public inbox for [email protected]
help / color / mirror / Atom feed[PATCH 1/2] review
97+ messages / 11 participants
[nested] [flat]
* [PATCH 1/2] review
@ 2021-01-09 02:17 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw)
---
doc/src/sgml/mvcc.sgml | 20 +++++++++++---------
doc/src/sgml/ref/create_policy.sgml | 6 +++---
doc/src/sgml/ref/merge.sgml | 12 +++++++++---
src/backend/commands/explain.c | 2 +-
src/backend/executor/execMerge.c | 5 ++++-
src/backend/executor/nodeModifyTable.c | 3 ++-
src/backend/optimizer/prep/preptlist.c | 6 ++++++
src/backend/parser/parse_agg.c | 1 +
src/backend/replication/walsender.c | 4 ++--
src/backend/rewrite/rewriteHandler.c | 4 ++++
src/backend/utils/adt/varlena.c | 3 +++
src/include/nodes/parsenodes.h | 7 ++++---
12 files changed, 50 insertions(+), 23 deletions(-)
diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml
index 9ec8474185..02c9f3fdea 100644
--- a/doc/src/sgml/mvcc.sgml
+++ b/doc/src/sgml/mvcc.sgml
@@ -429,22 +429,24 @@ COMMIT;
with both <command>INSERT</command> and <command>UPDATE</command>
subcommands looks similar to <command>INSERT</command> with an
<literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee
- that either <command>INSERT</command> and <command>UPDATE</command> will occur.
+ that either <command>INSERT</command> or <command>UPDATE</command> will occur.
- If MERGE attempts an UPDATE or DELETE and the row is concurrently updated
+ /* XXX This is a very long and hard to understand sentence :-( */
+ /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */
+ If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated
but the join condition still passes for the current target and the current
- source tuple, then MERGE will behave the same as the UPDATE or DELETE commands
+ source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands
and perform its action on the latest version of the row, using standard
- EvalPlanQual. MERGE actions can be conditional, so conditions must be
+ EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be
re-evaluated on the latest row, starting from the first action.
On the other hand, if the row is concurrently updated or deleted so that
- the join condition fails, then MERGE will execute a NOT MATCHED action, if it
- exists and the AND WHEN qual evaluates to true.
+ the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it
+ exists and the <literal>AND WHEN</literal> qual evaluates to true.
- If MERGE attempts an INSERT and a unique index is present and a duplicate
- row is concurrently inserted then a uniqueness violation is raised. MERGE
- does not attempt to avoid the ERROR by attempting an UPDATE.
+ If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate
+ row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command>
+ does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>.
</para>
<para>
diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml
index ad20230c58..0a64674f2d 100644
--- a/doc/src/sgml/ref/create_policy.sgml
+++ b/doc/src/sgml/ref/create_policy.sgml
@@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable
<para>
No separate policy exists for <command>MERGE</command>. Instead policies
- defined for <literal>SELECT</literal>, <literal>INSERT</literal>,
- <literal>UPDATE</literal> and <literal>DELETE</literal> are applied
- while executing MERGE, depending on the actions that are activated.
+ defined for <command>SELECT</command>, <command>INSERT</command>,
+ <command>UPDATE</command> and <command>DELETE</command> are applied
+ while executing <command>MERGE</command>, depending on the actions that are activated.
</para>
</refsect1>
diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml
index b2a9f67cfa..c2901b0d58 100644
--- a/doc/src/sgml/ref/merge.sgml
+++ b/doc/src/sgml/ref/merge.sgml
@@ -98,7 +98,7 @@ DELETE
</para>
<para>
- There is no MERGE privilege.
+ There is no <literal>MERGE</literal> privilege.
You must have the <literal>UPDATE</literal> privilege on the column(s)
of the <replaceable class="parameter">target_table_name</replaceable>
referred to in the <literal>SET</literal> clause
@@ -217,6 +217,7 @@ DELETE
At least one <literal>WHEN</literal> clause is required.
</para>
<para>
+ /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */
If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal>
and the candidate change row matches a row in the
<replaceable class="parameter">target_table_name</replaceable>
@@ -242,6 +243,7 @@ DELETE
clause will be activated and the corresponding action will occur for
that row. The expression may not contain functions that possibly performs
writes to the database.
+ /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */
</para>
<para>
A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns
@@ -379,6 +381,7 @@ DELETE
<para>
An expression to assign to the column. The expression can use the
old values of this and other columns in the table.
+ /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */
</para>
</listitem>
</varlistentry>
@@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable>
In summary, statement triggers for an event type (say, INSERT) will
be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level
triggers will fire only for the one event type <emphasis>activated</emphasis>.
+ /* XXX What does "activated" mean? Maybe "executed" would be better? */
So a <command>MERGE</command> might fire statement triggers for both
<command>UPDATE</command> and <command>INSERT</command>, even though only
<command>UPDATE</command> row triggers were fired.
@@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable>
rows will be used to modify the target row, later attempts to modify will
cause an error. This can also occur if row triggers make changes to the
target table which are then subsequently modified by <command>MERGE</command>.
+ /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */
+ /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */
If the repeated action is an <command>INSERT</command> this will
cause a uniqueness violation while a repeated <command>UPDATE</command> or
<command>DELETE</command> will cause a cardinality violation; the latter behavior
@@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable>
<title>Examples</title>
<para>
- Perform maintenance on CustomerAccounts based upon new Transactions.
+ Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>.
<programlisting>
MERGE CustomerAccount CA
@@ -599,7 +605,7 @@ WHEN MATCHED THEN
DELETE;
</programlisting>
- The wine_stock_changes table might be, for example, a temporary table
+ The <literal>wine_stock_changes</literal> table might be, for example, a temporary table
recently loaded into the database.
</para>
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index fbaf50c258..f914a00e9f 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors,
break;
case CMD_MERGE:
operation = "Merge";
- foperation = "Foreign Merge";
+ foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */
break;
default:
operation = "???";
diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c
index 0e245e1361..a7492a6c4b 100644
--- a/src/backend/executor/execMerge.c
+++ b/src/backend/executor/execMerge.c
@@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo,
* A concurrent update can:
*
* 1. modify the target tuple so that it no longer satisfies the
- * additional quals attached to the current WHEN MATCHED action OR
+ * additional quals attached to the current WHEN MATCHED action
*
* In this case, we are still dealing with a WHEN MATCHED case, but we
* should recheck the list of WHEN MATCHED actions and choose the first
@@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo,
* tuple, so we now instead find a qualifying WHEN NOT MATCHED action to
* execute.
*
+ * XXX Hmmm, what if the updated tuple would now match one that was
+ * considered NOT MATCHED so far?
+ *
* A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED.
*
* ExecMergeMatched takes care of following the update chain and
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index c0a97eba91..9c14709e47 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate)
{
/* advance to next subplan if any */
node->mt_whichplan++;
-
if (node->mt_whichplan < node->mt_nplans)
{
resultRelInfo++;
@@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate)
EvalPlanQualSetSlot(&node->mt_epqstate, planSlot);
slot = planSlot;
+ /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together
+ * with the other MT operations? This seems a bit out of place. */
if (operation == CMD_MERGE)
{
ExecMerge(node, resultRelInfo, estate, slot, junkfilter);
diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c
index 8848e9a03f..b2543f6814 100644
--- a/src/backend/optimizer/prep/preptlist.c
+++ b/src/backend/optimizer/prep/preptlist.c
@@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root)
tlist = expand_targetlist(tlist, command_type,
result_relation, target_relation);
+ /*
+ * For MERGE we need to handle the target list for the target relation,
+ * and also target list for each action (only INSERT/UPDATE matter).
+ */
if (command_type == CMD_MERGE)
{
ListCell *l;
@@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type,
* generate a NULL for dropped columns (we want to drop any old
* values).
*
+ * XXX Should this explain why MERGE has the same logic as UPDATE?
+ *
* When generating a NULL constant for a dropped column, we label
* it INT4 (any other guaranteed-to-exist datatype would do as
* well). We can't label it with the dropped column's datatype
diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c
index f7c152beb4..5d49c8c37e 100644
--- a/src/backend/parser/parse_agg.c
+++ b/src/backend/parser/parse_agg.c
@@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr)
break;
case EXPR_KIND_MERGE_WHEN_AND:
if (isAgg)
+ /* XXX "WHEN AND" seems rather strange. ... */
err = _("aggregate functions are not allowed in WHEN AND conditions");
else
err = _("grouping operations are not allowed in WHEN AND conditions");
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index d59f4fde95..d50543b1ba 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd)
/* Also update the sent position status in shared memory */
SpinLockAcquire(&MyWalSnd->mutex);
+ /* XXX Huh? Why does MERGE patch change walsender? */
MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush;
SpinLockRelease(&MyWalSnd->mutex);
@@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data)
replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ?
MyWalSnd->write : MyWalSnd->flush;
- if (WalSndCaughtUp &&
- sentPtr == replicatedPtr &&
+ if (WalSndCaughtUp && sentPtr == replicatedPtr &&
!pq_is_send_pending())
{
QueryCompletion qc;
diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c
index e2706c181c..80ae946d17 100644
--- a/src/backend/rewrite/rewriteHandler.c
+++ b/src/backend/rewrite/rewriteHandler.c
@@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation)
}
}
+
/*
* matchLocks -
* match the list of locks and returns the matching rules
@@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events)
/*
* XXX MERGE doesn't support write rules because they would violate
* the SQL Standard spec and would be unclear how they should work.
+ *
+ * XXX So does't support means 'ignores'? Should that either fail
+ * or at least print some warning?
*/
if (event == CMD_MERGE)
product_queries = NIL;
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c
index 0281351dcf..7a768a7b5b 100644
--- a/src/backend/utils/adt/varlena.c
+++ b/src/backend/utils/adt/varlena.c
@@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup)
int result;
bool arg1_match;
+ /* XXX Huh? Why is this in MERGE patch? */
if (len1 < 0 || len2 < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
@@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup)
memset(pres, 0, sizeof(Datum));
len = VARSIZE_ANY_EXHDR(authoritative);
+ /* XXX Huh? Why is this in MERGE patch? */
if (len < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
@@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2)
len1 = VARSIZE_ANY_EXHDR(t1);
len2 = VARSIZE_ANY_EXHDR(t2);
+ /* XXX Huh? Why is this in MERGE patch? */
if (len1 < 0 || len2 < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 3cba38044e..08fea9b8f7 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -172,6 +172,7 @@ typedef struct Query
List *withCheckOptions; /* a list of WithCheckOption's (added
* during rewrite) */
+ /* XXX Why not mergeTragetRelation? */
int mergeTarget_relation;
List *mergeSourceTargetList;
List *mergeActionList; /* list of actions for MERGE (only) */
@@ -1581,11 +1582,11 @@ typedef struct UpdateStmt
typedef struct MergeStmt
{
NodeTag type;
- RangeVar *relation; /* target relation to merge into */
+ RangeVar *relation; /* target relation to merge into */
Node *source_relation; /* source relation */
- Node *join_condition; /* join condition between source and target */
+ Node *join_condition; /* join condition between source and target */
List *mergeWhenClauses; /* list of MergeWhenClause(es) */
- WithClause *withClause; /* WITH clause */
+ WithClause *withClause; /* WITH clause */
} MergeStmt;
typedef struct MergeWhenClause
--
2.26.2
--------------B1AA4DC5EAE9BAF8B146244A
Content-Type: text/plain; charset=UTF-8;
name="0002-fix-valgrind-failure.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="0002-fix-valgrind-failure.txt"
^ permalink raw reply [nested|flat] 97+ messages in thread
* [PATCH 1/2] review
@ 2021-01-09 02:17 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw)
---
doc/src/sgml/mvcc.sgml | 20 +++++++++++---------
doc/src/sgml/ref/create_policy.sgml | 6 +++---
doc/src/sgml/ref/merge.sgml | 12 +++++++++---
src/backend/commands/explain.c | 2 +-
src/backend/executor/execMerge.c | 5 ++++-
src/backend/executor/nodeModifyTable.c | 3 ++-
src/backend/optimizer/prep/preptlist.c | 6 ++++++
src/backend/parser/parse_agg.c | 1 +
src/backend/replication/walsender.c | 4 ++--
src/backend/rewrite/rewriteHandler.c | 4 ++++
src/backend/utils/adt/varlena.c | 3 +++
src/include/nodes/parsenodes.h | 7 ++++---
12 files changed, 50 insertions(+), 23 deletions(-)
diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml
index 9ec8474185..02c9f3fdea 100644
--- a/doc/src/sgml/mvcc.sgml
+++ b/doc/src/sgml/mvcc.sgml
@@ -429,22 +429,24 @@ COMMIT;
with both <command>INSERT</command> and <command>UPDATE</command>
subcommands looks similar to <command>INSERT</command> with an
<literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee
- that either <command>INSERT</command> and <command>UPDATE</command> will occur.
+ that either <command>INSERT</command> or <command>UPDATE</command> will occur.
- If MERGE attempts an UPDATE or DELETE and the row is concurrently updated
+ /* XXX This is a very long and hard to understand sentence :-( */
+ /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */
+ If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated
but the join condition still passes for the current target and the current
- source tuple, then MERGE will behave the same as the UPDATE or DELETE commands
+ source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands
and perform its action on the latest version of the row, using standard
- EvalPlanQual. MERGE actions can be conditional, so conditions must be
+ EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be
re-evaluated on the latest row, starting from the first action.
On the other hand, if the row is concurrently updated or deleted so that
- the join condition fails, then MERGE will execute a NOT MATCHED action, if it
- exists and the AND WHEN qual evaluates to true.
+ the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it
+ exists and the <literal>AND WHEN</literal> qual evaluates to true.
- If MERGE attempts an INSERT and a unique index is present and a duplicate
- row is concurrently inserted then a uniqueness violation is raised. MERGE
- does not attempt to avoid the ERROR by attempting an UPDATE.
+ If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate
+ row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command>
+ does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>.
</para>
<para>
diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml
index ad20230c58..0a64674f2d 100644
--- a/doc/src/sgml/ref/create_policy.sgml
+++ b/doc/src/sgml/ref/create_policy.sgml
@@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable
<para>
No separate policy exists for <command>MERGE</command>. Instead policies
- defined for <literal>SELECT</literal>, <literal>INSERT</literal>,
- <literal>UPDATE</literal> and <literal>DELETE</literal> are applied
- while executing MERGE, depending on the actions that are activated.
+ defined for <command>SELECT</command>, <command>INSERT</command>,
+ <command>UPDATE</command> and <command>DELETE</command> are applied
+ while executing <command>MERGE</command>, depending on the actions that are activated.
</para>
</refsect1>
diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml
index b2a9f67cfa..c2901b0d58 100644
--- a/doc/src/sgml/ref/merge.sgml
+++ b/doc/src/sgml/ref/merge.sgml
@@ -98,7 +98,7 @@ DELETE
</para>
<para>
- There is no MERGE privilege.
+ There is no <literal>MERGE</literal> privilege.
You must have the <literal>UPDATE</literal> privilege on the column(s)
of the <replaceable class="parameter">target_table_name</replaceable>
referred to in the <literal>SET</literal> clause
@@ -217,6 +217,7 @@ DELETE
At least one <literal>WHEN</literal> clause is required.
</para>
<para>
+ /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */
If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal>
and the candidate change row matches a row in the
<replaceable class="parameter">target_table_name</replaceable>
@@ -242,6 +243,7 @@ DELETE
clause will be activated and the corresponding action will occur for
that row. The expression may not contain functions that possibly performs
writes to the database.
+ /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */
</para>
<para>
A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns
@@ -379,6 +381,7 @@ DELETE
<para>
An expression to assign to the column. The expression can use the
old values of this and other columns in the table.
+ /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */
</para>
</listitem>
</varlistentry>
@@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable>
In summary, statement triggers for an event type (say, INSERT) will
be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level
triggers will fire only for the one event type <emphasis>activated</emphasis>.
+ /* XXX What does "activated" mean? Maybe "executed" would be better? */
So a <command>MERGE</command> might fire statement triggers for both
<command>UPDATE</command> and <command>INSERT</command>, even though only
<command>UPDATE</command> row triggers were fired.
@@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable>
rows will be used to modify the target row, later attempts to modify will
cause an error. This can also occur if row triggers make changes to the
target table which are then subsequently modified by <command>MERGE</command>.
+ /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */
+ /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */
If the repeated action is an <command>INSERT</command> this will
cause a uniqueness violation while a repeated <command>UPDATE</command> or
<command>DELETE</command> will cause a cardinality violation; the latter behavior
@@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable>
<title>Examples</title>
<para>
- Perform maintenance on CustomerAccounts based upon new Transactions.
+ Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>.
<programlisting>
MERGE CustomerAccount CA
@@ -599,7 +605,7 @@ WHEN MATCHED THEN
DELETE;
</programlisting>
- The wine_stock_changes table might be, for example, a temporary table
+ The <literal>wine_stock_changes</literal> table might be, for example, a temporary table
recently loaded into the database.
</para>
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index fbaf50c258..f914a00e9f 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors,
break;
case CMD_MERGE:
operation = "Merge";
- foperation = "Foreign Merge";
+ foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */
break;
default:
operation = "???";
diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c
index 0e245e1361..a7492a6c4b 100644
--- a/src/backend/executor/execMerge.c
+++ b/src/backend/executor/execMerge.c
@@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo,
* A concurrent update can:
*
* 1. modify the target tuple so that it no longer satisfies the
- * additional quals attached to the current WHEN MATCHED action OR
+ * additional quals attached to the current WHEN MATCHED action
*
* In this case, we are still dealing with a WHEN MATCHED case, but we
* should recheck the list of WHEN MATCHED actions and choose the first
@@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo,
* tuple, so we now instead find a qualifying WHEN NOT MATCHED action to
* execute.
*
+ * XXX Hmmm, what if the updated tuple would now match one that was
+ * considered NOT MATCHED so far?
+ *
* A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED.
*
* ExecMergeMatched takes care of following the update chain and
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index c0a97eba91..9c14709e47 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate)
{
/* advance to next subplan if any */
node->mt_whichplan++;
-
if (node->mt_whichplan < node->mt_nplans)
{
resultRelInfo++;
@@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate)
EvalPlanQualSetSlot(&node->mt_epqstate, planSlot);
slot = planSlot;
+ /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together
+ * with the other MT operations? This seems a bit out of place. */
if (operation == CMD_MERGE)
{
ExecMerge(node, resultRelInfo, estate, slot, junkfilter);
diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c
index 8848e9a03f..b2543f6814 100644
--- a/src/backend/optimizer/prep/preptlist.c
+++ b/src/backend/optimizer/prep/preptlist.c
@@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root)
tlist = expand_targetlist(tlist, command_type,
result_relation, target_relation);
+ /*
+ * For MERGE we need to handle the target list for the target relation,
+ * and also target list for each action (only INSERT/UPDATE matter).
+ */
if (command_type == CMD_MERGE)
{
ListCell *l;
@@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type,
* generate a NULL for dropped columns (we want to drop any old
* values).
*
+ * XXX Should this explain why MERGE has the same logic as UPDATE?
+ *
* When generating a NULL constant for a dropped column, we label
* it INT4 (any other guaranteed-to-exist datatype would do as
* well). We can't label it with the dropped column's datatype
diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c
index f7c152beb4..5d49c8c37e 100644
--- a/src/backend/parser/parse_agg.c
+++ b/src/backend/parser/parse_agg.c
@@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr)
break;
case EXPR_KIND_MERGE_WHEN_AND:
if (isAgg)
+ /* XXX "WHEN AND" seems rather strange. ... */
err = _("aggregate functions are not allowed in WHEN AND conditions");
else
err = _("grouping operations are not allowed in WHEN AND conditions");
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index d59f4fde95..d50543b1ba 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd)
/* Also update the sent position status in shared memory */
SpinLockAcquire(&MyWalSnd->mutex);
+ /* XXX Huh? Why does MERGE patch change walsender? */
MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush;
SpinLockRelease(&MyWalSnd->mutex);
@@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data)
replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ?
MyWalSnd->write : MyWalSnd->flush;
- if (WalSndCaughtUp &&
- sentPtr == replicatedPtr &&
+ if (WalSndCaughtUp && sentPtr == replicatedPtr &&
!pq_is_send_pending())
{
QueryCompletion qc;
diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c
index e2706c181c..80ae946d17 100644
--- a/src/backend/rewrite/rewriteHandler.c
+++ b/src/backend/rewrite/rewriteHandler.c
@@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation)
}
}
+
/*
* matchLocks -
* match the list of locks and returns the matching rules
@@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events)
/*
* XXX MERGE doesn't support write rules because they would violate
* the SQL Standard spec and would be unclear how they should work.
+ *
+ * XXX So does't support means 'ignores'? Should that either fail
+ * or at least print some warning?
*/
if (event == CMD_MERGE)
product_queries = NIL;
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c
index 0281351dcf..7a768a7b5b 100644
--- a/src/backend/utils/adt/varlena.c
+++ b/src/backend/utils/adt/varlena.c
@@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup)
int result;
bool arg1_match;
+ /* XXX Huh? Why is this in MERGE patch? */
if (len1 < 0 || len2 < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
@@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup)
memset(pres, 0, sizeof(Datum));
len = VARSIZE_ANY_EXHDR(authoritative);
+ /* XXX Huh? Why is this in MERGE patch? */
if (len < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
@@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2)
len1 = VARSIZE_ANY_EXHDR(t1);
len2 = VARSIZE_ANY_EXHDR(t2);
+ /* XXX Huh? Why is this in MERGE patch? */
if (len1 < 0 || len2 < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 3cba38044e..08fea9b8f7 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -172,6 +172,7 @@ typedef struct Query
List *withCheckOptions; /* a list of WithCheckOption's (added
* during rewrite) */
+ /* XXX Why not mergeTragetRelation? */
int mergeTarget_relation;
List *mergeSourceTargetList;
List *mergeActionList; /* list of actions for MERGE (only) */
@@ -1581,11 +1582,11 @@ typedef struct UpdateStmt
typedef struct MergeStmt
{
NodeTag type;
- RangeVar *relation; /* target relation to merge into */
+ RangeVar *relation; /* target relation to merge into */
Node *source_relation; /* source relation */
- Node *join_condition; /* join condition between source and target */
+ Node *join_condition; /* join condition between source and target */
List *mergeWhenClauses; /* list of MergeWhenClause(es) */
- WithClause *withClause; /* WITH clause */
+ WithClause *withClause; /* WITH clause */
} MergeStmt;
typedef struct MergeWhenClause
--
2.26.2
--------------B1AA4DC5EAE9BAF8B146244A
Content-Type: text/plain; charset=UTF-8;
name="0002-fix-valgrind-failure.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="0002-fix-valgrind-failure.txt"
^ permalink raw reply [nested|flat] 97+ messages in thread
* [PATCH 1/2] review
@ 2021-01-09 02:17 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw)
---
doc/src/sgml/mvcc.sgml | 20 +++++++++++---------
doc/src/sgml/ref/create_policy.sgml | 6 +++---
doc/src/sgml/ref/merge.sgml | 12 +++++++++---
src/backend/commands/explain.c | 2 +-
src/backend/executor/execMerge.c | 5 ++++-
src/backend/executor/nodeModifyTable.c | 3 ++-
src/backend/optimizer/prep/preptlist.c | 6 ++++++
src/backend/parser/parse_agg.c | 1 +
src/backend/replication/walsender.c | 4 ++--
src/backend/rewrite/rewriteHandler.c | 4 ++++
src/backend/utils/adt/varlena.c | 3 +++
src/include/nodes/parsenodes.h | 7 ++++---
12 files changed, 50 insertions(+), 23 deletions(-)
diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml
index 9ec8474185..02c9f3fdea 100644
--- a/doc/src/sgml/mvcc.sgml
+++ b/doc/src/sgml/mvcc.sgml
@@ -429,22 +429,24 @@ COMMIT;
with both <command>INSERT</command> and <command>UPDATE</command>
subcommands looks similar to <command>INSERT</command> with an
<literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee
- that either <command>INSERT</command> and <command>UPDATE</command> will occur.
+ that either <command>INSERT</command> or <command>UPDATE</command> will occur.
- If MERGE attempts an UPDATE or DELETE and the row is concurrently updated
+ /* XXX This is a very long and hard to understand sentence :-( */
+ /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */
+ If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated
but the join condition still passes for the current target and the current
- source tuple, then MERGE will behave the same as the UPDATE or DELETE commands
+ source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands
and perform its action on the latest version of the row, using standard
- EvalPlanQual. MERGE actions can be conditional, so conditions must be
+ EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be
re-evaluated on the latest row, starting from the first action.
On the other hand, if the row is concurrently updated or deleted so that
- the join condition fails, then MERGE will execute a NOT MATCHED action, if it
- exists and the AND WHEN qual evaluates to true.
+ the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it
+ exists and the <literal>AND WHEN</literal> qual evaluates to true.
- If MERGE attempts an INSERT and a unique index is present and a duplicate
- row is concurrently inserted then a uniqueness violation is raised. MERGE
- does not attempt to avoid the ERROR by attempting an UPDATE.
+ If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate
+ row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command>
+ does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>.
</para>
<para>
diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml
index ad20230c58..0a64674f2d 100644
--- a/doc/src/sgml/ref/create_policy.sgml
+++ b/doc/src/sgml/ref/create_policy.sgml
@@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable
<para>
No separate policy exists for <command>MERGE</command>. Instead policies
- defined for <literal>SELECT</literal>, <literal>INSERT</literal>,
- <literal>UPDATE</literal> and <literal>DELETE</literal> are applied
- while executing MERGE, depending on the actions that are activated.
+ defined for <command>SELECT</command>, <command>INSERT</command>,
+ <command>UPDATE</command> and <command>DELETE</command> are applied
+ while executing <command>MERGE</command>, depending on the actions that are activated.
</para>
</refsect1>
diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml
index b2a9f67cfa..c2901b0d58 100644
--- a/doc/src/sgml/ref/merge.sgml
+++ b/doc/src/sgml/ref/merge.sgml
@@ -98,7 +98,7 @@ DELETE
</para>
<para>
- There is no MERGE privilege.
+ There is no <literal>MERGE</literal> privilege.
You must have the <literal>UPDATE</literal> privilege on the column(s)
of the <replaceable class="parameter">target_table_name</replaceable>
referred to in the <literal>SET</literal> clause
@@ -217,6 +217,7 @@ DELETE
At least one <literal>WHEN</literal> clause is required.
</para>
<para>
+ /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */
If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal>
and the candidate change row matches a row in the
<replaceable class="parameter">target_table_name</replaceable>
@@ -242,6 +243,7 @@ DELETE
clause will be activated and the corresponding action will occur for
that row. The expression may not contain functions that possibly performs
writes to the database.
+ /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */
</para>
<para>
A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns
@@ -379,6 +381,7 @@ DELETE
<para>
An expression to assign to the column. The expression can use the
old values of this and other columns in the table.
+ /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */
</para>
</listitem>
</varlistentry>
@@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable>
In summary, statement triggers for an event type (say, INSERT) will
be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level
triggers will fire only for the one event type <emphasis>activated</emphasis>.
+ /* XXX What does "activated" mean? Maybe "executed" would be better? */
So a <command>MERGE</command> might fire statement triggers for both
<command>UPDATE</command> and <command>INSERT</command>, even though only
<command>UPDATE</command> row triggers were fired.
@@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable>
rows will be used to modify the target row, later attempts to modify will
cause an error. This can also occur if row triggers make changes to the
target table which are then subsequently modified by <command>MERGE</command>.
+ /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */
+ /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */
If the repeated action is an <command>INSERT</command> this will
cause a uniqueness violation while a repeated <command>UPDATE</command> or
<command>DELETE</command> will cause a cardinality violation; the latter behavior
@@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable>
<title>Examples</title>
<para>
- Perform maintenance on CustomerAccounts based upon new Transactions.
+ Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>.
<programlisting>
MERGE CustomerAccount CA
@@ -599,7 +605,7 @@ WHEN MATCHED THEN
DELETE;
</programlisting>
- The wine_stock_changes table might be, for example, a temporary table
+ The <literal>wine_stock_changes</literal> table might be, for example, a temporary table
recently loaded into the database.
</para>
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index fbaf50c258..f914a00e9f 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors,
break;
case CMD_MERGE:
operation = "Merge";
- foperation = "Foreign Merge";
+ foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */
break;
default:
operation = "???";
diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c
index 0e245e1361..a7492a6c4b 100644
--- a/src/backend/executor/execMerge.c
+++ b/src/backend/executor/execMerge.c
@@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo,
* A concurrent update can:
*
* 1. modify the target tuple so that it no longer satisfies the
- * additional quals attached to the current WHEN MATCHED action OR
+ * additional quals attached to the current WHEN MATCHED action
*
* In this case, we are still dealing with a WHEN MATCHED case, but we
* should recheck the list of WHEN MATCHED actions and choose the first
@@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo,
* tuple, so we now instead find a qualifying WHEN NOT MATCHED action to
* execute.
*
+ * XXX Hmmm, what if the updated tuple would now match one that was
+ * considered NOT MATCHED so far?
+ *
* A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED.
*
* ExecMergeMatched takes care of following the update chain and
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index c0a97eba91..9c14709e47 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate)
{
/* advance to next subplan if any */
node->mt_whichplan++;
-
if (node->mt_whichplan < node->mt_nplans)
{
resultRelInfo++;
@@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate)
EvalPlanQualSetSlot(&node->mt_epqstate, planSlot);
slot = planSlot;
+ /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together
+ * with the other MT operations? This seems a bit out of place. */
if (operation == CMD_MERGE)
{
ExecMerge(node, resultRelInfo, estate, slot, junkfilter);
diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c
index 8848e9a03f..b2543f6814 100644
--- a/src/backend/optimizer/prep/preptlist.c
+++ b/src/backend/optimizer/prep/preptlist.c
@@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root)
tlist = expand_targetlist(tlist, command_type,
result_relation, target_relation);
+ /*
+ * For MERGE we need to handle the target list for the target relation,
+ * and also target list for each action (only INSERT/UPDATE matter).
+ */
if (command_type == CMD_MERGE)
{
ListCell *l;
@@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type,
* generate a NULL for dropped columns (we want to drop any old
* values).
*
+ * XXX Should this explain why MERGE has the same logic as UPDATE?
+ *
* When generating a NULL constant for a dropped column, we label
* it INT4 (any other guaranteed-to-exist datatype would do as
* well). We can't label it with the dropped column's datatype
diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c
index f7c152beb4..5d49c8c37e 100644
--- a/src/backend/parser/parse_agg.c
+++ b/src/backend/parser/parse_agg.c
@@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr)
break;
case EXPR_KIND_MERGE_WHEN_AND:
if (isAgg)
+ /* XXX "WHEN AND" seems rather strange. ... */
err = _("aggregate functions are not allowed in WHEN AND conditions");
else
err = _("grouping operations are not allowed in WHEN AND conditions");
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index d59f4fde95..d50543b1ba 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd)
/* Also update the sent position status in shared memory */
SpinLockAcquire(&MyWalSnd->mutex);
+ /* XXX Huh? Why does MERGE patch change walsender? */
MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush;
SpinLockRelease(&MyWalSnd->mutex);
@@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data)
replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ?
MyWalSnd->write : MyWalSnd->flush;
- if (WalSndCaughtUp &&
- sentPtr == replicatedPtr &&
+ if (WalSndCaughtUp && sentPtr == replicatedPtr &&
!pq_is_send_pending())
{
QueryCompletion qc;
diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c
index e2706c181c..80ae946d17 100644
--- a/src/backend/rewrite/rewriteHandler.c
+++ b/src/backend/rewrite/rewriteHandler.c
@@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation)
}
}
+
/*
* matchLocks -
* match the list of locks and returns the matching rules
@@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events)
/*
* XXX MERGE doesn't support write rules because they would violate
* the SQL Standard spec and would be unclear how they should work.
+ *
+ * XXX So does't support means 'ignores'? Should that either fail
+ * or at least print some warning?
*/
if (event == CMD_MERGE)
product_queries = NIL;
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c
index 0281351dcf..7a768a7b5b 100644
--- a/src/backend/utils/adt/varlena.c
+++ b/src/backend/utils/adt/varlena.c
@@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup)
int result;
bool arg1_match;
+ /* XXX Huh? Why is this in MERGE patch? */
if (len1 < 0 || len2 < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
@@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup)
memset(pres, 0, sizeof(Datum));
len = VARSIZE_ANY_EXHDR(authoritative);
+ /* XXX Huh? Why is this in MERGE patch? */
if (len < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
@@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2)
len1 = VARSIZE_ANY_EXHDR(t1);
len2 = VARSIZE_ANY_EXHDR(t2);
+ /* XXX Huh? Why is this in MERGE patch? */
if (len1 < 0 || len2 < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 3cba38044e..08fea9b8f7 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -172,6 +172,7 @@ typedef struct Query
List *withCheckOptions; /* a list of WithCheckOption's (added
* during rewrite) */
+ /* XXX Why not mergeTragetRelation? */
int mergeTarget_relation;
List *mergeSourceTargetList;
List *mergeActionList; /* list of actions for MERGE (only) */
@@ -1581,11 +1582,11 @@ typedef struct UpdateStmt
typedef struct MergeStmt
{
NodeTag type;
- RangeVar *relation; /* target relation to merge into */
+ RangeVar *relation; /* target relation to merge into */
Node *source_relation; /* source relation */
- Node *join_condition; /* join condition between source and target */
+ Node *join_condition; /* join condition between source and target */
List *mergeWhenClauses; /* list of MergeWhenClause(es) */
- WithClause *withClause; /* WITH clause */
+ WithClause *withClause; /* WITH clause */
} MergeStmt;
typedef struct MergeWhenClause
--
2.26.2
--------------B1AA4DC5EAE9BAF8B146244A
Content-Type: text/plain; charset=UTF-8;
name="0002-fix-valgrind-failure.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="0002-fix-valgrind-failure.txt"
^ permalink raw reply [nested|flat] 97+ messages in thread
* [PATCH 1/2] review
@ 2021-01-09 02:17 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw)
---
doc/src/sgml/mvcc.sgml | 20 +++++++++++---------
doc/src/sgml/ref/create_policy.sgml | 6 +++---
doc/src/sgml/ref/merge.sgml | 12 +++++++++---
src/backend/commands/explain.c | 2 +-
src/backend/executor/execMerge.c | 5 ++++-
src/backend/executor/nodeModifyTable.c | 3 ++-
src/backend/optimizer/prep/preptlist.c | 6 ++++++
src/backend/parser/parse_agg.c | 1 +
src/backend/replication/walsender.c | 4 ++--
src/backend/rewrite/rewriteHandler.c | 4 ++++
src/backend/utils/adt/varlena.c | 3 +++
src/include/nodes/parsenodes.h | 7 ++++---
12 files changed, 50 insertions(+), 23 deletions(-)
diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml
index 9ec8474185..02c9f3fdea 100644
--- a/doc/src/sgml/mvcc.sgml
+++ b/doc/src/sgml/mvcc.sgml
@@ -429,22 +429,24 @@ COMMIT;
with both <command>INSERT</command> and <command>UPDATE</command>
subcommands looks similar to <command>INSERT</command> with an
<literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee
- that either <command>INSERT</command> and <command>UPDATE</command> will occur.
+ that either <command>INSERT</command> or <command>UPDATE</command> will occur.
- If MERGE attempts an UPDATE or DELETE and the row is concurrently updated
+ /* XXX This is a very long and hard to understand sentence :-( */
+ /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */
+ If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated
but the join condition still passes for the current target and the current
- source tuple, then MERGE will behave the same as the UPDATE or DELETE commands
+ source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands
and perform its action on the latest version of the row, using standard
- EvalPlanQual. MERGE actions can be conditional, so conditions must be
+ EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be
re-evaluated on the latest row, starting from the first action.
On the other hand, if the row is concurrently updated or deleted so that
- the join condition fails, then MERGE will execute a NOT MATCHED action, if it
- exists and the AND WHEN qual evaluates to true.
+ the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it
+ exists and the <literal>AND WHEN</literal> qual evaluates to true.
- If MERGE attempts an INSERT and a unique index is present and a duplicate
- row is concurrently inserted then a uniqueness violation is raised. MERGE
- does not attempt to avoid the ERROR by attempting an UPDATE.
+ If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate
+ row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command>
+ does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>.
</para>
<para>
diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml
index ad20230c58..0a64674f2d 100644
--- a/doc/src/sgml/ref/create_policy.sgml
+++ b/doc/src/sgml/ref/create_policy.sgml
@@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable
<para>
No separate policy exists for <command>MERGE</command>. Instead policies
- defined for <literal>SELECT</literal>, <literal>INSERT</literal>,
- <literal>UPDATE</literal> and <literal>DELETE</literal> are applied
- while executing MERGE, depending on the actions that are activated.
+ defined for <command>SELECT</command>, <command>INSERT</command>,
+ <command>UPDATE</command> and <command>DELETE</command> are applied
+ while executing <command>MERGE</command>, depending on the actions that are activated.
</para>
</refsect1>
diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml
index b2a9f67cfa..c2901b0d58 100644
--- a/doc/src/sgml/ref/merge.sgml
+++ b/doc/src/sgml/ref/merge.sgml
@@ -98,7 +98,7 @@ DELETE
</para>
<para>
- There is no MERGE privilege.
+ There is no <literal>MERGE</literal> privilege.
You must have the <literal>UPDATE</literal> privilege on the column(s)
of the <replaceable class="parameter">target_table_name</replaceable>
referred to in the <literal>SET</literal> clause
@@ -217,6 +217,7 @@ DELETE
At least one <literal>WHEN</literal> clause is required.
</para>
<para>
+ /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */
If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal>
and the candidate change row matches a row in the
<replaceable class="parameter">target_table_name</replaceable>
@@ -242,6 +243,7 @@ DELETE
clause will be activated and the corresponding action will occur for
that row. The expression may not contain functions that possibly performs
writes to the database.
+ /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */
</para>
<para>
A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns
@@ -379,6 +381,7 @@ DELETE
<para>
An expression to assign to the column. The expression can use the
old values of this and other columns in the table.
+ /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */
</para>
</listitem>
</varlistentry>
@@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable>
In summary, statement triggers for an event type (say, INSERT) will
be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level
triggers will fire only for the one event type <emphasis>activated</emphasis>.
+ /* XXX What does "activated" mean? Maybe "executed" would be better? */
So a <command>MERGE</command> might fire statement triggers for both
<command>UPDATE</command> and <command>INSERT</command>, even though only
<command>UPDATE</command> row triggers were fired.
@@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable>
rows will be used to modify the target row, later attempts to modify will
cause an error. This can also occur if row triggers make changes to the
target table which are then subsequently modified by <command>MERGE</command>.
+ /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */
+ /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */
If the repeated action is an <command>INSERT</command> this will
cause a uniqueness violation while a repeated <command>UPDATE</command> or
<command>DELETE</command> will cause a cardinality violation; the latter behavior
@@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable>
<title>Examples</title>
<para>
- Perform maintenance on CustomerAccounts based upon new Transactions.
+ Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>.
<programlisting>
MERGE CustomerAccount CA
@@ -599,7 +605,7 @@ WHEN MATCHED THEN
DELETE;
</programlisting>
- The wine_stock_changes table might be, for example, a temporary table
+ The <literal>wine_stock_changes</literal> table might be, for example, a temporary table
recently loaded into the database.
</para>
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index fbaf50c258..f914a00e9f 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors,
break;
case CMD_MERGE:
operation = "Merge";
- foperation = "Foreign Merge";
+ foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */
break;
default:
operation = "???";
diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c
index 0e245e1361..a7492a6c4b 100644
--- a/src/backend/executor/execMerge.c
+++ b/src/backend/executor/execMerge.c
@@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo,
* A concurrent update can:
*
* 1. modify the target tuple so that it no longer satisfies the
- * additional quals attached to the current WHEN MATCHED action OR
+ * additional quals attached to the current WHEN MATCHED action
*
* In this case, we are still dealing with a WHEN MATCHED case, but we
* should recheck the list of WHEN MATCHED actions and choose the first
@@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo,
* tuple, so we now instead find a qualifying WHEN NOT MATCHED action to
* execute.
*
+ * XXX Hmmm, what if the updated tuple would now match one that was
+ * considered NOT MATCHED so far?
+ *
* A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED.
*
* ExecMergeMatched takes care of following the update chain and
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index c0a97eba91..9c14709e47 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate)
{
/* advance to next subplan if any */
node->mt_whichplan++;
-
if (node->mt_whichplan < node->mt_nplans)
{
resultRelInfo++;
@@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate)
EvalPlanQualSetSlot(&node->mt_epqstate, planSlot);
slot = planSlot;
+ /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together
+ * with the other MT operations? This seems a bit out of place. */
if (operation == CMD_MERGE)
{
ExecMerge(node, resultRelInfo, estate, slot, junkfilter);
diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c
index 8848e9a03f..b2543f6814 100644
--- a/src/backend/optimizer/prep/preptlist.c
+++ b/src/backend/optimizer/prep/preptlist.c
@@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root)
tlist = expand_targetlist(tlist, command_type,
result_relation, target_relation);
+ /*
+ * For MERGE we need to handle the target list for the target relation,
+ * and also target list for each action (only INSERT/UPDATE matter).
+ */
if (command_type == CMD_MERGE)
{
ListCell *l;
@@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type,
* generate a NULL for dropped columns (we want to drop any old
* values).
*
+ * XXX Should this explain why MERGE has the same logic as UPDATE?
+ *
* When generating a NULL constant for a dropped column, we label
* it INT4 (any other guaranteed-to-exist datatype would do as
* well). We can't label it with the dropped column's datatype
diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c
index f7c152beb4..5d49c8c37e 100644
--- a/src/backend/parser/parse_agg.c
+++ b/src/backend/parser/parse_agg.c
@@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr)
break;
case EXPR_KIND_MERGE_WHEN_AND:
if (isAgg)
+ /* XXX "WHEN AND" seems rather strange. ... */
err = _("aggregate functions are not allowed in WHEN AND conditions");
else
err = _("grouping operations are not allowed in WHEN AND conditions");
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index d59f4fde95..d50543b1ba 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd)
/* Also update the sent position status in shared memory */
SpinLockAcquire(&MyWalSnd->mutex);
+ /* XXX Huh? Why does MERGE patch change walsender? */
MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush;
SpinLockRelease(&MyWalSnd->mutex);
@@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data)
replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ?
MyWalSnd->write : MyWalSnd->flush;
- if (WalSndCaughtUp &&
- sentPtr == replicatedPtr &&
+ if (WalSndCaughtUp && sentPtr == replicatedPtr &&
!pq_is_send_pending())
{
QueryCompletion qc;
diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c
index e2706c181c..80ae946d17 100644
--- a/src/backend/rewrite/rewriteHandler.c
+++ b/src/backend/rewrite/rewriteHandler.c
@@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation)
}
}
+
/*
* matchLocks -
* match the list of locks and returns the matching rules
@@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events)
/*
* XXX MERGE doesn't support write rules because they would violate
* the SQL Standard spec and would be unclear how they should work.
+ *
+ * XXX So does't support means 'ignores'? Should that either fail
+ * or at least print some warning?
*/
if (event == CMD_MERGE)
product_queries = NIL;
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c
index 0281351dcf..7a768a7b5b 100644
--- a/src/backend/utils/adt/varlena.c
+++ b/src/backend/utils/adt/varlena.c
@@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup)
int result;
bool arg1_match;
+ /* XXX Huh? Why is this in MERGE patch? */
if (len1 < 0 || len2 < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
@@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup)
memset(pres, 0, sizeof(Datum));
len = VARSIZE_ANY_EXHDR(authoritative);
+ /* XXX Huh? Why is this in MERGE patch? */
if (len < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
@@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2)
len1 = VARSIZE_ANY_EXHDR(t1);
len2 = VARSIZE_ANY_EXHDR(t2);
+ /* XXX Huh? Why is this in MERGE patch? */
if (len1 < 0 || len2 < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 3cba38044e..08fea9b8f7 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -172,6 +172,7 @@ typedef struct Query
List *withCheckOptions; /* a list of WithCheckOption's (added
* during rewrite) */
+ /* XXX Why not mergeTragetRelation? */
int mergeTarget_relation;
List *mergeSourceTargetList;
List *mergeActionList; /* list of actions for MERGE (only) */
@@ -1581,11 +1582,11 @@ typedef struct UpdateStmt
typedef struct MergeStmt
{
NodeTag type;
- RangeVar *relation; /* target relation to merge into */
+ RangeVar *relation; /* target relation to merge into */
Node *source_relation; /* source relation */
- Node *join_condition; /* join condition between source and target */
+ Node *join_condition; /* join condition between source and target */
List *mergeWhenClauses; /* list of MergeWhenClause(es) */
- WithClause *withClause; /* WITH clause */
+ WithClause *withClause; /* WITH clause */
} MergeStmt;
typedef struct MergeWhenClause
--
2.26.2
--------------B1AA4DC5EAE9BAF8B146244A
Content-Type: text/plain; charset=UTF-8;
name="0002-fix-valgrind-failure.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="0002-fix-valgrind-failure.txt"
^ permalink raw reply [nested|flat] 97+ messages in thread
* [PATCH 1/2] review
@ 2021-01-09 02:17 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw)
---
doc/src/sgml/mvcc.sgml | 20 +++++++++++---------
doc/src/sgml/ref/create_policy.sgml | 6 +++---
doc/src/sgml/ref/merge.sgml | 12 +++++++++---
src/backend/commands/explain.c | 2 +-
src/backend/executor/execMerge.c | 5 ++++-
src/backend/executor/nodeModifyTable.c | 3 ++-
src/backend/optimizer/prep/preptlist.c | 6 ++++++
src/backend/parser/parse_agg.c | 1 +
src/backend/replication/walsender.c | 4 ++--
src/backend/rewrite/rewriteHandler.c | 4 ++++
src/backend/utils/adt/varlena.c | 3 +++
src/include/nodes/parsenodes.h | 7 ++++---
12 files changed, 50 insertions(+), 23 deletions(-)
diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml
index 9ec8474185..02c9f3fdea 100644
--- a/doc/src/sgml/mvcc.sgml
+++ b/doc/src/sgml/mvcc.sgml
@@ -429,22 +429,24 @@ COMMIT;
with both <command>INSERT</command> and <command>UPDATE</command>
subcommands looks similar to <command>INSERT</command> with an
<literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee
- that either <command>INSERT</command> and <command>UPDATE</command> will occur.
+ that either <command>INSERT</command> or <command>UPDATE</command> will occur.
- If MERGE attempts an UPDATE or DELETE and the row is concurrently updated
+ /* XXX This is a very long and hard to understand sentence :-( */
+ /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */
+ If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated
but the join condition still passes for the current target and the current
- source tuple, then MERGE will behave the same as the UPDATE or DELETE commands
+ source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands
and perform its action on the latest version of the row, using standard
- EvalPlanQual. MERGE actions can be conditional, so conditions must be
+ EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be
re-evaluated on the latest row, starting from the first action.
On the other hand, if the row is concurrently updated or deleted so that
- the join condition fails, then MERGE will execute a NOT MATCHED action, if it
- exists and the AND WHEN qual evaluates to true.
+ the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it
+ exists and the <literal>AND WHEN</literal> qual evaluates to true.
- If MERGE attempts an INSERT and a unique index is present and a duplicate
- row is concurrently inserted then a uniqueness violation is raised. MERGE
- does not attempt to avoid the ERROR by attempting an UPDATE.
+ If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate
+ row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command>
+ does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>.
</para>
<para>
diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml
index ad20230c58..0a64674f2d 100644
--- a/doc/src/sgml/ref/create_policy.sgml
+++ b/doc/src/sgml/ref/create_policy.sgml
@@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable
<para>
No separate policy exists for <command>MERGE</command>. Instead policies
- defined for <literal>SELECT</literal>, <literal>INSERT</literal>,
- <literal>UPDATE</literal> and <literal>DELETE</literal> are applied
- while executing MERGE, depending on the actions that are activated.
+ defined for <command>SELECT</command>, <command>INSERT</command>,
+ <command>UPDATE</command> and <command>DELETE</command> are applied
+ while executing <command>MERGE</command>, depending on the actions that are activated.
</para>
</refsect1>
diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml
index b2a9f67cfa..c2901b0d58 100644
--- a/doc/src/sgml/ref/merge.sgml
+++ b/doc/src/sgml/ref/merge.sgml
@@ -98,7 +98,7 @@ DELETE
</para>
<para>
- There is no MERGE privilege.
+ There is no <literal>MERGE</literal> privilege.
You must have the <literal>UPDATE</literal> privilege on the column(s)
of the <replaceable class="parameter">target_table_name</replaceable>
referred to in the <literal>SET</literal> clause
@@ -217,6 +217,7 @@ DELETE
At least one <literal>WHEN</literal> clause is required.
</para>
<para>
+ /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */
If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal>
and the candidate change row matches a row in the
<replaceable class="parameter">target_table_name</replaceable>
@@ -242,6 +243,7 @@ DELETE
clause will be activated and the corresponding action will occur for
that row. The expression may not contain functions that possibly performs
writes to the database.
+ /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */
</para>
<para>
A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns
@@ -379,6 +381,7 @@ DELETE
<para>
An expression to assign to the column. The expression can use the
old values of this and other columns in the table.
+ /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */
</para>
</listitem>
</varlistentry>
@@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable>
In summary, statement triggers for an event type (say, INSERT) will
be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level
triggers will fire only for the one event type <emphasis>activated</emphasis>.
+ /* XXX What does "activated" mean? Maybe "executed" would be better? */
So a <command>MERGE</command> might fire statement triggers for both
<command>UPDATE</command> and <command>INSERT</command>, even though only
<command>UPDATE</command> row triggers were fired.
@@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable>
rows will be used to modify the target row, later attempts to modify will
cause an error. This can also occur if row triggers make changes to the
target table which are then subsequently modified by <command>MERGE</command>.
+ /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */
+ /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */
If the repeated action is an <command>INSERT</command> this will
cause a uniqueness violation while a repeated <command>UPDATE</command> or
<command>DELETE</command> will cause a cardinality violation; the latter behavior
@@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable>
<title>Examples</title>
<para>
- Perform maintenance on CustomerAccounts based upon new Transactions.
+ Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>.
<programlisting>
MERGE CustomerAccount CA
@@ -599,7 +605,7 @@ WHEN MATCHED THEN
DELETE;
</programlisting>
- The wine_stock_changes table might be, for example, a temporary table
+ The <literal>wine_stock_changes</literal> table might be, for example, a temporary table
recently loaded into the database.
</para>
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index fbaf50c258..f914a00e9f 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors,
break;
case CMD_MERGE:
operation = "Merge";
- foperation = "Foreign Merge";
+ foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */
break;
default:
operation = "???";
diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c
index 0e245e1361..a7492a6c4b 100644
--- a/src/backend/executor/execMerge.c
+++ b/src/backend/executor/execMerge.c
@@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo,
* A concurrent update can:
*
* 1. modify the target tuple so that it no longer satisfies the
- * additional quals attached to the current WHEN MATCHED action OR
+ * additional quals attached to the current WHEN MATCHED action
*
* In this case, we are still dealing with a WHEN MATCHED case, but we
* should recheck the list of WHEN MATCHED actions and choose the first
@@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo,
* tuple, so we now instead find a qualifying WHEN NOT MATCHED action to
* execute.
*
+ * XXX Hmmm, what if the updated tuple would now match one that was
+ * considered NOT MATCHED so far?
+ *
* A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED.
*
* ExecMergeMatched takes care of following the update chain and
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index c0a97eba91..9c14709e47 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate)
{
/* advance to next subplan if any */
node->mt_whichplan++;
-
if (node->mt_whichplan < node->mt_nplans)
{
resultRelInfo++;
@@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate)
EvalPlanQualSetSlot(&node->mt_epqstate, planSlot);
slot = planSlot;
+ /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together
+ * with the other MT operations? This seems a bit out of place. */
if (operation == CMD_MERGE)
{
ExecMerge(node, resultRelInfo, estate, slot, junkfilter);
diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c
index 8848e9a03f..b2543f6814 100644
--- a/src/backend/optimizer/prep/preptlist.c
+++ b/src/backend/optimizer/prep/preptlist.c
@@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root)
tlist = expand_targetlist(tlist, command_type,
result_relation, target_relation);
+ /*
+ * For MERGE we need to handle the target list for the target relation,
+ * and also target list for each action (only INSERT/UPDATE matter).
+ */
if (command_type == CMD_MERGE)
{
ListCell *l;
@@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type,
* generate a NULL for dropped columns (we want to drop any old
* values).
*
+ * XXX Should this explain why MERGE has the same logic as UPDATE?
+ *
* When generating a NULL constant for a dropped column, we label
* it INT4 (any other guaranteed-to-exist datatype would do as
* well). We can't label it with the dropped column's datatype
diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c
index f7c152beb4..5d49c8c37e 100644
--- a/src/backend/parser/parse_agg.c
+++ b/src/backend/parser/parse_agg.c
@@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr)
break;
case EXPR_KIND_MERGE_WHEN_AND:
if (isAgg)
+ /* XXX "WHEN AND" seems rather strange. ... */
err = _("aggregate functions are not allowed in WHEN AND conditions");
else
err = _("grouping operations are not allowed in WHEN AND conditions");
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index d59f4fde95..d50543b1ba 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd)
/* Also update the sent position status in shared memory */
SpinLockAcquire(&MyWalSnd->mutex);
+ /* XXX Huh? Why does MERGE patch change walsender? */
MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush;
SpinLockRelease(&MyWalSnd->mutex);
@@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data)
replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ?
MyWalSnd->write : MyWalSnd->flush;
- if (WalSndCaughtUp &&
- sentPtr == replicatedPtr &&
+ if (WalSndCaughtUp && sentPtr == replicatedPtr &&
!pq_is_send_pending())
{
QueryCompletion qc;
diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c
index e2706c181c..80ae946d17 100644
--- a/src/backend/rewrite/rewriteHandler.c
+++ b/src/backend/rewrite/rewriteHandler.c
@@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation)
}
}
+
/*
* matchLocks -
* match the list of locks and returns the matching rules
@@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events)
/*
* XXX MERGE doesn't support write rules because they would violate
* the SQL Standard spec and would be unclear how they should work.
+ *
+ * XXX So does't support means 'ignores'? Should that either fail
+ * or at least print some warning?
*/
if (event == CMD_MERGE)
product_queries = NIL;
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c
index 0281351dcf..7a768a7b5b 100644
--- a/src/backend/utils/adt/varlena.c
+++ b/src/backend/utils/adt/varlena.c
@@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup)
int result;
bool arg1_match;
+ /* XXX Huh? Why is this in MERGE patch? */
if (len1 < 0 || len2 < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
@@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup)
memset(pres, 0, sizeof(Datum));
len = VARSIZE_ANY_EXHDR(authoritative);
+ /* XXX Huh? Why is this in MERGE patch? */
if (len < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
@@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2)
len1 = VARSIZE_ANY_EXHDR(t1);
len2 = VARSIZE_ANY_EXHDR(t2);
+ /* XXX Huh? Why is this in MERGE patch? */
if (len1 < 0 || len2 < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 3cba38044e..08fea9b8f7 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -172,6 +172,7 @@ typedef struct Query
List *withCheckOptions; /* a list of WithCheckOption's (added
* during rewrite) */
+ /* XXX Why not mergeTragetRelation? */
int mergeTarget_relation;
List *mergeSourceTargetList;
List *mergeActionList; /* list of actions for MERGE (only) */
@@ -1581,11 +1582,11 @@ typedef struct UpdateStmt
typedef struct MergeStmt
{
NodeTag type;
- RangeVar *relation; /* target relation to merge into */
+ RangeVar *relation; /* target relation to merge into */
Node *source_relation; /* source relation */
- Node *join_condition; /* join condition between source and target */
+ Node *join_condition; /* join condition between source and target */
List *mergeWhenClauses; /* list of MergeWhenClause(es) */
- WithClause *withClause; /* WITH clause */
+ WithClause *withClause; /* WITH clause */
} MergeStmt;
typedef struct MergeWhenClause
--
2.26.2
--------------B1AA4DC5EAE9BAF8B146244A
Content-Type: text/plain; charset=UTF-8;
name="0002-fix-valgrind-failure.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="0002-fix-valgrind-failure.txt"
^ permalink raw reply [nested|flat] 97+ messages in thread
* [PATCH 1/2] review
@ 2021-01-09 02:17 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw)
---
doc/src/sgml/mvcc.sgml | 20 +++++++++++---------
doc/src/sgml/ref/create_policy.sgml | 6 +++---
doc/src/sgml/ref/merge.sgml | 12 +++++++++---
src/backend/commands/explain.c | 2 +-
src/backend/executor/execMerge.c | 5 ++++-
src/backend/executor/nodeModifyTable.c | 3 ++-
src/backend/optimizer/prep/preptlist.c | 6 ++++++
src/backend/parser/parse_agg.c | 1 +
src/backend/replication/walsender.c | 4 ++--
src/backend/rewrite/rewriteHandler.c | 4 ++++
src/backend/utils/adt/varlena.c | 3 +++
src/include/nodes/parsenodes.h | 7 ++++---
12 files changed, 50 insertions(+), 23 deletions(-)
diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml
index 9ec8474185..02c9f3fdea 100644
--- a/doc/src/sgml/mvcc.sgml
+++ b/doc/src/sgml/mvcc.sgml
@@ -429,22 +429,24 @@ COMMIT;
with both <command>INSERT</command> and <command>UPDATE</command>
subcommands looks similar to <command>INSERT</command> with an
<literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee
- that either <command>INSERT</command> and <command>UPDATE</command> will occur.
+ that either <command>INSERT</command> or <command>UPDATE</command> will occur.
- If MERGE attempts an UPDATE or DELETE and the row is concurrently updated
+ /* XXX This is a very long and hard to understand sentence :-( */
+ /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */
+ If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated
but the join condition still passes for the current target and the current
- source tuple, then MERGE will behave the same as the UPDATE or DELETE commands
+ source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands
and perform its action on the latest version of the row, using standard
- EvalPlanQual. MERGE actions can be conditional, so conditions must be
+ EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be
re-evaluated on the latest row, starting from the first action.
On the other hand, if the row is concurrently updated or deleted so that
- the join condition fails, then MERGE will execute a NOT MATCHED action, if it
- exists and the AND WHEN qual evaluates to true.
+ the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it
+ exists and the <literal>AND WHEN</literal> qual evaluates to true.
- If MERGE attempts an INSERT and a unique index is present and a duplicate
- row is concurrently inserted then a uniqueness violation is raised. MERGE
- does not attempt to avoid the ERROR by attempting an UPDATE.
+ If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate
+ row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command>
+ does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>.
</para>
<para>
diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml
index ad20230c58..0a64674f2d 100644
--- a/doc/src/sgml/ref/create_policy.sgml
+++ b/doc/src/sgml/ref/create_policy.sgml
@@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable
<para>
No separate policy exists for <command>MERGE</command>. Instead policies
- defined for <literal>SELECT</literal>, <literal>INSERT</literal>,
- <literal>UPDATE</literal> and <literal>DELETE</literal> are applied
- while executing MERGE, depending on the actions that are activated.
+ defined for <command>SELECT</command>, <command>INSERT</command>,
+ <command>UPDATE</command> and <command>DELETE</command> are applied
+ while executing <command>MERGE</command>, depending on the actions that are activated.
</para>
</refsect1>
diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml
index b2a9f67cfa..c2901b0d58 100644
--- a/doc/src/sgml/ref/merge.sgml
+++ b/doc/src/sgml/ref/merge.sgml
@@ -98,7 +98,7 @@ DELETE
</para>
<para>
- There is no MERGE privilege.
+ There is no <literal>MERGE</literal> privilege.
You must have the <literal>UPDATE</literal> privilege on the column(s)
of the <replaceable class="parameter">target_table_name</replaceable>
referred to in the <literal>SET</literal> clause
@@ -217,6 +217,7 @@ DELETE
At least one <literal>WHEN</literal> clause is required.
</para>
<para>
+ /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */
If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal>
and the candidate change row matches a row in the
<replaceable class="parameter">target_table_name</replaceable>
@@ -242,6 +243,7 @@ DELETE
clause will be activated and the corresponding action will occur for
that row. The expression may not contain functions that possibly performs
writes to the database.
+ /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */
</para>
<para>
A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns
@@ -379,6 +381,7 @@ DELETE
<para>
An expression to assign to the column. The expression can use the
old values of this and other columns in the table.
+ /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */
</para>
</listitem>
</varlistentry>
@@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable>
In summary, statement triggers for an event type (say, INSERT) will
be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level
triggers will fire only for the one event type <emphasis>activated</emphasis>.
+ /* XXX What does "activated" mean? Maybe "executed" would be better? */
So a <command>MERGE</command> might fire statement triggers for both
<command>UPDATE</command> and <command>INSERT</command>, even though only
<command>UPDATE</command> row triggers were fired.
@@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable>
rows will be used to modify the target row, later attempts to modify will
cause an error. This can also occur if row triggers make changes to the
target table which are then subsequently modified by <command>MERGE</command>.
+ /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */
+ /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */
If the repeated action is an <command>INSERT</command> this will
cause a uniqueness violation while a repeated <command>UPDATE</command> or
<command>DELETE</command> will cause a cardinality violation; the latter behavior
@@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable>
<title>Examples</title>
<para>
- Perform maintenance on CustomerAccounts based upon new Transactions.
+ Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>.
<programlisting>
MERGE CustomerAccount CA
@@ -599,7 +605,7 @@ WHEN MATCHED THEN
DELETE;
</programlisting>
- The wine_stock_changes table might be, for example, a temporary table
+ The <literal>wine_stock_changes</literal> table might be, for example, a temporary table
recently loaded into the database.
</para>
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index fbaf50c258..f914a00e9f 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors,
break;
case CMD_MERGE:
operation = "Merge";
- foperation = "Foreign Merge";
+ foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */
break;
default:
operation = "???";
diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c
index 0e245e1361..a7492a6c4b 100644
--- a/src/backend/executor/execMerge.c
+++ b/src/backend/executor/execMerge.c
@@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo,
* A concurrent update can:
*
* 1. modify the target tuple so that it no longer satisfies the
- * additional quals attached to the current WHEN MATCHED action OR
+ * additional quals attached to the current WHEN MATCHED action
*
* In this case, we are still dealing with a WHEN MATCHED case, but we
* should recheck the list of WHEN MATCHED actions and choose the first
@@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo,
* tuple, so we now instead find a qualifying WHEN NOT MATCHED action to
* execute.
*
+ * XXX Hmmm, what if the updated tuple would now match one that was
+ * considered NOT MATCHED so far?
+ *
* A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED.
*
* ExecMergeMatched takes care of following the update chain and
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index c0a97eba91..9c14709e47 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate)
{
/* advance to next subplan if any */
node->mt_whichplan++;
-
if (node->mt_whichplan < node->mt_nplans)
{
resultRelInfo++;
@@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate)
EvalPlanQualSetSlot(&node->mt_epqstate, planSlot);
slot = planSlot;
+ /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together
+ * with the other MT operations? This seems a bit out of place. */
if (operation == CMD_MERGE)
{
ExecMerge(node, resultRelInfo, estate, slot, junkfilter);
diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c
index 8848e9a03f..b2543f6814 100644
--- a/src/backend/optimizer/prep/preptlist.c
+++ b/src/backend/optimizer/prep/preptlist.c
@@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root)
tlist = expand_targetlist(tlist, command_type,
result_relation, target_relation);
+ /*
+ * For MERGE we need to handle the target list for the target relation,
+ * and also target list for each action (only INSERT/UPDATE matter).
+ */
if (command_type == CMD_MERGE)
{
ListCell *l;
@@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type,
* generate a NULL for dropped columns (we want to drop any old
* values).
*
+ * XXX Should this explain why MERGE has the same logic as UPDATE?
+ *
* When generating a NULL constant for a dropped column, we label
* it INT4 (any other guaranteed-to-exist datatype would do as
* well). We can't label it with the dropped column's datatype
diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c
index f7c152beb4..5d49c8c37e 100644
--- a/src/backend/parser/parse_agg.c
+++ b/src/backend/parser/parse_agg.c
@@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr)
break;
case EXPR_KIND_MERGE_WHEN_AND:
if (isAgg)
+ /* XXX "WHEN AND" seems rather strange. ... */
err = _("aggregate functions are not allowed in WHEN AND conditions");
else
err = _("grouping operations are not allowed in WHEN AND conditions");
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index d59f4fde95..d50543b1ba 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd)
/* Also update the sent position status in shared memory */
SpinLockAcquire(&MyWalSnd->mutex);
+ /* XXX Huh? Why does MERGE patch change walsender? */
MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush;
SpinLockRelease(&MyWalSnd->mutex);
@@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data)
replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ?
MyWalSnd->write : MyWalSnd->flush;
- if (WalSndCaughtUp &&
- sentPtr == replicatedPtr &&
+ if (WalSndCaughtUp && sentPtr == replicatedPtr &&
!pq_is_send_pending())
{
QueryCompletion qc;
diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c
index e2706c181c..80ae946d17 100644
--- a/src/backend/rewrite/rewriteHandler.c
+++ b/src/backend/rewrite/rewriteHandler.c
@@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation)
}
}
+
/*
* matchLocks -
* match the list of locks and returns the matching rules
@@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events)
/*
* XXX MERGE doesn't support write rules because they would violate
* the SQL Standard spec and would be unclear how they should work.
+ *
+ * XXX So does't support means 'ignores'? Should that either fail
+ * or at least print some warning?
*/
if (event == CMD_MERGE)
product_queries = NIL;
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c
index 0281351dcf..7a768a7b5b 100644
--- a/src/backend/utils/adt/varlena.c
+++ b/src/backend/utils/adt/varlena.c
@@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup)
int result;
bool arg1_match;
+ /* XXX Huh? Why is this in MERGE patch? */
if (len1 < 0 || len2 < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
@@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup)
memset(pres, 0, sizeof(Datum));
len = VARSIZE_ANY_EXHDR(authoritative);
+ /* XXX Huh? Why is this in MERGE patch? */
if (len < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
@@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2)
len1 = VARSIZE_ANY_EXHDR(t1);
len2 = VARSIZE_ANY_EXHDR(t2);
+ /* XXX Huh? Why is this in MERGE patch? */
if (len1 < 0 || len2 < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 3cba38044e..08fea9b8f7 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -172,6 +172,7 @@ typedef struct Query
List *withCheckOptions; /* a list of WithCheckOption's (added
* during rewrite) */
+ /* XXX Why not mergeTragetRelation? */
int mergeTarget_relation;
List *mergeSourceTargetList;
List *mergeActionList; /* list of actions for MERGE (only) */
@@ -1581,11 +1582,11 @@ typedef struct UpdateStmt
typedef struct MergeStmt
{
NodeTag type;
- RangeVar *relation; /* target relation to merge into */
+ RangeVar *relation; /* target relation to merge into */
Node *source_relation; /* source relation */
- Node *join_condition; /* join condition between source and target */
+ Node *join_condition; /* join condition between source and target */
List *mergeWhenClauses; /* list of MergeWhenClause(es) */
- WithClause *withClause; /* WITH clause */
+ WithClause *withClause; /* WITH clause */
} MergeStmt;
typedef struct MergeWhenClause
--
2.26.2
--------------B1AA4DC5EAE9BAF8B146244A
Content-Type: text/plain; charset=UTF-8;
name="0002-fix-valgrind-failure.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="0002-fix-valgrind-failure.txt"
^ permalink raw reply [nested|flat] 97+ messages in thread
* [PATCH 1/2] review
@ 2021-01-09 02:17 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw)
---
doc/src/sgml/mvcc.sgml | 20 +++++++++++---------
doc/src/sgml/ref/create_policy.sgml | 6 +++---
doc/src/sgml/ref/merge.sgml | 12 +++++++++---
src/backend/commands/explain.c | 2 +-
src/backend/executor/execMerge.c | 5 ++++-
src/backend/executor/nodeModifyTable.c | 3 ++-
src/backend/optimizer/prep/preptlist.c | 6 ++++++
src/backend/parser/parse_agg.c | 1 +
src/backend/replication/walsender.c | 4 ++--
src/backend/rewrite/rewriteHandler.c | 4 ++++
src/backend/utils/adt/varlena.c | 3 +++
src/include/nodes/parsenodes.h | 7 ++++---
12 files changed, 50 insertions(+), 23 deletions(-)
diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml
index 9ec8474185..02c9f3fdea 100644
--- a/doc/src/sgml/mvcc.sgml
+++ b/doc/src/sgml/mvcc.sgml
@@ -429,22 +429,24 @@ COMMIT;
with both <command>INSERT</command> and <command>UPDATE</command>
subcommands looks similar to <command>INSERT</command> with an
<literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee
- that either <command>INSERT</command> and <command>UPDATE</command> will occur.
+ that either <command>INSERT</command> or <command>UPDATE</command> will occur.
- If MERGE attempts an UPDATE or DELETE and the row is concurrently updated
+ /* XXX This is a very long and hard to understand sentence :-( */
+ /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */
+ If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated
but the join condition still passes for the current target and the current
- source tuple, then MERGE will behave the same as the UPDATE or DELETE commands
+ source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands
and perform its action on the latest version of the row, using standard
- EvalPlanQual. MERGE actions can be conditional, so conditions must be
+ EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be
re-evaluated on the latest row, starting from the first action.
On the other hand, if the row is concurrently updated or deleted so that
- the join condition fails, then MERGE will execute a NOT MATCHED action, if it
- exists and the AND WHEN qual evaluates to true.
+ the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it
+ exists and the <literal>AND WHEN</literal> qual evaluates to true.
- If MERGE attempts an INSERT and a unique index is present and a duplicate
- row is concurrently inserted then a uniqueness violation is raised. MERGE
- does not attempt to avoid the ERROR by attempting an UPDATE.
+ If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate
+ row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command>
+ does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>.
</para>
<para>
diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml
index ad20230c58..0a64674f2d 100644
--- a/doc/src/sgml/ref/create_policy.sgml
+++ b/doc/src/sgml/ref/create_policy.sgml
@@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable
<para>
No separate policy exists for <command>MERGE</command>. Instead policies
- defined for <literal>SELECT</literal>, <literal>INSERT</literal>,
- <literal>UPDATE</literal> and <literal>DELETE</literal> are applied
- while executing MERGE, depending on the actions that are activated.
+ defined for <command>SELECT</command>, <command>INSERT</command>,
+ <command>UPDATE</command> and <command>DELETE</command> are applied
+ while executing <command>MERGE</command>, depending on the actions that are activated.
</para>
</refsect1>
diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml
index b2a9f67cfa..c2901b0d58 100644
--- a/doc/src/sgml/ref/merge.sgml
+++ b/doc/src/sgml/ref/merge.sgml
@@ -98,7 +98,7 @@ DELETE
</para>
<para>
- There is no MERGE privilege.
+ There is no <literal>MERGE</literal> privilege.
You must have the <literal>UPDATE</literal> privilege on the column(s)
of the <replaceable class="parameter">target_table_name</replaceable>
referred to in the <literal>SET</literal> clause
@@ -217,6 +217,7 @@ DELETE
At least one <literal>WHEN</literal> clause is required.
</para>
<para>
+ /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */
If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal>
and the candidate change row matches a row in the
<replaceable class="parameter">target_table_name</replaceable>
@@ -242,6 +243,7 @@ DELETE
clause will be activated and the corresponding action will occur for
that row. The expression may not contain functions that possibly performs
writes to the database.
+ /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */
</para>
<para>
A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns
@@ -379,6 +381,7 @@ DELETE
<para>
An expression to assign to the column. The expression can use the
old values of this and other columns in the table.
+ /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */
</para>
</listitem>
</varlistentry>
@@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable>
In summary, statement triggers for an event type (say, INSERT) will
be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level
triggers will fire only for the one event type <emphasis>activated</emphasis>.
+ /* XXX What does "activated" mean? Maybe "executed" would be better? */
So a <command>MERGE</command> might fire statement triggers for both
<command>UPDATE</command> and <command>INSERT</command>, even though only
<command>UPDATE</command> row triggers were fired.
@@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable>
rows will be used to modify the target row, later attempts to modify will
cause an error. This can also occur if row triggers make changes to the
target table which are then subsequently modified by <command>MERGE</command>.
+ /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */
+ /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */
If the repeated action is an <command>INSERT</command> this will
cause a uniqueness violation while a repeated <command>UPDATE</command> or
<command>DELETE</command> will cause a cardinality violation; the latter behavior
@@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable>
<title>Examples</title>
<para>
- Perform maintenance on CustomerAccounts based upon new Transactions.
+ Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>.
<programlisting>
MERGE CustomerAccount CA
@@ -599,7 +605,7 @@ WHEN MATCHED THEN
DELETE;
</programlisting>
- The wine_stock_changes table might be, for example, a temporary table
+ The <literal>wine_stock_changes</literal> table might be, for example, a temporary table
recently loaded into the database.
</para>
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index fbaf50c258..f914a00e9f 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors,
break;
case CMD_MERGE:
operation = "Merge";
- foperation = "Foreign Merge";
+ foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */
break;
default:
operation = "???";
diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c
index 0e245e1361..a7492a6c4b 100644
--- a/src/backend/executor/execMerge.c
+++ b/src/backend/executor/execMerge.c
@@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo,
* A concurrent update can:
*
* 1. modify the target tuple so that it no longer satisfies the
- * additional quals attached to the current WHEN MATCHED action OR
+ * additional quals attached to the current WHEN MATCHED action
*
* In this case, we are still dealing with a WHEN MATCHED case, but we
* should recheck the list of WHEN MATCHED actions and choose the first
@@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo,
* tuple, so we now instead find a qualifying WHEN NOT MATCHED action to
* execute.
*
+ * XXX Hmmm, what if the updated tuple would now match one that was
+ * considered NOT MATCHED so far?
+ *
* A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED.
*
* ExecMergeMatched takes care of following the update chain and
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index c0a97eba91..9c14709e47 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate)
{
/* advance to next subplan if any */
node->mt_whichplan++;
-
if (node->mt_whichplan < node->mt_nplans)
{
resultRelInfo++;
@@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate)
EvalPlanQualSetSlot(&node->mt_epqstate, planSlot);
slot = planSlot;
+ /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together
+ * with the other MT operations? This seems a bit out of place. */
if (operation == CMD_MERGE)
{
ExecMerge(node, resultRelInfo, estate, slot, junkfilter);
diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c
index 8848e9a03f..b2543f6814 100644
--- a/src/backend/optimizer/prep/preptlist.c
+++ b/src/backend/optimizer/prep/preptlist.c
@@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root)
tlist = expand_targetlist(tlist, command_type,
result_relation, target_relation);
+ /*
+ * For MERGE we need to handle the target list for the target relation,
+ * and also target list for each action (only INSERT/UPDATE matter).
+ */
if (command_type == CMD_MERGE)
{
ListCell *l;
@@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type,
* generate a NULL for dropped columns (we want to drop any old
* values).
*
+ * XXX Should this explain why MERGE has the same logic as UPDATE?
+ *
* When generating a NULL constant for a dropped column, we label
* it INT4 (any other guaranteed-to-exist datatype would do as
* well). We can't label it with the dropped column's datatype
diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c
index f7c152beb4..5d49c8c37e 100644
--- a/src/backend/parser/parse_agg.c
+++ b/src/backend/parser/parse_agg.c
@@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr)
break;
case EXPR_KIND_MERGE_WHEN_AND:
if (isAgg)
+ /* XXX "WHEN AND" seems rather strange. ... */
err = _("aggregate functions are not allowed in WHEN AND conditions");
else
err = _("grouping operations are not allowed in WHEN AND conditions");
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index d59f4fde95..d50543b1ba 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd)
/* Also update the sent position status in shared memory */
SpinLockAcquire(&MyWalSnd->mutex);
+ /* XXX Huh? Why does MERGE patch change walsender? */
MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush;
SpinLockRelease(&MyWalSnd->mutex);
@@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data)
replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ?
MyWalSnd->write : MyWalSnd->flush;
- if (WalSndCaughtUp &&
- sentPtr == replicatedPtr &&
+ if (WalSndCaughtUp && sentPtr == replicatedPtr &&
!pq_is_send_pending())
{
QueryCompletion qc;
diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c
index e2706c181c..80ae946d17 100644
--- a/src/backend/rewrite/rewriteHandler.c
+++ b/src/backend/rewrite/rewriteHandler.c
@@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation)
}
}
+
/*
* matchLocks -
* match the list of locks and returns the matching rules
@@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events)
/*
* XXX MERGE doesn't support write rules because they would violate
* the SQL Standard spec and would be unclear how they should work.
+ *
+ * XXX So does't support means 'ignores'? Should that either fail
+ * or at least print some warning?
*/
if (event == CMD_MERGE)
product_queries = NIL;
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c
index 0281351dcf..7a768a7b5b 100644
--- a/src/backend/utils/adt/varlena.c
+++ b/src/backend/utils/adt/varlena.c
@@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup)
int result;
bool arg1_match;
+ /* XXX Huh? Why is this in MERGE patch? */
if (len1 < 0 || len2 < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
@@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup)
memset(pres, 0, sizeof(Datum));
len = VARSIZE_ANY_EXHDR(authoritative);
+ /* XXX Huh? Why is this in MERGE patch? */
if (len < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
@@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2)
len1 = VARSIZE_ANY_EXHDR(t1);
len2 = VARSIZE_ANY_EXHDR(t2);
+ /* XXX Huh? Why is this in MERGE patch? */
if (len1 < 0 || len2 < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 3cba38044e..08fea9b8f7 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -172,6 +172,7 @@ typedef struct Query
List *withCheckOptions; /* a list of WithCheckOption's (added
* during rewrite) */
+ /* XXX Why not mergeTragetRelation? */
int mergeTarget_relation;
List *mergeSourceTargetList;
List *mergeActionList; /* list of actions for MERGE (only) */
@@ -1581,11 +1582,11 @@ typedef struct UpdateStmt
typedef struct MergeStmt
{
NodeTag type;
- RangeVar *relation; /* target relation to merge into */
+ RangeVar *relation; /* target relation to merge into */
Node *source_relation; /* source relation */
- Node *join_condition; /* join condition between source and target */
+ Node *join_condition; /* join condition between source and target */
List *mergeWhenClauses; /* list of MergeWhenClause(es) */
- WithClause *withClause; /* WITH clause */
+ WithClause *withClause; /* WITH clause */
} MergeStmt;
typedef struct MergeWhenClause
--
2.26.2
--------------B1AA4DC5EAE9BAF8B146244A
Content-Type: text/plain; charset=UTF-8;
name="0002-fix-valgrind-failure.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="0002-fix-valgrind-failure.txt"
^ permalink raw reply [nested|flat] 97+ messages in thread
* [PATCH 1/2] review
@ 2021-01-09 02:17 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw)
---
doc/src/sgml/mvcc.sgml | 20 +++++++++++---------
doc/src/sgml/ref/create_policy.sgml | 6 +++---
doc/src/sgml/ref/merge.sgml | 12 +++++++++---
src/backend/commands/explain.c | 2 +-
src/backend/executor/execMerge.c | 5 ++++-
src/backend/executor/nodeModifyTable.c | 3 ++-
src/backend/optimizer/prep/preptlist.c | 6 ++++++
src/backend/parser/parse_agg.c | 1 +
src/backend/replication/walsender.c | 4 ++--
src/backend/rewrite/rewriteHandler.c | 4 ++++
src/backend/utils/adt/varlena.c | 3 +++
src/include/nodes/parsenodes.h | 7 ++++---
12 files changed, 50 insertions(+), 23 deletions(-)
diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml
index 9ec8474185..02c9f3fdea 100644
--- a/doc/src/sgml/mvcc.sgml
+++ b/doc/src/sgml/mvcc.sgml
@@ -429,22 +429,24 @@ COMMIT;
with both <command>INSERT</command> and <command>UPDATE</command>
subcommands looks similar to <command>INSERT</command> with an
<literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee
- that either <command>INSERT</command> and <command>UPDATE</command> will occur.
+ that either <command>INSERT</command> or <command>UPDATE</command> will occur.
- If MERGE attempts an UPDATE or DELETE and the row is concurrently updated
+ /* XXX This is a very long and hard to understand sentence :-( */
+ /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */
+ If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated
but the join condition still passes for the current target and the current
- source tuple, then MERGE will behave the same as the UPDATE or DELETE commands
+ source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands
and perform its action on the latest version of the row, using standard
- EvalPlanQual. MERGE actions can be conditional, so conditions must be
+ EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be
re-evaluated on the latest row, starting from the first action.
On the other hand, if the row is concurrently updated or deleted so that
- the join condition fails, then MERGE will execute a NOT MATCHED action, if it
- exists and the AND WHEN qual evaluates to true.
+ the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it
+ exists and the <literal>AND WHEN</literal> qual evaluates to true.
- If MERGE attempts an INSERT and a unique index is present and a duplicate
- row is concurrently inserted then a uniqueness violation is raised. MERGE
- does not attempt to avoid the ERROR by attempting an UPDATE.
+ If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate
+ row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command>
+ does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>.
</para>
<para>
diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml
index ad20230c58..0a64674f2d 100644
--- a/doc/src/sgml/ref/create_policy.sgml
+++ b/doc/src/sgml/ref/create_policy.sgml
@@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable
<para>
No separate policy exists for <command>MERGE</command>. Instead policies
- defined for <literal>SELECT</literal>, <literal>INSERT</literal>,
- <literal>UPDATE</literal> and <literal>DELETE</literal> are applied
- while executing MERGE, depending on the actions that are activated.
+ defined for <command>SELECT</command>, <command>INSERT</command>,
+ <command>UPDATE</command> and <command>DELETE</command> are applied
+ while executing <command>MERGE</command>, depending on the actions that are activated.
</para>
</refsect1>
diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml
index b2a9f67cfa..c2901b0d58 100644
--- a/doc/src/sgml/ref/merge.sgml
+++ b/doc/src/sgml/ref/merge.sgml
@@ -98,7 +98,7 @@ DELETE
</para>
<para>
- There is no MERGE privilege.
+ There is no <literal>MERGE</literal> privilege.
You must have the <literal>UPDATE</literal> privilege on the column(s)
of the <replaceable class="parameter">target_table_name</replaceable>
referred to in the <literal>SET</literal> clause
@@ -217,6 +217,7 @@ DELETE
At least one <literal>WHEN</literal> clause is required.
</para>
<para>
+ /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */
If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal>
and the candidate change row matches a row in the
<replaceable class="parameter">target_table_name</replaceable>
@@ -242,6 +243,7 @@ DELETE
clause will be activated and the corresponding action will occur for
that row. The expression may not contain functions that possibly performs
writes to the database.
+ /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */
</para>
<para>
A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns
@@ -379,6 +381,7 @@ DELETE
<para>
An expression to assign to the column. The expression can use the
old values of this and other columns in the table.
+ /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */
</para>
</listitem>
</varlistentry>
@@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable>
In summary, statement triggers for an event type (say, INSERT) will
be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level
triggers will fire only for the one event type <emphasis>activated</emphasis>.
+ /* XXX What does "activated" mean? Maybe "executed" would be better? */
So a <command>MERGE</command> might fire statement triggers for both
<command>UPDATE</command> and <command>INSERT</command>, even though only
<command>UPDATE</command> row triggers were fired.
@@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable>
rows will be used to modify the target row, later attempts to modify will
cause an error. This can also occur if row triggers make changes to the
target table which are then subsequently modified by <command>MERGE</command>.
+ /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */
+ /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */
If the repeated action is an <command>INSERT</command> this will
cause a uniqueness violation while a repeated <command>UPDATE</command> or
<command>DELETE</command> will cause a cardinality violation; the latter behavior
@@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable>
<title>Examples</title>
<para>
- Perform maintenance on CustomerAccounts based upon new Transactions.
+ Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>.
<programlisting>
MERGE CustomerAccount CA
@@ -599,7 +605,7 @@ WHEN MATCHED THEN
DELETE;
</programlisting>
- The wine_stock_changes table might be, for example, a temporary table
+ The <literal>wine_stock_changes</literal> table might be, for example, a temporary table
recently loaded into the database.
</para>
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index fbaf50c258..f914a00e9f 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors,
break;
case CMD_MERGE:
operation = "Merge";
- foperation = "Foreign Merge";
+ foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */
break;
default:
operation = "???";
diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c
index 0e245e1361..a7492a6c4b 100644
--- a/src/backend/executor/execMerge.c
+++ b/src/backend/executor/execMerge.c
@@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo,
* A concurrent update can:
*
* 1. modify the target tuple so that it no longer satisfies the
- * additional quals attached to the current WHEN MATCHED action OR
+ * additional quals attached to the current WHEN MATCHED action
*
* In this case, we are still dealing with a WHEN MATCHED case, but we
* should recheck the list of WHEN MATCHED actions and choose the first
@@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo,
* tuple, so we now instead find a qualifying WHEN NOT MATCHED action to
* execute.
*
+ * XXX Hmmm, what if the updated tuple would now match one that was
+ * considered NOT MATCHED so far?
+ *
* A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED.
*
* ExecMergeMatched takes care of following the update chain and
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index c0a97eba91..9c14709e47 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate)
{
/* advance to next subplan if any */
node->mt_whichplan++;
-
if (node->mt_whichplan < node->mt_nplans)
{
resultRelInfo++;
@@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate)
EvalPlanQualSetSlot(&node->mt_epqstate, planSlot);
slot = planSlot;
+ /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together
+ * with the other MT operations? This seems a bit out of place. */
if (operation == CMD_MERGE)
{
ExecMerge(node, resultRelInfo, estate, slot, junkfilter);
diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c
index 8848e9a03f..b2543f6814 100644
--- a/src/backend/optimizer/prep/preptlist.c
+++ b/src/backend/optimizer/prep/preptlist.c
@@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root)
tlist = expand_targetlist(tlist, command_type,
result_relation, target_relation);
+ /*
+ * For MERGE we need to handle the target list for the target relation,
+ * and also target list for each action (only INSERT/UPDATE matter).
+ */
if (command_type == CMD_MERGE)
{
ListCell *l;
@@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type,
* generate a NULL for dropped columns (we want to drop any old
* values).
*
+ * XXX Should this explain why MERGE has the same logic as UPDATE?
+ *
* When generating a NULL constant for a dropped column, we label
* it INT4 (any other guaranteed-to-exist datatype would do as
* well). We can't label it with the dropped column's datatype
diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c
index f7c152beb4..5d49c8c37e 100644
--- a/src/backend/parser/parse_agg.c
+++ b/src/backend/parser/parse_agg.c
@@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr)
break;
case EXPR_KIND_MERGE_WHEN_AND:
if (isAgg)
+ /* XXX "WHEN AND" seems rather strange. ... */
err = _("aggregate functions are not allowed in WHEN AND conditions");
else
err = _("grouping operations are not allowed in WHEN AND conditions");
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index d59f4fde95..d50543b1ba 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd)
/* Also update the sent position status in shared memory */
SpinLockAcquire(&MyWalSnd->mutex);
+ /* XXX Huh? Why does MERGE patch change walsender? */
MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush;
SpinLockRelease(&MyWalSnd->mutex);
@@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data)
replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ?
MyWalSnd->write : MyWalSnd->flush;
- if (WalSndCaughtUp &&
- sentPtr == replicatedPtr &&
+ if (WalSndCaughtUp && sentPtr == replicatedPtr &&
!pq_is_send_pending())
{
QueryCompletion qc;
diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c
index e2706c181c..80ae946d17 100644
--- a/src/backend/rewrite/rewriteHandler.c
+++ b/src/backend/rewrite/rewriteHandler.c
@@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation)
}
}
+
/*
* matchLocks -
* match the list of locks and returns the matching rules
@@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events)
/*
* XXX MERGE doesn't support write rules because they would violate
* the SQL Standard spec and would be unclear how they should work.
+ *
+ * XXX So does't support means 'ignores'? Should that either fail
+ * or at least print some warning?
*/
if (event == CMD_MERGE)
product_queries = NIL;
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c
index 0281351dcf..7a768a7b5b 100644
--- a/src/backend/utils/adt/varlena.c
+++ b/src/backend/utils/adt/varlena.c
@@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup)
int result;
bool arg1_match;
+ /* XXX Huh? Why is this in MERGE patch? */
if (len1 < 0 || len2 < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
@@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup)
memset(pres, 0, sizeof(Datum));
len = VARSIZE_ANY_EXHDR(authoritative);
+ /* XXX Huh? Why is this in MERGE patch? */
if (len < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
@@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2)
len1 = VARSIZE_ANY_EXHDR(t1);
len2 = VARSIZE_ANY_EXHDR(t2);
+ /* XXX Huh? Why is this in MERGE patch? */
if (len1 < 0 || len2 < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 3cba38044e..08fea9b8f7 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -172,6 +172,7 @@ typedef struct Query
List *withCheckOptions; /* a list of WithCheckOption's (added
* during rewrite) */
+ /* XXX Why not mergeTragetRelation? */
int mergeTarget_relation;
List *mergeSourceTargetList;
List *mergeActionList; /* list of actions for MERGE (only) */
@@ -1581,11 +1582,11 @@ typedef struct UpdateStmt
typedef struct MergeStmt
{
NodeTag type;
- RangeVar *relation; /* target relation to merge into */
+ RangeVar *relation; /* target relation to merge into */
Node *source_relation; /* source relation */
- Node *join_condition; /* join condition between source and target */
+ Node *join_condition; /* join condition between source and target */
List *mergeWhenClauses; /* list of MergeWhenClause(es) */
- WithClause *withClause; /* WITH clause */
+ WithClause *withClause; /* WITH clause */
} MergeStmt;
typedef struct MergeWhenClause
--
2.26.2
--------------B1AA4DC5EAE9BAF8B146244A
Content-Type: text/plain; charset=UTF-8;
name="0002-fix-valgrind-failure.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="0002-fix-valgrind-failure.txt"
^ permalink raw reply [nested|flat] 97+ messages in thread
* [PATCH 1/2] review
@ 2021-01-09 02:17 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw)
---
doc/src/sgml/mvcc.sgml | 20 +++++++++++---------
doc/src/sgml/ref/create_policy.sgml | 6 +++---
doc/src/sgml/ref/merge.sgml | 12 +++++++++---
src/backend/commands/explain.c | 2 +-
src/backend/executor/execMerge.c | 5 ++++-
src/backend/executor/nodeModifyTable.c | 3 ++-
src/backend/optimizer/prep/preptlist.c | 6 ++++++
src/backend/parser/parse_agg.c | 1 +
src/backend/replication/walsender.c | 4 ++--
src/backend/rewrite/rewriteHandler.c | 4 ++++
src/backend/utils/adt/varlena.c | 3 +++
src/include/nodes/parsenodes.h | 7 ++++---
12 files changed, 50 insertions(+), 23 deletions(-)
diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml
index 9ec8474185..02c9f3fdea 100644
--- a/doc/src/sgml/mvcc.sgml
+++ b/doc/src/sgml/mvcc.sgml
@@ -429,22 +429,24 @@ COMMIT;
with both <command>INSERT</command> and <command>UPDATE</command>
subcommands looks similar to <command>INSERT</command> with an
<literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee
- that either <command>INSERT</command> and <command>UPDATE</command> will occur.
+ that either <command>INSERT</command> or <command>UPDATE</command> will occur.
- If MERGE attempts an UPDATE or DELETE and the row is concurrently updated
+ /* XXX This is a very long and hard to understand sentence :-( */
+ /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */
+ If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated
but the join condition still passes for the current target and the current
- source tuple, then MERGE will behave the same as the UPDATE or DELETE commands
+ source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands
and perform its action on the latest version of the row, using standard
- EvalPlanQual. MERGE actions can be conditional, so conditions must be
+ EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be
re-evaluated on the latest row, starting from the first action.
On the other hand, if the row is concurrently updated or deleted so that
- the join condition fails, then MERGE will execute a NOT MATCHED action, if it
- exists and the AND WHEN qual evaluates to true.
+ the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it
+ exists and the <literal>AND WHEN</literal> qual evaluates to true.
- If MERGE attempts an INSERT and a unique index is present and a duplicate
- row is concurrently inserted then a uniqueness violation is raised. MERGE
- does not attempt to avoid the ERROR by attempting an UPDATE.
+ If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate
+ row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command>
+ does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>.
</para>
<para>
diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml
index ad20230c58..0a64674f2d 100644
--- a/doc/src/sgml/ref/create_policy.sgml
+++ b/doc/src/sgml/ref/create_policy.sgml
@@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable
<para>
No separate policy exists for <command>MERGE</command>. Instead policies
- defined for <literal>SELECT</literal>, <literal>INSERT</literal>,
- <literal>UPDATE</literal> and <literal>DELETE</literal> are applied
- while executing MERGE, depending on the actions that are activated.
+ defined for <command>SELECT</command>, <command>INSERT</command>,
+ <command>UPDATE</command> and <command>DELETE</command> are applied
+ while executing <command>MERGE</command>, depending on the actions that are activated.
</para>
</refsect1>
diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml
index b2a9f67cfa..c2901b0d58 100644
--- a/doc/src/sgml/ref/merge.sgml
+++ b/doc/src/sgml/ref/merge.sgml
@@ -98,7 +98,7 @@ DELETE
</para>
<para>
- There is no MERGE privilege.
+ There is no <literal>MERGE</literal> privilege.
You must have the <literal>UPDATE</literal> privilege on the column(s)
of the <replaceable class="parameter">target_table_name</replaceable>
referred to in the <literal>SET</literal> clause
@@ -217,6 +217,7 @@ DELETE
At least one <literal>WHEN</literal> clause is required.
</para>
<para>
+ /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */
If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal>
and the candidate change row matches a row in the
<replaceable class="parameter">target_table_name</replaceable>
@@ -242,6 +243,7 @@ DELETE
clause will be activated and the corresponding action will occur for
that row. The expression may not contain functions that possibly performs
writes to the database.
+ /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */
</para>
<para>
A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns
@@ -379,6 +381,7 @@ DELETE
<para>
An expression to assign to the column. The expression can use the
old values of this and other columns in the table.
+ /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */
</para>
</listitem>
</varlistentry>
@@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable>
In summary, statement triggers for an event type (say, INSERT) will
be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level
triggers will fire only for the one event type <emphasis>activated</emphasis>.
+ /* XXX What does "activated" mean? Maybe "executed" would be better? */
So a <command>MERGE</command> might fire statement triggers for both
<command>UPDATE</command> and <command>INSERT</command>, even though only
<command>UPDATE</command> row triggers were fired.
@@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable>
rows will be used to modify the target row, later attempts to modify will
cause an error. This can also occur if row triggers make changes to the
target table which are then subsequently modified by <command>MERGE</command>.
+ /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */
+ /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */
If the repeated action is an <command>INSERT</command> this will
cause a uniqueness violation while a repeated <command>UPDATE</command> or
<command>DELETE</command> will cause a cardinality violation; the latter behavior
@@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable>
<title>Examples</title>
<para>
- Perform maintenance on CustomerAccounts based upon new Transactions.
+ Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>.
<programlisting>
MERGE CustomerAccount CA
@@ -599,7 +605,7 @@ WHEN MATCHED THEN
DELETE;
</programlisting>
- The wine_stock_changes table might be, for example, a temporary table
+ The <literal>wine_stock_changes</literal> table might be, for example, a temporary table
recently loaded into the database.
</para>
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index fbaf50c258..f914a00e9f 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors,
break;
case CMD_MERGE:
operation = "Merge";
- foperation = "Foreign Merge";
+ foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */
break;
default:
operation = "???";
diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c
index 0e245e1361..a7492a6c4b 100644
--- a/src/backend/executor/execMerge.c
+++ b/src/backend/executor/execMerge.c
@@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo,
* A concurrent update can:
*
* 1. modify the target tuple so that it no longer satisfies the
- * additional quals attached to the current WHEN MATCHED action OR
+ * additional quals attached to the current WHEN MATCHED action
*
* In this case, we are still dealing with a WHEN MATCHED case, but we
* should recheck the list of WHEN MATCHED actions and choose the first
@@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo,
* tuple, so we now instead find a qualifying WHEN NOT MATCHED action to
* execute.
*
+ * XXX Hmmm, what if the updated tuple would now match one that was
+ * considered NOT MATCHED so far?
+ *
* A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED.
*
* ExecMergeMatched takes care of following the update chain and
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index c0a97eba91..9c14709e47 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate)
{
/* advance to next subplan if any */
node->mt_whichplan++;
-
if (node->mt_whichplan < node->mt_nplans)
{
resultRelInfo++;
@@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate)
EvalPlanQualSetSlot(&node->mt_epqstate, planSlot);
slot = planSlot;
+ /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together
+ * with the other MT operations? This seems a bit out of place. */
if (operation == CMD_MERGE)
{
ExecMerge(node, resultRelInfo, estate, slot, junkfilter);
diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c
index 8848e9a03f..b2543f6814 100644
--- a/src/backend/optimizer/prep/preptlist.c
+++ b/src/backend/optimizer/prep/preptlist.c
@@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root)
tlist = expand_targetlist(tlist, command_type,
result_relation, target_relation);
+ /*
+ * For MERGE we need to handle the target list for the target relation,
+ * and also target list for each action (only INSERT/UPDATE matter).
+ */
if (command_type == CMD_MERGE)
{
ListCell *l;
@@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type,
* generate a NULL for dropped columns (we want to drop any old
* values).
*
+ * XXX Should this explain why MERGE has the same logic as UPDATE?
+ *
* When generating a NULL constant for a dropped column, we label
* it INT4 (any other guaranteed-to-exist datatype would do as
* well). We can't label it with the dropped column's datatype
diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c
index f7c152beb4..5d49c8c37e 100644
--- a/src/backend/parser/parse_agg.c
+++ b/src/backend/parser/parse_agg.c
@@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr)
break;
case EXPR_KIND_MERGE_WHEN_AND:
if (isAgg)
+ /* XXX "WHEN AND" seems rather strange. ... */
err = _("aggregate functions are not allowed in WHEN AND conditions");
else
err = _("grouping operations are not allowed in WHEN AND conditions");
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index d59f4fde95..d50543b1ba 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd)
/* Also update the sent position status in shared memory */
SpinLockAcquire(&MyWalSnd->mutex);
+ /* XXX Huh? Why does MERGE patch change walsender? */
MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush;
SpinLockRelease(&MyWalSnd->mutex);
@@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data)
replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ?
MyWalSnd->write : MyWalSnd->flush;
- if (WalSndCaughtUp &&
- sentPtr == replicatedPtr &&
+ if (WalSndCaughtUp && sentPtr == replicatedPtr &&
!pq_is_send_pending())
{
QueryCompletion qc;
diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c
index e2706c181c..80ae946d17 100644
--- a/src/backend/rewrite/rewriteHandler.c
+++ b/src/backend/rewrite/rewriteHandler.c
@@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation)
}
}
+
/*
* matchLocks -
* match the list of locks and returns the matching rules
@@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events)
/*
* XXX MERGE doesn't support write rules because they would violate
* the SQL Standard spec and would be unclear how they should work.
+ *
+ * XXX So does't support means 'ignores'? Should that either fail
+ * or at least print some warning?
*/
if (event == CMD_MERGE)
product_queries = NIL;
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c
index 0281351dcf..7a768a7b5b 100644
--- a/src/backend/utils/adt/varlena.c
+++ b/src/backend/utils/adt/varlena.c
@@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup)
int result;
bool arg1_match;
+ /* XXX Huh? Why is this in MERGE patch? */
if (len1 < 0 || len2 < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
@@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup)
memset(pres, 0, sizeof(Datum));
len = VARSIZE_ANY_EXHDR(authoritative);
+ /* XXX Huh? Why is this in MERGE patch? */
if (len < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
@@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2)
len1 = VARSIZE_ANY_EXHDR(t1);
len2 = VARSIZE_ANY_EXHDR(t2);
+ /* XXX Huh? Why is this in MERGE patch? */
if (len1 < 0 || len2 < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 3cba38044e..08fea9b8f7 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -172,6 +172,7 @@ typedef struct Query
List *withCheckOptions; /* a list of WithCheckOption's (added
* during rewrite) */
+ /* XXX Why not mergeTragetRelation? */
int mergeTarget_relation;
List *mergeSourceTargetList;
List *mergeActionList; /* list of actions for MERGE (only) */
@@ -1581,11 +1582,11 @@ typedef struct UpdateStmt
typedef struct MergeStmt
{
NodeTag type;
- RangeVar *relation; /* target relation to merge into */
+ RangeVar *relation; /* target relation to merge into */
Node *source_relation; /* source relation */
- Node *join_condition; /* join condition between source and target */
+ Node *join_condition; /* join condition between source and target */
List *mergeWhenClauses; /* list of MergeWhenClause(es) */
- WithClause *withClause; /* WITH clause */
+ WithClause *withClause; /* WITH clause */
} MergeStmt;
typedef struct MergeWhenClause
--
2.26.2
--------------B1AA4DC5EAE9BAF8B146244A
Content-Type: text/plain; charset=UTF-8;
name="0002-fix-valgrind-failure.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="0002-fix-valgrind-failure.txt"
^ permalink raw reply [nested|flat] 97+ messages in thread
* [PATCH 1/2] review
@ 2021-01-09 02:17 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw)
---
doc/src/sgml/mvcc.sgml | 20 +++++++++++---------
doc/src/sgml/ref/create_policy.sgml | 6 +++---
doc/src/sgml/ref/merge.sgml | 12 +++++++++---
src/backend/commands/explain.c | 2 +-
src/backend/executor/execMerge.c | 5 ++++-
src/backend/executor/nodeModifyTable.c | 3 ++-
src/backend/optimizer/prep/preptlist.c | 6 ++++++
src/backend/parser/parse_agg.c | 1 +
src/backend/replication/walsender.c | 4 ++--
src/backend/rewrite/rewriteHandler.c | 4 ++++
src/backend/utils/adt/varlena.c | 3 +++
src/include/nodes/parsenodes.h | 7 ++++---
12 files changed, 50 insertions(+), 23 deletions(-)
diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml
index 9ec8474185..02c9f3fdea 100644
--- a/doc/src/sgml/mvcc.sgml
+++ b/doc/src/sgml/mvcc.sgml
@@ -429,22 +429,24 @@ COMMIT;
with both <command>INSERT</command> and <command>UPDATE</command>
subcommands looks similar to <command>INSERT</command> with an
<literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee
- that either <command>INSERT</command> and <command>UPDATE</command> will occur.
+ that either <command>INSERT</command> or <command>UPDATE</command> will occur.
- If MERGE attempts an UPDATE or DELETE and the row is concurrently updated
+ /* XXX This is a very long and hard to understand sentence :-( */
+ /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */
+ If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated
but the join condition still passes for the current target and the current
- source tuple, then MERGE will behave the same as the UPDATE or DELETE commands
+ source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands
and perform its action on the latest version of the row, using standard
- EvalPlanQual. MERGE actions can be conditional, so conditions must be
+ EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be
re-evaluated on the latest row, starting from the first action.
On the other hand, if the row is concurrently updated or deleted so that
- the join condition fails, then MERGE will execute a NOT MATCHED action, if it
- exists and the AND WHEN qual evaluates to true.
+ the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it
+ exists and the <literal>AND WHEN</literal> qual evaluates to true.
- If MERGE attempts an INSERT and a unique index is present and a duplicate
- row is concurrently inserted then a uniqueness violation is raised. MERGE
- does not attempt to avoid the ERROR by attempting an UPDATE.
+ If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate
+ row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command>
+ does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>.
</para>
<para>
diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml
index ad20230c58..0a64674f2d 100644
--- a/doc/src/sgml/ref/create_policy.sgml
+++ b/doc/src/sgml/ref/create_policy.sgml
@@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable
<para>
No separate policy exists for <command>MERGE</command>. Instead policies
- defined for <literal>SELECT</literal>, <literal>INSERT</literal>,
- <literal>UPDATE</literal> and <literal>DELETE</literal> are applied
- while executing MERGE, depending on the actions that are activated.
+ defined for <command>SELECT</command>, <command>INSERT</command>,
+ <command>UPDATE</command> and <command>DELETE</command> are applied
+ while executing <command>MERGE</command>, depending on the actions that are activated.
</para>
</refsect1>
diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml
index b2a9f67cfa..c2901b0d58 100644
--- a/doc/src/sgml/ref/merge.sgml
+++ b/doc/src/sgml/ref/merge.sgml
@@ -98,7 +98,7 @@ DELETE
</para>
<para>
- There is no MERGE privilege.
+ There is no <literal>MERGE</literal> privilege.
You must have the <literal>UPDATE</literal> privilege on the column(s)
of the <replaceable class="parameter">target_table_name</replaceable>
referred to in the <literal>SET</literal> clause
@@ -217,6 +217,7 @@ DELETE
At least one <literal>WHEN</literal> clause is required.
</para>
<para>
+ /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */
If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal>
and the candidate change row matches a row in the
<replaceable class="parameter">target_table_name</replaceable>
@@ -242,6 +243,7 @@ DELETE
clause will be activated and the corresponding action will occur for
that row. The expression may not contain functions that possibly performs
writes to the database.
+ /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */
</para>
<para>
A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns
@@ -379,6 +381,7 @@ DELETE
<para>
An expression to assign to the column. The expression can use the
old values of this and other columns in the table.
+ /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */
</para>
</listitem>
</varlistentry>
@@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable>
In summary, statement triggers for an event type (say, INSERT) will
be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level
triggers will fire only for the one event type <emphasis>activated</emphasis>.
+ /* XXX What does "activated" mean? Maybe "executed" would be better? */
So a <command>MERGE</command> might fire statement triggers for both
<command>UPDATE</command> and <command>INSERT</command>, even though only
<command>UPDATE</command> row triggers were fired.
@@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable>
rows will be used to modify the target row, later attempts to modify will
cause an error. This can also occur if row triggers make changes to the
target table which are then subsequently modified by <command>MERGE</command>.
+ /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */
+ /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */
If the repeated action is an <command>INSERT</command> this will
cause a uniqueness violation while a repeated <command>UPDATE</command> or
<command>DELETE</command> will cause a cardinality violation; the latter behavior
@@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable>
<title>Examples</title>
<para>
- Perform maintenance on CustomerAccounts based upon new Transactions.
+ Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>.
<programlisting>
MERGE CustomerAccount CA
@@ -599,7 +605,7 @@ WHEN MATCHED THEN
DELETE;
</programlisting>
- The wine_stock_changes table might be, for example, a temporary table
+ The <literal>wine_stock_changes</literal> table might be, for example, a temporary table
recently loaded into the database.
</para>
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index fbaf50c258..f914a00e9f 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors,
break;
case CMD_MERGE:
operation = "Merge";
- foperation = "Foreign Merge";
+ foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */
break;
default:
operation = "???";
diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c
index 0e245e1361..a7492a6c4b 100644
--- a/src/backend/executor/execMerge.c
+++ b/src/backend/executor/execMerge.c
@@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo,
* A concurrent update can:
*
* 1. modify the target tuple so that it no longer satisfies the
- * additional quals attached to the current WHEN MATCHED action OR
+ * additional quals attached to the current WHEN MATCHED action
*
* In this case, we are still dealing with a WHEN MATCHED case, but we
* should recheck the list of WHEN MATCHED actions and choose the first
@@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo,
* tuple, so we now instead find a qualifying WHEN NOT MATCHED action to
* execute.
*
+ * XXX Hmmm, what if the updated tuple would now match one that was
+ * considered NOT MATCHED so far?
+ *
* A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED.
*
* ExecMergeMatched takes care of following the update chain and
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index c0a97eba91..9c14709e47 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate)
{
/* advance to next subplan if any */
node->mt_whichplan++;
-
if (node->mt_whichplan < node->mt_nplans)
{
resultRelInfo++;
@@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate)
EvalPlanQualSetSlot(&node->mt_epqstate, planSlot);
slot = planSlot;
+ /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together
+ * with the other MT operations? This seems a bit out of place. */
if (operation == CMD_MERGE)
{
ExecMerge(node, resultRelInfo, estate, slot, junkfilter);
diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c
index 8848e9a03f..b2543f6814 100644
--- a/src/backend/optimizer/prep/preptlist.c
+++ b/src/backend/optimizer/prep/preptlist.c
@@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root)
tlist = expand_targetlist(tlist, command_type,
result_relation, target_relation);
+ /*
+ * For MERGE we need to handle the target list for the target relation,
+ * and also target list for each action (only INSERT/UPDATE matter).
+ */
if (command_type == CMD_MERGE)
{
ListCell *l;
@@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type,
* generate a NULL for dropped columns (we want to drop any old
* values).
*
+ * XXX Should this explain why MERGE has the same logic as UPDATE?
+ *
* When generating a NULL constant for a dropped column, we label
* it INT4 (any other guaranteed-to-exist datatype would do as
* well). We can't label it with the dropped column's datatype
diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c
index f7c152beb4..5d49c8c37e 100644
--- a/src/backend/parser/parse_agg.c
+++ b/src/backend/parser/parse_agg.c
@@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr)
break;
case EXPR_KIND_MERGE_WHEN_AND:
if (isAgg)
+ /* XXX "WHEN AND" seems rather strange. ... */
err = _("aggregate functions are not allowed in WHEN AND conditions");
else
err = _("grouping operations are not allowed in WHEN AND conditions");
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index d59f4fde95..d50543b1ba 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd)
/* Also update the sent position status in shared memory */
SpinLockAcquire(&MyWalSnd->mutex);
+ /* XXX Huh? Why does MERGE patch change walsender? */
MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush;
SpinLockRelease(&MyWalSnd->mutex);
@@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data)
replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ?
MyWalSnd->write : MyWalSnd->flush;
- if (WalSndCaughtUp &&
- sentPtr == replicatedPtr &&
+ if (WalSndCaughtUp && sentPtr == replicatedPtr &&
!pq_is_send_pending())
{
QueryCompletion qc;
diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c
index e2706c181c..80ae946d17 100644
--- a/src/backend/rewrite/rewriteHandler.c
+++ b/src/backend/rewrite/rewriteHandler.c
@@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation)
}
}
+
/*
* matchLocks -
* match the list of locks and returns the matching rules
@@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events)
/*
* XXX MERGE doesn't support write rules because they would violate
* the SQL Standard spec and would be unclear how they should work.
+ *
+ * XXX So does't support means 'ignores'? Should that either fail
+ * or at least print some warning?
*/
if (event == CMD_MERGE)
product_queries = NIL;
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c
index 0281351dcf..7a768a7b5b 100644
--- a/src/backend/utils/adt/varlena.c
+++ b/src/backend/utils/adt/varlena.c
@@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup)
int result;
bool arg1_match;
+ /* XXX Huh? Why is this in MERGE patch? */
if (len1 < 0 || len2 < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
@@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup)
memset(pres, 0, sizeof(Datum));
len = VARSIZE_ANY_EXHDR(authoritative);
+ /* XXX Huh? Why is this in MERGE patch? */
if (len < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
@@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2)
len1 = VARSIZE_ANY_EXHDR(t1);
len2 = VARSIZE_ANY_EXHDR(t2);
+ /* XXX Huh? Why is this in MERGE patch? */
if (len1 < 0 || len2 < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 3cba38044e..08fea9b8f7 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -172,6 +172,7 @@ typedef struct Query
List *withCheckOptions; /* a list of WithCheckOption's (added
* during rewrite) */
+ /* XXX Why not mergeTragetRelation? */
int mergeTarget_relation;
List *mergeSourceTargetList;
List *mergeActionList; /* list of actions for MERGE (only) */
@@ -1581,11 +1582,11 @@ typedef struct UpdateStmt
typedef struct MergeStmt
{
NodeTag type;
- RangeVar *relation; /* target relation to merge into */
+ RangeVar *relation; /* target relation to merge into */
Node *source_relation; /* source relation */
- Node *join_condition; /* join condition between source and target */
+ Node *join_condition; /* join condition between source and target */
List *mergeWhenClauses; /* list of MergeWhenClause(es) */
- WithClause *withClause; /* WITH clause */
+ WithClause *withClause; /* WITH clause */
} MergeStmt;
typedef struct MergeWhenClause
--
2.26.2
--------------B1AA4DC5EAE9BAF8B146244A
Content-Type: text/plain; charset=UTF-8;
name="0002-fix-valgrind-failure.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="0002-fix-valgrind-failure.txt"
^ permalink raw reply [nested|flat] 97+ messages in thread
* [PATCH 1/2] review
@ 2021-01-09 02:17 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw)
---
doc/src/sgml/mvcc.sgml | 20 +++++++++++---------
doc/src/sgml/ref/create_policy.sgml | 6 +++---
doc/src/sgml/ref/merge.sgml | 12 +++++++++---
src/backend/commands/explain.c | 2 +-
src/backend/executor/execMerge.c | 5 ++++-
src/backend/executor/nodeModifyTable.c | 3 ++-
src/backend/optimizer/prep/preptlist.c | 6 ++++++
src/backend/parser/parse_agg.c | 1 +
src/backend/replication/walsender.c | 4 ++--
src/backend/rewrite/rewriteHandler.c | 4 ++++
src/backend/utils/adt/varlena.c | 3 +++
src/include/nodes/parsenodes.h | 7 ++++---
12 files changed, 50 insertions(+), 23 deletions(-)
diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml
index 9ec8474185..02c9f3fdea 100644
--- a/doc/src/sgml/mvcc.sgml
+++ b/doc/src/sgml/mvcc.sgml
@@ -429,22 +429,24 @@ COMMIT;
with both <command>INSERT</command> and <command>UPDATE</command>
subcommands looks similar to <command>INSERT</command> with an
<literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee
- that either <command>INSERT</command> and <command>UPDATE</command> will occur.
+ that either <command>INSERT</command> or <command>UPDATE</command> will occur.
- If MERGE attempts an UPDATE or DELETE and the row is concurrently updated
+ /* XXX This is a very long and hard to understand sentence :-( */
+ /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */
+ If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated
but the join condition still passes for the current target and the current
- source tuple, then MERGE will behave the same as the UPDATE or DELETE commands
+ source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands
and perform its action on the latest version of the row, using standard
- EvalPlanQual. MERGE actions can be conditional, so conditions must be
+ EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be
re-evaluated on the latest row, starting from the first action.
On the other hand, if the row is concurrently updated or deleted so that
- the join condition fails, then MERGE will execute a NOT MATCHED action, if it
- exists and the AND WHEN qual evaluates to true.
+ the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it
+ exists and the <literal>AND WHEN</literal> qual evaluates to true.
- If MERGE attempts an INSERT and a unique index is present and a duplicate
- row is concurrently inserted then a uniqueness violation is raised. MERGE
- does not attempt to avoid the ERROR by attempting an UPDATE.
+ If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate
+ row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command>
+ does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>.
</para>
<para>
diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml
index ad20230c58..0a64674f2d 100644
--- a/doc/src/sgml/ref/create_policy.sgml
+++ b/doc/src/sgml/ref/create_policy.sgml
@@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable
<para>
No separate policy exists for <command>MERGE</command>. Instead policies
- defined for <literal>SELECT</literal>, <literal>INSERT</literal>,
- <literal>UPDATE</literal> and <literal>DELETE</literal> are applied
- while executing MERGE, depending on the actions that are activated.
+ defined for <command>SELECT</command>, <command>INSERT</command>,
+ <command>UPDATE</command> and <command>DELETE</command> are applied
+ while executing <command>MERGE</command>, depending on the actions that are activated.
</para>
</refsect1>
diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml
index b2a9f67cfa..c2901b0d58 100644
--- a/doc/src/sgml/ref/merge.sgml
+++ b/doc/src/sgml/ref/merge.sgml
@@ -98,7 +98,7 @@ DELETE
</para>
<para>
- There is no MERGE privilege.
+ There is no <literal>MERGE</literal> privilege.
You must have the <literal>UPDATE</literal> privilege on the column(s)
of the <replaceable class="parameter">target_table_name</replaceable>
referred to in the <literal>SET</literal> clause
@@ -217,6 +217,7 @@ DELETE
At least one <literal>WHEN</literal> clause is required.
</para>
<para>
+ /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */
If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal>
and the candidate change row matches a row in the
<replaceable class="parameter">target_table_name</replaceable>
@@ -242,6 +243,7 @@ DELETE
clause will be activated and the corresponding action will occur for
that row. The expression may not contain functions that possibly performs
writes to the database.
+ /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */
</para>
<para>
A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns
@@ -379,6 +381,7 @@ DELETE
<para>
An expression to assign to the column. The expression can use the
old values of this and other columns in the table.
+ /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */
</para>
</listitem>
</varlistentry>
@@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable>
In summary, statement triggers for an event type (say, INSERT) will
be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level
triggers will fire only for the one event type <emphasis>activated</emphasis>.
+ /* XXX What does "activated" mean? Maybe "executed" would be better? */
So a <command>MERGE</command> might fire statement triggers for both
<command>UPDATE</command> and <command>INSERT</command>, even though only
<command>UPDATE</command> row triggers were fired.
@@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable>
rows will be used to modify the target row, later attempts to modify will
cause an error. This can also occur if row triggers make changes to the
target table which are then subsequently modified by <command>MERGE</command>.
+ /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */
+ /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */
If the repeated action is an <command>INSERT</command> this will
cause a uniqueness violation while a repeated <command>UPDATE</command> or
<command>DELETE</command> will cause a cardinality violation; the latter behavior
@@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable>
<title>Examples</title>
<para>
- Perform maintenance on CustomerAccounts based upon new Transactions.
+ Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>.
<programlisting>
MERGE CustomerAccount CA
@@ -599,7 +605,7 @@ WHEN MATCHED THEN
DELETE;
</programlisting>
- The wine_stock_changes table might be, for example, a temporary table
+ The <literal>wine_stock_changes</literal> table might be, for example, a temporary table
recently loaded into the database.
</para>
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index fbaf50c258..f914a00e9f 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors,
break;
case CMD_MERGE:
operation = "Merge";
- foperation = "Foreign Merge";
+ foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */
break;
default:
operation = "???";
diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c
index 0e245e1361..a7492a6c4b 100644
--- a/src/backend/executor/execMerge.c
+++ b/src/backend/executor/execMerge.c
@@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo,
* A concurrent update can:
*
* 1. modify the target tuple so that it no longer satisfies the
- * additional quals attached to the current WHEN MATCHED action OR
+ * additional quals attached to the current WHEN MATCHED action
*
* In this case, we are still dealing with a WHEN MATCHED case, but we
* should recheck the list of WHEN MATCHED actions and choose the first
@@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo,
* tuple, so we now instead find a qualifying WHEN NOT MATCHED action to
* execute.
*
+ * XXX Hmmm, what if the updated tuple would now match one that was
+ * considered NOT MATCHED so far?
+ *
* A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED.
*
* ExecMergeMatched takes care of following the update chain and
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index c0a97eba91..9c14709e47 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate)
{
/* advance to next subplan if any */
node->mt_whichplan++;
-
if (node->mt_whichplan < node->mt_nplans)
{
resultRelInfo++;
@@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate)
EvalPlanQualSetSlot(&node->mt_epqstate, planSlot);
slot = planSlot;
+ /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together
+ * with the other MT operations? This seems a bit out of place. */
if (operation == CMD_MERGE)
{
ExecMerge(node, resultRelInfo, estate, slot, junkfilter);
diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c
index 8848e9a03f..b2543f6814 100644
--- a/src/backend/optimizer/prep/preptlist.c
+++ b/src/backend/optimizer/prep/preptlist.c
@@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root)
tlist = expand_targetlist(tlist, command_type,
result_relation, target_relation);
+ /*
+ * For MERGE we need to handle the target list for the target relation,
+ * and also target list for each action (only INSERT/UPDATE matter).
+ */
if (command_type == CMD_MERGE)
{
ListCell *l;
@@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type,
* generate a NULL for dropped columns (we want to drop any old
* values).
*
+ * XXX Should this explain why MERGE has the same logic as UPDATE?
+ *
* When generating a NULL constant for a dropped column, we label
* it INT4 (any other guaranteed-to-exist datatype would do as
* well). We can't label it with the dropped column's datatype
diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c
index f7c152beb4..5d49c8c37e 100644
--- a/src/backend/parser/parse_agg.c
+++ b/src/backend/parser/parse_agg.c
@@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr)
break;
case EXPR_KIND_MERGE_WHEN_AND:
if (isAgg)
+ /* XXX "WHEN AND" seems rather strange. ... */
err = _("aggregate functions are not allowed in WHEN AND conditions");
else
err = _("grouping operations are not allowed in WHEN AND conditions");
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index d59f4fde95..d50543b1ba 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd)
/* Also update the sent position status in shared memory */
SpinLockAcquire(&MyWalSnd->mutex);
+ /* XXX Huh? Why does MERGE patch change walsender? */
MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush;
SpinLockRelease(&MyWalSnd->mutex);
@@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data)
replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ?
MyWalSnd->write : MyWalSnd->flush;
- if (WalSndCaughtUp &&
- sentPtr == replicatedPtr &&
+ if (WalSndCaughtUp && sentPtr == replicatedPtr &&
!pq_is_send_pending())
{
QueryCompletion qc;
diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c
index e2706c181c..80ae946d17 100644
--- a/src/backend/rewrite/rewriteHandler.c
+++ b/src/backend/rewrite/rewriteHandler.c
@@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation)
}
}
+
/*
* matchLocks -
* match the list of locks and returns the matching rules
@@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events)
/*
* XXX MERGE doesn't support write rules because they would violate
* the SQL Standard spec and would be unclear how they should work.
+ *
+ * XXX So does't support means 'ignores'? Should that either fail
+ * or at least print some warning?
*/
if (event == CMD_MERGE)
product_queries = NIL;
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c
index 0281351dcf..7a768a7b5b 100644
--- a/src/backend/utils/adt/varlena.c
+++ b/src/backend/utils/adt/varlena.c
@@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup)
int result;
bool arg1_match;
+ /* XXX Huh? Why is this in MERGE patch? */
if (len1 < 0 || len2 < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
@@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup)
memset(pres, 0, sizeof(Datum));
len = VARSIZE_ANY_EXHDR(authoritative);
+ /* XXX Huh? Why is this in MERGE patch? */
if (len < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
@@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2)
len1 = VARSIZE_ANY_EXHDR(t1);
len2 = VARSIZE_ANY_EXHDR(t2);
+ /* XXX Huh? Why is this in MERGE patch? */
if (len1 < 0 || len2 < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 3cba38044e..08fea9b8f7 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -172,6 +172,7 @@ typedef struct Query
List *withCheckOptions; /* a list of WithCheckOption's (added
* during rewrite) */
+ /* XXX Why not mergeTragetRelation? */
int mergeTarget_relation;
List *mergeSourceTargetList;
List *mergeActionList; /* list of actions for MERGE (only) */
@@ -1581,11 +1582,11 @@ typedef struct UpdateStmt
typedef struct MergeStmt
{
NodeTag type;
- RangeVar *relation; /* target relation to merge into */
+ RangeVar *relation; /* target relation to merge into */
Node *source_relation; /* source relation */
- Node *join_condition; /* join condition between source and target */
+ Node *join_condition; /* join condition between source and target */
List *mergeWhenClauses; /* list of MergeWhenClause(es) */
- WithClause *withClause; /* WITH clause */
+ WithClause *withClause; /* WITH clause */
} MergeStmt;
typedef struct MergeWhenClause
--
2.26.2
--------------B1AA4DC5EAE9BAF8B146244A
Content-Type: text/plain; charset=UTF-8;
name="0002-fix-valgrind-failure.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="0002-fix-valgrind-failure.txt"
^ permalink raw reply [nested|flat] 97+ messages in thread
* [PATCH 1/2] review
@ 2021-01-09 02:17 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw)
---
doc/src/sgml/mvcc.sgml | 20 +++++++++++---------
doc/src/sgml/ref/create_policy.sgml | 6 +++---
doc/src/sgml/ref/merge.sgml | 12 +++++++++---
src/backend/commands/explain.c | 2 +-
src/backend/executor/execMerge.c | 5 ++++-
src/backend/executor/nodeModifyTable.c | 3 ++-
src/backend/optimizer/prep/preptlist.c | 6 ++++++
src/backend/parser/parse_agg.c | 1 +
src/backend/replication/walsender.c | 4 ++--
src/backend/rewrite/rewriteHandler.c | 4 ++++
src/backend/utils/adt/varlena.c | 3 +++
src/include/nodes/parsenodes.h | 7 ++++---
12 files changed, 50 insertions(+), 23 deletions(-)
diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml
index 9ec8474185..02c9f3fdea 100644
--- a/doc/src/sgml/mvcc.sgml
+++ b/doc/src/sgml/mvcc.sgml
@@ -429,22 +429,24 @@ COMMIT;
with both <command>INSERT</command> and <command>UPDATE</command>
subcommands looks similar to <command>INSERT</command> with an
<literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee
- that either <command>INSERT</command> and <command>UPDATE</command> will occur.
+ that either <command>INSERT</command> or <command>UPDATE</command> will occur.
- If MERGE attempts an UPDATE or DELETE and the row is concurrently updated
+ /* XXX This is a very long and hard to understand sentence :-( */
+ /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */
+ If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated
but the join condition still passes for the current target and the current
- source tuple, then MERGE will behave the same as the UPDATE or DELETE commands
+ source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands
and perform its action on the latest version of the row, using standard
- EvalPlanQual. MERGE actions can be conditional, so conditions must be
+ EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be
re-evaluated on the latest row, starting from the first action.
On the other hand, if the row is concurrently updated or deleted so that
- the join condition fails, then MERGE will execute a NOT MATCHED action, if it
- exists and the AND WHEN qual evaluates to true.
+ the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it
+ exists and the <literal>AND WHEN</literal> qual evaluates to true.
- If MERGE attempts an INSERT and a unique index is present and a duplicate
- row is concurrently inserted then a uniqueness violation is raised. MERGE
- does not attempt to avoid the ERROR by attempting an UPDATE.
+ If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate
+ row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command>
+ does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>.
</para>
<para>
diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml
index ad20230c58..0a64674f2d 100644
--- a/doc/src/sgml/ref/create_policy.sgml
+++ b/doc/src/sgml/ref/create_policy.sgml
@@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable
<para>
No separate policy exists for <command>MERGE</command>. Instead policies
- defined for <literal>SELECT</literal>, <literal>INSERT</literal>,
- <literal>UPDATE</literal> and <literal>DELETE</literal> are applied
- while executing MERGE, depending on the actions that are activated.
+ defined for <command>SELECT</command>, <command>INSERT</command>,
+ <command>UPDATE</command> and <command>DELETE</command> are applied
+ while executing <command>MERGE</command>, depending on the actions that are activated.
</para>
</refsect1>
diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml
index b2a9f67cfa..c2901b0d58 100644
--- a/doc/src/sgml/ref/merge.sgml
+++ b/doc/src/sgml/ref/merge.sgml
@@ -98,7 +98,7 @@ DELETE
</para>
<para>
- There is no MERGE privilege.
+ There is no <literal>MERGE</literal> privilege.
You must have the <literal>UPDATE</literal> privilege on the column(s)
of the <replaceable class="parameter">target_table_name</replaceable>
referred to in the <literal>SET</literal> clause
@@ -217,6 +217,7 @@ DELETE
At least one <literal>WHEN</literal> clause is required.
</para>
<para>
+ /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */
If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal>
and the candidate change row matches a row in the
<replaceable class="parameter">target_table_name</replaceable>
@@ -242,6 +243,7 @@ DELETE
clause will be activated and the corresponding action will occur for
that row. The expression may not contain functions that possibly performs
writes to the database.
+ /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */
</para>
<para>
A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns
@@ -379,6 +381,7 @@ DELETE
<para>
An expression to assign to the column. The expression can use the
old values of this and other columns in the table.
+ /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */
</para>
</listitem>
</varlistentry>
@@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable>
In summary, statement triggers for an event type (say, INSERT) will
be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level
triggers will fire only for the one event type <emphasis>activated</emphasis>.
+ /* XXX What does "activated" mean? Maybe "executed" would be better? */
So a <command>MERGE</command> might fire statement triggers for both
<command>UPDATE</command> and <command>INSERT</command>, even though only
<command>UPDATE</command> row triggers were fired.
@@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable>
rows will be used to modify the target row, later attempts to modify will
cause an error. This can also occur if row triggers make changes to the
target table which are then subsequently modified by <command>MERGE</command>.
+ /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */
+ /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */
If the repeated action is an <command>INSERT</command> this will
cause a uniqueness violation while a repeated <command>UPDATE</command> or
<command>DELETE</command> will cause a cardinality violation; the latter behavior
@@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable>
<title>Examples</title>
<para>
- Perform maintenance on CustomerAccounts based upon new Transactions.
+ Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>.
<programlisting>
MERGE CustomerAccount CA
@@ -599,7 +605,7 @@ WHEN MATCHED THEN
DELETE;
</programlisting>
- The wine_stock_changes table might be, for example, a temporary table
+ The <literal>wine_stock_changes</literal> table might be, for example, a temporary table
recently loaded into the database.
</para>
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index fbaf50c258..f914a00e9f 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors,
break;
case CMD_MERGE:
operation = "Merge";
- foperation = "Foreign Merge";
+ foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */
break;
default:
operation = "???";
diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c
index 0e245e1361..a7492a6c4b 100644
--- a/src/backend/executor/execMerge.c
+++ b/src/backend/executor/execMerge.c
@@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo,
* A concurrent update can:
*
* 1. modify the target tuple so that it no longer satisfies the
- * additional quals attached to the current WHEN MATCHED action OR
+ * additional quals attached to the current WHEN MATCHED action
*
* In this case, we are still dealing with a WHEN MATCHED case, but we
* should recheck the list of WHEN MATCHED actions and choose the first
@@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo,
* tuple, so we now instead find a qualifying WHEN NOT MATCHED action to
* execute.
*
+ * XXX Hmmm, what if the updated tuple would now match one that was
+ * considered NOT MATCHED so far?
+ *
* A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED.
*
* ExecMergeMatched takes care of following the update chain and
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index c0a97eba91..9c14709e47 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate)
{
/* advance to next subplan if any */
node->mt_whichplan++;
-
if (node->mt_whichplan < node->mt_nplans)
{
resultRelInfo++;
@@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate)
EvalPlanQualSetSlot(&node->mt_epqstate, planSlot);
slot = planSlot;
+ /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together
+ * with the other MT operations? This seems a bit out of place. */
if (operation == CMD_MERGE)
{
ExecMerge(node, resultRelInfo, estate, slot, junkfilter);
diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c
index 8848e9a03f..b2543f6814 100644
--- a/src/backend/optimizer/prep/preptlist.c
+++ b/src/backend/optimizer/prep/preptlist.c
@@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root)
tlist = expand_targetlist(tlist, command_type,
result_relation, target_relation);
+ /*
+ * For MERGE we need to handle the target list for the target relation,
+ * and also target list for each action (only INSERT/UPDATE matter).
+ */
if (command_type == CMD_MERGE)
{
ListCell *l;
@@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type,
* generate a NULL for dropped columns (we want to drop any old
* values).
*
+ * XXX Should this explain why MERGE has the same logic as UPDATE?
+ *
* When generating a NULL constant for a dropped column, we label
* it INT4 (any other guaranteed-to-exist datatype would do as
* well). We can't label it with the dropped column's datatype
diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c
index f7c152beb4..5d49c8c37e 100644
--- a/src/backend/parser/parse_agg.c
+++ b/src/backend/parser/parse_agg.c
@@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr)
break;
case EXPR_KIND_MERGE_WHEN_AND:
if (isAgg)
+ /* XXX "WHEN AND" seems rather strange. ... */
err = _("aggregate functions are not allowed in WHEN AND conditions");
else
err = _("grouping operations are not allowed in WHEN AND conditions");
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index d59f4fde95..d50543b1ba 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd)
/* Also update the sent position status in shared memory */
SpinLockAcquire(&MyWalSnd->mutex);
+ /* XXX Huh? Why does MERGE patch change walsender? */
MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush;
SpinLockRelease(&MyWalSnd->mutex);
@@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data)
replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ?
MyWalSnd->write : MyWalSnd->flush;
- if (WalSndCaughtUp &&
- sentPtr == replicatedPtr &&
+ if (WalSndCaughtUp && sentPtr == replicatedPtr &&
!pq_is_send_pending())
{
QueryCompletion qc;
diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c
index e2706c181c..80ae946d17 100644
--- a/src/backend/rewrite/rewriteHandler.c
+++ b/src/backend/rewrite/rewriteHandler.c
@@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation)
}
}
+
/*
* matchLocks -
* match the list of locks and returns the matching rules
@@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events)
/*
* XXX MERGE doesn't support write rules because they would violate
* the SQL Standard spec and would be unclear how they should work.
+ *
+ * XXX So does't support means 'ignores'? Should that either fail
+ * or at least print some warning?
*/
if (event == CMD_MERGE)
product_queries = NIL;
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c
index 0281351dcf..7a768a7b5b 100644
--- a/src/backend/utils/adt/varlena.c
+++ b/src/backend/utils/adt/varlena.c
@@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup)
int result;
bool arg1_match;
+ /* XXX Huh? Why is this in MERGE patch? */
if (len1 < 0 || len2 < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
@@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup)
memset(pres, 0, sizeof(Datum));
len = VARSIZE_ANY_EXHDR(authoritative);
+ /* XXX Huh? Why is this in MERGE patch? */
if (len < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
@@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2)
len1 = VARSIZE_ANY_EXHDR(t1);
len2 = VARSIZE_ANY_EXHDR(t2);
+ /* XXX Huh? Why is this in MERGE patch? */
if (len1 < 0 || len2 < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 3cba38044e..08fea9b8f7 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -172,6 +172,7 @@ typedef struct Query
List *withCheckOptions; /* a list of WithCheckOption's (added
* during rewrite) */
+ /* XXX Why not mergeTragetRelation? */
int mergeTarget_relation;
List *mergeSourceTargetList;
List *mergeActionList; /* list of actions for MERGE (only) */
@@ -1581,11 +1582,11 @@ typedef struct UpdateStmt
typedef struct MergeStmt
{
NodeTag type;
- RangeVar *relation; /* target relation to merge into */
+ RangeVar *relation; /* target relation to merge into */
Node *source_relation; /* source relation */
- Node *join_condition; /* join condition between source and target */
+ Node *join_condition; /* join condition between source and target */
List *mergeWhenClauses; /* list of MergeWhenClause(es) */
- WithClause *withClause; /* WITH clause */
+ WithClause *withClause; /* WITH clause */
} MergeStmt;
typedef struct MergeWhenClause
--
2.26.2
--------------B1AA4DC5EAE9BAF8B146244A
Content-Type: text/plain; charset=UTF-8;
name="0002-fix-valgrind-failure.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="0002-fix-valgrind-failure.txt"
^ permalink raw reply [nested|flat] 97+ messages in thread
* [PATCH 1/2] review
@ 2021-01-09 02:17 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw)
---
doc/src/sgml/mvcc.sgml | 20 +++++++++++---------
doc/src/sgml/ref/create_policy.sgml | 6 +++---
doc/src/sgml/ref/merge.sgml | 12 +++++++++---
src/backend/commands/explain.c | 2 +-
src/backend/executor/execMerge.c | 5 ++++-
src/backend/executor/nodeModifyTable.c | 3 ++-
src/backend/optimizer/prep/preptlist.c | 6 ++++++
src/backend/parser/parse_agg.c | 1 +
src/backend/replication/walsender.c | 4 ++--
src/backend/rewrite/rewriteHandler.c | 4 ++++
src/backend/utils/adt/varlena.c | 3 +++
src/include/nodes/parsenodes.h | 7 ++++---
12 files changed, 50 insertions(+), 23 deletions(-)
diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml
index 9ec8474185..02c9f3fdea 100644
--- a/doc/src/sgml/mvcc.sgml
+++ b/doc/src/sgml/mvcc.sgml
@@ -429,22 +429,24 @@ COMMIT;
with both <command>INSERT</command> and <command>UPDATE</command>
subcommands looks similar to <command>INSERT</command> with an
<literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee
- that either <command>INSERT</command> and <command>UPDATE</command> will occur.
+ that either <command>INSERT</command> or <command>UPDATE</command> will occur.
- If MERGE attempts an UPDATE or DELETE and the row is concurrently updated
+ /* XXX This is a very long and hard to understand sentence :-( */
+ /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */
+ If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated
but the join condition still passes for the current target and the current
- source tuple, then MERGE will behave the same as the UPDATE or DELETE commands
+ source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands
and perform its action on the latest version of the row, using standard
- EvalPlanQual. MERGE actions can be conditional, so conditions must be
+ EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be
re-evaluated on the latest row, starting from the first action.
On the other hand, if the row is concurrently updated or deleted so that
- the join condition fails, then MERGE will execute a NOT MATCHED action, if it
- exists and the AND WHEN qual evaluates to true.
+ the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it
+ exists and the <literal>AND WHEN</literal> qual evaluates to true.
- If MERGE attempts an INSERT and a unique index is present and a duplicate
- row is concurrently inserted then a uniqueness violation is raised. MERGE
- does not attempt to avoid the ERROR by attempting an UPDATE.
+ If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate
+ row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command>
+ does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>.
</para>
<para>
diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml
index ad20230c58..0a64674f2d 100644
--- a/doc/src/sgml/ref/create_policy.sgml
+++ b/doc/src/sgml/ref/create_policy.sgml
@@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable
<para>
No separate policy exists for <command>MERGE</command>. Instead policies
- defined for <literal>SELECT</literal>, <literal>INSERT</literal>,
- <literal>UPDATE</literal> and <literal>DELETE</literal> are applied
- while executing MERGE, depending on the actions that are activated.
+ defined for <command>SELECT</command>, <command>INSERT</command>,
+ <command>UPDATE</command> and <command>DELETE</command> are applied
+ while executing <command>MERGE</command>, depending on the actions that are activated.
</para>
</refsect1>
diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml
index b2a9f67cfa..c2901b0d58 100644
--- a/doc/src/sgml/ref/merge.sgml
+++ b/doc/src/sgml/ref/merge.sgml
@@ -98,7 +98,7 @@ DELETE
</para>
<para>
- There is no MERGE privilege.
+ There is no <literal>MERGE</literal> privilege.
You must have the <literal>UPDATE</literal> privilege on the column(s)
of the <replaceable class="parameter">target_table_name</replaceable>
referred to in the <literal>SET</literal> clause
@@ -217,6 +217,7 @@ DELETE
At least one <literal>WHEN</literal> clause is required.
</para>
<para>
+ /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */
If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal>
and the candidate change row matches a row in the
<replaceable class="parameter">target_table_name</replaceable>
@@ -242,6 +243,7 @@ DELETE
clause will be activated and the corresponding action will occur for
that row. The expression may not contain functions that possibly performs
writes to the database.
+ /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */
</para>
<para>
A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns
@@ -379,6 +381,7 @@ DELETE
<para>
An expression to assign to the column. The expression can use the
old values of this and other columns in the table.
+ /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */
</para>
</listitem>
</varlistentry>
@@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable>
In summary, statement triggers for an event type (say, INSERT) will
be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level
triggers will fire only for the one event type <emphasis>activated</emphasis>.
+ /* XXX What does "activated" mean? Maybe "executed" would be better? */
So a <command>MERGE</command> might fire statement triggers for both
<command>UPDATE</command> and <command>INSERT</command>, even though only
<command>UPDATE</command> row triggers were fired.
@@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable>
rows will be used to modify the target row, later attempts to modify will
cause an error. This can also occur if row triggers make changes to the
target table which are then subsequently modified by <command>MERGE</command>.
+ /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */
+ /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */
If the repeated action is an <command>INSERT</command> this will
cause a uniqueness violation while a repeated <command>UPDATE</command> or
<command>DELETE</command> will cause a cardinality violation; the latter behavior
@@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable>
<title>Examples</title>
<para>
- Perform maintenance on CustomerAccounts based upon new Transactions.
+ Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>.
<programlisting>
MERGE CustomerAccount CA
@@ -599,7 +605,7 @@ WHEN MATCHED THEN
DELETE;
</programlisting>
- The wine_stock_changes table might be, for example, a temporary table
+ The <literal>wine_stock_changes</literal> table might be, for example, a temporary table
recently loaded into the database.
</para>
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index fbaf50c258..f914a00e9f 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors,
break;
case CMD_MERGE:
operation = "Merge";
- foperation = "Foreign Merge";
+ foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */
break;
default:
operation = "???";
diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c
index 0e245e1361..a7492a6c4b 100644
--- a/src/backend/executor/execMerge.c
+++ b/src/backend/executor/execMerge.c
@@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo,
* A concurrent update can:
*
* 1. modify the target tuple so that it no longer satisfies the
- * additional quals attached to the current WHEN MATCHED action OR
+ * additional quals attached to the current WHEN MATCHED action
*
* In this case, we are still dealing with a WHEN MATCHED case, but we
* should recheck the list of WHEN MATCHED actions and choose the first
@@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo,
* tuple, so we now instead find a qualifying WHEN NOT MATCHED action to
* execute.
*
+ * XXX Hmmm, what if the updated tuple would now match one that was
+ * considered NOT MATCHED so far?
+ *
* A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED.
*
* ExecMergeMatched takes care of following the update chain and
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index c0a97eba91..9c14709e47 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate)
{
/* advance to next subplan if any */
node->mt_whichplan++;
-
if (node->mt_whichplan < node->mt_nplans)
{
resultRelInfo++;
@@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate)
EvalPlanQualSetSlot(&node->mt_epqstate, planSlot);
slot = planSlot;
+ /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together
+ * with the other MT operations? This seems a bit out of place. */
if (operation == CMD_MERGE)
{
ExecMerge(node, resultRelInfo, estate, slot, junkfilter);
diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c
index 8848e9a03f..b2543f6814 100644
--- a/src/backend/optimizer/prep/preptlist.c
+++ b/src/backend/optimizer/prep/preptlist.c
@@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root)
tlist = expand_targetlist(tlist, command_type,
result_relation, target_relation);
+ /*
+ * For MERGE we need to handle the target list for the target relation,
+ * and also target list for each action (only INSERT/UPDATE matter).
+ */
if (command_type == CMD_MERGE)
{
ListCell *l;
@@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type,
* generate a NULL for dropped columns (we want to drop any old
* values).
*
+ * XXX Should this explain why MERGE has the same logic as UPDATE?
+ *
* When generating a NULL constant for a dropped column, we label
* it INT4 (any other guaranteed-to-exist datatype would do as
* well). We can't label it with the dropped column's datatype
diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c
index f7c152beb4..5d49c8c37e 100644
--- a/src/backend/parser/parse_agg.c
+++ b/src/backend/parser/parse_agg.c
@@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr)
break;
case EXPR_KIND_MERGE_WHEN_AND:
if (isAgg)
+ /* XXX "WHEN AND" seems rather strange. ... */
err = _("aggregate functions are not allowed in WHEN AND conditions");
else
err = _("grouping operations are not allowed in WHEN AND conditions");
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index d59f4fde95..d50543b1ba 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd)
/* Also update the sent position status in shared memory */
SpinLockAcquire(&MyWalSnd->mutex);
+ /* XXX Huh? Why does MERGE patch change walsender? */
MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush;
SpinLockRelease(&MyWalSnd->mutex);
@@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data)
replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ?
MyWalSnd->write : MyWalSnd->flush;
- if (WalSndCaughtUp &&
- sentPtr == replicatedPtr &&
+ if (WalSndCaughtUp && sentPtr == replicatedPtr &&
!pq_is_send_pending())
{
QueryCompletion qc;
diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c
index e2706c181c..80ae946d17 100644
--- a/src/backend/rewrite/rewriteHandler.c
+++ b/src/backend/rewrite/rewriteHandler.c
@@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation)
}
}
+
/*
* matchLocks -
* match the list of locks and returns the matching rules
@@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events)
/*
* XXX MERGE doesn't support write rules because they would violate
* the SQL Standard spec and would be unclear how they should work.
+ *
+ * XXX So does't support means 'ignores'? Should that either fail
+ * or at least print some warning?
*/
if (event == CMD_MERGE)
product_queries = NIL;
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c
index 0281351dcf..7a768a7b5b 100644
--- a/src/backend/utils/adt/varlena.c
+++ b/src/backend/utils/adt/varlena.c
@@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup)
int result;
bool arg1_match;
+ /* XXX Huh? Why is this in MERGE patch? */
if (len1 < 0 || len2 < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
@@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup)
memset(pres, 0, sizeof(Datum));
len = VARSIZE_ANY_EXHDR(authoritative);
+ /* XXX Huh? Why is this in MERGE patch? */
if (len < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
@@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2)
len1 = VARSIZE_ANY_EXHDR(t1);
len2 = VARSIZE_ANY_EXHDR(t2);
+ /* XXX Huh? Why is this in MERGE patch? */
if (len1 < 0 || len2 < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 3cba38044e..08fea9b8f7 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -172,6 +172,7 @@ typedef struct Query
List *withCheckOptions; /* a list of WithCheckOption's (added
* during rewrite) */
+ /* XXX Why not mergeTragetRelation? */
int mergeTarget_relation;
List *mergeSourceTargetList;
List *mergeActionList; /* list of actions for MERGE (only) */
@@ -1581,11 +1582,11 @@ typedef struct UpdateStmt
typedef struct MergeStmt
{
NodeTag type;
- RangeVar *relation; /* target relation to merge into */
+ RangeVar *relation; /* target relation to merge into */
Node *source_relation; /* source relation */
- Node *join_condition; /* join condition between source and target */
+ Node *join_condition; /* join condition between source and target */
List *mergeWhenClauses; /* list of MergeWhenClause(es) */
- WithClause *withClause; /* WITH clause */
+ WithClause *withClause; /* WITH clause */
} MergeStmt;
typedef struct MergeWhenClause
--
2.26.2
--------------B1AA4DC5EAE9BAF8B146244A
Content-Type: text/plain; charset=UTF-8;
name="0002-fix-valgrind-failure.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="0002-fix-valgrind-failure.txt"
^ permalink raw reply [nested|flat] 97+ messages in thread
* [PATCH 1/2] review
@ 2021-01-09 02:17 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw)
---
doc/src/sgml/mvcc.sgml | 20 +++++++++++---------
doc/src/sgml/ref/create_policy.sgml | 6 +++---
doc/src/sgml/ref/merge.sgml | 12 +++++++++---
src/backend/commands/explain.c | 2 +-
src/backend/executor/execMerge.c | 5 ++++-
src/backend/executor/nodeModifyTable.c | 3 ++-
src/backend/optimizer/prep/preptlist.c | 6 ++++++
src/backend/parser/parse_agg.c | 1 +
src/backend/replication/walsender.c | 4 ++--
src/backend/rewrite/rewriteHandler.c | 4 ++++
src/backend/utils/adt/varlena.c | 3 +++
src/include/nodes/parsenodes.h | 7 ++++---
12 files changed, 50 insertions(+), 23 deletions(-)
diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml
index 9ec8474185..02c9f3fdea 100644
--- a/doc/src/sgml/mvcc.sgml
+++ b/doc/src/sgml/mvcc.sgml
@@ -429,22 +429,24 @@ COMMIT;
with both <command>INSERT</command> and <command>UPDATE</command>
subcommands looks similar to <command>INSERT</command> with an
<literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee
- that either <command>INSERT</command> and <command>UPDATE</command> will occur.
+ that either <command>INSERT</command> or <command>UPDATE</command> will occur.
- If MERGE attempts an UPDATE or DELETE and the row is concurrently updated
+ /* XXX This is a very long and hard to understand sentence :-( */
+ /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */
+ If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated
but the join condition still passes for the current target and the current
- source tuple, then MERGE will behave the same as the UPDATE or DELETE commands
+ source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands
and perform its action on the latest version of the row, using standard
- EvalPlanQual. MERGE actions can be conditional, so conditions must be
+ EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be
re-evaluated on the latest row, starting from the first action.
On the other hand, if the row is concurrently updated or deleted so that
- the join condition fails, then MERGE will execute a NOT MATCHED action, if it
- exists and the AND WHEN qual evaluates to true.
+ the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it
+ exists and the <literal>AND WHEN</literal> qual evaluates to true.
- If MERGE attempts an INSERT and a unique index is present and a duplicate
- row is concurrently inserted then a uniqueness violation is raised. MERGE
- does not attempt to avoid the ERROR by attempting an UPDATE.
+ If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate
+ row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command>
+ does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>.
</para>
<para>
diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml
index ad20230c58..0a64674f2d 100644
--- a/doc/src/sgml/ref/create_policy.sgml
+++ b/doc/src/sgml/ref/create_policy.sgml
@@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable
<para>
No separate policy exists for <command>MERGE</command>. Instead policies
- defined for <literal>SELECT</literal>, <literal>INSERT</literal>,
- <literal>UPDATE</literal> and <literal>DELETE</literal> are applied
- while executing MERGE, depending on the actions that are activated.
+ defined for <command>SELECT</command>, <command>INSERT</command>,
+ <command>UPDATE</command> and <command>DELETE</command> are applied
+ while executing <command>MERGE</command>, depending on the actions that are activated.
</para>
</refsect1>
diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml
index b2a9f67cfa..c2901b0d58 100644
--- a/doc/src/sgml/ref/merge.sgml
+++ b/doc/src/sgml/ref/merge.sgml
@@ -98,7 +98,7 @@ DELETE
</para>
<para>
- There is no MERGE privilege.
+ There is no <literal>MERGE</literal> privilege.
You must have the <literal>UPDATE</literal> privilege on the column(s)
of the <replaceable class="parameter">target_table_name</replaceable>
referred to in the <literal>SET</literal> clause
@@ -217,6 +217,7 @@ DELETE
At least one <literal>WHEN</literal> clause is required.
</para>
<para>
+ /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */
If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal>
and the candidate change row matches a row in the
<replaceable class="parameter">target_table_name</replaceable>
@@ -242,6 +243,7 @@ DELETE
clause will be activated and the corresponding action will occur for
that row. The expression may not contain functions that possibly performs
writes to the database.
+ /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */
</para>
<para>
A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns
@@ -379,6 +381,7 @@ DELETE
<para>
An expression to assign to the column. The expression can use the
old values of this and other columns in the table.
+ /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */
</para>
</listitem>
</varlistentry>
@@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable>
In summary, statement triggers for an event type (say, INSERT) will
be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level
triggers will fire only for the one event type <emphasis>activated</emphasis>.
+ /* XXX What does "activated" mean? Maybe "executed" would be better? */
So a <command>MERGE</command> might fire statement triggers for both
<command>UPDATE</command> and <command>INSERT</command>, even though only
<command>UPDATE</command> row triggers were fired.
@@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable>
rows will be used to modify the target row, later attempts to modify will
cause an error. This can also occur if row triggers make changes to the
target table which are then subsequently modified by <command>MERGE</command>.
+ /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */
+ /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */
If the repeated action is an <command>INSERT</command> this will
cause a uniqueness violation while a repeated <command>UPDATE</command> or
<command>DELETE</command> will cause a cardinality violation; the latter behavior
@@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable>
<title>Examples</title>
<para>
- Perform maintenance on CustomerAccounts based upon new Transactions.
+ Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>.
<programlisting>
MERGE CustomerAccount CA
@@ -599,7 +605,7 @@ WHEN MATCHED THEN
DELETE;
</programlisting>
- The wine_stock_changes table might be, for example, a temporary table
+ The <literal>wine_stock_changes</literal> table might be, for example, a temporary table
recently loaded into the database.
</para>
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index fbaf50c258..f914a00e9f 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors,
break;
case CMD_MERGE:
operation = "Merge";
- foperation = "Foreign Merge";
+ foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */
break;
default:
operation = "???";
diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c
index 0e245e1361..a7492a6c4b 100644
--- a/src/backend/executor/execMerge.c
+++ b/src/backend/executor/execMerge.c
@@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo,
* A concurrent update can:
*
* 1. modify the target tuple so that it no longer satisfies the
- * additional quals attached to the current WHEN MATCHED action OR
+ * additional quals attached to the current WHEN MATCHED action
*
* In this case, we are still dealing with a WHEN MATCHED case, but we
* should recheck the list of WHEN MATCHED actions and choose the first
@@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo,
* tuple, so we now instead find a qualifying WHEN NOT MATCHED action to
* execute.
*
+ * XXX Hmmm, what if the updated tuple would now match one that was
+ * considered NOT MATCHED so far?
+ *
* A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED.
*
* ExecMergeMatched takes care of following the update chain and
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index c0a97eba91..9c14709e47 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate)
{
/* advance to next subplan if any */
node->mt_whichplan++;
-
if (node->mt_whichplan < node->mt_nplans)
{
resultRelInfo++;
@@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate)
EvalPlanQualSetSlot(&node->mt_epqstate, planSlot);
slot = planSlot;
+ /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together
+ * with the other MT operations? This seems a bit out of place. */
if (operation == CMD_MERGE)
{
ExecMerge(node, resultRelInfo, estate, slot, junkfilter);
diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c
index 8848e9a03f..b2543f6814 100644
--- a/src/backend/optimizer/prep/preptlist.c
+++ b/src/backend/optimizer/prep/preptlist.c
@@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root)
tlist = expand_targetlist(tlist, command_type,
result_relation, target_relation);
+ /*
+ * For MERGE we need to handle the target list for the target relation,
+ * and also target list for each action (only INSERT/UPDATE matter).
+ */
if (command_type == CMD_MERGE)
{
ListCell *l;
@@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type,
* generate a NULL for dropped columns (we want to drop any old
* values).
*
+ * XXX Should this explain why MERGE has the same logic as UPDATE?
+ *
* When generating a NULL constant for a dropped column, we label
* it INT4 (any other guaranteed-to-exist datatype would do as
* well). We can't label it with the dropped column's datatype
diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c
index f7c152beb4..5d49c8c37e 100644
--- a/src/backend/parser/parse_agg.c
+++ b/src/backend/parser/parse_agg.c
@@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr)
break;
case EXPR_KIND_MERGE_WHEN_AND:
if (isAgg)
+ /* XXX "WHEN AND" seems rather strange. ... */
err = _("aggregate functions are not allowed in WHEN AND conditions");
else
err = _("grouping operations are not allowed in WHEN AND conditions");
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index d59f4fde95..d50543b1ba 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd)
/* Also update the sent position status in shared memory */
SpinLockAcquire(&MyWalSnd->mutex);
+ /* XXX Huh? Why does MERGE patch change walsender? */
MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush;
SpinLockRelease(&MyWalSnd->mutex);
@@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data)
replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ?
MyWalSnd->write : MyWalSnd->flush;
- if (WalSndCaughtUp &&
- sentPtr == replicatedPtr &&
+ if (WalSndCaughtUp && sentPtr == replicatedPtr &&
!pq_is_send_pending())
{
QueryCompletion qc;
diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c
index e2706c181c..80ae946d17 100644
--- a/src/backend/rewrite/rewriteHandler.c
+++ b/src/backend/rewrite/rewriteHandler.c
@@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation)
}
}
+
/*
* matchLocks -
* match the list of locks and returns the matching rules
@@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events)
/*
* XXX MERGE doesn't support write rules because they would violate
* the SQL Standard spec and would be unclear how they should work.
+ *
+ * XXX So does't support means 'ignores'? Should that either fail
+ * or at least print some warning?
*/
if (event == CMD_MERGE)
product_queries = NIL;
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c
index 0281351dcf..7a768a7b5b 100644
--- a/src/backend/utils/adt/varlena.c
+++ b/src/backend/utils/adt/varlena.c
@@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup)
int result;
bool arg1_match;
+ /* XXX Huh? Why is this in MERGE patch? */
if (len1 < 0 || len2 < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
@@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup)
memset(pres, 0, sizeof(Datum));
len = VARSIZE_ANY_EXHDR(authoritative);
+ /* XXX Huh? Why is this in MERGE patch? */
if (len < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
@@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2)
len1 = VARSIZE_ANY_EXHDR(t1);
len2 = VARSIZE_ANY_EXHDR(t2);
+ /* XXX Huh? Why is this in MERGE patch? */
if (len1 < 0 || len2 < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 3cba38044e..08fea9b8f7 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -172,6 +172,7 @@ typedef struct Query
List *withCheckOptions; /* a list of WithCheckOption's (added
* during rewrite) */
+ /* XXX Why not mergeTragetRelation? */
int mergeTarget_relation;
List *mergeSourceTargetList;
List *mergeActionList; /* list of actions for MERGE (only) */
@@ -1581,11 +1582,11 @@ typedef struct UpdateStmt
typedef struct MergeStmt
{
NodeTag type;
- RangeVar *relation; /* target relation to merge into */
+ RangeVar *relation; /* target relation to merge into */
Node *source_relation; /* source relation */
- Node *join_condition; /* join condition between source and target */
+ Node *join_condition; /* join condition between source and target */
List *mergeWhenClauses; /* list of MergeWhenClause(es) */
- WithClause *withClause; /* WITH clause */
+ WithClause *withClause; /* WITH clause */
} MergeStmt;
typedef struct MergeWhenClause
--
2.26.2
--------------B1AA4DC5EAE9BAF8B146244A
Content-Type: text/plain; charset=UTF-8;
name="0002-fix-valgrind-failure.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="0002-fix-valgrind-failure.txt"
^ permalink raw reply [nested|flat] 97+ messages in thread
* [PATCH 1/2] review
@ 2021-01-09 02:17 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw)
---
doc/src/sgml/mvcc.sgml | 20 +++++++++++---------
doc/src/sgml/ref/create_policy.sgml | 6 +++---
doc/src/sgml/ref/merge.sgml | 12 +++++++++---
src/backend/commands/explain.c | 2 +-
src/backend/executor/execMerge.c | 5 ++++-
src/backend/executor/nodeModifyTable.c | 3 ++-
src/backend/optimizer/prep/preptlist.c | 6 ++++++
src/backend/parser/parse_agg.c | 1 +
src/backend/replication/walsender.c | 4 ++--
src/backend/rewrite/rewriteHandler.c | 4 ++++
src/backend/utils/adt/varlena.c | 3 +++
src/include/nodes/parsenodes.h | 7 ++++---
12 files changed, 50 insertions(+), 23 deletions(-)
diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml
index 9ec8474185..02c9f3fdea 100644
--- a/doc/src/sgml/mvcc.sgml
+++ b/doc/src/sgml/mvcc.sgml
@@ -429,22 +429,24 @@ COMMIT;
with both <command>INSERT</command> and <command>UPDATE</command>
subcommands looks similar to <command>INSERT</command> with an
<literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee
- that either <command>INSERT</command> and <command>UPDATE</command> will occur.
+ that either <command>INSERT</command> or <command>UPDATE</command> will occur.
- If MERGE attempts an UPDATE or DELETE and the row is concurrently updated
+ /* XXX This is a very long and hard to understand sentence :-( */
+ /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */
+ If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated
but the join condition still passes for the current target and the current
- source tuple, then MERGE will behave the same as the UPDATE or DELETE commands
+ source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands
and perform its action on the latest version of the row, using standard
- EvalPlanQual. MERGE actions can be conditional, so conditions must be
+ EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be
re-evaluated on the latest row, starting from the first action.
On the other hand, if the row is concurrently updated or deleted so that
- the join condition fails, then MERGE will execute a NOT MATCHED action, if it
- exists and the AND WHEN qual evaluates to true.
+ the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it
+ exists and the <literal>AND WHEN</literal> qual evaluates to true.
- If MERGE attempts an INSERT and a unique index is present and a duplicate
- row is concurrently inserted then a uniqueness violation is raised. MERGE
- does not attempt to avoid the ERROR by attempting an UPDATE.
+ If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate
+ row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command>
+ does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>.
</para>
<para>
diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml
index ad20230c58..0a64674f2d 100644
--- a/doc/src/sgml/ref/create_policy.sgml
+++ b/doc/src/sgml/ref/create_policy.sgml
@@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable
<para>
No separate policy exists for <command>MERGE</command>. Instead policies
- defined for <literal>SELECT</literal>, <literal>INSERT</literal>,
- <literal>UPDATE</literal> and <literal>DELETE</literal> are applied
- while executing MERGE, depending on the actions that are activated.
+ defined for <command>SELECT</command>, <command>INSERT</command>,
+ <command>UPDATE</command> and <command>DELETE</command> are applied
+ while executing <command>MERGE</command>, depending on the actions that are activated.
</para>
</refsect1>
diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml
index b2a9f67cfa..c2901b0d58 100644
--- a/doc/src/sgml/ref/merge.sgml
+++ b/doc/src/sgml/ref/merge.sgml
@@ -98,7 +98,7 @@ DELETE
</para>
<para>
- There is no MERGE privilege.
+ There is no <literal>MERGE</literal> privilege.
You must have the <literal>UPDATE</literal> privilege on the column(s)
of the <replaceable class="parameter">target_table_name</replaceable>
referred to in the <literal>SET</literal> clause
@@ -217,6 +217,7 @@ DELETE
At least one <literal>WHEN</literal> clause is required.
</para>
<para>
+ /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */
If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal>
and the candidate change row matches a row in the
<replaceable class="parameter">target_table_name</replaceable>
@@ -242,6 +243,7 @@ DELETE
clause will be activated and the corresponding action will occur for
that row. The expression may not contain functions that possibly performs
writes to the database.
+ /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */
</para>
<para>
A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns
@@ -379,6 +381,7 @@ DELETE
<para>
An expression to assign to the column. The expression can use the
old values of this and other columns in the table.
+ /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */
</para>
</listitem>
</varlistentry>
@@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable>
In summary, statement triggers for an event type (say, INSERT) will
be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level
triggers will fire only for the one event type <emphasis>activated</emphasis>.
+ /* XXX What does "activated" mean? Maybe "executed" would be better? */
So a <command>MERGE</command> might fire statement triggers for both
<command>UPDATE</command> and <command>INSERT</command>, even though only
<command>UPDATE</command> row triggers were fired.
@@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable>
rows will be used to modify the target row, later attempts to modify will
cause an error. This can also occur if row triggers make changes to the
target table which are then subsequently modified by <command>MERGE</command>.
+ /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */
+ /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */
If the repeated action is an <command>INSERT</command> this will
cause a uniqueness violation while a repeated <command>UPDATE</command> or
<command>DELETE</command> will cause a cardinality violation; the latter behavior
@@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable>
<title>Examples</title>
<para>
- Perform maintenance on CustomerAccounts based upon new Transactions.
+ Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>.
<programlisting>
MERGE CustomerAccount CA
@@ -599,7 +605,7 @@ WHEN MATCHED THEN
DELETE;
</programlisting>
- The wine_stock_changes table might be, for example, a temporary table
+ The <literal>wine_stock_changes</literal> table might be, for example, a temporary table
recently loaded into the database.
</para>
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index fbaf50c258..f914a00e9f 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors,
break;
case CMD_MERGE:
operation = "Merge";
- foperation = "Foreign Merge";
+ foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */
break;
default:
operation = "???";
diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c
index 0e245e1361..a7492a6c4b 100644
--- a/src/backend/executor/execMerge.c
+++ b/src/backend/executor/execMerge.c
@@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo,
* A concurrent update can:
*
* 1. modify the target tuple so that it no longer satisfies the
- * additional quals attached to the current WHEN MATCHED action OR
+ * additional quals attached to the current WHEN MATCHED action
*
* In this case, we are still dealing with a WHEN MATCHED case, but we
* should recheck the list of WHEN MATCHED actions and choose the first
@@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo,
* tuple, so we now instead find a qualifying WHEN NOT MATCHED action to
* execute.
*
+ * XXX Hmmm, what if the updated tuple would now match one that was
+ * considered NOT MATCHED so far?
+ *
* A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED.
*
* ExecMergeMatched takes care of following the update chain and
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index c0a97eba91..9c14709e47 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate)
{
/* advance to next subplan if any */
node->mt_whichplan++;
-
if (node->mt_whichplan < node->mt_nplans)
{
resultRelInfo++;
@@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate)
EvalPlanQualSetSlot(&node->mt_epqstate, planSlot);
slot = planSlot;
+ /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together
+ * with the other MT operations? This seems a bit out of place. */
if (operation == CMD_MERGE)
{
ExecMerge(node, resultRelInfo, estate, slot, junkfilter);
diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c
index 8848e9a03f..b2543f6814 100644
--- a/src/backend/optimizer/prep/preptlist.c
+++ b/src/backend/optimizer/prep/preptlist.c
@@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root)
tlist = expand_targetlist(tlist, command_type,
result_relation, target_relation);
+ /*
+ * For MERGE we need to handle the target list for the target relation,
+ * and also target list for each action (only INSERT/UPDATE matter).
+ */
if (command_type == CMD_MERGE)
{
ListCell *l;
@@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type,
* generate a NULL for dropped columns (we want to drop any old
* values).
*
+ * XXX Should this explain why MERGE has the same logic as UPDATE?
+ *
* When generating a NULL constant for a dropped column, we label
* it INT4 (any other guaranteed-to-exist datatype would do as
* well). We can't label it with the dropped column's datatype
diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c
index f7c152beb4..5d49c8c37e 100644
--- a/src/backend/parser/parse_agg.c
+++ b/src/backend/parser/parse_agg.c
@@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr)
break;
case EXPR_KIND_MERGE_WHEN_AND:
if (isAgg)
+ /* XXX "WHEN AND" seems rather strange. ... */
err = _("aggregate functions are not allowed in WHEN AND conditions");
else
err = _("grouping operations are not allowed in WHEN AND conditions");
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index d59f4fde95..d50543b1ba 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd)
/* Also update the sent position status in shared memory */
SpinLockAcquire(&MyWalSnd->mutex);
+ /* XXX Huh? Why does MERGE patch change walsender? */
MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush;
SpinLockRelease(&MyWalSnd->mutex);
@@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data)
replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ?
MyWalSnd->write : MyWalSnd->flush;
- if (WalSndCaughtUp &&
- sentPtr == replicatedPtr &&
+ if (WalSndCaughtUp && sentPtr == replicatedPtr &&
!pq_is_send_pending())
{
QueryCompletion qc;
diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c
index e2706c181c..80ae946d17 100644
--- a/src/backend/rewrite/rewriteHandler.c
+++ b/src/backend/rewrite/rewriteHandler.c
@@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation)
}
}
+
/*
* matchLocks -
* match the list of locks and returns the matching rules
@@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events)
/*
* XXX MERGE doesn't support write rules because they would violate
* the SQL Standard spec and would be unclear how they should work.
+ *
+ * XXX So does't support means 'ignores'? Should that either fail
+ * or at least print some warning?
*/
if (event == CMD_MERGE)
product_queries = NIL;
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c
index 0281351dcf..7a768a7b5b 100644
--- a/src/backend/utils/adt/varlena.c
+++ b/src/backend/utils/adt/varlena.c
@@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup)
int result;
bool arg1_match;
+ /* XXX Huh? Why is this in MERGE patch? */
if (len1 < 0 || len2 < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
@@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup)
memset(pres, 0, sizeof(Datum));
len = VARSIZE_ANY_EXHDR(authoritative);
+ /* XXX Huh? Why is this in MERGE patch? */
if (len < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
@@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2)
len1 = VARSIZE_ANY_EXHDR(t1);
len2 = VARSIZE_ANY_EXHDR(t2);
+ /* XXX Huh? Why is this in MERGE patch? */
if (len1 < 0 || len2 < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 3cba38044e..08fea9b8f7 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -172,6 +172,7 @@ typedef struct Query
List *withCheckOptions; /* a list of WithCheckOption's (added
* during rewrite) */
+ /* XXX Why not mergeTragetRelation? */
int mergeTarget_relation;
List *mergeSourceTargetList;
List *mergeActionList; /* list of actions for MERGE (only) */
@@ -1581,11 +1582,11 @@ typedef struct UpdateStmt
typedef struct MergeStmt
{
NodeTag type;
- RangeVar *relation; /* target relation to merge into */
+ RangeVar *relation; /* target relation to merge into */
Node *source_relation; /* source relation */
- Node *join_condition; /* join condition between source and target */
+ Node *join_condition; /* join condition between source and target */
List *mergeWhenClauses; /* list of MergeWhenClause(es) */
- WithClause *withClause; /* WITH clause */
+ WithClause *withClause; /* WITH clause */
} MergeStmt;
typedef struct MergeWhenClause
--
2.26.2
--------------B1AA4DC5EAE9BAF8B146244A
Content-Type: text/plain; charset=UTF-8;
name="0002-fix-valgrind-failure.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="0002-fix-valgrind-failure.txt"
^ permalink raw reply [nested|flat] 97+ messages in thread
* [PATCH 1/2] review
@ 2021-01-09 02:17 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw)
---
doc/src/sgml/mvcc.sgml | 20 +++++++++++---------
doc/src/sgml/ref/create_policy.sgml | 6 +++---
doc/src/sgml/ref/merge.sgml | 12 +++++++++---
src/backend/commands/explain.c | 2 +-
src/backend/executor/execMerge.c | 5 ++++-
src/backend/executor/nodeModifyTable.c | 3 ++-
src/backend/optimizer/prep/preptlist.c | 6 ++++++
src/backend/parser/parse_agg.c | 1 +
src/backend/replication/walsender.c | 4 ++--
src/backend/rewrite/rewriteHandler.c | 4 ++++
src/backend/utils/adt/varlena.c | 3 +++
src/include/nodes/parsenodes.h | 7 ++++---
12 files changed, 50 insertions(+), 23 deletions(-)
diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml
index 9ec8474185..02c9f3fdea 100644
--- a/doc/src/sgml/mvcc.sgml
+++ b/doc/src/sgml/mvcc.sgml
@@ -429,22 +429,24 @@ COMMIT;
with both <command>INSERT</command> and <command>UPDATE</command>
subcommands looks similar to <command>INSERT</command> with an
<literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee
- that either <command>INSERT</command> and <command>UPDATE</command> will occur.
+ that either <command>INSERT</command> or <command>UPDATE</command> will occur.
- If MERGE attempts an UPDATE or DELETE and the row is concurrently updated
+ /* XXX This is a very long and hard to understand sentence :-( */
+ /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */
+ If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated
but the join condition still passes for the current target and the current
- source tuple, then MERGE will behave the same as the UPDATE or DELETE commands
+ source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands
and perform its action on the latest version of the row, using standard
- EvalPlanQual. MERGE actions can be conditional, so conditions must be
+ EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be
re-evaluated on the latest row, starting from the first action.
On the other hand, if the row is concurrently updated or deleted so that
- the join condition fails, then MERGE will execute a NOT MATCHED action, if it
- exists and the AND WHEN qual evaluates to true.
+ the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it
+ exists and the <literal>AND WHEN</literal> qual evaluates to true.
- If MERGE attempts an INSERT and a unique index is present and a duplicate
- row is concurrently inserted then a uniqueness violation is raised. MERGE
- does not attempt to avoid the ERROR by attempting an UPDATE.
+ If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate
+ row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command>
+ does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>.
</para>
<para>
diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml
index ad20230c58..0a64674f2d 100644
--- a/doc/src/sgml/ref/create_policy.sgml
+++ b/doc/src/sgml/ref/create_policy.sgml
@@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable
<para>
No separate policy exists for <command>MERGE</command>. Instead policies
- defined for <literal>SELECT</literal>, <literal>INSERT</literal>,
- <literal>UPDATE</literal> and <literal>DELETE</literal> are applied
- while executing MERGE, depending on the actions that are activated.
+ defined for <command>SELECT</command>, <command>INSERT</command>,
+ <command>UPDATE</command> and <command>DELETE</command> are applied
+ while executing <command>MERGE</command>, depending on the actions that are activated.
</para>
</refsect1>
diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml
index b2a9f67cfa..c2901b0d58 100644
--- a/doc/src/sgml/ref/merge.sgml
+++ b/doc/src/sgml/ref/merge.sgml
@@ -98,7 +98,7 @@ DELETE
</para>
<para>
- There is no MERGE privilege.
+ There is no <literal>MERGE</literal> privilege.
You must have the <literal>UPDATE</literal> privilege on the column(s)
of the <replaceable class="parameter">target_table_name</replaceable>
referred to in the <literal>SET</literal> clause
@@ -217,6 +217,7 @@ DELETE
At least one <literal>WHEN</literal> clause is required.
</para>
<para>
+ /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */
If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal>
and the candidate change row matches a row in the
<replaceable class="parameter">target_table_name</replaceable>
@@ -242,6 +243,7 @@ DELETE
clause will be activated and the corresponding action will occur for
that row. The expression may not contain functions that possibly performs
writes to the database.
+ /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */
</para>
<para>
A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns
@@ -379,6 +381,7 @@ DELETE
<para>
An expression to assign to the column. The expression can use the
old values of this and other columns in the table.
+ /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */
</para>
</listitem>
</varlistentry>
@@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable>
In summary, statement triggers for an event type (say, INSERT) will
be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level
triggers will fire only for the one event type <emphasis>activated</emphasis>.
+ /* XXX What does "activated" mean? Maybe "executed" would be better? */
So a <command>MERGE</command> might fire statement triggers for both
<command>UPDATE</command> and <command>INSERT</command>, even though only
<command>UPDATE</command> row triggers were fired.
@@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable>
rows will be used to modify the target row, later attempts to modify will
cause an error. This can also occur if row triggers make changes to the
target table which are then subsequently modified by <command>MERGE</command>.
+ /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */
+ /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */
If the repeated action is an <command>INSERT</command> this will
cause a uniqueness violation while a repeated <command>UPDATE</command> or
<command>DELETE</command> will cause a cardinality violation; the latter behavior
@@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable>
<title>Examples</title>
<para>
- Perform maintenance on CustomerAccounts based upon new Transactions.
+ Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>.
<programlisting>
MERGE CustomerAccount CA
@@ -599,7 +605,7 @@ WHEN MATCHED THEN
DELETE;
</programlisting>
- The wine_stock_changes table might be, for example, a temporary table
+ The <literal>wine_stock_changes</literal> table might be, for example, a temporary table
recently loaded into the database.
</para>
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index fbaf50c258..f914a00e9f 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors,
break;
case CMD_MERGE:
operation = "Merge";
- foperation = "Foreign Merge";
+ foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */
break;
default:
operation = "???";
diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c
index 0e245e1361..a7492a6c4b 100644
--- a/src/backend/executor/execMerge.c
+++ b/src/backend/executor/execMerge.c
@@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo,
* A concurrent update can:
*
* 1. modify the target tuple so that it no longer satisfies the
- * additional quals attached to the current WHEN MATCHED action OR
+ * additional quals attached to the current WHEN MATCHED action
*
* In this case, we are still dealing with a WHEN MATCHED case, but we
* should recheck the list of WHEN MATCHED actions and choose the first
@@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo,
* tuple, so we now instead find a qualifying WHEN NOT MATCHED action to
* execute.
*
+ * XXX Hmmm, what if the updated tuple would now match one that was
+ * considered NOT MATCHED so far?
+ *
* A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED.
*
* ExecMergeMatched takes care of following the update chain and
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index c0a97eba91..9c14709e47 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate)
{
/* advance to next subplan if any */
node->mt_whichplan++;
-
if (node->mt_whichplan < node->mt_nplans)
{
resultRelInfo++;
@@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate)
EvalPlanQualSetSlot(&node->mt_epqstate, planSlot);
slot = planSlot;
+ /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together
+ * with the other MT operations? This seems a bit out of place. */
if (operation == CMD_MERGE)
{
ExecMerge(node, resultRelInfo, estate, slot, junkfilter);
diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c
index 8848e9a03f..b2543f6814 100644
--- a/src/backend/optimizer/prep/preptlist.c
+++ b/src/backend/optimizer/prep/preptlist.c
@@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root)
tlist = expand_targetlist(tlist, command_type,
result_relation, target_relation);
+ /*
+ * For MERGE we need to handle the target list for the target relation,
+ * and also target list for each action (only INSERT/UPDATE matter).
+ */
if (command_type == CMD_MERGE)
{
ListCell *l;
@@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type,
* generate a NULL for dropped columns (we want to drop any old
* values).
*
+ * XXX Should this explain why MERGE has the same logic as UPDATE?
+ *
* When generating a NULL constant for a dropped column, we label
* it INT4 (any other guaranteed-to-exist datatype would do as
* well). We can't label it with the dropped column's datatype
diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c
index f7c152beb4..5d49c8c37e 100644
--- a/src/backend/parser/parse_agg.c
+++ b/src/backend/parser/parse_agg.c
@@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr)
break;
case EXPR_KIND_MERGE_WHEN_AND:
if (isAgg)
+ /* XXX "WHEN AND" seems rather strange. ... */
err = _("aggregate functions are not allowed in WHEN AND conditions");
else
err = _("grouping operations are not allowed in WHEN AND conditions");
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index d59f4fde95..d50543b1ba 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd)
/* Also update the sent position status in shared memory */
SpinLockAcquire(&MyWalSnd->mutex);
+ /* XXX Huh? Why does MERGE patch change walsender? */
MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush;
SpinLockRelease(&MyWalSnd->mutex);
@@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data)
replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ?
MyWalSnd->write : MyWalSnd->flush;
- if (WalSndCaughtUp &&
- sentPtr == replicatedPtr &&
+ if (WalSndCaughtUp && sentPtr == replicatedPtr &&
!pq_is_send_pending())
{
QueryCompletion qc;
diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c
index e2706c181c..80ae946d17 100644
--- a/src/backend/rewrite/rewriteHandler.c
+++ b/src/backend/rewrite/rewriteHandler.c
@@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation)
}
}
+
/*
* matchLocks -
* match the list of locks and returns the matching rules
@@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events)
/*
* XXX MERGE doesn't support write rules because they would violate
* the SQL Standard spec and would be unclear how they should work.
+ *
+ * XXX So does't support means 'ignores'? Should that either fail
+ * or at least print some warning?
*/
if (event == CMD_MERGE)
product_queries = NIL;
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c
index 0281351dcf..7a768a7b5b 100644
--- a/src/backend/utils/adt/varlena.c
+++ b/src/backend/utils/adt/varlena.c
@@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup)
int result;
bool arg1_match;
+ /* XXX Huh? Why is this in MERGE patch? */
if (len1 < 0 || len2 < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
@@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup)
memset(pres, 0, sizeof(Datum));
len = VARSIZE_ANY_EXHDR(authoritative);
+ /* XXX Huh? Why is this in MERGE patch? */
if (len < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
@@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2)
len1 = VARSIZE_ANY_EXHDR(t1);
len2 = VARSIZE_ANY_EXHDR(t2);
+ /* XXX Huh? Why is this in MERGE patch? */
if (len1 < 0 || len2 < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 3cba38044e..08fea9b8f7 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -172,6 +172,7 @@ typedef struct Query
List *withCheckOptions; /* a list of WithCheckOption's (added
* during rewrite) */
+ /* XXX Why not mergeTragetRelation? */
int mergeTarget_relation;
List *mergeSourceTargetList;
List *mergeActionList; /* list of actions for MERGE (only) */
@@ -1581,11 +1582,11 @@ typedef struct UpdateStmt
typedef struct MergeStmt
{
NodeTag type;
- RangeVar *relation; /* target relation to merge into */
+ RangeVar *relation; /* target relation to merge into */
Node *source_relation; /* source relation */
- Node *join_condition; /* join condition between source and target */
+ Node *join_condition; /* join condition between source and target */
List *mergeWhenClauses; /* list of MergeWhenClause(es) */
- WithClause *withClause; /* WITH clause */
+ WithClause *withClause; /* WITH clause */
} MergeStmt;
typedef struct MergeWhenClause
--
2.26.2
--------------B1AA4DC5EAE9BAF8B146244A
Content-Type: text/plain; charset=UTF-8;
name="0002-fix-valgrind-failure.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="0002-fix-valgrind-failure.txt"
^ permalink raw reply [nested|flat] 97+ messages in thread
* [PATCH 1/2] review
@ 2021-01-09 02:17 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw)
---
doc/src/sgml/mvcc.sgml | 20 +++++++++++---------
doc/src/sgml/ref/create_policy.sgml | 6 +++---
doc/src/sgml/ref/merge.sgml | 12 +++++++++---
src/backend/commands/explain.c | 2 +-
src/backend/executor/execMerge.c | 5 ++++-
src/backend/executor/nodeModifyTable.c | 3 ++-
src/backend/optimizer/prep/preptlist.c | 6 ++++++
src/backend/parser/parse_agg.c | 1 +
src/backend/replication/walsender.c | 4 ++--
src/backend/rewrite/rewriteHandler.c | 4 ++++
src/backend/utils/adt/varlena.c | 3 +++
src/include/nodes/parsenodes.h | 7 ++++---
12 files changed, 50 insertions(+), 23 deletions(-)
diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml
index 9ec8474185..02c9f3fdea 100644
--- a/doc/src/sgml/mvcc.sgml
+++ b/doc/src/sgml/mvcc.sgml
@@ -429,22 +429,24 @@ COMMIT;
with both <command>INSERT</command> and <command>UPDATE</command>
subcommands looks similar to <command>INSERT</command> with an
<literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee
- that either <command>INSERT</command> and <command>UPDATE</command> will occur.
+ that either <command>INSERT</command> or <command>UPDATE</command> will occur.
- If MERGE attempts an UPDATE or DELETE and the row is concurrently updated
+ /* XXX This is a very long and hard to understand sentence :-( */
+ /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */
+ If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated
but the join condition still passes for the current target and the current
- source tuple, then MERGE will behave the same as the UPDATE or DELETE commands
+ source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands
and perform its action on the latest version of the row, using standard
- EvalPlanQual. MERGE actions can be conditional, so conditions must be
+ EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be
re-evaluated on the latest row, starting from the first action.
On the other hand, if the row is concurrently updated or deleted so that
- the join condition fails, then MERGE will execute a NOT MATCHED action, if it
- exists and the AND WHEN qual evaluates to true.
+ the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it
+ exists and the <literal>AND WHEN</literal> qual evaluates to true.
- If MERGE attempts an INSERT and a unique index is present and a duplicate
- row is concurrently inserted then a uniqueness violation is raised. MERGE
- does not attempt to avoid the ERROR by attempting an UPDATE.
+ If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate
+ row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command>
+ does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>.
</para>
<para>
diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml
index ad20230c58..0a64674f2d 100644
--- a/doc/src/sgml/ref/create_policy.sgml
+++ b/doc/src/sgml/ref/create_policy.sgml
@@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable
<para>
No separate policy exists for <command>MERGE</command>. Instead policies
- defined for <literal>SELECT</literal>, <literal>INSERT</literal>,
- <literal>UPDATE</literal> and <literal>DELETE</literal> are applied
- while executing MERGE, depending on the actions that are activated.
+ defined for <command>SELECT</command>, <command>INSERT</command>,
+ <command>UPDATE</command> and <command>DELETE</command> are applied
+ while executing <command>MERGE</command>, depending on the actions that are activated.
</para>
</refsect1>
diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml
index b2a9f67cfa..c2901b0d58 100644
--- a/doc/src/sgml/ref/merge.sgml
+++ b/doc/src/sgml/ref/merge.sgml
@@ -98,7 +98,7 @@ DELETE
</para>
<para>
- There is no MERGE privilege.
+ There is no <literal>MERGE</literal> privilege.
You must have the <literal>UPDATE</literal> privilege on the column(s)
of the <replaceable class="parameter">target_table_name</replaceable>
referred to in the <literal>SET</literal> clause
@@ -217,6 +217,7 @@ DELETE
At least one <literal>WHEN</literal> clause is required.
</para>
<para>
+ /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */
If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal>
and the candidate change row matches a row in the
<replaceable class="parameter">target_table_name</replaceable>
@@ -242,6 +243,7 @@ DELETE
clause will be activated and the corresponding action will occur for
that row. The expression may not contain functions that possibly performs
writes to the database.
+ /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */
</para>
<para>
A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns
@@ -379,6 +381,7 @@ DELETE
<para>
An expression to assign to the column. The expression can use the
old values of this and other columns in the table.
+ /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */
</para>
</listitem>
</varlistentry>
@@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable>
In summary, statement triggers for an event type (say, INSERT) will
be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level
triggers will fire only for the one event type <emphasis>activated</emphasis>.
+ /* XXX What does "activated" mean? Maybe "executed" would be better? */
So a <command>MERGE</command> might fire statement triggers for both
<command>UPDATE</command> and <command>INSERT</command>, even though only
<command>UPDATE</command> row triggers were fired.
@@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable>
rows will be used to modify the target row, later attempts to modify will
cause an error. This can also occur if row triggers make changes to the
target table which are then subsequently modified by <command>MERGE</command>.
+ /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */
+ /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */
If the repeated action is an <command>INSERT</command> this will
cause a uniqueness violation while a repeated <command>UPDATE</command> or
<command>DELETE</command> will cause a cardinality violation; the latter behavior
@@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable>
<title>Examples</title>
<para>
- Perform maintenance on CustomerAccounts based upon new Transactions.
+ Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>.
<programlisting>
MERGE CustomerAccount CA
@@ -599,7 +605,7 @@ WHEN MATCHED THEN
DELETE;
</programlisting>
- The wine_stock_changes table might be, for example, a temporary table
+ The <literal>wine_stock_changes</literal> table might be, for example, a temporary table
recently loaded into the database.
</para>
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index fbaf50c258..f914a00e9f 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors,
break;
case CMD_MERGE:
operation = "Merge";
- foperation = "Foreign Merge";
+ foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */
break;
default:
operation = "???";
diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c
index 0e245e1361..a7492a6c4b 100644
--- a/src/backend/executor/execMerge.c
+++ b/src/backend/executor/execMerge.c
@@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo,
* A concurrent update can:
*
* 1. modify the target tuple so that it no longer satisfies the
- * additional quals attached to the current WHEN MATCHED action OR
+ * additional quals attached to the current WHEN MATCHED action
*
* In this case, we are still dealing with a WHEN MATCHED case, but we
* should recheck the list of WHEN MATCHED actions and choose the first
@@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo,
* tuple, so we now instead find a qualifying WHEN NOT MATCHED action to
* execute.
*
+ * XXX Hmmm, what if the updated tuple would now match one that was
+ * considered NOT MATCHED so far?
+ *
* A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED.
*
* ExecMergeMatched takes care of following the update chain and
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index c0a97eba91..9c14709e47 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate)
{
/* advance to next subplan if any */
node->mt_whichplan++;
-
if (node->mt_whichplan < node->mt_nplans)
{
resultRelInfo++;
@@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate)
EvalPlanQualSetSlot(&node->mt_epqstate, planSlot);
slot = planSlot;
+ /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together
+ * with the other MT operations? This seems a bit out of place. */
if (operation == CMD_MERGE)
{
ExecMerge(node, resultRelInfo, estate, slot, junkfilter);
diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c
index 8848e9a03f..b2543f6814 100644
--- a/src/backend/optimizer/prep/preptlist.c
+++ b/src/backend/optimizer/prep/preptlist.c
@@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root)
tlist = expand_targetlist(tlist, command_type,
result_relation, target_relation);
+ /*
+ * For MERGE we need to handle the target list for the target relation,
+ * and also target list for each action (only INSERT/UPDATE matter).
+ */
if (command_type == CMD_MERGE)
{
ListCell *l;
@@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type,
* generate a NULL for dropped columns (we want to drop any old
* values).
*
+ * XXX Should this explain why MERGE has the same logic as UPDATE?
+ *
* When generating a NULL constant for a dropped column, we label
* it INT4 (any other guaranteed-to-exist datatype would do as
* well). We can't label it with the dropped column's datatype
diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c
index f7c152beb4..5d49c8c37e 100644
--- a/src/backend/parser/parse_agg.c
+++ b/src/backend/parser/parse_agg.c
@@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr)
break;
case EXPR_KIND_MERGE_WHEN_AND:
if (isAgg)
+ /* XXX "WHEN AND" seems rather strange. ... */
err = _("aggregate functions are not allowed in WHEN AND conditions");
else
err = _("grouping operations are not allowed in WHEN AND conditions");
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index d59f4fde95..d50543b1ba 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd)
/* Also update the sent position status in shared memory */
SpinLockAcquire(&MyWalSnd->mutex);
+ /* XXX Huh? Why does MERGE patch change walsender? */
MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush;
SpinLockRelease(&MyWalSnd->mutex);
@@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data)
replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ?
MyWalSnd->write : MyWalSnd->flush;
- if (WalSndCaughtUp &&
- sentPtr == replicatedPtr &&
+ if (WalSndCaughtUp && sentPtr == replicatedPtr &&
!pq_is_send_pending())
{
QueryCompletion qc;
diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c
index e2706c181c..80ae946d17 100644
--- a/src/backend/rewrite/rewriteHandler.c
+++ b/src/backend/rewrite/rewriteHandler.c
@@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation)
}
}
+
/*
* matchLocks -
* match the list of locks and returns the matching rules
@@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events)
/*
* XXX MERGE doesn't support write rules because they would violate
* the SQL Standard spec and would be unclear how they should work.
+ *
+ * XXX So does't support means 'ignores'? Should that either fail
+ * or at least print some warning?
*/
if (event == CMD_MERGE)
product_queries = NIL;
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c
index 0281351dcf..7a768a7b5b 100644
--- a/src/backend/utils/adt/varlena.c
+++ b/src/backend/utils/adt/varlena.c
@@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup)
int result;
bool arg1_match;
+ /* XXX Huh? Why is this in MERGE patch? */
if (len1 < 0 || len2 < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
@@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup)
memset(pres, 0, sizeof(Datum));
len = VARSIZE_ANY_EXHDR(authoritative);
+ /* XXX Huh? Why is this in MERGE patch? */
if (len < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
@@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2)
len1 = VARSIZE_ANY_EXHDR(t1);
len2 = VARSIZE_ANY_EXHDR(t2);
+ /* XXX Huh? Why is this in MERGE patch? */
if (len1 < 0 || len2 < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 3cba38044e..08fea9b8f7 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -172,6 +172,7 @@ typedef struct Query
List *withCheckOptions; /* a list of WithCheckOption's (added
* during rewrite) */
+ /* XXX Why not mergeTragetRelation? */
int mergeTarget_relation;
List *mergeSourceTargetList;
List *mergeActionList; /* list of actions for MERGE (only) */
@@ -1581,11 +1582,11 @@ typedef struct UpdateStmt
typedef struct MergeStmt
{
NodeTag type;
- RangeVar *relation; /* target relation to merge into */
+ RangeVar *relation; /* target relation to merge into */
Node *source_relation; /* source relation */
- Node *join_condition; /* join condition between source and target */
+ Node *join_condition; /* join condition between source and target */
List *mergeWhenClauses; /* list of MergeWhenClause(es) */
- WithClause *withClause; /* WITH clause */
+ WithClause *withClause; /* WITH clause */
} MergeStmt;
typedef struct MergeWhenClause
--
2.26.2
--------------B1AA4DC5EAE9BAF8B146244A
Content-Type: text/plain; charset=UTF-8;
name="0002-fix-valgrind-failure.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="0002-fix-valgrind-failure.txt"
^ permalink raw reply [nested|flat] 97+ messages in thread
* [PATCH 1/2] review
@ 2021-01-09 02:17 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw)
---
doc/src/sgml/mvcc.sgml | 20 +++++++++++---------
doc/src/sgml/ref/create_policy.sgml | 6 +++---
doc/src/sgml/ref/merge.sgml | 12 +++++++++---
src/backend/commands/explain.c | 2 +-
src/backend/executor/execMerge.c | 5 ++++-
src/backend/executor/nodeModifyTable.c | 3 ++-
src/backend/optimizer/prep/preptlist.c | 6 ++++++
src/backend/parser/parse_agg.c | 1 +
src/backend/replication/walsender.c | 4 ++--
src/backend/rewrite/rewriteHandler.c | 4 ++++
src/backend/utils/adt/varlena.c | 3 +++
src/include/nodes/parsenodes.h | 7 ++++---
12 files changed, 50 insertions(+), 23 deletions(-)
diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml
index 9ec8474185..02c9f3fdea 100644
--- a/doc/src/sgml/mvcc.sgml
+++ b/doc/src/sgml/mvcc.sgml
@@ -429,22 +429,24 @@ COMMIT;
with both <command>INSERT</command> and <command>UPDATE</command>
subcommands looks similar to <command>INSERT</command> with an
<literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee
- that either <command>INSERT</command> and <command>UPDATE</command> will occur.
+ that either <command>INSERT</command> or <command>UPDATE</command> will occur.
- If MERGE attempts an UPDATE or DELETE and the row is concurrently updated
+ /* XXX This is a very long and hard to understand sentence :-( */
+ /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */
+ If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated
but the join condition still passes for the current target and the current
- source tuple, then MERGE will behave the same as the UPDATE or DELETE commands
+ source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands
and perform its action on the latest version of the row, using standard
- EvalPlanQual. MERGE actions can be conditional, so conditions must be
+ EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be
re-evaluated on the latest row, starting from the first action.
On the other hand, if the row is concurrently updated or deleted so that
- the join condition fails, then MERGE will execute a NOT MATCHED action, if it
- exists and the AND WHEN qual evaluates to true.
+ the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it
+ exists and the <literal>AND WHEN</literal> qual evaluates to true.
- If MERGE attempts an INSERT and a unique index is present and a duplicate
- row is concurrently inserted then a uniqueness violation is raised. MERGE
- does not attempt to avoid the ERROR by attempting an UPDATE.
+ If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate
+ row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command>
+ does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>.
</para>
<para>
diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml
index ad20230c58..0a64674f2d 100644
--- a/doc/src/sgml/ref/create_policy.sgml
+++ b/doc/src/sgml/ref/create_policy.sgml
@@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable
<para>
No separate policy exists for <command>MERGE</command>. Instead policies
- defined for <literal>SELECT</literal>, <literal>INSERT</literal>,
- <literal>UPDATE</literal> and <literal>DELETE</literal> are applied
- while executing MERGE, depending on the actions that are activated.
+ defined for <command>SELECT</command>, <command>INSERT</command>,
+ <command>UPDATE</command> and <command>DELETE</command> are applied
+ while executing <command>MERGE</command>, depending on the actions that are activated.
</para>
</refsect1>
diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml
index b2a9f67cfa..c2901b0d58 100644
--- a/doc/src/sgml/ref/merge.sgml
+++ b/doc/src/sgml/ref/merge.sgml
@@ -98,7 +98,7 @@ DELETE
</para>
<para>
- There is no MERGE privilege.
+ There is no <literal>MERGE</literal> privilege.
You must have the <literal>UPDATE</literal> privilege on the column(s)
of the <replaceable class="parameter">target_table_name</replaceable>
referred to in the <literal>SET</literal> clause
@@ -217,6 +217,7 @@ DELETE
At least one <literal>WHEN</literal> clause is required.
</para>
<para>
+ /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */
If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal>
and the candidate change row matches a row in the
<replaceable class="parameter">target_table_name</replaceable>
@@ -242,6 +243,7 @@ DELETE
clause will be activated and the corresponding action will occur for
that row. The expression may not contain functions that possibly performs
writes to the database.
+ /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */
</para>
<para>
A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns
@@ -379,6 +381,7 @@ DELETE
<para>
An expression to assign to the column. The expression can use the
old values of this and other columns in the table.
+ /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */
</para>
</listitem>
</varlistentry>
@@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable>
In summary, statement triggers for an event type (say, INSERT) will
be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level
triggers will fire only for the one event type <emphasis>activated</emphasis>.
+ /* XXX What does "activated" mean? Maybe "executed" would be better? */
So a <command>MERGE</command> might fire statement triggers for both
<command>UPDATE</command> and <command>INSERT</command>, even though only
<command>UPDATE</command> row triggers were fired.
@@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable>
rows will be used to modify the target row, later attempts to modify will
cause an error. This can also occur if row triggers make changes to the
target table which are then subsequently modified by <command>MERGE</command>.
+ /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */
+ /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */
If the repeated action is an <command>INSERT</command> this will
cause a uniqueness violation while a repeated <command>UPDATE</command> or
<command>DELETE</command> will cause a cardinality violation; the latter behavior
@@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable>
<title>Examples</title>
<para>
- Perform maintenance on CustomerAccounts based upon new Transactions.
+ Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>.
<programlisting>
MERGE CustomerAccount CA
@@ -599,7 +605,7 @@ WHEN MATCHED THEN
DELETE;
</programlisting>
- The wine_stock_changes table might be, for example, a temporary table
+ The <literal>wine_stock_changes</literal> table might be, for example, a temporary table
recently loaded into the database.
</para>
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index fbaf50c258..f914a00e9f 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors,
break;
case CMD_MERGE:
operation = "Merge";
- foperation = "Foreign Merge";
+ foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */
break;
default:
operation = "???";
diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c
index 0e245e1361..a7492a6c4b 100644
--- a/src/backend/executor/execMerge.c
+++ b/src/backend/executor/execMerge.c
@@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo,
* A concurrent update can:
*
* 1. modify the target tuple so that it no longer satisfies the
- * additional quals attached to the current WHEN MATCHED action OR
+ * additional quals attached to the current WHEN MATCHED action
*
* In this case, we are still dealing with a WHEN MATCHED case, but we
* should recheck the list of WHEN MATCHED actions and choose the first
@@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo,
* tuple, so we now instead find a qualifying WHEN NOT MATCHED action to
* execute.
*
+ * XXX Hmmm, what if the updated tuple would now match one that was
+ * considered NOT MATCHED so far?
+ *
* A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED.
*
* ExecMergeMatched takes care of following the update chain and
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index c0a97eba91..9c14709e47 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate)
{
/* advance to next subplan if any */
node->mt_whichplan++;
-
if (node->mt_whichplan < node->mt_nplans)
{
resultRelInfo++;
@@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate)
EvalPlanQualSetSlot(&node->mt_epqstate, planSlot);
slot = planSlot;
+ /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together
+ * with the other MT operations? This seems a bit out of place. */
if (operation == CMD_MERGE)
{
ExecMerge(node, resultRelInfo, estate, slot, junkfilter);
diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c
index 8848e9a03f..b2543f6814 100644
--- a/src/backend/optimizer/prep/preptlist.c
+++ b/src/backend/optimizer/prep/preptlist.c
@@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root)
tlist = expand_targetlist(tlist, command_type,
result_relation, target_relation);
+ /*
+ * For MERGE we need to handle the target list for the target relation,
+ * and also target list for each action (only INSERT/UPDATE matter).
+ */
if (command_type == CMD_MERGE)
{
ListCell *l;
@@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type,
* generate a NULL for dropped columns (we want to drop any old
* values).
*
+ * XXX Should this explain why MERGE has the same logic as UPDATE?
+ *
* When generating a NULL constant for a dropped column, we label
* it INT4 (any other guaranteed-to-exist datatype would do as
* well). We can't label it with the dropped column's datatype
diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c
index f7c152beb4..5d49c8c37e 100644
--- a/src/backend/parser/parse_agg.c
+++ b/src/backend/parser/parse_agg.c
@@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr)
break;
case EXPR_KIND_MERGE_WHEN_AND:
if (isAgg)
+ /* XXX "WHEN AND" seems rather strange. ... */
err = _("aggregate functions are not allowed in WHEN AND conditions");
else
err = _("grouping operations are not allowed in WHEN AND conditions");
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index d59f4fde95..d50543b1ba 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd)
/* Also update the sent position status in shared memory */
SpinLockAcquire(&MyWalSnd->mutex);
+ /* XXX Huh? Why does MERGE patch change walsender? */
MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush;
SpinLockRelease(&MyWalSnd->mutex);
@@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data)
replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ?
MyWalSnd->write : MyWalSnd->flush;
- if (WalSndCaughtUp &&
- sentPtr == replicatedPtr &&
+ if (WalSndCaughtUp && sentPtr == replicatedPtr &&
!pq_is_send_pending())
{
QueryCompletion qc;
diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c
index e2706c181c..80ae946d17 100644
--- a/src/backend/rewrite/rewriteHandler.c
+++ b/src/backend/rewrite/rewriteHandler.c
@@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation)
}
}
+
/*
* matchLocks -
* match the list of locks and returns the matching rules
@@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events)
/*
* XXX MERGE doesn't support write rules because they would violate
* the SQL Standard spec and would be unclear how they should work.
+ *
+ * XXX So does't support means 'ignores'? Should that either fail
+ * or at least print some warning?
*/
if (event == CMD_MERGE)
product_queries = NIL;
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c
index 0281351dcf..7a768a7b5b 100644
--- a/src/backend/utils/adt/varlena.c
+++ b/src/backend/utils/adt/varlena.c
@@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup)
int result;
bool arg1_match;
+ /* XXX Huh? Why is this in MERGE patch? */
if (len1 < 0 || len2 < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
@@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup)
memset(pres, 0, sizeof(Datum));
len = VARSIZE_ANY_EXHDR(authoritative);
+ /* XXX Huh? Why is this in MERGE patch? */
if (len < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
@@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2)
len1 = VARSIZE_ANY_EXHDR(t1);
len2 = VARSIZE_ANY_EXHDR(t2);
+ /* XXX Huh? Why is this in MERGE patch? */
if (len1 < 0 || len2 < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 3cba38044e..08fea9b8f7 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -172,6 +172,7 @@ typedef struct Query
List *withCheckOptions; /* a list of WithCheckOption's (added
* during rewrite) */
+ /* XXX Why not mergeTragetRelation? */
int mergeTarget_relation;
List *mergeSourceTargetList;
List *mergeActionList; /* list of actions for MERGE (only) */
@@ -1581,11 +1582,11 @@ typedef struct UpdateStmt
typedef struct MergeStmt
{
NodeTag type;
- RangeVar *relation; /* target relation to merge into */
+ RangeVar *relation; /* target relation to merge into */
Node *source_relation; /* source relation */
- Node *join_condition; /* join condition between source and target */
+ Node *join_condition; /* join condition between source and target */
List *mergeWhenClauses; /* list of MergeWhenClause(es) */
- WithClause *withClause; /* WITH clause */
+ WithClause *withClause; /* WITH clause */
} MergeStmt;
typedef struct MergeWhenClause
--
2.26.2
--------------B1AA4DC5EAE9BAF8B146244A
Content-Type: text/plain; charset=UTF-8;
name="0002-fix-valgrind-failure.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="0002-fix-valgrind-failure.txt"
^ permalink raw reply [nested|flat] 97+ messages in thread
* [PATCH 1/2] review
@ 2021-01-09 02:17 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw)
---
doc/src/sgml/mvcc.sgml | 20 +++++++++++---------
doc/src/sgml/ref/create_policy.sgml | 6 +++---
doc/src/sgml/ref/merge.sgml | 12 +++++++++---
src/backend/commands/explain.c | 2 +-
src/backend/executor/execMerge.c | 5 ++++-
src/backend/executor/nodeModifyTable.c | 3 ++-
src/backend/optimizer/prep/preptlist.c | 6 ++++++
src/backend/parser/parse_agg.c | 1 +
src/backend/replication/walsender.c | 4 ++--
src/backend/rewrite/rewriteHandler.c | 4 ++++
src/backend/utils/adt/varlena.c | 3 +++
src/include/nodes/parsenodes.h | 7 ++++---
12 files changed, 50 insertions(+), 23 deletions(-)
diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml
index 9ec8474185..02c9f3fdea 100644
--- a/doc/src/sgml/mvcc.sgml
+++ b/doc/src/sgml/mvcc.sgml
@@ -429,22 +429,24 @@ COMMIT;
with both <command>INSERT</command> and <command>UPDATE</command>
subcommands looks similar to <command>INSERT</command> with an
<literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee
- that either <command>INSERT</command> and <command>UPDATE</command> will occur.
+ that either <command>INSERT</command> or <command>UPDATE</command> will occur.
- If MERGE attempts an UPDATE or DELETE and the row is concurrently updated
+ /* XXX This is a very long and hard to understand sentence :-( */
+ /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */
+ If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated
but the join condition still passes for the current target and the current
- source tuple, then MERGE will behave the same as the UPDATE or DELETE commands
+ source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands
and perform its action on the latest version of the row, using standard
- EvalPlanQual. MERGE actions can be conditional, so conditions must be
+ EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be
re-evaluated on the latest row, starting from the first action.
On the other hand, if the row is concurrently updated or deleted so that
- the join condition fails, then MERGE will execute a NOT MATCHED action, if it
- exists and the AND WHEN qual evaluates to true.
+ the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it
+ exists and the <literal>AND WHEN</literal> qual evaluates to true.
- If MERGE attempts an INSERT and a unique index is present and a duplicate
- row is concurrently inserted then a uniqueness violation is raised. MERGE
- does not attempt to avoid the ERROR by attempting an UPDATE.
+ If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate
+ row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command>
+ does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>.
</para>
<para>
diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml
index ad20230c58..0a64674f2d 100644
--- a/doc/src/sgml/ref/create_policy.sgml
+++ b/doc/src/sgml/ref/create_policy.sgml
@@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable
<para>
No separate policy exists for <command>MERGE</command>. Instead policies
- defined for <literal>SELECT</literal>, <literal>INSERT</literal>,
- <literal>UPDATE</literal> and <literal>DELETE</literal> are applied
- while executing MERGE, depending on the actions that are activated.
+ defined for <command>SELECT</command>, <command>INSERT</command>,
+ <command>UPDATE</command> and <command>DELETE</command> are applied
+ while executing <command>MERGE</command>, depending on the actions that are activated.
</para>
</refsect1>
diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml
index b2a9f67cfa..c2901b0d58 100644
--- a/doc/src/sgml/ref/merge.sgml
+++ b/doc/src/sgml/ref/merge.sgml
@@ -98,7 +98,7 @@ DELETE
</para>
<para>
- There is no MERGE privilege.
+ There is no <literal>MERGE</literal> privilege.
You must have the <literal>UPDATE</literal> privilege on the column(s)
of the <replaceable class="parameter">target_table_name</replaceable>
referred to in the <literal>SET</literal> clause
@@ -217,6 +217,7 @@ DELETE
At least one <literal>WHEN</literal> clause is required.
</para>
<para>
+ /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */
If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal>
and the candidate change row matches a row in the
<replaceable class="parameter">target_table_name</replaceable>
@@ -242,6 +243,7 @@ DELETE
clause will be activated and the corresponding action will occur for
that row. The expression may not contain functions that possibly performs
writes to the database.
+ /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */
</para>
<para>
A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns
@@ -379,6 +381,7 @@ DELETE
<para>
An expression to assign to the column. The expression can use the
old values of this and other columns in the table.
+ /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */
</para>
</listitem>
</varlistentry>
@@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable>
In summary, statement triggers for an event type (say, INSERT) will
be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level
triggers will fire only for the one event type <emphasis>activated</emphasis>.
+ /* XXX What does "activated" mean? Maybe "executed" would be better? */
So a <command>MERGE</command> might fire statement triggers for both
<command>UPDATE</command> and <command>INSERT</command>, even though only
<command>UPDATE</command> row triggers were fired.
@@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable>
rows will be used to modify the target row, later attempts to modify will
cause an error. This can also occur if row triggers make changes to the
target table which are then subsequently modified by <command>MERGE</command>.
+ /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */
+ /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */
If the repeated action is an <command>INSERT</command> this will
cause a uniqueness violation while a repeated <command>UPDATE</command> or
<command>DELETE</command> will cause a cardinality violation; the latter behavior
@@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable>
<title>Examples</title>
<para>
- Perform maintenance on CustomerAccounts based upon new Transactions.
+ Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>.
<programlisting>
MERGE CustomerAccount CA
@@ -599,7 +605,7 @@ WHEN MATCHED THEN
DELETE;
</programlisting>
- The wine_stock_changes table might be, for example, a temporary table
+ The <literal>wine_stock_changes</literal> table might be, for example, a temporary table
recently loaded into the database.
</para>
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index fbaf50c258..f914a00e9f 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors,
break;
case CMD_MERGE:
operation = "Merge";
- foperation = "Foreign Merge";
+ foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */
break;
default:
operation = "???";
diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c
index 0e245e1361..a7492a6c4b 100644
--- a/src/backend/executor/execMerge.c
+++ b/src/backend/executor/execMerge.c
@@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo,
* A concurrent update can:
*
* 1. modify the target tuple so that it no longer satisfies the
- * additional quals attached to the current WHEN MATCHED action OR
+ * additional quals attached to the current WHEN MATCHED action
*
* In this case, we are still dealing with a WHEN MATCHED case, but we
* should recheck the list of WHEN MATCHED actions and choose the first
@@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo,
* tuple, so we now instead find a qualifying WHEN NOT MATCHED action to
* execute.
*
+ * XXX Hmmm, what if the updated tuple would now match one that was
+ * considered NOT MATCHED so far?
+ *
* A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED.
*
* ExecMergeMatched takes care of following the update chain and
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index c0a97eba91..9c14709e47 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate)
{
/* advance to next subplan if any */
node->mt_whichplan++;
-
if (node->mt_whichplan < node->mt_nplans)
{
resultRelInfo++;
@@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate)
EvalPlanQualSetSlot(&node->mt_epqstate, planSlot);
slot = planSlot;
+ /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together
+ * with the other MT operations? This seems a bit out of place. */
if (operation == CMD_MERGE)
{
ExecMerge(node, resultRelInfo, estate, slot, junkfilter);
diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c
index 8848e9a03f..b2543f6814 100644
--- a/src/backend/optimizer/prep/preptlist.c
+++ b/src/backend/optimizer/prep/preptlist.c
@@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root)
tlist = expand_targetlist(tlist, command_type,
result_relation, target_relation);
+ /*
+ * For MERGE we need to handle the target list for the target relation,
+ * and also target list for each action (only INSERT/UPDATE matter).
+ */
if (command_type == CMD_MERGE)
{
ListCell *l;
@@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type,
* generate a NULL for dropped columns (we want to drop any old
* values).
*
+ * XXX Should this explain why MERGE has the same logic as UPDATE?
+ *
* When generating a NULL constant for a dropped column, we label
* it INT4 (any other guaranteed-to-exist datatype would do as
* well). We can't label it with the dropped column's datatype
diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c
index f7c152beb4..5d49c8c37e 100644
--- a/src/backend/parser/parse_agg.c
+++ b/src/backend/parser/parse_agg.c
@@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr)
break;
case EXPR_KIND_MERGE_WHEN_AND:
if (isAgg)
+ /* XXX "WHEN AND" seems rather strange. ... */
err = _("aggregate functions are not allowed in WHEN AND conditions");
else
err = _("grouping operations are not allowed in WHEN AND conditions");
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index d59f4fde95..d50543b1ba 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd)
/* Also update the sent position status in shared memory */
SpinLockAcquire(&MyWalSnd->mutex);
+ /* XXX Huh? Why does MERGE patch change walsender? */
MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush;
SpinLockRelease(&MyWalSnd->mutex);
@@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data)
replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ?
MyWalSnd->write : MyWalSnd->flush;
- if (WalSndCaughtUp &&
- sentPtr == replicatedPtr &&
+ if (WalSndCaughtUp && sentPtr == replicatedPtr &&
!pq_is_send_pending())
{
QueryCompletion qc;
diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c
index e2706c181c..80ae946d17 100644
--- a/src/backend/rewrite/rewriteHandler.c
+++ b/src/backend/rewrite/rewriteHandler.c
@@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation)
}
}
+
/*
* matchLocks -
* match the list of locks and returns the matching rules
@@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events)
/*
* XXX MERGE doesn't support write rules because they would violate
* the SQL Standard spec and would be unclear how they should work.
+ *
+ * XXX So does't support means 'ignores'? Should that either fail
+ * or at least print some warning?
*/
if (event == CMD_MERGE)
product_queries = NIL;
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c
index 0281351dcf..7a768a7b5b 100644
--- a/src/backend/utils/adt/varlena.c
+++ b/src/backend/utils/adt/varlena.c
@@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup)
int result;
bool arg1_match;
+ /* XXX Huh? Why is this in MERGE patch? */
if (len1 < 0 || len2 < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
@@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup)
memset(pres, 0, sizeof(Datum));
len = VARSIZE_ANY_EXHDR(authoritative);
+ /* XXX Huh? Why is this in MERGE patch? */
if (len < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
@@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2)
len1 = VARSIZE_ANY_EXHDR(t1);
len2 = VARSIZE_ANY_EXHDR(t2);
+ /* XXX Huh? Why is this in MERGE patch? */
if (len1 < 0 || len2 < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 3cba38044e..08fea9b8f7 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -172,6 +172,7 @@ typedef struct Query
List *withCheckOptions; /* a list of WithCheckOption's (added
* during rewrite) */
+ /* XXX Why not mergeTragetRelation? */
int mergeTarget_relation;
List *mergeSourceTargetList;
List *mergeActionList; /* list of actions for MERGE (only) */
@@ -1581,11 +1582,11 @@ typedef struct UpdateStmt
typedef struct MergeStmt
{
NodeTag type;
- RangeVar *relation; /* target relation to merge into */
+ RangeVar *relation; /* target relation to merge into */
Node *source_relation; /* source relation */
- Node *join_condition; /* join condition between source and target */
+ Node *join_condition; /* join condition between source and target */
List *mergeWhenClauses; /* list of MergeWhenClause(es) */
- WithClause *withClause; /* WITH clause */
+ WithClause *withClause; /* WITH clause */
} MergeStmt;
typedef struct MergeWhenClause
--
2.26.2
--------------B1AA4DC5EAE9BAF8B146244A
Content-Type: text/plain; charset=UTF-8;
name="0002-fix-valgrind-failure.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="0002-fix-valgrind-failure.txt"
^ permalink raw reply [nested|flat] 97+ messages in thread
* [PATCH 1/2] review
@ 2021-01-09 02:17 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw)
---
doc/src/sgml/mvcc.sgml | 20 +++++++++++---------
doc/src/sgml/ref/create_policy.sgml | 6 +++---
doc/src/sgml/ref/merge.sgml | 12 +++++++++---
src/backend/commands/explain.c | 2 +-
src/backend/executor/execMerge.c | 5 ++++-
src/backend/executor/nodeModifyTable.c | 3 ++-
src/backend/optimizer/prep/preptlist.c | 6 ++++++
src/backend/parser/parse_agg.c | 1 +
src/backend/replication/walsender.c | 4 ++--
src/backend/rewrite/rewriteHandler.c | 4 ++++
src/backend/utils/adt/varlena.c | 3 +++
src/include/nodes/parsenodes.h | 7 ++++---
12 files changed, 50 insertions(+), 23 deletions(-)
diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml
index 9ec8474185..02c9f3fdea 100644
--- a/doc/src/sgml/mvcc.sgml
+++ b/doc/src/sgml/mvcc.sgml
@@ -429,22 +429,24 @@ COMMIT;
with both <command>INSERT</command> and <command>UPDATE</command>
subcommands looks similar to <command>INSERT</command> with an
<literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee
- that either <command>INSERT</command> and <command>UPDATE</command> will occur.
+ that either <command>INSERT</command> or <command>UPDATE</command> will occur.
- If MERGE attempts an UPDATE or DELETE and the row is concurrently updated
+ /* XXX This is a very long and hard to understand sentence :-( */
+ /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */
+ If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated
but the join condition still passes for the current target and the current
- source tuple, then MERGE will behave the same as the UPDATE or DELETE commands
+ source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands
and perform its action on the latest version of the row, using standard
- EvalPlanQual. MERGE actions can be conditional, so conditions must be
+ EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be
re-evaluated on the latest row, starting from the first action.
On the other hand, if the row is concurrently updated or deleted so that
- the join condition fails, then MERGE will execute a NOT MATCHED action, if it
- exists and the AND WHEN qual evaluates to true.
+ the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it
+ exists and the <literal>AND WHEN</literal> qual evaluates to true.
- If MERGE attempts an INSERT and a unique index is present and a duplicate
- row is concurrently inserted then a uniqueness violation is raised. MERGE
- does not attempt to avoid the ERROR by attempting an UPDATE.
+ If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate
+ row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command>
+ does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>.
</para>
<para>
diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml
index ad20230c58..0a64674f2d 100644
--- a/doc/src/sgml/ref/create_policy.sgml
+++ b/doc/src/sgml/ref/create_policy.sgml
@@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable
<para>
No separate policy exists for <command>MERGE</command>. Instead policies
- defined for <literal>SELECT</literal>, <literal>INSERT</literal>,
- <literal>UPDATE</literal> and <literal>DELETE</literal> are applied
- while executing MERGE, depending on the actions that are activated.
+ defined for <command>SELECT</command>, <command>INSERT</command>,
+ <command>UPDATE</command> and <command>DELETE</command> are applied
+ while executing <command>MERGE</command>, depending on the actions that are activated.
</para>
</refsect1>
diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml
index b2a9f67cfa..c2901b0d58 100644
--- a/doc/src/sgml/ref/merge.sgml
+++ b/doc/src/sgml/ref/merge.sgml
@@ -98,7 +98,7 @@ DELETE
</para>
<para>
- There is no MERGE privilege.
+ There is no <literal>MERGE</literal> privilege.
You must have the <literal>UPDATE</literal> privilege on the column(s)
of the <replaceable class="parameter">target_table_name</replaceable>
referred to in the <literal>SET</literal> clause
@@ -217,6 +217,7 @@ DELETE
At least one <literal>WHEN</literal> clause is required.
</para>
<para>
+ /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */
If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal>
and the candidate change row matches a row in the
<replaceable class="parameter">target_table_name</replaceable>
@@ -242,6 +243,7 @@ DELETE
clause will be activated and the corresponding action will occur for
that row. The expression may not contain functions that possibly performs
writes to the database.
+ /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */
</para>
<para>
A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns
@@ -379,6 +381,7 @@ DELETE
<para>
An expression to assign to the column. The expression can use the
old values of this and other columns in the table.
+ /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */
</para>
</listitem>
</varlistentry>
@@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable>
In summary, statement triggers for an event type (say, INSERT) will
be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level
triggers will fire only for the one event type <emphasis>activated</emphasis>.
+ /* XXX What does "activated" mean? Maybe "executed" would be better? */
So a <command>MERGE</command> might fire statement triggers for both
<command>UPDATE</command> and <command>INSERT</command>, even though only
<command>UPDATE</command> row triggers were fired.
@@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable>
rows will be used to modify the target row, later attempts to modify will
cause an error. This can also occur if row triggers make changes to the
target table which are then subsequently modified by <command>MERGE</command>.
+ /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */
+ /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */
If the repeated action is an <command>INSERT</command> this will
cause a uniqueness violation while a repeated <command>UPDATE</command> or
<command>DELETE</command> will cause a cardinality violation; the latter behavior
@@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable>
<title>Examples</title>
<para>
- Perform maintenance on CustomerAccounts based upon new Transactions.
+ Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>.
<programlisting>
MERGE CustomerAccount CA
@@ -599,7 +605,7 @@ WHEN MATCHED THEN
DELETE;
</programlisting>
- The wine_stock_changes table might be, for example, a temporary table
+ The <literal>wine_stock_changes</literal> table might be, for example, a temporary table
recently loaded into the database.
</para>
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index fbaf50c258..f914a00e9f 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors,
break;
case CMD_MERGE:
operation = "Merge";
- foperation = "Foreign Merge";
+ foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */
break;
default:
operation = "???";
diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c
index 0e245e1361..a7492a6c4b 100644
--- a/src/backend/executor/execMerge.c
+++ b/src/backend/executor/execMerge.c
@@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo,
* A concurrent update can:
*
* 1. modify the target tuple so that it no longer satisfies the
- * additional quals attached to the current WHEN MATCHED action OR
+ * additional quals attached to the current WHEN MATCHED action
*
* In this case, we are still dealing with a WHEN MATCHED case, but we
* should recheck the list of WHEN MATCHED actions and choose the first
@@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo,
* tuple, so we now instead find a qualifying WHEN NOT MATCHED action to
* execute.
*
+ * XXX Hmmm, what if the updated tuple would now match one that was
+ * considered NOT MATCHED so far?
+ *
* A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED.
*
* ExecMergeMatched takes care of following the update chain and
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index c0a97eba91..9c14709e47 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate)
{
/* advance to next subplan if any */
node->mt_whichplan++;
-
if (node->mt_whichplan < node->mt_nplans)
{
resultRelInfo++;
@@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate)
EvalPlanQualSetSlot(&node->mt_epqstate, planSlot);
slot = planSlot;
+ /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together
+ * with the other MT operations? This seems a bit out of place. */
if (operation == CMD_MERGE)
{
ExecMerge(node, resultRelInfo, estate, slot, junkfilter);
diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c
index 8848e9a03f..b2543f6814 100644
--- a/src/backend/optimizer/prep/preptlist.c
+++ b/src/backend/optimizer/prep/preptlist.c
@@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root)
tlist = expand_targetlist(tlist, command_type,
result_relation, target_relation);
+ /*
+ * For MERGE we need to handle the target list for the target relation,
+ * and also target list for each action (only INSERT/UPDATE matter).
+ */
if (command_type == CMD_MERGE)
{
ListCell *l;
@@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type,
* generate a NULL for dropped columns (we want to drop any old
* values).
*
+ * XXX Should this explain why MERGE has the same logic as UPDATE?
+ *
* When generating a NULL constant for a dropped column, we label
* it INT4 (any other guaranteed-to-exist datatype would do as
* well). We can't label it with the dropped column's datatype
diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c
index f7c152beb4..5d49c8c37e 100644
--- a/src/backend/parser/parse_agg.c
+++ b/src/backend/parser/parse_agg.c
@@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr)
break;
case EXPR_KIND_MERGE_WHEN_AND:
if (isAgg)
+ /* XXX "WHEN AND" seems rather strange. ... */
err = _("aggregate functions are not allowed in WHEN AND conditions");
else
err = _("grouping operations are not allowed in WHEN AND conditions");
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index d59f4fde95..d50543b1ba 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd)
/* Also update the sent position status in shared memory */
SpinLockAcquire(&MyWalSnd->mutex);
+ /* XXX Huh? Why does MERGE patch change walsender? */
MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush;
SpinLockRelease(&MyWalSnd->mutex);
@@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data)
replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ?
MyWalSnd->write : MyWalSnd->flush;
- if (WalSndCaughtUp &&
- sentPtr == replicatedPtr &&
+ if (WalSndCaughtUp && sentPtr == replicatedPtr &&
!pq_is_send_pending())
{
QueryCompletion qc;
diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c
index e2706c181c..80ae946d17 100644
--- a/src/backend/rewrite/rewriteHandler.c
+++ b/src/backend/rewrite/rewriteHandler.c
@@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation)
}
}
+
/*
* matchLocks -
* match the list of locks and returns the matching rules
@@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events)
/*
* XXX MERGE doesn't support write rules because they would violate
* the SQL Standard spec and would be unclear how they should work.
+ *
+ * XXX So does't support means 'ignores'? Should that either fail
+ * or at least print some warning?
*/
if (event == CMD_MERGE)
product_queries = NIL;
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c
index 0281351dcf..7a768a7b5b 100644
--- a/src/backend/utils/adt/varlena.c
+++ b/src/backend/utils/adt/varlena.c
@@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup)
int result;
bool arg1_match;
+ /* XXX Huh? Why is this in MERGE patch? */
if (len1 < 0 || len2 < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
@@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup)
memset(pres, 0, sizeof(Datum));
len = VARSIZE_ANY_EXHDR(authoritative);
+ /* XXX Huh? Why is this in MERGE patch? */
if (len < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
@@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2)
len1 = VARSIZE_ANY_EXHDR(t1);
len2 = VARSIZE_ANY_EXHDR(t2);
+ /* XXX Huh? Why is this in MERGE patch? */
if (len1 < 0 || len2 < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 3cba38044e..08fea9b8f7 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -172,6 +172,7 @@ typedef struct Query
List *withCheckOptions; /* a list of WithCheckOption's (added
* during rewrite) */
+ /* XXX Why not mergeTragetRelation? */
int mergeTarget_relation;
List *mergeSourceTargetList;
List *mergeActionList; /* list of actions for MERGE (only) */
@@ -1581,11 +1582,11 @@ typedef struct UpdateStmt
typedef struct MergeStmt
{
NodeTag type;
- RangeVar *relation; /* target relation to merge into */
+ RangeVar *relation; /* target relation to merge into */
Node *source_relation; /* source relation */
- Node *join_condition; /* join condition between source and target */
+ Node *join_condition; /* join condition between source and target */
List *mergeWhenClauses; /* list of MergeWhenClause(es) */
- WithClause *withClause; /* WITH clause */
+ WithClause *withClause; /* WITH clause */
} MergeStmt;
typedef struct MergeWhenClause
--
2.26.2
--------------B1AA4DC5EAE9BAF8B146244A
Content-Type: text/plain; charset=UTF-8;
name="0002-fix-valgrind-failure.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="0002-fix-valgrind-failure.txt"
^ permalink raw reply [nested|flat] 97+ messages in thread
* [PATCH 1/2] review
@ 2021-01-09 02:17 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw)
---
doc/src/sgml/mvcc.sgml | 20 +++++++++++---------
doc/src/sgml/ref/create_policy.sgml | 6 +++---
doc/src/sgml/ref/merge.sgml | 12 +++++++++---
src/backend/commands/explain.c | 2 +-
src/backend/executor/execMerge.c | 5 ++++-
src/backend/executor/nodeModifyTable.c | 3 ++-
src/backend/optimizer/prep/preptlist.c | 6 ++++++
src/backend/parser/parse_agg.c | 1 +
src/backend/replication/walsender.c | 4 ++--
src/backend/rewrite/rewriteHandler.c | 4 ++++
src/backend/utils/adt/varlena.c | 3 +++
src/include/nodes/parsenodes.h | 7 ++++---
12 files changed, 50 insertions(+), 23 deletions(-)
diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml
index 9ec8474185..02c9f3fdea 100644
--- a/doc/src/sgml/mvcc.sgml
+++ b/doc/src/sgml/mvcc.sgml
@@ -429,22 +429,24 @@ COMMIT;
with both <command>INSERT</command> and <command>UPDATE</command>
subcommands looks similar to <command>INSERT</command> with an
<literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee
- that either <command>INSERT</command> and <command>UPDATE</command> will occur.
+ that either <command>INSERT</command> or <command>UPDATE</command> will occur.
- If MERGE attempts an UPDATE or DELETE and the row is concurrently updated
+ /* XXX This is a very long and hard to understand sentence :-( */
+ /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */
+ If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated
but the join condition still passes for the current target and the current
- source tuple, then MERGE will behave the same as the UPDATE or DELETE commands
+ source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands
and perform its action on the latest version of the row, using standard
- EvalPlanQual. MERGE actions can be conditional, so conditions must be
+ EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be
re-evaluated on the latest row, starting from the first action.
On the other hand, if the row is concurrently updated or deleted so that
- the join condition fails, then MERGE will execute a NOT MATCHED action, if it
- exists and the AND WHEN qual evaluates to true.
+ the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it
+ exists and the <literal>AND WHEN</literal> qual evaluates to true.
- If MERGE attempts an INSERT and a unique index is present and a duplicate
- row is concurrently inserted then a uniqueness violation is raised. MERGE
- does not attempt to avoid the ERROR by attempting an UPDATE.
+ If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate
+ row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command>
+ does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>.
</para>
<para>
diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml
index ad20230c58..0a64674f2d 100644
--- a/doc/src/sgml/ref/create_policy.sgml
+++ b/doc/src/sgml/ref/create_policy.sgml
@@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable
<para>
No separate policy exists for <command>MERGE</command>. Instead policies
- defined for <literal>SELECT</literal>, <literal>INSERT</literal>,
- <literal>UPDATE</literal> and <literal>DELETE</literal> are applied
- while executing MERGE, depending on the actions that are activated.
+ defined for <command>SELECT</command>, <command>INSERT</command>,
+ <command>UPDATE</command> and <command>DELETE</command> are applied
+ while executing <command>MERGE</command>, depending on the actions that are activated.
</para>
</refsect1>
diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml
index b2a9f67cfa..c2901b0d58 100644
--- a/doc/src/sgml/ref/merge.sgml
+++ b/doc/src/sgml/ref/merge.sgml
@@ -98,7 +98,7 @@ DELETE
</para>
<para>
- There is no MERGE privilege.
+ There is no <literal>MERGE</literal> privilege.
You must have the <literal>UPDATE</literal> privilege on the column(s)
of the <replaceable class="parameter">target_table_name</replaceable>
referred to in the <literal>SET</literal> clause
@@ -217,6 +217,7 @@ DELETE
At least one <literal>WHEN</literal> clause is required.
</para>
<para>
+ /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */
If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal>
and the candidate change row matches a row in the
<replaceable class="parameter">target_table_name</replaceable>
@@ -242,6 +243,7 @@ DELETE
clause will be activated and the corresponding action will occur for
that row. The expression may not contain functions that possibly performs
writes to the database.
+ /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */
</para>
<para>
A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns
@@ -379,6 +381,7 @@ DELETE
<para>
An expression to assign to the column. The expression can use the
old values of this and other columns in the table.
+ /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */
</para>
</listitem>
</varlistentry>
@@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable>
In summary, statement triggers for an event type (say, INSERT) will
be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level
triggers will fire only for the one event type <emphasis>activated</emphasis>.
+ /* XXX What does "activated" mean? Maybe "executed" would be better? */
So a <command>MERGE</command> might fire statement triggers for both
<command>UPDATE</command> and <command>INSERT</command>, even though only
<command>UPDATE</command> row triggers were fired.
@@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable>
rows will be used to modify the target row, later attempts to modify will
cause an error. This can also occur if row triggers make changes to the
target table which are then subsequently modified by <command>MERGE</command>.
+ /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */
+ /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */
If the repeated action is an <command>INSERT</command> this will
cause a uniqueness violation while a repeated <command>UPDATE</command> or
<command>DELETE</command> will cause a cardinality violation; the latter behavior
@@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable>
<title>Examples</title>
<para>
- Perform maintenance on CustomerAccounts based upon new Transactions.
+ Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>.
<programlisting>
MERGE CustomerAccount CA
@@ -599,7 +605,7 @@ WHEN MATCHED THEN
DELETE;
</programlisting>
- The wine_stock_changes table might be, for example, a temporary table
+ The <literal>wine_stock_changes</literal> table might be, for example, a temporary table
recently loaded into the database.
</para>
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index fbaf50c258..f914a00e9f 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors,
break;
case CMD_MERGE:
operation = "Merge";
- foperation = "Foreign Merge";
+ foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */
break;
default:
operation = "???";
diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c
index 0e245e1361..a7492a6c4b 100644
--- a/src/backend/executor/execMerge.c
+++ b/src/backend/executor/execMerge.c
@@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo,
* A concurrent update can:
*
* 1. modify the target tuple so that it no longer satisfies the
- * additional quals attached to the current WHEN MATCHED action OR
+ * additional quals attached to the current WHEN MATCHED action
*
* In this case, we are still dealing with a WHEN MATCHED case, but we
* should recheck the list of WHEN MATCHED actions and choose the first
@@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo,
* tuple, so we now instead find a qualifying WHEN NOT MATCHED action to
* execute.
*
+ * XXX Hmmm, what if the updated tuple would now match one that was
+ * considered NOT MATCHED so far?
+ *
* A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED.
*
* ExecMergeMatched takes care of following the update chain and
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index c0a97eba91..9c14709e47 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate)
{
/* advance to next subplan if any */
node->mt_whichplan++;
-
if (node->mt_whichplan < node->mt_nplans)
{
resultRelInfo++;
@@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate)
EvalPlanQualSetSlot(&node->mt_epqstate, planSlot);
slot = planSlot;
+ /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together
+ * with the other MT operations? This seems a bit out of place. */
if (operation == CMD_MERGE)
{
ExecMerge(node, resultRelInfo, estate, slot, junkfilter);
diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c
index 8848e9a03f..b2543f6814 100644
--- a/src/backend/optimizer/prep/preptlist.c
+++ b/src/backend/optimizer/prep/preptlist.c
@@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root)
tlist = expand_targetlist(tlist, command_type,
result_relation, target_relation);
+ /*
+ * For MERGE we need to handle the target list for the target relation,
+ * and also target list for each action (only INSERT/UPDATE matter).
+ */
if (command_type == CMD_MERGE)
{
ListCell *l;
@@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type,
* generate a NULL for dropped columns (we want to drop any old
* values).
*
+ * XXX Should this explain why MERGE has the same logic as UPDATE?
+ *
* When generating a NULL constant for a dropped column, we label
* it INT4 (any other guaranteed-to-exist datatype would do as
* well). We can't label it with the dropped column's datatype
diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c
index f7c152beb4..5d49c8c37e 100644
--- a/src/backend/parser/parse_agg.c
+++ b/src/backend/parser/parse_agg.c
@@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr)
break;
case EXPR_KIND_MERGE_WHEN_AND:
if (isAgg)
+ /* XXX "WHEN AND" seems rather strange. ... */
err = _("aggregate functions are not allowed in WHEN AND conditions");
else
err = _("grouping operations are not allowed in WHEN AND conditions");
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index d59f4fde95..d50543b1ba 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd)
/* Also update the sent position status in shared memory */
SpinLockAcquire(&MyWalSnd->mutex);
+ /* XXX Huh? Why does MERGE patch change walsender? */
MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush;
SpinLockRelease(&MyWalSnd->mutex);
@@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data)
replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ?
MyWalSnd->write : MyWalSnd->flush;
- if (WalSndCaughtUp &&
- sentPtr == replicatedPtr &&
+ if (WalSndCaughtUp && sentPtr == replicatedPtr &&
!pq_is_send_pending())
{
QueryCompletion qc;
diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c
index e2706c181c..80ae946d17 100644
--- a/src/backend/rewrite/rewriteHandler.c
+++ b/src/backend/rewrite/rewriteHandler.c
@@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation)
}
}
+
/*
* matchLocks -
* match the list of locks and returns the matching rules
@@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events)
/*
* XXX MERGE doesn't support write rules because they would violate
* the SQL Standard spec and would be unclear how they should work.
+ *
+ * XXX So does't support means 'ignores'? Should that either fail
+ * or at least print some warning?
*/
if (event == CMD_MERGE)
product_queries = NIL;
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c
index 0281351dcf..7a768a7b5b 100644
--- a/src/backend/utils/adt/varlena.c
+++ b/src/backend/utils/adt/varlena.c
@@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup)
int result;
bool arg1_match;
+ /* XXX Huh? Why is this in MERGE patch? */
if (len1 < 0 || len2 < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
@@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup)
memset(pres, 0, sizeof(Datum));
len = VARSIZE_ANY_EXHDR(authoritative);
+ /* XXX Huh? Why is this in MERGE patch? */
if (len < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
@@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2)
len1 = VARSIZE_ANY_EXHDR(t1);
len2 = VARSIZE_ANY_EXHDR(t2);
+ /* XXX Huh? Why is this in MERGE patch? */
if (len1 < 0 || len2 < 0)
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 3cba38044e..08fea9b8f7 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -172,6 +172,7 @@ typedef struct Query
List *withCheckOptions; /* a list of WithCheckOption's (added
* during rewrite) */
+ /* XXX Why not mergeTragetRelation? */
int mergeTarget_relation;
List *mergeSourceTargetList;
List *mergeActionList; /* list of actions for MERGE (only) */
@@ -1581,11 +1582,11 @@ typedef struct UpdateStmt
typedef struct MergeStmt
{
NodeTag type;
- RangeVar *relation; /* target relation to merge into */
+ RangeVar *relation; /* target relation to merge into */
Node *source_relation; /* source relation */
- Node *join_condition; /* join condition between source and target */
+ Node *join_condition; /* join condition between source and target */
List *mergeWhenClauses; /* list of MergeWhenClause(es) */
- WithClause *withClause; /* WITH clause */
+ WithClause *withClause; /* WITH clause */
} MergeStmt;
typedef struct MergeWhenClause
--
2.26.2
--------------B1AA4DC5EAE9BAF8B146244A
Content-Type: text/plain; charset=UTF-8;
name="0002-fix-valgrind-failure.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="0002-fix-valgrind-failure.txt"
^ permalink raw reply [nested|flat] 97+ messages in thread
* [PATCH 2/2] review
@ 2021-03-09 20:09 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw)
---
src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++-------
src/backend/optimizer/path/pathkeys.c | 14 ++++-
2 files changed, 69 insertions(+), 22 deletions(-)
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index 1639258aaf..f55a4f20e5 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm)
* is_fake_var
* Workaround for generate_append_tlist() which generates fake Vars with
* varno == 0, that will cause a fail of estimate_num_group() call
+ *
+ * XXX Ummm, why would estimate_num_group fail with this?
*/
static bool
is_fake_var(Expr *expr)
@@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr)
* compute_cpu_sort_cost
* compute CPU cost of sort (i.e. in-memory)
*
+ * The main thing we need to calculate to estimate sort CPU costs is the number
+ * of calls to the comparator functions. The difficulty is that for multi-column
+ * sorts there may be different data types involved (for some of which the calls
+ * may be much more expensive). Furthermore, the columns may have very different
+ * number of distinct values - the higher the number, the fewer comparisons will
+ * be needed for the following columns.
+ *
+ * The algoritm is incremental - we add pathkeys one by one, and at each step we
+ * estimate the number of necessary comparisons (based on the number of distinct
+ * groups in the current pathkey prefix and the new pathkey), and the comparison
+ * costs (which is data type specific).
+ *
+ * Estimation of the number of comparisons is based on ideas from two sources:
+ *
+ * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/]
+ *
+ * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild,
+ * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906]
+ *
+ * In term of that paper, let N - number of tuples, Xi - number of tuples with
+ * key Ki, then the estimate of number of comparisons is:
+ *
+ * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi))
+ *
+ * In our case all Xi are the same because now we don't have any estimation of
+ * group sizes, we have only know the estimate of number of groups (distinct
+ * values). In that case, formula becomes:
+ *
+ * N * log(NumberOfGroups)
+ *
+ * For multi-column sorts we need to estimate the number of comparisons for
+ * each individual column - for example with columns (c1, c2, ..., ck) we
+ * can estimate that number of comparions on ck is roughly
+ *
+ * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1))
+ *
+ * Let k be a column number, Gk - number of groups defined by k columns, and Fk
+ * the cost of the comparison is
+ *
+ * N * sum( Fk * log(Gk) )
+ *
+ * Note: We also consider column witdth, not just the comparator cost.
+ *
* NOTE: some callers currently pass NIL for pathkeys because they
* can't conveniently supply the sort keys. In this case, it will fallback to
* simple comparison cost estimate.
- *
- * Estimation algorithm is based on ideas from course Algorithms,
- * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper
- * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild,
- * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017.
- *
- * In term of that papers, let N - number of tuples, Xi - number of tuples with
- * key Ki, then estimation is:
- * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi))
- * In our case all Xi are the same because now we don't have an estimation of
- * group sizes, we have only estimation of number of groups. In this case,
- * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation
- * of multi-column sort we need separately compute each column, so, let k is a
- * column number, Gk - number of groups defined by k columns:
- * N * sum( Fk * log(Gk) )
- * Fk is a function costs (including width) for k columns.
*/
-
static Cost
compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
Cost comparison_cost, double tuples, double output_tuples,
@@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
bool has_fake_var = false;
int i = 0;
Oid prev_datatype = InvalidOid;
- Cost funcCost = 0.;
+ Cost funcCost = 0.0;
List *cache_varinfos = NIL;
/* fallback if pathkeys is unknown */
@@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* a total number of tuple comparisons of N log2 K; but the constant
* factor is a bit higher than for quicksort. Tweak it so that the
* cost curve is continuous at the crossover point.
+ *
+ * XXX I suppose the "quicksort factor" references to 1.5 at the end
+ * of this function, but I'm not sure. I suggest we introduce some simple
+ * constants for that, instead of magic values.
*/
output_tuples = (heapSort) ? 2.0 * output_tuples : tuples;
per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples);
@@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* - per column comparison function cost
* - we try to compute needed number of comparison per column
*/
-
foreach(lc, pathkeys)
{
PathKey *pathkey = (PathKey*)lfirst(lc);
@@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* Don't use DEFAULT_NUM_DISTINCT because it used for only one
* column while here we try to estimate number of groups over
* set of columns.
+ *
+ * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to
+ * limit the calculated values, somehow?
+ *
+ * XXX What's the logic of the following formula?
*/
nGroups = ceil(2.0 + sqrt(tuples) *
list_length(pathkeyExprs) / list_length(pathkeys));
@@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
{
if (tuplesPerPrevGroup < output_tuples)
/* comparing only inside output_tuples */
+ /* XXX why not to use the same multiplier (1.5)? */
correctedNGroups =
ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups));
else
@@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups);
/*
- * We could skip all followed columns for cost estimation, because we
+ * We could skip all following columns for cost estimation, because we
* believe that tuples are unique by set ot previous columns
*/
if (tuplesPerPrevGroup <= 1.0)
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 7beb32488a..b092c3e055 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b)
return 0;
return 1;
}
+
/*
* Order tail of list of group pathkeys by uniqueness descendetly. It allows to
* speedup sorting. Returns newly allocated lists, old ones stay untouched.
* n_preordered defines a head of list which order should be prevented.
+ *
+ * XXX But we're not generating this only based on uniqueness (that's a bad
+ * term anyway, because we're using ndistinct estimates, not uniqueness).
+ * We're also using the comparator cost to calculate the expected sort cost,
+ * and optimize that.
+ *
+ * XXX This should explain how we generate the values - all permutations for
+ * up to 4 values, etc.
*/
void
get_cheapest_group_keys_order(PlannerInfo *root, double nrows,
@@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows,
else
{
/*
- * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to
- * a new one which can then be cleaned safely if needed.
+ * Since v13 list_free() can clean list elements so for original list
+ * not to be modified it should be copied to a new one which can then
+ * be cleaned safely if needed.
*/
new_group_pathkeys = list_copy(*group_pathkeys);
nToPermute = nFreeKeys;
--
2.29.2
--------------A955BFDEDE369DCD57A45C5A--
^ permalink raw reply [nested|flat] 97+ messages in thread
* [PATCH 2/2] review
@ 2021-03-09 20:09 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw)
---
src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++-------
src/backend/optimizer/path/pathkeys.c | 14 ++++-
2 files changed, 69 insertions(+), 22 deletions(-)
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index 1639258aaf..f55a4f20e5 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm)
* is_fake_var
* Workaround for generate_append_tlist() which generates fake Vars with
* varno == 0, that will cause a fail of estimate_num_group() call
+ *
+ * XXX Ummm, why would estimate_num_group fail with this?
*/
static bool
is_fake_var(Expr *expr)
@@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr)
* compute_cpu_sort_cost
* compute CPU cost of sort (i.e. in-memory)
*
+ * The main thing we need to calculate to estimate sort CPU costs is the number
+ * of calls to the comparator functions. The difficulty is that for multi-column
+ * sorts there may be different data types involved (for some of which the calls
+ * may be much more expensive). Furthermore, the columns may have very different
+ * number of distinct values - the higher the number, the fewer comparisons will
+ * be needed for the following columns.
+ *
+ * The algoritm is incremental - we add pathkeys one by one, and at each step we
+ * estimate the number of necessary comparisons (based on the number of distinct
+ * groups in the current pathkey prefix and the new pathkey), and the comparison
+ * costs (which is data type specific).
+ *
+ * Estimation of the number of comparisons is based on ideas from two sources:
+ *
+ * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/]
+ *
+ * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild,
+ * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906]
+ *
+ * In term of that paper, let N - number of tuples, Xi - number of tuples with
+ * key Ki, then the estimate of number of comparisons is:
+ *
+ * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi))
+ *
+ * In our case all Xi are the same because now we don't have any estimation of
+ * group sizes, we have only know the estimate of number of groups (distinct
+ * values). In that case, formula becomes:
+ *
+ * N * log(NumberOfGroups)
+ *
+ * For multi-column sorts we need to estimate the number of comparisons for
+ * each individual column - for example with columns (c1, c2, ..., ck) we
+ * can estimate that number of comparions on ck is roughly
+ *
+ * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1))
+ *
+ * Let k be a column number, Gk - number of groups defined by k columns, and Fk
+ * the cost of the comparison is
+ *
+ * N * sum( Fk * log(Gk) )
+ *
+ * Note: We also consider column witdth, not just the comparator cost.
+ *
* NOTE: some callers currently pass NIL for pathkeys because they
* can't conveniently supply the sort keys. In this case, it will fallback to
* simple comparison cost estimate.
- *
- * Estimation algorithm is based on ideas from course Algorithms,
- * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper
- * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild,
- * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017.
- *
- * In term of that papers, let N - number of tuples, Xi - number of tuples with
- * key Ki, then estimation is:
- * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi))
- * In our case all Xi are the same because now we don't have an estimation of
- * group sizes, we have only estimation of number of groups. In this case,
- * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation
- * of multi-column sort we need separately compute each column, so, let k is a
- * column number, Gk - number of groups defined by k columns:
- * N * sum( Fk * log(Gk) )
- * Fk is a function costs (including width) for k columns.
*/
-
static Cost
compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
Cost comparison_cost, double tuples, double output_tuples,
@@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
bool has_fake_var = false;
int i = 0;
Oid prev_datatype = InvalidOid;
- Cost funcCost = 0.;
+ Cost funcCost = 0.0;
List *cache_varinfos = NIL;
/* fallback if pathkeys is unknown */
@@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* a total number of tuple comparisons of N log2 K; but the constant
* factor is a bit higher than for quicksort. Tweak it so that the
* cost curve is continuous at the crossover point.
+ *
+ * XXX I suppose the "quicksort factor" references to 1.5 at the end
+ * of this function, but I'm not sure. I suggest we introduce some simple
+ * constants for that, instead of magic values.
*/
output_tuples = (heapSort) ? 2.0 * output_tuples : tuples;
per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples);
@@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* - per column comparison function cost
* - we try to compute needed number of comparison per column
*/
-
foreach(lc, pathkeys)
{
PathKey *pathkey = (PathKey*)lfirst(lc);
@@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* Don't use DEFAULT_NUM_DISTINCT because it used for only one
* column while here we try to estimate number of groups over
* set of columns.
+ *
+ * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to
+ * limit the calculated values, somehow?
+ *
+ * XXX What's the logic of the following formula?
*/
nGroups = ceil(2.0 + sqrt(tuples) *
list_length(pathkeyExprs) / list_length(pathkeys));
@@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
{
if (tuplesPerPrevGroup < output_tuples)
/* comparing only inside output_tuples */
+ /* XXX why not to use the same multiplier (1.5)? */
correctedNGroups =
ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups));
else
@@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups);
/*
- * We could skip all followed columns for cost estimation, because we
+ * We could skip all following columns for cost estimation, because we
* believe that tuples are unique by set ot previous columns
*/
if (tuplesPerPrevGroup <= 1.0)
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 7beb32488a..b092c3e055 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b)
return 0;
return 1;
}
+
/*
* Order tail of list of group pathkeys by uniqueness descendetly. It allows to
* speedup sorting. Returns newly allocated lists, old ones stay untouched.
* n_preordered defines a head of list which order should be prevented.
+ *
+ * XXX But we're not generating this only based on uniqueness (that's a bad
+ * term anyway, because we're using ndistinct estimates, not uniqueness).
+ * We're also using the comparator cost to calculate the expected sort cost,
+ * and optimize that.
+ *
+ * XXX This should explain how we generate the values - all permutations for
+ * up to 4 values, etc.
*/
void
get_cheapest_group_keys_order(PlannerInfo *root, double nrows,
@@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows,
else
{
/*
- * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to
- * a new one which can then be cleaned safely if needed.
+ * Since v13 list_free() can clean list elements so for original list
+ * not to be modified it should be copied to a new one which can then
+ * be cleaned safely if needed.
*/
new_group_pathkeys = list_copy(*group_pathkeys);
nToPermute = nFreeKeys;
--
2.29.2
--------------A955BFDEDE369DCD57A45C5A--
^ permalink raw reply [nested|flat] 97+ messages in thread
* [PATCH 2/2] review
@ 2021-03-09 20:09 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw)
---
src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++-------
src/backend/optimizer/path/pathkeys.c | 14 ++++-
2 files changed, 69 insertions(+), 22 deletions(-)
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index 1639258aaf..f55a4f20e5 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm)
* is_fake_var
* Workaround for generate_append_tlist() which generates fake Vars with
* varno == 0, that will cause a fail of estimate_num_group() call
+ *
+ * XXX Ummm, why would estimate_num_group fail with this?
*/
static bool
is_fake_var(Expr *expr)
@@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr)
* compute_cpu_sort_cost
* compute CPU cost of sort (i.e. in-memory)
*
+ * The main thing we need to calculate to estimate sort CPU costs is the number
+ * of calls to the comparator functions. The difficulty is that for multi-column
+ * sorts there may be different data types involved (for some of which the calls
+ * may be much more expensive). Furthermore, the columns may have very different
+ * number of distinct values - the higher the number, the fewer comparisons will
+ * be needed for the following columns.
+ *
+ * The algoritm is incremental - we add pathkeys one by one, and at each step we
+ * estimate the number of necessary comparisons (based on the number of distinct
+ * groups in the current pathkey prefix and the new pathkey), and the comparison
+ * costs (which is data type specific).
+ *
+ * Estimation of the number of comparisons is based on ideas from two sources:
+ *
+ * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/]
+ *
+ * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild,
+ * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906]
+ *
+ * In term of that paper, let N - number of tuples, Xi - number of tuples with
+ * key Ki, then the estimate of number of comparisons is:
+ *
+ * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi))
+ *
+ * In our case all Xi are the same because now we don't have any estimation of
+ * group sizes, we have only know the estimate of number of groups (distinct
+ * values). In that case, formula becomes:
+ *
+ * N * log(NumberOfGroups)
+ *
+ * For multi-column sorts we need to estimate the number of comparisons for
+ * each individual column - for example with columns (c1, c2, ..., ck) we
+ * can estimate that number of comparions on ck is roughly
+ *
+ * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1))
+ *
+ * Let k be a column number, Gk - number of groups defined by k columns, and Fk
+ * the cost of the comparison is
+ *
+ * N * sum( Fk * log(Gk) )
+ *
+ * Note: We also consider column witdth, not just the comparator cost.
+ *
* NOTE: some callers currently pass NIL for pathkeys because they
* can't conveniently supply the sort keys. In this case, it will fallback to
* simple comparison cost estimate.
- *
- * Estimation algorithm is based on ideas from course Algorithms,
- * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper
- * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild,
- * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017.
- *
- * In term of that papers, let N - number of tuples, Xi - number of tuples with
- * key Ki, then estimation is:
- * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi))
- * In our case all Xi are the same because now we don't have an estimation of
- * group sizes, we have only estimation of number of groups. In this case,
- * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation
- * of multi-column sort we need separately compute each column, so, let k is a
- * column number, Gk - number of groups defined by k columns:
- * N * sum( Fk * log(Gk) )
- * Fk is a function costs (including width) for k columns.
*/
-
static Cost
compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
Cost comparison_cost, double tuples, double output_tuples,
@@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
bool has_fake_var = false;
int i = 0;
Oid prev_datatype = InvalidOid;
- Cost funcCost = 0.;
+ Cost funcCost = 0.0;
List *cache_varinfos = NIL;
/* fallback if pathkeys is unknown */
@@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* a total number of tuple comparisons of N log2 K; but the constant
* factor is a bit higher than for quicksort. Tweak it so that the
* cost curve is continuous at the crossover point.
+ *
+ * XXX I suppose the "quicksort factor" references to 1.5 at the end
+ * of this function, but I'm not sure. I suggest we introduce some simple
+ * constants for that, instead of magic values.
*/
output_tuples = (heapSort) ? 2.0 * output_tuples : tuples;
per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples);
@@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* - per column comparison function cost
* - we try to compute needed number of comparison per column
*/
-
foreach(lc, pathkeys)
{
PathKey *pathkey = (PathKey*)lfirst(lc);
@@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* Don't use DEFAULT_NUM_DISTINCT because it used for only one
* column while here we try to estimate number of groups over
* set of columns.
+ *
+ * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to
+ * limit the calculated values, somehow?
+ *
+ * XXX What's the logic of the following formula?
*/
nGroups = ceil(2.0 + sqrt(tuples) *
list_length(pathkeyExprs) / list_length(pathkeys));
@@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
{
if (tuplesPerPrevGroup < output_tuples)
/* comparing only inside output_tuples */
+ /* XXX why not to use the same multiplier (1.5)? */
correctedNGroups =
ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups));
else
@@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups);
/*
- * We could skip all followed columns for cost estimation, because we
+ * We could skip all following columns for cost estimation, because we
* believe that tuples are unique by set ot previous columns
*/
if (tuplesPerPrevGroup <= 1.0)
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 7beb32488a..b092c3e055 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b)
return 0;
return 1;
}
+
/*
* Order tail of list of group pathkeys by uniqueness descendetly. It allows to
* speedup sorting. Returns newly allocated lists, old ones stay untouched.
* n_preordered defines a head of list which order should be prevented.
+ *
+ * XXX But we're not generating this only based on uniqueness (that's a bad
+ * term anyway, because we're using ndistinct estimates, not uniqueness).
+ * We're also using the comparator cost to calculate the expected sort cost,
+ * and optimize that.
+ *
+ * XXX This should explain how we generate the values - all permutations for
+ * up to 4 values, etc.
*/
void
get_cheapest_group_keys_order(PlannerInfo *root, double nrows,
@@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows,
else
{
/*
- * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to
- * a new one which can then be cleaned safely if needed.
+ * Since v13 list_free() can clean list elements so for original list
+ * not to be modified it should be copied to a new one which can then
+ * be cleaned safely if needed.
*/
new_group_pathkeys = list_copy(*group_pathkeys);
nToPermute = nFreeKeys;
--
2.29.2
--------------A955BFDEDE369DCD57A45C5A--
^ permalink raw reply [nested|flat] 97+ messages in thread
* [PATCH 2/2] review
@ 2021-03-09 20:09 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw)
---
src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++-------
src/backend/optimizer/path/pathkeys.c | 14 ++++-
2 files changed, 69 insertions(+), 22 deletions(-)
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index 1639258aaf..f55a4f20e5 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm)
* is_fake_var
* Workaround for generate_append_tlist() which generates fake Vars with
* varno == 0, that will cause a fail of estimate_num_group() call
+ *
+ * XXX Ummm, why would estimate_num_group fail with this?
*/
static bool
is_fake_var(Expr *expr)
@@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr)
* compute_cpu_sort_cost
* compute CPU cost of sort (i.e. in-memory)
*
+ * The main thing we need to calculate to estimate sort CPU costs is the number
+ * of calls to the comparator functions. The difficulty is that for multi-column
+ * sorts there may be different data types involved (for some of which the calls
+ * may be much more expensive). Furthermore, the columns may have very different
+ * number of distinct values - the higher the number, the fewer comparisons will
+ * be needed for the following columns.
+ *
+ * The algoritm is incremental - we add pathkeys one by one, and at each step we
+ * estimate the number of necessary comparisons (based on the number of distinct
+ * groups in the current pathkey prefix and the new pathkey), and the comparison
+ * costs (which is data type specific).
+ *
+ * Estimation of the number of comparisons is based on ideas from two sources:
+ *
+ * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/]
+ *
+ * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild,
+ * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906]
+ *
+ * In term of that paper, let N - number of tuples, Xi - number of tuples with
+ * key Ki, then the estimate of number of comparisons is:
+ *
+ * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi))
+ *
+ * In our case all Xi are the same because now we don't have any estimation of
+ * group sizes, we have only know the estimate of number of groups (distinct
+ * values). In that case, formula becomes:
+ *
+ * N * log(NumberOfGroups)
+ *
+ * For multi-column sorts we need to estimate the number of comparisons for
+ * each individual column - for example with columns (c1, c2, ..., ck) we
+ * can estimate that number of comparions on ck is roughly
+ *
+ * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1))
+ *
+ * Let k be a column number, Gk - number of groups defined by k columns, and Fk
+ * the cost of the comparison is
+ *
+ * N * sum( Fk * log(Gk) )
+ *
+ * Note: We also consider column witdth, not just the comparator cost.
+ *
* NOTE: some callers currently pass NIL for pathkeys because they
* can't conveniently supply the sort keys. In this case, it will fallback to
* simple comparison cost estimate.
- *
- * Estimation algorithm is based on ideas from course Algorithms,
- * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper
- * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild,
- * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017.
- *
- * In term of that papers, let N - number of tuples, Xi - number of tuples with
- * key Ki, then estimation is:
- * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi))
- * In our case all Xi are the same because now we don't have an estimation of
- * group sizes, we have only estimation of number of groups. In this case,
- * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation
- * of multi-column sort we need separately compute each column, so, let k is a
- * column number, Gk - number of groups defined by k columns:
- * N * sum( Fk * log(Gk) )
- * Fk is a function costs (including width) for k columns.
*/
-
static Cost
compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
Cost comparison_cost, double tuples, double output_tuples,
@@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
bool has_fake_var = false;
int i = 0;
Oid prev_datatype = InvalidOid;
- Cost funcCost = 0.;
+ Cost funcCost = 0.0;
List *cache_varinfos = NIL;
/* fallback if pathkeys is unknown */
@@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* a total number of tuple comparisons of N log2 K; but the constant
* factor is a bit higher than for quicksort. Tweak it so that the
* cost curve is continuous at the crossover point.
+ *
+ * XXX I suppose the "quicksort factor" references to 1.5 at the end
+ * of this function, but I'm not sure. I suggest we introduce some simple
+ * constants for that, instead of magic values.
*/
output_tuples = (heapSort) ? 2.0 * output_tuples : tuples;
per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples);
@@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* - per column comparison function cost
* - we try to compute needed number of comparison per column
*/
-
foreach(lc, pathkeys)
{
PathKey *pathkey = (PathKey*)lfirst(lc);
@@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* Don't use DEFAULT_NUM_DISTINCT because it used for only one
* column while here we try to estimate number of groups over
* set of columns.
+ *
+ * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to
+ * limit the calculated values, somehow?
+ *
+ * XXX What's the logic of the following formula?
*/
nGroups = ceil(2.0 + sqrt(tuples) *
list_length(pathkeyExprs) / list_length(pathkeys));
@@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
{
if (tuplesPerPrevGroup < output_tuples)
/* comparing only inside output_tuples */
+ /* XXX why not to use the same multiplier (1.5)? */
correctedNGroups =
ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups));
else
@@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups);
/*
- * We could skip all followed columns for cost estimation, because we
+ * We could skip all following columns for cost estimation, because we
* believe that tuples are unique by set ot previous columns
*/
if (tuplesPerPrevGroup <= 1.0)
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 7beb32488a..b092c3e055 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b)
return 0;
return 1;
}
+
/*
* Order tail of list of group pathkeys by uniqueness descendetly. It allows to
* speedup sorting. Returns newly allocated lists, old ones stay untouched.
* n_preordered defines a head of list which order should be prevented.
+ *
+ * XXX But we're not generating this only based on uniqueness (that's a bad
+ * term anyway, because we're using ndistinct estimates, not uniqueness).
+ * We're also using the comparator cost to calculate the expected sort cost,
+ * and optimize that.
+ *
+ * XXX This should explain how we generate the values - all permutations for
+ * up to 4 values, etc.
*/
void
get_cheapest_group_keys_order(PlannerInfo *root, double nrows,
@@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows,
else
{
/*
- * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to
- * a new one which can then be cleaned safely if needed.
+ * Since v13 list_free() can clean list elements so for original list
+ * not to be modified it should be copied to a new one which can then
+ * be cleaned safely if needed.
*/
new_group_pathkeys = list_copy(*group_pathkeys);
nToPermute = nFreeKeys;
--
2.29.2
--------------A955BFDEDE369DCD57A45C5A--
^ permalink raw reply [nested|flat] 97+ messages in thread
* [PATCH 2/2] review
@ 2021-03-09 20:09 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw)
---
src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++-------
src/backend/optimizer/path/pathkeys.c | 14 ++++-
2 files changed, 69 insertions(+), 22 deletions(-)
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index 1639258aaf..f55a4f20e5 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm)
* is_fake_var
* Workaround for generate_append_tlist() which generates fake Vars with
* varno == 0, that will cause a fail of estimate_num_group() call
+ *
+ * XXX Ummm, why would estimate_num_group fail with this?
*/
static bool
is_fake_var(Expr *expr)
@@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr)
* compute_cpu_sort_cost
* compute CPU cost of sort (i.e. in-memory)
*
+ * The main thing we need to calculate to estimate sort CPU costs is the number
+ * of calls to the comparator functions. The difficulty is that for multi-column
+ * sorts there may be different data types involved (for some of which the calls
+ * may be much more expensive). Furthermore, the columns may have very different
+ * number of distinct values - the higher the number, the fewer comparisons will
+ * be needed for the following columns.
+ *
+ * The algoritm is incremental - we add pathkeys one by one, and at each step we
+ * estimate the number of necessary comparisons (based on the number of distinct
+ * groups in the current pathkey prefix and the new pathkey), and the comparison
+ * costs (which is data type specific).
+ *
+ * Estimation of the number of comparisons is based on ideas from two sources:
+ *
+ * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/]
+ *
+ * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild,
+ * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906]
+ *
+ * In term of that paper, let N - number of tuples, Xi - number of tuples with
+ * key Ki, then the estimate of number of comparisons is:
+ *
+ * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi))
+ *
+ * In our case all Xi are the same because now we don't have any estimation of
+ * group sizes, we have only know the estimate of number of groups (distinct
+ * values). In that case, formula becomes:
+ *
+ * N * log(NumberOfGroups)
+ *
+ * For multi-column sorts we need to estimate the number of comparisons for
+ * each individual column - for example with columns (c1, c2, ..., ck) we
+ * can estimate that number of comparions on ck is roughly
+ *
+ * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1))
+ *
+ * Let k be a column number, Gk - number of groups defined by k columns, and Fk
+ * the cost of the comparison is
+ *
+ * N * sum( Fk * log(Gk) )
+ *
+ * Note: We also consider column witdth, not just the comparator cost.
+ *
* NOTE: some callers currently pass NIL for pathkeys because they
* can't conveniently supply the sort keys. In this case, it will fallback to
* simple comparison cost estimate.
- *
- * Estimation algorithm is based on ideas from course Algorithms,
- * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper
- * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild,
- * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017.
- *
- * In term of that papers, let N - number of tuples, Xi - number of tuples with
- * key Ki, then estimation is:
- * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi))
- * In our case all Xi are the same because now we don't have an estimation of
- * group sizes, we have only estimation of number of groups. In this case,
- * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation
- * of multi-column sort we need separately compute each column, so, let k is a
- * column number, Gk - number of groups defined by k columns:
- * N * sum( Fk * log(Gk) )
- * Fk is a function costs (including width) for k columns.
*/
-
static Cost
compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
Cost comparison_cost, double tuples, double output_tuples,
@@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
bool has_fake_var = false;
int i = 0;
Oid prev_datatype = InvalidOid;
- Cost funcCost = 0.;
+ Cost funcCost = 0.0;
List *cache_varinfos = NIL;
/* fallback if pathkeys is unknown */
@@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* a total number of tuple comparisons of N log2 K; but the constant
* factor is a bit higher than for quicksort. Tweak it so that the
* cost curve is continuous at the crossover point.
+ *
+ * XXX I suppose the "quicksort factor" references to 1.5 at the end
+ * of this function, but I'm not sure. I suggest we introduce some simple
+ * constants for that, instead of magic values.
*/
output_tuples = (heapSort) ? 2.0 * output_tuples : tuples;
per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples);
@@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* - per column comparison function cost
* - we try to compute needed number of comparison per column
*/
-
foreach(lc, pathkeys)
{
PathKey *pathkey = (PathKey*)lfirst(lc);
@@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* Don't use DEFAULT_NUM_DISTINCT because it used for only one
* column while here we try to estimate number of groups over
* set of columns.
+ *
+ * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to
+ * limit the calculated values, somehow?
+ *
+ * XXX What's the logic of the following formula?
*/
nGroups = ceil(2.0 + sqrt(tuples) *
list_length(pathkeyExprs) / list_length(pathkeys));
@@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
{
if (tuplesPerPrevGroup < output_tuples)
/* comparing only inside output_tuples */
+ /* XXX why not to use the same multiplier (1.5)? */
correctedNGroups =
ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups));
else
@@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups);
/*
- * We could skip all followed columns for cost estimation, because we
+ * We could skip all following columns for cost estimation, because we
* believe that tuples are unique by set ot previous columns
*/
if (tuplesPerPrevGroup <= 1.0)
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 7beb32488a..b092c3e055 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b)
return 0;
return 1;
}
+
/*
* Order tail of list of group pathkeys by uniqueness descendetly. It allows to
* speedup sorting. Returns newly allocated lists, old ones stay untouched.
* n_preordered defines a head of list which order should be prevented.
+ *
+ * XXX But we're not generating this only based on uniqueness (that's a bad
+ * term anyway, because we're using ndistinct estimates, not uniqueness).
+ * We're also using the comparator cost to calculate the expected sort cost,
+ * and optimize that.
+ *
+ * XXX This should explain how we generate the values - all permutations for
+ * up to 4 values, etc.
*/
void
get_cheapest_group_keys_order(PlannerInfo *root, double nrows,
@@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows,
else
{
/*
- * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to
- * a new one which can then be cleaned safely if needed.
+ * Since v13 list_free() can clean list elements so for original list
+ * not to be modified it should be copied to a new one which can then
+ * be cleaned safely if needed.
*/
new_group_pathkeys = list_copy(*group_pathkeys);
nToPermute = nFreeKeys;
--
2.29.2
--------------A955BFDEDE369DCD57A45C5A--
^ permalink raw reply [nested|flat] 97+ messages in thread
* [PATCH 2/2] review
@ 2021-03-09 20:09 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw)
---
src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++-------
src/backend/optimizer/path/pathkeys.c | 14 ++++-
2 files changed, 69 insertions(+), 22 deletions(-)
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index 1639258aaf..f55a4f20e5 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm)
* is_fake_var
* Workaround for generate_append_tlist() which generates fake Vars with
* varno == 0, that will cause a fail of estimate_num_group() call
+ *
+ * XXX Ummm, why would estimate_num_group fail with this?
*/
static bool
is_fake_var(Expr *expr)
@@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr)
* compute_cpu_sort_cost
* compute CPU cost of sort (i.e. in-memory)
*
+ * The main thing we need to calculate to estimate sort CPU costs is the number
+ * of calls to the comparator functions. The difficulty is that for multi-column
+ * sorts there may be different data types involved (for some of which the calls
+ * may be much more expensive). Furthermore, the columns may have very different
+ * number of distinct values - the higher the number, the fewer comparisons will
+ * be needed for the following columns.
+ *
+ * The algoritm is incremental - we add pathkeys one by one, and at each step we
+ * estimate the number of necessary comparisons (based on the number of distinct
+ * groups in the current pathkey prefix and the new pathkey), and the comparison
+ * costs (which is data type specific).
+ *
+ * Estimation of the number of comparisons is based on ideas from two sources:
+ *
+ * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/]
+ *
+ * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild,
+ * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906]
+ *
+ * In term of that paper, let N - number of tuples, Xi - number of tuples with
+ * key Ki, then the estimate of number of comparisons is:
+ *
+ * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi))
+ *
+ * In our case all Xi are the same because now we don't have any estimation of
+ * group sizes, we have only know the estimate of number of groups (distinct
+ * values). In that case, formula becomes:
+ *
+ * N * log(NumberOfGroups)
+ *
+ * For multi-column sorts we need to estimate the number of comparisons for
+ * each individual column - for example with columns (c1, c2, ..., ck) we
+ * can estimate that number of comparions on ck is roughly
+ *
+ * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1))
+ *
+ * Let k be a column number, Gk - number of groups defined by k columns, and Fk
+ * the cost of the comparison is
+ *
+ * N * sum( Fk * log(Gk) )
+ *
+ * Note: We also consider column witdth, not just the comparator cost.
+ *
* NOTE: some callers currently pass NIL for pathkeys because they
* can't conveniently supply the sort keys. In this case, it will fallback to
* simple comparison cost estimate.
- *
- * Estimation algorithm is based on ideas from course Algorithms,
- * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper
- * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild,
- * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017.
- *
- * In term of that papers, let N - number of tuples, Xi - number of tuples with
- * key Ki, then estimation is:
- * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi))
- * In our case all Xi are the same because now we don't have an estimation of
- * group sizes, we have only estimation of number of groups. In this case,
- * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation
- * of multi-column sort we need separately compute each column, so, let k is a
- * column number, Gk - number of groups defined by k columns:
- * N * sum( Fk * log(Gk) )
- * Fk is a function costs (including width) for k columns.
*/
-
static Cost
compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
Cost comparison_cost, double tuples, double output_tuples,
@@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
bool has_fake_var = false;
int i = 0;
Oid prev_datatype = InvalidOid;
- Cost funcCost = 0.;
+ Cost funcCost = 0.0;
List *cache_varinfos = NIL;
/* fallback if pathkeys is unknown */
@@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* a total number of tuple comparisons of N log2 K; but the constant
* factor is a bit higher than for quicksort. Tweak it so that the
* cost curve is continuous at the crossover point.
+ *
+ * XXX I suppose the "quicksort factor" references to 1.5 at the end
+ * of this function, but I'm not sure. I suggest we introduce some simple
+ * constants for that, instead of magic values.
*/
output_tuples = (heapSort) ? 2.0 * output_tuples : tuples;
per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples);
@@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* - per column comparison function cost
* - we try to compute needed number of comparison per column
*/
-
foreach(lc, pathkeys)
{
PathKey *pathkey = (PathKey*)lfirst(lc);
@@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* Don't use DEFAULT_NUM_DISTINCT because it used for only one
* column while here we try to estimate number of groups over
* set of columns.
+ *
+ * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to
+ * limit the calculated values, somehow?
+ *
+ * XXX What's the logic of the following formula?
*/
nGroups = ceil(2.0 + sqrt(tuples) *
list_length(pathkeyExprs) / list_length(pathkeys));
@@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
{
if (tuplesPerPrevGroup < output_tuples)
/* comparing only inside output_tuples */
+ /* XXX why not to use the same multiplier (1.5)? */
correctedNGroups =
ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups));
else
@@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups);
/*
- * We could skip all followed columns for cost estimation, because we
+ * We could skip all following columns for cost estimation, because we
* believe that tuples are unique by set ot previous columns
*/
if (tuplesPerPrevGroup <= 1.0)
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 7beb32488a..b092c3e055 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b)
return 0;
return 1;
}
+
/*
* Order tail of list of group pathkeys by uniqueness descendetly. It allows to
* speedup sorting. Returns newly allocated lists, old ones stay untouched.
* n_preordered defines a head of list which order should be prevented.
+ *
+ * XXX But we're not generating this only based on uniqueness (that's a bad
+ * term anyway, because we're using ndistinct estimates, not uniqueness).
+ * We're also using the comparator cost to calculate the expected sort cost,
+ * and optimize that.
+ *
+ * XXX This should explain how we generate the values - all permutations for
+ * up to 4 values, etc.
*/
void
get_cheapest_group_keys_order(PlannerInfo *root, double nrows,
@@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows,
else
{
/*
- * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to
- * a new one which can then be cleaned safely if needed.
+ * Since v13 list_free() can clean list elements so for original list
+ * not to be modified it should be copied to a new one which can then
+ * be cleaned safely if needed.
*/
new_group_pathkeys = list_copy(*group_pathkeys);
nToPermute = nFreeKeys;
--
2.29.2
--------------A955BFDEDE369DCD57A45C5A--
^ permalink raw reply [nested|flat] 97+ messages in thread
* [PATCH 2/2] review
@ 2021-03-09 20:09 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw)
---
src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++-------
src/backend/optimizer/path/pathkeys.c | 14 ++++-
2 files changed, 69 insertions(+), 22 deletions(-)
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index 1639258aaf..f55a4f20e5 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm)
* is_fake_var
* Workaround for generate_append_tlist() which generates fake Vars with
* varno == 0, that will cause a fail of estimate_num_group() call
+ *
+ * XXX Ummm, why would estimate_num_group fail with this?
*/
static bool
is_fake_var(Expr *expr)
@@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr)
* compute_cpu_sort_cost
* compute CPU cost of sort (i.e. in-memory)
*
+ * The main thing we need to calculate to estimate sort CPU costs is the number
+ * of calls to the comparator functions. The difficulty is that for multi-column
+ * sorts there may be different data types involved (for some of which the calls
+ * may be much more expensive). Furthermore, the columns may have very different
+ * number of distinct values - the higher the number, the fewer comparisons will
+ * be needed for the following columns.
+ *
+ * The algoritm is incremental - we add pathkeys one by one, and at each step we
+ * estimate the number of necessary comparisons (based on the number of distinct
+ * groups in the current pathkey prefix and the new pathkey), and the comparison
+ * costs (which is data type specific).
+ *
+ * Estimation of the number of comparisons is based on ideas from two sources:
+ *
+ * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/]
+ *
+ * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild,
+ * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906]
+ *
+ * In term of that paper, let N - number of tuples, Xi - number of tuples with
+ * key Ki, then the estimate of number of comparisons is:
+ *
+ * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi))
+ *
+ * In our case all Xi are the same because now we don't have any estimation of
+ * group sizes, we have only know the estimate of number of groups (distinct
+ * values). In that case, formula becomes:
+ *
+ * N * log(NumberOfGroups)
+ *
+ * For multi-column sorts we need to estimate the number of comparisons for
+ * each individual column - for example with columns (c1, c2, ..., ck) we
+ * can estimate that number of comparions on ck is roughly
+ *
+ * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1))
+ *
+ * Let k be a column number, Gk - number of groups defined by k columns, and Fk
+ * the cost of the comparison is
+ *
+ * N * sum( Fk * log(Gk) )
+ *
+ * Note: We also consider column witdth, not just the comparator cost.
+ *
* NOTE: some callers currently pass NIL for pathkeys because they
* can't conveniently supply the sort keys. In this case, it will fallback to
* simple comparison cost estimate.
- *
- * Estimation algorithm is based on ideas from course Algorithms,
- * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper
- * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild,
- * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017.
- *
- * In term of that papers, let N - number of tuples, Xi - number of tuples with
- * key Ki, then estimation is:
- * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi))
- * In our case all Xi are the same because now we don't have an estimation of
- * group sizes, we have only estimation of number of groups. In this case,
- * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation
- * of multi-column sort we need separately compute each column, so, let k is a
- * column number, Gk - number of groups defined by k columns:
- * N * sum( Fk * log(Gk) )
- * Fk is a function costs (including width) for k columns.
*/
-
static Cost
compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
Cost comparison_cost, double tuples, double output_tuples,
@@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
bool has_fake_var = false;
int i = 0;
Oid prev_datatype = InvalidOid;
- Cost funcCost = 0.;
+ Cost funcCost = 0.0;
List *cache_varinfos = NIL;
/* fallback if pathkeys is unknown */
@@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* a total number of tuple comparisons of N log2 K; but the constant
* factor is a bit higher than for quicksort. Tweak it so that the
* cost curve is continuous at the crossover point.
+ *
+ * XXX I suppose the "quicksort factor" references to 1.5 at the end
+ * of this function, but I'm not sure. I suggest we introduce some simple
+ * constants for that, instead of magic values.
*/
output_tuples = (heapSort) ? 2.0 * output_tuples : tuples;
per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples);
@@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* - per column comparison function cost
* - we try to compute needed number of comparison per column
*/
-
foreach(lc, pathkeys)
{
PathKey *pathkey = (PathKey*)lfirst(lc);
@@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* Don't use DEFAULT_NUM_DISTINCT because it used for only one
* column while here we try to estimate number of groups over
* set of columns.
+ *
+ * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to
+ * limit the calculated values, somehow?
+ *
+ * XXX What's the logic of the following formula?
*/
nGroups = ceil(2.0 + sqrt(tuples) *
list_length(pathkeyExprs) / list_length(pathkeys));
@@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
{
if (tuplesPerPrevGroup < output_tuples)
/* comparing only inside output_tuples */
+ /* XXX why not to use the same multiplier (1.5)? */
correctedNGroups =
ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups));
else
@@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups);
/*
- * We could skip all followed columns for cost estimation, because we
+ * We could skip all following columns for cost estimation, because we
* believe that tuples are unique by set ot previous columns
*/
if (tuplesPerPrevGroup <= 1.0)
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 7beb32488a..b092c3e055 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b)
return 0;
return 1;
}
+
/*
* Order tail of list of group pathkeys by uniqueness descendetly. It allows to
* speedup sorting. Returns newly allocated lists, old ones stay untouched.
* n_preordered defines a head of list which order should be prevented.
+ *
+ * XXX But we're not generating this only based on uniqueness (that's a bad
+ * term anyway, because we're using ndistinct estimates, not uniqueness).
+ * We're also using the comparator cost to calculate the expected sort cost,
+ * and optimize that.
+ *
+ * XXX This should explain how we generate the values - all permutations for
+ * up to 4 values, etc.
*/
void
get_cheapest_group_keys_order(PlannerInfo *root, double nrows,
@@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows,
else
{
/*
- * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to
- * a new one which can then be cleaned safely if needed.
+ * Since v13 list_free() can clean list elements so for original list
+ * not to be modified it should be copied to a new one which can then
+ * be cleaned safely if needed.
*/
new_group_pathkeys = list_copy(*group_pathkeys);
nToPermute = nFreeKeys;
--
2.29.2
--------------A955BFDEDE369DCD57A45C5A--
^ permalink raw reply [nested|flat] 97+ messages in thread
* [PATCH 2/2] review
@ 2021-03-09 20:09 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw)
---
src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++-------
src/backend/optimizer/path/pathkeys.c | 14 ++++-
2 files changed, 69 insertions(+), 22 deletions(-)
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index 1639258aaf..f55a4f20e5 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm)
* is_fake_var
* Workaround for generate_append_tlist() which generates fake Vars with
* varno == 0, that will cause a fail of estimate_num_group() call
+ *
+ * XXX Ummm, why would estimate_num_group fail with this?
*/
static bool
is_fake_var(Expr *expr)
@@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr)
* compute_cpu_sort_cost
* compute CPU cost of sort (i.e. in-memory)
*
+ * The main thing we need to calculate to estimate sort CPU costs is the number
+ * of calls to the comparator functions. The difficulty is that for multi-column
+ * sorts there may be different data types involved (for some of which the calls
+ * may be much more expensive). Furthermore, the columns may have very different
+ * number of distinct values - the higher the number, the fewer comparisons will
+ * be needed for the following columns.
+ *
+ * The algoritm is incremental - we add pathkeys one by one, and at each step we
+ * estimate the number of necessary comparisons (based on the number of distinct
+ * groups in the current pathkey prefix and the new pathkey), and the comparison
+ * costs (which is data type specific).
+ *
+ * Estimation of the number of comparisons is based on ideas from two sources:
+ *
+ * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/]
+ *
+ * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild,
+ * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906]
+ *
+ * In term of that paper, let N - number of tuples, Xi - number of tuples with
+ * key Ki, then the estimate of number of comparisons is:
+ *
+ * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi))
+ *
+ * In our case all Xi are the same because now we don't have any estimation of
+ * group sizes, we have only know the estimate of number of groups (distinct
+ * values). In that case, formula becomes:
+ *
+ * N * log(NumberOfGroups)
+ *
+ * For multi-column sorts we need to estimate the number of comparisons for
+ * each individual column - for example with columns (c1, c2, ..., ck) we
+ * can estimate that number of comparions on ck is roughly
+ *
+ * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1))
+ *
+ * Let k be a column number, Gk - number of groups defined by k columns, and Fk
+ * the cost of the comparison is
+ *
+ * N * sum( Fk * log(Gk) )
+ *
+ * Note: We also consider column witdth, not just the comparator cost.
+ *
* NOTE: some callers currently pass NIL for pathkeys because they
* can't conveniently supply the sort keys. In this case, it will fallback to
* simple comparison cost estimate.
- *
- * Estimation algorithm is based on ideas from course Algorithms,
- * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper
- * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild,
- * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017.
- *
- * In term of that papers, let N - number of tuples, Xi - number of tuples with
- * key Ki, then estimation is:
- * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi))
- * In our case all Xi are the same because now we don't have an estimation of
- * group sizes, we have only estimation of number of groups. In this case,
- * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation
- * of multi-column sort we need separately compute each column, so, let k is a
- * column number, Gk - number of groups defined by k columns:
- * N * sum( Fk * log(Gk) )
- * Fk is a function costs (including width) for k columns.
*/
-
static Cost
compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
Cost comparison_cost, double tuples, double output_tuples,
@@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
bool has_fake_var = false;
int i = 0;
Oid prev_datatype = InvalidOid;
- Cost funcCost = 0.;
+ Cost funcCost = 0.0;
List *cache_varinfos = NIL;
/* fallback if pathkeys is unknown */
@@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* a total number of tuple comparisons of N log2 K; but the constant
* factor is a bit higher than for quicksort. Tweak it so that the
* cost curve is continuous at the crossover point.
+ *
+ * XXX I suppose the "quicksort factor" references to 1.5 at the end
+ * of this function, but I'm not sure. I suggest we introduce some simple
+ * constants for that, instead of magic values.
*/
output_tuples = (heapSort) ? 2.0 * output_tuples : tuples;
per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples);
@@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* - per column comparison function cost
* - we try to compute needed number of comparison per column
*/
-
foreach(lc, pathkeys)
{
PathKey *pathkey = (PathKey*)lfirst(lc);
@@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* Don't use DEFAULT_NUM_DISTINCT because it used for only one
* column while here we try to estimate number of groups over
* set of columns.
+ *
+ * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to
+ * limit the calculated values, somehow?
+ *
+ * XXX What's the logic of the following formula?
*/
nGroups = ceil(2.0 + sqrt(tuples) *
list_length(pathkeyExprs) / list_length(pathkeys));
@@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
{
if (tuplesPerPrevGroup < output_tuples)
/* comparing only inside output_tuples */
+ /* XXX why not to use the same multiplier (1.5)? */
correctedNGroups =
ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups));
else
@@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups);
/*
- * We could skip all followed columns for cost estimation, because we
+ * We could skip all following columns for cost estimation, because we
* believe that tuples are unique by set ot previous columns
*/
if (tuplesPerPrevGroup <= 1.0)
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 7beb32488a..b092c3e055 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b)
return 0;
return 1;
}
+
/*
* Order tail of list of group pathkeys by uniqueness descendetly. It allows to
* speedup sorting. Returns newly allocated lists, old ones stay untouched.
* n_preordered defines a head of list which order should be prevented.
+ *
+ * XXX But we're not generating this only based on uniqueness (that's a bad
+ * term anyway, because we're using ndistinct estimates, not uniqueness).
+ * We're also using the comparator cost to calculate the expected sort cost,
+ * and optimize that.
+ *
+ * XXX This should explain how we generate the values - all permutations for
+ * up to 4 values, etc.
*/
void
get_cheapest_group_keys_order(PlannerInfo *root, double nrows,
@@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows,
else
{
/*
- * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to
- * a new one which can then be cleaned safely if needed.
+ * Since v13 list_free() can clean list elements so for original list
+ * not to be modified it should be copied to a new one which can then
+ * be cleaned safely if needed.
*/
new_group_pathkeys = list_copy(*group_pathkeys);
nToPermute = nFreeKeys;
--
2.29.2
--------------A955BFDEDE369DCD57A45C5A--
^ permalink raw reply [nested|flat] 97+ messages in thread
* [PATCH 2/2] review
@ 2021-03-09 20:09 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw)
---
src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++-------
src/backend/optimizer/path/pathkeys.c | 14 ++++-
2 files changed, 69 insertions(+), 22 deletions(-)
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index 1639258aaf..f55a4f20e5 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm)
* is_fake_var
* Workaround for generate_append_tlist() which generates fake Vars with
* varno == 0, that will cause a fail of estimate_num_group() call
+ *
+ * XXX Ummm, why would estimate_num_group fail with this?
*/
static bool
is_fake_var(Expr *expr)
@@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr)
* compute_cpu_sort_cost
* compute CPU cost of sort (i.e. in-memory)
*
+ * The main thing we need to calculate to estimate sort CPU costs is the number
+ * of calls to the comparator functions. The difficulty is that for multi-column
+ * sorts there may be different data types involved (for some of which the calls
+ * may be much more expensive). Furthermore, the columns may have very different
+ * number of distinct values - the higher the number, the fewer comparisons will
+ * be needed for the following columns.
+ *
+ * The algoritm is incremental - we add pathkeys one by one, and at each step we
+ * estimate the number of necessary comparisons (based on the number of distinct
+ * groups in the current pathkey prefix and the new pathkey), and the comparison
+ * costs (which is data type specific).
+ *
+ * Estimation of the number of comparisons is based on ideas from two sources:
+ *
+ * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/]
+ *
+ * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild,
+ * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906]
+ *
+ * In term of that paper, let N - number of tuples, Xi - number of tuples with
+ * key Ki, then the estimate of number of comparisons is:
+ *
+ * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi))
+ *
+ * In our case all Xi are the same because now we don't have any estimation of
+ * group sizes, we have only know the estimate of number of groups (distinct
+ * values). In that case, formula becomes:
+ *
+ * N * log(NumberOfGroups)
+ *
+ * For multi-column sorts we need to estimate the number of comparisons for
+ * each individual column - for example with columns (c1, c2, ..., ck) we
+ * can estimate that number of comparions on ck is roughly
+ *
+ * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1))
+ *
+ * Let k be a column number, Gk - number of groups defined by k columns, and Fk
+ * the cost of the comparison is
+ *
+ * N * sum( Fk * log(Gk) )
+ *
+ * Note: We also consider column witdth, not just the comparator cost.
+ *
* NOTE: some callers currently pass NIL for pathkeys because they
* can't conveniently supply the sort keys. In this case, it will fallback to
* simple comparison cost estimate.
- *
- * Estimation algorithm is based on ideas from course Algorithms,
- * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper
- * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild,
- * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017.
- *
- * In term of that papers, let N - number of tuples, Xi - number of tuples with
- * key Ki, then estimation is:
- * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi))
- * In our case all Xi are the same because now we don't have an estimation of
- * group sizes, we have only estimation of number of groups. In this case,
- * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation
- * of multi-column sort we need separately compute each column, so, let k is a
- * column number, Gk - number of groups defined by k columns:
- * N * sum( Fk * log(Gk) )
- * Fk is a function costs (including width) for k columns.
*/
-
static Cost
compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
Cost comparison_cost, double tuples, double output_tuples,
@@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
bool has_fake_var = false;
int i = 0;
Oid prev_datatype = InvalidOid;
- Cost funcCost = 0.;
+ Cost funcCost = 0.0;
List *cache_varinfos = NIL;
/* fallback if pathkeys is unknown */
@@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* a total number of tuple comparisons of N log2 K; but the constant
* factor is a bit higher than for quicksort. Tweak it so that the
* cost curve is continuous at the crossover point.
+ *
+ * XXX I suppose the "quicksort factor" references to 1.5 at the end
+ * of this function, but I'm not sure. I suggest we introduce some simple
+ * constants for that, instead of magic values.
*/
output_tuples = (heapSort) ? 2.0 * output_tuples : tuples;
per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples);
@@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* - per column comparison function cost
* - we try to compute needed number of comparison per column
*/
-
foreach(lc, pathkeys)
{
PathKey *pathkey = (PathKey*)lfirst(lc);
@@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* Don't use DEFAULT_NUM_DISTINCT because it used for only one
* column while here we try to estimate number of groups over
* set of columns.
+ *
+ * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to
+ * limit the calculated values, somehow?
+ *
+ * XXX What's the logic of the following formula?
*/
nGroups = ceil(2.0 + sqrt(tuples) *
list_length(pathkeyExprs) / list_length(pathkeys));
@@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
{
if (tuplesPerPrevGroup < output_tuples)
/* comparing only inside output_tuples */
+ /* XXX why not to use the same multiplier (1.5)? */
correctedNGroups =
ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups));
else
@@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups);
/*
- * We could skip all followed columns for cost estimation, because we
+ * We could skip all following columns for cost estimation, because we
* believe that tuples are unique by set ot previous columns
*/
if (tuplesPerPrevGroup <= 1.0)
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 7beb32488a..b092c3e055 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b)
return 0;
return 1;
}
+
/*
* Order tail of list of group pathkeys by uniqueness descendetly. It allows to
* speedup sorting. Returns newly allocated lists, old ones stay untouched.
* n_preordered defines a head of list which order should be prevented.
+ *
+ * XXX But we're not generating this only based on uniqueness (that's a bad
+ * term anyway, because we're using ndistinct estimates, not uniqueness).
+ * We're also using the comparator cost to calculate the expected sort cost,
+ * and optimize that.
+ *
+ * XXX This should explain how we generate the values - all permutations for
+ * up to 4 values, etc.
*/
void
get_cheapest_group_keys_order(PlannerInfo *root, double nrows,
@@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows,
else
{
/*
- * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to
- * a new one which can then be cleaned safely if needed.
+ * Since v13 list_free() can clean list elements so for original list
+ * not to be modified it should be copied to a new one which can then
+ * be cleaned safely if needed.
*/
new_group_pathkeys = list_copy(*group_pathkeys);
nToPermute = nFreeKeys;
--
2.29.2
--------------A955BFDEDE369DCD57A45C5A--
^ permalink raw reply [nested|flat] 97+ messages in thread
* [PATCH 2/2] review
@ 2021-03-09 20:09 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw)
---
src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++-------
src/backend/optimizer/path/pathkeys.c | 14 ++++-
2 files changed, 69 insertions(+), 22 deletions(-)
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index 1639258aaf..f55a4f20e5 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm)
* is_fake_var
* Workaround for generate_append_tlist() which generates fake Vars with
* varno == 0, that will cause a fail of estimate_num_group() call
+ *
+ * XXX Ummm, why would estimate_num_group fail with this?
*/
static bool
is_fake_var(Expr *expr)
@@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr)
* compute_cpu_sort_cost
* compute CPU cost of sort (i.e. in-memory)
*
+ * The main thing we need to calculate to estimate sort CPU costs is the number
+ * of calls to the comparator functions. The difficulty is that for multi-column
+ * sorts there may be different data types involved (for some of which the calls
+ * may be much more expensive). Furthermore, the columns may have very different
+ * number of distinct values - the higher the number, the fewer comparisons will
+ * be needed for the following columns.
+ *
+ * The algoritm is incremental - we add pathkeys one by one, and at each step we
+ * estimate the number of necessary comparisons (based on the number of distinct
+ * groups in the current pathkey prefix and the new pathkey), and the comparison
+ * costs (which is data type specific).
+ *
+ * Estimation of the number of comparisons is based on ideas from two sources:
+ *
+ * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/]
+ *
+ * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild,
+ * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906]
+ *
+ * In term of that paper, let N - number of tuples, Xi - number of tuples with
+ * key Ki, then the estimate of number of comparisons is:
+ *
+ * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi))
+ *
+ * In our case all Xi are the same because now we don't have any estimation of
+ * group sizes, we have only know the estimate of number of groups (distinct
+ * values). In that case, formula becomes:
+ *
+ * N * log(NumberOfGroups)
+ *
+ * For multi-column sorts we need to estimate the number of comparisons for
+ * each individual column - for example with columns (c1, c2, ..., ck) we
+ * can estimate that number of comparions on ck is roughly
+ *
+ * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1))
+ *
+ * Let k be a column number, Gk - number of groups defined by k columns, and Fk
+ * the cost of the comparison is
+ *
+ * N * sum( Fk * log(Gk) )
+ *
+ * Note: We also consider column witdth, not just the comparator cost.
+ *
* NOTE: some callers currently pass NIL for pathkeys because they
* can't conveniently supply the sort keys. In this case, it will fallback to
* simple comparison cost estimate.
- *
- * Estimation algorithm is based on ideas from course Algorithms,
- * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper
- * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild,
- * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017.
- *
- * In term of that papers, let N - number of tuples, Xi - number of tuples with
- * key Ki, then estimation is:
- * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi))
- * In our case all Xi are the same because now we don't have an estimation of
- * group sizes, we have only estimation of number of groups. In this case,
- * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation
- * of multi-column sort we need separately compute each column, so, let k is a
- * column number, Gk - number of groups defined by k columns:
- * N * sum( Fk * log(Gk) )
- * Fk is a function costs (including width) for k columns.
*/
-
static Cost
compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
Cost comparison_cost, double tuples, double output_tuples,
@@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
bool has_fake_var = false;
int i = 0;
Oid prev_datatype = InvalidOid;
- Cost funcCost = 0.;
+ Cost funcCost = 0.0;
List *cache_varinfos = NIL;
/* fallback if pathkeys is unknown */
@@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* a total number of tuple comparisons of N log2 K; but the constant
* factor is a bit higher than for quicksort. Tweak it so that the
* cost curve is continuous at the crossover point.
+ *
+ * XXX I suppose the "quicksort factor" references to 1.5 at the end
+ * of this function, but I'm not sure. I suggest we introduce some simple
+ * constants for that, instead of magic values.
*/
output_tuples = (heapSort) ? 2.0 * output_tuples : tuples;
per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples);
@@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* - per column comparison function cost
* - we try to compute needed number of comparison per column
*/
-
foreach(lc, pathkeys)
{
PathKey *pathkey = (PathKey*)lfirst(lc);
@@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* Don't use DEFAULT_NUM_DISTINCT because it used for only one
* column while here we try to estimate number of groups over
* set of columns.
+ *
+ * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to
+ * limit the calculated values, somehow?
+ *
+ * XXX What's the logic of the following formula?
*/
nGroups = ceil(2.0 + sqrt(tuples) *
list_length(pathkeyExprs) / list_length(pathkeys));
@@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
{
if (tuplesPerPrevGroup < output_tuples)
/* comparing only inside output_tuples */
+ /* XXX why not to use the same multiplier (1.5)? */
correctedNGroups =
ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups));
else
@@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups);
/*
- * We could skip all followed columns for cost estimation, because we
+ * We could skip all following columns for cost estimation, because we
* believe that tuples are unique by set ot previous columns
*/
if (tuplesPerPrevGroup <= 1.0)
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 7beb32488a..b092c3e055 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b)
return 0;
return 1;
}
+
/*
* Order tail of list of group pathkeys by uniqueness descendetly. It allows to
* speedup sorting. Returns newly allocated lists, old ones stay untouched.
* n_preordered defines a head of list which order should be prevented.
+ *
+ * XXX But we're not generating this only based on uniqueness (that's a bad
+ * term anyway, because we're using ndistinct estimates, not uniqueness).
+ * We're also using the comparator cost to calculate the expected sort cost,
+ * and optimize that.
+ *
+ * XXX This should explain how we generate the values - all permutations for
+ * up to 4 values, etc.
*/
void
get_cheapest_group_keys_order(PlannerInfo *root, double nrows,
@@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows,
else
{
/*
- * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to
- * a new one which can then be cleaned safely if needed.
+ * Since v13 list_free() can clean list elements so for original list
+ * not to be modified it should be copied to a new one which can then
+ * be cleaned safely if needed.
*/
new_group_pathkeys = list_copy(*group_pathkeys);
nToPermute = nFreeKeys;
--
2.29.2
--------------A955BFDEDE369DCD57A45C5A--
^ permalink raw reply [nested|flat] 97+ messages in thread
* [PATCH 2/2] review
@ 2021-03-09 20:09 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw)
---
src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++-------
src/backend/optimizer/path/pathkeys.c | 14 ++++-
2 files changed, 69 insertions(+), 22 deletions(-)
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index 1639258aaf..f55a4f20e5 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm)
* is_fake_var
* Workaround for generate_append_tlist() which generates fake Vars with
* varno == 0, that will cause a fail of estimate_num_group() call
+ *
+ * XXX Ummm, why would estimate_num_group fail with this?
*/
static bool
is_fake_var(Expr *expr)
@@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr)
* compute_cpu_sort_cost
* compute CPU cost of sort (i.e. in-memory)
*
+ * The main thing we need to calculate to estimate sort CPU costs is the number
+ * of calls to the comparator functions. The difficulty is that for multi-column
+ * sorts there may be different data types involved (for some of which the calls
+ * may be much more expensive). Furthermore, the columns may have very different
+ * number of distinct values - the higher the number, the fewer comparisons will
+ * be needed for the following columns.
+ *
+ * The algoritm is incremental - we add pathkeys one by one, and at each step we
+ * estimate the number of necessary comparisons (based on the number of distinct
+ * groups in the current pathkey prefix and the new pathkey), and the comparison
+ * costs (which is data type specific).
+ *
+ * Estimation of the number of comparisons is based on ideas from two sources:
+ *
+ * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/]
+ *
+ * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild,
+ * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906]
+ *
+ * In term of that paper, let N - number of tuples, Xi - number of tuples with
+ * key Ki, then the estimate of number of comparisons is:
+ *
+ * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi))
+ *
+ * In our case all Xi are the same because now we don't have any estimation of
+ * group sizes, we have only know the estimate of number of groups (distinct
+ * values). In that case, formula becomes:
+ *
+ * N * log(NumberOfGroups)
+ *
+ * For multi-column sorts we need to estimate the number of comparisons for
+ * each individual column - for example with columns (c1, c2, ..., ck) we
+ * can estimate that number of comparions on ck is roughly
+ *
+ * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1))
+ *
+ * Let k be a column number, Gk - number of groups defined by k columns, and Fk
+ * the cost of the comparison is
+ *
+ * N * sum( Fk * log(Gk) )
+ *
+ * Note: We also consider column witdth, not just the comparator cost.
+ *
* NOTE: some callers currently pass NIL for pathkeys because they
* can't conveniently supply the sort keys. In this case, it will fallback to
* simple comparison cost estimate.
- *
- * Estimation algorithm is based on ideas from course Algorithms,
- * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper
- * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild,
- * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017.
- *
- * In term of that papers, let N - number of tuples, Xi - number of tuples with
- * key Ki, then estimation is:
- * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi))
- * In our case all Xi are the same because now we don't have an estimation of
- * group sizes, we have only estimation of number of groups. In this case,
- * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation
- * of multi-column sort we need separately compute each column, so, let k is a
- * column number, Gk - number of groups defined by k columns:
- * N * sum( Fk * log(Gk) )
- * Fk is a function costs (including width) for k columns.
*/
-
static Cost
compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
Cost comparison_cost, double tuples, double output_tuples,
@@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
bool has_fake_var = false;
int i = 0;
Oid prev_datatype = InvalidOid;
- Cost funcCost = 0.;
+ Cost funcCost = 0.0;
List *cache_varinfos = NIL;
/* fallback if pathkeys is unknown */
@@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* a total number of tuple comparisons of N log2 K; but the constant
* factor is a bit higher than for quicksort. Tweak it so that the
* cost curve is continuous at the crossover point.
+ *
+ * XXX I suppose the "quicksort factor" references to 1.5 at the end
+ * of this function, but I'm not sure. I suggest we introduce some simple
+ * constants for that, instead of magic values.
*/
output_tuples = (heapSort) ? 2.0 * output_tuples : tuples;
per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples);
@@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* - per column comparison function cost
* - we try to compute needed number of comparison per column
*/
-
foreach(lc, pathkeys)
{
PathKey *pathkey = (PathKey*)lfirst(lc);
@@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* Don't use DEFAULT_NUM_DISTINCT because it used for only one
* column while here we try to estimate number of groups over
* set of columns.
+ *
+ * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to
+ * limit the calculated values, somehow?
+ *
+ * XXX What's the logic of the following formula?
*/
nGroups = ceil(2.0 + sqrt(tuples) *
list_length(pathkeyExprs) / list_length(pathkeys));
@@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
{
if (tuplesPerPrevGroup < output_tuples)
/* comparing only inside output_tuples */
+ /* XXX why not to use the same multiplier (1.5)? */
correctedNGroups =
ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups));
else
@@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups);
/*
- * We could skip all followed columns for cost estimation, because we
+ * We could skip all following columns for cost estimation, because we
* believe that tuples are unique by set ot previous columns
*/
if (tuplesPerPrevGroup <= 1.0)
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 7beb32488a..b092c3e055 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b)
return 0;
return 1;
}
+
/*
* Order tail of list of group pathkeys by uniqueness descendetly. It allows to
* speedup sorting. Returns newly allocated lists, old ones stay untouched.
* n_preordered defines a head of list which order should be prevented.
+ *
+ * XXX But we're not generating this only based on uniqueness (that's a bad
+ * term anyway, because we're using ndistinct estimates, not uniqueness).
+ * We're also using the comparator cost to calculate the expected sort cost,
+ * and optimize that.
+ *
+ * XXX This should explain how we generate the values - all permutations for
+ * up to 4 values, etc.
*/
void
get_cheapest_group_keys_order(PlannerInfo *root, double nrows,
@@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows,
else
{
/*
- * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to
- * a new one which can then be cleaned safely if needed.
+ * Since v13 list_free() can clean list elements so for original list
+ * not to be modified it should be copied to a new one which can then
+ * be cleaned safely if needed.
*/
new_group_pathkeys = list_copy(*group_pathkeys);
nToPermute = nFreeKeys;
--
2.29.2
--------------A955BFDEDE369DCD57A45C5A--
^ permalink raw reply [nested|flat] 97+ messages in thread
* [PATCH 2/2] review
@ 2021-03-09 20:09 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw)
---
src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++-------
src/backend/optimizer/path/pathkeys.c | 14 ++++-
2 files changed, 69 insertions(+), 22 deletions(-)
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index 1639258aaf..f55a4f20e5 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm)
* is_fake_var
* Workaround for generate_append_tlist() which generates fake Vars with
* varno == 0, that will cause a fail of estimate_num_group() call
+ *
+ * XXX Ummm, why would estimate_num_group fail with this?
*/
static bool
is_fake_var(Expr *expr)
@@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr)
* compute_cpu_sort_cost
* compute CPU cost of sort (i.e. in-memory)
*
+ * The main thing we need to calculate to estimate sort CPU costs is the number
+ * of calls to the comparator functions. The difficulty is that for multi-column
+ * sorts there may be different data types involved (for some of which the calls
+ * may be much more expensive). Furthermore, the columns may have very different
+ * number of distinct values - the higher the number, the fewer comparisons will
+ * be needed for the following columns.
+ *
+ * The algoritm is incremental - we add pathkeys one by one, and at each step we
+ * estimate the number of necessary comparisons (based on the number of distinct
+ * groups in the current pathkey prefix and the new pathkey), and the comparison
+ * costs (which is data type specific).
+ *
+ * Estimation of the number of comparisons is based on ideas from two sources:
+ *
+ * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/]
+ *
+ * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild,
+ * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906]
+ *
+ * In term of that paper, let N - number of tuples, Xi - number of tuples with
+ * key Ki, then the estimate of number of comparisons is:
+ *
+ * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi))
+ *
+ * In our case all Xi are the same because now we don't have any estimation of
+ * group sizes, we have only know the estimate of number of groups (distinct
+ * values). In that case, formula becomes:
+ *
+ * N * log(NumberOfGroups)
+ *
+ * For multi-column sorts we need to estimate the number of comparisons for
+ * each individual column - for example with columns (c1, c2, ..., ck) we
+ * can estimate that number of comparions on ck is roughly
+ *
+ * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1))
+ *
+ * Let k be a column number, Gk - number of groups defined by k columns, and Fk
+ * the cost of the comparison is
+ *
+ * N * sum( Fk * log(Gk) )
+ *
+ * Note: We also consider column witdth, not just the comparator cost.
+ *
* NOTE: some callers currently pass NIL for pathkeys because they
* can't conveniently supply the sort keys. In this case, it will fallback to
* simple comparison cost estimate.
- *
- * Estimation algorithm is based on ideas from course Algorithms,
- * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper
- * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild,
- * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017.
- *
- * In term of that papers, let N - number of tuples, Xi - number of tuples with
- * key Ki, then estimation is:
- * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi))
- * In our case all Xi are the same because now we don't have an estimation of
- * group sizes, we have only estimation of number of groups. In this case,
- * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation
- * of multi-column sort we need separately compute each column, so, let k is a
- * column number, Gk - number of groups defined by k columns:
- * N * sum( Fk * log(Gk) )
- * Fk is a function costs (including width) for k columns.
*/
-
static Cost
compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
Cost comparison_cost, double tuples, double output_tuples,
@@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
bool has_fake_var = false;
int i = 0;
Oid prev_datatype = InvalidOid;
- Cost funcCost = 0.;
+ Cost funcCost = 0.0;
List *cache_varinfos = NIL;
/* fallback if pathkeys is unknown */
@@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* a total number of tuple comparisons of N log2 K; but the constant
* factor is a bit higher than for quicksort. Tweak it so that the
* cost curve is continuous at the crossover point.
+ *
+ * XXX I suppose the "quicksort factor" references to 1.5 at the end
+ * of this function, but I'm not sure. I suggest we introduce some simple
+ * constants for that, instead of magic values.
*/
output_tuples = (heapSort) ? 2.0 * output_tuples : tuples;
per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples);
@@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* - per column comparison function cost
* - we try to compute needed number of comparison per column
*/
-
foreach(lc, pathkeys)
{
PathKey *pathkey = (PathKey*)lfirst(lc);
@@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* Don't use DEFAULT_NUM_DISTINCT because it used for only one
* column while here we try to estimate number of groups over
* set of columns.
+ *
+ * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to
+ * limit the calculated values, somehow?
+ *
+ * XXX What's the logic of the following formula?
*/
nGroups = ceil(2.0 + sqrt(tuples) *
list_length(pathkeyExprs) / list_length(pathkeys));
@@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
{
if (tuplesPerPrevGroup < output_tuples)
/* comparing only inside output_tuples */
+ /* XXX why not to use the same multiplier (1.5)? */
correctedNGroups =
ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups));
else
@@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups);
/*
- * We could skip all followed columns for cost estimation, because we
+ * We could skip all following columns for cost estimation, because we
* believe that tuples are unique by set ot previous columns
*/
if (tuplesPerPrevGroup <= 1.0)
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 7beb32488a..b092c3e055 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b)
return 0;
return 1;
}
+
/*
* Order tail of list of group pathkeys by uniqueness descendetly. It allows to
* speedup sorting. Returns newly allocated lists, old ones stay untouched.
* n_preordered defines a head of list which order should be prevented.
+ *
+ * XXX But we're not generating this only based on uniqueness (that's a bad
+ * term anyway, because we're using ndistinct estimates, not uniqueness).
+ * We're also using the comparator cost to calculate the expected sort cost,
+ * and optimize that.
+ *
+ * XXX This should explain how we generate the values - all permutations for
+ * up to 4 values, etc.
*/
void
get_cheapest_group_keys_order(PlannerInfo *root, double nrows,
@@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows,
else
{
/*
- * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to
- * a new one which can then be cleaned safely if needed.
+ * Since v13 list_free() can clean list elements so for original list
+ * not to be modified it should be copied to a new one which can then
+ * be cleaned safely if needed.
*/
new_group_pathkeys = list_copy(*group_pathkeys);
nToPermute = nFreeKeys;
--
2.29.2
--------------A955BFDEDE369DCD57A45C5A--
^ permalink raw reply [nested|flat] 97+ messages in thread
* [PATCH 2/2] review
@ 2021-03-09 20:09 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw)
---
src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++-------
src/backend/optimizer/path/pathkeys.c | 14 ++++-
2 files changed, 69 insertions(+), 22 deletions(-)
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index 1639258aaf..f55a4f20e5 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm)
* is_fake_var
* Workaround for generate_append_tlist() which generates fake Vars with
* varno == 0, that will cause a fail of estimate_num_group() call
+ *
+ * XXX Ummm, why would estimate_num_group fail with this?
*/
static bool
is_fake_var(Expr *expr)
@@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr)
* compute_cpu_sort_cost
* compute CPU cost of sort (i.e. in-memory)
*
+ * The main thing we need to calculate to estimate sort CPU costs is the number
+ * of calls to the comparator functions. The difficulty is that for multi-column
+ * sorts there may be different data types involved (for some of which the calls
+ * may be much more expensive). Furthermore, the columns may have very different
+ * number of distinct values - the higher the number, the fewer comparisons will
+ * be needed for the following columns.
+ *
+ * The algoritm is incremental - we add pathkeys one by one, and at each step we
+ * estimate the number of necessary comparisons (based on the number of distinct
+ * groups in the current pathkey prefix and the new pathkey), and the comparison
+ * costs (which is data type specific).
+ *
+ * Estimation of the number of comparisons is based on ideas from two sources:
+ *
+ * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/]
+ *
+ * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild,
+ * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906]
+ *
+ * In term of that paper, let N - number of tuples, Xi - number of tuples with
+ * key Ki, then the estimate of number of comparisons is:
+ *
+ * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi))
+ *
+ * In our case all Xi are the same because now we don't have any estimation of
+ * group sizes, we have only know the estimate of number of groups (distinct
+ * values). In that case, formula becomes:
+ *
+ * N * log(NumberOfGroups)
+ *
+ * For multi-column sorts we need to estimate the number of comparisons for
+ * each individual column - for example with columns (c1, c2, ..., ck) we
+ * can estimate that number of comparions on ck is roughly
+ *
+ * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1))
+ *
+ * Let k be a column number, Gk - number of groups defined by k columns, and Fk
+ * the cost of the comparison is
+ *
+ * N * sum( Fk * log(Gk) )
+ *
+ * Note: We also consider column witdth, not just the comparator cost.
+ *
* NOTE: some callers currently pass NIL for pathkeys because they
* can't conveniently supply the sort keys. In this case, it will fallback to
* simple comparison cost estimate.
- *
- * Estimation algorithm is based on ideas from course Algorithms,
- * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper
- * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild,
- * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017.
- *
- * In term of that papers, let N - number of tuples, Xi - number of tuples with
- * key Ki, then estimation is:
- * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi))
- * In our case all Xi are the same because now we don't have an estimation of
- * group sizes, we have only estimation of number of groups. In this case,
- * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation
- * of multi-column sort we need separately compute each column, so, let k is a
- * column number, Gk - number of groups defined by k columns:
- * N * sum( Fk * log(Gk) )
- * Fk is a function costs (including width) for k columns.
*/
-
static Cost
compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
Cost comparison_cost, double tuples, double output_tuples,
@@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
bool has_fake_var = false;
int i = 0;
Oid prev_datatype = InvalidOid;
- Cost funcCost = 0.;
+ Cost funcCost = 0.0;
List *cache_varinfos = NIL;
/* fallback if pathkeys is unknown */
@@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* a total number of tuple comparisons of N log2 K; but the constant
* factor is a bit higher than for quicksort. Tweak it so that the
* cost curve is continuous at the crossover point.
+ *
+ * XXX I suppose the "quicksort factor" references to 1.5 at the end
+ * of this function, but I'm not sure. I suggest we introduce some simple
+ * constants for that, instead of magic values.
*/
output_tuples = (heapSort) ? 2.0 * output_tuples : tuples;
per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples);
@@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* - per column comparison function cost
* - we try to compute needed number of comparison per column
*/
-
foreach(lc, pathkeys)
{
PathKey *pathkey = (PathKey*)lfirst(lc);
@@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* Don't use DEFAULT_NUM_DISTINCT because it used for only one
* column while here we try to estimate number of groups over
* set of columns.
+ *
+ * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to
+ * limit the calculated values, somehow?
+ *
+ * XXX What's the logic of the following formula?
*/
nGroups = ceil(2.0 + sqrt(tuples) *
list_length(pathkeyExprs) / list_length(pathkeys));
@@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
{
if (tuplesPerPrevGroup < output_tuples)
/* comparing only inside output_tuples */
+ /* XXX why not to use the same multiplier (1.5)? */
correctedNGroups =
ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups));
else
@@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups);
/*
- * We could skip all followed columns for cost estimation, because we
+ * We could skip all following columns for cost estimation, because we
* believe that tuples are unique by set ot previous columns
*/
if (tuplesPerPrevGroup <= 1.0)
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 7beb32488a..b092c3e055 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b)
return 0;
return 1;
}
+
/*
* Order tail of list of group pathkeys by uniqueness descendetly. It allows to
* speedup sorting. Returns newly allocated lists, old ones stay untouched.
* n_preordered defines a head of list which order should be prevented.
+ *
+ * XXX But we're not generating this only based on uniqueness (that's a bad
+ * term anyway, because we're using ndistinct estimates, not uniqueness).
+ * We're also using the comparator cost to calculate the expected sort cost,
+ * and optimize that.
+ *
+ * XXX This should explain how we generate the values - all permutations for
+ * up to 4 values, etc.
*/
void
get_cheapest_group_keys_order(PlannerInfo *root, double nrows,
@@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows,
else
{
/*
- * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to
- * a new one which can then be cleaned safely if needed.
+ * Since v13 list_free() can clean list elements so for original list
+ * not to be modified it should be copied to a new one which can then
+ * be cleaned safely if needed.
*/
new_group_pathkeys = list_copy(*group_pathkeys);
nToPermute = nFreeKeys;
--
2.29.2
--------------A955BFDEDE369DCD57A45C5A--
^ permalink raw reply [nested|flat] 97+ messages in thread
* [PATCH 2/2] review
@ 2021-03-09 20:09 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw)
---
src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++-------
src/backend/optimizer/path/pathkeys.c | 14 ++++-
2 files changed, 69 insertions(+), 22 deletions(-)
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index 1639258aaf..f55a4f20e5 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm)
* is_fake_var
* Workaround for generate_append_tlist() which generates fake Vars with
* varno == 0, that will cause a fail of estimate_num_group() call
+ *
+ * XXX Ummm, why would estimate_num_group fail with this?
*/
static bool
is_fake_var(Expr *expr)
@@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr)
* compute_cpu_sort_cost
* compute CPU cost of sort (i.e. in-memory)
*
+ * The main thing we need to calculate to estimate sort CPU costs is the number
+ * of calls to the comparator functions. The difficulty is that for multi-column
+ * sorts there may be different data types involved (for some of which the calls
+ * may be much more expensive). Furthermore, the columns may have very different
+ * number of distinct values - the higher the number, the fewer comparisons will
+ * be needed for the following columns.
+ *
+ * The algoritm is incremental - we add pathkeys one by one, and at each step we
+ * estimate the number of necessary comparisons (based on the number of distinct
+ * groups in the current pathkey prefix and the new pathkey), and the comparison
+ * costs (which is data type specific).
+ *
+ * Estimation of the number of comparisons is based on ideas from two sources:
+ *
+ * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/]
+ *
+ * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild,
+ * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906]
+ *
+ * In term of that paper, let N - number of tuples, Xi - number of tuples with
+ * key Ki, then the estimate of number of comparisons is:
+ *
+ * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi))
+ *
+ * In our case all Xi are the same because now we don't have any estimation of
+ * group sizes, we have only know the estimate of number of groups (distinct
+ * values). In that case, formula becomes:
+ *
+ * N * log(NumberOfGroups)
+ *
+ * For multi-column sorts we need to estimate the number of comparisons for
+ * each individual column - for example with columns (c1, c2, ..., ck) we
+ * can estimate that number of comparions on ck is roughly
+ *
+ * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1))
+ *
+ * Let k be a column number, Gk - number of groups defined by k columns, and Fk
+ * the cost of the comparison is
+ *
+ * N * sum( Fk * log(Gk) )
+ *
+ * Note: We also consider column witdth, not just the comparator cost.
+ *
* NOTE: some callers currently pass NIL for pathkeys because they
* can't conveniently supply the sort keys. In this case, it will fallback to
* simple comparison cost estimate.
- *
- * Estimation algorithm is based on ideas from course Algorithms,
- * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper
- * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild,
- * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017.
- *
- * In term of that papers, let N - number of tuples, Xi - number of tuples with
- * key Ki, then estimation is:
- * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi))
- * In our case all Xi are the same because now we don't have an estimation of
- * group sizes, we have only estimation of number of groups. In this case,
- * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation
- * of multi-column sort we need separately compute each column, so, let k is a
- * column number, Gk - number of groups defined by k columns:
- * N * sum( Fk * log(Gk) )
- * Fk is a function costs (including width) for k columns.
*/
-
static Cost
compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
Cost comparison_cost, double tuples, double output_tuples,
@@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
bool has_fake_var = false;
int i = 0;
Oid prev_datatype = InvalidOid;
- Cost funcCost = 0.;
+ Cost funcCost = 0.0;
List *cache_varinfos = NIL;
/* fallback if pathkeys is unknown */
@@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* a total number of tuple comparisons of N log2 K; but the constant
* factor is a bit higher than for quicksort. Tweak it so that the
* cost curve is continuous at the crossover point.
+ *
+ * XXX I suppose the "quicksort factor" references to 1.5 at the end
+ * of this function, but I'm not sure. I suggest we introduce some simple
+ * constants for that, instead of magic values.
*/
output_tuples = (heapSort) ? 2.0 * output_tuples : tuples;
per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples);
@@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* - per column comparison function cost
* - we try to compute needed number of comparison per column
*/
-
foreach(lc, pathkeys)
{
PathKey *pathkey = (PathKey*)lfirst(lc);
@@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* Don't use DEFAULT_NUM_DISTINCT because it used for only one
* column while here we try to estimate number of groups over
* set of columns.
+ *
+ * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to
+ * limit the calculated values, somehow?
+ *
+ * XXX What's the logic of the following formula?
*/
nGroups = ceil(2.0 + sqrt(tuples) *
list_length(pathkeyExprs) / list_length(pathkeys));
@@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
{
if (tuplesPerPrevGroup < output_tuples)
/* comparing only inside output_tuples */
+ /* XXX why not to use the same multiplier (1.5)? */
correctedNGroups =
ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups));
else
@@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups);
/*
- * We could skip all followed columns for cost estimation, because we
+ * We could skip all following columns for cost estimation, because we
* believe that tuples are unique by set ot previous columns
*/
if (tuplesPerPrevGroup <= 1.0)
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 7beb32488a..b092c3e055 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b)
return 0;
return 1;
}
+
/*
* Order tail of list of group pathkeys by uniqueness descendetly. It allows to
* speedup sorting. Returns newly allocated lists, old ones stay untouched.
* n_preordered defines a head of list which order should be prevented.
+ *
+ * XXX But we're not generating this only based on uniqueness (that's a bad
+ * term anyway, because we're using ndistinct estimates, not uniqueness).
+ * We're also using the comparator cost to calculate the expected sort cost,
+ * and optimize that.
+ *
+ * XXX This should explain how we generate the values - all permutations for
+ * up to 4 values, etc.
*/
void
get_cheapest_group_keys_order(PlannerInfo *root, double nrows,
@@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows,
else
{
/*
- * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to
- * a new one which can then be cleaned safely if needed.
+ * Since v13 list_free() can clean list elements so for original list
+ * not to be modified it should be copied to a new one which can then
+ * be cleaned safely if needed.
*/
new_group_pathkeys = list_copy(*group_pathkeys);
nToPermute = nFreeKeys;
--
2.29.2
--------------A955BFDEDE369DCD57A45C5A--
^ permalink raw reply [nested|flat] 97+ messages in thread
* [PATCH 2/2] review
@ 2021-03-09 20:09 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw)
---
src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++-------
src/backend/optimizer/path/pathkeys.c | 14 ++++-
2 files changed, 69 insertions(+), 22 deletions(-)
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index 1639258aaf..f55a4f20e5 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm)
* is_fake_var
* Workaround for generate_append_tlist() which generates fake Vars with
* varno == 0, that will cause a fail of estimate_num_group() call
+ *
+ * XXX Ummm, why would estimate_num_group fail with this?
*/
static bool
is_fake_var(Expr *expr)
@@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr)
* compute_cpu_sort_cost
* compute CPU cost of sort (i.e. in-memory)
*
+ * The main thing we need to calculate to estimate sort CPU costs is the number
+ * of calls to the comparator functions. The difficulty is that for multi-column
+ * sorts there may be different data types involved (for some of which the calls
+ * may be much more expensive). Furthermore, the columns may have very different
+ * number of distinct values - the higher the number, the fewer comparisons will
+ * be needed for the following columns.
+ *
+ * The algoritm is incremental - we add pathkeys one by one, and at each step we
+ * estimate the number of necessary comparisons (based on the number of distinct
+ * groups in the current pathkey prefix and the new pathkey), and the comparison
+ * costs (which is data type specific).
+ *
+ * Estimation of the number of comparisons is based on ideas from two sources:
+ *
+ * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/]
+ *
+ * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild,
+ * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906]
+ *
+ * In term of that paper, let N - number of tuples, Xi - number of tuples with
+ * key Ki, then the estimate of number of comparisons is:
+ *
+ * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi))
+ *
+ * In our case all Xi are the same because now we don't have any estimation of
+ * group sizes, we have only know the estimate of number of groups (distinct
+ * values). In that case, formula becomes:
+ *
+ * N * log(NumberOfGroups)
+ *
+ * For multi-column sorts we need to estimate the number of comparisons for
+ * each individual column - for example with columns (c1, c2, ..., ck) we
+ * can estimate that number of comparions on ck is roughly
+ *
+ * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1))
+ *
+ * Let k be a column number, Gk - number of groups defined by k columns, and Fk
+ * the cost of the comparison is
+ *
+ * N * sum( Fk * log(Gk) )
+ *
+ * Note: We also consider column witdth, not just the comparator cost.
+ *
* NOTE: some callers currently pass NIL for pathkeys because they
* can't conveniently supply the sort keys. In this case, it will fallback to
* simple comparison cost estimate.
- *
- * Estimation algorithm is based on ideas from course Algorithms,
- * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper
- * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild,
- * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017.
- *
- * In term of that papers, let N - number of tuples, Xi - number of tuples with
- * key Ki, then estimation is:
- * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi))
- * In our case all Xi are the same because now we don't have an estimation of
- * group sizes, we have only estimation of number of groups. In this case,
- * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation
- * of multi-column sort we need separately compute each column, so, let k is a
- * column number, Gk - number of groups defined by k columns:
- * N * sum( Fk * log(Gk) )
- * Fk is a function costs (including width) for k columns.
*/
-
static Cost
compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
Cost comparison_cost, double tuples, double output_tuples,
@@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
bool has_fake_var = false;
int i = 0;
Oid prev_datatype = InvalidOid;
- Cost funcCost = 0.;
+ Cost funcCost = 0.0;
List *cache_varinfos = NIL;
/* fallback if pathkeys is unknown */
@@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* a total number of tuple comparisons of N log2 K; but the constant
* factor is a bit higher than for quicksort. Tweak it so that the
* cost curve is continuous at the crossover point.
+ *
+ * XXX I suppose the "quicksort factor" references to 1.5 at the end
+ * of this function, but I'm not sure. I suggest we introduce some simple
+ * constants for that, instead of magic values.
*/
output_tuples = (heapSort) ? 2.0 * output_tuples : tuples;
per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples);
@@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* - per column comparison function cost
* - we try to compute needed number of comparison per column
*/
-
foreach(lc, pathkeys)
{
PathKey *pathkey = (PathKey*)lfirst(lc);
@@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* Don't use DEFAULT_NUM_DISTINCT because it used for only one
* column while here we try to estimate number of groups over
* set of columns.
+ *
+ * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to
+ * limit the calculated values, somehow?
+ *
+ * XXX What's the logic of the following formula?
*/
nGroups = ceil(2.0 + sqrt(tuples) *
list_length(pathkeyExprs) / list_length(pathkeys));
@@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
{
if (tuplesPerPrevGroup < output_tuples)
/* comparing only inside output_tuples */
+ /* XXX why not to use the same multiplier (1.5)? */
correctedNGroups =
ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups));
else
@@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups);
/*
- * We could skip all followed columns for cost estimation, because we
+ * We could skip all following columns for cost estimation, because we
* believe that tuples are unique by set ot previous columns
*/
if (tuplesPerPrevGroup <= 1.0)
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 7beb32488a..b092c3e055 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b)
return 0;
return 1;
}
+
/*
* Order tail of list of group pathkeys by uniqueness descendetly. It allows to
* speedup sorting. Returns newly allocated lists, old ones stay untouched.
* n_preordered defines a head of list which order should be prevented.
+ *
+ * XXX But we're not generating this only based on uniqueness (that's a bad
+ * term anyway, because we're using ndistinct estimates, not uniqueness).
+ * We're also using the comparator cost to calculate the expected sort cost,
+ * and optimize that.
+ *
+ * XXX This should explain how we generate the values - all permutations for
+ * up to 4 values, etc.
*/
void
get_cheapest_group_keys_order(PlannerInfo *root, double nrows,
@@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows,
else
{
/*
- * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to
- * a new one which can then be cleaned safely if needed.
+ * Since v13 list_free() can clean list elements so for original list
+ * not to be modified it should be copied to a new one which can then
+ * be cleaned safely if needed.
*/
new_group_pathkeys = list_copy(*group_pathkeys);
nToPermute = nFreeKeys;
--
2.29.2
--------------A955BFDEDE369DCD57A45C5A--
^ permalink raw reply [nested|flat] 97+ messages in thread
* [PATCH 2/2] review
@ 2021-03-09 20:09 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw)
---
src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++-------
src/backend/optimizer/path/pathkeys.c | 14 ++++-
2 files changed, 69 insertions(+), 22 deletions(-)
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index 1639258aaf..f55a4f20e5 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm)
* is_fake_var
* Workaround for generate_append_tlist() which generates fake Vars with
* varno == 0, that will cause a fail of estimate_num_group() call
+ *
+ * XXX Ummm, why would estimate_num_group fail with this?
*/
static bool
is_fake_var(Expr *expr)
@@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr)
* compute_cpu_sort_cost
* compute CPU cost of sort (i.e. in-memory)
*
+ * The main thing we need to calculate to estimate sort CPU costs is the number
+ * of calls to the comparator functions. The difficulty is that for multi-column
+ * sorts there may be different data types involved (for some of which the calls
+ * may be much more expensive). Furthermore, the columns may have very different
+ * number of distinct values - the higher the number, the fewer comparisons will
+ * be needed for the following columns.
+ *
+ * The algoritm is incremental - we add pathkeys one by one, and at each step we
+ * estimate the number of necessary comparisons (based on the number of distinct
+ * groups in the current pathkey prefix and the new pathkey), and the comparison
+ * costs (which is data type specific).
+ *
+ * Estimation of the number of comparisons is based on ideas from two sources:
+ *
+ * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/]
+ *
+ * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild,
+ * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906]
+ *
+ * In term of that paper, let N - number of tuples, Xi - number of tuples with
+ * key Ki, then the estimate of number of comparisons is:
+ *
+ * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi))
+ *
+ * In our case all Xi are the same because now we don't have any estimation of
+ * group sizes, we have only know the estimate of number of groups (distinct
+ * values). In that case, formula becomes:
+ *
+ * N * log(NumberOfGroups)
+ *
+ * For multi-column sorts we need to estimate the number of comparisons for
+ * each individual column - for example with columns (c1, c2, ..., ck) we
+ * can estimate that number of comparions on ck is roughly
+ *
+ * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1))
+ *
+ * Let k be a column number, Gk - number of groups defined by k columns, and Fk
+ * the cost of the comparison is
+ *
+ * N * sum( Fk * log(Gk) )
+ *
+ * Note: We also consider column witdth, not just the comparator cost.
+ *
* NOTE: some callers currently pass NIL for pathkeys because they
* can't conveniently supply the sort keys. In this case, it will fallback to
* simple comparison cost estimate.
- *
- * Estimation algorithm is based on ideas from course Algorithms,
- * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper
- * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild,
- * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017.
- *
- * In term of that papers, let N - number of tuples, Xi - number of tuples with
- * key Ki, then estimation is:
- * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi))
- * In our case all Xi are the same because now we don't have an estimation of
- * group sizes, we have only estimation of number of groups. In this case,
- * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation
- * of multi-column sort we need separately compute each column, so, let k is a
- * column number, Gk - number of groups defined by k columns:
- * N * sum( Fk * log(Gk) )
- * Fk is a function costs (including width) for k columns.
*/
-
static Cost
compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
Cost comparison_cost, double tuples, double output_tuples,
@@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
bool has_fake_var = false;
int i = 0;
Oid prev_datatype = InvalidOid;
- Cost funcCost = 0.;
+ Cost funcCost = 0.0;
List *cache_varinfos = NIL;
/* fallback if pathkeys is unknown */
@@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* a total number of tuple comparisons of N log2 K; but the constant
* factor is a bit higher than for quicksort. Tweak it so that the
* cost curve is continuous at the crossover point.
+ *
+ * XXX I suppose the "quicksort factor" references to 1.5 at the end
+ * of this function, but I'm not sure. I suggest we introduce some simple
+ * constants for that, instead of magic values.
*/
output_tuples = (heapSort) ? 2.0 * output_tuples : tuples;
per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples);
@@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* - per column comparison function cost
* - we try to compute needed number of comparison per column
*/
-
foreach(lc, pathkeys)
{
PathKey *pathkey = (PathKey*)lfirst(lc);
@@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* Don't use DEFAULT_NUM_DISTINCT because it used for only one
* column while here we try to estimate number of groups over
* set of columns.
+ *
+ * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to
+ * limit the calculated values, somehow?
+ *
+ * XXX What's the logic of the following formula?
*/
nGroups = ceil(2.0 + sqrt(tuples) *
list_length(pathkeyExprs) / list_length(pathkeys));
@@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
{
if (tuplesPerPrevGroup < output_tuples)
/* comparing only inside output_tuples */
+ /* XXX why not to use the same multiplier (1.5)? */
correctedNGroups =
ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups));
else
@@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups);
/*
- * We could skip all followed columns for cost estimation, because we
+ * We could skip all following columns for cost estimation, because we
* believe that tuples are unique by set ot previous columns
*/
if (tuplesPerPrevGroup <= 1.0)
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 7beb32488a..b092c3e055 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b)
return 0;
return 1;
}
+
/*
* Order tail of list of group pathkeys by uniqueness descendetly. It allows to
* speedup sorting. Returns newly allocated lists, old ones stay untouched.
* n_preordered defines a head of list which order should be prevented.
+ *
+ * XXX But we're not generating this only based on uniqueness (that's a bad
+ * term anyway, because we're using ndistinct estimates, not uniqueness).
+ * We're also using the comparator cost to calculate the expected sort cost,
+ * and optimize that.
+ *
+ * XXX This should explain how we generate the values - all permutations for
+ * up to 4 values, etc.
*/
void
get_cheapest_group_keys_order(PlannerInfo *root, double nrows,
@@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows,
else
{
/*
- * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to
- * a new one which can then be cleaned safely if needed.
+ * Since v13 list_free() can clean list elements so for original list
+ * not to be modified it should be copied to a new one which can then
+ * be cleaned safely if needed.
*/
new_group_pathkeys = list_copy(*group_pathkeys);
nToPermute = nFreeKeys;
--
2.29.2
--------------A955BFDEDE369DCD57A45C5A--
^ permalink raw reply [nested|flat] 97+ messages in thread
* [PATCH 2/2] review
@ 2021-03-09 20:09 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw)
---
src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++-------
src/backend/optimizer/path/pathkeys.c | 14 ++++-
2 files changed, 69 insertions(+), 22 deletions(-)
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index 1639258aaf..f55a4f20e5 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm)
* is_fake_var
* Workaround for generate_append_tlist() which generates fake Vars with
* varno == 0, that will cause a fail of estimate_num_group() call
+ *
+ * XXX Ummm, why would estimate_num_group fail with this?
*/
static bool
is_fake_var(Expr *expr)
@@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr)
* compute_cpu_sort_cost
* compute CPU cost of sort (i.e. in-memory)
*
+ * The main thing we need to calculate to estimate sort CPU costs is the number
+ * of calls to the comparator functions. The difficulty is that for multi-column
+ * sorts there may be different data types involved (for some of which the calls
+ * may be much more expensive). Furthermore, the columns may have very different
+ * number of distinct values - the higher the number, the fewer comparisons will
+ * be needed for the following columns.
+ *
+ * The algoritm is incremental - we add pathkeys one by one, and at each step we
+ * estimate the number of necessary comparisons (based on the number of distinct
+ * groups in the current pathkey prefix and the new pathkey), and the comparison
+ * costs (which is data type specific).
+ *
+ * Estimation of the number of comparisons is based on ideas from two sources:
+ *
+ * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/]
+ *
+ * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild,
+ * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906]
+ *
+ * In term of that paper, let N - number of tuples, Xi - number of tuples with
+ * key Ki, then the estimate of number of comparisons is:
+ *
+ * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi))
+ *
+ * In our case all Xi are the same because now we don't have any estimation of
+ * group sizes, we have only know the estimate of number of groups (distinct
+ * values). In that case, formula becomes:
+ *
+ * N * log(NumberOfGroups)
+ *
+ * For multi-column sorts we need to estimate the number of comparisons for
+ * each individual column - for example with columns (c1, c2, ..., ck) we
+ * can estimate that number of comparions on ck is roughly
+ *
+ * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1))
+ *
+ * Let k be a column number, Gk - number of groups defined by k columns, and Fk
+ * the cost of the comparison is
+ *
+ * N * sum( Fk * log(Gk) )
+ *
+ * Note: We also consider column witdth, not just the comparator cost.
+ *
* NOTE: some callers currently pass NIL for pathkeys because they
* can't conveniently supply the sort keys. In this case, it will fallback to
* simple comparison cost estimate.
- *
- * Estimation algorithm is based on ideas from course Algorithms,
- * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper
- * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild,
- * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017.
- *
- * In term of that papers, let N - number of tuples, Xi - number of tuples with
- * key Ki, then estimation is:
- * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi))
- * In our case all Xi are the same because now we don't have an estimation of
- * group sizes, we have only estimation of number of groups. In this case,
- * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation
- * of multi-column sort we need separately compute each column, so, let k is a
- * column number, Gk - number of groups defined by k columns:
- * N * sum( Fk * log(Gk) )
- * Fk is a function costs (including width) for k columns.
*/
-
static Cost
compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
Cost comparison_cost, double tuples, double output_tuples,
@@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
bool has_fake_var = false;
int i = 0;
Oid prev_datatype = InvalidOid;
- Cost funcCost = 0.;
+ Cost funcCost = 0.0;
List *cache_varinfos = NIL;
/* fallback if pathkeys is unknown */
@@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* a total number of tuple comparisons of N log2 K; but the constant
* factor is a bit higher than for quicksort. Tweak it so that the
* cost curve is continuous at the crossover point.
+ *
+ * XXX I suppose the "quicksort factor" references to 1.5 at the end
+ * of this function, but I'm not sure. I suggest we introduce some simple
+ * constants for that, instead of magic values.
*/
output_tuples = (heapSort) ? 2.0 * output_tuples : tuples;
per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples);
@@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* - per column comparison function cost
* - we try to compute needed number of comparison per column
*/
-
foreach(lc, pathkeys)
{
PathKey *pathkey = (PathKey*)lfirst(lc);
@@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* Don't use DEFAULT_NUM_DISTINCT because it used for only one
* column while here we try to estimate number of groups over
* set of columns.
+ *
+ * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to
+ * limit the calculated values, somehow?
+ *
+ * XXX What's the logic of the following formula?
*/
nGroups = ceil(2.0 + sqrt(tuples) *
list_length(pathkeyExprs) / list_length(pathkeys));
@@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
{
if (tuplesPerPrevGroup < output_tuples)
/* comparing only inside output_tuples */
+ /* XXX why not to use the same multiplier (1.5)? */
correctedNGroups =
ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups));
else
@@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups);
/*
- * We could skip all followed columns for cost estimation, because we
+ * We could skip all following columns for cost estimation, because we
* believe that tuples are unique by set ot previous columns
*/
if (tuplesPerPrevGroup <= 1.0)
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 7beb32488a..b092c3e055 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b)
return 0;
return 1;
}
+
/*
* Order tail of list of group pathkeys by uniqueness descendetly. It allows to
* speedup sorting. Returns newly allocated lists, old ones stay untouched.
* n_preordered defines a head of list which order should be prevented.
+ *
+ * XXX But we're not generating this only based on uniqueness (that's a bad
+ * term anyway, because we're using ndistinct estimates, not uniqueness).
+ * We're also using the comparator cost to calculate the expected sort cost,
+ * and optimize that.
+ *
+ * XXX This should explain how we generate the values - all permutations for
+ * up to 4 values, etc.
*/
void
get_cheapest_group_keys_order(PlannerInfo *root, double nrows,
@@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows,
else
{
/*
- * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to
- * a new one which can then be cleaned safely if needed.
+ * Since v13 list_free() can clean list elements so for original list
+ * not to be modified it should be copied to a new one which can then
+ * be cleaned safely if needed.
*/
new_group_pathkeys = list_copy(*group_pathkeys);
nToPermute = nFreeKeys;
--
2.29.2
--------------A955BFDEDE369DCD57A45C5A--
^ permalink raw reply [nested|flat] 97+ messages in thread
* [PATCH 2/2] review
@ 2021-03-09 20:09 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw)
---
src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++-------
src/backend/optimizer/path/pathkeys.c | 14 ++++-
2 files changed, 69 insertions(+), 22 deletions(-)
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index 1639258aaf..f55a4f20e5 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm)
* is_fake_var
* Workaround for generate_append_tlist() which generates fake Vars with
* varno == 0, that will cause a fail of estimate_num_group() call
+ *
+ * XXX Ummm, why would estimate_num_group fail with this?
*/
static bool
is_fake_var(Expr *expr)
@@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr)
* compute_cpu_sort_cost
* compute CPU cost of sort (i.e. in-memory)
*
+ * The main thing we need to calculate to estimate sort CPU costs is the number
+ * of calls to the comparator functions. The difficulty is that for multi-column
+ * sorts there may be different data types involved (for some of which the calls
+ * may be much more expensive). Furthermore, the columns may have very different
+ * number of distinct values - the higher the number, the fewer comparisons will
+ * be needed for the following columns.
+ *
+ * The algoritm is incremental - we add pathkeys one by one, and at each step we
+ * estimate the number of necessary comparisons (based on the number of distinct
+ * groups in the current pathkey prefix and the new pathkey), and the comparison
+ * costs (which is data type specific).
+ *
+ * Estimation of the number of comparisons is based on ideas from two sources:
+ *
+ * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/]
+ *
+ * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild,
+ * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906]
+ *
+ * In term of that paper, let N - number of tuples, Xi - number of tuples with
+ * key Ki, then the estimate of number of comparisons is:
+ *
+ * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi))
+ *
+ * In our case all Xi are the same because now we don't have any estimation of
+ * group sizes, we have only know the estimate of number of groups (distinct
+ * values). In that case, formula becomes:
+ *
+ * N * log(NumberOfGroups)
+ *
+ * For multi-column sorts we need to estimate the number of comparisons for
+ * each individual column - for example with columns (c1, c2, ..., ck) we
+ * can estimate that number of comparions on ck is roughly
+ *
+ * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1))
+ *
+ * Let k be a column number, Gk - number of groups defined by k columns, and Fk
+ * the cost of the comparison is
+ *
+ * N * sum( Fk * log(Gk) )
+ *
+ * Note: We also consider column witdth, not just the comparator cost.
+ *
* NOTE: some callers currently pass NIL for pathkeys because they
* can't conveniently supply the sort keys. In this case, it will fallback to
* simple comparison cost estimate.
- *
- * Estimation algorithm is based on ideas from course Algorithms,
- * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper
- * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild,
- * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017.
- *
- * In term of that papers, let N - number of tuples, Xi - number of tuples with
- * key Ki, then estimation is:
- * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi))
- * In our case all Xi are the same because now we don't have an estimation of
- * group sizes, we have only estimation of number of groups. In this case,
- * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation
- * of multi-column sort we need separately compute each column, so, let k is a
- * column number, Gk - number of groups defined by k columns:
- * N * sum( Fk * log(Gk) )
- * Fk is a function costs (including width) for k columns.
*/
-
static Cost
compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
Cost comparison_cost, double tuples, double output_tuples,
@@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
bool has_fake_var = false;
int i = 0;
Oid prev_datatype = InvalidOid;
- Cost funcCost = 0.;
+ Cost funcCost = 0.0;
List *cache_varinfos = NIL;
/* fallback if pathkeys is unknown */
@@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* a total number of tuple comparisons of N log2 K; but the constant
* factor is a bit higher than for quicksort. Tweak it so that the
* cost curve is continuous at the crossover point.
+ *
+ * XXX I suppose the "quicksort factor" references to 1.5 at the end
+ * of this function, but I'm not sure. I suggest we introduce some simple
+ * constants for that, instead of magic values.
*/
output_tuples = (heapSort) ? 2.0 * output_tuples : tuples;
per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples);
@@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* - per column comparison function cost
* - we try to compute needed number of comparison per column
*/
-
foreach(lc, pathkeys)
{
PathKey *pathkey = (PathKey*)lfirst(lc);
@@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* Don't use DEFAULT_NUM_DISTINCT because it used for only one
* column while here we try to estimate number of groups over
* set of columns.
+ *
+ * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to
+ * limit the calculated values, somehow?
+ *
+ * XXX What's the logic of the following formula?
*/
nGroups = ceil(2.0 + sqrt(tuples) *
list_length(pathkeyExprs) / list_length(pathkeys));
@@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
{
if (tuplesPerPrevGroup < output_tuples)
/* comparing only inside output_tuples */
+ /* XXX why not to use the same multiplier (1.5)? */
correctedNGroups =
ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups));
else
@@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups);
/*
- * We could skip all followed columns for cost estimation, because we
+ * We could skip all following columns for cost estimation, because we
* believe that tuples are unique by set ot previous columns
*/
if (tuplesPerPrevGroup <= 1.0)
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 7beb32488a..b092c3e055 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b)
return 0;
return 1;
}
+
/*
* Order tail of list of group pathkeys by uniqueness descendetly. It allows to
* speedup sorting. Returns newly allocated lists, old ones stay untouched.
* n_preordered defines a head of list which order should be prevented.
+ *
+ * XXX But we're not generating this only based on uniqueness (that's a bad
+ * term anyway, because we're using ndistinct estimates, not uniqueness).
+ * We're also using the comparator cost to calculate the expected sort cost,
+ * and optimize that.
+ *
+ * XXX This should explain how we generate the values - all permutations for
+ * up to 4 values, etc.
*/
void
get_cheapest_group_keys_order(PlannerInfo *root, double nrows,
@@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows,
else
{
/*
- * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to
- * a new one which can then be cleaned safely if needed.
+ * Since v13 list_free() can clean list elements so for original list
+ * not to be modified it should be copied to a new one which can then
+ * be cleaned safely if needed.
*/
new_group_pathkeys = list_copy(*group_pathkeys);
nToPermute = nFreeKeys;
--
2.29.2
--------------A955BFDEDE369DCD57A45C5A--
^ permalink raw reply [nested|flat] 97+ messages in thread
* [PATCH 2/2] review
@ 2021-03-09 20:09 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw)
---
src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++-------
src/backend/optimizer/path/pathkeys.c | 14 ++++-
2 files changed, 69 insertions(+), 22 deletions(-)
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index 1639258aaf..f55a4f20e5 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm)
* is_fake_var
* Workaround for generate_append_tlist() which generates fake Vars with
* varno == 0, that will cause a fail of estimate_num_group() call
+ *
+ * XXX Ummm, why would estimate_num_group fail with this?
*/
static bool
is_fake_var(Expr *expr)
@@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr)
* compute_cpu_sort_cost
* compute CPU cost of sort (i.e. in-memory)
*
+ * The main thing we need to calculate to estimate sort CPU costs is the number
+ * of calls to the comparator functions. The difficulty is that for multi-column
+ * sorts there may be different data types involved (for some of which the calls
+ * may be much more expensive). Furthermore, the columns may have very different
+ * number of distinct values - the higher the number, the fewer comparisons will
+ * be needed for the following columns.
+ *
+ * The algoritm is incremental - we add pathkeys one by one, and at each step we
+ * estimate the number of necessary comparisons (based on the number of distinct
+ * groups in the current pathkey prefix and the new pathkey), and the comparison
+ * costs (which is data type specific).
+ *
+ * Estimation of the number of comparisons is based on ideas from two sources:
+ *
+ * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/]
+ *
+ * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild,
+ * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906]
+ *
+ * In term of that paper, let N - number of tuples, Xi - number of tuples with
+ * key Ki, then the estimate of number of comparisons is:
+ *
+ * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi))
+ *
+ * In our case all Xi are the same because now we don't have any estimation of
+ * group sizes, we have only know the estimate of number of groups (distinct
+ * values). In that case, formula becomes:
+ *
+ * N * log(NumberOfGroups)
+ *
+ * For multi-column sorts we need to estimate the number of comparisons for
+ * each individual column - for example with columns (c1, c2, ..., ck) we
+ * can estimate that number of comparions on ck is roughly
+ *
+ * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1))
+ *
+ * Let k be a column number, Gk - number of groups defined by k columns, and Fk
+ * the cost of the comparison is
+ *
+ * N * sum( Fk * log(Gk) )
+ *
+ * Note: We also consider column witdth, not just the comparator cost.
+ *
* NOTE: some callers currently pass NIL for pathkeys because they
* can't conveniently supply the sort keys. In this case, it will fallback to
* simple comparison cost estimate.
- *
- * Estimation algorithm is based on ideas from course Algorithms,
- * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper
- * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild,
- * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017.
- *
- * In term of that papers, let N - number of tuples, Xi - number of tuples with
- * key Ki, then estimation is:
- * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi))
- * In our case all Xi are the same because now we don't have an estimation of
- * group sizes, we have only estimation of number of groups. In this case,
- * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation
- * of multi-column sort we need separately compute each column, so, let k is a
- * column number, Gk - number of groups defined by k columns:
- * N * sum( Fk * log(Gk) )
- * Fk is a function costs (including width) for k columns.
*/
-
static Cost
compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
Cost comparison_cost, double tuples, double output_tuples,
@@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
bool has_fake_var = false;
int i = 0;
Oid prev_datatype = InvalidOid;
- Cost funcCost = 0.;
+ Cost funcCost = 0.0;
List *cache_varinfos = NIL;
/* fallback if pathkeys is unknown */
@@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* a total number of tuple comparisons of N log2 K; but the constant
* factor is a bit higher than for quicksort. Tweak it so that the
* cost curve is continuous at the crossover point.
+ *
+ * XXX I suppose the "quicksort factor" references to 1.5 at the end
+ * of this function, but I'm not sure. I suggest we introduce some simple
+ * constants for that, instead of magic values.
*/
output_tuples = (heapSort) ? 2.0 * output_tuples : tuples;
per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples);
@@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* - per column comparison function cost
* - we try to compute needed number of comparison per column
*/
-
foreach(lc, pathkeys)
{
PathKey *pathkey = (PathKey*)lfirst(lc);
@@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* Don't use DEFAULT_NUM_DISTINCT because it used for only one
* column while here we try to estimate number of groups over
* set of columns.
+ *
+ * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to
+ * limit the calculated values, somehow?
+ *
+ * XXX What's the logic of the following formula?
*/
nGroups = ceil(2.0 + sqrt(tuples) *
list_length(pathkeyExprs) / list_length(pathkeys));
@@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
{
if (tuplesPerPrevGroup < output_tuples)
/* comparing only inside output_tuples */
+ /* XXX why not to use the same multiplier (1.5)? */
correctedNGroups =
ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups));
else
@@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups);
/*
- * We could skip all followed columns for cost estimation, because we
+ * We could skip all following columns for cost estimation, because we
* believe that tuples are unique by set ot previous columns
*/
if (tuplesPerPrevGroup <= 1.0)
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 7beb32488a..b092c3e055 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b)
return 0;
return 1;
}
+
/*
* Order tail of list of group pathkeys by uniqueness descendetly. It allows to
* speedup sorting. Returns newly allocated lists, old ones stay untouched.
* n_preordered defines a head of list which order should be prevented.
+ *
+ * XXX But we're not generating this only based on uniqueness (that's a bad
+ * term anyway, because we're using ndistinct estimates, not uniqueness).
+ * We're also using the comparator cost to calculate the expected sort cost,
+ * and optimize that.
+ *
+ * XXX This should explain how we generate the values - all permutations for
+ * up to 4 values, etc.
*/
void
get_cheapest_group_keys_order(PlannerInfo *root, double nrows,
@@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows,
else
{
/*
- * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to
- * a new one which can then be cleaned safely if needed.
+ * Since v13 list_free() can clean list elements so for original list
+ * not to be modified it should be copied to a new one which can then
+ * be cleaned safely if needed.
*/
new_group_pathkeys = list_copy(*group_pathkeys);
nToPermute = nFreeKeys;
--
2.29.2
--------------A955BFDEDE369DCD57A45C5A--
^ permalink raw reply [nested|flat] 97+ messages in thread
* [PATCH 2/2] review
@ 2021-03-09 20:09 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw)
---
src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++-------
src/backend/optimizer/path/pathkeys.c | 14 ++++-
2 files changed, 69 insertions(+), 22 deletions(-)
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index 1639258aaf..f55a4f20e5 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm)
* is_fake_var
* Workaround for generate_append_tlist() which generates fake Vars with
* varno == 0, that will cause a fail of estimate_num_group() call
+ *
+ * XXX Ummm, why would estimate_num_group fail with this?
*/
static bool
is_fake_var(Expr *expr)
@@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr)
* compute_cpu_sort_cost
* compute CPU cost of sort (i.e. in-memory)
*
+ * The main thing we need to calculate to estimate sort CPU costs is the number
+ * of calls to the comparator functions. The difficulty is that for multi-column
+ * sorts there may be different data types involved (for some of which the calls
+ * may be much more expensive). Furthermore, the columns may have very different
+ * number of distinct values - the higher the number, the fewer comparisons will
+ * be needed for the following columns.
+ *
+ * The algoritm is incremental - we add pathkeys one by one, and at each step we
+ * estimate the number of necessary comparisons (based on the number of distinct
+ * groups in the current pathkey prefix and the new pathkey), and the comparison
+ * costs (which is data type specific).
+ *
+ * Estimation of the number of comparisons is based on ideas from two sources:
+ *
+ * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/]
+ *
+ * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild,
+ * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906]
+ *
+ * In term of that paper, let N - number of tuples, Xi - number of tuples with
+ * key Ki, then the estimate of number of comparisons is:
+ *
+ * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi))
+ *
+ * In our case all Xi are the same because now we don't have any estimation of
+ * group sizes, we have only know the estimate of number of groups (distinct
+ * values). In that case, formula becomes:
+ *
+ * N * log(NumberOfGroups)
+ *
+ * For multi-column sorts we need to estimate the number of comparisons for
+ * each individual column - for example with columns (c1, c2, ..., ck) we
+ * can estimate that number of comparions on ck is roughly
+ *
+ * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1))
+ *
+ * Let k be a column number, Gk - number of groups defined by k columns, and Fk
+ * the cost of the comparison is
+ *
+ * N * sum( Fk * log(Gk) )
+ *
+ * Note: We also consider column witdth, not just the comparator cost.
+ *
* NOTE: some callers currently pass NIL for pathkeys because they
* can't conveniently supply the sort keys. In this case, it will fallback to
* simple comparison cost estimate.
- *
- * Estimation algorithm is based on ideas from course Algorithms,
- * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper
- * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild,
- * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017.
- *
- * In term of that papers, let N - number of tuples, Xi - number of tuples with
- * key Ki, then estimation is:
- * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi))
- * In our case all Xi are the same because now we don't have an estimation of
- * group sizes, we have only estimation of number of groups. In this case,
- * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation
- * of multi-column sort we need separately compute each column, so, let k is a
- * column number, Gk - number of groups defined by k columns:
- * N * sum( Fk * log(Gk) )
- * Fk is a function costs (including width) for k columns.
*/
-
static Cost
compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
Cost comparison_cost, double tuples, double output_tuples,
@@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
bool has_fake_var = false;
int i = 0;
Oid prev_datatype = InvalidOid;
- Cost funcCost = 0.;
+ Cost funcCost = 0.0;
List *cache_varinfos = NIL;
/* fallback if pathkeys is unknown */
@@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* a total number of tuple comparisons of N log2 K; but the constant
* factor is a bit higher than for quicksort. Tweak it so that the
* cost curve is continuous at the crossover point.
+ *
+ * XXX I suppose the "quicksort factor" references to 1.5 at the end
+ * of this function, but I'm not sure. I suggest we introduce some simple
+ * constants for that, instead of magic values.
*/
output_tuples = (heapSort) ? 2.0 * output_tuples : tuples;
per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples);
@@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* - per column comparison function cost
* - we try to compute needed number of comparison per column
*/
-
foreach(lc, pathkeys)
{
PathKey *pathkey = (PathKey*)lfirst(lc);
@@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* Don't use DEFAULT_NUM_DISTINCT because it used for only one
* column while here we try to estimate number of groups over
* set of columns.
+ *
+ * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to
+ * limit the calculated values, somehow?
+ *
+ * XXX What's the logic of the following formula?
*/
nGroups = ceil(2.0 + sqrt(tuples) *
list_length(pathkeyExprs) / list_length(pathkeys));
@@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
{
if (tuplesPerPrevGroup < output_tuples)
/* comparing only inside output_tuples */
+ /* XXX why not to use the same multiplier (1.5)? */
correctedNGroups =
ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups));
else
@@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups);
/*
- * We could skip all followed columns for cost estimation, because we
+ * We could skip all following columns for cost estimation, because we
* believe that tuples are unique by set ot previous columns
*/
if (tuplesPerPrevGroup <= 1.0)
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 7beb32488a..b092c3e055 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b)
return 0;
return 1;
}
+
/*
* Order tail of list of group pathkeys by uniqueness descendetly. It allows to
* speedup sorting. Returns newly allocated lists, old ones stay untouched.
* n_preordered defines a head of list which order should be prevented.
+ *
+ * XXX But we're not generating this only based on uniqueness (that's a bad
+ * term anyway, because we're using ndistinct estimates, not uniqueness).
+ * We're also using the comparator cost to calculate the expected sort cost,
+ * and optimize that.
+ *
+ * XXX This should explain how we generate the values - all permutations for
+ * up to 4 values, etc.
*/
void
get_cheapest_group_keys_order(PlannerInfo *root, double nrows,
@@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows,
else
{
/*
- * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to
- * a new one which can then be cleaned safely if needed.
+ * Since v13 list_free() can clean list elements so for original list
+ * not to be modified it should be copied to a new one which can then
+ * be cleaned safely if needed.
*/
new_group_pathkeys = list_copy(*group_pathkeys);
nToPermute = nFreeKeys;
--
2.29.2
--------------A955BFDEDE369DCD57A45C5A--
^ permalink raw reply [nested|flat] 97+ messages in thread
* [PATCH 2/2] review
@ 2021-03-09 20:09 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw)
---
src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++-------
src/backend/optimizer/path/pathkeys.c | 14 ++++-
2 files changed, 69 insertions(+), 22 deletions(-)
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index 1639258aaf..f55a4f20e5 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm)
* is_fake_var
* Workaround for generate_append_tlist() which generates fake Vars with
* varno == 0, that will cause a fail of estimate_num_group() call
+ *
+ * XXX Ummm, why would estimate_num_group fail with this?
*/
static bool
is_fake_var(Expr *expr)
@@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr)
* compute_cpu_sort_cost
* compute CPU cost of sort (i.e. in-memory)
*
+ * The main thing we need to calculate to estimate sort CPU costs is the number
+ * of calls to the comparator functions. The difficulty is that for multi-column
+ * sorts there may be different data types involved (for some of which the calls
+ * may be much more expensive). Furthermore, the columns may have very different
+ * number of distinct values - the higher the number, the fewer comparisons will
+ * be needed for the following columns.
+ *
+ * The algoritm is incremental - we add pathkeys one by one, and at each step we
+ * estimate the number of necessary comparisons (based on the number of distinct
+ * groups in the current pathkey prefix and the new pathkey), and the comparison
+ * costs (which is data type specific).
+ *
+ * Estimation of the number of comparisons is based on ideas from two sources:
+ *
+ * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/]
+ *
+ * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild,
+ * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906]
+ *
+ * In term of that paper, let N - number of tuples, Xi - number of tuples with
+ * key Ki, then the estimate of number of comparisons is:
+ *
+ * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi))
+ *
+ * In our case all Xi are the same because now we don't have any estimation of
+ * group sizes, we have only know the estimate of number of groups (distinct
+ * values). In that case, formula becomes:
+ *
+ * N * log(NumberOfGroups)
+ *
+ * For multi-column sorts we need to estimate the number of comparisons for
+ * each individual column - for example with columns (c1, c2, ..., ck) we
+ * can estimate that number of comparions on ck is roughly
+ *
+ * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1))
+ *
+ * Let k be a column number, Gk - number of groups defined by k columns, and Fk
+ * the cost of the comparison is
+ *
+ * N * sum( Fk * log(Gk) )
+ *
+ * Note: We also consider column witdth, not just the comparator cost.
+ *
* NOTE: some callers currently pass NIL for pathkeys because they
* can't conveniently supply the sort keys. In this case, it will fallback to
* simple comparison cost estimate.
- *
- * Estimation algorithm is based on ideas from course Algorithms,
- * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper
- * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild,
- * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017.
- *
- * In term of that papers, let N - number of tuples, Xi - number of tuples with
- * key Ki, then estimation is:
- * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi))
- * In our case all Xi are the same because now we don't have an estimation of
- * group sizes, we have only estimation of number of groups. In this case,
- * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation
- * of multi-column sort we need separately compute each column, so, let k is a
- * column number, Gk - number of groups defined by k columns:
- * N * sum( Fk * log(Gk) )
- * Fk is a function costs (including width) for k columns.
*/
-
static Cost
compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
Cost comparison_cost, double tuples, double output_tuples,
@@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
bool has_fake_var = false;
int i = 0;
Oid prev_datatype = InvalidOid;
- Cost funcCost = 0.;
+ Cost funcCost = 0.0;
List *cache_varinfos = NIL;
/* fallback if pathkeys is unknown */
@@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* a total number of tuple comparisons of N log2 K; but the constant
* factor is a bit higher than for quicksort. Tweak it so that the
* cost curve is continuous at the crossover point.
+ *
+ * XXX I suppose the "quicksort factor" references to 1.5 at the end
+ * of this function, but I'm not sure. I suggest we introduce some simple
+ * constants for that, instead of magic values.
*/
output_tuples = (heapSort) ? 2.0 * output_tuples : tuples;
per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples);
@@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* - per column comparison function cost
* - we try to compute needed number of comparison per column
*/
-
foreach(lc, pathkeys)
{
PathKey *pathkey = (PathKey*)lfirst(lc);
@@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* Don't use DEFAULT_NUM_DISTINCT because it used for only one
* column while here we try to estimate number of groups over
* set of columns.
+ *
+ * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to
+ * limit the calculated values, somehow?
+ *
+ * XXX What's the logic of the following formula?
*/
nGroups = ceil(2.0 + sqrt(tuples) *
list_length(pathkeyExprs) / list_length(pathkeys));
@@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
{
if (tuplesPerPrevGroup < output_tuples)
/* comparing only inside output_tuples */
+ /* XXX why not to use the same multiplier (1.5)? */
correctedNGroups =
ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups));
else
@@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups);
/*
- * We could skip all followed columns for cost estimation, because we
+ * We could skip all following columns for cost estimation, because we
* believe that tuples are unique by set ot previous columns
*/
if (tuplesPerPrevGroup <= 1.0)
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 7beb32488a..b092c3e055 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b)
return 0;
return 1;
}
+
/*
* Order tail of list of group pathkeys by uniqueness descendetly. It allows to
* speedup sorting. Returns newly allocated lists, old ones stay untouched.
* n_preordered defines a head of list which order should be prevented.
+ *
+ * XXX But we're not generating this only based on uniqueness (that's a bad
+ * term anyway, because we're using ndistinct estimates, not uniqueness).
+ * We're also using the comparator cost to calculate the expected sort cost,
+ * and optimize that.
+ *
+ * XXX This should explain how we generate the values - all permutations for
+ * up to 4 values, etc.
*/
void
get_cheapest_group_keys_order(PlannerInfo *root, double nrows,
@@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows,
else
{
/*
- * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to
- * a new one which can then be cleaned safely if needed.
+ * Since v13 list_free() can clean list elements so for original list
+ * not to be modified it should be copied to a new one which can then
+ * be cleaned safely if needed.
*/
new_group_pathkeys = list_copy(*group_pathkeys);
nToPermute = nFreeKeys;
--
2.29.2
--------------A955BFDEDE369DCD57A45C5A--
^ permalink raw reply [nested|flat] 97+ messages in thread
* [PATCH 2/2] review
@ 2021-03-09 20:09 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw)
---
src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++-------
src/backend/optimizer/path/pathkeys.c | 14 ++++-
2 files changed, 69 insertions(+), 22 deletions(-)
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index 1639258aaf..f55a4f20e5 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm)
* is_fake_var
* Workaround for generate_append_tlist() which generates fake Vars with
* varno == 0, that will cause a fail of estimate_num_group() call
+ *
+ * XXX Ummm, why would estimate_num_group fail with this?
*/
static bool
is_fake_var(Expr *expr)
@@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr)
* compute_cpu_sort_cost
* compute CPU cost of sort (i.e. in-memory)
*
+ * The main thing we need to calculate to estimate sort CPU costs is the number
+ * of calls to the comparator functions. The difficulty is that for multi-column
+ * sorts there may be different data types involved (for some of which the calls
+ * may be much more expensive). Furthermore, the columns may have very different
+ * number of distinct values - the higher the number, the fewer comparisons will
+ * be needed for the following columns.
+ *
+ * The algoritm is incremental - we add pathkeys one by one, and at each step we
+ * estimate the number of necessary comparisons (based on the number of distinct
+ * groups in the current pathkey prefix and the new pathkey), and the comparison
+ * costs (which is data type specific).
+ *
+ * Estimation of the number of comparisons is based on ideas from two sources:
+ *
+ * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/]
+ *
+ * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild,
+ * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906]
+ *
+ * In term of that paper, let N - number of tuples, Xi - number of tuples with
+ * key Ki, then the estimate of number of comparisons is:
+ *
+ * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi))
+ *
+ * In our case all Xi are the same because now we don't have any estimation of
+ * group sizes, we have only know the estimate of number of groups (distinct
+ * values). In that case, formula becomes:
+ *
+ * N * log(NumberOfGroups)
+ *
+ * For multi-column sorts we need to estimate the number of comparisons for
+ * each individual column - for example with columns (c1, c2, ..., ck) we
+ * can estimate that number of comparions on ck is roughly
+ *
+ * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1))
+ *
+ * Let k be a column number, Gk - number of groups defined by k columns, and Fk
+ * the cost of the comparison is
+ *
+ * N * sum( Fk * log(Gk) )
+ *
+ * Note: We also consider column witdth, not just the comparator cost.
+ *
* NOTE: some callers currently pass NIL for pathkeys because they
* can't conveniently supply the sort keys. In this case, it will fallback to
* simple comparison cost estimate.
- *
- * Estimation algorithm is based on ideas from course Algorithms,
- * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper
- * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild,
- * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017.
- *
- * In term of that papers, let N - number of tuples, Xi - number of tuples with
- * key Ki, then estimation is:
- * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi))
- * In our case all Xi are the same because now we don't have an estimation of
- * group sizes, we have only estimation of number of groups. In this case,
- * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation
- * of multi-column sort we need separately compute each column, so, let k is a
- * column number, Gk - number of groups defined by k columns:
- * N * sum( Fk * log(Gk) )
- * Fk is a function costs (including width) for k columns.
*/
-
static Cost
compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
Cost comparison_cost, double tuples, double output_tuples,
@@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
bool has_fake_var = false;
int i = 0;
Oid prev_datatype = InvalidOid;
- Cost funcCost = 0.;
+ Cost funcCost = 0.0;
List *cache_varinfos = NIL;
/* fallback if pathkeys is unknown */
@@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* a total number of tuple comparisons of N log2 K; but the constant
* factor is a bit higher than for quicksort. Tweak it so that the
* cost curve is continuous at the crossover point.
+ *
+ * XXX I suppose the "quicksort factor" references to 1.5 at the end
+ * of this function, but I'm not sure. I suggest we introduce some simple
+ * constants for that, instead of magic values.
*/
output_tuples = (heapSort) ? 2.0 * output_tuples : tuples;
per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples);
@@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* - per column comparison function cost
* - we try to compute needed number of comparison per column
*/
-
foreach(lc, pathkeys)
{
PathKey *pathkey = (PathKey*)lfirst(lc);
@@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* Don't use DEFAULT_NUM_DISTINCT because it used for only one
* column while here we try to estimate number of groups over
* set of columns.
+ *
+ * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to
+ * limit the calculated values, somehow?
+ *
+ * XXX What's the logic of the following formula?
*/
nGroups = ceil(2.0 + sqrt(tuples) *
list_length(pathkeyExprs) / list_length(pathkeys));
@@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
{
if (tuplesPerPrevGroup < output_tuples)
/* comparing only inside output_tuples */
+ /* XXX why not to use the same multiplier (1.5)? */
correctedNGroups =
ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups));
else
@@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups);
/*
- * We could skip all followed columns for cost estimation, because we
+ * We could skip all following columns for cost estimation, because we
* believe that tuples are unique by set ot previous columns
*/
if (tuplesPerPrevGroup <= 1.0)
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 7beb32488a..b092c3e055 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b)
return 0;
return 1;
}
+
/*
* Order tail of list of group pathkeys by uniqueness descendetly. It allows to
* speedup sorting. Returns newly allocated lists, old ones stay untouched.
* n_preordered defines a head of list which order should be prevented.
+ *
+ * XXX But we're not generating this only based on uniqueness (that's a bad
+ * term anyway, because we're using ndistinct estimates, not uniqueness).
+ * We're also using the comparator cost to calculate the expected sort cost,
+ * and optimize that.
+ *
+ * XXX This should explain how we generate the values - all permutations for
+ * up to 4 values, etc.
*/
void
get_cheapest_group_keys_order(PlannerInfo *root, double nrows,
@@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows,
else
{
/*
- * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to
- * a new one which can then be cleaned safely if needed.
+ * Since v13 list_free() can clean list elements so for original list
+ * not to be modified it should be copied to a new one which can then
+ * be cleaned safely if needed.
*/
new_group_pathkeys = list_copy(*group_pathkeys);
nToPermute = nFreeKeys;
--
2.29.2
--------------A955BFDEDE369DCD57A45C5A--
^ permalink raw reply [nested|flat] 97+ messages in thread
* [PATCH 2/2] review
@ 2021-03-09 20:09 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw)
---
src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++-------
src/backend/optimizer/path/pathkeys.c | 14 ++++-
2 files changed, 69 insertions(+), 22 deletions(-)
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index 1639258aaf..f55a4f20e5 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm)
* is_fake_var
* Workaround for generate_append_tlist() which generates fake Vars with
* varno == 0, that will cause a fail of estimate_num_group() call
+ *
+ * XXX Ummm, why would estimate_num_group fail with this?
*/
static bool
is_fake_var(Expr *expr)
@@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr)
* compute_cpu_sort_cost
* compute CPU cost of sort (i.e. in-memory)
*
+ * The main thing we need to calculate to estimate sort CPU costs is the number
+ * of calls to the comparator functions. The difficulty is that for multi-column
+ * sorts there may be different data types involved (for some of which the calls
+ * may be much more expensive). Furthermore, the columns may have very different
+ * number of distinct values - the higher the number, the fewer comparisons will
+ * be needed for the following columns.
+ *
+ * The algoritm is incremental - we add pathkeys one by one, and at each step we
+ * estimate the number of necessary comparisons (based on the number of distinct
+ * groups in the current pathkey prefix and the new pathkey), and the comparison
+ * costs (which is data type specific).
+ *
+ * Estimation of the number of comparisons is based on ideas from two sources:
+ *
+ * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/]
+ *
+ * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild,
+ * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906]
+ *
+ * In term of that paper, let N - number of tuples, Xi - number of tuples with
+ * key Ki, then the estimate of number of comparisons is:
+ *
+ * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi))
+ *
+ * In our case all Xi are the same because now we don't have any estimation of
+ * group sizes, we have only know the estimate of number of groups (distinct
+ * values). In that case, formula becomes:
+ *
+ * N * log(NumberOfGroups)
+ *
+ * For multi-column sorts we need to estimate the number of comparisons for
+ * each individual column - for example with columns (c1, c2, ..., ck) we
+ * can estimate that number of comparions on ck is roughly
+ *
+ * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1))
+ *
+ * Let k be a column number, Gk - number of groups defined by k columns, and Fk
+ * the cost of the comparison is
+ *
+ * N * sum( Fk * log(Gk) )
+ *
+ * Note: We also consider column witdth, not just the comparator cost.
+ *
* NOTE: some callers currently pass NIL for pathkeys because they
* can't conveniently supply the sort keys. In this case, it will fallback to
* simple comparison cost estimate.
- *
- * Estimation algorithm is based on ideas from course Algorithms,
- * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper
- * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild,
- * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017.
- *
- * In term of that papers, let N - number of tuples, Xi - number of tuples with
- * key Ki, then estimation is:
- * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi))
- * In our case all Xi are the same because now we don't have an estimation of
- * group sizes, we have only estimation of number of groups. In this case,
- * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation
- * of multi-column sort we need separately compute each column, so, let k is a
- * column number, Gk - number of groups defined by k columns:
- * N * sum( Fk * log(Gk) )
- * Fk is a function costs (including width) for k columns.
*/
-
static Cost
compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
Cost comparison_cost, double tuples, double output_tuples,
@@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
bool has_fake_var = false;
int i = 0;
Oid prev_datatype = InvalidOid;
- Cost funcCost = 0.;
+ Cost funcCost = 0.0;
List *cache_varinfos = NIL;
/* fallback if pathkeys is unknown */
@@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* a total number of tuple comparisons of N log2 K; but the constant
* factor is a bit higher than for quicksort. Tweak it so that the
* cost curve is continuous at the crossover point.
+ *
+ * XXX I suppose the "quicksort factor" references to 1.5 at the end
+ * of this function, but I'm not sure. I suggest we introduce some simple
+ * constants for that, instead of magic values.
*/
output_tuples = (heapSort) ? 2.0 * output_tuples : tuples;
per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples);
@@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* - per column comparison function cost
* - we try to compute needed number of comparison per column
*/
-
foreach(lc, pathkeys)
{
PathKey *pathkey = (PathKey*)lfirst(lc);
@@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* Don't use DEFAULT_NUM_DISTINCT because it used for only one
* column while here we try to estimate number of groups over
* set of columns.
+ *
+ * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to
+ * limit the calculated values, somehow?
+ *
+ * XXX What's the logic of the following formula?
*/
nGroups = ceil(2.0 + sqrt(tuples) *
list_length(pathkeyExprs) / list_length(pathkeys));
@@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
{
if (tuplesPerPrevGroup < output_tuples)
/* comparing only inside output_tuples */
+ /* XXX why not to use the same multiplier (1.5)? */
correctedNGroups =
ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups));
else
@@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups);
/*
- * We could skip all followed columns for cost estimation, because we
+ * We could skip all following columns for cost estimation, because we
* believe that tuples are unique by set ot previous columns
*/
if (tuplesPerPrevGroup <= 1.0)
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 7beb32488a..b092c3e055 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b)
return 0;
return 1;
}
+
/*
* Order tail of list of group pathkeys by uniqueness descendetly. It allows to
* speedup sorting. Returns newly allocated lists, old ones stay untouched.
* n_preordered defines a head of list which order should be prevented.
+ *
+ * XXX But we're not generating this only based on uniqueness (that's a bad
+ * term anyway, because we're using ndistinct estimates, not uniqueness).
+ * We're also using the comparator cost to calculate the expected sort cost,
+ * and optimize that.
+ *
+ * XXX This should explain how we generate the values - all permutations for
+ * up to 4 values, etc.
*/
void
get_cheapest_group_keys_order(PlannerInfo *root, double nrows,
@@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows,
else
{
/*
- * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to
- * a new one which can then be cleaned safely if needed.
+ * Since v13 list_free() can clean list elements so for original list
+ * not to be modified it should be copied to a new one which can then
+ * be cleaned safely if needed.
*/
new_group_pathkeys = list_copy(*group_pathkeys);
nToPermute = nFreeKeys;
--
2.29.2
--------------A955BFDEDE369DCD57A45C5A--
^ permalink raw reply [nested|flat] 97+ messages in thread
* [PATCH 2/2] review
@ 2021-03-09 20:09 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw)
---
src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++-------
src/backend/optimizer/path/pathkeys.c | 14 ++++-
2 files changed, 69 insertions(+), 22 deletions(-)
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index 1639258aaf..f55a4f20e5 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm)
* is_fake_var
* Workaround for generate_append_tlist() which generates fake Vars with
* varno == 0, that will cause a fail of estimate_num_group() call
+ *
+ * XXX Ummm, why would estimate_num_group fail with this?
*/
static bool
is_fake_var(Expr *expr)
@@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr)
* compute_cpu_sort_cost
* compute CPU cost of sort (i.e. in-memory)
*
+ * The main thing we need to calculate to estimate sort CPU costs is the number
+ * of calls to the comparator functions. The difficulty is that for multi-column
+ * sorts there may be different data types involved (for some of which the calls
+ * may be much more expensive). Furthermore, the columns may have very different
+ * number of distinct values - the higher the number, the fewer comparisons will
+ * be needed for the following columns.
+ *
+ * The algoritm is incremental - we add pathkeys one by one, and at each step we
+ * estimate the number of necessary comparisons (based on the number of distinct
+ * groups in the current pathkey prefix and the new pathkey), and the comparison
+ * costs (which is data type specific).
+ *
+ * Estimation of the number of comparisons is based on ideas from two sources:
+ *
+ * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/]
+ *
+ * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild,
+ * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906]
+ *
+ * In term of that paper, let N - number of tuples, Xi - number of tuples with
+ * key Ki, then the estimate of number of comparisons is:
+ *
+ * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi))
+ *
+ * In our case all Xi are the same because now we don't have any estimation of
+ * group sizes, we have only know the estimate of number of groups (distinct
+ * values). In that case, formula becomes:
+ *
+ * N * log(NumberOfGroups)
+ *
+ * For multi-column sorts we need to estimate the number of comparisons for
+ * each individual column - for example with columns (c1, c2, ..., ck) we
+ * can estimate that number of comparions on ck is roughly
+ *
+ * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1))
+ *
+ * Let k be a column number, Gk - number of groups defined by k columns, and Fk
+ * the cost of the comparison is
+ *
+ * N * sum( Fk * log(Gk) )
+ *
+ * Note: We also consider column witdth, not just the comparator cost.
+ *
* NOTE: some callers currently pass NIL for pathkeys because they
* can't conveniently supply the sort keys. In this case, it will fallback to
* simple comparison cost estimate.
- *
- * Estimation algorithm is based on ideas from course Algorithms,
- * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper
- * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild,
- * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017.
- *
- * In term of that papers, let N - number of tuples, Xi - number of tuples with
- * key Ki, then estimation is:
- * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi))
- * In our case all Xi are the same because now we don't have an estimation of
- * group sizes, we have only estimation of number of groups. In this case,
- * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation
- * of multi-column sort we need separately compute each column, so, let k is a
- * column number, Gk - number of groups defined by k columns:
- * N * sum( Fk * log(Gk) )
- * Fk is a function costs (including width) for k columns.
*/
-
static Cost
compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
Cost comparison_cost, double tuples, double output_tuples,
@@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
bool has_fake_var = false;
int i = 0;
Oid prev_datatype = InvalidOid;
- Cost funcCost = 0.;
+ Cost funcCost = 0.0;
List *cache_varinfos = NIL;
/* fallback if pathkeys is unknown */
@@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* a total number of tuple comparisons of N log2 K; but the constant
* factor is a bit higher than for quicksort. Tweak it so that the
* cost curve is continuous at the crossover point.
+ *
+ * XXX I suppose the "quicksort factor" references to 1.5 at the end
+ * of this function, but I'm not sure. I suggest we introduce some simple
+ * constants for that, instead of magic values.
*/
output_tuples = (heapSort) ? 2.0 * output_tuples : tuples;
per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples);
@@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* - per column comparison function cost
* - we try to compute needed number of comparison per column
*/
-
foreach(lc, pathkeys)
{
PathKey *pathkey = (PathKey*)lfirst(lc);
@@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* Don't use DEFAULT_NUM_DISTINCT because it used for only one
* column while here we try to estimate number of groups over
* set of columns.
+ *
+ * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to
+ * limit the calculated values, somehow?
+ *
+ * XXX What's the logic of the following formula?
*/
nGroups = ceil(2.0 + sqrt(tuples) *
list_length(pathkeyExprs) / list_length(pathkeys));
@@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
{
if (tuplesPerPrevGroup < output_tuples)
/* comparing only inside output_tuples */
+ /* XXX why not to use the same multiplier (1.5)? */
correctedNGroups =
ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups));
else
@@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups);
/*
- * We could skip all followed columns for cost estimation, because we
+ * We could skip all following columns for cost estimation, because we
* believe that tuples are unique by set ot previous columns
*/
if (tuplesPerPrevGroup <= 1.0)
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 7beb32488a..b092c3e055 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b)
return 0;
return 1;
}
+
/*
* Order tail of list of group pathkeys by uniqueness descendetly. It allows to
* speedup sorting. Returns newly allocated lists, old ones stay untouched.
* n_preordered defines a head of list which order should be prevented.
+ *
+ * XXX But we're not generating this only based on uniqueness (that's a bad
+ * term anyway, because we're using ndistinct estimates, not uniqueness).
+ * We're also using the comparator cost to calculate the expected sort cost,
+ * and optimize that.
+ *
+ * XXX This should explain how we generate the values - all permutations for
+ * up to 4 values, etc.
*/
void
get_cheapest_group_keys_order(PlannerInfo *root, double nrows,
@@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows,
else
{
/*
- * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to
- * a new one which can then be cleaned safely if needed.
+ * Since v13 list_free() can clean list elements so for original list
+ * not to be modified it should be copied to a new one which can then
+ * be cleaned safely if needed.
*/
new_group_pathkeys = list_copy(*group_pathkeys);
nToPermute = nFreeKeys;
--
2.29.2
--------------A955BFDEDE369DCD57A45C5A--
^ permalink raw reply [nested|flat] 97+ messages in thread
* [PATCH 02/10] review
@ 2021-03-16 16:29 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: Tomas Vondra @ 2021-03-16 16:29 UTC (permalink / raw)
---
src/backend/optimizer/path/allpaths.c | 2 ++
src/backend/optimizer/plan/initsplan.c | 9 ++++++++-
src/backend/optimizer/util/plancat.c | 3 +++
3 files changed, 13 insertions(+), 1 deletion(-)
diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c
index 37b4223adb..fc1a3a68a2 100644
--- a/src/backend/optimizer/path/allpaths.c
+++ b/src/backend/optimizer/path/allpaths.c
@@ -1065,8 +1065,10 @@ set_append_rel_size(PlannerInfo *root, RelOptInfo *rel,
/* Whole row is not null, so must be same for child */
childrel->notnullattrs = bms_add_member(childrel->notnullattrs,
attno - FirstLowInvalidHeapAttributeNumber);
+ /* XXX shouldn't this be a continue, instead of a break? */
break;
}
+ /* XXX isn't this missing 'else'? */
if (attno < 0 )
/* no need to translate system column */
child_attno = attno;
diff --git a/src/backend/optimizer/plan/initsplan.c b/src/backend/optimizer/plan/initsplan.c
index d27167dc76..4ef876cf7b 100644
--- a/src/backend/optimizer/plan/initsplan.c
+++ b/src/backend/optimizer/plan/initsplan.c
@@ -829,7 +829,14 @@ deconstruct_recurse(PlannerInfo *root, Node *jtnode, bool below_outer_join,
{
Node *qual = (Node *) lfirst(l);
- /* Set the not null info now */
+ /* Set the not null info now
+ *
+ * XXX Why now? Why is this the right place to do this? Does it need
+ * to happen before distribute_qual_to_rels, for example?
+ *
+ * XXX Not clear to me why this looks at non-nullable vars? Shouldn't
+ * we already have the bitmap built from atnums (from get_relation_info)?
+ */
ListCell *lc;
List *non_nullable_vars = find_nonnullable_vars(qual);
foreach(lc, non_nullable_vars)
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index eebabcfccf..cd703e41ba 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -481,6 +481,9 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
if (inhparent && relation->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
set_relation_partition_info(root, rel, relation);
+ /*
+ * Build information about which attributes are marked as NOT NULL.
+ */
Assert(rel->notnullattrs == NULL);
for(i = 0; i < relation->rd_att->natts; i++)
{
--
2.30.2
--------------F495F6B18672582269F00FF9
Content-Type: text/x-patch; charset=UTF-8;
name="0003-Introduce-UniqueKey-attributes-on-RelOptInf-20210317.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename*0="0003-Introduce-UniqueKey-attributes-on-RelOptInf-20210317.pa";
filename*1="tch"
^ permalink raw reply [nested|flat] 97+ messages in thread
* [PATCH 02/10] review
@ 2021-03-16 16:29 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: Tomas Vondra @ 2021-03-16 16:29 UTC (permalink / raw)
---
src/backend/optimizer/path/allpaths.c | 2 ++
src/backend/optimizer/plan/initsplan.c | 9 ++++++++-
src/backend/optimizer/util/plancat.c | 3 +++
3 files changed, 13 insertions(+), 1 deletion(-)
diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c
index 37b4223adb..fc1a3a68a2 100644
--- a/src/backend/optimizer/path/allpaths.c
+++ b/src/backend/optimizer/path/allpaths.c
@@ -1065,8 +1065,10 @@ set_append_rel_size(PlannerInfo *root, RelOptInfo *rel,
/* Whole row is not null, so must be same for child */
childrel->notnullattrs = bms_add_member(childrel->notnullattrs,
attno - FirstLowInvalidHeapAttributeNumber);
+ /* XXX shouldn't this be a continue, instead of a break? */
break;
}
+ /* XXX isn't this missing 'else'? */
if (attno < 0 )
/* no need to translate system column */
child_attno = attno;
diff --git a/src/backend/optimizer/plan/initsplan.c b/src/backend/optimizer/plan/initsplan.c
index d27167dc76..4ef876cf7b 100644
--- a/src/backend/optimizer/plan/initsplan.c
+++ b/src/backend/optimizer/plan/initsplan.c
@@ -829,7 +829,14 @@ deconstruct_recurse(PlannerInfo *root, Node *jtnode, bool below_outer_join,
{
Node *qual = (Node *) lfirst(l);
- /* Set the not null info now */
+ /* Set the not null info now
+ *
+ * XXX Why now? Why is this the right place to do this? Does it need
+ * to happen before distribute_qual_to_rels, for example?
+ *
+ * XXX Not clear to me why this looks at non-nullable vars? Shouldn't
+ * we already have the bitmap built from atnums (from get_relation_info)?
+ */
ListCell *lc;
List *non_nullable_vars = find_nonnullable_vars(qual);
foreach(lc, non_nullable_vars)
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index eebabcfccf..cd703e41ba 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -481,6 +481,9 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
if (inhparent && relation->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
set_relation_partition_info(root, rel, relation);
+ /*
+ * Build information about which attributes are marked as NOT NULL.
+ */
Assert(rel->notnullattrs == NULL);
for(i = 0; i < relation->rd_att->natts; i++)
{
--
2.30.2
--------------F495F6B18672582269F00FF9
Content-Type: text/x-patch; charset=UTF-8;
name="0003-Introduce-UniqueKey-attributes-on-RelOptInf-20210317.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename*0="0003-Introduce-UniqueKey-attributes-on-RelOptInf-20210317.pa";
filename*1="tch"
^ permalink raw reply [nested|flat] 97+ messages in thread
* [PATCH 04/10] review
@ 2021-03-16 18:26 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: Tomas Vondra @ 2021-03-16 18:26 UTC (permalink / raw)
---
src/backend/nodes/copyfuncs.c | 4 +
src/backend/nodes/makefuncs.c | 1 -
src/backend/nodes/outfuncs.c | 1 +
src/backend/nodes/readfuncs.c | 2 +
src/backend/optimizer/path/README.uniquekey | 285 +++---
src/backend/optimizer/path/allpaths.c | 10 +-
src/backend/optimizer/path/joinrels.c | 7 +
src/backend/optimizer/path/uniquekeys.c | 906 ++++++++++++++------
src/backend/optimizer/plan/planner.c | 10 +
src/backend/optimizer/prep/prepunion.c | 1 +
src/backend/optimizer/util/inherit.c | 1 +
11 files changed, 863 insertions(+), 365 deletions(-)
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 75c1c5e824..9d832ddc03 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2296,6 +2296,9 @@ _copyPathKey(const PathKey *from)
return newnode;
}
+/*
+ * _copyUniqueKey
+ */
static UniqueKey *
_copyUniqueKey(const UniqueKey *from)
{
@@ -2306,6 +2309,7 @@ _copyUniqueKey(const UniqueKey *from)
return newnode;
}
+
/*
* _copyRestrictInfo
*/
diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c
index 40415d0f5b..e156c9cdf8 100644
--- a/src/backend/nodes/makefuncs.c
+++ b/src/backend/nodes/makefuncs.c
@@ -816,7 +816,6 @@ makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols)
return v;
}
-
/*
* makeUniqueKey
*/
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 44154cde6a..13905e6037 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2460,6 +2460,7 @@ static void
_outUniqueKey(StringInfo str, const UniqueKey *node)
{
WRITE_NODE_TYPE("UNIQUEKEY");
+
WRITE_NODE_FIELD(exprs);
WRITE_BOOL_FIELD(multi_nullvals);
}
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index b3e212bf1c..8830c8df99 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -496,8 +496,10 @@ static UniqueKey *
_readUniqueKey(void)
{
READ_LOCALS(UniqueKey);
+
READ_NODE_FIELD(exprs);
READ_BOOL_FIELD(multi_nullvals);
+
READ_DONE();
}
diff --git a/src/backend/optimizer/path/README.uniquekey b/src/backend/optimizer/path/README.uniquekey
index 5eac761995..31cdb5ed65 100644
--- a/src/backend/optimizer/path/README.uniquekey
+++ b/src/backend/optimizer/path/README.uniquekey
@@ -1,131 +1,208 @@
-1. What is UniqueKey?
-We can think UniqueKey is a set of exprs for a RelOptInfo, which we are insure
-that doesn't yields same result among all the rows. The simplest UniqueKey
-format is primary key.
+review comments:
+XXX Maybe move this to src/backend/optimizer/README.uniquekey?
+XXX multi_nullvals name seems a bit weird
+XXX no info about populate_distinctrel_uniquekeys, populate_grouprel_uniquekeys, populate_unionrel_uniquekeys
+-----
+src/backend/optimizer/path/README.uniquekey
+
+UniqueKey
+=========
+
+UniqueKey is a set of exprs for a RelOptInfo, which are known to have unique
+values on all the rows in the relation. A trivial example is a primary key
+defined on a relation - each attributes of the constraint is a unique key.
+
+We can use this knowledge to perform optimization in a number of places. Some
+of the optimizations are fairly obvious, others are less so:
+
+1. remove DISTINCT node if the clause is unique
+2. remove aggregation if group by clause is unique
+3. remove_useless_joins
+4. reduce_semianti_joins
+5. Index Skip Scan (WIP)
+6. Aggregation Push-Down without 2-phase aggregation if the join can't
+ duplicate the aggregated rows. (WIP)
+
-However we define the UnqiueKey as below.
+UniqueKey struct
+----------------
-typedef struct UniqueKey
-{
+A UnqiueKey is represented by the following struct:
+
+ typedef struct UniqueKey
+ {
NodeTag type;
List *exprs;
bool multi_nullvals;
-} UniqueKey;
-
-exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL
-is a special case of UniqueKey, which means there is only one row in that
-relation.it has a stronger semantic than others. like SELECT uk FROM t; uk is
-normal unique key and may have different values. SELECT colx FROM t WHERE uk =
-const. colx is unique AND we have only 1 value. This field can used for
-innerrel_is_unique. this logic is handled specially in add_uniquekey_for_onerow
-function.
-
-multi_nullvals: true means multi null values may exist in these exprs, so the
-uniqueness is not guaranteed in this case. This field is necessary for
-remove_useless_join & reduce_unique_semijoins where we don't mind these
-duplicated NULL values. It is set to true for 2 cases. One is a unique key
-from a unique index but the related column is nullable. The other one is for
-outer join. see populate_joinrel_uniquekeys for detail.
-
-
-The UniqueKey can be used at the following cases at least:
-1. remove_useless_joins.
-2. reduce_semianti_joins
-3. remove distinct node if distinct clause is unique.
-4. remove aggnode if group by clause is unique.
-5. Index Skip Scan (WIP)
-6. Aggregation Push Down without 2 phase aggregation if the join can't
- duplicated the aggregated rows. (work in progress feature)
-
-2. How is it maintained?
-
-We have a set of populate_xxx_unqiuekeys functions to maintain the uniquekey on
-various cases. xxx includes baserel, joinrel, partitionedrel, distinctrel,
-groupedrel, unionrel. and we also need to convert the uniquekey from subquery
-to outer relation, which is what convert_subquery_uniquekeys does.
-
-1. The first part is about baserel. We handled 3 cases. suppose we have Unique
-Index on (a, b).
-
-1. SELECT a, b FROM t. UniqueKey (a, b)
-2. SELECT a FROM t WHERE b = 1; UniqueKey (a)
-3. SELECT .. FROM t WHERE a = 1 AND b = 1; UniqueKey (NIL). onerow case, every
- column is Unique.
-
-2. The next part is joinrel, this part is most error-prone, we simplified the rules
-like below:
-1. If the relation's UniqueKey can't be duplicated after join, then is will be
- still valid for the join rel. The function we used here is
- innerrel_keeps_unique. The basic idea is innerrel.any_col = outer.uk.
-
-2. If the UnqiueKey can't keep valid via the rule 1, the combination of the
- UniqueKey from both sides are valid for sure. We can prove this as: if the
- unique exprs from rel1 is duplicated by rel2, the duplicated rows must
- contains different unique exprs from rel2.
-
-More considerations about onerow:
-1. If relation with one row and it can't be duplicated, it is still possible
- contains mulit_nullvas after outer join.
-2. If the either UniqueKey can be duplicated after join, the can get one row
- only when both side is one row AND there is no outer join.
-3. Whenever the onerow UniqueKey is not a valid any more, we need to convert one
- row UniqueKey to normal unique key since we don't store exprs for one-row
- relation. get_exprs_from_uniquekeys will be used here.
-
-
-More considerations about multi_nullvals after join:
+ } UniqueKey;
+
+exprs is a list of exprs which are know to be unique on current RelOptInfo.
+
+exprs = NIL is a special case, meaning there is only one row in the relation.
+This has has a stronger semantic than others. Consider for example
+
+ SELECT uk FROM t
+
+where 'uk' is a unique key. This guarantees uniqueness, but there may be mamy
+rows in the relation. On the other hand, consider this query
+
+ SELECT colx FROM t WHERE uk = const
+
+In this case we know there's only a single matching row (thanks to a condition
+on the unique key), which in turn guarantees uniqueness of the colx value, even
+if there is no constraint on the column itself.
+
+This knowledge is used in innerrel_is_unique, and is handled as a special case
+in add_uniquekey_for_onerow.
+
+
+The multi_nullvals field tracks whether the expressions may contain multiple
+NULL values. This can happen for example when the unique key is derived from
+a unique index with nullable columns, or because of outer joins (which may add
+NULL values to a known-unique list - see populate_joinrel_uniquekeys).
+
+In this case uniqueness is not guaranteed, but we can still use the information
+in places places where NULL values are harmless - when removing useless joins,
+reducing semijoins, and so on.
+
+
+How is it maintained?
+---------------------
+
+Deducing the unique keys depends on the type of the relation - for each case
+there's a separate "populate" function:
+
+
+populate_baserel_uniquekeys
+---------------------------
+
+There are three cases, all assuming there's a unique index (e.g. on (a,b)):
+
+1. SELECT a, b FROM t => UniqueKey (a, b)
+2. SELECT a FROM t WHERE b = 1 => UniqueKey (a)
+3. SELECT .. FROM t WHERE a = 1 AND b = 1; => UniqueKey (NIL)
+
+The last query is the "one row" case, in which case every column is Unique.
+
+
+populate_joinrel_uniquekeys
+---------------------------
+
+For joins, deducing the unique keys may be fairly complex and error-prone.
+We've simplified the rules like this:
+
+1. If the UniqueKey on an input relation can't be duplicated by the join, then
+it will be valid for the join rel. A typical example is a join like this:
+
+ inner_rel.any_col = outer_rel.unique_key
+
+The function used to detect this is innerrel_keeps_unique.
+
+2. Any combination of unique keys on each side of the join is a unique key
+for the join relation. This can be proved by contradiction - assume we have
+unique key on either side of the join - uk1 and uk2. If the values in uk1 get
+duplicated by the join with uk2 (by matching the row to multiple rows), the
+duplicated rows must have different values in the uk2.
+
+We can also leverage information about the "one row" case:
+
+1. If one of the input relations is known to have a single row, and the join
+can't duplicate the row (e.g. semi/anti join), we can keep the unique keys.
+It may however contain multi_nullvals after an outer join.
+
+XXX Not sure I understand the original logic/wording :-(
+
+2. If either UniqueKey can be duplicated after a join, there can be only one
+row only when both sides are "one row" AND there is no outer join.
+
+XXX Why the restriction on not allowing outer joins?
+
+3. Whenever the one row UniqueKey is not a valid any more, we need to convert
+UniqueKey to normal unique key since we don't store exprs for one-row relation.
+This is done by get_exprs_from_uniquekeys.
+
+The join case needs to be careful about multi_nullvals too:
+
1. If the original UnqiueKey has multi_nullvals, the final UniqueKey will have
- mulit_nullvals in any case.
-2. If a unique key doesn't allow mulit_nullvals, after some outer join, it
- allows some outer join.
+mulit_nullvals in any case too.
+
+2. If the original unique key doesn't allow multi_nullvals, the unique key for
+the join relation may allow multi_nullvals after an outer join.
+
+
+subqueries
+----------
+
+It's necessary to "translate" unique keys between a subquery and the outer rels,
+which is what convert_subquery_uniquekeys does. This does almost exactly what
+convert_subquery_pathkeys does for pathkeys. It keeps only unique keys matching
+Vars in the outer relation. The relationship between outerrel.Var and
+subquery.exprs is built from outerel->subroot->processed_tlist.
-3. When we comes to subquery, we need to convert_subquery_unqiuekeys just like
-convert_subquery_pathkeys. Only the UniqueKey insides subquery is referenced as
-a Var in outer relation will be reused. The relationship between the outerrel.Var
-and subquery.exprs is built with outerel->subroot->processed_tlist.
+set-returning functions
+------------------------
+As for the SRF functions, it will break the uniqueness of uniquekey, However it
+is handled in adjust_paths_for_srfs, which happens after the query_planner. So
+we will maintain the UniqueKey until there and reset it to NIL at that place.
-4. As for the SRF functions, it will break the uniqueness of uniquekey, However it
-is handled in adjust_paths_for_srfs, which happens after the query_planner. so
-we will maintain the UniqueKey until there and reset it to NIL at that
-places. This can't help on distinct/group by elimination cases but probably help
-in some other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is
-semantic correctly.
+This can't help on distinct/group by elimination cases but probably help in some
+other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is correct.
-5. As for inherit table, we first main the UnqiueKey on childrel as well. But for
-partitioned table we need to maintain 2 different kinds of
-UnqiueKey. 1). UniqueKey on the parent relation 2). UniqueKey on child
-relation for partition wise query.
+populate_partitionedrel_uniquekeys
+----------------------------------
+
+As for inherit table, we first build the UnqiueKey on childrel as well. But for
+partitioned table we need to maintain two different kinds of UniqueKey:
+
+1) UniqueKey on the parent relation
+
+2) UniqueKey on child
+
+This is needed because a unique key from the partition may not be be unique key
+on the partitioned table.
+
Example:
-CREATE TABLE p (a int not null, b int not null) partition by list (a);
+
+CREATE TABLE p (a INT NOT NULL, b INT NOT NULL) PARTITION BY LIST (a);
+
CREATE TABLE p0 partition of p for values in (1);
CREATE TABLE p1 partition of p for values in (2);
-create unique index p0_b on p0(b);
-create unique index p1_b on p1(b);
+CREATE UNIQUE INDEX p0_b ON p0(b);
+CREATE UNIQUE INDEX p1_b ON p1(b);
-Now b is only unique on partition level, so the distinct can't be removed on
-the following cases. SELECT DISTINCT b FROM p;
+SELECT DISTINCT b FROM p;
-Another example is SELECT DISTINCT a, b FROM p WHERE a = 1; Since only one
-partition is chosen, the UniqueKey on child relation is same as the UniqueKey on
-parent relation.
+Now "b" is only unique on partition level, but the two partitions may contain
+duplicate values for the "b" column (with different values in "a"). That means
+the DISTINCT clause can't be removed.
-Another usage of UniqueKey on partition level is it be helpful for
-partition-wise join.
+Now consider:
-As for the UniqueKey on parent table level, it comes with 2 different ways,
-1). the UniqueKey is also derived in UniqueKey index, but the index must be same
-in all the related children relations and the unique index must contains
-Partition Key in it. Example:
+SELECT DISTINCT a, b FROM p WHERE a = 1
+
+In this case, the optimizer eliminates all partitions except for one, so that
+the UniqueKey is valid for the parent relation too.
+
+UniqueKey at a partition level is useful for partition-wise join too.
+
+XXX Explain why is it useful?
+
+A UniqueKey from a partition can be transferred to the parent relation, in two
+cases. A trivial case is if there's a single child relation (e.g. thanks to
+partition elimination). In that case all unique keys on the child relation are
+automatically valid for the parent relation. If there are multiple relations,
+the unique key must be defived from an index present in all partitions, and the
+index has to include the partition key.
+
+Example:
CREATE UNIQUE INDEX p_ab ON p(a, b); -- where a is the partition key.
-- Query
SELECT a, b FROM p; the (a, b) is a UniqueKey of p.
-2). If the parent relation has only one childrel, the UniqueKey on childrel is
- the UniqueKey on parent as well.
diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c
index 66bf6f19f7..a801707eaa 100644
--- a/src/backend/optimizer/path/allpaths.c
+++ b/src/backend/optimizer/path/allpaths.c
@@ -581,7 +581,8 @@ set_plain_rel_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte)
/*
* Now that we've marked which partial indexes are suitable, we can now
- * build the relation's unique keys.
+ * build the relation's unique keys. We need to do it in this order,
+ * so that we don't deduce unique keys from inapplicable partial indexes.
*/
populate_baserel_uniquekeys(root, rel, rel->indexlist);
@@ -1305,6 +1306,12 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel,
/* Add paths to the append relation. */
add_paths_to_append_rel(root, rel, live_childrels);
+
+ /*
+ * XXX Maybe move the check into populate populate_partitionedrel_uniquekeys?
+ * XXX What if it's append rel (but not partitioned one), but there's only one
+ * child relation? We could still deduce unique keys, no?
+ */
if (IS_PARTITIONED_REL(rel))
populate_partitionedrel_uniquekeys(root, rel, live_childrels);
}
@@ -2314,6 +2321,7 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel,
pathkeys, required_outer));
}
+ /* Convert subpath's unique keys to outer representation */
convert_subquery_uniquekeys(root, rel, sub_final_rel);
/* If outer rel allows parallelism, do same for partial paths. */
diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c
index 7271f044ec..eefba449d6 100644
--- a/src/backend/optimizer/path/joinrels.c
+++ b/src/backend/optimizer/path/joinrels.c
@@ -925,6 +925,13 @@ populate_joinrel_with_paths(PlannerInfo *root, RelOptInfo *rel1,
/* Apply partitionwise join technique, if possible. */
try_partitionwise_join(root, rel1, rel2, joinrel, sjinfo, restrictlist);
+ /*
+ * Determine which of the unique keys from input relations are applicable
+ * for the join result.
+ *
+ * XXX We do this after trying the partitionwise join, because that may allow
+ * using additional unique keys.
+ */
populate_joinrel_uniquekeys(root, joinrel, rel1, rel2, restrictlist, sjinfo->jointype);
}
diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c
index 77ed2b2eff..114e8334f5 100644
--- a/src/backend/optimizer/path/uniquekeys.c
+++ b/src/backend/optimizer/path/uniquekeys.c
@@ -36,7 +36,7 @@ typedef struct UniqueKeyContextData
bool useful;
} *UniqueKeyContext;
-static List *initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel);
+static List *initialize_uniquecontext_for_joinrel(RelOptInfo *inputrel);
static bool innerrel_keeps_unique(PlannerInfo *root,
RelOptInfo *outerrel,
RelOptInfo *innerrel,
@@ -80,8 +80,20 @@ static void add_uniquekey_from_sortgroups(PlannerInfo *root,
/*
* populate_baserel_uniquekeys
- * Populate 'baserel' uniquekeys list by looking at the rel's unique index
- * and baserestrictinfo
+ * Build list of unique keys for the base relation.
+ *
+ * Inspects unique indexes defined on the relation and determines what
+ * unique keys are valid. Partial indexes are considered too, if the
+ * predicate is valid.
+ *
+ * This also inspects baserestrictinfo, because we need to determine
+ * which opclass families are interesting when inspecting indexes. If we
+ * have a unique index and distinct clause with a mismatching opclasses,
+ * we should not use that.
+ *
+ * XXX Why does this look at baserestrictinfo?
+ *
+ * XXX What about collations?
*/
void
populate_baserel_uniquekeys(PlannerInfo *root,
@@ -99,22 +111,48 @@ populate_baserel_uniquekeys(PlannerInfo *root,
Assert(baserel->rtekind == RTE_RELATION);
+ if (!indexlist)
+ return;
+
+ /*
+ * Determine which unique indexes to use to build the unique keys.
+ * We have to skip partial with predicates not matched by the query,
+ * and unique indexes that are not immediately enforced.
+ *
+ * XXX Do we actually skip indexes that are not immediate?
+ * XXX What about hypothetical indexes?
+ */
foreach(lc, indexlist)
{
IndexOptInfo *ind = (IndexOptInfo *) lfirst(lc);
+
if (!ind->unique || !ind->immediate ||
(ind->indpred != NIL && !ind->predOK))
continue;
+
matched_uniq_indexes = lappend(matched_uniq_indexes, ind);
}
+ /* If there are not applicable unique indexes, we're done. */
if (matched_uniq_indexes == NIL)
return;
- /* Check which attrs is used in baserel->reltarget */
- pull_varattnos((Node *)baserel->reltarget->exprs, baserel->relid, &used_attrs);
+ /*
+ * Determine which attrs are referenced in baserel->reltarget. To use the
+ * unique key info, we need all the columns - a unique index on (a,b) may
+ * not be unique on (a). If a column is missing in reltarget, the nodes
+ * above can't possibly use it, and we can just ignore any matching index.
+ */
+ pull_varattnos((Node *) baserel->reltarget->exprs, baserel->relid, &used_attrs);
- /* Check which attrno is used at a mergeable const filter */
+ /*
+ * Check which attrno is used at a mergeable const filter
+ *
+ * XXX This is not lookint att attrno at all, maybe obsolete comment?
+ *
+ * Seems the primary purpose of this is determining which opclass
+ * families to use when matching unique indexes in the next loop?
+ */
foreach(lc, baserel->baserestrictinfo)
{
RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc);
@@ -122,6 +160,10 @@ populate_baserel_uniquekeys(PlannerInfo *root,
if (rinfo->mergeopfamilies == NIL)
continue;
+ /*
+ * XXX What if bms_is_empty is true for both left_relids/right_relids?
+ * Or what if it's false in both cases?
+ */
if (bms_is_empty(rinfo->left_relids))
{
const_exprs = lappend(const_exprs, get_rightop(rinfo->clause));
@@ -136,40 +178,69 @@ populate_baserel_uniquekeys(PlannerInfo *root,
expr_opfamilies = lappend(expr_opfamilies, rinfo->mergeopfamilies);
}
+ /*
+ * Now try to match unique indexes to attributes in reltarget, and to
+ * merge operator families. The index may be on the right attributes,
+ * but if it's not matching the opfamily it's useless.
+ *
+ * XXX Can we have multiple baserestrictinfo for the same attribute,
+ * with different opfamilies? Probably not.
+ */
foreach(lc, matched_uniq_indexes)
{
- bool multi_nullvals, useful;
- List *exprs = get_exprs_from_uniqueindex(lfirst_node(IndexOptInfo, lc),
- const_exprs,
- expr_opfamilies,
- used_attrs,
- &useful,
- &multi_nullvals);
- if (useful)
+ bool multi_nullvals,
+ useful;
+
+ IndexOptInfo *index_info = (IndexOptInfo *) lfirst_node(IndexOptInfo, lc);
+
+ List *exprs = get_exprs_from_uniqueindex(index_info,
+ const_exprs,
+ expr_opfamilies,
+ used_attrs,
+ &useful,
+ &multi_nullvals);
+
+ if (!useful)
+ continue;
+
+ /*
+ * All the columns in Unique Index matched with a restrictinfo, so
+ * that we know there's just a one row in the result. If we find
+ * such index, we're done - we discard all other unique keys and
+ * keep just this special one. In principle, this is a stronger
+ * guarantee, because all subsets of one row are still unique.
+ *
+ * XXX Is it correct to just return? Doesn't that prevent some
+ * optimizations that might be possible with the other keys?
+ */
+ if (exprs == NIL)
{
- if (exprs == NIL)
- {
- /* All the columns in Unique Index matched with a restrictinfo */
- add_uniquekey_for_onerow(baserel);
- return;
- }
- baserel->uniquekeys = lappend(baserel->uniquekeys,
- makeUniqueKey(exprs, multi_nullvals));
+ /* discards all previous uniquekeys */
+ add_uniquekey_for_onerow(baserel);
+ return;
}
+
+ baserel->uniquekeys = lappend(baserel->uniquekeys,
+ makeUniqueKey(exprs, multi_nullvals));
}
}
/*
* populate_partitionedrel_uniquekeys
- * The UniqueKey on partitionrel comes from 2 cases:
- * 1). Only one partition is involved in this query, the unique key can be
- * copied to parent rel from childrel.
- * 2). There are some unique index which includes partition key and exists
- * in all the related partitions.
- * We never mind rule 2 if we hit rule 1.
+ * Determine unique keys for a partitioned relation.
+ *
+ * Inspects unique keys for all partitions and derives unique keys that
+ * are valid for the whole partitioned table. There are two basic cases:
+ *
+ * 1) There's only one remaining partition (thanks to pruning all other
+ * partitions). In this case all the unique keys from the partition are
+ * trivially valid for the partitioned table.
+ *
+ * 2) All the partitions have the same unique index (on the same set of
+ * columns), and the index includes the partition key. This ensures the
+ * combination of values is unique for the whole partitioned table.
*/
-
void
populate_partitionedrel_uniquekeys(PlannerInfo *root,
RelOptInfo *rel,
@@ -180,110 +251,181 @@ populate_partitionedrel_uniquekeys(PlannerInfo *root,
RelOptInfo *childrel;
bool is_first = true;
+ /* XXX What about append rels? At least for the one-child case? */
Assert(IS_PARTITIONED_REL(rel));
+ /* if there are no child relations, we're done. */
if (childrels == NIL)
return;
/*
- * If there is only one partition used in this query, the UniqueKey in childrel is
- * still valid in parent level, but we need convert the format from child expr to
- * parent expr.
+ * If there is only one partition used in this query, the UniqueKey for
+ * a child relation is still valid for the parent level. We need to
+ * convert the format from child expr to parent expr.
*/
if (list_length(childrels) == 1)
{
- /* Check for Rule 1 */
RelOptInfo *childrel = linitial_node(RelOptInfo, childrels);
ListCell *lc;
+
Assert(childrel->reloptkind == RELOPT_OTHER_MEMBER_REL);
+
+ /* If the partition has a single row, so does the parent. */
if (relation_is_onerow(childrel))
{
add_uniquekey_for_onerow(rel);
return;
}
+ /*
+ * Inspect the unique keys one by one, try reusing them for the
+ * parent relation.
+ *
+ * FIXME This needs more work to handle expressions and not just
+ * simple Vars.
+ */
foreach(lc, childrel->uniquekeys)
{
+ ListCell *lc2;
+ List *parent_exprs = NIL;
+ bool can_reuse = true;
+
UniqueKey *ukey = lfirst_node(UniqueKey, lc);
AppendRelInfo *appinfo = find_appinfo_by_child(root, childrel->relid);
- List *parent_exprs = NIL;
- bool can_reuse = true;
- ListCell *lc2;
+
+ /*
+ * XXX Not sure what exactly we do here. Surely we deal with
+ * expressions at child/parent level elsewhere? Can't we just
+ * copy the code from there?
+ */
foreach(lc2, ukey->exprs)
{
- Var *var = (Var *)lfirst(lc2);
+ Var *var = (Var *) lfirst(lc2);
+
/*
- * If the expr comes from a expression, it is hard to build the expression
- * in parent so ignore that case for now.
+ * XXX For now this only supports simple Var expressions,
+ * so if there's a more complex expression we'll not copy
+ * the unique key to the parent.
*/
if(!IsA(var, Var))
{
can_reuse = false;
break;
}
+
/* Convert it to parent var */
- parent_exprs = lappend(parent_exprs, find_parent_var(appinfo, var));
+ parent_exprs = lappend(parent_exprs,
+ find_parent_var(appinfo, var));
}
- if (can_reuse)
- rel->uniquekeys = lappend(rel->uniquekeys,
- makeUniqueKey(parent_exprs,
- ukey->multi_nullvals));
+
+ /* ignore unique keys with complex expressions */
+ if (!can_reuse)
+ continue;
+
+ rel->uniquekeys = lappend(rel->uniquekeys,
+ makeUniqueKey(parent_exprs,
+ ukey->multi_nullvals));
}
+
+ return;
}
- else
+
+ /*
+ * A parent with multiple child relations. We only care about indexes that
+ * are in all child relations, so we loop through indexes on the first one
+ * and check that they exist in the other child relations too.
+ */
+
+ childrel = linitial_node(RelOptInfo, childrels);
+ foreach(lc, childrel->indexlist)
{
- /* Check for rule 2 */
- childrel = linitial_node(RelOptInfo, childrels);
- foreach(lc, childrel->indexlist)
- {
- IndexOptInfo *ind = lfirst(lc);
- IndexOptInfo *modified_index;
- if (!ind->unique || !ind->immediate ||
- (ind->indpred != NIL && !ind->predOK))
- continue;
+ IndexOptInfo *ind = lfirst(lc);
+ IndexOptInfo *modified_index;
- /*
- * During simple_copy_indexinfo_to_parent, we need to convert var from
- * child var to parent var, index on expression is too complex to handle.
- * so ignore it for now.
- */
- if (ind->indexprs != NIL)
- continue;
+ /*
+ * Ignore indexes that are not unique, immediately enforced. Partial
+ * indexes with mismatched predicate are useless too.
+ */
+ if (!ind->unique || !ind->immediate ||
+ (ind->indpred != NIL && !ind->predOK))
+ continue;
- modified_index = simple_copy_indexinfo_to_parent(root, rel, ind);
- /*
- * If the unique index doesn't contain partkey, then it is unique
- * on this partition only, so it is useless for us.
- */
- if (!index_constains_partkey(rel, modified_index))
- continue;
+ /*
+ * During simple_copy_indexinfo_to_parent, we need to convert var from
+ * child var to parent var, index on expression is too complex to handle.
+ * so ignore it for now.
+ *
+ * FIXME We should support indexes on expressions.
+ */
+ if (ind->indexprs != NIL)
+ continue;
- global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index);
- }
+ /*
+ * Adopt the index definition for the parent.
+ *
+ * XXX This seems rather weird. We're constructing "artificial" index
+ * for the partitioned table (kinda like a global index). Can't we
+ * just have some simpler struct representing it?
+ */
+ modified_index = simple_copy_indexinfo_to_parent(root, rel, ind);
+
+ /*
+ * If the unique index doesn't contain partkey, then it is unique
+ * on this partition only, so it is useless for us.
+ *
+ * XXX Can't we do this check before simple_copy_indexinfo_to_parent?
+ */
+ if (!index_constains_partkey(rel, modified_index))
+ continue;
- if (global_uniq_indexlist != NIL)
+ global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index);
+ }
+
+ /* if there are no applicable unique indexes, we're done */
+ if (!global_uniq_indexlist)
+ return;
+
+ /*
+ * We iterate over the child relations first, and inspect the unique
+ * indexes for each hild, because this way we can stop early if we
+ * happen to eliminate all the unique indexes.
+ */
+ foreach(lc, childrels)
+ {
+ RelOptInfo *child = lfirst(lc);
+
+ /* skip the first index, which is where we got the list from */
+ if (is_first)
{
- foreach(lc, childrels)
- {
- RelOptInfo *child = lfirst(lc);
- if (is_first)
- {
- is_first = false;
- continue;
- }
- adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist);
- }
- /* Now we have a list of unique index which are exactly same on all childrels,
- * Set the UniqueKey just like it is non-partition table
- */
- populate_baserel_uniquekeys(root, rel, global_uniq_indexlist);
+ is_first = false;
+ continue;
}
+
+ /* match the unique keys to indexes on this child */
+ adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist);
+
+ /*
+ * If we have eliminated all unique indexes, no point in looking at
+ * the remaining child relations.
+ */
+ if (!global_uniq_indexlist)
+ break;
}
+
+ /* Now we have a list of unique index which are exactly same on all child
+ * relations. Set the UniqueKey just like it is non-partition table.
+ */
+ populate_baserel_uniquekeys(root, rel, global_uniq_indexlist);
}
/*
* populate_distinctrel_uniquekeys
+ * Update unique keys for relation produced by DISTINCT.
+ *
+ * We can keep all unique keys from the input relations, because DISTINCT
+ * can only remove rows - it can't duplicate them. Also, the DISTINCT clause
+ * itself is a unique key, so add that.
*/
void
populate_distinctrel_uniquekeys(PlannerInfo *root,
@@ -292,11 +434,13 @@ populate_distinctrel_uniquekeys(PlannerInfo *root,
{
/* The unique key before the distinct is still valid. */
distinctrel->uniquekeys = list_copy(inputrel->uniquekeys);
+
add_uniquekey_from_sortgroups(root, distinctrel, root->parse->distinctClause);
}
/*
* populate_grouprel_uniquekeys
+ *
*/
void
populate_grouprel_uniquekeys(PlannerInfo *root,
@@ -305,54 +449,76 @@ populate_grouprel_uniquekeys(PlannerInfo *root,
{
Query *parse = root->parse;
- bool input_ukey_added = false;
ListCell *lc;
+ /*
+ * XXX Is this actually valid, before checking fro grouping sets?
+ * The grouping sets may produce duplicate row even with just a single
+ * input row, I think.
+ */
if (relation_is_onerow(inputrel))
{
add_uniquekey_for_onerow(grouprel);
return;
}
+
+ /*
+ * Bail out if there are grouping sets.
+ *
+ * XXX Could we maybe inspect the grouping sets and determine if this
+ * generates distinct combinations? In some cases that's clearly not
+ * the case (rollup, cube), but for some simple cases it might.
+ */
if (parse->groupingSets)
return;
- /* A Normal group by without grouping set. */
- if (parse->groupClause)
+ /* It has aggregation but without a group by, so only one row returned */
+ if (!parse->groupClause)
+ add_uniquekey_for_onerow(grouprel);
+
+ /*
+ * A regular group by, without grouping sets.
+ *
+ * Obviously, the whole group clause determines a unique key. But if
+ * there are smaller unique keys on the input rel, we prefer those
+ * because those are more flexible. If (a,b) is unique, (a,b,c) is
+ * unique too. Only when there are no such smaller unique keys, we
+ * add the unique key derived from the group clause.
+ */
+ foreach(lc, inputrel->uniquekeys)
{
+ UniqueKey *ukey = lfirst_node(UniqueKey, lc);
+
/*
- * Current even the groupby clause is Unique already, but if query has aggref
- * We have to create grouprel still. To keep the UnqiueKey short, we will check
- * the UniqueKey of input_rel still valid, if so we reuse it.
+ * Ignore unique keys on the input that are not subset of the
+ * group clause. We can't use incomplete unique keys.
*/
- foreach(lc, inputrel->uniquekeys)
- {
- UniqueKey *ukey = lfirst_node(UniqueKey, lc);
- if (list_is_subset(ukey->exprs, grouprel->reltarget->exprs))
- {
- grouprel->uniquekeys = lappend(grouprel->uniquekeys,
- ukey);
- input_ukey_added = true;
- }
- }
- if (!input_ukey_added)
- /*
- * group by clause must be a super-set of grouprel->reltarget->exprs except the
- * aggregation expr, so if such exprs is unique already, no bother to generate
- * new uniquekey for group by exprs.
- */
- add_uniquekey_from_sortgroups(root,
- grouprel,
- root->parse->groupClause);
+ if (!list_is_subset(ukey->exprs, grouprel->reltarget->exprs))
+ continue;
+
+ grouprel->uniquekeys = lappend(grouprel->uniquekeys, ukey);
}
- else
- /* It has aggregation but without a group by, so only one row returned */
- add_uniquekey_for_onerow(grouprel);
+
+ /*
+ * Group clause must be a super-set of of grouprel->reltarget->exprs,
+ * except for the aggregation expressions. So if we found a smaller
+ * unique key on the input relation, don't bother adding a unique key
+ * for the group clause.
+ */
+ if (!grouprel->uniquekeys)
+ add_uniquekey_from_sortgroups(root,
+ grouprel,
+ root->parse->groupClause);
}
/*
* simple_copy_uniquekeys
- * Using a function for the one-line code makes us easy to check where we simply
- * copied the uniquekey.
+ * Copy yhe unique keys between relations.
+ *
+ * Using a function for the one-line code makes us easy to check where we
+ * simply copied the uniquekey.
+ *
+ * XXX Seems like an overkill, not sure what's the purpose?
*/
void
simple_copy_uniquekeys(RelOptInfo *oldrel,
@@ -362,24 +528,27 @@ simple_copy_uniquekeys(RelOptInfo *oldrel,
}
/*
- * populate_unionrel_uniquekeys
+ * populate_unionrel_uniquekeys
+ * Determine unique keys for UNION relation.
+ *
+ * XXX Does this need to care about UNION vs. UNION ALL? At least in the
+ * one-row code path?
*/
void
populate_unionrel_uniquekeys(PlannerInfo *root,
- RelOptInfo *unionrel)
+ RelOptInfo *unionrel)
{
- ListCell *lc;
- List *exprs = NIL;
+ ListCell *lc;
+ List *exprs = NIL;
Assert(unionrel->uniquekeys == NIL);
+ /* XXX Why are we copying the expressions? */
foreach(lc, unionrel->reltarget->exprs)
- {
exprs = lappend(exprs, lfirst(lc));
- }
+ /* SQL: select union select; is valid, we need to handle it here. */
if (exprs == NIL)
- /* SQL: select union select; is valid, we need to handle it here. */
add_uniquekey_for_onerow(unionrel);
else
unionrel->uniquekeys = lappend(unionrel->uniquekeys,
@@ -389,6 +558,7 @@ populate_unionrel_uniquekeys(PlannerInfo *root,
/*
* populate_joinrel_uniquekeys
+ * Determine unique keys for a join relation.
*
* populate uniquekeys for joinrel. We will check each relation to see if its
* UniqueKey is still valid via innerrel_keeps_unique, if so, we add it to
@@ -404,70 +574,99 @@ populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel,
RelOptInfo *outerrel, RelOptInfo *innerrel,
List *restrictlist, JoinType jointype)
{
- ListCell *lc, *lc2;
- List *clause_list = NIL;
- List *outerrel_ukey_ctx;
- List *innerrel_ukey_ctx;
- bool inner_onerow, outer_onerow;
- bool mergejoin_allowed;
-
- /* Care about the outerrel relation only for SEMI/ANTI join */
+ ListCell *lc,
+ *lc2;
+ List *clause_list = NIL;
+ List *outerrel_ukey_ctx;
+ List *innerrel_ukey_ctx;
+ bool inner_onerow,
+ outer_onerow;
+ bool mergejoin_allowed;
+
+ /* For SEMI/ANTI join, we care only about the outerrel unique keys. */
if (jointype == JOIN_SEMI || jointype == JOIN_ANTI)
{
foreach(lc, outerrel->uniquekeys)
{
UniqueKey *uniquekey = lfirst_node(UniqueKey, lc);
+
+ /* Keep the unique key if it's included in the joinrel. */
if (list_is_subset(uniquekey->exprs, joinrel->reltarget->exprs))
joinrel->uniquekeys = lappend(joinrel->uniquekeys, uniquekey);
}
+
return;
}
+ /* XXX What about JOIN_RIGHT? */
Assert(jointype == JOIN_LEFT || jointype == JOIN_FULL || jointype == JOIN_INNER);
- /* Fast path */
+ /*
+ * For regular joins, we need to combine unique keys from both sides
+ * of the join, to get a new unique key for the join relation. So if
+ * either side does not have a unique key, bail out.
+ */
if (innerrel->uniquekeys == NIL || outerrel->uniquekeys == NIL)
return;
+ /* XXX maybe move to the if blocks? Not needed outside. */
inner_onerow = relation_is_onerow(innerrel);
outer_onerow = relation_is_onerow(outerrel);
- outerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(outerrel);
- innerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(innerrel);
+ outerrel_ukey_ctx = initialize_uniquecontext_for_joinrel(outerrel);
+ innerrel_ukey_ctx = initialize_uniquecontext_for_joinrel(innerrel);
- clause_list = select_mergejoin_clauses(root, joinrel, outerrel, innerrel,
+ clause_list = select_mergejoin_clauses(root,
+ joinrel, outerrel, innerrel,
restrictlist, jointype,
&mergejoin_allowed);
- if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true /* reverse */))
+ /*
+ * XXX Seems a bit weird that it's called innerrel_keeps_unique but we
+ * seem to use it in both directions. Or what's the "reverse" for? The
+ * "reverse" name is not particularly descriptive.
+ */
+ if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true))
{
- bool outer_impact = jointype == JOIN_FULL;
+ bool outer_impact = (jointype == JOIN_FULL);
+
+ /* Inspect unique keys on the outer relation. */
foreach(lc, outerrel_ukey_ctx)
{
UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc);
+ /*
+ * If the output of the join does not include all the parts of the
+ * unique key, it's useless, so mark it accordingly and ignore it.
+ */
if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs))
{
ctx->useful = false;
continue;
}
- /* Outer relation has one row, and the unique key is not duplicated after join,
- * the joinrel will still has one row unless the jointype == JOIN_FULL.
+ /*
+ * When the outer relation has one row, and the unique key is not
+ * duplicated after join, so the joinrel will still have just one
+ * row unless the jointype == JOIN_FULL. In that case we're done,
+ * it's the strictest unique key possible.
+ *
+ * If it's one-row with a JOIN_FULL, it might produce multiple
+ * rows with NULLs, so set multi_nullvals. We also need to set
+ * the exprs correctly since it can't be NIL any more.
+ *
+ * For other cases (not one-row relation), we just reuse the
+ * unique key, but we may need to tweak the multi_nullvals.
*/
if (outer_onerow && !outer_impact)
{
add_uniquekey_for_onerow(joinrel);
return;
}
- else if (outer_onerow)
+ else if (outer_onerow) /* one-row and FULL join */
{
- /*
- * The onerow outerrel becomes multi rows and multi_nullvals
- * will be changed to true. We also need to set the exprs correctly since it
- * can't be NIL any more.
- */
ListCell *lc2;
+
foreach(lc2, get_exprs_from_uniquekey(root, joinrel, outerrel, NULL))
{
joinrel->uniquekeys = lappend(joinrel->uniquekeys,
@@ -485,18 +684,38 @@ populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel,
joinrel->uniquekeys = lappend(joinrel->uniquekeys,
ctx->uniquekey);
}
+
+ /*
+ * Mark the unique key as added, so that we can ignore it later
+ * when combining unique keys from both sides of the join.
+ */
ctx->added_to_joinrel = true;
}
}
+ /*
+ * XXX Seems this actually checks if "outerrel keeps unique" so the name
+ * is misleading. Of maybe it's the previous block, not sure.
+ *
+ * XXX So why does this consider JOIN_FULL and JOIN_LEFT, while the previous
+ * block only cares about JOIN_FULL?
+ *
+ * XXX This is almost exact copy of the previous block, so maybe make it
+ * a separate function and just call it twice?
+ */
if (innerrel_keeps_unique(root, outerrel, innerrel, clause_list, false))
{
- bool outer_impact = jointype == JOIN_FULL || jointype == JOIN_LEFT;;
+ bool outer_impact = (jointype == JOIN_FULL || jointype == JOIN_LEFT);
+ /* Inspect unique keys on the inner relation. */
foreach(lc, innerrel_ukey_ctx)
{
UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc);
+ /*
+ * If the output of the join does not include all the parts of the
+ * unique key, it's useless, so mark it accordingly and ignore it.
+ */
if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs))
{
ctx->useful = false;
@@ -529,29 +748,52 @@ populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel,
ctx->uniquekey);
}
+
+ /*
+ * Mark the unique key as added, so that we can ignore it later
+ * when combining unique keys from both sides of the join.
+ */
ctx->added_to_joinrel = true;
}
}
/*
- * The combination of the UniqueKey from both sides is unique as well regardless
- * of join type, but no bother to add it if its subset has been added to joinrel
- * already or it is not useful for the joinrel.
+ * XXX What if either of the previous two conditions did not match? In
+ * that case we haven't updated the useful flag, and maybe the unique
+ * key is not useful, but we don't know, right? So we should not be
+ * using it in the next loop. Or maybe we should evaluate the flag
+ * before the loops.
+ */
+
+ /*
+ * The combination of the UniqueKey from both sides is unique as well,
+ * regardless of the join type. But don't bother to add it if its
+ * subset has been added to joinrel already or when it's not useful for
+ * the joinrel.
+ *
+ * XXX Maybe we should have a flag that both sides have useful keys?
+ * Or maybe the loops are short/cheap?
*/
foreach(lc, outerrel_ukey_ctx)
{
UniqueKeyContext ctx1 = (UniqueKeyContext) lfirst(lc);
+
+ /* when not useful or already added to the joinrel, skip it */
if (ctx1->added_to_joinrel || !ctx1->useful)
continue;
+
foreach(lc2, innerrel_ukey_ctx)
{
UniqueKeyContext ctx2 = (UniqueKeyContext) lfirst(lc2);
+
+ /* when not useful or already added to the joinrel, skip it */
if (ctx2->added_to_joinrel || !ctx2->useful)
continue;
+
+ /* If we add a onerow UniqueKey, we don't need another key. */
if (add_combined_uniquekey(root, joinrel, outerrel, innerrel,
ctx1->uniquekey, ctx2->uniquekey,
jointype))
- /* If we set a onerow UniqueKey to joinrel, we don't need other. */
return;
}
}
@@ -560,8 +802,9 @@ populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel,
/*
* convert_subquery_uniquekeys
+ * Covert the UniqueKey in subquery to outer relation.
*
- * Covert the UniqueKey in subquery to outer relation.
+ * XXX Explain what exactly does the conversion do?
*/
void convert_subquery_uniquekeys(PlannerInfo *root,
RelOptInfo *currel,
@@ -618,12 +861,14 @@ void convert_subquery_uniquekeys(PlannerInfo *root,
/*
* innerrel_keeps_unique
+ * Check if Unique key on the innerrel is valid after join.
*
- * Check if Unique key of the innerrel is valid after join. innerrel's UniqueKey
- * will be still valid if innerrel's any-column mergeop outrerel's uniquekey
- * exists in clause_list.
+ * innerrel's UniqueKey will be still valid if innerrel's any-column mergeop
+ * outrerel's uniquekey exists in clause_list
*
* Note: the clause_list must be a list of mergeable restrictinfo already.
+ *
+ * XXX Misleading name? We seem to use it for "outerrel_keeps_unique" too.
*/
static bool
innerrel_keeps_unique(PlannerInfo *root,
@@ -634,26 +879,32 @@ innerrel_keeps_unique(PlannerInfo *root,
{
ListCell *lc, *lc2, *lc3;
+ /* XXX probably not needed, duplicate with the check in the caller
+ * (populate_joinrel_uniquekeys). But it's cheap. */
if (outerrel->uniquekeys == NIL || innerrel->uniquekeys == NIL)
return false;
/* Check if there is outerrel's uniquekey in mergeable clause. */
foreach(lc, outerrel->uniquekeys)
{
- List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs;
- bool clauselist_matchs_all_exprs = true;
+ List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs;
+ bool clauselist_matchs_all_exprs = true;
+
foreach(lc2, outer_uq_exprs)
{
Node *outer_uq_expr = lfirst(lc2);
bool find_uq_expr_in_clauselist = false;
+
foreach(lc3, clause_list)
{
RestrictInfo *rinfo = lfirst_node(RestrictInfo, lc3);
Node *outer_expr;
+
if (reverse)
outer_expr = rinfo->outer_is_left ? get_rightop(rinfo->clause) : get_leftop(rinfo->clause);
else
outer_expr = rinfo->outer_is_left ? get_leftop(rinfo->clause) : get_rightop(rinfo->clause);
+
if (equal(outer_expr, outer_uq_expr))
{
find_uq_expr_in_clauselist = true;
@@ -677,22 +928,37 @@ innerrel_keeps_unique(PlannerInfo *root,
/*
* relation_is_onerow
- * Check if it is a one-row relation by checking UniqueKey.
+ * Check if it is a one-row relation by checking UniqueKey.
+ *
+ * The one-row is a special case - there has to be just a single unique key,
+ * with no expressions.
*/
bool
relation_is_onerow(RelOptInfo *rel)
{
UniqueKey *ukey;
- if (rel->uniquekeys == NIL)
+
+ /* there has to be exactly one unique key */
+ if (list_length(rel->uniquekeys) != 1)
return false;
+
ukey = linitial_node(UniqueKey, rel->uniquekeys);
- return ukey->exprs == NIL && list_length(rel->uniquekeys) == 1;
+
+ /* the unique key must have no expressions */
+ return (ukey->exprs == NIL);
}
/*
* relation_has_uniquekeys_for
- * Returns true if we have proofs that 'rel' cannot return multiple rows with
- * the same values in each of 'exprs'. Otherwise returns false.
+ * Determines if the relation has unique key for a list of expressions.
+ *
+ * Returns true iff we can prove that the relation cannot return multiple rows
+ * with the same values in the provided expression.
+ *
+ * allow_multinulls determines whether we allow multiple NULL values or not.
+ *
+ * The special "one-row" unique key is considered incompatible with all
+ * possible expressions.
*/
bool
relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel,
@@ -710,20 +976,39 @@ relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel,
foreach(lc, rel->uniquekeys)
{
UniqueKey *ukey = lfirst_node(UniqueKey, lc);
+
if (ukey->multi_nullvals && !allow_multinulls)
continue;
+
if (list_is_subset(ukey->exprs, exprs))
return true;
}
+
return false;
}
/*
* get_exprs_from_uniqueindex
+ * Return a list of expressions from a unique index.
+ *
+ * Provided with a list of expressions and opclass families, we try to match
+ * it to the index. If useful, we produce a list of index expressions (subset
+ * of the list we provided).
+ *
+ * We simply walk through the index expressions, and for each expression we
+ * check three things:
*
- * Return a list of exprs which is unique. set useful to false if this
- * unique index is not useful for us.
+ * 1) If there's a matching (expr = Const) clause, we can simply ignore the
+ * expressions. Unique index on (a,b,c) guarantees uniqueness on (a,b) when
+ * there's condition (c=1).
+ *
+ * 2) Check that the index expression is present in the relation we're
+ * dealing with. If not, the unique key would be useless anyway, and the
+ * index can't produce unique key.
+ *
+ * XXX Shouldn't it be enough to return NULL when the index is not useful?
+ * The extra flag seems a bit unnecessary.
*/
static List *
get_exprs_from_uniqueindex(IndexOptInfo *unique_index,
@@ -743,18 +1028,19 @@ get_exprs_from_uniqueindex(IndexOptInfo *unique_index,
indexpr_item = list_head(unique_index->indexprs);
for(c = 0; c < unique_index->ncolumns; c++)
{
- int attr = unique_index->indexkeys[c];
- Expr *expr;
- bool matched_const = false;
- ListCell *lc1, *lc2;
+ int attr = unique_index->indexkeys[c];
+ Expr *expr;
+ bool matched_const = false;
+ ListCell *lc1, *lc2;
- if(attr > 0)
+ if (attr > 0)
{
+ /* regular attribute, just use the expression from index tlist */
expr = list_nth_node(TargetEntry, unique_index->indextlist, c)->expr;
}
else if (attr == 0)
{
- /* Expression index */
+ /* expression from the index */
expr = lfirst(indexpr_item);
indexpr_item = lnext(unique_index->indexprs, indexpr_item);
}
@@ -764,29 +1050,43 @@ get_exprs_from_uniqueindex(IndexOptInfo *unique_index,
Assert(false);
}
+ /* should have a valid expression now */
+ Assert(expr);
+
/*
- * Check index_col = Const case with regarding to opfamily checking
- * If we can remove the index_col from the final UniqueKey->exprs.
+ * Check if there's (index_col = Const) condition, and that it's using
+ * a compatible opfamily. If yes, we can remove the index_col from the
+ * final UniqueKey->exprs, because the value is constant (so removing
+ * it can't introduce duplicities).
*/
forboth(lc1, const_exprs, lc2, const_expr_opfamilies)
{
- if (list_member_oid((List *)lfirst(lc2), unique_index->opfamily[c])
- && match_index_to_operand((Node *) lfirst(lc1), c, unique_index))
+ List *opfamilies = (List *) lfirst(lc2);
+ Node *cexpr = (Node *) lfirst(lc1);
+
+ if (list_member_oid(opfamilies, unique_index->opfamily[c]) &&
+ match_index_to_operand(cexpr, c, unique_index))
{
matched_const = true;
break;
}
}
+ /* it's constant, so ignore the expression */
if (matched_const)
continue;
- /* Check if the indexed expr is used in rel */
+ /*
+ * Check if the indexed expr is used in rel. We do this after the
+ * (col = Const) check, because nn expression may be in a a restrict
+ * clause and not in the reltarget. So we don't want to rule out an
+ * index unnecessarily.
+ */
if (attr > 0)
{
/*
- * Normal Indexed column, if the col is not used, then the index is useless
- * for uniquekey.
+ * Normal indexed column, if the col is not used, then the index
+ * is useless for uniquekey.
*/
attr -= FirstLowInvalidHeapAttributeNumber;
@@ -806,67 +1106,85 @@ get_exprs_from_uniqueindex(IndexOptInfo *unique_index,
/* check not null property. */
if (attr == 0)
{
- /* We never know if a expression yields null or not */
+ /* We never know if an expression yields null or not */
*multi_nullvals = true;
}
- else if (!bms_is_member(attr, unique_index->rel->notnullattrs)
- && !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber,
- unique_index->rel->notnullattrs))
+ else if (!bms_is_member(attr, unique_index->rel->notnullattrs) &&
+ !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber,
+ unique_index->rel->notnullattrs))
{
*multi_nullvals = true;
}
exprs = lappend(exprs, expr);
}
+
return exprs;
}
/*
* add_uniquekey_for_onerow
- * If we are sure that the relation only returns one row, then all the columns
- * are unique. However we don't need to create UniqueKey for every column, we
- * just set exprs = NIL and overwrites all the other UniqueKey on this RelOptInfo
- * since this one has strongest semantics.
+ * Create a special unique key signifying that the rel has one row.
+ *
+ * If we are sure that the relation only returns one row (it might return
+ * no rows, but we still consider that unique), then all the columns are
+ * trivially unique.
+ *
+ * However we don't need to create UniqueKey with every column, we just
+ * set exprs = NIL, because that's easier to identify. We don't want to
+ * add unnecessary unique keys (such that we already have a unique key
+ * for a subset of the expressions), and with (exprs == NIL) we can just
+ * assume we have one unique key for each column in the rel.
+ *
+ * We discard all other unique keys, since it has the strongest semantics.
*/
void
add_uniquekey_for_onerow(RelOptInfo *rel)
{
/*
- * We overwrite the previous UniqueKey on purpose since this one has the
- * strongest semantic.
+ * We overwrite the previous UniqueKey on purpose since this one has
+ * the strongest semantic (all other unique keys are implied by it).
*/
rel->uniquekeys = list_make1(makeUniqueKey(NIL, false));
}
/*
- * initililze_uniquecontext_for_joinrel
- * Return a List of UniqueKeyContext for an inputrel
+ * initialize_uniquecontext_for_joinrel
+ * Return a List of UniqueKeyContext for an inputrel.
*/
static List *
-initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel)
+initialize_uniquecontext_for_joinrel(RelOptInfo *inputrel)
{
- List *res = NIL;
- ListCell *lc;
- foreach(lc, inputrel->uniquekeys)
+ List *res = NIL;
+ ListCell *lc;
+
+ foreach(lc, inputrel->uniquekeys)
{
UniqueKeyContext context;
+
context = palloc(sizeof(struct UniqueKeyContextData));
context->uniquekey = lfirst_node(UniqueKey, lc);
context->added_to_joinrel = false;
context->useful = true;
+
res = lappend(res, context);
}
+
return res;
}
-
/*
* get_exprs_from_uniquekey
- * Unify the way of get List of exprs from a one-row UniqueKey or
- * normal UniqueKey. for the onerow case, every expr in rel1 is a valid
- * UniqueKey. Return a List of exprs.
+ * Extract expressions that are part of a unique key.
+ *
+ * The meaning of the result is a bit different in regular and one-row cases.
+ * For the regular case, the list of expressions form a single unique key,
+ * i.e. the combination of values is unique.
+ *
+ * For the one-row case, each individual expression is known to be unique
+ * (simply because in a single row everything is unique).
*
* rel1: The relation which you want to get the exprs.
* ukey: The UniqueKey you want to get the exprs.
@@ -875,27 +1193,29 @@ static List *
get_exprs_from_uniquekey(PlannerInfo *root, RelOptInfo *joinrel,
RelOptInfo *rel1, UniqueKey *ukey)
{
- ListCell *lc;
- bool onerow = rel1 != NULL && relation_is_onerow(rel1);
+ ListCell *lc;
+ List *res = NIL;
+ bool onerow = (rel1 != NULL) && relation_is_onerow(rel1);
- List *res = NIL;
+ /* We require at least one of those to be true. */
Assert(onerow || ukey);
- if (onerow)
- {
- /* Only cares about the exprs still exist in joinrel */
- foreach(lc, joinrel->reltarget->exprs)
- {
- Bitmapset *relids = pull_varnos(root, lfirst(lc));
- if (bms_is_subset(relids, rel1->relids))
- {
- res = lappend(res, list_make1(lfirst(lc)));
- }
- }
- }
- else
+
+ /* if not a one-row unique key, just return the key's expressions */
+ if (!onerow)
+ return list_make1(ukey->exprs);
+
+ /*
+ * If it's a one-row relation, we simply extract the expressions that
+ * still exist in the reltarget.
+ */
+ foreach(lc, joinrel->reltarget->exprs)
{
- res = list_make1(ukey->exprs);
+ Bitmapset *relids = pull_varnos(root, lfirst(lc));
+
+ if (bms_is_subset(relids, rel1->relids))
+ res = lappend(res, list_make1(lfirst(lc)));
}
+
return res;
}
@@ -910,55 +1230,67 @@ get_exprs_from_uniquekey(PlannerInfo *root, RelOptInfo *joinrel,
/*
* index_constains_partkey
- * return true if the index contains the partiton key.
+ * Determines if the index includes a partition key.
+ *
+ * XXX Surely we already have a code doing this already? E.g. when creating
+ * a unique index on a partitioned table we define that.
*/
static bool
-index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind)
+index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind)
{
ListCell *lc;
int i;
+
Assert(IS_PARTITIONED_REL(partrel));
Assert(partrel->part_scheme->partnatts > 0);
for(i = 0; i < partrel->part_scheme->partnatts; i++)
{
- Node *part_expr = linitial(partrel->partexprs[i]);
- bool found_in_index = false;
+ Node *part_expr = linitial(partrel->partexprs[i]);
+ bool found_in_index = false;
+
foreach(lc, ind->indextlist)
{
- Expr *index_expr = lfirst_node(TargetEntry, lc)->expr;
+ Expr *index_expr = lfirst_node(TargetEntry, lc)->expr;
+
if (equal(index_expr, part_expr))
{
found_in_index = true;
break;
}
}
+
if (!found_in_index)
return false;
}
+
return true;
}
/*
* simple_indexinfo_equal
+ * Compare two indexes to determine if they are the same.
+ *
+ * We need to do this because simple_copy_indexinfo_to_parent does change
+ * some elements. So this is not exactly the same as calling equal().
*
- * Used to check if the 2 index is same as each other. The index here
- * is COPIED from childrel and did some tiny changes(see
- * simple_copy_indexinfo_to_parent)
+ * XXX I wonder if we could simply use equal(), somehow? In fact, we should
+ * probably build something much simpler than IndexOptInfo, just enough to
+ * do the checks.
*/
static bool
simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2)
{
Size oid_cmp_len = sizeof(Oid) * ind1->ncolumns;
- return ind1->ncolumns == ind2->ncolumns &&
- ind1->unique == ind2->unique &&
- memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0 &&
- memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0 &&
- memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0 &&
- memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0 &&
- equal(get_tlist_exprs(ind1->indextlist, true),
- get_tlist_exprs(ind2->indextlist, true));
+ return ((ind1->ncolumns == ind2->ncolumns) &&
+ (ind1->unique == ind2->unique) &&
+ (memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0) &&
+ (memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0) &&
+ (memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0) &&
+ (memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0) &&
+ (equal(get_tlist_exprs(ind1->indextlist, true),
+ get_tlist_exprs(ind2->indextlist, true))));
}
@@ -981,11 +1313,21 @@ simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2)
/*
- * simple_copy_indexinfo_to_parent (from partition)
- * Copy the IndexInfo from child relation to parent relation with some modification,
- * which is used to test:
- * 1. If the same index exists in all the childrels.
+ * simple_copy_indexinfo_to_parent
+ * Copy index info from child to parent, with necessary tweaks.
+ *
+ * We use this copy to check:
+ *
+ * 1. If the same/matching index exists in all the childrels.
* 2. If the parentrel->reltarget/basicrestrict info matches this index.
+ *
+ * XXX IMHO we should probably build something much simpler than a full
+ * IndexOptInfo copy, just enough to do the checks.
+ *
+ * XXX The fact that we copy so much data seems wrong, and having to
+ * define macros from copyfuncs.c seems like a very suspicious thing.
+ * One reason is that IndeOptInfo is fairly large struct, especially
+ * with all the fields, and we allocate it very often.
*/
static IndexOptInfo *
simple_copy_indexinfo_to_parent(PlannerInfo *root,
@@ -1027,20 +1369,24 @@ simple_copy_indexinfo_to_parent(PlannerInfo *root,
/*
* adjust_partition_unique_indexlist
+ * Checks and eliminates indexes that do not exist on the child relation.
*
- * global_unique_indexes: At the beginning, it contains the copy & modified
- * unique index from the first partition. And then check if each index in it still
- * exists in the following partitions. If no, remove it. at last, it has an
- * index list which exists in all the partitions.
+ * Walks the list of unique indexes, and eliminates those that don't match
+ * the child relation (i.e. where a matching child index does not exist).
+ * This is used to iteratively filter the list of candidate unique keys.
+ *
+ * After processing all child relations, the list contains only indexes that
+ * exist in all the child relations.
*/
static void
adjust_partition_unique_indexlist(PlannerInfo *root,
RelOptInfo *parentrel,
RelOptInfo *childrel,
- List **global_unique_indexes)
+ List **indexes)
{
ListCell *lc, *lc2;
- foreach(lc, *global_unique_indexes)
+
+ foreach(lc, *indexes)
{
IndexOptInfo *g_ind = lfirst_node(IndexOptInfo, lc);
bool found_in_child = false;
@@ -1049,23 +1395,45 @@ adjust_partition_unique_indexlist(PlannerInfo *root,
{
IndexOptInfo *p_ind = lfirst_node(IndexOptInfo, lc2);
IndexOptInfo *p_ind_copy;
- if (!p_ind->unique || !p_ind->immediate ||
- (p_ind->indpred != NIL && !p_ind->predOK))
+
+ /*
+ * Ignore child indexes that can't possibly match (not unique or
+ * immediate, etc.)
+ *
+ * XXX We do these checks in many places, so maybe turn it into
+ * a reusable macro?
+ */
+ if ((!p_ind->unique) || (!p_ind->immediate) ||
+ (p_ind->indpred != NIL) && (!p_ind->predOK))
continue;
+
+ /*
+ * XXX This seems possibly quite expensive. Imagine there are many
+ * child relations, with a bunch of unique indexes each. Then this
+ * generates a copy for each unique index in each child relation,
+ * something like O(N^2/2) copies.
+ */
p_ind_copy = simple_copy_indexinfo_to_parent(root, parentrel, p_ind);
+
+ /* Found a matching index for the child relation, we're done. */
if (simple_indexinfo_equal(p_ind_copy, g_ind))
{
found_in_child = true;
break;
}
}
+
+ /* No matching index in the child, so remove it from the list. */
if (!found_in_child)
- /* The index doesn't exist in childrel, remove it from global_unique_indexes */
- *global_unique_indexes = foreach_delete_current(*global_unique_indexes, lc);
+ *indexes = foreach_delete_current(*indexes, lc);
}
}
-/* Helper function for groupres/distinctrel */
+/*
+ * Helper function for groupres/distinctrel
+ *
+ * FIXME Not sure about this.
+ */
static void
add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgroups)
{
@@ -1073,27 +1441,32 @@ add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgrou
List *exprs;
/*
- * XXX: If there are some vars which is not in current levelsup, the semantic is
- * imprecise, should we avoid it or not? levelsup = 1 is just a demo, maybe we need to
- * check every level other than 0, if so, looks we have to write another
- * pull_var_walker.
+ * XXX: If there are some vars which are not in the current levelsup, the
+ * semantic is imprecise, should we avoid it or not? levelsup = 1 is just
+ * a demo, maybe we need to check every level other than 0, if so, looks
+ * we have to write another pull_var_walker.
*/
List *upper_vars = pull_vars_of_level((Node*)sortgroups, 1);
if (upper_vars != NIL)
return;
+ /* sortgroupclause can't be multi_nullvals */
exprs = get_sortgrouplist_exprs(sortgroups, parse->targetList);
rel->uniquekeys = lappend(rel->uniquekeys,
- makeUniqueKey(exprs,
- false /* sortgroupclause can't be multi_nullvals */));
+ makeUniqueKey(exprs, false));
}
/*
* add_combined_uniquekey
- * The combination of both UniqueKeys is a valid UniqueKey for joinrel no matter
- * the jointype.
+ * Add a unique key for a join, combined from keys on inner/outer side.
+ *
+ * The combination of both UniqueKeys is a valid UniqueKey for joinrel no
+ * matter what's the exact jointype.
+ *
+ * Returns true if the unique key is "one-row" variant, so that the caller
+ * can stop considering further combinations.
*/
bool
add_combined_uniquekey(PlannerInfo *root,
@@ -1104,32 +1477,47 @@ add_combined_uniquekey(PlannerInfo *root,
UniqueKey *inner_ukey,
JoinType jointype)
{
+ bool multi_nullvals;
+ ListCell *lc1, *lc2;
- ListCell *lc1, *lc2;
-
- /* Either side has multi_nullvals or we have outer join,
- * the combined UniqueKey has multi_nullvals */
- bool multi_nullvals = outer_ukey->multi_nullvals ||
+ /*
+ * If either side has multi_nullvals, or we are dealing with an outer join,
+ * the combined UniqueKey has multi_nullvals too.
+ */
+ multi_nullvals = outer_ukey->multi_nullvals ||
inner_ukey->multi_nullvals || IS_OUTER_JOIN(jointype);
/* The only case we can get onerow joinrel after join */
- if (relation_is_onerow(outer_rel)
- && relation_is_onerow(inner_rel)
- && jointype == JOIN_INNER)
+ if (relation_is_onerow(outer_rel) &&
+ relation_is_onerow(inner_rel) &&
+ jointype == JOIN_INNER)
{
add_uniquekey_for_onerow(joinrel);
return true;
}
+ /*
+ * XXX Isn't this wrong? Why is it combining expressions that are part
+ * of the two unique keys? Imagine we have outer unique key on (a1, a2)
+ * and inner outer key on (b1, b2). Then this adds four unique keys
+ * for the join (a1,b1), (a1,b2), (a2,b1) and (a2,b2). Shouldn't it
+ * just add (a1,a2,b1,b2)?
+ */
foreach(lc1, get_exprs_from_uniquekey(root, joinrel, outer_rel, outer_ukey))
{
+ /*
+ * XXX This calls get_exprs_from_uniquekey repeatedly for each outer
+ * loop. Maybe we should calculate it just once before the loop.
+ */
foreach(lc2, get_exprs_from_uniquekey(root, joinrel, inner_rel, inner_ukey))
{
List *exprs = list_concat_copy(lfirst_node(List, lc1), lfirst_node(List, lc2));
+
joinrel->uniquekeys = lappend(joinrel->uniquekeys,
makeUniqueKey(exprs,
multi_nullvals));
}
}
+
return false;
}
diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c
index 8d8e493f5c..f29b65c07b 100644
--- a/src/backend/optimizer/plan/planner.c
+++ b/src/backend/optimizer/plan/planner.c
@@ -2387,6 +2387,7 @@ grouping_planner(PlannerInfo *root, bool inheritance_update,
add_path(final_rel, path);
}
+ /* XXX comment? can we simply just copy the unique keys to the final relation? */
simple_copy_uniquekeys(current_rel, final_rel);
/*
@@ -3902,7 +3903,9 @@ create_grouping_paths(PlannerInfo *root,
set_cheapest(grouped_rel);
+ /* XXX does this apply to grouping sets too? */
populate_grouprel_uniquekeys(root, grouped_rel, input_rel);
+
return grouped_rel;
}
@@ -4625,7 +4628,10 @@ create_window_paths(PlannerInfo *root,
/* Now choose the best path(s) */
set_cheapest(window_rel);
+
+ /* XXX comment? */
simple_copy_uniquekeys(input_rel, window_rel);
+
return window_rel;
}
@@ -4939,7 +4945,10 @@ create_distinct_paths(PlannerInfo *root,
/* Now choose the best path(s) */
set_cheapest(distinct_rel);
+
+ /* XXX comment */
populate_distinctrel_uniquekeys(root, input_rel, distinct_rel);
+
return distinct_rel;
}
@@ -5200,6 +5209,7 @@ create_ordered_paths(PlannerInfo *root,
*/
Assert(ordered_rel->pathlist != NIL);
+ /* XXX comment */
simple_copy_uniquekeys(input_rel, ordered_rel);
return ordered_rel;
diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c
index b7626545bf..72a3f3c598 100644
--- a/src/backend/optimizer/prep/prepunion.c
+++ b/src/backend/optimizer/prep/prepunion.c
@@ -691,6 +691,7 @@ generate_union_paths(SetOperationStmt *op, PlannerInfo *root,
/* Add the UniqueKeys */
populate_unionrel_uniquekeys(root, result_rel);
+
return result_rel;
}
diff --git a/src/backend/optimizer/util/inherit.c b/src/backend/optimizer/util/inherit.c
index 3eec1f4d74..c9829c5fc4 100644
--- a/src/backend/optimizer/util/inherit.c
+++ b/src/backend/optimizer/util/inherit.c
@@ -755,6 +755,7 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel,
pseudoconstant,
rinfo->security_level,
NULL, NULL, NULL);
+ /* XXX This is a bit weird, doing this outside make_restrictinfo */
child_rinfo->mergeopfamilies = rinfo->mergeopfamilies;
childquals = lappend(childquals, child_rinfo);
/* track minimum security level among child quals */
--
2.30.2
--------------F495F6B18672582269F00FF9
Content-Type: text/x-patch; charset=UTF-8;
name="0005-Extend-UniqueKeys-20210317.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="0005-Extend-UniqueKeys-20210317.patch"
^ permalink raw reply [nested|flat] 97+ messages in thread
* [PATCH 04/10] review
@ 2021-03-16 18:26 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: Tomas Vondra @ 2021-03-16 18:26 UTC (permalink / raw)
---
src/backend/nodes/copyfuncs.c | 4 +
src/backend/nodes/makefuncs.c | 1 -
src/backend/nodes/outfuncs.c | 1 +
src/backend/nodes/readfuncs.c | 2 +
src/backend/optimizer/path/README.uniquekey | 285 +++---
src/backend/optimizer/path/allpaths.c | 10 +-
src/backend/optimizer/path/joinrels.c | 7 +
src/backend/optimizer/path/uniquekeys.c | 906 ++++++++++++++------
src/backend/optimizer/plan/planner.c | 10 +
src/backend/optimizer/prep/prepunion.c | 1 +
src/backend/optimizer/util/inherit.c | 1 +
11 files changed, 863 insertions(+), 365 deletions(-)
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 75c1c5e824..9d832ddc03 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2296,6 +2296,9 @@ _copyPathKey(const PathKey *from)
return newnode;
}
+/*
+ * _copyUniqueKey
+ */
static UniqueKey *
_copyUniqueKey(const UniqueKey *from)
{
@@ -2306,6 +2309,7 @@ _copyUniqueKey(const UniqueKey *from)
return newnode;
}
+
/*
* _copyRestrictInfo
*/
diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c
index 40415d0f5b..e156c9cdf8 100644
--- a/src/backend/nodes/makefuncs.c
+++ b/src/backend/nodes/makefuncs.c
@@ -816,7 +816,6 @@ makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols)
return v;
}
-
/*
* makeUniqueKey
*/
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 44154cde6a..13905e6037 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2460,6 +2460,7 @@ static void
_outUniqueKey(StringInfo str, const UniqueKey *node)
{
WRITE_NODE_TYPE("UNIQUEKEY");
+
WRITE_NODE_FIELD(exprs);
WRITE_BOOL_FIELD(multi_nullvals);
}
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index b3e212bf1c..8830c8df99 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -496,8 +496,10 @@ static UniqueKey *
_readUniqueKey(void)
{
READ_LOCALS(UniqueKey);
+
READ_NODE_FIELD(exprs);
READ_BOOL_FIELD(multi_nullvals);
+
READ_DONE();
}
diff --git a/src/backend/optimizer/path/README.uniquekey b/src/backend/optimizer/path/README.uniquekey
index 5eac761995..31cdb5ed65 100644
--- a/src/backend/optimizer/path/README.uniquekey
+++ b/src/backend/optimizer/path/README.uniquekey
@@ -1,131 +1,208 @@
-1. What is UniqueKey?
-We can think UniqueKey is a set of exprs for a RelOptInfo, which we are insure
-that doesn't yields same result among all the rows. The simplest UniqueKey
-format is primary key.
+review comments:
+XXX Maybe move this to src/backend/optimizer/README.uniquekey?
+XXX multi_nullvals name seems a bit weird
+XXX no info about populate_distinctrel_uniquekeys, populate_grouprel_uniquekeys, populate_unionrel_uniquekeys
+-----
+src/backend/optimizer/path/README.uniquekey
+
+UniqueKey
+=========
+
+UniqueKey is a set of exprs for a RelOptInfo, which are known to have unique
+values on all the rows in the relation. A trivial example is a primary key
+defined on a relation - each attributes of the constraint is a unique key.
+
+We can use this knowledge to perform optimization in a number of places. Some
+of the optimizations are fairly obvious, others are less so:
+
+1. remove DISTINCT node if the clause is unique
+2. remove aggregation if group by clause is unique
+3. remove_useless_joins
+4. reduce_semianti_joins
+5. Index Skip Scan (WIP)
+6. Aggregation Push-Down without 2-phase aggregation if the join can't
+ duplicate the aggregated rows. (WIP)
+
-However we define the UnqiueKey as below.
+UniqueKey struct
+----------------
-typedef struct UniqueKey
-{
+A UnqiueKey is represented by the following struct:
+
+ typedef struct UniqueKey
+ {
NodeTag type;
List *exprs;
bool multi_nullvals;
-} UniqueKey;
-
-exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL
-is a special case of UniqueKey, which means there is only one row in that
-relation.it has a stronger semantic than others. like SELECT uk FROM t; uk is
-normal unique key and may have different values. SELECT colx FROM t WHERE uk =
-const. colx is unique AND we have only 1 value. This field can used for
-innerrel_is_unique. this logic is handled specially in add_uniquekey_for_onerow
-function.
-
-multi_nullvals: true means multi null values may exist in these exprs, so the
-uniqueness is not guaranteed in this case. This field is necessary for
-remove_useless_join & reduce_unique_semijoins where we don't mind these
-duplicated NULL values. It is set to true for 2 cases. One is a unique key
-from a unique index but the related column is nullable. The other one is for
-outer join. see populate_joinrel_uniquekeys for detail.
-
-
-The UniqueKey can be used at the following cases at least:
-1. remove_useless_joins.
-2. reduce_semianti_joins
-3. remove distinct node if distinct clause is unique.
-4. remove aggnode if group by clause is unique.
-5. Index Skip Scan (WIP)
-6. Aggregation Push Down without 2 phase aggregation if the join can't
- duplicated the aggregated rows. (work in progress feature)
-
-2. How is it maintained?
-
-We have a set of populate_xxx_unqiuekeys functions to maintain the uniquekey on
-various cases. xxx includes baserel, joinrel, partitionedrel, distinctrel,
-groupedrel, unionrel. and we also need to convert the uniquekey from subquery
-to outer relation, which is what convert_subquery_uniquekeys does.
-
-1. The first part is about baserel. We handled 3 cases. suppose we have Unique
-Index on (a, b).
-
-1. SELECT a, b FROM t. UniqueKey (a, b)
-2. SELECT a FROM t WHERE b = 1; UniqueKey (a)
-3. SELECT .. FROM t WHERE a = 1 AND b = 1; UniqueKey (NIL). onerow case, every
- column is Unique.
-
-2. The next part is joinrel, this part is most error-prone, we simplified the rules
-like below:
-1. If the relation's UniqueKey can't be duplicated after join, then is will be
- still valid for the join rel. The function we used here is
- innerrel_keeps_unique. The basic idea is innerrel.any_col = outer.uk.
-
-2. If the UnqiueKey can't keep valid via the rule 1, the combination of the
- UniqueKey from both sides are valid for sure. We can prove this as: if the
- unique exprs from rel1 is duplicated by rel2, the duplicated rows must
- contains different unique exprs from rel2.
-
-More considerations about onerow:
-1. If relation with one row and it can't be duplicated, it is still possible
- contains mulit_nullvas after outer join.
-2. If the either UniqueKey can be duplicated after join, the can get one row
- only when both side is one row AND there is no outer join.
-3. Whenever the onerow UniqueKey is not a valid any more, we need to convert one
- row UniqueKey to normal unique key since we don't store exprs for one-row
- relation. get_exprs_from_uniquekeys will be used here.
-
-
-More considerations about multi_nullvals after join:
+ } UniqueKey;
+
+exprs is a list of exprs which are know to be unique on current RelOptInfo.
+
+exprs = NIL is a special case, meaning there is only one row in the relation.
+This has has a stronger semantic than others. Consider for example
+
+ SELECT uk FROM t
+
+where 'uk' is a unique key. This guarantees uniqueness, but there may be mamy
+rows in the relation. On the other hand, consider this query
+
+ SELECT colx FROM t WHERE uk = const
+
+In this case we know there's only a single matching row (thanks to a condition
+on the unique key), which in turn guarantees uniqueness of the colx value, even
+if there is no constraint on the column itself.
+
+This knowledge is used in innerrel_is_unique, and is handled as a special case
+in add_uniquekey_for_onerow.
+
+
+The multi_nullvals field tracks whether the expressions may contain multiple
+NULL values. This can happen for example when the unique key is derived from
+a unique index with nullable columns, or because of outer joins (which may add
+NULL values to a known-unique list - see populate_joinrel_uniquekeys).
+
+In this case uniqueness is not guaranteed, but we can still use the information
+in places places where NULL values are harmless - when removing useless joins,
+reducing semijoins, and so on.
+
+
+How is it maintained?
+---------------------
+
+Deducing the unique keys depends on the type of the relation - for each case
+there's a separate "populate" function:
+
+
+populate_baserel_uniquekeys
+---------------------------
+
+There are three cases, all assuming there's a unique index (e.g. on (a,b)):
+
+1. SELECT a, b FROM t => UniqueKey (a, b)
+2. SELECT a FROM t WHERE b = 1 => UniqueKey (a)
+3. SELECT .. FROM t WHERE a = 1 AND b = 1; => UniqueKey (NIL)
+
+The last query is the "one row" case, in which case every column is Unique.
+
+
+populate_joinrel_uniquekeys
+---------------------------
+
+For joins, deducing the unique keys may be fairly complex and error-prone.
+We've simplified the rules like this:
+
+1. If the UniqueKey on an input relation can't be duplicated by the join, then
+it will be valid for the join rel. A typical example is a join like this:
+
+ inner_rel.any_col = outer_rel.unique_key
+
+The function used to detect this is innerrel_keeps_unique.
+
+2. Any combination of unique keys on each side of the join is a unique key
+for the join relation. This can be proved by contradiction - assume we have
+unique key on either side of the join - uk1 and uk2. If the values in uk1 get
+duplicated by the join with uk2 (by matching the row to multiple rows), the
+duplicated rows must have different values in the uk2.
+
+We can also leverage information about the "one row" case:
+
+1. If one of the input relations is known to have a single row, and the join
+can't duplicate the row (e.g. semi/anti join), we can keep the unique keys.
+It may however contain multi_nullvals after an outer join.
+
+XXX Not sure I understand the original logic/wording :-(
+
+2. If either UniqueKey can be duplicated after a join, there can be only one
+row only when both sides are "one row" AND there is no outer join.
+
+XXX Why the restriction on not allowing outer joins?
+
+3. Whenever the one row UniqueKey is not a valid any more, we need to convert
+UniqueKey to normal unique key since we don't store exprs for one-row relation.
+This is done by get_exprs_from_uniquekeys.
+
+The join case needs to be careful about multi_nullvals too:
+
1. If the original UnqiueKey has multi_nullvals, the final UniqueKey will have
- mulit_nullvals in any case.
-2. If a unique key doesn't allow mulit_nullvals, after some outer join, it
- allows some outer join.
+mulit_nullvals in any case too.
+
+2. If the original unique key doesn't allow multi_nullvals, the unique key for
+the join relation may allow multi_nullvals after an outer join.
+
+
+subqueries
+----------
+
+It's necessary to "translate" unique keys between a subquery and the outer rels,
+which is what convert_subquery_uniquekeys does. This does almost exactly what
+convert_subquery_pathkeys does for pathkeys. It keeps only unique keys matching
+Vars in the outer relation. The relationship between outerrel.Var and
+subquery.exprs is built from outerel->subroot->processed_tlist.
-3. When we comes to subquery, we need to convert_subquery_unqiuekeys just like
-convert_subquery_pathkeys. Only the UniqueKey insides subquery is referenced as
-a Var in outer relation will be reused. The relationship between the outerrel.Var
-and subquery.exprs is built with outerel->subroot->processed_tlist.
+set-returning functions
+------------------------
+As for the SRF functions, it will break the uniqueness of uniquekey, However it
+is handled in adjust_paths_for_srfs, which happens after the query_planner. So
+we will maintain the UniqueKey until there and reset it to NIL at that place.
-4. As for the SRF functions, it will break the uniqueness of uniquekey, However it
-is handled in adjust_paths_for_srfs, which happens after the query_planner. so
-we will maintain the UniqueKey until there and reset it to NIL at that
-places. This can't help on distinct/group by elimination cases but probably help
-in some other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is
-semantic correctly.
+This can't help on distinct/group by elimination cases but probably help in some
+other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is correct.
-5. As for inherit table, we first main the UnqiueKey on childrel as well. But for
-partitioned table we need to maintain 2 different kinds of
-UnqiueKey. 1). UniqueKey on the parent relation 2). UniqueKey on child
-relation for partition wise query.
+populate_partitionedrel_uniquekeys
+----------------------------------
+
+As for inherit table, we first build the UnqiueKey on childrel as well. But for
+partitioned table we need to maintain two different kinds of UniqueKey:
+
+1) UniqueKey on the parent relation
+
+2) UniqueKey on child
+
+This is needed because a unique key from the partition may not be be unique key
+on the partitioned table.
+
Example:
-CREATE TABLE p (a int not null, b int not null) partition by list (a);
+
+CREATE TABLE p (a INT NOT NULL, b INT NOT NULL) PARTITION BY LIST (a);
+
CREATE TABLE p0 partition of p for values in (1);
CREATE TABLE p1 partition of p for values in (2);
-create unique index p0_b on p0(b);
-create unique index p1_b on p1(b);
+CREATE UNIQUE INDEX p0_b ON p0(b);
+CREATE UNIQUE INDEX p1_b ON p1(b);
-Now b is only unique on partition level, so the distinct can't be removed on
-the following cases. SELECT DISTINCT b FROM p;
+SELECT DISTINCT b FROM p;
-Another example is SELECT DISTINCT a, b FROM p WHERE a = 1; Since only one
-partition is chosen, the UniqueKey on child relation is same as the UniqueKey on
-parent relation.
+Now "b" is only unique on partition level, but the two partitions may contain
+duplicate values for the "b" column (with different values in "a"). That means
+the DISTINCT clause can't be removed.
-Another usage of UniqueKey on partition level is it be helpful for
-partition-wise join.
+Now consider:
-As for the UniqueKey on parent table level, it comes with 2 different ways,
-1). the UniqueKey is also derived in UniqueKey index, but the index must be same
-in all the related children relations and the unique index must contains
-Partition Key in it. Example:
+SELECT DISTINCT a, b FROM p WHERE a = 1
+
+In this case, the optimizer eliminates all partitions except for one, so that
+the UniqueKey is valid for the parent relation too.
+
+UniqueKey at a partition level is useful for partition-wise join too.
+
+XXX Explain why is it useful?
+
+A UniqueKey from a partition can be transferred to the parent relation, in two
+cases. A trivial case is if there's a single child relation (e.g. thanks to
+partition elimination). In that case all unique keys on the child relation are
+automatically valid for the parent relation. If there are multiple relations,
+the unique key must be defived from an index present in all partitions, and the
+index has to include the partition key.
+
+Example:
CREATE UNIQUE INDEX p_ab ON p(a, b); -- where a is the partition key.
-- Query
SELECT a, b FROM p; the (a, b) is a UniqueKey of p.
-2). If the parent relation has only one childrel, the UniqueKey on childrel is
- the UniqueKey on parent as well.
diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c
index 66bf6f19f7..a801707eaa 100644
--- a/src/backend/optimizer/path/allpaths.c
+++ b/src/backend/optimizer/path/allpaths.c
@@ -581,7 +581,8 @@ set_plain_rel_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte)
/*
* Now that we've marked which partial indexes are suitable, we can now
- * build the relation's unique keys.
+ * build the relation's unique keys. We need to do it in this order,
+ * so that we don't deduce unique keys from inapplicable partial indexes.
*/
populate_baserel_uniquekeys(root, rel, rel->indexlist);
@@ -1305,6 +1306,12 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel,
/* Add paths to the append relation. */
add_paths_to_append_rel(root, rel, live_childrels);
+
+ /*
+ * XXX Maybe move the check into populate populate_partitionedrel_uniquekeys?
+ * XXX What if it's append rel (but not partitioned one), but there's only one
+ * child relation? We could still deduce unique keys, no?
+ */
if (IS_PARTITIONED_REL(rel))
populate_partitionedrel_uniquekeys(root, rel, live_childrels);
}
@@ -2314,6 +2321,7 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel,
pathkeys, required_outer));
}
+ /* Convert subpath's unique keys to outer representation */
convert_subquery_uniquekeys(root, rel, sub_final_rel);
/* If outer rel allows parallelism, do same for partial paths. */
diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c
index 7271f044ec..eefba449d6 100644
--- a/src/backend/optimizer/path/joinrels.c
+++ b/src/backend/optimizer/path/joinrels.c
@@ -925,6 +925,13 @@ populate_joinrel_with_paths(PlannerInfo *root, RelOptInfo *rel1,
/* Apply partitionwise join technique, if possible. */
try_partitionwise_join(root, rel1, rel2, joinrel, sjinfo, restrictlist);
+ /*
+ * Determine which of the unique keys from input relations are applicable
+ * for the join result.
+ *
+ * XXX We do this after trying the partitionwise join, because that may allow
+ * using additional unique keys.
+ */
populate_joinrel_uniquekeys(root, joinrel, rel1, rel2, restrictlist, sjinfo->jointype);
}
diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c
index 77ed2b2eff..114e8334f5 100644
--- a/src/backend/optimizer/path/uniquekeys.c
+++ b/src/backend/optimizer/path/uniquekeys.c
@@ -36,7 +36,7 @@ typedef struct UniqueKeyContextData
bool useful;
} *UniqueKeyContext;
-static List *initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel);
+static List *initialize_uniquecontext_for_joinrel(RelOptInfo *inputrel);
static bool innerrel_keeps_unique(PlannerInfo *root,
RelOptInfo *outerrel,
RelOptInfo *innerrel,
@@ -80,8 +80,20 @@ static void add_uniquekey_from_sortgroups(PlannerInfo *root,
/*
* populate_baserel_uniquekeys
- * Populate 'baserel' uniquekeys list by looking at the rel's unique index
- * and baserestrictinfo
+ * Build list of unique keys for the base relation.
+ *
+ * Inspects unique indexes defined on the relation and determines what
+ * unique keys are valid. Partial indexes are considered too, if the
+ * predicate is valid.
+ *
+ * This also inspects baserestrictinfo, because we need to determine
+ * which opclass families are interesting when inspecting indexes. If we
+ * have a unique index and distinct clause with a mismatching opclasses,
+ * we should not use that.
+ *
+ * XXX Why does this look at baserestrictinfo?
+ *
+ * XXX What about collations?
*/
void
populate_baserel_uniquekeys(PlannerInfo *root,
@@ -99,22 +111,48 @@ populate_baserel_uniquekeys(PlannerInfo *root,
Assert(baserel->rtekind == RTE_RELATION);
+ if (!indexlist)
+ return;
+
+ /*
+ * Determine which unique indexes to use to build the unique keys.
+ * We have to skip partial with predicates not matched by the query,
+ * and unique indexes that are not immediately enforced.
+ *
+ * XXX Do we actually skip indexes that are not immediate?
+ * XXX What about hypothetical indexes?
+ */
foreach(lc, indexlist)
{
IndexOptInfo *ind = (IndexOptInfo *) lfirst(lc);
+
if (!ind->unique || !ind->immediate ||
(ind->indpred != NIL && !ind->predOK))
continue;
+
matched_uniq_indexes = lappend(matched_uniq_indexes, ind);
}
+ /* If there are not applicable unique indexes, we're done. */
if (matched_uniq_indexes == NIL)
return;
- /* Check which attrs is used in baserel->reltarget */
- pull_varattnos((Node *)baserel->reltarget->exprs, baserel->relid, &used_attrs);
+ /*
+ * Determine which attrs are referenced in baserel->reltarget. To use the
+ * unique key info, we need all the columns - a unique index on (a,b) may
+ * not be unique on (a). If a column is missing in reltarget, the nodes
+ * above can't possibly use it, and we can just ignore any matching index.
+ */
+ pull_varattnos((Node *) baserel->reltarget->exprs, baserel->relid, &used_attrs);
- /* Check which attrno is used at a mergeable const filter */
+ /*
+ * Check which attrno is used at a mergeable const filter
+ *
+ * XXX This is not lookint att attrno at all, maybe obsolete comment?
+ *
+ * Seems the primary purpose of this is determining which opclass
+ * families to use when matching unique indexes in the next loop?
+ */
foreach(lc, baserel->baserestrictinfo)
{
RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc);
@@ -122,6 +160,10 @@ populate_baserel_uniquekeys(PlannerInfo *root,
if (rinfo->mergeopfamilies == NIL)
continue;
+ /*
+ * XXX What if bms_is_empty is true for both left_relids/right_relids?
+ * Or what if it's false in both cases?
+ */
if (bms_is_empty(rinfo->left_relids))
{
const_exprs = lappend(const_exprs, get_rightop(rinfo->clause));
@@ -136,40 +178,69 @@ populate_baserel_uniquekeys(PlannerInfo *root,
expr_opfamilies = lappend(expr_opfamilies, rinfo->mergeopfamilies);
}
+ /*
+ * Now try to match unique indexes to attributes in reltarget, and to
+ * merge operator families. The index may be on the right attributes,
+ * but if it's not matching the opfamily it's useless.
+ *
+ * XXX Can we have multiple baserestrictinfo for the same attribute,
+ * with different opfamilies? Probably not.
+ */
foreach(lc, matched_uniq_indexes)
{
- bool multi_nullvals, useful;
- List *exprs = get_exprs_from_uniqueindex(lfirst_node(IndexOptInfo, lc),
- const_exprs,
- expr_opfamilies,
- used_attrs,
- &useful,
- &multi_nullvals);
- if (useful)
+ bool multi_nullvals,
+ useful;
+
+ IndexOptInfo *index_info = (IndexOptInfo *) lfirst_node(IndexOptInfo, lc);
+
+ List *exprs = get_exprs_from_uniqueindex(index_info,
+ const_exprs,
+ expr_opfamilies,
+ used_attrs,
+ &useful,
+ &multi_nullvals);
+
+ if (!useful)
+ continue;
+
+ /*
+ * All the columns in Unique Index matched with a restrictinfo, so
+ * that we know there's just a one row in the result. If we find
+ * such index, we're done - we discard all other unique keys and
+ * keep just this special one. In principle, this is a stronger
+ * guarantee, because all subsets of one row are still unique.
+ *
+ * XXX Is it correct to just return? Doesn't that prevent some
+ * optimizations that might be possible with the other keys?
+ */
+ if (exprs == NIL)
{
- if (exprs == NIL)
- {
- /* All the columns in Unique Index matched with a restrictinfo */
- add_uniquekey_for_onerow(baserel);
- return;
- }
- baserel->uniquekeys = lappend(baserel->uniquekeys,
- makeUniqueKey(exprs, multi_nullvals));
+ /* discards all previous uniquekeys */
+ add_uniquekey_for_onerow(baserel);
+ return;
}
+
+ baserel->uniquekeys = lappend(baserel->uniquekeys,
+ makeUniqueKey(exprs, multi_nullvals));
}
}
/*
* populate_partitionedrel_uniquekeys
- * The UniqueKey on partitionrel comes from 2 cases:
- * 1). Only one partition is involved in this query, the unique key can be
- * copied to parent rel from childrel.
- * 2). There are some unique index which includes partition key and exists
- * in all the related partitions.
- * We never mind rule 2 if we hit rule 1.
+ * Determine unique keys for a partitioned relation.
+ *
+ * Inspects unique keys for all partitions and derives unique keys that
+ * are valid for the whole partitioned table. There are two basic cases:
+ *
+ * 1) There's only one remaining partition (thanks to pruning all other
+ * partitions). In this case all the unique keys from the partition are
+ * trivially valid for the partitioned table.
+ *
+ * 2) All the partitions have the same unique index (on the same set of
+ * columns), and the index includes the partition key. This ensures the
+ * combination of values is unique for the whole partitioned table.
*/
-
void
populate_partitionedrel_uniquekeys(PlannerInfo *root,
RelOptInfo *rel,
@@ -180,110 +251,181 @@ populate_partitionedrel_uniquekeys(PlannerInfo *root,
RelOptInfo *childrel;
bool is_first = true;
+ /* XXX What about append rels? At least for the one-child case? */
Assert(IS_PARTITIONED_REL(rel));
+ /* if there are no child relations, we're done. */
if (childrels == NIL)
return;
/*
- * If there is only one partition used in this query, the UniqueKey in childrel is
- * still valid in parent level, but we need convert the format from child expr to
- * parent expr.
+ * If there is only one partition used in this query, the UniqueKey for
+ * a child relation is still valid for the parent level. We need to
+ * convert the format from child expr to parent expr.
*/
if (list_length(childrels) == 1)
{
- /* Check for Rule 1 */
RelOptInfo *childrel = linitial_node(RelOptInfo, childrels);
ListCell *lc;
+
Assert(childrel->reloptkind == RELOPT_OTHER_MEMBER_REL);
+
+ /* If the partition has a single row, so does the parent. */
if (relation_is_onerow(childrel))
{
add_uniquekey_for_onerow(rel);
return;
}
+ /*
+ * Inspect the unique keys one by one, try reusing them for the
+ * parent relation.
+ *
+ * FIXME This needs more work to handle expressions and not just
+ * simple Vars.
+ */
foreach(lc, childrel->uniquekeys)
{
+ ListCell *lc2;
+ List *parent_exprs = NIL;
+ bool can_reuse = true;
+
UniqueKey *ukey = lfirst_node(UniqueKey, lc);
AppendRelInfo *appinfo = find_appinfo_by_child(root, childrel->relid);
- List *parent_exprs = NIL;
- bool can_reuse = true;
- ListCell *lc2;
+
+ /*
+ * XXX Not sure what exactly we do here. Surely we deal with
+ * expressions at child/parent level elsewhere? Can't we just
+ * copy the code from there?
+ */
foreach(lc2, ukey->exprs)
{
- Var *var = (Var *)lfirst(lc2);
+ Var *var = (Var *) lfirst(lc2);
+
/*
- * If the expr comes from a expression, it is hard to build the expression
- * in parent so ignore that case for now.
+ * XXX For now this only supports simple Var expressions,
+ * so if there's a more complex expression we'll not copy
+ * the unique key to the parent.
*/
if(!IsA(var, Var))
{
can_reuse = false;
break;
}
+
/* Convert it to parent var */
- parent_exprs = lappend(parent_exprs, find_parent_var(appinfo, var));
+ parent_exprs = lappend(parent_exprs,
+ find_parent_var(appinfo, var));
}
- if (can_reuse)
- rel->uniquekeys = lappend(rel->uniquekeys,
- makeUniqueKey(parent_exprs,
- ukey->multi_nullvals));
+
+ /* ignore unique keys with complex expressions */
+ if (!can_reuse)
+ continue;
+
+ rel->uniquekeys = lappend(rel->uniquekeys,
+ makeUniqueKey(parent_exprs,
+ ukey->multi_nullvals));
}
+
+ return;
}
- else
+
+ /*
+ * A parent with multiple child relations. We only care about indexes that
+ * are in all child relations, so we loop through indexes on the first one
+ * and check that they exist in the other child relations too.
+ */
+
+ childrel = linitial_node(RelOptInfo, childrels);
+ foreach(lc, childrel->indexlist)
{
- /* Check for rule 2 */
- childrel = linitial_node(RelOptInfo, childrels);
- foreach(lc, childrel->indexlist)
- {
- IndexOptInfo *ind = lfirst(lc);
- IndexOptInfo *modified_index;
- if (!ind->unique || !ind->immediate ||
- (ind->indpred != NIL && !ind->predOK))
- continue;
+ IndexOptInfo *ind = lfirst(lc);
+ IndexOptInfo *modified_index;
- /*
- * During simple_copy_indexinfo_to_parent, we need to convert var from
- * child var to parent var, index on expression is too complex to handle.
- * so ignore it for now.
- */
- if (ind->indexprs != NIL)
- continue;
+ /*
+ * Ignore indexes that are not unique, immediately enforced. Partial
+ * indexes with mismatched predicate are useless too.
+ */
+ if (!ind->unique || !ind->immediate ||
+ (ind->indpred != NIL && !ind->predOK))
+ continue;
- modified_index = simple_copy_indexinfo_to_parent(root, rel, ind);
- /*
- * If the unique index doesn't contain partkey, then it is unique
- * on this partition only, so it is useless for us.
- */
- if (!index_constains_partkey(rel, modified_index))
- continue;
+ /*
+ * During simple_copy_indexinfo_to_parent, we need to convert var from
+ * child var to parent var, index on expression is too complex to handle.
+ * so ignore it for now.
+ *
+ * FIXME We should support indexes on expressions.
+ */
+ if (ind->indexprs != NIL)
+ continue;
- global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index);
- }
+ /*
+ * Adopt the index definition for the parent.
+ *
+ * XXX This seems rather weird. We're constructing "artificial" index
+ * for the partitioned table (kinda like a global index). Can't we
+ * just have some simpler struct representing it?
+ */
+ modified_index = simple_copy_indexinfo_to_parent(root, rel, ind);
+
+ /*
+ * If the unique index doesn't contain partkey, then it is unique
+ * on this partition only, so it is useless for us.
+ *
+ * XXX Can't we do this check before simple_copy_indexinfo_to_parent?
+ */
+ if (!index_constains_partkey(rel, modified_index))
+ continue;
- if (global_uniq_indexlist != NIL)
+ global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index);
+ }
+
+ /* if there are no applicable unique indexes, we're done */
+ if (!global_uniq_indexlist)
+ return;
+
+ /*
+ * We iterate over the child relations first, and inspect the unique
+ * indexes for each hild, because this way we can stop early if we
+ * happen to eliminate all the unique indexes.
+ */
+ foreach(lc, childrels)
+ {
+ RelOptInfo *child = lfirst(lc);
+
+ /* skip the first index, which is where we got the list from */
+ if (is_first)
{
- foreach(lc, childrels)
- {
- RelOptInfo *child = lfirst(lc);
- if (is_first)
- {
- is_first = false;
- continue;
- }
- adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist);
- }
- /* Now we have a list of unique index which are exactly same on all childrels,
- * Set the UniqueKey just like it is non-partition table
- */
- populate_baserel_uniquekeys(root, rel, global_uniq_indexlist);
+ is_first = false;
+ continue;
}
+
+ /* match the unique keys to indexes on this child */
+ adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist);
+
+ /*
+ * If we have eliminated all unique indexes, no point in looking at
+ * the remaining child relations.
+ */
+ if (!global_uniq_indexlist)
+ break;
}
+
+ /* Now we have a list of unique index which are exactly same on all child
+ * relations. Set the UniqueKey just like it is non-partition table.
+ */
+ populate_baserel_uniquekeys(root, rel, global_uniq_indexlist);
}
/*
* populate_distinctrel_uniquekeys
+ * Update unique keys for relation produced by DISTINCT.
+ *
+ * We can keep all unique keys from the input relations, because DISTINCT
+ * can only remove rows - it can't duplicate them. Also, the DISTINCT clause
+ * itself is a unique key, so add that.
*/
void
populate_distinctrel_uniquekeys(PlannerInfo *root,
@@ -292,11 +434,13 @@ populate_distinctrel_uniquekeys(PlannerInfo *root,
{
/* The unique key before the distinct is still valid. */
distinctrel->uniquekeys = list_copy(inputrel->uniquekeys);
+
add_uniquekey_from_sortgroups(root, distinctrel, root->parse->distinctClause);
}
/*
* populate_grouprel_uniquekeys
+ *
*/
void
populate_grouprel_uniquekeys(PlannerInfo *root,
@@ -305,54 +449,76 @@ populate_grouprel_uniquekeys(PlannerInfo *root,
{
Query *parse = root->parse;
- bool input_ukey_added = false;
ListCell *lc;
+ /*
+ * XXX Is this actually valid, before checking fro grouping sets?
+ * The grouping sets may produce duplicate row even with just a single
+ * input row, I think.
+ */
if (relation_is_onerow(inputrel))
{
add_uniquekey_for_onerow(grouprel);
return;
}
+
+ /*
+ * Bail out if there are grouping sets.
+ *
+ * XXX Could we maybe inspect the grouping sets and determine if this
+ * generates distinct combinations? In some cases that's clearly not
+ * the case (rollup, cube), but for some simple cases it might.
+ */
if (parse->groupingSets)
return;
- /* A Normal group by without grouping set. */
- if (parse->groupClause)
+ /* It has aggregation but without a group by, so only one row returned */
+ if (!parse->groupClause)
+ add_uniquekey_for_onerow(grouprel);
+
+ /*
+ * A regular group by, without grouping sets.
+ *
+ * Obviously, the whole group clause determines a unique key. But if
+ * there are smaller unique keys on the input rel, we prefer those
+ * because those are more flexible. If (a,b) is unique, (a,b,c) is
+ * unique too. Only when there are no such smaller unique keys, we
+ * add the unique key derived from the group clause.
+ */
+ foreach(lc, inputrel->uniquekeys)
{
+ UniqueKey *ukey = lfirst_node(UniqueKey, lc);
+
/*
- * Current even the groupby clause is Unique already, but if query has aggref
- * We have to create grouprel still. To keep the UnqiueKey short, we will check
- * the UniqueKey of input_rel still valid, if so we reuse it.
+ * Ignore unique keys on the input that are not subset of the
+ * group clause. We can't use incomplete unique keys.
*/
- foreach(lc, inputrel->uniquekeys)
- {
- UniqueKey *ukey = lfirst_node(UniqueKey, lc);
- if (list_is_subset(ukey->exprs, grouprel->reltarget->exprs))
- {
- grouprel->uniquekeys = lappend(grouprel->uniquekeys,
- ukey);
- input_ukey_added = true;
- }
- }
- if (!input_ukey_added)
- /*
- * group by clause must be a super-set of grouprel->reltarget->exprs except the
- * aggregation expr, so if such exprs is unique already, no bother to generate
- * new uniquekey for group by exprs.
- */
- add_uniquekey_from_sortgroups(root,
- grouprel,
- root->parse->groupClause);
+ if (!list_is_subset(ukey->exprs, grouprel->reltarget->exprs))
+ continue;
+
+ grouprel->uniquekeys = lappend(grouprel->uniquekeys, ukey);
}
- else
- /* It has aggregation but without a group by, so only one row returned */
- add_uniquekey_for_onerow(grouprel);
+
+ /*
+ * Group clause must be a super-set of of grouprel->reltarget->exprs,
+ * except for the aggregation expressions. So if we found a smaller
+ * unique key on the input relation, don't bother adding a unique key
+ * for the group clause.
+ */
+ if (!grouprel->uniquekeys)
+ add_uniquekey_from_sortgroups(root,
+ grouprel,
+ root->parse->groupClause);
}
/*
* simple_copy_uniquekeys
- * Using a function for the one-line code makes us easy to check where we simply
- * copied the uniquekey.
+ * Copy yhe unique keys between relations.
+ *
+ * Using a function for the one-line code makes us easy to check where we
+ * simply copied the uniquekey.
+ *
+ * XXX Seems like an overkill, not sure what's the purpose?
*/
void
simple_copy_uniquekeys(RelOptInfo *oldrel,
@@ -362,24 +528,27 @@ simple_copy_uniquekeys(RelOptInfo *oldrel,
}
/*
- * populate_unionrel_uniquekeys
+ * populate_unionrel_uniquekeys
+ * Determine unique keys for UNION relation.
+ *
+ * XXX Does this need to care about UNION vs. UNION ALL? At least in the
+ * one-row code path?
*/
void
populate_unionrel_uniquekeys(PlannerInfo *root,
- RelOptInfo *unionrel)
+ RelOptInfo *unionrel)
{
- ListCell *lc;
- List *exprs = NIL;
+ ListCell *lc;
+ List *exprs = NIL;
Assert(unionrel->uniquekeys == NIL);
+ /* XXX Why are we copying the expressions? */
foreach(lc, unionrel->reltarget->exprs)
- {
exprs = lappend(exprs, lfirst(lc));
- }
+ /* SQL: select union select; is valid, we need to handle it here. */
if (exprs == NIL)
- /* SQL: select union select; is valid, we need to handle it here. */
add_uniquekey_for_onerow(unionrel);
else
unionrel->uniquekeys = lappend(unionrel->uniquekeys,
@@ -389,6 +558,7 @@ populate_unionrel_uniquekeys(PlannerInfo *root,
/*
* populate_joinrel_uniquekeys
+ * Determine unique keys for a join relation.
*
* populate uniquekeys for joinrel. We will check each relation to see if its
* UniqueKey is still valid via innerrel_keeps_unique, if so, we add it to
@@ -404,70 +574,99 @@ populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel,
RelOptInfo *outerrel, RelOptInfo *innerrel,
List *restrictlist, JoinType jointype)
{
- ListCell *lc, *lc2;
- List *clause_list = NIL;
- List *outerrel_ukey_ctx;
- List *innerrel_ukey_ctx;
- bool inner_onerow, outer_onerow;
- bool mergejoin_allowed;
-
- /* Care about the outerrel relation only for SEMI/ANTI join */
+ ListCell *lc,
+ *lc2;
+ List *clause_list = NIL;
+ List *outerrel_ukey_ctx;
+ List *innerrel_ukey_ctx;
+ bool inner_onerow,
+ outer_onerow;
+ bool mergejoin_allowed;
+
+ /* For SEMI/ANTI join, we care only about the outerrel unique keys. */
if (jointype == JOIN_SEMI || jointype == JOIN_ANTI)
{
foreach(lc, outerrel->uniquekeys)
{
UniqueKey *uniquekey = lfirst_node(UniqueKey, lc);
+
+ /* Keep the unique key if it's included in the joinrel. */
if (list_is_subset(uniquekey->exprs, joinrel->reltarget->exprs))
joinrel->uniquekeys = lappend(joinrel->uniquekeys, uniquekey);
}
+
return;
}
+ /* XXX What about JOIN_RIGHT? */
Assert(jointype == JOIN_LEFT || jointype == JOIN_FULL || jointype == JOIN_INNER);
- /* Fast path */
+ /*
+ * For regular joins, we need to combine unique keys from both sides
+ * of the join, to get a new unique key for the join relation. So if
+ * either side does not have a unique key, bail out.
+ */
if (innerrel->uniquekeys == NIL || outerrel->uniquekeys == NIL)
return;
+ /* XXX maybe move to the if blocks? Not needed outside. */
inner_onerow = relation_is_onerow(innerrel);
outer_onerow = relation_is_onerow(outerrel);
- outerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(outerrel);
- innerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(innerrel);
+ outerrel_ukey_ctx = initialize_uniquecontext_for_joinrel(outerrel);
+ innerrel_ukey_ctx = initialize_uniquecontext_for_joinrel(innerrel);
- clause_list = select_mergejoin_clauses(root, joinrel, outerrel, innerrel,
+ clause_list = select_mergejoin_clauses(root,
+ joinrel, outerrel, innerrel,
restrictlist, jointype,
&mergejoin_allowed);
- if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true /* reverse */))
+ /*
+ * XXX Seems a bit weird that it's called innerrel_keeps_unique but we
+ * seem to use it in both directions. Or what's the "reverse" for? The
+ * "reverse" name is not particularly descriptive.
+ */
+ if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true))
{
- bool outer_impact = jointype == JOIN_FULL;
+ bool outer_impact = (jointype == JOIN_FULL);
+
+ /* Inspect unique keys on the outer relation. */
foreach(lc, outerrel_ukey_ctx)
{
UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc);
+ /*
+ * If the output of the join does not include all the parts of the
+ * unique key, it's useless, so mark it accordingly and ignore it.
+ */
if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs))
{
ctx->useful = false;
continue;
}
- /* Outer relation has one row, and the unique key is not duplicated after join,
- * the joinrel will still has one row unless the jointype == JOIN_FULL.
+ /*
+ * When the outer relation has one row, and the unique key is not
+ * duplicated after join, so the joinrel will still have just one
+ * row unless the jointype == JOIN_FULL. In that case we're done,
+ * it's the strictest unique key possible.
+ *
+ * If it's one-row with a JOIN_FULL, it might produce multiple
+ * rows with NULLs, so set multi_nullvals. We also need to set
+ * the exprs correctly since it can't be NIL any more.
+ *
+ * For other cases (not one-row relation), we just reuse the
+ * unique key, but we may need to tweak the multi_nullvals.
*/
if (outer_onerow && !outer_impact)
{
add_uniquekey_for_onerow(joinrel);
return;
}
- else if (outer_onerow)
+ else if (outer_onerow) /* one-row and FULL join */
{
- /*
- * The onerow outerrel becomes multi rows and multi_nullvals
- * will be changed to true. We also need to set the exprs correctly since it
- * can't be NIL any more.
- */
ListCell *lc2;
+
foreach(lc2, get_exprs_from_uniquekey(root, joinrel, outerrel, NULL))
{
joinrel->uniquekeys = lappend(joinrel->uniquekeys,
@@ -485,18 +684,38 @@ populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel,
joinrel->uniquekeys = lappend(joinrel->uniquekeys,
ctx->uniquekey);
}
+
+ /*
+ * Mark the unique key as added, so that we can ignore it later
+ * when combining unique keys from both sides of the join.
+ */
ctx->added_to_joinrel = true;
}
}
+ /*
+ * XXX Seems this actually checks if "outerrel keeps unique" so the name
+ * is misleading. Of maybe it's the previous block, not sure.
+ *
+ * XXX So why does this consider JOIN_FULL and JOIN_LEFT, while the previous
+ * block only cares about JOIN_FULL?
+ *
+ * XXX This is almost exact copy of the previous block, so maybe make it
+ * a separate function and just call it twice?
+ */
if (innerrel_keeps_unique(root, outerrel, innerrel, clause_list, false))
{
- bool outer_impact = jointype == JOIN_FULL || jointype == JOIN_LEFT;;
+ bool outer_impact = (jointype == JOIN_FULL || jointype == JOIN_LEFT);
+ /* Inspect unique keys on the inner relation. */
foreach(lc, innerrel_ukey_ctx)
{
UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc);
+ /*
+ * If the output of the join does not include all the parts of the
+ * unique key, it's useless, so mark it accordingly and ignore it.
+ */
if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs))
{
ctx->useful = false;
@@ -529,29 +748,52 @@ populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel,
ctx->uniquekey);
}
+
+ /*
+ * Mark the unique key as added, so that we can ignore it later
+ * when combining unique keys from both sides of the join.
+ */
ctx->added_to_joinrel = true;
}
}
/*
- * The combination of the UniqueKey from both sides is unique as well regardless
- * of join type, but no bother to add it if its subset has been added to joinrel
- * already or it is not useful for the joinrel.
+ * XXX What if either of the previous two conditions did not match? In
+ * that case we haven't updated the useful flag, and maybe the unique
+ * key is not useful, but we don't know, right? So we should not be
+ * using it in the next loop. Or maybe we should evaluate the flag
+ * before the loops.
+ */
+
+ /*
+ * The combination of the UniqueKey from both sides is unique as well,
+ * regardless of the join type. But don't bother to add it if its
+ * subset has been added to joinrel already or when it's not useful for
+ * the joinrel.
+ *
+ * XXX Maybe we should have a flag that both sides have useful keys?
+ * Or maybe the loops are short/cheap?
*/
foreach(lc, outerrel_ukey_ctx)
{
UniqueKeyContext ctx1 = (UniqueKeyContext) lfirst(lc);
+
+ /* when not useful or already added to the joinrel, skip it */
if (ctx1->added_to_joinrel || !ctx1->useful)
continue;
+
foreach(lc2, innerrel_ukey_ctx)
{
UniqueKeyContext ctx2 = (UniqueKeyContext) lfirst(lc2);
+
+ /* when not useful or already added to the joinrel, skip it */
if (ctx2->added_to_joinrel || !ctx2->useful)
continue;
+
+ /* If we add a onerow UniqueKey, we don't need another key. */
if (add_combined_uniquekey(root, joinrel, outerrel, innerrel,
ctx1->uniquekey, ctx2->uniquekey,
jointype))
- /* If we set a onerow UniqueKey to joinrel, we don't need other. */
return;
}
}
@@ -560,8 +802,9 @@ populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel,
/*
* convert_subquery_uniquekeys
+ * Covert the UniqueKey in subquery to outer relation.
*
- * Covert the UniqueKey in subquery to outer relation.
+ * XXX Explain what exactly does the conversion do?
*/
void convert_subquery_uniquekeys(PlannerInfo *root,
RelOptInfo *currel,
@@ -618,12 +861,14 @@ void convert_subquery_uniquekeys(PlannerInfo *root,
/*
* innerrel_keeps_unique
+ * Check if Unique key on the innerrel is valid after join.
*
- * Check if Unique key of the innerrel is valid after join. innerrel's UniqueKey
- * will be still valid if innerrel's any-column mergeop outrerel's uniquekey
- * exists in clause_list.
+ * innerrel's UniqueKey will be still valid if innerrel's any-column mergeop
+ * outrerel's uniquekey exists in clause_list
*
* Note: the clause_list must be a list of mergeable restrictinfo already.
+ *
+ * XXX Misleading name? We seem to use it for "outerrel_keeps_unique" too.
*/
static bool
innerrel_keeps_unique(PlannerInfo *root,
@@ -634,26 +879,32 @@ innerrel_keeps_unique(PlannerInfo *root,
{
ListCell *lc, *lc2, *lc3;
+ /* XXX probably not needed, duplicate with the check in the caller
+ * (populate_joinrel_uniquekeys). But it's cheap. */
if (outerrel->uniquekeys == NIL || innerrel->uniquekeys == NIL)
return false;
/* Check if there is outerrel's uniquekey in mergeable clause. */
foreach(lc, outerrel->uniquekeys)
{
- List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs;
- bool clauselist_matchs_all_exprs = true;
+ List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs;
+ bool clauselist_matchs_all_exprs = true;
+
foreach(lc2, outer_uq_exprs)
{
Node *outer_uq_expr = lfirst(lc2);
bool find_uq_expr_in_clauselist = false;
+
foreach(lc3, clause_list)
{
RestrictInfo *rinfo = lfirst_node(RestrictInfo, lc3);
Node *outer_expr;
+
if (reverse)
outer_expr = rinfo->outer_is_left ? get_rightop(rinfo->clause) : get_leftop(rinfo->clause);
else
outer_expr = rinfo->outer_is_left ? get_leftop(rinfo->clause) : get_rightop(rinfo->clause);
+
if (equal(outer_expr, outer_uq_expr))
{
find_uq_expr_in_clauselist = true;
@@ -677,22 +928,37 @@ innerrel_keeps_unique(PlannerInfo *root,
/*
* relation_is_onerow
- * Check if it is a one-row relation by checking UniqueKey.
+ * Check if it is a one-row relation by checking UniqueKey.
+ *
+ * The one-row is a special case - there has to be just a single unique key,
+ * with no expressions.
*/
bool
relation_is_onerow(RelOptInfo *rel)
{
UniqueKey *ukey;
- if (rel->uniquekeys == NIL)
+
+ /* there has to be exactly one unique key */
+ if (list_length(rel->uniquekeys) != 1)
return false;
+
ukey = linitial_node(UniqueKey, rel->uniquekeys);
- return ukey->exprs == NIL && list_length(rel->uniquekeys) == 1;
+
+ /* the unique key must have no expressions */
+ return (ukey->exprs == NIL);
}
/*
* relation_has_uniquekeys_for
- * Returns true if we have proofs that 'rel' cannot return multiple rows with
- * the same values in each of 'exprs'. Otherwise returns false.
+ * Determines if the relation has unique key for a list of expressions.
+ *
+ * Returns true iff we can prove that the relation cannot return multiple rows
+ * with the same values in the provided expression.
+ *
+ * allow_multinulls determines whether we allow multiple NULL values or not.
+ *
+ * The special "one-row" unique key is considered incompatible with all
+ * possible expressions.
*/
bool
relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel,
@@ -710,20 +976,39 @@ relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel,
foreach(lc, rel->uniquekeys)
{
UniqueKey *ukey = lfirst_node(UniqueKey, lc);
+
if (ukey->multi_nullvals && !allow_multinulls)
continue;
+
if (list_is_subset(ukey->exprs, exprs))
return true;
}
+
return false;
}
/*
* get_exprs_from_uniqueindex
+ * Return a list of expressions from a unique index.
+ *
+ * Provided with a list of expressions and opclass families, we try to match
+ * it to the index. If useful, we produce a list of index expressions (subset
+ * of the list we provided).
+ *
+ * We simply walk through the index expressions, and for each expression we
+ * check three things:
*
- * Return a list of exprs which is unique. set useful to false if this
- * unique index is not useful for us.
+ * 1) If there's a matching (expr = Const) clause, we can simply ignore the
+ * expressions. Unique index on (a,b,c) guarantees uniqueness on (a,b) when
+ * there's condition (c=1).
+ *
+ * 2) Check that the index expression is present in the relation we're
+ * dealing with. If not, the unique key would be useless anyway, and the
+ * index can't produce unique key.
+ *
+ * XXX Shouldn't it be enough to return NULL when the index is not useful?
+ * The extra flag seems a bit unnecessary.
*/
static List *
get_exprs_from_uniqueindex(IndexOptInfo *unique_index,
@@ -743,18 +1028,19 @@ get_exprs_from_uniqueindex(IndexOptInfo *unique_index,
indexpr_item = list_head(unique_index->indexprs);
for(c = 0; c < unique_index->ncolumns; c++)
{
- int attr = unique_index->indexkeys[c];
- Expr *expr;
- bool matched_const = false;
- ListCell *lc1, *lc2;
+ int attr = unique_index->indexkeys[c];
+ Expr *expr;
+ bool matched_const = false;
+ ListCell *lc1, *lc2;
- if(attr > 0)
+ if (attr > 0)
{
+ /* regular attribute, just use the expression from index tlist */
expr = list_nth_node(TargetEntry, unique_index->indextlist, c)->expr;
}
else if (attr == 0)
{
- /* Expression index */
+ /* expression from the index */
expr = lfirst(indexpr_item);
indexpr_item = lnext(unique_index->indexprs, indexpr_item);
}
@@ -764,29 +1050,43 @@ get_exprs_from_uniqueindex(IndexOptInfo *unique_index,
Assert(false);
}
+ /* should have a valid expression now */
+ Assert(expr);
+
/*
- * Check index_col = Const case with regarding to opfamily checking
- * If we can remove the index_col from the final UniqueKey->exprs.
+ * Check if there's (index_col = Const) condition, and that it's using
+ * a compatible opfamily. If yes, we can remove the index_col from the
+ * final UniqueKey->exprs, because the value is constant (so removing
+ * it can't introduce duplicities).
*/
forboth(lc1, const_exprs, lc2, const_expr_opfamilies)
{
- if (list_member_oid((List *)lfirst(lc2), unique_index->opfamily[c])
- && match_index_to_operand((Node *) lfirst(lc1), c, unique_index))
+ List *opfamilies = (List *) lfirst(lc2);
+ Node *cexpr = (Node *) lfirst(lc1);
+
+ if (list_member_oid(opfamilies, unique_index->opfamily[c]) &&
+ match_index_to_operand(cexpr, c, unique_index))
{
matched_const = true;
break;
}
}
+ /* it's constant, so ignore the expression */
if (matched_const)
continue;
- /* Check if the indexed expr is used in rel */
+ /*
+ * Check if the indexed expr is used in rel. We do this after the
+ * (col = Const) check, because nn expression may be in a a restrict
+ * clause and not in the reltarget. So we don't want to rule out an
+ * index unnecessarily.
+ */
if (attr > 0)
{
/*
- * Normal Indexed column, if the col is not used, then the index is useless
- * for uniquekey.
+ * Normal indexed column, if the col is not used, then the index
+ * is useless for uniquekey.
*/
attr -= FirstLowInvalidHeapAttributeNumber;
@@ -806,67 +1106,85 @@ get_exprs_from_uniqueindex(IndexOptInfo *unique_index,
/* check not null property. */
if (attr == 0)
{
- /* We never know if a expression yields null or not */
+ /* We never know if an expression yields null or not */
*multi_nullvals = true;
}
- else if (!bms_is_member(attr, unique_index->rel->notnullattrs)
- && !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber,
- unique_index->rel->notnullattrs))
+ else if (!bms_is_member(attr, unique_index->rel->notnullattrs) &&
+ !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber,
+ unique_index->rel->notnullattrs))
{
*multi_nullvals = true;
}
exprs = lappend(exprs, expr);
}
+
return exprs;
}
/*
* add_uniquekey_for_onerow
- * If we are sure that the relation only returns one row, then all the columns
- * are unique. However we don't need to create UniqueKey for every column, we
- * just set exprs = NIL and overwrites all the other UniqueKey on this RelOptInfo
- * since this one has strongest semantics.
+ * Create a special unique key signifying that the rel has one row.
+ *
+ * If we are sure that the relation only returns one row (it might return
+ * no rows, but we still consider that unique), then all the columns are
+ * trivially unique.
+ *
+ * However we don't need to create UniqueKey with every column, we just
+ * set exprs = NIL, because that's easier to identify. We don't want to
+ * add unnecessary unique keys (such that we already have a unique key
+ * for a subset of the expressions), and with (exprs == NIL) we can just
+ * assume we have one unique key for each column in the rel.
+ *
+ * We discard all other unique keys, since it has the strongest semantics.
*/
void
add_uniquekey_for_onerow(RelOptInfo *rel)
{
/*
- * We overwrite the previous UniqueKey on purpose since this one has the
- * strongest semantic.
+ * We overwrite the previous UniqueKey on purpose since this one has
+ * the strongest semantic (all other unique keys are implied by it).
*/
rel->uniquekeys = list_make1(makeUniqueKey(NIL, false));
}
/*
- * initililze_uniquecontext_for_joinrel
- * Return a List of UniqueKeyContext for an inputrel
+ * initialize_uniquecontext_for_joinrel
+ * Return a List of UniqueKeyContext for an inputrel.
*/
static List *
-initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel)
+initialize_uniquecontext_for_joinrel(RelOptInfo *inputrel)
{
- List *res = NIL;
- ListCell *lc;
- foreach(lc, inputrel->uniquekeys)
+ List *res = NIL;
+ ListCell *lc;
+
+ foreach(lc, inputrel->uniquekeys)
{
UniqueKeyContext context;
+
context = palloc(sizeof(struct UniqueKeyContextData));
context->uniquekey = lfirst_node(UniqueKey, lc);
context->added_to_joinrel = false;
context->useful = true;
+
res = lappend(res, context);
}
+
return res;
}
-
/*
* get_exprs_from_uniquekey
- * Unify the way of get List of exprs from a one-row UniqueKey or
- * normal UniqueKey. for the onerow case, every expr in rel1 is a valid
- * UniqueKey. Return a List of exprs.
+ * Extract expressions that are part of a unique key.
+ *
+ * The meaning of the result is a bit different in regular and one-row cases.
+ * For the regular case, the list of expressions form a single unique key,
+ * i.e. the combination of values is unique.
+ *
+ * For the one-row case, each individual expression is known to be unique
+ * (simply because in a single row everything is unique).
*
* rel1: The relation which you want to get the exprs.
* ukey: The UniqueKey you want to get the exprs.
@@ -875,27 +1193,29 @@ static List *
get_exprs_from_uniquekey(PlannerInfo *root, RelOptInfo *joinrel,
RelOptInfo *rel1, UniqueKey *ukey)
{
- ListCell *lc;
- bool onerow = rel1 != NULL && relation_is_onerow(rel1);
+ ListCell *lc;
+ List *res = NIL;
+ bool onerow = (rel1 != NULL) && relation_is_onerow(rel1);
- List *res = NIL;
+ /* We require at least one of those to be true. */
Assert(onerow || ukey);
- if (onerow)
- {
- /* Only cares about the exprs still exist in joinrel */
- foreach(lc, joinrel->reltarget->exprs)
- {
- Bitmapset *relids = pull_varnos(root, lfirst(lc));
- if (bms_is_subset(relids, rel1->relids))
- {
- res = lappend(res, list_make1(lfirst(lc)));
- }
- }
- }
- else
+
+ /* if not a one-row unique key, just return the key's expressions */
+ if (!onerow)
+ return list_make1(ukey->exprs);
+
+ /*
+ * If it's a one-row relation, we simply extract the expressions that
+ * still exist in the reltarget.
+ */
+ foreach(lc, joinrel->reltarget->exprs)
{
- res = list_make1(ukey->exprs);
+ Bitmapset *relids = pull_varnos(root, lfirst(lc));
+
+ if (bms_is_subset(relids, rel1->relids))
+ res = lappend(res, list_make1(lfirst(lc)));
}
+
return res;
}
@@ -910,55 +1230,67 @@ get_exprs_from_uniquekey(PlannerInfo *root, RelOptInfo *joinrel,
/*
* index_constains_partkey
- * return true if the index contains the partiton key.
+ * Determines if the index includes a partition key.
+ *
+ * XXX Surely we already have a code doing this already? E.g. when creating
+ * a unique index on a partitioned table we define that.
*/
static bool
-index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind)
+index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind)
{
ListCell *lc;
int i;
+
Assert(IS_PARTITIONED_REL(partrel));
Assert(partrel->part_scheme->partnatts > 0);
for(i = 0; i < partrel->part_scheme->partnatts; i++)
{
- Node *part_expr = linitial(partrel->partexprs[i]);
- bool found_in_index = false;
+ Node *part_expr = linitial(partrel->partexprs[i]);
+ bool found_in_index = false;
+
foreach(lc, ind->indextlist)
{
- Expr *index_expr = lfirst_node(TargetEntry, lc)->expr;
+ Expr *index_expr = lfirst_node(TargetEntry, lc)->expr;
+
if (equal(index_expr, part_expr))
{
found_in_index = true;
break;
}
}
+
if (!found_in_index)
return false;
}
+
return true;
}
/*
* simple_indexinfo_equal
+ * Compare two indexes to determine if they are the same.
+ *
+ * We need to do this because simple_copy_indexinfo_to_parent does change
+ * some elements. So this is not exactly the same as calling equal().
*
- * Used to check if the 2 index is same as each other. The index here
- * is COPIED from childrel and did some tiny changes(see
- * simple_copy_indexinfo_to_parent)
+ * XXX I wonder if we could simply use equal(), somehow? In fact, we should
+ * probably build something much simpler than IndexOptInfo, just enough to
+ * do the checks.
*/
static bool
simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2)
{
Size oid_cmp_len = sizeof(Oid) * ind1->ncolumns;
- return ind1->ncolumns == ind2->ncolumns &&
- ind1->unique == ind2->unique &&
- memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0 &&
- memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0 &&
- memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0 &&
- memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0 &&
- equal(get_tlist_exprs(ind1->indextlist, true),
- get_tlist_exprs(ind2->indextlist, true));
+ return ((ind1->ncolumns == ind2->ncolumns) &&
+ (ind1->unique == ind2->unique) &&
+ (memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0) &&
+ (memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0) &&
+ (memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0) &&
+ (memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0) &&
+ (equal(get_tlist_exprs(ind1->indextlist, true),
+ get_tlist_exprs(ind2->indextlist, true))));
}
@@ -981,11 +1313,21 @@ simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2)
/*
- * simple_copy_indexinfo_to_parent (from partition)
- * Copy the IndexInfo from child relation to parent relation with some modification,
- * which is used to test:
- * 1. If the same index exists in all the childrels.
+ * simple_copy_indexinfo_to_parent
+ * Copy index info from child to parent, with necessary tweaks.
+ *
+ * We use this copy to check:
+ *
+ * 1. If the same/matching index exists in all the childrels.
* 2. If the parentrel->reltarget/basicrestrict info matches this index.
+ *
+ * XXX IMHO we should probably build something much simpler than a full
+ * IndexOptInfo copy, just enough to do the checks.
+ *
+ * XXX The fact that we copy so much data seems wrong, and having to
+ * define macros from copyfuncs.c seems like a very suspicious thing.
+ * One reason is that IndeOptInfo is fairly large struct, especially
+ * with all the fields, and we allocate it very often.
*/
static IndexOptInfo *
simple_copy_indexinfo_to_parent(PlannerInfo *root,
@@ -1027,20 +1369,24 @@ simple_copy_indexinfo_to_parent(PlannerInfo *root,
/*
* adjust_partition_unique_indexlist
+ * Checks and eliminates indexes that do not exist on the child relation.
*
- * global_unique_indexes: At the beginning, it contains the copy & modified
- * unique index from the first partition. And then check if each index in it still
- * exists in the following partitions. If no, remove it. at last, it has an
- * index list which exists in all the partitions.
+ * Walks the list of unique indexes, and eliminates those that don't match
+ * the child relation (i.e. where a matching child index does not exist).
+ * This is used to iteratively filter the list of candidate unique keys.
+ *
+ * After processing all child relations, the list contains only indexes that
+ * exist in all the child relations.
*/
static void
adjust_partition_unique_indexlist(PlannerInfo *root,
RelOptInfo *parentrel,
RelOptInfo *childrel,
- List **global_unique_indexes)
+ List **indexes)
{
ListCell *lc, *lc2;
- foreach(lc, *global_unique_indexes)
+
+ foreach(lc, *indexes)
{
IndexOptInfo *g_ind = lfirst_node(IndexOptInfo, lc);
bool found_in_child = false;
@@ -1049,23 +1395,45 @@ adjust_partition_unique_indexlist(PlannerInfo *root,
{
IndexOptInfo *p_ind = lfirst_node(IndexOptInfo, lc2);
IndexOptInfo *p_ind_copy;
- if (!p_ind->unique || !p_ind->immediate ||
- (p_ind->indpred != NIL && !p_ind->predOK))
+
+ /*
+ * Ignore child indexes that can't possibly match (not unique or
+ * immediate, etc.)
+ *
+ * XXX We do these checks in many places, so maybe turn it into
+ * a reusable macro?
+ */
+ if ((!p_ind->unique) || (!p_ind->immediate) ||
+ (p_ind->indpred != NIL) && (!p_ind->predOK))
continue;
+
+ /*
+ * XXX This seems possibly quite expensive. Imagine there are many
+ * child relations, with a bunch of unique indexes each. Then this
+ * generates a copy for each unique index in each child relation,
+ * something like O(N^2/2) copies.
+ */
p_ind_copy = simple_copy_indexinfo_to_parent(root, parentrel, p_ind);
+
+ /* Found a matching index for the child relation, we're done. */
if (simple_indexinfo_equal(p_ind_copy, g_ind))
{
found_in_child = true;
break;
}
}
+
+ /* No matching index in the child, so remove it from the list. */
if (!found_in_child)
- /* The index doesn't exist in childrel, remove it from global_unique_indexes */
- *global_unique_indexes = foreach_delete_current(*global_unique_indexes, lc);
+ *indexes = foreach_delete_current(*indexes, lc);
}
}
-/* Helper function for groupres/distinctrel */
+/*
+ * Helper function for groupres/distinctrel
+ *
+ * FIXME Not sure about this.
+ */
static void
add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgroups)
{
@@ -1073,27 +1441,32 @@ add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgrou
List *exprs;
/*
- * XXX: If there are some vars which is not in current levelsup, the semantic is
- * imprecise, should we avoid it or not? levelsup = 1 is just a demo, maybe we need to
- * check every level other than 0, if so, looks we have to write another
- * pull_var_walker.
+ * XXX: If there are some vars which are not in the current levelsup, the
+ * semantic is imprecise, should we avoid it or not? levelsup = 1 is just
+ * a demo, maybe we need to check every level other than 0, if so, looks
+ * we have to write another pull_var_walker.
*/
List *upper_vars = pull_vars_of_level((Node*)sortgroups, 1);
if (upper_vars != NIL)
return;
+ /* sortgroupclause can't be multi_nullvals */
exprs = get_sortgrouplist_exprs(sortgroups, parse->targetList);
rel->uniquekeys = lappend(rel->uniquekeys,
- makeUniqueKey(exprs,
- false /* sortgroupclause can't be multi_nullvals */));
+ makeUniqueKey(exprs, false));
}
/*
* add_combined_uniquekey
- * The combination of both UniqueKeys is a valid UniqueKey for joinrel no matter
- * the jointype.
+ * Add a unique key for a join, combined from keys on inner/outer side.
+ *
+ * The combination of both UniqueKeys is a valid UniqueKey for joinrel no
+ * matter what's the exact jointype.
+ *
+ * Returns true if the unique key is "one-row" variant, so that the caller
+ * can stop considering further combinations.
*/
bool
add_combined_uniquekey(PlannerInfo *root,
@@ -1104,32 +1477,47 @@ add_combined_uniquekey(PlannerInfo *root,
UniqueKey *inner_ukey,
JoinType jointype)
{
+ bool multi_nullvals;
+ ListCell *lc1, *lc2;
- ListCell *lc1, *lc2;
-
- /* Either side has multi_nullvals or we have outer join,
- * the combined UniqueKey has multi_nullvals */
- bool multi_nullvals = outer_ukey->multi_nullvals ||
+ /*
+ * If either side has multi_nullvals, or we are dealing with an outer join,
+ * the combined UniqueKey has multi_nullvals too.
+ */
+ multi_nullvals = outer_ukey->multi_nullvals ||
inner_ukey->multi_nullvals || IS_OUTER_JOIN(jointype);
/* The only case we can get onerow joinrel after join */
- if (relation_is_onerow(outer_rel)
- && relation_is_onerow(inner_rel)
- && jointype == JOIN_INNER)
+ if (relation_is_onerow(outer_rel) &&
+ relation_is_onerow(inner_rel) &&
+ jointype == JOIN_INNER)
{
add_uniquekey_for_onerow(joinrel);
return true;
}
+ /*
+ * XXX Isn't this wrong? Why is it combining expressions that are part
+ * of the two unique keys? Imagine we have outer unique key on (a1, a2)
+ * and inner outer key on (b1, b2). Then this adds four unique keys
+ * for the join (a1,b1), (a1,b2), (a2,b1) and (a2,b2). Shouldn't it
+ * just add (a1,a2,b1,b2)?
+ */
foreach(lc1, get_exprs_from_uniquekey(root, joinrel, outer_rel, outer_ukey))
{
+ /*
+ * XXX This calls get_exprs_from_uniquekey repeatedly for each outer
+ * loop. Maybe we should calculate it just once before the loop.
+ */
foreach(lc2, get_exprs_from_uniquekey(root, joinrel, inner_rel, inner_ukey))
{
List *exprs = list_concat_copy(lfirst_node(List, lc1), lfirst_node(List, lc2));
+
joinrel->uniquekeys = lappend(joinrel->uniquekeys,
makeUniqueKey(exprs,
multi_nullvals));
}
}
+
return false;
}
diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c
index 8d8e493f5c..f29b65c07b 100644
--- a/src/backend/optimizer/plan/planner.c
+++ b/src/backend/optimizer/plan/planner.c
@@ -2387,6 +2387,7 @@ grouping_planner(PlannerInfo *root, bool inheritance_update,
add_path(final_rel, path);
}
+ /* XXX comment? can we simply just copy the unique keys to the final relation? */
simple_copy_uniquekeys(current_rel, final_rel);
/*
@@ -3902,7 +3903,9 @@ create_grouping_paths(PlannerInfo *root,
set_cheapest(grouped_rel);
+ /* XXX does this apply to grouping sets too? */
populate_grouprel_uniquekeys(root, grouped_rel, input_rel);
+
return grouped_rel;
}
@@ -4625,7 +4628,10 @@ create_window_paths(PlannerInfo *root,
/* Now choose the best path(s) */
set_cheapest(window_rel);
+
+ /* XXX comment? */
simple_copy_uniquekeys(input_rel, window_rel);
+
return window_rel;
}
@@ -4939,7 +4945,10 @@ create_distinct_paths(PlannerInfo *root,
/* Now choose the best path(s) */
set_cheapest(distinct_rel);
+
+ /* XXX comment */
populate_distinctrel_uniquekeys(root, input_rel, distinct_rel);
+
return distinct_rel;
}
@@ -5200,6 +5209,7 @@ create_ordered_paths(PlannerInfo *root,
*/
Assert(ordered_rel->pathlist != NIL);
+ /* XXX comment */
simple_copy_uniquekeys(input_rel, ordered_rel);
return ordered_rel;
diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c
index b7626545bf..72a3f3c598 100644
--- a/src/backend/optimizer/prep/prepunion.c
+++ b/src/backend/optimizer/prep/prepunion.c
@@ -691,6 +691,7 @@ generate_union_paths(SetOperationStmt *op, PlannerInfo *root,
/* Add the UniqueKeys */
populate_unionrel_uniquekeys(root, result_rel);
+
return result_rel;
}
diff --git a/src/backend/optimizer/util/inherit.c b/src/backend/optimizer/util/inherit.c
index 3eec1f4d74..c9829c5fc4 100644
--- a/src/backend/optimizer/util/inherit.c
+++ b/src/backend/optimizer/util/inherit.c
@@ -755,6 +755,7 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel,
pseudoconstant,
rinfo->security_level,
NULL, NULL, NULL);
+ /* XXX This is a bit weird, doing this outside make_restrictinfo */
child_rinfo->mergeopfamilies = rinfo->mergeopfamilies;
childquals = lappend(childquals, child_rinfo);
/* track minimum security level among child quals */
--
2.30.2
--------------F495F6B18672582269F00FF9
Content-Type: text/x-patch; charset=UTF-8;
name="0005-Extend-UniqueKeys-20210317.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="0005-Extend-UniqueKeys-20210317.patch"
^ permalink raw reply [nested|flat] 97+ messages in thread
* [PATCH 06/10] review
@ 2021-03-17 01:19 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: Tomas Vondra @ 2021-03-17 01:19 UTC (permalink / raw)
---
src/backend/optimizer/path/uniquekeys.c | 2 ++
src/backend/optimizer/plan/planner.c | 2 ++
2 files changed, 4 insertions(+)
diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c
index b9567ca5c8..0e6c826fcd 100644
--- a/src/backend/optimizer/path/uniquekeys.c
+++ b/src/backend/optimizer/path/uniquekeys.c
@@ -1522,6 +1522,7 @@ add_combined_uniquekey(PlannerInfo *root,
return false;
}
+/* FIXME comment */
List*
build_uniquekeys(PlannerInfo *root, List *sortclauses)
{
@@ -1548,6 +1549,7 @@ build_uniquekeys(PlannerInfo *root, List *sortclauses)
return result;
}
+/* FIXME comment */
bool
query_has_uniquekeys_for(PlannerInfo *root, List *pathuniquekeys,
bool allow_multinulls)
diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c
index 62c0e46c49..f66b9d4898 100644
--- a/src/backend/optimizer/plan/planner.c
+++ b/src/backend/optimizer/plan/planner.c
@@ -4857,11 +4857,13 @@ create_distinct_paths(PlannerInfo *root,
Path *path = (Path *) lfirst(lc);
if (pathkeys_contained_in(needed_pathkeys, path->pathkeys))
+ {
add_path(distinct_rel, (Path *)
create_upper_unique_path(root, distinct_rel,
path,
list_length(root->distinct_pathkeys),
numDistinctRows));
+ }
}
foreach(lc, input_rel->unique_pathlist)
--
2.30.2
--------------F495F6B18672582269F00FF9
Content-Type: text/x-patch; charset=UTF-8;
name="0007-Index-skip-scan-20210317.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="0007-Index-skip-scan-20210317.patch"
^ permalink raw reply [nested|flat] 97+ messages in thread
* [PATCH 06/10] review
@ 2021-03-17 01:19 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: Tomas Vondra @ 2021-03-17 01:19 UTC (permalink / raw)
---
src/backend/optimizer/path/uniquekeys.c | 2 ++
src/backend/optimizer/plan/planner.c | 2 ++
2 files changed, 4 insertions(+)
diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c
index b9567ca5c8..0e6c826fcd 100644
--- a/src/backend/optimizer/path/uniquekeys.c
+++ b/src/backend/optimizer/path/uniquekeys.c
@@ -1522,6 +1522,7 @@ add_combined_uniquekey(PlannerInfo *root,
return false;
}
+/* FIXME comment */
List*
build_uniquekeys(PlannerInfo *root, List *sortclauses)
{
@@ -1548,6 +1549,7 @@ build_uniquekeys(PlannerInfo *root, List *sortclauses)
return result;
}
+/* FIXME comment */
bool
query_has_uniquekeys_for(PlannerInfo *root, List *pathuniquekeys,
bool allow_multinulls)
diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c
index 62c0e46c49..f66b9d4898 100644
--- a/src/backend/optimizer/plan/planner.c
+++ b/src/backend/optimizer/plan/planner.c
@@ -4857,11 +4857,13 @@ create_distinct_paths(PlannerInfo *root,
Path *path = (Path *) lfirst(lc);
if (pathkeys_contained_in(needed_pathkeys, path->pathkeys))
+ {
add_path(distinct_rel, (Path *)
create_upper_unique_path(root, distinct_rel,
path,
list_length(root->distinct_pathkeys),
numDistinctRows));
+ }
}
foreach(lc, input_rel->unique_pathlist)
--
2.30.2
--------------F495F6B18672582269F00FF9
Content-Type: text/x-patch; charset=UTF-8;
name="0007-Index-skip-scan-20210317.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="0007-Index-skip-scan-20210317.patch"
^ permalink raw reply [nested|flat] 97+ messages in thread
* [PATCH 08/10] review
@ 2021-03-17 01:36 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: Tomas Vondra @ 2021-03-17 01:36 UTC (permalink / raw)
---
src/backend/commands/explain.c | 1 +
src/backend/executor/nodeIndexscan.c | 2 ++
2 files changed, 3 insertions(+)
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index a160de5493..3b0b38099f 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -1110,6 +1110,7 @@ ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
{
if (skipPrefixSize > 0)
{
+ /* FIXME Why not to show this for TEXT output? */
if (es->format != EXPLAIN_FORMAT_TEXT)
ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
}
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index 71aac4493d..720442b731 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -124,6 +124,8 @@ IndexNext(IndexScanState *node)
node->iss_ScanDesc = scandesc;
+ /* FIXME Is the else branch necessary? */
+
/* Index skip scan assumes xs_want_itup, so set it to true */
if (indexscan->indexskipprefixsize > 0)
node->iss_ScanDesc->xs_want_itup = true;
--
2.30.2
--------------F495F6B18672582269F00FF9
Content-Type: text/x-patch; charset=UTF-8;
name="0009-Btree-implementation-of-skipping-20210317.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="0009-Btree-implementation-of-skipping-20210317.patch"
^ permalink raw reply [nested|flat] 97+ messages in thread
* [PATCH 08/10] review
@ 2021-03-17 01:36 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: Tomas Vondra @ 2021-03-17 01:36 UTC (permalink / raw)
---
src/backend/commands/explain.c | 1 +
src/backend/executor/nodeIndexscan.c | 2 ++
2 files changed, 3 insertions(+)
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index a160de5493..3b0b38099f 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -1110,6 +1110,7 @@ ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
{
if (skipPrefixSize > 0)
{
+ /* FIXME Why not to show this for TEXT output? */
if (es->format != EXPLAIN_FORMAT_TEXT)
ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
}
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index 71aac4493d..720442b731 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -124,6 +124,8 @@ IndexNext(IndexScanState *node)
node->iss_ScanDesc = scandesc;
+ /* FIXME Is the else branch necessary? */
+
/* Index skip scan assumes xs_want_itup, so set it to true */
if (indexscan->indexskipprefixsize > 0)
node->iss_ScanDesc->xs_want_itup = true;
--
2.30.2
--------------F495F6B18672582269F00FF9
Content-Type: text/x-patch; charset=UTF-8;
name="0009-Btree-implementation-of-skipping-20210317.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="0009-Btree-implementation-of-skipping-20210317.patch"
^ permalink raw reply [nested|flat] 97+ messages in thread
* Re: [PATCH] Preserve replication origin OIDs in pg_upgrade
@ 2026-05-18 10:42 Ajin Cherian <[email protected]>
0 siblings, 1 reply; 97+ messages in thread
From: Ajin Cherian @ 2026-05-18 10:42 UTC (permalink / raw)
To: Zsolt Parragi <[email protected]>; +Cc: vignesh C <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; PostgreSQL Hackers <[email protected]>
Rebased the patch as it was no longer applying.
regards,
Ajin Cherian
Fujitsu Australia
Attachments:
[application/octet-stream] v6-0002-Preserve-replication-origin-OIDs-during-pg_upgrad.patch (22.0K, ../../CAFPTHDaftSwzGTGbFEw8rwDBsub0XqcDm1wxQcquj-Y3PC2qrg@mail.gmail.com/2-v6-0002-Preserve-replication-origin-OIDs-during-pg_upgrad.patch)
download | inline diff:
From c03447597e366d13fee22b288a822a66233329a7 Mon Sep 17 00:00:00 2001
From: Ajin Cherian <[email protected]>
Date: Mon, 18 May 2026 20:34:44 +1000
Subject: [PATCH v6 2/2] Preserve replication origin OIDs during pg_upgrade
When pg_upgrade migrates a subscriber, replication origin OIDs
(roident) can change across the upgrade. This is a problem because
commit-timestamp records embed roident and are copied directly from
the old cluster's pg_commit_ts directory, causing spurious
"update_origin_differs" conflicts after the upgrade.
Fix this by dumping replication origins as global objects via
pg_dumpall during binary upgrade, using a new function
binary_upgrade_create_replication_origin(oid, name, lsn) to recreate
each origin with its preserved roident and remote_lsn. To avoid
conflicts with this, CreateSubscription() skips replorigin_create()
in binary-upgrade mode since the origin is already created by the
time the subscription is restored.
Author: Ajin Cherian <[email protected]>
Reviewer: Hayato Kuroda (Fujitsu) <[email protected]>
Reviewer: Zsolt Parragi <[email protected]>
---
src/backend/commands/subscriptioncmds.c | 11 +-
src/backend/utils/adt/pg_upgrade_support.c | 159 +++++++++++++++------
src/bin/pg_dump/pg_dump.c | 38 ++---
src/bin/pg_dump/pg_dumpall.c | 64 +++++++++
src/bin/pg_upgrade/check.c | 13 +-
src/bin/pg_upgrade/info.c | 9 ++
src/bin/pg_upgrade/pg_upgrade.h | 1 +
src/bin/pg_upgrade/t/004_subscription.pl | 42 +++++-
src/include/catalog/pg_proc.dat | 8 +-
9 files changed, 253 insertions(+), 92 deletions(-)
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 7c1f05a5fd5..b31e15256a2 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -19,6 +19,7 @@
#include "access/table.h"
#include "access/twophase.h"
#include "access/xact.h"
+#include "catalog/binary_upgrade.h"
#include "catalog/catalog.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
@@ -867,9 +868,15 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
* apply workers initialization, and to handle origin creation dynamically
* when tables are added to the subscription. It is not clear whether
* preventing creation of origins is worth additional complexity.
+ *
+ * In binary-upgrade mode, skip origin creation here. This is required to
+ * preserve the roident from the old cluster for this subscription.
*/
- ReplicationOriginNameForLogicalRep(subid, InvalidOid, originname, sizeof(originname));
- replorigin_create(originname);
+ if (!IsBinaryUpgrade)
+ {
+ ReplicationOriginNameForLogicalRep(subid, InvalidOid, originname, sizeof(originname));
+ replorigin_create(originname);
+ }
/*
* Connect to remote side to execute requested commands and fetch table
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index 59c3e7f0146..41fde3f09fc 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -11,10 +11,13 @@
#include "postgres.h"
+#include "access/genam.h"
#include "access/relation.h"
+#include "access/skey.h"
#include "access/table.h"
#include "catalog/binary_upgrade.h"
#include "catalog/heap.h"
+#include "catalog/indexing.h"
#include "catalog/namespace.h"
#include "catalog/pg_subscription_rel.h"
#include "catalog/pg_type.h"
@@ -27,8 +30,10 @@
#include "storage/lmgr.h"
#include "utils/array.h"
#include "utils/builtins.h"
+#include "utils/fmgroids.h"
#include "utils/lsyscache.h"
#include "utils/pg_lsn.h"
+#include "utils/snapmgr.h"
#define CHECK_IS_BINARY_UPGRADE \
@@ -377,71 +382,133 @@ binary_upgrade_add_sub_rel_state(PG_FUNCTION_ARGS)
}
/*
- * binary_upgrade_replorigin_advance
+ * binary_upgrade_create_conflict_detection_slot
*
- * Update the remote_lsn for the subscriber's replication origin.
+ * Create a replication slot to retain information necessary for conflict
+ * detection such as dead tuples, commit timestamps, and origins.
*/
Datum
-binary_upgrade_replorigin_advance(PG_FUNCTION_ARGS)
+binary_upgrade_create_conflict_detection_slot(PG_FUNCTION_ARGS)
{
- Relation rel;
- Oid subid;
- char *subname;
- char originname[NAMEDATALEN];
- ReplOriginId node;
- XLogRecPtr remote_commit;
-
CHECK_IS_BINARY_UPGRADE;
- /*
- * We must ensure a non-NULL subscription name before dereferencing the
- * arguments.
- */
- if (PG_ARGISNULL(0))
- elog(ERROR, "null argument to binary_upgrade_replorigin_advance is not allowed");
-
- subname = text_to_cstring(PG_GETARG_TEXT_PP(0));
- remote_commit = PG_ARGISNULL(1) ? InvalidXLogRecPtr : PG_GETARG_LSN(1);
-
- rel = table_open(SubscriptionRelationId, RowExclusiveLock);
- subid = get_subscription_oid(subname, false);
-
- ReplicationOriginNameForLogicalRep(subid, InvalidOid, originname, sizeof(originname));
-
- /* Lock to prevent the replication origin from vanishing */
- LockRelationOid(ReplicationOriginRelationId, RowExclusiveLock);
- node = replorigin_by_name(originname, false);
-
- /*
- * The server will be stopped after setting up the objects in the new
- * cluster and the origins will be flushed during the shutdown checkpoint.
- * This will ensure that the latest LSN values for origin will be
- * available after the upgrade.
- */
- replorigin_advance(node, remote_commit, InvalidXLogRecPtr,
- false /* backward */ ,
- false /* WAL log */ );
+ CreateConflictDetectionSlot();
- UnlockRelationOid(ReplicationOriginRelationId, RowExclusiveLock);
- table_close(rel, RowExclusiveLock);
+ ReplicationSlotRelease();
PG_RETURN_VOID();
}
/*
- * binary_upgrade_create_conflict_detection_slot
+ * binary_upgrade_create_replication_origin
*
- * Create a replication slot to retain information necessary for conflict
- * detection such as dead tuples, commit timestamps, and origins.
+ * Create a replication origin with a specific OID and name, optionally
+ * restoring its remote_lsn. Used by pg_upgrade to preserve replication
+ * origin OIDs across the upgrade.
*/
Datum
-binary_upgrade_create_conflict_detection_slot(PG_FUNCTION_ARGS)
+binary_upgrade_create_replication_origin(PG_FUNCTION_ARGS)
{
+ Oid node_oid;
+ ReplOriginId node;
+ Name originname;
+ Relation rel;
+ HeapTuple tuple;
+ Datum roname_d;
+ SysScanDesc scan;
+ ScanKeyData key;
+ bool nulls[Natts_pg_replication_origin];
+ Datum values[Natts_pg_replication_origin];
+ bool collides;
+
CHECK_IS_BINARY_UPGRADE;
- CreateConflictDetectionSlot();
+ if (PG_ARGISNULL(0) || PG_ARGISNULL(1))
+ elog(ERROR,
+ "null argument to binary_upgrade_create_replication_origin is not allowed");
+
+ node_oid = PG_GETARG_OID(0);
+
+ if (node_oid == InvalidOid || node_oid > PG_UINT16_MAX)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("replication origin ID %u is out of range", node_oid)));
+
+ node = (ReplOriginId) node_oid;
+ originname = PG_GETARG_NAME(1);
+
+ if (strlen(NameStr(*originname)) > MAX_RONAME_LEN)
+ ereport(ERROR,
+ (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
+ errmsg("replication origin name is too long"),
+ errdetail("Replication origin names must be no longer than %d bytes.",
+ MAX_RONAME_LEN)));
+
+ roname_d = CStringGetTextDatum(NameStr(*originname));
+
+ Assert(IsTransactionState());
+
+ rel = table_open(ReplicationOriginRelationId, RowExclusiveLock);
+
+ Assert(!OidIsValid(rel->rd_rel->reltoastrelid));
+
+ /* Check for OID collision */
+ ScanKeyInit(&key,
+ Anum_pg_replication_origin_roident,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(node));
+ scan = systable_beginscan(rel, ReplicationOriginIdentIndex,
+ true /* indexOK */,
+ SnapshotSelf,
+ 1, &key);
+ collides = HeapTupleIsValid(systable_getnext(scan));
+ systable_endscan(scan);
+
+ if (collides)
+ ereport(ERROR,
+ (errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("replication origin with ID %u already exists", node_oid)));
+
+ /* Check for name collision */
+ ScanKeyInit(&key,
+ Anum_pg_replication_origin_roname,
+ BTEqualStrategyNumber, F_TEXTEQ,
+ roname_d);
+ scan = systable_beginscan(rel, ReplicationOriginNameIndex,
+ true /* indexOK */,
+ SnapshotSelf,
+ 1, &key);
+ collides = HeapTupleIsValid(systable_getnext(scan));
+ systable_endscan(scan);
+
+ if (collides)
+ ereport(ERROR,
+ (errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("replication origin \"%s\" already exists",
+ NameStr(*originname))));
+
+ memset(&nulls, 0, sizeof(nulls));
+ memset(&values, 0, sizeof(values));
+
+ values[Anum_pg_replication_origin_roident - 1] = ObjectIdGetDatum(node);
+ values[Anum_pg_replication_origin_roname - 1] = roname_d;
+
+ tuple = heap_form_tuple(RelationGetDescr(rel), values, nulls);
+ CatalogTupleInsert(rel, tuple);
+ heap_freetuple(tuple);
+ CommandCounterIncrement();
+
+ /* Restore the remote_lsn if provided, while still holding the lock */
+ if (!PG_ARGISNULL(2))
+ {
+ XLogRecPtr remote_commit = PG_GETARG_LSN(2);
- ReplicationSlotRelease();
+ replorigin_advance(node, remote_commit, InvalidXLogRecPtr,
+ false /* backward */,
+ false /* WAL log */);
+ }
+
+ table_close(rel, RowExclusiveLock);
PG_RETURN_VOID();
}
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 452d0b5e98a..a5fb2b42c3d 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -5668,37 +5668,15 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
* In binary-upgrade mode, we allow the replication to continue after the
* upgrade.
*/
- if (dopt->binary_upgrade && fout->remoteVersion >= 170000)
+ if (dopt->binary_upgrade && subinfo->subenabled && fout->remoteVersion >= 170000)
{
- if (subinfo->suboriginremotelsn)
- {
- /*
- * Preserve the remote_lsn for the subscriber's replication
- * origin. This value is required to start the replication from
- * the position before the upgrade. This value will be stale if
- * the publisher gets upgraded before the subscriber node.
- * However, this shouldn't be a problem as the upgrade of the
- * publisher ensures that all the transactions were replicated
- * before upgrading it.
- */
- appendPQExpBufferStr(query,
- "\n-- For binary upgrade, must preserve the remote_lsn for the subscriber's replication origin.\n");
- appendPQExpBufferStr(query,
- "SELECT pg_catalog.binary_upgrade_replorigin_advance(");
- appendStringLiteralAH(query, subinfo->dobj.name, fout);
- appendPQExpBuffer(query, ", '%s');\n", subinfo->suboriginremotelsn);
- }
-
- if (subinfo->subenabled)
- {
- /*
- * Enable the subscription to allow the replication to continue
- * after the upgrade.
- */
- appendPQExpBufferStr(query,
- "\n-- For binary upgrade, must preserve the subscriber's running state.\n");
- appendPQExpBuffer(query, "ALTER SUBSCRIPTION %s ENABLE;\n", qsubname);
- }
+ /*
+ * Enable the subscription to allow the replication to continue
+ * after the upgrade.
+ */
+ appendPQExpBufferStr(query,
+ "\n-- For binary upgrade, must preserve the subscriber's running state.\n");
+ appendPQExpBuffer(query, "ALTER SUBSCRIPTION %s ENABLE;\n", qsubname);
}
if (subinfo->dobj.dump & DUMP_COMPONENT_DEFINITION)
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index c1f43113c53..e16918feccf 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -25,6 +25,7 @@
#include <time.h>
#include <unistd.h>
+#include "access/xlogdefs.h"
#include "catalog/pg_authid_d.h"
#include "common/connect.h"
#include "common/file_perm.h"
@@ -76,6 +77,7 @@ static void dropDBs(PGconn *conn);
static void dumpUserConfig(PGconn *conn, const char *username);
static void dumpDatabases(PGconn *conn);
static void dumpTimestamp(const char *msg);
+static void dumpReplicationOrigins(PGconn *conn);
static int runPgDump(const char *dbname, const char *create_opts, char *dbfile);
static void buildShSecLabels(PGconn *conn,
const char *catalog_name, Oid objectId,
@@ -813,6 +815,10 @@ main(int argc, char *argv[])
/* Dump role GUC privileges */
if (server_version >= 150000 && !skip_acls)
dumpRoleGUCPrivs(conn);
+
+ /* Dump replication origins */
+ if (server_version >= 170000 && binary_upgrade && archDumpFormat == archNull)
+ dumpReplicationOrigins(conn);
}
/* Dump tablespaces */
@@ -2339,6 +2345,64 @@ dumpTimestamp(const char *msg)
fprintf(OPF, "-- %s %s\n\n", msg, buf);
}
+static void
+dumpReplicationOrigins(PGconn *conn)
+{
+ PQExpBuffer buf = createPQExpBuffer();
+ PGresult *res;
+ int i_roident;
+ int i_roname;
+ int i_remotelsn;
+
+ /* Get replication origins from catalogs */
+ appendPQExpBufferStr(buf,
+ "SELECT o.*, os.remote_lsn "
+ "FROM pg_catalog.pg_replication_origin o "
+ "LEFT OUTER JOIN pg_catalog.pg_replication_origin_status os ON o.roident = os.local_id ");
+
+ res = executeQuery(conn, buf->data);
+
+ i_roident = PQfnumber(res, "roident");
+ i_roname = PQfnumber(res, "roname");
+ i_remotelsn = PQfnumber(res, "remote_lsn");
+
+ if (PQntuples(res) > 0)
+ fprintf(OPF, "--\n-- Replication Origins \n--\n\n");
+
+ for (int i = 0; i < PQntuples(res); i++)
+ {
+ ReplOriginId roident;
+ const char *roname;
+
+ roident = atooid(PQgetvalue(res, i, i_roident));
+ roname = PQgetvalue(res, i, i_roname);
+
+ resetPQExpBuffer(buf);
+
+ appendPQExpBufferStr(buf, "\n-- For binary upgrade, must preserve replication origin roident and remote_lsn\n");
+ appendPQExpBuffer(buf,
+ "SELECT pg_catalog.binary_upgrade_create_replication_origin("
+ "'%u'::pg_catalog.oid, ", roident);
+ appendStringLiteralConn(buf, roname, conn);
+ appendPQExpBufferStr(buf, "::pg_catalog.name");
+
+ if (!PQgetisnull(res, i, i_remotelsn))
+ {
+ appendPQExpBufferStr(buf, ", ");
+ appendStringLiteralConn(buf, PQgetvalue(res, i, i_remotelsn), conn);
+ appendPQExpBufferStr(buf, "::pg_catalog.pg_lsn");
+ }
+ else
+ appendPQExpBufferStr(buf, ", NULL");
+
+ appendPQExpBufferStr(buf, ");\n");
+ fprintf(OPF, "%s", buf->data);
+ }
+
+ PQclear(res);
+ destroyPQExpBuffer(buf);
+}
+
/*
* read_dumpall_filters - retrieve database identifier patterns from file
*
diff --git a/src/bin/pg_upgrade/check.c b/src/bin/pg_upgrade/check.c
index f5c93e611d2..a3c027c7cd0 100644
--- a/src/bin/pg_upgrade/check.c
+++ b/src/bin/pg_upgrade/check.c
@@ -2306,8 +2306,7 @@ check_new_cluster_replication_slots(void)
* check_new_cluster_subscription_configuration()
*
* Verify that the max_active_replication_origins configuration specified is
- * enough for creating the subscriptions. This is required to create the
- * replication origin for each subscription.
+ * enough for creating all the replication origins.
*/
static void
check_new_cluster_subscription_configuration(void)
@@ -2320,8 +2319,8 @@ check_new_cluster_subscription_configuration(void)
if (GET_MAJOR_VERSION(old_cluster.major_version) < 1700)
return;
- /* Quick return if there are no subscriptions to be migrated. */
- if (old_cluster.nsubs == 0)
+ /* Quick return if there are no replication origins to be migrated. */
+ if (old_cluster.nrepl_origins == 0)
return;
prep_status("Checking new cluster configuration for subscriptions");
@@ -2335,10 +2334,10 @@ check_new_cluster_subscription_configuration(void)
pg_fatal("could not determine parameter settings on new cluster");
max_active_replication_origins = atoi(PQgetvalue(res, 0, 0));
- if (old_cluster.nsubs > max_active_replication_origins)
+ if (old_cluster.nrepl_origins > max_active_replication_origins)
pg_fatal("\"max_active_replication_origins\" (%d) must be greater than or equal to the number of "
- "subscriptions (%d) on the old cluster",
- max_active_replication_origins, old_cluster.nsubs);
+ "replication origins (%d) on the old cluster",
+ max_active_replication_origins, old_cluster.nrepl_origins);
PQclear(res);
PQfinish(conn);
diff --git a/src/bin/pg_upgrade/info.c b/src/bin/pg_upgrade/info.c
index 37fff93892f..630f3f06e24 100644
--- a/src/bin/pg_upgrade/info.c
+++ b/src/bin/pg_upgrade/info.c
@@ -843,6 +843,7 @@ get_subscription_info(ClusterInfo *cluster)
PGconn *conn;
PGresult *res;
int i_nsub;
+ int i_nrepl_origins;
int i_retain_dead_tuples;
conn = connectToServer(cluster, "template1");
@@ -862,6 +863,14 @@ get_subscription_info(ClusterInfo *cluster)
cluster->sub_retain_dead_tuples = (strcmp(PQgetvalue(res, 0, i_retain_dead_tuples), "t") == 0);
PQclear(res);
+
+ res = executeQueryOrDie(conn,
+ "SELECT count(*) AS nrepl_origins "
+ "FROM pg_catalog.pg_replication_origin");
+ i_nrepl_origins = PQfnumber(res, "nrepl_origins");
+ cluster->nrepl_origins = atoi(PQgetvalue(res, 0, i_nrepl_origins));
+ PQclear(res);
+
PQfinish(conn);
}
diff --git a/src/bin/pg_upgrade/pg_upgrade.h b/src/bin/pg_upgrade/pg_upgrade.h
index ccd1ac0d013..77e7ca1b4cd 100644
--- a/src/bin/pg_upgrade/pg_upgrade.h
+++ b/src/bin/pg_upgrade/pg_upgrade.h
@@ -311,6 +311,7 @@ typedef struct
int num_tablespaces;
const char *tablespace_suffix; /* directory specification */
int nsubs; /* number of subscriptions */
+ int nrepl_origins; /* number of replication origins */
bool sub_retain_dead_tuples; /* whether a subscription enables
* retain_dead_tuples. */
} ClusterInfo;
diff --git a/src/bin/pg_upgrade/t/004_subscription.pl b/src/bin/pg_upgrade/t/004_subscription.pl
index 646767f2a65..e8b11d39dd0 100644
--- a/src/bin/pg_upgrade/t/004_subscription.pl
+++ b/src/bin/pg_upgrade/t/004_subscription.pl
@@ -42,7 +42,7 @@ my $connstr = $publisher->connstr . ' dbname=postgres';
# ------------------------------------------------------
# Check that pg_upgrade fails when max_active_replication_origins configured
-# in the new cluster is less than the number of subscriptions in the old
+# in the new cluster is less than the number of replication origins in the old
# cluster.
# ------------------------------------------------------
# It is sufficient to use disabled subscription to test upgrade failure.
@@ -74,7 +74,7 @@ command_checks_all(
],
1,
[
- qr/"max_active_replication_origins" \(0\) must be greater than or equal to the number of subscriptions \(1\) on the old cluster/
+ qr/"max_active_replication_origins" \(0\) must be greater than or equal to the number of replication origins \(1\) on the old cluster/
],
[qr//],
'run of pg_upgrade where the new cluster has insufficient max_active_replication_origins'
@@ -301,8 +301,30 @@ is($result, qq(t), "Check that the table is in init state");
# Get the replication origin's remote_lsn of the old subscriber
my $remote_lsn = $old_sub->safe_psql('postgres',
- "SELECT remote_lsn FROM pg_replication_origin_status os, pg_subscription s WHERE os.external_id = 'pg_' || s.oid AND s.subname = 'regress_sub4'"
+ "SELECT os.remote_lsn
+ FROM pg_replication_origin_status os
+ JOIN pg_replication_origin o ON o.roident = os.local_id
+ JOIN pg_subscription s ON o.roname = 'pg_' || s.oid::text
+ WHERE s.subname = 'regress_sub4'"
);
+
+# Get the replication origin OIDs (roident) for all subscriptions, keyed by
+# subscription name (which is stable across upgrade, unlike suboid). These
+# must be preserved after upgrade. A mismatch would cause spurious
+# update_origin_differs conflicts.
+my %pre_upgrade_roident;
+my $roident_rows = $old_sub->safe_psql('postgres',
+ "SELECT s.subname, o.roident
+ FROM pg_subscription s
+ JOIN pg_replication_origin o ON o.roname = 'pg_' || s.oid::text
+ ORDER BY s.subname"
+);
+for my $row (split /\n/, $roident_rows)
+{
+ my ($subname, $roident) = split /\|/, $row;
+ $pre_upgrade_roident{$subname} = $roident;
+}
+
# Have the subscription in disabled state before upgrade
$old_sub->safe_psql('postgres', "ALTER SUBSCRIPTION regress_sub5 DISABLE");
@@ -378,6 +400,20 @@ regress_sub5|f|f|f),
"check that the subscription's running status, failover, and retain_dead_tuples are preserved"
);
+# Verify that replication origin OIDs are preserved after upgrade.
+my $post_roident_rows = $new_sub->safe_psql('postgres',
+ "SELECT s.subname, o.roident
+ FROM pg_subscription s
+ JOIN pg_replication_origin o ON o.roname = 'pg_' || s.oid::text
+ ORDER BY s.subname"
+);
+for my $row (split /\n/, $post_roident_rows)
+{
+ my ($subname, $roident) = split /\|/, $row;
+ is($roident, $pre_upgrade_roident{$subname},
+ "roident preserved for subscription '$subname' after upgrade");
+}
+
# Subscription relations should be preserved
$result = $new_sub->safe_psql('postgres',
"SELECT srrelid, srsubstate FROM pg_subscription_rel ORDER BY srrelid");
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 3a28406981d..1d2d624392e 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -11960,10 +11960,6 @@
provolatile => 'v', proparallel => 'u', prorettype => 'void',
proargtypes => 'text oid char pg_lsn',
prosrc => 'binary_upgrade_add_sub_rel_state' },
-{ oid => '6320', descr => 'for use by pg_upgrade (remote_lsn for origin)',
- proname => 'binary_upgrade_replorigin_advance', proisstrict => 'f',
- provolatile => 'v', proparallel => 'u', prorettype => 'void',
- proargtypes => 'text pg_lsn', prosrc => 'binary_upgrade_replorigin_advance' },
{ oid => '6505', descr => 'for use by pg_upgrade (conflict detection slot)',
proname => 'binary_upgrade_create_conflict_detection_slot',
proisstrict => 'f', provolatile => 'v', proparallel => 'u',
@@ -11973,6 +11969,10 @@
proname => 'binary_upgrade_set_next_pg_subscription_oid', provolatile => 'v',
proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
prosrc => 'binary_upgrade_set_next_pg_subscription_oid' },
+{ oid => '9161', descr => 'for use by pg_upgrade (replication origin)',
+ proname => 'binary_upgrade_create_replication_origin', proisstrict => 'f',
+ provolatile => 'v', proparallel => 'u', prorettype => 'void',
+ proargtypes => 'oid name pg_lsn', prosrc => 'binary_upgrade_create_replication_origin' },
# conversion functions
{ oid => '4310', descr => 'internal conversion function for KOI8R to WIN1251',
--
2.47.3
[application/octet-stream] v6-0001-Preserve-subscription-OIDs-during-pg_upgrade.patch (9.1K, ../../CAFPTHDaftSwzGTGbFEw8rwDBsub0XqcDm1wxQcquj-Y3PC2qrg@mail.gmail.com/3-v6-0001-Preserve-subscription-OIDs-during-pg_upgrade.patch)
download | inline diff:
From 9be4a488196d12cac3ccaba132b0efcef14a7a87 Mon Sep 17 00:00:00 2001
From: Ajin Cherian <[email protected]>
Date: Mon, 18 May 2026 18:41:05 +1000
Subject: [PATCH v6 1/2] Preserve subscription OIDs during pg_upgrade
Currently subscription OIDs can be changed when a cluster is upgraded
using pg_upgrade. This is required for a subsequent patch which will
preserve the replication oids after upgrade.
Author: Vignesh C <[email protected]>
---
src/backend/commands/subscriptioncmds.c | 25 +++++++++++++++++--
src/backend/utils/adt/pg_upgrade_support.c | 10 ++++++++
src/bin/pg_dump/pg_dump.c | 8 ++++++
src/bin/pg_upgrade/pg_upgrade.c | 3 +++
src/bin/pg_upgrade/t/004_subscription.pl | 7 ++++++
src/include/catalog/binary_upgrade.h | 1 +
src/include/catalog/pg_proc.dat | 4 +++
.../expected/spgist_name_ops.out | 6 +++--
8 files changed, 60 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 523959ba0ce..7c1f05a5fd5 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -83,6 +83,12 @@
/* check if the 'val' has 'bits' set */
#define IsSet(val, bits) (((val) & (bits)) == (bits))
+/*
+ * This will be set by the pg_upgrade_support function --
+ * binary_upgrade_set_next_pg_subscription_oid().
+ */
+Oid binary_upgrade_next_pg_subscription_oid = InvalidOid;
+
/*
* Structure to hold a bitmap representing the user-provided CREATE/ALTER
* SUBSCRIPTION command options and the parsed/default values of each of them.
@@ -772,8 +778,23 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
memset(values, 0, sizeof(values));
memset(nulls, false, sizeof(nulls));
- subid = GetNewOidWithIndex(rel, SubscriptionObjectIndexId,
- Anum_pg_subscription_oid);
+ /* Use binary-upgrade override for pg_subscription.oid? */
+ if (IsBinaryUpgrade)
+ {
+ if (!OidIsValid(binary_upgrade_next_pg_subscription_oid))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("pg_subscription OID value not set when in binary upgrade mode")));
+
+ subid = binary_upgrade_next_pg_subscription_oid;
+ binary_upgrade_next_pg_subscription_oid = InvalidOid;
+ }
+ else
+ {
+ subid = GetNewOidWithIndex(rel, SubscriptionObjectIndexId,
+ Anum_pg_subscription_oid);
+ }
+
values[Anum_pg_subscription_oid - 1] = ObjectIdGetDatum(subid);
values[Anum_pg_subscription_subdbid - 1] = ObjectIdGetDatum(MyDatabaseId);
values[Anum_pg_subscription_subskiplsn - 1] = LSNGetDatum(InvalidXLogRecPtr);
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index b505a6b4fee..59c3e7f0146 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -181,6 +181,16 @@ binary_upgrade_set_next_pg_authid_oid(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
+Datum
+binary_upgrade_set_next_pg_subscription_oid(PG_FUNCTION_ARGS)
+{
+ Oid subid = PG_GETARG_OID(0);
+
+ CHECK_IS_BINARY_UPGRADE;
+ binary_upgrade_next_pg_subscription_oid = subid;
+ PG_RETURN_VOID();
+}
+
Datum
binary_upgrade_create_empty_extension(PG_FUNCTION_ARGS)
{
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index d56dcc701ce..452d0b5e98a 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -5583,6 +5583,14 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
appendPQExpBuffer(delq, "DROP SUBSCRIPTION %s;\n",
qsubname);
+ if (dopt->binary_upgrade)
+ {
+ appendPQExpBufferStr(query, "\n-- For binary upgrade, must preserve pg_subscription.oid\n");
+ appendPQExpBuffer(query,
+ "SELECT pg_catalog.binary_upgrade_set_next_pg_subscription_oid('%u'::pg_catalog.oid);\n\n",
+ subinfo->dobj.catId.oid);
+ }
+
appendPQExpBuffer(query, "CREATE SUBSCRIPTION %s ",
qsubname);
if (subinfo->subservername)
diff --git a/src/bin/pg_upgrade/pg_upgrade.c b/src/bin/pg_upgrade/pg_upgrade.c
index 2127d297bfe..4e853096698 100644
--- a/src/bin/pg_upgrade/pg_upgrade.c
+++ b/src/bin/pg_upgrade/pg_upgrade.c
@@ -35,6 +35,9 @@
*
* We control all assignments of pg_database.oid because we want the directory
* names to match between the old and new cluster.
+ *
+ * We control assignment of pg_subscription.oid because we want the oid to
+ * match between the old and new cluster.
*/
diff --git a/src/bin/pg_upgrade/t/004_subscription.pl b/src/bin/pg_upgrade/t/004_subscription.pl
index c94a82deae0..646767f2a65 100644
--- a/src/bin/pg_upgrade/t/004_subscription.pl
+++ b/src/bin/pg_upgrade/t/004_subscription.pl
@@ -313,6 +313,9 @@ my $tab_upgraded1_oid = $old_sub->safe_psql('postgres',
my $tab_upgraded2_oid = $old_sub->safe_psql('postgres',
"SELECT oid FROM pg_class WHERE relname = 'tab_upgraded2'");
+$sub_oid = $old_sub->safe_psql('postgres',
+ "SELECT oid FROM pg_subscription ORDER BY subname");
+
$old_sub->stop;
# Change configuration so that initial table sync does not get started
@@ -359,6 +362,10 @@ $publisher->safe_psql(
$new_sub->start;
+# The subscription oid should be preserved
+$result = $new_sub->safe_psql('postgres', "SELECT oid FROM pg_subscription ORDER BY subname");
+is($result, qq($sub_oid), "subscription oid should have been preserved");
+
# The subscription's running status, failover option, and retain_dead_tuples
# option should be preserved in the upgraded instance. So regress_sub4 should
# still have subenabled, subfailover, and subretaindeadtuples set to true,
diff --git a/src/include/catalog/binary_upgrade.h b/src/include/catalog/binary_upgrade.h
index 7bf7ae44385..b15b18e7dc9 100644
--- a/src/include/catalog/binary_upgrade.h
+++ b/src/include/catalog/binary_upgrade.h
@@ -32,6 +32,7 @@ extern PGDLLIMPORT RelFileNumber binary_upgrade_next_toast_pg_class_relfilenumbe
extern PGDLLIMPORT Oid binary_upgrade_next_pg_enum_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_authid_oid;
+extern PGDLLIMPORT Oid binary_upgrade_next_pg_subscription_oid;
extern PGDLLIMPORT bool binary_upgrade_record_init_privs;
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index be157a5fbe9..3a28406981d 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -11969,6 +11969,10 @@
proisstrict => 'f', provolatile => 'v', proparallel => 'u',
prorettype => 'void', proargtypes => '',
prosrc => 'binary_upgrade_create_conflict_detection_slot' },
+{ oid => '9160', descr => 'for use by pg_upgrade',
+ proname => 'binary_upgrade_set_next_pg_subscription_oid', provolatile => 'v',
+ proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
+ prosrc => 'binary_upgrade_set_next_pg_subscription_oid' },
# conversion functions
{ oid => '4310', descr => 'internal conversion function for KOI8R to WIN1251',
diff --git a/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out b/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out
index 1ee65ede243..39d43368c42 100644
--- a/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out
+++ b/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out
@@ -59,11 +59,12 @@ select * from t
binary_upgrade_set_next_multirange_pg_type_oid | 1 | binary_upgrade_set_next_multirange_pg_type_oid
binary_upgrade_set_next_pg_authid_oid | | binary_upgrade_set_next_pg_authid_oid
binary_upgrade_set_next_pg_enum_oid | | binary_upgrade_set_next_pg_enum_oid
+ binary_upgrade_set_next_pg_subscription_oid | | binary_upgrade_set_next_pg_subscription_oid
binary_upgrade_set_next_pg_tablespace_oid | | binary_upgrade_set_next_pg_tablespace_oid
binary_upgrade_set_next_pg_type_oid | | binary_upgrade_set_next_pg_type_oid
binary_upgrade_set_next_toast_pg_class_oid | 1 | binary_upgrade_set_next_toast_pg_class_oid
binary_upgrade_set_next_toast_relfilenode | | binary_upgrade_set_next_toast_relfilenode
-(13 rows)
+(14 rows)
-- Verify clean failure when INCLUDE'd columns result in overlength tuple
-- The error message details are platform-dependent, so show only SQLSTATE
@@ -108,11 +109,12 @@ select * from t
binary_upgrade_set_next_multirange_pg_type_oid | 1 | binary_upgrade_set_next_multirange_pg_type_oid
binary_upgrade_set_next_pg_authid_oid | | binary_upgrade_set_next_pg_authid_oid
binary_upgrade_set_next_pg_enum_oid | | binary_upgrade_set_next_pg_enum_oid
+ binary_upgrade_set_next_pg_subscription_oid | | binary_upgrade_set_next_pg_subscription_oid
binary_upgrade_set_next_pg_tablespace_oid | | binary_upgrade_set_next_pg_tablespace_oid
binary_upgrade_set_next_pg_type_oid | | binary_upgrade_set_next_pg_type_oid
binary_upgrade_set_next_toast_pg_class_oid | 1 | binary_upgrade_set_next_toast_pg_class_oid
binary_upgrade_set_next_toast_relfilenode | | binary_upgrade_set_next_toast_relfilenode
-(13 rows)
+(14 rows)
\set VERBOSITY sqlstate
insert into t values(repeat('xyzzy', 12), 42, repeat('xyzzy', 4000));
--
2.47.3
^ permalink raw reply [nested|flat] 97+ messages in thread
* Re: [PATCH] Preserve replication origin OIDs in pg_upgrade
@ 2026-05-22 09:46 Shlok Kyal <[email protected]>
parent: Ajin Cherian <[email protected]>
0 siblings, 1 reply; 97+ messages in thread
From: Shlok Kyal @ 2026-05-22 09:46 UTC (permalink / raw)
To: Ajin Cherian <[email protected]>; +Cc: Zsolt Parragi <[email protected]>; vignesh C <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; PostgreSQL Hackers <[email protected]>
On Mon, 18 May 2026 at 16:13, Ajin Cherian <[email protected]> wrote:
>
> Rebased the patch as it was no longer applying.
>
Hi Ajin,
I have started reviewing the patch. Here is my comment for v6-0002 patch:
Suppose we have a replication setup: publisher -> subscriber
and we are upgrading subscriber to subscriber_new.
And if initially 'subscriber_new' has a replication origin, upgrading
the cluster can error out.
Example:
We set up a logical replication between publisher node and subscriber node.
On subscriber node:
postgres=# SELECT * FROM pg_replication_origin;
roident | roname
---------+----------
1 | pg_16393
(1 row)
And initially subscriber_new has a replication origin:
postgres=# select pg_replication_origin_create('myname');
pg_replication_origin_create
------------------------------
1
(1 row)
postgres=# SELECT * FROM pg_replication_origin;
roident | roname
---------+--------
1 | myname
(1 row)
Now, if we run pg_upgrade to upgrade subscriber node to subscriber_new
node, we get an error:
```
SELECT pg_catalog.binary_upgrade_create_replication_origin('1'::pg_catalog.oid,
'pg_16393'::pg_catalog.name, '0/01743078'::pg_catalog.pg_lsn);
psql:subscriber_new/pg_upgrade_output.d/20260522T140312.807/dump/pg_upgrade_dump_globals.sql:37:
ERROR: replication origin with ID 1 already exists
```
This error occurs in "Performing Upgrade" stage. Should we add a check
in the "Performing Consistency Checks" stage so that we don't need to
re-initdb the new cluster to perform the upgrade?
Maybe we can add a check similar to
check_new_cluster_replication_slots(), where pg_upgrade errors out if
the new cluster already contains replication origins. Thoughts?
Thanks,
Shlok Kyal
^ permalink raw reply [nested|flat] 97+ messages in thread
* Re: [PATCH] Preserve replication origin OIDs in pg_upgrade
@ 2026-05-22 10:27 shveta malik <[email protected]>
parent: Shlok Kyal <[email protected]>
0 siblings, 2 replies; 97+ messages in thread
From: shveta malik @ 2026-05-22 10:27 UTC (permalink / raw)
To: Shlok Kyal <[email protected]>; Ajin Cherian <[email protected]>; +Cc: Zsolt Parragi <[email protected]>; vignesh C <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; PostgreSQL Hackers <[email protected]>; shveta malik <[email protected]>
On Fri, May 22, 2026 at 3:16 PM Shlok Kyal <[email protected]> wrote:
>
> On Mon, 18 May 2026 at 16:13, Ajin Cherian <[email protected]> wrote:
> >
> > Rebased the patch as it was no longer applying.
> >
> Hi Ajin,
>
> I have started reviewing the patch. Here is my comment for v6-0002 patch:
>
> Suppose we have a replication setup: publisher -> subscriber
> and we are upgrading subscriber to subscriber_new.
> And if initially 'subscriber_new' has a replication origin, upgrading
> the cluster can error out.
>
> Example:
> We set up a logical replication between publisher node and subscriber node.
>
> On subscriber node:
> postgres=# SELECT * FROM pg_replication_origin;
> roident | roname
> ---------+----------
> 1 | pg_16393
> (1 row)
>
> And initially subscriber_new has a replication origin:
> postgres=# select pg_replication_origin_create('myname');
> pg_replication_origin_create
> ------------------------------
> 1
> (1 row)
>
> postgres=# SELECT * FROM pg_replication_origin;
> roident | roname
> ---------+--------
> 1 | myname
> (1 row)
>
> Now, if we run pg_upgrade to upgrade subscriber node to subscriber_new
> node, we get an error:
> ```
> SELECT pg_catalog.binary_upgrade_create_replication_origin('1'::pg_catalog.oid,
> 'pg_16393'::pg_catalog.name, '0/01743078'::pg_catalog.pg_lsn);
> psql:subscriber_new/pg_upgrade_output.d/20260522T140312.807/dump/pg_upgrade_dump_globals.sql:37:
> ERROR: replication origin with ID 1 already exists
> ```
>
> This error occurs in "Performing Upgrade" stage. Should we add a check
> in the "Performing Consistency Checks" stage so that we don't need to
> re-initdb the new cluster to perform the upgrade?
> Maybe we can add a check similar to
> check_new_cluster_replication_slots(), where pg_upgrade errors out if
> the new cluster already contains replication origins. Thoughts?
+1. I had the same thought while reviewing the patch today. We should
have it unless there is a reason we have avoided it??
Few trivial comments:
1)
+#include "access/skey.h"
+#include "catalog/indexing.h"
pg_upgrade_support.c compiles without above.
2)
+ Assert(!OidIsValid(rel->rd_rel->reltoastrelid));
Is there a reason for this sanity check? I generally do not see a
Null-Toast table sanity check after every table_open.
3)
+
+ /* Dump replication origins */
+ if (server_version >= 170000 && binary_upgrade && archDumpFormat == archNull)
+ dumpReplicationOrigins(conn);
why the check is for PG17 specifically?
thanks
Shveta
^ permalink raw reply [nested|flat] 97+ messages in thread
* Re: [PATCH] Preserve replication origin OIDs in pg_upgrade
@ 2026-05-25 07:13 shveta malik <[email protected]>
parent: shveta malik <[email protected]>
1 sibling, 0 replies; 97+ messages in thread
From: shveta malik @ 2026-05-25 07:13 UTC (permalink / raw)
To: Shlok Kyal <[email protected]>; Ajin Cherian <[email protected]>; +Cc: Zsolt Parragi <[email protected]>; vignesh C <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; PostgreSQL Hackers <[email protected]>; shveta malik <[email protected]>
On Fri, May 22, 2026 at 3:57 PM shveta malik <[email protected]> wrote:
>
> On Fri, May 22, 2026 at 3:16 PM Shlok Kyal <[email protected]> wrote:
> >
> > On Mon, 18 May 2026 at 16:13, Ajin Cherian <[email protected]> wrote:
> > >
> > > Rebased the patch as it was no longer applying.
> > >
> > Hi Ajin,
> >
> > I have started reviewing the patch. Here is my comment for v6-0002 patch:
> >
> > Suppose we have a replication setup: publisher -> subscriber
> > and we are upgrading subscriber to subscriber_new.
> > And if initially 'subscriber_new' has a replication origin, upgrading
> > the cluster can error out.
> >
> > Example:
> > We set up a logical replication between publisher node and subscriber node.
> >
> > On subscriber node:
> > postgres=# SELECT * FROM pg_replication_origin;
> > roident | roname
> > ---------+----------
> > 1 | pg_16393
> > (1 row)
> >
> > And initially subscriber_new has a replication origin:
> > postgres=# select pg_replication_origin_create('myname');
> > pg_replication_origin_create
> > ------------------------------
> > 1
> > (1 row)
> >
> > postgres=# SELECT * FROM pg_replication_origin;
> > roident | roname
> > ---------+--------
> > 1 | myname
> > (1 row)
> >
> > Now, if we run pg_upgrade to upgrade subscriber node to subscriber_new
> > node, we get an error:
> > ```
> > SELECT pg_catalog.binary_upgrade_create_replication_origin('1'::pg_catalog.oid,
> > 'pg_16393'::pg_catalog.name, '0/01743078'::pg_catalog.pg_lsn);
> > psql:subscriber_new/pg_upgrade_output.d/20260522T140312.807/dump/pg_upgrade_dump_globals.sql:37:
> > ERROR: replication origin with ID 1 already exists
> > ```
> >
> > This error occurs in "Performing Upgrade" stage. Should we add a check
> > in the "Performing Consistency Checks" stage so that we don't need to
> > re-initdb the new cluster to perform the upgrade?
> > Maybe we can add a check similar to
> > check_new_cluster_replication_slots(), where pg_upgrade errors out if
> > the new cluster already contains replication origins. Thoughts?
>
> +1. I had the same thought while reviewing the patch today. We should
> have it unless there is a reason we have avoided it??
>
> Few trivial comments:
>
> 1)
>
> +#include "access/skey.h"
> +#include "catalog/indexing.h"
>
> pg_upgrade_support.c compiles without above.
>
> 2)
> + Assert(!OidIsValid(rel->rd_rel->reltoastrelid));
>
> Is there a reason for this sanity check? I generally do not see a
> Null-Toast table sanity check after every table_open.
>
> 3)
>
> +
> + /* Dump replication origins */
> + if (server_version >= 170000 && binary_upgrade && archDumpFormat == archNull)
> + dumpReplicationOrigins(conn);
>
> why the check is for PG17 specifically?
>
One issue in 002:
binary_upgrade_create_replication_origin() has this:
+ originname = PG_GETARG_NAME(1);
+
+ roname_d = CStringGetTextDatum(NameStr(*originname));
+
We are getting origin-name (text) into Name-type which can not be more
than 64 bytes. So if an origin has name more than 64, it will end up
trimming the name post-upgrade.
I tried this:
Old-setup:
postgres=# SELECT
pg_replication_origin_create('this_is_a_very_long_replication_origin_name_that_exceeds_the_limit_of_64');
pg_replication_origin_create
------------------------------
1
postgres=# select * from pg_replication_origin;
roident | roname
---------+----------------------------------------------------------------------
1 | this_is_a_very_long_replication_origin_name_that_exceeds_the_limit_of_64
Post-upgrade: name got trimmed to 64 length.
-------------------------
postgres=# select * from pg_replication_origin;
roident | roname
---------+-----------------------------------------------------------------
1 | this_is_a_very_long_replication_origin_name_that_exceeds_the_li
thanks
Shveta
^ permalink raw reply [nested|flat] 97+ messages in thread
* Re: [PATCH] Preserve replication origin OIDs in pg_upgrade
@ 2026-05-28 02:48 Ajin Cherian <[email protected]>
parent: shveta malik <[email protected]>
1 sibling, 3 replies; 97+ messages in thread
From: Ajin Cherian @ 2026-05-28 02:48 UTC (permalink / raw)
To: shveta malik <[email protected]>; +Cc: Shlok Kyal <[email protected]>; Zsolt Parragi <[email protected]>; vignesh C <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; PostgreSQL Hackers <[email protected]>; shveta malik <[email protected]>
On Fri, May 22, 2026 at 8:27 PM shveta malik <[email protected]> wrote:
> On Fri, May 22, 2026 at 3:16 PM Shlok Kyal <[email protected]>
> wrote:
> >
> > On Mon, 18 May 2026 at 16:13, Ajin Cherian <[email protected]> wrote:
> > >
> > > Rebased the patch as it was no longer applying.
> > >
> > Hi Ajin,
> >
> > I have started reviewing the patch. Here is my comment for v6-0002 patch:
> >
> > Suppose we have a replication setup: publisher -> subscriber
> > and we are upgrading subscriber to subscriber_new.
> > And if initially 'subscriber_new' has a replication origin, upgrading
> > the cluster can error out.
> >
> > Example:
> > We set up a logical replication between publisher node and subscriber
> node.
> >
> > On subscriber node:
> > postgres=# SELECT * FROM pg_replication_origin;
> > roident | roname
> > ---------+----------
> > 1 | pg_16393
> > (1 row)
> >
> > And initially subscriber_new has a replication origin:
> > postgres=# select pg_replication_origin_create('myname');
> > pg_replication_origin_create
> > ------------------------------
> > 1
> > (1 row)
> >
> > postgres=# SELECT * FROM pg_replication_origin;
> > roident | roname
> > ---------+--------
> > 1 | myname
> > (1 row)
> >
> > Now, if we run pg_upgrade to upgrade subscriber node to subscriber_new
> > node, we get an error:
> > ```
> > SELECT
> pg_catalog.binary_upgrade_create_replication_origin('1'::pg_catalog.oid,
> > 'pg_16393'::pg_catalog.name, '0/01743078'::pg_catalog.pg_lsn);
> >
> psql:subscriber_new/pg_upgrade_output.d/20260522T140312.807/dump/pg_upgrade_dump_globals.sql:37:
> > ERROR: replication origin with ID 1 already exists
> > ```
> >
> > This error occurs in "Performing Upgrade" stage. Should we add a check
> > in the "Performing Consistency Checks" stage so that we don't need to
> > re-initdb the new cluster to perform the upgrade?
> > Maybe we can add a check similar to
> > check_new_cluster_replication_slots(), where pg_upgrade errors out if
> > the new cluster already contains replication origins. Thoughts?
>
> +1. I had the same thought while reviewing the patch today. We should
> have it unless there is a reason we have avoided it??
>
Fixed this. Added a new check for replication origins and if the new
cluster has any existing replication origins, then the check will fail.
>
> Few trivial comments:
>
> 1)
>
> +#include "access/skey.h"
> +#include "catalog/indexing.h"
>
> pg_upgrade_support.c compiles without above.
>
>
Removed.
> 2)
> + Assert(!OidIsValid(rel->rd_rel->reltoastrelid));
>
> Is there a reason for this sanity check? I generally do not see a
> Null-Toast table sanity check after every table_open.
>
>
Removed.
> 3)
>
> +
> + /* Dump replication origins */
> + if (server_version >= 170000 && binary_upgrade && archDumpFormat ==
> archNull)
> + dumpReplicationOrigins(conn);
>
> why the check is for PG17 specifically?
>
>
In PG17, we started migrating pg_subscription_rel and the remote LSN during
upgrades; prior to that, these were not migrated. Given that change, it
also makes sense to migrate replication origins from them. Otherwise, when
upgrading from PG17 to a later version, you could end up with a
subscription where pg_subscription_rel and the remote LSN are migrated, but
the corresponding replication origin is not created.
On Mon, May 25, 2026 at 5:13 PM shveta malik <[email protected]> wrote:
>
> One issue in 002:
>
> binary_upgrade_create_replication_origin() has this:
>
> + originname = PG_GETARG_NAME(1);
> +
> + roname_d = CStringGetTextDatum(NameStr(*originname));
> +
>
> We are getting origin-name (text) into Name-type which can not be more
> than 64 bytes. So if an origin has name more than 64, it will end up
> trimming the name post-upgrade.
>
> I tried this:
>
> Old-setup:
> postgres=# SELECT
>
> pg_replication_origin_create('this_is_a_very_long_replication_origin_name_that_exceeds_the_limit_of_64');
> pg_replication_origin_create
> ------------------------------
> 1
> postgres=# select * from pg_replication_origin;
> roident | roname
>
> ---------+----------------------------------------------------------------------
> 1 |
> this_is_a_very_long_replication_origin_name_that_exceeds_the_limit_of_64
>
>
> Post-upgrade: name got trimmed to 64 length.
> -------------------------
> postgres=# select * from pg_replication_origin;
> roident | roname
> ---------+-----------------------------------------------------------------
> 1 | this_is_a_very_long_replication_origin_name_that_exceeds_the_li
>
> thanks
> Shveta
Fixed this. Now binary_upgrade_create_replication_origin handles it
similarly to the way pg_replication_origin_create handles the name of the
origin.
Here's an updated version v7 containing these fixes.
regards,
Ajin Cherian
Fujitsu Australia
Attachments:
[application/octet-stream] v7-0001-Preserve-subscription-OIDs-during-pg_upgrade.patch (9.1K, ../../CAFPTHDavPyxsWjK0cRO3yOaP4u8FGYrOJuXJB-4wzneAY3H3Ug@mail.gmail.com/3-v7-0001-Preserve-subscription-OIDs-during-pg_upgrade.patch)
download | inline diff:
From 5c22e45a176695c16b8b22a764642327e666e868 Mon Sep 17 00:00:00 2001
From: Ajin Cherian <[email protected]>
Date: Wed, 27 May 2026 14:10:02 +1000
Subject: [PATCH v7 1/2] Preserve subscription OIDs during pg_upgrade
Currently subscription OIDs can be changed when a cluster is upgraded
using pg_upgrade. This is required for a subsequent patch which will
preserve the replication oids after upgrade.
Author: Vignesh C <[email protected]>
---
src/backend/commands/subscriptioncmds.c | 25 +++++++++++++++++--
src/backend/utils/adt/pg_upgrade_support.c | 10 ++++++++
src/bin/pg_dump/pg_dump.c | 8 ++++++
src/bin/pg_upgrade/pg_upgrade.c | 3 +++
src/bin/pg_upgrade/t/004_subscription.pl | 7 ++++++
src/include/catalog/binary_upgrade.h | 1 +
src/include/catalog/pg_proc.dat | 4 +++
.../expected/spgist_name_ops.out | 6 +++--
8 files changed, 60 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 523959ba0ce..7c1f05a5fd5 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -83,6 +83,12 @@
/* check if the 'val' has 'bits' set */
#define IsSet(val, bits) (((val) & (bits)) == (bits))
+/*
+ * This will be set by the pg_upgrade_support function --
+ * binary_upgrade_set_next_pg_subscription_oid().
+ */
+Oid binary_upgrade_next_pg_subscription_oid = InvalidOid;
+
/*
* Structure to hold a bitmap representing the user-provided CREATE/ALTER
* SUBSCRIPTION command options and the parsed/default values of each of them.
@@ -772,8 +778,23 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
memset(values, 0, sizeof(values));
memset(nulls, false, sizeof(nulls));
- subid = GetNewOidWithIndex(rel, SubscriptionObjectIndexId,
- Anum_pg_subscription_oid);
+ /* Use binary-upgrade override for pg_subscription.oid? */
+ if (IsBinaryUpgrade)
+ {
+ if (!OidIsValid(binary_upgrade_next_pg_subscription_oid))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("pg_subscription OID value not set when in binary upgrade mode")));
+
+ subid = binary_upgrade_next_pg_subscription_oid;
+ binary_upgrade_next_pg_subscription_oid = InvalidOid;
+ }
+ else
+ {
+ subid = GetNewOidWithIndex(rel, SubscriptionObjectIndexId,
+ Anum_pg_subscription_oid);
+ }
+
values[Anum_pg_subscription_oid - 1] = ObjectIdGetDatum(subid);
values[Anum_pg_subscription_subdbid - 1] = ObjectIdGetDatum(MyDatabaseId);
values[Anum_pg_subscription_subskiplsn - 1] = LSNGetDatum(InvalidXLogRecPtr);
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index b505a6b4fee..59c3e7f0146 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -181,6 +181,16 @@ binary_upgrade_set_next_pg_authid_oid(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
+Datum
+binary_upgrade_set_next_pg_subscription_oid(PG_FUNCTION_ARGS)
+{
+ Oid subid = PG_GETARG_OID(0);
+
+ CHECK_IS_BINARY_UPGRADE;
+ binary_upgrade_next_pg_subscription_oid = subid;
+ PG_RETURN_VOID();
+}
+
Datum
binary_upgrade_create_empty_extension(PG_FUNCTION_ARGS)
{
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index d56dcc701ce..452d0b5e98a 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -5583,6 +5583,14 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
appendPQExpBuffer(delq, "DROP SUBSCRIPTION %s;\n",
qsubname);
+ if (dopt->binary_upgrade)
+ {
+ appendPQExpBufferStr(query, "\n-- For binary upgrade, must preserve pg_subscription.oid\n");
+ appendPQExpBuffer(query,
+ "SELECT pg_catalog.binary_upgrade_set_next_pg_subscription_oid('%u'::pg_catalog.oid);\n\n",
+ subinfo->dobj.catId.oid);
+ }
+
appendPQExpBuffer(query, "CREATE SUBSCRIPTION %s ",
qsubname);
if (subinfo->subservername)
diff --git a/src/bin/pg_upgrade/pg_upgrade.c b/src/bin/pg_upgrade/pg_upgrade.c
index 2127d297bfe..4e853096698 100644
--- a/src/bin/pg_upgrade/pg_upgrade.c
+++ b/src/bin/pg_upgrade/pg_upgrade.c
@@ -35,6 +35,9 @@
*
* We control all assignments of pg_database.oid because we want the directory
* names to match between the old and new cluster.
+ *
+ * We control assignment of pg_subscription.oid because we want the oid to
+ * match between the old and new cluster.
*/
diff --git a/src/bin/pg_upgrade/t/004_subscription.pl b/src/bin/pg_upgrade/t/004_subscription.pl
index c94a82deae0..646767f2a65 100644
--- a/src/bin/pg_upgrade/t/004_subscription.pl
+++ b/src/bin/pg_upgrade/t/004_subscription.pl
@@ -313,6 +313,9 @@ my $tab_upgraded1_oid = $old_sub->safe_psql('postgres',
my $tab_upgraded2_oid = $old_sub->safe_psql('postgres',
"SELECT oid FROM pg_class WHERE relname = 'tab_upgraded2'");
+$sub_oid = $old_sub->safe_psql('postgres',
+ "SELECT oid FROM pg_subscription ORDER BY subname");
+
$old_sub->stop;
# Change configuration so that initial table sync does not get started
@@ -359,6 +362,10 @@ $publisher->safe_psql(
$new_sub->start;
+# The subscription oid should be preserved
+$result = $new_sub->safe_psql('postgres', "SELECT oid FROM pg_subscription ORDER BY subname");
+is($result, qq($sub_oid), "subscription oid should have been preserved");
+
# The subscription's running status, failover option, and retain_dead_tuples
# option should be preserved in the upgraded instance. So regress_sub4 should
# still have subenabled, subfailover, and subretaindeadtuples set to true,
diff --git a/src/include/catalog/binary_upgrade.h b/src/include/catalog/binary_upgrade.h
index 7bf7ae44385..b15b18e7dc9 100644
--- a/src/include/catalog/binary_upgrade.h
+++ b/src/include/catalog/binary_upgrade.h
@@ -32,6 +32,7 @@ extern PGDLLIMPORT RelFileNumber binary_upgrade_next_toast_pg_class_relfilenumbe
extern PGDLLIMPORT Oid binary_upgrade_next_pg_enum_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_authid_oid;
+extern PGDLLIMPORT Oid binary_upgrade_next_pg_subscription_oid;
extern PGDLLIMPORT bool binary_upgrade_record_init_privs;
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index be157a5fbe9..3a28406981d 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -11969,6 +11969,10 @@
proisstrict => 'f', provolatile => 'v', proparallel => 'u',
prorettype => 'void', proargtypes => '',
prosrc => 'binary_upgrade_create_conflict_detection_slot' },
+{ oid => '9160', descr => 'for use by pg_upgrade',
+ proname => 'binary_upgrade_set_next_pg_subscription_oid', provolatile => 'v',
+ proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
+ prosrc => 'binary_upgrade_set_next_pg_subscription_oid' },
# conversion functions
{ oid => '4310', descr => 'internal conversion function for KOI8R to WIN1251',
diff --git a/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out b/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out
index 1ee65ede243..39d43368c42 100644
--- a/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out
+++ b/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out
@@ -59,11 +59,12 @@ select * from t
binary_upgrade_set_next_multirange_pg_type_oid | 1 | binary_upgrade_set_next_multirange_pg_type_oid
binary_upgrade_set_next_pg_authid_oid | | binary_upgrade_set_next_pg_authid_oid
binary_upgrade_set_next_pg_enum_oid | | binary_upgrade_set_next_pg_enum_oid
+ binary_upgrade_set_next_pg_subscription_oid | | binary_upgrade_set_next_pg_subscription_oid
binary_upgrade_set_next_pg_tablespace_oid | | binary_upgrade_set_next_pg_tablespace_oid
binary_upgrade_set_next_pg_type_oid | | binary_upgrade_set_next_pg_type_oid
binary_upgrade_set_next_toast_pg_class_oid | 1 | binary_upgrade_set_next_toast_pg_class_oid
binary_upgrade_set_next_toast_relfilenode | | binary_upgrade_set_next_toast_relfilenode
-(13 rows)
+(14 rows)
-- Verify clean failure when INCLUDE'd columns result in overlength tuple
-- The error message details are platform-dependent, so show only SQLSTATE
@@ -108,11 +109,12 @@ select * from t
binary_upgrade_set_next_multirange_pg_type_oid | 1 | binary_upgrade_set_next_multirange_pg_type_oid
binary_upgrade_set_next_pg_authid_oid | | binary_upgrade_set_next_pg_authid_oid
binary_upgrade_set_next_pg_enum_oid | | binary_upgrade_set_next_pg_enum_oid
+ binary_upgrade_set_next_pg_subscription_oid | | binary_upgrade_set_next_pg_subscription_oid
binary_upgrade_set_next_pg_tablespace_oid | | binary_upgrade_set_next_pg_tablespace_oid
binary_upgrade_set_next_pg_type_oid | | binary_upgrade_set_next_pg_type_oid
binary_upgrade_set_next_toast_pg_class_oid | 1 | binary_upgrade_set_next_toast_pg_class_oid
binary_upgrade_set_next_toast_relfilenode | | binary_upgrade_set_next_toast_relfilenode
-(13 rows)
+(14 rows)
\set VERBOSITY sqlstate
insert into t values(repeat('xyzzy', 12), 42, repeat('xyzzy', 4000));
--
2.47.3
[application/octet-stream] v7-0002-Preserve-replication-origin-OIDs-during-pg_upgrad.patch (23.6K, ../../CAFPTHDavPyxsWjK0cRO3yOaP4u8FGYrOJuXJB-4wzneAY3H3Ug@mail.gmail.com/4-v7-0002-Preserve-replication-origin-OIDs-during-pg_upgrad.patch)
download | inline diff:
From 1c02466991377300c0ca819dca6ba627621d4eb1 Mon Sep 17 00:00:00 2001
From: Ajin Cherian <[email protected]>
Date: Thu, 28 May 2026 12:32:27 +1000
Subject: [PATCH v7 2/2] Preserve replication origin OIDs during pg_upgrade
When pg_upgrade migrates a subscriber, replication origin OIDs
(roident) can change across the upgrade. This is a problem because
commit-timestamp records embed roident and are copied directly from
the old cluster's pg_commit_ts directory, causing spurious
"update_origin_differs" conflicts after the upgrade.
Fix this by dumping replication origins as global objects via
pg_dumpall during binary upgrade, using a new function
binary_upgrade_create_replication_origin(oid, name, lsn) to recreate
each origin with its preserved roident and remote_lsn. To avoid
conflicts with this, CreateSubscription() skips replorigin_create()
in binary-upgrade mode since the origin is already created by the
time the subscription is restored.
Author: Ajin Cherian <[email protected]>
Reviewer: Hayato Kuroda (Fujitsu) <[email protected]>
Reviewer: Zsolt Parragi <[email protected]>
Reviewer: Shlok Kyal <[email protected]>
Reviewer: Shveta malik <[email protected]>
---
src/backend/commands/subscriptioncmds.c | 11 +-
src/backend/utils/adt/pg_upgrade_support.c | 155 +++++++++++++++------
src/bin/pg_dump/pg_dump.c | 38 ++---
src/bin/pg_dump/pg_dumpall.c | 64 +++++++++
src/bin/pg_upgrade/check.c | 55 +++++++-
src/bin/pg_upgrade/info.c | 9 ++
src/bin/pg_upgrade/pg_upgrade.h | 1 +
src/bin/pg_upgrade/t/004_subscription.pl | 42 +++++-
src/include/catalog/pg_proc.dat | 8 +-
9 files changed, 291 insertions(+), 92 deletions(-)
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 7c1f05a5fd5..b31e15256a2 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -19,6 +19,7 @@
#include "access/table.h"
#include "access/twophase.h"
#include "access/xact.h"
+#include "catalog/binary_upgrade.h"
#include "catalog/catalog.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
@@ -867,9 +868,15 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
* apply workers initialization, and to handle origin creation dynamically
* when tables are added to the subscription. It is not clear whether
* preventing creation of origins is worth additional complexity.
+ *
+ * In binary-upgrade mode, skip origin creation here. This is required to
+ * preserve the roident from the old cluster for this subscription.
*/
- ReplicationOriginNameForLogicalRep(subid, InvalidOid, originname, sizeof(originname));
- replorigin_create(originname);
+ if (!IsBinaryUpgrade)
+ {
+ ReplicationOriginNameForLogicalRep(subid, InvalidOid, originname, sizeof(originname));
+ replorigin_create(originname);
+ }
/*
* Connect to remote side to execute requested commands and fetch table
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index 59c3e7f0146..173bbdc764a 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -11,6 +11,7 @@
#include "postgres.h"
+#include "access/genam.h"
#include "access/relation.h"
#include "access/table.h"
#include "catalog/binary_upgrade.h"
@@ -27,8 +28,10 @@
#include "storage/lmgr.h"
#include "utils/array.h"
#include "utils/builtins.h"
+#include "utils/fmgroids.h"
#include "utils/lsyscache.h"
#include "utils/pg_lsn.h"
+#include "utils/snapmgr.h"
#define CHECK_IS_BINARY_UPGRADE \
@@ -377,71 +380,131 @@ binary_upgrade_add_sub_rel_state(PG_FUNCTION_ARGS)
}
/*
- * binary_upgrade_replorigin_advance
+ * binary_upgrade_create_conflict_detection_slot
*
- * Update the remote_lsn for the subscriber's replication origin.
+ * Create a replication slot to retain information necessary for conflict
+ * detection such as dead tuples, commit timestamps, and origins.
*/
Datum
-binary_upgrade_replorigin_advance(PG_FUNCTION_ARGS)
+binary_upgrade_create_conflict_detection_slot(PG_FUNCTION_ARGS)
{
- Relation rel;
- Oid subid;
- char *subname;
- char originname[NAMEDATALEN];
- ReplOriginId node;
- XLogRecPtr remote_commit;
-
CHECK_IS_BINARY_UPGRADE;
- /*
- * We must ensure a non-NULL subscription name before dereferencing the
- * arguments.
- */
- if (PG_ARGISNULL(0))
- elog(ERROR, "null argument to binary_upgrade_replorigin_advance is not allowed");
-
- subname = text_to_cstring(PG_GETARG_TEXT_PP(0));
- remote_commit = PG_ARGISNULL(1) ? InvalidXLogRecPtr : PG_GETARG_LSN(1);
-
- rel = table_open(SubscriptionRelationId, RowExclusiveLock);
- subid = get_subscription_oid(subname, false);
-
- ReplicationOriginNameForLogicalRep(subid, InvalidOid, originname, sizeof(originname));
-
- /* Lock to prevent the replication origin from vanishing */
- LockRelationOid(ReplicationOriginRelationId, RowExclusiveLock);
- node = replorigin_by_name(originname, false);
-
- /*
- * The server will be stopped after setting up the objects in the new
- * cluster and the origins will be flushed during the shutdown checkpoint.
- * This will ensure that the latest LSN values for origin will be
- * available after the upgrade.
- */
- replorigin_advance(node, remote_commit, InvalidXLogRecPtr,
- false /* backward */ ,
- false /* WAL log */ );
+ CreateConflictDetectionSlot();
- UnlockRelationOid(ReplicationOriginRelationId, RowExclusiveLock);
- table_close(rel, RowExclusiveLock);
+ ReplicationSlotRelease();
PG_RETURN_VOID();
}
/*
- * binary_upgrade_create_conflict_detection_slot
+ * binary_upgrade_create_replication_origin
*
- * Create a replication slot to retain information necessary for conflict
- * detection such as dead tuples, commit timestamps, and origins.
+ * Create a replication origin with a specific OID and name, optionally
+ * restoring its remote_lsn. Used by pg_upgrade to preserve replication
+ * origin OIDs across the upgrade.
*/
Datum
-binary_upgrade_create_conflict_detection_slot(PG_FUNCTION_ARGS)
+binary_upgrade_create_replication_origin(PG_FUNCTION_ARGS)
{
+ Oid node_oid;
+ ReplOriginId node;
+ char *originname;
+ Relation rel;
+ HeapTuple tuple;
+ Datum roname_d;
+ SysScanDesc scan;
+ ScanKeyData key;
+ bool nulls[Natts_pg_replication_origin];
+ Datum values[Natts_pg_replication_origin];
+ bool collides;
+
CHECK_IS_BINARY_UPGRADE;
- CreateConflictDetectionSlot();
+ if (PG_ARGISNULL(0) || PG_ARGISNULL(1))
+ elog(ERROR,
+ "null argument to binary_upgrade_create_replication_origin is not allowed");
+
+ node_oid = PG_GETARG_OID(0);
+
+ if (node_oid == InvalidOid || node_oid > PG_UINT16_MAX)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("replication origin ID %u is out of range", node_oid)));
+
+ node = (ReplOriginId) node_oid;
+ originname = text_to_cstring(PG_GETARG_TEXT_PP(1));
+
+ if (strlen(originname) > MAX_RONAME_LEN)
+ ereport(ERROR,
+ (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
+ errmsg("replication origin name is too long"),
+ errdetail("Replication origin names must be no longer than %d bytes.",
+ MAX_RONAME_LEN)));
+
+ roname_d = CStringGetTextDatum(originname);
+
+ Assert(IsTransactionState());
+
+ rel = table_open(ReplicationOriginRelationId, RowExclusiveLock);
+
+ /* Check for OID collision */
+ ScanKeyInit(&key,
+ Anum_pg_replication_origin_roident,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(node));
+ scan = systable_beginscan(rel, ReplicationOriginIdentIndex,
+ true /* indexOK */,
+ SnapshotSelf,
+ 1, &key);
+ collides = HeapTupleIsValid(systable_getnext(scan));
+ systable_endscan(scan);
+
+ if (collides)
+ ereport(ERROR,
+ (errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("replication origin with ID %u already exists", node_oid)));
+
+ /* Check for name collision */
+ ScanKeyInit(&key,
+ Anum_pg_replication_origin_roname,
+ BTEqualStrategyNumber, F_TEXTEQ,
+ roname_d);
+ scan = systable_beginscan(rel, ReplicationOriginNameIndex,
+ true /* indexOK */,
+ SnapshotSelf,
+ 1, &key);
+ collides = HeapTupleIsValid(systable_getnext(scan));
+ systable_endscan(scan);
+
+ if (collides)
+ ereport(ERROR,
+ (errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("replication origin \"%s\" already exists",
+ originname)));
+
+ memset(&nulls, 0, sizeof(nulls));
+ memset(&values, 0, sizeof(values));
+
+ values[Anum_pg_replication_origin_roident - 1] = ObjectIdGetDatum(node);
+ values[Anum_pg_replication_origin_roname - 1] = roname_d;
+
+ tuple = heap_form_tuple(RelationGetDescr(rel), values, nulls);
+ CatalogTupleInsert(rel, tuple);
+ heap_freetuple(tuple);
+ CommandCounterIncrement();
+
+ /* Restore the remote_lsn if provided, while still holding the lock */
+ if (!PG_ARGISNULL(2))
+ {
+ XLogRecPtr remote_commit = PG_GETARG_LSN(2);
- ReplicationSlotRelease();
+ replorigin_advance(node, remote_commit, InvalidXLogRecPtr,
+ false /* backward */,
+ false /* WAL log */);
+ }
+
+ table_close(rel, RowExclusiveLock);
PG_RETURN_VOID();
}
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 452d0b5e98a..a5fb2b42c3d 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -5668,37 +5668,15 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
* In binary-upgrade mode, we allow the replication to continue after the
* upgrade.
*/
- if (dopt->binary_upgrade && fout->remoteVersion >= 170000)
+ if (dopt->binary_upgrade && subinfo->subenabled && fout->remoteVersion >= 170000)
{
- if (subinfo->suboriginremotelsn)
- {
- /*
- * Preserve the remote_lsn for the subscriber's replication
- * origin. This value is required to start the replication from
- * the position before the upgrade. This value will be stale if
- * the publisher gets upgraded before the subscriber node.
- * However, this shouldn't be a problem as the upgrade of the
- * publisher ensures that all the transactions were replicated
- * before upgrading it.
- */
- appendPQExpBufferStr(query,
- "\n-- For binary upgrade, must preserve the remote_lsn for the subscriber's replication origin.\n");
- appendPQExpBufferStr(query,
- "SELECT pg_catalog.binary_upgrade_replorigin_advance(");
- appendStringLiteralAH(query, subinfo->dobj.name, fout);
- appendPQExpBuffer(query, ", '%s');\n", subinfo->suboriginremotelsn);
- }
-
- if (subinfo->subenabled)
- {
- /*
- * Enable the subscription to allow the replication to continue
- * after the upgrade.
- */
- appendPQExpBufferStr(query,
- "\n-- For binary upgrade, must preserve the subscriber's running state.\n");
- appendPQExpBuffer(query, "ALTER SUBSCRIPTION %s ENABLE;\n", qsubname);
- }
+ /*
+ * Enable the subscription to allow the replication to continue
+ * after the upgrade.
+ */
+ appendPQExpBufferStr(query,
+ "\n-- For binary upgrade, must preserve the subscriber's running state.\n");
+ appendPQExpBuffer(query, "ALTER SUBSCRIPTION %s ENABLE;\n", qsubname);
}
if (subinfo->dobj.dump & DUMP_COMPONENT_DEFINITION)
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index c1f43113c53..efa8da3e627 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -25,6 +25,7 @@
#include <time.h>
#include <unistd.h>
+#include "access/xlogdefs.h"
#include "catalog/pg_authid_d.h"
#include "common/connect.h"
#include "common/file_perm.h"
@@ -76,6 +77,7 @@ static void dropDBs(PGconn *conn);
static void dumpUserConfig(PGconn *conn, const char *username);
static void dumpDatabases(PGconn *conn);
static void dumpTimestamp(const char *msg);
+static void dumpReplicationOrigins(PGconn *conn);
static int runPgDump(const char *dbname, const char *create_opts, char *dbfile);
static void buildShSecLabels(PGconn *conn,
const char *catalog_name, Oid objectId,
@@ -813,6 +815,10 @@ main(int argc, char *argv[])
/* Dump role GUC privileges */
if (server_version >= 150000 && !skip_acls)
dumpRoleGUCPrivs(conn);
+
+ /* Dump replication origins */
+ if (server_version >= 170000 && binary_upgrade && archDumpFormat == archNull)
+ dumpReplicationOrigins(conn);
}
/* Dump tablespaces */
@@ -2339,6 +2345,64 @@ dumpTimestamp(const char *msg)
fprintf(OPF, "-- %s %s\n\n", msg, buf);
}
+static void
+dumpReplicationOrigins(PGconn *conn)
+{
+ PQExpBuffer buf = createPQExpBuffer();
+ PGresult *res;
+ int i_roident;
+ int i_roname;
+ int i_remotelsn;
+
+ /* Get replication origins from catalogs */
+ appendPQExpBufferStr(buf,
+ "SELECT o.*, os.remote_lsn "
+ "FROM pg_catalog.pg_replication_origin o "
+ "LEFT OUTER JOIN pg_catalog.pg_replication_origin_status os ON o.roident = os.local_id ");
+
+ res = executeQuery(conn, buf->data);
+
+ i_roident = PQfnumber(res, "roident");
+ i_roname = PQfnumber(res, "roname");
+ i_remotelsn = PQfnumber(res, "remote_lsn");
+
+ if (PQntuples(res) > 0)
+ fprintf(OPF, "--\n-- Replication Origins \n--\n\n");
+
+ for (int i = 0; i < PQntuples(res); i++)
+ {
+ ReplOriginId roident;
+ const char *roname;
+
+ roident = atooid(PQgetvalue(res, i, i_roident));
+ roname = PQgetvalue(res, i, i_roname);
+
+ resetPQExpBuffer(buf);
+
+ appendPQExpBufferStr(buf, "\n-- For binary upgrade, must preserve replication origin roident and remote_lsn\n");
+ appendPQExpBuffer(buf,
+ "SELECT pg_catalog.binary_upgrade_create_replication_origin("
+ "'%u'::pg_catalog.oid, ", roident);
+ appendStringLiteralConn(buf, roname, conn);
+ appendPQExpBufferStr(buf, "::pg_catalog.text");
+
+ if (!PQgetisnull(res, i, i_remotelsn))
+ {
+ appendPQExpBufferStr(buf, ", ");
+ appendStringLiteralConn(buf, PQgetvalue(res, i, i_remotelsn), conn);
+ appendPQExpBufferStr(buf, "::pg_catalog.pg_lsn");
+ }
+ else
+ appendPQExpBufferStr(buf, ", NULL");
+
+ appendPQExpBufferStr(buf, ");\n");
+ fprintf(OPF, "%s", buf->data);
+ }
+
+ PQclear(res);
+ destroyPQExpBuffer(buf);
+}
+
/*
* read_dumpall_filters - retrieve database identifier patterns from file
*
diff --git a/src/bin/pg_upgrade/check.c b/src/bin/pg_upgrade/check.c
index f5c93e611d2..303814fce9c 100644
--- a/src/bin/pg_upgrade/check.c
+++ b/src/bin/pg_upgrade/check.c
@@ -33,6 +33,7 @@ static void check_for_new_tablespace_dir(void);
static void check_for_user_defined_encoding_conversions(ClusterInfo *cluster);
static void check_for_unicode_update(ClusterInfo *cluster);
static void check_new_cluster_replication_slots(void);
+static void check_new_cluster_replication_origins(void);
static void check_new_cluster_subscription_configuration(void);
static void check_old_cluster_for_valid_slots(void);
static void check_old_cluster_subscription_state(void);
@@ -797,6 +798,8 @@ check_new_cluster(void)
check_new_cluster_replication_slots();
+ check_new_cluster_replication_origins();
+
check_new_cluster_subscription_configuration();
}
@@ -2302,12 +2305,50 @@ check_new_cluster_replication_slots(void)
check_ok();
}
+/*
+ * check_new_cluster_replication_origins()
+ *
+ * Verify that the new cluster has no replication origins. During upgrade,
+ * pg_upgrade restores replication origins from the old cluster with their
+ * original OIDs. If the new cluster already contains origins, those OIDs
+ * may collide, causing the upgrade to fail mid-way.
+ */
+static void
+check_new_cluster_replication_origins(void)
+{
+ PGconn *conn;
+ PGresult *res;
+ int norigins;
+
+ /* Quick return if there are no replication origins to migrate. */
+ if (old_cluster.nrepl_origins == 0)
+ return;
+
+ prep_status("Checking for replication origins in new cluster");
+
+ conn = connectToServer(&new_cluster, "template1");
+ res = executeQueryOrDie(conn,
+ "SELECT count(*) "
+ "FROM pg_catalog.pg_replication_origin");
+
+ if (PQntuples(res) != 1)
+ pg_fatal("could not count the number of replication origins");
+
+ norigins = atoi(PQgetvalue(res, 0, 0));
+ PQclear(res);
+ PQfinish(conn);
+
+ if (norigins > 0)
+ pg_fatal("expected 0 replication origins but found %d", norigins);
+
+ check_ok();
+}
+
/*
* check_new_cluster_subscription_configuration()
*
* Verify that the max_active_replication_origins configuration specified is
- * enough for creating the subscriptions. This is required to create the
- * replication origin for each subscription.
+ * enough for creating all the replication origins.
*/
static void
check_new_cluster_subscription_configuration(void)
@@ -2320,8 +2361,8 @@ check_new_cluster_subscription_configuration(void)
if (GET_MAJOR_VERSION(old_cluster.major_version) < 1700)
return;
- /* Quick return if there are no subscriptions to be migrated. */
- if (old_cluster.nsubs == 0)
+ /* Quick return if there are no replication origins to be migrated. */
+ if (old_cluster.nrepl_origins == 0)
return;
prep_status("Checking new cluster configuration for subscriptions");
@@ -2335,10 +2376,10 @@ check_new_cluster_subscription_configuration(void)
pg_fatal("could not determine parameter settings on new cluster");
max_active_replication_origins = atoi(PQgetvalue(res, 0, 0));
- if (old_cluster.nsubs > max_active_replication_origins)
+ if (old_cluster.nrepl_origins > max_active_replication_origins)
pg_fatal("\"max_active_replication_origins\" (%d) must be greater than or equal to the number of "
- "subscriptions (%d) on the old cluster",
- max_active_replication_origins, old_cluster.nsubs);
+ "replication origins (%d) on the old cluster",
+ max_active_replication_origins, old_cluster.nrepl_origins);
PQclear(res);
PQfinish(conn);
diff --git a/src/bin/pg_upgrade/info.c b/src/bin/pg_upgrade/info.c
index 37fff93892f..630f3f06e24 100644
--- a/src/bin/pg_upgrade/info.c
+++ b/src/bin/pg_upgrade/info.c
@@ -843,6 +843,7 @@ get_subscription_info(ClusterInfo *cluster)
PGconn *conn;
PGresult *res;
int i_nsub;
+ int i_nrepl_origins;
int i_retain_dead_tuples;
conn = connectToServer(cluster, "template1");
@@ -862,6 +863,14 @@ get_subscription_info(ClusterInfo *cluster)
cluster->sub_retain_dead_tuples = (strcmp(PQgetvalue(res, 0, i_retain_dead_tuples), "t") == 0);
PQclear(res);
+
+ res = executeQueryOrDie(conn,
+ "SELECT count(*) AS nrepl_origins "
+ "FROM pg_catalog.pg_replication_origin");
+ i_nrepl_origins = PQfnumber(res, "nrepl_origins");
+ cluster->nrepl_origins = atoi(PQgetvalue(res, 0, i_nrepl_origins));
+ PQclear(res);
+
PQfinish(conn);
}
diff --git a/src/bin/pg_upgrade/pg_upgrade.h b/src/bin/pg_upgrade/pg_upgrade.h
index ccd1ac0d013..77e7ca1b4cd 100644
--- a/src/bin/pg_upgrade/pg_upgrade.h
+++ b/src/bin/pg_upgrade/pg_upgrade.h
@@ -311,6 +311,7 @@ typedef struct
int num_tablespaces;
const char *tablespace_suffix; /* directory specification */
int nsubs; /* number of subscriptions */
+ int nrepl_origins; /* number of replication origins */
bool sub_retain_dead_tuples; /* whether a subscription enables
* retain_dead_tuples. */
} ClusterInfo;
diff --git a/src/bin/pg_upgrade/t/004_subscription.pl b/src/bin/pg_upgrade/t/004_subscription.pl
index 646767f2a65..e8b11d39dd0 100644
--- a/src/bin/pg_upgrade/t/004_subscription.pl
+++ b/src/bin/pg_upgrade/t/004_subscription.pl
@@ -42,7 +42,7 @@ my $connstr = $publisher->connstr . ' dbname=postgres';
# ------------------------------------------------------
# Check that pg_upgrade fails when max_active_replication_origins configured
-# in the new cluster is less than the number of subscriptions in the old
+# in the new cluster is less than the number of replication origins in the old
# cluster.
# ------------------------------------------------------
# It is sufficient to use disabled subscription to test upgrade failure.
@@ -74,7 +74,7 @@ command_checks_all(
],
1,
[
- qr/"max_active_replication_origins" \(0\) must be greater than or equal to the number of subscriptions \(1\) on the old cluster/
+ qr/"max_active_replication_origins" \(0\) must be greater than or equal to the number of replication origins \(1\) on the old cluster/
],
[qr//],
'run of pg_upgrade where the new cluster has insufficient max_active_replication_origins'
@@ -301,8 +301,30 @@ is($result, qq(t), "Check that the table is in init state");
# Get the replication origin's remote_lsn of the old subscriber
my $remote_lsn = $old_sub->safe_psql('postgres',
- "SELECT remote_lsn FROM pg_replication_origin_status os, pg_subscription s WHERE os.external_id = 'pg_' || s.oid AND s.subname = 'regress_sub4'"
+ "SELECT os.remote_lsn
+ FROM pg_replication_origin_status os
+ JOIN pg_replication_origin o ON o.roident = os.local_id
+ JOIN pg_subscription s ON o.roname = 'pg_' || s.oid::text
+ WHERE s.subname = 'regress_sub4'"
);
+
+# Get the replication origin OIDs (roident) for all subscriptions, keyed by
+# subscription name (which is stable across upgrade, unlike suboid). These
+# must be preserved after upgrade. A mismatch would cause spurious
+# update_origin_differs conflicts.
+my %pre_upgrade_roident;
+my $roident_rows = $old_sub->safe_psql('postgres',
+ "SELECT s.subname, o.roident
+ FROM pg_subscription s
+ JOIN pg_replication_origin o ON o.roname = 'pg_' || s.oid::text
+ ORDER BY s.subname"
+);
+for my $row (split /\n/, $roident_rows)
+{
+ my ($subname, $roident) = split /\|/, $row;
+ $pre_upgrade_roident{$subname} = $roident;
+}
+
# Have the subscription in disabled state before upgrade
$old_sub->safe_psql('postgres', "ALTER SUBSCRIPTION regress_sub5 DISABLE");
@@ -378,6 +400,20 @@ regress_sub5|f|f|f),
"check that the subscription's running status, failover, and retain_dead_tuples are preserved"
);
+# Verify that replication origin OIDs are preserved after upgrade.
+my $post_roident_rows = $new_sub->safe_psql('postgres',
+ "SELECT s.subname, o.roident
+ FROM pg_subscription s
+ JOIN pg_replication_origin o ON o.roname = 'pg_' || s.oid::text
+ ORDER BY s.subname"
+);
+for my $row (split /\n/, $post_roident_rows)
+{
+ my ($subname, $roident) = split /\|/, $row;
+ is($roident, $pre_upgrade_roident{$subname},
+ "roident preserved for subscription '$subname' after upgrade");
+}
+
# Subscription relations should be preserved
$result = $new_sub->safe_psql('postgres',
"SELECT srrelid, srsubstate FROM pg_subscription_rel ORDER BY srrelid");
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 3a28406981d..21466d926f1 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -11960,10 +11960,6 @@
provolatile => 'v', proparallel => 'u', prorettype => 'void',
proargtypes => 'text oid char pg_lsn',
prosrc => 'binary_upgrade_add_sub_rel_state' },
-{ oid => '6320', descr => 'for use by pg_upgrade (remote_lsn for origin)',
- proname => 'binary_upgrade_replorigin_advance', proisstrict => 'f',
- provolatile => 'v', proparallel => 'u', prorettype => 'void',
- proargtypes => 'text pg_lsn', prosrc => 'binary_upgrade_replorigin_advance' },
{ oid => '6505', descr => 'for use by pg_upgrade (conflict detection slot)',
proname => 'binary_upgrade_create_conflict_detection_slot',
proisstrict => 'f', provolatile => 'v', proparallel => 'u',
@@ -11973,6 +11969,10 @@
proname => 'binary_upgrade_set_next_pg_subscription_oid', provolatile => 'v',
proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
prosrc => 'binary_upgrade_set_next_pg_subscription_oid' },
+{ oid => '9161', descr => 'for use by pg_upgrade (replication origin)',
+ proname => 'binary_upgrade_create_replication_origin', proisstrict => 'f',
+ provolatile => 'v', proparallel => 'u', prorettype => 'void',
+ proargtypes => 'oid text pg_lsn', prosrc => 'binary_upgrade_create_replication_origin' },
# conversion functions
{ oid => '4310', descr => 'internal conversion function for KOI8R to WIN1251',
--
2.47.3
^ permalink raw reply [nested|flat] 97+ messages in thread
* Re: [PATCH] Preserve replication origin OIDs in pg_upgrade
@ 2026-06-05 10:09 vignesh C <[email protected]>
parent: Ajin Cherian <[email protected]>
2 siblings, 0 replies; 97+ messages in thread
From: vignesh C @ 2026-06-05 10:09 UTC (permalink / raw)
To: Ajin Cherian <[email protected]>; +Cc: shveta malik <[email protected]>; Shlok Kyal <[email protected]>; Zsolt Parragi <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; PostgreSQL Hackers <[email protected]>; shveta malik <[email protected]>
On Thu, 28 May 2026 at 08:18, Ajin Cherian <[email protected]> wrote:
>
> Fixed this. Added a new check for replication origins and if the new cluster has any existing replication origins, then the check will fail.
>>
>>
>> Few trivial comments:
>>
>> 1)
>>
>> +#include "access/skey.h"
>> +#include "catalog/indexing.h"
>>
>> pg_upgrade_support.c compiles without above.
>>
>
> Removed.
>
>>
>> 2)
>> + Assert(!OidIsValid(rel->rd_rel->reltoastrelid));
>>
>> Is there a reason for this sanity check? I generally do not see a
>> Null-Toast table sanity check after every table_open.
>>
>
> Removed.
>
>>
>> 3)
>>
>> +
>> + /* Dump replication origins */
>> + if (server_version >= 170000 && binary_upgrade && archDumpFormat == archNull)
>> + dumpReplicationOrigins(conn);
>>
>> why the check is for PG17 specifically?
>>
>
> In PG17, we started migrating pg_subscription_rel and the remote LSN during upgrades; prior to that, these were not migrated. Given that change, it also makes sense to migrate replication origins from them. Otherwise, when upgrading from PG17 to a later version, you could end up with a subscription where pg_subscription_rel and the remote LSN are migrated, but the corresponding replication origin is not created.
>
>
>
> On Mon, May 25, 2026 at 5:13 PM shveta malik <[email protected]> wrote:
>>
>>
>> One issue in 002:
>>
>> binary_upgrade_create_replication_origin() has this:
>>
>> + originname = PG_GETARG_NAME(1);
>> +
>> + roname_d = CStringGetTextDatum(NameStr(*originname));
>> +
>>
>> We are getting origin-name (text) into Name-type which can not be more
>> than 64 bytes. So if an origin has name more than 64, it will end up
>> trimming the name post-upgrade.
>>
>> I tried this:
>>
>> Old-setup:
>> postgres=# SELECT
>> pg_replication_origin_create('this_is_a_very_long_replication_origin_name_that_exceeds_the_limit_of_64');
>> pg_replication_origin_create
>> ------------------------------
>> 1
>> postgres=# select * from pg_replication_origin;
>> roident | roname
>> ---------+----------------------------------------------------------------------
>> 1 | this_is_a_very_long_replication_origin_name_that_exceeds_the_limit_of_64
>>
>>
>> Post-upgrade: name got trimmed to 64 length.
>> -------------------------
>> postgres=# select * from pg_replication_origin;
>> roident | roname
>> ---------+-----------------------------------------------------------------
>> 1 | this_is_a_very_long_replication_origin_name_that_exceeds_the_li
>>
>> thanks
>> Shveta
>
>
> Fixed this. Now binary_upgrade_create_replication_origin handles it similarly to the way pg_replication_origin_create handles the name of the origin.
Few comments:
1) pg_dumpall dumps all replication origins, not just subscription
origins and recreates every origin:
SELECT o.*, os.remote_lsn
FROM pg_replication_origin o
LEFT JOIN pg_replication_origin_status os
ON o.roident = os.local_id
Is pg_upgrade expected to preserve user-created origins?
If so, you can mention this in the commit message
2) Should the table lock be released here, if so should we mention
comments similar to how it is mentioned in
binary_upgarde_add_sub_rel_state:
+ /* Restore the remote_lsn if provided, while still holding the lock */
+ if (!PG_ARGISNULL(2))
+ {
+ XLogRecPtr remote_commit = PG_GETARG_LSN(2);
- ReplicationSlotRelease();
+ replorigin_advance(node, remote_commit, InvalidXLogRecPtr,
+ false /* backward */,
+ false /* WAL log */);
+ }
+
+ table_close(rel, RowExclusiveLock);
3) We can use new style of ereport to exclude '(' before errcode:
3.a)
+ ereport(ERROR,
+ (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
+ errmsg("replication origin name is too long"),
+ errdetail("Replication origin names must be no longer than %d bytes.",
+ MAX_RONAME_LEN)));
3.b) Similarly here:
+ ereport(ERROR,
+ (errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("replication origin with ID %u already exists", node_oid)));
3.c) There are many others, you can search other ereport in the patch
and change it too.
4) Since only one column will be returned here, can we use
"PQgetvalue(res, 0, 0) and remove PQfnumber:
+ res = executeQueryOrDie(conn,
+ "SELECT count(*) AS nrepl_origins "
+ "FROM pg_catalog.pg_replication_origin");
+ i_nrepl_origins = PQfnumber(res, "nrepl_origins");
+ cluster->nrepl_origins = atoi(PQgetvalue(res, 0, i_nrepl_origins));
+ PQclear(res);
5) replorigin_create also has some duplicate code like below, will it
be possible to have a common function so that both of them can use?
/* Check for name collision */
+ ScanKeyInit(&key,
+ Anum_pg_replication_origin_roname,
+ BTEqualStrategyNumber, F_TEXTEQ,
+ roname_d);
+ scan = systable_beginscan(rel, ReplicationOriginNameIndex,
+ true /* indexOK */,
+ SnapshotSelf,
+ 1, &key);
+ collides = HeapTupleIsValid(systable_getnext(scan));
+ systable_endscan(scan);
+
+ if (collides)
+ ereport(ERROR,
+ (errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("replication origin \"%s\" already exists",
+ originname)));
+
+ memset(&nulls, 0, sizeof(nulls));
+ memset(&values, 0, sizeof(values));
+
+ values[Anum_pg_replication_origin_roident - 1] = ObjectIdGetDatum(node);
+ values[Anum_pg_replication_origin_roname - 1] = roname_d;
+
+ tuple = heap_form_tuple(RelationGetDescr(rel), values, nulls);
+ CatalogTupleInsert(rel, tuple);
+ heap_freetuple(tuple);
+ CommandCounterIncrement();
Regards,
Vignesh
^ permalink raw reply [nested|flat] 97+ messages in thread
* Re: [PATCH] Preserve replication origin OIDs in pg_upgrade
@ 2026-06-08 17:06 Shlok Kyal <[email protected]>
parent: Ajin Cherian <[email protected]>
2 siblings, 3 replies; 97+ messages in thread
From: Shlok Kyal @ 2026-06-08 17:06 UTC (permalink / raw)
To: Ajin Cherian <[email protected]>; +Cc: shveta malik <[email protected]>; Zsolt Parragi <[email protected]>; vignesh C <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; PostgreSQL Hackers <[email protected]>; shveta malik <[email protected]>
On Thu, 28 May 2026 at 08:18, Ajin Cherian <[email protected]> wrote:
>
>
>
> On Fri, May 22, 2026 at 8:27 PM shveta malik <[email protected]> wrote:
>>
>> On Fri, May 22, 2026 at 3:16 PM Shlok Kyal <[email protected]> wrote:
>> >
>> > On Mon, 18 May 2026 at 16:13, Ajin Cherian <[email protected]> wrote:
>> > >
>> > > Rebased the patch as it was no longer applying.
>> > >
>> > Hi Ajin,
>> >
>> > I have started reviewing the patch. Here is my comment for v6-0002 patch:
>> >
>> > Suppose we have a replication setup: publisher -> subscriber
>> > and we are upgrading subscriber to subscriber_new.
>> > And if initially 'subscriber_new' has a replication origin, upgrading
>> > the cluster can error out.
>> >
>> > Example:
>> > We set up a logical replication between publisher node and subscriber node.
>> >
>> > On subscriber node:
>> > postgres=# SELECT * FROM pg_replication_origin;
>> > roident | roname
>> > ---------+----------
>> > 1 | pg_16393
>> > (1 row)
>> >
>> > And initially subscriber_new has a replication origin:
>> > postgres=# select pg_replication_origin_create('myname');
>> > pg_replication_origin_create
>> > ------------------------------
>> > 1
>> > (1 row)
>> >
>> > postgres=# SELECT * FROM pg_replication_origin;
>> > roident | roname
>> > ---------+--------
>> > 1 | myname
>> > (1 row)
>> >
>> > Now, if we run pg_upgrade to upgrade subscriber node to subscriber_new
>> > node, we get an error:
>> > ```
>> > SELECT pg_catalog.binary_upgrade_create_replication_origin('1'::pg_catalog.oid,
>> > 'pg_16393'::pg_catalog.name, '0/01743078'::pg_catalog.pg_lsn);
>> > psql:subscriber_new/pg_upgrade_output.d/20260522T140312.807/dump/pg_upgrade_dump_globals.sql:37:
>> > ERROR: replication origin with ID 1 already exists
>> > ```
>> >
>> > This error occurs in "Performing Upgrade" stage. Should we add a check
>> > in the "Performing Consistency Checks" stage so that we don't need to
>> > re-initdb the new cluster to perform the upgrade?
>> > Maybe we can add a check similar to
>> > check_new_cluster_replication_slots(), where pg_upgrade errors out if
>> > the new cluster already contains replication origins. Thoughts?
>>
>> +1. I had the same thought while reviewing the patch today. We should
>> have it unless there is a reason we have avoided it??
>
>
> Fixed this. Added a new check for replication origins and if the new cluster has any existing replication origins, then the check will fail.
>>
>>
>> Few trivial comments:
>>
>> 1)
>>
>> +#include "access/skey.h"
>> +#include "catalog/indexing.h"
>>
>> pg_upgrade_support.c compiles without above.
>>
>
> Removed.
>
>>
>> 2)
>> + Assert(!OidIsValid(rel->rd_rel->reltoastrelid));
>>
>> Is there a reason for this sanity check? I generally do not see a
>> Null-Toast table sanity check after every table_open.
>>
>
> Removed.
>
>>
>> 3)
>>
>> +
>> + /* Dump replication origins */
>> + if (server_version >= 170000 && binary_upgrade && archDumpFormat == archNull)
>> + dumpReplicationOrigins(conn);
>>
>> why the check is for PG17 specifically?
>>
>
> In PG17, we started migrating pg_subscription_rel and the remote LSN during upgrades; prior to that, these were not migrated. Given that change, it also makes sense to migrate replication origins from them. Otherwise, when upgrading from PG17 to a later version, you could end up with a subscription where pg_subscription_rel and the remote LSN are migrated, but the corresponding replication origin is not created.
>
>
>
> On Mon, May 25, 2026 at 5:13 PM shveta malik <[email protected]> wrote:
>>
>>
>> One issue in 002:
>>
>> binary_upgrade_create_replication_origin() has this:
>>
>> + originname = PG_GETARG_NAME(1);
>> +
>> + roname_d = CStringGetTextDatum(NameStr(*originname));
>> +
>>
>> We are getting origin-name (text) into Name-type which can not be more
>> than 64 bytes. So if an origin has name more than 64, it will end up
>> trimming the name post-upgrade.
>>
>> I tried this:
>>
>> Old-setup:
>> postgres=# SELECT
>> pg_replication_origin_create('this_is_a_very_long_replication_origin_name_that_exceeds_the_limit_of_64');
>> pg_replication_origin_create
>> ------------------------------
>> 1
>> postgres=# select * from pg_replication_origin;
>> roident | roname
>> ---------+----------------------------------------------------------------------
>> 1 | this_is_a_very_long_replication_origin_name_that_exceeds_the_limit_of_64
>>
>>
>> Post-upgrade: name got trimmed to 64 length.
>> -------------------------
>> postgres=# select * from pg_replication_origin;
>> roident | roname
>> ---------+-----------------------------------------------------------------
>> 1 | this_is_a_very_long_replication_origin_name_that_exceeds_the_li
>>
>> thanks
>> Shveta
>
>
> Fixed this. Now binary_upgrade_create_replication_origin handles it similarly to the way pg_replication_origin_create handles the name of the origin.
>
> Here's an updated version v7 containing these fixes.
Hi Ajin,
I have reviewed the patch and here are some comments:
1. I want to clarify, what is the expected behaviour when we upgrade
the postgres instance from version <= 16 to "HEAD + patch"?
- ReplicationOriginNameForLogicalRep(subid, InvalidOid, originname,
sizeof(originname));
- replorigin_create(originname);
+ if (!IsBinaryUpgrade)
+ {
+ ReplicationOriginNameForLogicalRep(subid, InvalidOid,
originname, sizeof(originname));
+ replorigin_create(originname);
+ }
and
+ /* Dump replication origins */
+ if (server_version >= 170000 && binary_upgrade &&
archDumpFormat == archNull)
+ dumpReplicationOrigins(conn);
When the postgres instance with subscription is updated from PG 16 (or
less) to HEAD, the replication origin is created as we do not have "if
(!IsBinaryUpgrade)" check in HEAD,
whereas when the similar instance is upgraded to "HEAD + patch", no
replication origin is present after the upgrade.
Is this difference in the behaviour between HEAD and "HEAD + patch" expected?
2.a) Can we replace:
+ /* Check for OID collision */
+ ScanKeyInit(&key,
+ Anum_pg_replication_origin_roident,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(node));
+ scan = systable_beginscan(rel, ReplicationOriginIdentIndex,
+ true /* indexOK */,
+ SnapshotSelf,
+ 1, &key);
+ collides = HeapTupleIsValid(systable_getnext(scan));
+ systable_endscan(scan);
+
+ if (collides)
+ ereport(ERROR,
+ (errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("replication origin with ID %u already
exists", node_oid)));
with:
if (SearchSysCacheExists1(REPLORIGIDENT, ObjectIdGetDatum(node)))
ereport(ERROR, errcode(ERRCODE_DUPLICATE_OBJECT),
errmsg("replication origin with ID %u already
exists", node_oid));
2.b) Similarly, Can we replace:
+ /* Check for name collision */
+ ScanKeyInit(&key,
+ Anum_pg_replication_origin_roname,
+ BTEqualStrategyNumber, F_TEXTEQ,
+ roname_d);
+ scan = systable_beginscan(rel, ReplicationOriginNameIndex,
+ true /* indexOK */,
+ SnapshotSelf,
+ 1, &key);
+ collides = HeapTupleIsValid(systable_getnext(scan));
+ systable_endscan(scan);
+
+ if (collides)
+ ereport(ERROR,
+ (errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("replication origin \"%s\" already exists",
+ originname)));
with:
if (SearchSysCacheExists1(REPLORIGNAME, roname_d))
ereport(ERROR, errcode(ERRCODE_DUPLICATE_OBJECT),
errmsg("replication origin \"%s\" already
exists", originname));
This will make the code smaller and easy to read. Thoughts?
Thanks,
Shlok Kyal
^ permalink raw reply [nested|flat] 97+ messages in thread
* Re: [PATCH] Preserve replication origin OIDs in pg_upgrade
@ 2026-06-09 04:13 shveta malik <[email protected]>
parent: Ajin Cherian <[email protected]>
2 siblings, 0 replies; 97+ messages in thread
From: shveta malik @ 2026-06-09 04:13 UTC (permalink / raw)
To: Ajin Cherian <[email protected]>; +Cc: Shlok Kyal <[email protected]>; Zsolt Parragi <[email protected]>; vignesh C <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; PostgreSQL Hackers <[email protected]>; shveta malik <[email protected]>
On Thu, May 28, 2026 at 8:18 AM Ajin Cherian <[email protected]> wrote:
>
>
>
>> 3)
>>
>> +
>> + /* Dump replication origins */
>> + if (server_version >= 170000 && binary_upgrade && archDumpFormat == archNull)
>> + dumpReplicationOrigins(conn);
>>
>> why the check is for PG17 specifically?
>>
>
> In PG17, we started migrating pg_subscription_rel and the remote LSN during upgrades; prior to that, these were not migrated. Given that change, it also makes sense to migrate replication origins from them. Otherwise, when upgrading from PG17 to a later version, you could end up with a subscription where pg_subscription_rel and the remote LSN are migrated, but the corresponding replication origin is not created.
>
Okay, please add a comment in code for this.
Please find a few comments on v007:
1)
+ node = (ReplOriginId) node_oid;
+ originname = text_to_cstring(PG_GETARG_TEXT_PP(1));
+
+ if (strlen(originname) > MAX_RONAME_LEN)
+ ereport(ERROR,
+ (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
+ errmsg("replication origin name is too long"),
+ errdetail("Replication origin names must be no longer than %d bytes.",
+ MAX_RONAME_LEN)));
+
+ roname_d = CStringGetTextDatum(originname);
CStringGetTextDatum is:
#define CStringGetTextDatum(s) PointerGetDatum(cstring_to_text(s))
---------
We are first converting text to C-string , then Cstring to text and
then getting Datum. Double allocation and double conversion. Can we
please try this instead and verify if it works:
text *origin_text = PG_GETARG_TEXT_PP(1);
originname = text_to_cstring(origin_text);
roname_d = PointerGetDatum(origin_text);
2)
+check_new_cluster_replication_origins(void)
+{
+ PGconn *conn;
+ PGresult *res;
+ int norigins;
+
+ /* Quick return if there are no replication origins to migrate. */
+ if (old_cluster.nrepl_origins == 0)
+ return;
Don't we need a version check simialr to what we have introduced in
pg_dumpall for origins? (Similar to
check_new_cluster_replication_slots as well)
3)
Since we have introduced check_new_cluster_replication_origins, do we
even need below checks in binary_upgrade_create_replication_origin()?
In which conditions will they be hit?
+ /* Check for OID collision */
+ ....
+ if (collides)
+ ereport(ERROR,
+ (errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("replication origin with ID %u already exists", node_oid)));
+
+ /* Check for name collision */
+ ....
+ if (collides)
+ ereport(ERROR,
+ (errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("replication origin \"%s\" already exists",
+ originname)));
4)
Now since check_new_cluster_subscription_configuration() is purely
about checking max-origin GUC configuration, I think the name is
misleading. IMO, we should merge it to
check_new_cluster_replication_origins(). See how
check_new_cluster_replication_slots() does it: checking both
new-cluster's count and the max-configuration for slots.
thanks
Shveta
^ permalink raw reply [nested|flat] 97+ messages in thread
* Re: [PATCH] Preserve replication origin OIDs in pg_upgrade
@ 2026-06-09 04:17 shveta malik <[email protected]>
parent: Shlok Kyal <[email protected]>
2 siblings, 0 replies; 97+ messages in thread
From: shveta malik @ 2026-06-09 04:17 UTC (permalink / raw)
To: Shlok Kyal <[email protected]>; +Cc: Ajin Cherian <[email protected]>; Zsolt Parragi <[email protected]>; vignesh C <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; PostgreSQL Hackers <[email protected]>; shveta malik <[email protected]>
On Mon, Jun 8, 2026 at 10:36 PM Shlok Kyal <[email protected]> wrote:
>
>
> Hi Ajin,
> I have reviewed the patch and here are some comments:
>
> 1. I want to clarify, what is the expected behaviour when we upgrade
> the postgres instance from version <= 16 to "HEAD + patch"?
> - ReplicationOriginNameForLogicalRep(subid, InvalidOid, originname,
> sizeof(originname));
> - replorigin_create(originname);
> + if (!IsBinaryUpgrade)
> + {
> + ReplicationOriginNameForLogicalRep(subid, InvalidOid,
> originname, sizeof(originname));
> + replorigin_create(originname);
> + }
> and
> + /* Dump replication origins */
> + if (server_version >= 170000 && binary_upgrade &&
> archDumpFormat == archNull)
> + dumpReplicationOrigins(conn);
>
> When the postgres instance with subscription is updated from PG 16 (or
> less) to HEAD, the replication origin is created as we do not have "if
> (!IsBinaryUpgrade)" check in HEAD,
> whereas when the similar instance is upgraded to "HEAD + patch", no
> replication origin is present after the upgrade.
> Is this difference in the behaviour between HEAD and "HEAD + patch" expected?
>
Good catch Shlok. IMO, the behaviour should be same even if we upgrade
to HEAD+patch, it should not result in the loss of origins.
thanks
Shveta
^ permalink raw reply [nested|flat] 97+ messages in thread
* Re: [PATCH] Preserve replication origin OIDs in pg_upgrade
@ 2026-06-09 11:32 Shlok Kyal <[email protected]>
parent: Shlok Kyal <[email protected]>
2 siblings, 1 reply; 97+ messages in thread
From: Shlok Kyal @ 2026-06-09 11:32 UTC (permalink / raw)
To: Ajin Cherian <[email protected]>; +Cc: shveta malik <[email protected]>; Zsolt Parragi <[email protected]>; vignesh C <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; PostgreSQL Hackers <[email protected]>; shveta malik <[email protected]>
On Mon, 8 Jun 2026 at 22:36, Shlok Kyal <[email protected]> wrote:
>
> On Thu, 28 May 2026 at 08:18, Ajin Cherian <[email protected]> wrote:
> >
> >
> >
> > On Fri, May 22, 2026 at 8:27 PM shveta malik <[email protected]> wrote:
> >>
> >> On Fri, May 22, 2026 at 3:16 PM Shlok Kyal <[email protected]> wrote:
> >> >
> >> > On Mon, 18 May 2026 at 16:13, Ajin Cherian <[email protected]> wrote:
> >> > >
> >> > > Rebased the patch as it was no longer applying.
> >> > >
> >> > Hi Ajin,
> >> >
> >> > I have started reviewing the patch. Here is my comment for v6-0002 patch:
> >> >
> >> > Suppose we have a replication setup: publisher -> subscriber
> >> > and we are upgrading subscriber to subscriber_new.
> >> > And if initially 'subscriber_new' has a replication origin, upgrading
> >> > the cluster can error out.
> >> >
> >> > Example:
> >> > We set up a logical replication between publisher node and subscriber node.
> >> >
> >> > On subscriber node:
> >> > postgres=# SELECT * FROM pg_replication_origin;
> >> > roident | roname
> >> > ---------+----------
> >> > 1 | pg_16393
> >> > (1 row)
> >> >
> >> > And initially subscriber_new has a replication origin:
> >> > postgres=# select pg_replication_origin_create('myname');
> >> > pg_replication_origin_create
> >> > ------------------------------
> >> > 1
> >> > (1 row)
> >> >
> >> > postgres=# SELECT * FROM pg_replication_origin;
> >> > roident | roname
> >> > ---------+--------
> >> > 1 | myname
> >> > (1 row)
> >> >
> >> > Now, if we run pg_upgrade to upgrade subscriber node to subscriber_new
> >> > node, we get an error:
> >> > ```
> >> > SELECT pg_catalog.binary_upgrade_create_replication_origin('1'::pg_catalog.oid,
> >> > 'pg_16393'::pg_catalog.name, '0/01743078'::pg_catalog.pg_lsn);
> >> > psql:subscriber_new/pg_upgrade_output.d/20260522T140312.807/dump/pg_upgrade_dump_globals.sql:37:
> >> > ERROR: replication origin with ID 1 already exists
> >> > ```
> >> >
> >> > This error occurs in "Performing Upgrade" stage. Should we add a check
> >> > in the "Performing Consistency Checks" stage so that we don't need to
> >> > re-initdb the new cluster to perform the upgrade?
> >> > Maybe we can add a check similar to
> >> > check_new_cluster_replication_slots(), where pg_upgrade errors out if
> >> > the new cluster already contains replication origins. Thoughts?
> >>
> >> +1. I had the same thought while reviewing the patch today. We should
> >> have it unless there is a reason we have avoided it??
> >
> >
> > Fixed this. Added a new check for replication origins and if the new cluster has any existing replication origins, then the check will fail.
> >>
> >>
> >> Few trivial comments:
> >>
> >> 1)
> >>
> >> +#include "access/skey.h"
> >> +#include "catalog/indexing.h"
> >>
> >> pg_upgrade_support.c compiles without above.
> >>
> >
> > Removed.
> >
> >>
> >> 2)
> >> + Assert(!OidIsValid(rel->rd_rel->reltoastrelid));
> >>
> >> Is there a reason for this sanity check? I generally do not see a
> >> Null-Toast table sanity check after every table_open.
> >>
> >
> > Removed.
> >
> >>
> >> 3)
> >>
> >> +
> >> + /* Dump replication origins */
> >> + if (server_version >= 170000 && binary_upgrade && archDumpFormat == archNull)
> >> + dumpReplicationOrigins(conn);
> >>
> >> why the check is for PG17 specifically?
> >>
> >
> > In PG17, we started migrating pg_subscription_rel and the remote LSN during upgrades; prior to that, these were not migrated. Given that change, it also makes sense to migrate replication origins from them. Otherwise, when upgrading from PG17 to a later version, you could end up with a subscription where pg_subscription_rel and the remote LSN are migrated, but the corresponding replication origin is not created.
> >
> >
> >
> > On Mon, May 25, 2026 at 5:13 PM shveta malik <[email protected]> wrote:
> >>
> >>
> >> One issue in 002:
> >>
> >> binary_upgrade_create_replication_origin() has this:
> >>
> >> + originname = PG_GETARG_NAME(1);
> >> +
> >> + roname_d = CStringGetTextDatum(NameStr(*originname));
> >> +
> >>
> >> We are getting origin-name (text) into Name-type which can not be more
> >> than 64 bytes. So if an origin has name more than 64, it will end up
> >> trimming the name post-upgrade.
> >>
> >> I tried this:
> >>
> >> Old-setup:
> >> postgres=# SELECT
> >> pg_replication_origin_create('this_is_a_very_long_replication_origin_name_that_exceeds_the_limit_of_64');
> >> pg_replication_origin_create
> >> ------------------------------
> >> 1
> >> postgres=# select * from pg_replication_origin;
> >> roident | roname
> >> ---------+----------------------------------------------------------------------
> >> 1 | this_is_a_very_long_replication_origin_name_that_exceeds_the_limit_of_64
> >>
> >>
> >> Post-upgrade: name got trimmed to 64 length.
> >> -------------------------
> >> postgres=# select * from pg_replication_origin;
> >> roident | roname
> >> ---------+-----------------------------------------------------------------
> >> 1 | this_is_a_very_long_replication_origin_name_that_exceeds_the_li
> >>
> >> thanks
> >> Shveta
> >
> >
> > Fixed this. Now binary_upgrade_create_replication_origin handles it similarly to the way pg_replication_origin_create handles the name of the origin.
> >
> > Here's an updated version v7 containing these fixes.
>
> Hi Ajin,
> I have reviewed the patch and here are some comments:
>
> 1. I want to clarify, what is the expected behaviour when we upgrade
> the postgres instance from version <= 16 to "HEAD + patch"?
> - ReplicationOriginNameForLogicalRep(subid, InvalidOid, originname,
> sizeof(originname));
> - replorigin_create(originname);
> + if (!IsBinaryUpgrade)
> + {
> + ReplicationOriginNameForLogicalRep(subid, InvalidOid,
> originname, sizeof(originname));
> + replorigin_create(originname);
> + }
> and
> + /* Dump replication origins */
> + if (server_version >= 170000 && binary_upgrade &&
> archDumpFormat == archNull)
> + dumpReplicationOrigins(conn);
>
> When the postgres instance with subscription is updated from PG 16 (or
> less) to HEAD, the replication origin is created as we do not have "if
> (!IsBinaryUpgrade)" check in HEAD,
> whereas when the similar instance is upgraded to "HEAD + patch", no
> replication origin is present after the upgrade.
> Is this difference in the behaviour between HEAD and "HEAD + patch" expected?
>
> 2.a) Can we replace:
> + /* Check for OID collision */
> + ScanKeyInit(&key,
> + Anum_pg_replication_origin_roident,
> + BTEqualStrategyNumber, F_OIDEQ,
> + ObjectIdGetDatum(node));
> + scan = systable_beginscan(rel, ReplicationOriginIdentIndex,
> + true /* indexOK */,
> + SnapshotSelf,
> + 1, &key);
> + collides = HeapTupleIsValid(systable_getnext(scan));
> + systable_endscan(scan);
> +
> + if (collides)
> + ereport(ERROR,
> + (errcode(ERRCODE_DUPLICATE_OBJECT),
> + errmsg("replication origin with ID %u already
> exists", node_oid)));
>
> with:
> if (SearchSysCacheExists1(REPLORIGIDENT, ObjectIdGetDatum(node)))
> ereport(ERROR, errcode(ERRCODE_DUPLICATE_OBJECT),
> errmsg("replication origin with ID %u already
> exists", node_oid));
>
> 2.b) Similarly, Can we replace:
> + /* Check for name collision */
> + ScanKeyInit(&key,
> + Anum_pg_replication_origin_roname,
> + BTEqualStrategyNumber, F_TEXTEQ,
> + roname_d);
> + scan = systable_beginscan(rel, ReplicationOriginNameIndex,
> + true /* indexOK */,
> + SnapshotSelf,
> + 1, &key);
> + collides = HeapTupleIsValid(systable_getnext(scan));
> + systable_endscan(scan);
> +
> + if (collides)
> + ereport(ERROR,
> + (errcode(ERRCODE_DUPLICATE_OBJECT),
> + errmsg("replication origin \"%s\" already exists",
> + originname)));
> with:
> if (SearchSysCacheExists1(REPLORIGNAME, roname_d))
> ereport(ERROR, errcode(ERRCODE_DUPLICATE_OBJECT),
> errmsg("replication origin \"%s\" already
> exists", originname));
>
> This will make the code smaller and easy to read. Thoughts?
>
One more comment:
3. In logical-replication.sgml, we have documentation as:
"Commit timestamps and origin data are not preserved during the upgrade."
Now since the origin related data is preserved after the upgrade,
should we update the documentation here?
Thanks,
Shlok Kyal
^ permalink raw reply [nested|flat] 97+ messages in thread
* Re: [PATCH] Preserve replication origin OIDs in pg_upgrade
@ 2026-06-11 06:38 shveta malik <[email protected]>
parent: Shlok Kyal <[email protected]>
2 siblings, 2 replies; 97+ messages in thread
From: shveta malik @ 2026-06-11 06:38 UTC (permalink / raw)
To: Shlok Kyal <[email protected]>; Amit Kapila <[email protected]>; Dilip Kumar <[email protected]>; +Cc: Ajin Cherian <[email protected]>; Zsolt Parragi <[email protected]>; vignesh C <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; PostgreSQL Hackers <[email protected]>; shveta malik <[email protected]>
On Mon, Jun 8, 2026 at 10:36 PM Shlok Kyal <[email protected]> wrote:
>
>
> 1. I want to clarify, what is the expected behaviour when we upgrade
> the postgres instance from version <= 16 to "HEAD + patch"?
> - ReplicationOriginNameForLogicalRep(subid, InvalidOid, originname,
> sizeof(originname));
> - replorigin_create(originname);
> + if (!IsBinaryUpgrade)
> + {
> + ReplicationOriginNameForLogicalRep(subid, InvalidOid,
> originname, sizeof(originname));
> + replorigin_create(originname);
> + }
> and
> + /* Dump replication origins */
> + if (server_version >= 170000 && binary_upgrade &&
> archDumpFormat == archNull)
> + dumpReplicationOrigins(conn);
>
> When the postgres instance with subscription is updated from PG 16 (or
> less) to HEAD, the replication origin is created as we do not have "if
> (!IsBinaryUpgrade)" check in HEAD,
> whereas when the similar instance is upgraded to "HEAD + patch", no
> replication origin is present after the upgrade.
> Is this difference in the behaviour between HEAD and "HEAD + patch" expected?
>
No, this difference is not expected.
The issue is that in patch 002 we intentionally avoid
replication-origin creation during CreateSubscription() by adding the
'!IsBinaryUpgrade' check. The idea is that origin creation should
instead happen through
'binary_upgrade_create_conflict_detection_slot()', where we can also
preserve the
original origin ID.
Origin-ID preservation is currently implemented only for upgrades from
PG17 and later. (see the check in pg_dumpall.c in 002). As a result,
when upgrading from PG16, origin creation is skipped in
CreateSubscription(), but the origin is also not created through
binary_upgrade_create_conflict_detection_slot() because of the PG17
version check. Therefore, no replication origin exists after the
upgrade in this particular case.
We do not have access to the source-node version during the restore
stage (i.e., when CreateSubscription() is executed), so we cannot
perform the version check directly there. Therefore, any information
needed to make this decision must be preserved during the dump stage.
Ajin, Kuroda-san, and I discussed several possible approaches to address this:
1)
Add a new subscription option, for example 'skip_origin_creation'.
When set to true, CREATE SUBSCRIPTION would skip replication-origin
creation. During binary upgrade, pg_dump would emit
skip_origin_creation = true for source clusters running PG17 or later,
and false otherwise.
However, this approach requires introducing a new user-visible CREATE
SUBSCRIPTION option.
2)
Another idea could be to have a
'binary_upgrade_skip_subs_origin_creation' variable. We populate it
during dumpSubscription if 'source_version >= PG17' and
'IsBinaryUpgrade' is true. So this will be dumped
SELECT pg_catalog.binary_upgrade_set_skip_subs_origin_creation(true);
CREATE SUBSCRIPTION ...
SELECT pg_catalog.binary_upgrade_set_skip_subs_origin_creation(false);
First call will set 'binary_upgrade_skip_subs_origin_creation' which
will be used by Create Subscription to decide to skip origin. This is
similar to how dumpACl decides whether to record priveleges into
'pg_init_privs' for objectes created by extensions. See
binary_upgrade_set_record_init_privs() which sets
binary_upgrade_record_init_privs which is then used by
recordExtensionInitPriv().
3)
Create origins and preserve origin IDs regardless of source version
The current patch preserves origin IDs for upgrades from PG17 and later.
One reason for choosing PG17 as the starting point for preserving
origin-ids is that complete logical-replication setup migration was
introduced in PG17. Prior to PG17, subscription state
(pg_subscription_rel) and replication slots were not preserved during
upgrade, so limiting origin-ID preservation to PG17 and later aligns
with the broader logical-replication migration support.
OTOH, replication-origin migration itself is supported for PG16
upgrades. For PG16- upgrades (without this patch), subscriptions
itself can be migrated, and corresponding origins can be created at
that time. Therefore, even on versions prior to Pg17, it may also be
reasonable to consider preserving origin IDs independently of the rest
of the logical-replication configuration migration, and not tie it to
the PG17 boundary.
If we choose that approach, check_new_cluster_replication_origins()
would likely need to run for upgrades from PG16 (and prior) as well.
Subscription-ids also need to be preserved for the same versions. For
reference, check_new_cluster_replication_slots() and
check_new_cluster_subscription_configuration() currently run only for
PG17 and later.
~~
Given the need for a new user-visible parameter, I would rule out
option 1. Between options 2 and 3, I prefer option 2, but I would be
interested in hearing others' opinions. Any other ideas are also
welcome.
thanks
Shveta
^ permalink raw reply [nested|flat] 97+ messages in thread
* RE: [PATCH] Preserve replication origin OIDs in pg_upgrade
@ 2026-06-11 06:52 Hayato Kuroda (Fujitsu) <[email protected]>
parent: shveta malik <[email protected]>
1 sibling, 1 reply; 97+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2026-06-11 06:52 UTC (permalink / raw)
To: 'shveta malik' <[email protected]>; Shlok Kyal <[email protected]>; Amit Kapila <[email protected]>; Dilip Kumar <[email protected]>; +Cc: Ajin Cherian <[email protected]>; Zsolt Parragi <[email protected]>; vignesh C <[email protected]>; PostgreSQL Hackers <[email protected]>
Dear Shveta,
> No, this difference is not expected.
Yes it must be fixed.
> 2)
> Another idea could be to have a
> 'binary_upgrade_skip_subs_origin_creation' variable. We populate it
> during dumpSubscription if 'source_version >= PG17' and
> 'IsBinaryUpgrade' is true. So this will be dumped
>
> SELECT pg_catalog.binary_upgrade_set_skip_subs_origin_creation(true);
> CREATE SUBSCRIPTION ...
> SELECT pg_catalog.binary_upgrade_set_skip_subs_origin_creation(false);
>
> First call will set 'binary_upgrade_skip_subs_origin_creation' which
> will be used by Create Subscription to decide to skip origin. This is
> similar to how dumpACl decides whether to record priveleges into
> 'pg_init_privs' for objectes created by extensions. See
> binary_upgrade_set_record_init_privs() which sets
> binary_upgrade_record_init_privs which is then used by
> recordExtensionInitPriv().
To clarify, I prefer the 2nd idea. According to 29d0a77, which migrates
replication slots from PG17+, it does not change the behavior when the source is
PG16-. We should follow the manner.
In case of third idea, origin id can be also preserved for PG16-, but the
version was out-of-scope of our patch [1]; in that version pg_subscription_rel
cannot be migrated, thus the logical replication cannot work in the first place.
Also, if we want to keep the existing behavior, we should allow upgrading the
below case.
Source: PG16, two subscriptions and origins exist (id = 1, 2).
Target: an origin exists (id = 1)
But we must reject upgrading if we preserves the origin-id, right?
[1]: https://www.postgresql.org/message-id/CAFPTHDYqpuZ6o2-HuCJDYqJ7GY3%2BzV%2BXo-gT7PAgi4Bkz%2BoTxw%40ma...
Best regards,
Hayato Kuroda
FUJITSU LIMITED
^ permalink raw reply [nested|flat] 97+ messages in thread
* Re: [PATCH] Preserve replication origin OIDs in pg_upgrade
@ 2026-06-11 10:37 shveta malik <[email protected]>
parent: Hayato Kuroda (Fujitsu) <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: shveta malik @ 2026-06-11 10:37 UTC (permalink / raw)
To: Hayato Kuroda (Fujitsu) <[email protected]>; +Cc: Shlok Kyal <[email protected]>; Amit Kapila <[email protected]>; Dilip Kumar <[email protected]>; Ajin Cherian <[email protected]>; Zsolt Parragi <[email protected]>; vignesh C <[email protected]>; PostgreSQL Hackers <[email protected]>; shveta malik <[email protected]>
On Thu, Jun 11, 2026 at 12:22 PM Hayato Kuroda (Fujitsu)
<[email protected]> wrote:
>
> Dear Shveta,
>
> > No, this difference is not expected.
>
> Yes it must be fixed.
>
> > 2)
> > Another idea could be to have a
> > 'binary_upgrade_skip_subs_origin_creation' variable. We populate it
> > during dumpSubscription if 'source_version >= PG17' and
> > 'IsBinaryUpgrade' is true. So this will be dumped
> >
> > SELECT pg_catalog.binary_upgrade_set_skip_subs_origin_creation(true);
> > CREATE SUBSCRIPTION ...
> > SELECT pg_catalog.binary_upgrade_set_skip_subs_origin_creation(false);
> >
> > First call will set 'binary_upgrade_skip_subs_origin_creation' which
> > will be used by Create Subscription to decide to skip origin. This is
> > similar to how dumpACl decides whether to record priveleges into
> > 'pg_init_privs' for objectes created by extensions. See
> > binary_upgrade_set_record_init_privs() which sets
> > binary_upgrade_record_init_privs which is then used by
> > recordExtensionInitPriv().
>
> To clarify, I prefer the 2nd idea. According to 29d0a77, which migrates
> replication slots from PG17+, it does not change the behavior when the source is
> PG16-. We should follow the manner.
>
> In case of third idea, origin id can be also preserved for PG16-, but the
> version was out-of-scope of our patch [1]; in that version pg_subscription_rel
> cannot be migrated, thus the logical replication cannot work in the first place.
I agree.
> Also, if we want to keep the existing behavior, we should allow upgrading the
> below case.
>
> Source: PG16, two subscriptions and origins exist (id = 1, 2).
> Target: an origin exists (id = 1)
>
> But we must reject upgrading if we preserves the origin-id, right?
Right. That is why I commented that if we choose option3, we will need
to implement the check (check_new_cluster_replication_origins) even
on PG16 to reject the upgrade in the case you suggested.
thanks
Shveta
^ permalink raw reply [nested|flat] 97+ messages in thread
* Re: [PATCH] Preserve replication origin OIDs in pg_upgrade
@ 2026-06-15 11:31 Amit Kapila <[email protected]>
parent: shveta malik <[email protected]>
1 sibling, 1 reply; 97+ messages in thread
From: Amit Kapila @ 2026-06-15 11:31 UTC (permalink / raw)
To: shveta malik <[email protected]>; +Cc: Shlok Kyal <[email protected]>; Dilip Kumar <[email protected]>; Ajin Cherian <[email protected]>; Zsolt Parragi <[email protected]>; vignesh C <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; PostgreSQL Hackers <[email protected]>
On Thu, Jun 11, 2026 at 12:08 PM shveta malik <[email protected]> wrote:
>
> Ajin, Kuroda-san, and I discussed several possible approaches to address this:
>
> 1)
> Add a new subscription option, for example 'skip_origin_creation'.
> When set to true, CREATE SUBSCRIPTION would skip replication-origin
> creation. During binary upgrade, pg_dump would emit
> skip_origin_creation = true for source clusters running PG17 or later,
> and false otherwise.
>
> However, this approach requires introducing a new user-visible CREATE
> SUBSCRIPTION option.
>
> 2)
> Another idea could be to have a
> 'binary_upgrade_skip_subs_origin_creation' variable. We populate it
> during dumpSubscription if 'source_version >= PG17' and
> 'IsBinaryUpgrade' is true. So this will be dumped
>
> SELECT pg_catalog.binary_upgrade_set_skip_subs_origin_creation(true);
> CREATE SUBSCRIPTION ...
> SELECT pg_catalog.binary_upgrade_set_skip_subs_origin_creation(false);
>
> First call will set 'binary_upgrade_skip_subs_origin_creation' which
> will be used by Create Subscription to decide to skip origin. This is
> similar to how dumpACl decides whether to record priveleges into
> 'pg_init_privs' for objectes created by extensions. See
> binary_upgrade_set_record_init_privs() which sets
> binary_upgrade_record_init_privs which is then used by
> recordExtensionInitPriv().
>
> 3)
> Create origins and preserve origin IDs regardless of source version
>
> The current patch preserves origin IDs for upgrades from PG17 and later.
>
> One reason for choosing PG17 as the starting point for preserving
> origin-ids is that complete logical-replication setup migration was
> introduced in PG17. Prior to PG17, subscription state
> (pg_subscription_rel) and replication slots were not preserved during
> upgrade, so limiting origin-ID preservation to PG17 and later aligns
> with the broader logical-replication migration support.
>
> OTOH, replication-origin migration itself is supported for PG16
> upgrades. For PG16- upgrades (without this patch), subscriptions
> itself can be migrated, and corresponding origins can be created at
> that time. Therefore, even on versions prior to Pg17, it may also be
> reasonable to consider preserving origin IDs independently of the rest
> of the logical-replication configuration migration, and not tie it to
> the PG17 boundary.
>
> If we choose that approach, check_new_cluster_replication_origins()
> would likely need to run for upgrades from PG16 (and prior) as well.
> Subscription-ids also need to be preserved for the same versions. For
> reference, check_new_cluster_replication_slots() and
> check_new_cluster_subscription_configuration() currently run only for
> PG17 and later.
> ~~
>
> Given the need for a new user-visible parameter, I would rule out
> option 1. Between options 2 and 3, I prefer option 2, but I would be
> interested in hearing others' opinions.
>
I'd prefer option-3 (preserve origin IDs uniformly, including upgrades
from PG16), for two reasons. (a) First, the new-cluster check ("reject
if the new cluster already has replication origins") is a restriction
on the target, and adding restrictions on a newer target version
regardless of source version is well within how pg_upgrade already
behaves. In practice the new cluster is a fresh initdb with no
origins, so the check effectively never fires; the only scenario it
rejects is one where the user manually created subscriptions/origins
on the new cluster before upgrading, which we discourage for other
object types anyway. So I don't think preserving existing behavior
here is worth much. (b) Second, option-2 reintroduces source-version
branching at dump time (the skip_subs_origin_creation flag gated on
source >= 17). Since pg_commit_ts is only copied from PG19+, the exact
roident value only matters from PG19; for PG16-18 the value is
don't-care, so preserving it everywhere is harmless where it isn't
needed and required where it is. That lets us keep a single
always-preserve path instead of branching per source version.
Separately, I don't think we actually need to preserve the
subscription OID to meet the goal here. pg_commit_ts stores the
numeric roident, not the origin name or the sub OID, and conflict
detection compares roidents. So the invariant we must keep is "each
subscription owns the same roident after upgrade", which doesn't
depend on the name being stable.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 97+ messages in thread
* Re: [PATCH] Preserve replication origin OIDs in pg_upgrade
@ 2026-06-16 09:20 Ajin Cherian <[email protected]>
parent: Amit Kapila <[email protected]>
0 siblings, 2 replies; 97+ messages in thread
From: Ajin Cherian @ 2026-06-16 09:20 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: shveta malik <[email protected]>; Shlok Kyal <[email protected]>; Dilip Kumar <[email protected]>; Zsolt Parragi <[email protected]>; vignesh C <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; PostgreSQL Hackers <[email protected]>
On Mon, Jun 15, 2026 at 9:31 PM Amit Kapila <[email protected]> wrote:
>
>
> I'd prefer option-3 (preserve origin IDs uniformly, including upgrades
> from PG16), for two reasons. (a) First, the new-cluster check ("reject
> if the new cluster already has replication origins") is a restriction
> on the target, and adding restrictions on a newer target version
> regardless of source version is well within how pg_upgrade already
> behaves. In practice the new cluster is a fresh initdb with no
> origins, so the check effectively never fires; the only scenario it
> rejects is one where the user manually created subscriptions/origins
> on the new cluster before upgrading, which we discourage for other
> object types anyway. So I don't think preserving existing behavior
> here is worth much. (b) Second, option-2 reintroduces source-version
> branching at dump time (the skip_subs_origin_creation flag gated on
> source >= 17). Since pg_commit_ts is only copied from PG19+, the exact
> roident value only matters from PG19; for PG16-18 the value is
> don't-care, so preserving it everywhere is harmless where it isn't
> needed and required where it is. That lets us keep a single
> always-preserve path instead of branching per source version.
>
>
Ok.
> Separately, I don't think we actually need to preserve the
> subscription OID to meet the goal here. pg_commit_ts stores the
> numeric roident, not the origin name or the sub OID, and conflict
> detection compares roidents. So the invariant we must keep is "each
> subscription owns the same roident after upgrade", which doesn't
> depend on the name being stable.
>
>
>
Replication origins are global objects and are therefore dumped by
pg_dumpall, while subscriptions are database-specific and are dumped by
pg_dump. In pg_upgrade, pg_dumpall runs before pg_dump.
The complication if subscription OID is not preserved is that
subscription-associated origins derive their name from the subscription's
OID. If the subscription OID is not preserved across upgrades, the origin
name on the new cluster will differ from the old one, making it impossible
to restore the origin independently. For these origins, creation must
happen after CreateSubscription has established the new OID and not upfront
in pg_dumpall.
Non-subscription origins have no such dependency and continue to be created
separately as before. The subscription code also cannot be left untouched:
even once the subscription is created, rather than origin being created
inside CreateSubscription, its associated origin must still be explicitly
created via binary_upgrade_create_replication_origin to ensure the roident
matches the original node, which means CreateSubscription needs to be
modified anyways. So, there is an advantage of reducing code if
subscription OID is preserved and all origins keep the same name as the old
node.
regards,
Ajin Cherian
Fujitsu Australia
^ permalink raw reply [nested|flat] 97+ messages in thread
* Re: [PATCH] Preserve replication origin OIDs in pg_upgrade
@ 2026-06-17 09:19 Ajin Cherian <[email protected]>
parent: Shlok Kyal <[email protected]>
0 siblings, 3 replies; 97+ messages in thread
From: Ajin Cherian @ 2026-06-17 09:19 UTC (permalink / raw)
To: Shlok Kyal <[email protected]>; +Cc: shveta malik <[email protected]>; Zsolt Parragi <[email protected]>; vignesh C <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; PostgreSQL Hackers <[email protected]>; shveta malik <[email protected]>
On Fri, Jun 5, 2026 at 8:10 PM vignesh C <[email protected]> wrote:
>
> Few comments:
> 1) pg_dumpall dumps all replication origins, not just subscription
> origins and recreates every origin:
> SELECT o.*, os.remote_lsn
> FROM pg_replication_origin o
> LEFT JOIN pg_replication_origin_status os
> ON o.roident = os.local_id
>
> Is pg_upgrade expected to preserve user-created origins?
> If so, you can mention this in the commit message
>
I explicitly state that all replication origins are migrated in the
commit message, nothing specific about subscription origins.
> 2) Should the table lock be released here, if so should we mention
> comments similar to how it is mentioned in
> binary_upgarde_add_sub_rel_state:
> + /* Restore the remote_lsn if provided, while still holding the lock */
> + if (!PG_ARGISNULL(2))
> + {
> + XLogRecPtr remote_commit = PG_GETARG_LSN(2);
>
> - ReplicationSlotRelease();
> + replorigin_advance(node, remote_commit, InvalidXLogRecPtr,
> + false /* backward */,
> + false /* WAL log */);
> + }
> +
> + table_close(rel, RowExclusiveLock);
>
Is that relevant when creating a new row? or only when updating an existing one?
> 3) We can use new style of ereport to exclude '(' before errcode:
> 3.a)
> + ereport(ERROR,
> + (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
> + errmsg("replication origin name is too long"),
> + errdetail("Replication origin names must be no longer than %d bytes.",
> + MAX_RONAME_LEN)));
>
> 3.b) Similarly here:
> + ereport(ERROR,
> + (errcode(ERRCODE_DUPLICATE_OBJECT),
> + errmsg("replication origin with ID %u already exists", node_oid)));
>
> 3.c) There are many others, you can search other ereport in the patch
> and change it too.
>
Done.
> 4) Since only one column will be returned here, can we use
> "PQgetvalue(res, 0, 0) and remove PQfnumber:
> + res = executeQueryOrDie(conn,
> + "SELECT count(*) AS nrepl_origins "
> + "FROM pg_catalog.pg_replication_origin");
> + i_nrepl_origins = PQfnumber(res, "nrepl_origins");
> + cluster->nrepl_origins = atoi(PQgetvalue(res, 0, i_nrepl_origins));
> + PQclear(res);
>
Done.
> 5) replorigin_create also has some duplicate code like below, will it
> be possible to have a common function so that both of them can use?
> /* Check for name collision */
> + ScanKeyInit(&key,
> + Anum_pg_replication_origin_roname,
> + BTEqualStrategyNumber, F_TEXTEQ,
> + roname_d);
> + scan = systable_beginscan(rel, ReplicationOriginNameIndex,
> + true /* indexOK */,
> + SnapshotSelf,
> + 1, &key);
> + collides = HeapTupleIsValid(systable_getnext(scan));
> + systable_endscan(scan);
> +
> + if (collides)
> + ereport(ERROR,
> + (errcode(ERRCODE_DUPLICATE_OBJECT),
> + errmsg("replication origin \"%s\" already exists",
> + originname)));
> +
> + memset(&nulls, 0, sizeof(nulls));
> + memset(&values, 0, sizeof(values));
> +
> + values[Anum_pg_replication_origin_roident - 1] = ObjectIdGetDatum(node);
> + values[Anum_pg_replication_origin_roname - 1] = roname_d;
> +
> + tuple = heap_form_tuple(RelationGetDescr(rel), values, nulls);
> + CatalogTupleInsert(rel, tuple);
> + heap_freetuple(tuple);
> + CommandCounterIncrement();
>
I have created a new function replorigin_create_with_id() which has
some common code and called that from both replorigin_create() and
binary_upgrade_create_replication_origin()
On Tue, Jun 9, 2026 at 3:06 AM Shlok Kyal <[email protected]> wrote:
>
> Hi Ajin,
> I have reviewed the patch and here are some comments:
>
> 1. I want to clarify, what is the expected behaviour when we upgrade
> the postgres instance from version <= 16 to "HEAD + patch"?
> - ReplicationOriginNameForLogicalRep(subid, InvalidOid, originname,
> sizeof(originname));
> - replorigin_create(originname);
> + if (!IsBinaryUpgrade)
> + {
> + ReplicationOriginNameForLogicalRep(subid, InvalidOid,
> originname, sizeof(originname));
> + replorigin_create(originname);
> + }
> and
> + /* Dump replication origins */
> + if (server_version >= 170000 && binary_upgrade &&
> archDumpFormat == archNull)
> + dumpReplicationOrigins(conn);
>
> When the postgres instance with subscription is updated from PG 16 (or
> less) to HEAD, the replication origin is created as we do not have "if
> (!IsBinaryUpgrade)" check in HEAD,
> whereas when the similar instance is upgraded to "HEAD + patch", no
> replication origin is present after the upgrade.
> Is this difference in the behaviour between HEAD and "HEAD + patch" expected?
>
I have modified this as option 3 suggested by Shveta and Amit and
removed the PG17 restriction for migrating replication origins, now it
does it for all versions that have replication origins.
> 2.a) Can we replace:
> + /* Check for OID collision */
> + ScanKeyInit(&key,
> + Anum_pg_replication_origin_roident,
> + BTEqualStrategyNumber, F_OIDEQ,
> + ObjectIdGetDatum(node));
> + scan = systable_beginscan(rel, ReplicationOriginIdentIndex,
> + true /* indexOK */,
> + SnapshotSelf,
> + 1, &key);
> + collides = HeapTupleIsValid(systable_getnext(scan));
> + systable_endscan(scan);
> +
> + if (collides)
> + ereport(ERROR,
> + (errcode(ERRCODE_DUPLICATE_OBJECT),
> + errmsg("replication origin with ID %u already
> exists", node_oid)));
>
> with:
> if (SearchSysCacheExists1(REPLORIGIDENT, ObjectIdGetDatum(node)))
> ereport(ERROR, errcode(ERRCODE_DUPLICATE_OBJECT),
> errmsg("replication origin with ID %u already
> exists", node_oid));
>
> 2.b) Similarly, Can we replace:
> + /* Check for name collision */
> + ScanKeyInit(&key,
> + Anum_pg_replication_origin_roname,
> + BTEqualStrategyNumber, F_TEXTEQ,
> + roname_d);
> + scan = systable_beginscan(rel, ReplicationOriginNameIndex,
> + true /* indexOK */,
> + SnapshotSelf,
> + 1, &key);
> + collides = HeapTupleIsValid(systable_getnext(scan));
> + systable_endscan(scan);
> +
> + if (collides)
> + ereport(ERROR,
> + (errcode(ERRCODE_DUPLICATE_OBJECT),
> + errmsg("replication origin \"%s\" already exists",
> + originname)));
> with:
> if (SearchSysCacheExists1(REPLORIGNAME, roname_d))
> ereport(ERROR, errcode(ERRCODE_DUPLICATE_OBJECT),
> errmsg("replication origin \"%s\" already
> exists", originname));
>
> This will make the code smaller and easy to read. Thoughts?
Done
On Tue, Jun 9, 2026 at 2:13 PM shveta malik <[email protected]> wrote:
>
>
> Please find a few comments on v007:
>
> 1)
>
> + node = (ReplOriginId) node_oid;
> + originname = text_to_cstring(PG_GETARG_TEXT_PP(1));
> +
> + if (strlen(originname) > MAX_RONAME_LEN)
> + ereport(ERROR,
> + (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
> + errmsg("replication origin name is too long"),
> + errdetail("Replication origin names must be no longer than %d bytes.",
> + MAX_RONAME_LEN)));
> +
> + roname_d = CStringGetTextDatum(originname);
>
> CStringGetTextDatum is:
> #define CStringGetTextDatum(s) PointerGetDatum(cstring_to_text(s))
> ---------
>
> We are first converting text to C-string , then Cstring to text and
> then getting Datum. Double allocation and double conversion. Can we
> please try this instead and verify if it works:
>
> text *origin_text = PG_GETARG_TEXT_PP(1);
> originname = text_to_cstring(origin_text);
> roname_d = PointerGetDatum(origin_text);
>
I've restructured the code now, its a bit tough to do this now.
> 2)
> +check_new_cluster_replication_origins(void)
> +{
> + PGconn *conn;
> + PGresult *res;
> + int norigins;
> +
> + /* Quick return if there are no replication origins to migrate. */
> + if (old_cluster.nrepl_origins == 0)
> + return;
>
> Don't we need a version check simialr to what we have introduced in
> pg_dumpall for origins? (Similar to
> check_new_cluster_replication_slots as well)
>
I've removed the version check for PG17 now.
> 3)
> Since we have introduced check_new_cluster_replication_origins, do we
> even need below checks in binary_upgrade_create_replication_origin()?
> In which conditions will they be hit?
>
> + /* Check for OID collision */
> + ....
> + if (collides)
> + ereport(ERROR,
> + (errcode(ERRCODE_DUPLICATE_OBJECT),
> + errmsg("replication origin with ID %u already exists", node_oid)));
> +
> + /* Check for name collision */
> + ....
> + if (collides)
> + ereport(ERROR,
> + (errcode(ERRCODE_DUPLICATE_OBJECT),
> + errmsg("replication origin \"%s\" already exists",
> + originname)));
>
With the restructured code, its not easy to do this now.
>
> 4)
> Now since check_new_cluster_subscription_configuration() is purely
> about checking max-origin GUC configuration, I think the name is
> misleading. IMO, we should merge it to
> check_new_cluster_replication_origins(). See how
> check_new_cluster_replication_slots() does it: checking both
> new-cluster's count and the max-configuration for slots.
>
Done.
On Tue, Jun 9, 2026 at 9:32 PM Shlok Kyal <[email protected]> wrote:
>
> >
> One more comment:
>
> 3. In logical-replication.sgml, we have documentation as:
> "Commit timestamps and origin data are not preserved during the upgrade."
> Now since the origin related data is preserved after the upgrade,
> should we update the documentation here?
>
Updated.
I have addressed the comments in v8 (attached). I've also now taken
out the check for PG17 on old cluster and now all clusters which have
replication origins are migrated.
regards,
Ajin Cherian
Fujitsu Australia
Attachments:
[application/octet-stream] v8-0001-Preserve-subscription-OIDs-during-pg_upgrade.patch (9.1K, ../../CAFPTHDbyVt_ggdi+rOP4=jBmgU=DDgYWuLxc4oQ6ux-T0H-ZBg@mail.gmail.com/2-v8-0001-Preserve-subscription-OIDs-during-pg_upgrade.patch)
download | inline diff:
From d7864a45be2d647ca3f58cdf6407c06cb53b5ff0 Mon Sep 17 00:00:00 2001
From: Ajin Cherian <[email protected]>
Date: Thu, 11 Jun 2026 12:35:13 +1000
Subject: [PATCH v8 1/2] Preserve subscription OIDs during pg_upgrade
Currently subscription OIDs can be changed when a cluster is upgraded
using pg_upgrade. This is required for a subsequent patch which will
preserve the replication oids after upgrade.
Author: Vignesh C <[email protected]>
---
src/backend/commands/subscriptioncmds.c | 25 +++++++++++++++++--
src/backend/utils/adt/pg_upgrade_support.c | 10 ++++++++
src/bin/pg_dump/pg_dump.c | 8 ++++++
src/bin/pg_upgrade/pg_upgrade.c | 3 +++
src/bin/pg_upgrade/t/004_subscription.pl | 7 ++++++
src/include/catalog/binary_upgrade.h | 1 +
src/include/catalog/pg_proc.dat | 4 +++
.../expected/spgist_name_ops.out | 6 +++--
8 files changed, 60 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index fd026b304c2..71386fa7d9d 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -83,6 +83,12 @@
/* check if the 'val' has 'bits' set */
#define IsSet(val, bits) (((val) & (bits)) == (bits))
+/*
+ * This will be set by the pg_upgrade_support function --
+ * binary_upgrade_set_next_pg_subscription_oid().
+ */
+Oid binary_upgrade_next_pg_subscription_oid = InvalidOid;
+
/*
* Structure to hold a bitmap representing the user-provided CREATE/ALTER
* SUBSCRIPTION command options and the parsed/default values of each of them.
@@ -772,8 +778,23 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
memset(values, 0, sizeof(values));
memset(nulls, false, sizeof(nulls));
- subid = GetNewOidWithIndex(rel, SubscriptionObjectIndexId,
- Anum_pg_subscription_oid);
+ /* Use binary-upgrade override for pg_subscription.oid? */
+ if (IsBinaryUpgrade)
+ {
+ if (!OidIsValid(binary_upgrade_next_pg_subscription_oid))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("pg_subscription OID value not set when in binary upgrade mode")));
+
+ subid = binary_upgrade_next_pg_subscription_oid;
+ binary_upgrade_next_pg_subscription_oid = InvalidOid;
+ }
+ else
+ {
+ subid = GetNewOidWithIndex(rel, SubscriptionObjectIndexId,
+ Anum_pg_subscription_oid);
+ }
+
values[Anum_pg_subscription_oid - 1] = ObjectIdGetDatum(subid);
values[Anum_pg_subscription_subdbid - 1] = ObjectIdGetDatum(MyDatabaseId);
values[Anum_pg_subscription_subskiplsn - 1] = LSNGetDatum(InvalidXLogRecPtr);
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index b505a6b4fee..59c3e7f0146 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -181,6 +181,16 @@ binary_upgrade_set_next_pg_authid_oid(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
+Datum
+binary_upgrade_set_next_pg_subscription_oid(PG_FUNCTION_ARGS)
+{
+ Oid subid = PG_GETARG_OID(0);
+
+ CHECK_IS_BINARY_UPGRADE;
+ binary_upgrade_next_pg_subscription_oid = subid;
+ PG_RETURN_VOID();
+}
+
Datum
binary_upgrade_create_empty_extension(PG_FUNCTION_ARGS)
{
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index a0f7f8e2168..049337de62a 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -5583,6 +5583,14 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
appendPQExpBuffer(delq, "DROP SUBSCRIPTION %s;\n",
qsubname);
+ if (dopt->binary_upgrade)
+ {
+ appendPQExpBufferStr(query, "\n-- For binary upgrade, must preserve pg_subscription.oid\n");
+ appendPQExpBuffer(query,
+ "SELECT pg_catalog.binary_upgrade_set_next_pg_subscription_oid('%u'::pg_catalog.oid);\n\n",
+ subinfo->dobj.catId.oid);
+ }
+
appendPQExpBuffer(query, "CREATE SUBSCRIPTION %s ",
qsubname);
if (subinfo->subservername)
diff --git a/src/bin/pg_upgrade/pg_upgrade.c b/src/bin/pg_upgrade/pg_upgrade.c
index 2127d297bfe..4e853096698 100644
--- a/src/bin/pg_upgrade/pg_upgrade.c
+++ b/src/bin/pg_upgrade/pg_upgrade.c
@@ -35,6 +35,9 @@
*
* We control all assignments of pg_database.oid because we want the directory
* names to match between the old and new cluster.
+ *
+ * We control assignment of pg_subscription.oid because we want the oid to
+ * match between the old and new cluster.
*/
diff --git a/src/bin/pg_upgrade/t/004_subscription.pl b/src/bin/pg_upgrade/t/004_subscription.pl
index c94a82deae0..646767f2a65 100644
--- a/src/bin/pg_upgrade/t/004_subscription.pl
+++ b/src/bin/pg_upgrade/t/004_subscription.pl
@@ -313,6 +313,9 @@ my $tab_upgraded1_oid = $old_sub->safe_psql('postgres',
my $tab_upgraded2_oid = $old_sub->safe_psql('postgres',
"SELECT oid FROM pg_class WHERE relname = 'tab_upgraded2'");
+$sub_oid = $old_sub->safe_psql('postgres',
+ "SELECT oid FROM pg_subscription ORDER BY subname");
+
$old_sub->stop;
# Change configuration so that initial table sync does not get started
@@ -359,6 +362,10 @@ $publisher->safe_psql(
$new_sub->start;
+# The subscription oid should be preserved
+$result = $new_sub->safe_psql('postgres', "SELECT oid FROM pg_subscription ORDER BY subname");
+is($result, qq($sub_oid), "subscription oid should have been preserved");
+
# The subscription's running status, failover option, and retain_dead_tuples
# option should be preserved in the upgraded instance. So regress_sub4 should
# still have subenabled, subfailover, and subretaindeadtuples set to true,
diff --git a/src/include/catalog/binary_upgrade.h b/src/include/catalog/binary_upgrade.h
index 7bf7ae44385..b15b18e7dc9 100644
--- a/src/include/catalog/binary_upgrade.h
+++ b/src/include/catalog/binary_upgrade.h
@@ -32,6 +32,7 @@ extern PGDLLIMPORT RelFileNumber binary_upgrade_next_toast_pg_class_relfilenumbe
extern PGDLLIMPORT Oid binary_upgrade_next_pg_enum_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_authid_oid;
+extern PGDLLIMPORT Oid binary_upgrade_next_pg_subscription_oid;
extern PGDLLIMPORT bool binary_upgrade_record_init_privs;
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index be157a5fbe9..3a28406981d 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -11969,6 +11969,10 @@
proisstrict => 'f', provolatile => 'v', proparallel => 'u',
prorettype => 'void', proargtypes => '',
prosrc => 'binary_upgrade_create_conflict_detection_slot' },
+{ oid => '9160', descr => 'for use by pg_upgrade',
+ proname => 'binary_upgrade_set_next_pg_subscription_oid', provolatile => 'v',
+ proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
+ prosrc => 'binary_upgrade_set_next_pg_subscription_oid' },
# conversion functions
{ oid => '4310', descr => 'internal conversion function for KOI8R to WIN1251',
diff --git a/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out b/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out
index 1ee65ede243..39d43368c42 100644
--- a/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out
+++ b/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out
@@ -59,11 +59,12 @@ select * from t
binary_upgrade_set_next_multirange_pg_type_oid | 1 | binary_upgrade_set_next_multirange_pg_type_oid
binary_upgrade_set_next_pg_authid_oid | | binary_upgrade_set_next_pg_authid_oid
binary_upgrade_set_next_pg_enum_oid | | binary_upgrade_set_next_pg_enum_oid
+ binary_upgrade_set_next_pg_subscription_oid | | binary_upgrade_set_next_pg_subscription_oid
binary_upgrade_set_next_pg_tablespace_oid | | binary_upgrade_set_next_pg_tablespace_oid
binary_upgrade_set_next_pg_type_oid | | binary_upgrade_set_next_pg_type_oid
binary_upgrade_set_next_toast_pg_class_oid | 1 | binary_upgrade_set_next_toast_pg_class_oid
binary_upgrade_set_next_toast_relfilenode | | binary_upgrade_set_next_toast_relfilenode
-(13 rows)
+(14 rows)
-- Verify clean failure when INCLUDE'd columns result in overlength tuple
-- The error message details are platform-dependent, so show only SQLSTATE
@@ -108,11 +109,12 @@ select * from t
binary_upgrade_set_next_multirange_pg_type_oid | 1 | binary_upgrade_set_next_multirange_pg_type_oid
binary_upgrade_set_next_pg_authid_oid | | binary_upgrade_set_next_pg_authid_oid
binary_upgrade_set_next_pg_enum_oid | | binary_upgrade_set_next_pg_enum_oid
+ binary_upgrade_set_next_pg_subscription_oid | | binary_upgrade_set_next_pg_subscription_oid
binary_upgrade_set_next_pg_tablespace_oid | | binary_upgrade_set_next_pg_tablespace_oid
binary_upgrade_set_next_pg_type_oid | | binary_upgrade_set_next_pg_type_oid
binary_upgrade_set_next_toast_pg_class_oid | 1 | binary_upgrade_set_next_toast_pg_class_oid
binary_upgrade_set_next_toast_relfilenode | | binary_upgrade_set_next_toast_relfilenode
-(13 rows)
+(14 rows)
\set VERBOSITY sqlstate
insert into t values(repeat('xyzzy', 12), 42, repeat('xyzzy', 4000));
--
2.47.3
[application/octet-stream] v8-0002-Preserve-replication-origin-OIDs-during-pg_upgrad.patch (28.4K, ../../CAFPTHDbyVt_ggdi+rOP4=jBmgU=DDgYWuLxc4oQ6ux-T0H-ZBg@mail.gmail.com/3-v8-0002-Preserve-replication-origin-OIDs-during-pg_upgrad.patch)
download | inline diff:
From 1e750a58fd4e247364b7c606869aae67287f84a1 Mon Sep 17 00:00:00 2001
From: Ajin Cherian <[email protected]>
Date: Wed, 17 Jun 2026 18:16:13 +1000
Subject: [PATCH v8 2/2] Preserve replication origin OIDs during pg_upgrade
When pg_upgrade migrates a subscriber, replication origin OIDs
(roident) can change across the upgrade. This is a problem because
commit-timestamp records embed roident and are copied directly from
the old cluster's pg_commit_ts directory, causing spurious
"update_origin_differs" conflicts after the upgrade.
Fix this by dumping replication origins as global objects via
pg_dumpall during binary upgrade, using a new function
binary_upgrade_create_replication_origin(oid, name, lsn) to recreate
each origin with its preserved roident and remote_lsn. To avoid
conflicts with this, CreateSubscription() skips replorigin_create()
in binary-upgrade mode since the origin is already created by the
time the subscription is restored.
Author: Ajin Cherian <[email protected]>
Reviewer: Hayato Kuroda (Fujitsu) <[email protected]>
Reviewer: Zsolt Parragi <[email protected]>
Reviewer: Shlok Kyal <[email protected]>
Reviewer: Shveta malik <[email protected]>
Reviewer: Vignesh C <[email protected]>
---
doc/src/sgml/logical-replication.sgml | 2 +-
src/backend/commands/subscriptioncmds.c | 11 +-
src/backend/replication/logical/origin.c | 117 +++++++++++++++------
src/backend/utils/adt/pg_upgrade_support.c | 88 +++++++---------
src/bin/pg_dump/pg_dump.c | 38 ++-----
src/bin/pg_dump/pg_dumpall.c | 64 +++++++++++
src/bin/pg_upgrade/check.c | 54 ++++++----
src/bin/pg_upgrade/info.c | 7 ++
src/bin/pg_upgrade/pg_upgrade.h | 1 +
src/bin/pg_upgrade/t/004_subscription.pl | 42 +++++++-
src/include/catalog/pg_proc.dat | 8 +-
src/include/replication/origin.h | 2 +
12 files changed, 293 insertions(+), 141 deletions(-)
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index 9e7868487de..0e484800fca 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -2819,7 +2819,7 @@ CONTEXT: processing remote data for replication origin "pg_16395" during "INSER
<note>
<para>
- Commit timestamps and origin data are not preserved during the upgrade.
+ Commit timestamps are not preserved during the upgrade.
As a result, even if
<link linkend="sql-createsubscription-params-with-retain-dead-tuples"><literal>retain_dead_tuples</literal></link>
is enabled, the upgraded subscriber may be unable to detect conflicts or
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 71386fa7d9d..81b0ce22ee9 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -19,6 +19,7 @@
#include "access/table.h"
#include "access/twophase.h"
#include "access/xact.h"
+#include "catalog/binary_upgrade.h"
#include "catalog/catalog.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
@@ -867,9 +868,15 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
* apply workers initialization, and to handle origin creation dynamically
* when tables are added to the subscription. It is not clear whether
* preventing creation of origins is worth additional complexity.
+ *
+ * In binary-upgrade mode, skip origin creation here. This is required to
+ * preserve the roident from the old cluster for this subscription.
*/
- ReplicationOriginNameForLogicalRep(subid, InvalidOid, originname, sizeof(originname));
- replorigin_create(originname);
+ if (!IsBinaryUpgrade)
+ {
+ ReplicationOriginNameForLogicalRep(subid, InvalidOid, originname, sizeof(originname));
+ replorigin_create(originname);
+ }
/*
* Connect to remote side to execute requested commands and fetch table
diff --git a/src/backend/replication/logical/origin.c b/src/backend/replication/logical/origin.c
index c9dfb094c2b..b5b87f3e93b 100644
--- a/src/backend/replication/logical/origin.c
+++ b/src/backend/replication/logical/origin.c
@@ -265,6 +265,73 @@ replorigin_by_name(const char *roname, bool missing_ok)
return roident;
}
+/*
+ * replorigin_create_with_id
+ *
+ * Create a replication origin with a specific ID and name, optionally
+ * restoring its remote_lsn. Used by pg_upgrade to preserve replication
+ * origin IDs across the upgrade.
+ *
+ * Needs to be called in a transaction.
+ */
+void
+replorigin_create_with_id(ReplOriginId roident, const char *roname,
+ XLogRecPtr remote_lsn)
+{
+ Datum roname_d;
+ bool nulls[Natts_pg_replication_origin];
+ Datum values[Natts_pg_replication_origin];
+ HeapTuple tuple;
+ Relation rel;
+
+ Assert(IsTransactionState());
+
+ /*
+ * To avoid needing a TOAST table for pg_replication_origin, we limit
+ * replication origin names to 512 bytes. This should be more than enough
+ * for all practical use.
+ */
+ if (strlen(roname) > MAX_RONAME_LEN)
+ ereport(ERROR,
+ errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
+ errmsg("replication origin name is too long"),
+ errdetail("Replication origin names must be no longer than %d bytes.",
+ MAX_RONAME_LEN));
+
+ roname_d = CStringGetTextDatum(roname);
+
+ if (SearchSysCacheExists1(REPLORIGIDENT, ObjectIdGetDatum(roident)))
+ ereport(ERROR,
+ errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("replication origin with ID %u already exists",
+ (Oid) roident));
+
+ if (SearchSysCacheExists1(REPLORIGNAME, roname_d))
+ ereport(ERROR,
+ errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("replication origin \"%s\" already exists", roname));
+
+ rel = table_open(ReplicationOriginRelationId, RowExclusiveLock);
+
+ memset(&nulls, 0, sizeof(nulls));
+ memset(&values, 0, sizeof(values));
+
+ values[Anum_pg_replication_origin_roident - 1] = ObjectIdGetDatum(roident);
+ values[Anum_pg_replication_origin_roname - 1] = roname_d;
+
+ tuple = heap_form_tuple(RelationGetDescr(rel), values, nulls);
+ CatalogTupleInsert(rel, tuple);
+ heap_freetuple(tuple);
+ CommandCounterIncrement();
+
+ if (remote_lsn != InvalidXLogRecPtr)
+ replorigin_advance(roident, remote_lsn, InvalidXLogRecPtr,
+ false /* backward */,
+ false /* WAL log */);
+
+ table_close(rel, RowExclusiveLock);
+}
+
/*
* Create a replication origin.
*
@@ -273,13 +340,12 @@ replorigin_by_name(const char *roname, bool missing_ok)
ReplOriginId
replorigin_create(const char *roname)
{
- Oid roident;
- HeapTuple tuple = NULL;
- Relation rel;
- Datum roname_d;
- SnapshotData SnapshotDirty;
- SysScanDesc scan;
- ScanKeyData key;
+ Oid roident;
+ Relation rel;
+ SnapshotData SnapshotDirty;
+ SysScanDesc scan;
+ ScanKeyData key;
+ bool found = false;
/*
* To avoid needing a TOAST table for pg_replication_origin, we limit
@@ -293,8 +359,6 @@ replorigin_create(const char *roname)
errdetail("Replication origin names must be no longer than %d bytes.",
MAX_RONAME_LEN)));
- roname_d = CStringGetTextDatum(roname);
-
Assert(IsTransactionState());
/*
@@ -321,17 +385,15 @@ replorigin_create(const char *roname)
* snapshot. To make that safe, it needs to not have a TOAST table, since
* TOASTed data cannot be fetched without a snapshot. As of this writing,
* its only varlena column is roname, which we limit to 512 bytes to avoid
- * needing out-of-line storage. If you add a TOAST table to this catalog,
- * be sure to set up a snapshot everywhere it might be needed. For more
+ * needing out-of-line storage. If you add a TOAST table to this catalog,
+ * be sure to set up a snapshot everywhere it might be needed. For more
* information, see https://postgr.es/m/ZvMSUPOqUU-VNADN%40nathan.
*/
Assert(!OidIsValid(rel->rd_rel->reltoastrelid));
for (roident = InvalidOid + 1; roident < PG_UINT16_MAX; roident++)
{
- bool nulls[Natts_pg_replication_origin];
- Datum values[Natts_pg_replication_origin];
- bool collides;
+ bool collides;
CHECK_FOR_INTERRUPTS();
@@ -341,41 +403,28 @@ replorigin_create(const char *roname)
ObjectIdGetDatum(roident));
scan = systable_beginscan(rel, ReplicationOriginIdentIndex,
- true /* indexOK */ ,
+ true /* indexOK */,
&SnapshotDirty,
1, &key);
-
collides = HeapTupleIsValid(systable_getnext(scan));
-
systable_endscan(scan);
if (!collides)
{
- /*
- * Ok, found an unused roident, insert the new row and do a CCI,
- * so our callers can look it up if they want to.
- */
- memset(&nulls, 0, sizeof(nulls));
-
- values[Anum_pg_replication_origin_roident - 1] = ObjectIdGetDatum(roident);
- values[Anum_pg_replication_origin_roname - 1] = roname_d;
-
- tuple = heap_form_tuple(RelationGetDescr(rel), values, nulls);
- CatalogTupleInsert(rel, tuple);
- CommandCounterIncrement();
+ found = true;
break;
}
}
- /* now release lock again, */
- table_close(rel, ExclusiveLock);
-
- if (tuple == NULL)
+ if (!found)
ereport(ERROR,
(errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
errmsg("could not find free replication origin ID")));
- heap_freetuple(tuple);
+ table_close(rel, ExclusiveLock);
+
+ replorigin_create_with_id(roident, roname, InvalidXLogRecPtr);
+
return roident;
}
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index 59c3e7f0146..16b7408416e 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -11,6 +11,7 @@
#include "postgres.h"
+#include "access/genam.h"
#include "access/relation.h"
#include "access/table.h"
#include "catalog/binary_upgrade.h"
@@ -27,8 +28,10 @@
#include "storage/lmgr.h"
#include "utils/array.h"
#include "utils/builtins.h"
+#include "utils/fmgroids.h"
#include "utils/lsyscache.h"
#include "utils/pg_lsn.h"
+#include "utils/snapmgr.h"
#define CHECK_IS_BINARY_UPGRADE \
@@ -377,71 +380,60 @@ binary_upgrade_add_sub_rel_state(PG_FUNCTION_ARGS)
}
/*
- * binary_upgrade_replorigin_advance
+ * binary_upgrade_create_conflict_detection_slot
*
- * Update the remote_lsn for the subscriber's replication origin.
+ * Create a replication slot to retain information necessary for conflict
+ * detection such as dead tuples, commit timestamps, and origins.
*/
Datum
-binary_upgrade_replorigin_advance(PG_FUNCTION_ARGS)
+binary_upgrade_create_conflict_detection_slot(PG_FUNCTION_ARGS)
{
- Relation rel;
- Oid subid;
- char *subname;
- char originname[NAMEDATALEN];
- ReplOriginId node;
- XLogRecPtr remote_commit;
-
CHECK_IS_BINARY_UPGRADE;
- /*
- * We must ensure a non-NULL subscription name before dereferencing the
- * arguments.
- */
- if (PG_ARGISNULL(0))
- elog(ERROR, "null argument to binary_upgrade_replorigin_advance is not allowed");
-
- subname = text_to_cstring(PG_GETARG_TEXT_PP(0));
- remote_commit = PG_ARGISNULL(1) ? InvalidXLogRecPtr : PG_GETARG_LSN(1);
-
- rel = table_open(SubscriptionRelationId, RowExclusiveLock);
- subid = get_subscription_oid(subname, false);
-
- ReplicationOriginNameForLogicalRep(subid, InvalidOid, originname, sizeof(originname));
-
- /* Lock to prevent the replication origin from vanishing */
- LockRelationOid(ReplicationOriginRelationId, RowExclusiveLock);
- node = replorigin_by_name(originname, false);
-
- /*
- * The server will be stopped after setting up the objects in the new
- * cluster and the origins will be flushed during the shutdown checkpoint.
- * This will ensure that the latest LSN values for origin will be
- * available after the upgrade.
- */
- replorigin_advance(node, remote_commit, InvalidXLogRecPtr,
- false /* backward */ ,
- false /* WAL log */ );
+ CreateConflictDetectionSlot();
- UnlockRelationOid(ReplicationOriginRelationId, RowExclusiveLock);
- table_close(rel, RowExclusiveLock);
+ ReplicationSlotRelease();
PG_RETURN_VOID();
}
/*
- * binary_upgrade_create_conflict_detection_slot
+ * binary_upgrade_create_replication_origin
*
- * Create a replication slot to retain information necessary for conflict
- * detection such as dead tuples, commit timestamps, and origins.
+ * Create a replication origin with a specific OID and name, optionally
+ * restoring its remote_lsn. Used by pg_upgrade to preserve replication
+ * origin OIDs across the upgrade.
*/
Datum
-binary_upgrade_create_conflict_detection_slot(PG_FUNCTION_ARGS)
+binary_upgrade_create_replication_origin(PG_FUNCTION_ARGS)
{
- CHECK_IS_BINARY_UPGRADE;
+ Oid node_oid;
+ ReplOriginId node;
+ char *originname;
+ XLogRecPtr remote_lsn = InvalidXLogRecPtr;
- CreateConflictDetectionSlot();
+ CHECK_IS_BINARY_UPGRADE;
- ReplicationSlotRelease();
+ if (PG_ARGISNULL(0) || PG_ARGISNULL(1))
+ elog(ERROR,
+ "null argument to binary_upgrade_create_replication_origin is not allowed");
- PG_RETURN_VOID();
+ node_oid = PG_GETARG_OID(0);
+
+ if (node_oid == InvalidOid || node_oid > PG_UINT16_MAX)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("replication origin ID %u is out of range", node_oid));
+
+ node = (ReplOriginId) node_oid;
+ originname = text_to_cstring(PG_GETARG_TEXT_PP(1));
+
+ if (!PG_ARGISNULL(2))
+ remote_lsn = PG_GETARG_LSN(2);
+
+ Assert(IsTransactionState());
+
+ replorigin_create_with_id(node, originname, remote_lsn);
+
+ PG_RETURN_VOID();
}
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 049337de62a..fd13e165f7b 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -5668,37 +5668,15 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
* In binary-upgrade mode, we allow the replication to continue after the
* upgrade.
*/
- if (dopt->binary_upgrade && fout->remoteVersion >= 170000)
+ if (dopt->binary_upgrade && subinfo->subenabled && fout->remoteVersion >= 170000)
{
- if (subinfo->suboriginremotelsn)
- {
- /*
- * Preserve the remote_lsn for the subscriber's replication
- * origin. This value is required to start the replication from
- * the position before the upgrade. This value will be stale if
- * the publisher gets upgraded before the subscriber node.
- * However, this shouldn't be a problem as the upgrade of the
- * publisher ensures that all the transactions were replicated
- * before upgrading it.
- */
- appendPQExpBufferStr(query,
- "\n-- For binary upgrade, must preserve the remote_lsn for the subscriber's replication origin.\n");
- appendPQExpBufferStr(query,
- "SELECT pg_catalog.binary_upgrade_replorigin_advance(");
- appendStringLiteralAH(query, subinfo->dobj.name, fout);
- appendPQExpBuffer(query, ", '%s');\n", subinfo->suboriginremotelsn);
- }
-
- if (subinfo->subenabled)
- {
- /*
- * Enable the subscription to allow the replication to continue
- * after the upgrade.
- */
- appendPQExpBufferStr(query,
- "\n-- For binary upgrade, must preserve the subscriber's running state.\n");
- appendPQExpBuffer(query, "ALTER SUBSCRIPTION %s ENABLE;\n", qsubname);
- }
+ /*
+ * Enable the subscription to allow the replication to continue
+ * after the upgrade.
+ */
+ appendPQExpBufferStr(query,
+ "\n-- For binary upgrade, must preserve the subscriber's running state.\n");
+ appendPQExpBuffer(query, "ALTER SUBSCRIPTION %s ENABLE;\n", qsubname);
}
if (subinfo->dobj.dump & DUMP_COMPONENT_DEFINITION)
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index c1f43113c53..b72dcb18c8f 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -25,6 +25,7 @@
#include <time.h>
#include <unistd.h>
+#include "access/xlogdefs.h"
#include "catalog/pg_authid_d.h"
#include "common/connect.h"
#include "common/file_perm.h"
@@ -76,6 +77,7 @@ static void dropDBs(PGconn *conn);
static void dumpUserConfig(PGconn *conn, const char *username);
static void dumpDatabases(PGconn *conn);
static void dumpTimestamp(const char *msg);
+static void dumpReplicationOrigins(PGconn *conn);
static int runPgDump(const char *dbname, const char *create_opts, char *dbfile);
static void buildShSecLabels(PGconn *conn,
const char *catalog_name, Oid objectId,
@@ -813,6 +815,10 @@ main(int argc, char *argv[])
/* Dump role GUC privileges */
if (server_version >= 150000 && !skip_acls)
dumpRoleGUCPrivs(conn);
+
+ /* Dump replication origins */
+ if (binary_upgrade && archDumpFormat == archNull)
+ dumpReplicationOrigins(conn);
}
/* Dump tablespaces */
@@ -2339,6 +2345,64 @@ dumpTimestamp(const char *msg)
fprintf(OPF, "-- %s %s\n\n", msg, buf);
}
+static void
+dumpReplicationOrigins(PGconn *conn)
+{
+ PQExpBuffer buf = createPQExpBuffer();
+ PGresult *res;
+ int i_roident;
+ int i_roname;
+ int i_remotelsn;
+
+ /* Get replication origins from catalogs */
+ appendPQExpBufferStr(buf,
+ "SELECT o.*, os.remote_lsn "
+ "FROM pg_catalog.pg_replication_origin o "
+ "LEFT OUTER JOIN pg_catalog.pg_replication_origin_status os ON o.roident = os.local_id ");
+
+ res = executeQuery(conn, buf->data);
+
+ i_roident = PQfnumber(res, "roident");
+ i_roname = PQfnumber(res, "roname");
+ i_remotelsn = PQfnumber(res, "remote_lsn");
+
+ if (PQntuples(res) > 0)
+ fprintf(OPF, "--\n-- Replication Origins \n--\n\n");
+
+ for (int i = 0; i < PQntuples(res); i++)
+ {
+ ReplOriginId roident;
+ const char *roname;
+
+ roident = atooid(PQgetvalue(res, i, i_roident));
+ roname = PQgetvalue(res, i, i_roname);
+
+ resetPQExpBuffer(buf);
+
+ appendPQExpBufferStr(buf, "\n-- For binary upgrade, must preserve replication origin roident and remote_lsn\n");
+ appendPQExpBuffer(buf,
+ "SELECT pg_catalog.binary_upgrade_create_replication_origin("
+ "'%u'::pg_catalog.oid, ", roident);
+ appendStringLiteralConn(buf, roname, conn);
+ appendPQExpBufferStr(buf, "::pg_catalog.text");
+
+ if (!PQgetisnull(res, i, i_remotelsn))
+ {
+ appendPQExpBufferStr(buf, ", ");
+ appendStringLiteralConn(buf, PQgetvalue(res, i, i_remotelsn), conn);
+ appendPQExpBufferStr(buf, "::pg_catalog.pg_lsn");
+ }
+ else
+ appendPQExpBufferStr(buf, ", NULL");
+
+ appendPQExpBufferStr(buf, ");\n");
+ fprintf(OPF, "%s", buf->data);
+ }
+
+ PQclear(res);
+ destroyPQExpBuffer(buf);
+}
+
/*
* read_dumpall_filters - retrieve database identifier patterns from file
*
diff --git a/src/bin/pg_upgrade/check.c b/src/bin/pg_upgrade/check.c
index f5c93e611d2..2ea2be2f422 100644
--- a/src/bin/pg_upgrade/check.c
+++ b/src/bin/pg_upgrade/check.c
@@ -33,7 +33,7 @@ static void check_for_new_tablespace_dir(void);
static void check_for_user_defined_encoding_conversions(ClusterInfo *cluster);
static void check_for_unicode_update(ClusterInfo *cluster);
static void check_new_cluster_replication_slots(void);
-static void check_new_cluster_subscription_configuration(void);
+static void check_new_cluster_replication_origins(void);
static void check_old_cluster_for_valid_slots(void);
static void check_old_cluster_subscription_state(void);
static void check_old_cluster_global_names(ClusterInfo *cluster);
@@ -797,7 +797,8 @@ check_new_cluster(void)
check_new_cluster_replication_slots();
- check_new_cluster_subscription_configuration();
+ check_new_cluster_replication_origins();
+
}
@@ -2303,31 +2304,45 @@ check_new_cluster_replication_slots(void)
}
/*
- * check_new_cluster_subscription_configuration()
+ * check_new_cluster_replication_origins()
+ *
+ * Verify that the new cluster has no replication origins. During upgrade,
+ * pg_upgrade restores replication origins from the old cluster with their
+ * original OIDs. If the new cluster already contains origins, those OIDs
+ * may collide, causing the upgrade to fail mid-way.
*
- * Verify that the max_active_replication_origins configuration specified is
- * enough for creating the subscriptions. This is required to create the
- * replication origin for each subscription.
+ * Also verify that the max_active_replication_origins configuration is
+ * enough for creating all the replication origins.
*/
static void
-check_new_cluster_subscription_configuration(void)
+check_new_cluster_replication_origins(void)
{
- PGresult *res;
PGconn *conn;
+ PGresult *res;
+ int norigins;
int max_active_replication_origins;
- /* Subscriptions and their dependencies can be migrated since PG17. */
- if (GET_MAJOR_VERSION(old_cluster.major_version) < 1700)
+ /* Quick return if there are no replication origins to migrate. */
+ if (old_cluster.nrepl_origins == 0)
return;
- /* Quick return if there are no subscriptions to be migrated. */
- if (old_cluster.nsubs == 0)
- return;
-
- prep_status("Checking new cluster configuration for subscriptions");
+ prep_status("Checking replication origins in new cluster");
conn = connectToServer(&new_cluster, "template1");
+ res = executeQueryOrDie(conn,
+ "SELECT count(*) "
+ "FROM pg_catalog.pg_replication_origin");
+
+ if (PQntuples(res) != 1)
+ pg_fatal("could not count the number of replication origins");
+
+ norigins = atoi(PQgetvalue(res, 0, 0));
+ PQclear(res);
+
+ if (norigins > 0)
+ pg_fatal("expected 0 replication origins but found %d", norigins);
+
res = executeQueryOrDie(conn, "SELECT setting FROM pg_settings "
"WHERE name = 'max_active_replication_origins';");
@@ -2335,12 +2350,13 @@ check_new_cluster_subscription_configuration(void)
pg_fatal("could not determine parameter settings on new cluster");
max_active_replication_origins = atoi(PQgetvalue(res, 0, 0));
- if (old_cluster.nsubs > max_active_replication_origins)
+ PQclear(res);
+
+ if (old_cluster.nrepl_origins > max_active_replication_origins)
pg_fatal("\"max_active_replication_origins\" (%d) must be greater than or equal to the number of "
- "subscriptions (%d) on the old cluster",
- max_active_replication_origins, old_cluster.nsubs);
+ "replication origins (%d) on the old cluster",
+ max_active_replication_origins, old_cluster.nrepl_origins);
- PQclear(res);
PQfinish(conn);
check_ok();
diff --git a/src/bin/pg_upgrade/info.c b/src/bin/pg_upgrade/info.c
index 37fff93892f..0ed8625eb6b 100644
--- a/src/bin/pg_upgrade/info.c
+++ b/src/bin/pg_upgrade/info.c
@@ -862,6 +862,13 @@ get_subscription_info(ClusterInfo *cluster)
cluster->sub_retain_dead_tuples = (strcmp(PQgetvalue(res, 0, i_retain_dead_tuples), "t") == 0);
PQclear(res);
+
+ res = executeQueryOrDie(conn,
+ "SELECT count(*) AS nrepl_origins "
+ "FROM pg_catalog.pg_replication_origin");
+ cluster->nrepl_origins = atoi(PQgetvalue(res, 0, 0));
+ PQclear(res);
+
PQfinish(conn);
}
diff --git a/src/bin/pg_upgrade/pg_upgrade.h b/src/bin/pg_upgrade/pg_upgrade.h
index ccd1ac0d013..77e7ca1b4cd 100644
--- a/src/bin/pg_upgrade/pg_upgrade.h
+++ b/src/bin/pg_upgrade/pg_upgrade.h
@@ -311,6 +311,7 @@ typedef struct
int num_tablespaces;
const char *tablespace_suffix; /* directory specification */
int nsubs; /* number of subscriptions */
+ int nrepl_origins; /* number of replication origins */
bool sub_retain_dead_tuples; /* whether a subscription enables
* retain_dead_tuples. */
} ClusterInfo;
diff --git a/src/bin/pg_upgrade/t/004_subscription.pl b/src/bin/pg_upgrade/t/004_subscription.pl
index 646767f2a65..e8b11d39dd0 100644
--- a/src/bin/pg_upgrade/t/004_subscription.pl
+++ b/src/bin/pg_upgrade/t/004_subscription.pl
@@ -42,7 +42,7 @@ my $connstr = $publisher->connstr . ' dbname=postgres';
# ------------------------------------------------------
# Check that pg_upgrade fails when max_active_replication_origins configured
-# in the new cluster is less than the number of subscriptions in the old
+# in the new cluster is less than the number of replication origins in the old
# cluster.
# ------------------------------------------------------
# It is sufficient to use disabled subscription to test upgrade failure.
@@ -74,7 +74,7 @@ command_checks_all(
],
1,
[
- qr/"max_active_replication_origins" \(0\) must be greater than or equal to the number of subscriptions \(1\) on the old cluster/
+ qr/"max_active_replication_origins" \(0\) must be greater than or equal to the number of replication origins \(1\) on the old cluster/
],
[qr//],
'run of pg_upgrade where the new cluster has insufficient max_active_replication_origins'
@@ -301,8 +301,30 @@ is($result, qq(t), "Check that the table is in init state");
# Get the replication origin's remote_lsn of the old subscriber
my $remote_lsn = $old_sub->safe_psql('postgres',
- "SELECT remote_lsn FROM pg_replication_origin_status os, pg_subscription s WHERE os.external_id = 'pg_' || s.oid AND s.subname = 'regress_sub4'"
+ "SELECT os.remote_lsn
+ FROM pg_replication_origin_status os
+ JOIN pg_replication_origin o ON o.roident = os.local_id
+ JOIN pg_subscription s ON o.roname = 'pg_' || s.oid::text
+ WHERE s.subname = 'regress_sub4'"
);
+
+# Get the replication origin OIDs (roident) for all subscriptions, keyed by
+# subscription name (which is stable across upgrade, unlike suboid). These
+# must be preserved after upgrade. A mismatch would cause spurious
+# update_origin_differs conflicts.
+my %pre_upgrade_roident;
+my $roident_rows = $old_sub->safe_psql('postgres',
+ "SELECT s.subname, o.roident
+ FROM pg_subscription s
+ JOIN pg_replication_origin o ON o.roname = 'pg_' || s.oid::text
+ ORDER BY s.subname"
+);
+for my $row (split /\n/, $roident_rows)
+{
+ my ($subname, $roident) = split /\|/, $row;
+ $pre_upgrade_roident{$subname} = $roident;
+}
+
# Have the subscription in disabled state before upgrade
$old_sub->safe_psql('postgres', "ALTER SUBSCRIPTION regress_sub5 DISABLE");
@@ -378,6 +400,20 @@ regress_sub5|f|f|f),
"check that the subscription's running status, failover, and retain_dead_tuples are preserved"
);
+# Verify that replication origin OIDs are preserved after upgrade.
+my $post_roident_rows = $new_sub->safe_psql('postgres',
+ "SELECT s.subname, o.roident
+ FROM pg_subscription s
+ JOIN pg_replication_origin o ON o.roname = 'pg_' || s.oid::text
+ ORDER BY s.subname"
+);
+for my $row (split /\n/, $post_roident_rows)
+{
+ my ($subname, $roident) = split /\|/, $row;
+ is($roident, $pre_upgrade_roident{$subname},
+ "roident preserved for subscription '$subname' after upgrade");
+}
+
# Subscription relations should be preserved
$result = $new_sub->safe_psql('postgres',
"SELECT srrelid, srsubstate FROM pg_subscription_rel ORDER BY srrelid");
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 3a28406981d..21466d926f1 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -11960,10 +11960,6 @@
provolatile => 'v', proparallel => 'u', prorettype => 'void',
proargtypes => 'text oid char pg_lsn',
prosrc => 'binary_upgrade_add_sub_rel_state' },
-{ oid => '6320', descr => 'for use by pg_upgrade (remote_lsn for origin)',
- proname => 'binary_upgrade_replorigin_advance', proisstrict => 'f',
- provolatile => 'v', proparallel => 'u', prorettype => 'void',
- proargtypes => 'text pg_lsn', prosrc => 'binary_upgrade_replorigin_advance' },
{ oid => '6505', descr => 'for use by pg_upgrade (conflict detection slot)',
proname => 'binary_upgrade_create_conflict_detection_slot',
proisstrict => 'f', provolatile => 'v', proparallel => 'u',
@@ -11973,6 +11969,10 @@
proname => 'binary_upgrade_set_next_pg_subscription_oid', provolatile => 'v',
proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
prosrc => 'binary_upgrade_set_next_pg_subscription_oid' },
+{ oid => '9161', descr => 'for use by pg_upgrade (replication origin)',
+ proname => 'binary_upgrade_create_replication_origin', proisstrict => 'f',
+ provolatile => 'v', proparallel => 'u', prorettype => 'void',
+ proargtypes => 'oid text pg_lsn', prosrc => 'binary_upgrade_create_replication_origin' },
# conversion functions
{ oid => '4310', descr => 'internal conversion function for KOI8R to WIN1251',
diff --git a/src/include/replication/origin.h b/src/include/replication/origin.h
index a69faf6eaaf..6b25a967fe9 100644
--- a/src/include/replication/origin.h
+++ b/src/include/replication/origin.h
@@ -55,6 +55,8 @@ extern PGDLLIMPORT int max_active_replication_origins;
/* API for querying & manipulating replication origins */
extern ReplOriginId replorigin_by_name(const char *roname, bool missing_ok);
extern ReplOriginId replorigin_create(const char *roname);
+extern void replorigin_create_with_id(ReplOriginId roident, const char *roname,
+ XLogRecPtr remote_lsn);
extern void replorigin_drop_by_name(const char *name, bool missing_ok, bool nowait);
extern bool replorigin_by_oid(ReplOriginId roident, bool missing_ok,
char **roname);
--
2.47.3
^ permalink raw reply [nested|flat] 97+ messages in thread
* Re: [PATCH] Preserve replication origin OIDs in pg_upgrade
@ 2026-06-17 10:59 Nisha Moond <[email protected]>
parent: Ajin Cherian <[email protected]>
2 siblings, 1 reply; 97+ messages in thread
From: Nisha Moond @ 2026-06-17 10:59 UTC (permalink / raw)
To: Ajin Cherian <[email protected]>; +Cc: Shlok Kyal <[email protected]>; shveta malik <[email protected]>; Zsolt Parragi <[email protected]>; vignesh C <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; PostgreSQL Hackers <[email protected]>; shveta malik <[email protected]>
On Wed, Jun 17, 2026 at 2:49 PM Ajin Cherian <[email protected]> wrote:
>
> I have addressed the comments in v8 (attached). I've also now taken
> out the check for PG17 on old cluster and now all clusters which have
> replication origins are migrated.
Hi Ajin, Thanks for the patches.
I have started reviewing the patches. Please find a few initial
comments below (from v7 review, still applicable to v8):
1) Patch-002: pg_dump.c
+ if (dopt->binary_upgrade && subinfo->subenabled &&
fout->remoteVersion >= 170000)
{
- if (subinfo->suboriginremotelsn)
- {
- /*
- * Preserve the remote_lsn for the subscriber's replication
I could not find any usage of subinfo[i].suboriginremotelsn anymore.
It seems to be dead code now. Can we remove it?
~~~
2) Patch-002 : binary_upgrade_create_replication_origin()
- PG_RETURN_VOID();
+ node_oid = PG_GETARG_OID(0);
+
+ if (node_oid == InvalidOid || node_oid > PG_UINT16_MAX)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("replication origin ID %u is out of range", node_oid));
+
The above range check allows PG_UINT16_MAX, but that value is not part
of the valid replication origin ID range.
In origin.h:
#define DoNotReplicateId PG_UINT16_MAX
And we already ensure "ReplOriginId != DoNotReplicateId" everywhere
else. Would something like the below be more appropriate?
if (node_oid == InvalidReplOriginId || node_oid >= DoNotReplicateId)
ereport(ERROR, ...
~~~
3) I think we should add some additional details in
logical-replication.sgml, especially under <title>Prepare for
Subscriber Upgrades</title>.
Some suggestions:
3a) Since all replication origins are now migrated, this:
"...greater than or equal to the number of subscriptions present in
the old cluster."
could become:
"...greater than or equal to the number of replication origins
present in the old cluster."
3b) We should also mention:
The new cluster must contain no replication origins.
--
Thanks,
Nisha
^ permalink raw reply [nested|flat] 97+ messages in thread
* Re: [PATCH] Preserve replication origin OIDs in pg_upgrade
@ 2026-06-17 11:42 Nisha Moond <[email protected]>
parent: Nisha Moond <[email protected]>
0 siblings, 1 reply; 97+ messages in thread
From: Nisha Moond @ 2026-06-17 11:42 UTC (permalink / raw)
To: Ajin Cherian <[email protected]>; +Cc: Shlok Kyal <[email protected]>; shveta malik <[email protected]>; Zsolt Parragi <[email protected]>; vignesh C <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; PostgreSQL Hackers <[email protected]>; shveta malik <[email protected]>
On Wed, Jun 17, 2026 at 4:29 PM Nisha Moond <[email protected]> wrote:
>
> On Wed, Jun 17, 2026 at 2:49 PM Ajin Cherian <[email protected]> wrote:
> >
> > I have addressed the comments in v8 (attached). I've also now taken
> > out the check for PG17 on old cluster and now all clusters which have
> > replication origins are migrated.
>
> Hi Ajin, Thanks for the patches.
> I have started reviewing the patches. Please find a few initial
> comments below (from v7 review, still applicable to v8):
Please find a couple more comments for the v8-002 patch:
4) I was able to build successfully without all three new header
inclusions below, so we may not need them.
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -11,6 +11,7 @@
+#include "access/genam.h"
...
+#include "utils/fmgroids.h"
...
+#include "utils/snapmgr.h"
~~~
5) Race condition in origin.c: replorigin_create()
- heap_freetuple(tuple);
+ table_close(rel, ExclusiveLock);
+
+ replorigin_create_with_id(roident, roname, InvalidXLogRecPtr);
+
Due to the above refactoring, CREATE SUBSCRIPTION can now fail because
of a race between two concurrent CREATE SUBSCRIPTION commands:
postgres=# create subscription sub2 connection 'dbname=postgres
host=localhost port=7733' publication pub1;
ERROR: replication origin with ID 2 already exists
I think this happens because the table lock is released, allowing
another session to create a pg_origin with the next available roident,
which replorigin_create_with_id() later attempts to use.
Would it make sense to call replorigin_create_with_id() before
table_close() so the lock is still held? I tested this and it works,
though I'm not fully sure whether self-locking could be a concern
here, since replorigin_create_with_id() again acquires a
RowExclusiveLock on the same table.
~~~
--
Thanks,
Nisha
^ permalink raw reply [nested|flat] 97+ messages in thread
* Re: [PATCH] Preserve replication origin OIDs in pg_upgrade
@ 2026-06-18 09:16 shveta malik <[email protected]>
parent: Nisha Moond <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: shveta malik @ 2026-06-18 09:16 UTC (permalink / raw)
To: Nisha Moond <[email protected]>; +Cc: Ajin Cherian <[email protected]>; Shlok Kyal <[email protected]>; Zsolt Parragi <[email protected]>; vignesh C <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; PostgreSQL Hackers <[email protected]>; shveta malik <[email protected]>
On Wed, Jun 17, 2026 at 5:12 PM Nisha Moond <[email protected]> wrote:
>
> On Wed, Jun 17, 2026 at 4:29 PM Nisha Moond <[email protected]> wrote:
> >
> > On Wed, Jun 17, 2026 at 2:49 PM Ajin Cherian <[email protected]> wrote:
> > >
> > > I have addressed the comments in v8 (attached). I've also now taken
> > > out the check for PG17 on old cluster and now all clusters which have
> > > replication origins are migrated.
> >
> > Hi Ajin, Thanks for the patches.
> > I have started reviewing the patches. Please find a few initial
> > comments below (from v7 review, still applicable to v8):
>
> Please find a couple more comments for the v8-002 patch:
>
> 4) I was able to build successfully without all three new header
> inclusions below, so we may not need them.
> --- a/src/backend/utils/adt/pg_upgrade_support.c
> +++ b/src/backend/utils/adt/pg_upgrade_support.c
> @@ -11,6 +11,7 @@
> +#include "access/genam.h"
> ...
> +#include "utils/fmgroids.h"
> ...
> +#include "utils/snapmgr.h"
> ~~~
>
> 5) Race condition in origin.c: replorigin_create()
> - heap_freetuple(tuple);
> + table_close(rel, ExclusiveLock);
> +
> + replorigin_create_with_id(roident, roname, InvalidXLogRecPtr);
> +
>
> Due to the above refactoring, CREATE SUBSCRIPTION can now fail because
> of a race between two concurrent CREATE SUBSCRIPTION commands:
>
> postgres=# create subscription sub2 connection 'dbname=postgres
> host=localhost port=7733' publication pub1;
> ERROR: replication origin with ID 2 already exists
>
> I think this happens because the table lock is released, allowing
> another session to create a pg_origin with the next available roident,
> which replorigin_create_with_id() later attempts to use.
>
> Would it make sense to call replorigin_create_with_id() before
> table_close() so the lock is still held? I tested this and it works,
> though I'm not fully sure whether self-locking could be a concern
> here, since replorigin_create_with_id() again acquires a
> RowExclusiveLock on the same table.
We can handle this race-condition by moving the table-open to callers
rather than internal call replorigin_create_with_id(). But we also
need the following changes first:
1)
replorigin_create_with_id has:
+ /*
+ * To avoid needing a TOAST table for pg_replication_origin, we limit
+ * replication origin names to 512 bytes. This should be more than enough
+ * for all practical use.
+ */
+ if (strlen(roname) > MAX_RONAME_LEN)
+ ereport(ERROR,
+ errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
+ errmsg("replication origin name is too long"),
+ errdetail("Replication origin names must be no longer than %d bytes.",
+ MAX_RONAME_LEN));
replorigin_create_with_id() is called by replorigin_create() and
binary_upgrade_create_replication_origin(). replorigin_create()
already has this check. We should move above check to
binary_upgrade_create_replication_origin() and get rid of it from
internal function replorigin_create_with_id().
2)
Also the checks:
+ if (SearchSysCacheExists1(REPLORIGIDENT, ObjectIdGetDatum(roident)))
+ ereport(ERROR,
+ errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("replication origin with ID %u already exists",
+ (Oid) roident));
+
+ if (SearchSysCacheExists1(REPLORIGNAME, roname_d))
+ ereport(ERROR,
+ errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("replication origin \"%s\" already exists", roname));
should be moved to caller binary_upgrade_create_replication_origin().
IMO, these checks are not needed for replorigin_create(), it alread
has this, which should suffice.
ereport(ERROR,
(errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
errmsg("could not find free
replication origin ID")));
~~
Once we correctly place these checks, we can move the 'table_open with
lock' to caller (before name and id collision checks) instead of
replorigin_create_with_id(), which will take care of race-conditions
reported by Nisha.
thanks
Shveta
^ permalink raw reply [nested|flat] 97+ messages in thread
* Re: [PATCH] Preserve replication origin OIDs in pg_upgrade
@ 2026-06-19 03:59 shveta malik <[email protected]>
parent: Ajin Cherian <[email protected]>
1 sibling, 0 replies; 97+ messages in thread
From: shveta malik @ 2026-06-19 03:59 UTC (permalink / raw)
To: Ajin Cherian <[email protected]>; Amit Kapila <[email protected]>; +Cc: Shlok Kyal <[email protected]>; Dilip Kumar <[email protected]>; Zsolt Parragi <[email protected]>; vignesh C <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; PostgreSQL Hackers <[email protected]>; shveta malik <[email protected]>
On Tue, Jun 16, 2026 at 2:50 PM Ajin Cherian <[email protected]> wrote:
>
>
>
> On Mon, Jun 15, 2026 at 9:31 PM Amit Kapila <[email protected]> wrote:
>>
>>
>>
>> I'd prefer option-3 (preserve origin IDs uniformly, including upgrades
>> from PG16), for two reasons. (a) First, the new-cluster check ("reject
>> if the new cluster already has replication origins") is a restriction
>> on the target, and adding restrictions on a newer target version
>> regardless of source version is well within how pg_upgrade already
>> behaves. In practice the new cluster is a fresh initdb with no
>> origins, so the check effectively never fires; the only scenario it
>> rejects is one where the user manually created subscriptions/origins
>> on the new cluster before upgrading, which we discourage for other
>> object types anyway. So I don't think preserving existing behavior
>> here is worth much. (b) Second, option-2 reintroduces source-version
>> branching at dump time (the skip_subs_origin_creation flag gated on
>> source >= 17). Since pg_commit_ts is only copied from PG19+, the exact
>> roident value only matters from PG19; for PG16-18 the value is
>> don't-care, so preserving it everywhere is harmless where it isn't
>> needed and required where it is. That lets us keep a single
>> always-preserve path instead of branching per source version.
>>
>
> Ok.
>
>>
>> Separately, I don't think we actually need to preserve the
>> subscription OID to meet the goal here. pg_commit_ts stores the
>> numeric roident, not the origin name or the sub OID, and conflict
>> detection compares roidents. So the invariant we must keep is "each
>> subscription owns the same roident after upgrade", which doesn't
>> depend on the name being stable.
>>
>>
>
> Replication origins are global objects and are therefore dumped by pg_dumpall, while subscriptions are database-specific and are dumped by pg_dump. In pg_upgrade, pg_dumpall runs before pg_dump.
>
> The complication if subscription OID is not preserved is that subscription-associated origins derive their name from the subscription's OID. If the subscription OID is not preserved across upgrades, the origin name on the new cluster will differ from the old one, making it impossible to restore the origin independently. For these origins, creation must happen after CreateSubscription has established the new OID and not upfront in pg_dumpall.
>
> Non-subscription origins have no such dependency and continue to be created separately as before. The subscription code also cannot be left untouched: even once the subscription is created, rather than origin being created inside CreateSubscription, its associated origin must still be explicitly created via binary_upgrade_create_replication_origin to ensure the roident matches the original node, which means CreateSubscription needs to be modified anyways. So, there is an advantage of reducing code if subscription OID is preserved and all origins keep the same name as the old node.
>
IIUC, the conclusion (and what's implemented in v8) is:
1) The origin-ids are preserved on upgraded version irrespective of
source version.
2) Since origin-ids preservation requires sub-oid preservation due to
the origin-name's dependency on the sub-oid, we preserve sub-oid too
irrespective of source-version.
Let's see what Amit has to say about this.
thanks
Shveta
^ permalink raw reply [nested|flat] 97+ messages in thread
* Re: [PATCH] Preserve replication origin OIDs in pg_upgrade
@ 2026-06-19 07:48 Shlok Kyal <[email protected]>
parent: Ajin Cherian <[email protected]>
2 siblings, 0 replies; 97+ messages in thread
From: Shlok Kyal @ 2026-06-19 07:48 UTC (permalink / raw)
To: Ajin Cherian <[email protected]>; +Cc: shveta malik <[email protected]>; Zsolt Parragi <[email protected]>; vignesh C <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; PostgreSQL Hackers <[email protected]>; shveta malik <[email protected]>
On Wed, 17 Jun 2026 at 14:49, Ajin Cherian <[email protected]> wrote:
>
> On Fri, Jun 5, 2026 at 8:10 PM vignesh C <[email protected]> wrote:
> >
> > Few comments:
> > 1) pg_dumpall dumps all replication origins, not just subscription
> > origins and recreates every origin:
> > SELECT o.*, os.remote_lsn
> > FROM pg_replication_origin o
> > LEFT JOIN pg_replication_origin_status os
> > ON o.roident = os.local_id
> >
> > Is pg_upgrade expected to preserve user-created origins?
> > If so, you can mention this in the commit message
> >
>
> I explicitly state that all replication origins are migrated in the
> commit message, nothing specific about subscription origins.
>
> > 2) Should the table lock be released here, if so should we mention
> > comments similar to how it is mentioned in
> > binary_upgarde_add_sub_rel_state:
> > + /* Restore the remote_lsn if provided, while still holding the lock */
> > + if (!PG_ARGISNULL(2))
> > + {
> > + XLogRecPtr remote_commit = PG_GETARG_LSN(2);
> >
> > - ReplicationSlotRelease();
> > + replorigin_advance(node, remote_commit, InvalidXLogRecPtr,
> > + false /* backward */,
> > + false /* WAL log */);
> > + }
> > +
> > + table_close(rel, RowExclusiveLock);
> >
>
> Is that relevant when creating a new row? or only when updating an existing one?
>
> > 3) We can use new style of ereport to exclude '(' before errcode:
> > 3.a)
> > + ereport(ERROR,
> > + (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
> > + errmsg("replication origin name is too long"),
> > + errdetail("Replication origin names must be no longer than %d bytes.",
> > + MAX_RONAME_LEN)));
> >
> > 3.b) Similarly here:
> > + ereport(ERROR,
> > + (errcode(ERRCODE_DUPLICATE_OBJECT),
> > + errmsg("replication origin with ID %u already exists", node_oid)));
> >
> > 3.c) There are many others, you can search other ereport in the patch
> > and change it too.
> >
>
> Done.
>
> > 4) Since only one column will be returned here, can we use
> > "PQgetvalue(res, 0, 0) and remove PQfnumber:
> > + res = executeQueryOrDie(conn,
> > + "SELECT count(*) AS nrepl_origins "
> > + "FROM pg_catalog.pg_replication_origin");
> > + i_nrepl_origins = PQfnumber(res, "nrepl_origins");
> > + cluster->nrepl_origins = atoi(PQgetvalue(res, 0, i_nrepl_origins));
> > + PQclear(res);
> >
>
> Done.
>
> > 5) replorigin_create also has some duplicate code like below, will it
> > be possible to have a common function so that both of them can use?
> > /* Check for name collision */
> > + ScanKeyInit(&key,
> > + Anum_pg_replication_origin_roname,
> > + BTEqualStrategyNumber, F_TEXTEQ,
> > + roname_d);
> > + scan = systable_beginscan(rel, ReplicationOriginNameIndex,
> > + true /* indexOK */,
> > + SnapshotSelf,
> > + 1, &key);
> > + collides = HeapTupleIsValid(systable_getnext(scan));
> > + systable_endscan(scan);
> > +
> > + if (collides)
> > + ereport(ERROR,
> > + (errcode(ERRCODE_DUPLICATE_OBJECT),
> > + errmsg("replication origin \"%s\" already exists",
> > + originname)));
> > +
> > + memset(&nulls, 0, sizeof(nulls));
> > + memset(&values, 0, sizeof(values));
> > +
> > + values[Anum_pg_replication_origin_roident - 1] = ObjectIdGetDatum(node);
> > + values[Anum_pg_replication_origin_roname - 1] = roname_d;
> > +
> > + tuple = heap_form_tuple(RelationGetDescr(rel), values, nulls);
> > + CatalogTupleInsert(rel, tuple);
> > + heap_freetuple(tuple);
> > + CommandCounterIncrement();
> >
>
> I have created a new function replorigin_create_with_id() which has
> some common code and called that from both replorigin_create() and
> binary_upgrade_create_replication_origin()
>
>
> On Tue, Jun 9, 2026 at 3:06 AM Shlok Kyal <[email protected]> wrote:
> >
> > Hi Ajin,
> > I have reviewed the patch and here are some comments:
> >
> > 1. I want to clarify, what is the expected behaviour when we upgrade
> > the postgres instance from version <= 16 to "HEAD + patch"?
> > - ReplicationOriginNameForLogicalRep(subid, InvalidOid, originname,
> > sizeof(originname));
> > - replorigin_create(originname);
> > + if (!IsBinaryUpgrade)
> > + {
> > + ReplicationOriginNameForLogicalRep(subid, InvalidOid,
> > originname, sizeof(originname));
> > + replorigin_create(originname);
> > + }
> > and
> > + /* Dump replication origins */
> > + if (server_version >= 170000 && binary_upgrade &&
> > archDumpFormat == archNull)
> > + dumpReplicationOrigins(conn);
> >
> > When the postgres instance with subscription is updated from PG 16 (or
> > less) to HEAD, the replication origin is created as we do not have "if
> > (!IsBinaryUpgrade)" check in HEAD,
> > whereas when the similar instance is upgraded to "HEAD + patch", no
> > replication origin is present after the upgrade.
> > Is this difference in the behaviour between HEAD and "HEAD + patch" expected?
> >
>
> I have modified this as option 3 suggested by Shveta and Amit and
> removed the PG17 restriction for migrating replication origins, now it
> does it for all versions that have replication origins.
>
> > 2.a) Can we replace:
> > + /* Check for OID collision */
> > + ScanKeyInit(&key,
> > + Anum_pg_replication_origin_roident,
> > + BTEqualStrategyNumber, F_OIDEQ,
> > + ObjectIdGetDatum(node));
> > + scan = systable_beginscan(rel, ReplicationOriginIdentIndex,
> > + true /* indexOK */,
> > + SnapshotSelf,
> > + 1, &key);
> > + collides = HeapTupleIsValid(systable_getnext(scan));
> > + systable_endscan(scan);
> > +
> > + if (collides)
> > + ereport(ERROR,
> > + (errcode(ERRCODE_DUPLICATE_OBJECT),
> > + errmsg("replication origin with ID %u already
> > exists", node_oid)));
> >
> > with:
> > if (SearchSysCacheExists1(REPLORIGIDENT, ObjectIdGetDatum(node)))
> > ereport(ERROR, errcode(ERRCODE_DUPLICATE_OBJECT),
> > errmsg("replication origin with ID %u already
> > exists", node_oid));
> >
> > 2.b) Similarly, Can we replace:
> > + /* Check for name collision */
> > + ScanKeyInit(&key,
> > + Anum_pg_replication_origin_roname,
> > + BTEqualStrategyNumber, F_TEXTEQ,
> > + roname_d);
> > + scan = systable_beginscan(rel, ReplicationOriginNameIndex,
> > + true /* indexOK */,
> > + SnapshotSelf,
> > + 1, &key);
> > + collides = HeapTupleIsValid(systable_getnext(scan));
> > + systable_endscan(scan);
> > +
> > + if (collides)
> > + ereport(ERROR,
> > + (errcode(ERRCODE_DUPLICATE_OBJECT),
> > + errmsg("replication origin \"%s\" already exists",
> > + originname)));
> > with:
> > if (SearchSysCacheExists1(REPLORIGNAME, roname_d))
> > ereport(ERROR, errcode(ERRCODE_DUPLICATE_OBJECT),
> > errmsg("replication origin \"%s\" already
> > exists", originname));
> >
> > This will make the code smaller and easy to read. Thoughts?
>
> Done
>
>
> On Tue, Jun 9, 2026 at 2:13 PM shveta malik <[email protected]> wrote:
> >
> >
> > Please find a few comments on v007:
> >
> > 1)
> >
> > + node = (ReplOriginId) node_oid;
> > + originname = text_to_cstring(PG_GETARG_TEXT_PP(1));
> > +
> > + if (strlen(originname) > MAX_RONAME_LEN)
> > + ereport(ERROR,
> > + (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
> > + errmsg("replication origin name is too long"),
> > + errdetail("Replication origin names must be no longer than %d bytes.",
> > + MAX_RONAME_LEN)));
> > +
> > + roname_d = CStringGetTextDatum(originname);
> >
> > CStringGetTextDatum is:
> > #define CStringGetTextDatum(s) PointerGetDatum(cstring_to_text(s))
> > ---------
> >
> > We are first converting text to C-string , then Cstring to text and
> > then getting Datum. Double allocation and double conversion. Can we
> > please try this instead and verify if it works:
> >
> > text *origin_text = PG_GETARG_TEXT_PP(1);
> > originname = text_to_cstring(origin_text);
> > roname_d = PointerGetDatum(origin_text);
> >
>
> I've restructured the code now, its a bit tough to do this now.
>
> > 2)
> > +check_new_cluster_replication_origins(void)
> > +{
> > + PGconn *conn;
> > + PGresult *res;
> > + int norigins;
> > +
> > + /* Quick return if there are no replication origins to migrate. */
> > + if (old_cluster.nrepl_origins == 0)
> > + return;
> >
> > Don't we need a version check simialr to what we have introduced in
> > pg_dumpall for origins? (Similar to
> > check_new_cluster_replication_slots as well)
> >
>
> I've removed the version check for PG17 now.
>
> > 3)
> > Since we have introduced check_new_cluster_replication_origins, do we
> > even need below checks in binary_upgrade_create_replication_origin()?
> > In which conditions will they be hit?
> >
> > + /* Check for OID collision */
> > + ....
> > + if (collides)
> > + ereport(ERROR,
> > + (errcode(ERRCODE_DUPLICATE_OBJECT),
> > + errmsg("replication origin with ID %u already exists", node_oid)));
> > +
> > + /* Check for name collision */
> > + ....
> > + if (collides)
> > + ereport(ERROR,
> > + (errcode(ERRCODE_DUPLICATE_OBJECT),
> > + errmsg("replication origin \"%s\" already exists",
> > + originname)));
> >
>
> With the restructured code, its not easy to do this now.
>
> >
> > 4)
> > Now since check_new_cluster_subscription_configuration() is purely
> > about checking max-origin GUC configuration, I think the name is
> > misleading. IMO, we should merge it to
> > check_new_cluster_replication_origins(). See how
> > check_new_cluster_replication_slots() does it: checking both
> > new-cluster's count and the max-configuration for slots.
> >
>
> Done.
>
>
> On Tue, Jun 9, 2026 at 9:32 PM Shlok Kyal <[email protected]> wrote:
> >
> > >
> > One more comment:
> >
> > 3. In logical-replication.sgml, we have documentation as:
> > "Commit timestamps and origin data are not preserved during the upgrade."
> > Now since the origin related data is preserved after the upgrade,
> > should we update the documentation here?
> >
>
> Updated.
>
> I have addressed the comments in v8 (attached). I've also now taken
> out the check for PG17 on old cluster and now all clusters which have
> replication origins are migrated.
Hi Ajin,
Thanks for providing the updated patch, I reviewed the patch and I
have some comments:
1. patch is not applying on HEAD and needs a rebase.
2. New line after 'check_new_cluster_replication_origins' is not required:
+ check_new_cluster_replication_origins();
+
3. pg_dumpall with option '--roles-only' also dumps replication origins
eg:
./pg_dumpall -p 9000 --binary-upgrade --roles-only > op.sql
output contains:
-- For binary upgrade, must preserve replication origin roident and remote_lsn
SELECT pg_catalog.binary_upgrade_create_replication_origin('1'::pg_catalog.oid,
'pg_16387'::pg_catalog.text, '0/00000000'::pg_catalog.pg_lsn);
I think in this case we should not dump the replication origins as we
explicitly gave the option of '--roles-only'
4. pg_dumpall with --globals-only option also dumps replication origins.
eg:
./pg_dumpall -p 9000 --binary-upgrade --globals-only> op.sql and
output contains:
-- For binary upgrade, must preserve replication origin roident and remote_lsn
SELECT pg_catalog.binary_upgrade_create_replication_origin('1'::pg_catalog.oid,
'pg_16387'::pg_catalog.text, '0/00000000'::pg_catalog.pg_lsn);
Is this behaviour intentional? Other logical replication objects
(e.g., publications and subscriptions) are not included in pg_dumpall
--globals-only, so is it okay to dump replication origin?
Thanks,
Shlok Kyal
^ permalink raw reply [nested|flat] 97+ messages in thread
* Re: [PATCH] Preserve replication origin OIDs in pg_upgrade
@ 2026-06-21 16:22 Rui Zhao <[email protected]>
parent: Ajin Cherian <[email protected]>
2 siblings, 1 reply; 97+ messages in thread
From: Rui Zhao @ 2026-06-21 16:22 UTC (permalink / raw)
To: Ajin Cherian <[email protected]>; +Cc: Shlok Kyal <[email protected]>; shveta malik <[email protected]>; Zsolt Parragi <[email protected]>; vignesh C <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; PostgreSQL Hackers <[email protected]>; shveta malik <[email protected]>
>
> Hi Ajin,
>
> Thanks for v8. A couple of comments:
>
> 1) The PG17 gating seems inconsistent between the dump and the check now.
> dumpReplicationOrigins() is called unconditionally:
>
> if (binary_upgrade && archDumpFormat == archNull)
> dumpReplicationOrigins(conn);
>
> but get_subscription_info(), which sets nrepl_origins, is still only
> called for old clusters >= 17 in check_new_cluster():
>
> if (GET_MAJOR_VERSION(old_cluster.major_version) >= 1700)
> ...
> get_subscription_info(&old_cluster);
>
> and check_new_cluster_replication_origins() returns early on it:
>
> if (old_cluster.nrepl_origins == 0)
> return;
>
> For a < 17 old cluster get_subscription_info() is never called, so
> nrepl_origins stays 0 and this returns early. So when such a cluster has a
> (user-created) origin, the origin still gets dumped and recreated, while
> the "new cluster must have no origins" and max_active_replication_origins
> checks are skipped, and the upgrade fails during restore instead of at the
> check step. nrepl_origins is just a count(*) on pg_replication_origin, so
> moving its collection out of the >= 1700 block should be enough.
>
> 2) 004_subscription.pl only checks subscription origins (pg_<oid>). Since
> the patch migrates all origins, should we also create a user origin
> (pg_replication_origin_create()) and verify its roident/remote_lsn after
> the upgrade? That would also cover the case in (1).
>
> Thanks,
> Rui
>
^ permalink raw reply [nested|flat] 97+ messages in thread
* Re: [PATCH] Preserve replication origin OIDs in pg_upgrade
@ 2026-06-23 11:50 Ajin Cherian <[email protected]>
parent: Rui Zhao <[email protected]>
0 siblings, 4 replies; 97+ messages in thread
From: Ajin Cherian @ 2026-06-23 11:50 UTC (permalink / raw)
To: Rui Zhao <[email protected]>; +Cc: Shlok Kyal <[email protected]>; shveta malik <[email protected]>; Zsolt Parragi <[email protected]>; vignesh C <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; PostgreSQL Hackers <[email protected]>; shveta malik <[email protected]>
On Wed, Jun 17, 2026 at 8:59 PM Nisha Moond <[email protected]> wrote:
>
> Hi Ajin, Thanks for the patches.
> I have started reviewing the patches. Please find a few initial
> comments below (from v7 review, still applicable to v8):
>
> 1) Patch-002: pg_dump.c
> + if (dopt->binary_upgrade && subinfo->subenabled &&
> fout->remoteVersion >= 170000)
> {
> - if (subinfo->suboriginremotelsn)
> - {
> - /*
> - * Preserve the remote_lsn for the subscriber's replication
>
> I could not find any usage of subinfo[i].suboriginremotelsn anymore.
> It seems to be dead code now. Can we remove it?
Removed it.
> ~~~
>
> 2) Patch-002 : binary_upgrade_create_replication_origin()
> - PG_RETURN_VOID();
> + node_oid = PG_GETARG_OID(0);
> +
> + if (node_oid == InvalidOid || node_oid > PG_UINT16_MAX)
> + ereport(ERROR,
> + errcode(ERRCODE_INVALID_PARAMETER_VALUE),
> + errmsg("replication origin ID %u is out of range", node_oid));
> +
>
> The above range check allows PG_UINT16_MAX, but that value is not part
> of the valid replication origin ID range.
>
> In origin.h:
> #define DoNotReplicateId PG_UINT16_MAX
>
> And we already ensure "ReplOriginId != DoNotReplicateId" everywhere
> else. Would something like the below be more appropriate?
> if (node_oid == InvalidReplOriginId || node_oid >= DoNotReplicateId)
> ereport(ERROR, ...
> ~~~
Done.
>
> 3) I think we should add some additional details in
> logical-replication.sgml, especially under <title>Prepare for
> Subscriber Upgrades</title>.
> Some suggestions:
> 3a) Since all replication origins are now migrated, this:
> "...greater than or equal to the number of subscriptions present in
> the old cluster."
> could become:
> "...greater than or equal to the number of replication origins
> present in the old cluster."
>
> 3b) We should also mention:
> The new cluster must contain no replication origins.
>
Done.
On Wed, Jun 17, 2026 at 9:42 PM Nisha Moond <[email protected]> wrote:
>
> 4) I was able to build successfully without all three new header
> inclusions below, so we may not need them.
> --- a/src/backend/utils/adt/pg_upgrade_support.c
> +++ b/src/backend/utils/adt/pg_upgrade_support.c
> @@ -11,6 +11,7 @@
> +#include "access/genam.h"
> ...
> +#include "utils/fmgroids.h"
> ...
> +#include "utils/snapmgr.h"
> ~~~
>
Removed.
> 5) Race condition in origin.c: replorigin_create()
> - heap_freetuple(tuple);
> + table_close(rel, ExclusiveLock);
> +
> + replorigin_create_with_id(roident, roname, InvalidXLogRecPtr);
> +
>
> Due to the above refactoring, CREATE SUBSCRIPTION can now fail because
> of a race between two concurrent CREATE SUBSCRIPTION commands:
>
> postgres=# create subscription sub2 connection 'dbname=postgres
> host=localhost port=7733' publication pub1;
> ERROR: replication origin with ID 2 already exists
>
> I think this happens because the table lock is released, allowing
> another session to create a pg_origin with the next available roident,
> which replorigin_create_with_id() later attempts to use.
>
> Would it make sense to call replorigin_create_with_id() before
> table_close() so the lock is still held? I tested this and it works,
> though I'm not fully sure whether self-locking could be a concern
> here, since replorigin_create_with_id() again acquires a
> RowExclusiveLock on the same table.
> ~~~
>
I have fixed this as suggested by Shveta
~~~~~~~~~~
On Thu, Jun 18, 2026 at 7:17 PM shveta malik <[email protected]> wrote:
>
> > I think this happens because the table lock is released, allowing
> > another session to create a pg_origin with the next available roident,
> > which replorigin_create_with_id() later attempts to use.
> >
> > Would it make sense to call replorigin_create_with_id() before
> > table_close() so the lock is still held? I tested this and it works,
> > though I'm not fully sure whether self-locking could be a concern
> > here, since replorigin_create_with_id() again acquires a
> > RowExclusiveLock on the same table.
>
> We can handle this race-condition by moving the table-open to callers
> rather than internal call replorigin_create_with_id(). But we also
> need the following changes first:
>
Yes, did it that way.
> 1)
> replorigin_create_with_id has:
>
> + /*
> + * To avoid needing a TOAST table for pg_replication_origin, we limit
> + * replication origin names to 512 bytes. This should be more than enough
> + * for all practical use.
> + */
> + if (strlen(roname) > MAX_RONAME_LEN)
> + ereport(ERROR,
> + errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
> + errmsg("replication origin name is too long"),
> + errdetail("Replication origin names must be no longer than %d bytes.",
> + MAX_RONAME_LEN));
>
> replorigin_create_with_id() is called by replorigin_create() and
> binary_upgrade_create_replication_origin(). replorigin_create()
> already has this check. We should move above check to
> binary_upgrade_create_replication_origin() and get rid of it from
> internal function replorigin_create_with_id().
Moved.
>
> 2)
>
> Also the checks:
> + if (SearchSysCacheExists1(REPLORIGIDENT, ObjectIdGetDatum(roident)))
> + ereport(ERROR,
> + errcode(ERRCODE_DUPLICATE_OBJECT),
> + errmsg("replication origin with ID %u already exists",
> + (Oid) roident));
> +
> + if (SearchSysCacheExists1(REPLORIGNAME, roname_d))
> + ereport(ERROR,
> + errcode(ERRCODE_DUPLICATE_OBJECT),
> + errmsg("replication origin \"%s\" already exists", roname));
>
> should be moved to caller binary_upgrade_create_replication_origin().
> IMO, these checks are not needed for replorigin_create(), it alread
> has this, which should suffice.
>
> ereport(ERROR,
> (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
> errmsg("could not find free
> replication origin ID")));
>
> ~~
>
> Once we correctly place these checks, we can move the 'table_open with
> lock' to caller (before name and id collision checks) instead of
> replorigin_create_with_id(), which will take care of race-conditions
> reported by Nisha.
>
I have moved the roident check out of replorigin_create_with_id(), but
kept the roname check there as that is not there in
replorigin_create()
On Fri, Jun 19, 2026 at 5:48 PM Shlok Kyal <[email protected]> wrote:
> Hi Ajin,
>
> Thanks for providing the updated patch, I reviewed the patch and I
> have some comments:
>
> 1. patch is not applying on HEAD and needs a rebase.
>
> 2. New line after 'check_new_cluster_replication_origins' is not required:
> + check_new_cluster_replication_origins();
> +
>
> 3. pg_dumpall with option '--roles-only' also dumps replication origins
> eg:
> ./pg_dumpall -p 9000 --binary-upgrade --roles-only > op.sql
> output contains:
> -- For binary upgrade, must preserve replication origin roident and remote_lsn
> SELECT pg_catalog.binary_upgrade_create_replication_origin('1'::pg_catalog.oid,
> 'pg_16387'::pg_catalog.text, '0/00000000'::pg_catalog.pg_lsn);
>
> I think in this case we should not dump the replication origins as we
> explicitly gave the option of '--roles-only'
>
I have moved the dump_replication_origins under !roles_only and
!tablespaces_only checks, I think "globals only" is valid for
replication origins.
> 4. pg_dumpall with --globals-only option also dumps replication origins.
> eg:
> ./pg_dumpall -p 9000 --binary-upgrade --globals-only> op.sql and
> output contains:
> -- For binary upgrade, must preserve replication origin roident and remote_lsn
> SELECT pg_catalog.binary_upgrade_create_replication_origin('1'::pg_catalog.oid,
> 'pg_16387'::pg_catalog.text, '0/00000000'::pg_catalog.pg_lsn);
>
> Is this behaviour intentional? Other logical replication objects
> (e.g., publications and subscriptions) are not included in pg_dumpall
> --globals-only, so is it okay to dump replication origin?
>
This is expected as publications and subscriptions are database
specific and not globals, while replication origins are global.
On Mon, Jun 22, 2026 at 2:22 AM Rui Zhao <[email protected]> wrote:
>>
>> Hi Ajin,
>>
>> Thanks for v8. A couple of comments:
>>
>> 1) The PG17 gating seems inconsistent between the dump and the check now.
>> dumpReplicationOrigins() is called unconditionally:
>>
>> if (binary_upgrade && archDumpFormat == archNull)
>> dumpReplicationOrigins(conn);
>>
>> but get_subscription_info(), which sets nrepl_origins, is still only
>> called for old clusters >= 17 in check_new_cluster():
>>
>> if (GET_MAJOR_VERSION(old_cluster.major_version) >= 1700)
>> ...
>> get_subscription_info(&old_cluster);
>>
>> and check_new_cluster_replication_origins() returns early on it:
>>
>> if (old_cluster.nrepl_origins == 0)
>> return;
>>
>> For a < 17 old cluster get_subscription_info() is never called, so
>> nrepl_origins stays 0 and this returns early. So when such a cluster has a
>> (user-created) origin, the origin still gets dumped and recreated, while
>> the "new cluster must have no origins" and max_active_replication_origins
>> checks are skipped, and the upgrade fails during restore instead of at the
>> check step. nrepl_origins is just a count(*) on pg_replication_origin, so
>> moving its collection out of the >= 1700 block should be enough.
>>
Yes, good catch. I have separated out the replication origin speciifc
checks into a different function and moved it oustide the >= 1700
block
>> 2) 004_subscription.pl only checks subscription origins (pg_<oid>). Since
>> the patch migrates all origins, should we also create a user origin
>> (pg_replication_origin_create()) and verify its roident/remote_lsn after
>> the upgrade? That would also cover the case in (1).
Added.
All the above changes have been addressed in patch v9.
regards,
Ajin Cherian
Fujitsu Australia
Attachments:
[application/octet-stream] v9-0002-Preserve-replication-origin-OIDs-during-pg_upgrad.patch (32.6K, ../../CAFPTHDbWKQjv4QfHaNiXosZ1TD0ZHc2hFyFFNB2RvLsJeJoH6g@mail.gmail.com/2-v9-0002-Preserve-replication-origin-OIDs-during-pg_upgrad.patch)
download | inline diff:
From 8d1953a78051c08cd610d2ca543bf15095e8fc9d Mon Sep 17 00:00:00 2001
From: Ajin Cherian <[email protected]>
Date: Tue, 23 Jun 2026 20:07:53 +1000
Subject: [PATCH v9 2/2] Preserve replication origin OIDs during pg_upgrade
When pg_upgrade migrates a subscriber, replication origin OIDs
(roident) can change across the upgrade. This is a problem because
commit-timestamp records embed roident and are copied directly from
the old cluster's pg_commit_ts directory, causing spurious
"update_origin_differs" conflicts after the upgrade.
Fix this by dumping replication origins as global objects via
pg_dumpall during binary upgrade, using a new function
binary_upgrade_create_replication_origin(oid, name, lsn) to recreate
each origin with its preserved roident and remote_lsn. To avoid
conflicts with this, CreateSubscription() skips replorigin_create()
in binary-upgrade mode since the origin is already created by the
time the subscription is restored.
Author: Ajin Cherian <[email protected]>
Reviewer: Hayato Kuroda (Fujitsu) <[email protected]>
Reviewer: Zsolt Parragi <[email protected]>
Reviewer: Shlok Kyal <[email protected]>
Reviewer: Shveta malik <[email protected]>
Reviewer: Vignesh C <[email protected]>
Reviewer: Nisha Moond <[email protected]>
Reviewer: Rui Zhao <[email protected]>
---
doc/src/sgml/logical-replication.sgml | 4 +-
src/backend/commands/subscriptioncmds.c | 11 ++-
src/backend/replication/logical/origin.c | 98 ++++++++++++-------
src/backend/utils/adt/pg_upgrade_support.c | 107 ++++++++++++---------
src/bin/pg_dump/pg_dump.c | 51 ++--------
src/bin/pg_dump/pg_dump.h | 1 -
src/bin/pg_dump/pg_dumpall.c | 67 +++++++++++++
src/bin/pg_upgrade/check.c | 57 +++++++----
src/bin/pg_upgrade/info.c | 23 +++++
src/bin/pg_upgrade/pg_upgrade.h | 2 +
src/bin/pg_upgrade/t/004_subscription.pl | 58 ++++++++++-
src/include/catalog/pg_proc.dat | 8 +-
src/include/replication/origin.h | 3 +
13 files changed, 338 insertions(+), 152 deletions(-)
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index 9e7868487de..b4ccd44a327 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -2819,7 +2819,7 @@ CONTEXT: processing remote data for replication origin "pg_16395" during "INSER
<note>
<para>
- Commit timestamps and origin data are not preserved during the upgrade.
+ Commit timestamps are not preserved during the upgrade.
As a result, even if
<link linkend="sql-createsubscription-params-with-retain-dead-tuples"><literal>retain_dead_tuples</literal></link>
is enabled, the upgraded subscriber may be unable to detect conflicts or
@@ -2861,7 +2861,7 @@ CONTEXT: processing remote data for replication origin "pg_16395" during "INSER
The new cluster must have
<link linkend="guc-max-active-replication-origins"><varname>max_active_replication_origins</varname></link>
configured to a value greater than or equal to the number of
- subscriptions present in the old cluster.
+ replication origins present in the old cluster. The new cluster must contain no replication origins.
</para>
</listitem>
<listitem>
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 5fe7440418c..4191d78aebd 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -19,6 +19,7 @@
#include "access/table.h"
#include "access/twophase.h"
#include "access/xact.h"
+#include "catalog/binary_upgrade.h"
#include "catalog/catalog.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
@@ -898,9 +899,15 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
* apply workers initialization, and to handle origin creation dynamically
* when tables are added to the subscription. It is not clear whether
* preventing creation of origins is worth additional complexity.
+ *
+ * In binary-upgrade mode, skip origin creation here. This is required to
+ * preserve the roident from the old cluster for this subscription.
*/
- ReplicationOriginNameForLogicalRep(subid, InvalidOid, originname, sizeof(originname));
- replorigin_create(originname);
+ if (!IsBinaryUpgrade)
+ {
+ ReplicationOriginNameForLogicalRep(subid, InvalidOid, originname, sizeof(originname));
+ replorigin_create(originname);
+ }
/*
* Connect to remote side to execute requested commands and fetch table
diff --git a/src/backend/replication/logical/origin.c b/src/backend/replication/logical/origin.c
index c9dfb094c2b..e3a3e816f24 100644
--- a/src/backend/replication/logical/origin.c
+++ b/src/backend/replication/logical/origin.c
@@ -265,6 +265,54 @@ replorigin_by_name(const char *roname, bool missing_ok)
return roident;
}
+/*
+ * replorigin_create_with_id
+ *
+ * Create a replication origin with a specific ID and name, optionally
+ * restoring its remote_lsn. Used by pg_upgrade to preserve replication
+ * origin IDs across the upgrade.
+ *
+ * Caller must hold an exclusive lock on ReplicationOriginRelationId.
+ *
+ * Needs to be called in a transaction.
+ */
+void
+replorigin_create_with_id(ReplOriginId roident, const char *roname,
+ XLogRecPtr remote_lsn, Relation rel)
+{
+ Datum roname_d;
+ bool nulls[Natts_pg_replication_origin];
+ Datum values[Natts_pg_replication_origin];
+ HeapTuple tuple;
+
+ Assert(IsTransactionState());
+ Assert(CheckRelationLockedByMe(rel, ExclusiveLock, false));
+
+ roname_d = CStringGetTextDatum(roname);
+
+ if (SearchSysCacheExists1(REPLORIGNAME, roname_d))
+ ereport(ERROR,
+ errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("replication origin \"%s\" already exists", roname));
+
+ memset(&nulls, 0, sizeof(nulls));
+ memset(&values, 0, sizeof(values));
+
+ values[Anum_pg_replication_origin_roident - 1] = ObjectIdGetDatum(roident);
+ values[Anum_pg_replication_origin_roname - 1] = roname_d;
+
+ tuple = heap_form_tuple(RelationGetDescr(rel), values, nulls);
+ CatalogTupleInsert(rel, tuple);
+ heap_freetuple(tuple);
+ CommandCounterIncrement();
+
+ if (remote_lsn != InvalidXLogRecPtr)
+ replorigin_advance(roident, remote_lsn, InvalidXLogRecPtr,
+ false /* backward */,
+ false /* WAL log */);
+
+}
+
/*
* Create a replication origin.
*
@@ -273,13 +321,12 @@ replorigin_by_name(const char *roname, bool missing_ok)
ReplOriginId
replorigin_create(const char *roname)
{
- Oid roident;
- HeapTuple tuple = NULL;
- Relation rel;
- Datum roname_d;
- SnapshotData SnapshotDirty;
- SysScanDesc scan;
- ScanKeyData key;
+ Oid roident;
+ Relation rel;
+ SnapshotData SnapshotDirty;
+ SysScanDesc scan;
+ ScanKeyData key;
+ bool found = false;
/*
* To avoid needing a TOAST table for pg_replication_origin, we limit
@@ -293,8 +340,6 @@ replorigin_create(const char *roname)
errdetail("Replication origin names must be no longer than %d bytes.",
MAX_RONAME_LEN)));
- roname_d = CStringGetTextDatum(roname);
-
Assert(IsTransactionState());
/*
@@ -321,17 +366,15 @@ replorigin_create(const char *roname)
* snapshot. To make that safe, it needs to not have a TOAST table, since
* TOASTed data cannot be fetched without a snapshot. As of this writing,
* its only varlena column is roname, which we limit to 512 bytes to avoid
- * needing out-of-line storage. If you add a TOAST table to this catalog,
- * be sure to set up a snapshot everywhere it might be needed. For more
+ * needing out-of-line storage. If you add a TOAST table to this catalog,
+ * be sure to set up a snapshot everywhere it might be needed. For more
* information, see https://postgr.es/m/ZvMSUPOqUU-VNADN%40nathan.
*/
Assert(!OidIsValid(rel->rd_rel->reltoastrelid));
for (roident = InvalidOid + 1; roident < PG_UINT16_MAX; roident++)
{
- bool nulls[Natts_pg_replication_origin];
- Datum values[Natts_pg_replication_origin];
- bool collides;
+ bool collides;
CHECK_FOR_INTERRUPTS();
@@ -341,41 +384,28 @@ replorigin_create(const char *roname)
ObjectIdGetDatum(roident));
scan = systable_beginscan(rel, ReplicationOriginIdentIndex,
- true /* indexOK */ ,
+ true /* indexOK */,
&SnapshotDirty,
1, &key);
-
collides = HeapTupleIsValid(systable_getnext(scan));
-
systable_endscan(scan);
if (!collides)
{
- /*
- * Ok, found an unused roident, insert the new row and do a CCI,
- * so our callers can look it up if they want to.
- */
- memset(&nulls, 0, sizeof(nulls));
-
- values[Anum_pg_replication_origin_roident - 1] = ObjectIdGetDatum(roident);
- values[Anum_pg_replication_origin_roname - 1] = roname_d;
-
- tuple = heap_form_tuple(RelationGetDescr(rel), values, nulls);
- CatalogTupleInsert(rel, tuple);
- CommandCounterIncrement();
+ found = true;
break;
}
}
- /* now release lock again, */
- table_close(rel, ExclusiveLock);
-
- if (tuple == NULL)
+ if (!found)
ereport(ERROR,
(errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
errmsg("could not find free replication origin ID")));
- heap_freetuple(tuple);
+ replorigin_create_with_id(roident, roname, InvalidXLogRecPtr, rel);
+
+ table_close(rel, ExclusiveLock);
+
return roident;
}
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index 59c3e7f0146..eb3bb081503 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -28,6 +28,7 @@
#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
+#include "utils/syscache.h"
#include "utils/pg_lsn.h"
@@ -377,71 +378,85 @@ binary_upgrade_add_sub_rel_state(PG_FUNCTION_ARGS)
}
/*
- * binary_upgrade_replorigin_advance
+ * binary_upgrade_create_conflict_detection_slot
*
- * Update the remote_lsn for the subscriber's replication origin.
+ * Create a replication slot to retain information necessary for conflict
+ * detection such as dead tuples, commit timestamps, and origins.
*/
Datum
-binary_upgrade_replorigin_advance(PG_FUNCTION_ARGS)
+binary_upgrade_create_conflict_detection_slot(PG_FUNCTION_ARGS)
{
- Relation rel;
- Oid subid;
- char *subname;
- char originname[NAMEDATALEN];
- ReplOriginId node;
- XLogRecPtr remote_commit;
-
CHECK_IS_BINARY_UPGRADE;
- /*
- * We must ensure a non-NULL subscription name before dereferencing the
- * arguments.
- */
- if (PG_ARGISNULL(0))
- elog(ERROR, "null argument to binary_upgrade_replorigin_advance is not allowed");
-
- subname = text_to_cstring(PG_GETARG_TEXT_PP(0));
- remote_commit = PG_ARGISNULL(1) ? InvalidXLogRecPtr : PG_GETARG_LSN(1);
-
- rel = table_open(SubscriptionRelationId, RowExclusiveLock);
- subid = get_subscription_oid(subname, false);
-
- ReplicationOriginNameForLogicalRep(subid, InvalidOid, originname, sizeof(originname));
-
- /* Lock to prevent the replication origin from vanishing */
- LockRelationOid(ReplicationOriginRelationId, RowExclusiveLock);
- node = replorigin_by_name(originname, false);
-
- /*
- * The server will be stopped after setting up the objects in the new
- * cluster and the origins will be flushed during the shutdown checkpoint.
- * This will ensure that the latest LSN values for origin will be
- * available after the upgrade.
- */
- replorigin_advance(node, remote_commit, InvalidXLogRecPtr,
- false /* backward */ ,
- false /* WAL log */ );
+ CreateConflictDetectionSlot();
- UnlockRelationOid(ReplicationOriginRelationId, RowExclusiveLock);
- table_close(rel, RowExclusiveLock);
+ ReplicationSlotRelease();
PG_RETURN_VOID();
}
/*
- * binary_upgrade_create_conflict_detection_slot
+ * binary_upgrade_create_replication_origin
*
- * Create a replication slot to retain information necessary for conflict
- * detection such as dead tuples, commit timestamps, and origins.
+ * Create a replication origin with a specific OID and name, optionally
+ * restoring its remote_lsn. Used by pg_upgrade to preserve replication
+ * origin OIDs across the upgrade.
*/
Datum
-binary_upgrade_create_conflict_detection_slot(PG_FUNCTION_ARGS)
+binary_upgrade_create_replication_origin(PG_FUNCTION_ARGS)
{
+ Oid node_oid;
+ ReplOriginId node;
+ Relation rel;
+ char *originname;
+ XLogRecPtr remote_lsn = InvalidXLogRecPtr;
+
CHECK_IS_BINARY_UPGRADE;
- CreateConflictDetectionSlot();
+ if (PG_ARGISNULL(0) || PG_ARGISNULL(1))
+ elog(ERROR,
+ "null argument to binary_upgrade_create_replication_origin is not allowed");
- ReplicationSlotRelease();
+ node_oid = PG_GETARG_OID(0);
+
+ if (node_oid == InvalidOid || node_oid >= DoNotReplicateId)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("replication origin ID %u is out of range", node_oid));
+
+ node = (ReplOriginId) node_oid;
+
+ if (SearchSysCacheExists1(REPLORIGIDENT, ObjectIdGetDatum(node)))
+ ereport(ERROR,
+ errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("replication origin with ID %u already exists",
+ (Oid) node));
+
+ originname = text_to_cstring(PG_GETARG_TEXT_PP(1));
+
+ /*
+ * To avoid needing a TOAST table for pg_replication_origin, we limit
+ * replication origin names to 512 bytes. This should be more than enough
+ * for all practical use.
+ */
+ if (strlen(originname) > MAX_RONAME_LEN)
+ ereport(ERROR,
+ errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
+ errmsg("replication origin name is too long"),
+ errdetail("Replication origin names must be no longer than %d bytes.",
+ MAX_RONAME_LEN));
+
+ if (!PG_ARGISNULL(2))
+ remote_lsn = PG_GETARG_LSN(2);
+
+ Assert(IsTransactionState());
+
+ /* Acquire an exclusive lock before inserting the new origin. */
+ rel = table_open(ReplicationOriginRelationId, ExclusiveLock);
+
+ replorigin_create_with_id(node, originname, remote_lsn, rel);
+
+ table_close(rel, ExclusiveLock);
PG_RETURN_VOID();
}
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index ea22b0604c1..e157214b926 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -5179,7 +5179,6 @@ getSubscriptions(Archive *fout)
int i_subwalrcvtimeout;
int i_subpublications;
int i_suborigin;
- int i_suboriginremotelsn;
int i_subenabled;
int i_subfailover;
int i_subretaindeadtuples;
@@ -5248,11 +5247,9 @@ getSubscriptions(Archive *fout)
LOGICALREP_ORIGIN_ANY);
if (dopt->binary_upgrade && fout->remoteVersion >= 170000)
- appendPQExpBufferStr(query, " o.remote_lsn AS suboriginremotelsn,\n"
- " s.subenabled,\n");
+ appendPQExpBufferStr(query, " s.subenabled,\n");
else
- appendPQExpBufferStr(query, " NULL AS suboriginremotelsn,\n"
- " false AS subenabled,\n");
+ appendPQExpBufferStr(query, " false AS subenabled,\n");
if (fout->remoteVersion >= 170000)
appendPQExpBufferStr(query,
@@ -5332,7 +5329,6 @@ getSubscriptions(Archive *fout)
i_subwalrcvtimeout = PQfnumber(res, "subwalrcvtimeout");
i_subpublications = PQfnumber(res, "subpublications");
i_suborigin = PQfnumber(res, "suborigin");
- i_suboriginremotelsn = PQfnumber(res, "suboriginremotelsn");
subinfo = pg_malloc_array(SubscriptionInfo, ntups);
@@ -5385,11 +5381,6 @@ getSubscriptions(Archive *fout)
subinfo[i].subpublications =
pg_strdup(PQgetvalue(res, i, i_subpublications));
subinfo[i].suborigin = pg_strdup(PQgetvalue(res, i, i_suborigin));
- if (PQgetisnull(res, i, i_suboriginremotelsn))
- subinfo[i].suboriginremotelsn = NULL;
- else
- subinfo[i].suboriginremotelsn =
- pg_strdup(PQgetvalue(res, i, i_suboriginremotelsn));
/* Decide whether we want to dump it */
selectDumpableObject(&(subinfo[i].dobj), fout);
@@ -5668,37 +5659,15 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
* In binary-upgrade mode, we allow the replication to continue after the
* upgrade.
*/
- if (dopt->binary_upgrade && fout->remoteVersion >= 170000)
+ if (dopt->binary_upgrade && subinfo->subenabled && fout->remoteVersion >= 170000)
{
- if (subinfo->suboriginremotelsn)
- {
- /*
- * Preserve the remote_lsn for the subscriber's replication
- * origin. This value is required to start the replication from
- * the position before the upgrade. This value will be stale if
- * the publisher gets upgraded before the subscriber node.
- * However, this shouldn't be a problem as the upgrade of the
- * publisher ensures that all the transactions were replicated
- * before upgrading it.
- */
- appendPQExpBufferStr(query,
- "\n-- For binary upgrade, must preserve the remote_lsn for the subscriber's replication origin.\n");
- appendPQExpBufferStr(query,
- "SELECT pg_catalog.binary_upgrade_replorigin_advance(");
- appendStringLiteralAH(query, subinfo->dobj.name, fout);
- appendPQExpBuffer(query, ", '%s');\n", subinfo->suboriginremotelsn);
- }
-
- if (subinfo->subenabled)
- {
- /*
- * Enable the subscription to allow the replication to continue
- * after the upgrade.
- */
- appendPQExpBufferStr(query,
- "\n-- For binary upgrade, must preserve the subscriber's running state.\n");
- appendPQExpBuffer(query, "ALTER SUBSCRIPTION %s ENABLE;\n", qsubname);
- }
+ /*
+ * Enable the subscription to allow the replication to continue
+ * after the upgrade.
+ */
+ appendPQExpBufferStr(query,
+ "\n-- For binary upgrade, must preserve the subscriber's running state.\n");
+ appendPQExpBuffer(query, "ALTER SUBSCRIPTION %s ENABLE;\n", qsubname);
}
if (subinfo->dobj.dump & DUMP_COMPONENT_DEFINITION)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 5a6726d8b12..8f1252b854e 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -729,7 +729,6 @@ typedef struct _SubscriptionInfo
char *subwalrcvtimeout;
char *subpublications;
char *suborigin;
- char *suboriginremotelsn;
} SubscriptionInfo;
/*
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index b9653f0aefe..711fd66b3f8 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -18,6 +18,7 @@
#include <time.h>
#include <unistd.h>
+#include "access/xlogdefs.h"
#include "catalog/pg_authid_d.h"
#include "common/connect.h"
#include "common/file_perm.h"
@@ -67,6 +68,7 @@ static void dropDBs(PGconn *conn);
static void dumpUserConfig(PGconn *conn, const char *username);
static void dumpDatabases(PGconn *conn);
static void dumpTimestamp(const char *msg);
+static void dumpReplicationOrigins(PGconn *conn);
static int runPgDump(const char *dbname, const char *create_opts);
static void buildShSecLabels(PGconn *conn,
const char *catalog_name, Oid objectId,
@@ -663,6 +665,13 @@ main(int argc, char *argv[])
/* Dump role GUC privileges */
if (server_version >= 150000 && !skip_acls)
dumpRoleGUCPrivs(conn);
+
+ }
+
+ if (!tablespaces_only && !roles_only && binary_upgrade)
+ {
+ /* Dump replication origins */
+ dumpReplicationOrigins(conn);
}
/* Dump tablespaces */
@@ -1832,6 +1841,64 @@ dumpTimestamp(const char *msg)
fprintf(OPF, "-- %s %s\n\n", msg, buf);
}
+static void
+dumpReplicationOrigins(PGconn *conn)
+{
+ PQExpBuffer buf = createPQExpBuffer();
+ PGresult *res;
+ int i_roident;
+ int i_roname;
+ int i_remotelsn;
+
+ /* Get replication origins from catalogs */
+ appendPQExpBufferStr(buf,
+ "SELECT o.*, os.remote_lsn "
+ "FROM pg_catalog.pg_replication_origin o "
+ "LEFT OUTER JOIN pg_catalog.pg_replication_origin_status os ON o.roident = os.local_id ");
+
+ res = executeQuery(conn, buf->data);
+
+ i_roident = PQfnumber(res, "roident");
+ i_roname = PQfnumber(res, "roname");
+ i_remotelsn = PQfnumber(res, "remote_lsn");
+
+ if (PQntuples(res) > 0)
+ fprintf(OPF, "--\n-- Replication Origins \n--\n\n");
+
+ for (int i = 0; i < PQntuples(res); i++)
+ {
+ ReplOriginId roident;
+ const char *roname;
+
+ roident = atooid(PQgetvalue(res, i, i_roident));
+ roname = PQgetvalue(res, i, i_roname);
+
+ resetPQExpBuffer(buf);
+
+ appendPQExpBufferStr(buf, "\n-- For binary upgrade, must preserve replication origin roident and remote_lsn\n");
+ appendPQExpBuffer(buf,
+ "SELECT pg_catalog.binary_upgrade_create_replication_origin("
+ "'%u'::pg_catalog.oid, ", roident);
+ appendStringLiteralConn(buf, roname, conn);
+ appendPQExpBufferStr(buf, "::pg_catalog.text");
+
+ if (!PQgetisnull(res, i, i_remotelsn))
+ {
+ appendPQExpBufferStr(buf, ", ");
+ appendStringLiteralConn(buf, PQgetvalue(res, i, i_remotelsn), conn);
+ appendPQExpBufferStr(buf, "::pg_catalog.pg_lsn");
+ }
+ else
+ appendPQExpBufferStr(buf, ", NULL");
+
+ appendPQExpBufferStr(buf, ");\n");
+ fprintf(OPF, "%s", buf->data);
+ }
+
+ PQclear(res);
+ destroyPQExpBuffer(buf);
+}
+
/*
* read_dumpall_filters - retrieve database identifier patterns from file
*
diff --git a/src/bin/pg_upgrade/check.c b/src/bin/pg_upgrade/check.c
index f5c93e611d2..613f64e1e15 100644
--- a/src/bin/pg_upgrade/check.c
+++ b/src/bin/pg_upgrade/check.c
@@ -33,7 +33,7 @@ static void check_for_new_tablespace_dir(void);
static void check_for_user_defined_encoding_conversions(ClusterInfo *cluster);
static void check_for_unicode_update(ClusterInfo *cluster);
static void check_new_cluster_replication_slots(void);
-static void check_new_cluster_subscription_configuration(void);
+static void check_new_cluster_replication_origins(void);
static void check_old_cluster_for_valid_slots(void);
static void check_old_cluster_subscription_state(void);
static void check_old_cluster_global_names(ClusterInfo *cluster);
@@ -651,6 +651,9 @@ check_and_dump_old_cluster(void)
check_old_cluster_subscription_state();
}
+ /* Get replication origin information */
+ get_replication_origin_info(&old_cluster);
+
check_for_data_types_usage(&old_cluster);
/*
@@ -797,7 +800,8 @@ check_new_cluster(void)
check_new_cluster_replication_slots();
- check_new_cluster_subscription_configuration();
+ check_new_cluster_replication_origins();
+
}
@@ -2303,31 +2307,45 @@ check_new_cluster_replication_slots(void)
}
/*
- * check_new_cluster_subscription_configuration()
+ * check_new_cluster_replication_origins()
+ *
+ * Verify that the new cluster has no replication origins. During upgrade,
+ * pg_upgrade restores replication origins from the old cluster with their
+ * original OIDs. If the new cluster already contains origins, those OIDs
+ * may collide, causing the upgrade to fail mid-way.
*
- * Verify that the max_active_replication_origins configuration specified is
- * enough for creating the subscriptions. This is required to create the
- * replication origin for each subscription.
+ * Also verify that the max_active_replication_origins configuration is
+ * enough for creating all the replication origins.
*/
static void
-check_new_cluster_subscription_configuration(void)
+check_new_cluster_replication_origins(void)
{
- PGresult *res;
PGconn *conn;
+ PGresult *res;
+ int norigins;
int max_active_replication_origins;
- /* Subscriptions and their dependencies can be migrated since PG17. */
- if (GET_MAJOR_VERSION(old_cluster.major_version) < 1700)
+ /* Quick return if there are no replication origins to migrate. */
+ if (old_cluster.nrepl_origins == 0)
return;
- /* Quick return if there are no subscriptions to be migrated. */
- if (old_cluster.nsubs == 0)
- return;
-
- prep_status("Checking new cluster configuration for subscriptions");
+ prep_status("Checking replication origins in new cluster");
conn = connectToServer(&new_cluster, "template1");
+ res = executeQueryOrDie(conn,
+ "SELECT count(*) "
+ "FROM pg_catalog.pg_replication_origin");
+
+ if (PQntuples(res) != 1)
+ pg_fatal("could not count the number of replication origins");
+
+ norigins = atoi(PQgetvalue(res, 0, 0));
+ PQclear(res);
+
+ if (norigins > 0)
+ pg_fatal("expected 0 replication origins but found %d", norigins);
+
res = executeQueryOrDie(conn, "SELECT setting FROM pg_settings "
"WHERE name = 'max_active_replication_origins';");
@@ -2335,12 +2353,13 @@ check_new_cluster_subscription_configuration(void)
pg_fatal("could not determine parameter settings on new cluster");
max_active_replication_origins = atoi(PQgetvalue(res, 0, 0));
- if (old_cluster.nsubs > max_active_replication_origins)
+ PQclear(res);
+
+ if (old_cluster.nrepl_origins > max_active_replication_origins)
pg_fatal("\"max_active_replication_origins\" (%d) must be greater than or equal to the number of "
- "subscriptions (%d) on the old cluster",
- max_active_replication_origins, old_cluster.nsubs);
+ "replication origins (%d) on the old cluster",
+ max_active_replication_origins, old_cluster.nrepl_origins);
- PQclear(res);
PQfinish(conn);
check_ok();
diff --git a/src/bin/pg_upgrade/info.c b/src/bin/pg_upgrade/info.c
index 37fff93892f..36d8570a9ef 100644
--- a/src/bin/pg_upgrade/info.c
+++ b/src/bin/pg_upgrade/info.c
@@ -832,6 +832,29 @@ count_old_cluster_logical_slots(void)
return slot_count;
}
+/*
+ * get_replication_origin_info()
+ *
+ * Gets the information of replication origins in the cluster.
+ */
+void
+get_replication_origin_info(ClusterInfo *cluster)
+{
+ PGconn *conn;
+ PGresult *res;
+ int i_norigins;
+
+ conn = connectToServer(cluster, "template1");
+ res = executeQueryOrDie(conn, "SELECT count(*) AS norigins "
+ "FROM pg_catalog.pg_replication_origin");
+ i_norigins = PQfnumber(res, "norigins");
+
+ cluster->nrepl_origins = atoi(PQgetvalue(res, 0, i_norigins));
+ PQclear(res);
+
+ PQfinish(conn);
+}
+
/*
* get_subscription_info()
*
diff --git a/src/bin/pg_upgrade/pg_upgrade.h b/src/bin/pg_upgrade/pg_upgrade.h
index ccd1ac0d013..1dec4cffbfb 100644
--- a/src/bin/pg_upgrade/pg_upgrade.h
+++ b/src/bin/pg_upgrade/pg_upgrade.h
@@ -311,6 +311,7 @@ typedef struct
int num_tablespaces;
const char *tablespace_suffix; /* directory specification */
int nsubs; /* number of subscriptions */
+ int nrepl_origins; /* number of replication origins */
bool sub_retain_dead_tuples; /* whether a subscription enables
* retain_dead_tuples. */
} ClusterInfo;
@@ -451,6 +452,7 @@ FileNameMap *gen_db_file_maps(DbInfo *old_db,
void get_db_rel_and_slot_infos(ClusterInfo *cluster);
int count_old_cluster_logical_slots(void);
void get_subscription_info(ClusterInfo *cluster);
+void get_replication_origin_info(ClusterInfo *cluster);
/* option.c */
diff --git a/src/bin/pg_upgrade/t/004_subscription.pl b/src/bin/pg_upgrade/t/004_subscription.pl
index 646767f2a65..2e07277fea9 100644
--- a/src/bin/pg_upgrade/t/004_subscription.pl
+++ b/src/bin/pg_upgrade/t/004_subscription.pl
@@ -42,7 +42,7 @@ my $connstr = $publisher->connstr . ' dbname=postgres';
# ------------------------------------------------------
# Check that pg_upgrade fails when max_active_replication_origins configured
-# in the new cluster is less than the number of subscriptions in the old
+# in the new cluster is less than the number of replication origins in the old
# cluster.
# ------------------------------------------------------
# It is sufficient to use disabled subscription to test upgrade failure.
@@ -74,7 +74,7 @@ command_checks_all(
],
1,
[
- qr/"max_active_replication_origins" \(0\) must be greater than or equal to the number of subscriptions \(1\) on the old cluster/
+ qr/"max_active_replication_origins" \(0\) must be greater than or equal to the number of replication origins \(1\) on the old cluster/
],
[qr//],
'run of pg_upgrade where the new cluster has insufficient max_active_replication_origins'
@@ -301,8 +301,38 @@ is($result, qq(t), "Check that the table is in init state");
# Get the replication origin's remote_lsn of the old subscriber
my $remote_lsn = $old_sub->safe_psql('postgres',
- "SELECT remote_lsn FROM pg_replication_origin_status os, pg_subscription s WHERE os.external_id = 'pg_' || s.oid AND s.subname = 'regress_sub4'"
+ "SELECT os.remote_lsn
+ FROM pg_replication_origin_status os
+ JOIN pg_replication_origin o ON o.roident = os.local_id
+ JOIN pg_subscription s ON o.roname = 'pg_' || s.oid::text
+ WHERE s.subname = 'regress_sub4'"
);
+
+# Get the replication origin ids (roident) for all subscriptions, keyed by
+# subscription name (which is stable across upgrade, unlike suboid). These
+# must be preserved after upgrade. A mismatch would cause spurious
+# update_origin_differs conflicts.
+my %pre_upgrade_roident;
+my $roident_rows = $old_sub->safe_psql('postgres',
+ "SELECT s.subname, o.roident
+ FROM pg_subscription s
+ JOIN pg_replication_origin o ON o.roname = 'pg_' || s.oid::text
+ ORDER BY s.subname"
+);
+for my $row (split /\n/, $roident_rows)
+{
+ my ($subname, $roident) = split /\|/, $row;
+ $pre_upgrade_roident{$subname} = $roident;
+}
+
+# Create a user created replication origin, which should also be preserved after upgrade.
+my $user_origin_name = 'regress_user_origin';
+$old_sub->safe_psql('postgres',
+ "SELECT pg_replication_origin_create('$user_origin_name')");
+$pre_upgrade_roident{$user_origin_name} = $old_sub->safe_psql('postgres',
+ "SELECT roident FROM pg_replication_origin WHERE roname = '$user_origin_name'"
+);
+
# Have the subscription in disabled state before upgrade
$old_sub->safe_psql('postgres', "ALTER SUBSCRIPTION regress_sub5 DISABLE");
@@ -378,6 +408,28 @@ regress_sub5|f|f|f),
"check that the subscription's running status, failover, and retain_dead_tuples are preserved"
);
+# Verify that the subscription related replication origins are preserved after upgrade.
+my $post_roident_rows = $new_sub->safe_psql('postgres',
+ "SELECT s.subname, o.roident
+ FROM pg_subscription s
+ JOIN pg_replication_origin o ON o.roname = 'pg_' || s.oid::text
+ ORDER BY s.subname"
+);
+for my $row (split /\n/, $post_roident_rows)
+{
+ my ($subname, $roident) = split /\|/, $row;
+ is($roident, $pre_upgrade_roident{$subname},
+ "roident preserved for subscription '$subname' after upgrade");
+}
+
+# Verify that user created replication origins are preserved after upgrade.
+my $post_user_roident = $new_sub->safe_psql('postgres',
+ "SELECT roident FROM pg_replication_origin WHERE roname = '$user_origin_name'"
+);
+is($post_user_roident, $pre_upgrade_roident{$user_origin_name},
+ "roident preserved for user-created origin '$user_origin_name' after upgrade"
+);
+
# Subscription relations should be preserved
$result = $new_sub->safe_psql('postgres',
"SELECT srrelid, srsubstate FROM pg_subscription_rel ORDER BY srrelid");
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 3a28406981d..21466d926f1 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -11960,10 +11960,6 @@
provolatile => 'v', proparallel => 'u', prorettype => 'void',
proargtypes => 'text oid char pg_lsn',
prosrc => 'binary_upgrade_add_sub_rel_state' },
-{ oid => '6320', descr => 'for use by pg_upgrade (remote_lsn for origin)',
- proname => 'binary_upgrade_replorigin_advance', proisstrict => 'f',
- provolatile => 'v', proparallel => 'u', prorettype => 'void',
- proargtypes => 'text pg_lsn', prosrc => 'binary_upgrade_replorigin_advance' },
{ oid => '6505', descr => 'for use by pg_upgrade (conflict detection slot)',
proname => 'binary_upgrade_create_conflict_detection_slot',
proisstrict => 'f', provolatile => 'v', proparallel => 'u',
@@ -11973,6 +11969,10 @@
proname => 'binary_upgrade_set_next_pg_subscription_oid', provolatile => 'v',
proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
prosrc => 'binary_upgrade_set_next_pg_subscription_oid' },
+{ oid => '9161', descr => 'for use by pg_upgrade (replication origin)',
+ proname => 'binary_upgrade_create_replication_origin', proisstrict => 'f',
+ provolatile => 'v', proparallel => 'u', prorettype => 'void',
+ proargtypes => 'oid text pg_lsn', prosrc => 'binary_upgrade_create_replication_origin' },
# conversion functions
{ oid => '4310', descr => 'internal conversion function for KOI8R to WIN1251',
diff --git a/src/include/replication/origin.h b/src/include/replication/origin.h
index a69faf6eaaf..11ee630fb28 100644
--- a/src/include/replication/origin.h
+++ b/src/include/replication/origin.h
@@ -14,6 +14,7 @@
#include "access/xlogdefs.h"
#include "access/xlogreader.h"
#include "catalog/pg_replication_origin.h"
+#include "utils/relcache.h"
typedef struct xl_replorigin_set
{
@@ -55,6 +56,8 @@ extern PGDLLIMPORT int max_active_replication_origins;
/* API for querying & manipulating replication origins */
extern ReplOriginId replorigin_by_name(const char *roname, bool missing_ok);
extern ReplOriginId replorigin_create(const char *roname);
+extern void replorigin_create_with_id(ReplOriginId roident, const char *roname,
+ XLogRecPtr remote_lsn, Relation rel);
extern void replorigin_drop_by_name(const char *name, bool missing_ok, bool nowait);
extern bool replorigin_by_oid(ReplOriginId roident, bool missing_ok,
char **roname);
--
2.47.3
[application/octet-stream] v9-0001-Preserve-subscription-OIDs-during-pg_upgrade.patch (9.1K, ../../CAFPTHDbWKQjv4QfHaNiXosZ1TD0ZHc2hFyFFNB2RvLsJeJoH6g@mail.gmail.com/3-v9-0001-Preserve-subscription-OIDs-during-pg_upgrade.patch)
download | inline diff:
From 9abd26e21ed10b0889f55e2299519a551f1e8bd0 Mon Sep 17 00:00:00 2001
From: Ajin Cherian <[email protected]>
Date: Mon, 22 Jun 2026 15:18:56 +1000
Subject: [PATCH v9 1/2] Preserve subscription OIDs during pg_upgrade
Currently subscription OIDs can be changed when a cluster is upgraded
using pg_upgrade. This is required for a subsequent patch which will
preserve the replication oids after upgrade.
Author: Vignesh C <[email protected]>
---
src/backend/commands/subscriptioncmds.c | 25 +++++++++++++++++--
src/backend/utils/adt/pg_upgrade_support.c | 10 ++++++++
src/bin/pg_dump/pg_dump.c | 8 ++++++
src/bin/pg_upgrade/pg_upgrade.c | 3 +++
src/bin/pg_upgrade/t/004_subscription.pl | 7 ++++++
src/include/catalog/binary_upgrade.h | 1 +
src/include/catalog/pg_proc.dat | 4 +++
.../expected/spgist_name_ops.out | 6 +++--
8 files changed, 60 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index ee06a726f42..5fe7440418c 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -83,6 +83,12 @@
/* check if the 'val' has 'bits' set */
#define IsSet(val, bits) (((val) & (bits)) == (bits))
+/*
+ * This will be set by the pg_upgrade_support function --
+ * binary_upgrade_set_next_pg_subscription_oid().
+ */
+Oid binary_upgrade_next_pg_subscription_oid = InvalidOid;
+
/*
* Structure to hold a bitmap representing the user-provided CREATE/ALTER
* SUBSCRIPTION command options and the parsed/default values of each of them.
@@ -803,8 +809,23 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
memset(values, 0, sizeof(values));
memset(nulls, false, sizeof(nulls));
- subid = GetNewOidWithIndex(rel, SubscriptionObjectIndexId,
- Anum_pg_subscription_oid);
+ /* Use binary-upgrade override for pg_subscription.oid? */
+ if (IsBinaryUpgrade)
+ {
+ if (!OidIsValid(binary_upgrade_next_pg_subscription_oid))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("pg_subscription OID value not set when in binary upgrade mode")));
+
+ subid = binary_upgrade_next_pg_subscription_oid;
+ binary_upgrade_next_pg_subscription_oid = InvalidOid;
+ }
+ else
+ {
+ subid = GetNewOidWithIndex(rel, SubscriptionObjectIndexId,
+ Anum_pg_subscription_oid);
+ }
+
values[Anum_pg_subscription_oid - 1] = ObjectIdGetDatum(subid);
values[Anum_pg_subscription_subdbid - 1] = ObjectIdGetDatum(MyDatabaseId);
values[Anum_pg_subscription_subskiplsn - 1] = LSNGetDatum(InvalidXLogRecPtr);
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index b505a6b4fee..59c3e7f0146 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -181,6 +181,16 @@ binary_upgrade_set_next_pg_authid_oid(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
+Datum
+binary_upgrade_set_next_pg_subscription_oid(PG_FUNCTION_ARGS)
+{
+ Oid subid = PG_GETARG_OID(0);
+
+ CHECK_IS_BINARY_UPGRADE;
+ binary_upgrade_next_pg_subscription_oid = subid;
+ PG_RETURN_VOID();
+}
+
Datum
binary_upgrade_create_empty_extension(PG_FUNCTION_ARGS)
{
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index c56437d6057..ea22b0604c1 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -5583,6 +5583,14 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
appendPQExpBuffer(delq, "DROP SUBSCRIPTION %s;\n",
qsubname);
+ if (dopt->binary_upgrade)
+ {
+ appendPQExpBufferStr(query, "\n-- For binary upgrade, must preserve pg_subscription.oid\n");
+ appendPQExpBuffer(query,
+ "SELECT pg_catalog.binary_upgrade_set_next_pg_subscription_oid('%u'::pg_catalog.oid);\n\n",
+ subinfo->dobj.catId.oid);
+ }
+
appendPQExpBuffer(query, "CREATE SUBSCRIPTION %s ",
qsubname);
if (subinfo->subservername)
diff --git a/src/bin/pg_upgrade/pg_upgrade.c b/src/bin/pg_upgrade/pg_upgrade.c
index 2127d297bfe..4e853096698 100644
--- a/src/bin/pg_upgrade/pg_upgrade.c
+++ b/src/bin/pg_upgrade/pg_upgrade.c
@@ -35,6 +35,9 @@
*
* We control all assignments of pg_database.oid because we want the directory
* names to match between the old and new cluster.
+ *
+ * We control assignment of pg_subscription.oid because we want the oid to
+ * match between the old and new cluster.
*/
diff --git a/src/bin/pg_upgrade/t/004_subscription.pl b/src/bin/pg_upgrade/t/004_subscription.pl
index c94a82deae0..646767f2a65 100644
--- a/src/bin/pg_upgrade/t/004_subscription.pl
+++ b/src/bin/pg_upgrade/t/004_subscription.pl
@@ -313,6 +313,9 @@ my $tab_upgraded1_oid = $old_sub->safe_psql('postgres',
my $tab_upgraded2_oid = $old_sub->safe_psql('postgres',
"SELECT oid FROM pg_class WHERE relname = 'tab_upgraded2'");
+$sub_oid = $old_sub->safe_psql('postgres',
+ "SELECT oid FROM pg_subscription ORDER BY subname");
+
$old_sub->stop;
# Change configuration so that initial table sync does not get started
@@ -359,6 +362,10 @@ $publisher->safe_psql(
$new_sub->start;
+# The subscription oid should be preserved
+$result = $new_sub->safe_psql('postgres', "SELECT oid FROM pg_subscription ORDER BY subname");
+is($result, qq($sub_oid), "subscription oid should have been preserved");
+
# The subscription's running status, failover option, and retain_dead_tuples
# option should be preserved in the upgraded instance. So regress_sub4 should
# still have subenabled, subfailover, and subretaindeadtuples set to true,
diff --git a/src/include/catalog/binary_upgrade.h b/src/include/catalog/binary_upgrade.h
index 7bf7ae44385..b15b18e7dc9 100644
--- a/src/include/catalog/binary_upgrade.h
+++ b/src/include/catalog/binary_upgrade.h
@@ -32,6 +32,7 @@ extern PGDLLIMPORT RelFileNumber binary_upgrade_next_toast_pg_class_relfilenumbe
extern PGDLLIMPORT Oid binary_upgrade_next_pg_enum_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_authid_oid;
+extern PGDLLIMPORT Oid binary_upgrade_next_pg_subscription_oid;
extern PGDLLIMPORT bool binary_upgrade_record_init_privs;
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index be157a5fbe9..3a28406981d 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -11969,6 +11969,10 @@
proisstrict => 'f', provolatile => 'v', proparallel => 'u',
prorettype => 'void', proargtypes => '',
prosrc => 'binary_upgrade_create_conflict_detection_slot' },
+{ oid => '9160', descr => 'for use by pg_upgrade',
+ proname => 'binary_upgrade_set_next_pg_subscription_oid', provolatile => 'v',
+ proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
+ prosrc => 'binary_upgrade_set_next_pg_subscription_oid' },
# conversion functions
{ oid => '4310', descr => 'internal conversion function for KOI8R to WIN1251',
diff --git a/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out b/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out
index 1ee65ede243..39d43368c42 100644
--- a/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out
+++ b/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out
@@ -59,11 +59,12 @@ select * from t
binary_upgrade_set_next_multirange_pg_type_oid | 1 | binary_upgrade_set_next_multirange_pg_type_oid
binary_upgrade_set_next_pg_authid_oid | | binary_upgrade_set_next_pg_authid_oid
binary_upgrade_set_next_pg_enum_oid | | binary_upgrade_set_next_pg_enum_oid
+ binary_upgrade_set_next_pg_subscription_oid | | binary_upgrade_set_next_pg_subscription_oid
binary_upgrade_set_next_pg_tablespace_oid | | binary_upgrade_set_next_pg_tablespace_oid
binary_upgrade_set_next_pg_type_oid | | binary_upgrade_set_next_pg_type_oid
binary_upgrade_set_next_toast_pg_class_oid | 1 | binary_upgrade_set_next_toast_pg_class_oid
binary_upgrade_set_next_toast_relfilenode | | binary_upgrade_set_next_toast_relfilenode
-(13 rows)
+(14 rows)
-- Verify clean failure when INCLUDE'd columns result in overlength tuple
-- The error message details are platform-dependent, so show only SQLSTATE
@@ -108,11 +109,12 @@ select * from t
binary_upgrade_set_next_multirange_pg_type_oid | 1 | binary_upgrade_set_next_multirange_pg_type_oid
binary_upgrade_set_next_pg_authid_oid | | binary_upgrade_set_next_pg_authid_oid
binary_upgrade_set_next_pg_enum_oid | | binary_upgrade_set_next_pg_enum_oid
+ binary_upgrade_set_next_pg_subscription_oid | | binary_upgrade_set_next_pg_subscription_oid
binary_upgrade_set_next_pg_tablespace_oid | | binary_upgrade_set_next_pg_tablespace_oid
binary_upgrade_set_next_pg_type_oid | | binary_upgrade_set_next_pg_type_oid
binary_upgrade_set_next_toast_pg_class_oid | 1 | binary_upgrade_set_next_toast_pg_class_oid
binary_upgrade_set_next_toast_relfilenode | | binary_upgrade_set_next_toast_relfilenode
-(13 rows)
+(14 rows)
\set VERBOSITY sqlstate
insert into t values(repeat('xyzzy', 12), 42, repeat('xyzzy', 4000));
--
2.47.3
^ permalink raw reply [nested|flat] 97+ messages in thread
* Re: [PATCH] Preserve replication origin OIDs in pg_upgrade
@ 2026-06-23 16:07 Rui Zhao <[email protected]>
parent: Ajin Cherian <[email protected]>
3 siblings, 0 replies; 97+ messages in thread
From: Rui Zhao @ 2026-06-23 16:07 UTC (permalink / raw)
To: Ajin Cherian <[email protected]>; +Cc: Shlok Kyal <[email protected]>; shveta malik <[email protected]>; Zsolt Parragi <[email protected]>; vignesh C <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; PostgreSQL Hackers <[email protected]>; shveta malik <[email protected]>
Hi Ajin,
Thanks for v9, and for addressing both my earlier comments. I also went
through the rest of v9 -- the other reviewers' comments and the
corresponding fixes (including the replorigin_create() locking rework) look
good to me; I didn't find any issues there.
A follow-up on the version gating, though: I think it now moved a bit too
far. get_replication_origin_info() is called unconditionally (outside the
>= 1700 block):
/* Get replication origin information */
get_replication_origin_info(&old_cluster);
and it runs "SELECT count(*) FROM pg_catalog.pg_replication_origin".
pg_dumpall's dumpReplicationOrigins() queries the same catalog with no
version guard either. But pg_replication_origin only exists from 9.5
(commit 5aa2350426), while pg_upgrade still accepts old clusters back to
9.2:
if (GET_MAJOR_VERSION(old_cluster.major_version) < 902)
pg_fatal("This utility can only upgrade from PostgreSQL version %s
and later.", "9.2");
So upgrading from 9.2/9.3/9.4 now fails: the query hits a non-existent
catalog and pg_fatal()s (and dumpReplicationOrigins would too).
It should be gated on the version where origins were introduced, e.g.
if (GET_MAJOR_VERSION(old_cluster.major_version) >= 905)
get_replication_origin_info(&old_cluster);
with the matching server_version >= 90500 check on the pg_dumpall side.
That keeps the < 17 case I flagged working while not breaking 9.2-9.4
(which have no origins to migrate anyway).
One nit while here: there is an extra blank line after the
check_new_cluster_replication_origins() call in check_new_cluster().
Thanks,
Rui
Ajin Cherian <[email protected]> 于2026年6月23日周二 19:50写道：
>
> On Wed, Jun 17, 2026 at 8:59 PM Nisha Moond <[email protected]> wrote:
> >
> > Hi Ajin, Thanks for the patches.
> > I have started reviewing the patches. Please find a few initial
> > comments below (from v7 review, still applicable to v8):
> >
> > 1) Patch-002: pg_dump.c
> > + if (dopt->binary_upgrade && subinfo->subenabled &&
> > fout->remoteVersion >= 170000)
> > {
> > - if (subinfo->suboriginremotelsn)
> > - {
> > - /*
> > - * Preserve the remote_lsn for the subscriber's replication
> >
> > I could not find any usage of subinfo[i].suboriginremotelsn anymore.
> > It seems to be dead code now. Can we remove it?
>
> Removed it.
>
>
> > ~~~
> >
> > 2) Patch-002 : binary_upgrade_create_replication_origin()
> > - PG_RETURN_VOID();
> > + node_oid = PG_GETARG_OID(0);
> > +
> > + if (node_oid == InvalidOid || node_oid > PG_UINT16_MAX)
> > + ereport(ERROR,
> > + errcode(ERRCODE_INVALID_PARAMETER_VALUE),
> > + errmsg("replication origin ID %u is out of range", node_oid));
> > +
> >
> > The above range check allows PG_UINT16_MAX, but that value is not part
> > of the valid replication origin ID range.
> >
> > In origin.h:
> > #define DoNotReplicateId PG_UINT16_MAX
> >
> > And we already ensure "ReplOriginId != DoNotReplicateId" everywhere
> > else. Would something like the below be more appropriate?
> > if (node_oid == InvalidReplOriginId || node_oid >= DoNotReplicateId)
> > ereport(ERROR, ...
> > ~~~
>
> Done.
>
>
> >
> > 3) I think we should add some additional details in
> > logical-replication.sgml, especially under <title>Prepare for
> > Subscriber Upgrades</title>.
> > Some suggestions:
> > 3a) Since all replication origins are now migrated, this:
> > "...greater than or equal to the number of subscriptions present in
> > the old cluster."
> > could become:
> > "...greater than or equal to the number of replication origins
> > present in the old cluster."
> >
> > 3b) We should also mention:
> > The new cluster must contain no replication origins.
> >
>
> Done.
>
>
>
> On Wed, Jun 17, 2026 at 9:42 PM Nisha Moond <[email protected]> wrote:
> >
> > 4) I was able to build successfully without all three new header
> > inclusions below, so we may not need them.
> > --- a/src/backend/utils/adt/pg_upgrade_support.c
> > +++ b/src/backend/utils/adt/pg_upgrade_support.c
> > @@ -11,6 +11,7 @@
> > +#include "access/genam.h"
> > ...
> > +#include "utils/fmgroids.h"
> > ...
> > +#include "utils/snapmgr.h"
> > ~~~
> >
>
> Removed.
>
>
> > 5) Race condition in origin.c: replorigin_create()
> > - heap_freetuple(tuple);
> > + table_close(rel, ExclusiveLock);
> > +
> > + replorigin_create_with_id(roident, roname, InvalidXLogRecPtr);
> > +
> >
> > Due to the above refactoring, CREATE SUBSCRIPTION can now fail because
> > of a race between two concurrent CREATE SUBSCRIPTION commands:
> >
> > postgres=# create subscription sub2 connection 'dbname=postgres
> > host=localhost port=7733' publication pub1;
> > ERROR: replication origin with ID 2 already exists
> >
> > I think this happens because the table lock is released, allowing
> > another session to create a pg_origin with the next available roident,
> > which replorigin_create_with_id() later attempts to use.
> >
> > Would it make sense to call replorigin_create_with_id() before
> > table_close() so the lock is still held? I tested this and it works,
> > though I'm not fully sure whether self-locking could be a concern
> > here, since replorigin_create_with_id() again acquires a
> > RowExclusiveLock on the same table.
> > ~~~
> >
>
> I have fixed this as suggested by Shveta
>
> ~~~~~~~~~~
>
>
>
> On Thu, Jun 18, 2026 at 7:17 PM shveta malik <[email protected]> wrote:
> >
> > > I think this happens because the table lock is released, allowing
> > > another session to create a pg_origin with the next available roident,
> > > which replorigin_create_with_id() later attempts to use.
> > >
> > > Would it make sense to call replorigin_create_with_id() before
> > > table_close() so the lock is still held? I tested this and it works,
> > > though I'm not fully sure whether self-locking could be a concern
> > > here, since replorigin_create_with_id() again acquires a
> > > RowExclusiveLock on the same table.
> >
> > We can handle this race-condition by moving the table-open to callers
> > rather than internal call replorigin_create_with_id(). But we also
> > need the following changes first:
> >
>
> Yes, did it that way.
>
>
> > 1)
> > replorigin_create_with_id has:
> >
> > + /*
> > + * To avoid needing a TOAST table for pg_replication_origin, we limit
> > + * replication origin names to 512 bytes. This should be more than enough
> > + * for all practical use.
> > + */
> > + if (strlen(roname) > MAX_RONAME_LEN)
> > + ereport(ERROR,
> > + errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
> > + errmsg("replication origin name is too long"),
> > + errdetail("Replication origin names must be no longer than %d bytes.",
> > + MAX_RONAME_LEN));
> >
> > replorigin_create_with_id() is called by replorigin_create() and
> > binary_upgrade_create_replication_origin(). replorigin_create()
> > already has this check. We should move above check to
> > binary_upgrade_create_replication_origin() and get rid of it from
> > internal function replorigin_create_with_id().
>
> Moved.
>
> >
> > 2)
> >
> > Also the checks:
> > + if (SearchSysCacheExists1(REPLORIGIDENT, ObjectIdGetDatum(roident)))
> > + ereport(ERROR,
> > + errcode(ERRCODE_DUPLICATE_OBJECT),
> > + errmsg("replication origin with ID %u already exists",
> > + (Oid) roident));
> > +
> > + if (SearchSysCacheExists1(REPLORIGNAME, roname_d))
> > + ereport(ERROR,
> > + errcode(ERRCODE_DUPLICATE_OBJECT),
> > + errmsg("replication origin \"%s\" already exists", roname));
> >
> > should be moved to caller binary_upgrade_create_replication_origin().
> > IMO, these checks are not needed for replorigin_create(), it alread
> > has this, which should suffice.
> >
> > ereport(ERROR,
> > (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
> > errmsg("could not find free
> > replication origin ID")));
> >
> > ~~
> >
> > Once we correctly place these checks, we can move the 'table_open with
> > lock' to caller (before name and id collision checks) instead of
> > replorigin_create_with_id(), which will take care of race-conditions
> > reported by Nisha.
> >
>
> I have moved the roident check out of replorigin_create_with_id(), but
> kept the roname check there as that is not there in
> replorigin_create()
>
> On Fri, Jun 19, 2026 at 5:48 PM Shlok Kyal <[email protected]> wrote:
> > Hi Ajin,
> >
> > Thanks for providing the updated patch, I reviewed the patch and I
> > have some comments:
> >
> > 1. patch is not applying on HEAD and needs a rebase.
> >
> > 2. New line after 'check_new_cluster_replication_origins' is not required:
> > + check_new_cluster_replication_origins();
> > +
> >
> > 3. pg_dumpall with option '--roles-only' also dumps replication origins
> > eg:
> > ./pg_dumpall -p 9000 --binary-upgrade --roles-only > op.sql
> > output contains:
> > -- For binary upgrade, must preserve replication origin roident and remote_lsn
> > SELECT pg_catalog.binary_upgrade_create_replication_origin('1'::pg_catalog.oid,
> > 'pg_16387'::pg_catalog.text, '0/00000000'::pg_catalog.pg_lsn);
> >
> > I think in this case we should not dump the replication origins as we
> > explicitly gave the option of '--roles-only'
> >
>
> I have moved the dump_replication_origins under !roles_only and
> !tablespaces_only checks, I think "globals only" is valid for
> replication origins.
>
>
> > 4. pg_dumpall with --globals-only option also dumps replication origins.
> > eg:
> > ./pg_dumpall -p 9000 --binary-upgrade --globals-only> op.sql and
> > output contains:
> > -- For binary upgrade, must preserve replication origin roident and remote_lsn
> > SELECT pg_catalog.binary_upgrade_create_replication_origin('1'::pg_catalog.oid,
> > 'pg_16387'::pg_catalog.text, '0/00000000'::pg_catalog.pg_lsn);
> >
> > Is this behaviour intentional? Other logical replication objects
> > (e.g., publications and subscriptions) are not included in pg_dumpall
> > --globals-only, so is it okay to dump replication origin?
> >
>
> This is expected as publications and subscriptions are database
> specific and not globals, while replication origins are global.
>
>
>
> On Mon, Jun 22, 2026 at 2:22 AM Rui Zhao <[email protected]> wrote:
> >>
> >> Hi Ajin,
> >>
> >> Thanks for v8. A couple of comments:
> >>
> >> 1) The PG17 gating seems inconsistent between the dump and the check now.
> >> dumpReplicationOrigins() is called unconditionally:
> >>
> >> if (binary_upgrade && archDumpFormat == archNull)
> >> dumpReplicationOrigins(conn);
> >>
> >> but get_subscription_info(), which sets nrepl_origins, is still only
> >> called for old clusters >= 17 in check_new_cluster():
> >>
> >> if (GET_MAJOR_VERSION(old_cluster.major_version) >= 1700)
> >> ...
> >> get_subscription_info(&old_cluster);
> >>
> >> and check_new_cluster_replication_origins() returns early on it:
> >>
> >> if (old_cluster.nrepl_origins == 0)
> >> return;
> >>
> >> For a < 17 old cluster get_subscription_info() is never called, so
> >> nrepl_origins stays 0 and this returns early. So when such a cluster has a
> >> (user-created) origin, the origin still gets dumped and recreated, while
> >> the "new cluster must have no origins" and max_active_replication_origins
> >> checks are skipped, and the upgrade fails during restore instead of at the
> >> check step. nrepl_origins is just a count(*) on pg_replication_origin, so
> >> moving its collection out of the >= 1700 block should be enough.
> >>
>
> Yes, good catch. I have separated out the replication origin speciifc
> checks into a different function and moved it oustide the >= 1700
> block
>
> >> 2) 004_subscription.pl only checks subscription origins (pg_<oid>). Since
> >> the patch migrates all origins, should we also create a user origin
> >> (pg_replication_origin_create()) and verify its roident/remote_lsn after
> >> the upgrade? That would also cover the case in (1).
>
>
> Added.
>
> All the above changes have been addressed in patch v9.
>
> regards,
> Ajin Cherian
> Fujitsu Australia
^ permalink raw reply [nested|flat] 97+ messages in thread
* Re: [PATCH] Preserve replication origin OIDs in pg_upgrade
@ 2026-06-24 05:09 shveta malik <[email protected]>
parent: Ajin Cherian <[email protected]>
3 siblings, 1 reply; 97+ messages in thread
From: shveta malik @ 2026-06-24 05:09 UTC (permalink / raw)
To: Ajin Cherian <[email protected]>; +Cc: Rui Zhao <[email protected]>; Shlok Kyal <[email protected]>; Zsolt Parragi <[email protected]>; vignesh C <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; PostgreSQL Hackers <[email protected]>; shveta malik <[email protected]>
On Tue, Jun 23, 2026 at 5:20 PM Ajin Cherian <[email protected]> wrote:
>
> All the above changes have been addressed in patch v9.
>
Thanks for the patches. A few ocmments:
1)
+ * In binary-upgrade mode, skip origin creation here. This is required to
+ * preserve the roident from the old cluster for this subscription.
subscription --> subscription's origin
2)
replorigin_create_with_id():
+ if (SearchSysCacheExists1(REPLORIGNAME, roname_d))
+ ereport(ERROR,
+ errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("replication origin \"%s\" already exists", roname));
Can this ever happen? IIUC, since new-cluster should not have any
origin (as per the check introduced) and old cluster can not have
duplicate names, we should not hit this error. Do we want to keep it
as sanity check? If so, can we put a comment atop it.
3)
replorigin_create_with_id:
+ if (remote_lsn != InvalidXLogRecPtr)
+ replorigin_advance(roident, remote_lsn, InvalidXLogRecPtr,
+ false /* backward */,
+ false /* WAL log */);
We do not want to call 'advance' from regular 'replorigin_create'
flow. It is only for binary-Upgrade flow. Do you think we should have
a sanity check here?
if (remote_lsn != InvalidXLogRecPtr)
{
Assert(IsBinaryUpgrade)
replorigin_advance(..)
}
4)
replorigin_create:
+ found = true;
+ if (!found)
ereport(ERROR,
(errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
errmsg("could not find free replication origin ID")));
Do you think we can make use of 'collides' itself here instead of
adding a new 'found' variable. We can move 'collides' outside of loop
and use it to emit above error.
5)
binary_upgrade_create_replication_origin:
+ node = (ReplOriginId) node_oid;
+
+ if (SearchSysCacheExists1(REPLORIGIDENT, ObjectIdGetDatum(node)))
+ ereport(ERROR,
+ errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("replication origin with ID %u already exists",
+ (Oid) node));
How can we hit this? Same comment as #2.
6)
-check_new_cluster_subscription_configuration(void)
+check_new_cluster_replication_origins(void)
- /* Quick return if there are no subscriptions to be migrated. */
- if (old_cluster.nsubs == 0)
- return;
- if (old_cluster.nsubs > max_active_replication_origins)
With these changes, do we even need to capture old-cluster's 'nsubs'
in get_subscription_info(). I do not see any other usage of nsubs. Let
me know if I am missing any case.
thanks
Shveta
^ permalink raw reply [nested|flat] 97+ messages in thread
* Re: [PATCH] Preserve replication origin OIDs in pg_upgrade
@ 2026-06-24 05:21 shveta malik <[email protected]>
parent: shveta malik <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: shveta malik @ 2026-06-24 05:21 UTC (permalink / raw)
To: Ajin Cherian <[email protected]>; +Cc: Rui Zhao <[email protected]>; Shlok Kyal <[email protected]>; Zsolt Parragi <[email protected]>; vignesh C <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; PostgreSQL Hackers <[email protected]>; shveta malik <[email protected]>
On Wed, Jun 24, 2026 at 10:39 AM shveta malik <[email protected]> wrote:
>
>
> 2)
> replorigin_create_with_id():
>
> + if (SearchSysCacheExists1(REPLORIGNAME, roname_d))
> + ereport(ERROR,
> + errcode(ERRCODE_DUPLICATE_OBJECT),
> + errmsg("replication origin \"%s\" already exists", roname));
>
> Can this ever happen? IIUC, since new-cluster should not have any
> origin (as per the check introduced) and old cluster can not have
> duplicate names, we should not hit this error. Do we want to keep it
> as sanity check? If so, can we put a comment atop it.
>
Analyzed a bit more, with this change in patch002, we see an error change:
Without patch:
postgres=# SELECT pg_replication_origin_create('abcd');
pg_replication_origin_create
------------------------------
1
postgres=# SELECT pg_replication_origin_create('abcd');
ERROR: duplicate key value violates unique constraint
"pg_replication_origin_roname_index"
DETAIL: Key (roname)=(abcd) already exists.
----
With patch:
postgres=# SELECT pg_replication_origin_create('abcd');
pg_replication_origin_create
------------------------------
1
postgres=# SELECT pg_replication_origin_create('abcd');
ERROR: replication origin "abcd" already exists
So without the new check also, we can prevent duplicate name insertion
(even if that happens during upgrade, which I can not see how). The
new error and the check is not wrong, but slightly redundant. Or let
me know if I am missing something.
thanks
Shveta
^ permalink raw reply [nested|flat] 97+ messages in thread
* Re: [PATCH] Preserve replication origin OIDs in pg_upgrade
@ 2026-06-24 13:27 Shlok Kyal <[email protected]>
parent: Ajin Cherian <[email protected]>
3 siblings, 0 replies; 97+ messages in thread
From: Shlok Kyal @ 2026-06-24 13:27 UTC (permalink / raw)
To: Ajin Cherian <[email protected]>; +Cc: Rui Zhao <[email protected]>; shveta malik <[email protected]>; Zsolt Parragi <[email protected]>; vignesh C <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; PostgreSQL Hackers <[email protected]>; shveta malik <[email protected]>
On Tue, 23 Jun 2026 at 17:20, Ajin Cherian <[email protected]> wrote:
>
> On Wed, Jun 17, 2026 at 8:59 PM Nisha Moond <[email protected]> wrote:
> >
> > Hi Ajin, Thanks for the patches.
> > I have started reviewing the patches. Please find a few initial
> > comments below (from v7 review, still applicable to v8):
> >
> > 1) Patch-002: pg_dump.c
> > + if (dopt->binary_upgrade && subinfo->subenabled &&
> > fout->remoteVersion >= 170000)
> > {
> > - if (subinfo->suboriginremotelsn)
> > - {
> > - /*
> > - * Preserve the remote_lsn for the subscriber's replication
> >
> > I could not find any usage of subinfo[i].suboriginremotelsn anymore.
> > It seems to be dead code now. Can we remove it?
>
> Removed it.
>
>
> > ~~~
> >
> > 2) Patch-002 : binary_upgrade_create_replication_origin()
> > - PG_RETURN_VOID();
> > + node_oid = PG_GETARG_OID(0);
> > +
> > + if (node_oid == InvalidOid || node_oid > PG_UINT16_MAX)
> > + ereport(ERROR,
> > + errcode(ERRCODE_INVALID_PARAMETER_VALUE),
> > + errmsg("replication origin ID %u is out of range", node_oid));
> > +
> >
> > The above range check allows PG_UINT16_MAX, but that value is not part
> > of the valid replication origin ID range.
> >
> > In origin.h:
> > #define DoNotReplicateId PG_UINT16_MAX
> >
> > And we already ensure "ReplOriginId != DoNotReplicateId" everywhere
> > else. Would something like the below be more appropriate?
> > if (node_oid == InvalidReplOriginId || node_oid >= DoNotReplicateId)
> > ereport(ERROR, ...
> > ~~~
>
> Done.
>
>
> >
> > 3) I think we should add some additional details in
> > logical-replication.sgml, especially under <title>Prepare for
> > Subscriber Upgrades</title>.
> > Some suggestions:
> > 3a) Since all replication origins are now migrated, this:
> > "...greater than or equal to the number of subscriptions present in
> > the old cluster."
> > could become:
> > "...greater than or equal to the number of replication origins
> > present in the old cluster."
> >
> > 3b) We should also mention:
> > The new cluster must contain no replication origins.
> >
>
> Done.
>
>
>
> On Wed, Jun 17, 2026 at 9:42 PM Nisha Moond <[email protected]> wrote:
> >
> > 4) I was able to build successfully without all three new header
> > inclusions below, so we may not need them.
> > --- a/src/backend/utils/adt/pg_upgrade_support.c
> > +++ b/src/backend/utils/adt/pg_upgrade_support.c
> > @@ -11,6 +11,7 @@
> > +#include "access/genam.h"
> > ...
> > +#include "utils/fmgroids.h"
> > ...
> > +#include "utils/snapmgr.h"
> > ~~~
> >
>
> Removed.
>
>
> > 5) Race condition in origin.c: replorigin_create()
> > - heap_freetuple(tuple);
> > + table_close(rel, ExclusiveLock);
> > +
> > + replorigin_create_with_id(roident, roname, InvalidXLogRecPtr);
> > +
> >
> > Due to the above refactoring, CREATE SUBSCRIPTION can now fail because
> > of a race between two concurrent CREATE SUBSCRIPTION commands:
> >
> > postgres=# create subscription sub2 connection 'dbname=postgres
> > host=localhost port=7733' publication pub1;
> > ERROR: replication origin with ID 2 already exists
> >
> > I think this happens because the table lock is released, allowing
> > another session to create a pg_origin with the next available roident,
> > which replorigin_create_with_id() later attempts to use.
> >
> > Would it make sense to call replorigin_create_with_id() before
> > table_close() so the lock is still held? I tested this and it works,
> > though I'm not fully sure whether self-locking could be a concern
> > here, since replorigin_create_with_id() again acquires a
> > RowExclusiveLock on the same table.
> > ~~~
> >
>
> I have fixed this as suggested by Shveta
>
> ~~~~~~~~~~
>
>
>
> On Thu, Jun 18, 2026 at 7:17 PM shveta malik <[email protected]> wrote:
> >
> > > I think this happens because the table lock is released, allowing
> > > another session to create a pg_origin with the next available roident,
> > > which replorigin_create_with_id() later attempts to use.
> > >
> > > Would it make sense to call replorigin_create_with_id() before
> > > table_close() so the lock is still held? I tested this and it works,
> > > though I'm not fully sure whether self-locking could be a concern
> > > here, since replorigin_create_with_id() again acquires a
> > > RowExclusiveLock on the same table.
> >
> > We can handle this race-condition by moving the table-open to callers
> > rather than internal call replorigin_create_with_id(). But we also
> > need the following changes first:
> >
>
> Yes, did it that way.
>
>
> > 1)
> > replorigin_create_with_id has:
> >
> > + /*
> > + * To avoid needing a TOAST table for pg_replication_origin, we limit
> > + * replication origin names to 512 bytes. This should be more than enough
> > + * for all practical use.
> > + */
> > + if (strlen(roname) > MAX_RONAME_LEN)
> > + ereport(ERROR,
> > + errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
> > + errmsg("replication origin name is too long"),
> > + errdetail("Replication origin names must be no longer than %d bytes.",
> > + MAX_RONAME_LEN));
> >
> > replorigin_create_with_id() is called by replorigin_create() and
> > binary_upgrade_create_replication_origin(). replorigin_create()
> > already has this check. We should move above check to
> > binary_upgrade_create_replication_origin() and get rid of it from
> > internal function replorigin_create_with_id().
>
> Moved.
>
> >
> > 2)
> >
> > Also the checks:
> > + if (SearchSysCacheExists1(REPLORIGIDENT, ObjectIdGetDatum(roident)))
> > + ereport(ERROR,
> > + errcode(ERRCODE_DUPLICATE_OBJECT),
> > + errmsg("replication origin with ID %u already exists",
> > + (Oid) roident));
> > +
> > + if (SearchSysCacheExists1(REPLORIGNAME, roname_d))
> > + ereport(ERROR,
> > + errcode(ERRCODE_DUPLICATE_OBJECT),
> > + errmsg("replication origin \"%s\" already exists", roname));
> >
> > should be moved to caller binary_upgrade_create_replication_origin().
> > IMO, these checks are not needed for replorigin_create(), it alread
> > has this, which should suffice.
> >
> > ereport(ERROR,
> > (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
> > errmsg("could not find free
> > replication origin ID")));
> >
> > ~~
> >
> > Once we correctly place these checks, we can move the 'table_open with
> > lock' to caller (before name and id collision checks) instead of
> > replorigin_create_with_id(), which will take care of race-conditions
> > reported by Nisha.
> >
>
> I have moved the roident check out of replorigin_create_with_id(), but
> kept the roname check there as that is not there in
> replorigin_create()
>
> On Fri, Jun 19, 2026 at 5:48 PM Shlok Kyal <[email protected]> wrote:
> > Hi Ajin,
> >
> > Thanks for providing the updated patch, I reviewed the patch and I
> > have some comments:
> >
> > 1. patch is not applying on HEAD and needs a rebase.
> >
> > 2. New line after 'check_new_cluster_replication_origins' is not required:
> > + check_new_cluster_replication_origins();
> > +
> >
> > 3. pg_dumpall with option '--roles-only' also dumps replication origins
> > eg:
> > ./pg_dumpall -p 9000 --binary-upgrade --roles-only > op.sql
> > output contains:
> > -- For binary upgrade, must preserve replication origin roident and remote_lsn
> > SELECT pg_catalog.binary_upgrade_create_replication_origin('1'::pg_catalog.oid,
> > 'pg_16387'::pg_catalog.text, '0/00000000'::pg_catalog.pg_lsn);
> >
> > I think in this case we should not dump the replication origins as we
> > explicitly gave the option of '--roles-only'
> >
>
> I have moved the dump_replication_origins under !roles_only and
> !tablespaces_only checks, I think "globals only" is valid for
> replication origins.
>
>
> > 4. pg_dumpall with --globals-only option also dumps replication origins.
> > eg:
> > ./pg_dumpall -p 9000 --binary-upgrade --globals-only> op.sql and
> > output contains:
> > -- For binary upgrade, must preserve replication origin roident and remote_lsn
> > SELECT pg_catalog.binary_upgrade_create_replication_origin('1'::pg_catalog.oid,
> > 'pg_16387'::pg_catalog.text, '0/00000000'::pg_catalog.pg_lsn);
> >
> > Is this behaviour intentional? Other logical replication objects
> > (e.g., publications and subscriptions) are not included in pg_dumpall
> > --globals-only, so is it okay to dump replication origin?
> >
>
> This is expected as publications and subscriptions are database
> specific and not globals, while replication origins are global.
>
>
>
> On Mon, Jun 22, 2026 at 2:22 AM Rui Zhao <[email protected]> wrote:
> >>
> >> Hi Ajin,
> >>
> >> Thanks for v8. A couple of comments:
> >>
> >> 1) The PG17 gating seems inconsistent between the dump and the check now.
> >> dumpReplicationOrigins() is called unconditionally:
> >>
> >> if (binary_upgrade && archDumpFormat == archNull)
> >> dumpReplicationOrigins(conn);
> >>
> >> but get_subscription_info(), which sets nrepl_origins, is still only
> >> called for old clusters >= 17 in check_new_cluster():
> >>
> >> if (GET_MAJOR_VERSION(old_cluster.major_version) >= 1700)
> >> ...
> >> get_subscription_info(&old_cluster);
> >>
> >> and check_new_cluster_replication_origins() returns early on it:
> >>
> >> if (old_cluster.nrepl_origins == 0)
> >> return;
> >>
> >> For a < 17 old cluster get_subscription_info() is never called, so
> >> nrepl_origins stays 0 and this returns early. So when such a cluster has a
> >> (user-created) origin, the origin still gets dumped and recreated, while
> >> the "new cluster must have no origins" and max_active_replication_origins
> >> checks are skipped, and the upgrade fails during restore instead of at the
> >> check step. nrepl_origins is just a count(*) on pg_replication_origin, so
> >> moving its collection out of the >= 1700 block should be enough.
> >>
>
> Yes, good catch. I have separated out the replication origin speciifc
> checks into a different function and moved it oustide the >= 1700
> block
>
> >> 2) 004_subscription.pl only checks subscription origins (pg_<oid>). Since
> >> the patch migrates all origins, should we also create a user origin
> >> (pg_replication_origin_create()) and verify its roident/remote_lsn after
> >> the upgrade? That would also cover the case in (1).
>
>
> Added.
>
> All the above changes have been addressed in patch v9.
>
Thanks for the updated patch. I have reviewed the patch and here are
some my comments:
1. Since we are quering only one column, do we need to use 'i_norigins'?
+void
+get_replication_origin_info(ClusterInfo *cluster)
+{
+ PGconn *conn;
+ PGresult *res;
+ int i_norigins;
+
+ conn = connectToServer(cluster, "template1");
+ res = executeQueryOrDie(conn, "SELECT count(*) AS norigins "
+ "FROM pg_catalog.pg_replication_origin");
+ i_norigins = PQfnumber(res, "norigins");
+
+ cluster->nrepl_origins = atoi(PQgetvalue(res, 0, i_norigins));
+ PQclear(res);
+
+ PQfinish(conn);
+}
I think we can directly use:
cluster->nrepl_origins = atoi(PQgetvalue(res, 0, 0));
Thoughts?
2. In the above code should we also add a check like:
if (PQntuples(res) != 1)
pg_fatal("could not count the number of replication origins");
3. The brackets are not required:
+ if (!tablespaces_only && !roles_only && binary_upgrade)
+ {
+ /* Dump replication origins */
+ dumpReplicationOrigins(conn);
}
We can move the comment '/* Dump replication origins */' above if condition.
It will make it consistent with the existing code.
4. I ran the tests locally and found that the test_decoding test suit
is failing:
not ok 10 - replorigin 249 ms
diff:
-- ensure duplicate creations fail
SELECT pg_replication_origin_create('regress_test_decoding: regression_slot');
-ERROR: duplicate key value violates unique constraint
"pg_replication_origin_roname_index"
-DETAIL: Key (roname)=(regress_test_decoding: regression_slot) already exists.
+ERROR: replication origin "regress_test_decoding: regression_slot"
already exists
-- ensure inactive origin cannot be set as session one if pid is specified
SELECT pg_replication_origin_session_setup('regress_test_decoding:
regression_slot', -1);
ERROR: cannot use PID -1 for inactive replication origin with ID 1
I think this is expected because this patch adds a new function
'replorigin_create_with_id'
which adds a check for the already existing replication origin.
In HEAD we don't have such a check. So I think we should update the
expected output file.
Thanks,
Shlok Kyal
^ permalink raw reply [nested|flat] 97+ messages in thread
* Re: [PATCH] Preserve replication origin OIDs in pg_upgrade
@ 2026-06-24 16:44 Zsolt Parragi <[email protected]>
parent: Ajin Cherian <[email protected]>
3 siblings, 1 reply; 97+ messages in thread
From: Zsolt Parragi @ 2026-06-24 16:44 UTC (permalink / raw)
To: [email protected]
Hello
+# Verify that the subscription related replication origins are
preserved after upgrade.
+my $post_roident_rows = $new_sub->safe_psql('postgres',
+ "SELECT s.subname, o.roident
+ FROM pg_subscription s
+ JOIN pg_replication_origin o ON o.roname = 'pg_' || s.oid::text
+ ORDER BY s.subname"
+);
+for my $row (split /\n/, $post_roident_rows)
+{
+ my ($subname, $roident) = split /\|/, $row;
+ is($roident, $pre_upgrade_roident{$subname},
+ "roident preserved for subscription '$subname' after upgrade");
+}
+
Won't this check miss (and silently pass) if an entry is missing
post-upgrade? It verifies that all entries that exists match the pre
upgrade entries, but it doesn't verify that the list is the same.
^ permalink raw reply [nested|flat] 97+ messages in thread
* Re: [PATCH] Preserve replication origin OIDs in pg_upgrade
@ 2026-07-01 02:49 Ajin Cherian <[email protected]>
parent: Zsolt Parragi <[email protected]>
0 siblings, 3 replies; 97+ messages in thread
From: Ajin Cherian @ 2026-07-01 02:49 UTC (permalink / raw)
To: Zsolt Parragi <[email protected]>; +Cc: [email protected]
On Wed, Jun 24, 2026 at 2:07 AM Rui Zhao <[email protected]> wrote:
>
> Hi Ajin,
>
> Thanks for v9, and for addressing both my earlier comments. I also went
> through the rest of v9 -- the other reviewers' comments and the
> corresponding fixes (including the replorigin_create() locking rework) look
> good to me; I didn't find any issues there.
>
> A follow-up on the version gating, though: I think it now moved a bit too
> far. get_replication_origin_info() is called unconditionally (outside the
> >= 1700 block):
>
> /* Get replication origin information */
> get_replication_origin_info(&old_cluster);
>
> and it runs "SELECT count(*) FROM pg_catalog.pg_replication_origin".
> pg_dumpall's dumpReplicationOrigins() queries the same catalog with no
> version guard either. But pg_replication_origin only exists from 9.5
> (commit 5aa2350426), while pg_upgrade still accepts old clusters back to
> 9.2:
>
> if (GET_MAJOR_VERSION(old_cluster.major_version) < 902)
> pg_fatal("This utility can only upgrade from PostgreSQL version %s
> and later.", "9.2");
>
> So upgrading from 9.2/9.3/9.4 now fails: the query hits a non-existent
> catalog and pg_fatal()s (and dumpReplicationOrigins would too).
> It should be gated on the version where origins were introduced, e.g.
>
> if (GET_MAJOR_VERSION(old_cluster.major_version) >= 905)
> get_replication_origin_info(&old_cluster);
>
> with the matching server_version >= 90500 check on the pg_dumpall side.
> That keeps the < 17 case I flagged working while not breaking 9.2-9.4
> (which have no origins to migrate anyway).
>
Thanks, I have added both the flags.
> One nit while here: there is an extra blank line after the
> check_new_cluster_replication_origins() call in check_new_cluster().
>
Removed.
On Wed, Jun 24, 2026 at 3:09 PM shveta malik <[email protected]> wrote:
>
> On Tue, Jun 23, 2026 at 5:20 PM Ajin Cherian <[email protected]> wrote:
> >
> > All the above changes have been addressed in patch v9.
> >
>
> Thanks for the patches. A few ocmments:
>
> 1)
>
> + * In binary-upgrade mode, skip origin creation here. This is required to
> + * preserve the roident from the old cluster for this subscription.
>
> subscription --> subscription's origin
>
Changed.
> 2)
> replorigin_create_with_id():
>
> + if (SearchSysCacheExists1(REPLORIGNAME, roname_d))
> + ereport(ERROR,
> + errcode(ERRCODE_DUPLICATE_OBJECT),
> + errmsg("replication origin \"%s\" already exists", roname));
>
> Can this ever happen? IIUC, since new-cluster should not have any
> origin (as per the check introduced) and old cluster can not have
> duplicate names, we should not hit this error. Do we want to keep it
> as sanity check? If so, can we put a comment atop it.
>
I've removed it.
> 3)
> replorigin_create_with_id:
>
> + if (remote_lsn != InvalidXLogRecPtr)
> + replorigin_advance(roident, remote_lsn, InvalidXLogRecPtr,
> + false /* backward */,
> + false /* WAL log */);
>
> We do not want to call 'advance' from regular 'replorigin_create'
> flow. It is only for binary-Upgrade flow. Do you think we should have
> a sanity check here?
>
> if (remote_lsn != InvalidXLogRecPtr)
> {
> Assert(IsBinaryUpgrade)
> replorigin_advance(..)
> }
>
Done.
> 4)
> replorigin_create:
>
> + found = true;
>
> + if (!found)
> ereport(ERROR,
> (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
> errmsg("could not find free replication origin ID")));
>
> Do you think we can make use of 'collides' itself here instead of
> adding a new 'found' variable. We can move 'collides' outside of loop
> and use it to emit above error.
>
I've removed found and instead checked if the roident has reached the max limit
> 5)
> binary_upgrade_create_replication_origin:
>
> + node = (ReplOriginId) node_oid;
> +
> + if (SearchSysCacheExists1(REPLORIGIDENT, ObjectIdGetDatum(node)))
> + ereport(ERROR,
> + errcode(ERRCODE_DUPLICATE_OBJECT),
> + errmsg("replication origin with ID %u already exists",
> + (Oid) node));
>
> How can we hit this? Same comment as #2.
>
Removed
> 6)
> -check_new_cluster_subscription_configuration(void)
> +check_new_cluster_replication_origins(void)
>
> - /* Quick return if there are no subscriptions to be migrated. */
> - if (old_cluster.nsubs == 0)
> - return;
>
> - if (old_cluster.nsubs > max_active_replication_origins)
>
> With these changes, do we even need to capture old-cluster's 'nsubs'
> in get_subscription_info(). I do not see any other usage of nsubs. Let
> me know if I am missing any case.
I've removed nsubs
On Wed, Jun 24, 2026 at 3:21 PM shveta malik <[email protected]> wrote:
>
> On Wed, Jun 24, 2026 at 10:39 AM shveta malik <[email protected]> wrote:
> >
> >
> > 2)
> > replorigin_create_with_id():
> >
> > + if (SearchSysCacheExists1(REPLORIGNAME, roname_d))
> > + ereport(ERROR,
> > + errcode(ERRCODE_DUPLICATE_OBJECT),
> > + errmsg("replication origin \"%s\" already exists", roname));
> >
> > Can this ever happen? IIUC, since new-cluster should not have any
> > origin (as per the check introduced) and old cluster can not have
> > duplicate names, we should not hit this error. Do we want to keep it
> > as sanity check? If so, can we put a comment atop it.
> >
>
> Analyzed a bit more, with this change in patch002, we see an error change:
>
> Without patch:
> postgres=# SELECT pg_replication_origin_create('abcd');
> pg_replication_origin_create
> ------------------------------
> 1
> postgres=# SELECT pg_replication_origin_create('abcd');
> ERROR: duplicate key value violates unique constraint
> "pg_replication_origin_roname_index"
> DETAIL: Key (roname)=(abcd) already exists.
>
> ----
>
> With patch:
> postgres=# SELECT pg_replication_origin_create('abcd');
> pg_replication_origin_create
> ------------------------------
> 1
>
> postgres=# SELECT pg_replication_origin_create('abcd');
> ERROR: replication origin "abcd" already exists
>
>
> So without the new check also, we can prevent duplicate name insertion
> (even if that happens during upgrade, which I can not see how). The
> new error and the check is not wrong, but slightly redundant. Or let
> me know if I am missing something.
>
I've removed that check.
On Wed, Jun 24, 2026 at 11:27 PM Shlok Kyal <[email protected]> wrote:
>
> Thanks for the updated patch. I have reviewed the patch and here are
> some my comments:
>
> 1. Since we are quering only one column, do we need to use 'i_norigins'?
>
> +void
> +get_replication_origin_info(ClusterInfo *cluster)
> +{
> + PGconn *conn;
> + PGresult *res;
> + int i_norigins;
> +
> + conn = connectToServer(cluster, "template1");
> + res = executeQueryOrDie(conn, "SELECT count(*) AS norigins "
> + "FROM pg_catalog.pg_replication_origin");
> + i_norigins = PQfnumber(res, "norigins");
> +
> + cluster->nrepl_origins = atoi(PQgetvalue(res, 0, i_norigins));
> + PQclear(res);
> +
> + PQfinish(conn);
> +}
> I think we can directly use:
> cluster->nrepl_origins = atoi(PQgetvalue(res, 0, 0));
>
> Thoughts?
Removed it.
>
> 2. In the above code should we also add a check like:
> if (PQntuples(res) != 1)
> pg_fatal("could not count the number of replication origins");
>
Added.
> 3. The brackets are not required:
> + if (!tablespaces_only && !roles_only && binary_upgrade)
> + {
> + /* Dump replication origins */
> + dumpReplicationOrigins(conn);
> }
> We can move the comment '/* Dump replication origins */' above if condition.
> It will make it consistent with the existing code.
>
I've moved the comment up but keeping the brackets as it has more than
one line, that is the recommendation from Tom Lane.
> 4. I ran the tests locally and found that the test_decoding test suit
> is failing:
> not ok 10 - replorigin 249 ms
>
> diff:
> -- ensure duplicate creations fail
> SELECT pg_replication_origin_create('regress_test_decoding: regression_slot');
> -ERROR: duplicate key value violates unique constraint
> "pg_replication_origin_roname_index"
> -DETAIL: Key (roname)=(regress_test_decoding: regression_slot) already exists.
> +ERROR: replication origin "regress_test_decoding: regression_slot"
> already exists
> -- ensure inactive origin cannot be set as session one if pid is specified
> SELECT pg_replication_origin_session_setup('regress_test_decoding:
> regression_slot', -1);
> ERROR: cannot use PID -1 for inactive replication origin with ID 1
>
> I think this is expected because this patch adds a new function
> 'replorigin_create_with_id'
> which adds a check for the already existing replication origin.
> In HEAD we don't have such a check. So I think we should update the
> expected output file.
>
Based on feedback from Shveta, I've removed that new error as changing
existing behaviour isn't required.
On Thu, Jun 25, 2026 at 2:45 AM Zsolt Parragi <[email protected]> wrote:
>
> Hello
>
> +# Verify that the subscription related replication origins are
> preserved after upgrade.
> +my $post_roident_rows = $new_sub->safe_psql('postgres',
> + "SELECT s.subname, o.roident
> + FROM pg_subscription s
> + JOIN pg_replication_origin o ON o.roname = 'pg_' || s.oid::text
> + ORDER BY s.subname"
> +);
> +for my $row (split /\n/, $post_roident_rows)
> +{
> + my ($subname, $roident) = split /\|/, $row;
> + is($roident, $pre_upgrade_roident{$subname},
> + "roident preserved for subscription '$subname' after upgrade");
> +}
> +
>
> Won't this check miss (and silently pass) if an entry is missing
> post-upgrade? It verifies that all entries that exists match the pre
> upgrade entries, but it doesn't verify that the list is the same.
Modified this to iterate over pre upgrade and post upgrade
subscriptions making sure both are present.
All the above changes are updated in patch v10.
regards,
Ajin Cherian
Fujitsu Australia
Attachments:
[application/octet-stream] v10-0001-Preserve-subscription-OIDs-during-pg_upgrade.patch (9.1K, ../../CAFPTHDYfDHQeQshXi7GDOQz3N8YdY6SNSmpoSAbnaQPBkVfv6A@mail.gmail.com/2-v10-0001-Preserve-subscription-OIDs-during-pg_upgrade.patch)
download | inline diff:
From cb9b9da7b14073924b635be0025fef9154203d96 Mon Sep 17 00:00:00 2001
From: Ajin Cherian <[email protected]>
Date: Mon, 29 Jun 2026 21:46:29 +1000
Subject: [PATCH v10 1/2] Preserve subscription OIDs during pg_upgrade
Currently subscription OIDs can be changed when a cluster is upgraded
using pg_upgrade. This is required for a subsequent patch which will
preserve the replication oids after upgrade.
Author: Vignesh C <[email protected]>
---
src/backend/commands/subscriptioncmds.c | 25 +++++++++++++++++--
src/backend/utils/adt/pg_upgrade_support.c | 10 ++++++++
src/bin/pg_dump/pg_dump.c | 8 ++++++
src/bin/pg_upgrade/pg_upgrade.c | 3 +++
src/bin/pg_upgrade/t/004_subscription.pl | 7 ++++++
src/include/catalog/binary_upgrade.h | 1 +
src/include/catalog/pg_proc.dat | 4 +++
.../expected/spgist_name_ops.out | 6 +++--
8 files changed, 60 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index ee06a726f42..5fe7440418c 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -83,6 +83,12 @@
/* check if the 'val' has 'bits' set */
#define IsSet(val, bits) (((val) & (bits)) == (bits))
+/*
+ * This will be set by the pg_upgrade_support function --
+ * binary_upgrade_set_next_pg_subscription_oid().
+ */
+Oid binary_upgrade_next_pg_subscription_oid = InvalidOid;
+
/*
* Structure to hold a bitmap representing the user-provided CREATE/ALTER
* SUBSCRIPTION command options and the parsed/default values of each of them.
@@ -803,8 +809,23 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
memset(values, 0, sizeof(values));
memset(nulls, false, sizeof(nulls));
- subid = GetNewOidWithIndex(rel, SubscriptionObjectIndexId,
- Anum_pg_subscription_oid);
+ /* Use binary-upgrade override for pg_subscription.oid? */
+ if (IsBinaryUpgrade)
+ {
+ if (!OidIsValid(binary_upgrade_next_pg_subscription_oid))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("pg_subscription OID value not set when in binary upgrade mode")));
+
+ subid = binary_upgrade_next_pg_subscription_oid;
+ binary_upgrade_next_pg_subscription_oid = InvalidOid;
+ }
+ else
+ {
+ subid = GetNewOidWithIndex(rel, SubscriptionObjectIndexId,
+ Anum_pg_subscription_oid);
+ }
+
values[Anum_pg_subscription_oid - 1] = ObjectIdGetDatum(subid);
values[Anum_pg_subscription_subdbid - 1] = ObjectIdGetDatum(MyDatabaseId);
values[Anum_pg_subscription_subskiplsn - 1] = LSNGetDatum(InvalidXLogRecPtr);
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index b505a6b4fee..59c3e7f0146 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -181,6 +181,16 @@ binary_upgrade_set_next_pg_authid_oid(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
+Datum
+binary_upgrade_set_next_pg_subscription_oid(PG_FUNCTION_ARGS)
+{
+ Oid subid = PG_GETARG_OID(0);
+
+ CHECK_IS_BINARY_UPGRADE;
+ binary_upgrade_next_pg_subscription_oid = subid;
+ PG_RETURN_VOID();
+}
+
Datum
binary_upgrade_create_empty_extension(PG_FUNCTION_ARGS)
{
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index c56437d6057..ea22b0604c1 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -5583,6 +5583,14 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
appendPQExpBuffer(delq, "DROP SUBSCRIPTION %s;\n",
qsubname);
+ if (dopt->binary_upgrade)
+ {
+ appendPQExpBufferStr(query, "\n-- For binary upgrade, must preserve pg_subscription.oid\n");
+ appendPQExpBuffer(query,
+ "SELECT pg_catalog.binary_upgrade_set_next_pg_subscription_oid('%u'::pg_catalog.oid);\n\n",
+ subinfo->dobj.catId.oid);
+ }
+
appendPQExpBuffer(query, "CREATE SUBSCRIPTION %s ",
qsubname);
if (subinfo->subservername)
diff --git a/src/bin/pg_upgrade/pg_upgrade.c b/src/bin/pg_upgrade/pg_upgrade.c
index 2127d297bfe..4e853096698 100644
--- a/src/bin/pg_upgrade/pg_upgrade.c
+++ b/src/bin/pg_upgrade/pg_upgrade.c
@@ -35,6 +35,9 @@
*
* We control all assignments of pg_database.oid because we want the directory
* names to match between the old and new cluster.
+ *
+ * We control assignment of pg_subscription.oid because we want the oid to
+ * match between the old and new cluster.
*/
diff --git a/src/bin/pg_upgrade/t/004_subscription.pl b/src/bin/pg_upgrade/t/004_subscription.pl
index 36f059faa2e..9286c1dc85e 100644
--- a/src/bin/pg_upgrade/t/004_subscription.pl
+++ b/src/bin/pg_upgrade/t/004_subscription.pl
@@ -313,6 +313,9 @@ my $tab_upgraded1_oid = $old_sub->safe_psql('postgres',
my $tab_upgraded2_oid = $old_sub->safe_psql('postgres',
"SELECT oid FROM pg_class WHERE relname = 'tab_upgraded2'");
+$sub_oid = $old_sub->safe_psql('postgres',
+ "SELECT oid FROM pg_subscription ORDER BY subname");
+
$old_sub->stop;
# Change configuration so that initial table sync does not get started
@@ -359,6 +362,10 @@ $publisher->safe_psql(
$new_sub->start;
+# The subscription oid should be preserved
+$result = $new_sub->safe_psql('postgres', "SELECT oid FROM pg_subscription ORDER BY subname");
+is($result, qq($sub_oid), "subscription oid should have been preserved");
+
# The subscription's running status, failover option, and retain_dead_tuples
# option should be preserved in the upgraded instance. So regress_sub4 should
# still have subenabled, subfailover, and subretaindeadtuples set to true,
diff --git a/src/include/catalog/binary_upgrade.h b/src/include/catalog/binary_upgrade.h
index 7bf7ae44385..b15b18e7dc9 100644
--- a/src/include/catalog/binary_upgrade.h
+++ b/src/include/catalog/binary_upgrade.h
@@ -32,6 +32,7 @@ extern PGDLLIMPORT RelFileNumber binary_upgrade_next_toast_pg_class_relfilenumbe
extern PGDLLIMPORT Oid binary_upgrade_next_pg_enum_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_authid_oid;
+extern PGDLLIMPORT Oid binary_upgrade_next_pg_subscription_oid;
extern PGDLLIMPORT bool binary_upgrade_record_init_privs;
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 402d869710b..231ce2ca91e 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -11968,6 +11968,10 @@
proisstrict => 'f', provolatile => 'v', proparallel => 'u',
prorettype => 'void', proargtypes => '',
prosrc => 'binary_upgrade_create_conflict_detection_slot' },
+{ oid => '9160', descr => 'for use by pg_upgrade',
+ proname => 'binary_upgrade_set_next_pg_subscription_oid', provolatile => 'v',
+ proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
+ prosrc => 'binary_upgrade_set_next_pg_subscription_oid' },
# conversion functions
{ oid => '4310', descr => 'internal conversion function for KOI8R to WIN1251',
diff --git a/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out b/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out
index 1ee65ede243..39d43368c42 100644
--- a/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out
+++ b/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out
@@ -59,11 +59,12 @@ select * from t
binary_upgrade_set_next_multirange_pg_type_oid | 1 | binary_upgrade_set_next_multirange_pg_type_oid
binary_upgrade_set_next_pg_authid_oid | | binary_upgrade_set_next_pg_authid_oid
binary_upgrade_set_next_pg_enum_oid | | binary_upgrade_set_next_pg_enum_oid
+ binary_upgrade_set_next_pg_subscription_oid | | binary_upgrade_set_next_pg_subscription_oid
binary_upgrade_set_next_pg_tablespace_oid | | binary_upgrade_set_next_pg_tablespace_oid
binary_upgrade_set_next_pg_type_oid | | binary_upgrade_set_next_pg_type_oid
binary_upgrade_set_next_toast_pg_class_oid | 1 | binary_upgrade_set_next_toast_pg_class_oid
binary_upgrade_set_next_toast_relfilenode | | binary_upgrade_set_next_toast_relfilenode
-(13 rows)
+(14 rows)
-- Verify clean failure when INCLUDE'd columns result in overlength tuple
-- The error message details are platform-dependent, so show only SQLSTATE
@@ -108,11 +109,12 @@ select * from t
binary_upgrade_set_next_multirange_pg_type_oid | 1 | binary_upgrade_set_next_multirange_pg_type_oid
binary_upgrade_set_next_pg_authid_oid | | binary_upgrade_set_next_pg_authid_oid
binary_upgrade_set_next_pg_enum_oid | | binary_upgrade_set_next_pg_enum_oid
+ binary_upgrade_set_next_pg_subscription_oid | | binary_upgrade_set_next_pg_subscription_oid
binary_upgrade_set_next_pg_tablespace_oid | | binary_upgrade_set_next_pg_tablespace_oid
binary_upgrade_set_next_pg_type_oid | | binary_upgrade_set_next_pg_type_oid
binary_upgrade_set_next_toast_pg_class_oid | 1 | binary_upgrade_set_next_toast_pg_class_oid
binary_upgrade_set_next_toast_relfilenode | | binary_upgrade_set_next_toast_relfilenode
-(13 rows)
+(14 rows)
\set VERBOSITY sqlstate
insert into t values(repeat('xyzzy', 12), 42, repeat('xyzzy', 4000));
--
2.47.3
[application/octet-stream] v10-0002-Preserve-replication-origin-OIDs-during-pg_upgra.patch (33.9K, ../../CAFPTHDYfDHQeQshXi7GDOQz3N8YdY6SNSmpoSAbnaQPBkVfv6A@mail.gmail.com/3-v10-0002-Preserve-replication-origin-OIDs-during-pg_upgra.patch)
download | inline diff:
From b89f8f95a970223a3fe1672946b31060c3551355 Mon Sep 17 00:00:00 2001
From: Ajin Cherian <[email protected]>
Date: Wed, 1 Jul 2026 12:39:16 +1000
Subject: [PATCH v10 2/2] Preserve replication origin OIDs during pg_upgrade
When pg_upgrade migrates a subscriber, replication origin OIDs
(roident) can change across the upgrade. This is a problem because
commit-timestamp records embed roident and are copied directly from
the old cluster's pg_commit_ts directory, causing spurious
"update_origin_differs" conflicts after the upgrade.
Fix this by dumping replication origins as global objects via
pg_dumpall during binary upgrade, using a new function
binary_upgrade_create_replication_origin(oid, name, lsn) to recreate
each origin with its preserved roident and remote_lsn. To avoid
conflicts with this, CreateSubscription() skips replorigin_create()
in binary-upgrade mode since the origin is already created by the
time the subscription is restored.
Author: Ajin Cherian <[email protected]>
Reviewer: Hayato Kuroda (Fujitsu) <[email protected]>
Reviewer: Zsolt Parragi <[email protected]>
Reviewer: Shlok Kyal <[email protected]>
Reviewer: Shveta malik <[email protected]>
Reviewer: Vignesh C <[email protected]>
Reviewer: Nisha Moond <[email protected]>
Reviewer: Rui Zhao <[email protected]>
---
doc/src/sgml/logical-replication.sgml | 4 +-
src/backend/commands/subscriptioncmds.c | 11 ++-
src/backend/replication/logical/origin.c | 96 ++++++++++++--------
src/backend/utils/adt/pg_upgrade_support.c | 101 +++++++++++----------
src/bin/pg_dump/pg_dump.c | 51 ++---------
src/bin/pg_dump/pg_dump.h | 1 -
src/bin/pg_dump/pg_dumpall.c | 68 ++++++++++++++
src/bin/pg_upgrade/check.c | 57 ++++++++----
src/bin/pg_upgrade/info.c | 35 +++++--
src/bin/pg_upgrade/pg_upgrade.h | 3 +-
src/bin/pg_upgrade/t/004_subscription.pl | 69 +++++++++++++-
src/include/catalog/pg_proc.dat | 8 +-
src/include/replication/origin.h | 3 +
13 files changed, 345 insertions(+), 162 deletions(-)
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index 5befefd9c5a..1eb64259ff3 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -2819,7 +2819,7 @@ CONTEXT: processing remote data for replication origin "pg_16395" during "INSER
<note>
<para>
- Commit timestamps and origin data are not preserved during the upgrade.
+ Commit timestamps are not preserved during the upgrade.
As a result, even if
<link linkend="sql-createsubscription-params-with-retain-dead-tuples"><literal>retain_dead_tuples</literal></link>
is enabled, the upgraded subscriber may be unable to detect conflicts or
@@ -2861,7 +2861,7 @@ CONTEXT: processing remote data for replication origin "pg_16395" during "INSER
The new cluster must have
<link linkend="guc-max-active-replication-origins"><varname>max_active_replication_origins</varname></link>
configured to a value greater than or equal to the number of
- subscriptions present in the old cluster.
+ replication origins present in the old cluster. The new cluster must contain no replication origins.
</para>
</listitem>
<listitem>
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 5fe7440418c..a10cda4e0e7 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -19,6 +19,7 @@
#include "access/table.h"
#include "access/twophase.h"
#include "access/xact.h"
+#include "catalog/binary_upgrade.h"
#include "catalog/catalog.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
@@ -898,9 +899,15 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
* apply workers initialization, and to handle origin creation dynamically
* when tables are added to the subscription. It is not clear whether
* preventing creation of origins is worth additional complexity.
+ *
+ * In binary-upgrade mode, skip origin creation here. This is required to
+ * preserve the roident from the old cluster for this subscription's origin.
*/
- ReplicationOriginNameForLogicalRep(subid, InvalidOid, originname, sizeof(originname));
- replorigin_create(originname);
+ if (!IsBinaryUpgrade)
+ {
+ ReplicationOriginNameForLogicalRep(subid, InvalidOid, originname, sizeof(originname));
+ replorigin_create(originname);
+ }
/*
* Connect to remote side to execute requested commands and fetch table
diff --git a/src/backend/replication/logical/origin.c b/src/backend/replication/logical/origin.c
index c9dfb094c2b..a9868a29b14 100644
--- a/src/backend/replication/logical/origin.c
+++ b/src/backend/replication/logical/origin.c
@@ -265,6 +265,52 @@ replorigin_by_name(const char *roname, bool missing_ok)
return roident;
}
+/*
+ * replorigin_create_with_id
+ *
+ * Create a replication origin with a specific ID and name, optionally
+ * restoring its remote_lsn. Used by pg_upgrade to preserve replication
+ * origin IDs across the upgrade.
+ *
+ * Caller must hold an exclusive lock on ReplicationOriginRelationId.
+ *
+ * Needs to be called in a transaction.
+ */
+void
+replorigin_create_with_id(ReplOriginId roident, const char *roname,
+ XLogRecPtr remote_lsn, Relation rel)
+{
+ Datum roname_d;
+ bool nulls[Natts_pg_replication_origin];
+ Datum values[Natts_pg_replication_origin];
+ HeapTuple tuple;
+
+ Assert(IsTransactionState());
+ Assert(CheckRelationLockedByMe(rel, ExclusiveLock, false));
+
+ roname_d = CStringGetTextDatum(roname);
+
+ memset(&nulls, 0, sizeof(nulls));
+ memset(&values, 0, sizeof(values));
+
+ values[Anum_pg_replication_origin_roident - 1] = ObjectIdGetDatum(roident);
+ values[Anum_pg_replication_origin_roname - 1] = roname_d;
+
+ tuple = heap_form_tuple(RelationGetDescr(rel), values, nulls);
+ CatalogTupleInsert(rel, tuple);
+ heap_freetuple(tuple);
+ CommandCounterIncrement();
+
+ if (remote_lsn != InvalidXLogRecPtr)
+ {
+ Assert(IsBinaryUpgrade);
+ replorigin_advance(roident, remote_lsn, InvalidXLogRecPtr,
+ false /* backward */,
+ false /* WAL log */);
+ }
+
+}
+
/*
* Create a replication origin.
*
@@ -273,13 +319,11 @@ replorigin_by_name(const char *roname, bool missing_ok)
ReplOriginId
replorigin_create(const char *roname)
{
- Oid roident;
- HeapTuple tuple = NULL;
- Relation rel;
- Datum roname_d;
- SnapshotData SnapshotDirty;
- SysScanDesc scan;
- ScanKeyData key;
+ Oid roident;
+ Relation rel;
+ SnapshotData SnapshotDirty;
+ SysScanDesc scan;
+ ScanKeyData key;
/*
* To avoid needing a TOAST table for pg_replication_origin, we limit
@@ -293,8 +337,6 @@ replorigin_create(const char *roname)
errdetail("Replication origin names must be no longer than %d bytes.",
MAX_RONAME_LEN)));
- roname_d = CStringGetTextDatum(roname);
-
Assert(IsTransactionState());
/*
@@ -321,17 +363,15 @@ replorigin_create(const char *roname)
* snapshot. To make that safe, it needs to not have a TOAST table, since
* TOASTed data cannot be fetched without a snapshot. As of this writing,
* its only varlena column is roname, which we limit to 512 bytes to avoid
- * needing out-of-line storage. If you add a TOAST table to this catalog,
- * be sure to set up a snapshot everywhere it might be needed. For more
+ * needing out-of-line storage. If you add a TOAST table to this catalog,
+ * be sure to set up a snapshot everywhere it might be needed. For more
* information, see https://postgr.es/m/ZvMSUPOqUU-VNADN%40nathan.
*/
Assert(!OidIsValid(rel->rd_rel->reltoastrelid));
for (roident = InvalidOid + 1; roident < PG_UINT16_MAX; roident++)
{
- bool nulls[Natts_pg_replication_origin];
- Datum values[Natts_pg_replication_origin];
- bool collides;
+ bool collides;
CHECK_FOR_INTERRUPTS();
@@ -341,41 +381,25 @@ replorigin_create(const char *roname)
ObjectIdGetDatum(roident));
scan = systable_beginscan(rel, ReplicationOriginIdentIndex,
- true /* indexOK */ ,
+ true /* indexOK */,
&SnapshotDirty,
1, &key);
-
collides = HeapTupleIsValid(systable_getnext(scan));
-
systable_endscan(scan);
if (!collides)
- {
- /*
- * Ok, found an unused roident, insert the new row and do a CCI,
- * so our callers can look it up if they want to.
- */
- memset(&nulls, 0, sizeof(nulls));
-
- values[Anum_pg_replication_origin_roident - 1] = ObjectIdGetDatum(roident);
- values[Anum_pg_replication_origin_roname - 1] = roname_d;
-
- tuple = heap_form_tuple(RelationGetDescr(rel), values, nulls);
- CatalogTupleInsert(rel, tuple);
- CommandCounterIncrement();
break;
- }
}
- /* now release lock again, */
- table_close(rel, ExclusiveLock);
-
- if (tuple == NULL)
+ if (roident >= PG_UINT16_MAX)
ereport(ERROR,
(errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
errmsg("could not find free replication origin ID")));
- heap_freetuple(tuple);
+ replorigin_create_with_id(roident, roname, InvalidXLogRecPtr, rel);
+
+ table_close(rel, ExclusiveLock);
+
return roident;
}
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index 59c3e7f0146..628b077ab69 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -28,6 +28,7 @@
#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
+#include "utils/syscache.h"
#include "utils/pg_lsn.h"
@@ -377,71 +378,79 @@ binary_upgrade_add_sub_rel_state(PG_FUNCTION_ARGS)
}
/*
- * binary_upgrade_replorigin_advance
+ * binary_upgrade_create_conflict_detection_slot
*
- * Update the remote_lsn for the subscriber's replication origin.
+ * Create a replication slot to retain information necessary for conflict
+ * detection such as dead tuples, commit timestamps, and origins.
*/
Datum
-binary_upgrade_replorigin_advance(PG_FUNCTION_ARGS)
+binary_upgrade_create_conflict_detection_slot(PG_FUNCTION_ARGS)
{
- Relation rel;
- Oid subid;
- char *subname;
- char originname[NAMEDATALEN];
- ReplOriginId node;
- XLogRecPtr remote_commit;
-
CHECK_IS_BINARY_UPGRADE;
- /*
- * We must ensure a non-NULL subscription name before dereferencing the
- * arguments.
- */
- if (PG_ARGISNULL(0))
- elog(ERROR, "null argument to binary_upgrade_replorigin_advance is not allowed");
-
- subname = text_to_cstring(PG_GETARG_TEXT_PP(0));
- remote_commit = PG_ARGISNULL(1) ? InvalidXLogRecPtr : PG_GETARG_LSN(1);
-
- rel = table_open(SubscriptionRelationId, RowExclusiveLock);
- subid = get_subscription_oid(subname, false);
-
- ReplicationOriginNameForLogicalRep(subid, InvalidOid, originname, sizeof(originname));
-
- /* Lock to prevent the replication origin from vanishing */
- LockRelationOid(ReplicationOriginRelationId, RowExclusiveLock);
- node = replorigin_by_name(originname, false);
-
- /*
- * The server will be stopped after setting up the objects in the new
- * cluster and the origins will be flushed during the shutdown checkpoint.
- * This will ensure that the latest LSN values for origin will be
- * available after the upgrade.
- */
- replorigin_advance(node, remote_commit, InvalidXLogRecPtr,
- false /* backward */ ,
- false /* WAL log */ );
+ CreateConflictDetectionSlot();
- UnlockRelationOid(ReplicationOriginRelationId, RowExclusiveLock);
- table_close(rel, RowExclusiveLock);
+ ReplicationSlotRelease();
PG_RETURN_VOID();
}
/*
- * binary_upgrade_create_conflict_detection_slot
+ * binary_upgrade_create_replication_origin
*
- * Create a replication slot to retain information necessary for conflict
- * detection such as dead tuples, commit timestamps, and origins.
+ * Create a replication origin with a specific OID and name, optionally
+ * restoring its remote_lsn. Used by pg_upgrade to preserve replication
+ * origin OIDs across the upgrade.
*/
Datum
-binary_upgrade_create_conflict_detection_slot(PG_FUNCTION_ARGS)
+binary_upgrade_create_replication_origin(PG_FUNCTION_ARGS)
{
+ Oid node_oid;
+ ReplOriginId node;
+ Relation rel;
+ char *originname;
+ XLogRecPtr remote_lsn = InvalidXLogRecPtr;
+
CHECK_IS_BINARY_UPGRADE;
- CreateConflictDetectionSlot();
+ if (PG_ARGISNULL(0) || PG_ARGISNULL(1))
+ elog(ERROR,
+ "null argument to binary_upgrade_create_replication_origin is not allowed");
- ReplicationSlotRelease();
+ node_oid = PG_GETARG_OID(0);
+
+ if (node_oid == InvalidOid || node_oid >= DoNotReplicateId)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("replication origin ID %u is out of range", node_oid));
+
+ node = (ReplOriginId) node_oid;
+
+ originname = text_to_cstring(PG_GETARG_TEXT_PP(1));
+
+ /*
+ * To avoid needing a TOAST table for pg_replication_origin, we limit
+ * replication origin names to 512 bytes. This should be more than enough
+ * for all practical use.
+ */
+ if (strlen(originname) > MAX_RONAME_LEN)
+ ereport(ERROR,
+ errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
+ errmsg("replication origin name is too long"),
+ errdetail("Replication origin names must be no longer than %d bytes.",
+ MAX_RONAME_LEN));
+
+ if (!PG_ARGISNULL(2))
+ remote_lsn = PG_GETARG_LSN(2);
+
+ Assert(IsTransactionState());
+
+ /* Acquire an exclusive lock before inserting the new origin. */
+ rel = table_open(ReplicationOriginRelationId, ExclusiveLock);
+
+ replorigin_create_with_id(node, originname, remote_lsn, rel);
+
+ table_close(rel, ExclusiveLock);
PG_RETURN_VOID();
}
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index ea22b0604c1..e157214b926 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -5179,7 +5179,6 @@ getSubscriptions(Archive *fout)
int i_subwalrcvtimeout;
int i_subpublications;
int i_suborigin;
- int i_suboriginremotelsn;
int i_subenabled;
int i_subfailover;
int i_subretaindeadtuples;
@@ -5248,11 +5247,9 @@ getSubscriptions(Archive *fout)
LOGICALREP_ORIGIN_ANY);
if (dopt->binary_upgrade && fout->remoteVersion >= 170000)
- appendPQExpBufferStr(query, " o.remote_lsn AS suboriginremotelsn,\n"
- " s.subenabled,\n");
+ appendPQExpBufferStr(query, " s.subenabled,\n");
else
- appendPQExpBufferStr(query, " NULL AS suboriginremotelsn,\n"
- " false AS subenabled,\n");
+ appendPQExpBufferStr(query, " false AS subenabled,\n");
if (fout->remoteVersion >= 170000)
appendPQExpBufferStr(query,
@@ -5332,7 +5329,6 @@ getSubscriptions(Archive *fout)
i_subwalrcvtimeout = PQfnumber(res, "subwalrcvtimeout");
i_subpublications = PQfnumber(res, "subpublications");
i_suborigin = PQfnumber(res, "suborigin");
- i_suboriginremotelsn = PQfnumber(res, "suboriginremotelsn");
subinfo = pg_malloc_array(SubscriptionInfo, ntups);
@@ -5385,11 +5381,6 @@ getSubscriptions(Archive *fout)
subinfo[i].subpublications =
pg_strdup(PQgetvalue(res, i, i_subpublications));
subinfo[i].suborigin = pg_strdup(PQgetvalue(res, i, i_suborigin));
- if (PQgetisnull(res, i, i_suboriginremotelsn))
- subinfo[i].suboriginremotelsn = NULL;
- else
- subinfo[i].suboriginremotelsn =
- pg_strdup(PQgetvalue(res, i, i_suboriginremotelsn));
/* Decide whether we want to dump it */
selectDumpableObject(&(subinfo[i].dobj), fout);
@@ -5668,37 +5659,15 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
* In binary-upgrade mode, we allow the replication to continue after the
* upgrade.
*/
- if (dopt->binary_upgrade && fout->remoteVersion >= 170000)
+ if (dopt->binary_upgrade && subinfo->subenabled && fout->remoteVersion >= 170000)
{
- if (subinfo->suboriginremotelsn)
- {
- /*
- * Preserve the remote_lsn for the subscriber's replication
- * origin. This value is required to start the replication from
- * the position before the upgrade. This value will be stale if
- * the publisher gets upgraded before the subscriber node.
- * However, this shouldn't be a problem as the upgrade of the
- * publisher ensures that all the transactions were replicated
- * before upgrading it.
- */
- appendPQExpBufferStr(query,
- "\n-- For binary upgrade, must preserve the remote_lsn for the subscriber's replication origin.\n");
- appendPQExpBufferStr(query,
- "SELECT pg_catalog.binary_upgrade_replorigin_advance(");
- appendStringLiteralAH(query, subinfo->dobj.name, fout);
- appendPQExpBuffer(query, ", '%s');\n", subinfo->suboriginremotelsn);
- }
-
- if (subinfo->subenabled)
- {
- /*
- * Enable the subscription to allow the replication to continue
- * after the upgrade.
- */
- appendPQExpBufferStr(query,
- "\n-- For binary upgrade, must preserve the subscriber's running state.\n");
- appendPQExpBuffer(query, "ALTER SUBSCRIPTION %s ENABLE;\n", qsubname);
- }
+ /*
+ * Enable the subscription to allow the replication to continue
+ * after the upgrade.
+ */
+ appendPQExpBufferStr(query,
+ "\n-- For binary upgrade, must preserve the subscriber's running state.\n");
+ appendPQExpBuffer(query, "ALTER SUBSCRIPTION %s ENABLE;\n", qsubname);
}
if (subinfo->dobj.dump & DUMP_COMPONENT_DEFINITION)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 5a6726d8b12..8f1252b854e 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -729,7 +729,6 @@ typedef struct _SubscriptionInfo
char *subwalrcvtimeout;
char *subpublications;
char *suborigin;
- char *suboriginremotelsn;
} SubscriptionInfo;
/*
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index b9653f0aefe..485140b9332 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -18,6 +18,7 @@
#include <time.h>
#include <unistd.h>
+#include "access/xlogdefs.h"
#include "catalog/pg_authid_d.h"
#include "common/connect.h"
#include "common/file_perm.h"
@@ -67,6 +68,7 @@ static void dropDBs(PGconn *conn);
static void dumpUserConfig(PGconn *conn, const char *username);
static void dumpDatabases(PGconn *conn);
static void dumpTimestamp(const char *msg);
+static void dumpReplicationOrigins(PGconn *conn);
static int runPgDump(const char *dbname, const char *create_opts);
static void buildShSecLabels(PGconn *conn,
const char *catalog_name, Oid objectId,
@@ -663,6 +665,14 @@ main(int argc, char *argv[])
/* Dump role GUC privileges */
if (server_version >= 150000 && !skip_acls)
dumpRoleGUCPrivs(conn);
+
+ }
+
+ /* Dump replication origins */
+ if (!tablespaces_only && !roles_only && binary_upgrade)
+ {
+ if (server_version >= 90500)
+ dumpReplicationOrigins(conn);
}
/* Dump tablespaces */
@@ -1832,6 +1842,64 @@ dumpTimestamp(const char *msg)
fprintf(OPF, "-- %s %s\n\n", msg, buf);
}
+static void
+dumpReplicationOrigins(PGconn *conn)
+{
+ PQExpBuffer buf = createPQExpBuffer();
+ PGresult *res;
+ int i_roident;
+ int i_roname;
+ int i_remotelsn;
+
+ /* Get replication origins from catalogs */
+ appendPQExpBufferStr(buf,
+ "SELECT o.*, os.remote_lsn "
+ "FROM pg_catalog.pg_replication_origin o "
+ "LEFT OUTER JOIN pg_catalog.pg_replication_origin_status os ON o.roident = os.local_id ");
+
+ res = executeQuery(conn, buf->data);
+
+ i_roident = PQfnumber(res, "roident");
+ i_roname = PQfnumber(res, "roname");
+ i_remotelsn = PQfnumber(res, "remote_lsn");
+
+ if (PQntuples(res) > 0)
+ fprintf(OPF, "--\n-- Replication Origins \n--\n\n");
+
+ for (int i = 0; i < PQntuples(res); i++)
+ {
+ ReplOriginId roident;
+ const char *roname;
+
+ roident = atooid(PQgetvalue(res, i, i_roident));
+ roname = PQgetvalue(res, i, i_roname);
+
+ resetPQExpBuffer(buf);
+
+ appendPQExpBufferStr(buf, "\n-- For binary upgrade, must preserve replication origin roident and remote_lsn\n");
+ appendPQExpBuffer(buf,
+ "SELECT pg_catalog.binary_upgrade_create_replication_origin("
+ "'%u'::pg_catalog.oid, ", roident);
+ appendStringLiteralConn(buf, roname, conn);
+ appendPQExpBufferStr(buf, "::pg_catalog.text");
+
+ if (!PQgetisnull(res, i, i_remotelsn))
+ {
+ appendPQExpBufferStr(buf, ", ");
+ appendStringLiteralConn(buf, PQgetvalue(res, i, i_remotelsn), conn);
+ appendPQExpBufferStr(buf, "::pg_catalog.pg_lsn");
+ }
+ else
+ appendPQExpBufferStr(buf, ", NULL");
+
+ appendPQExpBufferStr(buf, ");\n");
+ fprintf(OPF, "%s", buf->data);
+ }
+
+ PQclear(res);
+ destroyPQExpBuffer(buf);
+}
+
/*
* read_dumpall_filters - retrieve database identifier patterns from file
*
diff --git a/src/bin/pg_upgrade/check.c b/src/bin/pg_upgrade/check.c
index 7556fb3f22a..ea8451d6071 100644
--- a/src/bin/pg_upgrade/check.c
+++ b/src/bin/pg_upgrade/check.c
@@ -33,7 +33,7 @@ static void check_for_new_tablespace_dir(void);
static void check_for_user_defined_encoding_conversions(ClusterInfo *cluster);
static void check_for_unicode_update(ClusterInfo *cluster);
static void check_new_cluster_replication_slots(void);
-static void check_new_cluster_subscription_configuration(void);
+static void check_new_cluster_replication_origins(void);
static void check_old_cluster_for_valid_slots(void);
static void check_old_cluster_subscription_state(void);
static void check_old_cluster_global_names(ClusterInfo *cluster);
@@ -651,6 +651,10 @@ check_and_dump_old_cluster(void)
check_old_cluster_subscription_state();
}
+ /* Get replication origin information for PG 9.5 and above */
+ if (GET_MAJOR_VERSION(old_cluster.major_version) >= 905)
+ get_replication_origin_info(&old_cluster);
+
check_for_data_types_usage(&old_cluster);
/*
@@ -797,7 +801,7 @@ check_new_cluster(void)
check_new_cluster_replication_slots();
- check_new_cluster_subscription_configuration();
+ check_new_cluster_replication_origins();
}
@@ -2303,31 +2307,45 @@ check_new_cluster_replication_slots(void)
}
/*
- * check_new_cluster_subscription_configuration()
+ * check_new_cluster_replication_origins()
+ *
+ * Verify that the new cluster has no replication origins. During upgrade,
+ * pg_upgrade restores replication origins from the old cluster with their
+ * original OIDs. If the new cluster already contains origins, those OIDs
+ * may collide, causing the upgrade to fail mid-way.
*
- * Verify that the max_active_replication_origins configuration specified is
- * enough for creating the subscriptions. This is required to create the
- * replication origin for each subscription.
+ * Also verify that the max_active_replication_origins configuration is
+ * enough for creating all the replication origins.
*/
static void
-check_new_cluster_subscription_configuration(void)
+check_new_cluster_replication_origins(void)
{
- PGresult *res;
PGconn *conn;
+ PGresult *res;
+ int norigins;
int max_active_replication_origins;
- /* Subscriptions and their dependencies can be migrated since PG17. */
- if (GET_MAJOR_VERSION(old_cluster.major_version) < 1700)
+ /* Quick return if there are no replication origins to migrate. */
+ if (old_cluster.nrepl_origins == 0)
return;
- /* Quick return if there are no subscriptions to be migrated. */
- if (old_cluster.nsubs == 0)
- return;
-
- prep_status("Checking new cluster configuration for subscriptions");
+ prep_status("Checking replication origins in new cluster");
conn = connectToServer(&new_cluster, "template1");
+ res = executeQueryOrDie(conn,
+ "SELECT count(*) "
+ "FROM pg_catalog.pg_replication_origin");
+
+ if (PQntuples(res) != 1)
+ pg_fatal("could not count the number of replication origins");
+
+ norigins = atoi(PQgetvalue(res, 0, 0));
+ PQclear(res);
+
+ if (norigins > 0)
+ pg_fatal("expected 0 replication origins but found %d", norigins);
+
res = executeQueryOrDie(conn, "SELECT setting FROM pg_settings "
"WHERE name = 'max_active_replication_origins';");
@@ -2335,12 +2353,13 @@ check_new_cluster_subscription_configuration(void)
pg_fatal("could not determine parameter settings on new cluster");
max_active_replication_origins = atoi(PQgetvalue(res, 0, 0));
- if (old_cluster.nsubs > max_active_replication_origins)
+ PQclear(res);
+
+ if (old_cluster.nrepl_origins > max_active_replication_origins)
pg_fatal("\"max_active_replication_origins\" (%d) must be greater than or equal to the number of "
- "subscriptions (%d) in the old cluster",
- max_active_replication_origins, old_cluster.nsubs);
+ "replication origins (%d) in the old cluster",
+ max_active_replication_origins, old_cluster.nrepl_origins);
- PQclear(res);
PQfinish(conn);
check_ok();
diff --git a/src/bin/pg_upgrade/info.c b/src/bin/pg_upgrade/info.c
index 37fff93892f..ec5dd5b7e96 100644
--- a/src/bin/pg_upgrade/info.c
+++ b/src/bin/pg_upgrade/info.c
@@ -832,6 +832,30 @@ count_old_cluster_logical_slots(void)
return slot_count;
}
+/*
+ * get_replication_origin_info()
+ *
+ * Gets the information of replication origins in the cluster.
+ */
+void
+get_replication_origin_info(ClusterInfo *cluster)
+{
+ PGconn *conn;
+ PGresult *res;
+
+ conn = connectToServer(cluster, "template1");
+ res = executeQueryOrDie(conn, "SELECT count(*) AS norigins "
+ "FROM pg_catalog.pg_replication_origin");
+
+ if (PQntuples(res) != 1)
+ pg_fatal("could not get the number of replication origins");
+
+ cluster->nrepl_origins = atoi(PQgetvalue(res, 0, 0));
+ PQclear(res);
+
+ PQfinish(conn);
+}
+
/*
* get_subscription_info()
*
@@ -842,23 +866,20 @@ get_subscription_info(ClusterInfo *cluster)
{
PGconn *conn;
PGresult *res;
- int i_nsub;
int i_retain_dead_tuples;
conn = connectToServer(cluster, "template1");
if (GET_MAJOR_VERSION(cluster->major_version) >= 1900)
- res = executeQueryOrDie(conn, "SELECT count(*) AS nsub,"
- "COUNT(CASE WHEN subretaindeadtuples THEN 1 END) > 0 AS retain_dead_tuples "
+ res = executeQueryOrDie(conn,
+ "SELECT COUNT(CASE WHEN subretaindeadtuples THEN 1 END) > 0 AS retain_dead_tuples "
"FROM pg_catalog.pg_subscription");
else
- res = executeQueryOrDie(conn, "SELECT count(*) AS nsub,"
- "'f' AS retain_dead_tuples "
+ res = executeQueryOrDie(conn,
+ "SELECT 'f' AS retain_dead_tuples "
"FROM pg_catalog.pg_subscription");
- i_nsub = PQfnumber(res, "nsub");
i_retain_dead_tuples = PQfnumber(res, "retain_dead_tuples");
- cluster->nsubs = atoi(PQgetvalue(res, 0, i_nsub));
cluster->sub_retain_dead_tuples = (strcmp(PQgetvalue(res, 0, i_retain_dead_tuples), "t") == 0);
PQclear(res);
diff --git a/src/bin/pg_upgrade/pg_upgrade.h b/src/bin/pg_upgrade/pg_upgrade.h
index ccd1ac0d013..e79765bf4fe 100644
--- a/src/bin/pg_upgrade/pg_upgrade.h
+++ b/src/bin/pg_upgrade/pg_upgrade.h
@@ -310,7 +310,7 @@ typedef struct
char **tablespaces; /* tablespace directories */
int num_tablespaces;
const char *tablespace_suffix; /* directory specification */
- int nsubs; /* number of subscriptions */
+ int nrepl_origins; /* number of replication origins */
bool sub_retain_dead_tuples; /* whether a subscription enables
* retain_dead_tuples. */
} ClusterInfo;
@@ -451,6 +451,7 @@ FileNameMap *gen_db_file_maps(DbInfo *old_db,
void get_db_rel_and_slot_infos(ClusterInfo *cluster);
int count_old_cluster_logical_slots(void);
void get_subscription_info(ClusterInfo *cluster);
+void get_replication_origin_info(ClusterInfo *cluster);
/* option.c */
diff --git a/src/bin/pg_upgrade/t/004_subscription.pl b/src/bin/pg_upgrade/t/004_subscription.pl
index 9286c1dc85e..e09450fc335 100644
--- a/src/bin/pg_upgrade/t/004_subscription.pl
+++ b/src/bin/pg_upgrade/t/004_subscription.pl
@@ -42,7 +42,7 @@ my $connstr = $publisher->connstr . ' dbname=postgres';
# ------------------------------------------------------
# Check that pg_upgrade fails when max_active_replication_origins configured
-# in the new cluster is less than the number of subscriptions in the old
+# in the new cluster is less than the number of replication origins in the old
# cluster.
# ------------------------------------------------------
# It is sufficient to use disabled subscription to test upgrade failure.
@@ -74,7 +74,7 @@ command_checks_all(
],
1,
[
- qr/"max_active_replication_origins" \(0\) must be greater than or equal to the number of subscriptions \(1\) in the old cluster/
+ qr/"max_active_replication_origins" \(0\) must be greater than or equal to the number of replication origins \(1\) in the old cluster/
],
[qr//],
'run of pg_upgrade where the new cluster has insufficient max_active_replication_origins'
@@ -301,8 +301,38 @@ is($result, qq(t), "Check that the table is in init state");
# Get the replication origin's remote_lsn of the old subscriber
my $remote_lsn = $old_sub->safe_psql('postgres',
- "SELECT remote_lsn FROM pg_replication_origin_status os, pg_subscription s WHERE os.external_id = 'pg_' || s.oid AND s.subname = 'regress_sub4'"
+ "SELECT os.remote_lsn
+ FROM pg_replication_origin_status os
+ JOIN pg_replication_origin o ON o.roident = os.local_id
+ JOIN pg_subscription s ON o.roname = 'pg_' || s.oid::text
+ WHERE s.subname = 'regress_sub4'"
);
+
+# Get the replication origin ids (roident) for all subscriptions, keyed by
+# subscription name (which is stable across upgrade, unlike suboid). These
+# must be preserved after upgrade. A mismatch would cause spurious
+# update_origin_differs conflicts.
+my %pre_upgrade_roident;
+my $roident_rows = $old_sub->safe_psql('postgres',
+ "SELECT s.subname, o.roident
+ FROM pg_subscription s
+ JOIN pg_replication_origin o ON o.roname = 'pg_' || s.oid::text
+ ORDER BY s.subname"
+);
+for my $row (split /\n/, $roident_rows)
+{
+ my ($subname, $roident) = split /\|/, $row;
+ $pre_upgrade_roident{$subname} = $roident;
+}
+
+# Create a user created replication origin, which should also be preserved after upgrade.
+my $user_origin_name = 'regress_user_origin';
+$old_sub->safe_psql('postgres',
+ "SELECT pg_replication_origin_create('$user_origin_name')");
+$pre_upgrade_roident{$user_origin_name} = $old_sub->safe_psql('postgres',
+ "SELECT roident FROM pg_replication_origin WHERE roname = '$user_origin_name'"
+);
+
# Have the subscription in disabled state before upgrade
$old_sub->safe_psql('postgres', "ALTER SUBSCRIPTION regress_sub5 DISABLE");
@@ -378,6 +408,39 @@ regress_sub5|f|f|f),
"check that the subscription's running status, failover, and retain_dead_tuples are preserved"
);
+# Verify that the subscription related replication origins are preserved after upgrade.
+my $post_roident_rows = $new_sub->safe_psql('postgres',
+ "SELECT s.subname, o.roident
+ FROM pg_subscription s
+ JOIN pg_replication_origin o ON o.roname = 'pg_' || s.oid::text
+ ORDER BY s.subname"
+);
+
+my %post_upgrade_roident;
+for my $row (split /\n/, $post_roident_rows)
+{
+ my ($subname, $roident) = split /\|/, $row;
+ $post_upgrade_roident{$subname} = $roident;
+}
+
+# Iterate over the union of pre- and post-upgrade keys so a missing or
+# unexpected entry on either side is caught.
+my %all_subnames = (%pre_upgrade_roident, %post_upgrade_roident);
+for my $subname (sort keys %all_subnames)
+{
+ next if $subname eq $user_origin_name;
+ is($post_upgrade_roident{$subname}, $pre_upgrade_roident{$subname},
+ "roident preserved for subscription '$subname' after upgrade");
+}
+
+# Verify that user created replication origins are preserved after upgrade.
+my $post_user_roident = $new_sub->safe_psql('postgres',
+ "SELECT roident FROM pg_replication_origin WHERE roname = '$user_origin_name'"
+);
+is($post_user_roident, $pre_upgrade_roident{$user_origin_name},
+ "roident preserved for user-created origin '$user_origin_name' after upgrade"
+);
+
# Subscription relations should be preserved
$result = $new_sub->safe_psql('postgres',
"SELECT srrelid, srsubstate FROM pg_subscription_rel ORDER BY srrelid");
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 231ce2ca91e..4d930641522 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -11959,10 +11959,6 @@
provolatile => 'v', proparallel => 'u', prorettype => 'void',
proargtypes => 'text oid char pg_lsn',
prosrc => 'binary_upgrade_add_sub_rel_state' },
-{ oid => '6320', descr => 'for use by pg_upgrade (remote_lsn for origin)',
- proname => 'binary_upgrade_replorigin_advance', proisstrict => 'f',
- provolatile => 'v', proparallel => 'u', prorettype => 'void',
- proargtypes => 'text pg_lsn', prosrc => 'binary_upgrade_replorigin_advance' },
{ oid => '6505', descr => 'for use by pg_upgrade (conflict detection slot)',
proname => 'binary_upgrade_create_conflict_detection_slot',
proisstrict => 'f', provolatile => 'v', proparallel => 'u',
@@ -11972,6 +11968,10 @@
proname => 'binary_upgrade_set_next_pg_subscription_oid', provolatile => 'v',
proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
prosrc => 'binary_upgrade_set_next_pg_subscription_oid' },
+{ oid => '9161', descr => 'for use by pg_upgrade (replication origin)',
+ proname => 'binary_upgrade_create_replication_origin', proisstrict => 'f',
+ provolatile => 'v', proparallel => 'u', prorettype => 'void',
+ proargtypes => 'oid text pg_lsn', prosrc => 'binary_upgrade_create_replication_origin' },
# conversion functions
{ oid => '4310', descr => 'internal conversion function for KOI8R to WIN1251',
diff --git a/src/include/replication/origin.h b/src/include/replication/origin.h
index a69faf6eaaf..11ee630fb28 100644
--- a/src/include/replication/origin.h
+++ b/src/include/replication/origin.h
@@ -14,6 +14,7 @@
#include "access/xlogdefs.h"
#include "access/xlogreader.h"
#include "catalog/pg_replication_origin.h"
+#include "utils/relcache.h"
typedef struct xl_replorigin_set
{
@@ -55,6 +56,8 @@ extern PGDLLIMPORT int max_active_replication_origins;
/* API for querying & manipulating replication origins */
extern ReplOriginId replorigin_by_name(const char *roname, bool missing_ok);
extern ReplOriginId replorigin_create(const char *roname);
+extern void replorigin_create_with_id(ReplOriginId roident, const char *roname,
+ XLogRecPtr remote_lsn, Relation rel);
extern void replorigin_drop_by_name(const char *name, bool missing_ok, bool nowait);
extern bool replorigin_by_oid(ReplOriginId roident, bool missing_ok,
char **roname);
--
2.47.3
^ permalink raw reply [nested|flat] 97+ messages in thread
* Re: [PATCH] Preserve replication origin OIDs in pg_upgrade
@ 2026-07-05 16:09 Rui Zhao <[email protected]>
parent: Ajin Cherian <[email protected]>
2 siblings, 1 reply; 97+ messages in thread
From: Rui Zhao @ 2026-07-05 16:09 UTC (permalink / raw)
To: Ajin Cherian <[email protected]>; +Cc: [email protected]; Zsolt Parragi <[email protected]>
Hi Ajin,
Thanks for v10. On the version gating I asked about earlier -- I think it's
now moot, and the gates can come back out.
Since v10, 14d8418083 ("Remove pg_upgrade support for upgrading from pre-v10
servers") raised pg_upgrade's minimum source version to v10.
pg_replication_origin has existed since 9.5, so every cluster pg_upgrade now
accepts already has it, and the original unconditional
get_replication_origin_info() is safe again. Both gates that went into v10 are
now always true:
- check.c: if (GET_MAJOR_VERSION(old_cluster.major_version) >= 905)
- pg_dumpall.c: if (server_version >= 90500) -- only reached under
binary_upgrade, i.e. against the v10+ old cluster
14d8418083 was itself removing exactly this kind of pre-v10 version check
across pg_upgrade, so dropping these fits the same cleanup.
v10 doesn't apply on current master anyway (0002 fails on pg_proc.dat), so
could you post a rebase? That'd let it be built and tested against HEAD, and
it's a natural point to drop the two gates and go back to the unconditional
calls.
Minor, in that same hunk: the dumpReplicationOrigins(conn) call isn't indented
under its if.
Otherwise v10 reads well; the other reviewers' comments all look addressed.
Zsolt, a small request -- could you use reply-all on the thread? A couple of
times the CC list got narrowed, and when the reply chain continued from there
I dropped off it and missed the update. Keeping everyone CC'd helps the rest
of us stay in the loop.
Thanks,
Rui
^ permalink raw reply [nested|flat] 97+ messages in thread
* Re: [PATCH] Preserve replication origin OIDs in pg_upgrade
@ 2026-07-06 03:31 shveta malik <[email protected]>
parent: Rui Zhao <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: shveta malik @ 2026-07-06 03:31 UTC (permalink / raw)
To: Rui Zhao <[email protected]>; +Cc: Ajin Cherian <[email protected]>; [email protected]; Zsolt Parragi <[email protected]>; shveta malik <[email protected]>
On Sun, Jul 5, 2026 at 9:40 PM Rui Zhao <[email protected]> wrote:
>
> Hi Ajin,
>
> Thanks for v10. On the version gating I asked about earlier -- I think it's
> now moot, and the gates can come back out.
>
> Since v10, 14d8418083 ("Remove pg_upgrade support for upgrading from pre-v10
> servers") raised pg_upgrade's minimum source version to v10.
> pg_replication_origin has existed since 9.5, so every cluster pg_upgrade now
> accepts already has it, and the original unconditional
> get_replication_origin_info() is safe again. Both gates that went into v10 are
> now always true:
>
> - check.c: if (GET_MAJOR_VERSION(old_cluster.major_version) >= 905)
> - pg_dumpall.c: if (server_version >= 90500) -- only reached under
> binary_upgrade, i.e. against the v10+ old cluster
>
> 14d8418083 was itself removing exactly this kind of pre-v10 version check
> across pg_upgrade, so dropping these fits the same cleanup.
>
Thanks for pointing to the commit.
> v10 doesn't apply on current master anyway (0002 fails on pg_proc.dat), so
> could you post a rebase? That'd let it be built and tested against HEAD, and
> it's a natural point to drop the two gates and go back to the unconditional
> calls.
I can apply v10 on the latest HEAD (9d1188f298), can you please try it again?
thanks
Shveta
^ permalink raw reply [nested|flat] 97+ messages in thread
* Re: [PATCH] Preserve replication origin OIDs in pg_upgrade
@ 2026-07-06 05:12 shveta malik <[email protected]>
parent: Ajin Cherian <[email protected]>
2 siblings, 0 replies; 97+ messages in thread
From: shveta malik @ 2026-07-06 05:12 UTC (permalink / raw)
To: Ajin Cherian <[email protected]>; +Cc: Zsolt Parragi <[email protected]>; [email protected]; shveta malik <[email protected]>
On Wed, Jul 1, 2026 at 8:20 AM Ajin Cherian <[email protected]> wrote:
>
> All the above changes are updated in patch v10.
>
Thanks for the patch. I have not reviewed completely yet, but please
find comments so far:
1)
+ * In binary-upgrade mode, skip origin creation here. This is required to
+ * preserve the roident from the old cluster for this subscription's origin.
Shall we slightly tweak it to:
/*
* In binary-upgrade mode, skip origin creation here. It will be
* created separately so that the origin ID (roident) from the
* old cluster can be preserved for this subscription.
*/
dumpReplicationOrigins:
2)
+ appendPQExpBufferStr(buf,
+ "SELECT o.*, os.remote_lsn "
It will be better to be more specific here to make it more future proof:
SELECT o.roident, o.roname, os.remote_lsn
3)
+ if (PQntuples(res) > 0)
+ fprintf(OPF, "--\n-- Replication Origins \n--\n\n");
+
+ for (int i = 0; i < PQntuples(res); i++)
Instead of fetching it in loop, we can fetch it once and use it:
ntups = PQntuples(res)
4)
+ appendPQExpBufferStr(buf, "\n-- For binary upgrade, must preserve
replication origin roident and remote_lsn\n");
Seeing the pattern of other such comments in file, we can change
'origin' to be more specific 'pg_replication_origin'
5)
get_replication_origin_info:
+ res = executeQueryOrDie(conn, "SELECT count(*) AS norigins "
+ "FROM pg_catalog.pg_replication_origin");
I don't see alias 'norigins' being sued anywhere. We can get rid of it
if not needed.
6)
get_subscription_info: I think a bug is introduced with the new change.
Earlier query was below which will always return a row even if there
is no subscription entry:
postgres=# SELECT count(*) AS nsub,
'f' AS retain_dead_tuples
FROM pg_catalog.pg_subscription;
nsub | retain_dead_tuples
------+--------------------
0 | f
Now, this will not even return a result if there are no subscription entries:
postgres=# SELECT 'f' AS retain_dead_tuples
FROM pg_catalog.pg_subscription;
retain_dead_tuples
--------------------
(0 rows)
So this line of code will access a non existing result:
i_retain_dead_tuples = PQfnumber(res, "retain_dead_tuples");
Should we simply do:
res = executeQueryOrDie(conn,
"SELECT false AS retain_dead_tuples");
(false/'f' anything is fine)
instead of:
res = executeQueryOrDie(conn,
"SELECT 'f' AS retain_dead_tuples "
"FROM pg_catalog.pg_subscription");
thanks
Shveta
^ permalink raw reply [nested|flat] 97+ messages in thread
* Re: [PATCH] Preserve replication origin OIDs in pg_upgrade
@ 2026-07-06 06:40 Shlok Kyal <[email protected]>
parent: Ajin Cherian <[email protected]>
2 siblings, 2 replies; 97+ messages in thread
From: Shlok Kyal @ 2026-07-06 06:40 UTC (permalink / raw)
To: Ajin Cherian <[email protected]>; +Cc: Zsolt Parragi <[email protected]>; [email protected]
> All the above changes are updated in patch v10.
>
I noticed that we can create user defined replication origin even if the
total count of replication origins exceeds the max_active_replication_origins
Example:
postgres=# show max_active_replication_origins;
max_active_replication_origins
--------------------------------
1
postgres=# select * from pg_replication_origin;
roident | roname
---------+----------
1 | pg_16387
postgres=# SELECT pg_replication_origin_create('origin1');
pg_replication_origin_create
------------------------------
2
postgres=# select * from pg_replication_origin;
roident | roname
---------+----------
1 | pg_16387
2 | origin1
postgres=# select * from pg_replication_origin_status;
local_id | external_id | remote_lsn | local_lsn
----------+-------------+------------+------------
1 | pg_16387 | 0/00000000 | 0/00000000
The documentation for max_active_replication_origins says:
```
Specifies how many replication origins (see
<xref linkend="replication-origins"/>) can be tracked simultaneously,
effectively limiting how many logical replication subscriptions can
be created on the server. Setting it to a lower value than the current
number of tracked replication origins (reflected in
<link linkend="view-pg-replication-origin-status">pg_replication_origin_status</link>)
will prevent the server from starting. It defaults to 10. This parameter
can only be set at server start.
```
Since the user-created replication origin above is not
actively tracked in pg_replication_origin_status, no error is reported.
But with this patch, during pg_upgrade we are checking the
'max_active_replication_origins' against 'old_cluster.nrepl_origins'
+ if (old_cluster.nrepl_origins > max_active_replication_origins)
pg_fatal("\"max_active_replication_origins\" (%d) must be greater
than or equal to the number of "
- "subscriptions (%d) in the old cluster",
- max_active_replication_origins, old_cluster.nsubs);
+ "replication origins (%d) in the old cluster",
+ max_active_replication_origins, old_cluster.nrepl_origins);
Since old_cluster.nrepl_origins counts all replication origins, including
user-created ones that may not be tracked, should this check instead compare
max_active_replication_origins with 'the number of rows in
pg_replication_origin_status in old cluster' (i.e., the number of
tracked replication origins)?
Thanks
Shlok Kyal
^ permalink raw reply [nested|flat] 97+ messages in thread
* Re: [PATCH] Preserve replication origin OIDs in pg_upgrade
@ 2026-07-06 13:10 Rui Zhao <[email protected]>
parent: Shlok Kyal <[email protected]>
1 sibling, 1 reply; 97+ messages in thread
From: Rui Zhao @ 2026-07-06 13:10 UTC (permalink / raw)
To: Ajin Cherian <[email protected]>; +Cc: Zsolt Parragi <[email protected]>; [email protected]; Shlok Kyal <[email protected]>; shveta malik <[email protected]>
Hi Ajin,
Shveta's right, sorry for the noise -- git am --3way applies v10 cleanly on
HEAD (9d1188f298) and it builds. "git apply --check" was tripping on the
shifted pg_proc.dat hunk on my end.
Confirmed Shveta's (6), and it's a crash, not just a bad read:
cluster->sub_retain_dead_tuples = (strcmp(PQgetvalue(res, 0,
i_retain_dead_tuples), "t") == 0);
With no subscriptions the < 1900 query returns zero rows, PQgetvalue(res, 0,
...) returns NULL for a nonexistent row, and the strcmp segfaults (reproduced
with a small libpq program). Upgrading a v17/v18 cluster that has no
subscriptions hits it. Shveta's "SELECT false" fix is right.
A few more comments from re-reading v10:
1) The doc hunk updates this sentence:
- Commit timestamps and origin data are not preserved during
the upgrade.
+ Commit timestamps are not preserved during the upgrade.
but the twin comment in pg_upgrade.c (~line 222) is left unchanged and needs
the same edit:
* upgrade. Additionally, commit timestamps and origin data are not
* preserved during the upgrade. So, even after creating the slot, the
2) dumpReplicationOrigins:
+ appendPQExpBufferStr(buf,
+ "SELECT o.*, os.remote_lsn "
+ "FROM pg_catalog.pg_replication_origin o "
+ "LEFT OUTER JOIN pg_catalog.pg_replication_origin_status os ON
o.roident = os.local_id ");
No ORDER BY, so the output order is whatever the heap returns; every other
query in pg_dumpall.c has one. ORDER BY o.roident would keep the dump stable.
3) Trivia: trailing space in the section header --
+ fprintf(OPF, "--\n-- Replication Origins \n--\n\n");
and a stray blank line at the end of replorigin_create_with_id():
+ false /* WAL log */);
+ }
+
+}
Also, per my earlier mail, the >= 905 / >= 90500 gates can come out now that
14d8418083 raised the pg_upgrade floor to v10.
Thanks,
Rui
^ permalink raw reply [nested|flat] 97+ messages in thread
* Re: [PATCH] Preserve replication origin OIDs in pg_upgrade
@ 2026-07-08 04:01 shveta malik <[email protected]>
parent: Shlok Kyal <[email protected]>
1 sibling, 1 reply; 97+ messages in thread
From: shveta malik @ 2026-07-08 04:01 UTC (permalink / raw)
To: Shlok Kyal <[email protected]>; +Cc: Ajin Cherian <[email protected]>; Zsolt Parragi <[email protected]>; [email protected]; shveta malik <[email protected]>
On Mon, Jul 6, 2026 at 12:10 PM Shlok Kyal <[email protected]> wrote:
>
> > All the above changes are updated in patch v10.
> >
> I noticed that we can create user defined replication origin even if the
> total count of replication origins exceeds the max_active_replication_origins
> Example:
> postgres=# show max_active_replication_origins;
> max_active_replication_origins
> --------------------------------
> 1
>
> postgres=# select * from pg_replication_origin;
> roident | roname
> ---------+----------
> 1 | pg_16387
>
> postgres=# SELECT pg_replication_origin_create('origin1');
> pg_replication_origin_create
> ------------------------------
> 2
>
> postgres=# select * from pg_replication_origin;
> roident | roname
> ---------+----------
> 1 | pg_16387
> 2 | origin1
>
> postgres=# select * from pg_replication_origin_status;
> local_id | external_id | remote_lsn | local_lsn
> ----------+-------------+------------+------------
> 1 | pg_16387 | 0/00000000 | 0/00000000
>
> The documentation for max_active_replication_origins says:
> ```
> Specifies how many replication origins (see
> <xref linkend="replication-origins"/>) can be tracked simultaneously,
> effectively limiting how many logical replication subscriptions can
> be created on the server. Setting it to a lower value than the current
> number of tracked replication origins (reflected in
> <link linkend="view-pg-replication-origin-status">pg_replication_origin_status</link>)
> will prevent the server from starting. It defaults to 10. This parameter
> can only be set at server start.
> ```
> Since the user-created replication origin above is not
> actively tracked in pg_replication_origin_status, no error is reported.
>
> But with this patch, during pg_upgrade we are checking the
> 'max_active_replication_origins' against 'old_cluster.nrepl_origins'
> + if (old_cluster.nrepl_origins > max_active_replication_origins)
> pg_fatal("\"max_active_replication_origins\" (%d) must be greater
> than or equal to the number of "
> - "subscriptions (%d) in the old cluster",
> - max_active_replication_origins, old_cluster.nsubs);
> + "replication origins (%d) in the old cluster",
> + max_active_replication_origins, old_cluster.nrepl_origins);
>
> Since old_cluster.nrepl_origins counts all replication origins, including
> user-created ones that may not be tracked, should this check instead compare
> max_active_replication_origins with 'the number of rows in
> pg_replication_origin_status in old cluster' (i.e., the number of
> tracked replication origins)?
>
Thanks Shlok for exploring this scenario.
I had a look at it. Please find my analysis.
I was under the assumption that subscription creation (which
internally creates a replication origin) would fail if
max_active_replication_origins was exceeded. However, that is not the
case. See the following experiment:
postgres=# show max_active_replication_origins;
max_active_replication_origins
--------------------------------
2
Al these commands succeed:
postgres=# create subscription sub1 ...
postgres=# create subscription sub2 ..
postgres=# SELECT pg_replication_origin_create('user_origin1');
postgres=# SELECT pg_replication_origin_create('user_origin2');
postgres=# SELECT pg_replication_origin_create('user_origin3');
postgres=# SELECT pg_replication_origin_create('user_origin4');
postgres=# create subscription sub3 ..
postgres=# create subscription sub4...
postgres=# select * from pg_replication_origin_status;
local_id | external_id | remote_lsn | local_lsn
----------+-------------+------------+------------
1 | pg_16384 | 0/00000000 | 0/00000000
2 | pg_16385 | 0/00000000 | 0/00000000
(2 rows)
postgres=# select * from pg_replication_origin;
roident | roname
---------+--------------
1 | pg_16384
2 | pg_16385
3 | user_origin1
4 | user_origin2
5 | user_origin3
6 | user_origin4
7 | pg_16387
8 | pg_16388
(8 rows)
As shown above, sub3 and sub4 are created successfully, but their
replication origins are not tracked because there are no free active
replication origin slots. The apply workers for these subscriptions
also fail to start:
LOG: logical replication apply worker for subscription "sub3" has started
ERROR: could not find free replication state slot for replication
origin with ID 7
HINT: Increase "max_active_replication_origins" and try again.
LOG: background worker "logical replication apply worker" (PID
277193) exited with exit code 1
This means that if we implement the check suggested by Shlok:
Although pg_upgrade would succeed, it could leave some subscriptions
on the new cluster in non-functional state. I agree that this
situation (non-functional subscriptions) is also possible on the old
cluster. However, this would also deviate from the previous
implementation, where check_new_cluster_subscription_configuration()
compared old_cluster.nsubs against 'max_active_replication_origins' on
the new cluster, rather than comparing the number of entries in
pg_replication_origin_status.
For this reason, my inclination is to revert to the previous
implementation, where we compare old_cluster.nsubs with
max_active_replication_origins. But I'd like to hear what others think
before we make this change.
thanks
Shveta
^ permalink raw reply [nested|flat] 97+ messages in thread
* Re: [PATCH] Preserve replication origin OIDs in pg_upgrade
@ 2026-07-08 04:07 shveta malik <[email protected]>
parent: Rui Zhao <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: shveta malik @ 2026-07-08 04:07 UTC (permalink / raw)
To: Rui Zhao <[email protected]>; +Cc: Ajin Cherian <[email protected]>; Zsolt Parragi <[email protected]>; [email protected]; Shlok Kyal <[email protected]>; shveta malik <[email protected]>
On Mon, Jul 6, 2026 at 6:40 PM Rui Zhao <[email protected]> wrote:
>
> Hi Ajin,
>
> Shveta's right, sorry for the noise -- git am --3way applies v10 cleanly on
> HEAD (9d1188f298) and it builds. "git apply --check" was tripping on the
> shifted pg_proc.dat hunk on my end.
>
> Confirmed Shveta's (6), and it's a crash, not just a bad read:
Thanks for verifying it.
> cluster->sub_retain_dead_tuples = (strcmp(PQgetvalue(res, 0,
> i_retain_dead_tuples), "t") == 0);
>
> With no subscriptions the < 1900 query returns zero rows, PQgetvalue(res, 0,
> ...) returns NULL for a nonexistent row, and the strcmp segfaults (reproduced
> with a small libpq program). Upgrading a v17/v18 cluster that has no
> subscriptions hits it. Shveta's "SELECT false" fix is right.
>
> A few more comments from re-reading v10:
>
> 1) The doc hunk updates this sentence:
>
> - Commit timestamps and origin data are not preserved during
> the upgrade.
> + Commit timestamps are not preserved during the upgrade.
>
> but the twin comment in pg_upgrade.c (~line 222) is left unchanged and needs
> the same edit:
>
> * upgrade. Additionally, commit timestamps and origin data are not
> * preserved during the upgrade. So, even after creating the slot, the
>
> 2) dumpReplicationOrigins:
>
> + appendPQExpBufferStr(buf,
> + "SELECT o.*, os.remote_lsn "
> + "FROM pg_catalog.pg_replication_origin o "
> + "LEFT OUTER JOIN pg_catalog.pg_replication_origin_status os ON
> o.roident = os.local_id ");
>
> No ORDER BY, so the output order is whatever the heap returns; every other
> query in pg_dumpall.c has one. ORDER BY o.roident would keep the dump stable.
+1
>
> 3) Trivia: trailing space in the section header --
>
> + fprintf(OPF, "--\n-- Replication Origins \n--\n\n");
>
> and a stray blank line at the end of replorigin_create_with_id():
>
> + false /* WAL log */);
> + }
> +
> +}
>
> Also, per my earlier mail, the >= 905 / >= 90500 gates can come out now that
> 14d8418083 raised the pg_upgrade floor to v10.
>
> Thanks,
> Rui
^ permalink raw reply [nested|flat] 97+ messages in thread
* RE: [PATCH] Preserve replication origin OIDs in pg_upgrade
@ 2026-07-08 06:21 Hayato Kuroda (Fujitsu) <[email protected]>
parent: shveta malik <[email protected]>
0 siblings, 2 replies; 97+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2026-07-08 06:21 UTC (permalink / raw)
To: 'shveta malik' <[email protected]>; Shlok Kyal <[email protected]>; +Cc: Ajin Cherian <[email protected]>; Zsolt Parragi <[email protected]>; [email protected] <[email protected]>
Dear Shveta,
> For this reason, my inclination is to revert to the previous
> implementation, where we compare old_cluster.nsubs with
> max_active_replication_origins. But I'd like to hear what others think
> before we make this change.
But if origin created by users (or any extensions) have already been tracking the
Remote_lsn, they should be taken into account. If there are three enabled
subscriptions, two active user-created origins but the max_active_origin is set
to three on the new node, the upgrading should fail.
I prefer to preserve the current behavior of the patch, which compares
max_active_origin and total number of origins. It might cause the false-positive,
but even now it can happen. E.g., if there was a subscription which has never
been enabled.
Best regards,
Hayato Kuroda
FUJITSU LIMITED
^ permalink raw reply [nested|flat] 97+ messages in thread
* Re: [PATCH] Preserve replication origin OIDs in pg_upgrade
@ 2026-07-08 06:27 shveta malik <[email protected]>
parent: Hayato Kuroda (Fujitsu) <[email protected]>
1 sibling, 0 replies; 97+ messages in thread
From: shveta malik @ 2026-07-08 06:27 UTC (permalink / raw)
To: Hayato Kuroda (Fujitsu) <[email protected]>; +Cc: Shlok Kyal <[email protected]>; Ajin Cherian <[email protected]>; Zsolt Parragi <[email protected]>; [email protected] <[email protected]>; shveta malik <[email protected]>
On Wed, Jul 8, 2026 at 11:51 AM Hayato Kuroda (Fujitsu)
<[email protected]> wrote:
>
> Dear Shveta,
>
> > For this reason, my inclination is to revert to the previous
> > implementation, where we compare old_cluster.nsubs with
> > max_active_replication_origins. But I'd like to hear what others think
> > before we make this change.
>
> But if origin created by users (or any extensions) have already been tracking the
> Remote_lsn, they should be taken into account. If there are three enabled
> subscriptions, two active user-created origins but the max_active_origin is set
> to three on the new node, the upgrading should fail.
>
> I prefer to preserve the current behavior of the patch, which compares
> max_active_origin and total number of origins. It might cause the false-positive,
> but even now it can happen. E.g., if there was a subscription which has never
> been enabled.
>
Okay, I missed this point earlier. I agree, let's retain the current
behaviour itself.
thanks
Shveta
^ permalink raw reply [nested|flat] 97+ messages in thread
* Re: [PATCH] Preserve replication origin OIDs in pg_upgrade
@ 2026-07-08 09:09 Amit Kapila <[email protected]>
parent: Hayato Kuroda (Fujitsu) <[email protected]>
1 sibling, 1 reply; 97+ messages in thread
From: Amit Kapila @ 2026-07-08 09:09 UTC (permalink / raw)
To: Hayato Kuroda (Fujitsu) <[email protected]>; +Cc: shveta malik <[email protected]>; Shlok Kyal <[email protected]>; Ajin Cherian <[email protected]>; Zsolt Parragi <[email protected]>; [email protected] <[email protected]>
On Wed, Jul 8, 2026 at 11:51 AM Hayato Kuroda (Fujitsu)
<[email protected]> wrote:
>
> > For this reason, my inclination is to revert to the previous
> > implementation, where we compare old_cluster.nsubs with
> > max_active_replication_origins. But I'd like to hear what others think
> > before we make this change.
>
> But if origin created by users (or any extensions) have already been tracking the
> Remote_lsn, they should be taken into account. If there are three enabled
> subscriptions, two active user-created origins but the max_active_origin is set
> to three on the new node, the upgrading should fail.
>
Fair point but isn't it sufficient to check the tracked_origins rather
than all origins? In docs of max_active_replication_origins, we say:
"Setting it to a lower value than the current number of tracked
replication origins (reflected in pg_replication_origin_status) will
prevent the server from starting." which suggests here we should have
a tight check of tracked_origins against
max_active_replication_origins.
> I prefer to preserve the current behavior of the patch, which compares
> max_active_origin and total number of origins. It might cause the false-positive,
> but even now it can happen. E.g., if there was a subscription which has never
> been enabled.
>
I think it is possible that non-tracked origins (e.g for subscriptions
that get enabled only after upgrade) could lead to ERROR after upgrade
and starting the server but users can always increase
max_active_replication_origins to accommodate that which she anyway
needs to do even before upgrade.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 97+ messages in thread
* Re: [PATCH] Preserve replication origin OIDs in pg_upgrade
@ 2026-07-08 10:02 Amit Kapila <[email protected]>
parent: Ajin Cherian <[email protected]>
1 sibling, 0 replies; 97+ messages in thread
From: Amit Kapila @ 2026-07-08 10:02 UTC (permalink / raw)
To: Ajin Cherian <[email protected]>; +Cc: shveta malik <[email protected]>; Shlok Kyal <[email protected]>; Dilip Kumar <[email protected]>; Zsolt Parragi <[email protected]>; vignesh C <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; PostgreSQL Hackers <[email protected]>
On Tue, Jun 16, 2026 at 2:50 PM Ajin Cherian <[email protected]> wrote:
>
> On Mon, Jun 15, 2026 at 9:31 PM Amit Kapila <[email protected]> wrote:
>
>>
>> Separately, I don't think we actually need to preserve the
>> subscription OID to meet the goal here. pg_commit_ts stores the
>> numeric roident, not the origin name or the sub OID, and conflict
>> detection compares roidents. So the invariant we must keep is "each
>> subscription owns the same roident after upgrade", which doesn't
>> depend on the name being stable.
>>
>>
>
> Replication origins are global objects and are therefore dumped by pg_dumpall, while subscriptions are database-specific and are dumped by pg_dump. In pg_upgrade, pg_dumpall runs before pg_dump.
>
> The complication if subscription OID is not preserved is that subscription-associated origins derive their name from the subscription's OID. If the subscription OID is not preserved across upgrades, the origin name on the new cluster will differ from the old one, making it impossible to restore the origin independently. For these origins, creation must happen after CreateSubscription has established the new OID and not upfront in pg_dumpall.
>
> Non-subscription origins have no such dependency and continue to be created separately as before. The subscription code also cannot be left untouched: even once the subscription is created, rather than origin being created inside CreateSubscription, its associated origin must still be explicitly created via binary_upgrade_create_replication_origin to ensure the roident matches the original node, which means CreateSubscription needs to be modified anyways. So, there is an advantage of reducing code if subscription OID is preserved and all origins keep the same name as the old node.
>
I think here the key is not reducing the code foot print in
CreateSubscription rather the non-subscription origins anyway needs to
be created separately by pg_dumpall as they don't have any binding
with subscription. So, if we don't preserve sub OIDs, we will never be
able to match the names of origins created with pg_dumpall and then
later used by subscriptions. I feel this reasoning should be captured
in the commit message.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 97+ messages in thread
* RE: [PATCH] Preserve replication origin OIDs in pg_upgrade
@ 2026-07-08 10:07 Hayato Kuroda (Fujitsu) <[email protected]>
parent: Amit Kapila <[email protected]>
0 siblings, 1 reply; 97+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2026-07-08 10:07 UTC (permalink / raw)
To: 'Amit Kapila' <[email protected]>; +Cc: shveta malik <[email protected]>; Shlok Kyal <[email protected]>; Ajin Cherian <[email protected]>; Zsolt Parragi <[email protected]>; [email protected] <[email protected]>
Dear Amit,
> Fair point but isn't it sufficient to check the tracked_origins rather
> than all origins? In docs of max_active_replication_origins, we say:
> "Setting it to a lower value than the current number of tracked
> replication origins (reflected in pg_replication_origin_status) will
> prevent the server from starting." which suggests here we should have
> a tight check of tracked_origins against
> max_active_replication_origins.
Basically it sounds good, but it can slightly change the behavior if the old node
has some subscriptions which have never been enabled. Till now, we just counted
the number of subscriptions, thus we took into account them [1]. But after the
patch they can be ignored; IIUC pg_upgrade can succeed in the case.
If the change is OK, we can proceed with your (and Shlok-san's) idea.
[1]: reproducer is below.
1. initialized old node with max_active_replication_origins=1
2. created two subscriptions with (connect=false)
3. initialized new node with max_active_replication_origins=1
4. ran pg_upgrade and it did error-out
```
$ pg_upgrade -b /usr/local/pgsql/bin/ -d old/ -D new/ -U postgres
Performing Consistency Checks
-----------------------------
...
Checking new cluster configuration for subscriptions
"max_active_replication_origins" (1) must be greater than or equal to the number of subscriptions (2) in the old cluster
Failure, exiting
```
Best regards,
Hayato Kuroda
FUJITSU LIMITED
^ permalink raw reply [nested|flat] 97+ messages in thread
* Re: [PATCH] Preserve replication origin OIDs in pg_upgrade
@ 2026-07-08 10:42 Amit Kapila <[email protected]>
parent: Hayato Kuroda (Fujitsu) <[email protected]>
0 siblings, 1 reply; 97+ messages in thread
From: Amit Kapila @ 2026-07-08 10:42 UTC (permalink / raw)
To: Hayato Kuroda (Fujitsu) <[email protected]>; +Cc: shveta malik <[email protected]>; Shlok Kyal <[email protected]>; Ajin Cherian <[email protected]>; Zsolt Parragi <[email protected]>; [email protected] <[email protected]>
On Wed, Jul 8, 2026 at 3:37 PM Hayato Kuroda (Fujitsu)
<[email protected]> wrote:
>
> > Fair point but isn't it sufficient to check the tracked_origins rather
> > than all origins? In docs of max_active_replication_origins, we say:
> > "Setting it to a lower value than the current number of tracked
> > replication origins (reflected in pg_replication_origin_status) will
> > prevent the server from starting." which suggests here we should have
> > a tight check of tracked_origins against
> > max_active_replication_origins.
>
> Basically it sounds good, but it can slightly change the behavior if the old node
> has some subscriptions which have never been enabled. Till now, we just counted
> the number of subscriptions, thus we took into account them [1]. But after the
> patch they can be ignored; IIUC pg_upgrade can succeed in the case.
>
> If the change is OK, we can proceed with your (and Shlok-san's) idea.
>
> [1]: reproducer is below.
> 1. initialized old node with max_active_replication_origins=1
> 2. created two subscriptions with (connect=false)
> 3. initialized new node with max_active_replication_origins=1
> 4. ran pg_upgrade and it did error-out
>
> ```
> $ pg_upgrade -b /usr/local/pgsql/bin/ -d old/ -D new/ -U postgres
> Performing Consistency Checks
> -----------------------------
> ...
> Checking new cluster configuration for subscriptions
> "max_active_replication_origins" (1) must be greater than or equal to the number of subscriptions (2) in the old cluster
> Failure, exiting
> ```
>
Yes, I understand this case but I feel the old check is more
restrictive than required. We might want to even back-patch this small
part in future if we see any such user complaint.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 97+ messages in thread
* Re: [PATCH] Preserve replication origin OIDs in pg_upgrade
@ 2026-07-09 03:28 shveta malik <[email protected]>
parent: Amit Kapila <[email protected]>
0 siblings, 1 reply; 97+ messages in thread
From: shveta malik @ 2026-07-09 03:28 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Hayato Kuroda (Fujitsu) <[email protected]>; Shlok Kyal <[email protected]>; Ajin Cherian <[email protected]>; Zsolt Parragi <[email protected]>; [email protected] <[email protected]>; shveta malik <[email protected]>
On Wed, Jul 8, 2026 at 4:12 PM Amit Kapila <[email protected]> wrote:
>
> On Wed, Jul 8, 2026 at 3:37 PM Hayato Kuroda (Fujitsu)
> <[email protected]> wrote:
> >
> > > Fair point but isn't it sufficient to check the tracked_origins rather
> > > than all origins? In docs of max_active_replication_origins, we say:
> > > "Setting it to a lower value than the current number of tracked
> > > replication origins (reflected in pg_replication_origin_status) will
> > > prevent the server from starting." which suggests here we should have
> > > a tight check of tracked_origins against
> > > max_active_replication_origins.
> >
> > Basically it sounds good, but it can slightly change the behavior if the old node
> > has some subscriptions which have never been enabled. Till now, we just counted
> > the number of subscriptions, thus we took into account them [1]. But after the
> > patch they can be ignored; IIUC pg_upgrade can succeed in the case.
> >
> > If the change is OK, we can proceed with your (and Shlok-san's) idea.
> >
> > [1]: reproducer is below.
> > 1. initialized old node with max_active_replication_origins=1
> > 2. created two subscriptions with (connect=false)
> > 3. initialized new node with max_active_replication_origins=1
> > 4. ran pg_upgrade and it did error-out
> >
> > ```
> > $ pg_upgrade -b /usr/local/pgsql/bin/ -d old/ -D new/ -U postgres
> > Performing Consistency Checks
> > -----------------------------
> > ...
> > Checking new cluster configuration for subscriptions
> > "max_active_replication_origins" (1) must be greater than or equal to the number of subscriptions (2) in the old cluster
> > Failure, exiting
> > ```
> >
>
> Yes, I understand this case but I feel the old check is more
> restrictive than required. We might want to even back-patch this small
> part in future if we see any such user complaint.
>
Okay, makes sense.
thanks
Shveta
^ permalink raw reply [nested|flat] 97+ messages in thread
* Re: [PATCH] Preserve replication origin OIDs in pg_upgrade
@ 2026-07-09 11:37 Ajin Cherian <[email protected]>
parent: shveta malik <[email protected]>
0 siblings, 0 replies; 97+ messages in thread
From: Ajin Cherian @ 2026-07-09 11:37 UTC (permalink / raw)
To: shveta malik <[email protected]>; +Cc: Amit Kapila <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Shlok Kyal <[email protected]>; Zsolt Parragi <[email protected]>; [email protected] <[email protected]>
Thank you all for the comments. I have tried to address all the
comments. I am not responding to each of them here. Based on the
discussion above, I have retained the logic as mentioned by Shlok to
compare active origins against max_active_replication_origins.
Attaching v11 with all comments addressed.
regards,
Ajin Cherian
Fujitsu Australia
Attachments:
[application/octet-stream] v11-0001-Preserve-subscription-OIDs-during-pg_upgrade.patch (9.4K, ../../CAFPTHDbVLyri=pehK6dT+G+QE5Kh-Zv0XEBsvAQ4PEkgmJh5SA@mail.gmail.com/2-v11-0001-Preserve-subscription-OIDs-during-pg_upgrade.patch)
download | inline diff:
From d8ddf11c4877efa2f2a5bf183a8e0e02e9737275 Mon Sep 17 00:00:00 2001
From: Ajin Cherian <[email protected]>
Date: Thu, 9 Jul 2026 21:06:04 +1000
Subject: [PATCH v11 1/2] Preserve subscription OIDs during pg_upgrade
Replication origins are linked to their subscriptions only by
naming convention, where the origin name embeds the subscription's
OID (e.g. "pg_<suboid>"). pg_dumpall restores origins independently
of CreateSubscription, so if subscription OIDs change during
upgrade, restored origins will no longer match their subscriptions.
This patch preserves subscription OIDs so a follow-up patch can
correctly restore replication origins.
Author: Vignesh C <[email protected]>
Reviewer: Ajin Cherian <[email protected]>
---
src/backend/commands/subscriptioncmds.c | 25 +++++++++++++++++--
src/backend/utils/adt/pg_upgrade_support.c | 10 ++++++++
src/bin/pg_dump/pg_dump.c | 8 ++++++
src/bin/pg_upgrade/pg_upgrade.c | 3 +++
src/bin/pg_upgrade/t/004_subscription.pl | 7 ++++++
src/include/catalog/binary_upgrade.h | 1 +
src/include/catalog/pg_proc.dat | 4 +++
.../expected/spgist_name_ops.out | 6 +++--
8 files changed, 60 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 4292e7fb8f4..e87c18356de 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -86,6 +86,12 @@
/* check if the 'val' has 'bits' set */
#define IsSet(val, bits) (((val) & (bits)) == (bits))
+/*
+ * This will be set by the pg_upgrade_support function --
+ * binary_upgrade_set_next_pg_subscription_oid().
+ */
+Oid binary_upgrade_next_pg_subscription_oid = InvalidOid;
+
/*
* Structure to hold a bitmap representing the user-provided CREATE/ALTER
* SUBSCRIPTION command options and the parsed/default values of each of them.
@@ -828,8 +834,23 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
memset(values, 0, sizeof(values));
memset(nulls, false, sizeof(nulls));
- subid = GetNewOidWithIndex(rel, SubscriptionObjectIndexId,
- Anum_pg_subscription_oid);
+ /* Use binary-upgrade override for pg_subscription.oid? */
+ if (IsBinaryUpgrade)
+ {
+ if (!OidIsValid(binary_upgrade_next_pg_subscription_oid))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("pg_subscription OID value not set when in binary upgrade mode")));
+
+ subid = binary_upgrade_next_pg_subscription_oid;
+ binary_upgrade_next_pg_subscription_oid = InvalidOid;
+ }
+ else
+ {
+ subid = GetNewOidWithIndex(rel, SubscriptionObjectIndexId,
+ Anum_pg_subscription_oid);
+ }
+
values[Anum_pg_subscription_oid - 1] = ObjectIdGetDatum(subid);
values[Anum_pg_subscription_subdbid - 1] = ObjectIdGetDatum(MyDatabaseId);
values[Anum_pg_subscription_subskiplsn - 1] = LSNGetDatum(InvalidXLogRecPtr);
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index b505a6b4fee..59c3e7f0146 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -181,6 +181,16 @@ binary_upgrade_set_next_pg_authid_oid(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
+Datum
+binary_upgrade_set_next_pg_subscription_oid(PG_FUNCTION_ARGS)
+{
+ Oid subid = PG_GETARG_OID(0);
+
+ CHECK_IS_BINARY_UPGRADE;
+ binary_upgrade_next_pg_subscription_oid = subid;
+ PG_RETURN_VOID();
+}
+
Datum
binary_upgrade_create_empty_extension(PG_FUNCTION_ARGS)
{
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 4d660d14b4c..75075ae080b 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -5537,6 +5537,14 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
appendPQExpBuffer(delq, "DROP SUBSCRIPTION %s;\n",
qsubname);
+ if (dopt->binary_upgrade)
+ {
+ appendPQExpBufferStr(query, "\n-- For binary upgrade, must preserve pg_subscription.oid\n");
+ appendPQExpBuffer(query,
+ "SELECT pg_catalog.binary_upgrade_set_next_pg_subscription_oid('%u'::pg_catalog.oid);\n\n",
+ subinfo->dobj.catId.oid);
+ }
+
appendPQExpBuffer(query, "CREATE SUBSCRIPTION %s ",
qsubname);
if (subinfo->subservername)
diff --git a/src/bin/pg_upgrade/pg_upgrade.c b/src/bin/pg_upgrade/pg_upgrade.c
index 7366fd4627c..26b1c8362dd 100644
--- a/src/bin/pg_upgrade/pg_upgrade.c
+++ b/src/bin/pg_upgrade/pg_upgrade.c
@@ -35,6 +35,9 @@
*
* We control all assignments of pg_database.oid because we want the directory
* names to match between the old and new cluster.
+ *
+ * We control assignment of pg_subscription.oid because we want the oid to
+ * match between the old and new cluster.
*/
diff --git a/src/bin/pg_upgrade/t/004_subscription.pl b/src/bin/pg_upgrade/t/004_subscription.pl
index 36f059faa2e..9286c1dc85e 100644
--- a/src/bin/pg_upgrade/t/004_subscription.pl
+++ b/src/bin/pg_upgrade/t/004_subscription.pl
@@ -313,6 +313,9 @@ my $tab_upgraded1_oid = $old_sub->safe_psql('postgres',
my $tab_upgraded2_oid = $old_sub->safe_psql('postgres',
"SELECT oid FROM pg_class WHERE relname = 'tab_upgraded2'");
+$sub_oid = $old_sub->safe_psql('postgres',
+ "SELECT oid FROM pg_subscription ORDER BY subname");
+
$old_sub->stop;
# Change configuration so that initial table sync does not get started
@@ -359,6 +362,10 @@ $publisher->safe_psql(
$new_sub->start;
+# The subscription oid should be preserved
+$result = $new_sub->safe_psql('postgres', "SELECT oid FROM pg_subscription ORDER BY subname");
+is($result, qq($sub_oid), "subscription oid should have been preserved");
+
# The subscription's running status, failover option, and retain_dead_tuples
# option should be preserved in the upgraded instance. So regress_sub4 should
# still have subenabled, subfailover, and subretaindeadtuples set to true,
diff --git a/src/include/catalog/binary_upgrade.h b/src/include/catalog/binary_upgrade.h
index 7bf7ae44385..b15b18e7dc9 100644
--- a/src/include/catalog/binary_upgrade.h
+++ b/src/include/catalog/binary_upgrade.h
@@ -32,6 +32,7 @@ extern PGDLLIMPORT RelFileNumber binary_upgrade_next_toast_pg_class_relfilenumbe
extern PGDLLIMPORT Oid binary_upgrade_next_pg_enum_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_authid_oid;
+extern PGDLLIMPORT Oid binary_upgrade_next_pg_subscription_oid;
extern PGDLLIMPORT bool binary_upgrade_record_init_privs;
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 3cb84359adf..65d91e0d8de 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -11994,6 +11994,10 @@
proisstrict => 'f', provolatile => 'v', proparallel => 'u',
prorettype => 'void', proargtypes => '',
prosrc => 'binary_upgrade_create_conflict_detection_slot' },
+{ oid => '9160', descr => 'for use by pg_upgrade',
+ proname => 'binary_upgrade_set_next_pg_subscription_oid', provolatile => 'v',
+ proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
+ prosrc => 'binary_upgrade_set_next_pg_subscription_oid' },
# conversion functions
{ oid => '4310', descr => 'internal conversion function for KOI8R to WIN1251',
diff --git a/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out b/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out
index 1ee65ede243..39d43368c42 100644
--- a/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out
+++ b/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out
@@ -59,11 +59,12 @@ select * from t
binary_upgrade_set_next_multirange_pg_type_oid | 1 | binary_upgrade_set_next_multirange_pg_type_oid
binary_upgrade_set_next_pg_authid_oid | | binary_upgrade_set_next_pg_authid_oid
binary_upgrade_set_next_pg_enum_oid | | binary_upgrade_set_next_pg_enum_oid
+ binary_upgrade_set_next_pg_subscription_oid | | binary_upgrade_set_next_pg_subscription_oid
binary_upgrade_set_next_pg_tablespace_oid | | binary_upgrade_set_next_pg_tablespace_oid
binary_upgrade_set_next_pg_type_oid | | binary_upgrade_set_next_pg_type_oid
binary_upgrade_set_next_toast_pg_class_oid | 1 | binary_upgrade_set_next_toast_pg_class_oid
binary_upgrade_set_next_toast_relfilenode | | binary_upgrade_set_next_toast_relfilenode
-(13 rows)
+(14 rows)
-- Verify clean failure when INCLUDE'd columns result in overlength tuple
-- The error message details are platform-dependent, so show only SQLSTATE
@@ -108,11 +109,12 @@ select * from t
binary_upgrade_set_next_multirange_pg_type_oid | 1 | binary_upgrade_set_next_multirange_pg_type_oid
binary_upgrade_set_next_pg_authid_oid | | binary_upgrade_set_next_pg_authid_oid
binary_upgrade_set_next_pg_enum_oid | | binary_upgrade_set_next_pg_enum_oid
+ binary_upgrade_set_next_pg_subscription_oid | | binary_upgrade_set_next_pg_subscription_oid
binary_upgrade_set_next_pg_tablespace_oid | | binary_upgrade_set_next_pg_tablespace_oid
binary_upgrade_set_next_pg_type_oid | | binary_upgrade_set_next_pg_type_oid
binary_upgrade_set_next_toast_pg_class_oid | 1 | binary_upgrade_set_next_toast_pg_class_oid
binary_upgrade_set_next_toast_relfilenode | | binary_upgrade_set_next_toast_relfilenode
-(13 rows)
+(14 rows)
\set VERBOSITY sqlstate
insert into t values(repeat('xyzzy', 12), 42, repeat('xyzzy', 4000));
--
2.47.3
[application/octet-stream] v11-0002-Preserve-replication-origin-OIDs-during-pg_upgra.patch (36.8K, ../../CAFPTHDbVLyri=pehK6dT+G+QE5Kh-Zv0XEBsvAQ4PEkgmJh5SA@mail.gmail.com/3-v11-0002-Preserve-replication-origin-OIDs-during-pg_upgra.patch)
download | inline diff:
From 16f6719bcec4fb06a16d5d79527df68f63cbd63b Mon Sep 17 00:00:00 2001
From: Ajin Cherian <[email protected]>
Date: Thu, 9 Jul 2026 21:13:09 +1000
Subject: [PATCH v11 2/2] Preserve replication origin OIDs during pg_upgrade
When pg_upgrade migrates a subscriber, replication origin OIDs
(roident) can change across the upgrade. This is a problem because
commit-timestamp records embed roident and are copied directly from
the old cluster's pg_commit_ts directory, causing spurious
"update_origin_differs" conflicts after the upgrade.
Fix this by dumping replication origins as global objects via
pg_dumpall during binary upgrade, using a new function
binary_upgrade_create_replication_origin(oid, name, lsn) to recreate
each origin with its preserved roident and remote_lsn. To avoid
conflicts with this, CreateSubscription() skips replorigin_create()
in binary-upgrade mode since the origin is already created by the
time the subscription is restored.
Author: Ajin Cherian <[email protected]>
Reviewer: Hayato Kuroda (Fujitsu) <[email protected]>
Reviewer: Zsolt Parragi <[email protected]>
Reviewer: Shlok Kyal <[email protected]>
Reviewer: Shveta malik <[email protected]>
Reviewer: Vignesh C <[email protected]>
Reviewer: Nisha Moond <[email protected]>
Reviewer: Rui Zhao <[email protected]>
---
doc/src/sgml/logical-replication.sgml | 4 +-
src/backend/commands/subscriptioncmds.c | 12 ++-
src/backend/replication/logical/origin.c | 95 +++++++++++--------
src/backend/utils/adt/pg_upgrade_support.c | 101 +++++++++++----------
src/bin/pg_dump/pg_dump.c | 51 ++---------
src/bin/pg_dump/pg_dump.h | 1 -
src/bin/pg_dump/pg_dumpall.c | 68 ++++++++++++++
src/bin/pg_upgrade/check.c | 65 +++++++++----
src/bin/pg_upgrade/info.c | 49 ++++++++--
src/bin/pg_upgrade/pg_upgrade.c | 13 ++-
src/bin/pg_upgrade/pg_upgrade.h | 4 +-
src/bin/pg_upgrade/t/004_subscription.pl | 74 +++++++++++++--
src/include/catalog/pg_proc.dat | 8 +-
src/include/replication/origin.h | 3 +
14 files changed, 374 insertions(+), 174 deletions(-)
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index 690598bff98..4db729df283 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -2844,7 +2844,7 @@ CONTEXT: processing remote data for replication origin "pg_16395" during "INSER
<note>
<para>
- Commit timestamps and origin data are not preserved during the upgrade.
+ Commit timestamps are not preserved during the upgrade.
As a result, even if
<link linkend="sql-createsubscription-params-with-retain-dead-tuples"><literal>retain_dead_tuples</literal></link>
is enabled, the upgraded subscriber may be unable to detect conflicts or
@@ -2886,7 +2886,7 @@ CONTEXT: processing remote data for replication origin "pg_16395" during "INSER
The new cluster must have
<link linkend="guc-max-active-replication-origins"><varname>max_active_replication_origins</varname></link>
configured to a value greater than or equal to the number of
- subscriptions present in the old cluster.
+ replication origins present in the old cluster. The new cluster must contain no replication origins.
</para>
</listitem>
<listitem>
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index e87c18356de..28bd0e48f2c 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -19,6 +19,7 @@
#include "access/table.h"
#include "access/twophase.h"
#include "access/xact.h"
+#include "catalog/binary_upgrade.h"
#include "catalog/catalog.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
@@ -957,9 +958,16 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
* apply workers initialization, and to handle origin creation dynamically
* when tables are added to the subscription. It is not clear whether
* preventing creation of origins is worth additional complexity.
+ *
+ * In binary-upgrade mode, skip origin creation here. It will be created
+ * separately so that the origin ID (roident) from the old cluster can be
+ * preserved for this subscription.
*/
- ReplicationOriginNameForLogicalRep(subid, InvalidOid, originname, sizeof(originname));
- replorigin_create(originname);
+ if (!IsBinaryUpgrade)
+ {
+ ReplicationOriginNameForLogicalRep(subid, InvalidOid, originname, sizeof(originname));
+ replorigin_create(originname);
+ }
/*
* Connect to remote side to execute requested commands and fetch table
diff --git a/src/backend/replication/logical/origin.c b/src/backend/replication/logical/origin.c
index c9dfb094c2b..6ba1a9e6b61 100644
--- a/src/backend/replication/logical/origin.c
+++ b/src/backend/replication/logical/origin.c
@@ -265,6 +265,51 @@ replorigin_by_name(const char *roname, bool missing_ok)
return roident;
}
+/*
+ * replorigin_create_with_id
+ *
+ * Create a replication origin with a specific ID and name, optionally
+ * restoring its remote_lsn. Used by pg_upgrade to preserve replication
+ * origin IDs across the upgrade.
+ *
+ * Caller must hold an exclusive lock on ReplicationOriginRelationId.
+ *
+ * Needs to be called in a transaction.
+ */
+void
+replorigin_create_with_id(ReplOriginId roident, const char *roname,
+ XLogRecPtr remote_lsn, Relation rel)
+{
+ Datum roname_d;
+ bool nulls[Natts_pg_replication_origin];
+ Datum values[Natts_pg_replication_origin];
+ HeapTuple tuple;
+
+ Assert(IsTransactionState());
+ Assert(CheckRelationLockedByMe(rel, ExclusiveLock, false));
+
+ roname_d = CStringGetTextDatum(roname);
+
+ memset(&nulls, 0, sizeof(nulls));
+ memset(&values, 0, sizeof(values));
+
+ values[Anum_pg_replication_origin_roident - 1] = ObjectIdGetDatum(roident);
+ values[Anum_pg_replication_origin_roname - 1] = roname_d;
+
+ tuple = heap_form_tuple(RelationGetDescr(rel), values, nulls);
+ CatalogTupleInsert(rel, tuple);
+ heap_freetuple(tuple);
+ CommandCounterIncrement();
+
+ if (remote_lsn != InvalidXLogRecPtr)
+ {
+ Assert(IsBinaryUpgrade);
+ replorigin_advance(roident, remote_lsn, InvalidXLogRecPtr,
+ false /* backward */,
+ false /* WAL log */);
+ }
+}
+
/*
* Create a replication origin.
*
@@ -273,13 +318,11 @@ replorigin_by_name(const char *roname, bool missing_ok)
ReplOriginId
replorigin_create(const char *roname)
{
- Oid roident;
- HeapTuple tuple = NULL;
- Relation rel;
- Datum roname_d;
- SnapshotData SnapshotDirty;
- SysScanDesc scan;
- ScanKeyData key;
+ Oid roident;
+ Relation rel;
+ SnapshotData SnapshotDirty;
+ SysScanDesc scan;
+ ScanKeyData key;
/*
* To avoid needing a TOAST table for pg_replication_origin, we limit
@@ -293,8 +336,6 @@ replorigin_create(const char *roname)
errdetail("Replication origin names must be no longer than %d bytes.",
MAX_RONAME_LEN)));
- roname_d = CStringGetTextDatum(roname);
-
Assert(IsTransactionState());
/*
@@ -321,17 +362,15 @@ replorigin_create(const char *roname)
* snapshot. To make that safe, it needs to not have a TOAST table, since
* TOASTed data cannot be fetched without a snapshot. As of this writing,
* its only varlena column is roname, which we limit to 512 bytes to avoid
- * needing out-of-line storage. If you add a TOAST table to this catalog,
- * be sure to set up a snapshot everywhere it might be needed. For more
+ * needing out-of-line storage. If you add a TOAST table to this catalog,
+ * be sure to set up a snapshot everywhere it might be needed. For more
* information, see https://postgr.es/m/ZvMSUPOqUU-VNADN%40nathan.
*/
Assert(!OidIsValid(rel->rd_rel->reltoastrelid));
for (roident = InvalidOid + 1; roident < PG_UINT16_MAX; roident++)
{
- bool nulls[Natts_pg_replication_origin];
- Datum values[Natts_pg_replication_origin];
- bool collides;
+ bool collides;
CHECK_FOR_INTERRUPTS();
@@ -341,41 +380,25 @@ replorigin_create(const char *roname)
ObjectIdGetDatum(roident));
scan = systable_beginscan(rel, ReplicationOriginIdentIndex,
- true /* indexOK */ ,
+ true /* indexOK */,
&SnapshotDirty,
1, &key);
-
collides = HeapTupleIsValid(systable_getnext(scan));
-
systable_endscan(scan);
if (!collides)
- {
- /*
- * Ok, found an unused roident, insert the new row and do a CCI,
- * so our callers can look it up if they want to.
- */
- memset(&nulls, 0, sizeof(nulls));
-
- values[Anum_pg_replication_origin_roident - 1] = ObjectIdGetDatum(roident);
- values[Anum_pg_replication_origin_roname - 1] = roname_d;
-
- tuple = heap_form_tuple(RelationGetDescr(rel), values, nulls);
- CatalogTupleInsert(rel, tuple);
- CommandCounterIncrement();
break;
- }
}
- /* now release lock again, */
- table_close(rel, ExclusiveLock);
-
- if (tuple == NULL)
+ if (roident >= PG_UINT16_MAX)
ereport(ERROR,
(errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
errmsg("could not find free replication origin ID")));
- heap_freetuple(tuple);
+ replorigin_create_with_id(roident, roname, InvalidXLogRecPtr, rel);
+
+ table_close(rel, ExclusiveLock);
+
return roident;
}
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index 59c3e7f0146..628b077ab69 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -28,6 +28,7 @@
#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
+#include "utils/syscache.h"
#include "utils/pg_lsn.h"
@@ -377,71 +378,79 @@ binary_upgrade_add_sub_rel_state(PG_FUNCTION_ARGS)
}
/*
- * binary_upgrade_replorigin_advance
+ * binary_upgrade_create_conflict_detection_slot
*
- * Update the remote_lsn for the subscriber's replication origin.
+ * Create a replication slot to retain information necessary for conflict
+ * detection such as dead tuples, commit timestamps, and origins.
*/
Datum
-binary_upgrade_replorigin_advance(PG_FUNCTION_ARGS)
+binary_upgrade_create_conflict_detection_slot(PG_FUNCTION_ARGS)
{
- Relation rel;
- Oid subid;
- char *subname;
- char originname[NAMEDATALEN];
- ReplOriginId node;
- XLogRecPtr remote_commit;
-
CHECK_IS_BINARY_UPGRADE;
- /*
- * We must ensure a non-NULL subscription name before dereferencing the
- * arguments.
- */
- if (PG_ARGISNULL(0))
- elog(ERROR, "null argument to binary_upgrade_replorigin_advance is not allowed");
-
- subname = text_to_cstring(PG_GETARG_TEXT_PP(0));
- remote_commit = PG_ARGISNULL(1) ? InvalidXLogRecPtr : PG_GETARG_LSN(1);
-
- rel = table_open(SubscriptionRelationId, RowExclusiveLock);
- subid = get_subscription_oid(subname, false);
-
- ReplicationOriginNameForLogicalRep(subid, InvalidOid, originname, sizeof(originname));
-
- /* Lock to prevent the replication origin from vanishing */
- LockRelationOid(ReplicationOriginRelationId, RowExclusiveLock);
- node = replorigin_by_name(originname, false);
-
- /*
- * The server will be stopped after setting up the objects in the new
- * cluster and the origins will be flushed during the shutdown checkpoint.
- * This will ensure that the latest LSN values for origin will be
- * available after the upgrade.
- */
- replorigin_advance(node, remote_commit, InvalidXLogRecPtr,
- false /* backward */ ,
- false /* WAL log */ );
+ CreateConflictDetectionSlot();
- UnlockRelationOid(ReplicationOriginRelationId, RowExclusiveLock);
- table_close(rel, RowExclusiveLock);
+ ReplicationSlotRelease();
PG_RETURN_VOID();
}
/*
- * binary_upgrade_create_conflict_detection_slot
+ * binary_upgrade_create_replication_origin
*
- * Create a replication slot to retain information necessary for conflict
- * detection such as dead tuples, commit timestamps, and origins.
+ * Create a replication origin with a specific OID and name, optionally
+ * restoring its remote_lsn. Used by pg_upgrade to preserve replication
+ * origin OIDs across the upgrade.
*/
Datum
-binary_upgrade_create_conflict_detection_slot(PG_FUNCTION_ARGS)
+binary_upgrade_create_replication_origin(PG_FUNCTION_ARGS)
{
+ Oid node_oid;
+ ReplOriginId node;
+ Relation rel;
+ char *originname;
+ XLogRecPtr remote_lsn = InvalidXLogRecPtr;
+
CHECK_IS_BINARY_UPGRADE;
- CreateConflictDetectionSlot();
+ if (PG_ARGISNULL(0) || PG_ARGISNULL(1))
+ elog(ERROR,
+ "null argument to binary_upgrade_create_replication_origin is not allowed");
- ReplicationSlotRelease();
+ node_oid = PG_GETARG_OID(0);
+
+ if (node_oid == InvalidOid || node_oid >= DoNotReplicateId)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("replication origin ID %u is out of range", node_oid));
+
+ node = (ReplOriginId) node_oid;
+
+ originname = text_to_cstring(PG_GETARG_TEXT_PP(1));
+
+ /*
+ * To avoid needing a TOAST table for pg_replication_origin, we limit
+ * replication origin names to 512 bytes. This should be more than enough
+ * for all practical use.
+ */
+ if (strlen(originname) > MAX_RONAME_LEN)
+ ereport(ERROR,
+ errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
+ errmsg("replication origin name is too long"),
+ errdetail("Replication origin names must be no longer than %d bytes.",
+ MAX_RONAME_LEN));
+
+ if (!PG_ARGISNULL(2))
+ remote_lsn = PG_GETARG_LSN(2);
+
+ Assert(IsTransactionState());
+
+ /* Acquire an exclusive lock before inserting the new origin. */
+ rel = table_open(ReplicationOriginRelationId, ExclusiveLock);
+
+ replorigin_create_with_id(node, originname, remote_lsn, rel);
+
+ table_close(rel, ExclusiveLock);
PG_RETURN_VOID();
}
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 75075ae080b..a03599538ae 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -5133,7 +5133,6 @@ getSubscriptions(Archive *fout)
int i_subwalrcvtimeout;
int i_subpublications;
int i_suborigin;
- int i_suboriginremotelsn;
int i_subenabled;
int i_subfailover;
int i_subretaindeadtuples;
@@ -5202,11 +5201,9 @@ getSubscriptions(Archive *fout)
LOGICALREP_ORIGIN_ANY);
if (dopt->binary_upgrade && fout->remoteVersion >= 170000)
- appendPQExpBufferStr(query, " o.remote_lsn AS suboriginremotelsn,\n"
- " s.subenabled,\n");
+ appendPQExpBufferStr(query, " s.subenabled,\n");
else
- appendPQExpBufferStr(query, " NULL AS suboriginremotelsn,\n"
- " false AS subenabled,\n");
+ appendPQExpBufferStr(query, " false AS subenabled,\n");
if (fout->remoteVersion >= 170000)
appendPQExpBufferStr(query,
@@ -5286,7 +5283,6 @@ getSubscriptions(Archive *fout)
i_subwalrcvtimeout = PQfnumber(res, "subwalrcvtimeout");
i_subpublications = PQfnumber(res, "subpublications");
i_suborigin = PQfnumber(res, "suborigin");
- i_suboriginremotelsn = PQfnumber(res, "suboriginremotelsn");
subinfo = pg_malloc_array(SubscriptionInfo, ntups);
@@ -5339,11 +5335,6 @@ getSubscriptions(Archive *fout)
subinfo[i].subpublications =
pg_strdup(PQgetvalue(res, i, i_subpublications));
subinfo[i].suborigin = pg_strdup(PQgetvalue(res, i, i_suborigin));
- if (PQgetisnull(res, i, i_suboriginremotelsn))
- subinfo[i].suboriginremotelsn = NULL;
- else
- subinfo[i].suboriginremotelsn =
- pg_strdup(PQgetvalue(res, i, i_suboriginremotelsn));
/* Decide whether we want to dump it */
selectDumpableObject(&(subinfo[i].dobj), fout);
@@ -5622,37 +5613,15 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
* In binary-upgrade mode, we allow the replication to continue after the
* upgrade.
*/
- if (dopt->binary_upgrade && fout->remoteVersion >= 170000)
+ if (dopt->binary_upgrade && subinfo->subenabled && fout->remoteVersion >= 170000)
{
- if (subinfo->suboriginremotelsn)
- {
- /*
- * Preserve the remote_lsn for the subscriber's replication
- * origin. This value is required to start the replication from
- * the position before the upgrade. This value will be stale if
- * the publisher gets upgraded before the subscriber node.
- * However, this shouldn't be a problem as the upgrade of the
- * publisher ensures that all the transactions were replicated
- * before upgrading it.
- */
- appendPQExpBufferStr(query,
- "\n-- For binary upgrade, must preserve the remote_lsn for the subscriber's replication origin.\n");
- appendPQExpBufferStr(query,
- "SELECT pg_catalog.binary_upgrade_replorigin_advance(");
- appendStringLiteralAH(query, subinfo->dobj.name, fout);
- appendPQExpBuffer(query, ", '%s');\n", subinfo->suboriginremotelsn);
- }
-
- if (subinfo->subenabled)
- {
- /*
- * Enable the subscription to allow the replication to continue
- * after the upgrade.
- */
- appendPQExpBufferStr(query,
- "\n-- For binary upgrade, must preserve the subscriber's running state.\n");
- appendPQExpBuffer(query, "ALTER SUBSCRIPTION %s ENABLE;\n", qsubname);
- }
+ /*
+ * Enable the subscription to allow the replication to continue
+ * after the upgrade.
+ */
+ appendPQExpBufferStr(query,
+ "\n-- For binary upgrade, must preserve the subscriber's running state.\n");
+ appendPQExpBuffer(query, "ALTER SUBSCRIPTION %s ENABLE;\n", qsubname);
}
if (subinfo->dobj.dump & DUMP_COMPONENT_DEFINITION)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 5a6726d8b12..8f1252b854e 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -729,7 +729,6 @@ typedef struct _SubscriptionInfo
char *subwalrcvtimeout;
char *subpublications;
char *suborigin;
- char *suboriginremotelsn;
} SubscriptionInfo;
/*
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index c53e77c2878..6af5bb058cd 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -18,6 +18,7 @@
#include <time.h>
#include <unistd.h>
+#include "access/xlogdefs.h"
#include "catalog/pg_authid_d.h"
#include "common/connect.h"
#include "common/file_perm.h"
@@ -67,6 +68,7 @@ static void dropDBs(PGconn *conn);
static void dumpUserConfig(PGconn *conn, const char *username);
static void dumpDatabases(PGconn *conn);
static void dumpTimestamp(const char *msg);
+static void dumpReplicationOrigins(PGconn *conn);
static int runPgDump(const char *dbname, const char *create_opts);
static void buildShSecLabels(PGconn *conn,
const char *catalog_name, Oid objectId,
@@ -663,8 +665,13 @@ main(int argc, char *argv[])
/* Dump role GUC privileges */
if (server_version >= 150000 && !skip_acls)
dumpRoleGUCPrivs(conn);
+
}
+ /* Dump replication origins */
+ if (!tablespaces_only && !roles_only && binary_upgrade)
+ dumpReplicationOrigins(conn);
+
/* Dump tablespaces */
if (!roles_only && !no_tablespaces)
dumpTablespaces(conn);
@@ -1804,6 +1811,67 @@ dumpTimestamp(const char *msg)
fprintf(OPF, "-- %s %s\n\n", msg, buf);
}
+static void
+dumpReplicationOrigins(PGconn *conn)
+{
+ PQExpBuffer buf = createPQExpBuffer();
+ PGresult *res;
+ int i_roident;
+ int i_roname;
+ int i_remotelsn;
+ int ntups;
+
+ /* Get replication origins from catalogs */
+ appendPQExpBufferStr(buf,
+ "SELECT o.roident, o.roname, os.remote_lsn "
+ "FROM pg_catalog.pg_replication_origin o "
+ "LEFT OUTER JOIN pg_catalog.pg_replication_origin_status os ON o.roident = os.local_id "
+ "ORDER BY o.roident");
+
+ res = executeQuery(conn, buf->data);
+
+ i_roident = PQfnumber(res, "roident");
+ i_roname = PQfnumber(res, "roname");
+ i_remotelsn = PQfnumber(res, "remote_lsn");
+ ntups = PQntuples(res);
+
+ if (ntups > 0)
+ fprintf(OPF, "--\n-- Replication Origins \n--\n\n");
+
+ for (int i = 0; i < ntups; i++)
+ {
+ ReplOriginId roident;
+ const char *roname;
+
+ roident = atooid(PQgetvalue(res, i, i_roident));
+ roname = PQgetvalue(res, i, i_roname);
+
+ resetPQExpBuffer(buf);
+
+ appendPQExpBufferStr(buf, "\n-- For binary upgrade, must preserve pg_replication_origin roident and remote_lsn\n");
+ appendPQExpBuffer(buf,
+ "SELECT pg_catalog.binary_upgrade_create_replication_origin("
+ "'%u'::pg_catalog.oid, ", roident);
+ appendStringLiteralConn(buf, roname, conn);
+ appendPQExpBufferStr(buf, "::pg_catalog.text");
+
+ if (!PQgetisnull(res, i, i_remotelsn))
+ {
+ appendPQExpBufferStr(buf, ", ");
+ appendStringLiteralConn(buf, PQgetvalue(res, i, i_remotelsn), conn);
+ appendPQExpBufferStr(buf, "::pg_catalog.pg_lsn");
+ }
+ else
+ appendPQExpBufferStr(buf, ", NULL");
+
+ appendPQExpBufferStr(buf, ");\n");
+ fprintf(OPF, "%s", buf->data);
+ }
+
+ PQclear(res);
+ destroyPQExpBuffer(buf);
+}
+
/*
* read_dumpall_filters - retrieve database identifier patterns from file
*
diff --git a/src/bin/pg_upgrade/check.c b/src/bin/pg_upgrade/check.c
index 2155b01b11f..1d2db4527a8 100644
--- a/src/bin/pg_upgrade/check.c
+++ b/src/bin/pg_upgrade/check.c
@@ -32,7 +32,7 @@ static void check_for_new_tablespace_dir(void);
static void check_for_user_defined_encoding_conversions(ClusterInfo *cluster);
static void check_for_unicode_update(ClusterInfo *cluster);
static void check_new_cluster_replication_slots(void);
-static void check_new_cluster_subscription_configuration(void);
+static void check_new_cluster_replication_origins(void);
static void check_old_cluster_for_valid_slots(void);
static void check_old_cluster_subscription_state(void);
static void check_old_cluster_global_names(ClusterInfo *cluster);
@@ -589,6 +589,9 @@ check_and_dump_old_cluster(void)
check_old_cluster_subscription_state();
}
+ /* Get replication origin information */
+ get_replication_origin_info(&old_cluster);
+
check_for_data_types_usage(&old_cluster);
/*
@@ -707,7 +710,7 @@ check_new_cluster(void)
check_new_cluster_replication_slots();
- check_new_cluster_subscription_configuration();
+ check_new_cluster_replication_origins();
}
@@ -2150,31 +2153,45 @@ check_new_cluster_replication_slots(void)
}
/*
- * check_new_cluster_subscription_configuration()
+ * check_new_cluster_replication_origins()
+ *
+ * Verify that the new cluster has no replication origins. During upgrade,
+ * pg_upgrade restores replication origins from the old cluster with their
+ * original OIDs. If the new cluster already contains origins, those OIDs
+ * may collide, causing the upgrade to fail mid-way.
*
- * Verify that the max_active_replication_origins configuration specified is
- * enough for creating the subscriptions. This is required to create the
- * replication origin for each subscription.
+ * Also verify that the max_active_replication_origins configuration is
+ * enough for creating all the replication origins.
*/
static void
-check_new_cluster_subscription_configuration(void)
+check_new_cluster_replication_origins(void)
{
- PGresult *res;
PGconn *conn;
+ PGresult *res;
+ int norigins;
int max_active_replication_origins;
- /* Subscriptions and their dependencies can be migrated since PG17. */
- if (GET_MAJOR_VERSION(old_cluster.major_version) < 1700)
+ /* Quick return if there are no replication origins to migrate. */
+ if (old_cluster.nrepl_origins == 0)
return;
- /* Quick return if there are no subscriptions to be migrated. */
- if (old_cluster.nsubs == 0)
- return;
-
- prep_status("Checking new cluster configuration for subscriptions");
+ prep_status("Checking replication origins in new cluster");
conn = connectToServer(&new_cluster, "template1");
+ res = executeQueryOrDie(conn,
+ "SELECT count(*) "
+ "FROM pg_catalog.pg_replication_origin");
+
+ if (PQntuples(res) != 1)
+ pg_fatal("could not count the number of replication origins");
+
+ norigins = atoi(PQgetvalue(res, 0, 0));
+ PQclear(res);
+
+ if (norigins > 0)
+ pg_fatal("expected 0 replication origins but found %d", norigins);
+
res = executeQueryOrDie(conn, "SELECT setting FROM pg_settings "
"WHERE name = 'max_active_replication_origins';");
@@ -2182,12 +2199,20 @@ check_new_cluster_subscription_configuration(void)
pg_fatal("could not determine parameter settings on new cluster");
max_active_replication_origins = atoi(PQgetvalue(res, 0, 0));
- if (old_cluster.nsubs > max_active_replication_origins)
- pg_fatal("\"max_active_replication_origins\" (%d) must be greater than or equal to the number of "
- "subscriptions (%d) in the old cluster",
- max_active_replication_origins, old_cluster.nsubs);
-
PQclear(res);
+
+ /*
+ * Compare against the number of active origins in the old cluster not the
+ * total number of origins in pg_replication_origin. An origin can exist
+ * without ever becoming active, in which case it does not count against
+ * max_active_replication_origins.
+ */
+ if (old_cluster.nrepl_origins_active > max_active_replication_origins)
+ pg_fatal("\"max_active_replication_origins\" (%d) must be greater "
+ "than or equal to the number of active replication origins "
+ "(%d) in the old cluster",
+ max_active_replication_origins, old_cluster.nrepl_origins_active);
+
PQfinish(conn);
check_ok();
diff --git a/src/bin/pg_upgrade/info.c b/src/bin/pg_upgrade/info.c
index 37fff93892f..311d08fe233 100644
--- a/src/bin/pg_upgrade/info.c
+++ b/src/bin/pg_upgrade/info.c
@@ -832,6 +832,43 @@ count_old_cluster_logical_slots(void)
return slot_count;
}
+/*
+ * get_replication_origin_info()
+ *
+ * Gets the information of replication origins in the cluster.
+ */
+void
+get_replication_origin_info(ClusterInfo *cluster)
+{
+ PGconn *conn;
+ PGresult *res;
+
+ conn = connectToServer(cluster, "template1");
+
+ res = executeQueryOrDie(conn, "SELECT count(*) "
+ "FROM pg_catalog.pg_replication_origin");
+ if (PQntuples(res) != 1)
+ pg_fatal("could not get the number of replication origins");
+ cluster->nrepl_origins = atoi(PQgetvalue(res, 0, 0));
+ PQclear(res);
+
+ /*
+ * Get the number of actively tracked replication origins, i.e., those
+ * with an entry in pg_replication_origin_status. This can be less than
+ * nrepl_origins, since an origin can exist without ever becoming
+ * active. This count is what is actually bounded by
+ * max_active_replication_origins.
+ */
+ res = executeQueryOrDie(conn, "SELECT count(*) "
+ "FROM pg_catalog.pg_replication_origin_status");
+ if (PQntuples(res) != 1)
+ pg_fatal("could not get the number of active replication origins");
+ cluster->nrepl_origins_active = atoi(PQgetvalue(res, 0, 0));
+ PQclear(res);
+
+ PQfinish(conn);
+}
+
/*
* get_subscription_info()
*
@@ -842,23 +879,19 @@ get_subscription_info(ClusterInfo *cluster)
{
PGconn *conn;
PGresult *res;
- int i_nsub;
int i_retain_dead_tuples;
conn = connectToServer(cluster, "template1");
if (GET_MAJOR_VERSION(cluster->major_version) >= 1900)
- res = executeQueryOrDie(conn, "SELECT count(*) AS nsub,"
- "COUNT(CASE WHEN subretaindeadtuples THEN 1 END) > 0 AS retain_dead_tuples "
+ res = executeQueryOrDie(conn,
+ "SELECT COUNT(CASE WHEN subretaindeadtuples THEN 1 END) > 0 AS retain_dead_tuples "
"FROM pg_catalog.pg_subscription");
else
- res = executeQueryOrDie(conn, "SELECT count(*) AS nsub,"
- "'f' AS retain_dead_tuples "
- "FROM pg_catalog.pg_subscription");
+ res = executeQueryOrDie(conn,
+ "SELECT 'f' AS retain_dead_tuples ");
- i_nsub = PQfnumber(res, "nsub");
i_retain_dead_tuples = PQfnumber(res, "retain_dead_tuples");
- cluster->nsubs = atoi(PQgetvalue(res, 0, i_nsub));
cluster->sub_retain_dead_tuples = (strcmp(PQgetvalue(res, 0, i_retain_dead_tuples), "t") == 0);
PQclear(res);
diff --git a/src/bin/pg_upgrade/pg_upgrade.c b/src/bin/pg_upgrade/pg_upgrade.c
index 26b1c8362dd..9615460d4f6 100644
--- a/src/bin/pg_upgrade/pg_upgrade.c
+++ b/src/bin/pg_upgrade/pg_upgrade.c
@@ -219,13 +219,12 @@ main(int argc, char **argv)
* as it only retains the dead tuples. It is created here for consistency.
* Note that the new conflict detection slot uses the latest transaction
* ID as xmin, so it cannot protect dead tuples that existed before the
- * upgrade. Additionally, commit timestamps and origin data are not
- * preserved during the upgrade. So, even after creating the slot, the
- * upgraded subscriber may be unable to detect conflicts or log relevant
- * commit timestamps and origins when applying changes from the publisher
- * occurred before the upgrade especially if those changes were not
- * replicated. It can only protect tuples that might be deleted after the
- * new cluster starts.
+ * upgrade. Additionally, commit timestamps are not preserved during the
+ * upgrade. So, even after creating the slot, the upgraded subscriber may
+ * be unable to detect conflicts or log relevant commit timestamps when
+ * applying changes from the publisher occurred before the upgrade
+ * especially if those changes were not replicated. It can only protect
+ * tuples that might be deleted after the new cluster starts.
*/
if (migrate_logical_slots || old_cluster.sub_retain_dead_tuples)
{
diff --git a/src/bin/pg_upgrade/pg_upgrade.h b/src/bin/pg_upgrade/pg_upgrade.h
index d6e5bca5792..a09554ea3dc 100644
--- a/src/bin/pg_upgrade/pg_upgrade.h
+++ b/src/bin/pg_upgrade/pg_upgrade.h
@@ -286,7 +286,8 @@ typedef struct
char **tablespaces; /* tablespace directories */
int num_tablespaces;
const char *tablespace_suffix; /* directory specification */
- int nsubs; /* number of subscriptions */
+ int nrepl_origins; /* number of replication origins */
+ int nrepl_origins_active; /* number of replication origins that are active */
bool sub_retain_dead_tuples; /* whether a subscription enables
* retain_dead_tuples. */
} ClusterInfo;
@@ -425,6 +426,7 @@ FileNameMap *gen_db_file_maps(DbInfo *old_db,
void get_db_rel_and_slot_infos(ClusterInfo *cluster);
int count_old_cluster_logical_slots(void);
void get_subscription_info(ClusterInfo *cluster);
+void get_replication_origin_info(ClusterInfo *cluster);
/* option.c */
diff --git a/src/bin/pg_upgrade/t/004_subscription.pl b/src/bin/pg_upgrade/t/004_subscription.pl
index 9286c1dc85e..be93f0e3cbc 100644
--- a/src/bin/pg_upgrade/t/004_subscription.pl
+++ b/src/bin/pg_upgrade/t/004_subscription.pl
@@ -42,13 +42,12 @@ my $connstr = $publisher->connstr . ' dbname=postgres';
# ------------------------------------------------------
# Check that pg_upgrade fails when max_active_replication_origins configured
-# in the new cluster is less than the number of subscriptions in the old
-# cluster.
+# in the new cluster is less than the number of active replication origins in
+# the old cluster.
# ------------------------------------------------------
-# It is sufficient to use disabled subscription to test upgrade failure.
$publisher->safe_psql('postgres', "CREATE PUBLICATION regress_pub1");
$old_sub->safe_psql('postgres',
- "CREATE SUBSCRIPTION regress_sub1 CONNECTION '$connstr' PUBLICATION regress_pub1 WITH (enabled = false)"
+ "CREATE SUBSCRIPTION regress_sub1 CONNECTION '$connstr' PUBLICATION regress_pub1"
);
$old_sub->stop;
@@ -74,7 +73,7 @@ command_checks_all(
],
1,
[
- qr/"max_active_replication_origins" \(0\) must be greater than or equal to the number of subscriptions \(1\) in the old cluster/
+ qr/"max_active_replication_origins" \(0\) must be greater than or equal to the number of active replication origins \(1\) in the old cluster/
],
[qr//],
'run of pg_upgrade where the new cluster has insufficient max_active_replication_origins'
@@ -301,8 +300,38 @@ is($result, qq(t), "Check that the table is in init state");
# Get the replication origin's remote_lsn of the old subscriber
my $remote_lsn = $old_sub->safe_psql('postgres',
- "SELECT remote_lsn FROM pg_replication_origin_status os, pg_subscription s WHERE os.external_id = 'pg_' || s.oid AND s.subname = 'regress_sub4'"
+ "SELECT os.remote_lsn
+ FROM pg_replication_origin_status os
+ JOIN pg_replication_origin o ON o.roident = os.local_id
+ JOIN pg_subscription s ON o.roname = 'pg_' || s.oid::text
+ WHERE s.subname = 'regress_sub4'"
+);
+
+# Get the replication origin ids (roident) for all subscriptions, keyed by
+# subscription name (which is stable across upgrade, unlike suboid). These
+# must be preserved after upgrade. A mismatch would cause spurious
+# update_origin_differs conflicts.
+my %pre_upgrade_roident;
+my $roident_rows = $old_sub->safe_psql('postgres',
+ "SELECT s.subname, o.roident
+ FROM pg_subscription s
+ JOIN pg_replication_origin o ON o.roname = 'pg_' || s.oid::text
+ ORDER BY s.subname"
);
+for my $row (split /\n/, $roident_rows)
+{
+ my ($subname, $roident) = split /\|/, $row;
+ $pre_upgrade_roident{$subname} = $roident;
+}
+
+# Create a user created replication origin, which should also be preserved after upgrade.
+my $user_origin_name = 'regress_user_origin';
+$old_sub->safe_psql('postgres',
+ "SELECT pg_replication_origin_create('$user_origin_name')");
+$pre_upgrade_roident{$user_origin_name} = $old_sub->safe_psql('postgres',
+ "SELECT roident FROM pg_replication_origin WHERE roname = '$user_origin_name'"
+);
+
# Have the subscription in disabled state before upgrade
$old_sub->safe_psql('postgres', "ALTER SUBSCRIPTION regress_sub5 DISABLE");
@@ -378,6 +407,39 @@ regress_sub5|f|f|f),
"check that the subscription's running status, failover, and retain_dead_tuples are preserved"
);
+# Verify that the subscription related replication origins are preserved after upgrade.
+my $post_roident_rows = $new_sub->safe_psql('postgres',
+ "SELECT s.subname, o.roident
+ FROM pg_subscription s
+ JOIN pg_replication_origin o ON o.roname = 'pg_' || s.oid::text
+ ORDER BY s.subname"
+);
+
+my %post_upgrade_roident;
+for my $row (split /\n/, $post_roident_rows)
+{
+ my ($subname, $roident) = split /\|/, $row;
+ $post_upgrade_roident{$subname} = $roident;
+}
+
+# Iterate over the union of pre- and post-upgrade keys so a missing or
+# unexpected entry on either side is caught.
+my %all_subnames = (%pre_upgrade_roident, %post_upgrade_roident);
+for my $subname (sort keys %all_subnames)
+{
+ next if $subname eq $user_origin_name;
+ is($post_upgrade_roident{$subname}, $pre_upgrade_roident{$subname},
+ "roident preserved for subscription '$subname' after upgrade");
+}
+
+# Verify that user created replication origins are preserved after upgrade.
+my $post_user_roident = $new_sub->safe_psql('postgres',
+ "SELECT roident FROM pg_replication_origin WHERE roname = '$user_origin_name'"
+);
+is($post_user_roident, $pre_upgrade_roident{$user_origin_name},
+ "roident preserved for user-created origin '$user_origin_name' after upgrade"
+);
+
# Subscription relations should be preserved
$result = $new_sub->safe_psql('postgres',
"SELECT srrelid, srsubstate FROM pg_subscription_rel ORDER BY srrelid");
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 65d91e0d8de..bf56485c7f9 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -11985,10 +11985,6 @@
provolatile => 'v', proparallel => 'u', prorettype => 'void',
proargtypes => 'text oid char pg_lsn',
prosrc => 'binary_upgrade_add_sub_rel_state' },
-{ oid => '6320', descr => 'for use by pg_upgrade (remote_lsn for origin)',
- proname => 'binary_upgrade_replorigin_advance', proisstrict => 'f',
- provolatile => 'v', proparallel => 'u', prorettype => 'void',
- proargtypes => 'text pg_lsn', prosrc => 'binary_upgrade_replorigin_advance' },
{ oid => '6505', descr => 'for use by pg_upgrade (conflict detection slot)',
proname => 'binary_upgrade_create_conflict_detection_slot',
proisstrict => 'f', provolatile => 'v', proparallel => 'u',
@@ -11998,6 +11994,10 @@
proname => 'binary_upgrade_set_next_pg_subscription_oid', provolatile => 'v',
proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
prosrc => 'binary_upgrade_set_next_pg_subscription_oid' },
+{ oid => '9161', descr => 'for use by pg_upgrade (replication origin)',
+ proname => 'binary_upgrade_create_replication_origin', proisstrict => 'f',
+ provolatile => 'v', proparallel => 'u', prorettype => 'void',
+ proargtypes => 'oid text pg_lsn', prosrc => 'binary_upgrade_create_replication_origin' },
# conversion functions
{ oid => '4310', descr => 'internal conversion function for KOI8R to WIN1251',
diff --git a/src/include/replication/origin.h b/src/include/replication/origin.h
index a69faf6eaaf..11ee630fb28 100644
--- a/src/include/replication/origin.h
+++ b/src/include/replication/origin.h
@@ -14,6 +14,7 @@
#include "access/xlogdefs.h"
#include "access/xlogreader.h"
#include "catalog/pg_replication_origin.h"
+#include "utils/relcache.h"
typedef struct xl_replorigin_set
{
@@ -55,6 +56,8 @@ extern PGDLLIMPORT int max_active_replication_origins;
/* API for querying & manipulating replication origins */
extern ReplOriginId replorigin_by_name(const char *roname, bool missing_ok);
extern ReplOriginId replorigin_create(const char *roname);
+extern void replorigin_create_with_id(ReplOriginId roident, const char *roname,
+ XLogRecPtr remote_lsn, Relation rel);
extern void replorigin_drop_by_name(const char *name, bool missing_ok, bool nowait);
extern bool replorigin_by_oid(ReplOriginId roident, bool missing_ok,
char **roname);
--
2.47.3
^ permalink raw reply [nested|flat] 97+ messages in thread
end of thread, other threads:[~2026-07-09 11:37 UTC | newest]
Thread overview: 97+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]>
2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]>
2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]>
2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]>
2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]>
2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]>
2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]>
2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]>
2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]>
2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]>
2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]>
2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]>
2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]>
2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]>
2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]>
2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]>
2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]>
2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]>
2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]>
2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]>
2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]>
2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]>
2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]>
2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]>
2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]>
2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]>
2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]>
2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]>
2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]>
2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]>
2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]>
2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]>
2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]>
2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]>
2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]>
2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]>
2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]>
2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]>
2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]>
2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]>
2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]>
2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]>
2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]>
2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]>
2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]>
2021-03-16 16:29 [PATCH 02/10] review Tomas Vondra <[email protected]>
2021-03-16 16:29 [PATCH 02/10] review Tomas Vondra <[email protected]>
2021-03-16 18:26 [PATCH 04/10] review Tomas Vondra <[email protected]>
2021-03-16 18:26 [PATCH 04/10] review Tomas Vondra <[email protected]>
2021-03-17 01:19 [PATCH 06/10] review Tomas Vondra <[email protected]>
2021-03-17 01:19 [PATCH 06/10] review Tomas Vondra <[email protected]>
2021-03-17 01:36 [PATCH 08/10] review Tomas Vondra <[email protected]>
2021-03-17 01:36 [PATCH 08/10] review Tomas Vondra <[email protected]>
2026-05-18 10:42 Re: [PATCH] Preserve replication origin OIDs in pg_upgrade Ajin Cherian <[email protected]>
2026-05-22 09:46 ` Re: [PATCH] Preserve replication origin OIDs in pg_upgrade Shlok Kyal <[email protected]>
2026-05-22 10:27 ` Re: [PATCH] Preserve replication origin OIDs in pg_upgrade shveta malik <[email protected]>
2026-05-25 07:13 ` Re: [PATCH] Preserve replication origin OIDs in pg_upgrade shveta malik <[email protected]>
2026-05-28 02:48 ` Re: [PATCH] Preserve replication origin OIDs in pg_upgrade Ajin Cherian <[email protected]>
2026-06-05 10:09 ` Re: [PATCH] Preserve replication origin OIDs in pg_upgrade vignesh C <[email protected]>
2026-06-08 17:06 ` Re: [PATCH] Preserve replication origin OIDs in pg_upgrade Shlok Kyal <[email protected]>
2026-06-09 04:17 ` Re: [PATCH] Preserve replication origin OIDs in pg_upgrade shveta malik <[email protected]>
2026-06-09 11:32 ` Re: [PATCH] Preserve replication origin OIDs in pg_upgrade Shlok Kyal <[email protected]>
2026-06-17 09:19 ` Re: [PATCH] Preserve replication origin OIDs in pg_upgrade Ajin Cherian <[email protected]>
2026-06-17 10:59 ` Re: [PATCH] Preserve replication origin OIDs in pg_upgrade Nisha Moond <[email protected]>
2026-06-17 11:42 ` Re: [PATCH] Preserve replication origin OIDs in pg_upgrade Nisha Moond <[email protected]>
2026-06-18 09:16 ` Re: [PATCH] Preserve replication origin OIDs in pg_upgrade shveta malik <[email protected]>
2026-06-19 07:48 ` Re: [PATCH] Preserve replication origin OIDs in pg_upgrade Shlok Kyal <[email protected]>
2026-06-21 16:22 ` Re: [PATCH] Preserve replication origin OIDs in pg_upgrade Rui Zhao <[email protected]>
2026-06-23 11:50 ` Re: [PATCH] Preserve replication origin OIDs in pg_upgrade Ajin Cherian <[email protected]>
2026-06-23 16:07 ` Re: [PATCH] Preserve replication origin OIDs in pg_upgrade Rui Zhao <[email protected]>
2026-06-24 05:09 ` Re: [PATCH] Preserve replication origin OIDs in pg_upgrade shveta malik <[email protected]>
2026-06-24 05:21 ` Re: [PATCH] Preserve replication origin OIDs in pg_upgrade shveta malik <[email protected]>
2026-06-24 13:27 ` Re: [PATCH] Preserve replication origin OIDs in pg_upgrade Shlok Kyal <[email protected]>
2026-06-24 16:44 ` Re: [PATCH] Preserve replication origin OIDs in pg_upgrade Zsolt Parragi <[email protected]>
2026-07-01 02:49 ` Re: [PATCH] Preserve replication origin OIDs in pg_upgrade Ajin Cherian <[email protected]>
2026-07-05 16:09 ` Re: [PATCH] Preserve replication origin OIDs in pg_upgrade Rui Zhao <[email protected]>
2026-07-06 03:31 ` Re: [PATCH] Preserve replication origin OIDs in pg_upgrade shveta malik <[email protected]>
2026-07-06 05:12 ` Re: [PATCH] Preserve replication origin OIDs in pg_upgrade shveta malik <[email protected]>
2026-07-06 06:40 ` Re: [PATCH] Preserve replication origin OIDs in pg_upgrade Shlok Kyal <[email protected]>
2026-07-06 13:10 ` Re: [PATCH] Preserve replication origin OIDs in pg_upgrade Rui Zhao <[email protected]>
2026-07-08 04:07 ` Re: [PATCH] Preserve replication origin OIDs in pg_upgrade shveta malik <[email protected]>
2026-07-08 04:01 ` Re: [PATCH] Preserve replication origin OIDs in pg_upgrade shveta malik <[email protected]>
2026-07-08 06:21 ` RE: [PATCH] Preserve replication origin OIDs in pg_upgrade Hayato Kuroda (Fujitsu) <[email protected]>
2026-07-08 06:27 ` Re: [PATCH] Preserve replication origin OIDs in pg_upgrade shveta malik <[email protected]>
2026-07-08 09:09 ` Re: [PATCH] Preserve replication origin OIDs in pg_upgrade Amit Kapila <[email protected]>
2026-07-08 10:07 ` RE: [PATCH] Preserve replication origin OIDs in pg_upgrade Hayato Kuroda (Fujitsu) <[email protected]>
2026-07-08 10:42 ` Re: [PATCH] Preserve replication origin OIDs in pg_upgrade Amit Kapila <[email protected]>
2026-07-09 03:28 ` Re: [PATCH] Preserve replication origin OIDs in pg_upgrade shveta malik <[email protected]>
2026-07-09 11:37 ` Re: [PATCH] Preserve replication origin OIDs in pg_upgrade Ajin Cherian <[email protected]>
2026-06-11 06:38 ` Re: [PATCH] Preserve replication origin OIDs in pg_upgrade shveta malik <[email protected]>
2026-06-11 06:52 ` RE: [PATCH] Preserve replication origin OIDs in pg_upgrade Hayato Kuroda (Fujitsu) <[email protected]>
2026-06-11 10:37 ` Re: [PATCH] Preserve replication origin OIDs in pg_upgrade shveta malik <[email protected]>
2026-06-15 11:31 ` Re: [PATCH] Preserve replication origin OIDs in pg_upgrade Amit Kapila <[email protected]>
2026-06-16 09:20 ` Re: [PATCH] Preserve replication origin OIDs in pg_upgrade Ajin Cherian <[email protected]>
2026-06-19 03:59 ` Re: [PATCH] Preserve replication origin OIDs in pg_upgrade shveta malik <[email protected]>
2026-07-08 10:02 ` Re: [PATCH] Preserve replication origin OIDs in pg_upgrade Amit Kapila <[email protected]>
2026-06-09 04:13 ` Re: [PATCH] Preserve replication origin OIDs in pg_upgrade shveta malik <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox