public inbox for [email protected]help / color / mirror / Atom feed
[PATCH 1/2] review 56+ messages / 5 participants [nested] [flat]
* [PATCH 1/2] review @ 2021-01-09 02:17 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 56+ messages in thread From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw) --- doc/src/sgml/mvcc.sgml | 20 +++++++++++--------- doc/src/sgml/ref/create_policy.sgml | 6 +++--- doc/src/sgml/ref/merge.sgml | 12 +++++++++--- src/backend/commands/explain.c | 2 +- src/backend/executor/execMerge.c | 5 ++++- src/backend/executor/nodeModifyTable.c | 3 ++- src/backend/optimizer/prep/preptlist.c | 6 ++++++ src/backend/parser/parse_agg.c | 1 + src/backend/replication/walsender.c | 4 ++-- src/backend/rewrite/rewriteHandler.c | 4 ++++ src/backend/utils/adt/varlena.c | 3 +++ src/include/nodes/parsenodes.h | 7 ++++--- 12 files changed, 50 insertions(+), 23 deletions(-) diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml index 9ec8474185..02c9f3fdea 100644 --- a/doc/src/sgml/mvcc.sgml +++ b/doc/src/sgml/mvcc.sgml @@ -429,22 +429,24 @@ COMMIT; with both <command>INSERT</command> and <command>UPDATE</command> subcommands looks similar to <command>INSERT</command> with an <literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee - that either <command>INSERT</command> and <command>UPDATE</command> will occur. + that either <command>INSERT</command> or <command>UPDATE</command> will occur. - If MERGE attempts an UPDATE or DELETE and the row is concurrently updated + /* XXX This is a very long and hard to understand sentence :-( */ + /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */ + If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated but the join condition still passes for the current target and the current - source tuple, then MERGE will behave the same as the UPDATE or DELETE commands + source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands and perform its action on the latest version of the row, using standard - EvalPlanQual. MERGE actions can be conditional, so conditions must be + EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be re-evaluated on the latest row, starting from the first action. On the other hand, if the row is concurrently updated or deleted so that - the join condition fails, then MERGE will execute a NOT MATCHED action, if it - exists and the AND WHEN qual evaluates to true. + the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it + exists and the <literal>AND WHEN</literal> qual evaluates to true. - If MERGE attempts an INSERT and a unique index is present and a duplicate - row is concurrently inserted then a uniqueness violation is raised. MERGE - does not attempt to avoid the ERROR by attempting an UPDATE. + If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate + row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command> + does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>. </para> <para> diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml index ad20230c58..0a64674f2d 100644 --- a/doc/src/sgml/ref/create_policy.sgml +++ b/doc/src/sgml/ref/create_policy.sgml @@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable <para> No separate policy exists for <command>MERGE</command>. Instead policies - defined for <literal>SELECT</literal>, <literal>INSERT</literal>, - <literal>UPDATE</literal> and <literal>DELETE</literal> are applied - while executing MERGE, depending on the actions that are activated. + defined for <command>SELECT</command>, <command>INSERT</command>, + <command>UPDATE</command> and <command>DELETE</command> are applied + while executing <command>MERGE</command>, depending on the actions that are activated. </para> </refsect1> diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml index b2a9f67cfa..c2901b0d58 100644 --- a/doc/src/sgml/ref/merge.sgml +++ b/doc/src/sgml/ref/merge.sgml @@ -98,7 +98,7 @@ DELETE </para> <para> - There is no MERGE privilege. + There is no <literal>MERGE</literal> privilege. You must have the <literal>UPDATE</literal> privilege on the column(s) of the <replaceable class="parameter">target_table_name</replaceable> referred to in the <literal>SET</literal> clause @@ -217,6 +217,7 @@ DELETE At least one <literal>WHEN</literal> clause is required. </para> <para> + /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */ If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal> and the candidate change row matches a row in the <replaceable class="parameter">target_table_name</replaceable> @@ -242,6 +243,7 @@ DELETE clause will be activated and the corresponding action will occur for that row. The expression may not contain functions that possibly performs writes to the database. + /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */ </para> <para> A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns @@ -379,6 +381,7 @@ DELETE <para> An expression to assign to the column. The expression can use the old values of this and other columns in the table. + /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */ </para> </listitem> </varlistentry> @@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> In summary, statement triggers for an event type (say, INSERT) will be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level triggers will fire only for the one event type <emphasis>activated</emphasis>. + /* XXX What does "activated" mean? Maybe "executed" would be better? */ So a <command>MERGE</command> might fire statement triggers for both <command>UPDATE</command> and <command>INSERT</command>, even though only <command>UPDATE</command> row triggers were fired. @@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable> rows will be used to modify the target row, later attempts to modify will cause an error. This can also occur if row triggers make changes to the target table which are then subsequently modified by <command>MERGE</command>. + /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */ + /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */ If the repeated action is an <command>INSERT</command> this will cause a uniqueness violation while a repeated <command>UPDATE</command> or <command>DELETE</command> will cause a cardinality violation; the latter behavior @@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> <title>Examples</title> <para> - Perform maintenance on CustomerAccounts based upon new Transactions. + Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>. <programlisting> MERGE CustomerAccount CA @@ -599,7 +605,7 @@ WHEN MATCHED THEN DELETE; </programlisting> - The wine_stock_changes table might be, for example, a temporary table + The <literal>wine_stock_changes</literal> table might be, for example, a temporary table recently loaded into the database. </para> diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index fbaf50c258..f914a00e9f 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors, break; case CMD_MERGE: operation = "Merge"; - foperation = "Foreign Merge"; + foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */ break; default: operation = "???"; diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c index 0e245e1361..a7492a6c4b 100644 --- a/src/backend/executor/execMerge.c +++ b/src/backend/executor/execMerge.c @@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * A concurrent update can: * * 1. modify the target tuple so that it no longer satisfies the - * additional quals attached to the current WHEN MATCHED action OR + * additional quals attached to the current WHEN MATCHED action * * In this case, we are still dealing with a WHEN MATCHED case, but we * should recheck the list of WHEN MATCHED actions and choose the first @@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * tuple, so we now instead find a qualifying WHEN NOT MATCHED action to * execute. * + * XXX Hmmm, what if the updated tuple would now match one that was + * considered NOT MATCHED so far? + * * A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED. * * ExecMergeMatched takes care of following the update chain and diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index c0a97eba91..9c14709e47 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate) { /* advance to next subplan if any */ node->mt_whichplan++; - if (node->mt_whichplan < node->mt_nplans) { resultRelInfo++; @@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate) EvalPlanQualSetSlot(&node->mt_epqstate, planSlot); slot = planSlot; + /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together + * with the other MT operations? This seems a bit out of place. */ if (operation == CMD_MERGE) { ExecMerge(node, resultRelInfo, estate, slot, junkfilter); diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c index 8848e9a03f..b2543f6814 100644 --- a/src/backend/optimizer/prep/preptlist.c +++ b/src/backend/optimizer/prep/preptlist.c @@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root) tlist = expand_targetlist(tlist, command_type, result_relation, target_relation); + /* + * For MERGE we need to handle the target list for the target relation, + * and also target list for each action (only INSERT/UPDATE matter). + */ if (command_type == CMD_MERGE) { ListCell *l; @@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type, * generate a NULL for dropped columns (we want to drop any old * values). * + * XXX Should this explain why MERGE has the same logic as UPDATE? + * * When generating a NULL constant for a dropped column, we label * it INT4 (any other guaranteed-to-exist datatype would do as * well). We can't label it with the dropped column's datatype diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index f7c152beb4..5d49c8c37e 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) break; case EXPR_KIND_MERGE_WHEN_AND: if (isAgg) + /* XXX "WHEN AND" seems rather strange. ... */ err = _("aggregate functions are not allowed in WHEN AND conditions"); else err = _("grouping operations are not allowed in WHEN AND conditions"); diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index d59f4fde95..d50543b1ba 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd) /* Also update the sent position status in shared memory */ SpinLockAcquire(&MyWalSnd->mutex); + /* XXX Huh? Why does MERGE patch change walsender? */ MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush; SpinLockRelease(&MyWalSnd->mutex); @@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data) replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ? MyWalSnd->write : MyWalSnd->flush; - if (WalSndCaughtUp && - sentPtr == replicatedPtr && + if (WalSndCaughtUp && sentPtr == replicatedPtr && !pq_is_send_pending()) { QueryCompletion qc; diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c index e2706c181c..80ae946d17 100644 --- a/src/backend/rewrite/rewriteHandler.c +++ b/src/backend/rewrite/rewriteHandler.c @@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation) } } + /* * matchLocks - * match the list of locks and returns the matching rules @@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events) /* * XXX MERGE doesn't support write rules because they would violate * the SQL Standard spec and would be unclear how they should work. + * + * XXX So does't support means 'ignores'? Should that either fail + * or at least print some warning? */ if (event == CMD_MERGE) product_queries = NIL; diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 0281351dcf..7a768a7b5b 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup) int result; bool arg1_match; + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup) memset(pres, 0, sizeof(Datum)); len = VARSIZE_ANY_EXHDR(authoritative); + /* XXX Huh? Why is this in MERGE patch? */ if (len < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2) len1 = VARSIZE_ANY_EXHDR(t1); len2 = VARSIZE_ANY_EXHDR(t2); + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 3cba38044e..08fea9b8f7 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -172,6 +172,7 @@ typedef struct Query List *withCheckOptions; /* a list of WithCheckOption's (added * during rewrite) */ + /* XXX Why not mergeTragetRelation? */ int mergeTarget_relation; List *mergeSourceTargetList; List *mergeActionList; /* list of actions for MERGE (only) */ @@ -1581,11 +1582,11 @@ typedef struct UpdateStmt typedef struct MergeStmt { NodeTag type; - RangeVar *relation; /* target relation to merge into */ + RangeVar *relation; /* target relation to merge into */ Node *source_relation; /* source relation */ - Node *join_condition; /* join condition between source and target */ + Node *join_condition; /* join condition between source and target */ List *mergeWhenClauses; /* list of MergeWhenClause(es) */ - WithClause *withClause; /* WITH clause */ + WithClause *withClause; /* WITH clause */ } MergeStmt; typedef struct MergeWhenClause -- 2.26.2 --------------B1AA4DC5EAE9BAF8B146244A Content-Type: text/plain; charset=UTF-8; name="0002-fix-valgrind-failure.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-fix-valgrind-failure.txt" ^ permalink raw reply [nested|flat] 56+ messages in thread
* [PATCH 1/2] review @ 2021-01-09 02:17 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 56+ messages in thread From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw) --- doc/src/sgml/mvcc.sgml | 20 +++++++++++--------- doc/src/sgml/ref/create_policy.sgml | 6 +++--- doc/src/sgml/ref/merge.sgml | 12 +++++++++--- src/backend/commands/explain.c | 2 +- src/backend/executor/execMerge.c | 5 ++++- src/backend/executor/nodeModifyTable.c | 3 ++- src/backend/optimizer/prep/preptlist.c | 6 ++++++ src/backend/parser/parse_agg.c | 1 + src/backend/replication/walsender.c | 4 ++-- src/backend/rewrite/rewriteHandler.c | 4 ++++ src/backend/utils/adt/varlena.c | 3 +++ src/include/nodes/parsenodes.h | 7 ++++--- 12 files changed, 50 insertions(+), 23 deletions(-) diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml index 9ec8474185..02c9f3fdea 100644 --- a/doc/src/sgml/mvcc.sgml +++ b/doc/src/sgml/mvcc.sgml @@ -429,22 +429,24 @@ COMMIT; with both <command>INSERT</command> and <command>UPDATE</command> subcommands looks similar to <command>INSERT</command> with an <literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee - that either <command>INSERT</command> and <command>UPDATE</command> will occur. + that either <command>INSERT</command> or <command>UPDATE</command> will occur. - If MERGE attempts an UPDATE or DELETE and the row is concurrently updated + /* XXX This is a very long and hard to understand sentence :-( */ + /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */ + If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated but the join condition still passes for the current target and the current - source tuple, then MERGE will behave the same as the UPDATE or DELETE commands + source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands and perform its action on the latest version of the row, using standard - EvalPlanQual. MERGE actions can be conditional, so conditions must be + EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be re-evaluated on the latest row, starting from the first action. On the other hand, if the row is concurrently updated or deleted so that - the join condition fails, then MERGE will execute a NOT MATCHED action, if it - exists and the AND WHEN qual evaluates to true. + the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it + exists and the <literal>AND WHEN</literal> qual evaluates to true. - If MERGE attempts an INSERT and a unique index is present and a duplicate - row is concurrently inserted then a uniqueness violation is raised. MERGE - does not attempt to avoid the ERROR by attempting an UPDATE. + If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate + row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command> + does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>. </para> <para> diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml index ad20230c58..0a64674f2d 100644 --- a/doc/src/sgml/ref/create_policy.sgml +++ b/doc/src/sgml/ref/create_policy.sgml @@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable <para> No separate policy exists for <command>MERGE</command>. Instead policies - defined for <literal>SELECT</literal>, <literal>INSERT</literal>, - <literal>UPDATE</literal> and <literal>DELETE</literal> are applied - while executing MERGE, depending on the actions that are activated. + defined for <command>SELECT</command>, <command>INSERT</command>, + <command>UPDATE</command> and <command>DELETE</command> are applied + while executing <command>MERGE</command>, depending on the actions that are activated. </para> </refsect1> diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml index b2a9f67cfa..c2901b0d58 100644 --- a/doc/src/sgml/ref/merge.sgml +++ b/doc/src/sgml/ref/merge.sgml @@ -98,7 +98,7 @@ DELETE </para> <para> - There is no MERGE privilege. + There is no <literal>MERGE</literal> privilege. You must have the <literal>UPDATE</literal> privilege on the column(s) of the <replaceable class="parameter">target_table_name</replaceable> referred to in the <literal>SET</literal> clause @@ -217,6 +217,7 @@ DELETE At least one <literal>WHEN</literal> clause is required. </para> <para> + /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */ If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal> and the candidate change row matches a row in the <replaceable class="parameter">target_table_name</replaceable> @@ -242,6 +243,7 @@ DELETE clause will be activated and the corresponding action will occur for that row. The expression may not contain functions that possibly performs writes to the database. + /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */ </para> <para> A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns @@ -379,6 +381,7 @@ DELETE <para> An expression to assign to the column. The expression can use the old values of this and other columns in the table. + /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */ </para> </listitem> </varlistentry> @@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> In summary, statement triggers for an event type (say, INSERT) will be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level triggers will fire only for the one event type <emphasis>activated</emphasis>. + /* XXX What does "activated" mean? Maybe "executed" would be better? */ So a <command>MERGE</command> might fire statement triggers for both <command>UPDATE</command> and <command>INSERT</command>, even though only <command>UPDATE</command> row triggers were fired. @@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable> rows will be used to modify the target row, later attempts to modify will cause an error. This can also occur if row triggers make changes to the target table which are then subsequently modified by <command>MERGE</command>. + /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */ + /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */ If the repeated action is an <command>INSERT</command> this will cause a uniqueness violation while a repeated <command>UPDATE</command> or <command>DELETE</command> will cause a cardinality violation; the latter behavior @@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> <title>Examples</title> <para> - Perform maintenance on CustomerAccounts based upon new Transactions. + Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>. <programlisting> MERGE CustomerAccount CA @@ -599,7 +605,7 @@ WHEN MATCHED THEN DELETE; </programlisting> - The wine_stock_changes table might be, for example, a temporary table + The <literal>wine_stock_changes</literal> table might be, for example, a temporary table recently loaded into the database. </para> diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index fbaf50c258..f914a00e9f 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors, break; case CMD_MERGE: operation = "Merge"; - foperation = "Foreign Merge"; + foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */ break; default: operation = "???"; diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c index 0e245e1361..a7492a6c4b 100644 --- a/src/backend/executor/execMerge.c +++ b/src/backend/executor/execMerge.c @@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * A concurrent update can: * * 1. modify the target tuple so that it no longer satisfies the - * additional quals attached to the current WHEN MATCHED action OR + * additional quals attached to the current WHEN MATCHED action * * In this case, we are still dealing with a WHEN MATCHED case, but we * should recheck the list of WHEN MATCHED actions and choose the first @@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * tuple, so we now instead find a qualifying WHEN NOT MATCHED action to * execute. * + * XXX Hmmm, what if the updated tuple would now match one that was + * considered NOT MATCHED so far? + * * A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED. * * ExecMergeMatched takes care of following the update chain and diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index c0a97eba91..9c14709e47 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate) { /* advance to next subplan if any */ node->mt_whichplan++; - if (node->mt_whichplan < node->mt_nplans) { resultRelInfo++; @@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate) EvalPlanQualSetSlot(&node->mt_epqstate, planSlot); slot = planSlot; + /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together + * with the other MT operations? This seems a bit out of place. */ if (operation == CMD_MERGE) { ExecMerge(node, resultRelInfo, estate, slot, junkfilter); diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c index 8848e9a03f..b2543f6814 100644 --- a/src/backend/optimizer/prep/preptlist.c +++ b/src/backend/optimizer/prep/preptlist.c @@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root) tlist = expand_targetlist(tlist, command_type, result_relation, target_relation); + /* + * For MERGE we need to handle the target list for the target relation, + * and also target list for each action (only INSERT/UPDATE matter). + */ if (command_type == CMD_MERGE) { ListCell *l; @@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type, * generate a NULL for dropped columns (we want to drop any old * values). * + * XXX Should this explain why MERGE has the same logic as UPDATE? + * * When generating a NULL constant for a dropped column, we label * it INT4 (any other guaranteed-to-exist datatype would do as * well). We can't label it with the dropped column's datatype diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index f7c152beb4..5d49c8c37e 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) break; case EXPR_KIND_MERGE_WHEN_AND: if (isAgg) + /* XXX "WHEN AND" seems rather strange. ... */ err = _("aggregate functions are not allowed in WHEN AND conditions"); else err = _("grouping operations are not allowed in WHEN AND conditions"); diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index d59f4fde95..d50543b1ba 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd) /* Also update the sent position status in shared memory */ SpinLockAcquire(&MyWalSnd->mutex); + /* XXX Huh? Why does MERGE patch change walsender? */ MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush; SpinLockRelease(&MyWalSnd->mutex); @@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data) replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ? MyWalSnd->write : MyWalSnd->flush; - if (WalSndCaughtUp && - sentPtr == replicatedPtr && + if (WalSndCaughtUp && sentPtr == replicatedPtr && !pq_is_send_pending()) { QueryCompletion qc; diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c index e2706c181c..80ae946d17 100644 --- a/src/backend/rewrite/rewriteHandler.c +++ b/src/backend/rewrite/rewriteHandler.c @@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation) } } + /* * matchLocks - * match the list of locks and returns the matching rules @@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events) /* * XXX MERGE doesn't support write rules because they would violate * the SQL Standard spec and would be unclear how they should work. + * + * XXX So does't support means 'ignores'? Should that either fail + * or at least print some warning? */ if (event == CMD_MERGE) product_queries = NIL; diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 0281351dcf..7a768a7b5b 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup) int result; bool arg1_match; + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup) memset(pres, 0, sizeof(Datum)); len = VARSIZE_ANY_EXHDR(authoritative); + /* XXX Huh? Why is this in MERGE patch? */ if (len < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2) len1 = VARSIZE_ANY_EXHDR(t1); len2 = VARSIZE_ANY_EXHDR(t2); + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 3cba38044e..08fea9b8f7 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -172,6 +172,7 @@ typedef struct Query List *withCheckOptions; /* a list of WithCheckOption's (added * during rewrite) */ + /* XXX Why not mergeTragetRelation? */ int mergeTarget_relation; List *mergeSourceTargetList; List *mergeActionList; /* list of actions for MERGE (only) */ @@ -1581,11 +1582,11 @@ typedef struct UpdateStmt typedef struct MergeStmt { NodeTag type; - RangeVar *relation; /* target relation to merge into */ + RangeVar *relation; /* target relation to merge into */ Node *source_relation; /* source relation */ - Node *join_condition; /* join condition between source and target */ + Node *join_condition; /* join condition between source and target */ List *mergeWhenClauses; /* list of MergeWhenClause(es) */ - WithClause *withClause; /* WITH clause */ + WithClause *withClause; /* WITH clause */ } MergeStmt; typedef struct MergeWhenClause -- 2.26.2 --------------B1AA4DC5EAE9BAF8B146244A Content-Type: text/plain; charset=UTF-8; name="0002-fix-valgrind-failure.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-fix-valgrind-failure.txt" ^ permalink raw reply [nested|flat] 56+ messages in thread
* [PATCH 1/2] review @ 2021-01-09 02:17 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 56+ messages in thread From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw) --- doc/src/sgml/mvcc.sgml | 20 +++++++++++--------- doc/src/sgml/ref/create_policy.sgml | 6 +++--- doc/src/sgml/ref/merge.sgml | 12 +++++++++--- src/backend/commands/explain.c | 2 +- src/backend/executor/execMerge.c | 5 ++++- src/backend/executor/nodeModifyTable.c | 3 ++- src/backend/optimizer/prep/preptlist.c | 6 ++++++ src/backend/parser/parse_agg.c | 1 + src/backend/replication/walsender.c | 4 ++-- src/backend/rewrite/rewriteHandler.c | 4 ++++ src/backend/utils/adt/varlena.c | 3 +++ src/include/nodes/parsenodes.h | 7 ++++--- 12 files changed, 50 insertions(+), 23 deletions(-) diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml index 9ec8474185..02c9f3fdea 100644 --- a/doc/src/sgml/mvcc.sgml +++ b/doc/src/sgml/mvcc.sgml @@ -429,22 +429,24 @@ COMMIT; with both <command>INSERT</command> and <command>UPDATE</command> subcommands looks similar to <command>INSERT</command> with an <literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee - that either <command>INSERT</command> and <command>UPDATE</command> will occur. + that either <command>INSERT</command> or <command>UPDATE</command> will occur. - If MERGE attempts an UPDATE or DELETE and the row is concurrently updated + /* XXX This is a very long and hard to understand sentence :-( */ + /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */ + If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated but the join condition still passes for the current target and the current - source tuple, then MERGE will behave the same as the UPDATE or DELETE commands + source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands and perform its action on the latest version of the row, using standard - EvalPlanQual. MERGE actions can be conditional, so conditions must be + EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be re-evaluated on the latest row, starting from the first action. On the other hand, if the row is concurrently updated or deleted so that - the join condition fails, then MERGE will execute a NOT MATCHED action, if it - exists and the AND WHEN qual evaluates to true. + the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it + exists and the <literal>AND WHEN</literal> qual evaluates to true. - If MERGE attempts an INSERT and a unique index is present and a duplicate - row is concurrently inserted then a uniqueness violation is raised. MERGE - does not attempt to avoid the ERROR by attempting an UPDATE. + If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate + row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command> + does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>. </para> <para> diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml index ad20230c58..0a64674f2d 100644 --- a/doc/src/sgml/ref/create_policy.sgml +++ b/doc/src/sgml/ref/create_policy.sgml @@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable <para> No separate policy exists for <command>MERGE</command>. Instead policies - defined for <literal>SELECT</literal>, <literal>INSERT</literal>, - <literal>UPDATE</literal> and <literal>DELETE</literal> are applied - while executing MERGE, depending on the actions that are activated. + defined for <command>SELECT</command>, <command>INSERT</command>, + <command>UPDATE</command> and <command>DELETE</command> are applied + while executing <command>MERGE</command>, depending on the actions that are activated. </para> </refsect1> diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml index b2a9f67cfa..c2901b0d58 100644 --- a/doc/src/sgml/ref/merge.sgml +++ b/doc/src/sgml/ref/merge.sgml @@ -98,7 +98,7 @@ DELETE </para> <para> - There is no MERGE privilege. + There is no <literal>MERGE</literal> privilege. You must have the <literal>UPDATE</literal> privilege on the column(s) of the <replaceable class="parameter">target_table_name</replaceable> referred to in the <literal>SET</literal> clause @@ -217,6 +217,7 @@ DELETE At least one <literal>WHEN</literal> clause is required. </para> <para> + /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */ If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal> and the candidate change row matches a row in the <replaceable class="parameter">target_table_name</replaceable> @@ -242,6 +243,7 @@ DELETE clause will be activated and the corresponding action will occur for that row. The expression may not contain functions that possibly performs writes to the database. + /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */ </para> <para> A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns @@ -379,6 +381,7 @@ DELETE <para> An expression to assign to the column. The expression can use the old values of this and other columns in the table. + /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */ </para> </listitem> </varlistentry> @@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> In summary, statement triggers for an event type (say, INSERT) will be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level triggers will fire only for the one event type <emphasis>activated</emphasis>. + /* XXX What does "activated" mean? Maybe "executed" would be better? */ So a <command>MERGE</command> might fire statement triggers for both <command>UPDATE</command> and <command>INSERT</command>, even though only <command>UPDATE</command> row triggers were fired. @@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable> rows will be used to modify the target row, later attempts to modify will cause an error. This can also occur if row triggers make changes to the target table which are then subsequently modified by <command>MERGE</command>. + /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */ + /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */ If the repeated action is an <command>INSERT</command> this will cause a uniqueness violation while a repeated <command>UPDATE</command> or <command>DELETE</command> will cause a cardinality violation; the latter behavior @@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> <title>Examples</title> <para> - Perform maintenance on CustomerAccounts based upon new Transactions. + Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>. <programlisting> MERGE CustomerAccount CA @@ -599,7 +605,7 @@ WHEN MATCHED THEN DELETE; </programlisting> - The wine_stock_changes table might be, for example, a temporary table + The <literal>wine_stock_changes</literal> table might be, for example, a temporary table recently loaded into the database. </para> diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index fbaf50c258..f914a00e9f 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors, break; case CMD_MERGE: operation = "Merge"; - foperation = "Foreign Merge"; + foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */ break; default: operation = "???"; diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c index 0e245e1361..a7492a6c4b 100644 --- a/src/backend/executor/execMerge.c +++ b/src/backend/executor/execMerge.c @@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * A concurrent update can: * * 1. modify the target tuple so that it no longer satisfies the - * additional quals attached to the current WHEN MATCHED action OR + * additional quals attached to the current WHEN MATCHED action * * In this case, we are still dealing with a WHEN MATCHED case, but we * should recheck the list of WHEN MATCHED actions and choose the first @@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * tuple, so we now instead find a qualifying WHEN NOT MATCHED action to * execute. * + * XXX Hmmm, what if the updated tuple would now match one that was + * considered NOT MATCHED so far? + * * A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED. * * ExecMergeMatched takes care of following the update chain and diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index c0a97eba91..9c14709e47 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate) { /* advance to next subplan if any */ node->mt_whichplan++; - if (node->mt_whichplan < node->mt_nplans) { resultRelInfo++; @@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate) EvalPlanQualSetSlot(&node->mt_epqstate, planSlot); slot = planSlot; + /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together + * with the other MT operations? This seems a bit out of place. */ if (operation == CMD_MERGE) { ExecMerge(node, resultRelInfo, estate, slot, junkfilter); diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c index 8848e9a03f..b2543f6814 100644 --- a/src/backend/optimizer/prep/preptlist.c +++ b/src/backend/optimizer/prep/preptlist.c @@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root) tlist = expand_targetlist(tlist, command_type, result_relation, target_relation); + /* + * For MERGE we need to handle the target list for the target relation, + * and also target list for each action (only INSERT/UPDATE matter). + */ if (command_type == CMD_MERGE) { ListCell *l; @@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type, * generate a NULL for dropped columns (we want to drop any old * values). * + * XXX Should this explain why MERGE has the same logic as UPDATE? + * * When generating a NULL constant for a dropped column, we label * it INT4 (any other guaranteed-to-exist datatype would do as * well). We can't label it with the dropped column's datatype diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index f7c152beb4..5d49c8c37e 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) break; case EXPR_KIND_MERGE_WHEN_AND: if (isAgg) + /* XXX "WHEN AND" seems rather strange. ... */ err = _("aggregate functions are not allowed in WHEN AND conditions"); else err = _("grouping operations are not allowed in WHEN AND conditions"); diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index d59f4fde95..d50543b1ba 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd) /* Also update the sent position status in shared memory */ SpinLockAcquire(&MyWalSnd->mutex); + /* XXX Huh? Why does MERGE patch change walsender? */ MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush; SpinLockRelease(&MyWalSnd->mutex); @@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data) replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ? MyWalSnd->write : MyWalSnd->flush; - if (WalSndCaughtUp && - sentPtr == replicatedPtr && + if (WalSndCaughtUp && sentPtr == replicatedPtr && !pq_is_send_pending()) { QueryCompletion qc; diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c index e2706c181c..80ae946d17 100644 --- a/src/backend/rewrite/rewriteHandler.c +++ b/src/backend/rewrite/rewriteHandler.c @@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation) } } + /* * matchLocks - * match the list of locks and returns the matching rules @@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events) /* * XXX MERGE doesn't support write rules because they would violate * the SQL Standard spec and would be unclear how they should work. + * + * XXX So does't support means 'ignores'? Should that either fail + * or at least print some warning? */ if (event == CMD_MERGE) product_queries = NIL; diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 0281351dcf..7a768a7b5b 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup) int result; bool arg1_match; + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup) memset(pres, 0, sizeof(Datum)); len = VARSIZE_ANY_EXHDR(authoritative); + /* XXX Huh? Why is this in MERGE patch? */ if (len < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2) len1 = VARSIZE_ANY_EXHDR(t1); len2 = VARSIZE_ANY_EXHDR(t2); + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 3cba38044e..08fea9b8f7 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -172,6 +172,7 @@ typedef struct Query List *withCheckOptions; /* a list of WithCheckOption's (added * during rewrite) */ + /* XXX Why not mergeTragetRelation? */ int mergeTarget_relation; List *mergeSourceTargetList; List *mergeActionList; /* list of actions for MERGE (only) */ @@ -1581,11 +1582,11 @@ typedef struct UpdateStmt typedef struct MergeStmt { NodeTag type; - RangeVar *relation; /* target relation to merge into */ + RangeVar *relation; /* target relation to merge into */ Node *source_relation; /* source relation */ - Node *join_condition; /* join condition between source and target */ + Node *join_condition; /* join condition between source and target */ List *mergeWhenClauses; /* list of MergeWhenClause(es) */ - WithClause *withClause; /* WITH clause */ + WithClause *withClause; /* WITH clause */ } MergeStmt; typedef struct MergeWhenClause -- 2.26.2 --------------B1AA4DC5EAE9BAF8B146244A Content-Type: text/plain; charset=UTF-8; name="0002-fix-valgrind-failure.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-fix-valgrind-failure.txt" ^ permalink raw reply [nested|flat] 56+ messages in thread
* [PATCH 1/2] review @ 2021-01-09 02:17 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 56+ messages in thread From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw) --- doc/src/sgml/mvcc.sgml | 20 +++++++++++--------- doc/src/sgml/ref/create_policy.sgml | 6 +++--- doc/src/sgml/ref/merge.sgml | 12 +++++++++--- src/backend/commands/explain.c | 2 +- src/backend/executor/execMerge.c | 5 ++++- src/backend/executor/nodeModifyTable.c | 3 ++- src/backend/optimizer/prep/preptlist.c | 6 ++++++ src/backend/parser/parse_agg.c | 1 + src/backend/replication/walsender.c | 4 ++-- src/backend/rewrite/rewriteHandler.c | 4 ++++ src/backend/utils/adt/varlena.c | 3 +++ src/include/nodes/parsenodes.h | 7 ++++--- 12 files changed, 50 insertions(+), 23 deletions(-) diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml index 9ec8474185..02c9f3fdea 100644 --- a/doc/src/sgml/mvcc.sgml +++ b/doc/src/sgml/mvcc.sgml @@ -429,22 +429,24 @@ COMMIT; with both <command>INSERT</command> and <command>UPDATE</command> subcommands looks similar to <command>INSERT</command> with an <literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee - that either <command>INSERT</command> and <command>UPDATE</command> will occur. + that either <command>INSERT</command> or <command>UPDATE</command> will occur. - If MERGE attempts an UPDATE or DELETE and the row is concurrently updated + /* XXX This is a very long and hard to understand sentence :-( */ + /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */ + If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated but the join condition still passes for the current target and the current - source tuple, then MERGE will behave the same as the UPDATE or DELETE commands + source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands and perform its action on the latest version of the row, using standard - EvalPlanQual. MERGE actions can be conditional, so conditions must be + EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be re-evaluated on the latest row, starting from the first action. On the other hand, if the row is concurrently updated or deleted so that - the join condition fails, then MERGE will execute a NOT MATCHED action, if it - exists and the AND WHEN qual evaluates to true. + the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it + exists and the <literal>AND WHEN</literal> qual evaluates to true. - If MERGE attempts an INSERT and a unique index is present and a duplicate - row is concurrently inserted then a uniqueness violation is raised. MERGE - does not attempt to avoid the ERROR by attempting an UPDATE. + If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate + row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command> + does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>. </para> <para> diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml index ad20230c58..0a64674f2d 100644 --- a/doc/src/sgml/ref/create_policy.sgml +++ b/doc/src/sgml/ref/create_policy.sgml @@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable <para> No separate policy exists for <command>MERGE</command>. Instead policies - defined for <literal>SELECT</literal>, <literal>INSERT</literal>, - <literal>UPDATE</literal> and <literal>DELETE</literal> are applied - while executing MERGE, depending on the actions that are activated. + defined for <command>SELECT</command>, <command>INSERT</command>, + <command>UPDATE</command> and <command>DELETE</command> are applied + while executing <command>MERGE</command>, depending on the actions that are activated. </para> </refsect1> diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml index b2a9f67cfa..c2901b0d58 100644 --- a/doc/src/sgml/ref/merge.sgml +++ b/doc/src/sgml/ref/merge.sgml @@ -98,7 +98,7 @@ DELETE </para> <para> - There is no MERGE privilege. + There is no <literal>MERGE</literal> privilege. You must have the <literal>UPDATE</literal> privilege on the column(s) of the <replaceable class="parameter">target_table_name</replaceable> referred to in the <literal>SET</literal> clause @@ -217,6 +217,7 @@ DELETE At least one <literal>WHEN</literal> clause is required. </para> <para> + /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */ If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal> and the candidate change row matches a row in the <replaceable class="parameter">target_table_name</replaceable> @@ -242,6 +243,7 @@ DELETE clause will be activated and the corresponding action will occur for that row. The expression may not contain functions that possibly performs writes to the database. + /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */ </para> <para> A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns @@ -379,6 +381,7 @@ DELETE <para> An expression to assign to the column. The expression can use the old values of this and other columns in the table. + /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */ </para> </listitem> </varlistentry> @@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> In summary, statement triggers for an event type (say, INSERT) will be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level triggers will fire only for the one event type <emphasis>activated</emphasis>. + /* XXX What does "activated" mean? Maybe "executed" would be better? */ So a <command>MERGE</command> might fire statement triggers for both <command>UPDATE</command> and <command>INSERT</command>, even though only <command>UPDATE</command> row triggers were fired. @@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable> rows will be used to modify the target row, later attempts to modify will cause an error. This can also occur if row triggers make changes to the target table which are then subsequently modified by <command>MERGE</command>. + /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */ + /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */ If the repeated action is an <command>INSERT</command> this will cause a uniqueness violation while a repeated <command>UPDATE</command> or <command>DELETE</command> will cause a cardinality violation; the latter behavior @@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> <title>Examples</title> <para> - Perform maintenance on CustomerAccounts based upon new Transactions. + Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>. <programlisting> MERGE CustomerAccount CA @@ -599,7 +605,7 @@ WHEN MATCHED THEN DELETE; </programlisting> - The wine_stock_changes table might be, for example, a temporary table + The <literal>wine_stock_changes</literal> table might be, for example, a temporary table recently loaded into the database. </para> diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index fbaf50c258..f914a00e9f 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors, break; case CMD_MERGE: operation = "Merge"; - foperation = "Foreign Merge"; + foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */ break; default: operation = "???"; diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c index 0e245e1361..a7492a6c4b 100644 --- a/src/backend/executor/execMerge.c +++ b/src/backend/executor/execMerge.c @@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * A concurrent update can: * * 1. modify the target tuple so that it no longer satisfies the - * additional quals attached to the current WHEN MATCHED action OR + * additional quals attached to the current WHEN MATCHED action * * In this case, we are still dealing with a WHEN MATCHED case, but we * should recheck the list of WHEN MATCHED actions and choose the first @@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * tuple, so we now instead find a qualifying WHEN NOT MATCHED action to * execute. * + * XXX Hmmm, what if the updated tuple would now match one that was + * considered NOT MATCHED so far? + * * A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED. * * ExecMergeMatched takes care of following the update chain and diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index c0a97eba91..9c14709e47 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate) { /* advance to next subplan if any */ node->mt_whichplan++; - if (node->mt_whichplan < node->mt_nplans) { resultRelInfo++; @@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate) EvalPlanQualSetSlot(&node->mt_epqstate, planSlot); slot = planSlot; + /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together + * with the other MT operations? This seems a bit out of place. */ if (operation == CMD_MERGE) { ExecMerge(node, resultRelInfo, estate, slot, junkfilter); diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c index 8848e9a03f..b2543f6814 100644 --- a/src/backend/optimizer/prep/preptlist.c +++ b/src/backend/optimizer/prep/preptlist.c @@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root) tlist = expand_targetlist(tlist, command_type, result_relation, target_relation); + /* + * For MERGE we need to handle the target list for the target relation, + * and also target list for each action (only INSERT/UPDATE matter). + */ if (command_type == CMD_MERGE) { ListCell *l; @@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type, * generate a NULL for dropped columns (we want to drop any old * values). * + * XXX Should this explain why MERGE has the same logic as UPDATE? + * * When generating a NULL constant for a dropped column, we label * it INT4 (any other guaranteed-to-exist datatype would do as * well). We can't label it with the dropped column's datatype diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index f7c152beb4..5d49c8c37e 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) break; case EXPR_KIND_MERGE_WHEN_AND: if (isAgg) + /* XXX "WHEN AND" seems rather strange. ... */ err = _("aggregate functions are not allowed in WHEN AND conditions"); else err = _("grouping operations are not allowed in WHEN AND conditions"); diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index d59f4fde95..d50543b1ba 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd) /* Also update the sent position status in shared memory */ SpinLockAcquire(&MyWalSnd->mutex); + /* XXX Huh? Why does MERGE patch change walsender? */ MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush; SpinLockRelease(&MyWalSnd->mutex); @@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data) replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ? MyWalSnd->write : MyWalSnd->flush; - if (WalSndCaughtUp && - sentPtr == replicatedPtr && + if (WalSndCaughtUp && sentPtr == replicatedPtr && !pq_is_send_pending()) { QueryCompletion qc; diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c index e2706c181c..80ae946d17 100644 --- a/src/backend/rewrite/rewriteHandler.c +++ b/src/backend/rewrite/rewriteHandler.c @@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation) } } + /* * matchLocks - * match the list of locks and returns the matching rules @@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events) /* * XXX MERGE doesn't support write rules because they would violate * the SQL Standard spec and would be unclear how they should work. + * + * XXX So does't support means 'ignores'? Should that either fail + * or at least print some warning? */ if (event == CMD_MERGE) product_queries = NIL; diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 0281351dcf..7a768a7b5b 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup) int result; bool arg1_match; + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup) memset(pres, 0, sizeof(Datum)); len = VARSIZE_ANY_EXHDR(authoritative); + /* XXX Huh? Why is this in MERGE patch? */ if (len < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2) len1 = VARSIZE_ANY_EXHDR(t1); len2 = VARSIZE_ANY_EXHDR(t2); + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 3cba38044e..08fea9b8f7 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -172,6 +172,7 @@ typedef struct Query List *withCheckOptions; /* a list of WithCheckOption's (added * during rewrite) */ + /* XXX Why not mergeTragetRelation? */ int mergeTarget_relation; List *mergeSourceTargetList; List *mergeActionList; /* list of actions for MERGE (only) */ @@ -1581,11 +1582,11 @@ typedef struct UpdateStmt typedef struct MergeStmt { NodeTag type; - RangeVar *relation; /* target relation to merge into */ + RangeVar *relation; /* target relation to merge into */ Node *source_relation; /* source relation */ - Node *join_condition; /* join condition between source and target */ + Node *join_condition; /* join condition between source and target */ List *mergeWhenClauses; /* list of MergeWhenClause(es) */ - WithClause *withClause; /* WITH clause */ + WithClause *withClause; /* WITH clause */ } MergeStmt; typedef struct MergeWhenClause -- 2.26.2 --------------B1AA4DC5EAE9BAF8B146244A Content-Type: text/plain; charset=UTF-8; name="0002-fix-valgrind-failure.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-fix-valgrind-failure.txt" ^ permalink raw reply [nested|flat] 56+ messages in thread
* [PATCH 1/2] review @ 2021-01-09 02:17 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 56+ messages in thread From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw) --- doc/src/sgml/mvcc.sgml | 20 +++++++++++--------- doc/src/sgml/ref/create_policy.sgml | 6 +++--- doc/src/sgml/ref/merge.sgml | 12 +++++++++--- src/backend/commands/explain.c | 2 +- src/backend/executor/execMerge.c | 5 ++++- src/backend/executor/nodeModifyTable.c | 3 ++- src/backend/optimizer/prep/preptlist.c | 6 ++++++ src/backend/parser/parse_agg.c | 1 + src/backend/replication/walsender.c | 4 ++-- src/backend/rewrite/rewriteHandler.c | 4 ++++ src/backend/utils/adt/varlena.c | 3 +++ src/include/nodes/parsenodes.h | 7 ++++--- 12 files changed, 50 insertions(+), 23 deletions(-) diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml index 9ec8474185..02c9f3fdea 100644 --- a/doc/src/sgml/mvcc.sgml +++ b/doc/src/sgml/mvcc.sgml @@ -429,22 +429,24 @@ COMMIT; with both <command>INSERT</command> and <command>UPDATE</command> subcommands looks similar to <command>INSERT</command> with an <literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee - that either <command>INSERT</command> and <command>UPDATE</command> will occur. + that either <command>INSERT</command> or <command>UPDATE</command> will occur. - If MERGE attempts an UPDATE or DELETE and the row is concurrently updated + /* XXX This is a very long and hard to understand sentence :-( */ + /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */ + If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated but the join condition still passes for the current target and the current - source tuple, then MERGE will behave the same as the UPDATE or DELETE commands + source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands and perform its action on the latest version of the row, using standard - EvalPlanQual. MERGE actions can be conditional, so conditions must be + EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be re-evaluated on the latest row, starting from the first action. On the other hand, if the row is concurrently updated or deleted so that - the join condition fails, then MERGE will execute a NOT MATCHED action, if it - exists and the AND WHEN qual evaluates to true. + the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it + exists and the <literal>AND WHEN</literal> qual evaluates to true. - If MERGE attempts an INSERT and a unique index is present and a duplicate - row is concurrently inserted then a uniqueness violation is raised. MERGE - does not attempt to avoid the ERROR by attempting an UPDATE. + If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate + row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command> + does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>. </para> <para> diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml index ad20230c58..0a64674f2d 100644 --- a/doc/src/sgml/ref/create_policy.sgml +++ b/doc/src/sgml/ref/create_policy.sgml @@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable <para> No separate policy exists for <command>MERGE</command>. Instead policies - defined for <literal>SELECT</literal>, <literal>INSERT</literal>, - <literal>UPDATE</literal> and <literal>DELETE</literal> are applied - while executing MERGE, depending on the actions that are activated. + defined for <command>SELECT</command>, <command>INSERT</command>, + <command>UPDATE</command> and <command>DELETE</command> are applied + while executing <command>MERGE</command>, depending on the actions that are activated. </para> </refsect1> diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml index b2a9f67cfa..c2901b0d58 100644 --- a/doc/src/sgml/ref/merge.sgml +++ b/doc/src/sgml/ref/merge.sgml @@ -98,7 +98,7 @@ DELETE </para> <para> - There is no MERGE privilege. + There is no <literal>MERGE</literal> privilege. You must have the <literal>UPDATE</literal> privilege on the column(s) of the <replaceable class="parameter">target_table_name</replaceable> referred to in the <literal>SET</literal> clause @@ -217,6 +217,7 @@ DELETE At least one <literal>WHEN</literal> clause is required. </para> <para> + /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */ If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal> and the candidate change row matches a row in the <replaceable class="parameter">target_table_name</replaceable> @@ -242,6 +243,7 @@ DELETE clause will be activated and the corresponding action will occur for that row. The expression may not contain functions that possibly performs writes to the database. + /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */ </para> <para> A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns @@ -379,6 +381,7 @@ DELETE <para> An expression to assign to the column. The expression can use the old values of this and other columns in the table. + /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */ </para> </listitem> </varlistentry> @@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> In summary, statement triggers for an event type (say, INSERT) will be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level triggers will fire only for the one event type <emphasis>activated</emphasis>. + /* XXX What does "activated" mean? Maybe "executed" would be better? */ So a <command>MERGE</command> might fire statement triggers for both <command>UPDATE</command> and <command>INSERT</command>, even though only <command>UPDATE</command> row triggers were fired. @@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable> rows will be used to modify the target row, later attempts to modify will cause an error. This can also occur if row triggers make changes to the target table which are then subsequently modified by <command>MERGE</command>. + /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */ + /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */ If the repeated action is an <command>INSERT</command> this will cause a uniqueness violation while a repeated <command>UPDATE</command> or <command>DELETE</command> will cause a cardinality violation; the latter behavior @@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> <title>Examples</title> <para> - Perform maintenance on CustomerAccounts based upon new Transactions. + Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>. <programlisting> MERGE CustomerAccount CA @@ -599,7 +605,7 @@ WHEN MATCHED THEN DELETE; </programlisting> - The wine_stock_changes table might be, for example, a temporary table + The <literal>wine_stock_changes</literal> table might be, for example, a temporary table recently loaded into the database. </para> diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index fbaf50c258..f914a00e9f 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors, break; case CMD_MERGE: operation = "Merge"; - foperation = "Foreign Merge"; + foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */ break; default: operation = "???"; diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c index 0e245e1361..a7492a6c4b 100644 --- a/src/backend/executor/execMerge.c +++ b/src/backend/executor/execMerge.c @@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * A concurrent update can: * * 1. modify the target tuple so that it no longer satisfies the - * additional quals attached to the current WHEN MATCHED action OR + * additional quals attached to the current WHEN MATCHED action * * In this case, we are still dealing with a WHEN MATCHED case, but we * should recheck the list of WHEN MATCHED actions and choose the first @@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * tuple, so we now instead find a qualifying WHEN NOT MATCHED action to * execute. * + * XXX Hmmm, what if the updated tuple would now match one that was + * considered NOT MATCHED so far? + * * A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED. * * ExecMergeMatched takes care of following the update chain and diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index c0a97eba91..9c14709e47 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate) { /* advance to next subplan if any */ node->mt_whichplan++; - if (node->mt_whichplan < node->mt_nplans) { resultRelInfo++; @@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate) EvalPlanQualSetSlot(&node->mt_epqstate, planSlot); slot = planSlot; + /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together + * with the other MT operations? This seems a bit out of place. */ if (operation == CMD_MERGE) { ExecMerge(node, resultRelInfo, estate, slot, junkfilter); diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c index 8848e9a03f..b2543f6814 100644 --- a/src/backend/optimizer/prep/preptlist.c +++ b/src/backend/optimizer/prep/preptlist.c @@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root) tlist = expand_targetlist(tlist, command_type, result_relation, target_relation); + /* + * For MERGE we need to handle the target list for the target relation, + * and also target list for each action (only INSERT/UPDATE matter). + */ if (command_type == CMD_MERGE) { ListCell *l; @@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type, * generate a NULL for dropped columns (we want to drop any old * values). * + * XXX Should this explain why MERGE has the same logic as UPDATE? + * * When generating a NULL constant for a dropped column, we label * it INT4 (any other guaranteed-to-exist datatype would do as * well). We can't label it with the dropped column's datatype diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index f7c152beb4..5d49c8c37e 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) break; case EXPR_KIND_MERGE_WHEN_AND: if (isAgg) + /* XXX "WHEN AND" seems rather strange. ... */ err = _("aggregate functions are not allowed in WHEN AND conditions"); else err = _("grouping operations are not allowed in WHEN AND conditions"); diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index d59f4fde95..d50543b1ba 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd) /* Also update the sent position status in shared memory */ SpinLockAcquire(&MyWalSnd->mutex); + /* XXX Huh? Why does MERGE patch change walsender? */ MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush; SpinLockRelease(&MyWalSnd->mutex); @@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data) replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ? MyWalSnd->write : MyWalSnd->flush; - if (WalSndCaughtUp && - sentPtr == replicatedPtr && + if (WalSndCaughtUp && sentPtr == replicatedPtr && !pq_is_send_pending()) { QueryCompletion qc; diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c index e2706c181c..80ae946d17 100644 --- a/src/backend/rewrite/rewriteHandler.c +++ b/src/backend/rewrite/rewriteHandler.c @@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation) } } + /* * matchLocks - * match the list of locks and returns the matching rules @@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events) /* * XXX MERGE doesn't support write rules because they would violate * the SQL Standard spec and would be unclear how they should work. + * + * XXX So does't support means 'ignores'? Should that either fail + * or at least print some warning? */ if (event == CMD_MERGE) product_queries = NIL; diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 0281351dcf..7a768a7b5b 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup) int result; bool arg1_match; + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup) memset(pres, 0, sizeof(Datum)); len = VARSIZE_ANY_EXHDR(authoritative); + /* XXX Huh? Why is this in MERGE patch? */ if (len < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2) len1 = VARSIZE_ANY_EXHDR(t1); len2 = VARSIZE_ANY_EXHDR(t2); + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 3cba38044e..08fea9b8f7 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -172,6 +172,7 @@ typedef struct Query List *withCheckOptions; /* a list of WithCheckOption's (added * during rewrite) */ + /* XXX Why not mergeTragetRelation? */ int mergeTarget_relation; List *mergeSourceTargetList; List *mergeActionList; /* list of actions for MERGE (only) */ @@ -1581,11 +1582,11 @@ typedef struct UpdateStmt typedef struct MergeStmt { NodeTag type; - RangeVar *relation; /* target relation to merge into */ + RangeVar *relation; /* target relation to merge into */ Node *source_relation; /* source relation */ - Node *join_condition; /* join condition between source and target */ + Node *join_condition; /* join condition between source and target */ List *mergeWhenClauses; /* list of MergeWhenClause(es) */ - WithClause *withClause; /* WITH clause */ + WithClause *withClause; /* WITH clause */ } MergeStmt; typedef struct MergeWhenClause -- 2.26.2 --------------B1AA4DC5EAE9BAF8B146244A Content-Type: text/plain; charset=UTF-8; name="0002-fix-valgrind-failure.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-fix-valgrind-failure.txt" ^ permalink raw reply [nested|flat] 56+ messages in thread
* [PATCH 1/2] review @ 2021-01-09 02:17 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 56+ messages in thread From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw) --- doc/src/sgml/mvcc.sgml | 20 +++++++++++--------- doc/src/sgml/ref/create_policy.sgml | 6 +++--- doc/src/sgml/ref/merge.sgml | 12 +++++++++--- src/backend/commands/explain.c | 2 +- src/backend/executor/execMerge.c | 5 ++++- src/backend/executor/nodeModifyTable.c | 3 ++- src/backend/optimizer/prep/preptlist.c | 6 ++++++ src/backend/parser/parse_agg.c | 1 + src/backend/replication/walsender.c | 4 ++-- src/backend/rewrite/rewriteHandler.c | 4 ++++ src/backend/utils/adt/varlena.c | 3 +++ src/include/nodes/parsenodes.h | 7 ++++--- 12 files changed, 50 insertions(+), 23 deletions(-) diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml index 9ec8474185..02c9f3fdea 100644 --- a/doc/src/sgml/mvcc.sgml +++ b/doc/src/sgml/mvcc.sgml @@ -429,22 +429,24 @@ COMMIT; with both <command>INSERT</command> and <command>UPDATE</command> subcommands looks similar to <command>INSERT</command> with an <literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee - that either <command>INSERT</command> and <command>UPDATE</command> will occur. + that either <command>INSERT</command> or <command>UPDATE</command> will occur. - If MERGE attempts an UPDATE or DELETE and the row is concurrently updated + /* XXX This is a very long and hard to understand sentence :-( */ + /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */ + If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated but the join condition still passes for the current target and the current - source tuple, then MERGE will behave the same as the UPDATE or DELETE commands + source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands and perform its action on the latest version of the row, using standard - EvalPlanQual. MERGE actions can be conditional, so conditions must be + EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be re-evaluated on the latest row, starting from the first action. On the other hand, if the row is concurrently updated or deleted so that - the join condition fails, then MERGE will execute a NOT MATCHED action, if it - exists and the AND WHEN qual evaluates to true. + the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it + exists and the <literal>AND WHEN</literal> qual evaluates to true. - If MERGE attempts an INSERT and a unique index is present and a duplicate - row is concurrently inserted then a uniqueness violation is raised. MERGE - does not attempt to avoid the ERROR by attempting an UPDATE. + If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate + row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command> + does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>. </para> <para> diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml index ad20230c58..0a64674f2d 100644 --- a/doc/src/sgml/ref/create_policy.sgml +++ b/doc/src/sgml/ref/create_policy.sgml @@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable <para> No separate policy exists for <command>MERGE</command>. Instead policies - defined for <literal>SELECT</literal>, <literal>INSERT</literal>, - <literal>UPDATE</literal> and <literal>DELETE</literal> are applied - while executing MERGE, depending on the actions that are activated. + defined for <command>SELECT</command>, <command>INSERT</command>, + <command>UPDATE</command> and <command>DELETE</command> are applied + while executing <command>MERGE</command>, depending on the actions that are activated. </para> </refsect1> diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml index b2a9f67cfa..c2901b0d58 100644 --- a/doc/src/sgml/ref/merge.sgml +++ b/doc/src/sgml/ref/merge.sgml @@ -98,7 +98,7 @@ DELETE </para> <para> - There is no MERGE privilege. + There is no <literal>MERGE</literal> privilege. You must have the <literal>UPDATE</literal> privilege on the column(s) of the <replaceable class="parameter">target_table_name</replaceable> referred to in the <literal>SET</literal> clause @@ -217,6 +217,7 @@ DELETE At least one <literal>WHEN</literal> clause is required. </para> <para> + /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */ If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal> and the candidate change row matches a row in the <replaceable class="parameter">target_table_name</replaceable> @@ -242,6 +243,7 @@ DELETE clause will be activated and the corresponding action will occur for that row. The expression may not contain functions that possibly performs writes to the database. + /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */ </para> <para> A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns @@ -379,6 +381,7 @@ DELETE <para> An expression to assign to the column. The expression can use the old values of this and other columns in the table. + /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */ </para> </listitem> </varlistentry> @@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> In summary, statement triggers for an event type (say, INSERT) will be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level triggers will fire only for the one event type <emphasis>activated</emphasis>. + /* XXX What does "activated" mean? Maybe "executed" would be better? */ So a <command>MERGE</command> might fire statement triggers for both <command>UPDATE</command> and <command>INSERT</command>, even though only <command>UPDATE</command> row triggers were fired. @@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable> rows will be used to modify the target row, later attempts to modify will cause an error. This can also occur if row triggers make changes to the target table which are then subsequently modified by <command>MERGE</command>. + /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */ + /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */ If the repeated action is an <command>INSERT</command> this will cause a uniqueness violation while a repeated <command>UPDATE</command> or <command>DELETE</command> will cause a cardinality violation; the latter behavior @@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> <title>Examples</title> <para> - Perform maintenance on CustomerAccounts based upon new Transactions. + Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>. <programlisting> MERGE CustomerAccount CA @@ -599,7 +605,7 @@ WHEN MATCHED THEN DELETE; </programlisting> - The wine_stock_changes table might be, for example, a temporary table + The <literal>wine_stock_changes</literal> table might be, for example, a temporary table recently loaded into the database. </para> diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index fbaf50c258..f914a00e9f 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors, break; case CMD_MERGE: operation = "Merge"; - foperation = "Foreign Merge"; + foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */ break; default: operation = "???"; diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c index 0e245e1361..a7492a6c4b 100644 --- a/src/backend/executor/execMerge.c +++ b/src/backend/executor/execMerge.c @@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * A concurrent update can: * * 1. modify the target tuple so that it no longer satisfies the - * additional quals attached to the current WHEN MATCHED action OR + * additional quals attached to the current WHEN MATCHED action * * In this case, we are still dealing with a WHEN MATCHED case, but we * should recheck the list of WHEN MATCHED actions and choose the first @@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * tuple, so we now instead find a qualifying WHEN NOT MATCHED action to * execute. * + * XXX Hmmm, what if the updated tuple would now match one that was + * considered NOT MATCHED so far? + * * A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED. * * ExecMergeMatched takes care of following the update chain and diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index c0a97eba91..9c14709e47 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate) { /* advance to next subplan if any */ node->mt_whichplan++; - if (node->mt_whichplan < node->mt_nplans) { resultRelInfo++; @@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate) EvalPlanQualSetSlot(&node->mt_epqstate, planSlot); slot = planSlot; + /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together + * with the other MT operations? This seems a bit out of place. */ if (operation == CMD_MERGE) { ExecMerge(node, resultRelInfo, estate, slot, junkfilter); diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c index 8848e9a03f..b2543f6814 100644 --- a/src/backend/optimizer/prep/preptlist.c +++ b/src/backend/optimizer/prep/preptlist.c @@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root) tlist = expand_targetlist(tlist, command_type, result_relation, target_relation); + /* + * For MERGE we need to handle the target list for the target relation, + * and also target list for each action (only INSERT/UPDATE matter). + */ if (command_type == CMD_MERGE) { ListCell *l; @@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type, * generate a NULL for dropped columns (we want to drop any old * values). * + * XXX Should this explain why MERGE has the same logic as UPDATE? + * * When generating a NULL constant for a dropped column, we label * it INT4 (any other guaranteed-to-exist datatype would do as * well). We can't label it with the dropped column's datatype diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index f7c152beb4..5d49c8c37e 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) break; case EXPR_KIND_MERGE_WHEN_AND: if (isAgg) + /* XXX "WHEN AND" seems rather strange. ... */ err = _("aggregate functions are not allowed in WHEN AND conditions"); else err = _("grouping operations are not allowed in WHEN AND conditions"); diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index d59f4fde95..d50543b1ba 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd) /* Also update the sent position status in shared memory */ SpinLockAcquire(&MyWalSnd->mutex); + /* XXX Huh? Why does MERGE patch change walsender? */ MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush; SpinLockRelease(&MyWalSnd->mutex); @@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data) replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ? MyWalSnd->write : MyWalSnd->flush; - if (WalSndCaughtUp && - sentPtr == replicatedPtr && + if (WalSndCaughtUp && sentPtr == replicatedPtr && !pq_is_send_pending()) { QueryCompletion qc; diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c index e2706c181c..80ae946d17 100644 --- a/src/backend/rewrite/rewriteHandler.c +++ b/src/backend/rewrite/rewriteHandler.c @@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation) } } + /* * matchLocks - * match the list of locks and returns the matching rules @@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events) /* * XXX MERGE doesn't support write rules because they would violate * the SQL Standard spec and would be unclear how they should work. + * + * XXX So does't support means 'ignores'? Should that either fail + * or at least print some warning? */ if (event == CMD_MERGE) product_queries = NIL; diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 0281351dcf..7a768a7b5b 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup) int result; bool arg1_match; + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup) memset(pres, 0, sizeof(Datum)); len = VARSIZE_ANY_EXHDR(authoritative); + /* XXX Huh? Why is this in MERGE patch? */ if (len < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2) len1 = VARSIZE_ANY_EXHDR(t1); len2 = VARSIZE_ANY_EXHDR(t2); + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 3cba38044e..08fea9b8f7 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -172,6 +172,7 @@ typedef struct Query List *withCheckOptions; /* a list of WithCheckOption's (added * during rewrite) */ + /* XXX Why not mergeTragetRelation? */ int mergeTarget_relation; List *mergeSourceTargetList; List *mergeActionList; /* list of actions for MERGE (only) */ @@ -1581,11 +1582,11 @@ typedef struct UpdateStmt typedef struct MergeStmt { NodeTag type; - RangeVar *relation; /* target relation to merge into */ + RangeVar *relation; /* target relation to merge into */ Node *source_relation; /* source relation */ - Node *join_condition; /* join condition between source and target */ + Node *join_condition; /* join condition between source and target */ List *mergeWhenClauses; /* list of MergeWhenClause(es) */ - WithClause *withClause; /* WITH clause */ + WithClause *withClause; /* WITH clause */ } MergeStmt; typedef struct MergeWhenClause -- 2.26.2 --------------B1AA4DC5EAE9BAF8B146244A Content-Type: text/plain; charset=UTF-8; name="0002-fix-valgrind-failure.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-fix-valgrind-failure.txt" ^ permalink raw reply [nested|flat] 56+ messages in thread
* [PATCH 1/2] review @ 2021-01-09 02:17 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 56+ messages in thread From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw) --- doc/src/sgml/mvcc.sgml | 20 +++++++++++--------- doc/src/sgml/ref/create_policy.sgml | 6 +++--- doc/src/sgml/ref/merge.sgml | 12 +++++++++--- src/backend/commands/explain.c | 2 +- src/backend/executor/execMerge.c | 5 ++++- src/backend/executor/nodeModifyTable.c | 3 ++- src/backend/optimizer/prep/preptlist.c | 6 ++++++ src/backend/parser/parse_agg.c | 1 + src/backend/replication/walsender.c | 4 ++-- src/backend/rewrite/rewriteHandler.c | 4 ++++ src/backend/utils/adt/varlena.c | 3 +++ src/include/nodes/parsenodes.h | 7 ++++--- 12 files changed, 50 insertions(+), 23 deletions(-) diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml index 9ec8474185..02c9f3fdea 100644 --- a/doc/src/sgml/mvcc.sgml +++ b/doc/src/sgml/mvcc.sgml @@ -429,22 +429,24 @@ COMMIT; with both <command>INSERT</command> and <command>UPDATE</command> subcommands looks similar to <command>INSERT</command> with an <literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee - that either <command>INSERT</command> and <command>UPDATE</command> will occur. + that either <command>INSERT</command> or <command>UPDATE</command> will occur. - If MERGE attempts an UPDATE or DELETE and the row is concurrently updated + /* XXX This is a very long and hard to understand sentence :-( */ + /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */ + If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated but the join condition still passes for the current target and the current - source tuple, then MERGE will behave the same as the UPDATE or DELETE commands + source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands and perform its action on the latest version of the row, using standard - EvalPlanQual. MERGE actions can be conditional, so conditions must be + EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be re-evaluated on the latest row, starting from the first action. On the other hand, if the row is concurrently updated or deleted so that - the join condition fails, then MERGE will execute a NOT MATCHED action, if it - exists and the AND WHEN qual evaluates to true. + the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it + exists and the <literal>AND WHEN</literal> qual evaluates to true. - If MERGE attempts an INSERT and a unique index is present and a duplicate - row is concurrently inserted then a uniqueness violation is raised. MERGE - does not attempt to avoid the ERROR by attempting an UPDATE. + If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate + row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command> + does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>. </para> <para> diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml index ad20230c58..0a64674f2d 100644 --- a/doc/src/sgml/ref/create_policy.sgml +++ b/doc/src/sgml/ref/create_policy.sgml @@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable <para> No separate policy exists for <command>MERGE</command>. Instead policies - defined for <literal>SELECT</literal>, <literal>INSERT</literal>, - <literal>UPDATE</literal> and <literal>DELETE</literal> are applied - while executing MERGE, depending on the actions that are activated. + defined for <command>SELECT</command>, <command>INSERT</command>, + <command>UPDATE</command> and <command>DELETE</command> are applied + while executing <command>MERGE</command>, depending on the actions that are activated. </para> </refsect1> diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml index b2a9f67cfa..c2901b0d58 100644 --- a/doc/src/sgml/ref/merge.sgml +++ b/doc/src/sgml/ref/merge.sgml @@ -98,7 +98,7 @@ DELETE </para> <para> - There is no MERGE privilege. + There is no <literal>MERGE</literal> privilege. You must have the <literal>UPDATE</literal> privilege on the column(s) of the <replaceable class="parameter">target_table_name</replaceable> referred to in the <literal>SET</literal> clause @@ -217,6 +217,7 @@ DELETE At least one <literal>WHEN</literal> clause is required. </para> <para> + /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */ If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal> and the candidate change row matches a row in the <replaceable class="parameter">target_table_name</replaceable> @@ -242,6 +243,7 @@ DELETE clause will be activated and the corresponding action will occur for that row. The expression may not contain functions that possibly performs writes to the database. + /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */ </para> <para> A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns @@ -379,6 +381,7 @@ DELETE <para> An expression to assign to the column. The expression can use the old values of this and other columns in the table. + /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */ </para> </listitem> </varlistentry> @@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> In summary, statement triggers for an event type (say, INSERT) will be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level triggers will fire only for the one event type <emphasis>activated</emphasis>. + /* XXX What does "activated" mean? Maybe "executed" would be better? */ So a <command>MERGE</command> might fire statement triggers for both <command>UPDATE</command> and <command>INSERT</command>, even though only <command>UPDATE</command> row triggers were fired. @@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable> rows will be used to modify the target row, later attempts to modify will cause an error. This can also occur if row triggers make changes to the target table which are then subsequently modified by <command>MERGE</command>. + /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */ + /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */ If the repeated action is an <command>INSERT</command> this will cause a uniqueness violation while a repeated <command>UPDATE</command> or <command>DELETE</command> will cause a cardinality violation; the latter behavior @@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> <title>Examples</title> <para> - Perform maintenance on CustomerAccounts based upon new Transactions. + Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>. <programlisting> MERGE CustomerAccount CA @@ -599,7 +605,7 @@ WHEN MATCHED THEN DELETE; </programlisting> - The wine_stock_changes table might be, for example, a temporary table + The <literal>wine_stock_changes</literal> table might be, for example, a temporary table recently loaded into the database. </para> diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index fbaf50c258..f914a00e9f 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors, break; case CMD_MERGE: operation = "Merge"; - foperation = "Foreign Merge"; + foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */ break; default: operation = "???"; diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c index 0e245e1361..a7492a6c4b 100644 --- a/src/backend/executor/execMerge.c +++ b/src/backend/executor/execMerge.c @@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * A concurrent update can: * * 1. modify the target tuple so that it no longer satisfies the - * additional quals attached to the current WHEN MATCHED action OR + * additional quals attached to the current WHEN MATCHED action * * In this case, we are still dealing with a WHEN MATCHED case, but we * should recheck the list of WHEN MATCHED actions and choose the first @@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * tuple, so we now instead find a qualifying WHEN NOT MATCHED action to * execute. * + * XXX Hmmm, what if the updated tuple would now match one that was + * considered NOT MATCHED so far? + * * A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED. * * ExecMergeMatched takes care of following the update chain and diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index c0a97eba91..9c14709e47 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate) { /* advance to next subplan if any */ node->mt_whichplan++; - if (node->mt_whichplan < node->mt_nplans) { resultRelInfo++; @@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate) EvalPlanQualSetSlot(&node->mt_epqstate, planSlot); slot = planSlot; + /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together + * with the other MT operations? This seems a bit out of place. */ if (operation == CMD_MERGE) { ExecMerge(node, resultRelInfo, estate, slot, junkfilter); diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c index 8848e9a03f..b2543f6814 100644 --- a/src/backend/optimizer/prep/preptlist.c +++ b/src/backend/optimizer/prep/preptlist.c @@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root) tlist = expand_targetlist(tlist, command_type, result_relation, target_relation); + /* + * For MERGE we need to handle the target list for the target relation, + * and also target list for each action (only INSERT/UPDATE matter). + */ if (command_type == CMD_MERGE) { ListCell *l; @@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type, * generate a NULL for dropped columns (we want to drop any old * values). * + * XXX Should this explain why MERGE has the same logic as UPDATE? + * * When generating a NULL constant for a dropped column, we label * it INT4 (any other guaranteed-to-exist datatype would do as * well). We can't label it with the dropped column's datatype diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index f7c152beb4..5d49c8c37e 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) break; case EXPR_KIND_MERGE_WHEN_AND: if (isAgg) + /* XXX "WHEN AND" seems rather strange. ... */ err = _("aggregate functions are not allowed in WHEN AND conditions"); else err = _("grouping operations are not allowed in WHEN AND conditions"); diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index d59f4fde95..d50543b1ba 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd) /* Also update the sent position status in shared memory */ SpinLockAcquire(&MyWalSnd->mutex); + /* XXX Huh? Why does MERGE patch change walsender? */ MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush; SpinLockRelease(&MyWalSnd->mutex); @@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data) replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ? MyWalSnd->write : MyWalSnd->flush; - if (WalSndCaughtUp && - sentPtr == replicatedPtr && + if (WalSndCaughtUp && sentPtr == replicatedPtr && !pq_is_send_pending()) { QueryCompletion qc; diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c index e2706c181c..80ae946d17 100644 --- a/src/backend/rewrite/rewriteHandler.c +++ b/src/backend/rewrite/rewriteHandler.c @@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation) } } + /* * matchLocks - * match the list of locks and returns the matching rules @@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events) /* * XXX MERGE doesn't support write rules because they would violate * the SQL Standard spec and would be unclear how they should work. + * + * XXX So does't support means 'ignores'? Should that either fail + * or at least print some warning? */ if (event == CMD_MERGE) product_queries = NIL; diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 0281351dcf..7a768a7b5b 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup) int result; bool arg1_match; + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup) memset(pres, 0, sizeof(Datum)); len = VARSIZE_ANY_EXHDR(authoritative); + /* XXX Huh? Why is this in MERGE patch? */ if (len < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2) len1 = VARSIZE_ANY_EXHDR(t1); len2 = VARSIZE_ANY_EXHDR(t2); + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 3cba38044e..08fea9b8f7 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -172,6 +172,7 @@ typedef struct Query List *withCheckOptions; /* a list of WithCheckOption's (added * during rewrite) */ + /* XXX Why not mergeTragetRelation? */ int mergeTarget_relation; List *mergeSourceTargetList; List *mergeActionList; /* list of actions for MERGE (only) */ @@ -1581,11 +1582,11 @@ typedef struct UpdateStmt typedef struct MergeStmt { NodeTag type; - RangeVar *relation; /* target relation to merge into */ + RangeVar *relation; /* target relation to merge into */ Node *source_relation; /* source relation */ - Node *join_condition; /* join condition between source and target */ + Node *join_condition; /* join condition between source and target */ List *mergeWhenClauses; /* list of MergeWhenClause(es) */ - WithClause *withClause; /* WITH clause */ + WithClause *withClause; /* WITH clause */ } MergeStmt; typedef struct MergeWhenClause -- 2.26.2 --------------B1AA4DC5EAE9BAF8B146244A Content-Type: text/plain; charset=UTF-8; name="0002-fix-valgrind-failure.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-fix-valgrind-failure.txt" ^ permalink raw reply [nested|flat] 56+ messages in thread
* [PATCH 1/2] review @ 2021-01-09 02:17 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 56+ messages in thread From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw) --- doc/src/sgml/mvcc.sgml | 20 +++++++++++--------- doc/src/sgml/ref/create_policy.sgml | 6 +++--- doc/src/sgml/ref/merge.sgml | 12 +++++++++--- src/backend/commands/explain.c | 2 +- src/backend/executor/execMerge.c | 5 ++++- src/backend/executor/nodeModifyTable.c | 3 ++- src/backend/optimizer/prep/preptlist.c | 6 ++++++ src/backend/parser/parse_agg.c | 1 + src/backend/replication/walsender.c | 4 ++-- src/backend/rewrite/rewriteHandler.c | 4 ++++ src/backend/utils/adt/varlena.c | 3 +++ src/include/nodes/parsenodes.h | 7 ++++--- 12 files changed, 50 insertions(+), 23 deletions(-) diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml index 9ec8474185..02c9f3fdea 100644 --- a/doc/src/sgml/mvcc.sgml +++ b/doc/src/sgml/mvcc.sgml @@ -429,22 +429,24 @@ COMMIT; with both <command>INSERT</command> and <command>UPDATE</command> subcommands looks similar to <command>INSERT</command> with an <literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee - that either <command>INSERT</command> and <command>UPDATE</command> will occur. + that either <command>INSERT</command> or <command>UPDATE</command> will occur. - If MERGE attempts an UPDATE or DELETE and the row is concurrently updated + /* XXX This is a very long and hard to understand sentence :-( */ + /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */ + If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated but the join condition still passes for the current target and the current - source tuple, then MERGE will behave the same as the UPDATE or DELETE commands + source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands and perform its action on the latest version of the row, using standard - EvalPlanQual. MERGE actions can be conditional, so conditions must be + EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be re-evaluated on the latest row, starting from the first action. On the other hand, if the row is concurrently updated or deleted so that - the join condition fails, then MERGE will execute a NOT MATCHED action, if it - exists and the AND WHEN qual evaluates to true. + the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it + exists and the <literal>AND WHEN</literal> qual evaluates to true. - If MERGE attempts an INSERT and a unique index is present and a duplicate - row is concurrently inserted then a uniqueness violation is raised. MERGE - does not attempt to avoid the ERROR by attempting an UPDATE. + If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate + row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command> + does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>. </para> <para> diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml index ad20230c58..0a64674f2d 100644 --- a/doc/src/sgml/ref/create_policy.sgml +++ b/doc/src/sgml/ref/create_policy.sgml @@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable <para> No separate policy exists for <command>MERGE</command>. Instead policies - defined for <literal>SELECT</literal>, <literal>INSERT</literal>, - <literal>UPDATE</literal> and <literal>DELETE</literal> are applied - while executing MERGE, depending on the actions that are activated. + defined for <command>SELECT</command>, <command>INSERT</command>, + <command>UPDATE</command> and <command>DELETE</command> are applied + while executing <command>MERGE</command>, depending on the actions that are activated. </para> </refsect1> diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml index b2a9f67cfa..c2901b0d58 100644 --- a/doc/src/sgml/ref/merge.sgml +++ b/doc/src/sgml/ref/merge.sgml @@ -98,7 +98,7 @@ DELETE </para> <para> - There is no MERGE privilege. + There is no <literal>MERGE</literal> privilege. You must have the <literal>UPDATE</literal> privilege on the column(s) of the <replaceable class="parameter">target_table_name</replaceable> referred to in the <literal>SET</literal> clause @@ -217,6 +217,7 @@ DELETE At least one <literal>WHEN</literal> clause is required. </para> <para> + /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */ If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal> and the candidate change row matches a row in the <replaceable class="parameter">target_table_name</replaceable> @@ -242,6 +243,7 @@ DELETE clause will be activated and the corresponding action will occur for that row. The expression may not contain functions that possibly performs writes to the database. + /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */ </para> <para> A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns @@ -379,6 +381,7 @@ DELETE <para> An expression to assign to the column. The expression can use the old values of this and other columns in the table. + /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */ </para> </listitem> </varlistentry> @@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> In summary, statement triggers for an event type (say, INSERT) will be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level triggers will fire only for the one event type <emphasis>activated</emphasis>. + /* XXX What does "activated" mean? Maybe "executed" would be better? */ So a <command>MERGE</command> might fire statement triggers for both <command>UPDATE</command> and <command>INSERT</command>, even though only <command>UPDATE</command> row triggers were fired. @@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable> rows will be used to modify the target row, later attempts to modify will cause an error. This can also occur if row triggers make changes to the target table which are then subsequently modified by <command>MERGE</command>. + /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */ + /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */ If the repeated action is an <command>INSERT</command> this will cause a uniqueness violation while a repeated <command>UPDATE</command> or <command>DELETE</command> will cause a cardinality violation; the latter behavior @@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> <title>Examples</title> <para> - Perform maintenance on CustomerAccounts based upon new Transactions. + Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>. <programlisting> MERGE CustomerAccount CA @@ -599,7 +605,7 @@ WHEN MATCHED THEN DELETE; </programlisting> - The wine_stock_changes table might be, for example, a temporary table + The <literal>wine_stock_changes</literal> table might be, for example, a temporary table recently loaded into the database. </para> diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index fbaf50c258..f914a00e9f 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors, break; case CMD_MERGE: operation = "Merge"; - foperation = "Foreign Merge"; + foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */ break; default: operation = "???"; diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c index 0e245e1361..a7492a6c4b 100644 --- a/src/backend/executor/execMerge.c +++ b/src/backend/executor/execMerge.c @@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * A concurrent update can: * * 1. modify the target tuple so that it no longer satisfies the - * additional quals attached to the current WHEN MATCHED action OR + * additional quals attached to the current WHEN MATCHED action * * In this case, we are still dealing with a WHEN MATCHED case, but we * should recheck the list of WHEN MATCHED actions and choose the first @@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * tuple, so we now instead find a qualifying WHEN NOT MATCHED action to * execute. * + * XXX Hmmm, what if the updated tuple would now match one that was + * considered NOT MATCHED so far? + * * A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED. * * ExecMergeMatched takes care of following the update chain and diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index c0a97eba91..9c14709e47 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate) { /* advance to next subplan if any */ node->mt_whichplan++; - if (node->mt_whichplan < node->mt_nplans) { resultRelInfo++; @@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate) EvalPlanQualSetSlot(&node->mt_epqstate, planSlot); slot = planSlot; + /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together + * with the other MT operations? This seems a bit out of place. */ if (operation == CMD_MERGE) { ExecMerge(node, resultRelInfo, estate, slot, junkfilter); diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c index 8848e9a03f..b2543f6814 100644 --- a/src/backend/optimizer/prep/preptlist.c +++ b/src/backend/optimizer/prep/preptlist.c @@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root) tlist = expand_targetlist(tlist, command_type, result_relation, target_relation); + /* + * For MERGE we need to handle the target list for the target relation, + * and also target list for each action (only INSERT/UPDATE matter). + */ if (command_type == CMD_MERGE) { ListCell *l; @@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type, * generate a NULL for dropped columns (we want to drop any old * values). * + * XXX Should this explain why MERGE has the same logic as UPDATE? + * * When generating a NULL constant for a dropped column, we label * it INT4 (any other guaranteed-to-exist datatype would do as * well). We can't label it with the dropped column's datatype diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index f7c152beb4..5d49c8c37e 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) break; case EXPR_KIND_MERGE_WHEN_AND: if (isAgg) + /* XXX "WHEN AND" seems rather strange. ... */ err = _("aggregate functions are not allowed in WHEN AND conditions"); else err = _("grouping operations are not allowed in WHEN AND conditions"); diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index d59f4fde95..d50543b1ba 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd) /* Also update the sent position status in shared memory */ SpinLockAcquire(&MyWalSnd->mutex); + /* XXX Huh? Why does MERGE patch change walsender? */ MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush; SpinLockRelease(&MyWalSnd->mutex); @@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data) replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ? MyWalSnd->write : MyWalSnd->flush; - if (WalSndCaughtUp && - sentPtr == replicatedPtr && + if (WalSndCaughtUp && sentPtr == replicatedPtr && !pq_is_send_pending()) { QueryCompletion qc; diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c index e2706c181c..80ae946d17 100644 --- a/src/backend/rewrite/rewriteHandler.c +++ b/src/backend/rewrite/rewriteHandler.c @@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation) } } + /* * matchLocks - * match the list of locks and returns the matching rules @@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events) /* * XXX MERGE doesn't support write rules because they would violate * the SQL Standard spec and would be unclear how they should work. + * + * XXX So does't support means 'ignores'? Should that either fail + * or at least print some warning? */ if (event == CMD_MERGE) product_queries = NIL; diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 0281351dcf..7a768a7b5b 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup) int result; bool arg1_match; + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup) memset(pres, 0, sizeof(Datum)); len = VARSIZE_ANY_EXHDR(authoritative); + /* XXX Huh? Why is this in MERGE patch? */ if (len < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2) len1 = VARSIZE_ANY_EXHDR(t1); len2 = VARSIZE_ANY_EXHDR(t2); + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 3cba38044e..08fea9b8f7 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -172,6 +172,7 @@ typedef struct Query List *withCheckOptions; /* a list of WithCheckOption's (added * during rewrite) */ + /* XXX Why not mergeTragetRelation? */ int mergeTarget_relation; List *mergeSourceTargetList; List *mergeActionList; /* list of actions for MERGE (only) */ @@ -1581,11 +1582,11 @@ typedef struct UpdateStmt typedef struct MergeStmt { NodeTag type; - RangeVar *relation; /* target relation to merge into */ + RangeVar *relation; /* target relation to merge into */ Node *source_relation; /* source relation */ - Node *join_condition; /* join condition between source and target */ + Node *join_condition; /* join condition between source and target */ List *mergeWhenClauses; /* list of MergeWhenClause(es) */ - WithClause *withClause; /* WITH clause */ + WithClause *withClause; /* WITH clause */ } MergeStmt; typedef struct MergeWhenClause -- 2.26.2 --------------B1AA4DC5EAE9BAF8B146244A Content-Type: text/plain; charset=UTF-8; name="0002-fix-valgrind-failure.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-fix-valgrind-failure.txt" ^ permalink raw reply [nested|flat] 56+ messages in thread
* [PATCH 1/2] review @ 2021-01-09 02:17 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 56+ messages in thread From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw) --- doc/src/sgml/mvcc.sgml | 20 +++++++++++--------- doc/src/sgml/ref/create_policy.sgml | 6 +++--- doc/src/sgml/ref/merge.sgml | 12 +++++++++--- src/backend/commands/explain.c | 2 +- src/backend/executor/execMerge.c | 5 ++++- src/backend/executor/nodeModifyTable.c | 3 ++- src/backend/optimizer/prep/preptlist.c | 6 ++++++ src/backend/parser/parse_agg.c | 1 + src/backend/replication/walsender.c | 4 ++-- src/backend/rewrite/rewriteHandler.c | 4 ++++ src/backend/utils/adt/varlena.c | 3 +++ src/include/nodes/parsenodes.h | 7 ++++--- 12 files changed, 50 insertions(+), 23 deletions(-) diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml index 9ec8474185..02c9f3fdea 100644 --- a/doc/src/sgml/mvcc.sgml +++ b/doc/src/sgml/mvcc.sgml @@ -429,22 +429,24 @@ COMMIT; with both <command>INSERT</command> and <command>UPDATE</command> subcommands looks similar to <command>INSERT</command> with an <literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee - that either <command>INSERT</command> and <command>UPDATE</command> will occur. + that either <command>INSERT</command> or <command>UPDATE</command> will occur. - If MERGE attempts an UPDATE or DELETE and the row is concurrently updated + /* XXX This is a very long and hard to understand sentence :-( */ + /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */ + If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated but the join condition still passes for the current target and the current - source tuple, then MERGE will behave the same as the UPDATE or DELETE commands + source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands and perform its action on the latest version of the row, using standard - EvalPlanQual. MERGE actions can be conditional, so conditions must be + EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be re-evaluated on the latest row, starting from the first action. On the other hand, if the row is concurrently updated or deleted so that - the join condition fails, then MERGE will execute a NOT MATCHED action, if it - exists and the AND WHEN qual evaluates to true. + the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it + exists and the <literal>AND WHEN</literal> qual evaluates to true. - If MERGE attempts an INSERT and a unique index is present and a duplicate - row is concurrently inserted then a uniqueness violation is raised. MERGE - does not attempt to avoid the ERROR by attempting an UPDATE. + If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate + row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command> + does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>. </para> <para> diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml index ad20230c58..0a64674f2d 100644 --- a/doc/src/sgml/ref/create_policy.sgml +++ b/doc/src/sgml/ref/create_policy.sgml @@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable <para> No separate policy exists for <command>MERGE</command>. Instead policies - defined for <literal>SELECT</literal>, <literal>INSERT</literal>, - <literal>UPDATE</literal> and <literal>DELETE</literal> are applied - while executing MERGE, depending on the actions that are activated. + defined for <command>SELECT</command>, <command>INSERT</command>, + <command>UPDATE</command> and <command>DELETE</command> are applied + while executing <command>MERGE</command>, depending on the actions that are activated. </para> </refsect1> diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml index b2a9f67cfa..c2901b0d58 100644 --- a/doc/src/sgml/ref/merge.sgml +++ b/doc/src/sgml/ref/merge.sgml @@ -98,7 +98,7 @@ DELETE </para> <para> - There is no MERGE privilege. + There is no <literal>MERGE</literal> privilege. You must have the <literal>UPDATE</literal> privilege on the column(s) of the <replaceable class="parameter">target_table_name</replaceable> referred to in the <literal>SET</literal> clause @@ -217,6 +217,7 @@ DELETE At least one <literal>WHEN</literal> clause is required. </para> <para> + /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */ If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal> and the candidate change row matches a row in the <replaceable class="parameter">target_table_name</replaceable> @@ -242,6 +243,7 @@ DELETE clause will be activated and the corresponding action will occur for that row. The expression may not contain functions that possibly performs writes to the database. + /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */ </para> <para> A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns @@ -379,6 +381,7 @@ DELETE <para> An expression to assign to the column. The expression can use the old values of this and other columns in the table. + /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */ </para> </listitem> </varlistentry> @@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> In summary, statement triggers for an event type (say, INSERT) will be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level triggers will fire only for the one event type <emphasis>activated</emphasis>. + /* XXX What does "activated" mean? Maybe "executed" would be better? */ So a <command>MERGE</command> might fire statement triggers for both <command>UPDATE</command> and <command>INSERT</command>, even though only <command>UPDATE</command> row triggers were fired. @@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable> rows will be used to modify the target row, later attempts to modify will cause an error. This can also occur if row triggers make changes to the target table which are then subsequently modified by <command>MERGE</command>. + /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */ + /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */ If the repeated action is an <command>INSERT</command> this will cause a uniqueness violation while a repeated <command>UPDATE</command> or <command>DELETE</command> will cause a cardinality violation; the latter behavior @@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> <title>Examples</title> <para> - Perform maintenance on CustomerAccounts based upon new Transactions. + Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>. <programlisting> MERGE CustomerAccount CA @@ -599,7 +605,7 @@ WHEN MATCHED THEN DELETE; </programlisting> - The wine_stock_changes table might be, for example, a temporary table + The <literal>wine_stock_changes</literal> table might be, for example, a temporary table recently loaded into the database. </para> diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index fbaf50c258..f914a00e9f 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors, break; case CMD_MERGE: operation = "Merge"; - foperation = "Foreign Merge"; + foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */ break; default: operation = "???"; diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c index 0e245e1361..a7492a6c4b 100644 --- a/src/backend/executor/execMerge.c +++ b/src/backend/executor/execMerge.c @@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * A concurrent update can: * * 1. modify the target tuple so that it no longer satisfies the - * additional quals attached to the current WHEN MATCHED action OR + * additional quals attached to the current WHEN MATCHED action * * In this case, we are still dealing with a WHEN MATCHED case, but we * should recheck the list of WHEN MATCHED actions and choose the first @@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * tuple, so we now instead find a qualifying WHEN NOT MATCHED action to * execute. * + * XXX Hmmm, what if the updated tuple would now match one that was + * considered NOT MATCHED so far? + * * A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED. * * ExecMergeMatched takes care of following the update chain and diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index c0a97eba91..9c14709e47 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate) { /* advance to next subplan if any */ node->mt_whichplan++; - if (node->mt_whichplan < node->mt_nplans) { resultRelInfo++; @@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate) EvalPlanQualSetSlot(&node->mt_epqstate, planSlot); slot = planSlot; + /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together + * with the other MT operations? This seems a bit out of place. */ if (operation == CMD_MERGE) { ExecMerge(node, resultRelInfo, estate, slot, junkfilter); diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c index 8848e9a03f..b2543f6814 100644 --- a/src/backend/optimizer/prep/preptlist.c +++ b/src/backend/optimizer/prep/preptlist.c @@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root) tlist = expand_targetlist(tlist, command_type, result_relation, target_relation); + /* + * For MERGE we need to handle the target list for the target relation, + * and also target list for each action (only INSERT/UPDATE matter). + */ if (command_type == CMD_MERGE) { ListCell *l; @@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type, * generate a NULL for dropped columns (we want to drop any old * values). * + * XXX Should this explain why MERGE has the same logic as UPDATE? + * * When generating a NULL constant for a dropped column, we label * it INT4 (any other guaranteed-to-exist datatype would do as * well). We can't label it with the dropped column's datatype diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index f7c152beb4..5d49c8c37e 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) break; case EXPR_KIND_MERGE_WHEN_AND: if (isAgg) + /* XXX "WHEN AND" seems rather strange. ... */ err = _("aggregate functions are not allowed in WHEN AND conditions"); else err = _("grouping operations are not allowed in WHEN AND conditions"); diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index d59f4fde95..d50543b1ba 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd) /* Also update the sent position status in shared memory */ SpinLockAcquire(&MyWalSnd->mutex); + /* XXX Huh? Why does MERGE patch change walsender? */ MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush; SpinLockRelease(&MyWalSnd->mutex); @@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data) replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ? MyWalSnd->write : MyWalSnd->flush; - if (WalSndCaughtUp && - sentPtr == replicatedPtr && + if (WalSndCaughtUp && sentPtr == replicatedPtr && !pq_is_send_pending()) { QueryCompletion qc; diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c index e2706c181c..80ae946d17 100644 --- a/src/backend/rewrite/rewriteHandler.c +++ b/src/backend/rewrite/rewriteHandler.c @@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation) } } + /* * matchLocks - * match the list of locks and returns the matching rules @@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events) /* * XXX MERGE doesn't support write rules because they would violate * the SQL Standard spec and would be unclear how they should work. + * + * XXX So does't support means 'ignores'? Should that either fail + * or at least print some warning? */ if (event == CMD_MERGE) product_queries = NIL; diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 0281351dcf..7a768a7b5b 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup) int result; bool arg1_match; + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup) memset(pres, 0, sizeof(Datum)); len = VARSIZE_ANY_EXHDR(authoritative); + /* XXX Huh? Why is this in MERGE patch? */ if (len < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2) len1 = VARSIZE_ANY_EXHDR(t1); len2 = VARSIZE_ANY_EXHDR(t2); + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 3cba38044e..08fea9b8f7 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -172,6 +172,7 @@ typedef struct Query List *withCheckOptions; /* a list of WithCheckOption's (added * during rewrite) */ + /* XXX Why not mergeTragetRelation? */ int mergeTarget_relation; List *mergeSourceTargetList; List *mergeActionList; /* list of actions for MERGE (only) */ @@ -1581,11 +1582,11 @@ typedef struct UpdateStmt typedef struct MergeStmt { NodeTag type; - RangeVar *relation; /* target relation to merge into */ + RangeVar *relation; /* target relation to merge into */ Node *source_relation; /* source relation */ - Node *join_condition; /* join condition between source and target */ + Node *join_condition; /* join condition between source and target */ List *mergeWhenClauses; /* list of MergeWhenClause(es) */ - WithClause *withClause; /* WITH clause */ + WithClause *withClause; /* WITH clause */ } MergeStmt; typedef struct MergeWhenClause -- 2.26.2 --------------B1AA4DC5EAE9BAF8B146244A Content-Type: text/plain; charset=UTF-8; name="0002-fix-valgrind-failure.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-fix-valgrind-failure.txt" ^ permalink raw reply [nested|flat] 56+ messages in thread
* [PATCH 1/2] review @ 2021-01-09 02:17 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 56+ messages in thread From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw) --- doc/src/sgml/mvcc.sgml | 20 +++++++++++--------- doc/src/sgml/ref/create_policy.sgml | 6 +++--- doc/src/sgml/ref/merge.sgml | 12 +++++++++--- src/backend/commands/explain.c | 2 +- src/backend/executor/execMerge.c | 5 ++++- src/backend/executor/nodeModifyTable.c | 3 ++- src/backend/optimizer/prep/preptlist.c | 6 ++++++ src/backend/parser/parse_agg.c | 1 + src/backend/replication/walsender.c | 4 ++-- src/backend/rewrite/rewriteHandler.c | 4 ++++ src/backend/utils/adt/varlena.c | 3 +++ src/include/nodes/parsenodes.h | 7 ++++--- 12 files changed, 50 insertions(+), 23 deletions(-) diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml index 9ec8474185..02c9f3fdea 100644 --- a/doc/src/sgml/mvcc.sgml +++ b/doc/src/sgml/mvcc.sgml @@ -429,22 +429,24 @@ COMMIT; with both <command>INSERT</command> and <command>UPDATE</command> subcommands looks similar to <command>INSERT</command> with an <literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee - that either <command>INSERT</command> and <command>UPDATE</command> will occur. + that either <command>INSERT</command> or <command>UPDATE</command> will occur. - If MERGE attempts an UPDATE or DELETE and the row is concurrently updated + /* XXX This is a very long and hard to understand sentence :-( */ + /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */ + If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated but the join condition still passes for the current target and the current - source tuple, then MERGE will behave the same as the UPDATE or DELETE commands + source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands and perform its action on the latest version of the row, using standard - EvalPlanQual. MERGE actions can be conditional, so conditions must be + EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be re-evaluated on the latest row, starting from the first action. On the other hand, if the row is concurrently updated or deleted so that - the join condition fails, then MERGE will execute a NOT MATCHED action, if it - exists and the AND WHEN qual evaluates to true. + the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it + exists and the <literal>AND WHEN</literal> qual evaluates to true. - If MERGE attempts an INSERT and a unique index is present and a duplicate - row is concurrently inserted then a uniqueness violation is raised. MERGE - does not attempt to avoid the ERROR by attempting an UPDATE. + If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate + row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command> + does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>. </para> <para> diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml index ad20230c58..0a64674f2d 100644 --- a/doc/src/sgml/ref/create_policy.sgml +++ b/doc/src/sgml/ref/create_policy.sgml @@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable <para> No separate policy exists for <command>MERGE</command>. Instead policies - defined for <literal>SELECT</literal>, <literal>INSERT</literal>, - <literal>UPDATE</literal> and <literal>DELETE</literal> are applied - while executing MERGE, depending on the actions that are activated. + defined for <command>SELECT</command>, <command>INSERT</command>, + <command>UPDATE</command> and <command>DELETE</command> are applied + while executing <command>MERGE</command>, depending on the actions that are activated. </para> </refsect1> diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml index b2a9f67cfa..c2901b0d58 100644 --- a/doc/src/sgml/ref/merge.sgml +++ b/doc/src/sgml/ref/merge.sgml @@ -98,7 +98,7 @@ DELETE </para> <para> - There is no MERGE privilege. + There is no <literal>MERGE</literal> privilege. You must have the <literal>UPDATE</literal> privilege on the column(s) of the <replaceable class="parameter">target_table_name</replaceable> referred to in the <literal>SET</literal> clause @@ -217,6 +217,7 @@ DELETE At least one <literal>WHEN</literal> clause is required. </para> <para> + /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */ If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal> and the candidate change row matches a row in the <replaceable class="parameter">target_table_name</replaceable> @@ -242,6 +243,7 @@ DELETE clause will be activated and the corresponding action will occur for that row. The expression may not contain functions that possibly performs writes to the database. + /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */ </para> <para> A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns @@ -379,6 +381,7 @@ DELETE <para> An expression to assign to the column. The expression can use the old values of this and other columns in the table. + /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */ </para> </listitem> </varlistentry> @@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> In summary, statement triggers for an event type (say, INSERT) will be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level triggers will fire only for the one event type <emphasis>activated</emphasis>. + /* XXX What does "activated" mean? Maybe "executed" would be better? */ So a <command>MERGE</command> might fire statement triggers for both <command>UPDATE</command> and <command>INSERT</command>, even though only <command>UPDATE</command> row triggers were fired. @@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable> rows will be used to modify the target row, later attempts to modify will cause an error. This can also occur if row triggers make changes to the target table which are then subsequently modified by <command>MERGE</command>. + /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */ + /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */ If the repeated action is an <command>INSERT</command> this will cause a uniqueness violation while a repeated <command>UPDATE</command> or <command>DELETE</command> will cause a cardinality violation; the latter behavior @@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> <title>Examples</title> <para> - Perform maintenance on CustomerAccounts based upon new Transactions. + Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>. <programlisting> MERGE CustomerAccount CA @@ -599,7 +605,7 @@ WHEN MATCHED THEN DELETE; </programlisting> - The wine_stock_changes table might be, for example, a temporary table + The <literal>wine_stock_changes</literal> table might be, for example, a temporary table recently loaded into the database. </para> diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index fbaf50c258..f914a00e9f 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors, break; case CMD_MERGE: operation = "Merge"; - foperation = "Foreign Merge"; + foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */ break; default: operation = "???"; diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c index 0e245e1361..a7492a6c4b 100644 --- a/src/backend/executor/execMerge.c +++ b/src/backend/executor/execMerge.c @@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * A concurrent update can: * * 1. modify the target tuple so that it no longer satisfies the - * additional quals attached to the current WHEN MATCHED action OR + * additional quals attached to the current WHEN MATCHED action * * In this case, we are still dealing with a WHEN MATCHED case, but we * should recheck the list of WHEN MATCHED actions and choose the first @@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * tuple, so we now instead find a qualifying WHEN NOT MATCHED action to * execute. * + * XXX Hmmm, what if the updated tuple would now match one that was + * considered NOT MATCHED so far? + * * A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED. * * ExecMergeMatched takes care of following the update chain and diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index c0a97eba91..9c14709e47 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate) { /* advance to next subplan if any */ node->mt_whichplan++; - if (node->mt_whichplan < node->mt_nplans) { resultRelInfo++; @@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate) EvalPlanQualSetSlot(&node->mt_epqstate, planSlot); slot = planSlot; + /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together + * with the other MT operations? This seems a bit out of place. */ if (operation == CMD_MERGE) { ExecMerge(node, resultRelInfo, estate, slot, junkfilter); diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c index 8848e9a03f..b2543f6814 100644 --- a/src/backend/optimizer/prep/preptlist.c +++ b/src/backend/optimizer/prep/preptlist.c @@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root) tlist = expand_targetlist(tlist, command_type, result_relation, target_relation); + /* + * For MERGE we need to handle the target list for the target relation, + * and also target list for each action (only INSERT/UPDATE matter). + */ if (command_type == CMD_MERGE) { ListCell *l; @@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type, * generate a NULL for dropped columns (we want to drop any old * values). * + * XXX Should this explain why MERGE has the same logic as UPDATE? + * * When generating a NULL constant for a dropped column, we label * it INT4 (any other guaranteed-to-exist datatype would do as * well). We can't label it with the dropped column's datatype diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index f7c152beb4..5d49c8c37e 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) break; case EXPR_KIND_MERGE_WHEN_AND: if (isAgg) + /* XXX "WHEN AND" seems rather strange. ... */ err = _("aggregate functions are not allowed in WHEN AND conditions"); else err = _("grouping operations are not allowed in WHEN AND conditions"); diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index d59f4fde95..d50543b1ba 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd) /* Also update the sent position status in shared memory */ SpinLockAcquire(&MyWalSnd->mutex); + /* XXX Huh? Why does MERGE patch change walsender? */ MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush; SpinLockRelease(&MyWalSnd->mutex); @@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data) replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ? MyWalSnd->write : MyWalSnd->flush; - if (WalSndCaughtUp && - sentPtr == replicatedPtr && + if (WalSndCaughtUp && sentPtr == replicatedPtr && !pq_is_send_pending()) { QueryCompletion qc; diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c index e2706c181c..80ae946d17 100644 --- a/src/backend/rewrite/rewriteHandler.c +++ b/src/backend/rewrite/rewriteHandler.c @@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation) } } + /* * matchLocks - * match the list of locks and returns the matching rules @@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events) /* * XXX MERGE doesn't support write rules because they would violate * the SQL Standard spec and would be unclear how they should work. + * + * XXX So does't support means 'ignores'? Should that either fail + * or at least print some warning? */ if (event == CMD_MERGE) product_queries = NIL; diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 0281351dcf..7a768a7b5b 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup) int result; bool arg1_match; + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup) memset(pres, 0, sizeof(Datum)); len = VARSIZE_ANY_EXHDR(authoritative); + /* XXX Huh? Why is this in MERGE patch? */ if (len < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2) len1 = VARSIZE_ANY_EXHDR(t1); len2 = VARSIZE_ANY_EXHDR(t2); + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 3cba38044e..08fea9b8f7 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -172,6 +172,7 @@ typedef struct Query List *withCheckOptions; /* a list of WithCheckOption's (added * during rewrite) */ + /* XXX Why not mergeTragetRelation? */ int mergeTarget_relation; List *mergeSourceTargetList; List *mergeActionList; /* list of actions for MERGE (only) */ @@ -1581,11 +1582,11 @@ typedef struct UpdateStmt typedef struct MergeStmt { NodeTag type; - RangeVar *relation; /* target relation to merge into */ + RangeVar *relation; /* target relation to merge into */ Node *source_relation; /* source relation */ - Node *join_condition; /* join condition between source and target */ + Node *join_condition; /* join condition between source and target */ List *mergeWhenClauses; /* list of MergeWhenClause(es) */ - WithClause *withClause; /* WITH clause */ + WithClause *withClause; /* WITH clause */ } MergeStmt; typedef struct MergeWhenClause -- 2.26.2 --------------B1AA4DC5EAE9BAF8B146244A Content-Type: text/plain; charset=UTF-8; name="0002-fix-valgrind-failure.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-fix-valgrind-failure.txt" ^ permalink raw reply [nested|flat] 56+ messages in thread
* [PATCH 1/2] review @ 2021-01-09 02:17 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 56+ messages in thread From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw) --- doc/src/sgml/mvcc.sgml | 20 +++++++++++--------- doc/src/sgml/ref/create_policy.sgml | 6 +++--- doc/src/sgml/ref/merge.sgml | 12 +++++++++--- src/backend/commands/explain.c | 2 +- src/backend/executor/execMerge.c | 5 ++++- src/backend/executor/nodeModifyTable.c | 3 ++- src/backend/optimizer/prep/preptlist.c | 6 ++++++ src/backend/parser/parse_agg.c | 1 + src/backend/replication/walsender.c | 4 ++-- src/backend/rewrite/rewriteHandler.c | 4 ++++ src/backend/utils/adt/varlena.c | 3 +++ src/include/nodes/parsenodes.h | 7 ++++--- 12 files changed, 50 insertions(+), 23 deletions(-) diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml index 9ec8474185..02c9f3fdea 100644 --- a/doc/src/sgml/mvcc.sgml +++ b/doc/src/sgml/mvcc.sgml @@ -429,22 +429,24 @@ COMMIT; with both <command>INSERT</command> and <command>UPDATE</command> subcommands looks similar to <command>INSERT</command> with an <literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee - that either <command>INSERT</command> and <command>UPDATE</command> will occur. + that either <command>INSERT</command> or <command>UPDATE</command> will occur. - If MERGE attempts an UPDATE or DELETE and the row is concurrently updated + /* XXX This is a very long and hard to understand sentence :-( */ + /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */ + If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated but the join condition still passes for the current target and the current - source tuple, then MERGE will behave the same as the UPDATE or DELETE commands + source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands and perform its action on the latest version of the row, using standard - EvalPlanQual. MERGE actions can be conditional, so conditions must be + EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be re-evaluated on the latest row, starting from the first action. On the other hand, if the row is concurrently updated or deleted so that - the join condition fails, then MERGE will execute a NOT MATCHED action, if it - exists and the AND WHEN qual evaluates to true. + the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it + exists and the <literal>AND WHEN</literal> qual evaluates to true. - If MERGE attempts an INSERT and a unique index is present and a duplicate - row is concurrently inserted then a uniqueness violation is raised. MERGE - does not attempt to avoid the ERROR by attempting an UPDATE. + If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate + row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command> + does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>. </para> <para> diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml index ad20230c58..0a64674f2d 100644 --- a/doc/src/sgml/ref/create_policy.sgml +++ b/doc/src/sgml/ref/create_policy.sgml @@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable <para> No separate policy exists for <command>MERGE</command>. Instead policies - defined for <literal>SELECT</literal>, <literal>INSERT</literal>, - <literal>UPDATE</literal> and <literal>DELETE</literal> are applied - while executing MERGE, depending on the actions that are activated. + defined for <command>SELECT</command>, <command>INSERT</command>, + <command>UPDATE</command> and <command>DELETE</command> are applied + while executing <command>MERGE</command>, depending on the actions that are activated. </para> </refsect1> diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml index b2a9f67cfa..c2901b0d58 100644 --- a/doc/src/sgml/ref/merge.sgml +++ b/doc/src/sgml/ref/merge.sgml @@ -98,7 +98,7 @@ DELETE </para> <para> - There is no MERGE privilege. + There is no <literal>MERGE</literal> privilege. You must have the <literal>UPDATE</literal> privilege on the column(s) of the <replaceable class="parameter">target_table_name</replaceable> referred to in the <literal>SET</literal> clause @@ -217,6 +217,7 @@ DELETE At least one <literal>WHEN</literal> clause is required. </para> <para> + /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */ If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal> and the candidate change row matches a row in the <replaceable class="parameter">target_table_name</replaceable> @@ -242,6 +243,7 @@ DELETE clause will be activated and the corresponding action will occur for that row. The expression may not contain functions that possibly performs writes to the database. + /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */ </para> <para> A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns @@ -379,6 +381,7 @@ DELETE <para> An expression to assign to the column. The expression can use the old values of this and other columns in the table. + /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */ </para> </listitem> </varlistentry> @@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> In summary, statement triggers for an event type (say, INSERT) will be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level triggers will fire only for the one event type <emphasis>activated</emphasis>. + /* XXX What does "activated" mean? Maybe "executed" would be better? */ So a <command>MERGE</command> might fire statement triggers for both <command>UPDATE</command> and <command>INSERT</command>, even though only <command>UPDATE</command> row triggers were fired. @@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable> rows will be used to modify the target row, later attempts to modify will cause an error. This can also occur if row triggers make changes to the target table which are then subsequently modified by <command>MERGE</command>. + /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */ + /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */ If the repeated action is an <command>INSERT</command> this will cause a uniqueness violation while a repeated <command>UPDATE</command> or <command>DELETE</command> will cause a cardinality violation; the latter behavior @@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> <title>Examples</title> <para> - Perform maintenance on CustomerAccounts based upon new Transactions. + Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>. <programlisting> MERGE CustomerAccount CA @@ -599,7 +605,7 @@ WHEN MATCHED THEN DELETE; </programlisting> - The wine_stock_changes table might be, for example, a temporary table + The <literal>wine_stock_changes</literal> table might be, for example, a temporary table recently loaded into the database. </para> diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index fbaf50c258..f914a00e9f 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors, break; case CMD_MERGE: operation = "Merge"; - foperation = "Foreign Merge"; + foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */ break; default: operation = "???"; diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c index 0e245e1361..a7492a6c4b 100644 --- a/src/backend/executor/execMerge.c +++ b/src/backend/executor/execMerge.c @@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * A concurrent update can: * * 1. modify the target tuple so that it no longer satisfies the - * additional quals attached to the current WHEN MATCHED action OR + * additional quals attached to the current WHEN MATCHED action * * In this case, we are still dealing with a WHEN MATCHED case, but we * should recheck the list of WHEN MATCHED actions and choose the first @@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * tuple, so we now instead find a qualifying WHEN NOT MATCHED action to * execute. * + * XXX Hmmm, what if the updated tuple would now match one that was + * considered NOT MATCHED so far? + * * A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED. * * ExecMergeMatched takes care of following the update chain and diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index c0a97eba91..9c14709e47 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate) { /* advance to next subplan if any */ node->mt_whichplan++; - if (node->mt_whichplan < node->mt_nplans) { resultRelInfo++; @@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate) EvalPlanQualSetSlot(&node->mt_epqstate, planSlot); slot = planSlot; + /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together + * with the other MT operations? This seems a bit out of place. */ if (operation == CMD_MERGE) { ExecMerge(node, resultRelInfo, estate, slot, junkfilter); diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c index 8848e9a03f..b2543f6814 100644 --- a/src/backend/optimizer/prep/preptlist.c +++ b/src/backend/optimizer/prep/preptlist.c @@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root) tlist = expand_targetlist(tlist, command_type, result_relation, target_relation); + /* + * For MERGE we need to handle the target list for the target relation, + * and also target list for each action (only INSERT/UPDATE matter). + */ if (command_type == CMD_MERGE) { ListCell *l; @@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type, * generate a NULL for dropped columns (we want to drop any old * values). * + * XXX Should this explain why MERGE has the same logic as UPDATE? + * * When generating a NULL constant for a dropped column, we label * it INT4 (any other guaranteed-to-exist datatype would do as * well). We can't label it with the dropped column's datatype diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index f7c152beb4..5d49c8c37e 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) break; case EXPR_KIND_MERGE_WHEN_AND: if (isAgg) + /* XXX "WHEN AND" seems rather strange. ... */ err = _("aggregate functions are not allowed in WHEN AND conditions"); else err = _("grouping operations are not allowed in WHEN AND conditions"); diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index d59f4fde95..d50543b1ba 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd) /* Also update the sent position status in shared memory */ SpinLockAcquire(&MyWalSnd->mutex); + /* XXX Huh? Why does MERGE patch change walsender? */ MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush; SpinLockRelease(&MyWalSnd->mutex); @@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data) replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ? MyWalSnd->write : MyWalSnd->flush; - if (WalSndCaughtUp && - sentPtr == replicatedPtr && + if (WalSndCaughtUp && sentPtr == replicatedPtr && !pq_is_send_pending()) { QueryCompletion qc; diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c index e2706c181c..80ae946d17 100644 --- a/src/backend/rewrite/rewriteHandler.c +++ b/src/backend/rewrite/rewriteHandler.c @@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation) } } + /* * matchLocks - * match the list of locks and returns the matching rules @@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events) /* * XXX MERGE doesn't support write rules because they would violate * the SQL Standard spec and would be unclear how they should work. + * + * XXX So does't support means 'ignores'? Should that either fail + * or at least print some warning? */ if (event == CMD_MERGE) product_queries = NIL; diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 0281351dcf..7a768a7b5b 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup) int result; bool arg1_match; + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup) memset(pres, 0, sizeof(Datum)); len = VARSIZE_ANY_EXHDR(authoritative); + /* XXX Huh? Why is this in MERGE patch? */ if (len < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2) len1 = VARSIZE_ANY_EXHDR(t1); len2 = VARSIZE_ANY_EXHDR(t2); + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 3cba38044e..08fea9b8f7 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -172,6 +172,7 @@ typedef struct Query List *withCheckOptions; /* a list of WithCheckOption's (added * during rewrite) */ + /* XXX Why not mergeTragetRelation? */ int mergeTarget_relation; List *mergeSourceTargetList; List *mergeActionList; /* list of actions for MERGE (only) */ @@ -1581,11 +1582,11 @@ typedef struct UpdateStmt typedef struct MergeStmt { NodeTag type; - RangeVar *relation; /* target relation to merge into */ + RangeVar *relation; /* target relation to merge into */ Node *source_relation; /* source relation */ - Node *join_condition; /* join condition between source and target */ + Node *join_condition; /* join condition between source and target */ List *mergeWhenClauses; /* list of MergeWhenClause(es) */ - WithClause *withClause; /* WITH clause */ + WithClause *withClause; /* WITH clause */ } MergeStmt; typedef struct MergeWhenClause -- 2.26.2 --------------B1AA4DC5EAE9BAF8B146244A Content-Type: text/plain; charset=UTF-8; name="0002-fix-valgrind-failure.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-fix-valgrind-failure.txt" ^ permalink raw reply [nested|flat] 56+ messages in thread
* [PATCH 1/2] review @ 2021-01-09 02:17 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 56+ messages in thread From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw) --- doc/src/sgml/mvcc.sgml | 20 +++++++++++--------- doc/src/sgml/ref/create_policy.sgml | 6 +++--- doc/src/sgml/ref/merge.sgml | 12 +++++++++--- src/backend/commands/explain.c | 2 +- src/backend/executor/execMerge.c | 5 ++++- src/backend/executor/nodeModifyTable.c | 3 ++- src/backend/optimizer/prep/preptlist.c | 6 ++++++ src/backend/parser/parse_agg.c | 1 + src/backend/replication/walsender.c | 4 ++-- src/backend/rewrite/rewriteHandler.c | 4 ++++ src/backend/utils/adt/varlena.c | 3 +++ src/include/nodes/parsenodes.h | 7 ++++--- 12 files changed, 50 insertions(+), 23 deletions(-) diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml index 9ec8474185..02c9f3fdea 100644 --- a/doc/src/sgml/mvcc.sgml +++ b/doc/src/sgml/mvcc.sgml @@ -429,22 +429,24 @@ COMMIT; with both <command>INSERT</command> and <command>UPDATE</command> subcommands looks similar to <command>INSERT</command> with an <literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee - that either <command>INSERT</command> and <command>UPDATE</command> will occur. + that either <command>INSERT</command> or <command>UPDATE</command> will occur. - If MERGE attempts an UPDATE or DELETE and the row is concurrently updated + /* XXX This is a very long and hard to understand sentence :-( */ + /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */ + If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated but the join condition still passes for the current target and the current - source tuple, then MERGE will behave the same as the UPDATE or DELETE commands + source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands and perform its action on the latest version of the row, using standard - EvalPlanQual. MERGE actions can be conditional, so conditions must be + EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be re-evaluated on the latest row, starting from the first action. On the other hand, if the row is concurrently updated or deleted so that - the join condition fails, then MERGE will execute a NOT MATCHED action, if it - exists and the AND WHEN qual evaluates to true. + the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it + exists and the <literal>AND WHEN</literal> qual evaluates to true. - If MERGE attempts an INSERT and a unique index is present and a duplicate - row is concurrently inserted then a uniqueness violation is raised. MERGE - does not attempt to avoid the ERROR by attempting an UPDATE. + If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate + row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command> + does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>. </para> <para> diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml index ad20230c58..0a64674f2d 100644 --- a/doc/src/sgml/ref/create_policy.sgml +++ b/doc/src/sgml/ref/create_policy.sgml @@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable <para> No separate policy exists for <command>MERGE</command>. Instead policies - defined for <literal>SELECT</literal>, <literal>INSERT</literal>, - <literal>UPDATE</literal> and <literal>DELETE</literal> are applied - while executing MERGE, depending on the actions that are activated. + defined for <command>SELECT</command>, <command>INSERT</command>, + <command>UPDATE</command> and <command>DELETE</command> are applied + while executing <command>MERGE</command>, depending on the actions that are activated. </para> </refsect1> diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml index b2a9f67cfa..c2901b0d58 100644 --- a/doc/src/sgml/ref/merge.sgml +++ b/doc/src/sgml/ref/merge.sgml @@ -98,7 +98,7 @@ DELETE </para> <para> - There is no MERGE privilege. + There is no <literal>MERGE</literal> privilege. You must have the <literal>UPDATE</literal> privilege on the column(s) of the <replaceable class="parameter">target_table_name</replaceable> referred to in the <literal>SET</literal> clause @@ -217,6 +217,7 @@ DELETE At least one <literal>WHEN</literal> clause is required. </para> <para> + /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */ If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal> and the candidate change row matches a row in the <replaceable class="parameter">target_table_name</replaceable> @@ -242,6 +243,7 @@ DELETE clause will be activated and the corresponding action will occur for that row. The expression may not contain functions that possibly performs writes to the database. + /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */ </para> <para> A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns @@ -379,6 +381,7 @@ DELETE <para> An expression to assign to the column. The expression can use the old values of this and other columns in the table. + /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */ </para> </listitem> </varlistentry> @@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> In summary, statement triggers for an event type (say, INSERT) will be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level triggers will fire only for the one event type <emphasis>activated</emphasis>. + /* XXX What does "activated" mean? Maybe "executed" would be better? */ So a <command>MERGE</command> might fire statement triggers for both <command>UPDATE</command> and <command>INSERT</command>, even though only <command>UPDATE</command> row triggers were fired. @@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable> rows will be used to modify the target row, later attempts to modify will cause an error. This can also occur if row triggers make changes to the target table which are then subsequently modified by <command>MERGE</command>. + /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */ + /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */ If the repeated action is an <command>INSERT</command> this will cause a uniqueness violation while a repeated <command>UPDATE</command> or <command>DELETE</command> will cause a cardinality violation; the latter behavior @@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> <title>Examples</title> <para> - Perform maintenance on CustomerAccounts based upon new Transactions. + Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>. <programlisting> MERGE CustomerAccount CA @@ -599,7 +605,7 @@ WHEN MATCHED THEN DELETE; </programlisting> - The wine_stock_changes table might be, for example, a temporary table + The <literal>wine_stock_changes</literal> table might be, for example, a temporary table recently loaded into the database. </para> diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index fbaf50c258..f914a00e9f 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors, break; case CMD_MERGE: operation = "Merge"; - foperation = "Foreign Merge"; + foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */ break; default: operation = "???"; diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c index 0e245e1361..a7492a6c4b 100644 --- a/src/backend/executor/execMerge.c +++ b/src/backend/executor/execMerge.c @@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * A concurrent update can: * * 1. modify the target tuple so that it no longer satisfies the - * additional quals attached to the current WHEN MATCHED action OR + * additional quals attached to the current WHEN MATCHED action * * In this case, we are still dealing with a WHEN MATCHED case, but we * should recheck the list of WHEN MATCHED actions and choose the first @@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * tuple, so we now instead find a qualifying WHEN NOT MATCHED action to * execute. * + * XXX Hmmm, what if the updated tuple would now match one that was + * considered NOT MATCHED so far? + * * A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED. * * ExecMergeMatched takes care of following the update chain and diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index c0a97eba91..9c14709e47 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate) { /* advance to next subplan if any */ node->mt_whichplan++; - if (node->mt_whichplan < node->mt_nplans) { resultRelInfo++; @@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate) EvalPlanQualSetSlot(&node->mt_epqstate, planSlot); slot = planSlot; + /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together + * with the other MT operations? This seems a bit out of place. */ if (operation == CMD_MERGE) { ExecMerge(node, resultRelInfo, estate, slot, junkfilter); diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c index 8848e9a03f..b2543f6814 100644 --- a/src/backend/optimizer/prep/preptlist.c +++ b/src/backend/optimizer/prep/preptlist.c @@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root) tlist = expand_targetlist(tlist, command_type, result_relation, target_relation); + /* + * For MERGE we need to handle the target list for the target relation, + * and also target list for each action (only INSERT/UPDATE matter). + */ if (command_type == CMD_MERGE) { ListCell *l; @@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type, * generate a NULL for dropped columns (we want to drop any old * values). * + * XXX Should this explain why MERGE has the same logic as UPDATE? + * * When generating a NULL constant for a dropped column, we label * it INT4 (any other guaranteed-to-exist datatype would do as * well). We can't label it with the dropped column's datatype diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index f7c152beb4..5d49c8c37e 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) break; case EXPR_KIND_MERGE_WHEN_AND: if (isAgg) + /* XXX "WHEN AND" seems rather strange. ... */ err = _("aggregate functions are not allowed in WHEN AND conditions"); else err = _("grouping operations are not allowed in WHEN AND conditions"); diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index d59f4fde95..d50543b1ba 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd) /* Also update the sent position status in shared memory */ SpinLockAcquire(&MyWalSnd->mutex); + /* XXX Huh? Why does MERGE patch change walsender? */ MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush; SpinLockRelease(&MyWalSnd->mutex); @@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data) replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ? MyWalSnd->write : MyWalSnd->flush; - if (WalSndCaughtUp && - sentPtr == replicatedPtr && + if (WalSndCaughtUp && sentPtr == replicatedPtr && !pq_is_send_pending()) { QueryCompletion qc; diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c index e2706c181c..80ae946d17 100644 --- a/src/backend/rewrite/rewriteHandler.c +++ b/src/backend/rewrite/rewriteHandler.c @@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation) } } + /* * matchLocks - * match the list of locks and returns the matching rules @@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events) /* * XXX MERGE doesn't support write rules because they would violate * the SQL Standard spec and would be unclear how they should work. + * + * XXX So does't support means 'ignores'? Should that either fail + * or at least print some warning? */ if (event == CMD_MERGE) product_queries = NIL; diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 0281351dcf..7a768a7b5b 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup) int result; bool arg1_match; + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup) memset(pres, 0, sizeof(Datum)); len = VARSIZE_ANY_EXHDR(authoritative); + /* XXX Huh? Why is this in MERGE patch? */ if (len < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2) len1 = VARSIZE_ANY_EXHDR(t1); len2 = VARSIZE_ANY_EXHDR(t2); + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 3cba38044e..08fea9b8f7 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -172,6 +172,7 @@ typedef struct Query List *withCheckOptions; /* a list of WithCheckOption's (added * during rewrite) */ + /* XXX Why not mergeTragetRelation? */ int mergeTarget_relation; List *mergeSourceTargetList; List *mergeActionList; /* list of actions for MERGE (only) */ @@ -1581,11 +1582,11 @@ typedef struct UpdateStmt typedef struct MergeStmt { NodeTag type; - RangeVar *relation; /* target relation to merge into */ + RangeVar *relation; /* target relation to merge into */ Node *source_relation; /* source relation */ - Node *join_condition; /* join condition between source and target */ + Node *join_condition; /* join condition between source and target */ List *mergeWhenClauses; /* list of MergeWhenClause(es) */ - WithClause *withClause; /* WITH clause */ + WithClause *withClause; /* WITH clause */ } MergeStmt; typedef struct MergeWhenClause -- 2.26.2 --------------B1AA4DC5EAE9BAF8B146244A Content-Type: text/plain; charset=UTF-8; name="0002-fix-valgrind-failure.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-fix-valgrind-failure.txt" ^ permalink raw reply [nested|flat] 56+ messages in thread
* [PATCH 1/2] review @ 2021-01-09 02:17 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 56+ messages in thread From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw) --- doc/src/sgml/mvcc.sgml | 20 +++++++++++--------- doc/src/sgml/ref/create_policy.sgml | 6 +++--- doc/src/sgml/ref/merge.sgml | 12 +++++++++--- src/backend/commands/explain.c | 2 +- src/backend/executor/execMerge.c | 5 ++++- src/backend/executor/nodeModifyTable.c | 3 ++- src/backend/optimizer/prep/preptlist.c | 6 ++++++ src/backend/parser/parse_agg.c | 1 + src/backend/replication/walsender.c | 4 ++-- src/backend/rewrite/rewriteHandler.c | 4 ++++ src/backend/utils/adt/varlena.c | 3 +++ src/include/nodes/parsenodes.h | 7 ++++--- 12 files changed, 50 insertions(+), 23 deletions(-) diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml index 9ec8474185..02c9f3fdea 100644 --- a/doc/src/sgml/mvcc.sgml +++ b/doc/src/sgml/mvcc.sgml @@ -429,22 +429,24 @@ COMMIT; with both <command>INSERT</command> and <command>UPDATE</command> subcommands looks similar to <command>INSERT</command> with an <literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee - that either <command>INSERT</command> and <command>UPDATE</command> will occur. + that either <command>INSERT</command> or <command>UPDATE</command> will occur. - If MERGE attempts an UPDATE or DELETE and the row is concurrently updated + /* XXX This is a very long and hard to understand sentence :-( */ + /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */ + If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated but the join condition still passes for the current target and the current - source tuple, then MERGE will behave the same as the UPDATE or DELETE commands + source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands and perform its action on the latest version of the row, using standard - EvalPlanQual. MERGE actions can be conditional, so conditions must be + EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be re-evaluated on the latest row, starting from the first action. On the other hand, if the row is concurrently updated or deleted so that - the join condition fails, then MERGE will execute a NOT MATCHED action, if it - exists and the AND WHEN qual evaluates to true. + the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it + exists and the <literal>AND WHEN</literal> qual evaluates to true. - If MERGE attempts an INSERT and a unique index is present and a duplicate - row is concurrently inserted then a uniqueness violation is raised. MERGE - does not attempt to avoid the ERROR by attempting an UPDATE. + If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate + row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command> + does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>. </para> <para> diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml index ad20230c58..0a64674f2d 100644 --- a/doc/src/sgml/ref/create_policy.sgml +++ b/doc/src/sgml/ref/create_policy.sgml @@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable <para> No separate policy exists for <command>MERGE</command>. Instead policies - defined for <literal>SELECT</literal>, <literal>INSERT</literal>, - <literal>UPDATE</literal> and <literal>DELETE</literal> are applied - while executing MERGE, depending on the actions that are activated. + defined for <command>SELECT</command>, <command>INSERT</command>, + <command>UPDATE</command> and <command>DELETE</command> are applied + while executing <command>MERGE</command>, depending on the actions that are activated. </para> </refsect1> diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml index b2a9f67cfa..c2901b0d58 100644 --- a/doc/src/sgml/ref/merge.sgml +++ b/doc/src/sgml/ref/merge.sgml @@ -98,7 +98,7 @@ DELETE </para> <para> - There is no MERGE privilege. + There is no <literal>MERGE</literal> privilege. You must have the <literal>UPDATE</literal> privilege on the column(s) of the <replaceable class="parameter">target_table_name</replaceable> referred to in the <literal>SET</literal> clause @@ -217,6 +217,7 @@ DELETE At least one <literal>WHEN</literal> clause is required. </para> <para> + /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */ If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal> and the candidate change row matches a row in the <replaceable class="parameter">target_table_name</replaceable> @@ -242,6 +243,7 @@ DELETE clause will be activated and the corresponding action will occur for that row. The expression may not contain functions that possibly performs writes to the database. + /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */ </para> <para> A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns @@ -379,6 +381,7 @@ DELETE <para> An expression to assign to the column. The expression can use the old values of this and other columns in the table. + /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */ </para> </listitem> </varlistentry> @@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> In summary, statement triggers for an event type (say, INSERT) will be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level triggers will fire only for the one event type <emphasis>activated</emphasis>. + /* XXX What does "activated" mean? Maybe "executed" would be better? */ So a <command>MERGE</command> might fire statement triggers for both <command>UPDATE</command> and <command>INSERT</command>, even though only <command>UPDATE</command> row triggers were fired. @@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable> rows will be used to modify the target row, later attempts to modify will cause an error. This can also occur if row triggers make changes to the target table which are then subsequently modified by <command>MERGE</command>. + /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */ + /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */ If the repeated action is an <command>INSERT</command> this will cause a uniqueness violation while a repeated <command>UPDATE</command> or <command>DELETE</command> will cause a cardinality violation; the latter behavior @@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> <title>Examples</title> <para> - Perform maintenance on CustomerAccounts based upon new Transactions. + Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>. <programlisting> MERGE CustomerAccount CA @@ -599,7 +605,7 @@ WHEN MATCHED THEN DELETE; </programlisting> - The wine_stock_changes table might be, for example, a temporary table + The <literal>wine_stock_changes</literal> table might be, for example, a temporary table recently loaded into the database. </para> diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index fbaf50c258..f914a00e9f 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors, break; case CMD_MERGE: operation = "Merge"; - foperation = "Foreign Merge"; + foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */ break; default: operation = "???"; diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c index 0e245e1361..a7492a6c4b 100644 --- a/src/backend/executor/execMerge.c +++ b/src/backend/executor/execMerge.c @@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * A concurrent update can: * * 1. modify the target tuple so that it no longer satisfies the - * additional quals attached to the current WHEN MATCHED action OR + * additional quals attached to the current WHEN MATCHED action * * In this case, we are still dealing with a WHEN MATCHED case, but we * should recheck the list of WHEN MATCHED actions and choose the first @@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * tuple, so we now instead find a qualifying WHEN NOT MATCHED action to * execute. * + * XXX Hmmm, what if the updated tuple would now match one that was + * considered NOT MATCHED so far? + * * A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED. * * ExecMergeMatched takes care of following the update chain and diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index c0a97eba91..9c14709e47 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate) { /* advance to next subplan if any */ node->mt_whichplan++; - if (node->mt_whichplan < node->mt_nplans) { resultRelInfo++; @@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate) EvalPlanQualSetSlot(&node->mt_epqstate, planSlot); slot = planSlot; + /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together + * with the other MT operations? This seems a bit out of place. */ if (operation == CMD_MERGE) { ExecMerge(node, resultRelInfo, estate, slot, junkfilter); diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c index 8848e9a03f..b2543f6814 100644 --- a/src/backend/optimizer/prep/preptlist.c +++ b/src/backend/optimizer/prep/preptlist.c @@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root) tlist = expand_targetlist(tlist, command_type, result_relation, target_relation); + /* + * For MERGE we need to handle the target list for the target relation, + * and also target list for each action (only INSERT/UPDATE matter). + */ if (command_type == CMD_MERGE) { ListCell *l; @@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type, * generate a NULL for dropped columns (we want to drop any old * values). * + * XXX Should this explain why MERGE has the same logic as UPDATE? + * * When generating a NULL constant for a dropped column, we label * it INT4 (any other guaranteed-to-exist datatype would do as * well). We can't label it with the dropped column's datatype diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index f7c152beb4..5d49c8c37e 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) break; case EXPR_KIND_MERGE_WHEN_AND: if (isAgg) + /* XXX "WHEN AND" seems rather strange. ... */ err = _("aggregate functions are not allowed in WHEN AND conditions"); else err = _("grouping operations are not allowed in WHEN AND conditions"); diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index d59f4fde95..d50543b1ba 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd) /* Also update the sent position status in shared memory */ SpinLockAcquire(&MyWalSnd->mutex); + /* XXX Huh? Why does MERGE patch change walsender? */ MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush; SpinLockRelease(&MyWalSnd->mutex); @@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data) replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ? MyWalSnd->write : MyWalSnd->flush; - if (WalSndCaughtUp && - sentPtr == replicatedPtr && + if (WalSndCaughtUp && sentPtr == replicatedPtr && !pq_is_send_pending()) { QueryCompletion qc; diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c index e2706c181c..80ae946d17 100644 --- a/src/backend/rewrite/rewriteHandler.c +++ b/src/backend/rewrite/rewriteHandler.c @@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation) } } + /* * matchLocks - * match the list of locks and returns the matching rules @@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events) /* * XXX MERGE doesn't support write rules because they would violate * the SQL Standard spec and would be unclear how they should work. + * + * XXX So does't support means 'ignores'? Should that either fail + * or at least print some warning? */ if (event == CMD_MERGE) product_queries = NIL; diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 0281351dcf..7a768a7b5b 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup) int result; bool arg1_match; + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup) memset(pres, 0, sizeof(Datum)); len = VARSIZE_ANY_EXHDR(authoritative); + /* XXX Huh? Why is this in MERGE patch? */ if (len < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2) len1 = VARSIZE_ANY_EXHDR(t1); len2 = VARSIZE_ANY_EXHDR(t2); + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 3cba38044e..08fea9b8f7 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -172,6 +172,7 @@ typedef struct Query List *withCheckOptions; /* a list of WithCheckOption's (added * during rewrite) */ + /* XXX Why not mergeTragetRelation? */ int mergeTarget_relation; List *mergeSourceTargetList; List *mergeActionList; /* list of actions for MERGE (only) */ @@ -1581,11 +1582,11 @@ typedef struct UpdateStmt typedef struct MergeStmt { NodeTag type; - RangeVar *relation; /* target relation to merge into */ + RangeVar *relation; /* target relation to merge into */ Node *source_relation; /* source relation */ - Node *join_condition; /* join condition between source and target */ + Node *join_condition; /* join condition between source and target */ List *mergeWhenClauses; /* list of MergeWhenClause(es) */ - WithClause *withClause; /* WITH clause */ + WithClause *withClause; /* WITH clause */ } MergeStmt; typedef struct MergeWhenClause -- 2.26.2 --------------B1AA4DC5EAE9BAF8B146244A Content-Type: text/plain; charset=UTF-8; name="0002-fix-valgrind-failure.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-fix-valgrind-failure.txt" ^ permalink raw reply [nested|flat] 56+ messages in thread
* [PATCH 1/2] review @ 2021-01-09 02:17 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 56+ messages in thread From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw) --- doc/src/sgml/mvcc.sgml | 20 +++++++++++--------- doc/src/sgml/ref/create_policy.sgml | 6 +++--- doc/src/sgml/ref/merge.sgml | 12 +++++++++--- src/backend/commands/explain.c | 2 +- src/backend/executor/execMerge.c | 5 ++++- src/backend/executor/nodeModifyTable.c | 3 ++- src/backend/optimizer/prep/preptlist.c | 6 ++++++ src/backend/parser/parse_agg.c | 1 + src/backend/replication/walsender.c | 4 ++-- src/backend/rewrite/rewriteHandler.c | 4 ++++ src/backend/utils/adt/varlena.c | 3 +++ src/include/nodes/parsenodes.h | 7 ++++--- 12 files changed, 50 insertions(+), 23 deletions(-) diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml index 9ec8474185..02c9f3fdea 100644 --- a/doc/src/sgml/mvcc.sgml +++ b/doc/src/sgml/mvcc.sgml @@ -429,22 +429,24 @@ COMMIT; with both <command>INSERT</command> and <command>UPDATE</command> subcommands looks similar to <command>INSERT</command> with an <literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee - that either <command>INSERT</command> and <command>UPDATE</command> will occur. + that either <command>INSERT</command> or <command>UPDATE</command> will occur. - If MERGE attempts an UPDATE or DELETE and the row is concurrently updated + /* XXX This is a very long and hard to understand sentence :-( */ + /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */ + If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated but the join condition still passes for the current target and the current - source tuple, then MERGE will behave the same as the UPDATE or DELETE commands + source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands and perform its action on the latest version of the row, using standard - EvalPlanQual. MERGE actions can be conditional, so conditions must be + EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be re-evaluated on the latest row, starting from the first action. On the other hand, if the row is concurrently updated or deleted so that - the join condition fails, then MERGE will execute a NOT MATCHED action, if it - exists and the AND WHEN qual evaluates to true. + the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it + exists and the <literal>AND WHEN</literal> qual evaluates to true. - If MERGE attempts an INSERT and a unique index is present and a duplicate - row is concurrently inserted then a uniqueness violation is raised. MERGE - does not attempt to avoid the ERROR by attempting an UPDATE. + If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate + row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command> + does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>. </para> <para> diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml index ad20230c58..0a64674f2d 100644 --- a/doc/src/sgml/ref/create_policy.sgml +++ b/doc/src/sgml/ref/create_policy.sgml @@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable <para> No separate policy exists for <command>MERGE</command>. Instead policies - defined for <literal>SELECT</literal>, <literal>INSERT</literal>, - <literal>UPDATE</literal> and <literal>DELETE</literal> are applied - while executing MERGE, depending on the actions that are activated. + defined for <command>SELECT</command>, <command>INSERT</command>, + <command>UPDATE</command> and <command>DELETE</command> are applied + while executing <command>MERGE</command>, depending on the actions that are activated. </para> </refsect1> diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml index b2a9f67cfa..c2901b0d58 100644 --- a/doc/src/sgml/ref/merge.sgml +++ b/doc/src/sgml/ref/merge.sgml @@ -98,7 +98,7 @@ DELETE </para> <para> - There is no MERGE privilege. + There is no <literal>MERGE</literal> privilege. You must have the <literal>UPDATE</literal> privilege on the column(s) of the <replaceable class="parameter">target_table_name</replaceable> referred to in the <literal>SET</literal> clause @@ -217,6 +217,7 @@ DELETE At least one <literal>WHEN</literal> clause is required. </para> <para> + /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */ If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal> and the candidate change row matches a row in the <replaceable class="parameter">target_table_name</replaceable> @@ -242,6 +243,7 @@ DELETE clause will be activated and the corresponding action will occur for that row. The expression may not contain functions that possibly performs writes to the database. + /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */ </para> <para> A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns @@ -379,6 +381,7 @@ DELETE <para> An expression to assign to the column. The expression can use the old values of this and other columns in the table. + /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */ </para> </listitem> </varlistentry> @@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> In summary, statement triggers for an event type (say, INSERT) will be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level triggers will fire only for the one event type <emphasis>activated</emphasis>. + /* XXX What does "activated" mean? Maybe "executed" would be better? */ So a <command>MERGE</command> might fire statement triggers for both <command>UPDATE</command> and <command>INSERT</command>, even though only <command>UPDATE</command> row triggers were fired. @@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable> rows will be used to modify the target row, later attempts to modify will cause an error. This can also occur if row triggers make changes to the target table which are then subsequently modified by <command>MERGE</command>. + /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */ + /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */ If the repeated action is an <command>INSERT</command> this will cause a uniqueness violation while a repeated <command>UPDATE</command> or <command>DELETE</command> will cause a cardinality violation; the latter behavior @@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> <title>Examples</title> <para> - Perform maintenance on CustomerAccounts based upon new Transactions. + Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>. <programlisting> MERGE CustomerAccount CA @@ -599,7 +605,7 @@ WHEN MATCHED THEN DELETE; </programlisting> - The wine_stock_changes table might be, for example, a temporary table + The <literal>wine_stock_changes</literal> table might be, for example, a temporary table recently loaded into the database. </para> diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index fbaf50c258..f914a00e9f 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors, break; case CMD_MERGE: operation = "Merge"; - foperation = "Foreign Merge"; + foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */ break; default: operation = "???"; diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c index 0e245e1361..a7492a6c4b 100644 --- a/src/backend/executor/execMerge.c +++ b/src/backend/executor/execMerge.c @@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * A concurrent update can: * * 1. modify the target tuple so that it no longer satisfies the - * additional quals attached to the current WHEN MATCHED action OR + * additional quals attached to the current WHEN MATCHED action * * In this case, we are still dealing with a WHEN MATCHED case, but we * should recheck the list of WHEN MATCHED actions and choose the first @@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * tuple, so we now instead find a qualifying WHEN NOT MATCHED action to * execute. * + * XXX Hmmm, what if the updated tuple would now match one that was + * considered NOT MATCHED so far? + * * A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED. * * ExecMergeMatched takes care of following the update chain and diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index c0a97eba91..9c14709e47 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate) { /* advance to next subplan if any */ node->mt_whichplan++; - if (node->mt_whichplan < node->mt_nplans) { resultRelInfo++; @@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate) EvalPlanQualSetSlot(&node->mt_epqstate, planSlot); slot = planSlot; + /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together + * with the other MT operations? This seems a bit out of place. */ if (operation == CMD_MERGE) { ExecMerge(node, resultRelInfo, estate, slot, junkfilter); diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c index 8848e9a03f..b2543f6814 100644 --- a/src/backend/optimizer/prep/preptlist.c +++ b/src/backend/optimizer/prep/preptlist.c @@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root) tlist = expand_targetlist(tlist, command_type, result_relation, target_relation); + /* + * For MERGE we need to handle the target list for the target relation, + * and also target list for each action (only INSERT/UPDATE matter). + */ if (command_type == CMD_MERGE) { ListCell *l; @@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type, * generate a NULL for dropped columns (we want to drop any old * values). * + * XXX Should this explain why MERGE has the same logic as UPDATE? + * * When generating a NULL constant for a dropped column, we label * it INT4 (any other guaranteed-to-exist datatype would do as * well). We can't label it with the dropped column's datatype diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index f7c152beb4..5d49c8c37e 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) break; case EXPR_KIND_MERGE_WHEN_AND: if (isAgg) + /* XXX "WHEN AND" seems rather strange. ... */ err = _("aggregate functions are not allowed in WHEN AND conditions"); else err = _("grouping operations are not allowed in WHEN AND conditions"); diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index d59f4fde95..d50543b1ba 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd) /* Also update the sent position status in shared memory */ SpinLockAcquire(&MyWalSnd->mutex); + /* XXX Huh? Why does MERGE patch change walsender? */ MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush; SpinLockRelease(&MyWalSnd->mutex); @@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data) replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ? MyWalSnd->write : MyWalSnd->flush; - if (WalSndCaughtUp && - sentPtr == replicatedPtr && + if (WalSndCaughtUp && sentPtr == replicatedPtr && !pq_is_send_pending()) { QueryCompletion qc; diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c index e2706c181c..80ae946d17 100644 --- a/src/backend/rewrite/rewriteHandler.c +++ b/src/backend/rewrite/rewriteHandler.c @@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation) } } + /* * matchLocks - * match the list of locks and returns the matching rules @@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events) /* * XXX MERGE doesn't support write rules because they would violate * the SQL Standard spec and would be unclear how they should work. + * + * XXX So does't support means 'ignores'? Should that either fail + * or at least print some warning? */ if (event == CMD_MERGE) product_queries = NIL; diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 0281351dcf..7a768a7b5b 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup) int result; bool arg1_match; + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup) memset(pres, 0, sizeof(Datum)); len = VARSIZE_ANY_EXHDR(authoritative); + /* XXX Huh? Why is this in MERGE patch? */ if (len < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2) len1 = VARSIZE_ANY_EXHDR(t1); len2 = VARSIZE_ANY_EXHDR(t2); + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 3cba38044e..08fea9b8f7 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -172,6 +172,7 @@ typedef struct Query List *withCheckOptions; /* a list of WithCheckOption's (added * during rewrite) */ + /* XXX Why not mergeTragetRelation? */ int mergeTarget_relation; List *mergeSourceTargetList; List *mergeActionList; /* list of actions for MERGE (only) */ @@ -1581,11 +1582,11 @@ typedef struct UpdateStmt typedef struct MergeStmt { NodeTag type; - RangeVar *relation; /* target relation to merge into */ + RangeVar *relation; /* target relation to merge into */ Node *source_relation; /* source relation */ - Node *join_condition; /* join condition between source and target */ + Node *join_condition; /* join condition between source and target */ List *mergeWhenClauses; /* list of MergeWhenClause(es) */ - WithClause *withClause; /* WITH clause */ + WithClause *withClause; /* WITH clause */ } MergeStmt; typedef struct MergeWhenClause -- 2.26.2 --------------B1AA4DC5EAE9BAF8B146244A Content-Type: text/plain; charset=UTF-8; name="0002-fix-valgrind-failure.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-fix-valgrind-failure.txt" ^ permalink raw reply [nested|flat] 56+ messages in thread
* [PATCH 1/2] review @ 2021-01-09 02:17 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 56+ messages in thread From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw) --- doc/src/sgml/mvcc.sgml | 20 +++++++++++--------- doc/src/sgml/ref/create_policy.sgml | 6 +++--- doc/src/sgml/ref/merge.sgml | 12 +++++++++--- src/backend/commands/explain.c | 2 +- src/backend/executor/execMerge.c | 5 ++++- src/backend/executor/nodeModifyTable.c | 3 ++- src/backend/optimizer/prep/preptlist.c | 6 ++++++ src/backend/parser/parse_agg.c | 1 + src/backend/replication/walsender.c | 4 ++-- src/backend/rewrite/rewriteHandler.c | 4 ++++ src/backend/utils/adt/varlena.c | 3 +++ src/include/nodes/parsenodes.h | 7 ++++--- 12 files changed, 50 insertions(+), 23 deletions(-) diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml index 9ec8474185..02c9f3fdea 100644 --- a/doc/src/sgml/mvcc.sgml +++ b/doc/src/sgml/mvcc.sgml @@ -429,22 +429,24 @@ COMMIT; with both <command>INSERT</command> and <command>UPDATE</command> subcommands looks similar to <command>INSERT</command> with an <literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee - that either <command>INSERT</command> and <command>UPDATE</command> will occur. + that either <command>INSERT</command> or <command>UPDATE</command> will occur. - If MERGE attempts an UPDATE or DELETE and the row is concurrently updated + /* XXX This is a very long and hard to understand sentence :-( */ + /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */ + If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated but the join condition still passes for the current target and the current - source tuple, then MERGE will behave the same as the UPDATE or DELETE commands + source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands and perform its action on the latest version of the row, using standard - EvalPlanQual. MERGE actions can be conditional, so conditions must be + EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be re-evaluated on the latest row, starting from the first action. On the other hand, if the row is concurrently updated or deleted so that - the join condition fails, then MERGE will execute a NOT MATCHED action, if it - exists and the AND WHEN qual evaluates to true. + the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it + exists and the <literal>AND WHEN</literal> qual evaluates to true. - If MERGE attempts an INSERT and a unique index is present and a duplicate - row is concurrently inserted then a uniqueness violation is raised. MERGE - does not attempt to avoid the ERROR by attempting an UPDATE. + If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate + row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command> + does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>. </para> <para> diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml index ad20230c58..0a64674f2d 100644 --- a/doc/src/sgml/ref/create_policy.sgml +++ b/doc/src/sgml/ref/create_policy.sgml @@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable <para> No separate policy exists for <command>MERGE</command>. Instead policies - defined for <literal>SELECT</literal>, <literal>INSERT</literal>, - <literal>UPDATE</literal> and <literal>DELETE</literal> are applied - while executing MERGE, depending on the actions that are activated. + defined for <command>SELECT</command>, <command>INSERT</command>, + <command>UPDATE</command> and <command>DELETE</command> are applied + while executing <command>MERGE</command>, depending on the actions that are activated. </para> </refsect1> diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml index b2a9f67cfa..c2901b0d58 100644 --- a/doc/src/sgml/ref/merge.sgml +++ b/doc/src/sgml/ref/merge.sgml @@ -98,7 +98,7 @@ DELETE </para> <para> - There is no MERGE privilege. + There is no <literal>MERGE</literal> privilege. You must have the <literal>UPDATE</literal> privilege on the column(s) of the <replaceable class="parameter">target_table_name</replaceable> referred to in the <literal>SET</literal> clause @@ -217,6 +217,7 @@ DELETE At least one <literal>WHEN</literal> clause is required. </para> <para> + /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */ If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal> and the candidate change row matches a row in the <replaceable class="parameter">target_table_name</replaceable> @@ -242,6 +243,7 @@ DELETE clause will be activated and the corresponding action will occur for that row. The expression may not contain functions that possibly performs writes to the database. + /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */ </para> <para> A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns @@ -379,6 +381,7 @@ DELETE <para> An expression to assign to the column. The expression can use the old values of this and other columns in the table. + /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */ </para> </listitem> </varlistentry> @@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> In summary, statement triggers for an event type (say, INSERT) will be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level triggers will fire only for the one event type <emphasis>activated</emphasis>. + /* XXX What does "activated" mean? Maybe "executed" would be better? */ So a <command>MERGE</command> might fire statement triggers for both <command>UPDATE</command> and <command>INSERT</command>, even though only <command>UPDATE</command> row triggers were fired. @@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable> rows will be used to modify the target row, later attempts to modify will cause an error. This can also occur if row triggers make changes to the target table which are then subsequently modified by <command>MERGE</command>. + /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */ + /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */ If the repeated action is an <command>INSERT</command> this will cause a uniqueness violation while a repeated <command>UPDATE</command> or <command>DELETE</command> will cause a cardinality violation; the latter behavior @@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> <title>Examples</title> <para> - Perform maintenance on CustomerAccounts based upon new Transactions. + Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>. <programlisting> MERGE CustomerAccount CA @@ -599,7 +605,7 @@ WHEN MATCHED THEN DELETE; </programlisting> - The wine_stock_changes table might be, for example, a temporary table + The <literal>wine_stock_changes</literal> table might be, for example, a temporary table recently loaded into the database. </para> diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index fbaf50c258..f914a00e9f 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors, break; case CMD_MERGE: operation = "Merge"; - foperation = "Foreign Merge"; + foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */ break; default: operation = "???"; diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c index 0e245e1361..a7492a6c4b 100644 --- a/src/backend/executor/execMerge.c +++ b/src/backend/executor/execMerge.c @@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * A concurrent update can: * * 1. modify the target tuple so that it no longer satisfies the - * additional quals attached to the current WHEN MATCHED action OR + * additional quals attached to the current WHEN MATCHED action * * In this case, we are still dealing with a WHEN MATCHED case, but we * should recheck the list of WHEN MATCHED actions and choose the first @@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * tuple, so we now instead find a qualifying WHEN NOT MATCHED action to * execute. * + * XXX Hmmm, what if the updated tuple would now match one that was + * considered NOT MATCHED so far? + * * A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED. * * ExecMergeMatched takes care of following the update chain and diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index c0a97eba91..9c14709e47 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate) { /* advance to next subplan if any */ node->mt_whichplan++; - if (node->mt_whichplan < node->mt_nplans) { resultRelInfo++; @@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate) EvalPlanQualSetSlot(&node->mt_epqstate, planSlot); slot = planSlot; + /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together + * with the other MT operations? This seems a bit out of place. */ if (operation == CMD_MERGE) { ExecMerge(node, resultRelInfo, estate, slot, junkfilter); diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c index 8848e9a03f..b2543f6814 100644 --- a/src/backend/optimizer/prep/preptlist.c +++ b/src/backend/optimizer/prep/preptlist.c @@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root) tlist = expand_targetlist(tlist, command_type, result_relation, target_relation); + /* + * For MERGE we need to handle the target list for the target relation, + * and also target list for each action (only INSERT/UPDATE matter). + */ if (command_type == CMD_MERGE) { ListCell *l; @@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type, * generate a NULL for dropped columns (we want to drop any old * values). * + * XXX Should this explain why MERGE has the same logic as UPDATE? + * * When generating a NULL constant for a dropped column, we label * it INT4 (any other guaranteed-to-exist datatype would do as * well). We can't label it with the dropped column's datatype diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index f7c152beb4..5d49c8c37e 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) break; case EXPR_KIND_MERGE_WHEN_AND: if (isAgg) + /* XXX "WHEN AND" seems rather strange. ... */ err = _("aggregate functions are not allowed in WHEN AND conditions"); else err = _("grouping operations are not allowed in WHEN AND conditions"); diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index d59f4fde95..d50543b1ba 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd) /* Also update the sent position status in shared memory */ SpinLockAcquire(&MyWalSnd->mutex); + /* XXX Huh? Why does MERGE patch change walsender? */ MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush; SpinLockRelease(&MyWalSnd->mutex); @@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data) replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ? MyWalSnd->write : MyWalSnd->flush; - if (WalSndCaughtUp && - sentPtr == replicatedPtr && + if (WalSndCaughtUp && sentPtr == replicatedPtr && !pq_is_send_pending()) { QueryCompletion qc; diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c index e2706c181c..80ae946d17 100644 --- a/src/backend/rewrite/rewriteHandler.c +++ b/src/backend/rewrite/rewriteHandler.c @@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation) } } + /* * matchLocks - * match the list of locks and returns the matching rules @@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events) /* * XXX MERGE doesn't support write rules because they would violate * the SQL Standard spec and would be unclear how they should work. + * + * XXX So does't support means 'ignores'? Should that either fail + * or at least print some warning? */ if (event == CMD_MERGE) product_queries = NIL; diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 0281351dcf..7a768a7b5b 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup) int result; bool arg1_match; + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup) memset(pres, 0, sizeof(Datum)); len = VARSIZE_ANY_EXHDR(authoritative); + /* XXX Huh? Why is this in MERGE patch? */ if (len < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2) len1 = VARSIZE_ANY_EXHDR(t1); len2 = VARSIZE_ANY_EXHDR(t2); + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 3cba38044e..08fea9b8f7 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -172,6 +172,7 @@ typedef struct Query List *withCheckOptions; /* a list of WithCheckOption's (added * during rewrite) */ + /* XXX Why not mergeTragetRelation? */ int mergeTarget_relation; List *mergeSourceTargetList; List *mergeActionList; /* list of actions for MERGE (only) */ @@ -1581,11 +1582,11 @@ typedef struct UpdateStmt typedef struct MergeStmt { NodeTag type; - RangeVar *relation; /* target relation to merge into */ + RangeVar *relation; /* target relation to merge into */ Node *source_relation; /* source relation */ - Node *join_condition; /* join condition between source and target */ + Node *join_condition; /* join condition between source and target */ List *mergeWhenClauses; /* list of MergeWhenClause(es) */ - WithClause *withClause; /* WITH clause */ + WithClause *withClause; /* WITH clause */ } MergeStmt; typedef struct MergeWhenClause -- 2.26.2 --------------B1AA4DC5EAE9BAF8B146244A Content-Type: text/plain; charset=UTF-8; name="0002-fix-valgrind-failure.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-fix-valgrind-failure.txt" ^ permalink raw reply [nested|flat] 56+ messages in thread
* [PATCH 1/2] review @ 2021-01-09 02:17 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 56+ messages in thread From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw) --- doc/src/sgml/mvcc.sgml | 20 +++++++++++--------- doc/src/sgml/ref/create_policy.sgml | 6 +++--- doc/src/sgml/ref/merge.sgml | 12 +++++++++--- src/backend/commands/explain.c | 2 +- src/backend/executor/execMerge.c | 5 ++++- src/backend/executor/nodeModifyTable.c | 3 ++- src/backend/optimizer/prep/preptlist.c | 6 ++++++ src/backend/parser/parse_agg.c | 1 + src/backend/replication/walsender.c | 4 ++-- src/backend/rewrite/rewriteHandler.c | 4 ++++ src/backend/utils/adt/varlena.c | 3 +++ src/include/nodes/parsenodes.h | 7 ++++--- 12 files changed, 50 insertions(+), 23 deletions(-) diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml index 9ec8474185..02c9f3fdea 100644 --- a/doc/src/sgml/mvcc.sgml +++ b/doc/src/sgml/mvcc.sgml @@ -429,22 +429,24 @@ COMMIT; with both <command>INSERT</command> and <command>UPDATE</command> subcommands looks similar to <command>INSERT</command> with an <literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee - that either <command>INSERT</command> and <command>UPDATE</command> will occur. + that either <command>INSERT</command> or <command>UPDATE</command> will occur. - If MERGE attempts an UPDATE or DELETE and the row is concurrently updated + /* XXX This is a very long and hard to understand sentence :-( */ + /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */ + If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated but the join condition still passes for the current target and the current - source tuple, then MERGE will behave the same as the UPDATE or DELETE commands + source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands and perform its action on the latest version of the row, using standard - EvalPlanQual. MERGE actions can be conditional, so conditions must be + EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be re-evaluated on the latest row, starting from the first action. On the other hand, if the row is concurrently updated or deleted so that - the join condition fails, then MERGE will execute a NOT MATCHED action, if it - exists and the AND WHEN qual evaluates to true. + the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it + exists and the <literal>AND WHEN</literal> qual evaluates to true. - If MERGE attempts an INSERT and a unique index is present and a duplicate - row is concurrently inserted then a uniqueness violation is raised. MERGE - does not attempt to avoid the ERROR by attempting an UPDATE. + If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate + row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command> + does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>. </para> <para> diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml index ad20230c58..0a64674f2d 100644 --- a/doc/src/sgml/ref/create_policy.sgml +++ b/doc/src/sgml/ref/create_policy.sgml @@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable <para> No separate policy exists for <command>MERGE</command>. Instead policies - defined for <literal>SELECT</literal>, <literal>INSERT</literal>, - <literal>UPDATE</literal> and <literal>DELETE</literal> are applied - while executing MERGE, depending on the actions that are activated. + defined for <command>SELECT</command>, <command>INSERT</command>, + <command>UPDATE</command> and <command>DELETE</command> are applied + while executing <command>MERGE</command>, depending on the actions that are activated. </para> </refsect1> diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml index b2a9f67cfa..c2901b0d58 100644 --- a/doc/src/sgml/ref/merge.sgml +++ b/doc/src/sgml/ref/merge.sgml @@ -98,7 +98,7 @@ DELETE </para> <para> - There is no MERGE privilege. + There is no <literal>MERGE</literal> privilege. You must have the <literal>UPDATE</literal> privilege on the column(s) of the <replaceable class="parameter">target_table_name</replaceable> referred to in the <literal>SET</literal> clause @@ -217,6 +217,7 @@ DELETE At least one <literal>WHEN</literal> clause is required. </para> <para> + /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */ If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal> and the candidate change row matches a row in the <replaceable class="parameter">target_table_name</replaceable> @@ -242,6 +243,7 @@ DELETE clause will be activated and the corresponding action will occur for that row. The expression may not contain functions that possibly performs writes to the database. + /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */ </para> <para> A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns @@ -379,6 +381,7 @@ DELETE <para> An expression to assign to the column. The expression can use the old values of this and other columns in the table. + /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */ </para> </listitem> </varlistentry> @@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> In summary, statement triggers for an event type (say, INSERT) will be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level triggers will fire only for the one event type <emphasis>activated</emphasis>. + /* XXX What does "activated" mean? Maybe "executed" would be better? */ So a <command>MERGE</command> might fire statement triggers for both <command>UPDATE</command> and <command>INSERT</command>, even though only <command>UPDATE</command> row triggers were fired. @@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable> rows will be used to modify the target row, later attempts to modify will cause an error. This can also occur if row triggers make changes to the target table which are then subsequently modified by <command>MERGE</command>. + /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */ + /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */ If the repeated action is an <command>INSERT</command> this will cause a uniqueness violation while a repeated <command>UPDATE</command> or <command>DELETE</command> will cause a cardinality violation; the latter behavior @@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> <title>Examples</title> <para> - Perform maintenance on CustomerAccounts based upon new Transactions. + Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>. <programlisting> MERGE CustomerAccount CA @@ -599,7 +605,7 @@ WHEN MATCHED THEN DELETE; </programlisting> - The wine_stock_changes table might be, for example, a temporary table + The <literal>wine_stock_changes</literal> table might be, for example, a temporary table recently loaded into the database. </para> diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index fbaf50c258..f914a00e9f 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors, break; case CMD_MERGE: operation = "Merge"; - foperation = "Foreign Merge"; + foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */ break; default: operation = "???"; diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c index 0e245e1361..a7492a6c4b 100644 --- a/src/backend/executor/execMerge.c +++ b/src/backend/executor/execMerge.c @@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * A concurrent update can: * * 1. modify the target tuple so that it no longer satisfies the - * additional quals attached to the current WHEN MATCHED action OR + * additional quals attached to the current WHEN MATCHED action * * In this case, we are still dealing with a WHEN MATCHED case, but we * should recheck the list of WHEN MATCHED actions and choose the first @@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * tuple, so we now instead find a qualifying WHEN NOT MATCHED action to * execute. * + * XXX Hmmm, what if the updated tuple would now match one that was + * considered NOT MATCHED so far? + * * A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED. * * ExecMergeMatched takes care of following the update chain and diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index c0a97eba91..9c14709e47 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate) { /* advance to next subplan if any */ node->mt_whichplan++; - if (node->mt_whichplan < node->mt_nplans) { resultRelInfo++; @@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate) EvalPlanQualSetSlot(&node->mt_epqstate, planSlot); slot = planSlot; + /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together + * with the other MT operations? This seems a bit out of place. */ if (operation == CMD_MERGE) { ExecMerge(node, resultRelInfo, estate, slot, junkfilter); diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c index 8848e9a03f..b2543f6814 100644 --- a/src/backend/optimizer/prep/preptlist.c +++ b/src/backend/optimizer/prep/preptlist.c @@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root) tlist = expand_targetlist(tlist, command_type, result_relation, target_relation); + /* + * For MERGE we need to handle the target list for the target relation, + * and also target list for each action (only INSERT/UPDATE matter). + */ if (command_type == CMD_MERGE) { ListCell *l; @@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type, * generate a NULL for dropped columns (we want to drop any old * values). * + * XXX Should this explain why MERGE has the same logic as UPDATE? + * * When generating a NULL constant for a dropped column, we label * it INT4 (any other guaranteed-to-exist datatype would do as * well). We can't label it with the dropped column's datatype diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index f7c152beb4..5d49c8c37e 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) break; case EXPR_KIND_MERGE_WHEN_AND: if (isAgg) + /* XXX "WHEN AND" seems rather strange. ... */ err = _("aggregate functions are not allowed in WHEN AND conditions"); else err = _("grouping operations are not allowed in WHEN AND conditions"); diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index d59f4fde95..d50543b1ba 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd) /* Also update the sent position status in shared memory */ SpinLockAcquire(&MyWalSnd->mutex); + /* XXX Huh? Why does MERGE patch change walsender? */ MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush; SpinLockRelease(&MyWalSnd->mutex); @@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data) replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ? MyWalSnd->write : MyWalSnd->flush; - if (WalSndCaughtUp && - sentPtr == replicatedPtr && + if (WalSndCaughtUp && sentPtr == replicatedPtr && !pq_is_send_pending()) { QueryCompletion qc; diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c index e2706c181c..80ae946d17 100644 --- a/src/backend/rewrite/rewriteHandler.c +++ b/src/backend/rewrite/rewriteHandler.c @@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation) } } + /* * matchLocks - * match the list of locks and returns the matching rules @@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events) /* * XXX MERGE doesn't support write rules because they would violate * the SQL Standard spec and would be unclear how they should work. + * + * XXX So does't support means 'ignores'? Should that either fail + * or at least print some warning? */ if (event == CMD_MERGE) product_queries = NIL; diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 0281351dcf..7a768a7b5b 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup) int result; bool arg1_match; + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup) memset(pres, 0, sizeof(Datum)); len = VARSIZE_ANY_EXHDR(authoritative); + /* XXX Huh? Why is this in MERGE patch? */ if (len < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2) len1 = VARSIZE_ANY_EXHDR(t1); len2 = VARSIZE_ANY_EXHDR(t2); + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 3cba38044e..08fea9b8f7 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -172,6 +172,7 @@ typedef struct Query List *withCheckOptions; /* a list of WithCheckOption's (added * during rewrite) */ + /* XXX Why not mergeTragetRelation? */ int mergeTarget_relation; List *mergeSourceTargetList; List *mergeActionList; /* list of actions for MERGE (only) */ @@ -1581,11 +1582,11 @@ typedef struct UpdateStmt typedef struct MergeStmt { NodeTag type; - RangeVar *relation; /* target relation to merge into */ + RangeVar *relation; /* target relation to merge into */ Node *source_relation; /* source relation */ - Node *join_condition; /* join condition between source and target */ + Node *join_condition; /* join condition between source and target */ List *mergeWhenClauses; /* list of MergeWhenClause(es) */ - WithClause *withClause; /* WITH clause */ + WithClause *withClause; /* WITH clause */ } MergeStmt; typedef struct MergeWhenClause -- 2.26.2 --------------B1AA4DC5EAE9BAF8B146244A Content-Type: text/plain; charset=UTF-8; name="0002-fix-valgrind-failure.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-fix-valgrind-failure.txt" ^ permalink raw reply [nested|flat] 56+ messages in thread
* [PATCH 1/2] review @ 2021-01-09 02:17 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 56+ messages in thread From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw) --- doc/src/sgml/mvcc.sgml | 20 +++++++++++--------- doc/src/sgml/ref/create_policy.sgml | 6 +++--- doc/src/sgml/ref/merge.sgml | 12 +++++++++--- src/backend/commands/explain.c | 2 +- src/backend/executor/execMerge.c | 5 ++++- src/backend/executor/nodeModifyTable.c | 3 ++- src/backend/optimizer/prep/preptlist.c | 6 ++++++ src/backend/parser/parse_agg.c | 1 + src/backend/replication/walsender.c | 4 ++-- src/backend/rewrite/rewriteHandler.c | 4 ++++ src/backend/utils/adt/varlena.c | 3 +++ src/include/nodes/parsenodes.h | 7 ++++--- 12 files changed, 50 insertions(+), 23 deletions(-) diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml index 9ec8474185..02c9f3fdea 100644 --- a/doc/src/sgml/mvcc.sgml +++ b/doc/src/sgml/mvcc.sgml @@ -429,22 +429,24 @@ COMMIT; with both <command>INSERT</command> and <command>UPDATE</command> subcommands looks similar to <command>INSERT</command> with an <literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee - that either <command>INSERT</command> and <command>UPDATE</command> will occur. + that either <command>INSERT</command> or <command>UPDATE</command> will occur. - If MERGE attempts an UPDATE or DELETE and the row is concurrently updated + /* XXX This is a very long and hard to understand sentence :-( */ + /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */ + If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated but the join condition still passes for the current target and the current - source tuple, then MERGE will behave the same as the UPDATE or DELETE commands + source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands and perform its action on the latest version of the row, using standard - EvalPlanQual. MERGE actions can be conditional, so conditions must be + EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be re-evaluated on the latest row, starting from the first action. On the other hand, if the row is concurrently updated or deleted so that - the join condition fails, then MERGE will execute a NOT MATCHED action, if it - exists and the AND WHEN qual evaluates to true. + the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it + exists and the <literal>AND WHEN</literal> qual evaluates to true. - If MERGE attempts an INSERT and a unique index is present and a duplicate - row is concurrently inserted then a uniqueness violation is raised. MERGE - does not attempt to avoid the ERROR by attempting an UPDATE. + If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate + row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command> + does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>. </para> <para> diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml index ad20230c58..0a64674f2d 100644 --- a/doc/src/sgml/ref/create_policy.sgml +++ b/doc/src/sgml/ref/create_policy.sgml @@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable <para> No separate policy exists for <command>MERGE</command>. Instead policies - defined for <literal>SELECT</literal>, <literal>INSERT</literal>, - <literal>UPDATE</literal> and <literal>DELETE</literal> are applied - while executing MERGE, depending on the actions that are activated. + defined for <command>SELECT</command>, <command>INSERT</command>, + <command>UPDATE</command> and <command>DELETE</command> are applied + while executing <command>MERGE</command>, depending on the actions that are activated. </para> </refsect1> diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml index b2a9f67cfa..c2901b0d58 100644 --- a/doc/src/sgml/ref/merge.sgml +++ b/doc/src/sgml/ref/merge.sgml @@ -98,7 +98,7 @@ DELETE </para> <para> - There is no MERGE privilege. + There is no <literal>MERGE</literal> privilege. You must have the <literal>UPDATE</literal> privilege on the column(s) of the <replaceable class="parameter">target_table_name</replaceable> referred to in the <literal>SET</literal> clause @@ -217,6 +217,7 @@ DELETE At least one <literal>WHEN</literal> clause is required. </para> <para> + /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */ If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal> and the candidate change row matches a row in the <replaceable class="parameter">target_table_name</replaceable> @@ -242,6 +243,7 @@ DELETE clause will be activated and the corresponding action will occur for that row. The expression may not contain functions that possibly performs writes to the database. + /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */ </para> <para> A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns @@ -379,6 +381,7 @@ DELETE <para> An expression to assign to the column. The expression can use the old values of this and other columns in the table. + /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */ </para> </listitem> </varlistentry> @@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> In summary, statement triggers for an event type (say, INSERT) will be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level triggers will fire only for the one event type <emphasis>activated</emphasis>. + /* XXX What does "activated" mean? Maybe "executed" would be better? */ So a <command>MERGE</command> might fire statement triggers for both <command>UPDATE</command> and <command>INSERT</command>, even though only <command>UPDATE</command> row triggers were fired. @@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable> rows will be used to modify the target row, later attempts to modify will cause an error. This can also occur if row triggers make changes to the target table which are then subsequently modified by <command>MERGE</command>. + /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */ + /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */ If the repeated action is an <command>INSERT</command> this will cause a uniqueness violation while a repeated <command>UPDATE</command> or <command>DELETE</command> will cause a cardinality violation; the latter behavior @@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> <title>Examples</title> <para> - Perform maintenance on CustomerAccounts based upon new Transactions. + Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>. <programlisting> MERGE CustomerAccount CA @@ -599,7 +605,7 @@ WHEN MATCHED THEN DELETE; </programlisting> - The wine_stock_changes table might be, for example, a temporary table + The <literal>wine_stock_changes</literal> table might be, for example, a temporary table recently loaded into the database. </para> diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index fbaf50c258..f914a00e9f 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors, break; case CMD_MERGE: operation = "Merge"; - foperation = "Foreign Merge"; + foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */ break; default: operation = "???"; diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c index 0e245e1361..a7492a6c4b 100644 --- a/src/backend/executor/execMerge.c +++ b/src/backend/executor/execMerge.c @@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * A concurrent update can: * * 1. modify the target tuple so that it no longer satisfies the - * additional quals attached to the current WHEN MATCHED action OR + * additional quals attached to the current WHEN MATCHED action * * In this case, we are still dealing with a WHEN MATCHED case, but we * should recheck the list of WHEN MATCHED actions and choose the first @@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * tuple, so we now instead find a qualifying WHEN NOT MATCHED action to * execute. * + * XXX Hmmm, what if the updated tuple would now match one that was + * considered NOT MATCHED so far? + * * A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED. * * ExecMergeMatched takes care of following the update chain and diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index c0a97eba91..9c14709e47 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate) { /* advance to next subplan if any */ node->mt_whichplan++; - if (node->mt_whichplan < node->mt_nplans) { resultRelInfo++; @@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate) EvalPlanQualSetSlot(&node->mt_epqstate, planSlot); slot = planSlot; + /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together + * with the other MT operations? This seems a bit out of place. */ if (operation == CMD_MERGE) { ExecMerge(node, resultRelInfo, estate, slot, junkfilter); diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c index 8848e9a03f..b2543f6814 100644 --- a/src/backend/optimizer/prep/preptlist.c +++ b/src/backend/optimizer/prep/preptlist.c @@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root) tlist = expand_targetlist(tlist, command_type, result_relation, target_relation); + /* + * For MERGE we need to handle the target list for the target relation, + * and also target list for each action (only INSERT/UPDATE matter). + */ if (command_type == CMD_MERGE) { ListCell *l; @@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type, * generate a NULL for dropped columns (we want to drop any old * values). * + * XXX Should this explain why MERGE has the same logic as UPDATE? + * * When generating a NULL constant for a dropped column, we label * it INT4 (any other guaranteed-to-exist datatype would do as * well). We can't label it with the dropped column's datatype diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index f7c152beb4..5d49c8c37e 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) break; case EXPR_KIND_MERGE_WHEN_AND: if (isAgg) + /* XXX "WHEN AND" seems rather strange. ... */ err = _("aggregate functions are not allowed in WHEN AND conditions"); else err = _("grouping operations are not allowed in WHEN AND conditions"); diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index d59f4fde95..d50543b1ba 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd) /* Also update the sent position status in shared memory */ SpinLockAcquire(&MyWalSnd->mutex); + /* XXX Huh? Why does MERGE patch change walsender? */ MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush; SpinLockRelease(&MyWalSnd->mutex); @@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data) replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ? MyWalSnd->write : MyWalSnd->flush; - if (WalSndCaughtUp && - sentPtr == replicatedPtr && + if (WalSndCaughtUp && sentPtr == replicatedPtr && !pq_is_send_pending()) { QueryCompletion qc; diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c index e2706c181c..80ae946d17 100644 --- a/src/backend/rewrite/rewriteHandler.c +++ b/src/backend/rewrite/rewriteHandler.c @@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation) } } + /* * matchLocks - * match the list of locks and returns the matching rules @@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events) /* * XXX MERGE doesn't support write rules because they would violate * the SQL Standard spec and would be unclear how they should work. + * + * XXX So does't support means 'ignores'? Should that either fail + * or at least print some warning? */ if (event == CMD_MERGE) product_queries = NIL; diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 0281351dcf..7a768a7b5b 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup) int result; bool arg1_match; + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup) memset(pres, 0, sizeof(Datum)); len = VARSIZE_ANY_EXHDR(authoritative); + /* XXX Huh? Why is this in MERGE patch? */ if (len < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2) len1 = VARSIZE_ANY_EXHDR(t1); len2 = VARSIZE_ANY_EXHDR(t2); + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 3cba38044e..08fea9b8f7 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -172,6 +172,7 @@ typedef struct Query List *withCheckOptions; /* a list of WithCheckOption's (added * during rewrite) */ + /* XXX Why not mergeTragetRelation? */ int mergeTarget_relation; List *mergeSourceTargetList; List *mergeActionList; /* list of actions for MERGE (only) */ @@ -1581,11 +1582,11 @@ typedef struct UpdateStmt typedef struct MergeStmt { NodeTag type; - RangeVar *relation; /* target relation to merge into */ + RangeVar *relation; /* target relation to merge into */ Node *source_relation; /* source relation */ - Node *join_condition; /* join condition between source and target */ + Node *join_condition; /* join condition between source and target */ List *mergeWhenClauses; /* list of MergeWhenClause(es) */ - WithClause *withClause; /* WITH clause */ + WithClause *withClause; /* WITH clause */ } MergeStmt; typedef struct MergeWhenClause -- 2.26.2 --------------B1AA4DC5EAE9BAF8B146244A Content-Type: text/plain; charset=UTF-8; name="0002-fix-valgrind-failure.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-fix-valgrind-failure.txt" ^ permalink raw reply [nested|flat] 56+ messages in thread
* [PATCH 1/2] review @ 2021-01-09 02:17 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 56+ messages in thread From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw) --- doc/src/sgml/mvcc.sgml | 20 +++++++++++--------- doc/src/sgml/ref/create_policy.sgml | 6 +++--- doc/src/sgml/ref/merge.sgml | 12 +++++++++--- src/backend/commands/explain.c | 2 +- src/backend/executor/execMerge.c | 5 ++++- src/backend/executor/nodeModifyTable.c | 3 ++- src/backend/optimizer/prep/preptlist.c | 6 ++++++ src/backend/parser/parse_agg.c | 1 + src/backend/replication/walsender.c | 4 ++-- src/backend/rewrite/rewriteHandler.c | 4 ++++ src/backend/utils/adt/varlena.c | 3 +++ src/include/nodes/parsenodes.h | 7 ++++--- 12 files changed, 50 insertions(+), 23 deletions(-) diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml index 9ec8474185..02c9f3fdea 100644 --- a/doc/src/sgml/mvcc.sgml +++ b/doc/src/sgml/mvcc.sgml @@ -429,22 +429,24 @@ COMMIT; with both <command>INSERT</command> and <command>UPDATE</command> subcommands looks similar to <command>INSERT</command> with an <literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee - that either <command>INSERT</command> and <command>UPDATE</command> will occur. + that either <command>INSERT</command> or <command>UPDATE</command> will occur. - If MERGE attempts an UPDATE or DELETE and the row is concurrently updated + /* XXX This is a very long and hard to understand sentence :-( */ + /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */ + If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated but the join condition still passes for the current target and the current - source tuple, then MERGE will behave the same as the UPDATE or DELETE commands + source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands and perform its action on the latest version of the row, using standard - EvalPlanQual. MERGE actions can be conditional, so conditions must be + EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be re-evaluated on the latest row, starting from the first action. On the other hand, if the row is concurrently updated or deleted so that - the join condition fails, then MERGE will execute a NOT MATCHED action, if it - exists and the AND WHEN qual evaluates to true. + the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it + exists and the <literal>AND WHEN</literal> qual evaluates to true. - If MERGE attempts an INSERT and a unique index is present and a duplicate - row is concurrently inserted then a uniqueness violation is raised. MERGE - does not attempt to avoid the ERROR by attempting an UPDATE. + If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate + row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command> + does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>. </para> <para> diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml index ad20230c58..0a64674f2d 100644 --- a/doc/src/sgml/ref/create_policy.sgml +++ b/doc/src/sgml/ref/create_policy.sgml @@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable <para> No separate policy exists for <command>MERGE</command>. Instead policies - defined for <literal>SELECT</literal>, <literal>INSERT</literal>, - <literal>UPDATE</literal> and <literal>DELETE</literal> are applied - while executing MERGE, depending on the actions that are activated. + defined for <command>SELECT</command>, <command>INSERT</command>, + <command>UPDATE</command> and <command>DELETE</command> are applied + while executing <command>MERGE</command>, depending on the actions that are activated. </para> </refsect1> diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml index b2a9f67cfa..c2901b0d58 100644 --- a/doc/src/sgml/ref/merge.sgml +++ b/doc/src/sgml/ref/merge.sgml @@ -98,7 +98,7 @@ DELETE </para> <para> - There is no MERGE privilege. + There is no <literal>MERGE</literal> privilege. You must have the <literal>UPDATE</literal> privilege on the column(s) of the <replaceable class="parameter">target_table_name</replaceable> referred to in the <literal>SET</literal> clause @@ -217,6 +217,7 @@ DELETE At least one <literal>WHEN</literal> clause is required. </para> <para> + /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */ If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal> and the candidate change row matches a row in the <replaceable class="parameter">target_table_name</replaceable> @@ -242,6 +243,7 @@ DELETE clause will be activated and the corresponding action will occur for that row. The expression may not contain functions that possibly performs writes to the database. + /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */ </para> <para> A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns @@ -379,6 +381,7 @@ DELETE <para> An expression to assign to the column. The expression can use the old values of this and other columns in the table. + /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */ </para> </listitem> </varlistentry> @@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> In summary, statement triggers for an event type (say, INSERT) will be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level triggers will fire only for the one event type <emphasis>activated</emphasis>. + /* XXX What does "activated" mean? Maybe "executed" would be better? */ So a <command>MERGE</command> might fire statement triggers for both <command>UPDATE</command> and <command>INSERT</command>, even though only <command>UPDATE</command> row triggers were fired. @@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable> rows will be used to modify the target row, later attempts to modify will cause an error. This can also occur if row triggers make changes to the target table which are then subsequently modified by <command>MERGE</command>. + /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */ + /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */ If the repeated action is an <command>INSERT</command> this will cause a uniqueness violation while a repeated <command>UPDATE</command> or <command>DELETE</command> will cause a cardinality violation; the latter behavior @@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> <title>Examples</title> <para> - Perform maintenance on CustomerAccounts based upon new Transactions. + Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>. <programlisting> MERGE CustomerAccount CA @@ -599,7 +605,7 @@ WHEN MATCHED THEN DELETE; </programlisting> - The wine_stock_changes table might be, for example, a temporary table + The <literal>wine_stock_changes</literal> table might be, for example, a temporary table recently loaded into the database. </para> diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index fbaf50c258..f914a00e9f 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors, break; case CMD_MERGE: operation = "Merge"; - foperation = "Foreign Merge"; + foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */ break; default: operation = "???"; diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c index 0e245e1361..a7492a6c4b 100644 --- a/src/backend/executor/execMerge.c +++ b/src/backend/executor/execMerge.c @@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * A concurrent update can: * * 1. modify the target tuple so that it no longer satisfies the - * additional quals attached to the current WHEN MATCHED action OR + * additional quals attached to the current WHEN MATCHED action * * In this case, we are still dealing with a WHEN MATCHED case, but we * should recheck the list of WHEN MATCHED actions and choose the first @@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * tuple, so we now instead find a qualifying WHEN NOT MATCHED action to * execute. * + * XXX Hmmm, what if the updated tuple would now match one that was + * considered NOT MATCHED so far? + * * A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED. * * ExecMergeMatched takes care of following the update chain and diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index c0a97eba91..9c14709e47 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate) { /* advance to next subplan if any */ node->mt_whichplan++; - if (node->mt_whichplan < node->mt_nplans) { resultRelInfo++; @@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate) EvalPlanQualSetSlot(&node->mt_epqstate, planSlot); slot = planSlot; + /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together + * with the other MT operations? This seems a bit out of place. */ if (operation == CMD_MERGE) { ExecMerge(node, resultRelInfo, estate, slot, junkfilter); diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c index 8848e9a03f..b2543f6814 100644 --- a/src/backend/optimizer/prep/preptlist.c +++ b/src/backend/optimizer/prep/preptlist.c @@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root) tlist = expand_targetlist(tlist, command_type, result_relation, target_relation); + /* + * For MERGE we need to handle the target list for the target relation, + * and also target list for each action (only INSERT/UPDATE matter). + */ if (command_type == CMD_MERGE) { ListCell *l; @@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type, * generate a NULL for dropped columns (we want to drop any old * values). * + * XXX Should this explain why MERGE has the same logic as UPDATE? + * * When generating a NULL constant for a dropped column, we label * it INT4 (any other guaranteed-to-exist datatype would do as * well). We can't label it with the dropped column's datatype diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index f7c152beb4..5d49c8c37e 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) break; case EXPR_KIND_MERGE_WHEN_AND: if (isAgg) + /* XXX "WHEN AND" seems rather strange. ... */ err = _("aggregate functions are not allowed in WHEN AND conditions"); else err = _("grouping operations are not allowed in WHEN AND conditions"); diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index d59f4fde95..d50543b1ba 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd) /* Also update the sent position status in shared memory */ SpinLockAcquire(&MyWalSnd->mutex); + /* XXX Huh? Why does MERGE patch change walsender? */ MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush; SpinLockRelease(&MyWalSnd->mutex); @@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data) replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ? MyWalSnd->write : MyWalSnd->flush; - if (WalSndCaughtUp && - sentPtr == replicatedPtr && + if (WalSndCaughtUp && sentPtr == replicatedPtr && !pq_is_send_pending()) { QueryCompletion qc; diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c index e2706c181c..80ae946d17 100644 --- a/src/backend/rewrite/rewriteHandler.c +++ b/src/backend/rewrite/rewriteHandler.c @@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation) } } + /* * matchLocks - * match the list of locks and returns the matching rules @@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events) /* * XXX MERGE doesn't support write rules because they would violate * the SQL Standard spec and would be unclear how they should work. + * + * XXX So does't support means 'ignores'? Should that either fail + * or at least print some warning? */ if (event == CMD_MERGE) product_queries = NIL; diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 0281351dcf..7a768a7b5b 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup) int result; bool arg1_match; + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup) memset(pres, 0, sizeof(Datum)); len = VARSIZE_ANY_EXHDR(authoritative); + /* XXX Huh? Why is this in MERGE patch? */ if (len < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2) len1 = VARSIZE_ANY_EXHDR(t1); len2 = VARSIZE_ANY_EXHDR(t2); + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 3cba38044e..08fea9b8f7 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -172,6 +172,7 @@ typedef struct Query List *withCheckOptions; /* a list of WithCheckOption's (added * during rewrite) */ + /* XXX Why not mergeTragetRelation? */ int mergeTarget_relation; List *mergeSourceTargetList; List *mergeActionList; /* list of actions for MERGE (only) */ @@ -1581,11 +1582,11 @@ typedef struct UpdateStmt typedef struct MergeStmt { NodeTag type; - RangeVar *relation; /* target relation to merge into */ + RangeVar *relation; /* target relation to merge into */ Node *source_relation; /* source relation */ - Node *join_condition; /* join condition between source and target */ + Node *join_condition; /* join condition between source and target */ List *mergeWhenClauses; /* list of MergeWhenClause(es) */ - WithClause *withClause; /* WITH clause */ + WithClause *withClause; /* WITH clause */ } MergeStmt; typedef struct MergeWhenClause -- 2.26.2 --------------B1AA4DC5EAE9BAF8B146244A Content-Type: text/plain; charset=UTF-8; name="0002-fix-valgrind-failure.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-fix-valgrind-failure.txt" ^ permalink raw reply [nested|flat] 56+ messages in thread
* [PATCH 1/2] review @ 2021-01-09 02:17 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 56+ messages in thread From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw) --- doc/src/sgml/mvcc.sgml | 20 +++++++++++--------- doc/src/sgml/ref/create_policy.sgml | 6 +++--- doc/src/sgml/ref/merge.sgml | 12 +++++++++--- src/backend/commands/explain.c | 2 +- src/backend/executor/execMerge.c | 5 ++++- src/backend/executor/nodeModifyTable.c | 3 ++- src/backend/optimizer/prep/preptlist.c | 6 ++++++ src/backend/parser/parse_agg.c | 1 + src/backend/replication/walsender.c | 4 ++-- src/backend/rewrite/rewriteHandler.c | 4 ++++ src/backend/utils/adt/varlena.c | 3 +++ src/include/nodes/parsenodes.h | 7 ++++--- 12 files changed, 50 insertions(+), 23 deletions(-) diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml index 9ec8474185..02c9f3fdea 100644 --- a/doc/src/sgml/mvcc.sgml +++ b/doc/src/sgml/mvcc.sgml @@ -429,22 +429,24 @@ COMMIT; with both <command>INSERT</command> and <command>UPDATE</command> subcommands looks similar to <command>INSERT</command> with an <literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee - that either <command>INSERT</command> and <command>UPDATE</command> will occur. + that either <command>INSERT</command> or <command>UPDATE</command> will occur. - If MERGE attempts an UPDATE or DELETE and the row is concurrently updated + /* XXX This is a very long and hard to understand sentence :-( */ + /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */ + If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated but the join condition still passes for the current target and the current - source tuple, then MERGE will behave the same as the UPDATE or DELETE commands + source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands and perform its action on the latest version of the row, using standard - EvalPlanQual. MERGE actions can be conditional, so conditions must be + EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be re-evaluated on the latest row, starting from the first action. On the other hand, if the row is concurrently updated or deleted so that - the join condition fails, then MERGE will execute a NOT MATCHED action, if it - exists and the AND WHEN qual evaluates to true. + the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it + exists and the <literal>AND WHEN</literal> qual evaluates to true. - If MERGE attempts an INSERT and a unique index is present and a duplicate - row is concurrently inserted then a uniqueness violation is raised. MERGE - does not attempt to avoid the ERROR by attempting an UPDATE. + If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate + row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command> + does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>. </para> <para> diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml index ad20230c58..0a64674f2d 100644 --- a/doc/src/sgml/ref/create_policy.sgml +++ b/doc/src/sgml/ref/create_policy.sgml @@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable <para> No separate policy exists for <command>MERGE</command>. Instead policies - defined for <literal>SELECT</literal>, <literal>INSERT</literal>, - <literal>UPDATE</literal> and <literal>DELETE</literal> are applied - while executing MERGE, depending on the actions that are activated. + defined for <command>SELECT</command>, <command>INSERT</command>, + <command>UPDATE</command> and <command>DELETE</command> are applied + while executing <command>MERGE</command>, depending on the actions that are activated. </para> </refsect1> diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml index b2a9f67cfa..c2901b0d58 100644 --- a/doc/src/sgml/ref/merge.sgml +++ b/doc/src/sgml/ref/merge.sgml @@ -98,7 +98,7 @@ DELETE </para> <para> - There is no MERGE privilege. + There is no <literal>MERGE</literal> privilege. You must have the <literal>UPDATE</literal> privilege on the column(s) of the <replaceable class="parameter">target_table_name</replaceable> referred to in the <literal>SET</literal> clause @@ -217,6 +217,7 @@ DELETE At least one <literal>WHEN</literal> clause is required. </para> <para> + /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */ If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal> and the candidate change row matches a row in the <replaceable class="parameter">target_table_name</replaceable> @@ -242,6 +243,7 @@ DELETE clause will be activated and the corresponding action will occur for that row. The expression may not contain functions that possibly performs writes to the database. + /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */ </para> <para> A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns @@ -379,6 +381,7 @@ DELETE <para> An expression to assign to the column. The expression can use the old values of this and other columns in the table. + /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */ </para> </listitem> </varlistentry> @@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> In summary, statement triggers for an event type (say, INSERT) will be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level triggers will fire only for the one event type <emphasis>activated</emphasis>. + /* XXX What does "activated" mean? Maybe "executed" would be better? */ So a <command>MERGE</command> might fire statement triggers for both <command>UPDATE</command> and <command>INSERT</command>, even though only <command>UPDATE</command> row triggers were fired. @@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable> rows will be used to modify the target row, later attempts to modify will cause an error. This can also occur if row triggers make changes to the target table which are then subsequently modified by <command>MERGE</command>. + /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */ + /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */ If the repeated action is an <command>INSERT</command> this will cause a uniqueness violation while a repeated <command>UPDATE</command> or <command>DELETE</command> will cause a cardinality violation; the latter behavior @@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> <title>Examples</title> <para> - Perform maintenance on CustomerAccounts based upon new Transactions. + Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>. <programlisting> MERGE CustomerAccount CA @@ -599,7 +605,7 @@ WHEN MATCHED THEN DELETE; </programlisting> - The wine_stock_changes table might be, for example, a temporary table + The <literal>wine_stock_changes</literal> table might be, for example, a temporary table recently loaded into the database. </para> diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index fbaf50c258..f914a00e9f 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors, break; case CMD_MERGE: operation = "Merge"; - foperation = "Foreign Merge"; + foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */ break; default: operation = "???"; diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c index 0e245e1361..a7492a6c4b 100644 --- a/src/backend/executor/execMerge.c +++ b/src/backend/executor/execMerge.c @@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * A concurrent update can: * * 1. modify the target tuple so that it no longer satisfies the - * additional quals attached to the current WHEN MATCHED action OR + * additional quals attached to the current WHEN MATCHED action * * In this case, we are still dealing with a WHEN MATCHED case, but we * should recheck the list of WHEN MATCHED actions and choose the first @@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * tuple, so we now instead find a qualifying WHEN NOT MATCHED action to * execute. * + * XXX Hmmm, what if the updated tuple would now match one that was + * considered NOT MATCHED so far? + * * A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED. * * ExecMergeMatched takes care of following the update chain and diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index c0a97eba91..9c14709e47 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate) { /* advance to next subplan if any */ node->mt_whichplan++; - if (node->mt_whichplan < node->mt_nplans) { resultRelInfo++; @@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate) EvalPlanQualSetSlot(&node->mt_epqstate, planSlot); slot = planSlot; + /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together + * with the other MT operations? This seems a bit out of place. */ if (operation == CMD_MERGE) { ExecMerge(node, resultRelInfo, estate, slot, junkfilter); diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c index 8848e9a03f..b2543f6814 100644 --- a/src/backend/optimizer/prep/preptlist.c +++ b/src/backend/optimizer/prep/preptlist.c @@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root) tlist = expand_targetlist(tlist, command_type, result_relation, target_relation); + /* + * For MERGE we need to handle the target list for the target relation, + * and also target list for each action (only INSERT/UPDATE matter). + */ if (command_type == CMD_MERGE) { ListCell *l; @@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type, * generate a NULL for dropped columns (we want to drop any old * values). * + * XXX Should this explain why MERGE has the same logic as UPDATE? + * * When generating a NULL constant for a dropped column, we label * it INT4 (any other guaranteed-to-exist datatype would do as * well). We can't label it with the dropped column's datatype diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index f7c152beb4..5d49c8c37e 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) break; case EXPR_KIND_MERGE_WHEN_AND: if (isAgg) + /* XXX "WHEN AND" seems rather strange. ... */ err = _("aggregate functions are not allowed in WHEN AND conditions"); else err = _("grouping operations are not allowed in WHEN AND conditions"); diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index d59f4fde95..d50543b1ba 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd) /* Also update the sent position status in shared memory */ SpinLockAcquire(&MyWalSnd->mutex); + /* XXX Huh? Why does MERGE patch change walsender? */ MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush; SpinLockRelease(&MyWalSnd->mutex); @@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data) replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ? MyWalSnd->write : MyWalSnd->flush; - if (WalSndCaughtUp && - sentPtr == replicatedPtr && + if (WalSndCaughtUp && sentPtr == replicatedPtr && !pq_is_send_pending()) { QueryCompletion qc; diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c index e2706c181c..80ae946d17 100644 --- a/src/backend/rewrite/rewriteHandler.c +++ b/src/backend/rewrite/rewriteHandler.c @@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation) } } + /* * matchLocks - * match the list of locks and returns the matching rules @@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events) /* * XXX MERGE doesn't support write rules because they would violate * the SQL Standard spec and would be unclear how they should work. + * + * XXX So does't support means 'ignores'? Should that either fail + * or at least print some warning? */ if (event == CMD_MERGE) product_queries = NIL; diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 0281351dcf..7a768a7b5b 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup) int result; bool arg1_match; + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup) memset(pres, 0, sizeof(Datum)); len = VARSIZE_ANY_EXHDR(authoritative); + /* XXX Huh? Why is this in MERGE patch? */ if (len < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2) len1 = VARSIZE_ANY_EXHDR(t1); len2 = VARSIZE_ANY_EXHDR(t2); + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 3cba38044e..08fea9b8f7 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -172,6 +172,7 @@ typedef struct Query List *withCheckOptions; /* a list of WithCheckOption's (added * during rewrite) */ + /* XXX Why not mergeTragetRelation? */ int mergeTarget_relation; List *mergeSourceTargetList; List *mergeActionList; /* list of actions for MERGE (only) */ @@ -1581,11 +1582,11 @@ typedef struct UpdateStmt typedef struct MergeStmt { NodeTag type; - RangeVar *relation; /* target relation to merge into */ + RangeVar *relation; /* target relation to merge into */ Node *source_relation; /* source relation */ - Node *join_condition; /* join condition between source and target */ + Node *join_condition; /* join condition between source and target */ List *mergeWhenClauses; /* list of MergeWhenClause(es) */ - WithClause *withClause; /* WITH clause */ + WithClause *withClause; /* WITH clause */ } MergeStmt; typedef struct MergeWhenClause -- 2.26.2 --------------B1AA4DC5EAE9BAF8B146244A Content-Type: text/plain; charset=UTF-8; name="0002-fix-valgrind-failure.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-fix-valgrind-failure.txt" ^ permalink raw reply [nested|flat] 56+ messages in thread
* [PATCH 1/2] review @ 2021-01-09 02:17 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 56+ messages in thread From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw) --- doc/src/sgml/mvcc.sgml | 20 +++++++++++--------- doc/src/sgml/ref/create_policy.sgml | 6 +++--- doc/src/sgml/ref/merge.sgml | 12 +++++++++--- src/backend/commands/explain.c | 2 +- src/backend/executor/execMerge.c | 5 ++++- src/backend/executor/nodeModifyTable.c | 3 ++- src/backend/optimizer/prep/preptlist.c | 6 ++++++ src/backend/parser/parse_agg.c | 1 + src/backend/replication/walsender.c | 4 ++-- src/backend/rewrite/rewriteHandler.c | 4 ++++ src/backend/utils/adt/varlena.c | 3 +++ src/include/nodes/parsenodes.h | 7 ++++--- 12 files changed, 50 insertions(+), 23 deletions(-) diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml index 9ec8474185..02c9f3fdea 100644 --- a/doc/src/sgml/mvcc.sgml +++ b/doc/src/sgml/mvcc.sgml @@ -429,22 +429,24 @@ COMMIT; with both <command>INSERT</command> and <command>UPDATE</command> subcommands looks similar to <command>INSERT</command> with an <literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee - that either <command>INSERT</command> and <command>UPDATE</command> will occur. + that either <command>INSERT</command> or <command>UPDATE</command> will occur. - If MERGE attempts an UPDATE or DELETE and the row is concurrently updated + /* XXX This is a very long and hard to understand sentence :-( */ + /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */ + If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated but the join condition still passes for the current target and the current - source tuple, then MERGE will behave the same as the UPDATE or DELETE commands + source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands and perform its action on the latest version of the row, using standard - EvalPlanQual. MERGE actions can be conditional, so conditions must be + EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be re-evaluated on the latest row, starting from the first action. On the other hand, if the row is concurrently updated or deleted so that - the join condition fails, then MERGE will execute a NOT MATCHED action, if it - exists and the AND WHEN qual evaluates to true. + the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it + exists and the <literal>AND WHEN</literal> qual evaluates to true. - If MERGE attempts an INSERT and a unique index is present and a duplicate - row is concurrently inserted then a uniqueness violation is raised. MERGE - does not attempt to avoid the ERROR by attempting an UPDATE. + If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate + row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command> + does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>. </para> <para> diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml index ad20230c58..0a64674f2d 100644 --- a/doc/src/sgml/ref/create_policy.sgml +++ b/doc/src/sgml/ref/create_policy.sgml @@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable <para> No separate policy exists for <command>MERGE</command>. Instead policies - defined for <literal>SELECT</literal>, <literal>INSERT</literal>, - <literal>UPDATE</literal> and <literal>DELETE</literal> are applied - while executing MERGE, depending on the actions that are activated. + defined for <command>SELECT</command>, <command>INSERT</command>, + <command>UPDATE</command> and <command>DELETE</command> are applied + while executing <command>MERGE</command>, depending on the actions that are activated. </para> </refsect1> diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml index b2a9f67cfa..c2901b0d58 100644 --- a/doc/src/sgml/ref/merge.sgml +++ b/doc/src/sgml/ref/merge.sgml @@ -98,7 +98,7 @@ DELETE </para> <para> - There is no MERGE privilege. + There is no <literal>MERGE</literal> privilege. You must have the <literal>UPDATE</literal> privilege on the column(s) of the <replaceable class="parameter">target_table_name</replaceable> referred to in the <literal>SET</literal> clause @@ -217,6 +217,7 @@ DELETE At least one <literal>WHEN</literal> clause is required. </para> <para> + /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */ If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal> and the candidate change row matches a row in the <replaceable class="parameter">target_table_name</replaceable> @@ -242,6 +243,7 @@ DELETE clause will be activated and the corresponding action will occur for that row. The expression may not contain functions that possibly performs writes to the database. + /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */ </para> <para> A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns @@ -379,6 +381,7 @@ DELETE <para> An expression to assign to the column. The expression can use the old values of this and other columns in the table. + /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */ </para> </listitem> </varlistentry> @@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> In summary, statement triggers for an event type (say, INSERT) will be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level triggers will fire only for the one event type <emphasis>activated</emphasis>. + /* XXX What does "activated" mean? Maybe "executed" would be better? */ So a <command>MERGE</command> might fire statement triggers for both <command>UPDATE</command> and <command>INSERT</command>, even though only <command>UPDATE</command> row triggers were fired. @@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable> rows will be used to modify the target row, later attempts to modify will cause an error. This can also occur if row triggers make changes to the target table which are then subsequently modified by <command>MERGE</command>. + /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */ + /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */ If the repeated action is an <command>INSERT</command> this will cause a uniqueness violation while a repeated <command>UPDATE</command> or <command>DELETE</command> will cause a cardinality violation; the latter behavior @@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> <title>Examples</title> <para> - Perform maintenance on CustomerAccounts based upon new Transactions. + Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>. <programlisting> MERGE CustomerAccount CA @@ -599,7 +605,7 @@ WHEN MATCHED THEN DELETE; </programlisting> - The wine_stock_changes table might be, for example, a temporary table + The <literal>wine_stock_changes</literal> table might be, for example, a temporary table recently loaded into the database. </para> diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index fbaf50c258..f914a00e9f 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors, break; case CMD_MERGE: operation = "Merge"; - foperation = "Foreign Merge"; + foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */ break; default: operation = "???"; diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c index 0e245e1361..a7492a6c4b 100644 --- a/src/backend/executor/execMerge.c +++ b/src/backend/executor/execMerge.c @@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * A concurrent update can: * * 1. modify the target tuple so that it no longer satisfies the - * additional quals attached to the current WHEN MATCHED action OR + * additional quals attached to the current WHEN MATCHED action * * In this case, we are still dealing with a WHEN MATCHED case, but we * should recheck the list of WHEN MATCHED actions and choose the first @@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * tuple, so we now instead find a qualifying WHEN NOT MATCHED action to * execute. * + * XXX Hmmm, what if the updated tuple would now match one that was + * considered NOT MATCHED so far? + * * A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED. * * ExecMergeMatched takes care of following the update chain and diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index c0a97eba91..9c14709e47 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate) { /* advance to next subplan if any */ node->mt_whichplan++; - if (node->mt_whichplan < node->mt_nplans) { resultRelInfo++; @@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate) EvalPlanQualSetSlot(&node->mt_epqstate, planSlot); slot = planSlot; + /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together + * with the other MT operations? This seems a bit out of place. */ if (operation == CMD_MERGE) { ExecMerge(node, resultRelInfo, estate, slot, junkfilter); diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c index 8848e9a03f..b2543f6814 100644 --- a/src/backend/optimizer/prep/preptlist.c +++ b/src/backend/optimizer/prep/preptlist.c @@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root) tlist = expand_targetlist(tlist, command_type, result_relation, target_relation); + /* + * For MERGE we need to handle the target list for the target relation, + * and also target list for each action (only INSERT/UPDATE matter). + */ if (command_type == CMD_MERGE) { ListCell *l; @@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type, * generate a NULL for dropped columns (we want to drop any old * values). * + * XXX Should this explain why MERGE has the same logic as UPDATE? + * * When generating a NULL constant for a dropped column, we label * it INT4 (any other guaranteed-to-exist datatype would do as * well). We can't label it with the dropped column's datatype diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index f7c152beb4..5d49c8c37e 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) break; case EXPR_KIND_MERGE_WHEN_AND: if (isAgg) + /* XXX "WHEN AND" seems rather strange. ... */ err = _("aggregate functions are not allowed in WHEN AND conditions"); else err = _("grouping operations are not allowed in WHEN AND conditions"); diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index d59f4fde95..d50543b1ba 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd) /* Also update the sent position status in shared memory */ SpinLockAcquire(&MyWalSnd->mutex); + /* XXX Huh? Why does MERGE patch change walsender? */ MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush; SpinLockRelease(&MyWalSnd->mutex); @@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data) replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ? MyWalSnd->write : MyWalSnd->flush; - if (WalSndCaughtUp && - sentPtr == replicatedPtr && + if (WalSndCaughtUp && sentPtr == replicatedPtr && !pq_is_send_pending()) { QueryCompletion qc; diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c index e2706c181c..80ae946d17 100644 --- a/src/backend/rewrite/rewriteHandler.c +++ b/src/backend/rewrite/rewriteHandler.c @@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation) } } + /* * matchLocks - * match the list of locks and returns the matching rules @@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events) /* * XXX MERGE doesn't support write rules because they would violate * the SQL Standard spec and would be unclear how they should work. + * + * XXX So does't support means 'ignores'? Should that either fail + * or at least print some warning? */ if (event == CMD_MERGE) product_queries = NIL; diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 0281351dcf..7a768a7b5b 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup) int result; bool arg1_match; + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup) memset(pres, 0, sizeof(Datum)); len = VARSIZE_ANY_EXHDR(authoritative); + /* XXX Huh? Why is this in MERGE patch? */ if (len < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2) len1 = VARSIZE_ANY_EXHDR(t1); len2 = VARSIZE_ANY_EXHDR(t2); + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 3cba38044e..08fea9b8f7 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -172,6 +172,7 @@ typedef struct Query List *withCheckOptions; /* a list of WithCheckOption's (added * during rewrite) */ + /* XXX Why not mergeTragetRelation? */ int mergeTarget_relation; List *mergeSourceTargetList; List *mergeActionList; /* list of actions for MERGE (only) */ @@ -1581,11 +1582,11 @@ typedef struct UpdateStmt typedef struct MergeStmt { NodeTag type; - RangeVar *relation; /* target relation to merge into */ + RangeVar *relation; /* target relation to merge into */ Node *source_relation; /* source relation */ - Node *join_condition; /* join condition between source and target */ + Node *join_condition; /* join condition between source and target */ List *mergeWhenClauses; /* list of MergeWhenClause(es) */ - WithClause *withClause; /* WITH clause */ + WithClause *withClause; /* WITH clause */ } MergeStmt; typedef struct MergeWhenClause -- 2.26.2 --------------B1AA4DC5EAE9BAF8B146244A Content-Type: text/plain; charset=UTF-8; name="0002-fix-valgrind-failure.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-fix-valgrind-failure.txt" ^ permalink raw reply [nested|flat] 56+ messages in thread
* [PATCH 1/2] review @ 2021-01-09 02:17 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 56+ messages in thread From: Tomas Vondra @ 2021-01-09 02:17 UTC (permalink / raw) --- doc/src/sgml/mvcc.sgml | 20 +++++++++++--------- doc/src/sgml/ref/create_policy.sgml | 6 +++--- doc/src/sgml/ref/merge.sgml | 12 +++++++++--- src/backend/commands/explain.c | 2 +- src/backend/executor/execMerge.c | 5 ++++- src/backend/executor/nodeModifyTable.c | 3 ++- src/backend/optimizer/prep/preptlist.c | 6 ++++++ src/backend/parser/parse_agg.c | 1 + src/backend/replication/walsender.c | 4 ++-- src/backend/rewrite/rewriteHandler.c | 4 ++++ src/backend/utils/adt/varlena.c | 3 +++ src/include/nodes/parsenodes.h | 7 ++++--- 12 files changed, 50 insertions(+), 23 deletions(-) diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml index 9ec8474185..02c9f3fdea 100644 --- a/doc/src/sgml/mvcc.sgml +++ b/doc/src/sgml/mvcc.sgml @@ -429,22 +429,24 @@ COMMIT; with both <command>INSERT</command> and <command>UPDATE</command> subcommands looks similar to <command>INSERT</command> with an <literal>ON CONFLICT DO UPDATE</literal> clause but does not guarantee - that either <command>INSERT</command> and <command>UPDATE</command> will occur. + that either <command>INSERT</command> or <command>UPDATE</command> will occur. - If MERGE attempts an UPDATE or DELETE and the row is concurrently updated + /* XXX This is a very long and hard to understand sentence :-( */ + /* XXX Do we want to mention EvalPlanQual here? There's no explanation what it does in this file, so maybe elaborate what it does or leave that detail for a README? */ + If MERGE attempts an <command>UPDATE</command> or <command>DELETE</command> and the row is concurrently updated but the join condition still passes for the current target and the current - source tuple, then MERGE will behave the same as the UPDATE or DELETE commands + source tuple, then <command>MERGE</command> will behave the same as the <command>UPDATE</command> or <command>DELETE</command> commands and perform its action on the latest version of the row, using standard - EvalPlanQual. MERGE actions can be conditional, so conditions must be + EvalPlanQual. <command>MERGE</command> actions can be conditional, so conditions must be re-evaluated on the latest row, starting from the first action. On the other hand, if the row is concurrently updated or deleted so that - the join condition fails, then MERGE will execute a NOT MATCHED action, if it - exists and the AND WHEN qual evaluates to true. + the join condition fails, then <command>MERGE</command> will execute a <literal>NOT MATCHED</literal> action, if it + exists and the <literal>AND WHEN</literal> qual evaluates to true. - If MERGE attempts an INSERT and a unique index is present and a duplicate - row is concurrently inserted then a uniqueness violation is raised. MERGE - does not attempt to avoid the ERROR by attempting an UPDATE. + If <command>MERGE</command> attempts an <command>INSERT</command> and a unique index is present and a duplicate + row is concurrently inserted then a uniqueness violation is raised. <command>MERGE</command> + does not attempt to avoid the <literal>ERROR</literal> by attempting an <command>UPDATE</command>. </para> <para> diff --git a/doc/src/sgml/ref/create_policy.sgml b/doc/src/sgml/ref/create_policy.sgml index ad20230c58..0a64674f2d 100644 --- a/doc/src/sgml/ref/create_policy.sgml +++ b/doc/src/sgml/ref/create_policy.sgml @@ -97,9 +97,9 @@ CREATE POLICY <replaceable class="parameter">name</replaceable> ON <replaceable <para> No separate policy exists for <command>MERGE</command>. Instead policies - defined for <literal>SELECT</literal>, <literal>INSERT</literal>, - <literal>UPDATE</literal> and <literal>DELETE</literal> are applied - while executing MERGE, depending on the actions that are activated. + defined for <command>SELECT</command>, <command>INSERT</command>, + <command>UPDATE</command> and <command>DELETE</command> are applied + while executing <command>MERGE</command>, depending on the actions that are activated. </para> </refsect1> diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml index b2a9f67cfa..c2901b0d58 100644 --- a/doc/src/sgml/ref/merge.sgml +++ b/doc/src/sgml/ref/merge.sgml @@ -98,7 +98,7 @@ DELETE </para> <para> - There is no MERGE privilege. + There is no <literal>MERGE</literal> privilege. You must have the <literal>UPDATE</literal> privilege on the column(s) of the <replaceable class="parameter">target_table_name</replaceable> referred to in the <literal>SET</literal> clause @@ -217,6 +217,7 @@ DELETE At least one <literal>WHEN</literal> clause is required. </para> <para> + /* XXX Do we need to repeat the same thing for WHEN MATCHED and WHEN NOT MATCHED? Could this say that it applies to both? */ If the <literal>WHEN</literal> clause specifies <literal>WHEN MATCHED</literal> and the candidate change row matches a row in the <replaceable class="parameter">target_table_name</replaceable> @@ -242,6 +243,7 @@ DELETE clause will be activated and the corresponding action will occur for that row. The expression may not contain functions that possibly performs writes to the database. + /* XXX Does it mean that it has to be marked as STABLE, or that a write crashes the DB? */ </para> <para> A condition on a <literal>WHEN MATCHED</literal> clause can refer to columns @@ -379,6 +381,7 @@ DELETE <para> An expression to assign to the column. The expression can use the old values of this and other columns in the table. + /* XXX Which table? Source or target, or both? What if it's NOT MATCHED? */ </para> </listitem> </varlistentry> @@ -492,6 +495,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> In summary, statement triggers for an event type (say, INSERT) will be fired whenever we <emphasis>specify</emphasis> an action of that kind. Row-level triggers will fire only for the one event type <emphasis>activated</emphasis>. + /* XXX What does "activated" mean? Maybe "executed" would be better? */ So a <command>MERGE</command> might fire statement triggers for both <command>UPDATE</command> and <command>INSERT</command>, even though only <command>UPDATE</command> row triggers were fired. @@ -504,6 +508,8 @@ MERGE <replaceable class="parameter">total-count</replaceable> rows will be used to modify the target row, later attempts to modify will cause an error. This can also occur if row triggers make changes to the target table which are then subsequently modified by <command>MERGE</command>. + /* XXX Not sure the preceding sentence makes sense. What does the 'which' refer to? Row triggers, changes, or what? */ + /* XXX It seems the rule is that INSERT actions are <literal> while INSERT statements (in SQL sense) are <command>. But this mixes that up? */ If the repeated action is an <command>INSERT</command> this will cause a uniqueness violation while a repeated <command>UPDATE</command> or <command>DELETE</command> will cause a cardinality violation; the latter behavior @@ -554,7 +560,7 @@ MERGE <replaceable class="parameter">total-count</replaceable> <title>Examples</title> <para> - Perform maintenance on CustomerAccounts based upon new Transactions. + Perform maintenance on <literal>CustomerAccounts</literal> based upon new <literal>Transactions</literal>. <programlisting> MERGE CustomerAccount CA @@ -599,7 +605,7 @@ WHEN MATCHED THEN DELETE; </programlisting> - The wine_stock_changes table might be, for example, a temporary table + The <literal>wine_stock_changes</literal> table might be, for example, a temporary table recently loaded into the database. </para> diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index fbaf50c258..f914a00e9f 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -3690,7 +3690,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors, break; case CMD_MERGE: operation = "Merge"; - foperation = "Foreign Merge"; + foperation = "Foreign Merge"; /* XXX Doesn't the commit message say foreign tables are not yet supported? */ break; default: operation = "???"; diff --git a/src/backend/executor/execMerge.c b/src/backend/executor/execMerge.c index 0e245e1361..a7492a6c4b 100644 --- a/src/backend/executor/execMerge.c +++ b/src/backend/executor/execMerge.c @@ -105,7 +105,7 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * A concurrent update can: * * 1. modify the target tuple so that it no longer satisfies the - * additional quals attached to the current WHEN MATCHED action OR + * additional quals attached to the current WHEN MATCHED action * * In this case, we are still dealing with a WHEN MATCHED case, but we * should recheck the list of WHEN MATCHED actions and choose the first @@ -118,6 +118,9 @@ ExecMerge(ModifyTableState *mtstate, ResultRelInfo *resultRelInfo, * tuple, so we now instead find a qualifying WHEN NOT MATCHED action to * execute. * + * XXX Hmmm, what if the updated tuple would now match one that was + * considered NOT MATCHED so far? + * * A concurrent delete, changes a WHEN MATCHED case to WHEN NOT MATCHED. * * ExecMergeMatched takes care of following the update chain and diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index c0a97eba91..9c14709e47 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -2124,7 +2124,6 @@ ExecModifyTable(PlanState *pstate) { /* advance to next subplan if any */ node->mt_whichplan++; - if (node->mt_whichplan < node->mt_nplans) { resultRelInfo++; @@ -2169,6 +2168,8 @@ ExecModifyTable(PlanState *pstate) EvalPlanQualSetSlot(&node->mt_epqstate, planSlot); slot = planSlot; + /* XXX Wouldn't it be "nicer" to handle MERGE in the switch, together + * with the other MT operations? This seems a bit out of place. */ if (operation == CMD_MERGE) { ExecMerge(node, resultRelInfo, estate, slot, junkfilter); diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c index 8848e9a03f..b2543f6814 100644 --- a/src/backend/optimizer/prep/preptlist.c +++ b/src/backend/optimizer/prep/preptlist.c @@ -117,6 +117,10 @@ preprocess_targetlist(PlannerInfo *root) tlist = expand_targetlist(tlist, command_type, result_relation, target_relation); + /* + * For MERGE we need to handle the target list for the target relation, + * and also target list for each action (only INSERT/UPDATE matter). + */ if (command_type == CMD_MERGE) { ListCell *l; @@ -343,6 +347,8 @@ expand_targetlist(List *tlist, int command_type, * generate a NULL for dropped columns (we want to drop any old * values). * + * XXX Should this explain why MERGE has the same logic as UPDATE? + * * When generating a NULL constant for a dropped column, we label * it INT4 (any other guaranteed-to-exist datatype would do as * well). We can't label it with the dropped column's datatype diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index f7c152beb4..5d49c8c37e 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -436,6 +436,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) break; case EXPR_KIND_MERGE_WHEN_AND: if (isAgg) + /* XXX "WHEN AND" seems rather strange. ... */ err = _("aggregate functions are not allowed in WHEN AND conditions"); else err = _("grouping operations are not allowed in WHEN AND conditions"); diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index d59f4fde95..d50543b1ba 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1198,6 +1198,7 @@ StartLogicalReplication(StartReplicationCmd *cmd) /* Also update the sent position status in shared memory */ SpinLockAcquire(&MyWalSnd->mutex); + /* XXX Huh? Why does MERGE patch change walsender? */ MyWalSnd->sentPtr = MyReplicationSlot->data.confirmed_flush; SpinLockRelease(&MyWalSnd->mutex); @@ -2930,8 +2931,7 @@ WalSndDone(WalSndSendDataCallback send_data) replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ? MyWalSnd->write : MyWalSnd->flush; - if (WalSndCaughtUp && - sentPtr == replicatedPtr && + if (WalSndCaughtUp && sentPtr == replicatedPtr && !pq_is_send_pending()) { QueryCompletion qc; diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c index e2706c181c..80ae946d17 100644 --- a/src/backend/rewrite/rewriteHandler.c +++ b/src/backend/rewrite/rewriteHandler.c @@ -1774,6 +1774,7 @@ fill_extraUpdatedCols(RangeTblEntry *target_rte, Relation target_relation) } } + /* * matchLocks - * match the list of locks and returns the matching rules @@ -3946,6 +3947,9 @@ RewriteQuery(Query *parsetree, List *rewrite_events) /* * XXX MERGE doesn't support write rules because they would violate * the SQL Standard spec and would be unclear how they should work. + * + * XXX So does't support means 'ignores'? Should that either fail + * or at least print some warning? */ if (event == CMD_MERGE) product_queries = NIL; diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 0281351dcf..7a768a7b5b 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -2312,6 +2312,7 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup) int result; bool arg1_match; + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -2505,6 +2506,7 @@ varstr_abbrev_convert(Datum original, SortSupport ssup) memset(pres, 0, sizeof(Datum)); len = VARSIZE_ANY_EXHDR(authoritative); + /* XXX Huh? Why is this in MERGE patch? */ if (len < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), @@ -3260,6 +3262,7 @@ bytea_catenate(bytea *t1, bytea *t2) len1 = VARSIZE_ANY_EXHDR(t1); len2 = VARSIZE_ANY_EXHDR(t2); + /* XXX Huh? Why is this in MERGE patch? */ if (len1 < 0 || len2 < 0) ereport(ERROR, (errcode(ERRCODE_DATA_CORRUPTED), diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 3cba38044e..08fea9b8f7 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -172,6 +172,7 @@ typedef struct Query List *withCheckOptions; /* a list of WithCheckOption's (added * during rewrite) */ + /* XXX Why not mergeTragetRelation? */ int mergeTarget_relation; List *mergeSourceTargetList; List *mergeActionList; /* list of actions for MERGE (only) */ @@ -1581,11 +1582,11 @@ typedef struct UpdateStmt typedef struct MergeStmt { NodeTag type; - RangeVar *relation; /* target relation to merge into */ + RangeVar *relation; /* target relation to merge into */ Node *source_relation; /* source relation */ - Node *join_condition; /* join condition between source and target */ + Node *join_condition; /* join condition between source and target */ List *mergeWhenClauses; /* list of MergeWhenClause(es) */ - WithClause *withClause; /* WITH clause */ + WithClause *withClause; /* WITH clause */ } MergeStmt; typedef struct MergeWhenClause -- 2.26.2 --------------B1AA4DC5EAE9BAF8B146244A Content-Type: text/plain; charset=UTF-8; name="0002-fix-valgrind-failure.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-fix-valgrind-failure.txt" ^ permalink raw reply [nested|flat] 56+ messages in thread
* [PATCH 2/2] review @ 2021-03-09 20:09 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 56+ messages in thread From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw) --- src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++------- src/backend/optimizer/path/pathkeys.c | 14 ++++- 2 files changed, 69 insertions(+), 22 deletions(-) diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index 1639258aaf..f55a4f20e5 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm) * is_fake_var * Workaround for generate_append_tlist() which generates fake Vars with * varno == 0, that will cause a fail of estimate_num_group() call + * + * XXX Ummm, why would estimate_num_group fail with this? */ static bool is_fake_var(Expr *expr) @@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr) * compute_cpu_sort_cost * compute CPU cost of sort (i.e. in-memory) * + * The main thing we need to calculate to estimate sort CPU costs is the number + * of calls to the comparator functions. The difficulty is that for multi-column + * sorts there may be different data types involved (for some of which the calls + * may be much more expensive). Furthermore, the columns may have very different + * number of distinct values - the higher the number, the fewer comparisons will + * be needed for the following columns. + * + * The algoritm is incremental - we add pathkeys one by one, and at each step we + * estimate the number of necessary comparisons (based on the number of distinct + * groups in the current pathkey prefix and the new pathkey), and the comparison + * costs (which is data type specific). + * + * Estimation of the number of comparisons is based on ideas from two sources: + * + * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/] + * + * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild, + * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906] + * + * In term of that paper, let N - number of tuples, Xi - number of tuples with + * key Ki, then the estimate of number of comparisons is: + * + * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) + * + * In our case all Xi are the same because now we don't have any estimation of + * group sizes, we have only know the estimate of number of groups (distinct + * values). In that case, formula becomes: + * + * N * log(NumberOfGroups) + * + * For multi-column sorts we need to estimate the number of comparisons for + * each individual column - for example with columns (c1, c2, ..., ck) we + * can estimate that number of comparions on ck is roughly + * + * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1)) + * + * Let k be a column number, Gk - number of groups defined by k columns, and Fk + * the cost of the comparison is + * + * N * sum( Fk * log(Gk) ) + * + * Note: We also consider column witdth, not just the comparator cost. + * * NOTE: some callers currently pass NIL for pathkeys because they * can't conveniently supply the sort keys. In this case, it will fallback to * simple comparison cost estimate. - * - * Estimation algorithm is based on ideas from course Algorithms, - * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper - * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild, - * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. - * - * In term of that papers, let N - number of tuples, Xi - number of tuples with - * key Ki, then estimation is: - * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) - * In our case all Xi are the same because now we don't have an estimation of - * group sizes, we have only estimation of number of groups. In this case, - * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation - * of multi-column sort we need separately compute each column, so, let k is a - * column number, Gk - number of groups defined by k columns: - * N * sum( Fk * log(Gk) ) - * Fk is a function costs (including width) for k columns. */ - static Cost compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, Cost comparison_cost, double tuples, double output_tuples, @@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, bool has_fake_var = false; int i = 0; Oid prev_datatype = InvalidOid; - Cost funcCost = 0.; + Cost funcCost = 0.0; List *cache_varinfos = NIL; /* fallback if pathkeys is unknown */ @@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * a total number of tuple comparisons of N log2 K; but the constant * factor is a bit higher than for quicksort. Tweak it so that the * cost curve is continuous at the crossover point. + * + * XXX I suppose the "quicksort factor" references to 1.5 at the end + * of this function, but I'm not sure. I suggest we introduce some simple + * constants for that, instead of magic values. */ output_tuples = (heapSort) ? 2.0 * output_tuples : tuples; per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples); @@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * - per column comparison function cost * - we try to compute needed number of comparison per column */ - foreach(lc, pathkeys) { PathKey *pathkey = (PathKey*)lfirst(lc); @@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * Don't use DEFAULT_NUM_DISTINCT because it used for only one * column while here we try to estimate number of groups over * set of columns. + * + * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to + * limit the calculated values, somehow? + * + * XXX What's the logic of the following formula? */ nGroups = ceil(2.0 + sqrt(tuples) * list_length(pathkeyExprs) / list_length(pathkeys)); @@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, { if (tuplesPerPrevGroup < output_tuples) /* comparing only inside output_tuples */ + /* XXX why not to use the same multiplier (1.5)? */ correctedNGroups = ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups)); else @@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups); /* - * We could skip all followed columns for cost estimation, because we + * We could skip all following columns for cost estimation, because we * believe that tuples are unique by set ot previous columns */ if (tuplesPerPrevGroup <= 1.0) diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index 7beb32488a..b092c3e055 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b) return 0; return 1; } + /* * Order tail of list of group pathkeys by uniqueness descendetly. It allows to * speedup sorting. Returns newly allocated lists, old ones stay untouched. * n_preordered defines a head of list which order should be prevented. + * + * XXX But we're not generating this only based on uniqueness (that's a bad + * term anyway, because we're using ndistinct estimates, not uniqueness). + * We're also using the comparator cost to calculate the expected sort cost, + * and optimize that. + * + * XXX This should explain how we generate the values - all permutations for + * up to 4 values, etc. */ void get_cheapest_group_keys_order(PlannerInfo *root, double nrows, @@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows, else { /* - * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to - * a new one which can then be cleaned safely if needed. + * Since v13 list_free() can clean list elements so for original list + * not to be modified it should be copied to a new one which can then + * be cleaned safely if needed. */ new_group_pathkeys = list_copy(*group_pathkeys); nToPermute = nFreeKeys; -- 2.29.2 --------------A955BFDEDE369DCD57A45C5A-- ^ permalink raw reply [nested|flat] 56+ messages in thread
* [PATCH 2/2] review @ 2021-03-09 20:09 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 56+ messages in thread From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw) --- src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++------- src/backend/optimizer/path/pathkeys.c | 14 ++++- 2 files changed, 69 insertions(+), 22 deletions(-) diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index 1639258aaf..f55a4f20e5 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm) * is_fake_var * Workaround for generate_append_tlist() which generates fake Vars with * varno == 0, that will cause a fail of estimate_num_group() call + * + * XXX Ummm, why would estimate_num_group fail with this? */ static bool is_fake_var(Expr *expr) @@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr) * compute_cpu_sort_cost * compute CPU cost of sort (i.e. in-memory) * + * The main thing we need to calculate to estimate sort CPU costs is the number + * of calls to the comparator functions. The difficulty is that for multi-column + * sorts there may be different data types involved (for some of which the calls + * may be much more expensive). Furthermore, the columns may have very different + * number of distinct values - the higher the number, the fewer comparisons will + * be needed for the following columns. + * + * The algoritm is incremental - we add pathkeys one by one, and at each step we + * estimate the number of necessary comparisons (based on the number of distinct + * groups in the current pathkey prefix and the new pathkey), and the comparison + * costs (which is data type specific). + * + * Estimation of the number of comparisons is based on ideas from two sources: + * + * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/] + * + * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild, + * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906] + * + * In term of that paper, let N - number of tuples, Xi - number of tuples with + * key Ki, then the estimate of number of comparisons is: + * + * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) + * + * In our case all Xi are the same because now we don't have any estimation of + * group sizes, we have only know the estimate of number of groups (distinct + * values). In that case, formula becomes: + * + * N * log(NumberOfGroups) + * + * For multi-column sorts we need to estimate the number of comparisons for + * each individual column - for example with columns (c1, c2, ..., ck) we + * can estimate that number of comparions on ck is roughly + * + * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1)) + * + * Let k be a column number, Gk - number of groups defined by k columns, and Fk + * the cost of the comparison is + * + * N * sum( Fk * log(Gk) ) + * + * Note: We also consider column witdth, not just the comparator cost. + * * NOTE: some callers currently pass NIL for pathkeys because they * can't conveniently supply the sort keys. In this case, it will fallback to * simple comparison cost estimate. - * - * Estimation algorithm is based on ideas from course Algorithms, - * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper - * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild, - * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. - * - * In term of that papers, let N - number of tuples, Xi - number of tuples with - * key Ki, then estimation is: - * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) - * In our case all Xi are the same because now we don't have an estimation of - * group sizes, we have only estimation of number of groups. In this case, - * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation - * of multi-column sort we need separately compute each column, so, let k is a - * column number, Gk - number of groups defined by k columns: - * N * sum( Fk * log(Gk) ) - * Fk is a function costs (including width) for k columns. */ - static Cost compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, Cost comparison_cost, double tuples, double output_tuples, @@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, bool has_fake_var = false; int i = 0; Oid prev_datatype = InvalidOid; - Cost funcCost = 0.; + Cost funcCost = 0.0; List *cache_varinfos = NIL; /* fallback if pathkeys is unknown */ @@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * a total number of tuple comparisons of N log2 K; but the constant * factor is a bit higher than for quicksort. Tweak it so that the * cost curve is continuous at the crossover point. + * + * XXX I suppose the "quicksort factor" references to 1.5 at the end + * of this function, but I'm not sure. I suggest we introduce some simple + * constants for that, instead of magic values. */ output_tuples = (heapSort) ? 2.0 * output_tuples : tuples; per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples); @@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * - per column comparison function cost * - we try to compute needed number of comparison per column */ - foreach(lc, pathkeys) { PathKey *pathkey = (PathKey*)lfirst(lc); @@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * Don't use DEFAULT_NUM_DISTINCT because it used for only one * column while here we try to estimate number of groups over * set of columns. + * + * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to + * limit the calculated values, somehow? + * + * XXX What's the logic of the following formula? */ nGroups = ceil(2.0 + sqrt(tuples) * list_length(pathkeyExprs) / list_length(pathkeys)); @@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, { if (tuplesPerPrevGroup < output_tuples) /* comparing only inside output_tuples */ + /* XXX why not to use the same multiplier (1.5)? */ correctedNGroups = ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups)); else @@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups); /* - * We could skip all followed columns for cost estimation, because we + * We could skip all following columns for cost estimation, because we * believe that tuples are unique by set ot previous columns */ if (tuplesPerPrevGroup <= 1.0) diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index 7beb32488a..b092c3e055 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b) return 0; return 1; } + /* * Order tail of list of group pathkeys by uniqueness descendetly. It allows to * speedup sorting. Returns newly allocated lists, old ones stay untouched. * n_preordered defines a head of list which order should be prevented. + * + * XXX But we're not generating this only based on uniqueness (that's a bad + * term anyway, because we're using ndistinct estimates, not uniqueness). + * We're also using the comparator cost to calculate the expected sort cost, + * and optimize that. + * + * XXX This should explain how we generate the values - all permutations for + * up to 4 values, etc. */ void get_cheapest_group_keys_order(PlannerInfo *root, double nrows, @@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows, else { /* - * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to - * a new one which can then be cleaned safely if needed. + * Since v13 list_free() can clean list elements so for original list + * not to be modified it should be copied to a new one which can then + * be cleaned safely if needed. */ new_group_pathkeys = list_copy(*group_pathkeys); nToPermute = nFreeKeys; -- 2.29.2 --------------A955BFDEDE369DCD57A45C5A-- ^ permalink raw reply [nested|flat] 56+ messages in thread
* [PATCH 2/2] review @ 2021-03-09 20:09 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 56+ messages in thread From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw) --- src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++------- src/backend/optimizer/path/pathkeys.c | 14 ++++- 2 files changed, 69 insertions(+), 22 deletions(-) diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index 1639258aaf..f55a4f20e5 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm) * is_fake_var * Workaround for generate_append_tlist() which generates fake Vars with * varno == 0, that will cause a fail of estimate_num_group() call + * + * XXX Ummm, why would estimate_num_group fail with this? */ static bool is_fake_var(Expr *expr) @@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr) * compute_cpu_sort_cost * compute CPU cost of sort (i.e. in-memory) * + * The main thing we need to calculate to estimate sort CPU costs is the number + * of calls to the comparator functions. The difficulty is that for multi-column + * sorts there may be different data types involved (for some of which the calls + * may be much more expensive). Furthermore, the columns may have very different + * number of distinct values - the higher the number, the fewer comparisons will + * be needed for the following columns. + * + * The algoritm is incremental - we add pathkeys one by one, and at each step we + * estimate the number of necessary comparisons (based on the number of distinct + * groups in the current pathkey prefix and the new pathkey), and the comparison + * costs (which is data type specific). + * + * Estimation of the number of comparisons is based on ideas from two sources: + * + * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/] + * + * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild, + * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906] + * + * In term of that paper, let N - number of tuples, Xi - number of tuples with + * key Ki, then the estimate of number of comparisons is: + * + * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) + * + * In our case all Xi are the same because now we don't have any estimation of + * group sizes, we have only know the estimate of number of groups (distinct + * values). In that case, formula becomes: + * + * N * log(NumberOfGroups) + * + * For multi-column sorts we need to estimate the number of comparisons for + * each individual column - for example with columns (c1, c2, ..., ck) we + * can estimate that number of comparions on ck is roughly + * + * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1)) + * + * Let k be a column number, Gk - number of groups defined by k columns, and Fk + * the cost of the comparison is + * + * N * sum( Fk * log(Gk) ) + * + * Note: We also consider column witdth, not just the comparator cost. + * * NOTE: some callers currently pass NIL for pathkeys because they * can't conveniently supply the sort keys. In this case, it will fallback to * simple comparison cost estimate. - * - * Estimation algorithm is based on ideas from course Algorithms, - * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper - * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild, - * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. - * - * In term of that papers, let N - number of tuples, Xi - number of tuples with - * key Ki, then estimation is: - * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) - * In our case all Xi are the same because now we don't have an estimation of - * group sizes, we have only estimation of number of groups. In this case, - * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation - * of multi-column sort we need separately compute each column, so, let k is a - * column number, Gk - number of groups defined by k columns: - * N * sum( Fk * log(Gk) ) - * Fk is a function costs (including width) for k columns. */ - static Cost compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, Cost comparison_cost, double tuples, double output_tuples, @@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, bool has_fake_var = false; int i = 0; Oid prev_datatype = InvalidOid; - Cost funcCost = 0.; + Cost funcCost = 0.0; List *cache_varinfos = NIL; /* fallback if pathkeys is unknown */ @@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * a total number of tuple comparisons of N log2 K; but the constant * factor is a bit higher than for quicksort. Tweak it so that the * cost curve is continuous at the crossover point. + * + * XXX I suppose the "quicksort factor" references to 1.5 at the end + * of this function, but I'm not sure. I suggest we introduce some simple + * constants for that, instead of magic values. */ output_tuples = (heapSort) ? 2.0 * output_tuples : tuples; per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples); @@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * - per column comparison function cost * - we try to compute needed number of comparison per column */ - foreach(lc, pathkeys) { PathKey *pathkey = (PathKey*)lfirst(lc); @@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * Don't use DEFAULT_NUM_DISTINCT because it used for only one * column while here we try to estimate number of groups over * set of columns. + * + * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to + * limit the calculated values, somehow? + * + * XXX What's the logic of the following formula? */ nGroups = ceil(2.0 + sqrt(tuples) * list_length(pathkeyExprs) / list_length(pathkeys)); @@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, { if (tuplesPerPrevGroup < output_tuples) /* comparing only inside output_tuples */ + /* XXX why not to use the same multiplier (1.5)? */ correctedNGroups = ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups)); else @@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups); /* - * We could skip all followed columns for cost estimation, because we + * We could skip all following columns for cost estimation, because we * believe that tuples are unique by set ot previous columns */ if (tuplesPerPrevGroup <= 1.0) diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index 7beb32488a..b092c3e055 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b) return 0; return 1; } + /* * Order tail of list of group pathkeys by uniqueness descendetly. It allows to * speedup sorting. Returns newly allocated lists, old ones stay untouched. * n_preordered defines a head of list which order should be prevented. + * + * XXX But we're not generating this only based on uniqueness (that's a bad + * term anyway, because we're using ndistinct estimates, not uniqueness). + * We're also using the comparator cost to calculate the expected sort cost, + * and optimize that. + * + * XXX This should explain how we generate the values - all permutations for + * up to 4 values, etc. */ void get_cheapest_group_keys_order(PlannerInfo *root, double nrows, @@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows, else { /* - * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to - * a new one which can then be cleaned safely if needed. + * Since v13 list_free() can clean list elements so for original list + * not to be modified it should be copied to a new one which can then + * be cleaned safely if needed. */ new_group_pathkeys = list_copy(*group_pathkeys); nToPermute = nFreeKeys; -- 2.29.2 --------------A955BFDEDE369DCD57A45C5A-- ^ permalink raw reply [nested|flat] 56+ messages in thread
* [PATCH 2/2] review @ 2021-03-09 20:09 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 56+ messages in thread From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw) --- src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++------- src/backend/optimizer/path/pathkeys.c | 14 ++++- 2 files changed, 69 insertions(+), 22 deletions(-) diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index 1639258aaf..f55a4f20e5 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm) * is_fake_var * Workaround for generate_append_tlist() which generates fake Vars with * varno == 0, that will cause a fail of estimate_num_group() call + * + * XXX Ummm, why would estimate_num_group fail with this? */ static bool is_fake_var(Expr *expr) @@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr) * compute_cpu_sort_cost * compute CPU cost of sort (i.e. in-memory) * + * The main thing we need to calculate to estimate sort CPU costs is the number + * of calls to the comparator functions. The difficulty is that for multi-column + * sorts there may be different data types involved (for some of which the calls + * may be much more expensive). Furthermore, the columns may have very different + * number of distinct values - the higher the number, the fewer comparisons will + * be needed for the following columns. + * + * The algoritm is incremental - we add pathkeys one by one, and at each step we + * estimate the number of necessary comparisons (based on the number of distinct + * groups in the current pathkey prefix and the new pathkey), and the comparison + * costs (which is data type specific). + * + * Estimation of the number of comparisons is based on ideas from two sources: + * + * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/] + * + * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild, + * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906] + * + * In term of that paper, let N - number of tuples, Xi - number of tuples with + * key Ki, then the estimate of number of comparisons is: + * + * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) + * + * In our case all Xi are the same because now we don't have any estimation of + * group sizes, we have only know the estimate of number of groups (distinct + * values). In that case, formula becomes: + * + * N * log(NumberOfGroups) + * + * For multi-column sorts we need to estimate the number of comparisons for + * each individual column - for example with columns (c1, c2, ..., ck) we + * can estimate that number of comparions on ck is roughly + * + * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1)) + * + * Let k be a column number, Gk - number of groups defined by k columns, and Fk + * the cost of the comparison is + * + * N * sum( Fk * log(Gk) ) + * + * Note: We also consider column witdth, not just the comparator cost. + * * NOTE: some callers currently pass NIL for pathkeys because they * can't conveniently supply the sort keys. In this case, it will fallback to * simple comparison cost estimate. - * - * Estimation algorithm is based on ideas from course Algorithms, - * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper - * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild, - * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. - * - * In term of that papers, let N - number of tuples, Xi - number of tuples with - * key Ki, then estimation is: - * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) - * In our case all Xi are the same because now we don't have an estimation of - * group sizes, we have only estimation of number of groups. In this case, - * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation - * of multi-column sort we need separately compute each column, so, let k is a - * column number, Gk - number of groups defined by k columns: - * N * sum( Fk * log(Gk) ) - * Fk is a function costs (including width) for k columns. */ - static Cost compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, Cost comparison_cost, double tuples, double output_tuples, @@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, bool has_fake_var = false; int i = 0; Oid prev_datatype = InvalidOid; - Cost funcCost = 0.; + Cost funcCost = 0.0; List *cache_varinfos = NIL; /* fallback if pathkeys is unknown */ @@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * a total number of tuple comparisons of N log2 K; but the constant * factor is a bit higher than for quicksort. Tweak it so that the * cost curve is continuous at the crossover point. + * + * XXX I suppose the "quicksort factor" references to 1.5 at the end + * of this function, but I'm not sure. I suggest we introduce some simple + * constants for that, instead of magic values. */ output_tuples = (heapSort) ? 2.0 * output_tuples : tuples; per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples); @@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * - per column comparison function cost * - we try to compute needed number of comparison per column */ - foreach(lc, pathkeys) { PathKey *pathkey = (PathKey*)lfirst(lc); @@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * Don't use DEFAULT_NUM_DISTINCT because it used for only one * column while here we try to estimate number of groups over * set of columns. + * + * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to + * limit the calculated values, somehow? + * + * XXX What's the logic of the following formula? */ nGroups = ceil(2.0 + sqrt(tuples) * list_length(pathkeyExprs) / list_length(pathkeys)); @@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, { if (tuplesPerPrevGroup < output_tuples) /* comparing only inside output_tuples */ + /* XXX why not to use the same multiplier (1.5)? */ correctedNGroups = ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups)); else @@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups); /* - * We could skip all followed columns for cost estimation, because we + * We could skip all following columns for cost estimation, because we * believe that tuples are unique by set ot previous columns */ if (tuplesPerPrevGroup <= 1.0) diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index 7beb32488a..b092c3e055 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b) return 0; return 1; } + /* * Order tail of list of group pathkeys by uniqueness descendetly. It allows to * speedup sorting. Returns newly allocated lists, old ones stay untouched. * n_preordered defines a head of list which order should be prevented. + * + * XXX But we're not generating this only based on uniqueness (that's a bad + * term anyway, because we're using ndistinct estimates, not uniqueness). + * We're also using the comparator cost to calculate the expected sort cost, + * and optimize that. + * + * XXX This should explain how we generate the values - all permutations for + * up to 4 values, etc. */ void get_cheapest_group_keys_order(PlannerInfo *root, double nrows, @@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows, else { /* - * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to - * a new one which can then be cleaned safely if needed. + * Since v13 list_free() can clean list elements so for original list + * not to be modified it should be copied to a new one which can then + * be cleaned safely if needed. */ new_group_pathkeys = list_copy(*group_pathkeys); nToPermute = nFreeKeys; -- 2.29.2 --------------A955BFDEDE369DCD57A45C5A-- ^ permalink raw reply [nested|flat] 56+ messages in thread
* [PATCH 2/2] review @ 2021-03-09 20:09 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 56+ messages in thread From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw) --- src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++------- src/backend/optimizer/path/pathkeys.c | 14 ++++- 2 files changed, 69 insertions(+), 22 deletions(-) diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index 1639258aaf..f55a4f20e5 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm) * is_fake_var * Workaround for generate_append_tlist() which generates fake Vars with * varno == 0, that will cause a fail of estimate_num_group() call + * + * XXX Ummm, why would estimate_num_group fail with this? */ static bool is_fake_var(Expr *expr) @@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr) * compute_cpu_sort_cost * compute CPU cost of sort (i.e. in-memory) * + * The main thing we need to calculate to estimate sort CPU costs is the number + * of calls to the comparator functions. The difficulty is that for multi-column + * sorts there may be different data types involved (for some of which the calls + * may be much more expensive). Furthermore, the columns may have very different + * number of distinct values - the higher the number, the fewer comparisons will + * be needed for the following columns. + * + * The algoritm is incremental - we add pathkeys one by one, and at each step we + * estimate the number of necessary comparisons (based on the number of distinct + * groups in the current pathkey prefix and the new pathkey), and the comparison + * costs (which is data type specific). + * + * Estimation of the number of comparisons is based on ideas from two sources: + * + * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/] + * + * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild, + * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906] + * + * In term of that paper, let N - number of tuples, Xi - number of tuples with + * key Ki, then the estimate of number of comparisons is: + * + * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) + * + * In our case all Xi are the same because now we don't have any estimation of + * group sizes, we have only know the estimate of number of groups (distinct + * values). In that case, formula becomes: + * + * N * log(NumberOfGroups) + * + * For multi-column sorts we need to estimate the number of comparisons for + * each individual column - for example with columns (c1, c2, ..., ck) we + * can estimate that number of comparions on ck is roughly + * + * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1)) + * + * Let k be a column number, Gk - number of groups defined by k columns, and Fk + * the cost of the comparison is + * + * N * sum( Fk * log(Gk) ) + * + * Note: We also consider column witdth, not just the comparator cost. + * * NOTE: some callers currently pass NIL for pathkeys because they * can't conveniently supply the sort keys. In this case, it will fallback to * simple comparison cost estimate. - * - * Estimation algorithm is based on ideas from course Algorithms, - * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper - * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild, - * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. - * - * In term of that papers, let N - number of tuples, Xi - number of tuples with - * key Ki, then estimation is: - * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) - * In our case all Xi are the same because now we don't have an estimation of - * group sizes, we have only estimation of number of groups. In this case, - * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation - * of multi-column sort we need separately compute each column, so, let k is a - * column number, Gk - number of groups defined by k columns: - * N * sum( Fk * log(Gk) ) - * Fk is a function costs (including width) for k columns. */ - static Cost compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, Cost comparison_cost, double tuples, double output_tuples, @@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, bool has_fake_var = false; int i = 0; Oid prev_datatype = InvalidOid; - Cost funcCost = 0.; + Cost funcCost = 0.0; List *cache_varinfos = NIL; /* fallback if pathkeys is unknown */ @@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * a total number of tuple comparisons of N log2 K; but the constant * factor is a bit higher than for quicksort. Tweak it so that the * cost curve is continuous at the crossover point. + * + * XXX I suppose the "quicksort factor" references to 1.5 at the end + * of this function, but I'm not sure. I suggest we introduce some simple + * constants for that, instead of magic values. */ output_tuples = (heapSort) ? 2.0 * output_tuples : tuples; per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples); @@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * - per column comparison function cost * - we try to compute needed number of comparison per column */ - foreach(lc, pathkeys) { PathKey *pathkey = (PathKey*)lfirst(lc); @@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * Don't use DEFAULT_NUM_DISTINCT because it used for only one * column while here we try to estimate number of groups over * set of columns. + * + * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to + * limit the calculated values, somehow? + * + * XXX What's the logic of the following formula? */ nGroups = ceil(2.0 + sqrt(tuples) * list_length(pathkeyExprs) / list_length(pathkeys)); @@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, { if (tuplesPerPrevGroup < output_tuples) /* comparing only inside output_tuples */ + /* XXX why not to use the same multiplier (1.5)? */ correctedNGroups = ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups)); else @@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups); /* - * We could skip all followed columns for cost estimation, because we + * We could skip all following columns for cost estimation, because we * believe that tuples are unique by set ot previous columns */ if (tuplesPerPrevGroup <= 1.0) diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index 7beb32488a..b092c3e055 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b) return 0; return 1; } + /* * Order tail of list of group pathkeys by uniqueness descendetly. It allows to * speedup sorting. Returns newly allocated lists, old ones stay untouched. * n_preordered defines a head of list which order should be prevented. + * + * XXX But we're not generating this only based on uniqueness (that's a bad + * term anyway, because we're using ndistinct estimates, not uniqueness). + * We're also using the comparator cost to calculate the expected sort cost, + * and optimize that. + * + * XXX This should explain how we generate the values - all permutations for + * up to 4 values, etc. */ void get_cheapest_group_keys_order(PlannerInfo *root, double nrows, @@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows, else { /* - * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to - * a new one which can then be cleaned safely if needed. + * Since v13 list_free() can clean list elements so for original list + * not to be modified it should be copied to a new one which can then + * be cleaned safely if needed. */ new_group_pathkeys = list_copy(*group_pathkeys); nToPermute = nFreeKeys; -- 2.29.2 --------------A955BFDEDE369DCD57A45C5A-- ^ permalink raw reply [nested|flat] 56+ messages in thread
* [PATCH 2/2] review @ 2021-03-09 20:09 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 56+ messages in thread From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw) --- src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++------- src/backend/optimizer/path/pathkeys.c | 14 ++++- 2 files changed, 69 insertions(+), 22 deletions(-) diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index 1639258aaf..f55a4f20e5 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm) * is_fake_var * Workaround for generate_append_tlist() which generates fake Vars with * varno == 0, that will cause a fail of estimate_num_group() call + * + * XXX Ummm, why would estimate_num_group fail with this? */ static bool is_fake_var(Expr *expr) @@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr) * compute_cpu_sort_cost * compute CPU cost of sort (i.e. in-memory) * + * The main thing we need to calculate to estimate sort CPU costs is the number + * of calls to the comparator functions. The difficulty is that for multi-column + * sorts there may be different data types involved (for some of which the calls + * may be much more expensive). Furthermore, the columns may have very different + * number of distinct values - the higher the number, the fewer comparisons will + * be needed for the following columns. + * + * The algoritm is incremental - we add pathkeys one by one, and at each step we + * estimate the number of necessary comparisons (based on the number of distinct + * groups in the current pathkey prefix and the new pathkey), and the comparison + * costs (which is data type specific). + * + * Estimation of the number of comparisons is based on ideas from two sources: + * + * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/] + * + * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild, + * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906] + * + * In term of that paper, let N - number of tuples, Xi - number of tuples with + * key Ki, then the estimate of number of comparisons is: + * + * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) + * + * In our case all Xi are the same because now we don't have any estimation of + * group sizes, we have only know the estimate of number of groups (distinct + * values). In that case, formula becomes: + * + * N * log(NumberOfGroups) + * + * For multi-column sorts we need to estimate the number of comparisons for + * each individual column - for example with columns (c1, c2, ..., ck) we + * can estimate that number of comparions on ck is roughly + * + * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1)) + * + * Let k be a column number, Gk - number of groups defined by k columns, and Fk + * the cost of the comparison is + * + * N * sum( Fk * log(Gk) ) + * + * Note: We also consider column witdth, not just the comparator cost. + * * NOTE: some callers currently pass NIL for pathkeys because they * can't conveniently supply the sort keys. In this case, it will fallback to * simple comparison cost estimate. - * - * Estimation algorithm is based on ideas from course Algorithms, - * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper - * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild, - * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. - * - * In term of that papers, let N - number of tuples, Xi - number of tuples with - * key Ki, then estimation is: - * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) - * In our case all Xi are the same because now we don't have an estimation of - * group sizes, we have only estimation of number of groups. In this case, - * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation - * of multi-column sort we need separately compute each column, so, let k is a - * column number, Gk - number of groups defined by k columns: - * N * sum( Fk * log(Gk) ) - * Fk is a function costs (including width) for k columns. */ - static Cost compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, Cost comparison_cost, double tuples, double output_tuples, @@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, bool has_fake_var = false; int i = 0; Oid prev_datatype = InvalidOid; - Cost funcCost = 0.; + Cost funcCost = 0.0; List *cache_varinfos = NIL; /* fallback if pathkeys is unknown */ @@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * a total number of tuple comparisons of N log2 K; but the constant * factor is a bit higher than for quicksort. Tweak it so that the * cost curve is continuous at the crossover point. + * + * XXX I suppose the "quicksort factor" references to 1.5 at the end + * of this function, but I'm not sure. I suggest we introduce some simple + * constants for that, instead of magic values. */ output_tuples = (heapSort) ? 2.0 * output_tuples : tuples; per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples); @@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * - per column comparison function cost * - we try to compute needed number of comparison per column */ - foreach(lc, pathkeys) { PathKey *pathkey = (PathKey*)lfirst(lc); @@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * Don't use DEFAULT_NUM_DISTINCT because it used for only one * column while here we try to estimate number of groups over * set of columns. + * + * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to + * limit the calculated values, somehow? + * + * XXX What's the logic of the following formula? */ nGroups = ceil(2.0 + sqrt(tuples) * list_length(pathkeyExprs) / list_length(pathkeys)); @@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, { if (tuplesPerPrevGroup < output_tuples) /* comparing only inside output_tuples */ + /* XXX why not to use the same multiplier (1.5)? */ correctedNGroups = ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups)); else @@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups); /* - * We could skip all followed columns for cost estimation, because we + * We could skip all following columns for cost estimation, because we * believe that tuples are unique by set ot previous columns */ if (tuplesPerPrevGroup <= 1.0) diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index 7beb32488a..b092c3e055 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b) return 0; return 1; } + /* * Order tail of list of group pathkeys by uniqueness descendetly. It allows to * speedup sorting. Returns newly allocated lists, old ones stay untouched. * n_preordered defines a head of list which order should be prevented. + * + * XXX But we're not generating this only based on uniqueness (that's a bad + * term anyway, because we're using ndistinct estimates, not uniqueness). + * We're also using the comparator cost to calculate the expected sort cost, + * and optimize that. + * + * XXX This should explain how we generate the values - all permutations for + * up to 4 values, etc. */ void get_cheapest_group_keys_order(PlannerInfo *root, double nrows, @@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows, else { /* - * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to - * a new one which can then be cleaned safely if needed. + * Since v13 list_free() can clean list elements so for original list + * not to be modified it should be copied to a new one which can then + * be cleaned safely if needed. */ new_group_pathkeys = list_copy(*group_pathkeys); nToPermute = nFreeKeys; -- 2.29.2 --------------A955BFDEDE369DCD57A45C5A-- ^ permalink raw reply [nested|flat] 56+ messages in thread
* [PATCH 2/2] review @ 2021-03-09 20:09 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 56+ messages in thread From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw) --- src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++------- src/backend/optimizer/path/pathkeys.c | 14 ++++- 2 files changed, 69 insertions(+), 22 deletions(-) diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index 1639258aaf..f55a4f20e5 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm) * is_fake_var * Workaround for generate_append_tlist() which generates fake Vars with * varno == 0, that will cause a fail of estimate_num_group() call + * + * XXX Ummm, why would estimate_num_group fail with this? */ static bool is_fake_var(Expr *expr) @@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr) * compute_cpu_sort_cost * compute CPU cost of sort (i.e. in-memory) * + * The main thing we need to calculate to estimate sort CPU costs is the number + * of calls to the comparator functions. The difficulty is that for multi-column + * sorts there may be different data types involved (for some of which the calls + * may be much more expensive). Furthermore, the columns may have very different + * number of distinct values - the higher the number, the fewer comparisons will + * be needed for the following columns. + * + * The algoritm is incremental - we add pathkeys one by one, and at each step we + * estimate the number of necessary comparisons (based on the number of distinct + * groups in the current pathkey prefix and the new pathkey), and the comparison + * costs (which is data type specific). + * + * Estimation of the number of comparisons is based on ideas from two sources: + * + * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/] + * + * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild, + * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906] + * + * In term of that paper, let N - number of tuples, Xi - number of tuples with + * key Ki, then the estimate of number of comparisons is: + * + * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) + * + * In our case all Xi are the same because now we don't have any estimation of + * group sizes, we have only know the estimate of number of groups (distinct + * values). In that case, formula becomes: + * + * N * log(NumberOfGroups) + * + * For multi-column sorts we need to estimate the number of comparisons for + * each individual column - for example with columns (c1, c2, ..., ck) we + * can estimate that number of comparions on ck is roughly + * + * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1)) + * + * Let k be a column number, Gk - number of groups defined by k columns, and Fk + * the cost of the comparison is + * + * N * sum( Fk * log(Gk) ) + * + * Note: We also consider column witdth, not just the comparator cost. + * * NOTE: some callers currently pass NIL for pathkeys because they * can't conveniently supply the sort keys. In this case, it will fallback to * simple comparison cost estimate. - * - * Estimation algorithm is based on ideas from course Algorithms, - * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper - * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild, - * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. - * - * In term of that papers, let N - number of tuples, Xi - number of tuples with - * key Ki, then estimation is: - * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) - * In our case all Xi are the same because now we don't have an estimation of - * group sizes, we have only estimation of number of groups. In this case, - * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation - * of multi-column sort we need separately compute each column, so, let k is a - * column number, Gk - number of groups defined by k columns: - * N * sum( Fk * log(Gk) ) - * Fk is a function costs (including width) for k columns. */ - static Cost compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, Cost comparison_cost, double tuples, double output_tuples, @@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, bool has_fake_var = false; int i = 0; Oid prev_datatype = InvalidOid; - Cost funcCost = 0.; + Cost funcCost = 0.0; List *cache_varinfos = NIL; /* fallback if pathkeys is unknown */ @@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * a total number of tuple comparisons of N log2 K; but the constant * factor is a bit higher than for quicksort. Tweak it so that the * cost curve is continuous at the crossover point. + * + * XXX I suppose the "quicksort factor" references to 1.5 at the end + * of this function, but I'm not sure. I suggest we introduce some simple + * constants for that, instead of magic values. */ output_tuples = (heapSort) ? 2.0 * output_tuples : tuples; per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples); @@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * - per column comparison function cost * - we try to compute needed number of comparison per column */ - foreach(lc, pathkeys) { PathKey *pathkey = (PathKey*)lfirst(lc); @@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * Don't use DEFAULT_NUM_DISTINCT because it used for only one * column while here we try to estimate number of groups over * set of columns. + * + * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to + * limit the calculated values, somehow? + * + * XXX What's the logic of the following formula? */ nGroups = ceil(2.0 + sqrt(tuples) * list_length(pathkeyExprs) / list_length(pathkeys)); @@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, { if (tuplesPerPrevGroup < output_tuples) /* comparing only inside output_tuples */ + /* XXX why not to use the same multiplier (1.5)? */ correctedNGroups = ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups)); else @@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups); /* - * We could skip all followed columns for cost estimation, because we + * We could skip all following columns for cost estimation, because we * believe that tuples are unique by set ot previous columns */ if (tuplesPerPrevGroup <= 1.0) diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index 7beb32488a..b092c3e055 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b) return 0; return 1; } + /* * Order tail of list of group pathkeys by uniqueness descendetly. It allows to * speedup sorting. Returns newly allocated lists, old ones stay untouched. * n_preordered defines a head of list which order should be prevented. + * + * XXX But we're not generating this only based on uniqueness (that's a bad + * term anyway, because we're using ndistinct estimates, not uniqueness). + * We're also using the comparator cost to calculate the expected sort cost, + * and optimize that. + * + * XXX This should explain how we generate the values - all permutations for + * up to 4 values, etc. */ void get_cheapest_group_keys_order(PlannerInfo *root, double nrows, @@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows, else { /* - * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to - * a new one which can then be cleaned safely if needed. + * Since v13 list_free() can clean list elements so for original list + * not to be modified it should be copied to a new one which can then + * be cleaned safely if needed. */ new_group_pathkeys = list_copy(*group_pathkeys); nToPermute = nFreeKeys; -- 2.29.2 --------------A955BFDEDE369DCD57A45C5A-- ^ permalink raw reply [nested|flat] 56+ messages in thread
* [PATCH 2/2] review @ 2021-03-09 20:09 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 56+ messages in thread From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw) --- src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++------- src/backend/optimizer/path/pathkeys.c | 14 ++++- 2 files changed, 69 insertions(+), 22 deletions(-) diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index 1639258aaf..f55a4f20e5 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm) * is_fake_var * Workaround for generate_append_tlist() which generates fake Vars with * varno == 0, that will cause a fail of estimate_num_group() call + * + * XXX Ummm, why would estimate_num_group fail with this? */ static bool is_fake_var(Expr *expr) @@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr) * compute_cpu_sort_cost * compute CPU cost of sort (i.e. in-memory) * + * The main thing we need to calculate to estimate sort CPU costs is the number + * of calls to the comparator functions. The difficulty is that for multi-column + * sorts there may be different data types involved (for some of which the calls + * may be much more expensive). Furthermore, the columns may have very different + * number of distinct values - the higher the number, the fewer comparisons will + * be needed for the following columns. + * + * The algoritm is incremental - we add pathkeys one by one, and at each step we + * estimate the number of necessary comparisons (based on the number of distinct + * groups in the current pathkey prefix and the new pathkey), and the comparison + * costs (which is data type specific). + * + * Estimation of the number of comparisons is based on ideas from two sources: + * + * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/] + * + * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild, + * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906] + * + * In term of that paper, let N - number of tuples, Xi - number of tuples with + * key Ki, then the estimate of number of comparisons is: + * + * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) + * + * In our case all Xi are the same because now we don't have any estimation of + * group sizes, we have only know the estimate of number of groups (distinct + * values). In that case, formula becomes: + * + * N * log(NumberOfGroups) + * + * For multi-column sorts we need to estimate the number of comparisons for + * each individual column - for example with columns (c1, c2, ..., ck) we + * can estimate that number of comparions on ck is roughly + * + * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1)) + * + * Let k be a column number, Gk - number of groups defined by k columns, and Fk + * the cost of the comparison is + * + * N * sum( Fk * log(Gk) ) + * + * Note: We also consider column witdth, not just the comparator cost. + * * NOTE: some callers currently pass NIL for pathkeys because they * can't conveniently supply the sort keys. In this case, it will fallback to * simple comparison cost estimate. - * - * Estimation algorithm is based on ideas from course Algorithms, - * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper - * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild, - * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. - * - * In term of that papers, let N - number of tuples, Xi - number of tuples with - * key Ki, then estimation is: - * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) - * In our case all Xi are the same because now we don't have an estimation of - * group sizes, we have only estimation of number of groups. In this case, - * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation - * of multi-column sort we need separately compute each column, so, let k is a - * column number, Gk - number of groups defined by k columns: - * N * sum( Fk * log(Gk) ) - * Fk is a function costs (including width) for k columns. */ - static Cost compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, Cost comparison_cost, double tuples, double output_tuples, @@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, bool has_fake_var = false; int i = 0; Oid prev_datatype = InvalidOid; - Cost funcCost = 0.; + Cost funcCost = 0.0; List *cache_varinfos = NIL; /* fallback if pathkeys is unknown */ @@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * a total number of tuple comparisons of N log2 K; but the constant * factor is a bit higher than for quicksort. Tweak it so that the * cost curve is continuous at the crossover point. + * + * XXX I suppose the "quicksort factor" references to 1.5 at the end + * of this function, but I'm not sure. I suggest we introduce some simple + * constants for that, instead of magic values. */ output_tuples = (heapSort) ? 2.0 * output_tuples : tuples; per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples); @@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * - per column comparison function cost * - we try to compute needed number of comparison per column */ - foreach(lc, pathkeys) { PathKey *pathkey = (PathKey*)lfirst(lc); @@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * Don't use DEFAULT_NUM_DISTINCT because it used for only one * column while here we try to estimate number of groups over * set of columns. + * + * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to + * limit the calculated values, somehow? + * + * XXX What's the logic of the following formula? */ nGroups = ceil(2.0 + sqrt(tuples) * list_length(pathkeyExprs) / list_length(pathkeys)); @@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, { if (tuplesPerPrevGroup < output_tuples) /* comparing only inside output_tuples */ + /* XXX why not to use the same multiplier (1.5)? */ correctedNGroups = ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups)); else @@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups); /* - * We could skip all followed columns for cost estimation, because we + * We could skip all following columns for cost estimation, because we * believe that tuples are unique by set ot previous columns */ if (tuplesPerPrevGroup <= 1.0) diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index 7beb32488a..b092c3e055 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b) return 0; return 1; } + /* * Order tail of list of group pathkeys by uniqueness descendetly. It allows to * speedup sorting. Returns newly allocated lists, old ones stay untouched. * n_preordered defines a head of list which order should be prevented. + * + * XXX But we're not generating this only based on uniqueness (that's a bad + * term anyway, because we're using ndistinct estimates, not uniqueness). + * We're also using the comparator cost to calculate the expected sort cost, + * and optimize that. + * + * XXX This should explain how we generate the values - all permutations for + * up to 4 values, etc. */ void get_cheapest_group_keys_order(PlannerInfo *root, double nrows, @@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows, else { /* - * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to - * a new one which can then be cleaned safely if needed. + * Since v13 list_free() can clean list elements so for original list + * not to be modified it should be copied to a new one which can then + * be cleaned safely if needed. */ new_group_pathkeys = list_copy(*group_pathkeys); nToPermute = nFreeKeys; -- 2.29.2 --------------A955BFDEDE369DCD57A45C5A-- ^ permalink raw reply [nested|flat] 56+ messages in thread
* [PATCH 2/2] review @ 2021-03-09 20:09 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 56+ messages in thread From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw) --- src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++------- src/backend/optimizer/path/pathkeys.c | 14 ++++- 2 files changed, 69 insertions(+), 22 deletions(-) diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index 1639258aaf..f55a4f20e5 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm) * is_fake_var * Workaround for generate_append_tlist() which generates fake Vars with * varno == 0, that will cause a fail of estimate_num_group() call + * + * XXX Ummm, why would estimate_num_group fail with this? */ static bool is_fake_var(Expr *expr) @@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr) * compute_cpu_sort_cost * compute CPU cost of sort (i.e. in-memory) * + * The main thing we need to calculate to estimate sort CPU costs is the number + * of calls to the comparator functions. The difficulty is that for multi-column + * sorts there may be different data types involved (for some of which the calls + * may be much more expensive). Furthermore, the columns may have very different + * number of distinct values - the higher the number, the fewer comparisons will + * be needed for the following columns. + * + * The algoritm is incremental - we add pathkeys one by one, and at each step we + * estimate the number of necessary comparisons (based on the number of distinct + * groups in the current pathkey prefix and the new pathkey), and the comparison + * costs (which is data type specific). + * + * Estimation of the number of comparisons is based on ideas from two sources: + * + * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/] + * + * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild, + * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906] + * + * In term of that paper, let N - number of tuples, Xi - number of tuples with + * key Ki, then the estimate of number of comparisons is: + * + * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) + * + * In our case all Xi are the same because now we don't have any estimation of + * group sizes, we have only know the estimate of number of groups (distinct + * values). In that case, formula becomes: + * + * N * log(NumberOfGroups) + * + * For multi-column sorts we need to estimate the number of comparisons for + * each individual column - for example with columns (c1, c2, ..., ck) we + * can estimate that number of comparions on ck is roughly + * + * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1)) + * + * Let k be a column number, Gk - number of groups defined by k columns, and Fk + * the cost of the comparison is + * + * N * sum( Fk * log(Gk) ) + * + * Note: We also consider column witdth, not just the comparator cost. + * * NOTE: some callers currently pass NIL for pathkeys because they * can't conveniently supply the sort keys. In this case, it will fallback to * simple comparison cost estimate. - * - * Estimation algorithm is based on ideas from course Algorithms, - * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper - * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild, - * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. - * - * In term of that papers, let N - number of tuples, Xi - number of tuples with - * key Ki, then estimation is: - * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) - * In our case all Xi are the same because now we don't have an estimation of - * group sizes, we have only estimation of number of groups. In this case, - * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation - * of multi-column sort we need separately compute each column, so, let k is a - * column number, Gk - number of groups defined by k columns: - * N * sum( Fk * log(Gk) ) - * Fk is a function costs (including width) for k columns. */ - static Cost compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, Cost comparison_cost, double tuples, double output_tuples, @@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, bool has_fake_var = false; int i = 0; Oid prev_datatype = InvalidOid; - Cost funcCost = 0.; + Cost funcCost = 0.0; List *cache_varinfos = NIL; /* fallback if pathkeys is unknown */ @@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * a total number of tuple comparisons of N log2 K; but the constant * factor is a bit higher than for quicksort. Tweak it so that the * cost curve is continuous at the crossover point. + * + * XXX I suppose the "quicksort factor" references to 1.5 at the end + * of this function, but I'm not sure. I suggest we introduce some simple + * constants for that, instead of magic values. */ output_tuples = (heapSort) ? 2.0 * output_tuples : tuples; per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples); @@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * - per column comparison function cost * - we try to compute needed number of comparison per column */ - foreach(lc, pathkeys) { PathKey *pathkey = (PathKey*)lfirst(lc); @@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * Don't use DEFAULT_NUM_DISTINCT because it used for only one * column while here we try to estimate number of groups over * set of columns. + * + * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to + * limit the calculated values, somehow? + * + * XXX What's the logic of the following formula? */ nGroups = ceil(2.0 + sqrt(tuples) * list_length(pathkeyExprs) / list_length(pathkeys)); @@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, { if (tuplesPerPrevGroup < output_tuples) /* comparing only inside output_tuples */ + /* XXX why not to use the same multiplier (1.5)? */ correctedNGroups = ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups)); else @@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups); /* - * We could skip all followed columns for cost estimation, because we + * We could skip all following columns for cost estimation, because we * believe that tuples are unique by set ot previous columns */ if (tuplesPerPrevGroup <= 1.0) diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index 7beb32488a..b092c3e055 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b) return 0; return 1; } + /* * Order tail of list of group pathkeys by uniqueness descendetly. It allows to * speedup sorting. Returns newly allocated lists, old ones stay untouched. * n_preordered defines a head of list which order should be prevented. + * + * XXX But we're not generating this only based on uniqueness (that's a bad + * term anyway, because we're using ndistinct estimates, not uniqueness). + * We're also using the comparator cost to calculate the expected sort cost, + * and optimize that. + * + * XXX This should explain how we generate the values - all permutations for + * up to 4 values, etc. */ void get_cheapest_group_keys_order(PlannerInfo *root, double nrows, @@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows, else { /* - * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to - * a new one which can then be cleaned safely if needed. + * Since v13 list_free() can clean list elements so for original list + * not to be modified it should be copied to a new one which can then + * be cleaned safely if needed. */ new_group_pathkeys = list_copy(*group_pathkeys); nToPermute = nFreeKeys; -- 2.29.2 --------------A955BFDEDE369DCD57A45C5A-- ^ permalink raw reply [nested|flat] 56+ messages in thread
* [PATCH 2/2] review @ 2021-03-09 20:09 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 56+ messages in thread From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw) --- src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++------- src/backend/optimizer/path/pathkeys.c | 14 ++++- 2 files changed, 69 insertions(+), 22 deletions(-) diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index 1639258aaf..f55a4f20e5 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm) * is_fake_var * Workaround for generate_append_tlist() which generates fake Vars with * varno == 0, that will cause a fail of estimate_num_group() call + * + * XXX Ummm, why would estimate_num_group fail with this? */ static bool is_fake_var(Expr *expr) @@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr) * compute_cpu_sort_cost * compute CPU cost of sort (i.e. in-memory) * + * The main thing we need to calculate to estimate sort CPU costs is the number + * of calls to the comparator functions. The difficulty is that for multi-column + * sorts there may be different data types involved (for some of which the calls + * may be much more expensive). Furthermore, the columns may have very different + * number of distinct values - the higher the number, the fewer comparisons will + * be needed for the following columns. + * + * The algoritm is incremental - we add pathkeys one by one, and at each step we + * estimate the number of necessary comparisons (based on the number of distinct + * groups in the current pathkey prefix and the new pathkey), and the comparison + * costs (which is data type specific). + * + * Estimation of the number of comparisons is based on ideas from two sources: + * + * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/] + * + * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild, + * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906] + * + * In term of that paper, let N - number of tuples, Xi - number of tuples with + * key Ki, then the estimate of number of comparisons is: + * + * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) + * + * In our case all Xi are the same because now we don't have any estimation of + * group sizes, we have only know the estimate of number of groups (distinct + * values). In that case, formula becomes: + * + * N * log(NumberOfGroups) + * + * For multi-column sorts we need to estimate the number of comparisons for + * each individual column - for example with columns (c1, c2, ..., ck) we + * can estimate that number of comparions on ck is roughly + * + * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1)) + * + * Let k be a column number, Gk - number of groups defined by k columns, and Fk + * the cost of the comparison is + * + * N * sum( Fk * log(Gk) ) + * + * Note: We also consider column witdth, not just the comparator cost. + * * NOTE: some callers currently pass NIL for pathkeys because they * can't conveniently supply the sort keys. In this case, it will fallback to * simple comparison cost estimate. - * - * Estimation algorithm is based on ideas from course Algorithms, - * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper - * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild, - * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. - * - * In term of that papers, let N - number of tuples, Xi - number of tuples with - * key Ki, then estimation is: - * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) - * In our case all Xi are the same because now we don't have an estimation of - * group sizes, we have only estimation of number of groups. In this case, - * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation - * of multi-column sort we need separately compute each column, so, let k is a - * column number, Gk - number of groups defined by k columns: - * N * sum( Fk * log(Gk) ) - * Fk is a function costs (including width) for k columns. */ - static Cost compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, Cost comparison_cost, double tuples, double output_tuples, @@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, bool has_fake_var = false; int i = 0; Oid prev_datatype = InvalidOid; - Cost funcCost = 0.; + Cost funcCost = 0.0; List *cache_varinfos = NIL; /* fallback if pathkeys is unknown */ @@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * a total number of tuple comparisons of N log2 K; but the constant * factor is a bit higher than for quicksort. Tweak it so that the * cost curve is continuous at the crossover point. + * + * XXX I suppose the "quicksort factor" references to 1.5 at the end + * of this function, but I'm not sure. I suggest we introduce some simple + * constants for that, instead of magic values. */ output_tuples = (heapSort) ? 2.0 * output_tuples : tuples; per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples); @@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * - per column comparison function cost * - we try to compute needed number of comparison per column */ - foreach(lc, pathkeys) { PathKey *pathkey = (PathKey*)lfirst(lc); @@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * Don't use DEFAULT_NUM_DISTINCT because it used for only one * column while here we try to estimate number of groups over * set of columns. + * + * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to + * limit the calculated values, somehow? + * + * XXX What's the logic of the following formula? */ nGroups = ceil(2.0 + sqrt(tuples) * list_length(pathkeyExprs) / list_length(pathkeys)); @@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, { if (tuplesPerPrevGroup < output_tuples) /* comparing only inside output_tuples */ + /* XXX why not to use the same multiplier (1.5)? */ correctedNGroups = ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups)); else @@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups); /* - * We could skip all followed columns for cost estimation, because we + * We could skip all following columns for cost estimation, because we * believe that tuples are unique by set ot previous columns */ if (tuplesPerPrevGroup <= 1.0) diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index 7beb32488a..b092c3e055 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b) return 0; return 1; } + /* * Order tail of list of group pathkeys by uniqueness descendetly. It allows to * speedup sorting. Returns newly allocated lists, old ones stay untouched. * n_preordered defines a head of list which order should be prevented. + * + * XXX But we're not generating this only based on uniqueness (that's a bad + * term anyway, because we're using ndistinct estimates, not uniqueness). + * We're also using the comparator cost to calculate the expected sort cost, + * and optimize that. + * + * XXX This should explain how we generate the values - all permutations for + * up to 4 values, etc. */ void get_cheapest_group_keys_order(PlannerInfo *root, double nrows, @@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows, else { /* - * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to - * a new one which can then be cleaned safely if needed. + * Since v13 list_free() can clean list elements so for original list + * not to be modified it should be copied to a new one which can then + * be cleaned safely if needed. */ new_group_pathkeys = list_copy(*group_pathkeys); nToPermute = nFreeKeys; -- 2.29.2 --------------A955BFDEDE369DCD57A45C5A-- ^ permalink raw reply [nested|flat] 56+ messages in thread
* [PATCH 2/2] review @ 2021-03-09 20:09 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 56+ messages in thread From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw) --- src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++------- src/backend/optimizer/path/pathkeys.c | 14 ++++- 2 files changed, 69 insertions(+), 22 deletions(-) diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index 1639258aaf..f55a4f20e5 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm) * is_fake_var * Workaround for generate_append_tlist() which generates fake Vars with * varno == 0, that will cause a fail of estimate_num_group() call + * + * XXX Ummm, why would estimate_num_group fail with this? */ static bool is_fake_var(Expr *expr) @@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr) * compute_cpu_sort_cost * compute CPU cost of sort (i.e. in-memory) * + * The main thing we need to calculate to estimate sort CPU costs is the number + * of calls to the comparator functions. The difficulty is that for multi-column + * sorts there may be different data types involved (for some of which the calls + * may be much more expensive). Furthermore, the columns may have very different + * number of distinct values - the higher the number, the fewer comparisons will + * be needed for the following columns. + * + * The algoritm is incremental - we add pathkeys one by one, and at each step we + * estimate the number of necessary comparisons (based on the number of distinct + * groups in the current pathkey prefix and the new pathkey), and the comparison + * costs (which is data type specific). + * + * Estimation of the number of comparisons is based on ideas from two sources: + * + * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/] + * + * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild, + * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906] + * + * In term of that paper, let N - number of tuples, Xi - number of tuples with + * key Ki, then the estimate of number of comparisons is: + * + * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) + * + * In our case all Xi are the same because now we don't have any estimation of + * group sizes, we have only know the estimate of number of groups (distinct + * values). In that case, formula becomes: + * + * N * log(NumberOfGroups) + * + * For multi-column sorts we need to estimate the number of comparisons for + * each individual column - for example with columns (c1, c2, ..., ck) we + * can estimate that number of comparions on ck is roughly + * + * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1)) + * + * Let k be a column number, Gk - number of groups defined by k columns, and Fk + * the cost of the comparison is + * + * N * sum( Fk * log(Gk) ) + * + * Note: We also consider column witdth, not just the comparator cost. + * * NOTE: some callers currently pass NIL for pathkeys because they * can't conveniently supply the sort keys. In this case, it will fallback to * simple comparison cost estimate. - * - * Estimation algorithm is based on ideas from course Algorithms, - * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper - * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild, - * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. - * - * In term of that papers, let N - number of tuples, Xi - number of tuples with - * key Ki, then estimation is: - * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) - * In our case all Xi are the same because now we don't have an estimation of - * group sizes, we have only estimation of number of groups. In this case, - * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation - * of multi-column sort we need separately compute each column, so, let k is a - * column number, Gk - number of groups defined by k columns: - * N * sum( Fk * log(Gk) ) - * Fk is a function costs (including width) for k columns. */ - static Cost compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, Cost comparison_cost, double tuples, double output_tuples, @@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, bool has_fake_var = false; int i = 0; Oid prev_datatype = InvalidOid; - Cost funcCost = 0.; + Cost funcCost = 0.0; List *cache_varinfos = NIL; /* fallback if pathkeys is unknown */ @@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * a total number of tuple comparisons of N log2 K; but the constant * factor is a bit higher than for quicksort. Tweak it so that the * cost curve is continuous at the crossover point. + * + * XXX I suppose the "quicksort factor" references to 1.5 at the end + * of this function, but I'm not sure. I suggest we introduce some simple + * constants for that, instead of magic values. */ output_tuples = (heapSort) ? 2.0 * output_tuples : tuples; per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples); @@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * - per column comparison function cost * - we try to compute needed number of comparison per column */ - foreach(lc, pathkeys) { PathKey *pathkey = (PathKey*)lfirst(lc); @@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * Don't use DEFAULT_NUM_DISTINCT because it used for only one * column while here we try to estimate number of groups over * set of columns. + * + * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to + * limit the calculated values, somehow? + * + * XXX What's the logic of the following formula? */ nGroups = ceil(2.0 + sqrt(tuples) * list_length(pathkeyExprs) / list_length(pathkeys)); @@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, { if (tuplesPerPrevGroup < output_tuples) /* comparing only inside output_tuples */ + /* XXX why not to use the same multiplier (1.5)? */ correctedNGroups = ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups)); else @@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups); /* - * We could skip all followed columns for cost estimation, because we + * We could skip all following columns for cost estimation, because we * believe that tuples are unique by set ot previous columns */ if (tuplesPerPrevGroup <= 1.0) diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index 7beb32488a..b092c3e055 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b) return 0; return 1; } + /* * Order tail of list of group pathkeys by uniqueness descendetly. It allows to * speedup sorting. Returns newly allocated lists, old ones stay untouched. * n_preordered defines a head of list which order should be prevented. + * + * XXX But we're not generating this only based on uniqueness (that's a bad + * term anyway, because we're using ndistinct estimates, not uniqueness). + * We're also using the comparator cost to calculate the expected sort cost, + * and optimize that. + * + * XXX This should explain how we generate the values - all permutations for + * up to 4 values, etc. */ void get_cheapest_group_keys_order(PlannerInfo *root, double nrows, @@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows, else { /* - * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to - * a new one which can then be cleaned safely if needed. + * Since v13 list_free() can clean list elements so for original list + * not to be modified it should be copied to a new one which can then + * be cleaned safely if needed. */ new_group_pathkeys = list_copy(*group_pathkeys); nToPermute = nFreeKeys; -- 2.29.2 --------------A955BFDEDE369DCD57A45C5A-- ^ permalink raw reply [nested|flat] 56+ messages in thread
* [PATCH 2/2] review @ 2021-03-09 20:09 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 56+ messages in thread From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw) --- src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++------- src/backend/optimizer/path/pathkeys.c | 14 ++++- 2 files changed, 69 insertions(+), 22 deletions(-) diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index 1639258aaf..f55a4f20e5 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm) * is_fake_var * Workaround for generate_append_tlist() which generates fake Vars with * varno == 0, that will cause a fail of estimate_num_group() call + * + * XXX Ummm, why would estimate_num_group fail with this? */ static bool is_fake_var(Expr *expr) @@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr) * compute_cpu_sort_cost * compute CPU cost of sort (i.e. in-memory) * + * The main thing we need to calculate to estimate sort CPU costs is the number + * of calls to the comparator functions. The difficulty is that for multi-column + * sorts there may be different data types involved (for some of which the calls + * may be much more expensive). Furthermore, the columns may have very different + * number of distinct values - the higher the number, the fewer comparisons will + * be needed for the following columns. + * + * The algoritm is incremental - we add pathkeys one by one, and at each step we + * estimate the number of necessary comparisons (based on the number of distinct + * groups in the current pathkey prefix and the new pathkey), and the comparison + * costs (which is data type specific). + * + * Estimation of the number of comparisons is based on ideas from two sources: + * + * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/] + * + * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild, + * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906] + * + * In term of that paper, let N - number of tuples, Xi - number of tuples with + * key Ki, then the estimate of number of comparisons is: + * + * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) + * + * In our case all Xi are the same because now we don't have any estimation of + * group sizes, we have only know the estimate of number of groups (distinct + * values). In that case, formula becomes: + * + * N * log(NumberOfGroups) + * + * For multi-column sorts we need to estimate the number of comparisons for + * each individual column - for example with columns (c1, c2, ..., ck) we + * can estimate that number of comparions on ck is roughly + * + * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1)) + * + * Let k be a column number, Gk - number of groups defined by k columns, and Fk + * the cost of the comparison is + * + * N * sum( Fk * log(Gk) ) + * + * Note: We also consider column witdth, not just the comparator cost. + * * NOTE: some callers currently pass NIL for pathkeys because they * can't conveniently supply the sort keys. In this case, it will fallback to * simple comparison cost estimate. - * - * Estimation algorithm is based on ideas from course Algorithms, - * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper - * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild, - * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. - * - * In term of that papers, let N - number of tuples, Xi - number of tuples with - * key Ki, then estimation is: - * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) - * In our case all Xi are the same because now we don't have an estimation of - * group sizes, we have only estimation of number of groups. In this case, - * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation - * of multi-column sort we need separately compute each column, so, let k is a - * column number, Gk - number of groups defined by k columns: - * N * sum( Fk * log(Gk) ) - * Fk is a function costs (including width) for k columns. */ - static Cost compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, Cost comparison_cost, double tuples, double output_tuples, @@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, bool has_fake_var = false; int i = 0; Oid prev_datatype = InvalidOid; - Cost funcCost = 0.; + Cost funcCost = 0.0; List *cache_varinfos = NIL; /* fallback if pathkeys is unknown */ @@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * a total number of tuple comparisons of N log2 K; but the constant * factor is a bit higher than for quicksort. Tweak it so that the * cost curve is continuous at the crossover point. + * + * XXX I suppose the "quicksort factor" references to 1.5 at the end + * of this function, but I'm not sure. I suggest we introduce some simple + * constants for that, instead of magic values. */ output_tuples = (heapSort) ? 2.0 * output_tuples : tuples; per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples); @@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * - per column comparison function cost * - we try to compute needed number of comparison per column */ - foreach(lc, pathkeys) { PathKey *pathkey = (PathKey*)lfirst(lc); @@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * Don't use DEFAULT_NUM_DISTINCT because it used for only one * column while here we try to estimate number of groups over * set of columns. + * + * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to + * limit the calculated values, somehow? + * + * XXX What's the logic of the following formula? */ nGroups = ceil(2.0 + sqrt(tuples) * list_length(pathkeyExprs) / list_length(pathkeys)); @@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, { if (tuplesPerPrevGroup < output_tuples) /* comparing only inside output_tuples */ + /* XXX why not to use the same multiplier (1.5)? */ correctedNGroups = ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups)); else @@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups); /* - * We could skip all followed columns for cost estimation, because we + * We could skip all following columns for cost estimation, because we * believe that tuples are unique by set ot previous columns */ if (tuplesPerPrevGroup <= 1.0) diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index 7beb32488a..b092c3e055 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b) return 0; return 1; } + /* * Order tail of list of group pathkeys by uniqueness descendetly. It allows to * speedup sorting. Returns newly allocated lists, old ones stay untouched. * n_preordered defines a head of list which order should be prevented. + * + * XXX But we're not generating this only based on uniqueness (that's a bad + * term anyway, because we're using ndistinct estimates, not uniqueness). + * We're also using the comparator cost to calculate the expected sort cost, + * and optimize that. + * + * XXX This should explain how we generate the values - all permutations for + * up to 4 values, etc. */ void get_cheapest_group_keys_order(PlannerInfo *root, double nrows, @@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows, else { /* - * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to - * a new one which can then be cleaned safely if needed. + * Since v13 list_free() can clean list elements so for original list + * not to be modified it should be copied to a new one which can then + * be cleaned safely if needed. */ new_group_pathkeys = list_copy(*group_pathkeys); nToPermute = nFreeKeys; -- 2.29.2 --------------A955BFDEDE369DCD57A45C5A-- ^ permalink raw reply [nested|flat] 56+ messages in thread
* [PATCH 2/2] review @ 2021-03-09 20:09 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 56+ messages in thread From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw) --- src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++------- src/backend/optimizer/path/pathkeys.c | 14 ++++- 2 files changed, 69 insertions(+), 22 deletions(-) diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index 1639258aaf..f55a4f20e5 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm) * is_fake_var * Workaround for generate_append_tlist() which generates fake Vars with * varno == 0, that will cause a fail of estimate_num_group() call + * + * XXX Ummm, why would estimate_num_group fail with this? */ static bool is_fake_var(Expr *expr) @@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr) * compute_cpu_sort_cost * compute CPU cost of sort (i.e. in-memory) * + * The main thing we need to calculate to estimate sort CPU costs is the number + * of calls to the comparator functions. The difficulty is that for multi-column + * sorts there may be different data types involved (for some of which the calls + * may be much more expensive). Furthermore, the columns may have very different + * number of distinct values - the higher the number, the fewer comparisons will + * be needed for the following columns. + * + * The algoritm is incremental - we add pathkeys one by one, and at each step we + * estimate the number of necessary comparisons (based on the number of distinct + * groups in the current pathkey prefix and the new pathkey), and the comparison + * costs (which is data type specific). + * + * Estimation of the number of comparisons is based on ideas from two sources: + * + * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/] + * + * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild, + * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906] + * + * In term of that paper, let N - number of tuples, Xi - number of tuples with + * key Ki, then the estimate of number of comparisons is: + * + * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) + * + * In our case all Xi are the same because now we don't have any estimation of + * group sizes, we have only know the estimate of number of groups (distinct + * values). In that case, formula becomes: + * + * N * log(NumberOfGroups) + * + * For multi-column sorts we need to estimate the number of comparisons for + * each individual column - for example with columns (c1, c2, ..., ck) we + * can estimate that number of comparions on ck is roughly + * + * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1)) + * + * Let k be a column number, Gk - number of groups defined by k columns, and Fk + * the cost of the comparison is + * + * N * sum( Fk * log(Gk) ) + * + * Note: We also consider column witdth, not just the comparator cost. + * * NOTE: some callers currently pass NIL for pathkeys because they * can't conveniently supply the sort keys. In this case, it will fallback to * simple comparison cost estimate. - * - * Estimation algorithm is based on ideas from course Algorithms, - * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper - * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild, - * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. - * - * In term of that papers, let N - number of tuples, Xi - number of tuples with - * key Ki, then estimation is: - * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) - * In our case all Xi are the same because now we don't have an estimation of - * group sizes, we have only estimation of number of groups. In this case, - * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation - * of multi-column sort we need separately compute each column, so, let k is a - * column number, Gk - number of groups defined by k columns: - * N * sum( Fk * log(Gk) ) - * Fk is a function costs (including width) for k columns. */ - static Cost compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, Cost comparison_cost, double tuples, double output_tuples, @@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, bool has_fake_var = false; int i = 0; Oid prev_datatype = InvalidOid; - Cost funcCost = 0.; + Cost funcCost = 0.0; List *cache_varinfos = NIL; /* fallback if pathkeys is unknown */ @@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * a total number of tuple comparisons of N log2 K; but the constant * factor is a bit higher than for quicksort. Tweak it so that the * cost curve is continuous at the crossover point. + * + * XXX I suppose the "quicksort factor" references to 1.5 at the end + * of this function, but I'm not sure. I suggest we introduce some simple + * constants for that, instead of magic values. */ output_tuples = (heapSort) ? 2.0 * output_tuples : tuples; per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples); @@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * - per column comparison function cost * - we try to compute needed number of comparison per column */ - foreach(lc, pathkeys) { PathKey *pathkey = (PathKey*)lfirst(lc); @@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * Don't use DEFAULT_NUM_DISTINCT because it used for only one * column while here we try to estimate number of groups over * set of columns. + * + * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to + * limit the calculated values, somehow? + * + * XXX What's the logic of the following formula? */ nGroups = ceil(2.0 + sqrt(tuples) * list_length(pathkeyExprs) / list_length(pathkeys)); @@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, { if (tuplesPerPrevGroup < output_tuples) /* comparing only inside output_tuples */ + /* XXX why not to use the same multiplier (1.5)? */ correctedNGroups = ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups)); else @@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups); /* - * We could skip all followed columns for cost estimation, because we + * We could skip all following columns for cost estimation, because we * believe that tuples are unique by set ot previous columns */ if (tuplesPerPrevGroup <= 1.0) diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index 7beb32488a..b092c3e055 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b) return 0; return 1; } + /* * Order tail of list of group pathkeys by uniqueness descendetly. It allows to * speedup sorting. Returns newly allocated lists, old ones stay untouched. * n_preordered defines a head of list which order should be prevented. + * + * XXX But we're not generating this only based on uniqueness (that's a bad + * term anyway, because we're using ndistinct estimates, not uniqueness). + * We're also using the comparator cost to calculate the expected sort cost, + * and optimize that. + * + * XXX This should explain how we generate the values - all permutations for + * up to 4 values, etc. */ void get_cheapest_group_keys_order(PlannerInfo *root, double nrows, @@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows, else { /* - * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to - * a new one which can then be cleaned safely if needed. + * Since v13 list_free() can clean list elements so for original list + * not to be modified it should be copied to a new one which can then + * be cleaned safely if needed. */ new_group_pathkeys = list_copy(*group_pathkeys); nToPermute = nFreeKeys; -- 2.29.2 --------------A955BFDEDE369DCD57A45C5A-- ^ permalink raw reply [nested|flat] 56+ messages in thread
* [PATCH 2/2] review @ 2021-03-09 20:09 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 56+ messages in thread From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw) --- src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++------- src/backend/optimizer/path/pathkeys.c | 14 ++++- 2 files changed, 69 insertions(+), 22 deletions(-) diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index 1639258aaf..f55a4f20e5 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm) * is_fake_var * Workaround for generate_append_tlist() which generates fake Vars with * varno == 0, that will cause a fail of estimate_num_group() call + * + * XXX Ummm, why would estimate_num_group fail with this? */ static bool is_fake_var(Expr *expr) @@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr) * compute_cpu_sort_cost * compute CPU cost of sort (i.e. in-memory) * + * The main thing we need to calculate to estimate sort CPU costs is the number + * of calls to the comparator functions. The difficulty is that for multi-column + * sorts there may be different data types involved (for some of which the calls + * may be much more expensive). Furthermore, the columns may have very different + * number of distinct values - the higher the number, the fewer comparisons will + * be needed for the following columns. + * + * The algoritm is incremental - we add pathkeys one by one, and at each step we + * estimate the number of necessary comparisons (based on the number of distinct + * groups in the current pathkey prefix and the new pathkey), and the comparison + * costs (which is data type specific). + * + * Estimation of the number of comparisons is based on ideas from two sources: + * + * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/] + * + * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild, + * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906] + * + * In term of that paper, let N - number of tuples, Xi - number of tuples with + * key Ki, then the estimate of number of comparisons is: + * + * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) + * + * In our case all Xi are the same because now we don't have any estimation of + * group sizes, we have only know the estimate of number of groups (distinct + * values). In that case, formula becomes: + * + * N * log(NumberOfGroups) + * + * For multi-column sorts we need to estimate the number of comparisons for + * each individual column - for example with columns (c1, c2, ..., ck) we + * can estimate that number of comparions on ck is roughly + * + * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1)) + * + * Let k be a column number, Gk - number of groups defined by k columns, and Fk + * the cost of the comparison is + * + * N * sum( Fk * log(Gk) ) + * + * Note: We also consider column witdth, not just the comparator cost. + * * NOTE: some callers currently pass NIL for pathkeys because they * can't conveniently supply the sort keys. In this case, it will fallback to * simple comparison cost estimate. - * - * Estimation algorithm is based on ideas from course Algorithms, - * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper - * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild, - * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. - * - * In term of that papers, let N - number of tuples, Xi - number of tuples with - * key Ki, then estimation is: - * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) - * In our case all Xi are the same because now we don't have an estimation of - * group sizes, we have only estimation of number of groups. In this case, - * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation - * of multi-column sort we need separately compute each column, so, let k is a - * column number, Gk - number of groups defined by k columns: - * N * sum( Fk * log(Gk) ) - * Fk is a function costs (including width) for k columns. */ - static Cost compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, Cost comparison_cost, double tuples, double output_tuples, @@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, bool has_fake_var = false; int i = 0; Oid prev_datatype = InvalidOid; - Cost funcCost = 0.; + Cost funcCost = 0.0; List *cache_varinfos = NIL; /* fallback if pathkeys is unknown */ @@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * a total number of tuple comparisons of N log2 K; but the constant * factor is a bit higher than for quicksort. Tweak it so that the * cost curve is continuous at the crossover point. + * + * XXX I suppose the "quicksort factor" references to 1.5 at the end + * of this function, but I'm not sure. I suggest we introduce some simple + * constants for that, instead of magic values. */ output_tuples = (heapSort) ? 2.0 * output_tuples : tuples; per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples); @@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * - per column comparison function cost * - we try to compute needed number of comparison per column */ - foreach(lc, pathkeys) { PathKey *pathkey = (PathKey*)lfirst(lc); @@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * Don't use DEFAULT_NUM_DISTINCT because it used for only one * column while here we try to estimate number of groups over * set of columns. + * + * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to + * limit the calculated values, somehow? + * + * XXX What's the logic of the following formula? */ nGroups = ceil(2.0 + sqrt(tuples) * list_length(pathkeyExprs) / list_length(pathkeys)); @@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, { if (tuplesPerPrevGroup < output_tuples) /* comparing only inside output_tuples */ + /* XXX why not to use the same multiplier (1.5)? */ correctedNGroups = ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups)); else @@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups); /* - * We could skip all followed columns for cost estimation, because we + * We could skip all following columns for cost estimation, because we * believe that tuples are unique by set ot previous columns */ if (tuplesPerPrevGroup <= 1.0) diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index 7beb32488a..b092c3e055 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b) return 0; return 1; } + /* * Order tail of list of group pathkeys by uniqueness descendetly. It allows to * speedup sorting. Returns newly allocated lists, old ones stay untouched. * n_preordered defines a head of list which order should be prevented. + * + * XXX But we're not generating this only based on uniqueness (that's a bad + * term anyway, because we're using ndistinct estimates, not uniqueness). + * We're also using the comparator cost to calculate the expected sort cost, + * and optimize that. + * + * XXX This should explain how we generate the values - all permutations for + * up to 4 values, etc. */ void get_cheapest_group_keys_order(PlannerInfo *root, double nrows, @@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows, else { /* - * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to - * a new one which can then be cleaned safely if needed. + * Since v13 list_free() can clean list elements so for original list + * not to be modified it should be copied to a new one which can then + * be cleaned safely if needed. */ new_group_pathkeys = list_copy(*group_pathkeys); nToPermute = nFreeKeys; -- 2.29.2 --------------A955BFDEDE369DCD57A45C5A-- ^ permalink raw reply [nested|flat] 56+ messages in thread
* [PATCH 2/2] review @ 2021-03-09 20:09 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 56+ messages in thread From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw) --- src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++------- src/backend/optimizer/path/pathkeys.c | 14 ++++- 2 files changed, 69 insertions(+), 22 deletions(-) diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index 1639258aaf..f55a4f20e5 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm) * is_fake_var * Workaround for generate_append_tlist() which generates fake Vars with * varno == 0, that will cause a fail of estimate_num_group() call + * + * XXX Ummm, why would estimate_num_group fail with this? */ static bool is_fake_var(Expr *expr) @@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr) * compute_cpu_sort_cost * compute CPU cost of sort (i.e. in-memory) * + * The main thing we need to calculate to estimate sort CPU costs is the number + * of calls to the comparator functions. The difficulty is that for multi-column + * sorts there may be different data types involved (for some of which the calls + * may be much more expensive). Furthermore, the columns may have very different + * number of distinct values - the higher the number, the fewer comparisons will + * be needed for the following columns. + * + * The algoritm is incremental - we add pathkeys one by one, and at each step we + * estimate the number of necessary comparisons (based on the number of distinct + * groups in the current pathkey prefix and the new pathkey), and the comparison + * costs (which is data type specific). + * + * Estimation of the number of comparisons is based on ideas from two sources: + * + * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/] + * + * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild, + * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906] + * + * In term of that paper, let N - number of tuples, Xi - number of tuples with + * key Ki, then the estimate of number of comparisons is: + * + * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) + * + * In our case all Xi are the same because now we don't have any estimation of + * group sizes, we have only know the estimate of number of groups (distinct + * values). In that case, formula becomes: + * + * N * log(NumberOfGroups) + * + * For multi-column sorts we need to estimate the number of comparisons for + * each individual column - for example with columns (c1, c2, ..., ck) we + * can estimate that number of comparions on ck is roughly + * + * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1)) + * + * Let k be a column number, Gk - number of groups defined by k columns, and Fk + * the cost of the comparison is + * + * N * sum( Fk * log(Gk) ) + * + * Note: We also consider column witdth, not just the comparator cost. + * * NOTE: some callers currently pass NIL for pathkeys because they * can't conveniently supply the sort keys. In this case, it will fallback to * simple comparison cost estimate. - * - * Estimation algorithm is based on ideas from course Algorithms, - * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper - * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild, - * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. - * - * In term of that papers, let N - number of tuples, Xi - number of tuples with - * key Ki, then estimation is: - * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) - * In our case all Xi are the same because now we don't have an estimation of - * group sizes, we have only estimation of number of groups. In this case, - * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation - * of multi-column sort we need separately compute each column, so, let k is a - * column number, Gk - number of groups defined by k columns: - * N * sum( Fk * log(Gk) ) - * Fk is a function costs (including width) for k columns. */ - static Cost compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, Cost comparison_cost, double tuples, double output_tuples, @@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, bool has_fake_var = false; int i = 0; Oid prev_datatype = InvalidOid; - Cost funcCost = 0.; + Cost funcCost = 0.0; List *cache_varinfos = NIL; /* fallback if pathkeys is unknown */ @@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * a total number of tuple comparisons of N log2 K; but the constant * factor is a bit higher than for quicksort. Tweak it so that the * cost curve is continuous at the crossover point. + * + * XXX I suppose the "quicksort factor" references to 1.5 at the end + * of this function, but I'm not sure. I suggest we introduce some simple + * constants for that, instead of magic values. */ output_tuples = (heapSort) ? 2.0 * output_tuples : tuples; per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples); @@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * - per column comparison function cost * - we try to compute needed number of comparison per column */ - foreach(lc, pathkeys) { PathKey *pathkey = (PathKey*)lfirst(lc); @@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * Don't use DEFAULT_NUM_DISTINCT because it used for only one * column while here we try to estimate number of groups over * set of columns. + * + * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to + * limit the calculated values, somehow? + * + * XXX What's the logic of the following formula? */ nGroups = ceil(2.0 + sqrt(tuples) * list_length(pathkeyExprs) / list_length(pathkeys)); @@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, { if (tuplesPerPrevGroup < output_tuples) /* comparing only inside output_tuples */ + /* XXX why not to use the same multiplier (1.5)? */ correctedNGroups = ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups)); else @@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups); /* - * We could skip all followed columns for cost estimation, because we + * We could skip all following columns for cost estimation, because we * believe that tuples are unique by set ot previous columns */ if (tuplesPerPrevGroup <= 1.0) diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index 7beb32488a..b092c3e055 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b) return 0; return 1; } + /* * Order tail of list of group pathkeys by uniqueness descendetly. It allows to * speedup sorting. Returns newly allocated lists, old ones stay untouched. * n_preordered defines a head of list which order should be prevented. + * + * XXX But we're not generating this only based on uniqueness (that's a bad + * term anyway, because we're using ndistinct estimates, not uniqueness). + * We're also using the comparator cost to calculate the expected sort cost, + * and optimize that. + * + * XXX This should explain how we generate the values - all permutations for + * up to 4 values, etc. */ void get_cheapest_group_keys_order(PlannerInfo *root, double nrows, @@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows, else { /* - * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to - * a new one which can then be cleaned safely if needed. + * Since v13 list_free() can clean list elements so for original list + * not to be modified it should be copied to a new one which can then + * be cleaned safely if needed. */ new_group_pathkeys = list_copy(*group_pathkeys); nToPermute = nFreeKeys; -- 2.29.2 --------------A955BFDEDE369DCD57A45C5A-- ^ permalink raw reply [nested|flat] 56+ messages in thread
* [PATCH 2/2] review @ 2021-03-09 20:09 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 56+ messages in thread From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw) --- src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++------- src/backend/optimizer/path/pathkeys.c | 14 ++++- 2 files changed, 69 insertions(+), 22 deletions(-) diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index 1639258aaf..f55a4f20e5 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm) * is_fake_var * Workaround for generate_append_tlist() which generates fake Vars with * varno == 0, that will cause a fail of estimate_num_group() call + * + * XXX Ummm, why would estimate_num_group fail with this? */ static bool is_fake_var(Expr *expr) @@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr) * compute_cpu_sort_cost * compute CPU cost of sort (i.e. in-memory) * + * The main thing we need to calculate to estimate sort CPU costs is the number + * of calls to the comparator functions. The difficulty is that for multi-column + * sorts there may be different data types involved (for some of which the calls + * may be much more expensive). Furthermore, the columns may have very different + * number of distinct values - the higher the number, the fewer comparisons will + * be needed for the following columns. + * + * The algoritm is incremental - we add pathkeys one by one, and at each step we + * estimate the number of necessary comparisons (based on the number of distinct + * groups in the current pathkey prefix and the new pathkey), and the comparison + * costs (which is data type specific). + * + * Estimation of the number of comparisons is based on ideas from two sources: + * + * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/] + * + * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild, + * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906] + * + * In term of that paper, let N - number of tuples, Xi - number of tuples with + * key Ki, then the estimate of number of comparisons is: + * + * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) + * + * In our case all Xi are the same because now we don't have any estimation of + * group sizes, we have only know the estimate of number of groups (distinct + * values). In that case, formula becomes: + * + * N * log(NumberOfGroups) + * + * For multi-column sorts we need to estimate the number of comparisons for + * each individual column - for example with columns (c1, c2, ..., ck) we + * can estimate that number of comparions on ck is roughly + * + * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1)) + * + * Let k be a column number, Gk - number of groups defined by k columns, and Fk + * the cost of the comparison is + * + * N * sum( Fk * log(Gk) ) + * + * Note: We also consider column witdth, not just the comparator cost. + * * NOTE: some callers currently pass NIL for pathkeys because they * can't conveniently supply the sort keys. In this case, it will fallback to * simple comparison cost estimate. - * - * Estimation algorithm is based on ideas from course Algorithms, - * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper - * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild, - * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. - * - * In term of that papers, let N - number of tuples, Xi - number of tuples with - * key Ki, then estimation is: - * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) - * In our case all Xi are the same because now we don't have an estimation of - * group sizes, we have only estimation of number of groups. In this case, - * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation - * of multi-column sort we need separately compute each column, so, let k is a - * column number, Gk - number of groups defined by k columns: - * N * sum( Fk * log(Gk) ) - * Fk is a function costs (including width) for k columns. */ - static Cost compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, Cost comparison_cost, double tuples, double output_tuples, @@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, bool has_fake_var = false; int i = 0; Oid prev_datatype = InvalidOid; - Cost funcCost = 0.; + Cost funcCost = 0.0; List *cache_varinfos = NIL; /* fallback if pathkeys is unknown */ @@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * a total number of tuple comparisons of N log2 K; but the constant * factor is a bit higher than for quicksort. Tweak it so that the * cost curve is continuous at the crossover point. + * + * XXX I suppose the "quicksort factor" references to 1.5 at the end + * of this function, but I'm not sure. I suggest we introduce some simple + * constants for that, instead of magic values. */ output_tuples = (heapSort) ? 2.0 * output_tuples : tuples; per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples); @@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * - per column comparison function cost * - we try to compute needed number of comparison per column */ - foreach(lc, pathkeys) { PathKey *pathkey = (PathKey*)lfirst(lc); @@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * Don't use DEFAULT_NUM_DISTINCT because it used for only one * column while here we try to estimate number of groups over * set of columns. + * + * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to + * limit the calculated values, somehow? + * + * XXX What's the logic of the following formula? */ nGroups = ceil(2.0 + sqrt(tuples) * list_length(pathkeyExprs) / list_length(pathkeys)); @@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, { if (tuplesPerPrevGroup < output_tuples) /* comparing only inside output_tuples */ + /* XXX why not to use the same multiplier (1.5)? */ correctedNGroups = ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups)); else @@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups); /* - * We could skip all followed columns for cost estimation, because we + * We could skip all following columns for cost estimation, because we * believe that tuples are unique by set ot previous columns */ if (tuplesPerPrevGroup <= 1.0) diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index 7beb32488a..b092c3e055 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b) return 0; return 1; } + /* * Order tail of list of group pathkeys by uniqueness descendetly. It allows to * speedup sorting. Returns newly allocated lists, old ones stay untouched. * n_preordered defines a head of list which order should be prevented. + * + * XXX But we're not generating this only based on uniqueness (that's a bad + * term anyway, because we're using ndistinct estimates, not uniqueness). + * We're also using the comparator cost to calculate the expected sort cost, + * and optimize that. + * + * XXX This should explain how we generate the values - all permutations for + * up to 4 values, etc. */ void get_cheapest_group_keys_order(PlannerInfo *root, double nrows, @@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows, else { /* - * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to - * a new one which can then be cleaned safely if needed. + * Since v13 list_free() can clean list elements so for original list + * not to be modified it should be copied to a new one which can then + * be cleaned safely if needed. */ new_group_pathkeys = list_copy(*group_pathkeys); nToPermute = nFreeKeys; -- 2.29.2 --------------A955BFDEDE369DCD57A45C5A-- ^ permalink raw reply [nested|flat] 56+ messages in thread
* [PATCH 2/2] review @ 2021-03-09 20:09 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 56+ messages in thread From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw) --- src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++------- src/backend/optimizer/path/pathkeys.c | 14 ++++- 2 files changed, 69 insertions(+), 22 deletions(-) diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index 1639258aaf..f55a4f20e5 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm) * is_fake_var * Workaround for generate_append_tlist() which generates fake Vars with * varno == 0, that will cause a fail of estimate_num_group() call + * + * XXX Ummm, why would estimate_num_group fail with this? */ static bool is_fake_var(Expr *expr) @@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr) * compute_cpu_sort_cost * compute CPU cost of sort (i.e. in-memory) * + * The main thing we need to calculate to estimate sort CPU costs is the number + * of calls to the comparator functions. The difficulty is that for multi-column + * sorts there may be different data types involved (for some of which the calls + * may be much more expensive). Furthermore, the columns may have very different + * number of distinct values - the higher the number, the fewer comparisons will + * be needed for the following columns. + * + * The algoritm is incremental - we add pathkeys one by one, and at each step we + * estimate the number of necessary comparisons (based on the number of distinct + * groups in the current pathkey prefix and the new pathkey), and the comparison + * costs (which is data type specific). + * + * Estimation of the number of comparisons is based on ideas from two sources: + * + * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/] + * + * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild, + * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906] + * + * In term of that paper, let N - number of tuples, Xi - number of tuples with + * key Ki, then the estimate of number of comparisons is: + * + * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) + * + * In our case all Xi are the same because now we don't have any estimation of + * group sizes, we have only know the estimate of number of groups (distinct + * values). In that case, formula becomes: + * + * N * log(NumberOfGroups) + * + * For multi-column sorts we need to estimate the number of comparisons for + * each individual column - for example with columns (c1, c2, ..., ck) we + * can estimate that number of comparions on ck is roughly + * + * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1)) + * + * Let k be a column number, Gk - number of groups defined by k columns, and Fk + * the cost of the comparison is + * + * N * sum( Fk * log(Gk) ) + * + * Note: We also consider column witdth, not just the comparator cost. + * * NOTE: some callers currently pass NIL for pathkeys because they * can't conveniently supply the sort keys. In this case, it will fallback to * simple comparison cost estimate. - * - * Estimation algorithm is based on ideas from course Algorithms, - * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper - * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild, - * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. - * - * In term of that papers, let N - number of tuples, Xi - number of tuples with - * key Ki, then estimation is: - * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) - * In our case all Xi are the same because now we don't have an estimation of - * group sizes, we have only estimation of number of groups. In this case, - * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation - * of multi-column sort we need separately compute each column, so, let k is a - * column number, Gk - number of groups defined by k columns: - * N * sum( Fk * log(Gk) ) - * Fk is a function costs (including width) for k columns. */ - static Cost compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, Cost comparison_cost, double tuples, double output_tuples, @@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, bool has_fake_var = false; int i = 0; Oid prev_datatype = InvalidOid; - Cost funcCost = 0.; + Cost funcCost = 0.0; List *cache_varinfos = NIL; /* fallback if pathkeys is unknown */ @@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * a total number of tuple comparisons of N log2 K; but the constant * factor is a bit higher than for quicksort. Tweak it so that the * cost curve is continuous at the crossover point. + * + * XXX I suppose the "quicksort factor" references to 1.5 at the end + * of this function, but I'm not sure. I suggest we introduce some simple + * constants for that, instead of magic values. */ output_tuples = (heapSort) ? 2.0 * output_tuples : tuples; per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples); @@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * - per column comparison function cost * - we try to compute needed number of comparison per column */ - foreach(lc, pathkeys) { PathKey *pathkey = (PathKey*)lfirst(lc); @@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * Don't use DEFAULT_NUM_DISTINCT because it used for only one * column while here we try to estimate number of groups over * set of columns. + * + * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to + * limit the calculated values, somehow? + * + * XXX What's the logic of the following formula? */ nGroups = ceil(2.0 + sqrt(tuples) * list_length(pathkeyExprs) / list_length(pathkeys)); @@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, { if (tuplesPerPrevGroup < output_tuples) /* comparing only inside output_tuples */ + /* XXX why not to use the same multiplier (1.5)? */ correctedNGroups = ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups)); else @@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups); /* - * We could skip all followed columns for cost estimation, because we + * We could skip all following columns for cost estimation, because we * believe that tuples are unique by set ot previous columns */ if (tuplesPerPrevGroup <= 1.0) diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index 7beb32488a..b092c3e055 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b) return 0; return 1; } + /* * Order tail of list of group pathkeys by uniqueness descendetly. It allows to * speedup sorting. Returns newly allocated lists, old ones stay untouched. * n_preordered defines a head of list which order should be prevented. + * + * XXX But we're not generating this only based on uniqueness (that's a bad + * term anyway, because we're using ndistinct estimates, not uniqueness). + * We're also using the comparator cost to calculate the expected sort cost, + * and optimize that. + * + * XXX This should explain how we generate the values - all permutations for + * up to 4 values, etc. */ void get_cheapest_group_keys_order(PlannerInfo *root, double nrows, @@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows, else { /* - * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to - * a new one which can then be cleaned safely if needed. + * Since v13 list_free() can clean list elements so for original list + * not to be modified it should be copied to a new one which can then + * be cleaned safely if needed. */ new_group_pathkeys = list_copy(*group_pathkeys); nToPermute = nFreeKeys; -- 2.29.2 --------------A955BFDEDE369DCD57A45C5A-- ^ permalink raw reply [nested|flat] 56+ messages in thread
* [PATCH 2/2] review @ 2021-03-09 20:09 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 56+ messages in thread From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw) --- src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++------- src/backend/optimizer/path/pathkeys.c | 14 ++++- 2 files changed, 69 insertions(+), 22 deletions(-) diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index 1639258aaf..f55a4f20e5 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm) * is_fake_var * Workaround for generate_append_tlist() which generates fake Vars with * varno == 0, that will cause a fail of estimate_num_group() call + * + * XXX Ummm, why would estimate_num_group fail with this? */ static bool is_fake_var(Expr *expr) @@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr) * compute_cpu_sort_cost * compute CPU cost of sort (i.e. in-memory) * + * The main thing we need to calculate to estimate sort CPU costs is the number + * of calls to the comparator functions. The difficulty is that for multi-column + * sorts there may be different data types involved (for some of which the calls + * may be much more expensive). Furthermore, the columns may have very different + * number of distinct values - the higher the number, the fewer comparisons will + * be needed for the following columns. + * + * The algoritm is incremental - we add pathkeys one by one, and at each step we + * estimate the number of necessary comparisons (based on the number of distinct + * groups in the current pathkey prefix and the new pathkey), and the comparison + * costs (which is data type specific). + * + * Estimation of the number of comparisons is based on ideas from two sources: + * + * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/] + * + * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild, + * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906] + * + * In term of that paper, let N - number of tuples, Xi - number of tuples with + * key Ki, then the estimate of number of comparisons is: + * + * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) + * + * In our case all Xi are the same because now we don't have any estimation of + * group sizes, we have only know the estimate of number of groups (distinct + * values). In that case, formula becomes: + * + * N * log(NumberOfGroups) + * + * For multi-column sorts we need to estimate the number of comparisons for + * each individual column - for example with columns (c1, c2, ..., ck) we + * can estimate that number of comparions on ck is roughly + * + * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1)) + * + * Let k be a column number, Gk - number of groups defined by k columns, and Fk + * the cost of the comparison is + * + * N * sum( Fk * log(Gk) ) + * + * Note: We also consider column witdth, not just the comparator cost. + * * NOTE: some callers currently pass NIL for pathkeys because they * can't conveniently supply the sort keys. In this case, it will fallback to * simple comparison cost estimate. - * - * Estimation algorithm is based on ideas from course Algorithms, - * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper - * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild, - * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. - * - * In term of that papers, let N - number of tuples, Xi - number of tuples with - * key Ki, then estimation is: - * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) - * In our case all Xi are the same because now we don't have an estimation of - * group sizes, we have only estimation of number of groups. In this case, - * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation - * of multi-column sort we need separately compute each column, so, let k is a - * column number, Gk - number of groups defined by k columns: - * N * sum( Fk * log(Gk) ) - * Fk is a function costs (including width) for k columns. */ - static Cost compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, Cost comparison_cost, double tuples, double output_tuples, @@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, bool has_fake_var = false; int i = 0; Oid prev_datatype = InvalidOid; - Cost funcCost = 0.; + Cost funcCost = 0.0; List *cache_varinfos = NIL; /* fallback if pathkeys is unknown */ @@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * a total number of tuple comparisons of N log2 K; but the constant * factor is a bit higher than for quicksort. Tweak it so that the * cost curve is continuous at the crossover point. + * + * XXX I suppose the "quicksort factor" references to 1.5 at the end + * of this function, but I'm not sure. I suggest we introduce some simple + * constants for that, instead of magic values. */ output_tuples = (heapSort) ? 2.0 * output_tuples : tuples; per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples); @@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * - per column comparison function cost * - we try to compute needed number of comparison per column */ - foreach(lc, pathkeys) { PathKey *pathkey = (PathKey*)lfirst(lc); @@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * Don't use DEFAULT_NUM_DISTINCT because it used for only one * column while here we try to estimate number of groups over * set of columns. + * + * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to + * limit the calculated values, somehow? + * + * XXX What's the logic of the following formula? */ nGroups = ceil(2.0 + sqrt(tuples) * list_length(pathkeyExprs) / list_length(pathkeys)); @@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, { if (tuplesPerPrevGroup < output_tuples) /* comparing only inside output_tuples */ + /* XXX why not to use the same multiplier (1.5)? */ correctedNGroups = ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups)); else @@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups); /* - * We could skip all followed columns for cost estimation, because we + * We could skip all following columns for cost estimation, because we * believe that tuples are unique by set ot previous columns */ if (tuplesPerPrevGroup <= 1.0) diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index 7beb32488a..b092c3e055 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b) return 0; return 1; } + /* * Order tail of list of group pathkeys by uniqueness descendetly. It allows to * speedup sorting. Returns newly allocated lists, old ones stay untouched. * n_preordered defines a head of list which order should be prevented. + * + * XXX But we're not generating this only based on uniqueness (that's a bad + * term anyway, because we're using ndistinct estimates, not uniqueness). + * We're also using the comparator cost to calculate the expected sort cost, + * and optimize that. + * + * XXX This should explain how we generate the values - all permutations for + * up to 4 values, etc. */ void get_cheapest_group_keys_order(PlannerInfo *root, double nrows, @@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows, else { /* - * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to - * a new one which can then be cleaned safely if needed. + * Since v13 list_free() can clean list elements so for original list + * not to be modified it should be copied to a new one which can then + * be cleaned safely if needed. */ new_group_pathkeys = list_copy(*group_pathkeys); nToPermute = nFreeKeys; -- 2.29.2 --------------A955BFDEDE369DCD57A45C5A-- ^ permalink raw reply [nested|flat] 56+ messages in thread
* [PATCH 2/2] review @ 2021-03-09 20:09 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 56+ messages in thread From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw) --- src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++------- src/backend/optimizer/path/pathkeys.c | 14 ++++- 2 files changed, 69 insertions(+), 22 deletions(-) diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index 1639258aaf..f55a4f20e5 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm) * is_fake_var * Workaround for generate_append_tlist() which generates fake Vars with * varno == 0, that will cause a fail of estimate_num_group() call + * + * XXX Ummm, why would estimate_num_group fail with this? */ static bool is_fake_var(Expr *expr) @@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr) * compute_cpu_sort_cost * compute CPU cost of sort (i.e. in-memory) * + * The main thing we need to calculate to estimate sort CPU costs is the number + * of calls to the comparator functions. The difficulty is that for multi-column + * sorts there may be different data types involved (for some of which the calls + * may be much more expensive). Furthermore, the columns may have very different + * number of distinct values - the higher the number, the fewer comparisons will + * be needed for the following columns. + * + * The algoritm is incremental - we add pathkeys one by one, and at each step we + * estimate the number of necessary comparisons (based on the number of distinct + * groups in the current pathkey prefix and the new pathkey), and the comparison + * costs (which is data type specific). + * + * Estimation of the number of comparisons is based on ideas from two sources: + * + * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/] + * + * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild, + * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906] + * + * In term of that paper, let N - number of tuples, Xi - number of tuples with + * key Ki, then the estimate of number of comparisons is: + * + * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) + * + * In our case all Xi are the same because now we don't have any estimation of + * group sizes, we have only know the estimate of number of groups (distinct + * values). In that case, formula becomes: + * + * N * log(NumberOfGroups) + * + * For multi-column sorts we need to estimate the number of comparisons for + * each individual column - for example with columns (c1, c2, ..., ck) we + * can estimate that number of comparions on ck is roughly + * + * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1)) + * + * Let k be a column number, Gk - number of groups defined by k columns, and Fk + * the cost of the comparison is + * + * N * sum( Fk * log(Gk) ) + * + * Note: We also consider column witdth, not just the comparator cost. + * * NOTE: some callers currently pass NIL for pathkeys because they * can't conveniently supply the sort keys. In this case, it will fallback to * simple comparison cost estimate. - * - * Estimation algorithm is based on ideas from course Algorithms, - * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper - * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild, - * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. - * - * In term of that papers, let N - number of tuples, Xi - number of tuples with - * key Ki, then estimation is: - * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) - * In our case all Xi are the same because now we don't have an estimation of - * group sizes, we have only estimation of number of groups. In this case, - * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation - * of multi-column sort we need separately compute each column, so, let k is a - * column number, Gk - number of groups defined by k columns: - * N * sum( Fk * log(Gk) ) - * Fk is a function costs (including width) for k columns. */ - static Cost compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, Cost comparison_cost, double tuples, double output_tuples, @@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, bool has_fake_var = false; int i = 0; Oid prev_datatype = InvalidOid; - Cost funcCost = 0.; + Cost funcCost = 0.0; List *cache_varinfos = NIL; /* fallback if pathkeys is unknown */ @@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * a total number of tuple comparisons of N log2 K; but the constant * factor is a bit higher than for quicksort. Tweak it so that the * cost curve is continuous at the crossover point. + * + * XXX I suppose the "quicksort factor" references to 1.5 at the end + * of this function, but I'm not sure. I suggest we introduce some simple + * constants for that, instead of magic values. */ output_tuples = (heapSort) ? 2.0 * output_tuples : tuples; per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples); @@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * - per column comparison function cost * - we try to compute needed number of comparison per column */ - foreach(lc, pathkeys) { PathKey *pathkey = (PathKey*)lfirst(lc); @@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * Don't use DEFAULT_NUM_DISTINCT because it used for only one * column while here we try to estimate number of groups over * set of columns. + * + * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to + * limit the calculated values, somehow? + * + * XXX What's the logic of the following formula? */ nGroups = ceil(2.0 + sqrt(tuples) * list_length(pathkeyExprs) / list_length(pathkeys)); @@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, { if (tuplesPerPrevGroup < output_tuples) /* comparing only inside output_tuples */ + /* XXX why not to use the same multiplier (1.5)? */ correctedNGroups = ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups)); else @@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups); /* - * We could skip all followed columns for cost estimation, because we + * We could skip all following columns for cost estimation, because we * believe that tuples are unique by set ot previous columns */ if (tuplesPerPrevGroup <= 1.0) diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index 7beb32488a..b092c3e055 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b) return 0; return 1; } + /* * Order tail of list of group pathkeys by uniqueness descendetly. It allows to * speedup sorting. Returns newly allocated lists, old ones stay untouched. * n_preordered defines a head of list which order should be prevented. + * + * XXX But we're not generating this only based on uniqueness (that's a bad + * term anyway, because we're using ndistinct estimates, not uniqueness). + * We're also using the comparator cost to calculate the expected sort cost, + * and optimize that. + * + * XXX This should explain how we generate the values - all permutations for + * up to 4 values, etc. */ void get_cheapest_group_keys_order(PlannerInfo *root, double nrows, @@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows, else { /* - * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to - * a new one which can then be cleaned safely if needed. + * Since v13 list_free() can clean list elements so for original list + * not to be modified it should be copied to a new one which can then + * be cleaned safely if needed. */ new_group_pathkeys = list_copy(*group_pathkeys); nToPermute = nFreeKeys; -- 2.29.2 --------------A955BFDEDE369DCD57A45C5A-- ^ permalink raw reply [nested|flat] 56+ messages in thread
* [PATCH 2/2] review @ 2021-03-09 20:09 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 56+ messages in thread From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw) --- src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++------- src/backend/optimizer/path/pathkeys.c | 14 ++++- 2 files changed, 69 insertions(+), 22 deletions(-) diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index 1639258aaf..f55a4f20e5 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm) * is_fake_var * Workaround for generate_append_tlist() which generates fake Vars with * varno == 0, that will cause a fail of estimate_num_group() call + * + * XXX Ummm, why would estimate_num_group fail with this? */ static bool is_fake_var(Expr *expr) @@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr) * compute_cpu_sort_cost * compute CPU cost of sort (i.e. in-memory) * + * The main thing we need to calculate to estimate sort CPU costs is the number + * of calls to the comparator functions. The difficulty is that for multi-column + * sorts there may be different data types involved (for some of which the calls + * may be much more expensive). Furthermore, the columns may have very different + * number of distinct values - the higher the number, the fewer comparisons will + * be needed for the following columns. + * + * The algoritm is incremental - we add pathkeys one by one, and at each step we + * estimate the number of necessary comparisons (based on the number of distinct + * groups in the current pathkey prefix and the new pathkey), and the comparison + * costs (which is data type specific). + * + * Estimation of the number of comparisons is based on ideas from two sources: + * + * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/] + * + * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild, + * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906] + * + * In term of that paper, let N - number of tuples, Xi - number of tuples with + * key Ki, then the estimate of number of comparisons is: + * + * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) + * + * In our case all Xi are the same because now we don't have any estimation of + * group sizes, we have only know the estimate of number of groups (distinct + * values). In that case, formula becomes: + * + * N * log(NumberOfGroups) + * + * For multi-column sorts we need to estimate the number of comparisons for + * each individual column - for example with columns (c1, c2, ..., ck) we + * can estimate that number of comparions on ck is roughly + * + * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1)) + * + * Let k be a column number, Gk - number of groups defined by k columns, and Fk + * the cost of the comparison is + * + * N * sum( Fk * log(Gk) ) + * + * Note: We also consider column witdth, not just the comparator cost. + * * NOTE: some callers currently pass NIL for pathkeys because they * can't conveniently supply the sort keys. In this case, it will fallback to * simple comparison cost estimate. - * - * Estimation algorithm is based on ideas from course Algorithms, - * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper - * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild, - * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. - * - * In term of that papers, let N - number of tuples, Xi - number of tuples with - * key Ki, then estimation is: - * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) - * In our case all Xi are the same because now we don't have an estimation of - * group sizes, we have only estimation of number of groups. In this case, - * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation - * of multi-column sort we need separately compute each column, so, let k is a - * column number, Gk - number of groups defined by k columns: - * N * sum( Fk * log(Gk) ) - * Fk is a function costs (including width) for k columns. */ - static Cost compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, Cost comparison_cost, double tuples, double output_tuples, @@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, bool has_fake_var = false; int i = 0; Oid prev_datatype = InvalidOid; - Cost funcCost = 0.; + Cost funcCost = 0.0; List *cache_varinfos = NIL; /* fallback if pathkeys is unknown */ @@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * a total number of tuple comparisons of N log2 K; but the constant * factor is a bit higher than for quicksort. Tweak it so that the * cost curve is continuous at the crossover point. + * + * XXX I suppose the "quicksort factor" references to 1.5 at the end + * of this function, but I'm not sure. I suggest we introduce some simple + * constants for that, instead of magic values. */ output_tuples = (heapSort) ? 2.0 * output_tuples : tuples; per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples); @@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * - per column comparison function cost * - we try to compute needed number of comparison per column */ - foreach(lc, pathkeys) { PathKey *pathkey = (PathKey*)lfirst(lc); @@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * Don't use DEFAULT_NUM_DISTINCT because it used for only one * column while here we try to estimate number of groups over * set of columns. + * + * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to + * limit the calculated values, somehow? + * + * XXX What's the logic of the following formula? */ nGroups = ceil(2.0 + sqrt(tuples) * list_length(pathkeyExprs) / list_length(pathkeys)); @@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, { if (tuplesPerPrevGroup < output_tuples) /* comparing only inside output_tuples */ + /* XXX why not to use the same multiplier (1.5)? */ correctedNGroups = ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups)); else @@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups); /* - * We could skip all followed columns for cost estimation, because we + * We could skip all following columns for cost estimation, because we * believe that tuples are unique by set ot previous columns */ if (tuplesPerPrevGroup <= 1.0) diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index 7beb32488a..b092c3e055 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b) return 0; return 1; } + /* * Order tail of list of group pathkeys by uniqueness descendetly. It allows to * speedup sorting. Returns newly allocated lists, old ones stay untouched. * n_preordered defines a head of list which order should be prevented. + * + * XXX But we're not generating this only based on uniqueness (that's a bad + * term anyway, because we're using ndistinct estimates, not uniqueness). + * We're also using the comparator cost to calculate the expected sort cost, + * and optimize that. + * + * XXX This should explain how we generate the values - all permutations for + * up to 4 values, etc. */ void get_cheapest_group_keys_order(PlannerInfo *root, double nrows, @@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows, else { /* - * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to - * a new one which can then be cleaned safely if needed. + * Since v13 list_free() can clean list elements so for original list + * not to be modified it should be copied to a new one which can then + * be cleaned safely if needed. */ new_group_pathkeys = list_copy(*group_pathkeys); nToPermute = nFreeKeys; -- 2.29.2 --------------A955BFDEDE369DCD57A45C5A-- ^ permalink raw reply [nested|flat] 56+ messages in thread
* [PATCH 2/2] review @ 2021-03-09 20:09 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 56+ messages in thread From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw) --- src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++------- src/backend/optimizer/path/pathkeys.c | 14 ++++- 2 files changed, 69 insertions(+), 22 deletions(-) diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index 1639258aaf..f55a4f20e5 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm) * is_fake_var * Workaround for generate_append_tlist() which generates fake Vars with * varno == 0, that will cause a fail of estimate_num_group() call + * + * XXX Ummm, why would estimate_num_group fail with this? */ static bool is_fake_var(Expr *expr) @@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr) * compute_cpu_sort_cost * compute CPU cost of sort (i.e. in-memory) * + * The main thing we need to calculate to estimate sort CPU costs is the number + * of calls to the comparator functions. The difficulty is that for multi-column + * sorts there may be different data types involved (for some of which the calls + * may be much more expensive). Furthermore, the columns may have very different + * number of distinct values - the higher the number, the fewer comparisons will + * be needed for the following columns. + * + * The algoritm is incremental - we add pathkeys one by one, and at each step we + * estimate the number of necessary comparisons (based on the number of distinct + * groups in the current pathkey prefix and the new pathkey), and the comparison + * costs (which is data type specific). + * + * Estimation of the number of comparisons is based on ideas from two sources: + * + * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/] + * + * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild, + * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906] + * + * In term of that paper, let N - number of tuples, Xi - number of tuples with + * key Ki, then the estimate of number of comparisons is: + * + * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) + * + * In our case all Xi are the same because now we don't have any estimation of + * group sizes, we have only know the estimate of number of groups (distinct + * values). In that case, formula becomes: + * + * N * log(NumberOfGroups) + * + * For multi-column sorts we need to estimate the number of comparisons for + * each individual column - for example with columns (c1, c2, ..., ck) we + * can estimate that number of comparions on ck is roughly + * + * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1)) + * + * Let k be a column number, Gk - number of groups defined by k columns, and Fk + * the cost of the comparison is + * + * N * sum( Fk * log(Gk) ) + * + * Note: We also consider column witdth, not just the comparator cost. + * * NOTE: some callers currently pass NIL for pathkeys because they * can't conveniently supply the sort keys. In this case, it will fallback to * simple comparison cost estimate. - * - * Estimation algorithm is based on ideas from course Algorithms, - * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper - * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild, - * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. - * - * In term of that papers, let N - number of tuples, Xi - number of tuples with - * key Ki, then estimation is: - * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) - * In our case all Xi are the same because now we don't have an estimation of - * group sizes, we have only estimation of number of groups. In this case, - * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation - * of multi-column sort we need separately compute each column, so, let k is a - * column number, Gk - number of groups defined by k columns: - * N * sum( Fk * log(Gk) ) - * Fk is a function costs (including width) for k columns. */ - static Cost compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, Cost comparison_cost, double tuples, double output_tuples, @@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, bool has_fake_var = false; int i = 0; Oid prev_datatype = InvalidOid; - Cost funcCost = 0.; + Cost funcCost = 0.0; List *cache_varinfos = NIL; /* fallback if pathkeys is unknown */ @@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * a total number of tuple comparisons of N log2 K; but the constant * factor is a bit higher than for quicksort. Tweak it so that the * cost curve is continuous at the crossover point. + * + * XXX I suppose the "quicksort factor" references to 1.5 at the end + * of this function, but I'm not sure. I suggest we introduce some simple + * constants for that, instead of magic values. */ output_tuples = (heapSort) ? 2.0 * output_tuples : tuples; per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples); @@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * - per column comparison function cost * - we try to compute needed number of comparison per column */ - foreach(lc, pathkeys) { PathKey *pathkey = (PathKey*)lfirst(lc); @@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * Don't use DEFAULT_NUM_DISTINCT because it used for only one * column while here we try to estimate number of groups over * set of columns. + * + * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to + * limit the calculated values, somehow? + * + * XXX What's the logic of the following formula? */ nGroups = ceil(2.0 + sqrt(tuples) * list_length(pathkeyExprs) / list_length(pathkeys)); @@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, { if (tuplesPerPrevGroup < output_tuples) /* comparing only inside output_tuples */ + /* XXX why not to use the same multiplier (1.5)? */ correctedNGroups = ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups)); else @@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups); /* - * We could skip all followed columns for cost estimation, because we + * We could skip all following columns for cost estimation, because we * believe that tuples are unique by set ot previous columns */ if (tuplesPerPrevGroup <= 1.0) diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index 7beb32488a..b092c3e055 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b) return 0; return 1; } + /* * Order tail of list of group pathkeys by uniqueness descendetly. It allows to * speedup sorting. Returns newly allocated lists, old ones stay untouched. * n_preordered defines a head of list which order should be prevented. + * + * XXX But we're not generating this only based on uniqueness (that's a bad + * term anyway, because we're using ndistinct estimates, not uniqueness). + * We're also using the comparator cost to calculate the expected sort cost, + * and optimize that. + * + * XXX This should explain how we generate the values - all permutations for + * up to 4 values, etc. */ void get_cheapest_group_keys_order(PlannerInfo *root, double nrows, @@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows, else { /* - * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to - * a new one which can then be cleaned safely if needed. + * Since v13 list_free() can clean list elements so for original list + * not to be modified it should be copied to a new one which can then + * be cleaned safely if needed. */ new_group_pathkeys = list_copy(*group_pathkeys); nToPermute = nFreeKeys; -- 2.29.2 --------------A955BFDEDE369DCD57A45C5A-- ^ permalink raw reply [nested|flat] 56+ messages in thread
* [PATCH 2/2] review @ 2021-03-09 20:09 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 56+ messages in thread From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw) --- src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++------- src/backend/optimizer/path/pathkeys.c | 14 ++++- 2 files changed, 69 insertions(+), 22 deletions(-) diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index 1639258aaf..f55a4f20e5 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm) * is_fake_var * Workaround for generate_append_tlist() which generates fake Vars with * varno == 0, that will cause a fail of estimate_num_group() call + * + * XXX Ummm, why would estimate_num_group fail with this? */ static bool is_fake_var(Expr *expr) @@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr) * compute_cpu_sort_cost * compute CPU cost of sort (i.e. in-memory) * + * The main thing we need to calculate to estimate sort CPU costs is the number + * of calls to the comparator functions. The difficulty is that for multi-column + * sorts there may be different data types involved (for some of which the calls + * may be much more expensive). Furthermore, the columns may have very different + * number of distinct values - the higher the number, the fewer comparisons will + * be needed for the following columns. + * + * The algoritm is incremental - we add pathkeys one by one, and at each step we + * estimate the number of necessary comparisons (based on the number of distinct + * groups in the current pathkey prefix and the new pathkey), and the comparison + * costs (which is data type specific). + * + * Estimation of the number of comparisons is based on ideas from two sources: + * + * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/] + * + * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild, + * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906] + * + * In term of that paper, let N - number of tuples, Xi - number of tuples with + * key Ki, then the estimate of number of comparisons is: + * + * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) + * + * In our case all Xi are the same because now we don't have any estimation of + * group sizes, we have only know the estimate of number of groups (distinct + * values). In that case, formula becomes: + * + * N * log(NumberOfGroups) + * + * For multi-column sorts we need to estimate the number of comparisons for + * each individual column - for example with columns (c1, c2, ..., ck) we + * can estimate that number of comparions on ck is roughly + * + * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1)) + * + * Let k be a column number, Gk - number of groups defined by k columns, and Fk + * the cost of the comparison is + * + * N * sum( Fk * log(Gk) ) + * + * Note: We also consider column witdth, not just the comparator cost. + * * NOTE: some callers currently pass NIL for pathkeys because they * can't conveniently supply the sort keys. In this case, it will fallback to * simple comparison cost estimate. - * - * Estimation algorithm is based on ideas from course Algorithms, - * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper - * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild, - * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. - * - * In term of that papers, let N - number of tuples, Xi - number of tuples with - * key Ki, then estimation is: - * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) - * In our case all Xi are the same because now we don't have an estimation of - * group sizes, we have only estimation of number of groups. In this case, - * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation - * of multi-column sort we need separately compute each column, so, let k is a - * column number, Gk - number of groups defined by k columns: - * N * sum( Fk * log(Gk) ) - * Fk is a function costs (including width) for k columns. */ - static Cost compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, Cost comparison_cost, double tuples, double output_tuples, @@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, bool has_fake_var = false; int i = 0; Oid prev_datatype = InvalidOid; - Cost funcCost = 0.; + Cost funcCost = 0.0; List *cache_varinfos = NIL; /* fallback if pathkeys is unknown */ @@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * a total number of tuple comparisons of N log2 K; but the constant * factor is a bit higher than for quicksort. Tweak it so that the * cost curve is continuous at the crossover point. + * + * XXX I suppose the "quicksort factor" references to 1.5 at the end + * of this function, but I'm not sure. I suggest we introduce some simple + * constants for that, instead of magic values. */ output_tuples = (heapSort) ? 2.0 * output_tuples : tuples; per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples); @@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * - per column comparison function cost * - we try to compute needed number of comparison per column */ - foreach(lc, pathkeys) { PathKey *pathkey = (PathKey*)lfirst(lc); @@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * Don't use DEFAULT_NUM_DISTINCT because it used for only one * column while here we try to estimate number of groups over * set of columns. + * + * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to + * limit the calculated values, somehow? + * + * XXX What's the logic of the following formula? */ nGroups = ceil(2.0 + sqrt(tuples) * list_length(pathkeyExprs) / list_length(pathkeys)); @@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, { if (tuplesPerPrevGroup < output_tuples) /* comparing only inside output_tuples */ + /* XXX why not to use the same multiplier (1.5)? */ correctedNGroups = ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups)); else @@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups); /* - * We could skip all followed columns for cost estimation, because we + * We could skip all following columns for cost estimation, because we * believe that tuples are unique by set ot previous columns */ if (tuplesPerPrevGroup <= 1.0) diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index 7beb32488a..b092c3e055 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b) return 0; return 1; } + /* * Order tail of list of group pathkeys by uniqueness descendetly. It allows to * speedup sorting. Returns newly allocated lists, old ones stay untouched. * n_preordered defines a head of list which order should be prevented. + * + * XXX But we're not generating this only based on uniqueness (that's a bad + * term anyway, because we're using ndistinct estimates, not uniqueness). + * We're also using the comparator cost to calculate the expected sort cost, + * and optimize that. + * + * XXX This should explain how we generate the values - all permutations for + * up to 4 values, etc. */ void get_cheapest_group_keys_order(PlannerInfo *root, double nrows, @@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows, else { /* - * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to - * a new one which can then be cleaned safely if needed. + * Since v13 list_free() can clean list elements so for original list + * not to be modified it should be copied to a new one which can then + * be cleaned safely if needed. */ new_group_pathkeys = list_copy(*group_pathkeys); nToPermute = nFreeKeys; -- 2.29.2 --------------A955BFDEDE369DCD57A45C5A-- ^ permalink raw reply [nested|flat] 56+ messages in thread
* [PATCH 2/2] review @ 2021-03-09 20:09 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 56+ messages in thread From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw) --- src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++------- src/backend/optimizer/path/pathkeys.c | 14 ++++- 2 files changed, 69 insertions(+), 22 deletions(-) diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index 1639258aaf..f55a4f20e5 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm) * is_fake_var * Workaround for generate_append_tlist() which generates fake Vars with * varno == 0, that will cause a fail of estimate_num_group() call + * + * XXX Ummm, why would estimate_num_group fail with this? */ static bool is_fake_var(Expr *expr) @@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr) * compute_cpu_sort_cost * compute CPU cost of sort (i.e. in-memory) * + * The main thing we need to calculate to estimate sort CPU costs is the number + * of calls to the comparator functions. The difficulty is that for multi-column + * sorts there may be different data types involved (for some of which the calls + * may be much more expensive). Furthermore, the columns may have very different + * number of distinct values - the higher the number, the fewer comparisons will + * be needed for the following columns. + * + * The algoritm is incremental - we add pathkeys one by one, and at each step we + * estimate the number of necessary comparisons (based on the number of distinct + * groups in the current pathkey prefix and the new pathkey), and the comparison + * costs (which is data type specific). + * + * Estimation of the number of comparisons is based on ideas from two sources: + * + * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/] + * + * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild, + * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906] + * + * In term of that paper, let N - number of tuples, Xi - number of tuples with + * key Ki, then the estimate of number of comparisons is: + * + * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) + * + * In our case all Xi are the same because now we don't have any estimation of + * group sizes, we have only know the estimate of number of groups (distinct + * values). In that case, formula becomes: + * + * N * log(NumberOfGroups) + * + * For multi-column sorts we need to estimate the number of comparisons for + * each individual column - for example with columns (c1, c2, ..., ck) we + * can estimate that number of comparions on ck is roughly + * + * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1)) + * + * Let k be a column number, Gk - number of groups defined by k columns, and Fk + * the cost of the comparison is + * + * N * sum( Fk * log(Gk) ) + * + * Note: We also consider column witdth, not just the comparator cost. + * * NOTE: some callers currently pass NIL for pathkeys because they * can't conveniently supply the sort keys. In this case, it will fallback to * simple comparison cost estimate. - * - * Estimation algorithm is based on ideas from course Algorithms, - * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper - * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild, - * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. - * - * In term of that papers, let N - number of tuples, Xi - number of tuples with - * key Ki, then estimation is: - * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) - * In our case all Xi are the same because now we don't have an estimation of - * group sizes, we have only estimation of number of groups. In this case, - * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation - * of multi-column sort we need separately compute each column, so, let k is a - * column number, Gk - number of groups defined by k columns: - * N * sum( Fk * log(Gk) ) - * Fk is a function costs (including width) for k columns. */ - static Cost compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, Cost comparison_cost, double tuples, double output_tuples, @@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, bool has_fake_var = false; int i = 0; Oid prev_datatype = InvalidOid; - Cost funcCost = 0.; + Cost funcCost = 0.0; List *cache_varinfos = NIL; /* fallback if pathkeys is unknown */ @@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * a total number of tuple comparisons of N log2 K; but the constant * factor is a bit higher than for quicksort. Tweak it so that the * cost curve is continuous at the crossover point. + * + * XXX I suppose the "quicksort factor" references to 1.5 at the end + * of this function, but I'm not sure. I suggest we introduce some simple + * constants for that, instead of magic values. */ output_tuples = (heapSort) ? 2.0 * output_tuples : tuples; per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples); @@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * - per column comparison function cost * - we try to compute needed number of comparison per column */ - foreach(lc, pathkeys) { PathKey *pathkey = (PathKey*)lfirst(lc); @@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * Don't use DEFAULT_NUM_DISTINCT because it used for only one * column while here we try to estimate number of groups over * set of columns. + * + * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to + * limit the calculated values, somehow? + * + * XXX What's the logic of the following formula? */ nGroups = ceil(2.0 + sqrt(tuples) * list_length(pathkeyExprs) / list_length(pathkeys)); @@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, { if (tuplesPerPrevGroup < output_tuples) /* comparing only inside output_tuples */ + /* XXX why not to use the same multiplier (1.5)? */ correctedNGroups = ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups)); else @@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups); /* - * We could skip all followed columns for cost estimation, because we + * We could skip all following columns for cost estimation, because we * believe that tuples are unique by set ot previous columns */ if (tuplesPerPrevGroup <= 1.0) diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index 7beb32488a..b092c3e055 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b) return 0; return 1; } + /* * Order tail of list of group pathkeys by uniqueness descendetly. It allows to * speedup sorting. Returns newly allocated lists, old ones stay untouched. * n_preordered defines a head of list which order should be prevented. + * + * XXX But we're not generating this only based on uniqueness (that's a bad + * term anyway, because we're using ndistinct estimates, not uniqueness). + * We're also using the comparator cost to calculate the expected sort cost, + * and optimize that. + * + * XXX This should explain how we generate the values - all permutations for + * up to 4 values, etc. */ void get_cheapest_group_keys_order(PlannerInfo *root, double nrows, @@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows, else { /* - * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to - * a new one which can then be cleaned safely if needed. + * Since v13 list_free() can clean list elements so for original list + * not to be modified it should be copied to a new one which can then + * be cleaned safely if needed. */ new_group_pathkeys = list_copy(*group_pathkeys); nToPermute = nFreeKeys; -- 2.29.2 --------------A955BFDEDE369DCD57A45C5A-- ^ permalink raw reply [nested|flat] 56+ messages in thread
* [PATCH 2/2] review @ 2021-03-09 20:09 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 56+ messages in thread From: Tomas Vondra @ 2021-03-09 20:09 UTC (permalink / raw) --- src/backend/optimizer/path/costsize.c | 77 ++++++++++++++++++++------- src/backend/optimizer/path/pathkeys.c | 14 ++++- 2 files changed, 69 insertions(+), 22 deletions(-) diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index 1639258aaf..f55a4f20e5 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -1755,6 +1755,8 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm) * is_fake_var * Workaround for generate_append_tlist() which generates fake Vars with * varno == 0, that will cause a fail of estimate_num_group() call + * + * XXX Ummm, why would estimate_num_group fail with this? */ static bool is_fake_var(Expr *expr) @@ -1828,27 +1830,53 @@ get_width_cost_multiplier(PlannerInfo *root, Expr *expr) * compute_cpu_sort_cost * compute CPU cost of sort (i.e. in-memory) * + * The main thing we need to calculate to estimate sort CPU costs is the number + * of calls to the comparator functions. The difficulty is that for multi-column + * sorts there may be different data types involved (for some of which the calls + * may be much more expensive). Furthermore, the columns may have very different + * number of distinct values - the higher the number, the fewer comparisons will + * be needed for the following columns. + * + * The algoritm is incremental - we add pathkeys one by one, and at each step we + * estimate the number of necessary comparisons (based on the number of distinct + * groups in the current pathkey prefix and the new pathkey), and the comparison + * costs (which is data type specific). + * + * Estimation of the number of comparisons is based on ideas from two sources: + * + * 1) "Algorithms" (course), Robert Sedgewick, Kevin Wayne [https://algs4.cs.princeton.edu/home/] + * + * 2) "Quicksort Is Optimal For Many Equal Keys" (paper), Sebastian Wild, + * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. [https://arxiv.org/abs/1608.04906] + * + * In term of that paper, let N - number of tuples, Xi - number of tuples with + * key Ki, then the estimate of number of comparisons is: + * + * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) + * + * In our case all Xi are the same because now we don't have any estimation of + * group sizes, we have only know the estimate of number of groups (distinct + * values). In that case, formula becomes: + * + * N * log(NumberOfGroups) + * + * For multi-column sorts we need to estimate the number of comparisons for + * each individual column - for example with columns (c1, c2, ..., ck) we + * can estimate that number of comparions on ck is roughly + * + * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1)) + * + * Let k be a column number, Gk - number of groups defined by k columns, and Fk + * the cost of the comparison is + * + * N * sum( Fk * log(Gk) ) + * + * Note: We also consider column witdth, not just the comparator cost. + * * NOTE: some callers currently pass NIL for pathkeys because they * can't conveniently supply the sort keys. In this case, it will fallback to * simple comparison cost estimate. - * - * Estimation algorithm is based on ideas from course Algorithms, - * Robert Sedgewick, Kevin Wayne, https://algs4.cs.princeton.edu/home/ and paper - * "Quicksort Is Optimal For Many Equal Keys", Sebastian Wild, - * arXiv:1608.04906v4 [cs.DS] 1 Nov 2017. - * - * In term of that papers, let N - number of tuples, Xi - number of tuples with - * key Ki, then estimation is: - * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi)) - * In our case all Xi are the same because now we don't have an estimation of - * group sizes, we have only estimation of number of groups. In this case, - * formula becomes: N * log(NumberOfGroups). Next, to support correct estimation - * of multi-column sort we need separately compute each column, so, let k is a - * column number, Gk - number of groups defined by k columns: - * N * sum( Fk * log(Gk) ) - * Fk is a function costs (including width) for k columns. */ - static Cost compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, Cost comparison_cost, double tuples, double output_tuples, @@ -1862,7 +1890,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, bool has_fake_var = false; int i = 0; Oid prev_datatype = InvalidOid; - Cost funcCost = 0.; + Cost funcCost = 0.0; List *cache_varinfos = NIL; /* fallback if pathkeys is unknown */ @@ -1873,6 +1901,10 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * a total number of tuple comparisons of N log2 K; but the constant * factor is a bit higher than for quicksort. Tweak it so that the * cost curve is continuous at the crossover point. + * + * XXX I suppose the "quicksort factor" references to 1.5 at the end + * of this function, but I'm not sure. I suggest we introduce some simple + * constants for that, instead of magic values. */ output_tuples = (heapSort) ? 2.0 * output_tuples : tuples; per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples); @@ -1888,7 +1920,6 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * - per column comparison function cost * - we try to compute needed number of comparison per column */ - foreach(lc, pathkeys) { PathKey *pathkey = (PathKey*)lfirst(lc); @@ -1952,6 +1983,11 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, * Don't use DEFAULT_NUM_DISTINCT because it used for only one * column while here we try to estimate number of groups over * set of columns. + * + * XXX Perhaps this should use DEFAULT_NUM_DISTINCT at least to + * limit the calculated values, somehow? + * + * XXX What's the logic of the following formula? */ nGroups = ceil(2.0 + sqrt(tuples) * list_length(pathkeyExprs) / list_length(pathkeys)); @@ -1968,6 +2004,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, { if (tuplesPerPrevGroup < output_tuples) /* comparing only inside output_tuples */ + /* XXX why not to use the same multiplier (1.5)? */ correctedNGroups = ceil(2.0 * output_tuples / (tuplesPerPrevGroup / nGroups)); else @@ -1993,7 +2030,7 @@ compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys, tuplesPerPrevGroup = ceil(1.5 * tuplesPerPrevGroup / nGroups); /* - * We could skip all followed columns for cost estimation, because we + * We could skip all following columns for cost estimation, because we * believe that tuples are unique by set ot previous columns */ if (tuplesPerPrevGroup <= 1.0) diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index 7beb32488a..b092c3e055 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -515,10 +515,19 @@ pathkey_sort_cost_comparator(const void *_a, const void *_b) return 0; return 1; } + /* * Order tail of list of group pathkeys by uniqueness descendetly. It allows to * speedup sorting. Returns newly allocated lists, old ones stay untouched. * n_preordered defines a head of list which order should be prevented. + * + * XXX But we're not generating this only based on uniqueness (that's a bad + * term anyway, because we're using ndistinct estimates, not uniqueness). + * We're also using the comparator cost to calculate the expected sort cost, + * and optimize that. + * + * XXX This should explain how we generate the values - all permutations for + * up to 4 values, etc. */ void get_cheapest_group_keys_order(PlannerInfo *root, double nrows, @@ -597,8 +606,9 @@ get_cheapest_group_keys_order(PlannerInfo *root, double nrows, else { /* - * Since v13 list_free() can clean list elements so for original list not to be modified it should be copied to - * a new one which can then be cleaned safely if needed. + * Since v13 list_free() can clean list elements so for original list + * not to be modified it should be copied to a new one which can then + * be cleaned safely if needed. */ new_group_pathkeys = list_copy(*group_pathkeys); nToPermute = nFreeKeys; -- 2.29.2 --------------A955BFDEDE369DCD57A45C5A-- ^ permalink raw reply [nested|flat] 56+ messages in thread
* [PATCH 02/10] review @ 2021-03-16 16:29 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 56+ messages in thread From: Tomas Vondra @ 2021-03-16 16:29 UTC (permalink / raw) --- src/backend/optimizer/path/allpaths.c | 2 ++ src/backend/optimizer/plan/initsplan.c | 9 ++++++++- src/backend/optimizer/util/plancat.c | 3 +++ 3 files changed, 13 insertions(+), 1 deletion(-) diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index 37b4223adb..fc1a3a68a2 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -1065,8 +1065,10 @@ set_append_rel_size(PlannerInfo *root, RelOptInfo *rel, /* Whole row is not null, so must be same for child */ childrel->notnullattrs = bms_add_member(childrel->notnullattrs, attno - FirstLowInvalidHeapAttributeNumber); + /* XXX shouldn't this be a continue, instead of a break? */ break; } + /* XXX isn't this missing 'else'? */ if (attno < 0 ) /* no need to translate system column */ child_attno = attno; diff --git a/src/backend/optimizer/plan/initsplan.c b/src/backend/optimizer/plan/initsplan.c index d27167dc76..4ef876cf7b 100644 --- a/src/backend/optimizer/plan/initsplan.c +++ b/src/backend/optimizer/plan/initsplan.c @@ -829,7 +829,14 @@ deconstruct_recurse(PlannerInfo *root, Node *jtnode, bool below_outer_join, { Node *qual = (Node *) lfirst(l); - /* Set the not null info now */ + /* Set the not null info now + * + * XXX Why now? Why is this the right place to do this? Does it need + * to happen before distribute_qual_to_rels, for example? + * + * XXX Not clear to me why this looks at non-nullable vars? Shouldn't + * we already have the bitmap built from atnums (from get_relation_info)? + */ ListCell *lc; List *non_nullable_vars = find_nonnullable_vars(qual); foreach(lc, non_nullable_vars) diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index eebabcfccf..cd703e41ba 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -481,6 +481,9 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, if (inhparent && relation->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) set_relation_partition_info(root, rel, relation); + /* + * Build information about which attributes are marked as NOT NULL. + */ Assert(rel->notnullattrs == NULL); for(i = 0; i < relation->rd_att->natts; i++) { -- 2.30.2 --------------F495F6B18672582269F00FF9 Content-Type: text/x-patch; charset=UTF-8; name="0003-Introduce-UniqueKey-attributes-on-RelOptInf-20210317.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename*0="0003-Introduce-UniqueKey-attributes-on-RelOptInf-20210317.pa"; filename*1="tch" ^ permalink raw reply [nested|flat] 56+ messages in thread
* [PATCH 02/10] review @ 2021-03-16 16:29 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 56+ messages in thread From: Tomas Vondra @ 2021-03-16 16:29 UTC (permalink / raw) --- src/backend/optimizer/path/allpaths.c | 2 ++ src/backend/optimizer/plan/initsplan.c | 9 ++++++++- src/backend/optimizer/util/plancat.c | 3 +++ 3 files changed, 13 insertions(+), 1 deletion(-) diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index 37b4223adb..fc1a3a68a2 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -1065,8 +1065,10 @@ set_append_rel_size(PlannerInfo *root, RelOptInfo *rel, /* Whole row is not null, so must be same for child */ childrel->notnullattrs = bms_add_member(childrel->notnullattrs, attno - FirstLowInvalidHeapAttributeNumber); + /* XXX shouldn't this be a continue, instead of a break? */ break; } + /* XXX isn't this missing 'else'? */ if (attno < 0 ) /* no need to translate system column */ child_attno = attno; diff --git a/src/backend/optimizer/plan/initsplan.c b/src/backend/optimizer/plan/initsplan.c index d27167dc76..4ef876cf7b 100644 --- a/src/backend/optimizer/plan/initsplan.c +++ b/src/backend/optimizer/plan/initsplan.c @@ -829,7 +829,14 @@ deconstruct_recurse(PlannerInfo *root, Node *jtnode, bool below_outer_join, { Node *qual = (Node *) lfirst(l); - /* Set the not null info now */ + /* Set the not null info now + * + * XXX Why now? Why is this the right place to do this? Does it need + * to happen before distribute_qual_to_rels, for example? + * + * XXX Not clear to me why this looks at non-nullable vars? Shouldn't + * we already have the bitmap built from atnums (from get_relation_info)? + */ ListCell *lc; List *non_nullable_vars = find_nonnullable_vars(qual); foreach(lc, non_nullable_vars) diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index eebabcfccf..cd703e41ba 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -481,6 +481,9 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, if (inhparent && relation->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) set_relation_partition_info(root, rel, relation); + /* + * Build information about which attributes are marked as NOT NULL. + */ Assert(rel->notnullattrs == NULL); for(i = 0; i < relation->rd_att->natts; i++) { -- 2.30.2 --------------F495F6B18672582269F00FF9 Content-Type: text/x-patch; charset=UTF-8; name="0003-Introduce-UniqueKey-attributes-on-RelOptInf-20210317.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename*0="0003-Introduce-UniqueKey-attributes-on-RelOptInf-20210317.pa"; filename*1="tch" ^ permalink raw reply [nested|flat] 56+ messages in thread
* [PATCH 04/10] review @ 2021-03-16 18:26 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 56+ messages in thread From: Tomas Vondra @ 2021-03-16 18:26 UTC (permalink / raw) --- src/backend/nodes/copyfuncs.c | 4 + src/backend/nodes/makefuncs.c | 1 - src/backend/nodes/outfuncs.c | 1 + src/backend/nodes/readfuncs.c | 2 + src/backend/optimizer/path/README.uniquekey | 285 +++--- src/backend/optimizer/path/allpaths.c | 10 +- src/backend/optimizer/path/joinrels.c | 7 + src/backend/optimizer/path/uniquekeys.c | 906 ++++++++++++++------ src/backend/optimizer/plan/planner.c | 10 + src/backend/optimizer/prep/prepunion.c | 1 + src/backend/optimizer/util/inherit.c | 1 + 11 files changed, 863 insertions(+), 365 deletions(-) diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 75c1c5e824..9d832ddc03 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2296,6 +2296,9 @@ _copyPathKey(const PathKey *from) return newnode; } +/* + * _copyUniqueKey + */ static UniqueKey * _copyUniqueKey(const UniqueKey *from) { @@ -2306,6 +2309,7 @@ _copyUniqueKey(const UniqueKey *from) return newnode; } + /* * _copyRestrictInfo */ diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 40415d0f5b..e156c9cdf8 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -816,7 +816,6 @@ makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols) return v; } - /* * makeUniqueKey */ diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 44154cde6a..13905e6037 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2460,6 +2460,7 @@ static void _outUniqueKey(StringInfo str, const UniqueKey *node) { WRITE_NODE_TYPE("UNIQUEKEY"); + WRITE_NODE_FIELD(exprs); WRITE_BOOL_FIELD(multi_nullvals); } diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index b3e212bf1c..8830c8df99 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -496,8 +496,10 @@ static UniqueKey * _readUniqueKey(void) { READ_LOCALS(UniqueKey); + READ_NODE_FIELD(exprs); READ_BOOL_FIELD(multi_nullvals); + READ_DONE(); } diff --git a/src/backend/optimizer/path/README.uniquekey b/src/backend/optimizer/path/README.uniquekey index 5eac761995..31cdb5ed65 100644 --- a/src/backend/optimizer/path/README.uniquekey +++ b/src/backend/optimizer/path/README.uniquekey @@ -1,131 +1,208 @@ -1. What is UniqueKey? -We can think UniqueKey is a set of exprs for a RelOptInfo, which we are insure -that doesn't yields same result among all the rows. The simplest UniqueKey -format is primary key. +review comments: +XXX Maybe move this to src/backend/optimizer/README.uniquekey? +XXX multi_nullvals name seems a bit weird +XXX no info about populate_distinctrel_uniquekeys, populate_grouprel_uniquekeys, populate_unionrel_uniquekeys +----- +src/backend/optimizer/path/README.uniquekey + +UniqueKey +========= + +UniqueKey is a set of exprs for a RelOptInfo, which are known to have unique +values on all the rows in the relation. A trivial example is a primary key +defined on a relation - each attributes of the constraint is a unique key. + +We can use this knowledge to perform optimization in a number of places. Some +of the optimizations are fairly obvious, others are less so: + +1. remove DISTINCT node if the clause is unique +2. remove aggregation if group by clause is unique +3. remove_useless_joins +4. reduce_semianti_joins +5. Index Skip Scan (WIP) +6. Aggregation Push-Down without 2-phase aggregation if the join can't + duplicate the aggregated rows. (WIP) + -However we define the UnqiueKey as below. +UniqueKey struct +---------------- -typedef struct UniqueKey -{ +A UnqiueKey is represented by the following struct: + + typedef struct UniqueKey + { NodeTag type; List *exprs; bool multi_nullvals; -} UniqueKey; - -exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL -is a special case of UniqueKey, which means there is only one row in that -relation.it has a stronger semantic than others. like SELECT uk FROM t; uk is -normal unique key and may have different values. SELECT colx FROM t WHERE uk = -const. colx is unique AND we have only 1 value. This field can used for -innerrel_is_unique. this logic is handled specially in add_uniquekey_for_onerow -function. - -multi_nullvals: true means multi null values may exist in these exprs, so the -uniqueness is not guaranteed in this case. This field is necessary for -remove_useless_join & reduce_unique_semijoins where we don't mind these -duplicated NULL values. It is set to true for 2 cases. One is a unique key -from a unique index but the related column is nullable. The other one is for -outer join. see populate_joinrel_uniquekeys for detail. - - -The UniqueKey can be used at the following cases at least: -1. remove_useless_joins. -2. reduce_semianti_joins -3. remove distinct node if distinct clause is unique. -4. remove aggnode if group by clause is unique. -5. Index Skip Scan (WIP) -6. Aggregation Push Down without 2 phase aggregation if the join can't - duplicated the aggregated rows. (work in progress feature) - -2. How is it maintained? - -We have a set of populate_xxx_unqiuekeys functions to maintain the uniquekey on -various cases. xxx includes baserel, joinrel, partitionedrel, distinctrel, -groupedrel, unionrel. and we also need to convert the uniquekey from subquery -to outer relation, which is what convert_subquery_uniquekeys does. - -1. The first part is about baserel. We handled 3 cases. suppose we have Unique -Index on (a, b). - -1. SELECT a, b FROM t. UniqueKey (a, b) -2. SELECT a FROM t WHERE b = 1; UniqueKey (a) -3. SELECT .. FROM t WHERE a = 1 AND b = 1; UniqueKey (NIL). onerow case, every - column is Unique. - -2. The next part is joinrel, this part is most error-prone, we simplified the rules -like below: -1. If the relation's UniqueKey can't be duplicated after join, then is will be - still valid for the join rel. The function we used here is - innerrel_keeps_unique. The basic idea is innerrel.any_col = outer.uk. - -2. If the UnqiueKey can't keep valid via the rule 1, the combination of the - UniqueKey from both sides are valid for sure. We can prove this as: if the - unique exprs from rel1 is duplicated by rel2, the duplicated rows must - contains different unique exprs from rel2. - -More considerations about onerow: -1. If relation with one row and it can't be duplicated, it is still possible - contains mulit_nullvas after outer join. -2. If the either UniqueKey can be duplicated after join, the can get one row - only when both side is one row AND there is no outer join. -3. Whenever the onerow UniqueKey is not a valid any more, we need to convert one - row UniqueKey to normal unique key since we don't store exprs for one-row - relation. get_exprs_from_uniquekeys will be used here. - - -More considerations about multi_nullvals after join: + } UniqueKey; + +exprs is a list of exprs which are know to be unique on current RelOptInfo. + +exprs = NIL is a special case, meaning there is only one row in the relation. +This has has a stronger semantic than others. Consider for example + + SELECT uk FROM t + +where 'uk' is a unique key. This guarantees uniqueness, but there may be mamy +rows in the relation. On the other hand, consider this query + + SELECT colx FROM t WHERE uk = const + +In this case we know there's only a single matching row (thanks to a condition +on the unique key), which in turn guarantees uniqueness of the colx value, even +if there is no constraint on the column itself. + +This knowledge is used in innerrel_is_unique, and is handled as a special case +in add_uniquekey_for_onerow. + + +The multi_nullvals field tracks whether the expressions may contain multiple +NULL values. This can happen for example when the unique key is derived from +a unique index with nullable columns, or because of outer joins (which may add +NULL values to a known-unique list - see populate_joinrel_uniquekeys). + +In this case uniqueness is not guaranteed, but we can still use the information +in places places where NULL values are harmless - when removing useless joins, +reducing semijoins, and so on. + + +How is it maintained? +--------------------- + +Deducing the unique keys depends on the type of the relation - for each case +there's a separate "populate" function: + + +populate_baserel_uniquekeys +--------------------------- + +There are three cases, all assuming there's a unique index (e.g. on (a,b)): + +1. SELECT a, b FROM t => UniqueKey (a, b) +2. SELECT a FROM t WHERE b = 1 => UniqueKey (a) +3. SELECT .. FROM t WHERE a = 1 AND b = 1; => UniqueKey (NIL) + +The last query is the "one row" case, in which case every column is Unique. + + +populate_joinrel_uniquekeys +--------------------------- + +For joins, deducing the unique keys may be fairly complex and error-prone. +We've simplified the rules like this: + +1. If the UniqueKey on an input relation can't be duplicated by the join, then +it will be valid for the join rel. A typical example is a join like this: + + inner_rel.any_col = outer_rel.unique_key + +The function used to detect this is innerrel_keeps_unique. + +2. Any combination of unique keys on each side of the join is a unique key +for the join relation. This can be proved by contradiction - assume we have +unique key on either side of the join - uk1 and uk2. If the values in uk1 get +duplicated by the join with uk2 (by matching the row to multiple rows), the +duplicated rows must have different values in the uk2. + +We can also leverage information about the "one row" case: + +1. If one of the input relations is known to have a single row, and the join +can't duplicate the row (e.g. semi/anti join), we can keep the unique keys. +It may however contain multi_nullvals after an outer join. + +XXX Not sure I understand the original logic/wording :-( + +2. If either UniqueKey can be duplicated after a join, there can be only one +row only when both sides are "one row" AND there is no outer join. + +XXX Why the restriction on not allowing outer joins? + +3. Whenever the one row UniqueKey is not a valid any more, we need to convert +UniqueKey to normal unique key since we don't store exprs for one-row relation. +This is done by get_exprs_from_uniquekeys. + +The join case needs to be careful about multi_nullvals too: + 1. If the original UnqiueKey has multi_nullvals, the final UniqueKey will have - mulit_nullvals in any case. -2. If a unique key doesn't allow mulit_nullvals, after some outer join, it - allows some outer join. +mulit_nullvals in any case too. + +2. If the original unique key doesn't allow multi_nullvals, the unique key for +the join relation may allow multi_nullvals after an outer join. + + +subqueries +---------- + +It's necessary to "translate" unique keys between a subquery and the outer rels, +which is what convert_subquery_uniquekeys does. This does almost exactly what +convert_subquery_pathkeys does for pathkeys. It keeps only unique keys matching +Vars in the outer relation. The relationship between outerrel.Var and +subquery.exprs is built from outerel->subroot->processed_tlist. -3. When we comes to subquery, we need to convert_subquery_unqiuekeys just like -convert_subquery_pathkeys. Only the UniqueKey insides subquery is referenced as -a Var in outer relation will be reused. The relationship between the outerrel.Var -and subquery.exprs is built with outerel->subroot->processed_tlist. +set-returning functions +------------------------ +As for the SRF functions, it will break the uniqueness of uniquekey, However it +is handled in adjust_paths_for_srfs, which happens after the query_planner. So +we will maintain the UniqueKey until there and reset it to NIL at that place. -4. As for the SRF functions, it will break the uniqueness of uniquekey, However it -is handled in adjust_paths_for_srfs, which happens after the query_planner. so -we will maintain the UniqueKey until there and reset it to NIL at that -places. This can't help on distinct/group by elimination cases but probably help -in some other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is -semantic correctly. +This can't help on distinct/group by elimination cases but probably help in some +other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is correct. -5. As for inherit table, we first main the UnqiueKey on childrel as well. But for -partitioned table we need to maintain 2 different kinds of -UnqiueKey. 1). UniqueKey on the parent relation 2). UniqueKey on child -relation for partition wise query. +populate_partitionedrel_uniquekeys +---------------------------------- + +As for inherit table, we first build the UnqiueKey on childrel as well. But for +partitioned table we need to maintain two different kinds of UniqueKey: + +1) UniqueKey on the parent relation + +2) UniqueKey on child + +This is needed because a unique key from the partition may not be be unique key +on the partitioned table. + Example: -CREATE TABLE p (a int not null, b int not null) partition by list (a); + +CREATE TABLE p (a INT NOT NULL, b INT NOT NULL) PARTITION BY LIST (a); + CREATE TABLE p0 partition of p for values in (1); CREATE TABLE p1 partition of p for values in (2); -create unique index p0_b on p0(b); -create unique index p1_b on p1(b); +CREATE UNIQUE INDEX p0_b ON p0(b); +CREATE UNIQUE INDEX p1_b ON p1(b); -Now b is only unique on partition level, so the distinct can't be removed on -the following cases. SELECT DISTINCT b FROM p; +SELECT DISTINCT b FROM p; -Another example is SELECT DISTINCT a, b FROM p WHERE a = 1; Since only one -partition is chosen, the UniqueKey on child relation is same as the UniqueKey on -parent relation. +Now "b" is only unique on partition level, but the two partitions may contain +duplicate values for the "b" column (with different values in "a"). That means +the DISTINCT clause can't be removed. -Another usage of UniqueKey on partition level is it be helpful for -partition-wise join. +Now consider: -As for the UniqueKey on parent table level, it comes with 2 different ways, -1). the UniqueKey is also derived in UniqueKey index, but the index must be same -in all the related children relations and the unique index must contains -Partition Key in it. Example: +SELECT DISTINCT a, b FROM p WHERE a = 1 + +In this case, the optimizer eliminates all partitions except for one, so that +the UniqueKey is valid for the parent relation too. + +UniqueKey at a partition level is useful for partition-wise join too. + +XXX Explain why is it useful? + +A UniqueKey from a partition can be transferred to the parent relation, in two +cases. A trivial case is if there's a single child relation (e.g. thanks to +partition elimination). In that case all unique keys on the child relation are +automatically valid for the parent relation. If there are multiple relations, +the unique key must be defived from an index present in all partitions, and the +index has to include the partition key. + +Example: CREATE UNIQUE INDEX p_ab ON p(a, b); -- where a is the partition key. -- Query SELECT a, b FROM p; the (a, b) is a UniqueKey of p. -2). If the parent relation has only one childrel, the UniqueKey on childrel is - the UniqueKey on parent as well. diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index 66bf6f19f7..a801707eaa 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -581,7 +581,8 @@ set_plain_rel_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) /* * Now that we've marked which partial indexes are suitable, we can now - * build the relation's unique keys. + * build the relation's unique keys. We need to do it in this order, + * so that we don't deduce unique keys from inapplicable partial indexes. */ populate_baserel_uniquekeys(root, rel, rel->indexlist); @@ -1305,6 +1306,12 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, /* Add paths to the append relation. */ add_paths_to_append_rel(root, rel, live_childrels); + + /* + * XXX Maybe move the check into populate populate_partitionedrel_uniquekeys? + * XXX What if it's append rel (but not partitioned one), but there's only one + * child relation? We could still deduce unique keys, no? + */ if (IS_PARTITIONED_REL(rel)) populate_partitionedrel_uniquekeys(root, rel, live_childrels); } @@ -2314,6 +2321,7 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, pathkeys, required_outer)); } + /* Convert subpath's unique keys to outer representation */ convert_subquery_uniquekeys(root, rel, sub_final_rel); /* If outer rel allows parallelism, do same for partial paths. */ diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c index 7271f044ec..eefba449d6 100644 --- a/src/backend/optimizer/path/joinrels.c +++ b/src/backend/optimizer/path/joinrels.c @@ -925,6 +925,13 @@ populate_joinrel_with_paths(PlannerInfo *root, RelOptInfo *rel1, /* Apply partitionwise join technique, if possible. */ try_partitionwise_join(root, rel1, rel2, joinrel, sjinfo, restrictlist); + /* + * Determine which of the unique keys from input relations are applicable + * for the join result. + * + * XXX We do this after trying the partitionwise join, because that may allow + * using additional unique keys. + */ populate_joinrel_uniquekeys(root, joinrel, rel1, rel2, restrictlist, sjinfo->jointype); } diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c index 77ed2b2eff..114e8334f5 100644 --- a/src/backend/optimizer/path/uniquekeys.c +++ b/src/backend/optimizer/path/uniquekeys.c @@ -36,7 +36,7 @@ typedef struct UniqueKeyContextData bool useful; } *UniqueKeyContext; -static List *initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel); +static List *initialize_uniquecontext_for_joinrel(RelOptInfo *inputrel); static bool innerrel_keeps_unique(PlannerInfo *root, RelOptInfo *outerrel, RelOptInfo *innerrel, @@ -80,8 +80,20 @@ static void add_uniquekey_from_sortgroups(PlannerInfo *root, /* * populate_baserel_uniquekeys - * Populate 'baserel' uniquekeys list by looking at the rel's unique index - * and baserestrictinfo + * Build list of unique keys for the base relation. + * + * Inspects unique indexes defined on the relation and determines what + * unique keys are valid. Partial indexes are considered too, if the + * predicate is valid. + * + * This also inspects baserestrictinfo, because we need to determine + * which opclass families are interesting when inspecting indexes. If we + * have a unique index and distinct clause with a mismatching opclasses, + * we should not use that. + * + * XXX Why does this look at baserestrictinfo? + * + * XXX What about collations? */ void populate_baserel_uniquekeys(PlannerInfo *root, @@ -99,22 +111,48 @@ populate_baserel_uniquekeys(PlannerInfo *root, Assert(baserel->rtekind == RTE_RELATION); + if (!indexlist) + return; + + /* + * Determine which unique indexes to use to build the unique keys. + * We have to skip partial with predicates not matched by the query, + * and unique indexes that are not immediately enforced. + * + * XXX Do we actually skip indexes that are not immediate? + * XXX What about hypothetical indexes? + */ foreach(lc, indexlist) { IndexOptInfo *ind = (IndexOptInfo *) lfirst(lc); + if (!ind->unique || !ind->immediate || (ind->indpred != NIL && !ind->predOK)) continue; + matched_uniq_indexes = lappend(matched_uniq_indexes, ind); } + /* If there are not applicable unique indexes, we're done. */ if (matched_uniq_indexes == NIL) return; - /* Check which attrs is used in baserel->reltarget */ - pull_varattnos((Node *)baserel->reltarget->exprs, baserel->relid, &used_attrs); + /* + * Determine which attrs are referenced in baserel->reltarget. To use the + * unique key info, we need all the columns - a unique index on (a,b) may + * not be unique on (a). If a column is missing in reltarget, the nodes + * above can't possibly use it, and we can just ignore any matching index. + */ + pull_varattnos((Node *) baserel->reltarget->exprs, baserel->relid, &used_attrs); - /* Check which attrno is used at a mergeable const filter */ + /* + * Check which attrno is used at a mergeable const filter + * + * XXX This is not lookint att attrno at all, maybe obsolete comment? + * + * Seems the primary purpose of this is determining which opclass + * families to use when matching unique indexes in the next loop? + */ foreach(lc, baserel->baserestrictinfo) { RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); @@ -122,6 +160,10 @@ populate_baserel_uniquekeys(PlannerInfo *root, if (rinfo->mergeopfamilies == NIL) continue; + /* + * XXX What if bms_is_empty is true for both left_relids/right_relids? + * Or what if it's false in both cases? + */ if (bms_is_empty(rinfo->left_relids)) { const_exprs = lappend(const_exprs, get_rightop(rinfo->clause)); @@ -136,40 +178,69 @@ populate_baserel_uniquekeys(PlannerInfo *root, expr_opfamilies = lappend(expr_opfamilies, rinfo->mergeopfamilies); } + /* + * Now try to match unique indexes to attributes in reltarget, and to + * merge operator families. The index may be on the right attributes, + * but if it's not matching the opfamily it's useless. + * + * XXX Can we have multiple baserestrictinfo for the same attribute, + * with different opfamilies? Probably not. + */ foreach(lc, matched_uniq_indexes) { - bool multi_nullvals, useful; - List *exprs = get_exprs_from_uniqueindex(lfirst_node(IndexOptInfo, lc), - const_exprs, - expr_opfamilies, - used_attrs, - &useful, - &multi_nullvals); - if (useful) + bool multi_nullvals, + useful; + + IndexOptInfo *index_info = (IndexOptInfo *) lfirst_node(IndexOptInfo, lc); + + List *exprs = get_exprs_from_uniqueindex(index_info, + const_exprs, + expr_opfamilies, + used_attrs, + &useful, + &multi_nullvals); + + if (!useful) + continue; + + /* + * All the columns in Unique Index matched with a restrictinfo, so + * that we know there's just a one row in the result. If we find + * such index, we're done - we discard all other unique keys and + * keep just this special one. In principle, this is a stronger + * guarantee, because all subsets of one row are still unique. + * + * XXX Is it correct to just return? Doesn't that prevent some + * optimizations that might be possible with the other keys? + */ + if (exprs == NIL) { - if (exprs == NIL) - { - /* All the columns in Unique Index matched with a restrictinfo */ - add_uniquekey_for_onerow(baserel); - return; - } - baserel->uniquekeys = lappend(baserel->uniquekeys, - makeUniqueKey(exprs, multi_nullvals)); + /* discards all previous uniquekeys */ + add_uniquekey_for_onerow(baserel); + return; } + + baserel->uniquekeys = lappend(baserel->uniquekeys, + makeUniqueKey(exprs, multi_nullvals)); } } /* * populate_partitionedrel_uniquekeys - * The UniqueKey on partitionrel comes from 2 cases: - * 1). Only one partition is involved in this query, the unique key can be - * copied to parent rel from childrel. - * 2). There are some unique index which includes partition key and exists - * in all the related partitions. - * We never mind rule 2 if we hit rule 1. + * Determine unique keys for a partitioned relation. + * + * Inspects unique keys for all partitions and derives unique keys that + * are valid for the whole partitioned table. There are two basic cases: + * + * 1) There's only one remaining partition (thanks to pruning all other + * partitions). In this case all the unique keys from the partition are + * trivially valid for the partitioned table. + * + * 2) All the partitions have the same unique index (on the same set of + * columns), and the index includes the partition key. This ensures the + * combination of values is unique for the whole partitioned table. */ - void populate_partitionedrel_uniquekeys(PlannerInfo *root, RelOptInfo *rel, @@ -180,110 +251,181 @@ populate_partitionedrel_uniquekeys(PlannerInfo *root, RelOptInfo *childrel; bool is_first = true; + /* XXX What about append rels? At least for the one-child case? */ Assert(IS_PARTITIONED_REL(rel)); + /* if there are no child relations, we're done. */ if (childrels == NIL) return; /* - * If there is only one partition used in this query, the UniqueKey in childrel is - * still valid in parent level, but we need convert the format from child expr to - * parent expr. + * If there is only one partition used in this query, the UniqueKey for + * a child relation is still valid for the parent level. We need to + * convert the format from child expr to parent expr. */ if (list_length(childrels) == 1) { - /* Check for Rule 1 */ RelOptInfo *childrel = linitial_node(RelOptInfo, childrels); ListCell *lc; + Assert(childrel->reloptkind == RELOPT_OTHER_MEMBER_REL); + + /* If the partition has a single row, so does the parent. */ if (relation_is_onerow(childrel)) { add_uniquekey_for_onerow(rel); return; } + /* + * Inspect the unique keys one by one, try reusing them for the + * parent relation. + * + * FIXME This needs more work to handle expressions and not just + * simple Vars. + */ foreach(lc, childrel->uniquekeys) { + ListCell *lc2; + List *parent_exprs = NIL; + bool can_reuse = true; + UniqueKey *ukey = lfirst_node(UniqueKey, lc); AppendRelInfo *appinfo = find_appinfo_by_child(root, childrel->relid); - List *parent_exprs = NIL; - bool can_reuse = true; - ListCell *lc2; + + /* + * XXX Not sure what exactly we do here. Surely we deal with + * expressions at child/parent level elsewhere? Can't we just + * copy the code from there? + */ foreach(lc2, ukey->exprs) { - Var *var = (Var *)lfirst(lc2); + Var *var = (Var *) lfirst(lc2); + /* - * If the expr comes from a expression, it is hard to build the expression - * in parent so ignore that case for now. + * XXX For now this only supports simple Var expressions, + * so if there's a more complex expression we'll not copy + * the unique key to the parent. */ if(!IsA(var, Var)) { can_reuse = false; break; } + /* Convert it to parent var */ - parent_exprs = lappend(parent_exprs, find_parent_var(appinfo, var)); + parent_exprs = lappend(parent_exprs, + find_parent_var(appinfo, var)); } - if (can_reuse) - rel->uniquekeys = lappend(rel->uniquekeys, - makeUniqueKey(parent_exprs, - ukey->multi_nullvals)); + + /* ignore unique keys with complex expressions */ + if (!can_reuse) + continue; + + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(parent_exprs, + ukey->multi_nullvals)); } + + return; } - else + + /* + * A parent with multiple child relations. We only care about indexes that + * are in all child relations, so we loop through indexes on the first one + * and check that they exist in the other child relations too. + */ + + childrel = linitial_node(RelOptInfo, childrels); + foreach(lc, childrel->indexlist) { - /* Check for rule 2 */ - childrel = linitial_node(RelOptInfo, childrels); - foreach(lc, childrel->indexlist) - { - IndexOptInfo *ind = lfirst(lc); - IndexOptInfo *modified_index; - if (!ind->unique || !ind->immediate || - (ind->indpred != NIL && !ind->predOK)) - continue; + IndexOptInfo *ind = lfirst(lc); + IndexOptInfo *modified_index; - /* - * During simple_copy_indexinfo_to_parent, we need to convert var from - * child var to parent var, index on expression is too complex to handle. - * so ignore it for now. - */ - if (ind->indexprs != NIL) - continue; + /* + * Ignore indexes that are not unique, immediately enforced. Partial + * indexes with mismatched predicate are useless too. + */ + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; - modified_index = simple_copy_indexinfo_to_parent(root, rel, ind); - /* - * If the unique index doesn't contain partkey, then it is unique - * on this partition only, so it is useless for us. - */ - if (!index_constains_partkey(rel, modified_index)) - continue; + /* + * During simple_copy_indexinfo_to_parent, we need to convert var from + * child var to parent var, index on expression is too complex to handle. + * so ignore it for now. + * + * FIXME We should support indexes on expressions. + */ + if (ind->indexprs != NIL) + continue; - global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index); - } + /* + * Adopt the index definition for the parent. + * + * XXX This seems rather weird. We're constructing "artificial" index + * for the partitioned table (kinda like a global index). Can't we + * just have some simpler struct representing it? + */ + modified_index = simple_copy_indexinfo_to_parent(root, rel, ind); + + /* + * If the unique index doesn't contain partkey, then it is unique + * on this partition only, so it is useless for us. + * + * XXX Can't we do this check before simple_copy_indexinfo_to_parent? + */ + if (!index_constains_partkey(rel, modified_index)) + continue; - if (global_uniq_indexlist != NIL) + global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index); + } + + /* if there are no applicable unique indexes, we're done */ + if (!global_uniq_indexlist) + return; + + /* + * We iterate over the child relations first, and inspect the unique + * indexes for each hild, because this way we can stop early if we + * happen to eliminate all the unique indexes. + */ + foreach(lc, childrels) + { + RelOptInfo *child = lfirst(lc); + + /* skip the first index, which is where we got the list from */ + if (is_first) { - foreach(lc, childrels) - { - RelOptInfo *child = lfirst(lc); - if (is_first) - { - is_first = false; - continue; - } - adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist); - } - /* Now we have a list of unique index which are exactly same on all childrels, - * Set the UniqueKey just like it is non-partition table - */ - populate_baserel_uniquekeys(root, rel, global_uniq_indexlist); + is_first = false; + continue; } + + /* match the unique keys to indexes on this child */ + adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist); + + /* + * If we have eliminated all unique indexes, no point in looking at + * the remaining child relations. + */ + if (!global_uniq_indexlist) + break; } + + /* Now we have a list of unique index which are exactly same on all child + * relations. Set the UniqueKey just like it is non-partition table. + */ + populate_baserel_uniquekeys(root, rel, global_uniq_indexlist); } /* * populate_distinctrel_uniquekeys + * Update unique keys for relation produced by DISTINCT. + * + * We can keep all unique keys from the input relations, because DISTINCT + * can only remove rows - it can't duplicate them. Also, the DISTINCT clause + * itself is a unique key, so add that. */ void populate_distinctrel_uniquekeys(PlannerInfo *root, @@ -292,11 +434,13 @@ populate_distinctrel_uniquekeys(PlannerInfo *root, { /* The unique key before the distinct is still valid. */ distinctrel->uniquekeys = list_copy(inputrel->uniquekeys); + add_uniquekey_from_sortgroups(root, distinctrel, root->parse->distinctClause); } /* * populate_grouprel_uniquekeys + * */ void populate_grouprel_uniquekeys(PlannerInfo *root, @@ -305,54 +449,76 @@ populate_grouprel_uniquekeys(PlannerInfo *root, { Query *parse = root->parse; - bool input_ukey_added = false; ListCell *lc; + /* + * XXX Is this actually valid, before checking fro grouping sets? + * The grouping sets may produce duplicate row even with just a single + * input row, I think. + */ if (relation_is_onerow(inputrel)) { add_uniquekey_for_onerow(grouprel); return; } + + /* + * Bail out if there are grouping sets. + * + * XXX Could we maybe inspect the grouping sets and determine if this + * generates distinct combinations? In some cases that's clearly not + * the case (rollup, cube), but for some simple cases it might. + */ if (parse->groupingSets) return; - /* A Normal group by without grouping set. */ - if (parse->groupClause) + /* It has aggregation but without a group by, so only one row returned */ + if (!parse->groupClause) + add_uniquekey_for_onerow(grouprel); + + /* + * A regular group by, without grouping sets. + * + * Obviously, the whole group clause determines a unique key. But if + * there are smaller unique keys on the input rel, we prefer those + * because those are more flexible. If (a,b) is unique, (a,b,c) is + * unique too. Only when there are no such smaller unique keys, we + * add the unique key derived from the group clause. + */ + foreach(lc, inputrel->uniquekeys) { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + /* - * Current even the groupby clause is Unique already, but if query has aggref - * We have to create grouprel still. To keep the UnqiueKey short, we will check - * the UniqueKey of input_rel still valid, if so we reuse it. + * Ignore unique keys on the input that are not subset of the + * group clause. We can't use incomplete unique keys. */ - foreach(lc, inputrel->uniquekeys) - { - UniqueKey *ukey = lfirst_node(UniqueKey, lc); - if (list_is_subset(ukey->exprs, grouprel->reltarget->exprs)) - { - grouprel->uniquekeys = lappend(grouprel->uniquekeys, - ukey); - input_ukey_added = true; - } - } - if (!input_ukey_added) - /* - * group by clause must be a super-set of grouprel->reltarget->exprs except the - * aggregation expr, so if such exprs is unique already, no bother to generate - * new uniquekey for group by exprs. - */ - add_uniquekey_from_sortgroups(root, - grouprel, - root->parse->groupClause); + if (!list_is_subset(ukey->exprs, grouprel->reltarget->exprs)) + continue; + + grouprel->uniquekeys = lappend(grouprel->uniquekeys, ukey); } - else - /* It has aggregation but without a group by, so only one row returned */ - add_uniquekey_for_onerow(grouprel); + + /* + * Group clause must be a super-set of of grouprel->reltarget->exprs, + * except for the aggregation expressions. So if we found a smaller + * unique key on the input relation, don't bother adding a unique key + * for the group clause. + */ + if (!grouprel->uniquekeys) + add_uniquekey_from_sortgroups(root, + grouprel, + root->parse->groupClause); } /* * simple_copy_uniquekeys - * Using a function for the one-line code makes us easy to check where we simply - * copied the uniquekey. + * Copy yhe unique keys between relations. + * + * Using a function for the one-line code makes us easy to check where we + * simply copied the uniquekey. + * + * XXX Seems like an overkill, not sure what's the purpose? */ void simple_copy_uniquekeys(RelOptInfo *oldrel, @@ -362,24 +528,27 @@ simple_copy_uniquekeys(RelOptInfo *oldrel, } /* - * populate_unionrel_uniquekeys + * populate_unionrel_uniquekeys + * Determine unique keys for UNION relation. + * + * XXX Does this need to care about UNION vs. UNION ALL? At least in the + * one-row code path? */ void populate_unionrel_uniquekeys(PlannerInfo *root, - RelOptInfo *unionrel) + RelOptInfo *unionrel) { - ListCell *lc; - List *exprs = NIL; + ListCell *lc; + List *exprs = NIL; Assert(unionrel->uniquekeys == NIL); + /* XXX Why are we copying the expressions? */ foreach(lc, unionrel->reltarget->exprs) - { exprs = lappend(exprs, lfirst(lc)); - } + /* SQL: select union select; is valid, we need to handle it here. */ if (exprs == NIL) - /* SQL: select union select; is valid, we need to handle it here. */ add_uniquekey_for_onerow(unionrel); else unionrel->uniquekeys = lappend(unionrel->uniquekeys, @@ -389,6 +558,7 @@ populate_unionrel_uniquekeys(PlannerInfo *root, /* * populate_joinrel_uniquekeys + * Determine unique keys for a join relation. * * populate uniquekeys for joinrel. We will check each relation to see if its * UniqueKey is still valid via innerrel_keeps_unique, if so, we add it to @@ -404,70 +574,99 @@ populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, RelOptInfo *innerrel, List *restrictlist, JoinType jointype) { - ListCell *lc, *lc2; - List *clause_list = NIL; - List *outerrel_ukey_ctx; - List *innerrel_ukey_ctx; - bool inner_onerow, outer_onerow; - bool mergejoin_allowed; - - /* Care about the outerrel relation only for SEMI/ANTI join */ + ListCell *lc, + *lc2; + List *clause_list = NIL; + List *outerrel_ukey_ctx; + List *innerrel_ukey_ctx; + bool inner_onerow, + outer_onerow; + bool mergejoin_allowed; + + /* For SEMI/ANTI join, we care only about the outerrel unique keys. */ if (jointype == JOIN_SEMI || jointype == JOIN_ANTI) { foreach(lc, outerrel->uniquekeys) { UniqueKey *uniquekey = lfirst_node(UniqueKey, lc); + + /* Keep the unique key if it's included in the joinrel. */ if (list_is_subset(uniquekey->exprs, joinrel->reltarget->exprs)) joinrel->uniquekeys = lappend(joinrel->uniquekeys, uniquekey); } + return; } + /* XXX What about JOIN_RIGHT? */ Assert(jointype == JOIN_LEFT || jointype == JOIN_FULL || jointype == JOIN_INNER); - /* Fast path */ + /* + * For regular joins, we need to combine unique keys from both sides + * of the join, to get a new unique key for the join relation. So if + * either side does not have a unique key, bail out. + */ if (innerrel->uniquekeys == NIL || outerrel->uniquekeys == NIL) return; + /* XXX maybe move to the if blocks? Not needed outside. */ inner_onerow = relation_is_onerow(innerrel); outer_onerow = relation_is_onerow(outerrel); - outerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(outerrel); - innerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(innerrel); + outerrel_ukey_ctx = initialize_uniquecontext_for_joinrel(outerrel); + innerrel_ukey_ctx = initialize_uniquecontext_for_joinrel(innerrel); - clause_list = select_mergejoin_clauses(root, joinrel, outerrel, innerrel, + clause_list = select_mergejoin_clauses(root, + joinrel, outerrel, innerrel, restrictlist, jointype, &mergejoin_allowed); - if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true /* reverse */)) + /* + * XXX Seems a bit weird that it's called innerrel_keeps_unique but we + * seem to use it in both directions. Or what's the "reverse" for? The + * "reverse" name is not particularly descriptive. + */ + if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true)) { - bool outer_impact = jointype == JOIN_FULL; + bool outer_impact = (jointype == JOIN_FULL); + + /* Inspect unique keys on the outer relation. */ foreach(lc, outerrel_ukey_ctx) { UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + /* + * If the output of the join does not include all the parts of the + * unique key, it's useless, so mark it accordingly and ignore it. + */ if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) { ctx->useful = false; continue; } - /* Outer relation has one row, and the unique key is not duplicated after join, - * the joinrel will still has one row unless the jointype == JOIN_FULL. + /* + * When the outer relation has one row, and the unique key is not + * duplicated after join, so the joinrel will still have just one + * row unless the jointype == JOIN_FULL. In that case we're done, + * it's the strictest unique key possible. + * + * If it's one-row with a JOIN_FULL, it might produce multiple + * rows with NULLs, so set multi_nullvals. We also need to set + * the exprs correctly since it can't be NIL any more. + * + * For other cases (not one-row relation), we just reuse the + * unique key, but we may need to tweak the multi_nullvals. */ if (outer_onerow && !outer_impact) { add_uniquekey_for_onerow(joinrel); return; } - else if (outer_onerow) + else if (outer_onerow) /* one-row and FULL join */ { - /* - * The onerow outerrel becomes multi rows and multi_nullvals - * will be changed to true. We also need to set the exprs correctly since it - * can't be NIL any more. - */ ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, outerrel, NULL)) { joinrel->uniquekeys = lappend(joinrel->uniquekeys, @@ -485,18 +684,38 @@ populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, joinrel->uniquekeys = lappend(joinrel->uniquekeys, ctx->uniquekey); } + + /* + * Mark the unique key as added, so that we can ignore it later + * when combining unique keys from both sides of the join. + */ ctx->added_to_joinrel = true; } } + /* + * XXX Seems this actually checks if "outerrel keeps unique" so the name + * is misleading. Of maybe it's the previous block, not sure. + * + * XXX So why does this consider JOIN_FULL and JOIN_LEFT, while the previous + * block only cares about JOIN_FULL? + * + * XXX This is almost exact copy of the previous block, so maybe make it + * a separate function and just call it twice? + */ if (innerrel_keeps_unique(root, outerrel, innerrel, clause_list, false)) { - bool outer_impact = jointype == JOIN_FULL || jointype == JOIN_LEFT;; + bool outer_impact = (jointype == JOIN_FULL || jointype == JOIN_LEFT); + /* Inspect unique keys on the inner relation. */ foreach(lc, innerrel_ukey_ctx) { UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + /* + * If the output of the join does not include all the parts of the + * unique key, it's useless, so mark it accordingly and ignore it. + */ if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) { ctx->useful = false; @@ -529,29 +748,52 @@ populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, ctx->uniquekey); } + + /* + * Mark the unique key as added, so that we can ignore it later + * when combining unique keys from both sides of the join. + */ ctx->added_to_joinrel = true; } } /* - * The combination of the UniqueKey from both sides is unique as well regardless - * of join type, but no bother to add it if its subset has been added to joinrel - * already or it is not useful for the joinrel. + * XXX What if either of the previous two conditions did not match? In + * that case we haven't updated the useful flag, and maybe the unique + * key is not useful, but we don't know, right? So we should not be + * using it in the next loop. Or maybe we should evaluate the flag + * before the loops. + */ + + /* + * The combination of the UniqueKey from both sides is unique as well, + * regardless of the join type. But don't bother to add it if its + * subset has been added to joinrel already or when it's not useful for + * the joinrel. + * + * XXX Maybe we should have a flag that both sides have useful keys? + * Or maybe the loops are short/cheap? */ foreach(lc, outerrel_ukey_ctx) { UniqueKeyContext ctx1 = (UniqueKeyContext) lfirst(lc); + + /* when not useful or already added to the joinrel, skip it */ if (ctx1->added_to_joinrel || !ctx1->useful) continue; + foreach(lc2, innerrel_ukey_ctx) { UniqueKeyContext ctx2 = (UniqueKeyContext) lfirst(lc2); + + /* when not useful or already added to the joinrel, skip it */ if (ctx2->added_to_joinrel || !ctx2->useful) continue; + + /* If we add a onerow UniqueKey, we don't need another key. */ if (add_combined_uniquekey(root, joinrel, outerrel, innerrel, ctx1->uniquekey, ctx2->uniquekey, jointype)) - /* If we set a onerow UniqueKey to joinrel, we don't need other. */ return; } } @@ -560,8 +802,9 @@ populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, /* * convert_subquery_uniquekeys + * Covert the UniqueKey in subquery to outer relation. * - * Covert the UniqueKey in subquery to outer relation. + * XXX Explain what exactly does the conversion do? */ void convert_subquery_uniquekeys(PlannerInfo *root, RelOptInfo *currel, @@ -618,12 +861,14 @@ void convert_subquery_uniquekeys(PlannerInfo *root, /* * innerrel_keeps_unique + * Check if Unique key on the innerrel is valid after join. * - * Check if Unique key of the innerrel is valid after join. innerrel's UniqueKey - * will be still valid if innerrel's any-column mergeop outrerel's uniquekey - * exists in clause_list. + * innerrel's UniqueKey will be still valid if innerrel's any-column mergeop + * outrerel's uniquekey exists in clause_list * * Note: the clause_list must be a list of mergeable restrictinfo already. + * + * XXX Misleading name? We seem to use it for "outerrel_keeps_unique" too. */ static bool innerrel_keeps_unique(PlannerInfo *root, @@ -634,26 +879,32 @@ innerrel_keeps_unique(PlannerInfo *root, { ListCell *lc, *lc2, *lc3; + /* XXX probably not needed, duplicate with the check in the caller + * (populate_joinrel_uniquekeys). But it's cheap. */ if (outerrel->uniquekeys == NIL || innerrel->uniquekeys == NIL) return false; /* Check if there is outerrel's uniquekey in mergeable clause. */ foreach(lc, outerrel->uniquekeys) { - List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs; - bool clauselist_matchs_all_exprs = true; + List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs; + bool clauselist_matchs_all_exprs = true; + foreach(lc2, outer_uq_exprs) { Node *outer_uq_expr = lfirst(lc2); bool find_uq_expr_in_clauselist = false; + foreach(lc3, clause_list) { RestrictInfo *rinfo = lfirst_node(RestrictInfo, lc3); Node *outer_expr; + if (reverse) outer_expr = rinfo->outer_is_left ? get_rightop(rinfo->clause) : get_leftop(rinfo->clause); else outer_expr = rinfo->outer_is_left ? get_leftop(rinfo->clause) : get_rightop(rinfo->clause); + if (equal(outer_expr, outer_uq_expr)) { find_uq_expr_in_clauselist = true; @@ -677,22 +928,37 @@ innerrel_keeps_unique(PlannerInfo *root, /* * relation_is_onerow - * Check if it is a one-row relation by checking UniqueKey. + * Check if it is a one-row relation by checking UniqueKey. + * + * The one-row is a special case - there has to be just a single unique key, + * with no expressions. */ bool relation_is_onerow(RelOptInfo *rel) { UniqueKey *ukey; - if (rel->uniquekeys == NIL) + + /* there has to be exactly one unique key */ + if (list_length(rel->uniquekeys) != 1) return false; + ukey = linitial_node(UniqueKey, rel->uniquekeys); - return ukey->exprs == NIL && list_length(rel->uniquekeys) == 1; + + /* the unique key must have no expressions */ + return (ukey->exprs == NIL); } /* * relation_has_uniquekeys_for - * Returns true if we have proofs that 'rel' cannot return multiple rows with - * the same values in each of 'exprs'. Otherwise returns false. + * Determines if the relation has unique key for a list of expressions. + * + * Returns true iff we can prove that the relation cannot return multiple rows + * with the same values in the provided expression. + * + * allow_multinulls determines whether we allow multiple NULL values or not. + * + * The special "one-row" unique key is considered incompatible with all + * possible expressions. */ bool relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel, @@ -710,20 +976,39 @@ relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel, foreach(lc, rel->uniquekeys) { UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (ukey->multi_nullvals && !allow_multinulls) continue; + if (list_is_subset(ukey->exprs, exprs)) return true; } + return false; } /* * get_exprs_from_uniqueindex + * Return a list of expressions from a unique index. + * + * Provided with a list of expressions and opclass families, we try to match + * it to the index. If useful, we produce a list of index expressions (subset + * of the list we provided). + * + * We simply walk through the index expressions, and for each expression we + * check three things: * - * Return a list of exprs which is unique. set useful to false if this - * unique index is not useful for us. + * 1) If there's a matching (expr = Const) clause, we can simply ignore the + * expressions. Unique index on (a,b,c) guarantees uniqueness on (a,b) when + * there's condition (c=1). + * + * 2) Check that the index expression is present in the relation we're + * dealing with. If not, the unique key would be useless anyway, and the + * index can't produce unique key. + * + * XXX Shouldn't it be enough to return NULL when the index is not useful? + * The extra flag seems a bit unnecessary. */ static List * get_exprs_from_uniqueindex(IndexOptInfo *unique_index, @@ -743,18 +1028,19 @@ get_exprs_from_uniqueindex(IndexOptInfo *unique_index, indexpr_item = list_head(unique_index->indexprs); for(c = 0; c < unique_index->ncolumns; c++) { - int attr = unique_index->indexkeys[c]; - Expr *expr; - bool matched_const = false; - ListCell *lc1, *lc2; + int attr = unique_index->indexkeys[c]; + Expr *expr; + bool matched_const = false; + ListCell *lc1, *lc2; - if(attr > 0) + if (attr > 0) { + /* regular attribute, just use the expression from index tlist */ expr = list_nth_node(TargetEntry, unique_index->indextlist, c)->expr; } else if (attr == 0) { - /* Expression index */ + /* expression from the index */ expr = lfirst(indexpr_item); indexpr_item = lnext(unique_index->indexprs, indexpr_item); } @@ -764,29 +1050,43 @@ get_exprs_from_uniqueindex(IndexOptInfo *unique_index, Assert(false); } + /* should have a valid expression now */ + Assert(expr); + /* - * Check index_col = Const case with regarding to opfamily checking - * If we can remove the index_col from the final UniqueKey->exprs. + * Check if there's (index_col = Const) condition, and that it's using + * a compatible opfamily. If yes, we can remove the index_col from the + * final UniqueKey->exprs, because the value is constant (so removing + * it can't introduce duplicities). */ forboth(lc1, const_exprs, lc2, const_expr_opfamilies) { - if (list_member_oid((List *)lfirst(lc2), unique_index->opfamily[c]) - && match_index_to_operand((Node *) lfirst(lc1), c, unique_index)) + List *opfamilies = (List *) lfirst(lc2); + Node *cexpr = (Node *) lfirst(lc1); + + if (list_member_oid(opfamilies, unique_index->opfamily[c]) && + match_index_to_operand(cexpr, c, unique_index)) { matched_const = true; break; } } + /* it's constant, so ignore the expression */ if (matched_const) continue; - /* Check if the indexed expr is used in rel */ + /* + * Check if the indexed expr is used in rel. We do this after the + * (col = Const) check, because nn expression may be in a a restrict + * clause and not in the reltarget. So we don't want to rule out an + * index unnecessarily. + */ if (attr > 0) { /* - * Normal Indexed column, if the col is not used, then the index is useless - * for uniquekey. + * Normal indexed column, if the col is not used, then the index + * is useless for uniquekey. */ attr -= FirstLowInvalidHeapAttributeNumber; @@ -806,67 +1106,85 @@ get_exprs_from_uniqueindex(IndexOptInfo *unique_index, /* check not null property. */ if (attr == 0) { - /* We never know if a expression yields null or not */ + /* We never know if an expression yields null or not */ *multi_nullvals = true; } - else if (!bms_is_member(attr, unique_index->rel->notnullattrs) - && !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber, - unique_index->rel->notnullattrs)) + else if (!bms_is_member(attr, unique_index->rel->notnullattrs) && + !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber, + unique_index->rel->notnullattrs)) { *multi_nullvals = true; } exprs = lappend(exprs, expr); } + return exprs; } /* * add_uniquekey_for_onerow - * If we are sure that the relation only returns one row, then all the columns - * are unique. However we don't need to create UniqueKey for every column, we - * just set exprs = NIL and overwrites all the other UniqueKey on this RelOptInfo - * since this one has strongest semantics. + * Create a special unique key signifying that the rel has one row. + * + * If we are sure that the relation only returns one row (it might return + * no rows, but we still consider that unique), then all the columns are + * trivially unique. + * + * However we don't need to create UniqueKey with every column, we just + * set exprs = NIL, because that's easier to identify. We don't want to + * add unnecessary unique keys (such that we already have a unique key + * for a subset of the expressions), and with (exprs == NIL) we can just + * assume we have one unique key for each column in the rel. + * + * We discard all other unique keys, since it has the strongest semantics. */ void add_uniquekey_for_onerow(RelOptInfo *rel) { /* - * We overwrite the previous UniqueKey on purpose since this one has the - * strongest semantic. + * We overwrite the previous UniqueKey on purpose since this one has + * the strongest semantic (all other unique keys are implied by it). */ rel->uniquekeys = list_make1(makeUniqueKey(NIL, false)); } /* - * initililze_uniquecontext_for_joinrel - * Return a List of UniqueKeyContext for an inputrel + * initialize_uniquecontext_for_joinrel + * Return a List of UniqueKeyContext for an inputrel. */ static List * -initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel) +initialize_uniquecontext_for_joinrel(RelOptInfo *inputrel) { - List *res = NIL; - ListCell *lc; - foreach(lc, inputrel->uniquekeys) + List *res = NIL; + ListCell *lc; + + foreach(lc, inputrel->uniquekeys) { UniqueKeyContext context; + context = palloc(sizeof(struct UniqueKeyContextData)); context->uniquekey = lfirst_node(UniqueKey, lc); context->added_to_joinrel = false; context->useful = true; + res = lappend(res, context); } + return res; } - /* * get_exprs_from_uniquekey - * Unify the way of get List of exprs from a one-row UniqueKey or - * normal UniqueKey. for the onerow case, every expr in rel1 is a valid - * UniqueKey. Return a List of exprs. + * Extract expressions that are part of a unique key. + * + * The meaning of the result is a bit different in regular and one-row cases. + * For the regular case, the list of expressions form a single unique key, + * i.e. the combination of values is unique. + * + * For the one-row case, each individual expression is known to be unique + * (simply because in a single row everything is unique). * * rel1: The relation which you want to get the exprs. * ukey: The UniqueKey you want to get the exprs. @@ -875,27 +1193,29 @@ static List * get_exprs_from_uniquekey(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *rel1, UniqueKey *ukey) { - ListCell *lc; - bool onerow = rel1 != NULL && relation_is_onerow(rel1); + ListCell *lc; + List *res = NIL; + bool onerow = (rel1 != NULL) && relation_is_onerow(rel1); - List *res = NIL; + /* We require at least one of those to be true. */ Assert(onerow || ukey); - if (onerow) - { - /* Only cares about the exprs still exist in joinrel */ - foreach(lc, joinrel->reltarget->exprs) - { - Bitmapset *relids = pull_varnos(root, lfirst(lc)); - if (bms_is_subset(relids, rel1->relids)) - { - res = lappend(res, list_make1(lfirst(lc))); - } - } - } - else + + /* if not a one-row unique key, just return the key's expressions */ + if (!onerow) + return list_make1(ukey->exprs); + + /* + * If it's a one-row relation, we simply extract the expressions that + * still exist in the reltarget. + */ + foreach(lc, joinrel->reltarget->exprs) { - res = list_make1(ukey->exprs); + Bitmapset *relids = pull_varnos(root, lfirst(lc)); + + if (bms_is_subset(relids, rel1->relids)) + res = lappend(res, list_make1(lfirst(lc))); } + return res; } @@ -910,55 +1230,67 @@ get_exprs_from_uniquekey(PlannerInfo *root, RelOptInfo *joinrel, /* * index_constains_partkey - * return true if the index contains the partiton key. + * Determines if the index includes a partition key. + * + * XXX Surely we already have a code doing this already? E.g. when creating + * a unique index on a partitioned table we define that. */ static bool -index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind) +index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind) { ListCell *lc; int i; + Assert(IS_PARTITIONED_REL(partrel)); Assert(partrel->part_scheme->partnatts > 0); for(i = 0; i < partrel->part_scheme->partnatts; i++) { - Node *part_expr = linitial(partrel->partexprs[i]); - bool found_in_index = false; + Node *part_expr = linitial(partrel->partexprs[i]); + bool found_in_index = false; + foreach(lc, ind->indextlist) { - Expr *index_expr = lfirst_node(TargetEntry, lc)->expr; + Expr *index_expr = lfirst_node(TargetEntry, lc)->expr; + if (equal(index_expr, part_expr)) { found_in_index = true; break; } } + if (!found_in_index) return false; } + return true; } /* * simple_indexinfo_equal + * Compare two indexes to determine if they are the same. + * + * We need to do this because simple_copy_indexinfo_to_parent does change + * some elements. So this is not exactly the same as calling equal(). * - * Used to check if the 2 index is same as each other. The index here - * is COPIED from childrel and did some tiny changes(see - * simple_copy_indexinfo_to_parent) + * XXX I wonder if we could simply use equal(), somehow? In fact, we should + * probably build something much simpler than IndexOptInfo, just enough to + * do the checks. */ static bool simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2) { Size oid_cmp_len = sizeof(Oid) * ind1->ncolumns; - return ind1->ncolumns == ind2->ncolumns && - ind1->unique == ind2->unique && - memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0 && - memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0 && - memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0 && - memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0 && - equal(get_tlist_exprs(ind1->indextlist, true), - get_tlist_exprs(ind2->indextlist, true)); + return ((ind1->ncolumns == ind2->ncolumns) && + (ind1->unique == ind2->unique) && + (memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0) && + (memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0) && + (memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0) && + (memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0) && + (equal(get_tlist_exprs(ind1->indextlist, true), + get_tlist_exprs(ind2->indextlist, true)))); } @@ -981,11 +1313,21 @@ simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2) /* - * simple_copy_indexinfo_to_parent (from partition) - * Copy the IndexInfo from child relation to parent relation with some modification, - * which is used to test: - * 1. If the same index exists in all the childrels. + * simple_copy_indexinfo_to_parent + * Copy index info from child to parent, with necessary tweaks. + * + * We use this copy to check: + * + * 1. If the same/matching index exists in all the childrels. * 2. If the parentrel->reltarget/basicrestrict info matches this index. + * + * XXX IMHO we should probably build something much simpler than a full + * IndexOptInfo copy, just enough to do the checks. + * + * XXX The fact that we copy so much data seems wrong, and having to + * define macros from copyfuncs.c seems like a very suspicious thing. + * One reason is that IndeOptInfo is fairly large struct, especially + * with all the fields, and we allocate it very often. */ static IndexOptInfo * simple_copy_indexinfo_to_parent(PlannerInfo *root, @@ -1027,20 +1369,24 @@ simple_copy_indexinfo_to_parent(PlannerInfo *root, /* * adjust_partition_unique_indexlist + * Checks and eliminates indexes that do not exist on the child relation. * - * global_unique_indexes: At the beginning, it contains the copy & modified - * unique index from the first partition. And then check if each index in it still - * exists in the following partitions. If no, remove it. at last, it has an - * index list which exists in all the partitions. + * Walks the list of unique indexes, and eliminates those that don't match + * the child relation (i.e. where a matching child index does not exist). + * This is used to iteratively filter the list of candidate unique keys. + * + * After processing all child relations, the list contains only indexes that + * exist in all the child relations. */ static void adjust_partition_unique_indexlist(PlannerInfo *root, RelOptInfo *parentrel, RelOptInfo *childrel, - List **global_unique_indexes) + List **indexes) { ListCell *lc, *lc2; - foreach(lc, *global_unique_indexes) + + foreach(lc, *indexes) { IndexOptInfo *g_ind = lfirst_node(IndexOptInfo, lc); bool found_in_child = false; @@ -1049,23 +1395,45 @@ adjust_partition_unique_indexlist(PlannerInfo *root, { IndexOptInfo *p_ind = lfirst_node(IndexOptInfo, lc2); IndexOptInfo *p_ind_copy; - if (!p_ind->unique || !p_ind->immediate || - (p_ind->indpred != NIL && !p_ind->predOK)) + + /* + * Ignore child indexes that can't possibly match (not unique or + * immediate, etc.) + * + * XXX We do these checks in many places, so maybe turn it into + * a reusable macro? + */ + if ((!p_ind->unique) || (!p_ind->immediate) || + (p_ind->indpred != NIL) && (!p_ind->predOK)) continue; + + /* + * XXX This seems possibly quite expensive. Imagine there are many + * child relations, with a bunch of unique indexes each. Then this + * generates a copy for each unique index in each child relation, + * something like O(N^2/2) copies. + */ p_ind_copy = simple_copy_indexinfo_to_parent(root, parentrel, p_ind); + + /* Found a matching index for the child relation, we're done. */ if (simple_indexinfo_equal(p_ind_copy, g_ind)) { found_in_child = true; break; } } + + /* No matching index in the child, so remove it from the list. */ if (!found_in_child) - /* The index doesn't exist in childrel, remove it from global_unique_indexes */ - *global_unique_indexes = foreach_delete_current(*global_unique_indexes, lc); + *indexes = foreach_delete_current(*indexes, lc); } } -/* Helper function for groupres/distinctrel */ +/* + * Helper function for groupres/distinctrel + * + * FIXME Not sure about this. + */ static void add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgroups) { @@ -1073,27 +1441,32 @@ add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgrou List *exprs; /* - * XXX: If there are some vars which is not in current levelsup, the semantic is - * imprecise, should we avoid it or not? levelsup = 1 is just a demo, maybe we need to - * check every level other than 0, if so, looks we have to write another - * pull_var_walker. + * XXX: If there are some vars which are not in the current levelsup, the + * semantic is imprecise, should we avoid it or not? levelsup = 1 is just + * a demo, maybe we need to check every level other than 0, if so, looks + * we have to write another pull_var_walker. */ List *upper_vars = pull_vars_of_level((Node*)sortgroups, 1); if (upper_vars != NIL) return; + /* sortgroupclause can't be multi_nullvals */ exprs = get_sortgrouplist_exprs(sortgroups, parse->targetList); rel->uniquekeys = lappend(rel->uniquekeys, - makeUniqueKey(exprs, - false /* sortgroupclause can't be multi_nullvals */)); + makeUniqueKey(exprs, false)); } /* * add_combined_uniquekey - * The combination of both UniqueKeys is a valid UniqueKey for joinrel no matter - * the jointype. + * Add a unique key for a join, combined from keys on inner/outer side. + * + * The combination of both UniqueKeys is a valid UniqueKey for joinrel no + * matter what's the exact jointype. + * + * Returns true if the unique key is "one-row" variant, so that the caller + * can stop considering further combinations. */ bool add_combined_uniquekey(PlannerInfo *root, @@ -1104,32 +1477,47 @@ add_combined_uniquekey(PlannerInfo *root, UniqueKey *inner_ukey, JoinType jointype) { + bool multi_nullvals; + ListCell *lc1, *lc2; - ListCell *lc1, *lc2; - - /* Either side has multi_nullvals or we have outer join, - * the combined UniqueKey has multi_nullvals */ - bool multi_nullvals = outer_ukey->multi_nullvals || + /* + * If either side has multi_nullvals, or we are dealing with an outer join, + * the combined UniqueKey has multi_nullvals too. + */ + multi_nullvals = outer_ukey->multi_nullvals || inner_ukey->multi_nullvals || IS_OUTER_JOIN(jointype); /* The only case we can get onerow joinrel after join */ - if (relation_is_onerow(outer_rel) - && relation_is_onerow(inner_rel) - && jointype == JOIN_INNER) + if (relation_is_onerow(outer_rel) && + relation_is_onerow(inner_rel) && + jointype == JOIN_INNER) { add_uniquekey_for_onerow(joinrel); return true; } + /* + * XXX Isn't this wrong? Why is it combining expressions that are part + * of the two unique keys? Imagine we have outer unique key on (a1, a2) + * and inner outer key on (b1, b2). Then this adds four unique keys + * for the join (a1,b1), (a1,b2), (a2,b1) and (a2,b2). Shouldn't it + * just add (a1,a2,b1,b2)? + */ foreach(lc1, get_exprs_from_uniquekey(root, joinrel, outer_rel, outer_ukey)) { + /* + * XXX This calls get_exprs_from_uniquekey repeatedly for each outer + * loop. Maybe we should calculate it just once before the loop. + */ foreach(lc2, get_exprs_from_uniquekey(root, joinrel, inner_rel, inner_ukey)) { List *exprs = list_concat_copy(lfirst_node(List, lc1), lfirst_node(List, lc2)); + joinrel->uniquekeys = lappend(joinrel->uniquekeys, makeUniqueKey(exprs, multi_nullvals)); } } + return false; } diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index 8d8e493f5c..f29b65c07b 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -2387,6 +2387,7 @@ grouping_planner(PlannerInfo *root, bool inheritance_update, add_path(final_rel, path); } + /* XXX comment? can we simply just copy the unique keys to the final relation? */ simple_copy_uniquekeys(current_rel, final_rel); /* @@ -3902,7 +3903,9 @@ create_grouping_paths(PlannerInfo *root, set_cheapest(grouped_rel); + /* XXX does this apply to grouping sets too? */ populate_grouprel_uniquekeys(root, grouped_rel, input_rel); + return grouped_rel; } @@ -4625,7 +4628,10 @@ create_window_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(window_rel); + + /* XXX comment? */ simple_copy_uniquekeys(input_rel, window_rel); + return window_rel; } @@ -4939,7 +4945,10 @@ create_distinct_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(distinct_rel); + + /* XXX comment */ populate_distinctrel_uniquekeys(root, input_rel, distinct_rel); + return distinct_rel; } @@ -5200,6 +5209,7 @@ create_ordered_paths(PlannerInfo *root, */ Assert(ordered_rel->pathlist != NIL); + /* XXX comment */ simple_copy_uniquekeys(input_rel, ordered_rel); return ordered_rel; diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c index b7626545bf..72a3f3c598 100644 --- a/src/backend/optimizer/prep/prepunion.c +++ b/src/backend/optimizer/prep/prepunion.c @@ -691,6 +691,7 @@ generate_union_paths(SetOperationStmt *op, PlannerInfo *root, /* Add the UniqueKeys */ populate_unionrel_uniquekeys(root, result_rel); + return result_rel; } diff --git a/src/backend/optimizer/util/inherit.c b/src/backend/optimizer/util/inherit.c index 3eec1f4d74..c9829c5fc4 100644 --- a/src/backend/optimizer/util/inherit.c +++ b/src/backend/optimizer/util/inherit.c @@ -755,6 +755,7 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, pseudoconstant, rinfo->security_level, NULL, NULL, NULL); + /* XXX This is a bit weird, doing this outside make_restrictinfo */ child_rinfo->mergeopfamilies = rinfo->mergeopfamilies; childquals = lappend(childquals, child_rinfo); /* track minimum security level among child quals */ -- 2.30.2 --------------F495F6B18672582269F00FF9 Content-Type: text/x-patch; charset=UTF-8; name="0005-Extend-UniqueKeys-20210317.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0005-Extend-UniqueKeys-20210317.patch" ^ permalink raw reply [nested|flat] 56+ messages in thread
* [PATCH 04/10] review @ 2021-03-16 18:26 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 56+ messages in thread From: Tomas Vondra @ 2021-03-16 18:26 UTC (permalink / raw) --- src/backend/nodes/copyfuncs.c | 4 + src/backend/nodes/makefuncs.c | 1 - src/backend/nodes/outfuncs.c | 1 + src/backend/nodes/readfuncs.c | 2 + src/backend/optimizer/path/README.uniquekey | 285 +++--- src/backend/optimizer/path/allpaths.c | 10 +- src/backend/optimizer/path/joinrels.c | 7 + src/backend/optimizer/path/uniquekeys.c | 906 ++++++++++++++------ src/backend/optimizer/plan/planner.c | 10 + src/backend/optimizer/prep/prepunion.c | 1 + src/backend/optimizer/util/inherit.c | 1 + 11 files changed, 863 insertions(+), 365 deletions(-) diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 75c1c5e824..9d832ddc03 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2296,6 +2296,9 @@ _copyPathKey(const PathKey *from) return newnode; } +/* + * _copyUniqueKey + */ static UniqueKey * _copyUniqueKey(const UniqueKey *from) { @@ -2306,6 +2309,7 @@ _copyUniqueKey(const UniqueKey *from) return newnode; } + /* * _copyRestrictInfo */ diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 40415d0f5b..e156c9cdf8 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -816,7 +816,6 @@ makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols) return v; } - /* * makeUniqueKey */ diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 44154cde6a..13905e6037 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2460,6 +2460,7 @@ static void _outUniqueKey(StringInfo str, const UniqueKey *node) { WRITE_NODE_TYPE("UNIQUEKEY"); + WRITE_NODE_FIELD(exprs); WRITE_BOOL_FIELD(multi_nullvals); } diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index b3e212bf1c..8830c8df99 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -496,8 +496,10 @@ static UniqueKey * _readUniqueKey(void) { READ_LOCALS(UniqueKey); + READ_NODE_FIELD(exprs); READ_BOOL_FIELD(multi_nullvals); + READ_DONE(); } diff --git a/src/backend/optimizer/path/README.uniquekey b/src/backend/optimizer/path/README.uniquekey index 5eac761995..31cdb5ed65 100644 --- a/src/backend/optimizer/path/README.uniquekey +++ b/src/backend/optimizer/path/README.uniquekey @@ -1,131 +1,208 @@ -1. What is UniqueKey? -We can think UniqueKey is a set of exprs for a RelOptInfo, which we are insure -that doesn't yields same result among all the rows. The simplest UniqueKey -format is primary key. +review comments: +XXX Maybe move this to src/backend/optimizer/README.uniquekey? +XXX multi_nullvals name seems a bit weird +XXX no info about populate_distinctrel_uniquekeys, populate_grouprel_uniquekeys, populate_unionrel_uniquekeys +----- +src/backend/optimizer/path/README.uniquekey + +UniqueKey +========= + +UniqueKey is a set of exprs for a RelOptInfo, which are known to have unique +values on all the rows in the relation. A trivial example is a primary key +defined on a relation - each attributes of the constraint is a unique key. + +We can use this knowledge to perform optimization in a number of places. Some +of the optimizations are fairly obvious, others are less so: + +1. remove DISTINCT node if the clause is unique +2. remove aggregation if group by clause is unique +3. remove_useless_joins +4. reduce_semianti_joins +5. Index Skip Scan (WIP) +6. Aggregation Push-Down without 2-phase aggregation if the join can't + duplicate the aggregated rows. (WIP) + -However we define the UnqiueKey as below. +UniqueKey struct +---------------- -typedef struct UniqueKey -{ +A UnqiueKey is represented by the following struct: + + typedef struct UniqueKey + { NodeTag type; List *exprs; bool multi_nullvals; -} UniqueKey; - -exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL -is a special case of UniqueKey, which means there is only one row in that -relation.it has a stronger semantic than others. like SELECT uk FROM t; uk is -normal unique key and may have different values. SELECT colx FROM t WHERE uk = -const. colx is unique AND we have only 1 value. This field can used for -innerrel_is_unique. this logic is handled specially in add_uniquekey_for_onerow -function. - -multi_nullvals: true means multi null values may exist in these exprs, so the -uniqueness is not guaranteed in this case. This field is necessary for -remove_useless_join & reduce_unique_semijoins where we don't mind these -duplicated NULL values. It is set to true for 2 cases. One is a unique key -from a unique index but the related column is nullable. The other one is for -outer join. see populate_joinrel_uniquekeys for detail. - - -The UniqueKey can be used at the following cases at least: -1. remove_useless_joins. -2. reduce_semianti_joins -3. remove distinct node if distinct clause is unique. -4. remove aggnode if group by clause is unique. -5. Index Skip Scan (WIP) -6. Aggregation Push Down without 2 phase aggregation if the join can't - duplicated the aggregated rows. (work in progress feature) - -2. How is it maintained? - -We have a set of populate_xxx_unqiuekeys functions to maintain the uniquekey on -various cases. xxx includes baserel, joinrel, partitionedrel, distinctrel, -groupedrel, unionrel. and we also need to convert the uniquekey from subquery -to outer relation, which is what convert_subquery_uniquekeys does. - -1. The first part is about baserel. We handled 3 cases. suppose we have Unique -Index on (a, b). - -1. SELECT a, b FROM t. UniqueKey (a, b) -2. SELECT a FROM t WHERE b = 1; UniqueKey (a) -3. SELECT .. FROM t WHERE a = 1 AND b = 1; UniqueKey (NIL). onerow case, every - column is Unique. - -2. The next part is joinrel, this part is most error-prone, we simplified the rules -like below: -1. If the relation's UniqueKey can't be duplicated after join, then is will be - still valid for the join rel. The function we used here is - innerrel_keeps_unique. The basic idea is innerrel.any_col = outer.uk. - -2. If the UnqiueKey can't keep valid via the rule 1, the combination of the - UniqueKey from both sides are valid for sure. We can prove this as: if the - unique exprs from rel1 is duplicated by rel2, the duplicated rows must - contains different unique exprs from rel2. - -More considerations about onerow: -1. If relation with one row and it can't be duplicated, it is still possible - contains mulit_nullvas after outer join. -2. If the either UniqueKey can be duplicated after join, the can get one row - only when both side is one row AND there is no outer join. -3. Whenever the onerow UniqueKey is not a valid any more, we need to convert one - row UniqueKey to normal unique key since we don't store exprs for one-row - relation. get_exprs_from_uniquekeys will be used here. - - -More considerations about multi_nullvals after join: + } UniqueKey; + +exprs is a list of exprs which are know to be unique on current RelOptInfo. + +exprs = NIL is a special case, meaning there is only one row in the relation. +This has has a stronger semantic than others. Consider for example + + SELECT uk FROM t + +where 'uk' is a unique key. This guarantees uniqueness, but there may be mamy +rows in the relation. On the other hand, consider this query + + SELECT colx FROM t WHERE uk = const + +In this case we know there's only a single matching row (thanks to a condition +on the unique key), which in turn guarantees uniqueness of the colx value, even +if there is no constraint on the column itself. + +This knowledge is used in innerrel_is_unique, and is handled as a special case +in add_uniquekey_for_onerow. + + +The multi_nullvals field tracks whether the expressions may contain multiple +NULL values. This can happen for example when the unique key is derived from +a unique index with nullable columns, or because of outer joins (which may add +NULL values to a known-unique list - see populate_joinrel_uniquekeys). + +In this case uniqueness is not guaranteed, but we can still use the information +in places places where NULL values are harmless - when removing useless joins, +reducing semijoins, and so on. + + +How is it maintained? +--------------------- + +Deducing the unique keys depends on the type of the relation - for each case +there's a separate "populate" function: + + +populate_baserel_uniquekeys +--------------------------- + +There are three cases, all assuming there's a unique index (e.g. on (a,b)): + +1. SELECT a, b FROM t => UniqueKey (a, b) +2. SELECT a FROM t WHERE b = 1 => UniqueKey (a) +3. SELECT .. FROM t WHERE a = 1 AND b = 1; => UniqueKey (NIL) + +The last query is the "one row" case, in which case every column is Unique. + + +populate_joinrel_uniquekeys +--------------------------- + +For joins, deducing the unique keys may be fairly complex and error-prone. +We've simplified the rules like this: + +1. If the UniqueKey on an input relation can't be duplicated by the join, then +it will be valid for the join rel. A typical example is a join like this: + + inner_rel.any_col = outer_rel.unique_key + +The function used to detect this is innerrel_keeps_unique. + +2. Any combination of unique keys on each side of the join is a unique key +for the join relation. This can be proved by contradiction - assume we have +unique key on either side of the join - uk1 and uk2. If the values in uk1 get +duplicated by the join with uk2 (by matching the row to multiple rows), the +duplicated rows must have different values in the uk2. + +We can also leverage information about the "one row" case: + +1. If one of the input relations is known to have a single row, and the join +can't duplicate the row (e.g. semi/anti join), we can keep the unique keys. +It may however contain multi_nullvals after an outer join. + +XXX Not sure I understand the original logic/wording :-( + +2. If either UniqueKey can be duplicated after a join, there can be only one +row only when both sides are "one row" AND there is no outer join. + +XXX Why the restriction on not allowing outer joins? + +3. Whenever the one row UniqueKey is not a valid any more, we need to convert +UniqueKey to normal unique key since we don't store exprs for one-row relation. +This is done by get_exprs_from_uniquekeys. + +The join case needs to be careful about multi_nullvals too: + 1. If the original UnqiueKey has multi_nullvals, the final UniqueKey will have - mulit_nullvals in any case. -2. If a unique key doesn't allow mulit_nullvals, after some outer join, it - allows some outer join. +mulit_nullvals in any case too. + +2. If the original unique key doesn't allow multi_nullvals, the unique key for +the join relation may allow multi_nullvals after an outer join. + + +subqueries +---------- + +It's necessary to "translate" unique keys between a subquery and the outer rels, +which is what convert_subquery_uniquekeys does. This does almost exactly what +convert_subquery_pathkeys does for pathkeys. It keeps only unique keys matching +Vars in the outer relation. The relationship between outerrel.Var and +subquery.exprs is built from outerel->subroot->processed_tlist. -3. When we comes to subquery, we need to convert_subquery_unqiuekeys just like -convert_subquery_pathkeys. Only the UniqueKey insides subquery is referenced as -a Var in outer relation will be reused. The relationship between the outerrel.Var -and subquery.exprs is built with outerel->subroot->processed_tlist. +set-returning functions +------------------------ +As for the SRF functions, it will break the uniqueness of uniquekey, However it +is handled in adjust_paths_for_srfs, which happens after the query_planner. So +we will maintain the UniqueKey until there and reset it to NIL at that place. -4. As for the SRF functions, it will break the uniqueness of uniquekey, However it -is handled in adjust_paths_for_srfs, which happens after the query_planner. so -we will maintain the UniqueKey until there and reset it to NIL at that -places. This can't help on distinct/group by elimination cases but probably help -in some other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is -semantic correctly. +This can't help on distinct/group by elimination cases but probably help in some +other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is correct. -5. As for inherit table, we first main the UnqiueKey on childrel as well. But for -partitioned table we need to maintain 2 different kinds of -UnqiueKey. 1). UniqueKey on the parent relation 2). UniqueKey on child -relation for partition wise query. +populate_partitionedrel_uniquekeys +---------------------------------- + +As for inherit table, we first build the UnqiueKey on childrel as well. But for +partitioned table we need to maintain two different kinds of UniqueKey: + +1) UniqueKey on the parent relation + +2) UniqueKey on child + +This is needed because a unique key from the partition may not be be unique key +on the partitioned table. + Example: -CREATE TABLE p (a int not null, b int not null) partition by list (a); + +CREATE TABLE p (a INT NOT NULL, b INT NOT NULL) PARTITION BY LIST (a); + CREATE TABLE p0 partition of p for values in (1); CREATE TABLE p1 partition of p for values in (2); -create unique index p0_b on p0(b); -create unique index p1_b on p1(b); +CREATE UNIQUE INDEX p0_b ON p0(b); +CREATE UNIQUE INDEX p1_b ON p1(b); -Now b is only unique on partition level, so the distinct can't be removed on -the following cases. SELECT DISTINCT b FROM p; +SELECT DISTINCT b FROM p; -Another example is SELECT DISTINCT a, b FROM p WHERE a = 1; Since only one -partition is chosen, the UniqueKey on child relation is same as the UniqueKey on -parent relation. +Now "b" is only unique on partition level, but the two partitions may contain +duplicate values for the "b" column (with different values in "a"). That means +the DISTINCT clause can't be removed. -Another usage of UniqueKey on partition level is it be helpful for -partition-wise join. +Now consider: -As for the UniqueKey on parent table level, it comes with 2 different ways, -1). the UniqueKey is also derived in UniqueKey index, but the index must be same -in all the related children relations and the unique index must contains -Partition Key in it. Example: +SELECT DISTINCT a, b FROM p WHERE a = 1 + +In this case, the optimizer eliminates all partitions except for one, so that +the UniqueKey is valid for the parent relation too. + +UniqueKey at a partition level is useful for partition-wise join too. + +XXX Explain why is it useful? + +A UniqueKey from a partition can be transferred to the parent relation, in two +cases. A trivial case is if there's a single child relation (e.g. thanks to +partition elimination). In that case all unique keys on the child relation are +automatically valid for the parent relation. If there are multiple relations, +the unique key must be defived from an index present in all partitions, and the +index has to include the partition key. + +Example: CREATE UNIQUE INDEX p_ab ON p(a, b); -- where a is the partition key. -- Query SELECT a, b FROM p; the (a, b) is a UniqueKey of p. -2). If the parent relation has only one childrel, the UniqueKey on childrel is - the UniqueKey on parent as well. diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index 66bf6f19f7..a801707eaa 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -581,7 +581,8 @@ set_plain_rel_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) /* * Now that we've marked which partial indexes are suitable, we can now - * build the relation's unique keys. + * build the relation's unique keys. We need to do it in this order, + * so that we don't deduce unique keys from inapplicable partial indexes. */ populate_baserel_uniquekeys(root, rel, rel->indexlist); @@ -1305,6 +1306,12 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, /* Add paths to the append relation. */ add_paths_to_append_rel(root, rel, live_childrels); + + /* + * XXX Maybe move the check into populate populate_partitionedrel_uniquekeys? + * XXX What if it's append rel (but not partitioned one), but there's only one + * child relation? We could still deduce unique keys, no? + */ if (IS_PARTITIONED_REL(rel)) populate_partitionedrel_uniquekeys(root, rel, live_childrels); } @@ -2314,6 +2321,7 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, pathkeys, required_outer)); } + /* Convert subpath's unique keys to outer representation */ convert_subquery_uniquekeys(root, rel, sub_final_rel); /* If outer rel allows parallelism, do same for partial paths. */ diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c index 7271f044ec..eefba449d6 100644 --- a/src/backend/optimizer/path/joinrels.c +++ b/src/backend/optimizer/path/joinrels.c @@ -925,6 +925,13 @@ populate_joinrel_with_paths(PlannerInfo *root, RelOptInfo *rel1, /* Apply partitionwise join technique, if possible. */ try_partitionwise_join(root, rel1, rel2, joinrel, sjinfo, restrictlist); + /* + * Determine which of the unique keys from input relations are applicable + * for the join result. + * + * XXX We do this after trying the partitionwise join, because that may allow + * using additional unique keys. + */ populate_joinrel_uniquekeys(root, joinrel, rel1, rel2, restrictlist, sjinfo->jointype); } diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c index 77ed2b2eff..114e8334f5 100644 --- a/src/backend/optimizer/path/uniquekeys.c +++ b/src/backend/optimizer/path/uniquekeys.c @@ -36,7 +36,7 @@ typedef struct UniqueKeyContextData bool useful; } *UniqueKeyContext; -static List *initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel); +static List *initialize_uniquecontext_for_joinrel(RelOptInfo *inputrel); static bool innerrel_keeps_unique(PlannerInfo *root, RelOptInfo *outerrel, RelOptInfo *innerrel, @@ -80,8 +80,20 @@ static void add_uniquekey_from_sortgroups(PlannerInfo *root, /* * populate_baserel_uniquekeys - * Populate 'baserel' uniquekeys list by looking at the rel's unique index - * and baserestrictinfo + * Build list of unique keys for the base relation. + * + * Inspects unique indexes defined on the relation and determines what + * unique keys are valid. Partial indexes are considered too, if the + * predicate is valid. + * + * This also inspects baserestrictinfo, because we need to determine + * which opclass families are interesting when inspecting indexes. If we + * have a unique index and distinct clause with a mismatching opclasses, + * we should not use that. + * + * XXX Why does this look at baserestrictinfo? + * + * XXX What about collations? */ void populate_baserel_uniquekeys(PlannerInfo *root, @@ -99,22 +111,48 @@ populate_baserel_uniquekeys(PlannerInfo *root, Assert(baserel->rtekind == RTE_RELATION); + if (!indexlist) + return; + + /* + * Determine which unique indexes to use to build the unique keys. + * We have to skip partial with predicates not matched by the query, + * and unique indexes that are not immediately enforced. + * + * XXX Do we actually skip indexes that are not immediate? + * XXX What about hypothetical indexes? + */ foreach(lc, indexlist) { IndexOptInfo *ind = (IndexOptInfo *) lfirst(lc); + if (!ind->unique || !ind->immediate || (ind->indpred != NIL && !ind->predOK)) continue; + matched_uniq_indexes = lappend(matched_uniq_indexes, ind); } + /* If there are not applicable unique indexes, we're done. */ if (matched_uniq_indexes == NIL) return; - /* Check which attrs is used in baserel->reltarget */ - pull_varattnos((Node *)baserel->reltarget->exprs, baserel->relid, &used_attrs); + /* + * Determine which attrs are referenced in baserel->reltarget. To use the + * unique key info, we need all the columns - a unique index on (a,b) may + * not be unique on (a). If a column is missing in reltarget, the nodes + * above can't possibly use it, and we can just ignore any matching index. + */ + pull_varattnos((Node *) baserel->reltarget->exprs, baserel->relid, &used_attrs); - /* Check which attrno is used at a mergeable const filter */ + /* + * Check which attrno is used at a mergeable const filter + * + * XXX This is not lookint att attrno at all, maybe obsolete comment? + * + * Seems the primary purpose of this is determining which opclass + * families to use when matching unique indexes in the next loop? + */ foreach(lc, baserel->baserestrictinfo) { RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); @@ -122,6 +160,10 @@ populate_baserel_uniquekeys(PlannerInfo *root, if (rinfo->mergeopfamilies == NIL) continue; + /* + * XXX What if bms_is_empty is true for both left_relids/right_relids? + * Or what if it's false in both cases? + */ if (bms_is_empty(rinfo->left_relids)) { const_exprs = lappend(const_exprs, get_rightop(rinfo->clause)); @@ -136,40 +178,69 @@ populate_baserel_uniquekeys(PlannerInfo *root, expr_opfamilies = lappend(expr_opfamilies, rinfo->mergeopfamilies); } + /* + * Now try to match unique indexes to attributes in reltarget, and to + * merge operator families. The index may be on the right attributes, + * but if it's not matching the opfamily it's useless. + * + * XXX Can we have multiple baserestrictinfo for the same attribute, + * with different opfamilies? Probably not. + */ foreach(lc, matched_uniq_indexes) { - bool multi_nullvals, useful; - List *exprs = get_exprs_from_uniqueindex(lfirst_node(IndexOptInfo, lc), - const_exprs, - expr_opfamilies, - used_attrs, - &useful, - &multi_nullvals); - if (useful) + bool multi_nullvals, + useful; + + IndexOptInfo *index_info = (IndexOptInfo *) lfirst_node(IndexOptInfo, lc); + + List *exprs = get_exprs_from_uniqueindex(index_info, + const_exprs, + expr_opfamilies, + used_attrs, + &useful, + &multi_nullvals); + + if (!useful) + continue; + + /* + * All the columns in Unique Index matched with a restrictinfo, so + * that we know there's just a one row in the result. If we find + * such index, we're done - we discard all other unique keys and + * keep just this special one. In principle, this is a stronger + * guarantee, because all subsets of one row are still unique. + * + * XXX Is it correct to just return? Doesn't that prevent some + * optimizations that might be possible with the other keys? + */ + if (exprs == NIL) { - if (exprs == NIL) - { - /* All the columns in Unique Index matched with a restrictinfo */ - add_uniquekey_for_onerow(baserel); - return; - } - baserel->uniquekeys = lappend(baserel->uniquekeys, - makeUniqueKey(exprs, multi_nullvals)); + /* discards all previous uniquekeys */ + add_uniquekey_for_onerow(baserel); + return; } + + baserel->uniquekeys = lappend(baserel->uniquekeys, + makeUniqueKey(exprs, multi_nullvals)); } } /* * populate_partitionedrel_uniquekeys - * The UniqueKey on partitionrel comes from 2 cases: - * 1). Only one partition is involved in this query, the unique key can be - * copied to parent rel from childrel. - * 2). There are some unique index which includes partition key and exists - * in all the related partitions. - * We never mind rule 2 if we hit rule 1. + * Determine unique keys for a partitioned relation. + * + * Inspects unique keys for all partitions and derives unique keys that + * are valid for the whole partitioned table. There are two basic cases: + * + * 1) There's only one remaining partition (thanks to pruning all other + * partitions). In this case all the unique keys from the partition are + * trivially valid for the partitioned table. + * + * 2) All the partitions have the same unique index (on the same set of + * columns), and the index includes the partition key. This ensures the + * combination of values is unique for the whole partitioned table. */ - void populate_partitionedrel_uniquekeys(PlannerInfo *root, RelOptInfo *rel, @@ -180,110 +251,181 @@ populate_partitionedrel_uniquekeys(PlannerInfo *root, RelOptInfo *childrel; bool is_first = true; + /* XXX What about append rels? At least for the one-child case? */ Assert(IS_PARTITIONED_REL(rel)); + /* if there are no child relations, we're done. */ if (childrels == NIL) return; /* - * If there is only one partition used in this query, the UniqueKey in childrel is - * still valid in parent level, but we need convert the format from child expr to - * parent expr. + * If there is only one partition used in this query, the UniqueKey for + * a child relation is still valid for the parent level. We need to + * convert the format from child expr to parent expr. */ if (list_length(childrels) == 1) { - /* Check for Rule 1 */ RelOptInfo *childrel = linitial_node(RelOptInfo, childrels); ListCell *lc; + Assert(childrel->reloptkind == RELOPT_OTHER_MEMBER_REL); + + /* If the partition has a single row, so does the parent. */ if (relation_is_onerow(childrel)) { add_uniquekey_for_onerow(rel); return; } + /* + * Inspect the unique keys one by one, try reusing them for the + * parent relation. + * + * FIXME This needs more work to handle expressions and not just + * simple Vars. + */ foreach(lc, childrel->uniquekeys) { + ListCell *lc2; + List *parent_exprs = NIL; + bool can_reuse = true; + UniqueKey *ukey = lfirst_node(UniqueKey, lc); AppendRelInfo *appinfo = find_appinfo_by_child(root, childrel->relid); - List *parent_exprs = NIL; - bool can_reuse = true; - ListCell *lc2; + + /* + * XXX Not sure what exactly we do here. Surely we deal with + * expressions at child/parent level elsewhere? Can't we just + * copy the code from there? + */ foreach(lc2, ukey->exprs) { - Var *var = (Var *)lfirst(lc2); + Var *var = (Var *) lfirst(lc2); + /* - * If the expr comes from a expression, it is hard to build the expression - * in parent so ignore that case for now. + * XXX For now this only supports simple Var expressions, + * so if there's a more complex expression we'll not copy + * the unique key to the parent. */ if(!IsA(var, Var)) { can_reuse = false; break; } + /* Convert it to parent var */ - parent_exprs = lappend(parent_exprs, find_parent_var(appinfo, var)); + parent_exprs = lappend(parent_exprs, + find_parent_var(appinfo, var)); } - if (can_reuse) - rel->uniquekeys = lappend(rel->uniquekeys, - makeUniqueKey(parent_exprs, - ukey->multi_nullvals)); + + /* ignore unique keys with complex expressions */ + if (!can_reuse) + continue; + + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(parent_exprs, + ukey->multi_nullvals)); } + + return; } - else + + /* + * A parent with multiple child relations. We only care about indexes that + * are in all child relations, so we loop through indexes on the first one + * and check that they exist in the other child relations too. + */ + + childrel = linitial_node(RelOptInfo, childrels); + foreach(lc, childrel->indexlist) { - /* Check for rule 2 */ - childrel = linitial_node(RelOptInfo, childrels); - foreach(lc, childrel->indexlist) - { - IndexOptInfo *ind = lfirst(lc); - IndexOptInfo *modified_index; - if (!ind->unique || !ind->immediate || - (ind->indpred != NIL && !ind->predOK)) - continue; + IndexOptInfo *ind = lfirst(lc); + IndexOptInfo *modified_index; - /* - * During simple_copy_indexinfo_to_parent, we need to convert var from - * child var to parent var, index on expression is too complex to handle. - * so ignore it for now. - */ - if (ind->indexprs != NIL) - continue; + /* + * Ignore indexes that are not unique, immediately enforced. Partial + * indexes with mismatched predicate are useless too. + */ + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; - modified_index = simple_copy_indexinfo_to_parent(root, rel, ind); - /* - * If the unique index doesn't contain partkey, then it is unique - * on this partition only, so it is useless for us. - */ - if (!index_constains_partkey(rel, modified_index)) - continue; + /* + * During simple_copy_indexinfo_to_parent, we need to convert var from + * child var to parent var, index on expression is too complex to handle. + * so ignore it for now. + * + * FIXME We should support indexes on expressions. + */ + if (ind->indexprs != NIL) + continue; - global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index); - } + /* + * Adopt the index definition for the parent. + * + * XXX This seems rather weird. We're constructing "artificial" index + * for the partitioned table (kinda like a global index). Can't we + * just have some simpler struct representing it? + */ + modified_index = simple_copy_indexinfo_to_parent(root, rel, ind); + + /* + * If the unique index doesn't contain partkey, then it is unique + * on this partition only, so it is useless for us. + * + * XXX Can't we do this check before simple_copy_indexinfo_to_parent? + */ + if (!index_constains_partkey(rel, modified_index)) + continue; - if (global_uniq_indexlist != NIL) + global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index); + } + + /* if there are no applicable unique indexes, we're done */ + if (!global_uniq_indexlist) + return; + + /* + * We iterate over the child relations first, and inspect the unique + * indexes for each hild, because this way we can stop early if we + * happen to eliminate all the unique indexes. + */ + foreach(lc, childrels) + { + RelOptInfo *child = lfirst(lc); + + /* skip the first index, which is where we got the list from */ + if (is_first) { - foreach(lc, childrels) - { - RelOptInfo *child = lfirst(lc); - if (is_first) - { - is_first = false; - continue; - } - adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist); - } - /* Now we have a list of unique index which are exactly same on all childrels, - * Set the UniqueKey just like it is non-partition table - */ - populate_baserel_uniquekeys(root, rel, global_uniq_indexlist); + is_first = false; + continue; } + + /* match the unique keys to indexes on this child */ + adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist); + + /* + * If we have eliminated all unique indexes, no point in looking at + * the remaining child relations. + */ + if (!global_uniq_indexlist) + break; } + + /* Now we have a list of unique index which are exactly same on all child + * relations. Set the UniqueKey just like it is non-partition table. + */ + populate_baserel_uniquekeys(root, rel, global_uniq_indexlist); } /* * populate_distinctrel_uniquekeys + * Update unique keys for relation produced by DISTINCT. + * + * We can keep all unique keys from the input relations, because DISTINCT + * can only remove rows - it can't duplicate them. Also, the DISTINCT clause + * itself is a unique key, so add that. */ void populate_distinctrel_uniquekeys(PlannerInfo *root, @@ -292,11 +434,13 @@ populate_distinctrel_uniquekeys(PlannerInfo *root, { /* The unique key before the distinct is still valid. */ distinctrel->uniquekeys = list_copy(inputrel->uniquekeys); + add_uniquekey_from_sortgroups(root, distinctrel, root->parse->distinctClause); } /* * populate_grouprel_uniquekeys + * */ void populate_grouprel_uniquekeys(PlannerInfo *root, @@ -305,54 +449,76 @@ populate_grouprel_uniquekeys(PlannerInfo *root, { Query *parse = root->parse; - bool input_ukey_added = false; ListCell *lc; + /* + * XXX Is this actually valid, before checking fro grouping sets? + * The grouping sets may produce duplicate row even with just a single + * input row, I think. + */ if (relation_is_onerow(inputrel)) { add_uniquekey_for_onerow(grouprel); return; } + + /* + * Bail out if there are grouping sets. + * + * XXX Could we maybe inspect the grouping sets and determine if this + * generates distinct combinations? In some cases that's clearly not + * the case (rollup, cube), but for some simple cases it might. + */ if (parse->groupingSets) return; - /* A Normal group by without grouping set. */ - if (parse->groupClause) + /* It has aggregation but without a group by, so only one row returned */ + if (!parse->groupClause) + add_uniquekey_for_onerow(grouprel); + + /* + * A regular group by, without grouping sets. + * + * Obviously, the whole group clause determines a unique key. But if + * there are smaller unique keys on the input rel, we prefer those + * because those are more flexible. If (a,b) is unique, (a,b,c) is + * unique too. Only when there are no such smaller unique keys, we + * add the unique key derived from the group clause. + */ + foreach(lc, inputrel->uniquekeys) { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + /* - * Current even the groupby clause is Unique already, but if query has aggref - * We have to create grouprel still. To keep the UnqiueKey short, we will check - * the UniqueKey of input_rel still valid, if so we reuse it. + * Ignore unique keys on the input that are not subset of the + * group clause. We can't use incomplete unique keys. */ - foreach(lc, inputrel->uniquekeys) - { - UniqueKey *ukey = lfirst_node(UniqueKey, lc); - if (list_is_subset(ukey->exprs, grouprel->reltarget->exprs)) - { - grouprel->uniquekeys = lappend(grouprel->uniquekeys, - ukey); - input_ukey_added = true; - } - } - if (!input_ukey_added) - /* - * group by clause must be a super-set of grouprel->reltarget->exprs except the - * aggregation expr, so if such exprs is unique already, no bother to generate - * new uniquekey for group by exprs. - */ - add_uniquekey_from_sortgroups(root, - grouprel, - root->parse->groupClause); + if (!list_is_subset(ukey->exprs, grouprel->reltarget->exprs)) + continue; + + grouprel->uniquekeys = lappend(grouprel->uniquekeys, ukey); } - else - /* It has aggregation but without a group by, so only one row returned */ - add_uniquekey_for_onerow(grouprel); + + /* + * Group clause must be a super-set of of grouprel->reltarget->exprs, + * except for the aggregation expressions. So if we found a smaller + * unique key on the input relation, don't bother adding a unique key + * for the group clause. + */ + if (!grouprel->uniquekeys) + add_uniquekey_from_sortgroups(root, + grouprel, + root->parse->groupClause); } /* * simple_copy_uniquekeys - * Using a function for the one-line code makes us easy to check where we simply - * copied the uniquekey. + * Copy yhe unique keys between relations. + * + * Using a function for the one-line code makes us easy to check where we + * simply copied the uniquekey. + * + * XXX Seems like an overkill, not sure what's the purpose? */ void simple_copy_uniquekeys(RelOptInfo *oldrel, @@ -362,24 +528,27 @@ simple_copy_uniquekeys(RelOptInfo *oldrel, } /* - * populate_unionrel_uniquekeys + * populate_unionrel_uniquekeys + * Determine unique keys for UNION relation. + * + * XXX Does this need to care about UNION vs. UNION ALL? At least in the + * one-row code path? */ void populate_unionrel_uniquekeys(PlannerInfo *root, - RelOptInfo *unionrel) + RelOptInfo *unionrel) { - ListCell *lc; - List *exprs = NIL; + ListCell *lc; + List *exprs = NIL; Assert(unionrel->uniquekeys == NIL); + /* XXX Why are we copying the expressions? */ foreach(lc, unionrel->reltarget->exprs) - { exprs = lappend(exprs, lfirst(lc)); - } + /* SQL: select union select; is valid, we need to handle it here. */ if (exprs == NIL) - /* SQL: select union select; is valid, we need to handle it here. */ add_uniquekey_for_onerow(unionrel); else unionrel->uniquekeys = lappend(unionrel->uniquekeys, @@ -389,6 +558,7 @@ populate_unionrel_uniquekeys(PlannerInfo *root, /* * populate_joinrel_uniquekeys + * Determine unique keys for a join relation. * * populate uniquekeys for joinrel. We will check each relation to see if its * UniqueKey is still valid via innerrel_keeps_unique, if so, we add it to @@ -404,70 +574,99 @@ populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, RelOptInfo *innerrel, List *restrictlist, JoinType jointype) { - ListCell *lc, *lc2; - List *clause_list = NIL; - List *outerrel_ukey_ctx; - List *innerrel_ukey_ctx; - bool inner_onerow, outer_onerow; - bool mergejoin_allowed; - - /* Care about the outerrel relation only for SEMI/ANTI join */ + ListCell *lc, + *lc2; + List *clause_list = NIL; + List *outerrel_ukey_ctx; + List *innerrel_ukey_ctx; + bool inner_onerow, + outer_onerow; + bool mergejoin_allowed; + + /* For SEMI/ANTI join, we care only about the outerrel unique keys. */ if (jointype == JOIN_SEMI || jointype == JOIN_ANTI) { foreach(lc, outerrel->uniquekeys) { UniqueKey *uniquekey = lfirst_node(UniqueKey, lc); + + /* Keep the unique key if it's included in the joinrel. */ if (list_is_subset(uniquekey->exprs, joinrel->reltarget->exprs)) joinrel->uniquekeys = lappend(joinrel->uniquekeys, uniquekey); } + return; } + /* XXX What about JOIN_RIGHT? */ Assert(jointype == JOIN_LEFT || jointype == JOIN_FULL || jointype == JOIN_INNER); - /* Fast path */ + /* + * For regular joins, we need to combine unique keys from both sides + * of the join, to get a new unique key for the join relation. So if + * either side does not have a unique key, bail out. + */ if (innerrel->uniquekeys == NIL || outerrel->uniquekeys == NIL) return; + /* XXX maybe move to the if blocks? Not needed outside. */ inner_onerow = relation_is_onerow(innerrel); outer_onerow = relation_is_onerow(outerrel); - outerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(outerrel); - innerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(innerrel); + outerrel_ukey_ctx = initialize_uniquecontext_for_joinrel(outerrel); + innerrel_ukey_ctx = initialize_uniquecontext_for_joinrel(innerrel); - clause_list = select_mergejoin_clauses(root, joinrel, outerrel, innerrel, + clause_list = select_mergejoin_clauses(root, + joinrel, outerrel, innerrel, restrictlist, jointype, &mergejoin_allowed); - if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true /* reverse */)) + /* + * XXX Seems a bit weird that it's called innerrel_keeps_unique but we + * seem to use it in both directions. Or what's the "reverse" for? The + * "reverse" name is not particularly descriptive. + */ + if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true)) { - bool outer_impact = jointype == JOIN_FULL; + bool outer_impact = (jointype == JOIN_FULL); + + /* Inspect unique keys on the outer relation. */ foreach(lc, outerrel_ukey_ctx) { UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + /* + * If the output of the join does not include all the parts of the + * unique key, it's useless, so mark it accordingly and ignore it. + */ if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) { ctx->useful = false; continue; } - /* Outer relation has one row, and the unique key is not duplicated after join, - * the joinrel will still has one row unless the jointype == JOIN_FULL. + /* + * When the outer relation has one row, and the unique key is not + * duplicated after join, so the joinrel will still have just one + * row unless the jointype == JOIN_FULL. In that case we're done, + * it's the strictest unique key possible. + * + * If it's one-row with a JOIN_FULL, it might produce multiple + * rows with NULLs, so set multi_nullvals. We also need to set + * the exprs correctly since it can't be NIL any more. + * + * For other cases (not one-row relation), we just reuse the + * unique key, but we may need to tweak the multi_nullvals. */ if (outer_onerow && !outer_impact) { add_uniquekey_for_onerow(joinrel); return; } - else if (outer_onerow) + else if (outer_onerow) /* one-row and FULL join */ { - /* - * The onerow outerrel becomes multi rows and multi_nullvals - * will be changed to true. We also need to set the exprs correctly since it - * can't be NIL any more. - */ ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, outerrel, NULL)) { joinrel->uniquekeys = lappend(joinrel->uniquekeys, @@ -485,18 +684,38 @@ populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, joinrel->uniquekeys = lappend(joinrel->uniquekeys, ctx->uniquekey); } + + /* + * Mark the unique key as added, so that we can ignore it later + * when combining unique keys from both sides of the join. + */ ctx->added_to_joinrel = true; } } + /* + * XXX Seems this actually checks if "outerrel keeps unique" so the name + * is misleading. Of maybe it's the previous block, not sure. + * + * XXX So why does this consider JOIN_FULL and JOIN_LEFT, while the previous + * block only cares about JOIN_FULL? + * + * XXX This is almost exact copy of the previous block, so maybe make it + * a separate function and just call it twice? + */ if (innerrel_keeps_unique(root, outerrel, innerrel, clause_list, false)) { - bool outer_impact = jointype == JOIN_FULL || jointype == JOIN_LEFT;; + bool outer_impact = (jointype == JOIN_FULL || jointype == JOIN_LEFT); + /* Inspect unique keys on the inner relation. */ foreach(lc, innerrel_ukey_ctx) { UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + /* + * If the output of the join does not include all the parts of the + * unique key, it's useless, so mark it accordingly and ignore it. + */ if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) { ctx->useful = false; @@ -529,29 +748,52 @@ populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, ctx->uniquekey); } + + /* + * Mark the unique key as added, so that we can ignore it later + * when combining unique keys from both sides of the join. + */ ctx->added_to_joinrel = true; } } /* - * The combination of the UniqueKey from both sides is unique as well regardless - * of join type, but no bother to add it if its subset has been added to joinrel - * already or it is not useful for the joinrel. + * XXX What if either of the previous two conditions did not match? In + * that case we haven't updated the useful flag, and maybe the unique + * key is not useful, but we don't know, right? So we should not be + * using it in the next loop. Or maybe we should evaluate the flag + * before the loops. + */ + + /* + * The combination of the UniqueKey from both sides is unique as well, + * regardless of the join type. But don't bother to add it if its + * subset has been added to joinrel already or when it's not useful for + * the joinrel. + * + * XXX Maybe we should have a flag that both sides have useful keys? + * Or maybe the loops are short/cheap? */ foreach(lc, outerrel_ukey_ctx) { UniqueKeyContext ctx1 = (UniqueKeyContext) lfirst(lc); + + /* when not useful or already added to the joinrel, skip it */ if (ctx1->added_to_joinrel || !ctx1->useful) continue; + foreach(lc2, innerrel_ukey_ctx) { UniqueKeyContext ctx2 = (UniqueKeyContext) lfirst(lc2); + + /* when not useful or already added to the joinrel, skip it */ if (ctx2->added_to_joinrel || !ctx2->useful) continue; + + /* If we add a onerow UniqueKey, we don't need another key. */ if (add_combined_uniquekey(root, joinrel, outerrel, innerrel, ctx1->uniquekey, ctx2->uniquekey, jointype)) - /* If we set a onerow UniqueKey to joinrel, we don't need other. */ return; } } @@ -560,8 +802,9 @@ populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, /* * convert_subquery_uniquekeys + * Covert the UniqueKey in subquery to outer relation. * - * Covert the UniqueKey in subquery to outer relation. + * XXX Explain what exactly does the conversion do? */ void convert_subquery_uniquekeys(PlannerInfo *root, RelOptInfo *currel, @@ -618,12 +861,14 @@ void convert_subquery_uniquekeys(PlannerInfo *root, /* * innerrel_keeps_unique + * Check if Unique key on the innerrel is valid after join. * - * Check if Unique key of the innerrel is valid after join. innerrel's UniqueKey - * will be still valid if innerrel's any-column mergeop outrerel's uniquekey - * exists in clause_list. + * innerrel's UniqueKey will be still valid if innerrel's any-column mergeop + * outrerel's uniquekey exists in clause_list * * Note: the clause_list must be a list of mergeable restrictinfo already. + * + * XXX Misleading name? We seem to use it for "outerrel_keeps_unique" too. */ static bool innerrel_keeps_unique(PlannerInfo *root, @@ -634,26 +879,32 @@ innerrel_keeps_unique(PlannerInfo *root, { ListCell *lc, *lc2, *lc3; + /* XXX probably not needed, duplicate with the check in the caller + * (populate_joinrel_uniquekeys). But it's cheap. */ if (outerrel->uniquekeys == NIL || innerrel->uniquekeys == NIL) return false; /* Check if there is outerrel's uniquekey in mergeable clause. */ foreach(lc, outerrel->uniquekeys) { - List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs; - bool clauselist_matchs_all_exprs = true; + List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs; + bool clauselist_matchs_all_exprs = true; + foreach(lc2, outer_uq_exprs) { Node *outer_uq_expr = lfirst(lc2); bool find_uq_expr_in_clauselist = false; + foreach(lc3, clause_list) { RestrictInfo *rinfo = lfirst_node(RestrictInfo, lc3); Node *outer_expr; + if (reverse) outer_expr = rinfo->outer_is_left ? get_rightop(rinfo->clause) : get_leftop(rinfo->clause); else outer_expr = rinfo->outer_is_left ? get_leftop(rinfo->clause) : get_rightop(rinfo->clause); + if (equal(outer_expr, outer_uq_expr)) { find_uq_expr_in_clauselist = true; @@ -677,22 +928,37 @@ innerrel_keeps_unique(PlannerInfo *root, /* * relation_is_onerow - * Check if it is a one-row relation by checking UniqueKey. + * Check if it is a one-row relation by checking UniqueKey. + * + * The one-row is a special case - there has to be just a single unique key, + * with no expressions. */ bool relation_is_onerow(RelOptInfo *rel) { UniqueKey *ukey; - if (rel->uniquekeys == NIL) + + /* there has to be exactly one unique key */ + if (list_length(rel->uniquekeys) != 1) return false; + ukey = linitial_node(UniqueKey, rel->uniquekeys); - return ukey->exprs == NIL && list_length(rel->uniquekeys) == 1; + + /* the unique key must have no expressions */ + return (ukey->exprs == NIL); } /* * relation_has_uniquekeys_for - * Returns true if we have proofs that 'rel' cannot return multiple rows with - * the same values in each of 'exprs'. Otherwise returns false. + * Determines if the relation has unique key for a list of expressions. + * + * Returns true iff we can prove that the relation cannot return multiple rows + * with the same values in the provided expression. + * + * allow_multinulls determines whether we allow multiple NULL values or not. + * + * The special "one-row" unique key is considered incompatible with all + * possible expressions. */ bool relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel, @@ -710,20 +976,39 @@ relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel, foreach(lc, rel->uniquekeys) { UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (ukey->multi_nullvals && !allow_multinulls) continue; + if (list_is_subset(ukey->exprs, exprs)) return true; } + return false; } /* * get_exprs_from_uniqueindex + * Return a list of expressions from a unique index. + * + * Provided with a list of expressions and opclass families, we try to match + * it to the index. If useful, we produce a list of index expressions (subset + * of the list we provided). + * + * We simply walk through the index expressions, and for each expression we + * check three things: * - * Return a list of exprs which is unique. set useful to false if this - * unique index is not useful for us. + * 1) If there's a matching (expr = Const) clause, we can simply ignore the + * expressions. Unique index on (a,b,c) guarantees uniqueness on (a,b) when + * there's condition (c=1). + * + * 2) Check that the index expression is present in the relation we're + * dealing with. If not, the unique key would be useless anyway, and the + * index can't produce unique key. + * + * XXX Shouldn't it be enough to return NULL when the index is not useful? + * The extra flag seems a bit unnecessary. */ static List * get_exprs_from_uniqueindex(IndexOptInfo *unique_index, @@ -743,18 +1028,19 @@ get_exprs_from_uniqueindex(IndexOptInfo *unique_index, indexpr_item = list_head(unique_index->indexprs); for(c = 0; c < unique_index->ncolumns; c++) { - int attr = unique_index->indexkeys[c]; - Expr *expr; - bool matched_const = false; - ListCell *lc1, *lc2; + int attr = unique_index->indexkeys[c]; + Expr *expr; + bool matched_const = false; + ListCell *lc1, *lc2; - if(attr > 0) + if (attr > 0) { + /* regular attribute, just use the expression from index tlist */ expr = list_nth_node(TargetEntry, unique_index->indextlist, c)->expr; } else if (attr == 0) { - /* Expression index */ + /* expression from the index */ expr = lfirst(indexpr_item); indexpr_item = lnext(unique_index->indexprs, indexpr_item); } @@ -764,29 +1050,43 @@ get_exprs_from_uniqueindex(IndexOptInfo *unique_index, Assert(false); } + /* should have a valid expression now */ + Assert(expr); + /* - * Check index_col = Const case with regarding to opfamily checking - * If we can remove the index_col from the final UniqueKey->exprs. + * Check if there's (index_col = Const) condition, and that it's using + * a compatible opfamily. If yes, we can remove the index_col from the + * final UniqueKey->exprs, because the value is constant (so removing + * it can't introduce duplicities). */ forboth(lc1, const_exprs, lc2, const_expr_opfamilies) { - if (list_member_oid((List *)lfirst(lc2), unique_index->opfamily[c]) - && match_index_to_operand((Node *) lfirst(lc1), c, unique_index)) + List *opfamilies = (List *) lfirst(lc2); + Node *cexpr = (Node *) lfirst(lc1); + + if (list_member_oid(opfamilies, unique_index->opfamily[c]) && + match_index_to_operand(cexpr, c, unique_index)) { matched_const = true; break; } } + /* it's constant, so ignore the expression */ if (matched_const) continue; - /* Check if the indexed expr is used in rel */ + /* + * Check if the indexed expr is used in rel. We do this after the + * (col = Const) check, because nn expression may be in a a restrict + * clause and not in the reltarget. So we don't want to rule out an + * index unnecessarily. + */ if (attr > 0) { /* - * Normal Indexed column, if the col is not used, then the index is useless - * for uniquekey. + * Normal indexed column, if the col is not used, then the index + * is useless for uniquekey. */ attr -= FirstLowInvalidHeapAttributeNumber; @@ -806,67 +1106,85 @@ get_exprs_from_uniqueindex(IndexOptInfo *unique_index, /* check not null property. */ if (attr == 0) { - /* We never know if a expression yields null or not */ + /* We never know if an expression yields null or not */ *multi_nullvals = true; } - else if (!bms_is_member(attr, unique_index->rel->notnullattrs) - && !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber, - unique_index->rel->notnullattrs)) + else if (!bms_is_member(attr, unique_index->rel->notnullattrs) && + !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber, + unique_index->rel->notnullattrs)) { *multi_nullvals = true; } exprs = lappend(exprs, expr); } + return exprs; } /* * add_uniquekey_for_onerow - * If we are sure that the relation only returns one row, then all the columns - * are unique. However we don't need to create UniqueKey for every column, we - * just set exprs = NIL and overwrites all the other UniqueKey on this RelOptInfo - * since this one has strongest semantics. + * Create a special unique key signifying that the rel has one row. + * + * If we are sure that the relation only returns one row (it might return + * no rows, but we still consider that unique), then all the columns are + * trivially unique. + * + * However we don't need to create UniqueKey with every column, we just + * set exprs = NIL, because that's easier to identify. We don't want to + * add unnecessary unique keys (such that we already have a unique key + * for a subset of the expressions), and with (exprs == NIL) we can just + * assume we have one unique key for each column in the rel. + * + * We discard all other unique keys, since it has the strongest semantics. */ void add_uniquekey_for_onerow(RelOptInfo *rel) { /* - * We overwrite the previous UniqueKey on purpose since this one has the - * strongest semantic. + * We overwrite the previous UniqueKey on purpose since this one has + * the strongest semantic (all other unique keys are implied by it). */ rel->uniquekeys = list_make1(makeUniqueKey(NIL, false)); } /* - * initililze_uniquecontext_for_joinrel - * Return a List of UniqueKeyContext for an inputrel + * initialize_uniquecontext_for_joinrel + * Return a List of UniqueKeyContext for an inputrel. */ static List * -initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel) +initialize_uniquecontext_for_joinrel(RelOptInfo *inputrel) { - List *res = NIL; - ListCell *lc; - foreach(lc, inputrel->uniquekeys) + List *res = NIL; + ListCell *lc; + + foreach(lc, inputrel->uniquekeys) { UniqueKeyContext context; + context = palloc(sizeof(struct UniqueKeyContextData)); context->uniquekey = lfirst_node(UniqueKey, lc); context->added_to_joinrel = false; context->useful = true; + res = lappend(res, context); } + return res; } - /* * get_exprs_from_uniquekey - * Unify the way of get List of exprs from a one-row UniqueKey or - * normal UniqueKey. for the onerow case, every expr in rel1 is a valid - * UniqueKey. Return a List of exprs. + * Extract expressions that are part of a unique key. + * + * The meaning of the result is a bit different in regular and one-row cases. + * For the regular case, the list of expressions form a single unique key, + * i.e. the combination of values is unique. + * + * For the one-row case, each individual expression is known to be unique + * (simply because in a single row everything is unique). * * rel1: The relation which you want to get the exprs. * ukey: The UniqueKey you want to get the exprs. @@ -875,27 +1193,29 @@ static List * get_exprs_from_uniquekey(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *rel1, UniqueKey *ukey) { - ListCell *lc; - bool onerow = rel1 != NULL && relation_is_onerow(rel1); + ListCell *lc; + List *res = NIL; + bool onerow = (rel1 != NULL) && relation_is_onerow(rel1); - List *res = NIL; + /* We require at least one of those to be true. */ Assert(onerow || ukey); - if (onerow) - { - /* Only cares about the exprs still exist in joinrel */ - foreach(lc, joinrel->reltarget->exprs) - { - Bitmapset *relids = pull_varnos(root, lfirst(lc)); - if (bms_is_subset(relids, rel1->relids)) - { - res = lappend(res, list_make1(lfirst(lc))); - } - } - } - else + + /* if not a one-row unique key, just return the key's expressions */ + if (!onerow) + return list_make1(ukey->exprs); + + /* + * If it's a one-row relation, we simply extract the expressions that + * still exist in the reltarget. + */ + foreach(lc, joinrel->reltarget->exprs) { - res = list_make1(ukey->exprs); + Bitmapset *relids = pull_varnos(root, lfirst(lc)); + + if (bms_is_subset(relids, rel1->relids)) + res = lappend(res, list_make1(lfirst(lc))); } + return res; } @@ -910,55 +1230,67 @@ get_exprs_from_uniquekey(PlannerInfo *root, RelOptInfo *joinrel, /* * index_constains_partkey - * return true if the index contains the partiton key. + * Determines if the index includes a partition key. + * + * XXX Surely we already have a code doing this already? E.g. when creating + * a unique index on a partitioned table we define that. */ static bool -index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind) +index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind) { ListCell *lc; int i; + Assert(IS_PARTITIONED_REL(partrel)); Assert(partrel->part_scheme->partnatts > 0); for(i = 0; i < partrel->part_scheme->partnatts; i++) { - Node *part_expr = linitial(partrel->partexprs[i]); - bool found_in_index = false; + Node *part_expr = linitial(partrel->partexprs[i]); + bool found_in_index = false; + foreach(lc, ind->indextlist) { - Expr *index_expr = lfirst_node(TargetEntry, lc)->expr; + Expr *index_expr = lfirst_node(TargetEntry, lc)->expr; + if (equal(index_expr, part_expr)) { found_in_index = true; break; } } + if (!found_in_index) return false; } + return true; } /* * simple_indexinfo_equal + * Compare two indexes to determine if they are the same. + * + * We need to do this because simple_copy_indexinfo_to_parent does change + * some elements. So this is not exactly the same as calling equal(). * - * Used to check if the 2 index is same as each other. The index here - * is COPIED from childrel and did some tiny changes(see - * simple_copy_indexinfo_to_parent) + * XXX I wonder if we could simply use equal(), somehow? In fact, we should + * probably build something much simpler than IndexOptInfo, just enough to + * do the checks. */ static bool simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2) { Size oid_cmp_len = sizeof(Oid) * ind1->ncolumns; - return ind1->ncolumns == ind2->ncolumns && - ind1->unique == ind2->unique && - memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0 && - memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0 && - memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0 && - memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0 && - equal(get_tlist_exprs(ind1->indextlist, true), - get_tlist_exprs(ind2->indextlist, true)); + return ((ind1->ncolumns == ind2->ncolumns) && + (ind1->unique == ind2->unique) && + (memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0) && + (memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0) && + (memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0) && + (memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0) && + (equal(get_tlist_exprs(ind1->indextlist, true), + get_tlist_exprs(ind2->indextlist, true)))); } @@ -981,11 +1313,21 @@ simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2) /* - * simple_copy_indexinfo_to_parent (from partition) - * Copy the IndexInfo from child relation to parent relation with some modification, - * which is used to test: - * 1. If the same index exists in all the childrels. + * simple_copy_indexinfo_to_parent + * Copy index info from child to parent, with necessary tweaks. + * + * We use this copy to check: + * + * 1. If the same/matching index exists in all the childrels. * 2. If the parentrel->reltarget/basicrestrict info matches this index. + * + * XXX IMHO we should probably build something much simpler than a full + * IndexOptInfo copy, just enough to do the checks. + * + * XXX The fact that we copy so much data seems wrong, and having to + * define macros from copyfuncs.c seems like a very suspicious thing. + * One reason is that IndeOptInfo is fairly large struct, especially + * with all the fields, and we allocate it very often. */ static IndexOptInfo * simple_copy_indexinfo_to_parent(PlannerInfo *root, @@ -1027,20 +1369,24 @@ simple_copy_indexinfo_to_parent(PlannerInfo *root, /* * adjust_partition_unique_indexlist + * Checks and eliminates indexes that do not exist on the child relation. * - * global_unique_indexes: At the beginning, it contains the copy & modified - * unique index from the first partition. And then check if each index in it still - * exists in the following partitions. If no, remove it. at last, it has an - * index list which exists in all the partitions. + * Walks the list of unique indexes, and eliminates those that don't match + * the child relation (i.e. where a matching child index does not exist). + * This is used to iteratively filter the list of candidate unique keys. + * + * After processing all child relations, the list contains only indexes that + * exist in all the child relations. */ static void adjust_partition_unique_indexlist(PlannerInfo *root, RelOptInfo *parentrel, RelOptInfo *childrel, - List **global_unique_indexes) + List **indexes) { ListCell *lc, *lc2; - foreach(lc, *global_unique_indexes) + + foreach(lc, *indexes) { IndexOptInfo *g_ind = lfirst_node(IndexOptInfo, lc); bool found_in_child = false; @@ -1049,23 +1395,45 @@ adjust_partition_unique_indexlist(PlannerInfo *root, { IndexOptInfo *p_ind = lfirst_node(IndexOptInfo, lc2); IndexOptInfo *p_ind_copy; - if (!p_ind->unique || !p_ind->immediate || - (p_ind->indpred != NIL && !p_ind->predOK)) + + /* + * Ignore child indexes that can't possibly match (not unique or + * immediate, etc.) + * + * XXX We do these checks in many places, so maybe turn it into + * a reusable macro? + */ + if ((!p_ind->unique) || (!p_ind->immediate) || + (p_ind->indpred != NIL) && (!p_ind->predOK)) continue; + + /* + * XXX This seems possibly quite expensive. Imagine there are many + * child relations, with a bunch of unique indexes each. Then this + * generates a copy for each unique index in each child relation, + * something like O(N^2/2) copies. + */ p_ind_copy = simple_copy_indexinfo_to_parent(root, parentrel, p_ind); + + /* Found a matching index for the child relation, we're done. */ if (simple_indexinfo_equal(p_ind_copy, g_ind)) { found_in_child = true; break; } } + + /* No matching index in the child, so remove it from the list. */ if (!found_in_child) - /* The index doesn't exist in childrel, remove it from global_unique_indexes */ - *global_unique_indexes = foreach_delete_current(*global_unique_indexes, lc); + *indexes = foreach_delete_current(*indexes, lc); } } -/* Helper function for groupres/distinctrel */ +/* + * Helper function for groupres/distinctrel + * + * FIXME Not sure about this. + */ static void add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgroups) { @@ -1073,27 +1441,32 @@ add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgrou List *exprs; /* - * XXX: If there are some vars which is not in current levelsup, the semantic is - * imprecise, should we avoid it or not? levelsup = 1 is just a demo, maybe we need to - * check every level other than 0, if so, looks we have to write another - * pull_var_walker. + * XXX: If there are some vars which are not in the current levelsup, the + * semantic is imprecise, should we avoid it or not? levelsup = 1 is just + * a demo, maybe we need to check every level other than 0, if so, looks + * we have to write another pull_var_walker. */ List *upper_vars = pull_vars_of_level((Node*)sortgroups, 1); if (upper_vars != NIL) return; + /* sortgroupclause can't be multi_nullvals */ exprs = get_sortgrouplist_exprs(sortgroups, parse->targetList); rel->uniquekeys = lappend(rel->uniquekeys, - makeUniqueKey(exprs, - false /* sortgroupclause can't be multi_nullvals */)); + makeUniqueKey(exprs, false)); } /* * add_combined_uniquekey - * The combination of both UniqueKeys is a valid UniqueKey for joinrel no matter - * the jointype. + * Add a unique key for a join, combined from keys on inner/outer side. + * + * The combination of both UniqueKeys is a valid UniqueKey for joinrel no + * matter what's the exact jointype. + * + * Returns true if the unique key is "one-row" variant, so that the caller + * can stop considering further combinations. */ bool add_combined_uniquekey(PlannerInfo *root, @@ -1104,32 +1477,47 @@ add_combined_uniquekey(PlannerInfo *root, UniqueKey *inner_ukey, JoinType jointype) { + bool multi_nullvals; + ListCell *lc1, *lc2; - ListCell *lc1, *lc2; - - /* Either side has multi_nullvals or we have outer join, - * the combined UniqueKey has multi_nullvals */ - bool multi_nullvals = outer_ukey->multi_nullvals || + /* + * If either side has multi_nullvals, or we are dealing with an outer join, + * the combined UniqueKey has multi_nullvals too. + */ + multi_nullvals = outer_ukey->multi_nullvals || inner_ukey->multi_nullvals || IS_OUTER_JOIN(jointype); /* The only case we can get onerow joinrel after join */ - if (relation_is_onerow(outer_rel) - && relation_is_onerow(inner_rel) - && jointype == JOIN_INNER) + if (relation_is_onerow(outer_rel) && + relation_is_onerow(inner_rel) && + jointype == JOIN_INNER) { add_uniquekey_for_onerow(joinrel); return true; } + /* + * XXX Isn't this wrong? Why is it combining expressions that are part + * of the two unique keys? Imagine we have outer unique key on (a1, a2) + * and inner outer key on (b1, b2). Then this adds four unique keys + * for the join (a1,b1), (a1,b2), (a2,b1) and (a2,b2). Shouldn't it + * just add (a1,a2,b1,b2)? + */ foreach(lc1, get_exprs_from_uniquekey(root, joinrel, outer_rel, outer_ukey)) { + /* + * XXX This calls get_exprs_from_uniquekey repeatedly for each outer + * loop. Maybe we should calculate it just once before the loop. + */ foreach(lc2, get_exprs_from_uniquekey(root, joinrel, inner_rel, inner_ukey)) { List *exprs = list_concat_copy(lfirst_node(List, lc1), lfirst_node(List, lc2)); + joinrel->uniquekeys = lappend(joinrel->uniquekeys, makeUniqueKey(exprs, multi_nullvals)); } } + return false; } diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index 8d8e493f5c..f29b65c07b 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -2387,6 +2387,7 @@ grouping_planner(PlannerInfo *root, bool inheritance_update, add_path(final_rel, path); } + /* XXX comment? can we simply just copy the unique keys to the final relation? */ simple_copy_uniquekeys(current_rel, final_rel); /* @@ -3902,7 +3903,9 @@ create_grouping_paths(PlannerInfo *root, set_cheapest(grouped_rel); + /* XXX does this apply to grouping sets too? */ populate_grouprel_uniquekeys(root, grouped_rel, input_rel); + return grouped_rel; } @@ -4625,7 +4628,10 @@ create_window_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(window_rel); + + /* XXX comment? */ simple_copy_uniquekeys(input_rel, window_rel); + return window_rel; } @@ -4939,7 +4945,10 @@ create_distinct_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(distinct_rel); + + /* XXX comment */ populate_distinctrel_uniquekeys(root, input_rel, distinct_rel); + return distinct_rel; } @@ -5200,6 +5209,7 @@ create_ordered_paths(PlannerInfo *root, */ Assert(ordered_rel->pathlist != NIL); + /* XXX comment */ simple_copy_uniquekeys(input_rel, ordered_rel); return ordered_rel; diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c index b7626545bf..72a3f3c598 100644 --- a/src/backend/optimizer/prep/prepunion.c +++ b/src/backend/optimizer/prep/prepunion.c @@ -691,6 +691,7 @@ generate_union_paths(SetOperationStmt *op, PlannerInfo *root, /* Add the UniqueKeys */ populate_unionrel_uniquekeys(root, result_rel); + return result_rel; } diff --git a/src/backend/optimizer/util/inherit.c b/src/backend/optimizer/util/inherit.c index 3eec1f4d74..c9829c5fc4 100644 --- a/src/backend/optimizer/util/inherit.c +++ b/src/backend/optimizer/util/inherit.c @@ -755,6 +755,7 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, pseudoconstant, rinfo->security_level, NULL, NULL, NULL); + /* XXX This is a bit weird, doing this outside make_restrictinfo */ child_rinfo->mergeopfamilies = rinfo->mergeopfamilies; childquals = lappend(childquals, child_rinfo); /* track minimum security level among child quals */ -- 2.30.2 --------------F495F6B18672582269F00FF9 Content-Type: text/x-patch; charset=UTF-8; name="0005-Extend-UniqueKeys-20210317.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0005-Extend-UniqueKeys-20210317.patch" ^ permalink raw reply [nested|flat] 56+ messages in thread
* [PATCH 06/10] review @ 2021-03-17 01:19 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 56+ messages in thread From: Tomas Vondra @ 2021-03-17 01:19 UTC (permalink / raw) --- src/backend/optimizer/path/uniquekeys.c | 2 ++ src/backend/optimizer/plan/planner.c | 2 ++ 2 files changed, 4 insertions(+) diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c index b9567ca5c8..0e6c826fcd 100644 --- a/src/backend/optimizer/path/uniquekeys.c +++ b/src/backend/optimizer/path/uniquekeys.c @@ -1522,6 +1522,7 @@ add_combined_uniquekey(PlannerInfo *root, return false; } +/* FIXME comment */ List* build_uniquekeys(PlannerInfo *root, List *sortclauses) { @@ -1548,6 +1549,7 @@ build_uniquekeys(PlannerInfo *root, List *sortclauses) return result; } +/* FIXME comment */ bool query_has_uniquekeys_for(PlannerInfo *root, List *pathuniquekeys, bool allow_multinulls) diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index 62c0e46c49..f66b9d4898 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -4857,11 +4857,13 @@ create_distinct_paths(PlannerInfo *root, Path *path = (Path *) lfirst(lc); if (pathkeys_contained_in(needed_pathkeys, path->pathkeys)) + { add_path(distinct_rel, (Path *) create_upper_unique_path(root, distinct_rel, path, list_length(root->distinct_pathkeys), numDistinctRows)); + } } foreach(lc, input_rel->unique_pathlist) -- 2.30.2 --------------F495F6B18672582269F00FF9 Content-Type: text/x-patch; charset=UTF-8; name="0007-Index-skip-scan-20210317.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0007-Index-skip-scan-20210317.patch" ^ permalink raw reply [nested|flat] 56+ messages in thread
* [PATCH 06/10] review @ 2021-03-17 01:19 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 56+ messages in thread From: Tomas Vondra @ 2021-03-17 01:19 UTC (permalink / raw) --- src/backend/optimizer/path/uniquekeys.c | 2 ++ src/backend/optimizer/plan/planner.c | 2 ++ 2 files changed, 4 insertions(+) diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c index b9567ca5c8..0e6c826fcd 100644 --- a/src/backend/optimizer/path/uniquekeys.c +++ b/src/backend/optimizer/path/uniquekeys.c @@ -1522,6 +1522,7 @@ add_combined_uniquekey(PlannerInfo *root, return false; } +/* FIXME comment */ List* build_uniquekeys(PlannerInfo *root, List *sortclauses) { @@ -1548,6 +1549,7 @@ build_uniquekeys(PlannerInfo *root, List *sortclauses) return result; } +/* FIXME comment */ bool query_has_uniquekeys_for(PlannerInfo *root, List *pathuniquekeys, bool allow_multinulls) diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index 62c0e46c49..f66b9d4898 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -4857,11 +4857,13 @@ create_distinct_paths(PlannerInfo *root, Path *path = (Path *) lfirst(lc); if (pathkeys_contained_in(needed_pathkeys, path->pathkeys)) + { add_path(distinct_rel, (Path *) create_upper_unique_path(root, distinct_rel, path, list_length(root->distinct_pathkeys), numDistinctRows)); + } } foreach(lc, input_rel->unique_pathlist) -- 2.30.2 --------------F495F6B18672582269F00FF9 Content-Type: text/x-patch; charset=UTF-8; name="0007-Index-skip-scan-20210317.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0007-Index-skip-scan-20210317.patch" ^ permalink raw reply [nested|flat] 56+ messages in thread
* [PATCH 08/10] review @ 2021-03-17 01:36 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 56+ messages in thread From: Tomas Vondra @ 2021-03-17 01:36 UTC (permalink / raw) --- src/backend/commands/explain.c | 1 + src/backend/executor/nodeIndexscan.c | 2 ++ 2 files changed, 3 insertions(+) diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index a160de5493..3b0b38099f 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -1110,6 +1110,7 @@ ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es) { if (skipPrefixSize > 0) { + /* FIXME Why not to show this for TEXT output? */ if (es->format != EXPLAIN_FORMAT_TEXT) ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es); } diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c index 71aac4493d..720442b731 100644 --- a/src/backend/executor/nodeIndexscan.c +++ b/src/backend/executor/nodeIndexscan.c @@ -124,6 +124,8 @@ IndexNext(IndexScanState *node) node->iss_ScanDesc = scandesc; + /* FIXME Is the else branch necessary? */ + /* Index skip scan assumes xs_want_itup, so set it to true */ if (indexscan->indexskipprefixsize > 0) node->iss_ScanDesc->xs_want_itup = true; -- 2.30.2 --------------F495F6B18672582269F00FF9 Content-Type: text/x-patch; charset=UTF-8; name="0009-Btree-implementation-of-skipping-20210317.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0009-Btree-implementation-of-skipping-20210317.patch" ^ permalink raw reply [nested|flat] 56+ messages in thread
* [PATCH 08/10] review @ 2021-03-17 01:36 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 56+ messages in thread From: Tomas Vondra @ 2021-03-17 01:36 UTC (permalink / raw) --- src/backend/commands/explain.c | 1 + src/backend/executor/nodeIndexscan.c | 2 ++ 2 files changed, 3 insertions(+) diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index a160de5493..3b0b38099f 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -1110,6 +1110,7 @@ ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es) { if (skipPrefixSize > 0) { + /* FIXME Why not to show this for TEXT output? */ if (es->format != EXPLAIN_FORMAT_TEXT) ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es); } diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c index 71aac4493d..720442b731 100644 --- a/src/backend/executor/nodeIndexscan.c +++ b/src/backend/executor/nodeIndexscan.c @@ -124,6 +124,8 @@ IndexNext(IndexScanState *node) node->iss_ScanDesc = scandesc; + /* FIXME Is the else branch necessary? */ + /* Index skip scan assumes xs_want_itup, so set it to true */ if (indexscan->indexskipprefixsize > 0) node->iss_ScanDesc->xs_want_itup = true; -- 2.30.2 --------------F495F6B18672582269F00FF9 Content-Type: text/x-patch; charset=UTF-8; name="0009-Btree-implementation-of-skipping-20210317.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0009-Btree-implementation-of-skipping-20210317.patch" ^ permalink raw reply [nested|flat] 56+ messages in thread
* Re: [PoC] Reducing planning time when tables have many partitions @ 2023-01-30 12:14 David Rowley <[email protected]> 0 siblings, 1 reply; 56+ messages in thread From: David Rowley @ 2023-01-30 12:14 UTC (permalink / raw) To: Yuya Watari <[email protected]>; +Cc: Thom Brown <[email protected]>; Andrey Lepikhov <[email protected]>; Alvaro Herrera <[email protected]>; Zhang Mingli <[email protected]>; Tom Lane <[email protected]>; PostgreSQL Developers <[email protected]> isOn Mon, 30 Jan 2023 at 23:03, Yuya Watari <[email protected]> wrote: > 1. v13-0005 > > The first bug is in eclass_member_iterator_strict_next(). As I > mentioned in the commit message, the original code incorrectly missed > EquivalenceMembers with empty em_relids when 'with_norel_members' is > true. Yeah, I was also looking at this today and found the same issues after adding the verification code that checks we get the same members from the index and via the looking method. I ended up making some changes slightly different from what you had but wasn't quite ready to post them yet. I'm still a little unhappy with master's comments for the EquivalenceMember.em_relids field. It claims to be the relids for the em_expr, but that's not the case for em_is_child members. I've ended up adding an additional field named em_norel_expr that gets set to true when em_expr truly contains no Vars. I then adjusted the conditions in the iterator's loops to properly include members with no Vars when we ask for those. > 2. v13-0006 > > The second bug exists in get_ecmember_indexes_strict(). As I described > in the comment, if the empty relids is given, this function must > return all members because their em_relids are always superset. I am > concerned that this change may adversely affect performance. > Currently, I have not seen any degradation. I fixed this by adding a new field to the iterator struct named relids_empty. It's just set to bms_is_empty(iter->with_relids). The loop condition then just becomes: if (iter->relids_empty || !bms_is_subset(iter->with_relids, em->em_relids)) continue; > 3. v13-0007 > > The last one is in add_eq_member(). I am not sure why this change is > working, but it is probably related to the concerns David mentioned in > the previous mail. The v13-0007 may be wrong, so it should be > reconsidered. Unfortunately, we can't fix it that way. At a glance, what you have would only find var-less child members if you requested that the iterator also gave you with_norel_members==true. I've not looked, perhaps all current code locations request with_norel_members, so your change likely just words by accident. I've attached what I worked on today. I still want to do more cross-checking to make sure all code locations which use these new iterators get the same members as they used to get. In the attached I also changed the code that added a RelOptInfo to root->simple_rel_array[0] to allow the varno=0 Vars made in generate_append_tlist() to be indexed. That's now done via a new function (setup_append_rel_entry()) which is only called during plan_set_operations(). This means we're no longer wastefully creating that entry during the planning of normal queries. We could maybe consider giving this a more valid varno and expand simple_rel_array to make more room, but I'm not sure it's worth it or not. I'm happier that this simple_rel_array[0] entry now only exists when planning set operations, but I'd probably feel better if there was some other way that felt less like we're faking up a RelOptInfo to store EquivalenceMembers in. I've also included a slightly edited version of your code which checks that the members match when using and not using the new indexes. All the cross-checking seems to pass. David From 177d013928d2bb5c0c4ee565a433bbf62481b18c Mon Sep 17 00:00:00 2001 From: David Rowley <[email protected]> Date: Thu, 1 Dec 2022 20:36:10 +1300 Subject: [PATCH v15 1/2] Adjust bms_int_members so that it shortens the left input Prior to this change, if the left input to bms_int_members had fewer Bitmapwords than the right input, the additional words in the left input would be zeroed. Doing things this why only really makes sense if we expect to add additional words to the resulting bitmap set again later. Seemingly, most of our use cases don't add to the resulting Bitmapset, in fact, many cases are performing bms_int_members in a loop which is likely to even further increase the amount of all zero words in the set. Leaving these zero trailing words in place can cause inefficiencies in functions such as bms_is_empty and loops which loop over a set using bms_next_member when the set is left with a large number of trailing words. If there are use cases which need to add additional words again, then, repalloc has to do very little processing when the requested allocation size is <= the actual allocation size. Both AllocSetRealloc and GenerationRealloc simply just return the input pointer again. If there are any cases where we do increase the size of the set again, then this just shifts the zeroing of the additional words until then. We could give bms_del_members() and bms_difference similar treatment, but that would require additional code to record the last non-zero word after masking out the members being deleted. That seems less worthwhile as it will add cost when we're unable to trim down the number of words. --- src/backend/nodes/bitmapset.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/src/backend/nodes/bitmapset.c b/src/backend/nodes/bitmapset.c index 98660524ad..b13b4c378f 100644 --- a/src/backend/nodes/bitmapset.c +++ b/src/backend/nodes/bitmapset.c @@ -920,8 +920,16 @@ bms_int_members(Bitmapset *a, const Bitmapset *b) shortlen = Min(a->nwords, b->nwords); for (i = 0; i < shortlen; i++) a->words[i] &= b->words[i]; - for (; i < a->nwords; i++) - a->words[i] = 0; + + /* + * We simply shorten the left input to both remove the need to zero the + * trailing words and also to reduce the required processing if this + * function is being called in a loop. If more words are required later + * then AllocSetRealloc is likely not going to have to work very hard if + * the new requested size is >= the actual size of the allocation. + */ + a->nwords = shortlen; + return a; } -- 2.39.0.windows.1 From ba89effb12c3547bf843bd0e991b454388cfb88e Mon Sep 17 00:00:00 2001 From: David Rowley <[email protected]> Date: Sun, 4 Dec 2022 11:09:21 +1300 Subject: [PATCH v15 2/2] Add Bitmapset indexes for faster lookup of EquivalenceMembers For queries containing a large number of joins or for queries to partitioned tables, the looking up of EquivalenceMembers in an EquivalenceClass can become slow due to the large number of members. Until now, the searching for EquivalenceMembers has been done using a linear search over the EquivalenceClass's list of members. Here we effectively add an index to allow faster lookups of these members. This is done by way of adding all members to all classes into a List in PlannerInfo, then we have a Bitmapset in each EquivalenceClass with members for each index in PlannerInfo's list which belong to this class. Additionally, each RelOptInfo also maintains another Bitmapset which stores each index in PlannerInfo's list of EquivalenceMember for all members, in all classes, which mention this relation. This allows us to quickly find all EquivalenceMembers for a given RelOptInfo and a given EquivalenceClass by intersecting these two lists. When searching for members belonging to join rels, we must either intersect or union all base RelOptInfo's member indexes then intersect the result to the EquivalenceClass's indexes. Whether we intersect or union depends on if we need members that mention all relations or members that mention any of the base relations making up the join relation. This method of indexing can significantly speed up the planner when querying a partitioned table with a large number of partitions when the query contains a few joins. Tests have shown around a x10 increase in performance with 1000 partitions. --- contrib/postgres_fdw/postgres_fdw.c | 50 +- src/backend/nodes/list.c | 16 + src/backend/nodes/outfuncs.c | 7 +- src/backend/optimizer/path/costsize.c | 3 +- src/backend/optimizer/path/equivclass.c | 1295 +++++++++++++++++---- src/backend/optimizer/path/indxpath.c | 22 +- src/backend/optimizer/path/pathkeys.c | 26 +- src/backend/optimizer/plan/createplan.c | 66 +- src/backend/optimizer/plan/planagg.c | 1 + src/backend/optimizer/plan/planner.c | 3 + src/backend/optimizer/prep/prepjointree.c | 3 + src/backend/optimizer/prep/prepunion.c | 25 +- src/backend/optimizer/util/relnode.c | 9 + src/include/nodes/pathnodes.h | 81 +- src/include/nodes/pg_list.h | 1 + src/include/optimizer/paths.h | 67 +- src/tools/pgindent/typedefs.list | 1 + 17 files changed, 1347 insertions(+), 329 deletions(-) diff --git a/contrib/postgres_fdw/postgres_fdw.c b/contrib/postgres_fdw/postgres_fdw.c index 50d23f922c..b85c4a071e 100644 --- a/contrib/postgres_fdw/postgres_fdw.c +++ b/contrib/postgres_fdw/postgres_fdw.c @@ -7619,6 +7619,10 @@ conversion_error_callback(void *arg) varno = var->varno; colno = var->varattno; } + /* + * XXX why don't we break here? Surely there can't be another + * equal EquivalenceMember? + */ } if (varno > 0) @@ -7677,23 +7681,27 @@ conversion_error_callback(void *arg) EquivalenceMember * find_em_for_rel(PlannerInfo *root, EquivalenceClass *ec, RelOptInfo *rel) { - ListCell *lc; + EquivalenceMemberIterator iter; + EquivalenceMember *em; - foreach(lc, ec->ec_members) - { - EquivalenceMember *em = (EquivalenceMember *) lfirst(lc); + setup_eclass_member_iterator(&iter, root, ec, rel->relids, true, false); + while ((em = eclass_member_iterator_next(&iter)) != NULL) + { /* * Note we require !bms_is_empty, else we'd accept constant * expressions which are not suitable for the purpose. */ + Assert(!bms_is_empty(em->em_relids)); + if (bms_is_subset(em->em_relids, rel->relids) && - !bms_is_empty(em->em_relids) && is_foreign_expr(root, rel, em->em_expr)) - return em; + break; } - return NULL; + eclass_member_iterator_dispose(&iter); + + return em; } /* @@ -7720,7 +7728,9 @@ find_em_for_rel_target(PlannerInfo *root, EquivalenceClass *ec, { Expr *expr = (Expr *) lfirst(lc1); Index sgref = get_pathtarget_sortgroupref(target, i); - ListCell *lc2; + EquivalenceMemberIterator iter; + EquivalenceMember *em; + Relids expr_relids; /* Ignore non-sort expressions */ if (sgref == 0 || @@ -7735,19 +7745,20 @@ find_em_for_rel_target(PlannerInfo *root, EquivalenceClass *ec, while (expr && IsA(expr, RelabelType)) expr = ((RelabelType *) expr)->arg; + expr_relids = pull_varnos(root, (Node *) expr); + setup_eclass_member_strict_iterator(&iter, root, ec, expr_relids, + false, false); + /* Locate an EquivalenceClass member matching this expr, if any */ - foreach(lc2, ec->ec_members) + while ((em = eclass_member_iterator_strict_next(&iter)) != NULL) { - EquivalenceMember *em = (EquivalenceMember *) lfirst(lc2); Expr *em_expr; - /* Don't match constants */ - if (em->em_is_const) - continue; + /* don't expect constants */ + Assert(!em->em_is_const); - /* Ignore child members */ - if (em->em_is_child) - continue; + /* don't expect child members */ + Assert(!em->em_is_child); /* Match if same expression (after stripping relabel) */ em_expr = em->em_expr; @@ -7759,10 +7770,15 @@ find_em_for_rel_target(PlannerInfo *root, EquivalenceClass *ec, /* Check that expression (including relabels!) is shippable */ if (is_foreign_expr(root, rel, em->em_expr)) + { + bms_free(expr_relids); + eclass_member_iterator_dispose(&iter); return em; + } } - i++; + bms_free(expr_relids); + eclass_member_iterator_dispose(&iter); } return NULL; diff --git a/src/backend/nodes/list.c b/src/backend/nodes/list.c index a709d23ef1..b8972bbe5a 100644 --- a/src/backend/nodes/list.c +++ b/src/backend/nodes/list.c @@ -1683,6 +1683,22 @@ list_sort(List *list, list_sort_comparator cmp) qsort(list->elements, len, sizeof(ListCell), (qsort_comparator) cmp); } +/* + * list_sort comparator for sorting a list into ascending ptr order. + */ +int +list_ptr_cmp(const ListCell *p1, const ListCell *p2) +{ + void *v1 = lfirst(p1); + void *v2 = lfirst(p2); + + if (v1 < v2) + return -1; + if (v1 > v2) + return 1; + return 0; +} + /* * list_sort comparator for sorting a list into ascending int order. */ diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6b368b08b2..98becac509 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -463,8 +463,11 @@ _outEquivalenceClass(StringInfo str, const EquivalenceClass *node) WRITE_NODE_FIELD(ec_opfamilies); WRITE_OID_FIELD(ec_collation); WRITE_NODE_FIELD(ec_members); - WRITE_NODE_FIELD(ec_sources); - WRITE_NODE_FIELD(ec_derives); + WRITE_BITMAPSET_FIELD(ec_member_indexes); + WRITE_BITMAPSET_FIELD(ec_nonchild_indexes); + WRITE_BITMAPSET_FIELD(ec_norel_indexes); + WRITE_BITMAPSET_FIELD(ec_source_indexes); + WRITE_BITMAPSET_FIELD(ec_derive_indexes); WRITE_BITMAPSET_FIELD(ec_relids); WRITE_BOOL_FIELD(ec_has_const); WRITE_BOOL_FIELD(ec_has_volatile); diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index 29ae32d960..595aaf74f3 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -5501,7 +5501,8 @@ get_foreign_key_join_selectivity(PlannerInfo *root, if (ec && ec->ec_has_const) { EquivalenceMember *em = fkinfo->fk_eclass_member[i]; - RestrictInfo *rinfo = find_derived_clause_for_ec_member(ec, + RestrictInfo *rinfo = find_derived_clause_for_ec_member(root, + ec, em); if (rinfo) diff --git a/src/backend/optimizer/path/equivclass.c b/src/backend/optimizer/path/equivclass.c index 7d7e6facdf..4277548193 100644 --- a/src/backend/optimizer/path/equivclass.c +++ b/src/backend/optimizer/path/equivclass.c @@ -31,9 +31,14 @@ #include "optimizer/restrictinfo.h" #include "utils/lsyscache.h" - -static EquivalenceMember *add_eq_member(EquivalenceClass *ec, - Expr *expr, Relids relids, Relids nullable_relids, +static void add_eq_source(PlannerInfo *root, EquivalenceClass *ec, + RestrictInfo *rinfo); +static void add_eq_derive(PlannerInfo *root, EquivalenceClass *ec, + RestrictInfo *rinfo); +static EquivalenceMember *add_eq_member(PlannerInfo *root, + EquivalenceClass *ec, + Expr *expr, Relids relids, + Relids nullable_relids, bool is_child, Oid datatype); static bool is_exprlist_member(Expr *node, List *exprs); static void generate_base_implied_equalities_const(PlannerInfo *root, @@ -333,7 +338,6 @@ process_equivalence(PlannerInfo *root, /* If case 1, nothing to do, except add to sources */ if (ec1 == ec2) { - ec1->ec_sources = lappend(ec1->ec_sources, restrictinfo); ec1->ec_below_outer_join |= below_outer_join; ec1->ec_min_security = Min(ec1->ec_min_security, restrictinfo->security_level); @@ -345,6 +349,8 @@ process_equivalence(PlannerInfo *root, /* mark the RI as usable with this pair of EMs */ restrictinfo->left_em = em1; restrictinfo->right_em = em2; + + add_eq_source(root, ec1, restrictinfo); return true; } @@ -365,8 +371,16 @@ process_equivalence(PlannerInfo *root, * be found. */ ec1->ec_members = list_concat(ec1->ec_members, ec2->ec_members); - ec1->ec_sources = list_concat(ec1->ec_sources, ec2->ec_sources); - ec1->ec_derives = list_concat(ec1->ec_derives, ec2->ec_derives); + ec1->ec_member_indexes = bms_add_members(ec1->ec_member_indexes, + ec2->ec_member_indexes); + ec1->ec_nonchild_indexes = bms_add_members(ec1->ec_nonchild_indexes, + ec2->ec_nonchild_indexes); + ec1->ec_norel_indexes = bms_add_members(ec1->ec_norel_indexes, + ec2->ec_norel_indexes); + ec1->ec_source_indexes = bms_join(ec1->ec_source_indexes, + ec2->ec_source_indexes); + ec1->ec_derive_indexes = bms_join(ec1->ec_derive_indexes, + ec2->ec_derive_indexes); ec1->ec_relids = bms_join(ec1->ec_relids, ec2->ec_relids); ec1->ec_has_const |= ec2->ec_has_const; /* can't need to set has_volatile */ @@ -379,10 +393,12 @@ process_equivalence(PlannerInfo *root, root->eq_classes = list_delete_nth_cell(root->eq_classes, ec2_idx); /* just to avoid debugging confusion w/ dangling pointers: */ ec2->ec_members = NIL; - ec2->ec_sources = NIL; - ec2->ec_derives = NIL; + ec2->ec_member_indexes = NULL; + ec2->ec_nonchild_indexes = NULL; + ec2->ec_norel_indexes = NULL; + ec2->ec_source_indexes = NULL; + ec2->ec_derive_indexes = NULL; ec2->ec_relids = NULL; - ec1->ec_sources = lappend(ec1->ec_sources, restrictinfo); ec1->ec_below_outer_join |= below_outer_join; ec1->ec_min_security = Min(ec1->ec_min_security, restrictinfo->security_level); @@ -394,13 +410,14 @@ process_equivalence(PlannerInfo *root, /* mark the RI as usable with this pair of EMs */ restrictinfo->left_em = em1; restrictinfo->right_em = em2; + + add_eq_source(root, ec1, restrictinfo); } else if (ec1) { /* Case 3: add item2 to ec1 */ - em2 = add_eq_member(ec1, item2, item2_relids, item2_nullable_relids, - false, item2_type); - ec1->ec_sources = lappend(ec1->ec_sources, restrictinfo); + em2 = add_eq_member(root, ec1, item2, item2_relids, + item2_nullable_relids, false, item2_type); ec1->ec_below_outer_join |= below_outer_join; ec1->ec_min_security = Min(ec1->ec_min_security, restrictinfo->security_level); @@ -412,13 +429,14 @@ process_equivalence(PlannerInfo *root, /* mark the RI as usable with this pair of EMs */ restrictinfo->left_em = em1; restrictinfo->right_em = em2; + + add_eq_source(root, ec1, restrictinfo); } else if (ec2) { /* Case 3: add item1 to ec2 */ - em1 = add_eq_member(ec2, item1, item1_relids, item1_nullable_relids, - false, item1_type); - ec2->ec_sources = lappend(ec2->ec_sources, restrictinfo); + em1 = add_eq_member(root, ec2, item1, item1_relids, + item1_nullable_relids, false, item1_type); ec2->ec_below_outer_join |= below_outer_join; ec2->ec_min_security = Min(ec2->ec_min_security, restrictinfo->security_level); @@ -430,6 +448,8 @@ process_equivalence(PlannerInfo *root, /* mark the RI as usable with this pair of EMs */ restrictinfo->left_em = em1; restrictinfo->right_em = em2; + + add_eq_source(root, ec2, restrictinfo); } else { @@ -439,8 +459,11 @@ process_equivalence(PlannerInfo *root, ec->ec_opfamilies = opfamilies; ec->ec_collation = collation; ec->ec_members = NIL; - ec->ec_sources = list_make1(restrictinfo); - ec->ec_derives = NIL; + ec->ec_member_indexes = NULL; + ec->ec_nonchild_indexes = NULL; + ec->ec_norel_indexes = NULL; + ec->ec_source_indexes = NULL; + ec->ec_derive_indexes = NULL; ec->ec_relids = NULL; ec->ec_has_const = false; ec->ec_has_volatile = false; @@ -450,10 +473,10 @@ process_equivalence(PlannerInfo *root, ec->ec_min_security = restrictinfo->security_level; ec->ec_max_security = restrictinfo->security_level; ec->ec_merged = NULL; - em1 = add_eq_member(ec, item1, item1_relids, item1_nullable_relids, - false, item1_type); - em2 = add_eq_member(ec, item2, item2_relids, item2_nullable_relids, - false, item2_type); + em1 = add_eq_member(root, ec, item1, item1_relids, + item1_nullable_relids, false, item1_type); + em2 = add_eq_member(root, ec, item2, item2_relids, + item2_nullable_relids, false, item2_type); root->eq_classes = lappend(root->eq_classes, ec); @@ -463,6 +486,8 @@ process_equivalence(PlannerInfo *root, /* mark the RI as usable with this pair of EMs */ restrictinfo->left_em = em1; restrictinfo->right_em = em2; + + add_eq_source(root, ec, restrictinfo); } return true; @@ -538,14 +563,62 @@ canonicalize_ec_expression(Expr *expr, Oid req_type, Oid req_collation) return expr; } +/* + * add_eq_source - add 'rinfo' in eq_sources for this 'ec' + */ +static void +add_eq_source(PlannerInfo *root, EquivalenceClass *ec, RestrictInfo *rinfo) +{ + int source_idx = list_length(root->eq_sources); + int i; + + ec->ec_source_indexes = bms_add_member(ec->ec_source_indexes, source_idx); + root->eq_sources = lappend(root->eq_sources, rinfo); + + i = -1; + while ((i = bms_next_member(rinfo->clause_relids, i)) >= 0) + { + RelOptInfo *rel = root->simple_rel_array[i]; + + rel->eclass_source_indexes = bms_add_member(rel->eclass_source_indexes, + source_idx); + } +} + +/* + * add_eq_derive - add 'rinfo' in eq_derives for this 'ec' + */ +static void +add_eq_derive(PlannerInfo *root, EquivalenceClass *ec, RestrictInfo *rinfo) +{ + int derive_idx = list_length(root->eq_derives); + int i; + + ec->ec_derive_indexes = bms_add_member(ec->ec_derive_indexes, derive_idx); + root->eq_derives = lappend(root->eq_derives, rinfo); + + i = -1; + while ((i = bms_next_member(rinfo->clause_relids, i)) >= 0) + { + RelOptInfo *rel = root->simple_rel_array[i]; + + rel->eclass_derive_indexes = bms_add_member(rel->eclass_derive_indexes, + derive_idx); + } +} + /* * add_eq_member - build a new EquivalenceMember and add it to an EC */ static EquivalenceMember * -add_eq_member(EquivalenceClass *ec, Expr *expr, Relids relids, - Relids nullable_relids, bool is_child, Oid datatype) +add_eq_member(PlannerInfo *root, EquivalenceClass *ec, Expr *expr, + Relids relids, Relids nullable_relids, bool is_child, + Oid datatype) { EquivalenceMember *em = makeNode(EquivalenceMember); + Relids expr_relids; + int em_index = list_length(root->eq_members); + int i; em->em_expr = expr; em->em_relids = relids; @@ -554,6 +627,23 @@ add_eq_member(EquivalenceClass *ec, Expr *expr, Relids relids, em->em_is_child = is_child; em->em_datatype = datatype; + /* + * We must determine the exact set of relids in the expr for child + * EquivalenceMembers as what is given to us in 'relids' may differ from + * the relids mentioned in the expression. See add_child_rel_equivalences + */ + if (is_child) + expr_relids = pull_varnos(root, (Node *) expr); + else + { + expr_relids = relids; + /* We expect the relids to match for non-child members */ + Assert(bms_equal(pull_varnos(root, (Node *) expr), relids)); + } + + /* record the actual relids from 'expr' */ + em->em_norel_expr = bms_is_empty(expr_relids); + if (bms_is_empty(relids)) { /* @@ -572,9 +662,31 @@ add_eq_member(EquivalenceClass *ec, Expr *expr, Relids relids, else if (!is_child) /* child members don't add to ec_relids */ { ec->ec_relids = bms_add_members(ec->ec_relids, relids); + ec->ec_nonchild_indexes = bms_add_member(ec->ec_nonchild_indexes, em_index); } + + /* add the new member to the list */ ec->ec_members = lappend(ec->ec_members, em); + /* and add it to the index and PlannerInfo's list */ + ec->ec_member_indexes = bms_add_member(ec->ec_member_indexes, em_index); + root->eq_members = lappend(root->eq_members, em); + + /* record exprs with no relids */ + if (bms_is_empty(expr_relids)) + ec->ec_norel_indexes = bms_add_member(ec->ec_norel_indexes, em_index); + + if (is_child) + expr_relids = bms_add_members(expr_relids, relids); + + i = -1; + while ((i = bms_next_member(expr_relids, i)) >= 0) + { + RelOptInfo *rel = root->simple_rel_array[i]; + + rel->eclass_member_indexes = bms_add_member(rel->eclass_member_indexes, em_index); + } + return em; } @@ -638,6 +750,7 @@ get_eclass_for_sort_expr(PlannerInfo *root, * Ensure the expression exposes the correct type and collation. */ expr = canonicalize_ec_expression(expr, opcintype, collation); + expr_relids = pull_varnos(root, (Node *) expr); /* * Scan through the existing EquivalenceClasses for a match @@ -645,7 +758,8 @@ get_eclass_for_sort_expr(PlannerInfo *root, foreach(lc1, root->eq_classes) { EquivalenceClass *cur_ec = (EquivalenceClass *) lfirst(lc1); - ListCell *lc2; + EquivalenceMemberIterator iter; + EquivalenceMember *cur_em; /* * Never match to a volatile EC, except when we are looking at another @@ -660,10 +774,11 @@ get_eclass_for_sort_expr(PlannerInfo *root, if (!equal(opfamilies, cur_ec->ec_opfamilies)) continue; - foreach(lc2, cur_ec->ec_members) - { - EquivalenceMember *cur_em = (EquivalenceMember *) lfirst(lc2); + setup_eclass_member_strict_iterator(&iter, root, cur_ec, expr_relids, + true, true); + while ((cur_em = eclass_member_iterator_strict_next(&iter)) != NULL) + { /* * Ignore child members unless they match the request. */ @@ -683,6 +798,8 @@ get_eclass_for_sort_expr(PlannerInfo *root, equal(expr, cur_em->em_expr)) return cur_ec; /* Match! */ } + + eclass_member_iterator_dispose(&iter); } /* No match; does caller want a NULL result? */ @@ -700,8 +817,11 @@ get_eclass_for_sort_expr(PlannerInfo *root, newec->ec_opfamilies = list_copy(opfamilies); newec->ec_collation = collation; newec->ec_members = NIL; - newec->ec_sources = NIL; - newec->ec_derives = NIL; + newec->ec_member_indexes = NULL; + newec->ec_nonchild_indexes = NULL; + newec->ec_norel_indexes = NULL; + newec->ec_source_indexes = NULL; + newec->ec_derive_indexes = NULL; newec->ec_relids = NULL; newec->ec_has_const = false; newec->ec_has_volatile = contain_volatile_functions((Node *) expr); @@ -718,10 +838,9 @@ get_eclass_for_sort_expr(PlannerInfo *root, /* * Get the precise set of nullable relids appearing in the expression. */ - expr_relids = pull_varnos(root, (Node *) expr); nullable_relids = bms_intersect(nullable_relids, expr_relids); - newem = add_eq_member(newec, copyObject(expr), expr_relids, + newem = add_eq_member(root, newec, copyObject(expr), expr_relids, nullable_relids, false, opcintype); /* @@ -753,7 +872,7 @@ get_eclass_for_sort_expr(PlannerInfo *root, int ec_index = list_length(root->eq_classes) - 1; int i = -1; - while ((i = bms_next_member(newec->ec_relids, i)) > 0) + while ((i = bms_next_member(newec->ec_relids, i)) >= 0) { RelOptInfo *rel = root->simple_rel_array[i]; @@ -783,19 +902,25 @@ get_eclass_for_sort_expr(PlannerInfo *root, * Child EC members are ignored unless they belong to given 'relids'. */ EquivalenceMember * -find_ec_member_matching_expr(EquivalenceClass *ec, +find_ec_member_matching_expr(PlannerInfo *root, + EquivalenceClass *ec, Expr *expr, Relids relids) { - ListCell *lc; + EquivalenceMemberIterator iter; + Relids expr_relids; + EquivalenceMember *em; /* We ignore binary-compatible relabeling on both ends */ while (expr && IsA(expr, RelabelType)) expr = ((RelabelType *) expr)->arg; - foreach(lc, ec->ec_members) + expr_relids = pull_varnos(root, (Node *) expr); + setup_eclass_member_strict_iterator(&iter, root, ec, expr_relids, true, + true); + + while ((em = eclass_member_iterator_strict_next(&iter)) != NULL) { - EquivalenceMember *em = (EquivalenceMember *) lfirst(lc); Expr *emexpr; /* @@ -820,10 +945,13 @@ find_ec_member_matching_expr(EquivalenceClass *ec, emexpr = ((RelabelType *) emexpr)->arg; if (equal(emexpr, expr)) - return em; + break; } - return NULL; + bms_free(expr_relids); + eclass_member_iterator_dispose(&iter); + + return em; } /* @@ -965,7 +1093,7 @@ relation_can_be_sorted_early(PlannerInfo *root, RelOptInfo *rel, { Expr *targetexpr = (Expr *) lfirst(lc); - em = find_ec_member_matching_expr(ec, targetexpr, rel->relids); + em = find_ec_member_matching_expr(root, ec, targetexpr, rel->relids); if (!em) continue; @@ -1052,7 +1180,7 @@ relation_can_be_sorted_early(PlannerInfo *root, RelOptInfo *rel, * scanning of the quals and before Path construction begins. * * We make no attempt to avoid generating duplicate RestrictInfos here: we - * don't search ec_sources or ec_derives for matches. It doesn't really + * don't search eq_sources or eq_derives for matches. It doesn't really * seem worth the trouble to do so. */ void @@ -1109,7 +1237,7 @@ generate_base_implied_equalities(PlannerInfo *root) * this is a cheap version of has_relevant_eclass_joinclause(). */ i = -1; - while ((i = bms_next_member(ec->ec_relids, i)) > 0) + while ((i = bms_next_member(ec->ec_relids, i)) >= 0) { RelOptInfo *rel = root->simple_rel_array[i]; @@ -1135,6 +1263,7 @@ generate_base_implied_equalities_const(PlannerInfo *root, { EquivalenceMember *const_em = NULL; ListCell *lc; + int i; /* * In the trivial case where we just had one "var = const" clause, push @@ -1144,9 +1273,9 @@ generate_base_implied_equalities_const(PlannerInfo *root, * equivalent to the old one. */ if (list_length(ec->ec_members) == 2 && - list_length(ec->ec_sources) == 1) + bms_get_singleton_member(ec->ec_source_indexes, &i)) { - RestrictInfo *restrictinfo = (RestrictInfo *) linitial(ec->ec_sources); + RestrictInfo *restrictinfo = list_nth_node(RestrictInfo, root->eq_sources, i); if (bms_membership(restrictinfo->required_relids) != BMS_MULTIPLE) { @@ -1161,9 +1290,11 @@ generate_base_implied_equalities_const(PlannerInfo *root, * machinery might be able to exclude relations on the basis of generated * "var = const" equalities, but "var = param" won't work for that. */ - foreach(lc, ec->ec_members) + i = -1; + while ((i = bms_next_member(ec->ec_norel_indexes, i)) >= 0) { - EquivalenceMember *cur_em = (EquivalenceMember *) lfirst(lc); + EquivalenceMember *cur_em = list_nth_node(EquivalenceMember, + root->eq_members, i); if (cur_em->em_is_const) { @@ -1204,9 +1335,9 @@ generate_base_implied_equalities_const(PlannerInfo *root, /* * If the clause didn't degenerate to a constant, fill in the correct - * markings for a mergejoinable clause, and save it in ec_derives. (We + * markings for a mergejoinable clause, and save it in eq_derives. (We * will not re-use such clauses directly, but selectivity estimation - * may consult the list later. Note that this use of ec_derives does + * may consult the list later. Note that this use of eq_derives does * not overlap with its use for join clauses, since we never generate * join clauses from an ec_has_const eclass.) */ @@ -1216,7 +1347,8 @@ generate_base_implied_equalities_const(PlannerInfo *root, rinfo->left_ec = rinfo->right_ec = ec; rinfo->left_em = cur_em; rinfo->right_em = const_em; - ec->ec_derives = lappend(ec->ec_derives, rinfo); + + add_eq_derive(root, ec, rinfo); } } } @@ -1229,7 +1361,8 @@ generate_base_implied_equalities_no_const(PlannerInfo *root, EquivalenceClass *ec) { EquivalenceMember **prev_ems; - ListCell *lc; + Bitmapset *matching_ems; + int i; /* * We scan the EC members once and track the last-seen member for each @@ -1242,9 +1375,12 @@ generate_base_implied_equalities_no_const(PlannerInfo *root, prev_ems = (EquivalenceMember **) palloc0(root->simple_rel_array_size * sizeof(EquivalenceMember *)); - foreach(lc, ec->ec_members) + matching_ems = bms_difference(ec->ec_member_indexes, ec->ec_norel_indexes); + i = -1; + while ((i = bms_next_member(matching_ems, i)) >= 0) { - EquivalenceMember *cur_em = (EquivalenceMember *) lfirst(lc); + EquivalenceMember *cur_em = list_nth_node(EquivalenceMember, + root->eq_members, i); int relid; Assert(!cur_em->em_is_child); /* no children yet */ @@ -1279,7 +1415,7 @@ generate_base_implied_equalities_no_const(PlannerInfo *root, /* * If the clause didn't degenerate to a constant, fill in the * correct markings for a mergejoinable clause. We don't put it - * in ec_derives however; we don't currently need to re-find such + * in eq_derives however; we don't currently need to re-find such * clauses, and we don't want to clutter that list with non-join * clauses. */ @@ -1302,11 +1438,15 @@ generate_base_implied_equalities_no_const(PlannerInfo *root, * For the moment we force all the Vars to be available at all join nodes * for this eclass. Perhaps this could be improved by doing some * pre-analysis of which members we prefer to join, but it's no worse than - * what happened in the pre-8.3 code. + * what happened in the pre-8.3 code. We're able to make use of + * matching_ems from above. We're not going to find Vars in + * em_const_indexes. */ - foreach(lc, ec->ec_members) + i = -1; + while ((i = bms_next_member(matching_ems, i)) >= 0) { - EquivalenceMember *cur_em = (EquivalenceMember *) lfirst(lc); + EquivalenceMember *cur_em = list_nth_node(EquivalenceMember, + root->eq_members, i); List *vars = pull_var_clause((Node *) cur_em->em_expr, PVC_RECURSE_AGGREGATES | PVC_RECURSE_WINDOWFUNCS | @@ -1315,6 +1455,7 @@ generate_base_implied_equalities_no_const(PlannerInfo *root, add_vars_to_targetlist(root, vars, ec->ec_relids); list_free(vars); } + bms_free(matching_ems); } /* @@ -1335,11 +1476,12 @@ static void generate_base_implied_equalities_broken(PlannerInfo *root, EquivalenceClass *ec) { - ListCell *lc; + int i = -1; - foreach(lc, ec->ec_sources) + while ((i = bms_next_member(ec->ec_source_indexes, i)) >= 0) { - RestrictInfo *restrictinfo = (RestrictInfo *) lfirst(lc); + RestrictInfo *restrictinfo = list_nth_node(RestrictInfo, + root->eq_sources, i); if (ec->ec_has_const || bms_membership(restrictinfo->required_relids) != BMS_MULTIPLE) @@ -1380,11 +1522,11 @@ generate_base_implied_equalities_broken(PlannerInfo *root, * Because the same join clauses are likely to be needed multiple times as * we consider different join paths, we avoid generating multiple copies: * whenever we select a particular pair of EquivalenceMembers to join, - * we check to see if the pair matches any original clause (in ec_sources) - * or previously-built clause (in ec_derives). This saves memory and allows - * re-use of information cached in RestrictInfos. We also avoid generating - * commutative duplicates, i.e. if the algorithm selects "a.x = b.y" but - * we already have "b.y = a.x", we return the existing clause. + * we check to see if the pair matches any original clause in root's + * eq_sources or previously-built clause (in root's eq_derives). This saves + * memory and allows re-use of information cached in RestrictInfos. We also + * avoid generating commutative duplicates, i.e. if the algorithm selects + * "a.x = b.y" but we already have "b.y = a.x", we return the existing clause. * * join_relids should always equal bms_union(outer_relids, inner_rel->relids). * We could simplify this function's API by computing it internally, but in @@ -1546,6 +1688,8 @@ generate_join_implied_equalities_normal(PlannerInfo *root, Relids outer_relids, Relids inner_relids) { + EquivalenceMemberIterator iter; + EquivalenceMember *cur_em; List *result = NIL; List *new_members = NIL; List *outer_members = NIL; @@ -1561,10 +1705,10 @@ generate_join_implied_equalities_normal(PlannerInfo *root, * as well as to at least one input member, plus enforce at least one * outer-rel member equal to at least one inner-rel member. */ - foreach(lc1, ec->ec_members) - { - EquivalenceMember *cur_em = (EquivalenceMember *) lfirst(lc1); + setup_eclass_member_iterator(&iter, root, ec, join_relids, true, false); + while ((cur_em = eclass_member_iterator_next(&iter)) != NULL) + { /* * We don't need to check explicitly for child EC members. This test * against join_relids will cause them to be ignored except when @@ -1581,6 +1725,8 @@ generate_join_implied_equalities_normal(PlannerInfo *root, new_members = lappend(new_members, cur_em); } + eclass_member_iterator_dispose(&iter); + /* * First, select the joinclause if needed. We can equate any one outer * member to any one inner member, but we have to find a datatype @@ -1678,7 +1824,7 @@ generate_join_implied_equalities_normal(PlannerInfo *root, foreach(lc1, new_members) { - EquivalenceMember *cur_em = (EquivalenceMember *) lfirst(lc1); + cur_em = (EquivalenceMember *) lfirst(lc1); if (prev_em != NULL) { @@ -1723,12 +1869,16 @@ generate_join_implied_equalities_broken(PlannerInfo *root, Relids nominal_inner_relids, RelOptInfo *inner_rel) { + Bitmapset *matching_es; List *result = NIL; - ListCell *lc; + int i; - foreach(lc, ec->ec_sources) + matching_es = get_ec_source_indexes(root, ec, nominal_join_relids); + i = -1; + while ((i = bms_next_member(matching_es, i)) >= 0) { - RestrictInfo *restrictinfo = (RestrictInfo *) lfirst(lc); + RestrictInfo *restrictinfo = list_nth_node(RestrictInfo, + root->eq_sources, i); Relids clause_relids = restrictinfo->required_relids; if (bms_is_subset(clause_relids, nominal_join_relids) && @@ -1740,12 +1890,12 @@ generate_join_implied_equalities_broken(PlannerInfo *root, /* * If we have to translate, just brute-force apply adjust_appendrel_attrs * to all the RestrictInfos at once. This will result in returning - * RestrictInfos that are not listed in ec_derives, but there shouldn't be + * RestrictInfos that are not listed in eq_derives, but there shouldn't be * any duplication, and it's a sufficiently narrow corner case that we * shouldn't sweat too much over it anyway. * * Since inner_rel might be an indirect descendant of the baserel - * mentioned in the ec_sources clauses, we have to be prepared to apply + * mentioned in the eq_sources clauses, we have to be prepared to apply * multiple levels of Var translation. */ if (IS_OTHER_REL(inner_rel) && result != NIL) @@ -1807,9 +1957,10 @@ create_join_clause(PlannerInfo *root, EquivalenceMember *rightem, EquivalenceClass *parent_ec) { + Bitmapset *matches; RestrictInfo *rinfo; - ListCell *lc; MemoryContext oldcontext; + int i; /* * Search to see if we already built a RestrictInfo for this pair of @@ -1820,9 +1971,12 @@ create_join_clause(PlannerInfo *root, * it's not identical, it'd better have the same effects, or the operator * families we're using are broken. */ - foreach(lc, ec->ec_sources) + matches = bms_int_members(get_ec_source_indexes_strict(root, ec, leftem->em_relids), + get_ec_source_indexes_strict(root, ec, rightem->em_relids)); + i = -1; + while ((i = bms_next_member(matches, i)) >= 0) { - rinfo = (RestrictInfo *) lfirst(lc); + rinfo = list_nth_node(RestrictInfo, root->eq_sources, i); if (rinfo->left_em == leftem && rinfo->right_em == rightem && rinfo->parent_ec == parent_ec) @@ -1833,9 +1987,13 @@ create_join_clause(PlannerInfo *root, return rinfo; } - foreach(lc, ec->ec_derives) + matches = bms_int_members(get_ec_derive_indexes_strict(root, ec, leftem->em_relids), + get_ec_derive_indexes_strict(root, ec, rightem->em_relids)); + + i = -1; + while ((i = bms_next_member(matches, i)) >= 0) { - rinfo = (RestrictInfo *) lfirst(lc); + rinfo = list_nth_node(RestrictInfo, root->eq_derives, i); if (rinfo->left_em == leftem && rinfo->right_em == rightem && rinfo->parent_ec == parent_ec) @@ -1877,7 +2035,7 @@ create_join_clause(PlannerInfo *root, rinfo->left_em = leftem; rinfo->right_em = rightem; /* and save it for possible re-use */ - ec->ec_derives = lappend(ec->ec_derives, rinfo); + add_eq_derive(root, ec, rinfo); MemoryContextSwitchTo(oldcontext); @@ -2069,7 +2227,8 @@ reconsider_outer_join_clause(PlannerInfo *root, RestrictInfo *rinfo, left_type, right_type, inner_datatype; - Relids inner_relids, + Relids outer_relids, + inner_relids, inner_nullable_relids; ListCell *lc1; @@ -2088,6 +2247,7 @@ reconsider_outer_join_clause(PlannerInfo *root, RestrictInfo *rinfo, outervar = (Expr *) get_leftop(rinfo->clause); innervar = (Expr *) get_rightop(rinfo->clause); inner_datatype = right_type; + outer_relids = rinfo->left_relids; inner_relids = rinfo->right_relids; } else @@ -2095,6 +2255,7 @@ reconsider_outer_join_clause(PlannerInfo *root, RestrictInfo *rinfo, outervar = (Expr *) get_rightop(rinfo->clause); innervar = (Expr *) get_leftop(rinfo->clause); inner_datatype = left_type; + outer_relids = rinfo->right_relids; inner_relids = rinfo->left_relids; } inner_nullable_relids = bms_intersect(inner_relids, @@ -2104,8 +2265,10 @@ reconsider_outer_join_clause(PlannerInfo *root, RestrictInfo *rinfo, foreach(lc1, root->eq_classes) { EquivalenceClass *cur_ec = (EquivalenceClass *) lfirst(lc1); + EquivalenceMemberIterator iter; + EquivalenceMember *cur_em; bool match; - ListCell *lc2; + int i; /* Ignore EC unless it contains pseudoconstants */ if (!cur_ec->ec_has_const) @@ -2120,10 +2283,12 @@ reconsider_outer_join_clause(PlannerInfo *root, RestrictInfo *rinfo, continue; /* Does it contain a match to outervar? */ match = false; - foreach(lc2, cur_ec->ec_members) - { - EquivalenceMember *cur_em = (EquivalenceMember *) lfirst(lc2); + setup_eclass_member_strict_iterator(&iter, root, cur_ec, outer_relids, + false, true); + + while ((cur_em = eclass_member_iterator_strict_next(&iter)) != NULL) + { Assert(!cur_em->em_is_child); /* no children yet */ if (equal(outervar, cur_em->em_expr)) { @@ -2131,6 +2296,8 @@ reconsider_outer_join_clause(PlannerInfo *root, RestrictInfo *rinfo, break; } } + eclass_member_iterator_dispose(&iter); + if (!match) continue; /* no match, so ignore this EC */ @@ -2140,12 +2307,14 @@ reconsider_outer_join_clause(PlannerInfo *root, RestrictInfo *rinfo, * constant before we can decide to throw away the outer-join clause. */ match = false; - foreach(lc2, cur_ec->ec_members) + i = -1; + while ((i = bms_next_member(cur_ec->ec_norel_indexes, i)) >= 0) { - EquivalenceMember *cur_em = (EquivalenceMember *) lfirst(lc2); Oid eq_op; RestrictInfo *newrinfo; + cur_em = list_nth_node(EquivalenceMember, root->eq_members, i); + if (!cur_em->em_is_const) continue; /* ignore non-const members */ eq_op = select_equality_operator(cur_ec, @@ -2221,11 +2390,12 @@ reconsider_full_join_clause(PlannerInfo *root, RestrictInfo *rinfo) { EquivalenceClass *cur_ec = (EquivalenceClass *) lfirst(lc1); EquivalenceMember *coal_em = NULL; + Bitmapset *matching_ems; bool match; bool matchleft; bool matchright; - ListCell *lc2; int coal_idx = -1; + int i; /* Ignore EC unless it contains pseudoconstants */ if (!cur_ec->ec_has_const) @@ -2252,10 +2422,14 @@ reconsider_full_join_clause(PlannerInfo *root, RestrictInfo *rinfo) * the two column types). Is it OK to strip implicit coercions from * the COALESCE arguments? */ + matching_ems = get_ecmember_indexes_strict(root, cur_ec, + rinfo->clause_relids, true, + false); match = false; - foreach(lc2, cur_ec->ec_members) + i = -1; + while ((i = bms_next_member(matching_ems, i)) >= 0) { - coal_em = (EquivalenceMember *) lfirst(lc2); + coal_em = list_nth_node(EquivalenceMember, root->eq_members, i); Assert(!coal_em->em_is_child); /* no children yet */ if (IsA(coal_em->em_expr, CoalesceExpr)) { @@ -2270,7 +2444,7 @@ reconsider_full_join_clause(PlannerInfo *root, RestrictInfo *rinfo) if (equal(leftvar, cfirst) && equal(rightvar, csecond)) { - coal_idx = foreach_current_index(lc2); + coal_idx = i; match = true; break; } @@ -2286,9 +2460,11 @@ reconsider_full_join_clause(PlannerInfo *root, RestrictInfo *rinfo) * decide to throw away the outer-join clause. */ matchleft = matchright = false; - foreach(lc2, cur_ec->ec_members) + i = -1; + while ((i = bms_next_member(cur_ec->ec_norel_indexes, i)) >= 0) { - EquivalenceMember *cur_em = (EquivalenceMember *) lfirst(lc2); + EquivalenceMember *cur_em = list_nth_node(EquivalenceMember, + root->eq_members, i); Oid eq_op; RestrictInfo *newrinfo; @@ -2333,11 +2509,28 @@ reconsider_full_join_clause(PlannerInfo *root, RestrictInfo *rinfo) * we can throw away the full-join clause as redundant. Moreover, we * can remove the COALESCE entry from the EC, since the added * restrictions ensure it will always have the expected value. (We - * don't bother trying to update ec_relids or ec_sources.) + * don't bother trying to update ec_relids or root's eq_sources.) */ if (matchleft && matchright) { - cur_ec->ec_members = list_delete_nth_cell(cur_ec->ec_members, coal_idx); + /* XXX performance of list_delete()?? */ + cur_ec->ec_members = list_delete(cur_ec->ec_members, coal_em); + cur_ec->ec_member_indexes = bms_del_member(cur_ec->ec_member_indexes, + coal_idx); + cur_ec->ec_nonchild_indexes = bms_del_member(cur_ec->ec_nonchild_indexes, + coal_idx); + /* no need to adjust ec_norel_indexes */ + + /* Remove the member from each of the relations */ + i = -1; + while ((i = bms_next_member(coal_em->em_relids, i)) >= 0) + { + RelOptInfo *rel = root->simple_rel_array[i]; + + rel->eclass_member_indexes = bms_del_member(rel->eclass_member_indexes, + coal_idx); + } + return true; } @@ -2369,21 +2562,28 @@ bool exprs_known_equal(PlannerInfo *root, Node *item1, Node *item2) { ListCell *lc1; + Relids item1_relids = pull_varnos(root, item1); + Relids item2_relids = pull_varnos(root, item2); foreach(lc1, root->eq_classes) { EquivalenceClass *ec = (EquivalenceClass *) lfirst(lc1); bool item1member = false; bool item2member = false; - ListCell *lc2; + Bitmapset *matching_ems; + int i; /* Never match to a volatile EC */ if (ec->ec_has_volatile) continue; - foreach(lc2, ec->ec_members) + matching_ems = bms_join(get_ecmember_indexes_strict(root, ec, item1_relids, false, true), + get_ecmember_indexes_strict(root, ec, item2_relids, false, true)); + i = -1; + while ((i = bms_next_member(matching_ems, i)) >= 0) { - EquivalenceMember *em = (EquivalenceMember *) lfirst(lc2); + EquivalenceMember *em = list_nth_node(EquivalenceMember, + root->eq_members, i); if (em->em_is_child) continue; /* ignore children here */ @@ -2446,16 +2646,21 @@ match_eclasses_to_foreign_key_col(PlannerInfo *root, i); EquivalenceMember *item1_em = NULL; EquivalenceMember *item2_em = NULL; - ListCell *lc2; + Bitmapset *matching_ems; + int j; /* Never match to a volatile EC */ if (ec->ec_has_volatile) continue; /* Note: it seems okay to match to "broken" eclasses here */ + matching_ems = bms_join(get_ecmember_indexes_strict(root, ec, rel1->relids, false, false), + get_ecmember_indexes_strict(root, ec, rel2->relids, false, false)); - foreach(lc2, ec->ec_members) + j = -1; + while ((j = bms_next_member(matching_ems, j)) >= 0) { - EquivalenceMember *em = (EquivalenceMember *) lfirst(lc2); + EquivalenceMember *em = list_nth_node(EquivalenceMember, + root->eq_members, j); Var *var; if (em->em_is_child) @@ -2508,16 +2713,19 @@ match_eclasses_to_foreign_key_col(PlannerInfo *root, * Returns NULL if no such clause can be found. */ RestrictInfo * -find_derived_clause_for_ec_member(EquivalenceClass *ec, +find_derived_clause_for_ec_member(PlannerInfo *root, EquivalenceClass *ec, EquivalenceMember *em) { - ListCell *lc; + int i; Assert(ec->ec_has_const); Assert(!em->em_is_const); - foreach(lc, ec->ec_derives) + + i = -1; + while ((i = bms_next_member(ec->ec_derive_indexes, i)) >= 0) { - RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); + RestrictInfo *rinfo = list_nth_node(RestrictInfo, root->eq_derives, + i); /* * generate_base_implied_equalities_const will have put non-const @@ -2568,7 +2776,8 @@ add_child_rel_equivalences(PlannerInfo *root, while ((i = bms_next_member(parent_rel->eclass_indexes, i)) >= 0) { EquivalenceClass *cur_ec = (EquivalenceClass *) list_nth(root->eq_classes, i); - int num_members; + EquivalenceMemberIterator iter; + EquivalenceMember *cur_em; /* * If this EC contains a volatile expression, then generating child @@ -2582,86 +2791,72 @@ add_child_rel_equivalences(PlannerInfo *root, Assert(bms_is_subset(top_parent_relids, cur_ec->ec_relids)); /* - * We don't use foreach() here because there's no point in scanning - * newly-added child members, so we can stop after the last - * pre-existing EC member. + * Looping using the EquivalenceMemberIterator means we only loop over + * the members which existed when we set up the iterator, not newly + * added ones. */ - num_members = list_length(cur_ec->ec_members); - for (int pos = 0; pos < num_members; pos++) + setup_eclass_member_iterator(&iter, root, cur_ec, top_parent_relids, + false, false); + + while ((cur_em = eclass_member_iterator_next(&iter)) != NULL) { - EquivalenceMember *cur_em = (EquivalenceMember *) list_nth(cur_ec->ec_members, pos); + Expr *child_expr; + Relids new_relids; + Relids new_nullable_relids; - if (cur_em->em_is_const) - continue; /* ignore consts here */ + Assert(!cur_em->em_is_const); + Assert(!cur_em->em_is_child); + Assert(bms_overlap(cur_em->em_relids, top_parent_relids)); + + if (parent_rel->reloptkind == RELOPT_BASEREL) + { + /* Simple single-level transformation */ + child_expr = (Expr *) + adjust_appendrel_attrs(root, + (Node *) cur_em->em_expr, + 1, &appinfo); + } + else + { + /* Must do multi-level transformation */ + child_expr = (Expr *) + adjust_appendrel_attrs_multilevel(root, + (Node *) cur_em->em_expr, + child_rel, + child_rel->top_parent); + } /* - * We consider only original EC members here, not - * already-transformed child members. Otherwise, if some original - * member expression references more than one appendrel, we'd get - * an O(N^2) explosion of useless derived expressions for - * combinations of children. (But add_child_join_rel_equivalences - * may add targeted combinations for partitionwise-join purposes.) + * Transform em_relids to match. Note we do *not* do + * pull_varnos(child_expr) here, as for example the transformation + * might have substituted a constant, but we don't want the child + * member to be marked as constant. */ - if (cur_em->em_is_child) - continue; /* ignore children here */ + new_relids = bms_difference(cur_em->em_relids, + top_parent_relids); + new_relids = bms_add_members(new_relids, child_relids); - /* Does this member reference child's topmost parent rel? */ - if (bms_overlap(cur_em->em_relids, top_parent_relids)) + /* + * And likewise for nullable_relids. Note this code assumes + * parent and child relids are singletons. + */ + new_nullable_relids = cur_em->em_nullable_relids; + if (bms_overlap(new_nullable_relids, top_parent_relids)) { - /* Yes, generate transformed child version */ - Expr *child_expr; - Relids new_relids; - Relids new_nullable_relids; - - if (parent_rel->reloptkind == RELOPT_BASEREL) - { - /* Simple single-level transformation */ - child_expr = (Expr *) - adjust_appendrel_attrs(root, - (Node *) cur_em->em_expr, - 1, &appinfo); - } - else - { - /* Must do multi-level transformation */ - child_expr = (Expr *) - adjust_appendrel_attrs_multilevel(root, - (Node *) cur_em->em_expr, - child_rel, - child_rel->top_parent); - } - - /* - * Transform em_relids to match. Note we do *not* do - * pull_varnos(child_expr) here, as for example the - * transformation might have substituted a constant, but we - * don't want the child member to be marked as constant. - */ - new_relids = bms_difference(cur_em->em_relids, - top_parent_relids); - new_relids = bms_add_members(new_relids, child_relids); - - /* - * And likewise for nullable_relids. Note this code assumes - * parent and child relids are singletons. - */ - new_nullable_relids = cur_em->em_nullable_relids; - if (bms_overlap(new_nullable_relids, top_parent_relids)) - { - new_nullable_relids = bms_difference(new_nullable_relids, - top_parent_relids); - new_nullable_relids = bms_add_members(new_nullable_relids, - child_relids); - } + new_nullable_relids = bms_difference(new_nullable_relids, + top_parent_relids); + new_nullable_relids = bms_add_members(new_nullable_relids, + child_relids); + } - (void) add_eq_member(cur_ec, child_expr, - new_relids, new_nullable_relids, - true, cur_em->em_datatype); + (void) add_eq_member(root, cur_ec, child_expr, + new_relids, new_nullable_relids, + true, cur_em->em_datatype); - /* Record this EC index for the child rel */ - child_rel->eclass_indexes = bms_add_member(child_rel->eclass_indexes, i); - } + /* Record this EC index for the child rel */ + child_rel->eclass_indexes = bms_add_member(child_rel->eclass_indexes, i); } + eclass_member_iterator_dispose(&iter); } } @@ -2706,7 +2901,8 @@ add_child_join_rel_equivalences(PlannerInfo *root, while ((i = bms_next_member(matching_ecs, i)) >= 0) { EquivalenceClass *cur_ec = (EquivalenceClass *) list_nth(root->eq_classes, i); - int num_members; + EquivalenceMemberIterator iter; + EquivalenceMember *cur_em; /* * If this EC contains a volatile expression, then generating child @@ -2720,24 +2916,22 @@ add_child_join_rel_equivalences(PlannerInfo *root, Assert(bms_overlap(top_parent_relids, cur_ec->ec_relids)); /* - * We don't use foreach() here because there's no point in scanning - * newly-added child members, so we can stop after the last - * pre-existing EC member. + * Looping using the EquivalenceMemberIterator means we only loop over + * the members which existed when we set up the iterator, not newly + * added ones. */ - num_members = list_length(cur_ec->ec_members); - for (int pos = 0; pos < num_members; pos++) - { - EquivalenceMember *cur_em = (EquivalenceMember *) list_nth(cur_ec->ec_members, pos); + setup_eclass_member_iterator(&iter, root, cur_ec, top_parent_relids, + false, false); - if (cur_em->em_is_const) - continue; /* ignore consts here */ + while ((cur_em = eclass_member_iterator_next(&iter)) != NULL) + { + Expr *child_expr; + Relids new_relids; + Relids new_nullable_relids; - /* - * We consider only original EC members here, not - * already-transformed child members. - */ - if (cur_em->em_is_child) - continue; /* ignore children here */ + Assert(!cur_em->em_is_const); + Assert(!cur_em->em_is_child); + Assert(bms_overlap(cur_em->em_relids, top_parent_relids)); /* * We may ignore expressions that reference a single baserel, @@ -2746,60 +2940,52 @@ add_child_join_rel_equivalences(PlannerInfo *root, if (bms_membership(cur_em->em_relids) != BMS_MULTIPLE) continue; - /* Does this member reference child's topmost parent rel? */ - if (bms_overlap(cur_em->em_relids, top_parent_relids)) + if (parent_joinrel->reloptkind == RELOPT_JOINREL) { - /* Yes, generate transformed child version */ - Expr *child_expr; - Relids new_relids; - Relids new_nullable_relids; - - if (parent_joinrel->reloptkind == RELOPT_JOINREL) - { - /* Simple single-level transformation */ - child_expr = (Expr *) - adjust_appendrel_attrs(root, - (Node *) cur_em->em_expr, - nappinfos, appinfos); - } - else - { - /* Must do multi-level transformation */ - Assert(parent_joinrel->reloptkind == RELOPT_OTHER_JOINREL); - child_expr = (Expr *) - adjust_appendrel_attrs_multilevel(root, - (Node *) cur_em->em_expr, - child_joinrel, - child_joinrel->top_parent); - } + /* Simple single-level transformation */ + child_expr = (Expr *) + adjust_appendrel_attrs(root, + (Node *) cur_em->em_expr, + nappinfos, appinfos); + } + else + { + /* Must do multi-level transformation */ + Assert(parent_joinrel->reloptkind == RELOPT_OTHER_JOINREL); + child_expr = (Expr *) + adjust_appendrel_attrs_multilevel(root, + (Node *) cur_em->em_expr, + child_joinrel, + child_joinrel->top_parent); + } - /* - * Transform em_relids to match. Note we do *not* do - * pull_varnos(child_expr) here, as for example the - * transformation might have substituted a constant, but we - * don't want the child member to be marked as constant. - */ - new_relids = bms_difference(cur_em->em_relids, - top_parent_relids); - new_relids = bms_add_members(new_relids, child_relids); + /* + * Transform em_relids to match. Note we do *not* do + * pull_varnos(child_expr) here, as for example the transformation + * might have substituted a constant, but we don't want the child + * member to be marked as constant. + */ + new_relids = bms_difference(cur_em->em_relids, + top_parent_relids); + new_relids = bms_add_members(new_relids, child_relids); - /* - * For nullable_relids, we must selectively replace parent - * nullable relids with child ones. - */ - new_nullable_relids = cur_em->em_nullable_relids; - if (bms_overlap(new_nullable_relids, top_parent_relids)) - new_nullable_relids = - adjust_child_relids_multilevel(root, - new_nullable_relids, - child_joinrel, - child_joinrel->top_parent); - - (void) add_eq_member(cur_ec, child_expr, - new_relids, new_nullable_relids, - true, cur_em->em_datatype); - } + /* + * For nullable_relids, we must selectively replace parent + * nullable relids with child ones. + */ + new_nullable_relids = cur_em->em_nullable_relids; + if (bms_overlap(new_nullable_relids, top_parent_relids)) + new_nullable_relids = + adjust_child_relids_multilevel(root, + new_nullable_relids, + child_joinrel, + child_joinrel->top_parent); + + (void) add_eq_member(root, cur_ec, child_expr, + new_relids, new_nullable_relids, + true, cur_em->em_datatype); } + eclass_member_iterator_dispose(&iter); } MemoryContextSwitchTo(oldcontext); @@ -2858,7 +3044,8 @@ generate_implied_equalities_for_column(PlannerInfo *root, { EquivalenceClass *cur_ec = (EquivalenceClass *) list_nth(root->eq_classes, i); EquivalenceMember *cur_em; - ListCell *lc2; + EquivalenceMemberIterator iter; + int j; /* Sanity check eclass_indexes only contain ECs for rel */ Assert(is_child_rel || bms_is_subset(rel->relids, cur_ec->ec_relids)); @@ -2880,16 +3067,19 @@ generate_implied_equalities_for_column(PlannerInfo *root, * corner cases, so for now we live with just reporting the first * match. See also get_eclass_for_sort_expr.) */ - cur_em = NULL; - foreach(lc2, cur_ec->ec_members) + setup_eclass_member_iterator(&iter, root, cur_ec, rel->relids, true, + false); + + while ((cur_em = eclass_member_iterator_next(&iter)) != NULL) { - cur_em = (EquivalenceMember *) lfirst(lc2); if (bms_equal(cur_em->em_relids, rel->relids) && callback(root, rel, cur_ec, cur_em, callback_arg)) break; cur_em = NULL; } + eclass_member_iterator_dispose(&iter); + if (!cur_em) continue; @@ -2897,14 +3087,15 @@ generate_implied_equalities_for_column(PlannerInfo *root, * Found our match. Scan the other EC members and attempt to generate * joinclauses. */ - foreach(lc2, cur_ec->ec_members) + j = -1; + while ((j = bms_next_member(cur_ec->ec_nonchild_indexes, j)) >= 0) { - EquivalenceMember *other_em = (EquivalenceMember *) lfirst(lc2); + EquivalenceMember *other_em = list_nth_node(EquivalenceMember, + root->eq_members, j); Oid eq_op; RestrictInfo *rinfo; - if (other_em->em_is_child) - continue; /* ignore children here */ + Assert(!other_em->em_is_child); /* Make sure it'll be a join to a different rel */ if (other_em == cur_em || @@ -3001,7 +3192,7 @@ have_relevant_eclass_joinclause(PlannerInfo *root, * surely be true if both of them overlap ec_relids.) * * Note we don't test ec_broken; if we did, we'd need a separate code - * path to look through ec_sources. Checking the membership anyway is + * path to look through eq_sources. Checking the membership anyway is * OK as a possibly-overoptimistic heuristic. * * We don't test ec_has_const either, even though a const eclass won't @@ -3076,7 +3267,7 @@ eclass_useful_for_merging(PlannerInfo *root, RelOptInfo *rel) { Relids relids; - ListCell *lc; + int i; Assert(!eclass->ec_merged); @@ -3089,7 +3280,7 @@ eclass_useful_for_merging(PlannerInfo *root, /* * Note we don't test ec_broken; if we did, we'd need a separate code path - * to look through ec_sources. Checking the members anyway is OK as a + * to look through eq_sources. Checking the members anyway is OK as a * possibly-overoptimistic heuristic. */ @@ -3107,12 +3298,14 @@ eclass_useful_for_merging(PlannerInfo *root, return false; /* To join, we need a member not in the given rel */ - foreach(lc, eclass->ec_members) + i = -1; + while ((i = bms_next_member(eclass->ec_nonchild_indexes, i)) >= 0) { - EquivalenceMember *cur_em = (EquivalenceMember *) lfirst(lc); + EquivalenceMember *cur_em = + list_nth_node(EquivalenceMember, root->eq_members, i); - if (cur_em->em_is_child) - continue; /* ignore children here */ + /* we don't expect child members here */ + Assert(!cur_em->em_is_child); if (!bms_overlap(cur_em->em_relids, relids)) return true; @@ -3200,7 +3393,7 @@ get_eclass_indexes_for_relids(PlannerInfo *root, Relids relids) /* Should be OK to rely on eclass_indexes */ Assert(root->ec_merging_done); - while ((i = bms_next_member(relids, i)) > 0) + while ((i = bms_next_member(relids, i)) >= 0) { RelOptInfo *rel = root->simple_rel_array[i]; @@ -3236,3 +3429,595 @@ get_common_eclass_indexes(PlannerInfo *root, Relids relids1, Relids relids2) /* Calculate and return the common EC indexes, recycling the left input. */ return bms_int_members(rel1ecs, rel2ecs); } + +/* + * get_ecmember_indexes + * Returns a Bitmapset with indexes into root->eq_members for all + * EquivalenceMembers in 'ec' that have + * bms_overlap(em->em_relids, relids) as true. The returned indexes + * may reference EquivalenceMembers with em_relids containing members + * not mentioned in 'relids'. + * + * Returns a newly allocated Bitmapset which the caller is free to modify. + */ +Bitmapset * +get_ecmember_indexes(PlannerInfo *root, EquivalenceClass *ec, Relids relids, + bool with_children, bool with_norel_members) +{ + Bitmapset *matching_ems; + Bitmapset *rel_ems = NULL; + int i = -1; + + while ((i = bms_next_member(relids, i)) >= 0) + { + RelOptInfo *rel = root->simple_rel_array[i]; + + rel_ems = bms_add_members(rel_ems, rel->eclass_member_indexes); + } + +#ifdef USE_ASSERT_CHECKING + /* verify the results look sane */ + i = -1; + while ((i = bms_next_member(rel_ems, i)) >= 0) + { + EquivalenceMember *em = list_nth_node(EquivalenceMember, + root->eq_members, i); + + Assert(bms_overlap(em->em_relids, relids)); + } +#endif + + /* remove EC members not mentioning the required rels. */ + if (!with_children) + matching_ems = bms_int_members(rel_ems, ec->ec_nonchild_indexes); + else + matching_ems = bms_int_members(rel_ems, ec->ec_member_indexes); + + /* now add any members with that are not mentioned by any relation */ + if (with_norel_members) + matching_ems = bms_add_members(matching_ems, ec->ec_norel_indexes); + + return matching_ems; +} + +/* + * get_ecmember_indexes_strict + * Returns a Bitmapset with indexes into root->eq_members for all + * EquivalenceMembers in 'ec' that have + * bms_is_subset(relids, em->em_relids) as true. + * + * Returns a newly allocated Bitmapset which the caller is free to modify. + */ +Bitmapset * +get_ecmember_indexes_strict(PlannerInfo *root, EquivalenceClass *ec, + Relids relids, bool with_children, + bool with_norel_members) +{ + Bitmapset *matching_ems = NULL; + int i = bms_next_member(relids, -1); + + if (i >= 0) + { + RelOptInfo *rel = root->simple_rel_array[i]; + + /* + * bms_intersect to the first relation to try to keep the resulting + * Bitmapset as small as possible. This saves having to make a + * complete bms_copy() of one of them. One may contain significantly + * more words than the other. + */ + if (!with_children) + matching_ems = bms_intersect(rel->eclass_member_indexes, + ec->ec_nonchild_indexes); + else + matching_ems = bms_intersect(rel->eclass_member_indexes, + ec->ec_member_indexes); + + while ((i = bms_next_member(relids, i)) >= 0) + { + rel = root->simple_rel_array[i]; + matching_ems = bms_int_members(matching_ems, + rel->eclass_member_indexes); + } + } + +#ifdef USE_ASSERT_CHECKING + /* verify the results look sane */ + i = -1; + while ((i = bms_next_member(matching_ems, i)) >= 0) + { + EquivalenceMember *em = list_nth_node(EquivalenceMember, + root->eq_members, i); + + Assert(bms_is_subset(relids, em->em_relids)); + } +#endif + + /* optionally add members with that are not mentioned by any relation */ + if (with_norel_members) + matching_ems = bms_add_members(matching_ems, ec->ec_norel_indexes); + + return matching_ems; +} + +/* + * The threshold for the number of members that must be in an EquivalenceClass + * before we switch to searching for EquivalenceMember by using the Bitmapset + * indexes stored in EquivalenceClass and RelOptInfo. We don't want to make + * this too low as the manipulation of Bitmapsets slows this down for + * EquivalenceClasses with just a few members. The linear search becomes very + * slow when an EquivalenceClass has a large number of members, as can happen + * when planning queries to partitioned tables. + */ +#define ECMEMBER_INDEX_THRESHOLD 16 + +/* + * setup_eclass_member_iterator + * Setup 'iter' so it's ready for eclass_member_iterator_next to start + * searching for EquivalenceMembers matching the specified parameters. + * + * Once used, the iterator must be disposed of using + * eclass_member_iterator_dispose. + */ +void +setup_eclass_member_iterator(EquivalenceMemberIterator *iter, + PlannerInfo *root, EquivalenceClass *ec, + Relids relids, bool with_children, + bool with_norel_members) +{ + iter->orig_length = list_length(ec->ec_members); + iter->use_index = iter->orig_length >= ECMEMBER_INDEX_THRESHOLD; + iter->with_children = with_children; + iter->with_norel_members = with_norel_members; + iter->relids_empty = bms_is_empty(relids); +#ifdef USE_ASSERT_CHECKING + iter->isstrict = false; +#endif + iter->with_relids = relids; + iter->root = root; + iter->eclass = ec; + +#ifdef USE_ASSERT_CHECKING + if (1) +#else + if (iter->use_index) +#endif + iter->matching_ems = get_ecmember_indexes(root, ec, relids, + with_children, + with_norel_members); + else + iter->matching_ems = NULL; + + iter->current_index = -1; + +#ifdef USE_ASSERT_CHECKING + + /* + * Verify that an iterator that uses the index and one that does not both + * return the same EquivalenceMembers + */ + { + EquivalenceMemberIterator idx_iter; + EquivalenceMemberIterator noidx_iter; + EquivalenceMember *em; + List *list1 = NIL; + List *list2 = NIL; + ListCell *lc1, + *lc2; + + memcpy(&idx_iter, iter, sizeof(EquivalenceMemberIterator)); + memcpy(&noidx_iter, iter, sizeof(EquivalenceMemberIterator)); + + idx_iter.use_index = true; + noidx_iter.use_index = false; + + while ((em = eclass_member_iterator_next(&idx_iter)) != NULL) + list1 = lappend(list1, em); + + while ((em = eclass_member_iterator_next(&noidx_iter)) != NULL) + list2 = lappend(list2, em); + + list_sort(list1, list_ptr_cmp); + list_sort(list2, list_ptr_cmp); + + Assert(list_length(list1) == list_length(list2)); + + forboth(lc1, list1, lc2, list2) + Assert(lfirst(lc1) == lfirst(lc2)); + } +#endif + +} + +/* + * setup_eclass_member_strict_iterator + * Setup 'iter' so it's ready for eclass_member_iterator_strict_next to + * start searching for EquivalenceMembers matching the specified + * parameters. + * + * Once used, the iterator must be disposed of using + * eclass_member_iterator_dispose. + */ +void +setup_eclass_member_strict_iterator(EquivalenceMemberIterator *iter, + PlannerInfo *root, EquivalenceClass *ec, + Relids relids, bool with_children, + bool with_norel_members) +{ + iter->orig_length = list_length(ec->ec_members); + iter->use_index = iter->orig_length >= ECMEMBER_INDEX_THRESHOLD; + iter->with_children = with_children; + iter->with_norel_members = with_norel_members; + iter->relids_empty = bms_is_empty(relids); +#ifdef USE_ASSERT_CHECKING + iter->isstrict = true; +#endif + iter->with_relids = relids; + iter->root = root; + iter->eclass = ec; + +#ifdef USE_ASSERT_CHECKING + if (1) +#else + if (iter->use_index) +#endif + iter->matching_ems = get_ecmember_indexes_strict(root, ec, relids, + with_children, + with_norel_members); + else + iter->matching_ems = NULL; + + iter->current_index = -1; + +#ifdef USE_ASSERT_CHECKING + + /* + * Verify that an iterator that uses the index and one that does not both + * return the same EquivalenceMembers + */ + { + EquivalenceMemberIterator idx_iter; + EquivalenceMemberIterator noidx_iter; + EquivalenceMember *em; + List *list1 = NIL; + List *list2 = NIL; + ListCell *lc1, + *lc2; + + memcpy(&idx_iter, iter, sizeof(EquivalenceMemberIterator)); + memcpy(&noidx_iter, iter, sizeof(EquivalenceMemberIterator)); + + idx_iter.use_index = true; + noidx_iter.use_index = false; + + while ((em = eclass_member_iterator_strict_next(&idx_iter)) != NULL) + list1 = lappend(list1, em); + + while ((em = eclass_member_iterator_strict_next(&noidx_iter)) != NULL) + list2 = lappend(list2, em); + + list_sort(list1, list_ptr_cmp); + list_sort(list2, list_ptr_cmp); + + Assert(list_length(list1) == list_length(list2)); + + forboth(lc1, list1, lc2, list2) + Assert(lfirst(lc1) == lfirst(lc2)); + } +#endif +} + +/* + * eclass_member_iterator_next + * Fetch the next EquivalenceMember from an EquivalenceMemberIterator + * which was set up by setup_eclass_member_iterator(). Returns NULL when + * there are no more matching EquivalenceMembers. + */ +EquivalenceMember * +eclass_member_iterator_next(EquivalenceMemberIterator *iter) +{ + /* Fail if this was used instead of eclass_member_iterator_strict_next */ + Assert(!iter->isstrict); + + if (iter->use_index) + { + iter->current_index = bms_next_member(iter->matching_ems, + iter->current_index); + if (iter->current_index >= 0) + return list_nth_node(EquivalenceMember, iter->root->eq_members, + iter->current_index); + return NULL; + } + else + { + ListCell *lc; + + for_each_from(lc, iter->eclass->ec_members, iter->current_index + 1) + { + EquivalenceMember *em = lfirst_node(EquivalenceMember, lc); + + iter->current_index = foreach_current_index(lc); + + /* + * Some users of this iterator will be adding new + * EquivalenceMember during the loop. We must ensure we don't + * return those, so here we return NULL when the loop index goes + * beyond the original length of the ec_members list. + */ + if (iter->current_index >= iter->orig_length) + return NULL; + + /* don't return child members when with_children==false */ + if (!iter->with_children && em->em_is_child) + continue; + + /* + * When with_norel_members==true, make sure we return all members + * without Vars. + */ + if (iter->with_norel_members && em->em_norel_expr) + return em; + + /* + * Don't return members which have no common rels with with_relids + */ + if (!bms_overlap(em->em_relids, iter->with_relids)) + continue; + + return em; + } + return NULL; + } +} + +/* + * eclass_member_iterator_strict_next + * Fetch the next EquivalenceMember from an EquivalenceMemberIterator + * which was set up by setup_eclass_member_strict_iterator(). Returns + * NULL when there are no more matching EquivalenceMembers. + */ +EquivalenceMember * +eclass_member_iterator_strict_next(EquivalenceMemberIterator *iter) +{ + /* Fail if this was used instead of eclass_member_iterator_next */ + Assert(iter->isstrict); + + if (iter->use_index) + { + iter->current_index = bms_next_member(iter->matching_ems, + iter->current_index); + if (iter->current_index >= 0) + return list_nth_node(EquivalenceMember, iter->root->eq_members, + iter->current_index); + return NULL; + } + else + { + ListCell *lc; + + for_each_from(lc, iter->eclass->ec_members, iter->current_index + 1) + { + EquivalenceMember *em = lfirst_node(EquivalenceMember, lc); + + iter->current_index = foreach_current_index(lc); + + /* + * Some users of this iterator will be adding new + * EquivalenceMember during the loop. We must ensure we don't + * return those, so here we return NULL when the loop index goes + * beyond the original length of the ec_members list. + */ + if (iter->current_index >= iter->orig_length) + return NULL; + + /* don't return child members when with_children==false */ + if (!iter->with_children && em->em_is_child) + continue; + + /* + * When with_norel_members==true, make sure we return all members + * without Vars. + */ + if (iter->with_norel_members && em->em_norel_expr) + return em; + + /* + * Don't match members where em_relids that don't contain all rels + * mentioned in with_relids. + */ + if (iter->relids_empty || + !bms_is_subset(iter->with_relids, em->em_relids)) + continue; + + return em; + } + return NULL; + } +} + +/* + * eclass_member_iterator_dispose + * Free any memory allocated by the iterator + */ +void +eclass_member_iterator_dispose(EquivalenceMemberIterator *iter) +{ + bms_free(iter->matching_ems); +} + + +/* + * get_ec_source_indexes + * Returns a Bitmapset with indexes into root->eq_sources for all + * RestrictInfos in 'ec' that have + * bms_overlap(relids, rinfo->clause_relids) as true. + * + * Returns a newly allocated Bitmapset which the caller is free to modify. + */ +Bitmapset * +get_ec_source_indexes(PlannerInfo *root, EquivalenceClass *ec, Relids relids) +{ + Bitmapset *rel_esis = NULL; + int i = -1; + + while ((i = bms_next_member(relids, i)) >= 0) + { + RelOptInfo *rel = root->simple_rel_array[i]; + + rel_esis = bms_add_members(rel_esis, rel->eclass_source_indexes); + } + +#ifdef USE_ASSERT_CHECKING + /* verify the results look sane */ + i = -1; + while ((i = bms_next_member(rel_esis, i)) >= 0) + { + RestrictInfo *rinfo = list_nth_node(RestrictInfo, root->eq_sources, + i); + + Assert(bms_overlap(relids, rinfo->clause_relids)); + } +#endif + + /* bitwise-AND to leave only the ones for this EquivalenceClass */ + return bms_int_members(rel_esis, ec->ec_source_indexes); +} + +/* + * get_ec_source_indexes_strict + * Returns a Bitmapset with indexes into root->eq_sources for all + * RestrictInfos in 'ec' that have + * bms_is_subset(relids, rinfo->clause_relids) as true. + * + * Returns a newly allocated Bitmapset which the caller is free to modify. + */ +Bitmapset * +get_ec_source_indexes_strict(PlannerInfo *root, EquivalenceClass *ec, + Relids relids) +{ + Bitmapset *esis = NULL; + int i = bms_next_member(relids, -1); + + if (i >= 0) + { + RelOptInfo *rel = root->simple_rel_array[i]; + + /* + * bms_intersect to the first relation to try to keep the resulting + * Bitmapset as small as possible. This saves having to make a + * complete bms_copy() of one of them. One may contain significantly + * more words than the other. + */ + esis = bms_intersect(ec->ec_source_indexes, + rel->eclass_source_indexes); + + while ((i = bms_next_member(relids, i)) >= 0) + { + rel = root->simple_rel_array[i]; + esis = bms_int_members(esis, rel->eclass_source_indexes); + } + } + +#ifdef USE_ASSERT_CHECKING + /* verify the results look sane */ + i = -1; + while ((i = bms_next_member(esis, i)) >= 0) + { + RestrictInfo *rinfo = list_nth_node(RestrictInfo, root->eq_sources, + i); + + Assert(bms_is_subset(relids, rinfo->clause_relids)); + } +#endif + + return esis; +} + +/* + * get_ec_derive_indexes + * Returns a Bitmapset with indexes into root->eq_derives for all + * RestrictInfos in 'ec' that have + * bms_overlap(relids, rinfo->clause_relids) as true. + * + * Returns a newly allocated Bitmapset which the caller is free to modify. + * + * XXX is this function even needed? + */ +Bitmapset * +get_ec_derive_indexes(PlannerInfo *root, EquivalenceClass *ec, Relids relids) +{ + Bitmapset *rel_edis = NULL; + int i = -1; + + while ((i = bms_next_member(relids, i)) >= 0) + { + RelOptInfo *rel = root->simple_rel_array[i]; + + rel_edis = bms_add_members(rel_edis, rel->eclass_derive_indexes); + } + +#ifdef USE_ASSERT_CHECKING + /* verify the results look sane */ + i = -1; + while ((i = bms_next_member(rel_edis, i)) >= 0) + { + RestrictInfo *rinfo = list_nth_node(RestrictInfo, root->eq_derives, + i); + + Assert(bms_overlap(relids, rinfo->clause_relids)); + } +#endif + + /* bitwise-AND to leave only the ones for this EquivalenceClass */ + return bms_int_members(rel_edis, ec->ec_derive_indexes); +} + +/* + * get_ec_derive_indexes_strict + * Returns a Bitmapset with indexes into root->eq_derives for all + * RestrictInfos in 'ec' that have + * bms_is_subset(relids, rinfo->clause_relids) as true. + * + * Returns a newly allocated Bitmapset which the caller is free to modify. + */ +Bitmapset * +get_ec_derive_indexes_strict(PlannerInfo *root, EquivalenceClass *ec, + Relids relids) +{ + Bitmapset *edis = NULL; + int i = bms_next_member(relids, -1); + + if (i >= 0) + { + RelOptInfo *rel = root->simple_rel_array[i]; + + /* + * bms_intersect to the first relation to try to keep the resulting + * Bitmapset as small as possible. This saves having to make a + * complete bms_copy() of one of them. One may contain significantly + * more words than the other. + */ + edis = bms_intersect(ec->ec_derive_indexes, + rel->eclass_derive_indexes); + + while ((i = bms_next_member(relids, i)) >= 0) + { + rel = root->simple_rel_array[i]; + edis = bms_int_members(edis, rel->eclass_derive_indexes); + } + } + +#ifdef USE_ASSERT_CHECKING + /* verify the results look sane */ + i = -1; + while ((i = bms_next_member(edis, i)) >= 0) + { + RestrictInfo *rinfo = list_nth_node(RestrictInfo, root->eq_derives, + i); + + Assert(bms_is_subset(relids, rinfo->clause_relids)); + } +#endif + + return edis; +} diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c index e13c8f1914..e1b736c0c0 100644 --- a/src/backend/optimizer/path/indxpath.c +++ b/src/backend/optimizer/path/indxpath.c @@ -184,8 +184,8 @@ static IndexClause *expand_indexqual_rowcompare(PlannerInfo *root, IndexOptInfo *index, Oid expr_op, bool var_on_left); -static void match_pathkeys_to_index(IndexOptInfo *index, List *pathkeys, - List **orderby_clauses_p, +static void match_pathkeys_to_index(PlannerInfo *root, IndexOptInfo *index, + List *pathkeys, List **orderby_clauses_p, List **clause_columns_p); static Expr *match_clause_to_ordering_op(IndexOptInfo *index, int indexcol, Expr *clause, Oid pk_opfamily); @@ -978,7 +978,7 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel, else if (index->amcanorderbyop && pathkeys_possibly_useful) { /* see if we can generate ordering operators for query_pathkeys */ - match_pathkeys_to_index(index, root->query_pathkeys, + match_pathkeys_to_index(root, index, root->query_pathkeys, &orderbyclauses, &orderbyclausecols); if (orderbyclauses) @@ -3070,8 +3070,8 @@ expand_indexqual_rowcompare(PlannerInfo *root, * one-to-one with the requested pathkeys. */ static void -match_pathkeys_to_index(IndexOptInfo *index, List *pathkeys, - List **orderby_clauses_p, +match_pathkeys_to_index(PlannerInfo *root, IndexOptInfo *index, + List *pathkeys, List **orderby_clauses_p, List **clause_columns_p) { List *orderby_clauses = NIL; @@ -3088,8 +3088,9 @@ match_pathkeys_to_index(IndexOptInfo *index, List *pathkeys, foreach(lc1, pathkeys) { PathKey *pathkey = (PathKey *) lfirst(lc1); + Bitmapset *matching_ems; bool found = false; - ListCell *lc2; + int i; /* * Note: for any failure to match, we just return NIL immediately. @@ -3113,9 +3114,14 @@ match_pathkeys_to_index(IndexOptInfo *index, List *pathkeys, * be considered to match more than one pathkey list, which is OK * here. See also get_eclass_for_sort_expr.) */ - foreach(lc2, pathkey->pk_eclass->ec_members) + matching_ems = bms_intersect(pathkey->pk_eclass->ec_member_indexes, + index->rel->eclass_member_indexes); + + i = -1; + while ((i = bms_next_member(matching_ems, i)) >= 0) { - EquivalenceMember *member = (EquivalenceMember *) lfirst(lc2); + EquivalenceMember *member = list_nth_node(EquivalenceMember, + root->eq_members, i); int indexcol; /* No possibility of match if it references other relations */ diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index d2e241c983..e715d5135b 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -955,18 +955,19 @@ convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel, * outer query. */ int best_score = -1; - ListCell *j; + int j = -1; - foreach(j, sub_eclass->ec_members) + while ((j = bms_next_member(sub_eclass->ec_nonchild_indexes, j)) >= 0) { - EquivalenceMember *sub_member = (EquivalenceMember *) lfirst(j); + EquivalenceMember *sub_member = list_nth_node(EquivalenceMember, + rel->subroot->eq_members, + j); Expr *sub_expr = sub_member->em_expr; Oid sub_expr_type = sub_member->em_datatype; Oid sub_expr_coll = sub_eclass->ec_collation; ListCell *k; - if (sub_member->em_is_child) - continue; /* ignore children here */ + Assert(!sub_member->em_is_child); foreach(k, subquery_tlist) { @@ -1476,7 +1477,7 @@ select_outer_pathkeys_for_merge(PlannerInfo *root, RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); EquivalenceClass *oeclass; int score; - ListCell *lc2; + int i; /* get the outer eclass */ update_mergeclause_eclasses(root, rinfo); @@ -1497,13 +1498,16 @@ select_outer_pathkeys_for_merge(PlannerInfo *root, /* compute score */ score = 0; - foreach(lc2, oeclass->ec_members) + i = -1; + while ((i = bms_next_member(oeclass->ec_nonchild_indexes, i)) >= 0) { - EquivalenceMember *em = (EquivalenceMember *) lfirst(lc2); + EquivalenceMember *em = list_nth_node(EquivalenceMember, + root->eq_members, i); - /* Potential future join partner? */ - if (!em->em_is_const && !em->em_is_child && - !bms_overlap(em->em_relids, joinrel->relids)) + /* shouldn't be consts or child members in ec_nonchild_indexes */ + Assert(!em->em_is_const && !em->em_is_child); + + if (!bms_overlap(em->em_relids, joinrel->relids)) score++; } diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c index cd68942af0..d519b97dbf 100644 --- a/src/backend/optimizer/plan/createplan.c +++ b/src/backend/optimizer/plan/createplan.c @@ -260,8 +260,8 @@ static IncrementalSort *make_incrementalsort(Plan *lefttree, int numCols, int nPresortedCols, AttrNumber *sortColIdx, Oid *sortOperators, Oid *collations, bool *nullsFirst); -static Plan *prepare_sort_from_pathkeys(Plan *lefttree, List *pathkeys, - Relids relids, +static Plan *prepare_sort_from_pathkeys(PlannerInfo *root, Plan *lefttree, + List *pathkeys, Relids relids, const AttrNumber *reqColIdx, bool adjust_tlist_in_place, int *p_numsortkeys, @@ -269,10 +269,13 @@ static Plan *prepare_sort_from_pathkeys(Plan *lefttree, List *pathkeys, Oid **p_sortOperators, Oid **p_collations, bool **p_nullsFirst); -static Sort *make_sort_from_pathkeys(Plan *lefttree, List *pathkeys, - Relids relids); -static IncrementalSort *make_incrementalsort_from_pathkeys(Plan *lefttree, - List *pathkeys, Relids relids, int nPresortedCols); +static Sort *make_sort_from_pathkeys(PlannerInfo *root, Plan *lefttree, + List *pathkeys, Relids relids); +static IncrementalSort *make_incrementalsort_from_pathkeys(PlannerInfo *root, + Plan *lefttree, + List *pathkeys, + Relids relids, + int nPresortedCols); static Sort *make_sort_from_groupcols(List *groupcls, AttrNumber *grpColIdx, Plan *lefttree); @@ -293,7 +296,7 @@ static Group *make_group(List *tlist, List *qual, int numGroupCols, AttrNumber *grpColIdx, Oid *grpOperators, Oid *grpCollations, Plan *lefttree); static Unique *make_unique_from_sortclauses(Plan *lefttree, List *distinctList); -static Unique *make_unique_from_pathkeys(Plan *lefttree, +static Unique *make_unique_from_pathkeys(PlannerInfo *root, Plan *lefttree, List *pathkeys, int numCols); static Gather *make_gather(List *qptlist, List *qpqual, int nworkers, int rescan_param, bool single_copy, Plan *subplan); @@ -1260,7 +1263,8 @@ create_append_plan(PlannerInfo *root, AppendPath *best_path, int flags) * function result; it must be the same plan node. However, we then * need to detect whether any tlist entries were added. */ - (void) prepare_sort_from_pathkeys((Plan *) plan, pathkeys, + (void) prepare_sort_from_pathkeys(root, + (Plan *) plan, pathkeys, best_path->path.parent->relids, NULL, true, @@ -1304,7 +1308,7 @@ create_append_plan(PlannerInfo *root, AppendPath *best_path, int flags) * don't need an explicit sort, to make sure they are returning * the same sort key columns the Append expects. */ - subplan = prepare_sort_from_pathkeys(subplan, pathkeys, + subplan = prepare_sort_from_pathkeys(root, subplan, pathkeys, subpath->parent->relids, nodeSortColIdx, false, @@ -1445,7 +1449,7 @@ create_merge_append_plan(PlannerInfo *root, MergeAppendPath *best_path, * function result; it must be the same plan node. However, we then need * to detect whether any tlist entries were added. */ - (void) prepare_sort_from_pathkeys(plan, pathkeys, + (void) prepare_sort_from_pathkeys(root, plan, pathkeys, best_path->path.parent->relids, NULL, true, @@ -1476,7 +1480,7 @@ create_merge_append_plan(PlannerInfo *root, MergeAppendPath *best_path, subplan = create_plan_recurse(root, subpath, CP_EXACT_TLIST); /* Compute sort column info, and adjust subplan's tlist as needed */ - subplan = prepare_sort_from_pathkeys(subplan, pathkeys, + subplan = prepare_sort_from_pathkeys(root, subplan, pathkeys, subpath->parent->relids, node->sortColIdx, false, @@ -1966,7 +1970,7 @@ create_gather_merge_plan(PlannerInfo *root, GatherMergePath *best_path) Assert(pathkeys != NIL); /* Compute sort column info, and adjust subplan's tlist as needed */ - subplan = prepare_sort_from_pathkeys(subplan, pathkeys, + subplan = prepare_sort_from_pathkeys(root, subplan, pathkeys, best_path->subpath->parent->relids, gm_plan->sortColIdx, false, @@ -2185,7 +2189,7 @@ create_sort_plan(PlannerInfo *root, SortPath *best_path, int flags) * relids. Thus, if this sort path is based on a child relation, we must * pass its relids. */ - plan = make_sort_from_pathkeys(subplan, best_path->path.pathkeys, + plan = make_sort_from_pathkeys(root, subplan, best_path->path.pathkeys, IS_OTHER_REL(best_path->subpath->parent) ? best_path->path.parent->relids : NULL); @@ -2209,7 +2213,7 @@ create_incrementalsort_plan(PlannerInfo *root, IncrementalSortPath *best_path, /* See comments in create_sort_plan() above */ subplan = create_plan_recurse(root, best_path->spath.subpath, flags | CP_SMALL_TLIST); - plan = make_incrementalsort_from_pathkeys(subplan, + plan = make_incrementalsort_from_pathkeys(root, subplan, best_path->spath.path.pathkeys, IS_OTHER_REL(best_path->spath.subpath->parent) ? best_path->spath.path.parent->relids : NULL, @@ -2278,7 +2282,7 @@ create_upper_unique_plan(PlannerInfo *root, UpperUniquePath *best_path, int flag subplan = create_plan_recurse(root, best_path->subpath, flags | CP_LABEL_TLIST); - plan = make_unique_from_pathkeys(subplan, + plan = make_unique_from_pathkeys(root, subplan, best_path->path.pathkeys, best_path->numkeys); @@ -4483,7 +4487,7 @@ create_mergejoin_plan(PlannerInfo *root, if (best_path->outersortkeys) { Relids outer_relids = outer_path->parent->relids; - Sort *sort = make_sort_from_pathkeys(outer_plan, + Sort *sort = make_sort_from_pathkeys(root, outer_plan, best_path->outersortkeys, outer_relids); @@ -4497,7 +4501,7 @@ create_mergejoin_plan(PlannerInfo *root, if (best_path->innersortkeys) { Relids inner_relids = inner_path->parent->relids; - Sort *sort = make_sort_from_pathkeys(inner_plan, + Sort *sort = make_sort_from_pathkeys(root, inner_plan, best_path->innersortkeys, inner_relids); @@ -6117,7 +6121,8 @@ make_incrementalsort(Plan *lefttree, int numCols, int nPresortedCols, * or a Result stacked atop lefttree). */ static Plan * -prepare_sort_from_pathkeys(Plan *lefttree, List *pathkeys, +prepare_sort_from_pathkeys(PlannerInfo *root, + Plan *lefttree, List *pathkeys, Relids relids, const AttrNumber *reqColIdx, bool adjust_tlist_in_place, @@ -6184,7 +6189,7 @@ prepare_sort_from_pathkeys(Plan *lefttree, List *pathkeys, tle = get_tle_by_resno(tlist, reqColIdx[numsortkeys]); if (tle) { - em = find_ec_member_matching_expr(ec, tle->expr, relids); + em = find_ec_member_matching_expr(root, ec, tle->expr, relids); if (em) { /* found expr at right place in tlist */ @@ -6215,7 +6220,7 @@ prepare_sort_from_pathkeys(Plan *lefttree, List *pathkeys, foreach(j, tlist) { tle = (TargetEntry *) lfirst(j); - em = find_ec_member_matching_expr(ec, tle->expr, relids); + em = find_ec_member_matching_expr(root, ec, tle->expr, relids); if (em) { /* found expr already in tlist */ @@ -6231,7 +6236,7 @@ prepare_sort_from_pathkeys(Plan *lefttree, List *pathkeys, /* * No matching tlist item; look for a computable expression. */ - em = find_computable_ec_member(NULL, ec, tlist, relids, false); + em = find_computable_ec_member(root, ec, tlist, relids, false); if (!em) elog(ERROR, "could not find pathkey item to sort"); pk_datatype = em->em_datatype; @@ -6302,7 +6307,8 @@ prepare_sort_from_pathkeys(Plan *lefttree, List *pathkeys, * 'relids' is the set of relations required by prepare_sort_from_pathkeys() */ static Sort * -make_sort_from_pathkeys(Plan *lefttree, List *pathkeys, Relids relids) +make_sort_from_pathkeys(PlannerInfo *root, Plan *lefttree, List *pathkeys, + Relids relids) { int numsortkeys; AttrNumber *sortColIdx; @@ -6311,7 +6317,9 @@ make_sort_from_pathkeys(Plan *lefttree, List *pathkeys, Relids relids) bool *nullsFirst; /* Compute sort column info, and adjust lefttree as needed */ - lefttree = prepare_sort_from_pathkeys(lefttree, pathkeys, + lefttree = prepare_sort_from_pathkeys(root, + lefttree, + pathkeys, relids, NULL, false, @@ -6337,7 +6345,8 @@ make_sort_from_pathkeys(Plan *lefttree, List *pathkeys, Relids relids) * 'nPresortedCols' is the number of presorted columns in input tuples */ static IncrementalSort * -make_incrementalsort_from_pathkeys(Plan *lefttree, List *pathkeys, +make_incrementalsort_from_pathkeys(PlannerInfo *root, + Plan *lefttree, List *pathkeys, Relids relids, int nPresortedCols) { int numsortkeys; @@ -6347,7 +6356,9 @@ make_incrementalsort_from_pathkeys(Plan *lefttree, List *pathkeys, bool *nullsFirst; /* Compute sort column info, and adjust lefttree as needed */ - lefttree = prepare_sort_from_pathkeys(lefttree, pathkeys, + lefttree = prepare_sort_from_pathkeys(root, + lefttree, + pathkeys, relids, NULL, false, @@ -6697,7 +6708,8 @@ make_unique_from_sortclauses(Plan *lefttree, List *distinctList) * as above, but use pathkeys to identify the sort columns and semantics */ static Unique * -make_unique_from_pathkeys(Plan *lefttree, List *pathkeys, int numCols) +make_unique_from_pathkeys(PlannerInfo *root, Plan *lefttree, List *pathkeys, + int numCols) { Unique *node = makeNode(Unique); Plan *plan = &node->plan; @@ -6760,7 +6772,7 @@ make_unique_from_pathkeys(Plan *lefttree, List *pathkeys, int numCols) foreach(j, plan->targetlist) { tle = (TargetEntry *) lfirst(j); - em = find_ec_member_matching_expr(ec, tle->expr, NULL); + em = find_ec_member_matching_expr(root, ec, tle->expr, NULL); if (em) { /* found expr already in tlist */ diff --git a/src/backend/optimizer/plan/planagg.c b/src/backend/optimizer/plan/planagg.c index d168620665..9f0fa42045 100644 --- a/src/backend/optimizer/plan/planagg.c +++ b/src/backend/optimizer/plan/planagg.c @@ -356,6 +356,7 @@ build_minmax_path(PlannerInfo *root, MinMaxAggInfo *mminfo, Assert(subroot->join_info_list == NIL); /* and we haven't made equivalence classes, either */ Assert(subroot->eq_classes == NIL); + Assert(root->eq_members == NIL); /* and we haven't created PlaceHolderInfos, either */ Assert(subroot->placeholder_list == NIL); diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index 05f44faf6e..f12f7fa83b 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -626,6 +626,9 @@ subquery_planner(PlannerGlobal *glob, Query *parse, root->cte_plan_ids = NIL; root->multiexpr_params = NIL; root->eq_classes = NIL; + root->eq_members = NIL; + root->eq_sources = NIL; + root->eq_derives = NIL; root->ec_merging_done = false; root->all_result_relids = parse->resultRelation ? bms_make_singleton(parse->resultRelation) : NULL; diff --git a/src/backend/optimizer/prep/prepjointree.c b/src/backend/optimizer/prep/prepjointree.c index 37a7af8c66..b7eaeb6163 100644 --- a/src/backend/optimizer/prep/prepjointree.c +++ b/src/backend/optimizer/prep/prepjointree.c @@ -1000,6 +1000,9 @@ pull_up_simple_subquery(PlannerInfo *root, Node *jtnode, RangeTblEntry *rte, subroot->cte_plan_ids = NIL; subroot->multiexpr_params = NIL; subroot->eq_classes = NIL; + subroot->eq_members = NIL; + subroot->eq_sources = NIL; + subroot->eq_derives = NIL; subroot->ec_merging_done = false; subroot->all_result_relids = NULL; subroot->leaf_result_relids = NULL; diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c index 0c68ec011b..4c7d66311f 100644 --- a/src/backend/optimizer/prep/prepunion.c +++ b/src/backend/optimizer/prep/prepunion.c @@ -45,7 +45,7 @@ #include "utils/selfuncs.h" #include "utils/syscache.h" - +static void setup_append_rel_entry(PlannerInfo *root); static RelOptInfo *recurse_set_operations(Node *setOp, PlannerInfo *root, List *colTypes, List *colCollations, bool junkOK, @@ -128,6 +128,7 @@ plan_set_operations(PlannerInfo *root) * so that pathkeys.c won't complain. */ Assert(root->eq_classes == NIL); + Assert(root->eq_members == NIL); root->ec_merging_done = true; /* @@ -149,6 +150,8 @@ plan_set_operations(PlannerInfo *root) leftmostQuery = leftmostRTE->subquery; Assert(leftmostQuery != NULL); + setup_append_rel_entry(root); + /* * If the topmost node is a recursive union, it needs special processing. */ @@ -180,6 +183,26 @@ plan_set_operations(PlannerInfo *root) return setop_rel; } +/* + * setup_append_rel_entry + * Add entry into root's simple_rel_array at element 0. This is required + * because generate_append_tlist() makes Vars with varno=0. In + * add_eq_member() we need to index EquivalenceMembers belonging to this + * relation. + */ +static void +setup_append_rel_entry(PlannerInfo *root) +{ + RelOptInfo *rel = makeNode(RelOptInfo); + + memset(rel, 0, sizeof(RelOptInfo)); + rel->eclass_member_indexes = NULL; + + /* make sure nothing else has made use of this element */ + Assert(root->simple_rel_array[0] == NULL); + root->simple_rel_array[0] = rel; +} + /* * recurse_set_operations * Recursively handle one step in a tree of set operations diff --git a/src/backend/optimizer/util/relnode.c b/src/backend/optimizer/util/relnode.c index 0a5632699d..ace943c87d 100644 --- a/src/backend/optimizer/util/relnode.c +++ b/src/backend/optimizer/util/relnode.c @@ -219,6 +219,9 @@ build_simple_rel(PlannerInfo *root, int relid, RelOptInfo *parent) rel->tuples = 0; rel->allvisfrac = 0; rel->eclass_indexes = NULL; + rel->eclass_member_indexes = NULL; + rel->eclass_source_indexes = NULL; + rel->eclass_derive_indexes = NULL; rel->subroot = NULL; rel->subplan_params = NIL; rel->rel_parallel_workers = -1; /* set up in get_relation_info */ @@ -648,6 +651,9 @@ build_join_rel(PlannerInfo *root, joinrel->tuples = 0; joinrel->allvisfrac = 0; joinrel->eclass_indexes = NULL; + joinrel->eclass_member_indexes = NULL; + joinrel->eclass_source_indexes = NULL; + joinrel->eclass_derive_indexes = NULL; joinrel->subroot = NULL; joinrel->subplan_params = NIL; joinrel->rel_parallel_workers = -1; @@ -834,6 +840,9 @@ build_child_join_rel(PlannerInfo *root, RelOptInfo *outer_rel, joinrel->tuples = 0; joinrel->allvisfrac = 0; joinrel->eclass_indexes = NULL; + joinrel->eclass_member_indexes = NULL; + joinrel->eclass_source_indexes = NULL; + joinrel->eclass_derive_indexes = NULL; joinrel->subroot = NULL; joinrel->subplan_params = NIL; joinrel->amflags = 0; diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index 2d1d8f4bcd..d888cd54d3 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -306,6 +306,15 @@ struct PlannerInfo /* list of active EquivalenceClasses */ List *eq_classes; + /* list of each EquivalenceMember */ + List *eq_members; + + /* list of source RestrictInfos used to build EquivalenceClasses */ + List *eq_sources; + + /* list of RestrictInfos derived from EquivalenceClasses */ + List *eq_derives; + /* set true once ECs are canonical */ bool ec_merging_done; @@ -926,6 +935,24 @@ typedef struct RelOptInfo * Indexes in PlannerInfo's eq_classes list of ECs that mention this rel */ Bitmapset *eclass_indexes; + + /* + * Indexes in PlannerInfo's eq_members list of EMs that mention this rel + */ + Bitmapset *eclass_member_indexes; + + /* + * Indexes in PlannerInfo's eq_sources list for RestrictInfos that mention + * this relation. + */ + Bitmapset *eclass_source_indexes; + + /* + * Indexes in PlannerInfo's eq_derives list for RestrictInfos that mention + * this relation. + */ + Bitmapset *eclass_derive_indexes; + PlannerInfo *subroot; /* if subquery */ List *subplan_params; /* if subquery */ /* wanted number of parallel workers */ @@ -1305,6 +1332,39 @@ typedef struct StatisticExtInfo * the included values might be all NULL rather than all the same non-null * values. See src/backend/optimizer/README for more on that point. * + * At various locations in the query planner, we must search for + * EquivalenceMembers within a given EquivalenceClass. For the common case, + * an EquivalenceClass does not have a large number of EquivalenceMembers, + * however, in cases such as planning queries to partitioned tables, the number + * of members can become large. To maintain planning performance, we make use + * of a bitmap index to allow us to quickly find EquivalenceMembers in a given + * EquivalenceClass belonging to a given relation or set of relations. This + * is done by storing a list of EquivalenceMembers belonging to all + * EquivalenceClasses in PlannerInfo and storing a Bitmapset for each + * RelOptInfo which has a bit set for each EquivalenceMember in that list + * which relates to the given relation. We also store a Bitmapset to mark all + * of the indexes in the PlannerInfo's list of EquivalenceMembers in the + * EquivalenceClass. We can quickly find the interesting indexes into the + * PlannerInfo's list by performing a bit-wise AND on the RelOptInfo's + * Bitmapset and the EquivalenceClasses. + * + * To further speed up the lookup of EquivalenceMembers we also record the + * non-child indexes. This allows us to deal with fewer bits for searches + * that don't require "is_child" EquivalenceMembers. We must also store the + * indexes into EquivalenceMembers which have no relids mentioned as some + * searches require that. + * + * Additionally, we also store the EquivalenceMembers of a given + * EquivalenceClass in the ec_members list. Technically, we could obtain this + * from looking at the bits in ec_member_indexes, however, finding all members + * of a given EquivalenceClass is common enough that it pays to have fast + * access to a dense list containing all members. + * + * The source and derived RestrictInfos are indexed in a similar method, + * although we don't maintain a List in the EquivalenceClass for these. These + * must be looked up in PlannerInfo using the ec_source_indexes and + * ec_derive_indexes Bitmapsets. + * * NB: if ec_merged isn't NULL, this class has been merged into another, and * should be ignored in favor of using the pointed-to class. * @@ -1322,9 +1382,19 @@ typedef struct EquivalenceClass List *ec_opfamilies; /* btree operator family OIDs */ Oid ec_collation; /* collation, if datatypes are collatable */ - List *ec_members; /* list of EquivalenceMembers */ - List *ec_sources; /* list of generating RestrictInfos */ - List *ec_derives; /* list of derived RestrictInfos */ + List *ec_members; /* list of EquivalenceMembers in the class */ + Bitmapset *ec_member_indexes; /* Indexes into all PlannerInfos + * eq_members for this EquivalenceClass */ + Bitmapset *ec_nonchild_indexes; /* Indexes into PlannerInfo's + * eq_members with em_is_child == + * false */ + Bitmapset *ec_norel_indexes; /* Indexes into PlannerInfo's eq_members + * for members where pull_varno on the + * em_expr is an empty set */ + Bitmapset *ec_source_indexes; /* indexes into PlannerInfo's eq_sources + * list of generating RestrictInfos */ + Bitmapset *ec_derive_indexes; /* indexes into PlannerInfo's eq_derives + * list of derived RestrictInfos */ Relids ec_relids; /* all relids appearing in ec_members, except * for child members (see below) */ bool ec_has_const; /* any pseudoconstants in ec_members? */ @@ -1373,9 +1443,10 @@ typedef struct EquivalenceMember NodeTag type; Expr *em_expr; /* the expression represented */ - Relids em_relids; /* all relids appearing in em_expr */ + Relids em_relids; /* relids for this member */ Relids em_nullable_relids; /* nullable by lower outer joins */ - bool em_is_const; /* expression is pseudoconstant? */ + bool em_is_const; /* is em_relids empty? */ + bool em_norel_expr; /* true if em_expr contains no Vars */ bool em_is_child; /* derived version for a child relation? */ Oid em_datatype; /* the "nominal type" used by the opfamily */ } EquivalenceMember; diff --git a/src/include/nodes/pg_list.h b/src/include/nodes/pg_list.h index 529a382d28..77f26edf22 100644 --- a/src/include/nodes/pg_list.h +++ b/src/include/nodes/pg_list.h @@ -629,6 +629,7 @@ extern pg_nodiscard List *list_copy_deep(const List *oldlist); typedef int (*list_sort_comparator) (const ListCell *a, const ListCell *b); extern void list_sort(List *list, list_sort_comparator cmp); +extern int list_ptr_cmp(const ListCell *p1, const ListCell *p2); extern int list_int_cmp(const ListCell *p1, const ListCell *p2); extern int list_oid_cmp(const ListCell *p1, const ListCell *p2); diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h index 9b38627efd..4d2221c2d0 100644 --- a/src/include/optimizer/paths.h +++ b/src/include/optimizer/paths.h @@ -114,6 +114,31 @@ extern void mark_dummy_rel(RelOptInfo *rel); * equivclass.c * routines for managing EquivalenceClasses */ + +/* + * EquivalenceMemberIterator + * Data structure used for iteration over an EquivalenceClass's + * EquivalenceMember in order to quickly find the members matching our + * search pattern. + */ +typedef struct EquivalenceMemberIterator +{ + bool use_index; /* use matching_ems index? */ + bool relids_empty; /* is with_relids empty? */ + bool with_children; /* include em_is_child members? */ + bool with_norel_members; /* include members with empty em_relids */ +#ifdef USE_ASSERT_CHECKING + bool isstrict; /* ensure the correct next function is used */ +#endif + int orig_length; /* elements in eclass->ec_members at the start */ + int current_index; /* current iterator position, or -1. */ + Relids with_relids; /* relids to match in em_relids */ + PlannerInfo *root; /* PlannerInfo the eclass belongs to */ + EquivalenceClass *eclass; /* the EquivalenceClass we're looking at */ + Bitmapset *matching_ems; /* when use_index == true, these are the + * matching indexes in root eq_members */ +} EquivalenceMemberIterator; + typedef bool (*ec_matches_callback_type) (PlannerInfo *root, RelOptInfo *rel, EquivalenceClass *ec, @@ -135,7 +160,8 @@ extern EquivalenceClass *get_eclass_for_sort_expr(PlannerInfo *root, Index sortref, Relids rel, bool create_it); -extern EquivalenceMember *find_ec_member_matching_expr(EquivalenceClass *ec, +extern EquivalenceMember *find_ec_member_matching_expr(PlannerInfo *root, + EquivalenceClass *ec, Expr *expr, Relids relids); extern EquivalenceMember *find_computable_ec_member(PlannerInfo *root, @@ -160,7 +186,8 @@ extern bool exprs_known_equal(PlannerInfo *root, Node *item1, Node *item2); extern EquivalenceClass *match_eclasses_to_foreign_key_col(PlannerInfo *root, ForeignKeyOptInfo *fkinfo, int colno); -extern RestrictInfo *find_derived_clause_for_ec_member(EquivalenceClass *ec, +extern RestrictInfo *find_derived_clause_for_ec_member(PlannerInfo *root, + EquivalenceClass *ec, EquivalenceMember *em); extern void add_child_rel_equivalences(PlannerInfo *root, AppendRelInfo *appinfo, @@ -186,6 +213,42 @@ extern bool eclass_useful_for_merging(PlannerInfo *root, extern bool is_redundant_derived_clause(RestrictInfo *rinfo, List *clauselist); extern bool is_redundant_with_indexclauses(RestrictInfo *rinfo, List *indexclauses); +extern Bitmapset *get_ecmember_indexes(PlannerInfo *root, + EquivalenceClass *ec, + Relids relids, + bool with_children, + bool with_norel_members); +extern Bitmapset *get_ecmember_indexes_strict(PlannerInfo *root, + EquivalenceClass *ec, + Relids relids, + bool with_children, + bool with_norel_members); +extern void setup_eclass_member_iterator(EquivalenceMemberIterator *iter, + PlannerInfo *root, + EquivalenceClass *ec, Relids relids, + bool with_children, + bool with_norel_members); +extern void setup_eclass_member_strict_iterator(EquivalenceMemberIterator *iter, + PlannerInfo *root, + EquivalenceClass *ec, + Relids relids, + bool with_children, + bool with_norel_members); +extern EquivalenceMember *eclass_member_iterator_next(EquivalenceMemberIterator *iter); +extern EquivalenceMember *eclass_member_iterator_strict_next(EquivalenceMemberIterator *iter); +extern void eclass_member_iterator_dispose(EquivalenceMemberIterator *iter); +extern Bitmapset *get_ec_source_indexes(PlannerInfo *root, + EquivalenceClass *ec, + Relids relids); +extern Bitmapset *get_ec_source_indexes_strict(PlannerInfo *root, + EquivalenceClass *ec, + Relids relids); +extern Bitmapset *get_ec_derive_indexes(PlannerInfo *root, + EquivalenceClass *ec, + Relids relids); +extern Bitmapset *get_ec_derive_indexes_strict(PlannerInfo *root, + EquivalenceClass *ec, + Relids relids); /* * pathkeys.c diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 51484ca7e2..2e9c6c530f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -641,6 +641,7 @@ EphemeralNamedRelationMetadata EphemeralNamedRelationMetadataData EquivalenceClass EquivalenceMember +EquivalenceMemberIterator ErrorContextCallback ErrorData ErrorSaveContext -- 2.39.0.windows.1 Attachments: [text/plain] v15-0001-Adjust-bms_int_members-so-that-it-shortens-the-l.patch (2.5K, ../../CAApHDvoTcHHB2Fi-hDbsS4mkgxRC3S4SmMkEyXNdpmjUySFKuQ@mail.gmail.com/2-v15-0001-Adjust-bms_int_members-so-that-it-shortens-the-l.patch) download | inline diff: From 177d013928d2bb5c0c4ee565a433bbf62481b18c Mon Sep 17 00:00:00 2001 From: David Rowley <[email protected]> Date: Thu, 1 Dec 2022 20:36:10 +1300 Subject: [PATCH v15 1/2] Adjust bms_int_members so that it shortens the left input Prior to this change, if the left input to bms_int_members had fewer Bitmapwords than the right input, the additional words in the left input would be zeroed. Doing things this why only really makes sense if we expect to add additional words to the resulting bitmap set again later. Seemingly, most of our use cases don't add to the resulting Bitmapset, in fact, many cases are performing bms_int_members in a loop which is likely to even further increase the amount of all zero words in the set. Leaving these zero trailing words in place can cause inefficiencies in functions such as bms_is_empty and loops which loop over a set using bms_next_member when the set is left with a large number of trailing words. If there are use cases which need to add additional words again, then, repalloc has to do very little processing when the requested allocation size is <= the actual allocation size. Both AllocSetRealloc and GenerationRealloc simply just return the input pointer again. If there are any cases where we do increase the size of the set again, then this just shifts the zeroing of the additional words until then. We could give bms_del_members() and bms_difference similar treatment, but that would require additional code to record the last non-zero word after masking out the members being deleted. That seems less worthwhile as it will add cost when we're unable to trim down the number of words. --- src/backend/nodes/bitmapset.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/src/backend/nodes/bitmapset.c b/src/backend/nodes/bitmapset.c index 98660524ad..b13b4c378f 100644 --- a/src/backend/nodes/bitmapset.c +++ b/src/backend/nodes/bitmapset.c @@ -920,8 +920,16 @@ bms_int_members(Bitmapset *a, const Bitmapset *b) shortlen = Min(a->nwords, b->nwords); for (i = 0; i < shortlen; i++) a->words[i] &= b->words[i]; - for (; i < a->nwords; i++) - a->words[i] = 0; + + /* + * We simply shorten the left input to both remove the need to zero the + * trailing words and also to reduce the required processing if this + * function is being called in a loop. If more words are required later + * then AllocSetRealloc is likely not going to have to work very hard if + * the new requested size is >= the actual size of the allocation. + */ + a->nwords = shortlen; + return a; } -- 2.39.0.windows.1 [text/plain] v15-0002-Add-Bitmapset-indexes-for-faster-lookup-of-Equiv.patch (102.5K, ../../CAApHDvoTcHHB2Fi-hDbsS4mkgxRC3S4SmMkEyXNdpmjUySFKuQ@mail.gmail.com/3-v15-0002-Add-Bitmapset-indexes-for-faster-lookup-of-Equiv.patch) download | inline diff: From ba89effb12c3547bf843bd0e991b454388cfb88e Mon Sep 17 00:00:00 2001 From: David Rowley <[email protected]> Date: Sun, 4 Dec 2022 11:09:21 +1300 Subject: [PATCH v15 2/2] Add Bitmapset indexes for faster lookup of EquivalenceMembers For queries containing a large number of joins or for queries to partitioned tables, the looking up of EquivalenceMembers in an EquivalenceClass can become slow due to the large number of members. Until now, the searching for EquivalenceMembers has been done using a linear search over the EquivalenceClass's list of members. Here we effectively add an index to allow faster lookups of these members. This is done by way of adding all members to all classes into a List in PlannerInfo, then we have a Bitmapset in each EquivalenceClass with members for each index in PlannerInfo's list which belong to this class. Additionally, each RelOptInfo also maintains another Bitmapset which stores each index in PlannerInfo's list of EquivalenceMember for all members, in all classes, which mention this relation. This allows us to quickly find all EquivalenceMembers for a given RelOptInfo and a given EquivalenceClass by intersecting these two lists. When searching for members belonging to join rels, we must either intersect or union all base RelOptInfo's member indexes then intersect the result to the EquivalenceClass's indexes. Whether we intersect or union depends on if we need members that mention all relations or members that mention any of the base relations making up the join relation. This method of indexing can significantly speed up the planner when querying a partitioned table with a large number of partitions when the query contains a few joins. Tests have shown around a x10 increase in performance with 1000 partitions. --- contrib/postgres_fdw/postgres_fdw.c | 50 +- src/backend/nodes/list.c | 16 + src/backend/nodes/outfuncs.c | 7 +- src/backend/optimizer/path/costsize.c | 3 +- src/backend/optimizer/path/equivclass.c | 1295 +++++++++++++++++---- src/backend/optimizer/path/indxpath.c | 22 +- src/backend/optimizer/path/pathkeys.c | 26 +- src/backend/optimizer/plan/createplan.c | 66 +- src/backend/optimizer/plan/planagg.c | 1 + src/backend/optimizer/plan/planner.c | 3 + src/backend/optimizer/prep/prepjointree.c | 3 + src/backend/optimizer/prep/prepunion.c | 25 +- src/backend/optimizer/util/relnode.c | 9 + src/include/nodes/pathnodes.h | 81 +- src/include/nodes/pg_list.h | 1 + src/include/optimizer/paths.h | 67 +- src/tools/pgindent/typedefs.list | 1 + 17 files changed, 1347 insertions(+), 329 deletions(-) diff --git a/contrib/postgres_fdw/postgres_fdw.c b/contrib/postgres_fdw/postgres_fdw.c index 50d23f922c..b85c4a071e 100644 --- a/contrib/postgres_fdw/postgres_fdw.c +++ b/contrib/postgres_fdw/postgres_fdw.c @@ -7619,6 +7619,10 @@ conversion_error_callback(void *arg) varno = var->varno; colno = var->varattno; } + /* + * XXX why don't we break here? Surely there can't be another + * equal EquivalenceMember? + */ } if (varno > 0) @@ -7677,23 +7681,27 @@ conversion_error_callback(void *arg) EquivalenceMember * find_em_for_rel(PlannerInfo *root, EquivalenceClass *ec, RelOptInfo *rel) { - ListCell *lc; + EquivalenceMemberIterator iter; + EquivalenceMember *em; - foreach(lc, ec->ec_members) - { - EquivalenceMember *em = (EquivalenceMember *) lfirst(lc); + setup_eclass_member_iterator(&iter, root, ec, rel->relids, true, false); + while ((em = eclass_member_iterator_next(&iter)) != NULL) + { /* * Note we require !bms_is_empty, else we'd accept constant * expressions which are not suitable for the purpose. */ + Assert(!bms_is_empty(em->em_relids)); + if (bms_is_subset(em->em_relids, rel->relids) && - !bms_is_empty(em->em_relids) && is_foreign_expr(root, rel, em->em_expr)) - return em; + break; } - return NULL; + eclass_member_iterator_dispose(&iter); + + return em; } /* @@ -7720,7 +7728,9 @@ find_em_for_rel_target(PlannerInfo *root, EquivalenceClass *ec, { Expr *expr = (Expr *) lfirst(lc1); Index sgref = get_pathtarget_sortgroupref(target, i); - ListCell *lc2; + EquivalenceMemberIterator iter; + EquivalenceMember *em; + Relids expr_relids; /* Ignore non-sort expressions */ if (sgref == 0 || @@ -7735,19 +7745,20 @@ find_em_for_rel_target(PlannerInfo *root, EquivalenceClass *ec, while (expr && IsA(expr, RelabelType)) expr = ((RelabelType *) expr)->arg; + expr_relids = pull_varnos(root, (Node *) expr); + setup_eclass_member_strict_iterator(&iter, root, ec, expr_relids, + false, false); + /* Locate an EquivalenceClass member matching this expr, if any */ - foreach(lc2, ec->ec_members) + while ((em = eclass_member_iterator_strict_next(&iter)) != NULL) { - EquivalenceMember *em = (EquivalenceMember *) lfirst(lc2); Expr *em_expr; - /* Don't match constants */ - if (em->em_is_const) - continue; + /* don't expect constants */ + Assert(!em->em_is_const); - /* Ignore child members */ - if (em->em_is_child) - continue; + /* don't expect child members */ + Assert(!em->em_is_child); /* Match if same expression (after stripping relabel) */ em_expr = em->em_expr; @@ -7759,10 +7770,15 @@ find_em_for_rel_target(PlannerInfo *root, EquivalenceClass *ec, /* Check that expression (including relabels!) is shippable */ if (is_foreign_expr(root, rel, em->em_expr)) + { + bms_free(expr_relids); + eclass_member_iterator_dispose(&iter); return em; + } } - i++; + bms_free(expr_relids); + eclass_member_iterator_dispose(&iter); } return NULL; diff --git a/src/backend/nodes/list.c b/src/backend/nodes/list.c index a709d23ef1..b8972bbe5a 100644 --- a/src/backend/nodes/list.c +++ b/src/backend/nodes/list.c @@ -1683,6 +1683,22 @@ list_sort(List *list, list_sort_comparator cmp) qsort(list->elements, len, sizeof(ListCell), (qsort_comparator) cmp); } +/* + * list_sort comparator for sorting a list into ascending ptr order. + */ +int +list_ptr_cmp(const ListCell *p1, const ListCell *p2) +{ + void *v1 = lfirst(p1); + void *v2 = lfirst(p2); + + if (v1 < v2) + return -1; + if (v1 > v2) + return 1; + return 0; +} + /* * list_sort comparator for sorting a list into ascending int order. */ diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6b368b08b2..98becac509 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -463,8 +463,11 @@ _outEquivalenceClass(StringInfo str, const EquivalenceClass *node) WRITE_NODE_FIELD(ec_opfamilies); WRITE_OID_FIELD(ec_collation); WRITE_NODE_FIELD(ec_members); - WRITE_NODE_FIELD(ec_sources); - WRITE_NODE_FIELD(ec_derives); + WRITE_BITMAPSET_FIELD(ec_member_indexes); + WRITE_BITMAPSET_FIELD(ec_nonchild_indexes); + WRITE_BITMAPSET_FIELD(ec_norel_indexes); + WRITE_BITMAPSET_FIELD(ec_source_indexes); + WRITE_BITMAPSET_FIELD(ec_derive_indexes); WRITE_BITMAPSET_FIELD(ec_relids); WRITE_BOOL_FIELD(ec_has_const); WRITE_BOOL_FIELD(ec_has_volatile); diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index 29ae32d960..595aaf74f3 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -5501,7 +5501,8 @@ get_foreign_key_join_selectivity(PlannerInfo *root, if (ec && ec->ec_has_const) { EquivalenceMember *em = fkinfo->fk_eclass_member[i]; - RestrictInfo *rinfo = find_derived_clause_for_ec_member(ec, + RestrictInfo *rinfo = find_derived_clause_for_ec_member(root, + ec, em); if (rinfo) diff --git a/src/backend/optimizer/path/equivclass.c b/src/backend/optimizer/path/equivclass.c index 7d7e6facdf..4277548193 100644 --- a/src/backend/optimizer/path/equivclass.c +++ b/src/backend/optimizer/path/equivclass.c @@ -31,9 +31,14 @@ #include "optimizer/restrictinfo.h" #include "utils/lsyscache.h" - -static EquivalenceMember *add_eq_member(EquivalenceClass *ec, - Expr *expr, Relids relids, Relids nullable_relids, +static void add_eq_source(PlannerInfo *root, EquivalenceClass *ec, + RestrictInfo *rinfo); +static void add_eq_derive(PlannerInfo *root, EquivalenceClass *ec, + RestrictInfo *rinfo); +static EquivalenceMember *add_eq_member(PlannerInfo *root, + EquivalenceClass *ec, + Expr *expr, Relids relids, + Relids nullable_relids, bool is_child, Oid datatype); static bool is_exprlist_member(Expr *node, List *exprs); static void generate_base_implied_equalities_const(PlannerInfo *root, @@ -333,7 +338,6 @@ process_equivalence(PlannerInfo *root, /* If case 1, nothing to do, except add to sources */ if (ec1 == ec2) { - ec1->ec_sources = lappend(ec1->ec_sources, restrictinfo); ec1->ec_below_outer_join |= below_outer_join; ec1->ec_min_security = Min(ec1->ec_min_security, restrictinfo->security_level); @@ -345,6 +349,8 @@ process_equivalence(PlannerInfo *root, /* mark the RI as usable with this pair of EMs */ restrictinfo->left_em = em1; restrictinfo->right_em = em2; + + add_eq_source(root, ec1, restrictinfo); return true; } @@ -365,8 +371,16 @@ process_equivalence(PlannerInfo *root, * be found. */ ec1->ec_members = list_concat(ec1->ec_members, ec2->ec_members); - ec1->ec_sources = list_concat(ec1->ec_sources, ec2->ec_sources); - ec1->ec_derives = list_concat(ec1->ec_derives, ec2->ec_derives); + ec1->ec_member_indexes = bms_add_members(ec1->ec_member_indexes, + ec2->ec_member_indexes); + ec1->ec_nonchild_indexes = bms_add_members(ec1->ec_nonchild_indexes, + ec2->ec_nonchild_indexes); + ec1->ec_norel_indexes = bms_add_members(ec1->ec_norel_indexes, + ec2->ec_norel_indexes); + ec1->ec_source_indexes = bms_join(ec1->ec_source_indexes, + ec2->ec_source_indexes); + ec1->ec_derive_indexes = bms_join(ec1->ec_derive_indexes, + ec2->ec_derive_indexes); ec1->ec_relids = bms_join(ec1->ec_relids, ec2->ec_relids); ec1->ec_has_const |= ec2->ec_has_const; /* can't need to set has_volatile */ @@ -379,10 +393,12 @@ process_equivalence(PlannerInfo *root, root->eq_classes = list_delete_nth_cell(root->eq_classes, ec2_idx); /* just to avoid debugging confusion w/ dangling pointers: */ ec2->ec_members = NIL; - ec2->ec_sources = NIL; - ec2->ec_derives = NIL; + ec2->ec_member_indexes = NULL; + ec2->ec_nonchild_indexes = NULL; + ec2->ec_norel_indexes = NULL; + ec2->ec_source_indexes = NULL; + ec2->ec_derive_indexes = NULL; ec2->ec_relids = NULL; - ec1->ec_sources = lappend(ec1->ec_sources, restrictinfo); ec1->ec_below_outer_join |= below_outer_join; ec1->ec_min_security = Min(ec1->ec_min_security, restrictinfo->security_level); @@ -394,13 +410,14 @@ process_equivalence(PlannerInfo *root, /* mark the RI as usable with this pair of EMs */ restrictinfo->left_em = em1; restrictinfo->right_em = em2; + + add_eq_source(root, ec1, restrictinfo); } else if (ec1) { /* Case 3: add item2 to ec1 */ - em2 = add_eq_member(ec1, item2, item2_relids, item2_nullable_relids, - false, item2_type); - ec1->ec_sources = lappend(ec1->ec_sources, restrictinfo); + em2 = add_eq_member(root, ec1, item2, item2_relids, + item2_nullable_relids, false, item2_type); ec1->ec_below_outer_join |= below_outer_join; ec1->ec_min_security = Min(ec1->ec_min_security, restrictinfo->security_level); @@ -412,13 +429,14 @@ process_equivalence(PlannerInfo *root, /* mark the RI as usable with this pair of EMs */ restrictinfo->left_em = em1; restrictinfo->right_em = em2; + + add_eq_source(root, ec1, restrictinfo); } else if (ec2) { /* Case 3: add item1 to ec2 */ - em1 = add_eq_member(ec2, item1, item1_relids, item1_nullable_relids, - false, item1_type); - ec2->ec_sources = lappend(ec2->ec_sources, restrictinfo); + em1 = add_eq_member(root, ec2, item1, item1_relids, + item1_nullable_relids, false, item1_type); ec2->ec_below_outer_join |= below_outer_join; ec2->ec_min_security = Min(ec2->ec_min_security, restrictinfo->security_level); @@ -430,6 +448,8 @@ process_equivalence(PlannerInfo *root, /* mark the RI as usable with this pair of EMs */ restrictinfo->left_em = em1; restrictinfo->right_em = em2; + + add_eq_source(root, ec2, restrictinfo); } else { @@ -439,8 +459,11 @@ process_equivalence(PlannerInfo *root, ec->ec_opfamilies = opfamilies; ec->ec_collation = collation; ec->ec_members = NIL; - ec->ec_sources = list_make1(restrictinfo); - ec->ec_derives = NIL; + ec->ec_member_indexes = NULL; + ec->ec_nonchild_indexes = NULL; + ec->ec_norel_indexes = NULL; + ec->ec_source_indexes = NULL; + ec->ec_derive_indexes = NULL; ec->ec_relids = NULL; ec->ec_has_const = false; ec->ec_has_volatile = false; @@ -450,10 +473,10 @@ process_equivalence(PlannerInfo *root, ec->ec_min_security = restrictinfo->security_level; ec->ec_max_security = restrictinfo->security_level; ec->ec_merged = NULL; - em1 = add_eq_member(ec, item1, item1_relids, item1_nullable_relids, - false, item1_type); - em2 = add_eq_member(ec, item2, item2_relids, item2_nullable_relids, - false, item2_type); + em1 = add_eq_member(root, ec, item1, item1_relids, + item1_nullable_relids, false, item1_type); + em2 = add_eq_member(root, ec, item2, item2_relids, + item2_nullable_relids, false, item2_type); root->eq_classes = lappend(root->eq_classes, ec); @@ -463,6 +486,8 @@ process_equivalence(PlannerInfo *root, /* mark the RI as usable with this pair of EMs */ restrictinfo->left_em = em1; restrictinfo->right_em = em2; + + add_eq_source(root, ec, restrictinfo); } return true; @@ -538,14 +563,62 @@ canonicalize_ec_expression(Expr *expr, Oid req_type, Oid req_collation) return expr; } +/* + * add_eq_source - add 'rinfo' in eq_sources for this 'ec' + */ +static void +add_eq_source(PlannerInfo *root, EquivalenceClass *ec, RestrictInfo *rinfo) +{ + int source_idx = list_length(root->eq_sources); + int i; + + ec->ec_source_indexes = bms_add_member(ec->ec_source_indexes, source_idx); + root->eq_sources = lappend(root->eq_sources, rinfo); + + i = -1; + while ((i = bms_next_member(rinfo->clause_relids, i)) >= 0) + { + RelOptInfo *rel = root->simple_rel_array[i]; + + rel->eclass_source_indexes = bms_add_member(rel->eclass_source_indexes, + source_idx); + } +} + +/* + * add_eq_derive - add 'rinfo' in eq_derives for this 'ec' + */ +static void +add_eq_derive(PlannerInfo *root, EquivalenceClass *ec, RestrictInfo *rinfo) +{ + int derive_idx = list_length(root->eq_derives); + int i; + + ec->ec_derive_indexes = bms_add_member(ec->ec_derive_indexes, derive_idx); + root->eq_derives = lappend(root->eq_derives, rinfo); + + i = -1; + while ((i = bms_next_member(rinfo->clause_relids, i)) >= 0) + { + RelOptInfo *rel = root->simple_rel_array[i]; + + rel->eclass_derive_indexes = bms_add_member(rel->eclass_derive_indexes, + derive_idx); + } +} + /* * add_eq_member - build a new EquivalenceMember and add it to an EC */ static EquivalenceMember * -add_eq_member(EquivalenceClass *ec, Expr *expr, Relids relids, - Relids nullable_relids, bool is_child, Oid datatype) +add_eq_member(PlannerInfo *root, EquivalenceClass *ec, Expr *expr, + Relids relids, Relids nullable_relids, bool is_child, + Oid datatype) { EquivalenceMember *em = makeNode(EquivalenceMember); + Relids expr_relids; + int em_index = list_length(root->eq_members); + int i; em->em_expr = expr; em->em_relids = relids; @@ -554,6 +627,23 @@ add_eq_member(EquivalenceClass *ec, Expr *expr, Relids relids, em->em_is_child = is_child; em->em_datatype = datatype; + /* + * We must determine the exact set of relids in the expr for child + * EquivalenceMembers as what is given to us in 'relids' may differ from + * the relids mentioned in the expression. See add_child_rel_equivalences + */ + if (is_child) + expr_relids = pull_varnos(root, (Node *) expr); + else + { + expr_relids = relids; + /* We expect the relids to match for non-child members */ + Assert(bms_equal(pull_varnos(root, (Node *) expr), relids)); + } + + /* record the actual relids from 'expr' */ + em->em_norel_expr = bms_is_empty(expr_relids); + if (bms_is_empty(relids)) { /* @@ -572,9 +662,31 @@ add_eq_member(EquivalenceClass *ec, Expr *expr, Relids relids, else if (!is_child) /* child members don't add to ec_relids */ { ec->ec_relids = bms_add_members(ec->ec_relids, relids); + ec->ec_nonchild_indexes = bms_add_member(ec->ec_nonchild_indexes, em_index); } + + /* add the new member to the list */ ec->ec_members = lappend(ec->ec_members, em); + /* and add it to the index and PlannerInfo's list */ + ec->ec_member_indexes = bms_add_member(ec->ec_member_indexes, em_index); + root->eq_members = lappend(root->eq_members, em); + + /* record exprs with no relids */ + if (bms_is_empty(expr_relids)) + ec->ec_norel_indexes = bms_add_member(ec->ec_norel_indexes, em_index); + + if (is_child) + expr_relids = bms_add_members(expr_relids, relids); + + i = -1; + while ((i = bms_next_member(expr_relids, i)) >= 0) + { + RelOptInfo *rel = root->simple_rel_array[i]; + + rel->eclass_member_indexes = bms_add_member(rel->eclass_member_indexes, em_index); + } + return em; } @@ -638,6 +750,7 @@ get_eclass_for_sort_expr(PlannerInfo *root, * Ensure the expression exposes the correct type and collation. */ expr = canonicalize_ec_expression(expr, opcintype, collation); + expr_relids = pull_varnos(root, (Node *) expr); /* * Scan through the existing EquivalenceClasses for a match @@ -645,7 +758,8 @@ get_eclass_for_sort_expr(PlannerInfo *root, foreach(lc1, root->eq_classes) { EquivalenceClass *cur_ec = (EquivalenceClass *) lfirst(lc1); - ListCell *lc2; + EquivalenceMemberIterator iter; + EquivalenceMember *cur_em; /* * Never match to a volatile EC, except when we are looking at another @@ -660,10 +774,11 @@ get_eclass_for_sort_expr(PlannerInfo *root, if (!equal(opfamilies, cur_ec->ec_opfamilies)) continue; - foreach(lc2, cur_ec->ec_members) - { - EquivalenceMember *cur_em = (EquivalenceMember *) lfirst(lc2); + setup_eclass_member_strict_iterator(&iter, root, cur_ec, expr_relids, + true, true); + while ((cur_em = eclass_member_iterator_strict_next(&iter)) != NULL) + { /* * Ignore child members unless they match the request. */ @@ -683,6 +798,8 @@ get_eclass_for_sort_expr(PlannerInfo *root, equal(expr, cur_em->em_expr)) return cur_ec; /* Match! */ } + + eclass_member_iterator_dispose(&iter); } /* No match; does caller want a NULL result? */ @@ -700,8 +817,11 @@ get_eclass_for_sort_expr(PlannerInfo *root, newec->ec_opfamilies = list_copy(opfamilies); newec->ec_collation = collation; newec->ec_members = NIL; - newec->ec_sources = NIL; - newec->ec_derives = NIL; + newec->ec_member_indexes = NULL; + newec->ec_nonchild_indexes = NULL; + newec->ec_norel_indexes = NULL; + newec->ec_source_indexes = NULL; + newec->ec_derive_indexes = NULL; newec->ec_relids = NULL; newec->ec_has_const = false; newec->ec_has_volatile = contain_volatile_functions((Node *) expr); @@ -718,10 +838,9 @@ get_eclass_for_sort_expr(PlannerInfo *root, /* * Get the precise set of nullable relids appearing in the expression. */ - expr_relids = pull_varnos(root, (Node *) expr); nullable_relids = bms_intersect(nullable_relids, expr_relids); - newem = add_eq_member(newec, copyObject(expr), expr_relids, + newem = add_eq_member(root, newec, copyObject(expr), expr_relids, nullable_relids, false, opcintype); /* @@ -753,7 +872,7 @@ get_eclass_for_sort_expr(PlannerInfo *root, int ec_index = list_length(root->eq_classes) - 1; int i = -1; - while ((i = bms_next_member(newec->ec_relids, i)) > 0) + while ((i = bms_next_member(newec->ec_relids, i)) >= 0) { RelOptInfo *rel = root->simple_rel_array[i]; @@ -783,19 +902,25 @@ get_eclass_for_sort_expr(PlannerInfo *root, * Child EC members are ignored unless they belong to given 'relids'. */ EquivalenceMember * -find_ec_member_matching_expr(EquivalenceClass *ec, +find_ec_member_matching_expr(PlannerInfo *root, + EquivalenceClass *ec, Expr *expr, Relids relids) { - ListCell *lc; + EquivalenceMemberIterator iter; + Relids expr_relids; + EquivalenceMember *em; /* We ignore binary-compatible relabeling on both ends */ while (expr && IsA(expr, RelabelType)) expr = ((RelabelType *) expr)->arg; - foreach(lc, ec->ec_members) + expr_relids = pull_varnos(root, (Node *) expr); + setup_eclass_member_strict_iterator(&iter, root, ec, expr_relids, true, + true); + + while ((em = eclass_member_iterator_strict_next(&iter)) != NULL) { - EquivalenceMember *em = (EquivalenceMember *) lfirst(lc); Expr *emexpr; /* @@ -820,10 +945,13 @@ find_ec_member_matching_expr(EquivalenceClass *ec, emexpr = ((RelabelType *) emexpr)->arg; if (equal(emexpr, expr)) - return em; + break; } - return NULL; + bms_free(expr_relids); + eclass_member_iterator_dispose(&iter); + + return em; } /* @@ -965,7 +1093,7 @@ relation_can_be_sorted_early(PlannerInfo *root, RelOptInfo *rel, { Expr *targetexpr = (Expr *) lfirst(lc); - em = find_ec_member_matching_expr(ec, targetexpr, rel->relids); + em = find_ec_member_matching_expr(root, ec, targetexpr, rel->relids); if (!em) continue; @@ -1052,7 +1180,7 @@ relation_can_be_sorted_early(PlannerInfo *root, RelOptInfo *rel, * scanning of the quals and before Path construction begins. * * We make no attempt to avoid generating duplicate RestrictInfos here: we - * don't search ec_sources or ec_derives for matches. It doesn't really + * don't search eq_sources or eq_derives for matches. It doesn't really * seem worth the trouble to do so. */ void @@ -1109,7 +1237,7 @@ generate_base_implied_equalities(PlannerInfo *root) * this is a cheap version of has_relevant_eclass_joinclause(). */ i = -1; - while ((i = bms_next_member(ec->ec_relids, i)) > 0) + while ((i = bms_next_member(ec->ec_relids, i)) >= 0) { RelOptInfo *rel = root->simple_rel_array[i]; @@ -1135,6 +1263,7 @@ generate_base_implied_equalities_const(PlannerInfo *root, { EquivalenceMember *const_em = NULL; ListCell *lc; + int i; /* * In the trivial case where we just had one "var = const" clause, push @@ -1144,9 +1273,9 @@ generate_base_implied_equalities_const(PlannerInfo *root, * equivalent to the old one. */ if (list_length(ec->ec_members) == 2 && - list_length(ec->ec_sources) == 1) + bms_get_singleton_member(ec->ec_source_indexes, &i)) { - RestrictInfo *restrictinfo = (RestrictInfo *) linitial(ec->ec_sources); + RestrictInfo *restrictinfo = list_nth_node(RestrictInfo, root->eq_sources, i); if (bms_membership(restrictinfo->required_relids) != BMS_MULTIPLE) { @@ -1161,9 +1290,11 @@ generate_base_implied_equalities_const(PlannerInfo *root, * machinery might be able to exclude relations on the basis of generated * "var = const" equalities, but "var = param" won't work for that. */ - foreach(lc, ec->ec_members) + i = -1; + while ((i = bms_next_member(ec->ec_norel_indexes, i)) >= 0) { - EquivalenceMember *cur_em = (EquivalenceMember *) lfirst(lc); + EquivalenceMember *cur_em = list_nth_node(EquivalenceMember, + root->eq_members, i); if (cur_em->em_is_const) { @@ -1204,9 +1335,9 @@ generate_base_implied_equalities_const(PlannerInfo *root, /* * If the clause didn't degenerate to a constant, fill in the correct - * markings for a mergejoinable clause, and save it in ec_derives. (We + * markings for a mergejoinable clause, and save it in eq_derives. (We * will not re-use such clauses directly, but selectivity estimation - * may consult the list later. Note that this use of ec_derives does + * may consult the list later. Note that this use of eq_derives does * not overlap with its use for join clauses, since we never generate * join clauses from an ec_has_const eclass.) */ @@ -1216,7 +1347,8 @@ generate_base_implied_equalities_const(PlannerInfo *root, rinfo->left_ec = rinfo->right_ec = ec; rinfo->left_em = cur_em; rinfo->right_em = const_em; - ec->ec_derives = lappend(ec->ec_derives, rinfo); + + add_eq_derive(root, ec, rinfo); } } } @@ -1229,7 +1361,8 @@ generate_base_implied_equalities_no_const(PlannerInfo *root, EquivalenceClass *ec) { EquivalenceMember **prev_ems; - ListCell *lc; + Bitmapset *matching_ems; + int i; /* * We scan the EC members once and track the last-seen member for each @@ -1242,9 +1375,12 @@ generate_base_implied_equalities_no_const(PlannerInfo *root, prev_ems = (EquivalenceMember **) palloc0(root->simple_rel_array_size * sizeof(EquivalenceMember *)); - foreach(lc, ec->ec_members) + matching_ems = bms_difference(ec->ec_member_indexes, ec->ec_norel_indexes); + i = -1; + while ((i = bms_next_member(matching_ems, i)) >= 0) { - EquivalenceMember *cur_em = (EquivalenceMember *) lfirst(lc); + EquivalenceMember *cur_em = list_nth_node(EquivalenceMember, + root->eq_members, i); int relid; Assert(!cur_em->em_is_child); /* no children yet */ @@ -1279,7 +1415,7 @@ generate_base_implied_equalities_no_const(PlannerInfo *root, /* * If the clause didn't degenerate to a constant, fill in the * correct markings for a mergejoinable clause. We don't put it - * in ec_derives however; we don't currently need to re-find such + * in eq_derives however; we don't currently need to re-find such * clauses, and we don't want to clutter that list with non-join * clauses. */ @@ -1302,11 +1438,15 @@ generate_base_implied_equalities_no_const(PlannerInfo *root, * For the moment we force all the Vars to be available at all join nodes * for this eclass. Perhaps this could be improved by doing some * pre-analysis of which members we prefer to join, but it's no worse than - * what happened in the pre-8.3 code. + * what happened in the pre-8.3 code. We're able to make use of + * matching_ems from above. We're not going to find Vars in + * em_const_indexes. */ - foreach(lc, ec->ec_members) + i = -1; + while ((i = bms_next_member(matching_ems, i)) >= 0) { - EquivalenceMember *cur_em = (EquivalenceMember *) lfirst(lc); + EquivalenceMember *cur_em = list_nth_node(EquivalenceMember, + root->eq_members, i); List *vars = pull_var_clause((Node *) cur_em->em_expr, PVC_RECURSE_AGGREGATES | PVC_RECURSE_WINDOWFUNCS | @@ -1315,6 +1455,7 @@ generate_base_implied_equalities_no_const(PlannerInfo *root, add_vars_to_targetlist(root, vars, ec->ec_relids); list_free(vars); } + bms_free(matching_ems); } /* @@ -1335,11 +1476,12 @@ static void generate_base_implied_equalities_broken(PlannerInfo *root, EquivalenceClass *ec) { - ListCell *lc; + int i = -1; - foreach(lc, ec->ec_sources) + while ((i = bms_next_member(ec->ec_source_indexes, i)) >= 0) { - RestrictInfo *restrictinfo = (RestrictInfo *) lfirst(lc); + RestrictInfo *restrictinfo = list_nth_node(RestrictInfo, + root->eq_sources, i); if (ec->ec_has_const || bms_membership(restrictinfo->required_relids) != BMS_MULTIPLE) @@ -1380,11 +1522,11 @@ generate_base_implied_equalities_broken(PlannerInfo *root, * Because the same join clauses are likely to be needed multiple times as * we consider different join paths, we avoid generating multiple copies: * whenever we select a particular pair of EquivalenceMembers to join, - * we check to see if the pair matches any original clause (in ec_sources) - * or previously-built clause (in ec_derives). This saves memory and allows - * re-use of information cached in RestrictInfos. We also avoid generating - * commutative duplicates, i.e. if the algorithm selects "a.x = b.y" but - * we already have "b.y = a.x", we return the existing clause. + * we check to see if the pair matches any original clause in root's + * eq_sources or previously-built clause (in root's eq_derives). This saves + * memory and allows re-use of information cached in RestrictInfos. We also + * avoid generating commutative duplicates, i.e. if the algorithm selects + * "a.x = b.y" but we already have "b.y = a.x", we return the existing clause. * * join_relids should always equal bms_union(outer_relids, inner_rel->relids). * We could simplify this function's API by computing it internally, but in @@ -1546,6 +1688,8 @@ generate_join_implied_equalities_normal(PlannerInfo *root, Relids outer_relids, Relids inner_relids) { + EquivalenceMemberIterator iter; + EquivalenceMember *cur_em; List *result = NIL; List *new_members = NIL; List *outer_members = NIL; @@ -1561,10 +1705,10 @@ generate_join_implied_equalities_normal(PlannerInfo *root, * as well as to at least one input member, plus enforce at least one * outer-rel member equal to at least one inner-rel member. */ - foreach(lc1, ec->ec_members) - { - EquivalenceMember *cur_em = (EquivalenceMember *) lfirst(lc1); + setup_eclass_member_iterator(&iter, root, ec, join_relids, true, false); + while ((cur_em = eclass_member_iterator_next(&iter)) != NULL) + { /* * We don't need to check explicitly for child EC members. This test * against join_relids will cause them to be ignored except when @@ -1581,6 +1725,8 @@ generate_join_implied_equalities_normal(PlannerInfo *root, new_members = lappend(new_members, cur_em); } + eclass_member_iterator_dispose(&iter); + /* * First, select the joinclause if needed. We can equate any one outer * member to any one inner member, but we have to find a datatype @@ -1678,7 +1824,7 @@ generate_join_implied_equalities_normal(PlannerInfo *root, foreach(lc1, new_members) { - EquivalenceMember *cur_em = (EquivalenceMember *) lfirst(lc1); + cur_em = (EquivalenceMember *) lfirst(lc1); if (prev_em != NULL) { @@ -1723,12 +1869,16 @@ generate_join_implied_equalities_broken(PlannerInfo *root, Relids nominal_inner_relids, RelOptInfo *inner_rel) { + Bitmapset *matching_es; List *result = NIL; - ListCell *lc; + int i; - foreach(lc, ec->ec_sources) + matching_es = get_ec_source_indexes(root, ec, nominal_join_relids); + i = -1; + while ((i = bms_next_member(matching_es, i)) >= 0) { - RestrictInfo *restrictinfo = (RestrictInfo *) lfirst(lc); + RestrictInfo *restrictinfo = list_nth_node(RestrictInfo, + root->eq_sources, i); Relids clause_relids = restrictinfo->required_relids; if (bms_is_subset(clause_relids, nominal_join_relids) && @@ -1740,12 +1890,12 @@ generate_join_implied_equalities_broken(PlannerInfo *root, /* * If we have to translate, just brute-force apply adjust_appendrel_attrs * to all the RestrictInfos at once. This will result in returning - * RestrictInfos that are not listed in ec_derives, but there shouldn't be + * RestrictInfos that are not listed in eq_derives, but there shouldn't be * any duplication, and it's a sufficiently narrow corner case that we * shouldn't sweat too much over it anyway. * * Since inner_rel might be an indirect descendant of the baserel - * mentioned in the ec_sources clauses, we have to be prepared to apply + * mentioned in the eq_sources clauses, we have to be prepared to apply * multiple levels of Var translation. */ if (IS_OTHER_REL(inner_rel) && result != NIL) @@ -1807,9 +1957,10 @@ create_join_clause(PlannerInfo *root, EquivalenceMember *rightem, EquivalenceClass *parent_ec) { + Bitmapset *matches; RestrictInfo *rinfo; - ListCell *lc; MemoryContext oldcontext; + int i; /* * Search to see if we already built a RestrictInfo for this pair of @@ -1820,9 +1971,12 @@ create_join_clause(PlannerInfo *root, * it's not identical, it'd better have the same effects, or the operator * families we're using are broken. */ - foreach(lc, ec->ec_sources) + matches = bms_int_members(get_ec_source_indexes_strict(root, ec, leftem->em_relids), + get_ec_source_indexes_strict(root, ec, rightem->em_relids)); + i = -1; + while ((i = bms_next_member(matches, i)) >= 0) { - rinfo = (RestrictInfo *) lfirst(lc); + rinfo = list_nth_node(RestrictInfo, root->eq_sources, i); if (rinfo->left_em == leftem && rinfo->right_em == rightem && rinfo->parent_ec == parent_ec) @@ -1833,9 +1987,13 @@ create_join_clause(PlannerInfo *root, return rinfo; } - foreach(lc, ec->ec_derives) + matches = bms_int_members(get_ec_derive_indexes_strict(root, ec, leftem->em_relids), + get_ec_derive_indexes_strict(root, ec, rightem->em_relids)); + + i = -1; + while ((i = bms_next_member(matches, i)) >= 0) { - rinfo = (RestrictInfo *) lfirst(lc); + rinfo = list_nth_node(RestrictInfo, root->eq_derives, i); if (rinfo->left_em == leftem && rinfo->right_em == rightem && rinfo->parent_ec == parent_ec) @@ -1877,7 +2035,7 @@ create_join_clause(PlannerInfo *root, rinfo->left_em = leftem; rinfo->right_em = rightem; /* and save it for possible re-use */ - ec->ec_derives = lappend(ec->ec_derives, rinfo); + add_eq_derive(root, ec, rinfo); MemoryContextSwitchTo(oldcontext); @@ -2069,7 +2227,8 @@ reconsider_outer_join_clause(PlannerInfo *root, RestrictInfo *rinfo, left_type, right_type, inner_datatype; - Relids inner_relids, + Relids outer_relids, + inner_relids, inner_nullable_relids; ListCell *lc1; @@ -2088,6 +2247,7 @@ reconsider_outer_join_clause(PlannerInfo *root, RestrictInfo *rinfo, outervar = (Expr *) get_leftop(rinfo->clause); innervar = (Expr *) get_rightop(rinfo->clause); inner_datatype = right_type; + outer_relids = rinfo->left_relids; inner_relids = rinfo->right_relids; } else @@ -2095,6 +2255,7 @@ reconsider_outer_join_clause(PlannerInfo *root, RestrictInfo *rinfo, outervar = (Expr *) get_rightop(rinfo->clause); innervar = (Expr *) get_leftop(rinfo->clause); inner_datatype = left_type; + outer_relids = rinfo->right_relids; inner_relids = rinfo->left_relids; } inner_nullable_relids = bms_intersect(inner_relids, @@ -2104,8 +2265,10 @@ reconsider_outer_join_clause(PlannerInfo *root, RestrictInfo *rinfo, foreach(lc1, root->eq_classes) { EquivalenceClass *cur_ec = (EquivalenceClass *) lfirst(lc1); + EquivalenceMemberIterator iter; + EquivalenceMember *cur_em; bool match; - ListCell *lc2; + int i; /* Ignore EC unless it contains pseudoconstants */ if (!cur_ec->ec_has_const) @@ -2120,10 +2283,12 @@ reconsider_outer_join_clause(PlannerInfo *root, RestrictInfo *rinfo, continue; /* Does it contain a match to outervar? */ match = false; - foreach(lc2, cur_ec->ec_members) - { - EquivalenceMember *cur_em = (EquivalenceMember *) lfirst(lc2); + setup_eclass_member_strict_iterator(&iter, root, cur_ec, outer_relids, + false, true); + + while ((cur_em = eclass_member_iterator_strict_next(&iter)) != NULL) + { Assert(!cur_em->em_is_child); /* no children yet */ if (equal(outervar, cur_em->em_expr)) { @@ -2131,6 +2296,8 @@ reconsider_outer_join_clause(PlannerInfo *root, RestrictInfo *rinfo, break; } } + eclass_member_iterator_dispose(&iter); + if (!match) continue; /* no match, so ignore this EC */ @@ -2140,12 +2307,14 @@ reconsider_outer_join_clause(PlannerInfo *root, RestrictInfo *rinfo, * constant before we can decide to throw away the outer-join clause. */ match = false; - foreach(lc2, cur_ec->ec_members) + i = -1; + while ((i = bms_next_member(cur_ec->ec_norel_indexes, i)) >= 0) { - EquivalenceMember *cur_em = (EquivalenceMember *) lfirst(lc2); Oid eq_op; RestrictInfo *newrinfo; + cur_em = list_nth_node(EquivalenceMember, root->eq_members, i); + if (!cur_em->em_is_const) continue; /* ignore non-const members */ eq_op = select_equality_operator(cur_ec, @@ -2221,11 +2390,12 @@ reconsider_full_join_clause(PlannerInfo *root, RestrictInfo *rinfo) { EquivalenceClass *cur_ec = (EquivalenceClass *) lfirst(lc1); EquivalenceMember *coal_em = NULL; + Bitmapset *matching_ems; bool match; bool matchleft; bool matchright; - ListCell *lc2; int coal_idx = -1; + int i; /* Ignore EC unless it contains pseudoconstants */ if (!cur_ec->ec_has_const) @@ -2252,10 +2422,14 @@ reconsider_full_join_clause(PlannerInfo *root, RestrictInfo *rinfo) * the two column types). Is it OK to strip implicit coercions from * the COALESCE arguments? */ + matching_ems = get_ecmember_indexes_strict(root, cur_ec, + rinfo->clause_relids, true, + false); match = false; - foreach(lc2, cur_ec->ec_members) + i = -1; + while ((i = bms_next_member(matching_ems, i)) >= 0) { - coal_em = (EquivalenceMember *) lfirst(lc2); + coal_em = list_nth_node(EquivalenceMember, root->eq_members, i); Assert(!coal_em->em_is_child); /* no children yet */ if (IsA(coal_em->em_expr, CoalesceExpr)) { @@ -2270,7 +2444,7 @@ reconsider_full_join_clause(PlannerInfo *root, RestrictInfo *rinfo) if (equal(leftvar, cfirst) && equal(rightvar, csecond)) { - coal_idx = foreach_current_index(lc2); + coal_idx = i; match = true; break; } @@ -2286,9 +2460,11 @@ reconsider_full_join_clause(PlannerInfo *root, RestrictInfo *rinfo) * decide to throw away the outer-join clause. */ matchleft = matchright = false; - foreach(lc2, cur_ec->ec_members) + i = -1; + while ((i = bms_next_member(cur_ec->ec_norel_indexes, i)) >= 0) { - EquivalenceMember *cur_em = (EquivalenceMember *) lfirst(lc2); + EquivalenceMember *cur_em = list_nth_node(EquivalenceMember, + root->eq_members, i); Oid eq_op; RestrictInfo *newrinfo; @@ -2333,11 +2509,28 @@ reconsider_full_join_clause(PlannerInfo *root, RestrictInfo *rinfo) * we can throw away the full-join clause as redundant. Moreover, we * can remove the COALESCE entry from the EC, since the added * restrictions ensure it will always have the expected value. (We - * don't bother trying to update ec_relids or ec_sources.) + * don't bother trying to update ec_relids or root's eq_sources.) */ if (matchleft && matchright) { - cur_ec->ec_members = list_delete_nth_cell(cur_ec->ec_members, coal_idx); + /* XXX performance of list_delete()?? */ + cur_ec->ec_members = list_delete(cur_ec->ec_members, coal_em); + cur_ec->ec_member_indexes = bms_del_member(cur_ec->ec_member_indexes, + coal_idx); + cur_ec->ec_nonchild_indexes = bms_del_member(cur_ec->ec_nonchild_indexes, + coal_idx); + /* no need to adjust ec_norel_indexes */ + + /* Remove the member from each of the relations */ + i = -1; + while ((i = bms_next_member(coal_em->em_relids, i)) >= 0) + { + RelOptInfo *rel = root->simple_rel_array[i]; + + rel->eclass_member_indexes = bms_del_member(rel->eclass_member_indexes, + coal_idx); + } + return true; } @@ -2369,21 +2562,28 @@ bool exprs_known_equal(PlannerInfo *root, Node *item1, Node *item2) { ListCell *lc1; + Relids item1_relids = pull_varnos(root, item1); + Relids item2_relids = pull_varnos(root, item2); foreach(lc1, root->eq_classes) { EquivalenceClass *ec = (EquivalenceClass *) lfirst(lc1); bool item1member = false; bool item2member = false; - ListCell *lc2; + Bitmapset *matching_ems; + int i; /* Never match to a volatile EC */ if (ec->ec_has_volatile) continue; - foreach(lc2, ec->ec_members) + matching_ems = bms_join(get_ecmember_indexes_strict(root, ec, item1_relids, false, true), + get_ecmember_indexes_strict(root, ec, item2_relids, false, true)); + i = -1; + while ((i = bms_next_member(matching_ems, i)) >= 0) { - EquivalenceMember *em = (EquivalenceMember *) lfirst(lc2); + EquivalenceMember *em = list_nth_node(EquivalenceMember, + root->eq_members, i); if (em->em_is_child) continue; /* ignore children here */ @@ -2446,16 +2646,21 @@ match_eclasses_to_foreign_key_col(PlannerInfo *root, i); EquivalenceMember *item1_em = NULL; EquivalenceMember *item2_em = NULL; - ListCell *lc2; + Bitmapset *matching_ems; + int j; /* Never match to a volatile EC */ if (ec->ec_has_volatile) continue; /* Note: it seems okay to match to "broken" eclasses here */ + matching_ems = bms_join(get_ecmember_indexes_strict(root, ec, rel1->relids, false, false), + get_ecmember_indexes_strict(root, ec, rel2->relids, false, false)); - foreach(lc2, ec->ec_members) + j = -1; + while ((j = bms_next_member(matching_ems, j)) >= 0) { - EquivalenceMember *em = (EquivalenceMember *) lfirst(lc2); + EquivalenceMember *em = list_nth_node(EquivalenceMember, + root->eq_members, j); Var *var; if (em->em_is_child) @@ -2508,16 +2713,19 @@ match_eclasses_to_foreign_key_col(PlannerInfo *root, * Returns NULL if no such clause can be found. */ RestrictInfo * -find_derived_clause_for_ec_member(EquivalenceClass *ec, +find_derived_clause_for_ec_member(PlannerInfo *root, EquivalenceClass *ec, EquivalenceMember *em) { - ListCell *lc; + int i; Assert(ec->ec_has_const); Assert(!em->em_is_const); - foreach(lc, ec->ec_derives) + + i = -1; + while ((i = bms_next_member(ec->ec_derive_indexes, i)) >= 0) { - RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); + RestrictInfo *rinfo = list_nth_node(RestrictInfo, root->eq_derives, + i); /* * generate_base_implied_equalities_const will have put non-const @@ -2568,7 +2776,8 @@ add_child_rel_equivalences(PlannerInfo *root, while ((i = bms_next_member(parent_rel->eclass_indexes, i)) >= 0) { EquivalenceClass *cur_ec = (EquivalenceClass *) list_nth(root->eq_classes, i); - int num_members; + EquivalenceMemberIterator iter; + EquivalenceMember *cur_em; /* * If this EC contains a volatile expression, then generating child @@ -2582,86 +2791,72 @@ add_child_rel_equivalences(PlannerInfo *root, Assert(bms_is_subset(top_parent_relids, cur_ec->ec_relids)); /* - * We don't use foreach() here because there's no point in scanning - * newly-added child members, so we can stop after the last - * pre-existing EC member. + * Looping using the EquivalenceMemberIterator means we only loop over + * the members which existed when we set up the iterator, not newly + * added ones. */ - num_members = list_length(cur_ec->ec_members); - for (int pos = 0; pos < num_members; pos++) + setup_eclass_member_iterator(&iter, root, cur_ec, top_parent_relids, + false, false); + + while ((cur_em = eclass_member_iterator_next(&iter)) != NULL) { - EquivalenceMember *cur_em = (EquivalenceMember *) list_nth(cur_ec->ec_members, pos); + Expr *child_expr; + Relids new_relids; + Relids new_nullable_relids; - if (cur_em->em_is_const) - continue; /* ignore consts here */ + Assert(!cur_em->em_is_const); + Assert(!cur_em->em_is_child); + Assert(bms_overlap(cur_em->em_relids, top_parent_relids)); + + if (parent_rel->reloptkind == RELOPT_BASEREL) + { + /* Simple single-level transformation */ + child_expr = (Expr *) + adjust_appendrel_attrs(root, + (Node *) cur_em->em_expr, + 1, &appinfo); + } + else + { + /* Must do multi-level transformation */ + child_expr = (Expr *) + adjust_appendrel_attrs_multilevel(root, + (Node *) cur_em->em_expr, + child_rel, + child_rel->top_parent); + } /* - * We consider only original EC members here, not - * already-transformed child members. Otherwise, if some original - * member expression references more than one appendrel, we'd get - * an O(N^2) explosion of useless derived expressions for - * combinations of children. (But add_child_join_rel_equivalences - * may add targeted combinations for partitionwise-join purposes.) + * Transform em_relids to match. Note we do *not* do + * pull_varnos(child_expr) here, as for example the transformation + * might have substituted a constant, but we don't want the child + * member to be marked as constant. */ - if (cur_em->em_is_child) - continue; /* ignore children here */ + new_relids = bms_difference(cur_em->em_relids, + top_parent_relids); + new_relids = bms_add_members(new_relids, child_relids); - /* Does this member reference child's topmost parent rel? */ - if (bms_overlap(cur_em->em_relids, top_parent_relids)) + /* + * And likewise for nullable_relids. Note this code assumes + * parent and child relids are singletons. + */ + new_nullable_relids = cur_em->em_nullable_relids; + if (bms_overlap(new_nullable_relids, top_parent_relids)) { - /* Yes, generate transformed child version */ - Expr *child_expr; - Relids new_relids; - Relids new_nullable_relids; - - if (parent_rel->reloptkind == RELOPT_BASEREL) - { - /* Simple single-level transformation */ - child_expr = (Expr *) - adjust_appendrel_attrs(root, - (Node *) cur_em->em_expr, - 1, &appinfo); - } - else - { - /* Must do multi-level transformation */ - child_expr = (Expr *) - adjust_appendrel_attrs_multilevel(root, - (Node *) cur_em->em_expr, - child_rel, - child_rel->top_parent); - } - - /* - * Transform em_relids to match. Note we do *not* do - * pull_varnos(child_expr) here, as for example the - * transformation might have substituted a constant, but we - * don't want the child member to be marked as constant. - */ - new_relids = bms_difference(cur_em->em_relids, - top_parent_relids); - new_relids = bms_add_members(new_relids, child_relids); - - /* - * And likewise for nullable_relids. Note this code assumes - * parent and child relids are singletons. - */ - new_nullable_relids = cur_em->em_nullable_relids; - if (bms_overlap(new_nullable_relids, top_parent_relids)) - { - new_nullable_relids = bms_difference(new_nullable_relids, - top_parent_relids); - new_nullable_relids = bms_add_members(new_nullable_relids, - child_relids); - } + new_nullable_relids = bms_difference(new_nullable_relids, + top_parent_relids); + new_nullable_relids = bms_add_members(new_nullable_relids, + child_relids); + } - (void) add_eq_member(cur_ec, child_expr, - new_relids, new_nullable_relids, - true, cur_em->em_datatype); + (void) add_eq_member(root, cur_ec, child_expr, + new_relids, new_nullable_relids, + true, cur_em->em_datatype); - /* Record this EC index for the child rel */ - child_rel->eclass_indexes = bms_add_member(child_rel->eclass_indexes, i); - } + /* Record this EC index for the child rel */ + child_rel->eclass_indexes = bms_add_member(child_rel->eclass_indexes, i); } + eclass_member_iterator_dispose(&iter); } } @@ -2706,7 +2901,8 @@ add_child_join_rel_equivalences(PlannerInfo *root, while ((i = bms_next_member(matching_ecs, i)) >= 0) { EquivalenceClass *cur_ec = (EquivalenceClass *) list_nth(root->eq_classes, i); - int num_members; + EquivalenceMemberIterator iter; + EquivalenceMember *cur_em; /* * If this EC contains a volatile expression, then generating child @@ -2720,24 +2916,22 @@ add_child_join_rel_equivalences(PlannerInfo *root, Assert(bms_overlap(top_parent_relids, cur_ec->ec_relids)); /* - * We don't use foreach() here because there's no point in scanning - * newly-added child members, so we can stop after the last - * pre-existing EC member. + * Looping using the EquivalenceMemberIterator means we only loop over + * the members which existed when we set up the iterator, not newly + * added ones. */ - num_members = list_length(cur_ec->ec_members); - for (int pos = 0; pos < num_members; pos++) - { - EquivalenceMember *cur_em = (EquivalenceMember *) list_nth(cur_ec->ec_members, pos); + setup_eclass_member_iterator(&iter, root, cur_ec, top_parent_relids, + false, false); - if (cur_em->em_is_const) - continue; /* ignore consts here */ + while ((cur_em = eclass_member_iterator_next(&iter)) != NULL) + { + Expr *child_expr; + Relids new_relids; + Relids new_nullable_relids; - /* - * We consider only original EC members here, not - * already-transformed child members. - */ - if (cur_em->em_is_child) - continue; /* ignore children here */ + Assert(!cur_em->em_is_const); + Assert(!cur_em->em_is_child); + Assert(bms_overlap(cur_em->em_relids, top_parent_relids)); /* * We may ignore expressions that reference a single baserel, @@ -2746,60 +2940,52 @@ add_child_join_rel_equivalences(PlannerInfo *root, if (bms_membership(cur_em->em_relids) != BMS_MULTIPLE) continue; - /* Does this member reference child's topmost parent rel? */ - if (bms_overlap(cur_em->em_relids, top_parent_relids)) + if (parent_joinrel->reloptkind == RELOPT_JOINREL) { - /* Yes, generate transformed child version */ - Expr *child_expr; - Relids new_relids; - Relids new_nullable_relids; - - if (parent_joinrel->reloptkind == RELOPT_JOINREL) - { - /* Simple single-level transformation */ - child_expr = (Expr *) - adjust_appendrel_attrs(root, - (Node *) cur_em->em_expr, - nappinfos, appinfos); - } - else - { - /* Must do multi-level transformation */ - Assert(parent_joinrel->reloptkind == RELOPT_OTHER_JOINREL); - child_expr = (Expr *) - adjust_appendrel_attrs_multilevel(root, - (Node *) cur_em->em_expr, - child_joinrel, - child_joinrel->top_parent); - } + /* Simple single-level transformation */ + child_expr = (Expr *) + adjust_appendrel_attrs(root, + (Node *) cur_em->em_expr, + nappinfos, appinfos); + } + else + { + /* Must do multi-level transformation */ + Assert(parent_joinrel->reloptkind == RELOPT_OTHER_JOINREL); + child_expr = (Expr *) + adjust_appendrel_attrs_multilevel(root, + (Node *) cur_em->em_expr, + child_joinrel, + child_joinrel->top_parent); + } - /* - * Transform em_relids to match. Note we do *not* do - * pull_varnos(child_expr) here, as for example the - * transformation might have substituted a constant, but we - * don't want the child member to be marked as constant. - */ - new_relids = bms_difference(cur_em->em_relids, - top_parent_relids); - new_relids = bms_add_members(new_relids, child_relids); + /* + * Transform em_relids to match. Note we do *not* do + * pull_varnos(child_expr) here, as for example the transformation + * might have substituted a constant, but we don't want the child + * member to be marked as constant. + */ + new_relids = bms_difference(cur_em->em_relids, + top_parent_relids); + new_relids = bms_add_members(new_relids, child_relids); - /* - * For nullable_relids, we must selectively replace parent - * nullable relids with child ones. - */ - new_nullable_relids = cur_em->em_nullable_relids; - if (bms_overlap(new_nullable_relids, top_parent_relids)) - new_nullable_relids = - adjust_child_relids_multilevel(root, - new_nullable_relids, - child_joinrel, - child_joinrel->top_parent); - - (void) add_eq_member(cur_ec, child_expr, - new_relids, new_nullable_relids, - true, cur_em->em_datatype); - } + /* + * For nullable_relids, we must selectively replace parent + * nullable relids with child ones. + */ + new_nullable_relids = cur_em->em_nullable_relids; + if (bms_overlap(new_nullable_relids, top_parent_relids)) + new_nullable_relids = + adjust_child_relids_multilevel(root, + new_nullable_relids, + child_joinrel, + child_joinrel->top_parent); + + (void) add_eq_member(root, cur_ec, child_expr, + new_relids, new_nullable_relids, + true, cur_em->em_datatype); } + eclass_member_iterator_dispose(&iter); } MemoryContextSwitchTo(oldcontext); @@ -2858,7 +3044,8 @@ generate_implied_equalities_for_column(PlannerInfo *root, { EquivalenceClass *cur_ec = (EquivalenceClass *) list_nth(root->eq_classes, i); EquivalenceMember *cur_em; - ListCell *lc2; + EquivalenceMemberIterator iter; + int j; /* Sanity check eclass_indexes only contain ECs for rel */ Assert(is_child_rel || bms_is_subset(rel->relids, cur_ec->ec_relids)); @@ -2880,16 +3067,19 @@ generate_implied_equalities_for_column(PlannerInfo *root, * corner cases, so for now we live with just reporting the first * match. See also get_eclass_for_sort_expr.) */ - cur_em = NULL; - foreach(lc2, cur_ec->ec_members) + setup_eclass_member_iterator(&iter, root, cur_ec, rel->relids, true, + false); + + while ((cur_em = eclass_member_iterator_next(&iter)) != NULL) { - cur_em = (EquivalenceMember *) lfirst(lc2); if (bms_equal(cur_em->em_relids, rel->relids) && callback(root, rel, cur_ec, cur_em, callback_arg)) break; cur_em = NULL; } + eclass_member_iterator_dispose(&iter); + if (!cur_em) continue; @@ -2897,14 +3087,15 @@ generate_implied_equalities_for_column(PlannerInfo *root, * Found our match. Scan the other EC members and attempt to generate * joinclauses. */ - foreach(lc2, cur_ec->ec_members) + j = -1; + while ((j = bms_next_member(cur_ec->ec_nonchild_indexes, j)) >= 0) { - EquivalenceMember *other_em = (EquivalenceMember *) lfirst(lc2); + EquivalenceMember *other_em = list_nth_node(EquivalenceMember, + root->eq_members, j); Oid eq_op; RestrictInfo *rinfo; - if (other_em->em_is_child) - continue; /* ignore children here */ + Assert(!other_em->em_is_child); /* Make sure it'll be a join to a different rel */ if (other_em == cur_em || @@ -3001,7 +3192,7 @@ have_relevant_eclass_joinclause(PlannerInfo *root, * surely be true if both of them overlap ec_relids.) * * Note we don't test ec_broken; if we did, we'd need a separate code - * path to look through ec_sources. Checking the membership anyway is + * path to look through eq_sources. Checking the membership anyway is * OK as a possibly-overoptimistic heuristic. * * We don't test ec_has_const either, even though a const eclass won't @@ -3076,7 +3267,7 @@ eclass_useful_for_merging(PlannerInfo *root, RelOptInfo *rel) { Relids relids; - ListCell *lc; + int i; Assert(!eclass->ec_merged); @@ -3089,7 +3280,7 @@ eclass_useful_for_merging(PlannerInfo *root, /* * Note we don't test ec_broken; if we did, we'd need a separate code path - * to look through ec_sources. Checking the members anyway is OK as a + * to look through eq_sources. Checking the members anyway is OK as a * possibly-overoptimistic heuristic. */ @@ -3107,12 +3298,14 @@ eclass_useful_for_merging(PlannerInfo *root, return false; /* To join, we need a member not in the given rel */ - foreach(lc, eclass->ec_members) + i = -1; + while ((i = bms_next_member(eclass->ec_nonchild_indexes, i)) >= 0) { - EquivalenceMember *cur_em = (EquivalenceMember *) lfirst(lc); + EquivalenceMember *cur_em = + list_nth_node(EquivalenceMember, root->eq_members, i); - if (cur_em->em_is_child) - continue; /* ignore children here */ + /* we don't expect child members here */ + Assert(!cur_em->em_is_child); if (!bms_overlap(cur_em->em_relids, relids)) return true; @@ -3200,7 +3393,7 @@ get_eclass_indexes_for_relids(PlannerInfo *root, Relids relids) /* Should be OK to rely on eclass_indexes */ Assert(root->ec_merging_done); - while ((i = bms_next_member(relids, i)) > 0) + while ((i = bms_next_member(relids, i)) >= 0) { RelOptInfo *rel = root->simple_rel_array[i]; @@ -3236,3 +3429,595 @@ get_common_eclass_indexes(PlannerInfo *root, Relids relids1, Relids relids2) /* Calculate and return the common EC indexes, recycling the left input. */ return bms_int_members(rel1ecs, rel2ecs); } + +/* + * get_ecmember_indexes + * Returns a Bitmapset with indexes into root->eq_members for all + * EquivalenceMembers in 'ec' that have + * bms_overlap(em->em_relids, relids) as true. The returned indexes + * may reference EquivalenceMembers with em_relids containing members + * not mentioned in 'relids'. + * + * Returns a newly allocated Bitmapset which the caller is free to modify. + */ +Bitmapset * +get_ecmember_indexes(PlannerInfo *root, EquivalenceClass *ec, Relids relids, + bool with_children, bool with_norel_members) +{ + Bitmapset *matching_ems; + Bitmapset *rel_ems = NULL; + int i = -1; + + while ((i = bms_next_member(relids, i)) >= 0) + { + RelOptInfo *rel = root->simple_rel_array[i]; + + rel_ems = bms_add_members(rel_ems, rel->eclass_member_indexes); + } + +#ifdef USE_ASSERT_CHECKING + /* verify the results look sane */ + i = -1; + while ((i = bms_next_member(rel_ems, i)) >= 0) + { + EquivalenceMember *em = list_nth_node(EquivalenceMember, + root->eq_members, i); + + Assert(bms_overlap(em->em_relids, relids)); + } +#endif + + /* remove EC members not mentioning the required rels. */ + if (!with_children) + matching_ems = bms_int_members(rel_ems, ec->ec_nonchild_indexes); + else + matching_ems = bms_int_members(rel_ems, ec->ec_member_indexes); + + /* now add any members with that are not mentioned by any relation */ + if (with_norel_members) + matching_ems = bms_add_members(matching_ems, ec->ec_norel_indexes); + + return matching_ems; +} + +/* + * get_ecmember_indexes_strict + * Returns a Bitmapset with indexes into root->eq_members for all + * EquivalenceMembers in 'ec' that have + * bms_is_subset(relids, em->em_relids) as true. + * + * Returns a newly allocated Bitmapset which the caller is free to modify. + */ +Bitmapset * +get_ecmember_indexes_strict(PlannerInfo *root, EquivalenceClass *ec, + Relids relids, bool with_children, + bool with_norel_members) +{ + Bitmapset *matching_ems = NULL; + int i = bms_next_member(relids, -1); + + if (i >= 0) + { + RelOptInfo *rel = root->simple_rel_array[i]; + + /* + * bms_intersect to the first relation to try to keep the resulting + * Bitmapset as small as possible. This saves having to make a + * complete bms_copy() of one of them. One may contain significantly + * more words than the other. + */ + if (!with_children) + matching_ems = bms_intersect(rel->eclass_member_indexes, + ec->ec_nonchild_indexes); + else + matching_ems = bms_intersect(rel->eclass_member_indexes, + ec->ec_member_indexes); + + while ((i = bms_next_member(relids, i)) >= 0) + { + rel = root->simple_rel_array[i]; + matching_ems = bms_int_members(matching_ems, + rel->eclass_member_indexes); + } + } + +#ifdef USE_ASSERT_CHECKING + /* verify the results look sane */ + i = -1; + while ((i = bms_next_member(matching_ems, i)) >= 0) + { + EquivalenceMember *em = list_nth_node(EquivalenceMember, + root->eq_members, i); + + Assert(bms_is_subset(relids, em->em_relids)); + } +#endif + + /* optionally add members with that are not mentioned by any relation */ + if (with_norel_members) + matching_ems = bms_add_members(matching_ems, ec->ec_norel_indexes); + + return matching_ems; +} + +/* + * The threshold for the number of members that must be in an EquivalenceClass + * before we switch to searching for EquivalenceMember by using the Bitmapset + * indexes stored in EquivalenceClass and RelOptInfo. We don't want to make + * this too low as the manipulation of Bitmapsets slows this down for + * EquivalenceClasses with just a few members. The linear search becomes very + * slow when an EquivalenceClass has a large number of members, as can happen + * when planning queries to partitioned tables. + */ +#define ECMEMBER_INDEX_THRESHOLD 16 + +/* + * setup_eclass_member_iterator + * Setup 'iter' so it's ready for eclass_member_iterator_next to start + * searching for EquivalenceMembers matching the specified parameters. + * + * Once used, the iterator must be disposed of using + * eclass_member_iterator_dispose. + */ +void +setup_eclass_member_iterator(EquivalenceMemberIterator *iter, + PlannerInfo *root, EquivalenceClass *ec, + Relids relids, bool with_children, + bool with_norel_members) +{ + iter->orig_length = list_length(ec->ec_members); + iter->use_index = iter->orig_length >= ECMEMBER_INDEX_THRESHOLD; + iter->with_children = with_children; + iter->with_norel_members = with_norel_members; + iter->relids_empty = bms_is_empty(relids); +#ifdef USE_ASSERT_CHECKING + iter->isstrict = false; +#endif + iter->with_relids = relids; + iter->root = root; + iter->eclass = ec; + +#ifdef USE_ASSERT_CHECKING + if (1) +#else + if (iter->use_index) +#endif + iter->matching_ems = get_ecmember_indexes(root, ec, relids, + with_children, + with_norel_members); + else + iter->matching_ems = NULL; + + iter->current_index = -1; + +#ifdef USE_ASSERT_CHECKING + + /* + * Verify that an iterator that uses the index and one that does not both + * return the same EquivalenceMembers + */ + { + EquivalenceMemberIterator idx_iter; + EquivalenceMemberIterator noidx_iter; + EquivalenceMember *em; + List *list1 = NIL; + List *list2 = NIL; + ListCell *lc1, + *lc2; + + memcpy(&idx_iter, iter, sizeof(EquivalenceMemberIterator)); + memcpy(&noidx_iter, iter, sizeof(EquivalenceMemberIterator)); + + idx_iter.use_index = true; + noidx_iter.use_index = false; + + while ((em = eclass_member_iterator_next(&idx_iter)) != NULL) + list1 = lappend(list1, em); + + while ((em = eclass_member_iterator_next(&noidx_iter)) != NULL) + list2 = lappend(list2, em); + + list_sort(list1, list_ptr_cmp); + list_sort(list2, list_ptr_cmp); + + Assert(list_length(list1) == list_length(list2)); + + forboth(lc1, list1, lc2, list2) + Assert(lfirst(lc1) == lfirst(lc2)); + } +#endif + +} + +/* + * setup_eclass_member_strict_iterator + * Setup 'iter' so it's ready for eclass_member_iterator_strict_next to + * start searching for EquivalenceMembers matching the specified + * parameters. + * + * Once used, the iterator must be disposed of using + * eclass_member_iterator_dispose. + */ +void +setup_eclass_member_strict_iterator(EquivalenceMemberIterator *iter, + PlannerInfo *root, EquivalenceClass *ec, + Relids relids, bool with_children, + bool with_norel_members) +{ + iter->orig_length = list_length(ec->ec_members); + iter->use_index = iter->orig_length >= ECMEMBER_INDEX_THRESHOLD; + iter->with_children = with_children; + iter->with_norel_members = with_norel_members; + iter->relids_empty = bms_is_empty(relids); +#ifdef USE_ASSERT_CHECKING + iter->isstrict = true; +#endif + iter->with_relids = relids; + iter->root = root; + iter->eclass = ec; + +#ifdef USE_ASSERT_CHECKING + if (1) +#else + if (iter->use_index) +#endif + iter->matching_ems = get_ecmember_indexes_strict(root, ec, relids, + with_children, + with_norel_members); + else + iter->matching_ems = NULL; + + iter->current_index = -1; + +#ifdef USE_ASSERT_CHECKING + + /* + * Verify that an iterator that uses the index and one that does not both + * return the same EquivalenceMembers + */ + { + EquivalenceMemberIterator idx_iter; + EquivalenceMemberIterator noidx_iter; + EquivalenceMember *em; + List *list1 = NIL; + List *list2 = NIL; + ListCell *lc1, + *lc2; + + memcpy(&idx_iter, iter, sizeof(EquivalenceMemberIterator)); + memcpy(&noidx_iter, iter, sizeof(EquivalenceMemberIterator)); + + idx_iter.use_index = true; + noidx_iter.use_index = false; + + while ((em = eclass_member_iterator_strict_next(&idx_iter)) != NULL) + list1 = lappend(list1, em); + + while ((em = eclass_member_iterator_strict_next(&noidx_iter)) != NULL) + list2 = lappend(list2, em); + + list_sort(list1, list_ptr_cmp); + list_sort(list2, list_ptr_cmp); + + Assert(list_length(list1) == list_length(list2)); + + forboth(lc1, list1, lc2, list2) + Assert(lfirst(lc1) == lfirst(lc2)); + } +#endif +} + +/* + * eclass_member_iterator_next + * Fetch the next EquivalenceMember from an EquivalenceMemberIterator + * which was set up by setup_eclass_member_iterator(). Returns NULL when + * there are no more matching EquivalenceMembers. + */ +EquivalenceMember * +eclass_member_iterator_next(EquivalenceMemberIterator *iter) +{ + /* Fail if this was used instead of eclass_member_iterator_strict_next */ + Assert(!iter->isstrict); + + if (iter->use_index) + { + iter->current_index = bms_next_member(iter->matching_ems, + iter->current_index); + if (iter->current_index >= 0) + return list_nth_node(EquivalenceMember, iter->root->eq_members, + iter->current_index); + return NULL; + } + else + { + ListCell *lc; + + for_each_from(lc, iter->eclass->ec_members, iter->current_index + 1) + { + EquivalenceMember *em = lfirst_node(EquivalenceMember, lc); + + iter->current_index = foreach_current_index(lc); + + /* + * Some users of this iterator will be adding new + * EquivalenceMember during the loop. We must ensure we don't + * return those, so here we return NULL when the loop index goes + * beyond the original length of the ec_members list. + */ + if (iter->current_index >= iter->orig_length) + return NULL; + + /* don't return child members when with_children==false */ + if (!iter->with_children && em->em_is_child) + continue; + + /* + * When with_norel_members==true, make sure we return all members + * without Vars. + */ + if (iter->with_norel_members && em->em_norel_expr) + return em; + + /* + * Don't return members which have no common rels with with_relids + */ + if (!bms_overlap(em->em_relids, iter->with_relids)) + continue; + + return em; + } + return NULL; + } +} + +/* + * eclass_member_iterator_strict_next + * Fetch the next EquivalenceMember from an EquivalenceMemberIterator + * which was set up by setup_eclass_member_strict_iterator(). Returns + * NULL when there are no more matching EquivalenceMembers. + */ +EquivalenceMember * +eclass_member_iterator_strict_next(EquivalenceMemberIterator *iter) +{ + /* Fail if this was used instead of eclass_member_iterator_next */ + Assert(iter->isstrict); + + if (iter->use_index) + { + iter->current_index = bms_next_member(iter->matching_ems, + iter->current_index); + if (iter->current_index >= 0) + return list_nth_node(EquivalenceMember, iter->root->eq_members, + iter->current_index); + return NULL; + } + else + { + ListCell *lc; + + for_each_from(lc, iter->eclass->ec_members, iter->current_index + 1) + { + EquivalenceMember *em = lfirst_node(EquivalenceMember, lc); + + iter->current_index = foreach_current_index(lc); + + /* + * Some users of this iterator will be adding new + * EquivalenceMember during the loop. We must ensure we don't + * return those, so here we return NULL when the loop index goes + * beyond the original length of the ec_members list. + */ + if (iter->current_index >= iter->orig_length) + return NULL; + + /* don't return child members when with_children==false */ + if (!iter->with_children && em->em_is_child) + continue; + + /* + * When with_norel_members==true, make sure we return all members + * without Vars. + */ + if (iter->with_norel_members && em->em_norel_expr) + return em; + + /* + * Don't match members where em_relids that don't contain all rels + * mentioned in with_relids. + */ + if (iter->relids_empty || + !bms_is_subset(iter->with_relids, em->em_relids)) + continue; + + return em; + } + return NULL; + } +} + +/* + * eclass_member_iterator_dispose + * Free any memory allocated by the iterator + */ +void +eclass_member_iterator_dispose(EquivalenceMemberIterator *iter) +{ + bms_free(iter->matching_ems); +} + + +/* + * get_ec_source_indexes + * Returns a Bitmapset with indexes into root->eq_sources for all + * RestrictInfos in 'ec' that have + * bms_overlap(relids, rinfo->clause_relids) as true. + * + * Returns a newly allocated Bitmapset which the caller is free to modify. + */ +Bitmapset * +get_ec_source_indexes(PlannerInfo *root, EquivalenceClass *ec, Relids relids) +{ + Bitmapset *rel_esis = NULL; + int i = -1; + + while ((i = bms_next_member(relids, i)) >= 0) + { + RelOptInfo *rel = root->simple_rel_array[i]; + + rel_esis = bms_add_members(rel_esis, rel->eclass_source_indexes); + } + +#ifdef USE_ASSERT_CHECKING + /* verify the results look sane */ + i = -1; + while ((i = bms_next_member(rel_esis, i)) >= 0) + { + RestrictInfo *rinfo = list_nth_node(RestrictInfo, root->eq_sources, + i); + + Assert(bms_overlap(relids, rinfo->clause_relids)); + } +#endif + + /* bitwise-AND to leave only the ones for this EquivalenceClass */ + return bms_int_members(rel_esis, ec->ec_source_indexes); +} + +/* + * get_ec_source_indexes_strict + * Returns a Bitmapset with indexes into root->eq_sources for all + * RestrictInfos in 'ec' that have + * bms_is_subset(relids, rinfo->clause_relids) as true. + * + * Returns a newly allocated Bitmapset which the caller is free to modify. + */ +Bitmapset * +get_ec_source_indexes_strict(PlannerInfo *root, EquivalenceClass *ec, + Relids relids) +{ + Bitmapset *esis = NULL; + int i = bms_next_member(relids, -1); + + if (i >= 0) + { + RelOptInfo *rel = root->simple_rel_array[i]; + + /* + * bms_intersect to the first relation to try to keep the resulting + * Bitmapset as small as possible. This saves having to make a + * complete bms_copy() of one of them. One may contain significantly + * more words than the other. + */ + esis = bms_intersect(ec->ec_source_indexes, + rel->eclass_source_indexes); + + while ((i = bms_next_member(relids, i)) >= 0) + { + rel = root->simple_rel_array[i]; + esis = bms_int_members(esis, rel->eclass_source_indexes); + } + } + +#ifdef USE_ASSERT_CHECKING + /* verify the results look sane */ + i = -1; + while ((i = bms_next_member(esis, i)) >= 0) + { + RestrictInfo *rinfo = list_nth_node(RestrictInfo, root->eq_sources, + i); + + Assert(bms_is_subset(relids, rinfo->clause_relids)); + } +#endif + + return esis; +} + +/* + * get_ec_derive_indexes + * Returns a Bitmapset with indexes into root->eq_derives for all + * RestrictInfos in 'ec' that have + * bms_overlap(relids, rinfo->clause_relids) as true. + * + * Returns a newly allocated Bitmapset which the caller is free to modify. + * + * XXX is this function even needed? + */ +Bitmapset * +get_ec_derive_indexes(PlannerInfo *root, EquivalenceClass *ec, Relids relids) +{ + Bitmapset *rel_edis = NULL; + int i = -1; + + while ((i = bms_next_member(relids, i)) >= 0) + { + RelOptInfo *rel = root->simple_rel_array[i]; + + rel_edis = bms_add_members(rel_edis, rel->eclass_derive_indexes); + } + +#ifdef USE_ASSERT_CHECKING + /* verify the results look sane */ + i = -1; + while ((i = bms_next_member(rel_edis, i)) >= 0) + { + RestrictInfo *rinfo = list_nth_node(RestrictInfo, root->eq_derives, + i); + + Assert(bms_overlap(relids, rinfo->clause_relids)); + } +#endif + + /* bitwise-AND to leave only the ones for this EquivalenceClass */ + return bms_int_members(rel_edis, ec->ec_derive_indexes); +} + +/* + * get_ec_derive_indexes_strict + * Returns a Bitmapset with indexes into root->eq_derives for all + * RestrictInfos in 'ec' that have + * bms_is_subset(relids, rinfo->clause_relids) as true. + * + * Returns a newly allocated Bitmapset which the caller is free to modify. + */ +Bitmapset * +get_ec_derive_indexes_strict(PlannerInfo *root, EquivalenceClass *ec, + Relids relids) +{ + Bitmapset *edis = NULL; + int i = bms_next_member(relids, -1); + + if (i >= 0) + { + RelOptInfo *rel = root->simple_rel_array[i]; + + /* + * bms_intersect to the first relation to try to keep the resulting + * Bitmapset as small as possible. This saves having to make a + * complete bms_copy() of one of them. One may contain significantly + * more words than the other. + */ + edis = bms_intersect(ec->ec_derive_indexes, + rel->eclass_derive_indexes); + + while ((i = bms_next_member(relids, i)) >= 0) + { + rel = root->simple_rel_array[i]; + edis = bms_int_members(edis, rel->eclass_derive_indexes); + } + } + +#ifdef USE_ASSERT_CHECKING + /* verify the results look sane */ + i = -1; + while ((i = bms_next_member(edis, i)) >= 0) + { + RestrictInfo *rinfo = list_nth_node(RestrictInfo, root->eq_derives, + i); + + Assert(bms_is_subset(relids, rinfo->clause_relids)); + } +#endif + + return edis; +} diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c index e13c8f1914..e1b736c0c0 100644 --- a/src/backend/optimizer/path/indxpath.c +++ b/src/backend/optimizer/path/indxpath.c @@ -184,8 +184,8 @@ static IndexClause *expand_indexqual_rowcompare(PlannerInfo *root, IndexOptInfo *index, Oid expr_op, bool var_on_left); -static void match_pathkeys_to_index(IndexOptInfo *index, List *pathkeys, - List **orderby_clauses_p, +static void match_pathkeys_to_index(PlannerInfo *root, IndexOptInfo *index, + List *pathkeys, List **orderby_clauses_p, List **clause_columns_p); static Expr *match_clause_to_ordering_op(IndexOptInfo *index, int indexcol, Expr *clause, Oid pk_opfamily); @@ -978,7 +978,7 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel, else if (index->amcanorderbyop && pathkeys_possibly_useful) { /* see if we can generate ordering operators for query_pathkeys */ - match_pathkeys_to_index(index, root->query_pathkeys, + match_pathkeys_to_index(root, index, root->query_pathkeys, &orderbyclauses, &orderbyclausecols); if (orderbyclauses) @@ -3070,8 +3070,8 @@ expand_indexqual_rowcompare(PlannerInfo *root, * one-to-one with the requested pathkeys. */ static void -match_pathkeys_to_index(IndexOptInfo *index, List *pathkeys, - List **orderby_clauses_p, +match_pathkeys_to_index(PlannerInfo *root, IndexOptInfo *index, + List *pathkeys, List **orderby_clauses_p, List **clause_columns_p) { List *orderby_clauses = NIL; @@ -3088,8 +3088,9 @@ match_pathkeys_to_index(IndexOptInfo *index, List *pathkeys, foreach(lc1, pathkeys) { PathKey *pathkey = (PathKey *) lfirst(lc1); + Bitmapset *matching_ems; bool found = false; - ListCell *lc2; + int i; /* * Note: for any failure to match, we just return NIL immediately. @@ -3113,9 +3114,14 @@ match_pathkeys_to_index(IndexOptInfo *index, List *pathkeys, * be considered to match more than one pathkey list, which is OK * here. See also get_eclass_for_sort_expr.) */ - foreach(lc2, pathkey->pk_eclass->ec_members) + matching_ems = bms_intersect(pathkey->pk_eclass->ec_member_indexes, + index->rel->eclass_member_indexes); + + i = -1; + while ((i = bms_next_member(matching_ems, i)) >= 0) { - EquivalenceMember *member = (EquivalenceMember *) lfirst(lc2); + EquivalenceMember *member = list_nth_node(EquivalenceMember, + root->eq_members, i); int indexcol; /* No possibility of match if it references other relations */ diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index d2e241c983..e715d5135b 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -955,18 +955,19 @@ convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel, * outer query. */ int best_score = -1; - ListCell *j; + int j = -1; - foreach(j, sub_eclass->ec_members) + while ((j = bms_next_member(sub_eclass->ec_nonchild_indexes, j)) >= 0) { - EquivalenceMember *sub_member = (EquivalenceMember *) lfirst(j); + EquivalenceMember *sub_member = list_nth_node(EquivalenceMember, + rel->subroot->eq_members, + j); Expr *sub_expr = sub_member->em_expr; Oid sub_expr_type = sub_member->em_datatype; Oid sub_expr_coll = sub_eclass->ec_collation; ListCell *k; - if (sub_member->em_is_child) - continue; /* ignore children here */ + Assert(!sub_member->em_is_child); foreach(k, subquery_tlist) { @@ -1476,7 +1477,7 @@ select_outer_pathkeys_for_merge(PlannerInfo *root, RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); EquivalenceClass *oeclass; int score; - ListCell *lc2; + int i; /* get the outer eclass */ update_mergeclause_eclasses(root, rinfo); @@ -1497,13 +1498,16 @@ select_outer_pathkeys_for_merge(PlannerInfo *root, /* compute score */ score = 0; - foreach(lc2, oeclass->ec_members) + i = -1; + while ((i = bms_next_member(oeclass->ec_nonchild_indexes, i)) >= 0) { - EquivalenceMember *em = (EquivalenceMember *) lfirst(lc2); + EquivalenceMember *em = list_nth_node(EquivalenceMember, + root->eq_members, i); - /* Potential future join partner? */ - if (!em->em_is_const && !em->em_is_child && - !bms_overlap(em->em_relids, joinrel->relids)) + /* shouldn't be consts or child members in ec_nonchild_indexes */ + Assert(!em->em_is_const && !em->em_is_child); + + if (!bms_overlap(em->em_relids, joinrel->relids)) score++; } diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c index cd68942af0..d519b97dbf 100644 --- a/src/backend/optimizer/plan/createplan.c +++ b/src/backend/optimizer/plan/createplan.c @@ -260,8 +260,8 @@ static IncrementalSort *make_incrementalsort(Plan *lefttree, int numCols, int nPresortedCols, AttrNumber *sortColIdx, Oid *sortOperators, Oid *collations, bool *nullsFirst); -static Plan *prepare_sort_from_pathkeys(Plan *lefttree, List *pathkeys, - Relids relids, +static Plan *prepare_sort_from_pathkeys(PlannerInfo *root, Plan *lefttree, + List *pathkeys, Relids relids, const AttrNumber *reqColIdx, bool adjust_tlist_in_place, int *p_numsortkeys, @@ -269,10 +269,13 @@ static Plan *prepare_sort_from_pathkeys(Plan *lefttree, List *pathkeys, Oid **p_sortOperators, Oid **p_collations, bool **p_nullsFirst); -static Sort *make_sort_from_pathkeys(Plan *lefttree, List *pathkeys, - Relids relids); -static IncrementalSort *make_incrementalsort_from_pathkeys(Plan *lefttree, - List *pathkeys, Relids relids, int nPresortedCols); +static Sort *make_sort_from_pathkeys(PlannerInfo *root, Plan *lefttree, + List *pathkeys, Relids relids); +static IncrementalSort *make_incrementalsort_from_pathkeys(PlannerInfo *root, + Plan *lefttree, + List *pathkeys, + Relids relids, + int nPresortedCols); static Sort *make_sort_from_groupcols(List *groupcls, AttrNumber *grpColIdx, Plan *lefttree); @@ -293,7 +296,7 @@ static Group *make_group(List *tlist, List *qual, int numGroupCols, AttrNumber *grpColIdx, Oid *grpOperators, Oid *grpCollations, Plan *lefttree); static Unique *make_unique_from_sortclauses(Plan *lefttree, List *distinctList); -static Unique *make_unique_from_pathkeys(Plan *lefttree, +static Unique *make_unique_from_pathkeys(PlannerInfo *root, Plan *lefttree, List *pathkeys, int numCols); static Gather *make_gather(List *qptlist, List *qpqual, int nworkers, int rescan_param, bool single_copy, Plan *subplan); @@ -1260,7 +1263,8 @@ create_append_plan(PlannerInfo *root, AppendPath *best_path, int flags) * function result; it must be the same plan node. However, we then * need to detect whether any tlist entries were added. */ - (void) prepare_sort_from_pathkeys((Plan *) plan, pathkeys, + (void) prepare_sort_from_pathkeys(root, + (Plan *) plan, pathkeys, best_path->path.parent->relids, NULL, true, @@ -1304,7 +1308,7 @@ create_append_plan(PlannerInfo *root, AppendPath *best_path, int flags) * don't need an explicit sort, to make sure they are returning * the same sort key columns the Append expects. */ - subplan = prepare_sort_from_pathkeys(subplan, pathkeys, + subplan = prepare_sort_from_pathkeys(root, subplan, pathkeys, subpath->parent->relids, nodeSortColIdx, false, @@ -1445,7 +1449,7 @@ create_merge_append_plan(PlannerInfo *root, MergeAppendPath *best_path, * function result; it must be the same plan node. However, we then need * to detect whether any tlist entries were added. */ - (void) prepare_sort_from_pathkeys(plan, pathkeys, + (void) prepare_sort_from_pathkeys(root, plan, pathkeys, best_path->path.parent->relids, NULL, true, @@ -1476,7 +1480,7 @@ create_merge_append_plan(PlannerInfo *root, MergeAppendPath *best_path, subplan = create_plan_recurse(root, subpath, CP_EXACT_TLIST); /* Compute sort column info, and adjust subplan's tlist as needed */ - subplan = prepare_sort_from_pathkeys(subplan, pathkeys, + subplan = prepare_sort_from_pathkeys(root, subplan, pathkeys, subpath->parent->relids, node->sortColIdx, false, @@ -1966,7 +1970,7 @@ create_gather_merge_plan(PlannerInfo *root, GatherMergePath *best_path) Assert(pathkeys != NIL); /* Compute sort column info, and adjust subplan's tlist as needed */ - subplan = prepare_sort_from_pathkeys(subplan, pathkeys, + subplan = prepare_sort_from_pathkeys(root, subplan, pathkeys, best_path->subpath->parent->relids, gm_plan->sortColIdx, false, @@ -2185,7 +2189,7 @@ create_sort_plan(PlannerInfo *root, SortPath *best_path, int flags) * relids. Thus, if this sort path is based on a child relation, we must * pass its relids. */ - plan = make_sort_from_pathkeys(subplan, best_path->path.pathkeys, + plan = make_sort_from_pathkeys(root, subplan, best_path->path.pathkeys, IS_OTHER_REL(best_path->subpath->parent) ? best_path->path.parent->relids : NULL); @@ -2209,7 +2213,7 @@ create_incrementalsort_plan(PlannerInfo *root, IncrementalSortPath *best_path, /* See comments in create_sort_plan() above */ subplan = create_plan_recurse(root, best_path->spath.subpath, flags | CP_SMALL_TLIST); - plan = make_incrementalsort_from_pathkeys(subplan, + plan = make_incrementalsort_from_pathkeys(root, subplan, best_path->spath.path.pathkeys, IS_OTHER_REL(best_path->spath.subpath->parent) ? best_path->spath.path.parent->relids : NULL, @@ -2278,7 +2282,7 @@ create_upper_unique_plan(PlannerInfo *root, UpperUniquePath *best_path, int flag subplan = create_plan_recurse(root, best_path->subpath, flags | CP_LABEL_TLIST); - plan = make_unique_from_pathkeys(subplan, + plan = make_unique_from_pathkeys(root, subplan, best_path->path.pathkeys, best_path->numkeys); @@ -4483,7 +4487,7 @@ create_mergejoin_plan(PlannerInfo *root, if (best_path->outersortkeys) { Relids outer_relids = outer_path->parent->relids; - Sort *sort = make_sort_from_pathkeys(outer_plan, + Sort *sort = make_sort_from_pathkeys(root, outer_plan, best_path->outersortkeys, outer_relids); @@ -4497,7 +4501,7 @@ create_mergejoin_plan(PlannerInfo *root, if (best_path->innersortkeys) { Relids inner_relids = inner_path->parent->relids; - Sort *sort = make_sort_from_pathkeys(inner_plan, + Sort *sort = make_sort_from_pathkeys(root, inner_plan, best_path->innersortkeys, inner_relids); @@ -6117,7 +6121,8 @@ make_incrementalsort(Plan *lefttree, int numCols, int nPresortedCols, * or a Result stacked atop lefttree). */ static Plan * -prepare_sort_from_pathkeys(Plan *lefttree, List *pathkeys, +prepare_sort_from_pathkeys(PlannerInfo *root, + Plan *lefttree, List *pathkeys, Relids relids, const AttrNumber *reqColIdx, bool adjust_tlist_in_place, @@ -6184,7 +6189,7 @@ prepare_sort_from_pathkeys(Plan *lefttree, List *pathkeys, tle = get_tle_by_resno(tlist, reqColIdx[numsortkeys]); if (tle) { - em = find_ec_member_matching_expr(ec, tle->expr, relids); + em = find_ec_member_matching_expr(root, ec, tle->expr, relids); if (em) { /* found expr at right place in tlist */ @@ -6215,7 +6220,7 @@ prepare_sort_from_pathkeys(Plan *lefttree, List *pathkeys, foreach(j, tlist) { tle = (TargetEntry *) lfirst(j); - em = find_ec_member_matching_expr(ec, tle->expr, relids); + em = find_ec_member_matching_expr(root, ec, tle->expr, relids); if (em) { /* found expr already in tlist */ @@ -6231,7 +6236,7 @@ prepare_sort_from_pathkeys(Plan *lefttree, List *pathkeys, /* * No matching tlist item; look for a computable expression. */ - em = find_computable_ec_member(NULL, ec, tlist, relids, false); + em = find_computable_ec_member(root, ec, tlist, relids, false); if (!em) elog(ERROR, "could not find pathkey item to sort"); pk_datatype = em->em_datatype; @@ -6302,7 +6307,8 @@ prepare_sort_from_pathkeys(Plan *lefttree, List *pathkeys, * 'relids' is the set of relations required by prepare_sort_from_pathkeys() */ static Sort * -make_sort_from_pathkeys(Plan *lefttree, List *pathkeys, Relids relids) +make_sort_from_pathkeys(PlannerInfo *root, Plan *lefttree, List *pathkeys, + Relids relids) { int numsortkeys; AttrNumber *sortColIdx; @@ -6311,7 +6317,9 @@ make_sort_from_pathkeys(Plan *lefttree, List *pathkeys, Relids relids) bool *nullsFirst; /* Compute sort column info, and adjust lefttree as needed */ - lefttree = prepare_sort_from_pathkeys(lefttree, pathkeys, + lefttree = prepare_sort_from_pathkeys(root, + lefttree, + pathkeys, relids, NULL, false, @@ -6337,7 +6345,8 @@ make_sort_from_pathkeys(Plan *lefttree, List *pathkeys, Relids relids) * 'nPresortedCols' is the number of presorted columns in input tuples */ static IncrementalSort * -make_incrementalsort_from_pathkeys(Plan *lefttree, List *pathkeys, +make_incrementalsort_from_pathkeys(PlannerInfo *root, + Plan *lefttree, List *pathkeys, Relids relids, int nPresortedCols) { int numsortkeys; @@ -6347,7 +6356,9 @@ make_incrementalsort_from_pathkeys(Plan *lefttree, List *pathkeys, bool *nullsFirst; /* Compute sort column info, and adjust lefttree as needed */ - lefttree = prepare_sort_from_pathkeys(lefttree, pathkeys, + lefttree = prepare_sort_from_pathkeys(root, + lefttree, + pathkeys, relids, NULL, false, @@ -6697,7 +6708,8 @@ make_unique_from_sortclauses(Plan *lefttree, List *distinctList) * as above, but use pathkeys to identify the sort columns and semantics */ static Unique * -make_unique_from_pathkeys(Plan *lefttree, List *pathkeys, int numCols) +make_unique_from_pathkeys(PlannerInfo *root, Plan *lefttree, List *pathkeys, + int numCols) { Unique *node = makeNode(Unique); Plan *plan = &node->plan; @@ -6760,7 +6772,7 @@ make_unique_from_pathkeys(Plan *lefttree, List *pathkeys, int numCols) foreach(j, plan->targetlist) { tle = (TargetEntry *) lfirst(j); - em = find_ec_member_matching_expr(ec, tle->expr, NULL); + em = find_ec_member_matching_expr(root, ec, tle->expr, NULL); if (em) { /* found expr already in tlist */ diff --git a/src/backend/optimizer/plan/planagg.c b/src/backend/optimizer/plan/planagg.c index d168620665..9f0fa42045 100644 --- a/src/backend/optimizer/plan/planagg.c +++ b/src/backend/optimizer/plan/planagg.c @@ -356,6 +356,7 @@ build_minmax_path(PlannerInfo *root, MinMaxAggInfo *mminfo, Assert(subroot->join_info_list == NIL); /* and we haven't made equivalence classes, either */ Assert(subroot->eq_classes == NIL); + Assert(root->eq_members == NIL); /* and we haven't created PlaceHolderInfos, either */ Assert(subroot->placeholder_list == NIL); diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index 05f44faf6e..f12f7fa83b 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -626,6 +626,9 @@ subquery_planner(PlannerGlobal *glob, Query *parse, root->cte_plan_ids = NIL; root->multiexpr_params = NIL; root->eq_classes = NIL; + root->eq_members = NIL; + root->eq_sources = NIL; + root->eq_derives = NIL; root->ec_merging_done = false; root->all_result_relids = parse->resultRelation ? bms_make_singleton(parse->resultRelation) : NULL; diff --git a/src/backend/optimizer/prep/prepjointree.c b/src/backend/optimizer/prep/prepjointree.c index 37a7af8c66..b7eaeb6163 100644 --- a/src/backend/optimizer/prep/prepjointree.c +++ b/src/backend/optimizer/prep/prepjointree.c @@ -1000,6 +1000,9 @@ pull_up_simple_subquery(PlannerInfo *root, Node *jtnode, RangeTblEntry *rte, subroot->cte_plan_ids = NIL; subroot->multiexpr_params = NIL; subroot->eq_classes = NIL; + subroot->eq_members = NIL; + subroot->eq_sources = NIL; + subroot->eq_derives = NIL; subroot->ec_merging_done = false; subroot->all_result_relids = NULL; subroot->leaf_result_relids = NULL; diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c index 0c68ec011b..4c7d66311f 100644 --- a/src/backend/optimizer/prep/prepunion.c +++ b/src/backend/optimizer/prep/prepunion.c @@ -45,7 +45,7 @@ #include "utils/selfuncs.h" #include "utils/syscache.h" - +static void setup_append_rel_entry(PlannerInfo *root); static RelOptInfo *recurse_set_operations(Node *setOp, PlannerInfo *root, List *colTypes, List *colCollations, bool junkOK, @@ -128,6 +128,7 @@ plan_set_operations(PlannerInfo *root) * so that pathkeys.c won't complain. */ Assert(root->eq_classes == NIL); + Assert(root->eq_members == NIL); root->ec_merging_done = true; /* @@ -149,6 +150,8 @@ plan_set_operations(PlannerInfo *root) leftmostQuery = leftmostRTE->subquery; Assert(leftmostQuery != NULL); + setup_append_rel_entry(root); + /* * If the topmost node is a recursive union, it needs special processing. */ @@ -180,6 +183,26 @@ plan_set_operations(PlannerInfo *root) return setop_rel; } +/* + * setup_append_rel_entry + * Add entry into root's simple_rel_array at element 0. This is required + * because generate_append_tlist() makes Vars with varno=0. In + * add_eq_member() we need to index EquivalenceMembers belonging to this + * relation. + */ +static void +setup_append_rel_entry(PlannerInfo *root) +{ + RelOptInfo *rel = makeNode(RelOptInfo); + + memset(rel, 0, sizeof(RelOptInfo)); + rel->eclass_member_indexes = NULL; + + /* make sure nothing else has made use of this element */ + Assert(root->simple_rel_array[0] == NULL); + root->simple_rel_array[0] = rel; +} + /* * recurse_set_operations * Recursively handle one step in a tree of set operations diff --git a/src/backend/optimizer/util/relnode.c b/src/backend/optimizer/util/relnode.c index 0a5632699d..ace943c87d 100644 --- a/src/backend/optimizer/util/relnode.c +++ b/src/backend/optimizer/util/relnode.c @@ -219,6 +219,9 @@ build_simple_rel(PlannerInfo *root, int relid, RelOptInfo *parent) rel->tuples = 0; rel->allvisfrac = 0; rel->eclass_indexes = NULL; + rel->eclass_member_indexes = NULL; + rel->eclass_source_indexes = NULL; + rel->eclass_derive_indexes = NULL; rel->subroot = NULL; rel->subplan_params = NIL; rel->rel_parallel_workers = -1; /* set up in get_relation_info */ @@ -648,6 +651,9 @@ build_join_rel(PlannerInfo *root, joinrel->tuples = 0; joinrel->allvisfrac = 0; joinrel->eclass_indexes = NULL; + joinrel->eclass_member_indexes = NULL; + joinrel->eclass_source_indexes = NULL; + joinrel->eclass_derive_indexes = NULL; joinrel->subroot = NULL; joinrel->subplan_params = NIL; joinrel->rel_parallel_workers = -1; @@ -834,6 +840,9 @@ build_child_join_rel(PlannerInfo *root, RelOptInfo *outer_rel, joinrel->tuples = 0; joinrel->allvisfrac = 0; joinrel->eclass_indexes = NULL; + joinrel->eclass_member_indexes = NULL; + joinrel->eclass_source_indexes = NULL; + joinrel->eclass_derive_indexes = NULL; joinrel->subroot = NULL; joinrel->subplan_params = NIL; joinrel->amflags = 0; diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index 2d1d8f4bcd..d888cd54d3 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -306,6 +306,15 @@ struct PlannerInfo /* list of active EquivalenceClasses */ List *eq_classes; + /* list of each EquivalenceMember */ + List *eq_members; + + /* list of source RestrictInfos used to build EquivalenceClasses */ + List *eq_sources; + + /* list of RestrictInfos derived from EquivalenceClasses */ + List *eq_derives; + /* set true once ECs are canonical */ bool ec_merging_done; @@ -926,6 +935,24 @@ typedef struct RelOptInfo * Indexes in PlannerInfo's eq_classes list of ECs that mention this rel */ Bitmapset *eclass_indexes; + + /* + * Indexes in PlannerInfo's eq_members list of EMs that mention this rel + */ + Bitmapset *eclass_member_indexes; + + /* + * Indexes in PlannerInfo's eq_sources list for RestrictInfos that mention + * this relation. + */ + Bitmapset *eclass_source_indexes; + + /* + * Indexes in PlannerInfo's eq_derives list for RestrictInfos that mention + * this relation. + */ + Bitmapset *eclass_derive_indexes; + PlannerInfo *subroot; /* if subquery */ List *subplan_params; /* if subquery */ /* wanted number of parallel workers */ @@ -1305,6 +1332,39 @@ typedef struct StatisticExtInfo * the included values might be all NULL rather than all the same non-null * values. See src/backend/optimizer/README for more on that point. * + * At various locations in the query planner, we must search for + * EquivalenceMembers within a given EquivalenceClass. For the common case, + * an EquivalenceClass does not have a large number of EquivalenceMembers, + * however, in cases such as planning queries to partitioned tables, the number + * of members can become large. To maintain planning performance, we make use + * of a bitmap index to allow us to quickly find EquivalenceMembers in a given + * EquivalenceClass belonging to a given relation or set of relations. This + * is done by storing a list of EquivalenceMembers belonging to all + * EquivalenceClasses in PlannerInfo and storing a Bitmapset for each + * RelOptInfo which has a bit set for each EquivalenceMember in that list + * which relates to the given relation. We also store a Bitmapset to mark all + * of the indexes in the PlannerInfo's list of EquivalenceMembers in the + * EquivalenceClass. We can quickly find the interesting indexes into the + * PlannerInfo's list by performing a bit-wise AND on the RelOptInfo's + * Bitmapset and the EquivalenceClasses. + * + * To further speed up the lookup of EquivalenceMembers we also record the + * non-child indexes. This allows us to deal with fewer bits for searches + * that don't require "is_child" EquivalenceMembers. We must also store the + * indexes into EquivalenceMembers which have no relids mentioned as some + * searches require that. + * + * Additionally, we also store the EquivalenceMembers of a given + * EquivalenceClass in the ec_members list. Technically, we could obtain this + * from looking at the bits in ec_member_indexes, however, finding all members + * of a given EquivalenceClass is common enough that it pays to have fast + * access to a dense list containing all members. + * + * The source and derived RestrictInfos are indexed in a similar method, + * although we don't maintain a List in the EquivalenceClass for these. These + * must be looked up in PlannerInfo using the ec_source_indexes and + * ec_derive_indexes Bitmapsets. + * * NB: if ec_merged isn't NULL, this class has been merged into another, and * should be ignored in favor of using the pointed-to class. * @@ -1322,9 +1382,19 @@ typedef struct EquivalenceClass List *ec_opfamilies; /* btree operator family OIDs */ Oid ec_collation; /* collation, if datatypes are collatable */ - List *ec_members; /* list of EquivalenceMembers */ - List *ec_sources; /* list of generating RestrictInfos */ - List *ec_derives; /* list of derived RestrictInfos */ + List *ec_members; /* list of EquivalenceMembers in the class */ + Bitmapset *ec_member_indexes; /* Indexes into all PlannerInfos + * eq_members for this EquivalenceClass */ + Bitmapset *ec_nonchild_indexes; /* Indexes into PlannerInfo's + * eq_members with em_is_child == + * false */ + Bitmapset *ec_norel_indexes; /* Indexes into PlannerInfo's eq_members + * for members where pull_varno on the + * em_expr is an empty set */ + Bitmapset *ec_source_indexes; /* indexes into PlannerInfo's eq_sources + * list of generating RestrictInfos */ + Bitmapset *ec_derive_indexes; /* indexes into PlannerInfo's eq_derives + * list of derived RestrictInfos */ Relids ec_relids; /* all relids appearing in ec_members, except * for child members (see below) */ bool ec_has_const; /* any pseudoconstants in ec_members? */ @@ -1373,9 +1443,10 @@ typedef struct EquivalenceMember NodeTag type; Expr *em_expr; /* the expression represented */ - Relids em_relids; /* all relids appearing in em_expr */ + Relids em_relids; /* relids for this member */ Relids em_nullable_relids; /* nullable by lower outer joins */ - bool em_is_const; /* expression is pseudoconstant? */ + bool em_is_const; /* is em_relids empty? */ + bool em_norel_expr; /* true if em_expr contains no Vars */ bool em_is_child; /* derived version for a child relation? */ Oid em_datatype; /* the "nominal type" used by the opfamily */ } EquivalenceMember; diff --git a/src/include/nodes/pg_list.h b/src/include/nodes/pg_list.h index 529a382d28..77f26edf22 100644 --- a/src/include/nodes/pg_list.h +++ b/src/include/nodes/pg_list.h @@ -629,6 +629,7 @@ extern pg_nodiscard List *list_copy_deep(const List *oldlist); typedef int (*list_sort_comparator) (const ListCell *a, const ListCell *b); extern void list_sort(List *list, list_sort_comparator cmp); +extern int list_ptr_cmp(const ListCell *p1, const ListCell *p2); extern int list_int_cmp(const ListCell *p1, const ListCell *p2); extern int list_oid_cmp(const ListCell *p1, const ListCell *p2); diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h index 9b38627efd..4d2221c2d0 100644 --- a/src/include/optimizer/paths.h +++ b/src/include/optimizer/paths.h @@ -114,6 +114,31 @@ extern void mark_dummy_rel(RelOptInfo *rel); * equivclass.c * routines for managing EquivalenceClasses */ + +/* + * EquivalenceMemberIterator + * Data structure used for iteration over an EquivalenceClass's + * EquivalenceMember in order to quickly find the members matching our + * search pattern. + */ +typedef struct EquivalenceMemberIterator +{ + bool use_index; /* use matching_ems index? */ + bool relids_empty; /* is with_relids empty? */ + bool with_children; /* include em_is_child members? */ + bool with_norel_members; /* include members with empty em_relids */ +#ifdef USE_ASSERT_CHECKING + bool isstrict; /* ensure the correct next function is used */ +#endif + int orig_length; /* elements in eclass->ec_members at the start */ + int current_index; /* current iterator position, or -1. */ + Relids with_relids; /* relids to match in em_relids */ + PlannerInfo *root; /* PlannerInfo the eclass belongs to */ + EquivalenceClass *eclass; /* the EquivalenceClass we're looking at */ + Bitmapset *matching_ems; /* when use_index == true, these are the + * matching indexes in root eq_members */ +} EquivalenceMemberIterator; + typedef bool (*ec_matches_callback_type) (PlannerInfo *root, RelOptInfo *rel, EquivalenceClass *ec, @@ -135,7 +160,8 @@ extern EquivalenceClass *get_eclass_for_sort_expr(PlannerInfo *root, Index sortref, Relids rel, bool create_it); -extern EquivalenceMember *find_ec_member_matching_expr(EquivalenceClass *ec, +extern EquivalenceMember *find_ec_member_matching_expr(PlannerInfo *root, + EquivalenceClass *ec, Expr *expr, Relids relids); extern EquivalenceMember *find_computable_ec_member(PlannerInfo *root, @@ -160,7 +186,8 @@ extern bool exprs_known_equal(PlannerInfo *root, Node *item1, Node *item2); extern EquivalenceClass *match_eclasses_to_foreign_key_col(PlannerInfo *root, ForeignKeyOptInfo *fkinfo, int colno); -extern RestrictInfo *find_derived_clause_for_ec_member(EquivalenceClass *ec, +extern RestrictInfo *find_derived_clause_for_ec_member(PlannerInfo *root, + EquivalenceClass *ec, EquivalenceMember *em); extern void add_child_rel_equivalences(PlannerInfo *root, AppendRelInfo *appinfo, @@ -186,6 +213,42 @@ extern bool eclass_useful_for_merging(PlannerInfo *root, extern bool is_redundant_derived_clause(RestrictInfo *rinfo, List *clauselist); extern bool is_redundant_with_indexclauses(RestrictInfo *rinfo, List *indexclauses); +extern Bitmapset *get_ecmember_indexes(PlannerInfo *root, + EquivalenceClass *ec, + Relids relids, + bool with_children, + bool with_norel_members); +extern Bitmapset *get_ecmember_indexes_strict(PlannerInfo *root, + EquivalenceClass *ec, + Relids relids, + bool with_children, + bool with_norel_members); +extern void setup_eclass_member_iterator(EquivalenceMemberIterator *iter, + PlannerInfo *root, + EquivalenceClass *ec, Relids relids, + bool with_children, + bool with_norel_members); +extern void setup_eclass_member_strict_iterator(EquivalenceMemberIterator *iter, + PlannerInfo *root, + EquivalenceClass *ec, + Relids relids, + bool with_children, + bool with_norel_members); +extern EquivalenceMember *eclass_member_iterator_next(EquivalenceMemberIterator *iter); +extern EquivalenceMember *eclass_member_iterator_strict_next(EquivalenceMemberIterator *iter); +extern void eclass_member_iterator_dispose(EquivalenceMemberIterator *iter); +extern Bitmapset *get_ec_source_indexes(PlannerInfo *root, + EquivalenceClass *ec, + Relids relids); +extern Bitmapset *get_ec_source_indexes_strict(PlannerInfo *root, + EquivalenceClass *ec, + Relids relids); +extern Bitmapset *get_ec_derive_indexes(PlannerInfo *root, + EquivalenceClass *ec, + Relids relids); +extern Bitmapset *get_ec_derive_indexes_strict(PlannerInfo *root, + EquivalenceClass *ec, + Relids relids); /* * pathkeys.c diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 51484ca7e2..2e9c6c530f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -641,6 +641,7 @@ EphemeralNamedRelationMetadata EphemeralNamedRelationMetadataData EquivalenceClass EquivalenceMember +EquivalenceMemberIterator ErrorContextCallback ErrorData ErrorSaveContext -- 2.39.0.windows.1 ^ permalink raw reply [nested|flat] 56+ messages in thread
* Re: [PoC] Reducing planning time when tables have many partitions @ 2023-02-06 01:47 Yuya Watari <[email protected]> parent: David Rowley <[email protected]> 0 siblings, 1 reply; 56+ messages in thread From: Yuya Watari @ 2023-02-06 01:47 UTC (permalink / raw) To: David Rowley <[email protected]>; +Cc: Thom Brown <[email protected]>; Andrey Lepikhov <[email protected]>; Alvaro Herrera <[email protected]>; Zhang Mingli <[email protected]>; Tom Lane <[email protected]>; PostgreSQL Developers <[email protected]> Dear David, On Mon, Jan 30, 2023 at 9:14 PM David Rowley <[email protected]> wrote: > I've attached what I worked on today. I really appreciate your quick response and the v15 patches. The bug fixes in the v15 look good to me. After receiving your email, I realized that this version does not apply to the current master. This conflict is caused by commits of 2489d76c49 [1] and related. I have attached the rebased version, v16, to this email. Resolving many conflicts was a bit of hard work, so I may have made some mistakes. Unfortunately, the rebased version did not pass regression tests. This failure is due to segmentation faults regarding a null reference to RelOptInfo. I show the code snippet that leads to the segfault as follows. ===== @@ -572,9 +662,31 @@ add_eq_member(EquivalenceClass *ec, Expr *expr, Relids relids, + i = -1; + while ((i = bms_next_member(expr_relids, i)) >= 0) + { + RelOptInfo *rel = root->simple_rel_array[i]; + + rel->eclass_member_indexes = bms_add_member(rel->eclass_member_indexes, em_index); + } ===== The segfault occurred because root->simple_rel_array[i] is sometimes NULL. This issue is similar to the one regarding root->simple_rel_array[0]. Before the commit of 2489d76c49, we only had to consider the nullability of root->simple_rel_array[0]. We overcame this problem by creating the RelOptInfo in the setup_append_rel_entry() function. However, after the commit, root->simple_rel_array[i] with non-zero 'i' can also be NULL. I'm not confident with its cause, but is this because non-base relations appear in the expr_relids? Seeing the commit, I found the following change in pull_varnos_walker(): ===== @@ -153,7 +161,11 @@ pull_varnos_walker(Node *node, pull_varnos_context *context) Var *var = (Var *) node; if (var->varlevelsup == context->sublevels_up) + { context->varnos = bms_add_member(context->varnos, var->varno); + context->varnos = bms_add_members(context->varnos, + var->varnullingrels); + } return false; } if (IsA(node, CurrentOfExpr)) ===== We get the expr_relids by pull_varnos(). This commit adds var->varnullingrels to its result. From my observations, indices 'i' such that root->simple_rel_array[i] is null come from var->varnullingrels. This change is probably related to the segfault. I don't understand the commit well, so please let me know if I'm wrong. To address this problem, in v16-0003, I moved EquivalenceMember indexes in RelOptInfo to PlannerInfo. This change allows us to store indexes whose corresponding RelOptInfo is NULL. > I'm happier > that this simple_rel_array[0] entry now only exists when planning set > operations, but I'd probably feel better if there was some other way > that felt less like we're faking up a RelOptInfo to store > EquivalenceMembers in. Of course, I'm not sure if my approach in v16-0003 is ideal, but it may help solve your concern above. Since simple_rel_array[0] is no longer necessary with my patch, I removed the setup_append_rel_entry() function in v16-0004. However, to work the patch, I needed to change some assertions in v16-0005. For more details, please see the commit message of v16-0005. After these works, the attached patches passed all regression tests in my environment. Instead of my approach, imitating the following change to get_eclass_indexes_for_relids() is also a possible solution. Ignoring NULL RelOptInfos enables us to avoid the segfault, but we have to adjust EquivalenceMemberIterator to match the result, and I'm not sure if this idea is correct. ===== @@ -3204,6 +3268,12 @@ get_eclass_indexes_for_relids(PlannerInfo *root, Relids relids) { RelOptInfo *rel = root->simple_rel_array[i]; + if (rel == NULL) /* must be an outer join */ + { + Assert(bms_is_member(i, root->outer_join_rels)); + continue; + } + ec_indexes = bms_add_members(ec_indexes, rel->eclass_indexes); } return ec_indexes; ===== [1] https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=2489d76c4906f4461a364ca8ad7e0751ead8a... -- Best regards, Yuya Watari Attachments: [application/octet-stream] v16-0001-Adjust-bms_int_members-so-that-it-shortens-the-l.patch (2.5K, ../../CAJ2pMkYR_X-=pq+39-W5kc0OG7q9u5YUwDBCHnkPur17DXnxuQ@mail.gmail.com/2-v16-0001-Adjust-bms_int_members-so-that-it-shortens-the-l.patch) download | inline diff: From 4ff471f5e65955b1ce50d409d5abf8cebc181dda Mon Sep 17 00:00:00 2001 From: David Rowley <[email protected]> Date: Thu, 1 Dec 2022 20:36:10 +1300 Subject: [PATCH v16 1/5] Adjust bms_int_members so that it shortens the left input Prior to this change, if the left input to bms_int_members had fewer Bitmapwords than the right input, the additional words in the left input would be zeroed. Doing things this why only really makes sense if we expect to add additional words to the resulting bitmap set again later. Seemingly, most of our use cases don't add to the resulting Bitmapset, in fact, many cases are performing bms_int_members in a loop which is likely to even further increase the amount of all zero words in the set. Leaving these zero trailing words in place can cause inefficiencies in functions such as bms_is_empty and loops which loop over a set using bms_next_member when the set is left with a large number of trailing words. If there are use cases which need to add additional words again, then, repalloc has to do very little processing when the requested allocation size is <= the actual allocation size. Both AllocSetRealloc and GenerationRealloc simply just return the input pointer again. If there are any cases where we do increase the size of the set again, then this just shifts the zeroing of the additional words until then. We could give bms_del_members() and bms_difference similar treatment, but that would require additional code to record the last non-zero word after masking out the members being deleted. That seems less worthwhile as it will add cost when we're unable to trim down the number of words. --- src/backend/nodes/bitmapset.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/src/backend/nodes/bitmapset.c b/src/backend/nodes/bitmapset.c index 98660524ad..b13b4c378f 100644 --- a/src/backend/nodes/bitmapset.c +++ b/src/backend/nodes/bitmapset.c @@ -920,8 +920,16 @@ bms_int_members(Bitmapset *a, const Bitmapset *b) shortlen = Min(a->nwords, b->nwords); for (i = 0; i < shortlen; i++) a->words[i] &= b->words[i]; - for (; i < a->nwords; i++) - a->words[i] = 0; + + /* + * We simply shorten the left input to both remove the need to zero the + * trailing words and also to reduce the required processing if this + * function is being called in a loop. If more words are required later + * then AllocSetRealloc is likely not going to have to work very hard if + * the new requested size is >= the actual size of the allocation. + */ + a->nwords = shortlen; + return a; } -- 2.35.3.windows.1 [application/octet-stream] v16-0002-Add-Bitmapset-indexes-for-faster-lookup-of-Equiv.patch (99.0K, ../../CAJ2pMkYR_X-=pq+39-W5kc0OG7q9u5YUwDBCHnkPur17DXnxuQ@mail.gmail.com/3-v16-0002-Add-Bitmapset-indexes-for-faster-lookup-of-Equiv.patch) download | inline diff: From dac902c47557deff26b56345d9fc5486d7237453 Mon Sep 17 00:00:00 2001 From: David Rowley <[email protected]> Date: Sun, 4 Dec 2022 11:09:21 +1300 Subject: [PATCH v16 2/5] Add Bitmapset indexes for faster lookup of EquivalenceMembers For queries containing a large number of joins or for queries to partitioned tables, the looking up of EquivalenceMembers in an EquivalenceClass can become slow due to the large number of members. Until now, the searching for EquivalenceMembers has been done using a linear search over the EquivalenceClass's list of members. Here we effectively add an index to allow faster lookups of these members. This is done by way of adding all members to all classes into a List in PlannerInfo, then we have a Bitmapset in each EquivalenceClass with members for each index in PlannerInfo's list which belong to this class. Additionally, each RelOptInfo also maintains another Bitmapset which stores each index in PlannerInfo's list of EquivalenceMember for all members, in all classes, which mention this relation. This allows us to quickly find all EquivalenceMembers for a given RelOptInfo and a given EquivalenceClass by intersecting these two lists. When searching for members belonging to join rels, we must either intersect or union all base RelOptInfo's member indexes then intersect the result to the EquivalenceClass's indexes. Whether we intersect or union depends on if we need members that mention all relations or members that mention any of the base relations making up the join relation. This method of indexing can significantly speed up the planner when querying a partitioned table with a large number of partitions when the query contains a few joins. Tests have shown around a x10 increase in performance with 1000 partitions. --- contrib/postgres_fdw/postgres_fdw.c | 50 +- src/backend/nodes/list.c | 16 + src/backend/nodes/outfuncs.c | 7 +- src/backend/optimizer/path/costsize.c | 3 +- src/backend/optimizer/path/equivclass.c | 1173 +++++++++++++++++---- src/backend/optimizer/path/indxpath.c | 22 +- src/backend/optimizer/path/pathkeys.c | 26 +- src/backend/optimizer/plan/createplan.c | 66 +- src/backend/optimizer/plan/planagg.c | 1 + src/backend/optimizer/plan/planner.c | 3 + src/backend/optimizer/prep/prepjointree.c | 3 + src/backend/optimizer/prep/prepunion.c | 25 +- src/backend/optimizer/util/relnode.c | 9 + src/include/nodes/pathnodes.h | 81 +- src/include/nodes/pg_list.h | 1 + src/include/optimizer/paths.h | 67 +- src/tools/pgindent/typedefs.list | 1 + 17 files changed, 1285 insertions(+), 269 deletions(-) diff --git a/contrib/postgres_fdw/postgres_fdw.c b/contrib/postgres_fdw/postgres_fdw.c index f5926ab89d..8c48b7e479 100644 --- a/contrib/postgres_fdw/postgres_fdw.c +++ b/contrib/postgres_fdw/postgres_fdw.c @@ -7617,6 +7617,10 @@ conversion_error_callback(void *arg) varno = var->varno; colno = var->varattno; } + /* + * XXX why don't we break here? Surely there can't be another + * equal EquivalenceMember? + */ } if (varno > 0) @@ -7675,23 +7679,27 @@ conversion_error_callback(void *arg) EquivalenceMember * find_em_for_rel(PlannerInfo *root, EquivalenceClass *ec, RelOptInfo *rel) { - ListCell *lc; + EquivalenceMemberIterator iter; + EquivalenceMember *em; - foreach(lc, ec->ec_members) - { - EquivalenceMember *em = (EquivalenceMember *) lfirst(lc); + setup_eclass_member_iterator(&iter, root, ec, rel->relids, true, false); + while ((em = eclass_member_iterator_next(&iter)) != NULL) + { /* * Note we require !bms_is_empty, else we'd accept constant * expressions which are not suitable for the purpose. */ + Assert(!bms_is_empty(em->em_relids)); + if (bms_is_subset(em->em_relids, rel->relids) && - !bms_is_empty(em->em_relids) && is_foreign_expr(root, rel, em->em_expr)) - return em; + break; } - return NULL; + eclass_member_iterator_dispose(&iter); + + return em; } /* @@ -7718,7 +7726,9 @@ find_em_for_rel_target(PlannerInfo *root, EquivalenceClass *ec, { Expr *expr = (Expr *) lfirst(lc1); Index sgref = get_pathtarget_sortgroupref(target, i); - ListCell *lc2; + EquivalenceMemberIterator iter; + EquivalenceMember *em; + Relids expr_relids; /* Ignore non-sort expressions */ if (sgref == 0 || @@ -7733,19 +7743,20 @@ find_em_for_rel_target(PlannerInfo *root, EquivalenceClass *ec, while (expr && IsA(expr, RelabelType)) expr = ((RelabelType *) expr)->arg; + expr_relids = pull_varnos(root, (Node *) expr); + setup_eclass_member_strict_iterator(&iter, root, ec, expr_relids, + false, false); + /* Locate an EquivalenceClass member matching this expr, if any */ - foreach(lc2, ec->ec_members) + while ((em = eclass_member_iterator_strict_next(&iter)) != NULL) { - EquivalenceMember *em = (EquivalenceMember *) lfirst(lc2); Expr *em_expr; - /* Don't match constants */ - if (em->em_is_const) - continue; + /* don't expect constants */ + Assert(!em->em_is_const); - /* Ignore child members */ - if (em->em_is_child) - continue; + /* don't expect child members */ + Assert(!em->em_is_child); /* Match if same expression (after stripping relabel) */ em_expr = em->em_expr; @@ -7757,10 +7768,15 @@ find_em_for_rel_target(PlannerInfo *root, EquivalenceClass *ec, /* Check that expression (including relabels!) is shippable */ if (is_foreign_expr(root, rel, em->em_expr)) + { + bms_free(expr_relids); + eclass_member_iterator_dispose(&iter); return em; + } } - i++; + bms_free(expr_relids); + eclass_member_iterator_dispose(&iter); } return NULL; diff --git a/src/backend/nodes/list.c b/src/backend/nodes/list.c index a709d23ef1..b8972bbe5a 100644 --- a/src/backend/nodes/list.c +++ b/src/backend/nodes/list.c @@ -1683,6 +1683,22 @@ list_sort(List *list, list_sort_comparator cmp) qsort(list->elements, len, sizeof(ListCell), (qsort_comparator) cmp); } +/* + * list_sort comparator for sorting a list into ascending ptr order. + */ +int +list_ptr_cmp(const ListCell *p1, const ListCell *p2) +{ + void *v1 = lfirst(p1); + void *v2 = lfirst(p2); + + if (v1 < v2) + return -1; + if (v1 > v2) + return 1; + return 0; +} + /* * list_sort comparator for sorting a list into ascending int order. */ diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index ba00b99249..416c972b4e 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -463,8 +463,11 @@ _outEquivalenceClass(StringInfo str, const EquivalenceClass *node) WRITE_NODE_FIELD(ec_opfamilies); WRITE_OID_FIELD(ec_collation); WRITE_NODE_FIELD(ec_members); - WRITE_NODE_FIELD(ec_sources); - WRITE_NODE_FIELD(ec_derives); + WRITE_BITMAPSET_FIELD(ec_member_indexes); + WRITE_BITMAPSET_FIELD(ec_nonchild_indexes); + WRITE_BITMAPSET_FIELD(ec_norel_indexes); + WRITE_BITMAPSET_FIELD(ec_source_indexes); + WRITE_BITMAPSET_FIELD(ec_derive_indexes); WRITE_BITMAPSET_FIELD(ec_relids); WRITE_BOOL_FIELD(ec_has_const); WRITE_BOOL_FIELD(ec_has_volatile); diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index 7918bb6f0d..361de60cb5 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -5507,7 +5507,8 @@ get_foreign_key_join_selectivity(PlannerInfo *root, if (ec && ec->ec_has_const) { EquivalenceMember *em = fkinfo->fk_eclass_member[i]; - RestrictInfo *rinfo = find_derived_clause_for_ec_member(ec, + RestrictInfo *rinfo = find_derived_clause_for_ec_member(root, + ec, em); if (rinfo) diff --git a/src/backend/optimizer/path/equivclass.c b/src/backend/optimizer/path/equivclass.c index de335fdb4d..aad14c25ce 100644 --- a/src/backend/optimizer/path/equivclass.c +++ b/src/backend/optimizer/path/equivclass.c @@ -32,8 +32,12 @@ #include "rewrite/rewriteManip.h" #include "utils/lsyscache.h" - -static EquivalenceMember *add_eq_member(EquivalenceClass *ec, +static void add_eq_source(PlannerInfo *root, EquivalenceClass *ec, + RestrictInfo *rinfo); +static void add_eq_derive(PlannerInfo *root, EquivalenceClass *ec, + RestrictInfo *rinfo); +static EquivalenceMember *add_eq_member(PlannerInfo *root, + EquivalenceClass *ec, Expr *expr, Relids relids, JoinDomain *jdomain, EquivalenceMember *parent, @@ -308,7 +312,6 @@ process_equivalence(PlannerInfo *root, /* If case 1, nothing to do, except add to sources */ if (ec1 == ec2) { - ec1->ec_sources = lappend(ec1->ec_sources, restrictinfo); ec1->ec_min_security = Min(ec1->ec_min_security, restrictinfo->security_level); ec1->ec_max_security = Max(ec1->ec_max_security, @@ -319,6 +322,8 @@ process_equivalence(PlannerInfo *root, /* mark the RI as usable with this pair of EMs */ restrictinfo->left_em = em1; restrictinfo->right_em = em2; + + add_eq_source(root, ec1, restrictinfo); return true; } @@ -339,8 +344,16 @@ process_equivalence(PlannerInfo *root, * be found. */ ec1->ec_members = list_concat(ec1->ec_members, ec2->ec_members); - ec1->ec_sources = list_concat(ec1->ec_sources, ec2->ec_sources); - ec1->ec_derives = list_concat(ec1->ec_derives, ec2->ec_derives); + ec1->ec_member_indexes = bms_add_members(ec1->ec_member_indexes, + ec2->ec_member_indexes); + ec1->ec_nonchild_indexes = bms_add_members(ec1->ec_nonchild_indexes, + ec2->ec_nonchild_indexes); + ec1->ec_norel_indexes = bms_add_members(ec1->ec_norel_indexes, + ec2->ec_norel_indexes); + ec1->ec_source_indexes = bms_join(ec1->ec_source_indexes, + ec2->ec_source_indexes); + ec1->ec_derive_indexes = bms_join(ec1->ec_derive_indexes, + ec2->ec_derive_indexes); ec1->ec_relids = bms_join(ec1->ec_relids, ec2->ec_relids); ec1->ec_has_const |= ec2->ec_has_const; /* can't need to set has_volatile */ @@ -352,10 +365,12 @@ process_equivalence(PlannerInfo *root, root->eq_classes = list_delete_nth_cell(root->eq_classes, ec2_idx); /* just to avoid debugging confusion w/ dangling pointers: */ ec2->ec_members = NIL; - ec2->ec_sources = NIL; - ec2->ec_derives = NIL; + ec2->ec_member_indexes = NULL; + ec2->ec_nonchild_indexes = NULL; + ec2->ec_norel_indexes = NULL; + ec2->ec_source_indexes = NULL; + ec2->ec_derive_indexes = NULL; ec2->ec_relids = NULL; - ec1->ec_sources = lappend(ec1->ec_sources, restrictinfo); ec1->ec_min_security = Min(ec1->ec_min_security, restrictinfo->security_level); ec1->ec_max_security = Max(ec1->ec_max_security, @@ -366,13 +381,14 @@ process_equivalence(PlannerInfo *root, /* mark the RI as usable with this pair of EMs */ restrictinfo->left_em = em1; restrictinfo->right_em = em2; + + add_eq_source(root, ec1, restrictinfo); } else if (ec1) { /* Case 3: add item2 to ec1 */ - em2 = add_eq_member(ec1, item2, item2_relids, + em2 = add_eq_member(root, ec1, item2, item2_relids, jdomain, NULL, item2_type); - ec1->ec_sources = lappend(ec1->ec_sources, restrictinfo); ec1->ec_min_security = Min(ec1->ec_min_security, restrictinfo->security_level); ec1->ec_max_security = Max(ec1->ec_max_security, @@ -383,13 +399,14 @@ process_equivalence(PlannerInfo *root, /* mark the RI as usable with this pair of EMs */ restrictinfo->left_em = em1; restrictinfo->right_em = em2; + + add_eq_source(root, ec1, restrictinfo); } else if (ec2) { /* Case 3: add item1 to ec2 */ - em1 = add_eq_member(ec2, item1, item1_relids, + em1 = add_eq_member(root, ec2, item1, item1_relids, jdomain, NULL, item1_type); - ec2->ec_sources = lappend(ec2->ec_sources, restrictinfo); ec2->ec_min_security = Min(ec2->ec_min_security, restrictinfo->security_level); ec2->ec_max_security = Max(ec2->ec_max_security, @@ -400,6 +417,8 @@ process_equivalence(PlannerInfo *root, /* mark the RI as usable with this pair of EMs */ restrictinfo->left_em = em1; restrictinfo->right_em = em2; + + add_eq_source(root, ec2, restrictinfo); } else { @@ -409,8 +428,11 @@ process_equivalence(PlannerInfo *root, ec->ec_opfamilies = opfamilies; ec->ec_collation = collation; ec->ec_members = NIL; - ec->ec_sources = list_make1(restrictinfo); - ec->ec_derives = NIL; + ec->ec_member_indexes = NULL; + ec->ec_nonchild_indexes = NULL; + ec->ec_norel_indexes = NULL; + ec->ec_source_indexes = NULL; + ec->ec_derive_indexes = NULL; ec->ec_relids = NULL; ec->ec_has_const = false; ec->ec_has_volatile = false; @@ -419,9 +441,9 @@ process_equivalence(PlannerInfo *root, ec->ec_min_security = restrictinfo->security_level; ec->ec_max_security = restrictinfo->security_level; ec->ec_merged = NULL; - em1 = add_eq_member(ec, item1, item1_relids, + em1 = add_eq_member(root, ec, item1, item1_relids, jdomain, NULL, item1_type); - em2 = add_eq_member(ec, item2, item2_relids, + em2 = add_eq_member(root, ec, item2, item2_relids, jdomain, NULL, item2_type); root->eq_classes = lappend(root->eq_classes, ec); @@ -432,6 +454,8 @@ process_equivalence(PlannerInfo *root, /* mark the RI as usable with this pair of EMs */ restrictinfo->left_em = em1; restrictinfo->right_em = em2; + + add_eq_source(root, ec, restrictinfo); } return true; @@ -507,14 +531,61 @@ canonicalize_ec_expression(Expr *expr, Oid req_type, Oid req_collation) return expr; } +/* + * add_eq_source - add 'rinfo' in eq_sources for this 'ec' + */ +static void +add_eq_source(PlannerInfo *root, EquivalenceClass *ec, RestrictInfo *rinfo) +{ + int source_idx = list_length(root->eq_sources); + int i; + + ec->ec_source_indexes = bms_add_member(ec->ec_source_indexes, source_idx); + root->eq_sources = lappend(root->eq_sources, rinfo); + + i = -1; + while ((i = bms_next_member(rinfo->clause_relids, i)) >= 0) + { + RelOptInfo *rel = root->simple_rel_array[i]; + + rel->eclass_source_indexes = bms_add_member(rel->eclass_source_indexes, + source_idx); + } +} + +/* + * add_eq_derive - add 'rinfo' in eq_derives for this 'ec' + */ +static void +add_eq_derive(PlannerInfo *root, EquivalenceClass *ec, RestrictInfo *rinfo) +{ + int derive_idx = list_length(root->eq_derives); + int i; + + ec->ec_derive_indexes = bms_add_member(ec->ec_derive_indexes, derive_idx); + root->eq_derives = lappend(root->eq_derives, rinfo); + + i = -1; + while ((i = bms_next_member(rinfo->clause_relids, i)) >= 0) + { + RelOptInfo *rel = root->simple_rel_array[i]; + + rel->eclass_derive_indexes = bms_add_member(rel->eclass_derive_indexes, + derive_idx); + } +} + /* * add_eq_member - build a new EquivalenceMember and add it to an EC */ static EquivalenceMember * -add_eq_member(EquivalenceClass *ec, Expr *expr, Relids relids, +add_eq_member(PlannerInfo *root, EquivalenceClass *ec, Expr *expr, Relids relids, JoinDomain *jdomain, EquivalenceMember *parent, Oid datatype) { EquivalenceMember *em = makeNode(EquivalenceMember); + Relids expr_relids; + int em_index = list_length(root->eq_members); + int i; em->em_expr = expr; em->em_relids = relids; @@ -524,6 +595,23 @@ add_eq_member(EquivalenceClass *ec, Expr *expr, Relids relids, em->em_jdomain = jdomain; em->em_parent = parent; + /* + * We must determine the exact set of relids in the expr for child + * EquivalenceMembers as what is given to us in 'relids' may differ from + * the relids mentioned in the expression. See add_child_rel_equivalences + */ + if (parent != NULL) + expr_relids = pull_varnos(root, (Node *) expr); + else + { + expr_relids = relids; + /* We expect the relids to match for non-child members */ + Assert(bms_equal(pull_varnos(root, (Node *) expr), relids)); + } + + /* record the actual relids from 'expr' */ + em->em_norel_expr = bms_is_empty(expr_relids); + if (bms_is_empty(relids)) { /* @@ -542,9 +630,31 @@ add_eq_member(EquivalenceClass *ec, Expr *expr, Relids relids, else if (!parent) /* child members don't add to ec_relids */ { ec->ec_relids = bms_add_members(ec->ec_relids, relids); + ec->ec_nonchild_indexes = bms_add_member(ec->ec_nonchild_indexes, em_index); } + + /* add the new member to the list */ ec->ec_members = lappend(ec->ec_members, em); + /* and add it to the index and PlannerInfo's list */ + ec->ec_member_indexes = bms_add_member(ec->ec_member_indexes, em_index); + root->eq_members = lappend(root->eq_members, em); + + /* record exprs with no relids */ + if (bms_is_empty(expr_relids)) + ec->ec_norel_indexes = bms_add_member(ec->ec_norel_indexes, em_index); + + if (parent != NULL) + expr_relids = bms_add_members(expr_relids, relids); + + i = -1; + while ((i = bms_next_member(expr_relids, i)) >= 0) + { + RelOptInfo *rel = root->simple_rel_array[i]; + + rel->eclass_member_indexes = bms_add_member(rel->eclass_member_indexes, em_index); + } + return em; } @@ -601,6 +711,7 @@ get_eclass_for_sort_expr(PlannerInfo *root, * Ensure the expression exposes the correct type and collation. */ expr = canonicalize_ec_expression(expr, opcintype, collation); + expr_relids = pull_varnos(root, (Node *) expr); /* * Since SortGroupClause nodes are top-level expressions (GROUP BY, ORDER @@ -614,7 +725,8 @@ get_eclass_for_sort_expr(PlannerInfo *root, foreach(lc1, root->eq_classes) { EquivalenceClass *cur_ec = (EquivalenceClass *) lfirst(lc1); - ListCell *lc2; + EquivalenceMemberIterator iter; + EquivalenceMember *cur_em; /* * Never match to a volatile EC, except when we are looking at another @@ -629,10 +741,11 @@ get_eclass_for_sort_expr(PlannerInfo *root, if (!equal(opfamilies, cur_ec->ec_opfamilies)) continue; - foreach(lc2, cur_ec->ec_members) - { - EquivalenceMember *cur_em = (EquivalenceMember *) lfirst(lc2); + setup_eclass_member_strict_iterator(&iter, root, cur_ec, expr_relids, + true, true); + while ((cur_em = eclass_member_iterator_strict_next(&iter)) != NULL) + { /* * Ignore child members unless they match the request. */ @@ -651,6 +764,8 @@ get_eclass_for_sort_expr(PlannerInfo *root, equal(expr, cur_em->em_expr)) return cur_ec; /* Match! */ } + + eclass_member_iterator_dispose(&iter); } /* No match; does caller want a NULL result? */ @@ -668,8 +783,11 @@ get_eclass_for_sort_expr(PlannerInfo *root, newec->ec_opfamilies = list_copy(opfamilies); newec->ec_collation = collation; newec->ec_members = NIL; - newec->ec_sources = NIL; - newec->ec_derives = NIL; + newec->ec_member_indexes = NULL; + newec->ec_nonchild_indexes = NULL; + newec->ec_norel_indexes = NULL; + newec->ec_source_indexes = NULL; + newec->ec_derive_indexes = NULL; newec->ec_relids = NULL; newec->ec_has_const = false; newec->ec_has_volatile = contain_volatile_functions((Node *) expr); @@ -682,12 +800,7 @@ get_eclass_for_sort_expr(PlannerInfo *root, if (newec->ec_has_volatile && sortref == 0) /* should not happen */ elog(ERROR, "volatile EquivalenceClass has no sortref"); - /* - * Get the precise set of relids appearing in the expression. - */ - expr_relids = pull_varnos(root, (Node *) expr); - - newem = add_eq_member(newec, copyObject(expr), expr_relids, + newem = add_eq_member(root, newec, copyObject(expr), expr_relids, jdomain, NULL, opcintype); /* @@ -719,7 +832,7 @@ get_eclass_for_sort_expr(PlannerInfo *root, int ec_index = list_length(root->eq_classes) - 1; int i = -1; - while ((i = bms_next_member(newec->ec_relids, i)) > 0) + while ((i = bms_next_member(newec->ec_relids, i)) >= 0) { RelOptInfo *rel = root->simple_rel_array[i]; @@ -755,19 +868,25 @@ get_eclass_for_sort_expr(PlannerInfo *root, * Child EC members are ignored unless they belong to given 'relids'. */ EquivalenceMember * -find_ec_member_matching_expr(EquivalenceClass *ec, +find_ec_member_matching_expr(PlannerInfo *root, + EquivalenceClass *ec, Expr *expr, Relids relids) { - ListCell *lc; + EquivalenceMemberIterator iter; + Relids expr_relids; + EquivalenceMember *em; /* We ignore binary-compatible relabeling on both ends */ while (expr && IsA(expr, RelabelType)) expr = ((RelabelType *) expr)->arg; - foreach(lc, ec->ec_members) + expr_relids = pull_varnos(root, (Node *) expr); + setup_eclass_member_strict_iterator(&iter, root, ec, expr_relids, true, + true); + + while ((em = eclass_member_iterator_strict_next(&iter)) != NULL) { - EquivalenceMember *em = (EquivalenceMember *) lfirst(lc); Expr *emexpr; /* @@ -792,10 +911,13 @@ find_ec_member_matching_expr(EquivalenceClass *ec, emexpr = ((RelabelType *) emexpr)->arg; if (equal(emexpr, expr)) - return em; + break; } - return NULL; + bms_free(expr_relids); + eclass_member_iterator_dispose(&iter); + + return em; } /* @@ -937,7 +1059,7 @@ relation_can_be_sorted_early(PlannerInfo *root, RelOptInfo *rel, { Expr *targetexpr = (Expr *) lfirst(lc); - em = find_ec_member_matching_expr(ec, targetexpr, rel->relids); + em = find_ec_member_matching_expr(root, ec, targetexpr, rel->relids); if (!em) continue; @@ -1024,7 +1146,7 @@ relation_can_be_sorted_early(PlannerInfo *root, RelOptInfo *rel, * scanning of the quals and before Path construction begins. * * We make no attempt to avoid generating duplicate RestrictInfos here: we - * don't search ec_sources or ec_derives for matches. It doesn't really + * don't search eq_sources or eq_derives for matches. It doesn't really * seem worth the trouble to do so. */ void @@ -1081,7 +1203,7 @@ generate_base_implied_equalities(PlannerInfo *root) * this is a cheap version of has_relevant_eclass_joinclause(). */ i = -1; - while ((i = bms_next_member(ec->ec_relids, i)) > 0) + while ((i = bms_next_member(ec->ec_relids, i)) >= 0) { RelOptInfo *rel = root->simple_rel_array[i]; @@ -1113,6 +1235,7 @@ generate_base_implied_equalities_const(PlannerInfo *root, { EquivalenceMember *const_em = NULL; ListCell *lc; + int i; /* * In the trivial case where we just had one "var = const" clause, push @@ -1122,9 +1245,9 @@ generate_base_implied_equalities_const(PlannerInfo *root, * equivalent to the old one. */ if (list_length(ec->ec_members) == 2 && - list_length(ec->ec_sources) == 1) + bms_get_singleton_member(ec->ec_source_indexes, &i)) { - RestrictInfo *restrictinfo = (RestrictInfo *) linitial(ec->ec_sources); + RestrictInfo *restrictinfo = list_nth_node(RestrictInfo, root->eq_sources, i); distribute_restrictinfo_to_rels(root, restrictinfo); return; @@ -1136,9 +1259,11 @@ generate_base_implied_equalities_const(PlannerInfo *root, * machinery might be able to exclude relations on the basis of generated * "var = const" equalities, but "var = param" won't work for that. */ - foreach(lc, ec->ec_members) + i = -1; + while ((i = bms_next_member(ec->ec_norel_indexes, i)) >= 0) { - EquivalenceMember *cur_em = (EquivalenceMember *) lfirst(lc); + EquivalenceMember *cur_em = list_nth_node(EquivalenceMember, + root->eq_members, i); if (cur_em->em_is_const) { @@ -1182,9 +1307,9 @@ generate_base_implied_equalities_const(PlannerInfo *root, /* * If the clause didn't degenerate to a constant, fill in the correct - * markings for a mergejoinable clause, and save it in ec_derives. (We + * markings for a mergejoinable clause, and save it in eq_derives. (We * will not re-use such clauses directly, but selectivity estimation - * may consult the list later. Note that this use of ec_derives does + * may consult the list later. Note that this use of eq_derives does * not overlap with its use for join clauses, since we never generate * join clauses from an ec_has_const eclass.) */ @@ -1194,7 +1319,8 @@ generate_base_implied_equalities_const(PlannerInfo *root, rinfo->left_ec = rinfo->right_ec = ec; rinfo->left_em = cur_em; rinfo->right_em = const_em; - ec->ec_derives = lappend(ec->ec_derives, rinfo); + + add_eq_derive(root, ec, rinfo); } } } @@ -1207,7 +1333,8 @@ generate_base_implied_equalities_no_const(PlannerInfo *root, EquivalenceClass *ec) { EquivalenceMember **prev_ems; - ListCell *lc; + Bitmapset *matching_ems; + int i; /* * We scan the EC members once and track the last-seen member for each @@ -1220,9 +1347,12 @@ generate_base_implied_equalities_no_const(PlannerInfo *root, prev_ems = (EquivalenceMember **) palloc0(root->simple_rel_array_size * sizeof(EquivalenceMember *)); - foreach(lc, ec->ec_members) + matching_ems = bms_difference(ec->ec_member_indexes, ec->ec_norel_indexes); + i = -1; + while ((i = bms_next_member(matching_ems, i)) >= 0) { - EquivalenceMember *cur_em = (EquivalenceMember *) lfirst(lc); + EquivalenceMember *cur_em = list_nth_node(EquivalenceMember, + root->eq_members, i); int relid; Assert(!cur_em->em_is_child); /* no children yet */ @@ -1260,7 +1390,7 @@ generate_base_implied_equalities_no_const(PlannerInfo *root, /* * If the clause didn't degenerate to a constant, fill in the * correct markings for a mergejoinable clause. We don't put it - * in ec_derives however; we don't currently need to re-find such + * in eq_derives however; we don't currently need to re-find such * clauses, and we don't want to clutter that list with non-join * clauses. */ @@ -1283,11 +1413,15 @@ generate_base_implied_equalities_no_const(PlannerInfo *root, * For the moment we force all the Vars to be available at all join nodes * for this eclass. Perhaps this could be improved by doing some * pre-analysis of which members we prefer to join, but it's no worse than - * what happened in the pre-8.3 code. + * what happened in the pre-8.3 code. We're able to make use of + * matching_ems from above. We're not going to find Vars in + * em_const_indexes. */ - foreach(lc, ec->ec_members) + i = -1; + while ((i = bms_next_member(matching_ems, i)) >= 0) { - EquivalenceMember *cur_em = (EquivalenceMember *) lfirst(lc); + EquivalenceMember *cur_em = list_nth_node(EquivalenceMember, + root->eq_members, i); List *vars = pull_var_clause((Node *) cur_em->em_expr, PVC_RECURSE_AGGREGATES | PVC_RECURSE_WINDOWFUNCS | @@ -1296,6 +1430,7 @@ generate_base_implied_equalities_no_const(PlannerInfo *root, add_vars_to_targetlist(root, vars, ec->ec_relids); list_free(vars); } + bms_free(matching_ems); } /* @@ -1316,11 +1451,12 @@ static void generate_base_implied_equalities_broken(PlannerInfo *root, EquivalenceClass *ec) { - ListCell *lc; + int i = -1; - foreach(lc, ec->ec_sources) + while ((i = bms_next_member(ec->ec_source_indexes, i)) >= 0) { - RestrictInfo *restrictinfo = (RestrictInfo *) lfirst(lc); + RestrictInfo *restrictinfo = list_nth_node(RestrictInfo, + root->eq_sources, i); if (ec->ec_has_const || bms_membership(restrictinfo->required_relids) != BMS_MULTIPLE) @@ -1361,11 +1497,11 @@ generate_base_implied_equalities_broken(PlannerInfo *root, * Because the same join clauses are likely to be needed multiple times as * we consider different join paths, we avoid generating multiple copies: * whenever we select a particular pair of EquivalenceMembers to join, - * we check to see if the pair matches any original clause (in ec_sources) - * or previously-built clause (in ec_derives). This saves memory and allows - * re-use of information cached in RestrictInfos. We also avoid generating - * commutative duplicates, i.e. if the algorithm selects "a.x = b.y" but - * we already have "b.y = a.x", we return the existing clause. + * we check to see if the pair matches any original clause in root's + * eq_sources or previously-built clause (in root's eq_derives). This saves + * memory and allows re-use of information cached in RestrictInfos. We also + * avoid generating commutative duplicates, i.e. if the algorithm selects + * "a.x = b.y" but we already have "b.y = a.x", we return the existing clause. * * join_relids should always equal bms_union(outer_relids, inner_rel->relids). * We could simplify this function's API by computing it internally, but in @@ -1527,6 +1663,8 @@ generate_join_implied_equalities_normal(PlannerInfo *root, Relids outer_relids, Relids inner_relids) { + EquivalenceMemberIterator iter; + EquivalenceMember *cur_em; List *result = NIL; List *new_members = NIL; List *outer_members = NIL; @@ -1542,10 +1680,10 @@ generate_join_implied_equalities_normal(PlannerInfo *root, * as well as to at least one input member, plus enforce at least one * outer-rel member equal to at least one inner-rel member. */ - foreach(lc1, ec->ec_members) - { - EquivalenceMember *cur_em = (EquivalenceMember *) lfirst(lc1); + setup_eclass_member_iterator(&iter, root, ec, join_relids, true, false); + while ((cur_em = eclass_member_iterator_next(&iter)) != NULL) + { /* * We don't need to check explicitly for child EC members. This test * against join_relids will cause them to be ignored except when @@ -1562,6 +1700,8 @@ generate_join_implied_equalities_normal(PlannerInfo *root, new_members = lappend(new_members, cur_em); } + eclass_member_iterator_dispose(&iter); + /* * First, select the joinclause if needed. We can equate any one outer * member to any one inner member, but we have to find a datatype @@ -1659,7 +1799,7 @@ generate_join_implied_equalities_normal(PlannerInfo *root, foreach(lc1, new_members) { - EquivalenceMember *cur_em = (EquivalenceMember *) lfirst(lc1); + cur_em = (EquivalenceMember *) lfirst(lc1); if (prev_em != NULL) { @@ -1704,12 +1844,16 @@ generate_join_implied_equalities_broken(PlannerInfo *root, Relids nominal_inner_relids, RelOptInfo *inner_rel) { + Bitmapset *matching_es; List *result = NIL; - ListCell *lc; + int i; - foreach(lc, ec->ec_sources) + matching_es = get_ec_source_indexes(root, ec, nominal_join_relids); + i = -1; + while ((i = bms_next_member(matching_es, i)) >= 0) { - RestrictInfo *restrictinfo = (RestrictInfo *) lfirst(lc); + RestrictInfo *restrictinfo = list_nth_node(RestrictInfo, + root->eq_sources, i); Relids clause_relids = restrictinfo->required_relids; if (bms_is_subset(clause_relids, nominal_join_relids) && @@ -1721,12 +1865,12 @@ generate_join_implied_equalities_broken(PlannerInfo *root, /* * If we have to translate, just brute-force apply adjust_appendrel_attrs * to all the RestrictInfos at once. This will result in returning - * RestrictInfos that are not listed in ec_derives, but there shouldn't be + * RestrictInfos that are not listed in eq_derives, but there shouldn't be * any duplication, and it's a sufficiently narrow corner case that we * shouldn't sweat too much over it anyway. * * Since inner_rel might be an indirect descendant of the baserel - * mentioned in the ec_sources clauses, we have to be prepared to apply + * mentioned in the eq_sources clauses, we have to be prepared to apply * multiple levels of Var translation. */ if (IS_OTHER_REL(inner_rel) && result != NIL) @@ -1788,10 +1932,11 @@ create_join_clause(PlannerInfo *root, EquivalenceMember *rightem, EquivalenceClass *parent_ec) { + Bitmapset *matches; RestrictInfo *rinfo; RestrictInfo *parent_rinfo = NULL; - ListCell *lc; MemoryContext oldcontext; + int i; /* * Search to see if we already built a RestrictInfo for this pair of @@ -1802,9 +1947,12 @@ create_join_clause(PlannerInfo *root, * it's not identical, it'd better have the same effects, or the operator * families we're using are broken. */ - foreach(lc, ec->ec_sources) + matches = bms_int_members(get_ec_source_indexes_strict(root, ec, leftem->em_relids), + get_ec_source_indexes_strict(root, ec, rightem->em_relids)); + i = -1; + while ((i = bms_next_member(matches, i)) >= 0) { - rinfo = (RestrictInfo *) lfirst(lc); + rinfo = list_nth_node(RestrictInfo, root->eq_sources, i); if (rinfo->left_em == leftem && rinfo->right_em == rightem && rinfo->parent_ec == parent_ec) @@ -1815,9 +1963,13 @@ create_join_clause(PlannerInfo *root, return rinfo; } - foreach(lc, ec->ec_derives) + matches = bms_int_members(get_ec_derive_indexes_strict(root, ec, leftem->em_relids), + get_ec_derive_indexes_strict(root, ec, rightem->em_relids)); + + i = -1; + while ((i = bms_next_member(matches, i)) >= 0) { - rinfo = (RestrictInfo *) lfirst(lc); + rinfo = list_nth_node(RestrictInfo, root->eq_derives, i); if (rinfo->left_em == leftem && rinfo->right_em == rightem && rinfo->parent_ec == parent_ec) @@ -1875,7 +2027,7 @@ create_join_clause(PlannerInfo *root, rinfo->left_em = leftem; rinfo->right_em = rightem; /* and save it for possible re-use */ - ec->ec_derives = lappend(ec->ec_derives, rinfo); + add_eq_derive(root, ec, rinfo); MemoryContextSwitchTo(oldcontext); @@ -2075,7 +2227,8 @@ reconsider_outer_join_clause(PlannerInfo *root, OuterJoinClauseInfo *ojcinfo, left_type, right_type, inner_datatype; - Relids inner_relids; + Relids outer_relids, + inner_relids; ListCell *lc1; Assert(is_opclause(rinfo->clause)); @@ -2089,6 +2242,7 @@ reconsider_outer_join_clause(PlannerInfo *root, OuterJoinClauseInfo *ojcinfo, outervar = (Expr *) get_leftop(rinfo->clause); innervar = (Expr *) get_rightop(rinfo->clause); inner_datatype = right_type; + outer_relids = rinfo->left_relids; inner_relids = rinfo->right_relids; } else @@ -2096,6 +2250,7 @@ reconsider_outer_join_clause(PlannerInfo *root, OuterJoinClauseInfo *ojcinfo, outervar = (Expr *) get_rightop(rinfo->clause); innervar = (Expr *) get_leftop(rinfo->clause); inner_datatype = left_type; + outer_relids = rinfo->right_relids; inner_relids = rinfo->left_relids; } @@ -2103,8 +2258,10 @@ reconsider_outer_join_clause(PlannerInfo *root, OuterJoinClauseInfo *ojcinfo, foreach(lc1, root->eq_classes) { EquivalenceClass *cur_ec = (EquivalenceClass *) lfirst(lc1); + EquivalenceMemberIterator iter; + EquivalenceMember *cur_em; bool match; - ListCell *lc2; + int i; /* Ignore EC unless it contains pseudoconstants */ if (!cur_ec->ec_has_const) @@ -2119,10 +2276,12 @@ reconsider_outer_join_clause(PlannerInfo *root, OuterJoinClauseInfo *ojcinfo, continue; /* Does it contain a match to outervar? */ match = false; - foreach(lc2, cur_ec->ec_members) - { - EquivalenceMember *cur_em = (EquivalenceMember *) lfirst(lc2); + setup_eclass_member_strict_iterator(&iter, root, cur_ec, outer_relids, + false, true); + + while ((cur_em = eclass_member_iterator_strict_next(&iter)) != NULL) + { Assert(!cur_em->em_is_child); /* no children yet */ if (equal(outervar, cur_em->em_expr)) { @@ -2130,6 +2289,8 @@ reconsider_outer_join_clause(PlannerInfo *root, OuterJoinClauseInfo *ojcinfo, break; } } + eclass_member_iterator_dispose(&iter); + if (!match) continue; /* no match, so ignore this EC */ @@ -2139,13 +2300,15 @@ reconsider_outer_join_clause(PlannerInfo *root, OuterJoinClauseInfo *ojcinfo, * constant before we can decide to throw away the outer-join clause. */ match = false; - foreach(lc2, cur_ec->ec_members) + i = -1; + while ((i = bms_next_member(cur_ec->ec_norel_indexes, i)) >= 0) { - EquivalenceMember *cur_em = (EquivalenceMember *) lfirst(lc2); Oid eq_op; RestrictInfo *newrinfo; JoinDomain *jdomain; + cur_em = list_nth_node(EquivalenceMember, root->eq_members, i); + if (!cur_em->em_is_const) continue; /* ignore non-const members */ eq_op = select_equality_operator(cur_ec, @@ -2215,11 +2378,12 @@ reconsider_full_join_clause(PlannerInfo *root, OuterJoinClauseInfo *ojcinfo) { EquivalenceClass *cur_ec = (EquivalenceClass *) lfirst(lc1); EquivalenceMember *coal_em = NULL; + Bitmapset *matching_ems; bool match; bool matchleft; bool matchright; - ListCell *lc2; int coal_idx = -1; + int i; /* Ignore EC unless it contains pseudoconstants */ if (!cur_ec->ec_has_const) @@ -2246,10 +2410,14 @@ reconsider_full_join_clause(PlannerInfo *root, OuterJoinClauseInfo *ojcinfo) * the two column types). Is it OK to strip implicit coercions from * the COALESCE arguments? */ + matching_ems = get_ecmember_indexes_strict(root, cur_ec, + rinfo->clause_relids, true, + false); match = false; - foreach(lc2, cur_ec->ec_members) + i = -1; + while ((i = bms_next_member(matching_ems, i)) >= 0) { - coal_em = (EquivalenceMember *) lfirst(lc2); + coal_em = list_nth_node(EquivalenceMember, root->eq_members, i); Assert(!coal_em->em_is_child); /* no children yet */ if (IsA(coal_em->em_expr, CoalesceExpr)) { @@ -2276,7 +2444,7 @@ reconsider_full_join_clause(PlannerInfo *root, OuterJoinClauseInfo *ojcinfo) if (equal(leftvar, cfirst) && equal(rightvar, csecond)) { - coal_idx = foreach_current_index(lc2); + coal_idx = i; match = true; break; } @@ -2292,9 +2460,11 @@ reconsider_full_join_clause(PlannerInfo *root, OuterJoinClauseInfo *ojcinfo) * decide to throw away the outer-join clause. */ matchleft = matchright = false; - foreach(lc2, cur_ec->ec_members) + i = -1; + while ((i = bms_next_member(cur_ec->ec_norel_indexes, i)) >= 0) { - EquivalenceMember *cur_em = (EquivalenceMember *) lfirst(lc2); + EquivalenceMember *cur_em = list_nth_node(EquivalenceMember, + root->eq_members, i); Oid eq_op; RestrictInfo *newrinfo; JoinDomain *jdomain; @@ -2342,11 +2512,28 @@ reconsider_full_join_clause(PlannerInfo *root, OuterJoinClauseInfo *ojcinfo) * we can throw away the full-join clause as redundant. Moreover, we * can remove the COALESCE entry from the EC, since the added * restrictions ensure it will always have the expected value. (We - * don't bother trying to update ec_relids or ec_sources.) + * don't bother trying to update ec_relids or root's eq_sources.) */ if (matchleft && matchright) { - cur_ec->ec_members = list_delete_nth_cell(cur_ec->ec_members, coal_idx); + /* XXX performance of list_delete()?? */ + cur_ec->ec_members = list_delete(cur_ec->ec_members, coal_em); + cur_ec->ec_member_indexes = bms_del_member(cur_ec->ec_member_indexes, + coal_idx); + cur_ec->ec_nonchild_indexes = bms_del_member(cur_ec->ec_nonchild_indexes, + coal_idx); + /* no need to adjust ec_norel_indexes */ + + /* Remove the member from each of the relations */ + i = -1; + while ((i = bms_next_member(coal_em->em_relids, i)) >= 0) + { + RelOptInfo *rel = root->simple_rel_array[i]; + + rel->eclass_member_indexes = bms_del_member(rel->eclass_member_indexes, + coal_idx); + } + return true; } @@ -2401,21 +2588,28 @@ bool exprs_known_equal(PlannerInfo *root, Node *item1, Node *item2) { ListCell *lc1; + Relids item1_relids = pull_varnos(root, item1); + Relids item2_relids = pull_varnos(root, item2); foreach(lc1, root->eq_classes) { EquivalenceClass *ec = (EquivalenceClass *) lfirst(lc1); bool item1member = false; bool item2member = false; - ListCell *lc2; + Bitmapset *matching_ems; + int i; /* Never match to a volatile EC */ if (ec->ec_has_volatile) continue; - foreach(lc2, ec->ec_members) + matching_ems = bms_join(get_ecmember_indexes_strict(root, ec, item1_relids, false, true), + get_ecmember_indexes_strict(root, ec, item2_relids, false, true)); + i = -1; + while ((i = bms_next_member(matching_ems, i)) >= 0) { - EquivalenceMember *em = (EquivalenceMember *) lfirst(lc2); + EquivalenceMember *em = list_nth_node(EquivalenceMember, + root->eq_members, i); if (em->em_is_child) continue; /* ignore children here */ @@ -2478,16 +2672,21 @@ match_eclasses_to_foreign_key_col(PlannerInfo *root, i); EquivalenceMember *item1_em = NULL; EquivalenceMember *item2_em = NULL; - ListCell *lc2; + Bitmapset *matching_ems; + int j; /* Never match to a volatile EC */ if (ec->ec_has_volatile) continue; /* Note: it seems okay to match to "broken" eclasses here */ + matching_ems = bms_join(get_ecmember_indexes_strict(root, ec, rel1->relids, false, false), + get_ecmember_indexes_strict(root, ec, rel2->relids, false, false)); - foreach(lc2, ec->ec_members) + j = -1; + while ((j = bms_next_member(matching_ems, j)) >= 0) { - EquivalenceMember *em = (EquivalenceMember *) lfirst(lc2); + EquivalenceMember *em = list_nth_node(EquivalenceMember, + root->eq_members, j); Var *var; if (em->em_is_child) @@ -2540,16 +2739,19 @@ match_eclasses_to_foreign_key_col(PlannerInfo *root, * Returns NULL if no such clause can be found. */ RestrictInfo * -find_derived_clause_for_ec_member(EquivalenceClass *ec, +find_derived_clause_for_ec_member(PlannerInfo *root, EquivalenceClass *ec, EquivalenceMember *em) { - ListCell *lc; + int i; Assert(ec->ec_has_const); Assert(!em->em_is_const); - foreach(lc, ec->ec_derives) + + i = -1; + while ((i = bms_next_member(ec->ec_derive_indexes, i)) >= 0) { - RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); + RestrictInfo *rinfo = list_nth_node(RestrictInfo, root->eq_derives, + i); /* * generate_base_implied_equalities_const will have put non-const @@ -2600,7 +2802,8 @@ add_child_rel_equivalences(PlannerInfo *root, while ((i = bms_next_member(parent_rel->eclass_indexes, i)) >= 0) { EquivalenceClass *cur_ec = (EquivalenceClass *) list_nth(root->eq_classes, i); - int num_members; + EquivalenceMemberIterator iter; + EquivalenceMember *cur_em; /* * If this EC contains a volatile expression, then generating child @@ -2614,29 +2817,15 @@ add_child_rel_equivalences(PlannerInfo *root, Assert(bms_is_subset(top_parent_relids, cur_ec->ec_relids)); /* - * We don't use foreach() here because there's no point in scanning - * newly-added child members, so we can stop after the last - * pre-existing EC member. + * Looping using the EquivalenceMemberIterator means we only loop over + * the members which existed when we set up the iterator, not newly + * added ones. */ - num_members = list_length(cur_ec->ec_members); - for (int pos = 0; pos < num_members; pos++) - { - EquivalenceMember *cur_em = (EquivalenceMember *) list_nth(cur_ec->ec_members, pos); - - if (cur_em->em_is_const) - continue; /* ignore consts here */ - - /* - * We consider only original EC members here, not - * already-transformed child members. Otherwise, if some original - * member expression references more than one appendrel, we'd get - * an O(N^2) explosion of useless derived expressions for - * combinations of children. (But add_child_join_rel_equivalences - * may add targeted combinations for partitionwise-join purposes.) - */ - if (cur_em->em_is_child) - continue; /* ignore children here */ + setup_eclass_member_iterator(&iter, root, cur_ec, top_parent_relids, + false, false); + while ((cur_em = eclass_member_iterator_next(&iter)) != NULL) + { /* * Consider only members that reference and can be computed at * child's topmost parent rel. In particular we want to exclude @@ -2645,13 +2834,16 @@ add_child_rel_equivalences(PlannerInfo *root, * simple Var; and in any case it wouldn't produce a member that * has any use in creating plans for the child rel. */ - if (bms_is_subset(cur_em->em_relids, top_parent_relids) && - !bms_is_empty(cur_em->em_relids)) + if (bms_is_subset(cur_em->em_relids, top_parent_relids)) { /* OK, generate transformed child version */ Expr *child_expr; Relids new_relids; + Assert(!cur_em->em_is_const); + Assert(!cur_em->em_is_child); + Assert(!bms_is_empty(cur_em->em_relids)); + if (parent_rel->reloptkind == RELOPT_BASEREL) { /* Simple single-level transformation */ @@ -2680,7 +2872,7 @@ add_child_rel_equivalences(PlannerInfo *root, top_parent_relids); new_relids = bms_add_members(new_relids, child_relids); - (void) add_eq_member(cur_ec, child_expr, new_relids, + (void) add_eq_member(root, cur_ec, child_expr, new_relids, cur_em->em_jdomain, cur_em, cur_em->em_datatype); @@ -2688,6 +2880,7 @@ add_child_rel_equivalences(PlannerInfo *root, child_rel->eclass_indexes = bms_add_member(child_rel->eclass_indexes, i); } } + eclass_member_iterator_dispose(&iter); } } @@ -2732,7 +2925,8 @@ add_child_join_rel_equivalences(PlannerInfo *root, while ((i = bms_next_member(matching_ecs, i)) >= 0) { EquivalenceClass *cur_ec = (EquivalenceClass *) list_nth(root->eq_classes, i); - int num_members; + EquivalenceMemberIterator iter; + EquivalenceMember *cur_em; /* * If this EC contains a volatile expression, then generating child @@ -2746,24 +2940,21 @@ add_child_join_rel_equivalences(PlannerInfo *root, Assert(bms_overlap(top_parent_relids, cur_ec->ec_relids)); /* - * We don't use foreach() here because there's no point in scanning - * newly-added child members, so we can stop after the last - * pre-existing EC member. + * Looping using the EquivalenceMemberIterator means we only loop over + * the members which existed when we set up the iterator, not newly + * added ones. */ - num_members = list_length(cur_ec->ec_members); - for (int pos = 0; pos < num_members; pos++) + setup_eclass_member_iterator(&iter, root, cur_ec, top_parent_relids, + false, false); + + while ((cur_em = eclass_member_iterator_next(&iter)) != NULL) { - EquivalenceMember *cur_em = (EquivalenceMember *) list_nth(cur_ec->ec_members, pos); + Expr *child_expr; + Relids new_relids; - if (cur_em->em_is_const) - continue; /* ignore consts here */ - - /* - * We consider only original EC members here, not - * already-transformed child members. - */ - if (cur_em->em_is_child) - continue; /* ignore children here */ + Assert(!cur_em->em_is_const); + Assert(!cur_em->em_is_child); + Assert(bms_overlap(cur_em->em_relids, top_parent_relids)); /* * We may ignore expressions that reference a single baserel, @@ -2772,47 +2963,40 @@ add_child_join_rel_equivalences(PlannerInfo *root, if (bms_membership(cur_em->em_relids) != BMS_MULTIPLE) continue; - /* Does this member reference child's topmost parent rel? */ - if (bms_overlap(cur_em->em_relids, top_parent_relids)) + if (parent_joinrel->reloptkind == RELOPT_JOINREL) { - /* Yes, generate transformed child version */ - Expr *child_expr; - Relids new_relids; - - if (parent_joinrel->reloptkind == RELOPT_JOINREL) - { - /* Simple single-level transformation */ - child_expr = (Expr *) - adjust_appendrel_attrs(root, - (Node *) cur_em->em_expr, - nappinfos, appinfos); - } - else - { - /* Must do multi-level transformation */ - Assert(parent_joinrel->reloptkind == RELOPT_OTHER_JOINREL); - child_expr = (Expr *) - adjust_appendrel_attrs_multilevel(root, - (Node *) cur_em->em_expr, - child_joinrel, - child_joinrel->top_parent); - } + /* Simple single-level transformation */ + child_expr = (Expr *) + adjust_appendrel_attrs(root, + (Node *) cur_em->em_expr, + nappinfos, appinfos); + } + else + { + /* Must do multi-level transformation */ + Assert(parent_joinrel->reloptkind == RELOPT_OTHER_JOINREL); + child_expr = (Expr *) + adjust_appendrel_attrs_multilevel(root, + (Node *) cur_em->em_expr, + child_joinrel, + child_joinrel->top_parent); + } - /* - * Transform em_relids to match. Note we do *not* do - * pull_varnos(child_expr) here, as for example the - * transformation might have substituted a constant, but we - * don't want the child member to be marked as constant. - */ - new_relids = bms_difference(cur_em->em_relids, - top_parent_relids); - new_relids = bms_add_members(new_relids, child_relids); + /* + * Transform em_relids to match. Note we do *not* do + * pull_varnos(child_expr) here, as for example the + * transformation might have substituted a constant, but we + * don't want the child member to be marked as constant. + */ + new_relids = bms_difference(cur_em->em_relids, + top_parent_relids); + new_relids = bms_add_members(new_relids, child_relids); - (void) add_eq_member(cur_ec, child_expr, new_relids, - cur_em->em_jdomain, - cur_em, cur_em->em_datatype); - } + (void) add_eq_member(root, cur_ec, child_expr, new_relids, + cur_em->em_jdomain, + cur_em, cur_em->em_datatype); } + eclass_member_iterator_dispose(&iter); } MemoryContextSwitchTo(oldcontext); @@ -2871,7 +3055,8 @@ generate_implied_equalities_for_column(PlannerInfo *root, { EquivalenceClass *cur_ec = (EquivalenceClass *) list_nth(root->eq_classes, i); EquivalenceMember *cur_em; - ListCell *lc2; + EquivalenceMemberIterator iter; + int j; /* Sanity check eclass_indexes only contain ECs for rel */ Assert(is_child_rel || bms_is_subset(rel->relids, cur_ec->ec_relids)); @@ -2893,16 +3078,19 @@ generate_implied_equalities_for_column(PlannerInfo *root, * corner cases, so for now we live with just reporting the first * match. See also get_eclass_for_sort_expr.) */ - cur_em = NULL; - foreach(lc2, cur_ec->ec_members) + setup_eclass_member_iterator(&iter, root, cur_ec, rel->relids, true, + false); + + while ((cur_em = eclass_member_iterator_next(&iter)) != NULL) { - cur_em = (EquivalenceMember *) lfirst(lc2); if (bms_equal(cur_em->em_relids, rel->relids) && callback(root, rel, cur_ec, cur_em, callback_arg)) break; cur_em = NULL; } + eclass_member_iterator_dispose(&iter); + if (!cur_em) continue; @@ -2910,14 +3098,15 @@ generate_implied_equalities_for_column(PlannerInfo *root, * Found our match. Scan the other EC members and attempt to generate * joinclauses. */ - foreach(lc2, cur_ec->ec_members) + j = -1; + while ((j = bms_next_member(cur_ec->ec_nonchild_indexes, j)) >= 0) { - EquivalenceMember *other_em = (EquivalenceMember *) lfirst(lc2); + EquivalenceMember *other_em = list_nth_node(EquivalenceMember, + root->eq_members, j); Oid eq_op; RestrictInfo *rinfo; - if (other_em->em_is_child) - continue; /* ignore children here */ + Assert(!other_em->em_is_child); /* Make sure it'll be a join to a different rel */ if (other_em == cur_em || @@ -3014,7 +3203,7 @@ have_relevant_eclass_joinclause(PlannerInfo *root, * surely be true if both of them overlap ec_relids.) * * Note we don't test ec_broken; if we did, we'd need a separate code - * path to look through ec_sources. Checking the membership anyway is + * path to look through eq_sources. Checking the membership anyway is * OK as a possibly-overoptimistic heuristic. * * We don't test ec_has_const either, even though a const eclass won't @@ -3089,7 +3278,7 @@ eclass_useful_for_merging(PlannerInfo *root, RelOptInfo *rel) { Relids relids; - ListCell *lc; + int i; Assert(!eclass->ec_merged); @@ -3102,7 +3291,7 @@ eclass_useful_for_merging(PlannerInfo *root, /* * Note we don't test ec_broken; if we did, we'd need a separate code path - * to look through ec_sources. Checking the members anyway is OK as a + * to look through eq_sources. Checking the members anyway is OK as a * possibly-overoptimistic heuristic. */ @@ -3120,12 +3309,14 @@ eclass_useful_for_merging(PlannerInfo *root, return false; /* To join, we need a member not in the given rel */ - foreach(lc, eclass->ec_members) + i = -1; + while ((i = bms_next_member(eclass->ec_nonchild_indexes, i)) >= 0) { - EquivalenceMember *cur_em = (EquivalenceMember *) lfirst(lc); + EquivalenceMember *cur_em = + list_nth_node(EquivalenceMember, root->eq_members, i); - if (cur_em->em_is_child) - continue; /* ignore children here */ + /* we don't expect child members here */ + Assert(!cur_em->em_is_child); if (!bms_overlap(cur_em->em_relids, relids)) return true; @@ -3213,7 +3404,7 @@ get_eclass_indexes_for_relids(PlannerInfo *root, Relids relids) /* Should be OK to rely on eclass_indexes */ Assert(root->ec_merging_done); - while ((i = bms_next_member(relids, i)) > 0) + while ((i = bms_next_member(relids, i)) >= 0) { RelOptInfo *rel = root->simple_rel_array[i]; @@ -3255,3 +3446,595 @@ get_common_eclass_indexes(PlannerInfo *root, Relids relids1, Relids relids2) /* Calculate and return the common EC indexes, recycling the left input. */ return bms_int_members(rel1ecs, rel2ecs); } + +/* + * get_ecmember_indexes + * Returns a Bitmapset with indexes into root->eq_members for all + * EquivalenceMembers in 'ec' that have + * bms_overlap(em->em_relids, relids) as true. The returned indexes + * may reference EquivalenceMembers with em_relids containing members + * not mentioned in 'relids'. + * + * Returns a newly allocated Bitmapset which the caller is free to modify. + */ +Bitmapset * +get_ecmember_indexes(PlannerInfo *root, EquivalenceClass *ec, Relids relids, + bool with_children, bool with_norel_members) +{ + Bitmapset *matching_ems; + Bitmapset *rel_ems = NULL; + int i = -1; + + while ((i = bms_next_member(relids, i)) >= 0) + { + RelOptInfo *rel = root->simple_rel_array[i]; + + rel_ems = bms_add_members(rel_ems, rel->eclass_member_indexes); + } + +#ifdef USE_ASSERT_CHECKING + /* verify the results look sane */ + i = -1; + while ((i = bms_next_member(rel_ems, i)) >= 0) + { + EquivalenceMember *em = list_nth_node(EquivalenceMember, + root->eq_members, i); + + Assert(bms_overlap(em->em_relids, relids)); + } +#endif + + /* remove EC members not mentioning the required rels. */ + if (!with_children) + matching_ems = bms_int_members(rel_ems, ec->ec_nonchild_indexes); + else + matching_ems = bms_int_members(rel_ems, ec->ec_member_indexes); + + /* now add any members with that are not mentioned by any relation */ + if (with_norel_members) + matching_ems = bms_add_members(matching_ems, ec->ec_norel_indexes); + + return matching_ems; +} + +/* + * get_ecmember_indexes_strict + * Returns a Bitmapset with indexes into root->eq_members for all + * EquivalenceMembers in 'ec' that have + * bms_is_subset(relids, em->em_relids) as true. + * + * Returns a newly allocated Bitmapset which the caller is free to modify. + */ +Bitmapset * +get_ecmember_indexes_strict(PlannerInfo *root, EquivalenceClass *ec, + Relids relids, bool with_children, + bool with_norel_members) +{ + Bitmapset *matching_ems = NULL; + int i = bms_next_member(relids, -1); + + if (i >= 0) + { + RelOptInfo *rel = root->simple_rel_array[i]; + + /* + * bms_intersect to the first relation to try to keep the resulting + * Bitmapset as small as possible. This saves having to make a + * complete bms_copy() of one of them. One may contain significantly + * more words than the other. + */ + if (!with_children) + matching_ems = bms_intersect(rel->eclass_member_indexes, + ec->ec_nonchild_indexes); + else + matching_ems = bms_intersect(rel->eclass_member_indexes, + ec->ec_member_indexes); + + while ((i = bms_next_member(relids, i)) >= 0) + { + rel = root->simple_rel_array[i]; + matching_ems = bms_int_members(matching_ems, + rel->eclass_member_indexes); + } + } + +#ifdef USE_ASSERT_CHECKING + /* verify the results look sane */ + i = -1; + while ((i = bms_next_member(matching_ems, i)) >= 0) + { + EquivalenceMember *em = list_nth_node(EquivalenceMember, + root->eq_members, i); + + Assert(bms_is_subset(relids, em->em_relids)); + } +#endif + + /* optionally add members with that are not mentioned by any relation */ + if (with_norel_members) + matching_ems = bms_add_members(matching_ems, ec->ec_norel_indexes); + + return matching_ems; +} + +/* + * The threshold for the number of members that must be in an EquivalenceClass + * before we switch to searching for EquivalenceMember by using the Bitmapset + * indexes stored in EquivalenceClass and RelOptInfo. We don't want to make + * this too low as the manipulation of Bitmapsets slows this down for + * EquivalenceClasses with just a few members. The linear search becomes very + * slow when an EquivalenceClass has a large number of members, as can happen + * when planning queries to partitioned tables. + */ +#define ECMEMBER_INDEX_THRESHOLD 16 + +/* + * setup_eclass_member_iterator + * Setup 'iter' so it's ready for eclass_member_iterator_next to start + * searching for EquivalenceMembers matching the specified parameters. + * + * Once used, the iterator must be disposed of using + * eclass_member_iterator_dispose. + */ +void +setup_eclass_member_iterator(EquivalenceMemberIterator *iter, + PlannerInfo *root, EquivalenceClass *ec, + Relids relids, bool with_children, + bool with_norel_members) +{ + iter->orig_length = list_length(ec->ec_members); + iter->use_index = iter->orig_length >= ECMEMBER_INDEX_THRESHOLD; + iter->with_children = with_children; + iter->with_norel_members = with_norel_members; + iter->relids_empty = bms_is_empty(relids); +#ifdef USE_ASSERT_CHECKING + iter->isstrict = false; +#endif + iter->with_relids = relids; + iter->root = root; + iter->eclass = ec; + +#ifdef USE_ASSERT_CHECKING + if (1) +#else + if (iter->use_index) +#endif + iter->matching_ems = get_ecmember_indexes(root, ec, relids, + with_children, + with_norel_members); + else + iter->matching_ems = NULL; + + iter->current_index = -1; + +#ifdef USE_ASSERT_CHECKING + + /* + * Verify that an iterator that uses the index and one that does not both + * return the same EquivalenceMembers + */ + { + EquivalenceMemberIterator idx_iter; + EquivalenceMemberIterator noidx_iter; + EquivalenceMember *em; + List *list1 = NIL; + List *list2 = NIL; + ListCell *lc1, + *lc2; + + memcpy(&idx_iter, iter, sizeof(EquivalenceMemberIterator)); + memcpy(&noidx_iter, iter, sizeof(EquivalenceMemberIterator)); + + idx_iter.use_index = true; + noidx_iter.use_index = false; + + while ((em = eclass_member_iterator_next(&idx_iter)) != NULL) + list1 = lappend(list1, em); + + while ((em = eclass_member_iterator_next(&noidx_iter)) != NULL) + list2 = lappend(list2, em); + + list_sort(list1, list_ptr_cmp); + list_sort(list2, list_ptr_cmp); + + Assert(list_length(list1) == list_length(list2)); + + forboth(lc1, list1, lc2, list2) + Assert(lfirst(lc1) == lfirst(lc2)); + } +#endif + +} + +/* + * setup_eclass_member_strict_iterator + * Setup 'iter' so it's ready for eclass_member_iterator_strict_next to + * start searching for EquivalenceMembers matching the specified + * parameters. + * + * Once used, the iterator must be disposed of using + * eclass_member_iterator_dispose. + */ +void +setup_eclass_member_strict_iterator(EquivalenceMemberIterator *iter, + PlannerInfo *root, EquivalenceClass *ec, + Relids relids, bool with_children, + bool with_norel_members) +{ + iter->orig_length = list_length(ec->ec_members); + iter->use_index = iter->orig_length >= ECMEMBER_INDEX_THRESHOLD; + iter->with_children = with_children; + iter->with_norel_members = with_norel_members; + iter->relids_empty = bms_is_empty(relids); +#ifdef USE_ASSERT_CHECKING + iter->isstrict = true; +#endif + iter->with_relids = relids; + iter->root = root; + iter->eclass = ec; + +#ifdef USE_ASSERT_CHECKING + if (1) +#else + if (iter->use_index) +#endif + iter->matching_ems = get_ecmember_indexes_strict(root, ec, relids, + with_children, + with_norel_members); + else + iter->matching_ems = NULL; + + iter->current_index = -1; + +#ifdef USE_ASSERT_CHECKING + + /* + * Verify that an iterator that uses the index and one that does not both + * return the same EquivalenceMembers + */ + { + EquivalenceMemberIterator idx_iter; + EquivalenceMemberIterator noidx_iter; + EquivalenceMember *em; + List *list1 = NIL; + List *list2 = NIL; + ListCell *lc1, + *lc2; + + memcpy(&idx_iter, iter, sizeof(EquivalenceMemberIterator)); + memcpy(&noidx_iter, iter, sizeof(EquivalenceMemberIterator)); + + idx_iter.use_index = true; + noidx_iter.use_index = false; + + while ((em = eclass_member_iterator_strict_next(&idx_iter)) != NULL) + list1 = lappend(list1, em); + + while ((em = eclass_member_iterator_strict_next(&noidx_iter)) != NULL) + list2 = lappend(list2, em); + + list_sort(list1, list_ptr_cmp); + list_sort(list2, list_ptr_cmp); + + Assert(list_length(list1) == list_length(list2)); + + forboth(lc1, list1, lc2, list2) + Assert(lfirst(lc1) == lfirst(lc2)); + } +#endif +} + +/* + * eclass_member_iterator_next + * Fetch the next EquivalenceMember from an EquivalenceMemberIterator + * which was set up by setup_eclass_member_iterator(). Returns NULL when + * there are no more matching EquivalenceMembers. + */ +EquivalenceMember * +eclass_member_iterator_next(EquivalenceMemberIterator *iter) +{ + /* Fail if this was used instead of eclass_member_iterator_strict_next */ + Assert(!iter->isstrict); + + if (iter->use_index) + { + iter->current_index = bms_next_member(iter->matching_ems, + iter->current_index); + if (iter->current_index >= 0) + return list_nth_node(EquivalenceMember, iter->root->eq_members, + iter->current_index); + return NULL; + } + else + { + ListCell *lc; + + for_each_from(lc, iter->eclass->ec_members, iter->current_index + 1) + { + EquivalenceMember *em = lfirst_node(EquivalenceMember, lc); + + iter->current_index = foreach_current_index(lc); + + /* + * Some users of this iterator will be adding new + * EquivalenceMember during the loop. We must ensure we don't + * return those, so here we return NULL when the loop index goes + * beyond the original length of the ec_members list. + */ + if (iter->current_index >= iter->orig_length) + return NULL; + + /* don't return child members when with_children==false */ + if (!iter->with_children && em->em_is_child) + continue; + + /* + * When with_norel_members==true, make sure we return all members + * without Vars. + */ + if (iter->with_norel_members && em->em_norel_expr) + return em; + + /* + * Don't return members which have no common rels with with_relids + */ + if (!bms_overlap(em->em_relids, iter->with_relids)) + continue; + + return em; + } + return NULL; + } +} + +/* + * eclass_member_iterator_strict_next + * Fetch the next EquivalenceMember from an EquivalenceMemberIterator + * which was set up by setup_eclass_member_strict_iterator(). Returns + * NULL when there are no more matching EquivalenceMembers. + */ +EquivalenceMember * +eclass_member_iterator_strict_next(EquivalenceMemberIterator *iter) +{ + /* Fail if this was used instead of eclass_member_iterator_next */ + Assert(iter->isstrict); + + if (iter->use_index) + { + iter->current_index = bms_next_member(iter->matching_ems, + iter->current_index); + if (iter->current_index >= 0) + return list_nth_node(EquivalenceMember, iter->root->eq_members, + iter->current_index); + return NULL; + } + else + { + ListCell *lc; + + for_each_from(lc, iter->eclass->ec_members, iter->current_index + 1) + { + EquivalenceMember *em = lfirst_node(EquivalenceMember, lc); + + iter->current_index = foreach_current_index(lc); + + /* + * Some users of this iterator will be adding new + * EquivalenceMember during the loop. We must ensure we don't + * return those, so here we return NULL when the loop index goes + * beyond the original length of the ec_members list. + */ + if (iter->current_index >= iter->orig_length) + return NULL; + + /* don't return child members when with_children==false */ + if (!iter->with_children && em->em_is_child) + continue; + + /* + * When with_norel_members==true, make sure we return all members + * without Vars. + */ + if (iter->with_norel_members && em->em_norel_expr) + return em; + + /* + * Don't match members where em_relids that don't contain all rels + * mentioned in with_relids. + */ + if (iter->relids_empty || + !bms_is_subset(iter->with_relids, em->em_relids)) + continue; + + return em; + } + return NULL; + } +} + +/* + * eclass_member_iterator_dispose + * Free any memory allocated by the iterator + */ +void +eclass_member_iterator_dispose(EquivalenceMemberIterator *iter) +{ + bms_free(iter->matching_ems); +} + + +/* + * get_ec_source_indexes + * Returns a Bitmapset with indexes into root->eq_sources for all + * RestrictInfos in 'ec' that have + * bms_overlap(relids, rinfo->clause_relids) as true. + * + * Returns a newly allocated Bitmapset which the caller is free to modify. + */ +Bitmapset * +get_ec_source_indexes(PlannerInfo *root, EquivalenceClass *ec, Relids relids) +{ + Bitmapset *rel_esis = NULL; + int i = -1; + + while ((i = bms_next_member(relids, i)) >= 0) + { + RelOptInfo *rel = root->simple_rel_array[i]; + + rel_esis = bms_add_members(rel_esis, rel->eclass_source_indexes); + } + +#ifdef USE_ASSERT_CHECKING + /* verify the results look sane */ + i = -1; + while ((i = bms_next_member(rel_esis, i)) >= 0) + { + RestrictInfo *rinfo = list_nth_node(RestrictInfo, root->eq_sources, + i); + + Assert(bms_overlap(relids, rinfo->clause_relids)); + } +#endif + + /* bitwise-AND to leave only the ones for this EquivalenceClass */ + return bms_int_members(rel_esis, ec->ec_source_indexes); +} + +/* + * get_ec_source_indexes_strict + * Returns a Bitmapset with indexes into root->eq_sources for all + * RestrictInfos in 'ec' that have + * bms_is_subset(relids, rinfo->clause_relids) as true. + * + * Returns a newly allocated Bitmapset which the caller is free to modify. + */ +Bitmapset * +get_ec_source_indexes_strict(PlannerInfo *root, EquivalenceClass *ec, + Relids relids) +{ + Bitmapset *esis = NULL; + int i = bms_next_member(relids, -1); + + if (i >= 0) + { + RelOptInfo *rel = root->simple_rel_array[i]; + + /* + * bms_intersect to the first relation to try to keep the resulting + * Bitmapset as small as possible. This saves having to make a + * complete bms_copy() of one of them. One may contain significantly + * more words than the other. + */ + esis = bms_intersect(ec->ec_source_indexes, + rel->eclass_source_indexes); + + while ((i = bms_next_member(relids, i)) >= 0) + { + rel = root->simple_rel_array[i]; + esis = bms_int_members(esis, rel->eclass_source_indexes); + } + } + +#ifdef USE_ASSERT_CHECKING + /* verify the results look sane */ + i = -1; + while ((i = bms_next_member(esis, i)) >= 0) + { + RestrictInfo *rinfo = list_nth_node(RestrictInfo, root->eq_sources, + i); + + Assert(bms_is_subset(relids, rinfo->clause_relids)); + } +#endif + + return esis; +} + +/* + * get_ec_derive_indexes + * Returns a Bitmapset with indexes into root->eq_derives for all + * RestrictInfos in 'ec' that have + * bms_overlap(relids, rinfo->clause_relids) as true. + * + * Returns a newly allocated Bitmapset which the caller is free to modify. + * + * XXX is this function even needed? + */ +Bitmapset * +get_ec_derive_indexes(PlannerInfo *root, EquivalenceClass *ec, Relids relids) +{ + Bitmapset *rel_edis = NULL; + int i = -1; + + while ((i = bms_next_member(relids, i)) >= 0) + { + RelOptInfo *rel = root->simple_rel_array[i]; + + rel_edis = bms_add_members(rel_edis, rel->eclass_derive_indexes); + } + +#ifdef USE_ASSERT_CHECKING + /* verify the results look sane */ + i = -1; + while ((i = bms_next_member(rel_edis, i)) >= 0) + { + RestrictInfo *rinfo = list_nth_node(RestrictInfo, root->eq_derives, + i); + + Assert(bms_overlap(relids, rinfo->clause_relids)); + } +#endif + + /* bitwise-AND to leave only the ones for this EquivalenceClass */ + return bms_int_members(rel_edis, ec->ec_derive_indexes); +} + +/* + * get_ec_derive_indexes_strict + * Returns a Bitmapset with indexes into root->eq_derives for all + * RestrictInfos in 'ec' that have + * bms_is_subset(relids, rinfo->clause_relids) as true. + * + * Returns a newly allocated Bitmapset which the caller is free to modify. + */ +Bitmapset * +get_ec_derive_indexes_strict(PlannerInfo *root, EquivalenceClass *ec, + Relids relids) +{ + Bitmapset *edis = NULL; + int i = bms_next_member(relids, -1); + + if (i >= 0) + { + RelOptInfo *rel = root->simple_rel_array[i]; + + /* + * bms_intersect to the first relation to try to keep the resulting + * Bitmapset as small as possible. This saves having to make a + * complete bms_copy() of one of them. One may contain significantly + * more words than the other. + */ + edis = bms_intersect(ec->ec_derive_indexes, + rel->eclass_derive_indexes); + + while ((i = bms_next_member(relids, i)) >= 0) + { + rel = root->simple_rel_array[i]; + edis = bms_int_members(edis, rel->eclass_derive_indexes); + } + } + +#ifdef USE_ASSERT_CHECKING + /* verify the results look sane */ + i = -1; + while ((i = bms_next_member(edis, i)) >= 0) + { + RestrictInfo *rinfo = list_nth_node(RestrictInfo, root->eq_derives, + i); + + Assert(bms_is_subset(relids, rinfo->clause_relids)); + } +#endif + + return edis; +} diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c index 721a075201..5ebf944cb1 100644 --- a/src/backend/optimizer/path/indxpath.c +++ b/src/backend/optimizer/path/indxpath.c @@ -184,8 +184,8 @@ static IndexClause *expand_indexqual_rowcompare(PlannerInfo *root, IndexOptInfo *index, Oid expr_op, bool var_on_left); -static void match_pathkeys_to_index(IndexOptInfo *index, List *pathkeys, - List **orderby_clauses_p, +static void match_pathkeys_to_index(PlannerInfo *root, IndexOptInfo *index, + List *pathkeys, List **orderby_clauses_p, List **clause_columns_p); static Expr *match_clause_to_ordering_op(IndexOptInfo *index, int indexcol, Expr *clause, Oid pk_opfamily); @@ -978,7 +978,7 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel, else if (index->amcanorderbyop && pathkeys_possibly_useful) { /* see if we can generate ordering operators for query_pathkeys */ - match_pathkeys_to_index(index, root->query_pathkeys, + match_pathkeys_to_index(root, index, root->query_pathkeys, &orderbyclauses, &orderbyclausecols); if (orderbyclauses) @@ -3066,8 +3066,8 @@ expand_indexqual_rowcompare(PlannerInfo *root, * one-to-one with the requested pathkeys. */ static void -match_pathkeys_to_index(IndexOptInfo *index, List *pathkeys, - List **orderby_clauses_p, +match_pathkeys_to_index(PlannerInfo *root, IndexOptInfo *index, + List *pathkeys, List **orderby_clauses_p, List **clause_columns_p) { List *orderby_clauses = NIL; @@ -3084,8 +3084,9 @@ match_pathkeys_to_index(IndexOptInfo *index, List *pathkeys, foreach(lc1, pathkeys) { PathKey *pathkey = (PathKey *) lfirst(lc1); + Bitmapset *matching_ems; bool found = false; - ListCell *lc2; + int i; /* * Note: for any failure to match, we just return NIL immediately. @@ -3109,9 +3110,14 @@ match_pathkeys_to_index(IndexOptInfo *index, List *pathkeys, * be considered to match more than one pathkey list, which is OK * here. See also get_eclass_for_sort_expr.) */ - foreach(lc2, pathkey->pk_eclass->ec_members) + matching_ems = bms_intersect(pathkey->pk_eclass->ec_member_indexes, + index->rel->eclass_member_indexes); + + i = -1; + while ((i = bms_next_member(matching_ems, i)) >= 0) { - EquivalenceMember *member = (EquivalenceMember *) lfirst(lc2); + EquivalenceMember *member = list_nth_node(EquivalenceMember, + root->eq_members, i); int indexcol; /* No possibility of match if it references other relations */ diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index c4e7f97f68..0596532db4 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -940,18 +940,19 @@ convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel, * outer query. */ int best_score = -1; - ListCell *j; + int j = -1; - foreach(j, sub_eclass->ec_members) + while ((j = bms_next_member(sub_eclass->ec_nonchild_indexes, j)) >= 0) { - EquivalenceMember *sub_member = (EquivalenceMember *) lfirst(j); + EquivalenceMember *sub_member = list_nth_node(EquivalenceMember, + rel->subroot->eq_members, + j); Expr *sub_expr = sub_member->em_expr; Oid sub_expr_type = sub_member->em_datatype; Oid sub_expr_coll = sub_eclass->ec_collation; ListCell *k; - if (sub_member->em_is_child) - continue; /* ignore children here */ + Assert(!sub_member->em_is_child); foreach(k, subquery_tlist) { @@ -1450,7 +1451,7 @@ select_outer_pathkeys_for_merge(PlannerInfo *root, RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); EquivalenceClass *oeclass; int score; - ListCell *lc2; + int i; /* get the outer eclass */ update_mergeclause_eclasses(root, rinfo); @@ -1471,13 +1472,16 @@ select_outer_pathkeys_for_merge(PlannerInfo *root, /* compute score */ score = 0; - foreach(lc2, oeclass->ec_members) + i = -1; + while ((i = bms_next_member(oeclass->ec_nonchild_indexes, i)) >= 0) { - EquivalenceMember *em = (EquivalenceMember *) lfirst(lc2); + EquivalenceMember *em = list_nth_node(EquivalenceMember, + root->eq_members, i); - /* Potential future join partner? */ - if (!em->em_is_const && !em->em_is_child && - !bms_overlap(em->em_relids, joinrel->relids)) + /* shouldn't be consts or child members in ec_nonchild_indexes */ + Assert(!em->em_is_const && !em->em_is_child); + + if (!bms_overlap(em->em_relids, joinrel->relids)) score++; } diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c index 134130476e..ee4f3a9cee 100644 --- a/src/backend/optimizer/plan/createplan.c +++ b/src/backend/optimizer/plan/createplan.c @@ -260,8 +260,8 @@ static IncrementalSort *make_incrementalsort(Plan *lefttree, int numCols, int nPresortedCols, AttrNumber *sortColIdx, Oid *sortOperators, Oid *collations, bool *nullsFirst); -static Plan *prepare_sort_from_pathkeys(Plan *lefttree, List *pathkeys, - Relids relids, +static Plan *prepare_sort_from_pathkeys(PlannerInfo *root, Plan *lefttree, + List *pathkeys, Relids relids, const AttrNumber *reqColIdx, bool adjust_tlist_in_place, int *p_numsortkeys, @@ -269,10 +269,13 @@ static Plan *prepare_sort_from_pathkeys(Plan *lefttree, List *pathkeys, Oid **p_sortOperators, Oid **p_collations, bool **p_nullsFirst); -static Sort *make_sort_from_pathkeys(Plan *lefttree, List *pathkeys, - Relids relids); -static IncrementalSort *make_incrementalsort_from_pathkeys(Plan *lefttree, - List *pathkeys, Relids relids, int nPresortedCols); +static Sort *make_sort_from_pathkeys(PlannerInfo *root, Plan *lefttree, + List *pathkeys, Relids relids); +static IncrementalSort *make_incrementalsort_from_pathkeys(PlannerInfo *root, + Plan *lefttree, + List *pathkeys, + Relids relids, + int nPresortedCols); static Sort *make_sort_from_groupcols(List *groupcls, AttrNumber *grpColIdx, Plan *lefttree); @@ -293,7 +296,7 @@ static Group *make_group(List *tlist, List *qual, int numGroupCols, AttrNumber *grpColIdx, Oid *grpOperators, Oid *grpCollations, Plan *lefttree); static Unique *make_unique_from_sortclauses(Plan *lefttree, List *distinctList); -static Unique *make_unique_from_pathkeys(Plan *lefttree, +static Unique *make_unique_from_pathkeys(PlannerInfo *root, Plan *lefttree, List *pathkeys, int numCols); static Gather *make_gather(List *qptlist, List *qpqual, int nworkers, int rescan_param, bool single_copy, Plan *subplan); @@ -1260,7 +1263,8 @@ create_append_plan(PlannerInfo *root, AppendPath *best_path, int flags) * function result; it must be the same plan node. However, we then * need to detect whether any tlist entries were added. */ - (void) prepare_sort_from_pathkeys((Plan *) plan, pathkeys, + (void) prepare_sort_from_pathkeys(root, + (Plan *) plan, pathkeys, best_path->path.parent->relids, NULL, true, @@ -1304,7 +1308,7 @@ create_append_plan(PlannerInfo *root, AppendPath *best_path, int flags) * don't need an explicit sort, to make sure they are returning * the same sort key columns the Append expects. */ - subplan = prepare_sort_from_pathkeys(subplan, pathkeys, + subplan = prepare_sort_from_pathkeys(root, subplan, pathkeys, subpath->parent->relids, nodeSortColIdx, false, @@ -1445,7 +1449,7 @@ create_merge_append_plan(PlannerInfo *root, MergeAppendPath *best_path, * function result; it must be the same plan node. However, we then need * to detect whether any tlist entries were added. */ - (void) prepare_sort_from_pathkeys(plan, pathkeys, + (void) prepare_sort_from_pathkeys(root, plan, pathkeys, best_path->path.parent->relids, NULL, true, @@ -1476,7 +1480,7 @@ create_merge_append_plan(PlannerInfo *root, MergeAppendPath *best_path, subplan = create_plan_recurse(root, subpath, CP_EXACT_TLIST); /* Compute sort column info, and adjust subplan's tlist as needed */ - subplan = prepare_sort_from_pathkeys(subplan, pathkeys, + subplan = prepare_sort_from_pathkeys(root, subplan, pathkeys, subpath->parent->relids, node->sortColIdx, false, @@ -1966,7 +1970,7 @@ create_gather_merge_plan(PlannerInfo *root, GatherMergePath *best_path) Assert(pathkeys != NIL); /* Compute sort column info, and adjust subplan's tlist as needed */ - subplan = prepare_sort_from_pathkeys(subplan, pathkeys, + subplan = prepare_sort_from_pathkeys(root, subplan, pathkeys, best_path->subpath->parent->relids, gm_plan->sortColIdx, false, @@ -2185,7 +2189,7 @@ create_sort_plan(PlannerInfo *root, SortPath *best_path, int flags) * relids. Thus, if this sort path is based on a child relation, we must * pass its relids. */ - plan = make_sort_from_pathkeys(subplan, best_path->path.pathkeys, + plan = make_sort_from_pathkeys(root, subplan, best_path->path.pathkeys, IS_OTHER_REL(best_path->subpath->parent) ? best_path->path.parent->relids : NULL); @@ -2209,7 +2213,7 @@ create_incrementalsort_plan(PlannerInfo *root, IncrementalSortPath *best_path, /* See comments in create_sort_plan() above */ subplan = create_plan_recurse(root, best_path->spath.subpath, flags | CP_SMALL_TLIST); - plan = make_incrementalsort_from_pathkeys(subplan, + plan = make_incrementalsort_from_pathkeys(root, subplan, best_path->spath.path.pathkeys, IS_OTHER_REL(best_path->spath.subpath->parent) ? best_path->spath.path.parent->relids : NULL, @@ -2278,7 +2282,7 @@ create_upper_unique_plan(PlannerInfo *root, UpperUniquePath *best_path, int flag subplan = create_plan_recurse(root, best_path->subpath, flags | CP_LABEL_TLIST); - plan = make_unique_from_pathkeys(subplan, + plan = make_unique_from_pathkeys(root, subplan, best_path->path.pathkeys, best_path->numkeys); @@ -4494,7 +4498,7 @@ create_mergejoin_plan(PlannerInfo *root, if (best_path->outersortkeys) { Relids outer_relids = outer_path->parent->relids; - Sort *sort = make_sort_from_pathkeys(outer_plan, + Sort *sort = make_sort_from_pathkeys(root, outer_plan, best_path->outersortkeys, outer_relids); @@ -4508,7 +4512,7 @@ create_mergejoin_plan(PlannerInfo *root, if (best_path->innersortkeys) { Relids inner_relids = inner_path->parent->relids; - Sort *sort = make_sort_from_pathkeys(inner_plan, + Sort *sort = make_sort_from_pathkeys(root, inner_plan, best_path->innersortkeys, inner_relids); @@ -6129,7 +6133,8 @@ make_incrementalsort(Plan *lefttree, int numCols, int nPresortedCols, * or a Result stacked atop lefttree). */ static Plan * -prepare_sort_from_pathkeys(Plan *lefttree, List *pathkeys, +prepare_sort_from_pathkeys(PlannerInfo *root, + Plan *lefttree, List *pathkeys, Relids relids, const AttrNumber *reqColIdx, bool adjust_tlist_in_place, @@ -6196,7 +6201,7 @@ prepare_sort_from_pathkeys(Plan *lefttree, List *pathkeys, tle = get_tle_by_resno(tlist, reqColIdx[numsortkeys]); if (tle) { - em = find_ec_member_matching_expr(ec, tle->expr, relids); + em = find_ec_member_matching_expr(root, ec, tle->expr, relids); if (em) { /* found expr at right place in tlist */ @@ -6224,7 +6229,7 @@ prepare_sort_from_pathkeys(Plan *lefttree, List *pathkeys, foreach(j, tlist) { tle = (TargetEntry *) lfirst(j); - em = find_ec_member_matching_expr(ec, tle->expr, relids); + em = find_ec_member_matching_expr(root, ec, tle->expr, relids); if (em) { /* found expr already in tlist */ @@ -6240,7 +6245,7 @@ prepare_sort_from_pathkeys(Plan *lefttree, List *pathkeys, /* * No matching tlist item; look for a computable expression. */ - em = find_computable_ec_member(NULL, ec, tlist, relids, false); + em = find_computable_ec_member(root, ec, tlist, relids, false); if (!em) elog(ERROR, "could not find pathkey item to sort"); pk_datatype = em->em_datatype; @@ -6311,7 +6316,8 @@ prepare_sort_from_pathkeys(Plan *lefttree, List *pathkeys, * 'relids' is the set of relations required by prepare_sort_from_pathkeys() */ static Sort * -make_sort_from_pathkeys(Plan *lefttree, List *pathkeys, Relids relids) +make_sort_from_pathkeys(PlannerInfo *root, Plan *lefttree, List *pathkeys, + Relids relids) { int numsortkeys; AttrNumber *sortColIdx; @@ -6320,7 +6326,9 @@ make_sort_from_pathkeys(Plan *lefttree, List *pathkeys, Relids relids) bool *nullsFirst; /* Compute sort column info, and adjust lefttree as needed */ - lefttree = prepare_sort_from_pathkeys(lefttree, pathkeys, + lefttree = prepare_sort_from_pathkeys(root, + lefttree, + pathkeys, relids, NULL, false, @@ -6346,7 +6354,8 @@ make_sort_from_pathkeys(Plan *lefttree, List *pathkeys, Relids relids) * 'nPresortedCols' is the number of presorted columns in input tuples */ static IncrementalSort * -make_incrementalsort_from_pathkeys(Plan *lefttree, List *pathkeys, +make_incrementalsort_from_pathkeys(PlannerInfo *root, + Plan *lefttree, List *pathkeys, Relids relids, int nPresortedCols) { int numsortkeys; @@ -6356,7 +6365,9 @@ make_incrementalsort_from_pathkeys(Plan *lefttree, List *pathkeys, bool *nullsFirst; /* Compute sort column info, and adjust lefttree as needed */ - lefttree = prepare_sort_from_pathkeys(lefttree, pathkeys, + lefttree = prepare_sort_from_pathkeys(root, + lefttree, + pathkeys, relids, NULL, false, @@ -6706,7 +6717,8 @@ make_unique_from_sortclauses(Plan *lefttree, List *distinctList) * as above, but use pathkeys to identify the sort columns and semantics */ static Unique * -make_unique_from_pathkeys(Plan *lefttree, List *pathkeys, int numCols) +make_unique_from_pathkeys(PlannerInfo *root, Plan *lefttree, List *pathkeys, + int numCols) { Unique *node = makeNode(Unique); Plan *plan = &node->plan; @@ -6769,7 +6781,7 @@ make_unique_from_pathkeys(Plan *lefttree, List *pathkeys, int numCols) foreach(j, plan->targetlist) { tle = (TargetEntry *) lfirst(j); - em = find_ec_member_matching_expr(ec, tle->expr, NULL); + em = find_ec_member_matching_expr(root, ec, tle->expr, NULL); if (em) { /* found expr already in tlist */ diff --git a/src/backend/optimizer/plan/planagg.c b/src/backend/optimizer/plan/planagg.c index d168620665..9f0fa42045 100644 --- a/src/backend/optimizer/plan/planagg.c +++ b/src/backend/optimizer/plan/planagg.c @@ -356,6 +356,7 @@ build_minmax_path(PlannerInfo *root, MinMaxAggInfo *mminfo, Assert(subroot->join_info_list == NIL); /* and we haven't made equivalence classes, either */ Assert(subroot->eq_classes == NIL); + Assert(root->eq_members == NIL); /* and we haven't created PlaceHolderInfos, either */ Assert(subroot->placeholder_list == NIL); diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index db5ff6fdca..c27ee6fab1 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -627,6 +627,9 @@ subquery_planner(PlannerGlobal *glob, Query *parse, root->multiexpr_params = NIL; root->join_domains = NIL; root->eq_classes = NIL; + root->eq_members = NIL; + root->eq_sources = NIL; + root->eq_derives = NIL; root->ec_merging_done = false; root->last_rinfo_serial = 0; root->all_result_relids = diff --git a/src/backend/optimizer/prep/prepjointree.c b/src/backend/optimizer/prep/prepjointree.c index 870d84b29d..804ede97b1 100644 --- a/src/backend/optimizer/prep/prepjointree.c +++ b/src/backend/optimizer/prep/prepjointree.c @@ -993,6 +993,9 @@ pull_up_simple_subquery(PlannerInfo *root, Node *jtnode, RangeTblEntry *rte, subroot->multiexpr_params = NIL; subroot->join_domains = NIL; subroot->eq_classes = NIL; + subroot->eq_members = NIL; + subroot->eq_sources = NIL; + subroot->eq_derives = NIL; subroot->ec_merging_done = false; subroot->last_rinfo_serial = 0; subroot->all_result_relids = NULL; diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c index 0c68ec011b..4c7d66311f 100644 --- a/src/backend/optimizer/prep/prepunion.c +++ b/src/backend/optimizer/prep/prepunion.c @@ -45,7 +45,7 @@ #include "utils/selfuncs.h" #include "utils/syscache.h" - +static void setup_append_rel_entry(PlannerInfo *root); static RelOptInfo *recurse_set_operations(Node *setOp, PlannerInfo *root, List *colTypes, List *colCollations, bool junkOK, @@ -128,6 +128,7 @@ plan_set_operations(PlannerInfo *root) * so that pathkeys.c won't complain. */ Assert(root->eq_classes == NIL); + Assert(root->eq_members == NIL); root->ec_merging_done = true; /* @@ -149,6 +150,8 @@ plan_set_operations(PlannerInfo *root) leftmostQuery = leftmostRTE->subquery; Assert(leftmostQuery != NULL); + setup_append_rel_entry(root); + /* * If the topmost node is a recursive union, it needs special processing. */ @@ -180,6 +183,26 @@ plan_set_operations(PlannerInfo *root) return setop_rel; } +/* + * setup_append_rel_entry + * Add entry into root's simple_rel_array at element 0. This is required + * because generate_append_tlist() makes Vars with varno=0. In + * add_eq_member() we need to index EquivalenceMembers belonging to this + * relation. + */ +static void +setup_append_rel_entry(PlannerInfo *root) +{ + RelOptInfo *rel = makeNode(RelOptInfo); + + memset(rel, 0, sizeof(RelOptInfo)); + rel->eclass_member_indexes = NULL; + + /* make sure nothing else has made use of this element */ + Assert(root->simple_rel_array[0] == NULL); + root->simple_rel_array[0] = rel; +} + /* * recurse_set_operations * Recursively handle one step in a tree of set operations diff --git a/src/backend/optimizer/util/relnode.c b/src/backend/optimizer/util/relnode.c index ad84cc43e1..dfa374a8cf 100644 --- a/src/backend/optimizer/util/relnode.c +++ b/src/backend/optimizer/util/relnode.c @@ -226,6 +226,9 @@ build_simple_rel(PlannerInfo *root, int relid, RelOptInfo *parent) rel->tuples = 0; rel->allvisfrac = 0; rel->eclass_indexes = NULL; + rel->eclass_member_indexes = NULL; + rel->eclass_source_indexes = NULL; + rel->eclass_derive_indexes = NULL; rel->subroot = NULL; rel->subplan_params = NIL; rel->rel_parallel_workers = -1; /* set up in get_relation_info */ @@ -701,6 +704,9 @@ build_join_rel(PlannerInfo *root, joinrel->tuples = 0; joinrel->allvisfrac = 0; joinrel->eclass_indexes = NULL; + joinrel->eclass_member_indexes = NULL; + joinrel->eclass_source_indexes = NULL; + joinrel->eclass_derive_indexes = NULL; joinrel->subroot = NULL; joinrel->subplan_params = NIL; joinrel->rel_parallel_workers = -1; @@ -890,6 +896,9 @@ build_child_join_rel(PlannerInfo *root, RelOptInfo *outer_rel, joinrel->tuples = 0; joinrel->allvisfrac = 0; joinrel->eclass_indexes = NULL; + joinrel->eclass_member_indexes = NULL; + joinrel->eclass_source_indexes = NULL; + joinrel->eclass_derive_indexes = NULL; joinrel->subroot = NULL; joinrel->subplan_params = NIL; joinrel->amflags = 0; diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index 0d4b1ec4e4..cf43dd1385 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -313,6 +313,15 @@ struct PlannerInfo /* list of active EquivalenceClasses */ List *eq_classes; + /* list of each EquivalenceMember */ + List *eq_members; + + /* list of source RestrictInfos used to build EquivalenceClasses */ + List *eq_sources; + + /* list of RestrictInfos derived from EquivalenceClasses */ + List *eq_derives; + /* set true once ECs are canonical */ bool ec_merging_done; @@ -939,6 +948,24 @@ typedef struct RelOptInfo double allvisfrac; /* indexes in PlannerInfo's eq_classes list of ECs that mention this rel */ Bitmapset *eclass_indexes; + + /* + * Indexes in PlannerInfo's eq_members list of EMs that mention this rel + */ + Bitmapset *eclass_member_indexes; + + /* + * Indexes in PlannerInfo's eq_sources list for RestrictInfos that mention + * this relation. + */ + Bitmapset *eclass_source_indexes; + + /* + * Indexes in PlannerInfo's eq_derives list for RestrictInfos that mention + * this relation. + */ + Bitmapset *eclass_derive_indexes; + PlannerInfo *subroot; /* if subquery */ List *subplan_params; /* if subquery */ /* wanted number of parallel workers */ @@ -1360,6 +1387,39 @@ typedef struct JoinDomain * entry: consider SELECT random() AS a, random() AS b ... ORDER BY b,a. * So we record the SortGroupRef of the originating sort clause. * + * At various locations in the query planner, we must search for + * EquivalenceMembers within a given EquivalenceClass. For the common case, + * an EquivalenceClass does not have a large number of EquivalenceMembers, + * however, in cases such as planning queries to partitioned tables, the number + * of members can become large. To maintain planning performance, we make use + * of a bitmap index to allow us to quickly find EquivalenceMembers in a given + * EquivalenceClass belonging to a given relation or set of relations. This + * is done by storing a list of EquivalenceMembers belonging to all + * EquivalenceClasses in PlannerInfo and storing a Bitmapset for each + * RelOptInfo which has a bit set for each EquivalenceMember in that list + * which relates to the given relation. We also store a Bitmapset to mark all + * of the indexes in the PlannerInfo's list of EquivalenceMembers in the + * EquivalenceClass. We can quickly find the interesting indexes into the + * PlannerInfo's list by performing a bit-wise AND on the RelOptInfo's + * Bitmapset and the EquivalenceClasses. + * + * To further speed up the lookup of EquivalenceMembers we also record the + * non-child indexes. This allows us to deal with fewer bits for searches + * that don't require "is_child" EquivalenceMembers. We must also store the + * indexes into EquivalenceMembers which have no relids mentioned as some + * searches require that. + * + * Additionally, we also store the EquivalenceMembers of a given + * EquivalenceClass in the ec_members list. Technically, we could obtain this + * from looking at the bits in ec_member_indexes, however, finding all members + * of a given EquivalenceClass is common enough that it pays to have fast + * access to a dense list containing all members. + * + * The source and derived RestrictInfos are indexed in a similar method, + * although we don't maintain a List in the EquivalenceClass for these. These + * must be looked up in PlannerInfo using the ec_source_indexes and + * ec_derive_indexes Bitmapsets. + * * NB: if ec_merged isn't NULL, this class has been merged into another, and * should be ignored in favor of using the pointed-to class. * @@ -1377,9 +1437,19 @@ typedef struct EquivalenceClass List *ec_opfamilies; /* btree operator family OIDs */ Oid ec_collation; /* collation, if datatypes are collatable */ - List *ec_members; /* list of EquivalenceMembers */ - List *ec_sources; /* list of generating RestrictInfos */ - List *ec_derives; /* list of derived RestrictInfos */ + List *ec_members; /* list of EquivalenceMembers in the class */ + Bitmapset *ec_member_indexes; /* Indexes into all PlannerInfos + * eq_members for this EquivalenceClass */ + Bitmapset *ec_nonchild_indexes; /* Indexes into PlannerInfo's + * eq_members with em_is_child == + * false */ + Bitmapset *ec_norel_indexes; /* Indexes into PlannerInfo's eq_members + * for members where pull_varno on the + * em_expr is an empty set */ + Bitmapset *ec_source_indexes; /* indexes into PlannerInfo's eq_sources + * list of generating RestrictInfos */ + Bitmapset *ec_derive_indexes; /* indexes into PlannerInfo's eq_derives + * list of derived RestrictInfos */ Relids ec_relids; /* all relids appearing in ec_members, except * for child members (see below) */ bool ec_has_const; /* any pseudoconstants in ec_members? */ @@ -1427,8 +1497,9 @@ typedef struct EquivalenceMember NodeTag type; Expr *em_expr; /* the expression represented */ - Relids em_relids; /* all relids appearing in em_expr */ - bool em_is_const; /* expression is pseudoconstant? */ + Relids em_relids; /* relids for this member */ + bool em_is_const; /* is em_relids empty? */ + bool em_norel_expr; /* true if em_expr contains no Vars */ bool em_is_child; /* derived version for a child relation? */ Oid em_datatype; /* the "nominal type" used by the opfamily */ JoinDomain *em_jdomain; /* join domain containing the source clause */ diff --git a/src/include/nodes/pg_list.h b/src/include/nodes/pg_list.h index 529a382d28..77f26edf22 100644 --- a/src/include/nodes/pg_list.h +++ b/src/include/nodes/pg_list.h @@ -629,6 +629,7 @@ extern pg_nodiscard List *list_copy_deep(const List *oldlist); typedef int (*list_sort_comparator) (const ListCell *a, const ListCell *b); extern void list_sort(List *list, list_sort_comparator cmp); +extern int list_ptr_cmp(const ListCell *p1, const ListCell *p2); extern int list_int_cmp(const ListCell *p1, const ListCell *p2); extern int list_oid_cmp(const ListCell *p1, const ListCell *p2); diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h index 736d78ea4c..8b0bcbaf67 100644 --- a/src/include/optimizer/paths.h +++ b/src/include/optimizer/paths.h @@ -114,6 +114,31 @@ extern void mark_dummy_rel(RelOptInfo *rel); * equivclass.c * routines for managing EquivalenceClasses */ + +/* + * EquivalenceMemberIterator + * Data structure used for iteration over an EquivalenceClass's + * EquivalenceMember in order to quickly find the members matching our + * search pattern. + */ +typedef struct EquivalenceMemberIterator +{ + bool use_index; /* use matching_ems index? */ + bool relids_empty; /* is with_relids empty? */ + bool with_children; /* include em_is_child members? */ + bool with_norel_members; /* include members with empty em_relids */ +#ifdef USE_ASSERT_CHECKING + bool isstrict; /* ensure the correct next function is used */ +#endif + int orig_length; /* elements in eclass->ec_members at the start */ + int current_index; /* current iterator position, or -1. */ + Relids with_relids; /* relids to match in em_relids */ + PlannerInfo *root; /* PlannerInfo the eclass belongs to */ + EquivalenceClass *eclass; /* the EquivalenceClass we're looking at */ + Bitmapset *matching_ems; /* when use_index == true, these are the + * matching indexes in root eq_members */ +} EquivalenceMemberIterator; + typedef bool (*ec_matches_callback_type) (PlannerInfo *root, RelOptInfo *rel, EquivalenceClass *ec, @@ -134,7 +159,8 @@ extern EquivalenceClass *get_eclass_for_sort_expr(PlannerInfo *root, Index sortref, Relids rel, bool create_it); -extern EquivalenceMember *find_ec_member_matching_expr(EquivalenceClass *ec, +extern EquivalenceMember *find_ec_member_matching_expr(PlannerInfo *root, + EquivalenceClass *ec, Expr *expr, Relids relids); extern EquivalenceMember *find_computable_ec_member(PlannerInfo *root, @@ -159,7 +185,8 @@ extern bool exprs_known_equal(PlannerInfo *root, Node *item1, Node *item2); extern EquivalenceClass *match_eclasses_to_foreign_key_col(PlannerInfo *root, ForeignKeyOptInfo *fkinfo, int colno); -extern RestrictInfo *find_derived_clause_for_ec_member(EquivalenceClass *ec, +extern RestrictInfo *find_derived_clause_for_ec_member(PlannerInfo *root, + EquivalenceClass *ec, EquivalenceMember *em); extern void add_child_rel_equivalences(PlannerInfo *root, AppendRelInfo *appinfo, @@ -185,6 +212,42 @@ extern bool eclass_useful_for_merging(PlannerInfo *root, extern bool is_redundant_derived_clause(RestrictInfo *rinfo, List *clauselist); extern bool is_redundant_with_indexclauses(RestrictInfo *rinfo, List *indexclauses); +extern Bitmapset *get_ecmember_indexes(PlannerInfo *root, + EquivalenceClass *ec, + Relids relids, + bool with_children, + bool with_norel_members); +extern Bitmapset *get_ecmember_indexes_strict(PlannerInfo *root, + EquivalenceClass *ec, + Relids relids, + bool with_children, + bool with_norel_members); +extern void setup_eclass_member_iterator(EquivalenceMemberIterator *iter, + PlannerInfo *root, + EquivalenceClass *ec, Relids relids, + bool with_children, + bool with_norel_members); +extern void setup_eclass_member_strict_iterator(EquivalenceMemberIterator *iter, + PlannerInfo *root, + EquivalenceClass *ec, + Relids relids, + bool with_children, + bool with_norel_members); +extern EquivalenceMember *eclass_member_iterator_next(EquivalenceMemberIterator *iter); +extern EquivalenceMember *eclass_member_iterator_strict_next(EquivalenceMemberIterator *iter); +extern void eclass_member_iterator_dispose(EquivalenceMemberIterator *iter); +extern Bitmapset *get_ec_source_indexes(PlannerInfo *root, + EquivalenceClass *ec, + Relids relids); +extern Bitmapset *get_ec_source_indexes_strict(PlannerInfo *root, + EquivalenceClass *ec, + Relids relids); +extern Bitmapset *get_ec_derive_indexes(PlannerInfo *root, + EquivalenceClass *ec, + Relids relids); +extern Bitmapset *get_ec_derive_indexes_strict(PlannerInfo *root, + EquivalenceClass *ec, + Relids relids); /* * pathkeys.c diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 07fbb7ccf6..c219eea6dc 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -641,6 +641,7 @@ EphemeralNamedRelationMetadata EphemeralNamedRelationMetadataData EquivalenceClass EquivalenceMember +EquivalenceMemberIterator ErrorContextCallback ErrorData ErrorSaveContext -- 2.35.3.windows.1 [application/octet-stream] v16-0003-Move-EquivalenceMember-indexes-from-RelOptInfo-t.patch (12.3K, ../../CAJ2pMkYR_X-=pq+39-W5kc0OG7q9u5YUwDBCHnkPur17DXnxuQ@mail.gmail.com/4-v16-0003-Move-EquivalenceMember-indexes-from-RelOptInfo-t.patch) download | inline diff: From 8d4db63bb7b2cd9f5c992f635e2f004815d9bc5b Mon Sep 17 00:00:00 2001 From: Yuya Watari <[email protected]> Date: Fri, 3 Feb 2023 13:27:07 +0900 Subject: [PATCH v16 3/5] Move EquivalenceMember indexes from RelOptInfo to PlannerInfo --- src/backend/nodes/gen_node_support.pl | 10 ++++ src/backend/optimizer/path/equivclass.c | 65 ++++++++++--------------- src/backend/optimizer/path/indxpath.c | 2 +- src/backend/optimizer/prep/prepunion.c | 1 - src/backend/optimizer/util/relnode.c | 15 +++--- src/include/nodes/pathnodes.h | 38 ++++++++------- 6 files changed, 63 insertions(+), 68 deletions(-) diff --git a/src/backend/nodes/gen_node_support.pl b/src/backend/nodes/gen_node_support.pl index 19ed29657c..f40f4fec8f 100644 --- a/src/backend/nodes/gen_node_support.pl +++ b/src/backend/nodes/gen_node_support.pl @@ -862,6 +862,11 @@ _equal${n}(const $n *a, const $n *b) print $cff "\tCOPY_SCALAR_FIELD($f);\n" unless $copy_ignore; print $eff "\tCOMPARE_SCALAR_FIELD($f);\n" unless $equal_ignore; } + elsif ($t eq 'ECIndexes*') + { + # TODO: + # Do nothing + } else { die @@ -1211,6 +1216,11 @@ _read${n}(void) } ! unless $no_read; } + elsif ($t eq 'ECIndexes*') + { + # TODO: + # Do nothing + } else { die diff --git a/src/backend/optimizer/path/equivclass.c b/src/backend/optimizer/path/equivclass.c index aad14c25ce..c2b27e3ae1 100644 --- a/src/backend/optimizer/path/equivclass.c +++ b/src/backend/optimizer/path/equivclass.c @@ -546,10 +546,9 @@ add_eq_source(PlannerInfo *root, EquivalenceClass *ec, RestrictInfo *rinfo) i = -1; while ((i = bms_next_member(rinfo->clause_relids, i)) >= 0) { - RelOptInfo *rel = root->simple_rel_array[i]; - - rel->eclass_source_indexes = bms_add_member(rel->eclass_source_indexes, - source_idx); + root->eclass_indexes_array[i].source_indexes + = bms_add_member(root->eclass_indexes_array[i].source_indexes, + source_idx); } } @@ -568,10 +567,9 @@ add_eq_derive(PlannerInfo *root, EquivalenceClass *ec, RestrictInfo *rinfo) i = -1; while ((i = bms_next_member(rinfo->clause_relids, i)) >= 0) { - RelOptInfo *rel = root->simple_rel_array[i]; - - rel->eclass_derive_indexes = bms_add_member(rel->eclass_derive_indexes, - derive_idx); + root->eclass_indexes_array[i].derive_indexes + = bms_add_member(root->eclass_indexes_array[i].derive_indexes, + derive_idx); } } @@ -650,9 +648,9 @@ add_eq_member(PlannerInfo *root, EquivalenceClass *ec, Expr *expr, Relids relids i = -1; while ((i = bms_next_member(expr_relids, i)) >= 0) { - RelOptInfo *rel = root->simple_rel_array[i]; - - rel->eclass_member_indexes = bms_add_member(rel->eclass_member_indexes, em_index); + root->eclass_indexes_array[i].member_indexes + = bms_add_member(root->eclass_indexes_array[i].member_indexes, + em_index); } return em; @@ -2528,10 +2526,9 @@ reconsider_full_join_clause(PlannerInfo *root, OuterJoinClauseInfo *ojcinfo) i = -1; while ((i = bms_next_member(coal_em->em_relids, i)) >= 0) { - RelOptInfo *rel = root->simple_rel_array[i]; - - rel->eclass_member_indexes = bms_del_member(rel->eclass_member_indexes, - coal_idx); + root->eclass_indexes_array[i].member_indexes + = bms_del_member(root->eclass_indexes_array[i].member_indexes, + coal_idx); } return true; @@ -3467,9 +3464,8 @@ get_ecmember_indexes(PlannerInfo *root, EquivalenceClass *ec, Relids relids, while ((i = bms_next_member(relids, i)) >= 0) { - RelOptInfo *rel = root->simple_rel_array[i]; - - rel_ems = bms_add_members(rel_ems, rel->eclass_member_indexes); + rel_ems = bms_add_members(rel_ems, + root->eclass_indexes_array[i].member_indexes); } #ifdef USE_ASSERT_CHECKING @@ -3515,8 +3511,6 @@ get_ecmember_indexes_strict(PlannerInfo *root, EquivalenceClass *ec, if (i >= 0) { - RelOptInfo *rel = root->simple_rel_array[i]; - /* * bms_intersect to the first relation to try to keep the resulting * Bitmapset as small as possible. This saves having to make a @@ -3524,17 +3518,16 @@ get_ecmember_indexes_strict(PlannerInfo *root, EquivalenceClass *ec, * more words than the other. */ if (!with_children) - matching_ems = bms_intersect(rel->eclass_member_indexes, + matching_ems = bms_intersect(root->eclass_indexes_array[i].member_indexes, ec->ec_nonchild_indexes); else - matching_ems = bms_intersect(rel->eclass_member_indexes, + matching_ems = bms_intersect(root->eclass_indexes_array[i].member_indexes, ec->ec_member_indexes); while ((i = bms_next_member(relids, i)) >= 0) { - rel = root->simple_rel_array[i]; matching_ems = bms_int_members(matching_ems, - rel->eclass_member_indexes); + root->eclass_indexes_array[i].member_indexes); } } @@ -3879,9 +3872,8 @@ get_ec_source_indexes(PlannerInfo *root, EquivalenceClass *ec, Relids relids) while ((i = bms_next_member(relids, i)) >= 0) { - RelOptInfo *rel = root->simple_rel_array[i]; - - rel_esis = bms_add_members(rel_esis, rel->eclass_source_indexes); + rel_esis = bms_add_members(rel_esis, + root->eclass_indexes_array[i].source_indexes); } #ifdef USE_ASSERT_CHECKING @@ -3917,8 +3909,6 @@ get_ec_source_indexes_strict(PlannerInfo *root, EquivalenceClass *ec, if (i >= 0) { - RelOptInfo *rel = root->simple_rel_array[i]; - /* * bms_intersect to the first relation to try to keep the resulting * Bitmapset as small as possible. This saves having to make a @@ -3926,12 +3916,11 @@ get_ec_source_indexes_strict(PlannerInfo *root, EquivalenceClass *ec, * more words than the other. */ esis = bms_intersect(ec->ec_source_indexes, - rel->eclass_source_indexes); + root->eclass_indexes_array[i].source_indexes); while ((i = bms_next_member(relids, i)) >= 0) { - rel = root->simple_rel_array[i]; - esis = bms_int_members(esis, rel->eclass_source_indexes); + esis = bms_int_members(esis, root->eclass_indexes_array[i].source_indexes); } } @@ -3968,9 +3957,8 @@ get_ec_derive_indexes(PlannerInfo *root, EquivalenceClass *ec, Relids relids) while ((i = bms_next_member(relids, i)) >= 0) { - RelOptInfo *rel = root->simple_rel_array[i]; - - rel_edis = bms_add_members(rel_edis, rel->eclass_derive_indexes); + rel_edis = bms_add_members(rel_edis, + root->eclass_indexes_array[i].derive_indexes); } #ifdef USE_ASSERT_CHECKING @@ -4006,8 +3994,6 @@ get_ec_derive_indexes_strict(PlannerInfo *root, EquivalenceClass *ec, if (i >= 0) { - RelOptInfo *rel = root->simple_rel_array[i]; - /* * bms_intersect to the first relation to try to keep the resulting * Bitmapset as small as possible. This saves having to make a @@ -4015,12 +4001,11 @@ get_ec_derive_indexes_strict(PlannerInfo *root, EquivalenceClass *ec, * more words than the other. */ edis = bms_intersect(ec->ec_derive_indexes, - rel->eclass_derive_indexes); + root->eclass_indexes_array[i].derive_indexes); while ((i = bms_next_member(relids, i)) >= 0) { - rel = root->simple_rel_array[i]; - edis = bms_int_members(edis, rel->eclass_derive_indexes); + edis = bms_int_members(edis, root->eclass_indexes_array[i].derive_indexes); } } diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c index 5ebf944cb1..384932727b 100644 --- a/src/backend/optimizer/path/indxpath.c +++ b/src/backend/optimizer/path/indxpath.c @@ -3111,7 +3111,7 @@ match_pathkeys_to_index(PlannerInfo *root, IndexOptInfo *index, * here. See also get_eclass_for_sort_expr.) */ matching_ems = bms_intersect(pathkey->pk_eclass->ec_member_indexes, - index->rel->eclass_member_indexes); + root->eclass_indexes_array[index->rel->relid].member_indexes); i = -1; while ((i = bms_next_member(matching_ems, i)) >= 0) diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c index 4c7d66311f..938a1f581c 100644 --- a/src/backend/optimizer/prep/prepunion.c +++ b/src/backend/optimizer/prep/prepunion.c @@ -196,7 +196,6 @@ setup_append_rel_entry(PlannerInfo *root) RelOptInfo *rel = makeNode(RelOptInfo); memset(rel, 0, sizeof(RelOptInfo)); - rel->eclass_member_indexes = NULL; /* make sure nothing else has made use of this element */ Assert(root->simple_rel_array[0] == NULL); diff --git a/src/backend/optimizer/util/relnode.c b/src/backend/optimizer/util/relnode.c index dfa374a8cf..2131333297 100644 --- a/src/backend/optimizer/util/relnode.c +++ b/src/backend/optimizer/util/relnode.c @@ -105,6 +105,9 @@ setup_simple_rel_arrays(PlannerInfo *root) root->simple_rel_array = (RelOptInfo **) palloc0(size * sizeof(RelOptInfo *)); + root->eclass_indexes_array = (ECIndexes *) + palloc0(size * sizeof(ECIndexes)); + /* simple_rte_array is an array equivalent of the rtable list */ root->simple_rte_array = (RangeTblEntry **) palloc0(size * sizeof(RangeTblEntry *)); @@ -168,6 +171,9 @@ expand_planner_arrays(PlannerInfo *root, int add_size) root->simple_rel_array = repalloc0_array(root->simple_rel_array, RelOptInfo *, root->simple_rel_array_size, new_size); + root->eclass_indexes_array = + repalloc0_array(root->eclass_indexes_array, ECIndexes, root->simple_rel_array_size, new_size); + root->simple_rte_array = repalloc0_array(root->simple_rte_array, RangeTblEntry *, root->simple_rel_array_size, new_size); @@ -226,9 +232,6 @@ build_simple_rel(PlannerInfo *root, int relid, RelOptInfo *parent) rel->tuples = 0; rel->allvisfrac = 0; rel->eclass_indexes = NULL; - rel->eclass_member_indexes = NULL; - rel->eclass_source_indexes = NULL; - rel->eclass_derive_indexes = NULL; rel->subroot = NULL; rel->subplan_params = NIL; rel->rel_parallel_workers = -1; /* set up in get_relation_info */ @@ -704,9 +707,6 @@ build_join_rel(PlannerInfo *root, joinrel->tuples = 0; joinrel->allvisfrac = 0; joinrel->eclass_indexes = NULL; - joinrel->eclass_member_indexes = NULL; - joinrel->eclass_source_indexes = NULL; - joinrel->eclass_derive_indexes = NULL; joinrel->subroot = NULL; joinrel->subplan_params = NIL; joinrel->rel_parallel_workers = -1; @@ -896,9 +896,6 @@ build_child_join_rel(PlannerInfo *root, RelOptInfo *outer_rel, joinrel->tuples = 0; joinrel->allvisfrac = 0; joinrel->eclass_indexes = NULL; - joinrel->eclass_member_indexes = NULL; - joinrel->eclass_source_indexes = NULL; - joinrel->eclass_derive_indexes = NULL; joinrel->subroot = NULL; joinrel->subplan_params = NIL; joinrel->amflags = 0; diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index cf43dd1385..0b57a89f94 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -169,6 +169,25 @@ typedef struct PlannerGlobal #define planner_subplan_get_plan(root, subplan) \ ((Plan *) list_nth((root)->glob->subplans, (subplan)->plan_id - 1)) +typedef struct ECIndexes +{ + /* + * Indexes in PlannerInfo's eq_members list of EMs that mention this rel + */ + Bitmapset *member_indexes; + + /* + * Indexes in PlannerInfo's eq_sources list for RestrictInfos that mention + * this relation. + */ + Bitmapset *source_indexes; + + /* + * Indexes in PlannerInfo's eq_derives list for RestrictInfos that mention + * this relation. + */ + Bitmapset *derive_indexes; +} ECIndexes; /*---------- * PlannerInfo @@ -231,6 +250,8 @@ struct PlannerInfo /* allocated size of array */ int simple_rel_array_size; + ECIndexes *eclass_indexes_array pg_node_attr(array_size(simple_rel_array_size)); + /* * simple_rte_array is the same length as simple_rel_array and holds * pointers to the associated rangetable entries. Using this is a shade @@ -949,23 +970,6 @@ typedef struct RelOptInfo /* indexes in PlannerInfo's eq_classes list of ECs that mention this rel */ Bitmapset *eclass_indexes; - /* - * Indexes in PlannerInfo's eq_members list of EMs that mention this rel - */ - Bitmapset *eclass_member_indexes; - - /* - * Indexes in PlannerInfo's eq_sources list for RestrictInfos that mention - * this relation. - */ - Bitmapset *eclass_source_indexes; - - /* - * Indexes in PlannerInfo's eq_derives list for RestrictInfos that mention - * this relation. - */ - Bitmapset *eclass_derive_indexes; - PlannerInfo *subroot; /* if subquery */ List *subplan_params; /* if subquery */ /* wanted number of parallel workers */ -- 2.35.3.windows.1 [application/octet-stream] v16-0004-Remove-setup_append_rel_entry.patch (1.8K, ../../CAJ2pMkYR_X-=pq+39-W5kc0OG7q9u5YUwDBCHnkPur17DXnxuQ@mail.gmail.com/5-v16-0004-Remove-setup_append_rel_entry.patch) download | inline diff: From ae122f049f61a034ef961f3e75275abf20f2699d Mon Sep 17 00:00:00 2001 From: Yuya Watari <[email protected]> Date: Mon, 6 Feb 2023 09:25:24 +0900 Subject: [PATCH v16 4/5] Remove setup_append_rel_entry() --- src/backend/optimizer/prep/prepunion.c | 23 +---------------------- 1 file changed, 1 insertion(+), 22 deletions(-) diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c index 938a1f581c..31e5ed2c10 100644 --- a/src/backend/optimizer/prep/prepunion.c +++ b/src/backend/optimizer/prep/prepunion.c @@ -45,7 +45,7 @@ #include "utils/selfuncs.h" #include "utils/syscache.h" -static void setup_append_rel_entry(PlannerInfo *root); + static RelOptInfo *recurse_set_operations(Node *setOp, PlannerInfo *root, List *colTypes, List *colCollations, bool junkOK, @@ -150,8 +150,6 @@ plan_set_operations(PlannerInfo *root) leftmostQuery = leftmostRTE->subquery; Assert(leftmostQuery != NULL); - setup_append_rel_entry(root); - /* * If the topmost node is a recursive union, it needs special processing. */ @@ -183,25 +181,6 @@ plan_set_operations(PlannerInfo *root) return setop_rel; } -/* - * setup_append_rel_entry - * Add entry into root's simple_rel_array at element 0. This is required - * because generate_append_tlist() makes Vars with varno=0. In - * add_eq_member() we need to index EquivalenceMembers belonging to this - * relation. - */ -static void -setup_append_rel_entry(PlannerInfo *root) -{ - RelOptInfo *rel = makeNode(RelOptInfo); - - memset(rel, 0, sizeof(RelOptInfo)); - - /* make sure nothing else has made use of this element */ - Assert(root->simple_rel_array[0] == NULL); - root->simple_rel_array[0] = rel; -} - /* * recurse_set_operations * Recursively handle one step in a tree of set operations -- 2.35.3.windows.1 [application/octet-stream] v16-0005-Fix-an-assertion.patch (1.4K, ../../CAJ2pMkYR_X-=pq+39-W5kc0OG7q9u5YUwDBCHnkPur17DXnxuQ@mail.gmail.com/6-v16-0005-Fix-an-assertion.patch) download | inline diff: From 6438018135d46e8b32a295f017eaf649501e1996 Mon Sep 17 00:00:00 2001 From: Yuya Watari <[email protected]> Date: Mon, 6 Feb 2023 09:25:35 +0900 Subject: [PATCH v16 5/5] Fix an assertion Fix an assertion to deal with root->simple_rel_array[0]. This fix is required because the while condition for looping through relids has been changed in our patches as follows: - while ((i = bms_next_member(newec->ec_relids, i)) > 0) + while ((i = bms_next_member(newec->ec_relids, i)) >= 0) Originally, we stopped iterating when we we reached the root->simple_rel_array[0] member. However, we do not do so with our patches, so the assertion of Assert(bms_is_member(i, root->outer_join_rels)) may fail. To solve this problem, this commit changes its condition. TODO: Do we need to change the other occurances? --- src/backend/optimizer/path/equivclass.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/backend/optimizer/path/equivclass.c b/src/backend/optimizer/path/equivclass.c index c2b27e3ae1..0ab017788d 100644 --- a/src/backend/optimizer/path/equivclass.c +++ b/src/backend/optimizer/path/equivclass.c @@ -836,7 +836,7 @@ get_eclass_for_sort_expr(PlannerInfo *root, if (rel == NULL) /* must be an outer join */ { - Assert(bms_is_member(i, root->outer_join_rels)); + Assert(i == 0 || bms_is_member(i, root->outer_join_rels)); continue; } -- 2.35.3.windows.1 ^ permalink raw reply [nested|flat] 56+ messages in thread
* Re: [PoC] Reducing planning time when tables have many partitions @ 2023-02-14 10:01 Andrey Lepikhov <[email protected]> parent: Yuya Watari <[email protected]> 0 siblings, 0 replies; 56+ messages in thread From: Andrey Lepikhov @ 2023-02-14 10:01 UTC (permalink / raw) To: Yuya Watari <[email protected]>; David Rowley <[email protected]>; +Cc: Thom Brown <[email protected]>; Alvaro Herrera <[email protected]>; Zhang Mingli <[email protected]>; Tom Lane <[email protected]>; PostgreSQL Developers <[email protected]> On 2/6/23 06:47, Yuya Watari wrote: > Of course, I'm not sure if my approach in v16-0003 is ideal, but it > may help solve your concern above. Since simple_rel_array[0] is no > longer necessary with my patch, I removed the setup_append_rel_entry() > function in v16-0004. However, to work the patch, I needed to change > some assertions in v16-0005. For more details, please see the commit > message of v16-0005. After these works, the attached patches passed > all regression tests in my environment. > > Instead of my approach, imitating the following change to > get_eclass_indexes_for_relids() is also a possible solution. Ignoring > NULL RelOptInfos enables us to avoid the segfault, but we have to > adjust EquivalenceMemberIterator to match the result, and I'm not sure > if this idea is correct. As I see, You moved the indexes from RelOptInfo to PlannerInfo. May be better to move them into RangeTblEntry instead? -- Regards Andrey Lepikhov Postgres Professional ^ permalink raw reply [nested|flat] 56+ messages in thread
end of thread, other threads:[~2023-02-14 10:01 UTC | newest] Thread overview: 56+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]> 2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]> 2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]> 2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]> 2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]> 2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]> 2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]> 2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]> 2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]> 2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]> 2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]> 2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]> 2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]> 2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]> 2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]> 2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]> 2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]> 2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]> 2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]> 2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]> 2021-01-09 02:17 [PATCH 1/2] review Tomas Vondra <[email protected]> 2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]> 2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]> 2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]> 2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]> 2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]> 2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]> 2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]> 2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]> 2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]> 2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]> 2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]> 2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]> 2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]> 2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]> 2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]> 2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]> 2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]> 2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]> 2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]> 2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]> 2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]> 2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]> 2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]> 2021-03-09 20:09 [PATCH 2/2] review Tomas Vondra <[email protected]> 2021-03-16 16:29 [PATCH 02/10] review Tomas Vondra <[email protected]> 2021-03-16 16:29 [PATCH 02/10] review Tomas Vondra <[email protected]> 2021-03-16 18:26 [PATCH 04/10] review Tomas Vondra <[email protected]> 2021-03-16 18:26 [PATCH 04/10] review Tomas Vondra <[email protected]> 2021-03-17 01:19 [PATCH 06/10] review Tomas Vondra <[email protected]> 2021-03-17 01:19 [PATCH 06/10] review Tomas Vondra <[email protected]> 2021-03-17 01:36 [PATCH 08/10] review Tomas Vondra <[email protected]> 2021-03-17 01:36 [PATCH 08/10] review Tomas Vondra <[email protected]> 2023-01-30 12:14 Re: [PoC] Reducing planning time when tables have many partitions David Rowley <[email protected]> 2023-02-06 01:47 ` Re: [PoC] Reducing planning time when tables have many partitions Yuya Watari <[email protected]> 2023-02-14 10:01 ` Re: [PoC] Reducing planning time when tables have many partitions Andrey Lepikhov <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox