public inbox for [email protected]help / color / mirror / Atom feed
[PATCH 03/10] Introduce UniqueKey attributes on RelOptInfo struct. 91+ messages / 14 participants [nested] [flat]
* [PATCH 03/10] Introduce UniqueKey attributes on RelOptInfo struct. @ 2020-05-11 07:50 一挃 <[email protected]> 0 siblings, 0 replies; 91+ messages in thread From: 一挃 @ 2020-05-11 07:50 UTC (permalink / raw) UniqueKey is a set of exprs on RelOptInfo which represents the exprs will be unique on the given RelOptInfo. You can see README.uniquekey for more information. --- src/backend/nodes/copyfuncs.c | 13 + src/backend/nodes/list.c | 31 + src/backend/nodes/makefuncs.c | 13 + src/backend/nodes/outfuncs.c | 11 + src/backend/nodes/readfuncs.c | 10 + src/backend/optimizer/path/Makefile | 3 +- src/backend/optimizer/path/README.uniquekey | 131 +++ src/backend/optimizer/path/allpaths.c | 10 + src/backend/optimizer/path/joinpath.c | 9 +- src/backend/optimizer/path/joinrels.c | 2 + src/backend/optimizer/path/pathkeys.c | 3 +- src/backend/optimizer/path/uniquekeys.c | 1135 +++++++++++++++++++ src/backend/optimizer/plan/planner.c | 13 +- src/backend/optimizer/prep/prepunion.c | 2 + src/backend/optimizer/util/appendinfo.c | 44 + src/backend/optimizer/util/inherit.c | 18 +- src/include/nodes/makefuncs.h | 3 + src/include/nodes/nodes.h | 1 + src/include/nodes/pathnodes.h | 29 +- src/include/nodes/pg_list.h | 10 + src/include/optimizer/appendinfo.h | 3 + src/include/optimizer/optimizer.h | 2 + src/include/optimizer/paths.h | 43 + 23 files changed, 1515 insertions(+), 24 deletions(-) create mode 100644 src/backend/optimizer/path/README.uniquekey create mode 100644 src/backend/optimizer/path/uniquekeys.c diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..75c1c5e824 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2296,6 +2296,16 @@ _copyPathKey(const PathKey *from) return newnode; } +static UniqueKey * +_copyUniqueKey(const UniqueKey *from) +{ + UniqueKey *newnode = makeNode(UniqueKey); + + COPY_NODE_FIELD(exprs); + COPY_SCALAR_FIELD(multi_nullvals); + + return newnode; +} /* * _copyRestrictInfo */ @@ -5220,6 +5230,9 @@ copyObjectImpl(const void *from) case T_PathKey: retval = _copyPathKey(from); break; + case T_UniqueKey: + retval = _copyUniqueKey(from); + break; case T_RestrictInfo: retval = _copyRestrictInfo(from); break; diff --git a/src/backend/nodes/list.c b/src/backend/nodes/list.c index dbf6b30233..ca099495a1 100644 --- a/src/backend/nodes/list.c +++ b/src/backend/nodes/list.c @@ -702,6 +702,37 @@ list_member_oid(const List *list, Oid datum) return false; } +/* + * return true iff every entry in "members" list is also present + * in the "target" list. + */ +bool +list_is_subset(const List *members, const List *target) +{ + const ListCell *lc1, *lc2; + + Assert(IsPointerList(members)); + Assert(IsPointerList(target)); + check_list_invariants(members); + check_list_invariants(target); + + foreach(lc1, members) + { + bool found = false; + foreach(lc2, target) + { + if (equal(lfirst(lc1), lfirst(lc2))) + { + found = true; + break; + } + } + if (!found) + return false; + } + return true; +} + /* * Delete the n'th cell (counting from 0) in list. * diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 01c110cd2f..40415d0f5b 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -815,3 +815,16 @@ makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols) v->va_cols = va_cols; return v; } + + +/* + * makeUniqueKey + */ +UniqueKey* +makeUniqueKey(List *exprs, bool multi_nullvals) +{ + UniqueKey * ukey = makeNode(UniqueKey); + ukey->exprs = exprs; + ukey->multi_nullvals = multi_nullvals; + return ukey; +} diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..44154cde6a 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2456,6 +2456,14 @@ _outPathKey(StringInfo str, const PathKey *node) WRITE_BOOL_FIELD(pk_nulls_first); } +static void +_outUniqueKey(StringInfo str, const UniqueKey *node) +{ + WRITE_NODE_TYPE("UNIQUEKEY"); + WRITE_NODE_FIELD(exprs); + WRITE_BOOL_FIELD(multi_nullvals); +} + static void _outPathTarget(StringInfo str, const PathTarget *node) { @@ -4198,6 +4206,9 @@ outNode(StringInfo str, const void *obj) case T_PathKey: _outPathKey(str, obj); break; + case T_UniqueKey: + _outUniqueKey(str, obj); + break; case T_PathTarget: _outPathTarget(str, obj); break; diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index c5e136e9c3..b3e212bf1c 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -492,6 +492,14 @@ _readSetOperationStmt(void) READ_DONE(); } +static UniqueKey * +_readUniqueKey(void) +{ + READ_LOCALS(UniqueKey); + READ_NODE_FIELD(exprs); + READ_BOOL_FIELD(multi_nullvals); + READ_DONE(); +} /* * Stuff from primnodes.h. @@ -2717,6 +2725,8 @@ parseNodeString(void) return_value = _readCommonTableExpr(); else if (MATCH("SETOPERATIONSTMT", 16)) return_value = _readSetOperationStmt(); + else if (MATCH("UNIQUEKEY", 9)) + return_value = _readUniqueKey(); else if (MATCH("ALIAS", 5)) return_value = _readAlias(); else if (MATCH("RANGEVAR", 8)) diff --git a/src/backend/optimizer/path/Makefile b/src/backend/optimizer/path/Makefile index 1e199ff66f..7b9820c25f 100644 --- a/src/backend/optimizer/path/Makefile +++ b/src/backend/optimizer/path/Makefile @@ -21,6 +21,7 @@ OBJS = \ joinpath.o \ joinrels.o \ pathkeys.o \ - tidpath.o + tidpath.o \ + uniquekeys.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/optimizer/path/README.uniquekey b/src/backend/optimizer/path/README.uniquekey new file mode 100644 index 0000000000..5eac761995 --- /dev/null +++ b/src/backend/optimizer/path/README.uniquekey @@ -0,0 +1,131 @@ +1. What is UniqueKey? +We can think UniqueKey is a set of exprs for a RelOptInfo, which we are insure +that doesn't yields same result among all the rows. The simplest UniqueKey +format is primary key. + +However we define the UnqiueKey as below. + +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + +exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL +is a special case of UniqueKey, which means there is only one row in that +relation.it has a stronger semantic than others. like SELECT uk FROM t; uk is +normal unique key and may have different values. SELECT colx FROM t WHERE uk = +const. colx is unique AND we have only 1 value. This field can used for +innerrel_is_unique. this logic is handled specially in add_uniquekey_for_onerow +function. + +multi_nullvals: true means multi null values may exist in these exprs, so the +uniqueness is not guaranteed in this case. This field is necessary for +remove_useless_join & reduce_unique_semijoins where we don't mind these +duplicated NULL values. It is set to true for 2 cases. One is a unique key +from a unique index but the related column is nullable. The other one is for +outer join. see populate_joinrel_uniquekeys for detail. + + +The UniqueKey can be used at the following cases at least: +1. remove_useless_joins. +2. reduce_semianti_joins +3. remove distinct node if distinct clause is unique. +4. remove aggnode if group by clause is unique. +5. Index Skip Scan (WIP) +6. Aggregation Push Down without 2 phase aggregation if the join can't + duplicated the aggregated rows. (work in progress feature) + +2. How is it maintained? + +We have a set of populate_xxx_unqiuekeys functions to maintain the uniquekey on +various cases. xxx includes baserel, joinrel, partitionedrel, distinctrel, +groupedrel, unionrel. and we also need to convert the uniquekey from subquery +to outer relation, which is what convert_subquery_uniquekeys does. + +1. The first part is about baserel. We handled 3 cases. suppose we have Unique +Index on (a, b). + +1. SELECT a, b FROM t. UniqueKey (a, b) +2. SELECT a FROM t WHERE b = 1; UniqueKey (a) +3. SELECT .. FROM t WHERE a = 1 AND b = 1; UniqueKey (NIL). onerow case, every + column is Unique. + +2. The next part is joinrel, this part is most error-prone, we simplified the rules +like below: +1. If the relation's UniqueKey can't be duplicated after join, then is will be + still valid for the join rel. The function we used here is + innerrel_keeps_unique. The basic idea is innerrel.any_col = outer.uk. + +2. If the UnqiueKey can't keep valid via the rule 1, the combination of the + UniqueKey from both sides are valid for sure. We can prove this as: if the + unique exprs from rel1 is duplicated by rel2, the duplicated rows must + contains different unique exprs from rel2. + +More considerations about onerow: +1. If relation with one row and it can't be duplicated, it is still possible + contains mulit_nullvas after outer join. +2. If the either UniqueKey can be duplicated after join, the can get one row + only when both side is one row AND there is no outer join. +3. Whenever the onerow UniqueKey is not a valid any more, we need to convert one + row UniqueKey to normal unique key since we don't store exprs for one-row + relation. get_exprs_from_uniquekeys will be used here. + + +More considerations about multi_nullvals after join: +1. If the original UnqiueKey has multi_nullvals, the final UniqueKey will have + mulit_nullvals in any case. +2. If a unique key doesn't allow mulit_nullvals, after some outer join, it + allows some outer join. + + +3. When we comes to subquery, we need to convert_subquery_unqiuekeys just like +convert_subquery_pathkeys. Only the UniqueKey insides subquery is referenced as +a Var in outer relation will be reused. The relationship between the outerrel.Var +and subquery.exprs is built with outerel->subroot->processed_tlist. + + +4. As for the SRF functions, it will break the uniqueness of uniquekey, However it +is handled in adjust_paths_for_srfs, which happens after the query_planner. so +we will maintain the UniqueKey until there and reset it to NIL at that +places. This can't help on distinct/group by elimination cases but probably help +in some other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is +semantic correctly. + + +5. As for inherit table, we first main the UnqiueKey on childrel as well. But for +partitioned table we need to maintain 2 different kinds of +UnqiueKey. 1). UniqueKey on the parent relation 2). UniqueKey on child +relation for partition wise query. + +Example: +CREATE TABLE p (a int not null, b int not null) partition by list (a); +CREATE TABLE p0 partition of p for values in (1); +CREATE TABLE p1 partition of p for values in (2); + +create unique index p0_b on p0(b); +create unique index p1_b on p1(b); + +Now b is only unique on partition level, so the distinct can't be removed on +the following cases. SELECT DISTINCT b FROM p; + +Another example is SELECT DISTINCT a, b FROM p WHERE a = 1; Since only one +partition is chosen, the UniqueKey on child relation is same as the UniqueKey on +parent relation. + +Another usage of UniqueKey on partition level is it be helpful for +partition-wise join. + +As for the UniqueKey on parent table level, it comes with 2 different ways, +1). the UniqueKey is also derived in UniqueKey index, but the index must be same +in all the related children relations and the unique index must contains +Partition Key in it. Example: + +CREATE UNIQUE INDEX p_ab ON p(a, b); -- where a is the partition key. + +-- Query +SELECT a, b FROM p; the (a, b) is a UniqueKey of p. + +2). If the parent relation has only one childrel, the UniqueKey on childrel is + the UniqueKey on parent as well. diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index fc1a3a68a2..66bf6f19f7 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -579,6 +579,12 @@ set_plain_rel_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) */ check_index_predicates(root, rel); + /* + * Now that we've marked which partial indexes are suitable, we can now + * build the relation's unique keys. + */ + populate_baserel_uniquekeys(root, rel, rel->indexlist); + /* Mark rel with estimated output rows, width, etc */ set_baserel_size_estimates(root, rel); } @@ -1299,6 +1305,8 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, /* Add paths to the append relation. */ add_paths_to_append_rel(root, rel, live_childrels); + if (IS_PARTITIONED_REL(rel)) + populate_partitionedrel_uniquekeys(root, rel, live_childrels); } @@ -2306,6 +2314,8 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, pathkeys, required_outer)); } + convert_subquery_uniquekeys(root, rel, sub_final_rel); + /* If outer rel allows parallelism, do same for partial paths. */ if (rel->consider_parallel && bms_is_empty(required_outer)) { diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c index 57ce97fd53..697ac047fb 100644 --- a/src/backend/optimizer/path/joinpath.c +++ b/src/backend/optimizer/path/joinpath.c @@ -71,13 +71,6 @@ static void consider_parallel_mergejoin(PlannerInfo *root, static void hash_inner_and_outer(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, RelOptInfo *innerrel, JoinType jointype, JoinPathExtraData *extra); -static List *select_mergejoin_clauses(PlannerInfo *root, - RelOptInfo *joinrel, - RelOptInfo *outerrel, - RelOptInfo *innerrel, - List *restrictlist, - JoinType jointype, - bool *mergejoin_allowed); static void generate_mergejoin_paths(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *innerrel, @@ -1927,7 +1920,7 @@ hash_inner_and_outer(PlannerInfo *root, * if it is mergejoinable and involves vars from the two sub-relations * currently of interest. */ -static List * +List * select_mergejoin_clauses(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c index 0dbe2ac726..7271f044ec 100644 --- a/src/backend/optimizer/path/joinrels.c +++ b/src/backend/optimizer/path/joinrels.c @@ -924,6 +924,8 @@ populate_joinrel_with_paths(PlannerInfo *root, RelOptInfo *rel1, /* Apply partitionwise join technique, if possible. */ try_partitionwise_join(root, rel1, rel2, joinrel, sjinfo, restrictlist); + + populate_joinrel_uniquekeys(root, joinrel, rel1, rel2, restrictlist, sjinfo->jointype); } diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index bd9a176d7d..139278829b 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -33,7 +33,6 @@ static bool pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys); static bool matches_boolean_partition_clause(RestrictInfo *rinfo, RelOptInfo *partrel, int partkeycol); -static Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); static bool right_merge_direction(PlannerInfo *root, PathKey *pathkey); @@ -1035,7 +1034,7 @@ convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel, * We need this to ensure that we don't return pathkeys describing values * that are unavailable above the level of the subquery scan. */ -static Var * +Var * find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle) { ListCell *lc; diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c new file mode 100644 index 0000000000..77ed2b2eff --- /dev/null +++ b/src/backend/optimizer/path/uniquekeys.c @@ -0,0 +1,1135 @@ +/*------------------------------------------------------------------------- + * + * uniquekeys.c + * Utilities for matching and building unique keys + * + * Portions Copyright (c) 2020, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/optimizer/path/uniquekeys.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "nodes/makefuncs.h" +#include "nodes/nodeFuncs.h" +#include "optimizer/pathnode.h" +#include "optimizer/paths.h" +#include "optimizer/appendinfo.h" +#include "optimizer/optimizer.h" +#include "optimizer/tlist.h" +#include "rewrite/rewriteManip.h" + + +/* + * This struct is used to help populate_joinrel_uniquekeys. + * + * added_to_joinrel is true if a uniquekey (from outerrel or innerrel) + * has been added to joinrel. + * useful is true if the exprs of the uniquekey still appears in joinrel. + */ +typedef struct UniqueKeyContextData +{ + UniqueKey *uniquekey; + bool added_to_joinrel; + bool useful; +} *UniqueKeyContext; + +static List *initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel); +static bool innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + bool reverse); + +static List *get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals); +static List *get_exprs_from_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + UniqueKey *ukey); +static void add_uniquekey_for_onerow(RelOptInfo *rel); +static bool add_combined_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype); + +/* Used for unique indexes checking for partitioned table */ +static bool index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind); +static IndexOptInfo *simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from); +static bool simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2); +static void adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_index); + +/* Helper function for grouped relation and distinct relation. */ +static void add_uniquekey_from_sortgroups(PlannerInfo *root, + RelOptInfo *rel, + List *sortgroups); + +/* + * populate_baserel_uniquekeys + * Populate 'baserel' uniquekeys list by looking at the rel's unique index + * and baserestrictinfo + */ +void +populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List *indexlist) +{ + ListCell *lc; + List *matched_uniq_indexes = NIL; + + /* Attrs appears in rel->reltarget->exprs. */ + Bitmapset *used_attrs = NULL; + + List *const_exprs = NIL; + List *expr_opfamilies = NIL; + + Assert(baserel->rtekind == RTE_RELATION); + + foreach(lc, indexlist) + { + IndexOptInfo *ind = (IndexOptInfo *) lfirst(lc); + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + matched_uniq_indexes = lappend(matched_uniq_indexes, ind); + } + + if (matched_uniq_indexes == NIL) + return; + + /* Check which attrs is used in baserel->reltarget */ + pull_varattnos((Node *)baserel->reltarget->exprs, baserel->relid, &used_attrs); + + /* Check which attrno is used at a mergeable const filter */ + foreach(lc, baserel->baserestrictinfo) + { + RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); + + if (rinfo->mergeopfamilies == NIL) + continue; + + if (bms_is_empty(rinfo->left_relids)) + { + const_exprs = lappend(const_exprs, get_rightop(rinfo->clause)); + } + else if (bms_is_empty(rinfo->right_relids)) + { + const_exprs = lappend(const_exprs, get_leftop(rinfo->clause)); + } + else + continue; + + expr_opfamilies = lappend(expr_opfamilies, rinfo->mergeopfamilies); + } + + foreach(lc, matched_uniq_indexes) + { + bool multi_nullvals, useful; + List *exprs = get_exprs_from_uniqueindex(lfirst_node(IndexOptInfo, lc), + const_exprs, + expr_opfamilies, + used_attrs, + &useful, + &multi_nullvals); + if (useful) + { + if (exprs == NIL) + { + /* All the columns in Unique Index matched with a restrictinfo */ + add_uniquekey_for_onerow(baserel); + return; + } + baserel->uniquekeys = lappend(baserel->uniquekeys, + makeUniqueKey(exprs, multi_nullvals)); + } + } +} + + +/* + * populate_partitionedrel_uniquekeys + * The UniqueKey on partitionrel comes from 2 cases: + * 1). Only one partition is involved in this query, the unique key can be + * copied to parent rel from childrel. + * 2). There are some unique index which includes partition key and exists + * in all the related partitions. + * We never mind rule 2 if we hit rule 1. + */ + +void +populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels) +{ + ListCell *lc; + List *global_uniq_indexlist = NIL; + RelOptInfo *childrel; + bool is_first = true; + + Assert(IS_PARTITIONED_REL(rel)); + + if (childrels == NIL) + return; + + /* + * If there is only one partition used in this query, the UniqueKey in childrel is + * still valid in parent level, but we need convert the format from child expr to + * parent expr. + */ + if (list_length(childrels) == 1) + { + /* Check for Rule 1 */ + RelOptInfo *childrel = linitial_node(RelOptInfo, childrels); + ListCell *lc; + Assert(childrel->reloptkind == RELOPT_OTHER_MEMBER_REL); + if (relation_is_onerow(childrel)) + { + add_uniquekey_for_onerow(rel); + return; + } + + foreach(lc, childrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + AppendRelInfo *appinfo = find_appinfo_by_child(root, childrel->relid); + List *parent_exprs = NIL; + bool can_reuse = true; + ListCell *lc2; + foreach(lc2, ukey->exprs) + { + Var *var = (Var *)lfirst(lc2); + /* + * If the expr comes from a expression, it is hard to build the expression + * in parent so ignore that case for now. + */ + if(!IsA(var, Var)) + { + can_reuse = false; + break; + } + /* Convert it to parent var */ + parent_exprs = lappend(parent_exprs, find_parent_var(appinfo, var)); + } + if (can_reuse) + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(parent_exprs, + ukey->multi_nullvals)); + } + } + else + { + /* Check for rule 2 */ + childrel = linitial_node(RelOptInfo, childrels); + foreach(lc, childrel->indexlist) + { + IndexOptInfo *ind = lfirst(lc); + IndexOptInfo *modified_index; + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + + /* + * During simple_copy_indexinfo_to_parent, we need to convert var from + * child var to parent var, index on expression is too complex to handle. + * so ignore it for now. + */ + if (ind->indexprs != NIL) + continue; + + modified_index = simple_copy_indexinfo_to_parent(root, rel, ind); + /* + * If the unique index doesn't contain partkey, then it is unique + * on this partition only, so it is useless for us. + */ + if (!index_constains_partkey(rel, modified_index)) + continue; + + global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index); + } + + if (global_uniq_indexlist != NIL) + { + foreach(lc, childrels) + { + RelOptInfo *child = lfirst(lc); + if (is_first) + { + is_first = false; + continue; + } + adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist); + } + /* Now we have a list of unique index which are exactly same on all childrels, + * Set the UniqueKey just like it is non-partition table + */ + populate_baserel_uniquekeys(root, rel, global_uniq_indexlist); + } + } +} + + +/* + * populate_distinctrel_uniquekeys + */ +void +populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel) +{ + /* The unique key before the distinct is still valid. */ + distinctrel->uniquekeys = list_copy(inputrel->uniquekeys); + add_uniquekey_from_sortgroups(root, distinctrel, root->parse->distinctClause); +} + +/* + * populate_grouprel_uniquekeys + */ +void +populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel) + +{ + Query *parse = root->parse; + bool input_ukey_added = false; + ListCell *lc; + + if (relation_is_onerow(inputrel)) + { + add_uniquekey_for_onerow(grouprel); + return; + } + if (parse->groupingSets) + return; + + /* A Normal group by without grouping set. */ + if (parse->groupClause) + { + /* + * Current even the groupby clause is Unique already, but if query has aggref + * We have to create grouprel still. To keep the UnqiueKey short, we will check + * the UniqueKey of input_rel still valid, if so we reuse it. + */ + foreach(lc, inputrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (list_is_subset(ukey->exprs, grouprel->reltarget->exprs)) + { + grouprel->uniquekeys = lappend(grouprel->uniquekeys, + ukey); + input_ukey_added = true; + } + } + if (!input_ukey_added) + /* + * group by clause must be a super-set of grouprel->reltarget->exprs except the + * aggregation expr, so if such exprs is unique already, no bother to generate + * new uniquekey for group by exprs. + */ + add_uniquekey_from_sortgroups(root, + grouprel, + root->parse->groupClause); + } + else + /* It has aggregation but without a group by, so only one row returned */ + add_uniquekey_for_onerow(grouprel); +} + +/* + * simple_copy_uniquekeys + * Using a function for the one-line code makes us easy to check where we simply + * copied the uniquekey. + */ +void +simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel) +{ + newrel->uniquekeys = oldrel->uniquekeys; +} + +/* + * populate_unionrel_uniquekeys + */ +void +populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel) +{ + ListCell *lc; + List *exprs = NIL; + + Assert(unionrel->uniquekeys == NIL); + + foreach(lc, unionrel->reltarget->exprs) + { + exprs = lappend(exprs, lfirst(lc)); + } + + if (exprs == NIL) + /* SQL: select union select; is valid, we need to handle it here. */ + add_uniquekey_for_onerow(unionrel); + else + unionrel->uniquekeys = lappend(unionrel->uniquekeys, + makeUniqueKey(exprs,false)); + +} + +/* + * populate_joinrel_uniquekeys + * + * populate uniquekeys for joinrel. We will check each relation to see if its + * UniqueKey is still valid via innerrel_keeps_unique, if so, we add it to + * joinrel. The multi_nullvals field will be changed to true for some outer + * join cases and one-row UniqueKey needs to be converted to normal UniqueKey + * for the same case as well. + * For the uniquekey in either baserel which can't be unique after join, we still + * check to see if combination of UniqueKeys from both side is still useful for us. + * if yes, we add it to joinrel as well. + */ +void +populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *outerrel, RelOptInfo *innerrel, + List *restrictlist, JoinType jointype) +{ + ListCell *lc, *lc2; + List *clause_list = NIL; + List *outerrel_ukey_ctx; + List *innerrel_ukey_ctx; + bool inner_onerow, outer_onerow; + bool mergejoin_allowed; + + /* Care about the outerrel relation only for SEMI/ANTI join */ + if (jointype == JOIN_SEMI || jointype == JOIN_ANTI) + { + foreach(lc, outerrel->uniquekeys) + { + UniqueKey *uniquekey = lfirst_node(UniqueKey, lc); + if (list_is_subset(uniquekey->exprs, joinrel->reltarget->exprs)) + joinrel->uniquekeys = lappend(joinrel->uniquekeys, uniquekey); + } + return; + } + + Assert(jointype == JOIN_LEFT || jointype == JOIN_FULL || jointype == JOIN_INNER); + + /* Fast path */ + if (innerrel->uniquekeys == NIL || outerrel->uniquekeys == NIL) + return; + + inner_onerow = relation_is_onerow(innerrel); + outer_onerow = relation_is_onerow(outerrel); + + outerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(outerrel); + innerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(innerrel); + + clause_list = select_mergejoin_clauses(root, joinrel, outerrel, innerrel, + restrictlist, jointype, + &mergejoin_allowed); + + if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true /* reverse */)) + { + bool outer_impact = jointype == JOIN_FULL; + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + /* Outer relation has one row, and the unique key is not duplicated after join, + * the joinrel will still has one row unless the jointype == JOIN_FULL. + */ + if (outer_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (outer_onerow) + { + /* + * The onerow outerrel becomes multi rows and multi_nullvals + * will be changed to true. We also need to set the exprs correctly since it + * can't be NIL any more. + */ + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, outerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Change multi_nullvals to true due to the full join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, true)); + else + /* Just reuse it */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + } + ctx->added_to_joinrel = true; + } + } + + if (innerrel_keeps_unique(root, outerrel, innerrel, clause_list, false)) + { + bool outer_impact = jointype == JOIN_FULL || jointype == JOIN_LEFT;; + + foreach(lc, innerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + if (inner_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (inner_onerow) + { + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, innerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Need to change multi_nullvals to true due to the outer join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, + true)); + else + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + + } + ctx->added_to_joinrel = true; + } + } + + /* + * The combination of the UniqueKey from both sides is unique as well regardless + * of join type, but no bother to add it if its subset has been added to joinrel + * already or it is not useful for the joinrel. + */ + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx1 = (UniqueKeyContext) lfirst(lc); + if (ctx1->added_to_joinrel || !ctx1->useful) + continue; + foreach(lc2, innerrel_ukey_ctx) + { + UniqueKeyContext ctx2 = (UniqueKeyContext) lfirst(lc2); + if (ctx2->added_to_joinrel || !ctx2->useful) + continue; + if (add_combined_uniquekey(root, joinrel, outerrel, innerrel, + ctx1->uniquekey, ctx2->uniquekey, + jointype)) + /* If we set a onerow UniqueKey to joinrel, we don't need other. */ + return; + } + } +} + + +/* + * convert_subquery_uniquekeys + * + * Covert the UniqueKey in subquery to outer relation. + */ +void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel) +{ + ListCell *lc; + + if (sub_final_rel->uniquekeys == NIL) + return; + + if (relation_is_onerow(sub_final_rel)) + { + add_uniquekey_for_onerow(currel); + return; + } + + Assert(currel->subroot != NULL); + + foreach(lc, sub_final_rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + ListCell *lc; + List *exprs = NIL; + bool ukey_useful = true; + + /* One row case is handled above */ + Assert(ukey->exprs != NIL); + foreach(lc, ukey->exprs) + { + Var *var; + TargetEntry *tle = tlist_member(lfirst(lc), + currel->subroot->processed_tlist); + if (tle == NULL) + { + ukey_useful = false; + break; + } + var = find_var_for_subquery_tle(currel, tle); + if (var == NULL) + { + ukey_useful = false; + break; + } + exprs = lappend(exprs, var); + } + + if (ukey_useful) + currel->uniquekeys = lappend(currel->uniquekeys, + makeUniqueKey(exprs, + ukey->multi_nullvals)); + + } +} + +/* + * innerrel_keeps_unique + * + * Check if Unique key of the innerrel is valid after join. innerrel's UniqueKey + * will be still valid if innerrel's any-column mergeop outrerel's uniquekey + * exists in clause_list. + * + * Note: the clause_list must be a list of mergeable restrictinfo already. + */ +static bool +innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *clause_list, + bool reverse) +{ + ListCell *lc, *lc2, *lc3; + + if (outerrel->uniquekeys == NIL || innerrel->uniquekeys == NIL) + return false; + + /* Check if there is outerrel's uniquekey in mergeable clause. */ + foreach(lc, outerrel->uniquekeys) + { + List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs; + bool clauselist_matchs_all_exprs = true; + foreach(lc2, outer_uq_exprs) + { + Node *outer_uq_expr = lfirst(lc2); + bool find_uq_expr_in_clauselist = false; + foreach(lc3, clause_list) + { + RestrictInfo *rinfo = lfirst_node(RestrictInfo, lc3); + Node *outer_expr; + if (reverse) + outer_expr = rinfo->outer_is_left ? get_rightop(rinfo->clause) : get_leftop(rinfo->clause); + else + outer_expr = rinfo->outer_is_left ? get_leftop(rinfo->clause) : get_rightop(rinfo->clause); + if (equal(outer_expr, outer_uq_expr)) + { + find_uq_expr_in_clauselist = true; + break; + } + } + if (!find_uq_expr_in_clauselist) + { + /* No need to check the next exprs in the current uniquekey */ + clauselist_matchs_all_exprs = false; + break; + } + } + + if (clauselist_matchs_all_exprs) + return true; + } + return false; +} + + +/* + * relation_is_onerow + * Check if it is a one-row relation by checking UniqueKey. + */ +bool +relation_is_onerow(RelOptInfo *rel) +{ + UniqueKey *ukey; + if (rel->uniquekeys == NIL) + return false; + ukey = linitial_node(UniqueKey, rel->uniquekeys); + return ukey->exprs == NIL && list_length(rel->uniquekeys) == 1; +} + +/* + * relation_has_uniquekeys_for + * Returns true if we have proofs that 'rel' cannot return multiple rows with + * the same values in each of 'exprs'. Otherwise returns false. + */ +bool +relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel, + List *exprs, bool allow_multinulls) +{ + ListCell *lc; + + /* + * For UniqueKey->onerow case, the uniquekey->exprs is empty as well + * so we can't rely on list_is_subset to handle this special cases + */ + if (exprs == NIL) + return false; + + foreach(lc, rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (ukey->multi_nullvals && !allow_multinulls) + continue; + if (list_is_subset(ukey->exprs, exprs)) + return true; + } + return false; +} + + +/* + * get_exprs_from_uniqueindex + * + * Return a list of exprs which is unique. set useful to false if this + * unique index is not useful for us. + */ +static List * +get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals) +{ + List *exprs = NIL; + ListCell *indexpr_item; + int c = 0; + + *useful = true; + *multi_nullvals = false; + + indexpr_item = list_head(unique_index->indexprs); + for(c = 0; c < unique_index->ncolumns; c++) + { + int attr = unique_index->indexkeys[c]; + Expr *expr; + bool matched_const = false; + ListCell *lc1, *lc2; + + if(attr > 0) + { + expr = list_nth_node(TargetEntry, unique_index->indextlist, c)->expr; + } + else if (attr == 0) + { + /* Expression index */ + expr = lfirst(indexpr_item); + indexpr_item = lnext(unique_index->indexprs, indexpr_item); + } + else /* attr < 0 */ + { + /* Index on system column is not supported */ + Assert(false); + } + + /* + * Check index_col = Const case with regarding to opfamily checking + * If we can remove the index_col from the final UniqueKey->exprs. + */ + forboth(lc1, const_exprs, lc2, const_expr_opfamilies) + { + if (list_member_oid((List *)lfirst(lc2), unique_index->opfamily[c]) + && match_index_to_operand((Node *) lfirst(lc1), c, unique_index)) + { + matched_const = true; + break; + } + } + + if (matched_const) + continue; + + /* Check if the indexed expr is used in rel */ + if (attr > 0) + { + /* + * Normal Indexed column, if the col is not used, then the index is useless + * for uniquekey. + */ + attr -= FirstLowInvalidHeapAttributeNumber; + + if (!bms_is_member(attr, used_varattrs)) + { + *useful = false; + break; + } + } + else if (!list_member(unique_index->rel->reltarget->exprs, expr)) + { + /* Expression index but the expression is not used in rel */ + *useful = false; + break; + } + + /* check not null property. */ + if (attr == 0) + { + /* We never know if a expression yields null or not */ + *multi_nullvals = true; + } + else if (!bms_is_member(attr, unique_index->rel->notnullattrs) + && !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber, + unique_index->rel->notnullattrs)) + { + *multi_nullvals = true; + } + + exprs = lappend(exprs, expr); + } + return exprs; +} + + +/* + * add_uniquekey_for_onerow + * If we are sure that the relation only returns one row, then all the columns + * are unique. However we don't need to create UniqueKey for every column, we + * just set exprs = NIL and overwrites all the other UniqueKey on this RelOptInfo + * since this one has strongest semantics. + */ +void +add_uniquekey_for_onerow(RelOptInfo *rel) +{ + /* + * We overwrite the previous UniqueKey on purpose since this one has the + * strongest semantic. + */ + rel->uniquekeys = list_make1(makeUniqueKey(NIL, false)); +} + + +/* + * initililze_uniquecontext_for_joinrel + * Return a List of UniqueKeyContext for an inputrel + */ +static List * +initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel) +{ + List *res = NIL; + ListCell *lc; + foreach(lc, inputrel->uniquekeys) + { + UniqueKeyContext context; + context = palloc(sizeof(struct UniqueKeyContextData)); + context->uniquekey = lfirst_node(UniqueKey, lc); + context->added_to_joinrel = false; + context->useful = true; + res = lappend(res, context); + } + return res; +} + + +/* + * get_exprs_from_uniquekey + * Unify the way of get List of exprs from a one-row UniqueKey or + * normal UniqueKey. for the onerow case, every expr in rel1 is a valid + * UniqueKey. Return a List of exprs. + * + * rel1: The relation which you want to get the exprs. + * ukey: The UniqueKey you want to get the exprs. + */ +static List * +get_exprs_from_uniquekey(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *rel1, UniqueKey *ukey) +{ + ListCell *lc; + bool onerow = rel1 != NULL && relation_is_onerow(rel1); + + List *res = NIL; + Assert(onerow || ukey); + if (onerow) + { + /* Only cares about the exprs still exist in joinrel */ + foreach(lc, joinrel->reltarget->exprs) + { + Bitmapset *relids = pull_varnos(root, lfirst(lc)); + if (bms_is_subset(relids, rel1->relids)) + { + res = lappend(res, list_make1(lfirst(lc))); + } + } + } + else + { + res = list_make1(ukey->exprs); + } + return res; +} + +/* + * Partitioned table Unique Keys. + * The partition table unique key is maintained as: + * 1. The index must be unique as usual. + * 2. The index must contains partition key. + * 3. The index must exist on all the child rel. see simple_indexinfo_equal for + * how we compare it. + */ + +/* + * index_constains_partkey + * return true if the index contains the partiton key. + */ +static bool +index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind) +{ + ListCell *lc; + int i; + Assert(IS_PARTITIONED_REL(partrel)); + Assert(partrel->part_scheme->partnatts > 0); + + for(i = 0; i < partrel->part_scheme->partnatts; i++) + { + Node *part_expr = linitial(partrel->partexprs[i]); + bool found_in_index = false; + foreach(lc, ind->indextlist) + { + Expr *index_expr = lfirst_node(TargetEntry, lc)->expr; + if (equal(index_expr, part_expr)) + { + found_in_index = true; + break; + } + } + if (!found_in_index) + return false; + } + return true; +} + +/* + * simple_indexinfo_equal + * + * Used to check if the 2 index is same as each other. The index here + * is COPIED from childrel and did some tiny changes(see + * simple_copy_indexinfo_to_parent) + */ +static bool +simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2) +{ + Size oid_cmp_len = sizeof(Oid) * ind1->ncolumns; + + return ind1->ncolumns == ind2->ncolumns && + ind1->unique == ind2->unique && + memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0 && + memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0 && + memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0 && + memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0 && + equal(get_tlist_exprs(ind1->indextlist, true), + get_tlist_exprs(ind2->indextlist, true)); +} + + +/* + * The below macros are used for simple_copy_indexinfo_to_parent which is so + * customized that I don't want to put it to copyfuncs.c. So copy it here. + */ +#define COPY_POINTER_FIELD(fldname, sz) \ + do { \ + Size _size = (sz); \ + newnode->fldname = palloc(_size); \ + memcpy(newnode->fldname, from->fldname, _size); \ + } while (0) + +#define COPY_NODE_FIELD(fldname) \ + (newnode->fldname = copyObjectImpl(from->fldname)) + +#define COPY_SCALAR_FIELD(fldname) \ + (newnode->fldname = from->fldname) + + +/* + * simple_copy_indexinfo_to_parent (from partition) + * Copy the IndexInfo from child relation to parent relation with some modification, + * which is used to test: + * 1. If the same index exists in all the childrels. + * 2. If the parentrel->reltarget/basicrestrict info matches this index. + */ +static IndexOptInfo * +simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from) +{ + IndexOptInfo *newnode = makeNode(IndexOptInfo); + AppendRelInfo *appinfo = find_appinfo_by_child(root, from->rel->relid); + ListCell *lc; + int idx = 0; + + COPY_SCALAR_FIELD(ncolumns); + COPY_SCALAR_FIELD(nkeycolumns); + COPY_SCALAR_FIELD(unique); + COPY_SCALAR_FIELD(immediate); + /* We just need to know if it is NIL or not */ + COPY_SCALAR_FIELD(indpred); + COPY_SCALAR_FIELD(predOK); + COPY_POINTER_FIELD(indexkeys, from->ncolumns * sizeof(int)); + COPY_POINTER_FIELD(indexcollations, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opfamily, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opcintype, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(sortopfamily, from->ncolumns * sizeof(Oid)); + COPY_NODE_FIELD(indextlist); + + /* Convert index exprs on child expr to expr on parent */ + foreach(lc, newnode->indextlist) + { + TargetEntry *tle = lfirst_node(TargetEntry, lc); + /* Index on expression is ignored */ + Assert(IsA(tle->expr, Var)); + tle->expr = (Expr *) find_parent_var(appinfo, (Var *) tle->expr); + newnode->indexkeys[idx] = castNode(Var, tle->expr)->varattno; + idx++; + } + newnode->rel = parentrel; + return newnode; +} + +/* + * adjust_partition_unique_indexlist + * + * global_unique_indexes: At the beginning, it contains the copy & modified + * unique index from the first partition. And then check if each index in it still + * exists in the following partitions. If no, remove it. at last, it has an + * index list which exists in all the partitions. + */ +static void +adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_indexes) +{ + ListCell *lc, *lc2; + foreach(lc, *global_unique_indexes) + { + IndexOptInfo *g_ind = lfirst_node(IndexOptInfo, lc); + bool found_in_child = false; + + foreach(lc2, childrel->indexlist) + { + IndexOptInfo *p_ind = lfirst_node(IndexOptInfo, lc2); + IndexOptInfo *p_ind_copy; + if (!p_ind->unique || !p_ind->immediate || + (p_ind->indpred != NIL && !p_ind->predOK)) + continue; + p_ind_copy = simple_copy_indexinfo_to_parent(root, parentrel, p_ind); + if (simple_indexinfo_equal(p_ind_copy, g_ind)) + { + found_in_child = true; + break; + } + } + if (!found_in_child) + /* The index doesn't exist in childrel, remove it from global_unique_indexes */ + *global_unique_indexes = foreach_delete_current(*global_unique_indexes, lc); + } +} + +/* Helper function for groupres/distinctrel */ +static void +add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgroups) +{ + Query *parse = root->parse; + List *exprs; + + /* + * XXX: If there are some vars which is not in current levelsup, the semantic is + * imprecise, should we avoid it or not? levelsup = 1 is just a demo, maybe we need to + * check every level other than 0, if so, looks we have to write another + * pull_var_walker. + */ + List *upper_vars = pull_vars_of_level((Node*)sortgroups, 1); + + if (upper_vars != NIL) + return; + + exprs = get_sortgrouplist_exprs(sortgroups, parse->targetList); + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(exprs, + false /* sortgroupclause can't be multi_nullvals */)); +} + + +/* + * add_combined_uniquekey + * The combination of both UniqueKeys is a valid UniqueKey for joinrel no matter + * the jointype. + */ +bool +add_combined_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype) +{ + + ListCell *lc1, *lc2; + + /* Either side has multi_nullvals or we have outer join, + * the combined UniqueKey has multi_nullvals */ + bool multi_nullvals = outer_ukey->multi_nullvals || + inner_ukey->multi_nullvals || IS_OUTER_JOIN(jointype); + + /* The only case we can get onerow joinrel after join */ + if (relation_is_onerow(outer_rel) + && relation_is_onerow(inner_rel) + && jointype == JOIN_INNER) + { + add_uniquekey_for_onerow(joinrel); + return true; + } + + foreach(lc1, get_exprs_from_uniquekey(root, joinrel, outer_rel, outer_ukey)) + { + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, inner_rel, inner_ukey)) + { + List *exprs = list_concat_copy(lfirst_node(List, lc1), lfirst_node(List, lc2)); + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(exprs, + multi_nullvals)); + } + } + return false; +} diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index 424d25cbd5..8d8e493f5c 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -2387,6 +2387,8 @@ grouping_planner(PlannerInfo *root, bool inheritance_update, add_path(final_rel, path); } + simple_copy_uniquekeys(current_rel, final_rel); + /* * Generate partial paths for final_rel, too, if outer query levels might * be able to make use of them. @@ -3899,6 +3901,8 @@ create_grouping_paths(PlannerInfo *root, } set_cheapest(grouped_rel); + + populate_grouprel_uniquekeys(root, grouped_rel, input_rel); return grouped_rel; } @@ -4621,7 +4625,7 @@ create_window_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(window_rel); - + simple_copy_uniquekeys(input_rel, window_rel); return window_rel; } @@ -4935,7 +4939,7 @@ create_distinct_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(distinct_rel); - + populate_distinctrel_uniquekeys(root, input_rel, distinct_rel); return distinct_rel; } @@ -5196,6 +5200,8 @@ create_ordered_paths(PlannerInfo *root, */ Assert(ordered_rel->pathlist != NIL); + simple_copy_uniquekeys(input_rel, ordered_rel); + return ordered_rel; } @@ -6073,6 +6079,9 @@ adjust_paths_for_srfs(PlannerInfo *root, RelOptInfo *rel, if (list_length(targets) == 1) return; + /* UniqueKey is not valid after handling the SRF. */ + rel->uniquekeys = NIL; + /* * Stack SRF-evaluation nodes atop each path for the rel. * diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c index becdcbb872..b7626545bf 100644 --- a/src/backend/optimizer/prep/prepunion.c +++ b/src/backend/optimizer/prep/prepunion.c @@ -689,6 +689,8 @@ generate_union_paths(SetOperationStmt *op, PlannerInfo *root, /* Undo effects of possibly forcing tuple_fraction to 0 */ root->tuple_fraction = save_fraction; + /* Add the UniqueKeys */ + populate_unionrel_uniquekeys(root, result_rel); return result_rel; } diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c index 86922a273c..6817c9c787 100644 --- a/src/backend/optimizer/util/appendinfo.c +++ b/src/backend/optimizer/util/appendinfo.c @@ -746,3 +746,47 @@ find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos) } return appinfos; } + +/* + * find_appinfo_by_child + * + */ +AppendRelInfo * +find_appinfo_by_child(PlannerInfo *root, Index child_index) +{ + ListCell *lc; + foreach(lc, root->append_rel_list) + { + AppendRelInfo *appinfo = lfirst_node(AppendRelInfo, lc); + if (appinfo->child_relid == child_index) + return appinfo; + } + elog(ERROR, "parent relation cant be found"); + return NULL; +} + +/* + * find_parent_var + * + */ +Var * +find_parent_var(AppendRelInfo *appinfo, Var *child_var) +{ + ListCell *lc; + Var *res = NULL; + Index attno = 1; + foreach(lc, appinfo->translated_vars) + { + Node *child_node = lfirst(lc); + if (equal(child_node, child_var)) + { + res = copyObject(child_var); + res->varattno = attno; + res->varno = appinfo->parent_relid; + } + attno++; + } + if (res == NULL) + elog(ERROR, "parent var can't be found."); + return res; +} diff --git a/src/backend/optimizer/util/inherit.c b/src/backend/optimizer/util/inherit.c index be1c9ddd96..3eec1f4d74 100644 --- a/src/backend/optimizer/util/inherit.c +++ b/src/backend/optimizer/util/inherit.c @@ -736,6 +736,7 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, { Node *onecq = (Node *) lfirst(lc2); bool pseudoconstant; + RestrictInfo *child_rinfo; /* check for pseudoconstant (no Vars or volatile functions) */ pseudoconstant = @@ -747,14 +748,15 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, root->hasPseudoConstantQuals = true; } /* reconstitute RestrictInfo with appropriate properties */ - childquals = lappend(childquals, - make_restrictinfo(root, - (Expr *) onecq, - rinfo->is_pushed_down, - rinfo->outerjoin_delayed, - pseudoconstant, - rinfo->security_level, - NULL, NULL, NULL)); + child_rinfo = make_restrictinfo(root, + (Expr *) onecq, + rinfo->is_pushed_down, + rinfo->outerjoin_delayed, + pseudoconstant, + rinfo->security_level, + NULL, NULL, NULL); + child_rinfo->mergeopfamilies = rinfo->mergeopfamilies; + childquals = lappend(childquals, child_rinfo); /* track minimum security level among child quals */ cq_min_security = Min(cq_min_security, rinfo->security_level); } diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 48a7ebfe45..4fe1824eb0 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -16,6 +16,7 @@ #include "nodes/execnodes.h" #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" extern A_Expr *makeA_Expr(A_Expr_Kind kind, List *name, @@ -106,4 +107,6 @@ extern GroupingSet *makeGroupingSet(GroupingSetKind kind, List *content, int loc extern VacuumRelation *makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols); +extern UniqueKey* makeUniqueKey(List *exprs, bool multi_nullvals); + #endif /* MAKEFUNC_H */ diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index e22df890ef..3320273ac1 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -265,6 +265,7 @@ typedef enum NodeTag T_EquivalenceMember, T_PathKey, T_PathTarget, + T_UniqueKey, T_RestrictInfo, T_IndexClause, T_PlaceHolderVar, diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index 0d61f04d27..80561b1003 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -739,6 +739,7 @@ typedef struct RelOptInfo QualCost baserestrictcost; /* cost of evaluating the above */ Index baserestrict_min_security; /* min security_level found in * baserestrictinfo */ + List *uniquekeys; /* List of UniqueKey */ List *joininfo; /* RestrictInfo structures for join clauses * involving this rel */ bool has_eclass_joins; /* T means joininfo is incomplete */ @@ -1059,6 +1060,28 @@ typedef struct PathKey } PathKey; +/* + * UniqueKey + * + * Represents the unique properties held by a RelOptInfo. + * + * exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL + * is a special case of UniqueKey, which means there is only 1 row in that + * relation. + * multi_nullvals: true means multi null values may exist in these exprs, so the + * uniqueness is not guaranteed in this case. This field is necessary for + * remove_useless_join & reduce_unique_semijoins where we don't mind these + * duplicated NULL values. It is set to true for 2 cases. One is a unique key + * from a unique index but the related column is nullable. The other one is for + * outer join. see populate_joinrel_uniquekeys for detail. + */ +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + /* * PathTarget * @@ -2496,7 +2519,7 @@ typedef enum * * flags indicating what kinds of grouping are possible. * partial_costs_set is true if the agg_partial_costs and agg_final_costs - * have been initialized. + * have been initialized. * agg_partial_costs gives partial aggregation costs. * agg_final_costs gives finalization costs. * target_parallel_safe is true if target is parallel safe. @@ -2526,8 +2549,8 @@ typedef struct * limit_tuples is an estimated bound on the number of output tuples, * or -1 if no LIMIT or couldn't estimate. * count_est and offset_est are the estimated values of the LIMIT and OFFSET - * expressions computed by preprocess_limit() (see comments for - * preprocess_limit() for more information). + * expressions computed by preprocess_limit() (see comments for + * preprocess_limit() for more information). */ typedef struct { diff --git a/src/include/nodes/pg_list.h b/src/include/nodes/pg_list.h index 404e03f132..85c65e910c 100644 --- a/src/include/nodes/pg_list.h +++ b/src/include/nodes/pg_list.h @@ -566,6 +566,16 @@ extern pg_nodiscard List *list_delete_first(List *list); extern pg_nodiscard List *list_delete_last(List *list); extern pg_nodiscard List *list_delete_nth_cell(List *list, int n); extern pg_nodiscard List *list_delete_cell(List *list, ListCell *cell); +extern bool list_is_subset(const List *members, const List *target); + +extern List *list_delete(List *list, void *datum); +extern List *list_delete_ptr(List *list, void *datum); +extern List *list_delete_int(List *list, int datum); +extern List *list_delete_oid(List *list, Oid datum); +extern List *list_delete_first(List *list); +extern List *list_delete_last(List *list); +extern List *list_delete_nth_cell(List *list, int n); +extern List *list_delete_cell(List *list, ListCell *cell); extern List *list_union(const List *list1, const List *list2); extern List *list_union_ptr(const List *list1, const List *list2); diff --git a/src/include/optimizer/appendinfo.h b/src/include/optimizer/appendinfo.h index 4cbf8c26cc..43b8d05f69 100644 --- a/src/include/optimizer/appendinfo.h +++ b/src/include/optimizer/appendinfo.h @@ -32,4 +32,7 @@ extern Relids adjust_child_relids_multilevel(PlannerInfo *root, Relids relids, extern AppendRelInfo **find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos); +extern AppendRelInfo *find_appinfo_by_child(PlannerInfo *root, Index child_index); +extern Var *find_parent_var(AppendRelInfo *appinfo, Var *child_var); + #endif /* APPENDINFO_H */ diff --git a/src/include/optimizer/optimizer.h b/src/include/optimizer/optimizer.h index d587952b7d..843aafa51e 100644 --- a/src/include/optimizer/optimizer.h +++ b/src/include/optimizer/optimizer.h @@ -23,6 +23,7 @@ #define OPTIMIZER_H #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" /* Test if an expression node represents a SRF call. Beware multiple eval! */ #define IS_SRF_CALL(node) \ @@ -168,6 +169,7 @@ extern TargetEntry *get_sortgroupref_tle(Index sortref, List *targetList); extern TargetEntry *get_sortgroupclause_tle(SortGroupClause *sgClause, List *targetList); +extern Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); extern Node *get_sortgroupclause_expr(SortGroupClause *sgClause, List *targetList); extern List *get_sortgrouplist_exprs(List *sgClauses, diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h index 035d3e1206..1adf99c9ee 100644 --- a/src/include/optimizer/paths.h +++ b/src/include/optimizer/paths.h @@ -247,5 +247,48 @@ extern PathKey *make_canonical_pathkey(PlannerInfo *root, int strategy, bool nulls_first); extern void add_paths_to_append_rel(PlannerInfo *root, RelOptInfo *rel, List *live_childrels); +extern List *select_mergejoin_clauses(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + JoinType jointype, + bool *mergejoin_allowed); + +/* + * uniquekeys.c + * Utilities for matching and building unique keys + */ +extern void populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List* unique_index_list); +extern void populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels); +extern void populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel); +extern void populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel); +extern void populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel); +extern void simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel); +extern void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel); +extern void populate_joinrel_uniquekeys(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + RelOptInfo *rel2, + List *restrictlist, + JoinType jointype); + +extern bool relation_has_uniquekeys_for(PlannerInfo *root, + RelOptInfo *rel, + List *exprs, + bool allow_multinulls); +extern bool relation_is_onerow(RelOptInfo *rel); #endif /* PATHS_H */ -- 2.30.2 --------------F495F6B18672582269F00FF9 Content-Type: text/x-patch; charset=UTF-8; name="0004-review-20210317.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0004-review-20210317.patch" ^ permalink raw reply [nested|flat] 91+ messages in thread
* [PATCH v38 2/6] Introduce UniqueKey attributes on RelOptInfo struct. @ 2020-05-11 07:50 一挃 <[email protected]> 0 siblings, 0 replies; 91+ messages in thread From: 一挃 @ 2020-05-11 07:50 UTC (permalink / raw) UniqueKey is a set of exprs on RelOptInfo which represents the exprs will be unique on the given RelOptInfo. You can see README.uniquekey for more information. --- src/backend/nodes/copyfuncs.c | 13 + src/backend/nodes/list.c | 31 + src/backend/nodes/makefuncs.c | 13 + src/backend/nodes/outfuncs.c | 11 + src/backend/nodes/readfuncs.c | 10 + src/backend/optimizer/path/Makefile | 3 +- src/backend/optimizer/path/README.uniquekey | 131 +++ src/backend/optimizer/path/allpaths.c | 10 + src/backend/optimizer/path/joinpath.c | 9 +- src/backend/optimizer/path/joinrels.c | 2 + src/backend/optimizer/path/pathkeys.c | 3 +- src/backend/optimizer/path/uniquekeys.c | 1135 +++++++++++++++++++ src/backend/optimizer/plan/planner.c | 13 +- src/backend/optimizer/prep/prepunion.c | 2 + src/backend/optimizer/util/appendinfo.c | 44 + src/backend/optimizer/util/inherit.c | 18 +- src/include/nodes/makefuncs.h | 3 + src/include/nodes/nodes.h | 1 + src/include/nodes/pathnodes.h | 29 +- src/include/nodes/pg_list.h | 10 + src/include/optimizer/appendinfo.h | 3 + src/include/optimizer/optimizer.h | 2 + src/include/optimizer/paths.h | 43 + 23 files changed, 1515 insertions(+), 24 deletions(-) create mode 100644 src/backend/optimizer/path/README.uniquekey create mode 100644 src/backend/optimizer/path/uniquekeys.c diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..75c1c5e824 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2296,6 +2296,16 @@ _copyPathKey(const PathKey *from) return newnode; } +static UniqueKey * +_copyUniqueKey(const UniqueKey *from) +{ + UniqueKey *newnode = makeNode(UniqueKey); + + COPY_NODE_FIELD(exprs); + COPY_SCALAR_FIELD(multi_nullvals); + + return newnode; +} /* * _copyRestrictInfo */ @@ -5220,6 +5230,9 @@ copyObjectImpl(const void *from) case T_PathKey: retval = _copyPathKey(from); break; + case T_UniqueKey: + retval = _copyUniqueKey(from); + break; case T_RestrictInfo: retval = _copyRestrictInfo(from); break; diff --git a/src/backend/nodes/list.c b/src/backend/nodes/list.c index dbf6b30233..ca099495a1 100644 --- a/src/backend/nodes/list.c +++ b/src/backend/nodes/list.c @@ -702,6 +702,37 @@ list_member_oid(const List *list, Oid datum) return false; } +/* + * return true iff every entry in "members" list is also present + * in the "target" list. + */ +bool +list_is_subset(const List *members, const List *target) +{ + const ListCell *lc1, *lc2; + + Assert(IsPointerList(members)); + Assert(IsPointerList(target)); + check_list_invariants(members); + check_list_invariants(target); + + foreach(lc1, members) + { + bool found = false; + foreach(lc2, target) + { + if (equal(lfirst(lc1), lfirst(lc2))) + { + found = true; + break; + } + } + if (!found) + return false; + } + return true; +} + /* * Delete the n'th cell (counting from 0) in list. * diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 01c110cd2f..40415d0f5b 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -815,3 +815,16 @@ makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols) v->va_cols = va_cols; return v; } + + +/* + * makeUniqueKey + */ +UniqueKey* +makeUniqueKey(List *exprs, bool multi_nullvals) +{ + UniqueKey * ukey = makeNode(UniqueKey); + ukey->exprs = exprs; + ukey->multi_nullvals = multi_nullvals; + return ukey; +} diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..44154cde6a 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2456,6 +2456,14 @@ _outPathKey(StringInfo str, const PathKey *node) WRITE_BOOL_FIELD(pk_nulls_first); } +static void +_outUniqueKey(StringInfo str, const UniqueKey *node) +{ + WRITE_NODE_TYPE("UNIQUEKEY"); + WRITE_NODE_FIELD(exprs); + WRITE_BOOL_FIELD(multi_nullvals); +} + static void _outPathTarget(StringInfo str, const PathTarget *node) { @@ -4198,6 +4206,9 @@ outNode(StringInfo str, const void *obj) case T_PathKey: _outPathKey(str, obj); break; + case T_UniqueKey: + _outUniqueKey(str, obj); + break; case T_PathTarget: _outPathTarget(str, obj); break; diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index c5e136e9c3..b3e212bf1c 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -492,6 +492,14 @@ _readSetOperationStmt(void) READ_DONE(); } +static UniqueKey * +_readUniqueKey(void) +{ + READ_LOCALS(UniqueKey); + READ_NODE_FIELD(exprs); + READ_BOOL_FIELD(multi_nullvals); + READ_DONE(); +} /* * Stuff from primnodes.h. @@ -2717,6 +2725,8 @@ parseNodeString(void) return_value = _readCommonTableExpr(); else if (MATCH("SETOPERATIONSTMT", 16)) return_value = _readSetOperationStmt(); + else if (MATCH("UNIQUEKEY", 9)) + return_value = _readUniqueKey(); else if (MATCH("ALIAS", 5)) return_value = _readAlias(); else if (MATCH("RANGEVAR", 8)) diff --git a/src/backend/optimizer/path/Makefile b/src/backend/optimizer/path/Makefile index 1e199ff66f..7b9820c25f 100644 --- a/src/backend/optimizer/path/Makefile +++ b/src/backend/optimizer/path/Makefile @@ -21,6 +21,7 @@ OBJS = \ joinpath.o \ joinrels.o \ pathkeys.o \ - tidpath.o + tidpath.o \ + uniquekeys.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/optimizer/path/README.uniquekey b/src/backend/optimizer/path/README.uniquekey new file mode 100644 index 0000000000..5eac761995 --- /dev/null +++ b/src/backend/optimizer/path/README.uniquekey @@ -0,0 +1,131 @@ +1. What is UniqueKey? +We can think UniqueKey is a set of exprs for a RelOptInfo, which we are insure +that doesn't yields same result among all the rows. The simplest UniqueKey +format is primary key. + +However we define the UnqiueKey as below. + +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + +exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL +is a special case of UniqueKey, which means there is only one row in that +relation.it has a stronger semantic than others. like SELECT uk FROM t; uk is +normal unique key and may have different values. SELECT colx FROM t WHERE uk = +const. colx is unique AND we have only 1 value. This field can used for +innerrel_is_unique. this logic is handled specially in add_uniquekey_for_onerow +function. + +multi_nullvals: true means multi null values may exist in these exprs, so the +uniqueness is not guaranteed in this case. This field is necessary for +remove_useless_join & reduce_unique_semijoins where we don't mind these +duplicated NULL values. It is set to true for 2 cases. One is a unique key +from a unique index but the related column is nullable. The other one is for +outer join. see populate_joinrel_uniquekeys for detail. + + +The UniqueKey can be used at the following cases at least: +1. remove_useless_joins. +2. reduce_semianti_joins +3. remove distinct node if distinct clause is unique. +4. remove aggnode if group by clause is unique. +5. Index Skip Scan (WIP) +6. Aggregation Push Down without 2 phase aggregation if the join can't + duplicated the aggregated rows. (work in progress feature) + +2. How is it maintained? + +We have a set of populate_xxx_unqiuekeys functions to maintain the uniquekey on +various cases. xxx includes baserel, joinrel, partitionedrel, distinctrel, +groupedrel, unionrel. and we also need to convert the uniquekey from subquery +to outer relation, which is what convert_subquery_uniquekeys does. + +1. The first part is about baserel. We handled 3 cases. suppose we have Unique +Index on (a, b). + +1. SELECT a, b FROM t. UniqueKey (a, b) +2. SELECT a FROM t WHERE b = 1; UniqueKey (a) +3. SELECT .. FROM t WHERE a = 1 AND b = 1; UniqueKey (NIL). onerow case, every + column is Unique. + +2. The next part is joinrel, this part is most error-prone, we simplified the rules +like below: +1. If the relation's UniqueKey can't be duplicated after join, then is will be + still valid for the join rel. The function we used here is + innerrel_keeps_unique. The basic idea is innerrel.any_col = outer.uk. + +2. If the UnqiueKey can't keep valid via the rule 1, the combination of the + UniqueKey from both sides are valid for sure. We can prove this as: if the + unique exprs from rel1 is duplicated by rel2, the duplicated rows must + contains different unique exprs from rel2. + +More considerations about onerow: +1. If relation with one row and it can't be duplicated, it is still possible + contains mulit_nullvas after outer join. +2. If the either UniqueKey can be duplicated after join, the can get one row + only when both side is one row AND there is no outer join. +3. Whenever the onerow UniqueKey is not a valid any more, we need to convert one + row UniqueKey to normal unique key since we don't store exprs for one-row + relation. get_exprs_from_uniquekeys will be used here. + + +More considerations about multi_nullvals after join: +1. If the original UnqiueKey has multi_nullvals, the final UniqueKey will have + mulit_nullvals in any case. +2. If a unique key doesn't allow mulit_nullvals, after some outer join, it + allows some outer join. + + +3. When we comes to subquery, we need to convert_subquery_unqiuekeys just like +convert_subquery_pathkeys. Only the UniqueKey insides subquery is referenced as +a Var in outer relation will be reused. The relationship between the outerrel.Var +and subquery.exprs is built with outerel->subroot->processed_tlist. + + +4. As for the SRF functions, it will break the uniqueness of uniquekey, However it +is handled in adjust_paths_for_srfs, which happens after the query_planner. so +we will maintain the UniqueKey until there and reset it to NIL at that +places. This can't help on distinct/group by elimination cases but probably help +in some other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is +semantic correctly. + + +5. As for inherit table, we first main the UnqiueKey on childrel as well. But for +partitioned table we need to maintain 2 different kinds of +UnqiueKey. 1). UniqueKey on the parent relation 2). UniqueKey on child +relation for partition wise query. + +Example: +CREATE TABLE p (a int not null, b int not null) partition by list (a); +CREATE TABLE p0 partition of p for values in (1); +CREATE TABLE p1 partition of p for values in (2); + +create unique index p0_b on p0(b); +create unique index p1_b on p1(b); + +Now b is only unique on partition level, so the distinct can't be removed on +the following cases. SELECT DISTINCT b FROM p; + +Another example is SELECT DISTINCT a, b FROM p WHERE a = 1; Since only one +partition is chosen, the UniqueKey on child relation is same as the UniqueKey on +parent relation. + +Another usage of UniqueKey on partition level is it be helpful for +partition-wise join. + +As for the UniqueKey on parent table level, it comes with 2 different ways, +1). the UniqueKey is also derived in UniqueKey index, but the index must be same +in all the related children relations and the unique index must contains +Partition Key in it. Example: + +CREATE UNIQUE INDEX p_ab ON p(a, b); -- where a is the partition key. + +-- Query +SELECT a, b FROM p; the (a, b) is a UniqueKey of p. + +2). If the parent relation has only one childrel, the UniqueKey on childrel is + the UniqueKey on parent as well. diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index 37b4223adb..d52ad59f89 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -579,6 +579,12 @@ set_plain_rel_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) */ check_index_predicates(root, rel); + /* + * Now that we've marked which partial indexes are suitable, we can now + * build the relation's unique keys. + */ + populate_baserel_uniquekeys(root, rel, rel->indexlist); + /* Mark rel with estimated output rows, width, etc */ set_baserel_size_estimates(root, rel); } @@ -1297,6 +1303,8 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, /* Add paths to the append relation. */ add_paths_to_append_rel(root, rel, live_childrels); + if (IS_PARTITIONED_REL(rel)) + populate_partitionedrel_uniquekeys(root, rel, live_childrels); } @@ -2304,6 +2312,8 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, pathkeys, required_outer)); } + convert_subquery_uniquekeys(root, rel, sub_final_rel); + /* If outer rel allows parallelism, do same for partial paths. */ if (rel->consider_parallel && bms_is_empty(required_outer)) { diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c index 57ce97fd53..697ac047fb 100644 --- a/src/backend/optimizer/path/joinpath.c +++ b/src/backend/optimizer/path/joinpath.c @@ -71,13 +71,6 @@ static void consider_parallel_mergejoin(PlannerInfo *root, static void hash_inner_and_outer(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, RelOptInfo *innerrel, JoinType jointype, JoinPathExtraData *extra); -static List *select_mergejoin_clauses(PlannerInfo *root, - RelOptInfo *joinrel, - RelOptInfo *outerrel, - RelOptInfo *innerrel, - List *restrictlist, - JoinType jointype, - bool *mergejoin_allowed); static void generate_mergejoin_paths(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *innerrel, @@ -1927,7 +1920,7 @@ hash_inner_and_outer(PlannerInfo *root, * if it is mergejoinable and involves vars from the two sub-relations * currently of interest. */ -static List * +List * select_mergejoin_clauses(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c index 0dbe2ac726..7271f044ec 100644 --- a/src/backend/optimizer/path/joinrels.c +++ b/src/backend/optimizer/path/joinrels.c @@ -924,6 +924,8 @@ populate_joinrel_with_paths(PlannerInfo *root, RelOptInfo *rel1, /* Apply partitionwise join technique, if possible. */ try_partitionwise_join(root, rel1, rel2, joinrel, sjinfo, restrictlist); + + populate_joinrel_uniquekeys(root, joinrel, rel1, rel2, restrictlist, sjinfo->jointype); } diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index bd9a176d7d..139278829b 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -33,7 +33,6 @@ static bool pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys); static bool matches_boolean_partition_clause(RestrictInfo *rinfo, RelOptInfo *partrel, int partkeycol); -static Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); static bool right_merge_direction(PlannerInfo *root, PathKey *pathkey); @@ -1035,7 +1034,7 @@ convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel, * We need this to ensure that we don't return pathkeys describing values * that are unavailable above the level of the subquery scan. */ -static Var * +Var * find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle) { ListCell *lc; diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c new file mode 100644 index 0000000000..77ed2b2eff --- /dev/null +++ b/src/backend/optimizer/path/uniquekeys.c @@ -0,0 +1,1135 @@ +/*------------------------------------------------------------------------- + * + * uniquekeys.c + * Utilities for matching and building unique keys + * + * Portions Copyright (c) 2020, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/optimizer/path/uniquekeys.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "nodes/makefuncs.h" +#include "nodes/nodeFuncs.h" +#include "optimizer/pathnode.h" +#include "optimizer/paths.h" +#include "optimizer/appendinfo.h" +#include "optimizer/optimizer.h" +#include "optimizer/tlist.h" +#include "rewrite/rewriteManip.h" + + +/* + * This struct is used to help populate_joinrel_uniquekeys. + * + * added_to_joinrel is true if a uniquekey (from outerrel or innerrel) + * has been added to joinrel. + * useful is true if the exprs of the uniquekey still appears in joinrel. + */ +typedef struct UniqueKeyContextData +{ + UniqueKey *uniquekey; + bool added_to_joinrel; + bool useful; +} *UniqueKeyContext; + +static List *initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel); +static bool innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + bool reverse); + +static List *get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals); +static List *get_exprs_from_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + UniqueKey *ukey); +static void add_uniquekey_for_onerow(RelOptInfo *rel); +static bool add_combined_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype); + +/* Used for unique indexes checking for partitioned table */ +static bool index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind); +static IndexOptInfo *simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from); +static bool simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2); +static void adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_index); + +/* Helper function for grouped relation and distinct relation. */ +static void add_uniquekey_from_sortgroups(PlannerInfo *root, + RelOptInfo *rel, + List *sortgroups); + +/* + * populate_baserel_uniquekeys + * Populate 'baserel' uniquekeys list by looking at the rel's unique index + * and baserestrictinfo + */ +void +populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List *indexlist) +{ + ListCell *lc; + List *matched_uniq_indexes = NIL; + + /* Attrs appears in rel->reltarget->exprs. */ + Bitmapset *used_attrs = NULL; + + List *const_exprs = NIL; + List *expr_opfamilies = NIL; + + Assert(baserel->rtekind == RTE_RELATION); + + foreach(lc, indexlist) + { + IndexOptInfo *ind = (IndexOptInfo *) lfirst(lc); + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + matched_uniq_indexes = lappend(matched_uniq_indexes, ind); + } + + if (matched_uniq_indexes == NIL) + return; + + /* Check which attrs is used in baserel->reltarget */ + pull_varattnos((Node *)baserel->reltarget->exprs, baserel->relid, &used_attrs); + + /* Check which attrno is used at a mergeable const filter */ + foreach(lc, baserel->baserestrictinfo) + { + RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); + + if (rinfo->mergeopfamilies == NIL) + continue; + + if (bms_is_empty(rinfo->left_relids)) + { + const_exprs = lappend(const_exprs, get_rightop(rinfo->clause)); + } + else if (bms_is_empty(rinfo->right_relids)) + { + const_exprs = lappend(const_exprs, get_leftop(rinfo->clause)); + } + else + continue; + + expr_opfamilies = lappend(expr_opfamilies, rinfo->mergeopfamilies); + } + + foreach(lc, matched_uniq_indexes) + { + bool multi_nullvals, useful; + List *exprs = get_exprs_from_uniqueindex(lfirst_node(IndexOptInfo, lc), + const_exprs, + expr_opfamilies, + used_attrs, + &useful, + &multi_nullvals); + if (useful) + { + if (exprs == NIL) + { + /* All the columns in Unique Index matched with a restrictinfo */ + add_uniquekey_for_onerow(baserel); + return; + } + baserel->uniquekeys = lappend(baserel->uniquekeys, + makeUniqueKey(exprs, multi_nullvals)); + } + } +} + + +/* + * populate_partitionedrel_uniquekeys + * The UniqueKey on partitionrel comes from 2 cases: + * 1). Only one partition is involved in this query, the unique key can be + * copied to parent rel from childrel. + * 2). There are some unique index which includes partition key and exists + * in all the related partitions. + * We never mind rule 2 if we hit rule 1. + */ + +void +populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels) +{ + ListCell *lc; + List *global_uniq_indexlist = NIL; + RelOptInfo *childrel; + bool is_first = true; + + Assert(IS_PARTITIONED_REL(rel)); + + if (childrels == NIL) + return; + + /* + * If there is only one partition used in this query, the UniqueKey in childrel is + * still valid in parent level, but we need convert the format from child expr to + * parent expr. + */ + if (list_length(childrels) == 1) + { + /* Check for Rule 1 */ + RelOptInfo *childrel = linitial_node(RelOptInfo, childrels); + ListCell *lc; + Assert(childrel->reloptkind == RELOPT_OTHER_MEMBER_REL); + if (relation_is_onerow(childrel)) + { + add_uniquekey_for_onerow(rel); + return; + } + + foreach(lc, childrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + AppendRelInfo *appinfo = find_appinfo_by_child(root, childrel->relid); + List *parent_exprs = NIL; + bool can_reuse = true; + ListCell *lc2; + foreach(lc2, ukey->exprs) + { + Var *var = (Var *)lfirst(lc2); + /* + * If the expr comes from a expression, it is hard to build the expression + * in parent so ignore that case for now. + */ + if(!IsA(var, Var)) + { + can_reuse = false; + break; + } + /* Convert it to parent var */ + parent_exprs = lappend(parent_exprs, find_parent_var(appinfo, var)); + } + if (can_reuse) + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(parent_exprs, + ukey->multi_nullvals)); + } + } + else + { + /* Check for rule 2 */ + childrel = linitial_node(RelOptInfo, childrels); + foreach(lc, childrel->indexlist) + { + IndexOptInfo *ind = lfirst(lc); + IndexOptInfo *modified_index; + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + + /* + * During simple_copy_indexinfo_to_parent, we need to convert var from + * child var to parent var, index on expression is too complex to handle. + * so ignore it for now. + */ + if (ind->indexprs != NIL) + continue; + + modified_index = simple_copy_indexinfo_to_parent(root, rel, ind); + /* + * If the unique index doesn't contain partkey, then it is unique + * on this partition only, so it is useless for us. + */ + if (!index_constains_partkey(rel, modified_index)) + continue; + + global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index); + } + + if (global_uniq_indexlist != NIL) + { + foreach(lc, childrels) + { + RelOptInfo *child = lfirst(lc); + if (is_first) + { + is_first = false; + continue; + } + adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist); + } + /* Now we have a list of unique index which are exactly same on all childrels, + * Set the UniqueKey just like it is non-partition table + */ + populate_baserel_uniquekeys(root, rel, global_uniq_indexlist); + } + } +} + + +/* + * populate_distinctrel_uniquekeys + */ +void +populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel) +{ + /* The unique key before the distinct is still valid. */ + distinctrel->uniquekeys = list_copy(inputrel->uniquekeys); + add_uniquekey_from_sortgroups(root, distinctrel, root->parse->distinctClause); +} + +/* + * populate_grouprel_uniquekeys + */ +void +populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel) + +{ + Query *parse = root->parse; + bool input_ukey_added = false; + ListCell *lc; + + if (relation_is_onerow(inputrel)) + { + add_uniquekey_for_onerow(grouprel); + return; + } + if (parse->groupingSets) + return; + + /* A Normal group by without grouping set. */ + if (parse->groupClause) + { + /* + * Current even the groupby clause is Unique already, but if query has aggref + * We have to create grouprel still. To keep the UnqiueKey short, we will check + * the UniqueKey of input_rel still valid, if so we reuse it. + */ + foreach(lc, inputrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (list_is_subset(ukey->exprs, grouprel->reltarget->exprs)) + { + grouprel->uniquekeys = lappend(grouprel->uniquekeys, + ukey); + input_ukey_added = true; + } + } + if (!input_ukey_added) + /* + * group by clause must be a super-set of grouprel->reltarget->exprs except the + * aggregation expr, so if such exprs is unique already, no bother to generate + * new uniquekey for group by exprs. + */ + add_uniquekey_from_sortgroups(root, + grouprel, + root->parse->groupClause); + } + else + /* It has aggregation but without a group by, so only one row returned */ + add_uniquekey_for_onerow(grouprel); +} + +/* + * simple_copy_uniquekeys + * Using a function for the one-line code makes us easy to check where we simply + * copied the uniquekey. + */ +void +simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel) +{ + newrel->uniquekeys = oldrel->uniquekeys; +} + +/* + * populate_unionrel_uniquekeys + */ +void +populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel) +{ + ListCell *lc; + List *exprs = NIL; + + Assert(unionrel->uniquekeys == NIL); + + foreach(lc, unionrel->reltarget->exprs) + { + exprs = lappend(exprs, lfirst(lc)); + } + + if (exprs == NIL) + /* SQL: select union select; is valid, we need to handle it here. */ + add_uniquekey_for_onerow(unionrel); + else + unionrel->uniquekeys = lappend(unionrel->uniquekeys, + makeUniqueKey(exprs,false)); + +} + +/* + * populate_joinrel_uniquekeys + * + * populate uniquekeys for joinrel. We will check each relation to see if its + * UniqueKey is still valid via innerrel_keeps_unique, if so, we add it to + * joinrel. The multi_nullvals field will be changed to true for some outer + * join cases and one-row UniqueKey needs to be converted to normal UniqueKey + * for the same case as well. + * For the uniquekey in either baserel which can't be unique after join, we still + * check to see if combination of UniqueKeys from both side is still useful for us. + * if yes, we add it to joinrel as well. + */ +void +populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *outerrel, RelOptInfo *innerrel, + List *restrictlist, JoinType jointype) +{ + ListCell *lc, *lc2; + List *clause_list = NIL; + List *outerrel_ukey_ctx; + List *innerrel_ukey_ctx; + bool inner_onerow, outer_onerow; + bool mergejoin_allowed; + + /* Care about the outerrel relation only for SEMI/ANTI join */ + if (jointype == JOIN_SEMI || jointype == JOIN_ANTI) + { + foreach(lc, outerrel->uniquekeys) + { + UniqueKey *uniquekey = lfirst_node(UniqueKey, lc); + if (list_is_subset(uniquekey->exprs, joinrel->reltarget->exprs)) + joinrel->uniquekeys = lappend(joinrel->uniquekeys, uniquekey); + } + return; + } + + Assert(jointype == JOIN_LEFT || jointype == JOIN_FULL || jointype == JOIN_INNER); + + /* Fast path */ + if (innerrel->uniquekeys == NIL || outerrel->uniquekeys == NIL) + return; + + inner_onerow = relation_is_onerow(innerrel); + outer_onerow = relation_is_onerow(outerrel); + + outerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(outerrel); + innerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(innerrel); + + clause_list = select_mergejoin_clauses(root, joinrel, outerrel, innerrel, + restrictlist, jointype, + &mergejoin_allowed); + + if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true /* reverse */)) + { + bool outer_impact = jointype == JOIN_FULL; + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + /* Outer relation has one row, and the unique key is not duplicated after join, + * the joinrel will still has one row unless the jointype == JOIN_FULL. + */ + if (outer_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (outer_onerow) + { + /* + * The onerow outerrel becomes multi rows and multi_nullvals + * will be changed to true. We also need to set the exprs correctly since it + * can't be NIL any more. + */ + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, outerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Change multi_nullvals to true due to the full join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, true)); + else + /* Just reuse it */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + } + ctx->added_to_joinrel = true; + } + } + + if (innerrel_keeps_unique(root, outerrel, innerrel, clause_list, false)) + { + bool outer_impact = jointype == JOIN_FULL || jointype == JOIN_LEFT;; + + foreach(lc, innerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + if (inner_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (inner_onerow) + { + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, innerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Need to change multi_nullvals to true due to the outer join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, + true)); + else + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + + } + ctx->added_to_joinrel = true; + } + } + + /* + * The combination of the UniqueKey from both sides is unique as well regardless + * of join type, but no bother to add it if its subset has been added to joinrel + * already or it is not useful for the joinrel. + */ + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx1 = (UniqueKeyContext) lfirst(lc); + if (ctx1->added_to_joinrel || !ctx1->useful) + continue; + foreach(lc2, innerrel_ukey_ctx) + { + UniqueKeyContext ctx2 = (UniqueKeyContext) lfirst(lc2); + if (ctx2->added_to_joinrel || !ctx2->useful) + continue; + if (add_combined_uniquekey(root, joinrel, outerrel, innerrel, + ctx1->uniquekey, ctx2->uniquekey, + jointype)) + /* If we set a onerow UniqueKey to joinrel, we don't need other. */ + return; + } + } +} + + +/* + * convert_subquery_uniquekeys + * + * Covert the UniqueKey in subquery to outer relation. + */ +void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel) +{ + ListCell *lc; + + if (sub_final_rel->uniquekeys == NIL) + return; + + if (relation_is_onerow(sub_final_rel)) + { + add_uniquekey_for_onerow(currel); + return; + } + + Assert(currel->subroot != NULL); + + foreach(lc, sub_final_rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + ListCell *lc; + List *exprs = NIL; + bool ukey_useful = true; + + /* One row case is handled above */ + Assert(ukey->exprs != NIL); + foreach(lc, ukey->exprs) + { + Var *var; + TargetEntry *tle = tlist_member(lfirst(lc), + currel->subroot->processed_tlist); + if (tle == NULL) + { + ukey_useful = false; + break; + } + var = find_var_for_subquery_tle(currel, tle); + if (var == NULL) + { + ukey_useful = false; + break; + } + exprs = lappend(exprs, var); + } + + if (ukey_useful) + currel->uniquekeys = lappend(currel->uniquekeys, + makeUniqueKey(exprs, + ukey->multi_nullvals)); + + } +} + +/* + * innerrel_keeps_unique + * + * Check if Unique key of the innerrel is valid after join. innerrel's UniqueKey + * will be still valid if innerrel's any-column mergeop outrerel's uniquekey + * exists in clause_list. + * + * Note: the clause_list must be a list of mergeable restrictinfo already. + */ +static bool +innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *clause_list, + bool reverse) +{ + ListCell *lc, *lc2, *lc3; + + if (outerrel->uniquekeys == NIL || innerrel->uniquekeys == NIL) + return false; + + /* Check if there is outerrel's uniquekey in mergeable clause. */ + foreach(lc, outerrel->uniquekeys) + { + List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs; + bool clauselist_matchs_all_exprs = true; + foreach(lc2, outer_uq_exprs) + { + Node *outer_uq_expr = lfirst(lc2); + bool find_uq_expr_in_clauselist = false; + foreach(lc3, clause_list) + { + RestrictInfo *rinfo = lfirst_node(RestrictInfo, lc3); + Node *outer_expr; + if (reverse) + outer_expr = rinfo->outer_is_left ? get_rightop(rinfo->clause) : get_leftop(rinfo->clause); + else + outer_expr = rinfo->outer_is_left ? get_leftop(rinfo->clause) : get_rightop(rinfo->clause); + if (equal(outer_expr, outer_uq_expr)) + { + find_uq_expr_in_clauselist = true; + break; + } + } + if (!find_uq_expr_in_clauselist) + { + /* No need to check the next exprs in the current uniquekey */ + clauselist_matchs_all_exprs = false; + break; + } + } + + if (clauselist_matchs_all_exprs) + return true; + } + return false; +} + + +/* + * relation_is_onerow + * Check if it is a one-row relation by checking UniqueKey. + */ +bool +relation_is_onerow(RelOptInfo *rel) +{ + UniqueKey *ukey; + if (rel->uniquekeys == NIL) + return false; + ukey = linitial_node(UniqueKey, rel->uniquekeys); + return ukey->exprs == NIL && list_length(rel->uniquekeys) == 1; +} + +/* + * relation_has_uniquekeys_for + * Returns true if we have proofs that 'rel' cannot return multiple rows with + * the same values in each of 'exprs'. Otherwise returns false. + */ +bool +relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel, + List *exprs, bool allow_multinulls) +{ + ListCell *lc; + + /* + * For UniqueKey->onerow case, the uniquekey->exprs is empty as well + * so we can't rely on list_is_subset to handle this special cases + */ + if (exprs == NIL) + return false; + + foreach(lc, rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (ukey->multi_nullvals && !allow_multinulls) + continue; + if (list_is_subset(ukey->exprs, exprs)) + return true; + } + return false; +} + + +/* + * get_exprs_from_uniqueindex + * + * Return a list of exprs which is unique. set useful to false if this + * unique index is not useful for us. + */ +static List * +get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals) +{ + List *exprs = NIL; + ListCell *indexpr_item; + int c = 0; + + *useful = true; + *multi_nullvals = false; + + indexpr_item = list_head(unique_index->indexprs); + for(c = 0; c < unique_index->ncolumns; c++) + { + int attr = unique_index->indexkeys[c]; + Expr *expr; + bool matched_const = false; + ListCell *lc1, *lc2; + + if(attr > 0) + { + expr = list_nth_node(TargetEntry, unique_index->indextlist, c)->expr; + } + else if (attr == 0) + { + /* Expression index */ + expr = lfirst(indexpr_item); + indexpr_item = lnext(unique_index->indexprs, indexpr_item); + } + else /* attr < 0 */ + { + /* Index on system column is not supported */ + Assert(false); + } + + /* + * Check index_col = Const case with regarding to opfamily checking + * If we can remove the index_col from the final UniqueKey->exprs. + */ + forboth(lc1, const_exprs, lc2, const_expr_opfamilies) + { + if (list_member_oid((List *)lfirst(lc2), unique_index->opfamily[c]) + && match_index_to_operand((Node *) lfirst(lc1), c, unique_index)) + { + matched_const = true; + break; + } + } + + if (matched_const) + continue; + + /* Check if the indexed expr is used in rel */ + if (attr > 0) + { + /* + * Normal Indexed column, if the col is not used, then the index is useless + * for uniquekey. + */ + attr -= FirstLowInvalidHeapAttributeNumber; + + if (!bms_is_member(attr, used_varattrs)) + { + *useful = false; + break; + } + } + else if (!list_member(unique_index->rel->reltarget->exprs, expr)) + { + /* Expression index but the expression is not used in rel */ + *useful = false; + break; + } + + /* check not null property. */ + if (attr == 0) + { + /* We never know if a expression yields null or not */ + *multi_nullvals = true; + } + else if (!bms_is_member(attr, unique_index->rel->notnullattrs) + && !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber, + unique_index->rel->notnullattrs)) + { + *multi_nullvals = true; + } + + exprs = lappend(exprs, expr); + } + return exprs; +} + + +/* + * add_uniquekey_for_onerow + * If we are sure that the relation only returns one row, then all the columns + * are unique. However we don't need to create UniqueKey for every column, we + * just set exprs = NIL and overwrites all the other UniqueKey on this RelOptInfo + * since this one has strongest semantics. + */ +void +add_uniquekey_for_onerow(RelOptInfo *rel) +{ + /* + * We overwrite the previous UniqueKey on purpose since this one has the + * strongest semantic. + */ + rel->uniquekeys = list_make1(makeUniqueKey(NIL, false)); +} + + +/* + * initililze_uniquecontext_for_joinrel + * Return a List of UniqueKeyContext for an inputrel + */ +static List * +initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel) +{ + List *res = NIL; + ListCell *lc; + foreach(lc, inputrel->uniquekeys) + { + UniqueKeyContext context; + context = palloc(sizeof(struct UniqueKeyContextData)); + context->uniquekey = lfirst_node(UniqueKey, lc); + context->added_to_joinrel = false; + context->useful = true; + res = lappend(res, context); + } + return res; +} + + +/* + * get_exprs_from_uniquekey + * Unify the way of get List of exprs from a one-row UniqueKey or + * normal UniqueKey. for the onerow case, every expr in rel1 is a valid + * UniqueKey. Return a List of exprs. + * + * rel1: The relation which you want to get the exprs. + * ukey: The UniqueKey you want to get the exprs. + */ +static List * +get_exprs_from_uniquekey(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *rel1, UniqueKey *ukey) +{ + ListCell *lc; + bool onerow = rel1 != NULL && relation_is_onerow(rel1); + + List *res = NIL; + Assert(onerow || ukey); + if (onerow) + { + /* Only cares about the exprs still exist in joinrel */ + foreach(lc, joinrel->reltarget->exprs) + { + Bitmapset *relids = pull_varnos(root, lfirst(lc)); + if (bms_is_subset(relids, rel1->relids)) + { + res = lappend(res, list_make1(lfirst(lc))); + } + } + } + else + { + res = list_make1(ukey->exprs); + } + return res; +} + +/* + * Partitioned table Unique Keys. + * The partition table unique key is maintained as: + * 1. The index must be unique as usual. + * 2. The index must contains partition key. + * 3. The index must exist on all the child rel. see simple_indexinfo_equal for + * how we compare it. + */ + +/* + * index_constains_partkey + * return true if the index contains the partiton key. + */ +static bool +index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind) +{ + ListCell *lc; + int i; + Assert(IS_PARTITIONED_REL(partrel)); + Assert(partrel->part_scheme->partnatts > 0); + + for(i = 0; i < partrel->part_scheme->partnatts; i++) + { + Node *part_expr = linitial(partrel->partexprs[i]); + bool found_in_index = false; + foreach(lc, ind->indextlist) + { + Expr *index_expr = lfirst_node(TargetEntry, lc)->expr; + if (equal(index_expr, part_expr)) + { + found_in_index = true; + break; + } + } + if (!found_in_index) + return false; + } + return true; +} + +/* + * simple_indexinfo_equal + * + * Used to check if the 2 index is same as each other. The index here + * is COPIED from childrel and did some tiny changes(see + * simple_copy_indexinfo_to_parent) + */ +static bool +simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2) +{ + Size oid_cmp_len = sizeof(Oid) * ind1->ncolumns; + + return ind1->ncolumns == ind2->ncolumns && + ind1->unique == ind2->unique && + memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0 && + memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0 && + memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0 && + memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0 && + equal(get_tlist_exprs(ind1->indextlist, true), + get_tlist_exprs(ind2->indextlist, true)); +} + + +/* + * The below macros are used for simple_copy_indexinfo_to_parent which is so + * customized that I don't want to put it to copyfuncs.c. So copy it here. + */ +#define COPY_POINTER_FIELD(fldname, sz) \ + do { \ + Size _size = (sz); \ + newnode->fldname = palloc(_size); \ + memcpy(newnode->fldname, from->fldname, _size); \ + } while (0) + +#define COPY_NODE_FIELD(fldname) \ + (newnode->fldname = copyObjectImpl(from->fldname)) + +#define COPY_SCALAR_FIELD(fldname) \ + (newnode->fldname = from->fldname) + + +/* + * simple_copy_indexinfo_to_parent (from partition) + * Copy the IndexInfo from child relation to parent relation with some modification, + * which is used to test: + * 1. If the same index exists in all the childrels. + * 2. If the parentrel->reltarget/basicrestrict info matches this index. + */ +static IndexOptInfo * +simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from) +{ + IndexOptInfo *newnode = makeNode(IndexOptInfo); + AppendRelInfo *appinfo = find_appinfo_by_child(root, from->rel->relid); + ListCell *lc; + int idx = 0; + + COPY_SCALAR_FIELD(ncolumns); + COPY_SCALAR_FIELD(nkeycolumns); + COPY_SCALAR_FIELD(unique); + COPY_SCALAR_FIELD(immediate); + /* We just need to know if it is NIL or not */ + COPY_SCALAR_FIELD(indpred); + COPY_SCALAR_FIELD(predOK); + COPY_POINTER_FIELD(indexkeys, from->ncolumns * sizeof(int)); + COPY_POINTER_FIELD(indexcollations, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opfamily, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opcintype, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(sortopfamily, from->ncolumns * sizeof(Oid)); + COPY_NODE_FIELD(indextlist); + + /* Convert index exprs on child expr to expr on parent */ + foreach(lc, newnode->indextlist) + { + TargetEntry *tle = lfirst_node(TargetEntry, lc); + /* Index on expression is ignored */ + Assert(IsA(tle->expr, Var)); + tle->expr = (Expr *) find_parent_var(appinfo, (Var *) tle->expr); + newnode->indexkeys[idx] = castNode(Var, tle->expr)->varattno; + idx++; + } + newnode->rel = parentrel; + return newnode; +} + +/* + * adjust_partition_unique_indexlist + * + * global_unique_indexes: At the beginning, it contains the copy & modified + * unique index from the first partition. And then check if each index in it still + * exists in the following partitions. If no, remove it. at last, it has an + * index list which exists in all the partitions. + */ +static void +adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_indexes) +{ + ListCell *lc, *lc2; + foreach(lc, *global_unique_indexes) + { + IndexOptInfo *g_ind = lfirst_node(IndexOptInfo, lc); + bool found_in_child = false; + + foreach(lc2, childrel->indexlist) + { + IndexOptInfo *p_ind = lfirst_node(IndexOptInfo, lc2); + IndexOptInfo *p_ind_copy; + if (!p_ind->unique || !p_ind->immediate || + (p_ind->indpred != NIL && !p_ind->predOK)) + continue; + p_ind_copy = simple_copy_indexinfo_to_parent(root, parentrel, p_ind); + if (simple_indexinfo_equal(p_ind_copy, g_ind)) + { + found_in_child = true; + break; + } + } + if (!found_in_child) + /* The index doesn't exist in childrel, remove it from global_unique_indexes */ + *global_unique_indexes = foreach_delete_current(*global_unique_indexes, lc); + } +} + +/* Helper function for groupres/distinctrel */ +static void +add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgroups) +{ + Query *parse = root->parse; + List *exprs; + + /* + * XXX: If there are some vars which is not in current levelsup, the semantic is + * imprecise, should we avoid it or not? levelsup = 1 is just a demo, maybe we need to + * check every level other than 0, if so, looks we have to write another + * pull_var_walker. + */ + List *upper_vars = pull_vars_of_level((Node*)sortgroups, 1); + + if (upper_vars != NIL) + return; + + exprs = get_sortgrouplist_exprs(sortgroups, parse->targetList); + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(exprs, + false /* sortgroupclause can't be multi_nullvals */)); +} + + +/* + * add_combined_uniquekey + * The combination of both UniqueKeys is a valid UniqueKey for joinrel no matter + * the jointype. + */ +bool +add_combined_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype) +{ + + ListCell *lc1, *lc2; + + /* Either side has multi_nullvals or we have outer join, + * the combined UniqueKey has multi_nullvals */ + bool multi_nullvals = outer_ukey->multi_nullvals || + inner_ukey->multi_nullvals || IS_OUTER_JOIN(jointype); + + /* The only case we can get onerow joinrel after join */ + if (relation_is_onerow(outer_rel) + && relation_is_onerow(inner_rel) + && jointype == JOIN_INNER) + { + add_uniquekey_for_onerow(joinrel); + return true; + } + + foreach(lc1, get_exprs_from_uniquekey(root, joinrel, outer_rel, outer_ukey)) + { + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, inner_rel, inner_ukey)) + { + List *exprs = list_concat_copy(lfirst_node(List, lc1), lfirst_node(List, lc2)); + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(exprs, + multi_nullvals)); + } + } + return false; +} diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index 424d25cbd5..8d8e493f5c 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -2387,6 +2387,8 @@ grouping_planner(PlannerInfo *root, bool inheritance_update, add_path(final_rel, path); } + simple_copy_uniquekeys(current_rel, final_rel); + /* * Generate partial paths for final_rel, too, if outer query levels might * be able to make use of them. @@ -3899,6 +3901,8 @@ create_grouping_paths(PlannerInfo *root, } set_cheapest(grouped_rel); + + populate_grouprel_uniquekeys(root, grouped_rel, input_rel); return grouped_rel; } @@ -4621,7 +4625,7 @@ create_window_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(window_rel); - + simple_copy_uniquekeys(input_rel, window_rel); return window_rel; } @@ -4935,7 +4939,7 @@ create_distinct_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(distinct_rel); - + populate_distinctrel_uniquekeys(root, input_rel, distinct_rel); return distinct_rel; } @@ -5196,6 +5200,8 @@ create_ordered_paths(PlannerInfo *root, */ Assert(ordered_rel->pathlist != NIL); + simple_copy_uniquekeys(input_rel, ordered_rel); + return ordered_rel; } @@ -6073,6 +6079,9 @@ adjust_paths_for_srfs(PlannerInfo *root, RelOptInfo *rel, if (list_length(targets) == 1) return; + /* UniqueKey is not valid after handling the SRF. */ + rel->uniquekeys = NIL; + /* * Stack SRF-evaluation nodes atop each path for the rel. * diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c index becdcbb872..b7626545bf 100644 --- a/src/backend/optimizer/prep/prepunion.c +++ b/src/backend/optimizer/prep/prepunion.c @@ -689,6 +689,8 @@ generate_union_paths(SetOperationStmt *op, PlannerInfo *root, /* Undo effects of possibly forcing tuple_fraction to 0 */ root->tuple_fraction = save_fraction; + /* Add the UniqueKeys */ + populate_unionrel_uniquekeys(root, result_rel); return result_rel; } diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c index 86922a273c..6817c9c787 100644 --- a/src/backend/optimizer/util/appendinfo.c +++ b/src/backend/optimizer/util/appendinfo.c @@ -746,3 +746,47 @@ find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos) } return appinfos; } + +/* + * find_appinfo_by_child + * + */ +AppendRelInfo * +find_appinfo_by_child(PlannerInfo *root, Index child_index) +{ + ListCell *lc; + foreach(lc, root->append_rel_list) + { + AppendRelInfo *appinfo = lfirst_node(AppendRelInfo, lc); + if (appinfo->child_relid == child_index) + return appinfo; + } + elog(ERROR, "parent relation cant be found"); + return NULL; +} + +/* + * find_parent_var + * + */ +Var * +find_parent_var(AppendRelInfo *appinfo, Var *child_var) +{ + ListCell *lc; + Var *res = NULL; + Index attno = 1; + foreach(lc, appinfo->translated_vars) + { + Node *child_node = lfirst(lc); + if (equal(child_node, child_var)) + { + res = copyObject(child_var); + res->varattno = attno; + res->varno = appinfo->parent_relid; + } + attno++; + } + if (res == NULL) + elog(ERROR, "parent var can't be found."); + return res; +} diff --git a/src/backend/optimizer/util/inherit.c b/src/backend/optimizer/util/inherit.c index be1c9ddd96..3eec1f4d74 100644 --- a/src/backend/optimizer/util/inherit.c +++ b/src/backend/optimizer/util/inherit.c @@ -736,6 +736,7 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, { Node *onecq = (Node *) lfirst(lc2); bool pseudoconstant; + RestrictInfo *child_rinfo; /* check for pseudoconstant (no Vars or volatile functions) */ pseudoconstant = @@ -747,14 +748,15 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, root->hasPseudoConstantQuals = true; } /* reconstitute RestrictInfo with appropriate properties */ - childquals = lappend(childquals, - make_restrictinfo(root, - (Expr *) onecq, - rinfo->is_pushed_down, - rinfo->outerjoin_delayed, - pseudoconstant, - rinfo->security_level, - NULL, NULL, NULL)); + child_rinfo = make_restrictinfo(root, + (Expr *) onecq, + rinfo->is_pushed_down, + rinfo->outerjoin_delayed, + pseudoconstant, + rinfo->security_level, + NULL, NULL, NULL); + child_rinfo->mergeopfamilies = rinfo->mergeopfamilies; + childquals = lappend(childquals, child_rinfo); /* track minimum security level among child quals */ cq_min_security = Min(cq_min_security, rinfo->security_level); } diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 48a7ebfe45..4fe1824eb0 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -16,6 +16,7 @@ #include "nodes/execnodes.h" #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" extern A_Expr *makeA_Expr(A_Expr_Kind kind, List *name, @@ -106,4 +107,6 @@ extern GroupingSet *makeGroupingSet(GroupingSetKind kind, List *content, int loc extern VacuumRelation *makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols); +extern UniqueKey* makeUniqueKey(List *exprs, bool multi_nullvals); + #endif /* MAKEFUNC_H */ diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index e22df890ef..3320273ac1 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -265,6 +265,7 @@ typedef enum NodeTag T_EquivalenceMember, T_PathKey, T_PathTarget, + T_UniqueKey, T_RestrictInfo, T_IndexClause, T_PlaceHolderVar, diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index 0d61f04d27..80561b1003 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -739,6 +739,7 @@ typedef struct RelOptInfo QualCost baserestrictcost; /* cost of evaluating the above */ Index baserestrict_min_security; /* min security_level found in * baserestrictinfo */ + List *uniquekeys; /* List of UniqueKey */ List *joininfo; /* RestrictInfo structures for join clauses * involving this rel */ bool has_eclass_joins; /* T means joininfo is incomplete */ @@ -1059,6 +1060,28 @@ typedef struct PathKey } PathKey; +/* + * UniqueKey + * + * Represents the unique properties held by a RelOptInfo. + * + * exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL + * is a special case of UniqueKey, which means there is only 1 row in that + * relation. + * multi_nullvals: true means multi null values may exist in these exprs, so the + * uniqueness is not guaranteed in this case. This field is necessary for + * remove_useless_join & reduce_unique_semijoins where we don't mind these + * duplicated NULL values. It is set to true for 2 cases. One is a unique key + * from a unique index but the related column is nullable. The other one is for + * outer join. see populate_joinrel_uniquekeys for detail. + */ +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + /* * PathTarget * @@ -2496,7 +2519,7 @@ typedef enum * * flags indicating what kinds of grouping are possible. * partial_costs_set is true if the agg_partial_costs and agg_final_costs - * have been initialized. + * have been initialized. * agg_partial_costs gives partial aggregation costs. * agg_final_costs gives finalization costs. * target_parallel_safe is true if target is parallel safe. @@ -2526,8 +2549,8 @@ typedef struct * limit_tuples is an estimated bound on the number of output tuples, * or -1 if no LIMIT or couldn't estimate. * count_est and offset_est are the estimated values of the LIMIT and OFFSET - * expressions computed by preprocess_limit() (see comments for - * preprocess_limit() for more information). + * expressions computed by preprocess_limit() (see comments for + * preprocess_limit() for more information). */ typedef struct { diff --git a/src/include/nodes/pg_list.h b/src/include/nodes/pg_list.h index 404e03f132..85c65e910c 100644 --- a/src/include/nodes/pg_list.h +++ b/src/include/nodes/pg_list.h @@ -566,6 +566,16 @@ extern pg_nodiscard List *list_delete_first(List *list); extern pg_nodiscard List *list_delete_last(List *list); extern pg_nodiscard List *list_delete_nth_cell(List *list, int n); extern pg_nodiscard List *list_delete_cell(List *list, ListCell *cell); +extern bool list_is_subset(const List *members, const List *target); + +extern List *list_delete(List *list, void *datum); +extern List *list_delete_ptr(List *list, void *datum); +extern List *list_delete_int(List *list, int datum); +extern List *list_delete_oid(List *list, Oid datum); +extern List *list_delete_first(List *list); +extern List *list_delete_last(List *list); +extern List *list_delete_nth_cell(List *list, int n); +extern List *list_delete_cell(List *list, ListCell *cell); extern List *list_union(const List *list1, const List *list2); extern List *list_union_ptr(const List *list1, const List *list2); diff --git a/src/include/optimizer/appendinfo.h b/src/include/optimizer/appendinfo.h index 4cbf8c26cc..43b8d05f69 100644 --- a/src/include/optimizer/appendinfo.h +++ b/src/include/optimizer/appendinfo.h @@ -32,4 +32,7 @@ extern Relids adjust_child_relids_multilevel(PlannerInfo *root, Relids relids, extern AppendRelInfo **find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos); +extern AppendRelInfo *find_appinfo_by_child(PlannerInfo *root, Index child_index); +extern Var *find_parent_var(AppendRelInfo *appinfo, Var *child_var); + #endif /* APPENDINFO_H */ diff --git a/src/include/optimizer/optimizer.h b/src/include/optimizer/optimizer.h index d587952b7d..843aafa51e 100644 --- a/src/include/optimizer/optimizer.h +++ b/src/include/optimizer/optimizer.h @@ -23,6 +23,7 @@ #define OPTIMIZER_H #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" /* Test if an expression node represents a SRF call. Beware multiple eval! */ #define IS_SRF_CALL(node) \ @@ -168,6 +169,7 @@ extern TargetEntry *get_sortgroupref_tle(Index sortref, List *targetList); extern TargetEntry *get_sortgroupclause_tle(SortGroupClause *sgClause, List *targetList); +extern Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); extern Node *get_sortgroupclause_expr(SortGroupClause *sgClause, List *targetList); extern List *get_sortgrouplist_exprs(List *sgClauses, diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h index 035d3e1206..1adf99c9ee 100644 --- a/src/include/optimizer/paths.h +++ b/src/include/optimizer/paths.h @@ -247,5 +247,48 @@ extern PathKey *make_canonical_pathkey(PlannerInfo *root, int strategy, bool nulls_first); extern void add_paths_to_append_rel(PlannerInfo *root, RelOptInfo *rel, List *live_childrels); +extern List *select_mergejoin_clauses(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + JoinType jointype, + bool *mergejoin_allowed); + +/* + * uniquekeys.c + * Utilities for matching and building unique keys + */ +extern void populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List* unique_index_list); +extern void populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels); +extern void populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel); +extern void populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel); +extern void populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel); +extern void simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel); +extern void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel); +extern void populate_joinrel_uniquekeys(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + RelOptInfo *rel2, + List *restrictlist, + JoinType jointype); + +extern bool relation_has_uniquekeys_for(PlannerInfo *root, + RelOptInfo *rel, + List *exprs, + bool allow_multinulls); +extern bool relation_is_onerow(RelOptInfo *rel); #endif /* PATHS_H */ -- 2.26.2 --yh7cw345mbttadjz Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v38-0003-Extend-UniqueKeys.patch" ^ permalink raw reply [nested|flat] 91+ messages in thread
* [PATCH v38 2/6] Introduce UniqueKey attributes on RelOptInfo struct. @ 2020-05-11 07:50 一挃 <[email protected]> 0 siblings, 0 replies; 91+ messages in thread From: 一挃 @ 2020-05-11 07:50 UTC (permalink / raw) UniqueKey is a set of exprs on RelOptInfo which represents the exprs will be unique on the given RelOptInfo. You can see README.uniquekey for more information. --- src/backend/nodes/copyfuncs.c | 13 + src/backend/nodes/list.c | 31 + src/backend/nodes/makefuncs.c | 13 + src/backend/nodes/outfuncs.c | 11 + src/backend/nodes/readfuncs.c | 10 + src/backend/optimizer/path/Makefile | 3 +- src/backend/optimizer/path/README.uniquekey | 131 +++ src/backend/optimizer/path/allpaths.c | 10 + src/backend/optimizer/path/joinpath.c | 9 +- src/backend/optimizer/path/joinrels.c | 2 + src/backend/optimizer/path/pathkeys.c | 3 +- src/backend/optimizer/path/uniquekeys.c | 1135 +++++++++++++++++++ src/backend/optimizer/plan/planner.c | 13 +- src/backend/optimizer/prep/prepunion.c | 2 + src/backend/optimizer/util/appendinfo.c | 44 + src/backend/optimizer/util/inherit.c | 18 +- src/include/nodes/makefuncs.h | 3 + src/include/nodes/nodes.h | 1 + src/include/nodes/pathnodes.h | 29 +- src/include/nodes/pg_list.h | 10 + src/include/optimizer/appendinfo.h | 3 + src/include/optimizer/optimizer.h | 2 + src/include/optimizer/paths.h | 43 + 23 files changed, 1515 insertions(+), 24 deletions(-) create mode 100644 src/backend/optimizer/path/README.uniquekey create mode 100644 src/backend/optimizer/path/uniquekeys.c diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..75c1c5e824 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2296,6 +2296,16 @@ _copyPathKey(const PathKey *from) return newnode; } +static UniqueKey * +_copyUniqueKey(const UniqueKey *from) +{ + UniqueKey *newnode = makeNode(UniqueKey); + + COPY_NODE_FIELD(exprs); + COPY_SCALAR_FIELD(multi_nullvals); + + return newnode; +} /* * _copyRestrictInfo */ @@ -5220,6 +5230,9 @@ copyObjectImpl(const void *from) case T_PathKey: retval = _copyPathKey(from); break; + case T_UniqueKey: + retval = _copyUniqueKey(from); + break; case T_RestrictInfo: retval = _copyRestrictInfo(from); break; diff --git a/src/backend/nodes/list.c b/src/backend/nodes/list.c index dbf6b30233..ca099495a1 100644 --- a/src/backend/nodes/list.c +++ b/src/backend/nodes/list.c @@ -702,6 +702,37 @@ list_member_oid(const List *list, Oid datum) return false; } +/* + * return true iff every entry in "members" list is also present + * in the "target" list. + */ +bool +list_is_subset(const List *members, const List *target) +{ + const ListCell *lc1, *lc2; + + Assert(IsPointerList(members)); + Assert(IsPointerList(target)); + check_list_invariants(members); + check_list_invariants(target); + + foreach(lc1, members) + { + bool found = false; + foreach(lc2, target) + { + if (equal(lfirst(lc1), lfirst(lc2))) + { + found = true; + break; + } + } + if (!found) + return false; + } + return true; +} + /* * Delete the n'th cell (counting from 0) in list. * diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 01c110cd2f..40415d0f5b 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -815,3 +815,16 @@ makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols) v->va_cols = va_cols; return v; } + + +/* + * makeUniqueKey + */ +UniqueKey* +makeUniqueKey(List *exprs, bool multi_nullvals) +{ + UniqueKey * ukey = makeNode(UniqueKey); + ukey->exprs = exprs; + ukey->multi_nullvals = multi_nullvals; + return ukey; +} diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..44154cde6a 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2456,6 +2456,14 @@ _outPathKey(StringInfo str, const PathKey *node) WRITE_BOOL_FIELD(pk_nulls_first); } +static void +_outUniqueKey(StringInfo str, const UniqueKey *node) +{ + WRITE_NODE_TYPE("UNIQUEKEY"); + WRITE_NODE_FIELD(exprs); + WRITE_BOOL_FIELD(multi_nullvals); +} + static void _outPathTarget(StringInfo str, const PathTarget *node) { @@ -4198,6 +4206,9 @@ outNode(StringInfo str, const void *obj) case T_PathKey: _outPathKey(str, obj); break; + case T_UniqueKey: + _outUniqueKey(str, obj); + break; case T_PathTarget: _outPathTarget(str, obj); break; diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index c5e136e9c3..b3e212bf1c 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -492,6 +492,14 @@ _readSetOperationStmt(void) READ_DONE(); } +static UniqueKey * +_readUniqueKey(void) +{ + READ_LOCALS(UniqueKey); + READ_NODE_FIELD(exprs); + READ_BOOL_FIELD(multi_nullvals); + READ_DONE(); +} /* * Stuff from primnodes.h. @@ -2717,6 +2725,8 @@ parseNodeString(void) return_value = _readCommonTableExpr(); else if (MATCH("SETOPERATIONSTMT", 16)) return_value = _readSetOperationStmt(); + else if (MATCH("UNIQUEKEY", 9)) + return_value = _readUniqueKey(); else if (MATCH("ALIAS", 5)) return_value = _readAlias(); else if (MATCH("RANGEVAR", 8)) diff --git a/src/backend/optimizer/path/Makefile b/src/backend/optimizer/path/Makefile index 1e199ff66f..7b9820c25f 100644 --- a/src/backend/optimizer/path/Makefile +++ b/src/backend/optimizer/path/Makefile @@ -21,6 +21,7 @@ OBJS = \ joinpath.o \ joinrels.o \ pathkeys.o \ - tidpath.o + tidpath.o \ + uniquekeys.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/optimizer/path/README.uniquekey b/src/backend/optimizer/path/README.uniquekey new file mode 100644 index 0000000000..5eac761995 --- /dev/null +++ b/src/backend/optimizer/path/README.uniquekey @@ -0,0 +1,131 @@ +1. What is UniqueKey? +We can think UniqueKey is a set of exprs for a RelOptInfo, which we are insure +that doesn't yields same result among all the rows. The simplest UniqueKey +format is primary key. + +However we define the UnqiueKey as below. + +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + +exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL +is a special case of UniqueKey, which means there is only one row in that +relation.it has a stronger semantic than others. like SELECT uk FROM t; uk is +normal unique key and may have different values. SELECT colx FROM t WHERE uk = +const. colx is unique AND we have only 1 value. This field can used for +innerrel_is_unique. this logic is handled specially in add_uniquekey_for_onerow +function. + +multi_nullvals: true means multi null values may exist in these exprs, so the +uniqueness is not guaranteed in this case. This field is necessary for +remove_useless_join & reduce_unique_semijoins where we don't mind these +duplicated NULL values. It is set to true for 2 cases. One is a unique key +from a unique index but the related column is nullable. The other one is for +outer join. see populate_joinrel_uniquekeys for detail. + + +The UniqueKey can be used at the following cases at least: +1. remove_useless_joins. +2. reduce_semianti_joins +3. remove distinct node if distinct clause is unique. +4. remove aggnode if group by clause is unique. +5. Index Skip Scan (WIP) +6. Aggregation Push Down without 2 phase aggregation if the join can't + duplicated the aggregated rows. (work in progress feature) + +2. How is it maintained? + +We have a set of populate_xxx_unqiuekeys functions to maintain the uniquekey on +various cases. xxx includes baserel, joinrel, partitionedrel, distinctrel, +groupedrel, unionrel. and we also need to convert the uniquekey from subquery +to outer relation, which is what convert_subquery_uniquekeys does. + +1. The first part is about baserel. We handled 3 cases. suppose we have Unique +Index on (a, b). + +1. SELECT a, b FROM t. UniqueKey (a, b) +2. SELECT a FROM t WHERE b = 1; UniqueKey (a) +3. SELECT .. FROM t WHERE a = 1 AND b = 1; UniqueKey (NIL). onerow case, every + column is Unique. + +2. The next part is joinrel, this part is most error-prone, we simplified the rules +like below: +1. If the relation's UniqueKey can't be duplicated after join, then is will be + still valid for the join rel. The function we used here is + innerrel_keeps_unique. The basic idea is innerrel.any_col = outer.uk. + +2. If the UnqiueKey can't keep valid via the rule 1, the combination of the + UniqueKey from both sides are valid for sure. We can prove this as: if the + unique exprs from rel1 is duplicated by rel2, the duplicated rows must + contains different unique exprs from rel2. + +More considerations about onerow: +1. If relation with one row and it can't be duplicated, it is still possible + contains mulit_nullvas after outer join. +2. If the either UniqueKey can be duplicated after join, the can get one row + only when both side is one row AND there is no outer join. +3. Whenever the onerow UniqueKey is not a valid any more, we need to convert one + row UniqueKey to normal unique key since we don't store exprs for one-row + relation. get_exprs_from_uniquekeys will be used here. + + +More considerations about multi_nullvals after join: +1. If the original UnqiueKey has multi_nullvals, the final UniqueKey will have + mulit_nullvals in any case. +2. If a unique key doesn't allow mulit_nullvals, after some outer join, it + allows some outer join. + + +3. When we comes to subquery, we need to convert_subquery_unqiuekeys just like +convert_subquery_pathkeys. Only the UniqueKey insides subquery is referenced as +a Var in outer relation will be reused. The relationship between the outerrel.Var +and subquery.exprs is built with outerel->subroot->processed_tlist. + + +4. As for the SRF functions, it will break the uniqueness of uniquekey, However it +is handled in adjust_paths_for_srfs, which happens after the query_planner. so +we will maintain the UniqueKey until there and reset it to NIL at that +places. This can't help on distinct/group by elimination cases but probably help +in some other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is +semantic correctly. + + +5. As for inherit table, we first main the UnqiueKey on childrel as well. But for +partitioned table we need to maintain 2 different kinds of +UnqiueKey. 1). UniqueKey on the parent relation 2). UniqueKey on child +relation for partition wise query. + +Example: +CREATE TABLE p (a int not null, b int not null) partition by list (a); +CREATE TABLE p0 partition of p for values in (1); +CREATE TABLE p1 partition of p for values in (2); + +create unique index p0_b on p0(b); +create unique index p1_b on p1(b); + +Now b is only unique on partition level, so the distinct can't be removed on +the following cases. SELECT DISTINCT b FROM p; + +Another example is SELECT DISTINCT a, b FROM p WHERE a = 1; Since only one +partition is chosen, the UniqueKey on child relation is same as the UniqueKey on +parent relation. + +Another usage of UniqueKey on partition level is it be helpful for +partition-wise join. + +As for the UniqueKey on parent table level, it comes with 2 different ways, +1). the UniqueKey is also derived in UniqueKey index, but the index must be same +in all the related children relations and the unique index must contains +Partition Key in it. Example: + +CREATE UNIQUE INDEX p_ab ON p(a, b); -- where a is the partition key. + +-- Query +SELECT a, b FROM p; the (a, b) is a UniqueKey of p. + +2). If the parent relation has only one childrel, the UniqueKey on childrel is + the UniqueKey on parent as well. diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index 37b4223adb..d52ad59f89 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -579,6 +579,12 @@ set_plain_rel_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) */ check_index_predicates(root, rel); + /* + * Now that we've marked which partial indexes are suitable, we can now + * build the relation's unique keys. + */ + populate_baserel_uniquekeys(root, rel, rel->indexlist); + /* Mark rel with estimated output rows, width, etc */ set_baserel_size_estimates(root, rel); } @@ -1297,6 +1303,8 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, /* Add paths to the append relation. */ add_paths_to_append_rel(root, rel, live_childrels); + if (IS_PARTITIONED_REL(rel)) + populate_partitionedrel_uniquekeys(root, rel, live_childrels); } @@ -2304,6 +2312,8 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, pathkeys, required_outer)); } + convert_subquery_uniquekeys(root, rel, sub_final_rel); + /* If outer rel allows parallelism, do same for partial paths. */ if (rel->consider_parallel && bms_is_empty(required_outer)) { diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c index 57ce97fd53..697ac047fb 100644 --- a/src/backend/optimizer/path/joinpath.c +++ b/src/backend/optimizer/path/joinpath.c @@ -71,13 +71,6 @@ static void consider_parallel_mergejoin(PlannerInfo *root, static void hash_inner_and_outer(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, RelOptInfo *innerrel, JoinType jointype, JoinPathExtraData *extra); -static List *select_mergejoin_clauses(PlannerInfo *root, - RelOptInfo *joinrel, - RelOptInfo *outerrel, - RelOptInfo *innerrel, - List *restrictlist, - JoinType jointype, - bool *mergejoin_allowed); static void generate_mergejoin_paths(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *innerrel, @@ -1927,7 +1920,7 @@ hash_inner_and_outer(PlannerInfo *root, * if it is mergejoinable and involves vars from the two sub-relations * currently of interest. */ -static List * +List * select_mergejoin_clauses(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c index 0dbe2ac726..7271f044ec 100644 --- a/src/backend/optimizer/path/joinrels.c +++ b/src/backend/optimizer/path/joinrels.c @@ -924,6 +924,8 @@ populate_joinrel_with_paths(PlannerInfo *root, RelOptInfo *rel1, /* Apply partitionwise join technique, if possible. */ try_partitionwise_join(root, rel1, rel2, joinrel, sjinfo, restrictlist); + + populate_joinrel_uniquekeys(root, joinrel, rel1, rel2, restrictlist, sjinfo->jointype); } diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index bd9a176d7d..139278829b 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -33,7 +33,6 @@ static bool pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys); static bool matches_boolean_partition_clause(RestrictInfo *rinfo, RelOptInfo *partrel, int partkeycol); -static Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); static bool right_merge_direction(PlannerInfo *root, PathKey *pathkey); @@ -1035,7 +1034,7 @@ convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel, * We need this to ensure that we don't return pathkeys describing values * that are unavailable above the level of the subquery scan. */ -static Var * +Var * find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle) { ListCell *lc; diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c new file mode 100644 index 0000000000..77ed2b2eff --- /dev/null +++ b/src/backend/optimizer/path/uniquekeys.c @@ -0,0 +1,1135 @@ +/*------------------------------------------------------------------------- + * + * uniquekeys.c + * Utilities for matching and building unique keys + * + * Portions Copyright (c) 2020, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/optimizer/path/uniquekeys.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "nodes/makefuncs.h" +#include "nodes/nodeFuncs.h" +#include "optimizer/pathnode.h" +#include "optimizer/paths.h" +#include "optimizer/appendinfo.h" +#include "optimizer/optimizer.h" +#include "optimizer/tlist.h" +#include "rewrite/rewriteManip.h" + + +/* + * This struct is used to help populate_joinrel_uniquekeys. + * + * added_to_joinrel is true if a uniquekey (from outerrel or innerrel) + * has been added to joinrel. + * useful is true if the exprs of the uniquekey still appears in joinrel. + */ +typedef struct UniqueKeyContextData +{ + UniqueKey *uniquekey; + bool added_to_joinrel; + bool useful; +} *UniqueKeyContext; + +static List *initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel); +static bool innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + bool reverse); + +static List *get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals); +static List *get_exprs_from_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + UniqueKey *ukey); +static void add_uniquekey_for_onerow(RelOptInfo *rel); +static bool add_combined_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype); + +/* Used for unique indexes checking for partitioned table */ +static bool index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind); +static IndexOptInfo *simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from); +static bool simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2); +static void adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_index); + +/* Helper function for grouped relation and distinct relation. */ +static void add_uniquekey_from_sortgroups(PlannerInfo *root, + RelOptInfo *rel, + List *sortgroups); + +/* + * populate_baserel_uniquekeys + * Populate 'baserel' uniquekeys list by looking at the rel's unique index + * and baserestrictinfo + */ +void +populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List *indexlist) +{ + ListCell *lc; + List *matched_uniq_indexes = NIL; + + /* Attrs appears in rel->reltarget->exprs. */ + Bitmapset *used_attrs = NULL; + + List *const_exprs = NIL; + List *expr_opfamilies = NIL; + + Assert(baserel->rtekind == RTE_RELATION); + + foreach(lc, indexlist) + { + IndexOptInfo *ind = (IndexOptInfo *) lfirst(lc); + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + matched_uniq_indexes = lappend(matched_uniq_indexes, ind); + } + + if (matched_uniq_indexes == NIL) + return; + + /* Check which attrs is used in baserel->reltarget */ + pull_varattnos((Node *)baserel->reltarget->exprs, baserel->relid, &used_attrs); + + /* Check which attrno is used at a mergeable const filter */ + foreach(lc, baserel->baserestrictinfo) + { + RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); + + if (rinfo->mergeopfamilies == NIL) + continue; + + if (bms_is_empty(rinfo->left_relids)) + { + const_exprs = lappend(const_exprs, get_rightop(rinfo->clause)); + } + else if (bms_is_empty(rinfo->right_relids)) + { + const_exprs = lappend(const_exprs, get_leftop(rinfo->clause)); + } + else + continue; + + expr_opfamilies = lappend(expr_opfamilies, rinfo->mergeopfamilies); + } + + foreach(lc, matched_uniq_indexes) + { + bool multi_nullvals, useful; + List *exprs = get_exprs_from_uniqueindex(lfirst_node(IndexOptInfo, lc), + const_exprs, + expr_opfamilies, + used_attrs, + &useful, + &multi_nullvals); + if (useful) + { + if (exprs == NIL) + { + /* All the columns in Unique Index matched with a restrictinfo */ + add_uniquekey_for_onerow(baserel); + return; + } + baserel->uniquekeys = lappend(baserel->uniquekeys, + makeUniqueKey(exprs, multi_nullvals)); + } + } +} + + +/* + * populate_partitionedrel_uniquekeys + * The UniqueKey on partitionrel comes from 2 cases: + * 1). Only one partition is involved in this query, the unique key can be + * copied to parent rel from childrel. + * 2). There are some unique index which includes partition key and exists + * in all the related partitions. + * We never mind rule 2 if we hit rule 1. + */ + +void +populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels) +{ + ListCell *lc; + List *global_uniq_indexlist = NIL; + RelOptInfo *childrel; + bool is_first = true; + + Assert(IS_PARTITIONED_REL(rel)); + + if (childrels == NIL) + return; + + /* + * If there is only one partition used in this query, the UniqueKey in childrel is + * still valid in parent level, but we need convert the format from child expr to + * parent expr. + */ + if (list_length(childrels) == 1) + { + /* Check for Rule 1 */ + RelOptInfo *childrel = linitial_node(RelOptInfo, childrels); + ListCell *lc; + Assert(childrel->reloptkind == RELOPT_OTHER_MEMBER_REL); + if (relation_is_onerow(childrel)) + { + add_uniquekey_for_onerow(rel); + return; + } + + foreach(lc, childrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + AppendRelInfo *appinfo = find_appinfo_by_child(root, childrel->relid); + List *parent_exprs = NIL; + bool can_reuse = true; + ListCell *lc2; + foreach(lc2, ukey->exprs) + { + Var *var = (Var *)lfirst(lc2); + /* + * If the expr comes from a expression, it is hard to build the expression + * in parent so ignore that case for now. + */ + if(!IsA(var, Var)) + { + can_reuse = false; + break; + } + /* Convert it to parent var */ + parent_exprs = lappend(parent_exprs, find_parent_var(appinfo, var)); + } + if (can_reuse) + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(parent_exprs, + ukey->multi_nullvals)); + } + } + else + { + /* Check for rule 2 */ + childrel = linitial_node(RelOptInfo, childrels); + foreach(lc, childrel->indexlist) + { + IndexOptInfo *ind = lfirst(lc); + IndexOptInfo *modified_index; + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + + /* + * During simple_copy_indexinfo_to_parent, we need to convert var from + * child var to parent var, index on expression is too complex to handle. + * so ignore it for now. + */ + if (ind->indexprs != NIL) + continue; + + modified_index = simple_copy_indexinfo_to_parent(root, rel, ind); + /* + * If the unique index doesn't contain partkey, then it is unique + * on this partition only, so it is useless for us. + */ + if (!index_constains_partkey(rel, modified_index)) + continue; + + global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index); + } + + if (global_uniq_indexlist != NIL) + { + foreach(lc, childrels) + { + RelOptInfo *child = lfirst(lc); + if (is_first) + { + is_first = false; + continue; + } + adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist); + } + /* Now we have a list of unique index which are exactly same on all childrels, + * Set the UniqueKey just like it is non-partition table + */ + populate_baserel_uniquekeys(root, rel, global_uniq_indexlist); + } + } +} + + +/* + * populate_distinctrel_uniquekeys + */ +void +populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel) +{ + /* The unique key before the distinct is still valid. */ + distinctrel->uniquekeys = list_copy(inputrel->uniquekeys); + add_uniquekey_from_sortgroups(root, distinctrel, root->parse->distinctClause); +} + +/* + * populate_grouprel_uniquekeys + */ +void +populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel) + +{ + Query *parse = root->parse; + bool input_ukey_added = false; + ListCell *lc; + + if (relation_is_onerow(inputrel)) + { + add_uniquekey_for_onerow(grouprel); + return; + } + if (parse->groupingSets) + return; + + /* A Normal group by without grouping set. */ + if (parse->groupClause) + { + /* + * Current even the groupby clause is Unique already, but if query has aggref + * We have to create grouprel still. To keep the UnqiueKey short, we will check + * the UniqueKey of input_rel still valid, if so we reuse it. + */ + foreach(lc, inputrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (list_is_subset(ukey->exprs, grouprel->reltarget->exprs)) + { + grouprel->uniquekeys = lappend(grouprel->uniquekeys, + ukey); + input_ukey_added = true; + } + } + if (!input_ukey_added) + /* + * group by clause must be a super-set of grouprel->reltarget->exprs except the + * aggregation expr, so if such exprs is unique already, no bother to generate + * new uniquekey for group by exprs. + */ + add_uniquekey_from_sortgroups(root, + grouprel, + root->parse->groupClause); + } + else + /* It has aggregation but without a group by, so only one row returned */ + add_uniquekey_for_onerow(grouprel); +} + +/* + * simple_copy_uniquekeys + * Using a function for the one-line code makes us easy to check where we simply + * copied the uniquekey. + */ +void +simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel) +{ + newrel->uniquekeys = oldrel->uniquekeys; +} + +/* + * populate_unionrel_uniquekeys + */ +void +populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel) +{ + ListCell *lc; + List *exprs = NIL; + + Assert(unionrel->uniquekeys == NIL); + + foreach(lc, unionrel->reltarget->exprs) + { + exprs = lappend(exprs, lfirst(lc)); + } + + if (exprs == NIL) + /* SQL: select union select; is valid, we need to handle it here. */ + add_uniquekey_for_onerow(unionrel); + else + unionrel->uniquekeys = lappend(unionrel->uniquekeys, + makeUniqueKey(exprs,false)); + +} + +/* + * populate_joinrel_uniquekeys + * + * populate uniquekeys for joinrel. We will check each relation to see if its + * UniqueKey is still valid via innerrel_keeps_unique, if so, we add it to + * joinrel. The multi_nullvals field will be changed to true for some outer + * join cases and one-row UniqueKey needs to be converted to normal UniqueKey + * for the same case as well. + * For the uniquekey in either baserel which can't be unique after join, we still + * check to see if combination of UniqueKeys from both side is still useful for us. + * if yes, we add it to joinrel as well. + */ +void +populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *outerrel, RelOptInfo *innerrel, + List *restrictlist, JoinType jointype) +{ + ListCell *lc, *lc2; + List *clause_list = NIL; + List *outerrel_ukey_ctx; + List *innerrel_ukey_ctx; + bool inner_onerow, outer_onerow; + bool mergejoin_allowed; + + /* Care about the outerrel relation only for SEMI/ANTI join */ + if (jointype == JOIN_SEMI || jointype == JOIN_ANTI) + { + foreach(lc, outerrel->uniquekeys) + { + UniqueKey *uniquekey = lfirst_node(UniqueKey, lc); + if (list_is_subset(uniquekey->exprs, joinrel->reltarget->exprs)) + joinrel->uniquekeys = lappend(joinrel->uniquekeys, uniquekey); + } + return; + } + + Assert(jointype == JOIN_LEFT || jointype == JOIN_FULL || jointype == JOIN_INNER); + + /* Fast path */ + if (innerrel->uniquekeys == NIL || outerrel->uniquekeys == NIL) + return; + + inner_onerow = relation_is_onerow(innerrel); + outer_onerow = relation_is_onerow(outerrel); + + outerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(outerrel); + innerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(innerrel); + + clause_list = select_mergejoin_clauses(root, joinrel, outerrel, innerrel, + restrictlist, jointype, + &mergejoin_allowed); + + if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true /* reverse */)) + { + bool outer_impact = jointype == JOIN_FULL; + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + /* Outer relation has one row, and the unique key is not duplicated after join, + * the joinrel will still has one row unless the jointype == JOIN_FULL. + */ + if (outer_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (outer_onerow) + { + /* + * The onerow outerrel becomes multi rows and multi_nullvals + * will be changed to true. We also need to set the exprs correctly since it + * can't be NIL any more. + */ + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, outerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Change multi_nullvals to true due to the full join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, true)); + else + /* Just reuse it */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + } + ctx->added_to_joinrel = true; + } + } + + if (innerrel_keeps_unique(root, outerrel, innerrel, clause_list, false)) + { + bool outer_impact = jointype == JOIN_FULL || jointype == JOIN_LEFT;; + + foreach(lc, innerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + if (inner_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (inner_onerow) + { + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, innerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Need to change multi_nullvals to true due to the outer join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, + true)); + else + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + + } + ctx->added_to_joinrel = true; + } + } + + /* + * The combination of the UniqueKey from both sides is unique as well regardless + * of join type, but no bother to add it if its subset has been added to joinrel + * already or it is not useful for the joinrel. + */ + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx1 = (UniqueKeyContext) lfirst(lc); + if (ctx1->added_to_joinrel || !ctx1->useful) + continue; + foreach(lc2, innerrel_ukey_ctx) + { + UniqueKeyContext ctx2 = (UniqueKeyContext) lfirst(lc2); + if (ctx2->added_to_joinrel || !ctx2->useful) + continue; + if (add_combined_uniquekey(root, joinrel, outerrel, innerrel, + ctx1->uniquekey, ctx2->uniquekey, + jointype)) + /* If we set a onerow UniqueKey to joinrel, we don't need other. */ + return; + } + } +} + + +/* + * convert_subquery_uniquekeys + * + * Covert the UniqueKey in subquery to outer relation. + */ +void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel) +{ + ListCell *lc; + + if (sub_final_rel->uniquekeys == NIL) + return; + + if (relation_is_onerow(sub_final_rel)) + { + add_uniquekey_for_onerow(currel); + return; + } + + Assert(currel->subroot != NULL); + + foreach(lc, sub_final_rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + ListCell *lc; + List *exprs = NIL; + bool ukey_useful = true; + + /* One row case is handled above */ + Assert(ukey->exprs != NIL); + foreach(lc, ukey->exprs) + { + Var *var; + TargetEntry *tle = tlist_member(lfirst(lc), + currel->subroot->processed_tlist); + if (tle == NULL) + { + ukey_useful = false; + break; + } + var = find_var_for_subquery_tle(currel, tle); + if (var == NULL) + { + ukey_useful = false; + break; + } + exprs = lappend(exprs, var); + } + + if (ukey_useful) + currel->uniquekeys = lappend(currel->uniquekeys, + makeUniqueKey(exprs, + ukey->multi_nullvals)); + + } +} + +/* + * innerrel_keeps_unique + * + * Check if Unique key of the innerrel is valid after join. innerrel's UniqueKey + * will be still valid if innerrel's any-column mergeop outrerel's uniquekey + * exists in clause_list. + * + * Note: the clause_list must be a list of mergeable restrictinfo already. + */ +static bool +innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *clause_list, + bool reverse) +{ + ListCell *lc, *lc2, *lc3; + + if (outerrel->uniquekeys == NIL || innerrel->uniquekeys == NIL) + return false; + + /* Check if there is outerrel's uniquekey in mergeable clause. */ + foreach(lc, outerrel->uniquekeys) + { + List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs; + bool clauselist_matchs_all_exprs = true; + foreach(lc2, outer_uq_exprs) + { + Node *outer_uq_expr = lfirst(lc2); + bool find_uq_expr_in_clauselist = false; + foreach(lc3, clause_list) + { + RestrictInfo *rinfo = lfirst_node(RestrictInfo, lc3); + Node *outer_expr; + if (reverse) + outer_expr = rinfo->outer_is_left ? get_rightop(rinfo->clause) : get_leftop(rinfo->clause); + else + outer_expr = rinfo->outer_is_left ? get_leftop(rinfo->clause) : get_rightop(rinfo->clause); + if (equal(outer_expr, outer_uq_expr)) + { + find_uq_expr_in_clauselist = true; + break; + } + } + if (!find_uq_expr_in_clauselist) + { + /* No need to check the next exprs in the current uniquekey */ + clauselist_matchs_all_exprs = false; + break; + } + } + + if (clauselist_matchs_all_exprs) + return true; + } + return false; +} + + +/* + * relation_is_onerow + * Check if it is a one-row relation by checking UniqueKey. + */ +bool +relation_is_onerow(RelOptInfo *rel) +{ + UniqueKey *ukey; + if (rel->uniquekeys == NIL) + return false; + ukey = linitial_node(UniqueKey, rel->uniquekeys); + return ukey->exprs == NIL && list_length(rel->uniquekeys) == 1; +} + +/* + * relation_has_uniquekeys_for + * Returns true if we have proofs that 'rel' cannot return multiple rows with + * the same values in each of 'exprs'. Otherwise returns false. + */ +bool +relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel, + List *exprs, bool allow_multinulls) +{ + ListCell *lc; + + /* + * For UniqueKey->onerow case, the uniquekey->exprs is empty as well + * so we can't rely on list_is_subset to handle this special cases + */ + if (exprs == NIL) + return false; + + foreach(lc, rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (ukey->multi_nullvals && !allow_multinulls) + continue; + if (list_is_subset(ukey->exprs, exprs)) + return true; + } + return false; +} + + +/* + * get_exprs_from_uniqueindex + * + * Return a list of exprs which is unique. set useful to false if this + * unique index is not useful for us. + */ +static List * +get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals) +{ + List *exprs = NIL; + ListCell *indexpr_item; + int c = 0; + + *useful = true; + *multi_nullvals = false; + + indexpr_item = list_head(unique_index->indexprs); + for(c = 0; c < unique_index->ncolumns; c++) + { + int attr = unique_index->indexkeys[c]; + Expr *expr; + bool matched_const = false; + ListCell *lc1, *lc2; + + if(attr > 0) + { + expr = list_nth_node(TargetEntry, unique_index->indextlist, c)->expr; + } + else if (attr == 0) + { + /* Expression index */ + expr = lfirst(indexpr_item); + indexpr_item = lnext(unique_index->indexprs, indexpr_item); + } + else /* attr < 0 */ + { + /* Index on system column is not supported */ + Assert(false); + } + + /* + * Check index_col = Const case with regarding to opfamily checking + * If we can remove the index_col from the final UniqueKey->exprs. + */ + forboth(lc1, const_exprs, lc2, const_expr_opfamilies) + { + if (list_member_oid((List *)lfirst(lc2), unique_index->opfamily[c]) + && match_index_to_operand((Node *) lfirst(lc1), c, unique_index)) + { + matched_const = true; + break; + } + } + + if (matched_const) + continue; + + /* Check if the indexed expr is used in rel */ + if (attr > 0) + { + /* + * Normal Indexed column, if the col is not used, then the index is useless + * for uniquekey. + */ + attr -= FirstLowInvalidHeapAttributeNumber; + + if (!bms_is_member(attr, used_varattrs)) + { + *useful = false; + break; + } + } + else if (!list_member(unique_index->rel->reltarget->exprs, expr)) + { + /* Expression index but the expression is not used in rel */ + *useful = false; + break; + } + + /* check not null property. */ + if (attr == 0) + { + /* We never know if a expression yields null or not */ + *multi_nullvals = true; + } + else if (!bms_is_member(attr, unique_index->rel->notnullattrs) + && !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber, + unique_index->rel->notnullattrs)) + { + *multi_nullvals = true; + } + + exprs = lappend(exprs, expr); + } + return exprs; +} + + +/* + * add_uniquekey_for_onerow + * If we are sure that the relation only returns one row, then all the columns + * are unique. However we don't need to create UniqueKey for every column, we + * just set exprs = NIL and overwrites all the other UniqueKey on this RelOptInfo + * since this one has strongest semantics. + */ +void +add_uniquekey_for_onerow(RelOptInfo *rel) +{ + /* + * We overwrite the previous UniqueKey on purpose since this one has the + * strongest semantic. + */ + rel->uniquekeys = list_make1(makeUniqueKey(NIL, false)); +} + + +/* + * initililze_uniquecontext_for_joinrel + * Return a List of UniqueKeyContext for an inputrel + */ +static List * +initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel) +{ + List *res = NIL; + ListCell *lc; + foreach(lc, inputrel->uniquekeys) + { + UniqueKeyContext context; + context = palloc(sizeof(struct UniqueKeyContextData)); + context->uniquekey = lfirst_node(UniqueKey, lc); + context->added_to_joinrel = false; + context->useful = true; + res = lappend(res, context); + } + return res; +} + + +/* + * get_exprs_from_uniquekey + * Unify the way of get List of exprs from a one-row UniqueKey or + * normal UniqueKey. for the onerow case, every expr in rel1 is a valid + * UniqueKey. Return a List of exprs. + * + * rel1: The relation which you want to get the exprs. + * ukey: The UniqueKey you want to get the exprs. + */ +static List * +get_exprs_from_uniquekey(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *rel1, UniqueKey *ukey) +{ + ListCell *lc; + bool onerow = rel1 != NULL && relation_is_onerow(rel1); + + List *res = NIL; + Assert(onerow || ukey); + if (onerow) + { + /* Only cares about the exprs still exist in joinrel */ + foreach(lc, joinrel->reltarget->exprs) + { + Bitmapset *relids = pull_varnos(root, lfirst(lc)); + if (bms_is_subset(relids, rel1->relids)) + { + res = lappend(res, list_make1(lfirst(lc))); + } + } + } + else + { + res = list_make1(ukey->exprs); + } + return res; +} + +/* + * Partitioned table Unique Keys. + * The partition table unique key is maintained as: + * 1. The index must be unique as usual. + * 2. The index must contains partition key. + * 3. The index must exist on all the child rel. see simple_indexinfo_equal for + * how we compare it. + */ + +/* + * index_constains_partkey + * return true if the index contains the partiton key. + */ +static bool +index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind) +{ + ListCell *lc; + int i; + Assert(IS_PARTITIONED_REL(partrel)); + Assert(partrel->part_scheme->partnatts > 0); + + for(i = 0; i < partrel->part_scheme->partnatts; i++) + { + Node *part_expr = linitial(partrel->partexprs[i]); + bool found_in_index = false; + foreach(lc, ind->indextlist) + { + Expr *index_expr = lfirst_node(TargetEntry, lc)->expr; + if (equal(index_expr, part_expr)) + { + found_in_index = true; + break; + } + } + if (!found_in_index) + return false; + } + return true; +} + +/* + * simple_indexinfo_equal + * + * Used to check if the 2 index is same as each other. The index here + * is COPIED from childrel and did some tiny changes(see + * simple_copy_indexinfo_to_parent) + */ +static bool +simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2) +{ + Size oid_cmp_len = sizeof(Oid) * ind1->ncolumns; + + return ind1->ncolumns == ind2->ncolumns && + ind1->unique == ind2->unique && + memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0 && + memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0 && + memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0 && + memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0 && + equal(get_tlist_exprs(ind1->indextlist, true), + get_tlist_exprs(ind2->indextlist, true)); +} + + +/* + * The below macros are used for simple_copy_indexinfo_to_parent which is so + * customized that I don't want to put it to copyfuncs.c. So copy it here. + */ +#define COPY_POINTER_FIELD(fldname, sz) \ + do { \ + Size _size = (sz); \ + newnode->fldname = palloc(_size); \ + memcpy(newnode->fldname, from->fldname, _size); \ + } while (0) + +#define COPY_NODE_FIELD(fldname) \ + (newnode->fldname = copyObjectImpl(from->fldname)) + +#define COPY_SCALAR_FIELD(fldname) \ + (newnode->fldname = from->fldname) + + +/* + * simple_copy_indexinfo_to_parent (from partition) + * Copy the IndexInfo from child relation to parent relation with some modification, + * which is used to test: + * 1. If the same index exists in all the childrels. + * 2. If the parentrel->reltarget/basicrestrict info matches this index. + */ +static IndexOptInfo * +simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from) +{ + IndexOptInfo *newnode = makeNode(IndexOptInfo); + AppendRelInfo *appinfo = find_appinfo_by_child(root, from->rel->relid); + ListCell *lc; + int idx = 0; + + COPY_SCALAR_FIELD(ncolumns); + COPY_SCALAR_FIELD(nkeycolumns); + COPY_SCALAR_FIELD(unique); + COPY_SCALAR_FIELD(immediate); + /* We just need to know if it is NIL or not */ + COPY_SCALAR_FIELD(indpred); + COPY_SCALAR_FIELD(predOK); + COPY_POINTER_FIELD(indexkeys, from->ncolumns * sizeof(int)); + COPY_POINTER_FIELD(indexcollations, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opfamily, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opcintype, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(sortopfamily, from->ncolumns * sizeof(Oid)); + COPY_NODE_FIELD(indextlist); + + /* Convert index exprs on child expr to expr on parent */ + foreach(lc, newnode->indextlist) + { + TargetEntry *tle = lfirst_node(TargetEntry, lc); + /* Index on expression is ignored */ + Assert(IsA(tle->expr, Var)); + tle->expr = (Expr *) find_parent_var(appinfo, (Var *) tle->expr); + newnode->indexkeys[idx] = castNode(Var, tle->expr)->varattno; + idx++; + } + newnode->rel = parentrel; + return newnode; +} + +/* + * adjust_partition_unique_indexlist + * + * global_unique_indexes: At the beginning, it contains the copy & modified + * unique index from the first partition. And then check if each index in it still + * exists in the following partitions. If no, remove it. at last, it has an + * index list which exists in all the partitions. + */ +static void +adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_indexes) +{ + ListCell *lc, *lc2; + foreach(lc, *global_unique_indexes) + { + IndexOptInfo *g_ind = lfirst_node(IndexOptInfo, lc); + bool found_in_child = false; + + foreach(lc2, childrel->indexlist) + { + IndexOptInfo *p_ind = lfirst_node(IndexOptInfo, lc2); + IndexOptInfo *p_ind_copy; + if (!p_ind->unique || !p_ind->immediate || + (p_ind->indpred != NIL && !p_ind->predOK)) + continue; + p_ind_copy = simple_copy_indexinfo_to_parent(root, parentrel, p_ind); + if (simple_indexinfo_equal(p_ind_copy, g_ind)) + { + found_in_child = true; + break; + } + } + if (!found_in_child) + /* The index doesn't exist in childrel, remove it from global_unique_indexes */ + *global_unique_indexes = foreach_delete_current(*global_unique_indexes, lc); + } +} + +/* Helper function for groupres/distinctrel */ +static void +add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgroups) +{ + Query *parse = root->parse; + List *exprs; + + /* + * XXX: If there are some vars which is not in current levelsup, the semantic is + * imprecise, should we avoid it or not? levelsup = 1 is just a demo, maybe we need to + * check every level other than 0, if so, looks we have to write another + * pull_var_walker. + */ + List *upper_vars = pull_vars_of_level((Node*)sortgroups, 1); + + if (upper_vars != NIL) + return; + + exprs = get_sortgrouplist_exprs(sortgroups, parse->targetList); + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(exprs, + false /* sortgroupclause can't be multi_nullvals */)); +} + + +/* + * add_combined_uniquekey + * The combination of both UniqueKeys is a valid UniqueKey for joinrel no matter + * the jointype. + */ +bool +add_combined_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype) +{ + + ListCell *lc1, *lc2; + + /* Either side has multi_nullvals or we have outer join, + * the combined UniqueKey has multi_nullvals */ + bool multi_nullvals = outer_ukey->multi_nullvals || + inner_ukey->multi_nullvals || IS_OUTER_JOIN(jointype); + + /* The only case we can get onerow joinrel after join */ + if (relation_is_onerow(outer_rel) + && relation_is_onerow(inner_rel) + && jointype == JOIN_INNER) + { + add_uniquekey_for_onerow(joinrel); + return true; + } + + foreach(lc1, get_exprs_from_uniquekey(root, joinrel, outer_rel, outer_ukey)) + { + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, inner_rel, inner_ukey)) + { + List *exprs = list_concat_copy(lfirst_node(List, lc1), lfirst_node(List, lc2)); + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(exprs, + multi_nullvals)); + } + } + return false; +} diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index 424d25cbd5..8d8e493f5c 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -2387,6 +2387,8 @@ grouping_planner(PlannerInfo *root, bool inheritance_update, add_path(final_rel, path); } + simple_copy_uniquekeys(current_rel, final_rel); + /* * Generate partial paths for final_rel, too, if outer query levels might * be able to make use of them. @@ -3899,6 +3901,8 @@ create_grouping_paths(PlannerInfo *root, } set_cheapest(grouped_rel); + + populate_grouprel_uniquekeys(root, grouped_rel, input_rel); return grouped_rel; } @@ -4621,7 +4625,7 @@ create_window_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(window_rel); - + simple_copy_uniquekeys(input_rel, window_rel); return window_rel; } @@ -4935,7 +4939,7 @@ create_distinct_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(distinct_rel); - + populate_distinctrel_uniquekeys(root, input_rel, distinct_rel); return distinct_rel; } @@ -5196,6 +5200,8 @@ create_ordered_paths(PlannerInfo *root, */ Assert(ordered_rel->pathlist != NIL); + simple_copy_uniquekeys(input_rel, ordered_rel); + return ordered_rel; } @@ -6073,6 +6079,9 @@ adjust_paths_for_srfs(PlannerInfo *root, RelOptInfo *rel, if (list_length(targets) == 1) return; + /* UniqueKey is not valid after handling the SRF. */ + rel->uniquekeys = NIL; + /* * Stack SRF-evaluation nodes atop each path for the rel. * diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c index becdcbb872..b7626545bf 100644 --- a/src/backend/optimizer/prep/prepunion.c +++ b/src/backend/optimizer/prep/prepunion.c @@ -689,6 +689,8 @@ generate_union_paths(SetOperationStmt *op, PlannerInfo *root, /* Undo effects of possibly forcing tuple_fraction to 0 */ root->tuple_fraction = save_fraction; + /* Add the UniqueKeys */ + populate_unionrel_uniquekeys(root, result_rel); return result_rel; } diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c index 86922a273c..6817c9c787 100644 --- a/src/backend/optimizer/util/appendinfo.c +++ b/src/backend/optimizer/util/appendinfo.c @@ -746,3 +746,47 @@ find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos) } return appinfos; } + +/* + * find_appinfo_by_child + * + */ +AppendRelInfo * +find_appinfo_by_child(PlannerInfo *root, Index child_index) +{ + ListCell *lc; + foreach(lc, root->append_rel_list) + { + AppendRelInfo *appinfo = lfirst_node(AppendRelInfo, lc); + if (appinfo->child_relid == child_index) + return appinfo; + } + elog(ERROR, "parent relation cant be found"); + return NULL; +} + +/* + * find_parent_var + * + */ +Var * +find_parent_var(AppendRelInfo *appinfo, Var *child_var) +{ + ListCell *lc; + Var *res = NULL; + Index attno = 1; + foreach(lc, appinfo->translated_vars) + { + Node *child_node = lfirst(lc); + if (equal(child_node, child_var)) + { + res = copyObject(child_var); + res->varattno = attno; + res->varno = appinfo->parent_relid; + } + attno++; + } + if (res == NULL) + elog(ERROR, "parent var can't be found."); + return res; +} diff --git a/src/backend/optimizer/util/inherit.c b/src/backend/optimizer/util/inherit.c index be1c9ddd96..3eec1f4d74 100644 --- a/src/backend/optimizer/util/inherit.c +++ b/src/backend/optimizer/util/inherit.c @@ -736,6 +736,7 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, { Node *onecq = (Node *) lfirst(lc2); bool pseudoconstant; + RestrictInfo *child_rinfo; /* check for pseudoconstant (no Vars or volatile functions) */ pseudoconstant = @@ -747,14 +748,15 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, root->hasPseudoConstantQuals = true; } /* reconstitute RestrictInfo with appropriate properties */ - childquals = lappend(childquals, - make_restrictinfo(root, - (Expr *) onecq, - rinfo->is_pushed_down, - rinfo->outerjoin_delayed, - pseudoconstant, - rinfo->security_level, - NULL, NULL, NULL)); + child_rinfo = make_restrictinfo(root, + (Expr *) onecq, + rinfo->is_pushed_down, + rinfo->outerjoin_delayed, + pseudoconstant, + rinfo->security_level, + NULL, NULL, NULL); + child_rinfo->mergeopfamilies = rinfo->mergeopfamilies; + childquals = lappend(childquals, child_rinfo); /* track minimum security level among child quals */ cq_min_security = Min(cq_min_security, rinfo->security_level); } diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 48a7ebfe45..4fe1824eb0 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -16,6 +16,7 @@ #include "nodes/execnodes.h" #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" extern A_Expr *makeA_Expr(A_Expr_Kind kind, List *name, @@ -106,4 +107,6 @@ extern GroupingSet *makeGroupingSet(GroupingSetKind kind, List *content, int loc extern VacuumRelation *makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols); +extern UniqueKey* makeUniqueKey(List *exprs, bool multi_nullvals); + #endif /* MAKEFUNC_H */ diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index e22df890ef..3320273ac1 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -265,6 +265,7 @@ typedef enum NodeTag T_EquivalenceMember, T_PathKey, T_PathTarget, + T_UniqueKey, T_RestrictInfo, T_IndexClause, T_PlaceHolderVar, diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index 0d61f04d27..80561b1003 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -739,6 +739,7 @@ typedef struct RelOptInfo QualCost baserestrictcost; /* cost of evaluating the above */ Index baserestrict_min_security; /* min security_level found in * baserestrictinfo */ + List *uniquekeys; /* List of UniqueKey */ List *joininfo; /* RestrictInfo structures for join clauses * involving this rel */ bool has_eclass_joins; /* T means joininfo is incomplete */ @@ -1059,6 +1060,28 @@ typedef struct PathKey } PathKey; +/* + * UniqueKey + * + * Represents the unique properties held by a RelOptInfo. + * + * exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL + * is a special case of UniqueKey, which means there is only 1 row in that + * relation. + * multi_nullvals: true means multi null values may exist in these exprs, so the + * uniqueness is not guaranteed in this case. This field is necessary for + * remove_useless_join & reduce_unique_semijoins where we don't mind these + * duplicated NULL values. It is set to true for 2 cases. One is a unique key + * from a unique index but the related column is nullable. The other one is for + * outer join. see populate_joinrel_uniquekeys for detail. + */ +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + /* * PathTarget * @@ -2496,7 +2519,7 @@ typedef enum * * flags indicating what kinds of grouping are possible. * partial_costs_set is true if the agg_partial_costs and agg_final_costs - * have been initialized. + * have been initialized. * agg_partial_costs gives partial aggregation costs. * agg_final_costs gives finalization costs. * target_parallel_safe is true if target is parallel safe. @@ -2526,8 +2549,8 @@ typedef struct * limit_tuples is an estimated bound on the number of output tuples, * or -1 if no LIMIT or couldn't estimate. * count_est and offset_est are the estimated values of the LIMIT and OFFSET - * expressions computed by preprocess_limit() (see comments for - * preprocess_limit() for more information). + * expressions computed by preprocess_limit() (see comments for + * preprocess_limit() for more information). */ typedef struct { diff --git a/src/include/nodes/pg_list.h b/src/include/nodes/pg_list.h index 404e03f132..85c65e910c 100644 --- a/src/include/nodes/pg_list.h +++ b/src/include/nodes/pg_list.h @@ -566,6 +566,16 @@ extern pg_nodiscard List *list_delete_first(List *list); extern pg_nodiscard List *list_delete_last(List *list); extern pg_nodiscard List *list_delete_nth_cell(List *list, int n); extern pg_nodiscard List *list_delete_cell(List *list, ListCell *cell); +extern bool list_is_subset(const List *members, const List *target); + +extern List *list_delete(List *list, void *datum); +extern List *list_delete_ptr(List *list, void *datum); +extern List *list_delete_int(List *list, int datum); +extern List *list_delete_oid(List *list, Oid datum); +extern List *list_delete_first(List *list); +extern List *list_delete_last(List *list); +extern List *list_delete_nth_cell(List *list, int n); +extern List *list_delete_cell(List *list, ListCell *cell); extern List *list_union(const List *list1, const List *list2); extern List *list_union_ptr(const List *list1, const List *list2); diff --git a/src/include/optimizer/appendinfo.h b/src/include/optimizer/appendinfo.h index 4cbf8c26cc..43b8d05f69 100644 --- a/src/include/optimizer/appendinfo.h +++ b/src/include/optimizer/appendinfo.h @@ -32,4 +32,7 @@ extern Relids adjust_child_relids_multilevel(PlannerInfo *root, Relids relids, extern AppendRelInfo **find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos); +extern AppendRelInfo *find_appinfo_by_child(PlannerInfo *root, Index child_index); +extern Var *find_parent_var(AppendRelInfo *appinfo, Var *child_var); + #endif /* APPENDINFO_H */ diff --git a/src/include/optimizer/optimizer.h b/src/include/optimizer/optimizer.h index d587952b7d..843aafa51e 100644 --- a/src/include/optimizer/optimizer.h +++ b/src/include/optimizer/optimizer.h @@ -23,6 +23,7 @@ #define OPTIMIZER_H #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" /* Test if an expression node represents a SRF call. Beware multiple eval! */ #define IS_SRF_CALL(node) \ @@ -168,6 +169,7 @@ extern TargetEntry *get_sortgroupref_tle(Index sortref, List *targetList); extern TargetEntry *get_sortgroupclause_tle(SortGroupClause *sgClause, List *targetList); +extern Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); extern Node *get_sortgroupclause_expr(SortGroupClause *sgClause, List *targetList); extern List *get_sortgrouplist_exprs(List *sgClauses, diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h index 035d3e1206..1adf99c9ee 100644 --- a/src/include/optimizer/paths.h +++ b/src/include/optimizer/paths.h @@ -247,5 +247,48 @@ extern PathKey *make_canonical_pathkey(PlannerInfo *root, int strategy, bool nulls_first); extern void add_paths_to_append_rel(PlannerInfo *root, RelOptInfo *rel, List *live_childrels); +extern List *select_mergejoin_clauses(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + JoinType jointype, + bool *mergejoin_allowed); + +/* + * uniquekeys.c + * Utilities for matching and building unique keys + */ +extern void populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List* unique_index_list); +extern void populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels); +extern void populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel); +extern void populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel); +extern void populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel); +extern void simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel); +extern void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel); +extern void populate_joinrel_uniquekeys(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + RelOptInfo *rel2, + List *restrictlist, + JoinType jointype); + +extern bool relation_has_uniquekeys_for(PlannerInfo *root, + RelOptInfo *rel, + List *exprs, + bool allow_multinulls); +extern bool relation_is_onerow(RelOptInfo *rel); #endif /* PATHS_H */ -- 2.26.2 --yh7cw345mbttadjz Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v38-0003-Extend-UniqueKeys.patch" ^ permalink raw reply [nested|flat] 91+ messages in thread
* [PATCH v37 2/6] Introduce UniqueKey attributes on RelOptInfo struct. @ 2020-05-11 07:50 一挃 <[email protected]> 0 siblings, 0 replies; 91+ messages in thread From: 一挃 @ 2020-05-11 07:50 UTC (permalink / raw) UniqueKey is a set of exprs on RelOptInfo which represents the exprs will be unique on the given RelOptInfo. You can see README.uniquekey for more information. --- src/backend/nodes/copyfuncs.c | 13 + src/backend/nodes/list.c | 31 + src/backend/nodes/makefuncs.c | 13 + src/backend/nodes/outfuncs.c | 11 + src/backend/nodes/readfuncs.c | 10 + src/backend/optimizer/path/Makefile | 3 +- src/backend/optimizer/path/README.uniquekey | 131 +++ src/backend/optimizer/path/allpaths.c | 10 + src/backend/optimizer/path/joinpath.c | 9 +- src/backend/optimizer/path/joinrels.c | 2 + src/backend/optimizer/path/pathkeys.c | 3 +- src/backend/optimizer/path/uniquekeys.c | 1131 +++++++++++++++++++ src/backend/optimizer/plan/planner.c | 13 +- src/backend/optimizer/prep/prepunion.c | 2 + src/backend/optimizer/util/appendinfo.c | 44 + src/backend/optimizer/util/inherit.c | 16 +- src/include/nodes/makefuncs.h | 3 + src/include/nodes/nodes.h | 1 + src/include/nodes/pathnodes.h | 29 +- src/include/nodes/pg_list.h | 2 + src/include/optimizer/appendinfo.h | 3 + src/include/optimizer/optimizer.h | 2 + src/include/optimizer/paths.h | 43 + 23 files changed, 1502 insertions(+), 23 deletions(-) create mode 100644 src/backend/optimizer/path/README.uniquekey create mode 100644 src/backend/optimizer/path/uniquekeys.c diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 2b4d7654cc..34392f5553 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2271,6 +2271,16 @@ _copyPathKey(const PathKey *from) return newnode; } +static UniqueKey * +_copyUniqueKey(const UniqueKey *from) +{ + UniqueKey *newnode = makeNode(UniqueKey); + + COPY_NODE_FIELD(exprs); + COPY_SCALAR_FIELD(multi_nullvals); + + return newnode; +} /* * _copyRestrictInfo */ @@ -5149,6 +5159,9 @@ copyObjectImpl(const void *from) case T_PathKey: retval = _copyPathKey(from); break; + case T_UniqueKey: + retval = _copyUniqueKey(from); + break; case T_RestrictInfo: retval = _copyRestrictInfo(from); break; diff --git a/src/backend/nodes/list.c b/src/backend/nodes/list.c index efa44342c4..20daf4a9fd 100644 --- a/src/backend/nodes/list.c +++ b/src/backend/nodes/list.c @@ -687,6 +687,37 @@ list_member_oid(const List *list, Oid datum) return false; } +/* + * return true iff every entry in "members" list is also present + * in the "target" list. + */ +bool +list_is_subset(const List *members, const List *target) +{ + const ListCell *lc1, *lc2; + + Assert(IsPointerList(members)); + Assert(IsPointerList(target)); + check_list_invariants(members); + check_list_invariants(target); + + foreach(lc1, members) + { + bool found = false; + foreach(lc2, target) + { + if (equal(lfirst(lc1), lfirst(lc2))) + { + found = true; + break; + } + } + if (!found) + return false; + } + return true; +} + /* * Delete the n'th cell (counting from 0) in list. * diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 49de285f01..646cf7c9a1 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -814,3 +814,16 @@ makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols) v->va_cols = va_cols; return v; } + + +/* + * makeUniqueKey + */ +UniqueKey* +makeUniqueKey(List *exprs, bool multi_nullvals) +{ + UniqueKey * ukey = makeNode(UniqueKey); + ukey->exprs = exprs; + ukey->multi_nullvals = multi_nullvals; + return ukey; +} diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 08a049232e..53cf4fcfa1 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2426,6 +2426,14 @@ _outPathKey(StringInfo str, const PathKey *node) WRITE_BOOL_FIELD(pk_nulls_first); } +static void +_outUniqueKey(StringInfo str, const UniqueKey *node) +{ + WRITE_NODE_TYPE("UNIQUEKEY"); + WRITE_NODE_FIELD(exprs); + WRITE_BOOL_FIELD(multi_nullvals); +} + static void _outPathTarget(StringInfo str, const PathTarget *node) { @@ -4125,6 +4133,9 @@ outNode(StringInfo str, const void *obj) case T_PathKey: _outPathKey(str, obj); break; + case T_UniqueKey: + _outUniqueKey(str, obj); + break; case T_PathTarget: _outPathTarget(str, obj); break; diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index ab7b535caa..7b9e8c3292 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -452,6 +452,14 @@ _readSetOperationStmt(void) READ_DONE(); } +static UniqueKey * +_readUniqueKey(void) +{ + READ_LOCALS(UniqueKey); + READ_NODE_FIELD(exprs); + READ_BOOL_FIELD(multi_nullvals); + READ_DONE(); +} /* * Stuff from primnodes.h. @@ -2654,6 +2662,8 @@ parseNodeString(void) return_value = _readCommonTableExpr(); else if (MATCH("SETOPERATIONSTMT", 16)) return_value = _readSetOperationStmt(); + else if (MATCH("UNIQUEKEY", 9)) + return_value = _readUniqueKey(); else if (MATCH("ALIAS", 5)) return_value = _readAlias(); else if (MATCH("RANGEVAR", 8)) diff --git a/src/backend/optimizer/path/Makefile b/src/backend/optimizer/path/Makefile index 1e199ff66f..7b9820c25f 100644 --- a/src/backend/optimizer/path/Makefile +++ b/src/backend/optimizer/path/Makefile @@ -21,6 +21,7 @@ OBJS = \ joinpath.o \ joinrels.o \ pathkeys.o \ - tidpath.o + tidpath.o \ + uniquekeys.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/optimizer/path/README.uniquekey b/src/backend/optimizer/path/README.uniquekey new file mode 100644 index 0000000000..5eac761995 --- /dev/null +++ b/src/backend/optimizer/path/README.uniquekey @@ -0,0 +1,131 @@ +1. What is UniqueKey? +We can think UniqueKey is a set of exprs for a RelOptInfo, which we are insure +that doesn't yields same result among all the rows. The simplest UniqueKey +format is primary key. + +However we define the UnqiueKey as below. + +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + +exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL +is a special case of UniqueKey, which means there is only one row in that +relation.it has a stronger semantic than others. like SELECT uk FROM t; uk is +normal unique key and may have different values. SELECT colx FROM t WHERE uk = +const. colx is unique AND we have only 1 value. This field can used for +innerrel_is_unique. this logic is handled specially in add_uniquekey_for_onerow +function. + +multi_nullvals: true means multi null values may exist in these exprs, so the +uniqueness is not guaranteed in this case. This field is necessary for +remove_useless_join & reduce_unique_semijoins where we don't mind these +duplicated NULL values. It is set to true for 2 cases. One is a unique key +from a unique index but the related column is nullable. The other one is for +outer join. see populate_joinrel_uniquekeys for detail. + + +The UniqueKey can be used at the following cases at least: +1. remove_useless_joins. +2. reduce_semianti_joins +3. remove distinct node if distinct clause is unique. +4. remove aggnode if group by clause is unique. +5. Index Skip Scan (WIP) +6. Aggregation Push Down without 2 phase aggregation if the join can't + duplicated the aggregated rows. (work in progress feature) + +2. How is it maintained? + +We have a set of populate_xxx_unqiuekeys functions to maintain the uniquekey on +various cases. xxx includes baserel, joinrel, partitionedrel, distinctrel, +groupedrel, unionrel. and we also need to convert the uniquekey from subquery +to outer relation, which is what convert_subquery_uniquekeys does. + +1. The first part is about baserel. We handled 3 cases. suppose we have Unique +Index on (a, b). + +1. SELECT a, b FROM t. UniqueKey (a, b) +2. SELECT a FROM t WHERE b = 1; UniqueKey (a) +3. SELECT .. FROM t WHERE a = 1 AND b = 1; UniqueKey (NIL). onerow case, every + column is Unique. + +2. The next part is joinrel, this part is most error-prone, we simplified the rules +like below: +1. If the relation's UniqueKey can't be duplicated after join, then is will be + still valid for the join rel. The function we used here is + innerrel_keeps_unique. The basic idea is innerrel.any_col = outer.uk. + +2. If the UnqiueKey can't keep valid via the rule 1, the combination of the + UniqueKey from both sides are valid for sure. We can prove this as: if the + unique exprs from rel1 is duplicated by rel2, the duplicated rows must + contains different unique exprs from rel2. + +More considerations about onerow: +1. If relation with one row and it can't be duplicated, it is still possible + contains mulit_nullvas after outer join. +2. If the either UniqueKey can be duplicated after join, the can get one row + only when both side is one row AND there is no outer join. +3. Whenever the onerow UniqueKey is not a valid any more, we need to convert one + row UniqueKey to normal unique key since we don't store exprs for one-row + relation. get_exprs_from_uniquekeys will be used here. + + +More considerations about multi_nullvals after join: +1. If the original UnqiueKey has multi_nullvals, the final UniqueKey will have + mulit_nullvals in any case. +2. If a unique key doesn't allow mulit_nullvals, after some outer join, it + allows some outer join. + + +3. When we comes to subquery, we need to convert_subquery_unqiuekeys just like +convert_subquery_pathkeys. Only the UniqueKey insides subquery is referenced as +a Var in outer relation will be reused. The relationship between the outerrel.Var +and subquery.exprs is built with outerel->subroot->processed_tlist. + + +4. As for the SRF functions, it will break the uniqueness of uniquekey, However it +is handled in adjust_paths_for_srfs, which happens after the query_planner. so +we will maintain the UniqueKey until there and reset it to NIL at that +places. This can't help on distinct/group by elimination cases but probably help +in some other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is +semantic correctly. + + +5. As for inherit table, we first main the UnqiueKey on childrel as well. But for +partitioned table we need to maintain 2 different kinds of +UnqiueKey. 1). UniqueKey on the parent relation 2). UniqueKey on child +relation for partition wise query. + +Example: +CREATE TABLE p (a int not null, b int not null) partition by list (a); +CREATE TABLE p0 partition of p for values in (1); +CREATE TABLE p1 partition of p for values in (2); + +create unique index p0_b on p0(b); +create unique index p1_b on p1(b); + +Now b is only unique on partition level, so the distinct can't be removed on +the following cases. SELECT DISTINCT b FROM p; + +Another example is SELECT DISTINCT a, b FROM p WHERE a = 1; Since only one +partition is chosen, the UniqueKey on child relation is same as the UniqueKey on +parent relation. + +Another usage of UniqueKey on partition level is it be helpful for +partition-wise join. + +As for the UniqueKey on parent table level, it comes with 2 different ways, +1). the UniqueKey is also derived in UniqueKey index, but the index must be same +in all the related children relations and the unique index must contains +Partition Key in it. Example: + +CREATE UNIQUE INDEX p_ab ON p(a, b); -- where a is the partition key. + +-- Query +SELECT a, b FROM p; the (a, b) is a UniqueKey of p. + +2). If the parent relation has only one childrel, the UniqueKey on childrel is + the UniqueKey on parent as well. diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index 754f6d64f6..66d246fa1a 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -579,6 +579,12 @@ set_plain_rel_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) */ check_index_predicates(root, rel); + /* + * Now that we've marked which partial indexes are suitable, we can now + * build the relation's unique keys. + */ + populate_baserel_uniquekeys(root, rel, rel->indexlist); + /* Mark rel with estimated output rows, width, etc */ set_baserel_size_estimates(root, rel); } @@ -1314,6 +1320,8 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, /* Add paths to the append relation. */ add_paths_to_append_rel(root, rel, live_childrels); + if (IS_PARTITIONED_REL(rel)) + populate_partitionedrel_uniquekeys(root, rel, live_childrels); } @@ -2387,6 +2395,8 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, pathkeys, required_outer)); } + convert_subquery_uniquekeys(root, rel, sub_final_rel); + /* If outer rel allows parallelism, do same for partial paths. */ if (rel->consider_parallel && bms_is_empty(required_outer)) { diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c index 4a35903b29..f41be18e82 100644 --- a/src/backend/optimizer/path/joinpath.c +++ b/src/backend/optimizer/path/joinpath.c @@ -71,13 +71,6 @@ static void consider_parallel_mergejoin(PlannerInfo *root, static void hash_inner_and_outer(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, RelOptInfo *innerrel, JoinType jointype, JoinPathExtraData *extra); -static List *select_mergejoin_clauses(PlannerInfo *root, - RelOptInfo *joinrel, - RelOptInfo *outerrel, - RelOptInfo *innerrel, - List *restrictlist, - JoinType jointype, - bool *mergejoin_allowed); static void generate_mergejoin_paths(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *innerrel, @@ -1927,7 +1920,7 @@ hash_inner_and_outer(PlannerInfo *root, * if it is mergejoinable and involves vars from the two sub-relations * currently of interest. */ -static List * +List * select_mergejoin_clauses(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c index 2d343cd293..b9163ee8ff 100644 --- a/src/backend/optimizer/path/joinrels.c +++ b/src/backend/optimizer/path/joinrels.c @@ -924,6 +924,8 @@ populate_joinrel_with_paths(PlannerInfo *root, RelOptInfo *rel1, /* Apply partitionwise join technique, if possible. */ try_partitionwise_join(root, rel1, rel2, joinrel, sjinfo, restrictlist); + + populate_joinrel_uniquekeys(root, joinrel, rel1, rel2, restrictlist, sjinfo->jointype); } diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index ce9bf87e9b..7e596d4194 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -33,7 +33,6 @@ static bool pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys); static bool matches_boolean_partition_clause(RestrictInfo *rinfo, RelOptInfo *partrel, int partkeycol); -static Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); static bool right_merge_direction(PlannerInfo *root, PathKey *pathkey); @@ -1035,7 +1034,7 @@ convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel, * We need this to ensure that we don't return pathkeys describing values * that are unavailable above the level of the subquery scan. */ -static Var * +Var * find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle) { ListCell *lc; diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c new file mode 100644 index 0000000000..c7ad76d28f --- /dev/null +++ b/src/backend/optimizer/path/uniquekeys.c @@ -0,0 +1,1131 @@ +/*------------------------------------------------------------------------- + * + * uniquekeys.c + * Utilities for matching and building unique keys + * + * Portions Copyright (c) 2020, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/optimizer/path/uniquekeys.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "nodes/makefuncs.h" +#include "nodes/nodeFuncs.h" +#include "optimizer/pathnode.h" +#include "optimizer/paths.h" +#include "optimizer/appendinfo.h" +#include "optimizer/optimizer.h" +#include "optimizer/tlist.h" +#include "rewrite/rewriteManip.h" + + +/* + * This struct is used to help populate_joinrel_uniquekeys. + * + * added_to_joinrel is true if a uniquekey (from outerrel or innerrel) + * has been added to joinrel. + * useful is true if the exprs of the uniquekey still appears in joinrel. + */ +typedef struct UniqueKeyContextData +{ + UniqueKey *uniquekey; + bool added_to_joinrel; + bool useful; +} *UniqueKeyContext; + +static List *initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel); +static bool innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + bool reverse); + +static List *get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals); +static List *get_exprs_from_uniquekey(RelOptInfo *joinrel, + RelOptInfo *rel1, + UniqueKey *ukey); +static void add_uniquekey_for_onerow(RelOptInfo *rel); +static bool add_combined_uniquekey(RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype); + +/* Used for unique indexes checking for partitioned table */ +static bool index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind); +static IndexOptInfo *simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from); +static bool simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2); +static void adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_index); + +/* Helper function for grouped relation and distinct relation. */ +static void add_uniquekey_from_sortgroups(PlannerInfo *root, + RelOptInfo *rel, + List *sortgroups); + +/* + * populate_baserel_uniquekeys + * Populate 'baserel' uniquekeys list by looking at the rel's unique index + * and baserestrictinfo + */ +void +populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List *indexlist) +{ + ListCell *lc; + List *matched_uniq_indexes = NIL; + + /* Attrs appears in rel->reltarget->exprs. */ + Bitmapset *used_attrs = NULL; + + List *const_exprs = NIL; + List *expr_opfamilies = NIL; + + Assert(baserel->rtekind == RTE_RELATION); + + foreach(lc, indexlist) + { + IndexOptInfo *ind = (IndexOptInfo *) lfirst(lc); + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + matched_uniq_indexes = lappend(matched_uniq_indexes, ind); + } + + if (matched_uniq_indexes == NIL) + return; + + /* Check which attrs is used in baserel->reltarget */ + pull_varattnos((Node *)baserel->reltarget->exprs, baserel->relid, &used_attrs); + + /* Check which attrno is used at a mergeable const filter */ + foreach(lc, baserel->baserestrictinfo) + { + RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); + + if (rinfo->mergeopfamilies == NIL) + continue; + + if (bms_is_empty(rinfo->left_relids)) + { + const_exprs = lappend(const_exprs, get_rightop(rinfo->clause)); + } + else if (bms_is_empty(rinfo->right_relids)) + { + const_exprs = lappend(const_exprs, get_leftop(rinfo->clause)); + } + else + continue; + + expr_opfamilies = lappend(expr_opfamilies, rinfo->mergeopfamilies); + } + + foreach(lc, matched_uniq_indexes) + { + bool multi_nullvals, useful; + List *exprs = get_exprs_from_uniqueindex(lfirst_node(IndexOptInfo, lc), + const_exprs, + expr_opfamilies, + used_attrs, + &useful, + &multi_nullvals); + if (useful) + { + if (exprs == NIL) + { + /* All the columns in Unique Index matched with a restrictinfo */ + add_uniquekey_for_onerow(baserel); + return; + } + baserel->uniquekeys = lappend(baserel->uniquekeys, + makeUniqueKey(exprs, multi_nullvals)); + } + } +} + + +/* + * populate_partitionedrel_uniquekeys + * The UniqueKey on partitionrel comes from 2 cases: + * 1). Only one partition is involved in this query, the unique key can be + * copied to parent rel from childrel. + * 2). There are some unique index which includes partition key and exists + * in all the related partitions. + * We never mind rule 2 if we hit rule 1. + */ + +void +populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels) +{ + ListCell *lc; + List *global_uniq_indexlist = NIL; + RelOptInfo *childrel; + bool is_first = true; + + Assert(IS_PARTITIONED_REL(rel)); + + if (childrels == NIL) + return; + + /* + * If there is only one partition used in this query, the UniqueKey in childrel is + * still valid in parent level, but we need convert the format from child expr to + * parent expr. + */ + if (list_length(childrels) == 1) + { + /* Check for Rule 1 */ + RelOptInfo *childrel = linitial_node(RelOptInfo, childrels); + ListCell *lc; + Assert(childrel->reloptkind == RELOPT_OTHER_MEMBER_REL); + if (relation_is_onerow(childrel)) + { + add_uniquekey_for_onerow(rel); + return; + } + + foreach(lc, childrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + AppendRelInfo *appinfo = find_appinfo_by_child(root, childrel->relid); + List *parent_exprs = NIL; + bool can_reuse = true; + ListCell *lc2; + foreach(lc2, ukey->exprs) + { + Var *var = (Var *)lfirst(lc2); + /* + * If the expr comes from a expression, it is hard to build the expression + * in parent so ignore that case for now. + */ + if(!IsA(var, Var)) + { + can_reuse = false; + break; + } + /* Convert it to parent var */ + parent_exprs = lappend(parent_exprs, find_parent_var(appinfo, var)); + } + if (can_reuse) + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(parent_exprs, + ukey->multi_nullvals)); + } + } + else + { + /* Check for rule 2 */ + childrel = linitial_node(RelOptInfo, childrels); + foreach(lc, childrel->indexlist) + { + IndexOptInfo *ind = lfirst(lc); + IndexOptInfo *modified_index; + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + + /* + * During simple_copy_indexinfo_to_parent, we need to convert var from + * child var to parent var, index on expression is too complex to handle. + * so ignore it for now. + */ + if (ind->indexprs != NIL) + continue; + + modified_index = simple_copy_indexinfo_to_parent(root, rel, ind); + /* + * If the unique index doesn't contain partkey, then it is unique + * on this partition only, so it is useless for us. + */ + if (!index_constains_partkey(rel, modified_index)) + continue; + + global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index); + } + + if (global_uniq_indexlist != NIL) + { + foreach(lc, childrels) + { + RelOptInfo *child = lfirst(lc); + if (is_first) + { + is_first = false; + continue; + } + adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist); + } + /* Now we have a list of unique index which are exactly same on all childrels, + * Set the UniqueKey just like it is non-partition table + */ + populate_baserel_uniquekeys(root, rel, global_uniq_indexlist); + } + } +} + + +/* + * populate_distinctrel_uniquekeys + */ +void +populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel) +{ + /* The unique key before the distinct is still valid. */ + distinctrel->uniquekeys = list_copy(inputrel->uniquekeys); + add_uniquekey_from_sortgroups(root, distinctrel, root->parse->distinctClause); +} + +/* + * populate_grouprel_uniquekeys + */ +void +populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel) + +{ + Query *parse = root->parse; + bool input_ukey_added = false; + ListCell *lc; + + if (relation_is_onerow(inputrel)) + { + add_uniquekey_for_onerow(grouprel); + return; + } + if (parse->groupingSets) + return; + + /* A Normal group by without grouping set. */ + if (parse->groupClause) + { + /* + * Current even the groupby clause is Unique already, but if query has aggref + * We have to create grouprel still. To keep the UnqiueKey short, we will check + * the UniqueKey of input_rel still valid, if so we reuse it. + */ + foreach(lc, inputrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (list_is_subset(ukey->exprs, grouprel->reltarget->exprs)) + { + grouprel->uniquekeys = lappend(grouprel->uniquekeys, + ukey); + input_ukey_added = true; + } + } + if (!input_ukey_added) + /* + * group by clause must be a super-set of grouprel->reltarget->exprs except the + * aggregation expr, so if such exprs is unique already, no bother to generate + * new uniquekey for group by exprs. + */ + add_uniquekey_from_sortgroups(root, + grouprel, + root->parse->groupClause); + } + else + /* It has aggregation but without a group by, so only one row returned */ + add_uniquekey_for_onerow(grouprel); +} + +/* + * simple_copy_uniquekeys + * Using a function for the one-line code makes us easy to check where we simply + * copied the uniquekey. + */ +void +simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel) +{ + newrel->uniquekeys = oldrel->uniquekeys; +} + +/* + * populate_unionrel_uniquekeys + */ +void +populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel) +{ + ListCell *lc; + List *exprs = NIL; + + Assert(unionrel->uniquekeys == NIL); + + foreach(lc, unionrel->reltarget->exprs) + { + exprs = lappend(exprs, lfirst(lc)); + } + + if (exprs == NIL) + /* SQL: select union select; is valid, we need to handle it here. */ + add_uniquekey_for_onerow(unionrel); + else + unionrel->uniquekeys = lappend(unionrel->uniquekeys, + makeUniqueKey(exprs,false)); + +} + +/* + * populate_joinrel_uniquekeys + * + * populate uniquekeys for joinrel. We will check each relation to see if its + * UniqueKey is still valid via innerrel_keeps_unique, if so, we add it to + * joinrel. The multi_nullvals field will be changed to true for some outer + * join cases and one-row UniqueKey needs to be converted to normal UniqueKey + * for the same case as well. + * For the uniquekey in either baserel which can't be unique after join, we still + * check to see if combination of UniqueKeys from both side is still useful for us. + * if yes, we add it to joinrel as well. + */ +void +populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *outerrel, RelOptInfo *innerrel, + List *restrictlist, JoinType jointype) +{ + ListCell *lc, *lc2; + List *clause_list = NIL; + List *outerrel_ukey_ctx; + List *innerrel_ukey_ctx; + bool inner_onerow, outer_onerow; + bool mergejoin_allowed; + + /* Care about the outerrel relation only for SEMI/ANTI join */ + if (jointype == JOIN_SEMI || jointype == JOIN_ANTI) + { + foreach(lc, outerrel->uniquekeys) + { + UniqueKey *uniquekey = lfirst_node(UniqueKey, lc); + if (list_is_subset(uniquekey->exprs, joinrel->reltarget->exprs)) + joinrel->uniquekeys = lappend(joinrel->uniquekeys, uniquekey); + } + return; + } + + Assert(jointype == JOIN_LEFT || jointype == JOIN_FULL || jointype == JOIN_INNER); + + /* Fast path */ + if (innerrel->uniquekeys == NIL || outerrel->uniquekeys == NIL) + return; + + inner_onerow = relation_is_onerow(innerrel); + outer_onerow = relation_is_onerow(outerrel); + + outerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(outerrel); + innerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(innerrel); + + clause_list = select_mergejoin_clauses(root, joinrel, outerrel, innerrel, + restrictlist, jointype, + &mergejoin_allowed); + + if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true /* reverse */)) + { + bool outer_impact = jointype == JOIN_FULL; + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + /* Outer relation has one row, and the unique key is not duplicated after join, + * the joinrel will still has one row unless the jointype == JOIN_FULL. + */ + if (outer_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (outer_onerow) + { + /* + * The onerow outerrel becomes multi rows and multi_nullvals + * will be changed to true. We also need to set the exprs correctly since it + * can't be NIL any more. + */ + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(joinrel, outerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Change multi_nullvals to true due to the full join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, true)); + else + /* Just reuse it */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + } + ctx->added_to_joinrel = true; + } + } + + if (innerrel_keeps_unique(root, outerrel, innerrel, clause_list, false)) + { + bool outer_impact = jointype == JOIN_FULL || jointype == JOIN_LEFT;; + + foreach(lc, innerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + if (inner_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (inner_onerow) + { + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(joinrel, innerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Need to change multi_nullvals to true due to the outer join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, + true)); + else + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + + } + ctx->added_to_joinrel = true; + } + } + + /* + * The combination of the UniqueKey from both sides is unique as well regardless + * of join type, but no bother to add it if its subset has been added to joinrel + * already or it is not useful for the joinrel. + */ + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx1 = (UniqueKeyContext) lfirst(lc); + if (ctx1->added_to_joinrel || !ctx1->useful) + continue; + foreach(lc2, innerrel_ukey_ctx) + { + UniqueKeyContext ctx2 = (UniqueKeyContext) lfirst(lc2); + if (ctx2->added_to_joinrel || !ctx2->useful) + continue; + if (add_combined_uniquekey(joinrel, outerrel, innerrel, + ctx1->uniquekey, ctx2->uniquekey, + jointype)) + /* If we set a onerow UniqueKey to joinrel, we don't need other. */ + return; + } + } +} + + +/* + * convert_subquery_uniquekeys + * + * Covert the UniqueKey in subquery to outer relation. + */ +void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel) +{ + ListCell *lc; + + if (sub_final_rel->uniquekeys == NIL) + return; + + if (relation_is_onerow(sub_final_rel)) + { + add_uniquekey_for_onerow(currel); + return; + } + + Assert(currel->subroot != NULL); + + foreach(lc, sub_final_rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + ListCell *lc; + List *exprs = NIL; + bool ukey_useful = true; + + /* One row case is handled above */ + Assert(ukey->exprs != NIL); + foreach(lc, ukey->exprs) + { + Var *var; + TargetEntry *tle = tlist_member(lfirst(lc), + currel->subroot->processed_tlist); + if (tle == NULL) + { + ukey_useful = false; + break; + } + var = find_var_for_subquery_tle(currel, tle); + if (var == NULL) + { + ukey_useful = false; + break; + } + exprs = lappend(exprs, var); + } + + if (ukey_useful) + currel->uniquekeys = lappend(currel->uniquekeys, + makeUniqueKey(exprs, + ukey->multi_nullvals)); + + } +} + +/* + * innerrel_keeps_unique + * + * Check if Unique key of the innerrel is valid after join. innerrel's UniqueKey + * will be still valid if innerrel's any-column mergeop outrerel's uniquekey + * exists in clause_list. + * + * Note: the clause_list must be a list of mergeable restrictinfo already. + */ +static bool +innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *clause_list, + bool reverse) +{ + ListCell *lc, *lc2, *lc3; + + if (outerrel->uniquekeys == NIL || innerrel->uniquekeys == NIL) + return false; + + /* Check if there is outerrel's uniquekey in mergeable clause. */ + foreach(lc, outerrel->uniquekeys) + { + List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs; + bool clauselist_matchs_all_exprs = true; + foreach(lc2, outer_uq_exprs) + { + Node *outer_uq_expr = lfirst(lc2); + bool find_uq_expr_in_clauselist = false; + foreach(lc3, clause_list) + { + RestrictInfo *rinfo = lfirst_node(RestrictInfo, lc3); + Node *outer_expr; + if (reverse) + outer_expr = rinfo->outer_is_left ? get_rightop(rinfo->clause) : get_leftop(rinfo->clause); + else + outer_expr = rinfo->outer_is_left ? get_leftop(rinfo->clause) : get_rightop(rinfo->clause); + if (equal(outer_expr, outer_uq_expr)) + { + find_uq_expr_in_clauselist = true; + break; + } + } + if (!find_uq_expr_in_clauselist) + { + /* No need to check the next exprs in the current uniquekey */ + clauselist_matchs_all_exprs = false; + break; + } + } + + if (clauselist_matchs_all_exprs) + return true; + } + return false; +} + + +/* + * relation_is_onerow + * Check if it is a one-row relation by checking UniqueKey. + */ +bool +relation_is_onerow(RelOptInfo *rel) +{ + UniqueKey *ukey; + if (rel->uniquekeys == NIL) + return false; + ukey = linitial_node(UniqueKey, rel->uniquekeys); + return ukey->exprs == NIL && list_length(rel->uniquekeys) == 1; +} + +/* + * relation_has_uniquekeys_for + * Returns true if we have proofs that 'rel' cannot return multiple rows with + * the same values in each of 'exprs'. Otherwise returns false. + */ +bool +relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel, + List *exprs, bool allow_multinulls) +{ + ListCell *lc; + + /* + * For UniqueKey->onerow case, the uniquekey->exprs is empty as well + * so we can't rely on list_is_subset to handle this special cases + */ + if (exprs == NIL) + return false; + + foreach(lc, rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (ukey->multi_nullvals && !allow_multinulls) + continue; + if (list_is_subset(ukey->exprs, exprs)) + return true; + } + return false; +} + + +/* + * get_exprs_from_uniqueindex + * + * Return a list of exprs which is unique. set useful to false if this + * unique index is not useful for us. + */ +static List * +get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals) +{ + List *exprs = NIL; + ListCell *indexpr_item; + int c = 0; + + *useful = true; + *multi_nullvals = false; + + indexpr_item = list_head(unique_index->indexprs); + for(c = 0; c < unique_index->nkeycolumns; c++) + { + int attr = unique_index->indexkeys[c]; + Expr *expr; + bool matched_const = false; + ListCell *lc1, *lc2; + + if(attr > 0) + { + expr = list_nth_node(TargetEntry, unique_index->indextlist, c)->expr; + } + else if (attr == 0) + { + /* Expression index */ + expr = lfirst(indexpr_item); + indexpr_item = lnext(unique_index->indexprs, indexpr_item); + } + else /* attr < 0 */ + { + /* Index on system column is not supported */ + Assert(false); + } + + /* + * Check index_col = Const case with regarding to opfamily checking + * If we can remove the index_col from the final UniqueKey->exprs. + */ + forboth(lc1, const_exprs, lc2, const_expr_opfamilies) + { + if (list_member_oid((List *)lfirst(lc2), unique_index->opfamily[c]) + && match_index_to_operand((Node *) lfirst(lc1), c, unique_index)) + { + matched_const = true; + break; + } + } + + if (matched_const) + continue; + + /* Check if the indexed expr is used in rel */ + if (attr > 0) + { + /* + * Normal Indexed column, if the col is not used, then the index is useless + * for uniquekey. + */ + attr -= FirstLowInvalidHeapAttributeNumber; + + if (!bms_is_member(attr, used_varattrs)) + { + *useful = false; + break; + } + } + else if (!list_member(unique_index->rel->reltarget->exprs, expr)) + { + /* Expression index but the expression is not used in rel */ + *useful = false; + break; + } + + /* check not null property. */ + if (attr == 0) + { + /* We never know if a expression yields null or not */ + *multi_nullvals = true; + } + else if (!bms_is_member(attr, unique_index->rel->notnullattrs) + && !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber, + unique_index->rel->notnullattrs)) + { + *multi_nullvals = true; + } + + exprs = lappend(exprs, expr); + } + return exprs; +} + + +/* + * add_uniquekey_for_onerow + * If we are sure that the relation only returns one row, then all the columns + * are unique. However we don't need to create UniqueKey for every column, we + * just set exprs = NIL and overwrites all the other UniqueKey on this RelOptInfo + * since this one has strongest semantics. + */ +void +add_uniquekey_for_onerow(RelOptInfo *rel) +{ + /* + * We overwrite the previous UniqueKey on purpose since this one has the + * strongest semantic. + */ + rel->uniquekeys = list_make1(makeUniqueKey(NIL, false)); +} + + +/* + * initililze_uniquecontext_for_joinrel + * Return a List of UniqueKeyContext for an inputrel + */ +static List * +initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel) +{ + List *res = NIL; + ListCell *lc; + foreach(lc, inputrel->uniquekeys) + { + UniqueKeyContext context; + context = palloc(sizeof(struct UniqueKeyContextData)); + context->uniquekey = lfirst_node(UniqueKey, lc); + context->added_to_joinrel = false; + context->useful = true; + res = lappend(res, context); + } + return res; +} + + +/* + * get_exprs_from_uniquekey + * Unify the way of get List of exprs from a one-row UniqueKey or + * normal UniqueKey. for the onerow case, every expr in rel1 is a valid + * UniqueKey. Return a List of exprs. + * + * rel1: The relation which you want to get the exprs. + * ukey: The UniqueKey you want to get the exprs. + */ +static List * +get_exprs_from_uniquekey(RelOptInfo *joinrel, RelOptInfo *rel1, UniqueKey *ukey) +{ + ListCell *lc; + bool onerow = rel1 != NULL && relation_is_onerow(rel1); + + List *res = NIL; + Assert(onerow || ukey); + if (onerow) + { + /* Only cares about the exprs still exist in joinrel */ + foreach(lc, joinrel->reltarget->exprs) + { + Bitmapset *relids = pull_varnos(lfirst(lc)); + if (bms_is_subset(relids, rel1->relids)) + { + res = lappend(res, list_make1(lfirst(lc))); + } + } + } + else + { + res = list_make1(ukey->exprs); + } + return res; +} + +/* + * Partitioned table Unique Keys. + * The partition table unique key is maintained as: + * 1. The index must be unique as usual. + * 2. The index must contains partition key. + * 3. The index must exist on all the child rel. see simple_indexinfo_equal for + * how we compare it. + */ + +/* + * index_constains_partkey + * return true if the index contains the partiton key. + */ +static bool +index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind) +{ + ListCell *lc; + int i; + Assert(IS_PARTITIONED_REL(partrel)); + Assert(partrel->part_scheme->partnatts > 0); + + for(i = 0; i < partrel->part_scheme->partnatts; i++) + { + Node *part_expr = linitial(partrel->partexprs[i]); + bool found_in_index = false; + foreach(lc, ind->indextlist) + { + Expr *index_expr = lfirst_node(TargetEntry, lc)->expr; + if (equal(index_expr, part_expr)) + { + found_in_index = true; + break; + } + } + if (!found_in_index) + return false; + } + return true; +} + +/* + * simple_indexinfo_equal + * + * Used to check if the 2 index is same as each other. The index here + * is COPIED from childrel and did some tiny changes(see + * simple_copy_indexinfo_to_parent) + */ +static bool +simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2) +{ + Size oid_cmp_len = sizeof(Oid) * ind1->ncolumns; + + return ind1->ncolumns == ind2->ncolumns && + ind1->unique == ind2->unique && + memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0 && + memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0 && + memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0 && + memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0 && + equal(get_tlist_exprs(ind1->indextlist, true), + get_tlist_exprs(ind2->indextlist, true)); +} + + +/* + * The below macros are used for simple_copy_indexinfo_to_parent which is so + * customized that I don't want to put it to copyfuncs.c. So copy it here. + */ +#define COPY_POINTER_FIELD(fldname, sz) \ + do { \ + Size _size = (sz); \ + newnode->fldname = palloc(_size); \ + memcpy(newnode->fldname, from->fldname, _size); \ + } while (0) + +#define COPY_NODE_FIELD(fldname) \ + (newnode->fldname = copyObjectImpl(from->fldname)) + +#define COPY_SCALAR_FIELD(fldname) \ + (newnode->fldname = from->fldname) + + +/* + * simple_copy_indexinfo_to_parent (from partition) + * Copy the IndexInfo from child relation to parent relation with some modification, + * which is used to test: + * 1. If the same index exists in all the childrels. + * 2. If the parentrel->reltarget/basicrestrict info matches this index. + */ +static IndexOptInfo * +simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from) +{ + IndexOptInfo *newnode = makeNode(IndexOptInfo); + AppendRelInfo *appinfo = find_appinfo_by_child(root, from->rel->relid); + ListCell *lc; + int idx = 0; + + COPY_SCALAR_FIELD(ncolumns); + COPY_SCALAR_FIELD(nkeycolumns); + COPY_SCALAR_FIELD(unique); + COPY_SCALAR_FIELD(immediate); + /* We just need to know if it is NIL or not */ + COPY_SCALAR_FIELD(indpred); + COPY_SCALAR_FIELD(predOK); + COPY_POINTER_FIELD(indexkeys, from->ncolumns * sizeof(int)); + COPY_POINTER_FIELD(indexcollations, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opfamily, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opcintype, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(sortopfamily, from->ncolumns * sizeof(Oid)); + COPY_NODE_FIELD(indextlist); + + /* Convert index exprs on child expr to expr on parent */ + foreach(lc, newnode->indextlist) + { + TargetEntry *tle = lfirst_node(TargetEntry, lc); + /* Index on expression is ignored */ + Assert(IsA(tle->expr, Var)); + tle->expr = (Expr *) find_parent_var(appinfo, (Var *) tle->expr); + newnode->indexkeys[idx] = castNode(Var, tle->expr)->varattno; + idx++; + } + newnode->rel = parentrel; + return newnode; +} + +/* + * adjust_partition_unique_indexlist + * + * global_unique_indexes: At the beginning, it contains the copy & modified + * unique index from the first partition. And then check if each index in it still + * exists in the following partitions. If no, remove it. at last, it has an + * index list which exists in all the partitions. + */ +static void +adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_indexes) +{ + ListCell *lc, *lc2; + foreach(lc, *global_unique_indexes) + { + IndexOptInfo *g_ind = lfirst_node(IndexOptInfo, lc); + bool found_in_child = false; + + foreach(lc2, childrel->indexlist) + { + IndexOptInfo *p_ind = lfirst_node(IndexOptInfo, lc2); + IndexOptInfo *p_ind_copy; + if (!p_ind->unique || !p_ind->immediate || + (p_ind->indpred != NIL && !p_ind->predOK)) + continue; + p_ind_copy = simple_copy_indexinfo_to_parent(root, parentrel, p_ind); + if (simple_indexinfo_equal(p_ind_copy, g_ind)) + { + found_in_child = true; + break; + } + } + if (!found_in_child) + /* The index doesn't exist in childrel, remove it from global_unique_indexes */ + *global_unique_indexes = foreach_delete_current(*global_unique_indexes, lc); + } +} + +/* Helper function for groupres/distinctrel */ +static void +add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgroups) +{ + Query *parse = root->parse; + List *exprs; + + /* + * XXX: If there are some vars which is not in current levelsup, the semantic is + * imprecise, should we avoid it or not? levelsup = 1 is just a demo, maybe we need to + * check every level other than 0, if so, looks we have to write another + * pull_var_walker. + */ + List *upper_vars = pull_vars_of_level((Node*)sortgroups, 1); + + if (upper_vars != NIL) + return; + + exprs = get_sortgrouplist_exprs(sortgroups, parse->targetList); + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(exprs, + false /* sortgroupclause can't be multi_nullvals */)); +} + + +/* + * add_combined_uniquekey + * The combination of both UniqueKeys is a valid UniqueKey for joinrel no matter + * the jointype. + */ +bool +add_combined_uniquekey(RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype) +{ + + ListCell *lc1, *lc2; + + /* Either side has multi_nullvals or we have outer join, + * the combined UniqueKey has multi_nullvals */ + bool multi_nullvals = outer_ukey->multi_nullvals || + inner_ukey->multi_nullvals || IS_OUTER_JOIN(jointype); + + /* The only case we can get onerow joinrel after join */ + if (relation_is_onerow(outer_rel) + && relation_is_onerow(inner_rel) + && jointype == JOIN_INNER) + { + add_uniquekey_for_onerow(joinrel); + return true; + } + + foreach(lc1, get_exprs_from_uniquekey(joinrel, outer_rel, outer_ukey)) + { + foreach(lc2, get_exprs_from_uniquekey(joinrel, inner_rel, inner_ukey)) + { + List *exprs = list_concat_copy(lfirst_node(List, lc1), lfirst_node(List, lc2)); + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(exprs, + multi_nullvals)); + } + } + return false; +} diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index 986d7a52e3..f1bbb8c427 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -2385,6 +2385,8 @@ grouping_planner(PlannerInfo *root, bool inheritance_update, add_path(final_rel, path); } + simple_copy_uniquekeys(current_rel, final_rel); + /* * Generate partial paths for final_rel, too, if outer query levels might * be able to make use of them. @@ -3895,6 +3897,8 @@ create_grouping_paths(PlannerInfo *root, } set_cheapest(grouped_rel); + + populate_grouprel_uniquekeys(root, grouped_rel, input_rel); return grouped_rel; } @@ -4615,7 +4619,7 @@ create_window_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(window_rel); - + simple_copy_uniquekeys(input_rel, window_rel); return window_rel; } @@ -4929,7 +4933,7 @@ create_distinct_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(distinct_rel); - + populate_distinctrel_uniquekeys(root, input_rel, distinct_rel); return distinct_rel; } @@ -5190,6 +5194,8 @@ create_ordered_paths(PlannerInfo *root, */ Assert(ordered_rel->pathlist != NIL); + simple_copy_uniquekeys(input_rel, ordered_rel); + return ordered_rel; } @@ -6067,6 +6073,9 @@ adjust_paths_for_srfs(PlannerInfo *root, RelOptInfo *rel, if (list_length(targets) == 1) return; + /* UniqueKey is not valid after handling the SRF. */ + rel->uniquekeys = NIL; + /* * Stack SRF-evaluation nodes atop each path for the rel. * diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c index 745f443e5c..ce290cb97b 100644 --- a/src/backend/optimizer/prep/prepunion.c +++ b/src/backend/optimizer/prep/prepunion.c @@ -689,6 +689,8 @@ generate_union_paths(SetOperationStmt *op, PlannerInfo *root, /* Undo effects of possibly forcing tuple_fraction to 0 */ root->tuple_fraction = save_fraction; + /* Add the UniqueKeys */ + populate_unionrel_uniquekeys(root, result_rel); return result_rel; } diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c index d722063cf3..44c37ecffc 100644 --- a/src/backend/optimizer/util/appendinfo.c +++ b/src/backend/optimizer/util/appendinfo.c @@ -746,3 +746,47 @@ find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos) } return appinfos; } + +/* + * find_appinfo_by_child + * + */ +AppendRelInfo * +find_appinfo_by_child(PlannerInfo *root, Index child_index) +{ + ListCell *lc; + foreach(lc, root->append_rel_list) + { + AppendRelInfo *appinfo = lfirst_node(AppendRelInfo, lc); + if (appinfo->child_relid == child_index) + return appinfo; + } + elog(ERROR, "parent relation cant be found"); + return NULL; +} + +/* + * find_parent_var + * + */ +Var * +find_parent_var(AppendRelInfo *appinfo, Var *child_var) +{ + ListCell *lc; + Var *res = NULL; + Index attno = 1; + foreach(lc, appinfo->translated_vars) + { + Node *child_node = lfirst(lc); + if (equal(child_node, child_var)) + { + res = copyObject(child_var); + res->varattno = attno; + res->varno = appinfo->parent_relid; + } + attno++; + } + if (res == NULL) + elog(ERROR, "parent var can't be found."); + return res; +} diff --git a/src/backend/optimizer/util/inherit.c b/src/backend/optimizer/util/inherit.c index 3132fd35a5..d66b40ec50 100644 --- a/src/backend/optimizer/util/inherit.c +++ b/src/backend/optimizer/util/inherit.c @@ -736,6 +736,7 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, { Node *onecq = (Node *) lfirst(lc2); bool pseudoconstant; + RestrictInfo *child_rinfo; /* check for pseudoconstant (no Vars or volatile functions) */ pseudoconstant = @@ -747,13 +748,14 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, root->hasPseudoConstantQuals = true; } /* reconstitute RestrictInfo with appropriate properties */ - childquals = lappend(childquals, - make_restrictinfo((Expr *) onecq, - rinfo->is_pushed_down, - rinfo->outerjoin_delayed, - pseudoconstant, - rinfo->security_level, - NULL, NULL, NULL)); + child_rinfo = make_restrictinfo((Expr *) onecq, + rinfo->is_pushed_down, + rinfo->outerjoin_delayed, + pseudoconstant, + rinfo->security_level, + NULL, NULL, NULL); + child_rinfo->mergeopfamilies = rinfo->mergeopfamilies; + childquals = lappend(childquals, child_rinfo); /* track minimum security level among child quals */ cq_min_security = Min(cq_min_security, rinfo->security_level); } diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 31d9aedeeb..c83f17acb7 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -16,6 +16,7 @@ #include "nodes/execnodes.h" #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" extern A_Expr *makeA_Expr(A_Expr_Kind kind, List *name, @@ -105,4 +106,6 @@ extern GroupingSet *makeGroupingSet(GroupingSetKind kind, List *content, int loc extern VacuumRelation *makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols); +extern UniqueKey* makeUniqueKey(List *exprs, bool multi_nullvals); + #endif /* MAKEFUNC_H */ diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 7ddd8c011b..2bfbd353c7 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -263,6 +263,7 @@ typedef enum NodeTag T_EquivalenceMember, T_PathKey, T_PathTarget, + T_UniqueKey, T_RestrictInfo, T_IndexClause, T_PlaceHolderVar, diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index 925f2eac3f..5737cd76ce 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -729,6 +729,7 @@ typedef struct RelOptInfo QualCost baserestrictcost; /* cost of evaluating the above */ Index baserestrict_min_security; /* min security_level found in * baserestrictinfo */ + List *uniquekeys; /* List of UniqueKey */ List *joininfo; /* RestrictInfo structures for join clauses * involving this rel */ bool has_eclass_joins; /* T means joininfo is incomplete */ @@ -1046,6 +1047,28 @@ typedef struct PathKey } PathKey; +/* + * UniqueKey + * + * Represents the unique properties held by a RelOptInfo. + * + * exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL + * is a special case of UniqueKey, which means there is only 1 row in that + * relation. + * multi_nullvals: true means multi null values may exist in these exprs, so the + * uniqueness is not guaranteed in this case. This field is necessary for + * remove_useless_join & reduce_unique_semijoins where we don't mind these + * duplicated NULL values. It is set to true for 2 cases. One is a unique key + * from a unique index but the related column is nullable. The other one is for + * outer join. see populate_joinrel_uniquekeys for detail. + */ +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + /* * PathTarget * @@ -2472,7 +2495,7 @@ typedef enum * * flags indicating what kinds of grouping are possible. * partial_costs_set is true if the agg_partial_costs and agg_final_costs - * have been initialized. + * have been initialized. * agg_partial_costs gives partial aggregation costs. * agg_final_costs gives finalization costs. * target_parallel_safe is true if target is parallel safe. @@ -2502,8 +2525,8 @@ typedef struct * limit_tuples is an estimated bound on the number of output tuples, * or -1 if no LIMIT or couldn't estimate. * count_est and offset_est are the estimated values of the LIMIT and OFFSET - * expressions computed by preprocess_limit() (see comments for - * preprocess_limit() for more information). + * expressions computed by preprocess_limit() (see comments for + * preprocess_limit() for more information). */ typedef struct { diff --git a/src/include/nodes/pg_list.h b/src/include/nodes/pg_list.h index ec231010ce..a1e279815c 100644 --- a/src/include/nodes/pg_list.h +++ b/src/include/nodes/pg_list.h @@ -543,6 +543,8 @@ extern bool list_member_ptr(const List *list, const void *datum); extern bool list_member_int(const List *list, int datum); extern bool list_member_oid(const List *list, Oid datum); +extern bool list_is_subset(const List *members, const List *target); + extern List *list_delete(List *list, void *datum); extern List *list_delete_ptr(List *list, void *datum); extern List *list_delete_int(List *list, int datum); diff --git a/src/include/optimizer/appendinfo.h b/src/include/optimizer/appendinfo.h index d6a27a60dd..e87c92a054 100644 --- a/src/include/optimizer/appendinfo.h +++ b/src/include/optimizer/appendinfo.h @@ -32,4 +32,7 @@ extern Relids adjust_child_relids_multilevel(PlannerInfo *root, Relids relids, extern AppendRelInfo **find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos); +extern AppendRelInfo *find_appinfo_by_child(PlannerInfo *root, Index child_index); +extern Var *find_parent_var(AppendRelInfo *appinfo, Var *child_var); + #endif /* APPENDINFO_H */ diff --git a/src/include/optimizer/optimizer.h b/src/include/optimizer/optimizer.h index 3e4171056e..9445141263 100644 --- a/src/include/optimizer/optimizer.h +++ b/src/include/optimizer/optimizer.h @@ -23,6 +23,7 @@ #define OPTIMIZER_H #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" /* * We don't want to include nodes/pathnodes.h here, because non-planner @@ -156,6 +157,7 @@ extern TargetEntry *get_sortgroupref_tle(Index sortref, List *targetList); extern TargetEntry *get_sortgroupclause_tle(SortGroupClause *sgClause, List *targetList); +extern Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); extern Node *get_sortgroupclause_expr(SortGroupClause *sgClause, List *targetList); extern List *get_sortgrouplist_exprs(List *sgClauses, diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h index 10b6e81079..9217a8d6c6 100644 --- a/src/include/optimizer/paths.h +++ b/src/include/optimizer/paths.h @@ -240,5 +240,48 @@ extern PathKey *make_canonical_pathkey(PlannerInfo *root, int strategy, bool nulls_first); extern void add_paths_to_append_rel(PlannerInfo *root, RelOptInfo *rel, List *live_childrels); +extern List *select_mergejoin_clauses(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + JoinType jointype, + bool *mergejoin_allowed); + +/* + * uniquekeys.c + * Utilities for matching and building unique keys + */ +extern void populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List* unique_index_list); +extern void populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels); +extern void populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel); +extern void populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel); +extern void populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel); +extern void simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel); +extern void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel); +extern void populate_joinrel_uniquekeys(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + RelOptInfo *rel2, + List *restrictlist, + JoinType jointype); + +extern bool relation_has_uniquekeys_for(PlannerInfo *root, + RelOptInfo *rel, + List *exprs, + bool allow_multinulls); +extern bool relation_is_onerow(RelOptInfo *rel); #endif /* PATHS_H */ -- 2.21.0 --snesplnlompos6d4 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v37-0003-Extend-UniqueKeys.patch" ^ permalink raw reply [nested|flat] 91+ messages in thread
* [PATCH v38 2/6] Introduce UniqueKey attributes on RelOptInfo struct. @ 2020-05-11 07:50 一挃 <[email protected]> 0 siblings, 0 replies; 91+ messages in thread From: 一挃 @ 2020-05-11 07:50 UTC (permalink / raw) UniqueKey is a set of exprs on RelOptInfo which represents the exprs will be unique on the given RelOptInfo. You can see README.uniquekey for more information. --- src/backend/nodes/copyfuncs.c | 13 + src/backend/nodes/list.c | 31 + src/backend/nodes/makefuncs.c | 13 + src/backend/nodes/outfuncs.c | 11 + src/backend/nodes/readfuncs.c | 10 + src/backend/optimizer/path/Makefile | 3 +- src/backend/optimizer/path/README.uniquekey | 131 +++ src/backend/optimizer/path/allpaths.c | 10 + src/backend/optimizer/path/joinpath.c | 9 +- src/backend/optimizer/path/joinrels.c | 2 + src/backend/optimizer/path/pathkeys.c | 3 +- src/backend/optimizer/path/uniquekeys.c | 1135 +++++++++++++++++++ src/backend/optimizer/plan/planner.c | 13 +- src/backend/optimizer/prep/prepunion.c | 2 + src/backend/optimizer/util/appendinfo.c | 44 + src/backend/optimizer/util/inherit.c | 18 +- src/include/nodes/makefuncs.h | 3 + src/include/nodes/nodes.h | 1 + src/include/nodes/pathnodes.h | 29 +- src/include/nodes/pg_list.h | 10 + src/include/optimizer/appendinfo.h | 3 + src/include/optimizer/optimizer.h | 2 + src/include/optimizer/paths.h | 43 + 23 files changed, 1515 insertions(+), 24 deletions(-) create mode 100644 src/backend/optimizer/path/README.uniquekey create mode 100644 src/backend/optimizer/path/uniquekeys.c diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..75c1c5e824 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2296,6 +2296,16 @@ _copyPathKey(const PathKey *from) return newnode; } +static UniqueKey * +_copyUniqueKey(const UniqueKey *from) +{ + UniqueKey *newnode = makeNode(UniqueKey); + + COPY_NODE_FIELD(exprs); + COPY_SCALAR_FIELD(multi_nullvals); + + return newnode; +} /* * _copyRestrictInfo */ @@ -5220,6 +5230,9 @@ copyObjectImpl(const void *from) case T_PathKey: retval = _copyPathKey(from); break; + case T_UniqueKey: + retval = _copyUniqueKey(from); + break; case T_RestrictInfo: retval = _copyRestrictInfo(from); break; diff --git a/src/backend/nodes/list.c b/src/backend/nodes/list.c index dbf6b30233..ca099495a1 100644 --- a/src/backend/nodes/list.c +++ b/src/backend/nodes/list.c @@ -702,6 +702,37 @@ list_member_oid(const List *list, Oid datum) return false; } +/* + * return true iff every entry in "members" list is also present + * in the "target" list. + */ +bool +list_is_subset(const List *members, const List *target) +{ + const ListCell *lc1, *lc2; + + Assert(IsPointerList(members)); + Assert(IsPointerList(target)); + check_list_invariants(members); + check_list_invariants(target); + + foreach(lc1, members) + { + bool found = false; + foreach(lc2, target) + { + if (equal(lfirst(lc1), lfirst(lc2))) + { + found = true; + break; + } + } + if (!found) + return false; + } + return true; +} + /* * Delete the n'th cell (counting from 0) in list. * diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 01c110cd2f..40415d0f5b 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -815,3 +815,16 @@ makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols) v->va_cols = va_cols; return v; } + + +/* + * makeUniqueKey + */ +UniqueKey* +makeUniqueKey(List *exprs, bool multi_nullvals) +{ + UniqueKey * ukey = makeNode(UniqueKey); + ukey->exprs = exprs; + ukey->multi_nullvals = multi_nullvals; + return ukey; +} diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..44154cde6a 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2456,6 +2456,14 @@ _outPathKey(StringInfo str, const PathKey *node) WRITE_BOOL_FIELD(pk_nulls_first); } +static void +_outUniqueKey(StringInfo str, const UniqueKey *node) +{ + WRITE_NODE_TYPE("UNIQUEKEY"); + WRITE_NODE_FIELD(exprs); + WRITE_BOOL_FIELD(multi_nullvals); +} + static void _outPathTarget(StringInfo str, const PathTarget *node) { @@ -4198,6 +4206,9 @@ outNode(StringInfo str, const void *obj) case T_PathKey: _outPathKey(str, obj); break; + case T_UniqueKey: + _outUniqueKey(str, obj); + break; case T_PathTarget: _outPathTarget(str, obj); break; diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index c5e136e9c3..b3e212bf1c 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -492,6 +492,14 @@ _readSetOperationStmt(void) READ_DONE(); } +static UniqueKey * +_readUniqueKey(void) +{ + READ_LOCALS(UniqueKey); + READ_NODE_FIELD(exprs); + READ_BOOL_FIELD(multi_nullvals); + READ_DONE(); +} /* * Stuff from primnodes.h. @@ -2717,6 +2725,8 @@ parseNodeString(void) return_value = _readCommonTableExpr(); else if (MATCH("SETOPERATIONSTMT", 16)) return_value = _readSetOperationStmt(); + else if (MATCH("UNIQUEKEY", 9)) + return_value = _readUniqueKey(); else if (MATCH("ALIAS", 5)) return_value = _readAlias(); else if (MATCH("RANGEVAR", 8)) diff --git a/src/backend/optimizer/path/Makefile b/src/backend/optimizer/path/Makefile index 1e199ff66f..7b9820c25f 100644 --- a/src/backend/optimizer/path/Makefile +++ b/src/backend/optimizer/path/Makefile @@ -21,6 +21,7 @@ OBJS = \ joinpath.o \ joinrels.o \ pathkeys.o \ - tidpath.o + tidpath.o \ + uniquekeys.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/optimizer/path/README.uniquekey b/src/backend/optimizer/path/README.uniquekey new file mode 100644 index 0000000000..5eac761995 --- /dev/null +++ b/src/backend/optimizer/path/README.uniquekey @@ -0,0 +1,131 @@ +1. What is UniqueKey? +We can think UniqueKey is a set of exprs for a RelOptInfo, which we are insure +that doesn't yields same result among all the rows. The simplest UniqueKey +format is primary key. + +However we define the UnqiueKey as below. + +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + +exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL +is a special case of UniqueKey, which means there is only one row in that +relation.it has a stronger semantic than others. like SELECT uk FROM t; uk is +normal unique key and may have different values. SELECT colx FROM t WHERE uk = +const. colx is unique AND we have only 1 value. This field can used for +innerrel_is_unique. this logic is handled specially in add_uniquekey_for_onerow +function. + +multi_nullvals: true means multi null values may exist in these exprs, so the +uniqueness is not guaranteed in this case. This field is necessary for +remove_useless_join & reduce_unique_semijoins where we don't mind these +duplicated NULL values. It is set to true for 2 cases. One is a unique key +from a unique index but the related column is nullable. The other one is for +outer join. see populate_joinrel_uniquekeys for detail. + + +The UniqueKey can be used at the following cases at least: +1. remove_useless_joins. +2. reduce_semianti_joins +3. remove distinct node if distinct clause is unique. +4. remove aggnode if group by clause is unique. +5. Index Skip Scan (WIP) +6. Aggregation Push Down without 2 phase aggregation if the join can't + duplicated the aggregated rows. (work in progress feature) + +2. How is it maintained? + +We have a set of populate_xxx_unqiuekeys functions to maintain the uniquekey on +various cases. xxx includes baserel, joinrel, partitionedrel, distinctrel, +groupedrel, unionrel. and we also need to convert the uniquekey from subquery +to outer relation, which is what convert_subquery_uniquekeys does. + +1. The first part is about baserel. We handled 3 cases. suppose we have Unique +Index on (a, b). + +1. SELECT a, b FROM t. UniqueKey (a, b) +2. SELECT a FROM t WHERE b = 1; UniqueKey (a) +3. SELECT .. FROM t WHERE a = 1 AND b = 1; UniqueKey (NIL). onerow case, every + column is Unique. + +2. The next part is joinrel, this part is most error-prone, we simplified the rules +like below: +1. If the relation's UniqueKey can't be duplicated after join, then is will be + still valid for the join rel. The function we used here is + innerrel_keeps_unique. The basic idea is innerrel.any_col = outer.uk. + +2. If the UnqiueKey can't keep valid via the rule 1, the combination of the + UniqueKey from both sides are valid for sure. We can prove this as: if the + unique exprs from rel1 is duplicated by rel2, the duplicated rows must + contains different unique exprs from rel2. + +More considerations about onerow: +1. If relation with one row and it can't be duplicated, it is still possible + contains mulit_nullvas after outer join. +2. If the either UniqueKey can be duplicated after join, the can get one row + only when both side is one row AND there is no outer join. +3. Whenever the onerow UniqueKey is not a valid any more, we need to convert one + row UniqueKey to normal unique key since we don't store exprs for one-row + relation. get_exprs_from_uniquekeys will be used here. + + +More considerations about multi_nullvals after join: +1. If the original UnqiueKey has multi_nullvals, the final UniqueKey will have + mulit_nullvals in any case. +2. If a unique key doesn't allow mulit_nullvals, after some outer join, it + allows some outer join. + + +3. When we comes to subquery, we need to convert_subquery_unqiuekeys just like +convert_subquery_pathkeys. Only the UniqueKey insides subquery is referenced as +a Var in outer relation will be reused. The relationship between the outerrel.Var +and subquery.exprs is built with outerel->subroot->processed_tlist. + + +4. As for the SRF functions, it will break the uniqueness of uniquekey, However it +is handled in adjust_paths_for_srfs, which happens after the query_planner. so +we will maintain the UniqueKey until there and reset it to NIL at that +places. This can't help on distinct/group by elimination cases but probably help +in some other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is +semantic correctly. + + +5. As for inherit table, we first main the UnqiueKey on childrel as well. But for +partitioned table we need to maintain 2 different kinds of +UnqiueKey. 1). UniqueKey on the parent relation 2). UniqueKey on child +relation for partition wise query. + +Example: +CREATE TABLE p (a int not null, b int not null) partition by list (a); +CREATE TABLE p0 partition of p for values in (1); +CREATE TABLE p1 partition of p for values in (2); + +create unique index p0_b on p0(b); +create unique index p1_b on p1(b); + +Now b is only unique on partition level, so the distinct can't be removed on +the following cases. SELECT DISTINCT b FROM p; + +Another example is SELECT DISTINCT a, b FROM p WHERE a = 1; Since only one +partition is chosen, the UniqueKey on child relation is same as the UniqueKey on +parent relation. + +Another usage of UniqueKey on partition level is it be helpful for +partition-wise join. + +As for the UniqueKey on parent table level, it comes with 2 different ways, +1). the UniqueKey is also derived in UniqueKey index, but the index must be same +in all the related children relations and the unique index must contains +Partition Key in it. Example: + +CREATE UNIQUE INDEX p_ab ON p(a, b); -- where a is the partition key. + +-- Query +SELECT a, b FROM p; the (a, b) is a UniqueKey of p. + +2). If the parent relation has only one childrel, the UniqueKey on childrel is + the UniqueKey on parent as well. diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index 37b4223adb..d52ad59f89 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -579,6 +579,12 @@ set_plain_rel_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) */ check_index_predicates(root, rel); + /* + * Now that we've marked which partial indexes are suitable, we can now + * build the relation's unique keys. + */ + populate_baserel_uniquekeys(root, rel, rel->indexlist); + /* Mark rel with estimated output rows, width, etc */ set_baserel_size_estimates(root, rel); } @@ -1297,6 +1303,8 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, /* Add paths to the append relation. */ add_paths_to_append_rel(root, rel, live_childrels); + if (IS_PARTITIONED_REL(rel)) + populate_partitionedrel_uniquekeys(root, rel, live_childrels); } @@ -2304,6 +2312,8 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, pathkeys, required_outer)); } + convert_subquery_uniquekeys(root, rel, sub_final_rel); + /* If outer rel allows parallelism, do same for partial paths. */ if (rel->consider_parallel && bms_is_empty(required_outer)) { diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c index 57ce97fd53..697ac047fb 100644 --- a/src/backend/optimizer/path/joinpath.c +++ b/src/backend/optimizer/path/joinpath.c @@ -71,13 +71,6 @@ static void consider_parallel_mergejoin(PlannerInfo *root, static void hash_inner_and_outer(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, RelOptInfo *innerrel, JoinType jointype, JoinPathExtraData *extra); -static List *select_mergejoin_clauses(PlannerInfo *root, - RelOptInfo *joinrel, - RelOptInfo *outerrel, - RelOptInfo *innerrel, - List *restrictlist, - JoinType jointype, - bool *mergejoin_allowed); static void generate_mergejoin_paths(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *innerrel, @@ -1927,7 +1920,7 @@ hash_inner_and_outer(PlannerInfo *root, * if it is mergejoinable and involves vars from the two sub-relations * currently of interest. */ -static List * +List * select_mergejoin_clauses(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c index 0dbe2ac726..7271f044ec 100644 --- a/src/backend/optimizer/path/joinrels.c +++ b/src/backend/optimizer/path/joinrels.c @@ -924,6 +924,8 @@ populate_joinrel_with_paths(PlannerInfo *root, RelOptInfo *rel1, /* Apply partitionwise join technique, if possible. */ try_partitionwise_join(root, rel1, rel2, joinrel, sjinfo, restrictlist); + + populate_joinrel_uniquekeys(root, joinrel, rel1, rel2, restrictlist, sjinfo->jointype); } diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index bd9a176d7d..139278829b 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -33,7 +33,6 @@ static bool pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys); static bool matches_boolean_partition_clause(RestrictInfo *rinfo, RelOptInfo *partrel, int partkeycol); -static Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); static bool right_merge_direction(PlannerInfo *root, PathKey *pathkey); @@ -1035,7 +1034,7 @@ convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel, * We need this to ensure that we don't return pathkeys describing values * that are unavailable above the level of the subquery scan. */ -static Var * +Var * find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle) { ListCell *lc; diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c new file mode 100644 index 0000000000..77ed2b2eff --- /dev/null +++ b/src/backend/optimizer/path/uniquekeys.c @@ -0,0 +1,1135 @@ +/*------------------------------------------------------------------------- + * + * uniquekeys.c + * Utilities for matching and building unique keys + * + * Portions Copyright (c) 2020, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/optimizer/path/uniquekeys.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "nodes/makefuncs.h" +#include "nodes/nodeFuncs.h" +#include "optimizer/pathnode.h" +#include "optimizer/paths.h" +#include "optimizer/appendinfo.h" +#include "optimizer/optimizer.h" +#include "optimizer/tlist.h" +#include "rewrite/rewriteManip.h" + + +/* + * This struct is used to help populate_joinrel_uniquekeys. + * + * added_to_joinrel is true if a uniquekey (from outerrel or innerrel) + * has been added to joinrel. + * useful is true if the exprs of the uniquekey still appears in joinrel. + */ +typedef struct UniqueKeyContextData +{ + UniqueKey *uniquekey; + bool added_to_joinrel; + bool useful; +} *UniqueKeyContext; + +static List *initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel); +static bool innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + bool reverse); + +static List *get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals); +static List *get_exprs_from_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + UniqueKey *ukey); +static void add_uniquekey_for_onerow(RelOptInfo *rel); +static bool add_combined_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype); + +/* Used for unique indexes checking for partitioned table */ +static bool index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind); +static IndexOptInfo *simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from); +static bool simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2); +static void adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_index); + +/* Helper function for grouped relation and distinct relation. */ +static void add_uniquekey_from_sortgroups(PlannerInfo *root, + RelOptInfo *rel, + List *sortgroups); + +/* + * populate_baserel_uniquekeys + * Populate 'baserel' uniquekeys list by looking at the rel's unique index + * and baserestrictinfo + */ +void +populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List *indexlist) +{ + ListCell *lc; + List *matched_uniq_indexes = NIL; + + /* Attrs appears in rel->reltarget->exprs. */ + Bitmapset *used_attrs = NULL; + + List *const_exprs = NIL; + List *expr_opfamilies = NIL; + + Assert(baserel->rtekind == RTE_RELATION); + + foreach(lc, indexlist) + { + IndexOptInfo *ind = (IndexOptInfo *) lfirst(lc); + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + matched_uniq_indexes = lappend(matched_uniq_indexes, ind); + } + + if (matched_uniq_indexes == NIL) + return; + + /* Check which attrs is used in baserel->reltarget */ + pull_varattnos((Node *)baserel->reltarget->exprs, baserel->relid, &used_attrs); + + /* Check which attrno is used at a mergeable const filter */ + foreach(lc, baserel->baserestrictinfo) + { + RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); + + if (rinfo->mergeopfamilies == NIL) + continue; + + if (bms_is_empty(rinfo->left_relids)) + { + const_exprs = lappend(const_exprs, get_rightop(rinfo->clause)); + } + else if (bms_is_empty(rinfo->right_relids)) + { + const_exprs = lappend(const_exprs, get_leftop(rinfo->clause)); + } + else + continue; + + expr_opfamilies = lappend(expr_opfamilies, rinfo->mergeopfamilies); + } + + foreach(lc, matched_uniq_indexes) + { + bool multi_nullvals, useful; + List *exprs = get_exprs_from_uniqueindex(lfirst_node(IndexOptInfo, lc), + const_exprs, + expr_opfamilies, + used_attrs, + &useful, + &multi_nullvals); + if (useful) + { + if (exprs == NIL) + { + /* All the columns in Unique Index matched with a restrictinfo */ + add_uniquekey_for_onerow(baserel); + return; + } + baserel->uniquekeys = lappend(baserel->uniquekeys, + makeUniqueKey(exprs, multi_nullvals)); + } + } +} + + +/* + * populate_partitionedrel_uniquekeys + * The UniqueKey on partitionrel comes from 2 cases: + * 1). Only one partition is involved in this query, the unique key can be + * copied to parent rel from childrel. + * 2). There are some unique index which includes partition key and exists + * in all the related partitions. + * We never mind rule 2 if we hit rule 1. + */ + +void +populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels) +{ + ListCell *lc; + List *global_uniq_indexlist = NIL; + RelOptInfo *childrel; + bool is_first = true; + + Assert(IS_PARTITIONED_REL(rel)); + + if (childrels == NIL) + return; + + /* + * If there is only one partition used in this query, the UniqueKey in childrel is + * still valid in parent level, but we need convert the format from child expr to + * parent expr. + */ + if (list_length(childrels) == 1) + { + /* Check for Rule 1 */ + RelOptInfo *childrel = linitial_node(RelOptInfo, childrels); + ListCell *lc; + Assert(childrel->reloptkind == RELOPT_OTHER_MEMBER_REL); + if (relation_is_onerow(childrel)) + { + add_uniquekey_for_onerow(rel); + return; + } + + foreach(lc, childrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + AppendRelInfo *appinfo = find_appinfo_by_child(root, childrel->relid); + List *parent_exprs = NIL; + bool can_reuse = true; + ListCell *lc2; + foreach(lc2, ukey->exprs) + { + Var *var = (Var *)lfirst(lc2); + /* + * If the expr comes from a expression, it is hard to build the expression + * in parent so ignore that case for now. + */ + if(!IsA(var, Var)) + { + can_reuse = false; + break; + } + /* Convert it to parent var */ + parent_exprs = lappend(parent_exprs, find_parent_var(appinfo, var)); + } + if (can_reuse) + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(parent_exprs, + ukey->multi_nullvals)); + } + } + else + { + /* Check for rule 2 */ + childrel = linitial_node(RelOptInfo, childrels); + foreach(lc, childrel->indexlist) + { + IndexOptInfo *ind = lfirst(lc); + IndexOptInfo *modified_index; + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + + /* + * During simple_copy_indexinfo_to_parent, we need to convert var from + * child var to parent var, index on expression is too complex to handle. + * so ignore it for now. + */ + if (ind->indexprs != NIL) + continue; + + modified_index = simple_copy_indexinfo_to_parent(root, rel, ind); + /* + * If the unique index doesn't contain partkey, then it is unique + * on this partition only, so it is useless for us. + */ + if (!index_constains_partkey(rel, modified_index)) + continue; + + global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index); + } + + if (global_uniq_indexlist != NIL) + { + foreach(lc, childrels) + { + RelOptInfo *child = lfirst(lc); + if (is_first) + { + is_first = false; + continue; + } + adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist); + } + /* Now we have a list of unique index which are exactly same on all childrels, + * Set the UniqueKey just like it is non-partition table + */ + populate_baserel_uniquekeys(root, rel, global_uniq_indexlist); + } + } +} + + +/* + * populate_distinctrel_uniquekeys + */ +void +populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel) +{ + /* The unique key before the distinct is still valid. */ + distinctrel->uniquekeys = list_copy(inputrel->uniquekeys); + add_uniquekey_from_sortgroups(root, distinctrel, root->parse->distinctClause); +} + +/* + * populate_grouprel_uniquekeys + */ +void +populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel) + +{ + Query *parse = root->parse; + bool input_ukey_added = false; + ListCell *lc; + + if (relation_is_onerow(inputrel)) + { + add_uniquekey_for_onerow(grouprel); + return; + } + if (parse->groupingSets) + return; + + /* A Normal group by without grouping set. */ + if (parse->groupClause) + { + /* + * Current even the groupby clause is Unique already, but if query has aggref + * We have to create grouprel still. To keep the UnqiueKey short, we will check + * the UniqueKey of input_rel still valid, if so we reuse it. + */ + foreach(lc, inputrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (list_is_subset(ukey->exprs, grouprel->reltarget->exprs)) + { + grouprel->uniquekeys = lappend(grouprel->uniquekeys, + ukey); + input_ukey_added = true; + } + } + if (!input_ukey_added) + /* + * group by clause must be a super-set of grouprel->reltarget->exprs except the + * aggregation expr, so if such exprs is unique already, no bother to generate + * new uniquekey for group by exprs. + */ + add_uniquekey_from_sortgroups(root, + grouprel, + root->parse->groupClause); + } + else + /* It has aggregation but without a group by, so only one row returned */ + add_uniquekey_for_onerow(grouprel); +} + +/* + * simple_copy_uniquekeys + * Using a function for the one-line code makes us easy to check where we simply + * copied the uniquekey. + */ +void +simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel) +{ + newrel->uniquekeys = oldrel->uniquekeys; +} + +/* + * populate_unionrel_uniquekeys + */ +void +populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel) +{ + ListCell *lc; + List *exprs = NIL; + + Assert(unionrel->uniquekeys == NIL); + + foreach(lc, unionrel->reltarget->exprs) + { + exprs = lappend(exprs, lfirst(lc)); + } + + if (exprs == NIL) + /* SQL: select union select; is valid, we need to handle it here. */ + add_uniquekey_for_onerow(unionrel); + else + unionrel->uniquekeys = lappend(unionrel->uniquekeys, + makeUniqueKey(exprs,false)); + +} + +/* + * populate_joinrel_uniquekeys + * + * populate uniquekeys for joinrel. We will check each relation to see if its + * UniqueKey is still valid via innerrel_keeps_unique, if so, we add it to + * joinrel. The multi_nullvals field will be changed to true for some outer + * join cases and one-row UniqueKey needs to be converted to normal UniqueKey + * for the same case as well. + * For the uniquekey in either baserel which can't be unique after join, we still + * check to see if combination of UniqueKeys from both side is still useful for us. + * if yes, we add it to joinrel as well. + */ +void +populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *outerrel, RelOptInfo *innerrel, + List *restrictlist, JoinType jointype) +{ + ListCell *lc, *lc2; + List *clause_list = NIL; + List *outerrel_ukey_ctx; + List *innerrel_ukey_ctx; + bool inner_onerow, outer_onerow; + bool mergejoin_allowed; + + /* Care about the outerrel relation only for SEMI/ANTI join */ + if (jointype == JOIN_SEMI || jointype == JOIN_ANTI) + { + foreach(lc, outerrel->uniquekeys) + { + UniqueKey *uniquekey = lfirst_node(UniqueKey, lc); + if (list_is_subset(uniquekey->exprs, joinrel->reltarget->exprs)) + joinrel->uniquekeys = lappend(joinrel->uniquekeys, uniquekey); + } + return; + } + + Assert(jointype == JOIN_LEFT || jointype == JOIN_FULL || jointype == JOIN_INNER); + + /* Fast path */ + if (innerrel->uniquekeys == NIL || outerrel->uniquekeys == NIL) + return; + + inner_onerow = relation_is_onerow(innerrel); + outer_onerow = relation_is_onerow(outerrel); + + outerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(outerrel); + innerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(innerrel); + + clause_list = select_mergejoin_clauses(root, joinrel, outerrel, innerrel, + restrictlist, jointype, + &mergejoin_allowed); + + if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true /* reverse */)) + { + bool outer_impact = jointype == JOIN_FULL; + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + /* Outer relation has one row, and the unique key is not duplicated after join, + * the joinrel will still has one row unless the jointype == JOIN_FULL. + */ + if (outer_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (outer_onerow) + { + /* + * The onerow outerrel becomes multi rows and multi_nullvals + * will be changed to true. We also need to set the exprs correctly since it + * can't be NIL any more. + */ + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, outerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Change multi_nullvals to true due to the full join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, true)); + else + /* Just reuse it */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + } + ctx->added_to_joinrel = true; + } + } + + if (innerrel_keeps_unique(root, outerrel, innerrel, clause_list, false)) + { + bool outer_impact = jointype == JOIN_FULL || jointype == JOIN_LEFT;; + + foreach(lc, innerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + if (inner_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (inner_onerow) + { + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, innerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Need to change multi_nullvals to true due to the outer join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, + true)); + else + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + + } + ctx->added_to_joinrel = true; + } + } + + /* + * The combination of the UniqueKey from both sides is unique as well regardless + * of join type, but no bother to add it if its subset has been added to joinrel + * already or it is not useful for the joinrel. + */ + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx1 = (UniqueKeyContext) lfirst(lc); + if (ctx1->added_to_joinrel || !ctx1->useful) + continue; + foreach(lc2, innerrel_ukey_ctx) + { + UniqueKeyContext ctx2 = (UniqueKeyContext) lfirst(lc2); + if (ctx2->added_to_joinrel || !ctx2->useful) + continue; + if (add_combined_uniquekey(root, joinrel, outerrel, innerrel, + ctx1->uniquekey, ctx2->uniquekey, + jointype)) + /* If we set a onerow UniqueKey to joinrel, we don't need other. */ + return; + } + } +} + + +/* + * convert_subquery_uniquekeys + * + * Covert the UniqueKey in subquery to outer relation. + */ +void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel) +{ + ListCell *lc; + + if (sub_final_rel->uniquekeys == NIL) + return; + + if (relation_is_onerow(sub_final_rel)) + { + add_uniquekey_for_onerow(currel); + return; + } + + Assert(currel->subroot != NULL); + + foreach(lc, sub_final_rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + ListCell *lc; + List *exprs = NIL; + bool ukey_useful = true; + + /* One row case is handled above */ + Assert(ukey->exprs != NIL); + foreach(lc, ukey->exprs) + { + Var *var; + TargetEntry *tle = tlist_member(lfirst(lc), + currel->subroot->processed_tlist); + if (tle == NULL) + { + ukey_useful = false; + break; + } + var = find_var_for_subquery_tle(currel, tle); + if (var == NULL) + { + ukey_useful = false; + break; + } + exprs = lappend(exprs, var); + } + + if (ukey_useful) + currel->uniquekeys = lappend(currel->uniquekeys, + makeUniqueKey(exprs, + ukey->multi_nullvals)); + + } +} + +/* + * innerrel_keeps_unique + * + * Check if Unique key of the innerrel is valid after join. innerrel's UniqueKey + * will be still valid if innerrel's any-column mergeop outrerel's uniquekey + * exists in clause_list. + * + * Note: the clause_list must be a list of mergeable restrictinfo already. + */ +static bool +innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *clause_list, + bool reverse) +{ + ListCell *lc, *lc2, *lc3; + + if (outerrel->uniquekeys == NIL || innerrel->uniquekeys == NIL) + return false; + + /* Check if there is outerrel's uniquekey in mergeable clause. */ + foreach(lc, outerrel->uniquekeys) + { + List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs; + bool clauselist_matchs_all_exprs = true; + foreach(lc2, outer_uq_exprs) + { + Node *outer_uq_expr = lfirst(lc2); + bool find_uq_expr_in_clauselist = false; + foreach(lc3, clause_list) + { + RestrictInfo *rinfo = lfirst_node(RestrictInfo, lc3); + Node *outer_expr; + if (reverse) + outer_expr = rinfo->outer_is_left ? get_rightop(rinfo->clause) : get_leftop(rinfo->clause); + else + outer_expr = rinfo->outer_is_left ? get_leftop(rinfo->clause) : get_rightop(rinfo->clause); + if (equal(outer_expr, outer_uq_expr)) + { + find_uq_expr_in_clauselist = true; + break; + } + } + if (!find_uq_expr_in_clauselist) + { + /* No need to check the next exprs in the current uniquekey */ + clauselist_matchs_all_exprs = false; + break; + } + } + + if (clauselist_matchs_all_exprs) + return true; + } + return false; +} + + +/* + * relation_is_onerow + * Check if it is a one-row relation by checking UniqueKey. + */ +bool +relation_is_onerow(RelOptInfo *rel) +{ + UniqueKey *ukey; + if (rel->uniquekeys == NIL) + return false; + ukey = linitial_node(UniqueKey, rel->uniquekeys); + return ukey->exprs == NIL && list_length(rel->uniquekeys) == 1; +} + +/* + * relation_has_uniquekeys_for + * Returns true if we have proofs that 'rel' cannot return multiple rows with + * the same values in each of 'exprs'. Otherwise returns false. + */ +bool +relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel, + List *exprs, bool allow_multinulls) +{ + ListCell *lc; + + /* + * For UniqueKey->onerow case, the uniquekey->exprs is empty as well + * so we can't rely on list_is_subset to handle this special cases + */ + if (exprs == NIL) + return false; + + foreach(lc, rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (ukey->multi_nullvals && !allow_multinulls) + continue; + if (list_is_subset(ukey->exprs, exprs)) + return true; + } + return false; +} + + +/* + * get_exprs_from_uniqueindex + * + * Return a list of exprs which is unique. set useful to false if this + * unique index is not useful for us. + */ +static List * +get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals) +{ + List *exprs = NIL; + ListCell *indexpr_item; + int c = 0; + + *useful = true; + *multi_nullvals = false; + + indexpr_item = list_head(unique_index->indexprs); + for(c = 0; c < unique_index->ncolumns; c++) + { + int attr = unique_index->indexkeys[c]; + Expr *expr; + bool matched_const = false; + ListCell *lc1, *lc2; + + if(attr > 0) + { + expr = list_nth_node(TargetEntry, unique_index->indextlist, c)->expr; + } + else if (attr == 0) + { + /* Expression index */ + expr = lfirst(indexpr_item); + indexpr_item = lnext(unique_index->indexprs, indexpr_item); + } + else /* attr < 0 */ + { + /* Index on system column is not supported */ + Assert(false); + } + + /* + * Check index_col = Const case with regarding to opfamily checking + * If we can remove the index_col from the final UniqueKey->exprs. + */ + forboth(lc1, const_exprs, lc2, const_expr_opfamilies) + { + if (list_member_oid((List *)lfirst(lc2), unique_index->opfamily[c]) + && match_index_to_operand((Node *) lfirst(lc1), c, unique_index)) + { + matched_const = true; + break; + } + } + + if (matched_const) + continue; + + /* Check if the indexed expr is used in rel */ + if (attr > 0) + { + /* + * Normal Indexed column, if the col is not used, then the index is useless + * for uniquekey. + */ + attr -= FirstLowInvalidHeapAttributeNumber; + + if (!bms_is_member(attr, used_varattrs)) + { + *useful = false; + break; + } + } + else if (!list_member(unique_index->rel->reltarget->exprs, expr)) + { + /* Expression index but the expression is not used in rel */ + *useful = false; + break; + } + + /* check not null property. */ + if (attr == 0) + { + /* We never know if a expression yields null or not */ + *multi_nullvals = true; + } + else if (!bms_is_member(attr, unique_index->rel->notnullattrs) + && !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber, + unique_index->rel->notnullattrs)) + { + *multi_nullvals = true; + } + + exprs = lappend(exprs, expr); + } + return exprs; +} + + +/* + * add_uniquekey_for_onerow + * If we are sure that the relation only returns one row, then all the columns + * are unique. However we don't need to create UniqueKey for every column, we + * just set exprs = NIL and overwrites all the other UniqueKey on this RelOptInfo + * since this one has strongest semantics. + */ +void +add_uniquekey_for_onerow(RelOptInfo *rel) +{ + /* + * We overwrite the previous UniqueKey on purpose since this one has the + * strongest semantic. + */ + rel->uniquekeys = list_make1(makeUniqueKey(NIL, false)); +} + + +/* + * initililze_uniquecontext_for_joinrel + * Return a List of UniqueKeyContext for an inputrel + */ +static List * +initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel) +{ + List *res = NIL; + ListCell *lc; + foreach(lc, inputrel->uniquekeys) + { + UniqueKeyContext context; + context = palloc(sizeof(struct UniqueKeyContextData)); + context->uniquekey = lfirst_node(UniqueKey, lc); + context->added_to_joinrel = false; + context->useful = true; + res = lappend(res, context); + } + return res; +} + + +/* + * get_exprs_from_uniquekey + * Unify the way of get List of exprs from a one-row UniqueKey or + * normal UniqueKey. for the onerow case, every expr in rel1 is a valid + * UniqueKey. Return a List of exprs. + * + * rel1: The relation which you want to get the exprs. + * ukey: The UniqueKey you want to get the exprs. + */ +static List * +get_exprs_from_uniquekey(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *rel1, UniqueKey *ukey) +{ + ListCell *lc; + bool onerow = rel1 != NULL && relation_is_onerow(rel1); + + List *res = NIL; + Assert(onerow || ukey); + if (onerow) + { + /* Only cares about the exprs still exist in joinrel */ + foreach(lc, joinrel->reltarget->exprs) + { + Bitmapset *relids = pull_varnos(root, lfirst(lc)); + if (bms_is_subset(relids, rel1->relids)) + { + res = lappend(res, list_make1(lfirst(lc))); + } + } + } + else + { + res = list_make1(ukey->exprs); + } + return res; +} + +/* + * Partitioned table Unique Keys. + * The partition table unique key is maintained as: + * 1. The index must be unique as usual. + * 2. The index must contains partition key. + * 3. The index must exist on all the child rel. see simple_indexinfo_equal for + * how we compare it. + */ + +/* + * index_constains_partkey + * return true if the index contains the partiton key. + */ +static bool +index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind) +{ + ListCell *lc; + int i; + Assert(IS_PARTITIONED_REL(partrel)); + Assert(partrel->part_scheme->partnatts > 0); + + for(i = 0; i < partrel->part_scheme->partnatts; i++) + { + Node *part_expr = linitial(partrel->partexprs[i]); + bool found_in_index = false; + foreach(lc, ind->indextlist) + { + Expr *index_expr = lfirst_node(TargetEntry, lc)->expr; + if (equal(index_expr, part_expr)) + { + found_in_index = true; + break; + } + } + if (!found_in_index) + return false; + } + return true; +} + +/* + * simple_indexinfo_equal + * + * Used to check if the 2 index is same as each other. The index here + * is COPIED from childrel and did some tiny changes(see + * simple_copy_indexinfo_to_parent) + */ +static bool +simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2) +{ + Size oid_cmp_len = sizeof(Oid) * ind1->ncolumns; + + return ind1->ncolumns == ind2->ncolumns && + ind1->unique == ind2->unique && + memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0 && + memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0 && + memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0 && + memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0 && + equal(get_tlist_exprs(ind1->indextlist, true), + get_tlist_exprs(ind2->indextlist, true)); +} + + +/* + * The below macros are used for simple_copy_indexinfo_to_parent which is so + * customized that I don't want to put it to copyfuncs.c. So copy it here. + */ +#define COPY_POINTER_FIELD(fldname, sz) \ + do { \ + Size _size = (sz); \ + newnode->fldname = palloc(_size); \ + memcpy(newnode->fldname, from->fldname, _size); \ + } while (0) + +#define COPY_NODE_FIELD(fldname) \ + (newnode->fldname = copyObjectImpl(from->fldname)) + +#define COPY_SCALAR_FIELD(fldname) \ + (newnode->fldname = from->fldname) + + +/* + * simple_copy_indexinfo_to_parent (from partition) + * Copy the IndexInfo from child relation to parent relation with some modification, + * which is used to test: + * 1. If the same index exists in all the childrels. + * 2. If the parentrel->reltarget/basicrestrict info matches this index. + */ +static IndexOptInfo * +simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from) +{ + IndexOptInfo *newnode = makeNode(IndexOptInfo); + AppendRelInfo *appinfo = find_appinfo_by_child(root, from->rel->relid); + ListCell *lc; + int idx = 0; + + COPY_SCALAR_FIELD(ncolumns); + COPY_SCALAR_FIELD(nkeycolumns); + COPY_SCALAR_FIELD(unique); + COPY_SCALAR_FIELD(immediate); + /* We just need to know if it is NIL or not */ + COPY_SCALAR_FIELD(indpred); + COPY_SCALAR_FIELD(predOK); + COPY_POINTER_FIELD(indexkeys, from->ncolumns * sizeof(int)); + COPY_POINTER_FIELD(indexcollations, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opfamily, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opcintype, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(sortopfamily, from->ncolumns * sizeof(Oid)); + COPY_NODE_FIELD(indextlist); + + /* Convert index exprs on child expr to expr on parent */ + foreach(lc, newnode->indextlist) + { + TargetEntry *tle = lfirst_node(TargetEntry, lc); + /* Index on expression is ignored */ + Assert(IsA(tle->expr, Var)); + tle->expr = (Expr *) find_parent_var(appinfo, (Var *) tle->expr); + newnode->indexkeys[idx] = castNode(Var, tle->expr)->varattno; + idx++; + } + newnode->rel = parentrel; + return newnode; +} + +/* + * adjust_partition_unique_indexlist + * + * global_unique_indexes: At the beginning, it contains the copy & modified + * unique index from the first partition. And then check if each index in it still + * exists in the following partitions. If no, remove it. at last, it has an + * index list which exists in all the partitions. + */ +static void +adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_indexes) +{ + ListCell *lc, *lc2; + foreach(lc, *global_unique_indexes) + { + IndexOptInfo *g_ind = lfirst_node(IndexOptInfo, lc); + bool found_in_child = false; + + foreach(lc2, childrel->indexlist) + { + IndexOptInfo *p_ind = lfirst_node(IndexOptInfo, lc2); + IndexOptInfo *p_ind_copy; + if (!p_ind->unique || !p_ind->immediate || + (p_ind->indpred != NIL && !p_ind->predOK)) + continue; + p_ind_copy = simple_copy_indexinfo_to_parent(root, parentrel, p_ind); + if (simple_indexinfo_equal(p_ind_copy, g_ind)) + { + found_in_child = true; + break; + } + } + if (!found_in_child) + /* The index doesn't exist in childrel, remove it from global_unique_indexes */ + *global_unique_indexes = foreach_delete_current(*global_unique_indexes, lc); + } +} + +/* Helper function for groupres/distinctrel */ +static void +add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgroups) +{ + Query *parse = root->parse; + List *exprs; + + /* + * XXX: If there are some vars which is not in current levelsup, the semantic is + * imprecise, should we avoid it or not? levelsup = 1 is just a demo, maybe we need to + * check every level other than 0, if so, looks we have to write another + * pull_var_walker. + */ + List *upper_vars = pull_vars_of_level((Node*)sortgroups, 1); + + if (upper_vars != NIL) + return; + + exprs = get_sortgrouplist_exprs(sortgroups, parse->targetList); + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(exprs, + false /* sortgroupclause can't be multi_nullvals */)); +} + + +/* + * add_combined_uniquekey + * The combination of both UniqueKeys is a valid UniqueKey for joinrel no matter + * the jointype. + */ +bool +add_combined_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype) +{ + + ListCell *lc1, *lc2; + + /* Either side has multi_nullvals or we have outer join, + * the combined UniqueKey has multi_nullvals */ + bool multi_nullvals = outer_ukey->multi_nullvals || + inner_ukey->multi_nullvals || IS_OUTER_JOIN(jointype); + + /* The only case we can get onerow joinrel after join */ + if (relation_is_onerow(outer_rel) + && relation_is_onerow(inner_rel) + && jointype == JOIN_INNER) + { + add_uniquekey_for_onerow(joinrel); + return true; + } + + foreach(lc1, get_exprs_from_uniquekey(root, joinrel, outer_rel, outer_ukey)) + { + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, inner_rel, inner_ukey)) + { + List *exprs = list_concat_copy(lfirst_node(List, lc1), lfirst_node(List, lc2)); + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(exprs, + multi_nullvals)); + } + } + return false; +} diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index 424d25cbd5..8d8e493f5c 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -2387,6 +2387,8 @@ grouping_planner(PlannerInfo *root, bool inheritance_update, add_path(final_rel, path); } + simple_copy_uniquekeys(current_rel, final_rel); + /* * Generate partial paths for final_rel, too, if outer query levels might * be able to make use of them. @@ -3899,6 +3901,8 @@ create_grouping_paths(PlannerInfo *root, } set_cheapest(grouped_rel); + + populate_grouprel_uniquekeys(root, grouped_rel, input_rel); return grouped_rel; } @@ -4621,7 +4625,7 @@ create_window_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(window_rel); - + simple_copy_uniquekeys(input_rel, window_rel); return window_rel; } @@ -4935,7 +4939,7 @@ create_distinct_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(distinct_rel); - + populate_distinctrel_uniquekeys(root, input_rel, distinct_rel); return distinct_rel; } @@ -5196,6 +5200,8 @@ create_ordered_paths(PlannerInfo *root, */ Assert(ordered_rel->pathlist != NIL); + simple_copy_uniquekeys(input_rel, ordered_rel); + return ordered_rel; } @@ -6073,6 +6079,9 @@ adjust_paths_for_srfs(PlannerInfo *root, RelOptInfo *rel, if (list_length(targets) == 1) return; + /* UniqueKey is not valid after handling the SRF. */ + rel->uniquekeys = NIL; + /* * Stack SRF-evaluation nodes atop each path for the rel. * diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c index becdcbb872..b7626545bf 100644 --- a/src/backend/optimizer/prep/prepunion.c +++ b/src/backend/optimizer/prep/prepunion.c @@ -689,6 +689,8 @@ generate_union_paths(SetOperationStmt *op, PlannerInfo *root, /* Undo effects of possibly forcing tuple_fraction to 0 */ root->tuple_fraction = save_fraction; + /* Add the UniqueKeys */ + populate_unionrel_uniquekeys(root, result_rel); return result_rel; } diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c index 86922a273c..6817c9c787 100644 --- a/src/backend/optimizer/util/appendinfo.c +++ b/src/backend/optimizer/util/appendinfo.c @@ -746,3 +746,47 @@ find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos) } return appinfos; } + +/* + * find_appinfo_by_child + * + */ +AppendRelInfo * +find_appinfo_by_child(PlannerInfo *root, Index child_index) +{ + ListCell *lc; + foreach(lc, root->append_rel_list) + { + AppendRelInfo *appinfo = lfirst_node(AppendRelInfo, lc); + if (appinfo->child_relid == child_index) + return appinfo; + } + elog(ERROR, "parent relation cant be found"); + return NULL; +} + +/* + * find_parent_var + * + */ +Var * +find_parent_var(AppendRelInfo *appinfo, Var *child_var) +{ + ListCell *lc; + Var *res = NULL; + Index attno = 1; + foreach(lc, appinfo->translated_vars) + { + Node *child_node = lfirst(lc); + if (equal(child_node, child_var)) + { + res = copyObject(child_var); + res->varattno = attno; + res->varno = appinfo->parent_relid; + } + attno++; + } + if (res == NULL) + elog(ERROR, "parent var can't be found."); + return res; +} diff --git a/src/backend/optimizer/util/inherit.c b/src/backend/optimizer/util/inherit.c index be1c9ddd96..3eec1f4d74 100644 --- a/src/backend/optimizer/util/inherit.c +++ b/src/backend/optimizer/util/inherit.c @@ -736,6 +736,7 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, { Node *onecq = (Node *) lfirst(lc2); bool pseudoconstant; + RestrictInfo *child_rinfo; /* check for pseudoconstant (no Vars or volatile functions) */ pseudoconstant = @@ -747,14 +748,15 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, root->hasPseudoConstantQuals = true; } /* reconstitute RestrictInfo with appropriate properties */ - childquals = lappend(childquals, - make_restrictinfo(root, - (Expr *) onecq, - rinfo->is_pushed_down, - rinfo->outerjoin_delayed, - pseudoconstant, - rinfo->security_level, - NULL, NULL, NULL)); + child_rinfo = make_restrictinfo(root, + (Expr *) onecq, + rinfo->is_pushed_down, + rinfo->outerjoin_delayed, + pseudoconstant, + rinfo->security_level, + NULL, NULL, NULL); + child_rinfo->mergeopfamilies = rinfo->mergeopfamilies; + childquals = lappend(childquals, child_rinfo); /* track minimum security level among child quals */ cq_min_security = Min(cq_min_security, rinfo->security_level); } diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 48a7ebfe45..4fe1824eb0 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -16,6 +16,7 @@ #include "nodes/execnodes.h" #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" extern A_Expr *makeA_Expr(A_Expr_Kind kind, List *name, @@ -106,4 +107,6 @@ extern GroupingSet *makeGroupingSet(GroupingSetKind kind, List *content, int loc extern VacuumRelation *makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols); +extern UniqueKey* makeUniqueKey(List *exprs, bool multi_nullvals); + #endif /* MAKEFUNC_H */ diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index e22df890ef..3320273ac1 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -265,6 +265,7 @@ typedef enum NodeTag T_EquivalenceMember, T_PathKey, T_PathTarget, + T_UniqueKey, T_RestrictInfo, T_IndexClause, T_PlaceHolderVar, diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index 0d61f04d27..80561b1003 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -739,6 +739,7 @@ typedef struct RelOptInfo QualCost baserestrictcost; /* cost of evaluating the above */ Index baserestrict_min_security; /* min security_level found in * baserestrictinfo */ + List *uniquekeys; /* List of UniqueKey */ List *joininfo; /* RestrictInfo structures for join clauses * involving this rel */ bool has_eclass_joins; /* T means joininfo is incomplete */ @@ -1059,6 +1060,28 @@ typedef struct PathKey } PathKey; +/* + * UniqueKey + * + * Represents the unique properties held by a RelOptInfo. + * + * exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL + * is a special case of UniqueKey, which means there is only 1 row in that + * relation. + * multi_nullvals: true means multi null values may exist in these exprs, so the + * uniqueness is not guaranteed in this case. This field is necessary for + * remove_useless_join & reduce_unique_semijoins where we don't mind these + * duplicated NULL values. It is set to true for 2 cases. One is a unique key + * from a unique index but the related column is nullable. The other one is for + * outer join. see populate_joinrel_uniquekeys for detail. + */ +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + /* * PathTarget * @@ -2496,7 +2519,7 @@ typedef enum * * flags indicating what kinds of grouping are possible. * partial_costs_set is true if the agg_partial_costs and agg_final_costs - * have been initialized. + * have been initialized. * agg_partial_costs gives partial aggregation costs. * agg_final_costs gives finalization costs. * target_parallel_safe is true if target is parallel safe. @@ -2526,8 +2549,8 @@ typedef struct * limit_tuples is an estimated bound on the number of output tuples, * or -1 if no LIMIT or couldn't estimate. * count_est and offset_est are the estimated values of the LIMIT and OFFSET - * expressions computed by preprocess_limit() (see comments for - * preprocess_limit() for more information). + * expressions computed by preprocess_limit() (see comments for + * preprocess_limit() for more information). */ typedef struct { diff --git a/src/include/nodes/pg_list.h b/src/include/nodes/pg_list.h index 404e03f132..85c65e910c 100644 --- a/src/include/nodes/pg_list.h +++ b/src/include/nodes/pg_list.h @@ -566,6 +566,16 @@ extern pg_nodiscard List *list_delete_first(List *list); extern pg_nodiscard List *list_delete_last(List *list); extern pg_nodiscard List *list_delete_nth_cell(List *list, int n); extern pg_nodiscard List *list_delete_cell(List *list, ListCell *cell); +extern bool list_is_subset(const List *members, const List *target); + +extern List *list_delete(List *list, void *datum); +extern List *list_delete_ptr(List *list, void *datum); +extern List *list_delete_int(List *list, int datum); +extern List *list_delete_oid(List *list, Oid datum); +extern List *list_delete_first(List *list); +extern List *list_delete_last(List *list); +extern List *list_delete_nth_cell(List *list, int n); +extern List *list_delete_cell(List *list, ListCell *cell); extern List *list_union(const List *list1, const List *list2); extern List *list_union_ptr(const List *list1, const List *list2); diff --git a/src/include/optimizer/appendinfo.h b/src/include/optimizer/appendinfo.h index 4cbf8c26cc..43b8d05f69 100644 --- a/src/include/optimizer/appendinfo.h +++ b/src/include/optimizer/appendinfo.h @@ -32,4 +32,7 @@ extern Relids adjust_child_relids_multilevel(PlannerInfo *root, Relids relids, extern AppendRelInfo **find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos); +extern AppendRelInfo *find_appinfo_by_child(PlannerInfo *root, Index child_index); +extern Var *find_parent_var(AppendRelInfo *appinfo, Var *child_var); + #endif /* APPENDINFO_H */ diff --git a/src/include/optimizer/optimizer.h b/src/include/optimizer/optimizer.h index d587952b7d..843aafa51e 100644 --- a/src/include/optimizer/optimizer.h +++ b/src/include/optimizer/optimizer.h @@ -23,6 +23,7 @@ #define OPTIMIZER_H #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" /* Test if an expression node represents a SRF call. Beware multiple eval! */ #define IS_SRF_CALL(node) \ @@ -168,6 +169,7 @@ extern TargetEntry *get_sortgroupref_tle(Index sortref, List *targetList); extern TargetEntry *get_sortgroupclause_tle(SortGroupClause *sgClause, List *targetList); +extern Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); extern Node *get_sortgroupclause_expr(SortGroupClause *sgClause, List *targetList); extern List *get_sortgrouplist_exprs(List *sgClauses, diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h index 035d3e1206..1adf99c9ee 100644 --- a/src/include/optimizer/paths.h +++ b/src/include/optimizer/paths.h @@ -247,5 +247,48 @@ extern PathKey *make_canonical_pathkey(PlannerInfo *root, int strategy, bool nulls_first); extern void add_paths_to_append_rel(PlannerInfo *root, RelOptInfo *rel, List *live_childrels); +extern List *select_mergejoin_clauses(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + JoinType jointype, + bool *mergejoin_allowed); + +/* + * uniquekeys.c + * Utilities for matching and building unique keys + */ +extern void populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List* unique_index_list); +extern void populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels); +extern void populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel); +extern void populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel); +extern void populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel); +extern void simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel); +extern void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel); +extern void populate_joinrel_uniquekeys(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + RelOptInfo *rel2, + List *restrictlist, + JoinType jointype); + +extern bool relation_has_uniquekeys_for(PlannerInfo *root, + RelOptInfo *rel, + List *exprs, + bool allow_multinulls); +extern bool relation_is_onerow(RelOptInfo *rel); #endif /* PATHS_H */ -- 2.26.2 --yh7cw345mbttadjz Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v38-0003-Extend-UniqueKeys.patch" ^ permalink raw reply [nested|flat] 91+ messages in thread
* [PATCH v36 2/6] Introduce UniqueKey attributes on RelOptInfo struct. @ 2020-05-11 07:50 一挃 <[email protected]> 0 siblings, 0 replies; 91+ messages in thread From: 一挃 @ 2020-05-11 07:50 UTC (permalink / raw) UniqueKey is a set of exprs on RelOptInfo which represents the exprs will be unique on the given RelOptInfo. You can see README.uniquekey for more information. --- src/backend/nodes/copyfuncs.c | 13 + src/backend/nodes/list.c | 31 + src/backend/nodes/makefuncs.c | 13 + src/backend/nodes/outfuncs.c | 11 + src/backend/nodes/readfuncs.c | 10 + src/backend/optimizer/path/Makefile | 3 +- src/backend/optimizer/path/README.uniquekey | 131 +++ src/backend/optimizer/path/allpaths.c | 10 + src/backend/optimizer/path/joinpath.c | 9 +- src/backend/optimizer/path/joinrels.c | 2 + src/backend/optimizer/path/pathkeys.c | 3 +- src/backend/optimizer/path/uniquekeys.c | 1131 +++++++++++++++++++ src/backend/optimizer/plan/planner.c | 13 +- src/backend/optimizer/prep/prepunion.c | 2 + src/backend/optimizer/util/appendinfo.c | 44 + src/backend/optimizer/util/inherit.c | 16 +- src/include/nodes/makefuncs.h | 3 + src/include/nodes/nodes.h | 1 + src/include/nodes/pathnodes.h | 29 +- src/include/nodes/pg_list.h | 2 + src/include/optimizer/appendinfo.h | 3 + src/include/optimizer/optimizer.h | 2 + src/include/optimizer/paths.h | 43 + 23 files changed, 1502 insertions(+), 23 deletions(-) create mode 100644 src/backend/optimizer/path/README.uniquekey create mode 100644 src/backend/optimizer/path/uniquekeys.c diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 89c409de66..1f50400fd2 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2273,6 +2273,16 @@ _copyPathKey(const PathKey *from) return newnode; } +static UniqueKey * +_copyUniqueKey(const UniqueKey *from) +{ + UniqueKey *newnode = makeNode(UniqueKey); + + COPY_NODE_FIELD(exprs); + COPY_SCALAR_FIELD(multi_nullvals); + + return newnode; +} /* * _copyRestrictInfo */ @@ -5152,6 +5162,9 @@ copyObjectImpl(const void *from) case T_PathKey: retval = _copyPathKey(from); break; + case T_UniqueKey: + retval = _copyUniqueKey(from); + break; case T_RestrictInfo: retval = _copyRestrictInfo(from); break; diff --git a/src/backend/nodes/list.c b/src/backend/nodes/list.c index 80fa8c84e4..a7a99b70f2 100644 --- a/src/backend/nodes/list.c +++ b/src/backend/nodes/list.c @@ -687,6 +687,37 @@ list_member_oid(const List *list, Oid datum) return false; } +/* + * return true iff every entry in "members" list is also present + * in the "target" list. + */ +bool +list_is_subset(const List *members, const List *target) +{ + const ListCell *lc1, *lc2; + + Assert(IsPointerList(members)); + Assert(IsPointerList(target)); + check_list_invariants(members); + check_list_invariants(target); + + foreach(lc1, members) + { + bool found = false; + foreach(lc2, target) + { + if (equal(lfirst(lc1), lfirst(lc2))) + { + found = true; + break; + } + } + if (!found) + return false; + } + return true; +} + /* * Delete the n'th cell (counting from 0) in list. * diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 49de285f01..646cf7c9a1 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -814,3 +814,16 @@ makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols) v->va_cols = va_cols; return v; } + + +/* + * makeUniqueKey + */ +UniqueKey* +makeUniqueKey(List *exprs, bool multi_nullvals) +{ + UniqueKey * ukey = makeNode(UniqueKey); + ukey->exprs = exprs; + ukey->multi_nullvals = multi_nullvals; + return ukey; +} diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index e2f177515d..c3a9632992 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2428,6 +2428,14 @@ _outPathKey(StringInfo str, const PathKey *node) WRITE_BOOL_FIELD(pk_nulls_first); } +static void +_outUniqueKey(StringInfo str, const UniqueKey *node) +{ + WRITE_NODE_TYPE("UNIQUEKEY"); + WRITE_NODE_FIELD(exprs); + WRITE_BOOL_FIELD(multi_nullvals); +} + static void _outPathTarget(StringInfo str, const PathTarget *node) { @@ -4127,6 +4135,9 @@ outNode(StringInfo str, const void *obj) case T_PathKey: _outPathKey(str, obj); break; + case T_UniqueKey: + _outUniqueKey(str, obj); + break; case T_PathTarget: _outPathTarget(str, obj); break; diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index 42050ab719..3a18571d0c 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -452,6 +452,14 @@ _readSetOperationStmt(void) READ_DONE(); } +static UniqueKey * +_readUniqueKey(void) +{ + READ_LOCALS(UniqueKey); + READ_NODE_FIELD(exprs); + READ_BOOL_FIELD(multi_nullvals); + READ_DONE(); +} /* * Stuff from primnodes.h. @@ -2656,6 +2664,8 @@ parseNodeString(void) return_value = _readCommonTableExpr(); else if (MATCH("SETOPERATIONSTMT", 16)) return_value = _readSetOperationStmt(); + else if (MATCH("UNIQUEKEY", 9)) + return_value = _readUniqueKey(); else if (MATCH("ALIAS", 5)) return_value = _readAlias(); else if (MATCH("RANGEVAR", 8)) diff --git a/src/backend/optimizer/path/Makefile b/src/backend/optimizer/path/Makefile index 1e199ff66f..7b9820c25f 100644 --- a/src/backend/optimizer/path/Makefile +++ b/src/backend/optimizer/path/Makefile @@ -21,6 +21,7 @@ OBJS = \ joinpath.o \ joinrels.o \ pathkeys.o \ - tidpath.o + tidpath.o \ + uniquekeys.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/optimizer/path/README.uniquekey b/src/backend/optimizer/path/README.uniquekey new file mode 100644 index 0000000000..5eac761995 --- /dev/null +++ b/src/backend/optimizer/path/README.uniquekey @@ -0,0 +1,131 @@ +1. What is UniqueKey? +We can think UniqueKey is a set of exprs for a RelOptInfo, which we are insure +that doesn't yields same result among all the rows. The simplest UniqueKey +format is primary key. + +However we define the UnqiueKey as below. + +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + +exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL +is a special case of UniqueKey, which means there is only one row in that +relation.it has a stronger semantic than others. like SELECT uk FROM t; uk is +normal unique key and may have different values. SELECT colx FROM t WHERE uk = +const. colx is unique AND we have only 1 value. This field can used for +innerrel_is_unique. this logic is handled specially in add_uniquekey_for_onerow +function. + +multi_nullvals: true means multi null values may exist in these exprs, so the +uniqueness is not guaranteed in this case. This field is necessary for +remove_useless_join & reduce_unique_semijoins where we don't mind these +duplicated NULL values. It is set to true for 2 cases. One is a unique key +from a unique index but the related column is nullable. The other one is for +outer join. see populate_joinrel_uniquekeys for detail. + + +The UniqueKey can be used at the following cases at least: +1. remove_useless_joins. +2. reduce_semianti_joins +3. remove distinct node if distinct clause is unique. +4. remove aggnode if group by clause is unique. +5. Index Skip Scan (WIP) +6. Aggregation Push Down without 2 phase aggregation if the join can't + duplicated the aggregated rows. (work in progress feature) + +2. How is it maintained? + +We have a set of populate_xxx_unqiuekeys functions to maintain the uniquekey on +various cases. xxx includes baserel, joinrel, partitionedrel, distinctrel, +groupedrel, unionrel. and we also need to convert the uniquekey from subquery +to outer relation, which is what convert_subquery_uniquekeys does. + +1. The first part is about baserel. We handled 3 cases. suppose we have Unique +Index on (a, b). + +1. SELECT a, b FROM t. UniqueKey (a, b) +2. SELECT a FROM t WHERE b = 1; UniqueKey (a) +3. SELECT .. FROM t WHERE a = 1 AND b = 1; UniqueKey (NIL). onerow case, every + column is Unique. + +2. The next part is joinrel, this part is most error-prone, we simplified the rules +like below: +1. If the relation's UniqueKey can't be duplicated after join, then is will be + still valid for the join rel. The function we used here is + innerrel_keeps_unique. The basic idea is innerrel.any_col = outer.uk. + +2. If the UnqiueKey can't keep valid via the rule 1, the combination of the + UniqueKey from both sides are valid for sure. We can prove this as: if the + unique exprs from rel1 is duplicated by rel2, the duplicated rows must + contains different unique exprs from rel2. + +More considerations about onerow: +1. If relation with one row and it can't be duplicated, it is still possible + contains mulit_nullvas after outer join. +2. If the either UniqueKey can be duplicated after join, the can get one row + only when both side is one row AND there is no outer join. +3. Whenever the onerow UniqueKey is not a valid any more, we need to convert one + row UniqueKey to normal unique key since we don't store exprs for one-row + relation. get_exprs_from_uniquekeys will be used here. + + +More considerations about multi_nullvals after join: +1. If the original UnqiueKey has multi_nullvals, the final UniqueKey will have + mulit_nullvals in any case. +2. If a unique key doesn't allow mulit_nullvals, after some outer join, it + allows some outer join. + + +3. When we comes to subquery, we need to convert_subquery_unqiuekeys just like +convert_subquery_pathkeys. Only the UniqueKey insides subquery is referenced as +a Var in outer relation will be reused. The relationship between the outerrel.Var +and subquery.exprs is built with outerel->subroot->processed_tlist. + + +4. As for the SRF functions, it will break the uniqueness of uniquekey, However it +is handled in adjust_paths_for_srfs, which happens after the query_planner. so +we will maintain the UniqueKey until there and reset it to NIL at that +places. This can't help on distinct/group by elimination cases but probably help +in some other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is +semantic correctly. + + +5. As for inherit table, we first main the UnqiueKey on childrel as well. But for +partitioned table we need to maintain 2 different kinds of +UnqiueKey. 1). UniqueKey on the parent relation 2). UniqueKey on child +relation for partition wise query. + +Example: +CREATE TABLE p (a int not null, b int not null) partition by list (a); +CREATE TABLE p0 partition of p for values in (1); +CREATE TABLE p1 partition of p for values in (2); + +create unique index p0_b on p0(b); +create unique index p1_b on p1(b); + +Now b is only unique on partition level, so the distinct can't be removed on +the following cases. SELECT DISTINCT b FROM p; + +Another example is SELECT DISTINCT a, b FROM p WHERE a = 1; Since only one +partition is chosen, the UniqueKey on child relation is same as the UniqueKey on +parent relation. + +Another usage of UniqueKey on partition level is it be helpful for +partition-wise join. + +As for the UniqueKey on parent table level, it comes with 2 different ways, +1). the UniqueKey is also derived in UniqueKey index, but the index must be same +in all the related children relations and the unique index must contains +Partition Key in it. Example: + +CREATE UNIQUE INDEX p_ab ON p(a, b); -- where a is the partition key. + +-- Query +SELECT a, b FROM p; the (a, b) is a UniqueKey of p. + +2). If the parent relation has only one childrel, the UniqueKey on childrel is + the UniqueKey on parent as well. diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index 484dab0a1a..2ad9d06d7a 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -579,6 +579,12 @@ set_plain_rel_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) */ check_index_predicates(root, rel); + /* + * Now that we've marked which partial indexes are suitable, we can now + * build the relation's unique keys. + */ + populate_baserel_uniquekeys(root, rel, rel->indexlist); + /* Mark rel with estimated output rows, width, etc */ set_baserel_size_estimates(root, rel); } @@ -1310,6 +1316,8 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, /* Add paths to the append relation. */ add_paths_to_append_rel(root, rel, live_childrels); + if (IS_PARTITIONED_REL(rel)) + populate_partitionedrel_uniquekeys(root, rel, live_childrels); } @@ -2383,6 +2391,8 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, pathkeys, required_outer)); } + convert_subquery_uniquekeys(root, rel, sub_final_rel); + /* If outer rel allows parallelism, do same for partial paths. */ if (rel->consider_parallel && bms_is_empty(required_outer)) { diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c index db54a6ba2e..ef0fd2fb0b 100644 --- a/src/backend/optimizer/path/joinpath.c +++ b/src/backend/optimizer/path/joinpath.c @@ -71,13 +71,6 @@ static void consider_parallel_mergejoin(PlannerInfo *root, static void hash_inner_and_outer(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, RelOptInfo *innerrel, JoinType jointype, JoinPathExtraData *extra); -static List *select_mergejoin_clauses(PlannerInfo *root, - RelOptInfo *joinrel, - RelOptInfo *outerrel, - RelOptInfo *innerrel, - List *restrictlist, - JoinType jointype, - bool *mergejoin_allowed); static void generate_mergejoin_paths(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *innerrel, @@ -1927,7 +1920,7 @@ hash_inner_and_outer(PlannerInfo *root, * if it is mergejoinable and involves vars from the two sub-relations * currently of interest. */ -static List * +List * select_mergejoin_clauses(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c index 2d343cd293..b9163ee8ff 100644 --- a/src/backend/optimizer/path/joinrels.c +++ b/src/backend/optimizer/path/joinrels.c @@ -924,6 +924,8 @@ populate_joinrel_with_paths(PlannerInfo *root, RelOptInfo *rel1, /* Apply partitionwise join technique, if possible. */ try_partitionwise_join(root, rel1, rel2, joinrel, sjinfo, restrictlist); + + populate_joinrel_uniquekeys(root, joinrel, rel1, rel2, restrictlist, sjinfo->jointype); } diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index ce9bf87e9b..7e596d4194 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -33,7 +33,6 @@ static bool pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys); static bool matches_boolean_partition_clause(RestrictInfo *rinfo, RelOptInfo *partrel, int partkeycol); -static Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); static bool right_merge_direction(PlannerInfo *root, PathKey *pathkey); @@ -1035,7 +1034,7 @@ convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel, * We need this to ensure that we don't return pathkeys describing values * that are unavailable above the level of the subquery scan. */ -static Var * +Var * find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle) { ListCell *lc; diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c new file mode 100644 index 0000000000..b33bcd2f32 --- /dev/null +++ b/src/backend/optimizer/path/uniquekeys.c @@ -0,0 +1,1131 @@ +/*------------------------------------------------------------------------- + * + * uniquekeys.c + * Utilities for matching and building unique keys + * + * Portions Copyright (c) 2020, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/optimizer/path/uniquekeys.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "nodes/makefuncs.h" +#include "nodes/nodeFuncs.h" +#include "optimizer/pathnode.h" +#include "optimizer/paths.h" +#include "optimizer/appendinfo.h" +#include "optimizer/optimizer.h" +#include "optimizer/tlist.h" +#include "rewrite/rewriteManip.h" + + +/* + * This struct is used to help populate_joinrel_uniquekeys. + * + * added_to_joinrel is true if a uniquekey (from outerrel or innerrel) + * has been added to joinrel. + * useful is true if the exprs of the uniquekey still appears in joinrel. + */ +typedef struct UniqueKeyContextData +{ + UniqueKey *uniquekey; + bool added_to_joinrel; + bool useful; +} *UniqueKeyContext; + +static List *initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel); +static bool innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + bool reverse); + +static List *get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals); +static List *get_exprs_from_uniquekey(RelOptInfo *joinrel, + RelOptInfo *rel1, + UniqueKey *ukey); +static void add_uniquekey_for_onerow(RelOptInfo *rel); +static bool add_combined_uniquekey(RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype); + +/* Used for unique indexes checking for partitioned table */ +static bool index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind); +static IndexOptInfo *simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from); +static bool simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2); +static void adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_index); + +/* Helper function for grouped relation and distinct relation. */ +static void add_uniquekey_from_sortgroups(PlannerInfo *root, + RelOptInfo *rel, + List *sortgroups); + +/* + * populate_baserel_uniquekeys + * Populate 'baserel' uniquekeys list by looking at the rel's unique index + * and baserestrictinfo + */ +void +populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List *indexlist) +{ + ListCell *lc; + List *matched_uniq_indexes = NIL; + + /* Attrs appears in rel->reltarget->exprs. */ + Bitmapset *used_attrs = NULL; + + List *const_exprs = NIL; + List *expr_opfamilies = NIL; + + Assert(baserel->rtekind == RTE_RELATION); + + foreach(lc, indexlist) + { + IndexOptInfo *ind = (IndexOptInfo *) lfirst(lc); + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + matched_uniq_indexes = lappend(matched_uniq_indexes, ind); + } + + if (matched_uniq_indexes == NIL) + return; + + /* Check which attrs is used in baserel->reltarget */ + pull_varattnos((Node *)baserel->reltarget->exprs, baserel->relid, &used_attrs); + + /* Check which attrno is used at a mergeable const filter */ + foreach(lc, baserel->baserestrictinfo) + { + RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); + + if (rinfo->mergeopfamilies == NIL) + continue; + + if (bms_is_empty(rinfo->left_relids)) + { + const_exprs = lappend(const_exprs, get_rightop(rinfo->clause)); + } + else if (bms_is_empty(rinfo->right_relids)) + { + const_exprs = lappend(const_exprs, get_leftop(rinfo->clause)); + } + else + continue; + + expr_opfamilies = lappend(expr_opfamilies, rinfo->mergeopfamilies); + } + + foreach(lc, matched_uniq_indexes) + { + bool multi_nullvals, useful; + List *exprs = get_exprs_from_uniqueindex(lfirst_node(IndexOptInfo, lc), + const_exprs, + expr_opfamilies, + used_attrs, + &useful, + &multi_nullvals); + if (useful) + { + if (exprs == NIL) + { + /* All the columns in Unique Index matched with a restrictinfo */ + add_uniquekey_for_onerow(baserel); + return; + } + baserel->uniquekeys = lappend(baserel->uniquekeys, + makeUniqueKey(exprs, multi_nullvals)); + } + } +} + + +/* + * populate_partitionedrel_uniquekeys + * The UniqueKey on partitionrel comes from 2 cases: + * 1). Only one partition is involved in this query, the unique key can be + * copied to parent rel from childrel. + * 2). There are some unique index which includes partition key and exists + * in all the related partitions. + * We never mind rule 2 if we hit rule 1. + */ + +void +populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels) +{ + ListCell *lc; + List *global_uniq_indexlist = NIL; + RelOptInfo *childrel; + bool is_first = true; + + Assert(IS_PARTITIONED_REL(rel)); + + if (childrels == NIL) + return; + + /* + * If there is only one partition used in this query, the UniqueKey in childrel is + * still valid in parent level, but we need convert the format from child expr to + * parent expr. + */ + if (list_length(childrels) == 1) + { + /* Check for Rule 1 */ + RelOptInfo *childrel = linitial_node(RelOptInfo, childrels); + ListCell *lc; + Assert(childrel->reloptkind == RELOPT_OTHER_MEMBER_REL); + if (relation_is_onerow(childrel)) + { + add_uniquekey_for_onerow(rel); + return; + } + + foreach(lc, childrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + AppendRelInfo *appinfo = find_appinfo_by_child(root, childrel->relid); + List *parent_exprs = NIL; + bool can_reuse = true; + ListCell *lc2; + foreach(lc2, ukey->exprs) + { + Var *var = (Var *)lfirst(lc2); + /* + * If the expr comes from a expression, it is hard to build the expression + * in parent so ignore that case for now. + */ + if(!IsA(var, Var)) + { + can_reuse = false; + break; + } + /* Convert it to parent var */ + parent_exprs = lappend(parent_exprs, find_parent_var(appinfo, var)); + } + if (can_reuse) + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(parent_exprs, + ukey->multi_nullvals)); + } + } + else + { + /* Check for rule 2 */ + childrel = linitial_node(RelOptInfo, childrels); + foreach(lc, childrel->indexlist) + { + IndexOptInfo *ind = lfirst(lc); + IndexOptInfo *modified_index; + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + + /* + * During simple_copy_indexinfo_to_parent, we need to convert var from + * child var to parent var, index on expression is too complex to handle. + * so ignore it for now. + */ + if (ind->indexprs != NIL) + continue; + + modified_index = simple_copy_indexinfo_to_parent(root, rel, ind); + /* + * If the unique index doesn't contain partkey, then it is unique + * on this partition only, so it is useless for us. + */ + if (!index_constains_partkey(rel, modified_index)) + continue; + + global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index); + } + + if (global_uniq_indexlist != NIL) + { + foreach(lc, childrels) + { + RelOptInfo *child = lfirst(lc); + if (is_first) + { + is_first = false; + continue; + } + adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist); + } + /* Now we have a list of unique index which are exactly same on all childrels, + * Set the UniqueKey just like it is non-partition table + */ + populate_baserel_uniquekeys(root, rel, global_uniq_indexlist); + } + } +} + + +/* + * populate_distinctrel_uniquekeys + */ +void +populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel) +{ + /* The unique key before the distinct is still valid. */ + distinctrel->uniquekeys = list_copy(inputrel->uniquekeys); + add_uniquekey_from_sortgroups(root, distinctrel, root->parse->distinctClause); +} + +/* + * populate_grouprel_uniquekeys + */ +void +populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel) + +{ + Query *parse = root->parse; + bool input_ukey_added = false; + ListCell *lc; + + if (relation_is_onerow(inputrel)) + { + add_uniquekey_for_onerow(grouprel); + return; + } + if (parse->groupingSets) + return; + + /* A Normal group by without grouping set. */ + if (parse->groupClause) + { + /* + * Current even the groupby clause is Unique already, but if query has aggref + * We have to create grouprel still. To keep the UnqiueKey short, we will check + * the UniqueKey of input_rel still valid, if so we reuse it. + */ + foreach(lc, inputrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (list_is_subset(ukey->exprs, grouprel->reltarget->exprs)) + { + grouprel->uniquekeys = lappend(grouprel->uniquekeys, + ukey); + input_ukey_added = true; + } + } + if (!input_ukey_added) + /* + * group by clause must be a super-set of grouprel->reltarget->exprs except the + * aggregation expr, so if such exprs is unique already, no bother to generate + * new uniquekey for group by exprs. + */ + add_uniquekey_from_sortgroups(root, + grouprel, + root->parse->groupClause); + } + else + /* It has aggregation but without a group by, so only one row returned */ + add_uniquekey_for_onerow(grouprel); +} + +/* + * simple_copy_uniquekeys + * Using a function for the one-line code makes us easy to check where we simply + * copied the uniquekey. + */ +void +simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel) +{ + newrel->uniquekeys = oldrel->uniquekeys; +} + +/* + * populate_unionrel_uniquekeys + */ +void +populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel) +{ + ListCell *lc; + List *exprs = NIL; + + Assert(unionrel->uniquekeys == NIL); + + foreach(lc, unionrel->reltarget->exprs) + { + exprs = lappend(exprs, lfirst(lc)); + } + + if (exprs == NIL) + /* SQL: select union select; is valid, we need to handle it here. */ + add_uniquekey_for_onerow(unionrel); + else + unionrel->uniquekeys = lappend(unionrel->uniquekeys, + makeUniqueKey(exprs,false)); + +} + +/* + * populate_joinrel_uniquekeys + * + * populate uniquekeys for joinrel. We will check each relation to see if its + * UniqueKey is still valid via innerrel_keeps_unique, if so, we add it to + * joinrel. The multi_nullvals field will be changed to true for some outer + * join cases and one-row UniqueKey needs to be converted to normal UniqueKey + * for the same case as well. + * For the uniquekey in either baserel which can't be unique after join, we still + * check to see if combination of UniqueKeys from both side is still useful for us. + * if yes, we add it to joinrel as well. + */ +void +populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *outerrel, RelOptInfo *innerrel, + List *restrictlist, JoinType jointype) +{ + ListCell *lc, *lc2; + List *clause_list = NIL; + List *outerrel_ukey_ctx; + List *innerrel_ukey_ctx; + bool inner_onerow, outer_onerow; + bool mergejoin_allowed; + + /* Care about the outerrel relation only for SEMI/ANTI join */ + if (jointype == JOIN_SEMI || jointype == JOIN_ANTI) + { + foreach(lc, outerrel->uniquekeys) + { + UniqueKey *uniquekey = lfirst_node(UniqueKey, lc); + if (list_is_subset(uniquekey->exprs, joinrel->reltarget->exprs)) + joinrel->uniquekeys = lappend(joinrel->uniquekeys, uniquekey); + } + return; + } + + Assert(jointype == JOIN_LEFT || jointype == JOIN_FULL || jointype == JOIN_INNER); + + /* Fast path */ + if (innerrel->uniquekeys == NIL || outerrel->uniquekeys == NIL) + return; + + inner_onerow = relation_is_onerow(innerrel); + outer_onerow = relation_is_onerow(outerrel); + + outerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(outerrel); + innerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(innerrel); + + clause_list = select_mergejoin_clauses(root, joinrel, outerrel, innerrel, + restrictlist, jointype, + &mergejoin_allowed); + + if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true /* reverse */)) + { + bool outer_impact = jointype == JOIN_FULL; + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + /* Outer relation has one row, and the unique key is not duplicated after join, + * the joinrel will still has one row unless the jointype == JOIN_FULL. + */ + if (outer_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (outer_onerow) + { + /* + * The onerow outerrel becomes multi rows and multi_nullvals + * will be changed to true. We also need to set the exprs correctly since it + * can't be NIL any more. + */ + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(joinrel, outerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Change multi_nullvals to true due to the full join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, true)); + else + /* Just reuse it */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + } + ctx->added_to_joinrel = true; + } + } + + if (innerrel_keeps_unique(root, outerrel, innerrel, clause_list, false)) + { + bool outer_impact = jointype == JOIN_FULL || jointype == JOIN_LEFT;; + + foreach(lc, innerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + if (inner_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (inner_onerow) + { + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(joinrel, innerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Need to change multi_nullvals to true due to the outer join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, + true)); + else + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + + } + ctx->added_to_joinrel = true; + } + } + + /* + * The combination of the UniqueKey from both sides is unique as well regardless + * of join type, but no bother to add it if its subset has been added to joinrel + * already or it is not useful for the joinrel. + */ + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx1 = (UniqueKeyContext) lfirst(lc); + if (ctx1->added_to_joinrel || !ctx1->useful) + continue; + foreach(lc2, innerrel_ukey_ctx) + { + UniqueKeyContext ctx2 = (UniqueKeyContext) lfirst(lc2); + if (ctx2->added_to_joinrel || !ctx2->useful) + continue; + if (add_combined_uniquekey(joinrel, outerrel, innerrel, + ctx1->uniquekey, ctx2->uniquekey, + jointype)) + /* If we set a onerow UniqueKey to joinrel, we don't need other. */ + return; + } + } +} + + +/* + * convert_subquery_uniquekeys + * + * Covert the UniqueKey in subquery to outer relation. + */ +void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel) +{ + ListCell *lc; + + if (sub_final_rel->uniquekeys == NIL) + return; + + if (relation_is_onerow(sub_final_rel)) + { + add_uniquekey_for_onerow(currel); + return; + } + + Assert(currel->subroot != NULL); + + foreach(lc, sub_final_rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + ListCell *lc; + List *exprs = NIL; + bool ukey_useful = true; + + /* One row case is handled above */ + Assert(ukey->exprs != NIL); + foreach(lc, ukey->exprs) + { + Var *var; + TargetEntry *tle = tlist_member(lfirst(lc), + currel->subroot->processed_tlist); + if (tle == NULL) + { + ukey_useful = false; + break; + } + var = find_var_for_subquery_tle(currel, tle); + if (var == NULL) + { + ukey_useful = false; + break; + } + exprs = lappend(exprs, var); + } + + if (ukey_useful) + currel->uniquekeys = lappend(currel->uniquekeys, + makeUniqueKey(exprs, + ukey->multi_nullvals)); + + } +} + +/* + * innerrel_keeps_unique + * + * Check if Unique key of the innerrel is valid after join. innerrel's UniqueKey + * will be still valid if innerrel's any-column mergeop outrerel's uniquekey + * exists in clause_list. + * + * Note: the clause_list must be a list of mergeable restrictinfo already. + */ +static bool +innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *clause_list, + bool reverse) +{ + ListCell *lc, *lc2, *lc3; + + if (outerrel->uniquekeys == NIL || innerrel->uniquekeys == NIL) + return false; + + /* Check if there is outerrel's uniquekey in mergeable clause. */ + foreach(lc, outerrel->uniquekeys) + { + List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs; + bool clauselist_matchs_all_exprs = true; + foreach(lc2, outer_uq_exprs) + { + Node *outer_uq_expr = lfirst(lc2); + bool find_uq_expr_in_clauselist = false; + foreach(lc3, clause_list) + { + RestrictInfo *rinfo = lfirst_node(RestrictInfo, lc3); + Node *outer_expr; + if (reverse) + outer_expr = rinfo->outer_is_left ? get_rightop(rinfo->clause) : get_leftop(rinfo->clause); + else + outer_expr = rinfo->outer_is_left ? get_leftop(rinfo->clause) : get_rightop(rinfo->clause); + if (equal(outer_expr, outer_uq_expr)) + { + find_uq_expr_in_clauselist = true; + break; + } + } + if (!find_uq_expr_in_clauselist) + { + /* No need to check the next exprs in the current uniquekey */ + clauselist_matchs_all_exprs = false; + break; + } + } + + if (clauselist_matchs_all_exprs) + return true; + } + return false; +} + + +/* + * relation_is_onerow + * Check if it is a one-row relation by checking UniqueKey. + */ +bool +relation_is_onerow(RelOptInfo *rel) +{ + UniqueKey *ukey; + if (rel->uniquekeys == NIL) + return false; + ukey = linitial_node(UniqueKey, rel->uniquekeys); + return ukey->exprs == NIL && list_length(rel->uniquekeys) == 1; +} + +/* + * relation_has_uniquekeys_for + * Returns true if we have proofs that 'rel' cannot return multiple rows with + * the same values in each of 'exprs'. Otherwise returns false. + */ +bool +relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel, + List *exprs, bool allow_multinulls) +{ + ListCell *lc; + + /* + * For UniqueKey->onerow case, the uniquekey->exprs is empty as well + * so we can't rely on list_is_subset to handle this special cases + */ + if (exprs == NIL) + return false; + + foreach(lc, rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (ukey->multi_nullvals && !allow_multinulls) + continue; + if (list_is_subset(ukey->exprs, exprs)) + return true; + } + return false; +} + + +/* + * get_exprs_from_uniqueindex + * + * Return a list of exprs which is unique. set useful to false if this + * unique index is not useful for us. + */ +static List * +get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals) +{ + List *exprs = NIL; + ListCell *indexpr_item; + int c = 0; + + *useful = true; + *multi_nullvals = false; + + indexpr_item = list_head(unique_index->indexprs); + for(c = 0; c < unique_index->ncolumns; c++) + { + int attr = unique_index->indexkeys[c]; + Expr *expr; + bool matched_const = false; + ListCell *lc1, *lc2; + + if(attr > 0) + { + expr = list_nth_node(TargetEntry, unique_index->indextlist, c)->expr; + } + else if (attr == 0) + { + /* Expression index */ + expr = lfirst(indexpr_item); + indexpr_item = lnext(unique_index->indexprs, indexpr_item); + } + else /* attr < 0 */ + { + /* Index on system column is not supported */ + Assert(false); + } + + /* + * Check index_col = Const case with regarding to opfamily checking + * If we can remove the index_col from the final UniqueKey->exprs. + */ + forboth(lc1, const_exprs, lc2, const_expr_opfamilies) + { + if (list_member_oid((List *)lfirst(lc2), unique_index->opfamily[c]) + && match_index_to_operand((Node *) lfirst(lc1), c, unique_index)) + { + matched_const = true; + break; + } + } + + if (matched_const) + continue; + + /* Check if the indexed expr is used in rel */ + if (attr > 0) + { + /* + * Normal Indexed column, if the col is not used, then the index is useless + * for uniquekey. + */ + attr -= FirstLowInvalidHeapAttributeNumber; + + if (!bms_is_member(attr, used_varattrs)) + { + *useful = false; + break; + } + } + else if (!list_member(unique_index->rel->reltarget->exprs, expr)) + { + /* Expression index but the expression is not used in rel */ + *useful = false; + break; + } + + /* check not null property. */ + if (attr == 0) + { + /* We never know if a expression yields null or not */ + *multi_nullvals = true; + } + else if (!bms_is_member(attr, unique_index->rel->notnullattrs) + && !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber, + unique_index->rel->notnullattrs)) + { + *multi_nullvals = true; + } + + exprs = lappend(exprs, expr); + } + return exprs; +} + + +/* + * add_uniquekey_for_onerow + * If we are sure that the relation only returns one row, then all the columns + * are unique. However we don't need to create UniqueKey for every column, we + * just set exprs = NIL and overwrites all the other UniqueKey on this RelOptInfo + * since this one has strongest semantics. + */ +void +add_uniquekey_for_onerow(RelOptInfo *rel) +{ + /* + * We overwrite the previous UniqueKey on purpose since this one has the + * strongest semantic. + */ + rel->uniquekeys = list_make1(makeUniqueKey(NIL, false)); +} + + +/* + * initililze_uniquecontext_for_joinrel + * Return a List of UniqueKeyContext for an inputrel + */ +static List * +initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel) +{ + List *res = NIL; + ListCell *lc; + foreach(lc, inputrel->uniquekeys) + { + UniqueKeyContext context; + context = palloc(sizeof(struct UniqueKeyContextData)); + context->uniquekey = lfirst_node(UniqueKey, lc); + context->added_to_joinrel = false; + context->useful = true; + res = lappend(res, context); + } + return res; +} + + +/* + * get_exprs_from_uniquekey + * Unify the way of get List of exprs from a one-row UniqueKey or + * normal UniqueKey. for the onerow case, every expr in rel1 is a valid + * UniqueKey. Return a List of exprs. + * + * rel1: The relation which you want to get the exprs. + * ukey: The UniqueKey you want to get the exprs. + */ +static List * +get_exprs_from_uniquekey(RelOptInfo *joinrel, RelOptInfo *rel1, UniqueKey *ukey) +{ + ListCell *lc; + bool onerow = rel1 != NULL && relation_is_onerow(rel1); + + List *res = NIL; + Assert(onerow || ukey); + if (onerow) + { + /* Only cares about the exprs still exist in joinrel */ + foreach(lc, joinrel->reltarget->exprs) + { + Bitmapset *relids = pull_varnos(lfirst(lc)); + if (bms_is_subset(relids, rel1->relids)) + { + res = lappend(res, list_make1(lfirst(lc))); + } + } + } + else + { + res = list_make1(ukey->exprs); + } + return res; +} + +/* + * Partitioned table Unique Keys. + * The partition table unique key is maintained as: + * 1. The index must be unique as usual. + * 2. The index must contains partition key. + * 3. The index must exist on all the child rel. see simple_indexinfo_equal for + * how we compare it. + */ + +/* + * index_constains_partkey + * return true if the index contains the partiton key. + */ +static bool +index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind) +{ + ListCell *lc; + int i; + Assert(IS_PARTITIONED_REL(partrel)); + Assert(partrel->part_scheme->partnatts > 0); + + for(i = 0; i < partrel->part_scheme->partnatts; i++) + { + Node *part_expr = linitial(partrel->partexprs[i]); + bool found_in_index = false; + foreach(lc, ind->indextlist) + { + Expr *index_expr = lfirst_node(TargetEntry, lc)->expr; + if (equal(index_expr, part_expr)) + { + found_in_index = true; + break; + } + } + if (!found_in_index) + return false; + } + return true; +} + +/* + * simple_indexinfo_equal + * + * Used to check if the 2 index is same as each other. The index here + * is COPIED from childrel and did some tiny changes(see + * simple_copy_indexinfo_to_parent) + */ +static bool +simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2) +{ + Size oid_cmp_len = sizeof(Oid) * ind1->ncolumns; + + return ind1->ncolumns == ind2->ncolumns && + ind1->unique == ind2->unique && + memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0 && + memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0 && + memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0 && + memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0 && + equal(get_tlist_exprs(ind1->indextlist, true), + get_tlist_exprs(ind2->indextlist, true)); +} + + +/* + * The below macros are used for simple_copy_indexinfo_to_parent which is so + * customized that I don't want to put it to copyfuncs.c. So copy it here. + */ +#define COPY_POINTER_FIELD(fldname, sz) \ + do { \ + Size _size = (sz); \ + newnode->fldname = palloc(_size); \ + memcpy(newnode->fldname, from->fldname, _size); \ + } while (0) + +#define COPY_NODE_FIELD(fldname) \ + (newnode->fldname = copyObjectImpl(from->fldname)) + +#define COPY_SCALAR_FIELD(fldname) \ + (newnode->fldname = from->fldname) + + +/* + * simple_copy_indexinfo_to_parent (from partition) + * Copy the IndexInfo from child relation to parent relation with some modification, + * which is used to test: + * 1. If the same index exists in all the childrels. + * 2. If the parentrel->reltarget/basicrestrict info matches this index. + */ +static IndexOptInfo * +simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from) +{ + IndexOptInfo *newnode = makeNode(IndexOptInfo); + AppendRelInfo *appinfo = find_appinfo_by_child(root, from->rel->relid); + ListCell *lc; + int idx = 0; + + COPY_SCALAR_FIELD(ncolumns); + COPY_SCALAR_FIELD(nkeycolumns); + COPY_SCALAR_FIELD(unique); + COPY_SCALAR_FIELD(immediate); + /* We just need to know if it is NIL or not */ + COPY_SCALAR_FIELD(indpred); + COPY_SCALAR_FIELD(predOK); + COPY_POINTER_FIELD(indexkeys, from->ncolumns * sizeof(int)); + COPY_POINTER_FIELD(indexcollations, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opfamily, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opcintype, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(sortopfamily, from->ncolumns * sizeof(Oid)); + COPY_NODE_FIELD(indextlist); + + /* Convert index exprs on child expr to expr on parent */ + foreach(lc, newnode->indextlist) + { + TargetEntry *tle = lfirst_node(TargetEntry, lc); + /* Index on expression is ignored */ + Assert(IsA(tle->expr, Var)); + tle->expr = (Expr *) find_parent_var(appinfo, (Var *) tle->expr); + newnode->indexkeys[idx] = castNode(Var, tle->expr)->varattno; + idx++; + } + newnode->rel = parentrel; + return newnode; +} + +/* + * adjust_partition_unique_indexlist + * + * global_unique_indexes: At the beginning, it contains the copy & modified + * unique index from the first partition. And then check if each index in it still + * exists in the following partitions. If no, remove it. at last, it has an + * index list which exists in all the partitions. + */ +static void +adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_indexes) +{ + ListCell *lc, *lc2; + foreach(lc, *global_unique_indexes) + { + IndexOptInfo *g_ind = lfirst_node(IndexOptInfo, lc); + bool found_in_child = false; + + foreach(lc2, childrel->indexlist) + { + IndexOptInfo *p_ind = lfirst_node(IndexOptInfo, lc2); + IndexOptInfo *p_ind_copy; + if (!p_ind->unique || !p_ind->immediate || + (p_ind->indpred != NIL && !p_ind->predOK)) + continue; + p_ind_copy = simple_copy_indexinfo_to_parent(root, parentrel, p_ind); + if (simple_indexinfo_equal(p_ind_copy, g_ind)) + { + found_in_child = true; + break; + } + } + if (!found_in_child) + /* The index doesn't exist in childrel, remove it from global_unique_indexes */ + *global_unique_indexes = foreach_delete_current(*global_unique_indexes, lc); + } +} + +/* Helper function for groupres/distinctrel */ +static void +add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgroups) +{ + Query *parse = root->parse; + List *exprs; + + /* + * XXX: If there are some vars which is not in current levelsup, the semantic is + * imprecise, should we avoid it or not? levelsup = 1 is just a demo, maybe we need to + * check every level other than 0, if so, looks we have to write another + * pull_var_walker. + */ + List *upper_vars = pull_vars_of_level((Node*)sortgroups, 1); + + if (upper_vars != NIL) + return; + + exprs = get_sortgrouplist_exprs(sortgroups, parse->targetList); + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(exprs, + false /* sortgroupclause can't be multi_nullvals */)); +} + + +/* + * add_combined_uniquekey + * The combination of both UniqueKeys is a valid UniqueKey for joinrel no matter + * the jointype. + */ +bool +add_combined_uniquekey(RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype) +{ + + ListCell *lc1, *lc2; + + /* Either side has multi_nullvals or we have outer join, + * the combined UniqueKey has multi_nullvals */ + bool multi_nullvals = outer_ukey->multi_nullvals || + inner_ukey->multi_nullvals || IS_OUTER_JOIN(jointype); + + /* The only case we can get onerow joinrel after join */ + if (relation_is_onerow(outer_rel) + && relation_is_onerow(inner_rel) + && jointype == JOIN_INNER) + { + add_uniquekey_for_onerow(joinrel); + return true; + } + + foreach(lc1, get_exprs_from_uniquekey(joinrel, outer_rel, outer_ukey)) + { + foreach(lc2, get_exprs_from_uniquekey(joinrel, inner_rel, inner_ukey)) + { + List *exprs = list_concat_copy(lfirst_node(List, lc1), lfirst_node(List, lc2)); + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(exprs, + multi_nullvals)); + } + } + return false; +} diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index b406d41e91..0551ae0512 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -2389,6 +2389,8 @@ grouping_planner(PlannerInfo *root, bool inheritance_update, add_path(final_rel, path); } + simple_copy_uniquekeys(current_rel, final_rel); + /* * Generate partial paths for final_rel, too, if outer query levels might * be able to make use of them. @@ -3899,6 +3901,8 @@ create_grouping_paths(PlannerInfo *root, } set_cheapest(grouped_rel); + + populate_grouprel_uniquekeys(root, grouped_rel, input_rel); return grouped_rel; } @@ -4615,7 +4619,7 @@ create_window_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(window_rel); - + simple_copy_uniquekeys(input_rel, window_rel); return window_rel; } @@ -4911,7 +4915,7 @@ create_distinct_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(distinct_rel); - + populate_distinctrel_uniquekeys(root, input_rel, distinct_rel); return distinct_rel; } @@ -5172,6 +5176,8 @@ create_ordered_paths(PlannerInfo *root, */ Assert(ordered_rel->pathlist != NIL); + simple_copy_uniquekeys(input_rel, ordered_rel); + return ordered_rel; } @@ -6049,6 +6055,9 @@ adjust_paths_for_srfs(PlannerInfo *root, RelOptInfo *rel, if (list_length(targets) == 1) return; + /* UniqueKey is not valid after handling the SRF. */ + rel->uniquekeys = NIL; + /* * Stack SRF-evaluation nodes atop each path for the rel. * diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c index 951aed80e7..e94e92937c 100644 --- a/src/backend/optimizer/prep/prepunion.c +++ b/src/backend/optimizer/prep/prepunion.c @@ -689,6 +689,8 @@ generate_union_paths(SetOperationStmt *op, PlannerInfo *root, /* Undo effects of possibly forcing tuple_fraction to 0 */ root->tuple_fraction = save_fraction; + /* Add the UniqueKeys */ + populate_unionrel_uniquekeys(root, result_rel); return result_rel; } diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c index d722063cf3..44c37ecffc 100644 --- a/src/backend/optimizer/util/appendinfo.c +++ b/src/backend/optimizer/util/appendinfo.c @@ -746,3 +746,47 @@ find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos) } return appinfos; } + +/* + * find_appinfo_by_child + * + */ +AppendRelInfo * +find_appinfo_by_child(PlannerInfo *root, Index child_index) +{ + ListCell *lc; + foreach(lc, root->append_rel_list) + { + AppendRelInfo *appinfo = lfirst_node(AppendRelInfo, lc); + if (appinfo->child_relid == child_index) + return appinfo; + } + elog(ERROR, "parent relation cant be found"); + return NULL; +} + +/* + * find_parent_var + * + */ +Var * +find_parent_var(AppendRelInfo *appinfo, Var *child_var) +{ + ListCell *lc; + Var *res = NULL; + Index attno = 1; + foreach(lc, appinfo->translated_vars) + { + Node *child_node = lfirst(lc); + if (equal(child_node, child_var)) + { + res = copyObject(child_var); + res->varattno = attno; + res->varno = appinfo->parent_relid; + } + attno++; + } + if (res == NULL) + elog(ERROR, "parent var can't be found."); + return res; +} diff --git a/src/backend/optimizer/util/inherit.c b/src/backend/optimizer/util/inherit.c index 3132fd35a5..d66b40ec50 100644 --- a/src/backend/optimizer/util/inherit.c +++ b/src/backend/optimizer/util/inherit.c @@ -736,6 +736,7 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, { Node *onecq = (Node *) lfirst(lc2); bool pseudoconstant; + RestrictInfo *child_rinfo; /* check for pseudoconstant (no Vars or volatile functions) */ pseudoconstant = @@ -747,13 +748,14 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, root->hasPseudoConstantQuals = true; } /* reconstitute RestrictInfo with appropriate properties */ - childquals = lappend(childquals, - make_restrictinfo((Expr *) onecq, - rinfo->is_pushed_down, - rinfo->outerjoin_delayed, - pseudoconstant, - rinfo->security_level, - NULL, NULL, NULL)); + child_rinfo = make_restrictinfo((Expr *) onecq, + rinfo->is_pushed_down, + rinfo->outerjoin_delayed, + pseudoconstant, + rinfo->security_level, + NULL, NULL, NULL); + child_rinfo->mergeopfamilies = rinfo->mergeopfamilies; + childquals = lappend(childquals, child_rinfo); /* track minimum security level among child quals */ cq_min_security = Min(cq_min_security, rinfo->security_level); } diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 31d9aedeeb..c83f17acb7 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -16,6 +16,7 @@ #include "nodes/execnodes.h" #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" extern A_Expr *makeA_Expr(A_Expr_Kind kind, List *name, @@ -105,4 +106,6 @@ extern GroupingSet *makeGroupingSet(GroupingSetKind kind, List *content, int loc extern VacuumRelation *makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols); +extern UniqueKey* makeUniqueKey(List *exprs, bool multi_nullvals); + #endif /* MAKEFUNC_H */ diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 381d84b4e4..41110ed888 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -264,6 +264,7 @@ typedef enum NodeTag T_EquivalenceMember, T_PathKey, T_PathTarget, + T_UniqueKey, T_RestrictInfo, T_IndexClause, T_PlaceHolderVar, diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index 9e3ebd488a..02e4458bef 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -730,6 +730,7 @@ typedef struct RelOptInfo QualCost baserestrictcost; /* cost of evaluating the above */ Index baserestrict_min_security; /* min security_level found in * baserestrictinfo */ + List *uniquekeys; /* List of UniqueKey */ List *joininfo; /* RestrictInfo structures for join clauses * involving this rel */ bool has_eclass_joins; /* T means joininfo is incomplete */ @@ -1047,6 +1048,28 @@ typedef struct PathKey } PathKey; +/* + * UniqueKey + * + * Represents the unique properties held by a RelOptInfo. + * + * exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL + * is a special case of UniqueKey, which means there is only 1 row in that + * relation. + * multi_nullvals: true means multi null values may exist in these exprs, so the + * uniqueness is not guaranteed in this case. This field is necessary for + * remove_useless_join & reduce_unique_semijoins where we don't mind these + * duplicated NULL values. It is set to true for 2 cases. One is a unique key + * from a unique index but the related column is nullable. The other one is for + * outer join. see populate_joinrel_uniquekeys for detail. + */ +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + /* * PathTarget * @@ -2473,7 +2496,7 @@ typedef enum * * flags indicating what kinds of grouping are possible. * partial_costs_set is true if the agg_partial_costs and agg_final_costs - * have been initialized. + * have been initialized. * agg_partial_costs gives partial aggregation costs. * agg_final_costs gives finalization costs. * target_parallel_safe is true if target is parallel safe. @@ -2503,8 +2526,8 @@ typedef struct * limit_tuples is an estimated bound on the number of output tuples, * or -1 if no LIMIT or couldn't estimate. * count_est and offset_est are the estimated values of the LIMIT and OFFSET - * expressions computed by preprocess_limit() (see comments for - * preprocess_limit() for more information). + * expressions computed by preprocess_limit() (see comments for + * preprocess_limit() for more information). */ typedef struct { diff --git a/src/include/nodes/pg_list.h b/src/include/nodes/pg_list.h index 14ea2766ad..621f54a9f8 100644 --- a/src/include/nodes/pg_list.h +++ b/src/include/nodes/pg_list.h @@ -528,6 +528,8 @@ extern bool list_member_ptr(const List *list, const void *datum); extern bool list_member_int(const List *list, int datum); extern bool list_member_oid(const List *list, Oid datum); +extern bool list_is_subset(const List *members, const List *target); + extern List *list_delete(List *list, void *datum); extern List *list_delete_ptr(List *list, void *datum); extern List *list_delete_int(List *list, int datum); diff --git a/src/include/optimizer/appendinfo.h b/src/include/optimizer/appendinfo.h index d6a27a60dd..e87c92a054 100644 --- a/src/include/optimizer/appendinfo.h +++ b/src/include/optimizer/appendinfo.h @@ -32,4 +32,7 @@ extern Relids adjust_child_relids_multilevel(PlannerInfo *root, Relids relids, extern AppendRelInfo **find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos); +extern AppendRelInfo *find_appinfo_by_child(PlannerInfo *root, Index child_index); +extern Var *find_parent_var(AppendRelInfo *appinfo, Var *child_var); + #endif /* APPENDINFO_H */ diff --git a/src/include/optimizer/optimizer.h b/src/include/optimizer/optimizer.h index 3e4171056e..9445141263 100644 --- a/src/include/optimizer/optimizer.h +++ b/src/include/optimizer/optimizer.h @@ -23,6 +23,7 @@ #define OPTIMIZER_H #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" /* * We don't want to include nodes/pathnodes.h here, because non-planner @@ -156,6 +157,7 @@ extern TargetEntry *get_sortgroupref_tle(Index sortref, List *targetList); extern TargetEntry *get_sortgroupclause_tle(SortGroupClause *sgClause, List *targetList); +extern Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); extern Node *get_sortgroupclause_expr(SortGroupClause *sgClause, List *targetList); extern List *get_sortgrouplist_exprs(List *sgClauses, diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h index 10b6e81079..9217a8d6c6 100644 --- a/src/include/optimizer/paths.h +++ b/src/include/optimizer/paths.h @@ -240,5 +240,48 @@ extern PathKey *make_canonical_pathkey(PlannerInfo *root, int strategy, bool nulls_first); extern void add_paths_to_append_rel(PlannerInfo *root, RelOptInfo *rel, List *live_childrels); +extern List *select_mergejoin_clauses(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + JoinType jointype, + bool *mergejoin_allowed); + +/* + * uniquekeys.c + * Utilities for matching and building unique keys + */ +extern void populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List* unique_index_list); +extern void populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels); +extern void populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel); +extern void populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel); +extern void populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel); +extern void simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel); +extern void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel); +extern void populate_joinrel_uniquekeys(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + RelOptInfo *rel2, + List *restrictlist, + JoinType jointype); + +extern bool relation_has_uniquekeys_for(PlannerInfo *root, + RelOptInfo *rel, + List *exprs, + bool allow_multinulls); +extern bool relation_is_onerow(RelOptInfo *rel); #endif /* PATHS_H */ -- 2.21.0 --op7mf2i72xk5yu7c Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v36-0003-Extend-UniqueKeys.patch" ^ permalink raw reply [nested|flat] 91+ messages in thread
* [PATCH v38 2/6] Introduce UniqueKey attributes on RelOptInfo struct. @ 2020-05-11 07:50 一挃 <[email protected]> 0 siblings, 0 replies; 91+ messages in thread From: 一挃 @ 2020-05-11 07:50 UTC (permalink / raw) UniqueKey is a set of exprs on RelOptInfo which represents the exprs will be unique on the given RelOptInfo. You can see README.uniquekey for more information. --- src/backend/nodes/copyfuncs.c | 13 + src/backend/nodes/list.c | 31 + src/backend/nodes/makefuncs.c | 13 + src/backend/nodes/outfuncs.c | 11 + src/backend/nodes/readfuncs.c | 10 + src/backend/optimizer/path/Makefile | 3 +- src/backend/optimizer/path/README.uniquekey | 131 +++ src/backend/optimizer/path/allpaths.c | 10 + src/backend/optimizer/path/joinpath.c | 9 +- src/backend/optimizer/path/joinrels.c | 2 + src/backend/optimizer/path/pathkeys.c | 3 +- src/backend/optimizer/path/uniquekeys.c | 1135 +++++++++++++++++++ src/backend/optimizer/plan/planner.c | 13 +- src/backend/optimizer/prep/prepunion.c | 2 + src/backend/optimizer/util/appendinfo.c | 44 + src/backend/optimizer/util/inherit.c | 18 +- src/include/nodes/makefuncs.h | 3 + src/include/nodes/nodes.h | 1 + src/include/nodes/pathnodes.h | 29 +- src/include/nodes/pg_list.h | 10 + src/include/optimizer/appendinfo.h | 3 + src/include/optimizer/optimizer.h | 2 + src/include/optimizer/paths.h | 43 + 23 files changed, 1515 insertions(+), 24 deletions(-) create mode 100644 src/backend/optimizer/path/README.uniquekey create mode 100644 src/backend/optimizer/path/uniquekeys.c diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..75c1c5e824 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2296,6 +2296,16 @@ _copyPathKey(const PathKey *from) return newnode; } +static UniqueKey * +_copyUniqueKey(const UniqueKey *from) +{ + UniqueKey *newnode = makeNode(UniqueKey); + + COPY_NODE_FIELD(exprs); + COPY_SCALAR_FIELD(multi_nullvals); + + return newnode; +} /* * _copyRestrictInfo */ @@ -5220,6 +5230,9 @@ copyObjectImpl(const void *from) case T_PathKey: retval = _copyPathKey(from); break; + case T_UniqueKey: + retval = _copyUniqueKey(from); + break; case T_RestrictInfo: retval = _copyRestrictInfo(from); break; diff --git a/src/backend/nodes/list.c b/src/backend/nodes/list.c index dbf6b30233..ca099495a1 100644 --- a/src/backend/nodes/list.c +++ b/src/backend/nodes/list.c @@ -702,6 +702,37 @@ list_member_oid(const List *list, Oid datum) return false; } +/* + * return true iff every entry in "members" list is also present + * in the "target" list. + */ +bool +list_is_subset(const List *members, const List *target) +{ + const ListCell *lc1, *lc2; + + Assert(IsPointerList(members)); + Assert(IsPointerList(target)); + check_list_invariants(members); + check_list_invariants(target); + + foreach(lc1, members) + { + bool found = false; + foreach(lc2, target) + { + if (equal(lfirst(lc1), lfirst(lc2))) + { + found = true; + break; + } + } + if (!found) + return false; + } + return true; +} + /* * Delete the n'th cell (counting from 0) in list. * diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 01c110cd2f..40415d0f5b 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -815,3 +815,16 @@ makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols) v->va_cols = va_cols; return v; } + + +/* + * makeUniqueKey + */ +UniqueKey* +makeUniqueKey(List *exprs, bool multi_nullvals) +{ + UniqueKey * ukey = makeNode(UniqueKey); + ukey->exprs = exprs; + ukey->multi_nullvals = multi_nullvals; + return ukey; +} diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..44154cde6a 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2456,6 +2456,14 @@ _outPathKey(StringInfo str, const PathKey *node) WRITE_BOOL_FIELD(pk_nulls_first); } +static void +_outUniqueKey(StringInfo str, const UniqueKey *node) +{ + WRITE_NODE_TYPE("UNIQUEKEY"); + WRITE_NODE_FIELD(exprs); + WRITE_BOOL_FIELD(multi_nullvals); +} + static void _outPathTarget(StringInfo str, const PathTarget *node) { @@ -4198,6 +4206,9 @@ outNode(StringInfo str, const void *obj) case T_PathKey: _outPathKey(str, obj); break; + case T_UniqueKey: + _outUniqueKey(str, obj); + break; case T_PathTarget: _outPathTarget(str, obj); break; diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index c5e136e9c3..b3e212bf1c 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -492,6 +492,14 @@ _readSetOperationStmt(void) READ_DONE(); } +static UniqueKey * +_readUniqueKey(void) +{ + READ_LOCALS(UniqueKey); + READ_NODE_FIELD(exprs); + READ_BOOL_FIELD(multi_nullvals); + READ_DONE(); +} /* * Stuff from primnodes.h. @@ -2717,6 +2725,8 @@ parseNodeString(void) return_value = _readCommonTableExpr(); else if (MATCH("SETOPERATIONSTMT", 16)) return_value = _readSetOperationStmt(); + else if (MATCH("UNIQUEKEY", 9)) + return_value = _readUniqueKey(); else if (MATCH("ALIAS", 5)) return_value = _readAlias(); else if (MATCH("RANGEVAR", 8)) diff --git a/src/backend/optimizer/path/Makefile b/src/backend/optimizer/path/Makefile index 1e199ff66f..7b9820c25f 100644 --- a/src/backend/optimizer/path/Makefile +++ b/src/backend/optimizer/path/Makefile @@ -21,6 +21,7 @@ OBJS = \ joinpath.o \ joinrels.o \ pathkeys.o \ - tidpath.o + tidpath.o \ + uniquekeys.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/optimizer/path/README.uniquekey b/src/backend/optimizer/path/README.uniquekey new file mode 100644 index 0000000000..5eac761995 --- /dev/null +++ b/src/backend/optimizer/path/README.uniquekey @@ -0,0 +1,131 @@ +1. What is UniqueKey? +We can think UniqueKey is a set of exprs for a RelOptInfo, which we are insure +that doesn't yields same result among all the rows. The simplest UniqueKey +format is primary key. + +However we define the UnqiueKey as below. + +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + +exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL +is a special case of UniqueKey, which means there is only one row in that +relation.it has a stronger semantic than others. like SELECT uk FROM t; uk is +normal unique key and may have different values. SELECT colx FROM t WHERE uk = +const. colx is unique AND we have only 1 value. This field can used for +innerrel_is_unique. this logic is handled specially in add_uniquekey_for_onerow +function. + +multi_nullvals: true means multi null values may exist in these exprs, so the +uniqueness is not guaranteed in this case. This field is necessary for +remove_useless_join & reduce_unique_semijoins where we don't mind these +duplicated NULL values. It is set to true for 2 cases. One is a unique key +from a unique index but the related column is nullable. The other one is for +outer join. see populate_joinrel_uniquekeys for detail. + + +The UniqueKey can be used at the following cases at least: +1. remove_useless_joins. +2. reduce_semianti_joins +3. remove distinct node if distinct clause is unique. +4. remove aggnode if group by clause is unique. +5. Index Skip Scan (WIP) +6. Aggregation Push Down without 2 phase aggregation if the join can't + duplicated the aggregated rows. (work in progress feature) + +2. How is it maintained? + +We have a set of populate_xxx_unqiuekeys functions to maintain the uniquekey on +various cases. xxx includes baserel, joinrel, partitionedrel, distinctrel, +groupedrel, unionrel. and we also need to convert the uniquekey from subquery +to outer relation, which is what convert_subquery_uniquekeys does. + +1. The first part is about baserel. We handled 3 cases. suppose we have Unique +Index on (a, b). + +1. SELECT a, b FROM t. UniqueKey (a, b) +2. SELECT a FROM t WHERE b = 1; UniqueKey (a) +3. SELECT .. FROM t WHERE a = 1 AND b = 1; UniqueKey (NIL). onerow case, every + column is Unique. + +2. The next part is joinrel, this part is most error-prone, we simplified the rules +like below: +1. If the relation's UniqueKey can't be duplicated after join, then is will be + still valid for the join rel. The function we used here is + innerrel_keeps_unique. The basic idea is innerrel.any_col = outer.uk. + +2. If the UnqiueKey can't keep valid via the rule 1, the combination of the + UniqueKey from both sides are valid for sure. We can prove this as: if the + unique exprs from rel1 is duplicated by rel2, the duplicated rows must + contains different unique exprs from rel2. + +More considerations about onerow: +1. If relation with one row and it can't be duplicated, it is still possible + contains mulit_nullvas after outer join. +2. If the either UniqueKey can be duplicated after join, the can get one row + only when both side is one row AND there is no outer join. +3. Whenever the onerow UniqueKey is not a valid any more, we need to convert one + row UniqueKey to normal unique key since we don't store exprs for one-row + relation. get_exprs_from_uniquekeys will be used here. + + +More considerations about multi_nullvals after join: +1. If the original UnqiueKey has multi_nullvals, the final UniqueKey will have + mulit_nullvals in any case. +2. If a unique key doesn't allow mulit_nullvals, after some outer join, it + allows some outer join. + + +3. When we comes to subquery, we need to convert_subquery_unqiuekeys just like +convert_subquery_pathkeys. Only the UniqueKey insides subquery is referenced as +a Var in outer relation will be reused. The relationship between the outerrel.Var +and subquery.exprs is built with outerel->subroot->processed_tlist. + + +4. As for the SRF functions, it will break the uniqueness of uniquekey, However it +is handled in adjust_paths_for_srfs, which happens after the query_planner. so +we will maintain the UniqueKey until there and reset it to NIL at that +places. This can't help on distinct/group by elimination cases but probably help +in some other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is +semantic correctly. + + +5. As for inherit table, we first main the UnqiueKey on childrel as well. But for +partitioned table we need to maintain 2 different kinds of +UnqiueKey. 1). UniqueKey on the parent relation 2). UniqueKey on child +relation for partition wise query. + +Example: +CREATE TABLE p (a int not null, b int not null) partition by list (a); +CREATE TABLE p0 partition of p for values in (1); +CREATE TABLE p1 partition of p for values in (2); + +create unique index p0_b on p0(b); +create unique index p1_b on p1(b); + +Now b is only unique on partition level, so the distinct can't be removed on +the following cases. SELECT DISTINCT b FROM p; + +Another example is SELECT DISTINCT a, b FROM p WHERE a = 1; Since only one +partition is chosen, the UniqueKey on child relation is same as the UniqueKey on +parent relation. + +Another usage of UniqueKey on partition level is it be helpful for +partition-wise join. + +As for the UniqueKey on parent table level, it comes with 2 different ways, +1). the UniqueKey is also derived in UniqueKey index, but the index must be same +in all the related children relations and the unique index must contains +Partition Key in it. Example: + +CREATE UNIQUE INDEX p_ab ON p(a, b); -- where a is the partition key. + +-- Query +SELECT a, b FROM p; the (a, b) is a UniqueKey of p. + +2). If the parent relation has only one childrel, the UniqueKey on childrel is + the UniqueKey on parent as well. diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index 37b4223adb..d52ad59f89 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -579,6 +579,12 @@ set_plain_rel_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) */ check_index_predicates(root, rel); + /* + * Now that we've marked which partial indexes are suitable, we can now + * build the relation's unique keys. + */ + populate_baserel_uniquekeys(root, rel, rel->indexlist); + /* Mark rel with estimated output rows, width, etc */ set_baserel_size_estimates(root, rel); } @@ -1297,6 +1303,8 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, /* Add paths to the append relation. */ add_paths_to_append_rel(root, rel, live_childrels); + if (IS_PARTITIONED_REL(rel)) + populate_partitionedrel_uniquekeys(root, rel, live_childrels); } @@ -2304,6 +2312,8 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, pathkeys, required_outer)); } + convert_subquery_uniquekeys(root, rel, sub_final_rel); + /* If outer rel allows parallelism, do same for partial paths. */ if (rel->consider_parallel && bms_is_empty(required_outer)) { diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c index 57ce97fd53..697ac047fb 100644 --- a/src/backend/optimizer/path/joinpath.c +++ b/src/backend/optimizer/path/joinpath.c @@ -71,13 +71,6 @@ static void consider_parallel_mergejoin(PlannerInfo *root, static void hash_inner_and_outer(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, RelOptInfo *innerrel, JoinType jointype, JoinPathExtraData *extra); -static List *select_mergejoin_clauses(PlannerInfo *root, - RelOptInfo *joinrel, - RelOptInfo *outerrel, - RelOptInfo *innerrel, - List *restrictlist, - JoinType jointype, - bool *mergejoin_allowed); static void generate_mergejoin_paths(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *innerrel, @@ -1927,7 +1920,7 @@ hash_inner_and_outer(PlannerInfo *root, * if it is mergejoinable and involves vars from the two sub-relations * currently of interest. */ -static List * +List * select_mergejoin_clauses(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c index 0dbe2ac726..7271f044ec 100644 --- a/src/backend/optimizer/path/joinrels.c +++ b/src/backend/optimizer/path/joinrels.c @@ -924,6 +924,8 @@ populate_joinrel_with_paths(PlannerInfo *root, RelOptInfo *rel1, /* Apply partitionwise join technique, if possible. */ try_partitionwise_join(root, rel1, rel2, joinrel, sjinfo, restrictlist); + + populate_joinrel_uniquekeys(root, joinrel, rel1, rel2, restrictlist, sjinfo->jointype); } diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index bd9a176d7d..139278829b 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -33,7 +33,6 @@ static bool pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys); static bool matches_boolean_partition_clause(RestrictInfo *rinfo, RelOptInfo *partrel, int partkeycol); -static Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); static bool right_merge_direction(PlannerInfo *root, PathKey *pathkey); @@ -1035,7 +1034,7 @@ convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel, * We need this to ensure that we don't return pathkeys describing values * that are unavailable above the level of the subquery scan. */ -static Var * +Var * find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle) { ListCell *lc; diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c new file mode 100644 index 0000000000..77ed2b2eff --- /dev/null +++ b/src/backend/optimizer/path/uniquekeys.c @@ -0,0 +1,1135 @@ +/*------------------------------------------------------------------------- + * + * uniquekeys.c + * Utilities for matching and building unique keys + * + * Portions Copyright (c) 2020, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/optimizer/path/uniquekeys.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "nodes/makefuncs.h" +#include "nodes/nodeFuncs.h" +#include "optimizer/pathnode.h" +#include "optimizer/paths.h" +#include "optimizer/appendinfo.h" +#include "optimizer/optimizer.h" +#include "optimizer/tlist.h" +#include "rewrite/rewriteManip.h" + + +/* + * This struct is used to help populate_joinrel_uniquekeys. + * + * added_to_joinrel is true if a uniquekey (from outerrel or innerrel) + * has been added to joinrel. + * useful is true if the exprs of the uniquekey still appears in joinrel. + */ +typedef struct UniqueKeyContextData +{ + UniqueKey *uniquekey; + bool added_to_joinrel; + bool useful; +} *UniqueKeyContext; + +static List *initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel); +static bool innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + bool reverse); + +static List *get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals); +static List *get_exprs_from_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + UniqueKey *ukey); +static void add_uniquekey_for_onerow(RelOptInfo *rel); +static bool add_combined_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype); + +/* Used for unique indexes checking for partitioned table */ +static bool index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind); +static IndexOptInfo *simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from); +static bool simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2); +static void adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_index); + +/* Helper function for grouped relation and distinct relation. */ +static void add_uniquekey_from_sortgroups(PlannerInfo *root, + RelOptInfo *rel, + List *sortgroups); + +/* + * populate_baserel_uniquekeys + * Populate 'baserel' uniquekeys list by looking at the rel's unique index + * and baserestrictinfo + */ +void +populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List *indexlist) +{ + ListCell *lc; + List *matched_uniq_indexes = NIL; + + /* Attrs appears in rel->reltarget->exprs. */ + Bitmapset *used_attrs = NULL; + + List *const_exprs = NIL; + List *expr_opfamilies = NIL; + + Assert(baserel->rtekind == RTE_RELATION); + + foreach(lc, indexlist) + { + IndexOptInfo *ind = (IndexOptInfo *) lfirst(lc); + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + matched_uniq_indexes = lappend(matched_uniq_indexes, ind); + } + + if (matched_uniq_indexes == NIL) + return; + + /* Check which attrs is used in baserel->reltarget */ + pull_varattnos((Node *)baserel->reltarget->exprs, baserel->relid, &used_attrs); + + /* Check which attrno is used at a mergeable const filter */ + foreach(lc, baserel->baserestrictinfo) + { + RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); + + if (rinfo->mergeopfamilies == NIL) + continue; + + if (bms_is_empty(rinfo->left_relids)) + { + const_exprs = lappend(const_exprs, get_rightop(rinfo->clause)); + } + else if (bms_is_empty(rinfo->right_relids)) + { + const_exprs = lappend(const_exprs, get_leftop(rinfo->clause)); + } + else + continue; + + expr_opfamilies = lappend(expr_opfamilies, rinfo->mergeopfamilies); + } + + foreach(lc, matched_uniq_indexes) + { + bool multi_nullvals, useful; + List *exprs = get_exprs_from_uniqueindex(lfirst_node(IndexOptInfo, lc), + const_exprs, + expr_opfamilies, + used_attrs, + &useful, + &multi_nullvals); + if (useful) + { + if (exprs == NIL) + { + /* All the columns in Unique Index matched with a restrictinfo */ + add_uniquekey_for_onerow(baserel); + return; + } + baserel->uniquekeys = lappend(baserel->uniquekeys, + makeUniqueKey(exprs, multi_nullvals)); + } + } +} + + +/* + * populate_partitionedrel_uniquekeys + * The UniqueKey on partitionrel comes from 2 cases: + * 1). Only one partition is involved in this query, the unique key can be + * copied to parent rel from childrel. + * 2). There are some unique index which includes partition key and exists + * in all the related partitions. + * We never mind rule 2 if we hit rule 1. + */ + +void +populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels) +{ + ListCell *lc; + List *global_uniq_indexlist = NIL; + RelOptInfo *childrel; + bool is_first = true; + + Assert(IS_PARTITIONED_REL(rel)); + + if (childrels == NIL) + return; + + /* + * If there is only one partition used in this query, the UniqueKey in childrel is + * still valid in parent level, but we need convert the format from child expr to + * parent expr. + */ + if (list_length(childrels) == 1) + { + /* Check for Rule 1 */ + RelOptInfo *childrel = linitial_node(RelOptInfo, childrels); + ListCell *lc; + Assert(childrel->reloptkind == RELOPT_OTHER_MEMBER_REL); + if (relation_is_onerow(childrel)) + { + add_uniquekey_for_onerow(rel); + return; + } + + foreach(lc, childrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + AppendRelInfo *appinfo = find_appinfo_by_child(root, childrel->relid); + List *parent_exprs = NIL; + bool can_reuse = true; + ListCell *lc2; + foreach(lc2, ukey->exprs) + { + Var *var = (Var *)lfirst(lc2); + /* + * If the expr comes from a expression, it is hard to build the expression + * in parent so ignore that case for now. + */ + if(!IsA(var, Var)) + { + can_reuse = false; + break; + } + /* Convert it to parent var */ + parent_exprs = lappend(parent_exprs, find_parent_var(appinfo, var)); + } + if (can_reuse) + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(parent_exprs, + ukey->multi_nullvals)); + } + } + else + { + /* Check for rule 2 */ + childrel = linitial_node(RelOptInfo, childrels); + foreach(lc, childrel->indexlist) + { + IndexOptInfo *ind = lfirst(lc); + IndexOptInfo *modified_index; + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + + /* + * During simple_copy_indexinfo_to_parent, we need to convert var from + * child var to parent var, index on expression is too complex to handle. + * so ignore it for now. + */ + if (ind->indexprs != NIL) + continue; + + modified_index = simple_copy_indexinfo_to_parent(root, rel, ind); + /* + * If the unique index doesn't contain partkey, then it is unique + * on this partition only, so it is useless for us. + */ + if (!index_constains_partkey(rel, modified_index)) + continue; + + global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index); + } + + if (global_uniq_indexlist != NIL) + { + foreach(lc, childrels) + { + RelOptInfo *child = lfirst(lc); + if (is_first) + { + is_first = false; + continue; + } + adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist); + } + /* Now we have a list of unique index which are exactly same on all childrels, + * Set the UniqueKey just like it is non-partition table + */ + populate_baserel_uniquekeys(root, rel, global_uniq_indexlist); + } + } +} + + +/* + * populate_distinctrel_uniquekeys + */ +void +populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel) +{ + /* The unique key before the distinct is still valid. */ + distinctrel->uniquekeys = list_copy(inputrel->uniquekeys); + add_uniquekey_from_sortgroups(root, distinctrel, root->parse->distinctClause); +} + +/* + * populate_grouprel_uniquekeys + */ +void +populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel) + +{ + Query *parse = root->parse; + bool input_ukey_added = false; + ListCell *lc; + + if (relation_is_onerow(inputrel)) + { + add_uniquekey_for_onerow(grouprel); + return; + } + if (parse->groupingSets) + return; + + /* A Normal group by without grouping set. */ + if (parse->groupClause) + { + /* + * Current even the groupby clause is Unique already, but if query has aggref + * We have to create grouprel still. To keep the UnqiueKey short, we will check + * the UniqueKey of input_rel still valid, if so we reuse it. + */ + foreach(lc, inputrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (list_is_subset(ukey->exprs, grouprel->reltarget->exprs)) + { + grouprel->uniquekeys = lappend(grouprel->uniquekeys, + ukey); + input_ukey_added = true; + } + } + if (!input_ukey_added) + /* + * group by clause must be a super-set of grouprel->reltarget->exprs except the + * aggregation expr, so if such exprs is unique already, no bother to generate + * new uniquekey for group by exprs. + */ + add_uniquekey_from_sortgroups(root, + grouprel, + root->parse->groupClause); + } + else + /* It has aggregation but without a group by, so only one row returned */ + add_uniquekey_for_onerow(grouprel); +} + +/* + * simple_copy_uniquekeys + * Using a function for the one-line code makes us easy to check where we simply + * copied the uniquekey. + */ +void +simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel) +{ + newrel->uniquekeys = oldrel->uniquekeys; +} + +/* + * populate_unionrel_uniquekeys + */ +void +populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel) +{ + ListCell *lc; + List *exprs = NIL; + + Assert(unionrel->uniquekeys == NIL); + + foreach(lc, unionrel->reltarget->exprs) + { + exprs = lappend(exprs, lfirst(lc)); + } + + if (exprs == NIL) + /* SQL: select union select; is valid, we need to handle it here. */ + add_uniquekey_for_onerow(unionrel); + else + unionrel->uniquekeys = lappend(unionrel->uniquekeys, + makeUniqueKey(exprs,false)); + +} + +/* + * populate_joinrel_uniquekeys + * + * populate uniquekeys for joinrel. We will check each relation to see if its + * UniqueKey is still valid via innerrel_keeps_unique, if so, we add it to + * joinrel. The multi_nullvals field will be changed to true for some outer + * join cases and one-row UniqueKey needs to be converted to normal UniqueKey + * for the same case as well. + * For the uniquekey in either baserel which can't be unique after join, we still + * check to see if combination of UniqueKeys from both side is still useful for us. + * if yes, we add it to joinrel as well. + */ +void +populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *outerrel, RelOptInfo *innerrel, + List *restrictlist, JoinType jointype) +{ + ListCell *lc, *lc2; + List *clause_list = NIL; + List *outerrel_ukey_ctx; + List *innerrel_ukey_ctx; + bool inner_onerow, outer_onerow; + bool mergejoin_allowed; + + /* Care about the outerrel relation only for SEMI/ANTI join */ + if (jointype == JOIN_SEMI || jointype == JOIN_ANTI) + { + foreach(lc, outerrel->uniquekeys) + { + UniqueKey *uniquekey = lfirst_node(UniqueKey, lc); + if (list_is_subset(uniquekey->exprs, joinrel->reltarget->exprs)) + joinrel->uniquekeys = lappend(joinrel->uniquekeys, uniquekey); + } + return; + } + + Assert(jointype == JOIN_LEFT || jointype == JOIN_FULL || jointype == JOIN_INNER); + + /* Fast path */ + if (innerrel->uniquekeys == NIL || outerrel->uniquekeys == NIL) + return; + + inner_onerow = relation_is_onerow(innerrel); + outer_onerow = relation_is_onerow(outerrel); + + outerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(outerrel); + innerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(innerrel); + + clause_list = select_mergejoin_clauses(root, joinrel, outerrel, innerrel, + restrictlist, jointype, + &mergejoin_allowed); + + if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true /* reverse */)) + { + bool outer_impact = jointype == JOIN_FULL; + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + /* Outer relation has one row, and the unique key is not duplicated after join, + * the joinrel will still has one row unless the jointype == JOIN_FULL. + */ + if (outer_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (outer_onerow) + { + /* + * The onerow outerrel becomes multi rows and multi_nullvals + * will be changed to true. We also need to set the exprs correctly since it + * can't be NIL any more. + */ + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, outerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Change multi_nullvals to true due to the full join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, true)); + else + /* Just reuse it */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + } + ctx->added_to_joinrel = true; + } + } + + if (innerrel_keeps_unique(root, outerrel, innerrel, clause_list, false)) + { + bool outer_impact = jointype == JOIN_FULL || jointype == JOIN_LEFT;; + + foreach(lc, innerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + if (inner_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (inner_onerow) + { + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, innerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Need to change multi_nullvals to true due to the outer join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, + true)); + else + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + + } + ctx->added_to_joinrel = true; + } + } + + /* + * The combination of the UniqueKey from both sides is unique as well regardless + * of join type, but no bother to add it if its subset has been added to joinrel + * already or it is not useful for the joinrel. + */ + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx1 = (UniqueKeyContext) lfirst(lc); + if (ctx1->added_to_joinrel || !ctx1->useful) + continue; + foreach(lc2, innerrel_ukey_ctx) + { + UniqueKeyContext ctx2 = (UniqueKeyContext) lfirst(lc2); + if (ctx2->added_to_joinrel || !ctx2->useful) + continue; + if (add_combined_uniquekey(root, joinrel, outerrel, innerrel, + ctx1->uniquekey, ctx2->uniquekey, + jointype)) + /* If we set a onerow UniqueKey to joinrel, we don't need other. */ + return; + } + } +} + + +/* + * convert_subquery_uniquekeys + * + * Covert the UniqueKey in subquery to outer relation. + */ +void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel) +{ + ListCell *lc; + + if (sub_final_rel->uniquekeys == NIL) + return; + + if (relation_is_onerow(sub_final_rel)) + { + add_uniquekey_for_onerow(currel); + return; + } + + Assert(currel->subroot != NULL); + + foreach(lc, sub_final_rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + ListCell *lc; + List *exprs = NIL; + bool ukey_useful = true; + + /* One row case is handled above */ + Assert(ukey->exprs != NIL); + foreach(lc, ukey->exprs) + { + Var *var; + TargetEntry *tle = tlist_member(lfirst(lc), + currel->subroot->processed_tlist); + if (tle == NULL) + { + ukey_useful = false; + break; + } + var = find_var_for_subquery_tle(currel, tle); + if (var == NULL) + { + ukey_useful = false; + break; + } + exprs = lappend(exprs, var); + } + + if (ukey_useful) + currel->uniquekeys = lappend(currel->uniquekeys, + makeUniqueKey(exprs, + ukey->multi_nullvals)); + + } +} + +/* + * innerrel_keeps_unique + * + * Check if Unique key of the innerrel is valid after join. innerrel's UniqueKey + * will be still valid if innerrel's any-column mergeop outrerel's uniquekey + * exists in clause_list. + * + * Note: the clause_list must be a list of mergeable restrictinfo already. + */ +static bool +innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *clause_list, + bool reverse) +{ + ListCell *lc, *lc2, *lc3; + + if (outerrel->uniquekeys == NIL || innerrel->uniquekeys == NIL) + return false; + + /* Check if there is outerrel's uniquekey in mergeable clause. */ + foreach(lc, outerrel->uniquekeys) + { + List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs; + bool clauselist_matchs_all_exprs = true; + foreach(lc2, outer_uq_exprs) + { + Node *outer_uq_expr = lfirst(lc2); + bool find_uq_expr_in_clauselist = false; + foreach(lc3, clause_list) + { + RestrictInfo *rinfo = lfirst_node(RestrictInfo, lc3); + Node *outer_expr; + if (reverse) + outer_expr = rinfo->outer_is_left ? get_rightop(rinfo->clause) : get_leftop(rinfo->clause); + else + outer_expr = rinfo->outer_is_left ? get_leftop(rinfo->clause) : get_rightop(rinfo->clause); + if (equal(outer_expr, outer_uq_expr)) + { + find_uq_expr_in_clauselist = true; + break; + } + } + if (!find_uq_expr_in_clauselist) + { + /* No need to check the next exprs in the current uniquekey */ + clauselist_matchs_all_exprs = false; + break; + } + } + + if (clauselist_matchs_all_exprs) + return true; + } + return false; +} + + +/* + * relation_is_onerow + * Check if it is a one-row relation by checking UniqueKey. + */ +bool +relation_is_onerow(RelOptInfo *rel) +{ + UniqueKey *ukey; + if (rel->uniquekeys == NIL) + return false; + ukey = linitial_node(UniqueKey, rel->uniquekeys); + return ukey->exprs == NIL && list_length(rel->uniquekeys) == 1; +} + +/* + * relation_has_uniquekeys_for + * Returns true if we have proofs that 'rel' cannot return multiple rows with + * the same values in each of 'exprs'. Otherwise returns false. + */ +bool +relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel, + List *exprs, bool allow_multinulls) +{ + ListCell *lc; + + /* + * For UniqueKey->onerow case, the uniquekey->exprs is empty as well + * so we can't rely on list_is_subset to handle this special cases + */ + if (exprs == NIL) + return false; + + foreach(lc, rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (ukey->multi_nullvals && !allow_multinulls) + continue; + if (list_is_subset(ukey->exprs, exprs)) + return true; + } + return false; +} + + +/* + * get_exprs_from_uniqueindex + * + * Return a list of exprs which is unique. set useful to false if this + * unique index is not useful for us. + */ +static List * +get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals) +{ + List *exprs = NIL; + ListCell *indexpr_item; + int c = 0; + + *useful = true; + *multi_nullvals = false; + + indexpr_item = list_head(unique_index->indexprs); + for(c = 0; c < unique_index->ncolumns; c++) + { + int attr = unique_index->indexkeys[c]; + Expr *expr; + bool matched_const = false; + ListCell *lc1, *lc2; + + if(attr > 0) + { + expr = list_nth_node(TargetEntry, unique_index->indextlist, c)->expr; + } + else if (attr == 0) + { + /* Expression index */ + expr = lfirst(indexpr_item); + indexpr_item = lnext(unique_index->indexprs, indexpr_item); + } + else /* attr < 0 */ + { + /* Index on system column is not supported */ + Assert(false); + } + + /* + * Check index_col = Const case with regarding to opfamily checking + * If we can remove the index_col from the final UniqueKey->exprs. + */ + forboth(lc1, const_exprs, lc2, const_expr_opfamilies) + { + if (list_member_oid((List *)lfirst(lc2), unique_index->opfamily[c]) + && match_index_to_operand((Node *) lfirst(lc1), c, unique_index)) + { + matched_const = true; + break; + } + } + + if (matched_const) + continue; + + /* Check if the indexed expr is used in rel */ + if (attr > 0) + { + /* + * Normal Indexed column, if the col is not used, then the index is useless + * for uniquekey. + */ + attr -= FirstLowInvalidHeapAttributeNumber; + + if (!bms_is_member(attr, used_varattrs)) + { + *useful = false; + break; + } + } + else if (!list_member(unique_index->rel->reltarget->exprs, expr)) + { + /* Expression index but the expression is not used in rel */ + *useful = false; + break; + } + + /* check not null property. */ + if (attr == 0) + { + /* We never know if a expression yields null or not */ + *multi_nullvals = true; + } + else if (!bms_is_member(attr, unique_index->rel->notnullattrs) + && !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber, + unique_index->rel->notnullattrs)) + { + *multi_nullvals = true; + } + + exprs = lappend(exprs, expr); + } + return exprs; +} + + +/* + * add_uniquekey_for_onerow + * If we are sure that the relation only returns one row, then all the columns + * are unique. However we don't need to create UniqueKey for every column, we + * just set exprs = NIL and overwrites all the other UniqueKey on this RelOptInfo + * since this one has strongest semantics. + */ +void +add_uniquekey_for_onerow(RelOptInfo *rel) +{ + /* + * We overwrite the previous UniqueKey on purpose since this one has the + * strongest semantic. + */ + rel->uniquekeys = list_make1(makeUniqueKey(NIL, false)); +} + + +/* + * initililze_uniquecontext_for_joinrel + * Return a List of UniqueKeyContext for an inputrel + */ +static List * +initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel) +{ + List *res = NIL; + ListCell *lc; + foreach(lc, inputrel->uniquekeys) + { + UniqueKeyContext context; + context = palloc(sizeof(struct UniqueKeyContextData)); + context->uniquekey = lfirst_node(UniqueKey, lc); + context->added_to_joinrel = false; + context->useful = true; + res = lappend(res, context); + } + return res; +} + + +/* + * get_exprs_from_uniquekey + * Unify the way of get List of exprs from a one-row UniqueKey or + * normal UniqueKey. for the onerow case, every expr in rel1 is a valid + * UniqueKey. Return a List of exprs. + * + * rel1: The relation which you want to get the exprs. + * ukey: The UniqueKey you want to get the exprs. + */ +static List * +get_exprs_from_uniquekey(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *rel1, UniqueKey *ukey) +{ + ListCell *lc; + bool onerow = rel1 != NULL && relation_is_onerow(rel1); + + List *res = NIL; + Assert(onerow || ukey); + if (onerow) + { + /* Only cares about the exprs still exist in joinrel */ + foreach(lc, joinrel->reltarget->exprs) + { + Bitmapset *relids = pull_varnos(root, lfirst(lc)); + if (bms_is_subset(relids, rel1->relids)) + { + res = lappend(res, list_make1(lfirst(lc))); + } + } + } + else + { + res = list_make1(ukey->exprs); + } + return res; +} + +/* + * Partitioned table Unique Keys. + * The partition table unique key is maintained as: + * 1. The index must be unique as usual. + * 2. The index must contains partition key. + * 3. The index must exist on all the child rel. see simple_indexinfo_equal for + * how we compare it. + */ + +/* + * index_constains_partkey + * return true if the index contains the partiton key. + */ +static bool +index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind) +{ + ListCell *lc; + int i; + Assert(IS_PARTITIONED_REL(partrel)); + Assert(partrel->part_scheme->partnatts > 0); + + for(i = 0; i < partrel->part_scheme->partnatts; i++) + { + Node *part_expr = linitial(partrel->partexprs[i]); + bool found_in_index = false; + foreach(lc, ind->indextlist) + { + Expr *index_expr = lfirst_node(TargetEntry, lc)->expr; + if (equal(index_expr, part_expr)) + { + found_in_index = true; + break; + } + } + if (!found_in_index) + return false; + } + return true; +} + +/* + * simple_indexinfo_equal + * + * Used to check if the 2 index is same as each other. The index here + * is COPIED from childrel and did some tiny changes(see + * simple_copy_indexinfo_to_parent) + */ +static bool +simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2) +{ + Size oid_cmp_len = sizeof(Oid) * ind1->ncolumns; + + return ind1->ncolumns == ind2->ncolumns && + ind1->unique == ind2->unique && + memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0 && + memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0 && + memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0 && + memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0 && + equal(get_tlist_exprs(ind1->indextlist, true), + get_tlist_exprs(ind2->indextlist, true)); +} + + +/* + * The below macros are used for simple_copy_indexinfo_to_parent which is so + * customized that I don't want to put it to copyfuncs.c. So copy it here. + */ +#define COPY_POINTER_FIELD(fldname, sz) \ + do { \ + Size _size = (sz); \ + newnode->fldname = palloc(_size); \ + memcpy(newnode->fldname, from->fldname, _size); \ + } while (0) + +#define COPY_NODE_FIELD(fldname) \ + (newnode->fldname = copyObjectImpl(from->fldname)) + +#define COPY_SCALAR_FIELD(fldname) \ + (newnode->fldname = from->fldname) + + +/* + * simple_copy_indexinfo_to_parent (from partition) + * Copy the IndexInfo from child relation to parent relation with some modification, + * which is used to test: + * 1. If the same index exists in all the childrels. + * 2. If the parentrel->reltarget/basicrestrict info matches this index. + */ +static IndexOptInfo * +simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from) +{ + IndexOptInfo *newnode = makeNode(IndexOptInfo); + AppendRelInfo *appinfo = find_appinfo_by_child(root, from->rel->relid); + ListCell *lc; + int idx = 0; + + COPY_SCALAR_FIELD(ncolumns); + COPY_SCALAR_FIELD(nkeycolumns); + COPY_SCALAR_FIELD(unique); + COPY_SCALAR_FIELD(immediate); + /* We just need to know if it is NIL or not */ + COPY_SCALAR_FIELD(indpred); + COPY_SCALAR_FIELD(predOK); + COPY_POINTER_FIELD(indexkeys, from->ncolumns * sizeof(int)); + COPY_POINTER_FIELD(indexcollations, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opfamily, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opcintype, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(sortopfamily, from->ncolumns * sizeof(Oid)); + COPY_NODE_FIELD(indextlist); + + /* Convert index exprs on child expr to expr on parent */ + foreach(lc, newnode->indextlist) + { + TargetEntry *tle = lfirst_node(TargetEntry, lc); + /* Index on expression is ignored */ + Assert(IsA(tle->expr, Var)); + tle->expr = (Expr *) find_parent_var(appinfo, (Var *) tle->expr); + newnode->indexkeys[idx] = castNode(Var, tle->expr)->varattno; + idx++; + } + newnode->rel = parentrel; + return newnode; +} + +/* + * adjust_partition_unique_indexlist + * + * global_unique_indexes: At the beginning, it contains the copy & modified + * unique index from the first partition. And then check if each index in it still + * exists in the following partitions. If no, remove it. at last, it has an + * index list which exists in all the partitions. + */ +static void +adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_indexes) +{ + ListCell *lc, *lc2; + foreach(lc, *global_unique_indexes) + { + IndexOptInfo *g_ind = lfirst_node(IndexOptInfo, lc); + bool found_in_child = false; + + foreach(lc2, childrel->indexlist) + { + IndexOptInfo *p_ind = lfirst_node(IndexOptInfo, lc2); + IndexOptInfo *p_ind_copy; + if (!p_ind->unique || !p_ind->immediate || + (p_ind->indpred != NIL && !p_ind->predOK)) + continue; + p_ind_copy = simple_copy_indexinfo_to_parent(root, parentrel, p_ind); + if (simple_indexinfo_equal(p_ind_copy, g_ind)) + { + found_in_child = true; + break; + } + } + if (!found_in_child) + /* The index doesn't exist in childrel, remove it from global_unique_indexes */ + *global_unique_indexes = foreach_delete_current(*global_unique_indexes, lc); + } +} + +/* Helper function for groupres/distinctrel */ +static void +add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgroups) +{ + Query *parse = root->parse; + List *exprs; + + /* + * XXX: If there are some vars which is not in current levelsup, the semantic is + * imprecise, should we avoid it or not? levelsup = 1 is just a demo, maybe we need to + * check every level other than 0, if so, looks we have to write another + * pull_var_walker. + */ + List *upper_vars = pull_vars_of_level((Node*)sortgroups, 1); + + if (upper_vars != NIL) + return; + + exprs = get_sortgrouplist_exprs(sortgroups, parse->targetList); + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(exprs, + false /* sortgroupclause can't be multi_nullvals */)); +} + + +/* + * add_combined_uniquekey + * The combination of both UniqueKeys is a valid UniqueKey for joinrel no matter + * the jointype. + */ +bool +add_combined_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype) +{ + + ListCell *lc1, *lc2; + + /* Either side has multi_nullvals or we have outer join, + * the combined UniqueKey has multi_nullvals */ + bool multi_nullvals = outer_ukey->multi_nullvals || + inner_ukey->multi_nullvals || IS_OUTER_JOIN(jointype); + + /* The only case we can get onerow joinrel after join */ + if (relation_is_onerow(outer_rel) + && relation_is_onerow(inner_rel) + && jointype == JOIN_INNER) + { + add_uniquekey_for_onerow(joinrel); + return true; + } + + foreach(lc1, get_exprs_from_uniquekey(root, joinrel, outer_rel, outer_ukey)) + { + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, inner_rel, inner_ukey)) + { + List *exprs = list_concat_copy(lfirst_node(List, lc1), lfirst_node(List, lc2)); + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(exprs, + multi_nullvals)); + } + } + return false; +} diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index 424d25cbd5..8d8e493f5c 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -2387,6 +2387,8 @@ grouping_planner(PlannerInfo *root, bool inheritance_update, add_path(final_rel, path); } + simple_copy_uniquekeys(current_rel, final_rel); + /* * Generate partial paths for final_rel, too, if outer query levels might * be able to make use of them. @@ -3899,6 +3901,8 @@ create_grouping_paths(PlannerInfo *root, } set_cheapest(grouped_rel); + + populate_grouprel_uniquekeys(root, grouped_rel, input_rel); return grouped_rel; } @@ -4621,7 +4625,7 @@ create_window_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(window_rel); - + simple_copy_uniquekeys(input_rel, window_rel); return window_rel; } @@ -4935,7 +4939,7 @@ create_distinct_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(distinct_rel); - + populate_distinctrel_uniquekeys(root, input_rel, distinct_rel); return distinct_rel; } @@ -5196,6 +5200,8 @@ create_ordered_paths(PlannerInfo *root, */ Assert(ordered_rel->pathlist != NIL); + simple_copy_uniquekeys(input_rel, ordered_rel); + return ordered_rel; } @@ -6073,6 +6079,9 @@ adjust_paths_for_srfs(PlannerInfo *root, RelOptInfo *rel, if (list_length(targets) == 1) return; + /* UniqueKey is not valid after handling the SRF. */ + rel->uniquekeys = NIL; + /* * Stack SRF-evaluation nodes atop each path for the rel. * diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c index becdcbb872..b7626545bf 100644 --- a/src/backend/optimizer/prep/prepunion.c +++ b/src/backend/optimizer/prep/prepunion.c @@ -689,6 +689,8 @@ generate_union_paths(SetOperationStmt *op, PlannerInfo *root, /* Undo effects of possibly forcing tuple_fraction to 0 */ root->tuple_fraction = save_fraction; + /* Add the UniqueKeys */ + populate_unionrel_uniquekeys(root, result_rel); return result_rel; } diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c index 86922a273c..6817c9c787 100644 --- a/src/backend/optimizer/util/appendinfo.c +++ b/src/backend/optimizer/util/appendinfo.c @@ -746,3 +746,47 @@ find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos) } return appinfos; } + +/* + * find_appinfo_by_child + * + */ +AppendRelInfo * +find_appinfo_by_child(PlannerInfo *root, Index child_index) +{ + ListCell *lc; + foreach(lc, root->append_rel_list) + { + AppendRelInfo *appinfo = lfirst_node(AppendRelInfo, lc); + if (appinfo->child_relid == child_index) + return appinfo; + } + elog(ERROR, "parent relation cant be found"); + return NULL; +} + +/* + * find_parent_var + * + */ +Var * +find_parent_var(AppendRelInfo *appinfo, Var *child_var) +{ + ListCell *lc; + Var *res = NULL; + Index attno = 1; + foreach(lc, appinfo->translated_vars) + { + Node *child_node = lfirst(lc); + if (equal(child_node, child_var)) + { + res = copyObject(child_var); + res->varattno = attno; + res->varno = appinfo->parent_relid; + } + attno++; + } + if (res == NULL) + elog(ERROR, "parent var can't be found."); + return res; +} diff --git a/src/backend/optimizer/util/inherit.c b/src/backend/optimizer/util/inherit.c index be1c9ddd96..3eec1f4d74 100644 --- a/src/backend/optimizer/util/inherit.c +++ b/src/backend/optimizer/util/inherit.c @@ -736,6 +736,7 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, { Node *onecq = (Node *) lfirst(lc2); bool pseudoconstant; + RestrictInfo *child_rinfo; /* check for pseudoconstant (no Vars or volatile functions) */ pseudoconstant = @@ -747,14 +748,15 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, root->hasPseudoConstantQuals = true; } /* reconstitute RestrictInfo with appropriate properties */ - childquals = lappend(childquals, - make_restrictinfo(root, - (Expr *) onecq, - rinfo->is_pushed_down, - rinfo->outerjoin_delayed, - pseudoconstant, - rinfo->security_level, - NULL, NULL, NULL)); + child_rinfo = make_restrictinfo(root, + (Expr *) onecq, + rinfo->is_pushed_down, + rinfo->outerjoin_delayed, + pseudoconstant, + rinfo->security_level, + NULL, NULL, NULL); + child_rinfo->mergeopfamilies = rinfo->mergeopfamilies; + childquals = lappend(childquals, child_rinfo); /* track minimum security level among child quals */ cq_min_security = Min(cq_min_security, rinfo->security_level); } diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 48a7ebfe45..4fe1824eb0 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -16,6 +16,7 @@ #include "nodes/execnodes.h" #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" extern A_Expr *makeA_Expr(A_Expr_Kind kind, List *name, @@ -106,4 +107,6 @@ extern GroupingSet *makeGroupingSet(GroupingSetKind kind, List *content, int loc extern VacuumRelation *makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols); +extern UniqueKey* makeUniqueKey(List *exprs, bool multi_nullvals); + #endif /* MAKEFUNC_H */ diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index e22df890ef..3320273ac1 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -265,6 +265,7 @@ typedef enum NodeTag T_EquivalenceMember, T_PathKey, T_PathTarget, + T_UniqueKey, T_RestrictInfo, T_IndexClause, T_PlaceHolderVar, diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index 0d61f04d27..80561b1003 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -739,6 +739,7 @@ typedef struct RelOptInfo QualCost baserestrictcost; /* cost of evaluating the above */ Index baserestrict_min_security; /* min security_level found in * baserestrictinfo */ + List *uniquekeys; /* List of UniqueKey */ List *joininfo; /* RestrictInfo structures for join clauses * involving this rel */ bool has_eclass_joins; /* T means joininfo is incomplete */ @@ -1059,6 +1060,28 @@ typedef struct PathKey } PathKey; +/* + * UniqueKey + * + * Represents the unique properties held by a RelOptInfo. + * + * exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL + * is a special case of UniqueKey, which means there is only 1 row in that + * relation. + * multi_nullvals: true means multi null values may exist in these exprs, so the + * uniqueness is not guaranteed in this case. This field is necessary for + * remove_useless_join & reduce_unique_semijoins where we don't mind these + * duplicated NULL values. It is set to true for 2 cases. One is a unique key + * from a unique index but the related column is nullable. The other one is for + * outer join. see populate_joinrel_uniquekeys for detail. + */ +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + /* * PathTarget * @@ -2496,7 +2519,7 @@ typedef enum * * flags indicating what kinds of grouping are possible. * partial_costs_set is true if the agg_partial_costs and agg_final_costs - * have been initialized. + * have been initialized. * agg_partial_costs gives partial aggregation costs. * agg_final_costs gives finalization costs. * target_parallel_safe is true if target is parallel safe. @@ -2526,8 +2549,8 @@ typedef struct * limit_tuples is an estimated bound on the number of output tuples, * or -1 if no LIMIT or couldn't estimate. * count_est and offset_est are the estimated values of the LIMIT and OFFSET - * expressions computed by preprocess_limit() (see comments for - * preprocess_limit() for more information). + * expressions computed by preprocess_limit() (see comments for + * preprocess_limit() for more information). */ typedef struct { diff --git a/src/include/nodes/pg_list.h b/src/include/nodes/pg_list.h index 404e03f132..85c65e910c 100644 --- a/src/include/nodes/pg_list.h +++ b/src/include/nodes/pg_list.h @@ -566,6 +566,16 @@ extern pg_nodiscard List *list_delete_first(List *list); extern pg_nodiscard List *list_delete_last(List *list); extern pg_nodiscard List *list_delete_nth_cell(List *list, int n); extern pg_nodiscard List *list_delete_cell(List *list, ListCell *cell); +extern bool list_is_subset(const List *members, const List *target); + +extern List *list_delete(List *list, void *datum); +extern List *list_delete_ptr(List *list, void *datum); +extern List *list_delete_int(List *list, int datum); +extern List *list_delete_oid(List *list, Oid datum); +extern List *list_delete_first(List *list); +extern List *list_delete_last(List *list); +extern List *list_delete_nth_cell(List *list, int n); +extern List *list_delete_cell(List *list, ListCell *cell); extern List *list_union(const List *list1, const List *list2); extern List *list_union_ptr(const List *list1, const List *list2); diff --git a/src/include/optimizer/appendinfo.h b/src/include/optimizer/appendinfo.h index 4cbf8c26cc..43b8d05f69 100644 --- a/src/include/optimizer/appendinfo.h +++ b/src/include/optimizer/appendinfo.h @@ -32,4 +32,7 @@ extern Relids adjust_child_relids_multilevel(PlannerInfo *root, Relids relids, extern AppendRelInfo **find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos); +extern AppendRelInfo *find_appinfo_by_child(PlannerInfo *root, Index child_index); +extern Var *find_parent_var(AppendRelInfo *appinfo, Var *child_var); + #endif /* APPENDINFO_H */ diff --git a/src/include/optimizer/optimizer.h b/src/include/optimizer/optimizer.h index d587952b7d..843aafa51e 100644 --- a/src/include/optimizer/optimizer.h +++ b/src/include/optimizer/optimizer.h @@ -23,6 +23,7 @@ #define OPTIMIZER_H #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" /* Test if an expression node represents a SRF call. Beware multiple eval! */ #define IS_SRF_CALL(node) \ @@ -168,6 +169,7 @@ extern TargetEntry *get_sortgroupref_tle(Index sortref, List *targetList); extern TargetEntry *get_sortgroupclause_tle(SortGroupClause *sgClause, List *targetList); +extern Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); extern Node *get_sortgroupclause_expr(SortGroupClause *sgClause, List *targetList); extern List *get_sortgrouplist_exprs(List *sgClauses, diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h index 035d3e1206..1adf99c9ee 100644 --- a/src/include/optimizer/paths.h +++ b/src/include/optimizer/paths.h @@ -247,5 +247,48 @@ extern PathKey *make_canonical_pathkey(PlannerInfo *root, int strategy, bool nulls_first); extern void add_paths_to_append_rel(PlannerInfo *root, RelOptInfo *rel, List *live_childrels); +extern List *select_mergejoin_clauses(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + JoinType jointype, + bool *mergejoin_allowed); + +/* + * uniquekeys.c + * Utilities for matching and building unique keys + */ +extern void populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List* unique_index_list); +extern void populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels); +extern void populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel); +extern void populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel); +extern void populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel); +extern void simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel); +extern void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel); +extern void populate_joinrel_uniquekeys(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + RelOptInfo *rel2, + List *restrictlist, + JoinType jointype); + +extern bool relation_has_uniquekeys_for(PlannerInfo *root, + RelOptInfo *rel, + List *exprs, + bool allow_multinulls); +extern bool relation_is_onerow(RelOptInfo *rel); #endif /* PATHS_H */ -- 2.26.2 --yh7cw345mbttadjz Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v38-0003-Extend-UniqueKeys.patch" ^ permalink raw reply [nested|flat] 91+ messages in thread
* [PATCH v36 2/6] Introduce UniqueKey attributes on RelOptInfo struct. @ 2020-05-11 07:50 一挃 <[email protected]> 0 siblings, 0 replies; 91+ messages in thread From: 一挃 @ 2020-05-11 07:50 UTC (permalink / raw) UniqueKey is a set of exprs on RelOptInfo which represents the exprs will be unique on the given RelOptInfo. You can see README.uniquekey for more information. --- src/backend/nodes/copyfuncs.c | 13 + src/backend/nodes/list.c | 31 + src/backend/nodes/makefuncs.c | 13 + src/backend/nodes/outfuncs.c | 11 + src/backend/nodes/readfuncs.c | 10 + src/backend/optimizer/path/Makefile | 3 +- src/backend/optimizer/path/README.uniquekey | 131 +++ src/backend/optimizer/path/allpaths.c | 10 + src/backend/optimizer/path/joinpath.c | 9 +- src/backend/optimizer/path/joinrels.c | 2 + src/backend/optimizer/path/pathkeys.c | 3 +- src/backend/optimizer/path/uniquekeys.c | 1131 +++++++++++++++++++ src/backend/optimizer/plan/planner.c | 13 +- src/backend/optimizer/prep/prepunion.c | 2 + src/backend/optimizer/util/appendinfo.c | 44 + src/backend/optimizer/util/inherit.c | 16 +- src/include/nodes/makefuncs.h | 3 + src/include/nodes/nodes.h | 1 + src/include/nodes/pathnodes.h | 29 +- src/include/nodes/pg_list.h | 2 + src/include/optimizer/appendinfo.h | 3 + src/include/optimizer/optimizer.h | 2 + src/include/optimizer/paths.h | 43 + 23 files changed, 1502 insertions(+), 23 deletions(-) create mode 100644 src/backend/optimizer/path/README.uniquekey create mode 100644 src/backend/optimizer/path/uniquekeys.c diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 89c409de66..1f50400fd2 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2273,6 +2273,16 @@ _copyPathKey(const PathKey *from) return newnode; } +static UniqueKey * +_copyUniqueKey(const UniqueKey *from) +{ + UniqueKey *newnode = makeNode(UniqueKey); + + COPY_NODE_FIELD(exprs); + COPY_SCALAR_FIELD(multi_nullvals); + + return newnode; +} /* * _copyRestrictInfo */ @@ -5152,6 +5162,9 @@ copyObjectImpl(const void *from) case T_PathKey: retval = _copyPathKey(from); break; + case T_UniqueKey: + retval = _copyUniqueKey(from); + break; case T_RestrictInfo: retval = _copyRestrictInfo(from); break; diff --git a/src/backend/nodes/list.c b/src/backend/nodes/list.c index 80fa8c84e4..a7a99b70f2 100644 --- a/src/backend/nodes/list.c +++ b/src/backend/nodes/list.c @@ -687,6 +687,37 @@ list_member_oid(const List *list, Oid datum) return false; } +/* + * return true iff every entry in "members" list is also present + * in the "target" list. + */ +bool +list_is_subset(const List *members, const List *target) +{ + const ListCell *lc1, *lc2; + + Assert(IsPointerList(members)); + Assert(IsPointerList(target)); + check_list_invariants(members); + check_list_invariants(target); + + foreach(lc1, members) + { + bool found = false; + foreach(lc2, target) + { + if (equal(lfirst(lc1), lfirst(lc2))) + { + found = true; + break; + } + } + if (!found) + return false; + } + return true; +} + /* * Delete the n'th cell (counting from 0) in list. * diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 49de285f01..646cf7c9a1 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -814,3 +814,16 @@ makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols) v->va_cols = va_cols; return v; } + + +/* + * makeUniqueKey + */ +UniqueKey* +makeUniqueKey(List *exprs, bool multi_nullvals) +{ + UniqueKey * ukey = makeNode(UniqueKey); + ukey->exprs = exprs; + ukey->multi_nullvals = multi_nullvals; + return ukey; +} diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index e2f177515d..c3a9632992 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2428,6 +2428,14 @@ _outPathKey(StringInfo str, const PathKey *node) WRITE_BOOL_FIELD(pk_nulls_first); } +static void +_outUniqueKey(StringInfo str, const UniqueKey *node) +{ + WRITE_NODE_TYPE("UNIQUEKEY"); + WRITE_NODE_FIELD(exprs); + WRITE_BOOL_FIELD(multi_nullvals); +} + static void _outPathTarget(StringInfo str, const PathTarget *node) { @@ -4127,6 +4135,9 @@ outNode(StringInfo str, const void *obj) case T_PathKey: _outPathKey(str, obj); break; + case T_UniqueKey: + _outUniqueKey(str, obj); + break; case T_PathTarget: _outPathTarget(str, obj); break; diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index 42050ab719..3a18571d0c 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -452,6 +452,14 @@ _readSetOperationStmt(void) READ_DONE(); } +static UniqueKey * +_readUniqueKey(void) +{ + READ_LOCALS(UniqueKey); + READ_NODE_FIELD(exprs); + READ_BOOL_FIELD(multi_nullvals); + READ_DONE(); +} /* * Stuff from primnodes.h. @@ -2656,6 +2664,8 @@ parseNodeString(void) return_value = _readCommonTableExpr(); else if (MATCH("SETOPERATIONSTMT", 16)) return_value = _readSetOperationStmt(); + else if (MATCH("UNIQUEKEY", 9)) + return_value = _readUniqueKey(); else if (MATCH("ALIAS", 5)) return_value = _readAlias(); else if (MATCH("RANGEVAR", 8)) diff --git a/src/backend/optimizer/path/Makefile b/src/backend/optimizer/path/Makefile index 1e199ff66f..7b9820c25f 100644 --- a/src/backend/optimizer/path/Makefile +++ b/src/backend/optimizer/path/Makefile @@ -21,6 +21,7 @@ OBJS = \ joinpath.o \ joinrels.o \ pathkeys.o \ - tidpath.o + tidpath.o \ + uniquekeys.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/optimizer/path/README.uniquekey b/src/backend/optimizer/path/README.uniquekey new file mode 100644 index 0000000000..5eac761995 --- /dev/null +++ b/src/backend/optimizer/path/README.uniquekey @@ -0,0 +1,131 @@ +1. What is UniqueKey? +We can think UniqueKey is a set of exprs for a RelOptInfo, which we are insure +that doesn't yields same result among all the rows. The simplest UniqueKey +format is primary key. + +However we define the UnqiueKey as below. + +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + +exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL +is a special case of UniqueKey, which means there is only one row in that +relation.it has a stronger semantic than others. like SELECT uk FROM t; uk is +normal unique key and may have different values. SELECT colx FROM t WHERE uk = +const. colx is unique AND we have only 1 value. This field can used for +innerrel_is_unique. this logic is handled specially in add_uniquekey_for_onerow +function. + +multi_nullvals: true means multi null values may exist in these exprs, so the +uniqueness is not guaranteed in this case. This field is necessary for +remove_useless_join & reduce_unique_semijoins where we don't mind these +duplicated NULL values. It is set to true for 2 cases. One is a unique key +from a unique index but the related column is nullable. The other one is for +outer join. see populate_joinrel_uniquekeys for detail. + + +The UniqueKey can be used at the following cases at least: +1. remove_useless_joins. +2. reduce_semianti_joins +3. remove distinct node if distinct clause is unique. +4. remove aggnode if group by clause is unique. +5. Index Skip Scan (WIP) +6. Aggregation Push Down without 2 phase aggregation if the join can't + duplicated the aggregated rows. (work in progress feature) + +2. How is it maintained? + +We have a set of populate_xxx_unqiuekeys functions to maintain the uniquekey on +various cases. xxx includes baserel, joinrel, partitionedrel, distinctrel, +groupedrel, unionrel. and we also need to convert the uniquekey from subquery +to outer relation, which is what convert_subquery_uniquekeys does. + +1. The first part is about baserel. We handled 3 cases. suppose we have Unique +Index on (a, b). + +1. SELECT a, b FROM t. UniqueKey (a, b) +2. SELECT a FROM t WHERE b = 1; UniqueKey (a) +3. SELECT .. FROM t WHERE a = 1 AND b = 1; UniqueKey (NIL). onerow case, every + column is Unique. + +2. The next part is joinrel, this part is most error-prone, we simplified the rules +like below: +1. If the relation's UniqueKey can't be duplicated after join, then is will be + still valid for the join rel. The function we used here is + innerrel_keeps_unique. The basic idea is innerrel.any_col = outer.uk. + +2. If the UnqiueKey can't keep valid via the rule 1, the combination of the + UniqueKey from both sides are valid for sure. We can prove this as: if the + unique exprs from rel1 is duplicated by rel2, the duplicated rows must + contains different unique exprs from rel2. + +More considerations about onerow: +1. If relation with one row and it can't be duplicated, it is still possible + contains mulit_nullvas after outer join. +2. If the either UniqueKey can be duplicated after join, the can get one row + only when both side is one row AND there is no outer join. +3. Whenever the onerow UniqueKey is not a valid any more, we need to convert one + row UniqueKey to normal unique key since we don't store exprs for one-row + relation. get_exprs_from_uniquekeys will be used here. + + +More considerations about multi_nullvals after join: +1. If the original UnqiueKey has multi_nullvals, the final UniqueKey will have + mulit_nullvals in any case. +2. If a unique key doesn't allow mulit_nullvals, after some outer join, it + allows some outer join. + + +3. When we comes to subquery, we need to convert_subquery_unqiuekeys just like +convert_subquery_pathkeys. Only the UniqueKey insides subquery is referenced as +a Var in outer relation will be reused. The relationship between the outerrel.Var +and subquery.exprs is built with outerel->subroot->processed_tlist. + + +4. As for the SRF functions, it will break the uniqueness of uniquekey, However it +is handled in adjust_paths_for_srfs, which happens after the query_planner. so +we will maintain the UniqueKey until there and reset it to NIL at that +places. This can't help on distinct/group by elimination cases but probably help +in some other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is +semantic correctly. + + +5. As for inherit table, we first main the UnqiueKey on childrel as well. But for +partitioned table we need to maintain 2 different kinds of +UnqiueKey. 1). UniqueKey on the parent relation 2). UniqueKey on child +relation for partition wise query. + +Example: +CREATE TABLE p (a int not null, b int not null) partition by list (a); +CREATE TABLE p0 partition of p for values in (1); +CREATE TABLE p1 partition of p for values in (2); + +create unique index p0_b on p0(b); +create unique index p1_b on p1(b); + +Now b is only unique on partition level, so the distinct can't be removed on +the following cases. SELECT DISTINCT b FROM p; + +Another example is SELECT DISTINCT a, b FROM p WHERE a = 1; Since only one +partition is chosen, the UniqueKey on child relation is same as the UniqueKey on +parent relation. + +Another usage of UniqueKey on partition level is it be helpful for +partition-wise join. + +As for the UniqueKey on parent table level, it comes with 2 different ways, +1). the UniqueKey is also derived in UniqueKey index, but the index must be same +in all the related children relations and the unique index must contains +Partition Key in it. Example: + +CREATE UNIQUE INDEX p_ab ON p(a, b); -- where a is the partition key. + +-- Query +SELECT a, b FROM p; the (a, b) is a UniqueKey of p. + +2). If the parent relation has only one childrel, the UniqueKey on childrel is + the UniqueKey on parent as well. diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index 484dab0a1a..2ad9d06d7a 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -579,6 +579,12 @@ set_plain_rel_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) */ check_index_predicates(root, rel); + /* + * Now that we've marked which partial indexes are suitable, we can now + * build the relation's unique keys. + */ + populate_baserel_uniquekeys(root, rel, rel->indexlist); + /* Mark rel with estimated output rows, width, etc */ set_baserel_size_estimates(root, rel); } @@ -1310,6 +1316,8 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, /* Add paths to the append relation. */ add_paths_to_append_rel(root, rel, live_childrels); + if (IS_PARTITIONED_REL(rel)) + populate_partitionedrel_uniquekeys(root, rel, live_childrels); } @@ -2383,6 +2391,8 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, pathkeys, required_outer)); } + convert_subquery_uniquekeys(root, rel, sub_final_rel); + /* If outer rel allows parallelism, do same for partial paths. */ if (rel->consider_parallel && bms_is_empty(required_outer)) { diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c index db54a6ba2e..ef0fd2fb0b 100644 --- a/src/backend/optimizer/path/joinpath.c +++ b/src/backend/optimizer/path/joinpath.c @@ -71,13 +71,6 @@ static void consider_parallel_mergejoin(PlannerInfo *root, static void hash_inner_and_outer(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, RelOptInfo *innerrel, JoinType jointype, JoinPathExtraData *extra); -static List *select_mergejoin_clauses(PlannerInfo *root, - RelOptInfo *joinrel, - RelOptInfo *outerrel, - RelOptInfo *innerrel, - List *restrictlist, - JoinType jointype, - bool *mergejoin_allowed); static void generate_mergejoin_paths(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *innerrel, @@ -1927,7 +1920,7 @@ hash_inner_and_outer(PlannerInfo *root, * if it is mergejoinable and involves vars from the two sub-relations * currently of interest. */ -static List * +List * select_mergejoin_clauses(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c index 2d343cd293..b9163ee8ff 100644 --- a/src/backend/optimizer/path/joinrels.c +++ b/src/backend/optimizer/path/joinrels.c @@ -924,6 +924,8 @@ populate_joinrel_with_paths(PlannerInfo *root, RelOptInfo *rel1, /* Apply partitionwise join technique, if possible. */ try_partitionwise_join(root, rel1, rel2, joinrel, sjinfo, restrictlist); + + populate_joinrel_uniquekeys(root, joinrel, rel1, rel2, restrictlist, sjinfo->jointype); } diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index ce9bf87e9b..7e596d4194 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -33,7 +33,6 @@ static bool pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys); static bool matches_boolean_partition_clause(RestrictInfo *rinfo, RelOptInfo *partrel, int partkeycol); -static Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); static bool right_merge_direction(PlannerInfo *root, PathKey *pathkey); @@ -1035,7 +1034,7 @@ convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel, * We need this to ensure that we don't return pathkeys describing values * that are unavailable above the level of the subquery scan. */ -static Var * +Var * find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle) { ListCell *lc; diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c new file mode 100644 index 0000000000..b33bcd2f32 --- /dev/null +++ b/src/backend/optimizer/path/uniquekeys.c @@ -0,0 +1,1131 @@ +/*------------------------------------------------------------------------- + * + * uniquekeys.c + * Utilities for matching and building unique keys + * + * Portions Copyright (c) 2020, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/optimizer/path/uniquekeys.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "nodes/makefuncs.h" +#include "nodes/nodeFuncs.h" +#include "optimizer/pathnode.h" +#include "optimizer/paths.h" +#include "optimizer/appendinfo.h" +#include "optimizer/optimizer.h" +#include "optimizer/tlist.h" +#include "rewrite/rewriteManip.h" + + +/* + * This struct is used to help populate_joinrel_uniquekeys. + * + * added_to_joinrel is true if a uniquekey (from outerrel or innerrel) + * has been added to joinrel. + * useful is true if the exprs of the uniquekey still appears in joinrel. + */ +typedef struct UniqueKeyContextData +{ + UniqueKey *uniquekey; + bool added_to_joinrel; + bool useful; +} *UniqueKeyContext; + +static List *initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel); +static bool innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + bool reverse); + +static List *get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals); +static List *get_exprs_from_uniquekey(RelOptInfo *joinrel, + RelOptInfo *rel1, + UniqueKey *ukey); +static void add_uniquekey_for_onerow(RelOptInfo *rel); +static bool add_combined_uniquekey(RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype); + +/* Used for unique indexes checking for partitioned table */ +static bool index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind); +static IndexOptInfo *simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from); +static bool simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2); +static void adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_index); + +/* Helper function for grouped relation and distinct relation. */ +static void add_uniquekey_from_sortgroups(PlannerInfo *root, + RelOptInfo *rel, + List *sortgroups); + +/* + * populate_baserel_uniquekeys + * Populate 'baserel' uniquekeys list by looking at the rel's unique index + * and baserestrictinfo + */ +void +populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List *indexlist) +{ + ListCell *lc; + List *matched_uniq_indexes = NIL; + + /* Attrs appears in rel->reltarget->exprs. */ + Bitmapset *used_attrs = NULL; + + List *const_exprs = NIL; + List *expr_opfamilies = NIL; + + Assert(baserel->rtekind == RTE_RELATION); + + foreach(lc, indexlist) + { + IndexOptInfo *ind = (IndexOptInfo *) lfirst(lc); + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + matched_uniq_indexes = lappend(matched_uniq_indexes, ind); + } + + if (matched_uniq_indexes == NIL) + return; + + /* Check which attrs is used in baserel->reltarget */ + pull_varattnos((Node *)baserel->reltarget->exprs, baserel->relid, &used_attrs); + + /* Check which attrno is used at a mergeable const filter */ + foreach(lc, baserel->baserestrictinfo) + { + RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); + + if (rinfo->mergeopfamilies == NIL) + continue; + + if (bms_is_empty(rinfo->left_relids)) + { + const_exprs = lappend(const_exprs, get_rightop(rinfo->clause)); + } + else if (bms_is_empty(rinfo->right_relids)) + { + const_exprs = lappend(const_exprs, get_leftop(rinfo->clause)); + } + else + continue; + + expr_opfamilies = lappend(expr_opfamilies, rinfo->mergeopfamilies); + } + + foreach(lc, matched_uniq_indexes) + { + bool multi_nullvals, useful; + List *exprs = get_exprs_from_uniqueindex(lfirst_node(IndexOptInfo, lc), + const_exprs, + expr_opfamilies, + used_attrs, + &useful, + &multi_nullvals); + if (useful) + { + if (exprs == NIL) + { + /* All the columns in Unique Index matched with a restrictinfo */ + add_uniquekey_for_onerow(baserel); + return; + } + baserel->uniquekeys = lappend(baserel->uniquekeys, + makeUniqueKey(exprs, multi_nullvals)); + } + } +} + + +/* + * populate_partitionedrel_uniquekeys + * The UniqueKey on partitionrel comes from 2 cases: + * 1). Only one partition is involved in this query, the unique key can be + * copied to parent rel from childrel. + * 2). There are some unique index which includes partition key and exists + * in all the related partitions. + * We never mind rule 2 if we hit rule 1. + */ + +void +populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels) +{ + ListCell *lc; + List *global_uniq_indexlist = NIL; + RelOptInfo *childrel; + bool is_first = true; + + Assert(IS_PARTITIONED_REL(rel)); + + if (childrels == NIL) + return; + + /* + * If there is only one partition used in this query, the UniqueKey in childrel is + * still valid in parent level, but we need convert the format from child expr to + * parent expr. + */ + if (list_length(childrels) == 1) + { + /* Check for Rule 1 */ + RelOptInfo *childrel = linitial_node(RelOptInfo, childrels); + ListCell *lc; + Assert(childrel->reloptkind == RELOPT_OTHER_MEMBER_REL); + if (relation_is_onerow(childrel)) + { + add_uniquekey_for_onerow(rel); + return; + } + + foreach(lc, childrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + AppendRelInfo *appinfo = find_appinfo_by_child(root, childrel->relid); + List *parent_exprs = NIL; + bool can_reuse = true; + ListCell *lc2; + foreach(lc2, ukey->exprs) + { + Var *var = (Var *)lfirst(lc2); + /* + * If the expr comes from a expression, it is hard to build the expression + * in parent so ignore that case for now. + */ + if(!IsA(var, Var)) + { + can_reuse = false; + break; + } + /* Convert it to parent var */ + parent_exprs = lappend(parent_exprs, find_parent_var(appinfo, var)); + } + if (can_reuse) + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(parent_exprs, + ukey->multi_nullvals)); + } + } + else + { + /* Check for rule 2 */ + childrel = linitial_node(RelOptInfo, childrels); + foreach(lc, childrel->indexlist) + { + IndexOptInfo *ind = lfirst(lc); + IndexOptInfo *modified_index; + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + + /* + * During simple_copy_indexinfo_to_parent, we need to convert var from + * child var to parent var, index on expression is too complex to handle. + * so ignore it for now. + */ + if (ind->indexprs != NIL) + continue; + + modified_index = simple_copy_indexinfo_to_parent(root, rel, ind); + /* + * If the unique index doesn't contain partkey, then it is unique + * on this partition only, so it is useless for us. + */ + if (!index_constains_partkey(rel, modified_index)) + continue; + + global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index); + } + + if (global_uniq_indexlist != NIL) + { + foreach(lc, childrels) + { + RelOptInfo *child = lfirst(lc); + if (is_first) + { + is_first = false; + continue; + } + adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist); + } + /* Now we have a list of unique index which are exactly same on all childrels, + * Set the UniqueKey just like it is non-partition table + */ + populate_baserel_uniquekeys(root, rel, global_uniq_indexlist); + } + } +} + + +/* + * populate_distinctrel_uniquekeys + */ +void +populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel) +{ + /* The unique key before the distinct is still valid. */ + distinctrel->uniquekeys = list_copy(inputrel->uniquekeys); + add_uniquekey_from_sortgroups(root, distinctrel, root->parse->distinctClause); +} + +/* + * populate_grouprel_uniquekeys + */ +void +populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel) + +{ + Query *parse = root->parse; + bool input_ukey_added = false; + ListCell *lc; + + if (relation_is_onerow(inputrel)) + { + add_uniquekey_for_onerow(grouprel); + return; + } + if (parse->groupingSets) + return; + + /* A Normal group by without grouping set. */ + if (parse->groupClause) + { + /* + * Current even the groupby clause is Unique already, but if query has aggref + * We have to create grouprel still. To keep the UnqiueKey short, we will check + * the UniqueKey of input_rel still valid, if so we reuse it. + */ + foreach(lc, inputrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (list_is_subset(ukey->exprs, grouprel->reltarget->exprs)) + { + grouprel->uniquekeys = lappend(grouprel->uniquekeys, + ukey); + input_ukey_added = true; + } + } + if (!input_ukey_added) + /* + * group by clause must be a super-set of grouprel->reltarget->exprs except the + * aggregation expr, so if such exprs is unique already, no bother to generate + * new uniquekey for group by exprs. + */ + add_uniquekey_from_sortgroups(root, + grouprel, + root->parse->groupClause); + } + else + /* It has aggregation but without a group by, so only one row returned */ + add_uniquekey_for_onerow(grouprel); +} + +/* + * simple_copy_uniquekeys + * Using a function for the one-line code makes us easy to check where we simply + * copied the uniquekey. + */ +void +simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel) +{ + newrel->uniquekeys = oldrel->uniquekeys; +} + +/* + * populate_unionrel_uniquekeys + */ +void +populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel) +{ + ListCell *lc; + List *exprs = NIL; + + Assert(unionrel->uniquekeys == NIL); + + foreach(lc, unionrel->reltarget->exprs) + { + exprs = lappend(exprs, lfirst(lc)); + } + + if (exprs == NIL) + /* SQL: select union select; is valid, we need to handle it here. */ + add_uniquekey_for_onerow(unionrel); + else + unionrel->uniquekeys = lappend(unionrel->uniquekeys, + makeUniqueKey(exprs,false)); + +} + +/* + * populate_joinrel_uniquekeys + * + * populate uniquekeys for joinrel. We will check each relation to see if its + * UniqueKey is still valid via innerrel_keeps_unique, if so, we add it to + * joinrel. The multi_nullvals field will be changed to true for some outer + * join cases and one-row UniqueKey needs to be converted to normal UniqueKey + * for the same case as well. + * For the uniquekey in either baserel which can't be unique after join, we still + * check to see if combination of UniqueKeys from both side is still useful for us. + * if yes, we add it to joinrel as well. + */ +void +populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *outerrel, RelOptInfo *innerrel, + List *restrictlist, JoinType jointype) +{ + ListCell *lc, *lc2; + List *clause_list = NIL; + List *outerrel_ukey_ctx; + List *innerrel_ukey_ctx; + bool inner_onerow, outer_onerow; + bool mergejoin_allowed; + + /* Care about the outerrel relation only for SEMI/ANTI join */ + if (jointype == JOIN_SEMI || jointype == JOIN_ANTI) + { + foreach(lc, outerrel->uniquekeys) + { + UniqueKey *uniquekey = lfirst_node(UniqueKey, lc); + if (list_is_subset(uniquekey->exprs, joinrel->reltarget->exprs)) + joinrel->uniquekeys = lappend(joinrel->uniquekeys, uniquekey); + } + return; + } + + Assert(jointype == JOIN_LEFT || jointype == JOIN_FULL || jointype == JOIN_INNER); + + /* Fast path */ + if (innerrel->uniquekeys == NIL || outerrel->uniquekeys == NIL) + return; + + inner_onerow = relation_is_onerow(innerrel); + outer_onerow = relation_is_onerow(outerrel); + + outerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(outerrel); + innerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(innerrel); + + clause_list = select_mergejoin_clauses(root, joinrel, outerrel, innerrel, + restrictlist, jointype, + &mergejoin_allowed); + + if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true /* reverse */)) + { + bool outer_impact = jointype == JOIN_FULL; + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + /* Outer relation has one row, and the unique key is not duplicated after join, + * the joinrel will still has one row unless the jointype == JOIN_FULL. + */ + if (outer_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (outer_onerow) + { + /* + * The onerow outerrel becomes multi rows and multi_nullvals + * will be changed to true. We also need to set the exprs correctly since it + * can't be NIL any more. + */ + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(joinrel, outerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Change multi_nullvals to true due to the full join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, true)); + else + /* Just reuse it */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + } + ctx->added_to_joinrel = true; + } + } + + if (innerrel_keeps_unique(root, outerrel, innerrel, clause_list, false)) + { + bool outer_impact = jointype == JOIN_FULL || jointype == JOIN_LEFT;; + + foreach(lc, innerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + if (inner_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (inner_onerow) + { + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(joinrel, innerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Need to change multi_nullvals to true due to the outer join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, + true)); + else + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + + } + ctx->added_to_joinrel = true; + } + } + + /* + * The combination of the UniqueKey from both sides is unique as well regardless + * of join type, but no bother to add it if its subset has been added to joinrel + * already or it is not useful for the joinrel. + */ + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx1 = (UniqueKeyContext) lfirst(lc); + if (ctx1->added_to_joinrel || !ctx1->useful) + continue; + foreach(lc2, innerrel_ukey_ctx) + { + UniqueKeyContext ctx2 = (UniqueKeyContext) lfirst(lc2); + if (ctx2->added_to_joinrel || !ctx2->useful) + continue; + if (add_combined_uniquekey(joinrel, outerrel, innerrel, + ctx1->uniquekey, ctx2->uniquekey, + jointype)) + /* If we set a onerow UniqueKey to joinrel, we don't need other. */ + return; + } + } +} + + +/* + * convert_subquery_uniquekeys + * + * Covert the UniqueKey in subquery to outer relation. + */ +void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel) +{ + ListCell *lc; + + if (sub_final_rel->uniquekeys == NIL) + return; + + if (relation_is_onerow(sub_final_rel)) + { + add_uniquekey_for_onerow(currel); + return; + } + + Assert(currel->subroot != NULL); + + foreach(lc, sub_final_rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + ListCell *lc; + List *exprs = NIL; + bool ukey_useful = true; + + /* One row case is handled above */ + Assert(ukey->exprs != NIL); + foreach(lc, ukey->exprs) + { + Var *var; + TargetEntry *tle = tlist_member(lfirst(lc), + currel->subroot->processed_tlist); + if (tle == NULL) + { + ukey_useful = false; + break; + } + var = find_var_for_subquery_tle(currel, tle); + if (var == NULL) + { + ukey_useful = false; + break; + } + exprs = lappend(exprs, var); + } + + if (ukey_useful) + currel->uniquekeys = lappend(currel->uniquekeys, + makeUniqueKey(exprs, + ukey->multi_nullvals)); + + } +} + +/* + * innerrel_keeps_unique + * + * Check if Unique key of the innerrel is valid after join. innerrel's UniqueKey + * will be still valid if innerrel's any-column mergeop outrerel's uniquekey + * exists in clause_list. + * + * Note: the clause_list must be a list of mergeable restrictinfo already. + */ +static bool +innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *clause_list, + bool reverse) +{ + ListCell *lc, *lc2, *lc3; + + if (outerrel->uniquekeys == NIL || innerrel->uniquekeys == NIL) + return false; + + /* Check if there is outerrel's uniquekey in mergeable clause. */ + foreach(lc, outerrel->uniquekeys) + { + List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs; + bool clauselist_matchs_all_exprs = true; + foreach(lc2, outer_uq_exprs) + { + Node *outer_uq_expr = lfirst(lc2); + bool find_uq_expr_in_clauselist = false; + foreach(lc3, clause_list) + { + RestrictInfo *rinfo = lfirst_node(RestrictInfo, lc3); + Node *outer_expr; + if (reverse) + outer_expr = rinfo->outer_is_left ? get_rightop(rinfo->clause) : get_leftop(rinfo->clause); + else + outer_expr = rinfo->outer_is_left ? get_leftop(rinfo->clause) : get_rightop(rinfo->clause); + if (equal(outer_expr, outer_uq_expr)) + { + find_uq_expr_in_clauselist = true; + break; + } + } + if (!find_uq_expr_in_clauselist) + { + /* No need to check the next exprs in the current uniquekey */ + clauselist_matchs_all_exprs = false; + break; + } + } + + if (clauselist_matchs_all_exprs) + return true; + } + return false; +} + + +/* + * relation_is_onerow + * Check if it is a one-row relation by checking UniqueKey. + */ +bool +relation_is_onerow(RelOptInfo *rel) +{ + UniqueKey *ukey; + if (rel->uniquekeys == NIL) + return false; + ukey = linitial_node(UniqueKey, rel->uniquekeys); + return ukey->exprs == NIL && list_length(rel->uniquekeys) == 1; +} + +/* + * relation_has_uniquekeys_for + * Returns true if we have proofs that 'rel' cannot return multiple rows with + * the same values in each of 'exprs'. Otherwise returns false. + */ +bool +relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel, + List *exprs, bool allow_multinulls) +{ + ListCell *lc; + + /* + * For UniqueKey->onerow case, the uniquekey->exprs is empty as well + * so we can't rely on list_is_subset to handle this special cases + */ + if (exprs == NIL) + return false; + + foreach(lc, rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (ukey->multi_nullvals && !allow_multinulls) + continue; + if (list_is_subset(ukey->exprs, exprs)) + return true; + } + return false; +} + + +/* + * get_exprs_from_uniqueindex + * + * Return a list of exprs which is unique. set useful to false if this + * unique index is not useful for us. + */ +static List * +get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals) +{ + List *exprs = NIL; + ListCell *indexpr_item; + int c = 0; + + *useful = true; + *multi_nullvals = false; + + indexpr_item = list_head(unique_index->indexprs); + for(c = 0; c < unique_index->ncolumns; c++) + { + int attr = unique_index->indexkeys[c]; + Expr *expr; + bool matched_const = false; + ListCell *lc1, *lc2; + + if(attr > 0) + { + expr = list_nth_node(TargetEntry, unique_index->indextlist, c)->expr; + } + else if (attr == 0) + { + /* Expression index */ + expr = lfirst(indexpr_item); + indexpr_item = lnext(unique_index->indexprs, indexpr_item); + } + else /* attr < 0 */ + { + /* Index on system column is not supported */ + Assert(false); + } + + /* + * Check index_col = Const case with regarding to opfamily checking + * If we can remove the index_col from the final UniqueKey->exprs. + */ + forboth(lc1, const_exprs, lc2, const_expr_opfamilies) + { + if (list_member_oid((List *)lfirst(lc2), unique_index->opfamily[c]) + && match_index_to_operand((Node *) lfirst(lc1), c, unique_index)) + { + matched_const = true; + break; + } + } + + if (matched_const) + continue; + + /* Check if the indexed expr is used in rel */ + if (attr > 0) + { + /* + * Normal Indexed column, if the col is not used, then the index is useless + * for uniquekey. + */ + attr -= FirstLowInvalidHeapAttributeNumber; + + if (!bms_is_member(attr, used_varattrs)) + { + *useful = false; + break; + } + } + else if (!list_member(unique_index->rel->reltarget->exprs, expr)) + { + /* Expression index but the expression is not used in rel */ + *useful = false; + break; + } + + /* check not null property. */ + if (attr == 0) + { + /* We never know if a expression yields null or not */ + *multi_nullvals = true; + } + else if (!bms_is_member(attr, unique_index->rel->notnullattrs) + && !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber, + unique_index->rel->notnullattrs)) + { + *multi_nullvals = true; + } + + exprs = lappend(exprs, expr); + } + return exprs; +} + + +/* + * add_uniquekey_for_onerow + * If we are sure that the relation only returns one row, then all the columns + * are unique. However we don't need to create UniqueKey for every column, we + * just set exprs = NIL and overwrites all the other UniqueKey on this RelOptInfo + * since this one has strongest semantics. + */ +void +add_uniquekey_for_onerow(RelOptInfo *rel) +{ + /* + * We overwrite the previous UniqueKey on purpose since this one has the + * strongest semantic. + */ + rel->uniquekeys = list_make1(makeUniqueKey(NIL, false)); +} + + +/* + * initililze_uniquecontext_for_joinrel + * Return a List of UniqueKeyContext for an inputrel + */ +static List * +initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel) +{ + List *res = NIL; + ListCell *lc; + foreach(lc, inputrel->uniquekeys) + { + UniqueKeyContext context; + context = palloc(sizeof(struct UniqueKeyContextData)); + context->uniquekey = lfirst_node(UniqueKey, lc); + context->added_to_joinrel = false; + context->useful = true; + res = lappend(res, context); + } + return res; +} + + +/* + * get_exprs_from_uniquekey + * Unify the way of get List of exprs from a one-row UniqueKey or + * normal UniqueKey. for the onerow case, every expr in rel1 is a valid + * UniqueKey. Return a List of exprs. + * + * rel1: The relation which you want to get the exprs. + * ukey: The UniqueKey you want to get the exprs. + */ +static List * +get_exprs_from_uniquekey(RelOptInfo *joinrel, RelOptInfo *rel1, UniqueKey *ukey) +{ + ListCell *lc; + bool onerow = rel1 != NULL && relation_is_onerow(rel1); + + List *res = NIL; + Assert(onerow || ukey); + if (onerow) + { + /* Only cares about the exprs still exist in joinrel */ + foreach(lc, joinrel->reltarget->exprs) + { + Bitmapset *relids = pull_varnos(lfirst(lc)); + if (bms_is_subset(relids, rel1->relids)) + { + res = lappend(res, list_make1(lfirst(lc))); + } + } + } + else + { + res = list_make1(ukey->exprs); + } + return res; +} + +/* + * Partitioned table Unique Keys. + * The partition table unique key is maintained as: + * 1. The index must be unique as usual. + * 2. The index must contains partition key. + * 3. The index must exist on all the child rel. see simple_indexinfo_equal for + * how we compare it. + */ + +/* + * index_constains_partkey + * return true if the index contains the partiton key. + */ +static bool +index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind) +{ + ListCell *lc; + int i; + Assert(IS_PARTITIONED_REL(partrel)); + Assert(partrel->part_scheme->partnatts > 0); + + for(i = 0; i < partrel->part_scheme->partnatts; i++) + { + Node *part_expr = linitial(partrel->partexprs[i]); + bool found_in_index = false; + foreach(lc, ind->indextlist) + { + Expr *index_expr = lfirst_node(TargetEntry, lc)->expr; + if (equal(index_expr, part_expr)) + { + found_in_index = true; + break; + } + } + if (!found_in_index) + return false; + } + return true; +} + +/* + * simple_indexinfo_equal + * + * Used to check if the 2 index is same as each other. The index here + * is COPIED from childrel and did some tiny changes(see + * simple_copy_indexinfo_to_parent) + */ +static bool +simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2) +{ + Size oid_cmp_len = sizeof(Oid) * ind1->ncolumns; + + return ind1->ncolumns == ind2->ncolumns && + ind1->unique == ind2->unique && + memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0 && + memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0 && + memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0 && + memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0 && + equal(get_tlist_exprs(ind1->indextlist, true), + get_tlist_exprs(ind2->indextlist, true)); +} + + +/* + * The below macros are used for simple_copy_indexinfo_to_parent which is so + * customized that I don't want to put it to copyfuncs.c. So copy it here. + */ +#define COPY_POINTER_FIELD(fldname, sz) \ + do { \ + Size _size = (sz); \ + newnode->fldname = palloc(_size); \ + memcpy(newnode->fldname, from->fldname, _size); \ + } while (0) + +#define COPY_NODE_FIELD(fldname) \ + (newnode->fldname = copyObjectImpl(from->fldname)) + +#define COPY_SCALAR_FIELD(fldname) \ + (newnode->fldname = from->fldname) + + +/* + * simple_copy_indexinfo_to_parent (from partition) + * Copy the IndexInfo from child relation to parent relation with some modification, + * which is used to test: + * 1. If the same index exists in all the childrels. + * 2. If the parentrel->reltarget/basicrestrict info matches this index. + */ +static IndexOptInfo * +simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from) +{ + IndexOptInfo *newnode = makeNode(IndexOptInfo); + AppendRelInfo *appinfo = find_appinfo_by_child(root, from->rel->relid); + ListCell *lc; + int idx = 0; + + COPY_SCALAR_FIELD(ncolumns); + COPY_SCALAR_FIELD(nkeycolumns); + COPY_SCALAR_FIELD(unique); + COPY_SCALAR_FIELD(immediate); + /* We just need to know if it is NIL or not */ + COPY_SCALAR_FIELD(indpred); + COPY_SCALAR_FIELD(predOK); + COPY_POINTER_FIELD(indexkeys, from->ncolumns * sizeof(int)); + COPY_POINTER_FIELD(indexcollations, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opfamily, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opcintype, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(sortopfamily, from->ncolumns * sizeof(Oid)); + COPY_NODE_FIELD(indextlist); + + /* Convert index exprs on child expr to expr on parent */ + foreach(lc, newnode->indextlist) + { + TargetEntry *tle = lfirst_node(TargetEntry, lc); + /* Index on expression is ignored */ + Assert(IsA(tle->expr, Var)); + tle->expr = (Expr *) find_parent_var(appinfo, (Var *) tle->expr); + newnode->indexkeys[idx] = castNode(Var, tle->expr)->varattno; + idx++; + } + newnode->rel = parentrel; + return newnode; +} + +/* + * adjust_partition_unique_indexlist + * + * global_unique_indexes: At the beginning, it contains the copy & modified + * unique index from the first partition. And then check if each index in it still + * exists in the following partitions. If no, remove it. at last, it has an + * index list which exists in all the partitions. + */ +static void +adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_indexes) +{ + ListCell *lc, *lc2; + foreach(lc, *global_unique_indexes) + { + IndexOptInfo *g_ind = lfirst_node(IndexOptInfo, lc); + bool found_in_child = false; + + foreach(lc2, childrel->indexlist) + { + IndexOptInfo *p_ind = lfirst_node(IndexOptInfo, lc2); + IndexOptInfo *p_ind_copy; + if (!p_ind->unique || !p_ind->immediate || + (p_ind->indpred != NIL && !p_ind->predOK)) + continue; + p_ind_copy = simple_copy_indexinfo_to_parent(root, parentrel, p_ind); + if (simple_indexinfo_equal(p_ind_copy, g_ind)) + { + found_in_child = true; + break; + } + } + if (!found_in_child) + /* The index doesn't exist in childrel, remove it from global_unique_indexes */ + *global_unique_indexes = foreach_delete_current(*global_unique_indexes, lc); + } +} + +/* Helper function for groupres/distinctrel */ +static void +add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgroups) +{ + Query *parse = root->parse; + List *exprs; + + /* + * XXX: If there are some vars which is not in current levelsup, the semantic is + * imprecise, should we avoid it or not? levelsup = 1 is just a demo, maybe we need to + * check every level other than 0, if so, looks we have to write another + * pull_var_walker. + */ + List *upper_vars = pull_vars_of_level((Node*)sortgroups, 1); + + if (upper_vars != NIL) + return; + + exprs = get_sortgrouplist_exprs(sortgroups, parse->targetList); + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(exprs, + false /* sortgroupclause can't be multi_nullvals */)); +} + + +/* + * add_combined_uniquekey + * The combination of both UniqueKeys is a valid UniqueKey for joinrel no matter + * the jointype. + */ +bool +add_combined_uniquekey(RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype) +{ + + ListCell *lc1, *lc2; + + /* Either side has multi_nullvals or we have outer join, + * the combined UniqueKey has multi_nullvals */ + bool multi_nullvals = outer_ukey->multi_nullvals || + inner_ukey->multi_nullvals || IS_OUTER_JOIN(jointype); + + /* The only case we can get onerow joinrel after join */ + if (relation_is_onerow(outer_rel) + && relation_is_onerow(inner_rel) + && jointype == JOIN_INNER) + { + add_uniquekey_for_onerow(joinrel); + return true; + } + + foreach(lc1, get_exprs_from_uniquekey(joinrel, outer_rel, outer_ukey)) + { + foreach(lc2, get_exprs_from_uniquekey(joinrel, inner_rel, inner_ukey)) + { + List *exprs = list_concat_copy(lfirst_node(List, lc1), lfirst_node(List, lc2)); + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(exprs, + multi_nullvals)); + } + } + return false; +} diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index b406d41e91..0551ae0512 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -2389,6 +2389,8 @@ grouping_planner(PlannerInfo *root, bool inheritance_update, add_path(final_rel, path); } + simple_copy_uniquekeys(current_rel, final_rel); + /* * Generate partial paths for final_rel, too, if outer query levels might * be able to make use of them. @@ -3899,6 +3901,8 @@ create_grouping_paths(PlannerInfo *root, } set_cheapest(grouped_rel); + + populate_grouprel_uniquekeys(root, grouped_rel, input_rel); return grouped_rel; } @@ -4615,7 +4619,7 @@ create_window_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(window_rel); - + simple_copy_uniquekeys(input_rel, window_rel); return window_rel; } @@ -4911,7 +4915,7 @@ create_distinct_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(distinct_rel); - + populate_distinctrel_uniquekeys(root, input_rel, distinct_rel); return distinct_rel; } @@ -5172,6 +5176,8 @@ create_ordered_paths(PlannerInfo *root, */ Assert(ordered_rel->pathlist != NIL); + simple_copy_uniquekeys(input_rel, ordered_rel); + return ordered_rel; } @@ -6049,6 +6055,9 @@ adjust_paths_for_srfs(PlannerInfo *root, RelOptInfo *rel, if (list_length(targets) == 1) return; + /* UniqueKey is not valid after handling the SRF. */ + rel->uniquekeys = NIL; + /* * Stack SRF-evaluation nodes atop each path for the rel. * diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c index 951aed80e7..e94e92937c 100644 --- a/src/backend/optimizer/prep/prepunion.c +++ b/src/backend/optimizer/prep/prepunion.c @@ -689,6 +689,8 @@ generate_union_paths(SetOperationStmt *op, PlannerInfo *root, /* Undo effects of possibly forcing tuple_fraction to 0 */ root->tuple_fraction = save_fraction; + /* Add the UniqueKeys */ + populate_unionrel_uniquekeys(root, result_rel); return result_rel; } diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c index d722063cf3..44c37ecffc 100644 --- a/src/backend/optimizer/util/appendinfo.c +++ b/src/backend/optimizer/util/appendinfo.c @@ -746,3 +746,47 @@ find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos) } return appinfos; } + +/* + * find_appinfo_by_child + * + */ +AppendRelInfo * +find_appinfo_by_child(PlannerInfo *root, Index child_index) +{ + ListCell *lc; + foreach(lc, root->append_rel_list) + { + AppendRelInfo *appinfo = lfirst_node(AppendRelInfo, lc); + if (appinfo->child_relid == child_index) + return appinfo; + } + elog(ERROR, "parent relation cant be found"); + return NULL; +} + +/* + * find_parent_var + * + */ +Var * +find_parent_var(AppendRelInfo *appinfo, Var *child_var) +{ + ListCell *lc; + Var *res = NULL; + Index attno = 1; + foreach(lc, appinfo->translated_vars) + { + Node *child_node = lfirst(lc); + if (equal(child_node, child_var)) + { + res = copyObject(child_var); + res->varattno = attno; + res->varno = appinfo->parent_relid; + } + attno++; + } + if (res == NULL) + elog(ERROR, "parent var can't be found."); + return res; +} diff --git a/src/backend/optimizer/util/inherit.c b/src/backend/optimizer/util/inherit.c index 3132fd35a5..d66b40ec50 100644 --- a/src/backend/optimizer/util/inherit.c +++ b/src/backend/optimizer/util/inherit.c @@ -736,6 +736,7 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, { Node *onecq = (Node *) lfirst(lc2); bool pseudoconstant; + RestrictInfo *child_rinfo; /* check for pseudoconstant (no Vars or volatile functions) */ pseudoconstant = @@ -747,13 +748,14 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, root->hasPseudoConstantQuals = true; } /* reconstitute RestrictInfo with appropriate properties */ - childquals = lappend(childquals, - make_restrictinfo((Expr *) onecq, - rinfo->is_pushed_down, - rinfo->outerjoin_delayed, - pseudoconstant, - rinfo->security_level, - NULL, NULL, NULL)); + child_rinfo = make_restrictinfo((Expr *) onecq, + rinfo->is_pushed_down, + rinfo->outerjoin_delayed, + pseudoconstant, + rinfo->security_level, + NULL, NULL, NULL); + child_rinfo->mergeopfamilies = rinfo->mergeopfamilies; + childquals = lappend(childquals, child_rinfo); /* track minimum security level among child quals */ cq_min_security = Min(cq_min_security, rinfo->security_level); } diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 31d9aedeeb..c83f17acb7 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -16,6 +16,7 @@ #include "nodes/execnodes.h" #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" extern A_Expr *makeA_Expr(A_Expr_Kind kind, List *name, @@ -105,4 +106,6 @@ extern GroupingSet *makeGroupingSet(GroupingSetKind kind, List *content, int loc extern VacuumRelation *makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols); +extern UniqueKey* makeUniqueKey(List *exprs, bool multi_nullvals); + #endif /* MAKEFUNC_H */ diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 381d84b4e4..41110ed888 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -264,6 +264,7 @@ typedef enum NodeTag T_EquivalenceMember, T_PathKey, T_PathTarget, + T_UniqueKey, T_RestrictInfo, T_IndexClause, T_PlaceHolderVar, diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index 9e3ebd488a..02e4458bef 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -730,6 +730,7 @@ typedef struct RelOptInfo QualCost baserestrictcost; /* cost of evaluating the above */ Index baserestrict_min_security; /* min security_level found in * baserestrictinfo */ + List *uniquekeys; /* List of UniqueKey */ List *joininfo; /* RestrictInfo structures for join clauses * involving this rel */ bool has_eclass_joins; /* T means joininfo is incomplete */ @@ -1047,6 +1048,28 @@ typedef struct PathKey } PathKey; +/* + * UniqueKey + * + * Represents the unique properties held by a RelOptInfo. + * + * exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL + * is a special case of UniqueKey, which means there is only 1 row in that + * relation. + * multi_nullvals: true means multi null values may exist in these exprs, so the + * uniqueness is not guaranteed in this case. This field is necessary for + * remove_useless_join & reduce_unique_semijoins where we don't mind these + * duplicated NULL values. It is set to true for 2 cases. One is a unique key + * from a unique index but the related column is nullable. The other one is for + * outer join. see populate_joinrel_uniquekeys for detail. + */ +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + /* * PathTarget * @@ -2473,7 +2496,7 @@ typedef enum * * flags indicating what kinds of grouping are possible. * partial_costs_set is true if the agg_partial_costs and agg_final_costs - * have been initialized. + * have been initialized. * agg_partial_costs gives partial aggregation costs. * agg_final_costs gives finalization costs. * target_parallel_safe is true if target is parallel safe. @@ -2503,8 +2526,8 @@ typedef struct * limit_tuples is an estimated bound on the number of output tuples, * or -1 if no LIMIT or couldn't estimate. * count_est and offset_est are the estimated values of the LIMIT and OFFSET - * expressions computed by preprocess_limit() (see comments for - * preprocess_limit() for more information). + * expressions computed by preprocess_limit() (see comments for + * preprocess_limit() for more information). */ typedef struct { diff --git a/src/include/nodes/pg_list.h b/src/include/nodes/pg_list.h index 14ea2766ad..621f54a9f8 100644 --- a/src/include/nodes/pg_list.h +++ b/src/include/nodes/pg_list.h @@ -528,6 +528,8 @@ extern bool list_member_ptr(const List *list, const void *datum); extern bool list_member_int(const List *list, int datum); extern bool list_member_oid(const List *list, Oid datum); +extern bool list_is_subset(const List *members, const List *target); + extern List *list_delete(List *list, void *datum); extern List *list_delete_ptr(List *list, void *datum); extern List *list_delete_int(List *list, int datum); diff --git a/src/include/optimizer/appendinfo.h b/src/include/optimizer/appendinfo.h index d6a27a60dd..e87c92a054 100644 --- a/src/include/optimizer/appendinfo.h +++ b/src/include/optimizer/appendinfo.h @@ -32,4 +32,7 @@ extern Relids adjust_child_relids_multilevel(PlannerInfo *root, Relids relids, extern AppendRelInfo **find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos); +extern AppendRelInfo *find_appinfo_by_child(PlannerInfo *root, Index child_index); +extern Var *find_parent_var(AppendRelInfo *appinfo, Var *child_var); + #endif /* APPENDINFO_H */ diff --git a/src/include/optimizer/optimizer.h b/src/include/optimizer/optimizer.h index 3e4171056e..9445141263 100644 --- a/src/include/optimizer/optimizer.h +++ b/src/include/optimizer/optimizer.h @@ -23,6 +23,7 @@ #define OPTIMIZER_H #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" /* * We don't want to include nodes/pathnodes.h here, because non-planner @@ -156,6 +157,7 @@ extern TargetEntry *get_sortgroupref_tle(Index sortref, List *targetList); extern TargetEntry *get_sortgroupclause_tle(SortGroupClause *sgClause, List *targetList); +extern Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); extern Node *get_sortgroupclause_expr(SortGroupClause *sgClause, List *targetList); extern List *get_sortgrouplist_exprs(List *sgClauses, diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h index 10b6e81079..9217a8d6c6 100644 --- a/src/include/optimizer/paths.h +++ b/src/include/optimizer/paths.h @@ -240,5 +240,48 @@ extern PathKey *make_canonical_pathkey(PlannerInfo *root, int strategy, bool nulls_first); extern void add_paths_to_append_rel(PlannerInfo *root, RelOptInfo *rel, List *live_childrels); +extern List *select_mergejoin_clauses(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + JoinType jointype, + bool *mergejoin_allowed); + +/* + * uniquekeys.c + * Utilities for matching and building unique keys + */ +extern void populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List* unique_index_list); +extern void populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels); +extern void populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel); +extern void populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel); +extern void populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel); +extern void simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel); +extern void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel); +extern void populate_joinrel_uniquekeys(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + RelOptInfo *rel2, + List *restrictlist, + JoinType jointype); + +extern bool relation_has_uniquekeys_for(PlannerInfo *root, + RelOptInfo *rel, + List *exprs, + bool allow_multinulls); +extern bool relation_is_onerow(RelOptInfo *rel); #endif /* PATHS_H */ -- 2.21.0 --op7mf2i72xk5yu7c Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v36-0003-Extend-UniqueKeys.patch" ^ permalink raw reply [nested|flat] 91+ messages in thread
* [PATCH v38 2/6] Introduce UniqueKey attributes on RelOptInfo struct. @ 2020-05-11 07:50 一挃 <[email protected]> 0 siblings, 0 replies; 91+ messages in thread From: 一挃 @ 2020-05-11 07:50 UTC (permalink / raw) UniqueKey is a set of exprs on RelOptInfo which represents the exprs will be unique on the given RelOptInfo. You can see README.uniquekey for more information. --- src/backend/nodes/copyfuncs.c | 13 + src/backend/nodes/list.c | 31 + src/backend/nodes/makefuncs.c | 13 + src/backend/nodes/outfuncs.c | 11 + src/backend/nodes/readfuncs.c | 10 + src/backend/optimizer/path/Makefile | 3 +- src/backend/optimizer/path/README.uniquekey | 131 +++ src/backend/optimizer/path/allpaths.c | 10 + src/backend/optimizer/path/joinpath.c | 9 +- src/backend/optimizer/path/joinrels.c | 2 + src/backend/optimizer/path/pathkeys.c | 3 +- src/backend/optimizer/path/uniquekeys.c | 1135 +++++++++++++++++++ src/backend/optimizer/plan/planner.c | 13 +- src/backend/optimizer/prep/prepunion.c | 2 + src/backend/optimizer/util/appendinfo.c | 44 + src/backend/optimizer/util/inherit.c | 18 +- src/include/nodes/makefuncs.h | 3 + src/include/nodes/nodes.h | 1 + src/include/nodes/pathnodes.h | 29 +- src/include/nodes/pg_list.h | 10 + src/include/optimizer/appendinfo.h | 3 + src/include/optimizer/optimizer.h | 2 + src/include/optimizer/paths.h | 43 + 23 files changed, 1515 insertions(+), 24 deletions(-) create mode 100644 src/backend/optimizer/path/README.uniquekey create mode 100644 src/backend/optimizer/path/uniquekeys.c diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..75c1c5e824 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2296,6 +2296,16 @@ _copyPathKey(const PathKey *from) return newnode; } +static UniqueKey * +_copyUniqueKey(const UniqueKey *from) +{ + UniqueKey *newnode = makeNode(UniqueKey); + + COPY_NODE_FIELD(exprs); + COPY_SCALAR_FIELD(multi_nullvals); + + return newnode; +} /* * _copyRestrictInfo */ @@ -5220,6 +5230,9 @@ copyObjectImpl(const void *from) case T_PathKey: retval = _copyPathKey(from); break; + case T_UniqueKey: + retval = _copyUniqueKey(from); + break; case T_RestrictInfo: retval = _copyRestrictInfo(from); break; diff --git a/src/backend/nodes/list.c b/src/backend/nodes/list.c index dbf6b30233..ca099495a1 100644 --- a/src/backend/nodes/list.c +++ b/src/backend/nodes/list.c @@ -702,6 +702,37 @@ list_member_oid(const List *list, Oid datum) return false; } +/* + * return true iff every entry in "members" list is also present + * in the "target" list. + */ +bool +list_is_subset(const List *members, const List *target) +{ + const ListCell *lc1, *lc2; + + Assert(IsPointerList(members)); + Assert(IsPointerList(target)); + check_list_invariants(members); + check_list_invariants(target); + + foreach(lc1, members) + { + bool found = false; + foreach(lc2, target) + { + if (equal(lfirst(lc1), lfirst(lc2))) + { + found = true; + break; + } + } + if (!found) + return false; + } + return true; +} + /* * Delete the n'th cell (counting from 0) in list. * diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 01c110cd2f..40415d0f5b 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -815,3 +815,16 @@ makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols) v->va_cols = va_cols; return v; } + + +/* + * makeUniqueKey + */ +UniqueKey* +makeUniqueKey(List *exprs, bool multi_nullvals) +{ + UniqueKey * ukey = makeNode(UniqueKey); + ukey->exprs = exprs; + ukey->multi_nullvals = multi_nullvals; + return ukey; +} diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..44154cde6a 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2456,6 +2456,14 @@ _outPathKey(StringInfo str, const PathKey *node) WRITE_BOOL_FIELD(pk_nulls_first); } +static void +_outUniqueKey(StringInfo str, const UniqueKey *node) +{ + WRITE_NODE_TYPE("UNIQUEKEY"); + WRITE_NODE_FIELD(exprs); + WRITE_BOOL_FIELD(multi_nullvals); +} + static void _outPathTarget(StringInfo str, const PathTarget *node) { @@ -4198,6 +4206,9 @@ outNode(StringInfo str, const void *obj) case T_PathKey: _outPathKey(str, obj); break; + case T_UniqueKey: + _outUniqueKey(str, obj); + break; case T_PathTarget: _outPathTarget(str, obj); break; diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index c5e136e9c3..b3e212bf1c 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -492,6 +492,14 @@ _readSetOperationStmt(void) READ_DONE(); } +static UniqueKey * +_readUniqueKey(void) +{ + READ_LOCALS(UniqueKey); + READ_NODE_FIELD(exprs); + READ_BOOL_FIELD(multi_nullvals); + READ_DONE(); +} /* * Stuff from primnodes.h. @@ -2717,6 +2725,8 @@ parseNodeString(void) return_value = _readCommonTableExpr(); else if (MATCH("SETOPERATIONSTMT", 16)) return_value = _readSetOperationStmt(); + else if (MATCH("UNIQUEKEY", 9)) + return_value = _readUniqueKey(); else if (MATCH("ALIAS", 5)) return_value = _readAlias(); else if (MATCH("RANGEVAR", 8)) diff --git a/src/backend/optimizer/path/Makefile b/src/backend/optimizer/path/Makefile index 1e199ff66f..7b9820c25f 100644 --- a/src/backend/optimizer/path/Makefile +++ b/src/backend/optimizer/path/Makefile @@ -21,6 +21,7 @@ OBJS = \ joinpath.o \ joinrels.o \ pathkeys.o \ - tidpath.o + tidpath.o \ + uniquekeys.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/optimizer/path/README.uniquekey b/src/backend/optimizer/path/README.uniquekey new file mode 100644 index 0000000000..5eac761995 --- /dev/null +++ b/src/backend/optimizer/path/README.uniquekey @@ -0,0 +1,131 @@ +1. What is UniqueKey? +We can think UniqueKey is a set of exprs for a RelOptInfo, which we are insure +that doesn't yields same result among all the rows. The simplest UniqueKey +format is primary key. + +However we define the UnqiueKey as below. + +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + +exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL +is a special case of UniqueKey, which means there is only one row in that +relation.it has a stronger semantic than others. like SELECT uk FROM t; uk is +normal unique key and may have different values. SELECT colx FROM t WHERE uk = +const. colx is unique AND we have only 1 value. This field can used for +innerrel_is_unique. this logic is handled specially in add_uniquekey_for_onerow +function. + +multi_nullvals: true means multi null values may exist in these exprs, so the +uniqueness is not guaranteed in this case. This field is necessary for +remove_useless_join & reduce_unique_semijoins where we don't mind these +duplicated NULL values. It is set to true for 2 cases. One is a unique key +from a unique index but the related column is nullable. The other one is for +outer join. see populate_joinrel_uniquekeys for detail. + + +The UniqueKey can be used at the following cases at least: +1. remove_useless_joins. +2. reduce_semianti_joins +3. remove distinct node if distinct clause is unique. +4. remove aggnode if group by clause is unique. +5. Index Skip Scan (WIP) +6. Aggregation Push Down without 2 phase aggregation if the join can't + duplicated the aggregated rows. (work in progress feature) + +2. How is it maintained? + +We have a set of populate_xxx_unqiuekeys functions to maintain the uniquekey on +various cases. xxx includes baserel, joinrel, partitionedrel, distinctrel, +groupedrel, unionrel. and we also need to convert the uniquekey from subquery +to outer relation, which is what convert_subquery_uniquekeys does. + +1. The first part is about baserel. We handled 3 cases. suppose we have Unique +Index on (a, b). + +1. SELECT a, b FROM t. UniqueKey (a, b) +2. SELECT a FROM t WHERE b = 1; UniqueKey (a) +3. SELECT .. FROM t WHERE a = 1 AND b = 1; UniqueKey (NIL). onerow case, every + column is Unique. + +2. The next part is joinrel, this part is most error-prone, we simplified the rules +like below: +1. If the relation's UniqueKey can't be duplicated after join, then is will be + still valid for the join rel. The function we used here is + innerrel_keeps_unique. The basic idea is innerrel.any_col = outer.uk. + +2. If the UnqiueKey can't keep valid via the rule 1, the combination of the + UniqueKey from both sides are valid for sure. We can prove this as: if the + unique exprs from rel1 is duplicated by rel2, the duplicated rows must + contains different unique exprs from rel2. + +More considerations about onerow: +1. If relation with one row and it can't be duplicated, it is still possible + contains mulit_nullvas after outer join. +2. If the either UniqueKey can be duplicated after join, the can get one row + only when both side is one row AND there is no outer join. +3. Whenever the onerow UniqueKey is not a valid any more, we need to convert one + row UniqueKey to normal unique key since we don't store exprs for one-row + relation. get_exprs_from_uniquekeys will be used here. + + +More considerations about multi_nullvals after join: +1. If the original UnqiueKey has multi_nullvals, the final UniqueKey will have + mulit_nullvals in any case. +2. If a unique key doesn't allow mulit_nullvals, after some outer join, it + allows some outer join. + + +3. When we comes to subquery, we need to convert_subquery_unqiuekeys just like +convert_subquery_pathkeys. Only the UniqueKey insides subquery is referenced as +a Var in outer relation will be reused. The relationship between the outerrel.Var +and subquery.exprs is built with outerel->subroot->processed_tlist. + + +4. As for the SRF functions, it will break the uniqueness of uniquekey, However it +is handled in adjust_paths_for_srfs, which happens after the query_planner. so +we will maintain the UniqueKey until there and reset it to NIL at that +places. This can't help on distinct/group by elimination cases but probably help +in some other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is +semantic correctly. + + +5. As for inherit table, we first main the UnqiueKey on childrel as well. But for +partitioned table we need to maintain 2 different kinds of +UnqiueKey. 1). UniqueKey on the parent relation 2). UniqueKey on child +relation for partition wise query. + +Example: +CREATE TABLE p (a int not null, b int not null) partition by list (a); +CREATE TABLE p0 partition of p for values in (1); +CREATE TABLE p1 partition of p for values in (2); + +create unique index p0_b on p0(b); +create unique index p1_b on p1(b); + +Now b is only unique on partition level, so the distinct can't be removed on +the following cases. SELECT DISTINCT b FROM p; + +Another example is SELECT DISTINCT a, b FROM p WHERE a = 1; Since only one +partition is chosen, the UniqueKey on child relation is same as the UniqueKey on +parent relation. + +Another usage of UniqueKey on partition level is it be helpful for +partition-wise join. + +As for the UniqueKey on parent table level, it comes with 2 different ways, +1). the UniqueKey is also derived in UniqueKey index, but the index must be same +in all the related children relations and the unique index must contains +Partition Key in it. Example: + +CREATE UNIQUE INDEX p_ab ON p(a, b); -- where a is the partition key. + +-- Query +SELECT a, b FROM p; the (a, b) is a UniqueKey of p. + +2). If the parent relation has only one childrel, the UniqueKey on childrel is + the UniqueKey on parent as well. diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index 37b4223adb..d52ad59f89 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -579,6 +579,12 @@ set_plain_rel_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) */ check_index_predicates(root, rel); + /* + * Now that we've marked which partial indexes are suitable, we can now + * build the relation's unique keys. + */ + populate_baserel_uniquekeys(root, rel, rel->indexlist); + /* Mark rel with estimated output rows, width, etc */ set_baserel_size_estimates(root, rel); } @@ -1297,6 +1303,8 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, /* Add paths to the append relation. */ add_paths_to_append_rel(root, rel, live_childrels); + if (IS_PARTITIONED_REL(rel)) + populate_partitionedrel_uniquekeys(root, rel, live_childrels); } @@ -2304,6 +2312,8 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, pathkeys, required_outer)); } + convert_subquery_uniquekeys(root, rel, sub_final_rel); + /* If outer rel allows parallelism, do same for partial paths. */ if (rel->consider_parallel && bms_is_empty(required_outer)) { diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c index 57ce97fd53..697ac047fb 100644 --- a/src/backend/optimizer/path/joinpath.c +++ b/src/backend/optimizer/path/joinpath.c @@ -71,13 +71,6 @@ static void consider_parallel_mergejoin(PlannerInfo *root, static void hash_inner_and_outer(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, RelOptInfo *innerrel, JoinType jointype, JoinPathExtraData *extra); -static List *select_mergejoin_clauses(PlannerInfo *root, - RelOptInfo *joinrel, - RelOptInfo *outerrel, - RelOptInfo *innerrel, - List *restrictlist, - JoinType jointype, - bool *mergejoin_allowed); static void generate_mergejoin_paths(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *innerrel, @@ -1927,7 +1920,7 @@ hash_inner_and_outer(PlannerInfo *root, * if it is mergejoinable and involves vars from the two sub-relations * currently of interest. */ -static List * +List * select_mergejoin_clauses(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c index 0dbe2ac726..7271f044ec 100644 --- a/src/backend/optimizer/path/joinrels.c +++ b/src/backend/optimizer/path/joinrels.c @@ -924,6 +924,8 @@ populate_joinrel_with_paths(PlannerInfo *root, RelOptInfo *rel1, /* Apply partitionwise join technique, if possible. */ try_partitionwise_join(root, rel1, rel2, joinrel, sjinfo, restrictlist); + + populate_joinrel_uniquekeys(root, joinrel, rel1, rel2, restrictlist, sjinfo->jointype); } diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index bd9a176d7d..139278829b 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -33,7 +33,6 @@ static bool pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys); static bool matches_boolean_partition_clause(RestrictInfo *rinfo, RelOptInfo *partrel, int partkeycol); -static Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); static bool right_merge_direction(PlannerInfo *root, PathKey *pathkey); @@ -1035,7 +1034,7 @@ convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel, * We need this to ensure that we don't return pathkeys describing values * that are unavailable above the level of the subquery scan. */ -static Var * +Var * find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle) { ListCell *lc; diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c new file mode 100644 index 0000000000..77ed2b2eff --- /dev/null +++ b/src/backend/optimizer/path/uniquekeys.c @@ -0,0 +1,1135 @@ +/*------------------------------------------------------------------------- + * + * uniquekeys.c + * Utilities for matching and building unique keys + * + * Portions Copyright (c) 2020, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/optimizer/path/uniquekeys.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "nodes/makefuncs.h" +#include "nodes/nodeFuncs.h" +#include "optimizer/pathnode.h" +#include "optimizer/paths.h" +#include "optimizer/appendinfo.h" +#include "optimizer/optimizer.h" +#include "optimizer/tlist.h" +#include "rewrite/rewriteManip.h" + + +/* + * This struct is used to help populate_joinrel_uniquekeys. + * + * added_to_joinrel is true if a uniquekey (from outerrel or innerrel) + * has been added to joinrel. + * useful is true if the exprs of the uniquekey still appears in joinrel. + */ +typedef struct UniqueKeyContextData +{ + UniqueKey *uniquekey; + bool added_to_joinrel; + bool useful; +} *UniqueKeyContext; + +static List *initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel); +static bool innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + bool reverse); + +static List *get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals); +static List *get_exprs_from_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + UniqueKey *ukey); +static void add_uniquekey_for_onerow(RelOptInfo *rel); +static bool add_combined_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype); + +/* Used for unique indexes checking for partitioned table */ +static bool index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind); +static IndexOptInfo *simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from); +static bool simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2); +static void adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_index); + +/* Helper function for grouped relation and distinct relation. */ +static void add_uniquekey_from_sortgroups(PlannerInfo *root, + RelOptInfo *rel, + List *sortgroups); + +/* + * populate_baserel_uniquekeys + * Populate 'baserel' uniquekeys list by looking at the rel's unique index + * and baserestrictinfo + */ +void +populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List *indexlist) +{ + ListCell *lc; + List *matched_uniq_indexes = NIL; + + /* Attrs appears in rel->reltarget->exprs. */ + Bitmapset *used_attrs = NULL; + + List *const_exprs = NIL; + List *expr_opfamilies = NIL; + + Assert(baserel->rtekind == RTE_RELATION); + + foreach(lc, indexlist) + { + IndexOptInfo *ind = (IndexOptInfo *) lfirst(lc); + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + matched_uniq_indexes = lappend(matched_uniq_indexes, ind); + } + + if (matched_uniq_indexes == NIL) + return; + + /* Check which attrs is used in baserel->reltarget */ + pull_varattnos((Node *)baserel->reltarget->exprs, baserel->relid, &used_attrs); + + /* Check which attrno is used at a mergeable const filter */ + foreach(lc, baserel->baserestrictinfo) + { + RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); + + if (rinfo->mergeopfamilies == NIL) + continue; + + if (bms_is_empty(rinfo->left_relids)) + { + const_exprs = lappend(const_exprs, get_rightop(rinfo->clause)); + } + else if (bms_is_empty(rinfo->right_relids)) + { + const_exprs = lappend(const_exprs, get_leftop(rinfo->clause)); + } + else + continue; + + expr_opfamilies = lappend(expr_opfamilies, rinfo->mergeopfamilies); + } + + foreach(lc, matched_uniq_indexes) + { + bool multi_nullvals, useful; + List *exprs = get_exprs_from_uniqueindex(lfirst_node(IndexOptInfo, lc), + const_exprs, + expr_opfamilies, + used_attrs, + &useful, + &multi_nullvals); + if (useful) + { + if (exprs == NIL) + { + /* All the columns in Unique Index matched with a restrictinfo */ + add_uniquekey_for_onerow(baserel); + return; + } + baserel->uniquekeys = lappend(baserel->uniquekeys, + makeUniqueKey(exprs, multi_nullvals)); + } + } +} + + +/* + * populate_partitionedrel_uniquekeys + * The UniqueKey on partitionrel comes from 2 cases: + * 1). Only one partition is involved in this query, the unique key can be + * copied to parent rel from childrel. + * 2). There are some unique index which includes partition key and exists + * in all the related partitions. + * We never mind rule 2 if we hit rule 1. + */ + +void +populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels) +{ + ListCell *lc; + List *global_uniq_indexlist = NIL; + RelOptInfo *childrel; + bool is_first = true; + + Assert(IS_PARTITIONED_REL(rel)); + + if (childrels == NIL) + return; + + /* + * If there is only one partition used in this query, the UniqueKey in childrel is + * still valid in parent level, but we need convert the format from child expr to + * parent expr. + */ + if (list_length(childrels) == 1) + { + /* Check for Rule 1 */ + RelOptInfo *childrel = linitial_node(RelOptInfo, childrels); + ListCell *lc; + Assert(childrel->reloptkind == RELOPT_OTHER_MEMBER_REL); + if (relation_is_onerow(childrel)) + { + add_uniquekey_for_onerow(rel); + return; + } + + foreach(lc, childrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + AppendRelInfo *appinfo = find_appinfo_by_child(root, childrel->relid); + List *parent_exprs = NIL; + bool can_reuse = true; + ListCell *lc2; + foreach(lc2, ukey->exprs) + { + Var *var = (Var *)lfirst(lc2); + /* + * If the expr comes from a expression, it is hard to build the expression + * in parent so ignore that case for now. + */ + if(!IsA(var, Var)) + { + can_reuse = false; + break; + } + /* Convert it to parent var */ + parent_exprs = lappend(parent_exprs, find_parent_var(appinfo, var)); + } + if (can_reuse) + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(parent_exprs, + ukey->multi_nullvals)); + } + } + else + { + /* Check for rule 2 */ + childrel = linitial_node(RelOptInfo, childrels); + foreach(lc, childrel->indexlist) + { + IndexOptInfo *ind = lfirst(lc); + IndexOptInfo *modified_index; + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + + /* + * During simple_copy_indexinfo_to_parent, we need to convert var from + * child var to parent var, index on expression is too complex to handle. + * so ignore it for now. + */ + if (ind->indexprs != NIL) + continue; + + modified_index = simple_copy_indexinfo_to_parent(root, rel, ind); + /* + * If the unique index doesn't contain partkey, then it is unique + * on this partition only, so it is useless for us. + */ + if (!index_constains_partkey(rel, modified_index)) + continue; + + global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index); + } + + if (global_uniq_indexlist != NIL) + { + foreach(lc, childrels) + { + RelOptInfo *child = lfirst(lc); + if (is_first) + { + is_first = false; + continue; + } + adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist); + } + /* Now we have a list of unique index which are exactly same on all childrels, + * Set the UniqueKey just like it is non-partition table + */ + populate_baserel_uniquekeys(root, rel, global_uniq_indexlist); + } + } +} + + +/* + * populate_distinctrel_uniquekeys + */ +void +populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel) +{ + /* The unique key before the distinct is still valid. */ + distinctrel->uniquekeys = list_copy(inputrel->uniquekeys); + add_uniquekey_from_sortgroups(root, distinctrel, root->parse->distinctClause); +} + +/* + * populate_grouprel_uniquekeys + */ +void +populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel) + +{ + Query *parse = root->parse; + bool input_ukey_added = false; + ListCell *lc; + + if (relation_is_onerow(inputrel)) + { + add_uniquekey_for_onerow(grouprel); + return; + } + if (parse->groupingSets) + return; + + /* A Normal group by without grouping set. */ + if (parse->groupClause) + { + /* + * Current even the groupby clause is Unique already, but if query has aggref + * We have to create grouprel still. To keep the UnqiueKey short, we will check + * the UniqueKey of input_rel still valid, if so we reuse it. + */ + foreach(lc, inputrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (list_is_subset(ukey->exprs, grouprel->reltarget->exprs)) + { + grouprel->uniquekeys = lappend(grouprel->uniquekeys, + ukey); + input_ukey_added = true; + } + } + if (!input_ukey_added) + /* + * group by clause must be a super-set of grouprel->reltarget->exprs except the + * aggregation expr, so if such exprs is unique already, no bother to generate + * new uniquekey for group by exprs. + */ + add_uniquekey_from_sortgroups(root, + grouprel, + root->parse->groupClause); + } + else + /* It has aggregation but without a group by, so only one row returned */ + add_uniquekey_for_onerow(grouprel); +} + +/* + * simple_copy_uniquekeys + * Using a function for the one-line code makes us easy to check where we simply + * copied the uniquekey. + */ +void +simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel) +{ + newrel->uniquekeys = oldrel->uniquekeys; +} + +/* + * populate_unionrel_uniquekeys + */ +void +populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel) +{ + ListCell *lc; + List *exprs = NIL; + + Assert(unionrel->uniquekeys == NIL); + + foreach(lc, unionrel->reltarget->exprs) + { + exprs = lappend(exprs, lfirst(lc)); + } + + if (exprs == NIL) + /* SQL: select union select; is valid, we need to handle it here. */ + add_uniquekey_for_onerow(unionrel); + else + unionrel->uniquekeys = lappend(unionrel->uniquekeys, + makeUniqueKey(exprs,false)); + +} + +/* + * populate_joinrel_uniquekeys + * + * populate uniquekeys for joinrel. We will check each relation to see if its + * UniqueKey is still valid via innerrel_keeps_unique, if so, we add it to + * joinrel. The multi_nullvals field will be changed to true for some outer + * join cases and one-row UniqueKey needs to be converted to normal UniqueKey + * for the same case as well. + * For the uniquekey in either baserel which can't be unique after join, we still + * check to see if combination of UniqueKeys from both side is still useful for us. + * if yes, we add it to joinrel as well. + */ +void +populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *outerrel, RelOptInfo *innerrel, + List *restrictlist, JoinType jointype) +{ + ListCell *lc, *lc2; + List *clause_list = NIL; + List *outerrel_ukey_ctx; + List *innerrel_ukey_ctx; + bool inner_onerow, outer_onerow; + bool mergejoin_allowed; + + /* Care about the outerrel relation only for SEMI/ANTI join */ + if (jointype == JOIN_SEMI || jointype == JOIN_ANTI) + { + foreach(lc, outerrel->uniquekeys) + { + UniqueKey *uniquekey = lfirst_node(UniqueKey, lc); + if (list_is_subset(uniquekey->exprs, joinrel->reltarget->exprs)) + joinrel->uniquekeys = lappend(joinrel->uniquekeys, uniquekey); + } + return; + } + + Assert(jointype == JOIN_LEFT || jointype == JOIN_FULL || jointype == JOIN_INNER); + + /* Fast path */ + if (innerrel->uniquekeys == NIL || outerrel->uniquekeys == NIL) + return; + + inner_onerow = relation_is_onerow(innerrel); + outer_onerow = relation_is_onerow(outerrel); + + outerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(outerrel); + innerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(innerrel); + + clause_list = select_mergejoin_clauses(root, joinrel, outerrel, innerrel, + restrictlist, jointype, + &mergejoin_allowed); + + if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true /* reverse */)) + { + bool outer_impact = jointype == JOIN_FULL; + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + /* Outer relation has one row, and the unique key is not duplicated after join, + * the joinrel will still has one row unless the jointype == JOIN_FULL. + */ + if (outer_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (outer_onerow) + { + /* + * The onerow outerrel becomes multi rows and multi_nullvals + * will be changed to true. We also need to set the exprs correctly since it + * can't be NIL any more. + */ + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, outerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Change multi_nullvals to true due to the full join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, true)); + else + /* Just reuse it */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + } + ctx->added_to_joinrel = true; + } + } + + if (innerrel_keeps_unique(root, outerrel, innerrel, clause_list, false)) + { + bool outer_impact = jointype == JOIN_FULL || jointype == JOIN_LEFT;; + + foreach(lc, innerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + if (inner_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (inner_onerow) + { + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, innerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Need to change multi_nullvals to true due to the outer join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, + true)); + else + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + + } + ctx->added_to_joinrel = true; + } + } + + /* + * The combination of the UniqueKey from both sides is unique as well regardless + * of join type, but no bother to add it if its subset has been added to joinrel + * already or it is not useful for the joinrel. + */ + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx1 = (UniqueKeyContext) lfirst(lc); + if (ctx1->added_to_joinrel || !ctx1->useful) + continue; + foreach(lc2, innerrel_ukey_ctx) + { + UniqueKeyContext ctx2 = (UniqueKeyContext) lfirst(lc2); + if (ctx2->added_to_joinrel || !ctx2->useful) + continue; + if (add_combined_uniquekey(root, joinrel, outerrel, innerrel, + ctx1->uniquekey, ctx2->uniquekey, + jointype)) + /* If we set a onerow UniqueKey to joinrel, we don't need other. */ + return; + } + } +} + + +/* + * convert_subquery_uniquekeys + * + * Covert the UniqueKey in subquery to outer relation. + */ +void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel) +{ + ListCell *lc; + + if (sub_final_rel->uniquekeys == NIL) + return; + + if (relation_is_onerow(sub_final_rel)) + { + add_uniquekey_for_onerow(currel); + return; + } + + Assert(currel->subroot != NULL); + + foreach(lc, sub_final_rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + ListCell *lc; + List *exprs = NIL; + bool ukey_useful = true; + + /* One row case is handled above */ + Assert(ukey->exprs != NIL); + foreach(lc, ukey->exprs) + { + Var *var; + TargetEntry *tle = tlist_member(lfirst(lc), + currel->subroot->processed_tlist); + if (tle == NULL) + { + ukey_useful = false; + break; + } + var = find_var_for_subquery_tle(currel, tle); + if (var == NULL) + { + ukey_useful = false; + break; + } + exprs = lappend(exprs, var); + } + + if (ukey_useful) + currel->uniquekeys = lappend(currel->uniquekeys, + makeUniqueKey(exprs, + ukey->multi_nullvals)); + + } +} + +/* + * innerrel_keeps_unique + * + * Check if Unique key of the innerrel is valid after join. innerrel's UniqueKey + * will be still valid if innerrel's any-column mergeop outrerel's uniquekey + * exists in clause_list. + * + * Note: the clause_list must be a list of mergeable restrictinfo already. + */ +static bool +innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *clause_list, + bool reverse) +{ + ListCell *lc, *lc2, *lc3; + + if (outerrel->uniquekeys == NIL || innerrel->uniquekeys == NIL) + return false; + + /* Check if there is outerrel's uniquekey in mergeable clause. */ + foreach(lc, outerrel->uniquekeys) + { + List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs; + bool clauselist_matchs_all_exprs = true; + foreach(lc2, outer_uq_exprs) + { + Node *outer_uq_expr = lfirst(lc2); + bool find_uq_expr_in_clauselist = false; + foreach(lc3, clause_list) + { + RestrictInfo *rinfo = lfirst_node(RestrictInfo, lc3); + Node *outer_expr; + if (reverse) + outer_expr = rinfo->outer_is_left ? get_rightop(rinfo->clause) : get_leftop(rinfo->clause); + else + outer_expr = rinfo->outer_is_left ? get_leftop(rinfo->clause) : get_rightop(rinfo->clause); + if (equal(outer_expr, outer_uq_expr)) + { + find_uq_expr_in_clauselist = true; + break; + } + } + if (!find_uq_expr_in_clauselist) + { + /* No need to check the next exprs in the current uniquekey */ + clauselist_matchs_all_exprs = false; + break; + } + } + + if (clauselist_matchs_all_exprs) + return true; + } + return false; +} + + +/* + * relation_is_onerow + * Check if it is a one-row relation by checking UniqueKey. + */ +bool +relation_is_onerow(RelOptInfo *rel) +{ + UniqueKey *ukey; + if (rel->uniquekeys == NIL) + return false; + ukey = linitial_node(UniqueKey, rel->uniquekeys); + return ukey->exprs == NIL && list_length(rel->uniquekeys) == 1; +} + +/* + * relation_has_uniquekeys_for + * Returns true if we have proofs that 'rel' cannot return multiple rows with + * the same values in each of 'exprs'. Otherwise returns false. + */ +bool +relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel, + List *exprs, bool allow_multinulls) +{ + ListCell *lc; + + /* + * For UniqueKey->onerow case, the uniquekey->exprs is empty as well + * so we can't rely on list_is_subset to handle this special cases + */ + if (exprs == NIL) + return false; + + foreach(lc, rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (ukey->multi_nullvals && !allow_multinulls) + continue; + if (list_is_subset(ukey->exprs, exprs)) + return true; + } + return false; +} + + +/* + * get_exprs_from_uniqueindex + * + * Return a list of exprs which is unique. set useful to false if this + * unique index is not useful for us. + */ +static List * +get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals) +{ + List *exprs = NIL; + ListCell *indexpr_item; + int c = 0; + + *useful = true; + *multi_nullvals = false; + + indexpr_item = list_head(unique_index->indexprs); + for(c = 0; c < unique_index->ncolumns; c++) + { + int attr = unique_index->indexkeys[c]; + Expr *expr; + bool matched_const = false; + ListCell *lc1, *lc2; + + if(attr > 0) + { + expr = list_nth_node(TargetEntry, unique_index->indextlist, c)->expr; + } + else if (attr == 0) + { + /* Expression index */ + expr = lfirst(indexpr_item); + indexpr_item = lnext(unique_index->indexprs, indexpr_item); + } + else /* attr < 0 */ + { + /* Index on system column is not supported */ + Assert(false); + } + + /* + * Check index_col = Const case with regarding to opfamily checking + * If we can remove the index_col from the final UniqueKey->exprs. + */ + forboth(lc1, const_exprs, lc2, const_expr_opfamilies) + { + if (list_member_oid((List *)lfirst(lc2), unique_index->opfamily[c]) + && match_index_to_operand((Node *) lfirst(lc1), c, unique_index)) + { + matched_const = true; + break; + } + } + + if (matched_const) + continue; + + /* Check if the indexed expr is used in rel */ + if (attr > 0) + { + /* + * Normal Indexed column, if the col is not used, then the index is useless + * for uniquekey. + */ + attr -= FirstLowInvalidHeapAttributeNumber; + + if (!bms_is_member(attr, used_varattrs)) + { + *useful = false; + break; + } + } + else if (!list_member(unique_index->rel->reltarget->exprs, expr)) + { + /* Expression index but the expression is not used in rel */ + *useful = false; + break; + } + + /* check not null property. */ + if (attr == 0) + { + /* We never know if a expression yields null or not */ + *multi_nullvals = true; + } + else if (!bms_is_member(attr, unique_index->rel->notnullattrs) + && !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber, + unique_index->rel->notnullattrs)) + { + *multi_nullvals = true; + } + + exprs = lappend(exprs, expr); + } + return exprs; +} + + +/* + * add_uniquekey_for_onerow + * If we are sure that the relation only returns one row, then all the columns + * are unique. However we don't need to create UniqueKey for every column, we + * just set exprs = NIL and overwrites all the other UniqueKey on this RelOptInfo + * since this one has strongest semantics. + */ +void +add_uniquekey_for_onerow(RelOptInfo *rel) +{ + /* + * We overwrite the previous UniqueKey on purpose since this one has the + * strongest semantic. + */ + rel->uniquekeys = list_make1(makeUniqueKey(NIL, false)); +} + + +/* + * initililze_uniquecontext_for_joinrel + * Return a List of UniqueKeyContext for an inputrel + */ +static List * +initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel) +{ + List *res = NIL; + ListCell *lc; + foreach(lc, inputrel->uniquekeys) + { + UniqueKeyContext context; + context = palloc(sizeof(struct UniqueKeyContextData)); + context->uniquekey = lfirst_node(UniqueKey, lc); + context->added_to_joinrel = false; + context->useful = true; + res = lappend(res, context); + } + return res; +} + + +/* + * get_exprs_from_uniquekey + * Unify the way of get List of exprs from a one-row UniqueKey or + * normal UniqueKey. for the onerow case, every expr in rel1 is a valid + * UniqueKey. Return a List of exprs. + * + * rel1: The relation which you want to get the exprs. + * ukey: The UniqueKey you want to get the exprs. + */ +static List * +get_exprs_from_uniquekey(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *rel1, UniqueKey *ukey) +{ + ListCell *lc; + bool onerow = rel1 != NULL && relation_is_onerow(rel1); + + List *res = NIL; + Assert(onerow || ukey); + if (onerow) + { + /* Only cares about the exprs still exist in joinrel */ + foreach(lc, joinrel->reltarget->exprs) + { + Bitmapset *relids = pull_varnos(root, lfirst(lc)); + if (bms_is_subset(relids, rel1->relids)) + { + res = lappend(res, list_make1(lfirst(lc))); + } + } + } + else + { + res = list_make1(ukey->exprs); + } + return res; +} + +/* + * Partitioned table Unique Keys. + * The partition table unique key is maintained as: + * 1. The index must be unique as usual. + * 2. The index must contains partition key. + * 3. The index must exist on all the child rel. see simple_indexinfo_equal for + * how we compare it. + */ + +/* + * index_constains_partkey + * return true if the index contains the partiton key. + */ +static bool +index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind) +{ + ListCell *lc; + int i; + Assert(IS_PARTITIONED_REL(partrel)); + Assert(partrel->part_scheme->partnatts > 0); + + for(i = 0; i < partrel->part_scheme->partnatts; i++) + { + Node *part_expr = linitial(partrel->partexprs[i]); + bool found_in_index = false; + foreach(lc, ind->indextlist) + { + Expr *index_expr = lfirst_node(TargetEntry, lc)->expr; + if (equal(index_expr, part_expr)) + { + found_in_index = true; + break; + } + } + if (!found_in_index) + return false; + } + return true; +} + +/* + * simple_indexinfo_equal + * + * Used to check if the 2 index is same as each other. The index here + * is COPIED from childrel and did some tiny changes(see + * simple_copy_indexinfo_to_parent) + */ +static bool +simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2) +{ + Size oid_cmp_len = sizeof(Oid) * ind1->ncolumns; + + return ind1->ncolumns == ind2->ncolumns && + ind1->unique == ind2->unique && + memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0 && + memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0 && + memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0 && + memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0 && + equal(get_tlist_exprs(ind1->indextlist, true), + get_tlist_exprs(ind2->indextlist, true)); +} + + +/* + * The below macros are used for simple_copy_indexinfo_to_parent which is so + * customized that I don't want to put it to copyfuncs.c. So copy it here. + */ +#define COPY_POINTER_FIELD(fldname, sz) \ + do { \ + Size _size = (sz); \ + newnode->fldname = palloc(_size); \ + memcpy(newnode->fldname, from->fldname, _size); \ + } while (0) + +#define COPY_NODE_FIELD(fldname) \ + (newnode->fldname = copyObjectImpl(from->fldname)) + +#define COPY_SCALAR_FIELD(fldname) \ + (newnode->fldname = from->fldname) + + +/* + * simple_copy_indexinfo_to_parent (from partition) + * Copy the IndexInfo from child relation to parent relation with some modification, + * which is used to test: + * 1. If the same index exists in all the childrels. + * 2. If the parentrel->reltarget/basicrestrict info matches this index. + */ +static IndexOptInfo * +simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from) +{ + IndexOptInfo *newnode = makeNode(IndexOptInfo); + AppendRelInfo *appinfo = find_appinfo_by_child(root, from->rel->relid); + ListCell *lc; + int idx = 0; + + COPY_SCALAR_FIELD(ncolumns); + COPY_SCALAR_FIELD(nkeycolumns); + COPY_SCALAR_FIELD(unique); + COPY_SCALAR_FIELD(immediate); + /* We just need to know if it is NIL or not */ + COPY_SCALAR_FIELD(indpred); + COPY_SCALAR_FIELD(predOK); + COPY_POINTER_FIELD(indexkeys, from->ncolumns * sizeof(int)); + COPY_POINTER_FIELD(indexcollations, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opfamily, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opcintype, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(sortopfamily, from->ncolumns * sizeof(Oid)); + COPY_NODE_FIELD(indextlist); + + /* Convert index exprs on child expr to expr on parent */ + foreach(lc, newnode->indextlist) + { + TargetEntry *tle = lfirst_node(TargetEntry, lc); + /* Index on expression is ignored */ + Assert(IsA(tle->expr, Var)); + tle->expr = (Expr *) find_parent_var(appinfo, (Var *) tle->expr); + newnode->indexkeys[idx] = castNode(Var, tle->expr)->varattno; + idx++; + } + newnode->rel = parentrel; + return newnode; +} + +/* + * adjust_partition_unique_indexlist + * + * global_unique_indexes: At the beginning, it contains the copy & modified + * unique index from the first partition. And then check if each index in it still + * exists in the following partitions. If no, remove it. at last, it has an + * index list which exists in all the partitions. + */ +static void +adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_indexes) +{ + ListCell *lc, *lc2; + foreach(lc, *global_unique_indexes) + { + IndexOptInfo *g_ind = lfirst_node(IndexOptInfo, lc); + bool found_in_child = false; + + foreach(lc2, childrel->indexlist) + { + IndexOptInfo *p_ind = lfirst_node(IndexOptInfo, lc2); + IndexOptInfo *p_ind_copy; + if (!p_ind->unique || !p_ind->immediate || + (p_ind->indpred != NIL && !p_ind->predOK)) + continue; + p_ind_copy = simple_copy_indexinfo_to_parent(root, parentrel, p_ind); + if (simple_indexinfo_equal(p_ind_copy, g_ind)) + { + found_in_child = true; + break; + } + } + if (!found_in_child) + /* The index doesn't exist in childrel, remove it from global_unique_indexes */ + *global_unique_indexes = foreach_delete_current(*global_unique_indexes, lc); + } +} + +/* Helper function for groupres/distinctrel */ +static void +add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgroups) +{ + Query *parse = root->parse; + List *exprs; + + /* + * XXX: If there are some vars which is not in current levelsup, the semantic is + * imprecise, should we avoid it or not? levelsup = 1 is just a demo, maybe we need to + * check every level other than 0, if so, looks we have to write another + * pull_var_walker. + */ + List *upper_vars = pull_vars_of_level((Node*)sortgroups, 1); + + if (upper_vars != NIL) + return; + + exprs = get_sortgrouplist_exprs(sortgroups, parse->targetList); + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(exprs, + false /* sortgroupclause can't be multi_nullvals */)); +} + + +/* + * add_combined_uniquekey + * The combination of both UniqueKeys is a valid UniqueKey for joinrel no matter + * the jointype. + */ +bool +add_combined_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype) +{ + + ListCell *lc1, *lc2; + + /* Either side has multi_nullvals or we have outer join, + * the combined UniqueKey has multi_nullvals */ + bool multi_nullvals = outer_ukey->multi_nullvals || + inner_ukey->multi_nullvals || IS_OUTER_JOIN(jointype); + + /* The only case we can get onerow joinrel after join */ + if (relation_is_onerow(outer_rel) + && relation_is_onerow(inner_rel) + && jointype == JOIN_INNER) + { + add_uniquekey_for_onerow(joinrel); + return true; + } + + foreach(lc1, get_exprs_from_uniquekey(root, joinrel, outer_rel, outer_ukey)) + { + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, inner_rel, inner_ukey)) + { + List *exprs = list_concat_copy(lfirst_node(List, lc1), lfirst_node(List, lc2)); + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(exprs, + multi_nullvals)); + } + } + return false; +} diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index 424d25cbd5..8d8e493f5c 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -2387,6 +2387,8 @@ grouping_planner(PlannerInfo *root, bool inheritance_update, add_path(final_rel, path); } + simple_copy_uniquekeys(current_rel, final_rel); + /* * Generate partial paths for final_rel, too, if outer query levels might * be able to make use of them. @@ -3899,6 +3901,8 @@ create_grouping_paths(PlannerInfo *root, } set_cheapest(grouped_rel); + + populate_grouprel_uniquekeys(root, grouped_rel, input_rel); return grouped_rel; } @@ -4621,7 +4625,7 @@ create_window_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(window_rel); - + simple_copy_uniquekeys(input_rel, window_rel); return window_rel; } @@ -4935,7 +4939,7 @@ create_distinct_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(distinct_rel); - + populate_distinctrel_uniquekeys(root, input_rel, distinct_rel); return distinct_rel; } @@ -5196,6 +5200,8 @@ create_ordered_paths(PlannerInfo *root, */ Assert(ordered_rel->pathlist != NIL); + simple_copy_uniquekeys(input_rel, ordered_rel); + return ordered_rel; } @@ -6073,6 +6079,9 @@ adjust_paths_for_srfs(PlannerInfo *root, RelOptInfo *rel, if (list_length(targets) == 1) return; + /* UniqueKey is not valid after handling the SRF. */ + rel->uniquekeys = NIL; + /* * Stack SRF-evaluation nodes atop each path for the rel. * diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c index becdcbb872..b7626545bf 100644 --- a/src/backend/optimizer/prep/prepunion.c +++ b/src/backend/optimizer/prep/prepunion.c @@ -689,6 +689,8 @@ generate_union_paths(SetOperationStmt *op, PlannerInfo *root, /* Undo effects of possibly forcing tuple_fraction to 0 */ root->tuple_fraction = save_fraction; + /* Add the UniqueKeys */ + populate_unionrel_uniquekeys(root, result_rel); return result_rel; } diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c index 86922a273c..6817c9c787 100644 --- a/src/backend/optimizer/util/appendinfo.c +++ b/src/backend/optimizer/util/appendinfo.c @@ -746,3 +746,47 @@ find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos) } return appinfos; } + +/* + * find_appinfo_by_child + * + */ +AppendRelInfo * +find_appinfo_by_child(PlannerInfo *root, Index child_index) +{ + ListCell *lc; + foreach(lc, root->append_rel_list) + { + AppendRelInfo *appinfo = lfirst_node(AppendRelInfo, lc); + if (appinfo->child_relid == child_index) + return appinfo; + } + elog(ERROR, "parent relation cant be found"); + return NULL; +} + +/* + * find_parent_var + * + */ +Var * +find_parent_var(AppendRelInfo *appinfo, Var *child_var) +{ + ListCell *lc; + Var *res = NULL; + Index attno = 1; + foreach(lc, appinfo->translated_vars) + { + Node *child_node = lfirst(lc); + if (equal(child_node, child_var)) + { + res = copyObject(child_var); + res->varattno = attno; + res->varno = appinfo->parent_relid; + } + attno++; + } + if (res == NULL) + elog(ERROR, "parent var can't be found."); + return res; +} diff --git a/src/backend/optimizer/util/inherit.c b/src/backend/optimizer/util/inherit.c index be1c9ddd96..3eec1f4d74 100644 --- a/src/backend/optimizer/util/inherit.c +++ b/src/backend/optimizer/util/inherit.c @@ -736,6 +736,7 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, { Node *onecq = (Node *) lfirst(lc2); bool pseudoconstant; + RestrictInfo *child_rinfo; /* check for pseudoconstant (no Vars or volatile functions) */ pseudoconstant = @@ -747,14 +748,15 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, root->hasPseudoConstantQuals = true; } /* reconstitute RestrictInfo with appropriate properties */ - childquals = lappend(childquals, - make_restrictinfo(root, - (Expr *) onecq, - rinfo->is_pushed_down, - rinfo->outerjoin_delayed, - pseudoconstant, - rinfo->security_level, - NULL, NULL, NULL)); + child_rinfo = make_restrictinfo(root, + (Expr *) onecq, + rinfo->is_pushed_down, + rinfo->outerjoin_delayed, + pseudoconstant, + rinfo->security_level, + NULL, NULL, NULL); + child_rinfo->mergeopfamilies = rinfo->mergeopfamilies; + childquals = lappend(childquals, child_rinfo); /* track minimum security level among child quals */ cq_min_security = Min(cq_min_security, rinfo->security_level); } diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 48a7ebfe45..4fe1824eb0 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -16,6 +16,7 @@ #include "nodes/execnodes.h" #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" extern A_Expr *makeA_Expr(A_Expr_Kind kind, List *name, @@ -106,4 +107,6 @@ extern GroupingSet *makeGroupingSet(GroupingSetKind kind, List *content, int loc extern VacuumRelation *makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols); +extern UniqueKey* makeUniqueKey(List *exprs, bool multi_nullvals); + #endif /* MAKEFUNC_H */ diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index e22df890ef..3320273ac1 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -265,6 +265,7 @@ typedef enum NodeTag T_EquivalenceMember, T_PathKey, T_PathTarget, + T_UniqueKey, T_RestrictInfo, T_IndexClause, T_PlaceHolderVar, diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index 0d61f04d27..80561b1003 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -739,6 +739,7 @@ typedef struct RelOptInfo QualCost baserestrictcost; /* cost of evaluating the above */ Index baserestrict_min_security; /* min security_level found in * baserestrictinfo */ + List *uniquekeys; /* List of UniqueKey */ List *joininfo; /* RestrictInfo structures for join clauses * involving this rel */ bool has_eclass_joins; /* T means joininfo is incomplete */ @@ -1059,6 +1060,28 @@ typedef struct PathKey } PathKey; +/* + * UniqueKey + * + * Represents the unique properties held by a RelOptInfo. + * + * exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL + * is a special case of UniqueKey, which means there is only 1 row in that + * relation. + * multi_nullvals: true means multi null values may exist in these exprs, so the + * uniqueness is not guaranteed in this case. This field is necessary for + * remove_useless_join & reduce_unique_semijoins where we don't mind these + * duplicated NULL values. It is set to true for 2 cases. One is a unique key + * from a unique index but the related column is nullable. The other one is for + * outer join. see populate_joinrel_uniquekeys for detail. + */ +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + /* * PathTarget * @@ -2496,7 +2519,7 @@ typedef enum * * flags indicating what kinds of grouping are possible. * partial_costs_set is true if the agg_partial_costs and agg_final_costs - * have been initialized. + * have been initialized. * agg_partial_costs gives partial aggregation costs. * agg_final_costs gives finalization costs. * target_parallel_safe is true if target is parallel safe. @@ -2526,8 +2549,8 @@ typedef struct * limit_tuples is an estimated bound on the number of output tuples, * or -1 if no LIMIT or couldn't estimate. * count_est and offset_est are the estimated values of the LIMIT and OFFSET - * expressions computed by preprocess_limit() (see comments for - * preprocess_limit() for more information). + * expressions computed by preprocess_limit() (see comments for + * preprocess_limit() for more information). */ typedef struct { diff --git a/src/include/nodes/pg_list.h b/src/include/nodes/pg_list.h index 404e03f132..85c65e910c 100644 --- a/src/include/nodes/pg_list.h +++ b/src/include/nodes/pg_list.h @@ -566,6 +566,16 @@ extern pg_nodiscard List *list_delete_first(List *list); extern pg_nodiscard List *list_delete_last(List *list); extern pg_nodiscard List *list_delete_nth_cell(List *list, int n); extern pg_nodiscard List *list_delete_cell(List *list, ListCell *cell); +extern bool list_is_subset(const List *members, const List *target); + +extern List *list_delete(List *list, void *datum); +extern List *list_delete_ptr(List *list, void *datum); +extern List *list_delete_int(List *list, int datum); +extern List *list_delete_oid(List *list, Oid datum); +extern List *list_delete_first(List *list); +extern List *list_delete_last(List *list); +extern List *list_delete_nth_cell(List *list, int n); +extern List *list_delete_cell(List *list, ListCell *cell); extern List *list_union(const List *list1, const List *list2); extern List *list_union_ptr(const List *list1, const List *list2); diff --git a/src/include/optimizer/appendinfo.h b/src/include/optimizer/appendinfo.h index 4cbf8c26cc..43b8d05f69 100644 --- a/src/include/optimizer/appendinfo.h +++ b/src/include/optimizer/appendinfo.h @@ -32,4 +32,7 @@ extern Relids adjust_child_relids_multilevel(PlannerInfo *root, Relids relids, extern AppendRelInfo **find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos); +extern AppendRelInfo *find_appinfo_by_child(PlannerInfo *root, Index child_index); +extern Var *find_parent_var(AppendRelInfo *appinfo, Var *child_var); + #endif /* APPENDINFO_H */ diff --git a/src/include/optimizer/optimizer.h b/src/include/optimizer/optimizer.h index d587952b7d..843aafa51e 100644 --- a/src/include/optimizer/optimizer.h +++ b/src/include/optimizer/optimizer.h @@ -23,6 +23,7 @@ #define OPTIMIZER_H #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" /* Test if an expression node represents a SRF call. Beware multiple eval! */ #define IS_SRF_CALL(node) \ @@ -168,6 +169,7 @@ extern TargetEntry *get_sortgroupref_tle(Index sortref, List *targetList); extern TargetEntry *get_sortgroupclause_tle(SortGroupClause *sgClause, List *targetList); +extern Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); extern Node *get_sortgroupclause_expr(SortGroupClause *sgClause, List *targetList); extern List *get_sortgrouplist_exprs(List *sgClauses, diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h index 035d3e1206..1adf99c9ee 100644 --- a/src/include/optimizer/paths.h +++ b/src/include/optimizer/paths.h @@ -247,5 +247,48 @@ extern PathKey *make_canonical_pathkey(PlannerInfo *root, int strategy, bool nulls_first); extern void add_paths_to_append_rel(PlannerInfo *root, RelOptInfo *rel, List *live_childrels); +extern List *select_mergejoin_clauses(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + JoinType jointype, + bool *mergejoin_allowed); + +/* + * uniquekeys.c + * Utilities for matching and building unique keys + */ +extern void populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List* unique_index_list); +extern void populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels); +extern void populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel); +extern void populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel); +extern void populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel); +extern void simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel); +extern void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel); +extern void populate_joinrel_uniquekeys(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + RelOptInfo *rel2, + List *restrictlist, + JoinType jointype); + +extern bool relation_has_uniquekeys_for(PlannerInfo *root, + RelOptInfo *rel, + List *exprs, + bool allow_multinulls); +extern bool relation_is_onerow(RelOptInfo *rel); #endif /* PATHS_H */ -- 2.26.2 --yh7cw345mbttadjz Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v38-0003-Extend-UniqueKeys.patch" ^ permalink raw reply [nested|flat] 91+ messages in thread
* [PATCH v36 2/6] Introduce UniqueKey attributes on RelOptInfo struct. @ 2020-05-11 07:50 一挃 <[email protected]> 0 siblings, 0 replies; 91+ messages in thread From: 一挃 @ 2020-05-11 07:50 UTC (permalink / raw) UniqueKey is a set of exprs on RelOptInfo which represents the exprs will be unique on the given RelOptInfo. You can see README.uniquekey for more information. --- src/backend/nodes/copyfuncs.c | 13 + src/backend/nodes/list.c | 31 + src/backend/nodes/makefuncs.c | 13 + src/backend/nodes/outfuncs.c | 11 + src/backend/nodes/readfuncs.c | 10 + src/backend/optimizer/path/Makefile | 3 +- src/backend/optimizer/path/README.uniquekey | 131 +++ src/backend/optimizer/path/allpaths.c | 10 + src/backend/optimizer/path/joinpath.c | 9 +- src/backend/optimizer/path/joinrels.c | 2 + src/backend/optimizer/path/pathkeys.c | 3 +- src/backend/optimizer/path/uniquekeys.c | 1131 +++++++++++++++++++ src/backend/optimizer/plan/planner.c | 13 +- src/backend/optimizer/prep/prepunion.c | 2 + src/backend/optimizer/util/appendinfo.c | 44 + src/backend/optimizer/util/inherit.c | 16 +- src/include/nodes/makefuncs.h | 3 + src/include/nodes/nodes.h | 1 + src/include/nodes/pathnodes.h | 29 +- src/include/nodes/pg_list.h | 2 + src/include/optimizer/appendinfo.h | 3 + src/include/optimizer/optimizer.h | 2 + src/include/optimizer/paths.h | 43 + 23 files changed, 1502 insertions(+), 23 deletions(-) create mode 100644 src/backend/optimizer/path/README.uniquekey create mode 100644 src/backend/optimizer/path/uniquekeys.c diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 89c409de66..1f50400fd2 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2273,6 +2273,16 @@ _copyPathKey(const PathKey *from) return newnode; } +static UniqueKey * +_copyUniqueKey(const UniqueKey *from) +{ + UniqueKey *newnode = makeNode(UniqueKey); + + COPY_NODE_FIELD(exprs); + COPY_SCALAR_FIELD(multi_nullvals); + + return newnode; +} /* * _copyRestrictInfo */ @@ -5152,6 +5162,9 @@ copyObjectImpl(const void *from) case T_PathKey: retval = _copyPathKey(from); break; + case T_UniqueKey: + retval = _copyUniqueKey(from); + break; case T_RestrictInfo: retval = _copyRestrictInfo(from); break; diff --git a/src/backend/nodes/list.c b/src/backend/nodes/list.c index 80fa8c84e4..a7a99b70f2 100644 --- a/src/backend/nodes/list.c +++ b/src/backend/nodes/list.c @@ -687,6 +687,37 @@ list_member_oid(const List *list, Oid datum) return false; } +/* + * return true iff every entry in "members" list is also present + * in the "target" list. + */ +bool +list_is_subset(const List *members, const List *target) +{ + const ListCell *lc1, *lc2; + + Assert(IsPointerList(members)); + Assert(IsPointerList(target)); + check_list_invariants(members); + check_list_invariants(target); + + foreach(lc1, members) + { + bool found = false; + foreach(lc2, target) + { + if (equal(lfirst(lc1), lfirst(lc2))) + { + found = true; + break; + } + } + if (!found) + return false; + } + return true; +} + /* * Delete the n'th cell (counting from 0) in list. * diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 49de285f01..646cf7c9a1 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -814,3 +814,16 @@ makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols) v->va_cols = va_cols; return v; } + + +/* + * makeUniqueKey + */ +UniqueKey* +makeUniqueKey(List *exprs, bool multi_nullvals) +{ + UniqueKey * ukey = makeNode(UniqueKey); + ukey->exprs = exprs; + ukey->multi_nullvals = multi_nullvals; + return ukey; +} diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index e2f177515d..c3a9632992 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2428,6 +2428,14 @@ _outPathKey(StringInfo str, const PathKey *node) WRITE_BOOL_FIELD(pk_nulls_first); } +static void +_outUniqueKey(StringInfo str, const UniqueKey *node) +{ + WRITE_NODE_TYPE("UNIQUEKEY"); + WRITE_NODE_FIELD(exprs); + WRITE_BOOL_FIELD(multi_nullvals); +} + static void _outPathTarget(StringInfo str, const PathTarget *node) { @@ -4127,6 +4135,9 @@ outNode(StringInfo str, const void *obj) case T_PathKey: _outPathKey(str, obj); break; + case T_UniqueKey: + _outUniqueKey(str, obj); + break; case T_PathTarget: _outPathTarget(str, obj); break; diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index 42050ab719..3a18571d0c 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -452,6 +452,14 @@ _readSetOperationStmt(void) READ_DONE(); } +static UniqueKey * +_readUniqueKey(void) +{ + READ_LOCALS(UniqueKey); + READ_NODE_FIELD(exprs); + READ_BOOL_FIELD(multi_nullvals); + READ_DONE(); +} /* * Stuff from primnodes.h. @@ -2656,6 +2664,8 @@ parseNodeString(void) return_value = _readCommonTableExpr(); else if (MATCH("SETOPERATIONSTMT", 16)) return_value = _readSetOperationStmt(); + else if (MATCH("UNIQUEKEY", 9)) + return_value = _readUniqueKey(); else if (MATCH("ALIAS", 5)) return_value = _readAlias(); else if (MATCH("RANGEVAR", 8)) diff --git a/src/backend/optimizer/path/Makefile b/src/backend/optimizer/path/Makefile index 1e199ff66f..7b9820c25f 100644 --- a/src/backend/optimizer/path/Makefile +++ b/src/backend/optimizer/path/Makefile @@ -21,6 +21,7 @@ OBJS = \ joinpath.o \ joinrels.o \ pathkeys.o \ - tidpath.o + tidpath.o \ + uniquekeys.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/optimizer/path/README.uniquekey b/src/backend/optimizer/path/README.uniquekey new file mode 100644 index 0000000000..5eac761995 --- /dev/null +++ b/src/backend/optimizer/path/README.uniquekey @@ -0,0 +1,131 @@ +1. What is UniqueKey? +We can think UniqueKey is a set of exprs for a RelOptInfo, which we are insure +that doesn't yields same result among all the rows. The simplest UniqueKey +format is primary key. + +However we define the UnqiueKey as below. + +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + +exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL +is a special case of UniqueKey, which means there is only one row in that +relation.it has a stronger semantic than others. like SELECT uk FROM t; uk is +normal unique key and may have different values. SELECT colx FROM t WHERE uk = +const. colx is unique AND we have only 1 value. This field can used for +innerrel_is_unique. this logic is handled specially in add_uniquekey_for_onerow +function. + +multi_nullvals: true means multi null values may exist in these exprs, so the +uniqueness is not guaranteed in this case. This field is necessary for +remove_useless_join & reduce_unique_semijoins where we don't mind these +duplicated NULL values. It is set to true for 2 cases. One is a unique key +from a unique index but the related column is nullable. The other one is for +outer join. see populate_joinrel_uniquekeys for detail. + + +The UniqueKey can be used at the following cases at least: +1. remove_useless_joins. +2. reduce_semianti_joins +3. remove distinct node if distinct clause is unique. +4. remove aggnode if group by clause is unique. +5. Index Skip Scan (WIP) +6. Aggregation Push Down without 2 phase aggregation if the join can't + duplicated the aggregated rows. (work in progress feature) + +2. How is it maintained? + +We have a set of populate_xxx_unqiuekeys functions to maintain the uniquekey on +various cases. xxx includes baserel, joinrel, partitionedrel, distinctrel, +groupedrel, unionrel. and we also need to convert the uniquekey from subquery +to outer relation, which is what convert_subquery_uniquekeys does. + +1. The first part is about baserel. We handled 3 cases. suppose we have Unique +Index on (a, b). + +1. SELECT a, b FROM t. UniqueKey (a, b) +2. SELECT a FROM t WHERE b = 1; UniqueKey (a) +3. SELECT .. FROM t WHERE a = 1 AND b = 1; UniqueKey (NIL). onerow case, every + column is Unique. + +2. The next part is joinrel, this part is most error-prone, we simplified the rules +like below: +1. If the relation's UniqueKey can't be duplicated after join, then is will be + still valid for the join rel. The function we used here is + innerrel_keeps_unique. The basic idea is innerrel.any_col = outer.uk. + +2. If the UnqiueKey can't keep valid via the rule 1, the combination of the + UniqueKey from both sides are valid for sure. We can prove this as: if the + unique exprs from rel1 is duplicated by rel2, the duplicated rows must + contains different unique exprs from rel2. + +More considerations about onerow: +1. If relation with one row and it can't be duplicated, it is still possible + contains mulit_nullvas after outer join. +2. If the either UniqueKey can be duplicated after join, the can get one row + only when both side is one row AND there is no outer join. +3. Whenever the onerow UniqueKey is not a valid any more, we need to convert one + row UniqueKey to normal unique key since we don't store exprs for one-row + relation. get_exprs_from_uniquekeys will be used here. + + +More considerations about multi_nullvals after join: +1. If the original UnqiueKey has multi_nullvals, the final UniqueKey will have + mulit_nullvals in any case. +2. If a unique key doesn't allow mulit_nullvals, after some outer join, it + allows some outer join. + + +3. When we comes to subquery, we need to convert_subquery_unqiuekeys just like +convert_subquery_pathkeys. Only the UniqueKey insides subquery is referenced as +a Var in outer relation will be reused. The relationship between the outerrel.Var +and subquery.exprs is built with outerel->subroot->processed_tlist. + + +4. As for the SRF functions, it will break the uniqueness of uniquekey, However it +is handled in adjust_paths_for_srfs, which happens after the query_planner. so +we will maintain the UniqueKey until there and reset it to NIL at that +places. This can't help on distinct/group by elimination cases but probably help +in some other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is +semantic correctly. + + +5. As for inherit table, we first main the UnqiueKey on childrel as well. But for +partitioned table we need to maintain 2 different kinds of +UnqiueKey. 1). UniqueKey on the parent relation 2). UniqueKey on child +relation for partition wise query. + +Example: +CREATE TABLE p (a int not null, b int not null) partition by list (a); +CREATE TABLE p0 partition of p for values in (1); +CREATE TABLE p1 partition of p for values in (2); + +create unique index p0_b on p0(b); +create unique index p1_b on p1(b); + +Now b is only unique on partition level, so the distinct can't be removed on +the following cases. SELECT DISTINCT b FROM p; + +Another example is SELECT DISTINCT a, b FROM p WHERE a = 1; Since only one +partition is chosen, the UniqueKey on child relation is same as the UniqueKey on +parent relation. + +Another usage of UniqueKey on partition level is it be helpful for +partition-wise join. + +As for the UniqueKey on parent table level, it comes with 2 different ways, +1). the UniqueKey is also derived in UniqueKey index, but the index must be same +in all the related children relations and the unique index must contains +Partition Key in it. Example: + +CREATE UNIQUE INDEX p_ab ON p(a, b); -- where a is the partition key. + +-- Query +SELECT a, b FROM p; the (a, b) is a UniqueKey of p. + +2). If the parent relation has only one childrel, the UniqueKey on childrel is + the UniqueKey on parent as well. diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index 484dab0a1a..2ad9d06d7a 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -579,6 +579,12 @@ set_plain_rel_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) */ check_index_predicates(root, rel); + /* + * Now that we've marked which partial indexes are suitable, we can now + * build the relation's unique keys. + */ + populate_baserel_uniquekeys(root, rel, rel->indexlist); + /* Mark rel with estimated output rows, width, etc */ set_baserel_size_estimates(root, rel); } @@ -1310,6 +1316,8 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, /* Add paths to the append relation. */ add_paths_to_append_rel(root, rel, live_childrels); + if (IS_PARTITIONED_REL(rel)) + populate_partitionedrel_uniquekeys(root, rel, live_childrels); } @@ -2383,6 +2391,8 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, pathkeys, required_outer)); } + convert_subquery_uniquekeys(root, rel, sub_final_rel); + /* If outer rel allows parallelism, do same for partial paths. */ if (rel->consider_parallel && bms_is_empty(required_outer)) { diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c index db54a6ba2e..ef0fd2fb0b 100644 --- a/src/backend/optimizer/path/joinpath.c +++ b/src/backend/optimizer/path/joinpath.c @@ -71,13 +71,6 @@ static void consider_parallel_mergejoin(PlannerInfo *root, static void hash_inner_and_outer(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, RelOptInfo *innerrel, JoinType jointype, JoinPathExtraData *extra); -static List *select_mergejoin_clauses(PlannerInfo *root, - RelOptInfo *joinrel, - RelOptInfo *outerrel, - RelOptInfo *innerrel, - List *restrictlist, - JoinType jointype, - bool *mergejoin_allowed); static void generate_mergejoin_paths(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *innerrel, @@ -1927,7 +1920,7 @@ hash_inner_and_outer(PlannerInfo *root, * if it is mergejoinable and involves vars from the two sub-relations * currently of interest. */ -static List * +List * select_mergejoin_clauses(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c index 2d343cd293..b9163ee8ff 100644 --- a/src/backend/optimizer/path/joinrels.c +++ b/src/backend/optimizer/path/joinrels.c @@ -924,6 +924,8 @@ populate_joinrel_with_paths(PlannerInfo *root, RelOptInfo *rel1, /* Apply partitionwise join technique, if possible. */ try_partitionwise_join(root, rel1, rel2, joinrel, sjinfo, restrictlist); + + populate_joinrel_uniquekeys(root, joinrel, rel1, rel2, restrictlist, sjinfo->jointype); } diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index ce9bf87e9b..7e596d4194 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -33,7 +33,6 @@ static bool pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys); static bool matches_boolean_partition_clause(RestrictInfo *rinfo, RelOptInfo *partrel, int partkeycol); -static Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); static bool right_merge_direction(PlannerInfo *root, PathKey *pathkey); @@ -1035,7 +1034,7 @@ convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel, * We need this to ensure that we don't return pathkeys describing values * that are unavailable above the level of the subquery scan. */ -static Var * +Var * find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle) { ListCell *lc; diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c new file mode 100644 index 0000000000..b33bcd2f32 --- /dev/null +++ b/src/backend/optimizer/path/uniquekeys.c @@ -0,0 +1,1131 @@ +/*------------------------------------------------------------------------- + * + * uniquekeys.c + * Utilities for matching and building unique keys + * + * Portions Copyright (c) 2020, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/optimizer/path/uniquekeys.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "nodes/makefuncs.h" +#include "nodes/nodeFuncs.h" +#include "optimizer/pathnode.h" +#include "optimizer/paths.h" +#include "optimizer/appendinfo.h" +#include "optimizer/optimizer.h" +#include "optimizer/tlist.h" +#include "rewrite/rewriteManip.h" + + +/* + * This struct is used to help populate_joinrel_uniquekeys. + * + * added_to_joinrel is true if a uniquekey (from outerrel or innerrel) + * has been added to joinrel. + * useful is true if the exprs of the uniquekey still appears in joinrel. + */ +typedef struct UniqueKeyContextData +{ + UniqueKey *uniquekey; + bool added_to_joinrel; + bool useful; +} *UniqueKeyContext; + +static List *initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel); +static bool innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + bool reverse); + +static List *get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals); +static List *get_exprs_from_uniquekey(RelOptInfo *joinrel, + RelOptInfo *rel1, + UniqueKey *ukey); +static void add_uniquekey_for_onerow(RelOptInfo *rel); +static bool add_combined_uniquekey(RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype); + +/* Used for unique indexes checking for partitioned table */ +static bool index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind); +static IndexOptInfo *simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from); +static bool simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2); +static void adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_index); + +/* Helper function for grouped relation and distinct relation. */ +static void add_uniquekey_from_sortgroups(PlannerInfo *root, + RelOptInfo *rel, + List *sortgroups); + +/* + * populate_baserel_uniquekeys + * Populate 'baserel' uniquekeys list by looking at the rel's unique index + * and baserestrictinfo + */ +void +populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List *indexlist) +{ + ListCell *lc; + List *matched_uniq_indexes = NIL; + + /* Attrs appears in rel->reltarget->exprs. */ + Bitmapset *used_attrs = NULL; + + List *const_exprs = NIL; + List *expr_opfamilies = NIL; + + Assert(baserel->rtekind == RTE_RELATION); + + foreach(lc, indexlist) + { + IndexOptInfo *ind = (IndexOptInfo *) lfirst(lc); + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + matched_uniq_indexes = lappend(matched_uniq_indexes, ind); + } + + if (matched_uniq_indexes == NIL) + return; + + /* Check which attrs is used in baserel->reltarget */ + pull_varattnos((Node *)baserel->reltarget->exprs, baserel->relid, &used_attrs); + + /* Check which attrno is used at a mergeable const filter */ + foreach(lc, baserel->baserestrictinfo) + { + RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); + + if (rinfo->mergeopfamilies == NIL) + continue; + + if (bms_is_empty(rinfo->left_relids)) + { + const_exprs = lappend(const_exprs, get_rightop(rinfo->clause)); + } + else if (bms_is_empty(rinfo->right_relids)) + { + const_exprs = lappend(const_exprs, get_leftop(rinfo->clause)); + } + else + continue; + + expr_opfamilies = lappend(expr_opfamilies, rinfo->mergeopfamilies); + } + + foreach(lc, matched_uniq_indexes) + { + bool multi_nullvals, useful; + List *exprs = get_exprs_from_uniqueindex(lfirst_node(IndexOptInfo, lc), + const_exprs, + expr_opfamilies, + used_attrs, + &useful, + &multi_nullvals); + if (useful) + { + if (exprs == NIL) + { + /* All the columns in Unique Index matched with a restrictinfo */ + add_uniquekey_for_onerow(baserel); + return; + } + baserel->uniquekeys = lappend(baserel->uniquekeys, + makeUniqueKey(exprs, multi_nullvals)); + } + } +} + + +/* + * populate_partitionedrel_uniquekeys + * The UniqueKey on partitionrel comes from 2 cases: + * 1). Only one partition is involved in this query, the unique key can be + * copied to parent rel from childrel. + * 2). There are some unique index which includes partition key and exists + * in all the related partitions. + * We never mind rule 2 if we hit rule 1. + */ + +void +populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels) +{ + ListCell *lc; + List *global_uniq_indexlist = NIL; + RelOptInfo *childrel; + bool is_first = true; + + Assert(IS_PARTITIONED_REL(rel)); + + if (childrels == NIL) + return; + + /* + * If there is only one partition used in this query, the UniqueKey in childrel is + * still valid in parent level, but we need convert the format from child expr to + * parent expr. + */ + if (list_length(childrels) == 1) + { + /* Check for Rule 1 */ + RelOptInfo *childrel = linitial_node(RelOptInfo, childrels); + ListCell *lc; + Assert(childrel->reloptkind == RELOPT_OTHER_MEMBER_REL); + if (relation_is_onerow(childrel)) + { + add_uniquekey_for_onerow(rel); + return; + } + + foreach(lc, childrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + AppendRelInfo *appinfo = find_appinfo_by_child(root, childrel->relid); + List *parent_exprs = NIL; + bool can_reuse = true; + ListCell *lc2; + foreach(lc2, ukey->exprs) + { + Var *var = (Var *)lfirst(lc2); + /* + * If the expr comes from a expression, it is hard to build the expression + * in parent so ignore that case for now. + */ + if(!IsA(var, Var)) + { + can_reuse = false; + break; + } + /* Convert it to parent var */ + parent_exprs = lappend(parent_exprs, find_parent_var(appinfo, var)); + } + if (can_reuse) + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(parent_exprs, + ukey->multi_nullvals)); + } + } + else + { + /* Check for rule 2 */ + childrel = linitial_node(RelOptInfo, childrels); + foreach(lc, childrel->indexlist) + { + IndexOptInfo *ind = lfirst(lc); + IndexOptInfo *modified_index; + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + + /* + * During simple_copy_indexinfo_to_parent, we need to convert var from + * child var to parent var, index on expression is too complex to handle. + * so ignore it for now. + */ + if (ind->indexprs != NIL) + continue; + + modified_index = simple_copy_indexinfo_to_parent(root, rel, ind); + /* + * If the unique index doesn't contain partkey, then it is unique + * on this partition only, so it is useless for us. + */ + if (!index_constains_partkey(rel, modified_index)) + continue; + + global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index); + } + + if (global_uniq_indexlist != NIL) + { + foreach(lc, childrels) + { + RelOptInfo *child = lfirst(lc); + if (is_first) + { + is_first = false; + continue; + } + adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist); + } + /* Now we have a list of unique index which are exactly same on all childrels, + * Set the UniqueKey just like it is non-partition table + */ + populate_baserel_uniquekeys(root, rel, global_uniq_indexlist); + } + } +} + + +/* + * populate_distinctrel_uniquekeys + */ +void +populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel) +{ + /* The unique key before the distinct is still valid. */ + distinctrel->uniquekeys = list_copy(inputrel->uniquekeys); + add_uniquekey_from_sortgroups(root, distinctrel, root->parse->distinctClause); +} + +/* + * populate_grouprel_uniquekeys + */ +void +populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel) + +{ + Query *parse = root->parse; + bool input_ukey_added = false; + ListCell *lc; + + if (relation_is_onerow(inputrel)) + { + add_uniquekey_for_onerow(grouprel); + return; + } + if (parse->groupingSets) + return; + + /* A Normal group by without grouping set. */ + if (parse->groupClause) + { + /* + * Current even the groupby clause is Unique already, but if query has aggref + * We have to create grouprel still. To keep the UnqiueKey short, we will check + * the UniqueKey of input_rel still valid, if so we reuse it. + */ + foreach(lc, inputrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (list_is_subset(ukey->exprs, grouprel->reltarget->exprs)) + { + grouprel->uniquekeys = lappend(grouprel->uniquekeys, + ukey); + input_ukey_added = true; + } + } + if (!input_ukey_added) + /* + * group by clause must be a super-set of grouprel->reltarget->exprs except the + * aggregation expr, so if such exprs is unique already, no bother to generate + * new uniquekey for group by exprs. + */ + add_uniquekey_from_sortgroups(root, + grouprel, + root->parse->groupClause); + } + else + /* It has aggregation but without a group by, so only one row returned */ + add_uniquekey_for_onerow(grouprel); +} + +/* + * simple_copy_uniquekeys + * Using a function for the one-line code makes us easy to check where we simply + * copied the uniquekey. + */ +void +simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel) +{ + newrel->uniquekeys = oldrel->uniquekeys; +} + +/* + * populate_unionrel_uniquekeys + */ +void +populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel) +{ + ListCell *lc; + List *exprs = NIL; + + Assert(unionrel->uniquekeys == NIL); + + foreach(lc, unionrel->reltarget->exprs) + { + exprs = lappend(exprs, lfirst(lc)); + } + + if (exprs == NIL) + /* SQL: select union select; is valid, we need to handle it here. */ + add_uniquekey_for_onerow(unionrel); + else + unionrel->uniquekeys = lappend(unionrel->uniquekeys, + makeUniqueKey(exprs,false)); + +} + +/* + * populate_joinrel_uniquekeys + * + * populate uniquekeys for joinrel. We will check each relation to see if its + * UniqueKey is still valid via innerrel_keeps_unique, if so, we add it to + * joinrel. The multi_nullvals field will be changed to true for some outer + * join cases and one-row UniqueKey needs to be converted to normal UniqueKey + * for the same case as well. + * For the uniquekey in either baserel which can't be unique after join, we still + * check to see if combination of UniqueKeys from both side is still useful for us. + * if yes, we add it to joinrel as well. + */ +void +populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *outerrel, RelOptInfo *innerrel, + List *restrictlist, JoinType jointype) +{ + ListCell *lc, *lc2; + List *clause_list = NIL; + List *outerrel_ukey_ctx; + List *innerrel_ukey_ctx; + bool inner_onerow, outer_onerow; + bool mergejoin_allowed; + + /* Care about the outerrel relation only for SEMI/ANTI join */ + if (jointype == JOIN_SEMI || jointype == JOIN_ANTI) + { + foreach(lc, outerrel->uniquekeys) + { + UniqueKey *uniquekey = lfirst_node(UniqueKey, lc); + if (list_is_subset(uniquekey->exprs, joinrel->reltarget->exprs)) + joinrel->uniquekeys = lappend(joinrel->uniquekeys, uniquekey); + } + return; + } + + Assert(jointype == JOIN_LEFT || jointype == JOIN_FULL || jointype == JOIN_INNER); + + /* Fast path */ + if (innerrel->uniquekeys == NIL || outerrel->uniquekeys == NIL) + return; + + inner_onerow = relation_is_onerow(innerrel); + outer_onerow = relation_is_onerow(outerrel); + + outerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(outerrel); + innerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(innerrel); + + clause_list = select_mergejoin_clauses(root, joinrel, outerrel, innerrel, + restrictlist, jointype, + &mergejoin_allowed); + + if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true /* reverse */)) + { + bool outer_impact = jointype == JOIN_FULL; + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + /* Outer relation has one row, and the unique key is not duplicated after join, + * the joinrel will still has one row unless the jointype == JOIN_FULL. + */ + if (outer_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (outer_onerow) + { + /* + * The onerow outerrel becomes multi rows and multi_nullvals + * will be changed to true. We also need to set the exprs correctly since it + * can't be NIL any more. + */ + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(joinrel, outerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Change multi_nullvals to true due to the full join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, true)); + else + /* Just reuse it */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + } + ctx->added_to_joinrel = true; + } + } + + if (innerrel_keeps_unique(root, outerrel, innerrel, clause_list, false)) + { + bool outer_impact = jointype == JOIN_FULL || jointype == JOIN_LEFT;; + + foreach(lc, innerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + if (inner_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (inner_onerow) + { + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(joinrel, innerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Need to change multi_nullvals to true due to the outer join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, + true)); + else + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + + } + ctx->added_to_joinrel = true; + } + } + + /* + * The combination of the UniqueKey from both sides is unique as well regardless + * of join type, but no bother to add it if its subset has been added to joinrel + * already or it is not useful for the joinrel. + */ + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx1 = (UniqueKeyContext) lfirst(lc); + if (ctx1->added_to_joinrel || !ctx1->useful) + continue; + foreach(lc2, innerrel_ukey_ctx) + { + UniqueKeyContext ctx2 = (UniqueKeyContext) lfirst(lc2); + if (ctx2->added_to_joinrel || !ctx2->useful) + continue; + if (add_combined_uniquekey(joinrel, outerrel, innerrel, + ctx1->uniquekey, ctx2->uniquekey, + jointype)) + /* If we set a onerow UniqueKey to joinrel, we don't need other. */ + return; + } + } +} + + +/* + * convert_subquery_uniquekeys + * + * Covert the UniqueKey in subquery to outer relation. + */ +void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel) +{ + ListCell *lc; + + if (sub_final_rel->uniquekeys == NIL) + return; + + if (relation_is_onerow(sub_final_rel)) + { + add_uniquekey_for_onerow(currel); + return; + } + + Assert(currel->subroot != NULL); + + foreach(lc, sub_final_rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + ListCell *lc; + List *exprs = NIL; + bool ukey_useful = true; + + /* One row case is handled above */ + Assert(ukey->exprs != NIL); + foreach(lc, ukey->exprs) + { + Var *var; + TargetEntry *tle = tlist_member(lfirst(lc), + currel->subroot->processed_tlist); + if (tle == NULL) + { + ukey_useful = false; + break; + } + var = find_var_for_subquery_tle(currel, tle); + if (var == NULL) + { + ukey_useful = false; + break; + } + exprs = lappend(exprs, var); + } + + if (ukey_useful) + currel->uniquekeys = lappend(currel->uniquekeys, + makeUniqueKey(exprs, + ukey->multi_nullvals)); + + } +} + +/* + * innerrel_keeps_unique + * + * Check if Unique key of the innerrel is valid after join. innerrel's UniqueKey + * will be still valid if innerrel's any-column mergeop outrerel's uniquekey + * exists in clause_list. + * + * Note: the clause_list must be a list of mergeable restrictinfo already. + */ +static bool +innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *clause_list, + bool reverse) +{ + ListCell *lc, *lc2, *lc3; + + if (outerrel->uniquekeys == NIL || innerrel->uniquekeys == NIL) + return false; + + /* Check if there is outerrel's uniquekey in mergeable clause. */ + foreach(lc, outerrel->uniquekeys) + { + List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs; + bool clauselist_matchs_all_exprs = true; + foreach(lc2, outer_uq_exprs) + { + Node *outer_uq_expr = lfirst(lc2); + bool find_uq_expr_in_clauselist = false; + foreach(lc3, clause_list) + { + RestrictInfo *rinfo = lfirst_node(RestrictInfo, lc3); + Node *outer_expr; + if (reverse) + outer_expr = rinfo->outer_is_left ? get_rightop(rinfo->clause) : get_leftop(rinfo->clause); + else + outer_expr = rinfo->outer_is_left ? get_leftop(rinfo->clause) : get_rightop(rinfo->clause); + if (equal(outer_expr, outer_uq_expr)) + { + find_uq_expr_in_clauselist = true; + break; + } + } + if (!find_uq_expr_in_clauselist) + { + /* No need to check the next exprs in the current uniquekey */ + clauselist_matchs_all_exprs = false; + break; + } + } + + if (clauselist_matchs_all_exprs) + return true; + } + return false; +} + + +/* + * relation_is_onerow + * Check if it is a one-row relation by checking UniqueKey. + */ +bool +relation_is_onerow(RelOptInfo *rel) +{ + UniqueKey *ukey; + if (rel->uniquekeys == NIL) + return false; + ukey = linitial_node(UniqueKey, rel->uniquekeys); + return ukey->exprs == NIL && list_length(rel->uniquekeys) == 1; +} + +/* + * relation_has_uniquekeys_for + * Returns true if we have proofs that 'rel' cannot return multiple rows with + * the same values in each of 'exprs'. Otherwise returns false. + */ +bool +relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel, + List *exprs, bool allow_multinulls) +{ + ListCell *lc; + + /* + * For UniqueKey->onerow case, the uniquekey->exprs is empty as well + * so we can't rely on list_is_subset to handle this special cases + */ + if (exprs == NIL) + return false; + + foreach(lc, rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (ukey->multi_nullvals && !allow_multinulls) + continue; + if (list_is_subset(ukey->exprs, exprs)) + return true; + } + return false; +} + + +/* + * get_exprs_from_uniqueindex + * + * Return a list of exprs which is unique. set useful to false if this + * unique index is not useful for us. + */ +static List * +get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals) +{ + List *exprs = NIL; + ListCell *indexpr_item; + int c = 0; + + *useful = true; + *multi_nullvals = false; + + indexpr_item = list_head(unique_index->indexprs); + for(c = 0; c < unique_index->ncolumns; c++) + { + int attr = unique_index->indexkeys[c]; + Expr *expr; + bool matched_const = false; + ListCell *lc1, *lc2; + + if(attr > 0) + { + expr = list_nth_node(TargetEntry, unique_index->indextlist, c)->expr; + } + else if (attr == 0) + { + /* Expression index */ + expr = lfirst(indexpr_item); + indexpr_item = lnext(unique_index->indexprs, indexpr_item); + } + else /* attr < 0 */ + { + /* Index on system column is not supported */ + Assert(false); + } + + /* + * Check index_col = Const case with regarding to opfamily checking + * If we can remove the index_col from the final UniqueKey->exprs. + */ + forboth(lc1, const_exprs, lc2, const_expr_opfamilies) + { + if (list_member_oid((List *)lfirst(lc2), unique_index->opfamily[c]) + && match_index_to_operand((Node *) lfirst(lc1), c, unique_index)) + { + matched_const = true; + break; + } + } + + if (matched_const) + continue; + + /* Check if the indexed expr is used in rel */ + if (attr > 0) + { + /* + * Normal Indexed column, if the col is not used, then the index is useless + * for uniquekey. + */ + attr -= FirstLowInvalidHeapAttributeNumber; + + if (!bms_is_member(attr, used_varattrs)) + { + *useful = false; + break; + } + } + else if (!list_member(unique_index->rel->reltarget->exprs, expr)) + { + /* Expression index but the expression is not used in rel */ + *useful = false; + break; + } + + /* check not null property. */ + if (attr == 0) + { + /* We never know if a expression yields null or not */ + *multi_nullvals = true; + } + else if (!bms_is_member(attr, unique_index->rel->notnullattrs) + && !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber, + unique_index->rel->notnullattrs)) + { + *multi_nullvals = true; + } + + exprs = lappend(exprs, expr); + } + return exprs; +} + + +/* + * add_uniquekey_for_onerow + * If we are sure that the relation only returns one row, then all the columns + * are unique. However we don't need to create UniqueKey for every column, we + * just set exprs = NIL and overwrites all the other UniqueKey on this RelOptInfo + * since this one has strongest semantics. + */ +void +add_uniquekey_for_onerow(RelOptInfo *rel) +{ + /* + * We overwrite the previous UniqueKey on purpose since this one has the + * strongest semantic. + */ + rel->uniquekeys = list_make1(makeUniqueKey(NIL, false)); +} + + +/* + * initililze_uniquecontext_for_joinrel + * Return a List of UniqueKeyContext for an inputrel + */ +static List * +initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel) +{ + List *res = NIL; + ListCell *lc; + foreach(lc, inputrel->uniquekeys) + { + UniqueKeyContext context; + context = palloc(sizeof(struct UniqueKeyContextData)); + context->uniquekey = lfirst_node(UniqueKey, lc); + context->added_to_joinrel = false; + context->useful = true; + res = lappend(res, context); + } + return res; +} + + +/* + * get_exprs_from_uniquekey + * Unify the way of get List of exprs from a one-row UniqueKey or + * normal UniqueKey. for the onerow case, every expr in rel1 is a valid + * UniqueKey. Return a List of exprs. + * + * rel1: The relation which you want to get the exprs. + * ukey: The UniqueKey you want to get the exprs. + */ +static List * +get_exprs_from_uniquekey(RelOptInfo *joinrel, RelOptInfo *rel1, UniqueKey *ukey) +{ + ListCell *lc; + bool onerow = rel1 != NULL && relation_is_onerow(rel1); + + List *res = NIL; + Assert(onerow || ukey); + if (onerow) + { + /* Only cares about the exprs still exist in joinrel */ + foreach(lc, joinrel->reltarget->exprs) + { + Bitmapset *relids = pull_varnos(lfirst(lc)); + if (bms_is_subset(relids, rel1->relids)) + { + res = lappend(res, list_make1(lfirst(lc))); + } + } + } + else + { + res = list_make1(ukey->exprs); + } + return res; +} + +/* + * Partitioned table Unique Keys. + * The partition table unique key is maintained as: + * 1. The index must be unique as usual. + * 2. The index must contains partition key. + * 3. The index must exist on all the child rel. see simple_indexinfo_equal for + * how we compare it. + */ + +/* + * index_constains_partkey + * return true if the index contains the partiton key. + */ +static bool +index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind) +{ + ListCell *lc; + int i; + Assert(IS_PARTITIONED_REL(partrel)); + Assert(partrel->part_scheme->partnatts > 0); + + for(i = 0; i < partrel->part_scheme->partnatts; i++) + { + Node *part_expr = linitial(partrel->partexprs[i]); + bool found_in_index = false; + foreach(lc, ind->indextlist) + { + Expr *index_expr = lfirst_node(TargetEntry, lc)->expr; + if (equal(index_expr, part_expr)) + { + found_in_index = true; + break; + } + } + if (!found_in_index) + return false; + } + return true; +} + +/* + * simple_indexinfo_equal + * + * Used to check if the 2 index is same as each other. The index here + * is COPIED from childrel and did some tiny changes(see + * simple_copy_indexinfo_to_parent) + */ +static bool +simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2) +{ + Size oid_cmp_len = sizeof(Oid) * ind1->ncolumns; + + return ind1->ncolumns == ind2->ncolumns && + ind1->unique == ind2->unique && + memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0 && + memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0 && + memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0 && + memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0 && + equal(get_tlist_exprs(ind1->indextlist, true), + get_tlist_exprs(ind2->indextlist, true)); +} + + +/* + * The below macros are used for simple_copy_indexinfo_to_parent which is so + * customized that I don't want to put it to copyfuncs.c. So copy it here. + */ +#define COPY_POINTER_FIELD(fldname, sz) \ + do { \ + Size _size = (sz); \ + newnode->fldname = palloc(_size); \ + memcpy(newnode->fldname, from->fldname, _size); \ + } while (0) + +#define COPY_NODE_FIELD(fldname) \ + (newnode->fldname = copyObjectImpl(from->fldname)) + +#define COPY_SCALAR_FIELD(fldname) \ + (newnode->fldname = from->fldname) + + +/* + * simple_copy_indexinfo_to_parent (from partition) + * Copy the IndexInfo from child relation to parent relation with some modification, + * which is used to test: + * 1. If the same index exists in all the childrels. + * 2. If the parentrel->reltarget/basicrestrict info matches this index. + */ +static IndexOptInfo * +simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from) +{ + IndexOptInfo *newnode = makeNode(IndexOptInfo); + AppendRelInfo *appinfo = find_appinfo_by_child(root, from->rel->relid); + ListCell *lc; + int idx = 0; + + COPY_SCALAR_FIELD(ncolumns); + COPY_SCALAR_FIELD(nkeycolumns); + COPY_SCALAR_FIELD(unique); + COPY_SCALAR_FIELD(immediate); + /* We just need to know if it is NIL or not */ + COPY_SCALAR_FIELD(indpred); + COPY_SCALAR_FIELD(predOK); + COPY_POINTER_FIELD(indexkeys, from->ncolumns * sizeof(int)); + COPY_POINTER_FIELD(indexcollations, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opfamily, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opcintype, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(sortopfamily, from->ncolumns * sizeof(Oid)); + COPY_NODE_FIELD(indextlist); + + /* Convert index exprs on child expr to expr on parent */ + foreach(lc, newnode->indextlist) + { + TargetEntry *tle = lfirst_node(TargetEntry, lc); + /* Index on expression is ignored */ + Assert(IsA(tle->expr, Var)); + tle->expr = (Expr *) find_parent_var(appinfo, (Var *) tle->expr); + newnode->indexkeys[idx] = castNode(Var, tle->expr)->varattno; + idx++; + } + newnode->rel = parentrel; + return newnode; +} + +/* + * adjust_partition_unique_indexlist + * + * global_unique_indexes: At the beginning, it contains the copy & modified + * unique index from the first partition. And then check if each index in it still + * exists in the following partitions. If no, remove it. at last, it has an + * index list which exists in all the partitions. + */ +static void +adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_indexes) +{ + ListCell *lc, *lc2; + foreach(lc, *global_unique_indexes) + { + IndexOptInfo *g_ind = lfirst_node(IndexOptInfo, lc); + bool found_in_child = false; + + foreach(lc2, childrel->indexlist) + { + IndexOptInfo *p_ind = lfirst_node(IndexOptInfo, lc2); + IndexOptInfo *p_ind_copy; + if (!p_ind->unique || !p_ind->immediate || + (p_ind->indpred != NIL && !p_ind->predOK)) + continue; + p_ind_copy = simple_copy_indexinfo_to_parent(root, parentrel, p_ind); + if (simple_indexinfo_equal(p_ind_copy, g_ind)) + { + found_in_child = true; + break; + } + } + if (!found_in_child) + /* The index doesn't exist in childrel, remove it from global_unique_indexes */ + *global_unique_indexes = foreach_delete_current(*global_unique_indexes, lc); + } +} + +/* Helper function for groupres/distinctrel */ +static void +add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgroups) +{ + Query *parse = root->parse; + List *exprs; + + /* + * XXX: If there are some vars which is not in current levelsup, the semantic is + * imprecise, should we avoid it or not? levelsup = 1 is just a demo, maybe we need to + * check every level other than 0, if so, looks we have to write another + * pull_var_walker. + */ + List *upper_vars = pull_vars_of_level((Node*)sortgroups, 1); + + if (upper_vars != NIL) + return; + + exprs = get_sortgrouplist_exprs(sortgroups, parse->targetList); + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(exprs, + false /* sortgroupclause can't be multi_nullvals */)); +} + + +/* + * add_combined_uniquekey + * The combination of both UniqueKeys is a valid UniqueKey for joinrel no matter + * the jointype. + */ +bool +add_combined_uniquekey(RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype) +{ + + ListCell *lc1, *lc2; + + /* Either side has multi_nullvals or we have outer join, + * the combined UniqueKey has multi_nullvals */ + bool multi_nullvals = outer_ukey->multi_nullvals || + inner_ukey->multi_nullvals || IS_OUTER_JOIN(jointype); + + /* The only case we can get onerow joinrel after join */ + if (relation_is_onerow(outer_rel) + && relation_is_onerow(inner_rel) + && jointype == JOIN_INNER) + { + add_uniquekey_for_onerow(joinrel); + return true; + } + + foreach(lc1, get_exprs_from_uniquekey(joinrel, outer_rel, outer_ukey)) + { + foreach(lc2, get_exprs_from_uniquekey(joinrel, inner_rel, inner_ukey)) + { + List *exprs = list_concat_copy(lfirst_node(List, lc1), lfirst_node(List, lc2)); + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(exprs, + multi_nullvals)); + } + } + return false; +} diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index b406d41e91..0551ae0512 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -2389,6 +2389,8 @@ grouping_planner(PlannerInfo *root, bool inheritance_update, add_path(final_rel, path); } + simple_copy_uniquekeys(current_rel, final_rel); + /* * Generate partial paths for final_rel, too, if outer query levels might * be able to make use of them. @@ -3899,6 +3901,8 @@ create_grouping_paths(PlannerInfo *root, } set_cheapest(grouped_rel); + + populate_grouprel_uniquekeys(root, grouped_rel, input_rel); return grouped_rel; } @@ -4615,7 +4619,7 @@ create_window_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(window_rel); - + simple_copy_uniquekeys(input_rel, window_rel); return window_rel; } @@ -4911,7 +4915,7 @@ create_distinct_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(distinct_rel); - + populate_distinctrel_uniquekeys(root, input_rel, distinct_rel); return distinct_rel; } @@ -5172,6 +5176,8 @@ create_ordered_paths(PlannerInfo *root, */ Assert(ordered_rel->pathlist != NIL); + simple_copy_uniquekeys(input_rel, ordered_rel); + return ordered_rel; } @@ -6049,6 +6055,9 @@ adjust_paths_for_srfs(PlannerInfo *root, RelOptInfo *rel, if (list_length(targets) == 1) return; + /* UniqueKey is not valid after handling the SRF. */ + rel->uniquekeys = NIL; + /* * Stack SRF-evaluation nodes atop each path for the rel. * diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c index 951aed80e7..e94e92937c 100644 --- a/src/backend/optimizer/prep/prepunion.c +++ b/src/backend/optimizer/prep/prepunion.c @@ -689,6 +689,8 @@ generate_union_paths(SetOperationStmt *op, PlannerInfo *root, /* Undo effects of possibly forcing tuple_fraction to 0 */ root->tuple_fraction = save_fraction; + /* Add the UniqueKeys */ + populate_unionrel_uniquekeys(root, result_rel); return result_rel; } diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c index d722063cf3..44c37ecffc 100644 --- a/src/backend/optimizer/util/appendinfo.c +++ b/src/backend/optimizer/util/appendinfo.c @@ -746,3 +746,47 @@ find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos) } return appinfos; } + +/* + * find_appinfo_by_child + * + */ +AppendRelInfo * +find_appinfo_by_child(PlannerInfo *root, Index child_index) +{ + ListCell *lc; + foreach(lc, root->append_rel_list) + { + AppendRelInfo *appinfo = lfirst_node(AppendRelInfo, lc); + if (appinfo->child_relid == child_index) + return appinfo; + } + elog(ERROR, "parent relation cant be found"); + return NULL; +} + +/* + * find_parent_var + * + */ +Var * +find_parent_var(AppendRelInfo *appinfo, Var *child_var) +{ + ListCell *lc; + Var *res = NULL; + Index attno = 1; + foreach(lc, appinfo->translated_vars) + { + Node *child_node = lfirst(lc); + if (equal(child_node, child_var)) + { + res = copyObject(child_var); + res->varattno = attno; + res->varno = appinfo->parent_relid; + } + attno++; + } + if (res == NULL) + elog(ERROR, "parent var can't be found."); + return res; +} diff --git a/src/backend/optimizer/util/inherit.c b/src/backend/optimizer/util/inherit.c index 3132fd35a5..d66b40ec50 100644 --- a/src/backend/optimizer/util/inherit.c +++ b/src/backend/optimizer/util/inherit.c @@ -736,6 +736,7 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, { Node *onecq = (Node *) lfirst(lc2); bool pseudoconstant; + RestrictInfo *child_rinfo; /* check for pseudoconstant (no Vars or volatile functions) */ pseudoconstant = @@ -747,13 +748,14 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, root->hasPseudoConstantQuals = true; } /* reconstitute RestrictInfo with appropriate properties */ - childquals = lappend(childquals, - make_restrictinfo((Expr *) onecq, - rinfo->is_pushed_down, - rinfo->outerjoin_delayed, - pseudoconstant, - rinfo->security_level, - NULL, NULL, NULL)); + child_rinfo = make_restrictinfo((Expr *) onecq, + rinfo->is_pushed_down, + rinfo->outerjoin_delayed, + pseudoconstant, + rinfo->security_level, + NULL, NULL, NULL); + child_rinfo->mergeopfamilies = rinfo->mergeopfamilies; + childquals = lappend(childquals, child_rinfo); /* track minimum security level among child quals */ cq_min_security = Min(cq_min_security, rinfo->security_level); } diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 31d9aedeeb..c83f17acb7 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -16,6 +16,7 @@ #include "nodes/execnodes.h" #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" extern A_Expr *makeA_Expr(A_Expr_Kind kind, List *name, @@ -105,4 +106,6 @@ extern GroupingSet *makeGroupingSet(GroupingSetKind kind, List *content, int loc extern VacuumRelation *makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols); +extern UniqueKey* makeUniqueKey(List *exprs, bool multi_nullvals); + #endif /* MAKEFUNC_H */ diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 381d84b4e4..41110ed888 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -264,6 +264,7 @@ typedef enum NodeTag T_EquivalenceMember, T_PathKey, T_PathTarget, + T_UniqueKey, T_RestrictInfo, T_IndexClause, T_PlaceHolderVar, diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index 9e3ebd488a..02e4458bef 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -730,6 +730,7 @@ typedef struct RelOptInfo QualCost baserestrictcost; /* cost of evaluating the above */ Index baserestrict_min_security; /* min security_level found in * baserestrictinfo */ + List *uniquekeys; /* List of UniqueKey */ List *joininfo; /* RestrictInfo structures for join clauses * involving this rel */ bool has_eclass_joins; /* T means joininfo is incomplete */ @@ -1047,6 +1048,28 @@ typedef struct PathKey } PathKey; +/* + * UniqueKey + * + * Represents the unique properties held by a RelOptInfo. + * + * exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL + * is a special case of UniqueKey, which means there is only 1 row in that + * relation. + * multi_nullvals: true means multi null values may exist in these exprs, so the + * uniqueness is not guaranteed in this case. This field is necessary for + * remove_useless_join & reduce_unique_semijoins where we don't mind these + * duplicated NULL values. It is set to true for 2 cases. One is a unique key + * from a unique index but the related column is nullable. The other one is for + * outer join. see populate_joinrel_uniquekeys for detail. + */ +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + /* * PathTarget * @@ -2473,7 +2496,7 @@ typedef enum * * flags indicating what kinds of grouping are possible. * partial_costs_set is true if the agg_partial_costs and agg_final_costs - * have been initialized. + * have been initialized. * agg_partial_costs gives partial aggregation costs. * agg_final_costs gives finalization costs. * target_parallel_safe is true if target is parallel safe. @@ -2503,8 +2526,8 @@ typedef struct * limit_tuples is an estimated bound on the number of output tuples, * or -1 if no LIMIT or couldn't estimate. * count_est and offset_est are the estimated values of the LIMIT and OFFSET - * expressions computed by preprocess_limit() (see comments for - * preprocess_limit() for more information). + * expressions computed by preprocess_limit() (see comments for + * preprocess_limit() for more information). */ typedef struct { diff --git a/src/include/nodes/pg_list.h b/src/include/nodes/pg_list.h index 14ea2766ad..621f54a9f8 100644 --- a/src/include/nodes/pg_list.h +++ b/src/include/nodes/pg_list.h @@ -528,6 +528,8 @@ extern bool list_member_ptr(const List *list, const void *datum); extern bool list_member_int(const List *list, int datum); extern bool list_member_oid(const List *list, Oid datum); +extern bool list_is_subset(const List *members, const List *target); + extern List *list_delete(List *list, void *datum); extern List *list_delete_ptr(List *list, void *datum); extern List *list_delete_int(List *list, int datum); diff --git a/src/include/optimizer/appendinfo.h b/src/include/optimizer/appendinfo.h index d6a27a60dd..e87c92a054 100644 --- a/src/include/optimizer/appendinfo.h +++ b/src/include/optimizer/appendinfo.h @@ -32,4 +32,7 @@ extern Relids adjust_child_relids_multilevel(PlannerInfo *root, Relids relids, extern AppendRelInfo **find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos); +extern AppendRelInfo *find_appinfo_by_child(PlannerInfo *root, Index child_index); +extern Var *find_parent_var(AppendRelInfo *appinfo, Var *child_var); + #endif /* APPENDINFO_H */ diff --git a/src/include/optimizer/optimizer.h b/src/include/optimizer/optimizer.h index 3e4171056e..9445141263 100644 --- a/src/include/optimizer/optimizer.h +++ b/src/include/optimizer/optimizer.h @@ -23,6 +23,7 @@ #define OPTIMIZER_H #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" /* * We don't want to include nodes/pathnodes.h here, because non-planner @@ -156,6 +157,7 @@ extern TargetEntry *get_sortgroupref_tle(Index sortref, List *targetList); extern TargetEntry *get_sortgroupclause_tle(SortGroupClause *sgClause, List *targetList); +extern Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); extern Node *get_sortgroupclause_expr(SortGroupClause *sgClause, List *targetList); extern List *get_sortgrouplist_exprs(List *sgClauses, diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h index 10b6e81079..9217a8d6c6 100644 --- a/src/include/optimizer/paths.h +++ b/src/include/optimizer/paths.h @@ -240,5 +240,48 @@ extern PathKey *make_canonical_pathkey(PlannerInfo *root, int strategy, bool nulls_first); extern void add_paths_to_append_rel(PlannerInfo *root, RelOptInfo *rel, List *live_childrels); +extern List *select_mergejoin_clauses(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + JoinType jointype, + bool *mergejoin_allowed); + +/* + * uniquekeys.c + * Utilities for matching and building unique keys + */ +extern void populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List* unique_index_list); +extern void populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels); +extern void populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel); +extern void populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel); +extern void populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel); +extern void simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel); +extern void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel); +extern void populate_joinrel_uniquekeys(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + RelOptInfo *rel2, + List *restrictlist, + JoinType jointype); + +extern bool relation_has_uniquekeys_for(PlannerInfo *root, + RelOptInfo *rel, + List *exprs, + bool allow_multinulls); +extern bool relation_is_onerow(RelOptInfo *rel); #endif /* PATHS_H */ -- 2.21.0 --op7mf2i72xk5yu7c Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v36-0003-Extend-UniqueKeys.patch" ^ permalink raw reply [nested|flat] 91+ messages in thread
* [PATCH v38 2/6] Introduce UniqueKey attributes on RelOptInfo struct. @ 2020-05-11 07:50 一挃 <[email protected]> 0 siblings, 0 replies; 91+ messages in thread From: 一挃 @ 2020-05-11 07:50 UTC (permalink / raw) UniqueKey is a set of exprs on RelOptInfo which represents the exprs will be unique on the given RelOptInfo. You can see README.uniquekey for more information. --- src/backend/nodes/copyfuncs.c | 13 + src/backend/nodes/list.c | 31 + src/backend/nodes/makefuncs.c | 13 + src/backend/nodes/outfuncs.c | 11 + src/backend/nodes/readfuncs.c | 10 + src/backend/optimizer/path/Makefile | 3 +- src/backend/optimizer/path/README.uniquekey | 131 +++ src/backend/optimizer/path/allpaths.c | 10 + src/backend/optimizer/path/joinpath.c | 9 +- src/backend/optimizer/path/joinrels.c | 2 + src/backend/optimizer/path/pathkeys.c | 3 +- src/backend/optimizer/path/uniquekeys.c | 1135 +++++++++++++++++++ src/backend/optimizer/plan/planner.c | 13 +- src/backend/optimizer/prep/prepunion.c | 2 + src/backend/optimizer/util/appendinfo.c | 44 + src/backend/optimizer/util/inherit.c | 18 +- src/include/nodes/makefuncs.h | 3 + src/include/nodes/nodes.h | 1 + src/include/nodes/pathnodes.h | 29 +- src/include/nodes/pg_list.h | 10 + src/include/optimizer/appendinfo.h | 3 + src/include/optimizer/optimizer.h | 2 + src/include/optimizer/paths.h | 43 + 23 files changed, 1515 insertions(+), 24 deletions(-) create mode 100644 src/backend/optimizer/path/README.uniquekey create mode 100644 src/backend/optimizer/path/uniquekeys.c diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..75c1c5e824 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2296,6 +2296,16 @@ _copyPathKey(const PathKey *from) return newnode; } +static UniqueKey * +_copyUniqueKey(const UniqueKey *from) +{ + UniqueKey *newnode = makeNode(UniqueKey); + + COPY_NODE_FIELD(exprs); + COPY_SCALAR_FIELD(multi_nullvals); + + return newnode; +} /* * _copyRestrictInfo */ @@ -5220,6 +5230,9 @@ copyObjectImpl(const void *from) case T_PathKey: retval = _copyPathKey(from); break; + case T_UniqueKey: + retval = _copyUniqueKey(from); + break; case T_RestrictInfo: retval = _copyRestrictInfo(from); break; diff --git a/src/backend/nodes/list.c b/src/backend/nodes/list.c index dbf6b30233..ca099495a1 100644 --- a/src/backend/nodes/list.c +++ b/src/backend/nodes/list.c @@ -702,6 +702,37 @@ list_member_oid(const List *list, Oid datum) return false; } +/* + * return true iff every entry in "members" list is also present + * in the "target" list. + */ +bool +list_is_subset(const List *members, const List *target) +{ + const ListCell *lc1, *lc2; + + Assert(IsPointerList(members)); + Assert(IsPointerList(target)); + check_list_invariants(members); + check_list_invariants(target); + + foreach(lc1, members) + { + bool found = false; + foreach(lc2, target) + { + if (equal(lfirst(lc1), lfirst(lc2))) + { + found = true; + break; + } + } + if (!found) + return false; + } + return true; +} + /* * Delete the n'th cell (counting from 0) in list. * diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 01c110cd2f..40415d0f5b 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -815,3 +815,16 @@ makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols) v->va_cols = va_cols; return v; } + + +/* + * makeUniqueKey + */ +UniqueKey* +makeUniqueKey(List *exprs, bool multi_nullvals) +{ + UniqueKey * ukey = makeNode(UniqueKey); + ukey->exprs = exprs; + ukey->multi_nullvals = multi_nullvals; + return ukey; +} diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..44154cde6a 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2456,6 +2456,14 @@ _outPathKey(StringInfo str, const PathKey *node) WRITE_BOOL_FIELD(pk_nulls_first); } +static void +_outUniqueKey(StringInfo str, const UniqueKey *node) +{ + WRITE_NODE_TYPE("UNIQUEKEY"); + WRITE_NODE_FIELD(exprs); + WRITE_BOOL_FIELD(multi_nullvals); +} + static void _outPathTarget(StringInfo str, const PathTarget *node) { @@ -4198,6 +4206,9 @@ outNode(StringInfo str, const void *obj) case T_PathKey: _outPathKey(str, obj); break; + case T_UniqueKey: + _outUniqueKey(str, obj); + break; case T_PathTarget: _outPathTarget(str, obj); break; diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index c5e136e9c3..b3e212bf1c 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -492,6 +492,14 @@ _readSetOperationStmt(void) READ_DONE(); } +static UniqueKey * +_readUniqueKey(void) +{ + READ_LOCALS(UniqueKey); + READ_NODE_FIELD(exprs); + READ_BOOL_FIELD(multi_nullvals); + READ_DONE(); +} /* * Stuff from primnodes.h. @@ -2717,6 +2725,8 @@ parseNodeString(void) return_value = _readCommonTableExpr(); else if (MATCH("SETOPERATIONSTMT", 16)) return_value = _readSetOperationStmt(); + else if (MATCH("UNIQUEKEY", 9)) + return_value = _readUniqueKey(); else if (MATCH("ALIAS", 5)) return_value = _readAlias(); else if (MATCH("RANGEVAR", 8)) diff --git a/src/backend/optimizer/path/Makefile b/src/backend/optimizer/path/Makefile index 1e199ff66f..7b9820c25f 100644 --- a/src/backend/optimizer/path/Makefile +++ b/src/backend/optimizer/path/Makefile @@ -21,6 +21,7 @@ OBJS = \ joinpath.o \ joinrels.o \ pathkeys.o \ - tidpath.o + tidpath.o \ + uniquekeys.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/optimizer/path/README.uniquekey b/src/backend/optimizer/path/README.uniquekey new file mode 100644 index 0000000000..5eac761995 --- /dev/null +++ b/src/backend/optimizer/path/README.uniquekey @@ -0,0 +1,131 @@ +1. What is UniqueKey? +We can think UniqueKey is a set of exprs for a RelOptInfo, which we are insure +that doesn't yields same result among all the rows. The simplest UniqueKey +format is primary key. + +However we define the UnqiueKey as below. + +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + +exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL +is a special case of UniqueKey, which means there is only one row in that +relation.it has a stronger semantic than others. like SELECT uk FROM t; uk is +normal unique key and may have different values. SELECT colx FROM t WHERE uk = +const. colx is unique AND we have only 1 value. This field can used for +innerrel_is_unique. this logic is handled specially in add_uniquekey_for_onerow +function. + +multi_nullvals: true means multi null values may exist in these exprs, so the +uniqueness is not guaranteed in this case. This field is necessary for +remove_useless_join & reduce_unique_semijoins where we don't mind these +duplicated NULL values. It is set to true for 2 cases. One is a unique key +from a unique index but the related column is nullable. The other one is for +outer join. see populate_joinrel_uniquekeys for detail. + + +The UniqueKey can be used at the following cases at least: +1. remove_useless_joins. +2. reduce_semianti_joins +3. remove distinct node if distinct clause is unique. +4. remove aggnode if group by clause is unique. +5. Index Skip Scan (WIP) +6. Aggregation Push Down without 2 phase aggregation if the join can't + duplicated the aggregated rows. (work in progress feature) + +2. How is it maintained? + +We have a set of populate_xxx_unqiuekeys functions to maintain the uniquekey on +various cases. xxx includes baserel, joinrel, partitionedrel, distinctrel, +groupedrel, unionrel. and we also need to convert the uniquekey from subquery +to outer relation, which is what convert_subquery_uniquekeys does. + +1. The first part is about baserel. We handled 3 cases. suppose we have Unique +Index on (a, b). + +1. SELECT a, b FROM t. UniqueKey (a, b) +2. SELECT a FROM t WHERE b = 1; UniqueKey (a) +3. SELECT .. FROM t WHERE a = 1 AND b = 1; UniqueKey (NIL). onerow case, every + column is Unique. + +2. The next part is joinrel, this part is most error-prone, we simplified the rules +like below: +1. If the relation's UniqueKey can't be duplicated after join, then is will be + still valid for the join rel. The function we used here is + innerrel_keeps_unique. The basic idea is innerrel.any_col = outer.uk. + +2. If the UnqiueKey can't keep valid via the rule 1, the combination of the + UniqueKey from both sides are valid for sure. We can prove this as: if the + unique exprs from rel1 is duplicated by rel2, the duplicated rows must + contains different unique exprs from rel2. + +More considerations about onerow: +1. If relation with one row and it can't be duplicated, it is still possible + contains mulit_nullvas after outer join. +2. If the either UniqueKey can be duplicated after join, the can get one row + only when both side is one row AND there is no outer join. +3. Whenever the onerow UniqueKey is not a valid any more, we need to convert one + row UniqueKey to normal unique key since we don't store exprs for one-row + relation. get_exprs_from_uniquekeys will be used here. + + +More considerations about multi_nullvals after join: +1. If the original UnqiueKey has multi_nullvals, the final UniqueKey will have + mulit_nullvals in any case. +2. If a unique key doesn't allow mulit_nullvals, after some outer join, it + allows some outer join. + + +3. When we comes to subquery, we need to convert_subquery_unqiuekeys just like +convert_subquery_pathkeys. Only the UniqueKey insides subquery is referenced as +a Var in outer relation will be reused. The relationship between the outerrel.Var +and subquery.exprs is built with outerel->subroot->processed_tlist. + + +4. As for the SRF functions, it will break the uniqueness of uniquekey, However it +is handled in adjust_paths_for_srfs, which happens after the query_planner. so +we will maintain the UniqueKey until there and reset it to NIL at that +places. This can't help on distinct/group by elimination cases but probably help +in some other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is +semantic correctly. + + +5. As for inherit table, we first main the UnqiueKey on childrel as well. But for +partitioned table we need to maintain 2 different kinds of +UnqiueKey. 1). UniqueKey on the parent relation 2). UniqueKey on child +relation for partition wise query. + +Example: +CREATE TABLE p (a int not null, b int not null) partition by list (a); +CREATE TABLE p0 partition of p for values in (1); +CREATE TABLE p1 partition of p for values in (2); + +create unique index p0_b on p0(b); +create unique index p1_b on p1(b); + +Now b is only unique on partition level, so the distinct can't be removed on +the following cases. SELECT DISTINCT b FROM p; + +Another example is SELECT DISTINCT a, b FROM p WHERE a = 1; Since only one +partition is chosen, the UniqueKey on child relation is same as the UniqueKey on +parent relation. + +Another usage of UniqueKey on partition level is it be helpful for +partition-wise join. + +As for the UniqueKey on parent table level, it comes with 2 different ways, +1). the UniqueKey is also derived in UniqueKey index, but the index must be same +in all the related children relations and the unique index must contains +Partition Key in it. Example: + +CREATE UNIQUE INDEX p_ab ON p(a, b); -- where a is the partition key. + +-- Query +SELECT a, b FROM p; the (a, b) is a UniqueKey of p. + +2). If the parent relation has only one childrel, the UniqueKey on childrel is + the UniqueKey on parent as well. diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index 37b4223adb..d52ad59f89 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -579,6 +579,12 @@ set_plain_rel_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) */ check_index_predicates(root, rel); + /* + * Now that we've marked which partial indexes are suitable, we can now + * build the relation's unique keys. + */ + populate_baserel_uniquekeys(root, rel, rel->indexlist); + /* Mark rel with estimated output rows, width, etc */ set_baserel_size_estimates(root, rel); } @@ -1297,6 +1303,8 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, /* Add paths to the append relation. */ add_paths_to_append_rel(root, rel, live_childrels); + if (IS_PARTITIONED_REL(rel)) + populate_partitionedrel_uniquekeys(root, rel, live_childrels); } @@ -2304,6 +2312,8 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, pathkeys, required_outer)); } + convert_subquery_uniquekeys(root, rel, sub_final_rel); + /* If outer rel allows parallelism, do same for partial paths. */ if (rel->consider_parallel && bms_is_empty(required_outer)) { diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c index 57ce97fd53..697ac047fb 100644 --- a/src/backend/optimizer/path/joinpath.c +++ b/src/backend/optimizer/path/joinpath.c @@ -71,13 +71,6 @@ static void consider_parallel_mergejoin(PlannerInfo *root, static void hash_inner_and_outer(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, RelOptInfo *innerrel, JoinType jointype, JoinPathExtraData *extra); -static List *select_mergejoin_clauses(PlannerInfo *root, - RelOptInfo *joinrel, - RelOptInfo *outerrel, - RelOptInfo *innerrel, - List *restrictlist, - JoinType jointype, - bool *mergejoin_allowed); static void generate_mergejoin_paths(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *innerrel, @@ -1927,7 +1920,7 @@ hash_inner_and_outer(PlannerInfo *root, * if it is mergejoinable and involves vars from the two sub-relations * currently of interest. */ -static List * +List * select_mergejoin_clauses(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c index 0dbe2ac726..7271f044ec 100644 --- a/src/backend/optimizer/path/joinrels.c +++ b/src/backend/optimizer/path/joinrels.c @@ -924,6 +924,8 @@ populate_joinrel_with_paths(PlannerInfo *root, RelOptInfo *rel1, /* Apply partitionwise join technique, if possible. */ try_partitionwise_join(root, rel1, rel2, joinrel, sjinfo, restrictlist); + + populate_joinrel_uniquekeys(root, joinrel, rel1, rel2, restrictlist, sjinfo->jointype); } diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index bd9a176d7d..139278829b 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -33,7 +33,6 @@ static bool pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys); static bool matches_boolean_partition_clause(RestrictInfo *rinfo, RelOptInfo *partrel, int partkeycol); -static Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); static bool right_merge_direction(PlannerInfo *root, PathKey *pathkey); @@ -1035,7 +1034,7 @@ convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel, * We need this to ensure that we don't return pathkeys describing values * that are unavailable above the level of the subquery scan. */ -static Var * +Var * find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle) { ListCell *lc; diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c new file mode 100644 index 0000000000..77ed2b2eff --- /dev/null +++ b/src/backend/optimizer/path/uniquekeys.c @@ -0,0 +1,1135 @@ +/*------------------------------------------------------------------------- + * + * uniquekeys.c + * Utilities for matching and building unique keys + * + * Portions Copyright (c) 2020, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/optimizer/path/uniquekeys.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "nodes/makefuncs.h" +#include "nodes/nodeFuncs.h" +#include "optimizer/pathnode.h" +#include "optimizer/paths.h" +#include "optimizer/appendinfo.h" +#include "optimizer/optimizer.h" +#include "optimizer/tlist.h" +#include "rewrite/rewriteManip.h" + + +/* + * This struct is used to help populate_joinrel_uniquekeys. + * + * added_to_joinrel is true if a uniquekey (from outerrel or innerrel) + * has been added to joinrel. + * useful is true if the exprs of the uniquekey still appears in joinrel. + */ +typedef struct UniqueKeyContextData +{ + UniqueKey *uniquekey; + bool added_to_joinrel; + bool useful; +} *UniqueKeyContext; + +static List *initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel); +static bool innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + bool reverse); + +static List *get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals); +static List *get_exprs_from_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + UniqueKey *ukey); +static void add_uniquekey_for_onerow(RelOptInfo *rel); +static bool add_combined_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype); + +/* Used for unique indexes checking for partitioned table */ +static bool index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind); +static IndexOptInfo *simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from); +static bool simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2); +static void adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_index); + +/* Helper function for grouped relation and distinct relation. */ +static void add_uniquekey_from_sortgroups(PlannerInfo *root, + RelOptInfo *rel, + List *sortgroups); + +/* + * populate_baserel_uniquekeys + * Populate 'baserel' uniquekeys list by looking at the rel's unique index + * and baserestrictinfo + */ +void +populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List *indexlist) +{ + ListCell *lc; + List *matched_uniq_indexes = NIL; + + /* Attrs appears in rel->reltarget->exprs. */ + Bitmapset *used_attrs = NULL; + + List *const_exprs = NIL; + List *expr_opfamilies = NIL; + + Assert(baserel->rtekind == RTE_RELATION); + + foreach(lc, indexlist) + { + IndexOptInfo *ind = (IndexOptInfo *) lfirst(lc); + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + matched_uniq_indexes = lappend(matched_uniq_indexes, ind); + } + + if (matched_uniq_indexes == NIL) + return; + + /* Check which attrs is used in baserel->reltarget */ + pull_varattnos((Node *)baserel->reltarget->exprs, baserel->relid, &used_attrs); + + /* Check which attrno is used at a mergeable const filter */ + foreach(lc, baserel->baserestrictinfo) + { + RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); + + if (rinfo->mergeopfamilies == NIL) + continue; + + if (bms_is_empty(rinfo->left_relids)) + { + const_exprs = lappend(const_exprs, get_rightop(rinfo->clause)); + } + else if (bms_is_empty(rinfo->right_relids)) + { + const_exprs = lappend(const_exprs, get_leftop(rinfo->clause)); + } + else + continue; + + expr_opfamilies = lappend(expr_opfamilies, rinfo->mergeopfamilies); + } + + foreach(lc, matched_uniq_indexes) + { + bool multi_nullvals, useful; + List *exprs = get_exprs_from_uniqueindex(lfirst_node(IndexOptInfo, lc), + const_exprs, + expr_opfamilies, + used_attrs, + &useful, + &multi_nullvals); + if (useful) + { + if (exprs == NIL) + { + /* All the columns in Unique Index matched with a restrictinfo */ + add_uniquekey_for_onerow(baserel); + return; + } + baserel->uniquekeys = lappend(baserel->uniquekeys, + makeUniqueKey(exprs, multi_nullvals)); + } + } +} + + +/* + * populate_partitionedrel_uniquekeys + * The UniqueKey on partitionrel comes from 2 cases: + * 1). Only one partition is involved in this query, the unique key can be + * copied to parent rel from childrel. + * 2). There are some unique index which includes partition key and exists + * in all the related partitions. + * We never mind rule 2 if we hit rule 1. + */ + +void +populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels) +{ + ListCell *lc; + List *global_uniq_indexlist = NIL; + RelOptInfo *childrel; + bool is_first = true; + + Assert(IS_PARTITIONED_REL(rel)); + + if (childrels == NIL) + return; + + /* + * If there is only one partition used in this query, the UniqueKey in childrel is + * still valid in parent level, but we need convert the format from child expr to + * parent expr. + */ + if (list_length(childrels) == 1) + { + /* Check for Rule 1 */ + RelOptInfo *childrel = linitial_node(RelOptInfo, childrels); + ListCell *lc; + Assert(childrel->reloptkind == RELOPT_OTHER_MEMBER_REL); + if (relation_is_onerow(childrel)) + { + add_uniquekey_for_onerow(rel); + return; + } + + foreach(lc, childrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + AppendRelInfo *appinfo = find_appinfo_by_child(root, childrel->relid); + List *parent_exprs = NIL; + bool can_reuse = true; + ListCell *lc2; + foreach(lc2, ukey->exprs) + { + Var *var = (Var *)lfirst(lc2); + /* + * If the expr comes from a expression, it is hard to build the expression + * in parent so ignore that case for now. + */ + if(!IsA(var, Var)) + { + can_reuse = false; + break; + } + /* Convert it to parent var */ + parent_exprs = lappend(parent_exprs, find_parent_var(appinfo, var)); + } + if (can_reuse) + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(parent_exprs, + ukey->multi_nullvals)); + } + } + else + { + /* Check for rule 2 */ + childrel = linitial_node(RelOptInfo, childrels); + foreach(lc, childrel->indexlist) + { + IndexOptInfo *ind = lfirst(lc); + IndexOptInfo *modified_index; + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + + /* + * During simple_copy_indexinfo_to_parent, we need to convert var from + * child var to parent var, index on expression is too complex to handle. + * so ignore it for now. + */ + if (ind->indexprs != NIL) + continue; + + modified_index = simple_copy_indexinfo_to_parent(root, rel, ind); + /* + * If the unique index doesn't contain partkey, then it is unique + * on this partition only, so it is useless for us. + */ + if (!index_constains_partkey(rel, modified_index)) + continue; + + global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index); + } + + if (global_uniq_indexlist != NIL) + { + foreach(lc, childrels) + { + RelOptInfo *child = lfirst(lc); + if (is_first) + { + is_first = false; + continue; + } + adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist); + } + /* Now we have a list of unique index which are exactly same on all childrels, + * Set the UniqueKey just like it is non-partition table + */ + populate_baserel_uniquekeys(root, rel, global_uniq_indexlist); + } + } +} + + +/* + * populate_distinctrel_uniquekeys + */ +void +populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel) +{ + /* The unique key before the distinct is still valid. */ + distinctrel->uniquekeys = list_copy(inputrel->uniquekeys); + add_uniquekey_from_sortgroups(root, distinctrel, root->parse->distinctClause); +} + +/* + * populate_grouprel_uniquekeys + */ +void +populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel) + +{ + Query *parse = root->parse; + bool input_ukey_added = false; + ListCell *lc; + + if (relation_is_onerow(inputrel)) + { + add_uniquekey_for_onerow(grouprel); + return; + } + if (parse->groupingSets) + return; + + /* A Normal group by without grouping set. */ + if (parse->groupClause) + { + /* + * Current even the groupby clause is Unique already, but if query has aggref + * We have to create grouprel still. To keep the UnqiueKey short, we will check + * the UniqueKey of input_rel still valid, if so we reuse it. + */ + foreach(lc, inputrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (list_is_subset(ukey->exprs, grouprel->reltarget->exprs)) + { + grouprel->uniquekeys = lappend(grouprel->uniquekeys, + ukey); + input_ukey_added = true; + } + } + if (!input_ukey_added) + /* + * group by clause must be a super-set of grouprel->reltarget->exprs except the + * aggregation expr, so if such exprs is unique already, no bother to generate + * new uniquekey for group by exprs. + */ + add_uniquekey_from_sortgroups(root, + grouprel, + root->parse->groupClause); + } + else + /* It has aggregation but without a group by, so only one row returned */ + add_uniquekey_for_onerow(grouprel); +} + +/* + * simple_copy_uniquekeys + * Using a function for the one-line code makes us easy to check where we simply + * copied the uniquekey. + */ +void +simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel) +{ + newrel->uniquekeys = oldrel->uniquekeys; +} + +/* + * populate_unionrel_uniquekeys + */ +void +populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel) +{ + ListCell *lc; + List *exprs = NIL; + + Assert(unionrel->uniquekeys == NIL); + + foreach(lc, unionrel->reltarget->exprs) + { + exprs = lappend(exprs, lfirst(lc)); + } + + if (exprs == NIL) + /* SQL: select union select; is valid, we need to handle it here. */ + add_uniquekey_for_onerow(unionrel); + else + unionrel->uniquekeys = lappend(unionrel->uniquekeys, + makeUniqueKey(exprs,false)); + +} + +/* + * populate_joinrel_uniquekeys + * + * populate uniquekeys for joinrel. We will check each relation to see if its + * UniqueKey is still valid via innerrel_keeps_unique, if so, we add it to + * joinrel. The multi_nullvals field will be changed to true for some outer + * join cases and one-row UniqueKey needs to be converted to normal UniqueKey + * for the same case as well. + * For the uniquekey in either baserel which can't be unique after join, we still + * check to see if combination of UniqueKeys from both side is still useful for us. + * if yes, we add it to joinrel as well. + */ +void +populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *outerrel, RelOptInfo *innerrel, + List *restrictlist, JoinType jointype) +{ + ListCell *lc, *lc2; + List *clause_list = NIL; + List *outerrel_ukey_ctx; + List *innerrel_ukey_ctx; + bool inner_onerow, outer_onerow; + bool mergejoin_allowed; + + /* Care about the outerrel relation only for SEMI/ANTI join */ + if (jointype == JOIN_SEMI || jointype == JOIN_ANTI) + { + foreach(lc, outerrel->uniquekeys) + { + UniqueKey *uniquekey = lfirst_node(UniqueKey, lc); + if (list_is_subset(uniquekey->exprs, joinrel->reltarget->exprs)) + joinrel->uniquekeys = lappend(joinrel->uniquekeys, uniquekey); + } + return; + } + + Assert(jointype == JOIN_LEFT || jointype == JOIN_FULL || jointype == JOIN_INNER); + + /* Fast path */ + if (innerrel->uniquekeys == NIL || outerrel->uniquekeys == NIL) + return; + + inner_onerow = relation_is_onerow(innerrel); + outer_onerow = relation_is_onerow(outerrel); + + outerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(outerrel); + innerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(innerrel); + + clause_list = select_mergejoin_clauses(root, joinrel, outerrel, innerrel, + restrictlist, jointype, + &mergejoin_allowed); + + if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true /* reverse */)) + { + bool outer_impact = jointype == JOIN_FULL; + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + /* Outer relation has one row, and the unique key is not duplicated after join, + * the joinrel will still has one row unless the jointype == JOIN_FULL. + */ + if (outer_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (outer_onerow) + { + /* + * The onerow outerrel becomes multi rows and multi_nullvals + * will be changed to true. We also need to set the exprs correctly since it + * can't be NIL any more. + */ + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, outerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Change multi_nullvals to true due to the full join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, true)); + else + /* Just reuse it */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + } + ctx->added_to_joinrel = true; + } + } + + if (innerrel_keeps_unique(root, outerrel, innerrel, clause_list, false)) + { + bool outer_impact = jointype == JOIN_FULL || jointype == JOIN_LEFT;; + + foreach(lc, innerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + if (inner_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (inner_onerow) + { + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, innerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Need to change multi_nullvals to true due to the outer join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, + true)); + else + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + + } + ctx->added_to_joinrel = true; + } + } + + /* + * The combination of the UniqueKey from both sides is unique as well regardless + * of join type, but no bother to add it if its subset has been added to joinrel + * already or it is not useful for the joinrel. + */ + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx1 = (UniqueKeyContext) lfirst(lc); + if (ctx1->added_to_joinrel || !ctx1->useful) + continue; + foreach(lc2, innerrel_ukey_ctx) + { + UniqueKeyContext ctx2 = (UniqueKeyContext) lfirst(lc2); + if (ctx2->added_to_joinrel || !ctx2->useful) + continue; + if (add_combined_uniquekey(root, joinrel, outerrel, innerrel, + ctx1->uniquekey, ctx2->uniquekey, + jointype)) + /* If we set a onerow UniqueKey to joinrel, we don't need other. */ + return; + } + } +} + + +/* + * convert_subquery_uniquekeys + * + * Covert the UniqueKey in subquery to outer relation. + */ +void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel) +{ + ListCell *lc; + + if (sub_final_rel->uniquekeys == NIL) + return; + + if (relation_is_onerow(sub_final_rel)) + { + add_uniquekey_for_onerow(currel); + return; + } + + Assert(currel->subroot != NULL); + + foreach(lc, sub_final_rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + ListCell *lc; + List *exprs = NIL; + bool ukey_useful = true; + + /* One row case is handled above */ + Assert(ukey->exprs != NIL); + foreach(lc, ukey->exprs) + { + Var *var; + TargetEntry *tle = tlist_member(lfirst(lc), + currel->subroot->processed_tlist); + if (tle == NULL) + { + ukey_useful = false; + break; + } + var = find_var_for_subquery_tle(currel, tle); + if (var == NULL) + { + ukey_useful = false; + break; + } + exprs = lappend(exprs, var); + } + + if (ukey_useful) + currel->uniquekeys = lappend(currel->uniquekeys, + makeUniqueKey(exprs, + ukey->multi_nullvals)); + + } +} + +/* + * innerrel_keeps_unique + * + * Check if Unique key of the innerrel is valid after join. innerrel's UniqueKey + * will be still valid if innerrel's any-column mergeop outrerel's uniquekey + * exists in clause_list. + * + * Note: the clause_list must be a list of mergeable restrictinfo already. + */ +static bool +innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *clause_list, + bool reverse) +{ + ListCell *lc, *lc2, *lc3; + + if (outerrel->uniquekeys == NIL || innerrel->uniquekeys == NIL) + return false; + + /* Check if there is outerrel's uniquekey in mergeable clause. */ + foreach(lc, outerrel->uniquekeys) + { + List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs; + bool clauselist_matchs_all_exprs = true; + foreach(lc2, outer_uq_exprs) + { + Node *outer_uq_expr = lfirst(lc2); + bool find_uq_expr_in_clauselist = false; + foreach(lc3, clause_list) + { + RestrictInfo *rinfo = lfirst_node(RestrictInfo, lc3); + Node *outer_expr; + if (reverse) + outer_expr = rinfo->outer_is_left ? get_rightop(rinfo->clause) : get_leftop(rinfo->clause); + else + outer_expr = rinfo->outer_is_left ? get_leftop(rinfo->clause) : get_rightop(rinfo->clause); + if (equal(outer_expr, outer_uq_expr)) + { + find_uq_expr_in_clauselist = true; + break; + } + } + if (!find_uq_expr_in_clauselist) + { + /* No need to check the next exprs in the current uniquekey */ + clauselist_matchs_all_exprs = false; + break; + } + } + + if (clauselist_matchs_all_exprs) + return true; + } + return false; +} + + +/* + * relation_is_onerow + * Check if it is a one-row relation by checking UniqueKey. + */ +bool +relation_is_onerow(RelOptInfo *rel) +{ + UniqueKey *ukey; + if (rel->uniquekeys == NIL) + return false; + ukey = linitial_node(UniqueKey, rel->uniquekeys); + return ukey->exprs == NIL && list_length(rel->uniquekeys) == 1; +} + +/* + * relation_has_uniquekeys_for + * Returns true if we have proofs that 'rel' cannot return multiple rows with + * the same values in each of 'exprs'. Otherwise returns false. + */ +bool +relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel, + List *exprs, bool allow_multinulls) +{ + ListCell *lc; + + /* + * For UniqueKey->onerow case, the uniquekey->exprs is empty as well + * so we can't rely on list_is_subset to handle this special cases + */ + if (exprs == NIL) + return false; + + foreach(lc, rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (ukey->multi_nullvals && !allow_multinulls) + continue; + if (list_is_subset(ukey->exprs, exprs)) + return true; + } + return false; +} + + +/* + * get_exprs_from_uniqueindex + * + * Return a list of exprs which is unique. set useful to false if this + * unique index is not useful for us. + */ +static List * +get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals) +{ + List *exprs = NIL; + ListCell *indexpr_item; + int c = 0; + + *useful = true; + *multi_nullvals = false; + + indexpr_item = list_head(unique_index->indexprs); + for(c = 0; c < unique_index->ncolumns; c++) + { + int attr = unique_index->indexkeys[c]; + Expr *expr; + bool matched_const = false; + ListCell *lc1, *lc2; + + if(attr > 0) + { + expr = list_nth_node(TargetEntry, unique_index->indextlist, c)->expr; + } + else if (attr == 0) + { + /* Expression index */ + expr = lfirst(indexpr_item); + indexpr_item = lnext(unique_index->indexprs, indexpr_item); + } + else /* attr < 0 */ + { + /* Index on system column is not supported */ + Assert(false); + } + + /* + * Check index_col = Const case with regarding to opfamily checking + * If we can remove the index_col from the final UniqueKey->exprs. + */ + forboth(lc1, const_exprs, lc2, const_expr_opfamilies) + { + if (list_member_oid((List *)lfirst(lc2), unique_index->opfamily[c]) + && match_index_to_operand((Node *) lfirst(lc1), c, unique_index)) + { + matched_const = true; + break; + } + } + + if (matched_const) + continue; + + /* Check if the indexed expr is used in rel */ + if (attr > 0) + { + /* + * Normal Indexed column, if the col is not used, then the index is useless + * for uniquekey. + */ + attr -= FirstLowInvalidHeapAttributeNumber; + + if (!bms_is_member(attr, used_varattrs)) + { + *useful = false; + break; + } + } + else if (!list_member(unique_index->rel->reltarget->exprs, expr)) + { + /* Expression index but the expression is not used in rel */ + *useful = false; + break; + } + + /* check not null property. */ + if (attr == 0) + { + /* We never know if a expression yields null or not */ + *multi_nullvals = true; + } + else if (!bms_is_member(attr, unique_index->rel->notnullattrs) + && !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber, + unique_index->rel->notnullattrs)) + { + *multi_nullvals = true; + } + + exprs = lappend(exprs, expr); + } + return exprs; +} + + +/* + * add_uniquekey_for_onerow + * If we are sure that the relation only returns one row, then all the columns + * are unique. However we don't need to create UniqueKey for every column, we + * just set exprs = NIL and overwrites all the other UniqueKey on this RelOptInfo + * since this one has strongest semantics. + */ +void +add_uniquekey_for_onerow(RelOptInfo *rel) +{ + /* + * We overwrite the previous UniqueKey on purpose since this one has the + * strongest semantic. + */ + rel->uniquekeys = list_make1(makeUniqueKey(NIL, false)); +} + + +/* + * initililze_uniquecontext_for_joinrel + * Return a List of UniqueKeyContext for an inputrel + */ +static List * +initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel) +{ + List *res = NIL; + ListCell *lc; + foreach(lc, inputrel->uniquekeys) + { + UniqueKeyContext context; + context = palloc(sizeof(struct UniqueKeyContextData)); + context->uniquekey = lfirst_node(UniqueKey, lc); + context->added_to_joinrel = false; + context->useful = true; + res = lappend(res, context); + } + return res; +} + + +/* + * get_exprs_from_uniquekey + * Unify the way of get List of exprs from a one-row UniqueKey or + * normal UniqueKey. for the onerow case, every expr in rel1 is a valid + * UniqueKey. Return a List of exprs. + * + * rel1: The relation which you want to get the exprs. + * ukey: The UniqueKey you want to get the exprs. + */ +static List * +get_exprs_from_uniquekey(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *rel1, UniqueKey *ukey) +{ + ListCell *lc; + bool onerow = rel1 != NULL && relation_is_onerow(rel1); + + List *res = NIL; + Assert(onerow || ukey); + if (onerow) + { + /* Only cares about the exprs still exist in joinrel */ + foreach(lc, joinrel->reltarget->exprs) + { + Bitmapset *relids = pull_varnos(root, lfirst(lc)); + if (bms_is_subset(relids, rel1->relids)) + { + res = lappend(res, list_make1(lfirst(lc))); + } + } + } + else + { + res = list_make1(ukey->exprs); + } + return res; +} + +/* + * Partitioned table Unique Keys. + * The partition table unique key is maintained as: + * 1. The index must be unique as usual. + * 2. The index must contains partition key. + * 3. The index must exist on all the child rel. see simple_indexinfo_equal for + * how we compare it. + */ + +/* + * index_constains_partkey + * return true if the index contains the partiton key. + */ +static bool +index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind) +{ + ListCell *lc; + int i; + Assert(IS_PARTITIONED_REL(partrel)); + Assert(partrel->part_scheme->partnatts > 0); + + for(i = 0; i < partrel->part_scheme->partnatts; i++) + { + Node *part_expr = linitial(partrel->partexprs[i]); + bool found_in_index = false; + foreach(lc, ind->indextlist) + { + Expr *index_expr = lfirst_node(TargetEntry, lc)->expr; + if (equal(index_expr, part_expr)) + { + found_in_index = true; + break; + } + } + if (!found_in_index) + return false; + } + return true; +} + +/* + * simple_indexinfo_equal + * + * Used to check if the 2 index is same as each other. The index here + * is COPIED from childrel and did some tiny changes(see + * simple_copy_indexinfo_to_parent) + */ +static bool +simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2) +{ + Size oid_cmp_len = sizeof(Oid) * ind1->ncolumns; + + return ind1->ncolumns == ind2->ncolumns && + ind1->unique == ind2->unique && + memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0 && + memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0 && + memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0 && + memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0 && + equal(get_tlist_exprs(ind1->indextlist, true), + get_tlist_exprs(ind2->indextlist, true)); +} + + +/* + * The below macros are used for simple_copy_indexinfo_to_parent which is so + * customized that I don't want to put it to copyfuncs.c. So copy it here. + */ +#define COPY_POINTER_FIELD(fldname, sz) \ + do { \ + Size _size = (sz); \ + newnode->fldname = palloc(_size); \ + memcpy(newnode->fldname, from->fldname, _size); \ + } while (0) + +#define COPY_NODE_FIELD(fldname) \ + (newnode->fldname = copyObjectImpl(from->fldname)) + +#define COPY_SCALAR_FIELD(fldname) \ + (newnode->fldname = from->fldname) + + +/* + * simple_copy_indexinfo_to_parent (from partition) + * Copy the IndexInfo from child relation to parent relation with some modification, + * which is used to test: + * 1. If the same index exists in all the childrels. + * 2. If the parentrel->reltarget/basicrestrict info matches this index. + */ +static IndexOptInfo * +simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from) +{ + IndexOptInfo *newnode = makeNode(IndexOptInfo); + AppendRelInfo *appinfo = find_appinfo_by_child(root, from->rel->relid); + ListCell *lc; + int idx = 0; + + COPY_SCALAR_FIELD(ncolumns); + COPY_SCALAR_FIELD(nkeycolumns); + COPY_SCALAR_FIELD(unique); + COPY_SCALAR_FIELD(immediate); + /* We just need to know if it is NIL or not */ + COPY_SCALAR_FIELD(indpred); + COPY_SCALAR_FIELD(predOK); + COPY_POINTER_FIELD(indexkeys, from->ncolumns * sizeof(int)); + COPY_POINTER_FIELD(indexcollations, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opfamily, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opcintype, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(sortopfamily, from->ncolumns * sizeof(Oid)); + COPY_NODE_FIELD(indextlist); + + /* Convert index exprs on child expr to expr on parent */ + foreach(lc, newnode->indextlist) + { + TargetEntry *tle = lfirst_node(TargetEntry, lc); + /* Index on expression is ignored */ + Assert(IsA(tle->expr, Var)); + tle->expr = (Expr *) find_parent_var(appinfo, (Var *) tle->expr); + newnode->indexkeys[idx] = castNode(Var, tle->expr)->varattno; + idx++; + } + newnode->rel = parentrel; + return newnode; +} + +/* + * adjust_partition_unique_indexlist + * + * global_unique_indexes: At the beginning, it contains the copy & modified + * unique index from the first partition. And then check if each index in it still + * exists in the following partitions. If no, remove it. at last, it has an + * index list which exists in all the partitions. + */ +static void +adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_indexes) +{ + ListCell *lc, *lc2; + foreach(lc, *global_unique_indexes) + { + IndexOptInfo *g_ind = lfirst_node(IndexOptInfo, lc); + bool found_in_child = false; + + foreach(lc2, childrel->indexlist) + { + IndexOptInfo *p_ind = lfirst_node(IndexOptInfo, lc2); + IndexOptInfo *p_ind_copy; + if (!p_ind->unique || !p_ind->immediate || + (p_ind->indpred != NIL && !p_ind->predOK)) + continue; + p_ind_copy = simple_copy_indexinfo_to_parent(root, parentrel, p_ind); + if (simple_indexinfo_equal(p_ind_copy, g_ind)) + { + found_in_child = true; + break; + } + } + if (!found_in_child) + /* The index doesn't exist in childrel, remove it from global_unique_indexes */ + *global_unique_indexes = foreach_delete_current(*global_unique_indexes, lc); + } +} + +/* Helper function for groupres/distinctrel */ +static void +add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgroups) +{ + Query *parse = root->parse; + List *exprs; + + /* + * XXX: If there are some vars which is not in current levelsup, the semantic is + * imprecise, should we avoid it or not? levelsup = 1 is just a demo, maybe we need to + * check every level other than 0, if so, looks we have to write another + * pull_var_walker. + */ + List *upper_vars = pull_vars_of_level((Node*)sortgroups, 1); + + if (upper_vars != NIL) + return; + + exprs = get_sortgrouplist_exprs(sortgroups, parse->targetList); + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(exprs, + false /* sortgroupclause can't be multi_nullvals */)); +} + + +/* + * add_combined_uniquekey + * The combination of both UniqueKeys is a valid UniqueKey for joinrel no matter + * the jointype. + */ +bool +add_combined_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype) +{ + + ListCell *lc1, *lc2; + + /* Either side has multi_nullvals or we have outer join, + * the combined UniqueKey has multi_nullvals */ + bool multi_nullvals = outer_ukey->multi_nullvals || + inner_ukey->multi_nullvals || IS_OUTER_JOIN(jointype); + + /* The only case we can get onerow joinrel after join */ + if (relation_is_onerow(outer_rel) + && relation_is_onerow(inner_rel) + && jointype == JOIN_INNER) + { + add_uniquekey_for_onerow(joinrel); + return true; + } + + foreach(lc1, get_exprs_from_uniquekey(root, joinrel, outer_rel, outer_ukey)) + { + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, inner_rel, inner_ukey)) + { + List *exprs = list_concat_copy(lfirst_node(List, lc1), lfirst_node(List, lc2)); + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(exprs, + multi_nullvals)); + } + } + return false; +} diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index 424d25cbd5..8d8e493f5c 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -2387,6 +2387,8 @@ grouping_planner(PlannerInfo *root, bool inheritance_update, add_path(final_rel, path); } + simple_copy_uniquekeys(current_rel, final_rel); + /* * Generate partial paths for final_rel, too, if outer query levels might * be able to make use of them. @@ -3899,6 +3901,8 @@ create_grouping_paths(PlannerInfo *root, } set_cheapest(grouped_rel); + + populate_grouprel_uniquekeys(root, grouped_rel, input_rel); return grouped_rel; } @@ -4621,7 +4625,7 @@ create_window_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(window_rel); - + simple_copy_uniquekeys(input_rel, window_rel); return window_rel; } @@ -4935,7 +4939,7 @@ create_distinct_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(distinct_rel); - + populate_distinctrel_uniquekeys(root, input_rel, distinct_rel); return distinct_rel; } @@ -5196,6 +5200,8 @@ create_ordered_paths(PlannerInfo *root, */ Assert(ordered_rel->pathlist != NIL); + simple_copy_uniquekeys(input_rel, ordered_rel); + return ordered_rel; } @@ -6073,6 +6079,9 @@ adjust_paths_for_srfs(PlannerInfo *root, RelOptInfo *rel, if (list_length(targets) == 1) return; + /* UniqueKey is not valid after handling the SRF. */ + rel->uniquekeys = NIL; + /* * Stack SRF-evaluation nodes atop each path for the rel. * diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c index becdcbb872..b7626545bf 100644 --- a/src/backend/optimizer/prep/prepunion.c +++ b/src/backend/optimizer/prep/prepunion.c @@ -689,6 +689,8 @@ generate_union_paths(SetOperationStmt *op, PlannerInfo *root, /* Undo effects of possibly forcing tuple_fraction to 0 */ root->tuple_fraction = save_fraction; + /* Add the UniqueKeys */ + populate_unionrel_uniquekeys(root, result_rel); return result_rel; } diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c index 86922a273c..6817c9c787 100644 --- a/src/backend/optimizer/util/appendinfo.c +++ b/src/backend/optimizer/util/appendinfo.c @@ -746,3 +746,47 @@ find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos) } return appinfos; } + +/* + * find_appinfo_by_child + * + */ +AppendRelInfo * +find_appinfo_by_child(PlannerInfo *root, Index child_index) +{ + ListCell *lc; + foreach(lc, root->append_rel_list) + { + AppendRelInfo *appinfo = lfirst_node(AppendRelInfo, lc); + if (appinfo->child_relid == child_index) + return appinfo; + } + elog(ERROR, "parent relation cant be found"); + return NULL; +} + +/* + * find_parent_var + * + */ +Var * +find_parent_var(AppendRelInfo *appinfo, Var *child_var) +{ + ListCell *lc; + Var *res = NULL; + Index attno = 1; + foreach(lc, appinfo->translated_vars) + { + Node *child_node = lfirst(lc); + if (equal(child_node, child_var)) + { + res = copyObject(child_var); + res->varattno = attno; + res->varno = appinfo->parent_relid; + } + attno++; + } + if (res == NULL) + elog(ERROR, "parent var can't be found."); + return res; +} diff --git a/src/backend/optimizer/util/inherit.c b/src/backend/optimizer/util/inherit.c index be1c9ddd96..3eec1f4d74 100644 --- a/src/backend/optimizer/util/inherit.c +++ b/src/backend/optimizer/util/inherit.c @@ -736,6 +736,7 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, { Node *onecq = (Node *) lfirst(lc2); bool pseudoconstant; + RestrictInfo *child_rinfo; /* check for pseudoconstant (no Vars or volatile functions) */ pseudoconstant = @@ -747,14 +748,15 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, root->hasPseudoConstantQuals = true; } /* reconstitute RestrictInfo with appropriate properties */ - childquals = lappend(childquals, - make_restrictinfo(root, - (Expr *) onecq, - rinfo->is_pushed_down, - rinfo->outerjoin_delayed, - pseudoconstant, - rinfo->security_level, - NULL, NULL, NULL)); + child_rinfo = make_restrictinfo(root, + (Expr *) onecq, + rinfo->is_pushed_down, + rinfo->outerjoin_delayed, + pseudoconstant, + rinfo->security_level, + NULL, NULL, NULL); + child_rinfo->mergeopfamilies = rinfo->mergeopfamilies; + childquals = lappend(childquals, child_rinfo); /* track minimum security level among child quals */ cq_min_security = Min(cq_min_security, rinfo->security_level); } diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 48a7ebfe45..4fe1824eb0 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -16,6 +16,7 @@ #include "nodes/execnodes.h" #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" extern A_Expr *makeA_Expr(A_Expr_Kind kind, List *name, @@ -106,4 +107,6 @@ extern GroupingSet *makeGroupingSet(GroupingSetKind kind, List *content, int loc extern VacuumRelation *makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols); +extern UniqueKey* makeUniqueKey(List *exprs, bool multi_nullvals); + #endif /* MAKEFUNC_H */ diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index e22df890ef..3320273ac1 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -265,6 +265,7 @@ typedef enum NodeTag T_EquivalenceMember, T_PathKey, T_PathTarget, + T_UniqueKey, T_RestrictInfo, T_IndexClause, T_PlaceHolderVar, diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index 0d61f04d27..80561b1003 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -739,6 +739,7 @@ typedef struct RelOptInfo QualCost baserestrictcost; /* cost of evaluating the above */ Index baserestrict_min_security; /* min security_level found in * baserestrictinfo */ + List *uniquekeys; /* List of UniqueKey */ List *joininfo; /* RestrictInfo structures for join clauses * involving this rel */ bool has_eclass_joins; /* T means joininfo is incomplete */ @@ -1059,6 +1060,28 @@ typedef struct PathKey } PathKey; +/* + * UniqueKey + * + * Represents the unique properties held by a RelOptInfo. + * + * exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL + * is a special case of UniqueKey, which means there is only 1 row in that + * relation. + * multi_nullvals: true means multi null values may exist in these exprs, so the + * uniqueness is not guaranteed in this case. This field is necessary for + * remove_useless_join & reduce_unique_semijoins where we don't mind these + * duplicated NULL values. It is set to true for 2 cases. One is a unique key + * from a unique index but the related column is nullable. The other one is for + * outer join. see populate_joinrel_uniquekeys for detail. + */ +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + /* * PathTarget * @@ -2496,7 +2519,7 @@ typedef enum * * flags indicating what kinds of grouping are possible. * partial_costs_set is true if the agg_partial_costs and agg_final_costs - * have been initialized. + * have been initialized. * agg_partial_costs gives partial aggregation costs. * agg_final_costs gives finalization costs. * target_parallel_safe is true if target is parallel safe. @@ -2526,8 +2549,8 @@ typedef struct * limit_tuples is an estimated bound on the number of output tuples, * or -1 if no LIMIT or couldn't estimate. * count_est and offset_est are the estimated values of the LIMIT and OFFSET - * expressions computed by preprocess_limit() (see comments for - * preprocess_limit() for more information). + * expressions computed by preprocess_limit() (see comments for + * preprocess_limit() for more information). */ typedef struct { diff --git a/src/include/nodes/pg_list.h b/src/include/nodes/pg_list.h index 404e03f132..85c65e910c 100644 --- a/src/include/nodes/pg_list.h +++ b/src/include/nodes/pg_list.h @@ -566,6 +566,16 @@ extern pg_nodiscard List *list_delete_first(List *list); extern pg_nodiscard List *list_delete_last(List *list); extern pg_nodiscard List *list_delete_nth_cell(List *list, int n); extern pg_nodiscard List *list_delete_cell(List *list, ListCell *cell); +extern bool list_is_subset(const List *members, const List *target); + +extern List *list_delete(List *list, void *datum); +extern List *list_delete_ptr(List *list, void *datum); +extern List *list_delete_int(List *list, int datum); +extern List *list_delete_oid(List *list, Oid datum); +extern List *list_delete_first(List *list); +extern List *list_delete_last(List *list); +extern List *list_delete_nth_cell(List *list, int n); +extern List *list_delete_cell(List *list, ListCell *cell); extern List *list_union(const List *list1, const List *list2); extern List *list_union_ptr(const List *list1, const List *list2); diff --git a/src/include/optimizer/appendinfo.h b/src/include/optimizer/appendinfo.h index 4cbf8c26cc..43b8d05f69 100644 --- a/src/include/optimizer/appendinfo.h +++ b/src/include/optimizer/appendinfo.h @@ -32,4 +32,7 @@ extern Relids adjust_child_relids_multilevel(PlannerInfo *root, Relids relids, extern AppendRelInfo **find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos); +extern AppendRelInfo *find_appinfo_by_child(PlannerInfo *root, Index child_index); +extern Var *find_parent_var(AppendRelInfo *appinfo, Var *child_var); + #endif /* APPENDINFO_H */ diff --git a/src/include/optimizer/optimizer.h b/src/include/optimizer/optimizer.h index d587952b7d..843aafa51e 100644 --- a/src/include/optimizer/optimizer.h +++ b/src/include/optimizer/optimizer.h @@ -23,6 +23,7 @@ #define OPTIMIZER_H #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" /* Test if an expression node represents a SRF call. Beware multiple eval! */ #define IS_SRF_CALL(node) \ @@ -168,6 +169,7 @@ extern TargetEntry *get_sortgroupref_tle(Index sortref, List *targetList); extern TargetEntry *get_sortgroupclause_tle(SortGroupClause *sgClause, List *targetList); +extern Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); extern Node *get_sortgroupclause_expr(SortGroupClause *sgClause, List *targetList); extern List *get_sortgrouplist_exprs(List *sgClauses, diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h index 035d3e1206..1adf99c9ee 100644 --- a/src/include/optimizer/paths.h +++ b/src/include/optimizer/paths.h @@ -247,5 +247,48 @@ extern PathKey *make_canonical_pathkey(PlannerInfo *root, int strategy, bool nulls_first); extern void add_paths_to_append_rel(PlannerInfo *root, RelOptInfo *rel, List *live_childrels); +extern List *select_mergejoin_clauses(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + JoinType jointype, + bool *mergejoin_allowed); + +/* + * uniquekeys.c + * Utilities for matching and building unique keys + */ +extern void populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List* unique_index_list); +extern void populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels); +extern void populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel); +extern void populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel); +extern void populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel); +extern void simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel); +extern void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel); +extern void populate_joinrel_uniquekeys(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + RelOptInfo *rel2, + List *restrictlist, + JoinType jointype); + +extern bool relation_has_uniquekeys_for(PlannerInfo *root, + RelOptInfo *rel, + List *exprs, + bool allow_multinulls); +extern bool relation_is_onerow(RelOptInfo *rel); #endif /* PATHS_H */ -- 2.26.2 --yh7cw345mbttadjz Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v38-0003-Extend-UniqueKeys.patch" ^ permalink raw reply [nested|flat] 91+ messages in thread
* [PATCH v36 2/6] Introduce UniqueKey attributes on RelOptInfo struct. @ 2020-05-11 07:50 一挃 <[email protected]> 0 siblings, 0 replies; 91+ messages in thread From: 一挃 @ 2020-05-11 07:50 UTC (permalink / raw) UniqueKey is a set of exprs on RelOptInfo which represents the exprs will be unique on the given RelOptInfo. You can see README.uniquekey for more information. --- src/backend/nodes/copyfuncs.c | 13 + src/backend/nodes/list.c | 31 + src/backend/nodes/makefuncs.c | 13 + src/backend/nodes/outfuncs.c | 11 + src/backend/nodes/readfuncs.c | 10 + src/backend/optimizer/path/Makefile | 3 +- src/backend/optimizer/path/README.uniquekey | 131 +++ src/backend/optimizer/path/allpaths.c | 10 + src/backend/optimizer/path/joinpath.c | 9 +- src/backend/optimizer/path/joinrels.c | 2 + src/backend/optimizer/path/pathkeys.c | 3 +- src/backend/optimizer/path/uniquekeys.c | 1131 +++++++++++++++++++ src/backend/optimizer/plan/planner.c | 13 +- src/backend/optimizer/prep/prepunion.c | 2 + src/backend/optimizer/util/appendinfo.c | 44 + src/backend/optimizer/util/inherit.c | 16 +- src/include/nodes/makefuncs.h | 3 + src/include/nodes/nodes.h | 1 + src/include/nodes/pathnodes.h | 29 +- src/include/nodes/pg_list.h | 2 + src/include/optimizer/appendinfo.h | 3 + src/include/optimizer/optimizer.h | 2 + src/include/optimizer/paths.h | 43 + 23 files changed, 1502 insertions(+), 23 deletions(-) create mode 100644 src/backend/optimizer/path/README.uniquekey create mode 100644 src/backend/optimizer/path/uniquekeys.c diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 89c409de66..1f50400fd2 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2273,6 +2273,16 @@ _copyPathKey(const PathKey *from) return newnode; } +static UniqueKey * +_copyUniqueKey(const UniqueKey *from) +{ + UniqueKey *newnode = makeNode(UniqueKey); + + COPY_NODE_FIELD(exprs); + COPY_SCALAR_FIELD(multi_nullvals); + + return newnode; +} /* * _copyRestrictInfo */ @@ -5152,6 +5162,9 @@ copyObjectImpl(const void *from) case T_PathKey: retval = _copyPathKey(from); break; + case T_UniqueKey: + retval = _copyUniqueKey(from); + break; case T_RestrictInfo: retval = _copyRestrictInfo(from); break; diff --git a/src/backend/nodes/list.c b/src/backend/nodes/list.c index 80fa8c84e4..a7a99b70f2 100644 --- a/src/backend/nodes/list.c +++ b/src/backend/nodes/list.c @@ -687,6 +687,37 @@ list_member_oid(const List *list, Oid datum) return false; } +/* + * return true iff every entry in "members" list is also present + * in the "target" list. + */ +bool +list_is_subset(const List *members, const List *target) +{ + const ListCell *lc1, *lc2; + + Assert(IsPointerList(members)); + Assert(IsPointerList(target)); + check_list_invariants(members); + check_list_invariants(target); + + foreach(lc1, members) + { + bool found = false; + foreach(lc2, target) + { + if (equal(lfirst(lc1), lfirst(lc2))) + { + found = true; + break; + } + } + if (!found) + return false; + } + return true; +} + /* * Delete the n'th cell (counting from 0) in list. * diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 49de285f01..646cf7c9a1 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -814,3 +814,16 @@ makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols) v->va_cols = va_cols; return v; } + + +/* + * makeUniqueKey + */ +UniqueKey* +makeUniqueKey(List *exprs, bool multi_nullvals) +{ + UniqueKey * ukey = makeNode(UniqueKey); + ukey->exprs = exprs; + ukey->multi_nullvals = multi_nullvals; + return ukey; +} diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index e2f177515d..c3a9632992 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2428,6 +2428,14 @@ _outPathKey(StringInfo str, const PathKey *node) WRITE_BOOL_FIELD(pk_nulls_first); } +static void +_outUniqueKey(StringInfo str, const UniqueKey *node) +{ + WRITE_NODE_TYPE("UNIQUEKEY"); + WRITE_NODE_FIELD(exprs); + WRITE_BOOL_FIELD(multi_nullvals); +} + static void _outPathTarget(StringInfo str, const PathTarget *node) { @@ -4127,6 +4135,9 @@ outNode(StringInfo str, const void *obj) case T_PathKey: _outPathKey(str, obj); break; + case T_UniqueKey: + _outUniqueKey(str, obj); + break; case T_PathTarget: _outPathTarget(str, obj); break; diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index 42050ab719..3a18571d0c 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -452,6 +452,14 @@ _readSetOperationStmt(void) READ_DONE(); } +static UniqueKey * +_readUniqueKey(void) +{ + READ_LOCALS(UniqueKey); + READ_NODE_FIELD(exprs); + READ_BOOL_FIELD(multi_nullvals); + READ_DONE(); +} /* * Stuff from primnodes.h. @@ -2656,6 +2664,8 @@ parseNodeString(void) return_value = _readCommonTableExpr(); else if (MATCH("SETOPERATIONSTMT", 16)) return_value = _readSetOperationStmt(); + else if (MATCH("UNIQUEKEY", 9)) + return_value = _readUniqueKey(); else if (MATCH("ALIAS", 5)) return_value = _readAlias(); else if (MATCH("RANGEVAR", 8)) diff --git a/src/backend/optimizer/path/Makefile b/src/backend/optimizer/path/Makefile index 1e199ff66f..7b9820c25f 100644 --- a/src/backend/optimizer/path/Makefile +++ b/src/backend/optimizer/path/Makefile @@ -21,6 +21,7 @@ OBJS = \ joinpath.o \ joinrels.o \ pathkeys.o \ - tidpath.o + tidpath.o \ + uniquekeys.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/optimizer/path/README.uniquekey b/src/backend/optimizer/path/README.uniquekey new file mode 100644 index 0000000000..5eac761995 --- /dev/null +++ b/src/backend/optimizer/path/README.uniquekey @@ -0,0 +1,131 @@ +1. What is UniqueKey? +We can think UniqueKey is a set of exprs for a RelOptInfo, which we are insure +that doesn't yields same result among all the rows. The simplest UniqueKey +format is primary key. + +However we define the UnqiueKey as below. + +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + +exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL +is a special case of UniqueKey, which means there is only one row in that +relation.it has a stronger semantic than others. like SELECT uk FROM t; uk is +normal unique key and may have different values. SELECT colx FROM t WHERE uk = +const. colx is unique AND we have only 1 value. This field can used for +innerrel_is_unique. this logic is handled specially in add_uniquekey_for_onerow +function. + +multi_nullvals: true means multi null values may exist in these exprs, so the +uniqueness is not guaranteed in this case. This field is necessary for +remove_useless_join & reduce_unique_semijoins where we don't mind these +duplicated NULL values. It is set to true for 2 cases. One is a unique key +from a unique index but the related column is nullable. The other one is for +outer join. see populate_joinrel_uniquekeys for detail. + + +The UniqueKey can be used at the following cases at least: +1. remove_useless_joins. +2. reduce_semianti_joins +3. remove distinct node if distinct clause is unique. +4. remove aggnode if group by clause is unique. +5. Index Skip Scan (WIP) +6. Aggregation Push Down without 2 phase aggregation if the join can't + duplicated the aggregated rows. (work in progress feature) + +2. How is it maintained? + +We have a set of populate_xxx_unqiuekeys functions to maintain the uniquekey on +various cases. xxx includes baserel, joinrel, partitionedrel, distinctrel, +groupedrel, unionrel. and we also need to convert the uniquekey from subquery +to outer relation, which is what convert_subquery_uniquekeys does. + +1. The first part is about baserel. We handled 3 cases. suppose we have Unique +Index on (a, b). + +1. SELECT a, b FROM t. UniqueKey (a, b) +2. SELECT a FROM t WHERE b = 1; UniqueKey (a) +3. SELECT .. FROM t WHERE a = 1 AND b = 1; UniqueKey (NIL). onerow case, every + column is Unique. + +2. The next part is joinrel, this part is most error-prone, we simplified the rules +like below: +1. If the relation's UniqueKey can't be duplicated after join, then is will be + still valid for the join rel. The function we used here is + innerrel_keeps_unique. The basic idea is innerrel.any_col = outer.uk. + +2. If the UnqiueKey can't keep valid via the rule 1, the combination of the + UniqueKey from both sides are valid for sure. We can prove this as: if the + unique exprs from rel1 is duplicated by rel2, the duplicated rows must + contains different unique exprs from rel2. + +More considerations about onerow: +1. If relation with one row and it can't be duplicated, it is still possible + contains mulit_nullvas after outer join. +2. If the either UniqueKey can be duplicated after join, the can get one row + only when both side is one row AND there is no outer join. +3. Whenever the onerow UniqueKey is not a valid any more, we need to convert one + row UniqueKey to normal unique key since we don't store exprs for one-row + relation. get_exprs_from_uniquekeys will be used here. + + +More considerations about multi_nullvals after join: +1. If the original UnqiueKey has multi_nullvals, the final UniqueKey will have + mulit_nullvals in any case. +2. If a unique key doesn't allow mulit_nullvals, after some outer join, it + allows some outer join. + + +3. When we comes to subquery, we need to convert_subquery_unqiuekeys just like +convert_subquery_pathkeys. Only the UniqueKey insides subquery is referenced as +a Var in outer relation will be reused. The relationship between the outerrel.Var +and subquery.exprs is built with outerel->subroot->processed_tlist. + + +4. As for the SRF functions, it will break the uniqueness of uniquekey, However it +is handled in adjust_paths_for_srfs, which happens after the query_planner. so +we will maintain the UniqueKey until there and reset it to NIL at that +places. This can't help on distinct/group by elimination cases but probably help +in some other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is +semantic correctly. + + +5. As for inherit table, we first main the UnqiueKey on childrel as well. But for +partitioned table we need to maintain 2 different kinds of +UnqiueKey. 1). UniqueKey on the parent relation 2). UniqueKey on child +relation for partition wise query. + +Example: +CREATE TABLE p (a int not null, b int not null) partition by list (a); +CREATE TABLE p0 partition of p for values in (1); +CREATE TABLE p1 partition of p for values in (2); + +create unique index p0_b on p0(b); +create unique index p1_b on p1(b); + +Now b is only unique on partition level, so the distinct can't be removed on +the following cases. SELECT DISTINCT b FROM p; + +Another example is SELECT DISTINCT a, b FROM p WHERE a = 1; Since only one +partition is chosen, the UniqueKey on child relation is same as the UniqueKey on +parent relation. + +Another usage of UniqueKey on partition level is it be helpful for +partition-wise join. + +As for the UniqueKey on parent table level, it comes with 2 different ways, +1). the UniqueKey is also derived in UniqueKey index, but the index must be same +in all the related children relations and the unique index must contains +Partition Key in it. Example: + +CREATE UNIQUE INDEX p_ab ON p(a, b); -- where a is the partition key. + +-- Query +SELECT a, b FROM p; the (a, b) is a UniqueKey of p. + +2). If the parent relation has only one childrel, the UniqueKey on childrel is + the UniqueKey on parent as well. diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index 484dab0a1a..2ad9d06d7a 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -579,6 +579,12 @@ set_plain_rel_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) */ check_index_predicates(root, rel); + /* + * Now that we've marked which partial indexes are suitable, we can now + * build the relation's unique keys. + */ + populate_baserel_uniquekeys(root, rel, rel->indexlist); + /* Mark rel with estimated output rows, width, etc */ set_baserel_size_estimates(root, rel); } @@ -1310,6 +1316,8 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, /* Add paths to the append relation. */ add_paths_to_append_rel(root, rel, live_childrels); + if (IS_PARTITIONED_REL(rel)) + populate_partitionedrel_uniquekeys(root, rel, live_childrels); } @@ -2383,6 +2391,8 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, pathkeys, required_outer)); } + convert_subquery_uniquekeys(root, rel, sub_final_rel); + /* If outer rel allows parallelism, do same for partial paths. */ if (rel->consider_parallel && bms_is_empty(required_outer)) { diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c index db54a6ba2e..ef0fd2fb0b 100644 --- a/src/backend/optimizer/path/joinpath.c +++ b/src/backend/optimizer/path/joinpath.c @@ -71,13 +71,6 @@ static void consider_parallel_mergejoin(PlannerInfo *root, static void hash_inner_and_outer(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, RelOptInfo *innerrel, JoinType jointype, JoinPathExtraData *extra); -static List *select_mergejoin_clauses(PlannerInfo *root, - RelOptInfo *joinrel, - RelOptInfo *outerrel, - RelOptInfo *innerrel, - List *restrictlist, - JoinType jointype, - bool *mergejoin_allowed); static void generate_mergejoin_paths(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *innerrel, @@ -1927,7 +1920,7 @@ hash_inner_and_outer(PlannerInfo *root, * if it is mergejoinable and involves vars from the two sub-relations * currently of interest. */ -static List * +List * select_mergejoin_clauses(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c index 2d343cd293..b9163ee8ff 100644 --- a/src/backend/optimizer/path/joinrels.c +++ b/src/backend/optimizer/path/joinrels.c @@ -924,6 +924,8 @@ populate_joinrel_with_paths(PlannerInfo *root, RelOptInfo *rel1, /* Apply partitionwise join technique, if possible. */ try_partitionwise_join(root, rel1, rel2, joinrel, sjinfo, restrictlist); + + populate_joinrel_uniquekeys(root, joinrel, rel1, rel2, restrictlist, sjinfo->jointype); } diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index ce9bf87e9b..7e596d4194 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -33,7 +33,6 @@ static bool pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys); static bool matches_boolean_partition_clause(RestrictInfo *rinfo, RelOptInfo *partrel, int partkeycol); -static Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); static bool right_merge_direction(PlannerInfo *root, PathKey *pathkey); @@ -1035,7 +1034,7 @@ convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel, * We need this to ensure that we don't return pathkeys describing values * that are unavailable above the level of the subquery scan. */ -static Var * +Var * find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle) { ListCell *lc; diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c new file mode 100644 index 0000000000..b33bcd2f32 --- /dev/null +++ b/src/backend/optimizer/path/uniquekeys.c @@ -0,0 +1,1131 @@ +/*------------------------------------------------------------------------- + * + * uniquekeys.c + * Utilities for matching and building unique keys + * + * Portions Copyright (c) 2020, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/optimizer/path/uniquekeys.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "nodes/makefuncs.h" +#include "nodes/nodeFuncs.h" +#include "optimizer/pathnode.h" +#include "optimizer/paths.h" +#include "optimizer/appendinfo.h" +#include "optimizer/optimizer.h" +#include "optimizer/tlist.h" +#include "rewrite/rewriteManip.h" + + +/* + * This struct is used to help populate_joinrel_uniquekeys. + * + * added_to_joinrel is true if a uniquekey (from outerrel or innerrel) + * has been added to joinrel. + * useful is true if the exprs of the uniquekey still appears in joinrel. + */ +typedef struct UniqueKeyContextData +{ + UniqueKey *uniquekey; + bool added_to_joinrel; + bool useful; +} *UniqueKeyContext; + +static List *initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel); +static bool innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + bool reverse); + +static List *get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals); +static List *get_exprs_from_uniquekey(RelOptInfo *joinrel, + RelOptInfo *rel1, + UniqueKey *ukey); +static void add_uniquekey_for_onerow(RelOptInfo *rel); +static bool add_combined_uniquekey(RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype); + +/* Used for unique indexes checking for partitioned table */ +static bool index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind); +static IndexOptInfo *simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from); +static bool simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2); +static void adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_index); + +/* Helper function for grouped relation and distinct relation. */ +static void add_uniquekey_from_sortgroups(PlannerInfo *root, + RelOptInfo *rel, + List *sortgroups); + +/* + * populate_baserel_uniquekeys + * Populate 'baserel' uniquekeys list by looking at the rel's unique index + * and baserestrictinfo + */ +void +populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List *indexlist) +{ + ListCell *lc; + List *matched_uniq_indexes = NIL; + + /* Attrs appears in rel->reltarget->exprs. */ + Bitmapset *used_attrs = NULL; + + List *const_exprs = NIL; + List *expr_opfamilies = NIL; + + Assert(baserel->rtekind == RTE_RELATION); + + foreach(lc, indexlist) + { + IndexOptInfo *ind = (IndexOptInfo *) lfirst(lc); + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + matched_uniq_indexes = lappend(matched_uniq_indexes, ind); + } + + if (matched_uniq_indexes == NIL) + return; + + /* Check which attrs is used in baserel->reltarget */ + pull_varattnos((Node *)baserel->reltarget->exprs, baserel->relid, &used_attrs); + + /* Check which attrno is used at a mergeable const filter */ + foreach(lc, baserel->baserestrictinfo) + { + RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); + + if (rinfo->mergeopfamilies == NIL) + continue; + + if (bms_is_empty(rinfo->left_relids)) + { + const_exprs = lappend(const_exprs, get_rightop(rinfo->clause)); + } + else if (bms_is_empty(rinfo->right_relids)) + { + const_exprs = lappend(const_exprs, get_leftop(rinfo->clause)); + } + else + continue; + + expr_opfamilies = lappend(expr_opfamilies, rinfo->mergeopfamilies); + } + + foreach(lc, matched_uniq_indexes) + { + bool multi_nullvals, useful; + List *exprs = get_exprs_from_uniqueindex(lfirst_node(IndexOptInfo, lc), + const_exprs, + expr_opfamilies, + used_attrs, + &useful, + &multi_nullvals); + if (useful) + { + if (exprs == NIL) + { + /* All the columns in Unique Index matched with a restrictinfo */ + add_uniquekey_for_onerow(baserel); + return; + } + baserel->uniquekeys = lappend(baserel->uniquekeys, + makeUniqueKey(exprs, multi_nullvals)); + } + } +} + + +/* + * populate_partitionedrel_uniquekeys + * The UniqueKey on partitionrel comes from 2 cases: + * 1). Only one partition is involved in this query, the unique key can be + * copied to parent rel from childrel. + * 2). There are some unique index which includes partition key and exists + * in all the related partitions. + * We never mind rule 2 if we hit rule 1. + */ + +void +populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels) +{ + ListCell *lc; + List *global_uniq_indexlist = NIL; + RelOptInfo *childrel; + bool is_first = true; + + Assert(IS_PARTITIONED_REL(rel)); + + if (childrels == NIL) + return; + + /* + * If there is only one partition used in this query, the UniqueKey in childrel is + * still valid in parent level, but we need convert the format from child expr to + * parent expr. + */ + if (list_length(childrels) == 1) + { + /* Check for Rule 1 */ + RelOptInfo *childrel = linitial_node(RelOptInfo, childrels); + ListCell *lc; + Assert(childrel->reloptkind == RELOPT_OTHER_MEMBER_REL); + if (relation_is_onerow(childrel)) + { + add_uniquekey_for_onerow(rel); + return; + } + + foreach(lc, childrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + AppendRelInfo *appinfo = find_appinfo_by_child(root, childrel->relid); + List *parent_exprs = NIL; + bool can_reuse = true; + ListCell *lc2; + foreach(lc2, ukey->exprs) + { + Var *var = (Var *)lfirst(lc2); + /* + * If the expr comes from a expression, it is hard to build the expression + * in parent so ignore that case for now. + */ + if(!IsA(var, Var)) + { + can_reuse = false; + break; + } + /* Convert it to parent var */ + parent_exprs = lappend(parent_exprs, find_parent_var(appinfo, var)); + } + if (can_reuse) + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(parent_exprs, + ukey->multi_nullvals)); + } + } + else + { + /* Check for rule 2 */ + childrel = linitial_node(RelOptInfo, childrels); + foreach(lc, childrel->indexlist) + { + IndexOptInfo *ind = lfirst(lc); + IndexOptInfo *modified_index; + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + + /* + * During simple_copy_indexinfo_to_parent, we need to convert var from + * child var to parent var, index on expression is too complex to handle. + * so ignore it for now. + */ + if (ind->indexprs != NIL) + continue; + + modified_index = simple_copy_indexinfo_to_parent(root, rel, ind); + /* + * If the unique index doesn't contain partkey, then it is unique + * on this partition only, so it is useless for us. + */ + if (!index_constains_partkey(rel, modified_index)) + continue; + + global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index); + } + + if (global_uniq_indexlist != NIL) + { + foreach(lc, childrels) + { + RelOptInfo *child = lfirst(lc); + if (is_first) + { + is_first = false; + continue; + } + adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist); + } + /* Now we have a list of unique index which are exactly same on all childrels, + * Set the UniqueKey just like it is non-partition table + */ + populate_baserel_uniquekeys(root, rel, global_uniq_indexlist); + } + } +} + + +/* + * populate_distinctrel_uniquekeys + */ +void +populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel) +{ + /* The unique key before the distinct is still valid. */ + distinctrel->uniquekeys = list_copy(inputrel->uniquekeys); + add_uniquekey_from_sortgroups(root, distinctrel, root->parse->distinctClause); +} + +/* + * populate_grouprel_uniquekeys + */ +void +populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel) + +{ + Query *parse = root->parse; + bool input_ukey_added = false; + ListCell *lc; + + if (relation_is_onerow(inputrel)) + { + add_uniquekey_for_onerow(grouprel); + return; + } + if (parse->groupingSets) + return; + + /* A Normal group by without grouping set. */ + if (parse->groupClause) + { + /* + * Current even the groupby clause is Unique already, but if query has aggref + * We have to create grouprel still. To keep the UnqiueKey short, we will check + * the UniqueKey of input_rel still valid, if so we reuse it. + */ + foreach(lc, inputrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (list_is_subset(ukey->exprs, grouprel->reltarget->exprs)) + { + grouprel->uniquekeys = lappend(grouprel->uniquekeys, + ukey); + input_ukey_added = true; + } + } + if (!input_ukey_added) + /* + * group by clause must be a super-set of grouprel->reltarget->exprs except the + * aggregation expr, so if such exprs is unique already, no bother to generate + * new uniquekey for group by exprs. + */ + add_uniquekey_from_sortgroups(root, + grouprel, + root->parse->groupClause); + } + else + /* It has aggregation but without a group by, so only one row returned */ + add_uniquekey_for_onerow(grouprel); +} + +/* + * simple_copy_uniquekeys + * Using a function for the one-line code makes us easy to check where we simply + * copied the uniquekey. + */ +void +simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel) +{ + newrel->uniquekeys = oldrel->uniquekeys; +} + +/* + * populate_unionrel_uniquekeys + */ +void +populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel) +{ + ListCell *lc; + List *exprs = NIL; + + Assert(unionrel->uniquekeys == NIL); + + foreach(lc, unionrel->reltarget->exprs) + { + exprs = lappend(exprs, lfirst(lc)); + } + + if (exprs == NIL) + /* SQL: select union select; is valid, we need to handle it here. */ + add_uniquekey_for_onerow(unionrel); + else + unionrel->uniquekeys = lappend(unionrel->uniquekeys, + makeUniqueKey(exprs,false)); + +} + +/* + * populate_joinrel_uniquekeys + * + * populate uniquekeys for joinrel. We will check each relation to see if its + * UniqueKey is still valid via innerrel_keeps_unique, if so, we add it to + * joinrel. The multi_nullvals field will be changed to true for some outer + * join cases and one-row UniqueKey needs to be converted to normal UniqueKey + * for the same case as well. + * For the uniquekey in either baserel which can't be unique after join, we still + * check to see if combination of UniqueKeys from both side is still useful for us. + * if yes, we add it to joinrel as well. + */ +void +populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *outerrel, RelOptInfo *innerrel, + List *restrictlist, JoinType jointype) +{ + ListCell *lc, *lc2; + List *clause_list = NIL; + List *outerrel_ukey_ctx; + List *innerrel_ukey_ctx; + bool inner_onerow, outer_onerow; + bool mergejoin_allowed; + + /* Care about the outerrel relation only for SEMI/ANTI join */ + if (jointype == JOIN_SEMI || jointype == JOIN_ANTI) + { + foreach(lc, outerrel->uniquekeys) + { + UniqueKey *uniquekey = lfirst_node(UniqueKey, lc); + if (list_is_subset(uniquekey->exprs, joinrel->reltarget->exprs)) + joinrel->uniquekeys = lappend(joinrel->uniquekeys, uniquekey); + } + return; + } + + Assert(jointype == JOIN_LEFT || jointype == JOIN_FULL || jointype == JOIN_INNER); + + /* Fast path */ + if (innerrel->uniquekeys == NIL || outerrel->uniquekeys == NIL) + return; + + inner_onerow = relation_is_onerow(innerrel); + outer_onerow = relation_is_onerow(outerrel); + + outerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(outerrel); + innerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(innerrel); + + clause_list = select_mergejoin_clauses(root, joinrel, outerrel, innerrel, + restrictlist, jointype, + &mergejoin_allowed); + + if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true /* reverse */)) + { + bool outer_impact = jointype == JOIN_FULL; + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + /* Outer relation has one row, and the unique key is not duplicated after join, + * the joinrel will still has one row unless the jointype == JOIN_FULL. + */ + if (outer_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (outer_onerow) + { + /* + * The onerow outerrel becomes multi rows and multi_nullvals + * will be changed to true. We also need to set the exprs correctly since it + * can't be NIL any more. + */ + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(joinrel, outerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Change multi_nullvals to true due to the full join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, true)); + else + /* Just reuse it */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + } + ctx->added_to_joinrel = true; + } + } + + if (innerrel_keeps_unique(root, outerrel, innerrel, clause_list, false)) + { + bool outer_impact = jointype == JOIN_FULL || jointype == JOIN_LEFT;; + + foreach(lc, innerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + if (inner_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (inner_onerow) + { + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(joinrel, innerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Need to change multi_nullvals to true due to the outer join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, + true)); + else + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + + } + ctx->added_to_joinrel = true; + } + } + + /* + * The combination of the UniqueKey from both sides is unique as well regardless + * of join type, but no bother to add it if its subset has been added to joinrel + * already or it is not useful for the joinrel. + */ + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx1 = (UniqueKeyContext) lfirst(lc); + if (ctx1->added_to_joinrel || !ctx1->useful) + continue; + foreach(lc2, innerrel_ukey_ctx) + { + UniqueKeyContext ctx2 = (UniqueKeyContext) lfirst(lc2); + if (ctx2->added_to_joinrel || !ctx2->useful) + continue; + if (add_combined_uniquekey(joinrel, outerrel, innerrel, + ctx1->uniquekey, ctx2->uniquekey, + jointype)) + /* If we set a onerow UniqueKey to joinrel, we don't need other. */ + return; + } + } +} + + +/* + * convert_subquery_uniquekeys + * + * Covert the UniqueKey in subquery to outer relation. + */ +void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel) +{ + ListCell *lc; + + if (sub_final_rel->uniquekeys == NIL) + return; + + if (relation_is_onerow(sub_final_rel)) + { + add_uniquekey_for_onerow(currel); + return; + } + + Assert(currel->subroot != NULL); + + foreach(lc, sub_final_rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + ListCell *lc; + List *exprs = NIL; + bool ukey_useful = true; + + /* One row case is handled above */ + Assert(ukey->exprs != NIL); + foreach(lc, ukey->exprs) + { + Var *var; + TargetEntry *tle = tlist_member(lfirst(lc), + currel->subroot->processed_tlist); + if (tle == NULL) + { + ukey_useful = false; + break; + } + var = find_var_for_subquery_tle(currel, tle); + if (var == NULL) + { + ukey_useful = false; + break; + } + exprs = lappend(exprs, var); + } + + if (ukey_useful) + currel->uniquekeys = lappend(currel->uniquekeys, + makeUniqueKey(exprs, + ukey->multi_nullvals)); + + } +} + +/* + * innerrel_keeps_unique + * + * Check if Unique key of the innerrel is valid after join. innerrel's UniqueKey + * will be still valid if innerrel's any-column mergeop outrerel's uniquekey + * exists in clause_list. + * + * Note: the clause_list must be a list of mergeable restrictinfo already. + */ +static bool +innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *clause_list, + bool reverse) +{ + ListCell *lc, *lc2, *lc3; + + if (outerrel->uniquekeys == NIL || innerrel->uniquekeys == NIL) + return false; + + /* Check if there is outerrel's uniquekey in mergeable clause. */ + foreach(lc, outerrel->uniquekeys) + { + List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs; + bool clauselist_matchs_all_exprs = true; + foreach(lc2, outer_uq_exprs) + { + Node *outer_uq_expr = lfirst(lc2); + bool find_uq_expr_in_clauselist = false; + foreach(lc3, clause_list) + { + RestrictInfo *rinfo = lfirst_node(RestrictInfo, lc3); + Node *outer_expr; + if (reverse) + outer_expr = rinfo->outer_is_left ? get_rightop(rinfo->clause) : get_leftop(rinfo->clause); + else + outer_expr = rinfo->outer_is_left ? get_leftop(rinfo->clause) : get_rightop(rinfo->clause); + if (equal(outer_expr, outer_uq_expr)) + { + find_uq_expr_in_clauselist = true; + break; + } + } + if (!find_uq_expr_in_clauselist) + { + /* No need to check the next exprs in the current uniquekey */ + clauselist_matchs_all_exprs = false; + break; + } + } + + if (clauselist_matchs_all_exprs) + return true; + } + return false; +} + + +/* + * relation_is_onerow + * Check if it is a one-row relation by checking UniqueKey. + */ +bool +relation_is_onerow(RelOptInfo *rel) +{ + UniqueKey *ukey; + if (rel->uniquekeys == NIL) + return false; + ukey = linitial_node(UniqueKey, rel->uniquekeys); + return ukey->exprs == NIL && list_length(rel->uniquekeys) == 1; +} + +/* + * relation_has_uniquekeys_for + * Returns true if we have proofs that 'rel' cannot return multiple rows with + * the same values in each of 'exprs'. Otherwise returns false. + */ +bool +relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel, + List *exprs, bool allow_multinulls) +{ + ListCell *lc; + + /* + * For UniqueKey->onerow case, the uniquekey->exprs is empty as well + * so we can't rely on list_is_subset to handle this special cases + */ + if (exprs == NIL) + return false; + + foreach(lc, rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (ukey->multi_nullvals && !allow_multinulls) + continue; + if (list_is_subset(ukey->exprs, exprs)) + return true; + } + return false; +} + + +/* + * get_exprs_from_uniqueindex + * + * Return a list of exprs which is unique. set useful to false if this + * unique index is not useful for us. + */ +static List * +get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals) +{ + List *exprs = NIL; + ListCell *indexpr_item; + int c = 0; + + *useful = true; + *multi_nullvals = false; + + indexpr_item = list_head(unique_index->indexprs); + for(c = 0; c < unique_index->ncolumns; c++) + { + int attr = unique_index->indexkeys[c]; + Expr *expr; + bool matched_const = false; + ListCell *lc1, *lc2; + + if(attr > 0) + { + expr = list_nth_node(TargetEntry, unique_index->indextlist, c)->expr; + } + else if (attr == 0) + { + /* Expression index */ + expr = lfirst(indexpr_item); + indexpr_item = lnext(unique_index->indexprs, indexpr_item); + } + else /* attr < 0 */ + { + /* Index on system column is not supported */ + Assert(false); + } + + /* + * Check index_col = Const case with regarding to opfamily checking + * If we can remove the index_col from the final UniqueKey->exprs. + */ + forboth(lc1, const_exprs, lc2, const_expr_opfamilies) + { + if (list_member_oid((List *)lfirst(lc2), unique_index->opfamily[c]) + && match_index_to_operand((Node *) lfirst(lc1), c, unique_index)) + { + matched_const = true; + break; + } + } + + if (matched_const) + continue; + + /* Check if the indexed expr is used in rel */ + if (attr > 0) + { + /* + * Normal Indexed column, if the col is not used, then the index is useless + * for uniquekey. + */ + attr -= FirstLowInvalidHeapAttributeNumber; + + if (!bms_is_member(attr, used_varattrs)) + { + *useful = false; + break; + } + } + else if (!list_member(unique_index->rel->reltarget->exprs, expr)) + { + /* Expression index but the expression is not used in rel */ + *useful = false; + break; + } + + /* check not null property. */ + if (attr == 0) + { + /* We never know if a expression yields null or not */ + *multi_nullvals = true; + } + else if (!bms_is_member(attr, unique_index->rel->notnullattrs) + && !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber, + unique_index->rel->notnullattrs)) + { + *multi_nullvals = true; + } + + exprs = lappend(exprs, expr); + } + return exprs; +} + + +/* + * add_uniquekey_for_onerow + * If we are sure that the relation only returns one row, then all the columns + * are unique. However we don't need to create UniqueKey for every column, we + * just set exprs = NIL and overwrites all the other UniqueKey on this RelOptInfo + * since this one has strongest semantics. + */ +void +add_uniquekey_for_onerow(RelOptInfo *rel) +{ + /* + * We overwrite the previous UniqueKey on purpose since this one has the + * strongest semantic. + */ + rel->uniquekeys = list_make1(makeUniqueKey(NIL, false)); +} + + +/* + * initililze_uniquecontext_for_joinrel + * Return a List of UniqueKeyContext for an inputrel + */ +static List * +initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel) +{ + List *res = NIL; + ListCell *lc; + foreach(lc, inputrel->uniquekeys) + { + UniqueKeyContext context; + context = palloc(sizeof(struct UniqueKeyContextData)); + context->uniquekey = lfirst_node(UniqueKey, lc); + context->added_to_joinrel = false; + context->useful = true; + res = lappend(res, context); + } + return res; +} + + +/* + * get_exprs_from_uniquekey + * Unify the way of get List of exprs from a one-row UniqueKey or + * normal UniqueKey. for the onerow case, every expr in rel1 is a valid + * UniqueKey. Return a List of exprs. + * + * rel1: The relation which you want to get the exprs. + * ukey: The UniqueKey you want to get the exprs. + */ +static List * +get_exprs_from_uniquekey(RelOptInfo *joinrel, RelOptInfo *rel1, UniqueKey *ukey) +{ + ListCell *lc; + bool onerow = rel1 != NULL && relation_is_onerow(rel1); + + List *res = NIL; + Assert(onerow || ukey); + if (onerow) + { + /* Only cares about the exprs still exist in joinrel */ + foreach(lc, joinrel->reltarget->exprs) + { + Bitmapset *relids = pull_varnos(lfirst(lc)); + if (bms_is_subset(relids, rel1->relids)) + { + res = lappend(res, list_make1(lfirst(lc))); + } + } + } + else + { + res = list_make1(ukey->exprs); + } + return res; +} + +/* + * Partitioned table Unique Keys. + * The partition table unique key is maintained as: + * 1. The index must be unique as usual. + * 2. The index must contains partition key. + * 3. The index must exist on all the child rel. see simple_indexinfo_equal for + * how we compare it. + */ + +/* + * index_constains_partkey + * return true if the index contains the partiton key. + */ +static bool +index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind) +{ + ListCell *lc; + int i; + Assert(IS_PARTITIONED_REL(partrel)); + Assert(partrel->part_scheme->partnatts > 0); + + for(i = 0; i < partrel->part_scheme->partnatts; i++) + { + Node *part_expr = linitial(partrel->partexprs[i]); + bool found_in_index = false; + foreach(lc, ind->indextlist) + { + Expr *index_expr = lfirst_node(TargetEntry, lc)->expr; + if (equal(index_expr, part_expr)) + { + found_in_index = true; + break; + } + } + if (!found_in_index) + return false; + } + return true; +} + +/* + * simple_indexinfo_equal + * + * Used to check if the 2 index is same as each other. The index here + * is COPIED from childrel and did some tiny changes(see + * simple_copy_indexinfo_to_parent) + */ +static bool +simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2) +{ + Size oid_cmp_len = sizeof(Oid) * ind1->ncolumns; + + return ind1->ncolumns == ind2->ncolumns && + ind1->unique == ind2->unique && + memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0 && + memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0 && + memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0 && + memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0 && + equal(get_tlist_exprs(ind1->indextlist, true), + get_tlist_exprs(ind2->indextlist, true)); +} + + +/* + * The below macros are used for simple_copy_indexinfo_to_parent which is so + * customized that I don't want to put it to copyfuncs.c. So copy it here. + */ +#define COPY_POINTER_FIELD(fldname, sz) \ + do { \ + Size _size = (sz); \ + newnode->fldname = palloc(_size); \ + memcpy(newnode->fldname, from->fldname, _size); \ + } while (0) + +#define COPY_NODE_FIELD(fldname) \ + (newnode->fldname = copyObjectImpl(from->fldname)) + +#define COPY_SCALAR_FIELD(fldname) \ + (newnode->fldname = from->fldname) + + +/* + * simple_copy_indexinfo_to_parent (from partition) + * Copy the IndexInfo from child relation to parent relation with some modification, + * which is used to test: + * 1. If the same index exists in all the childrels. + * 2. If the parentrel->reltarget/basicrestrict info matches this index. + */ +static IndexOptInfo * +simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from) +{ + IndexOptInfo *newnode = makeNode(IndexOptInfo); + AppendRelInfo *appinfo = find_appinfo_by_child(root, from->rel->relid); + ListCell *lc; + int idx = 0; + + COPY_SCALAR_FIELD(ncolumns); + COPY_SCALAR_FIELD(nkeycolumns); + COPY_SCALAR_FIELD(unique); + COPY_SCALAR_FIELD(immediate); + /* We just need to know if it is NIL or not */ + COPY_SCALAR_FIELD(indpred); + COPY_SCALAR_FIELD(predOK); + COPY_POINTER_FIELD(indexkeys, from->ncolumns * sizeof(int)); + COPY_POINTER_FIELD(indexcollations, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opfamily, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opcintype, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(sortopfamily, from->ncolumns * sizeof(Oid)); + COPY_NODE_FIELD(indextlist); + + /* Convert index exprs on child expr to expr on parent */ + foreach(lc, newnode->indextlist) + { + TargetEntry *tle = lfirst_node(TargetEntry, lc); + /* Index on expression is ignored */ + Assert(IsA(tle->expr, Var)); + tle->expr = (Expr *) find_parent_var(appinfo, (Var *) tle->expr); + newnode->indexkeys[idx] = castNode(Var, tle->expr)->varattno; + idx++; + } + newnode->rel = parentrel; + return newnode; +} + +/* + * adjust_partition_unique_indexlist + * + * global_unique_indexes: At the beginning, it contains the copy & modified + * unique index from the first partition. And then check if each index in it still + * exists in the following partitions. If no, remove it. at last, it has an + * index list which exists in all the partitions. + */ +static void +adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_indexes) +{ + ListCell *lc, *lc2; + foreach(lc, *global_unique_indexes) + { + IndexOptInfo *g_ind = lfirst_node(IndexOptInfo, lc); + bool found_in_child = false; + + foreach(lc2, childrel->indexlist) + { + IndexOptInfo *p_ind = lfirst_node(IndexOptInfo, lc2); + IndexOptInfo *p_ind_copy; + if (!p_ind->unique || !p_ind->immediate || + (p_ind->indpred != NIL && !p_ind->predOK)) + continue; + p_ind_copy = simple_copy_indexinfo_to_parent(root, parentrel, p_ind); + if (simple_indexinfo_equal(p_ind_copy, g_ind)) + { + found_in_child = true; + break; + } + } + if (!found_in_child) + /* The index doesn't exist in childrel, remove it from global_unique_indexes */ + *global_unique_indexes = foreach_delete_current(*global_unique_indexes, lc); + } +} + +/* Helper function for groupres/distinctrel */ +static void +add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgroups) +{ + Query *parse = root->parse; + List *exprs; + + /* + * XXX: If there are some vars which is not in current levelsup, the semantic is + * imprecise, should we avoid it or not? levelsup = 1 is just a demo, maybe we need to + * check every level other than 0, if so, looks we have to write another + * pull_var_walker. + */ + List *upper_vars = pull_vars_of_level((Node*)sortgroups, 1); + + if (upper_vars != NIL) + return; + + exprs = get_sortgrouplist_exprs(sortgroups, parse->targetList); + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(exprs, + false /* sortgroupclause can't be multi_nullvals */)); +} + + +/* + * add_combined_uniquekey + * The combination of both UniqueKeys is a valid UniqueKey for joinrel no matter + * the jointype. + */ +bool +add_combined_uniquekey(RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype) +{ + + ListCell *lc1, *lc2; + + /* Either side has multi_nullvals or we have outer join, + * the combined UniqueKey has multi_nullvals */ + bool multi_nullvals = outer_ukey->multi_nullvals || + inner_ukey->multi_nullvals || IS_OUTER_JOIN(jointype); + + /* The only case we can get onerow joinrel after join */ + if (relation_is_onerow(outer_rel) + && relation_is_onerow(inner_rel) + && jointype == JOIN_INNER) + { + add_uniquekey_for_onerow(joinrel); + return true; + } + + foreach(lc1, get_exprs_from_uniquekey(joinrel, outer_rel, outer_ukey)) + { + foreach(lc2, get_exprs_from_uniquekey(joinrel, inner_rel, inner_ukey)) + { + List *exprs = list_concat_copy(lfirst_node(List, lc1), lfirst_node(List, lc2)); + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(exprs, + multi_nullvals)); + } + } + return false; +} diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index b406d41e91..0551ae0512 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -2389,6 +2389,8 @@ grouping_planner(PlannerInfo *root, bool inheritance_update, add_path(final_rel, path); } + simple_copy_uniquekeys(current_rel, final_rel); + /* * Generate partial paths for final_rel, too, if outer query levels might * be able to make use of them. @@ -3899,6 +3901,8 @@ create_grouping_paths(PlannerInfo *root, } set_cheapest(grouped_rel); + + populate_grouprel_uniquekeys(root, grouped_rel, input_rel); return grouped_rel; } @@ -4615,7 +4619,7 @@ create_window_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(window_rel); - + simple_copy_uniquekeys(input_rel, window_rel); return window_rel; } @@ -4911,7 +4915,7 @@ create_distinct_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(distinct_rel); - + populate_distinctrel_uniquekeys(root, input_rel, distinct_rel); return distinct_rel; } @@ -5172,6 +5176,8 @@ create_ordered_paths(PlannerInfo *root, */ Assert(ordered_rel->pathlist != NIL); + simple_copy_uniquekeys(input_rel, ordered_rel); + return ordered_rel; } @@ -6049,6 +6055,9 @@ adjust_paths_for_srfs(PlannerInfo *root, RelOptInfo *rel, if (list_length(targets) == 1) return; + /* UniqueKey is not valid after handling the SRF. */ + rel->uniquekeys = NIL; + /* * Stack SRF-evaluation nodes atop each path for the rel. * diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c index 951aed80e7..e94e92937c 100644 --- a/src/backend/optimizer/prep/prepunion.c +++ b/src/backend/optimizer/prep/prepunion.c @@ -689,6 +689,8 @@ generate_union_paths(SetOperationStmt *op, PlannerInfo *root, /* Undo effects of possibly forcing tuple_fraction to 0 */ root->tuple_fraction = save_fraction; + /* Add the UniqueKeys */ + populate_unionrel_uniquekeys(root, result_rel); return result_rel; } diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c index d722063cf3..44c37ecffc 100644 --- a/src/backend/optimizer/util/appendinfo.c +++ b/src/backend/optimizer/util/appendinfo.c @@ -746,3 +746,47 @@ find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos) } return appinfos; } + +/* + * find_appinfo_by_child + * + */ +AppendRelInfo * +find_appinfo_by_child(PlannerInfo *root, Index child_index) +{ + ListCell *lc; + foreach(lc, root->append_rel_list) + { + AppendRelInfo *appinfo = lfirst_node(AppendRelInfo, lc); + if (appinfo->child_relid == child_index) + return appinfo; + } + elog(ERROR, "parent relation cant be found"); + return NULL; +} + +/* + * find_parent_var + * + */ +Var * +find_parent_var(AppendRelInfo *appinfo, Var *child_var) +{ + ListCell *lc; + Var *res = NULL; + Index attno = 1; + foreach(lc, appinfo->translated_vars) + { + Node *child_node = lfirst(lc); + if (equal(child_node, child_var)) + { + res = copyObject(child_var); + res->varattno = attno; + res->varno = appinfo->parent_relid; + } + attno++; + } + if (res == NULL) + elog(ERROR, "parent var can't be found."); + return res; +} diff --git a/src/backend/optimizer/util/inherit.c b/src/backend/optimizer/util/inherit.c index 3132fd35a5..d66b40ec50 100644 --- a/src/backend/optimizer/util/inherit.c +++ b/src/backend/optimizer/util/inherit.c @@ -736,6 +736,7 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, { Node *onecq = (Node *) lfirst(lc2); bool pseudoconstant; + RestrictInfo *child_rinfo; /* check for pseudoconstant (no Vars or volatile functions) */ pseudoconstant = @@ -747,13 +748,14 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, root->hasPseudoConstantQuals = true; } /* reconstitute RestrictInfo with appropriate properties */ - childquals = lappend(childquals, - make_restrictinfo((Expr *) onecq, - rinfo->is_pushed_down, - rinfo->outerjoin_delayed, - pseudoconstant, - rinfo->security_level, - NULL, NULL, NULL)); + child_rinfo = make_restrictinfo((Expr *) onecq, + rinfo->is_pushed_down, + rinfo->outerjoin_delayed, + pseudoconstant, + rinfo->security_level, + NULL, NULL, NULL); + child_rinfo->mergeopfamilies = rinfo->mergeopfamilies; + childquals = lappend(childquals, child_rinfo); /* track minimum security level among child quals */ cq_min_security = Min(cq_min_security, rinfo->security_level); } diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 31d9aedeeb..c83f17acb7 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -16,6 +16,7 @@ #include "nodes/execnodes.h" #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" extern A_Expr *makeA_Expr(A_Expr_Kind kind, List *name, @@ -105,4 +106,6 @@ extern GroupingSet *makeGroupingSet(GroupingSetKind kind, List *content, int loc extern VacuumRelation *makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols); +extern UniqueKey* makeUniqueKey(List *exprs, bool multi_nullvals); + #endif /* MAKEFUNC_H */ diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 381d84b4e4..41110ed888 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -264,6 +264,7 @@ typedef enum NodeTag T_EquivalenceMember, T_PathKey, T_PathTarget, + T_UniqueKey, T_RestrictInfo, T_IndexClause, T_PlaceHolderVar, diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index 9e3ebd488a..02e4458bef 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -730,6 +730,7 @@ typedef struct RelOptInfo QualCost baserestrictcost; /* cost of evaluating the above */ Index baserestrict_min_security; /* min security_level found in * baserestrictinfo */ + List *uniquekeys; /* List of UniqueKey */ List *joininfo; /* RestrictInfo structures for join clauses * involving this rel */ bool has_eclass_joins; /* T means joininfo is incomplete */ @@ -1047,6 +1048,28 @@ typedef struct PathKey } PathKey; +/* + * UniqueKey + * + * Represents the unique properties held by a RelOptInfo. + * + * exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL + * is a special case of UniqueKey, which means there is only 1 row in that + * relation. + * multi_nullvals: true means multi null values may exist in these exprs, so the + * uniqueness is not guaranteed in this case. This field is necessary for + * remove_useless_join & reduce_unique_semijoins where we don't mind these + * duplicated NULL values. It is set to true for 2 cases. One is a unique key + * from a unique index but the related column is nullable. The other one is for + * outer join. see populate_joinrel_uniquekeys for detail. + */ +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + /* * PathTarget * @@ -2473,7 +2496,7 @@ typedef enum * * flags indicating what kinds of grouping are possible. * partial_costs_set is true if the agg_partial_costs and agg_final_costs - * have been initialized. + * have been initialized. * agg_partial_costs gives partial aggregation costs. * agg_final_costs gives finalization costs. * target_parallel_safe is true if target is parallel safe. @@ -2503,8 +2526,8 @@ typedef struct * limit_tuples is an estimated bound on the number of output tuples, * or -1 if no LIMIT or couldn't estimate. * count_est and offset_est are the estimated values of the LIMIT and OFFSET - * expressions computed by preprocess_limit() (see comments for - * preprocess_limit() for more information). + * expressions computed by preprocess_limit() (see comments for + * preprocess_limit() for more information). */ typedef struct { diff --git a/src/include/nodes/pg_list.h b/src/include/nodes/pg_list.h index 14ea2766ad..621f54a9f8 100644 --- a/src/include/nodes/pg_list.h +++ b/src/include/nodes/pg_list.h @@ -528,6 +528,8 @@ extern bool list_member_ptr(const List *list, const void *datum); extern bool list_member_int(const List *list, int datum); extern bool list_member_oid(const List *list, Oid datum); +extern bool list_is_subset(const List *members, const List *target); + extern List *list_delete(List *list, void *datum); extern List *list_delete_ptr(List *list, void *datum); extern List *list_delete_int(List *list, int datum); diff --git a/src/include/optimizer/appendinfo.h b/src/include/optimizer/appendinfo.h index d6a27a60dd..e87c92a054 100644 --- a/src/include/optimizer/appendinfo.h +++ b/src/include/optimizer/appendinfo.h @@ -32,4 +32,7 @@ extern Relids adjust_child_relids_multilevel(PlannerInfo *root, Relids relids, extern AppendRelInfo **find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos); +extern AppendRelInfo *find_appinfo_by_child(PlannerInfo *root, Index child_index); +extern Var *find_parent_var(AppendRelInfo *appinfo, Var *child_var); + #endif /* APPENDINFO_H */ diff --git a/src/include/optimizer/optimizer.h b/src/include/optimizer/optimizer.h index 3e4171056e..9445141263 100644 --- a/src/include/optimizer/optimizer.h +++ b/src/include/optimizer/optimizer.h @@ -23,6 +23,7 @@ #define OPTIMIZER_H #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" /* * We don't want to include nodes/pathnodes.h here, because non-planner @@ -156,6 +157,7 @@ extern TargetEntry *get_sortgroupref_tle(Index sortref, List *targetList); extern TargetEntry *get_sortgroupclause_tle(SortGroupClause *sgClause, List *targetList); +extern Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); extern Node *get_sortgroupclause_expr(SortGroupClause *sgClause, List *targetList); extern List *get_sortgrouplist_exprs(List *sgClauses, diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h index 10b6e81079..9217a8d6c6 100644 --- a/src/include/optimizer/paths.h +++ b/src/include/optimizer/paths.h @@ -240,5 +240,48 @@ extern PathKey *make_canonical_pathkey(PlannerInfo *root, int strategy, bool nulls_first); extern void add_paths_to_append_rel(PlannerInfo *root, RelOptInfo *rel, List *live_childrels); +extern List *select_mergejoin_clauses(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + JoinType jointype, + bool *mergejoin_allowed); + +/* + * uniquekeys.c + * Utilities for matching and building unique keys + */ +extern void populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List* unique_index_list); +extern void populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels); +extern void populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel); +extern void populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel); +extern void populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel); +extern void simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel); +extern void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel); +extern void populate_joinrel_uniquekeys(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + RelOptInfo *rel2, + List *restrictlist, + JoinType jointype); + +extern bool relation_has_uniquekeys_for(PlannerInfo *root, + RelOptInfo *rel, + List *exprs, + bool allow_multinulls); +extern bool relation_is_onerow(RelOptInfo *rel); #endif /* PATHS_H */ -- 2.21.0 --op7mf2i72xk5yu7c Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v36-0003-Extend-UniqueKeys.patch" ^ permalink raw reply [nested|flat] 91+ messages in thread
* [PATCH v38 2/6] Introduce UniqueKey attributes on RelOptInfo struct. @ 2020-05-11 07:50 一挃 <[email protected]> 0 siblings, 0 replies; 91+ messages in thread From: 一挃 @ 2020-05-11 07:50 UTC (permalink / raw) UniqueKey is a set of exprs on RelOptInfo which represents the exprs will be unique on the given RelOptInfo. You can see README.uniquekey for more information. --- src/backend/nodes/copyfuncs.c | 13 + src/backend/nodes/list.c | 31 + src/backend/nodes/makefuncs.c | 13 + src/backend/nodes/outfuncs.c | 11 + src/backend/nodes/readfuncs.c | 10 + src/backend/optimizer/path/Makefile | 3 +- src/backend/optimizer/path/README.uniquekey | 131 +++ src/backend/optimizer/path/allpaths.c | 10 + src/backend/optimizer/path/joinpath.c | 9 +- src/backend/optimizer/path/joinrels.c | 2 + src/backend/optimizer/path/pathkeys.c | 3 +- src/backend/optimizer/path/uniquekeys.c | 1135 +++++++++++++++++++ src/backend/optimizer/plan/planner.c | 13 +- src/backend/optimizer/prep/prepunion.c | 2 + src/backend/optimizer/util/appendinfo.c | 44 + src/backend/optimizer/util/inherit.c | 18 +- src/include/nodes/makefuncs.h | 3 + src/include/nodes/nodes.h | 1 + src/include/nodes/pathnodes.h | 29 +- src/include/nodes/pg_list.h | 10 + src/include/optimizer/appendinfo.h | 3 + src/include/optimizer/optimizer.h | 2 + src/include/optimizer/paths.h | 43 + 23 files changed, 1515 insertions(+), 24 deletions(-) create mode 100644 src/backend/optimizer/path/README.uniquekey create mode 100644 src/backend/optimizer/path/uniquekeys.c diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..75c1c5e824 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2296,6 +2296,16 @@ _copyPathKey(const PathKey *from) return newnode; } +static UniqueKey * +_copyUniqueKey(const UniqueKey *from) +{ + UniqueKey *newnode = makeNode(UniqueKey); + + COPY_NODE_FIELD(exprs); + COPY_SCALAR_FIELD(multi_nullvals); + + return newnode; +} /* * _copyRestrictInfo */ @@ -5220,6 +5230,9 @@ copyObjectImpl(const void *from) case T_PathKey: retval = _copyPathKey(from); break; + case T_UniqueKey: + retval = _copyUniqueKey(from); + break; case T_RestrictInfo: retval = _copyRestrictInfo(from); break; diff --git a/src/backend/nodes/list.c b/src/backend/nodes/list.c index dbf6b30233..ca099495a1 100644 --- a/src/backend/nodes/list.c +++ b/src/backend/nodes/list.c @@ -702,6 +702,37 @@ list_member_oid(const List *list, Oid datum) return false; } +/* + * return true iff every entry in "members" list is also present + * in the "target" list. + */ +bool +list_is_subset(const List *members, const List *target) +{ + const ListCell *lc1, *lc2; + + Assert(IsPointerList(members)); + Assert(IsPointerList(target)); + check_list_invariants(members); + check_list_invariants(target); + + foreach(lc1, members) + { + bool found = false; + foreach(lc2, target) + { + if (equal(lfirst(lc1), lfirst(lc2))) + { + found = true; + break; + } + } + if (!found) + return false; + } + return true; +} + /* * Delete the n'th cell (counting from 0) in list. * diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 01c110cd2f..40415d0f5b 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -815,3 +815,16 @@ makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols) v->va_cols = va_cols; return v; } + + +/* + * makeUniqueKey + */ +UniqueKey* +makeUniqueKey(List *exprs, bool multi_nullvals) +{ + UniqueKey * ukey = makeNode(UniqueKey); + ukey->exprs = exprs; + ukey->multi_nullvals = multi_nullvals; + return ukey; +} diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..44154cde6a 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2456,6 +2456,14 @@ _outPathKey(StringInfo str, const PathKey *node) WRITE_BOOL_FIELD(pk_nulls_first); } +static void +_outUniqueKey(StringInfo str, const UniqueKey *node) +{ + WRITE_NODE_TYPE("UNIQUEKEY"); + WRITE_NODE_FIELD(exprs); + WRITE_BOOL_FIELD(multi_nullvals); +} + static void _outPathTarget(StringInfo str, const PathTarget *node) { @@ -4198,6 +4206,9 @@ outNode(StringInfo str, const void *obj) case T_PathKey: _outPathKey(str, obj); break; + case T_UniqueKey: + _outUniqueKey(str, obj); + break; case T_PathTarget: _outPathTarget(str, obj); break; diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index c5e136e9c3..b3e212bf1c 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -492,6 +492,14 @@ _readSetOperationStmt(void) READ_DONE(); } +static UniqueKey * +_readUniqueKey(void) +{ + READ_LOCALS(UniqueKey); + READ_NODE_FIELD(exprs); + READ_BOOL_FIELD(multi_nullvals); + READ_DONE(); +} /* * Stuff from primnodes.h. @@ -2717,6 +2725,8 @@ parseNodeString(void) return_value = _readCommonTableExpr(); else if (MATCH("SETOPERATIONSTMT", 16)) return_value = _readSetOperationStmt(); + else if (MATCH("UNIQUEKEY", 9)) + return_value = _readUniqueKey(); else if (MATCH("ALIAS", 5)) return_value = _readAlias(); else if (MATCH("RANGEVAR", 8)) diff --git a/src/backend/optimizer/path/Makefile b/src/backend/optimizer/path/Makefile index 1e199ff66f..7b9820c25f 100644 --- a/src/backend/optimizer/path/Makefile +++ b/src/backend/optimizer/path/Makefile @@ -21,6 +21,7 @@ OBJS = \ joinpath.o \ joinrels.o \ pathkeys.o \ - tidpath.o + tidpath.o \ + uniquekeys.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/optimizer/path/README.uniquekey b/src/backend/optimizer/path/README.uniquekey new file mode 100644 index 0000000000..5eac761995 --- /dev/null +++ b/src/backend/optimizer/path/README.uniquekey @@ -0,0 +1,131 @@ +1. What is UniqueKey? +We can think UniqueKey is a set of exprs for a RelOptInfo, which we are insure +that doesn't yields same result among all the rows. The simplest UniqueKey +format is primary key. + +However we define the UnqiueKey as below. + +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + +exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL +is a special case of UniqueKey, which means there is only one row in that +relation.it has a stronger semantic than others. like SELECT uk FROM t; uk is +normal unique key and may have different values. SELECT colx FROM t WHERE uk = +const. colx is unique AND we have only 1 value. This field can used for +innerrel_is_unique. this logic is handled specially in add_uniquekey_for_onerow +function. + +multi_nullvals: true means multi null values may exist in these exprs, so the +uniqueness is not guaranteed in this case. This field is necessary for +remove_useless_join & reduce_unique_semijoins where we don't mind these +duplicated NULL values. It is set to true for 2 cases. One is a unique key +from a unique index but the related column is nullable. The other one is for +outer join. see populate_joinrel_uniquekeys for detail. + + +The UniqueKey can be used at the following cases at least: +1. remove_useless_joins. +2. reduce_semianti_joins +3. remove distinct node if distinct clause is unique. +4. remove aggnode if group by clause is unique. +5. Index Skip Scan (WIP) +6. Aggregation Push Down without 2 phase aggregation if the join can't + duplicated the aggregated rows. (work in progress feature) + +2. How is it maintained? + +We have a set of populate_xxx_unqiuekeys functions to maintain the uniquekey on +various cases. xxx includes baserel, joinrel, partitionedrel, distinctrel, +groupedrel, unionrel. and we also need to convert the uniquekey from subquery +to outer relation, which is what convert_subquery_uniquekeys does. + +1. The first part is about baserel. We handled 3 cases. suppose we have Unique +Index on (a, b). + +1. SELECT a, b FROM t. UniqueKey (a, b) +2. SELECT a FROM t WHERE b = 1; UniqueKey (a) +3. SELECT .. FROM t WHERE a = 1 AND b = 1; UniqueKey (NIL). onerow case, every + column is Unique. + +2. The next part is joinrel, this part is most error-prone, we simplified the rules +like below: +1. If the relation's UniqueKey can't be duplicated after join, then is will be + still valid for the join rel. The function we used here is + innerrel_keeps_unique. The basic idea is innerrel.any_col = outer.uk. + +2. If the UnqiueKey can't keep valid via the rule 1, the combination of the + UniqueKey from both sides are valid for sure. We can prove this as: if the + unique exprs from rel1 is duplicated by rel2, the duplicated rows must + contains different unique exprs from rel2. + +More considerations about onerow: +1. If relation with one row and it can't be duplicated, it is still possible + contains mulit_nullvas after outer join. +2. If the either UniqueKey can be duplicated after join, the can get one row + only when both side is one row AND there is no outer join. +3. Whenever the onerow UniqueKey is not a valid any more, we need to convert one + row UniqueKey to normal unique key since we don't store exprs for one-row + relation. get_exprs_from_uniquekeys will be used here. + + +More considerations about multi_nullvals after join: +1. If the original UnqiueKey has multi_nullvals, the final UniqueKey will have + mulit_nullvals in any case. +2. If a unique key doesn't allow mulit_nullvals, after some outer join, it + allows some outer join. + + +3. When we comes to subquery, we need to convert_subquery_unqiuekeys just like +convert_subquery_pathkeys. Only the UniqueKey insides subquery is referenced as +a Var in outer relation will be reused. The relationship between the outerrel.Var +and subquery.exprs is built with outerel->subroot->processed_tlist. + + +4. As for the SRF functions, it will break the uniqueness of uniquekey, However it +is handled in adjust_paths_for_srfs, which happens after the query_planner. so +we will maintain the UniqueKey until there and reset it to NIL at that +places. This can't help on distinct/group by elimination cases but probably help +in some other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is +semantic correctly. + + +5. As for inherit table, we first main the UnqiueKey on childrel as well. But for +partitioned table we need to maintain 2 different kinds of +UnqiueKey. 1). UniqueKey on the parent relation 2). UniqueKey on child +relation for partition wise query. + +Example: +CREATE TABLE p (a int not null, b int not null) partition by list (a); +CREATE TABLE p0 partition of p for values in (1); +CREATE TABLE p1 partition of p for values in (2); + +create unique index p0_b on p0(b); +create unique index p1_b on p1(b); + +Now b is only unique on partition level, so the distinct can't be removed on +the following cases. SELECT DISTINCT b FROM p; + +Another example is SELECT DISTINCT a, b FROM p WHERE a = 1; Since only one +partition is chosen, the UniqueKey on child relation is same as the UniqueKey on +parent relation. + +Another usage of UniqueKey on partition level is it be helpful for +partition-wise join. + +As for the UniqueKey on parent table level, it comes with 2 different ways, +1). the UniqueKey is also derived in UniqueKey index, but the index must be same +in all the related children relations and the unique index must contains +Partition Key in it. Example: + +CREATE UNIQUE INDEX p_ab ON p(a, b); -- where a is the partition key. + +-- Query +SELECT a, b FROM p; the (a, b) is a UniqueKey of p. + +2). If the parent relation has only one childrel, the UniqueKey on childrel is + the UniqueKey on parent as well. diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index 37b4223adb..d52ad59f89 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -579,6 +579,12 @@ set_plain_rel_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) */ check_index_predicates(root, rel); + /* + * Now that we've marked which partial indexes are suitable, we can now + * build the relation's unique keys. + */ + populate_baserel_uniquekeys(root, rel, rel->indexlist); + /* Mark rel with estimated output rows, width, etc */ set_baserel_size_estimates(root, rel); } @@ -1297,6 +1303,8 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, /* Add paths to the append relation. */ add_paths_to_append_rel(root, rel, live_childrels); + if (IS_PARTITIONED_REL(rel)) + populate_partitionedrel_uniquekeys(root, rel, live_childrels); } @@ -2304,6 +2312,8 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, pathkeys, required_outer)); } + convert_subquery_uniquekeys(root, rel, sub_final_rel); + /* If outer rel allows parallelism, do same for partial paths. */ if (rel->consider_parallel && bms_is_empty(required_outer)) { diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c index 57ce97fd53..697ac047fb 100644 --- a/src/backend/optimizer/path/joinpath.c +++ b/src/backend/optimizer/path/joinpath.c @@ -71,13 +71,6 @@ static void consider_parallel_mergejoin(PlannerInfo *root, static void hash_inner_and_outer(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, RelOptInfo *innerrel, JoinType jointype, JoinPathExtraData *extra); -static List *select_mergejoin_clauses(PlannerInfo *root, - RelOptInfo *joinrel, - RelOptInfo *outerrel, - RelOptInfo *innerrel, - List *restrictlist, - JoinType jointype, - bool *mergejoin_allowed); static void generate_mergejoin_paths(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *innerrel, @@ -1927,7 +1920,7 @@ hash_inner_and_outer(PlannerInfo *root, * if it is mergejoinable and involves vars from the two sub-relations * currently of interest. */ -static List * +List * select_mergejoin_clauses(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c index 0dbe2ac726..7271f044ec 100644 --- a/src/backend/optimizer/path/joinrels.c +++ b/src/backend/optimizer/path/joinrels.c @@ -924,6 +924,8 @@ populate_joinrel_with_paths(PlannerInfo *root, RelOptInfo *rel1, /* Apply partitionwise join technique, if possible. */ try_partitionwise_join(root, rel1, rel2, joinrel, sjinfo, restrictlist); + + populate_joinrel_uniquekeys(root, joinrel, rel1, rel2, restrictlist, sjinfo->jointype); } diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index bd9a176d7d..139278829b 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -33,7 +33,6 @@ static bool pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys); static bool matches_boolean_partition_clause(RestrictInfo *rinfo, RelOptInfo *partrel, int partkeycol); -static Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); static bool right_merge_direction(PlannerInfo *root, PathKey *pathkey); @@ -1035,7 +1034,7 @@ convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel, * We need this to ensure that we don't return pathkeys describing values * that are unavailable above the level of the subquery scan. */ -static Var * +Var * find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle) { ListCell *lc; diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c new file mode 100644 index 0000000000..77ed2b2eff --- /dev/null +++ b/src/backend/optimizer/path/uniquekeys.c @@ -0,0 +1,1135 @@ +/*------------------------------------------------------------------------- + * + * uniquekeys.c + * Utilities for matching and building unique keys + * + * Portions Copyright (c) 2020, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/optimizer/path/uniquekeys.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "nodes/makefuncs.h" +#include "nodes/nodeFuncs.h" +#include "optimizer/pathnode.h" +#include "optimizer/paths.h" +#include "optimizer/appendinfo.h" +#include "optimizer/optimizer.h" +#include "optimizer/tlist.h" +#include "rewrite/rewriteManip.h" + + +/* + * This struct is used to help populate_joinrel_uniquekeys. + * + * added_to_joinrel is true if a uniquekey (from outerrel or innerrel) + * has been added to joinrel. + * useful is true if the exprs of the uniquekey still appears in joinrel. + */ +typedef struct UniqueKeyContextData +{ + UniqueKey *uniquekey; + bool added_to_joinrel; + bool useful; +} *UniqueKeyContext; + +static List *initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel); +static bool innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + bool reverse); + +static List *get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals); +static List *get_exprs_from_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + UniqueKey *ukey); +static void add_uniquekey_for_onerow(RelOptInfo *rel); +static bool add_combined_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype); + +/* Used for unique indexes checking for partitioned table */ +static bool index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind); +static IndexOptInfo *simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from); +static bool simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2); +static void adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_index); + +/* Helper function for grouped relation and distinct relation. */ +static void add_uniquekey_from_sortgroups(PlannerInfo *root, + RelOptInfo *rel, + List *sortgroups); + +/* + * populate_baserel_uniquekeys + * Populate 'baserel' uniquekeys list by looking at the rel's unique index + * and baserestrictinfo + */ +void +populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List *indexlist) +{ + ListCell *lc; + List *matched_uniq_indexes = NIL; + + /* Attrs appears in rel->reltarget->exprs. */ + Bitmapset *used_attrs = NULL; + + List *const_exprs = NIL; + List *expr_opfamilies = NIL; + + Assert(baserel->rtekind == RTE_RELATION); + + foreach(lc, indexlist) + { + IndexOptInfo *ind = (IndexOptInfo *) lfirst(lc); + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + matched_uniq_indexes = lappend(matched_uniq_indexes, ind); + } + + if (matched_uniq_indexes == NIL) + return; + + /* Check which attrs is used in baserel->reltarget */ + pull_varattnos((Node *)baserel->reltarget->exprs, baserel->relid, &used_attrs); + + /* Check which attrno is used at a mergeable const filter */ + foreach(lc, baserel->baserestrictinfo) + { + RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); + + if (rinfo->mergeopfamilies == NIL) + continue; + + if (bms_is_empty(rinfo->left_relids)) + { + const_exprs = lappend(const_exprs, get_rightop(rinfo->clause)); + } + else if (bms_is_empty(rinfo->right_relids)) + { + const_exprs = lappend(const_exprs, get_leftop(rinfo->clause)); + } + else + continue; + + expr_opfamilies = lappend(expr_opfamilies, rinfo->mergeopfamilies); + } + + foreach(lc, matched_uniq_indexes) + { + bool multi_nullvals, useful; + List *exprs = get_exprs_from_uniqueindex(lfirst_node(IndexOptInfo, lc), + const_exprs, + expr_opfamilies, + used_attrs, + &useful, + &multi_nullvals); + if (useful) + { + if (exprs == NIL) + { + /* All the columns in Unique Index matched with a restrictinfo */ + add_uniquekey_for_onerow(baserel); + return; + } + baserel->uniquekeys = lappend(baserel->uniquekeys, + makeUniqueKey(exprs, multi_nullvals)); + } + } +} + + +/* + * populate_partitionedrel_uniquekeys + * The UniqueKey on partitionrel comes from 2 cases: + * 1). Only one partition is involved in this query, the unique key can be + * copied to parent rel from childrel. + * 2). There are some unique index which includes partition key and exists + * in all the related partitions. + * We never mind rule 2 if we hit rule 1. + */ + +void +populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels) +{ + ListCell *lc; + List *global_uniq_indexlist = NIL; + RelOptInfo *childrel; + bool is_first = true; + + Assert(IS_PARTITIONED_REL(rel)); + + if (childrels == NIL) + return; + + /* + * If there is only one partition used in this query, the UniqueKey in childrel is + * still valid in parent level, but we need convert the format from child expr to + * parent expr. + */ + if (list_length(childrels) == 1) + { + /* Check for Rule 1 */ + RelOptInfo *childrel = linitial_node(RelOptInfo, childrels); + ListCell *lc; + Assert(childrel->reloptkind == RELOPT_OTHER_MEMBER_REL); + if (relation_is_onerow(childrel)) + { + add_uniquekey_for_onerow(rel); + return; + } + + foreach(lc, childrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + AppendRelInfo *appinfo = find_appinfo_by_child(root, childrel->relid); + List *parent_exprs = NIL; + bool can_reuse = true; + ListCell *lc2; + foreach(lc2, ukey->exprs) + { + Var *var = (Var *)lfirst(lc2); + /* + * If the expr comes from a expression, it is hard to build the expression + * in parent so ignore that case for now. + */ + if(!IsA(var, Var)) + { + can_reuse = false; + break; + } + /* Convert it to parent var */ + parent_exprs = lappend(parent_exprs, find_parent_var(appinfo, var)); + } + if (can_reuse) + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(parent_exprs, + ukey->multi_nullvals)); + } + } + else + { + /* Check for rule 2 */ + childrel = linitial_node(RelOptInfo, childrels); + foreach(lc, childrel->indexlist) + { + IndexOptInfo *ind = lfirst(lc); + IndexOptInfo *modified_index; + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + + /* + * During simple_copy_indexinfo_to_parent, we need to convert var from + * child var to parent var, index on expression is too complex to handle. + * so ignore it for now. + */ + if (ind->indexprs != NIL) + continue; + + modified_index = simple_copy_indexinfo_to_parent(root, rel, ind); + /* + * If the unique index doesn't contain partkey, then it is unique + * on this partition only, so it is useless for us. + */ + if (!index_constains_partkey(rel, modified_index)) + continue; + + global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index); + } + + if (global_uniq_indexlist != NIL) + { + foreach(lc, childrels) + { + RelOptInfo *child = lfirst(lc); + if (is_first) + { + is_first = false; + continue; + } + adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist); + } + /* Now we have a list of unique index which are exactly same on all childrels, + * Set the UniqueKey just like it is non-partition table + */ + populate_baserel_uniquekeys(root, rel, global_uniq_indexlist); + } + } +} + + +/* + * populate_distinctrel_uniquekeys + */ +void +populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel) +{ + /* The unique key before the distinct is still valid. */ + distinctrel->uniquekeys = list_copy(inputrel->uniquekeys); + add_uniquekey_from_sortgroups(root, distinctrel, root->parse->distinctClause); +} + +/* + * populate_grouprel_uniquekeys + */ +void +populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel) + +{ + Query *parse = root->parse; + bool input_ukey_added = false; + ListCell *lc; + + if (relation_is_onerow(inputrel)) + { + add_uniquekey_for_onerow(grouprel); + return; + } + if (parse->groupingSets) + return; + + /* A Normal group by without grouping set. */ + if (parse->groupClause) + { + /* + * Current even the groupby clause is Unique already, but if query has aggref + * We have to create grouprel still. To keep the UnqiueKey short, we will check + * the UniqueKey of input_rel still valid, if so we reuse it. + */ + foreach(lc, inputrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (list_is_subset(ukey->exprs, grouprel->reltarget->exprs)) + { + grouprel->uniquekeys = lappend(grouprel->uniquekeys, + ukey); + input_ukey_added = true; + } + } + if (!input_ukey_added) + /* + * group by clause must be a super-set of grouprel->reltarget->exprs except the + * aggregation expr, so if such exprs is unique already, no bother to generate + * new uniquekey for group by exprs. + */ + add_uniquekey_from_sortgroups(root, + grouprel, + root->parse->groupClause); + } + else + /* It has aggregation but without a group by, so only one row returned */ + add_uniquekey_for_onerow(grouprel); +} + +/* + * simple_copy_uniquekeys + * Using a function for the one-line code makes us easy to check where we simply + * copied the uniquekey. + */ +void +simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel) +{ + newrel->uniquekeys = oldrel->uniquekeys; +} + +/* + * populate_unionrel_uniquekeys + */ +void +populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel) +{ + ListCell *lc; + List *exprs = NIL; + + Assert(unionrel->uniquekeys == NIL); + + foreach(lc, unionrel->reltarget->exprs) + { + exprs = lappend(exprs, lfirst(lc)); + } + + if (exprs == NIL) + /* SQL: select union select; is valid, we need to handle it here. */ + add_uniquekey_for_onerow(unionrel); + else + unionrel->uniquekeys = lappend(unionrel->uniquekeys, + makeUniqueKey(exprs,false)); + +} + +/* + * populate_joinrel_uniquekeys + * + * populate uniquekeys for joinrel. We will check each relation to see if its + * UniqueKey is still valid via innerrel_keeps_unique, if so, we add it to + * joinrel. The multi_nullvals field will be changed to true for some outer + * join cases and one-row UniqueKey needs to be converted to normal UniqueKey + * for the same case as well. + * For the uniquekey in either baserel which can't be unique after join, we still + * check to see if combination of UniqueKeys from both side is still useful for us. + * if yes, we add it to joinrel as well. + */ +void +populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *outerrel, RelOptInfo *innerrel, + List *restrictlist, JoinType jointype) +{ + ListCell *lc, *lc2; + List *clause_list = NIL; + List *outerrel_ukey_ctx; + List *innerrel_ukey_ctx; + bool inner_onerow, outer_onerow; + bool mergejoin_allowed; + + /* Care about the outerrel relation only for SEMI/ANTI join */ + if (jointype == JOIN_SEMI || jointype == JOIN_ANTI) + { + foreach(lc, outerrel->uniquekeys) + { + UniqueKey *uniquekey = lfirst_node(UniqueKey, lc); + if (list_is_subset(uniquekey->exprs, joinrel->reltarget->exprs)) + joinrel->uniquekeys = lappend(joinrel->uniquekeys, uniquekey); + } + return; + } + + Assert(jointype == JOIN_LEFT || jointype == JOIN_FULL || jointype == JOIN_INNER); + + /* Fast path */ + if (innerrel->uniquekeys == NIL || outerrel->uniquekeys == NIL) + return; + + inner_onerow = relation_is_onerow(innerrel); + outer_onerow = relation_is_onerow(outerrel); + + outerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(outerrel); + innerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(innerrel); + + clause_list = select_mergejoin_clauses(root, joinrel, outerrel, innerrel, + restrictlist, jointype, + &mergejoin_allowed); + + if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true /* reverse */)) + { + bool outer_impact = jointype == JOIN_FULL; + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + /* Outer relation has one row, and the unique key is not duplicated after join, + * the joinrel will still has one row unless the jointype == JOIN_FULL. + */ + if (outer_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (outer_onerow) + { + /* + * The onerow outerrel becomes multi rows and multi_nullvals + * will be changed to true. We also need to set the exprs correctly since it + * can't be NIL any more. + */ + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, outerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Change multi_nullvals to true due to the full join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, true)); + else + /* Just reuse it */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + } + ctx->added_to_joinrel = true; + } + } + + if (innerrel_keeps_unique(root, outerrel, innerrel, clause_list, false)) + { + bool outer_impact = jointype == JOIN_FULL || jointype == JOIN_LEFT;; + + foreach(lc, innerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + if (inner_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (inner_onerow) + { + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, innerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Need to change multi_nullvals to true due to the outer join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, + true)); + else + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + + } + ctx->added_to_joinrel = true; + } + } + + /* + * The combination of the UniqueKey from both sides is unique as well regardless + * of join type, but no bother to add it if its subset has been added to joinrel + * already or it is not useful for the joinrel. + */ + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx1 = (UniqueKeyContext) lfirst(lc); + if (ctx1->added_to_joinrel || !ctx1->useful) + continue; + foreach(lc2, innerrel_ukey_ctx) + { + UniqueKeyContext ctx2 = (UniqueKeyContext) lfirst(lc2); + if (ctx2->added_to_joinrel || !ctx2->useful) + continue; + if (add_combined_uniquekey(root, joinrel, outerrel, innerrel, + ctx1->uniquekey, ctx2->uniquekey, + jointype)) + /* If we set a onerow UniqueKey to joinrel, we don't need other. */ + return; + } + } +} + + +/* + * convert_subquery_uniquekeys + * + * Covert the UniqueKey in subquery to outer relation. + */ +void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel) +{ + ListCell *lc; + + if (sub_final_rel->uniquekeys == NIL) + return; + + if (relation_is_onerow(sub_final_rel)) + { + add_uniquekey_for_onerow(currel); + return; + } + + Assert(currel->subroot != NULL); + + foreach(lc, sub_final_rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + ListCell *lc; + List *exprs = NIL; + bool ukey_useful = true; + + /* One row case is handled above */ + Assert(ukey->exprs != NIL); + foreach(lc, ukey->exprs) + { + Var *var; + TargetEntry *tle = tlist_member(lfirst(lc), + currel->subroot->processed_tlist); + if (tle == NULL) + { + ukey_useful = false; + break; + } + var = find_var_for_subquery_tle(currel, tle); + if (var == NULL) + { + ukey_useful = false; + break; + } + exprs = lappend(exprs, var); + } + + if (ukey_useful) + currel->uniquekeys = lappend(currel->uniquekeys, + makeUniqueKey(exprs, + ukey->multi_nullvals)); + + } +} + +/* + * innerrel_keeps_unique + * + * Check if Unique key of the innerrel is valid after join. innerrel's UniqueKey + * will be still valid if innerrel's any-column mergeop outrerel's uniquekey + * exists in clause_list. + * + * Note: the clause_list must be a list of mergeable restrictinfo already. + */ +static bool +innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *clause_list, + bool reverse) +{ + ListCell *lc, *lc2, *lc3; + + if (outerrel->uniquekeys == NIL || innerrel->uniquekeys == NIL) + return false; + + /* Check if there is outerrel's uniquekey in mergeable clause. */ + foreach(lc, outerrel->uniquekeys) + { + List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs; + bool clauselist_matchs_all_exprs = true; + foreach(lc2, outer_uq_exprs) + { + Node *outer_uq_expr = lfirst(lc2); + bool find_uq_expr_in_clauselist = false; + foreach(lc3, clause_list) + { + RestrictInfo *rinfo = lfirst_node(RestrictInfo, lc3); + Node *outer_expr; + if (reverse) + outer_expr = rinfo->outer_is_left ? get_rightop(rinfo->clause) : get_leftop(rinfo->clause); + else + outer_expr = rinfo->outer_is_left ? get_leftop(rinfo->clause) : get_rightop(rinfo->clause); + if (equal(outer_expr, outer_uq_expr)) + { + find_uq_expr_in_clauselist = true; + break; + } + } + if (!find_uq_expr_in_clauselist) + { + /* No need to check the next exprs in the current uniquekey */ + clauselist_matchs_all_exprs = false; + break; + } + } + + if (clauselist_matchs_all_exprs) + return true; + } + return false; +} + + +/* + * relation_is_onerow + * Check if it is a one-row relation by checking UniqueKey. + */ +bool +relation_is_onerow(RelOptInfo *rel) +{ + UniqueKey *ukey; + if (rel->uniquekeys == NIL) + return false; + ukey = linitial_node(UniqueKey, rel->uniquekeys); + return ukey->exprs == NIL && list_length(rel->uniquekeys) == 1; +} + +/* + * relation_has_uniquekeys_for + * Returns true if we have proofs that 'rel' cannot return multiple rows with + * the same values in each of 'exprs'. Otherwise returns false. + */ +bool +relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel, + List *exprs, bool allow_multinulls) +{ + ListCell *lc; + + /* + * For UniqueKey->onerow case, the uniquekey->exprs is empty as well + * so we can't rely on list_is_subset to handle this special cases + */ + if (exprs == NIL) + return false; + + foreach(lc, rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (ukey->multi_nullvals && !allow_multinulls) + continue; + if (list_is_subset(ukey->exprs, exprs)) + return true; + } + return false; +} + + +/* + * get_exprs_from_uniqueindex + * + * Return a list of exprs which is unique. set useful to false if this + * unique index is not useful for us. + */ +static List * +get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals) +{ + List *exprs = NIL; + ListCell *indexpr_item; + int c = 0; + + *useful = true; + *multi_nullvals = false; + + indexpr_item = list_head(unique_index->indexprs); + for(c = 0; c < unique_index->ncolumns; c++) + { + int attr = unique_index->indexkeys[c]; + Expr *expr; + bool matched_const = false; + ListCell *lc1, *lc2; + + if(attr > 0) + { + expr = list_nth_node(TargetEntry, unique_index->indextlist, c)->expr; + } + else if (attr == 0) + { + /* Expression index */ + expr = lfirst(indexpr_item); + indexpr_item = lnext(unique_index->indexprs, indexpr_item); + } + else /* attr < 0 */ + { + /* Index on system column is not supported */ + Assert(false); + } + + /* + * Check index_col = Const case with regarding to opfamily checking + * If we can remove the index_col from the final UniqueKey->exprs. + */ + forboth(lc1, const_exprs, lc2, const_expr_opfamilies) + { + if (list_member_oid((List *)lfirst(lc2), unique_index->opfamily[c]) + && match_index_to_operand((Node *) lfirst(lc1), c, unique_index)) + { + matched_const = true; + break; + } + } + + if (matched_const) + continue; + + /* Check if the indexed expr is used in rel */ + if (attr > 0) + { + /* + * Normal Indexed column, if the col is not used, then the index is useless + * for uniquekey. + */ + attr -= FirstLowInvalidHeapAttributeNumber; + + if (!bms_is_member(attr, used_varattrs)) + { + *useful = false; + break; + } + } + else if (!list_member(unique_index->rel->reltarget->exprs, expr)) + { + /* Expression index but the expression is not used in rel */ + *useful = false; + break; + } + + /* check not null property. */ + if (attr == 0) + { + /* We never know if a expression yields null or not */ + *multi_nullvals = true; + } + else if (!bms_is_member(attr, unique_index->rel->notnullattrs) + && !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber, + unique_index->rel->notnullattrs)) + { + *multi_nullvals = true; + } + + exprs = lappend(exprs, expr); + } + return exprs; +} + + +/* + * add_uniquekey_for_onerow + * If we are sure that the relation only returns one row, then all the columns + * are unique. However we don't need to create UniqueKey for every column, we + * just set exprs = NIL and overwrites all the other UniqueKey on this RelOptInfo + * since this one has strongest semantics. + */ +void +add_uniquekey_for_onerow(RelOptInfo *rel) +{ + /* + * We overwrite the previous UniqueKey on purpose since this one has the + * strongest semantic. + */ + rel->uniquekeys = list_make1(makeUniqueKey(NIL, false)); +} + + +/* + * initililze_uniquecontext_for_joinrel + * Return a List of UniqueKeyContext for an inputrel + */ +static List * +initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel) +{ + List *res = NIL; + ListCell *lc; + foreach(lc, inputrel->uniquekeys) + { + UniqueKeyContext context; + context = palloc(sizeof(struct UniqueKeyContextData)); + context->uniquekey = lfirst_node(UniqueKey, lc); + context->added_to_joinrel = false; + context->useful = true; + res = lappend(res, context); + } + return res; +} + + +/* + * get_exprs_from_uniquekey + * Unify the way of get List of exprs from a one-row UniqueKey or + * normal UniqueKey. for the onerow case, every expr in rel1 is a valid + * UniqueKey. Return a List of exprs. + * + * rel1: The relation which you want to get the exprs. + * ukey: The UniqueKey you want to get the exprs. + */ +static List * +get_exprs_from_uniquekey(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *rel1, UniqueKey *ukey) +{ + ListCell *lc; + bool onerow = rel1 != NULL && relation_is_onerow(rel1); + + List *res = NIL; + Assert(onerow || ukey); + if (onerow) + { + /* Only cares about the exprs still exist in joinrel */ + foreach(lc, joinrel->reltarget->exprs) + { + Bitmapset *relids = pull_varnos(root, lfirst(lc)); + if (bms_is_subset(relids, rel1->relids)) + { + res = lappend(res, list_make1(lfirst(lc))); + } + } + } + else + { + res = list_make1(ukey->exprs); + } + return res; +} + +/* + * Partitioned table Unique Keys. + * The partition table unique key is maintained as: + * 1. The index must be unique as usual. + * 2. The index must contains partition key. + * 3. The index must exist on all the child rel. see simple_indexinfo_equal for + * how we compare it. + */ + +/* + * index_constains_partkey + * return true if the index contains the partiton key. + */ +static bool +index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind) +{ + ListCell *lc; + int i; + Assert(IS_PARTITIONED_REL(partrel)); + Assert(partrel->part_scheme->partnatts > 0); + + for(i = 0; i < partrel->part_scheme->partnatts; i++) + { + Node *part_expr = linitial(partrel->partexprs[i]); + bool found_in_index = false; + foreach(lc, ind->indextlist) + { + Expr *index_expr = lfirst_node(TargetEntry, lc)->expr; + if (equal(index_expr, part_expr)) + { + found_in_index = true; + break; + } + } + if (!found_in_index) + return false; + } + return true; +} + +/* + * simple_indexinfo_equal + * + * Used to check if the 2 index is same as each other. The index here + * is COPIED from childrel and did some tiny changes(see + * simple_copy_indexinfo_to_parent) + */ +static bool +simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2) +{ + Size oid_cmp_len = sizeof(Oid) * ind1->ncolumns; + + return ind1->ncolumns == ind2->ncolumns && + ind1->unique == ind2->unique && + memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0 && + memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0 && + memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0 && + memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0 && + equal(get_tlist_exprs(ind1->indextlist, true), + get_tlist_exprs(ind2->indextlist, true)); +} + + +/* + * The below macros are used for simple_copy_indexinfo_to_parent which is so + * customized that I don't want to put it to copyfuncs.c. So copy it here. + */ +#define COPY_POINTER_FIELD(fldname, sz) \ + do { \ + Size _size = (sz); \ + newnode->fldname = palloc(_size); \ + memcpy(newnode->fldname, from->fldname, _size); \ + } while (0) + +#define COPY_NODE_FIELD(fldname) \ + (newnode->fldname = copyObjectImpl(from->fldname)) + +#define COPY_SCALAR_FIELD(fldname) \ + (newnode->fldname = from->fldname) + + +/* + * simple_copy_indexinfo_to_parent (from partition) + * Copy the IndexInfo from child relation to parent relation with some modification, + * which is used to test: + * 1. If the same index exists in all the childrels. + * 2. If the parentrel->reltarget/basicrestrict info matches this index. + */ +static IndexOptInfo * +simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from) +{ + IndexOptInfo *newnode = makeNode(IndexOptInfo); + AppendRelInfo *appinfo = find_appinfo_by_child(root, from->rel->relid); + ListCell *lc; + int idx = 0; + + COPY_SCALAR_FIELD(ncolumns); + COPY_SCALAR_FIELD(nkeycolumns); + COPY_SCALAR_FIELD(unique); + COPY_SCALAR_FIELD(immediate); + /* We just need to know if it is NIL or not */ + COPY_SCALAR_FIELD(indpred); + COPY_SCALAR_FIELD(predOK); + COPY_POINTER_FIELD(indexkeys, from->ncolumns * sizeof(int)); + COPY_POINTER_FIELD(indexcollations, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opfamily, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opcintype, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(sortopfamily, from->ncolumns * sizeof(Oid)); + COPY_NODE_FIELD(indextlist); + + /* Convert index exprs on child expr to expr on parent */ + foreach(lc, newnode->indextlist) + { + TargetEntry *tle = lfirst_node(TargetEntry, lc); + /* Index on expression is ignored */ + Assert(IsA(tle->expr, Var)); + tle->expr = (Expr *) find_parent_var(appinfo, (Var *) tle->expr); + newnode->indexkeys[idx] = castNode(Var, tle->expr)->varattno; + idx++; + } + newnode->rel = parentrel; + return newnode; +} + +/* + * adjust_partition_unique_indexlist + * + * global_unique_indexes: At the beginning, it contains the copy & modified + * unique index from the first partition. And then check if each index in it still + * exists in the following partitions. If no, remove it. at last, it has an + * index list which exists in all the partitions. + */ +static void +adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_indexes) +{ + ListCell *lc, *lc2; + foreach(lc, *global_unique_indexes) + { + IndexOptInfo *g_ind = lfirst_node(IndexOptInfo, lc); + bool found_in_child = false; + + foreach(lc2, childrel->indexlist) + { + IndexOptInfo *p_ind = lfirst_node(IndexOptInfo, lc2); + IndexOptInfo *p_ind_copy; + if (!p_ind->unique || !p_ind->immediate || + (p_ind->indpred != NIL && !p_ind->predOK)) + continue; + p_ind_copy = simple_copy_indexinfo_to_parent(root, parentrel, p_ind); + if (simple_indexinfo_equal(p_ind_copy, g_ind)) + { + found_in_child = true; + break; + } + } + if (!found_in_child) + /* The index doesn't exist in childrel, remove it from global_unique_indexes */ + *global_unique_indexes = foreach_delete_current(*global_unique_indexes, lc); + } +} + +/* Helper function for groupres/distinctrel */ +static void +add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgroups) +{ + Query *parse = root->parse; + List *exprs; + + /* + * XXX: If there are some vars which is not in current levelsup, the semantic is + * imprecise, should we avoid it or not? levelsup = 1 is just a demo, maybe we need to + * check every level other than 0, if so, looks we have to write another + * pull_var_walker. + */ + List *upper_vars = pull_vars_of_level((Node*)sortgroups, 1); + + if (upper_vars != NIL) + return; + + exprs = get_sortgrouplist_exprs(sortgroups, parse->targetList); + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(exprs, + false /* sortgroupclause can't be multi_nullvals */)); +} + + +/* + * add_combined_uniquekey + * The combination of both UniqueKeys is a valid UniqueKey for joinrel no matter + * the jointype. + */ +bool +add_combined_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype) +{ + + ListCell *lc1, *lc2; + + /* Either side has multi_nullvals or we have outer join, + * the combined UniqueKey has multi_nullvals */ + bool multi_nullvals = outer_ukey->multi_nullvals || + inner_ukey->multi_nullvals || IS_OUTER_JOIN(jointype); + + /* The only case we can get onerow joinrel after join */ + if (relation_is_onerow(outer_rel) + && relation_is_onerow(inner_rel) + && jointype == JOIN_INNER) + { + add_uniquekey_for_onerow(joinrel); + return true; + } + + foreach(lc1, get_exprs_from_uniquekey(root, joinrel, outer_rel, outer_ukey)) + { + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, inner_rel, inner_ukey)) + { + List *exprs = list_concat_copy(lfirst_node(List, lc1), lfirst_node(List, lc2)); + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(exprs, + multi_nullvals)); + } + } + return false; +} diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index 424d25cbd5..8d8e493f5c 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -2387,6 +2387,8 @@ grouping_planner(PlannerInfo *root, bool inheritance_update, add_path(final_rel, path); } + simple_copy_uniquekeys(current_rel, final_rel); + /* * Generate partial paths for final_rel, too, if outer query levels might * be able to make use of them. @@ -3899,6 +3901,8 @@ create_grouping_paths(PlannerInfo *root, } set_cheapest(grouped_rel); + + populate_grouprel_uniquekeys(root, grouped_rel, input_rel); return grouped_rel; } @@ -4621,7 +4625,7 @@ create_window_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(window_rel); - + simple_copy_uniquekeys(input_rel, window_rel); return window_rel; } @@ -4935,7 +4939,7 @@ create_distinct_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(distinct_rel); - + populate_distinctrel_uniquekeys(root, input_rel, distinct_rel); return distinct_rel; } @@ -5196,6 +5200,8 @@ create_ordered_paths(PlannerInfo *root, */ Assert(ordered_rel->pathlist != NIL); + simple_copy_uniquekeys(input_rel, ordered_rel); + return ordered_rel; } @@ -6073,6 +6079,9 @@ adjust_paths_for_srfs(PlannerInfo *root, RelOptInfo *rel, if (list_length(targets) == 1) return; + /* UniqueKey is not valid after handling the SRF. */ + rel->uniquekeys = NIL; + /* * Stack SRF-evaluation nodes atop each path for the rel. * diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c index becdcbb872..b7626545bf 100644 --- a/src/backend/optimizer/prep/prepunion.c +++ b/src/backend/optimizer/prep/prepunion.c @@ -689,6 +689,8 @@ generate_union_paths(SetOperationStmt *op, PlannerInfo *root, /* Undo effects of possibly forcing tuple_fraction to 0 */ root->tuple_fraction = save_fraction; + /* Add the UniqueKeys */ + populate_unionrel_uniquekeys(root, result_rel); return result_rel; } diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c index 86922a273c..6817c9c787 100644 --- a/src/backend/optimizer/util/appendinfo.c +++ b/src/backend/optimizer/util/appendinfo.c @@ -746,3 +746,47 @@ find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos) } return appinfos; } + +/* + * find_appinfo_by_child + * + */ +AppendRelInfo * +find_appinfo_by_child(PlannerInfo *root, Index child_index) +{ + ListCell *lc; + foreach(lc, root->append_rel_list) + { + AppendRelInfo *appinfo = lfirst_node(AppendRelInfo, lc); + if (appinfo->child_relid == child_index) + return appinfo; + } + elog(ERROR, "parent relation cant be found"); + return NULL; +} + +/* + * find_parent_var + * + */ +Var * +find_parent_var(AppendRelInfo *appinfo, Var *child_var) +{ + ListCell *lc; + Var *res = NULL; + Index attno = 1; + foreach(lc, appinfo->translated_vars) + { + Node *child_node = lfirst(lc); + if (equal(child_node, child_var)) + { + res = copyObject(child_var); + res->varattno = attno; + res->varno = appinfo->parent_relid; + } + attno++; + } + if (res == NULL) + elog(ERROR, "parent var can't be found."); + return res; +} diff --git a/src/backend/optimizer/util/inherit.c b/src/backend/optimizer/util/inherit.c index be1c9ddd96..3eec1f4d74 100644 --- a/src/backend/optimizer/util/inherit.c +++ b/src/backend/optimizer/util/inherit.c @@ -736,6 +736,7 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, { Node *onecq = (Node *) lfirst(lc2); bool pseudoconstant; + RestrictInfo *child_rinfo; /* check for pseudoconstant (no Vars or volatile functions) */ pseudoconstant = @@ -747,14 +748,15 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, root->hasPseudoConstantQuals = true; } /* reconstitute RestrictInfo with appropriate properties */ - childquals = lappend(childquals, - make_restrictinfo(root, - (Expr *) onecq, - rinfo->is_pushed_down, - rinfo->outerjoin_delayed, - pseudoconstant, - rinfo->security_level, - NULL, NULL, NULL)); + child_rinfo = make_restrictinfo(root, + (Expr *) onecq, + rinfo->is_pushed_down, + rinfo->outerjoin_delayed, + pseudoconstant, + rinfo->security_level, + NULL, NULL, NULL); + child_rinfo->mergeopfamilies = rinfo->mergeopfamilies; + childquals = lappend(childquals, child_rinfo); /* track minimum security level among child quals */ cq_min_security = Min(cq_min_security, rinfo->security_level); } diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 48a7ebfe45..4fe1824eb0 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -16,6 +16,7 @@ #include "nodes/execnodes.h" #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" extern A_Expr *makeA_Expr(A_Expr_Kind kind, List *name, @@ -106,4 +107,6 @@ extern GroupingSet *makeGroupingSet(GroupingSetKind kind, List *content, int loc extern VacuumRelation *makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols); +extern UniqueKey* makeUniqueKey(List *exprs, bool multi_nullvals); + #endif /* MAKEFUNC_H */ diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index e22df890ef..3320273ac1 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -265,6 +265,7 @@ typedef enum NodeTag T_EquivalenceMember, T_PathKey, T_PathTarget, + T_UniqueKey, T_RestrictInfo, T_IndexClause, T_PlaceHolderVar, diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index 0d61f04d27..80561b1003 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -739,6 +739,7 @@ typedef struct RelOptInfo QualCost baserestrictcost; /* cost of evaluating the above */ Index baserestrict_min_security; /* min security_level found in * baserestrictinfo */ + List *uniquekeys; /* List of UniqueKey */ List *joininfo; /* RestrictInfo structures for join clauses * involving this rel */ bool has_eclass_joins; /* T means joininfo is incomplete */ @@ -1059,6 +1060,28 @@ typedef struct PathKey } PathKey; +/* + * UniqueKey + * + * Represents the unique properties held by a RelOptInfo. + * + * exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL + * is a special case of UniqueKey, which means there is only 1 row in that + * relation. + * multi_nullvals: true means multi null values may exist in these exprs, so the + * uniqueness is not guaranteed in this case. This field is necessary for + * remove_useless_join & reduce_unique_semijoins where we don't mind these + * duplicated NULL values. It is set to true for 2 cases. One is a unique key + * from a unique index but the related column is nullable. The other one is for + * outer join. see populate_joinrel_uniquekeys for detail. + */ +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + /* * PathTarget * @@ -2496,7 +2519,7 @@ typedef enum * * flags indicating what kinds of grouping are possible. * partial_costs_set is true if the agg_partial_costs and agg_final_costs - * have been initialized. + * have been initialized. * agg_partial_costs gives partial aggregation costs. * agg_final_costs gives finalization costs. * target_parallel_safe is true if target is parallel safe. @@ -2526,8 +2549,8 @@ typedef struct * limit_tuples is an estimated bound on the number of output tuples, * or -1 if no LIMIT or couldn't estimate. * count_est and offset_est are the estimated values of the LIMIT and OFFSET - * expressions computed by preprocess_limit() (see comments for - * preprocess_limit() for more information). + * expressions computed by preprocess_limit() (see comments for + * preprocess_limit() for more information). */ typedef struct { diff --git a/src/include/nodes/pg_list.h b/src/include/nodes/pg_list.h index 404e03f132..85c65e910c 100644 --- a/src/include/nodes/pg_list.h +++ b/src/include/nodes/pg_list.h @@ -566,6 +566,16 @@ extern pg_nodiscard List *list_delete_first(List *list); extern pg_nodiscard List *list_delete_last(List *list); extern pg_nodiscard List *list_delete_nth_cell(List *list, int n); extern pg_nodiscard List *list_delete_cell(List *list, ListCell *cell); +extern bool list_is_subset(const List *members, const List *target); + +extern List *list_delete(List *list, void *datum); +extern List *list_delete_ptr(List *list, void *datum); +extern List *list_delete_int(List *list, int datum); +extern List *list_delete_oid(List *list, Oid datum); +extern List *list_delete_first(List *list); +extern List *list_delete_last(List *list); +extern List *list_delete_nth_cell(List *list, int n); +extern List *list_delete_cell(List *list, ListCell *cell); extern List *list_union(const List *list1, const List *list2); extern List *list_union_ptr(const List *list1, const List *list2); diff --git a/src/include/optimizer/appendinfo.h b/src/include/optimizer/appendinfo.h index 4cbf8c26cc..43b8d05f69 100644 --- a/src/include/optimizer/appendinfo.h +++ b/src/include/optimizer/appendinfo.h @@ -32,4 +32,7 @@ extern Relids adjust_child_relids_multilevel(PlannerInfo *root, Relids relids, extern AppendRelInfo **find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos); +extern AppendRelInfo *find_appinfo_by_child(PlannerInfo *root, Index child_index); +extern Var *find_parent_var(AppendRelInfo *appinfo, Var *child_var); + #endif /* APPENDINFO_H */ diff --git a/src/include/optimizer/optimizer.h b/src/include/optimizer/optimizer.h index d587952b7d..843aafa51e 100644 --- a/src/include/optimizer/optimizer.h +++ b/src/include/optimizer/optimizer.h @@ -23,6 +23,7 @@ #define OPTIMIZER_H #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" /* Test if an expression node represents a SRF call. Beware multiple eval! */ #define IS_SRF_CALL(node) \ @@ -168,6 +169,7 @@ extern TargetEntry *get_sortgroupref_tle(Index sortref, List *targetList); extern TargetEntry *get_sortgroupclause_tle(SortGroupClause *sgClause, List *targetList); +extern Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); extern Node *get_sortgroupclause_expr(SortGroupClause *sgClause, List *targetList); extern List *get_sortgrouplist_exprs(List *sgClauses, diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h index 035d3e1206..1adf99c9ee 100644 --- a/src/include/optimizer/paths.h +++ b/src/include/optimizer/paths.h @@ -247,5 +247,48 @@ extern PathKey *make_canonical_pathkey(PlannerInfo *root, int strategy, bool nulls_first); extern void add_paths_to_append_rel(PlannerInfo *root, RelOptInfo *rel, List *live_childrels); +extern List *select_mergejoin_clauses(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + JoinType jointype, + bool *mergejoin_allowed); + +/* + * uniquekeys.c + * Utilities for matching and building unique keys + */ +extern void populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List* unique_index_list); +extern void populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels); +extern void populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel); +extern void populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel); +extern void populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel); +extern void simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel); +extern void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel); +extern void populate_joinrel_uniquekeys(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + RelOptInfo *rel2, + List *restrictlist, + JoinType jointype); + +extern bool relation_has_uniquekeys_for(PlannerInfo *root, + RelOptInfo *rel, + List *exprs, + bool allow_multinulls); +extern bool relation_is_onerow(RelOptInfo *rel); #endif /* PATHS_H */ -- 2.26.2 --yh7cw345mbttadjz Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v38-0003-Extend-UniqueKeys.patch" ^ permalink raw reply [nested|flat] 91+ messages in thread
* [PATCH v36 2/6] Introduce UniqueKey attributes on RelOptInfo struct. @ 2020-05-11 07:50 一挃 <[email protected]> 0 siblings, 0 replies; 91+ messages in thread From: 一挃 @ 2020-05-11 07:50 UTC (permalink / raw) UniqueKey is a set of exprs on RelOptInfo which represents the exprs will be unique on the given RelOptInfo. You can see README.uniquekey for more information. --- src/backend/nodes/copyfuncs.c | 13 + src/backend/nodes/list.c | 31 + src/backend/nodes/makefuncs.c | 13 + src/backend/nodes/outfuncs.c | 11 + src/backend/nodes/readfuncs.c | 10 + src/backend/optimizer/path/Makefile | 3 +- src/backend/optimizer/path/README.uniquekey | 131 +++ src/backend/optimizer/path/allpaths.c | 10 + src/backend/optimizer/path/joinpath.c | 9 +- src/backend/optimizer/path/joinrels.c | 2 + src/backend/optimizer/path/pathkeys.c | 3 +- src/backend/optimizer/path/uniquekeys.c | 1131 +++++++++++++++++++ src/backend/optimizer/plan/planner.c | 13 +- src/backend/optimizer/prep/prepunion.c | 2 + src/backend/optimizer/util/appendinfo.c | 44 + src/backend/optimizer/util/inherit.c | 16 +- src/include/nodes/makefuncs.h | 3 + src/include/nodes/nodes.h | 1 + src/include/nodes/pathnodes.h | 29 +- src/include/nodes/pg_list.h | 2 + src/include/optimizer/appendinfo.h | 3 + src/include/optimizer/optimizer.h | 2 + src/include/optimizer/paths.h | 43 + 23 files changed, 1502 insertions(+), 23 deletions(-) create mode 100644 src/backend/optimizer/path/README.uniquekey create mode 100644 src/backend/optimizer/path/uniquekeys.c diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 89c409de66..1f50400fd2 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2273,6 +2273,16 @@ _copyPathKey(const PathKey *from) return newnode; } +static UniqueKey * +_copyUniqueKey(const UniqueKey *from) +{ + UniqueKey *newnode = makeNode(UniqueKey); + + COPY_NODE_FIELD(exprs); + COPY_SCALAR_FIELD(multi_nullvals); + + return newnode; +} /* * _copyRestrictInfo */ @@ -5152,6 +5162,9 @@ copyObjectImpl(const void *from) case T_PathKey: retval = _copyPathKey(from); break; + case T_UniqueKey: + retval = _copyUniqueKey(from); + break; case T_RestrictInfo: retval = _copyRestrictInfo(from); break; diff --git a/src/backend/nodes/list.c b/src/backend/nodes/list.c index 80fa8c84e4..a7a99b70f2 100644 --- a/src/backend/nodes/list.c +++ b/src/backend/nodes/list.c @@ -687,6 +687,37 @@ list_member_oid(const List *list, Oid datum) return false; } +/* + * return true iff every entry in "members" list is also present + * in the "target" list. + */ +bool +list_is_subset(const List *members, const List *target) +{ + const ListCell *lc1, *lc2; + + Assert(IsPointerList(members)); + Assert(IsPointerList(target)); + check_list_invariants(members); + check_list_invariants(target); + + foreach(lc1, members) + { + bool found = false; + foreach(lc2, target) + { + if (equal(lfirst(lc1), lfirst(lc2))) + { + found = true; + break; + } + } + if (!found) + return false; + } + return true; +} + /* * Delete the n'th cell (counting from 0) in list. * diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 49de285f01..646cf7c9a1 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -814,3 +814,16 @@ makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols) v->va_cols = va_cols; return v; } + + +/* + * makeUniqueKey + */ +UniqueKey* +makeUniqueKey(List *exprs, bool multi_nullvals) +{ + UniqueKey * ukey = makeNode(UniqueKey); + ukey->exprs = exprs; + ukey->multi_nullvals = multi_nullvals; + return ukey; +} diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index e2f177515d..c3a9632992 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2428,6 +2428,14 @@ _outPathKey(StringInfo str, const PathKey *node) WRITE_BOOL_FIELD(pk_nulls_first); } +static void +_outUniqueKey(StringInfo str, const UniqueKey *node) +{ + WRITE_NODE_TYPE("UNIQUEKEY"); + WRITE_NODE_FIELD(exprs); + WRITE_BOOL_FIELD(multi_nullvals); +} + static void _outPathTarget(StringInfo str, const PathTarget *node) { @@ -4127,6 +4135,9 @@ outNode(StringInfo str, const void *obj) case T_PathKey: _outPathKey(str, obj); break; + case T_UniqueKey: + _outUniqueKey(str, obj); + break; case T_PathTarget: _outPathTarget(str, obj); break; diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index 42050ab719..3a18571d0c 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -452,6 +452,14 @@ _readSetOperationStmt(void) READ_DONE(); } +static UniqueKey * +_readUniqueKey(void) +{ + READ_LOCALS(UniqueKey); + READ_NODE_FIELD(exprs); + READ_BOOL_FIELD(multi_nullvals); + READ_DONE(); +} /* * Stuff from primnodes.h. @@ -2656,6 +2664,8 @@ parseNodeString(void) return_value = _readCommonTableExpr(); else if (MATCH("SETOPERATIONSTMT", 16)) return_value = _readSetOperationStmt(); + else if (MATCH("UNIQUEKEY", 9)) + return_value = _readUniqueKey(); else if (MATCH("ALIAS", 5)) return_value = _readAlias(); else if (MATCH("RANGEVAR", 8)) diff --git a/src/backend/optimizer/path/Makefile b/src/backend/optimizer/path/Makefile index 1e199ff66f..7b9820c25f 100644 --- a/src/backend/optimizer/path/Makefile +++ b/src/backend/optimizer/path/Makefile @@ -21,6 +21,7 @@ OBJS = \ joinpath.o \ joinrels.o \ pathkeys.o \ - tidpath.o + tidpath.o \ + uniquekeys.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/optimizer/path/README.uniquekey b/src/backend/optimizer/path/README.uniquekey new file mode 100644 index 0000000000..5eac761995 --- /dev/null +++ b/src/backend/optimizer/path/README.uniquekey @@ -0,0 +1,131 @@ +1. What is UniqueKey? +We can think UniqueKey is a set of exprs for a RelOptInfo, which we are insure +that doesn't yields same result among all the rows. The simplest UniqueKey +format is primary key. + +However we define the UnqiueKey as below. + +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + +exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL +is a special case of UniqueKey, which means there is only one row in that +relation.it has a stronger semantic than others. like SELECT uk FROM t; uk is +normal unique key and may have different values. SELECT colx FROM t WHERE uk = +const. colx is unique AND we have only 1 value. This field can used for +innerrel_is_unique. this logic is handled specially in add_uniquekey_for_onerow +function. + +multi_nullvals: true means multi null values may exist in these exprs, so the +uniqueness is not guaranteed in this case. This field is necessary for +remove_useless_join & reduce_unique_semijoins where we don't mind these +duplicated NULL values. It is set to true for 2 cases. One is a unique key +from a unique index but the related column is nullable. The other one is for +outer join. see populate_joinrel_uniquekeys for detail. + + +The UniqueKey can be used at the following cases at least: +1. remove_useless_joins. +2. reduce_semianti_joins +3. remove distinct node if distinct clause is unique. +4. remove aggnode if group by clause is unique. +5. Index Skip Scan (WIP) +6. Aggregation Push Down without 2 phase aggregation if the join can't + duplicated the aggregated rows. (work in progress feature) + +2. How is it maintained? + +We have a set of populate_xxx_unqiuekeys functions to maintain the uniquekey on +various cases. xxx includes baserel, joinrel, partitionedrel, distinctrel, +groupedrel, unionrel. and we also need to convert the uniquekey from subquery +to outer relation, which is what convert_subquery_uniquekeys does. + +1. The first part is about baserel. We handled 3 cases. suppose we have Unique +Index on (a, b). + +1. SELECT a, b FROM t. UniqueKey (a, b) +2. SELECT a FROM t WHERE b = 1; UniqueKey (a) +3. SELECT .. FROM t WHERE a = 1 AND b = 1; UniqueKey (NIL). onerow case, every + column is Unique. + +2. The next part is joinrel, this part is most error-prone, we simplified the rules +like below: +1. If the relation's UniqueKey can't be duplicated after join, then is will be + still valid for the join rel. The function we used here is + innerrel_keeps_unique. The basic idea is innerrel.any_col = outer.uk. + +2. If the UnqiueKey can't keep valid via the rule 1, the combination of the + UniqueKey from both sides are valid for sure. We can prove this as: if the + unique exprs from rel1 is duplicated by rel2, the duplicated rows must + contains different unique exprs from rel2. + +More considerations about onerow: +1. If relation with one row and it can't be duplicated, it is still possible + contains mulit_nullvas after outer join. +2. If the either UniqueKey can be duplicated after join, the can get one row + only when both side is one row AND there is no outer join. +3. Whenever the onerow UniqueKey is not a valid any more, we need to convert one + row UniqueKey to normal unique key since we don't store exprs for one-row + relation. get_exprs_from_uniquekeys will be used here. + + +More considerations about multi_nullvals after join: +1. If the original UnqiueKey has multi_nullvals, the final UniqueKey will have + mulit_nullvals in any case. +2. If a unique key doesn't allow mulit_nullvals, after some outer join, it + allows some outer join. + + +3. When we comes to subquery, we need to convert_subquery_unqiuekeys just like +convert_subquery_pathkeys. Only the UniqueKey insides subquery is referenced as +a Var in outer relation will be reused. The relationship between the outerrel.Var +and subquery.exprs is built with outerel->subroot->processed_tlist. + + +4. As for the SRF functions, it will break the uniqueness of uniquekey, However it +is handled in adjust_paths_for_srfs, which happens after the query_planner. so +we will maintain the UniqueKey until there and reset it to NIL at that +places. This can't help on distinct/group by elimination cases but probably help +in some other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is +semantic correctly. + + +5. As for inherit table, we first main the UnqiueKey on childrel as well. But for +partitioned table we need to maintain 2 different kinds of +UnqiueKey. 1). UniqueKey on the parent relation 2). UniqueKey on child +relation for partition wise query. + +Example: +CREATE TABLE p (a int not null, b int not null) partition by list (a); +CREATE TABLE p0 partition of p for values in (1); +CREATE TABLE p1 partition of p for values in (2); + +create unique index p0_b on p0(b); +create unique index p1_b on p1(b); + +Now b is only unique on partition level, so the distinct can't be removed on +the following cases. SELECT DISTINCT b FROM p; + +Another example is SELECT DISTINCT a, b FROM p WHERE a = 1; Since only one +partition is chosen, the UniqueKey on child relation is same as the UniqueKey on +parent relation. + +Another usage of UniqueKey on partition level is it be helpful for +partition-wise join. + +As for the UniqueKey on parent table level, it comes with 2 different ways, +1). the UniqueKey is also derived in UniqueKey index, but the index must be same +in all the related children relations and the unique index must contains +Partition Key in it. Example: + +CREATE UNIQUE INDEX p_ab ON p(a, b); -- where a is the partition key. + +-- Query +SELECT a, b FROM p; the (a, b) is a UniqueKey of p. + +2). If the parent relation has only one childrel, the UniqueKey on childrel is + the UniqueKey on parent as well. diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index 484dab0a1a..2ad9d06d7a 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -579,6 +579,12 @@ set_plain_rel_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) */ check_index_predicates(root, rel); + /* + * Now that we've marked which partial indexes are suitable, we can now + * build the relation's unique keys. + */ + populate_baserel_uniquekeys(root, rel, rel->indexlist); + /* Mark rel with estimated output rows, width, etc */ set_baserel_size_estimates(root, rel); } @@ -1310,6 +1316,8 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, /* Add paths to the append relation. */ add_paths_to_append_rel(root, rel, live_childrels); + if (IS_PARTITIONED_REL(rel)) + populate_partitionedrel_uniquekeys(root, rel, live_childrels); } @@ -2383,6 +2391,8 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, pathkeys, required_outer)); } + convert_subquery_uniquekeys(root, rel, sub_final_rel); + /* If outer rel allows parallelism, do same for partial paths. */ if (rel->consider_parallel && bms_is_empty(required_outer)) { diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c index db54a6ba2e..ef0fd2fb0b 100644 --- a/src/backend/optimizer/path/joinpath.c +++ b/src/backend/optimizer/path/joinpath.c @@ -71,13 +71,6 @@ static void consider_parallel_mergejoin(PlannerInfo *root, static void hash_inner_and_outer(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, RelOptInfo *innerrel, JoinType jointype, JoinPathExtraData *extra); -static List *select_mergejoin_clauses(PlannerInfo *root, - RelOptInfo *joinrel, - RelOptInfo *outerrel, - RelOptInfo *innerrel, - List *restrictlist, - JoinType jointype, - bool *mergejoin_allowed); static void generate_mergejoin_paths(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *innerrel, @@ -1927,7 +1920,7 @@ hash_inner_and_outer(PlannerInfo *root, * if it is mergejoinable and involves vars from the two sub-relations * currently of interest. */ -static List * +List * select_mergejoin_clauses(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c index 2d343cd293..b9163ee8ff 100644 --- a/src/backend/optimizer/path/joinrels.c +++ b/src/backend/optimizer/path/joinrels.c @@ -924,6 +924,8 @@ populate_joinrel_with_paths(PlannerInfo *root, RelOptInfo *rel1, /* Apply partitionwise join technique, if possible. */ try_partitionwise_join(root, rel1, rel2, joinrel, sjinfo, restrictlist); + + populate_joinrel_uniquekeys(root, joinrel, rel1, rel2, restrictlist, sjinfo->jointype); } diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index ce9bf87e9b..7e596d4194 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -33,7 +33,6 @@ static bool pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys); static bool matches_boolean_partition_clause(RestrictInfo *rinfo, RelOptInfo *partrel, int partkeycol); -static Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); static bool right_merge_direction(PlannerInfo *root, PathKey *pathkey); @@ -1035,7 +1034,7 @@ convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel, * We need this to ensure that we don't return pathkeys describing values * that are unavailable above the level of the subquery scan. */ -static Var * +Var * find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle) { ListCell *lc; diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c new file mode 100644 index 0000000000..b33bcd2f32 --- /dev/null +++ b/src/backend/optimizer/path/uniquekeys.c @@ -0,0 +1,1131 @@ +/*------------------------------------------------------------------------- + * + * uniquekeys.c + * Utilities for matching and building unique keys + * + * Portions Copyright (c) 2020, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/optimizer/path/uniquekeys.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "nodes/makefuncs.h" +#include "nodes/nodeFuncs.h" +#include "optimizer/pathnode.h" +#include "optimizer/paths.h" +#include "optimizer/appendinfo.h" +#include "optimizer/optimizer.h" +#include "optimizer/tlist.h" +#include "rewrite/rewriteManip.h" + + +/* + * This struct is used to help populate_joinrel_uniquekeys. + * + * added_to_joinrel is true if a uniquekey (from outerrel or innerrel) + * has been added to joinrel. + * useful is true if the exprs of the uniquekey still appears in joinrel. + */ +typedef struct UniqueKeyContextData +{ + UniqueKey *uniquekey; + bool added_to_joinrel; + bool useful; +} *UniqueKeyContext; + +static List *initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel); +static bool innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + bool reverse); + +static List *get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals); +static List *get_exprs_from_uniquekey(RelOptInfo *joinrel, + RelOptInfo *rel1, + UniqueKey *ukey); +static void add_uniquekey_for_onerow(RelOptInfo *rel); +static bool add_combined_uniquekey(RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype); + +/* Used for unique indexes checking for partitioned table */ +static bool index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind); +static IndexOptInfo *simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from); +static bool simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2); +static void adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_index); + +/* Helper function for grouped relation and distinct relation. */ +static void add_uniquekey_from_sortgroups(PlannerInfo *root, + RelOptInfo *rel, + List *sortgroups); + +/* + * populate_baserel_uniquekeys + * Populate 'baserel' uniquekeys list by looking at the rel's unique index + * and baserestrictinfo + */ +void +populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List *indexlist) +{ + ListCell *lc; + List *matched_uniq_indexes = NIL; + + /* Attrs appears in rel->reltarget->exprs. */ + Bitmapset *used_attrs = NULL; + + List *const_exprs = NIL; + List *expr_opfamilies = NIL; + + Assert(baserel->rtekind == RTE_RELATION); + + foreach(lc, indexlist) + { + IndexOptInfo *ind = (IndexOptInfo *) lfirst(lc); + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + matched_uniq_indexes = lappend(matched_uniq_indexes, ind); + } + + if (matched_uniq_indexes == NIL) + return; + + /* Check which attrs is used in baserel->reltarget */ + pull_varattnos((Node *)baserel->reltarget->exprs, baserel->relid, &used_attrs); + + /* Check which attrno is used at a mergeable const filter */ + foreach(lc, baserel->baserestrictinfo) + { + RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); + + if (rinfo->mergeopfamilies == NIL) + continue; + + if (bms_is_empty(rinfo->left_relids)) + { + const_exprs = lappend(const_exprs, get_rightop(rinfo->clause)); + } + else if (bms_is_empty(rinfo->right_relids)) + { + const_exprs = lappend(const_exprs, get_leftop(rinfo->clause)); + } + else + continue; + + expr_opfamilies = lappend(expr_opfamilies, rinfo->mergeopfamilies); + } + + foreach(lc, matched_uniq_indexes) + { + bool multi_nullvals, useful; + List *exprs = get_exprs_from_uniqueindex(lfirst_node(IndexOptInfo, lc), + const_exprs, + expr_opfamilies, + used_attrs, + &useful, + &multi_nullvals); + if (useful) + { + if (exprs == NIL) + { + /* All the columns in Unique Index matched with a restrictinfo */ + add_uniquekey_for_onerow(baserel); + return; + } + baserel->uniquekeys = lappend(baserel->uniquekeys, + makeUniqueKey(exprs, multi_nullvals)); + } + } +} + + +/* + * populate_partitionedrel_uniquekeys + * The UniqueKey on partitionrel comes from 2 cases: + * 1). Only one partition is involved in this query, the unique key can be + * copied to parent rel from childrel. + * 2). There are some unique index which includes partition key and exists + * in all the related partitions. + * We never mind rule 2 if we hit rule 1. + */ + +void +populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels) +{ + ListCell *lc; + List *global_uniq_indexlist = NIL; + RelOptInfo *childrel; + bool is_first = true; + + Assert(IS_PARTITIONED_REL(rel)); + + if (childrels == NIL) + return; + + /* + * If there is only one partition used in this query, the UniqueKey in childrel is + * still valid in parent level, but we need convert the format from child expr to + * parent expr. + */ + if (list_length(childrels) == 1) + { + /* Check for Rule 1 */ + RelOptInfo *childrel = linitial_node(RelOptInfo, childrels); + ListCell *lc; + Assert(childrel->reloptkind == RELOPT_OTHER_MEMBER_REL); + if (relation_is_onerow(childrel)) + { + add_uniquekey_for_onerow(rel); + return; + } + + foreach(lc, childrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + AppendRelInfo *appinfo = find_appinfo_by_child(root, childrel->relid); + List *parent_exprs = NIL; + bool can_reuse = true; + ListCell *lc2; + foreach(lc2, ukey->exprs) + { + Var *var = (Var *)lfirst(lc2); + /* + * If the expr comes from a expression, it is hard to build the expression + * in parent so ignore that case for now. + */ + if(!IsA(var, Var)) + { + can_reuse = false; + break; + } + /* Convert it to parent var */ + parent_exprs = lappend(parent_exprs, find_parent_var(appinfo, var)); + } + if (can_reuse) + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(parent_exprs, + ukey->multi_nullvals)); + } + } + else + { + /* Check for rule 2 */ + childrel = linitial_node(RelOptInfo, childrels); + foreach(lc, childrel->indexlist) + { + IndexOptInfo *ind = lfirst(lc); + IndexOptInfo *modified_index; + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + + /* + * During simple_copy_indexinfo_to_parent, we need to convert var from + * child var to parent var, index on expression is too complex to handle. + * so ignore it for now. + */ + if (ind->indexprs != NIL) + continue; + + modified_index = simple_copy_indexinfo_to_parent(root, rel, ind); + /* + * If the unique index doesn't contain partkey, then it is unique + * on this partition only, so it is useless for us. + */ + if (!index_constains_partkey(rel, modified_index)) + continue; + + global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index); + } + + if (global_uniq_indexlist != NIL) + { + foreach(lc, childrels) + { + RelOptInfo *child = lfirst(lc); + if (is_first) + { + is_first = false; + continue; + } + adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist); + } + /* Now we have a list of unique index which are exactly same on all childrels, + * Set the UniqueKey just like it is non-partition table + */ + populate_baserel_uniquekeys(root, rel, global_uniq_indexlist); + } + } +} + + +/* + * populate_distinctrel_uniquekeys + */ +void +populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel) +{ + /* The unique key before the distinct is still valid. */ + distinctrel->uniquekeys = list_copy(inputrel->uniquekeys); + add_uniquekey_from_sortgroups(root, distinctrel, root->parse->distinctClause); +} + +/* + * populate_grouprel_uniquekeys + */ +void +populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel) + +{ + Query *parse = root->parse; + bool input_ukey_added = false; + ListCell *lc; + + if (relation_is_onerow(inputrel)) + { + add_uniquekey_for_onerow(grouprel); + return; + } + if (parse->groupingSets) + return; + + /* A Normal group by without grouping set. */ + if (parse->groupClause) + { + /* + * Current even the groupby clause is Unique already, but if query has aggref + * We have to create grouprel still. To keep the UnqiueKey short, we will check + * the UniqueKey of input_rel still valid, if so we reuse it. + */ + foreach(lc, inputrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (list_is_subset(ukey->exprs, grouprel->reltarget->exprs)) + { + grouprel->uniquekeys = lappend(grouprel->uniquekeys, + ukey); + input_ukey_added = true; + } + } + if (!input_ukey_added) + /* + * group by clause must be a super-set of grouprel->reltarget->exprs except the + * aggregation expr, so if such exprs is unique already, no bother to generate + * new uniquekey for group by exprs. + */ + add_uniquekey_from_sortgroups(root, + grouprel, + root->parse->groupClause); + } + else + /* It has aggregation but without a group by, so only one row returned */ + add_uniquekey_for_onerow(grouprel); +} + +/* + * simple_copy_uniquekeys + * Using a function for the one-line code makes us easy to check where we simply + * copied the uniquekey. + */ +void +simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel) +{ + newrel->uniquekeys = oldrel->uniquekeys; +} + +/* + * populate_unionrel_uniquekeys + */ +void +populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel) +{ + ListCell *lc; + List *exprs = NIL; + + Assert(unionrel->uniquekeys == NIL); + + foreach(lc, unionrel->reltarget->exprs) + { + exprs = lappend(exprs, lfirst(lc)); + } + + if (exprs == NIL) + /* SQL: select union select; is valid, we need to handle it here. */ + add_uniquekey_for_onerow(unionrel); + else + unionrel->uniquekeys = lappend(unionrel->uniquekeys, + makeUniqueKey(exprs,false)); + +} + +/* + * populate_joinrel_uniquekeys + * + * populate uniquekeys for joinrel. We will check each relation to see if its + * UniqueKey is still valid via innerrel_keeps_unique, if so, we add it to + * joinrel. The multi_nullvals field will be changed to true for some outer + * join cases and one-row UniqueKey needs to be converted to normal UniqueKey + * for the same case as well. + * For the uniquekey in either baserel which can't be unique after join, we still + * check to see if combination of UniqueKeys from both side is still useful for us. + * if yes, we add it to joinrel as well. + */ +void +populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *outerrel, RelOptInfo *innerrel, + List *restrictlist, JoinType jointype) +{ + ListCell *lc, *lc2; + List *clause_list = NIL; + List *outerrel_ukey_ctx; + List *innerrel_ukey_ctx; + bool inner_onerow, outer_onerow; + bool mergejoin_allowed; + + /* Care about the outerrel relation only for SEMI/ANTI join */ + if (jointype == JOIN_SEMI || jointype == JOIN_ANTI) + { + foreach(lc, outerrel->uniquekeys) + { + UniqueKey *uniquekey = lfirst_node(UniqueKey, lc); + if (list_is_subset(uniquekey->exprs, joinrel->reltarget->exprs)) + joinrel->uniquekeys = lappend(joinrel->uniquekeys, uniquekey); + } + return; + } + + Assert(jointype == JOIN_LEFT || jointype == JOIN_FULL || jointype == JOIN_INNER); + + /* Fast path */ + if (innerrel->uniquekeys == NIL || outerrel->uniquekeys == NIL) + return; + + inner_onerow = relation_is_onerow(innerrel); + outer_onerow = relation_is_onerow(outerrel); + + outerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(outerrel); + innerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(innerrel); + + clause_list = select_mergejoin_clauses(root, joinrel, outerrel, innerrel, + restrictlist, jointype, + &mergejoin_allowed); + + if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true /* reverse */)) + { + bool outer_impact = jointype == JOIN_FULL; + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + /* Outer relation has one row, and the unique key is not duplicated after join, + * the joinrel will still has one row unless the jointype == JOIN_FULL. + */ + if (outer_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (outer_onerow) + { + /* + * The onerow outerrel becomes multi rows and multi_nullvals + * will be changed to true. We also need to set the exprs correctly since it + * can't be NIL any more. + */ + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(joinrel, outerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Change multi_nullvals to true due to the full join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, true)); + else + /* Just reuse it */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + } + ctx->added_to_joinrel = true; + } + } + + if (innerrel_keeps_unique(root, outerrel, innerrel, clause_list, false)) + { + bool outer_impact = jointype == JOIN_FULL || jointype == JOIN_LEFT;; + + foreach(lc, innerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + if (inner_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (inner_onerow) + { + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(joinrel, innerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Need to change multi_nullvals to true due to the outer join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, + true)); + else + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + + } + ctx->added_to_joinrel = true; + } + } + + /* + * The combination of the UniqueKey from both sides is unique as well regardless + * of join type, but no bother to add it if its subset has been added to joinrel + * already or it is not useful for the joinrel. + */ + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx1 = (UniqueKeyContext) lfirst(lc); + if (ctx1->added_to_joinrel || !ctx1->useful) + continue; + foreach(lc2, innerrel_ukey_ctx) + { + UniqueKeyContext ctx2 = (UniqueKeyContext) lfirst(lc2); + if (ctx2->added_to_joinrel || !ctx2->useful) + continue; + if (add_combined_uniquekey(joinrel, outerrel, innerrel, + ctx1->uniquekey, ctx2->uniquekey, + jointype)) + /* If we set a onerow UniqueKey to joinrel, we don't need other. */ + return; + } + } +} + + +/* + * convert_subquery_uniquekeys + * + * Covert the UniqueKey in subquery to outer relation. + */ +void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel) +{ + ListCell *lc; + + if (sub_final_rel->uniquekeys == NIL) + return; + + if (relation_is_onerow(sub_final_rel)) + { + add_uniquekey_for_onerow(currel); + return; + } + + Assert(currel->subroot != NULL); + + foreach(lc, sub_final_rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + ListCell *lc; + List *exprs = NIL; + bool ukey_useful = true; + + /* One row case is handled above */ + Assert(ukey->exprs != NIL); + foreach(lc, ukey->exprs) + { + Var *var; + TargetEntry *tle = tlist_member(lfirst(lc), + currel->subroot->processed_tlist); + if (tle == NULL) + { + ukey_useful = false; + break; + } + var = find_var_for_subquery_tle(currel, tle); + if (var == NULL) + { + ukey_useful = false; + break; + } + exprs = lappend(exprs, var); + } + + if (ukey_useful) + currel->uniquekeys = lappend(currel->uniquekeys, + makeUniqueKey(exprs, + ukey->multi_nullvals)); + + } +} + +/* + * innerrel_keeps_unique + * + * Check if Unique key of the innerrel is valid after join. innerrel's UniqueKey + * will be still valid if innerrel's any-column mergeop outrerel's uniquekey + * exists in clause_list. + * + * Note: the clause_list must be a list of mergeable restrictinfo already. + */ +static bool +innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *clause_list, + bool reverse) +{ + ListCell *lc, *lc2, *lc3; + + if (outerrel->uniquekeys == NIL || innerrel->uniquekeys == NIL) + return false; + + /* Check if there is outerrel's uniquekey in mergeable clause. */ + foreach(lc, outerrel->uniquekeys) + { + List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs; + bool clauselist_matchs_all_exprs = true; + foreach(lc2, outer_uq_exprs) + { + Node *outer_uq_expr = lfirst(lc2); + bool find_uq_expr_in_clauselist = false; + foreach(lc3, clause_list) + { + RestrictInfo *rinfo = lfirst_node(RestrictInfo, lc3); + Node *outer_expr; + if (reverse) + outer_expr = rinfo->outer_is_left ? get_rightop(rinfo->clause) : get_leftop(rinfo->clause); + else + outer_expr = rinfo->outer_is_left ? get_leftop(rinfo->clause) : get_rightop(rinfo->clause); + if (equal(outer_expr, outer_uq_expr)) + { + find_uq_expr_in_clauselist = true; + break; + } + } + if (!find_uq_expr_in_clauselist) + { + /* No need to check the next exprs in the current uniquekey */ + clauselist_matchs_all_exprs = false; + break; + } + } + + if (clauselist_matchs_all_exprs) + return true; + } + return false; +} + + +/* + * relation_is_onerow + * Check if it is a one-row relation by checking UniqueKey. + */ +bool +relation_is_onerow(RelOptInfo *rel) +{ + UniqueKey *ukey; + if (rel->uniquekeys == NIL) + return false; + ukey = linitial_node(UniqueKey, rel->uniquekeys); + return ukey->exprs == NIL && list_length(rel->uniquekeys) == 1; +} + +/* + * relation_has_uniquekeys_for + * Returns true if we have proofs that 'rel' cannot return multiple rows with + * the same values in each of 'exprs'. Otherwise returns false. + */ +bool +relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel, + List *exprs, bool allow_multinulls) +{ + ListCell *lc; + + /* + * For UniqueKey->onerow case, the uniquekey->exprs is empty as well + * so we can't rely on list_is_subset to handle this special cases + */ + if (exprs == NIL) + return false; + + foreach(lc, rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (ukey->multi_nullvals && !allow_multinulls) + continue; + if (list_is_subset(ukey->exprs, exprs)) + return true; + } + return false; +} + + +/* + * get_exprs_from_uniqueindex + * + * Return a list of exprs which is unique. set useful to false if this + * unique index is not useful for us. + */ +static List * +get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals) +{ + List *exprs = NIL; + ListCell *indexpr_item; + int c = 0; + + *useful = true; + *multi_nullvals = false; + + indexpr_item = list_head(unique_index->indexprs); + for(c = 0; c < unique_index->ncolumns; c++) + { + int attr = unique_index->indexkeys[c]; + Expr *expr; + bool matched_const = false; + ListCell *lc1, *lc2; + + if(attr > 0) + { + expr = list_nth_node(TargetEntry, unique_index->indextlist, c)->expr; + } + else if (attr == 0) + { + /* Expression index */ + expr = lfirst(indexpr_item); + indexpr_item = lnext(unique_index->indexprs, indexpr_item); + } + else /* attr < 0 */ + { + /* Index on system column is not supported */ + Assert(false); + } + + /* + * Check index_col = Const case with regarding to opfamily checking + * If we can remove the index_col from the final UniqueKey->exprs. + */ + forboth(lc1, const_exprs, lc2, const_expr_opfamilies) + { + if (list_member_oid((List *)lfirst(lc2), unique_index->opfamily[c]) + && match_index_to_operand((Node *) lfirst(lc1), c, unique_index)) + { + matched_const = true; + break; + } + } + + if (matched_const) + continue; + + /* Check if the indexed expr is used in rel */ + if (attr > 0) + { + /* + * Normal Indexed column, if the col is not used, then the index is useless + * for uniquekey. + */ + attr -= FirstLowInvalidHeapAttributeNumber; + + if (!bms_is_member(attr, used_varattrs)) + { + *useful = false; + break; + } + } + else if (!list_member(unique_index->rel->reltarget->exprs, expr)) + { + /* Expression index but the expression is not used in rel */ + *useful = false; + break; + } + + /* check not null property. */ + if (attr == 0) + { + /* We never know if a expression yields null or not */ + *multi_nullvals = true; + } + else if (!bms_is_member(attr, unique_index->rel->notnullattrs) + && !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber, + unique_index->rel->notnullattrs)) + { + *multi_nullvals = true; + } + + exprs = lappend(exprs, expr); + } + return exprs; +} + + +/* + * add_uniquekey_for_onerow + * If we are sure that the relation only returns one row, then all the columns + * are unique. However we don't need to create UniqueKey for every column, we + * just set exprs = NIL and overwrites all the other UniqueKey on this RelOptInfo + * since this one has strongest semantics. + */ +void +add_uniquekey_for_onerow(RelOptInfo *rel) +{ + /* + * We overwrite the previous UniqueKey on purpose since this one has the + * strongest semantic. + */ + rel->uniquekeys = list_make1(makeUniqueKey(NIL, false)); +} + + +/* + * initililze_uniquecontext_for_joinrel + * Return a List of UniqueKeyContext for an inputrel + */ +static List * +initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel) +{ + List *res = NIL; + ListCell *lc; + foreach(lc, inputrel->uniquekeys) + { + UniqueKeyContext context; + context = palloc(sizeof(struct UniqueKeyContextData)); + context->uniquekey = lfirst_node(UniqueKey, lc); + context->added_to_joinrel = false; + context->useful = true; + res = lappend(res, context); + } + return res; +} + + +/* + * get_exprs_from_uniquekey + * Unify the way of get List of exprs from a one-row UniqueKey or + * normal UniqueKey. for the onerow case, every expr in rel1 is a valid + * UniqueKey. Return a List of exprs. + * + * rel1: The relation which you want to get the exprs. + * ukey: The UniqueKey you want to get the exprs. + */ +static List * +get_exprs_from_uniquekey(RelOptInfo *joinrel, RelOptInfo *rel1, UniqueKey *ukey) +{ + ListCell *lc; + bool onerow = rel1 != NULL && relation_is_onerow(rel1); + + List *res = NIL; + Assert(onerow || ukey); + if (onerow) + { + /* Only cares about the exprs still exist in joinrel */ + foreach(lc, joinrel->reltarget->exprs) + { + Bitmapset *relids = pull_varnos(lfirst(lc)); + if (bms_is_subset(relids, rel1->relids)) + { + res = lappend(res, list_make1(lfirst(lc))); + } + } + } + else + { + res = list_make1(ukey->exprs); + } + return res; +} + +/* + * Partitioned table Unique Keys. + * The partition table unique key is maintained as: + * 1. The index must be unique as usual. + * 2. The index must contains partition key. + * 3. The index must exist on all the child rel. see simple_indexinfo_equal for + * how we compare it. + */ + +/* + * index_constains_partkey + * return true if the index contains the partiton key. + */ +static bool +index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind) +{ + ListCell *lc; + int i; + Assert(IS_PARTITIONED_REL(partrel)); + Assert(partrel->part_scheme->partnatts > 0); + + for(i = 0; i < partrel->part_scheme->partnatts; i++) + { + Node *part_expr = linitial(partrel->partexprs[i]); + bool found_in_index = false; + foreach(lc, ind->indextlist) + { + Expr *index_expr = lfirst_node(TargetEntry, lc)->expr; + if (equal(index_expr, part_expr)) + { + found_in_index = true; + break; + } + } + if (!found_in_index) + return false; + } + return true; +} + +/* + * simple_indexinfo_equal + * + * Used to check if the 2 index is same as each other. The index here + * is COPIED from childrel and did some tiny changes(see + * simple_copy_indexinfo_to_parent) + */ +static bool +simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2) +{ + Size oid_cmp_len = sizeof(Oid) * ind1->ncolumns; + + return ind1->ncolumns == ind2->ncolumns && + ind1->unique == ind2->unique && + memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0 && + memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0 && + memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0 && + memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0 && + equal(get_tlist_exprs(ind1->indextlist, true), + get_tlist_exprs(ind2->indextlist, true)); +} + + +/* + * The below macros are used for simple_copy_indexinfo_to_parent which is so + * customized that I don't want to put it to copyfuncs.c. So copy it here. + */ +#define COPY_POINTER_FIELD(fldname, sz) \ + do { \ + Size _size = (sz); \ + newnode->fldname = palloc(_size); \ + memcpy(newnode->fldname, from->fldname, _size); \ + } while (0) + +#define COPY_NODE_FIELD(fldname) \ + (newnode->fldname = copyObjectImpl(from->fldname)) + +#define COPY_SCALAR_FIELD(fldname) \ + (newnode->fldname = from->fldname) + + +/* + * simple_copy_indexinfo_to_parent (from partition) + * Copy the IndexInfo from child relation to parent relation with some modification, + * which is used to test: + * 1. If the same index exists in all the childrels. + * 2. If the parentrel->reltarget/basicrestrict info matches this index. + */ +static IndexOptInfo * +simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from) +{ + IndexOptInfo *newnode = makeNode(IndexOptInfo); + AppendRelInfo *appinfo = find_appinfo_by_child(root, from->rel->relid); + ListCell *lc; + int idx = 0; + + COPY_SCALAR_FIELD(ncolumns); + COPY_SCALAR_FIELD(nkeycolumns); + COPY_SCALAR_FIELD(unique); + COPY_SCALAR_FIELD(immediate); + /* We just need to know if it is NIL or not */ + COPY_SCALAR_FIELD(indpred); + COPY_SCALAR_FIELD(predOK); + COPY_POINTER_FIELD(indexkeys, from->ncolumns * sizeof(int)); + COPY_POINTER_FIELD(indexcollations, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opfamily, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opcintype, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(sortopfamily, from->ncolumns * sizeof(Oid)); + COPY_NODE_FIELD(indextlist); + + /* Convert index exprs on child expr to expr on parent */ + foreach(lc, newnode->indextlist) + { + TargetEntry *tle = lfirst_node(TargetEntry, lc); + /* Index on expression is ignored */ + Assert(IsA(tle->expr, Var)); + tle->expr = (Expr *) find_parent_var(appinfo, (Var *) tle->expr); + newnode->indexkeys[idx] = castNode(Var, tle->expr)->varattno; + idx++; + } + newnode->rel = parentrel; + return newnode; +} + +/* + * adjust_partition_unique_indexlist + * + * global_unique_indexes: At the beginning, it contains the copy & modified + * unique index from the first partition. And then check if each index in it still + * exists in the following partitions. If no, remove it. at last, it has an + * index list which exists in all the partitions. + */ +static void +adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_indexes) +{ + ListCell *lc, *lc2; + foreach(lc, *global_unique_indexes) + { + IndexOptInfo *g_ind = lfirst_node(IndexOptInfo, lc); + bool found_in_child = false; + + foreach(lc2, childrel->indexlist) + { + IndexOptInfo *p_ind = lfirst_node(IndexOptInfo, lc2); + IndexOptInfo *p_ind_copy; + if (!p_ind->unique || !p_ind->immediate || + (p_ind->indpred != NIL && !p_ind->predOK)) + continue; + p_ind_copy = simple_copy_indexinfo_to_parent(root, parentrel, p_ind); + if (simple_indexinfo_equal(p_ind_copy, g_ind)) + { + found_in_child = true; + break; + } + } + if (!found_in_child) + /* The index doesn't exist in childrel, remove it from global_unique_indexes */ + *global_unique_indexes = foreach_delete_current(*global_unique_indexes, lc); + } +} + +/* Helper function for groupres/distinctrel */ +static void +add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgroups) +{ + Query *parse = root->parse; + List *exprs; + + /* + * XXX: If there are some vars which is not in current levelsup, the semantic is + * imprecise, should we avoid it or not? levelsup = 1 is just a demo, maybe we need to + * check every level other than 0, if so, looks we have to write another + * pull_var_walker. + */ + List *upper_vars = pull_vars_of_level((Node*)sortgroups, 1); + + if (upper_vars != NIL) + return; + + exprs = get_sortgrouplist_exprs(sortgroups, parse->targetList); + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(exprs, + false /* sortgroupclause can't be multi_nullvals */)); +} + + +/* + * add_combined_uniquekey + * The combination of both UniqueKeys is a valid UniqueKey for joinrel no matter + * the jointype. + */ +bool +add_combined_uniquekey(RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype) +{ + + ListCell *lc1, *lc2; + + /* Either side has multi_nullvals or we have outer join, + * the combined UniqueKey has multi_nullvals */ + bool multi_nullvals = outer_ukey->multi_nullvals || + inner_ukey->multi_nullvals || IS_OUTER_JOIN(jointype); + + /* The only case we can get onerow joinrel after join */ + if (relation_is_onerow(outer_rel) + && relation_is_onerow(inner_rel) + && jointype == JOIN_INNER) + { + add_uniquekey_for_onerow(joinrel); + return true; + } + + foreach(lc1, get_exprs_from_uniquekey(joinrel, outer_rel, outer_ukey)) + { + foreach(lc2, get_exprs_from_uniquekey(joinrel, inner_rel, inner_ukey)) + { + List *exprs = list_concat_copy(lfirst_node(List, lc1), lfirst_node(List, lc2)); + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(exprs, + multi_nullvals)); + } + } + return false; +} diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index b406d41e91..0551ae0512 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -2389,6 +2389,8 @@ grouping_planner(PlannerInfo *root, bool inheritance_update, add_path(final_rel, path); } + simple_copy_uniquekeys(current_rel, final_rel); + /* * Generate partial paths for final_rel, too, if outer query levels might * be able to make use of them. @@ -3899,6 +3901,8 @@ create_grouping_paths(PlannerInfo *root, } set_cheapest(grouped_rel); + + populate_grouprel_uniquekeys(root, grouped_rel, input_rel); return grouped_rel; } @@ -4615,7 +4619,7 @@ create_window_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(window_rel); - + simple_copy_uniquekeys(input_rel, window_rel); return window_rel; } @@ -4911,7 +4915,7 @@ create_distinct_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(distinct_rel); - + populate_distinctrel_uniquekeys(root, input_rel, distinct_rel); return distinct_rel; } @@ -5172,6 +5176,8 @@ create_ordered_paths(PlannerInfo *root, */ Assert(ordered_rel->pathlist != NIL); + simple_copy_uniquekeys(input_rel, ordered_rel); + return ordered_rel; } @@ -6049,6 +6055,9 @@ adjust_paths_for_srfs(PlannerInfo *root, RelOptInfo *rel, if (list_length(targets) == 1) return; + /* UniqueKey is not valid after handling the SRF. */ + rel->uniquekeys = NIL; + /* * Stack SRF-evaluation nodes atop each path for the rel. * diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c index 951aed80e7..e94e92937c 100644 --- a/src/backend/optimizer/prep/prepunion.c +++ b/src/backend/optimizer/prep/prepunion.c @@ -689,6 +689,8 @@ generate_union_paths(SetOperationStmt *op, PlannerInfo *root, /* Undo effects of possibly forcing tuple_fraction to 0 */ root->tuple_fraction = save_fraction; + /* Add the UniqueKeys */ + populate_unionrel_uniquekeys(root, result_rel); return result_rel; } diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c index d722063cf3..44c37ecffc 100644 --- a/src/backend/optimizer/util/appendinfo.c +++ b/src/backend/optimizer/util/appendinfo.c @@ -746,3 +746,47 @@ find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos) } return appinfos; } + +/* + * find_appinfo_by_child + * + */ +AppendRelInfo * +find_appinfo_by_child(PlannerInfo *root, Index child_index) +{ + ListCell *lc; + foreach(lc, root->append_rel_list) + { + AppendRelInfo *appinfo = lfirst_node(AppendRelInfo, lc); + if (appinfo->child_relid == child_index) + return appinfo; + } + elog(ERROR, "parent relation cant be found"); + return NULL; +} + +/* + * find_parent_var + * + */ +Var * +find_parent_var(AppendRelInfo *appinfo, Var *child_var) +{ + ListCell *lc; + Var *res = NULL; + Index attno = 1; + foreach(lc, appinfo->translated_vars) + { + Node *child_node = lfirst(lc); + if (equal(child_node, child_var)) + { + res = copyObject(child_var); + res->varattno = attno; + res->varno = appinfo->parent_relid; + } + attno++; + } + if (res == NULL) + elog(ERROR, "parent var can't be found."); + return res; +} diff --git a/src/backend/optimizer/util/inherit.c b/src/backend/optimizer/util/inherit.c index 3132fd35a5..d66b40ec50 100644 --- a/src/backend/optimizer/util/inherit.c +++ b/src/backend/optimizer/util/inherit.c @@ -736,6 +736,7 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, { Node *onecq = (Node *) lfirst(lc2); bool pseudoconstant; + RestrictInfo *child_rinfo; /* check for pseudoconstant (no Vars or volatile functions) */ pseudoconstant = @@ -747,13 +748,14 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, root->hasPseudoConstantQuals = true; } /* reconstitute RestrictInfo with appropriate properties */ - childquals = lappend(childquals, - make_restrictinfo((Expr *) onecq, - rinfo->is_pushed_down, - rinfo->outerjoin_delayed, - pseudoconstant, - rinfo->security_level, - NULL, NULL, NULL)); + child_rinfo = make_restrictinfo((Expr *) onecq, + rinfo->is_pushed_down, + rinfo->outerjoin_delayed, + pseudoconstant, + rinfo->security_level, + NULL, NULL, NULL); + child_rinfo->mergeopfamilies = rinfo->mergeopfamilies; + childquals = lappend(childquals, child_rinfo); /* track minimum security level among child quals */ cq_min_security = Min(cq_min_security, rinfo->security_level); } diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 31d9aedeeb..c83f17acb7 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -16,6 +16,7 @@ #include "nodes/execnodes.h" #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" extern A_Expr *makeA_Expr(A_Expr_Kind kind, List *name, @@ -105,4 +106,6 @@ extern GroupingSet *makeGroupingSet(GroupingSetKind kind, List *content, int loc extern VacuumRelation *makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols); +extern UniqueKey* makeUniqueKey(List *exprs, bool multi_nullvals); + #endif /* MAKEFUNC_H */ diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 381d84b4e4..41110ed888 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -264,6 +264,7 @@ typedef enum NodeTag T_EquivalenceMember, T_PathKey, T_PathTarget, + T_UniqueKey, T_RestrictInfo, T_IndexClause, T_PlaceHolderVar, diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index 9e3ebd488a..02e4458bef 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -730,6 +730,7 @@ typedef struct RelOptInfo QualCost baserestrictcost; /* cost of evaluating the above */ Index baserestrict_min_security; /* min security_level found in * baserestrictinfo */ + List *uniquekeys; /* List of UniqueKey */ List *joininfo; /* RestrictInfo structures for join clauses * involving this rel */ bool has_eclass_joins; /* T means joininfo is incomplete */ @@ -1047,6 +1048,28 @@ typedef struct PathKey } PathKey; +/* + * UniqueKey + * + * Represents the unique properties held by a RelOptInfo. + * + * exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL + * is a special case of UniqueKey, which means there is only 1 row in that + * relation. + * multi_nullvals: true means multi null values may exist in these exprs, so the + * uniqueness is not guaranteed in this case. This field is necessary for + * remove_useless_join & reduce_unique_semijoins where we don't mind these + * duplicated NULL values. It is set to true for 2 cases. One is a unique key + * from a unique index but the related column is nullable. The other one is for + * outer join. see populate_joinrel_uniquekeys for detail. + */ +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + /* * PathTarget * @@ -2473,7 +2496,7 @@ typedef enum * * flags indicating what kinds of grouping are possible. * partial_costs_set is true if the agg_partial_costs and agg_final_costs - * have been initialized. + * have been initialized. * agg_partial_costs gives partial aggregation costs. * agg_final_costs gives finalization costs. * target_parallel_safe is true if target is parallel safe. @@ -2503,8 +2526,8 @@ typedef struct * limit_tuples is an estimated bound on the number of output tuples, * or -1 if no LIMIT or couldn't estimate. * count_est and offset_est are the estimated values of the LIMIT and OFFSET - * expressions computed by preprocess_limit() (see comments for - * preprocess_limit() for more information). + * expressions computed by preprocess_limit() (see comments for + * preprocess_limit() for more information). */ typedef struct { diff --git a/src/include/nodes/pg_list.h b/src/include/nodes/pg_list.h index 14ea2766ad..621f54a9f8 100644 --- a/src/include/nodes/pg_list.h +++ b/src/include/nodes/pg_list.h @@ -528,6 +528,8 @@ extern bool list_member_ptr(const List *list, const void *datum); extern bool list_member_int(const List *list, int datum); extern bool list_member_oid(const List *list, Oid datum); +extern bool list_is_subset(const List *members, const List *target); + extern List *list_delete(List *list, void *datum); extern List *list_delete_ptr(List *list, void *datum); extern List *list_delete_int(List *list, int datum); diff --git a/src/include/optimizer/appendinfo.h b/src/include/optimizer/appendinfo.h index d6a27a60dd..e87c92a054 100644 --- a/src/include/optimizer/appendinfo.h +++ b/src/include/optimizer/appendinfo.h @@ -32,4 +32,7 @@ extern Relids adjust_child_relids_multilevel(PlannerInfo *root, Relids relids, extern AppendRelInfo **find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos); +extern AppendRelInfo *find_appinfo_by_child(PlannerInfo *root, Index child_index); +extern Var *find_parent_var(AppendRelInfo *appinfo, Var *child_var); + #endif /* APPENDINFO_H */ diff --git a/src/include/optimizer/optimizer.h b/src/include/optimizer/optimizer.h index 3e4171056e..9445141263 100644 --- a/src/include/optimizer/optimizer.h +++ b/src/include/optimizer/optimizer.h @@ -23,6 +23,7 @@ #define OPTIMIZER_H #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" /* * We don't want to include nodes/pathnodes.h here, because non-planner @@ -156,6 +157,7 @@ extern TargetEntry *get_sortgroupref_tle(Index sortref, List *targetList); extern TargetEntry *get_sortgroupclause_tle(SortGroupClause *sgClause, List *targetList); +extern Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); extern Node *get_sortgroupclause_expr(SortGroupClause *sgClause, List *targetList); extern List *get_sortgrouplist_exprs(List *sgClauses, diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h index 10b6e81079..9217a8d6c6 100644 --- a/src/include/optimizer/paths.h +++ b/src/include/optimizer/paths.h @@ -240,5 +240,48 @@ extern PathKey *make_canonical_pathkey(PlannerInfo *root, int strategy, bool nulls_first); extern void add_paths_to_append_rel(PlannerInfo *root, RelOptInfo *rel, List *live_childrels); +extern List *select_mergejoin_clauses(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + JoinType jointype, + bool *mergejoin_allowed); + +/* + * uniquekeys.c + * Utilities for matching and building unique keys + */ +extern void populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List* unique_index_list); +extern void populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels); +extern void populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel); +extern void populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel); +extern void populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel); +extern void simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel); +extern void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel); +extern void populate_joinrel_uniquekeys(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + RelOptInfo *rel2, + List *restrictlist, + JoinType jointype); + +extern bool relation_has_uniquekeys_for(PlannerInfo *root, + RelOptInfo *rel, + List *exprs, + bool allow_multinulls); +extern bool relation_is_onerow(RelOptInfo *rel); #endif /* PATHS_H */ -- 2.21.0 --op7mf2i72xk5yu7c Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v36-0003-Extend-UniqueKeys.patch" ^ permalink raw reply [nested|flat] 91+ messages in thread
* [PATCH v38 2/6] Introduce UniqueKey attributes on RelOptInfo struct. @ 2020-05-11 07:50 一挃 <[email protected]> 0 siblings, 0 replies; 91+ messages in thread From: 一挃 @ 2020-05-11 07:50 UTC (permalink / raw) UniqueKey is a set of exprs on RelOptInfo which represents the exprs will be unique on the given RelOptInfo. You can see README.uniquekey for more information. --- src/backend/nodes/copyfuncs.c | 13 + src/backend/nodes/list.c | 31 + src/backend/nodes/makefuncs.c | 13 + src/backend/nodes/outfuncs.c | 11 + src/backend/nodes/readfuncs.c | 10 + src/backend/optimizer/path/Makefile | 3 +- src/backend/optimizer/path/README.uniquekey | 131 +++ src/backend/optimizer/path/allpaths.c | 10 + src/backend/optimizer/path/joinpath.c | 9 +- src/backend/optimizer/path/joinrels.c | 2 + src/backend/optimizer/path/pathkeys.c | 3 +- src/backend/optimizer/path/uniquekeys.c | 1135 +++++++++++++++++++ src/backend/optimizer/plan/planner.c | 13 +- src/backend/optimizer/prep/prepunion.c | 2 + src/backend/optimizer/util/appendinfo.c | 44 + src/backend/optimizer/util/inherit.c | 18 +- src/include/nodes/makefuncs.h | 3 + src/include/nodes/nodes.h | 1 + src/include/nodes/pathnodes.h | 29 +- src/include/nodes/pg_list.h | 10 + src/include/optimizer/appendinfo.h | 3 + src/include/optimizer/optimizer.h | 2 + src/include/optimizer/paths.h | 43 + 23 files changed, 1515 insertions(+), 24 deletions(-) create mode 100644 src/backend/optimizer/path/README.uniquekey create mode 100644 src/backend/optimizer/path/uniquekeys.c diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..75c1c5e824 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2296,6 +2296,16 @@ _copyPathKey(const PathKey *from) return newnode; } +static UniqueKey * +_copyUniqueKey(const UniqueKey *from) +{ + UniqueKey *newnode = makeNode(UniqueKey); + + COPY_NODE_FIELD(exprs); + COPY_SCALAR_FIELD(multi_nullvals); + + return newnode; +} /* * _copyRestrictInfo */ @@ -5220,6 +5230,9 @@ copyObjectImpl(const void *from) case T_PathKey: retval = _copyPathKey(from); break; + case T_UniqueKey: + retval = _copyUniqueKey(from); + break; case T_RestrictInfo: retval = _copyRestrictInfo(from); break; diff --git a/src/backend/nodes/list.c b/src/backend/nodes/list.c index dbf6b30233..ca099495a1 100644 --- a/src/backend/nodes/list.c +++ b/src/backend/nodes/list.c @@ -702,6 +702,37 @@ list_member_oid(const List *list, Oid datum) return false; } +/* + * return true iff every entry in "members" list is also present + * in the "target" list. + */ +bool +list_is_subset(const List *members, const List *target) +{ + const ListCell *lc1, *lc2; + + Assert(IsPointerList(members)); + Assert(IsPointerList(target)); + check_list_invariants(members); + check_list_invariants(target); + + foreach(lc1, members) + { + bool found = false; + foreach(lc2, target) + { + if (equal(lfirst(lc1), lfirst(lc2))) + { + found = true; + break; + } + } + if (!found) + return false; + } + return true; +} + /* * Delete the n'th cell (counting from 0) in list. * diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 01c110cd2f..40415d0f5b 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -815,3 +815,16 @@ makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols) v->va_cols = va_cols; return v; } + + +/* + * makeUniqueKey + */ +UniqueKey* +makeUniqueKey(List *exprs, bool multi_nullvals) +{ + UniqueKey * ukey = makeNode(UniqueKey); + ukey->exprs = exprs; + ukey->multi_nullvals = multi_nullvals; + return ukey; +} diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..44154cde6a 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2456,6 +2456,14 @@ _outPathKey(StringInfo str, const PathKey *node) WRITE_BOOL_FIELD(pk_nulls_first); } +static void +_outUniqueKey(StringInfo str, const UniqueKey *node) +{ + WRITE_NODE_TYPE("UNIQUEKEY"); + WRITE_NODE_FIELD(exprs); + WRITE_BOOL_FIELD(multi_nullvals); +} + static void _outPathTarget(StringInfo str, const PathTarget *node) { @@ -4198,6 +4206,9 @@ outNode(StringInfo str, const void *obj) case T_PathKey: _outPathKey(str, obj); break; + case T_UniqueKey: + _outUniqueKey(str, obj); + break; case T_PathTarget: _outPathTarget(str, obj); break; diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index c5e136e9c3..b3e212bf1c 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -492,6 +492,14 @@ _readSetOperationStmt(void) READ_DONE(); } +static UniqueKey * +_readUniqueKey(void) +{ + READ_LOCALS(UniqueKey); + READ_NODE_FIELD(exprs); + READ_BOOL_FIELD(multi_nullvals); + READ_DONE(); +} /* * Stuff from primnodes.h. @@ -2717,6 +2725,8 @@ parseNodeString(void) return_value = _readCommonTableExpr(); else if (MATCH("SETOPERATIONSTMT", 16)) return_value = _readSetOperationStmt(); + else if (MATCH("UNIQUEKEY", 9)) + return_value = _readUniqueKey(); else if (MATCH("ALIAS", 5)) return_value = _readAlias(); else if (MATCH("RANGEVAR", 8)) diff --git a/src/backend/optimizer/path/Makefile b/src/backend/optimizer/path/Makefile index 1e199ff66f..7b9820c25f 100644 --- a/src/backend/optimizer/path/Makefile +++ b/src/backend/optimizer/path/Makefile @@ -21,6 +21,7 @@ OBJS = \ joinpath.o \ joinrels.o \ pathkeys.o \ - tidpath.o + tidpath.o \ + uniquekeys.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/optimizer/path/README.uniquekey b/src/backend/optimizer/path/README.uniquekey new file mode 100644 index 0000000000..5eac761995 --- /dev/null +++ b/src/backend/optimizer/path/README.uniquekey @@ -0,0 +1,131 @@ +1. What is UniqueKey? +We can think UniqueKey is a set of exprs for a RelOptInfo, which we are insure +that doesn't yields same result among all the rows. The simplest UniqueKey +format is primary key. + +However we define the UnqiueKey as below. + +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + +exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL +is a special case of UniqueKey, which means there is only one row in that +relation.it has a stronger semantic than others. like SELECT uk FROM t; uk is +normal unique key and may have different values. SELECT colx FROM t WHERE uk = +const. colx is unique AND we have only 1 value. This field can used for +innerrel_is_unique. this logic is handled specially in add_uniquekey_for_onerow +function. + +multi_nullvals: true means multi null values may exist in these exprs, so the +uniqueness is not guaranteed in this case. This field is necessary for +remove_useless_join & reduce_unique_semijoins where we don't mind these +duplicated NULL values. It is set to true for 2 cases. One is a unique key +from a unique index but the related column is nullable. The other one is for +outer join. see populate_joinrel_uniquekeys for detail. + + +The UniqueKey can be used at the following cases at least: +1. remove_useless_joins. +2. reduce_semianti_joins +3. remove distinct node if distinct clause is unique. +4. remove aggnode if group by clause is unique. +5. Index Skip Scan (WIP) +6. Aggregation Push Down without 2 phase aggregation if the join can't + duplicated the aggregated rows. (work in progress feature) + +2. How is it maintained? + +We have a set of populate_xxx_unqiuekeys functions to maintain the uniquekey on +various cases. xxx includes baserel, joinrel, partitionedrel, distinctrel, +groupedrel, unionrel. and we also need to convert the uniquekey from subquery +to outer relation, which is what convert_subquery_uniquekeys does. + +1. The first part is about baserel. We handled 3 cases. suppose we have Unique +Index on (a, b). + +1. SELECT a, b FROM t. UniqueKey (a, b) +2. SELECT a FROM t WHERE b = 1; UniqueKey (a) +3. SELECT .. FROM t WHERE a = 1 AND b = 1; UniqueKey (NIL). onerow case, every + column is Unique. + +2. The next part is joinrel, this part is most error-prone, we simplified the rules +like below: +1. If the relation's UniqueKey can't be duplicated after join, then is will be + still valid for the join rel. The function we used here is + innerrel_keeps_unique. The basic idea is innerrel.any_col = outer.uk. + +2. If the UnqiueKey can't keep valid via the rule 1, the combination of the + UniqueKey from both sides are valid for sure. We can prove this as: if the + unique exprs from rel1 is duplicated by rel2, the duplicated rows must + contains different unique exprs from rel2. + +More considerations about onerow: +1. If relation with one row and it can't be duplicated, it is still possible + contains mulit_nullvas after outer join. +2. If the either UniqueKey can be duplicated after join, the can get one row + only when both side is one row AND there is no outer join. +3. Whenever the onerow UniqueKey is not a valid any more, we need to convert one + row UniqueKey to normal unique key since we don't store exprs for one-row + relation. get_exprs_from_uniquekeys will be used here. + + +More considerations about multi_nullvals after join: +1. If the original UnqiueKey has multi_nullvals, the final UniqueKey will have + mulit_nullvals in any case. +2. If a unique key doesn't allow mulit_nullvals, after some outer join, it + allows some outer join. + + +3. When we comes to subquery, we need to convert_subquery_unqiuekeys just like +convert_subquery_pathkeys. Only the UniqueKey insides subquery is referenced as +a Var in outer relation will be reused. The relationship between the outerrel.Var +and subquery.exprs is built with outerel->subroot->processed_tlist. + + +4. As for the SRF functions, it will break the uniqueness of uniquekey, However it +is handled in adjust_paths_for_srfs, which happens after the query_planner. so +we will maintain the UniqueKey until there and reset it to NIL at that +places. This can't help on distinct/group by elimination cases but probably help +in some other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is +semantic correctly. + + +5. As for inherit table, we first main the UnqiueKey on childrel as well. But for +partitioned table we need to maintain 2 different kinds of +UnqiueKey. 1). UniqueKey on the parent relation 2). UniqueKey on child +relation for partition wise query. + +Example: +CREATE TABLE p (a int not null, b int not null) partition by list (a); +CREATE TABLE p0 partition of p for values in (1); +CREATE TABLE p1 partition of p for values in (2); + +create unique index p0_b on p0(b); +create unique index p1_b on p1(b); + +Now b is only unique on partition level, so the distinct can't be removed on +the following cases. SELECT DISTINCT b FROM p; + +Another example is SELECT DISTINCT a, b FROM p WHERE a = 1; Since only one +partition is chosen, the UniqueKey on child relation is same as the UniqueKey on +parent relation. + +Another usage of UniqueKey on partition level is it be helpful for +partition-wise join. + +As for the UniqueKey on parent table level, it comes with 2 different ways, +1). the UniqueKey is also derived in UniqueKey index, but the index must be same +in all the related children relations and the unique index must contains +Partition Key in it. Example: + +CREATE UNIQUE INDEX p_ab ON p(a, b); -- where a is the partition key. + +-- Query +SELECT a, b FROM p; the (a, b) is a UniqueKey of p. + +2). If the parent relation has only one childrel, the UniqueKey on childrel is + the UniqueKey on parent as well. diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index 37b4223adb..d52ad59f89 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -579,6 +579,12 @@ set_plain_rel_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) */ check_index_predicates(root, rel); + /* + * Now that we've marked which partial indexes are suitable, we can now + * build the relation's unique keys. + */ + populate_baserel_uniquekeys(root, rel, rel->indexlist); + /* Mark rel with estimated output rows, width, etc */ set_baserel_size_estimates(root, rel); } @@ -1297,6 +1303,8 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, /* Add paths to the append relation. */ add_paths_to_append_rel(root, rel, live_childrels); + if (IS_PARTITIONED_REL(rel)) + populate_partitionedrel_uniquekeys(root, rel, live_childrels); } @@ -2304,6 +2312,8 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, pathkeys, required_outer)); } + convert_subquery_uniquekeys(root, rel, sub_final_rel); + /* If outer rel allows parallelism, do same for partial paths. */ if (rel->consider_parallel && bms_is_empty(required_outer)) { diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c index 57ce97fd53..697ac047fb 100644 --- a/src/backend/optimizer/path/joinpath.c +++ b/src/backend/optimizer/path/joinpath.c @@ -71,13 +71,6 @@ static void consider_parallel_mergejoin(PlannerInfo *root, static void hash_inner_and_outer(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, RelOptInfo *innerrel, JoinType jointype, JoinPathExtraData *extra); -static List *select_mergejoin_clauses(PlannerInfo *root, - RelOptInfo *joinrel, - RelOptInfo *outerrel, - RelOptInfo *innerrel, - List *restrictlist, - JoinType jointype, - bool *mergejoin_allowed); static void generate_mergejoin_paths(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *innerrel, @@ -1927,7 +1920,7 @@ hash_inner_and_outer(PlannerInfo *root, * if it is mergejoinable and involves vars from the two sub-relations * currently of interest. */ -static List * +List * select_mergejoin_clauses(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c index 0dbe2ac726..7271f044ec 100644 --- a/src/backend/optimizer/path/joinrels.c +++ b/src/backend/optimizer/path/joinrels.c @@ -924,6 +924,8 @@ populate_joinrel_with_paths(PlannerInfo *root, RelOptInfo *rel1, /* Apply partitionwise join technique, if possible. */ try_partitionwise_join(root, rel1, rel2, joinrel, sjinfo, restrictlist); + + populate_joinrel_uniquekeys(root, joinrel, rel1, rel2, restrictlist, sjinfo->jointype); } diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index bd9a176d7d..139278829b 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -33,7 +33,6 @@ static bool pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys); static bool matches_boolean_partition_clause(RestrictInfo *rinfo, RelOptInfo *partrel, int partkeycol); -static Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); static bool right_merge_direction(PlannerInfo *root, PathKey *pathkey); @@ -1035,7 +1034,7 @@ convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel, * We need this to ensure that we don't return pathkeys describing values * that are unavailable above the level of the subquery scan. */ -static Var * +Var * find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle) { ListCell *lc; diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c new file mode 100644 index 0000000000..77ed2b2eff --- /dev/null +++ b/src/backend/optimizer/path/uniquekeys.c @@ -0,0 +1,1135 @@ +/*------------------------------------------------------------------------- + * + * uniquekeys.c + * Utilities for matching and building unique keys + * + * Portions Copyright (c) 2020, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/optimizer/path/uniquekeys.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "nodes/makefuncs.h" +#include "nodes/nodeFuncs.h" +#include "optimizer/pathnode.h" +#include "optimizer/paths.h" +#include "optimizer/appendinfo.h" +#include "optimizer/optimizer.h" +#include "optimizer/tlist.h" +#include "rewrite/rewriteManip.h" + + +/* + * This struct is used to help populate_joinrel_uniquekeys. + * + * added_to_joinrel is true if a uniquekey (from outerrel or innerrel) + * has been added to joinrel. + * useful is true if the exprs of the uniquekey still appears in joinrel. + */ +typedef struct UniqueKeyContextData +{ + UniqueKey *uniquekey; + bool added_to_joinrel; + bool useful; +} *UniqueKeyContext; + +static List *initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel); +static bool innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + bool reverse); + +static List *get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals); +static List *get_exprs_from_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + UniqueKey *ukey); +static void add_uniquekey_for_onerow(RelOptInfo *rel); +static bool add_combined_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype); + +/* Used for unique indexes checking for partitioned table */ +static bool index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind); +static IndexOptInfo *simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from); +static bool simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2); +static void adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_index); + +/* Helper function for grouped relation and distinct relation. */ +static void add_uniquekey_from_sortgroups(PlannerInfo *root, + RelOptInfo *rel, + List *sortgroups); + +/* + * populate_baserel_uniquekeys + * Populate 'baserel' uniquekeys list by looking at the rel's unique index + * and baserestrictinfo + */ +void +populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List *indexlist) +{ + ListCell *lc; + List *matched_uniq_indexes = NIL; + + /* Attrs appears in rel->reltarget->exprs. */ + Bitmapset *used_attrs = NULL; + + List *const_exprs = NIL; + List *expr_opfamilies = NIL; + + Assert(baserel->rtekind == RTE_RELATION); + + foreach(lc, indexlist) + { + IndexOptInfo *ind = (IndexOptInfo *) lfirst(lc); + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + matched_uniq_indexes = lappend(matched_uniq_indexes, ind); + } + + if (matched_uniq_indexes == NIL) + return; + + /* Check which attrs is used in baserel->reltarget */ + pull_varattnos((Node *)baserel->reltarget->exprs, baserel->relid, &used_attrs); + + /* Check which attrno is used at a mergeable const filter */ + foreach(lc, baserel->baserestrictinfo) + { + RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); + + if (rinfo->mergeopfamilies == NIL) + continue; + + if (bms_is_empty(rinfo->left_relids)) + { + const_exprs = lappend(const_exprs, get_rightop(rinfo->clause)); + } + else if (bms_is_empty(rinfo->right_relids)) + { + const_exprs = lappend(const_exprs, get_leftop(rinfo->clause)); + } + else + continue; + + expr_opfamilies = lappend(expr_opfamilies, rinfo->mergeopfamilies); + } + + foreach(lc, matched_uniq_indexes) + { + bool multi_nullvals, useful; + List *exprs = get_exprs_from_uniqueindex(lfirst_node(IndexOptInfo, lc), + const_exprs, + expr_opfamilies, + used_attrs, + &useful, + &multi_nullvals); + if (useful) + { + if (exprs == NIL) + { + /* All the columns in Unique Index matched with a restrictinfo */ + add_uniquekey_for_onerow(baserel); + return; + } + baserel->uniquekeys = lappend(baserel->uniquekeys, + makeUniqueKey(exprs, multi_nullvals)); + } + } +} + + +/* + * populate_partitionedrel_uniquekeys + * The UniqueKey on partitionrel comes from 2 cases: + * 1). Only one partition is involved in this query, the unique key can be + * copied to parent rel from childrel. + * 2). There are some unique index which includes partition key and exists + * in all the related partitions. + * We never mind rule 2 if we hit rule 1. + */ + +void +populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels) +{ + ListCell *lc; + List *global_uniq_indexlist = NIL; + RelOptInfo *childrel; + bool is_first = true; + + Assert(IS_PARTITIONED_REL(rel)); + + if (childrels == NIL) + return; + + /* + * If there is only one partition used in this query, the UniqueKey in childrel is + * still valid in parent level, but we need convert the format from child expr to + * parent expr. + */ + if (list_length(childrels) == 1) + { + /* Check for Rule 1 */ + RelOptInfo *childrel = linitial_node(RelOptInfo, childrels); + ListCell *lc; + Assert(childrel->reloptkind == RELOPT_OTHER_MEMBER_REL); + if (relation_is_onerow(childrel)) + { + add_uniquekey_for_onerow(rel); + return; + } + + foreach(lc, childrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + AppendRelInfo *appinfo = find_appinfo_by_child(root, childrel->relid); + List *parent_exprs = NIL; + bool can_reuse = true; + ListCell *lc2; + foreach(lc2, ukey->exprs) + { + Var *var = (Var *)lfirst(lc2); + /* + * If the expr comes from a expression, it is hard to build the expression + * in parent so ignore that case for now. + */ + if(!IsA(var, Var)) + { + can_reuse = false; + break; + } + /* Convert it to parent var */ + parent_exprs = lappend(parent_exprs, find_parent_var(appinfo, var)); + } + if (can_reuse) + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(parent_exprs, + ukey->multi_nullvals)); + } + } + else + { + /* Check for rule 2 */ + childrel = linitial_node(RelOptInfo, childrels); + foreach(lc, childrel->indexlist) + { + IndexOptInfo *ind = lfirst(lc); + IndexOptInfo *modified_index; + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + + /* + * During simple_copy_indexinfo_to_parent, we need to convert var from + * child var to parent var, index on expression is too complex to handle. + * so ignore it for now. + */ + if (ind->indexprs != NIL) + continue; + + modified_index = simple_copy_indexinfo_to_parent(root, rel, ind); + /* + * If the unique index doesn't contain partkey, then it is unique + * on this partition only, so it is useless for us. + */ + if (!index_constains_partkey(rel, modified_index)) + continue; + + global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index); + } + + if (global_uniq_indexlist != NIL) + { + foreach(lc, childrels) + { + RelOptInfo *child = lfirst(lc); + if (is_first) + { + is_first = false; + continue; + } + adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist); + } + /* Now we have a list of unique index which are exactly same on all childrels, + * Set the UniqueKey just like it is non-partition table + */ + populate_baserel_uniquekeys(root, rel, global_uniq_indexlist); + } + } +} + + +/* + * populate_distinctrel_uniquekeys + */ +void +populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel) +{ + /* The unique key before the distinct is still valid. */ + distinctrel->uniquekeys = list_copy(inputrel->uniquekeys); + add_uniquekey_from_sortgroups(root, distinctrel, root->parse->distinctClause); +} + +/* + * populate_grouprel_uniquekeys + */ +void +populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel) + +{ + Query *parse = root->parse; + bool input_ukey_added = false; + ListCell *lc; + + if (relation_is_onerow(inputrel)) + { + add_uniquekey_for_onerow(grouprel); + return; + } + if (parse->groupingSets) + return; + + /* A Normal group by without grouping set. */ + if (parse->groupClause) + { + /* + * Current even the groupby clause is Unique already, but if query has aggref + * We have to create grouprel still. To keep the UnqiueKey short, we will check + * the UniqueKey of input_rel still valid, if so we reuse it. + */ + foreach(lc, inputrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (list_is_subset(ukey->exprs, grouprel->reltarget->exprs)) + { + grouprel->uniquekeys = lappend(grouprel->uniquekeys, + ukey); + input_ukey_added = true; + } + } + if (!input_ukey_added) + /* + * group by clause must be a super-set of grouprel->reltarget->exprs except the + * aggregation expr, so if such exprs is unique already, no bother to generate + * new uniquekey for group by exprs. + */ + add_uniquekey_from_sortgroups(root, + grouprel, + root->parse->groupClause); + } + else + /* It has aggregation but without a group by, so only one row returned */ + add_uniquekey_for_onerow(grouprel); +} + +/* + * simple_copy_uniquekeys + * Using a function for the one-line code makes us easy to check where we simply + * copied the uniquekey. + */ +void +simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel) +{ + newrel->uniquekeys = oldrel->uniquekeys; +} + +/* + * populate_unionrel_uniquekeys + */ +void +populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel) +{ + ListCell *lc; + List *exprs = NIL; + + Assert(unionrel->uniquekeys == NIL); + + foreach(lc, unionrel->reltarget->exprs) + { + exprs = lappend(exprs, lfirst(lc)); + } + + if (exprs == NIL) + /* SQL: select union select; is valid, we need to handle it here. */ + add_uniquekey_for_onerow(unionrel); + else + unionrel->uniquekeys = lappend(unionrel->uniquekeys, + makeUniqueKey(exprs,false)); + +} + +/* + * populate_joinrel_uniquekeys + * + * populate uniquekeys for joinrel. We will check each relation to see if its + * UniqueKey is still valid via innerrel_keeps_unique, if so, we add it to + * joinrel. The multi_nullvals field will be changed to true for some outer + * join cases and one-row UniqueKey needs to be converted to normal UniqueKey + * for the same case as well. + * For the uniquekey in either baserel which can't be unique after join, we still + * check to see if combination of UniqueKeys from both side is still useful for us. + * if yes, we add it to joinrel as well. + */ +void +populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *outerrel, RelOptInfo *innerrel, + List *restrictlist, JoinType jointype) +{ + ListCell *lc, *lc2; + List *clause_list = NIL; + List *outerrel_ukey_ctx; + List *innerrel_ukey_ctx; + bool inner_onerow, outer_onerow; + bool mergejoin_allowed; + + /* Care about the outerrel relation only for SEMI/ANTI join */ + if (jointype == JOIN_SEMI || jointype == JOIN_ANTI) + { + foreach(lc, outerrel->uniquekeys) + { + UniqueKey *uniquekey = lfirst_node(UniqueKey, lc); + if (list_is_subset(uniquekey->exprs, joinrel->reltarget->exprs)) + joinrel->uniquekeys = lappend(joinrel->uniquekeys, uniquekey); + } + return; + } + + Assert(jointype == JOIN_LEFT || jointype == JOIN_FULL || jointype == JOIN_INNER); + + /* Fast path */ + if (innerrel->uniquekeys == NIL || outerrel->uniquekeys == NIL) + return; + + inner_onerow = relation_is_onerow(innerrel); + outer_onerow = relation_is_onerow(outerrel); + + outerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(outerrel); + innerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(innerrel); + + clause_list = select_mergejoin_clauses(root, joinrel, outerrel, innerrel, + restrictlist, jointype, + &mergejoin_allowed); + + if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true /* reverse */)) + { + bool outer_impact = jointype == JOIN_FULL; + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + /* Outer relation has one row, and the unique key is not duplicated after join, + * the joinrel will still has one row unless the jointype == JOIN_FULL. + */ + if (outer_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (outer_onerow) + { + /* + * The onerow outerrel becomes multi rows and multi_nullvals + * will be changed to true. We also need to set the exprs correctly since it + * can't be NIL any more. + */ + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, outerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Change multi_nullvals to true due to the full join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, true)); + else + /* Just reuse it */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + } + ctx->added_to_joinrel = true; + } + } + + if (innerrel_keeps_unique(root, outerrel, innerrel, clause_list, false)) + { + bool outer_impact = jointype == JOIN_FULL || jointype == JOIN_LEFT;; + + foreach(lc, innerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + if (inner_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (inner_onerow) + { + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, innerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Need to change multi_nullvals to true due to the outer join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, + true)); + else + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + + } + ctx->added_to_joinrel = true; + } + } + + /* + * The combination of the UniqueKey from both sides is unique as well regardless + * of join type, but no bother to add it if its subset has been added to joinrel + * already or it is not useful for the joinrel. + */ + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx1 = (UniqueKeyContext) lfirst(lc); + if (ctx1->added_to_joinrel || !ctx1->useful) + continue; + foreach(lc2, innerrel_ukey_ctx) + { + UniqueKeyContext ctx2 = (UniqueKeyContext) lfirst(lc2); + if (ctx2->added_to_joinrel || !ctx2->useful) + continue; + if (add_combined_uniquekey(root, joinrel, outerrel, innerrel, + ctx1->uniquekey, ctx2->uniquekey, + jointype)) + /* If we set a onerow UniqueKey to joinrel, we don't need other. */ + return; + } + } +} + + +/* + * convert_subquery_uniquekeys + * + * Covert the UniqueKey in subquery to outer relation. + */ +void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel) +{ + ListCell *lc; + + if (sub_final_rel->uniquekeys == NIL) + return; + + if (relation_is_onerow(sub_final_rel)) + { + add_uniquekey_for_onerow(currel); + return; + } + + Assert(currel->subroot != NULL); + + foreach(lc, sub_final_rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + ListCell *lc; + List *exprs = NIL; + bool ukey_useful = true; + + /* One row case is handled above */ + Assert(ukey->exprs != NIL); + foreach(lc, ukey->exprs) + { + Var *var; + TargetEntry *tle = tlist_member(lfirst(lc), + currel->subroot->processed_tlist); + if (tle == NULL) + { + ukey_useful = false; + break; + } + var = find_var_for_subquery_tle(currel, tle); + if (var == NULL) + { + ukey_useful = false; + break; + } + exprs = lappend(exprs, var); + } + + if (ukey_useful) + currel->uniquekeys = lappend(currel->uniquekeys, + makeUniqueKey(exprs, + ukey->multi_nullvals)); + + } +} + +/* + * innerrel_keeps_unique + * + * Check if Unique key of the innerrel is valid after join. innerrel's UniqueKey + * will be still valid if innerrel's any-column mergeop outrerel's uniquekey + * exists in clause_list. + * + * Note: the clause_list must be a list of mergeable restrictinfo already. + */ +static bool +innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *clause_list, + bool reverse) +{ + ListCell *lc, *lc2, *lc3; + + if (outerrel->uniquekeys == NIL || innerrel->uniquekeys == NIL) + return false; + + /* Check if there is outerrel's uniquekey in mergeable clause. */ + foreach(lc, outerrel->uniquekeys) + { + List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs; + bool clauselist_matchs_all_exprs = true; + foreach(lc2, outer_uq_exprs) + { + Node *outer_uq_expr = lfirst(lc2); + bool find_uq_expr_in_clauselist = false; + foreach(lc3, clause_list) + { + RestrictInfo *rinfo = lfirst_node(RestrictInfo, lc3); + Node *outer_expr; + if (reverse) + outer_expr = rinfo->outer_is_left ? get_rightop(rinfo->clause) : get_leftop(rinfo->clause); + else + outer_expr = rinfo->outer_is_left ? get_leftop(rinfo->clause) : get_rightop(rinfo->clause); + if (equal(outer_expr, outer_uq_expr)) + { + find_uq_expr_in_clauselist = true; + break; + } + } + if (!find_uq_expr_in_clauselist) + { + /* No need to check the next exprs in the current uniquekey */ + clauselist_matchs_all_exprs = false; + break; + } + } + + if (clauselist_matchs_all_exprs) + return true; + } + return false; +} + + +/* + * relation_is_onerow + * Check if it is a one-row relation by checking UniqueKey. + */ +bool +relation_is_onerow(RelOptInfo *rel) +{ + UniqueKey *ukey; + if (rel->uniquekeys == NIL) + return false; + ukey = linitial_node(UniqueKey, rel->uniquekeys); + return ukey->exprs == NIL && list_length(rel->uniquekeys) == 1; +} + +/* + * relation_has_uniquekeys_for + * Returns true if we have proofs that 'rel' cannot return multiple rows with + * the same values in each of 'exprs'. Otherwise returns false. + */ +bool +relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel, + List *exprs, bool allow_multinulls) +{ + ListCell *lc; + + /* + * For UniqueKey->onerow case, the uniquekey->exprs is empty as well + * so we can't rely on list_is_subset to handle this special cases + */ + if (exprs == NIL) + return false; + + foreach(lc, rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (ukey->multi_nullvals && !allow_multinulls) + continue; + if (list_is_subset(ukey->exprs, exprs)) + return true; + } + return false; +} + + +/* + * get_exprs_from_uniqueindex + * + * Return a list of exprs which is unique. set useful to false if this + * unique index is not useful for us. + */ +static List * +get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals) +{ + List *exprs = NIL; + ListCell *indexpr_item; + int c = 0; + + *useful = true; + *multi_nullvals = false; + + indexpr_item = list_head(unique_index->indexprs); + for(c = 0; c < unique_index->ncolumns; c++) + { + int attr = unique_index->indexkeys[c]; + Expr *expr; + bool matched_const = false; + ListCell *lc1, *lc2; + + if(attr > 0) + { + expr = list_nth_node(TargetEntry, unique_index->indextlist, c)->expr; + } + else if (attr == 0) + { + /* Expression index */ + expr = lfirst(indexpr_item); + indexpr_item = lnext(unique_index->indexprs, indexpr_item); + } + else /* attr < 0 */ + { + /* Index on system column is not supported */ + Assert(false); + } + + /* + * Check index_col = Const case with regarding to opfamily checking + * If we can remove the index_col from the final UniqueKey->exprs. + */ + forboth(lc1, const_exprs, lc2, const_expr_opfamilies) + { + if (list_member_oid((List *)lfirst(lc2), unique_index->opfamily[c]) + && match_index_to_operand((Node *) lfirst(lc1), c, unique_index)) + { + matched_const = true; + break; + } + } + + if (matched_const) + continue; + + /* Check if the indexed expr is used in rel */ + if (attr > 0) + { + /* + * Normal Indexed column, if the col is not used, then the index is useless + * for uniquekey. + */ + attr -= FirstLowInvalidHeapAttributeNumber; + + if (!bms_is_member(attr, used_varattrs)) + { + *useful = false; + break; + } + } + else if (!list_member(unique_index->rel->reltarget->exprs, expr)) + { + /* Expression index but the expression is not used in rel */ + *useful = false; + break; + } + + /* check not null property. */ + if (attr == 0) + { + /* We never know if a expression yields null or not */ + *multi_nullvals = true; + } + else if (!bms_is_member(attr, unique_index->rel->notnullattrs) + && !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber, + unique_index->rel->notnullattrs)) + { + *multi_nullvals = true; + } + + exprs = lappend(exprs, expr); + } + return exprs; +} + + +/* + * add_uniquekey_for_onerow + * If we are sure that the relation only returns one row, then all the columns + * are unique. However we don't need to create UniqueKey for every column, we + * just set exprs = NIL and overwrites all the other UniqueKey on this RelOptInfo + * since this one has strongest semantics. + */ +void +add_uniquekey_for_onerow(RelOptInfo *rel) +{ + /* + * We overwrite the previous UniqueKey on purpose since this one has the + * strongest semantic. + */ + rel->uniquekeys = list_make1(makeUniqueKey(NIL, false)); +} + + +/* + * initililze_uniquecontext_for_joinrel + * Return a List of UniqueKeyContext for an inputrel + */ +static List * +initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel) +{ + List *res = NIL; + ListCell *lc; + foreach(lc, inputrel->uniquekeys) + { + UniqueKeyContext context; + context = palloc(sizeof(struct UniqueKeyContextData)); + context->uniquekey = lfirst_node(UniqueKey, lc); + context->added_to_joinrel = false; + context->useful = true; + res = lappend(res, context); + } + return res; +} + + +/* + * get_exprs_from_uniquekey + * Unify the way of get List of exprs from a one-row UniqueKey or + * normal UniqueKey. for the onerow case, every expr in rel1 is a valid + * UniqueKey. Return a List of exprs. + * + * rel1: The relation which you want to get the exprs. + * ukey: The UniqueKey you want to get the exprs. + */ +static List * +get_exprs_from_uniquekey(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *rel1, UniqueKey *ukey) +{ + ListCell *lc; + bool onerow = rel1 != NULL && relation_is_onerow(rel1); + + List *res = NIL; + Assert(onerow || ukey); + if (onerow) + { + /* Only cares about the exprs still exist in joinrel */ + foreach(lc, joinrel->reltarget->exprs) + { + Bitmapset *relids = pull_varnos(root, lfirst(lc)); + if (bms_is_subset(relids, rel1->relids)) + { + res = lappend(res, list_make1(lfirst(lc))); + } + } + } + else + { + res = list_make1(ukey->exprs); + } + return res; +} + +/* + * Partitioned table Unique Keys. + * The partition table unique key is maintained as: + * 1. The index must be unique as usual. + * 2. The index must contains partition key. + * 3. The index must exist on all the child rel. see simple_indexinfo_equal for + * how we compare it. + */ + +/* + * index_constains_partkey + * return true if the index contains the partiton key. + */ +static bool +index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind) +{ + ListCell *lc; + int i; + Assert(IS_PARTITIONED_REL(partrel)); + Assert(partrel->part_scheme->partnatts > 0); + + for(i = 0; i < partrel->part_scheme->partnatts; i++) + { + Node *part_expr = linitial(partrel->partexprs[i]); + bool found_in_index = false; + foreach(lc, ind->indextlist) + { + Expr *index_expr = lfirst_node(TargetEntry, lc)->expr; + if (equal(index_expr, part_expr)) + { + found_in_index = true; + break; + } + } + if (!found_in_index) + return false; + } + return true; +} + +/* + * simple_indexinfo_equal + * + * Used to check if the 2 index is same as each other. The index here + * is COPIED from childrel and did some tiny changes(see + * simple_copy_indexinfo_to_parent) + */ +static bool +simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2) +{ + Size oid_cmp_len = sizeof(Oid) * ind1->ncolumns; + + return ind1->ncolumns == ind2->ncolumns && + ind1->unique == ind2->unique && + memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0 && + memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0 && + memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0 && + memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0 && + equal(get_tlist_exprs(ind1->indextlist, true), + get_tlist_exprs(ind2->indextlist, true)); +} + + +/* + * The below macros are used for simple_copy_indexinfo_to_parent which is so + * customized that I don't want to put it to copyfuncs.c. So copy it here. + */ +#define COPY_POINTER_FIELD(fldname, sz) \ + do { \ + Size _size = (sz); \ + newnode->fldname = palloc(_size); \ + memcpy(newnode->fldname, from->fldname, _size); \ + } while (0) + +#define COPY_NODE_FIELD(fldname) \ + (newnode->fldname = copyObjectImpl(from->fldname)) + +#define COPY_SCALAR_FIELD(fldname) \ + (newnode->fldname = from->fldname) + + +/* + * simple_copy_indexinfo_to_parent (from partition) + * Copy the IndexInfo from child relation to parent relation with some modification, + * which is used to test: + * 1. If the same index exists in all the childrels. + * 2. If the parentrel->reltarget/basicrestrict info matches this index. + */ +static IndexOptInfo * +simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from) +{ + IndexOptInfo *newnode = makeNode(IndexOptInfo); + AppendRelInfo *appinfo = find_appinfo_by_child(root, from->rel->relid); + ListCell *lc; + int idx = 0; + + COPY_SCALAR_FIELD(ncolumns); + COPY_SCALAR_FIELD(nkeycolumns); + COPY_SCALAR_FIELD(unique); + COPY_SCALAR_FIELD(immediate); + /* We just need to know if it is NIL or not */ + COPY_SCALAR_FIELD(indpred); + COPY_SCALAR_FIELD(predOK); + COPY_POINTER_FIELD(indexkeys, from->ncolumns * sizeof(int)); + COPY_POINTER_FIELD(indexcollations, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opfamily, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opcintype, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(sortopfamily, from->ncolumns * sizeof(Oid)); + COPY_NODE_FIELD(indextlist); + + /* Convert index exprs on child expr to expr on parent */ + foreach(lc, newnode->indextlist) + { + TargetEntry *tle = lfirst_node(TargetEntry, lc); + /* Index on expression is ignored */ + Assert(IsA(tle->expr, Var)); + tle->expr = (Expr *) find_parent_var(appinfo, (Var *) tle->expr); + newnode->indexkeys[idx] = castNode(Var, tle->expr)->varattno; + idx++; + } + newnode->rel = parentrel; + return newnode; +} + +/* + * adjust_partition_unique_indexlist + * + * global_unique_indexes: At the beginning, it contains the copy & modified + * unique index from the first partition. And then check if each index in it still + * exists in the following partitions. If no, remove it. at last, it has an + * index list which exists in all the partitions. + */ +static void +adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_indexes) +{ + ListCell *lc, *lc2; + foreach(lc, *global_unique_indexes) + { + IndexOptInfo *g_ind = lfirst_node(IndexOptInfo, lc); + bool found_in_child = false; + + foreach(lc2, childrel->indexlist) + { + IndexOptInfo *p_ind = lfirst_node(IndexOptInfo, lc2); + IndexOptInfo *p_ind_copy; + if (!p_ind->unique || !p_ind->immediate || + (p_ind->indpred != NIL && !p_ind->predOK)) + continue; + p_ind_copy = simple_copy_indexinfo_to_parent(root, parentrel, p_ind); + if (simple_indexinfo_equal(p_ind_copy, g_ind)) + { + found_in_child = true; + break; + } + } + if (!found_in_child) + /* The index doesn't exist in childrel, remove it from global_unique_indexes */ + *global_unique_indexes = foreach_delete_current(*global_unique_indexes, lc); + } +} + +/* Helper function for groupres/distinctrel */ +static void +add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgroups) +{ + Query *parse = root->parse; + List *exprs; + + /* + * XXX: If there are some vars which is not in current levelsup, the semantic is + * imprecise, should we avoid it or not? levelsup = 1 is just a demo, maybe we need to + * check every level other than 0, if so, looks we have to write another + * pull_var_walker. + */ + List *upper_vars = pull_vars_of_level((Node*)sortgroups, 1); + + if (upper_vars != NIL) + return; + + exprs = get_sortgrouplist_exprs(sortgroups, parse->targetList); + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(exprs, + false /* sortgroupclause can't be multi_nullvals */)); +} + + +/* + * add_combined_uniquekey + * The combination of both UniqueKeys is a valid UniqueKey for joinrel no matter + * the jointype. + */ +bool +add_combined_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype) +{ + + ListCell *lc1, *lc2; + + /* Either side has multi_nullvals or we have outer join, + * the combined UniqueKey has multi_nullvals */ + bool multi_nullvals = outer_ukey->multi_nullvals || + inner_ukey->multi_nullvals || IS_OUTER_JOIN(jointype); + + /* The only case we can get onerow joinrel after join */ + if (relation_is_onerow(outer_rel) + && relation_is_onerow(inner_rel) + && jointype == JOIN_INNER) + { + add_uniquekey_for_onerow(joinrel); + return true; + } + + foreach(lc1, get_exprs_from_uniquekey(root, joinrel, outer_rel, outer_ukey)) + { + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, inner_rel, inner_ukey)) + { + List *exprs = list_concat_copy(lfirst_node(List, lc1), lfirst_node(List, lc2)); + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(exprs, + multi_nullvals)); + } + } + return false; +} diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index 424d25cbd5..8d8e493f5c 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -2387,6 +2387,8 @@ grouping_planner(PlannerInfo *root, bool inheritance_update, add_path(final_rel, path); } + simple_copy_uniquekeys(current_rel, final_rel); + /* * Generate partial paths for final_rel, too, if outer query levels might * be able to make use of them. @@ -3899,6 +3901,8 @@ create_grouping_paths(PlannerInfo *root, } set_cheapest(grouped_rel); + + populate_grouprel_uniquekeys(root, grouped_rel, input_rel); return grouped_rel; } @@ -4621,7 +4625,7 @@ create_window_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(window_rel); - + simple_copy_uniquekeys(input_rel, window_rel); return window_rel; } @@ -4935,7 +4939,7 @@ create_distinct_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(distinct_rel); - + populate_distinctrel_uniquekeys(root, input_rel, distinct_rel); return distinct_rel; } @@ -5196,6 +5200,8 @@ create_ordered_paths(PlannerInfo *root, */ Assert(ordered_rel->pathlist != NIL); + simple_copy_uniquekeys(input_rel, ordered_rel); + return ordered_rel; } @@ -6073,6 +6079,9 @@ adjust_paths_for_srfs(PlannerInfo *root, RelOptInfo *rel, if (list_length(targets) == 1) return; + /* UniqueKey is not valid after handling the SRF. */ + rel->uniquekeys = NIL; + /* * Stack SRF-evaluation nodes atop each path for the rel. * diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c index becdcbb872..b7626545bf 100644 --- a/src/backend/optimizer/prep/prepunion.c +++ b/src/backend/optimizer/prep/prepunion.c @@ -689,6 +689,8 @@ generate_union_paths(SetOperationStmt *op, PlannerInfo *root, /* Undo effects of possibly forcing tuple_fraction to 0 */ root->tuple_fraction = save_fraction; + /* Add the UniqueKeys */ + populate_unionrel_uniquekeys(root, result_rel); return result_rel; } diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c index 86922a273c..6817c9c787 100644 --- a/src/backend/optimizer/util/appendinfo.c +++ b/src/backend/optimizer/util/appendinfo.c @@ -746,3 +746,47 @@ find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos) } return appinfos; } + +/* + * find_appinfo_by_child + * + */ +AppendRelInfo * +find_appinfo_by_child(PlannerInfo *root, Index child_index) +{ + ListCell *lc; + foreach(lc, root->append_rel_list) + { + AppendRelInfo *appinfo = lfirst_node(AppendRelInfo, lc); + if (appinfo->child_relid == child_index) + return appinfo; + } + elog(ERROR, "parent relation cant be found"); + return NULL; +} + +/* + * find_parent_var + * + */ +Var * +find_parent_var(AppendRelInfo *appinfo, Var *child_var) +{ + ListCell *lc; + Var *res = NULL; + Index attno = 1; + foreach(lc, appinfo->translated_vars) + { + Node *child_node = lfirst(lc); + if (equal(child_node, child_var)) + { + res = copyObject(child_var); + res->varattno = attno; + res->varno = appinfo->parent_relid; + } + attno++; + } + if (res == NULL) + elog(ERROR, "parent var can't be found."); + return res; +} diff --git a/src/backend/optimizer/util/inherit.c b/src/backend/optimizer/util/inherit.c index be1c9ddd96..3eec1f4d74 100644 --- a/src/backend/optimizer/util/inherit.c +++ b/src/backend/optimizer/util/inherit.c @@ -736,6 +736,7 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, { Node *onecq = (Node *) lfirst(lc2); bool pseudoconstant; + RestrictInfo *child_rinfo; /* check for pseudoconstant (no Vars or volatile functions) */ pseudoconstant = @@ -747,14 +748,15 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, root->hasPseudoConstantQuals = true; } /* reconstitute RestrictInfo with appropriate properties */ - childquals = lappend(childquals, - make_restrictinfo(root, - (Expr *) onecq, - rinfo->is_pushed_down, - rinfo->outerjoin_delayed, - pseudoconstant, - rinfo->security_level, - NULL, NULL, NULL)); + child_rinfo = make_restrictinfo(root, + (Expr *) onecq, + rinfo->is_pushed_down, + rinfo->outerjoin_delayed, + pseudoconstant, + rinfo->security_level, + NULL, NULL, NULL); + child_rinfo->mergeopfamilies = rinfo->mergeopfamilies; + childquals = lappend(childquals, child_rinfo); /* track minimum security level among child quals */ cq_min_security = Min(cq_min_security, rinfo->security_level); } diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 48a7ebfe45..4fe1824eb0 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -16,6 +16,7 @@ #include "nodes/execnodes.h" #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" extern A_Expr *makeA_Expr(A_Expr_Kind kind, List *name, @@ -106,4 +107,6 @@ extern GroupingSet *makeGroupingSet(GroupingSetKind kind, List *content, int loc extern VacuumRelation *makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols); +extern UniqueKey* makeUniqueKey(List *exprs, bool multi_nullvals); + #endif /* MAKEFUNC_H */ diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index e22df890ef..3320273ac1 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -265,6 +265,7 @@ typedef enum NodeTag T_EquivalenceMember, T_PathKey, T_PathTarget, + T_UniqueKey, T_RestrictInfo, T_IndexClause, T_PlaceHolderVar, diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index 0d61f04d27..80561b1003 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -739,6 +739,7 @@ typedef struct RelOptInfo QualCost baserestrictcost; /* cost of evaluating the above */ Index baserestrict_min_security; /* min security_level found in * baserestrictinfo */ + List *uniquekeys; /* List of UniqueKey */ List *joininfo; /* RestrictInfo structures for join clauses * involving this rel */ bool has_eclass_joins; /* T means joininfo is incomplete */ @@ -1059,6 +1060,28 @@ typedef struct PathKey } PathKey; +/* + * UniqueKey + * + * Represents the unique properties held by a RelOptInfo. + * + * exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL + * is a special case of UniqueKey, which means there is only 1 row in that + * relation. + * multi_nullvals: true means multi null values may exist in these exprs, so the + * uniqueness is not guaranteed in this case. This field is necessary for + * remove_useless_join & reduce_unique_semijoins where we don't mind these + * duplicated NULL values. It is set to true for 2 cases. One is a unique key + * from a unique index but the related column is nullable. The other one is for + * outer join. see populate_joinrel_uniquekeys for detail. + */ +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + /* * PathTarget * @@ -2496,7 +2519,7 @@ typedef enum * * flags indicating what kinds of grouping are possible. * partial_costs_set is true if the agg_partial_costs and agg_final_costs - * have been initialized. + * have been initialized. * agg_partial_costs gives partial aggregation costs. * agg_final_costs gives finalization costs. * target_parallel_safe is true if target is parallel safe. @@ -2526,8 +2549,8 @@ typedef struct * limit_tuples is an estimated bound on the number of output tuples, * or -1 if no LIMIT or couldn't estimate. * count_est and offset_est are the estimated values of the LIMIT and OFFSET - * expressions computed by preprocess_limit() (see comments for - * preprocess_limit() for more information). + * expressions computed by preprocess_limit() (see comments for + * preprocess_limit() for more information). */ typedef struct { diff --git a/src/include/nodes/pg_list.h b/src/include/nodes/pg_list.h index 404e03f132..85c65e910c 100644 --- a/src/include/nodes/pg_list.h +++ b/src/include/nodes/pg_list.h @@ -566,6 +566,16 @@ extern pg_nodiscard List *list_delete_first(List *list); extern pg_nodiscard List *list_delete_last(List *list); extern pg_nodiscard List *list_delete_nth_cell(List *list, int n); extern pg_nodiscard List *list_delete_cell(List *list, ListCell *cell); +extern bool list_is_subset(const List *members, const List *target); + +extern List *list_delete(List *list, void *datum); +extern List *list_delete_ptr(List *list, void *datum); +extern List *list_delete_int(List *list, int datum); +extern List *list_delete_oid(List *list, Oid datum); +extern List *list_delete_first(List *list); +extern List *list_delete_last(List *list); +extern List *list_delete_nth_cell(List *list, int n); +extern List *list_delete_cell(List *list, ListCell *cell); extern List *list_union(const List *list1, const List *list2); extern List *list_union_ptr(const List *list1, const List *list2); diff --git a/src/include/optimizer/appendinfo.h b/src/include/optimizer/appendinfo.h index 4cbf8c26cc..43b8d05f69 100644 --- a/src/include/optimizer/appendinfo.h +++ b/src/include/optimizer/appendinfo.h @@ -32,4 +32,7 @@ extern Relids adjust_child_relids_multilevel(PlannerInfo *root, Relids relids, extern AppendRelInfo **find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos); +extern AppendRelInfo *find_appinfo_by_child(PlannerInfo *root, Index child_index); +extern Var *find_parent_var(AppendRelInfo *appinfo, Var *child_var); + #endif /* APPENDINFO_H */ diff --git a/src/include/optimizer/optimizer.h b/src/include/optimizer/optimizer.h index d587952b7d..843aafa51e 100644 --- a/src/include/optimizer/optimizer.h +++ b/src/include/optimizer/optimizer.h @@ -23,6 +23,7 @@ #define OPTIMIZER_H #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" /* Test if an expression node represents a SRF call. Beware multiple eval! */ #define IS_SRF_CALL(node) \ @@ -168,6 +169,7 @@ extern TargetEntry *get_sortgroupref_tle(Index sortref, List *targetList); extern TargetEntry *get_sortgroupclause_tle(SortGroupClause *sgClause, List *targetList); +extern Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); extern Node *get_sortgroupclause_expr(SortGroupClause *sgClause, List *targetList); extern List *get_sortgrouplist_exprs(List *sgClauses, diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h index 035d3e1206..1adf99c9ee 100644 --- a/src/include/optimizer/paths.h +++ b/src/include/optimizer/paths.h @@ -247,5 +247,48 @@ extern PathKey *make_canonical_pathkey(PlannerInfo *root, int strategy, bool nulls_first); extern void add_paths_to_append_rel(PlannerInfo *root, RelOptInfo *rel, List *live_childrels); +extern List *select_mergejoin_clauses(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + JoinType jointype, + bool *mergejoin_allowed); + +/* + * uniquekeys.c + * Utilities for matching and building unique keys + */ +extern void populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List* unique_index_list); +extern void populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels); +extern void populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel); +extern void populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel); +extern void populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel); +extern void simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel); +extern void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel); +extern void populate_joinrel_uniquekeys(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + RelOptInfo *rel2, + List *restrictlist, + JoinType jointype); + +extern bool relation_has_uniquekeys_for(PlannerInfo *root, + RelOptInfo *rel, + List *exprs, + bool allow_multinulls); +extern bool relation_is_onerow(RelOptInfo *rel); #endif /* PATHS_H */ -- 2.26.2 --yh7cw345mbttadjz Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v38-0003-Extend-UniqueKeys.patch" ^ permalink raw reply [nested|flat] 91+ messages in thread
* [PATCH v36 2/6] Introduce UniqueKey attributes on RelOptInfo struct. @ 2020-05-11 07:50 一挃 <[email protected]> 0 siblings, 0 replies; 91+ messages in thread From: 一挃 @ 2020-05-11 07:50 UTC (permalink / raw) UniqueKey is a set of exprs on RelOptInfo which represents the exprs will be unique on the given RelOptInfo. You can see README.uniquekey for more information. --- src/backend/nodes/copyfuncs.c | 13 + src/backend/nodes/list.c | 31 + src/backend/nodes/makefuncs.c | 13 + src/backend/nodes/outfuncs.c | 11 + src/backend/nodes/readfuncs.c | 10 + src/backend/optimizer/path/Makefile | 3 +- src/backend/optimizer/path/README.uniquekey | 131 +++ src/backend/optimizer/path/allpaths.c | 10 + src/backend/optimizer/path/joinpath.c | 9 +- src/backend/optimizer/path/joinrels.c | 2 + src/backend/optimizer/path/pathkeys.c | 3 +- src/backend/optimizer/path/uniquekeys.c | 1131 +++++++++++++++++++ src/backend/optimizer/plan/planner.c | 13 +- src/backend/optimizer/prep/prepunion.c | 2 + src/backend/optimizer/util/appendinfo.c | 44 + src/backend/optimizer/util/inherit.c | 16 +- src/include/nodes/makefuncs.h | 3 + src/include/nodes/nodes.h | 1 + src/include/nodes/pathnodes.h | 29 +- src/include/nodes/pg_list.h | 2 + src/include/optimizer/appendinfo.h | 3 + src/include/optimizer/optimizer.h | 2 + src/include/optimizer/paths.h | 43 + 23 files changed, 1502 insertions(+), 23 deletions(-) create mode 100644 src/backend/optimizer/path/README.uniquekey create mode 100644 src/backend/optimizer/path/uniquekeys.c diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 89c409de66..1f50400fd2 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2273,6 +2273,16 @@ _copyPathKey(const PathKey *from) return newnode; } +static UniqueKey * +_copyUniqueKey(const UniqueKey *from) +{ + UniqueKey *newnode = makeNode(UniqueKey); + + COPY_NODE_FIELD(exprs); + COPY_SCALAR_FIELD(multi_nullvals); + + return newnode; +} /* * _copyRestrictInfo */ @@ -5152,6 +5162,9 @@ copyObjectImpl(const void *from) case T_PathKey: retval = _copyPathKey(from); break; + case T_UniqueKey: + retval = _copyUniqueKey(from); + break; case T_RestrictInfo: retval = _copyRestrictInfo(from); break; diff --git a/src/backend/nodes/list.c b/src/backend/nodes/list.c index 80fa8c84e4..a7a99b70f2 100644 --- a/src/backend/nodes/list.c +++ b/src/backend/nodes/list.c @@ -687,6 +687,37 @@ list_member_oid(const List *list, Oid datum) return false; } +/* + * return true iff every entry in "members" list is also present + * in the "target" list. + */ +bool +list_is_subset(const List *members, const List *target) +{ + const ListCell *lc1, *lc2; + + Assert(IsPointerList(members)); + Assert(IsPointerList(target)); + check_list_invariants(members); + check_list_invariants(target); + + foreach(lc1, members) + { + bool found = false; + foreach(lc2, target) + { + if (equal(lfirst(lc1), lfirst(lc2))) + { + found = true; + break; + } + } + if (!found) + return false; + } + return true; +} + /* * Delete the n'th cell (counting from 0) in list. * diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 49de285f01..646cf7c9a1 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -814,3 +814,16 @@ makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols) v->va_cols = va_cols; return v; } + + +/* + * makeUniqueKey + */ +UniqueKey* +makeUniqueKey(List *exprs, bool multi_nullvals) +{ + UniqueKey * ukey = makeNode(UniqueKey); + ukey->exprs = exprs; + ukey->multi_nullvals = multi_nullvals; + return ukey; +} diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index e2f177515d..c3a9632992 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2428,6 +2428,14 @@ _outPathKey(StringInfo str, const PathKey *node) WRITE_BOOL_FIELD(pk_nulls_first); } +static void +_outUniqueKey(StringInfo str, const UniqueKey *node) +{ + WRITE_NODE_TYPE("UNIQUEKEY"); + WRITE_NODE_FIELD(exprs); + WRITE_BOOL_FIELD(multi_nullvals); +} + static void _outPathTarget(StringInfo str, const PathTarget *node) { @@ -4127,6 +4135,9 @@ outNode(StringInfo str, const void *obj) case T_PathKey: _outPathKey(str, obj); break; + case T_UniqueKey: + _outUniqueKey(str, obj); + break; case T_PathTarget: _outPathTarget(str, obj); break; diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index 42050ab719..3a18571d0c 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -452,6 +452,14 @@ _readSetOperationStmt(void) READ_DONE(); } +static UniqueKey * +_readUniqueKey(void) +{ + READ_LOCALS(UniqueKey); + READ_NODE_FIELD(exprs); + READ_BOOL_FIELD(multi_nullvals); + READ_DONE(); +} /* * Stuff from primnodes.h. @@ -2656,6 +2664,8 @@ parseNodeString(void) return_value = _readCommonTableExpr(); else if (MATCH("SETOPERATIONSTMT", 16)) return_value = _readSetOperationStmt(); + else if (MATCH("UNIQUEKEY", 9)) + return_value = _readUniqueKey(); else if (MATCH("ALIAS", 5)) return_value = _readAlias(); else if (MATCH("RANGEVAR", 8)) diff --git a/src/backend/optimizer/path/Makefile b/src/backend/optimizer/path/Makefile index 1e199ff66f..7b9820c25f 100644 --- a/src/backend/optimizer/path/Makefile +++ b/src/backend/optimizer/path/Makefile @@ -21,6 +21,7 @@ OBJS = \ joinpath.o \ joinrels.o \ pathkeys.o \ - tidpath.o + tidpath.o \ + uniquekeys.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/optimizer/path/README.uniquekey b/src/backend/optimizer/path/README.uniquekey new file mode 100644 index 0000000000..5eac761995 --- /dev/null +++ b/src/backend/optimizer/path/README.uniquekey @@ -0,0 +1,131 @@ +1. What is UniqueKey? +We can think UniqueKey is a set of exprs for a RelOptInfo, which we are insure +that doesn't yields same result among all the rows. The simplest UniqueKey +format is primary key. + +However we define the UnqiueKey as below. + +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + +exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL +is a special case of UniqueKey, which means there is only one row in that +relation.it has a stronger semantic than others. like SELECT uk FROM t; uk is +normal unique key and may have different values. SELECT colx FROM t WHERE uk = +const. colx is unique AND we have only 1 value. This field can used for +innerrel_is_unique. this logic is handled specially in add_uniquekey_for_onerow +function. + +multi_nullvals: true means multi null values may exist in these exprs, so the +uniqueness is not guaranteed in this case. This field is necessary for +remove_useless_join & reduce_unique_semijoins where we don't mind these +duplicated NULL values. It is set to true for 2 cases. One is a unique key +from a unique index but the related column is nullable. The other one is for +outer join. see populate_joinrel_uniquekeys for detail. + + +The UniqueKey can be used at the following cases at least: +1. remove_useless_joins. +2. reduce_semianti_joins +3. remove distinct node if distinct clause is unique. +4. remove aggnode if group by clause is unique. +5. Index Skip Scan (WIP) +6. Aggregation Push Down without 2 phase aggregation if the join can't + duplicated the aggregated rows. (work in progress feature) + +2. How is it maintained? + +We have a set of populate_xxx_unqiuekeys functions to maintain the uniquekey on +various cases. xxx includes baserel, joinrel, partitionedrel, distinctrel, +groupedrel, unionrel. and we also need to convert the uniquekey from subquery +to outer relation, which is what convert_subquery_uniquekeys does. + +1. The first part is about baserel. We handled 3 cases. suppose we have Unique +Index on (a, b). + +1. SELECT a, b FROM t. UniqueKey (a, b) +2. SELECT a FROM t WHERE b = 1; UniqueKey (a) +3. SELECT .. FROM t WHERE a = 1 AND b = 1; UniqueKey (NIL). onerow case, every + column is Unique. + +2. The next part is joinrel, this part is most error-prone, we simplified the rules +like below: +1. If the relation's UniqueKey can't be duplicated after join, then is will be + still valid for the join rel. The function we used here is + innerrel_keeps_unique. The basic idea is innerrel.any_col = outer.uk. + +2. If the UnqiueKey can't keep valid via the rule 1, the combination of the + UniqueKey from both sides are valid for sure. We can prove this as: if the + unique exprs from rel1 is duplicated by rel2, the duplicated rows must + contains different unique exprs from rel2. + +More considerations about onerow: +1. If relation with one row and it can't be duplicated, it is still possible + contains mulit_nullvas after outer join. +2. If the either UniqueKey can be duplicated after join, the can get one row + only when both side is one row AND there is no outer join. +3. Whenever the onerow UniqueKey is not a valid any more, we need to convert one + row UniqueKey to normal unique key since we don't store exprs for one-row + relation. get_exprs_from_uniquekeys will be used here. + + +More considerations about multi_nullvals after join: +1. If the original UnqiueKey has multi_nullvals, the final UniqueKey will have + mulit_nullvals in any case. +2. If a unique key doesn't allow mulit_nullvals, after some outer join, it + allows some outer join. + + +3. When we comes to subquery, we need to convert_subquery_unqiuekeys just like +convert_subquery_pathkeys. Only the UniqueKey insides subquery is referenced as +a Var in outer relation will be reused. The relationship between the outerrel.Var +and subquery.exprs is built with outerel->subroot->processed_tlist. + + +4. As for the SRF functions, it will break the uniqueness of uniquekey, However it +is handled in adjust_paths_for_srfs, which happens after the query_planner. so +we will maintain the UniqueKey until there and reset it to NIL at that +places. This can't help on distinct/group by elimination cases but probably help +in some other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is +semantic correctly. + + +5. As for inherit table, we first main the UnqiueKey on childrel as well. But for +partitioned table we need to maintain 2 different kinds of +UnqiueKey. 1). UniqueKey on the parent relation 2). UniqueKey on child +relation for partition wise query. + +Example: +CREATE TABLE p (a int not null, b int not null) partition by list (a); +CREATE TABLE p0 partition of p for values in (1); +CREATE TABLE p1 partition of p for values in (2); + +create unique index p0_b on p0(b); +create unique index p1_b on p1(b); + +Now b is only unique on partition level, so the distinct can't be removed on +the following cases. SELECT DISTINCT b FROM p; + +Another example is SELECT DISTINCT a, b FROM p WHERE a = 1; Since only one +partition is chosen, the UniqueKey on child relation is same as the UniqueKey on +parent relation. + +Another usage of UniqueKey on partition level is it be helpful for +partition-wise join. + +As for the UniqueKey on parent table level, it comes with 2 different ways, +1). the UniqueKey is also derived in UniqueKey index, but the index must be same +in all the related children relations and the unique index must contains +Partition Key in it. Example: + +CREATE UNIQUE INDEX p_ab ON p(a, b); -- where a is the partition key. + +-- Query +SELECT a, b FROM p; the (a, b) is a UniqueKey of p. + +2). If the parent relation has only one childrel, the UniqueKey on childrel is + the UniqueKey on parent as well. diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index 484dab0a1a..2ad9d06d7a 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -579,6 +579,12 @@ set_plain_rel_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) */ check_index_predicates(root, rel); + /* + * Now that we've marked which partial indexes are suitable, we can now + * build the relation's unique keys. + */ + populate_baserel_uniquekeys(root, rel, rel->indexlist); + /* Mark rel with estimated output rows, width, etc */ set_baserel_size_estimates(root, rel); } @@ -1310,6 +1316,8 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, /* Add paths to the append relation. */ add_paths_to_append_rel(root, rel, live_childrels); + if (IS_PARTITIONED_REL(rel)) + populate_partitionedrel_uniquekeys(root, rel, live_childrels); } @@ -2383,6 +2391,8 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, pathkeys, required_outer)); } + convert_subquery_uniquekeys(root, rel, sub_final_rel); + /* If outer rel allows parallelism, do same for partial paths. */ if (rel->consider_parallel && bms_is_empty(required_outer)) { diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c index db54a6ba2e..ef0fd2fb0b 100644 --- a/src/backend/optimizer/path/joinpath.c +++ b/src/backend/optimizer/path/joinpath.c @@ -71,13 +71,6 @@ static void consider_parallel_mergejoin(PlannerInfo *root, static void hash_inner_and_outer(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, RelOptInfo *innerrel, JoinType jointype, JoinPathExtraData *extra); -static List *select_mergejoin_clauses(PlannerInfo *root, - RelOptInfo *joinrel, - RelOptInfo *outerrel, - RelOptInfo *innerrel, - List *restrictlist, - JoinType jointype, - bool *mergejoin_allowed); static void generate_mergejoin_paths(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *innerrel, @@ -1927,7 +1920,7 @@ hash_inner_and_outer(PlannerInfo *root, * if it is mergejoinable and involves vars from the two sub-relations * currently of interest. */ -static List * +List * select_mergejoin_clauses(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c index 2d343cd293..b9163ee8ff 100644 --- a/src/backend/optimizer/path/joinrels.c +++ b/src/backend/optimizer/path/joinrels.c @@ -924,6 +924,8 @@ populate_joinrel_with_paths(PlannerInfo *root, RelOptInfo *rel1, /* Apply partitionwise join technique, if possible. */ try_partitionwise_join(root, rel1, rel2, joinrel, sjinfo, restrictlist); + + populate_joinrel_uniquekeys(root, joinrel, rel1, rel2, restrictlist, sjinfo->jointype); } diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index ce9bf87e9b..7e596d4194 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -33,7 +33,6 @@ static bool pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys); static bool matches_boolean_partition_clause(RestrictInfo *rinfo, RelOptInfo *partrel, int partkeycol); -static Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); static bool right_merge_direction(PlannerInfo *root, PathKey *pathkey); @@ -1035,7 +1034,7 @@ convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel, * We need this to ensure that we don't return pathkeys describing values * that are unavailable above the level of the subquery scan. */ -static Var * +Var * find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle) { ListCell *lc; diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c new file mode 100644 index 0000000000..b33bcd2f32 --- /dev/null +++ b/src/backend/optimizer/path/uniquekeys.c @@ -0,0 +1,1131 @@ +/*------------------------------------------------------------------------- + * + * uniquekeys.c + * Utilities for matching and building unique keys + * + * Portions Copyright (c) 2020, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/optimizer/path/uniquekeys.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "nodes/makefuncs.h" +#include "nodes/nodeFuncs.h" +#include "optimizer/pathnode.h" +#include "optimizer/paths.h" +#include "optimizer/appendinfo.h" +#include "optimizer/optimizer.h" +#include "optimizer/tlist.h" +#include "rewrite/rewriteManip.h" + + +/* + * This struct is used to help populate_joinrel_uniquekeys. + * + * added_to_joinrel is true if a uniquekey (from outerrel or innerrel) + * has been added to joinrel. + * useful is true if the exprs of the uniquekey still appears in joinrel. + */ +typedef struct UniqueKeyContextData +{ + UniqueKey *uniquekey; + bool added_to_joinrel; + bool useful; +} *UniqueKeyContext; + +static List *initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel); +static bool innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + bool reverse); + +static List *get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals); +static List *get_exprs_from_uniquekey(RelOptInfo *joinrel, + RelOptInfo *rel1, + UniqueKey *ukey); +static void add_uniquekey_for_onerow(RelOptInfo *rel); +static bool add_combined_uniquekey(RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype); + +/* Used for unique indexes checking for partitioned table */ +static bool index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind); +static IndexOptInfo *simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from); +static bool simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2); +static void adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_index); + +/* Helper function for grouped relation and distinct relation. */ +static void add_uniquekey_from_sortgroups(PlannerInfo *root, + RelOptInfo *rel, + List *sortgroups); + +/* + * populate_baserel_uniquekeys + * Populate 'baserel' uniquekeys list by looking at the rel's unique index + * and baserestrictinfo + */ +void +populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List *indexlist) +{ + ListCell *lc; + List *matched_uniq_indexes = NIL; + + /* Attrs appears in rel->reltarget->exprs. */ + Bitmapset *used_attrs = NULL; + + List *const_exprs = NIL; + List *expr_opfamilies = NIL; + + Assert(baserel->rtekind == RTE_RELATION); + + foreach(lc, indexlist) + { + IndexOptInfo *ind = (IndexOptInfo *) lfirst(lc); + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + matched_uniq_indexes = lappend(matched_uniq_indexes, ind); + } + + if (matched_uniq_indexes == NIL) + return; + + /* Check which attrs is used in baserel->reltarget */ + pull_varattnos((Node *)baserel->reltarget->exprs, baserel->relid, &used_attrs); + + /* Check which attrno is used at a mergeable const filter */ + foreach(lc, baserel->baserestrictinfo) + { + RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); + + if (rinfo->mergeopfamilies == NIL) + continue; + + if (bms_is_empty(rinfo->left_relids)) + { + const_exprs = lappend(const_exprs, get_rightop(rinfo->clause)); + } + else if (bms_is_empty(rinfo->right_relids)) + { + const_exprs = lappend(const_exprs, get_leftop(rinfo->clause)); + } + else + continue; + + expr_opfamilies = lappend(expr_opfamilies, rinfo->mergeopfamilies); + } + + foreach(lc, matched_uniq_indexes) + { + bool multi_nullvals, useful; + List *exprs = get_exprs_from_uniqueindex(lfirst_node(IndexOptInfo, lc), + const_exprs, + expr_opfamilies, + used_attrs, + &useful, + &multi_nullvals); + if (useful) + { + if (exprs == NIL) + { + /* All the columns in Unique Index matched with a restrictinfo */ + add_uniquekey_for_onerow(baserel); + return; + } + baserel->uniquekeys = lappend(baserel->uniquekeys, + makeUniqueKey(exprs, multi_nullvals)); + } + } +} + + +/* + * populate_partitionedrel_uniquekeys + * The UniqueKey on partitionrel comes from 2 cases: + * 1). Only one partition is involved in this query, the unique key can be + * copied to parent rel from childrel. + * 2). There are some unique index which includes partition key and exists + * in all the related partitions. + * We never mind rule 2 if we hit rule 1. + */ + +void +populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels) +{ + ListCell *lc; + List *global_uniq_indexlist = NIL; + RelOptInfo *childrel; + bool is_first = true; + + Assert(IS_PARTITIONED_REL(rel)); + + if (childrels == NIL) + return; + + /* + * If there is only one partition used in this query, the UniqueKey in childrel is + * still valid in parent level, but we need convert the format from child expr to + * parent expr. + */ + if (list_length(childrels) == 1) + { + /* Check for Rule 1 */ + RelOptInfo *childrel = linitial_node(RelOptInfo, childrels); + ListCell *lc; + Assert(childrel->reloptkind == RELOPT_OTHER_MEMBER_REL); + if (relation_is_onerow(childrel)) + { + add_uniquekey_for_onerow(rel); + return; + } + + foreach(lc, childrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + AppendRelInfo *appinfo = find_appinfo_by_child(root, childrel->relid); + List *parent_exprs = NIL; + bool can_reuse = true; + ListCell *lc2; + foreach(lc2, ukey->exprs) + { + Var *var = (Var *)lfirst(lc2); + /* + * If the expr comes from a expression, it is hard to build the expression + * in parent so ignore that case for now. + */ + if(!IsA(var, Var)) + { + can_reuse = false; + break; + } + /* Convert it to parent var */ + parent_exprs = lappend(parent_exprs, find_parent_var(appinfo, var)); + } + if (can_reuse) + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(parent_exprs, + ukey->multi_nullvals)); + } + } + else + { + /* Check for rule 2 */ + childrel = linitial_node(RelOptInfo, childrels); + foreach(lc, childrel->indexlist) + { + IndexOptInfo *ind = lfirst(lc); + IndexOptInfo *modified_index; + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + + /* + * During simple_copy_indexinfo_to_parent, we need to convert var from + * child var to parent var, index on expression is too complex to handle. + * so ignore it for now. + */ + if (ind->indexprs != NIL) + continue; + + modified_index = simple_copy_indexinfo_to_parent(root, rel, ind); + /* + * If the unique index doesn't contain partkey, then it is unique + * on this partition only, so it is useless for us. + */ + if (!index_constains_partkey(rel, modified_index)) + continue; + + global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index); + } + + if (global_uniq_indexlist != NIL) + { + foreach(lc, childrels) + { + RelOptInfo *child = lfirst(lc); + if (is_first) + { + is_first = false; + continue; + } + adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist); + } + /* Now we have a list of unique index which are exactly same on all childrels, + * Set the UniqueKey just like it is non-partition table + */ + populate_baserel_uniquekeys(root, rel, global_uniq_indexlist); + } + } +} + + +/* + * populate_distinctrel_uniquekeys + */ +void +populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel) +{ + /* The unique key before the distinct is still valid. */ + distinctrel->uniquekeys = list_copy(inputrel->uniquekeys); + add_uniquekey_from_sortgroups(root, distinctrel, root->parse->distinctClause); +} + +/* + * populate_grouprel_uniquekeys + */ +void +populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel) + +{ + Query *parse = root->parse; + bool input_ukey_added = false; + ListCell *lc; + + if (relation_is_onerow(inputrel)) + { + add_uniquekey_for_onerow(grouprel); + return; + } + if (parse->groupingSets) + return; + + /* A Normal group by without grouping set. */ + if (parse->groupClause) + { + /* + * Current even the groupby clause is Unique already, but if query has aggref + * We have to create grouprel still. To keep the UnqiueKey short, we will check + * the UniqueKey of input_rel still valid, if so we reuse it. + */ + foreach(lc, inputrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (list_is_subset(ukey->exprs, grouprel->reltarget->exprs)) + { + grouprel->uniquekeys = lappend(grouprel->uniquekeys, + ukey); + input_ukey_added = true; + } + } + if (!input_ukey_added) + /* + * group by clause must be a super-set of grouprel->reltarget->exprs except the + * aggregation expr, so if such exprs is unique already, no bother to generate + * new uniquekey for group by exprs. + */ + add_uniquekey_from_sortgroups(root, + grouprel, + root->parse->groupClause); + } + else + /* It has aggregation but without a group by, so only one row returned */ + add_uniquekey_for_onerow(grouprel); +} + +/* + * simple_copy_uniquekeys + * Using a function for the one-line code makes us easy to check where we simply + * copied the uniquekey. + */ +void +simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel) +{ + newrel->uniquekeys = oldrel->uniquekeys; +} + +/* + * populate_unionrel_uniquekeys + */ +void +populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel) +{ + ListCell *lc; + List *exprs = NIL; + + Assert(unionrel->uniquekeys == NIL); + + foreach(lc, unionrel->reltarget->exprs) + { + exprs = lappend(exprs, lfirst(lc)); + } + + if (exprs == NIL) + /* SQL: select union select; is valid, we need to handle it here. */ + add_uniquekey_for_onerow(unionrel); + else + unionrel->uniquekeys = lappend(unionrel->uniquekeys, + makeUniqueKey(exprs,false)); + +} + +/* + * populate_joinrel_uniquekeys + * + * populate uniquekeys for joinrel. We will check each relation to see if its + * UniqueKey is still valid via innerrel_keeps_unique, if so, we add it to + * joinrel. The multi_nullvals field will be changed to true for some outer + * join cases and one-row UniqueKey needs to be converted to normal UniqueKey + * for the same case as well. + * For the uniquekey in either baserel which can't be unique after join, we still + * check to see if combination of UniqueKeys from both side is still useful for us. + * if yes, we add it to joinrel as well. + */ +void +populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *outerrel, RelOptInfo *innerrel, + List *restrictlist, JoinType jointype) +{ + ListCell *lc, *lc2; + List *clause_list = NIL; + List *outerrel_ukey_ctx; + List *innerrel_ukey_ctx; + bool inner_onerow, outer_onerow; + bool mergejoin_allowed; + + /* Care about the outerrel relation only for SEMI/ANTI join */ + if (jointype == JOIN_SEMI || jointype == JOIN_ANTI) + { + foreach(lc, outerrel->uniquekeys) + { + UniqueKey *uniquekey = lfirst_node(UniqueKey, lc); + if (list_is_subset(uniquekey->exprs, joinrel->reltarget->exprs)) + joinrel->uniquekeys = lappend(joinrel->uniquekeys, uniquekey); + } + return; + } + + Assert(jointype == JOIN_LEFT || jointype == JOIN_FULL || jointype == JOIN_INNER); + + /* Fast path */ + if (innerrel->uniquekeys == NIL || outerrel->uniquekeys == NIL) + return; + + inner_onerow = relation_is_onerow(innerrel); + outer_onerow = relation_is_onerow(outerrel); + + outerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(outerrel); + innerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(innerrel); + + clause_list = select_mergejoin_clauses(root, joinrel, outerrel, innerrel, + restrictlist, jointype, + &mergejoin_allowed); + + if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true /* reverse */)) + { + bool outer_impact = jointype == JOIN_FULL; + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + /* Outer relation has one row, and the unique key is not duplicated after join, + * the joinrel will still has one row unless the jointype == JOIN_FULL. + */ + if (outer_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (outer_onerow) + { + /* + * The onerow outerrel becomes multi rows and multi_nullvals + * will be changed to true. We also need to set the exprs correctly since it + * can't be NIL any more. + */ + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(joinrel, outerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Change multi_nullvals to true due to the full join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, true)); + else + /* Just reuse it */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + } + ctx->added_to_joinrel = true; + } + } + + if (innerrel_keeps_unique(root, outerrel, innerrel, clause_list, false)) + { + bool outer_impact = jointype == JOIN_FULL || jointype == JOIN_LEFT;; + + foreach(lc, innerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + if (inner_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (inner_onerow) + { + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(joinrel, innerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Need to change multi_nullvals to true due to the outer join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, + true)); + else + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + + } + ctx->added_to_joinrel = true; + } + } + + /* + * The combination of the UniqueKey from both sides is unique as well regardless + * of join type, but no bother to add it if its subset has been added to joinrel + * already or it is not useful for the joinrel. + */ + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx1 = (UniqueKeyContext) lfirst(lc); + if (ctx1->added_to_joinrel || !ctx1->useful) + continue; + foreach(lc2, innerrel_ukey_ctx) + { + UniqueKeyContext ctx2 = (UniqueKeyContext) lfirst(lc2); + if (ctx2->added_to_joinrel || !ctx2->useful) + continue; + if (add_combined_uniquekey(joinrel, outerrel, innerrel, + ctx1->uniquekey, ctx2->uniquekey, + jointype)) + /* If we set a onerow UniqueKey to joinrel, we don't need other. */ + return; + } + } +} + + +/* + * convert_subquery_uniquekeys + * + * Covert the UniqueKey in subquery to outer relation. + */ +void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel) +{ + ListCell *lc; + + if (sub_final_rel->uniquekeys == NIL) + return; + + if (relation_is_onerow(sub_final_rel)) + { + add_uniquekey_for_onerow(currel); + return; + } + + Assert(currel->subroot != NULL); + + foreach(lc, sub_final_rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + ListCell *lc; + List *exprs = NIL; + bool ukey_useful = true; + + /* One row case is handled above */ + Assert(ukey->exprs != NIL); + foreach(lc, ukey->exprs) + { + Var *var; + TargetEntry *tle = tlist_member(lfirst(lc), + currel->subroot->processed_tlist); + if (tle == NULL) + { + ukey_useful = false; + break; + } + var = find_var_for_subquery_tle(currel, tle); + if (var == NULL) + { + ukey_useful = false; + break; + } + exprs = lappend(exprs, var); + } + + if (ukey_useful) + currel->uniquekeys = lappend(currel->uniquekeys, + makeUniqueKey(exprs, + ukey->multi_nullvals)); + + } +} + +/* + * innerrel_keeps_unique + * + * Check if Unique key of the innerrel is valid after join. innerrel's UniqueKey + * will be still valid if innerrel's any-column mergeop outrerel's uniquekey + * exists in clause_list. + * + * Note: the clause_list must be a list of mergeable restrictinfo already. + */ +static bool +innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *clause_list, + bool reverse) +{ + ListCell *lc, *lc2, *lc3; + + if (outerrel->uniquekeys == NIL || innerrel->uniquekeys == NIL) + return false; + + /* Check if there is outerrel's uniquekey in mergeable clause. */ + foreach(lc, outerrel->uniquekeys) + { + List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs; + bool clauselist_matchs_all_exprs = true; + foreach(lc2, outer_uq_exprs) + { + Node *outer_uq_expr = lfirst(lc2); + bool find_uq_expr_in_clauselist = false; + foreach(lc3, clause_list) + { + RestrictInfo *rinfo = lfirst_node(RestrictInfo, lc3); + Node *outer_expr; + if (reverse) + outer_expr = rinfo->outer_is_left ? get_rightop(rinfo->clause) : get_leftop(rinfo->clause); + else + outer_expr = rinfo->outer_is_left ? get_leftop(rinfo->clause) : get_rightop(rinfo->clause); + if (equal(outer_expr, outer_uq_expr)) + { + find_uq_expr_in_clauselist = true; + break; + } + } + if (!find_uq_expr_in_clauselist) + { + /* No need to check the next exprs in the current uniquekey */ + clauselist_matchs_all_exprs = false; + break; + } + } + + if (clauselist_matchs_all_exprs) + return true; + } + return false; +} + + +/* + * relation_is_onerow + * Check if it is a one-row relation by checking UniqueKey. + */ +bool +relation_is_onerow(RelOptInfo *rel) +{ + UniqueKey *ukey; + if (rel->uniquekeys == NIL) + return false; + ukey = linitial_node(UniqueKey, rel->uniquekeys); + return ukey->exprs == NIL && list_length(rel->uniquekeys) == 1; +} + +/* + * relation_has_uniquekeys_for + * Returns true if we have proofs that 'rel' cannot return multiple rows with + * the same values in each of 'exprs'. Otherwise returns false. + */ +bool +relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel, + List *exprs, bool allow_multinulls) +{ + ListCell *lc; + + /* + * For UniqueKey->onerow case, the uniquekey->exprs is empty as well + * so we can't rely on list_is_subset to handle this special cases + */ + if (exprs == NIL) + return false; + + foreach(lc, rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (ukey->multi_nullvals && !allow_multinulls) + continue; + if (list_is_subset(ukey->exprs, exprs)) + return true; + } + return false; +} + + +/* + * get_exprs_from_uniqueindex + * + * Return a list of exprs which is unique. set useful to false if this + * unique index is not useful for us. + */ +static List * +get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals) +{ + List *exprs = NIL; + ListCell *indexpr_item; + int c = 0; + + *useful = true; + *multi_nullvals = false; + + indexpr_item = list_head(unique_index->indexprs); + for(c = 0; c < unique_index->ncolumns; c++) + { + int attr = unique_index->indexkeys[c]; + Expr *expr; + bool matched_const = false; + ListCell *lc1, *lc2; + + if(attr > 0) + { + expr = list_nth_node(TargetEntry, unique_index->indextlist, c)->expr; + } + else if (attr == 0) + { + /* Expression index */ + expr = lfirst(indexpr_item); + indexpr_item = lnext(unique_index->indexprs, indexpr_item); + } + else /* attr < 0 */ + { + /* Index on system column is not supported */ + Assert(false); + } + + /* + * Check index_col = Const case with regarding to opfamily checking + * If we can remove the index_col from the final UniqueKey->exprs. + */ + forboth(lc1, const_exprs, lc2, const_expr_opfamilies) + { + if (list_member_oid((List *)lfirst(lc2), unique_index->opfamily[c]) + && match_index_to_operand((Node *) lfirst(lc1), c, unique_index)) + { + matched_const = true; + break; + } + } + + if (matched_const) + continue; + + /* Check if the indexed expr is used in rel */ + if (attr > 0) + { + /* + * Normal Indexed column, if the col is not used, then the index is useless + * for uniquekey. + */ + attr -= FirstLowInvalidHeapAttributeNumber; + + if (!bms_is_member(attr, used_varattrs)) + { + *useful = false; + break; + } + } + else if (!list_member(unique_index->rel->reltarget->exprs, expr)) + { + /* Expression index but the expression is not used in rel */ + *useful = false; + break; + } + + /* check not null property. */ + if (attr == 0) + { + /* We never know if a expression yields null or not */ + *multi_nullvals = true; + } + else if (!bms_is_member(attr, unique_index->rel->notnullattrs) + && !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber, + unique_index->rel->notnullattrs)) + { + *multi_nullvals = true; + } + + exprs = lappend(exprs, expr); + } + return exprs; +} + + +/* + * add_uniquekey_for_onerow + * If we are sure that the relation only returns one row, then all the columns + * are unique. However we don't need to create UniqueKey for every column, we + * just set exprs = NIL and overwrites all the other UniqueKey on this RelOptInfo + * since this one has strongest semantics. + */ +void +add_uniquekey_for_onerow(RelOptInfo *rel) +{ + /* + * We overwrite the previous UniqueKey on purpose since this one has the + * strongest semantic. + */ + rel->uniquekeys = list_make1(makeUniqueKey(NIL, false)); +} + + +/* + * initililze_uniquecontext_for_joinrel + * Return a List of UniqueKeyContext for an inputrel + */ +static List * +initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel) +{ + List *res = NIL; + ListCell *lc; + foreach(lc, inputrel->uniquekeys) + { + UniqueKeyContext context; + context = palloc(sizeof(struct UniqueKeyContextData)); + context->uniquekey = lfirst_node(UniqueKey, lc); + context->added_to_joinrel = false; + context->useful = true; + res = lappend(res, context); + } + return res; +} + + +/* + * get_exprs_from_uniquekey + * Unify the way of get List of exprs from a one-row UniqueKey or + * normal UniqueKey. for the onerow case, every expr in rel1 is a valid + * UniqueKey. Return a List of exprs. + * + * rel1: The relation which you want to get the exprs. + * ukey: The UniqueKey you want to get the exprs. + */ +static List * +get_exprs_from_uniquekey(RelOptInfo *joinrel, RelOptInfo *rel1, UniqueKey *ukey) +{ + ListCell *lc; + bool onerow = rel1 != NULL && relation_is_onerow(rel1); + + List *res = NIL; + Assert(onerow || ukey); + if (onerow) + { + /* Only cares about the exprs still exist in joinrel */ + foreach(lc, joinrel->reltarget->exprs) + { + Bitmapset *relids = pull_varnos(lfirst(lc)); + if (bms_is_subset(relids, rel1->relids)) + { + res = lappend(res, list_make1(lfirst(lc))); + } + } + } + else + { + res = list_make1(ukey->exprs); + } + return res; +} + +/* + * Partitioned table Unique Keys. + * The partition table unique key is maintained as: + * 1. The index must be unique as usual. + * 2. The index must contains partition key. + * 3. The index must exist on all the child rel. see simple_indexinfo_equal for + * how we compare it. + */ + +/* + * index_constains_partkey + * return true if the index contains the partiton key. + */ +static bool +index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind) +{ + ListCell *lc; + int i; + Assert(IS_PARTITIONED_REL(partrel)); + Assert(partrel->part_scheme->partnatts > 0); + + for(i = 0; i < partrel->part_scheme->partnatts; i++) + { + Node *part_expr = linitial(partrel->partexprs[i]); + bool found_in_index = false; + foreach(lc, ind->indextlist) + { + Expr *index_expr = lfirst_node(TargetEntry, lc)->expr; + if (equal(index_expr, part_expr)) + { + found_in_index = true; + break; + } + } + if (!found_in_index) + return false; + } + return true; +} + +/* + * simple_indexinfo_equal + * + * Used to check if the 2 index is same as each other. The index here + * is COPIED from childrel and did some tiny changes(see + * simple_copy_indexinfo_to_parent) + */ +static bool +simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2) +{ + Size oid_cmp_len = sizeof(Oid) * ind1->ncolumns; + + return ind1->ncolumns == ind2->ncolumns && + ind1->unique == ind2->unique && + memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0 && + memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0 && + memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0 && + memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0 && + equal(get_tlist_exprs(ind1->indextlist, true), + get_tlist_exprs(ind2->indextlist, true)); +} + + +/* + * The below macros are used for simple_copy_indexinfo_to_parent which is so + * customized that I don't want to put it to copyfuncs.c. So copy it here. + */ +#define COPY_POINTER_FIELD(fldname, sz) \ + do { \ + Size _size = (sz); \ + newnode->fldname = palloc(_size); \ + memcpy(newnode->fldname, from->fldname, _size); \ + } while (0) + +#define COPY_NODE_FIELD(fldname) \ + (newnode->fldname = copyObjectImpl(from->fldname)) + +#define COPY_SCALAR_FIELD(fldname) \ + (newnode->fldname = from->fldname) + + +/* + * simple_copy_indexinfo_to_parent (from partition) + * Copy the IndexInfo from child relation to parent relation with some modification, + * which is used to test: + * 1. If the same index exists in all the childrels. + * 2. If the parentrel->reltarget/basicrestrict info matches this index. + */ +static IndexOptInfo * +simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from) +{ + IndexOptInfo *newnode = makeNode(IndexOptInfo); + AppendRelInfo *appinfo = find_appinfo_by_child(root, from->rel->relid); + ListCell *lc; + int idx = 0; + + COPY_SCALAR_FIELD(ncolumns); + COPY_SCALAR_FIELD(nkeycolumns); + COPY_SCALAR_FIELD(unique); + COPY_SCALAR_FIELD(immediate); + /* We just need to know if it is NIL or not */ + COPY_SCALAR_FIELD(indpred); + COPY_SCALAR_FIELD(predOK); + COPY_POINTER_FIELD(indexkeys, from->ncolumns * sizeof(int)); + COPY_POINTER_FIELD(indexcollations, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opfamily, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opcintype, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(sortopfamily, from->ncolumns * sizeof(Oid)); + COPY_NODE_FIELD(indextlist); + + /* Convert index exprs on child expr to expr on parent */ + foreach(lc, newnode->indextlist) + { + TargetEntry *tle = lfirst_node(TargetEntry, lc); + /* Index on expression is ignored */ + Assert(IsA(tle->expr, Var)); + tle->expr = (Expr *) find_parent_var(appinfo, (Var *) tle->expr); + newnode->indexkeys[idx] = castNode(Var, tle->expr)->varattno; + idx++; + } + newnode->rel = parentrel; + return newnode; +} + +/* + * adjust_partition_unique_indexlist + * + * global_unique_indexes: At the beginning, it contains the copy & modified + * unique index from the first partition. And then check if each index in it still + * exists in the following partitions. If no, remove it. at last, it has an + * index list which exists in all the partitions. + */ +static void +adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_indexes) +{ + ListCell *lc, *lc2; + foreach(lc, *global_unique_indexes) + { + IndexOptInfo *g_ind = lfirst_node(IndexOptInfo, lc); + bool found_in_child = false; + + foreach(lc2, childrel->indexlist) + { + IndexOptInfo *p_ind = lfirst_node(IndexOptInfo, lc2); + IndexOptInfo *p_ind_copy; + if (!p_ind->unique || !p_ind->immediate || + (p_ind->indpred != NIL && !p_ind->predOK)) + continue; + p_ind_copy = simple_copy_indexinfo_to_parent(root, parentrel, p_ind); + if (simple_indexinfo_equal(p_ind_copy, g_ind)) + { + found_in_child = true; + break; + } + } + if (!found_in_child) + /* The index doesn't exist in childrel, remove it from global_unique_indexes */ + *global_unique_indexes = foreach_delete_current(*global_unique_indexes, lc); + } +} + +/* Helper function for groupres/distinctrel */ +static void +add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgroups) +{ + Query *parse = root->parse; + List *exprs; + + /* + * XXX: If there are some vars which is not in current levelsup, the semantic is + * imprecise, should we avoid it or not? levelsup = 1 is just a demo, maybe we need to + * check every level other than 0, if so, looks we have to write another + * pull_var_walker. + */ + List *upper_vars = pull_vars_of_level((Node*)sortgroups, 1); + + if (upper_vars != NIL) + return; + + exprs = get_sortgrouplist_exprs(sortgroups, parse->targetList); + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(exprs, + false /* sortgroupclause can't be multi_nullvals */)); +} + + +/* + * add_combined_uniquekey + * The combination of both UniqueKeys is a valid UniqueKey for joinrel no matter + * the jointype. + */ +bool +add_combined_uniquekey(RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype) +{ + + ListCell *lc1, *lc2; + + /* Either side has multi_nullvals or we have outer join, + * the combined UniqueKey has multi_nullvals */ + bool multi_nullvals = outer_ukey->multi_nullvals || + inner_ukey->multi_nullvals || IS_OUTER_JOIN(jointype); + + /* The only case we can get onerow joinrel after join */ + if (relation_is_onerow(outer_rel) + && relation_is_onerow(inner_rel) + && jointype == JOIN_INNER) + { + add_uniquekey_for_onerow(joinrel); + return true; + } + + foreach(lc1, get_exprs_from_uniquekey(joinrel, outer_rel, outer_ukey)) + { + foreach(lc2, get_exprs_from_uniquekey(joinrel, inner_rel, inner_ukey)) + { + List *exprs = list_concat_copy(lfirst_node(List, lc1), lfirst_node(List, lc2)); + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(exprs, + multi_nullvals)); + } + } + return false; +} diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index b406d41e91..0551ae0512 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -2389,6 +2389,8 @@ grouping_planner(PlannerInfo *root, bool inheritance_update, add_path(final_rel, path); } + simple_copy_uniquekeys(current_rel, final_rel); + /* * Generate partial paths for final_rel, too, if outer query levels might * be able to make use of them. @@ -3899,6 +3901,8 @@ create_grouping_paths(PlannerInfo *root, } set_cheapest(grouped_rel); + + populate_grouprel_uniquekeys(root, grouped_rel, input_rel); return grouped_rel; } @@ -4615,7 +4619,7 @@ create_window_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(window_rel); - + simple_copy_uniquekeys(input_rel, window_rel); return window_rel; } @@ -4911,7 +4915,7 @@ create_distinct_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(distinct_rel); - + populate_distinctrel_uniquekeys(root, input_rel, distinct_rel); return distinct_rel; } @@ -5172,6 +5176,8 @@ create_ordered_paths(PlannerInfo *root, */ Assert(ordered_rel->pathlist != NIL); + simple_copy_uniquekeys(input_rel, ordered_rel); + return ordered_rel; } @@ -6049,6 +6055,9 @@ adjust_paths_for_srfs(PlannerInfo *root, RelOptInfo *rel, if (list_length(targets) == 1) return; + /* UniqueKey is not valid after handling the SRF. */ + rel->uniquekeys = NIL; + /* * Stack SRF-evaluation nodes atop each path for the rel. * diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c index 951aed80e7..e94e92937c 100644 --- a/src/backend/optimizer/prep/prepunion.c +++ b/src/backend/optimizer/prep/prepunion.c @@ -689,6 +689,8 @@ generate_union_paths(SetOperationStmt *op, PlannerInfo *root, /* Undo effects of possibly forcing tuple_fraction to 0 */ root->tuple_fraction = save_fraction; + /* Add the UniqueKeys */ + populate_unionrel_uniquekeys(root, result_rel); return result_rel; } diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c index d722063cf3..44c37ecffc 100644 --- a/src/backend/optimizer/util/appendinfo.c +++ b/src/backend/optimizer/util/appendinfo.c @@ -746,3 +746,47 @@ find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos) } return appinfos; } + +/* + * find_appinfo_by_child + * + */ +AppendRelInfo * +find_appinfo_by_child(PlannerInfo *root, Index child_index) +{ + ListCell *lc; + foreach(lc, root->append_rel_list) + { + AppendRelInfo *appinfo = lfirst_node(AppendRelInfo, lc); + if (appinfo->child_relid == child_index) + return appinfo; + } + elog(ERROR, "parent relation cant be found"); + return NULL; +} + +/* + * find_parent_var + * + */ +Var * +find_parent_var(AppendRelInfo *appinfo, Var *child_var) +{ + ListCell *lc; + Var *res = NULL; + Index attno = 1; + foreach(lc, appinfo->translated_vars) + { + Node *child_node = lfirst(lc); + if (equal(child_node, child_var)) + { + res = copyObject(child_var); + res->varattno = attno; + res->varno = appinfo->parent_relid; + } + attno++; + } + if (res == NULL) + elog(ERROR, "parent var can't be found."); + return res; +} diff --git a/src/backend/optimizer/util/inherit.c b/src/backend/optimizer/util/inherit.c index 3132fd35a5..d66b40ec50 100644 --- a/src/backend/optimizer/util/inherit.c +++ b/src/backend/optimizer/util/inherit.c @@ -736,6 +736,7 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, { Node *onecq = (Node *) lfirst(lc2); bool pseudoconstant; + RestrictInfo *child_rinfo; /* check for pseudoconstant (no Vars or volatile functions) */ pseudoconstant = @@ -747,13 +748,14 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, root->hasPseudoConstantQuals = true; } /* reconstitute RestrictInfo with appropriate properties */ - childquals = lappend(childquals, - make_restrictinfo((Expr *) onecq, - rinfo->is_pushed_down, - rinfo->outerjoin_delayed, - pseudoconstant, - rinfo->security_level, - NULL, NULL, NULL)); + child_rinfo = make_restrictinfo((Expr *) onecq, + rinfo->is_pushed_down, + rinfo->outerjoin_delayed, + pseudoconstant, + rinfo->security_level, + NULL, NULL, NULL); + child_rinfo->mergeopfamilies = rinfo->mergeopfamilies; + childquals = lappend(childquals, child_rinfo); /* track minimum security level among child quals */ cq_min_security = Min(cq_min_security, rinfo->security_level); } diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 31d9aedeeb..c83f17acb7 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -16,6 +16,7 @@ #include "nodes/execnodes.h" #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" extern A_Expr *makeA_Expr(A_Expr_Kind kind, List *name, @@ -105,4 +106,6 @@ extern GroupingSet *makeGroupingSet(GroupingSetKind kind, List *content, int loc extern VacuumRelation *makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols); +extern UniqueKey* makeUniqueKey(List *exprs, bool multi_nullvals); + #endif /* MAKEFUNC_H */ diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 381d84b4e4..41110ed888 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -264,6 +264,7 @@ typedef enum NodeTag T_EquivalenceMember, T_PathKey, T_PathTarget, + T_UniqueKey, T_RestrictInfo, T_IndexClause, T_PlaceHolderVar, diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index 9e3ebd488a..02e4458bef 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -730,6 +730,7 @@ typedef struct RelOptInfo QualCost baserestrictcost; /* cost of evaluating the above */ Index baserestrict_min_security; /* min security_level found in * baserestrictinfo */ + List *uniquekeys; /* List of UniqueKey */ List *joininfo; /* RestrictInfo structures for join clauses * involving this rel */ bool has_eclass_joins; /* T means joininfo is incomplete */ @@ -1047,6 +1048,28 @@ typedef struct PathKey } PathKey; +/* + * UniqueKey + * + * Represents the unique properties held by a RelOptInfo. + * + * exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL + * is a special case of UniqueKey, which means there is only 1 row in that + * relation. + * multi_nullvals: true means multi null values may exist in these exprs, so the + * uniqueness is not guaranteed in this case. This field is necessary for + * remove_useless_join & reduce_unique_semijoins where we don't mind these + * duplicated NULL values. It is set to true for 2 cases. One is a unique key + * from a unique index but the related column is nullable. The other one is for + * outer join. see populate_joinrel_uniquekeys for detail. + */ +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + /* * PathTarget * @@ -2473,7 +2496,7 @@ typedef enum * * flags indicating what kinds of grouping are possible. * partial_costs_set is true if the agg_partial_costs and agg_final_costs - * have been initialized. + * have been initialized. * agg_partial_costs gives partial aggregation costs. * agg_final_costs gives finalization costs. * target_parallel_safe is true if target is parallel safe. @@ -2503,8 +2526,8 @@ typedef struct * limit_tuples is an estimated bound on the number of output tuples, * or -1 if no LIMIT or couldn't estimate. * count_est and offset_est are the estimated values of the LIMIT and OFFSET - * expressions computed by preprocess_limit() (see comments for - * preprocess_limit() for more information). + * expressions computed by preprocess_limit() (see comments for + * preprocess_limit() for more information). */ typedef struct { diff --git a/src/include/nodes/pg_list.h b/src/include/nodes/pg_list.h index 14ea2766ad..621f54a9f8 100644 --- a/src/include/nodes/pg_list.h +++ b/src/include/nodes/pg_list.h @@ -528,6 +528,8 @@ extern bool list_member_ptr(const List *list, const void *datum); extern bool list_member_int(const List *list, int datum); extern bool list_member_oid(const List *list, Oid datum); +extern bool list_is_subset(const List *members, const List *target); + extern List *list_delete(List *list, void *datum); extern List *list_delete_ptr(List *list, void *datum); extern List *list_delete_int(List *list, int datum); diff --git a/src/include/optimizer/appendinfo.h b/src/include/optimizer/appendinfo.h index d6a27a60dd..e87c92a054 100644 --- a/src/include/optimizer/appendinfo.h +++ b/src/include/optimizer/appendinfo.h @@ -32,4 +32,7 @@ extern Relids adjust_child_relids_multilevel(PlannerInfo *root, Relids relids, extern AppendRelInfo **find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos); +extern AppendRelInfo *find_appinfo_by_child(PlannerInfo *root, Index child_index); +extern Var *find_parent_var(AppendRelInfo *appinfo, Var *child_var); + #endif /* APPENDINFO_H */ diff --git a/src/include/optimizer/optimizer.h b/src/include/optimizer/optimizer.h index 3e4171056e..9445141263 100644 --- a/src/include/optimizer/optimizer.h +++ b/src/include/optimizer/optimizer.h @@ -23,6 +23,7 @@ #define OPTIMIZER_H #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" /* * We don't want to include nodes/pathnodes.h here, because non-planner @@ -156,6 +157,7 @@ extern TargetEntry *get_sortgroupref_tle(Index sortref, List *targetList); extern TargetEntry *get_sortgroupclause_tle(SortGroupClause *sgClause, List *targetList); +extern Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); extern Node *get_sortgroupclause_expr(SortGroupClause *sgClause, List *targetList); extern List *get_sortgrouplist_exprs(List *sgClauses, diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h index 10b6e81079..9217a8d6c6 100644 --- a/src/include/optimizer/paths.h +++ b/src/include/optimizer/paths.h @@ -240,5 +240,48 @@ extern PathKey *make_canonical_pathkey(PlannerInfo *root, int strategy, bool nulls_first); extern void add_paths_to_append_rel(PlannerInfo *root, RelOptInfo *rel, List *live_childrels); +extern List *select_mergejoin_clauses(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + JoinType jointype, + bool *mergejoin_allowed); + +/* + * uniquekeys.c + * Utilities for matching and building unique keys + */ +extern void populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List* unique_index_list); +extern void populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels); +extern void populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel); +extern void populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel); +extern void populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel); +extern void simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel); +extern void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel); +extern void populate_joinrel_uniquekeys(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + RelOptInfo *rel2, + List *restrictlist, + JoinType jointype); + +extern bool relation_has_uniquekeys_for(PlannerInfo *root, + RelOptInfo *rel, + List *exprs, + bool allow_multinulls); +extern bool relation_is_onerow(RelOptInfo *rel); #endif /* PATHS_H */ -- 2.21.0 --op7mf2i72xk5yu7c Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v36-0003-Extend-UniqueKeys.patch" ^ permalink raw reply [nested|flat] 91+ messages in thread
* [PATCH v38 2/6] Introduce UniqueKey attributes on RelOptInfo struct. @ 2020-05-11 07:50 一挃 <[email protected]> 0 siblings, 0 replies; 91+ messages in thread From: 一挃 @ 2020-05-11 07:50 UTC (permalink / raw) UniqueKey is a set of exprs on RelOptInfo which represents the exprs will be unique on the given RelOptInfo. You can see README.uniquekey for more information. --- src/backend/nodes/copyfuncs.c | 13 + src/backend/nodes/list.c | 31 + src/backend/nodes/makefuncs.c | 13 + src/backend/nodes/outfuncs.c | 11 + src/backend/nodes/readfuncs.c | 10 + src/backend/optimizer/path/Makefile | 3 +- src/backend/optimizer/path/README.uniquekey | 131 +++ src/backend/optimizer/path/allpaths.c | 10 + src/backend/optimizer/path/joinpath.c | 9 +- src/backend/optimizer/path/joinrels.c | 2 + src/backend/optimizer/path/pathkeys.c | 3 +- src/backend/optimizer/path/uniquekeys.c | 1135 +++++++++++++++++++ src/backend/optimizer/plan/planner.c | 13 +- src/backend/optimizer/prep/prepunion.c | 2 + src/backend/optimizer/util/appendinfo.c | 44 + src/backend/optimizer/util/inherit.c | 18 +- src/include/nodes/makefuncs.h | 3 + src/include/nodes/nodes.h | 1 + src/include/nodes/pathnodes.h | 29 +- src/include/nodes/pg_list.h | 10 + src/include/optimizer/appendinfo.h | 3 + src/include/optimizer/optimizer.h | 2 + src/include/optimizer/paths.h | 43 + 23 files changed, 1515 insertions(+), 24 deletions(-) create mode 100644 src/backend/optimizer/path/README.uniquekey create mode 100644 src/backend/optimizer/path/uniquekeys.c diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..75c1c5e824 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2296,6 +2296,16 @@ _copyPathKey(const PathKey *from) return newnode; } +static UniqueKey * +_copyUniqueKey(const UniqueKey *from) +{ + UniqueKey *newnode = makeNode(UniqueKey); + + COPY_NODE_FIELD(exprs); + COPY_SCALAR_FIELD(multi_nullvals); + + return newnode; +} /* * _copyRestrictInfo */ @@ -5220,6 +5230,9 @@ copyObjectImpl(const void *from) case T_PathKey: retval = _copyPathKey(from); break; + case T_UniqueKey: + retval = _copyUniqueKey(from); + break; case T_RestrictInfo: retval = _copyRestrictInfo(from); break; diff --git a/src/backend/nodes/list.c b/src/backend/nodes/list.c index dbf6b30233..ca099495a1 100644 --- a/src/backend/nodes/list.c +++ b/src/backend/nodes/list.c @@ -702,6 +702,37 @@ list_member_oid(const List *list, Oid datum) return false; } +/* + * return true iff every entry in "members" list is also present + * in the "target" list. + */ +bool +list_is_subset(const List *members, const List *target) +{ + const ListCell *lc1, *lc2; + + Assert(IsPointerList(members)); + Assert(IsPointerList(target)); + check_list_invariants(members); + check_list_invariants(target); + + foreach(lc1, members) + { + bool found = false; + foreach(lc2, target) + { + if (equal(lfirst(lc1), lfirst(lc2))) + { + found = true; + break; + } + } + if (!found) + return false; + } + return true; +} + /* * Delete the n'th cell (counting from 0) in list. * diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 01c110cd2f..40415d0f5b 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -815,3 +815,16 @@ makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols) v->va_cols = va_cols; return v; } + + +/* + * makeUniqueKey + */ +UniqueKey* +makeUniqueKey(List *exprs, bool multi_nullvals) +{ + UniqueKey * ukey = makeNode(UniqueKey); + ukey->exprs = exprs; + ukey->multi_nullvals = multi_nullvals; + return ukey; +} diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..44154cde6a 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2456,6 +2456,14 @@ _outPathKey(StringInfo str, const PathKey *node) WRITE_BOOL_FIELD(pk_nulls_first); } +static void +_outUniqueKey(StringInfo str, const UniqueKey *node) +{ + WRITE_NODE_TYPE("UNIQUEKEY"); + WRITE_NODE_FIELD(exprs); + WRITE_BOOL_FIELD(multi_nullvals); +} + static void _outPathTarget(StringInfo str, const PathTarget *node) { @@ -4198,6 +4206,9 @@ outNode(StringInfo str, const void *obj) case T_PathKey: _outPathKey(str, obj); break; + case T_UniqueKey: + _outUniqueKey(str, obj); + break; case T_PathTarget: _outPathTarget(str, obj); break; diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index c5e136e9c3..b3e212bf1c 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -492,6 +492,14 @@ _readSetOperationStmt(void) READ_DONE(); } +static UniqueKey * +_readUniqueKey(void) +{ + READ_LOCALS(UniqueKey); + READ_NODE_FIELD(exprs); + READ_BOOL_FIELD(multi_nullvals); + READ_DONE(); +} /* * Stuff from primnodes.h. @@ -2717,6 +2725,8 @@ parseNodeString(void) return_value = _readCommonTableExpr(); else if (MATCH("SETOPERATIONSTMT", 16)) return_value = _readSetOperationStmt(); + else if (MATCH("UNIQUEKEY", 9)) + return_value = _readUniqueKey(); else if (MATCH("ALIAS", 5)) return_value = _readAlias(); else if (MATCH("RANGEVAR", 8)) diff --git a/src/backend/optimizer/path/Makefile b/src/backend/optimizer/path/Makefile index 1e199ff66f..7b9820c25f 100644 --- a/src/backend/optimizer/path/Makefile +++ b/src/backend/optimizer/path/Makefile @@ -21,6 +21,7 @@ OBJS = \ joinpath.o \ joinrels.o \ pathkeys.o \ - tidpath.o + tidpath.o \ + uniquekeys.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/optimizer/path/README.uniquekey b/src/backend/optimizer/path/README.uniquekey new file mode 100644 index 0000000000..5eac761995 --- /dev/null +++ b/src/backend/optimizer/path/README.uniquekey @@ -0,0 +1,131 @@ +1. What is UniqueKey? +We can think UniqueKey is a set of exprs for a RelOptInfo, which we are insure +that doesn't yields same result among all the rows. The simplest UniqueKey +format is primary key. + +However we define the UnqiueKey as below. + +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + +exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL +is a special case of UniqueKey, which means there is only one row in that +relation.it has a stronger semantic than others. like SELECT uk FROM t; uk is +normal unique key and may have different values. SELECT colx FROM t WHERE uk = +const. colx is unique AND we have only 1 value. This field can used for +innerrel_is_unique. this logic is handled specially in add_uniquekey_for_onerow +function. + +multi_nullvals: true means multi null values may exist in these exprs, so the +uniqueness is not guaranteed in this case. This field is necessary for +remove_useless_join & reduce_unique_semijoins where we don't mind these +duplicated NULL values. It is set to true for 2 cases. One is a unique key +from a unique index but the related column is nullable. The other one is for +outer join. see populate_joinrel_uniquekeys for detail. + + +The UniqueKey can be used at the following cases at least: +1. remove_useless_joins. +2. reduce_semianti_joins +3. remove distinct node if distinct clause is unique. +4. remove aggnode if group by clause is unique. +5. Index Skip Scan (WIP) +6. Aggregation Push Down without 2 phase aggregation if the join can't + duplicated the aggregated rows. (work in progress feature) + +2. How is it maintained? + +We have a set of populate_xxx_unqiuekeys functions to maintain the uniquekey on +various cases. xxx includes baserel, joinrel, partitionedrel, distinctrel, +groupedrel, unionrel. and we also need to convert the uniquekey from subquery +to outer relation, which is what convert_subquery_uniquekeys does. + +1. The first part is about baserel. We handled 3 cases. suppose we have Unique +Index on (a, b). + +1. SELECT a, b FROM t. UniqueKey (a, b) +2. SELECT a FROM t WHERE b = 1; UniqueKey (a) +3. SELECT .. FROM t WHERE a = 1 AND b = 1; UniqueKey (NIL). onerow case, every + column is Unique. + +2. The next part is joinrel, this part is most error-prone, we simplified the rules +like below: +1. If the relation's UniqueKey can't be duplicated after join, then is will be + still valid for the join rel. The function we used here is + innerrel_keeps_unique. The basic idea is innerrel.any_col = outer.uk. + +2. If the UnqiueKey can't keep valid via the rule 1, the combination of the + UniqueKey from both sides are valid for sure. We can prove this as: if the + unique exprs from rel1 is duplicated by rel2, the duplicated rows must + contains different unique exprs from rel2. + +More considerations about onerow: +1. If relation with one row and it can't be duplicated, it is still possible + contains mulit_nullvas after outer join. +2. If the either UniqueKey can be duplicated after join, the can get one row + only when both side is one row AND there is no outer join. +3. Whenever the onerow UniqueKey is not a valid any more, we need to convert one + row UniqueKey to normal unique key since we don't store exprs for one-row + relation. get_exprs_from_uniquekeys will be used here. + + +More considerations about multi_nullvals after join: +1. If the original UnqiueKey has multi_nullvals, the final UniqueKey will have + mulit_nullvals in any case. +2. If a unique key doesn't allow mulit_nullvals, after some outer join, it + allows some outer join. + + +3. When we comes to subquery, we need to convert_subquery_unqiuekeys just like +convert_subquery_pathkeys. Only the UniqueKey insides subquery is referenced as +a Var in outer relation will be reused. The relationship between the outerrel.Var +and subquery.exprs is built with outerel->subroot->processed_tlist. + + +4. As for the SRF functions, it will break the uniqueness of uniquekey, However it +is handled in adjust_paths_for_srfs, which happens after the query_planner. so +we will maintain the UniqueKey until there and reset it to NIL at that +places. This can't help on distinct/group by elimination cases but probably help +in some other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is +semantic correctly. + + +5. As for inherit table, we first main the UnqiueKey on childrel as well. But for +partitioned table we need to maintain 2 different kinds of +UnqiueKey. 1). UniqueKey on the parent relation 2). UniqueKey on child +relation for partition wise query. + +Example: +CREATE TABLE p (a int not null, b int not null) partition by list (a); +CREATE TABLE p0 partition of p for values in (1); +CREATE TABLE p1 partition of p for values in (2); + +create unique index p0_b on p0(b); +create unique index p1_b on p1(b); + +Now b is only unique on partition level, so the distinct can't be removed on +the following cases. SELECT DISTINCT b FROM p; + +Another example is SELECT DISTINCT a, b FROM p WHERE a = 1; Since only one +partition is chosen, the UniqueKey on child relation is same as the UniqueKey on +parent relation. + +Another usage of UniqueKey on partition level is it be helpful for +partition-wise join. + +As for the UniqueKey on parent table level, it comes with 2 different ways, +1). the UniqueKey is also derived in UniqueKey index, but the index must be same +in all the related children relations and the unique index must contains +Partition Key in it. Example: + +CREATE UNIQUE INDEX p_ab ON p(a, b); -- where a is the partition key. + +-- Query +SELECT a, b FROM p; the (a, b) is a UniqueKey of p. + +2). If the parent relation has only one childrel, the UniqueKey on childrel is + the UniqueKey on parent as well. diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index 37b4223adb..d52ad59f89 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -579,6 +579,12 @@ set_plain_rel_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) */ check_index_predicates(root, rel); + /* + * Now that we've marked which partial indexes are suitable, we can now + * build the relation's unique keys. + */ + populate_baserel_uniquekeys(root, rel, rel->indexlist); + /* Mark rel with estimated output rows, width, etc */ set_baserel_size_estimates(root, rel); } @@ -1297,6 +1303,8 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, /* Add paths to the append relation. */ add_paths_to_append_rel(root, rel, live_childrels); + if (IS_PARTITIONED_REL(rel)) + populate_partitionedrel_uniquekeys(root, rel, live_childrels); } @@ -2304,6 +2312,8 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, pathkeys, required_outer)); } + convert_subquery_uniquekeys(root, rel, sub_final_rel); + /* If outer rel allows parallelism, do same for partial paths. */ if (rel->consider_parallel && bms_is_empty(required_outer)) { diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c index 57ce97fd53..697ac047fb 100644 --- a/src/backend/optimizer/path/joinpath.c +++ b/src/backend/optimizer/path/joinpath.c @@ -71,13 +71,6 @@ static void consider_parallel_mergejoin(PlannerInfo *root, static void hash_inner_and_outer(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, RelOptInfo *innerrel, JoinType jointype, JoinPathExtraData *extra); -static List *select_mergejoin_clauses(PlannerInfo *root, - RelOptInfo *joinrel, - RelOptInfo *outerrel, - RelOptInfo *innerrel, - List *restrictlist, - JoinType jointype, - bool *mergejoin_allowed); static void generate_mergejoin_paths(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *innerrel, @@ -1927,7 +1920,7 @@ hash_inner_and_outer(PlannerInfo *root, * if it is mergejoinable and involves vars from the two sub-relations * currently of interest. */ -static List * +List * select_mergejoin_clauses(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c index 0dbe2ac726..7271f044ec 100644 --- a/src/backend/optimizer/path/joinrels.c +++ b/src/backend/optimizer/path/joinrels.c @@ -924,6 +924,8 @@ populate_joinrel_with_paths(PlannerInfo *root, RelOptInfo *rel1, /* Apply partitionwise join technique, if possible. */ try_partitionwise_join(root, rel1, rel2, joinrel, sjinfo, restrictlist); + + populate_joinrel_uniquekeys(root, joinrel, rel1, rel2, restrictlist, sjinfo->jointype); } diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index bd9a176d7d..139278829b 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -33,7 +33,6 @@ static bool pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys); static bool matches_boolean_partition_clause(RestrictInfo *rinfo, RelOptInfo *partrel, int partkeycol); -static Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); static bool right_merge_direction(PlannerInfo *root, PathKey *pathkey); @@ -1035,7 +1034,7 @@ convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel, * We need this to ensure that we don't return pathkeys describing values * that are unavailable above the level of the subquery scan. */ -static Var * +Var * find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle) { ListCell *lc; diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c new file mode 100644 index 0000000000..77ed2b2eff --- /dev/null +++ b/src/backend/optimizer/path/uniquekeys.c @@ -0,0 +1,1135 @@ +/*------------------------------------------------------------------------- + * + * uniquekeys.c + * Utilities for matching and building unique keys + * + * Portions Copyright (c) 2020, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/optimizer/path/uniquekeys.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "nodes/makefuncs.h" +#include "nodes/nodeFuncs.h" +#include "optimizer/pathnode.h" +#include "optimizer/paths.h" +#include "optimizer/appendinfo.h" +#include "optimizer/optimizer.h" +#include "optimizer/tlist.h" +#include "rewrite/rewriteManip.h" + + +/* + * This struct is used to help populate_joinrel_uniquekeys. + * + * added_to_joinrel is true if a uniquekey (from outerrel or innerrel) + * has been added to joinrel. + * useful is true if the exprs of the uniquekey still appears in joinrel. + */ +typedef struct UniqueKeyContextData +{ + UniqueKey *uniquekey; + bool added_to_joinrel; + bool useful; +} *UniqueKeyContext; + +static List *initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel); +static bool innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + bool reverse); + +static List *get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals); +static List *get_exprs_from_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + UniqueKey *ukey); +static void add_uniquekey_for_onerow(RelOptInfo *rel); +static bool add_combined_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype); + +/* Used for unique indexes checking for partitioned table */ +static bool index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind); +static IndexOptInfo *simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from); +static bool simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2); +static void adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_index); + +/* Helper function for grouped relation and distinct relation. */ +static void add_uniquekey_from_sortgroups(PlannerInfo *root, + RelOptInfo *rel, + List *sortgroups); + +/* + * populate_baserel_uniquekeys + * Populate 'baserel' uniquekeys list by looking at the rel's unique index + * and baserestrictinfo + */ +void +populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List *indexlist) +{ + ListCell *lc; + List *matched_uniq_indexes = NIL; + + /* Attrs appears in rel->reltarget->exprs. */ + Bitmapset *used_attrs = NULL; + + List *const_exprs = NIL; + List *expr_opfamilies = NIL; + + Assert(baserel->rtekind == RTE_RELATION); + + foreach(lc, indexlist) + { + IndexOptInfo *ind = (IndexOptInfo *) lfirst(lc); + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + matched_uniq_indexes = lappend(matched_uniq_indexes, ind); + } + + if (matched_uniq_indexes == NIL) + return; + + /* Check which attrs is used in baserel->reltarget */ + pull_varattnos((Node *)baserel->reltarget->exprs, baserel->relid, &used_attrs); + + /* Check which attrno is used at a mergeable const filter */ + foreach(lc, baserel->baserestrictinfo) + { + RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); + + if (rinfo->mergeopfamilies == NIL) + continue; + + if (bms_is_empty(rinfo->left_relids)) + { + const_exprs = lappend(const_exprs, get_rightop(rinfo->clause)); + } + else if (bms_is_empty(rinfo->right_relids)) + { + const_exprs = lappend(const_exprs, get_leftop(rinfo->clause)); + } + else + continue; + + expr_opfamilies = lappend(expr_opfamilies, rinfo->mergeopfamilies); + } + + foreach(lc, matched_uniq_indexes) + { + bool multi_nullvals, useful; + List *exprs = get_exprs_from_uniqueindex(lfirst_node(IndexOptInfo, lc), + const_exprs, + expr_opfamilies, + used_attrs, + &useful, + &multi_nullvals); + if (useful) + { + if (exprs == NIL) + { + /* All the columns in Unique Index matched with a restrictinfo */ + add_uniquekey_for_onerow(baserel); + return; + } + baserel->uniquekeys = lappend(baserel->uniquekeys, + makeUniqueKey(exprs, multi_nullvals)); + } + } +} + + +/* + * populate_partitionedrel_uniquekeys + * The UniqueKey on partitionrel comes from 2 cases: + * 1). Only one partition is involved in this query, the unique key can be + * copied to parent rel from childrel. + * 2). There are some unique index which includes partition key and exists + * in all the related partitions. + * We never mind rule 2 if we hit rule 1. + */ + +void +populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels) +{ + ListCell *lc; + List *global_uniq_indexlist = NIL; + RelOptInfo *childrel; + bool is_first = true; + + Assert(IS_PARTITIONED_REL(rel)); + + if (childrels == NIL) + return; + + /* + * If there is only one partition used in this query, the UniqueKey in childrel is + * still valid in parent level, but we need convert the format from child expr to + * parent expr. + */ + if (list_length(childrels) == 1) + { + /* Check for Rule 1 */ + RelOptInfo *childrel = linitial_node(RelOptInfo, childrels); + ListCell *lc; + Assert(childrel->reloptkind == RELOPT_OTHER_MEMBER_REL); + if (relation_is_onerow(childrel)) + { + add_uniquekey_for_onerow(rel); + return; + } + + foreach(lc, childrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + AppendRelInfo *appinfo = find_appinfo_by_child(root, childrel->relid); + List *parent_exprs = NIL; + bool can_reuse = true; + ListCell *lc2; + foreach(lc2, ukey->exprs) + { + Var *var = (Var *)lfirst(lc2); + /* + * If the expr comes from a expression, it is hard to build the expression + * in parent so ignore that case for now. + */ + if(!IsA(var, Var)) + { + can_reuse = false; + break; + } + /* Convert it to parent var */ + parent_exprs = lappend(parent_exprs, find_parent_var(appinfo, var)); + } + if (can_reuse) + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(parent_exprs, + ukey->multi_nullvals)); + } + } + else + { + /* Check for rule 2 */ + childrel = linitial_node(RelOptInfo, childrels); + foreach(lc, childrel->indexlist) + { + IndexOptInfo *ind = lfirst(lc); + IndexOptInfo *modified_index; + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + + /* + * During simple_copy_indexinfo_to_parent, we need to convert var from + * child var to parent var, index on expression is too complex to handle. + * so ignore it for now. + */ + if (ind->indexprs != NIL) + continue; + + modified_index = simple_copy_indexinfo_to_parent(root, rel, ind); + /* + * If the unique index doesn't contain partkey, then it is unique + * on this partition only, so it is useless for us. + */ + if (!index_constains_partkey(rel, modified_index)) + continue; + + global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index); + } + + if (global_uniq_indexlist != NIL) + { + foreach(lc, childrels) + { + RelOptInfo *child = lfirst(lc); + if (is_first) + { + is_first = false; + continue; + } + adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist); + } + /* Now we have a list of unique index which are exactly same on all childrels, + * Set the UniqueKey just like it is non-partition table + */ + populate_baserel_uniquekeys(root, rel, global_uniq_indexlist); + } + } +} + + +/* + * populate_distinctrel_uniquekeys + */ +void +populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel) +{ + /* The unique key before the distinct is still valid. */ + distinctrel->uniquekeys = list_copy(inputrel->uniquekeys); + add_uniquekey_from_sortgroups(root, distinctrel, root->parse->distinctClause); +} + +/* + * populate_grouprel_uniquekeys + */ +void +populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel) + +{ + Query *parse = root->parse; + bool input_ukey_added = false; + ListCell *lc; + + if (relation_is_onerow(inputrel)) + { + add_uniquekey_for_onerow(grouprel); + return; + } + if (parse->groupingSets) + return; + + /* A Normal group by without grouping set. */ + if (parse->groupClause) + { + /* + * Current even the groupby clause is Unique already, but if query has aggref + * We have to create grouprel still. To keep the UnqiueKey short, we will check + * the UniqueKey of input_rel still valid, if so we reuse it. + */ + foreach(lc, inputrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (list_is_subset(ukey->exprs, grouprel->reltarget->exprs)) + { + grouprel->uniquekeys = lappend(grouprel->uniquekeys, + ukey); + input_ukey_added = true; + } + } + if (!input_ukey_added) + /* + * group by clause must be a super-set of grouprel->reltarget->exprs except the + * aggregation expr, so if such exprs is unique already, no bother to generate + * new uniquekey for group by exprs. + */ + add_uniquekey_from_sortgroups(root, + grouprel, + root->parse->groupClause); + } + else + /* It has aggregation but without a group by, so only one row returned */ + add_uniquekey_for_onerow(grouprel); +} + +/* + * simple_copy_uniquekeys + * Using a function for the one-line code makes us easy to check where we simply + * copied the uniquekey. + */ +void +simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel) +{ + newrel->uniquekeys = oldrel->uniquekeys; +} + +/* + * populate_unionrel_uniquekeys + */ +void +populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel) +{ + ListCell *lc; + List *exprs = NIL; + + Assert(unionrel->uniquekeys == NIL); + + foreach(lc, unionrel->reltarget->exprs) + { + exprs = lappend(exprs, lfirst(lc)); + } + + if (exprs == NIL) + /* SQL: select union select; is valid, we need to handle it here. */ + add_uniquekey_for_onerow(unionrel); + else + unionrel->uniquekeys = lappend(unionrel->uniquekeys, + makeUniqueKey(exprs,false)); + +} + +/* + * populate_joinrel_uniquekeys + * + * populate uniquekeys for joinrel. We will check each relation to see if its + * UniqueKey is still valid via innerrel_keeps_unique, if so, we add it to + * joinrel. The multi_nullvals field will be changed to true for some outer + * join cases and one-row UniqueKey needs to be converted to normal UniqueKey + * for the same case as well. + * For the uniquekey in either baserel which can't be unique after join, we still + * check to see if combination of UniqueKeys from both side is still useful for us. + * if yes, we add it to joinrel as well. + */ +void +populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *outerrel, RelOptInfo *innerrel, + List *restrictlist, JoinType jointype) +{ + ListCell *lc, *lc2; + List *clause_list = NIL; + List *outerrel_ukey_ctx; + List *innerrel_ukey_ctx; + bool inner_onerow, outer_onerow; + bool mergejoin_allowed; + + /* Care about the outerrel relation only for SEMI/ANTI join */ + if (jointype == JOIN_SEMI || jointype == JOIN_ANTI) + { + foreach(lc, outerrel->uniquekeys) + { + UniqueKey *uniquekey = lfirst_node(UniqueKey, lc); + if (list_is_subset(uniquekey->exprs, joinrel->reltarget->exprs)) + joinrel->uniquekeys = lappend(joinrel->uniquekeys, uniquekey); + } + return; + } + + Assert(jointype == JOIN_LEFT || jointype == JOIN_FULL || jointype == JOIN_INNER); + + /* Fast path */ + if (innerrel->uniquekeys == NIL || outerrel->uniquekeys == NIL) + return; + + inner_onerow = relation_is_onerow(innerrel); + outer_onerow = relation_is_onerow(outerrel); + + outerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(outerrel); + innerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(innerrel); + + clause_list = select_mergejoin_clauses(root, joinrel, outerrel, innerrel, + restrictlist, jointype, + &mergejoin_allowed); + + if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true /* reverse */)) + { + bool outer_impact = jointype == JOIN_FULL; + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + /* Outer relation has one row, and the unique key is not duplicated after join, + * the joinrel will still has one row unless the jointype == JOIN_FULL. + */ + if (outer_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (outer_onerow) + { + /* + * The onerow outerrel becomes multi rows and multi_nullvals + * will be changed to true. We also need to set the exprs correctly since it + * can't be NIL any more. + */ + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, outerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Change multi_nullvals to true due to the full join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, true)); + else + /* Just reuse it */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + } + ctx->added_to_joinrel = true; + } + } + + if (innerrel_keeps_unique(root, outerrel, innerrel, clause_list, false)) + { + bool outer_impact = jointype == JOIN_FULL || jointype == JOIN_LEFT;; + + foreach(lc, innerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + if (inner_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (inner_onerow) + { + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, innerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Need to change multi_nullvals to true due to the outer join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, + true)); + else + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + + } + ctx->added_to_joinrel = true; + } + } + + /* + * The combination of the UniqueKey from both sides is unique as well regardless + * of join type, but no bother to add it if its subset has been added to joinrel + * already or it is not useful for the joinrel. + */ + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx1 = (UniqueKeyContext) lfirst(lc); + if (ctx1->added_to_joinrel || !ctx1->useful) + continue; + foreach(lc2, innerrel_ukey_ctx) + { + UniqueKeyContext ctx2 = (UniqueKeyContext) lfirst(lc2); + if (ctx2->added_to_joinrel || !ctx2->useful) + continue; + if (add_combined_uniquekey(root, joinrel, outerrel, innerrel, + ctx1->uniquekey, ctx2->uniquekey, + jointype)) + /* If we set a onerow UniqueKey to joinrel, we don't need other. */ + return; + } + } +} + + +/* + * convert_subquery_uniquekeys + * + * Covert the UniqueKey in subquery to outer relation. + */ +void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel) +{ + ListCell *lc; + + if (sub_final_rel->uniquekeys == NIL) + return; + + if (relation_is_onerow(sub_final_rel)) + { + add_uniquekey_for_onerow(currel); + return; + } + + Assert(currel->subroot != NULL); + + foreach(lc, sub_final_rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + ListCell *lc; + List *exprs = NIL; + bool ukey_useful = true; + + /* One row case is handled above */ + Assert(ukey->exprs != NIL); + foreach(lc, ukey->exprs) + { + Var *var; + TargetEntry *tle = tlist_member(lfirst(lc), + currel->subroot->processed_tlist); + if (tle == NULL) + { + ukey_useful = false; + break; + } + var = find_var_for_subquery_tle(currel, tle); + if (var == NULL) + { + ukey_useful = false; + break; + } + exprs = lappend(exprs, var); + } + + if (ukey_useful) + currel->uniquekeys = lappend(currel->uniquekeys, + makeUniqueKey(exprs, + ukey->multi_nullvals)); + + } +} + +/* + * innerrel_keeps_unique + * + * Check if Unique key of the innerrel is valid after join. innerrel's UniqueKey + * will be still valid if innerrel's any-column mergeop outrerel's uniquekey + * exists in clause_list. + * + * Note: the clause_list must be a list of mergeable restrictinfo already. + */ +static bool +innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *clause_list, + bool reverse) +{ + ListCell *lc, *lc2, *lc3; + + if (outerrel->uniquekeys == NIL || innerrel->uniquekeys == NIL) + return false; + + /* Check if there is outerrel's uniquekey in mergeable clause. */ + foreach(lc, outerrel->uniquekeys) + { + List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs; + bool clauselist_matchs_all_exprs = true; + foreach(lc2, outer_uq_exprs) + { + Node *outer_uq_expr = lfirst(lc2); + bool find_uq_expr_in_clauselist = false; + foreach(lc3, clause_list) + { + RestrictInfo *rinfo = lfirst_node(RestrictInfo, lc3); + Node *outer_expr; + if (reverse) + outer_expr = rinfo->outer_is_left ? get_rightop(rinfo->clause) : get_leftop(rinfo->clause); + else + outer_expr = rinfo->outer_is_left ? get_leftop(rinfo->clause) : get_rightop(rinfo->clause); + if (equal(outer_expr, outer_uq_expr)) + { + find_uq_expr_in_clauselist = true; + break; + } + } + if (!find_uq_expr_in_clauselist) + { + /* No need to check the next exprs in the current uniquekey */ + clauselist_matchs_all_exprs = false; + break; + } + } + + if (clauselist_matchs_all_exprs) + return true; + } + return false; +} + + +/* + * relation_is_onerow + * Check if it is a one-row relation by checking UniqueKey. + */ +bool +relation_is_onerow(RelOptInfo *rel) +{ + UniqueKey *ukey; + if (rel->uniquekeys == NIL) + return false; + ukey = linitial_node(UniqueKey, rel->uniquekeys); + return ukey->exprs == NIL && list_length(rel->uniquekeys) == 1; +} + +/* + * relation_has_uniquekeys_for + * Returns true if we have proofs that 'rel' cannot return multiple rows with + * the same values in each of 'exprs'. Otherwise returns false. + */ +bool +relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel, + List *exprs, bool allow_multinulls) +{ + ListCell *lc; + + /* + * For UniqueKey->onerow case, the uniquekey->exprs is empty as well + * so we can't rely on list_is_subset to handle this special cases + */ + if (exprs == NIL) + return false; + + foreach(lc, rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (ukey->multi_nullvals && !allow_multinulls) + continue; + if (list_is_subset(ukey->exprs, exprs)) + return true; + } + return false; +} + + +/* + * get_exprs_from_uniqueindex + * + * Return a list of exprs which is unique. set useful to false if this + * unique index is not useful for us. + */ +static List * +get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals) +{ + List *exprs = NIL; + ListCell *indexpr_item; + int c = 0; + + *useful = true; + *multi_nullvals = false; + + indexpr_item = list_head(unique_index->indexprs); + for(c = 0; c < unique_index->ncolumns; c++) + { + int attr = unique_index->indexkeys[c]; + Expr *expr; + bool matched_const = false; + ListCell *lc1, *lc2; + + if(attr > 0) + { + expr = list_nth_node(TargetEntry, unique_index->indextlist, c)->expr; + } + else if (attr == 0) + { + /* Expression index */ + expr = lfirst(indexpr_item); + indexpr_item = lnext(unique_index->indexprs, indexpr_item); + } + else /* attr < 0 */ + { + /* Index on system column is not supported */ + Assert(false); + } + + /* + * Check index_col = Const case with regarding to opfamily checking + * If we can remove the index_col from the final UniqueKey->exprs. + */ + forboth(lc1, const_exprs, lc2, const_expr_opfamilies) + { + if (list_member_oid((List *)lfirst(lc2), unique_index->opfamily[c]) + && match_index_to_operand((Node *) lfirst(lc1), c, unique_index)) + { + matched_const = true; + break; + } + } + + if (matched_const) + continue; + + /* Check if the indexed expr is used in rel */ + if (attr > 0) + { + /* + * Normal Indexed column, if the col is not used, then the index is useless + * for uniquekey. + */ + attr -= FirstLowInvalidHeapAttributeNumber; + + if (!bms_is_member(attr, used_varattrs)) + { + *useful = false; + break; + } + } + else if (!list_member(unique_index->rel->reltarget->exprs, expr)) + { + /* Expression index but the expression is not used in rel */ + *useful = false; + break; + } + + /* check not null property. */ + if (attr == 0) + { + /* We never know if a expression yields null or not */ + *multi_nullvals = true; + } + else if (!bms_is_member(attr, unique_index->rel->notnullattrs) + && !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber, + unique_index->rel->notnullattrs)) + { + *multi_nullvals = true; + } + + exprs = lappend(exprs, expr); + } + return exprs; +} + + +/* + * add_uniquekey_for_onerow + * If we are sure that the relation only returns one row, then all the columns + * are unique. However we don't need to create UniqueKey for every column, we + * just set exprs = NIL and overwrites all the other UniqueKey on this RelOptInfo + * since this one has strongest semantics. + */ +void +add_uniquekey_for_onerow(RelOptInfo *rel) +{ + /* + * We overwrite the previous UniqueKey on purpose since this one has the + * strongest semantic. + */ + rel->uniquekeys = list_make1(makeUniqueKey(NIL, false)); +} + + +/* + * initililze_uniquecontext_for_joinrel + * Return a List of UniqueKeyContext for an inputrel + */ +static List * +initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel) +{ + List *res = NIL; + ListCell *lc; + foreach(lc, inputrel->uniquekeys) + { + UniqueKeyContext context; + context = palloc(sizeof(struct UniqueKeyContextData)); + context->uniquekey = lfirst_node(UniqueKey, lc); + context->added_to_joinrel = false; + context->useful = true; + res = lappend(res, context); + } + return res; +} + + +/* + * get_exprs_from_uniquekey + * Unify the way of get List of exprs from a one-row UniqueKey or + * normal UniqueKey. for the onerow case, every expr in rel1 is a valid + * UniqueKey. Return a List of exprs. + * + * rel1: The relation which you want to get the exprs. + * ukey: The UniqueKey you want to get the exprs. + */ +static List * +get_exprs_from_uniquekey(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *rel1, UniqueKey *ukey) +{ + ListCell *lc; + bool onerow = rel1 != NULL && relation_is_onerow(rel1); + + List *res = NIL; + Assert(onerow || ukey); + if (onerow) + { + /* Only cares about the exprs still exist in joinrel */ + foreach(lc, joinrel->reltarget->exprs) + { + Bitmapset *relids = pull_varnos(root, lfirst(lc)); + if (bms_is_subset(relids, rel1->relids)) + { + res = lappend(res, list_make1(lfirst(lc))); + } + } + } + else + { + res = list_make1(ukey->exprs); + } + return res; +} + +/* + * Partitioned table Unique Keys. + * The partition table unique key is maintained as: + * 1. The index must be unique as usual. + * 2. The index must contains partition key. + * 3. The index must exist on all the child rel. see simple_indexinfo_equal for + * how we compare it. + */ + +/* + * index_constains_partkey + * return true if the index contains the partiton key. + */ +static bool +index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind) +{ + ListCell *lc; + int i; + Assert(IS_PARTITIONED_REL(partrel)); + Assert(partrel->part_scheme->partnatts > 0); + + for(i = 0; i < partrel->part_scheme->partnatts; i++) + { + Node *part_expr = linitial(partrel->partexprs[i]); + bool found_in_index = false; + foreach(lc, ind->indextlist) + { + Expr *index_expr = lfirst_node(TargetEntry, lc)->expr; + if (equal(index_expr, part_expr)) + { + found_in_index = true; + break; + } + } + if (!found_in_index) + return false; + } + return true; +} + +/* + * simple_indexinfo_equal + * + * Used to check if the 2 index is same as each other. The index here + * is COPIED from childrel and did some tiny changes(see + * simple_copy_indexinfo_to_parent) + */ +static bool +simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2) +{ + Size oid_cmp_len = sizeof(Oid) * ind1->ncolumns; + + return ind1->ncolumns == ind2->ncolumns && + ind1->unique == ind2->unique && + memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0 && + memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0 && + memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0 && + memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0 && + equal(get_tlist_exprs(ind1->indextlist, true), + get_tlist_exprs(ind2->indextlist, true)); +} + + +/* + * The below macros are used for simple_copy_indexinfo_to_parent which is so + * customized that I don't want to put it to copyfuncs.c. So copy it here. + */ +#define COPY_POINTER_FIELD(fldname, sz) \ + do { \ + Size _size = (sz); \ + newnode->fldname = palloc(_size); \ + memcpy(newnode->fldname, from->fldname, _size); \ + } while (0) + +#define COPY_NODE_FIELD(fldname) \ + (newnode->fldname = copyObjectImpl(from->fldname)) + +#define COPY_SCALAR_FIELD(fldname) \ + (newnode->fldname = from->fldname) + + +/* + * simple_copy_indexinfo_to_parent (from partition) + * Copy the IndexInfo from child relation to parent relation with some modification, + * which is used to test: + * 1. If the same index exists in all the childrels. + * 2. If the parentrel->reltarget/basicrestrict info matches this index. + */ +static IndexOptInfo * +simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from) +{ + IndexOptInfo *newnode = makeNode(IndexOptInfo); + AppendRelInfo *appinfo = find_appinfo_by_child(root, from->rel->relid); + ListCell *lc; + int idx = 0; + + COPY_SCALAR_FIELD(ncolumns); + COPY_SCALAR_FIELD(nkeycolumns); + COPY_SCALAR_FIELD(unique); + COPY_SCALAR_FIELD(immediate); + /* We just need to know if it is NIL or not */ + COPY_SCALAR_FIELD(indpred); + COPY_SCALAR_FIELD(predOK); + COPY_POINTER_FIELD(indexkeys, from->ncolumns * sizeof(int)); + COPY_POINTER_FIELD(indexcollations, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opfamily, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opcintype, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(sortopfamily, from->ncolumns * sizeof(Oid)); + COPY_NODE_FIELD(indextlist); + + /* Convert index exprs on child expr to expr on parent */ + foreach(lc, newnode->indextlist) + { + TargetEntry *tle = lfirst_node(TargetEntry, lc); + /* Index on expression is ignored */ + Assert(IsA(tle->expr, Var)); + tle->expr = (Expr *) find_parent_var(appinfo, (Var *) tle->expr); + newnode->indexkeys[idx] = castNode(Var, tle->expr)->varattno; + idx++; + } + newnode->rel = parentrel; + return newnode; +} + +/* + * adjust_partition_unique_indexlist + * + * global_unique_indexes: At the beginning, it contains the copy & modified + * unique index from the first partition. And then check if each index in it still + * exists in the following partitions. If no, remove it. at last, it has an + * index list which exists in all the partitions. + */ +static void +adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_indexes) +{ + ListCell *lc, *lc2; + foreach(lc, *global_unique_indexes) + { + IndexOptInfo *g_ind = lfirst_node(IndexOptInfo, lc); + bool found_in_child = false; + + foreach(lc2, childrel->indexlist) + { + IndexOptInfo *p_ind = lfirst_node(IndexOptInfo, lc2); + IndexOptInfo *p_ind_copy; + if (!p_ind->unique || !p_ind->immediate || + (p_ind->indpred != NIL && !p_ind->predOK)) + continue; + p_ind_copy = simple_copy_indexinfo_to_parent(root, parentrel, p_ind); + if (simple_indexinfo_equal(p_ind_copy, g_ind)) + { + found_in_child = true; + break; + } + } + if (!found_in_child) + /* The index doesn't exist in childrel, remove it from global_unique_indexes */ + *global_unique_indexes = foreach_delete_current(*global_unique_indexes, lc); + } +} + +/* Helper function for groupres/distinctrel */ +static void +add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgroups) +{ + Query *parse = root->parse; + List *exprs; + + /* + * XXX: If there are some vars which is not in current levelsup, the semantic is + * imprecise, should we avoid it or not? levelsup = 1 is just a demo, maybe we need to + * check every level other than 0, if so, looks we have to write another + * pull_var_walker. + */ + List *upper_vars = pull_vars_of_level((Node*)sortgroups, 1); + + if (upper_vars != NIL) + return; + + exprs = get_sortgrouplist_exprs(sortgroups, parse->targetList); + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(exprs, + false /* sortgroupclause can't be multi_nullvals */)); +} + + +/* + * add_combined_uniquekey + * The combination of both UniqueKeys is a valid UniqueKey for joinrel no matter + * the jointype. + */ +bool +add_combined_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype) +{ + + ListCell *lc1, *lc2; + + /* Either side has multi_nullvals or we have outer join, + * the combined UniqueKey has multi_nullvals */ + bool multi_nullvals = outer_ukey->multi_nullvals || + inner_ukey->multi_nullvals || IS_OUTER_JOIN(jointype); + + /* The only case we can get onerow joinrel after join */ + if (relation_is_onerow(outer_rel) + && relation_is_onerow(inner_rel) + && jointype == JOIN_INNER) + { + add_uniquekey_for_onerow(joinrel); + return true; + } + + foreach(lc1, get_exprs_from_uniquekey(root, joinrel, outer_rel, outer_ukey)) + { + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, inner_rel, inner_ukey)) + { + List *exprs = list_concat_copy(lfirst_node(List, lc1), lfirst_node(List, lc2)); + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(exprs, + multi_nullvals)); + } + } + return false; +} diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index 424d25cbd5..8d8e493f5c 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -2387,6 +2387,8 @@ grouping_planner(PlannerInfo *root, bool inheritance_update, add_path(final_rel, path); } + simple_copy_uniquekeys(current_rel, final_rel); + /* * Generate partial paths for final_rel, too, if outer query levels might * be able to make use of them. @@ -3899,6 +3901,8 @@ create_grouping_paths(PlannerInfo *root, } set_cheapest(grouped_rel); + + populate_grouprel_uniquekeys(root, grouped_rel, input_rel); return grouped_rel; } @@ -4621,7 +4625,7 @@ create_window_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(window_rel); - + simple_copy_uniquekeys(input_rel, window_rel); return window_rel; } @@ -4935,7 +4939,7 @@ create_distinct_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(distinct_rel); - + populate_distinctrel_uniquekeys(root, input_rel, distinct_rel); return distinct_rel; } @@ -5196,6 +5200,8 @@ create_ordered_paths(PlannerInfo *root, */ Assert(ordered_rel->pathlist != NIL); + simple_copy_uniquekeys(input_rel, ordered_rel); + return ordered_rel; } @@ -6073,6 +6079,9 @@ adjust_paths_for_srfs(PlannerInfo *root, RelOptInfo *rel, if (list_length(targets) == 1) return; + /* UniqueKey is not valid after handling the SRF. */ + rel->uniquekeys = NIL; + /* * Stack SRF-evaluation nodes atop each path for the rel. * diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c index becdcbb872..b7626545bf 100644 --- a/src/backend/optimizer/prep/prepunion.c +++ b/src/backend/optimizer/prep/prepunion.c @@ -689,6 +689,8 @@ generate_union_paths(SetOperationStmt *op, PlannerInfo *root, /* Undo effects of possibly forcing tuple_fraction to 0 */ root->tuple_fraction = save_fraction; + /* Add the UniqueKeys */ + populate_unionrel_uniquekeys(root, result_rel); return result_rel; } diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c index 86922a273c..6817c9c787 100644 --- a/src/backend/optimizer/util/appendinfo.c +++ b/src/backend/optimizer/util/appendinfo.c @@ -746,3 +746,47 @@ find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos) } return appinfos; } + +/* + * find_appinfo_by_child + * + */ +AppendRelInfo * +find_appinfo_by_child(PlannerInfo *root, Index child_index) +{ + ListCell *lc; + foreach(lc, root->append_rel_list) + { + AppendRelInfo *appinfo = lfirst_node(AppendRelInfo, lc); + if (appinfo->child_relid == child_index) + return appinfo; + } + elog(ERROR, "parent relation cant be found"); + return NULL; +} + +/* + * find_parent_var + * + */ +Var * +find_parent_var(AppendRelInfo *appinfo, Var *child_var) +{ + ListCell *lc; + Var *res = NULL; + Index attno = 1; + foreach(lc, appinfo->translated_vars) + { + Node *child_node = lfirst(lc); + if (equal(child_node, child_var)) + { + res = copyObject(child_var); + res->varattno = attno; + res->varno = appinfo->parent_relid; + } + attno++; + } + if (res == NULL) + elog(ERROR, "parent var can't be found."); + return res; +} diff --git a/src/backend/optimizer/util/inherit.c b/src/backend/optimizer/util/inherit.c index be1c9ddd96..3eec1f4d74 100644 --- a/src/backend/optimizer/util/inherit.c +++ b/src/backend/optimizer/util/inherit.c @@ -736,6 +736,7 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, { Node *onecq = (Node *) lfirst(lc2); bool pseudoconstant; + RestrictInfo *child_rinfo; /* check for pseudoconstant (no Vars or volatile functions) */ pseudoconstant = @@ -747,14 +748,15 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, root->hasPseudoConstantQuals = true; } /* reconstitute RestrictInfo with appropriate properties */ - childquals = lappend(childquals, - make_restrictinfo(root, - (Expr *) onecq, - rinfo->is_pushed_down, - rinfo->outerjoin_delayed, - pseudoconstant, - rinfo->security_level, - NULL, NULL, NULL)); + child_rinfo = make_restrictinfo(root, + (Expr *) onecq, + rinfo->is_pushed_down, + rinfo->outerjoin_delayed, + pseudoconstant, + rinfo->security_level, + NULL, NULL, NULL); + child_rinfo->mergeopfamilies = rinfo->mergeopfamilies; + childquals = lappend(childquals, child_rinfo); /* track minimum security level among child quals */ cq_min_security = Min(cq_min_security, rinfo->security_level); } diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 48a7ebfe45..4fe1824eb0 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -16,6 +16,7 @@ #include "nodes/execnodes.h" #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" extern A_Expr *makeA_Expr(A_Expr_Kind kind, List *name, @@ -106,4 +107,6 @@ extern GroupingSet *makeGroupingSet(GroupingSetKind kind, List *content, int loc extern VacuumRelation *makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols); +extern UniqueKey* makeUniqueKey(List *exprs, bool multi_nullvals); + #endif /* MAKEFUNC_H */ diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index e22df890ef..3320273ac1 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -265,6 +265,7 @@ typedef enum NodeTag T_EquivalenceMember, T_PathKey, T_PathTarget, + T_UniqueKey, T_RestrictInfo, T_IndexClause, T_PlaceHolderVar, diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index 0d61f04d27..80561b1003 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -739,6 +739,7 @@ typedef struct RelOptInfo QualCost baserestrictcost; /* cost of evaluating the above */ Index baserestrict_min_security; /* min security_level found in * baserestrictinfo */ + List *uniquekeys; /* List of UniqueKey */ List *joininfo; /* RestrictInfo structures for join clauses * involving this rel */ bool has_eclass_joins; /* T means joininfo is incomplete */ @@ -1059,6 +1060,28 @@ typedef struct PathKey } PathKey; +/* + * UniqueKey + * + * Represents the unique properties held by a RelOptInfo. + * + * exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL + * is a special case of UniqueKey, which means there is only 1 row in that + * relation. + * multi_nullvals: true means multi null values may exist in these exprs, so the + * uniqueness is not guaranteed in this case. This field is necessary for + * remove_useless_join & reduce_unique_semijoins where we don't mind these + * duplicated NULL values. It is set to true for 2 cases. One is a unique key + * from a unique index but the related column is nullable. The other one is for + * outer join. see populate_joinrel_uniquekeys for detail. + */ +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + /* * PathTarget * @@ -2496,7 +2519,7 @@ typedef enum * * flags indicating what kinds of grouping are possible. * partial_costs_set is true if the agg_partial_costs and agg_final_costs - * have been initialized. + * have been initialized. * agg_partial_costs gives partial aggregation costs. * agg_final_costs gives finalization costs. * target_parallel_safe is true if target is parallel safe. @@ -2526,8 +2549,8 @@ typedef struct * limit_tuples is an estimated bound on the number of output tuples, * or -1 if no LIMIT or couldn't estimate. * count_est and offset_est are the estimated values of the LIMIT and OFFSET - * expressions computed by preprocess_limit() (see comments for - * preprocess_limit() for more information). + * expressions computed by preprocess_limit() (see comments for + * preprocess_limit() for more information). */ typedef struct { diff --git a/src/include/nodes/pg_list.h b/src/include/nodes/pg_list.h index 404e03f132..85c65e910c 100644 --- a/src/include/nodes/pg_list.h +++ b/src/include/nodes/pg_list.h @@ -566,6 +566,16 @@ extern pg_nodiscard List *list_delete_first(List *list); extern pg_nodiscard List *list_delete_last(List *list); extern pg_nodiscard List *list_delete_nth_cell(List *list, int n); extern pg_nodiscard List *list_delete_cell(List *list, ListCell *cell); +extern bool list_is_subset(const List *members, const List *target); + +extern List *list_delete(List *list, void *datum); +extern List *list_delete_ptr(List *list, void *datum); +extern List *list_delete_int(List *list, int datum); +extern List *list_delete_oid(List *list, Oid datum); +extern List *list_delete_first(List *list); +extern List *list_delete_last(List *list); +extern List *list_delete_nth_cell(List *list, int n); +extern List *list_delete_cell(List *list, ListCell *cell); extern List *list_union(const List *list1, const List *list2); extern List *list_union_ptr(const List *list1, const List *list2); diff --git a/src/include/optimizer/appendinfo.h b/src/include/optimizer/appendinfo.h index 4cbf8c26cc..43b8d05f69 100644 --- a/src/include/optimizer/appendinfo.h +++ b/src/include/optimizer/appendinfo.h @@ -32,4 +32,7 @@ extern Relids adjust_child_relids_multilevel(PlannerInfo *root, Relids relids, extern AppendRelInfo **find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos); +extern AppendRelInfo *find_appinfo_by_child(PlannerInfo *root, Index child_index); +extern Var *find_parent_var(AppendRelInfo *appinfo, Var *child_var); + #endif /* APPENDINFO_H */ diff --git a/src/include/optimizer/optimizer.h b/src/include/optimizer/optimizer.h index d587952b7d..843aafa51e 100644 --- a/src/include/optimizer/optimizer.h +++ b/src/include/optimizer/optimizer.h @@ -23,6 +23,7 @@ #define OPTIMIZER_H #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" /* Test if an expression node represents a SRF call. Beware multiple eval! */ #define IS_SRF_CALL(node) \ @@ -168,6 +169,7 @@ extern TargetEntry *get_sortgroupref_tle(Index sortref, List *targetList); extern TargetEntry *get_sortgroupclause_tle(SortGroupClause *sgClause, List *targetList); +extern Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); extern Node *get_sortgroupclause_expr(SortGroupClause *sgClause, List *targetList); extern List *get_sortgrouplist_exprs(List *sgClauses, diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h index 035d3e1206..1adf99c9ee 100644 --- a/src/include/optimizer/paths.h +++ b/src/include/optimizer/paths.h @@ -247,5 +247,48 @@ extern PathKey *make_canonical_pathkey(PlannerInfo *root, int strategy, bool nulls_first); extern void add_paths_to_append_rel(PlannerInfo *root, RelOptInfo *rel, List *live_childrels); +extern List *select_mergejoin_clauses(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + JoinType jointype, + bool *mergejoin_allowed); + +/* + * uniquekeys.c + * Utilities for matching and building unique keys + */ +extern void populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List* unique_index_list); +extern void populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels); +extern void populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel); +extern void populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel); +extern void populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel); +extern void simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel); +extern void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel); +extern void populate_joinrel_uniquekeys(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + RelOptInfo *rel2, + List *restrictlist, + JoinType jointype); + +extern bool relation_has_uniquekeys_for(PlannerInfo *root, + RelOptInfo *rel, + List *exprs, + bool allow_multinulls); +extern bool relation_is_onerow(RelOptInfo *rel); #endif /* PATHS_H */ -- 2.26.2 --yh7cw345mbttadjz Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v38-0003-Extend-UniqueKeys.patch" ^ permalink raw reply [nested|flat] 91+ messages in thread
* [PATCH v36 2/6] Introduce UniqueKey attributes on RelOptInfo struct. @ 2020-05-11 07:50 一挃 <[email protected]> 0 siblings, 0 replies; 91+ messages in thread From: 一挃 @ 2020-05-11 07:50 UTC (permalink / raw) UniqueKey is a set of exprs on RelOptInfo which represents the exprs will be unique on the given RelOptInfo. You can see README.uniquekey for more information. --- src/backend/nodes/copyfuncs.c | 13 + src/backend/nodes/list.c | 31 + src/backend/nodes/makefuncs.c | 13 + src/backend/nodes/outfuncs.c | 11 + src/backend/nodes/readfuncs.c | 10 + src/backend/optimizer/path/Makefile | 3 +- src/backend/optimizer/path/README.uniquekey | 131 +++ src/backend/optimizer/path/allpaths.c | 10 + src/backend/optimizer/path/joinpath.c | 9 +- src/backend/optimizer/path/joinrels.c | 2 + src/backend/optimizer/path/pathkeys.c | 3 +- src/backend/optimizer/path/uniquekeys.c | 1131 +++++++++++++++++++ src/backend/optimizer/plan/planner.c | 13 +- src/backend/optimizer/prep/prepunion.c | 2 + src/backend/optimizer/util/appendinfo.c | 44 + src/backend/optimizer/util/inherit.c | 16 +- src/include/nodes/makefuncs.h | 3 + src/include/nodes/nodes.h | 1 + src/include/nodes/pathnodes.h | 29 +- src/include/nodes/pg_list.h | 2 + src/include/optimizer/appendinfo.h | 3 + src/include/optimizer/optimizer.h | 2 + src/include/optimizer/paths.h | 43 + 23 files changed, 1502 insertions(+), 23 deletions(-) create mode 100644 src/backend/optimizer/path/README.uniquekey create mode 100644 src/backend/optimizer/path/uniquekeys.c diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 89c409de66..1f50400fd2 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2273,6 +2273,16 @@ _copyPathKey(const PathKey *from) return newnode; } +static UniqueKey * +_copyUniqueKey(const UniqueKey *from) +{ + UniqueKey *newnode = makeNode(UniqueKey); + + COPY_NODE_FIELD(exprs); + COPY_SCALAR_FIELD(multi_nullvals); + + return newnode; +} /* * _copyRestrictInfo */ @@ -5152,6 +5162,9 @@ copyObjectImpl(const void *from) case T_PathKey: retval = _copyPathKey(from); break; + case T_UniqueKey: + retval = _copyUniqueKey(from); + break; case T_RestrictInfo: retval = _copyRestrictInfo(from); break; diff --git a/src/backend/nodes/list.c b/src/backend/nodes/list.c index 80fa8c84e4..a7a99b70f2 100644 --- a/src/backend/nodes/list.c +++ b/src/backend/nodes/list.c @@ -687,6 +687,37 @@ list_member_oid(const List *list, Oid datum) return false; } +/* + * return true iff every entry in "members" list is also present + * in the "target" list. + */ +bool +list_is_subset(const List *members, const List *target) +{ + const ListCell *lc1, *lc2; + + Assert(IsPointerList(members)); + Assert(IsPointerList(target)); + check_list_invariants(members); + check_list_invariants(target); + + foreach(lc1, members) + { + bool found = false; + foreach(lc2, target) + { + if (equal(lfirst(lc1), lfirst(lc2))) + { + found = true; + break; + } + } + if (!found) + return false; + } + return true; +} + /* * Delete the n'th cell (counting from 0) in list. * diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 49de285f01..646cf7c9a1 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -814,3 +814,16 @@ makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols) v->va_cols = va_cols; return v; } + + +/* + * makeUniqueKey + */ +UniqueKey* +makeUniqueKey(List *exprs, bool multi_nullvals) +{ + UniqueKey * ukey = makeNode(UniqueKey); + ukey->exprs = exprs; + ukey->multi_nullvals = multi_nullvals; + return ukey; +} diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index e2f177515d..c3a9632992 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2428,6 +2428,14 @@ _outPathKey(StringInfo str, const PathKey *node) WRITE_BOOL_FIELD(pk_nulls_first); } +static void +_outUniqueKey(StringInfo str, const UniqueKey *node) +{ + WRITE_NODE_TYPE("UNIQUEKEY"); + WRITE_NODE_FIELD(exprs); + WRITE_BOOL_FIELD(multi_nullvals); +} + static void _outPathTarget(StringInfo str, const PathTarget *node) { @@ -4127,6 +4135,9 @@ outNode(StringInfo str, const void *obj) case T_PathKey: _outPathKey(str, obj); break; + case T_UniqueKey: + _outUniqueKey(str, obj); + break; case T_PathTarget: _outPathTarget(str, obj); break; diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index 42050ab719..3a18571d0c 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -452,6 +452,14 @@ _readSetOperationStmt(void) READ_DONE(); } +static UniqueKey * +_readUniqueKey(void) +{ + READ_LOCALS(UniqueKey); + READ_NODE_FIELD(exprs); + READ_BOOL_FIELD(multi_nullvals); + READ_DONE(); +} /* * Stuff from primnodes.h. @@ -2656,6 +2664,8 @@ parseNodeString(void) return_value = _readCommonTableExpr(); else if (MATCH("SETOPERATIONSTMT", 16)) return_value = _readSetOperationStmt(); + else if (MATCH("UNIQUEKEY", 9)) + return_value = _readUniqueKey(); else if (MATCH("ALIAS", 5)) return_value = _readAlias(); else if (MATCH("RANGEVAR", 8)) diff --git a/src/backend/optimizer/path/Makefile b/src/backend/optimizer/path/Makefile index 1e199ff66f..7b9820c25f 100644 --- a/src/backend/optimizer/path/Makefile +++ b/src/backend/optimizer/path/Makefile @@ -21,6 +21,7 @@ OBJS = \ joinpath.o \ joinrels.o \ pathkeys.o \ - tidpath.o + tidpath.o \ + uniquekeys.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/optimizer/path/README.uniquekey b/src/backend/optimizer/path/README.uniquekey new file mode 100644 index 0000000000..5eac761995 --- /dev/null +++ b/src/backend/optimizer/path/README.uniquekey @@ -0,0 +1,131 @@ +1. What is UniqueKey? +We can think UniqueKey is a set of exprs for a RelOptInfo, which we are insure +that doesn't yields same result among all the rows. The simplest UniqueKey +format is primary key. + +However we define the UnqiueKey as below. + +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + +exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL +is a special case of UniqueKey, which means there is only one row in that +relation.it has a stronger semantic than others. like SELECT uk FROM t; uk is +normal unique key and may have different values. SELECT colx FROM t WHERE uk = +const. colx is unique AND we have only 1 value. This field can used for +innerrel_is_unique. this logic is handled specially in add_uniquekey_for_onerow +function. + +multi_nullvals: true means multi null values may exist in these exprs, so the +uniqueness is not guaranteed in this case. This field is necessary for +remove_useless_join & reduce_unique_semijoins where we don't mind these +duplicated NULL values. It is set to true for 2 cases. One is a unique key +from a unique index but the related column is nullable. The other one is for +outer join. see populate_joinrel_uniquekeys for detail. + + +The UniqueKey can be used at the following cases at least: +1. remove_useless_joins. +2. reduce_semianti_joins +3. remove distinct node if distinct clause is unique. +4. remove aggnode if group by clause is unique. +5. Index Skip Scan (WIP) +6. Aggregation Push Down without 2 phase aggregation if the join can't + duplicated the aggregated rows. (work in progress feature) + +2. How is it maintained? + +We have a set of populate_xxx_unqiuekeys functions to maintain the uniquekey on +various cases. xxx includes baserel, joinrel, partitionedrel, distinctrel, +groupedrel, unionrel. and we also need to convert the uniquekey from subquery +to outer relation, which is what convert_subquery_uniquekeys does. + +1. The first part is about baserel. We handled 3 cases. suppose we have Unique +Index on (a, b). + +1. SELECT a, b FROM t. UniqueKey (a, b) +2. SELECT a FROM t WHERE b = 1; UniqueKey (a) +3. SELECT .. FROM t WHERE a = 1 AND b = 1; UniqueKey (NIL). onerow case, every + column is Unique. + +2. The next part is joinrel, this part is most error-prone, we simplified the rules +like below: +1. If the relation's UniqueKey can't be duplicated after join, then is will be + still valid for the join rel. The function we used here is + innerrel_keeps_unique. The basic idea is innerrel.any_col = outer.uk. + +2. If the UnqiueKey can't keep valid via the rule 1, the combination of the + UniqueKey from both sides are valid for sure. We can prove this as: if the + unique exprs from rel1 is duplicated by rel2, the duplicated rows must + contains different unique exprs from rel2. + +More considerations about onerow: +1. If relation with one row and it can't be duplicated, it is still possible + contains mulit_nullvas after outer join. +2. If the either UniqueKey can be duplicated after join, the can get one row + only when both side is one row AND there is no outer join. +3. Whenever the onerow UniqueKey is not a valid any more, we need to convert one + row UniqueKey to normal unique key since we don't store exprs for one-row + relation. get_exprs_from_uniquekeys will be used here. + + +More considerations about multi_nullvals after join: +1. If the original UnqiueKey has multi_nullvals, the final UniqueKey will have + mulit_nullvals in any case. +2. If a unique key doesn't allow mulit_nullvals, after some outer join, it + allows some outer join. + + +3. When we comes to subquery, we need to convert_subquery_unqiuekeys just like +convert_subquery_pathkeys. Only the UniqueKey insides subquery is referenced as +a Var in outer relation will be reused. The relationship between the outerrel.Var +and subquery.exprs is built with outerel->subroot->processed_tlist. + + +4. As for the SRF functions, it will break the uniqueness of uniquekey, However it +is handled in adjust_paths_for_srfs, which happens after the query_planner. so +we will maintain the UniqueKey until there and reset it to NIL at that +places. This can't help on distinct/group by elimination cases but probably help +in some other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is +semantic correctly. + + +5. As for inherit table, we first main the UnqiueKey on childrel as well. But for +partitioned table we need to maintain 2 different kinds of +UnqiueKey. 1). UniqueKey on the parent relation 2). UniqueKey on child +relation for partition wise query. + +Example: +CREATE TABLE p (a int not null, b int not null) partition by list (a); +CREATE TABLE p0 partition of p for values in (1); +CREATE TABLE p1 partition of p for values in (2); + +create unique index p0_b on p0(b); +create unique index p1_b on p1(b); + +Now b is only unique on partition level, so the distinct can't be removed on +the following cases. SELECT DISTINCT b FROM p; + +Another example is SELECT DISTINCT a, b FROM p WHERE a = 1; Since only one +partition is chosen, the UniqueKey on child relation is same as the UniqueKey on +parent relation. + +Another usage of UniqueKey on partition level is it be helpful for +partition-wise join. + +As for the UniqueKey on parent table level, it comes with 2 different ways, +1). the UniqueKey is also derived in UniqueKey index, but the index must be same +in all the related children relations and the unique index must contains +Partition Key in it. Example: + +CREATE UNIQUE INDEX p_ab ON p(a, b); -- where a is the partition key. + +-- Query +SELECT a, b FROM p; the (a, b) is a UniqueKey of p. + +2). If the parent relation has only one childrel, the UniqueKey on childrel is + the UniqueKey on parent as well. diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index 484dab0a1a..2ad9d06d7a 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -579,6 +579,12 @@ set_plain_rel_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) */ check_index_predicates(root, rel); + /* + * Now that we've marked which partial indexes are suitable, we can now + * build the relation's unique keys. + */ + populate_baserel_uniquekeys(root, rel, rel->indexlist); + /* Mark rel with estimated output rows, width, etc */ set_baserel_size_estimates(root, rel); } @@ -1310,6 +1316,8 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, /* Add paths to the append relation. */ add_paths_to_append_rel(root, rel, live_childrels); + if (IS_PARTITIONED_REL(rel)) + populate_partitionedrel_uniquekeys(root, rel, live_childrels); } @@ -2383,6 +2391,8 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, pathkeys, required_outer)); } + convert_subquery_uniquekeys(root, rel, sub_final_rel); + /* If outer rel allows parallelism, do same for partial paths. */ if (rel->consider_parallel && bms_is_empty(required_outer)) { diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c index db54a6ba2e..ef0fd2fb0b 100644 --- a/src/backend/optimizer/path/joinpath.c +++ b/src/backend/optimizer/path/joinpath.c @@ -71,13 +71,6 @@ static void consider_parallel_mergejoin(PlannerInfo *root, static void hash_inner_and_outer(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, RelOptInfo *innerrel, JoinType jointype, JoinPathExtraData *extra); -static List *select_mergejoin_clauses(PlannerInfo *root, - RelOptInfo *joinrel, - RelOptInfo *outerrel, - RelOptInfo *innerrel, - List *restrictlist, - JoinType jointype, - bool *mergejoin_allowed); static void generate_mergejoin_paths(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *innerrel, @@ -1927,7 +1920,7 @@ hash_inner_and_outer(PlannerInfo *root, * if it is mergejoinable and involves vars from the two sub-relations * currently of interest. */ -static List * +List * select_mergejoin_clauses(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c index 2d343cd293..b9163ee8ff 100644 --- a/src/backend/optimizer/path/joinrels.c +++ b/src/backend/optimizer/path/joinrels.c @@ -924,6 +924,8 @@ populate_joinrel_with_paths(PlannerInfo *root, RelOptInfo *rel1, /* Apply partitionwise join technique, if possible. */ try_partitionwise_join(root, rel1, rel2, joinrel, sjinfo, restrictlist); + + populate_joinrel_uniquekeys(root, joinrel, rel1, rel2, restrictlist, sjinfo->jointype); } diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index ce9bf87e9b..7e596d4194 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -33,7 +33,6 @@ static bool pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys); static bool matches_boolean_partition_clause(RestrictInfo *rinfo, RelOptInfo *partrel, int partkeycol); -static Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); static bool right_merge_direction(PlannerInfo *root, PathKey *pathkey); @@ -1035,7 +1034,7 @@ convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel, * We need this to ensure that we don't return pathkeys describing values * that are unavailable above the level of the subquery scan. */ -static Var * +Var * find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle) { ListCell *lc; diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c new file mode 100644 index 0000000000..b33bcd2f32 --- /dev/null +++ b/src/backend/optimizer/path/uniquekeys.c @@ -0,0 +1,1131 @@ +/*------------------------------------------------------------------------- + * + * uniquekeys.c + * Utilities for matching and building unique keys + * + * Portions Copyright (c) 2020, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/optimizer/path/uniquekeys.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "nodes/makefuncs.h" +#include "nodes/nodeFuncs.h" +#include "optimizer/pathnode.h" +#include "optimizer/paths.h" +#include "optimizer/appendinfo.h" +#include "optimizer/optimizer.h" +#include "optimizer/tlist.h" +#include "rewrite/rewriteManip.h" + + +/* + * This struct is used to help populate_joinrel_uniquekeys. + * + * added_to_joinrel is true if a uniquekey (from outerrel or innerrel) + * has been added to joinrel. + * useful is true if the exprs of the uniquekey still appears in joinrel. + */ +typedef struct UniqueKeyContextData +{ + UniqueKey *uniquekey; + bool added_to_joinrel; + bool useful; +} *UniqueKeyContext; + +static List *initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel); +static bool innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + bool reverse); + +static List *get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals); +static List *get_exprs_from_uniquekey(RelOptInfo *joinrel, + RelOptInfo *rel1, + UniqueKey *ukey); +static void add_uniquekey_for_onerow(RelOptInfo *rel); +static bool add_combined_uniquekey(RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype); + +/* Used for unique indexes checking for partitioned table */ +static bool index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind); +static IndexOptInfo *simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from); +static bool simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2); +static void adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_index); + +/* Helper function for grouped relation and distinct relation. */ +static void add_uniquekey_from_sortgroups(PlannerInfo *root, + RelOptInfo *rel, + List *sortgroups); + +/* + * populate_baserel_uniquekeys + * Populate 'baserel' uniquekeys list by looking at the rel's unique index + * and baserestrictinfo + */ +void +populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List *indexlist) +{ + ListCell *lc; + List *matched_uniq_indexes = NIL; + + /* Attrs appears in rel->reltarget->exprs. */ + Bitmapset *used_attrs = NULL; + + List *const_exprs = NIL; + List *expr_opfamilies = NIL; + + Assert(baserel->rtekind == RTE_RELATION); + + foreach(lc, indexlist) + { + IndexOptInfo *ind = (IndexOptInfo *) lfirst(lc); + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + matched_uniq_indexes = lappend(matched_uniq_indexes, ind); + } + + if (matched_uniq_indexes == NIL) + return; + + /* Check which attrs is used in baserel->reltarget */ + pull_varattnos((Node *)baserel->reltarget->exprs, baserel->relid, &used_attrs); + + /* Check which attrno is used at a mergeable const filter */ + foreach(lc, baserel->baserestrictinfo) + { + RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); + + if (rinfo->mergeopfamilies == NIL) + continue; + + if (bms_is_empty(rinfo->left_relids)) + { + const_exprs = lappend(const_exprs, get_rightop(rinfo->clause)); + } + else if (bms_is_empty(rinfo->right_relids)) + { + const_exprs = lappend(const_exprs, get_leftop(rinfo->clause)); + } + else + continue; + + expr_opfamilies = lappend(expr_opfamilies, rinfo->mergeopfamilies); + } + + foreach(lc, matched_uniq_indexes) + { + bool multi_nullvals, useful; + List *exprs = get_exprs_from_uniqueindex(lfirst_node(IndexOptInfo, lc), + const_exprs, + expr_opfamilies, + used_attrs, + &useful, + &multi_nullvals); + if (useful) + { + if (exprs == NIL) + { + /* All the columns in Unique Index matched with a restrictinfo */ + add_uniquekey_for_onerow(baserel); + return; + } + baserel->uniquekeys = lappend(baserel->uniquekeys, + makeUniqueKey(exprs, multi_nullvals)); + } + } +} + + +/* + * populate_partitionedrel_uniquekeys + * The UniqueKey on partitionrel comes from 2 cases: + * 1). Only one partition is involved in this query, the unique key can be + * copied to parent rel from childrel. + * 2). There are some unique index which includes partition key and exists + * in all the related partitions. + * We never mind rule 2 if we hit rule 1. + */ + +void +populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels) +{ + ListCell *lc; + List *global_uniq_indexlist = NIL; + RelOptInfo *childrel; + bool is_first = true; + + Assert(IS_PARTITIONED_REL(rel)); + + if (childrels == NIL) + return; + + /* + * If there is only one partition used in this query, the UniqueKey in childrel is + * still valid in parent level, but we need convert the format from child expr to + * parent expr. + */ + if (list_length(childrels) == 1) + { + /* Check for Rule 1 */ + RelOptInfo *childrel = linitial_node(RelOptInfo, childrels); + ListCell *lc; + Assert(childrel->reloptkind == RELOPT_OTHER_MEMBER_REL); + if (relation_is_onerow(childrel)) + { + add_uniquekey_for_onerow(rel); + return; + } + + foreach(lc, childrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + AppendRelInfo *appinfo = find_appinfo_by_child(root, childrel->relid); + List *parent_exprs = NIL; + bool can_reuse = true; + ListCell *lc2; + foreach(lc2, ukey->exprs) + { + Var *var = (Var *)lfirst(lc2); + /* + * If the expr comes from a expression, it is hard to build the expression + * in parent so ignore that case for now. + */ + if(!IsA(var, Var)) + { + can_reuse = false; + break; + } + /* Convert it to parent var */ + parent_exprs = lappend(parent_exprs, find_parent_var(appinfo, var)); + } + if (can_reuse) + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(parent_exprs, + ukey->multi_nullvals)); + } + } + else + { + /* Check for rule 2 */ + childrel = linitial_node(RelOptInfo, childrels); + foreach(lc, childrel->indexlist) + { + IndexOptInfo *ind = lfirst(lc); + IndexOptInfo *modified_index; + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + + /* + * During simple_copy_indexinfo_to_parent, we need to convert var from + * child var to parent var, index on expression is too complex to handle. + * so ignore it for now. + */ + if (ind->indexprs != NIL) + continue; + + modified_index = simple_copy_indexinfo_to_parent(root, rel, ind); + /* + * If the unique index doesn't contain partkey, then it is unique + * on this partition only, so it is useless for us. + */ + if (!index_constains_partkey(rel, modified_index)) + continue; + + global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index); + } + + if (global_uniq_indexlist != NIL) + { + foreach(lc, childrels) + { + RelOptInfo *child = lfirst(lc); + if (is_first) + { + is_first = false; + continue; + } + adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist); + } + /* Now we have a list of unique index which are exactly same on all childrels, + * Set the UniqueKey just like it is non-partition table + */ + populate_baserel_uniquekeys(root, rel, global_uniq_indexlist); + } + } +} + + +/* + * populate_distinctrel_uniquekeys + */ +void +populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel) +{ + /* The unique key before the distinct is still valid. */ + distinctrel->uniquekeys = list_copy(inputrel->uniquekeys); + add_uniquekey_from_sortgroups(root, distinctrel, root->parse->distinctClause); +} + +/* + * populate_grouprel_uniquekeys + */ +void +populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel) + +{ + Query *parse = root->parse; + bool input_ukey_added = false; + ListCell *lc; + + if (relation_is_onerow(inputrel)) + { + add_uniquekey_for_onerow(grouprel); + return; + } + if (parse->groupingSets) + return; + + /* A Normal group by without grouping set. */ + if (parse->groupClause) + { + /* + * Current even the groupby clause is Unique already, but if query has aggref + * We have to create grouprel still. To keep the UnqiueKey short, we will check + * the UniqueKey of input_rel still valid, if so we reuse it. + */ + foreach(lc, inputrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (list_is_subset(ukey->exprs, grouprel->reltarget->exprs)) + { + grouprel->uniquekeys = lappend(grouprel->uniquekeys, + ukey); + input_ukey_added = true; + } + } + if (!input_ukey_added) + /* + * group by clause must be a super-set of grouprel->reltarget->exprs except the + * aggregation expr, so if such exprs is unique already, no bother to generate + * new uniquekey for group by exprs. + */ + add_uniquekey_from_sortgroups(root, + grouprel, + root->parse->groupClause); + } + else + /* It has aggregation but without a group by, so only one row returned */ + add_uniquekey_for_onerow(grouprel); +} + +/* + * simple_copy_uniquekeys + * Using a function for the one-line code makes us easy to check where we simply + * copied the uniquekey. + */ +void +simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel) +{ + newrel->uniquekeys = oldrel->uniquekeys; +} + +/* + * populate_unionrel_uniquekeys + */ +void +populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel) +{ + ListCell *lc; + List *exprs = NIL; + + Assert(unionrel->uniquekeys == NIL); + + foreach(lc, unionrel->reltarget->exprs) + { + exprs = lappend(exprs, lfirst(lc)); + } + + if (exprs == NIL) + /* SQL: select union select; is valid, we need to handle it here. */ + add_uniquekey_for_onerow(unionrel); + else + unionrel->uniquekeys = lappend(unionrel->uniquekeys, + makeUniqueKey(exprs,false)); + +} + +/* + * populate_joinrel_uniquekeys + * + * populate uniquekeys for joinrel. We will check each relation to see if its + * UniqueKey is still valid via innerrel_keeps_unique, if so, we add it to + * joinrel. The multi_nullvals field will be changed to true for some outer + * join cases and one-row UniqueKey needs to be converted to normal UniqueKey + * for the same case as well. + * For the uniquekey in either baserel which can't be unique after join, we still + * check to see if combination of UniqueKeys from both side is still useful for us. + * if yes, we add it to joinrel as well. + */ +void +populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *outerrel, RelOptInfo *innerrel, + List *restrictlist, JoinType jointype) +{ + ListCell *lc, *lc2; + List *clause_list = NIL; + List *outerrel_ukey_ctx; + List *innerrel_ukey_ctx; + bool inner_onerow, outer_onerow; + bool mergejoin_allowed; + + /* Care about the outerrel relation only for SEMI/ANTI join */ + if (jointype == JOIN_SEMI || jointype == JOIN_ANTI) + { + foreach(lc, outerrel->uniquekeys) + { + UniqueKey *uniquekey = lfirst_node(UniqueKey, lc); + if (list_is_subset(uniquekey->exprs, joinrel->reltarget->exprs)) + joinrel->uniquekeys = lappend(joinrel->uniquekeys, uniquekey); + } + return; + } + + Assert(jointype == JOIN_LEFT || jointype == JOIN_FULL || jointype == JOIN_INNER); + + /* Fast path */ + if (innerrel->uniquekeys == NIL || outerrel->uniquekeys == NIL) + return; + + inner_onerow = relation_is_onerow(innerrel); + outer_onerow = relation_is_onerow(outerrel); + + outerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(outerrel); + innerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(innerrel); + + clause_list = select_mergejoin_clauses(root, joinrel, outerrel, innerrel, + restrictlist, jointype, + &mergejoin_allowed); + + if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true /* reverse */)) + { + bool outer_impact = jointype == JOIN_FULL; + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + /* Outer relation has one row, and the unique key is not duplicated after join, + * the joinrel will still has one row unless the jointype == JOIN_FULL. + */ + if (outer_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (outer_onerow) + { + /* + * The onerow outerrel becomes multi rows and multi_nullvals + * will be changed to true. We also need to set the exprs correctly since it + * can't be NIL any more. + */ + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(joinrel, outerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Change multi_nullvals to true due to the full join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, true)); + else + /* Just reuse it */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + } + ctx->added_to_joinrel = true; + } + } + + if (innerrel_keeps_unique(root, outerrel, innerrel, clause_list, false)) + { + bool outer_impact = jointype == JOIN_FULL || jointype == JOIN_LEFT;; + + foreach(lc, innerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + if (inner_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (inner_onerow) + { + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(joinrel, innerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Need to change multi_nullvals to true due to the outer join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, + true)); + else + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + + } + ctx->added_to_joinrel = true; + } + } + + /* + * The combination of the UniqueKey from both sides is unique as well regardless + * of join type, but no bother to add it if its subset has been added to joinrel + * already or it is not useful for the joinrel. + */ + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx1 = (UniqueKeyContext) lfirst(lc); + if (ctx1->added_to_joinrel || !ctx1->useful) + continue; + foreach(lc2, innerrel_ukey_ctx) + { + UniqueKeyContext ctx2 = (UniqueKeyContext) lfirst(lc2); + if (ctx2->added_to_joinrel || !ctx2->useful) + continue; + if (add_combined_uniquekey(joinrel, outerrel, innerrel, + ctx1->uniquekey, ctx2->uniquekey, + jointype)) + /* If we set a onerow UniqueKey to joinrel, we don't need other. */ + return; + } + } +} + + +/* + * convert_subquery_uniquekeys + * + * Covert the UniqueKey in subquery to outer relation. + */ +void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel) +{ + ListCell *lc; + + if (sub_final_rel->uniquekeys == NIL) + return; + + if (relation_is_onerow(sub_final_rel)) + { + add_uniquekey_for_onerow(currel); + return; + } + + Assert(currel->subroot != NULL); + + foreach(lc, sub_final_rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + ListCell *lc; + List *exprs = NIL; + bool ukey_useful = true; + + /* One row case is handled above */ + Assert(ukey->exprs != NIL); + foreach(lc, ukey->exprs) + { + Var *var; + TargetEntry *tle = tlist_member(lfirst(lc), + currel->subroot->processed_tlist); + if (tle == NULL) + { + ukey_useful = false; + break; + } + var = find_var_for_subquery_tle(currel, tle); + if (var == NULL) + { + ukey_useful = false; + break; + } + exprs = lappend(exprs, var); + } + + if (ukey_useful) + currel->uniquekeys = lappend(currel->uniquekeys, + makeUniqueKey(exprs, + ukey->multi_nullvals)); + + } +} + +/* + * innerrel_keeps_unique + * + * Check if Unique key of the innerrel is valid after join. innerrel's UniqueKey + * will be still valid if innerrel's any-column mergeop outrerel's uniquekey + * exists in clause_list. + * + * Note: the clause_list must be a list of mergeable restrictinfo already. + */ +static bool +innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *clause_list, + bool reverse) +{ + ListCell *lc, *lc2, *lc3; + + if (outerrel->uniquekeys == NIL || innerrel->uniquekeys == NIL) + return false; + + /* Check if there is outerrel's uniquekey in mergeable clause. */ + foreach(lc, outerrel->uniquekeys) + { + List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs; + bool clauselist_matchs_all_exprs = true; + foreach(lc2, outer_uq_exprs) + { + Node *outer_uq_expr = lfirst(lc2); + bool find_uq_expr_in_clauselist = false; + foreach(lc3, clause_list) + { + RestrictInfo *rinfo = lfirst_node(RestrictInfo, lc3); + Node *outer_expr; + if (reverse) + outer_expr = rinfo->outer_is_left ? get_rightop(rinfo->clause) : get_leftop(rinfo->clause); + else + outer_expr = rinfo->outer_is_left ? get_leftop(rinfo->clause) : get_rightop(rinfo->clause); + if (equal(outer_expr, outer_uq_expr)) + { + find_uq_expr_in_clauselist = true; + break; + } + } + if (!find_uq_expr_in_clauselist) + { + /* No need to check the next exprs in the current uniquekey */ + clauselist_matchs_all_exprs = false; + break; + } + } + + if (clauselist_matchs_all_exprs) + return true; + } + return false; +} + + +/* + * relation_is_onerow + * Check if it is a one-row relation by checking UniqueKey. + */ +bool +relation_is_onerow(RelOptInfo *rel) +{ + UniqueKey *ukey; + if (rel->uniquekeys == NIL) + return false; + ukey = linitial_node(UniqueKey, rel->uniquekeys); + return ukey->exprs == NIL && list_length(rel->uniquekeys) == 1; +} + +/* + * relation_has_uniquekeys_for + * Returns true if we have proofs that 'rel' cannot return multiple rows with + * the same values in each of 'exprs'. Otherwise returns false. + */ +bool +relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel, + List *exprs, bool allow_multinulls) +{ + ListCell *lc; + + /* + * For UniqueKey->onerow case, the uniquekey->exprs is empty as well + * so we can't rely on list_is_subset to handle this special cases + */ + if (exprs == NIL) + return false; + + foreach(lc, rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (ukey->multi_nullvals && !allow_multinulls) + continue; + if (list_is_subset(ukey->exprs, exprs)) + return true; + } + return false; +} + + +/* + * get_exprs_from_uniqueindex + * + * Return a list of exprs which is unique. set useful to false if this + * unique index is not useful for us. + */ +static List * +get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals) +{ + List *exprs = NIL; + ListCell *indexpr_item; + int c = 0; + + *useful = true; + *multi_nullvals = false; + + indexpr_item = list_head(unique_index->indexprs); + for(c = 0; c < unique_index->ncolumns; c++) + { + int attr = unique_index->indexkeys[c]; + Expr *expr; + bool matched_const = false; + ListCell *lc1, *lc2; + + if(attr > 0) + { + expr = list_nth_node(TargetEntry, unique_index->indextlist, c)->expr; + } + else if (attr == 0) + { + /* Expression index */ + expr = lfirst(indexpr_item); + indexpr_item = lnext(unique_index->indexprs, indexpr_item); + } + else /* attr < 0 */ + { + /* Index on system column is not supported */ + Assert(false); + } + + /* + * Check index_col = Const case with regarding to opfamily checking + * If we can remove the index_col from the final UniqueKey->exprs. + */ + forboth(lc1, const_exprs, lc2, const_expr_opfamilies) + { + if (list_member_oid((List *)lfirst(lc2), unique_index->opfamily[c]) + && match_index_to_operand((Node *) lfirst(lc1), c, unique_index)) + { + matched_const = true; + break; + } + } + + if (matched_const) + continue; + + /* Check if the indexed expr is used in rel */ + if (attr > 0) + { + /* + * Normal Indexed column, if the col is not used, then the index is useless + * for uniquekey. + */ + attr -= FirstLowInvalidHeapAttributeNumber; + + if (!bms_is_member(attr, used_varattrs)) + { + *useful = false; + break; + } + } + else if (!list_member(unique_index->rel->reltarget->exprs, expr)) + { + /* Expression index but the expression is not used in rel */ + *useful = false; + break; + } + + /* check not null property. */ + if (attr == 0) + { + /* We never know if a expression yields null or not */ + *multi_nullvals = true; + } + else if (!bms_is_member(attr, unique_index->rel->notnullattrs) + && !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber, + unique_index->rel->notnullattrs)) + { + *multi_nullvals = true; + } + + exprs = lappend(exprs, expr); + } + return exprs; +} + + +/* + * add_uniquekey_for_onerow + * If we are sure that the relation only returns one row, then all the columns + * are unique. However we don't need to create UniqueKey for every column, we + * just set exprs = NIL and overwrites all the other UniqueKey on this RelOptInfo + * since this one has strongest semantics. + */ +void +add_uniquekey_for_onerow(RelOptInfo *rel) +{ + /* + * We overwrite the previous UniqueKey on purpose since this one has the + * strongest semantic. + */ + rel->uniquekeys = list_make1(makeUniqueKey(NIL, false)); +} + + +/* + * initililze_uniquecontext_for_joinrel + * Return a List of UniqueKeyContext for an inputrel + */ +static List * +initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel) +{ + List *res = NIL; + ListCell *lc; + foreach(lc, inputrel->uniquekeys) + { + UniqueKeyContext context; + context = palloc(sizeof(struct UniqueKeyContextData)); + context->uniquekey = lfirst_node(UniqueKey, lc); + context->added_to_joinrel = false; + context->useful = true; + res = lappend(res, context); + } + return res; +} + + +/* + * get_exprs_from_uniquekey + * Unify the way of get List of exprs from a one-row UniqueKey or + * normal UniqueKey. for the onerow case, every expr in rel1 is a valid + * UniqueKey. Return a List of exprs. + * + * rel1: The relation which you want to get the exprs. + * ukey: The UniqueKey you want to get the exprs. + */ +static List * +get_exprs_from_uniquekey(RelOptInfo *joinrel, RelOptInfo *rel1, UniqueKey *ukey) +{ + ListCell *lc; + bool onerow = rel1 != NULL && relation_is_onerow(rel1); + + List *res = NIL; + Assert(onerow || ukey); + if (onerow) + { + /* Only cares about the exprs still exist in joinrel */ + foreach(lc, joinrel->reltarget->exprs) + { + Bitmapset *relids = pull_varnos(lfirst(lc)); + if (bms_is_subset(relids, rel1->relids)) + { + res = lappend(res, list_make1(lfirst(lc))); + } + } + } + else + { + res = list_make1(ukey->exprs); + } + return res; +} + +/* + * Partitioned table Unique Keys. + * The partition table unique key is maintained as: + * 1. The index must be unique as usual. + * 2. The index must contains partition key. + * 3. The index must exist on all the child rel. see simple_indexinfo_equal for + * how we compare it. + */ + +/* + * index_constains_partkey + * return true if the index contains the partiton key. + */ +static bool +index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind) +{ + ListCell *lc; + int i; + Assert(IS_PARTITIONED_REL(partrel)); + Assert(partrel->part_scheme->partnatts > 0); + + for(i = 0; i < partrel->part_scheme->partnatts; i++) + { + Node *part_expr = linitial(partrel->partexprs[i]); + bool found_in_index = false; + foreach(lc, ind->indextlist) + { + Expr *index_expr = lfirst_node(TargetEntry, lc)->expr; + if (equal(index_expr, part_expr)) + { + found_in_index = true; + break; + } + } + if (!found_in_index) + return false; + } + return true; +} + +/* + * simple_indexinfo_equal + * + * Used to check if the 2 index is same as each other. The index here + * is COPIED from childrel and did some tiny changes(see + * simple_copy_indexinfo_to_parent) + */ +static bool +simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2) +{ + Size oid_cmp_len = sizeof(Oid) * ind1->ncolumns; + + return ind1->ncolumns == ind2->ncolumns && + ind1->unique == ind2->unique && + memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0 && + memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0 && + memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0 && + memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0 && + equal(get_tlist_exprs(ind1->indextlist, true), + get_tlist_exprs(ind2->indextlist, true)); +} + + +/* + * The below macros are used for simple_copy_indexinfo_to_parent which is so + * customized that I don't want to put it to copyfuncs.c. So copy it here. + */ +#define COPY_POINTER_FIELD(fldname, sz) \ + do { \ + Size _size = (sz); \ + newnode->fldname = palloc(_size); \ + memcpy(newnode->fldname, from->fldname, _size); \ + } while (0) + +#define COPY_NODE_FIELD(fldname) \ + (newnode->fldname = copyObjectImpl(from->fldname)) + +#define COPY_SCALAR_FIELD(fldname) \ + (newnode->fldname = from->fldname) + + +/* + * simple_copy_indexinfo_to_parent (from partition) + * Copy the IndexInfo from child relation to parent relation with some modification, + * which is used to test: + * 1. If the same index exists in all the childrels. + * 2. If the parentrel->reltarget/basicrestrict info matches this index. + */ +static IndexOptInfo * +simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from) +{ + IndexOptInfo *newnode = makeNode(IndexOptInfo); + AppendRelInfo *appinfo = find_appinfo_by_child(root, from->rel->relid); + ListCell *lc; + int idx = 0; + + COPY_SCALAR_FIELD(ncolumns); + COPY_SCALAR_FIELD(nkeycolumns); + COPY_SCALAR_FIELD(unique); + COPY_SCALAR_FIELD(immediate); + /* We just need to know if it is NIL or not */ + COPY_SCALAR_FIELD(indpred); + COPY_SCALAR_FIELD(predOK); + COPY_POINTER_FIELD(indexkeys, from->ncolumns * sizeof(int)); + COPY_POINTER_FIELD(indexcollations, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opfamily, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opcintype, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(sortopfamily, from->ncolumns * sizeof(Oid)); + COPY_NODE_FIELD(indextlist); + + /* Convert index exprs on child expr to expr on parent */ + foreach(lc, newnode->indextlist) + { + TargetEntry *tle = lfirst_node(TargetEntry, lc); + /* Index on expression is ignored */ + Assert(IsA(tle->expr, Var)); + tle->expr = (Expr *) find_parent_var(appinfo, (Var *) tle->expr); + newnode->indexkeys[idx] = castNode(Var, tle->expr)->varattno; + idx++; + } + newnode->rel = parentrel; + return newnode; +} + +/* + * adjust_partition_unique_indexlist + * + * global_unique_indexes: At the beginning, it contains the copy & modified + * unique index from the first partition. And then check if each index in it still + * exists in the following partitions. If no, remove it. at last, it has an + * index list which exists in all the partitions. + */ +static void +adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_indexes) +{ + ListCell *lc, *lc2; + foreach(lc, *global_unique_indexes) + { + IndexOptInfo *g_ind = lfirst_node(IndexOptInfo, lc); + bool found_in_child = false; + + foreach(lc2, childrel->indexlist) + { + IndexOptInfo *p_ind = lfirst_node(IndexOptInfo, lc2); + IndexOptInfo *p_ind_copy; + if (!p_ind->unique || !p_ind->immediate || + (p_ind->indpred != NIL && !p_ind->predOK)) + continue; + p_ind_copy = simple_copy_indexinfo_to_parent(root, parentrel, p_ind); + if (simple_indexinfo_equal(p_ind_copy, g_ind)) + { + found_in_child = true; + break; + } + } + if (!found_in_child) + /* The index doesn't exist in childrel, remove it from global_unique_indexes */ + *global_unique_indexes = foreach_delete_current(*global_unique_indexes, lc); + } +} + +/* Helper function for groupres/distinctrel */ +static void +add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgroups) +{ + Query *parse = root->parse; + List *exprs; + + /* + * XXX: If there are some vars which is not in current levelsup, the semantic is + * imprecise, should we avoid it or not? levelsup = 1 is just a demo, maybe we need to + * check every level other than 0, if so, looks we have to write another + * pull_var_walker. + */ + List *upper_vars = pull_vars_of_level((Node*)sortgroups, 1); + + if (upper_vars != NIL) + return; + + exprs = get_sortgrouplist_exprs(sortgroups, parse->targetList); + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(exprs, + false /* sortgroupclause can't be multi_nullvals */)); +} + + +/* + * add_combined_uniquekey + * The combination of both UniqueKeys is a valid UniqueKey for joinrel no matter + * the jointype. + */ +bool +add_combined_uniquekey(RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype) +{ + + ListCell *lc1, *lc2; + + /* Either side has multi_nullvals or we have outer join, + * the combined UniqueKey has multi_nullvals */ + bool multi_nullvals = outer_ukey->multi_nullvals || + inner_ukey->multi_nullvals || IS_OUTER_JOIN(jointype); + + /* The only case we can get onerow joinrel after join */ + if (relation_is_onerow(outer_rel) + && relation_is_onerow(inner_rel) + && jointype == JOIN_INNER) + { + add_uniquekey_for_onerow(joinrel); + return true; + } + + foreach(lc1, get_exprs_from_uniquekey(joinrel, outer_rel, outer_ukey)) + { + foreach(lc2, get_exprs_from_uniquekey(joinrel, inner_rel, inner_ukey)) + { + List *exprs = list_concat_copy(lfirst_node(List, lc1), lfirst_node(List, lc2)); + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(exprs, + multi_nullvals)); + } + } + return false; +} diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index b406d41e91..0551ae0512 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -2389,6 +2389,8 @@ grouping_planner(PlannerInfo *root, bool inheritance_update, add_path(final_rel, path); } + simple_copy_uniquekeys(current_rel, final_rel); + /* * Generate partial paths for final_rel, too, if outer query levels might * be able to make use of them. @@ -3899,6 +3901,8 @@ create_grouping_paths(PlannerInfo *root, } set_cheapest(grouped_rel); + + populate_grouprel_uniquekeys(root, grouped_rel, input_rel); return grouped_rel; } @@ -4615,7 +4619,7 @@ create_window_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(window_rel); - + simple_copy_uniquekeys(input_rel, window_rel); return window_rel; } @@ -4911,7 +4915,7 @@ create_distinct_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(distinct_rel); - + populate_distinctrel_uniquekeys(root, input_rel, distinct_rel); return distinct_rel; } @@ -5172,6 +5176,8 @@ create_ordered_paths(PlannerInfo *root, */ Assert(ordered_rel->pathlist != NIL); + simple_copy_uniquekeys(input_rel, ordered_rel); + return ordered_rel; } @@ -6049,6 +6055,9 @@ adjust_paths_for_srfs(PlannerInfo *root, RelOptInfo *rel, if (list_length(targets) == 1) return; + /* UniqueKey is not valid after handling the SRF. */ + rel->uniquekeys = NIL; + /* * Stack SRF-evaluation nodes atop each path for the rel. * diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c index 951aed80e7..e94e92937c 100644 --- a/src/backend/optimizer/prep/prepunion.c +++ b/src/backend/optimizer/prep/prepunion.c @@ -689,6 +689,8 @@ generate_union_paths(SetOperationStmt *op, PlannerInfo *root, /* Undo effects of possibly forcing tuple_fraction to 0 */ root->tuple_fraction = save_fraction; + /* Add the UniqueKeys */ + populate_unionrel_uniquekeys(root, result_rel); return result_rel; } diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c index d722063cf3..44c37ecffc 100644 --- a/src/backend/optimizer/util/appendinfo.c +++ b/src/backend/optimizer/util/appendinfo.c @@ -746,3 +746,47 @@ find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos) } return appinfos; } + +/* + * find_appinfo_by_child + * + */ +AppendRelInfo * +find_appinfo_by_child(PlannerInfo *root, Index child_index) +{ + ListCell *lc; + foreach(lc, root->append_rel_list) + { + AppendRelInfo *appinfo = lfirst_node(AppendRelInfo, lc); + if (appinfo->child_relid == child_index) + return appinfo; + } + elog(ERROR, "parent relation cant be found"); + return NULL; +} + +/* + * find_parent_var + * + */ +Var * +find_parent_var(AppendRelInfo *appinfo, Var *child_var) +{ + ListCell *lc; + Var *res = NULL; + Index attno = 1; + foreach(lc, appinfo->translated_vars) + { + Node *child_node = lfirst(lc); + if (equal(child_node, child_var)) + { + res = copyObject(child_var); + res->varattno = attno; + res->varno = appinfo->parent_relid; + } + attno++; + } + if (res == NULL) + elog(ERROR, "parent var can't be found."); + return res; +} diff --git a/src/backend/optimizer/util/inherit.c b/src/backend/optimizer/util/inherit.c index 3132fd35a5..d66b40ec50 100644 --- a/src/backend/optimizer/util/inherit.c +++ b/src/backend/optimizer/util/inherit.c @@ -736,6 +736,7 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, { Node *onecq = (Node *) lfirst(lc2); bool pseudoconstant; + RestrictInfo *child_rinfo; /* check for pseudoconstant (no Vars or volatile functions) */ pseudoconstant = @@ -747,13 +748,14 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, root->hasPseudoConstantQuals = true; } /* reconstitute RestrictInfo with appropriate properties */ - childquals = lappend(childquals, - make_restrictinfo((Expr *) onecq, - rinfo->is_pushed_down, - rinfo->outerjoin_delayed, - pseudoconstant, - rinfo->security_level, - NULL, NULL, NULL)); + child_rinfo = make_restrictinfo((Expr *) onecq, + rinfo->is_pushed_down, + rinfo->outerjoin_delayed, + pseudoconstant, + rinfo->security_level, + NULL, NULL, NULL); + child_rinfo->mergeopfamilies = rinfo->mergeopfamilies; + childquals = lappend(childquals, child_rinfo); /* track minimum security level among child quals */ cq_min_security = Min(cq_min_security, rinfo->security_level); } diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 31d9aedeeb..c83f17acb7 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -16,6 +16,7 @@ #include "nodes/execnodes.h" #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" extern A_Expr *makeA_Expr(A_Expr_Kind kind, List *name, @@ -105,4 +106,6 @@ extern GroupingSet *makeGroupingSet(GroupingSetKind kind, List *content, int loc extern VacuumRelation *makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols); +extern UniqueKey* makeUniqueKey(List *exprs, bool multi_nullvals); + #endif /* MAKEFUNC_H */ diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 381d84b4e4..41110ed888 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -264,6 +264,7 @@ typedef enum NodeTag T_EquivalenceMember, T_PathKey, T_PathTarget, + T_UniqueKey, T_RestrictInfo, T_IndexClause, T_PlaceHolderVar, diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index 9e3ebd488a..02e4458bef 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -730,6 +730,7 @@ typedef struct RelOptInfo QualCost baserestrictcost; /* cost of evaluating the above */ Index baserestrict_min_security; /* min security_level found in * baserestrictinfo */ + List *uniquekeys; /* List of UniqueKey */ List *joininfo; /* RestrictInfo structures for join clauses * involving this rel */ bool has_eclass_joins; /* T means joininfo is incomplete */ @@ -1047,6 +1048,28 @@ typedef struct PathKey } PathKey; +/* + * UniqueKey + * + * Represents the unique properties held by a RelOptInfo. + * + * exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL + * is a special case of UniqueKey, which means there is only 1 row in that + * relation. + * multi_nullvals: true means multi null values may exist in these exprs, so the + * uniqueness is not guaranteed in this case. This field is necessary for + * remove_useless_join & reduce_unique_semijoins where we don't mind these + * duplicated NULL values. It is set to true for 2 cases. One is a unique key + * from a unique index but the related column is nullable. The other one is for + * outer join. see populate_joinrel_uniquekeys for detail. + */ +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + /* * PathTarget * @@ -2473,7 +2496,7 @@ typedef enum * * flags indicating what kinds of grouping are possible. * partial_costs_set is true if the agg_partial_costs and agg_final_costs - * have been initialized. + * have been initialized. * agg_partial_costs gives partial aggregation costs. * agg_final_costs gives finalization costs. * target_parallel_safe is true if target is parallel safe. @@ -2503,8 +2526,8 @@ typedef struct * limit_tuples is an estimated bound on the number of output tuples, * or -1 if no LIMIT or couldn't estimate. * count_est and offset_est are the estimated values of the LIMIT and OFFSET - * expressions computed by preprocess_limit() (see comments for - * preprocess_limit() for more information). + * expressions computed by preprocess_limit() (see comments for + * preprocess_limit() for more information). */ typedef struct { diff --git a/src/include/nodes/pg_list.h b/src/include/nodes/pg_list.h index 14ea2766ad..621f54a9f8 100644 --- a/src/include/nodes/pg_list.h +++ b/src/include/nodes/pg_list.h @@ -528,6 +528,8 @@ extern bool list_member_ptr(const List *list, const void *datum); extern bool list_member_int(const List *list, int datum); extern bool list_member_oid(const List *list, Oid datum); +extern bool list_is_subset(const List *members, const List *target); + extern List *list_delete(List *list, void *datum); extern List *list_delete_ptr(List *list, void *datum); extern List *list_delete_int(List *list, int datum); diff --git a/src/include/optimizer/appendinfo.h b/src/include/optimizer/appendinfo.h index d6a27a60dd..e87c92a054 100644 --- a/src/include/optimizer/appendinfo.h +++ b/src/include/optimizer/appendinfo.h @@ -32,4 +32,7 @@ extern Relids adjust_child_relids_multilevel(PlannerInfo *root, Relids relids, extern AppendRelInfo **find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos); +extern AppendRelInfo *find_appinfo_by_child(PlannerInfo *root, Index child_index); +extern Var *find_parent_var(AppendRelInfo *appinfo, Var *child_var); + #endif /* APPENDINFO_H */ diff --git a/src/include/optimizer/optimizer.h b/src/include/optimizer/optimizer.h index 3e4171056e..9445141263 100644 --- a/src/include/optimizer/optimizer.h +++ b/src/include/optimizer/optimizer.h @@ -23,6 +23,7 @@ #define OPTIMIZER_H #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" /* * We don't want to include nodes/pathnodes.h here, because non-planner @@ -156,6 +157,7 @@ extern TargetEntry *get_sortgroupref_tle(Index sortref, List *targetList); extern TargetEntry *get_sortgroupclause_tle(SortGroupClause *sgClause, List *targetList); +extern Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); extern Node *get_sortgroupclause_expr(SortGroupClause *sgClause, List *targetList); extern List *get_sortgrouplist_exprs(List *sgClauses, diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h index 10b6e81079..9217a8d6c6 100644 --- a/src/include/optimizer/paths.h +++ b/src/include/optimizer/paths.h @@ -240,5 +240,48 @@ extern PathKey *make_canonical_pathkey(PlannerInfo *root, int strategy, bool nulls_first); extern void add_paths_to_append_rel(PlannerInfo *root, RelOptInfo *rel, List *live_childrels); +extern List *select_mergejoin_clauses(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + JoinType jointype, + bool *mergejoin_allowed); + +/* + * uniquekeys.c + * Utilities for matching and building unique keys + */ +extern void populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List* unique_index_list); +extern void populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels); +extern void populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel); +extern void populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel); +extern void populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel); +extern void simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel); +extern void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel); +extern void populate_joinrel_uniquekeys(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + RelOptInfo *rel2, + List *restrictlist, + JoinType jointype); + +extern bool relation_has_uniquekeys_for(PlannerInfo *root, + RelOptInfo *rel, + List *exprs, + bool allow_multinulls); +extern bool relation_is_onerow(RelOptInfo *rel); #endif /* PATHS_H */ -- 2.21.0 --op7mf2i72xk5yu7c Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v36-0003-Extend-UniqueKeys.patch" ^ permalink raw reply [nested|flat] 91+ messages in thread
* [PATCH v38 2/6] Introduce UniqueKey attributes on RelOptInfo struct. @ 2020-05-11 07:50 一挃 <[email protected]> 0 siblings, 0 replies; 91+ messages in thread From: 一挃 @ 2020-05-11 07:50 UTC (permalink / raw) UniqueKey is a set of exprs on RelOptInfo which represents the exprs will be unique on the given RelOptInfo. You can see README.uniquekey for more information. --- src/backend/nodes/copyfuncs.c | 13 + src/backend/nodes/list.c | 31 + src/backend/nodes/makefuncs.c | 13 + src/backend/nodes/outfuncs.c | 11 + src/backend/nodes/readfuncs.c | 10 + src/backend/optimizer/path/Makefile | 3 +- src/backend/optimizer/path/README.uniquekey | 131 +++ src/backend/optimizer/path/allpaths.c | 10 + src/backend/optimizer/path/joinpath.c | 9 +- src/backend/optimizer/path/joinrels.c | 2 + src/backend/optimizer/path/pathkeys.c | 3 +- src/backend/optimizer/path/uniquekeys.c | 1135 +++++++++++++++++++ src/backend/optimizer/plan/planner.c | 13 +- src/backend/optimizer/prep/prepunion.c | 2 + src/backend/optimizer/util/appendinfo.c | 44 + src/backend/optimizer/util/inherit.c | 18 +- src/include/nodes/makefuncs.h | 3 + src/include/nodes/nodes.h | 1 + src/include/nodes/pathnodes.h | 29 +- src/include/nodes/pg_list.h | 10 + src/include/optimizer/appendinfo.h | 3 + src/include/optimizer/optimizer.h | 2 + src/include/optimizer/paths.h | 43 + 23 files changed, 1515 insertions(+), 24 deletions(-) create mode 100644 src/backend/optimizer/path/README.uniquekey create mode 100644 src/backend/optimizer/path/uniquekeys.c diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..75c1c5e824 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2296,6 +2296,16 @@ _copyPathKey(const PathKey *from) return newnode; } +static UniqueKey * +_copyUniqueKey(const UniqueKey *from) +{ + UniqueKey *newnode = makeNode(UniqueKey); + + COPY_NODE_FIELD(exprs); + COPY_SCALAR_FIELD(multi_nullvals); + + return newnode; +} /* * _copyRestrictInfo */ @@ -5220,6 +5230,9 @@ copyObjectImpl(const void *from) case T_PathKey: retval = _copyPathKey(from); break; + case T_UniqueKey: + retval = _copyUniqueKey(from); + break; case T_RestrictInfo: retval = _copyRestrictInfo(from); break; diff --git a/src/backend/nodes/list.c b/src/backend/nodes/list.c index dbf6b30233..ca099495a1 100644 --- a/src/backend/nodes/list.c +++ b/src/backend/nodes/list.c @@ -702,6 +702,37 @@ list_member_oid(const List *list, Oid datum) return false; } +/* + * return true iff every entry in "members" list is also present + * in the "target" list. + */ +bool +list_is_subset(const List *members, const List *target) +{ + const ListCell *lc1, *lc2; + + Assert(IsPointerList(members)); + Assert(IsPointerList(target)); + check_list_invariants(members); + check_list_invariants(target); + + foreach(lc1, members) + { + bool found = false; + foreach(lc2, target) + { + if (equal(lfirst(lc1), lfirst(lc2))) + { + found = true; + break; + } + } + if (!found) + return false; + } + return true; +} + /* * Delete the n'th cell (counting from 0) in list. * diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 01c110cd2f..40415d0f5b 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -815,3 +815,16 @@ makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols) v->va_cols = va_cols; return v; } + + +/* + * makeUniqueKey + */ +UniqueKey* +makeUniqueKey(List *exprs, bool multi_nullvals) +{ + UniqueKey * ukey = makeNode(UniqueKey); + ukey->exprs = exprs; + ukey->multi_nullvals = multi_nullvals; + return ukey; +} diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..44154cde6a 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2456,6 +2456,14 @@ _outPathKey(StringInfo str, const PathKey *node) WRITE_BOOL_FIELD(pk_nulls_first); } +static void +_outUniqueKey(StringInfo str, const UniqueKey *node) +{ + WRITE_NODE_TYPE("UNIQUEKEY"); + WRITE_NODE_FIELD(exprs); + WRITE_BOOL_FIELD(multi_nullvals); +} + static void _outPathTarget(StringInfo str, const PathTarget *node) { @@ -4198,6 +4206,9 @@ outNode(StringInfo str, const void *obj) case T_PathKey: _outPathKey(str, obj); break; + case T_UniqueKey: + _outUniqueKey(str, obj); + break; case T_PathTarget: _outPathTarget(str, obj); break; diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index c5e136e9c3..b3e212bf1c 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -492,6 +492,14 @@ _readSetOperationStmt(void) READ_DONE(); } +static UniqueKey * +_readUniqueKey(void) +{ + READ_LOCALS(UniqueKey); + READ_NODE_FIELD(exprs); + READ_BOOL_FIELD(multi_nullvals); + READ_DONE(); +} /* * Stuff from primnodes.h. @@ -2717,6 +2725,8 @@ parseNodeString(void) return_value = _readCommonTableExpr(); else if (MATCH("SETOPERATIONSTMT", 16)) return_value = _readSetOperationStmt(); + else if (MATCH("UNIQUEKEY", 9)) + return_value = _readUniqueKey(); else if (MATCH("ALIAS", 5)) return_value = _readAlias(); else if (MATCH("RANGEVAR", 8)) diff --git a/src/backend/optimizer/path/Makefile b/src/backend/optimizer/path/Makefile index 1e199ff66f..7b9820c25f 100644 --- a/src/backend/optimizer/path/Makefile +++ b/src/backend/optimizer/path/Makefile @@ -21,6 +21,7 @@ OBJS = \ joinpath.o \ joinrels.o \ pathkeys.o \ - tidpath.o + tidpath.o \ + uniquekeys.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/optimizer/path/README.uniquekey b/src/backend/optimizer/path/README.uniquekey new file mode 100644 index 0000000000..5eac761995 --- /dev/null +++ b/src/backend/optimizer/path/README.uniquekey @@ -0,0 +1,131 @@ +1. What is UniqueKey? +We can think UniqueKey is a set of exprs for a RelOptInfo, which we are insure +that doesn't yields same result among all the rows. The simplest UniqueKey +format is primary key. + +However we define the UnqiueKey as below. + +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + +exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL +is a special case of UniqueKey, which means there is only one row in that +relation.it has a stronger semantic than others. like SELECT uk FROM t; uk is +normal unique key and may have different values. SELECT colx FROM t WHERE uk = +const. colx is unique AND we have only 1 value. This field can used for +innerrel_is_unique. this logic is handled specially in add_uniquekey_for_onerow +function. + +multi_nullvals: true means multi null values may exist in these exprs, so the +uniqueness is not guaranteed in this case. This field is necessary for +remove_useless_join & reduce_unique_semijoins where we don't mind these +duplicated NULL values. It is set to true for 2 cases. One is a unique key +from a unique index but the related column is nullable. The other one is for +outer join. see populate_joinrel_uniquekeys for detail. + + +The UniqueKey can be used at the following cases at least: +1. remove_useless_joins. +2. reduce_semianti_joins +3. remove distinct node if distinct clause is unique. +4. remove aggnode if group by clause is unique. +5. Index Skip Scan (WIP) +6. Aggregation Push Down without 2 phase aggregation if the join can't + duplicated the aggregated rows. (work in progress feature) + +2. How is it maintained? + +We have a set of populate_xxx_unqiuekeys functions to maintain the uniquekey on +various cases. xxx includes baserel, joinrel, partitionedrel, distinctrel, +groupedrel, unionrel. and we also need to convert the uniquekey from subquery +to outer relation, which is what convert_subquery_uniquekeys does. + +1. The first part is about baserel. We handled 3 cases. suppose we have Unique +Index on (a, b). + +1. SELECT a, b FROM t. UniqueKey (a, b) +2. SELECT a FROM t WHERE b = 1; UniqueKey (a) +3. SELECT .. FROM t WHERE a = 1 AND b = 1; UniqueKey (NIL). onerow case, every + column is Unique. + +2. The next part is joinrel, this part is most error-prone, we simplified the rules +like below: +1. If the relation's UniqueKey can't be duplicated after join, then is will be + still valid for the join rel. The function we used here is + innerrel_keeps_unique. The basic idea is innerrel.any_col = outer.uk. + +2. If the UnqiueKey can't keep valid via the rule 1, the combination of the + UniqueKey from both sides are valid for sure. We can prove this as: if the + unique exprs from rel1 is duplicated by rel2, the duplicated rows must + contains different unique exprs from rel2. + +More considerations about onerow: +1. If relation with one row and it can't be duplicated, it is still possible + contains mulit_nullvas after outer join. +2. If the either UniqueKey can be duplicated after join, the can get one row + only when both side is one row AND there is no outer join. +3. Whenever the onerow UniqueKey is not a valid any more, we need to convert one + row UniqueKey to normal unique key since we don't store exprs for one-row + relation. get_exprs_from_uniquekeys will be used here. + + +More considerations about multi_nullvals after join: +1. If the original UnqiueKey has multi_nullvals, the final UniqueKey will have + mulit_nullvals in any case. +2. If a unique key doesn't allow mulit_nullvals, after some outer join, it + allows some outer join. + + +3. When we comes to subquery, we need to convert_subquery_unqiuekeys just like +convert_subquery_pathkeys. Only the UniqueKey insides subquery is referenced as +a Var in outer relation will be reused. The relationship between the outerrel.Var +and subquery.exprs is built with outerel->subroot->processed_tlist. + + +4. As for the SRF functions, it will break the uniqueness of uniquekey, However it +is handled in adjust_paths_for_srfs, which happens after the query_planner. so +we will maintain the UniqueKey until there and reset it to NIL at that +places. This can't help on distinct/group by elimination cases but probably help +in some other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is +semantic correctly. + + +5. As for inherit table, we first main the UnqiueKey on childrel as well. But for +partitioned table we need to maintain 2 different kinds of +UnqiueKey. 1). UniqueKey on the parent relation 2). UniqueKey on child +relation for partition wise query. + +Example: +CREATE TABLE p (a int not null, b int not null) partition by list (a); +CREATE TABLE p0 partition of p for values in (1); +CREATE TABLE p1 partition of p for values in (2); + +create unique index p0_b on p0(b); +create unique index p1_b on p1(b); + +Now b is only unique on partition level, so the distinct can't be removed on +the following cases. SELECT DISTINCT b FROM p; + +Another example is SELECT DISTINCT a, b FROM p WHERE a = 1; Since only one +partition is chosen, the UniqueKey on child relation is same as the UniqueKey on +parent relation. + +Another usage of UniqueKey on partition level is it be helpful for +partition-wise join. + +As for the UniqueKey on parent table level, it comes with 2 different ways, +1). the UniqueKey is also derived in UniqueKey index, but the index must be same +in all the related children relations and the unique index must contains +Partition Key in it. Example: + +CREATE UNIQUE INDEX p_ab ON p(a, b); -- where a is the partition key. + +-- Query +SELECT a, b FROM p; the (a, b) is a UniqueKey of p. + +2). If the parent relation has only one childrel, the UniqueKey on childrel is + the UniqueKey on parent as well. diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index 37b4223adb..d52ad59f89 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -579,6 +579,12 @@ set_plain_rel_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) */ check_index_predicates(root, rel); + /* + * Now that we've marked which partial indexes are suitable, we can now + * build the relation's unique keys. + */ + populate_baserel_uniquekeys(root, rel, rel->indexlist); + /* Mark rel with estimated output rows, width, etc */ set_baserel_size_estimates(root, rel); } @@ -1297,6 +1303,8 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, /* Add paths to the append relation. */ add_paths_to_append_rel(root, rel, live_childrels); + if (IS_PARTITIONED_REL(rel)) + populate_partitionedrel_uniquekeys(root, rel, live_childrels); } @@ -2304,6 +2312,8 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, pathkeys, required_outer)); } + convert_subquery_uniquekeys(root, rel, sub_final_rel); + /* If outer rel allows parallelism, do same for partial paths. */ if (rel->consider_parallel && bms_is_empty(required_outer)) { diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c index 57ce97fd53..697ac047fb 100644 --- a/src/backend/optimizer/path/joinpath.c +++ b/src/backend/optimizer/path/joinpath.c @@ -71,13 +71,6 @@ static void consider_parallel_mergejoin(PlannerInfo *root, static void hash_inner_and_outer(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, RelOptInfo *innerrel, JoinType jointype, JoinPathExtraData *extra); -static List *select_mergejoin_clauses(PlannerInfo *root, - RelOptInfo *joinrel, - RelOptInfo *outerrel, - RelOptInfo *innerrel, - List *restrictlist, - JoinType jointype, - bool *mergejoin_allowed); static void generate_mergejoin_paths(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *innerrel, @@ -1927,7 +1920,7 @@ hash_inner_and_outer(PlannerInfo *root, * if it is mergejoinable and involves vars from the two sub-relations * currently of interest. */ -static List * +List * select_mergejoin_clauses(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c index 0dbe2ac726..7271f044ec 100644 --- a/src/backend/optimizer/path/joinrels.c +++ b/src/backend/optimizer/path/joinrels.c @@ -924,6 +924,8 @@ populate_joinrel_with_paths(PlannerInfo *root, RelOptInfo *rel1, /* Apply partitionwise join technique, if possible. */ try_partitionwise_join(root, rel1, rel2, joinrel, sjinfo, restrictlist); + + populate_joinrel_uniquekeys(root, joinrel, rel1, rel2, restrictlist, sjinfo->jointype); } diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index bd9a176d7d..139278829b 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -33,7 +33,6 @@ static bool pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys); static bool matches_boolean_partition_clause(RestrictInfo *rinfo, RelOptInfo *partrel, int partkeycol); -static Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); static bool right_merge_direction(PlannerInfo *root, PathKey *pathkey); @@ -1035,7 +1034,7 @@ convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel, * We need this to ensure that we don't return pathkeys describing values * that are unavailable above the level of the subquery scan. */ -static Var * +Var * find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle) { ListCell *lc; diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c new file mode 100644 index 0000000000..77ed2b2eff --- /dev/null +++ b/src/backend/optimizer/path/uniquekeys.c @@ -0,0 +1,1135 @@ +/*------------------------------------------------------------------------- + * + * uniquekeys.c + * Utilities for matching and building unique keys + * + * Portions Copyright (c) 2020, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/optimizer/path/uniquekeys.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "nodes/makefuncs.h" +#include "nodes/nodeFuncs.h" +#include "optimizer/pathnode.h" +#include "optimizer/paths.h" +#include "optimizer/appendinfo.h" +#include "optimizer/optimizer.h" +#include "optimizer/tlist.h" +#include "rewrite/rewriteManip.h" + + +/* + * This struct is used to help populate_joinrel_uniquekeys. + * + * added_to_joinrel is true if a uniquekey (from outerrel or innerrel) + * has been added to joinrel. + * useful is true if the exprs of the uniquekey still appears in joinrel. + */ +typedef struct UniqueKeyContextData +{ + UniqueKey *uniquekey; + bool added_to_joinrel; + bool useful; +} *UniqueKeyContext; + +static List *initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel); +static bool innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + bool reverse); + +static List *get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals); +static List *get_exprs_from_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + UniqueKey *ukey); +static void add_uniquekey_for_onerow(RelOptInfo *rel); +static bool add_combined_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype); + +/* Used for unique indexes checking for partitioned table */ +static bool index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind); +static IndexOptInfo *simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from); +static bool simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2); +static void adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_index); + +/* Helper function for grouped relation and distinct relation. */ +static void add_uniquekey_from_sortgroups(PlannerInfo *root, + RelOptInfo *rel, + List *sortgroups); + +/* + * populate_baserel_uniquekeys + * Populate 'baserel' uniquekeys list by looking at the rel's unique index + * and baserestrictinfo + */ +void +populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List *indexlist) +{ + ListCell *lc; + List *matched_uniq_indexes = NIL; + + /* Attrs appears in rel->reltarget->exprs. */ + Bitmapset *used_attrs = NULL; + + List *const_exprs = NIL; + List *expr_opfamilies = NIL; + + Assert(baserel->rtekind == RTE_RELATION); + + foreach(lc, indexlist) + { + IndexOptInfo *ind = (IndexOptInfo *) lfirst(lc); + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + matched_uniq_indexes = lappend(matched_uniq_indexes, ind); + } + + if (matched_uniq_indexes == NIL) + return; + + /* Check which attrs is used in baserel->reltarget */ + pull_varattnos((Node *)baserel->reltarget->exprs, baserel->relid, &used_attrs); + + /* Check which attrno is used at a mergeable const filter */ + foreach(lc, baserel->baserestrictinfo) + { + RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); + + if (rinfo->mergeopfamilies == NIL) + continue; + + if (bms_is_empty(rinfo->left_relids)) + { + const_exprs = lappend(const_exprs, get_rightop(rinfo->clause)); + } + else if (bms_is_empty(rinfo->right_relids)) + { + const_exprs = lappend(const_exprs, get_leftop(rinfo->clause)); + } + else + continue; + + expr_opfamilies = lappend(expr_opfamilies, rinfo->mergeopfamilies); + } + + foreach(lc, matched_uniq_indexes) + { + bool multi_nullvals, useful; + List *exprs = get_exprs_from_uniqueindex(lfirst_node(IndexOptInfo, lc), + const_exprs, + expr_opfamilies, + used_attrs, + &useful, + &multi_nullvals); + if (useful) + { + if (exprs == NIL) + { + /* All the columns in Unique Index matched with a restrictinfo */ + add_uniquekey_for_onerow(baserel); + return; + } + baserel->uniquekeys = lappend(baserel->uniquekeys, + makeUniqueKey(exprs, multi_nullvals)); + } + } +} + + +/* + * populate_partitionedrel_uniquekeys + * The UniqueKey on partitionrel comes from 2 cases: + * 1). Only one partition is involved in this query, the unique key can be + * copied to parent rel from childrel. + * 2). There are some unique index which includes partition key and exists + * in all the related partitions. + * We never mind rule 2 if we hit rule 1. + */ + +void +populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels) +{ + ListCell *lc; + List *global_uniq_indexlist = NIL; + RelOptInfo *childrel; + bool is_first = true; + + Assert(IS_PARTITIONED_REL(rel)); + + if (childrels == NIL) + return; + + /* + * If there is only one partition used in this query, the UniqueKey in childrel is + * still valid in parent level, but we need convert the format from child expr to + * parent expr. + */ + if (list_length(childrels) == 1) + { + /* Check for Rule 1 */ + RelOptInfo *childrel = linitial_node(RelOptInfo, childrels); + ListCell *lc; + Assert(childrel->reloptkind == RELOPT_OTHER_MEMBER_REL); + if (relation_is_onerow(childrel)) + { + add_uniquekey_for_onerow(rel); + return; + } + + foreach(lc, childrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + AppendRelInfo *appinfo = find_appinfo_by_child(root, childrel->relid); + List *parent_exprs = NIL; + bool can_reuse = true; + ListCell *lc2; + foreach(lc2, ukey->exprs) + { + Var *var = (Var *)lfirst(lc2); + /* + * If the expr comes from a expression, it is hard to build the expression + * in parent so ignore that case for now. + */ + if(!IsA(var, Var)) + { + can_reuse = false; + break; + } + /* Convert it to parent var */ + parent_exprs = lappend(parent_exprs, find_parent_var(appinfo, var)); + } + if (can_reuse) + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(parent_exprs, + ukey->multi_nullvals)); + } + } + else + { + /* Check for rule 2 */ + childrel = linitial_node(RelOptInfo, childrels); + foreach(lc, childrel->indexlist) + { + IndexOptInfo *ind = lfirst(lc); + IndexOptInfo *modified_index; + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + + /* + * During simple_copy_indexinfo_to_parent, we need to convert var from + * child var to parent var, index on expression is too complex to handle. + * so ignore it for now. + */ + if (ind->indexprs != NIL) + continue; + + modified_index = simple_copy_indexinfo_to_parent(root, rel, ind); + /* + * If the unique index doesn't contain partkey, then it is unique + * on this partition only, so it is useless for us. + */ + if (!index_constains_partkey(rel, modified_index)) + continue; + + global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index); + } + + if (global_uniq_indexlist != NIL) + { + foreach(lc, childrels) + { + RelOptInfo *child = lfirst(lc); + if (is_first) + { + is_first = false; + continue; + } + adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist); + } + /* Now we have a list of unique index which are exactly same on all childrels, + * Set the UniqueKey just like it is non-partition table + */ + populate_baserel_uniquekeys(root, rel, global_uniq_indexlist); + } + } +} + + +/* + * populate_distinctrel_uniquekeys + */ +void +populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel) +{ + /* The unique key before the distinct is still valid. */ + distinctrel->uniquekeys = list_copy(inputrel->uniquekeys); + add_uniquekey_from_sortgroups(root, distinctrel, root->parse->distinctClause); +} + +/* + * populate_grouprel_uniquekeys + */ +void +populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel) + +{ + Query *parse = root->parse; + bool input_ukey_added = false; + ListCell *lc; + + if (relation_is_onerow(inputrel)) + { + add_uniquekey_for_onerow(grouprel); + return; + } + if (parse->groupingSets) + return; + + /* A Normal group by without grouping set. */ + if (parse->groupClause) + { + /* + * Current even the groupby clause is Unique already, but if query has aggref + * We have to create grouprel still. To keep the UnqiueKey short, we will check + * the UniqueKey of input_rel still valid, if so we reuse it. + */ + foreach(lc, inputrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (list_is_subset(ukey->exprs, grouprel->reltarget->exprs)) + { + grouprel->uniquekeys = lappend(grouprel->uniquekeys, + ukey); + input_ukey_added = true; + } + } + if (!input_ukey_added) + /* + * group by clause must be a super-set of grouprel->reltarget->exprs except the + * aggregation expr, so if such exprs is unique already, no bother to generate + * new uniquekey for group by exprs. + */ + add_uniquekey_from_sortgroups(root, + grouprel, + root->parse->groupClause); + } + else + /* It has aggregation but without a group by, so only one row returned */ + add_uniquekey_for_onerow(grouprel); +} + +/* + * simple_copy_uniquekeys + * Using a function for the one-line code makes us easy to check where we simply + * copied the uniquekey. + */ +void +simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel) +{ + newrel->uniquekeys = oldrel->uniquekeys; +} + +/* + * populate_unionrel_uniquekeys + */ +void +populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel) +{ + ListCell *lc; + List *exprs = NIL; + + Assert(unionrel->uniquekeys == NIL); + + foreach(lc, unionrel->reltarget->exprs) + { + exprs = lappend(exprs, lfirst(lc)); + } + + if (exprs == NIL) + /* SQL: select union select; is valid, we need to handle it here. */ + add_uniquekey_for_onerow(unionrel); + else + unionrel->uniquekeys = lappend(unionrel->uniquekeys, + makeUniqueKey(exprs,false)); + +} + +/* + * populate_joinrel_uniquekeys + * + * populate uniquekeys for joinrel. We will check each relation to see if its + * UniqueKey is still valid via innerrel_keeps_unique, if so, we add it to + * joinrel. The multi_nullvals field will be changed to true for some outer + * join cases and one-row UniqueKey needs to be converted to normal UniqueKey + * for the same case as well. + * For the uniquekey in either baserel which can't be unique after join, we still + * check to see if combination of UniqueKeys from both side is still useful for us. + * if yes, we add it to joinrel as well. + */ +void +populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *outerrel, RelOptInfo *innerrel, + List *restrictlist, JoinType jointype) +{ + ListCell *lc, *lc2; + List *clause_list = NIL; + List *outerrel_ukey_ctx; + List *innerrel_ukey_ctx; + bool inner_onerow, outer_onerow; + bool mergejoin_allowed; + + /* Care about the outerrel relation only for SEMI/ANTI join */ + if (jointype == JOIN_SEMI || jointype == JOIN_ANTI) + { + foreach(lc, outerrel->uniquekeys) + { + UniqueKey *uniquekey = lfirst_node(UniqueKey, lc); + if (list_is_subset(uniquekey->exprs, joinrel->reltarget->exprs)) + joinrel->uniquekeys = lappend(joinrel->uniquekeys, uniquekey); + } + return; + } + + Assert(jointype == JOIN_LEFT || jointype == JOIN_FULL || jointype == JOIN_INNER); + + /* Fast path */ + if (innerrel->uniquekeys == NIL || outerrel->uniquekeys == NIL) + return; + + inner_onerow = relation_is_onerow(innerrel); + outer_onerow = relation_is_onerow(outerrel); + + outerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(outerrel); + innerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(innerrel); + + clause_list = select_mergejoin_clauses(root, joinrel, outerrel, innerrel, + restrictlist, jointype, + &mergejoin_allowed); + + if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true /* reverse */)) + { + bool outer_impact = jointype == JOIN_FULL; + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + /* Outer relation has one row, and the unique key is not duplicated after join, + * the joinrel will still has one row unless the jointype == JOIN_FULL. + */ + if (outer_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (outer_onerow) + { + /* + * The onerow outerrel becomes multi rows and multi_nullvals + * will be changed to true. We also need to set the exprs correctly since it + * can't be NIL any more. + */ + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, outerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Change multi_nullvals to true due to the full join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, true)); + else + /* Just reuse it */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + } + ctx->added_to_joinrel = true; + } + } + + if (innerrel_keeps_unique(root, outerrel, innerrel, clause_list, false)) + { + bool outer_impact = jointype == JOIN_FULL || jointype == JOIN_LEFT;; + + foreach(lc, innerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + if (inner_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (inner_onerow) + { + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, innerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Need to change multi_nullvals to true due to the outer join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, + true)); + else + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + + } + ctx->added_to_joinrel = true; + } + } + + /* + * The combination of the UniqueKey from both sides is unique as well regardless + * of join type, but no bother to add it if its subset has been added to joinrel + * already or it is not useful for the joinrel. + */ + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx1 = (UniqueKeyContext) lfirst(lc); + if (ctx1->added_to_joinrel || !ctx1->useful) + continue; + foreach(lc2, innerrel_ukey_ctx) + { + UniqueKeyContext ctx2 = (UniqueKeyContext) lfirst(lc2); + if (ctx2->added_to_joinrel || !ctx2->useful) + continue; + if (add_combined_uniquekey(root, joinrel, outerrel, innerrel, + ctx1->uniquekey, ctx2->uniquekey, + jointype)) + /* If we set a onerow UniqueKey to joinrel, we don't need other. */ + return; + } + } +} + + +/* + * convert_subquery_uniquekeys + * + * Covert the UniqueKey in subquery to outer relation. + */ +void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel) +{ + ListCell *lc; + + if (sub_final_rel->uniquekeys == NIL) + return; + + if (relation_is_onerow(sub_final_rel)) + { + add_uniquekey_for_onerow(currel); + return; + } + + Assert(currel->subroot != NULL); + + foreach(lc, sub_final_rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + ListCell *lc; + List *exprs = NIL; + bool ukey_useful = true; + + /* One row case is handled above */ + Assert(ukey->exprs != NIL); + foreach(lc, ukey->exprs) + { + Var *var; + TargetEntry *tle = tlist_member(lfirst(lc), + currel->subroot->processed_tlist); + if (tle == NULL) + { + ukey_useful = false; + break; + } + var = find_var_for_subquery_tle(currel, tle); + if (var == NULL) + { + ukey_useful = false; + break; + } + exprs = lappend(exprs, var); + } + + if (ukey_useful) + currel->uniquekeys = lappend(currel->uniquekeys, + makeUniqueKey(exprs, + ukey->multi_nullvals)); + + } +} + +/* + * innerrel_keeps_unique + * + * Check if Unique key of the innerrel is valid after join. innerrel's UniqueKey + * will be still valid if innerrel's any-column mergeop outrerel's uniquekey + * exists in clause_list. + * + * Note: the clause_list must be a list of mergeable restrictinfo already. + */ +static bool +innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *clause_list, + bool reverse) +{ + ListCell *lc, *lc2, *lc3; + + if (outerrel->uniquekeys == NIL || innerrel->uniquekeys == NIL) + return false; + + /* Check if there is outerrel's uniquekey in mergeable clause. */ + foreach(lc, outerrel->uniquekeys) + { + List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs; + bool clauselist_matchs_all_exprs = true; + foreach(lc2, outer_uq_exprs) + { + Node *outer_uq_expr = lfirst(lc2); + bool find_uq_expr_in_clauselist = false; + foreach(lc3, clause_list) + { + RestrictInfo *rinfo = lfirst_node(RestrictInfo, lc3); + Node *outer_expr; + if (reverse) + outer_expr = rinfo->outer_is_left ? get_rightop(rinfo->clause) : get_leftop(rinfo->clause); + else + outer_expr = rinfo->outer_is_left ? get_leftop(rinfo->clause) : get_rightop(rinfo->clause); + if (equal(outer_expr, outer_uq_expr)) + { + find_uq_expr_in_clauselist = true; + break; + } + } + if (!find_uq_expr_in_clauselist) + { + /* No need to check the next exprs in the current uniquekey */ + clauselist_matchs_all_exprs = false; + break; + } + } + + if (clauselist_matchs_all_exprs) + return true; + } + return false; +} + + +/* + * relation_is_onerow + * Check if it is a one-row relation by checking UniqueKey. + */ +bool +relation_is_onerow(RelOptInfo *rel) +{ + UniqueKey *ukey; + if (rel->uniquekeys == NIL) + return false; + ukey = linitial_node(UniqueKey, rel->uniquekeys); + return ukey->exprs == NIL && list_length(rel->uniquekeys) == 1; +} + +/* + * relation_has_uniquekeys_for + * Returns true if we have proofs that 'rel' cannot return multiple rows with + * the same values in each of 'exprs'. Otherwise returns false. + */ +bool +relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel, + List *exprs, bool allow_multinulls) +{ + ListCell *lc; + + /* + * For UniqueKey->onerow case, the uniquekey->exprs is empty as well + * so we can't rely on list_is_subset to handle this special cases + */ + if (exprs == NIL) + return false; + + foreach(lc, rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (ukey->multi_nullvals && !allow_multinulls) + continue; + if (list_is_subset(ukey->exprs, exprs)) + return true; + } + return false; +} + + +/* + * get_exprs_from_uniqueindex + * + * Return a list of exprs which is unique. set useful to false if this + * unique index is not useful for us. + */ +static List * +get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals) +{ + List *exprs = NIL; + ListCell *indexpr_item; + int c = 0; + + *useful = true; + *multi_nullvals = false; + + indexpr_item = list_head(unique_index->indexprs); + for(c = 0; c < unique_index->ncolumns; c++) + { + int attr = unique_index->indexkeys[c]; + Expr *expr; + bool matched_const = false; + ListCell *lc1, *lc2; + + if(attr > 0) + { + expr = list_nth_node(TargetEntry, unique_index->indextlist, c)->expr; + } + else if (attr == 0) + { + /* Expression index */ + expr = lfirst(indexpr_item); + indexpr_item = lnext(unique_index->indexprs, indexpr_item); + } + else /* attr < 0 */ + { + /* Index on system column is not supported */ + Assert(false); + } + + /* + * Check index_col = Const case with regarding to opfamily checking + * If we can remove the index_col from the final UniqueKey->exprs. + */ + forboth(lc1, const_exprs, lc2, const_expr_opfamilies) + { + if (list_member_oid((List *)lfirst(lc2), unique_index->opfamily[c]) + && match_index_to_operand((Node *) lfirst(lc1), c, unique_index)) + { + matched_const = true; + break; + } + } + + if (matched_const) + continue; + + /* Check if the indexed expr is used in rel */ + if (attr > 0) + { + /* + * Normal Indexed column, if the col is not used, then the index is useless + * for uniquekey. + */ + attr -= FirstLowInvalidHeapAttributeNumber; + + if (!bms_is_member(attr, used_varattrs)) + { + *useful = false; + break; + } + } + else if (!list_member(unique_index->rel->reltarget->exprs, expr)) + { + /* Expression index but the expression is not used in rel */ + *useful = false; + break; + } + + /* check not null property. */ + if (attr == 0) + { + /* We never know if a expression yields null or not */ + *multi_nullvals = true; + } + else if (!bms_is_member(attr, unique_index->rel->notnullattrs) + && !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber, + unique_index->rel->notnullattrs)) + { + *multi_nullvals = true; + } + + exprs = lappend(exprs, expr); + } + return exprs; +} + + +/* + * add_uniquekey_for_onerow + * If we are sure that the relation only returns one row, then all the columns + * are unique. However we don't need to create UniqueKey for every column, we + * just set exprs = NIL and overwrites all the other UniqueKey on this RelOptInfo + * since this one has strongest semantics. + */ +void +add_uniquekey_for_onerow(RelOptInfo *rel) +{ + /* + * We overwrite the previous UniqueKey on purpose since this one has the + * strongest semantic. + */ + rel->uniquekeys = list_make1(makeUniqueKey(NIL, false)); +} + + +/* + * initililze_uniquecontext_for_joinrel + * Return a List of UniqueKeyContext for an inputrel + */ +static List * +initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel) +{ + List *res = NIL; + ListCell *lc; + foreach(lc, inputrel->uniquekeys) + { + UniqueKeyContext context; + context = palloc(sizeof(struct UniqueKeyContextData)); + context->uniquekey = lfirst_node(UniqueKey, lc); + context->added_to_joinrel = false; + context->useful = true; + res = lappend(res, context); + } + return res; +} + + +/* + * get_exprs_from_uniquekey + * Unify the way of get List of exprs from a one-row UniqueKey or + * normal UniqueKey. for the onerow case, every expr in rel1 is a valid + * UniqueKey. Return a List of exprs. + * + * rel1: The relation which you want to get the exprs. + * ukey: The UniqueKey you want to get the exprs. + */ +static List * +get_exprs_from_uniquekey(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *rel1, UniqueKey *ukey) +{ + ListCell *lc; + bool onerow = rel1 != NULL && relation_is_onerow(rel1); + + List *res = NIL; + Assert(onerow || ukey); + if (onerow) + { + /* Only cares about the exprs still exist in joinrel */ + foreach(lc, joinrel->reltarget->exprs) + { + Bitmapset *relids = pull_varnos(root, lfirst(lc)); + if (bms_is_subset(relids, rel1->relids)) + { + res = lappend(res, list_make1(lfirst(lc))); + } + } + } + else + { + res = list_make1(ukey->exprs); + } + return res; +} + +/* + * Partitioned table Unique Keys. + * The partition table unique key is maintained as: + * 1. The index must be unique as usual. + * 2. The index must contains partition key. + * 3. The index must exist on all the child rel. see simple_indexinfo_equal for + * how we compare it. + */ + +/* + * index_constains_partkey + * return true if the index contains the partiton key. + */ +static bool +index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind) +{ + ListCell *lc; + int i; + Assert(IS_PARTITIONED_REL(partrel)); + Assert(partrel->part_scheme->partnatts > 0); + + for(i = 0; i < partrel->part_scheme->partnatts; i++) + { + Node *part_expr = linitial(partrel->partexprs[i]); + bool found_in_index = false; + foreach(lc, ind->indextlist) + { + Expr *index_expr = lfirst_node(TargetEntry, lc)->expr; + if (equal(index_expr, part_expr)) + { + found_in_index = true; + break; + } + } + if (!found_in_index) + return false; + } + return true; +} + +/* + * simple_indexinfo_equal + * + * Used to check if the 2 index is same as each other. The index here + * is COPIED from childrel and did some tiny changes(see + * simple_copy_indexinfo_to_parent) + */ +static bool +simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2) +{ + Size oid_cmp_len = sizeof(Oid) * ind1->ncolumns; + + return ind1->ncolumns == ind2->ncolumns && + ind1->unique == ind2->unique && + memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0 && + memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0 && + memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0 && + memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0 && + equal(get_tlist_exprs(ind1->indextlist, true), + get_tlist_exprs(ind2->indextlist, true)); +} + + +/* + * The below macros are used for simple_copy_indexinfo_to_parent which is so + * customized that I don't want to put it to copyfuncs.c. So copy it here. + */ +#define COPY_POINTER_FIELD(fldname, sz) \ + do { \ + Size _size = (sz); \ + newnode->fldname = palloc(_size); \ + memcpy(newnode->fldname, from->fldname, _size); \ + } while (0) + +#define COPY_NODE_FIELD(fldname) \ + (newnode->fldname = copyObjectImpl(from->fldname)) + +#define COPY_SCALAR_FIELD(fldname) \ + (newnode->fldname = from->fldname) + + +/* + * simple_copy_indexinfo_to_parent (from partition) + * Copy the IndexInfo from child relation to parent relation with some modification, + * which is used to test: + * 1. If the same index exists in all the childrels. + * 2. If the parentrel->reltarget/basicrestrict info matches this index. + */ +static IndexOptInfo * +simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from) +{ + IndexOptInfo *newnode = makeNode(IndexOptInfo); + AppendRelInfo *appinfo = find_appinfo_by_child(root, from->rel->relid); + ListCell *lc; + int idx = 0; + + COPY_SCALAR_FIELD(ncolumns); + COPY_SCALAR_FIELD(nkeycolumns); + COPY_SCALAR_FIELD(unique); + COPY_SCALAR_FIELD(immediate); + /* We just need to know if it is NIL or not */ + COPY_SCALAR_FIELD(indpred); + COPY_SCALAR_FIELD(predOK); + COPY_POINTER_FIELD(indexkeys, from->ncolumns * sizeof(int)); + COPY_POINTER_FIELD(indexcollations, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opfamily, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opcintype, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(sortopfamily, from->ncolumns * sizeof(Oid)); + COPY_NODE_FIELD(indextlist); + + /* Convert index exprs on child expr to expr on parent */ + foreach(lc, newnode->indextlist) + { + TargetEntry *tle = lfirst_node(TargetEntry, lc); + /* Index on expression is ignored */ + Assert(IsA(tle->expr, Var)); + tle->expr = (Expr *) find_parent_var(appinfo, (Var *) tle->expr); + newnode->indexkeys[idx] = castNode(Var, tle->expr)->varattno; + idx++; + } + newnode->rel = parentrel; + return newnode; +} + +/* + * adjust_partition_unique_indexlist + * + * global_unique_indexes: At the beginning, it contains the copy & modified + * unique index from the first partition. And then check if each index in it still + * exists in the following partitions. If no, remove it. at last, it has an + * index list which exists in all the partitions. + */ +static void +adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_indexes) +{ + ListCell *lc, *lc2; + foreach(lc, *global_unique_indexes) + { + IndexOptInfo *g_ind = lfirst_node(IndexOptInfo, lc); + bool found_in_child = false; + + foreach(lc2, childrel->indexlist) + { + IndexOptInfo *p_ind = lfirst_node(IndexOptInfo, lc2); + IndexOptInfo *p_ind_copy; + if (!p_ind->unique || !p_ind->immediate || + (p_ind->indpred != NIL && !p_ind->predOK)) + continue; + p_ind_copy = simple_copy_indexinfo_to_parent(root, parentrel, p_ind); + if (simple_indexinfo_equal(p_ind_copy, g_ind)) + { + found_in_child = true; + break; + } + } + if (!found_in_child) + /* The index doesn't exist in childrel, remove it from global_unique_indexes */ + *global_unique_indexes = foreach_delete_current(*global_unique_indexes, lc); + } +} + +/* Helper function for groupres/distinctrel */ +static void +add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgroups) +{ + Query *parse = root->parse; + List *exprs; + + /* + * XXX: If there are some vars which is not in current levelsup, the semantic is + * imprecise, should we avoid it or not? levelsup = 1 is just a demo, maybe we need to + * check every level other than 0, if so, looks we have to write another + * pull_var_walker. + */ + List *upper_vars = pull_vars_of_level((Node*)sortgroups, 1); + + if (upper_vars != NIL) + return; + + exprs = get_sortgrouplist_exprs(sortgroups, parse->targetList); + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(exprs, + false /* sortgroupclause can't be multi_nullvals */)); +} + + +/* + * add_combined_uniquekey + * The combination of both UniqueKeys is a valid UniqueKey for joinrel no matter + * the jointype. + */ +bool +add_combined_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype) +{ + + ListCell *lc1, *lc2; + + /* Either side has multi_nullvals or we have outer join, + * the combined UniqueKey has multi_nullvals */ + bool multi_nullvals = outer_ukey->multi_nullvals || + inner_ukey->multi_nullvals || IS_OUTER_JOIN(jointype); + + /* The only case we can get onerow joinrel after join */ + if (relation_is_onerow(outer_rel) + && relation_is_onerow(inner_rel) + && jointype == JOIN_INNER) + { + add_uniquekey_for_onerow(joinrel); + return true; + } + + foreach(lc1, get_exprs_from_uniquekey(root, joinrel, outer_rel, outer_ukey)) + { + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, inner_rel, inner_ukey)) + { + List *exprs = list_concat_copy(lfirst_node(List, lc1), lfirst_node(List, lc2)); + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(exprs, + multi_nullvals)); + } + } + return false; +} diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index 424d25cbd5..8d8e493f5c 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -2387,6 +2387,8 @@ grouping_planner(PlannerInfo *root, bool inheritance_update, add_path(final_rel, path); } + simple_copy_uniquekeys(current_rel, final_rel); + /* * Generate partial paths for final_rel, too, if outer query levels might * be able to make use of them. @@ -3899,6 +3901,8 @@ create_grouping_paths(PlannerInfo *root, } set_cheapest(grouped_rel); + + populate_grouprel_uniquekeys(root, grouped_rel, input_rel); return grouped_rel; } @@ -4621,7 +4625,7 @@ create_window_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(window_rel); - + simple_copy_uniquekeys(input_rel, window_rel); return window_rel; } @@ -4935,7 +4939,7 @@ create_distinct_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(distinct_rel); - + populate_distinctrel_uniquekeys(root, input_rel, distinct_rel); return distinct_rel; } @@ -5196,6 +5200,8 @@ create_ordered_paths(PlannerInfo *root, */ Assert(ordered_rel->pathlist != NIL); + simple_copy_uniquekeys(input_rel, ordered_rel); + return ordered_rel; } @@ -6073,6 +6079,9 @@ adjust_paths_for_srfs(PlannerInfo *root, RelOptInfo *rel, if (list_length(targets) == 1) return; + /* UniqueKey is not valid after handling the SRF. */ + rel->uniquekeys = NIL; + /* * Stack SRF-evaluation nodes atop each path for the rel. * diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c index becdcbb872..b7626545bf 100644 --- a/src/backend/optimizer/prep/prepunion.c +++ b/src/backend/optimizer/prep/prepunion.c @@ -689,6 +689,8 @@ generate_union_paths(SetOperationStmt *op, PlannerInfo *root, /* Undo effects of possibly forcing tuple_fraction to 0 */ root->tuple_fraction = save_fraction; + /* Add the UniqueKeys */ + populate_unionrel_uniquekeys(root, result_rel); return result_rel; } diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c index 86922a273c..6817c9c787 100644 --- a/src/backend/optimizer/util/appendinfo.c +++ b/src/backend/optimizer/util/appendinfo.c @@ -746,3 +746,47 @@ find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos) } return appinfos; } + +/* + * find_appinfo_by_child + * + */ +AppendRelInfo * +find_appinfo_by_child(PlannerInfo *root, Index child_index) +{ + ListCell *lc; + foreach(lc, root->append_rel_list) + { + AppendRelInfo *appinfo = lfirst_node(AppendRelInfo, lc); + if (appinfo->child_relid == child_index) + return appinfo; + } + elog(ERROR, "parent relation cant be found"); + return NULL; +} + +/* + * find_parent_var + * + */ +Var * +find_parent_var(AppendRelInfo *appinfo, Var *child_var) +{ + ListCell *lc; + Var *res = NULL; + Index attno = 1; + foreach(lc, appinfo->translated_vars) + { + Node *child_node = lfirst(lc); + if (equal(child_node, child_var)) + { + res = copyObject(child_var); + res->varattno = attno; + res->varno = appinfo->parent_relid; + } + attno++; + } + if (res == NULL) + elog(ERROR, "parent var can't be found."); + return res; +} diff --git a/src/backend/optimizer/util/inherit.c b/src/backend/optimizer/util/inherit.c index be1c9ddd96..3eec1f4d74 100644 --- a/src/backend/optimizer/util/inherit.c +++ b/src/backend/optimizer/util/inherit.c @@ -736,6 +736,7 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, { Node *onecq = (Node *) lfirst(lc2); bool pseudoconstant; + RestrictInfo *child_rinfo; /* check for pseudoconstant (no Vars or volatile functions) */ pseudoconstant = @@ -747,14 +748,15 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, root->hasPseudoConstantQuals = true; } /* reconstitute RestrictInfo with appropriate properties */ - childquals = lappend(childquals, - make_restrictinfo(root, - (Expr *) onecq, - rinfo->is_pushed_down, - rinfo->outerjoin_delayed, - pseudoconstant, - rinfo->security_level, - NULL, NULL, NULL)); + child_rinfo = make_restrictinfo(root, + (Expr *) onecq, + rinfo->is_pushed_down, + rinfo->outerjoin_delayed, + pseudoconstant, + rinfo->security_level, + NULL, NULL, NULL); + child_rinfo->mergeopfamilies = rinfo->mergeopfamilies; + childquals = lappend(childquals, child_rinfo); /* track minimum security level among child quals */ cq_min_security = Min(cq_min_security, rinfo->security_level); } diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 48a7ebfe45..4fe1824eb0 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -16,6 +16,7 @@ #include "nodes/execnodes.h" #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" extern A_Expr *makeA_Expr(A_Expr_Kind kind, List *name, @@ -106,4 +107,6 @@ extern GroupingSet *makeGroupingSet(GroupingSetKind kind, List *content, int loc extern VacuumRelation *makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols); +extern UniqueKey* makeUniqueKey(List *exprs, bool multi_nullvals); + #endif /* MAKEFUNC_H */ diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index e22df890ef..3320273ac1 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -265,6 +265,7 @@ typedef enum NodeTag T_EquivalenceMember, T_PathKey, T_PathTarget, + T_UniqueKey, T_RestrictInfo, T_IndexClause, T_PlaceHolderVar, diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index 0d61f04d27..80561b1003 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -739,6 +739,7 @@ typedef struct RelOptInfo QualCost baserestrictcost; /* cost of evaluating the above */ Index baserestrict_min_security; /* min security_level found in * baserestrictinfo */ + List *uniquekeys; /* List of UniqueKey */ List *joininfo; /* RestrictInfo structures for join clauses * involving this rel */ bool has_eclass_joins; /* T means joininfo is incomplete */ @@ -1059,6 +1060,28 @@ typedef struct PathKey } PathKey; +/* + * UniqueKey + * + * Represents the unique properties held by a RelOptInfo. + * + * exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL + * is a special case of UniqueKey, which means there is only 1 row in that + * relation. + * multi_nullvals: true means multi null values may exist in these exprs, so the + * uniqueness is not guaranteed in this case. This field is necessary for + * remove_useless_join & reduce_unique_semijoins where we don't mind these + * duplicated NULL values. It is set to true for 2 cases. One is a unique key + * from a unique index but the related column is nullable. The other one is for + * outer join. see populate_joinrel_uniquekeys for detail. + */ +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + /* * PathTarget * @@ -2496,7 +2519,7 @@ typedef enum * * flags indicating what kinds of grouping are possible. * partial_costs_set is true if the agg_partial_costs and agg_final_costs - * have been initialized. + * have been initialized. * agg_partial_costs gives partial aggregation costs. * agg_final_costs gives finalization costs. * target_parallel_safe is true if target is parallel safe. @@ -2526,8 +2549,8 @@ typedef struct * limit_tuples is an estimated bound on the number of output tuples, * or -1 if no LIMIT or couldn't estimate. * count_est and offset_est are the estimated values of the LIMIT and OFFSET - * expressions computed by preprocess_limit() (see comments for - * preprocess_limit() for more information). + * expressions computed by preprocess_limit() (see comments for + * preprocess_limit() for more information). */ typedef struct { diff --git a/src/include/nodes/pg_list.h b/src/include/nodes/pg_list.h index 404e03f132..85c65e910c 100644 --- a/src/include/nodes/pg_list.h +++ b/src/include/nodes/pg_list.h @@ -566,6 +566,16 @@ extern pg_nodiscard List *list_delete_first(List *list); extern pg_nodiscard List *list_delete_last(List *list); extern pg_nodiscard List *list_delete_nth_cell(List *list, int n); extern pg_nodiscard List *list_delete_cell(List *list, ListCell *cell); +extern bool list_is_subset(const List *members, const List *target); + +extern List *list_delete(List *list, void *datum); +extern List *list_delete_ptr(List *list, void *datum); +extern List *list_delete_int(List *list, int datum); +extern List *list_delete_oid(List *list, Oid datum); +extern List *list_delete_first(List *list); +extern List *list_delete_last(List *list); +extern List *list_delete_nth_cell(List *list, int n); +extern List *list_delete_cell(List *list, ListCell *cell); extern List *list_union(const List *list1, const List *list2); extern List *list_union_ptr(const List *list1, const List *list2); diff --git a/src/include/optimizer/appendinfo.h b/src/include/optimizer/appendinfo.h index 4cbf8c26cc..43b8d05f69 100644 --- a/src/include/optimizer/appendinfo.h +++ b/src/include/optimizer/appendinfo.h @@ -32,4 +32,7 @@ extern Relids adjust_child_relids_multilevel(PlannerInfo *root, Relids relids, extern AppendRelInfo **find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos); +extern AppendRelInfo *find_appinfo_by_child(PlannerInfo *root, Index child_index); +extern Var *find_parent_var(AppendRelInfo *appinfo, Var *child_var); + #endif /* APPENDINFO_H */ diff --git a/src/include/optimizer/optimizer.h b/src/include/optimizer/optimizer.h index d587952b7d..843aafa51e 100644 --- a/src/include/optimizer/optimizer.h +++ b/src/include/optimizer/optimizer.h @@ -23,6 +23,7 @@ #define OPTIMIZER_H #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" /* Test if an expression node represents a SRF call. Beware multiple eval! */ #define IS_SRF_CALL(node) \ @@ -168,6 +169,7 @@ extern TargetEntry *get_sortgroupref_tle(Index sortref, List *targetList); extern TargetEntry *get_sortgroupclause_tle(SortGroupClause *sgClause, List *targetList); +extern Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); extern Node *get_sortgroupclause_expr(SortGroupClause *sgClause, List *targetList); extern List *get_sortgrouplist_exprs(List *sgClauses, diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h index 035d3e1206..1adf99c9ee 100644 --- a/src/include/optimizer/paths.h +++ b/src/include/optimizer/paths.h @@ -247,5 +247,48 @@ extern PathKey *make_canonical_pathkey(PlannerInfo *root, int strategy, bool nulls_first); extern void add_paths_to_append_rel(PlannerInfo *root, RelOptInfo *rel, List *live_childrels); +extern List *select_mergejoin_clauses(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + JoinType jointype, + bool *mergejoin_allowed); + +/* + * uniquekeys.c + * Utilities for matching and building unique keys + */ +extern void populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List* unique_index_list); +extern void populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels); +extern void populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel); +extern void populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel); +extern void populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel); +extern void simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel); +extern void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel); +extern void populate_joinrel_uniquekeys(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + RelOptInfo *rel2, + List *restrictlist, + JoinType jointype); + +extern bool relation_has_uniquekeys_for(PlannerInfo *root, + RelOptInfo *rel, + List *exprs, + bool allow_multinulls); +extern bool relation_is_onerow(RelOptInfo *rel); #endif /* PATHS_H */ -- 2.26.2 --yh7cw345mbttadjz Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v38-0003-Extend-UniqueKeys.patch" ^ permalink raw reply [nested|flat] 91+ messages in thread
* [PATCH v36 2/6] Introduce UniqueKey attributes on RelOptInfo struct. @ 2020-05-11 07:50 一挃 <[email protected]> 0 siblings, 0 replies; 91+ messages in thread From: 一挃 @ 2020-05-11 07:50 UTC (permalink / raw) UniqueKey is a set of exprs on RelOptInfo which represents the exprs will be unique on the given RelOptInfo. You can see README.uniquekey for more information. --- src/backend/nodes/copyfuncs.c | 13 + src/backend/nodes/list.c | 31 + src/backend/nodes/makefuncs.c | 13 + src/backend/nodes/outfuncs.c | 11 + src/backend/nodes/readfuncs.c | 10 + src/backend/optimizer/path/Makefile | 3 +- src/backend/optimizer/path/README.uniquekey | 131 +++ src/backend/optimizer/path/allpaths.c | 10 + src/backend/optimizer/path/joinpath.c | 9 +- src/backend/optimizer/path/joinrels.c | 2 + src/backend/optimizer/path/pathkeys.c | 3 +- src/backend/optimizer/path/uniquekeys.c | 1131 +++++++++++++++++++ src/backend/optimizer/plan/planner.c | 13 +- src/backend/optimizer/prep/prepunion.c | 2 + src/backend/optimizer/util/appendinfo.c | 44 + src/backend/optimizer/util/inherit.c | 16 +- src/include/nodes/makefuncs.h | 3 + src/include/nodes/nodes.h | 1 + src/include/nodes/pathnodes.h | 29 +- src/include/nodes/pg_list.h | 2 + src/include/optimizer/appendinfo.h | 3 + src/include/optimizer/optimizer.h | 2 + src/include/optimizer/paths.h | 43 + 23 files changed, 1502 insertions(+), 23 deletions(-) create mode 100644 src/backend/optimizer/path/README.uniquekey create mode 100644 src/backend/optimizer/path/uniquekeys.c diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 89c409de66..1f50400fd2 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2273,6 +2273,16 @@ _copyPathKey(const PathKey *from) return newnode; } +static UniqueKey * +_copyUniqueKey(const UniqueKey *from) +{ + UniqueKey *newnode = makeNode(UniqueKey); + + COPY_NODE_FIELD(exprs); + COPY_SCALAR_FIELD(multi_nullvals); + + return newnode; +} /* * _copyRestrictInfo */ @@ -5152,6 +5162,9 @@ copyObjectImpl(const void *from) case T_PathKey: retval = _copyPathKey(from); break; + case T_UniqueKey: + retval = _copyUniqueKey(from); + break; case T_RestrictInfo: retval = _copyRestrictInfo(from); break; diff --git a/src/backend/nodes/list.c b/src/backend/nodes/list.c index 80fa8c84e4..a7a99b70f2 100644 --- a/src/backend/nodes/list.c +++ b/src/backend/nodes/list.c @@ -687,6 +687,37 @@ list_member_oid(const List *list, Oid datum) return false; } +/* + * return true iff every entry in "members" list is also present + * in the "target" list. + */ +bool +list_is_subset(const List *members, const List *target) +{ + const ListCell *lc1, *lc2; + + Assert(IsPointerList(members)); + Assert(IsPointerList(target)); + check_list_invariants(members); + check_list_invariants(target); + + foreach(lc1, members) + { + bool found = false; + foreach(lc2, target) + { + if (equal(lfirst(lc1), lfirst(lc2))) + { + found = true; + break; + } + } + if (!found) + return false; + } + return true; +} + /* * Delete the n'th cell (counting from 0) in list. * diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 49de285f01..646cf7c9a1 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -814,3 +814,16 @@ makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols) v->va_cols = va_cols; return v; } + + +/* + * makeUniqueKey + */ +UniqueKey* +makeUniqueKey(List *exprs, bool multi_nullvals) +{ + UniqueKey * ukey = makeNode(UniqueKey); + ukey->exprs = exprs; + ukey->multi_nullvals = multi_nullvals; + return ukey; +} diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index e2f177515d..c3a9632992 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2428,6 +2428,14 @@ _outPathKey(StringInfo str, const PathKey *node) WRITE_BOOL_FIELD(pk_nulls_first); } +static void +_outUniqueKey(StringInfo str, const UniqueKey *node) +{ + WRITE_NODE_TYPE("UNIQUEKEY"); + WRITE_NODE_FIELD(exprs); + WRITE_BOOL_FIELD(multi_nullvals); +} + static void _outPathTarget(StringInfo str, const PathTarget *node) { @@ -4127,6 +4135,9 @@ outNode(StringInfo str, const void *obj) case T_PathKey: _outPathKey(str, obj); break; + case T_UniqueKey: + _outUniqueKey(str, obj); + break; case T_PathTarget: _outPathTarget(str, obj); break; diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index 42050ab719..3a18571d0c 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -452,6 +452,14 @@ _readSetOperationStmt(void) READ_DONE(); } +static UniqueKey * +_readUniqueKey(void) +{ + READ_LOCALS(UniqueKey); + READ_NODE_FIELD(exprs); + READ_BOOL_FIELD(multi_nullvals); + READ_DONE(); +} /* * Stuff from primnodes.h. @@ -2656,6 +2664,8 @@ parseNodeString(void) return_value = _readCommonTableExpr(); else if (MATCH("SETOPERATIONSTMT", 16)) return_value = _readSetOperationStmt(); + else if (MATCH("UNIQUEKEY", 9)) + return_value = _readUniqueKey(); else if (MATCH("ALIAS", 5)) return_value = _readAlias(); else if (MATCH("RANGEVAR", 8)) diff --git a/src/backend/optimizer/path/Makefile b/src/backend/optimizer/path/Makefile index 1e199ff66f..7b9820c25f 100644 --- a/src/backend/optimizer/path/Makefile +++ b/src/backend/optimizer/path/Makefile @@ -21,6 +21,7 @@ OBJS = \ joinpath.o \ joinrels.o \ pathkeys.o \ - tidpath.o + tidpath.o \ + uniquekeys.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/optimizer/path/README.uniquekey b/src/backend/optimizer/path/README.uniquekey new file mode 100644 index 0000000000..5eac761995 --- /dev/null +++ b/src/backend/optimizer/path/README.uniquekey @@ -0,0 +1,131 @@ +1. What is UniqueKey? +We can think UniqueKey is a set of exprs for a RelOptInfo, which we are insure +that doesn't yields same result among all the rows. The simplest UniqueKey +format is primary key. + +However we define the UnqiueKey as below. + +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + +exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL +is a special case of UniqueKey, which means there is only one row in that +relation.it has a stronger semantic than others. like SELECT uk FROM t; uk is +normal unique key and may have different values. SELECT colx FROM t WHERE uk = +const. colx is unique AND we have only 1 value. This field can used for +innerrel_is_unique. this logic is handled specially in add_uniquekey_for_onerow +function. + +multi_nullvals: true means multi null values may exist in these exprs, so the +uniqueness is not guaranteed in this case. This field is necessary for +remove_useless_join & reduce_unique_semijoins where we don't mind these +duplicated NULL values. It is set to true for 2 cases. One is a unique key +from a unique index but the related column is nullable. The other one is for +outer join. see populate_joinrel_uniquekeys for detail. + + +The UniqueKey can be used at the following cases at least: +1. remove_useless_joins. +2. reduce_semianti_joins +3. remove distinct node if distinct clause is unique. +4. remove aggnode if group by clause is unique. +5. Index Skip Scan (WIP) +6. Aggregation Push Down without 2 phase aggregation if the join can't + duplicated the aggregated rows. (work in progress feature) + +2. How is it maintained? + +We have a set of populate_xxx_unqiuekeys functions to maintain the uniquekey on +various cases. xxx includes baserel, joinrel, partitionedrel, distinctrel, +groupedrel, unionrel. and we also need to convert the uniquekey from subquery +to outer relation, which is what convert_subquery_uniquekeys does. + +1. The first part is about baserel. We handled 3 cases. suppose we have Unique +Index on (a, b). + +1. SELECT a, b FROM t. UniqueKey (a, b) +2. SELECT a FROM t WHERE b = 1; UniqueKey (a) +3. SELECT .. FROM t WHERE a = 1 AND b = 1; UniqueKey (NIL). onerow case, every + column is Unique. + +2. The next part is joinrel, this part is most error-prone, we simplified the rules +like below: +1. If the relation's UniqueKey can't be duplicated after join, then is will be + still valid for the join rel. The function we used here is + innerrel_keeps_unique. The basic idea is innerrel.any_col = outer.uk. + +2. If the UnqiueKey can't keep valid via the rule 1, the combination of the + UniqueKey from both sides are valid for sure. We can prove this as: if the + unique exprs from rel1 is duplicated by rel2, the duplicated rows must + contains different unique exprs from rel2. + +More considerations about onerow: +1. If relation with one row and it can't be duplicated, it is still possible + contains mulit_nullvas after outer join. +2. If the either UniqueKey can be duplicated after join, the can get one row + only when both side is one row AND there is no outer join. +3. Whenever the onerow UniqueKey is not a valid any more, we need to convert one + row UniqueKey to normal unique key since we don't store exprs for one-row + relation. get_exprs_from_uniquekeys will be used here. + + +More considerations about multi_nullvals after join: +1. If the original UnqiueKey has multi_nullvals, the final UniqueKey will have + mulit_nullvals in any case. +2. If a unique key doesn't allow mulit_nullvals, after some outer join, it + allows some outer join. + + +3. When we comes to subquery, we need to convert_subquery_unqiuekeys just like +convert_subquery_pathkeys. Only the UniqueKey insides subquery is referenced as +a Var in outer relation will be reused. The relationship between the outerrel.Var +and subquery.exprs is built with outerel->subroot->processed_tlist. + + +4. As for the SRF functions, it will break the uniqueness of uniquekey, However it +is handled in adjust_paths_for_srfs, which happens after the query_planner. so +we will maintain the UniqueKey until there and reset it to NIL at that +places. This can't help on distinct/group by elimination cases but probably help +in some other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is +semantic correctly. + + +5. As for inherit table, we first main the UnqiueKey on childrel as well. But for +partitioned table we need to maintain 2 different kinds of +UnqiueKey. 1). UniqueKey on the parent relation 2). UniqueKey on child +relation for partition wise query. + +Example: +CREATE TABLE p (a int not null, b int not null) partition by list (a); +CREATE TABLE p0 partition of p for values in (1); +CREATE TABLE p1 partition of p for values in (2); + +create unique index p0_b on p0(b); +create unique index p1_b on p1(b); + +Now b is only unique on partition level, so the distinct can't be removed on +the following cases. SELECT DISTINCT b FROM p; + +Another example is SELECT DISTINCT a, b FROM p WHERE a = 1; Since only one +partition is chosen, the UniqueKey on child relation is same as the UniqueKey on +parent relation. + +Another usage of UniqueKey on partition level is it be helpful for +partition-wise join. + +As for the UniqueKey on parent table level, it comes with 2 different ways, +1). the UniqueKey is also derived in UniqueKey index, but the index must be same +in all the related children relations and the unique index must contains +Partition Key in it. Example: + +CREATE UNIQUE INDEX p_ab ON p(a, b); -- where a is the partition key. + +-- Query +SELECT a, b FROM p; the (a, b) is a UniqueKey of p. + +2). If the parent relation has only one childrel, the UniqueKey on childrel is + the UniqueKey on parent as well. diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index 484dab0a1a..2ad9d06d7a 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -579,6 +579,12 @@ set_plain_rel_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) */ check_index_predicates(root, rel); + /* + * Now that we've marked which partial indexes are suitable, we can now + * build the relation's unique keys. + */ + populate_baserel_uniquekeys(root, rel, rel->indexlist); + /* Mark rel with estimated output rows, width, etc */ set_baserel_size_estimates(root, rel); } @@ -1310,6 +1316,8 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, /* Add paths to the append relation. */ add_paths_to_append_rel(root, rel, live_childrels); + if (IS_PARTITIONED_REL(rel)) + populate_partitionedrel_uniquekeys(root, rel, live_childrels); } @@ -2383,6 +2391,8 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, pathkeys, required_outer)); } + convert_subquery_uniquekeys(root, rel, sub_final_rel); + /* If outer rel allows parallelism, do same for partial paths. */ if (rel->consider_parallel && bms_is_empty(required_outer)) { diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c index db54a6ba2e..ef0fd2fb0b 100644 --- a/src/backend/optimizer/path/joinpath.c +++ b/src/backend/optimizer/path/joinpath.c @@ -71,13 +71,6 @@ static void consider_parallel_mergejoin(PlannerInfo *root, static void hash_inner_and_outer(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, RelOptInfo *innerrel, JoinType jointype, JoinPathExtraData *extra); -static List *select_mergejoin_clauses(PlannerInfo *root, - RelOptInfo *joinrel, - RelOptInfo *outerrel, - RelOptInfo *innerrel, - List *restrictlist, - JoinType jointype, - bool *mergejoin_allowed); static void generate_mergejoin_paths(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *innerrel, @@ -1927,7 +1920,7 @@ hash_inner_and_outer(PlannerInfo *root, * if it is mergejoinable and involves vars from the two sub-relations * currently of interest. */ -static List * +List * select_mergejoin_clauses(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c index 2d343cd293..b9163ee8ff 100644 --- a/src/backend/optimizer/path/joinrels.c +++ b/src/backend/optimizer/path/joinrels.c @@ -924,6 +924,8 @@ populate_joinrel_with_paths(PlannerInfo *root, RelOptInfo *rel1, /* Apply partitionwise join technique, if possible. */ try_partitionwise_join(root, rel1, rel2, joinrel, sjinfo, restrictlist); + + populate_joinrel_uniquekeys(root, joinrel, rel1, rel2, restrictlist, sjinfo->jointype); } diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index ce9bf87e9b..7e596d4194 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -33,7 +33,6 @@ static bool pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys); static bool matches_boolean_partition_clause(RestrictInfo *rinfo, RelOptInfo *partrel, int partkeycol); -static Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); static bool right_merge_direction(PlannerInfo *root, PathKey *pathkey); @@ -1035,7 +1034,7 @@ convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel, * We need this to ensure that we don't return pathkeys describing values * that are unavailable above the level of the subquery scan. */ -static Var * +Var * find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle) { ListCell *lc; diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c new file mode 100644 index 0000000000..b33bcd2f32 --- /dev/null +++ b/src/backend/optimizer/path/uniquekeys.c @@ -0,0 +1,1131 @@ +/*------------------------------------------------------------------------- + * + * uniquekeys.c + * Utilities for matching and building unique keys + * + * Portions Copyright (c) 2020, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/optimizer/path/uniquekeys.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "nodes/makefuncs.h" +#include "nodes/nodeFuncs.h" +#include "optimizer/pathnode.h" +#include "optimizer/paths.h" +#include "optimizer/appendinfo.h" +#include "optimizer/optimizer.h" +#include "optimizer/tlist.h" +#include "rewrite/rewriteManip.h" + + +/* + * This struct is used to help populate_joinrel_uniquekeys. + * + * added_to_joinrel is true if a uniquekey (from outerrel or innerrel) + * has been added to joinrel. + * useful is true if the exprs of the uniquekey still appears in joinrel. + */ +typedef struct UniqueKeyContextData +{ + UniqueKey *uniquekey; + bool added_to_joinrel; + bool useful; +} *UniqueKeyContext; + +static List *initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel); +static bool innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + bool reverse); + +static List *get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals); +static List *get_exprs_from_uniquekey(RelOptInfo *joinrel, + RelOptInfo *rel1, + UniqueKey *ukey); +static void add_uniquekey_for_onerow(RelOptInfo *rel); +static bool add_combined_uniquekey(RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype); + +/* Used for unique indexes checking for partitioned table */ +static bool index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind); +static IndexOptInfo *simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from); +static bool simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2); +static void adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_index); + +/* Helper function for grouped relation and distinct relation. */ +static void add_uniquekey_from_sortgroups(PlannerInfo *root, + RelOptInfo *rel, + List *sortgroups); + +/* + * populate_baserel_uniquekeys + * Populate 'baserel' uniquekeys list by looking at the rel's unique index + * and baserestrictinfo + */ +void +populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List *indexlist) +{ + ListCell *lc; + List *matched_uniq_indexes = NIL; + + /* Attrs appears in rel->reltarget->exprs. */ + Bitmapset *used_attrs = NULL; + + List *const_exprs = NIL; + List *expr_opfamilies = NIL; + + Assert(baserel->rtekind == RTE_RELATION); + + foreach(lc, indexlist) + { + IndexOptInfo *ind = (IndexOptInfo *) lfirst(lc); + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + matched_uniq_indexes = lappend(matched_uniq_indexes, ind); + } + + if (matched_uniq_indexes == NIL) + return; + + /* Check which attrs is used in baserel->reltarget */ + pull_varattnos((Node *)baserel->reltarget->exprs, baserel->relid, &used_attrs); + + /* Check which attrno is used at a mergeable const filter */ + foreach(lc, baserel->baserestrictinfo) + { + RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); + + if (rinfo->mergeopfamilies == NIL) + continue; + + if (bms_is_empty(rinfo->left_relids)) + { + const_exprs = lappend(const_exprs, get_rightop(rinfo->clause)); + } + else if (bms_is_empty(rinfo->right_relids)) + { + const_exprs = lappend(const_exprs, get_leftop(rinfo->clause)); + } + else + continue; + + expr_opfamilies = lappend(expr_opfamilies, rinfo->mergeopfamilies); + } + + foreach(lc, matched_uniq_indexes) + { + bool multi_nullvals, useful; + List *exprs = get_exprs_from_uniqueindex(lfirst_node(IndexOptInfo, lc), + const_exprs, + expr_opfamilies, + used_attrs, + &useful, + &multi_nullvals); + if (useful) + { + if (exprs == NIL) + { + /* All the columns in Unique Index matched with a restrictinfo */ + add_uniquekey_for_onerow(baserel); + return; + } + baserel->uniquekeys = lappend(baserel->uniquekeys, + makeUniqueKey(exprs, multi_nullvals)); + } + } +} + + +/* + * populate_partitionedrel_uniquekeys + * The UniqueKey on partitionrel comes from 2 cases: + * 1). Only one partition is involved in this query, the unique key can be + * copied to parent rel from childrel. + * 2). There are some unique index which includes partition key and exists + * in all the related partitions. + * We never mind rule 2 if we hit rule 1. + */ + +void +populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels) +{ + ListCell *lc; + List *global_uniq_indexlist = NIL; + RelOptInfo *childrel; + bool is_first = true; + + Assert(IS_PARTITIONED_REL(rel)); + + if (childrels == NIL) + return; + + /* + * If there is only one partition used in this query, the UniqueKey in childrel is + * still valid in parent level, but we need convert the format from child expr to + * parent expr. + */ + if (list_length(childrels) == 1) + { + /* Check for Rule 1 */ + RelOptInfo *childrel = linitial_node(RelOptInfo, childrels); + ListCell *lc; + Assert(childrel->reloptkind == RELOPT_OTHER_MEMBER_REL); + if (relation_is_onerow(childrel)) + { + add_uniquekey_for_onerow(rel); + return; + } + + foreach(lc, childrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + AppendRelInfo *appinfo = find_appinfo_by_child(root, childrel->relid); + List *parent_exprs = NIL; + bool can_reuse = true; + ListCell *lc2; + foreach(lc2, ukey->exprs) + { + Var *var = (Var *)lfirst(lc2); + /* + * If the expr comes from a expression, it is hard to build the expression + * in parent so ignore that case for now. + */ + if(!IsA(var, Var)) + { + can_reuse = false; + break; + } + /* Convert it to parent var */ + parent_exprs = lappend(parent_exprs, find_parent_var(appinfo, var)); + } + if (can_reuse) + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(parent_exprs, + ukey->multi_nullvals)); + } + } + else + { + /* Check for rule 2 */ + childrel = linitial_node(RelOptInfo, childrels); + foreach(lc, childrel->indexlist) + { + IndexOptInfo *ind = lfirst(lc); + IndexOptInfo *modified_index; + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + + /* + * During simple_copy_indexinfo_to_parent, we need to convert var from + * child var to parent var, index on expression is too complex to handle. + * so ignore it for now. + */ + if (ind->indexprs != NIL) + continue; + + modified_index = simple_copy_indexinfo_to_parent(root, rel, ind); + /* + * If the unique index doesn't contain partkey, then it is unique + * on this partition only, so it is useless for us. + */ + if (!index_constains_partkey(rel, modified_index)) + continue; + + global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index); + } + + if (global_uniq_indexlist != NIL) + { + foreach(lc, childrels) + { + RelOptInfo *child = lfirst(lc); + if (is_first) + { + is_first = false; + continue; + } + adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist); + } + /* Now we have a list of unique index which are exactly same on all childrels, + * Set the UniqueKey just like it is non-partition table + */ + populate_baserel_uniquekeys(root, rel, global_uniq_indexlist); + } + } +} + + +/* + * populate_distinctrel_uniquekeys + */ +void +populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel) +{ + /* The unique key before the distinct is still valid. */ + distinctrel->uniquekeys = list_copy(inputrel->uniquekeys); + add_uniquekey_from_sortgroups(root, distinctrel, root->parse->distinctClause); +} + +/* + * populate_grouprel_uniquekeys + */ +void +populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel) + +{ + Query *parse = root->parse; + bool input_ukey_added = false; + ListCell *lc; + + if (relation_is_onerow(inputrel)) + { + add_uniquekey_for_onerow(grouprel); + return; + } + if (parse->groupingSets) + return; + + /* A Normal group by without grouping set. */ + if (parse->groupClause) + { + /* + * Current even the groupby clause is Unique already, but if query has aggref + * We have to create grouprel still. To keep the UnqiueKey short, we will check + * the UniqueKey of input_rel still valid, if so we reuse it. + */ + foreach(lc, inputrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (list_is_subset(ukey->exprs, grouprel->reltarget->exprs)) + { + grouprel->uniquekeys = lappend(grouprel->uniquekeys, + ukey); + input_ukey_added = true; + } + } + if (!input_ukey_added) + /* + * group by clause must be a super-set of grouprel->reltarget->exprs except the + * aggregation expr, so if such exprs is unique already, no bother to generate + * new uniquekey for group by exprs. + */ + add_uniquekey_from_sortgroups(root, + grouprel, + root->parse->groupClause); + } + else + /* It has aggregation but without a group by, so only one row returned */ + add_uniquekey_for_onerow(grouprel); +} + +/* + * simple_copy_uniquekeys + * Using a function for the one-line code makes us easy to check where we simply + * copied the uniquekey. + */ +void +simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel) +{ + newrel->uniquekeys = oldrel->uniquekeys; +} + +/* + * populate_unionrel_uniquekeys + */ +void +populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel) +{ + ListCell *lc; + List *exprs = NIL; + + Assert(unionrel->uniquekeys == NIL); + + foreach(lc, unionrel->reltarget->exprs) + { + exprs = lappend(exprs, lfirst(lc)); + } + + if (exprs == NIL) + /* SQL: select union select; is valid, we need to handle it here. */ + add_uniquekey_for_onerow(unionrel); + else + unionrel->uniquekeys = lappend(unionrel->uniquekeys, + makeUniqueKey(exprs,false)); + +} + +/* + * populate_joinrel_uniquekeys + * + * populate uniquekeys for joinrel. We will check each relation to see if its + * UniqueKey is still valid via innerrel_keeps_unique, if so, we add it to + * joinrel. The multi_nullvals field will be changed to true for some outer + * join cases and one-row UniqueKey needs to be converted to normal UniqueKey + * for the same case as well. + * For the uniquekey in either baserel which can't be unique after join, we still + * check to see if combination of UniqueKeys from both side is still useful for us. + * if yes, we add it to joinrel as well. + */ +void +populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *outerrel, RelOptInfo *innerrel, + List *restrictlist, JoinType jointype) +{ + ListCell *lc, *lc2; + List *clause_list = NIL; + List *outerrel_ukey_ctx; + List *innerrel_ukey_ctx; + bool inner_onerow, outer_onerow; + bool mergejoin_allowed; + + /* Care about the outerrel relation only for SEMI/ANTI join */ + if (jointype == JOIN_SEMI || jointype == JOIN_ANTI) + { + foreach(lc, outerrel->uniquekeys) + { + UniqueKey *uniquekey = lfirst_node(UniqueKey, lc); + if (list_is_subset(uniquekey->exprs, joinrel->reltarget->exprs)) + joinrel->uniquekeys = lappend(joinrel->uniquekeys, uniquekey); + } + return; + } + + Assert(jointype == JOIN_LEFT || jointype == JOIN_FULL || jointype == JOIN_INNER); + + /* Fast path */ + if (innerrel->uniquekeys == NIL || outerrel->uniquekeys == NIL) + return; + + inner_onerow = relation_is_onerow(innerrel); + outer_onerow = relation_is_onerow(outerrel); + + outerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(outerrel); + innerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(innerrel); + + clause_list = select_mergejoin_clauses(root, joinrel, outerrel, innerrel, + restrictlist, jointype, + &mergejoin_allowed); + + if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true /* reverse */)) + { + bool outer_impact = jointype == JOIN_FULL; + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + /* Outer relation has one row, and the unique key is not duplicated after join, + * the joinrel will still has one row unless the jointype == JOIN_FULL. + */ + if (outer_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (outer_onerow) + { + /* + * The onerow outerrel becomes multi rows and multi_nullvals + * will be changed to true. We also need to set the exprs correctly since it + * can't be NIL any more. + */ + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(joinrel, outerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Change multi_nullvals to true due to the full join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, true)); + else + /* Just reuse it */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + } + ctx->added_to_joinrel = true; + } + } + + if (innerrel_keeps_unique(root, outerrel, innerrel, clause_list, false)) + { + bool outer_impact = jointype == JOIN_FULL || jointype == JOIN_LEFT;; + + foreach(lc, innerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + if (inner_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (inner_onerow) + { + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(joinrel, innerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Need to change multi_nullvals to true due to the outer join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, + true)); + else + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + + } + ctx->added_to_joinrel = true; + } + } + + /* + * The combination of the UniqueKey from both sides is unique as well regardless + * of join type, but no bother to add it if its subset has been added to joinrel + * already or it is not useful for the joinrel. + */ + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx1 = (UniqueKeyContext) lfirst(lc); + if (ctx1->added_to_joinrel || !ctx1->useful) + continue; + foreach(lc2, innerrel_ukey_ctx) + { + UniqueKeyContext ctx2 = (UniqueKeyContext) lfirst(lc2); + if (ctx2->added_to_joinrel || !ctx2->useful) + continue; + if (add_combined_uniquekey(joinrel, outerrel, innerrel, + ctx1->uniquekey, ctx2->uniquekey, + jointype)) + /* If we set a onerow UniqueKey to joinrel, we don't need other. */ + return; + } + } +} + + +/* + * convert_subquery_uniquekeys + * + * Covert the UniqueKey in subquery to outer relation. + */ +void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel) +{ + ListCell *lc; + + if (sub_final_rel->uniquekeys == NIL) + return; + + if (relation_is_onerow(sub_final_rel)) + { + add_uniquekey_for_onerow(currel); + return; + } + + Assert(currel->subroot != NULL); + + foreach(lc, sub_final_rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + ListCell *lc; + List *exprs = NIL; + bool ukey_useful = true; + + /* One row case is handled above */ + Assert(ukey->exprs != NIL); + foreach(lc, ukey->exprs) + { + Var *var; + TargetEntry *tle = tlist_member(lfirst(lc), + currel->subroot->processed_tlist); + if (tle == NULL) + { + ukey_useful = false; + break; + } + var = find_var_for_subquery_tle(currel, tle); + if (var == NULL) + { + ukey_useful = false; + break; + } + exprs = lappend(exprs, var); + } + + if (ukey_useful) + currel->uniquekeys = lappend(currel->uniquekeys, + makeUniqueKey(exprs, + ukey->multi_nullvals)); + + } +} + +/* + * innerrel_keeps_unique + * + * Check if Unique key of the innerrel is valid after join. innerrel's UniqueKey + * will be still valid if innerrel's any-column mergeop outrerel's uniquekey + * exists in clause_list. + * + * Note: the clause_list must be a list of mergeable restrictinfo already. + */ +static bool +innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *clause_list, + bool reverse) +{ + ListCell *lc, *lc2, *lc3; + + if (outerrel->uniquekeys == NIL || innerrel->uniquekeys == NIL) + return false; + + /* Check if there is outerrel's uniquekey in mergeable clause. */ + foreach(lc, outerrel->uniquekeys) + { + List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs; + bool clauselist_matchs_all_exprs = true; + foreach(lc2, outer_uq_exprs) + { + Node *outer_uq_expr = lfirst(lc2); + bool find_uq_expr_in_clauselist = false; + foreach(lc3, clause_list) + { + RestrictInfo *rinfo = lfirst_node(RestrictInfo, lc3); + Node *outer_expr; + if (reverse) + outer_expr = rinfo->outer_is_left ? get_rightop(rinfo->clause) : get_leftop(rinfo->clause); + else + outer_expr = rinfo->outer_is_left ? get_leftop(rinfo->clause) : get_rightop(rinfo->clause); + if (equal(outer_expr, outer_uq_expr)) + { + find_uq_expr_in_clauselist = true; + break; + } + } + if (!find_uq_expr_in_clauselist) + { + /* No need to check the next exprs in the current uniquekey */ + clauselist_matchs_all_exprs = false; + break; + } + } + + if (clauselist_matchs_all_exprs) + return true; + } + return false; +} + + +/* + * relation_is_onerow + * Check if it is a one-row relation by checking UniqueKey. + */ +bool +relation_is_onerow(RelOptInfo *rel) +{ + UniqueKey *ukey; + if (rel->uniquekeys == NIL) + return false; + ukey = linitial_node(UniqueKey, rel->uniquekeys); + return ukey->exprs == NIL && list_length(rel->uniquekeys) == 1; +} + +/* + * relation_has_uniquekeys_for + * Returns true if we have proofs that 'rel' cannot return multiple rows with + * the same values in each of 'exprs'. Otherwise returns false. + */ +bool +relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel, + List *exprs, bool allow_multinulls) +{ + ListCell *lc; + + /* + * For UniqueKey->onerow case, the uniquekey->exprs is empty as well + * so we can't rely on list_is_subset to handle this special cases + */ + if (exprs == NIL) + return false; + + foreach(lc, rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (ukey->multi_nullvals && !allow_multinulls) + continue; + if (list_is_subset(ukey->exprs, exprs)) + return true; + } + return false; +} + + +/* + * get_exprs_from_uniqueindex + * + * Return a list of exprs which is unique. set useful to false if this + * unique index is not useful for us. + */ +static List * +get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals) +{ + List *exprs = NIL; + ListCell *indexpr_item; + int c = 0; + + *useful = true; + *multi_nullvals = false; + + indexpr_item = list_head(unique_index->indexprs); + for(c = 0; c < unique_index->ncolumns; c++) + { + int attr = unique_index->indexkeys[c]; + Expr *expr; + bool matched_const = false; + ListCell *lc1, *lc2; + + if(attr > 0) + { + expr = list_nth_node(TargetEntry, unique_index->indextlist, c)->expr; + } + else if (attr == 0) + { + /* Expression index */ + expr = lfirst(indexpr_item); + indexpr_item = lnext(unique_index->indexprs, indexpr_item); + } + else /* attr < 0 */ + { + /* Index on system column is not supported */ + Assert(false); + } + + /* + * Check index_col = Const case with regarding to opfamily checking + * If we can remove the index_col from the final UniqueKey->exprs. + */ + forboth(lc1, const_exprs, lc2, const_expr_opfamilies) + { + if (list_member_oid((List *)lfirst(lc2), unique_index->opfamily[c]) + && match_index_to_operand((Node *) lfirst(lc1), c, unique_index)) + { + matched_const = true; + break; + } + } + + if (matched_const) + continue; + + /* Check if the indexed expr is used in rel */ + if (attr > 0) + { + /* + * Normal Indexed column, if the col is not used, then the index is useless + * for uniquekey. + */ + attr -= FirstLowInvalidHeapAttributeNumber; + + if (!bms_is_member(attr, used_varattrs)) + { + *useful = false; + break; + } + } + else if (!list_member(unique_index->rel->reltarget->exprs, expr)) + { + /* Expression index but the expression is not used in rel */ + *useful = false; + break; + } + + /* check not null property. */ + if (attr == 0) + { + /* We never know if a expression yields null or not */ + *multi_nullvals = true; + } + else if (!bms_is_member(attr, unique_index->rel->notnullattrs) + && !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber, + unique_index->rel->notnullattrs)) + { + *multi_nullvals = true; + } + + exprs = lappend(exprs, expr); + } + return exprs; +} + + +/* + * add_uniquekey_for_onerow + * If we are sure that the relation only returns one row, then all the columns + * are unique. However we don't need to create UniqueKey for every column, we + * just set exprs = NIL and overwrites all the other UniqueKey on this RelOptInfo + * since this one has strongest semantics. + */ +void +add_uniquekey_for_onerow(RelOptInfo *rel) +{ + /* + * We overwrite the previous UniqueKey on purpose since this one has the + * strongest semantic. + */ + rel->uniquekeys = list_make1(makeUniqueKey(NIL, false)); +} + + +/* + * initililze_uniquecontext_for_joinrel + * Return a List of UniqueKeyContext for an inputrel + */ +static List * +initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel) +{ + List *res = NIL; + ListCell *lc; + foreach(lc, inputrel->uniquekeys) + { + UniqueKeyContext context; + context = palloc(sizeof(struct UniqueKeyContextData)); + context->uniquekey = lfirst_node(UniqueKey, lc); + context->added_to_joinrel = false; + context->useful = true; + res = lappend(res, context); + } + return res; +} + + +/* + * get_exprs_from_uniquekey + * Unify the way of get List of exprs from a one-row UniqueKey or + * normal UniqueKey. for the onerow case, every expr in rel1 is a valid + * UniqueKey. Return a List of exprs. + * + * rel1: The relation which you want to get the exprs. + * ukey: The UniqueKey you want to get the exprs. + */ +static List * +get_exprs_from_uniquekey(RelOptInfo *joinrel, RelOptInfo *rel1, UniqueKey *ukey) +{ + ListCell *lc; + bool onerow = rel1 != NULL && relation_is_onerow(rel1); + + List *res = NIL; + Assert(onerow || ukey); + if (onerow) + { + /* Only cares about the exprs still exist in joinrel */ + foreach(lc, joinrel->reltarget->exprs) + { + Bitmapset *relids = pull_varnos(lfirst(lc)); + if (bms_is_subset(relids, rel1->relids)) + { + res = lappend(res, list_make1(lfirst(lc))); + } + } + } + else + { + res = list_make1(ukey->exprs); + } + return res; +} + +/* + * Partitioned table Unique Keys. + * The partition table unique key is maintained as: + * 1. The index must be unique as usual. + * 2. The index must contains partition key. + * 3. The index must exist on all the child rel. see simple_indexinfo_equal for + * how we compare it. + */ + +/* + * index_constains_partkey + * return true if the index contains the partiton key. + */ +static bool +index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind) +{ + ListCell *lc; + int i; + Assert(IS_PARTITIONED_REL(partrel)); + Assert(partrel->part_scheme->partnatts > 0); + + for(i = 0; i < partrel->part_scheme->partnatts; i++) + { + Node *part_expr = linitial(partrel->partexprs[i]); + bool found_in_index = false; + foreach(lc, ind->indextlist) + { + Expr *index_expr = lfirst_node(TargetEntry, lc)->expr; + if (equal(index_expr, part_expr)) + { + found_in_index = true; + break; + } + } + if (!found_in_index) + return false; + } + return true; +} + +/* + * simple_indexinfo_equal + * + * Used to check if the 2 index is same as each other. The index here + * is COPIED from childrel and did some tiny changes(see + * simple_copy_indexinfo_to_parent) + */ +static bool +simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2) +{ + Size oid_cmp_len = sizeof(Oid) * ind1->ncolumns; + + return ind1->ncolumns == ind2->ncolumns && + ind1->unique == ind2->unique && + memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0 && + memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0 && + memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0 && + memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0 && + equal(get_tlist_exprs(ind1->indextlist, true), + get_tlist_exprs(ind2->indextlist, true)); +} + + +/* + * The below macros are used for simple_copy_indexinfo_to_parent which is so + * customized that I don't want to put it to copyfuncs.c. So copy it here. + */ +#define COPY_POINTER_FIELD(fldname, sz) \ + do { \ + Size _size = (sz); \ + newnode->fldname = palloc(_size); \ + memcpy(newnode->fldname, from->fldname, _size); \ + } while (0) + +#define COPY_NODE_FIELD(fldname) \ + (newnode->fldname = copyObjectImpl(from->fldname)) + +#define COPY_SCALAR_FIELD(fldname) \ + (newnode->fldname = from->fldname) + + +/* + * simple_copy_indexinfo_to_parent (from partition) + * Copy the IndexInfo from child relation to parent relation with some modification, + * which is used to test: + * 1. If the same index exists in all the childrels. + * 2. If the parentrel->reltarget/basicrestrict info matches this index. + */ +static IndexOptInfo * +simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from) +{ + IndexOptInfo *newnode = makeNode(IndexOptInfo); + AppendRelInfo *appinfo = find_appinfo_by_child(root, from->rel->relid); + ListCell *lc; + int idx = 0; + + COPY_SCALAR_FIELD(ncolumns); + COPY_SCALAR_FIELD(nkeycolumns); + COPY_SCALAR_FIELD(unique); + COPY_SCALAR_FIELD(immediate); + /* We just need to know if it is NIL or not */ + COPY_SCALAR_FIELD(indpred); + COPY_SCALAR_FIELD(predOK); + COPY_POINTER_FIELD(indexkeys, from->ncolumns * sizeof(int)); + COPY_POINTER_FIELD(indexcollations, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opfamily, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opcintype, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(sortopfamily, from->ncolumns * sizeof(Oid)); + COPY_NODE_FIELD(indextlist); + + /* Convert index exprs on child expr to expr on parent */ + foreach(lc, newnode->indextlist) + { + TargetEntry *tle = lfirst_node(TargetEntry, lc); + /* Index on expression is ignored */ + Assert(IsA(tle->expr, Var)); + tle->expr = (Expr *) find_parent_var(appinfo, (Var *) tle->expr); + newnode->indexkeys[idx] = castNode(Var, tle->expr)->varattno; + idx++; + } + newnode->rel = parentrel; + return newnode; +} + +/* + * adjust_partition_unique_indexlist + * + * global_unique_indexes: At the beginning, it contains the copy & modified + * unique index from the first partition. And then check if each index in it still + * exists in the following partitions. If no, remove it. at last, it has an + * index list which exists in all the partitions. + */ +static void +adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_indexes) +{ + ListCell *lc, *lc2; + foreach(lc, *global_unique_indexes) + { + IndexOptInfo *g_ind = lfirst_node(IndexOptInfo, lc); + bool found_in_child = false; + + foreach(lc2, childrel->indexlist) + { + IndexOptInfo *p_ind = lfirst_node(IndexOptInfo, lc2); + IndexOptInfo *p_ind_copy; + if (!p_ind->unique || !p_ind->immediate || + (p_ind->indpred != NIL && !p_ind->predOK)) + continue; + p_ind_copy = simple_copy_indexinfo_to_parent(root, parentrel, p_ind); + if (simple_indexinfo_equal(p_ind_copy, g_ind)) + { + found_in_child = true; + break; + } + } + if (!found_in_child) + /* The index doesn't exist in childrel, remove it from global_unique_indexes */ + *global_unique_indexes = foreach_delete_current(*global_unique_indexes, lc); + } +} + +/* Helper function for groupres/distinctrel */ +static void +add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgroups) +{ + Query *parse = root->parse; + List *exprs; + + /* + * XXX: If there are some vars which is not in current levelsup, the semantic is + * imprecise, should we avoid it or not? levelsup = 1 is just a demo, maybe we need to + * check every level other than 0, if so, looks we have to write another + * pull_var_walker. + */ + List *upper_vars = pull_vars_of_level((Node*)sortgroups, 1); + + if (upper_vars != NIL) + return; + + exprs = get_sortgrouplist_exprs(sortgroups, parse->targetList); + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(exprs, + false /* sortgroupclause can't be multi_nullvals */)); +} + + +/* + * add_combined_uniquekey + * The combination of both UniqueKeys is a valid UniqueKey for joinrel no matter + * the jointype. + */ +bool +add_combined_uniquekey(RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype) +{ + + ListCell *lc1, *lc2; + + /* Either side has multi_nullvals or we have outer join, + * the combined UniqueKey has multi_nullvals */ + bool multi_nullvals = outer_ukey->multi_nullvals || + inner_ukey->multi_nullvals || IS_OUTER_JOIN(jointype); + + /* The only case we can get onerow joinrel after join */ + if (relation_is_onerow(outer_rel) + && relation_is_onerow(inner_rel) + && jointype == JOIN_INNER) + { + add_uniquekey_for_onerow(joinrel); + return true; + } + + foreach(lc1, get_exprs_from_uniquekey(joinrel, outer_rel, outer_ukey)) + { + foreach(lc2, get_exprs_from_uniquekey(joinrel, inner_rel, inner_ukey)) + { + List *exprs = list_concat_copy(lfirst_node(List, lc1), lfirst_node(List, lc2)); + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(exprs, + multi_nullvals)); + } + } + return false; +} diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index b406d41e91..0551ae0512 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -2389,6 +2389,8 @@ grouping_planner(PlannerInfo *root, bool inheritance_update, add_path(final_rel, path); } + simple_copy_uniquekeys(current_rel, final_rel); + /* * Generate partial paths for final_rel, too, if outer query levels might * be able to make use of them. @@ -3899,6 +3901,8 @@ create_grouping_paths(PlannerInfo *root, } set_cheapest(grouped_rel); + + populate_grouprel_uniquekeys(root, grouped_rel, input_rel); return grouped_rel; } @@ -4615,7 +4619,7 @@ create_window_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(window_rel); - + simple_copy_uniquekeys(input_rel, window_rel); return window_rel; } @@ -4911,7 +4915,7 @@ create_distinct_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(distinct_rel); - + populate_distinctrel_uniquekeys(root, input_rel, distinct_rel); return distinct_rel; } @@ -5172,6 +5176,8 @@ create_ordered_paths(PlannerInfo *root, */ Assert(ordered_rel->pathlist != NIL); + simple_copy_uniquekeys(input_rel, ordered_rel); + return ordered_rel; } @@ -6049,6 +6055,9 @@ adjust_paths_for_srfs(PlannerInfo *root, RelOptInfo *rel, if (list_length(targets) == 1) return; + /* UniqueKey is not valid after handling the SRF. */ + rel->uniquekeys = NIL; + /* * Stack SRF-evaluation nodes atop each path for the rel. * diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c index 951aed80e7..e94e92937c 100644 --- a/src/backend/optimizer/prep/prepunion.c +++ b/src/backend/optimizer/prep/prepunion.c @@ -689,6 +689,8 @@ generate_union_paths(SetOperationStmt *op, PlannerInfo *root, /* Undo effects of possibly forcing tuple_fraction to 0 */ root->tuple_fraction = save_fraction; + /* Add the UniqueKeys */ + populate_unionrel_uniquekeys(root, result_rel); return result_rel; } diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c index d722063cf3..44c37ecffc 100644 --- a/src/backend/optimizer/util/appendinfo.c +++ b/src/backend/optimizer/util/appendinfo.c @@ -746,3 +746,47 @@ find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos) } return appinfos; } + +/* + * find_appinfo_by_child + * + */ +AppendRelInfo * +find_appinfo_by_child(PlannerInfo *root, Index child_index) +{ + ListCell *lc; + foreach(lc, root->append_rel_list) + { + AppendRelInfo *appinfo = lfirst_node(AppendRelInfo, lc); + if (appinfo->child_relid == child_index) + return appinfo; + } + elog(ERROR, "parent relation cant be found"); + return NULL; +} + +/* + * find_parent_var + * + */ +Var * +find_parent_var(AppendRelInfo *appinfo, Var *child_var) +{ + ListCell *lc; + Var *res = NULL; + Index attno = 1; + foreach(lc, appinfo->translated_vars) + { + Node *child_node = lfirst(lc); + if (equal(child_node, child_var)) + { + res = copyObject(child_var); + res->varattno = attno; + res->varno = appinfo->parent_relid; + } + attno++; + } + if (res == NULL) + elog(ERROR, "parent var can't be found."); + return res; +} diff --git a/src/backend/optimizer/util/inherit.c b/src/backend/optimizer/util/inherit.c index 3132fd35a5..d66b40ec50 100644 --- a/src/backend/optimizer/util/inherit.c +++ b/src/backend/optimizer/util/inherit.c @@ -736,6 +736,7 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, { Node *onecq = (Node *) lfirst(lc2); bool pseudoconstant; + RestrictInfo *child_rinfo; /* check for pseudoconstant (no Vars or volatile functions) */ pseudoconstant = @@ -747,13 +748,14 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, root->hasPseudoConstantQuals = true; } /* reconstitute RestrictInfo with appropriate properties */ - childquals = lappend(childquals, - make_restrictinfo((Expr *) onecq, - rinfo->is_pushed_down, - rinfo->outerjoin_delayed, - pseudoconstant, - rinfo->security_level, - NULL, NULL, NULL)); + child_rinfo = make_restrictinfo((Expr *) onecq, + rinfo->is_pushed_down, + rinfo->outerjoin_delayed, + pseudoconstant, + rinfo->security_level, + NULL, NULL, NULL); + child_rinfo->mergeopfamilies = rinfo->mergeopfamilies; + childquals = lappend(childquals, child_rinfo); /* track minimum security level among child quals */ cq_min_security = Min(cq_min_security, rinfo->security_level); } diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 31d9aedeeb..c83f17acb7 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -16,6 +16,7 @@ #include "nodes/execnodes.h" #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" extern A_Expr *makeA_Expr(A_Expr_Kind kind, List *name, @@ -105,4 +106,6 @@ extern GroupingSet *makeGroupingSet(GroupingSetKind kind, List *content, int loc extern VacuumRelation *makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols); +extern UniqueKey* makeUniqueKey(List *exprs, bool multi_nullvals); + #endif /* MAKEFUNC_H */ diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 381d84b4e4..41110ed888 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -264,6 +264,7 @@ typedef enum NodeTag T_EquivalenceMember, T_PathKey, T_PathTarget, + T_UniqueKey, T_RestrictInfo, T_IndexClause, T_PlaceHolderVar, diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index 9e3ebd488a..02e4458bef 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -730,6 +730,7 @@ typedef struct RelOptInfo QualCost baserestrictcost; /* cost of evaluating the above */ Index baserestrict_min_security; /* min security_level found in * baserestrictinfo */ + List *uniquekeys; /* List of UniqueKey */ List *joininfo; /* RestrictInfo structures for join clauses * involving this rel */ bool has_eclass_joins; /* T means joininfo is incomplete */ @@ -1047,6 +1048,28 @@ typedef struct PathKey } PathKey; +/* + * UniqueKey + * + * Represents the unique properties held by a RelOptInfo. + * + * exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL + * is a special case of UniqueKey, which means there is only 1 row in that + * relation. + * multi_nullvals: true means multi null values may exist in these exprs, so the + * uniqueness is not guaranteed in this case. This field is necessary for + * remove_useless_join & reduce_unique_semijoins where we don't mind these + * duplicated NULL values. It is set to true for 2 cases. One is a unique key + * from a unique index but the related column is nullable. The other one is for + * outer join. see populate_joinrel_uniquekeys for detail. + */ +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + /* * PathTarget * @@ -2473,7 +2496,7 @@ typedef enum * * flags indicating what kinds of grouping are possible. * partial_costs_set is true if the agg_partial_costs and agg_final_costs - * have been initialized. + * have been initialized. * agg_partial_costs gives partial aggregation costs. * agg_final_costs gives finalization costs. * target_parallel_safe is true if target is parallel safe. @@ -2503,8 +2526,8 @@ typedef struct * limit_tuples is an estimated bound on the number of output tuples, * or -1 if no LIMIT or couldn't estimate. * count_est and offset_est are the estimated values of the LIMIT and OFFSET - * expressions computed by preprocess_limit() (see comments for - * preprocess_limit() for more information). + * expressions computed by preprocess_limit() (see comments for + * preprocess_limit() for more information). */ typedef struct { diff --git a/src/include/nodes/pg_list.h b/src/include/nodes/pg_list.h index 14ea2766ad..621f54a9f8 100644 --- a/src/include/nodes/pg_list.h +++ b/src/include/nodes/pg_list.h @@ -528,6 +528,8 @@ extern bool list_member_ptr(const List *list, const void *datum); extern bool list_member_int(const List *list, int datum); extern bool list_member_oid(const List *list, Oid datum); +extern bool list_is_subset(const List *members, const List *target); + extern List *list_delete(List *list, void *datum); extern List *list_delete_ptr(List *list, void *datum); extern List *list_delete_int(List *list, int datum); diff --git a/src/include/optimizer/appendinfo.h b/src/include/optimizer/appendinfo.h index d6a27a60dd..e87c92a054 100644 --- a/src/include/optimizer/appendinfo.h +++ b/src/include/optimizer/appendinfo.h @@ -32,4 +32,7 @@ extern Relids adjust_child_relids_multilevel(PlannerInfo *root, Relids relids, extern AppendRelInfo **find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos); +extern AppendRelInfo *find_appinfo_by_child(PlannerInfo *root, Index child_index); +extern Var *find_parent_var(AppendRelInfo *appinfo, Var *child_var); + #endif /* APPENDINFO_H */ diff --git a/src/include/optimizer/optimizer.h b/src/include/optimizer/optimizer.h index 3e4171056e..9445141263 100644 --- a/src/include/optimizer/optimizer.h +++ b/src/include/optimizer/optimizer.h @@ -23,6 +23,7 @@ #define OPTIMIZER_H #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" /* * We don't want to include nodes/pathnodes.h here, because non-planner @@ -156,6 +157,7 @@ extern TargetEntry *get_sortgroupref_tle(Index sortref, List *targetList); extern TargetEntry *get_sortgroupclause_tle(SortGroupClause *sgClause, List *targetList); +extern Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); extern Node *get_sortgroupclause_expr(SortGroupClause *sgClause, List *targetList); extern List *get_sortgrouplist_exprs(List *sgClauses, diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h index 10b6e81079..9217a8d6c6 100644 --- a/src/include/optimizer/paths.h +++ b/src/include/optimizer/paths.h @@ -240,5 +240,48 @@ extern PathKey *make_canonical_pathkey(PlannerInfo *root, int strategy, bool nulls_first); extern void add_paths_to_append_rel(PlannerInfo *root, RelOptInfo *rel, List *live_childrels); +extern List *select_mergejoin_clauses(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + JoinType jointype, + bool *mergejoin_allowed); + +/* + * uniquekeys.c + * Utilities for matching and building unique keys + */ +extern void populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List* unique_index_list); +extern void populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels); +extern void populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel); +extern void populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel); +extern void populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel); +extern void simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel); +extern void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel); +extern void populate_joinrel_uniquekeys(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + RelOptInfo *rel2, + List *restrictlist, + JoinType jointype); + +extern bool relation_has_uniquekeys_for(PlannerInfo *root, + RelOptInfo *rel, + List *exprs, + bool allow_multinulls); +extern bool relation_is_onerow(RelOptInfo *rel); #endif /* PATHS_H */ -- 2.21.0 --op7mf2i72xk5yu7c Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v36-0003-Extend-UniqueKeys.patch" ^ permalink raw reply [nested|flat] 91+ messages in thread
* [PATCH v38 2/6] Introduce UniqueKey attributes on RelOptInfo struct. @ 2020-05-11 07:50 一挃 <[email protected]> 0 siblings, 0 replies; 91+ messages in thread From: 一挃 @ 2020-05-11 07:50 UTC (permalink / raw) UniqueKey is a set of exprs on RelOptInfo which represents the exprs will be unique on the given RelOptInfo. You can see README.uniquekey for more information. --- src/backend/nodes/copyfuncs.c | 13 + src/backend/nodes/list.c | 31 + src/backend/nodes/makefuncs.c | 13 + src/backend/nodes/outfuncs.c | 11 + src/backend/nodes/readfuncs.c | 10 + src/backend/optimizer/path/Makefile | 3 +- src/backend/optimizer/path/README.uniquekey | 131 +++ src/backend/optimizer/path/allpaths.c | 10 + src/backend/optimizer/path/joinpath.c | 9 +- src/backend/optimizer/path/joinrels.c | 2 + src/backend/optimizer/path/pathkeys.c | 3 +- src/backend/optimizer/path/uniquekeys.c | 1135 +++++++++++++++++++ src/backend/optimizer/plan/planner.c | 13 +- src/backend/optimizer/prep/prepunion.c | 2 + src/backend/optimizer/util/appendinfo.c | 44 + src/backend/optimizer/util/inherit.c | 18 +- src/include/nodes/makefuncs.h | 3 + src/include/nodes/nodes.h | 1 + src/include/nodes/pathnodes.h | 29 +- src/include/nodes/pg_list.h | 10 + src/include/optimizer/appendinfo.h | 3 + src/include/optimizer/optimizer.h | 2 + src/include/optimizer/paths.h | 43 + 23 files changed, 1515 insertions(+), 24 deletions(-) create mode 100644 src/backend/optimizer/path/README.uniquekey create mode 100644 src/backend/optimizer/path/uniquekeys.c diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..75c1c5e824 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2296,6 +2296,16 @@ _copyPathKey(const PathKey *from) return newnode; } +static UniqueKey * +_copyUniqueKey(const UniqueKey *from) +{ + UniqueKey *newnode = makeNode(UniqueKey); + + COPY_NODE_FIELD(exprs); + COPY_SCALAR_FIELD(multi_nullvals); + + return newnode; +} /* * _copyRestrictInfo */ @@ -5220,6 +5230,9 @@ copyObjectImpl(const void *from) case T_PathKey: retval = _copyPathKey(from); break; + case T_UniqueKey: + retval = _copyUniqueKey(from); + break; case T_RestrictInfo: retval = _copyRestrictInfo(from); break; diff --git a/src/backend/nodes/list.c b/src/backend/nodes/list.c index dbf6b30233..ca099495a1 100644 --- a/src/backend/nodes/list.c +++ b/src/backend/nodes/list.c @@ -702,6 +702,37 @@ list_member_oid(const List *list, Oid datum) return false; } +/* + * return true iff every entry in "members" list is also present + * in the "target" list. + */ +bool +list_is_subset(const List *members, const List *target) +{ + const ListCell *lc1, *lc2; + + Assert(IsPointerList(members)); + Assert(IsPointerList(target)); + check_list_invariants(members); + check_list_invariants(target); + + foreach(lc1, members) + { + bool found = false; + foreach(lc2, target) + { + if (equal(lfirst(lc1), lfirst(lc2))) + { + found = true; + break; + } + } + if (!found) + return false; + } + return true; +} + /* * Delete the n'th cell (counting from 0) in list. * diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 01c110cd2f..40415d0f5b 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -815,3 +815,16 @@ makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols) v->va_cols = va_cols; return v; } + + +/* + * makeUniqueKey + */ +UniqueKey* +makeUniqueKey(List *exprs, bool multi_nullvals) +{ + UniqueKey * ukey = makeNode(UniqueKey); + ukey->exprs = exprs; + ukey->multi_nullvals = multi_nullvals; + return ukey; +} diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..44154cde6a 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2456,6 +2456,14 @@ _outPathKey(StringInfo str, const PathKey *node) WRITE_BOOL_FIELD(pk_nulls_first); } +static void +_outUniqueKey(StringInfo str, const UniqueKey *node) +{ + WRITE_NODE_TYPE("UNIQUEKEY"); + WRITE_NODE_FIELD(exprs); + WRITE_BOOL_FIELD(multi_nullvals); +} + static void _outPathTarget(StringInfo str, const PathTarget *node) { @@ -4198,6 +4206,9 @@ outNode(StringInfo str, const void *obj) case T_PathKey: _outPathKey(str, obj); break; + case T_UniqueKey: + _outUniqueKey(str, obj); + break; case T_PathTarget: _outPathTarget(str, obj); break; diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index c5e136e9c3..b3e212bf1c 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -492,6 +492,14 @@ _readSetOperationStmt(void) READ_DONE(); } +static UniqueKey * +_readUniqueKey(void) +{ + READ_LOCALS(UniqueKey); + READ_NODE_FIELD(exprs); + READ_BOOL_FIELD(multi_nullvals); + READ_DONE(); +} /* * Stuff from primnodes.h. @@ -2717,6 +2725,8 @@ parseNodeString(void) return_value = _readCommonTableExpr(); else if (MATCH("SETOPERATIONSTMT", 16)) return_value = _readSetOperationStmt(); + else if (MATCH("UNIQUEKEY", 9)) + return_value = _readUniqueKey(); else if (MATCH("ALIAS", 5)) return_value = _readAlias(); else if (MATCH("RANGEVAR", 8)) diff --git a/src/backend/optimizer/path/Makefile b/src/backend/optimizer/path/Makefile index 1e199ff66f..7b9820c25f 100644 --- a/src/backend/optimizer/path/Makefile +++ b/src/backend/optimizer/path/Makefile @@ -21,6 +21,7 @@ OBJS = \ joinpath.o \ joinrels.o \ pathkeys.o \ - tidpath.o + tidpath.o \ + uniquekeys.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/optimizer/path/README.uniquekey b/src/backend/optimizer/path/README.uniquekey new file mode 100644 index 0000000000..5eac761995 --- /dev/null +++ b/src/backend/optimizer/path/README.uniquekey @@ -0,0 +1,131 @@ +1. What is UniqueKey? +We can think UniqueKey is a set of exprs for a RelOptInfo, which we are insure +that doesn't yields same result among all the rows. The simplest UniqueKey +format is primary key. + +However we define the UnqiueKey as below. + +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + +exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL +is a special case of UniqueKey, which means there is only one row in that +relation.it has a stronger semantic than others. like SELECT uk FROM t; uk is +normal unique key and may have different values. SELECT colx FROM t WHERE uk = +const. colx is unique AND we have only 1 value. This field can used for +innerrel_is_unique. this logic is handled specially in add_uniquekey_for_onerow +function. + +multi_nullvals: true means multi null values may exist in these exprs, so the +uniqueness is not guaranteed in this case. This field is necessary for +remove_useless_join & reduce_unique_semijoins where we don't mind these +duplicated NULL values. It is set to true for 2 cases. One is a unique key +from a unique index but the related column is nullable. The other one is for +outer join. see populate_joinrel_uniquekeys for detail. + + +The UniqueKey can be used at the following cases at least: +1. remove_useless_joins. +2. reduce_semianti_joins +3. remove distinct node if distinct clause is unique. +4. remove aggnode if group by clause is unique. +5. Index Skip Scan (WIP) +6. Aggregation Push Down without 2 phase aggregation if the join can't + duplicated the aggregated rows. (work in progress feature) + +2. How is it maintained? + +We have a set of populate_xxx_unqiuekeys functions to maintain the uniquekey on +various cases. xxx includes baserel, joinrel, partitionedrel, distinctrel, +groupedrel, unionrel. and we also need to convert the uniquekey from subquery +to outer relation, which is what convert_subquery_uniquekeys does. + +1. The first part is about baserel. We handled 3 cases. suppose we have Unique +Index on (a, b). + +1. SELECT a, b FROM t. UniqueKey (a, b) +2. SELECT a FROM t WHERE b = 1; UniqueKey (a) +3. SELECT .. FROM t WHERE a = 1 AND b = 1; UniqueKey (NIL). onerow case, every + column is Unique. + +2. The next part is joinrel, this part is most error-prone, we simplified the rules +like below: +1. If the relation's UniqueKey can't be duplicated after join, then is will be + still valid for the join rel. The function we used here is + innerrel_keeps_unique. The basic idea is innerrel.any_col = outer.uk. + +2. If the UnqiueKey can't keep valid via the rule 1, the combination of the + UniqueKey from both sides are valid for sure. We can prove this as: if the + unique exprs from rel1 is duplicated by rel2, the duplicated rows must + contains different unique exprs from rel2. + +More considerations about onerow: +1. If relation with one row and it can't be duplicated, it is still possible + contains mulit_nullvas after outer join. +2. If the either UniqueKey can be duplicated after join, the can get one row + only when both side is one row AND there is no outer join. +3. Whenever the onerow UniqueKey is not a valid any more, we need to convert one + row UniqueKey to normal unique key since we don't store exprs for one-row + relation. get_exprs_from_uniquekeys will be used here. + + +More considerations about multi_nullvals after join: +1. If the original UnqiueKey has multi_nullvals, the final UniqueKey will have + mulit_nullvals in any case. +2. If a unique key doesn't allow mulit_nullvals, after some outer join, it + allows some outer join. + + +3. When we comes to subquery, we need to convert_subquery_unqiuekeys just like +convert_subquery_pathkeys. Only the UniqueKey insides subquery is referenced as +a Var in outer relation will be reused. The relationship between the outerrel.Var +and subquery.exprs is built with outerel->subroot->processed_tlist. + + +4. As for the SRF functions, it will break the uniqueness of uniquekey, However it +is handled in adjust_paths_for_srfs, which happens after the query_planner. so +we will maintain the UniqueKey until there and reset it to NIL at that +places. This can't help on distinct/group by elimination cases but probably help +in some other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is +semantic correctly. + + +5. As for inherit table, we first main the UnqiueKey on childrel as well. But for +partitioned table we need to maintain 2 different kinds of +UnqiueKey. 1). UniqueKey on the parent relation 2). UniqueKey on child +relation for partition wise query. + +Example: +CREATE TABLE p (a int not null, b int not null) partition by list (a); +CREATE TABLE p0 partition of p for values in (1); +CREATE TABLE p1 partition of p for values in (2); + +create unique index p0_b on p0(b); +create unique index p1_b on p1(b); + +Now b is only unique on partition level, so the distinct can't be removed on +the following cases. SELECT DISTINCT b FROM p; + +Another example is SELECT DISTINCT a, b FROM p WHERE a = 1; Since only one +partition is chosen, the UniqueKey on child relation is same as the UniqueKey on +parent relation. + +Another usage of UniqueKey on partition level is it be helpful for +partition-wise join. + +As for the UniqueKey on parent table level, it comes with 2 different ways, +1). the UniqueKey is also derived in UniqueKey index, but the index must be same +in all the related children relations and the unique index must contains +Partition Key in it. Example: + +CREATE UNIQUE INDEX p_ab ON p(a, b); -- where a is the partition key. + +-- Query +SELECT a, b FROM p; the (a, b) is a UniqueKey of p. + +2). If the parent relation has only one childrel, the UniqueKey on childrel is + the UniqueKey on parent as well. diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index 37b4223adb..d52ad59f89 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -579,6 +579,12 @@ set_plain_rel_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) */ check_index_predicates(root, rel); + /* + * Now that we've marked which partial indexes are suitable, we can now + * build the relation's unique keys. + */ + populate_baserel_uniquekeys(root, rel, rel->indexlist); + /* Mark rel with estimated output rows, width, etc */ set_baserel_size_estimates(root, rel); } @@ -1297,6 +1303,8 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, /* Add paths to the append relation. */ add_paths_to_append_rel(root, rel, live_childrels); + if (IS_PARTITIONED_REL(rel)) + populate_partitionedrel_uniquekeys(root, rel, live_childrels); } @@ -2304,6 +2312,8 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, pathkeys, required_outer)); } + convert_subquery_uniquekeys(root, rel, sub_final_rel); + /* If outer rel allows parallelism, do same for partial paths. */ if (rel->consider_parallel && bms_is_empty(required_outer)) { diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c index 57ce97fd53..697ac047fb 100644 --- a/src/backend/optimizer/path/joinpath.c +++ b/src/backend/optimizer/path/joinpath.c @@ -71,13 +71,6 @@ static void consider_parallel_mergejoin(PlannerInfo *root, static void hash_inner_and_outer(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, RelOptInfo *innerrel, JoinType jointype, JoinPathExtraData *extra); -static List *select_mergejoin_clauses(PlannerInfo *root, - RelOptInfo *joinrel, - RelOptInfo *outerrel, - RelOptInfo *innerrel, - List *restrictlist, - JoinType jointype, - bool *mergejoin_allowed); static void generate_mergejoin_paths(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *innerrel, @@ -1927,7 +1920,7 @@ hash_inner_and_outer(PlannerInfo *root, * if it is mergejoinable and involves vars from the two sub-relations * currently of interest. */ -static List * +List * select_mergejoin_clauses(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c index 0dbe2ac726..7271f044ec 100644 --- a/src/backend/optimizer/path/joinrels.c +++ b/src/backend/optimizer/path/joinrels.c @@ -924,6 +924,8 @@ populate_joinrel_with_paths(PlannerInfo *root, RelOptInfo *rel1, /* Apply partitionwise join technique, if possible. */ try_partitionwise_join(root, rel1, rel2, joinrel, sjinfo, restrictlist); + + populate_joinrel_uniquekeys(root, joinrel, rel1, rel2, restrictlist, sjinfo->jointype); } diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index bd9a176d7d..139278829b 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -33,7 +33,6 @@ static bool pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys); static bool matches_boolean_partition_clause(RestrictInfo *rinfo, RelOptInfo *partrel, int partkeycol); -static Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); static bool right_merge_direction(PlannerInfo *root, PathKey *pathkey); @@ -1035,7 +1034,7 @@ convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel, * We need this to ensure that we don't return pathkeys describing values * that are unavailable above the level of the subquery scan. */ -static Var * +Var * find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle) { ListCell *lc; diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c new file mode 100644 index 0000000000..77ed2b2eff --- /dev/null +++ b/src/backend/optimizer/path/uniquekeys.c @@ -0,0 +1,1135 @@ +/*------------------------------------------------------------------------- + * + * uniquekeys.c + * Utilities for matching and building unique keys + * + * Portions Copyright (c) 2020, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/optimizer/path/uniquekeys.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "nodes/makefuncs.h" +#include "nodes/nodeFuncs.h" +#include "optimizer/pathnode.h" +#include "optimizer/paths.h" +#include "optimizer/appendinfo.h" +#include "optimizer/optimizer.h" +#include "optimizer/tlist.h" +#include "rewrite/rewriteManip.h" + + +/* + * This struct is used to help populate_joinrel_uniquekeys. + * + * added_to_joinrel is true if a uniquekey (from outerrel or innerrel) + * has been added to joinrel. + * useful is true if the exprs of the uniquekey still appears in joinrel. + */ +typedef struct UniqueKeyContextData +{ + UniqueKey *uniquekey; + bool added_to_joinrel; + bool useful; +} *UniqueKeyContext; + +static List *initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel); +static bool innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + bool reverse); + +static List *get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals); +static List *get_exprs_from_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + UniqueKey *ukey); +static void add_uniquekey_for_onerow(RelOptInfo *rel); +static bool add_combined_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype); + +/* Used for unique indexes checking for partitioned table */ +static bool index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind); +static IndexOptInfo *simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from); +static bool simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2); +static void adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_index); + +/* Helper function for grouped relation and distinct relation. */ +static void add_uniquekey_from_sortgroups(PlannerInfo *root, + RelOptInfo *rel, + List *sortgroups); + +/* + * populate_baserel_uniquekeys + * Populate 'baserel' uniquekeys list by looking at the rel's unique index + * and baserestrictinfo + */ +void +populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List *indexlist) +{ + ListCell *lc; + List *matched_uniq_indexes = NIL; + + /* Attrs appears in rel->reltarget->exprs. */ + Bitmapset *used_attrs = NULL; + + List *const_exprs = NIL; + List *expr_opfamilies = NIL; + + Assert(baserel->rtekind == RTE_RELATION); + + foreach(lc, indexlist) + { + IndexOptInfo *ind = (IndexOptInfo *) lfirst(lc); + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + matched_uniq_indexes = lappend(matched_uniq_indexes, ind); + } + + if (matched_uniq_indexes == NIL) + return; + + /* Check which attrs is used in baserel->reltarget */ + pull_varattnos((Node *)baserel->reltarget->exprs, baserel->relid, &used_attrs); + + /* Check which attrno is used at a mergeable const filter */ + foreach(lc, baserel->baserestrictinfo) + { + RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); + + if (rinfo->mergeopfamilies == NIL) + continue; + + if (bms_is_empty(rinfo->left_relids)) + { + const_exprs = lappend(const_exprs, get_rightop(rinfo->clause)); + } + else if (bms_is_empty(rinfo->right_relids)) + { + const_exprs = lappend(const_exprs, get_leftop(rinfo->clause)); + } + else + continue; + + expr_opfamilies = lappend(expr_opfamilies, rinfo->mergeopfamilies); + } + + foreach(lc, matched_uniq_indexes) + { + bool multi_nullvals, useful; + List *exprs = get_exprs_from_uniqueindex(lfirst_node(IndexOptInfo, lc), + const_exprs, + expr_opfamilies, + used_attrs, + &useful, + &multi_nullvals); + if (useful) + { + if (exprs == NIL) + { + /* All the columns in Unique Index matched with a restrictinfo */ + add_uniquekey_for_onerow(baserel); + return; + } + baserel->uniquekeys = lappend(baserel->uniquekeys, + makeUniqueKey(exprs, multi_nullvals)); + } + } +} + + +/* + * populate_partitionedrel_uniquekeys + * The UniqueKey on partitionrel comes from 2 cases: + * 1). Only one partition is involved in this query, the unique key can be + * copied to parent rel from childrel. + * 2). There are some unique index which includes partition key and exists + * in all the related partitions. + * We never mind rule 2 if we hit rule 1. + */ + +void +populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels) +{ + ListCell *lc; + List *global_uniq_indexlist = NIL; + RelOptInfo *childrel; + bool is_first = true; + + Assert(IS_PARTITIONED_REL(rel)); + + if (childrels == NIL) + return; + + /* + * If there is only one partition used in this query, the UniqueKey in childrel is + * still valid in parent level, but we need convert the format from child expr to + * parent expr. + */ + if (list_length(childrels) == 1) + { + /* Check for Rule 1 */ + RelOptInfo *childrel = linitial_node(RelOptInfo, childrels); + ListCell *lc; + Assert(childrel->reloptkind == RELOPT_OTHER_MEMBER_REL); + if (relation_is_onerow(childrel)) + { + add_uniquekey_for_onerow(rel); + return; + } + + foreach(lc, childrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + AppendRelInfo *appinfo = find_appinfo_by_child(root, childrel->relid); + List *parent_exprs = NIL; + bool can_reuse = true; + ListCell *lc2; + foreach(lc2, ukey->exprs) + { + Var *var = (Var *)lfirst(lc2); + /* + * If the expr comes from a expression, it is hard to build the expression + * in parent so ignore that case for now. + */ + if(!IsA(var, Var)) + { + can_reuse = false; + break; + } + /* Convert it to parent var */ + parent_exprs = lappend(parent_exprs, find_parent_var(appinfo, var)); + } + if (can_reuse) + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(parent_exprs, + ukey->multi_nullvals)); + } + } + else + { + /* Check for rule 2 */ + childrel = linitial_node(RelOptInfo, childrels); + foreach(lc, childrel->indexlist) + { + IndexOptInfo *ind = lfirst(lc); + IndexOptInfo *modified_index; + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + + /* + * During simple_copy_indexinfo_to_parent, we need to convert var from + * child var to parent var, index on expression is too complex to handle. + * so ignore it for now. + */ + if (ind->indexprs != NIL) + continue; + + modified_index = simple_copy_indexinfo_to_parent(root, rel, ind); + /* + * If the unique index doesn't contain partkey, then it is unique + * on this partition only, so it is useless for us. + */ + if (!index_constains_partkey(rel, modified_index)) + continue; + + global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index); + } + + if (global_uniq_indexlist != NIL) + { + foreach(lc, childrels) + { + RelOptInfo *child = lfirst(lc); + if (is_first) + { + is_first = false; + continue; + } + adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist); + } + /* Now we have a list of unique index which are exactly same on all childrels, + * Set the UniqueKey just like it is non-partition table + */ + populate_baserel_uniquekeys(root, rel, global_uniq_indexlist); + } + } +} + + +/* + * populate_distinctrel_uniquekeys + */ +void +populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel) +{ + /* The unique key before the distinct is still valid. */ + distinctrel->uniquekeys = list_copy(inputrel->uniquekeys); + add_uniquekey_from_sortgroups(root, distinctrel, root->parse->distinctClause); +} + +/* + * populate_grouprel_uniquekeys + */ +void +populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel) + +{ + Query *parse = root->parse; + bool input_ukey_added = false; + ListCell *lc; + + if (relation_is_onerow(inputrel)) + { + add_uniquekey_for_onerow(grouprel); + return; + } + if (parse->groupingSets) + return; + + /* A Normal group by without grouping set. */ + if (parse->groupClause) + { + /* + * Current even the groupby clause is Unique already, but if query has aggref + * We have to create grouprel still. To keep the UnqiueKey short, we will check + * the UniqueKey of input_rel still valid, if so we reuse it. + */ + foreach(lc, inputrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (list_is_subset(ukey->exprs, grouprel->reltarget->exprs)) + { + grouprel->uniquekeys = lappend(grouprel->uniquekeys, + ukey); + input_ukey_added = true; + } + } + if (!input_ukey_added) + /* + * group by clause must be a super-set of grouprel->reltarget->exprs except the + * aggregation expr, so if such exprs is unique already, no bother to generate + * new uniquekey for group by exprs. + */ + add_uniquekey_from_sortgroups(root, + grouprel, + root->parse->groupClause); + } + else + /* It has aggregation but without a group by, so only one row returned */ + add_uniquekey_for_onerow(grouprel); +} + +/* + * simple_copy_uniquekeys + * Using a function for the one-line code makes us easy to check where we simply + * copied the uniquekey. + */ +void +simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel) +{ + newrel->uniquekeys = oldrel->uniquekeys; +} + +/* + * populate_unionrel_uniquekeys + */ +void +populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel) +{ + ListCell *lc; + List *exprs = NIL; + + Assert(unionrel->uniquekeys == NIL); + + foreach(lc, unionrel->reltarget->exprs) + { + exprs = lappend(exprs, lfirst(lc)); + } + + if (exprs == NIL) + /* SQL: select union select; is valid, we need to handle it here. */ + add_uniquekey_for_onerow(unionrel); + else + unionrel->uniquekeys = lappend(unionrel->uniquekeys, + makeUniqueKey(exprs,false)); + +} + +/* + * populate_joinrel_uniquekeys + * + * populate uniquekeys for joinrel. We will check each relation to see if its + * UniqueKey is still valid via innerrel_keeps_unique, if so, we add it to + * joinrel. The multi_nullvals field will be changed to true for some outer + * join cases and one-row UniqueKey needs to be converted to normal UniqueKey + * for the same case as well. + * For the uniquekey in either baserel which can't be unique after join, we still + * check to see if combination of UniqueKeys from both side is still useful for us. + * if yes, we add it to joinrel as well. + */ +void +populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *outerrel, RelOptInfo *innerrel, + List *restrictlist, JoinType jointype) +{ + ListCell *lc, *lc2; + List *clause_list = NIL; + List *outerrel_ukey_ctx; + List *innerrel_ukey_ctx; + bool inner_onerow, outer_onerow; + bool mergejoin_allowed; + + /* Care about the outerrel relation only for SEMI/ANTI join */ + if (jointype == JOIN_SEMI || jointype == JOIN_ANTI) + { + foreach(lc, outerrel->uniquekeys) + { + UniqueKey *uniquekey = lfirst_node(UniqueKey, lc); + if (list_is_subset(uniquekey->exprs, joinrel->reltarget->exprs)) + joinrel->uniquekeys = lappend(joinrel->uniquekeys, uniquekey); + } + return; + } + + Assert(jointype == JOIN_LEFT || jointype == JOIN_FULL || jointype == JOIN_INNER); + + /* Fast path */ + if (innerrel->uniquekeys == NIL || outerrel->uniquekeys == NIL) + return; + + inner_onerow = relation_is_onerow(innerrel); + outer_onerow = relation_is_onerow(outerrel); + + outerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(outerrel); + innerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(innerrel); + + clause_list = select_mergejoin_clauses(root, joinrel, outerrel, innerrel, + restrictlist, jointype, + &mergejoin_allowed); + + if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true /* reverse */)) + { + bool outer_impact = jointype == JOIN_FULL; + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + /* Outer relation has one row, and the unique key is not duplicated after join, + * the joinrel will still has one row unless the jointype == JOIN_FULL. + */ + if (outer_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (outer_onerow) + { + /* + * The onerow outerrel becomes multi rows and multi_nullvals + * will be changed to true. We also need to set the exprs correctly since it + * can't be NIL any more. + */ + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, outerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Change multi_nullvals to true due to the full join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, true)); + else + /* Just reuse it */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + } + ctx->added_to_joinrel = true; + } + } + + if (innerrel_keeps_unique(root, outerrel, innerrel, clause_list, false)) + { + bool outer_impact = jointype == JOIN_FULL || jointype == JOIN_LEFT;; + + foreach(lc, innerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + if (inner_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (inner_onerow) + { + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, innerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Need to change multi_nullvals to true due to the outer join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, + true)); + else + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + + } + ctx->added_to_joinrel = true; + } + } + + /* + * The combination of the UniqueKey from both sides is unique as well regardless + * of join type, but no bother to add it if its subset has been added to joinrel + * already or it is not useful for the joinrel. + */ + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx1 = (UniqueKeyContext) lfirst(lc); + if (ctx1->added_to_joinrel || !ctx1->useful) + continue; + foreach(lc2, innerrel_ukey_ctx) + { + UniqueKeyContext ctx2 = (UniqueKeyContext) lfirst(lc2); + if (ctx2->added_to_joinrel || !ctx2->useful) + continue; + if (add_combined_uniquekey(root, joinrel, outerrel, innerrel, + ctx1->uniquekey, ctx2->uniquekey, + jointype)) + /* If we set a onerow UniqueKey to joinrel, we don't need other. */ + return; + } + } +} + + +/* + * convert_subquery_uniquekeys + * + * Covert the UniqueKey in subquery to outer relation. + */ +void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel) +{ + ListCell *lc; + + if (sub_final_rel->uniquekeys == NIL) + return; + + if (relation_is_onerow(sub_final_rel)) + { + add_uniquekey_for_onerow(currel); + return; + } + + Assert(currel->subroot != NULL); + + foreach(lc, sub_final_rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + ListCell *lc; + List *exprs = NIL; + bool ukey_useful = true; + + /* One row case is handled above */ + Assert(ukey->exprs != NIL); + foreach(lc, ukey->exprs) + { + Var *var; + TargetEntry *tle = tlist_member(lfirst(lc), + currel->subroot->processed_tlist); + if (tle == NULL) + { + ukey_useful = false; + break; + } + var = find_var_for_subquery_tle(currel, tle); + if (var == NULL) + { + ukey_useful = false; + break; + } + exprs = lappend(exprs, var); + } + + if (ukey_useful) + currel->uniquekeys = lappend(currel->uniquekeys, + makeUniqueKey(exprs, + ukey->multi_nullvals)); + + } +} + +/* + * innerrel_keeps_unique + * + * Check if Unique key of the innerrel is valid after join. innerrel's UniqueKey + * will be still valid if innerrel's any-column mergeop outrerel's uniquekey + * exists in clause_list. + * + * Note: the clause_list must be a list of mergeable restrictinfo already. + */ +static bool +innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *clause_list, + bool reverse) +{ + ListCell *lc, *lc2, *lc3; + + if (outerrel->uniquekeys == NIL || innerrel->uniquekeys == NIL) + return false; + + /* Check if there is outerrel's uniquekey in mergeable clause. */ + foreach(lc, outerrel->uniquekeys) + { + List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs; + bool clauselist_matchs_all_exprs = true; + foreach(lc2, outer_uq_exprs) + { + Node *outer_uq_expr = lfirst(lc2); + bool find_uq_expr_in_clauselist = false; + foreach(lc3, clause_list) + { + RestrictInfo *rinfo = lfirst_node(RestrictInfo, lc3); + Node *outer_expr; + if (reverse) + outer_expr = rinfo->outer_is_left ? get_rightop(rinfo->clause) : get_leftop(rinfo->clause); + else + outer_expr = rinfo->outer_is_left ? get_leftop(rinfo->clause) : get_rightop(rinfo->clause); + if (equal(outer_expr, outer_uq_expr)) + { + find_uq_expr_in_clauselist = true; + break; + } + } + if (!find_uq_expr_in_clauselist) + { + /* No need to check the next exprs in the current uniquekey */ + clauselist_matchs_all_exprs = false; + break; + } + } + + if (clauselist_matchs_all_exprs) + return true; + } + return false; +} + + +/* + * relation_is_onerow + * Check if it is a one-row relation by checking UniqueKey. + */ +bool +relation_is_onerow(RelOptInfo *rel) +{ + UniqueKey *ukey; + if (rel->uniquekeys == NIL) + return false; + ukey = linitial_node(UniqueKey, rel->uniquekeys); + return ukey->exprs == NIL && list_length(rel->uniquekeys) == 1; +} + +/* + * relation_has_uniquekeys_for + * Returns true if we have proofs that 'rel' cannot return multiple rows with + * the same values in each of 'exprs'. Otherwise returns false. + */ +bool +relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel, + List *exprs, bool allow_multinulls) +{ + ListCell *lc; + + /* + * For UniqueKey->onerow case, the uniquekey->exprs is empty as well + * so we can't rely on list_is_subset to handle this special cases + */ + if (exprs == NIL) + return false; + + foreach(lc, rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (ukey->multi_nullvals && !allow_multinulls) + continue; + if (list_is_subset(ukey->exprs, exprs)) + return true; + } + return false; +} + + +/* + * get_exprs_from_uniqueindex + * + * Return a list of exprs which is unique. set useful to false if this + * unique index is not useful for us. + */ +static List * +get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals) +{ + List *exprs = NIL; + ListCell *indexpr_item; + int c = 0; + + *useful = true; + *multi_nullvals = false; + + indexpr_item = list_head(unique_index->indexprs); + for(c = 0; c < unique_index->ncolumns; c++) + { + int attr = unique_index->indexkeys[c]; + Expr *expr; + bool matched_const = false; + ListCell *lc1, *lc2; + + if(attr > 0) + { + expr = list_nth_node(TargetEntry, unique_index->indextlist, c)->expr; + } + else if (attr == 0) + { + /* Expression index */ + expr = lfirst(indexpr_item); + indexpr_item = lnext(unique_index->indexprs, indexpr_item); + } + else /* attr < 0 */ + { + /* Index on system column is not supported */ + Assert(false); + } + + /* + * Check index_col = Const case with regarding to opfamily checking + * If we can remove the index_col from the final UniqueKey->exprs. + */ + forboth(lc1, const_exprs, lc2, const_expr_opfamilies) + { + if (list_member_oid((List *)lfirst(lc2), unique_index->opfamily[c]) + && match_index_to_operand((Node *) lfirst(lc1), c, unique_index)) + { + matched_const = true; + break; + } + } + + if (matched_const) + continue; + + /* Check if the indexed expr is used in rel */ + if (attr > 0) + { + /* + * Normal Indexed column, if the col is not used, then the index is useless + * for uniquekey. + */ + attr -= FirstLowInvalidHeapAttributeNumber; + + if (!bms_is_member(attr, used_varattrs)) + { + *useful = false; + break; + } + } + else if (!list_member(unique_index->rel->reltarget->exprs, expr)) + { + /* Expression index but the expression is not used in rel */ + *useful = false; + break; + } + + /* check not null property. */ + if (attr == 0) + { + /* We never know if a expression yields null or not */ + *multi_nullvals = true; + } + else if (!bms_is_member(attr, unique_index->rel->notnullattrs) + && !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber, + unique_index->rel->notnullattrs)) + { + *multi_nullvals = true; + } + + exprs = lappend(exprs, expr); + } + return exprs; +} + + +/* + * add_uniquekey_for_onerow + * If we are sure that the relation only returns one row, then all the columns + * are unique. However we don't need to create UniqueKey for every column, we + * just set exprs = NIL and overwrites all the other UniqueKey on this RelOptInfo + * since this one has strongest semantics. + */ +void +add_uniquekey_for_onerow(RelOptInfo *rel) +{ + /* + * We overwrite the previous UniqueKey on purpose since this one has the + * strongest semantic. + */ + rel->uniquekeys = list_make1(makeUniqueKey(NIL, false)); +} + + +/* + * initililze_uniquecontext_for_joinrel + * Return a List of UniqueKeyContext for an inputrel + */ +static List * +initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel) +{ + List *res = NIL; + ListCell *lc; + foreach(lc, inputrel->uniquekeys) + { + UniqueKeyContext context; + context = palloc(sizeof(struct UniqueKeyContextData)); + context->uniquekey = lfirst_node(UniqueKey, lc); + context->added_to_joinrel = false; + context->useful = true; + res = lappend(res, context); + } + return res; +} + + +/* + * get_exprs_from_uniquekey + * Unify the way of get List of exprs from a one-row UniqueKey or + * normal UniqueKey. for the onerow case, every expr in rel1 is a valid + * UniqueKey. Return a List of exprs. + * + * rel1: The relation which you want to get the exprs. + * ukey: The UniqueKey you want to get the exprs. + */ +static List * +get_exprs_from_uniquekey(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *rel1, UniqueKey *ukey) +{ + ListCell *lc; + bool onerow = rel1 != NULL && relation_is_onerow(rel1); + + List *res = NIL; + Assert(onerow || ukey); + if (onerow) + { + /* Only cares about the exprs still exist in joinrel */ + foreach(lc, joinrel->reltarget->exprs) + { + Bitmapset *relids = pull_varnos(root, lfirst(lc)); + if (bms_is_subset(relids, rel1->relids)) + { + res = lappend(res, list_make1(lfirst(lc))); + } + } + } + else + { + res = list_make1(ukey->exprs); + } + return res; +} + +/* + * Partitioned table Unique Keys. + * The partition table unique key is maintained as: + * 1. The index must be unique as usual. + * 2. The index must contains partition key. + * 3. The index must exist on all the child rel. see simple_indexinfo_equal for + * how we compare it. + */ + +/* + * index_constains_partkey + * return true if the index contains the partiton key. + */ +static bool +index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind) +{ + ListCell *lc; + int i; + Assert(IS_PARTITIONED_REL(partrel)); + Assert(partrel->part_scheme->partnatts > 0); + + for(i = 0; i < partrel->part_scheme->partnatts; i++) + { + Node *part_expr = linitial(partrel->partexprs[i]); + bool found_in_index = false; + foreach(lc, ind->indextlist) + { + Expr *index_expr = lfirst_node(TargetEntry, lc)->expr; + if (equal(index_expr, part_expr)) + { + found_in_index = true; + break; + } + } + if (!found_in_index) + return false; + } + return true; +} + +/* + * simple_indexinfo_equal + * + * Used to check if the 2 index is same as each other. The index here + * is COPIED from childrel and did some tiny changes(see + * simple_copy_indexinfo_to_parent) + */ +static bool +simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2) +{ + Size oid_cmp_len = sizeof(Oid) * ind1->ncolumns; + + return ind1->ncolumns == ind2->ncolumns && + ind1->unique == ind2->unique && + memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0 && + memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0 && + memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0 && + memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0 && + equal(get_tlist_exprs(ind1->indextlist, true), + get_tlist_exprs(ind2->indextlist, true)); +} + + +/* + * The below macros are used for simple_copy_indexinfo_to_parent which is so + * customized that I don't want to put it to copyfuncs.c. So copy it here. + */ +#define COPY_POINTER_FIELD(fldname, sz) \ + do { \ + Size _size = (sz); \ + newnode->fldname = palloc(_size); \ + memcpy(newnode->fldname, from->fldname, _size); \ + } while (0) + +#define COPY_NODE_FIELD(fldname) \ + (newnode->fldname = copyObjectImpl(from->fldname)) + +#define COPY_SCALAR_FIELD(fldname) \ + (newnode->fldname = from->fldname) + + +/* + * simple_copy_indexinfo_to_parent (from partition) + * Copy the IndexInfo from child relation to parent relation with some modification, + * which is used to test: + * 1. If the same index exists in all the childrels. + * 2. If the parentrel->reltarget/basicrestrict info matches this index. + */ +static IndexOptInfo * +simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from) +{ + IndexOptInfo *newnode = makeNode(IndexOptInfo); + AppendRelInfo *appinfo = find_appinfo_by_child(root, from->rel->relid); + ListCell *lc; + int idx = 0; + + COPY_SCALAR_FIELD(ncolumns); + COPY_SCALAR_FIELD(nkeycolumns); + COPY_SCALAR_FIELD(unique); + COPY_SCALAR_FIELD(immediate); + /* We just need to know if it is NIL or not */ + COPY_SCALAR_FIELD(indpred); + COPY_SCALAR_FIELD(predOK); + COPY_POINTER_FIELD(indexkeys, from->ncolumns * sizeof(int)); + COPY_POINTER_FIELD(indexcollations, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opfamily, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opcintype, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(sortopfamily, from->ncolumns * sizeof(Oid)); + COPY_NODE_FIELD(indextlist); + + /* Convert index exprs on child expr to expr on parent */ + foreach(lc, newnode->indextlist) + { + TargetEntry *tle = lfirst_node(TargetEntry, lc); + /* Index on expression is ignored */ + Assert(IsA(tle->expr, Var)); + tle->expr = (Expr *) find_parent_var(appinfo, (Var *) tle->expr); + newnode->indexkeys[idx] = castNode(Var, tle->expr)->varattno; + idx++; + } + newnode->rel = parentrel; + return newnode; +} + +/* + * adjust_partition_unique_indexlist + * + * global_unique_indexes: At the beginning, it contains the copy & modified + * unique index from the first partition. And then check if each index in it still + * exists in the following partitions. If no, remove it. at last, it has an + * index list which exists in all the partitions. + */ +static void +adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_indexes) +{ + ListCell *lc, *lc2; + foreach(lc, *global_unique_indexes) + { + IndexOptInfo *g_ind = lfirst_node(IndexOptInfo, lc); + bool found_in_child = false; + + foreach(lc2, childrel->indexlist) + { + IndexOptInfo *p_ind = lfirst_node(IndexOptInfo, lc2); + IndexOptInfo *p_ind_copy; + if (!p_ind->unique || !p_ind->immediate || + (p_ind->indpred != NIL && !p_ind->predOK)) + continue; + p_ind_copy = simple_copy_indexinfo_to_parent(root, parentrel, p_ind); + if (simple_indexinfo_equal(p_ind_copy, g_ind)) + { + found_in_child = true; + break; + } + } + if (!found_in_child) + /* The index doesn't exist in childrel, remove it from global_unique_indexes */ + *global_unique_indexes = foreach_delete_current(*global_unique_indexes, lc); + } +} + +/* Helper function for groupres/distinctrel */ +static void +add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgroups) +{ + Query *parse = root->parse; + List *exprs; + + /* + * XXX: If there are some vars which is not in current levelsup, the semantic is + * imprecise, should we avoid it or not? levelsup = 1 is just a demo, maybe we need to + * check every level other than 0, if so, looks we have to write another + * pull_var_walker. + */ + List *upper_vars = pull_vars_of_level((Node*)sortgroups, 1); + + if (upper_vars != NIL) + return; + + exprs = get_sortgrouplist_exprs(sortgroups, parse->targetList); + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(exprs, + false /* sortgroupclause can't be multi_nullvals */)); +} + + +/* + * add_combined_uniquekey + * The combination of both UniqueKeys is a valid UniqueKey for joinrel no matter + * the jointype. + */ +bool +add_combined_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype) +{ + + ListCell *lc1, *lc2; + + /* Either side has multi_nullvals or we have outer join, + * the combined UniqueKey has multi_nullvals */ + bool multi_nullvals = outer_ukey->multi_nullvals || + inner_ukey->multi_nullvals || IS_OUTER_JOIN(jointype); + + /* The only case we can get onerow joinrel after join */ + if (relation_is_onerow(outer_rel) + && relation_is_onerow(inner_rel) + && jointype == JOIN_INNER) + { + add_uniquekey_for_onerow(joinrel); + return true; + } + + foreach(lc1, get_exprs_from_uniquekey(root, joinrel, outer_rel, outer_ukey)) + { + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, inner_rel, inner_ukey)) + { + List *exprs = list_concat_copy(lfirst_node(List, lc1), lfirst_node(List, lc2)); + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(exprs, + multi_nullvals)); + } + } + return false; +} diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index 424d25cbd5..8d8e493f5c 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -2387,6 +2387,8 @@ grouping_planner(PlannerInfo *root, bool inheritance_update, add_path(final_rel, path); } + simple_copy_uniquekeys(current_rel, final_rel); + /* * Generate partial paths for final_rel, too, if outer query levels might * be able to make use of them. @@ -3899,6 +3901,8 @@ create_grouping_paths(PlannerInfo *root, } set_cheapest(grouped_rel); + + populate_grouprel_uniquekeys(root, grouped_rel, input_rel); return grouped_rel; } @@ -4621,7 +4625,7 @@ create_window_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(window_rel); - + simple_copy_uniquekeys(input_rel, window_rel); return window_rel; } @@ -4935,7 +4939,7 @@ create_distinct_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(distinct_rel); - + populate_distinctrel_uniquekeys(root, input_rel, distinct_rel); return distinct_rel; } @@ -5196,6 +5200,8 @@ create_ordered_paths(PlannerInfo *root, */ Assert(ordered_rel->pathlist != NIL); + simple_copy_uniquekeys(input_rel, ordered_rel); + return ordered_rel; } @@ -6073,6 +6079,9 @@ adjust_paths_for_srfs(PlannerInfo *root, RelOptInfo *rel, if (list_length(targets) == 1) return; + /* UniqueKey is not valid after handling the SRF. */ + rel->uniquekeys = NIL; + /* * Stack SRF-evaluation nodes atop each path for the rel. * diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c index becdcbb872..b7626545bf 100644 --- a/src/backend/optimizer/prep/prepunion.c +++ b/src/backend/optimizer/prep/prepunion.c @@ -689,6 +689,8 @@ generate_union_paths(SetOperationStmt *op, PlannerInfo *root, /* Undo effects of possibly forcing tuple_fraction to 0 */ root->tuple_fraction = save_fraction; + /* Add the UniqueKeys */ + populate_unionrel_uniquekeys(root, result_rel); return result_rel; } diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c index 86922a273c..6817c9c787 100644 --- a/src/backend/optimizer/util/appendinfo.c +++ b/src/backend/optimizer/util/appendinfo.c @@ -746,3 +746,47 @@ find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos) } return appinfos; } + +/* + * find_appinfo_by_child + * + */ +AppendRelInfo * +find_appinfo_by_child(PlannerInfo *root, Index child_index) +{ + ListCell *lc; + foreach(lc, root->append_rel_list) + { + AppendRelInfo *appinfo = lfirst_node(AppendRelInfo, lc); + if (appinfo->child_relid == child_index) + return appinfo; + } + elog(ERROR, "parent relation cant be found"); + return NULL; +} + +/* + * find_parent_var + * + */ +Var * +find_parent_var(AppendRelInfo *appinfo, Var *child_var) +{ + ListCell *lc; + Var *res = NULL; + Index attno = 1; + foreach(lc, appinfo->translated_vars) + { + Node *child_node = lfirst(lc); + if (equal(child_node, child_var)) + { + res = copyObject(child_var); + res->varattno = attno; + res->varno = appinfo->parent_relid; + } + attno++; + } + if (res == NULL) + elog(ERROR, "parent var can't be found."); + return res; +} diff --git a/src/backend/optimizer/util/inherit.c b/src/backend/optimizer/util/inherit.c index be1c9ddd96..3eec1f4d74 100644 --- a/src/backend/optimizer/util/inherit.c +++ b/src/backend/optimizer/util/inherit.c @@ -736,6 +736,7 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, { Node *onecq = (Node *) lfirst(lc2); bool pseudoconstant; + RestrictInfo *child_rinfo; /* check for pseudoconstant (no Vars or volatile functions) */ pseudoconstant = @@ -747,14 +748,15 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, root->hasPseudoConstantQuals = true; } /* reconstitute RestrictInfo with appropriate properties */ - childquals = lappend(childquals, - make_restrictinfo(root, - (Expr *) onecq, - rinfo->is_pushed_down, - rinfo->outerjoin_delayed, - pseudoconstant, - rinfo->security_level, - NULL, NULL, NULL)); + child_rinfo = make_restrictinfo(root, + (Expr *) onecq, + rinfo->is_pushed_down, + rinfo->outerjoin_delayed, + pseudoconstant, + rinfo->security_level, + NULL, NULL, NULL); + child_rinfo->mergeopfamilies = rinfo->mergeopfamilies; + childquals = lappend(childquals, child_rinfo); /* track minimum security level among child quals */ cq_min_security = Min(cq_min_security, rinfo->security_level); } diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 48a7ebfe45..4fe1824eb0 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -16,6 +16,7 @@ #include "nodes/execnodes.h" #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" extern A_Expr *makeA_Expr(A_Expr_Kind kind, List *name, @@ -106,4 +107,6 @@ extern GroupingSet *makeGroupingSet(GroupingSetKind kind, List *content, int loc extern VacuumRelation *makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols); +extern UniqueKey* makeUniqueKey(List *exprs, bool multi_nullvals); + #endif /* MAKEFUNC_H */ diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index e22df890ef..3320273ac1 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -265,6 +265,7 @@ typedef enum NodeTag T_EquivalenceMember, T_PathKey, T_PathTarget, + T_UniqueKey, T_RestrictInfo, T_IndexClause, T_PlaceHolderVar, diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index 0d61f04d27..80561b1003 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -739,6 +739,7 @@ typedef struct RelOptInfo QualCost baserestrictcost; /* cost of evaluating the above */ Index baserestrict_min_security; /* min security_level found in * baserestrictinfo */ + List *uniquekeys; /* List of UniqueKey */ List *joininfo; /* RestrictInfo structures for join clauses * involving this rel */ bool has_eclass_joins; /* T means joininfo is incomplete */ @@ -1059,6 +1060,28 @@ typedef struct PathKey } PathKey; +/* + * UniqueKey + * + * Represents the unique properties held by a RelOptInfo. + * + * exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL + * is a special case of UniqueKey, which means there is only 1 row in that + * relation. + * multi_nullvals: true means multi null values may exist in these exprs, so the + * uniqueness is not guaranteed in this case. This field is necessary for + * remove_useless_join & reduce_unique_semijoins where we don't mind these + * duplicated NULL values. It is set to true for 2 cases. One is a unique key + * from a unique index but the related column is nullable. The other one is for + * outer join. see populate_joinrel_uniquekeys for detail. + */ +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + /* * PathTarget * @@ -2496,7 +2519,7 @@ typedef enum * * flags indicating what kinds of grouping are possible. * partial_costs_set is true if the agg_partial_costs and agg_final_costs - * have been initialized. + * have been initialized. * agg_partial_costs gives partial aggregation costs. * agg_final_costs gives finalization costs. * target_parallel_safe is true if target is parallel safe. @@ -2526,8 +2549,8 @@ typedef struct * limit_tuples is an estimated bound on the number of output tuples, * or -1 if no LIMIT or couldn't estimate. * count_est and offset_est are the estimated values of the LIMIT and OFFSET - * expressions computed by preprocess_limit() (see comments for - * preprocess_limit() for more information). + * expressions computed by preprocess_limit() (see comments for + * preprocess_limit() for more information). */ typedef struct { diff --git a/src/include/nodes/pg_list.h b/src/include/nodes/pg_list.h index 404e03f132..85c65e910c 100644 --- a/src/include/nodes/pg_list.h +++ b/src/include/nodes/pg_list.h @@ -566,6 +566,16 @@ extern pg_nodiscard List *list_delete_first(List *list); extern pg_nodiscard List *list_delete_last(List *list); extern pg_nodiscard List *list_delete_nth_cell(List *list, int n); extern pg_nodiscard List *list_delete_cell(List *list, ListCell *cell); +extern bool list_is_subset(const List *members, const List *target); + +extern List *list_delete(List *list, void *datum); +extern List *list_delete_ptr(List *list, void *datum); +extern List *list_delete_int(List *list, int datum); +extern List *list_delete_oid(List *list, Oid datum); +extern List *list_delete_first(List *list); +extern List *list_delete_last(List *list); +extern List *list_delete_nth_cell(List *list, int n); +extern List *list_delete_cell(List *list, ListCell *cell); extern List *list_union(const List *list1, const List *list2); extern List *list_union_ptr(const List *list1, const List *list2); diff --git a/src/include/optimizer/appendinfo.h b/src/include/optimizer/appendinfo.h index 4cbf8c26cc..43b8d05f69 100644 --- a/src/include/optimizer/appendinfo.h +++ b/src/include/optimizer/appendinfo.h @@ -32,4 +32,7 @@ extern Relids adjust_child_relids_multilevel(PlannerInfo *root, Relids relids, extern AppendRelInfo **find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos); +extern AppendRelInfo *find_appinfo_by_child(PlannerInfo *root, Index child_index); +extern Var *find_parent_var(AppendRelInfo *appinfo, Var *child_var); + #endif /* APPENDINFO_H */ diff --git a/src/include/optimizer/optimizer.h b/src/include/optimizer/optimizer.h index d587952b7d..843aafa51e 100644 --- a/src/include/optimizer/optimizer.h +++ b/src/include/optimizer/optimizer.h @@ -23,6 +23,7 @@ #define OPTIMIZER_H #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" /* Test if an expression node represents a SRF call. Beware multiple eval! */ #define IS_SRF_CALL(node) \ @@ -168,6 +169,7 @@ extern TargetEntry *get_sortgroupref_tle(Index sortref, List *targetList); extern TargetEntry *get_sortgroupclause_tle(SortGroupClause *sgClause, List *targetList); +extern Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); extern Node *get_sortgroupclause_expr(SortGroupClause *sgClause, List *targetList); extern List *get_sortgrouplist_exprs(List *sgClauses, diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h index 035d3e1206..1adf99c9ee 100644 --- a/src/include/optimizer/paths.h +++ b/src/include/optimizer/paths.h @@ -247,5 +247,48 @@ extern PathKey *make_canonical_pathkey(PlannerInfo *root, int strategy, bool nulls_first); extern void add_paths_to_append_rel(PlannerInfo *root, RelOptInfo *rel, List *live_childrels); +extern List *select_mergejoin_clauses(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + JoinType jointype, + bool *mergejoin_allowed); + +/* + * uniquekeys.c + * Utilities for matching and building unique keys + */ +extern void populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List* unique_index_list); +extern void populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels); +extern void populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel); +extern void populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel); +extern void populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel); +extern void simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel); +extern void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel); +extern void populate_joinrel_uniquekeys(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + RelOptInfo *rel2, + List *restrictlist, + JoinType jointype); + +extern bool relation_has_uniquekeys_for(PlannerInfo *root, + RelOptInfo *rel, + List *exprs, + bool allow_multinulls); +extern bool relation_is_onerow(RelOptInfo *rel); #endif /* PATHS_H */ -- 2.26.2 --yh7cw345mbttadjz Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v38-0003-Extend-UniqueKeys.patch" ^ permalink raw reply [nested|flat] 91+ messages in thread
* [PATCH v36 2/6] Introduce UniqueKey attributes on RelOptInfo struct. @ 2020-05-11 07:50 一挃 <[email protected]> 0 siblings, 0 replies; 91+ messages in thread From: 一挃 @ 2020-05-11 07:50 UTC (permalink / raw) UniqueKey is a set of exprs on RelOptInfo which represents the exprs will be unique on the given RelOptInfo. You can see README.uniquekey for more information. --- src/backend/nodes/copyfuncs.c | 13 + src/backend/nodes/list.c | 31 + src/backend/nodes/makefuncs.c | 13 + src/backend/nodes/outfuncs.c | 11 + src/backend/nodes/readfuncs.c | 10 + src/backend/optimizer/path/Makefile | 3 +- src/backend/optimizer/path/README.uniquekey | 131 +++ src/backend/optimizer/path/allpaths.c | 10 + src/backend/optimizer/path/joinpath.c | 9 +- src/backend/optimizer/path/joinrels.c | 2 + src/backend/optimizer/path/pathkeys.c | 3 +- src/backend/optimizer/path/uniquekeys.c | 1131 +++++++++++++++++++ src/backend/optimizer/plan/planner.c | 13 +- src/backend/optimizer/prep/prepunion.c | 2 + src/backend/optimizer/util/appendinfo.c | 44 + src/backend/optimizer/util/inherit.c | 16 +- src/include/nodes/makefuncs.h | 3 + src/include/nodes/nodes.h | 1 + src/include/nodes/pathnodes.h | 29 +- src/include/nodes/pg_list.h | 2 + src/include/optimizer/appendinfo.h | 3 + src/include/optimizer/optimizer.h | 2 + src/include/optimizer/paths.h | 43 + 23 files changed, 1502 insertions(+), 23 deletions(-) create mode 100644 src/backend/optimizer/path/README.uniquekey create mode 100644 src/backend/optimizer/path/uniquekeys.c diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 89c409de66..1f50400fd2 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2273,6 +2273,16 @@ _copyPathKey(const PathKey *from) return newnode; } +static UniqueKey * +_copyUniqueKey(const UniqueKey *from) +{ + UniqueKey *newnode = makeNode(UniqueKey); + + COPY_NODE_FIELD(exprs); + COPY_SCALAR_FIELD(multi_nullvals); + + return newnode; +} /* * _copyRestrictInfo */ @@ -5152,6 +5162,9 @@ copyObjectImpl(const void *from) case T_PathKey: retval = _copyPathKey(from); break; + case T_UniqueKey: + retval = _copyUniqueKey(from); + break; case T_RestrictInfo: retval = _copyRestrictInfo(from); break; diff --git a/src/backend/nodes/list.c b/src/backend/nodes/list.c index 80fa8c84e4..a7a99b70f2 100644 --- a/src/backend/nodes/list.c +++ b/src/backend/nodes/list.c @@ -687,6 +687,37 @@ list_member_oid(const List *list, Oid datum) return false; } +/* + * return true iff every entry in "members" list is also present + * in the "target" list. + */ +bool +list_is_subset(const List *members, const List *target) +{ + const ListCell *lc1, *lc2; + + Assert(IsPointerList(members)); + Assert(IsPointerList(target)); + check_list_invariants(members); + check_list_invariants(target); + + foreach(lc1, members) + { + bool found = false; + foreach(lc2, target) + { + if (equal(lfirst(lc1), lfirst(lc2))) + { + found = true; + break; + } + } + if (!found) + return false; + } + return true; +} + /* * Delete the n'th cell (counting from 0) in list. * diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 49de285f01..646cf7c9a1 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -814,3 +814,16 @@ makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols) v->va_cols = va_cols; return v; } + + +/* + * makeUniqueKey + */ +UniqueKey* +makeUniqueKey(List *exprs, bool multi_nullvals) +{ + UniqueKey * ukey = makeNode(UniqueKey); + ukey->exprs = exprs; + ukey->multi_nullvals = multi_nullvals; + return ukey; +} diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index e2f177515d..c3a9632992 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2428,6 +2428,14 @@ _outPathKey(StringInfo str, const PathKey *node) WRITE_BOOL_FIELD(pk_nulls_first); } +static void +_outUniqueKey(StringInfo str, const UniqueKey *node) +{ + WRITE_NODE_TYPE("UNIQUEKEY"); + WRITE_NODE_FIELD(exprs); + WRITE_BOOL_FIELD(multi_nullvals); +} + static void _outPathTarget(StringInfo str, const PathTarget *node) { @@ -4127,6 +4135,9 @@ outNode(StringInfo str, const void *obj) case T_PathKey: _outPathKey(str, obj); break; + case T_UniqueKey: + _outUniqueKey(str, obj); + break; case T_PathTarget: _outPathTarget(str, obj); break; diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index 42050ab719..3a18571d0c 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -452,6 +452,14 @@ _readSetOperationStmt(void) READ_DONE(); } +static UniqueKey * +_readUniqueKey(void) +{ + READ_LOCALS(UniqueKey); + READ_NODE_FIELD(exprs); + READ_BOOL_FIELD(multi_nullvals); + READ_DONE(); +} /* * Stuff from primnodes.h. @@ -2656,6 +2664,8 @@ parseNodeString(void) return_value = _readCommonTableExpr(); else if (MATCH("SETOPERATIONSTMT", 16)) return_value = _readSetOperationStmt(); + else if (MATCH("UNIQUEKEY", 9)) + return_value = _readUniqueKey(); else if (MATCH("ALIAS", 5)) return_value = _readAlias(); else if (MATCH("RANGEVAR", 8)) diff --git a/src/backend/optimizer/path/Makefile b/src/backend/optimizer/path/Makefile index 1e199ff66f..7b9820c25f 100644 --- a/src/backend/optimizer/path/Makefile +++ b/src/backend/optimizer/path/Makefile @@ -21,6 +21,7 @@ OBJS = \ joinpath.o \ joinrels.o \ pathkeys.o \ - tidpath.o + tidpath.o \ + uniquekeys.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/optimizer/path/README.uniquekey b/src/backend/optimizer/path/README.uniquekey new file mode 100644 index 0000000000..5eac761995 --- /dev/null +++ b/src/backend/optimizer/path/README.uniquekey @@ -0,0 +1,131 @@ +1. What is UniqueKey? +We can think UniqueKey is a set of exprs for a RelOptInfo, which we are insure +that doesn't yields same result among all the rows. The simplest UniqueKey +format is primary key. + +However we define the UnqiueKey as below. + +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + +exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL +is a special case of UniqueKey, which means there is only one row in that +relation.it has a stronger semantic than others. like SELECT uk FROM t; uk is +normal unique key and may have different values. SELECT colx FROM t WHERE uk = +const. colx is unique AND we have only 1 value. This field can used for +innerrel_is_unique. this logic is handled specially in add_uniquekey_for_onerow +function. + +multi_nullvals: true means multi null values may exist in these exprs, so the +uniqueness is not guaranteed in this case. This field is necessary for +remove_useless_join & reduce_unique_semijoins where we don't mind these +duplicated NULL values. It is set to true for 2 cases. One is a unique key +from a unique index but the related column is nullable. The other one is for +outer join. see populate_joinrel_uniquekeys for detail. + + +The UniqueKey can be used at the following cases at least: +1. remove_useless_joins. +2. reduce_semianti_joins +3. remove distinct node if distinct clause is unique. +4. remove aggnode if group by clause is unique. +5. Index Skip Scan (WIP) +6. Aggregation Push Down without 2 phase aggregation if the join can't + duplicated the aggregated rows. (work in progress feature) + +2. How is it maintained? + +We have a set of populate_xxx_unqiuekeys functions to maintain the uniquekey on +various cases. xxx includes baserel, joinrel, partitionedrel, distinctrel, +groupedrel, unionrel. and we also need to convert the uniquekey from subquery +to outer relation, which is what convert_subquery_uniquekeys does. + +1. The first part is about baserel. We handled 3 cases. suppose we have Unique +Index on (a, b). + +1. SELECT a, b FROM t. UniqueKey (a, b) +2. SELECT a FROM t WHERE b = 1; UniqueKey (a) +3. SELECT .. FROM t WHERE a = 1 AND b = 1; UniqueKey (NIL). onerow case, every + column is Unique. + +2. The next part is joinrel, this part is most error-prone, we simplified the rules +like below: +1. If the relation's UniqueKey can't be duplicated after join, then is will be + still valid for the join rel. The function we used here is + innerrel_keeps_unique. The basic idea is innerrel.any_col = outer.uk. + +2. If the UnqiueKey can't keep valid via the rule 1, the combination of the + UniqueKey from both sides are valid for sure. We can prove this as: if the + unique exprs from rel1 is duplicated by rel2, the duplicated rows must + contains different unique exprs from rel2. + +More considerations about onerow: +1. If relation with one row and it can't be duplicated, it is still possible + contains mulit_nullvas after outer join. +2. If the either UniqueKey can be duplicated after join, the can get one row + only when both side is one row AND there is no outer join. +3. Whenever the onerow UniqueKey is not a valid any more, we need to convert one + row UniqueKey to normal unique key since we don't store exprs for one-row + relation. get_exprs_from_uniquekeys will be used here. + + +More considerations about multi_nullvals after join: +1. If the original UnqiueKey has multi_nullvals, the final UniqueKey will have + mulit_nullvals in any case. +2. If a unique key doesn't allow mulit_nullvals, after some outer join, it + allows some outer join. + + +3. When we comes to subquery, we need to convert_subquery_unqiuekeys just like +convert_subquery_pathkeys. Only the UniqueKey insides subquery is referenced as +a Var in outer relation will be reused. The relationship between the outerrel.Var +and subquery.exprs is built with outerel->subroot->processed_tlist. + + +4. As for the SRF functions, it will break the uniqueness of uniquekey, However it +is handled in adjust_paths_for_srfs, which happens after the query_planner. so +we will maintain the UniqueKey until there and reset it to NIL at that +places. This can't help on distinct/group by elimination cases but probably help +in some other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is +semantic correctly. + + +5. As for inherit table, we first main the UnqiueKey on childrel as well. But for +partitioned table we need to maintain 2 different kinds of +UnqiueKey. 1). UniqueKey on the parent relation 2). UniqueKey on child +relation for partition wise query. + +Example: +CREATE TABLE p (a int not null, b int not null) partition by list (a); +CREATE TABLE p0 partition of p for values in (1); +CREATE TABLE p1 partition of p for values in (2); + +create unique index p0_b on p0(b); +create unique index p1_b on p1(b); + +Now b is only unique on partition level, so the distinct can't be removed on +the following cases. SELECT DISTINCT b FROM p; + +Another example is SELECT DISTINCT a, b FROM p WHERE a = 1; Since only one +partition is chosen, the UniqueKey on child relation is same as the UniqueKey on +parent relation. + +Another usage of UniqueKey on partition level is it be helpful for +partition-wise join. + +As for the UniqueKey on parent table level, it comes with 2 different ways, +1). the UniqueKey is also derived in UniqueKey index, but the index must be same +in all the related children relations and the unique index must contains +Partition Key in it. Example: + +CREATE UNIQUE INDEX p_ab ON p(a, b); -- where a is the partition key. + +-- Query +SELECT a, b FROM p; the (a, b) is a UniqueKey of p. + +2). If the parent relation has only one childrel, the UniqueKey on childrel is + the UniqueKey on parent as well. diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index 484dab0a1a..2ad9d06d7a 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -579,6 +579,12 @@ set_plain_rel_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) */ check_index_predicates(root, rel); + /* + * Now that we've marked which partial indexes are suitable, we can now + * build the relation's unique keys. + */ + populate_baserel_uniquekeys(root, rel, rel->indexlist); + /* Mark rel with estimated output rows, width, etc */ set_baserel_size_estimates(root, rel); } @@ -1310,6 +1316,8 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, /* Add paths to the append relation. */ add_paths_to_append_rel(root, rel, live_childrels); + if (IS_PARTITIONED_REL(rel)) + populate_partitionedrel_uniquekeys(root, rel, live_childrels); } @@ -2383,6 +2391,8 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, pathkeys, required_outer)); } + convert_subquery_uniquekeys(root, rel, sub_final_rel); + /* If outer rel allows parallelism, do same for partial paths. */ if (rel->consider_parallel && bms_is_empty(required_outer)) { diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c index db54a6ba2e..ef0fd2fb0b 100644 --- a/src/backend/optimizer/path/joinpath.c +++ b/src/backend/optimizer/path/joinpath.c @@ -71,13 +71,6 @@ static void consider_parallel_mergejoin(PlannerInfo *root, static void hash_inner_and_outer(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, RelOptInfo *innerrel, JoinType jointype, JoinPathExtraData *extra); -static List *select_mergejoin_clauses(PlannerInfo *root, - RelOptInfo *joinrel, - RelOptInfo *outerrel, - RelOptInfo *innerrel, - List *restrictlist, - JoinType jointype, - bool *mergejoin_allowed); static void generate_mergejoin_paths(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *innerrel, @@ -1927,7 +1920,7 @@ hash_inner_and_outer(PlannerInfo *root, * if it is mergejoinable and involves vars from the two sub-relations * currently of interest. */ -static List * +List * select_mergejoin_clauses(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c index 2d343cd293..b9163ee8ff 100644 --- a/src/backend/optimizer/path/joinrels.c +++ b/src/backend/optimizer/path/joinrels.c @@ -924,6 +924,8 @@ populate_joinrel_with_paths(PlannerInfo *root, RelOptInfo *rel1, /* Apply partitionwise join technique, if possible. */ try_partitionwise_join(root, rel1, rel2, joinrel, sjinfo, restrictlist); + + populate_joinrel_uniquekeys(root, joinrel, rel1, rel2, restrictlist, sjinfo->jointype); } diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index ce9bf87e9b..7e596d4194 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -33,7 +33,6 @@ static bool pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys); static bool matches_boolean_partition_clause(RestrictInfo *rinfo, RelOptInfo *partrel, int partkeycol); -static Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); static bool right_merge_direction(PlannerInfo *root, PathKey *pathkey); @@ -1035,7 +1034,7 @@ convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel, * We need this to ensure that we don't return pathkeys describing values * that are unavailable above the level of the subquery scan. */ -static Var * +Var * find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle) { ListCell *lc; diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c new file mode 100644 index 0000000000..b33bcd2f32 --- /dev/null +++ b/src/backend/optimizer/path/uniquekeys.c @@ -0,0 +1,1131 @@ +/*------------------------------------------------------------------------- + * + * uniquekeys.c + * Utilities for matching and building unique keys + * + * Portions Copyright (c) 2020, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/optimizer/path/uniquekeys.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "nodes/makefuncs.h" +#include "nodes/nodeFuncs.h" +#include "optimizer/pathnode.h" +#include "optimizer/paths.h" +#include "optimizer/appendinfo.h" +#include "optimizer/optimizer.h" +#include "optimizer/tlist.h" +#include "rewrite/rewriteManip.h" + + +/* + * This struct is used to help populate_joinrel_uniquekeys. + * + * added_to_joinrel is true if a uniquekey (from outerrel or innerrel) + * has been added to joinrel. + * useful is true if the exprs of the uniquekey still appears in joinrel. + */ +typedef struct UniqueKeyContextData +{ + UniqueKey *uniquekey; + bool added_to_joinrel; + bool useful; +} *UniqueKeyContext; + +static List *initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel); +static bool innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + bool reverse); + +static List *get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals); +static List *get_exprs_from_uniquekey(RelOptInfo *joinrel, + RelOptInfo *rel1, + UniqueKey *ukey); +static void add_uniquekey_for_onerow(RelOptInfo *rel); +static bool add_combined_uniquekey(RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype); + +/* Used for unique indexes checking for partitioned table */ +static bool index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind); +static IndexOptInfo *simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from); +static bool simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2); +static void adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_index); + +/* Helper function for grouped relation and distinct relation. */ +static void add_uniquekey_from_sortgroups(PlannerInfo *root, + RelOptInfo *rel, + List *sortgroups); + +/* + * populate_baserel_uniquekeys + * Populate 'baserel' uniquekeys list by looking at the rel's unique index + * and baserestrictinfo + */ +void +populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List *indexlist) +{ + ListCell *lc; + List *matched_uniq_indexes = NIL; + + /* Attrs appears in rel->reltarget->exprs. */ + Bitmapset *used_attrs = NULL; + + List *const_exprs = NIL; + List *expr_opfamilies = NIL; + + Assert(baserel->rtekind == RTE_RELATION); + + foreach(lc, indexlist) + { + IndexOptInfo *ind = (IndexOptInfo *) lfirst(lc); + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + matched_uniq_indexes = lappend(matched_uniq_indexes, ind); + } + + if (matched_uniq_indexes == NIL) + return; + + /* Check which attrs is used in baserel->reltarget */ + pull_varattnos((Node *)baserel->reltarget->exprs, baserel->relid, &used_attrs); + + /* Check which attrno is used at a mergeable const filter */ + foreach(lc, baserel->baserestrictinfo) + { + RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); + + if (rinfo->mergeopfamilies == NIL) + continue; + + if (bms_is_empty(rinfo->left_relids)) + { + const_exprs = lappend(const_exprs, get_rightop(rinfo->clause)); + } + else if (bms_is_empty(rinfo->right_relids)) + { + const_exprs = lappend(const_exprs, get_leftop(rinfo->clause)); + } + else + continue; + + expr_opfamilies = lappend(expr_opfamilies, rinfo->mergeopfamilies); + } + + foreach(lc, matched_uniq_indexes) + { + bool multi_nullvals, useful; + List *exprs = get_exprs_from_uniqueindex(lfirst_node(IndexOptInfo, lc), + const_exprs, + expr_opfamilies, + used_attrs, + &useful, + &multi_nullvals); + if (useful) + { + if (exprs == NIL) + { + /* All the columns in Unique Index matched with a restrictinfo */ + add_uniquekey_for_onerow(baserel); + return; + } + baserel->uniquekeys = lappend(baserel->uniquekeys, + makeUniqueKey(exprs, multi_nullvals)); + } + } +} + + +/* + * populate_partitionedrel_uniquekeys + * The UniqueKey on partitionrel comes from 2 cases: + * 1). Only one partition is involved in this query, the unique key can be + * copied to parent rel from childrel. + * 2). There are some unique index which includes partition key and exists + * in all the related partitions. + * We never mind rule 2 if we hit rule 1. + */ + +void +populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels) +{ + ListCell *lc; + List *global_uniq_indexlist = NIL; + RelOptInfo *childrel; + bool is_first = true; + + Assert(IS_PARTITIONED_REL(rel)); + + if (childrels == NIL) + return; + + /* + * If there is only one partition used in this query, the UniqueKey in childrel is + * still valid in parent level, but we need convert the format from child expr to + * parent expr. + */ + if (list_length(childrels) == 1) + { + /* Check for Rule 1 */ + RelOptInfo *childrel = linitial_node(RelOptInfo, childrels); + ListCell *lc; + Assert(childrel->reloptkind == RELOPT_OTHER_MEMBER_REL); + if (relation_is_onerow(childrel)) + { + add_uniquekey_for_onerow(rel); + return; + } + + foreach(lc, childrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + AppendRelInfo *appinfo = find_appinfo_by_child(root, childrel->relid); + List *parent_exprs = NIL; + bool can_reuse = true; + ListCell *lc2; + foreach(lc2, ukey->exprs) + { + Var *var = (Var *)lfirst(lc2); + /* + * If the expr comes from a expression, it is hard to build the expression + * in parent so ignore that case for now. + */ + if(!IsA(var, Var)) + { + can_reuse = false; + break; + } + /* Convert it to parent var */ + parent_exprs = lappend(parent_exprs, find_parent_var(appinfo, var)); + } + if (can_reuse) + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(parent_exprs, + ukey->multi_nullvals)); + } + } + else + { + /* Check for rule 2 */ + childrel = linitial_node(RelOptInfo, childrels); + foreach(lc, childrel->indexlist) + { + IndexOptInfo *ind = lfirst(lc); + IndexOptInfo *modified_index; + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + + /* + * During simple_copy_indexinfo_to_parent, we need to convert var from + * child var to parent var, index on expression is too complex to handle. + * so ignore it for now. + */ + if (ind->indexprs != NIL) + continue; + + modified_index = simple_copy_indexinfo_to_parent(root, rel, ind); + /* + * If the unique index doesn't contain partkey, then it is unique + * on this partition only, so it is useless for us. + */ + if (!index_constains_partkey(rel, modified_index)) + continue; + + global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index); + } + + if (global_uniq_indexlist != NIL) + { + foreach(lc, childrels) + { + RelOptInfo *child = lfirst(lc); + if (is_first) + { + is_first = false; + continue; + } + adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist); + } + /* Now we have a list of unique index which are exactly same on all childrels, + * Set the UniqueKey just like it is non-partition table + */ + populate_baserel_uniquekeys(root, rel, global_uniq_indexlist); + } + } +} + + +/* + * populate_distinctrel_uniquekeys + */ +void +populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel) +{ + /* The unique key before the distinct is still valid. */ + distinctrel->uniquekeys = list_copy(inputrel->uniquekeys); + add_uniquekey_from_sortgroups(root, distinctrel, root->parse->distinctClause); +} + +/* + * populate_grouprel_uniquekeys + */ +void +populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel) + +{ + Query *parse = root->parse; + bool input_ukey_added = false; + ListCell *lc; + + if (relation_is_onerow(inputrel)) + { + add_uniquekey_for_onerow(grouprel); + return; + } + if (parse->groupingSets) + return; + + /* A Normal group by without grouping set. */ + if (parse->groupClause) + { + /* + * Current even the groupby clause is Unique already, but if query has aggref + * We have to create grouprel still. To keep the UnqiueKey short, we will check + * the UniqueKey of input_rel still valid, if so we reuse it. + */ + foreach(lc, inputrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (list_is_subset(ukey->exprs, grouprel->reltarget->exprs)) + { + grouprel->uniquekeys = lappend(grouprel->uniquekeys, + ukey); + input_ukey_added = true; + } + } + if (!input_ukey_added) + /* + * group by clause must be a super-set of grouprel->reltarget->exprs except the + * aggregation expr, so if such exprs is unique already, no bother to generate + * new uniquekey for group by exprs. + */ + add_uniquekey_from_sortgroups(root, + grouprel, + root->parse->groupClause); + } + else + /* It has aggregation but without a group by, so only one row returned */ + add_uniquekey_for_onerow(grouprel); +} + +/* + * simple_copy_uniquekeys + * Using a function for the one-line code makes us easy to check where we simply + * copied the uniquekey. + */ +void +simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel) +{ + newrel->uniquekeys = oldrel->uniquekeys; +} + +/* + * populate_unionrel_uniquekeys + */ +void +populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel) +{ + ListCell *lc; + List *exprs = NIL; + + Assert(unionrel->uniquekeys == NIL); + + foreach(lc, unionrel->reltarget->exprs) + { + exprs = lappend(exprs, lfirst(lc)); + } + + if (exprs == NIL) + /* SQL: select union select; is valid, we need to handle it here. */ + add_uniquekey_for_onerow(unionrel); + else + unionrel->uniquekeys = lappend(unionrel->uniquekeys, + makeUniqueKey(exprs,false)); + +} + +/* + * populate_joinrel_uniquekeys + * + * populate uniquekeys for joinrel. We will check each relation to see if its + * UniqueKey is still valid via innerrel_keeps_unique, if so, we add it to + * joinrel. The multi_nullvals field will be changed to true for some outer + * join cases and one-row UniqueKey needs to be converted to normal UniqueKey + * for the same case as well. + * For the uniquekey in either baserel which can't be unique after join, we still + * check to see if combination of UniqueKeys from both side is still useful for us. + * if yes, we add it to joinrel as well. + */ +void +populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *outerrel, RelOptInfo *innerrel, + List *restrictlist, JoinType jointype) +{ + ListCell *lc, *lc2; + List *clause_list = NIL; + List *outerrel_ukey_ctx; + List *innerrel_ukey_ctx; + bool inner_onerow, outer_onerow; + bool mergejoin_allowed; + + /* Care about the outerrel relation only for SEMI/ANTI join */ + if (jointype == JOIN_SEMI || jointype == JOIN_ANTI) + { + foreach(lc, outerrel->uniquekeys) + { + UniqueKey *uniquekey = lfirst_node(UniqueKey, lc); + if (list_is_subset(uniquekey->exprs, joinrel->reltarget->exprs)) + joinrel->uniquekeys = lappend(joinrel->uniquekeys, uniquekey); + } + return; + } + + Assert(jointype == JOIN_LEFT || jointype == JOIN_FULL || jointype == JOIN_INNER); + + /* Fast path */ + if (innerrel->uniquekeys == NIL || outerrel->uniquekeys == NIL) + return; + + inner_onerow = relation_is_onerow(innerrel); + outer_onerow = relation_is_onerow(outerrel); + + outerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(outerrel); + innerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(innerrel); + + clause_list = select_mergejoin_clauses(root, joinrel, outerrel, innerrel, + restrictlist, jointype, + &mergejoin_allowed); + + if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true /* reverse */)) + { + bool outer_impact = jointype == JOIN_FULL; + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + /* Outer relation has one row, and the unique key is not duplicated after join, + * the joinrel will still has one row unless the jointype == JOIN_FULL. + */ + if (outer_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (outer_onerow) + { + /* + * The onerow outerrel becomes multi rows and multi_nullvals + * will be changed to true. We also need to set the exprs correctly since it + * can't be NIL any more. + */ + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(joinrel, outerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Change multi_nullvals to true due to the full join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, true)); + else + /* Just reuse it */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + } + ctx->added_to_joinrel = true; + } + } + + if (innerrel_keeps_unique(root, outerrel, innerrel, clause_list, false)) + { + bool outer_impact = jointype == JOIN_FULL || jointype == JOIN_LEFT;; + + foreach(lc, innerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + if (inner_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (inner_onerow) + { + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(joinrel, innerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Need to change multi_nullvals to true due to the outer join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, + true)); + else + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + + } + ctx->added_to_joinrel = true; + } + } + + /* + * The combination of the UniqueKey from both sides is unique as well regardless + * of join type, but no bother to add it if its subset has been added to joinrel + * already or it is not useful for the joinrel. + */ + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx1 = (UniqueKeyContext) lfirst(lc); + if (ctx1->added_to_joinrel || !ctx1->useful) + continue; + foreach(lc2, innerrel_ukey_ctx) + { + UniqueKeyContext ctx2 = (UniqueKeyContext) lfirst(lc2); + if (ctx2->added_to_joinrel || !ctx2->useful) + continue; + if (add_combined_uniquekey(joinrel, outerrel, innerrel, + ctx1->uniquekey, ctx2->uniquekey, + jointype)) + /* If we set a onerow UniqueKey to joinrel, we don't need other. */ + return; + } + } +} + + +/* + * convert_subquery_uniquekeys + * + * Covert the UniqueKey in subquery to outer relation. + */ +void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel) +{ + ListCell *lc; + + if (sub_final_rel->uniquekeys == NIL) + return; + + if (relation_is_onerow(sub_final_rel)) + { + add_uniquekey_for_onerow(currel); + return; + } + + Assert(currel->subroot != NULL); + + foreach(lc, sub_final_rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + ListCell *lc; + List *exprs = NIL; + bool ukey_useful = true; + + /* One row case is handled above */ + Assert(ukey->exprs != NIL); + foreach(lc, ukey->exprs) + { + Var *var; + TargetEntry *tle = tlist_member(lfirst(lc), + currel->subroot->processed_tlist); + if (tle == NULL) + { + ukey_useful = false; + break; + } + var = find_var_for_subquery_tle(currel, tle); + if (var == NULL) + { + ukey_useful = false; + break; + } + exprs = lappend(exprs, var); + } + + if (ukey_useful) + currel->uniquekeys = lappend(currel->uniquekeys, + makeUniqueKey(exprs, + ukey->multi_nullvals)); + + } +} + +/* + * innerrel_keeps_unique + * + * Check if Unique key of the innerrel is valid after join. innerrel's UniqueKey + * will be still valid if innerrel's any-column mergeop outrerel's uniquekey + * exists in clause_list. + * + * Note: the clause_list must be a list of mergeable restrictinfo already. + */ +static bool +innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *clause_list, + bool reverse) +{ + ListCell *lc, *lc2, *lc3; + + if (outerrel->uniquekeys == NIL || innerrel->uniquekeys == NIL) + return false; + + /* Check if there is outerrel's uniquekey in mergeable clause. */ + foreach(lc, outerrel->uniquekeys) + { + List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs; + bool clauselist_matchs_all_exprs = true; + foreach(lc2, outer_uq_exprs) + { + Node *outer_uq_expr = lfirst(lc2); + bool find_uq_expr_in_clauselist = false; + foreach(lc3, clause_list) + { + RestrictInfo *rinfo = lfirst_node(RestrictInfo, lc3); + Node *outer_expr; + if (reverse) + outer_expr = rinfo->outer_is_left ? get_rightop(rinfo->clause) : get_leftop(rinfo->clause); + else + outer_expr = rinfo->outer_is_left ? get_leftop(rinfo->clause) : get_rightop(rinfo->clause); + if (equal(outer_expr, outer_uq_expr)) + { + find_uq_expr_in_clauselist = true; + break; + } + } + if (!find_uq_expr_in_clauselist) + { + /* No need to check the next exprs in the current uniquekey */ + clauselist_matchs_all_exprs = false; + break; + } + } + + if (clauselist_matchs_all_exprs) + return true; + } + return false; +} + + +/* + * relation_is_onerow + * Check if it is a one-row relation by checking UniqueKey. + */ +bool +relation_is_onerow(RelOptInfo *rel) +{ + UniqueKey *ukey; + if (rel->uniquekeys == NIL) + return false; + ukey = linitial_node(UniqueKey, rel->uniquekeys); + return ukey->exprs == NIL && list_length(rel->uniquekeys) == 1; +} + +/* + * relation_has_uniquekeys_for + * Returns true if we have proofs that 'rel' cannot return multiple rows with + * the same values in each of 'exprs'. Otherwise returns false. + */ +bool +relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel, + List *exprs, bool allow_multinulls) +{ + ListCell *lc; + + /* + * For UniqueKey->onerow case, the uniquekey->exprs is empty as well + * so we can't rely on list_is_subset to handle this special cases + */ + if (exprs == NIL) + return false; + + foreach(lc, rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (ukey->multi_nullvals && !allow_multinulls) + continue; + if (list_is_subset(ukey->exprs, exprs)) + return true; + } + return false; +} + + +/* + * get_exprs_from_uniqueindex + * + * Return a list of exprs which is unique. set useful to false if this + * unique index is not useful for us. + */ +static List * +get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals) +{ + List *exprs = NIL; + ListCell *indexpr_item; + int c = 0; + + *useful = true; + *multi_nullvals = false; + + indexpr_item = list_head(unique_index->indexprs); + for(c = 0; c < unique_index->ncolumns; c++) + { + int attr = unique_index->indexkeys[c]; + Expr *expr; + bool matched_const = false; + ListCell *lc1, *lc2; + + if(attr > 0) + { + expr = list_nth_node(TargetEntry, unique_index->indextlist, c)->expr; + } + else if (attr == 0) + { + /* Expression index */ + expr = lfirst(indexpr_item); + indexpr_item = lnext(unique_index->indexprs, indexpr_item); + } + else /* attr < 0 */ + { + /* Index on system column is not supported */ + Assert(false); + } + + /* + * Check index_col = Const case with regarding to opfamily checking + * If we can remove the index_col from the final UniqueKey->exprs. + */ + forboth(lc1, const_exprs, lc2, const_expr_opfamilies) + { + if (list_member_oid((List *)lfirst(lc2), unique_index->opfamily[c]) + && match_index_to_operand((Node *) lfirst(lc1), c, unique_index)) + { + matched_const = true; + break; + } + } + + if (matched_const) + continue; + + /* Check if the indexed expr is used in rel */ + if (attr > 0) + { + /* + * Normal Indexed column, if the col is not used, then the index is useless + * for uniquekey. + */ + attr -= FirstLowInvalidHeapAttributeNumber; + + if (!bms_is_member(attr, used_varattrs)) + { + *useful = false; + break; + } + } + else if (!list_member(unique_index->rel->reltarget->exprs, expr)) + { + /* Expression index but the expression is not used in rel */ + *useful = false; + break; + } + + /* check not null property. */ + if (attr == 0) + { + /* We never know if a expression yields null or not */ + *multi_nullvals = true; + } + else if (!bms_is_member(attr, unique_index->rel->notnullattrs) + && !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber, + unique_index->rel->notnullattrs)) + { + *multi_nullvals = true; + } + + exprs = lappend(exprs, expr); + } + return exprs; +} + + +/* + * add_uniquekey_for_onerow + * If we are sure that the relation only returns one row, then all the columns + * are unique. However we don't need to create UniqueKey for every column, we + * just set exprs = NIL and overwrites all the other UniqueKey on this RelOptInfo + * since this one has strongest semantics. + */ +void +add_uniquekey_for_onerow(RelOptInfo *rel) +{ + /* + * We overwrite the previous UniqueKey on purpose since this one has the + * strongest semantic. + */ + rel->uniquekeys = list_make1(makeUniqueKey(NIL, false)); +} + + +/* + * initililze_uniquecontext_for_joinrel + * Return a List of UniqueKeyContext for an inputrel + */ +static List * +initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel) +{ + List *res = NIL; + ListCell *lc; + foreach(lc, inputrel->uniquekeys) + { + UniqueKeyContext context; + context = palloc(sizeof(struct UniqueKeyContextData)); + context->uniquekey = lfirst_node(UniqueKey, lc); + context->added_to_joinrel = false; + context->useful = true; + res = lappend(res, context); + } + return res; +} + + +/* + * get_exprs_from_uniquekey + * Unify the way of get List of exprs from a one-row UniqueKey or + * normal UniqueKey. for the onerow case, every expr in rel1 is a valid + * UniqueKey. Return a List of exprs. + * + * rel1: The relation which you want to get the exprs. + * ukey: The UniqueKey you want to get the exprs. + */ +static List * +get_exprs_from_uniquekey(RelOptInfo *joinrel, RelOptInfo *rel1, UniqueKey *ukey) +{ + ListCell *lc; + bool onerow = rel1 != NULL && relation_is_onerow(rel1); + + List *res = NIL; + Assert(onerow || ukey); + if (onerow) + { + /* Only cares about the exprs still exist in joinrel */ + foreach(lc, joinrel->reltarget->exprs) + { + Bitmapset *relids = pull_varnos(lfirst(lc)); + if (bms_is_subset(relids, rel1->relids)) + { + res = lappend(res, list_make1(lfirst(lc))); + } + } + } + else + { + res = list_make1(ukey->exprs); + } + return res; +} + +/* + * Partitioned table Unique Keys. + * The partition table unique key is maintained as: + * 1. The index must be unique as usual. + * 2. The index must contains partition key. + * 3. The index must exist on all the child rel. see simple_indexinfo_equal for + * how we compare it. + */ + +/* + * index_constains_partkey + * return true if the index contains the partiton key. + */ +static bool +index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind) +{ + ListCell *lc; + int i; + Assert(IS_PARTITIONED_REL(partrel)); + Assert(partrel->part_scheme->partnatts > 0); + + for(i = 0; i < partrel->part_scheme->partnatts; i++) + { + Node *part_expr = linitial(partrel->partexprs[i]); + bool found_in_index = false; + foreach(lc, ind->indextlist) + { + Expr *index_expr = lfirst_node(TargetEntry, lc)->expr; + if (equal(index_expr, part_expr)) + { + found_in_index = true; + break; + } + } + if (!found_in_index) + return false; + } + return true; +} + +/* + * simple_indexinfo_equal + * + * Used to check if the 2 index is same as each other. The index here + * is COPIED from childrel and did some tiny changes(see + * simple_copy_indexinfo_to_parent) + */ +static bool +simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2) +{ + Size oid_cmp_len = sizeof(Oid) * ind1->ncolumns; + + return ind1->ncolumns == ind2->ncolumns && + ind1->unique == ind2->unique && + memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0 && + memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0 && + memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0 && + memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0 && + equal(get_tlist_exprs(ind1->indextlist, true), + get_tlist_exprs(ind2->indextlist, true)); +} + + +/* + * The below macros are used for simple_copy_indexinfo_to_parent which is so + * customized that I don't want to put it to copyfuncs.c. So copy it here. + */ +#define COPY_POINTER_FIELD(fldname, sz) \ + do { \ + Size _size = (sz); \ + newnode->fldname = palloc(_size); \ + memcpy(newnode->fldname, from->fldname, _size); \ + } while (0) + +#define COPY_NODE_FIELD(fldname) \ + (newnode->fldname = copyObjectImpl(from->fldname)) + +#define COPY_SCALAR_FIELD(fldname) \ + (newnode->fldname = from->fldname) + + +/* + * simple_copy_indexinfo_to_parent (from partition) + * Copy the IndexInfo from child relation to parent relation with some modification, + * which is used to test: + * 1. If the same index exists in all the childrels. + * 2. If the parentrel->reltarget/basicrestrict info matches this index. + */ +static IndexOptInfo * +simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from) +{ + IndexOptInfo *newnode = makeNode(IndexOptInfo); + AppendRelInfo *appinfo = find_appinfo_by_child(root, from->rel->relid); + ListCell *lc; + int idx = 0; + + COPY_SCALAR_FIELD(ncolumns); + COPY_SCALAR_FIELD(nkeycolumns); + COPY_SCALAR_FIELD(unique); + COPY_SCALAR_FIELD(immediate); + /* We just need to know if it is NIL or not */ + COPY_SCALAR_FIELD(indpred); + COPY_SCALAR_FIELD(predOK); + COPY_POINTER_FIELD(indexkeys, from->ncolumns * sizeof(int)); + COPY_POINTER_FIELD(indexcollations, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opfamily, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opcintype, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(sortopfamily, from->ncolumns * sizeof(Oid)); + COPY_NODE_FIELD(indextlist); + + /* Convert index exprs on child expr to expr on parent */ + foreach(lc, newnode->indextlist) + { + TargetEntry *tle = lfirst_node(TargetEntry, lc); + /* Index on expression is ignored */ + Assert(IsA(tle->expr, Var)); + tle->expr = (Expr *) find_parent_var(appinfo, (Var *) tle->expr); + newnode->indexkeys[idx] = castNode(Var, tle->expr)->varattno; + idx++; + } + newnode->rel = parentrel; + return newnode; +} + +/* + * adjust_partition_unique_indexlist + * + * global_unique_indexes: At the beginning, it contains the copy & modified + * unique index from the first partition. And then check if each index in it still + * exists in the following partitions. If no, remove it. at last, it has an + * index list which exists in all the partitions. + */ +static void +adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_indexes) +{ + ListCell *lc, *lc2; + foreach(lc, *global_unique_indexes) + { + IndexOptInfo *g_ind = lfirst_node(IndexOptInfo, lc); + bool found_in_child = false; + + foreach(lc2, childrel->indexlist) + { + IndexOptInfo *p_ind = lfirst_node(IndexOptInfo, lc2); + IndexOptInfo *p_ind_copy; + if (!p_ind->unique || !p_ind->immediate || + (p_ind->indpred != NIL && !p_ind->predOK)) + continue; + p_ind_copy = simple_copy_indexinfo_to_parent(root, parentrel, p_ind); + if (simple_indexinfo_equal(p_ind_copy, g_ind)) + { + found_in_child = true; + break; + } + } + if (!found_in_child) + /* The index doesn't exist in childrel, remove it from global_unique_indexes */ + *global_unique_indexes = foreach_delete_current(*global_unique_indexes, lc); + } +} + +/* Helper function for groupres/distinctrel */ +static void +add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgroups) +{ + Query *parse = root->parse; + List *exprs; + + /* + * XXX: If there are some vars which is not in current levelsup, the semantic is + * imprecise, should we avoid it or not? levelsup = 1 is just a demo, maybe we need to + * check every level other than 0, if so, looks we have to write another + * pull_var_walker. + */ + List *upper_vars = pull_vars_of_level((Node*)sortgroups, 1); + + if (upper_vars != NIL) + return; + + exprs = get_sortgrouplist_exprs(sortgroups, parse->targetList); + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(exprs, + false /* sortgroupclause can't be multi_nullvals */)); +} + + +/* + * add_combined_uniquekey + * The combination of both UniqueKeys is a valid UniqueKey for joinrel no matter + * the jointype. + */ +bool +add_combined_uniquekey(RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype) +{ + + ListCell *lc1, *lc2; + + /* Either side has multi_nullvals or we have outer join, + * the combined UniqueKey has multi_nullvals */ + bool multi_nullvals = outer_ukey->multi_nullvals || + inner_ukey->multi_nullvals || IS_OUTER_JOIN(jointype); + + /* The only case we can get onerow joinrel after join */ + if (relation_is_onerow(outer_rel) + && relation_is_onerow(inner_rel) + && jointype == JOIN_INNER) + { + add_uniquekey_for_onerow(joinrel); + return true; + } + + foreach(lc1, get_exprs_from_uniquekey(joinrel, outer_rel, outer_ukey)) + { + foreach(lc2, get_exprs_from_uniquekey(joinrel, inner_rel, inner_ukey)) + { + List *exprs = list_concat_copy(lfirst_node(List, lc1), lfirst_node(List, lc2)); + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(exprs, + multi_nullvals)); + } + } + return false; +} diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index b406d41e91..0551ae0512 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -2389,6 +2389,8 @@ grouping_planner(PlannerInfo *root, bool inheritance_update, add_path(final_rel, path); } + simple_copy_uniquekeys(current_rel, final_rel); + /* * Generate partial paths for final_rel, too, if outer query levels might * be able to make use of them. @@ -3899,6 +3901,8 @@ create_grouping_paths(PlannerInfo *root, } set_cheapest(grouped_rel); + + populate_grouprel_uniquekeys(root, grouped_rel, input_rel); return grouped_rel; } @@ -4615,7 +4619,7 @@ create_window_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(window_rel); - + simple_copy_uniquekeys(input_rel, window_rel); return window_rel; } @@ -4911,7 +4915,7 @@ create_distinct_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(distinct_rel); - + populate_distinctrel_uniquekeys(root, input_rel, distinct_rel); return distinct_rel; } @@ -5172,6 +5176,8 @@ create_ordered_paths(PlannerInfo *root, */ Assert(ordered_rel->pathlist != NIL); + simple_copy_uniquekeys(input_rel, ordered_rel); + return ordered_rel; } @@ -6049,6 +6055,9 @@ adjust_paths_for_srfs(PlannerInfo *root, RelOptInfo *rel, if (list_length(targets) == 1) return; + /* UniqueKey is not valid after handling the SRF. */ + rel->uniquekeys = NIL; + /* * Stack SRF-evaluation nodes atop each path for the rel. * diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c index 951aed80e7..e94e92937c 100644 --- a/src/backend/optimizer/prep/prepunion.c +++ b/src/backend/optimizer/prep/prepunion.c @@ -689,6 +689,8 @@ generate_union_paths(SetOperationStmt *op, PlannerInfo *root, /* Undo effects of possibly forcing tuple_fraction to 0 */ root->tuple_fraction = save_fraction; + /* Add the UniqueKeys */ + populate_unionrel_uniquekeys(root, result_rel); return result_rel; } diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c index d722063cf3..44c37ecffc 100644 --- a/src/backend/optimizer/util/appendinfo.c +++ b/src/backend/optimizer/util/appendinfo.c @@ -746,3 +746,47 @@ find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos) } return appinfos; } + +/* + * find_appinfo_by_child + * + */ +AppendRelInfo * +find_appinfo_by_child(PlannerInfo *root, Index child_index) +{ + ListCell *lc; + foreach(lc, root->append_rel_list) + { + AppendRelInfo *appinfo = lfirst_node(AppendRelInfo, lc); + if (appinfo->child_relid == child_index) + return appinfo; + } + elog(ERROR, "parent relation cant be found"); + return NULL; +} + +/* + * find_parent_var + * + */ +Var * +find_parent_var(AppendRelInfo *appinfo, Var *child_var) +{ + ListCell *lc; + Var *res = NULL; + Index attno = 1; + foreach(lc, appinfo->translated_vars) + { + Node *child_node = lfirst(lc); + if (equal(child_node, child_var)) + { + res = copyObject(child_var); + res->varattno = attno; + res->varno = appinfo->parent_relid; + } + attno++; + } + if (res == NULL) + elog(ERROR, "parent var can't be found."); + return res; +} diff --git a/src/backend/optimizer/util/inherit.c b/src/backend/optimizer/util/inherit.c index 3132fd35a5..d66b40ec50 100644 --- a/src/backend/optimizer/util/inherit.c +++ b/src/backend/optimizer/util/inherit.c @@ -736,6 +736,7 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, { Node *onecq = (Node *) lfirst(lc2); bool pseudoconstant; + RestrictInfo *child_rinfo; /* check for pseudoconstant (no Vars or volatile functions) */ pseudoconstant = @@ -747,13 +748,14 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, root->hasPseudoConstantQuals = true; } /* reconstitute RestrictInfo with appropriate properties */ - childquals = lappend(childquals, - make_restrictinfo((Expr *) onecq, - rinfo->is_pushed_down, - rinfo->outerjoin_delayed, - pseudoconstant, - rinfo->security_level, - NULL, NULL, NULL)); + child_rinfo = make_restrictinfo((Expr *) onecq, + rinfo->is_pushed_down, + rinfo->outerjoin_delayed, + pseudoconstant, + rinfo->security_level, + NULL, NULL, NULL); + child_rinfo->mergeopfamilies = rinfo->mergeopfamilies; + childquals = lappend(childquals, child_rinfo); /* track minimum security level among child quals */ cq_min_security = Min(cq_min_security, rinfo->security_level); } diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 31d9aedeeb..c83f17acb7 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -16,6 +16,7 @@ #include "nodes/execnodes.h" #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" extern A_Expr *makeA_Expr(A_Expr_Kind kind, List *name, @@ -105,4 +106,6 @@ extern GroupingSet *makeGroupingSet(GroupingSetKind kind, List *content, int loc extern VacuumRelation *makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols); +extern UniqueKey* makeUniqueKey(List *exprs, bool multi_nullvals); + #endif /* MAKEFUNC_H */ diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 381d84b4e4..41110ed888 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -264,6 +264,7 @@ typedef enum NodeTag T_EquivalenceMember, T_PathKey, T_PathTarget, + T_UniqueKey, T_RestrictInfo, T_IndexClause, T_PlaceHolderVar, diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index 9e3ebd488a..02e4458bef 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -730,6 +730,7 @@ typedef struct RelOptInfo QualCost baserestrictcost; /* cost of evaluating the above */ Index baserestrict_min_security; /* min security_level found in * baserestrictinfo */ + List *uniquekeys; /* List of UniqueKey */ List *joininfo; /* RestrictInfo structures for join clauses * involving this rel */ bool has_eclass_joins; /* T means joininfo is incomplete */ @@ -1047,6 +1048,28 @@ typedef struct PathKey } PathKey; +/* + * UniqueKey + * + * Represents the unique properties held by a RelOptInfo. + * + * exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL + * is a special case of UniqueKey, which means there is only 1 row in that + * relation. + * multi_nullvals: true means multi null values may exist in these exprs, so the + * uniqueness is not guaranteed in this case. This field is necessary for + * remove_useless_join & reduce_unique_semijoins where we don't mind these + * duplicated NULL values. It is set to true for 2 cases. One is a unique key + * from a unique index but the related column is nullable. The other one is for + * outer join. see populate_joinrel_uniquekeys for detail. + */ +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + /* * PathTarget * @@ -2473,7 +2496,7 @@ typedef enum * * flags indicating what kinds of grouping are possible. * partial_costs_set is true if the agg_partial_costs and agg_final_costs - * have been initialized. + * have been initialized. * agg_partial_costs gives partial aggregation costs. * agg_final_costs gives finalization costs. * target_parallel_safe is true if target is parallel safe. @@ -2503,8 +2526,8 @@ typedef struct * limit_tuples is an estimated bound on the number of output tuples, * or -1 if no LIMIT or couldn't estimate. * count_est and offset_est are the estimated values of the LIMIT and OFFSET - * expressions computed by preprocess_limit() (see comments for - * preprocess_limit() for more information). + * expressions computed by preprocess_limit() (see comments for + * preprocess_limit() for more information). */ typedef struct { diff --git a/src/include/nodes/pg_list.h b/src/include/nodes/pg_list.h index 14ea2766ad..621f54a9f8 100644 --- a/src/include/nodes/pg_list.h +++ b/src/include/nodes/pg_list.h @@ -528,6 +528,8 @@ extern bool list_member_ptr(const List *list, const void *datum); extern bool list_member_int(const List *list, int datum); extern bool list_member_oid(const List *list, Oid datum); +extern bool list_is_subset(const List *members, const List *target); + extern List *list_delete(List *list, void *datum); extern List *list_delete_ptr(List *list, void *datum); extern List *list_delete_int(List *list, int datum); diff --git a/src/include/optimizer/appendinfo.h b/src/include/optimizer/appendinfo.h index d6a27a60dd..e87c92a054 100644 --- a/src/include/optimizer/appendinfo.h +++ b/src/include/optimizer/appendinfo.h @@ -32,4 +32,7 @@ extern Relids adjust_child_relids_multilevel(PlannerInfo *root, Relids relids, extern AppendRelInfo **find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos); +extern AppendRelInfo *find_appinfo_by_child(PlannerInfo *root, Index child_index); +extern Var *find_parent_var(AppendRelInfo *appinfo, Var *child_var); + #endif /* APPENDINFO_H */ diff --git a/src/include/optimizer/optimizer.h b/src/include/optimizer/optimizer.h index 3e4171056e..9445141263 100644 --- a/src/include/optimizer/optimizer.h +++ b/src/include/optimizer/optimizer.h @@ -23,6 +23,7 @@ #define OPTIMIZER_H #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" /* * We don't want to include nodes/pathnodes.h here, because non-planner @@ -156,6 +157,7 @@ extern TargetEntry *get_sortgroupref_tle(Index sortref, List *targetList); extern TargetEntry *get_sortgroupclause_tle(SortGroupClause *sgClause, List *targetList); +extern Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); extern Node *get_sortgroupclause_expr(SortGroupClause *sgClause, List *targetList); extern List *get_sortgrouplist_exprs(List *sgClauses, diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h index 10b6e81079..9217a8d6c6 100644 --- a/src/include/optimizer/paths.h +++ b/src/include/optimizer/paths.h @@ -240,5 +240,48 @@ extern PathKey *make_canonical_pathkey(PlannerInfo *root, int strategy, bool nulls_first); extern void add_paths_to_append_rel(PlannerInfo *root, RelOptInfo *rel, List *live_childrels); +extern List *select_mergejoin_clauses(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + JoinType jointype, + bool *mergejoin_allowed); + +/* + * uniquekeys.c + * Utilities for matching and building unique keys + */ +extern void populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List* unique_index_list); +extern void populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels); +extern void populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel); +extern void populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel); +extern void populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel); +extern void simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel); +extern void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel); +extern void populate_joinrel_uniquekeys(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + RelOptInfo *rel2, + List *restrictlist, + JoinType jointype); + +extern bool relation_has_uniquekeys_for(PlannerInfo *root, + RelOptInfo *rel, + List *exprs, + bool allow_multinulls); +extern bool relation_is_onerow(RelOptInfo *rel); #endif /* PATHS_H */ -- 2.21.0 --op7mf2i72xk5yu7c Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v36-0003-Extend-UniqueKeys.patch" ^ permalink raw reply [nested|flat] 91+ messages in thread
* [PATCH v38 2/6] Introduce UniqueKey attributes on RelOptInfo struct. @ 2020-05-11 07:50 一挃 <[email protected]> 0 siblings, 0 replies; 91+ messages in thread From: 一挃 @ 2020-05-11 07:50 UTC (permalink / raw) UniqueKey is a set of exprs on RelOptInfo which represents the exprs will be unique on the given RelOptInfo. You can see README.uniquekey for more information. --- src/backend/nodes/copyfuncs.c | 13 + src/backend/nodes/list.c | 31 + src/backend/nodes/makefuncs.c | 13 + src/backend/nodes/outfuncs.c | 11 + src/backend/nodes/readfuncs.c | 10 + src/backend/optimizer/path/Makefile | 3 +- src/backend/optimizer/path/README.uniquekey | 131 +++ src/backend/optimizer/path/allpaths.c | 10 + src/backend/optimizer/path/joinpath.c | 9 +- src/backend/optimizer/path/joinrels.c | 2 + src/backend/optimizer/path/pathkeys.c | 3 +- src/backend/optimizer/path/uniquekeys.c | 1135 +++++++++++++++++++ src/backend/optimizer/plan/planner.c | 13 +- src/backend/optimizer/prep/prepunion.c | 2 + src/backend/optimizer/util/appendinfo.c | 44 + src/backend/optimizer/util/inherit.c | 18 +- src/include/nodes/makefuncs.h | 3 + src/include/nodes/nodes.h | 1 + src/include/nodes/pathnodes.h | 29 +- src/include/nodes/pg_list.h | 10 + src/include/optimizer/appendinfo.h | 3 + src/include/optimizer/optimizer.h | 2 + src/include/optimizer/paths.h | 43 + 23 files changed, 1515 insertions(+), 24 deletions(-) create mode 100644 src/backend/optimizer/path/README.uniquekey create mode 100644 src/backend/optimizer/path/uniquekeys.c diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..75c1c5e824 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2296,6 +2296,16 @@ _copyPathKey(const PathKey *from) return newnode; } +static UniqueKey * +_copyUniqueKey(const UniqueKey *from) +{ + UniqueKey *newnode = makeNode(UniqueKey); + + COPY_NODE_FIELD(exprs); + COPY_SCALAR_FIELD(multi_nullvals); + + return newnode; +} /* * _copyRestrictInfo */ @@ -5220,6 +5230,9 @@ copyObjectImpl(const void *from) case T_PathKey: retval = _copyPathKey(from); break; + case T_UniqueKey: + retval = _copyUniqueKey(from); + break; case T_RestrictInfo: retval = _copyRestrictInfo(from); break; diff --git a/src/backend/nodes/list.c b/src/backend/nodes/list.c index dbf6b30233..ca099495a1 100644 --- a/src/backend/nodes/list.c +++ b/src/backend/nodes/list.c @@ -702,6 +702,37 @@ list_member_oid(const List *list, Oid datum) return false; } +/* + * return true iff every entry in "members" list is also present + * in the "target" list. + */ +bool +list_is_subset(const List *members, const List *target) +{ + const ListCell *lc1, *lc2; + + Assert(IsPointerList(members)); + Assert(IsPointerList(target)); + check_list_invariants(members); + check_list_invariants(target); + + foreach(lc1, members) + { + bool found = false; + foreach(lc2, target) + { + if (equal(lfirst(lc1), lfirst(lc2))) + { + found = true; + break; + } + } + if (!found) + return false; + } + return true; +} + /* * Delete the n'th cell (counting from 0) in list. * diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 01c110cd2f..40415d0f5b 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -815,3 +815,16 @@ makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols) v->va_cols = va_cols; return v; } + + +/* + * makeUniqueKey + */ +UniqueKey* +makeUniqueKey(List *exprs, bool multi_nullvals) +{ + UniqueKey * ukey = makeNode(UniqueKey); + ukey->exprs = exprs; + ukey->multi_nullvals = multi_nullvals; + return ukey; +} diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..44154cde6a 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2456,6 +2456,14 @@ _outPathKey(StringInfo str, const PathKey *node) WRITE_BOOL_FIELD(pk_nulls_first); } +static void +_outUniqueKey(StringInfo str, const UniqueKey *node) +{ + WRITE_NODE_TYPE("UNIQUEKEY"); + WRITE_NODE_FIELD(exprs); + WRITE_BOOL_FIELD(multi_nullvals); +} + static void _outPathTarget(StringInfo str, const PathTarget *node) { @@ -4198,6 +4206,9 @@ outNode(StringInfo str, const void *obj) case T_PathKey: _outPathKey(str, obj); break; + case T_UniqueKey: + _outUniqueKey(str, obj); + break; case T_PathTarget: _outPathTarget(str, obj); break; diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index c5e136e9c3..b3e212bf1c 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -492,6 +492,14 @@ _readSetOperationStmt(void) READ_DONE(); } +static UniqueKey * +_readUniqueKey(void) +{ + READ_LOCALS(UniqueKey); + READ_NODE_FIELD(exprs); + READ_BOOL_FIELD(multi_nullvals); + READ_DONE(); +} /* * Stuff from primnodes.h. @@ -2717,6 +2725,8 @@ parseNodeString(void) return_value = _readCommonTableExpr(); else if (MATCH("SETOPERATIONSTMT", 16)) return_value = _readSetOperationStmt(); + else if (MATCH("UNIQUEKEY", 9)) + return_value = _readUniqueKey(); else if (MATCH("ALIAS", 5)) return_value = _readAlias(); else if (MATCH("RANGEVAR", 8)) diff --git a/src/backend/optimizer/path/Makefile b/src/backend/optimizer/path/Makefile index 1e199ff66f..7b9820c25f 100644 --- a/src/backend/optimizer/path/Makefile +++ b/src/backend/optimizer/path/Makefile @@ -21,6 +21,7 @@ OBJS = \ joinpath.o \ joinrels.o \ pathkeys.o \ - tidpath.o + tidpath.o \ + uniquekeys.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/optimizer/path/README.uniquekey b/src/backend/optimizer/path/README.uniquekey new file mode 100644 index 0000000000..5eac761995 --- /dev/null +++ b/src/backend/optimizer/path/README.uniquekey @@ -0,0 +1,131 @@ +1. What is UniqueKey? +We can think UniqueKey is a set of exprs for a RelOptInfo, which we are insure +that doesn't yields same result among all the rows. The simplest UniqueKey +format is primary key. + +However we define the UnqiueKey as below. + +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + +exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL +is a special case of UniqueKey, which means there is only one row in that +relation.it has a stronger semantic than others. like SELECT uk FROM t; uk is +normal unique key and may have different values. SELECT colx FROM t WHERE uk = +const. colx is unique AND we have only 1 value. This field can used for +innerrel_is_unique. this logic is handled specially in add_uniquekey_for_onerow +function. + +multi_nullvals: true means multi null values may exist in these exprs, so the +uniqueness is not guaranteed in this case. This field is necessary for +remove_useless_join & reduce_unique_semijoins where we don't mind these +duplicated NULL values. It is set to true for 2 cases. One is a unique key +from a unique index but the related column is nullable. The other one is for +outer join. see populate_joinrel_uniquekeys for detail. + + +The UniqueKey can be used at the following cases at least: +1. remove_useless_joins. +2. reduce_semianti_joins +3. remove distinct node if distinct clause is unique. +4. remove aggnode if group by clause is unique. +5. Index Skip Scan (WIP) +6. Aggregation Push Down without 2 phase aggregation if the join can't + duplicated the aggregated rows. (work in progress feature) + +2. How is it maintained? + +We have a set of populate_xxx_unqiuekeys functions to maintain the uniquekey on +various cases. xxx includes baserel, joinrel, partitionedrel, distinctrel, +groupedrel, unionrel. and we also need to convert the uniquekey from subquery +to outer relation, which is what convert_subquery_uniquekeys does. + +1. The first part is about baserel. We handled 3 cases. suppose we have Unique +Index on (a, b). + +1. SELECT a, b FROM t. UniqueKey (a, b) +2. SELECT a FROM t WHERE b = 1; UniqueKey (a) +3. SELECT .. FROM t WHERE a = 1 AND b = 1; UniqueKey (NIL). onerow case, every + column is Unique. + +2. The next part is joinrel, this part is most error-prone, we simplified the rules +like below: +1. If the relation's UniqueKey can't be duplicated after join, then is will be + still valid for the join rel. The function we used here is + innerrel_keeps_unique. The basic idea is innerrel.any_col = outer.uk. + +2. If the UnqiueKey can't keep valid via the rule 1, the combination of the + UniqueKey from both sides are valid for sure. We can prove this as: if the + unique exprs from rel1 is duplicated by rel2, the duplicated rows must + contains different unique exprs from rel2. + +More considerations about onerow: +1. If relation with one row and it can't be duplicated, it is still possible + contains mulit_nullvas after outer join. +2. If the either UniqueKey can be duplicated after join, the can get one row + only when both side is one row AND there is no outer join. +3. Whenever the onerow UniqueKey is not a valid any more, we need to convert one + row UniqueKey to normal unique key since we don't store exprs for one-row + relation. get_exprs_from_uniquekeys will be used here. + + +More considerations about multi_nullvals after join: +1. If the original UnqiueKey has multi_nullvals, the final UniqueKey will have + mulit_nullvals in any case. +2. If a unique key doesn't allow mulit_nullvals, after some outer join, it + allows some outer join. + + +3. When we comes to subquery, we need to convert_subquery_unqiuekeys just like +convert_subquery_pathkeys. Only the UniqueKey insides subquery is referenced as +a Var in outer relation will be reused. The relationship between the outerrel.Var +and subquery.exprs is built with outerel->subroot->processed_tlist. + + +4. As for the SRF functions, it will break the uniqueness of uniquekey, However it +is handled in adjust_paths_for_srfs, which happens after the query_planner. so +we will maintain the UniqueKey until there and reset it to NIL at that +places. This can't help on distinct/group by elimination cases but probably help +in some other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is +semantic correctly. + + +5. As for inherit table, we first main the UnqiueKey on childrel as well. But for +partitioned table we need to maintain 2 different kinds of +UnqiueKey. 1). UniqueKey on the parent relation 2). UniqueKey on child +relation for partition wise query. + +Example: +CREATE TABLE p (a int not null, b int not null) partition by list (a); +CREATE TABLE p0 partition of p for values in (1); +CREATE TABLE p1 partition of p for values in (2); + +create unique index p0_b on p0(b); +create unique index p1_b on p1(b); + +Now b is only unique on partition level, so the distinct can't be removed on +the following cases. SELECT DISTINCT b FROM p; + +Another example is SELECT DISTINCT a, b FROM p WHERE a = 1; Since only one +partition is chosen, the UniqueKey on child relation is same as the UniqueKey on +parent relation. + +Another usage of UniqueKey on partition level is it be helpful for +partition-wise join. + +As for the UniqueKey on parent table level, it comes with 2 different ways, +1). the UniqueKey is also derived in UniqueKey index, but the index must be same +in all the related children relations and the unique index must contains +Partition Key in it. Example: + +CREATE UNIQUE INDEX p_ab ON p(a, b); -- where a is the partition key. + +-- Query +SELECT a, b FROM p; the (a, b) is a UniqueKey of p. + +2). If the parent relation has only one childrel, the UniqueKey on childrel is + the UniqueKey on parent as well. diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index 37b4223adb..d52ad59f89 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -579,6 +579,12 @@ set_plain_rel_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) */ check_index_predicates(root, rel); + /* + * Now that we've marked which partial indexes are suitable, we can now + * build the relation's unique keys. + */ + populate_baserel_uniquekeys(root, rel, rel->indexlist); + /* Mark rel with estimated output rows, width, etc */ set_baserel_size_estimates(root, rel); } @@ -1297,6 +1303,8 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, /* Add paths to the append relation. */ add_paths_to_append_rel(root, rel, live_childrels); + if (IS_PARTITIONED_REL(rel)) + populate_partitionedrel_uniquekeys(root, rel, live_childrels); } @@ -2304,6 +2312,8 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, pathkeys, required_outer)); } + convert_subquery_uniquekeys(root, rel, sub_final_rel); + /* If outer rel allows parallelism, do same for partial paths. */ if (rel->consider_parallel && bms_is_empty(required_outer)) { diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c index 57ce97fd53..697ac047fb 100644 --- a/src/backend/optimizer/path/joinpath.c +++ b/src/backend/optimizer/path/joinpath.c @@ -71,13 +71,6 @@ static void consider_parallel_mergejoin(PlannerInfo *root, static void hash_inner_and_outer(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, RelOptInfo *innerrel, JoinType jointype, JoinPathExtraData *extra); -static List *select_mergejoin_clauses(PlannerInfo *root, - RelOptInfo *joinrel, - RelOptInfo *outerrel, - RelOptInfo *innerrel, - List *restrictlist, - JoinType jointype, - bool *mergejoin_allowed); static void generate_mergejoin_paths(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *innerrel, @@ -1927,7 +1920,7 @@ hash_inner_and_outer(PlannerInfo *root, * if it is mergejoinable and involves vars from the two sub-relations * currently of interest. */ -static List * +List * select_mergejoin_clauses(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c index 0dbe2ac726..7271f044ec 100644 --- a/src/backend/optimizer/path/joinrels.c +++ b/src/backend/optimizer/path/joinrels.c @@ -924,6 +924,8 @@ populate_joinrel_with_paths(PlannerInfo *root, RelOptInfo *rel1, /* Apply partitionwise join technique, if possible. */ try_partitionwise_join(root, rel1, rel2, joinrel, sjinfo, restrictlist); + + populate_joinrel_uniquekeys(root, joinrel, rel1, rel2, restrictlist, sjinfo->jointype); } diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index bd9a176d7d..139278829b 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -33,7 +33,6 @@ static bool pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys); static bool matches_boolean_partition_clause(RestrictInfo *rinfo, RelOptInfo *partrel, int partkeycol); -static Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); static bool right_merge_direction(PlannerInfo *root, PathKey *pathkey); @@ -1035,7 +1034,7 @@ convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel, * We need this to ensure that we don't return pathkeys describing values * that are unavailable above the level of the subquery scan. */ -static Var * +Var * find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle) { ListCell *lc; diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c new file mode 100644 index 0000000000..77ed2b2eff --- /dev/null +++ b/src/backend/optimizer/path/uniquekeys.c @@ -0,0 +1,1135 @@ +/*------------------------------------------------------------------------- + * + * uniquekeys.c + * Utilities for matching and building unique keys + * + * Portions Copyright (c) 2020, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/optimizer/path/uniquekeys.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "nodes/makefuncs.h" +#include "nodes/nodeFuncs.h" +#include "optimizer/pathnode.h" +#include "optimizer/paths.h" +#include "optimizer/appendinfo.h" +#include "optimizer/optimizer.h" +#include "optimizer/tlist.h" +#include "rewrite/rewriteManip.h" + + +/* + * This struct is used to help populate_joinrel_uniquekeys. + * + * added_to_joinrel is true if a uniquekey (from outerrel or innerrel) + * has been added to joinrel. + * useful is true if the exprs of the uniquekey still appears in joinrel. + */ +typedef struct UniqueKeyContextData +{ + UniqueKey *uniquekey; + bool added_to_joinrel; + bool useful; +} *UniqueKeyContext; + +static List *initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel); +static bool innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + bool reverse); + +static List *get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals); +static List *get_exprs_from_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + UniqueKey *ukey); +static void add_uniquekey_for_onerow(RelOptInfo *rel); +static bool add_combined_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype); + +/* Used for unique indexes checking for partitioned table */ +static bool index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind); +static IndexOptInfo *simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from); +static bool simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2); +static void adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_index); + +/* Helper function for grouped relation and distinct relation. */ +static void add_uniquekey_from_sortgroups(PlannerInfo *root, + RelOptInfo *rel, + List *sortgroups); + +/* + * populate_baserel_uniquekeys + * Populate 'baserel' uniquekeys list by looking at the rel's unique index + * and baserestrictinfo + */ +void +populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List *indexlist) +{ + ListCell *lc; + List *matched_uniq_indexes = NIL; + + /* Attrs appears in rel->reltarget->exprs. */ + Bitmapset *used_attrs = NULL; + + List *const_exprs = NIL; + List *expr_opfamilies = NIL; + + Assert(baserel->rtekind == RTE_RELATION); + + foreach(lc, indexlist) + { + IndexOptInfo *ind = (IndexOptInfo *) lfirst(lc); + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + matched_uniq_indexes = lappend(matched_uniq_indexes, ind); + } + + if (matched_uniq_indexes == NIL) + return; + + /* Check which attrs is used in baserel->reltarget */ + pull_varattnos((Node *)baserel->reltarget->exprs, baserel->relid, &used_attrs); + + /* Check which attrno is used at a mergeable const filter */ + foreach(lc, baserel->baserestrictinfo) + { + RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); + + if (rinfo->mergeopfamilies == NIL) + continue; + + if (bms_is_empty(rinfo->left_relids)) + { + const_exprs = lappend(const_exprs, get_rightop(rinfo->clause)); + } + else if (bms_is_empty(rinfo->right_relids)) + { + const_exprs = lappend(const_exprs, get_leftop(rinfo->clause)); + } + else + continue; + + expr_opfamilies = lappend(expr_opfamilies, rinfo->mergeopfamilies); + } + + foreach(lc, matched_uniq_indexes) + { + bool multi_nullvals, useful; + List *exprs = get_exprs_from_uniqueindex(lfirst_node(IndexOptInfo, lc), + const_exprs, + expr_opfamilies, + used_attrs, + &useful, + &multi_nullvals); + if (useful) + { + if (exprs == NIL) + { + /* All the columns in Unique Index matched with a restrictinfo */ + add_uniquekey_for_onerow(baserel); + return; + } + baserel->uniquekeys = lappend(baserel->uniquekeys, + makeUniqueKey(exprs, multi_nullvals)); + } + } +} + + +/* + * populate_partitionedrel_uniquekeys + * The UniqueKey on partitionrel comes from 2 cases: + * 1). Only one partition is involved in this query, the unique key can be + * copied to parent rel from childrel. + * 2). There are some unique index which includes partition key and exists + * in all the related partitions. + * We never mind rule 2 if we hit rule 1. + */ + +void +populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels) +{ + ListCell *lc; + List *global_uniq_indexlist = NIL; + RelOptInfo *childrel; + bool is_first = true; + + Assert(IS_PARTITIONED_REL(rel)); + + if (childrels == NIL) + return; + + /* + * If there is only one partition used in this query, the UniqueKey in childrel is + * still valid in parent level, but we need convert the format from child expr to + * parent expr. + */ + if (list_length(childrels) == 1) + { + /* Check for Rule 1 */ + RelOptInfo *childrel = linitial_node(RelOptInfo, childrels); + ListCell *lc; + Assert(childrel->reloptkind == RELOPT_OTHER_MEMBER_REL); + if (relation_is_onerow(childrel)) + { + add_uniquekey_for_onerow(rel); + return; + } + + foreach(lc, childrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + AppendRelInfo *appinfo = find_appinfo_by_child(root, childrel->relid); + List *parent_exprs = NIL; + bool can_reuse = true; + ListCell *lc2; + foreach(lc2, ukey->exprs) + { + Var *var = (Var *)lfirst(lc2); + /* + * If the expr comes from a expression, it is hard to build the expression + * in parent so ignore that case for now. + */ + if(!IsA(var, Var)) + { + can_reuse = false; + break; + } + /* Convert it to parent var */ + parent_exprs = lappend(parent_exprs, find_parent_var(appinfo, var)); + } + if (can_reuse) + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(parent_exprs, + ukey->multi_nullvals)); + } + } + else + { + /* Check for rule 2 */ + childrel = linitial_node(RelOptInfo, childrels); + foreach(lc, childrel->indexlist) + { + IndexOptInfo *ind = lfirst(lc); + IndexOptInfo *modified_index; + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + + /* + * During simple_copy_indexinfo_to_parent, we need to convert var from + * child var to parent var, index on expression is too complex to handle. + * so ignore it for now. + */ + if (ind->indexprs != NIL) + continue; + + modified_index = simple_copy_indexinfo_to_parent(root, rel, ind); + /* + * If the unique index doesn't contain partkey, then it is unique + * on this partition only, so it is useless for us. + */ + if (!index_constains_partkey(rel, modified_index)) + continue; + + global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index); + } + + if (global_uniq_indexlist != NIL) + { + foreach(lc, childrels) + { + RelOptInfo *child = lfirst(lc); + if (is_first) + { + is_first = false; + continue; + } + adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist); + } + /* Now we have a list of unique index which are exactly same on all childrels, + * Set the UniqueKey just like it is non-partition table + */ + populate_baserel_uniquekeys(root, rel, global_uniq_indexlist); + } + } +} + + +/* + * populate_distinctrel_uniquekeys + */ +void +populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel) +{ + /* The unique key before the distinct is still valid. */ + distinctrel->uniquekeys = list_copy(inputrel->uniquekeys); + add_uniquekey_from_sortgroups(root, distinctrel, root->parse->distinctClause); +} + +/* + * populate_grouprel_uniquekeys + */ +void +populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel) + +{ + Query *parse = root->parse; + bool input_ukey_added = false; + ListCell *lc; + + if (relation_is_onerow(inputrel)) + { + add_uniquekey_for_onerow(grouprel); + return; + } + if (parse->groupingSets) + return; + + /* A Normal group by without grouping set. */ + if (parse->groupClause) + { + /* + * Current even the groupby clause is Unique already, but if query has aggref + * We have to create grouprel still. To keep the UnqiueKey short, we will check + * the UniqueKey of input_rel still valid, if so we reuse it. + */ + foreach(lc, inputrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (list_is_subset(ukey->exprs, grouprel->reltarget->exprs)) + { + grouprel->uniquekeys = lappend(grouprel->uniquekeys, + ukey); + input_ukey_added = true; + } + } + if (!input_ukey_added) + /* + * group by clause must be a super-set of grouprel->reltarget->exprs except the + * aggregation expr, so if such exprs is unique already, no bother to generate + * new uniquekey for group by exprs. + */ + add_uniquekey_from_sortgroups(root, + grouprel, + root->parse->groupClause); + } + else + /* It has aggregation but without a group by, so only one row returned */ + add_uniquekey_for_onerow(grouprel); +} + +/* + * simple_copy_uniquekeys + * Using a function for the one-line code makes us easy to check where we simply + * copied the uniquekey. + */ +void +simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel) +{ + newrel->uniquekeys = oldrel->uniquekeys; +} + +/* + * populate_unionrel_uniquekeys + */ +void +populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel) +{ + ListCell *lc; + List *exprs = NIL; + + Assert(unionrel->uniquekeys == NIL); + + foreach(lc, unionrel->reltarget->exprs) + { + exprs = lappend(exprs, lfirst(lc)); + } + + if (exprs == NIL) + /* SQL: select union select; is valid, we need to handle it here. */ + add_uniquekey_for_onerow(unionrel); + else + unionrel->uniquekeys = lappend(unionrel->uniquekeys, + makeUniqueKey(exprs,false)); + +} + +/* + * populate_joinrel_uniquekeys + * + * populate uniquekeys for joinrel. We will check each relation to see if its + * UniqueKey is still valid via innerrel_keeps_unique, if so, we add it to + * joinrel. The multi_nullvals field will be changed to true for some outer + * join cases and one-row UniqueKey needs to be converted to normal UniqueKey + * for the same case as well. + * For the uniquekey in either baserel which can't be unique after join, we still + * check to see if combination of UniqueKeys from both side is still useful for us. + * if yes, we add it to joinrel as well. + */ +void +populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *outerrel, RelOptInfo *innerrel, + List *restrictlist, JoinType jointype) +{ + ListCell *lc, *lc2; + List *clause_list = NIL; + List *outerrel_ukey_ctx; + List *innerrel_ukey_ctx; + bool inner_onerow, outer_onerow; + bool mergejoin_allowed; + + /* Care about the outerrel relation only for SEMI/ANTI join */ + if (jointype == JOIN_SEMI || jointype == JOIN_ANTI) + { + foreach(lc, outerrel->uniquekeys) + { + UniqueKey *uniquekey = lfirst_node(UniqueKey, lc); + if (list_is_subset(uniquekey->exprs, joinrel->reltarget->exprs)) + joinrel->uniquekeys = lappend(joinrel->uniquekeys, uniquekey); + } + return; + } + + Assert(jointype == JOIN_LEFT || jointype == JOIN_FULL || jointype == JOIN_INNER); + + /* Fast path */ + if (innerrel->uniquekeys == NIL || outerrel->uniquekeys == NIL) + return; + + inner_onerow = relation_is_onerow(innerrel); + outer_onerow = relation_is_onerow(outerrel); + + outerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(outerrel); + innerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(innerrel); + + clause_list = select_mergejoin_clauses(root, joinrel, outerrel, innerrel, + restrictlist, jointype, + &mergejoin_allowed); + + if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true /* reverse */)) + { + bool outer_impact = jointype == JOIN_FULL; + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + /* Outer relation has one row, and the unique key is not duplicated after join, + * the joinrel will still has one row unless the jointype == JOIN_FULL. + */ + if (outer_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (outer_onerow) + { + /* + * The onerow outerrel becomes multi rows and multi_nullvals + * will be changed to true. We also need to set the exprs correctly since it + * can't be NIL any more. + */ + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, outerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Change multi_nullvals to true due to the full join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, true)); + else + /* Just reuse it */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + } + ctx->added_to_joinrel = true; + } + } + + if (innerrel_keeps_unique(root, outerrel, innerrel, clause_list, false)) + { + bool outer_impact = jointype == JOIN_FULL || jointype == JOIN_LEFT;; + + foreach(lc, innerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + if (inner_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (inner_onerow) + { + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, innerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Need to change multi_nullvals to true due to the outer join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, + true)); + else + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + + } + ctx->added_to_joinrel = true; + } + } + + /* + * The combination of the UniqueKey from both sides is unique as well regardless + * of join type, but no bother to add it if its subset has been added to joinrel + * already or it is not useful for the joinrel. + */ + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx1 = (UniqueKeyContext) lfirst(lc); + if (ctx1->added_to_joinrel || !ctx1->useful) + continue; + foreach(lc2, innerrel_ukey_ctx) + { + UniqueKeyContext ctx2 = (UniqueKeyContext) lfirst(lc2); + if (ctx2->added_to_joinrel || !ctx2->useful) + continue; + if (add_combined_uniquekey(root, joinrel, outerrel, innerrel, + ctx1->uniquekey, ctx2->uniquekey, + jointype)) + /* If we set a onerow UniqueKey to joinrel, we don't need other. */ + return; + } + } +} + + +/* + * convert_subquery_uniquekeys + * + * Covert the UniqueKey in subquery to outer relation. + */ +void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel) +{ + ListCell *lc; + + if (sub_final_rel->uniquekeys == NIL) + return; + + if (relation_is_onerow(sub_final_rel)) + { + add_uniquekey_for_onerow(currel); + return; + } + + Assert(currel->subroot != NULL); + + foreach(lc, sub_final_rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + ListCell *lc; + List *exprs = NIL; + bool ukey_useful = true; + + /* One row case is handled above */ + Assert(ukey->exprs != NIL); + foreach(lc, ukey->exprs) + { + Var *var; + TargetEntry *tle = tlist_member(lfirst(lc), + currel->subroot->processed_tlist); + if (tle == NULL) + { + ukey_useful = false; + break; + } + var = find_var_for_subquery_tle(currel, tle); + if (var == NULL) + { + ukey_useful = false; + break; + } + exprs = lappend(exprs, var); + } + + if (ukey_useful) + currel->uniquekeys = lappend(currel->uniquekeys, + makeUniqueKey(exprs, + ukey->multi_nullvals)); + + } +} + +/* + * innerrel_keeps_unique + * + * Check if Unique key of the innerrel is valid after join. innerrel's UniqueKey + * will be still valid if innerrel's any-column mergeop outrerel's uniquekey + * exists in clause_list. + * + * Note: the clause_list must be a list of mergeable restrictinfo already. + */ +static bool +innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *clause_list, + bool reverse) +{ + ListCell *lc, *lc2, *lc3; + + if (outerrel->uniquekeys == NIL || innerrel->uniquekeys == NIL) + return false; + + /* Check if there is outerrel's uniquekey in mergeable clause. */ + foreach(lc, outerrel->uniquekeys) + { + List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs; + bool clauselist_matchs_all_exprs = true; + foreach(lc2, outer_uq_exprs) + { + Node *outer_uq_expr = lfirst(lc2); + bool find_uq_expr_in_clauselist = false; + foreach(lc3, clause_list) + { + RestrictInfo *rinfo = lfirst_node(RestrictInfo, lc3); + Node *outer_expr; + if (reverse) + outer_expr = rinfo->outer_is_left ? get_rightop(rinfo->clause) : get_leftop(rinfo->clause); + else + outer_expr = rinfo->outer_is_left ? get_leftop(rinfo->clause) : get_rightop(rinfo->clause); + if (equal(outer_expr, outer_uq_expr)) + { + find_uq_expr_in_clauselist = true; + break; + } + } + if (!find_uq_expr_in_clauselist) + { + /* No need to check the next exprs in the current uniquekey */ + clauselist_matchs_all_exprs = false; + break; + } + } + + if (clauselist_matchs_all_exprs) + return true; + } + return false; +} + + +/* + * relation_is_onerow + * Check if it is a one-row relation by checking UniqueKey. + */ +bool +relation_is_onerow(RelOptInfo *rel) +{ + UniqueKey *ukey; + if (rel->uniquekeys == NIL) + return false; + ukey = linitial_node(UniqueKey, rel->uniquekeys); + return ukey->exprs == NIL && list_length(rel->uniquekeys) == 1; +} + +/* + * relation_has_uniquekeys_for + * Returns true if we have proofs that 'rel' cannot return multiple rows with + * the same values in each of 'exprs'. Otherwise returns false. + */ +bool +relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel, + List *exprs, bool allow_multinulls) +{ + ListCell *lc; + + /* + * For UniqueKey->onerow case, the uniquekey->exprs is empty as well + * so we can't rely on list_is_subset to handle this special cases + */ + if (exprs == NIL) + return false; + + foreach(lc, rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (ukey->multi_nullvals && !allow_multinulls) + continue; + if (list_is_subset(ukey->exprs, exprs)) + return true; + } + return false; +} + + +/* + * get_exprs_from_uniqueindex + * + * Return a list of exprs which is unique. set useful to false if this + * unique index is not useful for us. + */ +static List * +get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals) +{ + List *exprs = NIL; + ListCell *indexpr_item; + int c = 0; + + *useful = true; + *multi_nullvals = false; + + indexpr_item = list_head(unique_index->indexprs); + for(c = 0; c < unique_index->ncolumns; c++) + { + int attr = unique_index->indexkeys[c]; + Expr *expr; + bool matched_const = false; + ListCell *lc1, *lc2; + + if(attr > 0) + { + expr = list_nth_node(TargetEntry, unique_index->indextlist, c)->expr; + } + else if (attr == 0) + { + /* Expression index */ + expr = lfirst(indexpr_item); + indexpr_item = lnext(unique_index->indexprs, indexpr_item); + } + else /* attr < 0 */ + { + /* Index on system column is not supported */ + Assert(false); + } + + /* + * Check index_col = Const case with regarding to opfamily checking + * If we can remove the index_col from the final UniqueKey->exprs. + */ + forboth(lc1, const_exprs, lc2, const_expr_opfamilies) + { + if (list_member_oid((List *)lfirst(lc2), unique_index->opfamily[c]) + && match_index_to_operand((Node *) lfirst(lc1), c, unique_index)) + { + matched_const = true; + break; + } + } + + if (matched_const) + continue; + + /* Check if the indexed expr is used in rel */ + if (attr > 0) + { + /* + * Normal Indexed column, if the col is not used, then the index is useless + * for uniquekey. + */ + attr -= FirstLowInvalidHeapAttributeNumber; + + if (!bms_is_member(attr, used_varattrs)) + { + *useful = false; + break; + } + } + else if (!list_member(unique_index->rel->reltarget->exprs, expr)) + { + /* Expression index but the expression is not used in rel */ + *useful = false; + break; + } + + /* check not null property. */ + if (attr == 0) + { + /* We never know if a expression yields null or not */ + *multi_nullvals = true; + } + else if (!bms_is_member(attr, unique_index->rel->notnullattrs) + && !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber, + unique_index->rel->notnullattrs)) + { + *multi_nullvals = true; + } + + exprs = lappend(exprs, expr); + } + return exprs; +} + + +/* + * add_uniquekey_for_onerow + * If we are sure that the relation only returns one row, then all the columns + * are unique. However we don't need to create UniqueKey for every column, we + * just set exprs = NIL and overwrites all the other UniqueKey on this RelOptInfo + * since this one has strongest semantics. + */ +void +add_uniquekey_for_onerow(RelOptInfo *rel) +{ + /* + * We overwrite the previous UniqueKey on purpose since this one has the + * strongest semantic. + */ + rel->uniquekeys = list_make1(makeUniqueKey(NIL, false)); +} + + +/* + * initililze_uniquecontext_for_joinrel + * Return a List of UniqueKeyContext for an inputrel + */ +static List * +initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel) +{ + List *res = NIL; + ListCell *lc; + foreach(lc, inputrel->uniquekeys) + { + UniqueKeyContext context; + context = palloc(sizeof(struct UniqueKeyContextData)); + context->uniquekey = lfirst_node(UniqueKey, lc); + context->added_to_joinrel = false; + context->useful = true; + res = lappend(res, context); + } + return res; +} + + +/* + * get_exprs_from_uniquekey + * Unify the way of get List of exprs from a one-row UniqueKey or + * normal UniqueKey. for the onerow case, every expr in rel1 is a valid + * UniqueKey. Return a List of exprs. + * + * rel1: The relation which you want to get the exprs. + * ukey: The UniqueKey you want to get the exprs. + */ +static List * +get_exprs_from_uniquekey(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *rel1, UniqueKey *ukey) +{ + ListCell *lc; + bool onerow = rel1 != NULL && relation_is_onerow(rel1); + + List *res = NIL; + Assert(onerow || ukey); + if (onerow) + { + /* Only cares about the exprs still exist in joinrel */ + foreach(lc, joinrel->reltarget->exprs) + { + Bitmapset *relids = pull_varnos(root, lfirst(lc)); + if (bms_is_subset(relids, rel1->relids)) + { + res = lappend(res, list_make1(lfirst(lc))); + } + } + } + else + { + res = list_make1(ukey->exprs); + } + return res; +} + +/* + * Partitioned table Unique Keys. + * The partition table unique key is maintained as: + * 1. The index must be unique as usual. + * 2. The index must contains partition key. + * 3. The index must exist on all the child rel. see simple_indexinfo_equal for + * how we compare it. + */ + +/* + * index_constains_partkey + * return true if the index contains the partiton key. + */ +static bool +index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind) +{ + ListCell *lc; + int i; + Assert(IS_PARTITIONED_REL(partrel)); + Assert(partrel->part_scheme->partnatts > 0); + + for(i = 0; i < partrel->part_scheme->partnatts; i++) + { + Node *part_expr = linitial(partrel->partexprs[i]); + bool found_in_index = false; + foreach(lc, ind->indextlist) + { + Expr *index_expr = lfirst_node(TargetEntry, lc)->expr; + if (equal(index_expr, part_expr)) + { + found_in_index = true; + break; + } + } + if (!found_in_index) + return false; + } + return true; +} + +/* + * simple_indexinfo_equal + * + * Used to check if the 2 index is same as each other. The index here + * is COPIED from childrel and did some tiny changes(see + * simple_copy_indexinfo_to_parent) + */ +static bool +simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2) +{ + Size oid_cmp_len = sizeof(Oid) * ind1->ncolumns; + + return ind1->ncolumns == ind2->ncolumns && + ind1->unique == ind2->unique && + memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0 && + memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0 && + memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0 && + memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0 && + equal(get_tlist_exprs(ind1->indextlist, true), + get_tlist_exprs(ind2->indextlist, true)); +} + + +/* + * The below macros are used for simple_copy_indexinfo_to_parent which is so + * customized that I don't want to put it to copyfuncs.c. So copy it here. + */ +#define COPY_POINTER_FIELD(fldname, sz) \ + do { \ + Size _size = (sz); \ + newnode->fldname = palloc(_size); \ + memcpy(newnode->fldname, from->fldname, _size); \ + } while (0) + +#define COPY_NODE_FIELD(fldname) \ + (newnode->fldname = copyObjectImpl(from->fldname)) + +#define COPY_SCALAR_FIELD(fldname) \ + (newnode->fldname = from->fldname) + + +/* + * simple_copy_indexinfo_to_parent (from partition) + * Copy the IndexInfo from child relation to parent relation with some modification, + * which is used to test: + * 1. If the same index exists in all the childrels. + * 2. If the parentrel->reltarget/basicrestrict info matches this index. + */ +static IndexOptInfo * +simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from) +{ + IndexOptInfo *newnode = makeNode(IndexOptInfo); + AppendRelInfo *appinfo = find_appinfo_by_child(root, from->rel->relid); + ListCell *lc; + int idx = 0; + + COPY_SCALAR_FIELD(ncolumns); + COPY_SCALAR_FIELD(nkeycolumns); + COPY_SCALAR_FIELD(unique); + COPY_SCALAR_FIELD(immediate); + /* We just need to know if it is NIL or not */ + COPY_SCALAR_FIELD(indpred); + COPY_SCALAR_FIELD(predOK); + COPY_POINTER_FIELD(indexkeys, from->ncolumns * sizeof(int)); + COPY_POINTER_FIELD(indexcollations, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opfamily, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opcintype, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(sortopfamily, from->ncolumns * sizeof(Oid)); + COPY_NODE_FIELD(indextlist); + + /* Convert index exprs on child expr to expr on parent */ + foreach(lc, newnode->indextlist) + { + TargetEntry *tle = lfirst_node(TargetEntry, lc); + /* Index on expression is ignored */ + Assert(IsA(tle->expr, Var)); + tle->expr = (Expr *) find_parent_var(appinfo, (Var *) tle->expr); + newnode->indexkeys[idx] = castNode(Var, tle->expr)->varattno; + idx++; + } + newnode->rel = parentrel; + return newnode; +} + +/* + * adjust_partition_unique_indexlist + * + * global_unique_indexes: At the beginning, it contains the copy & modified + * unique index from the first partition. And then check if each index in it still + * exists in the following partitions. If no, remove it. at last, it has an + * index list which exists in all the partitions. + */ +static void +adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_indexes) +{ + ListCell *lc, *lc2; + foreach(lc, *global_unique_indexes) + { + IndexOptInfo *g_ind = lfirst_node(IndexOptInfo, lc); + bool found_in_child = false; + + foreach(lc2, childrel->indexlist) + { + IndexOptInfo *p_ind = lfirst_node(IndexOptInfo, lc2); + IndexOptInfo *p_ind_copy; + if (!p_ind->unique || !p_ind->immediate || + (p_ind->indpred != NIL && !p_ind->predOK)) + continue; + p_ind_copy = simple_copy_indexinfo_to_parent(root, parentrel, p_ind); + if (simple_indexinfo_equal(p_ind_copy, g_ind)) + { + found_in_child = true; + break; + } + } + if (!found_in_child) + /* The index doesn't exist in childrel, remove it from global_unique_indexes */ + *global_unique_indexes = foreach_delete_current(*global_unique_indexes, lc); + } +} + +/* Helper function for groupres/distinctrel */ +static void +add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgroups) +{ + Query *parse = root->parse; + List *exprs; + + /* + * XXX: If there are some vars which is not in current levelsup, the semantic is + * imprecise, should we avoid it or not? levelsup = 1 is just a demo, maybe we need to + * check every level other than 0, if so, looks we have to write another + * pull_var_walker. + */ + List *upper_vars = pull_vars_of_level((Node*)sortgroups, 1); + + if (upper_vars != NIL) + return; + + exprs = get_sortgrouplist_exprs(sortgroups, parse->targetList); + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(exprs, + false /* sortgroupclause can't be multi_nullvals */)); +} + + +/* + * add_combined_uniquekey + * The combination of both UniqueKeys is a valid UniqueKey for joinrel no matter + * the jointype. + */ +bool +add_combined_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype) +{ + + ListCell *lc1, *lc2; + + /* Either side has multi_nullvals or we have outer join, + * the combined UniqueKey has multi_nullvals */ + bool multi_nullvals = outer_ukey->multi_nullvals || + inner_ukey->multi_nullvals || IS_OUTER_JOIN(jointype); + + /* The only case we can get onerow joinrel after join */ + if (relation_is_onerow(outer_rel) + && relation_is_onerow(inner_rel) + && jointype == JOIN_INNER) + { + add_uniquekey_for_onerow(joinrel); + return true; + } + + foreach(lc1, get_exprs_from_uniquekey(root, joinrel, outer_rel, outer_ukey)) + { + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, inner_rel, inner_ukey)) + { + List *exprs = list_concat_copy(lfirst_node(List, lc1), lfirst_node(List, lc2)); + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(exprs, + multi_nullvals)); + } + } + return false; +} diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index 424d25cbd5..8d8e493f5c 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -2387,6 +2387,8 @@ grouping_planner(PlannerInfo *root, bool inheritance_update, add_path(final_rel, path); } + simple_copy_uniquekeys(current_rel, final_rel); + /* * Generate partial paths for final_rel, too, if outer query levels might * be able to make use of them. @@ -3899,6 +3901,8 @@ create_grouping_paths(PlannerInfo *root, } set_cheapest(grouped_rel); + + populate_grouprel_uniquekeys(root, grouped_rel, input_rel); return grouped_rel; } @@ -4621,7 +4625,7 @@ create_window_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(window_rel); - + simple_copy_uniquekeys(input_rel, window_rel); return window_rel; } @@ -4935,7 +4939,7 @@ create_distinct_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(distinct_rel); - + populate_distinctrel_uniquekeys(root, input_rel, distinct_rel); return distinct_rel; } @@ -5196,6 +5200,8 @@ create_ordered_paths(PlannerInfo *root, */ Assert(ordered_rel->pathlist != NIL); + simple_copy_uniquekeys(input_rel, ordered_rel); + return ordered_rel; } @@ -6073,6 +6079,9 @@ adjust_paths_for_srfs(PlannerInfo *root, RelOptInfo *rel, if (list_length(targets) == 1) return; + /* UniqueKey is not valid after handling the SRF. */ + rel->uniquekeys = NIL; + /* * Stack SRF-evaluation nodes atop each path for the rel. * diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c index becdcbb872..b7626545bf 100644 --- a/src/backend/optimizer/prep/prepunion.c +++ b/src/backend/optimizer/prep/prepunion.c @@ -689,6 +689,8 @@ generate_union_paths(SetOperationStmt *op, PlannerInfo *root, /* Undo effects of possibly forcing tuple_fraction to 0 */ root->tuple_fraction = save_fraction; + /* Add the UniqueKeys */ + populate_unionrel_uniquekeys(root, result_rel); return result_rel; } diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c index 86922a273c..6817c9c787 100644 --- a/src/backend/optimizer/util/appendinfo.c +++ b/src/backend/optimizer/util/appendinfo.c @@ -746,3 +746,47 @@ find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos) } return appinfos; } + +/* + * find_appinfo_by_child + * + */ +AppendRelInfo * +find_appinfo_by_child(PlannerInfo *root, Index child_index) +{ + ListCell *lc; + foreach(lc, root->append_rel_list) + { + AppendRelInfo *appinfo = lfirst_node(AppendRelInfo, lc); + if (appinfo->child_relid == child_index) + return appinfo; + } + elog(ERROR, "parent relation cant be found"); + return NULL; +} + +/* + * find_parent_var + * + */ +Var * +find_parent_var(AppendRelInfo *appinfo, Var *child_var) +{ + ListCell *lc; + Var *res = NULL; + Index attno = 1; + foreach(lc, appinfo->translated_vars) + { + Node *child_node = lfirst(lc); + if (equal(child_node, child_var)) + { + res = copyObject(child_var); + res->varattno = attno; + res->varno = appinfo->parent_relid; + } + attno++; + } + if (res == NULL) + elog(ERROR, "parent var can't be found."); + return res; +} diff --git a/src/backend/optimizer/util/inherit.c b/src/backend/optimizer/util/inherit.c index be1c9ddd96..3eec1f4d74 100644 --- a/src/backend/optimizer/util/inherit.c +++ b/src/backend/optimizer/util/inherit.c @@ -736,6 +736,7 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, { Node *onecq = (Node *) lfirst(lc2); bool pseudoconstant; + RestrictInfo *child_rinfo; /* check for pseudoconstant (no Vars or volatile functions) */ pseudoconstant = @@ -747,14 +748,15 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, root->hasPseudoConstantQuals = true; } /* reconstitute RestrictInfo with appropriate properties */ - childquals = lappend(childquals, - make_restrictinfo(root, - (Expr *) onecq, - rinfo->is_pushed_down, - rinfo->outerjoin_delayed, - pseudoconstant, - rinfo->security_level, - NULL, NULL, NULL)); + child_rinfo = make_restrictinfo(root, + (Expr *) onecq, + rinfo->is_pushed_down, + rinfo->outerjoin_delayed, + pseudoconstant, + rinfo->security_level, + NULL, NULL, NULL); + child_rinfo->mergeopfamilies = rinfo->mergeopfamilies; + childquals = lappend(childquals, child_rinfo); /* track minimum security level among child quals */ cq_min_security = Min(cq_min_security, rinfo->security_level); } diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 48a7ebfe45..4fe1824eb0 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -16,6 +16,7 @@ #include "nodes/execnodes.h" #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" extern A_Expr *makeA_Expr(A_Expr_Kind kind, List *name, @@ -106,4 +107,6 @@ extern GroupingSet *makeGroupingSet(GroupingSetKind kind, List *content, int loc extern VacuumRelation *makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols); +extern UniqueKey* makeUniqueKey(List *exprs, bool multi_nullvals); + #endif /* MAKEFUNC_H */ diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index e22df890ef..3320273ac1 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -265,6 +265,7 @@ typedef enum NodeTag T_EquivalenceMember, T_PathKey, T_PathTarget, + T_UniqueKey, T_RestrictInfo, T_IndexClause, T_PlaceHolderVar, diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index 0d61f04d27..80561b1003 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -739,6 +739,7 @@ typedef struct RelOptInfo QualCost baserestrictcost; /* cost of evaluating the above */ Index baserestrict_min_security; /* min security_level found in * baserestrictinfo */ + List *uniquekeys; /* List of UniqueKey */ List *joininfo; /* RestrictInfo structures for join clauses * involving this rel */ bool has_eclass_joins; /* T means joininfo is incomplete */ @@ -1059,6 +1060,28 @@ typedef struct PathKey } PathKey; +/* + * UniqueKey + * + * Represents the unique properties held by a RelOptInfo. + * + * exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL + * is a special case of UniqueKey, which means there is only 1 row in that + * relation. + * multi_nullvals: true means multi null values may exist in these exprs, so the + * uniqueness is not guaranteed in this case. This field is necessary for + * remove_useless_join & reduce_unique_semijoins where we don't mind these + * duplicated NULL values. It is set to true for 2 cases. One is a unique key + * from a unique index but the related column is nullable. The other one is for + * outer join. see populate_joinrel_uniquekeys for detail. + */ +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + /* * PathTarget * @@ -2496,7 +2519,7 @@ typedef enum * * flags indicating what kinds of grouping are possible. * partial_costs_set is true if the agg_partial_costs and agg_final_costs - * have been initialized. + * have been initialized. * agg_partial_costs gives partial aggregation costs. * agg_final_costs gives finalization costs. * target_parallel_safe is true if target is parallel safe. @@ -2526,8 +2549,8 @@ typedef struct * limit_tuples is an estimated bound on the number of output tuples, * or -1 if no LIMIT or couldn't estimate. * count_est and offset_est are the estimated values of the LIMIT and OFFSET - * expressions computed by preprocess_limit() (see comments for - * preprocess_limit() for more information). + * expressions computed by preprocess_limit() (see comments for + * preprocess_limit() for more information). */ typedef struct { diff --git a/src/include/nodes/pg_list.h b/src/include/nodes/pg_list.h index 404e03f132..85c65e910c 100644 --- a/src/include/nodes/pg_list.h +++ b/src/include/nodes/pg_list.h @@ -566,6 +566,16 @@ extern pg_nodiscard List *list_delete_first(List *list); extern pg_nodiscard List *list_delete_last(List *list); extern pg_nodiscard List *list_delete_nth_cell(List *list, int n); extern pg_nodiscard List *list_delete_cell(List *list, ListCell *cell); +extern bool list_is_subset(const List *members, const List *target); + +extern List *list_delete(List *list, void *datum); +extern List *list_delete_ptr(List *list, void *datum); +extern List *list_delete_int(List *list, int datum); +extern List *list_delete_oid(List *list, Oid datum); +extern List *list_delete_first(List *list); +extern List *list_delete_last(List *list); +extern List *list_delete_nth_cell(List *list, int n); +extern List *list_delete_cell(List *list, ListCell *cell); extern List *list_union(const List *list1, const List *list2); extern List *list_union_ptr(const List *list1, const List *list2); diff --git a/src/include/optimizer/appendinfo.h b/src/include/optimizer/appendinfo.h index 4cbf8c26cc..43b8d05f69 100644 --- a/src/include/optimizer/appendinfo.h +++ b/src/include/optimizer/appendinfo.h @@ -32,4 +32,7 @@ extern Relids adjust_child_relids_multilevel(PlannerInfo *root, Relids relids, extern AppendRelInfo **find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos); +extern AppendRelInfo *find_appinfo_by_child(PlannerInfo *root, Index child_index); +extern Var *find_parent_var(AppendRelInfo *appinfo, Var *child_var); + #endif /* APPENDINFO_H */ diff --git a/src/include/optimizer/optimizer.h b/src/include/optimizer/optimizer.h index d587952b7d..843aafa51e 100644 --- a/src/include/optimizer/optimizer.h +++ b/src/include/optimizer/optimizer.h @@ -23,6 +23,7 @@ #define OPTIMIZER_H #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" /* Test if an expression node represents a SRF call. Beware multiple eval! */ #define IS_SRF_CALL(node) \ @@ -168,6 +169,7 @@ extern TargetEntry *get_sortgroupref_tle(Index sortref, List *targetList); extern TargetEntry *get_sortgroupclause_tle(SortGroupClause *sgClause, List *targetList); +extern Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); extern Node *get_sortgroupclause_expr(SortGroupClause *sgClause, List *targetList); extern List *get_sortgrouplist_exprs(List *sgClauses, diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h index 035d3e1206..1adf99c9ee 100644 --- a/src/include/optimizer/paths.h +++ b/src/include/optimizer/paths.h @@ -247,5 +247,48 @@ extern PathKey *make_canonical_pathkey(PlannerInfo *root, int strategy, bool nulls_first); extern void add_paths_to_append_rel(PlannerInfo *root, RelOptInfo *rel, List *live_childrels); +extern List *select_mergejoin_clauses(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + JoinType jointype, + bool *mergejoin_allowed); + +/* + * uniquekeys.c + * Utilities for matching and building unique keys + */ +extern void populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List* unique_index_list); +extern void populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels); +extern void populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel); +extern void populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel); +extern void populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel); +extern void simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel); +extern void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel); +extern void populate_joinrel_uniquekeys(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + RelOptInfo *rel2, + List *restrictlist, + JoinType jointype); + +extern bool relation_has_uniquekeys_for(PlannerInfo *root, + RelOptInfo *rel, + List *exprs, + bool allow_multinulls); +extern bool relation_is_onerow(RelOptInfo *rel); #endif /* PATHS_H */ -- 2.26.2 --yh7cw345mbttadjz Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v38-0003-Extend-UniqueKeys.patch" ^ permalink raw reply [nested|flat] 91+ messages in thread
* [PATCH v36 2/6] Introduce UniqueKey attributes on RelOptInfo struct. @ 2020-05-11 07:50 一挃 <[email protected]> 0 siblings, 0 replies; 91+ messages in thread From: 一挃 @ 2020-05-11 07:50 UTC (permalink / raw) UniqueKey is a set of exprs on RelOptInfo which represents the exprs will be unique on the given RelOptInfo. You can see README.uniquekey for more information. --- src/backend/nodes/copyfuncs.c | 13 + src/backend/nodes/list.c | 31 + src/backend/nodes/makefuncs.c | 13 + src/backend/nodes/outfuncs.c | 11 + src/backend/nodes/readfuncs.c | 10 + src/backend/optimizer/path/Makefile | 3 +- src/backend/optimizer/path/README.uniquekey | 131 +++ src/backend/optimizer/path/allpaths.c | 10 + src/backend/optimizer/path/joinpath.c | 9 +- src/backend/optimizer/path/joinrels.c | 2 + src/backend/optimizer/path/pathkeys.c | 3 +- src/backend/optimizer/path/uniquekeys.c | 1131 +++++++++++++++++++ src/backend/optimizer/plan/planner.c | 13 +- src/backend/optimizer/prep/prepunion.c | 2 + src/backend/optimizer/util/appendinfo.c | 44 + src/backend/optimizer/util/inherit.c | 16 +- src/include/nodes/makefuncs.h | 3 + src/include/nodes/nodes.h | 1 + src/include/nodes/pathnodes.h | 29 +- src/include/nodes/pg_list.h | 2 + src/include/optimizer/appendinfo.h | 3 + src/include/optimizer/optimizer.h | 2 + src/include/optimizer/paths.h | 43 + 23 files changed, 1502 insertions(+), 23 deletions(-) create mode 100644 src/backend/optimizer/path/README.uniquekey create mode 100644 src/backend/optimizer/path/uniquekeys.c diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 89c409de66..1f50400fd2 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2273,6 +2273,16 @@ _copyPathKey(const PathKey *from) return newnode; } +static UniqueKey * +_copyUniqueKey(const UniqueKey *from) +{ + UniqueKey *newnode = makeNode(UniqueKey); + + COPY_NODE_FIELD(exprs); + COPY_SCALAR_FIELD(multi_nullvals); + + return newnode; +} /* * _copyRestrictInfo */ @@ -5152,6 +5162,9 @@ copyObjectImpl(const void *from) case T_PathKey: retval = _copyPathKey(from); break; + case T_UniqueKey: + retval = _copyUniqueKey(from); + break; case T_RestrictInfo: retval = _copyRestrictInfo(from); break; diff --git a/src/backend/nodes/list.c b/src/backend/nodes/list.c index 80fa8c84e4..a7a99b70f2 100644 --- a/src/backend/nodes/list.c +++ b/src/backend/nodes/list.c @@ -687,6 +687,37 @@ list_member_oid(const List *list, Oid datum) return false; } +/* + * return true iff every entry in "members" list is also present + * in the "target" list. + */ +bool +list_is_subset(const List *members, const List *target) +{ + const ListCell *lc1, *lc2; + + Assert(IsPointerList(members)); + Assert(IsPointerList(target)); + check_list_invariants(members); + check_list_invariants(target); + + foreach(lc1, members) + { + bool found = false; + foreach(lc2, target) + { + if (equal(lfirst(lc1), lfirst(lc2))) + { + found = true; + break; + } + } + if (!found) + return false; + } + return true; +} + /* * Delete the n'th cell (counting from 0) in list. * diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 49de285f01..646cf7c9a1 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -814,3 +814,16 @@ makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols) v->va_cols = va_cols; return v; } + + +/* + * makeUniqueKey + */ +UniqueKey* +makeUniqueKey(List *exprs, bool multi_nullvals) +{ + UniqueKey * ukey = makeNode(UniqueKey); + ukey->exprs = exprs; + ukey->multi_nullvals = multi_nullvals; + return ukey; +} diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index e2f177515d..c3a9632992 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2428,6 +2428,14 @@ _outPathKey(StringInfo str, const PathKey *node) WRITE_BOOL_FIELD(pk_nulls_first); } +static void +_outUniqueKey(StringInfo str, const UniqueKey *node) +{ + WRITE_NODE_TYPE("UNIQUEKEY"); + WRITE_NODE_FIELD(exprs); + WRITE_BOOL_FIELD(multi_nullvals); +} + static void _outPathTarget(StringInfo str, const PathTarget *node) { @@ -4127,6 +4135,9 @@ outNode(StringInfo str, const void *obj) case T_PathKey: _outPathKey(str, obj); break; + case T_UniqueKey: + _outUniqueKey(str, obj); + break; case T_PathTarget: _outPathTarget(str, obj); break; diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index 42050ab719..3a18571d0c 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -452,6 +452,14 @@ _readSetOperationStmt(void) READ_DONE(); } +static UniqueKey * +_readUniqueKey(void) +{ + READ_LOCALS(UniqueKey); + READ_NODE_FIELD(exprs); + READ_BOOL_FIELD(multi_nullvals); + READ_DONE(); +} /* * Stuff from primnodes.h. @@ -2656,6 +2664,8 @@ parseNodeString(void) return_value = _readCommonTableExpr(); else if (MATCH("SETOPERATIONSTMT", 16)) return_value = _readSetOperationStmt(); + else if (MATCH("UNIQUEKEY", 9)) + return_value = _readUniqueKey(); else if (MATCH("ALIAS", 5)) return_value = _readAlias(); else if (MATCH("RANGEVAR", 8)) diff --git a/src/backend/optimizer/path/Makefile b/src/backend/optimizer/path/Makefile index 1e199ff66f..7b9820c25f 100644 --- a/src/backend/optimizer/path/Makefile +++ b/src/backend/optimizer/path/Makefile @@ -21,6 +21,7 @@ OBJS = \ joinpath.o \ joinrels.o \ pathkeys.o \ - tidpath.o + tidpath.o \ + uniquekeys.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/optimizer/path/README.uniquekey b/src/backend/optimizer/path/README.uniquekey new file mode 100644 index 0000000000..5eac761995 --- /dev/null +++ b/src/backend/optimizer/path/README.uniquekey @@ -0,0 +1,131 @@ +1. What is UniqueKey? +We can think UniqueKey is a set of exprs for a RelOptInfo, which we are insure +that doesn't yields same result among all the rows. The simplest UniqueKey +format is primary key. + +However we define the UnqiueKey as below. + +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + +exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL +is a special case of UniqueKey, which means there is only one row in that +relation.it has a stronger semantic than others. like SELECT uk FROM t; uk is +normal unique key and may have different values. SELECT colx FROM t WHERE uk = +const. colx is unique AND we have only 1 value. This field can used for +innerrel_is_unique. this logic is handled specially in add_uniquekey_for_onerow +function. + +multi_nullvals: true means multi null values may exist in these exprs, so the +uniqueness is not guaranteed in this case. This field is necessary for +remove_useless_join & reduce_unique_semijoins where we don't mind these +duplicated NULL values. It is set to true for 2 cases. One is a unique key +from a unique index but the related column is nullable. The other one is for +outer join. see populate_joinrel_uniquekeys for detail. + + +The UniqueKey can be used at the following cases at least: +1. remove_useless_joins. +2. reduce_semianti_joins +3. remove distinct node if distinct clause is unique. +4. remove aggnode if group by clause is unique. +5. Index Skip Scan (WIP) +6. Aggregation Push Down without 2 phase aggregation if the join can't + duplicated the aggregated rows. (work in progress feature) + +2. How is it maintained? + +We have a set of populate_xxx_unqiuekeys functions to maintain the uniquekey on +various cases. xxx includes baserel, joinrel, partitionedrel, distinctrel, +groupedrel, unionrel. and we also need to convert the uniquekey from subquery +to outer relation, which is what convert_subquery_uniquekeys does. + +1. The first part is about baserel. We handled 3 cases. suppose we have Unique +Index on (a, b). + +1. SELECT a, b FROM t. UniqueKey (a, b) +2. SELECT a FROM t WHERE b = 1; UniqueKey (a) +3. SELECT .. FROM t WHERE a = 1 AND b = 1; UniqueKey (NIL). onerow case, every + column is Unique. + +2. The next part is joinrel, this part is most error-prone, we simplified the rules +like below: +1. If the relation's UniqueKey can't be duplicated after join, then is will be + still valid for the join rel. The function we used here is + innerrel_keeps_unique. The basic idea is innerrel.any_col = outer.uk. + +2. If the UnqiueKey can't keep valid via the rule 1, the combination of the + UniqueKey from both sides are valid for sure. We can prove this as: if the + unique exprs from rel1 is duplicated by rel2, the duplicated rows must + contains different unique exprs from rel2. + +More considerations about onerow: +1. If relation with one row and it can't be duplicated, it is still possible + contains mulit_nullvas after outer join. +2. If the either UniqueKey can be duplicated after join, the can get one row + only when both side is one row AND there is no outer join. +3. Whenever the onerow UniqueKey is not a valid any more, we need to convert one + row UniqueKey to normal unique key since we don't store exprs for one-row + relation. get_exprs_from_uniquekeys will be used here. + + +More considerations about multi_nullvals after join: +1. If the original UnqiueKey has multi_nullvals, the final UniqueKey will have + mulit_nullvals in any case. +2. If a unique key doesn't allow mulit_nullvals, after some outer join, it + allows some outer join. + + +3. When we comes to subquery, we need to convert_subquery_unqiuekeys just like +convert_subquery_pathkeys. Only the UniqueKey insides subquery is referenced as +a Var in outer relation will be reused. The relationship between the outerrel.Var +and subquery.exprs is built with outerel->subroot->processed_tlist. + + +4. As for the SRF functions, it will break the uniqueness of uniquekey, However it +is handled in adjust_paths_for_srfs, which happens after the query_planner. so +we will maintain the UniqueKey until there and reset it to NIL at that +places. This can't help on distinct/group by elimination cases but probably help +in some other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is +semantic correctly. + + +5. As for inherit table, we first main the UnqiueKey on childrel as well. But for +partitioned table we need to maintain 2 different kinds of +UnqiueKey. 1). UniqueKey on the parent relation 2). UniqueKey on child +relation for partition wise query. + +Example: +CREATE TABLE p (a int not null, b int not null) partition by list (a); +CREATE TABLE p0 partition of p for values in (1); +CREATE TABLE p1 partition of p for values in (2); + +create unique index p0_b on p0(b); +create unique index p1_b on p1(b); + +Now b is only unique on partition level, so the distinct can't be removed on +the following cases. SELECT DISTINCT b FROM p; + +Another example is SELECT DISTINCT a, b FROM p WHERE a = 1; Since only one +partition is chosen, the UniqueKey on child relation is same as the UniqueKey on +parent relation. + +Another usage of UniqueKey on partition level is it be helpful for +partition-wise join. + +As for the UniqueKey on parent table level, it comes with 2 different ways, +1). the UniqueKey is also derived in UniqueKey index, but the index must be same +in all the related children relations and the unique index must contains +Partition Key in it. Example: + +CREATE UNIQUE INDEX p_ab ON p(a, b); -- where a is the partition key. + +-- Query +SELECT a, b FROM p; the (a, b) is a UniqueKey of p. + +2). If the parent relation has only one childrel, the UniqueKey on childrel is + the UniqueKey on parent as well. diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index 484dab0a1a..2ad9d06d7a 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -579,6 +579,12 @@ set_plain_rel_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) */ check_index_predicates(root, rel); + /* + * Now that we've marked which partial indexes are suitable, we can now + * build the relation's unique keys. + */ + populate_baserel_uniquekeys(root, rel, rel->indexlist); + /* Mark rel with estimated output rows, width, etc */ set_baserel_size_estimates(root, rel); } @@ -1310,6 +1316,8 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, /* Add paths to the append relation. */ add_paths_to_append_rel(root, rel, live_childrels); + if (IS_PARTITIONED_REL(rel)) + populate_partitionedrel_uniquekeys(root, rel, live_childrels); } @@ -2383,6 +2391,8 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, pathkeys, required_outer)); } + convert_subquery_uniquekeys(root, rel, sub_final_rel); + /* If outer rel allows parallelism, do same for partial paths. */ if (rel->consider_parallel && bms_is_empty(required_outer)) { diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c index db54a6ba2e..ef0fd2fb0b 100644 --- a/src/backend/optimizer/path/joinpath.c +++ b/src/backend/optimizer/path/joinpath.c @@ -71,13 +71,6 @@ static void consider_parallel_mergejoin(PlannerInfo *root, static void hash_inner_and_outer(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, RelOptInfo *innerrel, JoinType jointype, JoinPathExtraData *extra); -static List *select_mergejoin_clauses(PlannerInfo *root, - RelOptInfo *joinrel, - RelOptInfo *outerrel, - RelOptInfo *innerrel, - List *restrictlist, - JoinType jointype, - bool *mergejoin_allowed); static void generate_mergejoin_paths(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *innerrel, @@ -1927,7 +1920,7 @@ hash_inner_and_outer(PlannerInfo *root, * if it is mergejoinable and involves vars from the two sub-relations * currently of interest. */ -static List * +List * select_mergejoin_clauses(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c index 2d343cd293..b9163ee8ff 100644 --- a/src/backend/optimizer/path/joinrels.c +++ b/src/backend/optimizer/path/joinrels.c @@ -924,6 +924,8 @@ populate_joinrel_with_paths(PlannerInfo *root, RelOptInfo *rel1, /* Apply partitionwise join technique, if possible. */ try_partitionwise_join(root, rel1, rel2, joinrel, sjinfo, restrictlist); + + populate_joinrel_uniquekeys(root, joinrel, rel1, rel2, restrictlist, sjinfo->jointype); } diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index ce9bf87e9b..7e596d4194 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -33,7 +33,6 @@ static bool pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys); static bool matches_boolean_partition_clause(RestrictInfo *rinfo, RelOptInfo *partrel, int partkeycol); -static Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); static bool right_merge_direction(PlannerInfo *root, PathKey *pathkey); @@ -1035,7 +1034,7 @@ convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel, * We need this to ensure that we don't return pathkeys describing values * that are unavailable above the level of the subquery scan. */ -static Var * +Var * find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle) { ListCell *lc; diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c new file mode 100644 index 0000000000..b33bcd2f32 --- /dev/null +++ b/src/backend/optimizer/path/uniquekeys.c @@ -0,0 +1,1131 @@ +/*------------------------------------------------------------------------- + * + * uniquekeys.c + * Utilities for matching and building unique keys + * + * Portions Copyright (c) 2020, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/optimizer/path/uniquekeys.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "nodes/makefuncs.h" +#include "nodes/nodeFuncs.h" +#include "optimizer/pathnode.h" +#include "optimizer/paths.h" +#include "optimizer/appendinfo.h" +#include "optimizer/optimizer.h" +#include "optimizer/tlist.h" +#include "rewrite/rewriteManip.h" + + +/* + * This struct is used to help populate_joinrel_uniquekeys. + * + * added_to_joinrel is true if a uniquekey (from outerrel or innerrel) + * has been added to joinrel. + * useful is true if the exprs of the uniquekey still appears in joinrel. + */ +typedef struct UniqueKeyContextData +{ + UniqueKey *uniquekey; + bool added_to_joinrel; + bool useful; +} *UniqueKeyContext; + +static List *initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel); +static bool innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + bool reverse); + +static List *get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals); +static List *get_exprs_from_uniquekey(RelOptInfo *joinrel, + RelOptInfo *rel1, + UniqueKey *ukey); +static void add_uniquekey_for_onerow(RelOptInfo *rel); +static bool add_combined_uniquekey(RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype); + +/* Used for unique indexes checking for partitioned table */ +static bool index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind); +static IndexOptInfo *simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from); +static bool simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2); +static void adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_index); + +/* Helper function for grouped relation and distinct relation. */ +static void add_uniquekey_from_sortgroups(PlannerInfo *root, + RelOptInfo *rel, + List *sortgroups); + +/* + * populate_baserel_uniquekeys + * Populate 'baserel' uniquekeys list by looking at the rel's unique index + * and baserestrictinfo + */ +void +populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List *indexlist) +{ + ListCell *lc; + List *matched_uniq_indexes = NIL; + + /* Attrs appears in rel->reltarget->exprs. */ + Bitmapset *used_attrs = NULL; + + List *const_exprs = NIL; + List *expr_opfamilies = NIL; + + Assert(baserel->rtekind == RTE_RELATION); + + foreach(lc, indexlist) + { + IndexOptInfo *ind = (IndexOptInfo *) lfirst(lc); + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + matched_uniq_indexes = lappend(matched_uniq_indexes, ind); + } + + if (matched_uniq_indexes == NIL) + return; + + /* Check which attrs is used in baserel->reltarget */ + pull_varattnos((Node *)baserel->reltarget->exprs, baserel->relid, &used_attrs); + + /* Check which attrno is used at a mergeable const filter */ + foreach(lc, baserel->baserestrictinfo) + { + RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); + + if (rinfo->mergeopfamilies == NIL) + continue; + + if (bms_is_empty(rinfo->left_relids)) + { + const_exprs = lappend(const_exprs, get_rightop(rinfo->clause)); + } + else if (bms_is_empty(rinfo->right_relids)) + { + const_exprs = lappend(const_exprs, get_leftop(rinfo->clause)); + } + else + continue; + + expr_opfamilies = lappend(expr_opfamilies, rinfo->mergeopfamilies); + } + + foreach(lc, matched_uniq_indexes) + { + bool multi_nullvals, useful; + List *exprs = get_exprs_from_uniqueindex(lfirst_node(IndexOptInfo, lc), + const_exprs, + expr_opfamilies, + used_attrs, + &useful, + &multi_nullvals); + if (useful) + { + if (exprs == NIL) + { + /* All the columns in Unique Index matched with a restrictinfo */ + add_uniquekey_for_onerow(baserel); + return; + } + baserel->uniquekeys = lappend(baserel->uniquekeys, + makeUniqueKey(exprs, multi_nullvals)); + } + } +} + + +/* + * populate_partitionedrel_uniquekeys + * The UniqueKey on partitionrel comes from 2 cases: + * 1). Only one partition is involved in this query, the unique key can be + * copied to parent rel from childrel. + * 2). There are some unique index which includes partition key and exists + * in all the related partitions. + * We never mind rule 2 if we hit rule 1. + */ + +void +populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels) +{ + ListCell *lc; + List *global_uniq_indexlist = NIL; + RelOptInfo *childrel; + bool is_first = true; + + Assert(IS_PARTITIONED_REL(rel)); + + if (childrels == NIL) + return; + + /* + * If there is only one partition used in this query, the UniqueKey in childrel is + * still valid in parent level, but we need convert the format from child expr to + * parent expr. + */ + if (list_length(childrels) == 1) + { + /* Check for Rule 1 */ + RelOptInfo *childrel = linitial_node(RelOptInfo, childrels); + ListCell *lc; + Assert(childrel->reloptkind == RELOPT_OTHER_MEMBER_REL); + if (relation_is_onerow(childrel)) + { + add_uniquekey_for_onerow(rel); + return; + } + + foreach(lc, childrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + AppendRelInfo *appinfo = find_appinfo_by_child(root, childrel->relid); + List *parent_exprs = NIL; + bool can_reuse = true; + ListCell *lc2; + foreach(lc2, ukey->exprs) + { + Var *var = (Var *)lfirst(lc2); + /* + * If the expr comes from a expression, it is hard to build the expression + * in parent so ignore that case for now. + */ + if(!IsA(var, Var)) + { + can_reuse = false; + break; + } + /* Convert it to parent var */ + parent_exprs = lappend(parent_exprs, find_parent_var(appinfo, var)); + } + if (can_reuse) + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(parent_exprs, + ukey->multi_nullvals)); + } + } + else + { + /* Check for rule 2 */ + childrel = linitial_node(RelOptInfo, childrels); + foreach(lc, childrel->indexlist) + { + IndexOptInfo *ind = lfirst(lc); + IndexOptInfo *modified_index; + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + + /* + * During simple_copy_indexinfo_to_parent, we need to convert var from + * child var to parent var, index on expression is too complex to handle. + * so ignore it for now. + */ + if (ind->indexprs != NIL) + continue; + + modified_index = simple_copy_indexinfo_to_parent(root, rel, ind); + /* + * If the unique index doesn't contain partkey, then it is unique + * on this partition only, so it is useless for us. + */ + if (!index_constains_partkey(rel, modified_index)) + continue; + + global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index); + } + + if (global_uniq_indexlist != NIL) + { + foreach(lc, childrels) + { + RelOptInfo *child = lfirst(lc); + if (is_first) + { + is_first = false; + continue; + } + adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist); + } + /* Now we have a list of unique index which are exactly same on all childrels, + * Set the UniqueKey just like it is non-partition table + */ + populate_baserel_uniquekeys(root, rel, global_uniq_indexlist); + } + } +} + + +/* + * populate_distinctrel_uniquekeys + */ +void +populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel) +{ + /* The unique key before the distinct is still valid. */ + distinctrel->uniquekeys = list_copy(inputrel->uniquekeys); + add_uniquekey_from_sortgroups(root, distinctrel, root->parse->distinctClause); +} + +/* + * populate_grouprel_uniquekeys + */ +void +populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel) + +{ + Query *parse = root->parse; + bool input_ukey_added = false; + ListCell *lc; + + if (relation_is_onerow(inputrel)) + { + add_uniquekey_for_onerow(grouprel); + return; + } + if (parse->groupingSets) + return; + + /* A Normal group by without grouping set. */ + if (parse->groupClause) + { + /* + * Current even the groupby clause is Unique already, but if query has aggref + * We have to create grouprel still. To keep the UnqiueKey short, we will check + * the UniqueKey of input_rel still valid, if so we reuse it. + */ + foreach(lc, inputrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (list_is_subset(ukey->exprs, grouprel->reltarget->exprs)) + { + grouprel->uniquekeys = lappend(grouprel->uniquekeys, + ukey); + input_ukey_added = true; + } + } + if (!input_ukey_added) + /* + * group by clause must be a super-set of grouprel->reltarget->exprs except the + * aggregation expr, so if such exprs is unique already, no bother to generate + * new uniquekey for group by exprs. + */ + add_uniquekey_from_sortgroups(root, + grouprel, + root->parse->groupClause); + } + else + /* It has aggregation but without a group by, so only one row returned */ + add_uniquekey_for_onerow(grouprel); +} + +/* + * simple_copy_uniquekeys + * Using a function for the one-line code makes us easy to check where we simply + * copied the uniquekey. + */ +void +simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel) +{ + newrel->uniquekeys = oldrel->uniquekeys; +} + +/* + * populate_unionrel_uniquekeys + */ +void +populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel) +{ + ListCell *lc; + List *exprs = NIL; + + Assert(unionrel->uniquekeys == NIL); + + foreach(lc, unionrel->reltarget->exprs) + { + exprs = lappend(exprs, lfirst(lc)); + } + + if (exprs == NIL) + /* SQL: select union select; is valid, we need to handle it here. */ + add_uniquekey_for_onerow(unionrel); + else + unionrel->uniquekeys = lappend(unionrel->uniquekeys, + makeUniqueKey(exprs,false)); + +} + +/* + * populate_joinrel_uniquekeys + * + * populate uniquekeys for joinrel. We will check each relation to see if its + * UniqueKey is still valid via innerrel_keeps_unique, if so, we add it to + * joinrel. The multi_nullvals field will be changed to true for some outer + * join cases and one-row UniqueKey needs to be converted to normal UniqueKey + * for the same case as well. + * For the uniquekey in either baserel which can't be unique after join, we still + * check to see if combination of UniqueKeys from both side is still useful for us. + * if yes, we add it to joinrel as well. + */ +void +populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *outerrel, RelOptInfo *innerrel, + List *restrictlist, JoinType jointype) +{ + ListCell *lc, *lc2; + List *clause_list = NIL; + List *outerrel_ukey_ctx; + List *innerrel_ukey_ctx; + bool inner_onerow, outer_onerow; + bool mergejoin_allowed; + + /* Care about the outerrel relation only for SEMI/ANTI join */ + if (jointype == JOIN_SEMI || jointype == JOIN_ANTI) + { + foreach(lc, outerrel->uniquekeys) + { + UniqueKey *uniquekey = lfirst_node(UniqueKey, lc); + if (list_is_subset(uniquekey->exprs, joinrel->reltarget->exprs)) + joinrel->uniquekeys = lappend(joinrel->uniquekeys, uniquekey); + } + return; + } + + Assert(jointype == JOIN_LEFT || jointype == JOIN_FULL || jointype == JOIN_INNER); + + /* Fast path */ + if (innerrel->uniquekeys == NIL || outerrel->uniquekeys == NIL) + return; + + inner_onerow = relation_is_onerow(innerrel); + outer_onerow = relation_is_onerow(outerrel); + + outerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(outerrel); + innerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(innerrel); + + clause_list = select_mergejoin_clauses(root, joinrel, outerrel, innerrel, + restrictlist, jointype, + &mergejoin_allowed); + + if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true /* reverse */)) + { + bool outer_impact = jointype == JOIN_FULL; + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + /* Outer relation has one row, and the unique key is not duplicated after join, + * the joinrel will still has one row unless the jointype == JOIN_FULL. + */ + if (outer_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (outer_onerow) + { + /* + * The onerow outerrel becomes multi rows and multi_nullvals + * will be changed to true. We also need to set the exprs correctly since it + * can't be NIL any more. + */ + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(joinrel, outerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Change multi_nullvals to true due to the full join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, true)); + else + /* Just reuse it */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + } + ctx->added_to_joinrel = true; + } + } + + if (innerrel_keeps_unique(root, outerrel, innerrel, clause_list, false)) + { + bool outer_impact = jointype == JOIN_FULL || jointype == JOIN_LEFT;; + + foreach(lc, innerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + if (inner_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (inner_onerow) + { + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(joinrel, innerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Need to change multi_nullvals to true due to the outer join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, + true)); + else + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + + } + ctx->added_to_joinrel = true; + } + } + + /* + * The combination of the UniqueKey from both sides is unique as well regardless + * of join type, but no bother to add it if its subset has been added to joinrel + * already or it is not useful for the joinrel. + */ + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx1 = (UniqueKeyContext) lfirst(lc); + if (ctx1->added_to_joinrel || !ctx1->useful) + continue; + foreach(lc2, innerrel_ukey_ctx) + { + UniqueKeyContext ctx2 = (UniqueKeyContext) lfirst(lc2); + if (ctx2->added_to_joinrel || !ctx2->useful) + continue; + if (add_combined_uniquekey(joinrel, outerrel, innerrel, + ctx1->uniquekey, ctx2->uniquekey, + jointype)) + /* If we set a onerow UniqueKey to joinrel, we don't need other. */ + return; + } + } +} + + +/* + * convert_subquery_uniquekeys + * + * Covert the UniqueKey in subquery to outer relation. + */ +void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel) +{ + ListCell *lc; + + if (sub_final_rel->uniquekeys == NIL) + return; + + if (relation_is_onerow(sub_final_rel)) + { + add_uniquekey_for_onerow(currel); + return; + } + + Assert(currel->subroot != NULL); + + foreach(lc, sub_final_rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + ListCell *lc; + List *exprs = NIL; + bool ukey_useful = true; + + /* One row case is handled above */ + Assert(ukey->exprs != NIL); + foreach(lc, ukey->exprs) + { + Var *var; + TargetEntry *tle = tlist_member(lfirst(lc), + currel->subroot->processed_tlist); + if (tle == NULL) + { + ukey_useful = false; + break; + } + var = find_var_for_subquery_tle(currel, tle); + if (var == NULL) + { + ukey_useful = false; + break; + } + exprs = lappend(exprs, var); + } + + if (ukey_useful) + currel->uniquekeys = lappend(currel->uniquekeys, + makeUniqueKey(exprs, + ukey->multi_nullvals)); + + } +} + +/* + * innerrel_keeps_unique + * + * Check if Unique key of the innerrel is valid after join. innerrel's UniqueKey + * will be still valid if innerrel's any-column mergeop outrerel's uniquekey + * exists in clause_list. + * + * Note: the clause_list must be a list of mergeable restrictinfo already. + */ +static bool +innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *clause_list, + bool reverse) +{ + ListCell *lc, *lc2, *lc3; + + if (outerrel->uniquekeys == NIL || innerrel->uniquekeys == NIL) + return false; + + /* Check if there is outerrel's uniquekey in mergeable clause. */ + foreach(lc, outerrel->uniquekeys) + { + List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs; + bool clauselist_matchs_all_exprs = true; + foreach(lc2, outer_uq_exprs) + { + Node *outer_uq_expr = lfirst(lc2); + bool find_uq_expr_in_clauselist = false; + foreach(lc3, clause_list) + { + RestrictInfo *rinfo = lfirst_node(RestrictInfo, lc3); + Node *outer_expr; + if (reverse) + outer_expr = rinfo->outer_is_left ? get_rightop(rinfo->clause) : get_leftop(rinfo->clause); + else + outer_expr = rinfo->outer_is_left ? get_leftop(rinfo->clause) : get_rightop(rinfo->clause); + if (equal(outer_expr, outer_uq_expr)) + { + find_uq_expr_in_clauselist = true; + break; + } + } + if (!find_uq_expr_in_clauselist) + { + /* No need to check the next exprs in the current uniquekey */ + clauselist_matchs_all_exprs = false; + break; + } + } + + if (clauselist_matchs_all_exprs) + return true; + } + return false; +} + + +/* + * relation_is_onerow + * Check if it is a one-row relation by checking UniqueKey. + */ +bool +relation_is_onerow(RelOptInfo *rel) +{ + UniqueKey *ukey; + if (rel->uniquekeys == NIL) + return false; + ukey = linitial_node(UniqueKey, rel->uniquekeys); + return ukey->exprs == NIL && list_length(rel->uniquekeys) == 1; +} + +/* + * relation_has_uniquekeys_for + * Returns true if we have proofs that 'rel' cannot return multiple rows with + * the same values in each of 'exprs'. Otherwise returns false. + */ +bool +relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel, + List *exprs, bool allow_multinulls) +{ + ListCell *lc; + + /* + * For UniqueKey->onerow case, the uniquekey->exprs is empty as well + * so we can't rely on list_is_subset to handle this special cases + */ + if (exprs == NIL) + return false; + + foreach(lc, rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (ukey->multi_nullvals && !allow_multinulls) + continue; + if (list_is_subset(ukey->exprs, exprs)) + return true; + } + return false; +} + + +/* + * get_exprs_from_uniqueindex + * + * Return a list of exprs which is unique. set useful to false if this + * unique index is not useful for us. + */ +static List * +get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals) +{ + List *exprs = NIL; + ListCell *indexpr_item; + int c = 0; + + *useful = true; + *multi_nullvals = false; + + indexpr_item = list_head(unique_index->indexprs); + for(c = 0; c < unique_index->ncolumns; c++) + { + int attr = unique_index->indexkeys[c]; + Expr *expr; + bool matched_const = false; + ListCell *lc1, *lc2; + + if(attr > 0) + { + expr = list_nth_node(TargetEntry, unique_index->indextlist, c)->expr; + } + else if (attr == 0) + { + /* Expression index */ + expr = lfirst(indexpr_item); + indexpr_item = lnext(unique_index->indexprs, indexpr_item); + } + else /* attr < 0 */ + { + /* Index on system column is not supported */ + Assert(false); + } + + /* + * Check index_col = Const case with regarding to opfamily checking + * If we can remove the index_col from the final UniqueKey->exprs. + */ + forboth(lc1, const_exprs, lc2, const_expr_opfamilies) + { + if (list_member_oid((List *)lfirst(lc2), unique_index->opfamily[c]) + && match_index_to_operand((Node *) lfirst(lc1), c, unique_index)) + { + matched_const = true; + break; + } + } + + if (matched_const) + continue; + + /* Check if the indexed expr is used in rel */ + if (attr > 0) + { + /* + * Normal Indexed column, if the col is not used, then the index is useless + * for uniquekey. + */ + attr -= FirstLowInvalidHeapAttributeNumber; + + if (!bms_is_member(attr, used_varattrs)) + { + *useful = false; + break; + } + } + else if (!list_member(unique_index->rel->reltarget->exprs, expr)) + { + /* Expression index but the expression is not used in rel */ + *useful = false; + break; + } + + /* check not null property. */ + if (attr == 0) + { + /* We never know if a expression yields null or not */ + *multi_nullvals = true; + } + else if (!bms_is_member(attr, unique_index->rel->notnullattrs) + && !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber, + unique_index->rel->notnullattrs)) + { + *multi_nullvals = true; + } + + exprs = lappend(exprs, expr); + } + return exprs; +} + + +/* + * add_uniquekey_for_onerow + * If we are sure that the relation only returns one row, then all the columns + * are unique. However we don't need to create UniqueKey for every column, we + * just set exprs = NIL and overwrites all the other UniqueKey on this RelOptInfo + * since this one has strongest semantics. + */ +void +add_uniquekey_for_onerow(RelOptInfo *rel) +{ + /* + * We overwrite the previous UniqueKey on purpose since this one has the + * strongest semantic. + */ + rel->uniquekeys = list_make1(makeUniqueKey(NIL, false)); +} + + +/* + * initililze_uniquecontext_for_joinrel + * Return a List of UniqueKeyContext for an inputrel + */ +static List * +initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel) +{ + List *res = NIL; + ListCell *lc; + foreach(lc, inputrel->uniquekeys) + { + UniqueKeyContext context; + context = palloc(sizeof(struct UniqueKeyContextData)); + context->uniquekey = lfirst_node(UniqueKey, lc); + context->added_to_joinrel = false; + context->useful = true; + res = lappend(res, context); + } + return res; +} + + +/* + * get_exprs_from_uniquekey + * Unify the way of get List of exprs from a one-row UniqueKey or + * normal UniqueKey. for the onerow case, every expr in rel1 is a valid + * UniqueKey. Return a List of exprs. + * + * rel1: The relation which you want to get the exprs. + * ukey: The UniqueKey you want to get the exprs. + */ +static List * +get_exprs_from_uniquekey(RelOptInfo *joinrel, RelOptInfo *rel1, UniqueKey *ukey) +{ + ListCell *lc; + bool onerow = rel1 != NULL && relation_is_onerow(rel1); + + List *res = NIL; + Assert(onerow || ukey); + if (onerow) + { + /* Only cares about the exprs still exist in joinrel */ + foreach(lc, joinrel->reltarget->exprs) + { + Bitmapset *relids = pull_varnos(lfirst(lc)); + if (bms_is_subset(relids, rel1->relids)) + { + res = lappend(res, list_make1(lfirst(lc))); + } + } + } + else + { + res = list_make1(ukey->exprs); + } + return res; +} + +/* + * Partitioned table Unique Keys. + * The partition table unique key is maintained as: + * 1. The index must be unique as usual. + * 2. The index must contains partition key. + * 3. The index must exist on all the child rel. see simple_indexinfo_equal for + * how we compare it. + */ + +/* + * index_constains_partkey + * return true if the index contains the partiton key. + */ +static bool +index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind) +{ + ListCell *lc; + int i; + Assert(IS_PARTITIONED_REL(partrel)); + Assert(partrel->part_scheme->partnatts > 0); + + for(i = 0; i < partrel->part_scheme->partnatts; i++) + { + Node *part_expr = linitial(partrel->partexprs[i]); + bool found_in_index = false; + foreach(lc, ind->indextlist) + { + Expr *index_expr = lfirst_node(TargetEntry, lc)->expr; + if (equal(index_expr, part_expr)) + { + found_in_index = true; + break; + } + } + if (!found_in_index) + return false; + } + return true; +} + +/* + * simple_indexinfo_equal + * + * Used to check if the 2 index is same as each other. The index here + * is COPIED from childrel and did some tiny changes(see + * simple_copy_indexinfo_to_parent) + */ +static bool +simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2) +{ + Size oid_cmp_len = sizeof(Oid) * ind1->ncolumns; + + return ind1->ncolumns == ind2->ncolumns && + ind1->unique == ind2->unique && + memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0 && + memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0 && + memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0 && + memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0 && + equal(get_tlist_exprs(ind1->indextlist, true), + get_tlist_exprs(ind2->indextlist, true)); +} + + +/* + * The below macros are used for simple_copy_indexinfo_to_parent which is so + * customized that I don't want to put it to copyfuncs.c. So copy it here. + */ +#define COPY_POINTER_FIELD(fldname, sz) \ + do { \ + Size _size = (sz); \ + newnode->fldname = palloc(_size); \ + memcpy(newnode->fldname, from->fldname, _size); \ + } while (0) + +#define COPY_NODE_FIELD(fldname) \ + (newnode->fldname = copyObjectImpl(from->fldname)) + +#define COPY_SCALAR_FIELD(fldname) \ + (newnode->fldname = from->fldname) + + +/* + * simple_copy_indexinfo_to_parent (from partition) + * Copy the IndexInfo from child relation to parent relation with some modification, + * which is used to test: + * 1. If the same index exists in all the childrels. + * 2. If the parentrel->reltarget/basicrestrict info matches this index. + */ +static IndexOptInfo * +simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from) +{ + IndexOptInfo *newnode = makeNode(IndexOptInfo); + AppendRelInfo *appinfo = find_appinfo_by_child(root, from->rel->relid); + ListCell *lc; + int idx = 0; + + COPY_SCALAR_FIELD(ncolumns); + COPY_SCALAR_FIELD(nkeycolumns); + COPY_SCALAR_FIELD(unique); + COPY_SCALAR_FIELD(immediate); + /* We just need to know if it is NIL or not */ + COPY_SCALAR_FIELD(indpred); + COPY_SCALAR_FIELD(predOK); + COPY_POINTER_FIELD(indexkeys, from->ncolumns * sizeof(int)); + COPY_POINTER_FIELD(indexcollations, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opfamily, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opcintype, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(sortopfamily, from->ncolumns * sizeof(Oid)); + COPY_NODE_FIELD(indextlist); + + /* Convert index exprs on child expr to expr on parent */ + foreach(lc, newnode->indextlist) + { + TargetEntry *tle = lfirst_node(TargetEntry, lc); + /* Index on expression is ignored */ + Assert(IsA(tle->expr, Var)); + tle->expr = (Expr *) find_parent_var(appinfo, (Var *) tle->expr); + newnode->indexkeys[idx] = castNode(Var, tle->expr)->varattno; + idx++; + } + newnode->rel = parentrel; + return newnode; +} + +/* + * adjust_partition_unique_indexlist + * + * global_unique_indexes: At the beginning, it contains the copy & modified + * unique index from the first partition. And then check if each index in it still + * exists in the following partitions. If no, remove it. at last, it has an + * index list which exists in all the partitions. + */ +static void +adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_indexes) +{ + ListCell *lc, *lc2; + foreach(lc, *global_unique_indexes) + { + IndexOptInfo *g_ind = lfirst_node(IndexOptInfo, lc); + bool found_in_child = false; + + foreach(lc2, childrel->indexlist) + { + IndexOptInfo *p_ind = lfirst_node(IndexOptInfo, lc2); + IndexOptInfo *p_ind_copy; + if (!p_ind->unique || !p_ind->immediate || + (p_ind->indpred != NIL && !p_ind->predOK)) + continue; + p_ind_copy = simple_copy_indexinfo_to_parent(root, parentrel, p_ind); + if (simple_indexinfo_equal(p_ind_copy, g_ind)) + { + found_in_child = true; + break; + } + } + if (!found_in_child) + /* The index doesn't exist in childrel, remove it from global_unique_indexes */ + *global_unique_indexes = foreach_delete_current(*global_unique_indexes, lc); + } +} + +/* Helper function for groupres/distinctrel */ +static void +add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgroups) +{ + Query *parse = root->parse; + List *exprs; + + /* + * XXX: If there are some vars which is not in current levelsup, the semantic is + * imprecise, should we avoid it or not? levelsup = 1 is just a demo, maybe we need to + * check every level other than 0, if so, looks we have to write another + * pull_var_walker. + */ + List *upper_vars = pull_vars_of_level((Node*)sortgroups, 1); + + if (upper_vars != NIL) + return; + + exprs = get_sortgrouplist_exprs(sortgroups, parse->targetList); + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(exprs, + false /* sortgroupclause can't be multi_nullvals */)); +} + + +/* + * add_combined_uniquekey + * The combination of both UniqueKeys is a valid UniqueKey for joinrel no matter + * the jointype. + */ +bool +add_combined_uniquekey(RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype) +{ + + ListCell *lc1, *lc2; + + /* Either side has multi_nullvals or we have outer join, + * the combined UniqueKey has multi_nullvals */ + bool multi_nullvals = outer_ukey->multi_nullvals || + inner_ukey->multi_nullvals || IS_OUTER_JOIN(jointype); + + /* The only case we can get onerow joinrel after join */ + if (relation_is_onerow(outer_rel) + && relation_is_onerow(inner_rel) + && jointype == JOIN_INNER) + { + add_uniquekey_for_onerow(joinrel); + return true; + } + + foreach(lc1, get_exprs_from_uniquekey(joinrel, outer_rel, outer_ukey)) + { + foreach(lc2, get_exprs_from_uniquekey(joinrel, inner_rel, inner_ukey)) + { + List *exprs = list_concat_copy(lfirst_node(List, lc1), lfirst_node(List, lc2)); + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(exprs, + multi_nullvals)); + } + } + return false; +} diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index b406d41e91..0551ae0512 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -2389,6 +2389,8 @@ grouping_planner(PlannerInfo *root, bool inheritance_update, add_path(final_rel, path); } + simple_copy_uniquekeys(current_rel, final_rel); + /* * Generate partial paths for final_rel, too, if outer query levels might * be able to make use of them. @@ -3899,6 +3901,8 @@ create_grouping_paths(PlannerInfo *root, } set_cheapest(grouped_rel); + + populate_grouprel_uniquekeys(root, grouped_rel, input_rel); return grouped_rel; } @@ -4615,7 +4619,7 @@ create_window_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(window_rel); - + simple_copy_uniquekeys(input_rel, window_rel); return window_rel; } @@ -4911,7 +4915,7 @@ create_distinct_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(distinct_rel); - + populate_distinctrel_uniquekeys(root, input_rel, distinct_rel); return distinct_rel; } @@ -5172,6 +5176,8 @@ create_ordered_paths(PlannerInfo *root, */ Assert(ordered_rel->pathlist != NIL); + simple_copy_uniquekeys(input_rel, ordered_rel); + return ordered_rel; } @@ -6049,6 +6055,9 @@ adjust_paths_for_srfs(PlannerInfo *root, RelOptInfo *rel, if (list_length(targets) == 1) return; + /* UniqueKey is not valid after handling the SRF. */ + rel->uniquekeys = NIL; + /* * Stack SRF-evaluation nodes atop each path for the rel. * diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c index 951aed80e7..e94e92937c 100644 --- a/src/backend/optimizer/prep/prepunion.c +++ b/src/backend/optimizer/prep/prepunion.c @@ -689,6 +689,8 @@ generate_union_paths(SetOperationStmt *op, PlannerInfo *root, /* Undo effects of possibly forcing tuple_fraction to 0 */ root->tuple_fraction = save_fraction; + /* Add the UniqueKeys */ + populate_unionrel_uniquekeys(root, result_rel); return result_rel; } diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c index d722063cf3..44c37ecffc 100644 --- a/src/backend/optimizer/util/appendinfo.c +++ b/src/backend/optimizer/util/appendinfo.c @@ -746,3 +746,47 @@ find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos) } return appinfos; } + +/* + * find_appinfo_by_child + * + */ +AppendRelInfo * +find_appinfo_by_child(PlannerInfo *root, Index child_index) +{ + ListCell *lc; + foreach(lc, root->append_rel_list) + { + AppendRelInfo *appinfo = lfirst_node(AppendRelInfo, lc); + if (appinfo->child_relid == child_index) + return appinfo; + } + elog(ERROR, "parent relation cant be found"); + return NULL; +} + +/* + * find_parent_var + * + */ +Var * +find_parent_var(AppendRelInfo *appinfo, Var *child_var) +{ + ListCell *lc; + Var *res = NULL; + Index attno = 1; + foreach(lc, appinfo->translated_vars) + { + Node *child_node = lfirst(lc); + if (equal(child_node, child_var)) + { + res = copyObject(child_var); + res->varattno = attno; + res->varno = appinfo->parent_relid; + } + attno++; + } + if (res == NULL) + elog(ERROR, "parent var can't be found."); + return res; +} diff --git a/src/backend/optimizer/util/inherit.c b/src/backend/optimizer/util/inherit.c index 3132fd35a5..d66b40ec50 100644 --- a/src/backend/optimizer/util/inherit.c +++ b/src/backend/optimizer/util/inherit.c @@ -736,6 +736,7 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, { Node *onecq = (Node *) lfirst(lc2); bool pseudoconstant; + RestrictInfo *child_rinfo; /* check for pseudoconstant (no Vars or volatile functions) */ pseudoconstant = @@ -747,13 +748,14 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, root->hasPseudoConstantQuals = true; } /* reconstitute RestrictInfo with appropriate properties */ - childquals = lappend(childquals, - make_restrictinfo((Expr *) onecq, - rinfo->is_pushed_down, - rinfo->outerjoin_delayed, - pseudoconstant, - rinfo->security_level, - NULL, NULL, NULL)); + child_rinfo = make_restrictinfo((Expr *) onecq, + rinfo->is_pushed_down, + rinfo->outerjoin_delayed, + pseudoconstant, + rinfo->security_level, + NULL, NULL, NULL); + child_rinfo->mergeopfamilies = rinfo->mergeopfamilies; + childquals = lappend(childquals, child_rinfo); /* track minimum security level among child quals */ cq_min_security = Min(cq_min_security, rinfo->security_level); } diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 31d9aedeeb..c83f17acb7 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -16,6 +16,7 @@ #include "nodes/execnodes.h" #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" extern A_Expr *makeA_Expr(A_Expr_Kind kind, List *name, @@ -105,4 +106,6 @@ extern GroupingSet *makeGroupingSet(GroupingSetKind kind, List *content, int loc extern VacuumRelation *makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols); +extern UniqueKey* makeUniqueKey(List *exprs, bool multi_nullvals); + #endif /* MAKEFUNC_H */ diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 381d84b4e4..41110ed888 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -264,6 +264,7 @@ typedef enum NodeTag T_EquivalenceMember, T_PathKey, T_PathTarget, + T_UniqueKey, T_RestrictInfo, T_IndexClause, T_PlaceHolderVar, diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index 9e3ebd488a..02e4458bef 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -730,6 +730,7 @@ typedef struct RelOptInfo QualCost baserestrictcost; /* cost of evaluating the above */ Index baserestrict_min_security; /* min security_level found in * baserestrictinfo */ + List *uniquekeys; /* List of UniqueKey */ List *joininfo; /* RestrictInfo structures for join clauses * involving this rel */ bool has_eclass_joins; /* T means joininfo is incomplete */ @@ -1047,6 +1048,28 @@ typedef struct PathKey } PathKey; +/* + * UniqueKey + * + * Represents the unique properties held by a RelOptInfo. + * + * exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL + * is a special case of UniqueKey, which means there is only 1 row in that + * relation. + * multi_nullvals: true means multi null values may exist in these exprs, so the + * uniqueness is not guaranteed in this case. This field is necessary for + * remove_useless_join & reduce_unique_semijoins where we don't mind these + * duplicated NULL values. It is set to true for 2 cases. One is a unique key + * from a unique index but the related column is nullable. The other one is for + * outer join. see populate_joinrel_uniquekeys for detail. + */ +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + /* * PathTarget * @@ -2473,7 +2496,7 @@ typedef enum * * flags indicating what kinds of grouping are possible. * partial_costs_set is true if the agg_partial_costs and agg_final_costs - * have been initialized. + * have been initialized. * agg_partial_costs gives partial aggregation costs. * agg_final_costs gives finalization costs. * target_parallel_safe is true if target is parallel safe. @@ -2503,8 +2526,8 @@ typedef struct * limit_tuples is an estimated bound on the number of output tuples, * or -1 if no LIMIT or couldn't estimate. * count_est and offset_est are the estimated values of the LIMIT and OFFSET - * expressions computed by preprocess_limit() (see comments for - * preprocess_limit() for more information). + * expressions computed by preprocess_limit() (see comments for + * preprocess_limit() for more information). */ typedef struct { diff --git a/src/include/nodes/pg_list.h b/src/include/nodes/pg_list.h index 14ea2766ad..621f54a9f8 100644 --- a/src/include/nodes/pg_list.h +++ b/src/include/nodes/pg_list.h @@ -528,6 +528,8 @@ extern bool list_member_ptr(const List *list, const void *datum); extern bool list_member_int(const List *list, int datum); extern bool list_member_oid(const List *list, Oid datum); +extern bool list_is_subset(const List *members, const List *target); + extern List *list_delete(List *list, void *datum); extern List *list_delete_ptr(List *list, void *datum); extern List *list_delete_int(List *list, int datum); diff --git a/src/include/optimizer/appendinfo.h b/src/include/optimizer/appendinfo.h index d6a27a60dd..e87c92a054 100644 --- a/src/include/optimizer/appendinfo.h +++ b/src/include/optimizer/appendinfo.h @@ -32,4 +32,7 @@ extern Relids adjust_child_relids_multilevel(PlannerInfo *root, Relids relids, extern AppendRelInfo **find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos); +extern AppendRelInfo *find_appinfo_by_child(PlannerInfo *root, Index child_index); +extern Var *find_parent_var(AppendRelInfo *appinfo, Var *child_var); + #endif /* APPENDINFO_H */ diff --git a/src/include/optimizer/optimizer.h b/src/include/optimizer/optimizer.h index 3e4171056e..9445141263 100644 --- a/src/include/optimizer/optimizer.h +++ b/src/include/optimizer/optimizer.h @@ -23,6 +23,7 @@ #define OPTIMIZER_H #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" /* * We don't want to include nodes/pathnodes.h here, because non-planner @@ -156,6 +157,7 @@ extern TargetEntry *get_sortgroupref_tle(Index sortref, List *targetList); extern TargetEntry *get_sortgroupclause_tle(SortGroupClause *sgClause, List *targetList); +extern Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); extern Node *get_sortgroupclause_expr(SortGroupClause *sgClause, List *targetList); extern List *get_sortgrouplist_exprs(List *sgClauses, diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h index 10b6e81079..9217a8d6c6 100644 --- a/src/include/optimizer/paths.h +++ b/src/include/optimizer/paths.h @@ -240,5 +240,48 @@ extern PathKey *make_canonical_pathkey(PlannerInfo *root, int strategy, bool nulls_first); extern void add_paths_to_append_rel(PlannerInfo *root, RelOptInfo *rel, List *live_childrels); +extern List *select_mergejoin_clauses(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + JoinType jointype, + bool *mergejoin_allowed); + +/* + * uniquekeys.c + * Utilities for matching and building unique keys + */ +extern void populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List* unique_index_list); +extern void populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels); +extern void populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel); +extern void populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel); +extern void populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel); +extern void simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel); +extern void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel); +extern void populate_joinrel_uniquekeys(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + RelOptInfo *rel2, + List *restrictlist, + JoinType jointype); + +extern bool relation_has_uniquekeys_for(PlannerInfo *root, + RelOptInfo *rel, + List *exprs, + bool allow_multinulls); +extern bool relation_is_onerow(RelOptInfo *rel); #endif /* PATHS_H */ -- 2.21.0 --op7mf2i72xk5yu7c Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v36-0003-Extend-UniqueKeys.patch" ^ permalink raw reply [nested|flat] 91+ messages in thread
* [PATCH v38 2/6] Introduce UniqueKey attributes on RelOptInfo struct. @ 2020-05-11 07:50 一挃 <[email protected]> 0 siblings, 0 replies; 91+ messages in thread From: 一挃 @ 2020-05-11 07:50 UTC (permalink / raw) UniqueKey is a set of exprs on RelOptInfo which represents the exprs will be unique on the given RelOptInfo. You can see README.uniquekey for more information. --- src/backend/nodes/copyfuncs.c | 13 + src/backend/nodes/list.c | 31 + src/backend/nodes/makefuncs.c | 13 + src/backend/nodes/outfuncs.c | 11 + src/backend/nodes/readfuncs.c | 10 + src/backend/optimizer/path/Makefile | 3 +- src/backend/optimizer/path/README.uniquekey | 131 +++ src/backend/optimizer/path/allpaths.c | 10 + src/backend/optimizer/path/joinpath.c | 9 +- src/backend/optimizer/path/joinrels.c | 2 + src/backend/optimizer/path/pathkeys.c | 3 +- src/backend/optimizer/path/uniquekeys.c | 1135 +++++++++++++++++++ src/backend/optimizer/plan/planner.c | 13 +- src/backend/optimizer/prep/prepunion.c | 2 + src/backend/optimizer/util/appendinfo.c | 44 + src/backend/optimizer/util/inherit.c | 18 +- src/include/nodes/makefuncs.h | 3 + src/include/nodes/nodes.h | 1 + src/include/nodes/pathnodes.h | 29 +- src/include/nodes/pg_list.h | 10 + src/include/optimizer/appendinfo.h | 3 + src/include/optimizer/optimizer.h | 2 + src/include/optimizer/paths.h | 43 + 23 files changed, 1515 insertions(+), 24 deletions(-) create mode 100644 src/backend/optimizer/path/README.uniquekey create mode 100644 src/backend/optimizer/path/uniquekeys.c diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..75c1c5e824 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2296,6 +2296,16 @@ _copyPathKey(const PathKey *from) return newnode; } +static UniqueKey * +_copyUniqueKey(const UniqueKey *from) +{ + UniqueKey *newnode = makeNode(UniqueKey); + + COPY_NODE_FIELD(exprs); + COPY_SCALAR_FIELD(multi_nullvals); + + return newnode; +} /* * _copyRestrictInfo */ @@ -5220,6 +5230,9 @@ copyObjectImpl(const void *from) case T_PathKey: retval = _copyPathKey(from); break; + case T_UniqueKey: + retval = _copyUniqueKey(from); + break; case T_RestrictInfo: retval = _copyRestrictInfo(from); break; diff --git a/src/backend/nodes/list.c b/src/backend/nodes/list.c index dbf6b30233..ca099495a1 100644 --- a/src/backend/nodes/list.c +++ b/src/backend/nodes/list.c @@ -702,6 +702,37 @@ list_member_oid(const List *list, Oid datum) return false; } +/* + * return true iff every entry in "members" list is also present + * in the "target" list. + */ +bool +list_is_subset(const List *members, const List *target) +{ + const ListCell *lc1, *lc2; + + Assert(IsPointerList(members)); + Assert(IsPointerList(target)); + check_list_invariants(members); + check_list_invariants(target); + + foreach(lc1, members) + { + bool found = false; + foreach(lc2, target) + { + if (equal(lfirst(lc1), lfirst(lc2))) + { + found = true; + break; + } + } + if (!found) + return false; + } + return true; +} + /* * Delete the n'th cell (counting from 0) in list. * diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 01c110cd2f..40415d0f5b 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -815,3 +815,16 @@ makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols) v->va_cols = va_cols; return v; } + + +/* + * makeUniqueKey + */ +UniqueKey* +makeUniqueKey(List *exprs, bool multi_nullvals) +{ + UniqueKey * ukey = makeNode(UniqueKey); + ukey->exprs = exprs; + ukey->multi_nullvals = multi_nullvals; + return ukey; +} diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..44154cde6a 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2456,6 +2456,14 @@ _outPathKey(StringInfo str, const PathKey *node) WRITE_BOOL_FIELD(pk_nulls_first); } +static void +_outUniqueKey(StringInfo str, const UniqueKey *node) +{ + WRITE_NODE_TYPE("UNIQUEKEY"); + WRITE_NODE_FIELD(exprs); + WRITE_BOOL_FIELD(multi_nullvals); +} + static void _outPathTarget(StringInfo str, const PathTarget *node) { @@ -4198,6 +4206,9 @@ outNode(StringInfo str, const void *obj) case T_PathKey: _outPathKey(str, obj); break; + case T_UniqueKey: + _outUniqueKey(str, obj); + break; case T_PathTarget: _outPathTarget(str, obj); break; diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index c5e136e9c3..b3e212bf1c 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -492,6 +492,14 @@ _readSetOperationStmt(void) READ_DONE(); } +static UniqueKey * +_readUniqueKey(void) +{ + READ_LOCALS(UniqueKey); + READ_NODE_FIELD(exprs); + READ_BOOL_FIELD(multi_nullvals); + READ_DONE(); +} /* * Stuff from primnodes.h. @@ -2717,6 +2725,8 @@ parseNodeString(void) return_value = _readCommonTableExpr(); else if (MATCH("SETOPERATIONSTMT", 16)) return_value = _readSetOperationStmt(); + else if (MATCH("UNIQUEKEY", 9)) + return_value = _readUniqueKey(); else if (MATCH("ALIAS", 5)) return_value = _readAlias(); else if (MATCH("RANGEVAR", 8)) diff --git a/src/backend/optimizer/path/Makefile b/src/backend/optimizer/path/Makefile index 1e199ff66f..7b9820c25f 100644 --- a/src/backend/optimizer/path/Makefile +++ b/src/backend/optimizer/path/Makefile @@ -21,6 +21,7 @@ OBJS = \ joinpath.o \ joinrels.o \ pathkeys.o \ - tidpath.o + tidpath.o \ + uniquekeys.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/optimizer/path/README.uniquekey b/src/backend/optimizer/path/README.uniquekey new file mode 100644 index 0000000000..5eac761995 --- /dev/null +++ b/src/backend/optimizer/path/README.uniquekey @@ -0,0 +1,131 @@ +1. What is UniqueKey? +We can think UniqueKey is a set of exprs for a RelOptInfo, which we are insure +that doesn't yields same result among all the rows. The simplest UniqueKey +format is primary key. + +However we define the UnqiueKey as below. + +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + +exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL +is a special case of UniqueKey, which means there is only one row in that +relation.it has a stronger semantic than others. like SELECT uk FROM t; uk is +normal unique key and may have different values. SELECT colx FROM t WHERE uk = +const. colx is unique AND we have only 1 value. This field can used for +innerrel_is_unique. this logic is handled specially in add_uniquekey_for_onerow +function. + +multi_nullvals: true means multi null values may exist in these exprs, so the +uniqueness is not guaranteed in this case. This field is necessary for +remove_useless_join & reduce_unique_semijoins where we don't mind these +duplicated NULL values. It is set to true for 2 cases. One is a unique key +from a unique index but the related column is nullable. The other one is for +outer join. see populate_joinrel_uniquekeys for detail. + + +The UniqueKey can be used at the following cases at least: +1. remove_useless_joins. +2. reduce_semianti_joins +3. remove distinct node if distinct clause is unique. +4. remove aggnode if group by clause is unique. +5. Index Skip Scan (WIP) +6. Aggregation Push Down without 2 phase aggregation if the join can't + duplicated the aggregated rows. (work in progress feature) + +2. How is it maintained? + +We have a set of populate_xxx_unqiuekeys functions to maintain the uniquekey on +various cases. xxx includes baserel, joinrel, partitionedrel, distinctrel, +groupedrel, unionrel. and we also need to convert the uniquekey from subquery +to outer relation, which is what convert_subquery_uniquekeys does. + +1. The first part is about baserel. We handled 3 cases. suppose we have Unique +Index on (a, b). + +1. SELECT a, b FROM t. UniqueKey (a, b) +2. SELECT a FROM t WHERE b = 1; UniqueKey (a) +3. SELECT .. FROM t WHERE a = 1 AND b = 1; UniqueKey (NIL). onerow case, every + column is Unique. + +2. The next part is joinrel, this part is most error-prone, we simplified the rules +like below: +1. If the relation's UniqueKey can't be duplicated after join, then is will be + still valid for the join rel. The function we used here is + innerrel_keeps_unique. The basic idea is innerrel.any_col = outer.uk. + +2. If the UnqiueKey can't keep valid via the rule 1, the combination of the + UniqueKey from both sides are valid for sure. We can prove this as: if the + unique exprs from rel1 is duplicated by rel2, the duplicated rows must + contains different unique exprs from rel2. + +More considerations about onerow: +1. If relation with one row and it can't be duplicated, it is still possible + contains mulit_nullvas after outer join. +2. If the either UniqueKey can be duplicated after join, the can get one row + only when both side is one row AND there is no outer join. +3. Whenever the onerow UniqueKey is not a valid any more, we need to convert one + row UniqueKey to normal unique key since we don't store exprs for one-row + relation. get_exprs_from_uniquekeys will be used here. + + +More considerations about multi_nullvals after join: +1. If the original UnqiueKey has multi_nullvals, the final UniqueKey will have + mulit_nullvals in any case. +2. If a unique key doesn't allow mulit_nullvals, after some outer join, it + allows some outer join. + + +3. When we comes to subquery, we need to convert_subquery_unqiuekeys just like +convert_subquery_pathkeys. Only the UniqueKey insides subquery is referenced as +a Var in outer relation will be reused. The relationship between the outerrel.Var +and subquery.exprs is built with outerel->subroot->processed_tlist. + + +4. As for the SRF functions, it will break the uniqueness of uniquekey, However it +is handled in adjust_paths_for_srfs, which happens after the query_planner. so +we will maintain the UniqueKey until there and reset it to NIL at that +places. This can't help on distinct/group by elimination cases but probably help +in some other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is +semantic correctly. + + +5. As for inherit table, we first main the UnqiueKey on childrel as well. But for +partitioned table we need to maintain 2 different kinds of +UnqiueKey. 1). UniqueKey on the parent relation 2). UniqueKey on child +relation for partition wise query. + +Example: +CREATE TABLE p (a int not null, b int not null) partition by list (a); +CREATE TABLE p0 partition of p for values in (1); +CREATE TABLE p1 partition of p for values in (2); + +create unique index p0_b on p0(b); +create unique index p1_b on p1(b); + +Now b is only unique on partition level, so the distinct can't be removed on +the following cases. SELECT DISTINCT b FROM p; + +Another example is SELECT DISTINCT a, b FROM p WHERE a = 1; Since only one +partition is chosen, the UniqueKey on child relation is same as the UniqueKey on +parent relation. + +Another usage of UniqueKey on partition level is it be helpful for +partition-wise join. + +As for the UniqueKey on parent table level, it comes with 2 different ways, +1). the UniqueKey is also derived in UniqueKey index, but the index must be same +in all the related children relations and the unique index must contains +Partition Key in it. Example: + +CREATE UNIQUE INDEX p_ab ON p(a, b); -- where a is the partition key. + +-- Query +SELECT a, b FROM p; the (a, b) is a UniqueKey of p. + +2). If the parent relation has only one childrel, the UniqueKey on childrel is + the UniqueKey on parent as well. diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index 37b4223adb..d52ad59f89 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -579,6 +579,12 @@ set_plain_rel_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) */ check_index_predicates(root, rel); + /* + * Now that we've marked which partial indexes are suitable, we can now + * build the relation's unique keys. + */ + populate_baserel_uniquekeys(root, rel, rel->indexlist); + /* Mark rel with estimated output rows, width, etc */ set_baserel_size_estimates(root, rel); } @@ -1297,6 +1303,8 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, /* Add paths to the append relation. */ add_paths_to_append_rel(root, rel, live_childrels); + if (IS_PARTITIONED_REL(rel)) + populate_partitionedrel_uniquekeys(root, rel, live_childrels); } @@ -2304,6 +2312,8 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, pathkeys, required_outer)); } + convert_subquery_uniquekeys(root, rel, sub_final_rel); + /* If outer rel allows parallelism, do same for partial paths. */ if (rel->consider_parallel && bms_is_empty(required_outer)) { diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c index 57ce97fd53..697ac047fb 100644 --- a/src/backend/optimizer/path/joinpath.c +++ b/src/backend/optimizer/path/joinpath.c @@ -71,13 +71,6 @@ static void consider_parallel_mergejoin(PlannerInfo *root, static void hash_inner_and_outer(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, RelOptInfo *innerrel, JoinType jointype, JoinPathExtraData *extra); -static List *select_mergejoin_clauses(PlannerInfo *root, - RelOptInfo *joinrel, - RelOptInfo *outerrel, - RelOptInfo *innerrel, - List *restrictlist, - JoinType jointype, - bool *mergejoin_allowed); static void generate_mergejoin_paths(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *innerrel, @@ -1927,7 +1920,7 @@ hash_inner_and_outer(PlannerInfo *root, * if it is mergejoinable and involves vars from the two sub-relations * currently of interest. */ -static List * +List * select_mergejoin_clauses(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c index 0dbe2ac726..7271f044ec 100644 --- a/src/backend/optimizer/path/joinrels.c +++ b/src/backend/optimizer/path/joinrels.c @@ -924,6 +924,8 @@ populate_joinrel_with_paths(PlannerInfo *root, RelOptInfo *rel1, /* Apply partitionwise join technique, if possible. */ try_partitionwise_join(root, rel1, rel2, joinrel, sjinfo, restrictlist); + + populate_joinrel_uniquekeys(root, joinrel, rel1, rel2, restrictlist, sjinfo->jointype); } diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index bd9a176d7d..139278829b 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -33,7 +33,6 @@ static bool pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys); static bool matches_boolean_partition_clause(RestrictInfo *rinfo, RelOptInfo *partrel, int partkeycol); -static Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); static bool right_merge_direction(PlannerInfo *root, PathKey *pathkey); @@ -1035,7 +1034,7 @@ convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel, * We need this to ensure that we don't return pathkeys describing values * that are unavailable above the level of the subquery scan. */ -static Var * +Var * find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle) { ListCell *lc; diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c new file mode 100644 index 0000000000..77ed2b2eff --- /dev/null +++ b/src/backend/optimizer/path/uniquekeys.c @@ -0,0 +1,1135 @@ +/*------------------------------------------------------------------------- + * + * uniquekeys.c + * Utilities for matching and building unique keys + * + * Portions Copyright (c) 2020, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/optimizer/path/uniquekeys.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "nodes/makefuncs.h" +#include "nodes/nodeFuncs.h" +#include "optimizer/pathnode.h" +#include "optimizer/paths.h" +#include "optimizer/appendinfo.h" +#include "optimizer/optimizer.h" +#include "optimizer/tlist.h" +#include "rewrite/rewriteManip.h" + + +/* + * This struct is used to help populate_joinrel_uniquekeys. + * + * added_to_joinrel is true if a uniquekey (from outerrel or innerrel) + * has been added to joinrel. + * useful is true if the exprs of the uniquekey still appears in joinrel. + */ +typedef struct UniqueKeyContextData +{ + UniqueKey *uniquekey; + bool added_to_joinrel; + bool useful; +} *UniqueKeyContext; + +static List *initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel); +static bool innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + bool reverse); + +static List *get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals); +static List *get_exprs_from_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + UniqueKey *ukey); +static void add_uniquekey_for_onerow(RelOptInfo *rel); +static bool add_combined_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype); + +/* Used for unique indexes checking for partitioned table */ +static bool index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind); +static IndexOptInfo *simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from); +static bool simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2); +static void adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_index); + +/* Helper function for grouped relation and distinct relation. */ +static void add_uniquekey_from_sortgroups(PlannerInfo *root, + RelOptInfo *rel, + List *sortgroups); + +/* + * populate_baserel_uniquekeys + * Populate 'baserel' uniquekeys list by looking at the rel's unique index + * and baserestrictinfo + */ +void +populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List *indexlist) +{ + ListCell *lc; + List *matched_uniq_indexes = NIL; + + /* Attrs appears in rel->reltarget->exprs. */ + Bitmapset *used_attrs = NULL; + + List *const_exprs = NIL; + List *expr_opfamilies = NIL; + + Assert(baserel->rtekind == RTE_RELATION); + + foreach(lc, indexlist) + { + IndexOptInfo *ind = (IndexOptInfo *) lfirst(lc); + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + matched_uniq_indexes = lappend(matched_uniq_indexes, ind); + } + + if (matched_uniq_indexes == NIL) + return; + + /* Check which attrs is used in baserel->reltarget */ + pull_varattnos((Node *)baserel->reltarget->exprs, baserel->relid, &used_attrs); + + /* Check which attrno is used at a mergeable const filter */ + foreach(lc, baserel->baserestrictinfo) + { + RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); + + if (rinfo->mergeopfamilies == NIL) + continue; + + if (bms_is_empty(rinfo->left_relids)) + { + const_exprs = lappend(const_exprs, get_rightop(rinfo->clause)); + } + else if (bms_is_empty(rinfo->right_relids)) + { + const_exprs = lappend(const_exprs, get_leftop(rinfo->clause)); + } + else + continue; + + expr_opfamilies = lappend(expr_opfamilies, rinfo->mergeopfamilies); + } + + foreach(lc, matched_uniq_indexes) + { + bool multi_nullvals, useful; + List *exprs = get_exprs_from_uniqueindex(lfirst_node(IndexOptInfo, lc), + const_exprs, + expr_opfamilies, + used_attrs, + &useful, + &multi_nullvals); + if (useful) + { + if (exprs == NIL) + { + /* All the columns in Unique Index matched with a restrictinfo */ + add_uniquekey_for_onerow(baserel); + return; + } + baserel->uniquekeys = lappend(baserel->uniquekeys, + makeUniqueKey(exprs, multi_nullvals)); + } + } +} + + +/* + * populate_partitionedrel_uniquekeys + * The UniqueKey on partitionrel comes from 2 cases: + * 1). Only one partition is involved in this query, the unique key can be + * copied to parent rel from childrel. + * 2). There are some unique index which includes partition key and exists + * in all the related partitions. + * We never mind rule 2 if we hit rule 1. + */ + +void +populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels) +{ + ListCell *lc; + List *global_uniq_indexlist = NIL; + RelOptInfo *childrel; + bool is_first = true; + + Assert(IS_PARTITIONED_REL(rel)); + + if (childrels == NIL) + return; + + /* + * If there is only one partition used in this query, the UniqueKey in childrel is + * still valid in parent level, but we need convert the format from child expr to + * parent expr. + */ + if (list_length(childrels) == 1) + { + /* Check for Rule 1 */ + RelOptInfo *childrel = linitial_node(RelOptInfo, childrels); + ListCell *lc; + Assert(childrel->reloptkind == RELOPT_OTHER_MEMBER_REL); + if (relation_is_onerow(childrel)) + { + add_uniquekey_for_onerow(rel); + return; + } + + foreach(lc, childrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + AppendRelInfo *appinfo = find_appinfo_by_child(root, childrel->relid); + List *parent_exprs = NIL; + bool can_reuse = true; + ListCell *lc2; + foreach(lc2, ukey->exprs) + { + Var *var = (Var *)lfirst(lc2); + /* + * If the expr comes from a expression, it is hard to build the expression + * in parent so ignore that case for now. + */ + if(!IsA(var, Var)) + { + can_reuse = false; + break; + } + /* Convert it to parent var */ + parent_exprs = lappend(parent_exprs, find_parent_var(appinfo, var)); + } + if (can_reuse) + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(parent_exprs, + ukey->multi_nullvals)); + } + } + else + { + /* Check for rule 2 */ + childrel = linitial_node(RelOptInfo, childrels); + foreach(lc, childrel->indexlist) + { + IndexOptInfo *ind = lfirst(lc); + IndexOptInfo *modified_index; + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + + /* + * During simple_copy_indexinfo_to_parent, we need to convert var from + * child var to parent var, index on expression is too complex to handle. + * so ignore it for now. + */ + if (ind->indexprs != NIL) + continue; + + modified_index = simple_copy_indexinfo_to_parent(root, rel, ind); + /* + * If the unique index doesn't contain partkey, then it is unique + * on this partition only, so it is useless for us. + */ + if (!index_constains_partkey(rel, modified_index)) + continue; + + global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index); + } + + if (global_uniq_indexlist != NIL) + { + foreach(lc, childrels) + { + RelOptInfo *child = lfirst(lc); + if (is_first) + { + is_first = false; + continue; + } + adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist); + } + /* Now we have a list of unique index which are exactly same on all childrels, + * Set the UniqueKey just like it is non-partition table + */ + populate_baserel_uniquekeys(root, rel, global_uniq_indexlist); + } + } +} + + +/* + * populate_distinctrel_uniquekeys + */ +void +populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel) +{ + /* The unique key before the distinct is still valid. */ + distinctrel->uniquekeys = list_copy(inputrel->uniquekeys); + add_uniquekey_from_sortgroups(root, distinctrel, root->parse->distinctClause); +} + +/* + * populate_grouprel_uniquekeys + */ +void +populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel) + +{ + Query *parse = root->parse; + bool input_ukey_added = false; + ListCell *lc; + + if (relation_is_onerow(inputrel)) + { + add_uniquekey_for_onerow(grouprel); + return; + } + if (parse->groupingSets) + return; + + /* A Normal group by without grouping set. */ + if (parse->groupClause) + { + /* + * Current even the groupby clause is Unique already, but if query has aggref + * We have to create grouprel still. To keep the UnqiueKey short, we will check + * the UniqueKey of input_rel still valid, if so we reuse it. + */ + foreach(lc, inputrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (list_is_subset(ukey->exprs, grouprel->reltarget->exprs)) + { + grouprel->uniquekeys = lappend(grouprel->uniquekeys, + ukey); + input_ukey_added = true; + } + } + if (!input_ukey_added) + /* + * group by clause must be a super-set of grouprel->reltarget->exprs except the + * aggregation expr, so if such exprs is unique already, no bother to generate + * new uniquekey for group by exprs. + */ + add_uniquekey_from_sortgroups(root, + grouprel, + root->parse->groupClause); + } + else + /* It has aggregation but without a group by, so only one row returned */ + add_uniquekey_for_onerow(grouprel); +} + +/* + * simple_copy_uniquekeys + * Using a function for the one-line code makes us easy to check where we simply + * copied the uniquekey. + */ +void +simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel) +{ + newrel->uniquekeys = oldrel->uniquekeys; +} + +/* + * populate_unionrel_uniquekeys + */ +void +populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel) +{ + ListCell *lc; + List *exprs = NIL; + + Assert(unionrel->uniquekeys == NIL); + + foreach(lc, unionrel->reltarget->exprs) + { + exprs = lappend(exprs, lfirst(lc)); + } + + if (exprs == NIL) + /* SQL: select union select; is valid, we need to handle it here. */ + add_uniquekey_for_onerow(unionrel); + else + unionrel->uniquekeys = lappend(unionrel->uniquekeys, + makeUniqueKey(exprs,false)); + +} + +/* + * populate_joinrel_uniquekeys + * + * populate uniquekeys for joinrel. We will check each relation to see if its + * UniqueKey is still valid via innerrel_keeps_unique, if so, we add it to + * joinrel. The multi_nullvals field will be changed to true for some outer + * join cases and one-row UniqueKey needs to be converted to normal UniqueKey + * for the same case as well. + * For the uniquekey in either baserel which can't be unique after join, we still + * check to see if combination of UniqueKeys from both side is still useful for us. + * if yes, we add it to joinrel as well. + */ +void +populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *outerrel, RelOptInfo *innerrel, + List *restrictlist, JoinType jointype) +{ + ListCell *lc, *lc2; + List *clause_list = NIL; + List *outerrel_ukey_ctx; + List *innerrel_ukey_ctx; + bool inner_onerow, outer_onerow; + bool mergejoin_allowed; + + /* Care about the outerrel relation only for SEMI/ANTI join */ + if (jointype == JOIN_SEMI || jointype == JOIN_ANTI) + { + foreach(lc, outerrel->uniquekeys) + { + UniqueKey *uniquekey = lfirst_node(UniqueKey, lc); + if (list_is_subset(uniquekey->exprs, joinrel->reltarget->exprs)) + joinrel->uniquekeys = lappend(joinrel->uniquekeys, uniquekey); + } + return; + } + + Assert(jointype == JOIN_LEFT || jointype == JOIN_FULL || jointype == JOIN_INNER); + + /* Fast path */ + if (innerrel->uniquekeys == NIL || outerrel->uniquekeys == NIL) + return; + + inner_onerow = relation_is_onerow(innerrel); + outer_onerow = relation_is_onerow(outerrel); + + outerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(outerrel); + innerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(innerrel); + + clause_list = select_mergejoin_clauses(root, joinrel, outerrel, innerrel, + restrictlist, jointype, + &mergejoin_allowed); + + if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true /* reverse */)) + { + bool outer_impact = jointype == JOIN_FULL; + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + /* Outer relation has one row, and the unique key is not duplicated after join, + * the joinrel will still has one row unless the jointype == JOIN_FULL. + */ + if (outer_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (outer_onerow) + { + /* + * The onerow outerrel becomes multi rows and multi_nullvals + * will be changed to true. We also need to set the exprs correctly since it + * can't be NIL any more. + */ + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, outerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Change multi_nullvals to true due to the full join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, true)); + else + /* Just reuse it */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + } + ctx->added_to_joinrel = true; + } + } + + if (innerrel_keeps_unique(root, outerrel, innerrel, clause_list, false)) + { + bool outer_impact = jointype == JOIN_FULL || jointype == JOIN_LEFT;; + + foreach(lc, innerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + if (inner_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (inner_onerow) + { + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, innerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Need to change multi_nullvals to true due to the outer join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, + true)); + else + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + + } + ctx->added_to_joinrel = true; + } + } + + /* + * The combination of the UniqueKey from both sides is unique as well regardless + * of join type, but no bother to add it if its subset has been added to joinrel + * already or it is not useful for the joinrel. + */ + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx1 = (UniqueKeyContext) lfirst(lc); + if (ctx1->added_to_joinrel || !ctx1->useful) + continue; + foreach(lc2, innerrel_ukey_ctx) + { + UniqueKeyContext ctx2 = (UniqueKeyContext) lfirst(lc2); + if (ctx2->added_to_joinrel || !ctx2->useful) + continue; + if (add_combined_uniquekey(root, joinrel, outerrel, innerrel, + ctx1->uniquekey, ctx2->uniquekey, + jointype)) + /* If we set a onerow UniqueKey to joinrel, we don't need other. */ + return; + } + } +} + + +/* + * convert_subquery_uniquekeys + * + * Covert the UniqueKey in subquery to outer relation. + */ +void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel) +{ + ListCell *lc; + + if (sub_final_rel->uniquekeys == NIL) + return; + + if (relation_is_onerow(sub_final_rel)) + { + add_uniquekey_for_onerow(currel); + return; + } + + Assert(currel->subroot != NULL); + + foreach(lc, sub_final_rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + ListCell *lc; + List *exprs = NIL; + bool ukey_useful = true; + + /* One row case is handled above */ + Assert(ukey->exprs != NIL); + foreach(lc, ukey->exprs) + { + Var *var; + TargetEntry *tle = tlist_member(lfirst(lc), + currel->subroot->processed_tlist); + if (tle == NULL) + { + ukey_useful = false; + break; + } + var = find_var_for_subquery_tle(currel, tle); + if (var == NULL) + { + ukey_useful = false; + break; + } + exprs = lappend(exprs, var); + } + + if (ukey_useful) + currel->uniquekeys = lappend(currel->uniquekeys, + makeUniqueKey(exprs, + ukey->multi_nullvals)); + + } +} + +/* + * innerrel_keeps_unique + * + * Check if Unique key of the innerrel is valid after join. innerrel's UniqueKey + * will be still valid if innerrel's any-column mergeop outrerel's uniquekey + * exists in clause_list. + * + * Note: the clause_list must be a list of mergeable restrictinfo already. + */ +static bool +innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *clause_list, + bool reverse) +{ + ListCell *lc, *lc2, *lc3; + + if (outerrel->uniquekeys == NIL || innerrel->uniquekeys == NIL) + return false; + + /* Check if there is outerrel's uniquekey in mergeable clause. */ + foreach(lc, outerrel->uniquekeys) + { + List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs; + bool clauselist_matchs_all_exprs = true; + foreach(lc2, outer_uq_exprs) + { + Node *outer_uq_expr = lfirst(lc2); + bool find_uq_expr_in_clauselist = false; + foreach(lc3, clause_list) + { + RestrictInfo *rinfo = lfirst_node(RestrictInfo, lc3); + Node *outer_expr; + if (reverse) + outer_expr = rinfo->outer_is_left ? get_rightop(rinfo->clause) : get_leftop(rinfo->clause); + else + outer_expr = rinfo->outer_is_left ? get_leftop(rinfo->clause) : get_rightop(rinfo->clause); + if (equal(outer_expr, outer_uq_expr)) + { + find_uq_expr_in_clauselist = true; + break; + } + } + if (!find_uq_expr_in_clauselist) + { + /* No need to check the next exprs in the current uniquekey */ + clauselist_matchs_all_exprs = false; + break; + } + } + + if (clauselist_matchs_all_exprs) + return true; + } + return false; +} + + +/* + * relation_is_onerow + * Check if it is a one-row relation by checking UniqueKey. + */ +bool +relation_is_onerow(RelOptInfo *rel) +{ + UniqueKey *ukey; + if (rel->uniquekeys == NIL) + return false; + ukey = linitial_node(UniqueKey, rel->uniquekeys); + return ukey->exprs == NIL && list_length(rel->uniquekeys) == 1; +} + +/* + * relation_has_uniquekeys_for + * Returns true if we have proofs that 'rel' cannot return multiple rows with + * the same values in each of 'exprs'. Otherwise returns false. + */ +bool +relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel, + List *exprs, bool allow_multinulls) +{ + ListCell *lc; + + /* + * For UniqueKey->onerow case, the uniquekey->exprs is empty as well + * so we can't rely on list_is_subset to handle this special cases + */ + if (exprs == NIL) + return false; + + foreach(lc, rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (ukey->multi_nullvals && !allow_multinulls) + continue; + if (list_is_subset(ukey->exprs, exprs)) + return true; + } + return false; +} + + +/* + * get_exprs_from_uniqueindex + * + * Return a list of exprs which is unique. set useful to false if this + * unique index is not useful for us. + */ +static List * +get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals) +{ + List *exprs = NIL; + ListCell *indexpr_item; + int c = 0; + + *useful = true; + *multi_nullvals = false; + + indexpr_item = list_head(unique_index->indexprs); + for(c = 0; c < unique_index->ncolumns; c++) + { + int attr = unique_index->indexkeys[c]; + Expr *expr; + bool matched_const = false; + ListCell *lc1, *lc2; + + if(attr > 0) + { + expr = list_nth_node(TargetEntry, unique_index->indextlist, c)->expr; + } + else if (attr == 0) + { + /* Expression index */ + expr = lfirst(indexpr_item); + indexpr_item = lnext(unique_index->indexprs, indexpr_item); + } + else /* attr < 0 */ + { + /* Index on system column is not supported */ + Assert(false); + } + + /* + * Check index_col = Const case with regarding to opfamily checking + * If we can remove the index_col from the final UniqueKey->exprs. + */ + forboth(lc1, const_exprs, lc2, const_expr_opfamilies) + { + if (list_member_oid((List *)lfirst(lc2), unique_index->opfamily[c]) + && match_index_to_operand((Node *) lfirst(lc1), c, unique_index)) + { + matched_const = true; + break; + } + } + + if (matched_const) + continue; + + /* Check if the indexed expr is used in rel */ + if (attr > 0) + { + /* + * Normal Indexed column, if the col is not used, then the index is useless + * for uniquekey. + */ + attr -= FirstLowInvalidHeapAttributeNumber; + + if (!bms_is_member(attr, used_varattrs)) + { + *useful = false; + break; + } + } + else if (!list_member(unique_index->rel->reltarget->exprs, expr)) + { + /* Expression index but the expression is not used in rel */ + *useful = false; + break; + } + + /* check not null property. */ + if (attr == 0) + { + /* We never know if a expression yields null or not */ + *multi_nullvals = true; + } + else if (!bms_is_member(attr, unique_index->rel->notnullattrs) + && !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber, + unique_index->rel->notnullattrs)) + { + *multi_nullvals = true; + } + + exprs = lappend(exprs, expr); + } + return exprs; +} + + +/* + * add_uniquekey_for_onerow + * If we are sure that the relation only returns one row, then all the columns + * are unique. However we don't need to create UniqueKey for every column, we + * just set exprs = NIL and overwrites all the other UniqueKey on this RelOptInfo + * since this one has strongest semantics. + */ +void +add_uniquekey_for_onerow(RelOptInfo *rel) +{ + /* + * We overwrite the previous UniqueKey on purpose since this one has the + * strongest semantic. + */ + rel->uniquekeys = list_make1(makeUniqueKey(NIL, false)); +} + + +/* + * initililze_uniquecontext_for_joinrel + * Return a List of UniqueKeyContext for an inputrel + */ +static List * +initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel) +{ + List *res = NIL; + ListCell *lc; + foreach(lc, inputrel->uniquekeys) + { + UniqueKeyContext context; + context = palloc(sizeof(struct UniqueKeyContextData)); + context->uniquekey = lfirst_node(UniqueKey, lc); + context->added_to_joinrel = false; + context->useful = true; + res = lappend(res, context); + } + return res; +} + + +/* + * get_exprs_from_uniquekey + * Unify the way of get List of exprs from a one-row UniqueKey or + * normal UniqueKey. for the onerow case, every expr in rel1 is a valid + * UniqueKey. Return a List of exprs. + * + * rel1: The relation which you want to get the exprs. + * ukey: The UniqueKey you want to get the exprs. + */ +static List * +get_exprs_from_uniquekey(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *rel1, UniqueKey *ukey) +{ + ListCell *lc; + bool onerow = rel1 != NULL && relation_is_onerow(rel1); + + List *res = NIL; + Assert(onerow || ukey); + if (onerow) + { + /* Only cares about the exprs still exist in joinrel */ + foreach(lc, joinrel->reltarget->exprs) + { + Bitmapset *relids = pull_varnos(root, lfirst(lc)); + if (bms_is_subset(relids, rel1->relids)) + { + res = lappend(res, list_make1(lfirst(lc))); + } + } + } + else + { + res = list_make1(ukey->exprs); + } + return res; +} + +/* + * Partitioned table Unique Keys. + * The partition table unique key is maintained as: + * 1. The index must be unique as usual. + * 2. The index must contains partition key. + * 3. The index must exist on all the child rel. see simple_indexinfo_equal for + * how we compare it. + */ + +/* + * index_constains_partkey + * return true if the index contains the partiton key. + */ +static bool +index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind) +{ + ListCell *lc; + int i; + Assert(IS_PARTITIONED_REL(partrel)); + Assert(partrel->part_scheme->partnatts > 0); + + for(i = 0; i < partrel->part_scheme->partnatts; i++) + { + Node *part_expr = linitial(partrel->partexprs[i]); + bool found_in_index = false; + foreach(lc, ind->indextlist) + { + Expr *index_expr = lfirst_node(TargetEntry, lc)->expr; + if (equal(index_expr, part_expr)) + { + found_in_index = true; + break; + } + } + if (!found_in_index) + return false; + } + return true; +} + +/* + * simple_indexinfo_equal + * + * Used to check if the 2 index is same as each other. The index here + * is COPIED from childrel and did some tiny changes(see + * simple_copy_indexinfo_to_parent) + */ +static bool +simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2) +{ + Size oid_cmp_len = sizeof(Oid) * ind1->ncolumns; + + return ind1->ncolumns == ind2->ncolumns && + ind1->unique == ind2->unique && + memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0 && + memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0 && + memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0 && + memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0 && + equal(get_tlist_exprs(ind1->indextlist, true), + get_tlist_exprs(ind2->indextlist, true)); +} + + +/* + * The below macros are used for simple_copy_indexinfo_to_parent which is so + * customized that I don't want to put it to copyfuncs.c. So copy it here. + */ +#define COPY_POINTER_FIELD(fldname, sz) \ + do { \ + Size _size = (sz); \ + newnode->fldname = palloc(_size); \ + memcpy(newnode->fldname, from->fldname, _size); \ + } while (0) + +#define COPY_NODE_FIELD(fldname) \ + (newnode->fldname = copyObjectImpl(from->fldname)) + +#define COPY_SCALAR_FIELD(fldname) \ + (newnode->fldname = from->fldname) + + +/* + * simple_copy_indexinfo_to_parent (from partition) + * Copy the IndexInfo from child relation to parent relation with some modification, + * which is used to test: + * 1. If the same index exists in all the childrels. + * 2. If the parentrel->reltarget/basicrestrict info matches this index. + */ +static IndexOptInfo * +simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from) +{ + IndexOptInfo *newnode = makeNode(IndexOptInfo); + AppendRelInfo *appinfo = find_appinfo_by_child(root, from->rel->relid); + ListCell *lc; + int idx = 0; + + COPY_SCALAR_FIELD(ncolumns); + COPY_SCALAR_FIELD(nkeycolumns); + COPY_SCALAR_FIELD(unique); + COPY_SCALAR_FIELD(immediate); + /* We just need to know if it is NIL or not */ + COPY_SCALAR_FIELD(indpred); + COPY_SCALAR_FIELD(predOK); + COPY_POINTER_FIELD(indexkeys, from->ncolumns * sizeof(int)); + COPY_POINTER_FIELD(indexcollations, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opfamily, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opcintype, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(sortopfamily, from->ncolumns * sizeof(Oid)); + COPY_NODE_FIELD(indextlist); + + /* Convert index exprs on child expr to expr on parent */ + foreach(lc, newnode->indextlist) + { + TargetEntry *tle = lfirst_node(TargetEntry, lc); + /* Index on expression is ignored */ + Assert(IsA(tle->expr, Var)); + tle->expr = (Expr *) find_parent_var(appinfo, (Var *) tle->expr); + newnode->indexkeys[idx] = castNode(Var, tle->expr)->varattno; + idx++; + } + newnode->rel = parentrel; + return newnode; +} + +/* + * adjust_partition_unique_indexlist + * + * global_unique_indexes: At the beginning, it contains the copy & modified + * unique index from the first partition. And then check if each index in it still + * exists in the following partitions. If no, remove it. at last, it has an + * index list which exists in all the partitions. + */ +static void +adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_indexes) +{ + ListCell *lc, *lc2; + foreach(lc, *global_unique_indexes) + { + IndexOptInfo *g_ind = lfirst_node(IndexOptInfo, lc); + bool found_in_child = false; + + foreach(lc2, childrel->indexlist) + { + IndexOptInfo *p_ind = lfirst_node(IndexOptInfo, lc2); + IndexOptInfo *p_ind_copy; + if (!p_ind->unique || !p_ind->immediate || + (p_ind->indpred != NIL && !p_ind->predOK)) + continue; + p_ind_copy = simple_copy_indexinfo_to_parent(root, parentrel, p_ind); + if (simple_indexinfo_equal(p_ind_copy, g_ind)) + { + found_in_child = true; + break; + } + } + if (!found_in_child) + /* The index doesn't exist in childrel, remove it from global_unique_indexes */ + *global_unique_indexes = foreach_delete_current(*global_unique_indexes, lc); + } +} + +/* Helper function for groupres/distinctrel */ +static void +add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgroups) +{ + Query *parse = root->parse; + List *exprs; + + /* + * XXX: If there are some vars which is not in current levelsup, the semantic is + * imprecise, should we avoid it or not? levelsup = 1 is just a demo, maybe we need to + * check every level other than 0, if so, looks we have to write another + * pull_var_walker. + */ + List *upper_vars = pull_vars_of_level((Node*)sortgroups, 1); + + if (upper_vars != NIL) + return; + + exprs = get_sortgrouplist_exprs(sortgroups, parse->targetList); + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(exprs, + false /* sortgroupclause can't be multi_nullvals */)); +} + + +/* + * add_combined_uniquekey + * The combination of both UniqueKeys is a valid UniqueKey for joinrel no matter + * the jointype. + */ +bool +add_combined_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype) +{ + + ListCell *lc1, *lc2; + + /* Either side has multi_nullvals or we have outer join, + * the combined UniqueKey has multi_nullvals */ + bool multi_nullvals = outer_ukey->multi_nullvals || + inner_ukey->multi_nullvals || IS_OUTER_JOIN(jointype); + + /* The only case we can get onerow joinrel after join */ + if (relation_is_onerow(outer_rel) + && relation_is_onerow(inner_rel) + && jointype == JOIN_INNER) + { + add_uniquekey_for_onerow(joinrel); + return true; + } + + foreach(lc1, get_exprs_from_uniquekey(root, joinrel, outer_rel, outer_ukey)) + { + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, inner_rel, inner_ukey)) + { + List *exprs = list_concat_copy(lfirst_node(List, lc1), lfirst_node(List, lc2)); + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(exprs, + multi_nullvals)); + } + } + return false; +} diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index 424d25cbd5..8d8e493f5c 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -2387,6 +2387,8 @@ grouping_planner(PlannerInfo *root, bool inheritance_update, add_path(final_rel, path); } + simple_copy_uniquekeys(current_rel, final_rel); + /* * Generate partial paths for final_rel, too, if outer query levels might * be able to make use of them. @@ -3899,6 +3901,8 @@ create_grouping_paths(PlannerInfo *root, } set_cheapest(grouped_rel); + + populate_grouprel_uniquekeys(root, grouped_rel, input_rel); return grouped_rel; } @@ -4621,7 +4625,7 @@ create_window_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(window_rel); - + simple_copy_uniquekeys(input_rel, window_rel); return window_rel; } @@ -4935,7 +4939,7 @@ create_distinct_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(distinct_rel); - + populate_distinctrel_uniquekeys(root, input_rel, distinct_rel); return distinct_rel; } @@ -5196,6 +5200,8 @@ create_ordered_paths(PlannerInfo *root, */ Assert(ordered_rel->pathlist != NIL); + simple_copy_uniquekeys(input_rel, ordered_rel); + return ordered_rel; } @@ -6073,6 +6079,9 @@ adjust_paths_for_srfs(PlannerInfo *root, RelOptInfo *rel, if (list_length(targets) == 1) return; + /* UniqueKey is not valid after handling the SRF. */ + rel->uniquekeys = NIL; + /* * Stack SRF-evaluation nodes atop each path for the rel. * diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c index becdcbb872..b7626545bf 100644 --- a/src/backend/optimizer/prep/prepunion.c +++ b/src/backend/optimizer/prep/prepunion.c @@ -689,6 +689,8 @@ generate_union_paths(SetOperationStmt *op, PlannerInfo *root, /* Undo effects of possibly forcing tuple_fraction to 0 */ root->tuple_fraction = save_fraction; + /* Add the UniqueKeys */ + populate_unionrel_uniquekeys(root, result_rel); return result_rel; } diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c index 86922a273c..6817c9c787 100644 --- a/src/backend/optimizer/util/appendinfo.c +++ b/src/backend/optimizer/util/appendinfo.c @@ -746,3 +746,47 @@ find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos) } return appinfos; } + +/* + * find_appinfo_by_child + * + */ +AppendRelInfo * +find_appinfo_by_child(PlannerInfo *root, Index child_index) +{ + ListCell *lc; + foreach(lc, root->append_rel_list) + { + AppendRelInfo *appinfo = lfirst_node(AppendRelInfo, lc); + if (appinfo->child_relid == child_index) + return appinfo; + } + elog(ERROR, "parent relation cant be found"); + return NULL; +} + +/* + * find_parent_var + * + */ +Var * +find_parent_var(AppendRelInfo *appinfo, Var *child_var) +{ + ListCell *lc; + Var *res = NULL; + Index attno = 1; + foreach(lc, appinfo->translated_vars) + { + Node *child_node = lfirst(lc); + if (equal(child_node, child_var)) + { + res = copyObject(child_var); + res->varattno = attno; + res->varno = appinfo->parent_relid; + } + attno++; + } + if (res == NULL) + elog(ERROR, "parent var can't be found."); + return res; +} diff --git a/src/backend/optimizer/util/inherit.c b/src/backend/optimizer/util/inherit.c index be1c9ddd96..3eec1f4d74 100644 --- a/src/backend/optimizer/util/inherit.c +++ b/src/backend/optimizer/util/inherit.c @@ -736,6 +736,7 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, { Node *onecq = (Node *) lfirst(lc2); bool pseudoconstant; + RestrictInfo *child_rinfo; /* check for pseudoconstant (no Vars or volatile functions) */ pseudoconstant = @@ -747,14 +748,15 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, root->hasPseudoConstantQuals = true; } /* reconstitute RestrictInfo with appropriate properties */ - childquals = lappend(childquals, - make_restrictinfo(root, - (Expr *) onecq, - rinfo->is_pushed_down, - rinfo->outerjoin_delayed, - pseudoconstant, - rinfo->security_level, - NULL, NULL, NULL)); + child_rinfo = make_restrictinfo(root, + (Expr *) onecq, + rinfo->is_pushed_down, + rinfo->outerjoin_delayed, + pseudoconstant, + rinfo->security_level, + NULL, NULL, NULL); + child_rinfo->mergeopfamilies = rinfo->mergeopfamilies; + childquals = lappend(childquals, child_rinfo); /* track minimum security level among child quals */ cq_min_security = Min(cq_min_security, rinfo->security_level); } diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 48a7ebfe45..4fe1824eb0 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -16,6 +16,7 @@ #include "nodes/execnodes.h" #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" extern A_Expr *makeA_Expr(A_Expr_Kind kind, List *name, @@ -106,4 +107,6 @@ extern GroupingSet *makeGroupingSet(GroupingSetKind kind, List *content, int loc extern VacuumRelation *makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols); +extern UniqueKey* makeUniqueKey(List *exprs, bool multi_nullvals); + #endif /* MAKEFUNC_H */ diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index e22df890ef..3320273ac1 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -265,6 +265,7 @@ typedef enum NodeTag T_EquivalenceMember, T_PathKey, T_PathTarget, + T_UniqueKey, T_RestrictInfo, T_IndexClause, T_PlaceHolderVar, diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index 0d61f04d27..80561b1003 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -739,6 +739,7 @@ typedef struct RelOptInfo QualCost baserestrictcost; /* cost of evaluating the above */ Index baserestrict_min_security; /* min security_level found in * baserestrictinfo */ + List *uniquekeys; /* List of UniqueKey */ List *joininfo; /* RestrictInfo structures for join clauses * involving this rel */ bool has_eclass_joins; /* T means joininfo is incomplete */ @@ -1059,6 +1060,28 @@ typedef struct PathKey } PathKey; +/* + * UniqueKey + * + * Represents the unique properties held by a RelOptInfo. + * + * exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL + * is a special case of UniqueKey, which means there is only 1 row in that + * relation. + * multi_nullvals: true means multi null values may exist in these exprs, so the + * uniqueness is not guaranteed in this case. This field is necessary for + * remove_useless_join & reduce_unique_semijoins where we don't mind these + * duplicated NULL values. It is set to true for 2 cases. One is a unique key + * from a unique index but the related column is nullable. The other one is for + * outer join. see populate_joinrel_uniquekeys for detail. + */ +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + /* * PathTarget * @@ -2496,7 +2519,7 @@ typedef enum * * flags indicating what kinds of grouping are possible. * partial_costs_set is true if the agg_partial_costs and agg_final_costs - * have been initialized. + * have been initialized. * agg_partial_costs gives partial aggregation costs. * agg_final_costs gives finalization costs. * target_parallel_safe is true if target is parallel safe. @@ -2526,8 +2549,8 @@ typedef struct * limit_tuples is an estimated bound on the number of output tuples, * or -1 if no LIMIT or couldn't estimate. * count_est and offset_est are the estimated values of the LIMIT and OFFSET - * expressions computed by preprocess_limit() (see comments for - * preprocess_limit() for more information). + * expressions computed by preprocess_limit() (see comments for + * preprocess_limit() for more information). */ typedef struct { diff --git a/src/include/nodes/pg_list.h b/src/include/nodes/pg_list.h index 404e03f132..85c65e910c 100644 --- a/src/include/nodes/pg_list.h +++ b/src/include/nodes/pg_list.h @@ -566,6 +566,16 @@ extern pg_nodiscard List *list_delete_first(List *list); extern pg_nodiscard List *list_delete_last(List *list); extern pg_nodiscard List *list_delete_nth_cell(List *list, int n); extern pg_nodiscard List *list_delete_cell(List *list, ListCell *cell); +extern bool list_is_subset(const List *members, const List *target); + +extern List *list_delete(List *list, void *datum); +extern List *list_delete_ptr(List *list, void *datum); +extern List *list_delete_int(List *list, int datum); +extern List *list_delete_oid(List *list, Oid datum); +extern List *list_delete_first(List *list); +extern List *list_delete_last(List *list); +extern List *list_delete_nth_cell(List *list, int n); +extern List *list_delete_cell(List *list, ListCell *cell); extern List *list_union(const List *list1, const List *list2); extern List *list_union_ptr(const List *list1, const List *list2); diff --git a/src/include/optimizer/appendinfo.h b/src/include/optimizer/appendinfo.h index 4cbf8c26cc..43b8d05f69 100644 --- a/src/include/optimizer/appendinfo.h +++ b/src/include/optimizer/appendinfo.h @@ -32,4 +32,7 @@ extern Relids adjust_child_relids_multilevel(PlannerInfo *root, Relids relids, extern AppendRelInfo **find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos); +extern AppendRelInfo *find_appinfo_by_child(PlannerInfo *root, Index child_index); +extern Var *find_parent_var(AppendRelInfo *appinfo, Var *child_var); + #endif /* APPENDINFO_H */ diff --git a/src/include/optimizer/optimizer.h b/src/include/optimizer/optimizer.h index d587952b7d..843aafa51e 100644 --- a/src/include/optimizer/optimizer.h +++ b/src/include/optimizer/optimizer.h @@ -23,6 +23,7 @@ #define OPTIMIZER_H #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" /* Test if an expression node represents a SRF call. Beware multiple eval! */ #define IS_SRF_CALL(node) \ @@ -168,6 +169,7 @@ extern TargetEntry *get_sortgroupref_tle(Index sortref, List *targetList); extern TargetEntry *get_sortgroupclause_tle(SortGroupClause *sgClause, List *targetList); +extern Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); extern Node *get_sortgroupclause_expr(SortGroupClause *sgClause, List *targetList); extern List *get_sortgrouplist_exprs(List *sgClauses, diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h index 035d3e1206..1adf99c9ee 100644 --- a/src/include/optimizer/paths.h +++ b/src/include/optimizer/paths.h @@ -247,5 +247,48 @@ extern PathKey *make_canonical_pathkey(PlannerInfo *root, int strategy, bool nulls_first); extern void add_paths_to_append_rel(PlannerInfo *root, RelOptInfo *rel, List *live_childrels); +extern List *select_mergejoin_clauses(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + JoinType jointype, + bool *mergejoin_allowed); + +/* + * uniquekeys.c + * Utilities for matching and building unique keys + */ +extern void populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List* unique_index_list); +extern void populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels); +extern void populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel); +extern void populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel); +extern void populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel); +extern void simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel); +extern void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel); +extern void populate_joinrel_uniquekeys(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + RelOptInfo *rel2, + List *restrictlist, + JoinType jointype); + +extern bool relation_has_uniquekeys_for(PlannerInfo *root, + RelOptInfo *rel, + List *exprs, + bool allow_multinulls); +extern bool relation_is_onerow(RelOptInfo *rel); #endif /* PATHS_H */ -- 2.26.2 --yh7cw345mbttadjz Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v38-0003-Extend-UniqueKeys.patch" ^ permalink raw reply [nested|flat] 91+ messages in thread
* [PATCH v38 2/6] Introduce UniqueKey attributes on RelOptInfo struct. @ 2020-05-11 07:50 一挃 <[email protected]> 0 siblings, 0 replies; 91+ messages in thread From: 一挃 @ 2020-05-11 07:50 UTC (permalink / raw) UniqueKey is a set of exprs on RelOptInfo which represents the exprs will be unique on the given RelOptInfo. You can see README.uniquekey for more information. --- src/backend/nodes/copyfuncs.c | 13 + src/backend/nodes/list.c | 31 + src/backend/nodes/makefuncs.c | 13 + src/backend/nodes/outfuncs.c | 11 + src/backend/nodes/readfuncs.c | 10 + src/backend/optimizer/path/Makefile | 3 +- src/backend/optimizer/path/README.uniquekey | 131 +++ src/backend/optimizer/path/allpaths.c | 10 + src/backend/optimizer/path/joinpath.c | 9 +- src/backend/optimizer/path/joinrels.c | 2 + src/backend/optimizer/path/pathkeys.c | 3 +- src/backend/optimizer/path/uniquekeys.c | 1135 +++++++++++++++++++ src/backend/optimizer/plan/planner.c | 13 +- src/backend/optimizer/prep/prepunion.c | 2 + src/backend/optimizer/util/appendinfo.c | 44 + src/backend/optimizer/util/inherit.c | 18 +- src/include/nodes/makefuncs.h | 3 + src/include/nodes/nodes.h | 1 + src/include/nodes/pathnodes.h | 29 +- src/include/nodes/pg_list.h | 10 + src/include/optimizer/appendinfo.h | 3 + src/include/optimizer/optimizer.h | 2 + src/include/optimizer/paths.h | 43 + 23 files changed, 1515 insertions(+), 24 deletions(-) create mode 100644 src/backend/optimizer/path/README.uniquekey create mode 100644 src/backend/optimizer/path/uniquekeys.c diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..75c1c5e824 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2296,6 +2296,16 @@ _copyPathKey(const PathKey *from) return newnode; } +static UniqueKey * +_copyUniqueKey(const UniqueKey *from) +{ + UniqueKey *newnode = makeNode(UniqueKey); + + COPY_NODE_FIELD(exprs); + COPY_SCALAR_FIELD(multi_nullvals); + + return newnode; +} /* * _copyRestrictInfo */ @@ -5220,6 +5230,9 @@ copyObjectImpl(const void *from) case T_PathKey: retval = _copyPathKey(from); break; + case T_UniqueKey: + retval = _copyUniqueKey(from); + break; case T_RestrictInfo: retval = _copyRestrictInfo(from); break; diff --git a/src/backend/nodes/list.c b/src/backend/nodes/list.c index dbf6b30233..ca099495a1 100644 --- a/src/backend/nodes/list.c +++ b/src/backend/nodes/list.c @@ -702,6 +702,37 @@ list_member_oid(const List *list, Oid datum) return false; } +/* + * return true iff every entry in "members" list is also present + * in the "target" list. + */ +bool +list_is_subset(const List *members, const List *target) +{ + const ListCell *lc1, *lc2; + + Assert(IsPointerList(members)); + Assert(IsPointerList(target)); + check_list_invariants(members); + check_list_invariants(target); + + foreach(lc1, members) + { + bool found = false; + foreach(lc2, target) + { + if (equal(lfirst(lc1), lfirst(lc2))) + { + found = true; + break; + } + } + if (!found) + return false; + } + return true; +} + /* * Delete the n'th cell (counting from 0) in list. * diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 01c110cd2f..40415d0f5b 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -815,3 +815,16 @@ makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols) v->va_cols = va_cols; return v; } + + +/* + * makeUniqueKey + */ +UniqueKey* +makeUniqueKey(List *exprs, bool multi_nullvals) +{ + UniqueKey * ukey = makeNode(UniqueKey); + ukey->exprs = exprs; + ukey->multi_nullvals = multi_nullvals; + return ukey; +} diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..44154cde6a 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2456,6 +2456,14 @@ _outPathKey(StringInfo str, const PathKey *node) WRITE_BOOL_FIELD(pk_nulls_first); } +static void +_outUniqueKey(StringInfo str, const UniqueKey *node) +{ + WRITE_NODE_TYPE("UNIQUEKEY"); + WRITE_NODE_FIELD(exprs); + WRITE_BOOL_FIELD(multi_nullvals); +} + static void _outPathTarget(StringInfo str, const PathTarget *node) { @@ -4198,6 +4206,9 @@ outNode(StringInfo str, const void *obj) case T_PathKey: _outPathKey(str, obj); break; + case T_UniqueKey: + _outUniqueKey(str, obj); + break; case T_PathTarget: _outPathTarget(str, obj); break; diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index c5e136e9c3..b3e212bf1c 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -492,6 +492,14 @@ _readSetOperationStmt(void) READ_DONE(); } +static UniqueKey * +_readUniqueKey(void) +{ + READ_LOCALS(UniqueKey); + READ_NODE_FIELD(exprs); + READ_BOOL_FIELD(multi_nullvals); + READ_DONE(); +} /* * Stuff from primnodes.h. @@ -2717,6 +2725,8 @@ parseNodeString(void) return_value = _readCommonTableExpr(); else if (MATCH("SETOPERATIONSTMT", 16)) return_value = _readSetOperationStmt(); + else if (MATCH("UNIQUEKEY", 9)) + return_value = _readUniqueKey(); else if (MATCH("ALIAS", 5)) return_value = _readAlias(); else if (MATCH("RANGEVAR", 8)) diff --git a/src/backend/optimizer/path/Makefile b/src/backend/optimizer/path/Makefile index 1e199ff66f..7b9820c25f 100644 --- a/src/backend/optimizer/path/Makefile +++ b/src/backend/optimizer/path/Makefile @@ -21,6 +21,7 @@ OBJS = \ joinpath.o \ joinrels.o \ pathkeys.o \ - tidpath.o + tidpath.o \ + uniquekeys.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/optimizer/path/README.uniquekey b/src/backend/optimizer/path/README.uniquekey new file mode 100644 index 0000000000..5eac761995 --- /dev/null +++ b/src/backend/optimizer/path/README.uniquekey @@ -0,0 +1,131 @@ +1. What is UniqueKey? +We can think UniqueKey is a set of exprs for a RelOptInfo, which we are insure +that doesn't yields same result among all the rows. The simplest UniqueKey +format is primary key. + +However we define the UnqiueKey as below. + +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + +exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL +is a special case of UniqueKey, which means there is only one row in that +relation.it has a stronger semantic than others. like SELECT uk FROM t; uk is +normal unique key and may have different values. SELECT colx FROM t WHERE uk = +const. colx is unique AND we have only 1 value. This field can used for +innerrel_is_unique. this logic is handled specially in add_uniquekey_for_onerow +function. + +multi_nullvals: true means multi null values may exist in these exprs, so the +uniqueness is not guaranteed in this case. This field is necessary for +remove_useless_join & reduce_unique_semijoins where we don't mind these +duplicated NULL values. It is set to true for 2 cases. One is a unique key +from a unique index but the related column is nullable. The other one is for +outer join. see populate_joinrel_uniquekeys for detail. + + +The UniqueKey can be used at the following cases at least: +1. remove_useless_joins. +2. reduce_semianti_joins +3. remove distinct node if distinct clause is unique. +4. remove aggnode if group by clause is unique. +5. Index Skip Scan (WIP) +6. Aggregation Push Down without 2 phase aggregation if the join can't + duplicated the aggregated rows. (work in progress feature) + +2. How is it maintained? + +We have a set of populate_xxx_unqiuekeys functions to maintain the uniquekey on +various cases. xxx includes baserel, joinrel, partitionedrel, distinctrel, +groupedrel, unionrel. and we also need to convert the uniquekey from subquery +to outer relation, which is what convert_subquery_uniquekeys does. + +1. The first part is about baserel. We handled 3 cases. suppose we have Unique +Index on (a, b). + +1. SELECT a, b FROM t. UniqueKey (a, b) +2. SELECT a FROM t WHERE b = 1; UniqueKey (a) +3. SELECT .. FROM t WHERE a = 1 AND b = 1; UniqueKey (NIL). onerow case, every + column is Unique. + +2. The next part is joinrel, this part is most error-prone, we simplified the rules +like below: +1. If the relation's UniqueKey can't be duplicated after join, then is will be + still valid for the join rel. The function we used here is + innerrel_keeps_unique. The basic idea is innerrel.any_col = outer.uk. + +2. If the UnqiueKey can't keep valid via the rule 1, the combination of the + UniqueKey from both sides are valid for sure. We can prove this as: if the + unique exprs from rel1 is duplicated by rel2, the duplicated rows must + contains different unique exprs from rel2. + +More considerations about onerow: +1. If relation with one row and it can't be duplicated, it is still possible + contains mulit_nullvas after outer join. +2. If the either UniqueKey can be duplicated after join, the can get one row + only when both side is one row AND there is no outer join. +3. Whenever the onerow UniqueKey is not a valid any more, we need to convert one + row UniqueKey to normal unique key since we don't store exprs for one-row + relation. get_exprs_from_uniquekeys will be used here. + + +More considerations about multi_nullvals after join: +1. If the original UnqiueKey has multi_nullvals, the final UniqueKey will have + mulit_nullvals in any case. +2. If a unique key doesn't allow mulit_nullvals, after some outer join, it + allows some outer join. + + +3. When we comes to subquery, we need to convert_subquery_unqiuekeys just like +convert_subquery_pathkeys. Only the UniqueKey insides subquery is referenced as +a Var in outer relation will be reused. The relationship between the outerrel.Var +and subquery.exprs is built with outerel->subroot->processed_tlist. + + +4. As for the SRF functions, it will break the uniqueness of uniquekey, However it +is handled in adjust_paths_for_srfs, which happens after the query_planner. so +we will maintain the UniqueKey until there and reset it to NIL at that +places. This can't help on distinct/group by elimination cases but probably help +in some other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is +semantic correctly. + + +5. As for inherit table, we first main the UnqiueKey on childrel as well. But for +partitioned table we need to maintain 2 different kinds of +UnqiueKey. 1). UniqueKey on the parent relation 2). UniqueKey on child +relation for partition wise query. + +Example: +CREATE TABLE p (a int not null, b int not null) partition by list (a); +CREATE TABLE p0 partition of p for values in (1); +CREATE TABLE p1 partition of p for values in (2); + +create unique index p0_b on p0(b); +create unique index p1_b on p1(b); + +Now b is only unique on partition level, so the distinct can't be removed on +the following cases. SELECT DISTINCT b FROM p; + +Another example is SELECT DISTINCT a, b FROM p WHERE a = 1; Since only one +partition is chosen, the UniqueKey on child relation is same as the UniqueKey on +parent relation. + +Another usage of UniqueKey on partition level is it be helpful for +partition-wise join. + +As for the UniqueKey on parent table level, it comes with 2 different ways, +1). the UniqueKey is also derived in UniqueKey index, but the index must be same +in all the related children relations and the unique index must contains +Partition Key in it. Example: + +CREATE UNIQUE INDEX p_ab ON p(a, b); -- where a is the partition key. + +-- Query +SELECT a, b FROM p; the (a, b) is a UniqueKey of p. + +2). If the parent relation has only one childrel, the UniqueKey on childrel is + the UniqueKey on parent as well. diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index 37b4223adb..d52ad59f89 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -579,6 +579,12 @@ set_plain_rel_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) */ check_index_predicates(root, rel); + /* + * Now that we've marked which partial indexes are suitable, we can now + * build the relation's unique keys. + */ + populate_baserel_uniquekeys(root, rel, rel->indexlist); + /* Mark rel with estimated output rows, width, etc */ set_baserel_size_estimates(root, rel); } @@ -1297,6 +1303,8 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, /* Add paths to the append relation. */ add_paths_to_append_rel(root, rel, live_childrels); + if (IS_PARTITIONED_REL(rel)) + populate_partitionedrel_uniquekeys(root, rel, live_childrels); } @@ -2304,6 +2312,8 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, pathkeys, required_outer)); } + convert_subquery_uniquekeys(root, rel, sub_final_rel); + /* If outer rel allows parallelism, do same for partial paths. */ if (rel->consider_parallel && bms_is_empty(required_outer)) { diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c index 57ce97fd53..697ac047fb 100644 --- a/src/backend/optimizer/path/joinpath.c +++ b/src/backend/optimizer/path/joinpath.c @@ -71,13 +71,6 @@ static void consider_parallel_mergejoin(PlannerInfo *root, static void hash_inner_and_outer(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, RelOptInfo *innerrel, JoinType jointype, JoinPathExtraData *extra); -static List *select_mergejoin_clauses(PlannerInfo *root, - RelOptInfo *joinrel, - RelOptInfo *outerrel, - RelOptInfo *innerrel, - List *restrictlist, - JoinType jointype, - bool *mergejoin_allowed); static void generate_mergejoin_paths(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *innerrel, @@ -1927,7 +1920,7 @@ hash_inner_and_outer(PlannerInfo *root, * if it is mergejoinable and involves vars from the two sub-relations * currently of interest. */ -static List * +List * select_mergejoin_clauses(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c index 0dbe2ac726..7271f044ec 100644 --- a/src/backend/optimizer/path/joinrels.c +++ b/src/backend/optimizer/path/joinrels.c @@ -924,6 +924,8 @@ populate_joinrel_with_paths(PlannerInfo *root, RelOptInfo *rel1, /* Apply partitionwise join technique, if possible. */ try_partitionwise_join(root, rel1, rel2, joinrel, sjinfo, restrictlist); + + populate_joinrel_uniquekeys(root, joinrel, rel1, rel2, restrictlist, sjinfo->jointype); } diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index bd9a176d7d..139278829b 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -33,7 +33,6 @@ static bool pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys); static bool matches_boolean_partition_clause(RestrictInfo *rinfo, RelOptInfo *partrel, int partkeycol); -static Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); static bool right_merge_direction(PlannerInfo *root, PathKey *pathkey); @@ -1035,7 +1034,7 @@ convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel, * We need this to ensure that we don't return pathkeys describing values * that are unavailable above the level of the subquery scan. */ -static Var * +Var * find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle) { ListCell *lc; diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c new file mode 100644 index 0000000000..77ed2b2eff --- /dev/null +++ b/src/backend/optimizer/path/uniquekeys.c @@ -0,0 +1,1135 @@ +/*------------------------------------------------------------------------- + * + * uniquekeys.c + * Utilities for matching and building unique keys + * + * Portions Copyright (c) 2020, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/optimizer/path/uniquekeys.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "nodes/makefuncs.h" +#include "nodes/nodeFuncs.h" +#include "optimizer/pathnode.h" +#include "optimizer/paths.h" +#include "optimizer/appendinfo.h" +#include "optimizer/optimizer.h" +#include "optimizer/tlist.h" +#include "rewrite/rewriteManip.h" + + +/* + * This struct is used to help populate_joinrel_uniquekeys. + * + * added_to_joinrel is true if a uniquekey (from outerrel or innerrel) + * has been added to joinrel. + * useful is true if the exprs of the uniquekey still appears in joinrel. + */ +typedef struct UniqueKeyContextData +{ + UniqueKey *uniquekey; + bool added_to_joinrel; + bool useful; +} *UniqueKeyContext; + +static List *initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel); +static bool innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + bool reverse); + +static List *get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals); +static List *get_exprs_from_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + UniqueKey *ukey); +static void add_uniquekey_for_onerow(RelOptInfo *rel); +static bool add_combined_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype); + +/* Used for unique indexes checking for partitioned table */ +static bool index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind); +static IndexOptInfo *simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from); +static bool simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2); +static void adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_index); + +/* Helper function for grouped relation and distinct relation. */ +static void add_uniquekey_from_sortgroups(PlannerInfo *root, + RelOptInfo *rel, + List *sortgroups); + +/* + * populate_baserel_uniquekeys + * Populate 'baserel' uniquekeys list by looking at the rel's unique index + * and baserestrictinfo + */ +void +populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List *indexlist) +{ + ListCell *lc; + List *matched_uniq_indexes = NIL; + + /* Attrs appears in rel->reltarget->exprs. */ + Bitmapset *used_attrs = NULL; + + List *const_exprs = NIL; + List *expr_opfamilies = NIL; + + Assert(baserel->rtekind == RTE_RELATION); + + foreach(lc, indexlist) + { + IndexOptInfo *ind = (IndexOptInfo *) lfirst(lc); + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + matched_uniq_indexes = lappend(matched_uniq_indexes, ind); + } + + if (matched_uniq_indexes == NIL) + return; + + /* Check which attrs is used in baserel->reltarget */ + pull_varattnos((Node *)baserel->reltarget->exprs, baserel->relid, &used_attrs); + + /* Check which attrno is used at a mergeable const filter */ + foreach(lc, baserel->baserestrictinfo) + { + RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); + + if (rinfo->mergeopfamilies == NIL) + continue; + + if (bms_is_empty(rinfo->left_relids)) + { + const_exprs = lappend(const_exprs, get_rightop(rinfo->clause)); + } + else if (bms_is_empty(rinfo->right_relids)) + { + const_exprs = lappend(const_exprs, get_leftop(rinfo->clause)); + } + else + continue; + + expr_opfamilies = lappend(expr_opfamilies, rinfo->mergeopfamilies); + } + + foreach(lc, matched_uniq_indexes) + { + bool multi_nullvals, useful; + List *exprs = get_exprs_from_uniqueindex(lfirst_node(IndexOptInfo, lc), + const_exprs, + expr_opfamilies, + used_attrs, + &useful, + &multi_nullvals); + if (useful) + { + if (exprs == NIL) + { + /* All the columns in Unique Index matched with a restrictinfo */ + add_uniquekey_for_onerow(baserel); + return; + } + baserel->uniquekeys = lappend(baserel->uniquekeys, + makeUniqueKey(exprs, multi_nullvals)); + } + } +} + + +/* + * populate_partitionedrel_uniquekeys + * The UniqueKey on partitionrel comes from 2 cases: + * 1). Only one partition is involved in this query, the unique key can be + * copied to parent rel from childrel. + * 2). There are some unique index which includes partition key and exists + * in all the related partitions. + * We never mind rule 2 if we hit rule 1. + */ + +void +populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels) +{ + ListCell *lc; + List *global_uniq_indexlist = NIL; + RelOptInfo *childrel; + bool is_first = true; + + Assert(IS_PARTITIONED_REL(rel)); + + if (childrels == NIL) + return; + + /* + * If there is only one partition used in this query, the UniqueKey in childrel is + * still valid in parent level, but we need convert the format from child expr to + * parent expr. + */ + if (list_length(childrels) == 1) + { + /* Check for Rule 1 */ + RelOptInfo *childrel = linitial_node(RelOptInfo, childrels); + ListCell *lc; + Assert(childrel->reloptkind == RELOPT_OTHER_MEMBER_REL); + if (relation_is_onerow(childrel)) + { + add_uniquekey_for_onerow(rel); + return; + } + + foreach(lc, childrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + AppendRelInfo *appinfo = find_appinfo_by_child(root, childrel->relid); + List *parent_exprs = NIL; + bool can_reuse = true; + ListCell *lc2; + foreach(lc2, ukey->exprs) + { + Var *var = (Var *)lfirst(lc2); + /* + * If the expr comes from a expression, it is hard to build the expression + * in parent so ignore that case for now. + */ + if(!IsA(var, Var)) + { + can_reuse = false; + break; + } + /* Convert it to parent var */ + parent_exprs = lappend(parent_exprs, find_parent_var(appinfo, var)); + } + if (can_reuse) + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(parent_exprs, + ukey->multi_nullvals)); + } + } + else + { + /* Check for rule 2 */ + childrel = linitial_node(RelOptInfo, childrels); + foreach(lc, childrel->indexlist) + { + IndexOptInfo *ind = lfirst(lc); + IndexOptInfo *modified_index; + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + + /* + * During simple_copy_indexinfo_to_parent, we need to convert var from + * child var to parent var, index on expression is too complex to handle. + * so ignore it for now. + */ + if (ind->indexprs != NIL) + continue; + + modified_index = simple_copy_indexinfo_to_parent(root, rel, ind); + /* + * If the unique index doesn't contain partkey, then it is unique + * on this partition only, so it is useless for us. + */ + if (!index_constains_partkey(rel, modified_index)) + continue; + + global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index); + } + + if (global_uniq_indexlist != NIL) + { + foreach(lc, childrels) + { + RelOptInfo *child = lfirst(lc); + if (is_first) + { + is_first = false; + continue; + } + adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist); + } + /* Now we have a list of unique index which are exactly same on all childrels, + * Set the UniqueKey just like it is non-partition table + */ + populate_baserel_uniquekeys(root, rel, global_uniq_indexlist); + } + } +} + + +/* + * populate_distinctrel_uniquekeys + */ +void +populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel) +{ + /* The unique key before the distinct is still valid. */ + distinctrel->uniquekeys = list_copy(inputrel->uniquekeys); + add_uniquekey_from_sortgroups(root, distinctrel, root->parse->distinctClause); +} + +/* + * populate_grouprel_uniquekeys + */ +void +populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel) + +{ + Query *parse = root->parse; + bool input_ukey_added = false; + ListCell *lc; + + if (relation_is_onerow(inputrel)) + { + add_uniquekey_for_onerow(grouprel); + return; + } + if (parse->groupingSets) + return; + + /* A Normal group by without grouping set. */ + if (parse->groupClause) + { + /* + * Current even the groupby clause is Unique already, but if query has aggref + * We have to create grouprel still. To keep the UnqiueKey short, we will check + * the UniqueKey of input_rel still valid, if so we reuse it. + */ + foreach(lc, inputrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (list_is_subset(ukey->exprs, grouprel->reltarget->exprs)) + { + grouprel->uniquekeys = lappend(grouprel->uniquekeys, + ukey); + input_ukey_added = true; + } + } + if (!input_ukey_added) + /* + * group by clause must be a super-set of grouprel->reltarget->exprs except the + * aggregation expr, so if such exprs is unique already, no bother to generate + * new uniquekey for group by exprs. + */ + add_uniquekey_from_sortgroups(root, + grouprel, + root->parse->groupClause); + } + else + /* It has aggregation but without a group by, so only one row returned */ + add_uniquekey_for_onerow(grouprel); +} + +/* + * simple_copy_uniquekeys + * Using a function for the one-line code makes us easy to check where we simply + * copied the uniquekey. + */ +void +simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel) +{ + newrel->uniquekeys = oldrel->uniquekeys; +} + +/* + * populate_unionrel_uniquekeys + */ +void +populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel) +{ + ListCell *lc; + List *exprs = NIL; + + Assert(unionrel->uniquekeys == NIL); + + foreach(lc, unionrel->reltarget->exprs) + { + exprs = lappend(exprs, lfirst(lc)); + } + + if (exprs == NIL) + /* SQL: select union select; is valid, we need to handle it here. */ + add_uniquekey_for_onerow(unionrel); + else + unionrel->uniquekeys = lappend(unionrel->uniquekeys, + makeUniqueKey(exprs,false)); + +} + +/* + * populate_joinrel_uniquekeys + * + * populate uniquekeys for joinrel. We will check each relation to see if its + * UniqueKey is still valid via innerrel_keeps_unique, if so, we add it to + * joinrel. The multi_nullvals field will be changed to true for some outer + * join cases and one-row UniqueKey needs to be converted to normal UniqueKey + * for the same case as well. + * For the uniquekey in either baserel which can't be unique after join, we still + * check to see if combination of UniqueKeys from both side is still useful for us. + * if yes, we add it to joinrel as well. + */ +void +populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *outerrel, RelOptInfo *innerrel, + List *restrictlist, JoinType jointype) +{ + ListCell *lc, *lc2; + List *clause_list = NIL; + List *outerrel_ukey_ctx; + List *innerrel_ukey_ctx; + bool inner_onerow, outer_onerow; + bool mergejoin_allowed; + + /* Care about the outerrel relation only for SEMI/ANTI join */ + if (jointype == JOIN_SEMI || jointype == JOIN_ANTI) + { + foreach(lc, outerrel->uniquekeys) + { + UniqueKey *uniquekey = lfirst_node(UniqueKey, lc); + if (list_is_subset(uniquekey->exprs, joinrel->reltarget->exprs)) + joinrel->uniquekeys = lappend(joinrel->uniquekeys, uniquekey); + } + return; + } + + Assert(jointype == JOIN_LEFT || jointype == JOIN_FULL || jointype == JOIN_INNER); + + /* Fast path */ + if (innerrel->uniquekeys == NIL || outerrel->uniquekeys == NIL) + return; + + inner_onerow = relation_is_onerow(innerrel); + outer_onerow = relation_is_onerow(outerrel); + + outerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(outerrel); + innerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(innerrel); + + clause_list = select_mergejoin_clauses(root, joinrel, outerrel, innerrel, + restrictlist, jointype, + &mergejoin_allowed); + + if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true /* reverse */)) + { + bool outer_impact = jointype == JOIN_FULL; + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + /* Outer relation has one row, and the unique key is not duplicated after join, + * the joinrel will still has one row unless the jointype == JOIN_FULL. + */ + if (outer_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (outer_onerow) + { + /* + * The onerow outerrel becomes multi rows and multi_nullvals + * will be changed to true. We also need to set the exprs correctly since it + * can't be NIL any more. + */ + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, outerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Change multi_nullvals to true due to the full join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, true)); + else + /* Just reuse it */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + } + ctx->added_to_joinrel = true; + } + } + + if (innerrel_keeps_unique(root, outerrel, innerrel, clause_list, false)) + { + bool outer_impact = jointype == JOIN_FULL || jointype == JOIN_LEFT;; + + foreach(lc, innerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + if (inner_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (inner_onerow) + { + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, innerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Need to change multi_nullvals to true due to the outer join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, + true)); + else + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + + } + ctx->added_to_joinrel = true; + } + } + + /* + * The combination of the UniqueKey from both sides is unique as well regardless + * of join type, but no bother to add it if its subset has been added to joinrel + * already or it is not useful for the joinrel. + */ + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx1 = (UniqueKeyContext) lfirst(lc); + if (ctx1->added_to_joinrel || !ctx1->useful) + continue; + foreach(lc2, innerrel_ukey_ctx) + { + UniqueKeyContext ctx2 = (UniqueKeyContext) lfirst(lc2); + if (ctx2->added_to_joinrel || !ctx2->useful) + continue; + if (add_combined_uniquekey(root, joinrel, outerrel, innerrel, + ctx1->uniquekey, ctx2->uniquekey, + jointype)) + /* If we set a onerow UniqueKey to joinrel, we don't need other. */ + return; + } + } +} + + +/* + * convert_subquery_uniquekeys + * + * Covert the UniqueKey in subquery to outer relation. + */ +void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel) +{ + ListCell *lc; + + if (sub_final_rel->uniquekeys == NIL) + return; + + if (relation_is_onerow(sub_final_rel)) + { + add_uniquekey_for_onerow(currel); + return; + } + + Assert(currel->subroot != NULL); + + foreach(lc, sub_final_rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + ListCell *lc; + List *exprs = NIL; + bool ukey_useful = true; + + /* One row case is handled above */ + Assert(ukey->exprs != NIL); + foreach(lc, ukey->exprs) + { + Var *var; + TargetEntry *tle = tlist_member(lfirst(lc), + currel->subroot->processed_tlist); + if (tle == NULL) + { + ukey_useful = false; + break; + } + var = find_var_for_subquery_tle(currel, tle); + if (var == NULL) + { + ukey_useful = false; + break; + } + exprs = lappend(exprs, var); + } + + if (ukey_useful) + currel->uniquekeys = lappend(currel->uniquekeys, + makeUniqueKey(exprs, + ukey->multi_nullvals)); + + } +} + +/* + * innerrel_keeps_unique + * + * Check if Unique key of the innerrel is valid after join. innerrel's UniqueKey + * will be still valid if innerrel's any-column mergeop outrerel's uniquekey + * exists in clause_list. + * + * Note: the clause_list must be a list of mergeable restrictinfo already. + */ +static bool +innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *clause_list, + bool reverse) +{ + ListCell *lc, *lc2, *lc3; + + if (outerrel->uniquekeys == NIL || innerrel->uniquekeys == NIL) + return false; + + /* Check if there is outerrel's uniquekey in mergeable clause. */ + foreach(lc, outerrel->uniquekeys) + { + List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs; + bool clauselist_matchs_all_exprs = true; + foreach(lc2, outer_uq_exprs) + { + Node *outer_uq_expr = lfirst(lc2); + bool find_uq_expr_in_clauselist = false; + foreach(lc3, clause_list) + { + RestrictInfo *rinfo = lfirst_node(RestrictInfo, lc3); + Node *outer_expr; + if (reverse) + outer_expr = rinfo->outer_is_left ? get_rightop(rinfo->clause) : get_leftop(rinfo->clause); + else + outer_expr = rinfo->outer_is_left ? get_leftop(rinfo->clause) : get_rightop(rinfo->clause); + if (equal(outer_expr, outer_uq_expr)) + { + find_uq_expr_in_clauselist = true; + break; + } + } + if (!find_uq_expr_in_clauselist) + { + /* No need to check the next exprs in the current uniquekey */ + clauselist_matchs_all_exprs = false; + break; + } + } + + if (clauselist_matchs_all_exprs) + return true; + } + return false; +} + + +/* + * relation_is_onerow + * Check if it is a one-row relation by checking UniqueKey. + */ +bool +relation_is_onerow(RelOptInfo *rel) +{ + UniqueKey *ukey; + if (rel->uniquekeys == NIL) + return false; + ukey = linitial_node(UniqueKey, rel->uniquekeys); + return ukey->exprs == NIL && list_length(rel->uniquekeys) == 1; +} + +/* + * relation_has_uniquekeys_for + * Returns true if we have proofs that 'rel' cannot return multiple rows with + * the same values in each of 'exprs'. Otherwise returns false. + */ +bool +relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel, + List *exprs, bool allow_multinulls) +{ + ListCell *lc; + + /* + * For UniqueKey->onerow case, the uniquekey->exprs is empty as well + * so we can't rely on list_is_subset to handle this special cases + */ + if (exprs == NIL) + return false; + + foreach(lc, rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (ukey->multi_nullvals && !allow_multinulls) + continue; + if (list_is_subset(ukey->exprs, exprs)) + return true; + } + return false; +} + + +/* + * get_exprs_from_uniqueindex + * + * Return a list of exprs which is unique. set useful to false if this + * unique index is not useful for us. + */ +static List * +get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals) +{ + List *exprs = NIL; + ListCell *indexpr_item; + int c = 0; + + *useful = true; + *multi_nullvals = false; + + indexpr_item = list_head(unique_index->indexprs); + for(c = 0; c < unique_index->ncolumns; c++) + { + int attr = unique_index->indexkeys[c]; + Expr *expr; + bool matched_const = false; + ListCell *lc1, *lc2; + + if(attr > 0) + { + expr = list_nth_node(TargetEntry, unique_index->indextlist, c)->expr; + } + else if (attr == 0) + { + /* Expression index */ + expr = lfirst(indexpr_item); + indexpr_item = lnext(unique_index->indexprs, indexpr_item); + } + else /* attr < 0 */ + { + /* Index on system column is not supported */ + Assert(false); + } + + /* + * Check index_col = Const case with regarding to opfamily checking + * If we can remove the index_col from the final UniqueKey->exprs. + */ + forboth(lc1, const_exprs, lc2, const_expr_opfamilies) + { + if (list_member_oid((List *)lfirst(lc2), unique_index->opfamily[c]) + && match_index_to_operand((Node *) lfirst(lc1), c, unique_index)) + { + matched_const = true; + break; + } + } + + if (matched_const) + continue; + + /* Check if the indexed expr is used in rel */ + if (attr > 0) + { + /* + * Normal Indexed column, if the col is not used, then the index is useless + * for uniquekey. + */ + attr -= FirstLowInvalidHeapAttributeNumber; + + if (!bms_is_member(attr, used_varattrs)) + { + *useful = false; + break; + } + } + else if (!list_member(unique_index->rel->reltarget->exprs, expr)) + { + /* Expression index but the expression is not used in rel */ + *useful = false; + break; + } + + /* check not null property. */ + if (attr == 0) + { + /* We never know if a expression yields null or not */ + *multi_nullvals = true; + } + else if (!bms_is_member(attr, unique_index->rel->notnullattrs) + && !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber, + unique_index->rel->notnullattrs)) + { + *multi_nullvals = true; + } + + exprs = lappend(exprs, expr); + } + return exprs; +} + + +/* + * add_uniquekey_for_onerow + * If we are sure that the relation only returns one row, then all the columns + * are unique. However we don't need to create UniqueKey for every column, we + * just set exprs = NIL and overwrites all the other UniqueKey on this RelOptInfo + * since this one has strongest semantics. + */ +void +add_uniquekey_for_onerow(RelOptInfo *rel) +{ + /* + * We overwrite the previous UniqueKey on purpose since this one has the + * strongest semantic. + */ + rel->uniquekeys = list_make1(makeUniqueKey(NIL, false)); +} + + +/* + * initililze_uniquecontext_for_joinrel + * Return a List of UniqueKeyContext for an inputrel + */ +static List * +initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel) +{ + List *res = NIL; + ListCell *lc; + foreach(lc, inputrel->uniquekeys) + { + UniqueKeyContext context; + context = palloc(sizeof(struct UniqueKeyContextData)); + context->uniquekey = lfirst_node(UniqueKey, lc); + context->added_to_joinrel = false; + context->useful = true; + res = lappend(res, context); + } + return res; +} + + +/* + * get_exprs_from_uniquekey + * Unify the way of get List of exprs from a one-row UniqueKey or + * normal UniqueKey. for the onerow case, every expr in rel1 is a valid + * UniqueKey. Return a List of exprs. + * + * rel1: The relation which you want to get the exprs. + * ukey: The UniqueKey you want to get the exprs. + */ +static List * +get_exprs_from_uniquekey(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *rel1, UniqueKey *ukey) +{ + ListCell *lc; + bool onerow = rel1 != NULL && relation_is_onerow(rel1); + + List *res = NIL; + Assert(onerow || ukey); + if (onerow) + { + /* Only cares about the exprs still exist in joinrel */ + foreach(lc, joinrel->reltarget->exprs) + { + Bitmapset *relids = pull_varnos(root, lfirst(lc)); + if (bms_is_subset(relids, rel1->relids)) + { + res = lappend(res, list_make1(lfirst(lc))); + } + } + } + else + { + res = list_make1(ukey->exprs); + } + return res; +} + +/* + * Partitioned table Unique Keys. + * The partition table unique key is maintained as: + * 1. The index must be unique as usual. + * 2. The index must contains partition key. + * 3. The index must exist on all the child rel. see simple_indexinfo_equal for + * how we compare it. + */ + +/* + * index_constains_partkey + * return true if the index contains the partiton key. + */ +static bool +index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind) +{ + ListCell *lc; + int i; + Assert(IS_PARTITIONED_REL(partrel)); + Assert(partrel->part_scheme->partnatts > 0); + + for(i = 0; i < partrel->part_scheme->partnatts; i++) + { + Node *part_expr = linitial(partrel->partexprs[i]); + bool found_in_index = false; + foreach(lc, ind->indextlist) + { + Expr *index_expr = lfirst_node(TargetEntry, lc)->expr; + if (equal(index_expr, part_expr)) + { + found_in_index = true; + break; + } + } + if (!found_in_index) + return false; + } + return true; +} + +/* + * simple_indexinfo_equal + * + * Used to check if the 2 index is same as each other. The index here + * is COPIED from childrel and did some tiny changes(see + * simple_copy_indexinfo_to_parent) + */ +static bool +simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2) +{ + Size oid_cmp_len = sizeof(Oid) * ind1->ncolumns; + + return ind1->ncolumns == ind2->ncolumns && + ind1->unique == ind2->unique && + memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0 && + memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0 && + memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0 && + memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0 && + equal(get_tlist_exprs(ind1->indextlist, true), + get_tlist_exprs(ind2->indextlist, true)); +} + + +/* + * The below macros are used for simple_copy_indexinfo_to_parent which is so + * customized that I don't want to put it to copyfuncs.c. So copy it here. + */ +#define COPY_POINTER_FIELD(fldname, sz) \ + do { \ + Size _size = (sz); \ + newnode->fldname = palloc(_size); \ + memcpy(newnode->fldname, from->fldname, _size); \ + } while (0) + +#define COPY_NODE_FIELD(fldname) \ + (newnode->fldname = copyObjectImpl(from->fldname)) + +#define COPY_SCALAR_FIELD(fldname) \ + (newnode->fldname = from->fldname) + + +/* + * simple_copy_indexinfo_to_parent (from partition) + * Copy the IndexInfo from child relation to parent relation with some modification, + * which is used to test: + * 1. If the same index exists in all the childrels. + * 2. If the parentrel->reltarget/basicrestrict info matches this index. + */ +static IndexOptInfo * +simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from) +{ + IndexOptInfo *newnode = makeNode(IndexOptInfo); + AppendRelInfo *appinfo = find_appinfo_by_child(root, from->rel->relid); + ListCell *lc; + int idx = 0; + + COPY_SCALAR_FIELD(ncolumns); + COPY_SCALAR_FIELD(nkeycolumns); + COPY_SCALAR_FIELD(unique); + COPY_SCALAR_FIELD(immediate); + /* We just need to know if it is NIL or not */ + COPY_SCALAR_FIELD(indpred); + COPY_SCALAR_FIELD(predOK); + COPY_POINTER_FIELD(indexkeys, from->ncolumns * sizeof(int)); + COPY_POINTER_FIELD(indexcollations, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opfamily, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opcintype, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(sortopfamily, from->ncolumns * sizeof(Oid)); + COPY_NODE_FIELD(indextlist); + + /* Convert index exprs on child expr to expr on parent */ + foreach(lc, newnode->indextlist) + { + TargetEntry *tle = lfirst_node(TargetEntry, lc); + /* Index on expression is ignored */ + Assert(IsA(tle->expr, Var)); + tle->expr = (Expr *) find_parent_var(appinfo, (Var *) tle->expr); + newnode->indexkeys[idx] = castNode(Var, tle->expr)->varattno; + idx++; + } + newnode->rel = parentrel; + return newnode; +} + +/* + * adjust_partition_unique_indexlist + * + * global_unique_indexes: At the beginning, it contains the copy & modified + * unique index from the first partition. And then check if each index in it still + * exists in the following partitions. If no, remove it. at last, it has an + * index list which exists in all the partitions. + */ +static void +adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_indexes) +{ + ListCell *lc, *lc2; + foreach(lc, *global_unique_indexes) + { + IndexOptInfo *g_ind = lfirst_node(IndexOptInfo, lc); + bool found_in_child = false; + + foreach(lc2, childrel->indexlist) + { + IndexOptInfo *p_ind = lfirst_node(IndexOptInfo, lc2); + IndexOptInfo *p_ind_copy; + if (!p_ind->unique || !p_ind->immediate || + (p_ind->indpred != NIL && !p_ind->predOK)) + continue; + p_ind_copy = simple_copy_indexinfo_to_parent(root, parentrel, p_ind); + if (simple_indexinfo_equal(p_ind_copy, g_ind)) + { + found_in_child = true; + break; + } + } + if (!found_in_child) + /* The index doesn't exist in childrel, remove it from global_unique_indexes */ + *global_unique_indexes = foreach_delete_current(*global_unique_indexes, lc); + } +} + +/* Helper function for groupres/distinctrel */ +static void +add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgroups) +{ + Query *parse = root->parse; + List *exprs; + + /* + * XXX: If there are some vars which is not in current levelsup, the semantic is + * imprecise, should we avoid it or not? levelsup = 1 is just a demo, maybe we need to + * check every level other than 0, if so, looks we have to write another + * pull_var_walker. + */ + List *upper_vars = pull_vars_of_level((Node*)sortgroups, 1); + + if (upper_vars != NIL) + return; + + exprs = get_sortgrouplist_exprs(sortgroups, parse->targetList); + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(exprs, + false /* sortgroupclause can't be multi_nullvals */)); +} + + +/* + * add_combined_uniquekey + * The combination of both UniqueKeys is a valid UniqueKey for joinrel no matter + * the jointype. + */ +bool +add_combined_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype) +{ + + ListCell *lc1, *lc2; + + /* Either side has multi_nullvals or we have outer join, + * the combined UniqueKey has multi_nullvals */ + bool multi_nullvals = outer_ukey->multi_nullvals || + inner_ukey->multi_nullvals || IS_OUTER_JOIN(jointype); + + /* The only case we can get onerow joinrel after join */ + if (relation_is_onerow(outer_rel) + && relation_is_onerow(inner_rel) + && jointype == JOIN_INNER) + { + add_uniquekey_for_onerow(joinrel); + return true; + } + + foreach(lc1, get_exprs_from_uniquekey(root, joinrel, outer_rel, outer_ukey)) + { + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, inner_rel, inner_ukey)) + { + List *exprs = list_concat_copy(lfirst_node(List, lc1), lfirst_node(List, lc2)); + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(exprs, + multi_nullvals)); + } + } + return false; +} diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index 424d25cbd5..8d8e493f5c 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -2387,6 +2387,8 @@ grouping_planner(PlannerInfo *root, bool inheritance_update, add_path(final_rel, path); } + simple_copy_uniquekeys(current_rel, final_rel); + /* * Generate partial paths for final_rel, too, if outer query levels might * be able to make use of them. @@ -3899,6 +3901,8 @@ create_grouping_paths(PlannerInfo *root, } set_cheapest(grouped_rel); + + populate_grouprel_uniquekeys(root, grouped_rel, input_rel); return grouped_rel; } @@ -4621,7 +4625,7 @@ create_window_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(window_rel); - + simple_copy_uniquekeys(input_rel, window_rel); return window_rel; } @@ -4935,7 +4939,7 @@ create_distinct_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(distinct_rel); - + populate_distinctrel_uniquekeys(root, input_rel, distinct_rel); return distinct_rel; } @@ -5196,6 +5200,8 @@ create_ordered_paths(PlannerInfo *root, */ Assert(ordered_rel->pathlist != NIL); + simple_copy_uniquekeys(input_rel, ordered_rel); + return ordered_rel; } @@ -6073,6 +6079,9 @@ adjust_paths_for_srfs(PlannerInfo *root, RelOptInfo *rel, if (list_length(targets) == 1) return; + /* UniqueKey is not valid after handling the SRF. */ + rel->uniquekeys = NIL; + /* * Stack SRF-evaluation nodes atop each path for the rel. * diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c index becdcbb872..b7626545bf 100644 --- a/src/backend/optimizer/prep/prepunion.c +++ b/src/backend/optimizer/prep/prepunion.c @@ -689,6 +689,8 @@ generate_union_paths(SetOperationStmt *op, PlannerInfo *root, /* Undo effects of possibly forcing tuple_fraction to 0 */ root->tuple_fraction = save_fraction; + /* Add the UniqueKeys */ + populate_unionrel_uniquekeys(root, result_rel); return result_rel; } diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c index 86922a273c..6817c9c787 100644 --- a/src/backend/optimizer/util/appendinfo.c +++ b/src/backend/optimizer/util/appendinfo.c @@ -746,3 +746,47 @@ find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos) } return appinfos; } + +/* + * find_appinfo_by_child + * + */ +AppendRelInfo * +find_appinfo_by_child(PlannerInfo *root, Index child_index) +{ + ListCell *lc; + foreach(lc, root->append_rel_list) + { + AppendRelInfo *appinfo = lfirst_node(AppendRelInfo, lc); + if (appinfo->child_relid == child_index) + return appinfo; + } + elog(ERROR, "parent relation cant be found"); + return NULL; +} + +/* + * find_parent_var + * + */ +Var * +find_parent_var(AppendRelInfo *appinfo, Var *child_var) +{ + ListCell *lc; + Var *res = NULL; + Index attno = 1; + foreach(lc, appinfo->translated_vars) + { + Node *child_node = lfirst(lc); + if (equal(child_node, child_var)) + { + res = copyObject(child_var); + res->varattno = attno; + res->varno = appinfo->parent_relid; + } + attno++; + } + if (res == NULL) + elog(ERROR, "parent var can't be found."); + return res; +} diff --git a/src/backend/optimizer/util/inherit.c b/src/backend/optimizer/util/inherit.c index be1c9ddd96..3eec1f4d74 100644 --- a/src/backend/optimizer/util/inherit.c +++ b/src/backend/optimizer/util/inherit.c @@ -736,6 +736,7 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, { Node *onecq = (Node *) lfirst(lc2); bool pseudoconstant; + RestrictInfo *child_rinfo; /* check for pseudoconstant (no Vars or volatile functions) */ pseudoconstant = @@ -747,14 +748,15 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, root->hasPseudoConstantQuals = true; } /* reconstitute RestrictInfo with appropriate properties */ - childquals = lappend(childquals, - make_restrictinfo(root, - (Expr *) onecq, - rinfo->is_pushed_down, - rinfo->outerjoin_delayed, - pseudoconstant, - rinfo->security_level, - NULL, NULL, NULL)); + child_rinfo = make_restrictinfo(root, + (Expr *) onecq, + rinfo->is_pushed_down, + rinfo->outerjoin_delayed, + pseudoconstant, + rinfo->security_level, + NULL, NULL, NULL); + child_rinfo->mergeopfamilies = rinfo->mergeopfamilies; + childquals = lappend(childquals, child_rinfo); /* track minimum security level among child quals */ cq_min_security = Min(cq_min_security, rinfo->security_level); } diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 48a7ebfe45..4fe1824eb0 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -16,6 +16,7 @@ #include "nodes/execnodes.h" #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" extern A_Expr *makeA_Expr(A_Expr_Kind kind, List *name, @@ -106,4 +107,6 @@ extern GroupingSet *makeGroupingSet(GroupingSetKind kind, List *content, int loc extern VacuumRelation *makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols); +extern UniqueKey* makeUniqueKey(List *exprs, bool multi_nullvals); + #endif /* MAKEFUNC_H */ diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index e22df890ef..3320273ac1 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -265,6 +265,7 @@ typedef enum NodeTag T_EquivalenceMember, T_PathKey, T_PathTarget, + T_UniqueKey, T_RestrictInfo, T_IndexClause, T_PlaceHolderVar, diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index 0d61f04d27..80561b1003 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -739,6 +739,7 @@ typedef struct RelOptInfo QualCost baserestrictcost; /* cost of evaluating the above */ Index baserestrict_min_security; /* min security_level found in * baserestrictinfo */ + List *uniquekeys; /* List of UniqueKey */ List *joininfo; /* RestrictInfo structures for join clauses * involving this rel */ bool has_eclass_joins; /* T means joininfo is incomplete */ @@ -1059,6 +1060,28 @@ typedef struct PathKey } PathKey; +/* + * UniqueKey + * + * Represents the unique properties held by a RelOptInfo. + * + * exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL + * is a special case of UniqueKey, which means there is only 1 row in that + * relation. + * multi_nullvals: true means multi null values may exist in these exprs, so the + * uniqueness is not guaranteed in this case. This field is necessary for + * remove_useless_join & reduce_unique_semijoins where we don't mind these + * duplicated NULL values. It is set to true for 2 cases. One is a unique key + * from a unique index but the related column is nullable. The other one is for + * outer join. see populate_joinrel_uniquekeys for detail. + */ +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + /* * PathTarget * @@ -2496,7 +2519,7 @@ typedef enum * * flags indicating what kinds of grouping are possible. * partial_costs_set is true if the agg_partial_costs and agg_final_costs - * have been initialized. + * have been initialized. * agg_partial_costs gives partial aggregation costs. * agg_final_costs gives finalization costs. * target_parallel_safe is true if target is parallel safe. @@ -2526,8 +2549,8 @@ typedef struct * limit_tuples is an estimated bound on the number of output tuples, * or -1 if no LIMIT or couldn't estimate. * count_est and offset_est are the estimated values of the LIMIT and OFFSET - * expressions computed by preprocess_limit() (see comments for - * preprocess_limit() for more information). + * expressions computed by preprocess_limit() (see comments for + * preprocess_limit() for more information). */ typedef struct { diff --git a/src/include/nodes/pg_list.h b/src/include/nodes/pg_list.h index 404e03f132..85c65e910c 100644 --- a/src/include/nodes/pg_list.h +++ b/src/include/nodes/pg_list.h @@ -566,6 +566,16 @@ extern pg_nodiscard List *list_delete_first(List *list); extern pg_nodiscard List *list_delete_last(List *list); extern pg_nodiscard List *list_delete_nth_cell(List *list, int n); extern pg_nodiscard List *list_delete_cell(List *list, ListCell *cell); +extern bool list_is_subset(const List *members, const List *target); + +extern List *list_delete(List *list, void *datum); +extern List *list_delete_ptr(List *list, void *datum); +extern List *list_delete_int(List *list, int datum); +extern List *list_delete_oid(List *list, Oid datum); +extern List *list_delete_first(List *list); +extern List *list_delete_last(List *list); +extern List *list_delete_nth_cell(List *list, int n); +extern List *list_delete_cell(List *list, ListCell *cell); extern List *list_union(const List *list1, const List *list2); extern List *list_union_ptr(const List *list1, const List *list2); diff --git a/src/include/optimizer/appendinfo.h b/src/include/optimizer/appendinfo.h index 4cbf8c26cc..43b8d05f69 100644 --- a/src/include/optimizer/appendinfo.h +++ b/src/include/optimizer/appendinfo.h @@ -32,4 +32,7 @@ extern Relids adjust_child_relids_multilevel(PlannerInfo *root, Relids relids, extern AppendRelInfo **find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos); +extern AppendRelInfo *find_appinfo_by_child(PlannerInfo *root, Index child_index); +extern Var *find_parent_var(AppendRelInfo *appinfo, Var *child_var); + #endif /* APPENDINFO_H */ diff --git a/src/include/optimizer/optimizer.h b/src/include/optimizer/optimizer.h index d587952b7d..843aafa51e 100644 --- a/src/include/optimizer/optimizer.h +++ b/src/include/optimizer/optimizer.h @@ -23,6 +23,7 @@ #define OPTIMIZER_H #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" /* Test if an expression node represents a SRF call. Beware multiple eval! */ #define IS_SRF_CALL(node) \ @@ -168,6 +169,7 @@ extern TargetEntry *get_sortgroupref_tle(Index sortref, List *targetList); extern TargetEntry *get_sortgroupclause_tle(SortGroupClause *sgClause, List *targetList); +extern Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); extern Node *get_sortgroupclause_expr(SortGroupClause *sgClause, List *targetList); extern List *get_sortgrouplist_exprs(List *sgClauses, diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h index 035d3e1206..1adf99c9ee 100644 --- a/src/include/optimizer/paths.h +++ b/src/include/optimizer/paths.h @@ -247,5 +247,48 @@ extern PathKey *make_canonical_pathkey(PlannerInfo *root, int strategy, bool nulls_first); extern void add_paths_to_append_rel(PlannerInfo *root, RelOptInfo *rel, List *live_childrels); +extern List *select_mergejoin_clauses(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + JoinType jointype, + bool *mergejoin_allowed); + +/* + * uniquekeys.c + * Utilities for matching and building unique keys + */ +extern void populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List* unique_index_list); +extern void populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels); +extern void populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel); +extern void populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel); +extern void populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel); +extern void simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel); +extern void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel); +extern void populate_joinrel_uniquekeys(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + RelOptInfo *rel2, + List *restrictlist, + JoinType jointype); + +extern bool relation_has_uniquekeys_for(PlannerInfo *root, + RelOptInfo *rel, + List *exprs, + bool allow_multinulls); +extern bool relation_is_onerow(RelOptInfo *rel); #endif /* PATHS_H */ -- 2.26.2 --yh7cw345mbttadjz Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v38-0003-Extend-UniqueKeys.patch" ^ permalink raw reply [nested|flat] 91+ messages in thread
* [PATCH v36 2/6] Introduce UniqueKey attributes on RelOptInfo struct. @ 2020-05-11 07:50 一挃 <[email protected]> 0 siblings, 0 replies; 91+ messages in thread From: 一挃 @ 2020-05-11 07:50 UTC (permalink / raw) UniqueKey is a set of exprs on RelOptInfo which represents the exprs will be unique on the given RelOptInfo. You can see README.uniquekey for more information. --- src/backend/nodes/copyfuncs.c | 13 + src/backend/nodes/list.c | 31 + src/backend/nodes/makefuncs.c | 13 + src/backend/nodes/outfuncs.c | 11 + src/backend/nodes/readfuncs.c | 10 + src/backend/optimizer/path/Makefile | 3 +- src/backend/optimizer/path/README.uniquekey | 131 +++ src/backend/optimizer/path/allpaths.c | 10 + src/backend/optimizer/path/joinpath.c | 9 +- src/backend/optimizer/path/joinrels.c | 2 + src/backend/optimizer/path/pathkeys.c | 3 +- src/backend/optimizer/path/uniquekeys.c | 1131 +++++++++++++++++++ src/backend/optimizer/plan/planner.c | 13 +- src/backend/optimizer/prep/prepunion.c | 2 + src/backend/optimizer/util/appendinfo.c | 44 + src/backend/optimizer/util/inherit.c | 16 +- src/include/nodes/makefuncs.h | 3 + src/include/nodes/nodes.h | 1 + src/include/nodes/pathnodes.h | 29 +- src/include/nodes/pg_list.h | 2 + src/include/optimizer/appendinfo.h | 3 + src/include/optimizer/optimizer.h | 2 + src/include/optimizer/paths.h | 43 + 23 files changed, 1502 insertions(+), 23 deletions(-) create mode 100644 src/backend/optimizer/path/README.uniquekey create mode 100644 src/backend/optimizer/path/uniquekeys.c diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 89c409de66..1f50400fd2 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2273,6 +2273,16 @@ _copyPathKey(const PathKey *from) return newnode; } +static UniqueKey * +_copyUniqueKey(const UniqueKey *from) +{ + UniqueKey *newnode = makeNode(UniqueKey); + + COPY_NODE_FIELD(exprs); + COPY_SCALAR_FIELD(multi_nullvals); + + return newnode; +} /* * _copyRestrictInfo */ @@ -5152,6 +5162,9 @@ copyObjectImpl(const void *from) case T_PathKey: retval = _copyPathKey(from); break; + case T_UniqueKey: + retval = _copyUniqueKey(from); + break; case T_RestrictInfo: retval = _copyRestrictInfo(from); break; diff --git a/src/backend/nodes/list.c b/src/backend/nodes/list.c index 80fa8c84e4..a7a99b70f2 100644 --- a/src/backend/nodes/list.c +++ b/src/backend/nodes/list.c @@ -687,6 +687,37 @@ list_member_oid(const List *list, Oid datum) return false; } +/* + * return true iff every entry in "members" list is also present + * in the "target" list. + */ +bool +list_is_subset(const List *members, const List *target) +{ + const ListCell *lc1, *lc2; + + Assert(IsPointerList(members)); + Assert(IsPointerList(target)); + check_list_invariants(members); + check_list_invariants(target); + + foreach(lc1, members) + { + bool found = false; + foreach(lc2, target) + { + if (equal(lfirst(lc1), lfirst(lc2))) + { + found = true; + break; + } + } + if (!found) + return false; + } + return true; +} + /* * Delete the n'th cell (counting from 0) in list. * diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 49de285f01..646cf7c9a1 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -814,3 +814,16 @@ makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols) v->va_cols = va_cols; return v; } + + +/* + * makeUniqueKey + */ +UniqueKey* +makeUniqueKey(List *exprs, bool multi_nullvals) +{ + UniqueKey * ukey = makeNode(UniqueKey); + ukey->exprs = exprs; + ukey->multi_nullvals = multi_nullvals; + return ukey; +} diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index e2f177515d..c3a9632992 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2428,6 +2428,14 @@ _outPathKey(StringInfo str, const PathKey *node) WRITE_BOOL_FIELD(pk_nulls_first); } +static void +_outUniqueKey(StringInfo str, const UniqueKey *node) +{ + WRITE_NODE_TYPE("UNIQUEKEY"); + WRITE_NODE_FIELD(exprs); + WRITE_BOOL_FIELD(multi_nullvals); +} + static void _outPathTarget(StringInfo str, const PathTarget *node) { @@ -4127,6 +4135,9 @@ outNode(StringInfo str, const void *obj) case T_PathKey: _outPathKey(str, obj); break; + case T_UniqueKey: + _outUniqueKey(str, obj); + break; case T_PathTarget: _outPathTarget(str, obj); break; diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index 42050ab719..3a18571d0c 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -452,6 +452,14 @@ _readSetOperationStmt(void) READ_DONE(); } +static UniqueKey * +_readUniqueKey(void) +{ + READ_LOCALS(UniqueKey); + READ_NODE_FIELD(exprs); + READ_BOOL_FIELD(multi_nullvals); + READ_DONE(); +} /* * Stuff from primnodes.h. @@ -2656,6 +2664,8 @@ parseNodeString(void) return_value = _readCommonTableExpr(); else if (MATCH("SETOPERATIONSTMT", 16)) return_value = _readSetOperationStmt(); + else if (MATCH("UNIQUEKEY", 9)) + return_value = _readUniqueKey(); else if (MATCH("ALIAS", 5)) return_value = _readAlias(); else if (MATCH("RANGEVAR", 8)) diff --git a/src/backend/optimizer/path/Makefile b/src/backend/optimizer/path/Makefile index 1e199ff66f..7b9820c25f 100644 --- a/src/backend/optimizer/path/Makefile +++ b/src/backend/optimizer/path/Makefile @@ -21,6 +21,7 @@ OBJS = \ joinpath.o \ joinrels.o \ pathkeys.o \ - tidpath.o + tidpath.o \ + uniquekeys.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/optimizer/path/README.uniquekey b/src/backend/optimizer/path/README.uniquekey new file mode 100644 index 0000000000..5eac761995 --- /dev/null +++ b/src/backend/optimizer/path/README.uniquekey @@ -0,0 +1,131 @@ +1. What is UniqueKey? +We can think UniqueKey is a set of exprs for a RelOptInfo, which we are insure +that doesn't yields same result among all the rows. The simplest UniqueKey +format is primary key. + +However we define the UnqiueKey as below. + +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + +exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL +is a special case of UniqueKey, which means there is only one row in that +relation.it has a stronger semantic than others. like SELECT uk FROM t; uk is +normal unique key and may have different values. SELECT colx FROM t WHERE uk = +const. colx is unique AND we have only 1 value. This field can used for +innerrel_is_unique. this logic is handled specially in add_uniquekey_for_onerow +function. + +multi_nullvals: true means multi null values may exist in these exprs, so the +uniqueness is not guaranteed in this case. This field is necessary for +remove_useless_join & reduce_unique_semijoins where we don't mind these +duplicated NULL values. It is set to true for 2 cases. One is a unique key +from a unique index but the related column is nullable. The other one is for +outer join. see populate_joinrel_uniquekeys for detail. + + +The UniqueKey can be used at the following cases at least: +1. remove_useless_joins. +2. reduce_semianti_joins +3. remove distinct node if distinct clause is unique. +4. remove aggnode if group by clause is unique. +5. Index Skip Scan (WIP) +6. Aggregation Push Down without 2 phase aggregation if the join can't + duplicated the aggregated rows. (work in progress feature) + +2. How is it maintained? + +We have a set of populate_xxx_unqiuekeys functions to maintain the uniquekey on +various cases. xxx includes baserel, joinrel, partitionedrel, distinctrel, +groupedrel, unionrel. and we also need to convert the uniquekey from subquery +to outer relation, which is what convert_subquery_uniquekeys does. + +1. The first part is about baserel. We handled 3 cases. suppose we have Unique +Index on (a, b). + +1. SELECT a, b FROM t. UniqueKey (a, b) +2. SELECT a FROM t WHERE b = 1; UniqueKey (a) +3. SELECT .. FROM t WHERE a = 1 AND b = 1; UniqueKey (NIL). onerow case, every + column is Unique. + +2. The next part is joinrel, this part is most error-prone, we simplified the rules +like below: +1. If the relation's UniqueKey can't be duplicated after join, then is will be + still valid for the join rel. The function we used here is + innerrel_keeps_unique. The basic idea is innerrel.any_col = outer.uk. + +2. If the UnqiueKey can't keep valid via the rule 1, the combination of the + UniqueKey from both sides are valid for sure. We can prove this as: if the + unique exprs from rel1 is duplicated by rel2, the duplicated rows must + contains different unique exprs from rel2. + +More considerations about onerow: +1. If relation with one row and it can't be duplicated, it is still possible + contains mulit_nullvas after outer join. +2. If the either UniqueKey can be duplicated after join, the can get one row + only when both side is one row AND there is no outer join. +3. Whenever the onerow UniqueKey is not a valid any more, we need to convert one + row UniqueKey to normal unique key since we don't store exprs for one-row + relation. get_exprs_from_uniquekeys will be used here. + + +More considerations about multi_nullvals after join: +1. If the original UnqiueKey has multi_nullvals, the final UniqueKey will have + mulit_nullvals in any case. +2. If a unique key doesn't allow mulit_nullvals, after some outer join, it + allows some outer join. + + +3. When we comes to subquery, we need to convert_subquery_unqiuekeys just like +convert_subquery_pathkeys. Only the UniqueKey insides subquery is referenced as +a Var in outer relation will be reused. The relationship between the outerrel.Var +and subquery.exprs is built with outerel->subroot->processed_tlist. + + +4. As for the SRF functions, it will break the uniqueness of uniquekey, However it +is handled in adjust_paths_for_srfs, which happens after the query_planner. so +we will maintain the UniqueKey until there and reset it to NIL at that +places. This can't help on distinct/group by elimination cases but probably help +in some other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is +semantic correctly. + + +5. As for inherit table, we first main the UnqiueKey on childrel as well. But for +partitioned table we need to maintain 2 different kinds of +UnqiueKey. 1). UniqueKey on the parent relation 2). UniqueKey on child +relation for partition wise query. + +Example: +CREATE TABLE p (a int not null, b int not null) partition by list (a); +CREATE TABLE p0 partition of p for values in (1); +CREATE TABLE p1 partition of p for values in (2); + +create unique index p0_b on p0(b); +create unique index p1_b on p1(b); + +Now b is only unique on partition level, so the distinct can't be removed on +the following cases. SELECT DISTINCT b FROM p; + +Another example is SELECT DISTINCT a, b FROM p WHERE a = 1; Since only one +partition is chosen, the UniqueKey on child relation is same as the UniqueKey on +parent relation. + +Another usage of UniqueKey on partition level is it be helpful for +partition-wise join. + +As for the UniqueKey on parent table level, it comes with 2 different ways, +1). the UniqueKey is also derived in UniqueKey index, but the index must be same +in all the related children relations and the unique index must contains +Partition Key in it. Example: + +CREATE UNIQUE INDEX p_ab ON p(a, b); -- where a is the partition key. + +-- Query +SELECT a, b FROM p; the (a, b) is a UniqueKey of p. + +2). If the parent relation has only one childrel, the UniqueKey on childrel is + the UniqueKey on parent as well. diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index 484dab0a1a..2ad9d06d7a 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -579,6 +579,12 @@ set_plain_rel_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) */ check_index_predicates(root, rel); + /* + * Now that we've marked which partial indexes are suitable, we can now + * build the relation's unique keys. + */ + populate_baserel_uniquekeys(root, rel, rel->indexlist); + /* Mark rel with estimated output rows, width, etc */ set_baserel_size_estimates(root, rel); } @@ -1310,6 +1316,8 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, /* Add paths to the append relation. */ add_paths_to_append_rel(root, rel, live_childrels); + if (IS_PARTITIONED_REL(rel)) + populate_partitionedrel_uniquekeys(root, rel, live_childrels); } @@ -2383,6 +2391,8 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, pathkeys, required_outer)); } + convert_subquery_uniquekeys(root, rel, sub_final_rel); + /* If outer rel allows parallelism, do same for partial paths. */ if (rel->consider_parallel && bms_is_empty(required_outer)) { diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c index db54a6ba2e..ef0fd2fb0b 100644 --- a/src/backend/optimizer/path/joinpath.c +++ b/src/backend/optimizer/path/joinpath.c @@ -71,13 +71,6 @@ static void consider_parallel_mergejoin(PlannerInfo *root, static void hash_inner_and_outer(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, RelOptInfo *innerrel, JoinType jointype, JoinPathExtraData *extra); -static List *select_mergejoin_clauses(PlannerInfo *root, - RelOptInfo *joinrel, - RelOptInfo *outerrel, - RelOptInfo *innerrel, - List *restrictlist, - JoinType jointype, - bool *mergejoin_allowed); static void generate_mergejoin_paths(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *innerrel, @@ -1927,7 +1920,7 @@ hash_inner_and_outer(PlannerInfo *root, * if it is mergejoinable and involves vars from the two sub-relations * currently of interest. */ -static List * +List * select_mergejoin_clauses(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c index 2d343cd293..b9163ee8ff 100644 --- a/src/backend/optimizer/path/joinrels.c +++ b/src/backend/optimizer/path/joinrels.c @@ -924,6 +924,8 @@ populate_joinrel_with_paths(PlannerInfo *root, RelOptInfo *rel1, /* Apply partitionwise join technique, if possible. */ try_partitionwise_join(root, rel1, rel2, joinrel, sjinfo, restrictlist); + + populate_joinrel_uniquekeys(root, joinrel, rel1, rel2, restrictlist, sjinfo->jointype); } diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index ce9bf87e9b..7e596d4194 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -33,7 +33,6 @@ static bool pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys); static bool matches_boolean_partition_clause(RestrictInfo *rinfo, RelOptInfo *partrel, int partkeycol); -static Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); static bool right_merge_direction(PlannerInfo *root, PathKey *pathkey); @@ -1035,7 +1034,7 @@ convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel, * We need this to ensure that we don't return pathkeys describing values * that are unavailable above the level of the subquery scan. */ -static Var * +Var * find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle) { ListCell *lc; diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c new file mode 100644 index 0000000000..b33bcd2f32 --- /dev/null +++ b/src/backend/optimizer/path/uniquekeys.c @@ -0,0 +1,1131 @@ +/*------------------------------------------------------------------------- + * + * uniquekeys.c + * Utilities for matching and building unique keys + * + * Portions Copyright (c) 2020, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/optimizer/path/uniquekeys.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "nodes/makefuncs.h" +#include "nodes/nodeFuncs.h" +#include "optimizer/pathnode.h" +#include "optimizer/paths.h" +#include "optimizer/appendinfo.h" +#include "optimizer/optimizer.h" +#include "optimizer/tlist.h" +#include "rewrite/rewriteManip.h" + + +/* + * This struct is used to help populate_joinrel_uniquekeys. + * + * added_to_joinrel is true if a uniquekey (from outerrel or innerrel) + * has been added to joinrel. + * useful is true if the exprs of the uniquekey still appears in joinrel. + */ +typedef struct UniqueKeyContextData +{ + UniqueKey *uniquekey; + bool added_to_joinrel; + bool useful; +} *UniqueKeyContext; + +static List *initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel); +static bool innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + bool reverse); + +static List *get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals); +static List *get_exprs_from_uniquekey(RelOptInfo *joinrel, + RelOptInfo *rel1, + UniqueKey *ukey); +static void add_uniquekey_for_onerow(RelOptInfo *rel); +static bool add_combined_uniquekey(RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype); + +/* Used for unique indexes checking for partitioned table */ +static bool index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind); +static IndexOptInfo *simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from); +static bool simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2); +static void adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_index); + +/* Helper function for grouped relation and distinct relation. */ +static void add_uniquekey_from_sortgroups(PlannerInfo *root, + RelOptInfo *rel, + List *sortgroups); + +/* + * populate_baserel_uniquekeys + * Populate 'baserel' uniquekeys list by looking at the rel's unique index + * and baserestrictinfo + */ +void +populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List *indexlist) +{ + ListCell *lc; + List *matched_uniq_indexes = NIL; + + /* Attrs appears in rel->reltarget->exprs. */ + Bitmapset *used_attrs = NULL; + + List *const_exprs = NIL; + List *expr_opfamilies = NIL; + + Assert(baserel->rtekind == RTE_RELATION); + + foreach(lc, indexlist) + { + IndexOptInfo *ind = (IndexOptInfo *) lfirst(lc); + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + matched_uniq_indexes = lappend(matched_uniq_indexes, ind); + } + + if (matched_uniq_indexes == NIL) + return; + + /* Check which attrs is used in baserel->reltarget */ + pull_varattnos((Node *)baserel->reltarget->exprs, baserel->relid, &used_attrs); + + /* Check which attrno is used at a mergeable const filter */ + foreach(lc, baserel->baserestrictinfo) + { + RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); + + if (rinfo->mergeopfamilies == NIL) + continue; + + if (bms_is_empty(rinfo->left_relids)) + { + const_exprs = lappend(const_exprs, get_rightop(rinfo->clause)); + } + else if (bms_is_empty(rinfo->right_relids)) + { + const_exprs = lappend(const_exprs, get_leftop(rinfo->clause)); + } + else + continue; + + expr_opfamilies = lappend(expr_opfamilies, rinfo->mergeopfamilies); + } + + foreach(lc, matched_uniq_indexes) + { + bool multi_nullvals, useful; + List *exprs = get_exprs_from_uniqueindex(lfirst_node(IndexOptInfo, lc), + const_exprs, + expr_opfamilies, + used_attrs, + &useful, + &multi_nullvals); + if (useful) + { + if (exprs == NIL) + { + /* All the columns in Unique Index matched with a restrictinfo */ + add_uniquekey_for_onerow(baserel); + return; + } + baserel->uniquekeys = lappend(baserel->uniquekeys, + makeUniqueKey(exprs, multi_nullvals)); + } + } +} + + +/* + * populate_partitionedrel_uniquekeys + * The UniqueKey on partitionrel comes from 2 cases: + * 1). Only one partition is involved in this query, the unique key can be + * copied to parent rel from childrel. + * 2). There are some unique index which includes partition key and exists + * in all the related partitions. + * We never mind rule 2 if we hit rule 1. + */ + +void +populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels) +{ + ListCell *lc; + List *global_uniq_indexlist = NIL; + RelOptInfo *childrel; + bool is_first = true; + + Assert(IS_PARTITIONED_REL(rel)); + + if (childrels == NIL) + return; + + /* + * If there is only one partition used in this query, the UniqueKey in childrel is + * still valid in parent level, but we need convert the format from child expr to + * parent expr. + */ + if (list_length(childrels) == 1) + { + /* Check for Rule 1 */ + RelOptInfo *childrel = linitial_node(RelOptInfo, childrels); + ListCell *lc; + Assert(childrel->reloptkind == RELOPT_OTHER_MEMBER_REL); + if (relation_is_onerow(childrel)) + { + add_uniquekey_for_onerow(rel); + return; + } + + foreach(lc, childrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + AppendRelInfo *appinfo = find_appinfo_by_child(root, childrel->relid); + List *parent_exprs = NIL; + bool can_reuse = true; + ListCell *lc2; + foreach(lc2, ukey->exprs) + { + Var *var = (Var *)lfirst(lc2); + /* + * If the expr comes from a expression, it is hard to build the expression + * in parent so ignore that case for now. + */ + if(!IsA(var, Var)) + { + can_reuse = false; + break; + } + /* Convert it to parent var */ + parent_exprs = lappend(parent_exprs, find_parent_var(appinfo, var)); + } + if (can_reuse) + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(parent_exprs, + ukey->multi_nullvals)); + } + } + else + { + /* Check for rule 2 */ + childrel = linitial_node(RelOptInfo, childrels); + foreach(lc, childrel->indexlist) + { + IndexOptInfo *ind = lfirst(lc); + IndexOptInfo *modified_index; + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + + /* + * During simple_copy_indexinfo_to_parent, we need to convert var from + * child var to parent var, index on expression is too complex to handle. + * so ignore it for now. + */ + if (ind->indexprs != NIL) + continue; + + modified_index = simple_copy_indexinfo_to_parent(root, rel, ind); + /* + * If the unique index doesn't contain partkey, then it is unique + * on this partition only, so it is useless for us. + */ + if (!index_constains_partkey(rel, modified_index)) + continue; + + global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index); + } + + if (global_uniq_indexlist != NIL) + { + foreach(lc, childrels) + { + RelOptInfo *child = lfirst(lc); + if (is_first) + { + is_first = false; + continue; + } + adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist); + } + /* Now we have a list of unique index which are exactly same on all childrels, + * Set the UniqueKey just like it is non-partition table + */ + populate_baserel_uniquekeys(root, rel, global_uniq_indexlist); + } + } +} + + +/* + * populate_distinctrel_uniquekeys + */ +void +populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel) +{ + /* The unique key before the distinct is still valid. */ + distinctrel->uniquekeys = list_copy(inputrel->uniquekeys); + add_uniquekey_from_sortgroups(root, distinctrel, root->parse->distinctClause); +} + +/* + * populate_grouprel_uniquekeys + */ +void +populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel) + +{ + Query *parse = root->parse; + bool input_ukey_added = false; + ListCell *lc; + + if (relation_is_onerow(inputrel)) + { + add_uniquekey_for_onerow(grouprel); + return; + } + if (parse->groupingSets) + return; + + /* A Normal group by without grouping set. */ + if (parse->groupClause) + { + /* + * Current even the groupby clause is Unique already, but if query has aggref + * We have to create grouprel still. To keep the UnqiueKey short, we will check + * the UniqueKey of input_rel still valid, if so we reuse it. + */ + foreach(lc, inputrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (list_is_subset(ukey->exprs, grouprel->reltarget->exprs)) + { + grouprel->uniquekeys = lappend(grouprel->uniquekeys, + ukey); + input_ukey_added = true; + } + } + if (!input_ukey_added) + /* + * group by clause must be a super-set of grouprel->reltarget->exprs except the + * aggregation expr, so if such exprs is unique already, no bother to generate + * new uniquekey for group by exprs. + */ + add_uniquekey_from_sortgroups(root, + grouprel, + root->parse->groupClause); + } + else + /* It has aggregation but without a group by, so only one row returned */ + add_uniquekey_for_onerow(grouprel); +} + +/* + * simple_copy_uniquekeys + * Using a function for the one-line code makes us easy to check where we simply + * copied the uniquekey. + */ +void +simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel) +{ + newrel->uniquekeys = oldrel->uniquekeys; +} + +/* + * populate_unionrel_uniquekeys + */ +void +populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel) +{ + ListCell *lc; + List *exprs = NIL; + + Assert(unionrel->uniquekeys == NIL); + + foreach(lc, unionrel->reltarget->exprs) + { + exprs = lappend(exprs, lfirst(lc)); + } + + if (exprs == NIL) + /* SQL: select union select; is valid, we need to handle it here. */ + add_uniquekey_for_onerow(unionrel); + else + unionrel->uniquekeys = lappend(unionrel->uniquekeys, + makeUniqueKey(exprs,false)); + +} + +/* + * populate_joinrel_uniquekeys + * + * populate uniquekeys for joinrel. We will check each relation to see if its + * UniqueKey is still valid via innerrel_keeps_unique, if so, we add it to + * joinrel. The multi_nullvals field will be changed to true for some outer + * join cases and one-row UniqueKey needs to be converted to normal UniqueKey + * for the same case as well. + * For the uniquekey in either baserel which can't be unique after join, we still + * check to see if combination of UniqueKeys from both side is still useful for us. + * if yes, we add it to joinrel as well. + */ +void +populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *outerrel, RelOptInfo *innerrel, + List *restrictlist, JoinType jointype) +{ + ListCell *lc, *lc2; + List *clause_list = NIL; + List *outerrel_ukey_ctx; + List *innerrel_ukey_ctx; + bool inner_onerow, outer_onerow; + bool mergejoin_allowed; + + /* Care about the outerrel relation only for SEMI/ANTI join */ + if (jointype == JOIN_SEMI || jointype == JOIN_ANTI) + { + foreach(lc, outerrel->uniquekeys) + { + UniqueKey *uniquekey = lfirst_node(UniqueKey, lc); + if (list_is_subset(uniquekey->exprs, joinrel->reltarget->exprs)) + joinrel->uniquekeys = lappend(joinrel->uniquekeys, uniquekey); + } + return; + } + + Assert(jointype == JOIN_LEFT || jointype == JOIN_FULL || jointype == JOIN_INNER); + + /* Fast path */ + if (innerrel->uniquekeys == NIL || outerrel->uniquekeys == NIL) + return; + + inner_onerow = relation_is_onerow(innerrel); + outer_onerow = relation_is_onerow(outerrel); + + outerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(outerrel); + innerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(innerrel); + + clause_list = select_mergejoin_clauses(root, joinrel, outerrel, innerrel, + restrictlist, jointype, + &mergejoin_allowed); + + if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true /* reverse */)) + { + bool outer_impact = jointype == JOIN_FULL; + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + /* Outer relation has one row, and the unique key is not duplicated after join, + * the joinrel will still has one row unless the jointype == JOIN_FULL. + */ + if (outer_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (outer_onerow) + { + /* + * The onerow outerrel becomes multi rows and multi_nullvals + * will be changed to true. We also need to set the exprs correctly since it + * can't be NIL any more. + */ + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(joinrel, outerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Change multi_nullvals to true due to the full join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, true)); + else + /* Just reuse it */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + } + ctx->added_to_joinrel = true; + } + } + + if (innerrel_keeps_unique(root, outerrel, innerrel, clause_list, false)) + { + bool outer_impact = jointype == JOIN_FULL || jointype == JOIN_LEFT;; + + foreach(lc, innerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + if (inner_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (inner_onerow) + { + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(joinrel, innerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Need to change multi_nullvals to true due to the outer join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, + true)); + else + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + + } + ctx->added_to_joinrel = true; + } + } + + /* + * The combination of the UniqueKey from both sides is unique as well regardless + * of join type, but no bother to add it if its subset has been added to joinrel + * already or it is not useful for the joinrel. + */ + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx1 = (UniqueKeyContext) lfirst(lc); + if (ctx1->added_to_joinrel || !ctx1->useful) + continue; + foreach(lc2, innerrel_ukey_ctx) + { + UniqueKeyContext ctx2 = (UniqueKeyContext) lfirst(lc2); + if (ctx2->added_to_joinrel || !ctx2->useful) + continue; + if (add_combined_uniquekey(joinrel, outerrel, innerrel, + ctx1->uniquekey, ctx2->uniquekey, + jointype)) + /* If we set a onerow UniqueKey to joinrel, we don't need other. */ + return; + } + } +} + + +/* + * convert_subquery_uniquekeys + * + * Covert the UniqueKey in subquery to outer relation. + */ +void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel) +{ + ListCell *lc; + + if (sub_final_rel->uniquekeys == NIL) + return; + + if (relation_is_onerow(sub_final_rel)) + { + add_uniquekey_for_onerow(currel); + return; + } + + Assert(currel->subroot != NULL); + + foreach(lc, sub_final_rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + ListCell *lc; + List *exprs = NIL; + bool ukey_useful = true; + + /* One row case is handled above */ + Assert(ukey->exprs != NIL); + foreach(lc, ukey->exprs) + { + Var *var; + TargetEntry *tle = tlist_member(lfirst(lc), + currel->subroot->processed_tlist); + if (tle == NULL) + { + ukey_useful = false; + break; + } + var = find_var_for_subquery_tle(currel, tle); + if (var == NULL) + { + ukey_useful = false; + break; + } + exprs = lappend(exprs, var); + } + + if (ukey_useful) + currel->uniquekeys = lappend(currel->uniquekeys, + makeUniqueKey(exprs, + ukey->multi_nullvals)); + + } +} + +/* + * innerrel_keeps_unique + * + * Check if Unique key of the innerrel is valid after join. innerrel's UniqueKey + * will be still valid if innerrel's any-column mergeop outrerel's uniquekey + * exists in clause_list. + * + * Note: the clause_list must be a list of mergeable restrictinfo already. + */ +static bool +innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *clause_list, + bool reverse) +{ + ListCell *lc, *lc2, *lc3; + + if (outerrel->uniquekeys == NIL || innerrel->uniquekeys == NIL) + return false; + + /* Check if there is outerrel's uniquekey in mergeable clause. */ + foreach(lc, outerrel->uniquekeys) + { + List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs; + bool clauselist_matchs_all_exprs = true; + foreach(lc2, outer_uq_exprs) + { + Node *outer_uq_expr = lfirst(lc2); + bool find_uq_expr_in_clauselist = false; + foreach(lc3, clause_list) + { + RestrictInfo *rinfo = lfirst_node(RestrictInfo, lc3); + Node *outer_expr; + if (reverse) + outer_expr = rinfo->outer_is_left ? get_rightop(rinfo->clause) : get_leftop(rinfo->clause); + else + outer_expr = rinfo->outer_is_left ? get_leftop(rinfo->clause) : get_rightop(rinfo->clause); + if (equal(outer_expr, outer_uq_expr)) + { + find_uq_expr_in_clauselist = true; + break; + } + } + if (!find_uq_expr_in_clauselist) + { + /* No need to check the next exprs in the current uniquekey */ + clauselist_matchs_all_exprs = false; + break; + } + } + + if (clauselist_matchs_all_exprs) + return true; + } + return false; +} + + +/* + * relation_is_onerow + * Check if it is a one-row relation by checking UniqueKey. + */ +bool +relation_is_onerow(RelOptInfo *rel) +{ + UniqueKey *ukey; + if (rel->uniquekeys == NIL) + return false; + ukey = linitial_node(UniqueKey, rel->uniquekeys); + return ukey->exprs == NIL && list_length(rel->uniquekeys) == 1; +} + +/* + * relation_has_uniquekeys_for + * Returns true if we have proofs that 'rel' cannot return multiple rows with + * the same values in each of 'exprs'. Otherwise returns false. + */ +bool +relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel, + List *exprs, bool allow_multinulls) +{ + ListCell *lc; + + /* + * For UniqueKey->onerow case, the uniquekey->exprs is empty as well + * so we can't rely on list_is_subset to handle this special cases + */ + if (exprs == NIL) + return false; + + foreach(lc, rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (ukey->multi_nullvals && !allow_multinulls) + continue; + if (list_is_subset(ukey->exprs, exprs)) + return true; + } + return false; +} + + +/* + * get_exprs_from_uniqueindex + * + * Return a list of exprs which is unique. set useful to false if this + * unique index is not useful for us. + */ +static List * +get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals) +{ + List *exprs = NIL; + ListCell *indexpr_item; + int c = 0; + + *useful = true; + *multi_nullvals = false; + + indexpr_item = list_head(unique_index->indexprs); + for(c = 0; c < unique_index->ncolumns; c++) + { + int attr = unique_index->indexkeys[c]; + Expr *expr; + bool matched_const = false; + ListCell *lc1, *lc2; + + if(attr > 0) + { + expr = list_nth_node(TargetEntry, unique_index->indextlist, c)->expr; + } + else if (attr == 0) + { + /* Expression index */ + expr = lfirst(indexpr_item); + indexpr_item = lnext(unique_index->indexprs, indexpr_item); + } + else /* attr < 0 */ + { + /* Index on system column is not supported */ + Assert(false); + } + + /* + * Check index_col = Const case with regarding to opfamily checking + * If we can remove the index_col from the final UniqueKey->exprs. + */ + forboth(lc1, const_exprs, lc2, const_expr_opfamilies) + { + if (list_member_oid((List *)lfirst(lc2), unique_index->opfamily[c]) + && match_index_to_operand((Node *) lfirst(lc1), c, unique_index)) + { + matched_const = true; + break; + } + } + + if (matched_const) + continue; + + /* Check if the indexed expr is used in rel */ + if (attr > 0) + { + /* + * Normal Indexed column, if the col is not used, then the index is useless + * for uniquekey. + */ + attr -= FirstLowInvalidHeapAttributeNumber; + + if (!bms_is_member(attr, used_varattrs)) + { + *useful = false; + break; + } + } + else if (!list_member(unique_index->rel->reltarget->exprs, expr)) + { + /* Expression index but the expression is not used in rel */ + *useful = false; + break; + } + + /* check not null property. */ + if (attr == 0) + { + /* We never know if a expression yields null or not */ + *multi_nullvals = true; + } + else if (!bms_is_member(attr, unique_index->rel->notnullattrs) + && !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber, + unique_index->rel->notnullattrs)) + { + *multi_nullvals = true; + } + + exprs = lappend(exprs, expr); + } + return exprs; +} + + +/* + * add_uniquekey_for_onerow + * If we are sure that the relation only returns one row, then all the columns + * are unique. However we don't need to create UniqueKey for every column, we + * just set exprs = NIL and overwrites all the other UniqueKey on this RelOptInfo + * since this one has strongest semantics. + */ +void +add_uniquekey_for_onerow(RelOptInfo *rel) +{ + /* + * We overwrite the previous UniqueKey on purpose since this one has the + * strongest semantic. + */ + rel->uniquekeys = list_make1(makeUniqueKey(NIL, false)); +} + + +/* + * initililze_uniquecontext_for_joinrel + * Return a List of UniqueKeyContext for an inputrel + */ +static List * +initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel) +{ + List *res = NIL; + ListCell *lc; + foreach(lc, inputrel->uniquekeys) + { + UniqueKeyContext context; + context = palloc(sizeof(struct UniqueKeyContextData)); + context->uniquekey = lfirst_node(UniqueKey, lc); + context->added_to_joinrel = false; + context->useful = true; + res = lappend(res, context); + } + return res; +} + + +/* + * get_exprs_from_uniquekey + * Unify the way of get List of exprs from a one-row UniqueKey or + * normal UniqueKey. for the onerow case, every expr in rel1 is a valid + * UniqueKey. Return a List of exprs. + * + * rel1: The relation which you want to get the exprs. + * ukey: The UniqueKey you want to get the exprs. + */ +static List * +get_exprs_from_uniquekey(RelOptInfo *joinrel, RelOptInfo *rel1, UniqueKey *ukey) +{ + ListCell *lc; + bool onerow = rel1 != NULL && relation_is_onerow(rel1); + + List *res = NIL; + Assert(onerow || ukey); + if (onerow) + { + /* Only cares about the exprs still exist in joinrel */ + foreach(lc, joinrel->reltarget->exprs) + { + Bitmapset *relids = pull_varnos(lfirst(lc)); + if (bms_is_subset(relids, rel1->relids)) + { + res = lappend(res, list_make1(lfirst(lc))); + } + } + } + else + { + res = list_make1(ukey->exprs); + } + return res; +} + +/* + * Partitioned table Unique Keys. + * The partition table unique key is maintained as: + * 1. The index must be unique as usual. + * 2. The index must contains partition key. + * 3. The index must exist on all the child rel. see simple_indexinfo_equal for + * how we compare it. + */ + +/* + * index_constains_partkey + * return true if the index contains the partiton key. + */ +static bool +index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind) +{ + ListCell *lc; + int i; + Assert(IS_PARTITIONED_REL(partrel)); + Assert(partrel->part_scheme->partnatts > 0); + + for(i = 0; i < partrel->part_scheme->partnatts; i++) + { + Node *part_expr = linitial(partrel->partexprs[i]); + bool found_in_index = false; + foreach(lc, ind->indextlist) + { + Expr *index_expr = lfirst_node(TargetEntry, lc)->expr; + if (equal(index_expr, part_expr)) + { + found_in_index = true; + break; + } + } + if (!found_in_index) + return false; + } + return true; +} + +/* + * simple_indexinfo_equal + * + * Used to check if the 2 index is same as each other. The index here + * is COPIED from childrel and did some tiny changes(see + * simple_copy_indexinfo_to_parent) + */ +static bool +simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2) +{ + Size oid_cmp_len = sizeof(Oid) * ind1->ncolumns; + + return ind1->ncolumns == ind2->ncolumns && + ind1->unique == ind2->unique && + memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0 && + memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0 && + memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0 && + memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0 && + equal(get_tlist_exprs(ind1->indextlist, true), + get_tlist_exprs(ind2->indextlist, true)); +} + + +/* + * The below macros are used for simple_copy_indexinfo_to_parent which is so + * customized that I don't want to put it to copyfuncs.c. So copy it here. + */ +#define COPY_POINTER_FIELD(fldname, sz) \ + do { \ + Size _size = (sz); \ + newnode->fldname = palloc(_size); \ + memcpy(newnode->fldname, from->fldname, _size); \ + } while (0) + +#define COPY_NODE_FIELD(fldname) \ + (newnode->fldname = copyObjectImpl(from->fldname)) + +#define COPY_SCALAR_FIELD(fldname) \ + (newnode->fldname = from->fldname) + + +/* + * simple_copy_indexinfo_to_parent (from partition) + * Copy the IndexInfo from child relation to parent relation with some modification, + * which is used to test: + * 1. If the same index exists in all the childrels. + * 2. If the parentrel->reltarget/basicrestrict info matches this index. + */ +static IndexOptInfo * +simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from) +{ + IndexOptInfo *newnode = makeNode(IndexOptInfo); + AppendRelInfo *appinfo = find_appinfo_by_child(root, from->rel->relid); + ListCell *lc; + int idx = 0; + + COPY_SCALAR_FIELD(ncolumns); + COPY_SCALAR_FIELD(nkeycolumns); + COPY_SCALAR_FIELD(unique); + COPY_SCALAR_FIELD(immediate); + /* We just need to know if it is NIL or not */ + COPY_SCALAR_FIELD(indpred); + COPY_SCALAR_FIELD(predOK); + COPY_POINTER_FIELD(indexkeys, from->ncolumns * sizeof(int)); + COPY_POINTER_FIELD(indexcollations, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opfamily, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opcintype, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(sortopfamily, from->ncolumns * sizeof(Oid)); + COPY_NODE_FIELD(indextlist); + + /* Convert index exprs on child expr to expr on parent */ + foreach(lc, newnode->indextlist) + { + TargetEntry *tle = lfirst_node(TargetEntry, lc); + /* Index on expression is ignored */ + Assert(IsA(tle->expr, Var)); + tle->expr = (Expr *) find_parent_var(appinfo, (Var *) tle->expr); + newnode->indexkeys[idx] = castNode(Var, tle->expr)->varattno; + idx++; + } + newnode->rel = parentrel; + return newnode; +} + +/* + * adjust_partition_unique_indexlist + * + * global_unique_indexes: At the beginning, it contains the copy & modified + * unique index from the first partition. And then check if each index in it still + * exists in the following partitions. If no, remove it. at last, it has an + * index list which exists in all the partitions. + */ +static void +adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_indexes) +{ + ListCell *lc, *lc2; + foreach(lc, *global_unique_indexes) + { + IndexOptInfo *g_ind = lfirst_node(IndexOptInfo, lc); + bool found_in_child = false; + + foreach(lc2, childrel->indexlist) + { + IndexOptInfo *p_ind = lfirst_node(IndexOptInfo, lc2); + IndexOptInfo *p_ind_copy; + if (!p_ind->unique || !p_ind->immediate || + (p_ind->indpred != NIL && !p_ind->predOK)) + continue; + p_ind_copy = simple_copy_indexinfo_to_parent(root, parentrel, p_ind); + if (simple_indexinfo_equal(p_ind_copy, g_ind)) + { + found_in_child = true; + break; + } + } + if (!found_in_child) + /* The index doesn't exist in childrel, remove it from global_unique_indexes */ + *global_unique_indexes = foreach_delete_current(*global_unique_indexes, lc); + } +} + +/* Helper function for groupres/distinctrel */ +static void +add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgroups) +{ + Query *parse = root->parse; + List *exprs; + + /* + * XXX: If there are some vars which is not in current levelsup, the semantic is + * imprecise, should we avoid it or not? levelsup = 1 is just a demo, maybe we need to + * check every level other than 0, if so, looks we have to write another + * pull_var_walker. + */ + List *upper_vars = pull_vars_of_level((Node*)sortgroups, 1); + + if (upper_vars != NIL) + return; + + exprs = get_sortgrouplist_exprs(sortgroups, parse->targetList); + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(exprs, + false /* sortgroupclause can't be multi_nullvals */)); +} + + +/* + * add_combined_uniquekey + * The combination of both UniqueKeys is a valid UniqueKey for joinrel no matter + * the jointype. + */ +bool +add_combined_uniquekey(RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype) +{ + + ListCell *lc1, *lc2; + + /* Either side has multi_nullvals or we have outer join, + * the combined UniqueKey has multi_nullvals */ + bool multi_nullvals = outer_ukey->multi_nullvals || + inner_ukey->multi_nullvals || IS_OUTER_JOIN(jointype); + + /* The only case we can get onerow joinrel after join */ + if (relation_is_onerow(outer_rel) + && relation_is_onerow(inner_rel) + && jointype == JOIN_INNER) + { + add_uniquekey_for_onerow(joinrel); + return true; + } + + foreach(lc1, get_exprs_from_uniquekey(joinrel, outer_rel, outer_ukey)) + { + foreach(lc2, get_exprs_from_uniquekey(joinrel, inner_rel, inner_ukey)) + { + List *exprs = list_concat_copy(lfirst_node(List, lc1), lfirst_node(List, lc2)); + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(exprs, + multi_nullvals)); + } + } + return false; +} diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index b406d41e91..0551ae0512 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -2389,6 +2389,8 @@ grouping_planner(PlannerInfo *root, bool inheritance_update, add_path(final_rel, path); } + simple_copy_uniquekeys(current_rel, final_rel); + /* * Generate partial paths for final_rel, too, if outer query levels might * be able to make use of them. @@ -3899,6 +3901,8 @@ create_grouping_paths(PlannerInfo *root, } set_cheapest(grouped_rel); + + populate_grouprel_uniquekeys(root, grouped_rel, input_rel); return grouped_rel; } @@ -4615,7 +4619,7 @@ create_window_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(window_rel); - + simple_copy_uniquekeys(input_rel, window_rel); return window_rel; } @@ -4911,7 +4915,7 @@ create_distinct_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(distinct_rel); - + populate_distinctrel_uniquekeys(root, input_rel, distinct_rel); return distinct_rel; } @@ -5172,6 +5176,8 @@ create_ordered_paths(PlannerInfo *root, */ Assert(ordered_rel->pathlist != NIL); + simple_copy_uniquekeys(input_rel, ordered_rel); + return ordered_rel; } @@ -6049,6 +6055,9 @@ adjust_paths_for_srfs(PlannerInfo *root, RelOptInfo *rel, if (list_length(targets) == 1) return; + /* UniqueKey is not valid after handling the SRF. */ + rel->uniquekeys = NIL; + /* * Stack SRF-evaluation nodes atop each path for the rel. * diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c index 951aed80e7..e94e92937c 100644 --- a/src/backend/optimizer/prep/prepunion.c +++ b/src/backend/optimizer/prep/prepunion.c @@ -689,6 +689,8 @@ generate_union_paths(SetOperationStmt *op, PlannerInfo *root, /* Undo effects of possibly forcing tuple_fraction to 0 */ root->tuple_fraction = save_fraction; + /* Add the UniqueKeys */ + populate_unionrel_uniquekeys(root, result_rel); return result_rel; } diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c index d722063cf3..44c37ecffc 100644 --- a/src/backend/optimizer/util/appendinfo.c +++ b/src/backend/optimizer/util/appendinfo.c @@ -746,3 +746,47 @@ find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos) } return appinfos; } + +/* + * find_appinfo_by_child + * + */ +AppendRelInfo * +find_appinfo_by_child(PlannerInfo *root, Index child_index) +{ + ListCell *lc; + foreach(lc, root->append_rel_list) + { + AppendRelInfo *appinfo = lfirst_node(AppendRelInfo, lc); + if (appinfo->child_relid == child_index) + return appinfo; + } + elog(ERROR, "parent relation cant be found"); + return NULL; +} + +/* + * find_parent_var + * + */ +Var * +find_parent_var(AppendRelInfo *appinfo, Var *child_var) +{ + ListCell *lc; + Var *res = NULL; + Index attno = 1; + foreach(lc, appinfo->translated_vars) + { + Node *child_node = lfirst(lc); + if (equal(child_node, child_var)) + { + res = copyObject(child_var); + res->varattno = attno; + res->varno = appinfo->parent_relid; + } + attno++; + } + if (res == NULL) + elog(ERROR, "parent var can't be found."); + return res; +} diff --git a/src/backend/optimizer/util/inherit.c b/src/backend/optimizer/util/inherit.c index 3132fd35a5..d66b40ec50 100644 --- a/src/backend/optimizer/util/inherit.c +++ b/src/backend/optimizer/util/inherit.c @@ -736,6 +736,7 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, { Node *onecq = (Node *) lfirst(lc2); bool pseudoconstant; + RestrictInfo *child_rinfo; /* check for pseudoconstant (no Vars or volatile functions) */ pseudoconstant = @@ -747,13 +748,14 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, root->hasPseudoConstantQuals = true; } /* reconstitute RestrictInfo with appropriate properties */ - childquals = lappend(childquals, - make_restrictinfo((Expr *) onecq, - rinfo->is_pushed_down, - rinfo->outerjoin_delayed, - pseudoconstant, - rinfo->security_level, - NULL, NULL, NULL)); + child_rinfo = make_restrictinfo((Expr *) onecq, + rinfo->is_pushed_down, + rinfo->outerjoin_delayed, + pseudoconstant, + rinfo->security_level, + NULL, NULL, NULL); + child_rinfo->mergeopfamilies = rinfo->mergeopfamilies; + childquals = lappend(childquals, child_rinfo); /* track minimum security level among child quals */ cq_min_security = Min(cq_min_security, rinfo->security_level); } diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 31d9aedeeb..c83f17acb7 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -16,6 +16,7 @@ #include "nodes/execnodes.h" #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" extern A_Expr *makeA_Expr(A_Expr_Kind kind, List *name, @@ -105,4 +106,6 @@ extern GroupingSet *makeGroupingSet(GroupingSetKind kind, List *content, int loc extern VacuumRelation *makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols); +extern UniqueKey* makeUniqueKey(List *exprs, bool multi_nullvals); + #endif /* MAKEFUNC_H */ diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 381d84b4e4..41110ed888 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -264,6 +264,7 @@ typedef enum NodeTag T_EquivalenceMember, T_PathKey, T_PathTarget, + T_UniqueKey, T_RestrictInfo, T_IndexClause, T_PlaceHolderVar, diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index 9e3ebd488a..02e4458bef 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -730,6 +730,7 @@ typedef struct RelOptInfo QualCost baserestrictcost; /* cost of evaluating the above */ Index baserestrict_min_security; /* min security_level found in * baserestrictinfo */ + List *uniquekeys; /* List of UniqueKey */ List *joininfo; /* RestrictInfo structures for join clauses * involving this rel */ bool has_eclass_joins; /* T means joininfo is incomplete */ @@ -1047,6 +1048,28 @@ typedef struct PathKey } PathKey; +/* + * UniqueKey + * + * Represents the unique properties held by a RelOptInfo. + * + * exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL + * is a special case of UniqueKey, which means there is only 1 row in that + * relation. + * multi_nullvals: true means multi null values may exist in these exprs, so the + * uniqueness is not guaranteed in this case. This field is necessary for + * remove_useless_join & reduce_unique_semijoins where we don't mind these + * duplicated NULL values. It is set to true for 2 cases. One is a unique key + * from a unique index but the related column is nullable. The other one is for + * outer join. see populate_joinrel_uniquekeys for detail. + */ +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + /* * PathTarget * @@ -2473,7 +2496,7 @@ typedef enum * * flags indicating what kinds of grouping are possible. * partial_costs_set is true if the agg_partial_costs and agg_final_costs - * have been initialized. + * have been initialized. * agg_partial_costs gives partial aggregation costs. * agg_final_costs gives finalization costs. * target_parallel_safe is true if target is parallel safe. @@ -2503,8 +2526,8 @@ typedef struct * limit_tuples is an estimated bound on the number of output tuples, * or -1 if no LIMIT or couldn't estimate. * count_est and offset_est are the estimated values of the LIMIT and OFFSET - * expressions computed by preprocess_limit() (see comments for - * preprocess_limit() for more information). + * expressions computed by preprocess_limit() (see comments for + * preprocess_limit() for more information). */ typedef struct { diff --git a/src/include/nodes/pg_list.h b/src/include/nodes/pg_list.h index 14ea2766ad..621f54a9f8 100644 --- a/src/include/nodes/pg_list.h +++ b/src/include/nodes/pg_list.h @@ -528,6 +528,8 @@ extern bool list_member_ptr(const List *list, const void *datum); extern bool list_member_int(const List *list, int datum); extern bool list_member_oid(const List *list, Oid datum); +extern bool list_is_subset(const List *members, const List *target); + extern List *list_delete(List *list, void *datum); extern List *list_delete_ptr(List *list, void *datum); extern List *list_delete_int(List *list, int datum); diff --git a/src/include/optimizer/appendinfo.h b/src/include/optimizer/appendinfo.h index d6a27a60dd..e87c92a054 100644 --- a/src/include/optimizer/appendinfo.h +++ b/src/include/optimizer/appendinfo.h @@ -32,4 +32,7 @@ extern Relids adjust_child_relids_multilevel(PlannerInfo *root, Relids relids, extern AppendRelInfo **find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos); +extern AppendRelInfo *find_appinfo_by_child(PlannerInfo *root, Index child_index); +extern Var *find_parent_var(AppendRelInfo *appinfo, Var *child_var); + #endif /* APPENDINFO_H */ diff --git a/src/include/optimizer/optimizer.h b/src/include/optimizer/optimizer.h index 3e4171056e..9445141263 100644 --- a/src/include/optimizer/optimizer.h +++ b/src/include/optimizer/optimizer.h @@ -23,6 +23,7 @@ #define OPTIMIZER_H #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" /* * We don't want to include nodes/pathnodes.h here, because non-planner @@ -156,6 +157,7 @@ extern TargetEntry *get_sortgroupref_tle(Index sortref, List *targetList); extern TargetEntry *get_sortgroupclause_tle(SortGroupClause *sgClause, List *targetList); +extern Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); extern Node *get_sortgroupclause_expr(SortGroupClause *sgClause, List *targetList); extern List *get_sortgrouplist_exprs(List *sgClauses, diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h index 10b6e81079..9217a8d6c6 100644 --- a/src/include/optimizer/paths.h +++ b/src/include/optimizer/paths.h @@ -240,5 +240,48 @@ extern PathKey *make_canonical_pathkey(PlannerInfo *root, int strategy, bool nulls_first); extern void add_paths_to_append_rel(PlannerInfo *root, RelOptInfo *rel, List *live_childrels); +extern List *select_mergejoin_clauses(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + JoinType jointype, + bool *mergejoin_allowed); + +/* + * uniquekeys.c + * Utilities for matching and building unique keys + */ +extern void populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List* unique_index_list); +extern void populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels); +extern void populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel); +extern void populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel); +extern void populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel); +extern void simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel); +extern void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel); +extern void populate_joinrel_uniquekeys(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + RelOptInfo *rel2, + List *restrictlist, + JoinType jointype); + +extern bool relation_has_uniquekeys_for(PlannerInfo *root, + RelOptInfo *rel, + List *exprs, + bool allow_multinulls); +extern bool relation_is_onerow(RelOptInfo *rel); #endif /* PATHS_H */ -- 2.21.0 --op7mf2i72xk5yu7c Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v36-0003-Extend-UniqueKeys.patch" ^ permalink raw reply [nested|flat] 91+ messages in thread
* [PATCH v38 2/6] Introduce UniqueKey attributes on RelOptInfo struct. @ 2020-05-11 07:50 一挃 <[email protected]> 0 siblings, 0 replies; 91+ messages in thread From: 一挃 @ 2020-05-11 07:50 UTC (permalink / raw) UniqueKey is a set of exprs on RelOptInfo which represents the exprs will be unique on the given RelOptInfo. You can see README.uniquekey for more information. --- src/backend/nodes/copyfuncs.c | 13 + src/backend/nodes/list.c | 31 + src/backend/nodes/makefuncs.c | 13 + src/backend/nodes/outfuncs.c | 11 + src/backend/nodes/readfuncs.c | 10 + src/backend/optimizer/path/Makefile | 3 +- src/backend/optimizer/path/README.uniquekey | 131 +++ src/backend/optimizer/path/allpaths.c | 10 + src/backend/optimizer/path/joinpath.c | 9 +- src/backend/optimizer/path/joinrels.c | 2 + src/backend/optimizer/path/pathkeys.c | 3 +- src/backend/optimizer/path/uniquekeys.c | 1135 +++++++++++++++++++ src/backend/optimizer/plan/planner.c | 13 +- src/backend/optimizer/prep/prepunion.c | 2 + src/backend/optimizer/util/appendinfo.c | 44 + src/backend/optimizer/util/inherit.c | 18 +- src/include/nodes/makefuncs.h | 3 + src/include/nodes/nodes.h | 1 + src/include/nodes/pathnodes.h | 29 +- src/include/nodes/pg_list.h | 10 + src/include/optimizer/appendinfo.h | 3 + src/include/optimizer/optimizer.h | 2 + src/include/optimizer/paths.h | 43 + 23 files changed, 1515 insertions(+), 24 deletions(-) create mode 100644 src/backend/optimizer/path/README.uniquekey create mode 100644 src/backend/optimizer/path/uniquekeys.c diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..75c1c5e824 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2296,6 +2296,16 @@ _copyPathKey(const PathKey *from) return newnode; } +static UniqueKey * +_copyUniqueKey(const UniqueKey *from) +{ + UniqueKey *newnode = makeNode(UniqueKey); + + COPY_NODE_FIELD(exprs); + COPY_SCALAR_FIELD(multi_nullvals); + + return newnode; +} /* * _copyRestrictInfo */ @@ -5220,6 +5230,9 @@ copyObjectImpl(const void *from) case T_PathKey: retval = _copyPathKey(from); break; + case T_UniqueKey: + retval = _copyUniqueKey(from); + break; case T_RestrictInfo: retval = _copyRestrictInfo(from); break; diff --git a/src/backend/nodes/list.c b/src/backend/nodes/list.c index dbf6b30233..ca099495a1 100644 --- a/src/backend/nodes/list.c +++ b/src/backend/nodes/list.c @@ -702,6 +702,37 @@ list_member_oid(const List *list, Oid datum) return false; } +/* + * return true iff every entry in "members" list is also present + * in the "target" list. + */ +bool +list_is_subset(const List *members, const List *target) +{ + const ListCell *lc1, *lc2; + + Assert(IsPointerList(members)); + Assert(IsPointerList(target)); + check_list_invariants(members); + check_list_invariants(target); + + foreach(lc1, members) + { + bool found = false; + foreach(lc2, target) + { + if (equal(lfirst(lc1), lfirst(lc2))) + { + found = true; + break; + } + } + if (!found) + return false; + } + return true; +} + /* * Delete the n'th cell (counting from 0) in list. * diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 01c110cd2f..40415d0f5b 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -815,3 +815,16 @@ makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols) v->va_cols = va_cols; return v; } + + +/* + * makeUniqueKey + */ +UniqueKey* +makeUniqueKey(List *exprs, bool multi_nullvals) +{ + UniqueKey * ukey = makeNode(UniqueKey); + ukey->exprs = exprs; + ukey->multi_nullvals = multi_nullvals; + return ukey; +} diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..44154cde6a 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2456,6 +2456,14 @@ _outPathKey(StringInfo str, const PathKey *node) WRITE_BOOL_FIELD(pk_nulls_first); } +static void +_outUniqueKey(StringInfo str, const UniqueKey *node) +{ + WRITE_NODE_TYPE("UNIQUEKEY"); + WRITE_NODE_FIELD(exprs); + WRITE_BOOL_FIELD(multi_nullvals); +} + static void _outPathTarget(StringInfo str, const PathTarget *node) { @@ -4198,6 +4206,9 @@ outNode(StringInfo str, const void *obj) case T_PathKey: _outPathKey(str, obj); break; + case T_UniqueKey: + _outUniqueKey(str, obj); + break; case T_PathTarget: _outPathTarget(str, obj); break; diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index c5e136e9c3..b3e212bf1c 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -492,6 +492,14 @@ _readSetOperationStmt(void) READ_DONE(); } +static UniqueKey * +_readUniqueKey(void) +{ + READ_LOCALS(UniqueKey); + READ_NODE_FIELD(exprs); + READ_BOOL_FIELD(multi_nullvals); + READ_DONE(); +} /* * Stuff from primnodes.h. @@ -2717,6 +2725,8 @@ parseNodeString(void) return_value = _readCommonTableExpr(); else if (MATCH("SETOPERATIONSTMT", 16)) return_value = _readSetOperationStmt(); + else if (MATCH("UNIQUEKEY", 9)) + return_value = _readUniqueKey(); else if (MATCH("ALIAS", 5)) return_value = _readAlias(); else if (MATCH("RANGEVAR", 8)) diff --git a/src/backend/optimizer/path/Makefile b/src/backend/optimizer/path/Makefile index 1e199ff66f..7b9820c25f 100644 --- a/src/backend/optimizer/path/Makefile +++ b/src/backend/optimizer/path/Makefile @@ -21,6 +21,7 @@ OBJS = \ joinpath.o \ joinrels.o \ pathkeys.o \ - tidpath.o + tidpath.o \ + uniquekeys.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/optimizer/path/README.uniquekey b/src/backend/optimizer/path/README.uniquekey new file mode 100644 index 0000000000..5eac761995 --- /dev/null +++ b/src/backend/optimizer/path/README.uniquekey @@ -0,0 +1,131 @@ +1. What is UniqueKey? +We can think UniqueKey is a set of exprs for a RelOptInfo, which we are insure +that doesn't yields same result among all the rows. The simplest UniqueKey +format is primary key. + +However we define the UnqiueKey as below. + +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + +exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL +is a special case of UniqueKey, which means there is only one row in that +relation.it has a stronger semantic than others. like SELECT uk FROM t; uk is +normal unique key and may have different values. SELECT colx FROM t WHERE uk = +const. colx is unique AND we have only 1 value. This field can used for +innerrel_is_unique. this logic is handled specially in add_uniquekey_for_onerow +function. + +multi_nullvals: true means multi null values may exist in these exprs, so the +uniqueness is not guaranteed in this case. This field is necessary for +remove_useless_join & reduce_unique_semijoins where we don't mind these +duplicated NULL values. It is set to true for 2 cases. One is a unique key +from a unique index but the related column is nullable. The other one is for +outer join. see populate_joinrel_uniquekeys for detail. + + +The UniqueKey can be used at the following cases at least: +1. remove_useless_joins. +2. reduce_semianti_joins +3. remove distinct node if distinct clause is unique. +4. remove aggnode if group by clause is unique. +5. Index Skip Scan (WIP) +6. Aggregation Push Down without 2 phase aggregation if the join can't + duplicated the aggregated rows. (work in progress feature) + +2. How is it maintained? + +We have a set of populate_xxx_unqiuekeys functions to maintain the uniquekey on +various cases. xxx includes baserel, joinrel, partitionedrel, distinctrel, +groupedrel, unionrel. and we also need to convert the uniquekey from subquery +to outer relation, which is what convert_subquery_uniquekeys does. + +1. The first part is about baserel. We handled 3 cases. suppose we have Unique +Index on (a, b). + +1. SELECT a, b FROM t. UniqueKey (a, b) +2. SELECT a FROM t WHERE b = 1; UniqueKey (a) +3. SELECT .. FROM t WHERE a = 1 AND b = 1; UniqueKey (NIL). onerow case, every + column is Unique. + +2. The next part is joinrel, this part is most error-prone, we simplified the rules +like below: +1. If the relation's UniqueKey can't be duplicated after join, then is will be + still valid for the join rel. The function we used here is + innerrel_keeps_unique. The basic idea is innerrel.any_col = outer.uk. + +2. If the UnqiueKey can't keep valid via the rule 1, the combination of the + UniqueKey from both sides are valid for sure. We can prove this as: if the + unique exprs from rel1 is duplicated by rel2, the duplicated rows must + contains different unique exprs from rel2. + +More considerations about onerow: +1. If relation with one row and it can't be duplicated, it is still possible + contains mulit_nullvas after outer join. +2. If the either UniqueKey can be duplicated after join, the can get one row + only when both side is one row AND there is no outer join. +3. Whenever the onerow UniqueKey is not a valid any more, we need to convert one + row UniqueKey to normal unique key since we don't store exprs for one-row + relation. get_exprs_from_uniquekeys will be used here. + + +More considerations about multi_nullvals after join: +1. If the original UnqiueKey has multi_nullvals, the final UniqueKey will have + mulit_nullvals in any case. +2. If a unique key doesn't allow mulit_nullvals, after some outer join, it + allows some outer join. + + +3. When we comes to subquery, we need to convert_subquery_unqiuekeys just like +convert_subquery_pathkeys. Only the UniqueKey insides subquery is referenced as +a Var in outer relation will be reused. The relationship between the outerrel.Var +and subquery.exprs is built with outerel->subroot->processed_tlist. + + +4. As for the SRF functions, it will break the uniqueness of uniquekey, However it +is handled in adjust_paths_for_srfs, which happens after the query_planner. so +we will maintain the UniqueKey until there and reset it to NIL at that +places. This can't help on distinct/group by elimination cases but probably help +in some other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is +semantic correctly. + + +5. As for inherit table, we first main the UnqiueKey on childrel as well. But for +partitioned table we need to maintain 2 different kinds of +UnqiueKey. 1). UniqueKey on the parent relation 2). UniqueKey on child +relation for partition wise query. + +Example: +CREATE TABLE p (a int not null, b int not null) partition by list (a); +CREATE TABLE p0 partition of p for values in (1); +CREATE TABLE p1 partition of p for values in (2); + +create unique index p0_b on p0(b); +create unique index p1_b on p1(b); + +Now b is only unique on partition level, so the distinct can't be removed on +the following cases. SELECT DISTINCT b FROM p; + +Another example is SELECT DISTINCT a, b FROM p WHERE a = 1; Since only one +partition is chosen, the UniqueKey on child relation is same as the UniqueKey on +parent relation. + +Another usage of UniqueKey on partition level is it be helpful for +partition-wise join. + +As for the UniqueKey on parent table level, it comes with 2 different ways, +1). the UniqueKey is also derived in UniqueKey index, but the index must be same +in all the related children relations and the unique index must contains +Partition Key in it. Example: + +CREATE UNIQUE INDEX p_ab ON p(a, b); -- where a is the partition key. + +-- Query +SELECT a, b FROM p; the (a, b) is a UniqueKey of p. + +2). If the parent relation has only one childrel, the UniqueKey on childrel is + the UniqueKey on parent as well. diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index 37b4223adb..d52ad59f89 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -579,6 +579,12 @@ set_plain_rel_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) */ check_index_predicates(root, rel); + /* + * Now that we've marked which partial indexes are suitable, we can now + * build the relation's unique keys. + */ + populate_baserel_uniquekeys(root, rel, rel->indexlist); + /* Mark rel with estimated output rows, width, etc */ set_baserel_size_estimates(root, rel); } @@ -1297,6 +1303,8 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, /* Add paths to the append relation. */ add_paths_to_append_rel(root, rel, live_childrels); + if (IS_PARTITIONED_REL(rel)) + populate_partitionedrel_uniquekeys(root, rel, live_childrels); } @@ -2304,6 +2312,8 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, pathkeys, required_outer)); } + convert_subquery_uniquekeys(root, rel, sub_final_rel); + /* If outer rel allows parallelism, do same for partial paths. */ if (rel->consider_parallel && bms_is_empty(required_outer)) { diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c index 57ce97fd53..697ac047fb 100644 --- a/src/backend/optimizer/path/joinpath.c +++ b/src/backend/optimizer/path/joinpath.c @@ -71,13 +71,6 @@ static void consider_parallel_mergejoin(PlannerInfo *root, static void hash_inner_and_outer(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, RelOptInfo *innerrel, JoinType jointype, JoinPathExtraData *extra); -static List *select_mergejoin_clauses(PlannerInfo *root, - RelOptInfo *joinrel, - RelOptInfo *outerrel, - RelOptInfo *innerrel, - List *restrictlist, - JoinType jointype, - bool *mergejoin_allowed); static void generate_mergejoin_paths(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *innerrel, @@ -1927,7 +1920,7 @@ hash_inner_and_outer(PlannerInfo *root, * if it is mergejoinable and involves vars from the two sub-relations * currently of interest. */ -static List * +List * select_mergejoin_clauses(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c index 0dbe2ac726..7271f044ec 100644 --- a/src/backend/optimizer/path/joinrels.c +++ b/src/backend/optimizer/path/joinrels.c @@ -924,6 +924,8 @@ populate_joinrel_with_paths(PlannerInfo *root, RelOptInfo *rel1, /* Apply partitionwise join technique, if possible. */ try_partitionwise_join(root, rel1, rel2, joinrel, sjinfo, restrictlist); + + populate_joinrel_uniquekeys(root, joinrel, rel1, rel2, restrictlist, sjinfo->jointype); } diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index bd9a176d7d..139278829b 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -33,7 +33,6 @@ static bool pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys); static bool matches_boolean_partition_clause(RestrictInfo *rinfo, RelOptInfo *partrel, int partkeycol); -static Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); static bool right_merge_direction(PlannerInfo *root, PathKey *pathkey); @@ -1035,7 +1034,7 @@ convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel, * We need this to ensure that we don't return pathkeys describing values * that are unavailable above the level of the subquery scan. */ -static Var * +Var * find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle) { ListCell *lc; diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c new file mode 100644 index 0000000000..77ed2b2eff --- /dev/null +++ b/src/backend/optimizer/path/uniquekeys.c @@ -0,0 +1,1135 @@ +/*------------------------------------------------------------------------- + * + * uniquekeys.c + * Utilities for matching and building unique keys + * + * Portions Copyright (c) 2020, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/optimizer/path/uniquekeys.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "nodes/makefuncs.h" +#include "nodes/nodeFuncs.h" +#include "optimizer/pathnode.h" +#include "optimizer/paths.h" +#include "optimizer/appendinfo.h" +#include "optimizer/optimizer.h" +#include "optimizer/tlist.h" +#include "rewrite/rewriteManip.h" + + +/* + * This struct is used to help populate_joinrel_uniquekeys. + * + * added_to_joinrel is true if a uniquekey (from outerrel or innerrel) + * has been added to joinrel. + * useful is true if the exprs of the uniquekey still appears in joinrel. + */ +typedef struct UniqueKeyContextData +{ + UniqueKey *uniquekey; + bool added_to_joinrel; + bool useful; +} *UniqueKeyContext; + +static List *initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel); +static bool innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + bool reverse); + +static List *get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals); +static List *get_exprs_from_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + UniqueKey *ukey); +static void add_uniquekey_for_onerow(RelOptInfo *rel); +static bool add_combined_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype); + +/* Used for unique indexes checking for partitioned table */ +static bool index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind); +static IndexOptInfo *simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from); +static bool simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2); +static void adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_index); + +/* Helper function for grouped relation and distinct relation. */ +static void add_uniquekey_from_sortgroups(PlannerInfo *root, + RelOptInfo *rel, + List *sortgroups); + +/* + * populate_baserel_uniquekeys + * Populate 'baserel' uniquekeys list by looking at the rel's unique index + * and baserestrictinfo + */ +void +populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List *indexlist) +{ + ListCell *lc; + List *matched_uniq_indexes = NIL; + + /* Attrs appears in rel->reltarget->exprs. */ + Bitmapset *used_attrs = NULL; + + List *const_exprs = NIL; + List *expr_opfamilies = NIL; + + Assert(baserel->rtekind == RTE_RELATION); + + foreach(lc, indexlist) + { + IndexOptInfo *ind = (IndexOptInfo *) lfirst(lc); + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + matched_uniq_indexes = lappend(matched_uniq_indexes, ind); + } + + if (matched_uniq_indexes == NIL) + return; + + /* Check which attrs is used in baserel->reltarget */ + pull_varattnos((Node *)baserel->reltarget->exprs, baserel->relid, &used_attrs); + + /* Check which attrno is used at a mergeable const filter */ + foreach(lc, baserel->baserestrictinfo) + { + RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); + + if (rinfo->mergeopfamilies == NIL) + continue; + + if (bms_is_empty(rinfo->left_relids)) + { + const_exprs = lappend(const_exprs, get_rightop(rinfo->clause)); + } + else if (bms_is_empty(rinfo->right_relids)) + { + const_exprs = lappend(const_exprs, get_leftop(rinfo->clause)); + } + else + continue; + + expr_opfamilies = lappend(expr_opfamilies, rinfo->mergeopfamilies); + } + + foreach(lc, matched_uniq_indexes) + { + bool multi_nullvals, useful; + List *exprs = get_exprs_from_uniqueindex(lfirst_node(IndexOptInfo, lc), + const_exprs, + expr_opfamilies, + used_attrs, + &useful, + &multi_nullvals); + if (useful) + { + if (exprs == NIL) + { + /* All the columns in Unique Index matched with a restrictinfo */ + add_uniquekey_for_onerow(baserel); + return; + } + baserel->uniquekeys = lappend(baserel->uniquekeys, + makeUniqueKey(exprs, multi_nullvals)); + } + } +} + + +/* + * populate_partitionedrel_uniquekeys + * The UniqueKey on partitionrel comes from 2 cases: + * 1). Only one partition is involved in this query, the unique key can be + * copied to parent rel from childrel. + * 2). There are some unique index which includes partition key and exists + * in all the related partitions. + * We never mind rule 2 if we hit rule 1. + */ + +void +populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels) +{ + ListCell *lc; + List *global_uniq_indexlist = NIL; + RelOptInfo *childrel; + bool is_first = true; + + Assert(IS_PARTITIONED_REL(rel)); + + if (childrels == NIL) + return; + + /* + * If there is only one partition used in this query, the UniqueKey in childrel is + * still valid in parent level, but we need convert the format from child expr to + * parent expr. + */ + if (list_length(childrels) == 1) + { + /* Check for Rule 1 */ + RelOptInfo *childrel = linitial_node(RelOptInfo, childrels); + ListCell *lc; + Assert(childrel->reloptkind == RELOPT_OTHER_MEMBER_REL); + if (relation_is_onerow(childrel)) + { + add_uniquekey_for_onerow(rel); + return; + } + + foreach(lc, childrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + AppendRelInfo *appinfo = find_appinfo_by_child(root, childrel->relid); + List *parent_exprs = NIL; + bool can_reuse = true; + ListCell *lc2; + foreach(lc2, ukey->exprs) + { + Var *var = (Var *)lfirst(lc2); + /* + * If the expr comes from a expression, it is hard to build the expression + * in parent so ignore that case for now. + */ + if(!IsA(var, Var)) + { + can_reuse = false; + break; + } + /* Convert it to parent var */ + parent_exprs = lappend(parent_exprs, find_parent_var(appinfo, var)); + } + if (can_reuse) + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(parent_exprs, + ukey->multi_nullvals)); + } + } + else + { + /* Check for rule 2 */ + childrel = linitial_node(RelOptInfo, childrels); + foreach(lc, childrel->indexlist) + { + IndexOptInfo *ind = lfirst(lc); + IndexOptInfo *modified_index; + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + + /* + * During simple_copy_indexinfo_to_parent, we need to convert var from + * child var to parent var, index on expression is too complex to handle. + * so ignore it for now. + */ + if (ind->indexprs != NIL) + continue; + + modified_index = simple_copy_indexinfo_to_parent(root, rel, ind); + /* + * If the unique index doesn't contain partkey, then it is unique + * on this partition only, so it is useless for us. + */ + if (!index_constains_partkey(rel, modified_index)) + continue; + + global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index); + } + + if (global_uniq_indexlist != NIL) + { + foreach(lc, childrels) + { + RelOptInfo *child = lfirst(lc); + if (is_first) + { + is_first = false; + continue; + } + adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist); + } + /* Now we have a list of unique index which are exactly same on all childrels, + * Set the UniqueKey just like it is non-partition table + */ + populate_baserel_uniquekeys(root, rel, global_uniq_indexlist); + } + } +} + + +/* + * populate_distinctrel_uniquekeys + */ +void +populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel) +{ + /* The unique key before the distinct is still valid. */ + distinctrel->uniquekeys = list_copy(inputrel->uniquekeys); + add_uniquekey_from_sortgroups(root, distinctrel, root->parse->distinctClause); +} + +/* + * populate_grouprel_uniquekeys + */ +void +populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel) + +{ + Query *parse = root->parse; + bool input_ukey_added = false; + ListCell *lc; + + if (relation_is_onerow(inputrel)) + { + add_uniquekey_for_onerow(grouprel); + return; + } + if (parse->groupingSets) + return; + + /* A Normal group by without grouping set. */ + if (parse->groupClause) + { + /* + * Current even the groupby clause is Unique already, but if query has aggref + * We have to create grouprel still. To keep the UnqiueKey short, we will check + * the UniqueKey of input_rel still valid, if so we reuse it. + */ + foreach(lc, inputrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (list_is_subset(ukey->exprs, grouprel->reltarget->exprs)) + { + grouprel->uniquekeys = lappend(grouprel->uniquekeys, + ukey); + input_ukey_added = true; + } + } + if (!input_ukey_added) + /* + * group by clause must be a super-set of grouprel->reltarget->exprs except the + * aggregation expr, so if such exprs is unique already, no bother to generate + * new uniquekey for group by exprs. + */ + add_uniquekey_from_sortgroups(root, + grouprel, + root->parse->groupClause); + } + else + /* It has aggregation but without a group by, so only one row returned */ + add_uniquekey_for_onerow(grouprel); +} + +/* + * simple_copy_uniquekeys + * Using a function for the one-line code makes us easy to check where we simply + * copied the uniquekey. + */ +void +simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel) +{ + newrel->uniquekeys = oldrel->uniquekeys; +} + +/* + * populate_unionrel_uniquekeys + */ +void +populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel) +{ + ListCell *lc; + List *exprs = NIL; + + Assert(unionrel->uniquekeys == NIL); + + foreach(lc, unionrel->reltarget->exprs) + { + exprs = lappend(exprs, lfirst(lc)); + } + + if (exprs == NIL) + /* SQL: select union select; is valid, we need to handle it here. */ + add_uniquekey_for_onerow(unionrel); + else + unionrel->uniquekeys = lappend(unionrel->uniquekeys, + makeUniqueKey(exprs,false)); + +} + +/* + * populate_joinrel_uniquekeys + * + * populate uniquekeys for joinrel. We will check each relation to see if its + * UniqueKey is still valid via innerrel_keeps_unique, if so, we add it to + * joinrel. The multi_nullvals field will be changed to true for some outer + * join cases and one-row UniqueKey needs to be converted to normal UniqueKey + * for the same case as well. + * For the uniquekey in either baserel which can't be unique after join, we still + * check to see if combination of UniqueKeys from both side is still useful for us. + * if yes, we add it to joinrel as well. + */ +void +populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *outerrel, RelOptInfo *innerrel, + List *restrictlist, JoinType jointype) +{ + ListCell *lc, *lc2; + List *clause_list = NIL; + List *outerrel_ukey_ctx; + List *innerrel_ukey_ctx; + bool inner_onerow, outer_onerow; + bool mergejoin_allowed; + + /* Care about the outerrel relation only for SEMI/ANTI join */ + if (jointype == JOIN_SEMI || jointype == JOIN_ANTI) + { + foreach(lc, outerrel->uniquekeys) + { + UniqueKey *uniquekey = lfirst_node(UniqueKey, lc); + if (list_is_subset(uniquekey->exprs, joinrel->reltarget->exprs)) + joinrel->uniquekeys = lappend(joinrel->uniquekeys, uniquekey); + } + return; + } + + Assert(jointype == JOIN_LEFT || jointype == JOIN_FULL || jointype == JOIN_INNER); + + /* Fast path */ + if (innerrel->uniquekeys == NIL || outerrel->uniquekeys == NIL) + return; + + inner_onerow = relation_is_onerow(innerrel); + outer_onerow = relation_is_onerow(outerrel); + + outerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(outerrel); + innerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(innerrel); + + clause_list = select_mergejoin_clauses(root, joinrel, outerrel, innerrel, + restrictlist, jointype, + &mergejoin_allowed); + + if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true /* reverse */)) + { + bool outer_impact = jointype == JOIN_FULL; + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + /* Outer relation has one row, and the unique key is not duplicated after join, + * the joinrel will still has one row unless the jointype == JOIN_FULL. + */ + if (outer_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (outer_onerow) + { + /* + * The onerow outerrel becomes multi rows and multi_nullvals + * will be changed to true. We also need to set the exprs correctly since it + * can't be NIL any more. + */ + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, outerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Change multi_nullvals to true due to the full join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, true)); + else + /* Just reuse it */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + } + ctx->added_to_joinrel = true; + } + } + + if (innerrel_keeps_unique(root, outerrel, innerrel, clause_list, false)) + { + bool outer_impact = jointype == JOIN_FULL || jointype == JOIN_LEFT;; + + foreach(lc, innerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + if (inner_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (inner_onerow) + { + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, innerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Need to change multi_nullvals to true due to the outer join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, + true)); + else + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + + } + ctx->added_to_joinrel = true; + } + } + + /* + * The combination of the UniqueKey from both sides is unique as well regardless + * of join type, but no bother to add it if its subset has been added to joinrel + * already or it is not useful for the joinrel. + */ + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx1 = (UniqueKeyContext) lfirst(lc); + if (ctx1->added_to_joinrel || !ctx1->useful) + continue; + foreach(lc2, innerrel_ukey_ctx) + { + UniqueKeyContext ctx2 = (UniqueKeyContext) lfirst(lc2); + if (ctx2->added_to_joinrel || !ctx2->useful) + continue; + if (add_combined_uniquekey(root, joinrel, outerrel, innerrel, + ctx1->uniquekey, ctx2->uniquekey, + jointype)) + /* If we set a onerow UniqueKey to joinrel, we don't need other. */ + return; + } + } +} + + +/* + * convert_subquery_uniquekeys + * + * Covert the UniqueKey in subquery to outer relation. + */ +void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel) +{ + ListCell *lc; + + if (sub_final_rel->uniquekeys == NIL) + return; + + if (relation_is_onerow(sub_final_rel)) + { + add_uniquekey_for_onerow(currel); + return; + } + + Assert(currel->subroot != NULL); + + foreach(lc, sub_final_rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + ListCell *lc; + List *exprs = NIL; + bool ukey_useful = true; + + /* One row case is handled above */ + Assert(ukey->exprs != NIL); + foreach(lc, ukey->exprs) + { + Var *var; + TargetEntry *tle = tlist_member(lfirst(lc), + currel->subroot->processed_tlist); + if (tle == NULL) + { + ukey_useful = false; + break; + } + var = find_var_for_subquery_tle(currel, tle); + if (var == NULL) + { + ukey_useful = false; + break; + } + exprs = lappend(exprs, var); + } + + if (ukey_useful) + currel->uniquekeys = lappend(currel->uniquekeys, + makeUniqueKey(exprs, + ukey->multi_nullvals)); + + } +} + +/* + * innerrel_keeps_unique + * + * Check if Unique key of the innerrel is valid after join. innerrel's UniqueKey + * will be still valid if innerrel's any-column mergeop outrerel's uniquekey + * exists in clause_list. + * + * Note: the clause_list must be a list of mergeable restrictinfo already. + */ +static bool +innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *clause_list, + bool reverse) +{ + ListCell *lc, *lc2, *lc3; + + if (outerrel->uniquekeys == NIL || innerrel->uniquekeys == NIL) + return false; + + /* Check if there is outerrel's uniquekey in mergeable clause. */ + foreach(lc, outerrel->uniquekeys) + { + List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs; + bool clauselist_matchs_all_exprs = true; + foreach(lc2, outer_uq_exprs) + { + Node *outer_uq_expr = lfirst(lc2); + bool find_uq_expr_in_clauselist = false; + foreach(lc3, clause_list) + { + RestrictInfo *rinfo = lfirst_node(RestrictInfo, lc3); + Node *outer_expr; + if (reverse) + outer_expr = rinfo->outer_is_left ? get_rightop(rinfo->clause) : get_leftop(rinfo->clause); + else + outer_expr = rinfo->outer_is_left ? get_leftop(rinfo->clause) : get_rightop(rinfo->clause); + if (equal(outer_expr, outer_uq_expr)) + { + find_uq_expr_in_clauselist = true; + break; + } + } + if (!find_uq_expr_in_clauselist) + { + /* No need to check the next exprs in the current uniquekey */ + clauselist_matchs_all_exprs = false; + break; + } + } + + if (clauselist_matchs_all_exprs) + return true; + } + return false; +} + + +/* + * relation_is_onerow + * Check if it is a one-row relation by checking UniqueKey. + */ +bool +relation_is_onerow(RelOptInfo *rel) +{ + UniqueKey *ukey; + if (rel->uniquekeys == NIL) + return false; + ukey = linitial_node(UniqueKey, rel->uniquekeys); + return ukey->exprs == NIL && list_length(rel->uniquekeys) == 1; +} + +/* + * relation_has_uniquekeys_for + * Returns true if we have proofs that 'rel' cannot return multiple rows with + * the same values in each of 'exprs'. Otherwise returns false. + */ +bool +relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel, + List *exprs, bool allow_multinulls) +{ + ListCell *lc; + + /* + * For UniqueKey->onerow case, the uniquekey->exprs is empty as well + * so we can't rely on list_is_subset to handle this special cases + */ + if (exprs == NIL) + return false; + + foreach(lc, rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (ukey->multi_nullvals && !allow_multinulls) + continue; + if (list_is_subset(ukey->exprs, exprs)) + return true; + } + return false; +} + + +/* + * get_exprs_from_uniqueindex + * + * Return a list of exprs which is unique. set useful to false if this + * unique index is not useful for us. + */ +static List * +get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals) +{ + List *exprs = NIL; + ListCell *indexpr_item; + int c = 0; + + *useful = true; + *multi_nullvals = false; + + indexpr_item = list_head(unique_index->indexprs); + for(c = 0; c < unique_index->ncolumns; c++) + { + int attr = unique_index->indexkeys[c]; + Expr *expr; + bool matched_const = false; + ListCell *lc1, *lc2; + + if(attr > 0) + { + expr = list_nth_node(TargetEntry, unique_index->indextlist, c)->expr; + } + else if (attr == 0) + { + /* Expression index */ + expr = lfirst(indexpr_item); + indexpr_item = lnext(unique_index->indexprs, indexpr_item); + } + else /* attr < 0 */ + { + /* Index on system column is not supported */ + Assert(false); + } + + /* + * Check index_col = Const case with regarding to opfamily checking + * If we can remove the index_col from the final UniqueKey->exprs. + */ + forboth(lc1, const_exprs, lc2, const_expr_opfamilies) + { + if (list_member_oid((List *)lfirst(lc2), unique_index->opfamily[c]) + && match_index_to_operand((Node *) lfirst(lc1), c, unique_index)) + { + matched_const = true; + break; + } + } + + if (matched_const) + continue; + + /* Check if the indexed expr is used in rel */ + if (attr > 0) + { + /* + * Normal Indexed column, if the col is not used, then the index is useless + * for uniquekey. + */ + attr -= FirstLowInvalidHeapAttributeNumber; + + if (!bms_is_member(attr, used_varattrs)) + { + *useful = false; + break; + } + } + else if (!list_member(unique_index->rel->reltarget->exprs, expr)) + { + /* Expression index but the expression is not used in rel */ + *useful = false; + break; + } + + /* check not null property. */ + if (attr == 0) + { + /* We never know if a expression yields null or not */ + *multi_nullvals = true; + } + else if (!bms_is_member(attr, unique_index->rel->notnullattrs) + && !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber, + unique_index->rel->notnullattrs)) + { + *multi_nullvals = true; + } + + exprs = lappend(exprs, expr); + } + return exprs; +} + + +/* + * add_uniquekey_for_onerow + * If we are sure that the relation only returns one row, then all the columns + * are unique. However we don't need to create UniqueKey for every column, we + * just set exprs = NIL and overwrites all the other UniqueKey on this RelOptInfo + * since this one has strongest semantics. + */ +void +add_uniquekey_for_onerow(RelOptInfo *rel) +{ + /* + * We overwrite the previous UniqueKey on purpose since this one has the + * strongest semantic. + */ + rel->uniquekeys = list_make1(makeUniqueKey(NIL, false)); +} + + +/* + * initililze_uniquecontext_for_joinrel + * Return a List of UniqueKeyContext for an inputrel + */ +static List * +initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel) +{ + List *res = NIL; + ListCell *lc; + foreach(lc, inputrel->uniquekeys) + { + UniqueKeyContext context; + context = palloc(sizeof(struct UniqueKeyContextData)); + context->uniquekey = lfirst_node(UniqueKey, lc); + context->added_to_joinrel = false; + context->useful = true; + res = lappend(res, context); + } + return res; +} + + +/* + * get_exprs_from_uniquekey + * Unify the way of get List of exprs from a one-row UniqueKey or + * normal UniqueKey. for the onerow case, every expr in rel1 is a valid + * UniqueKey. Return a List of exprs. + * + * rel1: The relation which you want to get the exprs. + * ukey: The UniqueKey you want to get the exprs. + */ +static List * +get_exprs_from_uniquekey(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *rel1, UniqueKey *ukey) +{ + ListCell *lc; + bool onerow = rel1 != NULL && relation_is_onerow(rel1); + + List *res = NIL; + Assert(onerow || ukey); + if (onerow) + { + /* Only cares about the exprs still exist in joinrel */ + foreach(lc, joinrel->reltarget->exprs) + { + Bitmapset *relids = pull_varnos(root, lfirst(lc)); + if (bms_is_subset(relids, rel1->relids)) + { + res = lappend(res, list_make1(lfirst(lc))); + } + } + } + else + { + res = list_make1(ukey->exprs); + } + return res; +} + +/* + * Partitioned table Unique Keys. + * The partition table unique key is maintained as: + * 1. The index must be unique as usual. + * 2. The index must contains partition key. + * 3. The index must exist on all the child rel. see simple_indexinfo_equal for + * how we compare it. + */ + +/* + * index_constains_partkey + * return true if the index contains the partiton key. + */ +static bool +index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind) +{ + ListCell *lc; + int i; + Assert(IS_PARTITIONED_REL(partrel)); + Assert(partrel->part_scheme->partnatts > 0); + + for(i = 0; i < partrel->part_scheme->partnatts; i++) + { + Node *part_expr = linitial(partrel->partexprs[i]); + bool found_in_index = false; + foreach(lc, ind->indextlist) + { + Expr *index_expr = lfirst_node(TargetEntry, lc)->expr; + if (equal(index_expr, part_expr)) + { + found_in_index = true; + break; + } + } + if (!found_in_index) + return false; + } + return true; +} + +/* + * simple_indexinfo_equal + * + * Used to check if the 2 index is same as each other. The index here + * is COPIED from childrel and did some tiny changes(see + * simple_copy_indexinfo_to_parent) + */ +static bool +simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2) +{ + Size oid_cmp_len = sizeof(Oid) * ind1->ncolumns; + + return ind1->ncolumns == ind2->ncolumns && + ind1->unique == ind2->unique && + memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0 && + memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0 && + memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0 && + memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0 && + equal(get_tlist_exprs(ind1->indextlist, true), + get_tlist_exprs(ind2->indextlist, true)); +} + + +/* + * The below macros are used for simple_copy_indexinfo_to_parent which is so + * customized that I don't want to put it to copyfuncs.c. So copy it here. + */ +#define COPY_POINTER_FIELD(fldname, sz) \ + do { \ + Size _size = (sz); \ + newnode->fldname = palloc(_size); \ + memcpy(newnode->fldname, from->fldname, _size); \ + } while (0) + +#define COPY_NODE_FIELD(fldname) \ + (newnode->fldname = copyObjectImpl(from->fldname)) + +#define COPY_SCALAR_FIELD(fldname) \ + (newnode->fldname = from->fldname) + + +/* + * simple_copy_indexinfo_to_parent (from partition) + * Copy the IndexInfo from child relation to parent relation with some modification, + * which is used to test: + * 1. If the same index exists in all the childrels. + * 2. If the parentrel->reltarget/basicrestrict info matches this index. + */ +static IndexOptInfo * +simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from) +{ + IndexOptInfo *newnode = makeNode(IndexOptInfo); + AppendRelInfo *appinfo = find_appinfo_by_child(root, from->rel->relid); + ListCell *lc; + int idx = 0; + + COPY_SCALAR_FIELD(ncolumns); + COPY_SCALAR_FIELD(nkeycolumns); + COPY_SCALAR_FIELD(unique); + COPY_SCALAR_FIELD(immediate); + /* We just need to know if it is NIL or not */ + COPY_SCALAR_FIELD(indpred); + COPY_SCALAR_FIELD(predOK); + COPY_POINTER_FIELD(indexkeys, from->ncolumns * sizeof(int)); + COPY_POINTER_FIELD(indexcollations, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opfamily, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opcintype, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(sortopfamily, from->ncolumns * sizeof(Oid)); + COPY_NODE_FIELD(indextlist); + + /* Convert index exprs on child expr to expr on parent */ + foreach(lc, newnode->indextlist) + { + TargetEntry *tle = lfirst_node(TargetEntry, lc); + /* Index on expression is ignored */ + Assert(IsA(tle->expr, Var)); + tle->expr = (Expr *) find_parent_var(appinfo, (Var *) tle->expr); + newnode->indexkeys[idx] = castNode(Var, tle->expr)->varattno; + idx++; + } + newnode->rel = parentrel; + return newnode; +} + +/* + * adjust_partition_unique_indexlist + * + * global_unique_indexes: At the beginning, it contains the copy & modified + * unique index from the first partition. And then check if each index in it still + * exists in the following partitions. If no, remove it. at last, it has an + * index list which exists in all the partitions. + */ +static void +adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_indexes) +{ + ListCell *lc, *lc2; + foreach(lc, *global_unique_indexes) + { + IndexOptInfo *g_ind = lfirst_node(IndexOptInfo, lc); + bool found_in_child = false; + + foreach(lc2, childrel->indexlist) + { + IndexOptInfo *p_ind = lfirst_node(IndexOptInfo, lc2); + IndexOptInfo *p_ind_copy; + if (!p_ind->unique || !p_ind->immediate || + (p_ind->indpred != NIL && !p_ind->predOK)) + continue; + p_ind_copy = simple_copy_indexinfo_to_parent(root, parentrel, p_ind); + if (simple_indexinfo_equal(p_ind_copy, g_ind)) + { + found_in_child = true; + break; + } + } + if (!found_in_child) + /* The index doesn't exist in childrel, remove it from global_unique_indexes */ + *global_unique_indexes = foreach_delete_current(*global_unique_indexes, lc); + } +} + +/* Helper function for groupres/distinctrel */ +static void +add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgroups) +{ + Query *parse = root->parse; + List *exprs; + + /* + * XXX: If there are some vars which is not in current levelsup, the semantic is + * imprecise, should we avoid it or not? levelsup = 1 is just a demo, maybe we need to + * check every level other than 0, if so, looks we have to write another + * pull_var_walker. + */ + List *upper_vars = pull_vars_of_level((Node*)sortgroups, 1); + + if (upper_vars != NIL) + return; + + exprs = get_sortgrouplist_exprs(sortgroups, parse->targetList); + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(exprs, + false /* sortgroupclause can't be multi_nullvals */)); +} + + +/* + * add_combined_uniquekey + * The combination of both UniqueKeys is a valid UniqueKey for joinrel no matter + * the jointype. + */ +bool +add_combined_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype) +{ + + ListCell *lc1, *lc2; + + /* Either side has multi_nullvals or we have outer join, + * the combined UniqueKey has multi_nullvals */ + bool multi_nullvals = outer_ukey->multi_nullvals || + inner_ukey->multi_nullvals || IS_OUTER_JOIN(jointype); + + /* The only case we can get onerow joinrel after join */ + if (relation_is_onerow(outer_rel) + && relation_is_onerow(inner_rel) + && jointype == JOIN_INNER) + { + add_uniquekey_for_onerow(joinrel); + return true; + } + + foreach(lc1, get_exprs_from_uniquekey(root, joinrel, outer_rel, outer_ukey)) + { + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, inner_rel, inner_ukey)) + { + List *exprs = list_concat_copy(lfirst_node(List, lc1), lfirst_node(List, lc2)); + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(exprs, + multi_nullvals)); + } + } + return false; +} diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index 424d25cbd5..8d8e493f5c 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -2387,6 +2387,8 @@ grouping_planner(PlannerInfo *root, bool inheritance_update, add_path(final_rel, path); } + simple_copy_uniquekeys(current_rel, final_rel); + /* * Generate partial paths for final_rel, too, if outer query levels might * be able to make use of them. @@ -3899,6 +3901,8 @@ create_grouping_paths(PlannerInfo *root, } set_cheapest(grouped_rel); + + populate_grouprel_uniquekeys(root, grouped_rel, input_rel); return grouped_rel; } @@ -4621,7 +4625,7 @@ create_window_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(window_rel); - + simple_copy_uniquekeys(input_rel, window_rel); return window_rel; } @@ -4935,7 +4939,7 @@ create_distinct_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(distinct_rel); - + populate_distinctrel_uniquekeys(root, input_rel, distinct_rel); return distinct_rel; } @@ -5196,6 +5200,8 @@ create_ordered_paths(PlannerInfo *root, */ Assert(ordered_rel->pathlist != NIL); + simple_copy_uniquekeys(input_rel, ordered_rel); + return ordered_rel; } @@ -6073,6 +6079,9 @@ adjust_paths_for_srfs(PlannerInfo *root, RelOptInfo *rel, if (list_length(targets) == 1) return; + /* UniqueKey is not valid after handling the SRF. */ + rel->uniquekeys = NIL; + /* * Stack SRF-evaluation nodes atop each path for the rel. * diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c index becdcbb872..b7626545bf 100644 --- a/src/backend/optimizer/prep/prepunion.c +++ b/src/backend/optimizer/prep/prepunion.c @@ -689,6 +689,8 @@ generate_union_paths(SetOperationStmt *op, PlannerInfo *root, /* Undo effects of possibly forcing tuple_fraction to 0 */ root->tuple_fraction = save_fraction; + /* Add the UniqueKeys */ + populate_unionrel_uniquekeys(root, result_rel); return result_rel; } diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c index 86922a273c..6817c9c787 100644 --- a/src/backend/optimizer/util/appendinfo.c +++ b/src/backend/optimizer/util/appendinfo.c @@ -746,3 +746,47 @@ find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos) } return appinfos; } + +/* + * find_appinfo_by_child + * + */ +AppendRelInfo * +find_appinfo_by_child(PlannerInfo *root, Index child_index) +{ + ListCell *lc; + foreach(lc, root->append_rel_list) + { + AppendRelInfo *appinfo = lfirst_node(AppendRelInfo, lc); + if (appinfo->child_relid == child_index) + return appinfo; + } + elog(ERROR, "parent relation cant be found"); + return NULL; +} + +/* + * find_parent_var + * + */ +Var * +find_parent_var(AppendRelInfo *appinfo, Var *child_var) +{ + ListCell *lc; + Var *res = NULL; + Index attno = 1; + foreach(lc, appinfo->translated_vars) + { + Node *child_node = lfirst(lc); + if (equal(child_node, child_var)) + { + res = copyObject(child_var); + res->varattno = attno; + res->varno = appinfo->parent_relid; + } + attno++; + } + if (res == NULL) + elog(ERROR, "parent var can't be found."); + return res; +} diff --git a/src/backend/optimizer/util/inherit.c b/src/backend/optimizer/util/inherit.c index be1c9ddd96..3eec1f4d74 100644 --- a/src/backend/optimizer/util/inherit.c +++ b/src/backend/optimizer/util/inherit.c @@ -736,6 +736,7 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, { Node *onecq = (Node *) lfirst(lc2); bool pseudoconstant; + RestrictInfo *child_rinfo; /* check for pseudoconstant (no Vars or volatile functions) */ pseudoconstant = @@ -747,14 +748,15 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, root->hasPseudoConstantQuals = true; } /* reconstitute RestrictInfo with appropriate properties */ - childquals = lappend(childquals, - make_restrictinfo(root, - (Expr *) onecq, - rinfo->is_pushed_down, - rinfo->outerjoin_delayed, - pseudoconstant, - rinfo->security_level, - NULL, NULL, NULL)); + child_rinfo = make_restrictinfo(root, + (Expr *) onecq, + rinfo->is_pushed_down, + rinfo->outerjoin_delayed, + pseudoconstant, + rinfo->security_level, + NULL, NULL, NULL); + child_rinfo->mergeopfamilies = rinfo->mergeopfamilies; + childquals = lappend(childquals, child_rinfo); /* track minimum security level among child quals */ cq_min_security = Min(cq_min_security, rinfo->security_level); } diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 48a7ebfe45..4fe1824eb0 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -16,6 +16,7 @@ #include "nodes/execnodes.h" #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" extern A_Expr *makeA_Expr(A_Expr_Kind kind, List *name, @@ -106,4 +107,6 @@ extern GroupingSet *makeGroupingSet(GroupingSetKind kind, List *content, int loc extern VacuumRelation *makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols); +extern UniqueKey* makeUniqueKey(List *exprs, bool multi_nullvals); + #endif /* MAKEFUNC_H */ diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index e22df890ef..3320273ac1 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -265,6 +265,7 @@ typedef enum NodeTag T_EquivalenceMember, T_PathKey, T_PathTarget, + T_UniqueKey, T_RestrictInfo, T_IndexClause, T_PlaceHolderVar, diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index 0d61f04d27..80561b1003 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -739,6 +739,7 @@ typedef struct RelOptInfo QualCost baserestrictcost; /* cost of evaluating the above */ Index baserestrict_min_security; /* min security_level found in * baserestrictinfo */ + List *uniquekeys; /* List of UniqueKey */ List *joininfo; /* RestrictInfo structures for join clauses * involving this rel */ bool has_eclass_joins; /* T means joininfo is incomplete */ @@ -1059,6 +1060,28 @@ typedef struct PathKey } PathKey; +/* + * UniqueKey + * + * Represents the unique properties held by a RelOptInfo. + * + * exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL + * is a special case of UniqueKey, which means there is only 1 row in that + * relation. + * multi_nullvals: true means multi null values may exist in these exprs, so the + * uniqueness is not guaranteed in this case. This field is necessary for + * remove_useless_join & reduce_unique_semijoins where we don't mind these + * duplicated NULL values. It is set to true for 2 cases. One is a unique key + * from a unique index but the related column is nullable. The other one is for + * outer join. see populate_joinrel_uniquekeys for detail. + */ +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + /* * PathTarget * @@ -2496,7 +2519,7 @@ typedef enum * * flags indicating what kinds of grouping are possible. * partial_costs_set is true if the agg_partial_costs and agg_final_costs - * have been initialized. + * have been initialized. * agg_partial_costs gives partial aggregation costs. * agg_final_costs gives finalization costs. * target_parallel_safe is true if target is parallel safe. @@ -2526,8 +2549,8 @@ typedef struct * limit_tuples is an estimated bound on the number of output tuples, * or -1 if no LIMIT or couldn't estimate. * count_est and offset_est are the estimated values of the LIMIT and OFFSET - * expressions computed by preprocess_limit() (see comments for - * preprocess_limit() for more information). + * expressions computed by preprocess_limit() (see comments for + * preprocess_limit() for more information). */ typedef struct { diff --git a/src/include/nodes/pg_list.h b/src/include/nodes/pg_list.h index 404e03f132..85c65e910c 100644 --- a/src/include/nodes/pg_list.h +++ b/src/include/nodes/pg_list.h @@ -566,6 +566,16 @@ extern pg_nodiscard List *list_delete_first(List *list); extern pg_nodiscard List *list_delete_last(List *list); extern pg_nodiscard List *list_delete_nth_cell(List *list, int n); extern pg_nodiscard List *list_delete_cell(List *list, ListCell *cell); +extern bool list_is_subset(const List *members, const List *target); + +extern List *list_delete(List *list, void *datum); +extern List *list_delete_ptr(List *list, void *datum); +extern List *list_delete_int(List *list, int datum); +extern List *list_delete_oid(List *list, Oid datum); +extern List *list_delete_first(List *list); +extern List *list_delete_last(List *list); +extern List *list_delete_nth_cell(List *list, int n); +extern List *list_delete_cell(List *list, ListCell *cell); extern List *list_union(const List *list1, const List *list2); extern List *list_union_ptr(const List *list1, const List *list2); diff --git a/src/include/optimizer/appendinfo.h b/src/include/optimizer/appendinfo.h index 4cbf8c26cc..43b8d05f69 100644 --- a/src/include/optimizer/appendinfo.h +++ b/src/include/optimizer/appendinfo.h @@ -32,4 +32,7 @@ extern Relids adjust_child_relids_multilevel(PlannerInfo *root, Relids relids, extern AppendRelInfo **find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos); +extern AppendRelInfo *find_appinfo_by_child(PlannerInfo *root, Index child_index); +extern Var *find_parent_var(AppendRelInfo *appinfo, Var *child_var); + #endif /* APPENDINFO_H */ diff --git a/src/include/optimizer/optimizer.h b/src/include/optimizer/optimizer.h index d587952b7d..843aafa51e 100644 --- a/src/include/optimizer/optimizer.h +++ b/src/include/optimizer/optimizer.h @@ -23,6 +23,7 @@ #define OPTIMIZER_H #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" /* Test if an expression node represents a SRF call. Beware multiple eval! */ #define IS_SRF_CALL(node) \ @@ -168,6 +169,7 @@ extern TargetEntry *get_sortgroupref_tle(Index sortref, List *targetList); extern TargetEntry *get_sortgroupclause_tle(SortGroupClause *sgClause, List *targetList); +extern Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); extern Node *get_sortgroupclause_expr(SortGroupClause *sgClause, List *targetList); extern List *get_sortgrouplist_exprs(List *sgClauses, diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h index 035d3e1206..1adf99c9ee 100644 --- a/src/include/optimizer/paths.h +++ b/src/include/optimizer/paths.h @@ -247,5 +247,48 @@ extern PathKey *make_canonical_pathkey(PlannerInfo *root, int strategy, bool nulls_first); extern void add_paths_to_append_rel(PlannerInfo *root, RelOptInfo *rel, List *live_childrels); +extern List *select_mergejoin_clauses(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + JoinType jointype, + bool *mergejoin_allowed); + +/* + * uniquekeys.c + * Utilities for matching and building unique keys + */ +extern void populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List* unique_index_list); +extern void populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels); +extern void populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel); +extern void populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel); +extern void populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel); +extern void simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel); +extern void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel); +extern void populate_joinrel_uniquekeys(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + RelOptInfo *rel2, + List *restrictlist, + JoinType jointype); + +extern bool relation_has_uniquekeys_for(PlannerInfo *root, + RelOptInfo *rel, + List *exprs, + bool allow_multinulls); +extern bool relation_is_onerow(RelOptInfo *rel); #endif /* PATHS_H */ -- 2.26.2 --yh7cw345mbttadjz Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v38-0003-Extend-UniqueKeys.patch" ^ permalink raw reply [nested|flat] 91+ messages in thread
* [PATCH v36 2/6] Introduce UniqueKey attributes on RelOptInfo struct. @ 2020-05-11 07:50 一挃 <[email protected]> 0 siblings, 0 replies; 91+ messages in thread From: 一挃 @ 2020-05-11 07:50 UTC (permalink / raw) UniqueKey is a set of exprs on RelOptInfo which represents the exprs will be unique on the given RelOptInfo. You can see README.uniquekey for more information. --- src/backend/nodes/copyfuncs.c | 13 + src/backend/nodes/list.c | 31 + src/backend/nodes/makefuncs.c | 13 + src/backend/nodes/outfuncs.c | 11 + src/backend/nodes/readfuncs.c | 10 + src/backend/optimizer/path/Makefile | 3 +- src/backend/optimizer/path/README.uniquekey | 131 +++ src/backend/optimizer/path/allpaths.c | 10 + src/backend/optimizer/path/joinpath.c | 9 +- src/backend/optimizer/path/joinrels.c | 2 + src/backend/optimizer/path/pathkeys.c | 3 +- src/backend/optimizer/path/uniquekeys.c | 1131 +++++++++++++++++++ src/backend/optimizer/plan/planner.c | 13 +- src/backend/optimizer/prep/prepunion.c | 2 + src/backend/optimizer/util/appendinfo.c | 44 + src/backend/optimizer/util/inherit.c | 16 +- src/include/nodes/makefuncs.h | 3 + src/include/nodes/nodes.h | 1 + src/include/nodes/pathnodes.h | 29 +- src/include/nodes/pg_list.h | 2 + src/include/optimizer/appendinfo.h | 3 + src/include/optimizer/optimizer.h | 2 + src/include/optimizer/paths.h | 43 + 23 files changed, 1502 insertions(+), 23 deletions(-) create mode 100644 src/backend/optimizer/path/README.uniquekey create mode 100644 src/backend/optimizer/path/uniquekeys.c diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 89c409de66..1f50400fd2 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2273,6 +2273,16 @@ _copyPathKey(const PathKey *from) return newnode; } +static UniqueKey * +_copyUniqueKey(const UniqueKey *from) +{ + UniqueKey *newnode = makeNode(UniqueKey); + + COPY_NODE_FIELD(exprs); + COPY_SCALAR_FIELD(multi_nullvals); + + return newnode; +} /* * _copyRestrictInfo */ @@ -5152,6 +5162,9 @@ copyObjectImpl(const void *from) case T_PathKey: retval = _copyPathKey(from); break; + case T_UniqueKey: + retval = _copyUniqueKey(from); + break; case T_RestrictInfo: retval = _copyRestrictInfo(from); break; diff --git a/src/backend/nodes/list.c b/src/backend/nodes/list.c index 80fa8c84e4..a7a99b70f2 100644 --- a/src/backend/nodes/list.c +++ b/src/backend/nodes/list.c @@ -687,6 +687,37 @@ list_member_oid(const List *list, Oid datum) return false; } +/* + * return true iff every entry in "members" list is also present + * in the "target" list. + */ +bool +list_is_subset(const List *members, const List *target) +{ + const ListCell *lc1, *lc2; + + Assert(IsPointerList(members)); + Assert(IsPointerList(target)); + check_list_invariants(members); + check_list_invariants(target); + + foreach(lc1, members) + { + bool found = false; + foreach(lc2, target) + { + if (equal(lfirst(lc1), lfirst(lc2))) + { + found = true; + break; + } + } + if (!found) + return false; + } + return true; +} + /* * Delete the n'th cell (counting from 0) in list. * diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 49de285f01..646cf7c9a1 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -814,3 +814,16 @@ makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols) v->va_cols = va_cols; return v; } + + +/* + * makeUniqueKey + */ +UniqueKey* +makeUniqueKey(List *exprs, bool multi_nullvals) +{ + UniqueKey * ukey = makeNode(UniqueKey); + ukey->exprs = exprs; + ukey->multi_nullvals = multi_nullvals; + return ukey; +} diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index e2f177515d..c3a9632992 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2428,6 +2428,14 @@ _outPathKey(StringInfo str, const PathKey *node) WRITE_BOOL_FIELD(pk_nulls_first); } +static void +_outUniqueKey(StringInfo str, const UniqueKey *node) +{ + WRITE_NODE_TYPE("UNIQUEKEY"); + WRITE_NODE_FIELD(exprs); + WRITE_BOOL_FIELD(multi_nullvals); +} + static void _outPathTarget(StringInfo str, const PathTarget *node) { @@ -4127,6 +4135,9 @@ outNode(StringInfo str, const void *obj) case T_PathKey: _outPathKey(str, obj); break; + case T_UniqueKey: + _outUniqueKey(str, obj); + break; case T_PathTarget: _outPathTarget(str, obj); break; diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index 42050ab719..3a18571d0c 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -452,6 +452,14 @@ _readSetOperationStmt(void) READ_DONE(); } +static UniqueKey * +_readUniqueKey(void) +{ + READ_LOCALS(UniqueKey); + READ_NODE_FIELD(exprs); + READ_BOOL_FIELD(multi_nullvals); + READ_DONE(); +} /* * Stuff from primnodes.h. @@ -2656,6 +2664,8 @@ parseNodeString(void) return_value = _readCommonTableExpr(); else if (MATCH("SETOPERATIONSTMT", 16)) return_value = _readSetOperationStmt(); + else if (MATCH("UNIQUEKEY", 9)) + return_value = _readUniqueKey(); else if (MATCH("ALIAS", 5)) return_value = _readAlias(); else if (MATCH("RANGEVAR", 8)) diff --git a/src/backend/optimizer/path/Makefile b/src/backend/optimizer/path/Makefile index 1e199ff66f..7b9820c25f 100644 --- a/src/backend/optimizer/path/Makefile +++ b/src/backend/optimizer/path/Makefile @@ -21,6 +21,7 @@ OBJS = \ joinpath.o \ joinrels.o \ pathkeys.o \ - tidpath.o + tidpath.o \ + uniquekeys.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/optimizer/path/README.uniquekey b/src/backend/optimizer/path/README.uniquekey new file mode 100644 index 0000000000..5eac761995 --- /dev/null +++ b/src/backend/optimizer/path/README.uniquekey @@ -0,0 +1,131 @@ +1. What is UniqueKey? +We can think UniqueKey is a set of exprs for a RelOptInfo, which we are insure +that doesn't yields same result among all the rows. The simplest UniqueKey +format is primary key. + +However we define the UnqiueKey as below. + +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + +exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL +is a special case of UniqueKey, which means there is only one row in that +relation.it has a stronger semantic than others. like SELECT uk FROM t; uk is +normal unique key and may have different values. SELECT colx FROM t WHERE uk = +const. colx is unique AND we have only 1 value. This field can used for +innerrel_is_unique. this logic is handled specially in add_uniquekey_for_onerow +function. + +multi_nullvals: true means multi null values may exist in these exprs, so the +uniqueness is not guaranteed in this case. This field is necessary for +remove_useless_join & reduce_unique_semijoins where we don't mind these +duplicated NULL values. It is set to true for 2 cases. One is a unique key +from a unique index but the related column is nullable. The other one is for +outer join. see populate_joinrel_uniquekeys for detail. + + +The UniqueKey can be used at the following cases at least: +1. remove_useless_joins. +2. reduce_semianti_joins +3. remove distinct node if distinct clause is unique. +4. remove aggnode if group by clause is unique. +5. Index Skip Scan (WIP) +6. Aggregation Push Down without 2 phase aggregation if the join can't + duplicated the aggregated rows. (work in progress feature) + +2. How is it maintained? + +We have a set of populate_xxx_unqiuekeys functions to maintain the uniquekey on +various cases. xxx includes baserel, joinrel, partitionedrel, distinctrel, +groupedrel, unionrel. and we also need to convert the uniquekey from subquery +to outer relation, which is what convert_subquery_uniquekeys does. + +1. The first part is about baserel. We handled 3 cases. suppose we have Unique +Index on (a, b). + +1. SELECT a, b FROM t. UniqueKey (a, b) +2. SELECT a FROM t WHERE b = 1; UniqueKey (a) +3. SELECT .. FROM t WHERE a = 1 AND b = 1; UniqueKey (NIL). onerow case, every + column is Unique. + +2. The next part is joinrel, this part is most error-prone, we simplified the rules +like below: +1. If the relation's UniqueKey can't be duplicated after join, then is will be + still valid for the join rel. The function we used here is + innerrel_keeps_unique. The basic idea is innerrel.any_col = outer.uk. + +2. If the UnqiueKey can't keep valid via the rule 1, the combination of the + UniqueKey from both sides are valid for sure. We can prove this as: if the + unique exprs from rel1 is duplicated by rel2, the duplicated rows must + contains different unique exprs from rel2. + +More considerations about onerow: +1. If relation with one row and it can't be duplicated, it is still possible + contains mulit_nullvas after outer join. +2. If the either UniqueKey can be duplicated after join, the can get one row + only when both side is one row AND there is no outer join. +3. Whenever the onerow UniqueKey is not a valid any more, we need to convert one + row UniqueKey to normal unique key since we don't store exprs for one-row + relation. get_exprs_from_uniquekeys will be used here. + + +More considerations about multi_nullvals after join: +1. If the original UnqiueKey has multi_nullvals, the final UniqueKey will have + mulit_nullvals in any case. +2. If a unique key doesn't allow mulit_nullvals, after some outer join, it + allows some outer join. + + +3. When we comes to subquery, we need to convert_subquery_unqiuekeys just like +convert_subquery_pathkeys. Only the UniqueKey insides subquery is referenced as +a Var in outer relation will be reused. The relationship between the outerrel.Var +and subquery.exprs is built with outerel->subroot->processed_tlist. + + +4. As for the SRF functions, it will break the uniqueness of uniquekey, However it +is handled in adjust_paths_for_srfs, which happens after the query_planner. so +we will maintain the UniqueKey until there and reset it to NIL at that +places. This can't help on distinct/group by elimination cases but probably help +in some other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is +semantic correctly. + + +5. As for inherit table, we first main the UnqiueKey on childrel as well. But for +partitioned table we need to maintain 2 different kinds of +UnqiueKey. 1). UniqueKey on the parent relation 2). UniqueKey on child +relation for partition wise query. + +Example: +CREATE TABLE p (a int not null, b int not null) partition by list (a); +CREATE TABLE p0 partition of p for values in (1); +CREATE TABLE p1 partition of p for values in (2); + +create unique index p0_b on p0(b); +create unique index p1_b on p1(b); + +Now b is only unique on partition level, so the distinct can't be removed on +the following cases. SELECT DISTINCT b FROM p; + +Another example is SELECT DISTINCT a, b FROM p WHERE a = 1; Since only one +partition is chosen, the UniqueKey on child relation is same as the UniqueKey on +parent relation. + +Another usage of UniqueKey on partition level is it be helpful for +partition-wise join. + +As for the UniqueKey on parent table level, it comes with 2 different ways, +1). the UniqueKey is also derived in UniqueKey index, but the index must be same +in all the related children relations and the unique index must contains +Partition Key in it. Example: + +CREATE UNIQUE INDEX p_ab ON p(a, b); -- where a is the partition key. + +-- Query +SELECT a, b FROM p; the (a, b) is a UniqueKey of p. + +2). If the parent relation has only one childrel, the UniqueKey on childrel is + the UniqueKey on parent as well. diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index 484dab0a1a..2ad9d06d7a 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -579,6 +579,12 @@ set_plain_rel_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) */ check_index_predicates(root, rel); + /* + * Now that we've marked which partial indexes are suitable, we can now + * build the relation's unique keys. + */ + populate_baserel_uniquekeys(root, rel, rel->indexlist); + /* Mark rel with estimated output rows, width, etc */ set_baserel_size_estimates(root, rel); } @@ -1310,6 +1316,8 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, /* Add paths to the append relation. */ add_paths_to_append_rel(root, rel, live_childrels); + if (IS_PARTITIONED_REL(rel)) + populate_partitionedrel_uniquekeys(root, rel, live_childrels); } @@ -2383,6 +2391,8 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, pathkeys, required_outer)); } + convert_subquery_uniquekeys(root, rel, sub_final_rel); + /* If outer rel allows parallelism, do same for partial paths. */ if (rel->consider_parallel && bms_is_empty(required_outer)) { diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c index db54a6ba2e..ef0fd2fb0b 100644 --- a/src/backend/optimizer/path/joinpath.c +++ b/src/backend/optimizer/path/joinpath.c @@ -71,13 +71,6 @@ static void consider_parallel_mergejoin(PlannerInfo *root, static void hash_inner_and_outer(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, RelOptInfo *innerrel, JoinType jointype, JoinPathExtraData *extra); -static List *select_mergejoin_clauses(PlannerInfo *root, - RelOptInfo *joinrel, - RelOptInfo *outerrel, - RelOptInfo *innerrel, - List *restrictlist, - JoinType jointype, - bool *mergejoin_allowed); static void generate_mergejoin_paths(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *innerrel, @@ -1927,7 +1920,7 @@ hash_inner_and_outer(PlannerInfo *root, * if it is mergejoinable and involves vars from the two sub-relations * currently of interest. */ -static List * +List * select_mergejoin_clauses(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c index 2d343cd293..b9163ee8ff 100644 --- a/src/backend/optimizer/path/joinrels.c +++ b/src/backend/optimizer/path/joinrels.c @@ -924,6 +924,8 @@ populate_joinrel_with_paths(PlannerInfo *root, RelOptInfo *rel1, /* Apply partitionwise join technique, if possible. */ try_partitionwise_join(root, rel1, rel2, joinrel, sjinfo, restrictlist); + + populate_joinrel_uniquekeys(root, joinrel, rel1, rel2, restrictlist, sjinfo->jointype); } diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index ce9bf87e9b..7e596d4194 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -33,7 +33,6 @@ static bool pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys); static bool matches_boolean_partition_clause(RestrictInfo *rinfo, RelOptInfo *partrel, int partkeycol); -static Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); static bool right_merge_direction(PlannerInfo *root, PathKey *pathkey); @@ -1035,7 +1034,7 @@ convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel, * We need this to ensure that we don't return pathkeys describing values * that are unavailable above the level of the subquery scan. */ -static Var * +Var * find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle) { ListCell *lc; diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c new file mode 100644 index 0000000000..b33bcd2f32 --- /dev/null +++ b/src/backend/optimizer/path/uniquekeys.c @@ -0,0 +1,1131 @@ +/*------------------------------------------------------------------------- + * + * uniquekeys.c + * Utilities for matching and building unique keys + * + * Portions Copyright (c) 2020, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/optimizer/path/uniquekeys.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "nodes/makefuncs.h" +#include "nodes/nodeFuncs.h" +#include "optimizer/pathnode.h" +#include "optimizer/paths.h" +#include "optimizer/appendinfo.h" +#include "optimizer/optimizer.h" +#include "optimizer/tlist.h" +#include "rewrite/rewriteManip.h" + + +/* + * This struct is used to help populate_joinrel_uniquekeys. + * + * added_to_joinrel is true if a uniquekey (from outerrel or innerrel) + * has been added to joinrel. + * useful is true if the exprs of the uniquekey still appears in joinrel. + */ +typedef struct UniqueKeyContextData +{ + UniqueKey *uniquekey; + bool added_to_joinrel; + bool useful; +} *UniqueKeyContext; + +static List *initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel); +static bool innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + bool reverse); + +static List *get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals); +static List *get_exprs_from_uniquekey(RelOptInfo *joinrel, + RelOptInfo *rel1, + UniqueKey *ukey); +static void add_uniquekey_for_onerow(RelOptInfo *rel); +static bool add_combined_uniquekey(RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype); + +/* Used for unique indexes checking for partitioned table */ +static bool index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind); +static IndexOptInfo *simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from); +static bool simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2); +static void adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_index); + +/* Helper function for grouped relation and distinct relation. */ +static void add_uniquekey_from_sortgroups(PlannerInfo *root, + RelOptInfo *rel, + List *sortgroups); + +/* + * populate_baserel_uniquekeys + * Populate 'baserel' uniquekeys list by looking at the rel's unique index + * and baserestrictinfo + */ +void +populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List *indexlist) +{ + ListCell *lc; + List *matched_uniq_indexes = NIL; + + /* Attrs appears in rel->reltarget->exprs. */ + Bitmapset *used_attrs = NULL; + + List *const_exprs = NIL; + List *expr_opfamilies = NIL; + + Assert(baserel->rtekind == RTE_RELATION); + + foreach(lc, indexlist) + { + IndexOptInfo *ind = (IndexOptInfo *) lfirst(lc); + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + matched_uniq_indexes = lappend(matched_uniq_indexes, ind); + } + + if (matched_uniq_indexes == NIL) + return; + + /* Check which attrs is used in baserel->reltarget */ + pull_varattnos((Node *)baserel->reltarget->exprs, baserel->relid, &used_attrs); + + /* Check which attrno is used at a mergeable const filter */ + foreach(lc, baserel->baserestrictinfo) + { + RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); + + if (rinfo->mergeopfamilies == NIL) + continue; + + if (bms_is_empty(rinfo->left_relids)) + { + const_exprs = lappend(const_exprs, get_rightop(rinfo->clause)); + } + else if (bms_is_empty(rinfo->right_relids)) + { + const_exprs = lappend(const_exprs, get_leftop(rinfo->clause)); + } + else + continue; + + expr_opfamilies = lappend(expr_opfamilies, rinfo->mergeopfamilies); + } + + foreach(lc, matched_uniq_indexes) + { + bool multi_nullvals, useful; + List *exprs = get_exprs_from_uniqueindex(lfirst_node(IndexOptInfo, lc), + const_exprs, + expr_opfamilies, + used_attrs, + &useful, + &multi_nullvals); + if (useful) + { + if (exprs == NIL) + { + /* All the columns in Unique Index matched with a restrictinfo */ + add_uniquekey_for_onerow(baserel); + return; + } + baserel->uniquekeys = lappend(baserel->uniquekeys, + makeUniqueKey(exprs, multi_nullvals)); + } + } +} + + +/* + * populate_partitionedrel_uniquekeys + * The UniqueKey on partitionrel comes from 2 cases: + * 1). Only one partition is involved in this query, the unique key can be + * copied to parent rel from childrel. + * 2). There are some unique index which includes partition key and exists + * in all the related partitions. + * We never mind rule 2 if we hit rule 1. + */ + +void +populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels) +{ + ListCell *lc; + List *global_uniq_indexlist = NIL; + RelOptInfo *childrel; + bool is_first = true; + + Assert(IS_PARTITIONED_REL(rel)); + + if (childrels == NIL) + return; + + /* + * If there is only one partition used in this query, the UniqueKey in childrel is + * still valid in parent level, but we need convert the format from child expr to + * parent expr. + */ + if (list_length(childrels) == 1) + { + /* Check for Rule 1 */ + RelOptInfo *childrel = linitial_node(RelOptInfo, childrels); + ListCell *lc; + Assert(childrel->reloptkind == RELOPT_OTHER_MEMBER_REL); + if (relation_is_onerow(childrel)) + { + add_uniquekey_for_onerow(rel); + return; + } + + foreach(lc, childrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + AppendRelInfo *appinfo = find_appinfo_by_child(root, childrel->relid); + List *parent_exprs = NIL; + bool can_reuse = true; + ListCell *lc2; + foreach(lc2, ukey->exprs) + { + Var *var = (Var *)lfirst(lc2); + /* + * If the expr comes from a expression, it is hard to build the expression + * in parent so ignore that case for now. + */ + if(!IsA(var, Var)) + { + can_reuse = false; + break; + } + /* Convert it to parent var */ + parent_exprs = lappend(parent_exprs, find_parent_var(appinfo, var)); + } + if (can_reuse) + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(parent_exprs, + ukey->multi_nullvals)); + } + } + else + { + /* Check for rule 2 */ + childrel = linitial_node(RelOptInfo, childrels); + foreach(lc, childrel->indexlist) + { + IndexOptInfo *ind = lfirst(lc); + IndexOptInfo *modified_index; + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + + /* + * During simple_copy_indexinfo_to_parent, we need to convert var from + * child var to parent var, index on expression is too complex to handle. + * so ignore it for now. + */ + if (ind->indexprs != NIL) + continue; + + modified_index = simple_copy_indexinfo_to_parent(root, rel, ind); + /* + * If the unique index doesn't contain partkey, then it is unique + * on this partition only, so it is useless for us. + */ + if (!index_constains_partkey(rel, modified_index)) + continue; + + global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index); + } + + if (global_uniq_indexlist != NIL) + { + foreach(lc, childrels) + { + RelOptInfo *child = lfirst(lc); + if (is_first) + { + is_first = false; + continue; + } + adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist); + } + /* Now we have a list of unique index which are exactly same on all childrels, + * Set the UniqueKey just like it is non-partition table + */ + populate_baserel_uniquekeys(root, rel, global_uniq_indexlist); + } + } +} + + +/* + * populate_distinctrel_uniquekeys + */ +void +populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel) +{ + /* The unique key before the distinct is still valid. */ + distinctrel->uniquekeys = list_copy(inputrel->uniquekeys); + add_uniquekey_from_sortgroups(root, distinctrel, root->parse->distinctClause); +} + +/* + * populate_grouprel_uniquekeys + */ +void +populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel) + +{ + Query *parse = root->parse; + bool input_ukey_added = false; + ListCell *lc; + + if (relation_is_onerow(inputrel)) + { + add_uniquekey_for_onerow(grouprel); + return; + } + if (parse->groupingSets) + return; + + /* A Normal group by without grouping set. */ + if (parse->groupClause) + { + /* + * Current even the groupby clause is Unique already, but if query has aggref + * We have to create grouprel still. To keep the UnqiueKey short, we will check + * the UniqueKey of input_rel still valid, if so we reuse it. + */ + foreach(lc, inputrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (list_is_subset(ukey->exprs, grouprel->reltarget->exprs)) + { + grouprel->uniquekeys = lappend(grouprel->uniquekeys, + ukey); + input_ukey_added = true; + } + } + if (!input_ukey_added) + /* + * group by clause must be a super-set of grouprel->reltarget->exprs except the + * aggregation expr, so if such exprs is unique already, no bother to generate + * new uniquekey for group by exprs. + */ + add_uniquekey_from_sortgroups(root, + grouprel, + root->parse->groupClause); + } + else + /* It has aggregation but without a group by, so only one row returned */ + add_uniquekey_for_onerow(grouprel); +} + +/* + * simple_copy_uniquekeys + * Using a function for the one-line code makes us easy to check where we simply + * copied the uniquekey. + */ +void +simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel) +{ + newrel->uniquekeys = oldrel->uniquekeys; +} + +/* + * populate_unionrel_uniquekeys + */ +void +populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel) +{ + ListCell *lc; + List *exprs = NIL; + + Assert(unionrel->uniquekeys == NIL); + + foreach(lc, unionrel->reltarget->exprs) + { + exprs = lappend(exprs, lfirst(lc)); + } + + if (exprs == NIL) + /* SQL: select union select; is valid, we need to handle it here. */ + add_uniquekey_for_onerow(unionrel); + else + unionrel->uniquekeys = lappend(unionrel->uniquekeys, + makeUniqueKey(exprs,false)); + +} + +/* + * populate_joinrel_uniquekeys + * + * populate uniquekeys for joinrel. We will check each relation to see if its + * UniqueKey is still valid via innerrel_keeps_unique, if so, we add it to + * joinrel. The multi_nullvals field will be changed to true for some outer + * join cases and one-row UniqueKey needs to be converted to normal UniqueKey + * for the same case as well. + * For the uniquekey in either baserel which can't be unique after join, we still + * check to see if combination of UniqueKeys from both side is still useful for us. + * if yes, we add it to joinrel as well. + */ +void +populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *outerrel, RelOptInfo *innerrel, + List *restrictlist, JoinType jointype) +{ + ListCell *lc, *lc2; + List *clause_list = NIL; + List *outerrel_ukey_ctx; + List *innerrel_ukey_ctx; + bool inner_onerow, outer_onerow; + bool mergejoin_allowed; + + /* Care about the outerrel relation only for SEMI/ANTI join */ + if (jointype == JOIN_SEMI || jointype == JOIN_ANTI) + { + foreach(lc, outerrel->uniquekeys) + { + UniqueKey *uniquekey = lfirst_node(UniqueKey, lc); + if (list_is_subset(uniquekey->exprs, joinrel->reltarget->exprs)) + joinrel->uniquekeys = lappend(joinrel->uniquekeys, uniquekey); + } + return; + } + + Assert(jointype == JOIN_LEFT || jointype == JOIN_FULL || jointype == JOIN_INNER); + + /* Fast path */ + if (innerrel->uniquekeys == NIL || outerrel->uniquekeys == NIL) + return; + + inner_onerow = relation_is_onerow(innerrel); + outer_onerow = relation_is_onerow(outerrel); + + outerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(outerrel); + innerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(innerrel); + + clause_list = select_mergejoin_clauses(root, joinrel, outerrel, innerrel, + restrictlist, jointype, + &mergejoin_allowed); + + if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true /* reverse */)) + { + bool outer_impact = jointype == JOIN_FULL; + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + /* Outer relation has one row, and the unique key is not duplicated after join, + * the joinrel will still has one row unless the jointype == JOIN_FULL. + */ + if (outer_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (outer_onerow) + { + /* + * The onerow outerrel becomes multi rows and multi_nullvals + * will be changed to true. We also need to set the exprs correctly since it + * can't be NIL any more. + */ + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(joinrel, outerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Change multi_nullvals to true due to the full join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, true)); + else + /* Just reuse it */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + } + ctx->added_to_joinrel = true; + } + } + + if (innerrel_keeps_unique(root, outerrel, innerrel, clause_list, false)) + { + bool outer_impact = jointype == JOIN_FULL || jointype == JOIN_LEFT;; + + foreach(lc, innerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + if (inner_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (inner_onerow) + { + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(joinrel, innerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Need to change multi_nullvals to true due to the outer join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, + true)); + else + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + + } + ctx->added_to_joinrel = true; + } + } + + /* + * The combination of the UniqueKey from both sides is unique as well regardless + * of join type, but no bother to add it if its subset has been added to joinrel + * already or it is not useful for the joinrel. + */ + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx1 = (UniqueKeyContext) lfirst(lc); + if (ctx1->added_to_joinrel || !ctx1->useful) + continue; + foreach(lc2, innerrel_ukey_ctx) + { + UniqueKeyContext ctx2 = (UniqueKeyContext) lfirst(lc2); + if (ctx2->added_to_joinrel || !ctx2->useful) + continue; + if (add_combined_uniquekey(joinrel, outerrel, innerrel, + ctx1->uniquekey, ctx2->uniquekey, + jointype)) + /* If we set a onerow UniqueKey to joinrel, we don't need other. */ + return; + } + } +} + + +/* + * convert_subquery_uniquekeys + * + * Covert the UniqueKey in subquery to outer relation. + */ +void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel) +{ + ListCell *lc; + + if (sub_final_rel->uniquekeys == NIL) + return; + + if (relation_is_onerow(sub_final_rel)) + { + add_uniquekey_for_onerow(currel); + return; + } + + Assert(currel->subroot != NULL); + + foreach(lc, sub_final_rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + ListCell *lc; + List *exprs = NIL; + bool ukey_useful = true; + + /* One row case is handled above */ + Assert(ukey->exprs != NIL); + foreach(lc, ukey->exprs) + { + Var *var; + TargetEntry *tle = tlist_member(lfirst(lc), + currel->subroot->processed_tlist); + if (tle == NULL) + { + ukey_useful = false; + break; + } + var = find_var_for_subquery_tle(currel, tle); + if (var == NULL) + { + ukey_useful = false; + break; + } + exprs = lappend(exprs, var); + } + + if (ukey_useful) + currel->uniquekeys = lappend(currel->uniquekeys, + makeUniqueKey(exprs, + ukey->multi_nullvals)); + + } +} + +/* + * innerrel_keeps_unique + * + * Check if Unique key of the innerrel is valid after join. innerrel's UniqueKey + * will be still valid if innerrel's any-column mergeop outrerel's uniquekey + * exists in clause_list. + * + * Note: the clause_list must be a list of mergeable restrictinfo already. + */ +static bool +innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *clause_list, + bool reverse) +{ + ListCell *lc, *lc2, *lc3; + + if (outerrel->uniquekeys == NIL || innerrel->uniquekeys == NIL) + return false; + + /* Check if there is outerrel's uniquekey in mergeable clause. */ + foreach(lc, outerrel->uniquekeys) + { + List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs; + bool clauselist_matchs_all_exprs = true; + foreach(lc2, outer_uq_exprs) + { + Node *outer_uq_expr = lfirst(lc2); + bool find_uq_expr_in_clauselist = false; + foreach(lc3, clause_list) + { + RestrictInfo *rinfo = lfirst_node(RestrictInfo, lc3); + Node *outer_expr; + if (reverse) + outer_expr = rinfo->outer_is_left ? get_rightop(rinfo->clause) : get_leftop(rinfo->clause); + else + outer_expr = rinfo->outer_is_left ? get_leftop(rinfo->clause) : get_rightop(rinfo->clause); + if (equal(outer_expr, outer_uq_expr)) + { + find_uq_expr_in_clauselist = true; + break; + } + } + if (!find_uq_expr_in_clauselist) + { + /* No need to check the next exprs in the current uniquekey */ + clauselist_matchs_all_exprs = false; + break; + } + } + + if (clauselist_matchs_all_exprs) + return true; + } + return false; +} + + +/* + * relation_is_onerow + * Check if it is a one-row relation by checking UniqueKey. + */ +bool +relation_is_onerow(RelOptInfo *rel) +{ + UniqueKey *ukey; + if (rel->uniquekeys == NIL) + return false; + ukey = linitial_node(UniqueKey, rel->uniquekeys); + return ukey->exprs == NIL && list_length(rel->uniquekeys) == 1; +} + +/* + * relation_has_uniquekeys_for + * Returns true if we have proofs that 'rel' cannot return multiple rows with + * the same values in each of 'exprs'. Otherwise returns false. + */ +bool +relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel, + List *exprs, bool allow_multinulls) +{ + ListCell *lc; + + /* + * For UniqueKey->onerow case, the uniquekey->exprs is empty as well + * so we can't rely on list_is_subset to handle this special cases + */ + if (exprs == NIL) + return false; + + foreach(lc, rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (ukey->multi_nullvals && !allow_multinulls) + continue; + if (list_is_subset(ukey->exprs, exprs)) + return true; + } + return false; +} + + +/* + * get_exprs_from_uniqueindex + * + * Return a list of exprs which is unique. set useful to false if this + * unique index is not useful for us. + */ +static List * +get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals) +{ + List *exprs = NIL; + ListCell *indexpr_item; + int c = 0; + + *useful = true; + *multi_nullvals = false; + + indexpr_item = list_head(unique_index->indexprs); + for(c = 0; c < unique_index->ncolumns; c++) + { + int attr = unique_index->indexkeys[c]; + Expr *expr; + bool matched_const = false; + ListCell *lc1, *lc2; + + if(attr > 0) + { + expr = list_nth_node(TargetEntry, unique_index->indextlist, c)->expr; + } + else if (attr == 0) + { + /* Expression index */ + expr = lfirst(indexpr_item); + indexpr_item = lnext(unique_index->indexprs, indexpr_item); + } + else /* attr < 0 */ + { + /* Index on system column is not supported */ + Assert(false); + } + + /* + * Check index_col = Const case with regarding to opfamily checking + * If we can remove the index_col from the final UniqueKey->exprs. + */ + forboth(lc1, const_exprs, lc2, const_expr_opfamilies) + { + if (list_member_oid((List *)lfirst(lc2), unique_index->opfamily[c]) + && match_index_to_operand((Node *) lfirst(lc1), c, unique_index)) + { + matched_const = true; + break; + } + } + + if (matched_const) + continue; + + /* Check if the indexed expr is used in rel */ + if (attr > 0) + { + /* + * Normal Indexed column, if the col is not used, then the index is useless + * for uniquekey. + */ + attr -= FirstLowInvalidHeapAttributeNumber; + + if (!bms_is_member(attr, used_varattrs)) + { + *useful = false; + break; + } + } + else if (!list_member(unique_index->rel->reltarget->exprs, expr)) + { + /* Expression index but the expression is not used in rel */ + *useful = false; + break; + } + + /* check not null property. */ + if (attr == 0) + { + /* We never know if a expression yields null or not */ + *multi_nullvals = true; + } + else if (!bms_is_member(attr, unique_index->rel->notnullattrs) + && !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber, + unique_index->rel->notnullattrs)) + { + *multi_nullvals = true; + } + + exprs = lappend(exprs, expr); + } + return exprs; +} + + +/* + * add_uniquekey_for_onerow + * If we are sure that the relation only returns one row, then all the columns + * are unique. However we don't need to create UniqueKey for every column, we + * just set exprs = NIL and overwrites all the other UniqueKey on this RelOptInfo + * since this one has strongest semantics. + */ +void +add_uniquekey_for_onerow(RelOptInfo *rel) +{ + /* + * We overwrite the previous UniqueKey on purpose since this one has the + * strongest semantic. + */ + rel->uniquekeys = list_make1(makeUniqueKey(NIL, false)); +} + + +/* + * initililze_uniquecontext_for_joinrel + * Return a List of UniqueKeyContext for an inputrel + */ +static List * +initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel) +{ + List *res = NIL; + ListCell *lc; + foreach(lc, inputrel->uniquekeys) + { + UniqueKeyContext context; + context = palloc(sizeof(struct UniqueKeyContextData)); + context->uniquekey = lfirst_node(UniqueKey, lc); + context->added_to_joinrel = false; + context->useful = true; + res = lappend(res, context); + } + return res; +} + + +/* + * get_exprs_from_uniquekey + * Unify the way of get List of exprs from a one-row UniqueKey or + * normal UniqueKey. for the onerow case, every expr in rel1 is a valid + * UniqueKey. Return a List of exprs. + * + * rel1: The relation which you want to get the exprs. + * ukey: The UniqueKey you want to get the exprs. + */ +static List * +get_exprs_from_uniquekey(RelOptInfo *joinrel, RelOptInfo *rel1, UniqueKey *ukey) +{ + ListCell *lc; + bool onerow = rel1 != NULL && relation_is_onerow(rel1); + + List *res = NIL; + Assert(onerow || ukey); + if (onerow) + { + /* Only cares about the exprs still exist in joinrel */ + foreach(lc, joinrel->reltarget->exprs) + { + Bitmapset *relids = pull_varnos(lfirst(lc)); + if (bms_is_subset(relids, rel1->relids)) + { + res = lappend(res, list_make1(lfirst(lc))); + } + } + } + else + { + res = list_make1(ukey->exprs); + } + return res; +} + +/* + * Partitioned table Unique Keys. + * The partition table unique key is maintained as: + * 1. The index must be unique as usual. + * 2. The index must contains partition key. + * 3. The index must exist on all the child rel. see simple_indexinfo_equal for + * how we compare it. + */ + +/* + * index_constains_partkey + * return true if the index contains the partiton key. + */ +static bool +index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind) +{ + ListCell *lc; + int i; + Assert(IS_PARTITIONED_REL(partrel)); + Assert(partrel->part_scheme->partnatts > 0); + + for(i = 0; i < partrel->part_scheme->partnatts; i++) + { + Node *part_expr = linitial(partrel->partexprs[i]); + bool found_in_index = false; + foreach(lc, ind->indextlist) + { + Expr *index_expr = lfirst_node(TargetEntry, lc)->expr; + if (equal(index_expr, part_expr)) + { + found_in_index = true; + break; + } + } + if (!found_in_index) + return false; + } + return true; +} + +/* + * simple_indexinfo_equal + * + * Used to check if the 2 index is same as each other. The index here + * is COPIED from childrel and did some tiny changes(see + * simple_copy_indexinfo_to_parent) + */ +static bool +simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2) +{ + Size oid_cmp_len = sizeof(Oid) * ind1->ncolumns; + + return ind1->ncolumns == ind2->ncolumns && + ind1->unique == ind2->unique && + memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0 && + memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0 && + memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0 && + memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0 && + equal(get_tlist_exprs(ind1->indextlist, true), + get_tlist_exprs(ind2->indextlist, true)); +} + + +/* + * The below macros are used for simple_copy_indexinfo_to_parent which is so + * customized that I don't want to put it to copyfuncs.c. So copy it here. + */ +#define COPY_POINTER_FIELD(fldname, sz) \ + do { \ + Size _size = (sz); \ + newnode->fldname = palloc(_size); \ + memcpy(newnode->fldname, from->fldname, _size); \ + } while (0) + +#define COPY_NODE_FIELD(fldname) \ + (newnode->fldname = copyObjectImpl(from->fldname)) + +#define COPY_SCALAR_FIELD(fldname) \ + (newnode->fldname = from->fldname) + + +/* + * simple_copy_indexinfo_to_parent (from partition) + * Copy the IndexInfo from child relation to parent relation with some modification, + * which is used to test: + * 1. If the same index exists in all the childrels. + * 2. If the parentrel->reltarget/basicrestrict info matches this index. + */ +static IndexOptInfo * +simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from) +{ + IndexOptInfo *newnode = makeNode(IndexOptInfo); + AppendRelInfo *appinfo = find_appinfo_by_child(root, from->rel->relid); + ListCell *lc; + int idx = 0; + + COPY_SCALAR_FIELD(ncolumns); + COPY_SCALAR_FIELD(nkeycolumns); + COPY_SCALAR_FIELD(unique); + COPY_SCALAR_FIELD(immediate); + /* We just need to know if it is NIL or not */ + COPY_SCALAR_FIELD(indpred); + COPY_SCALAR_FIELD(predOK); + COPY_POINTER_FIELD(indexkeys, from->ncolumns * sizeof(int)); + COPY_POINTER_FIELD(indexcollations, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opfamily, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opcintype, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(sortopfamily, from->ncolumns * sizeof(Oid)); + COPY_NODE_FIELD(indextlist); + + /* Convert index exprs on child expr to expr on parent */ + foreach(lc, newnode->indextlist) + { + TargetEntry *tle = lfirst_node(TargetEntry, lc); + /* Index on expression is ignored */ + Assert(IsA(tle->expr, Var)); + tle->expr = (Expr *) find_parent_var(appinfo, (Var *) tle->expr); + newnode->indexkeys[idx] = castNode(Var, tle->expr)->varattno; + idx++; + } + newnode->rel = parentrel; + return newnode; +} + +/* + * adjust_partition_unique_indexlist + * + * global_unique_indexes: At the beginning, it contains the copy & modified + * unique index from the first partition. And then check if each index in it still + * exists in the following partitions. If no, remove it. at last, it has an + * index list which exists in all the partitions. + */ +static void +adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_indexes) +{ + ListCell *lc, *lc2; + foreach(lc, *global_unique_indexes) + { + IndexOptInfo *g_ind = lfirst_node(IndexOptInfo, lc); + bool found_in_child = false; + + foreach(lc2, childrel->indexlist) + { + IndexOptInfo *p_ind = lfirst_node(IndexOptInfo, lc2); + IndexOptInfo *p_ind_copy; + if (!p_ind->unique || !p_ind->immediate || + (p_ind->indpred != NIL && !p_ind->predOK)) + continue; + p_ind_copy = simple_copy_indexinfo_to_parent(root, parentrel, p_ind); + if (simple_indexinfo_equal(p_ind_copy, g_ind)) + { + found_in_child = true; + break; + } + } + if (!found_in_child) + /* The index doesn't exist in childrel, remove it from global_unique_indexes */ + *global_unique_indexes = foreach_delete_current(*global_unique_indexes, lc); + } +} + +/* Helper function for groupres/distinctrel */ +static void +add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgroups) +{ + Query *parse = root->parse; + List *exprs; + + /* + * XXX: If there are some vars which is not in current levelsup, the semantic is + * imprecise, should we avoid it or not? levelsup = 1 is just a demo, maybe we need to + * check every level other than 0, if so, looks we have to write another + * pull_var_walker. + */ + List *upper_vars = pull_vars_of_level((Node*)sortgroups, 1); + + if (upper_vars != NIL) + return; + + exprs = get_sortgrouplist_exprs(sortgroups, parse->targetList); + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(exprs, + false /* sortgroupclause can't be multi_nullvals */)); +} + + +/* + * add_combined_uniquekey + * The combination of both UniqueKeys is a valid UniqueKey for joinrel no matter + * the jointype. + */ +bool +add_combined_uniquekey(RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype) +{ + + ListCell *lc1, *lc2; + + /* Either side has multi_nullvals or we have outer join, + * the combined UniqueKey has multi_nullvals */ + bool multi_nullvals = outer_ukey->multi_nullvals || + inner_ukey->multi_nullvals || IS_OUTER_JOIN(jointype); + + /* The only case we can get onerow joinrel after join */ + if (relation_is_onerow(outer_rel) + && relation_is_onerow(inner_rel) + && jointype == JOIN_INNER) + { + add_uniquekey_for_onerow(joinrel); + return true; + } + + foreach(lc1, get_exprs_from_uniquekey(joinrel, outer_rel, outer_ukey)) + { + foreach(lc2, get_exprs_from_uniquekey(joinrel, inner_rel, inner_ukey)) + { + List *exprs = list_concat_copy(lfirst_node(List, lc1), lfirst_node(List, lc2)); + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(exprs, + multi_nullvals)); + } + } + return false; +} diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index b406d41e91..0551ae0512 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -2389,6 +2389,8 @@ grouping_planner(PlannerInfo *root, bool inheritance_update, add_path(final_rel, path); } + simple_copy_uniquekeys(current_rel, final_rel); + /* * Generate partial paths for final_rel, too, if outer query levels might * be able to make use of them. @@ -3899,6 +3901,8 @@ create_grouping_paths(PlannerInfo *root, } set_cheapest(grouped_rel); + + populate_grouprel_uniquekeys(root, grouped_rel, input_rel); return grouped_rel; } @@ -4615,7 +4619,7 @@ create_window_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(window_rel); - + simple_copy_uniquekeys(input_rel, window_rel); return window_rel; } @@ -4911,7 +4915,7 @@ create_distinct_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(distinct_rel); - + populate_distinctrel_uniquekeys(root, input_rel, distinct_rel); return distinct_rel; } @@ -5172,6 +5176,8 @@ create_ordered_paths(PlannerInfo *root, */ Assert(ordered_rel->pathlist != NIL); + simple_copy_uniquekeys(input_rel, ordered_rel); + return ordered_rel; } @@ -6049,6 +6055,9 @@ adjust_paths_for_srfs(PlannerInfo *root, RelOptInfo *rel, if (list_length(targets) == 1) return; + /* UniqueKey is not valid after handling the SRF. */ + rel->uniquekeys = NIL; + /* * Stack SRF-evaluation nodes atop each path for the rel. * diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c index 951aed80e7..e94e92937c 100644 --- a/src/backend/optimizer/prep/prepunion.c +++ b/src/backend/optimizer/prep/prepunion.c @@ -689,6 +689,8 @@ generate_union_paths(SetOperationStmt *op, PlannerInfo *root, /* Undo effects of possibly forcing tuple_fraction to 0 */ root->tuple_fraction = save_fraction; + /* Add the UniqueKeys */ + populate_unionrel_uniquekeys(root, result_rel); return result_rel; } diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c index d722063cf3..44c37ecffc 100644 --- a/src/backend/optimizer/util/appendinfo.c +++ b/src/backend/optimizer/util/appendinfo.c @@ -746,3 +746,47 @@ find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos) } return appinfos; } + +/* + * find_appinfo_by_child + * + */ +AppendRelInfo * +find_appinfo_by_child(PlannerInfo *root, Index child_index) +{ + ListCell *lc; + foreach(lc, root->append_rel_list) + { + AppendRelInfo *appinfo = lfirst_node(AppendRelInfo, lc); + if (appinfo->child_relid == child_index) + return appinfo; + } + elog(ERROR, "parent relation cant be found"); + return NULL; +} + +/* + * find_parent_var + * + */ +Var * +find_parent_var(AppendRelInfo *appinfo, Var *child_var) +{ + ListCell *lc; + Var *res = NULL; + Index attno = 1; + foreach(lc, appinfo->translated_vars) + { + Node *child_node = lfirst(lc); + if (equal(child_node, child_var)) + { + res = copyObject(child_var); + res->varattno = attno; + res->varno = appinfo->parent_relid; + } + attno++; + } + if (res == NULL) + elog(ERROR, "parent var can't be found."); + return res; +} diff --git a/src/backend/optimizer/util/inherit.c b/src/backend/optimizer/util/inherit.c index 3132fd35a5..d66b40ec50 100644 --- a/src/backend/optimizer/util/inherit.c +++ b/src/backend/optimizer/util/inherit.c @@ -736,6 +736,7 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, { Node *onecq = (Node *) lfirst(lc2); bool pseudoconstant; + RestrictInfo *child_rinfo; /* check for pseudoconstant (no Vars or volatile functions) */ pseudoconstant = @@ -747,13 +748,14 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, root->hasPseudoConstantQuals = true; } /* reconstitute RestrictInfo with appropriate properties */ - childquals = lappend(childquals, - make_restrictinfo((Expr *) onecq, - rinfo->is_pushed_down, - rinfo->outerjoin_delayed, - pseudoconstant, - rinfo->security_level, - NULL, NULL, NULL)); + child_rinfo = make_restrictinfo((Expr *) onecq, + rinfo->is_pushed_down, + rinfo->outerjoin_delayed, + pseudoconstant, + rinfo->security_level, + NULL, NULL, NULL); + child_rinfo->mergeopfamilies = rinfo->mergeopfamilies; + childquals = lappend(childquals, child_rinfo); /* track minimum security level among child quals */ cq_min_security = Min(cq_min_security, rinfo->security_level); } diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 31d9aedeeb..c83f17acb7 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -16,6 +16,7 @@ #include "nodes/execnodes.h" #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" extern A_Expr *makeA_Expr(A_Expr_Kind kind, List *name, @@ -105,4 +106,6 @@ extern GroupingSet *makeGroupingSet(GroupingSetKind kind, List *content, int loc extern VacuumRelation *makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols); +extern UniqueKey* makeUniqueKey(List *exprs, bool multi_nullvals); + #endif /* MAKEFUNC_H */ diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 381d84b4e4..41110ed888 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -264,6 +264,7 @@ typedef enum NodeTag T_EquivalenceMember, T_PathKey, T_PathTarget, + T_UniqueKey, T_RestrictInfo, T_IndexClause, T_PlaceHolderVar, diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index 9e3ebd488a..02e4458bef 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -730,6 +730,7 @@ typedef struct RelOptInfo QualCost baserestrictcost; /* cost of evaluating the above */ Index baserestrict_min_security; /* min security_level found in * baserestrictinfo */ + List *uniquekeys; /* List of UniqueKey */ List *joininfo; /* RestrictInfo structures for join clauses * involving this rel */ bool has_eclass_joins; /* T means joininfo is incomplete */ @@ -1047,6 +1048,28 @@ typedef struct PathKey } PathKey; +/* + * UniqueKey + * + * Represents the unique properties held by a RelOptInfo. + * + * exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL + * is a special case of UniqueKey, which means there is only 1 row in that + * relation. + * multi_nullvals: true means multi null values may exist in these exprs, so the + * uniqueness is not guaranteed in this case. This field is necessary for + * remove_useless_join & reduce_unique_semijoins where we don't mind these + * duplicated NULL values. It is set to true for 2 cases. One is a unique key + * from a unique index but the related column is nullable. The other one is for + * outer join. see populate_joinrel_uniquekeys for detail. + */ +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + /* * PathTarget * @@ -2473,7 +2496,7 @@ typedef enum * * flags indicating what kinds of grouping are possible. * partial_costs_set is true if the agg_partial_costs and agg_final_costs - * have been initialized. + * have been initialized. * agg_partial_costs gives partial aggregation costs. * agg_final_costs gives finalization costs. * target_parallel_safe is true if target is parallel safe. @@ -2503,8 +2526,8 @@ typedef struct * limit_tuples is an estimated bound on the number of output tuples, * or -1 if no LIMIT or couldn't estimate. * count_est and offset_est are the estimated values of the LIMIT and OFFSET - * expressions computed by preprocess_limit() (see comments for - * preprocess_limit() for more information). + * expressions computed by preprocess_limit() (see comments for + * preprocess_limit() for more information). */ typedef struct { diff --git a/src/include/nodes/pg_list.h b/src/include/nodes/pg_list.h index 14ea2766ad..621f54a9f8 100644 --- a/src/include/nodes/pg_list.h +++ b/src/include/nodes/pg_list.h @@ -528,6 +528,8 @@ extern bool list_member_ptr(const List *list, const void *datum); extern bool list_member_int(const List *list, int datum); extern bool list_member_oid(const List *list, Oid datum); +extern bool list_is_subset(const List *members, const List *target); + extern List *list_delete(List *list, void *datum); extern List *list_delete_ptr(List *list, void *datum); extern List *list_delete_int(List *list, int datum); diff --git a/src/include/optimizer/appendinfo.h b/src/include/optimizer/appendinfo.h index d6a27a60dd..e87c92a054 100644 --- a/src/include/optimizer/appendinfo.h +++ b/src/include/optimizer/appendinfo.h @@ -32,4 +32,7 @@ extern Relids adjust_child_relids_multilevel(PlannerInfo *root, Relids relids, extern AppendRelInfo **find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos); +extern AppendRelInfo *find_appinfo_by_child(PlannerInfo *root, Index child_index); +extern Var *find_parent_var(AppendRelInfo *appinfo, Var *child_var); + #endif /* APPENDINFO_H */ diff --git a/src/include/optimizer/optimizer.h b/src/include/optimizer/optimizer.h index 3e4171056e..9445141263 100644 --- a/src/include/optimizer/optimizer.h +++ b/src/include/optimizer/optimizer.h @@ -23,6 +23,7 @@ #define OPTIMIZER_H #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" /* * We don't want to include nodes/pathnodes.h here, because non-planner @@ -156,6 +157,7 @@ extern TargetEntry *get_sortgroupref_tle(Index sortref, List *targetList); extern TargetEntry *get_sortgroupclause_tle(SortGroupClause *sgClause, List *targetList); +extern Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); extern Node *get_sortgroupclause_expr(SortGroupClause *sgClause, List *targetList); extern List *get_sortgrouplist_exprs(List *sgClauses, diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h index 10b6e81079..9217a8d6c6 100644 --- a/src/include/optimizer/paths.h +++ b/src/include/optimizer/paths.h @@ -240,5 +240,48 @@ extern PathKey *make_canonical_pathkey(PlannerInfo *root, int strategy, bool nulls_first); extern void add_paths_to_append_rel(PlannerInfo *root, RelOptInfo *rel, List *live_childrels); +extern List *select_mergejoin_clauses(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + JoinType jointype, + bool *mergejoin_allowed); + +/* + * uniquekeys.c + * Utilities for matching and building unique keys + */ +extern void populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List* unique_index_list); +extern void populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels); +extern void populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel); +extern void populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel); +extern void populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel); +extern void simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel); +extern void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel); +extern void populate_joinrel_uniquekeys(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + RelOptInfo *rel2, + List *restrictlist, + JoinType jointype); + +extern bool relation_has_uniquekeys_for(PlannerInfo *root, + RelOptInfo *rel, + List *exprs, + bool allow_multinulls); +extern bool relation_is_onerow(RelOptInfo *rel); #endif /* PATHS_H */ -- 2.21.0 --op7mf2i72xk5yu7c Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v36-0003-Extend-UniqueKeys.patch" ^ permalink raw reply [nested|flat] 91+ messages in thread
* [PATCH v38 2/6] Introduce UniqueKey attributes on RelOptInfo struct. @ 2020-05-11 07:50 一挃 <[email protected]> 0 siblings, 0 replies; 91+ messages in thread From: 一挃 @ 2020-05-11 07:50 UTC (permalink / raw) UniqueKey is a set of exprs on RelOptInfo which represents the exprs will be unique on the given RelOptInfo. You can see README.uniquekey for more information. --- src/backend/nodes/copyfuncs.c | 13 + src/backend/nodes/list.c | 31 + src/backend/nodes/makefuncs.c | 13 + src/backend/nodes/outfuncs.c | 11 + src/backend/nodes/readfuncs.c | 10 + src/backend/optimizer/path/Makefile | 3 +- src/backend/optimizer/path/README.uniquekey | 131 +++ src/backend/optimizer/path/allpaths.c | 10 + src/backend/optimizer/path/joinpath.c | 9 +- src/backend/optimizer/path/joinrels.c | 2 + src/backend/optimizer/path/pathkeys.c | 3 +- src/backend/optimizer/path/uniquekeys.c | 1135 +++++++++++++++++++ src/backend/optimizer/plan/planner.c | 13 +- src/backend/optimizer/prep/prepunion.c | 2 + src/backend/optimizer/util/appendinfo.c | 44 + src/backend/optimizer/util/inherit.c | 18 +- src/include/nodes/makefuncs.h | 3 + src/include/nodes/nodes.h | 1 + src/include/nodes/pathnodes.h | 29 +- src/include/nodes/pg_list.h | 10 + src/include/optimizer/appendinfo.h | 3 + src/include/optimizer/optimizer.h | 2 + src/include/optimizer/paths.h | 43 + 23 files changed, 1515 insertions(+), 24 deletions(-) create mode 100644 src/backend/optimizer/path/README.uniquekey create mode 100644 src/backend/optimizer/path/uniquekeys.c diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..75c1c5e824 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2296,6 +2296,16 @@ _copyPathKey(const PathKey *from) return newnode; } +static UniqueKey * +_copyUniqueKey(const UniqueKey *from) +{ + UniqueKey *newnode = makeNode(UniqueKey); + + COPY_NODE_FIELD(exprs); + COPY_SCALAR_FIELD(multi_nullvals); + + return newnode; +} /* * _copyRestrictInfo */ @@ -5220,6 +5230,9 @@ copyObjectImpl(const void *from) case T_PathKey: retval = _copyPathKey(from); break; + case T_UniqueKey: + retval = _copyUniqueKey(from); + break; case T_RestrictInfo: retval = _copyRestrictInfo(from); break; diff --git a/src/backend/nodes/list.c b/src/backend/nodes/list.c index dbf6b30233..ca099495a1 100644 --- a/src/backend/nodes/list.c +++ b/src/backend/nodes/list.c @@ -702,6 +702,37 @@ list_member_oid(const List *list, Oid datum) return false; } +/* + * return true iff every entry in "members" list is also present + * in the "target" list. + */ +bool +list_is_subset(const List *members, const List *target) +{ + const ListCell *lc1, *lc2; + + Assert(IsPointerList(members)); + Assert(IsPointerList(target)); + check_list_invariants(members); + check_list_invariants(target); + + foreach(lc1, members) + { + bool found = false; + foreach(lc2, target) + { + if (equal(lfirst(lc1), lfirst(lc2))) + { + found = true; + break; + } + } + if (!found) + return false; + } + return true; +} + /* * Delete the n'th cell (counting from 0) in list. * diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 01c110cd2f..40415d0f5b 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -815,3 +815,16 @@ makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols) v->va_cols = va_cols; return v; } + + +/* + * makeUniqueKey + */ +UniqueKey* +makeUniqueKey(List *exprs, bool multi_nullvals) +{ + UniqueKey * ukey = makeNode(UniqueKey); + ukey->exprs = exprs; + ukey->multi_nullvals = multi_nullvals; + return ukey; +} diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..44154cde6a 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2456,6 +2456,14 @@ _outPathKey(StringInfo str, const PathKey *node) WRITE_BOOL_FIELD(pk_nulls_first); } +static void +_outUniqueKey(StringInfo str, const UniqueKey *node) +{ + WRITE_NODE_TYPE("UNIQUEKEY"); + WRITE_NODE_FIELD(exprs); + WRITE_BOOL_FIELD(multi_nullvals); +} + static void _outPathTarget(StringInfo str, const PathTarget *node) { @@ -4198,6 +4206,9 @@ outNode(StringInfo str, const void *obj) case T_PathKey: _outPathKey(str, obj); break; + case T_UniqueKey: + _outUniqueKey(str, obj); + break; case T_PathTarget: _outPathTarget(str, obj); break; diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index c5e136e9c3..b3e212bf1c 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -492,6 +492,14 @@ _readSetOperationStmt(void) READ_DONE(); } +static UniqueKey * +_readUniqueKey(void) +{ + READ_LOCALS(UniqueKey); + READ_NODE_FIELD(exprs); + READ_BOOL_FIELD(multi_nullvals); + READ_DONE(); +} /* * Stuff from primnodes.h. @@ -2717,6 +2725,8 @@ parseNodeString(void) return_value = _readCommonTableExpr(); else if (MATCH("SETOPERATIONSTMT", 16)) return_value = _readSetOperationStmt(); + else if (MATCH("UNIQUEKEY", 9)) + return_value = _readUniqueKey(); else if (MATCH("ALIAS", 5)) return_value = _readAlias(); else if (MATCH("RANGEVAR", 8)) diff --git a/src/backend/optimizer/path/Makefile b/src/backend/optimizer/path/Makefile index 1e199ff66f..7b9820c25f 100644 --- a/src/backend/optimizer/path/Makefile +++ b/src/backend/optimizer/path/Makefile @@ -21,6 +21,7 @@ OBJS = \ joinpath.o \ joinrels.o \ pathkeys.o \ - tidpath.o + tidpath.o \ + uniquekeys.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/optimizer/path/README.uniquekey b/src/backend/optimizer/path/README.uniquekey new file mode 100644 index 0000000000..5eac761995 --- /dev/null +++ b/src/backend/optimizer/path/README.uniquekey @@ -0,0 +1,131 @@ +1. What is UniqueKey? +We can think UniqueKey is a set of exprs for a RelOptInfo, which we are insure +that doesn't yields same result among all the rows. The simplest UniqueKey +format is primary key. + +However we define the UnqiueKey as below. + +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + +exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL +is a special case of UniqueKey, which means there is only one row in that +relation.it has a stronger semantic than others. like SELECT uk FROM t; uk is +normal unique key and may have different values. SELECT colx FROM t WHERE uk = +const. colx is unique AND we have only 1 value. This field can used for +innerrel_is_unique. this logic is handled specially in add_uniquekey_for_onerow +function. + +multi_nullvals: true means multi null values may exist in these exprs, so the +uniqueness is not guaranteed in this case. This field is necessary for +remove_useless_join & reduce_unique_semijoins where we don't mind these +duplicated NULL values. It is set to true for 2 cases. One is a unique key +from a unique index but the related column is nullable. The other one is for +outer join. see populate_joinrel_uniquekeys for detail. + + +The UniqueKey can be used at the following cases at least: +1. remove_useless_joins. +2. reduce_semianti_joins +3. remove distinct node if distinct clause is unique. +4. remove aggnode if group by clause is unique. +5. Index Skip Scan (WIP) +6. Aggregation Push Down without 2 phase aggregation if the join can't + duplicated the aggregated rows. (work in progress feature) + +2. How is it maintained? + +We have a set of populate_xxx_unqiuekeys functions to maintain the uniquekey on +various cases. xxx includes baserel, joinrel, partitionedrel, distinctrel, +groupedrel, unionrel. and we also need to convert the uniquekey from subquery +to outer relation, which is what convert_subquery_uniquekeys does. + +1. The first part is about baserel. We handled 3 cases. suppose we have Unique +Index on (a, b). + +1. SELECT a, b FROM t. UniqueKey (a, b) +2. SELECT a FROM t WHERE b = 1; UniqueKey (a) +3. SELECT .. FROM t WHERE a = 1 AND b = 1; UniqueKey (NIL). onerow case, every + column is Unique. + +2. The next part is joinrel, this part is most error-prone, we simplified the rules +like below: +1. If the relation's UniqueKey can't be duplicated after join, then is will be + still valid for the join rel. The function we used here is + innerrel_keeps_unique. The basic idea is innerrel.any_col = outer.uk. + +2. If the UnqiueKey can't keep valid via the rule 1, the combination of the + UniqueKey from both sides are valid for sure. We can prove this as: if the + unique exprs from rel1 is duplicated by rel2, the duplicated rows must + contains different unique exprs from rel2. + +More considerations about onerow: +1. If relation with one row and it can't be duplicated, it is still possible + contains mulit_nullvas after outer join. +2. If the either UniqueKey can be duplicated after join, the can get one row + only when both side is one row AND there is no outer join. +3. Whenever the onerow UniqueKey is not a valid any more, we need to convert one + row UniqueKey to normal unique key since we don't store exprs for one-row + relation. get_exprs_from_uniquekeys will be used here. + + +More considerations about multi_nullvals after join: +1. If the original UnqiueKey has multi_nullvals, the final UniqueKey will have + mulit_nullvals in any case. +2. If a unique key doesn't allow mulit_nullvals, after some outer join, it + allows some outer join. + + +3. When we comes to subquery, we need to convert_subquery_unqiuekeys just like +convert_subquery_pathkeys. Only the UniqueKey insides subquery is referenced as +a Var in outer relation will be reused. The relationship between the outerrel.Var +and subquery.exprs is built with outerel->subroot->processed_tlist. + + +4. As for the SRF functions, it will break the uniqueness of uniquekey, However it +is handled in adjust_paths_for_srfs, which happens after the query_planner. so +we will maintain the UniqueKey until there and reset it to NIL at that +places. This can't help on distinct/group by elimination cases but probably help +in some other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is +semantic correctly. + + +5. As for inherit table, we first main the UnqiueKey on childrel as well. But for +partitioned table we need to maintain 2 different kinds of +UnqiueKey. 1). UniqueKey on the parent relation 2). UniqueKey on child +relation for partition wise query. + +Example: +CREATE TABLE p (a int not null, b int not null) partition by list (a); +CREATE TABLE p0 partition of p for values in (1); +CREATE TABLE p1 partition of p for values in (2); + +create unique index p0_b on p0(b); +create unique index p1_b on p1(b); + +Now b is only unique on partition level, so the distinct can't be removed on +the following cases. SELECT DISTINCT b FROM p; + +Another example is SELECT DISTINCT a, b FROM p WHERE a = 1; Since only one +partition is chosen, the UniqueKey on child relation is same as the UniqueKey on +parent relation. + +Another usage of UniqueKey on partition level is it be helpful for +partition-wise join. + +As for the UniqueKey on parent table level, it comes with 2 different ways, +1). the UniqueKey is also derived in UniqueKey index, but the index must be same +in all the related children relations and the unique index must contains +Partition Key in it. Example: + +CREATE UNIQUE INDEX p_ab ON p(a, b); -- where a is the partition key. + +-- Query +SELECT a, b FROM p; the (a, b) is a UniqueKey of p. + +2). If the parent relation has only one childrel, the UniqueKey on childrel is + the UniqueKey on parent as well. diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index 37b4223adb..d52ad59f89 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -579,6 +579,12 @@ set_plain_rel_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) */ check_index_predicates(root, rel); + /* + * Now that we've marked which partial indexes are suitable, we can now + * build the relation's unique keys. + */ + populate_baserel_uniquekeys(root, rel, rel->indexlist); + /* Mark rel with estimated output rows, width, etc */ set_baserel_size_estimates(root, rel); } @@ -1297,6 +1303,8 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, /* Add paths to the append relation. */ add_paths_to_append_rel(root, rel, live_childrels); + if (IS_PARTITIONED_REL(rel)) + populate_partitionedrel_uniquekeys(root, rel, live_childrels); } @@ -2304,6 +2312,8 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, pathkeys, required_outer)); } + convert_subquery_uniquekeys(root, rel, sub_final_rel); + /* If outer rel allows parallelism, do same for partial paths. */ if (rel->consider_parallel && bms_is_empty(required_outer)) { diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c index 57ce97fd53..697ac047fb 100644 --- a/src/backend/optimizer/path/joinpath.c +++ b/src/backend/optimizer/path/joinpath.c @@ -71,13 +71,6 @@ static void consider_parallel_mergejoin(PlannerInfo *root, static void hash_inner_and_outer(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, RelOptInfo *innerrel, JoinType jointype, JoinPathExtraData *extra); -static List *select_mergejoin_clauses(PlannerInfo *root, - RelOptInfo *joinrel, - RelOptInfo *outerrel, - RelOptInfo *innerrel, - List *restrictlist, - JoinType jointype, - bool *mergejoin_allowed); static void generate_mergejoin_paths(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *innerrel, @@ -1927,7 +1920,7 @@ hash_inner_and_outer(PlannerInfo *root, * if it is mergejoinable and involves vars from the two sub-relations * currently of interest. */ -static List * +List * select_mergejoin_clauses(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c index 0dbe2ac726..7271f044ec 100644 --- a/src/backend/optimizer/path/joinrels.c +++ b/src/backend/optimizer/path/joinrels.c @@ -924,6 +924,8 @@ populate_joinrel_with_paths(PlannerInfo *root, RelOptInfo *rel1, /* Apply partitionwise join technique, if possible. */ try_partitionwise_join(root, rel1, rel2, joinrel, sjinfo, restrictlist); + + populate_joinrel_uniquekeys(root, joinrel, rel1, rel2, restrictlist, sjinfo->jointype); } diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index bd9a176d7d..139278829b 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -33,7 +33,6 @@ static bool pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys); static bool matches_boolean_partition_clause(RestrictInfo *rinfo, RelOptInfo *partrel, int partkeycol); -static Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); static bool right_merge_direction(PlannerInfo *root, PathKey *pathkey); @@ -1035,7 +1034,7 @@ convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel, * We need this to ensure that we don't return pathkeys describing values * that are unavailable above the level of the subquery scan. */ -static Var * +Var * find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle) { ListCell *lc; diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c new file mode 100644 index 0000000000..77ed2b2eff --- /dev/null +++ b/src/backend/optimizer/path/uniquekeys.c @@ -0,0 +1,1135 @@ +/*------------------------------------------------------------------------- + * + * uniquekeys.c + * Utilities for matching and building unique keys + * + * Portions Copyright (c) 2020, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/optimizer/path/uniquekeys.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "nodes/makefuncs.h" +#include "nodes/nodeFuncs.h" +#include "optimizer/pathnode.h" +#include "optimizer/paths.h" +#include "optimizer/appendinfo.h" +#include "optimizer/optimizer.h" +#include "optimizer/tlist.h" +#include "rewrite/rewriteManip.h" + + +/* + * This struct is used to help populate_joinrel_uniquekeys. + * + * added_to_joinrel is true if a uniquekey (from outerrel or innerrel) + * has been added to joinrel. + * useful is true if the exprs of the uniquekey still appears in joinrel. + */ +typedef struct UniqueKeyContextData +{ + UniqueKey *uniquekey; + bool added_to_joinrel; + bool useful; +} *UniqueKeyContext; + +static List *initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel); +static bool innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + bool reverse); + +static List *get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals); +static List *get_exprs_from_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + UniqueKey *ukey); +static void add_uniquekey_for_onerow(RelOptInfo *rel); +static bool add_combined_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype); + +/* Used for unique indexes checking for partitioned table */ +static bool index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind); +static IndexOptInfo *simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from); +static bool simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2); +static void adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_index); + +/* Helper function for grouped relation and distinct relation. */ +static void add_uniquekey_from_sortgroups(PlannerInfo *root, + RelOptInfo *rel, + List *sortgroups); + +/* + * populate_baserel_uniquekeys + * Populate 'baserel' uniquekeys list by looking at the rel's unique index + * and baserestrictinfo + */ +void +populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List *indexlist) +{ + ListCell *lc; + List *matched_uniq_indexes = NIL; + + /* Attrs appears in rel->reltarget->exprs. */ + Bitmapset *used_attrs = NULL; + + List *const_exprs = NIL; + List *expr_opfamilies = NIL; + + Assert(baserel->rtekind == RTE_RELATION); + + foreach(lc, indexlist) + { + IndexOptInfo *ind = (IndexOptInfo *) lfirst(lc); + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + matched_uniq_indexes = lappend(matched_uniq_indexes, ind); + } + + if (matched_uniq_indexes == NIL) + return; + + /* Check which attrs is used in baserel->reltarget */ + pull_varattnos((Node *)baserel->reltarget->exprs, baserel->relid, &used_attrs); + + /* Check which attrno is used at a mergeable const filter */ + foreach(lc, baserel->baserestrictinfo) + { + RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); + + if (rinfo->mergeopfamilies == NIL) + continue; + + if (bms_is_empty(rinfo->left_relids)) + { + const_exprs = lappend(const_exprs, get_rightop(rinfo->clause)); + } + else if (bms_is_empty(rinfo->right_relids)) + { + const_exprs = lappend(const_exprs, get_leftop(rinfo->clause)); + } + else + continue; + + expr_opfamilies = lappend(expr_opfamilies, rinfo->mergeopfamilies); + } + + foreach(lc, matched_uniq_indexes) + { + bool multi_nullvals, useful; + List *exprs = get_exprs_from_uniqueindex(lfirst_node(IndexOptInfo, lc), + const_exprs, + expr_opfamilies, + used_attrs, + &useful, + &multi_nullvals); + if (useful) + { + if (exprs == NIL) + { + /* All the columns in Unique Index matched with a restrictinfo */ + add_uniquekey_for_onerow(baserel); + return; + } + baserel->uniquekeys = lappend(baserel->uniquekeys, + makeUniqueKey(exprs, multi_nullvals)); + } + } +} + + +/* + * populate_partitionedrel_uniquekeys + * The UniqueKey on partitionrel comes from 2 cases: + * 1). Only one partition is involved in this query, the unique key can be + * copied to parent rel from childrel. + * 2). There are some unique index which includes partition key and exists + * in all the related partitions. + * We never mind rule 2 if we hit rule 1. + */ + +void +populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels) +{ + ListCell *lc; + List *global_uniq_indexlist = NIL; + RelOptInfo *childrel; + bool is_first = true; + + Assert(IS_PARTITIONED_REL(rel)); + + if (childrels == NIL) + return; + + /* + * If there is only one partition used in this query, the UniqueKey in childrel is + * still valid in parent level, but we need convert the format from child expr to + * parent expr. + */ + if (list_length(childrels) == 1) + { + /* Check for Rule 1 */ + RelOptInfo *childrel = linitial_node(RelOptInfo, childrels); + ListCell *lc; + Assert(childrel->reloptkind == RELOPT_OTHER_MEMBER_REL); + if (relation_is_onerow(childrel)) + { + add_uniquekey_for_onerow(rel); + return; + } + + foreach(lc, childrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + AppendRelInfo *appinfo = find_appinfo_by_child(root, childrel->relid); + List *parent_exprs = NIL; + bool can_reuse = true; + ListCell *lc2; + foreach(lc2, ukey->exprs) + { + Var *var = (Var *)lfirst(lc2); + /* + * If the expr comes from a expression, it is hard to build the expression + * in parent so ignore that case for now. + */ + if(!IsA(var, Var)) + { + can_reuse = false; + break; + } + /* Convert it to parent var */ + parent_exprs = lappend(parent_exprs, find_parent_var(appinfo, var)); + } + if (can_reuse) + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(parent_exprs, + ukey->multi_nullvals)); + } + } + else + { + /* Check for rule 2 */ + childrel = linitial_node(RelOptInfo, childrels); + foreach(lc, childrel->indexlist) + { + IndexOptInfo *ind = lfirst(lc); + IndexOptInfo *modified_index; + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + + /* + * During simple_copy_indexinfo_to_parent, we need to convert var from + * child var to parent var, index on expression is too complex to handle. + * so ignore it for now. + */ + if (ind->indexprs != NIL) + continue; + + modified_index = simple_copy_indexinfo_to_parent(root, rel, ind); + /* + * If the unique index doesn't contain partkey, then it is unique + * on this partition only, so it is useless for us. + */ + if (!index_constains_partkey(rel, modified_index)) + continue; + + global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index); + } + + if (global_uniq_indexlist != NIL) + { + foreach(lc, childrels) + { + RelOptInfo *child = lfirst(lc); + if (is_first) + { + is_first = false; + continue; + } + adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist); + } + /* Now we have a list of unique index which are exactly same on all childrels, + * Set the UniqueKey just like it is non-partition table + */ + populate_baserel_uniquekeys(root, rel, global_uniq_indexlist); + } + } +} + + +/* + * populate_distinctrel_uniquekeys + */ +void +populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel) +{ + /* The unique key before the distinct is still valid. */ + distinctrel->uniquekeys = list_copy(inputrel->uniquekeys); + add_uniquekey_from_sortgroups(root, distinctrel, root->parse->distinctClause); +} + +/* + * populate_grouprel_uniquekeys + */ +void +populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel) + +{ + Query *parse = root->parse; + bool input_ukey_added = false; + ListCell *lc; + + if (relation_is_onerow(inputrel)) + { + add_uniquekey_for_onerow(grouprel); + return; + } + if (parse->groupingSets) + return; + + /* A Normal group by without grouping set. */ + if (parse->groupClause) + { + /* + * Current even the groupby clause is Unique already, but if query has aggref + * We have to create grouprel still. To keep the UnqiueKey short, we will check + * the UniqueKey of input_rel still valid, if so we reuse it. + */ + foreach(lc, inputrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (list_is_subset(ukey->exprs, grouprel->reltarget->exprs)) + { + grouprel->uniquekeys = lappend(grouprel->uniquekeys, + ukey); + input_ukey_added = true; + } + } + if (!input_ukey_added) + /* + * group by clause must be a super-set of grouprel->reltarget->exprs except the + * aggregation expr, so if such exprs is unique already, no bother to generate + * new uniquekey for group by exprs. + */ + add_uniquekey_from_sortgroups(root, + grouprel, + root->parse->groupClause); + } + else + /* It has aggregation but without a group by, so only one row returned */ + add_uniquekey_for_onerow(grouprel); +} + +/* + * simple_copy_uniquekeys + * Using a function for the one-line code makes us easy to check where we simply + * copied the uniquekey. + */ +void +simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel) +{ + newrel->uniquekeys = oldrel->uniquekeys; +} + +/* + * populate_unionrel_uniquekeys + */ +void +populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel) +{ + ListCell *lc; + List *exprs = NIL; + + Assert(unionrel->uniquekeys == NIL); + + foreach(lc, unionrel->reltarget->exprs) + { + exprs = lappend(exprs, lfirst(lc)); + } + + if (exprs == NIL) + /* SQL: select union select; is valid, we need to handle it here. */ + add_uniquekey_for_onerow(unionrel); + else + unionrel->uniquekeys = lappend(unionrel->uniquekeys, + makeUniqueKey(exprs,false)); + +} + +/* + * populate_joinrel_uniquekeys + * + * populate uniquekeys for joinrel. We will check each relation to see if its + * UniqueKey is still valid via innerrel_keeps_unique, if so, we add it to + * joinrel. The multi_nullvals field will be changed to true for some outer + * join cases and one-row UniqueKey needs to be converted to normal UniqueKey + * for the same case as well. + * For the uniquekey in either baserel which can't be unique after join, we still + * check to see if combination of UniqueKeys from both side is still useful for us. + * if yes, we add it to joinrel as well. + */ +void +populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *outerrel, RelOptInfo *innerrel, + List *restrictlist, JoinType jointype) +{ + ListCell *lc, *lc2; + List *clause_list = NIL; + List *outerrel_ukey_ctx; + List *innerrel_ukey_ctx; + bool inner_onerow, outer_onerow; + bool mergejoin_allowed; + + /* Care about the outerrel relation only for SEMI/ANTI join */ + if (jointype == JOIN_SEMI || jointype == JOIN_ANTI) + { + foreach(lc, outerrel->uniquekeys) + { + UniqueKey *uniquekey = lfirst_node(UniqueKey, lc); + if (list_is_subset(uniquekey->exprs, joinrel->reltarget->exprs)) + joinrel->uniquekeys = lappend(joinrel->uniquekeys, uniquekey); + } + return; + } + + Assert(jointype == JOIN_LEFT || jointype == JOIN_FULL || jointype == JOIN_INNER); + + /* Fast path */ + if (innerrel->uniquekeys == NIL || outerrel->uniquekeys == NIL) + return; + + inner_onerow = relation_is_onerow(innerrel); + outer_onerow = relation_is_onerow(outerrel); + + outerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(outerrel); + innerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(innerrel); + + clause_list = select_mergejoin_clauses(root, joinrel, outerrel, innerrel, + restrictlist, jointype, + &mergejoin_allowed); + + if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true /* reverse */)) + { + bool outer_impact = jointype == JOIN_FULL; + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + /* Outer relation has one row, and the unique key is not duplicated after join, + * the joinrel will still has one row unless the jointype == JOIN_FULL. + */ + if (outer_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (outer_onerow) + { + /* + * The onerow outerrel becomes multi rows and multi_nullvals + * will be changed to true. We also need to set the exprs correctly since it + * can't be NIL any more. + */ + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, outerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Change multi_nullvals to true due to the full join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, true)); + else + /* Just reuse it */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + } + ctx->added_to_joinrel = true; + } + } + + if (innerrel_keeps_unique(root, outerrel, innerrel, clause_list, false)) + { + bool outer_impact = jointype == JOIN_FULL || jointype == JOIN_LEFT;; + + foreach(lc, innerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + if (inner_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (inner_onerow) + { + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, innerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Need to change multi_nullvals to true due to the outer join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, + true)); + else + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + + } + ctx->added_to_joinrel = true; + } + } + + /* + * The combination of the UniqueKey from both sides is unique as well regardless + * of join type, but no bother to add it if its subset has been added to joinrel + * already or it is not useful for the joinrel. + */ + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx1 = (UniqueKeyContext) lfirst(lc); + if (ctx1->added_to_joinrel || !ctx1->useful) + continue; + foreach(lc2, innerrel_ukey_ctx) + { + UniqueKeyContext ctx2 = (UniqueKeyContext) lfirst(lc2); + if (ctx2->added_to_joinrel || !ctx2->useful) + continue; + if (add_combined_uniquekey(root, joinrel, outerrel, innerrel, + ctx1->uniquekey, ctx2->uniquekey, + jointype)) + /* If we set a onerow UniqueKey to joinrel, we don't need other. */ + return; + } + } +} + + +/* + * convert_subquery_uniquekeys + * + * Covert the UniqueKey in subquery to outer relation. + */ +void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel) +{ + ListCell *lc; + + if (sub_final_rel->uniquekeys == NIL) + return; + + if (relation_is_onerow(sub_final_rel)) + { + add_uniquekey_for_onerow(currel); + return; + } + + Assert(currel->subroot != NULL); + + foreach(lc, sub_final_rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + ListCell *lc; + List *exprs = NIL; + bool ukey_useful = true; + + /* One row case is handled above */ + Assert(ukey->exprs != NIL); + foreach(lc, ukey->exprs) + { + Var *var; + TargetEntry *tle = tlist_member(lfirst(lc), + currel->subroot->processed_tlist); + if (tle == NULL) + { + ukey_useful = false; + break; + } + var = find_var_for_subquery_tle(currel, tle); + if (var == NULL) + { + ukey_useful = false; + break; + } + exprs = lappend(exprs, var); + } + + if (ukey_useful) + currel->uniquekeys = lappend(currel->uniquekeys, + makeUniqueKey(exprs, + ukey->multi_nullvals)); + + } +} + +/* + * innerrel_keeps_unique + * + * Check if Unique key of the innerrel is valid after join. innerrel's UniqueKey + * will be still valid if innerrel's any-column mergeop outrerel's uniquekey + * exists in clause_list. + * + * Note: the clause_list must be a list of mergeable restrictinfo already. + */ +static bool +innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *clause_list, + bool reverse) +{ + ListCell *lc, *lc2, *lc3; + + if (outerrel->uniquekeys == NIL || innerrel->uniquekeys == NIL) + return false; + + /* Check if there is outerrel's uniquekey in mergeable clause. */ + foreach(lc, outerrel->uniquekeys) + { + List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs; + bool clauselist_matchs_all_exprs = true; + foreach(lc2, outer_uq_exprs) + { + Node *outer_uq_expr = lfirst(lc2); + bool find_uq_expr_in_clauselist = false; + foreach(lc3, clause_list) + { + RestrictInfo *rinfo = lfirst_node(RestrictInfo, lc3); + Node *outer_expr; + if (reverse) + outer_expr = rinfo->outer_is_left ? get_rightop(rinfo->clause) : get_leftop(rinfo->clause); + else + outer_expr = rinfo->outer_is_left ? get_leftop(rinfo->clause) : get_rightop(rinfo->clause); + if (equal(outer_expr, outer_uq_expr)) + { + find_uq_expr_in_clauselist = true; + break; + } + } + if (!find_uq_expr_in_clauselist) + { + /* No need to check the next exprs in the current uniquekey */ + clauselist_matchs_all_exprs = false; + break; + } + } + + if (clauselist_matchs_all_exprs) + return true; + } + return false; +} + + +/* + * relation_is_onerow + * Check if it is a one-row relation by checking UniqueKey. + */ +bool +relation_is_onerow(RelOptInfo *rel) +{ + UniqueKey *ukey; + if (rel->uniquekeys == NIL) + return false; + ukey = linitial_node(UniqueKey, rel->uniquekeys); + return ukey->exprs == NIL && list_length(rel->uniquekeys) == 1; +} + +/* + * relation_has_uniquekeys_for + * Returns true if we have proofs that 'rel' cannot return multiple rows with + * the same values in each of 'exprs'. Otherwise returns false. + */ +bool +relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel, + List *exprs, bool allow_multinulls) +{ + ListCell *lc; + + /* + * For UniqueKey->onerow case, the uniquekey->exprs is empty as well + * so we can't rely on list_is_subset to handle this special cases + */ + if (exprs == NIL) + return false; + + foreach(lc, rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (ukey->multi_nullvals && !allow_multinulls) + continue; + if (list_is_subset(ukey->exprs, exprs)) + return true; + } + return false; +} + + +/* + * get_exprs_from_uniqueindex + * + * Return a list of exprs which is unique. set useful to false if this + * unique index is not useful for us. + */ +static List * +get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals) +{ + List *exprs = NIL; + ListCell *indexpr_item; + int c = 0; + + *useful = true; + *multi_nullvals = false; + + indexpr_item = list_head(unique_index->indexprs); + for(c = 0; c < unique_index->ncolumns; c++) + { + int attr = unique_index->indexkeys[c]; + Expr *expr; + bool matched_const = false; + ListCell *lc1, *lc2; + + if(attr > 0) + { + expr = list_nth_node(TargetEntry, unique_index->indextlist, c)->expr; + } + else if (attr == 0) + { + /* Expression index */ + expr = lfirst(indexpr_item); + indexpr_item = lnext(unique_index->indexprs, indexpr_item); + } + else /* attr < 0 */ + { + /* Index on system column is not supported */ + Assert(false); + } + + /* + * Check index_col = Const case with regarding to opfamily checking + * If we can remove the index_col from the final UniqueKey->exprs. + */ + forboth(lc1, const_exprs, lc2, const_expr_opfamilies) + { + if (list_member_oid((List *)lfirst(lc2), unique_index->opfamily[c]) + && match_index_to_operand((Node *) lfirst(lc1), c, unique_index)) + { + matched_const = true; + break; + } + } + + if (matched_const) + continue; + + /* Check if the indexed expr is used in rel */ + if (attr > 0) + { + /* + * Normal Indexed column, if the col is not used, then the index is useless + * for uniquekey. + */ + attr -= FirstLowInvalidHeapAttributeNumber; + + if (!bms_is_member(attr, used_varattrs)) + { + *useful = false; + break; + } + } + else if (!list_member(unique_index->rel->reltarget->exprs, expr)) + { + /* Expression index but the expression is not used in rel */ + *useful = false; + break; + } + + /* check not null property. */ + if (attr == 0) + { + /* We never know if a expression yields null or not */ + *multi_nullvals = true; + } + else if (!bms_is_member(attr, unique_index->rel->notnullattrs) + && !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber, + unique_index->rel->notnullattrs)) + { + *multi_nullvals = true; + } + + exprs = lappend(exprs, expr); + } + return exprs; +} + + +/* + * add_uniquekey_for_onerow + * If we are sure that the relation only returns one row, then all the columns + * are unique. However we don't need to create UniqueKey for every column, we + * just set exprs = NIL and overwrites all the other UniqueKey on this RelOptInfo + * since this one has strongest semantics. + */ +void +add_uniquekey_for_onerow(RelOptInfo *rel) +{ + /* + * We overwrite the previous UniqueKey on purpose since this one has the + * strongest semantic. + */ + rel->uniquekeys = list_make1(makeUniqueKey(NIL, false)); +} + + +/* + * initililze_uniquecontext_for_joinrel + * Return a List of UniqueKeyContext for an inputrel + */ +static List * +initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel) +{ + List *res = NIL; + ListCell *lc; + foreach(lc, inputrel->uniquekeys) + { + UniqueKeyContext context; + context = palloc(sizeof(struct UniqueKeyContextData)); + context->uniquekey = lfirst_node(UniqueKey, lc); + context->added_to_joinrel = false; + context->useful = true; + res = lappend(res, context); + } + return res; +} + + +/* + * get_exprs_from_uniquekey + * Unify the way of get List of exprs from a one-row UniqueKey or + * normal UniqueKey. for the onerow case, every expr in rel1 is a valid + * UniqueKey. Return a List of exprs. + * + * rel1: The relation which you want to get the exprs. + * ukey: The UniqueKey you want to get the exprs. + */ +static List * +get_exprs_from_uniquekey(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *rel1, UniqueKey *ukey) +{ + ListCell *lc; + bool onerow = rel1 != NULL && relation_is_onerow(rel1); + + List *res = NIL; + Assert(onerow || ukey); + if (onerow) + { + /* Only cares about the exprs still exist in joinrel */ + foreach(lc, joinrel->reltarget->exprs) + { + Bitmapset *relids = pull_varnos(root, lfirst(lc)); + if (bms_is_subset(relids, rel1->relids)) + { + res = lappend(res, list_make1(lfirst(lc))); + } + } + } + else + { + res = list_make1(ukey->exprs); + } + return res; +} + +/* + * Partitioned table Unique Keys. + * The partition table unique key is maintained as: + * 1. The index must be unique as usual. + * 2. The index must contains partition key. + * 3. The index must exist on all the child rel. see simple_indexinfo_equal for + * how we compare it. + */ + +/* + * index_constains_partkey + * return true if the index contains the partiton key. + */ +static bool +index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind) +{ + ListCell *lc; + int i; + Assert(IS_PARTITIONED_REL(partrel)); + Assert(partrel->part_scheme->partnatts > 0); + + for(i = 0; i < partrel->part_scheme->partnatts; i++) + { + Node *part_expr = linitial(partrel->partexprs[i]); + bool found_in_index = false; + foreach(lc, ind->indextlist) + { + Expr *index_expr = lfirst_node(TargetEntry, lc)->expr; + if (equal(index_expr, part_expr)) + { + found_in_index = true; + break; + } + } + if (!found_in_index) + return false; + } + return true; +} + +/* + * simple_indexinfo_equal + * + * Used to check if the 2 index is same as each other. The index here + * is COPIED from childrel and did some tiny changes(see + * simple_copy_indexinfo_to_parent) + */ +static bool +simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2) +{ + Size oid_cmp_len = sizeof(Oid) * ind1->ncolumns; + + return ind1->ncolumns == ind2->ncolumns && + ind1->unique == ind2->unique && + memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0 && + memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0 && + memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0 && + memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0 && + equal(get_tlist_exprs(ind1->indextlist, true), + get_tlist_exprs(ind2->indextlist, true)); +} + + +/* + * The below macros are used for simple_copy_indexinfo_to_parent which is so + * customized that I don't want to put it to copyfuncs.c. So copy it here. + */ +#define COPY_POINTER_FIELD(fldname, sz) \ + do { \ + Size _size = (sz); \ + newnode->fldname = palloc(_size); \ + memcpy(newnode->fldname, from->fldname, _size); \ + } while (0) + +#define COPY_NODE_FIELD(fldname) \ + (newnode->fldname = copyObjectImpl(from->fldname)) + +#define COPY_SCALAR_FIELD(fldname) \ + (newnode->fldname = from->fldname) + + +/* + * simple_copy_indexinfo_to_parent (from partition) + * Copy the IndexInfo from child relation to parent relation with some modification, + * which is used to test: + * 1. If the same index exists in all the childrels. + * 2. If the parentrel->reltarget/basicrestrict info matches this index. + */ +static IndexOptInfo * +simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from) +{ + IndexOptInfo *newnode = makeNode(IndexOptInfo); + AppendRelInfo *appinfo = find_appinfo_by_child(root, from->rel->relid); + ListCell *lc; + int idx = 0; + + COPY_SCALAR_FIELD(ncolumns); + COPY_SCALAR_FIELD(nkeycolumns); + COPY_SCALAR_FIELD(unique); + COPY_SCALAR_FIELD(immediate); + /* We just need to know if it is NIL or not */ + COPY_SCALAR_FIELD(indpred); + COPY_SCALAR_FIELD(predOK); + COPY_POINTER_FIELD(indexkeys, from->ncolumns * sizeof(int)); + COPY_POINTER_FIELD(indexcollations, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opfamily, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opcintype, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(sortopfamily, from->ncolumns * sizeof(Oid)); + COPY_NODE_FIELD(indextlist); + + /* Convert index exprs on child expr to expr on parent */ + foreach(lc, newnode->indextlist) + { + TargetEntry *tle = lfirst_node(TargetEntry, lc); + /* Index on expression is ignored */ + Assert(IsA(tle->expr, Var)); + tle->expr = (Expr *) find_parent_var(appinfo, (Var *) tle->expr); + newnode->indexkeys[idx] = castNode(Var, tle->expr)->varattno; + idx++; + } + newnode->rel = parentrel; + return newnode; +} + +/* + * adjust_partition_unique_indexlist + * + * global_unique_indexes: At the beginning, it contains the copy & modified + * unique index from the first partition. And then check if each index in it still + * exists in the following partitions. If no, remove it. at last, it has an + * index list which exists in all the partitions. + */ +static void +adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_indexes) +{ + ListCell *lc, *lc2; + foreach(lc, *global_unique_indexes) + { + IndexOptInfo *g_ind = lfirst_node(IndexOptInfo, lc); + bool found_in_child = false; + + foreach(lc2, childrel->indexlist) + { + IndexOptInfo *p_ind = lfirst_node(IndexOptInfo, lc2); + IndexOptInfo *p_ind_copy; + if (!p_ind->unique || !p_ind->immediate || + (p_ind->indpred != NIL && !p_ind->predOK)) + continue; + p_ind_copy = simple_copy_indexinfo_to_parent(root, parentrel, p_ind); + if (simple_indexinfo_equal(p_ind_copy, g_ind)) + { + found_in_child = true; + break; + } + } + if (!found_in_child) + /* The index doesn't exist in childrel, remove it from global_unique_indexes */ + *global_unique_indexes = foreach_delete_current(*global_unique_indexes, lc); + } +} + +/* Helper function for groupres/distinctrel */ +static void +add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgroups) +{ + Query *parse = root->parse; + List *exprs; + + /* + * XXX: If there are some vars which is not in current levelsup, the semantic is + * imprecise, should we avoid it or not? levelsup = 1 is just a demo, maybe we need to + * check every level other than 0, if so, looks we have to write another + * pull_var_walker. + */ + List *upper_vars = pull_vars_of_level((Node*)sortgroups, 1); + + if (upper_vars != NIL) + return; + + exprs = get_sortgrouplist_exprs(sortgroups, parse->targetList); + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(exprs, + false /* sortgroupclause can't be multi_nullvals */)); +} + + +/* + * add_combined_uniquekey + * The combination of both UniqueKeys is a valid UniqueKey for joinrel no matter + * the jointype. + */ +bool +add_combined_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype) +{ + + ListCell *lc1, *lc2; + + /* Either side has multi_nullvals or we have outer join, + * the combined UniqueKey has multi_nullvals */ + bool multi_nullvals = outer_ukey->multi_nullvals || + inner_ukey->multi_nullvals || IS_OUTER_JOIN(jointype); + + /* The only case we can get onerow joinrel after join */ + if (relation_is_onerow(outer_rel) + && relation_is_onerow(inner_rel) + && jointype == JOIN_INNER) + { + add_uniquekey_for_onerow(joinrel); + return true; + } + + foreach(lc1, get_exprs_from_uniquekey(root, joinrel, outer_rel, outer_ukey)) + { + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, inner_rel, inner_ukey)) + { + List *exprs = list_concat_copy(lfirst_node(List, lc1), lfirst_node(List, lc2)); + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(exprs, + multi_nullvals)); + } + } + return false; +} diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index 424d25cbd5..8d8e493f5c 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -2387,6 +2387,8 @@ grouping_planner(PlannerInfo *root, bool inheritance_update, add_path(final_rel, path); } + simple_copy_uniquekeys(current_rel, final_rel); + /* * Generate partial paths for final_rel, too, if outer query levels might * be able to make use of them. @@ -3899,6 +3901,8 @@ create_grouping_paths(PlannerInfo *root, } set_cheapest(grouped_rel); + + populate_grouprel_uniquekeys(root, grouped_rel, input_rel); return grouped_rel; } @@ -4621,7 +4625,7 @@ create_window_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(window_rel); - + simple_copy_uniquekeys(input_rel, window_rel); return window_rel; } @@ -4935,7 +4939,7 @@ create_distinct_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(distinct_rel); - + populate_distinctrel_uniquekeys(root, input_rel, distinct_rel); return distinct_rel; } @@ -5196,6 +5200,8 @@ create_ordered_paths(PlannerInfo *root, */ Assert(ordered_rel->pathlist != NIL); + simple_copy_uniquekeys(input_rel, ordered_rel); + return ordered_rel; } @@ -6073,6 +6079,9 @@ adjust_paths_for_srfs(PlannerInfo *root, RelOptInfo *rel, if (list_length(targets) == 1) return; + /* UniqueKey is not valid after handling the SRF. */ + rel->uniquekeys = NIL; + /* * Stack SRF-evaluation nodes atop each path for the rel. * diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c index becdcbb872..b7626545bf 100644 --- a/src/backend/optimizer/prep/prepunion.c +++ b/src/backend/optimizer/prep/prepunion.c @@ -689,6 +689,8 @@ generate_union_paths(SetOperationStmt *op, PlannerInfo *root, /* Undo effects of possibly forcing tuple_fraction to 0 */ root->tuple_fraction = save_fraction; + /* Add the UniqueKeys */ + populate_unionrel_uniquekeys(root, result_rel); return result_rel; } diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c index 86922a273c..6817c9c787 100644 --- a/src/backend/optimizer/util/appendinfo.c +++ b/src/backend/optimizer/util/appendinfo.c @@ -746,3 +746,47 @@ find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos) } return appinfos; } + +/* + * find_appinfo_by_child + * + */ +AppendRelInfo * +find_appinfo_by_child(PlannerInfo *root, Index child_index) +{ + ListCell *lc; + foreach(lc, root->append_rel_list) + { + AppendRelInfo *appinfo = lfirst_node(AppendRelInfo, lc); + if (appinfo->child_relid == child_index) + return appinfo; + } + elog(ERROR, "parent relation cant be found"); + return NULL; +} + +/* + * find_parent_var + * + */ +Var * +find_parent_var(AppendRelInfo *appinfo, Var *child_var) +{ + ListCell *lc; + Var *res = NULL; + Index attno = 1; + foreach(lc, appinfo->translated_vars) + { + Node *child_node = lfirst(lc); + if (equal(child_node, child_var)) + { + res = copyObject(child_var); + res->varattno = attno; + res->varno = appinfo->parent_relid; + } + attno++; + } + if (res == NULL) + elog(ERROR, "parent var can't be found."); + return res; +} diff --git a/src/backend/optimizer/util/inherit.c b/src/backend/optimizer/util/inherit.c index be1c9ddd96..3eec1f4d74 100644 --- a/src/backend/optimizer/util/inherit.c +++ b/src/backend/optimizer/util/inherit.c @@ -736,6 +736,7 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, { Node *onecq = (Node *) lfirst(lc2); bool pseudoconstant; + RestrictInfo *child_rinfo; /* check for pseudoconstant (no Vars or volatile functions) */ pseudoconstant = @@ -747,14 +748,15 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, root->hasPseudoConstantQuals = true; } /* reconstitute RestrictInfo with appropriate properties */ - childquals = lappend(childquals, - make_restrictinfo(root, - (Expr *) onecq, - rinfo->is_pushed_down, - rinfo->outerjoin_delayed, - pseudoconstant, - rinfo->security_level, - NULL, NULL, NULL)); + child_rinfo = make_restrictinfo(root, + (Expr *) onecq, + rinfo->is_pushed_down, + rinfo->outerjoin_delayed, + pseudoconstant, + rinfo->security_level, + NULL, NULL, NULL); + child_rinfo->mergeopfamilies = rinfo->mergeopfamilies; + childquals = lappend(childquals, child_rinfo); /* track minimum security level among child quals */ cq_min_security = Min(cq_min_security, rinfo->security_level); } diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 48a7ebfe45..4fe1824eb0 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -16,6 +16,7 @@ #include "nodes/execnodes.h" #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" extern A_Expr *makeA_Expr(A_Expr_Kind kind, List *name, @@ -106,4 +107,6 @@ extern GroupingSet *makeGroupingSet(GroupingSetKind kind, List *content, int loc extern VacuumRelation *makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols); +extern UniqueKey* makeUniqueKey(List *exprs, bool multi_nullvals); + #endif /* MAKEFUNC_H */ diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index e22df890ef..3320273ac1 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -265,6 +265,7 @@ typedef enum NodeTag T_EquivalenceMember, T_PathKey, T_PathTarget, + T_UniqueKey, T_RestrictInfo, T_IndexClause, T_PlaceHolderVar, diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index 0d61f04d27..80561b1003 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -739,6 +739,7 @@ typedef struct RelOptInfo QualCost baserestrictcost; /* cost of evaluating the above */ Index baserestrict_min_security; /* min security_level found in * baserestrictinfo */ + List *uniquekeys; /* List of UniqueKey */ List *joininfo; /* RestrictInfo structures for join clauses * involving this rel */ bool has_eclass_joins; /* T means joininfo is incomplete */ @@ -1059,6 +1060,28 @@ typedef struct PathKey } PathKey; +/* + * UniqueKey + * + * Represents the unique properties held by a RelOptInfo. + * + * exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL + * is a special case of UniqueKey, which means there is only 1 row in that + * relation. + * multi_nullvals: true means multi null values may exist in these exprs, so the + * uniqueness is not guaranteed in this case. This field is necessary for + * remove_useless_join & reduce_unique_semijoins where we don't mind these + * duplicated NULL values. It is set to true for 2 cases. One is a unique key + * from a unique index but the related column is nullable. The other one is for + * outer join. see populate_joinrel_uniquekeys for detail. + */ +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + /* * PathTarget * @@ -2496,7 +2519,7 @@ typedef enum * * flags indicating what kinds of grouping are possible. * partial_costs_set is true if the agg_partial_costs and agg_final_costs - * have been initialized. + * have been initialized. * agg_partial_costs gives partial aggregation costs. * agg_final_costs gives finalization costs. * target_parallel_safe is true if target is parallel safe. @@ -2526,8 +2549,8 @@ typedef struct * limit_tuples is an estimated bound on the number of output tuples, * or -1 if no LIMIT or couldn't estimate. * count_est and offset_est are the estimated values of the LIMIT and OFFSET - * expressions computed by preprocess_limit() (see comments for - * preprocess_limit() for more information). + * expressions computed by preprocess_limit() (see comments for + * preprocess_limit() for more information). */ typedef struct { diff --git a/src/include/nodes/pg_list.h b/src/include/nodes/pg_list.h index 404e03f132..85c65e910c 100644 --- a/src/include/nodes/pg_list.h +++ b/src/include/nodes/pg_list.h @@ -566,6 +566,16 @@ extern pg_nodiscard List *list_delete_first(List *list); extern pg_nodiscard List *list_delete_last(List *list); extern pg_nodiscard List *list_delete_nth_cell(List *list, int n); extern pg_nodiscard List *list_delete_cell(List *list, ListCell *cell); +extern bool list_is_subset(const List *members, const List *target); + +extern List *list_delete(List *list, void *datum); +extern List *list_delete_ptr(List *list, void *datum); +extern List *list_delete_int(List *list, int datum); +extern List *list_delete_oid(List *list, Oid datum); +extern List *list_delete_first(List *list); +extern List *list_delete_last(List *list); +extern List *list_delete_nth_cell(List *list, int n); +extern List *list_delete_cell(List *list, ListCell *cell); extern List *list_union(const List *list1, const List *list2); extern List *list_union_ptr(const List *list1, const List *list2); diff --git a/src/include/optimizer/appendinfo.h b/src/include/optimizer/appendinfo.h index 4cbf8c26cc..43b8d05f69 100644 --- a/src/include/optimizer/appendinfo.h +++ b/src/include/optimizer/appendinfo.h @@ -32,4 +32,7 @@ extern Relids adjust_child_relids_multilevel(PlannerInfo *root, Relids relids, extern AppendRelInfo **find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos); +extern AppendRelInfo *find_appinfo_by_child(PlannerInfo *root, Index child_index); +extern Var *find_parent_var(AppendRelInfo *appinfo, Var *child_var); + #endif /* APPENDINFO_H */ diff --git a/src/include/optimizer/optimizer.h b/src/include/optimizer/optimizer.h index d587952b7d..843aafa51e 100644 --- a/src/include/optimizer/optimizer.h +++ b/src/include/optimizer/optimizer.h @@ -23,6 +23,7 @@ #define OPTIMIZER_H #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" /* Test if an expression node represents a SRF call. Beware multiple eval! */ #define IS_SRF_CALL(node) \ @@ -168,6 +169,7 @@ extern TargetEntry *get_sortgroupref_tle(Index sortref, List *targetList); extern TargetEntry *get_sortgroupclause_tle(SortGroupClause *sgClause, List *targetList); +extern Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); extern Node *get_sortgroupclause_expr(SortGroupClause *sgClause, List *targetList); extern List *get_sortgrouplist_exprs(List *sgClauses, diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h index 035d3e1206..1adf99c9ee 100644 --- a/src/include/optimizer/paths.h +++ b/src/include/optimizer/paths.h @@ -247,5 +247,48 @@ extern PathKey *make_canonical_pathkey(PlannerInfo *root, int strategy, bool nulls_first); extern void add_paths_to_append_rel(PlannerInfo *root, RelOptInfo *rel, List *live_childrels); +extern List *select_mergejoin_clauses(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + JoinType jointype, + bool *mergejoin_allowed); + +/* + * uniquekeys.c + * Utilities for matching and building unique keys + */ +extern void populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List* unique_index_list); +extern void populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels); +extern void populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel); +extern void populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel); +extern void populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel); +extern void simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel); +extern void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel); +extern void populate_joinrel_uniquekeys(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + RelOptInfo *rel2, + List *restrictlist, + JoinType jointype); + +extern bool relation_has_uniquekeys_for(PlannerInfo *root, + RelOptInfo *rel, + List *exprs, + bool allow_multinulls); +extern bool relation_is_onerow(RelOptInfo *rel); #endif /* PATHS_H */ -- 2.26.2 --yh7cw345mbttadjz Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v38-0003-Extend-UniqueKeys.patch" ^ permalink raw reply [nested|flat] 91+ messages in thread
* [PATCH v36 2/6] Introduce UniqueKey attributes on RelOptInfo struct. @ 2020-05-11 07:50 一挃 <[email protected]> 0 siblings, 0 replies; 91+ messages in thread From: 一挃 @ 2020-05-11 07:50 UTC (permalink / raw) UniqueKey is a set of exprs on RelOptInfo which represents the exprs will be unique on the given RelOptInfo. You can see README.uniquekey for more information. --- src/backend/nodes/copyfuncs.c | 13 + src/backend/nodes/list.c | 31 + src/backend/nodes/makefuncs.c | 13 + src/backend/nodes/outfuncs.c | 11 + src/backend/nodes/readfuncs.c | 10 + src/backend/optimizer/path/Makefile | 3 +- src/backend/optimizer/path/README.uniquekey | 131 +++ src/backend/optimizer/path/allpaths.c | 10 + src/backend/optimizer/path/joinpath.c | 9 +- src/backend/optimizer/path/joinrels.c | 2 + src/backend/optimizer/path/pathkeys.c | 3 +- src/backend/optimizer/path/uniquekeys.c | 1131 +++++++++++++++++++ src/backend/optimizer/plan/planner.c | 13 +- src/backend/optimizer/prep/prepunion.c | 2 + src/backend/optimizer/util/appendinfo.c | 44 + src/backend/optimizer/util/inherit.c | 16 +- src/include/nodes/makefuncs.h | 3 + src/include/nodes/nodes.h | 1 + src/include/nodes/pathnodes.h | 29 +- src/include/nodes/pg_list.h | 2 + src/include/optimizer/appendinfo.h | 3 + src/include/optimizer/optimizer.h | 2 + src/include/optimizer/paths.h | 43 + 23 files changed, 1502 insertions(+), 23 deletions(-) create mode 100644 src/backend/optimizer/path/README.uniquekey create mode 100644 src/backend/optimizer/path/uniquekeys.c diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 89c409de66..1f50400fd2 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2273,6 +2273,16 @@ _copyPathKey(const PathKey *from) return newnode; } +static UniqueKey * +_copyUniqueKey(const UniqueKey *from) +{ + UniqueKey *newnode = makeNode(UniqueKey); + + COPY_NODE_FIELD(exprs); + COPY_SCALAR_FIELD(multi_nullvals); + + return newnode; +} /* * _copyRestrictInfo */ @@ -5152,6 +5162,9 @@ copyObjectImpl(const void *from) case T_PathKey: retval = _copyPathKey(from); break; + case T_UniqueKey: + retval = _copyUniqueKey(from); + break; case T_RestrictInfo: retval = _copyRestrictInfo(from); break; diff --git a/src/backend/nodes/list.c b/src/backend/nodes/list.c index 80fa8c84e4..a7a99b70f2 100644 --- a/src/backend/nodes/list.c +++ b/src/backend/nodes/list.c @@ -687,6 +687,37 @@ list_member_oid(const List *list, Oid datum) return false; } +/* + * return true iff every entry in "members" list is also present + * in the "target" list. + */ +bool +list_is_subset(const List *members, const List *target) +{ + const ListCell *lc1, *lc2; + + Assert(IsPointerList(members)); + Assert(IsPointerList(target)); + check_list_invariants(members); + check_list_invariants(target); + + foreach(lc1, members) + { + bool found = false; + foreach(lc2, target) + { + if (equal(lfirst(lc1), lfirst(lc2))) + { + found = true; + break; + } + } + if (!found) + return false; + } + return true; +} + /* * Delete the n'th cell (counting from 0) in list. * diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 49de285f01..646cf7c9a1 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -814,3 +814,16 @@ makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols) v->va_cols = va_cols; return v; } + + +/* + * makeUniqueKey + */ +UniqueKey* +makeUniqueKey(List *exprs, bool multi_nullvals) +{ + UniqueKey * ukey = makeNode(UniqueKey); + ukey->exprs = exprs; + ukey->multi_nullvals = multi_nullvals; + return ukey; +} diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index e2f177515d..c3a9632992 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2428,6 +2428,14 @@ _outPathKey(StringInfo str, const PathKey *node) WRITE_BOOL_FIELD(pk_nulls_first); } +static void +_outUniqueKey(StringInfo str, const UniqueKey *node) +{ + WRITE_NODE_TYPE("UNIQUEKEY"); + WRITE_NODE_FIELD(exprs); + WRITE_BOOL_FIELD(multi_nullvals); +} + static void _outPathTarget(StringInfo str, const PathTarget *node) { @@ -4127,6 +4135,9 @@ outNode(StringInfo str, const void *obj) case T_PathKey: _outPathKey(str, obj); break; + case T_UniqueKey: + _outUniqueKey(str, obj); + break; case T_PathTarget: _outPathTarget(str, obj); break; diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index 42050ab719..3a18571d0c 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -452,6 +452,14 @@ _readSetOperationStmt(void) READ_DONE(); } +static UniqueKey * +_readUniqueKey(void) +{ + READ_LOCALS(UniqueKey); + READ_NODE_FIELD(exprs); + READ_BOOL_FIELD(multi_nullvals); + READ_DONE(); +} /* * Stuff from primnodes.h. @@ -2656,6 +2664,8 @@ parseNodeString(void) return_value = _readCommonTableExpr(); else if (MATCH("SETOPERATIONSTMT", 16)) return_value = _readSetOperationStmt(); + else if (MATCH("UNIQUEKEY", 9)) + return_value = _readUniqueKey(); else if (MATCH("ALIAS", 5)) return_value = _readAlias(); else if (MATCH("RANGEVAR", 8)) diff --git a/src/backend/optimizer/path/Makefile b/src/backend/optimizer/path/Makefile index 1e199ff66f..7b9820c25f 100644 --- a/src/backend/optimizer/path/Makefile +++ b/src/backend/optimizer/path/Makefile @@ -21,6 +21,7 @@ OBJS = \ joinpath.o \ joinrels.o \ pathkeys.o \ - tidpath.o + tidpath.o \ + uniquekeys.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/optimizer/path/README.uniquekey b/src/backend/optimizer/path/README.uniquekey new file mode 100644 index 0000000000..5eac761995 --- /dev/null +++ b/src/backend/optimizer/path/README.uniquekey @@ -0,0 +1,131 @@ +1. What is UniqueKey? +We can think UniqueKey is a set of exprs for a RelOptInfo, which we are insure +that doesn't yields same result among all the rows. The simplest UniqueKey +format is primary key. + +However we define the UnqiueKey as below. + +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + +exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL +is a special case of UniqueKey, which means there is only one row in that +relation.it has a stronger semantic than others. like SELECT uk FROM t; uk is +normal unique key and may have different values. SELECT colx FROM t WHERE uk = +const. colx is unique AND we have only 1 value. This field can used for +innerrel_is_unique. this logic is handled specially in add_uniquekey_for_onerow +function. + +multi_nullvals: true means multi null values may exist in these exprs, so the +uniqueness is not guaranteed in this case. This field is necessary for +remove_useless_join & reduce_unique_semijoins where we don't mind these +duplicated NULL values. It is set to true for 2 cases. One is a unique key +from a unique index but the related column is nullable. The other one is for +outer join. see populate_joinrel_uniquekeys for detail. + + +The UniqueKey can be used at the following cases at least: +1. remove_useless_joins. +2. reduce_semianti_joins +3. remove distinct node if distinct clause is unique. +4. remove aggnode if group by clause is unique. +5. Index Skip Scan (WIP) +6. Aggregation Push Down without 2 phase aggregation if the join can't + duplicated the aggregated rows. (work in progress feature) + +2. How is it maintained? + +We have a set of populate_xxx_unqiuekeys functions to maintain the uniquekey on +various cases. xxx includes baserel, joinrel, partitionedrel, distinctrel, +groupedrel, unionrel. and we also need to convert the uniquekey from subquery +to outer relation, which is what convert_subquery_uniquekeys does. + +1. The first part is about baserel. We handled 3 cases. suppose we have Unique +Index on (a, b). + +1. SELECT a, b FROM t. UniqueKey (a, b) +2. SELECT a FROM t WHERE b = 1; UniqueKey (a) +3. SELECT .. FROM t WHERE a = 1 AND b = 1; UniqueKey (NIL). onerow case, every + column is Unique. + +2. The next part is joinrel, this part is most error-prone, we simplified the rules +like below: +1. If the relation's UniqueKey can't be duplicated after join, then is will be + still valid for the join rel. The function we used here is + innerrel_keeps_unique. The basic idea is innerrel.any_col = outer.uk. + +2. If the UnqiueKey can't keep valid via the rule 1, the combination of the + UniqueKey from both sides are valid for sure. We can prove this as: if the + unique exprs from rel1 is duplicated by rel2, the duplicated rows must + contains different unique exprs from rel2. + +More considerations about onerow: +1. If relation with one row and it can't be duplicated, it is still possible + contains mulit_nullvas after outer join. +2. If the either UniqueKey can be duplicated after join, the can get one row + only when both side is one row AND there is no outer join. +3. Whenever the onerow UniqueKey is not a valid any more, we need to convert one + row UniqueKey to normal unique key since we don't store exprs for one-row + relation. get_exprs_from_uniquekeys will be used here. + + +More considerations about multi_nullvals after join: +1. If the original UnqiueKey has multi_nullvals, the final UniqueKey will have + mulit_nullvals in any case. +2. If a unique key doesn't allow mulit_nullvals, after some outer join, it + allows some outer join. + + +3. When we comes to subquery, we need to convert_subquery_unqiuekeys just like +convert_subquery_pathkeys. Only the UniqueKey insides subquery is referenced as +a Var in outer relation will be reused. The relationship between the outerrel.Var +and subquery.exprs is built with outerel->subroot->processed_tlist. + + +4. As for the SRF functions, it will break the uniqueness of uniquekey, However it +is handled in adjust_paths_for_srfs, which happens after the query_planner. so +we will maintain the UniqueKey until there and reset it to NIL at that +places. This can't help on distinct/group by elimination cases but probably help +in some other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is +semantic correctly. + + +5. As for inherit table, we first main the UnqiueKey on childrel as well. But for +partitioned table we need to maintain 2 different kinds of +UnqiueKey. 1). UniqueKey on the parent relation 2). UniqueKey on child +relation for partition wise query. + +Example: +CREATE TABLE p (a int not null, b int not null) partition by list (a); +CREATE TABLE p0 partition of p for values in (1); +CREATE TABLE p1 partition of p for values in (2); + +create unique index p0_b on p0(b); +create unique index p1_b on p1(b); + +Now b is only unique on partition level, so the distinct can't be removed on +the following cases. SELECT DISTINCT b FROM p; + +Another example is SELECT DISTINCT a, b FROM p WHERE a = 1; Since only one +partition is chosen, the UniqueKey on child relation is same as the UniqueKey on +parent relation. + +Another usage of UniqueKey on partition level is it be helpful for +partition-wise join. + +As for the UniqueKey on parent table level, it comes with 2 different ways, +1). the UniqueKey is also derived in UniqueKey index, but the index must be same +in all the related children relations and the unique index must contains +Partition Key in it. Example: + +CREATE UNIQUE INDEX p_ab ON p(a, b); -- where a is the partition key. + +-- Query +SELECT a, b FROM p; the (a, b) is a UniqueKey of p. + +2). If the parent relation has only one childrel, the UniqueKey on childrel is + the UniqueKey on parent as well. diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index 484dab0a1a..2ad9d06d7a 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -579,6 +579,12 @@ set_plain_rel_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) */ check_index_predicates(root, rel); + /* + * Now that we've marked which partial indexes are suitable, we can now + * build the relation's unique keys. + */ + populate_baserel_uniquekeys(root, rel, rel->indexlist); + /* Mark rel with estimated output rows, width, etc */ set_baserel_size_estimates(root, rel); } @@ -1310,6 +1316,8 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, /* Add paths to the append relation. */ add_paths_to_append_rel(root, rel, live_childrels); + if (IS_PARTITIONED_REL(rel)) + populate_partitionedrel_uniquekeys(root, rel, live_childrels); } @@ -2383,6 +2391,8 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, pathkeys, required_outer)); } + convert_subquery_uniquekeys(root, rel, sub_final_rel); + /* If outer rel allows parallelism, do same for partial paths. */ if (rel->consider_parallel && bms_is_empty(required_outer)) { diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c index db54a6ba2e..ef0fd2fb0b 100644 --- a/src/backend/optimizer/path/joinpath.c +++ b/src/backend/optimizer/path/joinpath.c @@ -71,13 +71,6 @@ static void consider_parallel_mergejoin(PlannerInfo *root, static void hash_inner_and_outer(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, RelOptInfo *innerrel, JoinType jointype, JoinPathExtraData *extra); -static List *select_mergejoin_clauses(PlannerInfo *root, - RelOptInfo *joinrel, - RelOptInfo *outerrel, - RelOptInfo *innerrel, - List *restrictlist, - JoinType jointype, - bool *mergejoin_allowed); static void generate_mergejoin_paths(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *innerrel, @@ -1927,7 +1920,7 @@ hash_inner_and_outer(PlannerInfo *root, * if it is mergejoinable and involves vars from the two sub-relations * currently of interest. */ -static List * +List * select_mergejoin_clauses(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c index 2d343cd293..b9163ee8ff 100644 --- a/src/backend/optimizer/path/joinrels.c +++ b/src/backend/optimizer/path/joinrels.c @@ -924,6 +924,8 @@ populate_joinrel_with_paths(PlannerInfo *root, RelOptInfo *rel1, /* Apply partitionwise join technique, if possible. */ try_partitionwise_join(root, rel1, rel2, joinrel, sjinfo, restrictlist); + + populate_joinrel_uniquekeys(root, joinrel, rel1, rel2, restrictlist, sjinfo->jointype); } diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index ce9bf87e9b..7e596d4194 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -33,7 +33,6 @@ static bool pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys); static bool matches_boolean_partition_clause(RestrictInfo *rinfo, RelOptInfo *partrel, int partkeycol); -static Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); static bool right_merge_direction(PlannerInfo *root, PathKey *pathkey); @@ -1035,7 +1034,7 @@ convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel, * We need this to ensure that we don't return pathkeys describing values * that are unavailable above the level of the subquery scan. */ -static Var * +Var * find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle) { ListCell *lc; diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c new file mode 100644 index 0000000000..b33bcd2f32 --- /dev/null +++ b/src/backend/optimizer/path/uniquekeys.c @@ -0,0 +1,1131 @@ +/*------------------------------------------------------------------------- + * + * uniquekeys.c + * Utilities for matching and building unique keys + * + * Portions Copyright (c) 2020, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/optimizer/path/uniquekeys.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "nodes/makefuncs.h" +#include "nodes/nodeFuncs.h" +#include "optimizer/pathnode.h" +#include "optimizer/paths.h" +#include "optimizer/appendinfo.h" +#include "optimizer/optimizer.h" +#include "optimizer/tlist.h" +#include "rewrite/rewriteManip.h" + + +/* + * This struct is used to help populate_joinrel_uniquekeys. + * + * added_to_joinrel is true if a uniquekey (from outerrel or innerrel) + * has been added to joinrel. + * useful is true if the exprs of the uniquekey still appears in joinrel. + */ +typedef struct UniqueKeyContextData +{ + UniqueKey *uniquekey; + bool added_to_joinrel; + bool useful; +} *UniqueKeyContext; + +static List *initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel); +static bool innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + bool reverse); + +static List *get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals); +static List *get_exprs_from_uniquekey(RelOptInfo *joinrel, + RelOptInfo *rel1, + UniqueKey *ukey); +static void add_uniquekey_for_onerow(RelOptInfo *rel); +static bool add_combined_uniquekey(RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype); + +/* Used for unique indexes checking for partitioned table */ +static bool index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind); +static IndexOptInfo *simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from); +static bool simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2); +static void adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_index); + +/* Helper function for grouped relation and distinct relation. */ +static void add_uniquekey_from_sortgroups(PlannerInfo *root, + RelOptInfo *rel, + List *sortgroups); + +/* + * populate_baserel_uniquekeys + * Populate 'baserel' uniquekeys list by looking at the rel's unique index + * and baserestrictinfo + */ +void +populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List *indexlist) +{ + ListCell *lc; + List *matched_uniq_indexes = NIL; + + /* Attrs appears in rel->reltarget->exprs. */ + Bitmapset *used_attrs = NULL; + + List *const_exprs = NIL; + List *expr_opfamilies = NIL; + + Assert(baserel->rtekind == RTE_RELATION); + + foreach(lc, indexlist) + { + IndexOptInfo *ind = (IndexOptInfo *) lfirst(lc); + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + matched_uniq_indexes = lappend(matched_uniq_indexes, ind); + } + + if (matched_uniq_indexes == NIL) + return; + + /* Check which attrs is used in baserel->reltarget */ + pull_varattnos((Node *)baserel->reltarget->exprs, baserel->relid, &used_attrs); + + /* Check which attrno is used at a mergeable const filter */ + foreach(lc, baserel->baserestrictinfo) + { + RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); + + if (rinfo->mergeopfamilies == NIL) + continue; + + if (bms_is_empty(rinfo->left_relids)) + { + const_exprs = lappend(const_exprs, get_rightop(rinfo->clause)); + } + else if (bms_is_empty(rinfo->right_relids)) + { + const_exprs = lappend(const_exprs, get_leftop(rinfo->clause)); + } + else + continue; + + expr_opfamilies = lappend(expr_opfamilies, rinfo->mergeopfamilies); + } + + foreach(lc, matched_uniq_indexes) + { + bool multi_nullvals, useful; + List *exprs = get_exprs_from_uniqueindex(lfirst_node(IndexOptInfo, lc), + const_exprs, + expr_opfamilies, + used_attrs, + &useful, + &multi_nullvals); + if (useful) + { + if (exprs == NIL) + { + /* All the columns in Unique Index matched with a restrictinfo */ + add_uniquekey_for_onerow(baserel); + return; + } + baserel->uniquekeys = lappend(baserel->uniquekeys, + makeUniqueKey(exprs, multi_nullvals)); + } + } +} + + +/* + * populate_partitionedrel_uniquekeys + * The UniqueKey on partitionrel comes from 2 cases: + * 1). Only one partition is involved in this query, the unique key can be + * copied to parent rel from childrel. + * 2). There are some unique index which includes partition key and exists + * in all the related partitions. + * We never mind rule 2 if we hit rule 1. + */ + +void +populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels) +{ + ListCell *lc; + List *global_uniq_indexlist = NIL; + RelOptInfo *childrel; + bool is_first = true; + + Assert(IS_PARTITIONED_REL(rel)); + + if (childrels == NIL) + return; + + /* + * If there is only one partition used in this query, the UniqueKey in childrel is + * still valid in parent level, but we need convert the format from child expr to + * parent expr. + */ + if (list_length(childrels) == 1) + { + /* Check for Rule 1 */ + RelOptInfo *childrel = linitial_node(RelOptInfo, childrels); + ListCell *lc; + Assert(childrel->reloptkind == RELOPT_OTHER_MEMBER_REL); + if (relation_is_onerow(childrel)) + { + add_uniquekey_for_onerow(rel); + return; + } + + foreach(lc, childrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + AppendRelInfo *appinfo = find_appinfo_by_child(root, childrel->relid); + List *parent_exprs = NIL; + bool can_reuse = true; + ListCell *lc2; + foreach(lc2, ukey->exprs) + { + Var *var = (Var *)lfirst(lc2); + /* + * If the expr comes from a expression, it is hard to build the expression + * in parent so ignore that case for now. + */ + if(!IsA(var, Var)) + { + can_reuse = false; + break; + } + /* Convert it to parent var */ + parent_exprs = lappend(parent_exprs, find_parent_var(appinfo, var)); + } + if (can_reuse) + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(parent_exprs, + ukey->multi_nullvals)); + } + } + else + { + /* Check for rule 2 */ + childrel = linitial_node(RelOptInfo, childrels); + foreach(lc, childrel->indexlist) + { + IndexOptInfo *ind = lfirst(lc); + IndexOptInfo *modified_index; + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + + /* + * During simple_copy_indexinfo_to_parent, we need to convert var from + * child var to parent var, index on expression is too complex to handle. + * so ignore it for now. + */ + if (ind->indexprs != NIL) + continue; + + modified_index = simple_copy_indexinfo_to_parent(root, rel, ind); + /* + * If the unique index doesn't contain partkey, then it is unique + * on this partition only, so it is useless for us. + */ + if (!index_constains_partkey(rel, modified_index)) + continue; + + global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index); + } + + if (global_uniq_indexlist != NIL) + { + foreach(lc, childrels) + { + RelOptInfo *child = lfirst(lc); + if (is_first) + { + is_first = false; + continue; + } + adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist); + } + /* Now we have a list of unique index which are exactly same on all childrels, + * Set the UniqueKey just like it is non-partition table + */ + populate_baserel_uniquekeys(root, rel, global_uniq_indexlist); + } + } +} + + +/* + * populate_distinctrel_uniquekeys + */ +void +populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel) +{ + /* The unique key before the distinct is still valid. */ + distinctrel->uniquekeys = list_copy(inputrel->uniquekeys); + add_uniquekey_from_sortgroups(root, distinctrel, root->parse->distinctClause); +} + +/* + * populate_grouprel_uniquekeys + */ +void +populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel) + +{ + Query *parse = root->parse; + bool input_ukey_added = false; + ListCell *lc; + + if (relation_is_onerow(inputrel)) + { + add_uniquekey_for_onerow(grouprel); + return; + } + if (parse->groupingSets) + return; + + /* A Normal group by without grouping set. */ + if (parse->groupClause) + { + /* + * Current even the groupby clause is Unique already, but if query has aggref + * We have to create grouprel still. To keep the UnqiueKey short, we will check + * the UniqueKey of input_rel still valid, if so we reuse it. + */ + foreach(lc, inputrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (list_is_subset(ukey->exprs, grouprel->reltarget->exprs)) + { + grouprel->uniquekeys = lappend(grouprel->uniquekeys, + ukey); + input_ukey_added = true; + } + } + if (!input_ukey_added) + /* + * group by clause must be a super-set of grouprel->reltarget->exprs except the + * aggregation expr, so if such exprs is unique already, no bother to generate + * new uniquekey for group by exprs. + */ + add_uniquekey_from_sortgroups(root, + grouprel, + root->parse->groupClause); + } + else + /* It has aggregation but without a group by, so only one row returned */ + add_uniquekey_for_onerow(grouprel); +} + +/* + * simple_copy_uniquekeys + * Using a function for the one-line code makes us easy to check where we simply + * copied the uniquekey. + */ +void +simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel) +{ + newrel->uniquekeys = oldrel->uniquekeys; +} + +/* + * populate_unionrel_uniquekeys + */ +void +populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel) +{ + ListCell *lc; + List *exprs = NIL; + + Assert(unionrel->uniquekeys == NIL); + + foreach(lc, unionrel->reltarget->exprs) + { + exprs = lappend(exprs, lfirst(lc)); + } + + if (exprs == NIL) + /* SQL: select union select; is valid, we need to handle it here. */ + add_uniquekey_for_onerow(unionrel); + else + unionrel->uniquekeys = lappend(unionrel->uniquekeys, + makeUniqueKey(exprs,false)); + +} + +/* + * populate_joinrel_uniquekeys + * + * populate uniquekeys for joinrel. We will check each relation to see if its + * UniqueKey is still valid via innerrel_keeps_unique, if so, we add it to + * joinrel. The multi_nullvals field will be changed to true for some outer + * join cases and one-row UniqueKey needs to be converted to normal UniqueKey + * for the same case as well. + * For the uniquekey in either baserel which can't be unique after join, we still + * check to see if combination of UniqueKeys from both side is still useful for us. + * if yes, we add it to joinrel as well. + */ +void +populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *outerrel, RelOptInfo *innerrel, + List *restrictlist, JoinType jointype) +{ + ListCell *lc, *lc2; + List *clause_list = NIL; + List *outerrel_ukey_ctx; + List *innerrel_ukey_ctx; + bool inner_onerow, outer_onerow; + bool mergejoin_allowed; + + /* Care about the outerrel relation only for SEMI/ANTI join */ + if (jointype == JOIN_SEMI || jointype == JOIN_ANTI) + { + foreach(lc, outerrel->uniquekeys) + { + UniqueKey *uniquekey = lfirst_node(UniqueKey, lc); + if (list_is_subset(uniquekey->exprs, joinrel->reltarget->exprs)) + joinrel->uniquekeys = lappend(joinrel->uniquekeys, uniquekey); + } + return; + } + + Assert(jointype == JOIN_LEFT || jointype == JOIN_FULL || jointype == JOIN_INNER); + + /* Fast path */ + if (innerrel->uniquekeys == NIL || outerrel->uniquekeys == NIL) + return; + + inner_onerow = relation_is_onerow(innerrel); + outer_onerow = relation_is_onerow(outerrel); + + outerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(outerrel); + innerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(innerrel); + + clause_list = select_mergejoin_clauses(root, joinrel, outerrel, innerrel, + restrictlist, jointype, + &mergejoin_allowed); + + if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true /* reverse */)) + { + bool outer_impact = jointype == JOIN_FULL; + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + /* Outer relation has one row, and the unique key is not duplicated after join, + * the joinrel will still has one row unless the jointype == JOIN_FULL. + */ + if (outer_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (outer_onerow) + { + /* + * The onerow outerrel becomes multi rows and multi_nullvals + * will be changed to true. We also need to set the exprs correctly since it + * can't be NIL any more. + */ + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(joinrel, outerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Change multi_nullvals to true due to the full join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, true)); + else + /* Just reuse it */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + } + ctx->added_to_joinrel = true; + } + } + + if (innerrel_keeps_unique(root, outerrel, innerrel, clause_list, false)) + { + bool outer_impact = jointype == JOIN_FULL || jointype == JOIN_LEFT;; + + foreach(lc, innerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + if (inner_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (inner_onerow) + { + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(joinrel, innerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Need to change multi_nullvals to true due to the outer join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, + true)); + else + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + + } + ctx->added_to_joinrel = true; + } + } + + /* + * The combination of the UniqueKey from both sides is unique as well regardless + * of join type, but no bother to add it if its subset has been added to joinrel + * already or it is not useful for the joinrel. + */ + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx1 = (UniqueKeyContext) lfirst(lc); + if (ctx1->added_to_joinrel || !ctx1->useful) + continue; + foreach(lc2, innerrel_ukey_ctx) + { + UniqueKeyContext ctx2 = (UniqueKeyContext) lfirst(lc2); + if (ctx2->added_to_joinrel || !ctx2->useful) + continue; + if (add_combined_uniquekey(joinrel, outerrel, innerrel, + ctx1->uniquekey, ctx2->uniquekey, + jointype)) + /* If we set a onerow UniqueKey to joinrel, we don't need other. */ + return; + } + } +} + + +/* + * convert_subquery_uniquekeys + * + * Covert the UniqueKey in subquery to outer relation. + */ +void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel) +{ + ListCell *lc; + + if (sub_final_rel->uniquekeys == NIL) + return; + + if (relation_is_onerow(sub_final_rel)) + { + add_uniquekey_for_onerow(currel); + return; + } + + Assert(currel->subroot != NULL); + + foreach(lc, sub_final_rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + ListCell *lc; + List *exprs = NIL; + bool ukey_useful = true; + + /* One row case is handled above */ + Assert(ukey->exprs != NIL); + foreach(lc, ukey->exprs) + { + Var *var; + TargetEntry *tle = tlist_member(lfirst(lc), + currel->subroot->processed_tlist); + if (tle == NULL) + { + ukey_useful = false; + break; + } + var = find_var_for_subquery_tle(currel, tle); + if (var == NULL) + { + ukey_useful = false; + break; + } + exprs = lappend(exprs, var); + } + + if (ukey_useful) + currel->uniquekeys = lappend(currel->uniquekeys, + makeUniqueKey(exprs, + ukey->multi_nullvals)); + + } +} + +/* + * innerrel_keeps_unique + * + * Check if Unique key of the innerrel is valid after join. innerrel's UniqueKey + * will be still valid if innerrel's any-column mergeop outrerel's uniquekey + * exists in clause_list. + * + * Note: the clause_list must be a list of mergeable restrictinfo already. + */ +static bool +innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *clause_list, + bool reverse) +{ + ListCell *lc, *lc2, *lc3; + + if (outerrel->uniquekeys == NIL || innerrel->uniquekeys == NIL) + return false; + + /* Check if there is outerrel's uniquekey in mergeable clause. */ + foreach(lc, outerrel->uniquekeys) + { + List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs; + bool clauselist_matchs_all_exprs = true; + foreach(lc2, outer_uq_exprs) + { + Node *outer_uq_expr = lfirst(lc2); + bool find_uq_expr_in_clauselist = false; + foreach(lc3, clause_list) + { + RestrictInfo *rinfo = lfirst_node(RestrictInfo, lc3); + Node *outer_expr; + if (reverse) + outer_expr = rinfo->outer_is_left ? get_rightop(rinfo->clause) : get_leftop(rinfo->clause); + else + outer_expr = rinfo->outer_is_left ? get_leftop(rinfo->clause) : get_rightop(rinfo->clause); + if (equal(outer_expr, outer_uq_expr)) + { + find_uq_expr_in_clauselist = true; + break; + } + } + if (!find_uq_expr_in_clauselist) + { + /* No need to check the next exprs in the current uniquekey */ + clauselist_matchs_all_exprs = false; + break; + } + } + + if (clauselist_matchs_all_exprs) + return true; + } + return false; +} + + +/* + * relation_is_onerow + * Check if it is a one-row relation by checking UniqueKey. + */ +bool +relation_is_onerow(RelOptInfo *rel) +{ + UniqueKey *ukey; + if (rel->uniquekeys == NIL) + return false; + ukey = linitial_node(UniqueKey, rel->uniquekeys); + return ukey->exprs == NIL && list_length(rel->uniquekeys) == 1; +} + +/* + * relation_has_uniquekeys_for + * Returns true if we have proofs that 'rel' cannot return multiple rows with + * the same values in each of 'exprs'. Otherwise returns false. + */ +bool +relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel, + List *exprs, bool allow_multinulls) +{ + ListCell *lc; + + /* + * For UniqueKey->onerow case, the uniquekey->exprs is empty as well + * so we can't rely on list_is_subset to handle this special cases + */ + if (exprs == NIL) + return false; + + foreach(lc, rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (ukey->multi_nullvals && !allow_multinulls) + continue; + if (list_is_subset(ukey->exprs, exprs)) + return true; + } + return false; +} + + +/* + * get_exprs_from_uniqueindex + * + * Return a list of exprs which is unique. set useful to false if this + * unique index is not useful for us. + */ +static List * +get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals) +{ + List *exprs = NIL; + ListCell *indexpr_item; + int c = 0; + + *useful = true; + *multi_nullvals = false; + + indexpr_item = list_head(unique_index->indexprs); + for(c = 0; c < unique_index->ncolumns; c++) + { + int attr = unique_index->indexkeys[c]; + Expr *expr; + bool matched_const = false; + ListCell *lc1, *lc2; + + if(attr > 0) + { + expr = list_nth_node(TargetEntry, unique_index->indextlist, c)->expr; + } + else if (attr == 0) + { + /* Expression index */ + expr = lfirst(indexpr_item); + indexpr_item = lnext(unique_index->indexprs, indexpr_item); + } + else /* attr < 0 */ + { + /* Index on system column is not supported */ + Assert(false); + } + + /* + * Check index_col = Const case with regarding to opfamily checking + * If we can remove the index_col from the final UniqueKey->exprs. + */ + forboth(lc1, const_exprs, lc2, const_expr_opfamilies) + { + if (list_member_oid((List *)lfirst(lc2), unique_index->opfamily[c]) + && match_index_to_operand((Node *) lfirst(lc1), c, unique_index)) + { + matched_const = true; + break; + } + } + + if (matched_const) + continue; + + /* Check if the indexed expr is used in rel */ + if (attr > 0) + { + /* + * Normal Indexed column, if the col is not used, then the index is useless + * for uniquekey. + */ + attr -= FirstLowInvalidHeapAttributeNumber; + + if (!bms_is_member(attr, used_varattrs)) + { + *useful = false; + break; + } + } + else if (!list_member(unique_index->rel->reltarget->exprs, expr)) + { + /* Expression index but the expression is not used in rel */ + *useful = false; + break; + } + + /* check not null property. */ + if (attr == 0) + { + /* We never know if a expression yields null or not */ + *multi_nullvals = true; + } + else if (!bms_is_member(attr, unique_index->rel->notnullattrs) + && !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber, + unique_index->rel->notnullattrs)) + { + *multi_nullvals = true; + } + + exprs = lappend(exprs, expr); + } + return exprs; +} + + +/* + * add_uniquekey_for_onerow + * If we are sure that the relation only returns one row, then all the columns + * are unique. However we don't need to create UniqueKey for every column, we + * just set exprs = NIL and overwrites all the other UniqueKey on this RelOptInfo + * since this one has strongest semantics. + */ +void +add_uniquekey_for_onerow(RelOptInfo *rel) +{ + /* + * We overwrite the previous UniqueKey on purpose since this one has the + * strongest semantic. + */ + rel->uniquekeys = list_make1(makeUniqueKey(NIL, false)); +} + + +/* + * initililze_uniquecontext_for_joinrel + * Return a List of UniqueKeyContext for an inputrel + */ +static List * +initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel) +{ + List *res = NIL; + ListCell *lc; + foreach(lc, inputrel->uniquekeys) + { + UniqueKeyContext context; + context = palloc(sizeof(struct UniqueKeyContextData)); + context->uniquekey = lfirst_node(UniqueKey, lc); + context->added_to_joinrel = false; + context->useful = true; + res = lappend(res, context); + } + return res; +} + + +/* + * get_exprs_from_uniquekey + * Unify the way of get List of exprs from a one-row UniqueKey or + * normal UniqueKey. for the onerow case, every expr in rel1 is a valid + * UniqueKey. Return a List of exprs. + * + * rel1: The relation which you want to get the exprs. + * ukey: The UniqueKey you want to get the exprs. + */ +static List * +get_exprs_from_uniquekey(RelOptInfo *joinrel, RelOptInfo *rel1, UniqueKey *ukey) +{ + ListCell *lc; + bool onerow = rel1 != NULL && relation_is_onerow(rel1); + + List *res = NIL; + Assert(onerow || ukey); + if (onerow) + { + /* Only cares about the exprs still exist in joinrel */ + foreach(lc, joinrel->reltarget->exprs) + { + Bitmapset *relids = pull_varnos(lfirst(lc)); + if (bms_is_subset(relids, rel1->relids)) + { + res = lappend(res, list_make1(lfirst(lc))); + } + } + } + else + { + res = list_make1(ukey->exprs); + } + return res; +} + +/* + * Partitioned table Unique Keys. + * The partition table unique key is maintained as: + * 1. The index must be unique as usual. + * 2. The index must contains partition key. + * 3. The index must exist on all the child rel. see simple_indexinfo_equal for + * how we compare it. + */ + +/* + * index_constains_partkey + * return true if the index contains the partiton key. + */ +static bool +index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind) +{ + ListCell *lc; + int i; + Assert(IS_PARTITIONED_REL(partrel)); + Assert(partrel->part_scheme->partnatts > 0); + + for(i = 0; i < partrel->part_scheme->partnatts; i++) + { + Node *part_expr = linitial(partrel->partexprs[i]); + bool found_in_index = false; + foreach(lc, ind->indextlist) + { + Expr *index_expr = lfirst_node(TargetEntry, lc)->expr; + if (equal(index_expr, part_expr)) + { + found_in_index = true; + break; + } + } + if (!found_in_index) + return false; + } + return true; +} + +/* + * simple_indexinfo_equal + * + * Used to check if the 2 index is same as each other. The index here + * is COPIED from childrel and did some tiny changes(see + * simple_copy_indexinfo_to_parent) + */ +static bool +simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2) +{ + Size oid_cmp_len = sizeof(Oid) * ind1->ncolumns; + + return ind1->ncolumns == ind2->ncolumns && + ind1->unique == ind2->unique && + memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0 && + memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0 && + memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0 && + memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0 && + equal(get_tlist_exprs(ind1->indextlist, true), + get_tlist_exprs(ind2->indextlist, true)); +} + + +/* + * The below macros are used for simple_copy_indexinfo_to_parent which is so + * customized that I don't want to put it to copyfuncs.c. So copy it here. + */ +#define COPY_POINTER_FIELD(fldname, sz) \ + do { \ + Size _size = (sz); \ + newnode->fldname = palloc(_size); \ + memcpy(newnode->fldname, from->fldname, _size); \ + } while (0) + +#define COPY_NODE_FIELD(fldname) \ + (newnode->fldname = copyObjectImpl(from->fldname)) + +#define COPY_SCALAR_FIELD(fldname) \ + (newnode->fldname = from->fldname) + + +/* + * simple_copy_indexinfo_to_parent (from partition) + * Copy the IndexInfo from child relation to parent relation with some modification, + * which is used to test: + * 1. If the same index exists in all the childrels. + * 2. If the parentrel->reltarget/basicrestrict info matches this index. + */ +static IndexOptInfo * +simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from) +{ + IndexOptInfo *newnode = makeNode(IndexOptInfo); + AppendRelInfo *appinfo = find_appinfo_by_child(root, from->rel->relid); + ListCell *lc; + int idx = 0; + + COPY_SCALAR_FIELD(ncolumns); + COPY_SCALAR_FIELD(nkeycolumns); + COPY_SCALAR_FIELD(unique); + COPY_SCALAR_FIELD(immediate); + /* We just need to know if it is NIL or not */ + COPY_SCALAR_FIELD(indpred); + COPY_SCALAR_FIELD(predOK); + COPY_POINTER_FIELD(indexkeys, from->ncolumns * sizeof(int)); + COPY_POINTER_FIELD(indexcollations, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opfamily, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opcintype, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(sortopfamily, from->ncolumns * sizeof(Oid)); + COPY_NODE_FIELD(indextlist); + + /* Convert index exprs on child expr to expr on parent */ + foreach(lc, newnode->indextlist) + { + TargetEntry *tle = lfirst_node(TargetEntry, lc); + /* Index on expression is ignored */ + Assert(IsA(tle->expr, Var)); + tle->expr = (Expr *) find_parent_var(appinfo, (Var *) tle->expr); + newnode->indexkeys[idx] = castNode(Var, tle->expr)->varattno; + idx++; + } + newnode->rel = parentrel; + return newnode; +} + +/* + * adjust_partition_unique_indexlist + * + * global_unique_indexes: At the beginning, it contains the copy & modified + * unique index from the first partition. And then check if each index in it still + * exists in the following partitions. If no, remove it. at last, it has an + * index list which exists in all the partitions. + */ +static void +adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_indexes) +{ + ListCell *lc, *lc2; + foreach(lc, *global_unique_indexes) + { + IndexOptInfo *g_ind = lfirst_node(IndexOptInfo, lc); + bool found_in_child = false; + + foreach(lc2, childrel->indexlist) + { + IndexOptInfo *p_ind = lfirst_node(IndexOptInfo, lc2); + IndexOptInfo *p_ind_copy; + if (!p_ind->unique || !p_ind->immediate || + (p_ind->indpred != NIL && !p_ind->predOK)) + continue; + p_ind_copy = simple_copy_indexinfo_to_parent(root, parentrel, p_ind); + if (simple_indexinfo_equal(p_ind_copy, g_ind)) + { + found_in_child = true; + break; + } + } + if (!found_in_child) + /* The index doesn't exist in childrel, remove it from global_unique_indexes */ + *global_unique_indexes = foreach_delete_current(*global_unique_indexes, lc); + } +} + +/* Helper function for groupres/distinctrel */ +static void +add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgroups) +{ + Query *parse = root->parse; + List *exprs; + + /* + * XXX: If there are some vars which is not in current levelsup, the semantic is + * imprecise, should we avoid it or not? levelsup = 1 is just a demo, maybe we need to + * check every level other than 0, if so, looks we have to write another + * pull_var_walker. + */ + List *upper_vars = pull_vars_of_level((Node*)sortgroups, 1); + + if (upper_vars != NIL) + return; + + exprs = get_sortgrouplist_exprs(sortgroups, parse->targetList); + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(exprs, + false /* sortgroupclause can't be multi_nullvals */)); +} + + +/* + * add_combined_uniquekey + * The combination of both UniqueKeys is a valid UniqueKey for joinrel no matter + * the jointype. + */ +bool +add_combined_uniquekey(RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype) +{ + + ListCell *lc1, *lc2; + + /* Either side has multi_nullvals or we have outer join, + * the combined UniqueKey has multi_nullvals */ + bool multi_nullvals = outer_ukey->multi_nullvals || + inner_ukey->multi_nullvals || IS_OUTER_JOIN(jointype); + + /* The only case we can get onerow joinrel after join */ + if (relation_is_onerow(outer_rel) + && relation_is_onerow(inner_rel) + && jointype == JOIN_INNER) + { + add_uniquekey_for_onerow(joinrel); + return true; + } + + foreach(lc1, get_exprs_from_uniquekey(joinrel, outer_rel, outer_ukey)) + { + foreach(lc2, get_exprs_from_uniquekey(joinrel, inner_rel, inner_ukey)) + { + List *exprs = list_concat_copy(lfirst_node(List, lc1), lfirst_node(List, lc2)); + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(exprs, + multi_nullvals)); + } + } + return false; +} diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index b406d41e91..0551ae0512 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -2389,6 +2389,8 @@ grouping_planner(PlannerInfo *root, bool inheritance_update, add_path(final_rel, path); } + simple_copy_uniquekeys(current_rel, final_rel); + /* * Generate partial paths for final_rel, too, if outer query levels might * be able to make use of them. @@ -3899,6 +3901,8 @@ create_grouping_paths(PlannerInfo *root, } set_cheapest(grouped_rel); + + populate_grouprel_uniquekeys(root, grouped_rel, input_rel); return grouped_rel; } @@ -4615,7 +4619,7 @@ create_window_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(window_rel); - + simple_copy_uniquekeys(input_rel, window_rel); return window_rel; } @@ -4911,7 +4915,7 @@ create_distinct_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(distinct_rel); - + populate_distinctrel_uniquekeys(root, input_rel, distinct_rel); return distinct_rel; } @@ -5172,6 +5176,8 @@ create_ordered_paths(PlannerInfo *root, */ Assert(ordered_rel->pathlist != NIL); + simple_copy_uniquekeys(input_rel, ordered_rel); + return ordered_rel; } @@ -6049,6 +6055,9 @@ adjust_paths_for_srfs(PlannerInfo *root, RelOptInfo *rel, if (list_length(targets) == 1) return; + /* UniqueKey is not valid after handling the SRF. */ + rel->uniquekeys = NIL; + /* * Stack SRF-evaluation nodes atop each path for the rel. * diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c index 951aed80e7..e94e92937c 100644 --- a/src/backend/optimizer/prep/prepunion.c +++ b/src/backend/optimizer/prep/prepunion.c @@ -689,6 +689,8 @@ generate_union_paths(SetOperationStmt *op, PlannerInfo *root, /* Undo effects of possibly forcing tuple_fraction to 0 */ root->tuple_fraction = save_fraction; + /* Add the UniqueKeys */ + populate_unionrel_uniquekeys(root, result_rel); return result_rel; } diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c index d722063cf3..44c37ecffc 100644 --- a/src/backend/optimizer/util/appendinfo.c +++ b/src/backend/optimizer/util/appendinfo.c @@ -746,3 +746,47 @@ find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos) } return appinfos; } + +/* + * find_appinfo_by_child + * + */ +AppendRelInfo * +find_appinfo_by_child(PlannerInfo *root, Index child_index) +{ + ListCell *lc; + foreach(lc, root->append_rel_list) + { + AppendRelInfo *appinfo = lfirst_node(AppendRelInfo, lc); + if (appinfo->child_relid == child_index) + return appinfo; + } + elog(ERROR, "parent relation cant be found"); + return NULL; +} + +/* + * find_parent_var + * + */ +Var * +find_parent_var(AppendRelInfo *appinfo, Var *child_var) +{ + ListCell *lc; + Var *res = NULL; + Index attno = 1; + foreach(lc, appinfo->translated_vars) + { + Node *child_node = lfirst(lc); + if (equal(child_node, child_var)) + { + res = copyObject(child_var); + res->varattno = attno; + res->varno = appinfo->parent_relid; + } + attno++; + } + if (res == NULL) + elog(ERROR, "parent var can't be found."); + return res; +} diff --git a/src/backend/optimizer/util/inherit.c b/src/backend/optimizer/util/inherit.c index 3132fd35a5..d66b40ec50 100644 --- a/src/backend/optimizer/util/inherit.c +++ b/src/backend/optimizer/util/inherit.c @@ -736,6 +736,7 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, { Node *onecq = (Node *) lfirst(lc2); bool pseudoconstant; + RestrictInfo *child_rinfo; /* check for pseudoconstant (no Vars or volatile functions) */ pseudoconstant = @@ -747,13 +748,14 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, root->hasPseudoConstantQuals = true; } /* reconstitute RestrictInfo with appropriate properties */ - childquals = lappend(childquals, - make_restrictinfo((Expr *) onecq, - rinfo->is_pushed_down, - rinfo->outerjoin_delayed, - pseudoconstant, - rinfo->security_level, - NULL, NULL, NULL)); + child_rinfo = make_restrictinfo((Expr *) onecq, + rinfo->is_pushed_down, + rinfo->outerjoin_delayed, + pseudoconstant, + rinfo->security_level, + NULL, NULL, NULL); + child_rinfo->mergeopfamilies = rinfo->mergeopfamilies; + childquals = lappend(childquals, child_rinfo); /* track minimum security level among child quals */ cq_min_security = Min(cq_min_security, rinfo->security_level); } diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 31d9aedeeb..c83f17acb7 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -16,6 +16,7 @@ #include "nodes/execnodes.h" #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" extern A_Expr *makeA_Expr(A_Expr_Kind kind, List *name, @@ -105,4 +106,6 @@ extern GroupingSet *makeGroupingSet(GroupingSetKind kind, List *content, int loc extern VacuumRelation *makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols); +extern UniqueKey* makeUniqueKey(List *exprs, bool multi_nullvals); + #endif /* MAKEFUNC_H */ diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 381d84b4e4..41110ed888 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -264,6 +264,7 @@ typedef enum NodeTag T_EquivalenceMember, T_PathKey, T_PathTarget, + T_UniqueKey, T_RestrictInfo, T_IndexClause, T_PlaceHolderVar, diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index 9e3ebd488a..02e4458bef 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -730,6 +730,7 @@ typedef struct RelOptInfo QualCost baserestrictcost; /* cost of evaluating the above */ Index baserestrict_min_security; /* min security_level found in * baserestrictinfo */ + List *uniquekeys; /* List of UniqueKey */ List *joininfo; /* RestrictInfo structures for join clauses * involving this rel */ bool has_eclass_joins; /* T means joininfo is incomplete */ @@ -1047,6 +1048,28 @@ typedef struct PathKey } PathKey; +/* + * UniqueKey + * + * Represents the unique properties held by a RelOptInfo. + * + * exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL + * is a special case of UniqueKey, which means there is only 1 row in that + * relation. + * multi_nullvals: true means multi null values may exist in these exprs, so the + * uniqueness is not guaranteed in this case. This field is necessary for + * remove_useless_join & reduce_unique_semijoins where we don't mind these + * duplicated NULL values. It is set to true for 2 cases. One is a unique key + * from a unique index but the related column is nullable. The other one is for + * outer join. see populate_joinrel_uniquekeys for detail. + */ +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + /* * PathTarget * @@ -2473,7 +2496,7 @@ typedef enum * * flags indicating what kinds of grouping are possible. * partial_costs_set is true if the agg_partial_costs and agg_final_costs - * have been initialized. + * have been initialized. * agg_partial_costs gives partial aggregation costs. * agg_final_costs gives finalization costs. * target_parallel_safe is true if target is parallel safe. @@ -2503,8 +2526,8 @@ typedef struct * limit_tuples is an estimated bound on the number of output tuples, * or -1 if no LIMIT or couldn't estimate. * count_est and offset_est are the estimated values of the LIMIT and OFFSET - * expressions computed by preprocess_limit() (see comments for - * preprocess_limit() for more information). + * expressions computed by preprocess_limit() (see comments for + * preprocess_limit() for more information). */ typedef struct { diff --git a/src/include/nodes/pg_list.h b/src/include/nodes/pg_list.h index 14ea2766ad..621f54a9f8 100644 --- a/src/include/nodes/pg_list.h +++ b/src/include/nodes/pg_list.h @@ -528,6 +528,8 @@ extern bool list_member_ptr(const List *list, const void *datum); extern bool list_member_int(const List *list, int datum); extern bool list_member_oid(const List *list, Oid datum); +extern bool list_is_subset(const List *members, const List *target); + extern List *list_delete(List *list, void *datum); extern List *list_delete_ptr(List *list, void *datum); extern List *list_delete_int(List *list, int datum); diff --git a/src/include/optimizer/appendinfo.h b/src/include/optimizer/appendinfo.h index d6a27a60dd..e87c92a054 100644 --- a/src/include/optimizer/appendinfo.h +++ b/src/include/optimizer/appendinfo.h @@ -32,4 +32,7 @@ extern Relids adjust_child_relids_multilevel(PlannerInfo *root, Relids relids, extern AppendRelInfo **find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos); +extern AppendRelInfo *find_appinfo_by_child(PlannerInfo *root, Index child_index); +extern Var *find_parent_var(AppendRelInfo *appinfo, Var *child_var); + #endif /* APPENDINFO_H */ diff --git a/src/include/optimizer/optimizer.h b/src/include/optimizer/optimizer.h index 3e4171056e..9445141263 100644 --- a/src/include/optimizer/optimizer.h +++ b/src/include/optimizer/optimizer.h @@ -23,6 +23,7 @@ #define OPTIMIZER_H #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" /* * We don't want to include nodes/pathnodes.h here, because non-planner @@ -156,6 +157,7 @@ extern TargetEntry *get_sortgroupref_tle(Index sortref, List *targetList); extern TargetEntry *get_sortgroupclause_tle(SortGroupClause *sgClause, List *targetList); +extern Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); extern Node *get_sortgroupclause_expr(SortGroupClause *sgClause, List *targetList); extern List *get_sortgrouplist_exprs(List *sgClauses, diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h index 10b6e81079..9217a8d6c6 100644 --- a/src/include/optimizer/paths.h +++ b/src/include/optimizer/paths.h @@ -240,5 +240,48 @@ extern PathKey *make_canonical_pathkey(PlannerInfo *root, int strategy, bool nulls_first); extern void add_paths_to_append_rel(PlannerInfo *root, RelOptInfo *rel, List *live_childrels); +extern List *select_mergejoin_clauses(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + JoinType jointype, + bool *mergejoin_allowed); + +/* + * uniquekeys.c + * Utilities for matching and building unique keys + */ +extern void populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List* unique_index_list); +extern void populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels); +extern void populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel); +extern void populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel); +extern void populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel); +extern void simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel); +extern void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel); +extern void populate_joinrel_uniquekeys(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + RelOptInfo *rel2, + List *restrictlist, + JoinType jointype); + +extern bool relation_has_uniquekeys_for(PlannerInfo *root, + RelOptInfo *rel, + List *exprs, + bool allow_multinulls); +extern bool relation_is_onerow(RelOptInfo *rel); #endif /* PATHS_H */ -- 2.21.0 --op7mf2i72xk5yu7c Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v36-0003-Extend-UniqueKeys.patch" ^ permalink raw reply [nested|flat] 91+ messages in thread
* [PATCH v38 2/6] Introduce UniqueKey attributes on RelOptInfo struct. @ 2020-05-11 07:50 一挃 <[email protected]> 0 siblings, 0 replies; 91+ messages in thread From: 一挃 @ 2020-05-11 07:50 UTC (permalink / raw) UniqueKey is a set of exprs on RelOptInfo which represents the exprs will be unique on the given RelOptInfo. You can see README.uniquekey for more information. --- src/backend/nodes/copyfuncs.c | 13 + src/backend/nodes/list.c | 31 + src/backend/nodes/makefuncs.c | 13 + src/backend/nodes/outfuncs.c | 11 + src/backend/nodes/readfuncs.c | 10 + src/backend/optimizer/path/Makefile | 3 +- src/backend/optimizer/path/README.uniquekey | 131 +++ src/backend/optimizer/path/allpaths.c | 10 + src/backend/optimizer/path/joinpath.c | 9 +- src/backend/optimizer/path/joinrels.c | 2 + src/backend/optimizer/path/pathkeys.c | 3 +- src/backend/optimizer/path/uniquekeys.c | 1135 +++++++++++++++++++ src/backend/optimizer/plan/planner.c | 13 +- src/backend/optimizer/prep/prepunion.c | 2 + src/backend/optimizer/util/appendinfo.c | 44 + src/backend/optimizer/util/inherit.c | 18 +- src/include/nodes/makefuncs.h | 3 + src/include/nodes/nodes.h | 1 + src/include/nodes/pathnodes.h | 29 +- src/include/nodes/pg_list.h | 10 + src/include/optimizer/appendinfo.h | 3 + src/include/optimizer/optimizer.h | 2 + src/include/optimizer/paths.h | 43 + 23 files changed, 1515 insertions(+), 24 deletions(-) create mode 100644 src/backend/optimizer/path/README.uniquekey create mode 100644 src/backend/optimizer/path/uniquekeys.c diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..75c1c5e824 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2296,6 +2296,16 @@ _copyPathKey(const PathKey *from) return newnode; } +static UniqueKey * +_copyUniqueKey(const UniqueKey *from) +{ + UniqueKey *newnode = makeNode(UniqueKey); + + COPY_NODE_FIELD(exprs); + COPY_SCALAR_FIELD(multi_nullvals); + + return newnode; +} /* * _copyRestrictInfo */ @@ -5220,6 +5230,9 @@ copyObjectImpl(const void *from) case T_PathKey: retval = _copyPathKey(from); break; + case T_UniqueKey: + retval = _copyUniqueKey(from); + break; case T_RestrictInfo: retval = _copyRestrictInfo(from); break; diff --git a/src/backend/nodes/list.c b/src/backend/nodes/list.c index dbf6b30233..ca099495a1 100644 --- a/src/backend/nodes/list.c +++ b/src/backend/nodes/list.c @@ -702,6 +702,37 @@ list_member_oid(const List *list, Oid datum) return false; } +/* + * return true iff every entry in "members" list is also present + * in the "target" list. + */ +bool +list_is_subset(const List *members, const List *target) +{ + const ListCell *lc1, *lc2; + + Assert(IsPointerList(members)); + Assert(IsPointerList(target)); + check_list_invariants(members); + check_list_invariants(target); + + foreach(lc1, members) + { + bool found = false; + foreach(lc2, target) + { + if (equal(lfirst(lc1), lfirst(lc2))) + { + found = true; + break; + } + } + if (!found) + return false; + } + return true; +} + /* * Delete the n'th cell (counting from 0) in list. * diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 01c110cd2f..40415d0f5b 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -815,3 +815,16 @@ makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols) v->va_cols = va_cols; return v; } + + +/* + * makeUniqueKey + */ +UniqueKey* +makeUniqueKey(List *exprs, bool multi_nullvals) +{ + UniqueKey * ukey = makeNode(UniqueKey); + ukey->exprs = exprs; + ukey->multi_nullvals = multi_nullvals; + return ukey; +} diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..44154cde6a 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2456,6 +2456,14 @@ _outPathKey(StringInfo str, const PathKey *node) WRITE_BOOL_FIELD(pk_nulls_first); } +static void +_outUniqueKey(StringInfo str, const UniqueKey *node) +{ + WRITE_NODE_TYPE("UNIQUEKEY"); + WRITE_NODE_FIELD(exprs); + WRITE_BOOL_FIELD(multi_nullvals); +} + static void _outPathTarget(StringInfo str, const PathTarget *node) { @@ -4198,6 +4206,9 @@ outNode(StringInfo str, const void *obj) case T_PathKey: _outPathKey(str, obj); break; + case T_UniqueKey: + _outUniqueKey(str, obj); + break; case T_PathTarget: _outPathTarget(str, obj); break; diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index c5e136e9c3..b3e212bf1c 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -492,6 +492,14 @@ _readSetOperationStmt(void) READ_DONE(); } +static UniqueKey * +_readUniqueKey(void) +{ + READ_LOCALS(UniqueKey); + READ_NODE_FIELD(exprs); + READ_BOOL_FIELD(multi_nullvals); + READ_DONE(); +} /* * Stuff from primnodes.h. @@ -2717,6 +2725,8 @@ parseNodeString(void) return_value = _readCommonTableExpr(); else if (MATCH("SETOPERATIONSTMT", 16)) return_value = _readSetOperationStmt(); + else if (MATCH("UNIQUEKEY", 9)) + return_value = _readUniqueKey(); else if (MATCH("ALIAS", 5)) return_value = _readAlias(); else if (MATCH("RANGEVAR", 8)) diff --git a/src/backend/optimizer/path/Makefile b/src/backend/optimizer/path/Makefile index 1e199ff66f..7b9820c25f 100644 --- a/src/backend/optimizer/path/Makefile +++ b/src/backend/optimizer/path/Makefile @@ -21,6 +21,7 @@ OBJS = \ joinpath.o \ joinrels.o \ pathkeys.o \ - tidpath.o + tidpath.o \ + uniquekeys.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/optimizer/path/README.uniquekey b/src/backend/optimizer/path/README.uniquekey new file mode 100644 index 0000000000..5eac761995 --- /dev/null +++ b/src/backend/optimizer/path/README.uniquekey @@ -0,0 +1,131 @@ +1. What is UniqueKey? +We can think UniqueKey is a set of exprs for a RelOptInfo, which we are insure +that doesn't yields same result among all the rows. The simplest UniqueKey +format is primary key. + +However we define the UnqiueKey as below. + +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + +exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL +is a special case of UniqueKey, which means there is only one row in that +relation.it has a stronger semantic than others. like SELECT uk FROM t; uk is +normal unique key and may have different values. SELECT colx FROM t WHERE uk = +const. colx is unique AND we have only 1 value. This field can used for +innerrel_is_unique. this logic is handled specially in add_uniquekey_for_onerow +function. + +multi_nullvals: true means multi null values may exist in these exprs, so the +uniqueness is not guaranteed in this case. This field is necessary for +remove_useless_join & reduce_unique_semijoins where we don't mind these +duplicated NULL values. It is set to true for 2 cases. One is a unique key +from a unique index but the related column is nullable. The other one is for +outer join. see populate_joinrel_uniquekeys for detail. + + +The UniqueKey can be used at the following cases at least: +1. remove_useless_joins. +2. reduce_semianti_joins +3. remove distinct node if distinct clause is unique. +4. remove aggnode if group by clause is unique. +5. Index Skip Scan (WIP) +6. Aggregation Push Down without 2 phase aggregation if the join can't + duplicated the aggregated rows. (work in progress feature) + +2. How is it maintained? + +We have a set of populate_xxx_unqiuekeys functions to maintain the uniquekey on +various cases. xxx includes baserel, joinrel, partitionedrel, distinctrel, +groupedrel, unionrel. and we also need to convert the uniquekey from subquery +to outer relation, which is what convert_subquery_uniquekeys does. + +1. The first part is about baserel. We handled 3 cases. suppose we have Unique +Index on (a, b). + +1. SELECT a, b FROM t. UniqueKey (a, b) +2. SELECT a FROM t WHERE b = 1; UniqueKey (a) +3. SELECT .. FROM t WHERE a = 1 AND b = 1; UniqueKey (NIL). onerow case, every + column is Unique. + +2. The next part is joinrel, this part is most error-prone, we simplified the rules +like below: +1. If the relation's UniqueKey can't be duplicated after join, then is will be + still valid for the join rel. The function we used here is + innerrel_keeps_unique. The basic idea is innerrel.any_col = outer.uk. + +2. If the UnqiueKey can't keep valid via the rule 1, the combination of the + UniqueKey from both sides are valid for sure. We can prove this as: if the + unique exprs from rel1 is duplicated by rel2, the duplicated rows must + contains different unique exprs from rel2. + +More considerations about onerow: +1. If relation with one row and it can't be duplicated, it is still possible + contains mulit_nullvas after outer join. +2. If the either UniqueKey can be duplicated after join, the can get one row + only when both side is one row AND there is no outer join. +3. Whenever the onerow UniqueKey is not a valid any more, we need to convert one + row UniqueKey to normal unique key since we don't store exprs for one-row + relation. get_exprs_from_uniquekeys will be used here. + + +More considerations about multi_nullvals after join: +1. If the original UnqiueKey has multi_nullvals, the final UniqueKey will have + mulit_nullvals in any case. +2. If a unique key doesn't allow mulit_nullvals, after some outer join, it + allows some outer join. + + +3. When we comes to subquery, we need to convert_subquery_unqiuekeys just like +convert_subquery_pathkeys. Only the UniqueKey insides subquery is referenced as +a Var in outer relation will be reused. The relationship between the outerrel.Var +and subquery.exprs is built with outerel->subroot->processed_tlist. + + +4. As for the SRF functions, it will break the uniqueness of uniquekey, However it +is handled in adjust_paths_for_srfs, which happens after the query_planner. so +we will maintain the UniqueKey until there and reset it to NIL at that +places. This can't help on distinct/group by elimination cases but probably help +in some other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is +semantic correctly. + + +5. As for inherit table, we first main the UnqiueKey on childrel as well. But for +partitioned table we need to maintain 2 different kinds of +UnqiueKey. 1). UniqueKey on the parent relation 2). UniqueKey on child +relation for partition wise query. + +Example: +CREATE TABLE p (a int not null, b int not null) partition by list (a); +CREATE TABLE p0 partition of p for values in (1); +CREATE TABLE p1 partition of p for values in (2); + +create unique index p0_b on p0(b); +create unique index p1_b on p1(b); + +Now b is only unique on partition level, so the distinct can't be removed on +the following cases. SELECT DISTINCT b FROM p; + +Another example is SELECT DISTINCT a, b FROM p WHERE a = 1; Since only one +partition is chosen, the UniqueKey on child relation is same as the UniqueKey on +parent relation. + +Another usage of UniqueKey on partition level is it be helpful for +partition-wise join. + +As for the UniqueKey on parent table level, it comes with 2 different ways, +1). the UniqueKey is also derived in UniqueKey index, but the index must be same +in all the related children relations and the unique index must contains +Partition Key in it. Example: + +CREATE UNIQUE INDEX p_ab ON p(a, b); -- where a is the partition key. + +-- Query +SELECT a, b FROM p; the (a, b) is a UniqueKey of p. + +2). If the parent relation has only one childrel, the UniqueKey on childrel is + the UniqueKey on parent as well. diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index 37b4223adb..d52ad59f89 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -579,6 +579,12 @@ set_plain_rel_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) */ check_index_predicates(root, rel); + /* + * Now that we've marked which partial indexes are suitable, we can now + * build the relation's unique keys. + */ + populate_baserel_uniquekeys(root, rel, rel->indexlist); + /* Mark rel with estimated output rows, width, etc */ set_baserel_size_estimates(root, rel); } @@ -1297,6 +1303,8 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, /* Add paths to the append relation. */ add_paths_to_append_rel(root, rel, live_childrels); + if (IS_PARTITIONED_REL(rel)) + populate_partitionedrel_uniquekeys(root, rel, live_childrels); } @@ -2304,6 +2312,8 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, pathkeys, required_outer)); } + convert_subquery_uniquekeys(root, rel, sub_final_rel); + /* If outer rel allows parallelism, do same for partial paths. */ if (rel->consider_parallel && bms_is_empty(required_outer)) { diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c index 57ce97fd53..697ac047fb 100644 --- a/src/backend/optimizer/path/joinpath.c +++ b/src/backend/optimizer/path/joinpath.c @@ -71,13 +71,6 @@ static void consider_parallel_mergejoin(PlannerInfo *root, static void hash_inner_and_outer(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, RelOptInfo *innerrel, JoinType jointype, JoinPathExtraData *extra); -static List *select_mergejoin_clauses(PlannerInfo *root, - RelOptInfo *joinrel, - RelOptInfo *outerrel, - RelOptInfo *innerrel, - List *restrictlist, - JoinType jointype, - bool *mergejoin_allowed); static void generate_mergejoin_paths(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *innerrel, @@ -1927,7 +1920,7 @@ hash_inner_and_outer(PlannerInfo *root, * if it is mergejoinable and involves vars from the two sub-relations * currently of interest. */ -static List * +List * select_mergejoin_clauses(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c index 0dbe2ac726..7271f044ec 100644 --- a/src/backend/optimizer/path/joinrels.c +++ b/src/backend/optimizer/path/joinrels.c @@ -924,6 +924,8 @@ populate_joinrel_with_paths(PlannerInfo *root, RelOptInfo *rel1, /* Apply partitionwise join technique, if possible. */ try_partitionwise_join(root, rel1, rel2, joinrel, sjinfo, restrictlist); + + populate_joinrel_uniquekeys(root, joinrel, rel1, rel2, restrictlist, sjinfo->jointype); } diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index bd9a176d7d..139278829b 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -33,7 +33,6 @@ static bool pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys); static bool matches_boolean_partition_clause(RestrictInfo *rinfo, RelOptInfo *partrel, int partkeycol); -static Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); static bool right_merge_direction(PlannerInfo *root, PathKey *pathkey); @@ -1035,7 +1034,7 @@ convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel, * We need this to ensure that we don't return pathkeys describing values * that are unavailable above the level of the subquery scan. */ -static Var * +Var * find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle) { ListCell *lc; diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c new file mode 100644 index 0000000000..77ed2b2eff --- /dev/null +++ b/src/backend/optimizer/path/uniquekeys.c @@ -0,0 +1,1135 @@ +/*------------------------------------------------------------------------- + * + * uniquekeys.c + * Utilities for matching and building unique keys + * + * Portions Copyright (c) 2020, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/optimizer/path/uniquekeys.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "nodes/makefuncs.h" +#include "nodes/nodeFuncs.h" +#include "optimizer/pathnode.h" +#include "optimizer/paths.h" +#include "optimizer/appendinfo.h" +#include "optimizer/optimizer.h" +#include "optimizer/tlist.h" +#include "rewrite/rewriteManip.h" + + +/* + * This struct is used to help populate_joinrel_uniquekeys. + * + * added_to_joinrel is true if a uniquekey (from outerrel or innerrel) + * has been added to joinrel. + * useful is true if the exprs of the uniquekey still appears in joinrel. + */ +typedef struct UniqueKeyContextData +{ + UniqueKey *uniquekey; + bool added_to_joinrel; + bool useful; +} *UniqueKeyContext; + +static List *initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel); +static bool innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + bool reverse); + +static List *get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals); +static List *get_exprs_from_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + UniqueKey *ukey); +static void add_uniquekey_for_onerow(RelOptInfo *rel); +static bool add_combined_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype); + +/* Used for unique indexes checking for partitioned table */ +static bool index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind); +static IndexOptInfo *simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from); +static bool simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2); +static void adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_index); + +/* Helper function for grouped relation and distinct relation. */ +static void add_uniquekey_from_sortgroups(PlannerInfo *root, + RelOptInfo *rel, + List *sortgroups); + +/* + * populate_baserel_uniquekeys + * Populate 'baserel' uniquekeys list by looking at the rel's unique index + * and baserestrictinfo + */ +void +populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List *indexlist) +{ + ListCell *lc; + List *matched_uniq_indexes = NIL; + + /* Attrs appears in rel->reltarget->exprs. */ + Bitmapset *used_attrs = NULL; + + List *const_exprs = NIL; + List *expr_opfamilies = NIL; + + Assert(baserel->rtekind == RTE_RELATION); + + foreach(lc, indexlist) + { + IndexOptInfo *ind = (IndexOptInfo *) lfirst(lc); + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + matched_uniq_indexes = lappend(matched_uniq_indexes, ind); + } + + if (matched_uniq_indexes == NIL) + return; + + /* Check which attrs is used in baserel->reltarget */ + pull_varattnos((Node *)baserel->reltarget->exprs, baserel->relid, &used_attrs); + + /* Check which attrno is used at a mergeable const filter */ + foreach(lc, baserel->baserestrictinfo) + { + RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); + + if (rinfo->mergeopfamilies == NIL) + continue; + + if (bms_is_empty(rinfo->left_relids)) + { + const_exprs = lappend(const_exprs, get_rightop(rinfo->clause)); + } + else if (bms_is_empty(rinfo->right_relids)) + { + const_exprs = lappend(const_exprs, get_leftop(rinfo->clause)); + } + else + continue; + + expr_opfamilies = lappend(expr_opfamilies, rinfo->mergeopfamilies); + } + + foreach(lc, matched_uniq_indexes) + { + bool multi_nullvals, useful; + List *exprs = get_exprs_from_uniqueindex(lfirst_node(IndexOptInfo, lc), + const_exprs, + expr_opfamilies, + used_attrs, + &useful, + &multi_nullvals); + if (useful) + { + if (exprs == NIL) + { + /* All the columns in Unique Index matched with a restrictinfo */ + add_uniquekey_for_onerow(baserel); + return; + } + baserel->uniquekeys = lappend(baserel->uniquekeys, + makeUniqueKey(exprs, multi_nullvals)); + } + } +} + + +/* + * populate_partitionedrel_uniquekeys + * The UniqueKey on partitionrel comes from 2 cases: + * 1). Only one partition is involved in this query, the unique key can be + * copied to parent rel from childrel. + * 2). There are some unique index which includes partition key and exists + * in all the related partitions. + * We never mind rule 2 if we hit rule 1. + */ + +void +populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels) +{ + ListCell *lc; + List *global_uniq_indexlist = NIL; + RelOptInfo *childrel; + bool is_first = true; + + Assert(IS_PARTITIONED_REL(rel)); + + if (childrels == NIL) + return; + + /* + * If there is only one partition used in this query, the UniqueKey in childrel is + * still valid in parent level, but we need convert the format from child expr to + * parent expr. + */ + if (list_length(childrels) == 1) + { + /* Check for Rule 1 */ + RelOptInfo *childrel = linitial_node(RelOptInfo, childrels); + ListCell *lc; + Assert(childrel->reloptkind == RELOPT_OTHER_MEMBER_REL); + if (relation_is_onerow(childrel)) + { + add_uniquekey_for_onerow(rel); + return; + } + + foreach(lc, childrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + AppendRelInfo *appinfo = find_appinfo_by_child(root, childrel->relid); + List *parent_exprs = NIL; + bool can_reuse = true; + ListCell *lc2; + foreach(lc2, ukey->exprs) + { + Var *var = (Var *)lfirst(lc2); + /* + * If the expr comes from a expression, it is hard to build the expression + * in parent so ignore that case for now. + */ + if(!IsA(var, Var)) + { + can_reuse = false; + break; + } + /* Convert it to parent var */ + parent_exprs = lappend(parent_exprs, find_parent_var(appinfo, var)); + } + if (can_reuse) + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(parent_exprs, + ukey->multi_nullvals)); + } + } + else + { + /* Check for rule 2 */ + childrel = linitial_node(RelOptInfo, childrels); + foreach(lc, childrel->indexlist) + { + IndexOptInfo *ind = lfirst(lc); + IndexOptInfo *modified_index; + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + + /* + * During simple_copy_indexinfo_to_parent, we need to convert var from + * child var to parent var, index on expression is too complex to handle. + * so ignore it for now. + */ + if (ind->indexprs != NIL) + continue; + + modified_index = simple_copy_indexinfo_to_parent(root, rel, ind); + /* + * If the unique index doesn't contain partkey, then it is unique + * on this partition only, so it is useless for us. + */ + if (!index_constains_partkey(rel, modified_index)) + continue; + + global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index); + } + + if (global_uniq_indexlist != NIL) + { + foreach(lc, childrels) + { + RelOptInfo *child = lfirst(lc); + if (is_first) + { + is_first = false; + continue; + } + adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist); + } + /* Now we have a list of unique index which are exactly same on all childrels, + * Set the UniqueKey just like it is non-partition table + */ + populate_baserel_uniquekeys(root, rel, global_uniq_indexlist); + } + } +} + + +/* + * populate_distinctrel_uniquekeys + */ +void +populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel) +{ + /* The unique key before the distinct is still valid. */ + distinctrel->uniquekeys = list_copy(inputrel->uniquekeys); + add_uniquekey_from_sortgroups(root, distinctrel, root->parse->distinctClause); +} + +/* + * populate_grouprel_uniquekeys + */ +void +populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel) + +{ + Query *parse = root->parse; + bool input_ukey_added = false; + ListCell *lc; + + if (relation_is_onerow(inputrel)) + { + add_uniquekey_for_onerow(grouprel); + return; + } + if (parse->groupingSets) + return; + + /* A Normal group by without grouping set. */ + if (parse->groupClause) + { + /* + * Current even the groupby clause is Unique already, but if query has aggref + * We have to create grouprel still. To keep the UnqiueKey short, we will check + * the UniqueKey of input_rel still valid, if so we reuse it. + */ + foreach(lc, inputrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (list_is_subset(ukey->exprs, grouprel->reltarget->exprs)) + { + grouprel->uniquekeys = lappend(grouprel->uniquekeys, + ukey); + input_ukey_added = true; + } + } + if (!input_ukey_added) + /* + * group by clause must be a super-set of grouprel->reltarget->exprs except the + * aggregation expr, so if such exprs is unique already, no bother to generate + * new uniquekey for group by exprs. + */ + add_uniquekey_from_sortgroups(root, + grouprel, + root->parse->groupClause); + } + else + /* It has aggregation but without a group by, so only one row returned */ + add_uniquekey_for_onerow(grouprel); +} + +/* + * simple_copy_uniquekeys + * Using a function for the one-line code makes us easy to check where we simply + * copied the uniquekey. + */ +void +simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel) +{ + newrel->uniquekeys = oldrel->uniquekeys; +} + +/* + * populate_unionrel_uniquekeys + */ +void +populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel) +{ + ListCell *lc; + List *exprs = NIL; + + Assert(unionrel->uniquekeys == NIL); + + foreach(lc, unionrel->reltarget->exprs) + { + exprs = lappend(exprs, lfirst(lc)); + } + + if (exprs == NIL) + /* SQL: select union select; is valid, we need to handle it here. */ + add_uniquekey_for_onerow(unionrel); + else + unionrel->uniquekeys = lappend(unionrel->uniquekeys, + makeUniqueKey(exprs,false)); + +} + +/* + * populate_joinrel_uniquekeys + * + * populate uniquekeys for joinrel. We will check each relation to see if its + * UniqueKey is still valid via innerrel_keeps_unique, if so, we add it to + * joinrel. The multi_nullvals field will be changed to true for some outer + * join cases and one-row UniqueKey needs to be converted to normal UniqueKey + * for the same case as well. + * For the uniquekey in either baserel which can't be unique after join, we still + * check to see if combination of UniqueKeys from both side is still useful for us. + * if yes, we add it to joinrel as well. + */ +void +populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *outerrel, RelOptInfo *innerrel, + List *restrictlist, JoinType jointype) +{ + ListCell *lc, *lc2; + List *clause_list = NIL; + List *outerrel_ukey_ctx; + List *innerrel_ukey_ctx; + bool inner_onerow, outer_onerow; + bool mergejoin_allowed; + + /* Care about the outerrel relation only for SEMI/ANTI join */ + if (jointype == JOIN_SEMI || jointype == JOIN_ANTI) + { + foreach(lc, outerrel->uniquekeys) + { + UniqueKey *uniquekey = lfirst_node(UniqueKey, lc); + if (list_is_subset(uniquekey->exprs, joinrel->reltarget->exprs)) + joinrel->uniquekeys = lappend(joinrel->uniquekeys, uniquekey); + } + return; + } + + Assert(jointype == JOIN_LEFT || jointype == JOIN_FULL || jointype == JOIN_INNER); + + /* Fast path */ + if (innerrel->uniquekeys == NIL || outerrel->uniquekeys == NIL) + return; + + inner_onerow = relation_is_onerow(innerrel); + outer_onerow = relation_is_onerow(outerrel); + + outerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(outerrel); + innerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(innerrel); + + clause_list = select_mergejoin_clauses(root, joinrel, outerrel, innerrel, + restrictlist, jointype, + &mergejoin_allowed); + + if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true /* reverse */)) + { + bool outer_impact = jointype == JOIN_FULL; + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + /* Outer relation has one row, and the unique key is not duplicated after join, + * the joinrel will still has one row unless the jointype == JOIN_FULL. + */ + if (outer_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (outer_onerow) + { + /* + * The onerow outerrel becomes multi rows and multi_nullvals + * will be changed to true. We also need to set the exprs correctly since it + * can't be NIL any more. + */ + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, outerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Change multi_nullvals to true due to the full join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, true)); + else + /* Just reuse it */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + } + ctx->added_to_joinrel = true; + } + } + + if (innerrel_keeps_unique(root, outerrel, innerrel, clause_list, false)) + { + bool outer_impact = jointype == JOIN_FULL || jointype == JOIN_LEFT;; + + foreach(lc, innerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + if (inner_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (inner_onerow) + { + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, innerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Need to change multi_nullvals to true due to the outer join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, + true)); + else + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + + } + ctx->added_to_joinrel = true; + } + } + + /* + * The combination of the UniqueKey from both sides is unique as well regardless + * of join type, but no bother to add it if its subset has been added to joinrel + * already or it is not useful for the joinrel. + */ + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx1 = (UniqueKeyContext) lfirst(lc); + if (ctx1->added_to_joinrel || !ctx1->useful) + continue; + foreach(lc2, innerrel_ukey_ctx) + { + UniqueKeyContext ctx2 = (UniqueKeyContext) lfirst(lc2); + if (ctx2->added_to_joinrel || !ctx2->useful) + continue; + if (add_combined_uniquekey(root, joinrel, outerrel, innerrel, + ctx1->uniquekey, ctx2->uniquekey, + jointype)) + /* If we set a onerow UniqueKey to joinrel, we don't need other. */ + return; + } + } +} + + +/* + * convert_subquery_uniquekeys + * + * Covert the UniqueKey in subquery to outer relation. + */ +void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel) +{ + ListCell *lc; + + if (sub_final_rel->uniquekeys == NIL) + return; + + if (relation_is_onerow(sub_final_rel)) + { + add_uniquekey_for_onerow(currel); + return; + } + + Assert(currel->subroot != NULL); + + foreach(lc, sub_final_rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + ListCell *lc; + List *exprs = NIL; + bool ukey_useful = true; + + /* One row case is handled above */ + Assert(ukey->exprs != NIL); + foreach(lc, ukey->exprs) + { + Var *var; + TargetEntry *tle = tlist_member(lfirst(lc), + currel->subroot->processed_tlist); + if (tle == NULL) + { + ukey_useful = false; + break; + } + var = find_var_for_subquery_tle(currel, tle); + if (var == NULL) + { + ukey_useful = false; + break; + } + exprs = lappend(exprs, var); + } + + if (ukey_useful) + currel->uniquekeys = lappend(currel->uniquekeys, + makeUniqueKey(exprs, + ukey->multi_nullvals)); + + } +} + +/* + * innerrel_keeps_unique + * + * Check if Unique key of the innerrel is valid after join. innerrel's UniqueKey + * will be still valid if innerrel's any-column mergeop outrerel's uniquekey + * exists in clause_list. + * + * Note: the clause_list must be a list of mergeable restrictinfo already. + */ +static bool +innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *clause_list, + bool reverse) +{ + ListCell *lc, *lc2, *lc3; + + if (outerrel->uniquekeys == NIL || innerrel->uniquekeys == NIL) + return false; + + /* Check if there is outerrel's uniquekey in mergeable clause. */ + foreach(lc, outerrel->uniquekeys) + { + List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs; + bool clauselist_matchs_all_exprs = true; + foreach(lc2, outer_uq_exprs) + { + Node *outer_uq_expr = lfirst(lc2); + bool find_uq_expr_in_clauselist = false; + foreach(lc3, clause_list) + { + RestrictInfo *rinfo = lfirst_node(RestrictInfo, lc3); + Node *outer_expr; + if (reverse) + outer_expr = rinfo->outer_is_left ? get_rightop(rinfo->clause) : get_leftop(rinfo->clause); + else + outer_expr = rinfo->outer_is_left ? get_leftop(rinfo->clause) : get_rightop(rinfo->clause); + if (equal(outer_expr, outer_uq_expr)) + { + find_uq_expr_in_clauselist = true; + break; + } + } + if (!find_uq_expr_in_clauselist) + { + /* No need to check the next exprs in the current uniquekey */ + clauselist_matchs_all_exprs = false; + break; + } + } + + if (clauselist_matchs_all_exprs) + return true; + } + return false; +} + + +/* + * relation_is_onerow + * Check if it is a one-row relation by checking UniqueKey. + */ +bool +relation_is_onerow(RelOptInfo *rel) +{ + UniqueKey *ukey; + if (rel->uniquekeys == NIL) + return false; + ukey = linitial_node(UniqueKey, rel->uniquekeys); + return ukey->exprs == NIL && list_length(rel->uniquekeys) == 1; +} + +/* + * relation_has_uniquekeys_for + * Returns true if we have proofs that 'rel' cannot return multiple rows with + * the same values in each of 'exprs'. Otherwise returns false. + */ +bool +relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel, + List *exprs, bool allow_multinulls) +{ + ListCell *lc; + + /* + * For UniqueKey->onerow case, the uniquekey->exprs is empty as well + * so we can't rely on list_is_subset to handle this special cases + */ + if (exprs == NIL) + return false; + + foreach(lc, rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (ukey->multi_nullvals && !allow_multinulls) + continue; + if (list_is_subset(ukey->exprs, exprs)) + return true; + } + return false; +} + + +/* + * get_exprs_from_uniqueindex + * + * Return a list of exprs which is unique. set useful to false if this + * unique index is not useful for us. + */ +static List * +get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals) +{ + List *exprs = NIL; + ListCell *indexpr_item; + int c = 0; + + *useful = true; + *multi_nullvals = false; + + indexpr_item = list_head(unique_index->indexprs); + for(c = 0; c < unique_index->ncolumns; c++) + { + int attr = unique_index->indexkeys[c]; + Expr *expr; + bool matched_const = false; + ListCell *lc1, *lc2; + + if(attr > 0) + { + expr = list_nth_node(TargetEntry, unique_index->indextlist, c)->expr; + } + else if (attr == 0) + { + /* Expression index */ + expr = lfirst(indexpr_item); + indexpr_item = lnext(unique_index->indexprs, indexpr_item); + } + else /* attr < 0 */ + { + /* Index on system column is not supported */ + Assert(false); + } + + /* + * Check index_col = Const case with regarding to opfamily checking + * If we can remove the index_col from the final UniqueKey->exprs. + */ + forboth(lc1, const_exprs, lc2, const_expr_opfamilies) + { + if (list_member_oid((List *)lfirst(lc2), unique_index->opfamily[c]) + && match_index_to_operand((Node *) lfirst(lc1), c, unique_index)) + { + matched_const = true; + break; + } + } + + if (matched_const) + continue; + + /* Check if the indexed expr is used in rel */ + if (attr > 0) + { + /* + * Normal Indexed column, if the col is not used, then the index is useless + * for uniquekey. + */ + attr -= FirstLowInvalidHeapAttributeNumber; + + if (!bms_is_member(attr, used_varattrs)) + { + *useful = false; + break; + } + } + else if (!list_member(unique_index->rel->reltarget->exprs, expr)) + { + /* Expression index but the expression is not used in rel */ + *useful = false; + break; + } + + /* check not null property. */ + if (attr == 0) + { + /* We never know if a expression yields null or not */ + *multi_nullvals = true; + } + else if (!bms_is_member(attr, unique_index->rel->notnullattrs) + && !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber, + unique_index->rel->notnullattrs)) + { + *multi_nullvals = true; + } + + exprs = lappend(exprs, expr); + } + return exprs; +} + + +/* + * add_uniquekey_for_onerow + * If we are sure that the relation only returns one row, then all the columns + * are unique. However we don't need to create UniqueKey for every column, we + * just set exprs = NIL and overwrites all the other UniqueKey on this RelOptInfo + * since this one has strongest semantics. + */ +void +add_uniquekey_for_onerow(RelOptInfo *rel) +{ + /* + * We overwrite the previous UniqueKey on purpose since this one has the + * strongest semantic. + */ + rel->uniquekeys = list_make1(makeUniqueKey(NIL, false)); +} + + +/* + * initililze_uniquecontext_for_joinrel + * Return a List of UniqueKeyContext for an inputrel + */ +static List * +initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel) +{ + List *res = NIL; + ListCell *lc; + foreach(lc, inputrel->uniquekeys) + { + UniqueKeyContext context; + context = palloc(sizeof(struct UniqueKeyContextData)); + context->uniquekey = lfirst_node(UniqueKey, lc); + context->added_to_joinrel = false; + context->useful = true; + res = lappend(res, context); + } + return res; +} + + +/* + * get_exprs_from_uniquekey + * Unify the way of get List of exprs from a one-row UniqueKey or + * normal UniqueKey. for the onerow case, every expr in rel1 is a valid + * UniqueKey. Return a List of exprs. + * + * rel1: The relation which you want to get the exprs. + * ukey: The UniqueKey you want to get the exprs. + */ +static List * +get_exprs_from_uniquekey(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *rel1, UniqueKey *ukey) +{ + ListCell *lc; + bool onerow = rel1 != NULL && relation_is_onerow(rel1); + + List *res = NIL; + Assert(onerow || ukey); + if (onerow) + { + /* Only cares about the exprs still exist in joinrel */ + foreach(lc, joinrel->reltarget->exprs) + { + Bitmapset *relids = pull_varnos(root, lfirst(lc)); + if (bms_is_subset(relids, rel1->relids)) + { + res = lappend(res, list_make1(lfirst(lc))); + } + } + } + else + { + res = list_make1(ukey->exprs); + } + return res; +} + +/* + * Partitioned table Unique Keys. + * The partition table unique key is maintained as: + * 1. The index must be unique as usual. + * 2. The index must contains partition key. + * 3. The index must exist on all the child rel. see simple_indexinfo_equal for + * how we compare it. + */ + +/* + * index_constains_partkey + * return true if the index contains the partiton key. + */ +static bool +index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind) +{ + ListCell *lc; + int i; + Assert(IS_PARTITIONED_REL(partrel)); + Assert(partrel->part_scheme->partnatts > 0); + + for(i = 0; i < partrel->part_scheme->partnatts; i++) + { + Node *part_expr = linitial(partrel->partexprs[i]); + bool found_in_index = false; + foreach(lc, ind->indextlist) + { + Expr *index_expr = lfirst_node(TargetEntry, lc)->expr; + if (equal(index_expr, part_expr)) + { + found_in_index = true; + break; + } + } + if (!found_in_index) + return false; + } + return true; +} + +/* + * simple_indexinfo_equal + * + * Used to check if the 2 index is same as each other. The index here + * is COPIED from childrel and did some tiny changes(see + * simple_copy_indexinfo_to_parent) + */ +static bool +simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2) +{ + Size oid_cmp_len = sizeof(Oid) * ind1->ncolumns; + + return ind1->ncolumns == ind2->ncolumns && + ind1->unique == ind2->unique && + memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0 && + memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0 && + memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0 && + memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0 && + equal(get_tlist_exprs(ind1->indextlist, true), + get_tlist_exprs(ind2->indextlist, true)); +} + + +/* + * The below macros are used for simple_copy_indexinfo_to_parent which is so + * customized that I don't want to put it to copyfuncs.c. So copy it here. + */ +#define COPY_POINTER_FIELD(fldname, sz) \ + do { \ + Size _size = (sz); \ + newnode->fldname = palloc(_size); \ + memcpy(newnode->fldname, from->fldname, _size); \ + } while (0) + +#define COPY_NODE_FIELD(fldname) \ + (newnode->fldname = copyObjectImpl(from->fldname)) + +#define COPY_SCALAR_FIELD(fldname) \ + (newnode->fldname = from->fldname) + + +/* + * simple_copy_indexinfo_to_parent (from partition) + * Copy the IndexInfo from child relation to parent relation with some modification, + * which is used to test: + * 1. If the same index exists in all the childrels. + * 2. If the parentrel->reltarget/basicrestrict info matches this index. + */ +static IndexOptInfo * +simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from) +{ + IndexOptInfo *newnode = makeNode(IndexOptInfo); + AppendRelInfo *appinfo = find_appinfo_by_child(root, from->rel->relid); + ListCell *lc; + int idx = 0; + + COPY_SCALAR_FIELD(ncolumns); + COPY_SCALAR_FIELD(nkeycolumns); + COPY_SCALAR_FIELD(unique); + COPY_SCALAR_FIELD(immediate); + /* We just need to know if it is NIL or not */ + COPY_SCALAR_FIELD(indpred); + COPY_SCALAR_FIELD(predOK); + COPY_POINTER_FIELD(indexkeys, from->ncolumns * sizeof(int)); + COPY_POINTER_FIELD(indexcollations, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opfamily, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opcintype, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(sortopfamily, from->ncolumns * sizeof(Oid)); + COPY_NODE_FIELD(indextlist); + + /* Convert index exprs on child expr to expr on parent */ + foreach(lc, newnode->indextlist) + { + TargetEntry *tle = lfirst_node(TargetEntry, lc); + /* Index on expression is ignored */ + Assert(IsA(tle->expr, Var)); + tle->expr = (Expr *) find_parent_var(appinfo, (Var *) tle->expr); + newnode->indexkeys[idx] = castNode(Var, tle->expr)->varattno; + idx++; + } + newnode->rel = parentrel; + return newnode; +} + +/* + * adjust_partition_unique_indexlist + * + * global_unique_indexes: At the beginning, it contains the copy & modified + * unique index from the first partition. And then check if each index in it still + * exists in the following partitions. If no, remove it. at last, it has an + * index list which exists in all the partitions. + */ +static void +adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_indexes) +{ + ListCell *lc, *lc2; + foreach(lc, *global_unique_indexes) + { + IndexOptInfo *g_ind = lfirst_node(IndexOptInfo, lc); + bool found_in_child = false; + + foreach(lc2, childrel->indexlist) + { + IndexOptInfo *p_ind = lfirst_node(IndexOptInfo, lc2); + IndexOptInfo *p_ind_copy; + if (!p_ind->unique || !p_ind->immediate || + (p_ind->indpred != NIL && !p_ind->predOK)) + continue; + p_ind_copy = simple_copy_indexinfo_to_parent(root, parentrel, p_ind); + if (simple_indexinfo_equal(p_ind_copy, g_ind)) + { + found_in_child = true; + break; + } + } + if (!found_in_child) + /* The index doesn't exist in childrel, remove it from global_unique_indexes */ + *global_unique_indexes = foreach_delete_current(*global_unique_indexes, lc); + } +} + +/* Helper function for groupres/distinctrel */ +static void +add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgroups) +{ + Query *parse = root->parse; + List *exprs; + + /* + * XXX: If there are some vars which is not in current levelsup, the semantic is + * imprecise, should we avoid it or not? levelsup = 1 is just a demo, maybe we need to + * check every level other than 0, if so, looks we have to write another + * pull_var_walker. + */ + List *upper_vars = pull_vars_of_level((Node*)sortgroups, 1); + + if (upper_vars != NIL) + return; + + exprs = get_sortgrouplist_exprs(sortgroups, parse->targetList); + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(exprs, + false /* sortgroupclause can't be multi_nullvals */)); +} + + +/* + * add_combined_uniquekey + * The combination of both UniqueKeys is a valid UniqueKey for joinrel no matter + * the jointype. + */ +bool +add_combined_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype) +{ + + ListCell *lc1, *lc2; + + /* Either side has multi_nullvals or we have outer join, + * the combined UniqueKey has multi_nullvals */ + bool multi_nullvals = outer_ukey->multi_nullvals || + inner_ukey->multi_nullvals || IS_OUTER_JOIN(jointype); + + /* The only case we can get onerow joinrel after join */ + if (relation_is_onerow(outer_rel) + && relation_is_onerow(inner_rel) + && jointype == JOIN_INNER) + { + add_uniquekey_for_onerow(joinrel); + return true; + } + + foreach(lc1, get_exprs_from_uniquekey(root, joinrel, outer_rel, outer_ukey)) + { + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, inner_rel, inner_ukey)) + { + List *exprs = list_concat_copy(lfirst_node(List, lc1), lfirst_node(List, lc2)); + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(exprs, + multi_nullvals)); + } + } + return false; +} diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index 424d25cbd5..8d8e493f5c 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -2387,6 +2387,8 @@ grouping_planner(PlannerInfo *root, bool inheritance_update, add_path(final_rel, path); } + simple_copy_uniquekeys(current_rel, final_rel); + /* * Generate partial paths for final_rel, too, if outer query levels might * be able to make use of them. @@ -3899,6 +3901,8 @@ create_grouping_paths(PlannerInfo *root, } set_cheapest(grouped_rel); + + populate_grouprel_uniquekeys(root, grouped_rel, input_rel); return grouped_rel; } @@ -4621,7 +4625,7 @@ create_window_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(window_rel); - + simple_copy_uniquekeys(input_rel, window_rel); return window_rel; } @@ -4935,7 +4939,7 @@ create_distinct_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(distinct_rel); - + populate_distinctrel_uniquekeys(root, input_rel, distinct_rel); return distinct_rel; } @@ -5196,6 +5200,8 @@ create_ordered_paths(PlannerInfo *root, */ Assert(ordered_rel->pathlist != NIL); + simple_copy_uniquekeys(input_rel, ordered_rel); + return ordered_rel; } @@ -6073,6 +6079,9 @@ adjust_paths_for_srfs(PlannerInfo *root, RelOptInfo *rel, if (list_length(targets) == 1) return; + /* UniqueKey is not valid after handling the SRF. */ + rel->uniquekeys = NIL; + /* * Stack SRF-evaluation nodes atop each path for the rel. * diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c index becdcbb872..b7626545bf 100644 --- a/src/backend/optimizer/prep/prepunion.c +++ b/src/backend/optimizer/prep/prepunion.c @@ -689,6 +689,8 @@ generate_union_paths(SetOperationStmt *op, PlannerInfo *root, /* Undo effects of possibly forcing tuple_fraction to 0 */ root->tuple_fraction = save_fraction; + /* Add the UniqueKeys */ + populate_unionrel_uniquekeys(root, result_rel); return result_rel; } diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c index 86922a273c..6817c9c787 100644 --- a/src/backend/optimizer/util/appendinfo.c +++ b/src/backend/optimizer/util/appendinfo.c @@ -746,3 +746,47 @@ find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos) } return appinfos; } + +/* + * find_appinfo_by_child + * + */ +AppendRelInfo * +find_appinfo_by_child(PlannerInfo *root, Index child_index) +{ + ListCell *lc; + foreach(lc, root->append_rel_list) + { + AppendRelInfo *appinfo = lfirst_node(AppendRelInfo, lc); + if (appinfo->child_relid == child_index) + return appinfo; + } + elog(ERROR, "parent relation cant be found"); + return NULL; +} + +/* + * find_parent_var + * + */ +Var * +find_parent_var(AppendRelInfo *appinfo, Var *child_var) +{ + ListCell *lc; + Var *res = NULL; + Index attno = 1; + foreach(lc, appinfo->translated_vars) + { + Node *child_node = lfirst(lc); + if (equal(child_node, child_var)) + { + res = copyObject(child_var); + res->varattno = attno; + res->varno = appinfo->parent_relid; + } + attno++; + } + if (res == NULL) + elog(ERROR, "parent var can't be found."); + return res; +} diff --git a/src/backend/optimizer/util/inherit.c b/src/backend/optimizer/util/inherit.c index be1c9ddd96..3eec1f4d74 100644 --- a/src/backend/optimizer/util/inherit.c +++ b/src/backend/optimizer/util/inherit.c @@ -736,6 +736,7 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, { Node *onecq = (Node *) lfirst(lc2); bool pseudoconstant; + RestrictInfo *child_rinfo; /* check for pseudoconstant (no Vars or volatile functions) */ pseudoconstant = @@ -747,14 +748,15 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, root->hasPseudoConstantQuals = true; } /* reconstitute RestrictInfo with appropriate properties */ - childquals = lappend(childquals, - make_restrictinfo(root, - (Expr *) onecq, - rinfo->is_pushed_down, - rinfo->outerjoin_delayed, - pseudoconstant, - rinfo->security_level, - NULL, NULL, NULL)); + child_rinfo = make_restrictinfo(root, + (Expr *) onecq, + rinfo->is_pushed_down, + rinfo->outerjoin_delayed, + pseudoconstant, + rinfo->security_level, + NULL, NULL, NULL); + child_rinfo->mergeopfamilies = rinfo->mergeopfamilies; + childquals = lappend(childquals, child_rinfo); /* track minimum security level among child quals */ cq_min_security = Min(cq_min_security, rinfo->security_level); } diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 48a7ebfe45..4fe1824eb0 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -16,6 +16,7 @@ #include "nodes/execnodes.h" #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" extern A_Expr *makeA_Expr(A_Expr_Kind kind, List *name, @@ -106,4 +107,6 @@ extern GroupingSet *makeGroupingSet(GroupingSetKind kind, List *content, int loc extern VacuumRelation *makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols); +extern UniqueKey* makeUniqueKey(List *exprs, bool multi_nullvals); + #endif /* MAKEFUNC_H */ diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index e22df890ef..3320273ac1 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -265,6 +265,7 @@ typedef enum NodeTag T_EquivalenceMember, T_PathKey, T_PathTarget, + T_UniqueKey, T_RestrictInfo, T_IndexClause, T_PlaceHolderVar, diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index 0d61f04d27..80561b1003 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -739,6 +739,7 @@ typedef struct RelOptInfo QualCost baserestrictcost; /* cost of evaluating the above */ Index baserestrict_min_security; /* min security_level found in * baserestrictinfo */ + List *uniquekeys; /* List of UniqueKey */ List *joininfo; /* RestrictInfo structures for join clauses * involving this rel */ bool has_eclass_joins; /* T means joininfo is incomplete */ @@ -1059,6 +1060,28 @@ typedef struct PathKey } PathKey; +/* + * UniqueKey + * + * Represents the unique properties held by a RelOptInfo. + * + * exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL + * is a special case of UniqueKey, which means there is only 1 row in that + * relation. + * multi_nullvals: true means multi null values may exist in these exprs, so the + * uniqueness is not guaranteed in this case. This field is necessary for + * remove_useless_join & reduce_unique_semijoins where we don't mind these + * duplicated NULL values. It is set to true for 2 cases. One is a unique key + * from a unique index but the related column is nullable. The other one is for + * outer join. see populate_joinrel_uniquekeys for detail. + */ +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + /* * PathTarget * @@ -2496,7 +2519,7 @@ typedef enum * * flags indicating what kinds of grouping are possible. * partial_costs_set is true if the agg_partial_costs and agg_final_costs - * have been initialized. + * have been initialized. * agg_partial_costs gives partial aggregation costs. * agg_final_costs gives finalization costs. * target_parallel_safe is true if target is parallel safe. @@ -2526,8 +2549,8 @@ typedef struct * limit_tuples is an estimated bound on the number of output tuples, * or -1 if no LIMIT or couldn't estimate. * count_est and offset_est are the estimated values of the LIMIT and OFFSET - * expressions computed by preprocess_limit() (see comments for - * preprocess_limit() for more information). + * expressions computed by preprocess_limit() (see comments for + * preprocess_limit() for more information). */ typedef struct { diff --git a/src/include/nodes/pg_list.h b/src/include/nodes/pg_list.h index 404e03f132..85c65e910c 100644 --- a/src/include/nodes/pg_list.h +++ b/src/include/nodes/pg_list.h @@ -566,6 +566,16 @@ extern pg_nodiscard List *list_delete_first(List *list); extern pg_nodiscard List *list_delete_last(List *list); extern pg_nodiscard List *list_delete_nth_cell(List *list, int n); extern pg_nodiscard List *list_delete_cell(List *list, ListCell *cell); +extern bool list_is_subset(const List *members, const List *target); + +extern List *list_delete(List *list, void *datum); +extern List *list_delete_ptr(List *list, void *datum); +extern List *list_delete_int(List *list, int datum); +extern List *list_delete_oid(List *list, Oid datum); +extern List *list_delete_first(List *list); +extern List *list_delete_last(List *list); +extern List *list_delete_nth_cell(List *list, int n); +extern List *list_delete_cell(List *list, ListCell *cell); extern List *list_union(const List *list1, const List *list2); extern List *list_union_ptr(const List *list1, const List *list2); diff --git a/src/include/optimizer/appendinfo.h b/src/include/optimizer/appendinfo.h index 4cbf8c26cc..43b8d05f69 100644 --- a/src/include/optimizer/appendinfo.h +++ b/src/include/optimizer/appendinfo.h @@ -32,4 +32,7 @@ extern Relids adjust_child_relids_multilevel(PlannerInfo *root, Relids relids, extern AppendRelInfo **find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos); +extern AppendRelInfo *find_appinfo_by_child(PlannerInfo *root, Index child_index); +extern Var *find_parent_var(AppendRelInfo *appinfo, Var *child_var); + #endif /* APPENDINFO_H */ diff --git a/src/include/optimizer/optimizer.h b/src/include/optimizer/optimizer.h index d587952b7d..843aafa51e 100644 --- a/src/include/optimizer/optimizer.h +++ b/src/include/optimizer/optimizer.h @@ -23,6 +23,7 @@ #define OPTIMIZER_H #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" /* Test if an expression node represents a SRF call. Beware multiple eval! */ #define IS_SRF_CALL(node) \ @@ -168,6 +169,7 @@ extern TargetEntry *get_sortgroupref_tle(Index sortref, List *targetList); extern TargetEntry *get_sortgroupclause_tle(SortGroupClause *sgClause, List *targetList); +extern Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); extern Node *get_sortgroupclause_expr(SortGroupClause *sgClause, List *targetList); extern List *get_sortgrouplist_exprs(List *sgClauses, diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h index 035d3e1206..1adf99c9ee 100644 --- a/src/include/optimizer/paths.h +++ b/src/include/optimizer/paths.h @@ -247,5 +247,48 @@ extern PathKey *make_canonical_pathkey(PlannerInfo *root, int strategy, bool nulls_first); extern void add_paths_to_append_rel(PlannerInfo *root, RelOptInfo *rel, List *live_childrels); +extern List *select_mergejoin_clauses(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + JoinType jointype, + bool *mergejoin_allowed); + +/* + * uniquekeys.c + * Utilities for matching and building unique keys + */ +extern void populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List* unique_index_list); +extern void populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels); +extern void populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel); +extern void populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel); +extern void populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel); +extern void simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel); +extern void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel); +extern void populate_joinrel_uniquekeys(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + RelOptInfo *rel2, + List *restrictlist, + JoinType jointype); + +extern bool relation_has_uniquekeys_for(PlannerInfo *root, + RelOptInfo *rel, + List *exprs, + bool allow_multinulls); +extern bool relation_is_onerow(RelOptInfo *rel); #endif /* PATHS_H */ -- 2.26.2 --yh7cw345mbttadjz Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v38-0003-Extend-UniqueKeys.patch" ^ permalink raw reply [nested|flat] 91+ messages in thread
* [PATCH v36 2/6] Introduce UniqueKey attributes on RelOptInfo struct. @ 2020-05-11 07:50 一挃 <[email protected]> 0 siblings, 0 replies; 91+ messages in thread From: 一挃 @ 2020-05-11 07:50 UTC (permalink / raw) UniqueKey is a set of exprs on RelOptInfo which represents the exprs will be unique on the given RelOptInfo. You can see README.uniquekey for more information. --- src/backend/nodes/copyfuncs.c | 13 + src/backend/nodes/list.c | 31 + src/backend/nodes/makefuncs.c | 13 + src/backend/nodes/outfuncs.c | 11 + src/backend/nodes/readfuncs.c | 10 + src/backend/optimizer/path/Makefile | 3 +- src/backend/optimizer/path/README.uniquekey | 131 +++ src/backend/optimizer/path/allpaths.c | 10 + src/backend/optimizer/path/joinpath.c | 9 +- src/backend/optimizer/path/joinrels.c | 2 + src/backend/optimizer/path/pathkeys.c | 3 +- src/backend/optimizer/path/uniquekeys.c | 1131 +++++++++++++++++++ src/backend/optimizer/plan/planner.c | 13 +- src/backend/optimizer/prep/prepunion.c | 2 + src/backend/optimizer/util/appendinfo.c | 44 + src/backend/optimizer/util/inherit.c | 16 +- src/include/nodes/makefuncs.h | 3 + src/include/nodes/nodes.h | 1 + src/include/nodes/pathnodes.h | 29 +- src/include/nodes/pg_list.h | 2 + src/include/optimizer/appendinfo.h | 3 + src/include/optimizer/optimizer.h | 2 + src/include/optimizer/paths.h | 43 + 23 files changed, 1502 insertions(+), 23 deletions(-) create mode 100644 src/backend/optimizer/path/README.uniquekey create mode 100644 src/backend/optimizer/path/uniquekeys.c diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 89c409de66..1f50400fd2 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2273,6 +2273,16 @@ _copyPathKey(const PathKey *from) return newnode; } +static UniqueKey * +_copyUniqueKey(const UniqueKey *from) +{ + UniqueKey *newnode = makeNode(UniqueKey); + + COPY_NODE_FIELD(exprs); + COPY_SCALAR_FIELD(multi_nullvals); + + return newnode; +} /* * _copyRestrictInfo */ @@ -5152,6 +5162,9 @@ copyObjectImpl(const void *from) case T_PathKey: retval = _copyPathKey(from); break; + case T_UniqueKey: + retval = _copyUniqueKey(from); + break; case T_RestrictInfo: retval = _copyRestrictInfo(from); break; diff --git a/src/backend/nodes/list.c b/src/backend/nodes/list.c index 80fa8c84e4..a7a99b70f2 100644 --- a/src/backend/nodes/list.c +++ b/src/backend/nodes/list.c @@ -687,6 +687,37 @@ list_member_oid(const List *list, Oid datum) return false; } +/* + * return true iff every entry in "members" list is also present + * in the "target" list. + */ +bool +list_is_subset(const List *members, const List *target) +{ + const ListCell *lc1, *lc2; + + Assert(IsPointerList(members)); + Assert(IsPointerList(target)); + check_list_invariants(members); + check_list_invariants(target); + + foreach(lc1, members) + { + bool found = false; + foreach(lc2, target) + { + if (equal(lfirst(lc1), lfirst(lc2))) + { + found = true; + break; + } + } + if (!found) + return false; + } + return true; +} + /* * Delete the n'th cell (counting from 0) in list. * diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 49de285f01..646cf7c9a1 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -814,3 +814,16 @@ makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols) v->va_cols = va_cols; return v; } + + +/* + * makeUniqueKey + */ +UniqueKey* +makeUniqueKey(List *exprs, bool multi_nullvals) +{ + UniqueKey * ukey = makeNode(UniqueKey); + ukey->exprs = exprs; + ukey->multi_nullvals = multi_nullvals; + return ukey; +} diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index e2f177515d..c3a9632992 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2428,6 +2428,14 @@ _outPathKey(StringInfo str, const PathKey *node) WRITE_BOOL_FIELD(pk_nulls_first); } +static void +_outUniqueKey(StringInfo str, const UniqueKey *node) +{ + WRITE_NODE_TYPE("UNIQUEKEY"); + WRITE_NODE_FIELD(exprs); + WRITE_BOOL_FIELD(multi_nullvals); +} + static void _outPathTarget(StringInfo str, const PathTarget *node) { @@ -4127,6 +4135,9 @@ outNode(StringInfo str, const void *obj) case T_PathKey: _outPathKey(str, obj); break; + case T_UniqueKey: + _outUniqueKey(str, obj); + break; case T_PathTarget: _outPathTarget(str, obj); break; diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index 42050ab719..3a18571d0c 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -452,6 +452,14 @@ _readSetOperationStmt(void) READ_DONE(); } +static UniqueKey * +_readUniqueKey(void) +{ + READ_LOCALS(UniqueKey); + READ_NODE_FIELD(exprs); + READ_BOOL_FIELD(multi_nullvals); + READ_DONE(); +} /* * Stuff from primnodes.h. @@ -2656,6 +2664,8 @@ parseNodeString(void) return_value = _readCommonTableExpr(); else if (MATCH("SETOPERATIONSTMT", 16)) return_value = _readSetOperationStmt(); + else if (MATCH("UNIQUEKEY", 9)) + return_value = _readUniqueKey(); else if (MATCH("ALIAS", 5)) return_value = _readAlias(); else if (MATCH("RANGEVAR", 8)) diff --git a/src/backend/optimizer/path/Makefile b/src/backend/optimizer/path/Makefile index 1e199ff66f..7b9820c25f 100644 --- a/src/backend/optimizer/path/Makefile +++ b/src/backend/optimizer/path/Makefile @@ -21,6 +21,7 @@ OBJS = \ joinpath.o \ joinrels.o \ pathkeys.o \ - tidpath.o + tidpath.o \ + uniquekeys.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/optimizer/path/README.uniquekey b/src/backend/optimizer/path/README.uniquekey new file mode 100644 index 0000000000..5eac761995 --- /dev/null +++ b/src/backend/optimizer/path/README.uniquekey @@ -0,0 +1,131 @@ +1. What is UniqueKey? +We can think UniqueKey is a set of exprs for a RelOptInfo, which we are insure +that doesn't yields same result among all the rows. The simplest UniqueKey +format is primary key. + +However we define the UnqiueKey as below. + +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + +exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL +is a special case of UniqueKey, which means there is only one row in that +relation.it has a stronger semantic than others. like SELECT uk FROM t; uk is +normal unique key and may have different values. SELECT colx FROM t WHERE uk = +const. colx is unique AND we have only 1 value. This field can used for +innerrel_is_unique. this logic is handled specially in add_uniquekey_for_onerow +function. + +multi_nullvals: true means multi null values may exist in these exprs, so the +uniqueness is not guaranteed in this case. This field is necessary for +remove_useless_join & reduce_unique_semijoins where we don't mind these +duplicated NULL values. It is set to true for 2 cases. One is a unique key +from a unique index but the related column is nullable. The other one is for +outer join. see populate_joinrel_uniquekeys for detail. + + +The UniqueKey can be used at the following cases at least: +1. remove_useless_joins. +2. reduce_semianti_joins +3. remove distinct node if distinct clause is unique. +4. remove aggnode if group by clause is unique. +5. Index Skip Scan (WIP) +6. Aggregation Push Down without 2 phase aggregation if the join can't + duplicated the aggregated rows. (work in progress feature) + +2. How is it maintained? + +We have a set of populate_xxx_unqiuekeys functions to maintain the uniquekey on +various cases. xxx includes baserel, joinrel, partitionedrel, distinctrel, +groupedrel, unionrel. and we also need to convert the uniquekey from subquery +to outer relation, which is what convert_subquery_uniquekeys does. + +1. The first part is about baserel. We handled 3 cases. suppose we have Unique +Index on (a, b). + +1. SELECT a, b FROM t. UniqueKey (a, b) +2. SELECT a FROM t WHERE b = 1; UniqueKey (a) +3. SELECT .. FROM t WHERE a = 1 AND b = 1; UniqueKey (NIL). onerow case, every + column is Unique. + +2. The next part is joinrel, this part is most error-prone, we simplified the rules +like below: +1. If the relation's UniqueKey can't be duplicated after join, then is will be + still valid for the join rel. The function we used here is + innerrel_keeps_unique. The basic idea is innerrel.any_col = outer.uk. + +2. If the UnqiueKey can't keep valid via the rule 1, the combination of the + UniqueKey from both sides are valid for sure. We can prove this as: if the + unique exprs from rel1 is duplicated by rel2, the duplicated rows must + contains different unique exprs from rel2. + +More considerations about onerow: +1. If relation with one row and it can't be duplicated, it is still possible + contains mulit_nullvas after outer join. +2. If the either UniqueKey can be duplicated after join, the can get one row + only when both side is one row AND there is no outer join. +3. Whenever the onerow UniqueKey is not a valid any more, we need to convert one + row UniqueKey to normal unique key since we don't store exprs for one-row + relation. get_exprs_from_uniquekeys will be used here. + + +More considerations about multi_nullvals after join: +1. If the original UnqiueKey has multi_nullvals, the final UniqueKey will have + mulit_nullvals in any case. +2. If a unique key doesn't allow mulit_nullvals, after some outer join, it + allows some outer join. + + +3. When we comes to subquery, we need to convert_subquery_unqiuekeys just like +convert_subquery_pathkeys. Only the UniqueKey insides subquery is referenced as +a Var in outer relation will be reused. The relationship between the outerrel.Var +and subquery.exprs is built with outerel->subroot->processed_tlist. + + +4. As for the SRF functions, it will break the uniqueness of uniquekey, However it +is handled in adjust_paths_for_srfs, which happens after the query_planner. so +we will maintain the UniqueKey until there and reset it to NIL at that +places. This can't help on distinct/group by elimination cases but probably help +in some other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is +semantic correctly. + + +5. As for inherit table, we first main the UnqiueKey on childrel as well. But for +partitioned table we need to maintain 2 different kinds of +UnqiueKey. 1). UniqueKey on the parent relation 2). UniqueKey on child +relation for partition wise query. + +Example: +CREATE TABLE p (a int not null, b int not null) partition by list (a); +CREATE TABLE p0 partition of p for values in (1); +CREATE TABLE p1 partition of p for values in (2); + +create unique index p0_b on p0(b); +create unique index p1_b on p1(b); + +Now b is only unique on partition level, so the distinct can't be removed on +the following cases. SELECT DISTINCT b FROM p; + +Another example is SELECT DISTINCT a, b FROM p WHERE a = 1; Since only one +partition is chosen, the UniqueKey on child relation is same as the UniqueKey on +parent relation. + +Another usage of UniqueKey on partition level is it be helpful for +partition-wise join. + +As for the UniqueKey on parent table level, it comes with 2 different ways, +1). the UniqueKey is also derived in UniqueKey index, but the index must be same +in all the related children relations and the unique index must contains +Partition Key in it. Example: + +CREATE UNIQUE INDEX p_ab ON p(a, b); -- where a is the partition key. + +-- Query +SELECT a, b FROM p; the (a, b) is a UniqueKey of p. + +2). If the parent relation has only one childrel, the UniqueKey on childrel is + the UniqueKey on parent as well. diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index 484dab0a1a..2ad9d06d7a 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -579,6 +579,12 @@ set_plain_rel_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) */ check_index_predicates(root, rel); + /* + * Now that we've marked which partial indexes are suitable, we can now + * build the relation's unique keys. + */ + populate_baserel_uniquekeys(root, rel, rel->indexlist); + /* Mark rel with estimated output rows, width, etc */ set_baserel_size_estimates(root, rel); } @@ -1310,6 +1316,8 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, /* Add paths to the append relation. */ add_paths_to_append_rel(root, rel, live_childrels); + if (IS_PARTITIONED_REL(rel)) + populate_partitionedrel_uniquekeys(root, rel, live_childrels); } @@ -2383,6 +2391,8 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, pathkeys, required_outer)); } + convert_subquery_uniquekeys(root, rel, sub_final_rel); + /* If outer rel allows parallelism, do same for partial paths. */ if (rel->consider_parallel && bms_is_empty(required_outer)) { diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c index db54a6ba2e..ef0fd2fb0b 100644 --- a/src/backend/optimizer/path/joinpath.c +++ b/src/backend/optimizer/path/joinpath.c @@ -71,13 +71,6 @@ static void consider_parallel_mergejoin(PlannerInfo *root, static void hash_inner_and_outer(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, RelOptInfo *innerrel, JoinType jointype, JoinPathExtraData *extra); -static List *select_mergejoin_clauses(PlannerInfo *root, - RelOptInfo *joinrel, - RelOptInfo *outerrel, - RelOptInfo *innerrel, - List *restrictlist, - JoinType jointype, - bool *mergejoin_allowed); static void generate_mergejoin_paths(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *innerrel, @@ -1927,7 +1920,7 @@ hash_inner_and_outer(PlannerInfo *root, * if it is mergejoinable and involves vars from the two sub-relations * currently of interest. */ -static List * +List * select_mergejoin_clauses(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c index 2d343cd293..b9163ee8ff 100644 --- a/src/backend/optimizer/path/joinrels.c +++ b/src/backend/optimizer/path/joinrels.c @@ -924,6 +924,8 @@ populate_joinrel_with_paths(PlannerInfo *root, RelOptInfo *rel1, /* Apply partitionwise join technique, if possible. */ try_partitionwise_join(root, rel1, rel2, joinrel, sjinfo, restrictlist); + + populate_joinrel_uniquekeys(root, joinrel, rel1, rel2, restrictlist, sjinfo->jointype); } diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index ce9bf87e9b..7e596d4194 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -33,7 +33,6 @@ static bool pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys); static bool matches_boolean_partition_clause(RestrictInfo *rinfo, RelOptInfo *partrel, int partkeycol); -static Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); static bool right_merge_direction(PlannerInfo *root, PathKey *pathkey); @@ -1035,7 +1034,7 @@ convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel, * We need this to ensure that we don't return pathkeys describing values * that are unavailable above the level of the subquery scan. */ -static Var * +Var * find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle) { ListCell *lc; diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c new file mode 100644 index 0000000000..b33bcd2f32 --- /dev/null +++ b/src/backend/optimizer/path/uniquekeys.c @@ -0,0 +1,1131 @@ +/*------------------------------------------------------------------------- + * + * uniquekeys.c + * Utilities for matching and building unique keys + * + * Portions Copyright (c) 2020, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/optimizer/path/uniquekeys.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "nodes/makefuncs.h" +#include "nodes/nodeFuncs.h" +#include "optimizer/pathnode.h" +#include "optimizer/paths.h" +#include "optimizer/appendinfo.h" +#include "optimizer/optimizer.h" +#include "optimizer/tlist.h" +#include "rewrite/rewriteManip.h" + + +/* + * This struct is used to help populate_joinrel_uniquekeys. + * + * added_to_joinrel is true if a uniquekey (from outerrel or innerrel) + * has been added to joinrel. + * useful is true if the exprs of the uniquekey still appears in joinrel. + */ +typedef struct UniqueKeyContextData +{ + UniqueKey *uniquekey; + bool added_to_joinrel; + bool useful; +} *UniqueKeyContext; + +static List *initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel); +static bool innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + bool reverse); + +static List *get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals); +static List *get_exprs_from_uniquekey(RelOptInfo *joinrel, + RelOptInfo *rel1, + UniqueKey *ukey); +static void add_uniquekey_for_onerow(RelOptInfo *rel); +static bool add_combined_uniquekey(RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype); + +/* Used for unique indexes checking for partitioned table */ +static bool index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind); +static IndexOptInfo *simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from); +static bool simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2); +static void adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_index); + +/* Helper function for grouped relation and distinct relation. */ +static void add_uniquekey_from_sortgroups(PlannerInfo *root, + RelOptInfo *rel, + List *sortgroups); + +/* + * populate_baserel_uniquekeys + * Populate 'baserel' uniquekeys list by looking at the rel's unique index + * and baserestrictinfo + */ +void +populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List *indexlist) +{ + ListCell *lc; + List *matched_uniq_indexes = NIL; + + /* Attrs appears in rel->reltarget->exprs. */ + Bitmapset *used_attrs = NULL; + + List *const_exprs = NIL; + List *expr_opfamilies = NIL; + + Assert(baserel->rtekind == RTE_RELATION); + + foreach(lc, indexlist) + { + IndexOptInfo *ind = (IndexOptInfo *) lfirst(lc); + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + matched_uniq_indexes = lappend(matched_uniq_indexes, ind); + } + + if (matched_uniq_indexes == NIL) + return; + + /* Check which attrs is used in baserel->reltarget */ + pull_varattnos((Node *)baserel->reltarget->exprs, baserel->relid, &used_attrs); + + /* Check which attrno is used at a mergeable const filter */ + foreach(lc, baserel->baserestrictinfo) + { + RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); + + if (rinfo->mergeopfamilies == NIL) + continue; + + if (bms_is_empty(rinfo->left_relids)) + { + const_exprs = lappend(const_exprs, get_rightop(rinfo->clause)); + } + else if (bms_is_empty(rinfo->right_relids)) + { + const_exprs = lappend(const_exprs, get_leftop(rinfo->clause)); + } + else + continue; + + expr_opfamilies = lappend(expr_opfamilies, rinfo->mergeopfamilies); + } + + foreach(lc, matched_uniq_indexes) + { + bool multi_nullvals, useful; + List *exprs = get_exprs_from_uniqueindex(lfirst_node(IndexOptInfo, lc), + const_exprs, + expr_opfamilies, + used_attrs, + &useful, + &multi_nullvals); + if (useful) + { + if (exprs == NIL) + { + /* All the columns in Unique Index matched with a restrictinfo */ + add_uniquekey_for_onerow(baserel); + return; + } + baserel->uniquekeys = lappend(baserel->uniquekeys, + makeUniqueKey(exprs, multi_nullvals)); + } + } +} + + +/* + * populate_partitionedrel_uniquekeys + * The UniqueKey on partitionrel comes from 2 cases: + * 1). Only one partition is involved in this query, the unique key can be + * copied to parent rel from childrel. + * 2). There are some unique index which includes partition key and exists + * in all the related partitions. + * We never mind rule 2 if we hit rule 1. + */ + +void +populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels) +{ + ListCell *lc; + List *global_uniq_indexlist = NIL; + RelOptInfo *childrel; + bool is_first = true; + + Assert(IS_PARTITIONED_REL(rel)); + + if (childrels == NIL) + return; + + /* + * If there is only one partition used in this query, the UniqueKey in childrel is + * still valid in parent level, but we need convert the format from child expr to + * parent expr. + */ + if (list_length(childrels) == 1) + { + /* Check for Rule 1 */ + RelOptInfo *childrel = linitial_node(RelOptInfo, childrels); + ListCell *lc; + Assert(childrel->reloptkind == RELOPT_OTHER_MEMBER_REL); + if (relation_is_onerow(childrel)) + { + add_uniquekey_for_onerow(rel); + return; + } + + foreach(lc, childrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + AppendRelInfo *appinfo = find_appinfo_by_child(root, childrel->relid); + List *parent_exprs = NIL; + bool can_reuse = true; + ListCell *lc2; + foreach(lc2, ukey->exprs) + { + Var *var = (Var *)lfirst(lc2); + /* + * If the expr comes from a expression, it is hard to build the expression + * in parent so ignore that case for now. + */ + if(!IsA(var, Var)) + { + can_reuse = false; + break; + } + /* Convert it to parent var */ + parent_exprs = lappend(parent_exprs, find_parent_var(appinfo, var)); + } + if (can_reuse) + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(parent_exprs, + ukey->multi_nullvals)); + } + } + else + { + /* Check for rule 2 */ + childrel = linitial_node(RelOptInfo, childrels); + foreach(lc, childrel->indexlist) + { + IndexOptInfo *ind = lfirst(lc); + IndexOptInfo *modified_index; + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + + /* + * During simple_copy_indexinfo_to_parent, we need to convert var from + * child var to parent var, index on expression is too complex to handle. + * so ignore it for now. + */ + if (ind->indexprs != NIL) + continue; + + modified_index = simple_copy_indexinfo_to_parent(root, rel, ind); + /* + * If the unique index doesn't contain partkey, then it is unique + * on this partition only, so it is useless for us. + */ + if (!index_constains_partkey(rel, modified_index)) + continue; + + global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index); + } + + if (global_uniq_indexlist != NIL) + { + foreach(lc, childrels) + { + RelOptInfo *child = lfirst(lc); + if (is_first) + { + is_first = false; + continue; + } + adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist); + } + /* Now we have a list of unique index which are exactly same on all childrels, + * Set the UniqueKey just like it is non-partition table + */ + populate_baserel_uniquekeys(root, rel, global_uniq_indexlist); + } + } +} + + +/* + * populate_distinctrel_uniquekeys + */ +void +populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel) +{ + /* The unique key before the distinct is still valid. */ + distinctrel->uniquekeys = list_copy(inputrel->uniquekeys); + add_uniquekey_from_sortgroups(root, distinctrel, root->parse->distinctClause); +} + +/* + * populate_grouprel_uniquekeys + */ +void +populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel) + +{ + Query *parse = root->parse; + bool input_ukey_added = false; + ListCell *lc; + + if (relation_is_onerow(inputrel)) + { + add_uniquekey_for_onerow(grouprel); + return; + } + if (parse->groupingSets) + return; + + /* A Normal group by without grouping set. */ + if (parse->groupClause) + { + /* + * Current even the groupby clause is Unique already, but if query has aggref + * We have to create grouprel still. To keep the UnqiueKey short, we will check + * the UniqueKey of input_rel still valid, if so we reuse it. + */ + foreach(lc, inputrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (list_is_subset(ukey->exprs, grouprel->reltarget->exprs)) + { + grouprel->uniquekeys = lappend(grouprel->uniquekeys, + ukey); + input_ukey_added = true; + } + } + if (!input_ukey_added) + /* + * group by clause must be a super-set of grouprel->reltarget->exprs except the + * aggregation expr, so if such exprs is unique already, no bother to generate + * new uniquekey for group by exprs. + */ + add_uniquekey_from_sortgroups(root, + grouprel, + root->parse->groupClause); + } + else + /* It has aggregation but without a group by, so only one row returned */ + add_uniquekey_for_onerow(grouprel); +} + +/* + * simple_copy_uniquekeys + * Using a function for the one-line code makes us easy to check where we simply + * copied the uniquekey. + */ +void +simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel) +{ + newrel->uniquekeys = oldrel->uniquekeys; +} + +/* + * populate_unionrel_uniquekeys + */ +void +populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel) +{ + ListCell *lc; + List *exprs = NIL; + + Assert(unionrel->uniquekeys == NIL); + + foreach(lc, unionrel->reltarget->exprs) + { + exprs = lappend(exprs, lfirst(lc)); + } + + if (exprs == NIL) + /* SQL: select union select; is valid, we need to handle it here. */ + add_uniquekey_for_onerow(unionrel); + else + unionrel->uniquekeys = lappend(unionrel->uniquekeys, + makeUniqueKey(exprs,false)); + +} + +/* + * populate_joinrel_uniquekeys + * + * populate uniquekeys for joinrel. We will check each relation to see if its + * UniqueKey is still valid via innerrel_keeps_unique, if so, we add it to + * joinrel. The multi_nullvals field will be changed to true for some outer + * join cases and one-row UniqueKey needs to be converted to normal UniqueKey + * for the same case as well. + * For the uniquekey in either baserel which can't be unique after join, we still + * check to see if combination of UniqueKeys from both side is still useful for us. + * if yes, we add it to joinrel as well. + */ +void +populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *outerrel, RelOptInfo *innerrel, + List *restrictlist, JoinType jointype) +{ + ListCell *lc, *lc2; + List *clause_list = NIL; + List *outerrel_ukey_ctx; + List *innerrel_ukey_ctx; + bool inner_onerow, outer_onerow; + bool mergejoin_allowed; + + /* Care about the outerrel relation only for SEMI/ANTI join */ + if (jointype == JOIN_SEMI || jointype == JOIN_ANTI) + { + foreach(lc, outerrel->uniquekeys) + { + UniqueKey *uniquekey = lfirst_node(UniqueKey, lc); + if (list_is_subset(uniquekey->exprs, joinrel->reltarget->exprs)) + joinrel->uniquekeys = lappend(joinrel->uniquekeys, uniquekey); + } + return; + } + + Assert(jointype == JOIN_LEFT || jointype == JOIN_FULL || jointype == JOIN_INNER); + + /* Fast path */ + if (innerrel->uniquekeys == NIL || outerrel->uniquekeys == NIL) + return; + + inner_onerow = relation_is_onerow(innerrel); + outer_onerow = relation_is_onerow(outerrel); + + outerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(outerrel); + innerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(innerrel); + + clause_list = select_mergejoin_clauses(root, joinrel, outerrel, innerrel, + restrictlist, jointype, + &mergejoin_allowed); + + if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true /* reverse */)) + { + bool outer_impact = jointype == JOIN_FULL; + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + /* Outer relation has one row, and the unique key is not duplicated after join, + * the joinrel will still has one row unless the jointype == JOIN_FULL. + */ + if (outer_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (outer_onerow) + { + /* + * The onerow outerrel becomes multi rows and multi_nullvals + * will be changed to true. We also need to set the exprs correctly since it + * can't be NIL any more. + */ + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(joinrel, outerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Change multi_nullvals to true due to the full join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, true)); + else + /* Just reuse it */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + } + ctx->added_to_joinrel = true; + } + } + + if (innerrel_keeps_unique(root, outerrel, innerrel, clause_list, false)) + { + bool outer_impact = jointype == JOIN_FULL || jointype == JOIN_LEFT;; + + foreach(lc, innerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + if (inner_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (inner_onerow) + { + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(joinrel, innerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Need to change multi_nullvals to true due to the outer join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, + true)); + else + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + + } + ctx->added_to_joinrel = true; + } + } + + /* + * The combination of the UniqueKey from both sides is unique as well regardless + * of join type, but no bother to add it if its subset has been added to joinrel + * already or it is not useful for the joinrel. + */ + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx1 = (UniqueKeyContext) lfirst(lc); + if (ctx1->added_to_joinrel || !ctx1->useful) + continue; + foreach(lc2, innerrel_ukey_ctx) + { + UniqueKeyContext ctx2 = (UniqueKeyContext) lfirst(lc2); + if (ctx2->added_to_joinrel || !ctx2->useful) + continue; + if (add_combined_uniquekey(joinrel, outerrel, innerrel, + ctx1->uniquekey, ctx2->uniquekey, + jointype)) + /* If we set a onerow UniqueKey to joinrel, we don't need other. */ + return; + } + } +} + + +/* + * convert_subquery_uniquekeys + * + * Covert the UniqueKey in subquery to outer relation. + */ +void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel) +{ + ListCell *lc; + + if (sub_final_rel->uniquekeys == NIL) + return; + + if (relation_is_onerow(sub_final_rel)) + { + add_uniquekey_for_onerow(currel); + return; + } + + Assert(currel->subroot != NULL); + + foreach(lc, sub_final_rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + ListCell *lc; + List *exprs = NIL; + bool ukey_useful = true; + + /* One row case is handled above */ + Assert(ukey->exprs != NIL); + foreach(lc, ukey->exprs) + { + Var *var; + TargetEntry *tle = tlist_member(lfirst(lc), + currel->subroot->processed_tlist); + if (tle == NULL) + { + ukey_useful = false; + break; + } + var = find_var_for_subquery_tle(currel, tle); + if (var == NULL) + { + ukey_useful = false; + break; + } + exprs = lappend(exprs, var); + } + + if (ukey_useful) + currel->uniquekeys = lappend(currel->uniquekeys, + makeUniqueKey(exprs, + ukey->multi_nullvals)); + + } +} + +/* + * innerrel_keeps_unique + * + * Check if Unique key of the innerrel is valid after join. innerrel's UniqueKey + * will be still valid if innerrel's any-column mergeop outrerel's uniquekey + * exists in clause_list. + * + * Note: the clause_list must be a list of mergeable restrictinfo already. + */ +static bool +innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *clause_list, + bool reverse) +{ + ListCell *lc, *lc2, *lc3; + + if (outerrel->uniquekeys == NIL || innerrel->uniquekeys == NIL) + return false; + + /* Check if there is outerrel's uniquekey in mergeable clause. */ + foreach(lc, outerrel->uniquekeys) + { + List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs; + bool clauselist_matchs_all_exprs = true; + foreach(lc2, outer_uq_exprs) + { + Node *outer_uq_expr = lfirst(lc2); + bool find_uq_expr_in_clauselist = false; + foreach(lc3, clause_list) + { + RestrictInfo *rinfo = lfirst_node(RestrictInfo, lc3); + Node *outer_expr; + if (reverse) + outer_expr = rinfo->outer_is_left ? get_rightop(rinfo->clause) : get_leftop(rinfo->clause); + else + outer_expr = rinfo->outer_is_left ? get_leftop(rinfo->clause) : get_rightop(rinfo->clause); + if (equal(outer_expr, outer_uq_expr)) + { + find_uq_expr_in_clauselist = true; + break; + } + } + if (!find_uq_expr_in_clauselist) + { + /* No need to check the next exprs in the current uniquekey */ + clauselist_matchs_all_exprs = false; + break; + } + } + + if (clauselist_matchs_all_exprs) + return true; + } + return false; +} + + +/* + * relation_is_onerow + * Check if it is a one-row relation by checking UniqueKey. + */ +bool +relation_is_onerow(RelOptInfo *rel) +{ + UniqueKey *ukey; + if (rel->uniquekeys == NIL) + return false; + ukey = linitial_node(UniqueKey, rel->uniquekeys); + return ukey->exprs == NIL && list_length(rel->uniquekeys) == 1; +} + +/* + * relation_has_uniquekeys_for + * Returns true if we have proofs that 'rel' cannot return multiple rows with + * the same values in each of 'exprs'. Otherwise returns false. + */ +bool +relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel, + List *exprs, bool allow_multinulls) +{ + ListCell *lc; + + /* + * For UniqueKey->onerow case, the uniquekey->exprs is empty as well + * so we can't rely on list_is_subset to handle this special cases + */ + if (exprs == NIL) + return false; + + foreach(lc, rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (ukey->multi_nullvals && !allow_multinulls) + continue; + if (list_is_subset(ukey->exprs, exprs)) + return true; + } + return false; +} + + +/* + * get_exprs_from_uniqueindex + * + * Return a list of exprs which is unique. set useful to false if this + * unique index is not useful for us. + */ +static List * +get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals) +{ + List *exprs = NIL; + ListCell *indexpr_item; + int c = 0; + + *useful = true; + *multi_nullvals = false; + + indexpr_item = list_head(unique_index->indexprs); + for(c = 0; c < unique_index->ncolumns; c++) + { + int attr = unique_index->indexkeys[c]; + Expr *expr; + bool matched_const = false; + ListCell *lc1, *lc2; + + if(attr > 0) + { + expr = list_nth_node(TargetEntry, unique_index->indextlist, c)->expr; + } + else if (attr == 0) + { + /* Expression index */ + expr = lfirst(indexpr_item); + indexpr_item = lnext(unique_index->indexprs, indexpr_item); + } + else /* attr < 0 */ + { + /* Index on system column is not supported */ + Assert(false); + } + + /* + * Check index_col = Const case with regarding to opfamily checking + * If we can remove the index_col from the final UniqueKey->exprs. + */ + forboth(lc1, const_exprs, lc2, const_expr_opfamilies) + { + if (list_member_oid((List *)lfirst(lc2), unique_index->opfamily[c]) + && match_index_to_operand((Node *) lfirst(lc1), c, unique_index)) + { + matched_const = true; + break; + } + } + + if (matched_const) + continue; + + /* Check if the indexed expr is used in rel */ + if (attr > 0) + { + /* + * Normal Indexed column, if the col is not used, then the index is useless + * for uniquekey. + */ + attr -= FirstLowInvalidHeapAttributeNumber; + + if (!bms_is_member(attr, used_varattrs)) + { + *useful = false; + break; + } + } + else if (!list_member(unique_index->rel->reltarget->exprs, expr)) + { + /* Expression index but the expression is not used in rel */ + *useful = false; + break; + } + + /* check not null property. */ + if (attr == 0) + { + /* We never know if a expression yields null or not */ + *multi_nullvals = true; + } + else if (!bms_is_member(attr, unique_index->rel->notnullattrs) + && !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber, + unique_index->rel->notnullattrs)) + { + *multi_nullvals = true; + } + + exprs = lappend(exprs, expr); + } + return exprs; +} + + +/* + * add_uniquekey_for_onerow + * If we are sure that the relation only returns one row, then all the columns + * are unique. However we don't need to create UniqueKey for every column, we + * just set exprs = NIL and overwrites all the other UniqueKey on this RelOptInfo + * since this one has strongest semantics. + */ +void +add_uniquekey_for_onerow(RelOptInfo *rel) +{ + /* + * We overwrite the previous UniqueKey on purpose since this one has the + * strongest semantic. + */ + rel->uniquekeys = list_make1(makeUniqueKey(NIL, false)); +} + + +/* + * initililze_uniquecontext_for_joinrel + * Return a List of UniqueKeyContext for an inputrel + */ +static List * +initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel) +{ + List *res = NIL; + ListCell *lc; + foreach(lc, inputrel->uniquekeys) + { + UniqueKeyContext context; + context = palloc(sizeof(struct UniqueKeyContextData)); + context->uniquekey = lfirst_node(UniqueKey, lc); + context->added_to_joinrel = false; + context->useful = true; + res = lappend(res, context); + } + return res; +} + + +/* + * get_exprs_from_uniquekey + * Unify the way of get List of exprs from a one-row UniqueKey or + * normal UniqueKey. for the onerow case, every expr in rel1 is a valid + * UniqueKey. Return a List of exprs. + * + * rel1: The relation which you want to get the exprs. + * ukey: The UniqueKey you want to get the exprs. + */ +static List * +get_exprs_from_uniquekey(RelOptInfo *joinrel, RelOptInfo *rel1, UniqueKey *ukey) +{ + ListCell *lc; + bool onerow = rel1 != NULL && relation_is_onerow(rel1); + + List *res = NIL; + Assert(onerow || ukey); + if (onerow) + { + /* Only cares about the exprs still exist in joinrel */ + foreach(lc, joinrel->reltarget->exprs) + { + Bitmapset *relids = pull_varnos(lfirst(lc)); + if (bms_is_subset(relids, rel1->relids)) + { + res = lappend(res, list_make1(lfirst(lc))); + } + } + } + else + { + res = list_make1(ukey->exprs); + } + return res; +} + +/* + * Partitioned table Unique Keys. + * The partition table unique key is maintained as: + * 1. The index must be unique as usual. + * 2. The index must contains partition key. + * 3. The index must exist on all the child rel. see simple_indexinfo_equal for + * how we compare it. + */ + +/* + * index_constains_partkey + * return true if the index contains the partiton key. + */ +static bool +index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind) +{ + ListCell *lc; + int i; + Assert(IS_PARTITIONED_REL(partrel)); + Assert(partrel->part_scheme->partnatts > 0); + + for(i = 0; i < partrel->part_scheme->partnatts; i++) + { + Node *part_expr = linitial(partrel->partexprs[i]); + bool found_in_index = false; + foreach(lc, ind->indextlist) + { + Expr *index_expr = lfirst_node(TargetEntry, lc)->expr; + if (equal(index_expr, part_expr)) + { + found_in_index = true; + break; + } + } + if (!found_in_index) + return false; + } + return true; +} + +/* + * simple_indexinfo_equal + * + * Used to check if the 2 index is same as each other. The index here + * is COPIED from childrel and did some tiny changes(see + * simple_copy_indexinfo_to_parent) + */ +static bool +simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2) +{ + Size oid_cmp_len = sizeof(Oid) * ind1->ncolumns; + + return ind1->ncolumns == ind2->ncolumns && + ind1->unique == ind2->unique && + memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0 && + memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0 && + memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0 && + memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0 && + equal(get_tlist_exprs(ind1->indextlist, true), + get_tlist_exprs(ind2->indextlist, true)); +} + + +/* + * The below macros are used for simple_copy_indexinfo_to_parent which is so + * customized that I don't want to put it to copyfuncs.c. So copy it here. + */ +#define COPY_POINTER_FIELD(fldname, sz) \ + do { \ + Size _size = (sz); \ + newnode->fldname = palloc(_size); \ + memcpy(newnode->fldname, from->fldname, _size); \ + } while (0) + +#define COPY_NODE_FIELD(fldname) \ + (newnode->fldname = copyObjectImpl(from->fldname)) + +#define COPY_SCALAR_FIELD(fldname) \ + (newnode->fldname = from->fldname) + + +/* + * simple_copy_indexinfo_to_parent (from partition) + * Copy the IndexInfo from child relation to parent relation with some modification, + * which is used to test: + * 1. If the same index exists in all the childrels. + * 2. If the parentrel->reltarget/basicrestrict info matches this index. + */ +static IndexOptInfo * +simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from) +{ + IndexOptInfo *newnode = makeNode(IndexOptInfo); + AppendRelInfo *appinfo = find_appinfo_by_child(root, from->rel->relid); + ListCell *lc; + int idx = 0; + + COPY_SCALAR_FIELD(ncolumns); + COPY_SCALAR_FIELD(nkeycolumns); + COPY_SCALAR_FIELD(unique); + COPY_SCALAR_FIELD(immediate); + /* We just need to know if it is NIL or not */ + COPY_SCALAR_FIELD(indpred); + COPY_SCALAR_FIELD(predOK); + COPY_POINTER_FIELD(indexkeys, from->ncolumns * sizeof(int)); + COPY_POINTER_FIELD(indexcollations, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opfamily, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opcintype, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(sortopfamily, from->ncolumns * sizeof(Oid)); + COPY_NODE_FIELD(indextlist); + + /* Convert index exprs on child expr to expr on parent */ + foreach(lc, newnode->indextlist) + { + TargetEntry *tle = lfirst_node(TargetEntry, lc); + /* Index on expression is ignored */ + Assert(IsA(tle->expr, Var)); + tle->expr = (Expr *) find_parent_var(appinfo, (Var *) tle->expr); + newnode->indexkeys[idx] = castNode(Var, tle->expr)->varattno; + idx++; + } + newnode->rel = parentrel; + return newnode; +} + +/* + * adjust_partition_unique_indexlist + * + * global_unique_indexes: At the beginning, it contains the copy & modified + * unique index from the first partition. And then check if each index in it still + * exists in the following partitions. If no, remove it. at last, it has an + * index list which exists in all the partitions. + */ +static void +adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_indexes) +{ + ListCell *lc, *lc2; + foreach(lc, *global_unique_indexes) + { + IndexOptInfo *g_ind = lfirst_node(IndexOptInfo, lc); + bool found_in_child = false; + + foreach(lc2, childrel->indexlist) + { + IndexOptInfo *p_ind = lfirst_node(IndexOptInfo, lc2); + IndexOptInfo *p_ind_copy; + if (!p_ind->unique || !p_ind->immediate || + (p_ind->indpred != NIL && !p_ind->predOK)) + continue; + p_ind_copy = simple_copy_indexinfo_to_parent(root, parentrel, p_ind); + if (simple_indexinfo_equal(p_ind_copy, g_ind)) + { + found_in_child = true; + break; + } + } + if (!found_in_child) + /* The index doesn't exist in childrel, remove it from global_unique_indexes */ + *global_unique_indexes = foreach_delete_current(*global_unique_indexes, lc); + } +} + +/* Helper function for groupres/distinctrel */ +static void +add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgroups) +{ + Query *parse = root->parse; + List *exprs; + + /* + * XXX: If there are some vars which is not in current levelsup, the semantic is + * imprecise, should we avoid it or not? levelsup = 1 is just a demo, maybe we need to + * check every level other than 0, if so, looks we have to write another + * pull_var_walker. + */ + List *upper_vars = pull_vars_of_level((Node*)sortgroups, 1); + + if (upper_vars != NIL) + return; + + exprs = get_sortgrouplist_exprs(sortgroups, parse->targetList); + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(exprs, + false /* sortgroupclause can't be multi_nullvals */)); +} + + +/* + * add_combined_uniquekey + * The combination of both UniqueKeys is a valid UniqueKey for joinrel no matter + * the jointype. + */ +bool +add_combined_uniquekey(RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype) +{ + + ListCell *lc1, *lc2; + + /* Either side has multi_nullvals or we have outer join, + * the combined UniqueKey has multi_nullvals */ + bool multi_nullvals = outer_ukey->multi_nullvals || + inner_ukey->multi_nullvals || IS_OUTER_JOIN(jointype); + + /* The only case we can get onerow joinrel after join */ + if (relation_is_onerow(outer_rel) + && relation_is_onerow(inner_rel) + && jointype == JOIN_INNER) + { + add_uniquekey_for_onerow(joinrel); + return true; + } + + foreach(lc1, get_exprs_from_uniquekey(joinrel, outer_rel, outer_ukey)) + { + foreach(lc2, get_exprs_from_uniquekey(joinrel, inner_rel, inner_ukey)) + { + List *exprs = list_concat_copy(lfirst_node(List, lc1), lfirst_node(List, lc2)); + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(exprs, + multi_nullvals)); + } + } + return false; +} diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index b406d41e91..0551ae0512 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -2389,6 +2389,8 @@ grouping_planner(PlannerInfo *root, bool inheritance_update, add_path(final_rel, path); } + simple_copy_uniquekeys(current_rel, final_rel); + /* * Generate partial paths for final_rel, too, if outer query levels might * be able to make use of them. @@ -3899,6 +3901,8 @@ create_grouping_paths(PlannerInfo *root, } set_cheapest(grouped_rel); + + populate_grouprel_uniquekeys(root, grouped_rel, input_rel); return grouped_rel; } @@ -4615,7 +4619,7 @@ create_window_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(window_rel); - + simple_copy_uniquekeys(input_rel, window_rel); return window_rel; } @@ -4911,7 +4915,7 @@ create_distinct_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(distinct_rel); - + populate_distinctrel_uniquekeys(root, input_rel, distinct_rel); return distinct_rel; } @@ -5172,6 +5176,8 @@ create_ordered_paths(PlannerInfo *root, */ Assert(ordered_rel->pathlist != NIL); + simple_copy_uniquekeys(input_rel, ordered_rel); + return ordered_rel; } @@ -6049,6 +6055,9 @@ adjust_paths_for_srfs(PlannerInfo *root, RelOptInfo *rel, if (list_length(targets) == 1) return; + /* UniqueKey is not valid after handling the SRF. */ + rel->uniquekeys = NIL; + /* * Stack SRF-evaluation nodes atop each path for the rel. * diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c index 951aed80e7..e94e92937c 100644 --- a/src/backend/optimizer/prep/prepunion.c +++ b/src/backend/optimizer/prep/prepunion.c @@ -689,6 +689,8 @@ generate_union_paths(SetOperationStmt *op, PlannerInfo *root, /* Undo effects of possibly forcing tuple_fraction to 0 */ root->tuple_fraction = save_fraction; + /* Add the UniqueKeys */ + populate_unionrel_uniquekeys(root, result_rel); return result_rel; } diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c index d722063cf3..44c37ecffc 100644 --- a/src/backend/optimizer/util/appendinfo.c +++ b/src/backend/optimizer/util/appendinfo.c @@ -746,3 +746,47 @@ find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos) } return appinfos; } + +/* + * find_appinfo_by_child + * + */ +AppendRelInfo * +find_appinfo_by_child(PlannerInfo *root, Index child_index) +{ + ListCell *lc; + foreach(lc, root->append_rel_list) + { + AppendRelInfo *appinfo = lfirst_node(AppendRelInfo, lc); + if (appinfo->child_relid == child_index) + return appinfo; + } + elog(ERROR, "parent relation cant be found"); + return NULL; +} + +/* + * find_parent_var + * + */ +Var * +find_parent_var(AppendRelInfo *appinfo, Var *child_var) +{ + ListCell *lc; + Var *res = NULL; + Index attno = 1; + foreach(lc, appinfo->translated_vars) + { + Node *child_node = lfirst(lc); + if (equal(child_node, child_var)) + { + res = copyObject(child_var); + res->varattno = attno; + res->varno = appinfo->parent_relid; + } + attno++; + } + if (res == NULL) + elog(ERROR, "parent var can't be found."); + return res; +} diff --git a/src/backend/optimizer/util/inherit.c b/src/backend/optimizer/util/inherit.c index 3132fd35a5..d66b40ec50 100644 --- a/src/backend/optimizer/util/inherit.c +++ b/src/backend/optimizer/util/inherit.c @@ -736,6 +736,7 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, { Node *onecq = (Node *) lfirst(lc2); bool pseudoconstant; + RestrictInfo *child_rinfo; /* check for pseudoconstant (no Vars or volatile functions) */ pseudoconstant = @@ -747,13 +748,14 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, root->hasPseudoConstantQuals = true; } /* reconstitute RestrictInfo with appropriate properties */ - childquals = lappend(childquals, - make_restrictinfo((Expr *) onecq, - rinfo->is_pushed_down, - rinfo->outerjoin_delayed, - pseudoconstant, - rinfo->security_level, - NULL, NULL, NULL)); + child_rinfo = make_restrictinfo((Expr *) onecq, + rinfo->is_pushed_down, + rinfo->outerjoin_delayed, + pseudoconstant, + rinfo->security_level, + NULL, NULL, NULL); + child_rinfo->mergeopfamilies = rinfo->mergeopfamilies; + childquals = lappend(childquals, child_rinfo); /* track minimum security level among child quals */ cq_min_security = Min(cq_min_security, rinfo->security_level); } diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 31d9aedeeb..c83f17acb7 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -16,6 +16,7 @@ #include "nodes/execnodes.h" #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" extern A_Expr *makeA_Expr(A_Expr_Kind kind, List *name, @@ -105,4 +106,6 @@ extern GroupingSet *makeGroupingSet(GroupingSetKind kind, List *content, int loc extern VacuumRelation *makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols); +extern UniqueKey* makeUniqueKey(List *exprs, bool multi_nullvals); + #endif /* MAKEFUNC_H */ diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 381d84b4e4..41110ed888 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -264,6 +264,7 @@ typedef enum NodeTag T_EquivalenceMember, T_PathKey, T_PathTarget, + T_UniqueKey, T_RestrictInfo, T_IndexClause, T_PlaceHolderVar, diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index 9e3ebd488a..02e4458bef 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -730,6 +730,7 @@ typedef struct RelOptInfo QualCost baserestrictcost; /* cost of evaluating the above */ Index baserestrict_min_security; /* min security_level found in * baserestrictinfo */ + List *uniquekeys; /* List of UniqueKey */ List *joininfo; /* RestrictInfo structures for join clauses * involving this rel */ bool has_eclass_joins; /* T means joininfo is incomplete */ @@ -1047,6 +1048,28 @@ typedef struct PathKey } PathKey; +/* + * UniqueKey + * + * Represents the unique properties held by a RelOptInfo. + * + * exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL + * is a special case of UniqueKey, which means there is only 1 row in that + * relation. + * multi_nullvals: true means multi null values may exist in these exprs, so the + * uniqueness is not guaranteed in this case. This field is necessary for + * remove_useless_join & reduce_unique_semijoins where we don't mind these + * duplicated NULL values. It is set to true for 2 cases. One is a unique key + * from a unique index but the related column is nullable. The other one is for + * outer join. see populate_joinrel_uniquekeys for detail. + */ +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + /* * PathTarget * @@ -2473,7 +2496,7 @@ typedef enum * * flags indicating what kinds of grouping are possible. * partial_costs_set is true if the agg_partial_costs and agg_final_costs - * have been initialized. + * have been initialized. * agg_partial_costs gives partial aggregation costs. * agg_final_costs gives finalization costs. * target_parallel_safe is true if target is parallel safe. @@ -2503,8 +2526,8 @@ typedef struct * limit_tuples is an estimated bound on the number of output tuples, * or -1 if no LIMIT or couldn't estimate. * count_est and offset_est are the estimated values of the LIMIT and OFFSET - * expressions computed by preprocess_limit() (see comments for - * preprocess_limit() for more information). + * expressions computed by preprocess_limit() (see comments for + * preprocess_limit() for more information). */ typedef struct { diff --git a/src/include/nodes/pg_list.h b/src/include/nodes/pg_list.h index 14ea2766ad..621f54a9f8 100644 --- a/src/include/nodes/pg_list.h +++ b/src/include/nodes/pg_list.h @@ -528,6 +528,8 @@ extern bool list_member_ptr(const List *list, const void *datum); extern bool list_member_int(const List *list, int datum); extern bool list_member_oid(const List *list, Oid datum); +extern bool list_is_subset(const List *members, const List *target); + extern List *list_delete(List *list, void *datum); extern List *list_delete_ptr(List *list, void *datum); extern List *list_delete_int(List *list, int datum); diff --git a/src/include/optimizer/appendinfo.h b/src/include/optimizer/appendinfo.h index d6a27a60dd..e87c92a054 100644 --- a/src/include/optimizer/appendinfo.h +++ b/src/include/optimizer/appendinfo.h @@ -32,4 +32,7 @@ extern Relids adjust_child_relids_multilevel(PlannerInfo *root, Relids relids, extern AppendRelInfo **find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos); +extern AppendRelInfo *find_appinfo_by_child(PlannerInfo *root, Index child_index); +extern Var *find_parent_var(AppendRelInfo *appinfo, Var *child_var); + #endif /* APPENDINFO_H */ diff --git a/src/include/optimizer/optimizer.h b/src/include/optimizer/optimizer.h index 3e4171056e..9445141263 100644 --- a/src/include/optimizer/optimizer.h +++ b/src/include/optimizer/optimizer.h @@ -23,6 +23,7 @@ #define OPTIMIZER_H #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" /* * We don't want to include nodes/pathnodes.h here, because non-planner @@ -156,6 +157,7 @@ extern TargetEntry *get_sortgroupref_tle(Index sortref, List *targetList); extern TargetEntry *get_sortgroupclause_tle(SortGroupClause *sgClause, List *targetList); +extern Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); extern Node *get_sortgroupclause_expr(SortGroupClause *sgClause, List *targetList); extern List *get_sortgrouplist_exprs(List *sgClauses, diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h index 10b6e81079..9217a8d6c6 100644 --- a/src/include/optimizer/paths.h +++ b/src/include/optimizer/paths.h @@ -240,5 +240,48 @@ extern PathKey *make_canonical_pathkey(PlannerInfo *root, int strategy, bool nulls_first); extern void add_paths_to_append_rel(PlannerInfo *root, RelOptInfo *rel, List *live_childrels); +extern List *select_mergejoin_clauses(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + JoinType jointype, + bool *mergejoin_allowed); + +/* + * uniquekeys.c + * Utilities for matching and building unique keys + */ +extern void populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List* unique_index_list); +extern void populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels); +extern void populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel); +extern void populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel); +extern void populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel); +extern void simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel); +extern void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel); +extern void populate_joinrel_uniquekeys(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + RelOptInfo *rel2, + List *restrictlist, + JoinType jointype); + +extern bool relation_has_uniquekeys_for(PlannerInfo *root, + RelOptInfo *rel, + List *exprs, + bool allow_multinulls); +extern bool relation_is_onerow(RelOptInfo *rel); #endif /* PATHS_H */ -- 2.21.0 --op7mf2i72xk5yu7c Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v36-0003-Extend-UniqueKeys.patch" ^ permalink raw reply [nested|flat] 91+ messages in thread
* [PATCH v38 2/6] Introduce UniqueKey attributes on RelOptInfo struct. @ 2020-05-11 07:50 一挃 <[email protected]> 0 siblings, 0 replies; 91+ messages in thread From: 一挃 @ 2020-05-11 07:50 UTC (permalink / raw) UniqueKey is a set of exprs on RelOptInfo which represents the exprs will be unique on the given RelOptInfo. You can see README.uniquekey for more information. --- src/backend/nodes/copyfuncs.c | 13 + src/backend/nodes/list.c | 31 + src/backend/nodes/makefuncs.c | 13 + src/backend/nodes/outfuncs.c | 11 + src/backend/nodes/readfuncs.c | 10 + src/backend/optimizer/path/Makefile | 3 +- src/backend/optimizer/path/README.uniquekey | 131 +++ src/backend/optimizer/path/allpaths.c | 10 + src/backend/optimizer/path/joinpath.c | 9 +- src/backend/optimizer/path/joinrels.c | 2 + src/backend/optimizer/path/pathkeys.c | 3 +- src/backend/optimizer/path/uniquekeys.c | 1135 +++++++++++++++++++ src/backend/optimizer/plan/planner.c | 13 +- src/backend/optimizer/prep/prepunion.c | 2 + src/backend/optimizer/util/appendinfo.c | 44 + src/backend/optimizer/util/inherit.c | 18 +- src/include/nodes/makefuncs.h | 3 + src/include/nodes/nodes.h | 1 + src/include/nodes/pathnodes.h | 29 +- src/include/nodes/pg_list.h | 10 + src/include/optimizer/appendinfo.h | 3 + src/include/optimizer/optimizer.h | 2 + src/include/optimizer/paths.h | 43 + 23 files changed, 1515 insertions(+), 24 deletions(-) create mode 100644 src/backend/optimizer/path/README.uniquekey create mode 100644 src/backend/optimizer/path/uniquekeys.c diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..75c1c5e824 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2296,6 +2296,16 @@ _copyPathKey(const PathKey *from) return newnode; } +static UniqueKey * +_copyUniqueKey(const UniqueKey *from) +{ + UniqueKey *newnode = makeNode(UniqueKey); + + COPY_NODE_FIELD(exprs); + COPY_SCALAR_FIELD(multi_nullvals); + + return newnode; +} /* * _copyRestrictInfo */ @@ -5220,6 +5230,9 @@ copyObjectImpl(const void *from) case T_PathKey: retval = _copyPathKey(from); break; + case T_UniqueKey: + retval = _copyUniqueKey(from); + break; case T_RestrictInfo: retval = _copyRestrictInfo(from); break; diff --git a/src/backend/nodes/list.c b/src/backend/nodes/list.c index dbf6b30233..ca099495a1 100644 --- a/src/backend/nodes/list.c +++ b/src/backend/nodes/list.c @@ -702,6 +702,37 @@ list_member_oid(const List *list, Oid datum) return false; } +/* + * return true iff every entry in "members" list is also present + * in the "target" list. + */ +bool +list_is_subset(const List *members, const List *target) +{ + const ListCell *lc1, *lc2; + + Assert(IsPointerList(members)); + Assert(IsPointerList(target)); + check_list_invariants(members); + check_list_invariants(target); + + foreach(lc1, members) + { + bool found = false; + foreach(lc2, target) + { + if (equal(lfirst(lc1), lfirst(lc2))) + { + found = true; + break; + } + } + if (!found) + return false; + } + return true; +} + /* * Delete the n'th cell (counting from 0) in list. * diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 01c110cd2f..40415d0f5b 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -815,3 +815,16 @@ makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols) v->va_cols = va_cols; return v; } + + +/* + * makeUniqueKey + */ +UniqueKey* +makeUniqueKey(List *exprs, bool multi_nullvals) +{ + UniqueKey * ukey = makeNode(UniqueKey); + ukey->exprs = exprs; + ukey->multi_nullvals = multi_nullvals; + return ukey; +} diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..44154cde6a 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2456,6 +2456,14 @@ _outPathKey(StringInfo str, const PathKey *node) WRITE_BOOL_FIELD(pk_nulls_first); } +static void +_outUniqueKey(StringInfo str, const UniqueKey *node) +{ + WRITE_NODE_TYPE("UNIQUEKEY"); + WRITE_NODE_FIELD(exprs); + WRITE_BOOL_FIELD(multi_nullvals); +} + static void _outPathTarget(StringInfo str, const PathTarget *node) { @@ -4198,6 +4206,9 @@ outNode(StringInfo str, const void *obj) case T_PathKey: _outPathKey(str, obj); break; + case T_UniqueKey: + _outUniqueKey(str, obj); + break; case T_PathTarget: _outPathTarget(str, obj); break; diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index c5e136e9c3..b3e212bf1c 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -492,6 +492,14 @@ _readSetOperationStmt(void) READ_DONE(); } +static UniqueKey * +_readUniqueKey(void) +{ + READ_LOCALS(UniqueKey); + READ_NODE_FIELD(exprs); + READ_BOOL_FIELD(multi_nullvals); + READ_DONE(); +} /* * Stuff from primnodes.h. @@ -2717,6 +2725,8 @@ parseNodeString(void) return_value = _readCommonTableExpr(); else if (MATCH("SETOPERATIONSTMT", 16)) return_value = _readSetOperationStmt(); + else if (MATCH("UNIQUEKEY", 9)) + return_value = _readUniqueKey(); else if (MATCH("ALIAS", 5)) return_value = _readAlias(); else if (MATCH("RANGEVAR", 8)) diff --git a/src/backend/optimizer/path/Makefile b/src/backend/optimizer/path/Makefile index 1e199ff66f..7b9820c25f 100644 --- a/src/backend/optimizer/path/Makefile +++ b/src/backend/optimizer/path/Makefile @@ -21,6 +21,7 @@ OBJS = \ joinpath.o \ joinrels.o \ pathkeys.o \ - tidpath.o + tidpath.o \ + uniquekeys.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/optimizer/path/README.uniquekey b/src/backend/optimizer/path/README.uniquekey new file mode 100644 index 0000000000..5eac761995 --- /dev/null +++ b/src/backend/optimizer/path/README.uniquekey @@ -0,0 +1,131 @@ +1. What is UniqueKey? +We can think UniqueKey is a set of exprs for a RelOptInfo, which we are insure +that doesn't yields same result among all the rows. The simplest UniqueKey +format is primary key. + +However we define the UnqiueKey as below. + +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + +exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL +is a special case of UniqueKey, which means there is only one row in that +relation.it has a stronger semantic than others. like SELECT uk FROM t; uk is +normal unique key and may have different values. SELECT colx FROM t WHERE uk = +const. colx is unique AND we have only 1 value. This field can used for +innerrel_is_unique. this logic is handled specially in add_uniquekey_for_onerow +function. + +multi_nullvals: true means multi null values may exist in these exprs, so the +uniqueness is not guaranteed in this case. This field is necessary for +remove_useless_join & reduce_unique_semijoins where we don't mind these +duplicated NULL values. It is set to true for 2 cases. One is a unique key +from a unique index but the related column is nullable. The other one is for +outer join. see populate_joinrel_uniquekeys for detail. + + +The UniqueKey can be used at the following cases at least: +1. remove_useless_joins. +2. reduce_semianti_joins +3. remove distinct node if distinct clause is unique. +4. remove aggnode if group by clause is unique. +5. Index Skip Scan (WIP) +6. Aggregation Push Down without 2 phase aggregation if the join can't + duplicated the aggregated rows. (work in progress feature) + +2. How is it maintained? + +We have a set of populate_xxx_unqiuekeys functions to maintain the uniquekey on +various cases. xxx includes baserel, joinrel, partitionedrel, distinctrel, +groupedrel, unionrel. and we also need to convert the uniquekey from subquery +to outer relation, which is what convert_subquery_uniquekeys does. + +1. The first part is about baserel. We handled 3 cases. suppose we have Unique +Index on (a, b). + +1. SELECT a, b FROM t. UniqueKey (a, b) +2. SELECT a FROM t WHERE b = 1; UniqueKey (a) +3. SELECT .. FROM t WHERE a = 1 AND b = 1; UniqueKey (NIL). onerow case, every + column is Unique. + +2. The next part is joinrel, this part is most error-prone, we simplified the rules +like below: +1. If the relation's UniqueKey can't be duplicated after join, then is will be + still valid for the join rel. The function we used here is + innerrel_keeps_unique. The basic idea is innerrel.any_col = outer.uk. + +2. If the UnqiueKey can't keep valid via the rule 1, the combination of the + UniqueKey from both sides are valid for sure. We can prove this as: if the + unique exprs from rel1 is duplicated by rel2, the duplicated rows must + contains different unique exprs from rel2. + +More considerations about onerow: +1. If relation with one row and it can't be duplicated, it is still possible + contains mulit_nullvas after outer join. +2. If the either UniqueKey can be duplicated after join, the can get one row + only when both side is one row AND there is no outer join. +3. Whenever the onerow UniqueKey is not a valid any more, we need to convert one + row UniqueKey to normal unique key since we don't store exprs for one-row + relation. get_exprs_from_uniquekeys will be used here. + + +More considerations about multi_nullvals after join: +1. If the original UnqiueKey has multi_nullvals, the final UniqueKey will have + mulit_nullvals in any case. +2. If a unique key doesn't allow mulit_nullvals, after some outer join, it + allows some outer join. + + +3. When we comes to subquery, we need to convert_subquery_unqiuekeys just like +convert_subquery_pathkeys. Only the UniqueKey insides subquery is referenced as +a Var in outer relation will be reused. The relationship between the outerrel.Var +and subquery.exprs is built with outerel->subroot->processed_tlist. + + +4. As for the SRF functions, it will break the uniqueness of uniquekey, However it +is handled in adjust_paths_for_srfs, which happens after the query_planner. so +we will maintain the UniqueKey until there and reset it to NIL at that +places. This can't help on distinct/group by elimination cases but probably help +in some other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is +semantic correctly. + + +5. As for inherit table, we first main the UnqiueKey on childrel as well. But for +partitioned table we need to maintain 2 different kinds of +UnqiueKey. 1). UniqueKey on the parent relation 2). UniqueKey on child +relation for partition wise query. + +Example: +CREATE TABLE p (a int not null, b int not null) partition by list (a); +CREATE TABLE p0 partition of p for values in (1); +CREATE TABLE p1 partition of p for values in (2); + +create unique index p0_b on p0(b); +create unique index p1_b on p1(b); + +Now b is only unique on partition level, so the distinct can't be removed on +the following cases. SELECT DISTINCT b FROM p; + +Another example is SELECT DISTINCT a, b FROM p WHERE a = 1; Since only one +partition is chosen, the UniqueKey on child relation is same as the UniqueKey on +parent relation. + +Another usage of UniqueKey on partition level is it be helpful for +partition-wise join. + +As for the UniqueKey on parent table level, it comes with 2 different ways, +1). the UniqueKey is also derived in UniqueKey index, but the index must be same +in all the related children relations and the unique index must contains +Partition Key in it. Example: + +CREATE UNIQUE INDEX p_ab ON p(a, b); -- where a is the partition key. + +-- Query +SELECT a, b FROM p; the (a, b) is a UniqueKey of p. + +2). If the parent relation has only one childrel, the UniqueKey on childrel is + the UniqueKey on parent as well. diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index 37b4223adb..d52ad59f89 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -579,6 +579,12 @@ set_plain_rel_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) */ check_index_predicates(root, rel); + /* + * Now that we've marked which partial indexes are suitable, we can now + * build the relation's unique keys. + */ + populate_baserel_uniquekeys(root, rel, rel->indexlist); + /* Mark rel with estimated output rows, width, etc */ set_baserel_size_estimates(root, rel); } @@ -1297,6 +1303,8 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, /* Add paths to the append relation. */ add_paths_to_append_rel(root, rel, live_childrels); + if (IS_PARTITIONED_REL(rel)) + populate_partitionedrel_uniquekeys(root, rel, live_childrels); } @@ -2304,6 +2312,8 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, pathkeys, required_outer)); } + convert_subquery_uniquekeys(root, rel, sub_final_rel); + /* If outer rel allows parallelism, do same for partial paths. */ if (rel->consider_parallel && bms_is_empty(required_outer)) { diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c index 57ce97fd53..697ac047fb 100644 --- a/src/backend/optimizer/path/joinpath.c +++ b/src/backend/optimizer/path/joinpath.c @@ -71,13 +71,6 @@ static void consider_parallel_mergejoin(PlannerInfo *root, static void hash_inner_and_outer(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, RelOptInfo *innerrel, JoinType jointype, JoinPathExtraData *extra); -static List *select_mergejoin_clauses(PlannerInfo *root, - RelOptInfo *joinrel, - RelOptInfo *outerrel, - RelOptInfo *innerrel, - List *restrictlist, - JoinType jointype, - bool *mergejoin_allowed); static void generate_mergejoin_paths(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *innerrel, @@ -1927,7 +1920,7 @@ hash_inner_and_outer(PlannerInfo *root, * if it is mergejoinable and involves vars from the two sub-relations * currently of interest. */ -static List * +List * select_mergejoin_clauses(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c index 0dbe2ac726..7271f044ec 100644 --- a/src/backend/optimizer/path/joinrels.c +++ b/src/backend/optimizer/path/joinrels.c @@ -924,6 +924,8 @@ populate_joinrel_with_paths(PlannerInfo *root, RelOptInfo *rel1, /* Apply partitionwise join technique, if possible. */ try_partitionwise_join(root, rel1, rel2, joinrel, sjinfo, restrictlist); + + populate_joinrel_uniquekeys(root, joinrel, rel1, rel2, restrictlist, sjinfo->jointype); } diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index bd9a176d7d..139278829b 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -33,7 +33,6 @@ static bool pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys); static bool matches_boolean_partition_clause(RestrictInfo *rinfo, RelOptInfo *partrel, int partkeycol); -static Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); static bool right_merge_direction(PlannerInfo *root, PathKey *pathkey); @@ -1035,7 +1034,7 @@ convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel, * We need this to ensure that we don't return pathkeys describing values * that are unavailable above the level of the subquery scan. */ -static Var * +Var * find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle) { ListCell *lc; diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c new file mode 100644 index 0000000000..77ed2b2eff --- /dev/null +++ b/src/backend/optimizer/path/uniquekeys.c @@ -0,0 +1,1135 @@ +/*------------------------------------------------------------------------- + * + * uniquekeys.c + * Utilities for matching and building unique keys + * + * Portions Copyright (c) 2020, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/optimizer/path/uniquekeys.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "nodes/makefuncs.h" +#include "nodes/nodeFuncs.h" +#include "optimizer/pathnode.h" +#include "optimizer/paths.h" +#include "optimizer/appendinfo.h" +#include "optimizer/optimizer.h" +#include "optimizer/tlist.h" +#include "rewrite/rewriteManip.h" + + +/* + * This struct is used to help populate_joinrel_uniquekeys. + * + * added_to_joinrel is true if a uniquekey (from outerrel or innerrel) + * has been added to joinrel. + * useful is true if the exprs of the uniquekey still appears in joinrel. + */ +typedef struct UniqueKeyContextData +{ + UniqueKey *uniquekey; + bool added_to_joinrel; + bool useful; +} *UniqueKeyContext; + +static List *initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel); +static bool innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + bool reverse); + +static List *get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals); +static List *get_exprs_from_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + UniqueKey *ukey); +static void add_uniquekey_for_onerow(RelOptInfo *rel); +static bool add_combined_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype); + +/* Used for unique indexes checking for partitioned table */ +static bool index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind); +static IndexOptInfo *simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from); +static bool simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2); +static void adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_index); + +/* Helper function for grouped relation and distinct relation. */ +static void add_uniquekey_from_sortgroups(PlannerInfo *root, + RelOptInfo *rel, + List *sortgroups); + +/* + * populate_baserel_uniquekeys + * Populate 'baserel' uniquekeys list by looking at the rel's unique index + * and baserestrictinfo + */ +void +populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List *indexlist) +{ + ListCell *lc; + List *matched_uniq_indexes = NIL; + + /* Attrs appears in rel->reltarget->exprs. */ + Bitmapset *used_attrs = NULL; + + List *const_exprs = NIL; + List *expr_opfamilies = NIL; + + Assert(baserel->rtekind == RTE_RELATION); + + foreach(lc, indexlist) + { + IndexOptInfo *ind = (IndexOptInfo *) lfirst(lc); + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + matched_uniq_indexes = lappend(matched_uniq_indexes, ind); + } + + if (matched_uniq_indexes == NIL) + return; + + /* Check which attrs is used in baserel->reltarget */ + pull_varattnos((Node *)baserel->reltarget->exprs, baserel->relid, &used_attrs); + + /* Check which attrno is used at a mergeable const filter */ + foreach(lc, baserel->baserestrictinfo) + { + RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); + + if (rinfo->mergeopfamilies == NIL) + continue; + + if (bms_is_empty(rinfo->left_relids)) + { + const_exprs = lappend(const_exprs, get_rightop(rinfo->clause)); + } + else if (bms_is_empty(rinfo->right_relids)) + { + const_exprs = lappend(const_exprs, get_leftop(rinfo->clause)); + } + else + continue; + + expr_opfamilies = lappend(expr_opfamilies, rinfo->mergeopfamilies); + } + + foreach(lc, matched_uniq_indexes) + { + bool multi_nullvals, useful; + List *exprs = get_exprs_from_uniqueindex(lfirst_node(IndexOptInfo, lc), + const_exprs, + expr_opfamilies, + used_attrs, + &useful, + &multi_nullvals); + if (useful) + { + if (exprs == NIL) + { + /* All the columns in Unique Index matched with a restrictinfo */ + add_uniquekey_for_onerow(baserel); + return; + } + baserel->uniquekeys = lappend(baserel->uniquekeys, + makeUniqueKey(exprs, multi_nullvals)); + } + } +} + + +/* + * populate_partitionedrel_uniquekeys + * The UniqueKey on partitionrel comes from 2 cases: + * 1). Only one partition is involved in this query, the unique key can be + * copied to parent rel from childrel. + * 2). There are some unique index which includes partition key and exists + * in all the related partitions. + * We never mind rule 2 if we hit rule 1. + */ + +void +populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels) +{ + ListCell *lc; + List *global_uniq_indexlist = NIL; + RelOptInfo *childrel; + bool is_first = true; + + Assert(IS_PARTITIONED_REL(rel)); + + if (childrels == NIL) + return; + + /* + * If there is only one partition used in this query, the UniqueKey in childrel is + * still valid in parent level, but we need convert the format from child expr to + * parent expr. + */ + if (list_length(childrels) == 1) + { + /* Check for Rule 1 */ + RelOptInfo *childrel = linitial_node(RelOptInfo, childrels); + ListCell *lc; + Assert(childrel->reloptkind == RELOPT_OTHER_MEMBER_REL); + if (relation_is_onerow(childrel)) + { + add_uniquekey_for_onerow(rel); + return; + } + + foreach(lc, childrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + AppendRelInfo *appinfo = find_appinfo_by_child(root, childrel->relid); + List *parent_exprs = NIL; + bool can_reuse = true; + ListCell *lc2; + foreach(lc2, ukey->exprs) + { + Var *var = (Var *)lfirst(lc2); + /* + * If the expr comes from a expression, it is hard to build the expression + * in parent so ignore that case for now. + */ + if(!IsA(var, Var)) + { + can_reuse = false; + break; + } + /* Convert it to parent var */ + parent_exprs = lappend(parent_exprs, find_parent_var(appinfo, var)); + } + if (can_reuse) + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(parent_exprs, + ukey->multi_nullvals)); + } + } + else + { + /* Check for rule 2 */ + childrel = linitial_node(RelOptInfo, childrels); + foreach(lc, childrel->indexlist) + { + IndexOptInfo *ind = lfirst(lc); + IndexOptInfo *modified_index; + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + + /* + * During simple_copy_indexinfo_to_parent, we need to convert var from + * child var to parent var, index on expression is too complex to handle. + * so ignore it for now. + */ + if (ind->indexprs != NIL) + continue; + + modified_index = simple_copy_indexinfo_to_parent(root, rel, ind); + /* + * If the unique index doesn't contain partkey, then it is unique + * on this partition only, so it is useless for us. + */ + if (!index_constains_partkey(rel, modified_index)) + continue; + + global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index); + } + + if (global_uniq_indexlist != NIL) + { + foreach(lc, childrels) + { + RelOptInfo *child = lfirst(lc); + if (is_first) + { + is_first = false; + continue; + } + adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist); + } + /* Now we have a list of unique index which are exactly same on all childrels, + * Set the UniqueKey just like it is non-partition table + */ + populate_baserel_uniquekeys(root, rel, global_uniq_indexlist); + } + } +} + + +/* + * populate_distinctrel_uniquekeys + */ +void +populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel) +{ + /* The unique key before the distinct is still valid. */ + distinctrel->uniquekeys = list_copy(inputrel->uniquekeys); + add_uniquekey_from_sortgroups(root, distinctrel, root->parse->distinctClause); +} + +/* + * populate_grouprel_uniquekeys + */ +void +populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel) + +{ + Query *parse = root->parse; + bool input_ukey_added = false; + ListCell *lc; + + if (relation_is_onerow(inputrel)) + { + add_uniquekey_for_onerow(grouprel); + return; + } + if (parse->groupingSets) + return; + + /* A Normal group by without grouping set. */ + if (parse->groupClause) + { + /* + * Current even the groupby clause is Unique already, but if query has aggref + * We have to create grouprel still. To keep the UnqiueKey short, we will check + * the UniqueKey of input_rel still valid, if so we reuse it. + */ + foreach(lc, inputrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (list_is_subset(ukey->exprs, grouprel->reltarget->exprs)) + { + grouprel->uniquekeys = lappend(grouprel->uniquekeys, + ukey); + input_ukey_added = true; + } + } + if (!input_ukey_added) + /* + * group by clause must be a super-set of grouprel->reltarget->exprs except the + * aggregation expr, so if such exprs is unique already, no bother to generate + * new uniquekey for group by exprs. + */ + add_uniquekey_from_sortgroups(root, + grouprel, + root->parse->groupClause); + } + else + /* It has aggregation but without a group by, so only one row returned */ + add_uniquekey_for_onerow(grouprel); +} + +/* + * simple_copy_uniquekeys + * Using a function for the one-line code makes us easy to check where we simply + * copied the uniquekey. + */ +void +simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel) +{ + newrel->uniquekeys = oldrel->uniquekeys; +} + +/* + * populate_unionrel_uniquekeys + */ +void +populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel) +{ + ListCell *lc; + List *exprs = NIL; + + Assert(unionrel->uniquekeys == NIL); + + foreach(lc, unionrel->reltarget->exprs) + { + exprs = lappend(exprs, lfirst(lc)); + } + + if (exprs == NIL) + /* SQL: select union select; is valid, we need to handle it here. */ + add_uniquekey_for_onerow(unionrel); + else + unionrel->uniquekeys = lappend(unionrel->uniquekeys, + makeUniqueKey(exprs,false)); + +} + +/* + * populate_joinrel_uniquekeys + * + * populate uniquekeys for joinrel. We will check each relation to see if its + * UniqueKey is still valid via innerrel_keeps_unique, if so, we add it to + * joinrel. The multi_nullvals field will be changed to true for some outer + * join cases and one-row UniqueKey needs to be converted to normal UniqueKey + * for the same case as well. + * For the uniquekey in either baserel which can't be unique after join, we still + * check to see if combination of UniqueKeys from both side is still useful for us. + * if yes, we add it to joinrel as well. + */ +void +populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *outerrel, RelOptInfo *innerrel, + List *restrictlist, JoinType jointype) +{ + ListCell *lc, *lc2; + List *clause_list = NIL; + List *outerrel_ukey_ctx; + List *innerrel_ukey_ctx; + bool inner_onerow, outer_onerow; + bool mergejoin_allowed; + + /* Care about the outerrel relation only for SEMI/ANTI join */ + if (jointype == JOIN_SEMI || jointype == JOIN_ANTI) + { + foreach(lc, outerrel->uniquekeys) + { + UniqueKey *uniquekey = lfirst_node(UniqueKey, lc); + if (list_is_subset(uniquekey->exprs, joinrel->reltarget->exprs)) + joinrel->uniquekeys = lappend(joinrel->uniquekeys, uniquekey); + } + return; + } + + Assert(jointype == JOIN_LEFT || jointype == JOIN_FULL || jointype == JOIN_INNER); + + /* Fast path */ + if (innerrel->uniquekeys == NIL || outerrel->uniquekeys == NIL) + return; + + inner_onerow = relation_is_onerow(innerrel); + outer_onerow = relation_is_onerow(outerrel); + + outerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(outerrel); + innerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(innerrel); + + clause_list = select_mergejoin_clauses(root, joinrel, outerrel, innerrel, + restrictlist, jointype, + &mergejoin_allowed); + + if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true /* reverse */)) + { + bool outer_impact = jointype == JOIN_FULL; + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + /* Outer relation has one row, and the unique key is not duplicated after join, + * the joinrel will still has one row unless the jointype == JOIN_FULL. + */ + if (outer_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (outer_onerow) + { + /* + * The onerow outerrel becomes multi rows and multi_nullvals + * will be changed to true. We also need to set the exprs correctly since it + * can't be NIL any more. + */ + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, outerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Change multi_nullvals to true due to the full join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, true)); + else + /* Just reuse it */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + } + ctx->added_to_joinrel = true; + } + } + + if (innerrel_keeps_unique(root, outerrel, innerrel, clause_list, false)) + { + bool outer_impact = jointype == JOIN_FULL || jointype == JOIN_LEFT;; + + foreach(lc, innerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + if (inner_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (inner_onerow) + { + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, innerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Need to change multi_nullvals to true due to the outer join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, + true)); + else + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + + } + ctx->added_to_joinrel = true; + } + } + + /* + * The combination of the UniqueKey from both sides is unique as well regardless + * of join type, but no bother to add it if its subset has been added to joinrel + * already or it is not useful for the joinrel. + */ + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx1 = (UniqueKeyContext) lfirst(lc); + if (ctx1->added_to_joinrel || !ctx1->useful) + continue; + foreach(lc2, innerrel_ukey_ctx) + { + UniqueKeyContext ctx2 = (UniqueKeyContext) lfirst(lc2); + if (ctx2->added_to_joinrel || !ctx2->useful) + continue; + if (add_combined_uniquekey(root, joinrel, outerrel, innerrel, + ctx1->uniquekey, ctx2->uniquekey, + jointype)) + /* If we set a onerow UniqueKey to joinrel, we don't need other. */ + return; + } + } +} + + +/* + * convert_subquery_uniquekeys + * + * Covert the UniqueKey in subquery to outer relation. + */ +void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel) +{ + ListCell *lc; + + if (sub_final_rel->uniquekeys == NIL) + return; + + if (relation_is_onerow(sub_final_rel)) + { + add_uniquekey_for_onerow(currel); + return; + } + + Assert(currel->subroot != NULL); + + foreach(lc, sub_final_rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + ListCell *lc; + List *exprs = NIL; + bool ukey_useful = true; + + /* One row case is handled above */ + Assert(ukey->exprs != NIL); + foreach(lc, ukey->exprs) + { + Var *var; + TargetEntry *tle = tlist_member(lfirst(lc), + currel->subroot->processed_tlist); + if (tle == NULL) + { + ukey_useful = false; + break; + } + var = find_var_for_subquery_tle(currel, tle); + if (var == NULL) + { + ukey_useful = false; + break; + } + exprs = lappend(exprs, var); + } + + if (ukey_useful) + currel->uniquekeys = lappend(currel->uniquekeys, + makeUniqueKey(exprs, + ukey->multi_nullvals)); + + } +} + +/* + * innerrel_keeps_unique + * + * Check if Unique key of the innerrel is valid after join. innerrel's UniqueKey + * will be still valid if innerrel's any-column mergeop outrerel's uniquekey + * exists in clause_list. + * + * Note: the clause_list must be a list of mergeable restrictinfo already. + */ +static bool +innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *clause_list, + bool reverse) +{ + ListCell *lc, *lc2, *lc3; + + if (outerrel->uniquekeys == NIL || innerrel->uniquekeys == NIL) + return false; + + /* Check if there is outerrel's uniquekey in mergeable clause. */ + foreach(lc, outerrel->uniquekeys) + { + List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs; + bool clauselist_matchs_all_exprs = true; + foreach(lc2, outer_uq_exprs) + { + Node *outer_uq_expr = lfirst(lc2); + bool find_uq_expr_in_clauselist = false; + foreach(lc3, clause_list) + { + RestrictInfo *rinfo = lfirst_node(RestrictInfo, lc3); + Node *outer_expr; + if (reverse) + outer_expr = rinfo->outer_is_left ? get_rightop(rinfo->clause) : get_leftop(rinfo->clause); + else + outer_expr = rinfo->outer_is_left ? get_leftop(rinfo->clause) : get_rightop(rinfo->clause); + if (equal(outer_expr, outer_uq_expr)) + { + find_uq_expr_in_clauselist = true; + break; + } + } + if (!find_uq_expr_in_clauselist) + { + /* No need to check the next exprs in the current uniquekey */ + clauselist_matchs_all_exprs = false; + break; + } + } + + if (clauselist_matchs_all_exprs) + return true; + } + return false; +} + + +/* + * relation_is_onerow + * Check if it is a one-row relation by checking UniqueKey. + */ +bool +relation_is_onerow(RelOptInfo *rel) +{ + UniqueKey *ukey; + if (rel->uniquekeys == NIL) + return false; + ukey = linitial_node(UniqueKey, rel->uniquekeys); + return ukey->exprs == NIL && list_length(rel->uniquekeys) == 1; +} + +/* + * relation_has_uniquekeys_for + * Returns true if we have proofs that 'rel' cannot return multiple rows with + * the same values in each of 'exprs'. Otherwise returns false. + */ +bool +relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel, + List *exprs, bool allow_multinulls) +{ + ListCell *lc; + + /* + * For UniqueKey->onerow case, the uniquekey->exprs is empty as well + * so we can't rely on list_is_subset to handle this special cases + */ + if (exprs == NIL) + return false; + + foreach(lc, rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (ukey->multi_nullvals && !allow_multinulls) + continue; + if (list_is_subset(ukey->exprs, exprs)) + return true; + } + return false; +} + + +/* + * get_exprs_from_uniqueindex + * + * Return a list of exprs which is unique. set useful to false if this + * unique index is not useful for us. + */ +static List * +get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals) +{ + List *exprs = NIL; + ListCell *indexpr_item; + int c = 0; + + *useful = true; + *multi_nullvals = false; + + indexpr_item = list_head(unique_index->indexprs); + for(c = 0; c < unique_index->ncolumns; c++) + { + int attr = unique_index->indexkeys[c]; + Expr *expr; + bool matched_const = false; + ListCell *lc1, *lc2; + + if(attr > 0) + { + expr = list_nth_node(TargetEntry, unique_index->indextlist, c)->expr; + } + else if (attr == 0) + { + /* Expression index */ + expr = lfirst(indexpr_item); + indexpr_item = lnext(unique_index->indexprs, indexpr_item); + } + else /* attr < 0 */ + { + /* Index on system column is not supported */ + Assert(false); + } + + /* + * Check index_col = Const case with regarding to opfamily checking + * If we can remove the index_col from the final UniqueKey->exprs. + */ + forboth(lc1, const_exprs, lc2, const_expr_opfamilies) + { + if (list_member_oid((List *)lfirst(lc2), unique_index->opfamily[c]) + && match_index_to_operand((Node *) lfirst(lc1), c, unique_index)) + { + matched_const = true; + break; + } + } + + if (matched_const) + continue; + + /* Check if the indexed expr is used in rel */ + if (attr > 0) + { + /* + * Normal Indexed column, if the col is not used, then the index is useless + * for uniquekey. + */ + attr -= FirstLowInvalidHeapAttributeNumber; + + if (!bms_is_member(attr, used_varattrs)) + { + *useful = false; + break; + } + } + else if (!list_member(unique_index->rel->reltarget->exprs, expr)) + { + /* Expression index but the expression is not used in rel */ + *useful = false; + break; + } + + /* check not null property. */ + if (attr == 0) + { + /* We never know if a expression yields null or not */ + *multi_nullvals = true; + } + else if (!bms_is_member(attr, unique_index->rel->notnullattrs) + && !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber, + unique_index->rel->notnullattrs)) + { + *multi_nullvals = true; + } + + exprs = lappend(exprs, expr); + } + return exprs; +} + + +/* + * add_uniquekey_for_onerow + * If we are sure that the relation only returns one row, then all the columns + * are unique. However we don't need to create UniqueKey for every column, we + * just set exprs = NIL and overwrites all the other UniqueKey on this RelOptInfo + * since this one has strongest semantics. + */ +void +add_uniquekey_for_onerow(RelOptInfo *rel) +{ + /* + * We overwrite the previous UniqueKey on purpose since this one has the + * strongest semantic. + */ + rel->uniquekeys = list_make1(makeUniqueKey(NIL, false)); +} + + +/* + * initililze_uniquecontext_for_joinrel + * Return a List of UniqueKeyContext for an inputrel + */ +static List * +initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel) +{ + List *res = NIL; + ListCell *lc; + foreach(lc, inputrel->uniquekeys) + { + UniqueKeyContext context; + context = palloc(sizeof(struct UniqueKeyContextData)); + context->uniquekey = lfirst_node(UniqueKey, lc); + context->added_to_joinrel = false; + context->useful = true; + res = lappend(res, context); + } + return res; +} + + +/* + * get_exprs_from_uniquekey + * Unify the way of get List of exprs from a one-row UniqueKey or + * normal UniqueKey. for the onerow case, every expr in rel1 is a valid + * UniqueKey. Return a List of exprs. + * + * rel1: The relation which you want to get the exprs. + * ukey: The UniqueKey you want to get the exprs. + */ +static List * +get_exprs_from_uniquekey(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *rel1, UniqueKey *ukey) +{ + ListCell *lc; + bool onerow = rel1 != NULL && relation_is_onerow(rel1); + + List *res = NIL; + Assert(onerow || ukey); + if (onerow) + { + /* Only cares about the exprs still exist in joinrel */ + foreach(lc, joinrel->reltarget->exprs) + { + Bitmapset *relids = pull_varnos(root, lfirst(lc)); + if (bms_is_subset(relids, rel1->relids)) + { + res = lappend(res, list_make1(lfirst(lc))); + } + } + } + else + { + res = list_make1(ukey->exprs); + } + return res; +} + +/* + * Partitioned table Unique Keys. + * The partition table unique key is maintained as: + * 1. The index must be unique as usual. + * 2. The index must contains partition key. + * 3. The index must exist on all the child rel. see simple_indexinfo_equal for + * how we compare it. + */ + +/* + * index_constains_partkey + * return true if the index contains the partiton key. + */ +static bool +index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind) +{ + ListCell *lc; + int i; + Assert(IS_PARTITIONED_REL(partrel)); + Assert(partrel->part_scheme->partnatts > 0); + + for(i = 0; i < partrel->part_scheme->partnatts; i++) + { + Node *part_expr = linitial(partrel->partexprs[i]); + bool found_in_index = false; + foreach(lc, ind->indextlist) + { + Expr *index_expr = lfirst_node(TargetEntry, lc)->expr; + if (equal(index_expr, part_expr)) + { + found_in_index = true; + break; + } + } + if (!found_in_index) + return false; + } + return true; +} + +/* + * simple_indexinfo_equal + * + * Used to check if the 2 index is same as each other. The index here + * is COPIED from childrel and did some tiny changes(see + * simple_copy_indexinfo_to_parent) + */ +static bool +simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2) +{ + Size oid_cmp_len = sizeof(Oid) * ind1->ncolumns; + + return ind1->ncolumns == ind2->ncolumns && + ind1->unique == ind2->unique && + memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0 && + memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0 && + memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0 && + memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0 && + equal(get_tlist_exprs(ind1->indextlist, true), + get_tlist_exprs(ind2->indextlist, true)); +} + + +/* + * The below macros are used for simple_copy_indexinfo_to_parent which is so + * customized that I don't want to put it to copyfuncs.c. So copy it here. + */ +#define COPY_POINTER_FIELD(fldname, sz) \ + do { \ + Size _size = (sz); \ + newnode->fldname = palloc(_size); \ + memcpy(newnode->fldname, from->fldname, _size); \ + } while (0) + +#define COPY_NODE_FIELD(fldname) \ + (newnode->fldname = copyObjectImpl(from->fldname)) + +#define COPY_SCALAR_FIELD(fldname) \ + (newnode->fldname = from->fldname) + + +/* + * simple_copy_indexinfo_to_parent (from partition) + * Copy the IndexInfo from child relation to parent relation with some modification, + * which is used to test: + * 1. If the same index exists in all the childrels. + * 2. If the parentrel->reltarget/basicrestrict info matches this index. + */ +static IndexOptInfo * +simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from) +{ + IndexOptInfo *newnode = makeNode(IndexOptInfo); + AppendRelInfo *appinfo = find_appinfo_by_child(root, from->rel->relid); + ListCell *lc; + int idx = 0; + + COPY_SCALAR_FIELD(ncolumns); + COPY_SCALAR_FIELD(nkeycolumns); + COPY_SCALAR_FIELD(unique); + COPY_SCALAR_FIELD(immediate); + /* We just need to know if it is NIL or not */ + COPY_SCALAR_FIELD(indpred); + COPY_SCALAR_FIELD(predOK); + COPY_POINTER_FIELD(indexkeys, from->ncolumns * sizeof(int)); + COPY_POINTER_FIELD(indexcollations, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opfamily, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opcintype, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(sortopfamily, from->ncolumns * sizeof(Oid)); + COPY_NODE_FIELD(indextlist); + + /* Convert index exprs on child expr to expr on parent */ + foreach(lc, newnode->indextlist) + { + TargetEntry *tle = lfirst_node(TargetEntry, lc); + /* Index on expression is ignored */ + Assert(IsA(tle->expr, Var)); + tle->expr = (Expr *) find_parent_var(appinfo, (Var *) tle->expr); + newnode->indexkeys[idx] = castNode(Var, tle->expr)->varattno; + idx++; + } + newnode->rel = parentrel; + return newnode; +} + +/* + * adjust_partition_unique_indexlist + * + * global_unique_indexes: At the beginning, it contains the copy & modified + * unique index from the first partition. And then check if each index in it still + * exists in the following partitions. If no, remove it. at last, it has an + * index list which exists in all the partitions. + */ +static void +adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_indexes) +{ + ListCell *lc, *lc2; + foreach(lc, *global_unique_indexes) + { + IndexOptInfo *g_ind = lfirst_node(IndexOptInfo, lc); + bool found_in_child = false; + + foreach(lc2, childrel->indexlist) + { + IndexOptInfo *p_ind = lfirst_node(IndexOptInfo, lc2); + IndexOptInfo *p_ind_copy; + if (!p_ind->unique || !p_ind->immediate || + (p_ind->indpred != NIL && !p_ind->predOK)) + continue; + p_ind_copy = simple_copy_indexinfo_to_parent(root, parentrel, p_ind); + if (simple_indexinfo_equal(p_ind_copy, g_ind)) + { + found_in_child = true; + break; + } + } + if (!found_in_child) + /* The index doesn't exist in childrel, remove it from global_unique_indexes */ + *global_unique_indexes = foreach_delete_current(*global_unique_indexes, lc); + } +} + +/* Helper function for groupres/distinctrel */ +static void +add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgroups) +{ + Query *parse = root->parse; + List *exprs; + + /* + * XXX: If there are some vars which is not in current levelsup, the semantic is + * imprecise, should we avoid it or not? levelsup = 1 is just a demo, maybe we need to + * check every level other than 0, if so, looks we have to write another + * pull_var_walker. + */ + List *upper_vars = pull_vars_of_level((Node*)sortgroups, 1); + + if (upper_vars != NIL) + return; + + exprs = get_sortgrouplist_exprs(sortgroups, parse->targetList); + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(exprs, + false /* sortgroupclause can't be multi_nullvals */)); +} + + +/* + * add_combined_uniquekey + * The combination of both UniqueKeys is a valid UniqueKey for joinrel no matter + * the jointype. + */ +bool +add_combined_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype) +{ + + ListCell *lc1, *lc2; + + /* Either side has multi_nullvals or we have outer join, + * the combined UniqueKey has multi_nullvals */ + bool multi_nullvals = outer_ukey->multi_nullvals || + inner_ukey->multi_nullvals || IS_OUTER_JOIN(jointype); + + /* The only case we can get onerow joinrel after join */ + if (relation_is_onerow(outer_rel) + && relation_is_onerow(inner_rel) + && jointype == JOIN_INNER) + { + add_uniquekey_for_onerow(joinrel); + return true; + } + + foreach(lc1, get_exprs_from_uniquekey(root, joinrel, outer_rel, outer_ukey)) + { + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, inner_rel, inner_ukey)) + { + List *exprs = list_concat_copy(lfirst_node(List, lc1), lfirst_node(List, lc2)); + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(exprs, + multi_nullvals)); + } + } + return false; +} diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index 424d25cbd5..8d8e493f5c 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -2387,6 +2387,8 @@ grouping_planner(PlannerInfo *root, bool inheritance_update, add_path(final_rel, path); } + simple_copy_uniquekeys(current_rel, final_rel); + /* * Generate partial paths for final_rel, too, if outer query levels might * be able to make use of them. @@ -3899,6 +3901,8 @@ create_grouping_paths(PlannerInfo *root, } set_cheapest(grouped_rel); + + populate_grouprel_uniquekeys(root, grouped_rel, input_rel); return grouped_rel; } @@ -4621,7 +4625,7 @@ create_window_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(window_rel); - + simple_copy_uniquekeys(input_rel, window_rel); return window_rel; } @@ -4935,7 +4939,7 @@ create_distinct_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(distinct_rel); - + populate_distinctrel_uniquekeys(root, input_rel, distinct_rel); return distinct_rel; } @@ -5196,6 +5200,8 @@ create_ordered_paths(PlannerInfo *root, */ Assert(ordered_rel->pathlist != NIL); + simple_copy_uniquekeys(input_rel, ordered_rel); + return ordered_rel; } @@ -6073,6 +6079,9 @@ adjust_paths_for_srfs(PlannerInfo *root, RelOptInfo *rel, if (list_length(targets) == 1) return; + /* UniqueKey is not valid after handling the SRF. */ + rel->uniquekeys = NIL; + /* * Stack SRF-evaluation nodes atop each path for the rel. * diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c index becdcbb872..b7626545bf 100644 --- a/src/backend/optimizer/prep/prepunion.c +++ b/src/backend/optimizer/prep/prepunion.c @@ -689,6 +689,8 @@ generate_union_paths(SetOperationStmt *op, PlannerInfo *root, /* Undo effects of possibly forcing tuple_fraction to 0 */ root->tuple_fraction = save_fraction; + /* Add the UniqueKeys */ + populate_unionrel_uniquekeys(root, result_rel); return result_rel; } diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c index 86922a273c..6817c9c787 100644 --- a/src/backend/optimizer/util/appendinfo.c +++ b/src/backend/optimizer/util/appendinfo.c @@ -746,3 +746,47 @@ find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos) } return appinfos; } + +/* + * find_appinfo_by_child + * + */ +AppendRelInfo * +find_appinfo_by_child(PlannerInfo *root, Index child_index) +{ + ListCell *lc; + foreach(lc, root->append_rel_list) + { + AppendRelInfo *appinfo = lfirst_node(AppendRelInfo, lc); + if (appinfo->child_relid == child_index) + return appinfo; + } + elog(ERROR, "parent relation cant be found"); + return NULL; +} + +/* + * find_parent_var + * + */ +Var * +find_parent_var(AppendRelInfo *appinfo, Var *child_var) +{ + ListCell *lc; + Var *res = NULL; + Index attno = 1; + foreach(lc, appinfo->translated_vars) + { + Node *child_node = lfirst(lc); + if (equal(child_node, child_var)) + { + res = copyObject(child_var); + res->varattno = attno; + res->varno = appinfo->parent_relid; + } + attno++; + } + if (res == NULL) + elog(ERROR, "parent var can't be found."); + return res; +} diff --git a/src/backend/optimizer/util/inherit.c b/src/backend/optimizer/util/inherit.c index be1c9ddd96..3eec1f4d74 100644 --- a/src/backend/optimizer/util/inherit.c +++ b/src/backend/optimizer/util/inherit.c @@ -736,6 +736,7 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, { Node *onecq = (Node *) lfirst(lc2); bool pseudoconstant; + RestrictInfo *child_rinfo; /* check for pseudoconstant (no Vars or volatile functions) */ pseudoconstant = @@ -747,14 +748,15 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, root->hasPseudoConstantQuals = true; } /* reconstitute RestrictInfo with appropriate properties */ - childquals = lappend(childquals, - make_restrictinfo(root, - (Expr *) onecq, - rinfo->is_pushed_down, - rinfo->outerjoin_delayed, - pseudoconstant, - rinfo->security_level, - NULL, NULL, NULL)); + child_rinfo = make_restrictinfo(root, + (Expr *) onecq, + rinfo->is_pushed_down, + rinfo->outerjoin_delayed, + pseudoconstant, + rinfo->security_level, + NULL, NULL, NULL); + child_rinfo->mergeopfamilies = rinfo->mergeopfamilies; + childquals = lappend(childquals, child_rinfo); /* track minimum security level among child quals */ cq_min_security = Min(cq_min_security, rinfo->security_level); } diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 48a7ebfe45..4fe1824eb0 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -16,6 +16,7 @@ #include "nodes/execnodes.h" #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" extern A_Expr *makeA_Expr(A_Expr_Kind kind, List *name, @@ -106,4 +107,6 @@ extern GroupingSet *makeGroupingSet(GroupingSetKind kind, List *content, int loc extern VacuumRelation *makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols); +extern UniqueKey* makeUniqueKey(List *exprs, bool multi_nullvals); + #endif /* MAKEFUNC_H */ diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index e22df890ef..3320273ac1 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -265,6 +265,7 @@ typedef enum NodeTag T_EquivalenceMember, T_PathKey, T_PathTarget, + T_UniqueKey, T_RestrictInfo, T_IndexClause, T_PlaceHolderVar, diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index 0d61f04d27..80561b1003 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -739,6 +739,7 @@ typedef struct RelOptInfo QualCost baserestrictcost; /* cost of evaluating the above */ Index baserestrict_min_security; /* min security_level found in * baserestrictinfo */ + List *uniquekeys; /* List of UniqueKey */ List *joininfo; /* RestrictInfo structures for join clauses * involving this rel */ bool has_eclass_joins; /* T means joininfo is incomplete */ @@ -1059,6 +1060,28 @@ typedef struct PathKey } PathKey; +/* + * UniqueKey + * + * Represents the unique properties held by a RelOptInfo. + * + * exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL + * is a special case of UniqueKey, which means there is only 1 row in that + * relation. + * multi_nullvals: true means multi null values may exist in these exprs, so the + * uniqueness is not guaranteed in this case. This field is necessary for + * remove_useless_join & reduce_unique_semijoins where we don't mind these + * duplicated NULL values. It is set to true for 2 cases. One is a unique key + * from a unique index but the related column is nullable. The other one is for + * outer join. see populate_joinrel_uniquekeys for detail. + */ +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + /* * PathTarget * @@ -2496,7 +2519,7 @@ typedef enum * * flags indicating what kinds of grouping are possible. * partial_costs_set is true if the agg_partial_costs and agg_final_costs - * have been initialized. + * have been initialized. * agg_partial_costs gives partial aggregation costs. * agg_final_costs gives finalization costs. * target_parallel_safe is true if target is parallel safe. @@ -2526,8 +2549,8 @@ typedef struct * limit_tuples is an estimated bound on the number of output tuples, * or -1 if no LIMIT or couldn't estimate. * count_est and offset_est are the estimated values of the LIMIT and OFFSET - * expressions computed by preprocess_limit() (see comments for - * preprocess_limit() for more information). + * expressions computed by preprocess_limit() (see comments for + * preprocess_limit() for more information). */ typedef struct { diff --git a/src/include/nodes/pg_list.h b/src/include/nodes/pg_list.h index 404e03f132..85c65e910c 100644 --- a/src/include/nodes/pg_list.h +++ b/src/include/nodes/pg_list.h @@ -566,6 +566,16 @@ extern pg_nodiscard List *list_delete_first(List *list); extern pg_nodiscard List *list_delete_last(List *list); extern pg_nodiscard List *list_delete_nth_cell(List *list, int n); extern pg_nodiscard List *list_delete_cell(List *list, ListCell *cell); +extern bool list_is_subset(const List *members, const List *target); + +extern List *list_delete(List *list, void *datum); +extern List *list_delete_ptr(List *list, void *datum); +extern List *list_delete_int(List *list, int datum); +extern List *list_delete_oid(List *list, Oid datum); +extern List *list_delete_first(List *list); +extern List *list_delete_last(List *list); +extern List *list_delete_nth_cell(List *list, int n); +extern List *list_delete_cell(List *list, ListCell *cell); extern List *list_union(const List *list1, const List *list2); extern List *list_union_ptr(const List *list1, const List *list2); diff --git a/src/include/optimizer/appendinfo.h b/src/include/optimizer/appendinfo.h index 4cbf8c26cc..43b8d05f69 100644 --- a/src/include/optimizer/appendinfo.h +++ b/src/include/optimizer/appendinfo.h @@ -32,4 +32,7 @@ extern Relids adjust_child_relids_multilevel(PlannerInfo *root, Relids relids, extern AppendRelInfo **find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos); +extern AppendRelInfo *find_appinfo_by_child(PlannerInfo *root, Index child_index); +extern Var *find_parent_var(AppendRelInfo *appinfo, Var *child_var); + #endif /* APPENDINFO_H */ diff --git a/src/include/optimizer/optimizer.h b/src/include/optimizer/optimizer.h index d587952b7d..843aafa51e 100644 --- a/src/include/optimizer/optimizer.h +++ b/src/include/optimizer/optimizer.h @@ -23,6 +23,7 @@ #define OPTIMIZER_H #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" /* Test if an expression node represents a SRF call. Beware multiple eval! */ #define IS_SRF_CALL(node) \ @@ -168,6 +169,7 @@ extern TargetEntry *get_sortgroupref_tle(Index sortref, List *targetList); extern TargetEntry *get_sortgroupclause_tle(SortGroupClause *sgClause, List *targetList); +extern Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); extern Node *get_sortgroupclause_expr(SortGroupClause *sgClause, List *targetList); extern List *get_sortgrouplist_exprs(List *sgClauses, diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h index 035d3e1206..1adf99c9ee 100644 --- a/src/include/optimizer/paths.h +++ b/src/include/optimizer/paths.h @@ -247,5 +247,48 @@ extern PathKey *make_canonical_pathkey(PlannerInfo *root, int strategy, bool nulls_first); extern void add_paths_to_append_rel(PlannerInfo *root, RelOptInfo *rel, List *live_childrels); +extern List *select_mergejoin_clauses(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + JoinType jointype, + bool *mergejoin_allowed); + +/* + * uniquekeys.c + * Utilities for matching and building unique keys + */ +extern void populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List* unique_index_list); +extern void populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels); +extern void populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel); +extern void populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel); +extern void populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel); +extern void simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel); +extern void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel); +extern void populate_joinrel_uniquekeys(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + RelOptInfo *rel2, + List *restrictlist, + JoinType jointype); + +extern bool relation_has_uniquekeys_for(PlannerInfo *root, + RelOptInfo *rel, + List *exprs, + bool allow_multinulls); +extern bool relation_is_onerow(RelOptInfo *rel); #endif /* PATHS_H */ -- 2.26.2 --yh7cw345mbttadjz Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v38-0003-Extend-UniqueKeys.patch" ^ permalink raw reply [nested|flat] 91+ messages in thread
* [PATCH v36 2/6] Introduce UniqueKey attributes on RelOptInfo struct. @ 2020-05-11 07:50 一挃 <[email protected]> 0 siblings, 0 replies; 91+ messages in thread From: 一挃 @ 2020-05-11 07:50 UTC (permalink / raw) UniqueKey is a set of exprs on RelOptInfo which represents the exprs will be unique on the given RelOptInfo. You can see README.uniquekey for more information. --- src/backend/nodes/copyfuncs.c | 13 + src/backend/nodes/list.c | 31 + src/backend/nodes/makefuncs.c | 13 + src/backend/nodes/outfuncs.c | 11 + src/backend/nodes/readfuncs.c | 10 + src/backend/optimizer/path/Makefile | 3 +- src/backend/optimizer/path/README.uniquekey | 131 +++ src/backend/optimizer/path/allpaths.c | 10 + src/backend/optimizer/path/joinpath.c | 9 +- src/backend/optimizer/path/joinrels.c | 2 + src/backend/optimizer/path/pathkeys.c | 3 +- src/backend/optimizer/path/uniquekeys.c | 1131 +++++++++++++++++++ src/backend/optimizer/plan/planner.c | 13 +- src/backend/optimizer/prep/prepunion.c | 2 + src/backend/optimizer/util/appendinfo.c | 44 + src/backend/optimizer/util/inherit.c | 16 +- src/include/nodes/makefuncs.h | 3 + src/include/nodes/nodes.h | 1 + src/include/nodes/pathnodes.h | 29 +- src/include/nodes/pg_list.h | 2 + src/include/optimizer/appendinfo.h | 3 + src/include/optimizer/optimizer.h | 2 + src/include/optimizer/paths.h | 43 + 23 files changed, 1502 insertions(+), 23 deletions(-) create mode 100644 src/backend/optimizer/path/README.uniquekey create mode 100644 src/backend/optimizer/path/uniquekeys.c diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 89c409de66..1f50400fd2 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2273,6 +2273,16 @@ _copyPathKey(const PathKey *from) return newnode; } +static UniqueKey * +_copyUniqueKey(const UniqueKey *from) +{ + UniqueKey *newnode = makeNode(UniqueKey); + + COPY_NODE_FIELD(exprs); + COPY_SCALAR_FIELD(multi_nullvals); + + return newnode; +} /* * _copyRestrictInfo */ @@ -5152,6 +5162,9 @@ copyObjectImpl(const void *from) case T_PathKey: retval = _copyPathKey(from); break; + case T_UniqueKey: + retval = _copyUniqueKey(from); + break; case T_RestrictInfo: retval = _copyRestrictInfo(from); break; diff --git a/src/backend/nodes/list.c b/src/backend/nodes/list.c index 80fa8c84e4..a7a99b70f2 100644 --- a/src/backend/nodes/list.c +++ b/src/backend/nodes/list.c @@ -687,6 +687,37 @@ list_member_oid(const List *list, Oid datum) return false; } +/* + * return true iff every entry in "members" list is also present + * in the "target" list. + */ +bool +list_is_subset(const List *members, const List *target) +{ + const ListCell *lc1, *lc2; + + Assert(IsPointerList(members)); + Assert(IsPointerList(target)); + check_list_invariants(members); + check_list_invariants(target); + + foreach(lc1, members) + { + bool found = false; + foreach(lc2, target) + { + if (equal(lfirst(lc1), lfirst(lc2))) + { + found = true; + break; + } + } + if (!found) + return false; + } + return true; +} + /* * Delete the n'th cell (counting from 0) in list. * diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 49de285f01..646cf7c9a1 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -814,3 +814,16 @@ makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols) v->va_cols = va_cols; return v; } + + +/* + * makeUniqueKey + */ +UniqueKey* +makeUniqueKey(List *exprs, bool multi_nullvals) +{ + UniqueKey * ukey = makeNode(UniqueKey); + ukey->exprs = exprs; + ukey->multi_nullvals = multi_nullvals; + return ukey; +} diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index e2f177515d..c3a9632992 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2428,6 +2428,14 @@ _outPathKey(StringInfo str, const PathKey *node) WRITE_BOOL_FIELD(pk_nulls_first); } +static void +_outUniqueKey(StringInfo str, const UniqueKey *node) +{ + WRITE_NODE_TYPE("UNIQUEKEY"); + WRITE_NODE_FIELD(exprs); + WRITE_BOOL_FIELD(multi_nullvals); +} + static void _outPathTarget(StringInfo str, const PathTarget *node) { @@ -4127,6 +4135,9 @@ outNode(StringInfo str, const void *obj) case T_PathKey: _outPathKey(str, obj); break; + case T_UniqueKey: + _outUniqueKey(str, obj); + break; case T_PathTarget: _outPathTarget(str, obj); break; diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index 42050ab719..3a18571d0c 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -452,6 +452,14 @@ _readSetOperationStmt(void) READ_DONE(); } +static UniqueKey * +_readUniqueKey(void) +{ + READ_LOCALS(UniqueKey); + READ_NODE_FIELD(exprs); + READ_BOOL_FIELD(multi_nullvals); + READ_DONE(); +} /* * Stuff from primnodes.h. @@ -2656,6 +2664,8 @@ parseNodeString(void) return_value = _readCommonTableExpr(); else if (MATCH("SETOPERATIONSTMT", 16)) return_value = _readSetOperationStmt(); + else if (MATCH("UNIQUEKEY", 9)) + return_value = _readUniqueKey(); else if (MATCH("ALIAS", 5)) return_value = _readAlias(); else if (MATCH("RANGEVAR", 8)) diff --git a/src/backend/optimizer/path/Makefile b/src/backend/optimizer/path/Makefile index 1e199ff66f..7b9820c25f 100644 --- a/src/backend/optimizer/path/Makefile +++ b/src/backend/optimizer/path/Makefile @@ -21,6 +21,7 @@ OBJS = \ joinpath.o \ joinrels.o \ pathkeys.o \ - tidpath.o + tidpath.o \ + uniquekeys.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/optimizer/path/README.uniquekey b/src/backend/optimizer/path/README.uniquekey new file mode 100644 index 0000000000..5eac761995 --- /dev/null +++ b/src/backend/optimizer/path/README.uniquekey @@ -0,0 +1,131 @@ +1. What is UniqueKey? +We can think UniqueKey is a set of exprs for a RelOptInfo, which we are insure +that doesn't yields same result among all the rows. The simplest UniqueKey +format is primary key. + +However we define the UnqiueKey as below. + +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + +exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL +is a special case of UniqueKey, which means there is only one row in that +relation.it has a stronger semantic than others. like SELECT uk FROM t; uk is +normal unique key and may have different values. SELECT colx FROM t WHERE uk = +const. colx is unique AND we have only 1 value. This field can used for +innerrel_is_unique. this logic is handled specially in add_uniquekey_for_onerow +function. + +multi_nullvals: true means multi null values may exist in these exprs, so the +uniqueness is not guaranteed in this case. This field is necessary for +remove_useless_join & reduce_unique_semijoins where we don't mind these +duplicated NULL values. It is set to true for 2 cases. One is a unique key +from a unique index but the related column is nullable. The other one is for +outer join. see populate_joinrel_uniquekeys for detail. + + +The UniqueKey can be used at the following cases at least: +1. remove_useless_joins. +2. reduce_semianti_joins +3. remove distinct node if distinct clause is unique. +4. remove aggnode if group by clause is unique. +5. Index Skip Scan (WIP) +6. Aggregation Push Down without 2 phase aggregation if the join can't + duplicated the aggregated rows. (work in progress feature) + +2. How is it maintained? + +We have a set of populate_xxx_unqiuekeys functions to maintain the uniquekey on +various cases. xxx includes baserel, joinrel, partitionedrel, distinctrel, +groupedrel, unionrel. and we also need to convert the uniquekey from subquery +to outer relation, which is what convert_subquery_uniquekeys does. + +1. The first part is about baserel. We handled 3 cases. suppose we have Unique +Index on (a, b). + +1. SELECT a, b FROM t. UniqueKey (a, b) +2. SELECT a FROM t WHERE b = 1; UniqueKey (a) +3. SELECT .. FROM t WHERE a = 1 AND b = 1; UniqueKey (NIL). onerow case, every + column is Unique. + +2. The next part is joinrel, this part is most error-prone, we simplified the rules +like below: +1. If the relation's UniqueKey can't be duplicated after join, then is will be + still valid for the join rel. The function we used here is + innerrel_keeps_unique. The basic idea is innerrel.any_col = outer.uk. + +2. If the UnqiueKey can't keep valid via the rule 1, the combination of the + UniqueKey from both sides are valid for sure. We can prove this as: if the + unique exprs from rel1 is duplicated by rel2, the duplicated rows must + contains different unique exprs from rel2. + +More considerations about onerow: +1. If relation with one row and it can't be duplicated, it is still possible + contains mulit_nullvas after outer join. +2. If the either UniqueKey can be duplicated after join, the can get one row + only when both side is one row AND there is no outer join. +3. Whenever the onerow UniqueKey is not a valid any more, we need to convert one + row UniqueKey to normal unique key since we don't store exprs for one-row + relation. get_exprs_from_uniquekeys will be used here. + + +More considerations about multi_nullvals after join: +1. If the original UnqiueKey has multi_nullvals, the final UniqueKey will have + mulit_nullvals in any case. +2. If a unique key doesn't allow mulit_nullvals, after some outer join, it + allows some outer join. + + +3. When we comes to subquery, we need to convert_subquery_unqiuekeys just like +convert_subquery_pathkeys. Only the UniqueKey insides subquery is referenced as +a Var in outer relation will be reused. The relationship between the outerrel.Var +and subquery.exprs is built with outerel->subroot->processed_tlist. + + +4. As for the SRF functions, it will break the uniqueness of uniquekey, However it +is handled in adjust_paths_for_srfs, which happens after the query_planner. so +we will maintain the UniqueKey until there and reset it to NIL at that +places. This can't help on distinct/group by elimination cases but probably help +in some other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is +semantic correctly. + + +5. As for inherit table, we first main the UnqiueKey on childrel as well. But for +partitioned table we need to maintain 2 different kinds of +UnqiueKey. 1). UniqueKey on the parent relation 2). UniqueKey on child +relation for partition wise query. + +Example: +CREATE TABLE p (a int not null, b int not null) partition by list (a); +CREATE TABLE p0 partition of p for values in (1); +CREATE TABLE p1 partition of p for values in (2); + +create unique index p0_b on p0(b); +create unique index p1_b on p1(b); + +Now b is only unique on partition level, so the distinct can't be removed on +the following cases. SELECT DISTINCT b FROM p; + +Another example is SELECT DISTINCT a, b FROM p WHERE a = 1; Since only one +partition is chosen, the UniqueKey on child relation is same as the UniqueKey on +parent relation. + +Another usage of UniqueKey on partition level is it be helpful for +partition-wise join. + +As for the UniqueKey on parent table level, it comes with 2 different ways, +1). the UniqueKey is also derived in UniqueKey index, but the index must be same +in all the related children relations and the unique index must contains +Partition Key in it. Example: + +CREATE UNIQUE INDEX p_ab ON p(a, b); -- where a is the partition key. + +-- Query +SELECT a, b FROM p; the (a, b) is a UniqueKey of p. + +2). If the parent relation has only one childrel, the UniqueKey on childrel is + the UniqueKey on parent as well. diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index 484dab0a1a..2ad9d06d7a 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -579,6 +579,12 @@ set_plain_rel_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) */ check_index_predicates(root, rel); + /* + * Now that we've marked which partial indexes are suitable, we can now + * build the relation's unique keys. + */ + populate_baserel_uniquekeys(root, rel, rel->indexlist); + /* Mark rel with estimated output rows, width, etc */ set_baserel_size_estimates(root, rel); } @@ -1310,6 +1316,8 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, /* Add paths to the append relation. */ add_paths_to_append_rel(root, rel, live_childrels); + if (IS_PARTITIONED_REL(rel)) + populate_partitionedrel_uniquekeys(root, rel, live_childrels); } @@ -2383,6 +2391,8 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, pathkeys, required_outer)); } + convert_subquery_uniquekeys(root, rel, sub_final_rel); + /* If outer rel allows parallelism, do same for partial paths. */ if (rel->consider_parallel && bms_is_empty(required_outer)) { diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c index db54a6ba2e..ef0fd2fb0b 100644 --- a/src/backend/optimizer/path/joinpath.c +++ b/src/backend/optimizer/path/joinpath.c @@ -71,13 +71,6 @@ static void consider_parallel_mergejoin(PlannerInfo *root, static void hash_inner_and_outer(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, RelOptInfo *innerrel, JoinType jointype, JoinPathExtraData *extra); -static List *select_mergejoin_clauses(PlannerInfo *root, - RelOptInfo *joinrel, - RelOptInfo *outerrel, - RelOptInfo *innerrel, - List *restrictlist, - JoinType jointype, - bool *mergejoin_allowed); static void generate_mergejoin_paths(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *innerrel, @@ -1927,7 +1920,7 @@ hash_inner_and_outer(PlannerInfo *root, * if it is mergejoinable and involves vars from the two sub-relations * currently of interest. */ -static List * +List * select_mergejoin_clauses(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c index 2d343cd293..b9163ee8ff 100644 --- a/src/backend/optimizer/path/joinrels.c +++ b/src/backend/optimizer/path/joinrels.c @@ -924,6 +924,8 @@ populate_joinrel_with_paths(PlannerInfo *root, RelOptInfo *rel1, /* Apply partitionwise join technique, if possible. */ try_partitionwise_join(root, rel1, rel2, joinrel, sjinfo, restrictlist); + + populate_joinrel_uniquekeys(root, joinrel, rel1, rel2, restrictlist, sjinfo->jointype); } diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index ce9bf87e9b..7e596d4194 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -33,7 +33,6 @@ static bool pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys); static bool matches_boolean_partition_clause(RestrictInfo *rinfo, RelOptInfo *partrel, int partkeycol); -static Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); static bool right_merge_direction(PlannerInfo *root, PathKey *pathkey); @@ -1035,7 +1034,7 @@ convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel, * We need this to ensure that we don't return pathkeys describing values * that are unavailable above the level of the subquery scan. */ -static Var * +Var * find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle) { ListCell *lc; diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c new file mode 100644 index 0000000000..b33bcd2f32 --- /dev/null +++ b/src/backend/optimizer/path/uniquekeys.c @@ -0,0 +1,1131 @@ +/*------------------------------------------------------------------------- + * + * uniquekeys.c + * Utilities for matching and building unique keys + * + * Portions Copyright (c) 2020, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/optimizer/path/uniquekeys.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "nodes/makefuncs.h" +#include "nodes/nodeFuncs.h" +#include "optimizer/pathnode.h" +#include "optimizer/paths.h" +#include "optimizer/appendinfo.h" +#include "optimizer/optimizer.h" +#include "optimizer/tlist.h" +#include "rewrite/rewriteManip.h" + + +/* + * This struct is used to help populate_joinrel_uniquekeys. + * + * added_to_joinrel is true if a uniquekey (from outerrel or innerrel) + * has been added to joinrel. + * useful is true if the exprs of the uniquekey still appears in joinrel. + */ +typedef struct UniqueKeyContextData +{ + UniqueKey *uniquekey; + bool added_to_joinrel; + bool useful; +} *UniqueKeyContext; + +static List *initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel); +static bool innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + bool reverse); + +static List *get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals); +static List *get_exprs_from_uniquekey(RelOptInfo *joinrel, + RelOptInfo *rel1, + UniqueKey *ukey); +static void add_uniquekey_for_onerow(RelOptInfo *rel); +static bool add_combined_uniquekey(RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype); + +/* Used for unique indexes checking for partitioned table */ +static bool index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind); +static IndexOptInfo *simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from); +static bool simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2); +static void adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_index); + +/* Helper function for grouped relation and distinct relation. */ +static void add_uniquekey_from_sortgroups(PlannerInfo *root, + RelOptInfo *rel, + List *sortgroups); + +/* + * populate_baserel_uniquekeys + * Populate 'baserel' uniquekeys list by looking at the rel's unique index + * and baserestrictinfo + */ +void +populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List *indexlist) +{ + ListCell *lc; + List *matched_uniq_indexes = NIL; + + /* Attrs appears in rel->reltarget->exprs. */ + Bitmapset *used_attrs = NULL; + + List *const_exprs = NIL; + List *expr_opfamilies = NIL; + + Assert(baserel->rtekind == RTE_RELATION); + + foreach(lc, indexlist) + { + IndexOptInfo *ind = (IndexOptInfo *) lfirst(lc); + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + matched_uniq_indexes = lappend(matched_uniq_indexes, ind); + } + + if (matched_uniq_indexes == NIL) + return; + + /* Check which attrs is used in baserel->reltarget */ + pull_varattnos((Node *)baserel->reltarget->exprs, baserel->relid, &used_attrs); + + /* Check which attrno is used at a mergeable const filter */ + foreach(lc, baserel->baserestrictinfo) + { + RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); + + if (rinfo->mergeopfamilies == NIL) + continue; + + if (bms_is_empty(rinfo->left_relids)) + { + const_exprs = lappend(const_exprs, get_rightop(rinfo->clause)); + } + else if (bms_is_empty(rinfo->right_relids)) + { + const_exprs = lappend(const_exprs, get_leftop(rinfo->clause)); + } + else + continue; + + expr_opfamilies = lappend(expr_opfamilies, rinfo->mergeopfamilies); + } + + foreach(lc, matched_uniq_indexes) + { + bool multi_nullvals, useful; + List *exprs = get_exprs_from_uniqueindex(lfirst_node(IndexOptInfo, lc), + const_exprs, + expr_opfamilies, + used_attrs, + &useful, + &multi_nullvals); + if (useful) + { + if (exprs == NIL) + { + /* All the columns in Unique Index matched with a restrictinfo */ + add_uniquekey_for_onerow(baserel); + return; + } + baserel->uniquekeys = lappend(baserel->uniquekeys, + makeUniqueKey(exprs, multi_nullvals)); + } + } +} + + +/* + * populate_partitionedrel_uniquekeys + * The UniqueKey on partitionrel comes from 2 cases: + * 1). Only one partition is involved in this query, the unique key can be + * copied to parent rel from childrel. + * 2). There are some unique index which includes partition key and exists + * in all the related partitions. + * We never mind rule 2 if we hit rule 1. + */ + +void +populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels) +{ + ListCell *lc; + List *global_uniq_indexlist = NIL; + RelOptInfo *childrel; + bool is_first = true; + + Assert(IS_PARTITIONED_REL(rel)); + + if (childrels == NIL) + return; + + /* + * If there is only one partition used in this query, the UniqueKey in childrel is + * still valid in parent level, but we need convert the format from child expr to + * parent expr. + */ + if (list_length(childrels) == 1) + { + /* Check for Rule 1 */ + RelOptInfo *childrel = linitial_node(RelOptInfo, childrels); + ListCell *lc; + Assert(childrel->reloptkind == RELOPT_OTHER_MEMBER_REL); + if (relation_is_onerow(childrel)) + { + add_uniquekey_for_onerow(rel); + return; + } + + foreach(lc, childrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + AppendRelInfo *appinfo = find_appinfo_by_child(root, childrel->relid); + List *parent_exprs = NIL; + bool can_reuse = true; + ListCell *lc2; + foreach(lc2, ukey->exprs) + { + Var *var = (Var *)lfirst(lc2); + /* + * If the expr comes from a expression, it is hard to build the expression + * in parent so ignore that case for now. + */ + if(!IsA(var, Var)) + { + can_reuse = false; + break; + } + /* Convert it to parent var */ + parent_exprs = lappend(parent_exprs, find_parent_var(appinfo, var)); + } + if (can_reuse) + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(parent_exprs, + ukey->multi_nullvals)); + } + } + else + { + /* Check for rule 2 */ + childrel = linitial_node(RelOptInfo, childrels); + foreach(lc, childrel->indexlist) + { + IndexOptInfo *ind = lfirst(lc); + IndexOptInfo *modified_index; + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + + /* + * During simple_copy_indexinfo_to_parent, we need to convert var from + * child var to parent var, index on expression is too complex to handle. + * so ignore it for now. + */ + if (ind->indexprs != NIL) + continue; + + modified_index = simple_copy_indexinfo_to_parent(root, rel, ind); + /* + * If the unique index doesn't contain partkey, then it is unique + * on this partition only, so it is useless for us. + */ + if (!index_constains_partkey(rel, modified_index)) + continue; + + global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index); + } + + if (global_uniq_indexlist != NIL) + { + foreach(lc, childrels) + { + RelOptInfo *child = lfirst(lc); + if (is_first) + { + is_first = false; + continue; + } + adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist); + } + /* Now we have a list of unique index which are exactly same on all childrels, + * Set the UniqueKey just like it is non-partition table + */ + populate_baserel_uniquekeys(root, rel, global_uniq_indexlist); + } + } +} + + +/* + * populate_distinctrel_uniquekeys + */ +void +populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel) +{ + /* The unique key before the distinct is still valid. */ + distinctrel->uniquekeys = list_copy(inputrel->uniquekeys); + add_uniquekey_from_sortgroups(root, distinctrel, root->parse->distinctClause); +} + +/* + * populate_grouprel_uniquekeys + */ +void +populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel) + +{ + Query *parse = root->parse; + bool input_ukey_added = false; + ListCell *lc; + + if (relation_is_onerow(inputrel)) + { + add_uniquekey_for_onerow(grouprel); + return; + } + if (parse->groupingSets) + return; + + /* A Normal group by without grouping set. */ + if (parse->groupClause) + { + /* + * Current even the groupby clause is Unique already, but if query has aggref + * We have to create grouprel still. To keep the UnqiueKey short, we will check + * the UniqueKey of input_rel still valid, if so we reuse it. + */ + foreach(lc, inputrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (list_is_subset(ukey->exprs, grouprel->reltarget->exprs)) + { + grouprel->uniquekeys = lappend(grouprel->uniquekeys, + ukey); + input_ukey_added = true; + } + } + if (!input_ukey_added) + /* + * group by clause must be a super-set of grouprel->reltarget->exprs except the + * aggregation expr, so if such exprs is unique already, no bother to generate + * new uniquekey for group by exprs. + */ + add_uniquekey_from_sortgroups(root, + grouprel, + root->parse->groupClause); + } + else + /* It has aggregation but without a group by, so only one row returned */ + add_uniquekey_for_onerow(grouprel); +} + +/* + * simple_copy_uniquekeys + * Using a function for the one-line code makes us easy to check where we simply + * copied the uniquekey. + */ +void +simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel) +{ + newrel->uniquekeys = oldrel->uniquekeys; +} + +/* + * populate_unionrel_uniquekeys + */ +void +populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel) +{ + ListCell *lc; + List *exprs = NIL; + + Assert(unionrel->uniquekeys == NIL); + + foreach(lc, unionrel->reltarget->exprs) + { + exprs = lappend(exprs, lfirst(lc)); + } + + if (exprs == NIL) + /* SQL: select union select; is valid, we need to handle it here. */ + add_uniquekey_for_onerow(unionrel); + else + unionrel->uniquekeys = lappend(unionrel->uniquekeys, + makeUniqueKey(exprs,false)); + +} + +/* + * populate_joinrel_uniquekeys + * + * populate uniquekeys for joinrel. We will check each relation to see if its + * UniqueKey is still valid via innerrel_keeps_unique, if so, we add it to + * joinrel. The multi_nullvals field will be changed to true for some outer + * join cases and one-row UniqueKey needs to be converted to normal UniqueKey + * for the same case as well. + * For the uniquekey in either baserel which can't be unique after join, we still + * check to see if combination of UniqueKeys from both side is still useful for us. + * if yes, we add it to joinrel as well. + */ +void +populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *outerrel, RelOptInfo *innerrel, + List *restrictlist, JoinType jointype) +{ + ListCell *lc, *lc2; + List *clause_list = NIL; + List *outerrel_ukey_ctx; + List *innerrel_ukey_ctx; + bool inner_onerow, outer_onerow; + bool mergejoin_allowed; + + /* Care about the outerrel relation only for SEMI/ANTI join */ + if (jointype == JOIN_SEMI || jointype == JOIN_ANTI) + { + foreach(lc, outerrel->uniquekeys) + { + UniqueKey *uniquekey = lfirst_node(UniqueKey, lc); + if (list_is_subset(uniquekey->exprs, joinrel->reltarget->exprs)) + joinrel->uniquekeys = lappend(joinrel->uniquekeys, uniquekey); + } + return; + } + + Assert(jointype == JOIN_LEFT || jointype == JOIN_FULL || jointype == JOIN_INNER); + + /* Fast path */ + if (innerrel->uniquekeys == NIL || outerrel->uniquekeys == NIL) + return; + + inner_onerow = relation_is_onerow(innerrel); + outer_onerow = relation_is_onerow(outerrel); + + outerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(outerrel); + innerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(innerrel); + + clause_list = select_mergejoin_clauses(root, joinrel, outerrel, innerrel, + restrictlist, jointype, + &mergejoin_allowed); + + if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true /* reverse */)) + { + bool outer_impact = jointype == JOIN_FULL; + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + /* Outer relation has one row, and the unique key is not duplicated after join, + * the joinrel will still has one row unless the jointype == JOIN_FULL. + */ + if (outer_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (outer_onerow) + { + /* + * The onerow outerrel becomes multi rows and multi_nullvals + * will be changed to true. We also need to set the exprs correctly since it + * can't be NIL any more. + */ + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(joinrel, outerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Change multi_nullvals to true due to the full join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, true)); + else + /* Just reuse it */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + } + ctx->added_to_joinrel = true; + } + } + + if (innerrel_keeps_unique(root, outerrel, innerrel, clause_list, false)) + { + bool outer_impact = jointype == JOIN_FULL || jointype == JOIN_LEFT;; + + foreach(lc, innerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + if (inner_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (inner_onerow) + { + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(joinrel, innerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Need to change multi_nullvals to true due to the outer join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, + true)); + else + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + + } + ctx->added_to_joinrel = true; + } + } + + /* + * The combination of the UniqueKey from both sides is unique as well regardless + * of join type, but no bother to add it if its subset has been added to joinrel + * already or it is not useful for the joinrel. + */ + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx1 = (UniqueKeyContext) lfirst(lc); + if (ctx1->added_to_joinrel || !ctx1->useful) + continue; + foreach(lc2, innerrel_ukey_ctx) + { + UniqueKeyContext ctx2 = (UniqueKeyContext) lfirst(lc2); + if (ctx2->added_to_joinrel || !ctx2->useful) + continue; + if (add_combined_uniquekey(joinrel, outerrel, innerrel, + ctx1->uniquekey, ctx2->uniquekey, + jointype)) + /* If we set a onerow UniqueKey to joinrel, we don't need other. */ + return; + } + } +} + + +/* + * convert_subquery_uniquekeys + * + * Covert the UniqueKey in subquery to outer relation. + */ +void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel) +{ + ListCell *lc; + + if (sub_final_rel->uniquekeys == NIL) + return; + + if (relation_is_onerow(sub_final_rel)) + { + add_uniquekey_for_onerow(currel); + return; + } + + Assert(currel->subroot != NULL); + + foreach(lc, sub_final_rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + ListCell *lc; + List *exprs = NIL; + bool ukey_useful = true; + + /* One row case is handled above */ + Assert(ukey->exprs != NIL); + foreach(lc, ukey->exprs) + { + Var *var; + TargetEntry *tle = tlist_member(lfirst(lc), + currel->subroot->processed_tlist); + if (tle == NULL) + { + ukey_useful = false; + break; + } + var = find_var_for_subquery_tle(currel, tle); + if (var == NULL) + { + ukey_useful = false; + break; + } + exprs = lappend(exprs, var); + } + + if (ukey_useful) + currel->uniquekeys = lappend(currel->uniquekeys, + makeUniqueKey(exprs, + ukey->multi_nullvals)); + + } +} + +/* + * innerrel_keeps_unique + * + * Check if Unique key of the innerrel is valid after join. innerrel's UniqueKey + * will be still valid if innerrel's any-column mergeop outrerel's uniquekey + * exists in clause_list. + * + * Note: the clause_list must be a list of mergeable restrictinfo already. + */ +static bool +innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *clause_list, + bool reverse) +{ + ListCell *lc, *lc2, *lc3; + + if (outerrel->uniquekeys == NIL || innerrel->uniquekeys == NIL) + return false; + + /* Check if there is outerrel's uniquekey in mergeable clause. */ + foreach(lc, outerrel->uniquekeys) + { + List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs; + bool clauselist_matchs_all_exprs = true; + foreach(lc2, outer_uq_exprs) + { + Node *outer_uq_expr = lfirst(lc2); + bool find_uq_expr_in_clauselist = false; + foreach(lc3, clause_list) + { + RestrictInfo *rinfo = lfirst_node(RestrictInfo, lc3); + Node *outer_expr; + if (reverse) + outer_expr = rinfo->outer_is_left ? get_rightop(rinfo->clause) : get_leftop(rinfo->clause); + else + outer_expr = rinfo->outer_is_left ? get_leftop(rinfo->clause) : get_rightop(rinfo->clause); + if (equal(outer_expr, outer_uq_expr)) + { + find_uq_expr_in_clauselist = true; + break; + } + } + if (!find_uq_expr_in_clauselist) + { + /* No need to check the next exprs in the current uniquekey */ + clauselist_matchs_all_exprs = false; + break; + } + } + + if (clauselist_matchs_all_exprs) + return true; + } + return false; +} + + +/* + * relation_is_onerow + * Check if it is a one-row relation by checking UniqueKey. + */ +bool +relation_is_onerow(RelOptInfo *rel) +{ + UniqueKey *ukey; + if (rel->uniquekeys == NIL) + return false; + ukey = linitial_node(UniqueKey, rel->uniquekeys); + return ukey->exprs == NIL && list_length(rel->uniquekeys) == 1; +} + +/* + * relation_has_uniquekeys_for + * Returns true if we have proofs that 'rel' cannot return multiple rows with + * the same values in each of 'exprs'. Otherwise returns false. + */ +bool +relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel, + List *exprs, bool allow_multinulls) +{ + ListCell *lc; + + /* + * For UniqueKey->onerow case, the uniquekey->exprs is empty as well + * so we can't rely on list_is_subset to handle this special cases + */ + if (exprs == NIL) + return false; + + foreach(lc, rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (ukey->multi_nullvals && !allow_multinulls) + continue; + if (list_is_subset(ukey->exprs, exprs)) + return true; + } + return false; +} + + +/* + * get_exprs_from_uniqueindex + * + * Return a list of exprs which is unique. set useful to false if this + * unique index is not useful for us. + */ +static List * +get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals) +{ + List *exprs = NIL; + ListCell *indexpr_item; + int c = 0; + + *useful = true; + *multi_nullvals = false; + + indexpr_item = list_head(unique_index->indexprs); + for(c = 0; c < unique_index->ncolumns; c++) + { + int attr = unique_index->indexkeys[c]; + Expr *expr; + bool matched_const = false; + ListCell *lc1, *lc2; + + if(attr > 0) + { + expr = list_nth_node(TargetEntry, unique_index->indextlist, c)->expr; + } + else if (attr == 0) + { + /* Expression index */ + expr = lfirst(indexpr_item); + indexpr_item = lnext(unique_index->indexprs, indexpr_item); + } + else /* attr < 0 */ + { + /* Index on system column is not supported */ + Assert(false); + } + + /* + * Check index_col = Const case with regarding to opfamily checking + * If we can remove the index_col from the final UniqueKey->exprs. + */ + forboth(lc1, const_exprs, lc2, const_expr_opfamilies) + { + if (list_member_oid((List *)lfirst(lc2), unique_index->opfamily[c]) + && match_index_to_operand((Node *) lfirst(lc1), c, unique_index)) + { + matched_const = true; + break; + } + } + + if (matched_const) + continue; + + /* Check if the indexed expr is used in rel */ + if (attr > 0) + { + /* + * Normal Indexed column, if the col is not used, then the index is useless + * for uniquekey. + */ + attr -= FirstLowInvalidHeapAttributeNumber; + + if (!bms_is_member(attr, used_varattrs)) + { + *useful = false; + break; + } + } + else if (!list_member(unique_index->rel->reltarget->exprs, expr)) + { + /* Expression index but the expression is not used in rel */ + *useful = false; + break; + } + + /* check not null property. */ + if (attr == 0) + { + /* We never know if a expression yields null or not */ + *multi_nullvals = true; + } + else if (!bms_is_member(attr, unique_index->rel->notnullattrs) + && !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber, + unique_index->rel->notnullattrs)) + { + *multi_nullvals = true; + } + + exprs = lappend(exprs, expr); + } + return exprs; +} + + +/* + * add_uniquekey_for_onerow + * If we are sure that the relation only returns one row, then all the columns + * are unique. However we don't need to create UniqueKey for every column, we + * just set exprs = NIL and overwrites all the other UniqueKey on this RelOptInfo + * since this one has strongest semantics. + */ +void +add_uniquekey_for_onerow(RelOptInfo *rel) +{ + /* + * We overwrite the previous UniqueKey on purpose since this one has the + * strongest semantic. + */ + rel->uniquekeys = list_make1(makeUniqueKey(NIL, false)); +} + + +/* + * initililze_uniquecontext_for_joinrel + * Return a List of UniqueKeyContext for an inputrel + */ +static List * +initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel) +{ + List *res = NIL; + ListCell *lc; + foreach(lc, inputrel->uniquekeys) + { + UniqueKeyContext context; + context = palloc(sizeof(struct UniqueKeyContextData)); + context->uniquekey = lfirst_node(UniqueKey, lc); + context->added_to_joinrel = false; + context->useful = true; + res = lappend(res, context); + } + return res; +} + + +/* + * get_exprs_from_uniquekey + * Unify the way of get List of exprs from a one-row UniqueKey or + * normal UniqueKey. for the onerow case, every expr in rel1 is a valid + * UniqueKey. Return a List of exprs. + * + * rel1: The relation which you want to get the exprs. + * ukey: The UniqueKey you want to get the exprs. + */ +static List * +get_exprs_from_uniquekey(RelOptInfo *joinrel, RelOptInfo *rel1, UniqueKey *ukey) +{ + ListCell *lc; + bool onerow = rel1 != NULL && relation_is_onerow(rel1); + + List *res = NIL; + Assert(onerow || ukey); + if (onerow) + { + /* Only cares about the exprs still exist in joinrel */ + foreach(lc, joinrel->reltarget->exprs) + { + Bitmapset *relids = pull_varnos(lfirst(lc)); + if (bms_is_subset(relids, rel1->relids)) + { + res = lappend(res, list_make1(lfirst(lc))); + } + } + } + else + { + res = list_make1(ukey->exprs); + } + return res; +} + +/* + * Partitioned table Unique Keys. + * The partition table unique key is maintained as: + * 1. The index must be unique as usual. + * 2. The index must contains partition key. + * 3. The index must exist on all the child rel. see simple_indexinfo_equal for + * how we compare it. + */ + +/* + * index_constains_partkey + * return true if the index contains the partiton key. + */ +static bool +index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind) +{ + ListCell *lc; + int i; + Assert(IS_PARTITIONED_REL(partrel)); + Assert(partrel->part_scheme->partnatts > 0); + + for(i = 0; i < partrel->part_scheme->partnatts; i++) + { + Node *part_expr = linitial(partrel->partexprs[i]); + bool found_in_index = false; + foreach(lc, ind->indextlist) + { + Expr *index_expr = lfirst_node(TargetEntry, lc)->expr; + if (equal(index_expr, part_expr)) + { + found_in_index = true; + break; + } + } + if (!found_in_index) + return false; + } + return true; +} + +/* + * simple_indexinfo_equal + * + * Used to check if the 2 index is same as each other. The index here + * is COPIED from childrel and did some tiny changes(see + * simple_copy_indexinfo_to_parent) + */ +static bool +simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2) +{ + Size oid_cmp_len = sizeof(Oid) * ind1->ncolumns; + + return ind1->ncolumns == ind2->ncolumns && + ind1->unique == ind2->unique && + memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0 && + memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0 && + memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0 && + memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0 && + equal(get_tlist_exprs(ind1->indextlist, true), + get_tlist_exprs(ind2->indextlist, true)); +} + + +/* + * The below macros are used for simple_copy_indexinfo_to_parent which is so + * customized that I don't want to put it to copyfuncs.c. So copy it here. + */ +#define COPY_POINTER_FIELD(fldname, sz) \ + do { \ + Size _size = (sz); \ + newnode->fldname = palloc(_size); \ + memcpy(newnode->fldname, from->fldname, _size); \ + } while (0) + +#define COPY_NODE_FIELD(fldname) \ + (newnode->fldname = copyObjectImpl(from->fldname)) + +#define COPY_SCALAR_FIELD(fldname) \ + (newnode->fldname = from->fldname) + + +/* + * simple_copy_indexinfo_to_parent (from partition) + * Copy the IndexInfo from child relation to parent relation with some modification, + * which is used to test: + * 1. If the same index exists in all the childrels. + * 2. If the parentrel->reltarget/basicrestrict info matches this index. + */ +static IndexOptInfo * +simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from) +{ + IndexOptInfo *newnode = makeNode(IndexOptInfo); + AppendRelInfo *appinfo = find_appinfo_by_child(root, from->rel->relid); + ListCell *lc; + int idx = 0; + + COPY_SCALAR_FIELD(ncolumns); + COPY_SCALAR_FIELD(nkeycolumns); + COPY_SCALAR_FIELD(unique); + COPY_SCALAR_FIELD(immediate); + /* We just need to know if it is NIL or not */ + COPY_SCALAR_FIELD(indpred); + COPY_SCALAR_FIELD(predOK); + COPY_POINTER_FIELD(indexkeys, from->ncolumns * sizeof(int)); + COPY_POINTER_FIELD(indexcollations, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opfamily, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opcintype, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(sortopfamily, from->ncolumns * sizeof(Oid)); + COPY_NODE_FIELD(indextlist); + + /* Convert index exprs on child expr to expr on parent */ + foreach(lc, newnode->indextlist) + { + TargetEntry *tle = lfirst_node(TargetEntry, lc); + /* Index on expression is ignored */ + Assert(IsA(tle->expr, Var)); + tle->expr = (Expr *) find_parent_var(appinfo, (Var *) tle->expr); + newnode->indexkeys[idx] = castNode(Var, tle->expr)->varattno; + idx++; + } + newnode->rel = parentrel; + return newnode; +} + +/* + * adjust_partition_unique_indexlist + * + * global_unique_indexes: At the beginning, it contains the copy & modified + * unique index from the first partition. And then check if each index in it still + * exists in the following partitions. If no, remove it. at last, it has an + * index list which exists in all the partitions. + */ +static void +adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_indexes) +{ + ListCell *lc, *lc2; + foreach(lc, *global_unique_indexes) + { + IndexOptInfo *g_ind = lfirst_node(IndexOptInfo, lc); + bool found_in_child = false; + + foreach(lc2, childrel->indexlist) + { + IndexOptInfo *p_ind = lfirst_node(IndexOptInfo, lc2); + IndexOptInfo *p_ind_copy; + if (!p_ind->unique || !p_ind->immediate || + (p_ind->indpred != NIL && !p_ind->predOK)) + continue; + p_ind_copy = simple_copy_indexinfo_to_parent(root, parentrel, p_ind); + if (simple_indexinfo_equal(p_ind_copy, g_ind)) + { + found_in_child = true; + break; + } + } + if (!found_in_child) + /* The index doesn't exist in childrel, remove it from global_unique_indexes */ + *global_unique_indexes = foreach_delete_current(*global_unique_indexes, lc); + } +} + +/* Helper function for groupres/distinctrel */ +static void +add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgroups) +{ + Query *parse = root->parse; + List *exprs; + + /* + * XXX: If there are some vars which is not in current levelsup, the semantic is + * imprecise, should we avoid it or not? levelsup = 1 is just a demo, maybe we need to + * check every level other than 0, if so, looks we have to write another + * pull_var_walker. + */ + List *upper_vars = pull_vars_of_level((Node*)sortgroups, 1); + + if (upper_vars != NIL) + return; + + exprs = get_sortgrouplist_exprs(sortgroups, parse->targetList); + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(exprs, + false /* sortgroupclause can't be multi_nullvals */)); +} + + +/* + * add_combined_uniquekey + * The combination of both UniqueKeys is a valid UniqueKey for joinrel no matter + * the jointype. + */ +bool +add_combined_uniquekey(RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype) +{ + + ListCell *lc1, *lc2; + + /* Either side has multi_nullvals or we have outer join, + * the combined UniqueKey has multi_nullvals */ + bool multi_nullvals = outer_ukey->multi_nullvals || + inner_ukey->multi_nullvals || IS_OUTER_JOIN(jointype); + + /* The only case we can get onerow joinrel after join */ + if (relation_is_onerow(outer_rel) + && relation_is_onerow(inner_rel) + && jointype == JOIN_INNER) + { + add_uniquekey_for_onerow(joinrel); + return true; + } + + foreach(lc1, get_exprs_from_uniquekey(joinrel, outer_rel, outer_ukey)) + { + foreach(lc2, get_exprs_from_uniquekey(joinrel, inner_rel, inner_ukey)) + { + List *exprs = list_concat_copy(lfirst_node(List, lc1), lfirst_node(List, lc2)); + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(exprs, + multi_nullvals)); + } + } + return false; +} diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index b406d41e91..0551ae0512 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -2389,6 +2389,8 @@ grouping_planner(PlannerInfo *root, bool inheritance_update, add_path(final_rel, path); } + simple_copy_uniquekeys(current_rel, final_rel); + /* * Generate partial paths for final_rel, too, if outer query levels might * be able to make use of them. @@ -3899,6 +3901,8 @@ create_grouping_paths(PlannerInfo *root, } set_cheapest(grouped_rel); + + populate_grouprel_uniquekeys(root, grouped_rel, input_rel); return grouped_rel; } @@ -4615,7 +4619,7 @@ create_window_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(window_rel); - + simple_copy_uniquekeys(input_rel, window_rel); return window_rel; } @@ -4911,7 +4915,7 @@ create_distinct_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(distinct_rel); - + populate_distinctrel_uniquekeys(root, input_rel, distinct_rel); return distinct_rel; } @@ -5172,6 +5176,8 @@ create_ordered_paths(PlannerInfo *root, */ Assert(ordered_rel->pathlist != NIL); + simple_copy_uniquekeys(input_rel, ordered_rel); + return ordered_rel; } @@ -6049,6 +6055,9 @@ adjust_paths_for_srfs(PlannerInfo *root, RelOptInfo *rel, if (list_length(targets) == 1) return; + /* UniqueKey is not valid after handling the SRF. */ + rel->uniquekeys = NIL; + /* * Stack SRF-evaluation nodes atop each path for the rel. * diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c index 951aed80e7..e94e92937c 100644 --- a/src/backend/optimizer/prep/prepunion.c +++ b/src/backend/optimizer/prep/prepunion.c @@ -689,6 +689,8 @@ generate_union_paths(SetOperationStmt *op, PlannerInfo *root, /* Undo effects of possibly forcing tuple_fraction to 0 */ root->tuple_fraction = save_fraction; + /* Add the UniqueKeys */ + populate_unionrel_uniquekeys(root, result_rel); return result_rel; } diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c index d722063cf3..44c37ecffc 100644 --- a/src/backend/optimizer/util/appendinfo.c +++ b/src/backend/optimizer/util/appendinfo.c @@ -746,3 +746,47 @@ find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos) } return appinfos; } + +/* + * find_appinfo_by_child + * + */ +AppendRelInfo * +find_appinfo_by_child(PlannerInfo *root, Index child_index) +{ + ListCell *lc; + foreach(lc, root->append_rel_list) + { + AppendRelInfo *appinfo = lfirst_node(AppendRelInfo, lc); + if (appinfo->child_relid == child_index) + return appinfo; + } + elog(ERROR, "parent relation cant be found"); + return NULL; +} + +/* + * find_parent_var + * + */ +Var * +find_parent_var(AppendRelInfo *appinfo, Var *child_var) +{ + ListCell *lc; + Var *res = NULL; + Index attno = 1; + foreach(lc, appinfo->translated_vars) + { + Node *child_node = lfirst(lc); + if (equal(child_node, child_var)) + { + res = copyObject(child_var); + res->varattno = attno; + res->varno = appinfo->parent_relid; + } + attno++; + } + if (res == NULL) + elog(ERROR, "parent var can't be found."); + return res; +} diff --git a/src/backend/optimizer/util/inherit.c b/src/backend/optimizer/util/inherit.c index 3132fd35a5..d66b40ec50 100644 --- a/src/backend/optimizer/util/inherit.c +++ b/src/backend/optimizer/util/inherit.c @@ -736,6 +736,7 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, { Node *onecq = (Node *) lfirst(lc2); bool pseudoconstant; + RestrictInfo *child_rinfo; /* check for pseudoconstant (no Vars or volatile functions) */ pseudoconstant = @@ -747,13 +748,14 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, root->hasPseudoConstantQuals = true; } /* reconstitute RestrictInfo with appropriate properties */ - childquals = lappend(childquals, - make_restrictinfo((Expr *) onecq, - rinfo->is_pushed_down, - rinfo->outerjoin_delayed, - pseudoconstant, - rinfo->security_level, - NULL, NULL, NULL)); + child_rinfo = make_restrictinfo((Expr *) onecq, + rinfo->is_pushed_down, + rinfo->outerjoin_delayed, + pseudoconstant, + rinfo->security_level, + NULL, NULL, NULL); + child_rinfo->mergeopfamilies = rinfo->mergeopfamilies; + childquals = lappend(childquals, child_rinfo); /* track minimum security level among child quals */ cq_min_security = Min(cq_min_security, rinfo->security_level); } diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 31d9aedeeb..c83f17acb7 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -16,6 +16,7 @@ #include "nodes/execnodes.h" #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" extern A_Expr *makeA_Expr(A_Expr_Kind kind, List *name, @@ -105,4 +106,6 @@ extern GroupingSet *makeGroupingSet(GroupingSetKind kind, List *content, int loc extern VacuumRelation *makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols); +extern UniqueKey* makeUniqueKey(List *exprs, bool multi_nullvals); + #endif /* MAKEFUNC_H */ diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 381d84b4e4..41110ed888 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -264,6 +264,7 @@ typedef enum NodeTag T_EquivalenceMember, T_PathKey, T_PathTarget, + T_UniqueKey, T_RestrictInfo, T_IndexClause, T_PlaceHolderVar, diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index 9e3ebd488a..02e4458bef 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -730,6 +730,7 @@ typedef struct RelOptInfo QualCost baserestrictcost; /* cost of evaluating the above */ Index baserestrict_min_security; /* min security_level found in * baserestrictinfo */ + List *uniquekeys; /* List of UniqueKey */ List *joininfo; /* RestrictInfo structures for join clauses * involving this rel */ bool has_eclass_joins; /* T means joininfo is incomplete */ @@ -1047,6 +1048,28 @@ typedef struct PathKey } PathKey; +/* + * UniqueKey + * + * Represents the unique properties held by a RelOptInfo. + * + * exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL + * is a special case of UniqueKey, which means there is only 1 row in that + * relation. + * multi_nullvals: true means multi null values may exist in these exprs, so the + * uniqueness is not guaranteed in this case. This field is necessary for + * remove_useless_join & reduce_unique_semijoins where we don't mind these + * duplicated NULL values. It is set to true for 2 cases. One is a unique key + * from a unique index but the related column is nullable. The other one is for + * outer join. see populate_joinrel_uniquekeys for detail. + */ +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + /* * PathTarget * @@ -2473,7 +2496,7 @@ typedef enum * * flags indicating what kinds of grouping are possible. * partial_costs_set is true if the agg_partial_costs and agg_final_costs - * have been initialized. + * have been initialized. * agg_partial_costs gives partial aggregation costs. * agg_final_costs gives finalization costs. * target_parallel_safe is true if target is parallel safe. @@ -2503,8 +2526,8 @@ typedef struct * limit_tuples is an estimated bound on the number of output tuples, * or -1 if no LIMIT or couldn't estimate. * count_est and offset_est are the estimated values of the LIMIT and OFFSET - * expressions computed by preprocess_limit() (see comments for - * preprocess_limit() for more information). + * expressions computed by preprocess_limit() (see comments for + * preprocess_limit() for more information). */ typedef struct { diff --git a/src/include/nodes/pg_list.h b/src/include/nodes/pg_list.h index 14ea2766ad..621f54a9f8 100644 --- a/src/include/nodes/pg_list.h +++ b/src/include/nodes/pg_list.h @@ -528,6 +528,8 @@ extern bool list_member_ptr(const List *list, const void *datum); extern bool list_member_int(const List *list, int datum); extern bool list_member_oid(const List *list, Oid datum); +extern bool list_is_subset(const List *members, const List *target); + extern List *list_delete(List *list, void *datum); extern List *list_delete_ptr(List *list, void *datum); extern List *list_delete_int(List *list, int datum); diff --git a/src/include/optimizer/appendinfo.h b/src/include/optimizer/appendinfo.h index d6a27a60dd..e87c92a054 100644 --- a/src/include/optimizer/appendinfo.h +++ b/src/include/optimizer/appendinfo.h @@ -32,4 +32,7 @@ extern Relids adjust_child_relids_multilevel(PlannerInfo *root, Relids relids, extern AppendRelInfo **find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos); +extern AppendRelInfo *find_appinfo_by_child(PlannerInfo *root, Index child_index); +extern Var *find_parent_var(AppendRelInfo *appinfo, Var *child_var); + #endif /* APPENDINFO_H */ diff --git a/src/include/optimizer/optimizer.h b/src/include/optimizer/optimizer.h index 3e4171056e..9445141263 100644 --- a/src/include/optimizer/optimizer.h +++ b/src/include/optimizer/optimizer.h @@ -23,6 +23,7 @@ #define OPTIMIZER_H #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" /* * We don't want to include nodes/pathnodes.h here, because non-planner @@ -156,6 +157,7 @@ extern TargetEntry *get_sortgroupref_tle(Index sortref, List *targetList); extern TargetEntry *get_sortgroupclause_tle(SortGroupClause *sgClause, List *targetList); +extern Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); extern Node *get_sortgroupclause_expr(SortGroupClause *sgClause, List *targetList); extern List *get_sortgrouplist_exprs(List *sgClauses, diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h index 10b6e81079..9217a8d6c6 100644 --- a/src/include/optimizer/paths.h +++ b/src/include/optimizer/paths.h @@ -240,5 +240,48 @@ extern PathKey *make_canonical_pathkey(PlannerInfo *root, int strategy, bool nulls_first); extern void add_paths_to_append_rel(PlannerInfo *root, RelOptInfo *rel, List *live_childrels); +extern List *select_mergejoin_clauses(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + JoinType jointype, + bool *mergejoin_allowed); + +/* + * uniquekeys.c + * Utilities for matching and building unique keys + */ +extern void populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List* unique_index_list); +extern void populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels); +extern void populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel); +extern void populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel); +extern void populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel); +extern void simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel); +extern void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel); +extern void populate_joinrel_uniquekeys(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + RelOptInfo *rel2, + List *restrictlist, + JoinType jointype); + +extern bool relation_has_uniquekeys_for(PlannerInfo *root, + RelOptInfo *rel, + List *exprs, + bool allow_multinulls); +extern bool relation_is_onerow(RelOptInfo *rel); #endif /* PATHS_H */ -- 2.21.0 --op7mf2i72xk5yu7c Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v36-0003-Extend-UniqueKeys.patch" ^ permalink raw reply [nested|flat] 91+ messages in thread
* [PATCH v38 2/6] Introduce UniqueKey attributes on RelOptInfo struct. @ 2020-05-11 07:50 一挃 <[email protected]> 0 siblings, 0 replies; 91+ messages in thread From: 一挃 @ 2020-05-11 07:50 UTC (permalink / raw) UniqueKey is a set of exprs on RelOptInfo which represents the exprs will be unique on the given RelOptInfo. You can see README.uniquekey for more information. --- src/backend/nodes/copyfuncs.c | 13 + src/backend/nodes/list.c | 31 + src/backend/nodes/makefuncs.c | 13 + src/backend/nodes/outfuncs.c | 11 + src/backend/nodes/readfuncs.c | 10 + src/backend/optimizer/path/Makefile | 3 +- src/backend/optimizer/path/README.uniquekey | 131 +++ src/backend/optimizer/path/allpaths.c | 10 + src/backend/optimizer/path/joinpath.c | 9 +- src/backend/optimizer/path/joinrels.c | 2 + src/backend/optimizer/path/pathkeys.c | 3 +- src/backend/optimizer/path/uniquekeys.c | 1135 +++++++++++++++++++ src/backend/optimizer/plan/planner.c | 13 +- src/backend/optimizer/prep/prepunion.c | 2 + src/backend/optimizer/util/appendinfo.c | 44 + src/backend/optimizer/util/inherit.c | 18 +- src/include/nodes/makefuncs.h | 3 + src/include/nodes/nodes.h | 1 + src/include/nodes/pathnodes.h | 29 +- src/include/nodes/pg_list.h | 10 + src/include/optimizer/appendinfo.h | 3 + src/include/optimizer/optimizer.h | 2 + src/include/optimizer/paths.h | 43 + 23 files changed, 1515 insertions(+), 24 deletions(-) create mode 100644 src/backend/optimizer/path/README.uniquekey create mode 100644 src/backend/optimizer/path/uniquekeys.c diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..75c1c5e824 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2296,6 +2296,16 @@ _copyPathKey(const PathKey *from) return newnode; } +static UniqueKey * +_copyUniqueKey(const UniqueKey *from) +{ + UniqueKey *newnode = makeNode(UniqueKey); + + COPY_NODE_FIELD(exprs); + COPY_SCALAR_FIELD(multi_nullvals); + + return newnode; +} /* * _copyRestrictInfo */ @@ -5220,6 +5230,9 @@ copyObjectImpl(const void *from) case T_PathKey: retval = _copyPathKey(from); break; + case T_UniqueKey: + retval = _copyUniqueKey(from); + break; case T_RestrictInfo: retval = _copyRestrictInfo(from); break; diff --git a/src/backend/nodes/list.c b/src/backend/nodes/list.c index dbf6b30233..ca099495a1 100644 --- a/src/backend/nodes/list.c +++ b/src/backend/nodes/list.c @@ -702,6 +702,37 @@ list_member_oid(const List *list, Oid datum) return false; } +/* + * return true iff every entry in "members" list is also present + * in the "target" list. + */ +bool +list_is_subset(const List *members, const List *target) +{ + const ListCell *lc1, *lc2; + + Assert(IsPointerList(members)); + Assert(IsPointerList(target)); + check_list_invariants(members); + check_list_invariants(target); + + foreach(lc1, members) + { + bool found = false; + foreach(lc2, target) + { + if (equal(lfirst(lc1), lfirst(lc2))) + { + found = true; + break; + } + } + if (!found) + return false; + } + return true; +} + /* * Delete the n'th cell (counting from 0) in list. * diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 01c110cd2f..40415d0f5b 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -815,3 +815,16 @@ makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols) v->va_cols = va_cols; return v; } + + +/* + * makeUniqueKey + */ +UniqueKey* +makeUniqueKey(List *exprs, bool multi_nullvals) +{ + UniqueKey * ukey = makeNode(UniqueKey); + ukey->exprs = exprs; + ukey->multi_nullvals = multi_nullvals; + return ukey; +} diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..44154cde6a 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2456,6 +2456,14 @@ _outPathKey(StringInfo str, const PathKey *node) WRITE_BOOL_FIELD(pk_nulls_first); } +static void +_outUniqueKey(StringInfo str, const UniqueKey *node) +{ + WRITE_NODE_TYPE("UNIQUEKEY"); + WRITE_NODE_FIELD(exprs); + WRITE_BOOL_FIELD(multi_nullvals); +} + static void _outPathTarget(StringInfo str, const PathTarget *node) { @@ -4198,6 +4206,9 @@ outNode(StringInfo str, const void *obj) case T_PathKey: _outPathKey(str, obj); break; + case T_UniqueKey: + _outUniqueKey(str, obj); + break; case T_PathTarget: _outPathTarget(str, obj); break; diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index c5e136e9c3..b3e212bf1c 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -492,6 +492,14 @@ _readSetOperationStmt(void) READ_DONE(); } +static UniqueKey * +_readUniqueKey(void) +{ + READ_LOCALS(UniqueKey); + READ_NODE_FIELD(exprs); + READ_BOOL_FIELD(multi_nullvals); + READ_DONE(); +} /* * Stuff from primnodes.h. @@ -2717,6 +2725,8 @@ parseNodeString(void) return_value = _readCommonTableExpr(); else if (MATCH("SETOPERATIONSTMT", 16)) return_value = _readSetOperationStmt(); + else if (MATCH("UNIQUEKEY", 9)) + return_value = _readUniqueKey(); else if (MATCH("ALIAS", 5)) return_value = _readAlias(); else if (MATCH("RANGEVAR", 8)) diff --git a/src/backend/optimizer/path/Makefile b/src/backend/optimizer/path/Makefile index 1e199ff66f..7b9820c25f 100644 --- a/src/backend/optimizer/path/Makefile +++ b/src/backend/optimizer/path/Makefile @@ -21,6 +21,7 @@ OBJS = \ joinpath.o \ joinrels.o \ pathkeys.o \ - tidpath.o + tidpath.o \ + uniquekeys.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/optimizer/path/README.uniquekey b/src/backend/optimizer/path/README.uniquekey new file mode 100644 index 0000000000..5eac761995 --- /dev/null +++ b/src/backend/optimizer/path/README.uniquekey @@ -0,0 +1,131 @@ +1. What is UniqueKey? +We can think UniqueKey is a set of exprs for a RelOptInfo, which we are insure +that doesn't yields same result among all the rows. The simplest UniqueKey +format is primary key. + +However we define the UnqiueKey as below. + +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + +exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL +is a special case of UniqueKey, which means there is only one row in that +relation.it has a stronger semantic than others. like SELECT uk FROM t; uk is +normal unique key and may have different values. SELECT colx FROM t WHERE uk = +const. colx is unique AND we have only 1 value. This field can used for +innerrel_is_unique. this logic is handled specially in add_uniquekey_for_onerow +function. + +multi_nullvals: true means multi null values may exist in these exprs, so the +uniqueness is not guaranteed in this case. This field is necessary for +remove_useless_join & reduce_unique_semijoins where we don't mind these +duplicated NULL values. It is set to true for 2 cases. One is a unique key +from a unique index but the related column is nullable. The other one is for +outer join. see populate_joinrel_uniquekeys for detail. + + +The UniqueKey can be used at the following cases at least: +1. remove_useless_joins. +2. reduce_semianti_joins +3. remove distinct node if distinct clause is unique. +4. remove aggnode if group by clause is unique. +5. Index Skip Scan (WIP) +6. Aggregation Push Down without 2 phase aggregation if the join can't + duplicated the aggregated rows. (work in progress feature) + +2. How is it maintained? + +We have a set of populate_xxx_unqiuekeys functions to maintain the uniquekey on +various cases. xxx includes baserel, joinrel, partitionedrel, distinctrel, +groupedrel, unionrel. and we also need to convert the uniquekey from subquery +to outer relation, which is what convert_subquery_uniquekeys does. + +1. The first part is about baserel. We handled 3 cases. suppose we have Unique +Index on (a, b). + +1. SELECT a, b FROM t. UniqueKey (a, b) +2. SELECT a FROM t WHERE b = 1; UniqueKey (a) +3. SELECT .. FROM t WHERE a = 1 AND b = 1; UniqueKey (NIL). onerow case, every + column is Unique. + +2. The next part is joinrel, this part is most error-prone, we simplified the rules +like below: +1. If the relation's UniqueKey can't be duplicated after join, then is will be + still valid for the join rel. The function we used here is + innerrel_keeps_unique. The basic idea is innerrel.any_col = outer.uk. + +2. If the UnqiueKey can't keep valid via the rule 1, the combination of the + UniqueKey from both sides are valid for sure. We can prove this as: if the + unique exprs from rel1 is duplicated by rel2, the duplicated rows must + contains different unique exprs from rel2. + +More considerations about onerow: +1. If relation with one row and it can't be duplicated, it is still possible + contains mulit_nullvas after outer join. +2. If the either UniqueKey can be duplicated after join, the can get one row + only when both side is one row AND there is no outer join. +3. Whenever the onerow UniqueKey is not a valid any more, we need to convert one + row UniqueKey to normal unique key since we don't store exprs for one-row + relation. get_exprs_from_uniquekeys will be used here. + + +More considerations about multi_nullvals after join: +1. If the original UnqiueKey has multi_nullvals, the final UniqueKey will have + mulit_nullvals in any case. +2. If a unique key doesn't allow mulit_nullvals, after some outer join, it + allows some outer join. + + +3. When we comes to subquery, we need to convert_subquery_unqiuekeys just like +convert_subquery_pathkeys. Only the UniqueKey insides subquery is referenced as +a Var in outer relation will be reused. The relationship between the outerrel.Var +and subquery.exprs is built with outerel->subroot->processed_tlist. + + +4. As for the SRF functions, it will break the uniqueness of uniquekey, However it +is handled in adjust_paths_for_srfs, which happens after the query_planner. so +we will maintain the UniqueKey until there and reset it to NIL at that +places. This can't help on distinct/group by elimination cases but probably help +in some other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is +semantic correctly. + + +5. As for inherit table, we first main the UnqiueKey on childrel as well. But for +partitioned table we need to maintain 2 different kinds of +UnqiueKey. 1). UniqueKey on the parent relation 2). UniqueKey on child +relation for partition wise query. + +Example: +CREATE TABLE p (a int not null, b int not null) partition by list (a); +CREATE TABLE p0 partition of p for values in (1); +CREATE TABLE p1 partition of p for values in (2); + +create unique index p0_b on p0(b); +create unique index p1_b on p1(b); + +Now b is only unique on partition level, so the distinct can't be removed on +the following cases. SELECT DISTINCT b FROM p; + +Another example is SELECT DISTINCT a, b FROM p WHERE a = 1; Since only one +partition is chosen, the UniqueKey on child relation is same as the UniqueKey on +parent relation. + +Another usage of UniqueKey on partition level is it be helpful for +partition-wise join. + +As for the UniqueKey on parent table level, it comes with 2 different ways, +1). the UniqueKey is also derived in UniqueKey index, but the index must be same +in all the related children relations and the unique index must contains +Partition Key in it. Example: + +CREATE UNIQUE INDEX p_ab ON p(a, b); -- where a is the partition key. + +-- Query +SELECT a, b FROM p; the (a, b) is a UniqueKey of p. + +2). If the parent relation has only one childrel, the UniqueKey on childrel is + the UniqueKey on parent as well. diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index 37b4223adb..d52ad59f89 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -579,6 +579,12 @@ set_plain_rel_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) */ check_index_predicates(root, rel); + /* + * Now that we've marked which partial indexes are suitable, we can now + * build the relation's unique keys. + */ + populate_baserel_uniquekeys(root, rel, rel->indexlist); + /* Mark rel with estimated output rows, width, etc */ set_baserel_size_estimates(root, rel); } @@ -1297,6 +1303,8 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, /* Add paths to the append relation. */ add_paths_to_append_rel(root, rel, live_childrels); + if (IS_PARTITIONED_REL(rel)) + populate_partitionedrel_uniquekeys(root, rel, live_childrels); } @@ -2304,6 +2312,8 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, pathkeys, required_outer)); } + convert_subquery_uniquekeys(root, rel, sub_final_rel); + /* If outer rel allows parallelism, do same for partial paths. */ if (rel->consider_parallel && bms_is_empty(required_outer)) { diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c index 57ce97fd53..697ac047fb 100644 --- a/src/backend/optimizer/path/joinpath.c +++ b/src/backend/optimizer/path/joinpath.c @@ -71,13 +71,6 @@ static void consider_parallel_mergejoin(PlannerInfo *root, static void hash_inner_and_outer(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, RelOptInfo *innerrel, JoinType jointype, JoinPathExtraData *extra); -static List *select_mergejoin_clauses(PlannerInfo *root, - RelOptInfo *joinrel, - RelOptInfo *outerrel, - RelOptInfo *innerrel, - List *restrictlist, - JoinType jointype, - bool *mergejoin_allowed); static void generate_mergejoin_paths(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *innerrel, @@ -1927,7 +1920,7 @@ hash_inner_and_outer(PlannerInfo *root, * if it is mergejoinable and involves vars from the two sub-relations * currently of interest. */ -static List * +List * select_mergejoin_clauses(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c index 0dbe2ac726..7271f044ec 100644 --- a/src/backend/optimizer/path/joinrels.c +++ b/src/backend/optimizer/path/joinrels.c @@ -924,6 +924,8 @@ populate_joinrel_with_paths(PlannerInfo *root, RelOptInfo *rel1, /* Apply partitionwise join technique, if possible. */ try_partitionwise_join(root, rel1, rel2, joinrel, sjinfo, restrictlist); + + populate_joinrel_uniquekeys(root, joinrel, rel1, rel2, restrictlist, sjinfo->jointype); } diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index bd9a176d7d..139278829b 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -33,7 +33,6 @@ static bool pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys); static bool matches_boolean_partition_clause(RestrictInfo *rinfo, RelOptInfo *partrel, int partkeycol); -static Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); static bool right_merge_direction(PlannerInfo *root, PathKey *pathkey); @@ -1035,7 +1034,7 @@ convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel, * We need this to ensure that we don't return pathkeys describing values * that are unavailable above the level of the subquery scan. */ -static Var * +Var * find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle) { ListCell *lc; diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c new file mode 100644 index 0000000000..77ed2b2eff --- /dev/null +++ b/src/backend/optimizer/path/uniquekeys.c @@ -0,0 +1,1135 @@ +/*------------------------------------------------------------------------- + * + * uniquekeys.c + * Utilities for matching and building unique keys + * + * Portions Copyright (c) 2020, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/optimizer/path/uniquekeys.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "nodes/makefuncs.h" +#include "nodes/nodeFuncs.h" +#include "optimizer/pathnode.h" +#include "optimizer/paths.h" +#include "optimizer/appendinfo.h" +#include "optimizer/optimizer.h" +#include "optimizer/tlist.h" +#include "rewrite/rewriteManip.h" + + +/* + * This struct is used to help populate_joinrel_uniquekeys. + * + * added_to_joinrel is true if a uniquekey (from outerrel or innerrel) + * has been added to joinrel. + * useful is true if the exprs of the uniquekey still appears in joinrel. + */ +typedef struct UniqueKeyContextData +{ + UniqueKey *uniquekey; + bool added_to_joinrel; + bool useful; +} *UniqueKeyContext; + +static List *initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel); +static bool innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + bool reverse); + +static List *get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals); +static List *get_exprs_from_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + UniqueKey *ukey); +static void add_uniquekey_for_onerow(RelOptInfo *rel); +static bool add_combined_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype); + +/* Used for unique indexes checking for partitioned table */ +static bool index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind); +static IndexOptInfo *simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from); +static bool simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2); +static void adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_index); + +/* Helper function for grouped relation and distinct relation. */ +static void add_uniquekey_from_sortgroups(PlannerInfo *root, + RelOptInfo *rel, + List *sortgroups); + +/* + * populate_baserel_uniquekeys + * Populate 'baserel' uniquekeys list by looking at the rel's unique index + * and baserestrictinfo + */ +void +populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List *indexlist) +{ + ListCell *lc; + List *matched_uniq_indexes = NIL; + + /* Attrs appears in rel->reltarget->exprs. */ + Bitmapset *used_attrs = NULL; + + List *const_exprs = NIL; + List *expr_opfamilies = NIL; + + Assert(baserel->rtekind == RTE_RELATION); + + foreach(lc, indexlist) + { + IndexOptInfo *ind = (IndexOptInfo *) lfirst(lc); + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + matched_uniq_indexes = lappend(matched_uniq_indexes, ind); + } + + if (matched_uniq_indexes == NIL) + return; + + /* Check which attrs is used in baserel->reltarget */ + pull_varattnos((Node *)baserel->reltarget->exprs, baserel->relid, &used_attrs); + + /* Check which attrno is used at a mergeable const filter */ + foreach(lc, baserel->baserestrictinfo) + { + RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); + + if (rinfo->mergeopfamilies == NIL) + continue; + + if (bms_is_empty(rinfo->left_relids)) + { + const_exprs = lappend(const_exprs, get_rightop(rinfo->clause)); + } + else if (bms_is_empty(rinfo->right_relids)) + { + const_exprs = lappend(const_exprs, get_leftop(rinfo->clause)); + } + else + continue; + + expr_opfamilies = lappend(expr_opfamilies, rinfo->mergeopfamilies); + } + + foreach(lc, matched_uniq_indexes) + { + bool multi_nullvals, useful; + List *exprs = get_exprs_from_uniqueindex(lfirst_node(IndexOptInfo, lc), + const_exprs, + expr_opfamilies, + used_attrs, + &useful, + &multi_nullvals); + if (useful) + { + if (exprs == NIL) + { + /* All the columns in Unique Index matched with a restrictinfo */ + add_uniquekey_for_onerow(baserel); + return; + } + baserel->uniquekeys = lappend(baserel->uniquekeys, + makeUniqueKey(exprs, multi_nullvals)); + } + } +} + + +/* + * populate_partitionedrel_uniquekeys + * The UniqueKey on partitionrel comes from 2 cases: + * 1). Only one partition is involved in this query, the unique key can be + * copied to parent rel from childrel. + * 2). There are some unique index which includes partition key and exists + * in all the related partitions. + * We never mind rule 2 if we hit rule 1. + */ + +void +populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels) +{ + ListCell *lc; + List *global_uniq_indexlist = NIL; + RelOptInfo *childrel; + bool is_first = true; + + Assert(IS_PARTITIONED_REL(rel)); + + if (childrels == NIL) + return; + + /* + * If there is only one partition used in this query, the UniqueKey in childrel is + * still valid in parent level, but we need convert the format from child expr to + * parent expr. + */ + if (list_length(childrels) == 1) + { + /* Check for Rule 1 */ + RelOptInfo *childrel = linitial_node(RelOptInfo, childrels); + ListCell *lc; + Assert(childrel->reloptkind == RELOPT_OTHER_MEMBER_REL); + if (relation_is_onerow(childrel)) + { + add_uniquekey_for_onerow(rel); + return; + } + + foreach(lc, childrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + AppendRelInfo *appinfo = find_appinfo_by_child(root, childrel->relid); + List *parent_exprs = NIL; + bool can_reuse = true; + ListCell *lc2; + foreach(lc2, ukey->exprs) + { + Var *var = (Var *)lfirst(lc2); + /* + * If the expr comes from a expression, it is hard to build the expression + * in parent so ignore that case for now. + */ + if(!IsA(var, Var)) + { + can_reuse = false; + break; + } + /* Convert it to parent var */ + parent_exprs = lappend(parent_exprs, find_parent_var(appinfo, var)); + } + if (can_reuse) + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(parent_exprs, + ukey->multi_nullvals)); + } + } + else + { + /* Check for rule 2 */ + childrel = linitial_node(RelOptInfo, childrels); + foreach(lc, childrel->indexlist) + { + IndexOptInfo *ind = lfirst(lc); + IndexOptInfo *modified_index; + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + + /* + * During simple_copy_indexinfo_to_parent, we need to convert var from + * child var to parent var, index on expression is too complex to handle. + * so ignore it for now. + */ + if (ind->indexprs != NIL) + continue; + + modified_index = simple_copy_indexinfo_to_parent(root, rel, ind); + /* + * If the unique index doesn't contain partkey, then it is unique + * on this partition only, so it is useless for us. + */ + if (!index_constains_partkey(rel, modified_index)) + continue; + + global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index); + } + + if (global_uniq_indexlist != NIL) + { + foreach(lc, childrels) + { + RelOptInfo *child = lfirst(lc); + if (is_first) + { + is_first = false; + continue; + } + adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist); + } + /* Now we have a list of unique index which are exactly same on all childrels, + * Set the UniqueKey just like it is non-partition table + */ + populate_baserel_uniquekeys(root, rel, global_uniq_indexlist); + } + } +} + + +/* + * populate_distinctrel_uniquekeys + */ +void +populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel) +{ + /* The unique key before the distinct is still valid. */ + distinctrel->uniquekeys = list_copy(inputrel->uniquekeys); + add_uniquekey_from_sortgroups(root, distinctrel, root->parse->distinctClause); +} + +/* + * populate_grouprel_uniquekeys + */ +void +populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel) + +{ + Query *parse = root->parse; + bool input_ukey_added = false; + ListCell *lc; + + if (relation_is_onerow(inputrel)) + { + add_uniquekey_for_onerow(grouprel); + return; + } + if (parse->groupingSets) + return; + + /* A Normal group by without grouping set. */ + if (parse->groupClause) + { + /* + * Current even the groupby clause is Unique already, but if query has aggref + * We have to create grouprel still. To keep the UnqiueKey short, we will check + * the UniqueKey of input_rel still valid, if so we reuse it. + */ + foreach(lc, inputrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (list_is_subset(ukey->exprs, grouprel->reltarget->exprs)) + { + grouprel->uniquekeys = lappend(grouprel->uniquekeys, + ukey); + input_ukey_added = true; + } + } + if (!input_ukey_added) + /* + * group by clause must be a super-set of grouprel->reltarget->exprs except the + * aggregation expr, so if such exprs is unique already, no bother to generate + * new uniquekey for group by exprs. + */ + add_uniquekey_from_sortgroups(root, + grouprel, + root->parse->groupClause); + } + else + /* It has aggregation but without a group by, so only one row returned */ + add_uniquekey_for_onerow(grouprel); +} + +/* + * simple_copy_uniquekeys + * Using a function for the one-line code makes us easy to check where we simply + * copied the uniquekey. + */ +void +simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel) +{ + newrel->uniquekeys = oldrel->uniquekeys; +} + +/* + * populate_unionrel_uniquekeys + */ +void +populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel) +{ + ListCell *lc; + List *exprs = NIL; + + Assert(unionrel->uniquekeys == NIL); + + foreach(lc, unionrel->reltarget->exprs) + { + exprs = lappend(exprs, lfirst(lc)); + } + + if (exprs == NIL) + /* SQL: select union select; is valid, we need to handle it here. */ + add_uniquekey_for_onerow(unionrel); + else + unionrel->uniquekeys = lappend(unionrel->uniquekeys, + makeUniqueKey(exprs,false)); + +} + +/* + * populate_joinrel_uniquekeys + * + * populate uniquekeys for joinrel. We will check each relation to see if its + * UniqueKey is still valid via innerrel_keeps_unique, if so, we add it to + * joinrel. The multi_nullvals field will be changed to true for some outer + * join cases and one-row UniqueKey needs to be converted to normal UniqueKey + * for the same case as well. + * For the uniquekey in either baserel which can't be unique after join, we still + * check to see if combination of UniqueKeys from both side is still useful for us. + * if yes, we add it to joinrel as well. + */ +void +populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *outerrel, RelOptInfo *innerrel, + List *restrictlist, JoinType jointype) +{ + ListCell *lc, *lc2; + List *clause_list = NIL; + List *outerrel_ukey_ctx; + List *innerrel_ukey_ctx; + bool inner_onerow, outer_onerow; + bool mergejoin_allowed; + + /* Care about the outerrel relation only for SEMI/ANTI join */ + if (jointype == JOIN_SEMI || jointype == JOIN_ANTI) + { + foreach(lc, outerrel->uniquekeys) + { + UniqueKey *uniquekey = lfirst_node(UniqueKey, lc); + if (list_is_subset(uniquekey->exprs, joinrel->reltarget->exprs)) + joinrel->uniquekeys = lappend(joinrel->uniquekeys, uniquekey); + } + return; + } + + Assert(jointype == JOIN_LEFT || jointype == JOIN_FULL || jointype == JOIN_INNER); + + /* Fast path */ + if (innerrel->uniquekeys == NIL || outerrel->uniquekeys == NIL) + return; + + inner_onerow = relation_is_onerow(innerrel); + outer_onerow = relation_is_onerow(outerrel); + + outerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(outerrel); + innerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(innerrel); + + clause_list = select_mergejoin_clauses(root, joinrel, outerrel, innerrel, + restrictlist, jointype, + &mergejoin_allowed); + + if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true /* reverse */)) + { + bool outer_impact = jointype == JOIN_FULL; + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + /* Outer relation has one row, and the unique key is not duplicated after join, + * the joinrel will still has one row unless the jointype == JOIN_FULL. + */ + if (outer_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (outer_onerow) + { + /* + * The onerow outerrel becomes multi rows and multi_nullvals + * will be changed to true. We also need to set the exprs correctly since it + * can't be NIL any more. + */ + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, outerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Change multi_nullvals to true due to the full join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, true)); + else + /* Just reuse it */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + } + ctx->added_to_joinrel = true; + } + } + + if (innerrel_keeps_unique(root, outerrel, innerrel, clause_list, false)) + { + bool outer_impact = jointype == JOIN_FULL || jointype == JOIN_LEFT;; + + foreach(lc, innerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + if (inner_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (inner_onerow) + { + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, innerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Need to change multi_nullvals to true due to the outer join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, + true)); + else + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + + } + ctx->added_to_joinrel = true; + } + } + + /* + * The combination of the UniqueKey from both sides is unique as well regardless + * of join type, but no bother to add it if its subset has been added to joinrel + * already or it is not useful for the joinrel. + */ + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx1 = (UniqueKeyContext) lfirst(lc); + if (ctx1->added_to_joinrel || !ctx1->useful) + continue; + foreach(lc2, innerrel_ukey_ctx) + { + UniqueKeyContext ctx2 = (UniqueKeyContext) lfirst(lc2); + if (ctx2->added_to_joinrel || !ctx2->useful) + continue; + if (add_combined_uniquekey(root, joinrel, outerrel, innerrel, + ctx1->uniquekey, ctx2->uniquekey, + jointype)) + /* If we set a onerow UniqueKey to joinrel, we don't need other. */ + return; + } + } +} + + +/* + * convert_subquery_uniquekeys + * + * Covert the UniqueKey in subquery to outer relation. + */ +void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel) +{ + ListCell *lc; + + if (sub_final_rel->uniquekeys == NIL) + return; + + if (relation_is_onerow(sub_final_rel)) + { + add_uniquekey_for_onerow(currel); + return; + } + + Assert(currel->subroot != NULL); + + foreach(lc, sub_final_rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + ListCell *lc; + List *exprs = NIL; + bool ukey_useful = true; + + /* One row case is handled above */ + Assert(ukey->exprs != NIL); + foreach(lc, ukey->exprs) + { + Var *var; + TargetEntry *tle = tlist_member(lfirst(lc), + currel->subroot->processed_tlist); + if (tle == NULL) + { + ukey_useful = false; + break; + } + var = find_var_for_subquery_tle(currel, tle); + if (var == NULL) + { + ukey_useful = false; + break; + } + exprs = lappend(exprs, var); + } + + if (ukey_useful) + currel->uniquekeys = lappend(currel->uniquekeys, + makeUniqueKey(exprs, + ukey->multi_nullvals)); + + } +} + +/* + * innerrel_keeps_unique + * + * Check if Unique key of the innerrel is valid after join. innerrel's UniqueKey + * will be still valid if innerrel's any-column mergeop outrerel's uniquekey + * exists in clause_list. + * + * Note: the clause_list must be a list of mergeable restrictinfo already. + */ +static bool +innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *clause_list, + bool reverse) +{ + ListCell *lc, *lc2, *lc3; + + if (outerrel->uniquekeys == NIL || innerrel->uniquekeys == NIL) + return false; + + /* Check if there is outerrel's uniquekey in mergeable clause. */ + foreach(lc, outerrel->uniquekeys) + { + List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs; + bool clauselist_matchs_all_exprs = true; + foreach(lc2, outer_uq_exprs) + { + Node *outer_uq_expr = lfirst(lc2); + bool find_uq_expr_in_clauselist = false; + foreach(lc3, clause_list) + { + RestrictInfo *rinfo = lfirst_node(RestrictInfo, lc3); + Node *outer_expr; + if (reverse) + outer_expr = rinfo->outer_is_left ? get_rightop(rinfo->clause) : get_leftop(rinfo->clause); + else + outer_expr = rinfo->outer_is_left ? get_leftop(rinfo->clause) : get_rightop(rinfo->clause); + if (equal(outer_expr, outer_uq_expr)) + { + find_uq_expr_in_clauselist = true; + break; + } + } + if (!find_uq_expr_in_clauselist) + { + /* No need to check the next exprs in the current uniquekey */ + clauselist_matchs_all_exprs = false; + break; + } + } + + if (clauselist_matchs_all_exprs) + return true; + } + return false; +} + + +/* + * relation_is_onerow + * Check if it is a one-row relation by checking UniqueKey. + */ +bool +relation_is_onerow(RelOptInfo *rel) +{ + UniqueKey *ukey; + if (rel->uniquekeys == NIL) + return false; + ukey = linitial_node(UniqueKey, rel->uniquekeys); + return ukey->exprs == NIL && list_length(rel->uniquekeys) == 1; +} + +/* + * relation_has_uniquekeys_for + * Returns true if we have proofs that 'rel' cannot return multiple rows with + * the same values in each of 'exprs'. Otherwise returns false. + */ +bool +relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel, + List *exprs, bool allow_multinulls) +{ + ListCell *lc; + + /* + * For UniqueKey->onerow case, the uniquekey->exprs is empty as well + * so we can't rely on list_is_subset to handle this special cases + */ + if (exprs == NIL) + return false; + + foreach(lc, rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (ukey->multi_nullvals && !allow_multinulls) + continue; + if (list_is_subset(ukey->exprs, exprs)) + return true; + } + return false; +} + + +/* + * get_exprs_from_uniqueindex + * + * Return a list of exprs which is unique. set useful to false if this + * unique index is not useful for us. + */ +static List * +get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals) +{ + List *exprs = NIL; + ListCell *indexpr_item; + int c = 0; + + *useful = true; + *multi_nullvals = false; + + indexpr_item = list_head(unique_index->indexprs); + for(c = 0; c < unique_index->ncolumns; c++) + { + int attr = unique_index->indexkeys[c]; + Expr *expr; + bool matched_const = false; + ListCell *lc1, *lc2; + + if(attr > 0) + { + expr = list_nth_node(TargetEntry, unique_index->indextlist, c)->expr; + } + else if (attr == 0) + { + /* Expression index */ + expr = lfirst(indexpr_item); + indexpr_item = lnext(unique_index->indexprs, indexpr_item); + } + else /* attr < 0 */ + { + /* Index on system column is not supported */ + Assert(false); + } + + /* + * Check index_col = Const case with regarding to opfamily checking + * If we can remove the index_col from the final UniqueKey->exprs. + */ + forboth(lc1, const_exprs, lc2, const_expr_opfamilies) + { + if (list_member_oid((List *)lfirst(lc2), unique_index->opfamily[c]) + && match_index_to_operand((Node *) lfirst(lc1), c, unique_index)) + { + matched_const = true; + break; + } + } + + if (matched_const) + continue; + + /* Check if the indexed expr is used in rel */ + if (attr > 0) + { + /* + * Normal Indexed column, if the col is not used, then the index is useless + * for uniquekey. + */ + attr -= FirstLowInvalidHeapAttributeNumber; + + if (!bms_is_member(attr, used_varattrs)) + { + *useful = false; + break; + } + } + else if (!list_member(unique_index->rel->reltarget->exprs, expr)) + { + /* Expression index but the expression is not used in rel */ + *useful = false; + break; + } + + /* check not null property. */ + if (attr == 0) + { + /* We never know if a expression yields null or not */ + *multi_nullvals = true; + } + else if (!bms_is_member(attr, unique_index->rel->notnullattrs) + && !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber, + unique_index->rel->notnullattrs)) + { + *multi_nullvals = true; + } + + exprs = lappend(exprs, expr); + } + return exprs; +} + + +/* + * add_uniquekey_for_onerow + * If we are sure that the relation only returns one row, then all the columns + * are unique. However we don't need to create UniqueKey for every column, we + * just set exprs = NIL and overwrites all the other UniqueKey on this RelOptInfo + * since this one has strongest semantics. + */ +void +add_uniquekey_for_onerow(RelOptInfo *rel) +{ + /* + * We overwrite the previous UniqueKey on purpose since this one has the + * strongest semantic. + */ + rel->uniquekeys = list_make1(makeUniqueKey(NIL, false)); +} + + +/* + * initililze_uniquecontext_for_joinrel + * Return a List of UniqueKeyContext for an inputrel + */ +static List * +initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel) +{ + List *res = NIL; + ListCell *lc; + foreach(lc, inputrel->uniquekeys) + { + UniqueKeyContext context; + context = palloc(sizeof(struct UniqueKeyContextData)); + context->uniquekey = lfirst_node(UniqueKey, lc); + context->added_to_joinrel = false; + context->useful = true; + res = lappend(res, context); + } + return res; +} + + +/* + * get_exprs_from_uniquekey + * Unify the way of get List of exprs from a one-row UniqueKey or + * normal UniqueKey. for the onerow case, every expr in rel1 is a valid + * UniqueKey. Return a List of exprs. + * + * rel1: The relation which you want to get the exprs. + * ukey: The UniqueKey you want to get the exprs. + */ +static List * +get_exprs_from_uniquekey(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *rel1, UniqueKey *ukey) +{ + ListCell *lc; + bool onerow = rel1 != NULL && relation_is_onerow(rel1); + + List *res = NIL; + Assert(onerow || ukey); + if (onerow) + { + /* Only cares about the exprs still exist in joinrel */ + foreach(lc, joinrel->reltarget->exprs) + { + Bitmapset *relids = pull_varnos(root, lfirst(lc)); + if (bms_is_subset(relids, rel1->relids)) + { + res = lappend(res, list_make1(lfirst(lc))); + } + } + } + else + { + res = list_make1(ukey->exprs); + } + return res; +} + +/* + * Partitioned table Unique Keys. + * The partition table unique key is maintained as: + * 1. The index must be unique as usual. + * 2. The index must contains partition key. + * 3. The index must exist on all the child rel. see simple_indexinfo_equal for + * how we compare it. + */ + +/* + * index_constains_partkey + * return true if the index contains the partiton key. + */ +static bool +index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind) +{ + ListCell *lc; + int i; + Assert(IS_PARTITIONED_REL(partrel)); + Assert(partrel->part_scheme->partnatts > 0); + + for(i = 0; i < partrel->part_scheme->partnatts; i++) + { + Node *part_expr = linitial(partrel->partexprs[i]); + bool found_in_index = false; + foreach(lc, ind->indextlist) + { + Expr *index_expr = lfirst_node(TargetEntry, lc)->expr; + if (equal(index_expr, part_expr)) + { + found_in_index = true; + break; + } + } + if (!found_in_index) + return false; + } + return true; +} + +/* + * simple_indexinfo_equal + * + * Used to check if the 2 index is same as each other. The index here + * is COPIED from childrel and did some tiny changes(see + * simple_copy_indexinfo_to_parent) + */ +static bool +simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2) +{ + Size oid_cmp_len = sizeof(Oid) * ind1->ncolumns; + + return ind1->ncolumns == ind2->ncolumns && + ind1->unique == ind2->unique && + memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0 && + memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0 && + memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0 && + memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0 && + equal(get_tlist_exprs(ind1->indextlist, true), + get_tlist_exprs(ind2->indextlist, true)); +} + + +/* + * The below macros are used for simple_copy_indexinfo_to_parent which is so + * customized that I don't want to put it to copyfuncs.c. So copy it here. + */ +#define COPY_POINTER_FIELD(fldname, sz) \ + do { \ + Size _size = (sz); \ + newnode->fldname = palloc(_size); \ + memcpy(newnode->fldname, from->fldname, _size); \ + } while (0) + +#define COPY_NODE_FIELD(fldname) \ + (newnode->fldname = copyObjectImpl(from->fldname)) + +#define COPY_SCALAR_FIELD(fldname) \ + (newnode->fldname = from->fldname) + + +/* + * simple_copy_indexinfo_to_parent (from partition) + * Copy the IndexInfo from child relation to parent relation with some modification, + * which is used to test: + * 1. If the same index exists in all the childrels. + * 2. If the parentrel->reltarget/basicrestrict info matches this index. + */ +static IndexOptInfo * +simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from) +{ + IndexOptInfo *newnode = makeNode(IndexOptInfo); + AppendRelInfo *appinfo = find_appinfo_by_child(root, from->rel->relid); + ListCell *lc; + int idx = 0; + + COPY_SCALAR_FIELD(ncolumns); + COPY_SCALAR_FIELD(nkeycolumns); + COPY_SCALAR_FIELD(unique); + COPY_SCALAR_FIELD(immediate); + /* We just need to know if it is NIL or not */ + COPY_SCALAR_FIELD(indpred); + COPY_SCALAR_FIELD(predOK); + COPY_POINTER_FIELD(indexkeys, from->ncolumns * sizeof(int)); + COPY_POINTER_FIELD(indexcollations, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opfamily, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opcintype, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(sortopfamily, from->ncolumns * sizeof(Oid)); + COPY_NODE_FIELD(indextlist); + + /* Convert index exprs on child expr to expr on parent */ + foreach(lc, newnode->indextlist) + { + TargetEntry *tle = lfirst_node(TargetEntry, lc); + /* Index on expression is ignored */ + Assert(IsA(tle->expr, Var)); + tle->expr = (Expr *) find_parent_var(appinfo, (Var *) tle->expr); + newnode->indexkeys[idx] = castNode(Var, tle->expr)->varattno; + idx++; + } + newnode->rel = parentrel; + return newnode; +} + +/* + * adjust_partition_unique_indexlist + * + * global_unique_indexes: At the beginning, it contains the copy & modified + * unique index from the first partition. And then check if each index in it still + * exists in the following partitions. If no, remove it. at last, it has an + * index list which exists in all the partitions. + */ +static void +adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_indexes) +{ + ListCell *lc, *lc2; + foreach(lc, *global_unique_indexes) + { + IndexOptInfo *g_ind = lfirst_node(IndexOptInfo, lc); + bool found_in_child = false; + + foreach(lc2, childrel->indexlist) + { + IndexOptInfo *p_ind = lfirst_node(IndexOptInfo, lc2); + IndexOptInfo *p_ind_copy; + if (!p_ind->unique || !p_ind->immediate || + (p_ind->indpred != NIL && !p_ind->predOK)) + continue; + p_ind_copy = simple_copy_indexinfo_to_parent(root, parentrel, p_ind); + if (simple_indexinfo_equal(p_ind_copy, g_ind)) + { + found_in_child = true; + break; + } + } + if (!found_in_child) + /* The index doesn't exist in childrel, remove it from global_unique_indexes */ + *global_unique_indexes = foreach_delete_current(*global_unique_indexes, lc); + } +} + +/* Helper function for groupres/distinctrel */ +static void +add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgroups) +{ + Query *parse = root->parse; + List *exprs; + + /* + * XXX: If there are some vars which is not in current levelsup, the semantic is + * imprecise, should we avoid it or not? levelsup = 1 is just a demo, maybe we need to + * check every level other than 0, if so, looks we have to write another + * pull_var_walker. + */ + List *upper_vars = pull_vars_of_level((Node*)sortgroups, 1); + + if (upper_vars != NIL) + return; + + exprs = get_sortgrouplist_exprs(sortgroups, parse->targetList); + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(exprs, + false /* sortgroupclause can't be multi_nullvals */)); +} + + +/* + * add_combined_uniquekey + * The combination of both UniqueKeys is a valid UniqueKey for joinrel no matter + * the jointype. + */ +bool +add_combined_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype) +{ + + ListCell *lc1, *lc2; + + /* Either side has multi_nullvals or we have outer join, + * the combined UniqueKey has multi_nullvals */ + bool multi_nullvals = outer_ukey->multi_nullvals || + inner_ukey->multi_nullvals || IS_OUTER_JOIN(jointype); + + /* The only case we can get onerow joinrel after join */ + if (relation_is_onerow(outer_rel) + && relation_is_onerow(inner_rel) + && jointype == JOIN_INNER) + { + add_uniquekey_for_onerow(joinrel); + return true; + } + + foreach(lc1, get_exprs_from_uniquekey(root, joinrel, outer_rel, outer_ukey)) + { + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, inner_rel, inner_ukey)) + { + List *exprs = list_concat_copy(lfirst_node(List, lc1), lfirst_node(List, lc2)); + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(exprs, + multi_nullvals)); + } + } + return false; +} diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index 424d25cbd5..8d8e493f5c 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -2387,6 +2387,8 @@ grouping_planner(PlannerInfo *root, bool inheritance_update, add_path(final_rel, path); } + simple_copy_uniquekeys(current_rel, final_rel); + /* * Generate partial paths for final_rel, too, if outer query levels might * be able to make use of them. @@ -3899,6 +3901,8 @@ create_grouping_paths(PlannerInfo *root, } set_cheapest(grouped_rel); + + populate_grouprel_uniquekeys(root, grouped_rel, input_rel); return grouped_rel; } @@ -4621,7 +4625,7 @@ create_window_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(window_rel); - + simple_copy_uniquekeys(input_rel, window_rel); return window_rel; } @@ -4935,7 +4939,7 @@ create_distinct_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(distinct_rel); - + populate_distinctrel_uniquekeys(root, input_rel, distinct_rel); return distinct_rel; } @@ -5196,6 +5200,8 @@ create_ordered_paths(PlannerInfo *root, */ Assert(ordered_rel->pathlist != NIL); + simple_copy_uniquekeys(input_rel, ordered_rel); + return ordered_rel; } @@ -6073,6 +6079,9 @@ adjust_paths_for_srfs(PlannerInfo *root, RelOptInfo *rel, if (list_length(targets) == 1) return; + /* UniqueKey is not valid after handling the SRF. */ + rel->uniquekeys = NIL; + /* * Stack SRF-evaluation nodes atop each path for the rel. * diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c index becdcbb872..b7626545bf 100644 --- a/src/backend/optimizer/prep/prepunion.c +++ b/src/backend/optimizer/prep/prepunion.c @@ -689,6 +689,8 @@ generate_union_paths(SetOperationStmt *op, PlannerInfo *root, /* Undo effects of possibly forcing tuple_fraction to 0 */ root->tuple_fraction = save_fraction; + /* Add the UniqueKeys */ + populate_unionrel_uniquekeys(root, result_rel); return result_rel; } diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c index 86922a273c..6817c9c787 100644 --- a/src/backend/optimizer/util/appendinfo.c +++ b/src/backend/optimizer/util/appendinfo.c @@ -746,3 +746,47 @@ find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos) } return appinfos; } + +/* + * find_appinfo_by_child + * + */ +AppendRelInfo * +find_appinfo_by_child(PlannerInfo *root, Index child_index) +{ + ListCell *lc; + foreach(lc, root->append_rel_list) + { + AppendRelInfo *appinfo = lfirst_node(AppendRelInfo, lc); + if (appinfo->child_relid == child_index) + return appinfo; + } + elog(ERROR, "parent relation cant be found"); + return NULL; +} + +/* + * find_parent_var + * + */ +Var * +find_parent_var(AppendRelInfo *appinfo, Var *child_var) +{ + ListCell *lc; + Var *res = NULL; + Index attno = 1; + foreach(lc, appinfo->translated_vars) + { + Node *child_node = lfirst(lc); + if (equal(child_node, child_var)) + { + res = copyObject(child_var); + res->varattno = attno; + res->varno = appinfo->parent_relid; + } + attno++; + } + if (res == NULL) + elog(ERROR, "parent var can't be found."); + return res; +} diff --git a/src/backend/optimizer/util/inherit.c b/src/backend/optimizer/util/inherit.c index be1c9ddd96..3eec1f4d74 100644 --- a/src/backend/optimizer/util/inherit.c +++ b/src/backend/optimizer/util/inherit.c @@ -736,6 +736,7 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, { Node *onecq = (Node *) lfirst(lc2); bool pseudoconstant; + RestrictInfo *child_rinfo; /* check for pseudoconstant (no Vars or volatile functions) */ pseudoconstant = @@ -747,14 +748,15 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, root->hasPseudoConstantQuals = true; } /* reconstitute RestrictInfo with appropriate properties */ - childquals = lappend(childquals, - make_restrictinfo(root, - (Expr *) onecq, - rinfo->is_pushed_down, - rinfo->outerjoin_delayed, - pseudoconstant, - rinfo->security_level, - NULL, NULL, NULL)); + child_rinfo = make_restrictinfo(root, + (Expr *) onecq, + rinfo->is_pushed_down, + rinfo->outerjoin_delayed, + pseudoconstant, + rinfo->security_level, + NULL, NULL, NULL); + child_rinfo->mergeopfamilies = rinfo->mergeopfamilies; + childquals = lappend(childquals, child_rinfo); /* track minimum security level among child quals */ cq_min_security = Min(cq_min_security, rinfo->security_level); } diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 48a7ebfe45..4fe1824eb0 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -16,6 +16,7 @@ #include "nodes/execnodes.h" #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" extern A_Expr *makeA_Expr(A_Expr_Kind kind, List *name, @@ -106,4 +107,6 @@ extern GroupingSet *makeGroupingSet(GroupingSetKind kind, List *content, int loc extern VacuumRelation *makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols); +extern UniqueKey* makeUniqueKey(List *exprs, bool multi_nullvals); + #endif /* MAKEFUNC_H */ diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index e22df890ef..3320273ac1 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -265,6 +265,7 @@ typedef enum NodeTag T_EquivalenceMember, T_PathKey, T_PathTarget, + T_UniqueKey, T_RestrictInfo, T_IndexClause, T_PlaceHolderVar, diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index 0d61f04d27..80561b1003 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -739,6 +739,7 @@ typedef struct RelOptInfo QualCost baserestrictcost; /* cost of evaluating the above */ Index baserestrict_min_security; /* min security_level found in * baserestrictinfo */ + List *uniquekeys; /* List of UniqueKey */ List *joininfo; /* RestrictInfo structures for join clauses * involving this rel */ bool has_eclass_joins; /* T means joininfo is incomplete */ @@ -1059,6 +1060,28 @@ typedef struct PathKey } PathKey; +/* + * UniqueKey + * + * Represents the unique properties held by a RelOptInfo. + * + * exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL + * is a special case of UniqueKey, which means there is only 1 row in that + * relation. + * multi_nullvals: true means multi null values may exist in these exprs, so the + * uniqueness is not guaranteed in this case. This field is necessary for + * remove_useless_join & reduce_unique_semijoins where we don't mind these + * duplicated NULL values. It is set to true for 2 cases. One is a unique key + * from a unique index but the related column is nullable. The other one is for + * outer join. see populate_joinrel_uniquekeys for detail. + */ +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + /* * PathTarget * @@ -2496,7 +2519,7 @@ typedef enum * * flags indicating what kinds of grouping are possible. * partial_costs_set is true if the agg_partial_costs and agg_final_costs - * have been initialized. + * have been initialized. * agg_partial_costs gives partial aggregation costs. * agg_final_costs gives finalization costs. * target_parallel_safe is true if target is parallel safe. @@ -2526,8 +2549,8 @@ typedef struct * limit_tuples is an estimated bound on the number of output tuples, * or -1 if no LIMIT or couldn't estimate. * count_est and offset_est are the estimated values of the LIMIT and OFFSET - * expressions computed by preprocess_limit() (see comments for - * preprocess_limit() for more information). + * expressions computed by preprocess_limit() (see comments for + * preprocess_limit() for more information). */ typedef struct { diff --git a/src/include/nodes/pg_list.h b/src/include/nodes/pg_list.h index 404e03f132..85c65e910c 100644 --- a/src/include/nodes/pg_list.h +++ b/src/include/nodes/pg_list.h @@ -566,6 +566,16 @@ extern pg_nodiscard List *list_delete_first(List *list); extern pg_nodiscard List *list_delete_last(List *list); extern pg_nodiscard List *list_delete_nth_cell(List *list, int n); extern pg_nodiscard List *list_delete_cell(List *list, ListCell *cell); +extern bool list_is_subset(const List *members, const List *target); + +extern List *list_delete(List *list, void *datum); +extern List *list_delete_ptr(List *list, void *datum); +extern List *list_delete_int(List *list, int datum); +extern List *list_delete_oid(List *list, Oid datum); +extern List *list_delete_first(List *list); +extern List *list_delete_last(List *list); +extern List *list_delete_nth_cell(List *list, int n); +extern List *list_delete_cell(List *list, ListCell *cell); extern List *list_union(const List *list1, const List *list2); extern List *list_union_ptr(const List *list1, const List *list2); diff --git a/src/include/optimizer/appendinfo.h b/src/include/optimizer/appendinfo.h index 4cbf8c26cc..43b8d05f69 100644 --- a/src/include/optimizer/appendinfo.h +++ b/src/include/optimizer/appendinfo.h @@ -32,4 +32,7 @@ extern Relids adjust_child_relids_multilevel(PlannerInfo *root, Relids relids, extern AppendRelInfo **find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos); +extern AppendRelInfo *find_appinfo_by_child(PlannerInfo *root, Index child_index); +extern Var *find_parent_var(AppendRelInfo *appinfo, Var *child_var); + #endif /* APPENDINFO_H */ diff --git a/src/include/optimizer/optimizer.h b/src/include/optimizer/optimizer.h index d587952b7d..843aafa51e 100644 --- a/src/include/optimizer/optimizer.h +++ b/src/include/optimizer/optimizer.h @@ -23,6 +23,7 @@ #define OPTIMIZER_H #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" /* Test if an expression node represents a SRF call. Beware multiple eval! */ #define IS_SRF_CALL(node) \ @@ -168,6 +169,7 @@ extern TargetEntry *get_sortgroupref_tle(Index sortref, List *targetList); extern TargetEntry *get_sortgroupclause_tle(SortGroupClause *sgClause, List *targetList); +extern Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); extern Node *get_sortgroupclause_expr(SortGroupClause *sgClause, List *targetList); extern List *get_sortgrouplist_exprs(List *sgClauses, diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h index 035d3e1206..1adf99c9ee 100644 --- a/src/include/optimizer/paths.h +++ b/src/include/optimizer/paths.h @@ -247,5 +247,48 @@ extern PathKey *make_canonical_pathkey(PlannerInfo *root, int strategy, bool nulls_first); extern void add_paths_to_append_rel(PlannerInfo *root, RelOptInfo *rel, List *live_childrels); +extern List *select_mergejoin_clauses(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + JoinType jointype, + bool *mergejoin_allowed); + +/* + * uniquekeys.c + * Utilities for matching and building unique keys + */ +extern void populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List* unique_index_list); +extern void populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels); +extern void populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel); +extern void populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel); +extern void populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel); +extern void simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel); +extern void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel); +extern void populate_joinrel_uniquekeys(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + RelOptInfo *rel2, + List *restrictlist, + JoinType jointype); + +extern bool relation_has_uniquekeys_for(PlannerInfo *root, + RelOptInfo *rel, + List *exprs, + bool allow_multinulls); +extern bool relation_is_onerow(RelOptInfo *rel); #endif /* PATHS_H */ -- 2.26.2 --yh7cw345mbttadjz Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v38-0003-Extend-UniqueKeys.patch" ^ permalink raw reply [nested|flat] 91+ messages in thread
* [PATCH v36 2/6] Introduce UniqueKey attributes on RelOptInfo struct. @ 2020-05-11 07:50 一挃 <[email protected]> 0 siblings, 0 replies; 91+ messages in thread From: 一挃 @ 2020-05-11 07:50 UTC (permalink / raw) UniqueKey is a set of exprs on RelOptInfo which represents the exprs will be unique on the given RelOptInfo. You can see README.uniquekey for more information. --- src/backend/nodes/copyfuncs.c | 13 + src/backend/nodes/list.c | 31 + src/backend/nodes/makefuncs.c | 13 + src/backend/nodes/outfuncs.c | 11 + src/backend/nodes/readfuncs.c | 10 + src/backend/optimizer/path/Makefile | 3 +- src/backend/optimizer/path/README.uniquekey | 131 +++ src/backend/optimizer/path/allpaths.c | 10 + src/backend/optimizer/path/joinpath.c | 9 +- src/backend/optimizer/path/joinrels.c | 2 + src/backend/optimizer/path/pathkeys.c | 3 +- src/backend/optimizer/path/uniquekeys.c | 1131 +++++++++++++++++++ src/backend/optimizer/plan/planner.c | 13 +- src/backend/optimizer/prep/prepunion.c | 2 + src/backend/optimizer/util/appendinfo.c | 44 + src/backend/optimizer/util/inherit.c | 16 +- src/include/nodes/makefuncs.h | 3 + src/include/nodes/nodes.h | 1 + src/include/nodes/pathnodes.h | 29 +- src/include/nodes/pg_list.h | 2 + src/include/optimizer/appendinfo.h | 3 + src/include/optimizer/optimizer.h | 2 + src/include/optimizer/paths.h | 43 + 23 files changed, 1502 insertions(+), 23 deletions(-) create mode 100644 src/backend/optimizer/path/README.uniquekey create mode 100644 src/backend/optimizer/path/uniquekeys.c diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 89c409de66..1f50400fd2 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2273,6 +2273,16 @@ _copyPathKey(const PathKey *from) return newnode; } +static UniqueKey * +_copyUniqueKey(const UniqueKey *from) +{ + UniqueKey *newnode = makeNode(UniqueKey); + + COPY_NODE_FIELD(exprs); + COPY_SCALAR_FIELD(multi_nullvals); + + return newnode; +} /* * _copyRestrictInfo */ @@ -5152,6 +5162,9 @@ copyObjectImpl(const void *from) case T_PathKey: retval = _copyPathKey(from); break; + case T_UniqueKey: + retval = _copyUniqueKey(from); + break; case T_RestrictInfo: retval = _copyRestrictInfo(from); break; diff --git a/src/backend/nodes/list.c b/src/backend/nodes/list.c index 80fa8c84e4..a7a99b70f2 100644 --- a/src/backend/nodes/list.c +++ b/src/backend/nodes/list.c @@ -687,6 +687,37 @@ list_member_oid(const List *list, Oid datum) return false; } +/* + * return true iff every entry in "members" list is also present + * in the "target" list. + */ +bool +list_is_subset(const List *members, const List *target) +{ + const ListCell *lc1, *lc2; + + Assert(IsPointerList(members)); + Assert(IsPointerList(target)); + check_list_invariants(members); + check_list_invariants(target); + + foreach(lc1, members) + { + bool found = false; + foreach(lc2, target) + { + if (equal(lfirst(lc1), lfirst(lc2))) + { + found = true; + break; + } + } + if (!found) + return false; + } + return true; +} + /* * Delete the n'th cell (counting from 0) in list. * diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 49de285f01..646cf7c9a1 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -814,3 +814,16 @@ makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols) v->va_cols = va_cols; return v; } + + +/* + * makeUniqueKey + */ +UniqueKey* +makeUniqueKey(List *exprs, bool multi_nullvals) +{ + UniqueKey * ukey = makeNode(UniqueKey); + ukey->exprs = exprs; + ukey->multi_nullvals = multi_nullvals; + return ukey; +} diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index e2f177515d..c3a9632992 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2428,6 +2428,14 @@ _outPathKey(StringInfo str, const PathKey *node) WRITE_BOOL_FIELD(pk_nulls_first); } +static void +_outUniqueKey(StringInfo str, const UniqueKey *node) +{ + WRITE_NODE_TYPE("UNIQUEKEY"); + WRITE_NODE_FIELD(exprs); + WRITE_BOOL_FIELD(multi_nullvals); +} + static void _outPathTarget(StringInfo str, const PathTarget *node) { @@ -4127,6 +4135,9 @@ outNode(StringInfo str, const void *obj) case T_PathKey: _outPathKey(str, obj); break; + case T_UniqueKey: + _outUniqueKey(str, obj); + break; case T_PathTarget: _outPathTarget(str, obj); break; diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index 42050ab719..3a18571d0c 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -452,6 +452,14 @@ _readSetOperationStmt(void) READ_DONE(); } +static UniqueKey * +_readUniqueKey(void) +{ + READ_LOCALS(UniqueKey); + READ_NODE_FIELD(exprs); + READ_BOOL_FIELD(multi_nullvals); + READ_DONE(); +} /* * Stuff from primnodes.h. @@ -2656,6 +2664,8 @@ parseNodeString(void) return_value = _readCommonTableExpr(); else if (MATCH("SETOPERATIONSTMT", 16)) return_value = _readSetOperationStmt(); + else if (MATCH("UNIQUEKEY", 9)) + return_value = _readUniqueKey(); else if (MATCH("ALIAS", 5)) return_value = _readAlias(); else if (MATCH("RANGEVAR", 8)) diff --git a/src/backend/optimizer/path/Makefile b/src/backend/optimizer/path/Makefile index 1e199ff66f..7b9820c25f 100644 --- a/src/backend/optimizer/path/Makefile +++ b/src/backend/optimizer/path/Makefile @@ -21,6 +21,7 @@ OBJS = \ joinpath.o \ joinrels.o \ pathkeys.o \ - tidpath.o + tidpath.o \ + uniquekeys.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/optimizer/path/README.uniquekey b/src/backend/optimizer/path/README.uniquekey new file mode 100644 index 0000000000..5eac761995 --- /dev/null +++ b/src/backend/optimizer/path/README.uniquekey @@ -0,0 +1,131 @@ +1. What is UniqueKey? +We can think UniqueKey is a set of exprs for a RelOptInfo, which we are insure +that doesn't yields same result among all the rows. The simplest UniqueKey +format is primary key. + +However we define the UnqiueKey as below. + +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + +exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL +is a special case of UniqueKey, which means there is only one row in that +relation.it has a stronger semantic than others. like SELECT uk FROM t; uk is +normal unique key and may have different values. SELECT colx FROM t WHERE uk = +const. colx is unique AND we have only 1 value. This field can used for +innerrel_is_unique. this logic is handled specially in add_uniquekey_for_onerow +function. + +multi_nullvals: true means multi null values may exist in these exprs, so the +uniqueness is not guaranteed in this case. This field is necessary for +remove_useless_join & reduce_unique_semijoins where we don't mind these +duplicated NULL values. It is set to true for 2 cases. One is a unique key +from a unique index but the related column is nullable. The other one is for +outer join. see populate_joinrel_uniquekeys for detail. + + +The UniqueKey can be used at the following cases at least: +1. remove_useless_joins. +2. reduce_semianti_joins +3. remove distinct node if distinct clause is unique. +4. remove aggnode if group by clause is unique. +5. Index Skip Scan (WIP) +6. Aggregation Push Down without 2 phase aggregation if the join can't + duplicated the aggregated rows. (work in progress feature) + +2. How is it maintained? + +We have a set of populate_xxx_unqiuekeys functions to maintain the uniquekey on +various cases. xxx includes baserel, joinrel, partitionedrel, distinctrel, +groupedrel, unionrel. and we also need to convert the uniquekey from subquery +to outer relation, which is what convert_subquery_uniquekeys does. + +1. The first part is about baserel. We handled 3 cases. suppose we have Unique +Index on (a, b). + +1. SELECT a, b FROM t. UniqueKey (a, b) +2. SELECT a FROM t WHERE b = 1; UniqueKey (a) +3. SELECT .. FROM t WHERE a = 1 AND b = 1; UniqueKey (NIL). onerow case, every + column is Unique. + +2. The next part is joinrel, this part is most error-prone, we simplified the rules +like below: +1. If the relation's UniqueKey can't be duplicated after join, then is will be + still valid for the join rel. The function we used here is + innerrel_keeps_unique. The basic idea is innerrel.any_col = outer.uk. + +2. If the UnqiueKey can't keep valid via the rule 1, the combination of the + UniqueKey from both sides are valid for sure. We can prove this as: if the + unique exprs from rel1 is duplicated by rel2, the duplicated rows must + contains different unique exprs from rel2. + +More considerations about onerow: +1. If relation with one row and it can't be duplicated, it is still possible + contains mulit_nullvas after outer join. +2. If the either UniqueKey can be duplicated after join, the can get one row + only when both side is one row AND there is no outer join. +3. Whenever the onerow UniqueKey is not a valid any more, we need to convert one + row UniqueKey to normal unique key since we don't store exprs for one-row + relation. get_exprs_from_uniquekeys will be used here. + + +More considerations about multi_nullvals after join: +1. If the original UnqiueKey has multi_nullvals, the final UniqueKey will have + mulit_nullvals in any case. +2. If a unique key doesn't allow mulit_nullvals, after some outer join, it + allows some outer join. + + +3. When we comes to subquery, we need to convert_subquery_unqiuekeys just like +convert_subquery_pathkeys. Only the UniqueKey insides subquery is referenced as +a Var in outer relation will be reused. The relationship between the outerrel.Var +and subquery.exprs is built with outerel->subroot->processed_tlist. + + +4. As for the SRF functions, it will break the uniqueness of uniquekey, However it +is handled in adjust_paths_for_srfs, which happens after the query_planner. so +we will maintain the UniqueKey until there and reset it to NIL at that +places. This can't help on distinct/group by elimination cases but probably help +in some other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is +semantic correctly. + + +5. As for inherit table, we first main the UnqiueKey on childrel as well. But for +partitioned table we need to maintain 2 different kinds of +UnqiueKey. 1). UniqueKey on the parent relation 2). UniqueKey on child +relation for partition wise query. + +Example: +CREATE TABLE p (a int not null, b int not null) partition by list (a); +CREATE TABLE p0 partition of p for values in (1); +CREATE TABLE p1 partition of p for values in (2); + +create unique index p0_b on p0(b); +create unique index p1_b on p1(b); + +Now b is only unique on partition level, so the distinct can't be removed on +the following cases. SELECT DISTINCT b FROM p; + +Another example is SELECT DISTINCT a, b FROM p WHERE a = 1; Since only one +partition is chosen, the UniqueKey on child relation is same as the UniqueKey on +parent relation. + +Another usage of UniqueKey on partition level is it be helpful for +partition-wise join. + +As for the UniqueKey on parent table level, it comes with 2 different ways, +1). the UniqueKey is also derived in UniqueKey index, but the index must be same +in all the related children relations and the unique index must contains +Partition Key in it. Example: + +CREATE UNIQUE INDEX p_ab ON p(a, b); -- where a is the partition key. + +-- Query +SELECT a, b FROM p; the (a, b) is a UniqueKey of p. + +2). If the parent relation has only one childrel, the UniqueKey on childrel is + the UniqueKey on parent as well. diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index 484dab0a1a..2ad9d06d7a 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -579,6 +579,12 @@ set_plain_rel_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) */ check_index_predicates(root, rel); + /* + * Now that we've marked which partial indexes are suitable, we can now + * build the relation's unique keys. + */ + populate_baserel_uniquekeys(root, rel, rel->indexlist); + /* Mark rel with estimated output rows, width, etc */ set_baserel_size_estimates(root, rel); } @@ -1310,6 +1316,8 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, /* Add paths to the append relation. */ add_paths_to_append_rel(root, rel, live_childrels); + if (IS_PARTITIONED_REL(rel)) + populate_partitionedrel_uniquekeys(root, rel, live_childrels); } @@ -2383,6 +2391,8 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, pathkeys, required_outer)); } + convert_subquery_uniquekeys(root, rel, sub_final_rel); + /* If outer rel allows parallelism, do same for partial paths. */ if (rel->consider_parallel && bms_is_empty(required_outer)) { diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c index db54a6ba2e..ef0fd2fb0b 100644 --- a/src/backend/optimizer/path/joinpath.c +++ b/src/backend/optimizer/path/joinpath.c @@ -71,13 +71,6 @@ static void consider_parallel_mergejoin(PlannerInfo *root, static void hash_inner_and_outer(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, RelOptInfo *innerrel, JoinType jointype, JoinPathExtraData *extra); -static List *select_mergejoin_clauses(PlannerInfo *root, - RelOptInfo *joinrel, - RelOptInfo *outerrel, - RelOptInfo *innerrel, - List *restrictlist, - JoinType jointype, - bool *mergejoin_allowed); static void generate_mergejoin_paths(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *innerrel, @@ -1927,7 +1920,7 @@ hash_inner_and_outer(PlannerInfo *root, * if it is mergejoinable and involves vars from the two sub-relations * currently of interest. */ -static List * +List * select_mergejoin_clauses(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c index 2d343cd293..b9163ee8ff 100644 --- a/src/backend/optimizer/path/joinrels.c +++ b/src/backend/optimizer/path/joinrels.c @@ -924,6 +924,8 @@ populate_joinrel_with_paths(PlannerInfo *root, RelOptInfo *rel1, /* Apply partitionwise join technique, if possible. */ try_partitionwise_join(root, rel1, rel2, joinrel, sjinfo, restrictlist); + + populate_joinrel_uniquekeys(root, joinrel, rel1, rel2, restrictlist, sjinfo->jointype); } diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index ce9bf87e9b..7e596d4194 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -33,7 +33,6 @@ static bool pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys); static bool matches_boolean_partition_clause(RestrictInfo *rinfo, RelOptInfo *partrel, int partkeycol); -static Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); static bool right_merge_direction(PlannerInfo *root, PathKey *pathkey); @@ -1035,7 +1034,7 @@ convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel, * We need this to ensure that we don't return pathkeys describing values * that are unavailable above the level of the subquery scan. */ -static Var * +Var * find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle) { ListCell *lc; diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c new file mode 100644 index 0000000000..b33bcd2f32 --- /dev/null +++ b/src/backend/optimizer/path/uniquekeys.c @@ -0,0 +1,1131 @@ +/*------------------------------------------------------------------------- + * + * uniquekeys.c + * Utilities for matching and building unique keys + * + * Portions Copyright (c) 2020, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/optimizer/path/uniquekeys.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "nodes/makefuncs.h" +#include "nodes/nodeFuncs.h" +#include "optimizer/pathnode.h" +#include "optimizer/paths.h" +#include "optimizer/appendinfo.h" +#include "optimizer/optimizer.h" +#include "optimizer/tlist.h" +#include "rewrite/rewriteManip.h" + + +/* + * This struct is used to help populate_joinrel_uniquekeys. + * + * added_to_joinrel is true if a uniquekey (from outerrel or innerrel) + * has been added to joinrel. + * useful is true if the exprs of the uniquekey still appears in joinrel. + */ +typedef struct UniqueKeyContextData +{ + UniqueKey *uniquekey; + bool added_to_joinrel; + bool useful; +} *UniqueKeyContext; + +static List *initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel); +static bool innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + bool reverse); + +static List *get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals); +static List *get_exprs_from_uniquekey(RelOptInfo *joinrel, + RelOptInfo *rel1, + UniqueKey *ukey); +static void add_uniquekey_for_onerow(RelOptInfo *rel); +static bool add_combined_uniquekey(RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype); + +/* Used for unique indexes checking for partitioned table */ +static bool index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind); +static IndexOptInfo *simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from); +static bool simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2); +static void adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_index); + +/* Helper function for grouped relation and distinct relation. */ +static void add_uniquekey_from_sortgroups(PlannerInfo *root, + RelOptInfo *rel, + List *sortgroups); + +/* + * populate_baserel_uniquekeys + * Populate 'baserel' uniquekeys list by looking at the rel's unique index + * and baserestrictinfo + */ +void +populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List *indexlist) +{ + ListCell *lc; + List *matched_uniq_indexes = NIL; + + /* Attrs appears in rel->reltarget->exprs. */ + Bitmapset *used_attrs = NULL; + + List *const_exprs = NIL; + List *expr_opfamilies = NIL; + + Assert(baserel->rtekind == RTE_RELATION); + + foreach(lc, indexlist) + { + IndexOptInfo *ind = (IndexOptInfo *) lfirst(lc); + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + matched_uniq_indexes = lappend(matched_uniq_indexes, ind); + } + + if (matched_uniq_indexes == NIL) + return; + + /* Check which attrs is used in baserel->reltarget */ + pull_varattnos((Node *)baserel->reltarget->exprs, baserel->relid, &used_attrs); + + /* Check which attrno is used at a mergeable const filter */ + foreach(lc, baserel->baserestrictinfo) + { + RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); + + if (rinfo->mergeopfamilies == NIL) + continue; + + if (bms_is_empty(rinfo->left_relids)) + { + const_exprs = lappend(const_exprs, get_rightop(rinfo->clause)); + } + else if (bms_is_empty(rinfo->right_relids)) + { + const_exprs = lappend(const_exprs, get_leftop(rinfo->clause)); + } + else + continue; + + expr_opfamilies = lappend(expr_opfamilies, rinfo->mergeopfamilies); + } + + foreach(lc, matched_uniq_indexes) + { + bool multi_nullvals, useful; + List *exprs = get_exprs_from_uniqueindex(lfirst_node(IndexOptInfo, lc), + const_exprs, + expr_opfamilies, + used_attrs, + &useful, + &multi_nullvals); + if (useful) + { + if (exprs == NIL) + { + /* All the columns in Unique Index matched with a restrictinfo */ + add_uniquekey_for_onerow(baserel); + return; + } + baserel->uniquekeys = lappend(baserel->uniquekeys, + makeUniqueKey(exprs, multi_nullvals)); + } + } +} + + +/* + * populate_partitionedrel_uniquekeys + * The UniqueKey on partitionrel comes from 2 cases: + * 1). Only one partition is involved in this query, the unique key can be + * copied to parent rel from childrel. + * 2). There are some unique index which includes partition key and exists + * in all the related partitions. + * We never mind rule 2 if we hit rule 1. + */ + +void +populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels) +{ + ListCell *lc; + List *global_uniq_indexlist = NIL; + RelOptInfo *childrel; + bool is_first = true; + + Assert(IS_PARTITIONED_REL(rel)); + + if (childrels == NIL) + return; + + /* + * If there is only one partition used in this query, the UniqueKey in childrel is + * still valid in parent level, but we need convert the format from child expr to + * parent expr. + */ + if (list_length(childrels) == 1) + { + /* Check for Rule 1 */ + RelOptInfo *childrel = linitial_node(RelOptInfo, childrels); + ListCell *lc; + Assert(childrel->reloptkind == RELOPT_OTHER_MEMBER_REL); + if (relation_is_onerow(childrel)) + { + add_uniquekey_for_onerow(rel); + return; + } + + foreach(lc, childrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + AppendRelInfo *appinfo = find_appinfo_by_child(root, childrel->relid); + List *parent_exprs = NIL; + bool can_reuse = true; + ListCell *lc2; + foreach(lc2, ukey->exprs) + { + Var *var = (Var *)lfirst(lc2); + /* + * If the expr comes from a expression, it is hard to build the expression + * in parent so ignore that case for now. + */ + if(!IsA(var, Var)) + { + can_reuse = false; + break; + } + /* Convert it to parent var */ + parent_exprs = lappend(parent_exprs, find_parent_var(appinfo, var)); + } + if (can_reuse) + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(parent_exprs, + ukey->multi_nullvals)); + } + } + else + { + /* Check for rule 2 */ + childrel = linitial_node(RelOptInfo, childrels); + foreach(lc, childrel->indexlist) + { + IndexOptInfo *ind = lfirst(lc); + IndexOptInfo *modified_index; + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + + /* + * During simple_copy_indexinfo_to_parent, we need to convert var from + * child var to parent var, index on expression is too complex to handle. + * so ignore it for now. + */ + if (ind->indexprs != NIL) + continue; + + modified_index = simple_copy_indexinfo_to_parent(root, rel, ind); + /* + * If the unique index doesn't contain partkey, then it is unique + * on this partition only, so it is useless for us. + */ + if (!index_constains_partkey(rel, modified_index)) + continue; + + global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index); + } + + if (global_uniq_indexlist != NIL) + { + foreach(lc, childrels) + { + RelOptInfo *child = lfirst(lc); + if (is_first) + { + is_first = false; + continue; + } + adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist); + } + /* Now we have a list of unique index which are exactly same on all childrels, + * Set the UniqueKey just like it is non-partition table + */ + populate_baserel_uniquekeys(root, rel, global_uniq_indexlist); + } + } +} + + +/* + * populate_distinctrel_uniquekeys + */ +void +populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel) +{ + /* The unique key before the distinct is still valid. */ + distinctrel->uniquekeys = list_copy(inputrel->uniquekeys); + add_uniquekey_from_sortgroups(root, distinctrel, root->parse->distinctClause); +} + +/* + * populate_grouprel_uniquekeys + */ +void +populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel) + +{ + Query *parse = root->parse; + bool input_ukey_added = false; + ListCell *lc; + + if (relation_is_onerow(inputrel)) + { + add_uniquekey_for_onerow(grouprel); + return; + } + if (parse->groupingSets) + return; + + /* A Normal group by without grouping set. */ + if (parse->groupClause) + { + /* + * Current even the groupby clause is Unique already, but if query has aggref + * We have to create grouprel still. To keep the UnqiueKey short, we will check + * the UniqueKey of input_rel still valid, if so we reuse it. + */ + foreach(lc, inputrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (list_is_subset(ukey->exprs, grouprel->reltarget->exprs)) + { + grouprel->uniquekeys = lappend(grouprel->uniquekeys, + ukey); + input_ukey_added = true; + } + } + if (!input_ukey_added) + /* + * group by clause must be a super-set of grouprel->reltarget->exprs except the + * aggregation expr, so if such exprs is unique already, no bother to generate + * new uniquekey for group by exprs. + */ + add_uniquekey_from_sortgroups(root, + grouprel, + root->parse->groupClause); + } + else + /* It has aggregation but without a group by, so only one row returned */ + add_uniquekey_for_onerow(grouprel); +} + +/* + * simple_copy_uniquekeys + * Using a function for the one-line code makes us easy to check where we simply + * copied the uniquekey. + */ +void +simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel) +{ + newrel->uniquekeys = oldrel->uniquekeys; +} + +/* + * populate_unionrel_uniquekeys + */ +void +populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel) +{ + ListCell *lc; + List *exprs = NIL; + + Assert(unionrel->uniquekeys == NIL); + + foreach(lc, unionrel->reltarget->exprs) + { + exprs = lappend(exprs, lfirst(lc)); + } + + if (exprs == NIL) + /* SQL: select union select; is valid, we need to handle it here. */ + add_uniquekey_for_onerow(unionrel); + else + unionrel->uniquekeys = lappend(unionrel->uniquekeys, + makeUniqueKey(exprs,false)); + +} + +/* + * populate_joinrel_uniquekeys + * + * populate uniquekeys for joinrel. We will check each relation to see if its + * UniqueKey is still valid via innerrel_keeps_unique, if so, we add it to + * joinrel. The multi_nullvals field will be changed to true for some outer + * join cases and one-row UniqueKey needs to be converted to normal UniqueKey + * for the same case as well. + * For the uniquekey in either baserel which can't be unique after join, we still + * check to see if combination of UniqueKeys from both side is still useful for us. + * if yes, we add it to joinrel as well. + */ +void +populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *outerrel, RelOptInfo *innerrel, + List *restrictlist, JoinType jointype) +{ + ListCell *lc, *lc2; + List *clause_list = NIL; + List *outerrel_ukey_ctx; + List *innerrel_ukey_ctx; + bool inner_onerow, outer_onerow; + bool mergejoin_allowed; + + /* Care about the outerrel relation only for SEMI/ANTI join */ + if (jointype == JOIN_SEMI || jointype == JOIN_ANTI) + { + foreach(lc, outerrel->uniquekeys) + { + UniqueKey *uniquekey = lfirst_node(UniqueKey, lc); + if (list_is_subset(uniquekey->exprs, joinrel->reltarget->exprs)) + joinrel->uniquekeys = lappend(joinrel->uniquekeys, uniquekey); + } + return; + } + + Assert(jointype == JOIN_LEFT || jointype == JOIN_FULL || jointype == JOIN_INNER); + + /* Fast path */ + if (innerrel->uniquekeys == NIL || outerrel->uniquekeys == NIL) + return; + + inner_onerow = relation_is_onerow(innerrel); + outer_onerow = relation_is_onerow(outerrel); + + outerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(outerrel); + innerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(innerrel); + + clause_list = select_mergejoin_clauses(root, joinrel, outerrel, innerrel, + restrictlist, jointype, + &mergejoin_allowed); + + if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true /* reverse */)) + { + bool outer_impact = jointype == JOIN_FULL; + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + /* Outer relation has one row, and the unique key is not duplicated after join, + * the joinrel will still has one row unless the jointype == JOIN_FULL. + */ + if (outer_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (outer_onerow) + { + /* + * The onerow outerrel becomes multi rows and multi_nullvals + * will be changed to true. We also need to set the exprs correctly since it + * can't be NIL any more. + */ + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(joinrel, outerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Change multi_nullvals to true due to the full join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, true)); + else + /* Just reuse it */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + } + ctx->added_to_joinrel = true; + } + } + + if (innerrel_keeps_unique(root, outerrel, innerrel, clause_list, false)) + { + bool outer_impact = jointype == JOIN_FULL || jointype == JOIN_LEFT;; + + foreach(lc, innerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + if (inner_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (inner_onerow) + { + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(joinrel, innerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Need to change multi_nullvals to true due to the outer join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, + true)); + else + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + + } + ctx->added_to_joinrel = true; + } + } + + /* + * The combination of the UniqueKey from both sides is unique as well regardless + * of join type, but no bother to add it if its subset has been added to joinrel + * already or it is not useful for the joinrel. + */ + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx1 = (UniqueKeyContext) lfirst(lc); + if (ctx1->added_to_joinrel || !ctx1->useful) + continue; + foreach(lc2, innerrel_ukey_ctx) + { + UniqueKeyContext ctx2 = (UniqueKeyContext) lfirst(lc2); + if (ctx2->added_to_joinrel || !ctx2->useful) + continue; + if (add_combined_uniquekey(joinrel, outerrel, innerrel, + ctx1->uniquekey, ctx2->uniquekey, + jointype)) + /* If we set a onerow UniqueKey to joinrel, we don't need other. */ + return; + } + } +} + + +/* + * convert_subquery_uniquekeys + * + * Covert the UniqueKey in subquery to outer relation. + */ +void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel) +{ + ListCell *lc; + + if (sub_final_rel->uniquekeys == NIL) + return; + + if (relation_is_onerow(sub_final_rel)) + { + add_uniquekey_for_onerow(currel); + return; + } + + Assert(currel->subroot != NULL); + + foreach(lc, sub_final_rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + ListCell *lc; + List *exprs = NIL; + bool ukey_useful = true; + + /* One row case is handled above */ + Assert(ukey->exprs != NIL); + foreach(lc, ukey->exprs) + { + Var *var; + TargetEntry *tle = tlist_member(lfirst(lc), + currel->subroot->processed_tlist); + if (tle == NULL) + { + ukey_useful = false; + break; + } + var = find_var_for_subquery_tle(currel, tle); + if (var == NULL) + { + ukey_useful = false; + break; + } + exprs = lappend(exprs, var); + } + + if (ukey_useful) + currel->uniquekeys = lappend(currel->uniquekeys, + makeUniqueKey(exprs, + ukey->multi_nullvals)); + + } +} + +/* + * innerrel_keeps_unique + * + * Check if Unique key of the innerrel is valid after join. innerrel's UniqueKey + * will be still valid if innerrel's any-column mergeop outrerel's uniquekey + * exists in clause_list. + * + * Note: the clause_list must be a list of mergeable restrictinfo already. + */ +static bool +innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *clause_list, + bool reverse) +{ + ListCell *lc, *lc2, *lc3; + + if (outerrel->uniquekeys == NIL || innerrel->uniquekeys == NIL) + return false; + + /* Check if there is outerrel's uniquekey in mergeable clause. */ + foreach(lc, outerrel->uniquekeys) + { + List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs; + bool clauselist_matchs_all_exprs = true; + foreach(lc2, outer_uq_exprs) + { + Node *outer_uq_expr = lfirst(lc2); + bool find_uq_expr_in_clauselist = false; + foreach(lc3, clause_list) + { + RestrictInfo *rinfo = lfirst_node(RestrictInfo, lc3); + Node *outer_expr; + if (reverse) + outer_expr = rinfo->outer_is_left ? get_rightop(rinfo->clause) : get_leftop(rinfo->clause); + else + outer_expr = rinfo->outer_is_left ? get_leftop(rinfo->clause) : get_rightop(rinfo->clause); + if (equal(outer_expr, outer_uq_expr)) + { + find_uq_expr_in_clauselist = true; + break; + } + } + if (!find_uq_expr_in_clauselist) + { + /* No need to check the next exprs in the current uniquekey */ + clauselist_matchs_all_exprs = false; + break; + } + } + + if (clauselist_matchs_all_exprs) + return true; + } + return false; +} + + +/* + * relation_is_onerow + * Check if it is a one-row relation by checking UniqueKey. + */ +bool +relation_is_onerow(RelOptInfo *rel) +{ + UniqueKey *ukey; + if (rel->uniquekeys == NIL) + return false; + ukey = linitial_node(UniqueKey, rel->uniquekeys); + return ukey->exprs == NIL && list_length(rel->uniquekeys) == 1; +} + +/* + * relation_has_uniquekeys_for + * Returns true if we have proofs that 'rel' cannot return multiple rows with + * the same values in each of 'exprs'. Otherwise returns false. + */ +bool +relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel, + List *exprs, bool allow_multinulls) +{ + ListCell *lc; + + /* + * For UniqueKey->onerow case, the uniquekey->exprs is empty as well + * so we can't rely on list_is_subset to handle this special cases + */ + if (exprs == NIL) + return false; + + foreach(lc, rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (ukey->multi_nullvals && !allow_multinulls) + continue; + if (list_is_subset(ukey->exprs, exprs)) + return true; + } + return false; +} + + +/* + * get_exprs_from_uniqueindex + * + * Return a list of exprs which is unique. set useful to false if this + * unique index is not useful for us. + */ +static List * +get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals) +{ + List *exprs = NIL; + ListCell *indexpr_item; + int c = 0; + + *useful = true; + *multi_nullvals = false; + + indexpr_item = list_head(unique_index->indexprs); + for(c = 0; c < unique_index->ncolumns; c++) + { + int attr = unique_index->indexkeys[c]; + Expr *expr; + bool matched_const = false; + ListCell *lc1, *lc2; + + if(attr > 0) + { + expr = list_nth_node(TargetEntry, unique_index->indextlist, c)->expr; + } + else if (attr == 0) + { + /* Expression index */ + expr = lfirst(indexpr_item); + indexpr_item = lnext(unique_index->indexprs, indexpr_item); + } + else /* attr < 0 */ + { + /* Index on system column is not supported */ + Assert(false); + } + + /* + * Check index_col = Const case with regarding to opfamily checking + * If we can remove the index_col from the final UniqueKey->exprs. + */ + forboth(lc1, const_exprs, lc2, const_expr_opfamilies) + { + if (list_member_oid((List *)lfirst(lc2), unique_index->opfamily[c]) + && match_index_to_operand((Node *) lfirst(lc1), c, unique_index)) + { + matched_const = true; + break; + } + } + + if (matched_const) + continue; + + /* Check if the indexed expr is used in rel */ + if (attr > 0) + { + /* + * Normal Indexed column, if the col is not used, then the index is useless + * for uniquekey. + */ + attr -= FirstLowInvalidHeapAttributeNumber; + + if (!bms_is_member(attr, used_varattrs)) + { + *useful = false; + break; + } + } + else if (!list_member(unique_index->rel->reltarget->exprs, expr)) + { + /* Expression index but the expression is not used in rel */ + *useful = false; + break; + } + + /* check not null property. */ + if (attr == 0) + { + /* We never know if a expression yields null or not */ + *multi_nullvals = true; + } + else if (!bms_is_member(attr, unique_index->rel->notnullattrs) + && !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber, + unique_index->rel->notnullattrs)) + { + *multi_nullvals = true; + } + + exprs = lappend(exprs, expr); + } + return exprs; +} + + +/* + * add_uniquekey_for_onerow + * If we are sure that the relation only returns one row, then all the columns + * are unique. However we don't need to create UniqueKey for every column, we + * just set exprs = NIL and overwrites all the other UniqueKey on this RelOptInfo + * since this one has strongest semantics. + */ +void +add_uniquekey_for_onerow(RelOptInfo *rel) +{ + /* + * We overwrite the previous UniqueKey on purpose since this one has the + * strongest semantic. + */ + rel->uniquekeys = list_make1(makeUniqueKey(NIL, false)); +} + + +/* + * initililze_uniquecontext_for_joinrel + * Return a List of UniqueKeyContext for an inputrel + */ +static List * +initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel) +{ + List *res = NIL; + ListCell *lc; + foreach(lc, inputrel->uniquekeys) + { + UniqueKeyContext context; + context = palloc(sizeof(struct UniqueKeyContextData)); + context->uniquekey = lfirst_node(UniqueKey, lc); + context->added_to_joinrel = false; + context->useful = true; + res = lappend(res, context); + } + return res; +} + + +/* + * get_exprs_from_uniquekey + * Unify the way of get List of exprs from a one-row UniqueKey or + * normal UniqueKey. for the onerow case, every expr in rel1 is a valid + * UniqueKey. Return a List of exprs. + * + * rel1: The relation which you want to get the exprs. + * ukey: The UniqueKey you want to get the exprs. + */ +static List * +get_exprs_from_uniquekey(RelOptInfo *joinrel, RelOptInfo *rel1, UniqueKey *ukey) +{ + ListCell *lc; + bool onerow = rel1 != NULL && relation_is_onerow(rel1); + + List *res = NIL; + Assert(onerow || ukey); + if (onerow) + { + /* Only cares about the exprs still exist in joinrel */ + foreach(lc, joinrel->reltarget->exprs) + { + Bitmapset *relids = pull_varnos(lfirst(lc)); + if (bms_is_subset(relids, rel1->relids)) + { + res = lappend(res, list_make1(lfirst(lc))); + } + } + } + else + { + res = list_make1(ukey->exprs); + } + return res; +} + +/* + * Partitioned table Unique Keys. + * The partition table unique key is maintained as: + * 1. The index must be unique as usual. + * 2. The index must contains partition key. + * 3. The index must exist on all the child rel. see simple_indexinfo_equal for + * how we compare it. + */ + +/* + * index_constains_partkey + * return true if the index contains the partiton key. + */ +static bool +index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind) +{ + ListCell *lc; + int i; + Assert(IS_PARTITIONED_REL(partrel)); + Assert(partrel->part_scheme->partnatts > 0); + + for(i = 0; i < partrel->part_scheme->partnatts; i++) + { + Node *part_expr = linitial(partrel->partexprs[i]); + bool found_in_index = false; + foreach(lc, ind->indextlist) + { + Expr *index_expr = lfirst_node(TargetEntry, lc)->expr; + if (equal(index_expr, part_expr)) + { + found_in_index = true; + break; + } + } + if (!found_in_index) + return false; + } + return true; +} + +/* + * simple_indexinfo_equal + * + * Used to check if the 2 index is same as each other. The index here + * is COPIED from childrel and did some tiny changes(see + * simple_copy_indexinfo_to_parent) + */ +static bool +simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2) +{ + Size oid_cmp_len = sizeof(Oid) * ind1->ncolumns; + + return ind1->ncolumns == ind2->ncolumns && + ind1->unique == ind2->unique && + memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0 && + memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0 && + memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0 && + memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0 && + equal(get_tlist_exprs(ind1->indextlist, true), + get_tlist_exprs(ind2->indextlist, true)); +} + + +/* + * The below macros are used for simple_copy_indexinfo_to_parent which is so + * customized that I don't want to put it to copyfuncs.c. So copy it here. + */ +#define COPY_POINTER_FIELD(fldname, sz) \ + do { \ + Size _size = (sz); \ + newnode->fldname = palloc(_size); \ + memcpy(newnode->fldname, from->fldname, _size); \ + } while (0) + +#define COPY_NODE_FIELD(fldname) \ + (newnode->fldname = copyObjectImpl(from->fldname)) + +#define COPY_SCALAR_FIELD(fldname) \ + (newnode->fldname = from->fldname) + + +/* + * simple_copy_indexinfo_to_parent (from partition) + * Copy the IndexInfo from child relation to parent relation with some modification, + * which is used to test: + * 1. If the same index exists in all the childrels. + * 2. If the parentrel->reltarget/basicrestrict info matches this index. + */ +static IndexOptInfo * +simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from) +{ + IndexOptInfo *newnode = makeNode(IndexOptInfo); + AppendRelInfo *appinfo = find_appinfo_by_child(root, from->rel->relid); + ListCell *lc; + int idx = 0; + + COPY_SCALAR_FIELD(ncolumns); + COPY_SCALAR_FIELD(nkeycolumns); + COPY_SCALAR_FIELD(unique); + COPY_SCALAR_FIELD(immediate); + /* We just need to know if it is NIL or not */ + COPY_SCALAR_FIELD(indpred); + COPY_SCALAR_FIELD(predOK); + COPY_POINTER_FIELD(indexkeys, from->ncolumns * sizeof(int)); + COPY_POINTER_FIELD(indexcollations, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opfamily, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opcintype, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(sortopfamily, from->ncolumns * sizeof(Oid)); + COPY_NODE_FIELD(indextlist); + + /* Convert index exprs on child expr to expr on parent */ + foreach(lc, newnode->indextlist) + { + TargetEntry *tle = lfirst_node(TargetEntry, lc); + /* Index on expression is ignored */ + Assert(IsA(tle->expr, Var)); + tle->expr = (Expr *) find_parent_var(appinfo, (Var *) tle->expr); + newnode->indexkeys[idx] = castNode(Var, tle->expr)->varattno; + idx++; + } + newnode->rel = parentrel; + return newnode; +} + +/* + * adjust_partition_unique_indexlist + * + * global_unique_indexes: At the beginning, it contains the copy & modified + * unique index from the first partition. And then check if each index in it still + * exists in the following partitions. If no, remove it. at last, it has an + * index list which exists in all the partitions. + */ +static void +adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_indexes) +{ + ListCell *lc, *lc2; + foreach(lc, *global_unique_indexes) + { + IndexOptInfo *g_ind = lfirst_node(IndexOptInfo, lc); + bool found_in_child = false; + + foreach(lc2, childrel->indexlist) + { + IndexOptInfo *p_ind = lfirst_node(IndexOptInfo, lc2); + IndexOptInfo *p_ind_copy; + if (!p_ind->unique || !p_ind->immediate || + (p_ind->indpred != NIL && !p_ind->predOK)) + continue; + p_ind_copy = simple_copy_indexinfo_to_parent(root, parentrel, p_ind); + if (simple_indexinfo_equal(p_ind_copy, g_ind)) + { + found_in_child = true; + break; + } + } + if (!found_in_child) + /* The index doesn't exist in childrel, remove it from global_unique_indexes */ + *global_unique_indexes = foreach_delete_current(*global_unique_indexes, lc); + } +} + +/* Helper function for groupres/distinctrel */ +static void +add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgroups) +{ + Query *parse = root->parse; + List *exprs; + + /* + * XXX: If there are some vars which is not in current levelsup, the semantic is + * imprecise, should we avoid it or not? levelsup = 1 is just a demo, maybe we need to + * check every level other than 0, if so, looks we have to write another + * pull_var_walker. + */ + List *upper_vars = pull_vars_of_level((Node*)sortgroups, 1); + + if (upper_vars != NIL) + return; + + exprs = get_sortgrouplist_exprs(sortgroups, parse->targetList); + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(exprs, + false /* sortgroupclause can't be multi_nullvals */)); +} + + +/* + * add_combined_uniquekey + * The combination of both UniqueKeys is a valid UniqueKey for joinrel no matter + * the jointype. + */ +bool +add_combined_uniquekey(RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype) +{ + + ListCell *lc1, *lc2; + + /* Either side has multi_nullvals or we have outer join, + * the combined UniqueKey has multi_nullvals */ + bool multi_nullvals = outer_ukey->multi_nullvals || + inner_ukey->multi_nullvals || IS_OUTER_JOIN(jointype); + + /* The only case we can get onerow joinrel after join */ + if (relation_is_onerow(outer_rel) + && relation_is_onerow(inner_rel) + && jointype == JOIN_INNER) + { + add_uniquekey_for_onerow(joinrel); + return true; + } + + foreach(lc1, get_exprs_from_uniquekey(joinrel, outer_rel, outer_ukey)) + { + foreach(lc2, get_exprs_from_uniquekey(joinrel, inner_rel, inner_ukey)) + { + List *exprs = list_concat_copy(lfirst_node(List, lc1), lfirst_node(List, lc2)); + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(exprs, + multi_nullvals)); + } + } + return false; +} diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index b406d41e91..0551ae0512 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -2389,6 +2389,8 @@ grouping_planner(PlannerInfo *root, bool inheritance_update, add_path(final_rel, path); } + simple_copy_uniquekeys(current_rel, final_rel); + /* * Generate partial paths for final_rel, too, if outer query levels might * be able to make use of them. @@ -3899,6 +3901,8 @@ create_grouping_paths(PlannerInfo *root, } set_cheapest(grouped_rel); + + populate_grouprel_uniquekeys(root, grouped_rel, input_rel); return grouped_rel; } @@ -4615,7 +4619,7 @@ create_window_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(window_rel); - + simple_copy_uniquekeys(input_rel, window_rel); return window_rel; } @@ -4911,7 +4915,7 @@ create_distinct_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(distinct_rel); - + populate_distinctrel_uniquekeys(root, input_rel, distinct_rel); return distinct_rel; } @@ -5172,6 +5176,8 @@ create_ordered_paths(PlannerInfo *root, */ Assert(ordered_rel->pathlist != NIL); + simple_copy_uniquekeys(input_rel, ordered_rel); + return ordered_rel; } @@ -6049,6 +6055,9 @@ adjust_paths_for_srfs(PlannerInfo *root, RelOptInfo *rel, if (list_length(targets) == 1) return; + /* UniqueKey is not valid after handling the SRF. */ + rel->uniquekeys = NIL; + /* * Stack SRF-evaluation nodes atop each path for the rel. * diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c index 951aed80e7..e94e92937c 100644 --- a/src/backend/optimizer/prep/prepunion.c +++ b/src/backend/optimizer/prep/prepunion.c @@ -689,6 +689,8 @@ generate_union_paths(SetOperationStmt *op, PlannerInfo *root, /* Undo effects of possibly forcing tuple_fraction to 0 */ root->tuple_fraction = save_fraction; + /* Add the UniqueKeys */ + populate_unionrel_uniquekeys(root, result_rel); return result_rel; } diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c index d722063cf3..44c37ecffc 100644 --- a/src/backend/optimizer/util/appendinfo.c +++ b/src/backend/optimizer/util/appendinfo.c @@ -746,3 +746,47 @@ find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos) } return appinfos; } + +/* + * find_appinfo_by_child + * + */ +AppendRelInfo * +find_appinfo_by_child(PlannerInfo *root, Index child_index) +{ + ListCell *lc; + foreach(lc, root->append_rel_list) + { + AppendRelInfo *appinfo = lfirst_node(AppendRelInfo, lc); + if (appinfo->child_relid == child_index) + return appinfo; + } + elog(ERROR, "parent relation cant be found"); + return NULL; +} + +/* + * find_parent_var + * + */ +Var * +find_parent_var(AppendRelInfo *appinfo, Var *child_var) +{ + ListCell *lc; + Var *res = NULL; + Index attno = 1; + foreach(lc, appinfo->translated_vars) + { + Node *child_node = lfirst(lc); + if (equal(child_node, child_var)) + { + res = copyObject(child_var); + res->varattno = attno; + res->varno = appinfo->parent_relid; + } + attno++; + } + if (res == NULL) + elog(ERROR, "parent var can't be found."); + return res; +} diff --git a/src/backend/optimizer/util/inherit.c b/src/backend/optimizer/util/inherit.c index 3132fd35a5..d66b40ec50 100644 --- a/src/backend/optimizer/util/inherit.c +++ b/src/backend/optimizer/util/inherit.c @@ -736,6 +736,7 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, { Node *onecq = (Node *) lfirst(lc2); bool pseudoconstant; + RestrictInfo *child_rinfo; /* check for pseudoconstant (no Vars or volatile functions) */ pseudoconstant = @@ -747,13 +748,14 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, root->hasPseudoConstantQuals = true; } /* reconstitute RestrictInfo with appropriate properties */ - childquals = lappend(childquals, - make_restrictinfo((Expr *) onecq, - rinfo->is_pushed_down, - rinfo->outerjoin_delayed, - pseudoconstant, - rinfo->security_level, - NULL, NULL, NULL)); + child_rinfo = make_restrictinfo((Expr *) onecq, + rinfo->is_pushed_down, + rinfo->outerjoin_delayed, + pseudoconstant, + rinfo->security_level, + NULL, NULL, NULL); + child_rinfo->mergeopfamilies = rinfo->mergeopfamilies; + childquals = lappend(childquals, child_rinfo); /* track minimum security level among child quals */ cq_min_security = Min(cq_min_security, rinfo->security_level); } diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 31d9aedeeb..c83f17acb7 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -16,6 +16,7 @@ #include "nodes/execnodes.h" #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" extern A_Expr *makeA_Expr(A_Expr_Kind kind, List *name, @@ -105,4 +106,6 @@ extern GroupingSet *makeGroupingSet(GroupingSetKind kind, List *content, int loc extern VacuumRelation *makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols); +extern UniqueKey* makeUniqueKey(List *exprs, bool multi_nullvals); + #endif /* MAKEFUNC_H */ diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 381d84b4e4..41110ed888 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -264,6 +264,7 @@ typedef enum NodeTag T_EquivalenceMember, T_PathKey, T_PathTarget, + T_UniqueKey, T_RestrictInfo, T_IndexClause, T_PlaceHolderVar, diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index 9e3ebd488a..02e4458bef 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -730,6 +730,7 @@ typedef struct RelOptInfo QualCost baserestrictcost; /* cost of evaluating the above */ Index baserestrict_min_security; /* min security_level found in * baserestrictinfo */ + List *uniquekeys; /* List of UniqueKey */ List *joininfo; /* RestrictInfo structures for join clauses * involving this rel */ bool has_eclass_joins; /* T means joininfo is incomplete */ @@ -1047,6 +1048,28 @@ typedef struct PathKey } PathKey; +/* + * UniqueKey + * + * Represents the unique properties held by a RelOptInfo. + * + * exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL + * is a special case of UniqueKey, which means there is only 1 row in that + * relation. + * multi_nullvals: true means multi null values may exist in these exprs, so the + * uniqueness is not guaranteed in this case. This field is necessary for + * remove_useless_join & reduce_unique_semijoins where we don't mind these + * duplicated NULL values. It is set to true for 2 cases. One is a unique key + * from a unique index but the related column is nullable. The other one is for + * outer join. see populate_joinrel_uniquekeys for detail. + */ +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + /* * PathTarget * @@ -2473,7 +2496,7 @@ typedef enum * * flags indicating what kinds of grouping are possible. * partial_costs_set is true if the agg_partial_costs and agg_final_costs - * have been initialized. + * have been initialized. * agg_partial_costs gives partial aggregation costs. * agg_final_costs gives finalization costs. * target_parallel_safe is true if target is parallel safe. @@ -2503,8 +2526,8 @@ typedef struct * limit_tuples is an estimated bound on the number of output tuples, * or -1 if no LIMIT or couldn't estimate. * count_est and offset_est are the estimated values of the LIMIT and OFFSET - * expressions computed by preprocess_limit() (see comments for - * preprocess_limit() for more information). + * expressions computed by preprocess_limit() (see comments for + * preprocess_limit() for more information). */ typedef struct { diff --git a/src/include/nodes/pg_list.h b/src/include/nodes/pg_list.h index 14ea2766ad..621f54a9f8 100644 --- a/src/include/nodes/pg_list.h +++ b/src/include/nodes/pg_list.h @@ -528,6 +528,8 @@ extern bool list_member_ptr(const List *list, const void *datum); extern bool list_member_int(const List *list, int datum); extern bool list_member_oid(const List *list, Oid datum); +extern bool list_is_subset(const List *members, const List *target); + extern List *list_delete(List *list, void *datum); extern List *list_delete_ptr(List *list, void *datum); extern List *list_delete_int(List *list, int datum); diff --git a/src/include/optimizer/appendinfo.h b/src/include/optimizer/appendinfo.h index d6a27a60dd..e87c92a054 100644 --- a/src/include/optimizer/appendinfo.h +++ b/src/include/optimizer/appendinfo.h @@ -32,4 +32,7 @@ extern Relids adjust_child_relids_multilevel(PlannerInfo *root, Relids relids, extern AppendRelInfo **find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos); +extern AppendRelInfo *find_appinfo_by_child(PlannerInfo *root, Index child_index); +extern Var *find_parent_var(AppendRelInfo *appinfo, Var *child_var); + #endif /* APPENDINFO_H */ diff --git a/src/include/optimizer/optimizer.h b/src/include/optimizer/optimizer.h index 3e4171056e..9445141263 100644 --- a/src/include/optimizer/optimizer.h +++ b/src/include/optimizer/optimizer.h @@ -23,6 +23,7 @@ #define OPTIMIZER_H #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" /* * We don't want to include nodes/pathnodes.h here, because non-planner @@ -156,6 +157,7 @@ extern TargetEntry *get_sortgroupref_tle(Index sortref, List *targetList); extern TargetEntry *get_sortgroupclause_tle(SortGroupClause *sgClause, List *targetList); +extern Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); extern Node *get_sortgroupclause_expr(SortGroupClause *sgClause, List *targetList); extern List *get_sortgrouplist_exprs(List *sgClauses, diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h index 10b6e81079..9217a8d6c6 100644 --- a/src/include/optimizer/paths.h +++ b/src/include/optimizer/paths.h @@ -240,5 +240,48 @@ extern PathKey *make_canonical_pathkey(PlannerInfo *root, int strategy, bool nulls_first); extern void add_paths_to_append_rel(PlannerInfo *root, RelOptInfo *rel, List *live_childrels); +extern List *select_mergejoin_clauses(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + JoinType jointype, + bool *mergejoin_allowed); + +/* + * uniquekeys.c + * Utilities for matching and building unique keys + */ +extern void populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List* unique_index_list); +extern void populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels); +extern void populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel); +extern void populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel); +extern void populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel); +extern void simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel); +extern void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel); +extern void populate_joinrel_uniquekeys(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + RelOptInfo *rel2, + List *restrictlist, + JoinType jointype); + +extern bool relation_has_uniquekeys_for(PlannerInfo *root, + RelOptInfo *rel, + List *exprs, + bool allow_multinulls); +extern bool relation_is_onerow(RelOptInfo *rel); #endif /* PATHS_H */ -- 2.21.0 --op7mf2i72xk5yu7c Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v36-0003-Extend-UniqueKeys.patch" ^ permalink raw reply [nested|flat] 91+ messages in thread
* [PATCH v38 2/6] Introduce UniqueKey attributes on RelOptInfo struct. @ 2020-05-11 07:50 一挃 <[email protected]> 0 siblings, 0 replies; 91+ messages in thread From: 一挃 @ 2020-05-11 07:50 UTC (permalink / raw) UniqueKey is a set of exprs on RelOptInfo which represents the exprs will be unique on the given RelOptInfo. You can see README.uniquekey for more information. --- src/backend/nodes/copyfuncs.c | 13 + src/backend/nodes/list.c | 31 + src/backend/nodes/makefuncs.c | 13 + src/backend/nodes/outfuncs.c | 11 + src/backend/nodes/readfuncs.c | 10 + src/backend/optimizer/path/Makefile | 3 +- src/backend/optimizer/path/README.uniquekey | 131 +++ src/backend/optimizer/path/allpaths.c | 10 + src/backend/optimizer/path/joinpath.c | 9 +- src/backend/optimizer/path/joinrels.c | 2 + src/backend/optimizer/path/pathkeys.c | 3 +- src/backend/optimizer/path/uniquekeys.c | 1135 +++++++++++++++++++ src/backend/optimizer/plan/planner.c | 13 +- src/backend/optimizer/prep/prepunion.c | 2 + src/backend/optimizer/util/appendinfo.c | 44 + src/backend/optimizer/util/inherit.c | 18 +- src/include/nodes/makefuncs.h | 3 + src/include/nodes/nodes.h | 1 + src/include/nodes/pathnodes.h | 29 +- src/include/nodes/pg_list.h | 10 + src/include/optimizer/appendinfo.h | 3 + src/include/optimizer/optimizer.h | 2 + src/include/optimizer/paths.h | 43 + 23 files changed, 1515 insertions(+), 24 deletions(-) create mode 100644 src/backend/optimizer/path/README.uniquekey create mode 100644 src/backend/optimizer/path/uniquekeys.c diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..75c1c5e824 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2296,6 +2296,16 @@ _copyPathKey(const PathKey *from) return newnode; } +static UniqueKey * +_copyUniqueKey(const UniqueKey *from) +{ + UniqueKey *newnode = makeNode(UniqueKey); + + COPY_NODE_FIELD(exprs); + COPY_SCALAR_FIELD(multi_nullvals); + + return newnode; +} /* * _copyRestrictInfo */ @@ -5220,6 +5230,9 @@ copyObjectImpl(const void *from) case T_PathKey: retval = _copyPathKey(from); break; + case T_UniqueKey: + retval = _copyUniqueKey(from); + break; case T_RestrictInfo: retval = _copyRestrictInfo(from); break; diff --git a/src/backend/nodes/list.c b/src/backend/nodes/list.c index dbf6b30233..ca099495a1 100644 --- a/src/backend/nodes/list.c +++ b/src/backend/nodes/list.c @@ -702,6 +702,37 @@ list_member_oid(const List *list, Oid datum) return false; } +/* + * return true iff every entry in "members" list is also present + * in the "target" list. + */ +bool +list_is_subset(const List *members, const List *target) +{ + const ListCell *lc1, *lc2; + + Assert(IsPointerList(members)); + Assert(IsPointerList(target)); + check_list_invariants(members); + check_list_invariants(target); + + foreach(lc1, members) + { + bool found = false; + foreach(lc2, target) + { + if (equal(lfirst(lc1), lfirst(lc2))) + { + found = true; + break; + } + } + if (!found) + return false; + } + return true; +} + /* * Delete the n'th cell (counting from 0) in list. * diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 01c110cd2f..40415d0f5b 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -815,3 +815,16 @@ makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols) v->va_cols = va_cols; return v; } + + +/* + * makeUniqueKey + */ +UniqueKey* +makeUniqueKey(List *exprs, bool multi_nullvals) +{ + UniqueKey * ukey = makeNode(UniqueKey); + ukey->exprs = exprs; + ukey->multi_nullvals = multi_nullvals; + return ukey; +} diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..44154cde6a 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2456,6 +2456,14 @@ _outPathKey(StringInfo str, const PathKey *node) WRITE_BOOL_FIELD(pk_nulls_first); } +static void +_outUniqueKey(StringInfo str, const UniqueKey *node) +{ + WRITE_NODE_TYPE("UNIQUEKEY"); + WRITE_NODE_FIELD(exprs); + WRITE_BOOL_FIELD(multi_nullvals); +} + static void _outPathTarget(StringInfo str, const PathTarget *node) { @@ -4198,6 +4206,9 @@ outNode(StringInfo str, const void *obj) case T_PathKey: _outPathKey(str, obj); break; + case T_UniqueKey: + _outUniqueKey(str, obj); + break; case T_PathTarget: _outPathTarget(str, obj); break; diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index c5e136e9c3..b3e212bf1c 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -492,6 +492,14 @@ _readSetOperationStmt(void) READ_DONE(); } +static UniqueKey * +_readUniqueKey(void) +{ + READ_LOCALS(UniqueKey); + READ_NODE_FIELD(exprs); + READ_BOOL_FIELD(multi_nullvals); + READ_DONE(); +} /* * Stuff from primnodes.h. @@ -2717,6 +2725,8 @@ parseNodeString(void) return_value = _readCommonTableExpr(); else if (MATCH("SETOPERATIONSTMT", 16)) return_value = _readSetOperationStmt(); + else if (MATCH("UNIQUEKEY", 9)) + return_value = _readUniqueKey(); else if (MATCH("ALIAS", 5)) return_value = _readAlias(); else if (MATCH("RANGEVAR", 8)) diff --git a/src/backend/optimizer/path/Makefile b/src/backend/optimizer/path/Makefile index 1e199ff66f..7b9820c25f 100644 --- a/src/backend/optimizer/path/Makefile +++ b/src/backend/optimizer/path/Makefile @@ -21,6 +21,7 @@ OBJS = \ joinpath.o \ joinrels.o \ pathkeys.o \ - tidpath.o + tidpath.o \ + uniquekeys.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/optimizer/path/README.uniquekey b/src/backend/optimizer/path/README.uniquekey new file mode 100644 index 0000000000..5eac761995 --- /dev/null +++ b/src/backend/optimizer/path/README.uniquekey @@ -0,0 +1,131 @@ +1. What is UniqueKey? +We can think UniqueKey is a set of exprs for a RelOptInfo, which we are insure +that doesn't yields same result among all the rows. The simplest UniqueKey +format is primary key. + +However we define the UnqiueKey as below. + +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + +exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL +is a special case of UniqueKey, which means there is only one row in that +relation.it has a stronger semantic than others. like SELECT uk FROM t; uk is +normal unique key and may have different values. SELECT colx FROM t WHERE uk = +const. colx is unique AND we have only 1 value. This field can used for +innerrel_is_unique. this logic is handled specially in add_uniquekey_for_onerow +function. + +multi_nullvals: true means multi null values may exist in these exprs, so the +uniqueness is not guaranteed in this case. This field is necessary for +remove_useless_join & reduce_unique_semijoins where we don't mind these +duplicated NULL values. It is set to true for 2 cases. One is a unique key +from a unique index but the related column is nullable. The other one is for +outer join. see populate_joinrel_uniquekeys for detail. + + +The UniqueKey can be used at the following cases at least: +1. remove_useless_joins. +2. reduce_semianti_joins +3. remove distinct node if distinct clause is unique. +4. remove aggnode if group by clause is unique. +5. Index Skip Scan (WIP) +6. Aggregation Push Down without 2 phase aggregation if the join can't + duplicated the aggregated rows. (work in progress feature) + +2. How is it maintained? + +We have a set of populate_xxx_unqiuekeys functions to maintain the uniquekey on +various cases. xxx includes baserel, joinrel, partitionedrel, distinctrel, +groupedrel, unionrel. and we also need to convert the uniquekey from subquery +to outer relation, which is what convert_subquery_uniquekeys does. + +1. The first part is about baserel. We handled 3 cases. suppose we have Unique +Index on (a, b). + +1. SELECT a, b FROM t. UniqueKey (a, b) +2. SELECT a FROM t WHERE b = 1; UniqueKey (a) +3. SELECT .. FROM t WHERE a = 1 AND b = 1; UniqueKey (NIL). onerow case, every + column is Unique. + +2. The next part is joinrel, this part is most error-prone, we simplified the rules +like below: +1. If the relation's UniqueKey can't be duplicated after join, then is will be + still valid for the join rel. The function we used here is + innerrel_keeps_unique. The basic idea is innerrel.any_col = outer.uk. + +2. If the UnqiueKey can't keep valid via the rule 1, the combination of the + UniqueKey from both sides are valid for sure. We can prove this as: if the + unique exprs from rel1 is duplicated by rel2, the duplicated rows must + contains different unique exprs from rel2. + +More considerations about onerow: +1. If relation with one row and it can't be duplicated, it is still possible + contains mulit_nullvas after outer join. +2. If the either UniqueKey can be duplicated after join, the can get one row + only when both side is one row AND there is no outer join. +3. Whenever the onerow UniqueKey is not a valid any more, we need to convert one + row UniqueKey to normal unique key since we don't store exprs for one-row + relation. get_exprs_from_uniquekeys will be used here. + + +More considerations about multi_nullvals after join: +1. If the original UnqiueKey has multi_nullvals, the final UniqueKey will have + mulit_nullvals in any case. +2. If a unique key doesn't allow mulit_nullvals, after some outer join, it + allows some outer join. + + +3. When we comes to subquery, we need to convert_subquery_unqiuekeys just like +convert_subquery_pathkeys. Only the UniqueKey insides subquery is referenced as +a Var in outer relation will be reused. The relationship between the outerrel.Var +and subquery.exprs is built with outerel->subroot->processed_tlist. + + +4. As for the SRF functions, it will break the uniqueness of uniquekey, However it +is handled in adjust_paths_for_srfs, which happens after the query_planner. so +we will maintain the UniqueKey until there and reset it to NIL at that +places. This can't help on distinct/group by elimination cases but probably help +in some other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is +semantic correctly. + + +5. As for inherit table, we first main the UnqiueKey on childrel as well. But for +partitioned table we need to maintain 2 different kinds of +UnqiueKey. 1). UniqueKey on the parent relation 2). UniqueKey on child +relation for partition wise query. + +Example: +CREATE TABLE p (a int not null, b int not null) partition by list (a); +CREATE TABLE p0 partition of p for values in (1); +CREATE TABLE p1 partition of p for values in (2); + +create unique index p0_b on p0(b); +create unique index p1_b on p1(b); + +Now b is only unique on partition level, so the distinct can't be removed on +the following cases. SELECT DISTINCT b FROM p; + +Another example is SELECT DISTINCT a, b FROM p WHERE a = 1; Since only one +partition is chosen, the UniqueKey on child relation is same as the UniqueKey on +parent relation. + +Another usage of UniqueKey on partition level is it be helpful for +partition-wise join. + +As for the UniqueKey on parent table level, it comes with 2 different ways, +1). the UniqueKey is also derived in UniqueKey index, but the index must be same +in all the related children relations and the unique index must contains +Partition Key in it. Example: + +CREATE UNIQUE INDEX p_ab ON p(a, b); -- where a is the partition key. + +-- Query +SELECT a, b FROM p; the (a, b) is a UniqueKey of p. + +2). If the parent relation has only one childrel, the UniqueKey on childrel is + the UniqueKey on parent as well. diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index 37b4223adb..d52ad59f89 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -579,6 +579,12 @@ set_plain_rel_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) */ check_index_predicates(root, rel); + /* + * Now that we've marked which partial indexes are suitable, we can now + * build the relation's unique keys. + */ + populate_baserel_uniquekeys(root, rel, rel->indexlist); + /* Mark rel with estimated output rows, width, etc */ set_baserel_size_estimates(root, rel); } @@ -1297,6 +1303,8 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, /* Add paths to the append relation. */ add_paths_to_append_rel(root, rel, live_childrels); + if (IS_PARTITIONED_REL(rel)) + populate_partitionedrel_uniquekeys(root, rel, live_childrels); } @@ -2304,6 +2312,8 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, pathkeys, required_outer)); } + convert_subquery_uniquekeys(root, rel, sub_final_rel); + /* If outer rel allows parallelism, do same for partial paths. */ if (rel->consider_parallel && bms_is_empty(required_outer)) { diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c index 57ce97fd53..697ac047fb 100644 --- a/src/backend/optimizer/path/joinpath.c +++ b/src/backend/optimizer/path/joinpath.c @@ -71,13 +71,6 @@ static void consider_parallel_mergejoin(PlannerInfo *root, static void hash_inner_and_outer(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, RelOptInfo *innerrel, JoinType jointype, JoinPathExtraData *extra); -static List *select_mergejoin_clauses(PlannerInfo *root, - RelOptInfo *joinrel, - RelOptInfo *outerrel, - RelOptInfo *innerrel, - List *restrictlist, - JoinType jointype, - bool *mergejoin_allowed); static void generate_mergejoin_paths(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *innerrel, @@ -1927,7 +1920,7 @@ hash_inner_and_outer(PlannerInfo *root, * if it is mergejoinable and involves vars from the two sub-relations * currently of interest. */ -static List * +List * select_mergejoin_clauses(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c index 0dbe2ac726..7271f044ec 100644 --- a/src/backend/optimizer/path/joinrels.c +++ b/src/backend/optimizer/path/joinrels.c @@ -924,6 +924,8 @@ populate_joinrel_with_paths(PlannerInfo *root, RelOptInfo *rel1, /* Apply partitionwise join technique, if possible. */ try_partitionwise_join(root, rel1, rel2, joinrel, sjinfo, restrictlist); + + populate_joinrel_uniquekeys(root, joinrel, rel1, rel2, restrictlist, sjinfo->jointype); } diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index bd9a176d7d..139278829b 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -33,7 +33,6 @@ static bool pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys); static bool matches_boolean_partition_clause(RestrictInfo *rinfo, RelOptInfo *partrel, int partkeycol); -static Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); static bool right_merge_direction(PlannerInfo *root, PathKey *pathkey); @@ -1035,7 +1034,7 @@ convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel, * We need this to ensure that we don't return pathkeys describing values * that are unavailable above the level of the subquery scan. */ -static Var * +Var * find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle) { ListCell *lc; diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c new file mode 100644 index 0000000000..77ed2b2eff --- /dev/null +++ b/src/backend/optimizer/path/uniquekeys.c @@ -0,0 +1,1135 @@ +/*------------------------------------------------------------------------- + * + * uniquekeys.c + * Utilities for matching and building unique keys + * + * Portions Copyright (c) 2020, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/optimizer/path/uniquekeys.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "nodes/makefuncs.h" +#include "nodes/nodeFuncs.h" +#include "optimizer/pathnode.h" +#include "optimizer/paths.h" +#include "optimizer/appendinfo.h" +#include "optimizer/optimizer.h" +#include "optimizer/tlist.h" +#include "rewrite/rewriteManip.h" + + +/* + * This struct is used to help populate_joinrel_uniquekeys. + * + * added_to_joinrel is true if a uniquekey (from outerrel or innerrel) + * has been added to joinrel. + * useful is true if the exprs of the uniquekey still appears in joinrel. + */ +typedef struct UniqueKeyContextData +{ + UniqueKey *uniquekey; + bool added_to_joinrel; + bool useful; +} *UniqueKeyContext; + +static List *initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel); +static bool innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + bool reverse); + +static List *get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals); +static List *get_exprs_from_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + UniqueKey *ukey); +static void add_uniquekey_for_onerow(RelOptInfo *rel); +static bool add_combined_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype); + +/* Used for unique indexes checking for partitioned table */ +static bool index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind); +static IndexOptInfo *simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from); +static bool simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2); +static void adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_index); + +/* Helper function for grouped relation and distinct relation. */ +static void add_uniquekey_from_sortgroups(PlannerInfo *root, + RelOptInfo *rel, + List *sortgroups); + +/* + * populate_baserel_uniquekeys + * Populate 'baserel' uniquekeys list by looking at the rel's unique index + * and baserestrictinfo + */ +void +populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List *indexlist) +{ + ListCell *lc; + List *matched_uniq_indexes = NIL; + + /* Attrs appears in rel->reltarget->exprs. */ + Bitmapset *used_attrs = NULL; + + List *const_exprs = NIL; + List *expr_opfamilies = NIL; + + Assert(baserel->rtekind == RTE_RELATION); + + foreach(lc, indexlist) + { + IndexOptInfo *ind = (IndexOptInfo *) lfirst(lc); + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + matched_uniq_indexes = lappend(matched_uniq_indexes, ind); + } + + if (matched_uniq_indexes == NIL) + return; + + /* Check which attrs is used in baserel->reltarget */ + pull_varattnos((Node *)baserel->reltarget->exprs, baserel->relid, &used_attrs); + + /* Check which attrno is used at a mergeable const filter */ + foreach(lc, baserel->baserestrictinfo) + { + RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); + + if (rinfo->mergeopfamilies == NIL) + continue; + + if (bms_is_empty(rinfo->left_relids)) + { + const_exprs = lappend(const_exprs, get_rightop(rinfo->clause)); + } + else if (bms_is_empty(rinfo->right_relids)) + { + const_exprs = lappend(const_exprs, get_leftop(rinfo->clause)); + } + else + continue; + + expr_opfamilies = lappend(expr_opfamilies, rinfo->mergeopfamilies); + } + + foreach(lc, matched_uniq_indexes) + { + bool multi_nullvals, useful; + List *exprs = get_exprs_from_uniqueindex(lfirst_node(IndexOptInfo, lc), + const_exprs, + expr_opfamilies, + used_attrs, + &useful, + &multi_nullvals); + if (useful) + { + if (exprs == NIL) + { + /* All the columns in Unique Index matched with a restrictinfo */ + add_uniquekey_for_onerow(baserel); + return; + } + baserel->uniquekeys = lappend(baserel->uniquekeys, + makeUniqueKey(exprs, multi_nullvals)); + } + } +} + + +/* + * populate_partitionedrel_uniquekeys + * The UniqueKey on partitionrel comes from 2 cases: + * 1). Only one partition is involved in this query, the unique key can be + * copied to parent rel from childrel. + * 2). There are some unique index which includes partition key and exists + * in all the related partitions. + * We never mind rule 2 if we hit rule 1. + */ + +void +populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels) +{ + ListCell *lc; + List *global_uniq_indexlist = NIL; + RelOptInfo *childrel; + bool is_first = true; + + Assert(IS_PARTITIONED_REL(rel)); + + if (childrels == NIL) + return; + + /* + * If there is only one partition used in this query, the UniqueKey in childrel is + * still valid in parent level, but we need convert the format from child expr to + * parent expr. + */ + if (list_length(childrels) == 1) + { + /* Check for Rule 1 */ + RelOptInfo *childrel = linitial_node(RelOptInfo, childrels); + ListCell *lc; + Assert(childrel->reloptkind == RELOPT_OTHER_MEMBER_REL); + if (relation_is_onerow(childrel)) + { + add_uniquekey_for_onerow(rel); + return; + } + + foreach(lc, childrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + AppendRelInfo *appinfo = find_appinfo_by_child(root, childrel->relid); + List *parent_exprs = NIL; + bool can_reuse = true; + ListCell *lc2; + foreach(lc2, ukey->exprs) + { + Var *var = (Var *)lfirst(lc2); + /* + * If the expr comes from a expression, it is hard to build the expression + * in parent so ignore that case for now. + */ + if(!IsA(var, Var)) + { + can_reuse = false; + break; + } + /* Convert it to parent var */ + parent_exprs = lappend(parent_exprs, find_parent_var(appinfo, var)); + } + if (can_reuse) + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(parent_exprs, + ukey->multi_nullvals)); + } + } + else + { + /* Check for rule 2 */ + childrel = linitial_node(RelOptInfo, childrels); + foreach(lc, childrel->indexlist) + { + IndexOptInfo *ind = lfirst(lc); + IndexOptInfo *modified_index; + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + + /* + * During simple_copy_indexinfo_to_parent, we need to convert var from + * child var to parent var, index on expression is too complex to handle. + * so ignore it for now. + */ + if (ind->indexprs != NIL) + continue; + + modified_index = simple_copy_indexinfo_to_parent(root, rel, ind); + /* + * If the unique index doesn't contain partkey, then it is unique + * on this partition only, so it is useless for us. + */ + if (!index_constains_partkey(rel, modified_index)) + continue; + + global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index); + } + + if (global_uniq_indexlist != NIL) + { + foreach(lc, childrels) + { + RelOptInfo *child = lfirst(lc); + if (is_first) + { + is_first = false; + continue; + } + adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist); + } + /* Now we have a list of unique index which are exactly same on all childrels, + * Set the UniqueKey just like it is non-partition table + */ + populate_baserel_uniquekeys(root, rel, global_uniq_indexlist); + } + } +} + + +/* + * populate_distinctrel_uniquekeys + */ +void +populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel) +{ + /* The unique key before the distinct is still valid. */ + distinctrel->uniquekeys = list_copy(inputrel->uniquekeys); + add_uniquekey_from_sortgroups(root, distinctrel, root->parse->distinctClause); +} + +/* + * populate_grouprel_uniquekeys + */ +void +populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel) + +{ + Query *parse = root->parse; + bool input_ukey_added = false; + ListCell *lc; + + if (relation_is_onerow(inputrel)) + { + add_uniquekey_for_onerow(grouprel); + return; + } + if (parse->groupingSets) + return; + + /* A Normal group by without grouping set. */ + if (parse->groupClause) + { + /* + * Current even the groupby clause is Unique already, but if query has aggref + * We have to create grouprel still. To keep the UnqiueKey short, we will check + * the UniqueKey of input_rel still valid, if so we reuse it. + */ + foreach(lc, inputrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (list_is_subset(ukey->exprs, grouprel->reltarget->exprs)) + { + grouprel->uniquekeys = lappend(grouprel->uniquekeys, + ukey); + input_ukey_added = true; + } + } + if (!input_ukey_added) + /* + * group by clause must be a super-set of grouprel->reltarget->exprs except the + * aggregation expr, so if such exprs is unique already, no bother to generate + * new uniquekey for group by exprs. + */ + add_uniquekey_from_sortgroups(root, + grouprel, + root->parse->groupClause); + } + else + /* It has aggregation but without a group by, so only one row returned */ + add_uniquekey_for_onerow(grouprel); +} + +/* + * simple_copy_uniquekeys + * Using a function for the one-line code makes us easy to check where we simply + * copied the uniquekey. + */ +void +simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel) +{ + newrel->uniquekeys = oldrel->uniquekeys; +} + +/* + * populate_unionrel_uniquekeys + */ +void +populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel) +{ + ListCell *lc; + List *exprs = NIL; + + Assert(unionrel->uniquekeys == NIL); + + foreach(lc, unionrel->reltarget->exprs) + { + exprs = lappend(exprs, lfirst(lc)); + } + + if (exprs == NIL) + /* SQL: select union select; is valid, we need to handle it here. */ + add_uniquekey_for_onerow(unionrel); + else + unionrel->uniquekeys = lappend(unionrel->uniquekeys, + makeUniqueKey(exprs,false)); + +} + +/* + * populate_joinrel_uniquekeys + * + * populate uniquekeys for joinrel. We will check each relation to see if its + * UniqueKey is still valid via innerrel_keeps_unique, if so, we add it to + * joinrel. The multi_nullvals field will be changed to true for some outer + * join cases and one-row UniqueKey needs to be converted to normal UniqueKey + * for the same case as well. + * For the uniquekey in either baserel which can't be unique after join, we still + * check to see if combination of UniqueKeys from both side is still useful for us. + * if yes, we add it to joinrel as well. + */ +void +populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *outerrel, RelOptInfo *innerrel, + List *restrictlist, JoinType jointype) +{ + ListCell *lc, *lc2; + List *clause_list = NIL; + List *outerrel_ukey_ctx; + List *innerrel_ukey_ctx; + bool inner_onerow, outer_onerow; + bool mergejoin_allowed; + + /* Care about the outerrel relation only for SEMI/ANTI join */ + if (jointype == JOIN_SEMI || jointype == JOIN_ANTI) + { + foreach(lc, outerrel->uniquekeys) + { + UniqueKey *uniquekey = lfirst_node(UniqueKey, lc); + if (list_is_subset(uniquekey->exprs, joinrel->reltarget->exprs)) + joinrel->uniquekeys = lappend(joinrel->uniquekeys, uniquekey); + } + return; + } + + Assert(jointype == JOIN_LEFT || jointype == JOIN_FULL || jointype == JOIN_INNER); + + /* Fast path */ + if (innerrel->uniquekeys == NIL || outerrel->uniquekeys == NIL) + return; + + inner_onerow = relation_is_onerow(innerrel); + outer_onerow = relation_is_onerow(outerrel); + + outerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(outerrel); + innerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(innerrel); + + clause_list = select_mergejoin_clauses(root, joinrel, outerrel, innerrel, + restrictlist, jointype, + &mergejoin_allowed); + + if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true /* reverse */)) + { + bool outer_impact = jointype == JOIN_FULL; + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + /* Outer relation has one row, and the unique key is not duplicated after join, + * the joinrel will still has one row unless the jointype == JOIN_FULL. + */ + if (outer_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (outer_onerow) + { + /* + * The onerow outerrel becomes multi rows and multi_nullvals + * will be changed to true. We also need to set the exprs correctly since it + * can't be NIL any more. + */ + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, outerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Change multi_nullvals to true due to the full join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, true)); + else + /* Just reuse it */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + } + ctx->added_to_joinrel = true; + } + } + + if (innerrel_keeps_unique(root, outerrel, innerrel, clause_list, false)) + { + bool outer_impact = jointype == JOIN_FULL || jointype == JOIN_LEFT;; + + foreach(lc, innerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + if (inner_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (inner_onerow) + { + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, innerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Need to change multi_nullvals to true due to the outer join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, + true)); + else + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + + } + ctx->added_to_joinrel = true; + } + } + + /* + * The combination of the UniqueKey from both sides is unique as well regardless + * of join type, but no bother to add it if its subset has been added to joinrel + * already or it is not useful for the joinrel. + */ + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx1 = (UniqueKeyContext) lfirst(lc); + if (ctx1->added_to_joinrel || !ctx1->useful) + continue; + foreach(lc2, innerrel_ukey_ctx) + { + UniqueKeyContext ctx2 = (UniqueKeyContext) lfirst(lc2); + if (ctx2->added_to_joinrel || !ctx2->useful) + continue; + if (add_combined_uniquekey(root, joinrel, outerrel, innerrel, + ctx1->uniquekey, ctx2->uniquekey, + jointype)) + /* If we set a onerow UniqueKey to joinrel, we don't need other. */ + return; + } + } +} + + +/* + * convert_subquery_uniquekeys + * + * Covert the UniqueKey in subquery to outer relation. + */ +void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel) +{ + ListCell *lc; + + if (sub_final_rel->uniquekeys == NIL) + return; + + if (relation_is_onerow(sub_final_rel)) + { + add_uniquekey_for_onerow(currel); + return; + } + + Assert(currel->subroot != NULL); + + foreach(lc, sub_final_rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + ListCell *lc; + List *exprs = NIL; + bool ukey_useful = true; + + /* One row case is handled above */ + Assert(ukey->exprs != NIL); + foreach(lc, ukey->exprs) + { + Var *var; + TargetEntry *tle = tlist_member(lfirst(lc), + currel->subroot->processed_tlist); + if (tle == NULL) + { + ukey_useful = false; + break; + } + var = find_var_for_subquery_tle(currel, tle); + if (var == NULL) + { + ukey_useful = false; + break; + } + exprs = lappend(exprs, var); + } + + if (ukey_useful) + currel->uniquekeys = lappend(currel->uniquekeys, + makeUniqueKey(exprs, + ukey->multi_nullvals)); + + } +} + +/* + * innerrel_keeps_unique + * + * Check if Unique key of the innerrel is valid after join. innerrel's UniqueKey + * will be still valid if innerrel's any-column mergeop outrerel's uniquekey + * exists in clause_list. + * + * Note: the clause_list must be a list of mergeable restrictinfo already. + */ +static bool +innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *clause_list, + bool reverse) +{ + ListCell *lc, *lc2, *lc3; + + if (outerrel->uniquekeys == NIL || innerrel->uniquekeys == NIL) + return false; + + /* Check if there is outerrel's uniquekey in mergeable clause. */ + foreach(lc, outerrel->uniquekeys) + { + List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs; + bool clauselist_matchs_all_exprs = true; + foreach(lc2, outer_uq_exprs) + { + Node *outer_uq_expr = lfirst(lc2); + bool find_uq_expr_in_clauselist = false; + foreach(lc3, clause_list) + { + RestrictInfo *rinfo = lfirst_node(RestrictInfo, lc3); + Node *outer_expr; + if (reverse) + outer_expr = rinfo->outer_is_left ? get_rightop(rinfo->clause) : get_leftop(rinfo->clause); + else + outer_expr = rinfo->outer_is_left ? get_leftop(rinfo->clause) : get_rightop(rinfo->clause); + if (equal(outer_expr, outer_uq_expr)) + { + find_uq_expr_in_clauselist = true; + break; + } + } + if (!find_uq_expr_in_clauselist) + { + /* No need to check the next exprs in the current uniquekey */ + clauselist_matchs_all_exprs = false; + break; + } + } + + if (clauselist_matchs_all_exprs) + return true; + } + return false; +} + + +/* + * relation_is_onerow + * Check if it is a one-row relation by checking UniqueKey. + */ +bool +relation_is_onerow(RelOptInfo *rel) +{ + UniqueKey *ukey; + if (rel->uniquekeys == NIL) + return false; + ukey = linitial_node(UniqueKey, rel->uniquekeys); + return ukey->exprs == NIL && list_length(rel->uniquekeys) == 1; +} + +/* + * relation_has_uniquekeys_for + * Returns true if we have proofs that 'rel' cannot return multiple rows with + * the same values in each of 'exprs'. Otherwise returns false. + */ +bool +relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel, + List *exprs, bool allow_multinulls) +{ + ListCell *lc; + + /* + * For UniqueKey->onerow case, the uniquekey->exprs is empty as well + * so we can't rely on list_is_subset to handle this special cases + */ + if (exprs == NIL) + return false; + + foreach(lc, rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (ukey->multi_nullvals && !allow_multinulls) + continue; + if (list_is_subset(ukey->exprs, exprs)) + return true; + } + return false; +} + + +/* + * get_exprs_from_uniqueindex + * + * Return a list of exprs which is unique. set useful to false if this + * unique index is not useful for us. + */ +static List * +get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals) +{ + List *exprs = NIL; + ListCell *indexpr_item; + int c = 0; + + *useful = true; + *multi_nullvals = false; + + indexpr_item = list_head(unique_index->indexprs); + for(c = 0; c < unique_index->ncolumns; c++) + { + int attr = unique_index->indexkeys[c]; + Expr *expr; + bool matched_const = false; + ListCell *lc1, *lc2; + + if(attr > 0) + { + expr = list_nth_node(TargetEntry, unique_index->indextlist, c)->expr; + } + else if (attr == 0) + { + /* Expression index */ + expr = lfirst(indexpr_item); + indexpr_item = lnext(unique_index->indexprs, indexpr_item); + } + else /* attr < 0 */ + { + /* Index on system column is not supported */ + Assert(false); + } + + /* + * Check index_col = Const case with regarding to opfamily checking + * If we can remove the index_col from the final UniqueKey->exprs. + */ + forboth(lc1, const_exprs, lc2, const_expr_opfamilies) + { + if (list_member_oid((List *)lfirst(lc2), unique_index->opfamily[c]) + && match_index_to_operand((Node *) lfirst(lc1), c, unique_index)) + { + matched_const = true; + break; + } + } + + if (matched_const) + continue; + + /* Check if the indexed expr is used in rel */ + if (attr > 0) + { + /* + * Normal Indexed column, if the col is not used, then the index is useless + * for uniquekey. + */ + attr -= FirstLowInvalidHeapAttributeNumber; + + if (!bms_is_member(attr, used_varattrs)) + { + *useful = false; + break; + } + } + else if (!list_member(unique_index->rel->reltarget->exprs, expr)) + { + /* Expression index but the expression is not used in rel */ + *useful = false; + break; + } + + /* check not null property. */ + if (attr == 0) + { + /* We never know if a expression yields null or not */ + *multi_nullvals = true; + } + else if (!bms_is_member(attr, unique_index->rel->notnullattrs) + && !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber, + unique_index->rel->notnullattrs)) + { + *multi_nullvals = true; + } + + exprs = lappend(exprs, expr); + } + return exprs; +} + + +/* + * add_uniquekey_for_onerow + * If we are sure that the relation only returns one row, then all the columns + * are unique. However we don't need to create UniqueKey for every column, we + * just set exprs = NIL and overwrites all the other UniqueKey on this RelOptInfo + * since this one has strongest semantics. + */ +void +add_uniquekey_for_onerow(RelOptInfo *rel) +{ + /* + * We overwrite the previous UniqueKey on purpose since this one has the + * strongest semantic. + */ + rel->uniquekeys = list_make1(makeUniqueKey(NIL, false)); +} + + +/* + * initililze_uniquecontext_for_joinrel + * Return a List of UniqueKeyContext for an inputrel + */ +static List * +initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel) +{ + List *res = NIL; + ListCell *lc; + foreach(lc, inputrel->uniquekeys) + { + UniqueKeyContext context; + context = palloc(sizeof(struct UniqueKeyContextData)); + context->uniquekey = lfirst_node(UniqueKey, lc); + context->added_to_joinrel = false; + context->useful = true; + res = lappend(res, context); + } + return res; +} + + +/* + * get_exprs_from_uniquekey + * Unify the way of get List of exprs from a one-row UniqueKey or + * normal UniqueKey. for the onerow case, every expr in rel1 is a valid + * UniqueKey. Return a List of exprs. + * + * rel1: The relation which you want to get the exprs. + * ukey: The UniqueKey you want to get the exprs. + */ +static List * +get_exprs_from_uniquekey(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *rel1, UniqueKey *ukey) +{ + ListCell *lc; + bool onerow = rel1 != NULL && relation_is_onerow(rel1); + + List *res = NIL; + Assert(onerow || ukey); + if (onerow) + { + /* Only cares about the exprs still exist in joinrel */ + foreach(lc, joinrel->reltarget->exprs) + { + Bitmapset *relids = pull_varnos(root, lfirst(lc)); + if (bms_is_subset(relids, rel1->relids)) + { + res = lappend(res, list_make1(lfirst(lc))); + } + } + } + else + { + res = list_make1(ukey->exprs); + } + return res; +} + +/* + * Partitioned table Unique Keys. + * The partition table unique key is maintained as: + * 1. The index must be unique as usual. + * 2. The index must contains partition key. + * 3. The index must exist on all the child rel. see simple_indexinfo_equal for + * how we compare it. + */ + +/* + * index_constains_partkey + * return true if the index contains the partiton key. + */ +static bool +index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind) +{ + ListCell *lc; + int i; + Assert(IS_PARTITIONED_REL(partrel)); + Assert(partrel->part_scheme->partnatts > 0); + + for(i = 0; i < partrel->part_scheme->partnatts; i++) + { + Node *part_expr = linitial(partrel->partexprs[i]); + bool found_in_index = false; + foreach(lc, ind->indextlist) + { + Expr *index_expr = lfirst_node(TargetEntry, lc)->expr; + if (equal(index_expr, part_expr)) + { + found_in_index = true; + break; + } + } + if (!found_in_index) + return false; + } + return true; +} + +/* + * simple_indexinfo_equal + * + * Used to check if the 2 index is same as each other. The index here + * is COPIED from childrel and did some tiny changes(see + * simple_copy_indexinfo_to_parent) + */ +static bool +simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2) +{ + Size oid_cmp_len = sizeof(Oid) * ind1->ncolumns; + + return ind1->ncolumns == ind2->ncolumns && + ind1->unique == ind2->unique && + memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0 && + memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0 && + memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0 && + memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0 && + equal(get_tlist_exprs(ind1->indextlist, true), + get_tlist_exprs(ind2->indextlist, true)); +} + + +/* + * The below macros are used for simple_copy_indexinfo_to_parent which is so + * customized that I don't want to put it to copyfuncs.c. So copy it here. + */ +#define COPY_POINTER_FIELD(fldname, sz) \ + do { \ + Size _size = (sz); \ + newnode->fldname = palloc(_size); \ + memcpy(newnode->fldname, from->fldname, _size); \ + } while (0) + +#define COPY_NODE_FIELD(fldname) \ + (newnode->fldname = copyObjectImpl(from->fldname)) + +#define COPY_SCALAR_FIELD(fldname) \ + (newnode->fldname = from->fldname) + + +/* + * simple_copy_indexinfo_to_parent (from partition) + * Copy the IndexInfo from child relation to parent relation with some modification, + * which is used to test: + * 1. If the same index exists in all the childrels. + * 2. If the parentrel->reltarget/basicrestrict info matches this index. + */ +static IndexOptInfo * +simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from) +{ + IndexOptInfo *newnode = makeNode(IndexOptInfo); + AppendRelInfo *appinfo = find_appinfo_by_child(root, from->rel->relid); + ListCell *lc; + int idx = 0; + + COPY_SCALAR_FIELD(ncolumns); + COPY_SCALAR_FIELD(nkeycolumns); + COPY_SCALAR_FIELD(unique); + COPY_SCALAR_FIELD(immediate); + /* We just need to know if it is NIL or not */ + COPY_SCALAR_FIELD(indpred); + COPY_SCALAR_FIELD(predOK); + COPY_POINTER_FIELD(indexkeys, from->ncolumns * sizeof(int)); + COPY_POINTER_FIELD(indexcollations, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opfamily, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opcintype, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(sortopfamily, from->ncolumns * sizeof(Oid)); + COPY_NODE_FIELD(indextlist); + + /* Convert index exprs on child expr to expr on parent */ + foreach(lc, newnode->indextlist) + { + TargetEntry *tle = lfirst_node(TargetEntry, lc); + /* Index on expression is ignored */ + Assert(IsA(tle->expr, Var)); + tle->expr = (Expr *) find_parent_var(appinfo, (Var *) tle->expr); + newnode->indexkeys[idx] = castNode(Var, tle->expr)->varattno; + idx++; + } + newnode->rel = parentrel; + return newnode; +} + +/* + * adjust_partition_unique_indexlist + * + * global_unique_indexes: At the beginning, it contains the copy & modified + * unique index from the first partition. And then check if each index in it still + * exists in the following partitions. If no, remove it. at last, it has an + * index list which exists in all the partitions. + */ +static void +adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_indexes) +{ + ListCell *lc, *lc2; + foreach(lc, *global_unique_indexes) + { + IndexOptInfo *g_ind = lfirst_node(IndexOptInfo, lc); + bool found_in_child = false; + + foreach(lc2, childrel->indexlist) + { + IndexOptInfo *p_ind = lfirst_node(IndexOptInfo, lc2); + IndexOptInfo *p_ind_copy; + if (!p_ind->unique || !p_ind->immediate || + (p_ind->indpred != NIL && !p_ind->predOK)) + continue; + p_ind_copy = simple_copy_indexinfo_to_parent(root, parentrel, p_ind); + if (simple_indexinfo_equal(p_ind_copy, g_ind)) + { + found_in_child = true; + break; + } + } + if (!found_in_child) + /* The index doesn't exist in childrel, remove it from global_unique_indexes */ + *global_unique_indexes = foreach_delete_current(*global_unique_indexes, lc); + } +} + +/* Helper function for groupres/distinctrel */ +static void +add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgroups) +{ + Query *parse = root->parse; + List *exprs; + + /* + * XXX: If there are some vars which is not in current levelsup, the semantic is + * imprecise, should we avoid it or not? levelsup = 1 is just a demo, maybe we need to + * check every level other than 0, if so, looks we have to write another + * pull_var_walker. + */ + List *upper_vars = pull_vars_of_level((Node*)sortgroups, 1); + + if (upper_vars != NIL) + return; + + exprs = get_sortgrouplist_exprs(sortgroups, parse->targetList); + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(exprs, + false /* sortgroupclause can't be multi_nullvals */)); +} + + +/* + * add_combined_uniquekey + * The combination of both UniqueKeys is a valid UniqueKey for joinrel no matter + * the jointype. + */ +bool +add_combined_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype) +{ + + ListCell *lc1, *lc2; + + /* Either side has multi_nullvals or we have outer join, + * the combined UniqueKey has multi_nullvals */ + bool multi_nullvals = outer_ukey->multi_nullvals || + inner_ukey->multi_nullvals || IS_OUTER_JOIN(jointype); + + /* The only case we can get onerow joinrel after join */ + if (relation_is_onerow(outer_rel) + && relation_is_onerow(inner_rel) + && jointype == JOIN_INNER) + { + add_uniquekey_for_onerow(joinrel); + return true; + } + + foreach(lc1, get_exprs_from_uniquekey(root, joinrel, outer_rel, outer_ukey)) + { + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, inner_rel, inner_ukey)) + { + List *exprs = list_concat_copy(lfirst_node(List, lc1), lfirst_node(List, lc2)); + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(exprs, + multi_nullvals)); + } + } + return false; +} diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index 424d25cbd5..8d8e493f5c 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -2387,6 +2387,8 @@ grouping_planner(PlannerInfo *root, bool inheritance_update, add_path(final_rel, path); } + simple_copy_uniquekeys(current_rel, final_rel); + /* * Generate partial paths for final_rel, too, if outer query levels might * be able to make use of them. @@ -3899,6 +3901,8 @@ create_grouping_paths(PlannerInfo *root, } set_cheapest(grouped_rel); + + populate_grouprel_uniquekeys(root, grouped_rel, input_rel); return grouped_rel; } @@ -4621,7 +4625,7 @@ create_window_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(window_rel); - + simple_copy_uniquekeys(input_rel, window_rel); return window_rel; } @@ -4935,7 +4939,7 @@ create_distinct_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(distinct_rel); - + populate_distinctrel_uniquekeys(root, input_rel, distinct_rel); return distinct_rel; } @@ -5196,6 +5200,8 @@ create_ordered_paths(PlannerInfo *root, */ Assert(ordered_rel->pathlist != NIL); + simple_copy_uniquekeys(input_rel, ordered_rel); + return ordered_rel; } @@ -6073,6 +6079,9 @@ adjust_paths_for_srfs(PlannerInfo *root, RelOptInfo *rel, if (list_length(targets) == 1) return; + /* UniqueKey is not valid after handling the SRF. */ + rel->uniquekeys = NIL; + /* * Stack SRF-evaluation nodes atop each path for the rel. * diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c index becdcbb872..b7626545bf 100644 --- a/src/backend/optimizer/prep/prepunion.c +++ b/src/backend/optimizer/prep/prepunion.c @@ -689,6 +689,8 @@ generate_union_paths(SetOperationStmt *op, PlannerInfo *root, /* Undo effects of possibly forcing tuple_fraction to 0 */ root->tuple_fraction = save_fraction; + /* Add the UniqueKeys */ + populate_unionrel_uniquekeys(root, result_rel); return result_rel; } diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c index 86922a273c..6817c9c787 100644 --- a/src/backend/optimizer/util/appendinfo.c +++ b/src/backend/optimizer/util/appendinfo.c @@ -746,3 +746,47 @@ find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos) } return appinfos; } + +/* + * find_appinfo_by_child + * + */ +AppendRelInfo * +find_appinfo_by_child(PlannerInfo *root, Index child_index) +{ + ListCell *lc; + foreach(lc, root->append_rel_list) + { + AppendRelInfo *appinfo = lfirst_node(AppendRelInfo, lc); + if (appinfo->child_relid == child_index) + return appinfo; + } + elog(ERROR, "parent relation cant be found"); + return NULL; +} + +/* + * find_parent_var + * + */ +Var * +find_parent_var(AppendRelInfo *appinfo, Var *child_var) +{ + ListCell *lc; + Var *res = NULL; + Index attno = 1; + foreach(lc, appinfo->translated_vars) + { + Node *child_node = lfirst(lc); + if (equal(child_node, child_var)) + { + res = copyObject(child_var); + res->varattno = attno; + res->varno = appinfo->parent_relid; + } + attno++; + } + if (res == NULL) + elog(ERROR, "parent var can't be found."); + return res; +} diff --git a/src/backend/optimizer/util/inherit.c b/src/backend/optimizer/util/inherit.c index be1c9ddd96..3eec1f4d74 100644 --- a/src/backend/optimizer/util/inherit.c +++ b/src/backend/optimizer/util/inherit.c @@ -736,6 +736,7 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, { Node *onecq = (Node *) lfirst(lc2); bool pseudoconstant; + RestrictInfo *child_rinfo; /* check for pseudoconstant (no Vars or volatile functions) */ pseudoconstant = @@ -747,14 +748,15 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, root->hasPseudoConstantQuals = true; } /* reconstitute RestrictInfo with appropriate properties */ - childquals = lappend(childquals, - make_restrictinfo(root, - (Expr *) onecq, - rinfo->is_pushed_down, - rinfo->outerjoin_delayed, - pseudoconstant, - rinfo->security_level, - NULL, NULL, NULL)); + child_rinfo = make_restrictinfo(root, + (Expr *) onecq, + rinfo->is_pushed_down, + rinfo->outerjoin_delayed, + pseudoconstant, + rinfo->security_level, + NULL, NULL, NULL); + child_rinfo->mergeopfamilies = rinfo->mergeopfamilies; + childquals = lappend(childquals, child_rinfo); /* track minimum security level among child quals */ cq_min_security = Min(cq_min_security, rinfo->security_level); } diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 48a7ebfe45..4fe1824eb0 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -16,6 +16,7 @@ #include "nodes/execnodes.h" #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" extern A_Expr *makeA_Expr(A_Expr_Kind kind, List *name, @@ -106,4 +107,6 @@ extern GroupingSet *makeGroupingSet(GroupingSetKind kind, List *content, int loc extern VacuumRelation *makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols); +extern UniqueKey* makeUniqueKey(List *exprs, bool multi_nullvals); + #endif /* MAKEFUNC_H */ diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index e22df890ef..3320273ac1 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -265,6 +265,7 @@ typedef enum NodeTag T_EquivalenceMember, T_PathKey, T_PathTarget, + T_UniqueKey, T_RestrictInfo, T_IndexClause, T_PlaceHolderVar, diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index 0d61f04d27..80561b1003 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -739,6 +739,7 @@ typedef struct RelOptInfo QualCost baserestrictcost; /* cost of evaluating the above */ Index baserestrict_min_security; /* min security_level found in * baserestrictinfo */ + List *uniquekeys; /* List of UniqueKey */ List *joininfo; /* RestrictInfo structures for join clauses * involving this rel */ bool has_eclass_joins; /* T means joininfo is incomplete */ @@ -1059,6 +1060,28 @@ typedef struct PathKey } PathKey; +/* + * UniqueKey + * + * Represents the unique properties held by a RelOptInfo. + * + * exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL + * is a special case of UniqueKey, which means there is only 1 row in that + * relation. + * multi_nullvals: true means multi null values may exist in these exprs, so the + * uniqueness is not guaranteed in this case. This field is necessary for + * remove_useless_join & reduce_unique_semijoins where we don't mind these + * duplicated NULL values. It is set to true for 2 cases. One is a unique key + * from a unique index but the related column is nullable. The other one is for + * outer join. see populate_joinrel_uniquekeys for detail. + */ +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + /* * PathTarget * @@ -2496,7 +2519,7 @@ typedef enum * * flags indicating what kinds of grouping are possible. * partial_costs_set is true if the agg_partial_costs and agg_final_costs - * have been initialized. + * have been initialized. * agg_partial_costs gives partial aggregation costs. * agg_final_costs gives finalization costs. * target_parallel_safe is true if target is parallel safe. @@ -2526,8 +2549,8 @@ typedef struct * limit_tuples is an estimated bound on the number of output tuples, * or -1 if no LIMIT or couldn't estimate. * count_est and offset_est are the estimated values of the LIMIT and OFFSET - * expressions computed by preprocess_limit() (see comments for - * preprocess_limit() for more information). + * expressions computed by preprocess_limit() (see comments for + * preprocess_limit() for more information). */ typedef struct { diff --git a/src/include/nodes/pg_list.h b/src/include/nodes/pg_list.h index 404e03f132..85c65e910c 100644 --- a/src/include/nodes/pg_list.h +++ b/src/include/nodes/pg_list.h @@ -566,6 +566,16 @@ extern pg_nodiscard List *list_delete_first(List *list); extern pg_nodiscard List *list_delete_last(List *list); extern pg_nodiscard List *list_delete_nth_cell(List *list, int n); extern pg_nodiscard List *list_delete_cell(List *list, ListCell *cell); +extern bool list_is_subset(const List *members, const List *target); + +extern List *list_delete(List *list, void *datum); +extern List *list_delete_ptr(List *list, void *datum); +extern List *list_delete_int(List *list, int datum); +extern List *list_delete_oid(List *list, Oid datum); +extern List *list_delete_first(List *list); +extern List *list_delete_last(List *list); +extern List *list_delete_nth_cell(List *list, int n); +extern List *list_delete_cell(List *list, ListCell *cell); extern List *list_union(const List *list1, const List *list2); extern List *list_union_ptr(const List *list1, const List *list2); diff --git a/src/include/optimizer/appendinfo.h b/src/include/optimizer/appendinfo.h index 4cbf8c26cc..43b8d05f69 100644 --- a/src/include/optimizer/appendinfo.h +++ b/src/include/optimizer/appendinfo.h @@ -32,4 +32,7 @@ extern Relids adjust_child_relids_multilevel(PlannerInfo *root, Relids relids, extern AppendRelInfo **find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos); +extern AppendRelInfo *find_appinfo_by_child(PlannerInfo *root, Index child_index); +extern Var *find_parent_var(AppendRelInfo *appinfo, Var *child_var); + #endif /* APPENDINFO_H */ diff --git a/src/include/optimizer/optimizer.h b/src/include/optimizer/optimizer.h index d587952b7d..843aafa51e 100644 --- a/src/include/optimizer/optimizer.h +++ b/src/include/optimizer/optimizer.h @@ -23,6 +23,7 @@ #define OPTIMIZER_H #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" /* Test if an expression node represents a SRF call. Beware multiple eval! */ #define IS_SRF_CALL(node) \ @@ -168,6 +169,7 @@ extern TargetEntry *get_sortgroupref_tle(Index sortref, List *targetList); extern TargetEntry *get_sortgroupclause_tle(SortGroupClause *sgClause, List *targetList); +extern Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); extern Node *get_sortgroupclause_expr(SortGroupClause *sgClause, List *targetList); extern List *get_sortgrouplist_exprs(List *sgClauses, diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h index 035d3e1206..1adf99c9ee 100644 --- a/src/include/optimizer/paths.h +++ b/src/include/optimizer/paths.h @@ -247,5 +247,48 @@ extern PathKey *make_canonical_pathkey(PlannerInfo *root, int strategy, bool nulls_first); extern void add_paths_to_append_rel(PlannerInfo *root, RelOptInfo *rel, List *live_childrels); +extern List *select_mergejoin_clauses(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + JoinType jointype, + bool *mergejoin_allowed); + +/* + * uniquekeys.c + * Utilities for matching and building unique keys + */ +extern void populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List* unique_index_list); +extern void populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels); +extern void populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel); +extern void populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel); +extern void populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel); +extern void simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel); +extern void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel); +extern void populate_joinrel_uniquekeys(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + RelOptInfo *rel2, + List *restrictlist, + JoinType jointype); + +extern bool relation_has_uniquekeys_for(PlannerInfo *root, + RelOptInfo *rel, + List *exprs, + bool allow_multinulls); +extern bool relation_is_onerow(RelOptInfo *rel); #endif /* PATHS_H */ -- 2.26.2 --yh7cw345mbttadjz Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v38-0003-Extend-UniqueKeys.patch" ^ permalink raw reply [nested|flat] 91+ messages in thread
* [PATCH v36 2/6] Introduce UniqueKey attributes on RelOptInfo struct. @ 2020-05-11 07:50 一挃 <[email protected]> 0 siblings, 0 replies; 91+ messages in thread From: 一挃 @ 2020-05-11 07:50 UTC (permalink / raw) UniqueKey is a set of exprs on RelOptInfo which represents the exprs will be unique on the given RelOptInfo. You can see README.uniquekey for more information. --- src/backend/nodes/copyfuncs.c | 13 + src/backend/nodes/list.c | 31 + src/backend/nodes/makefuncs.c | 13 + src/backend/nodes/outfuncs.c | 11 + src/backend/nodes/readfuncs.c | 10 + src/backend/optimizer/path/Makefile | 3 +- src/backend/optimizer/path/README.uniquekey | 131 +++ src/backend/optimizer/path/allpaths.c | 10 + src/backend/optimizer/path/joinpath.c | 9 +- src/backend/optimizer/path/joinrels.c | 2 + src/backend/optimizer/path/pathkeys.c | 3 +- src/backend/optimizer/path/uniquekeys.c | 1131 +++++++++++++++++++ src/backend/optimizer/plan/planner.c | 13 +- src/backend/optimizer/prep/prepunion.c | 2 + src/backend/optimizer/util/appendinfo.c | 44 + src/backend/optimizer/util/inherit.c | 16 +- src/include/nodes/makefuncs.h | 3 + src/include/nodes/nodes.h | 1 + src/include/nodes/pathnodes.h | 29 +- src/include/nodes/pg_list.h | 2 + src/include/optimizer/appendinfo.h | 3 + src/include/optimizer/optimizer.h | 2 + src/include/optimizer/paths.h | 43 + 23 files changed, 1502 insertions(+), 23 deletions(-) create mode 100644 src/backend/optimizer/path/README.uniquekey create mode 100644 src/backend/optimizer/path/uniquekeys.c diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 89c409de66..1f50400fd2 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2273,6 +2273,16 @@ _copyPathKey(const PathKey *from) return newnode; } +static UniqueKey * +_copyUniqueKey(const UniqueKey *from) +{ + UniqueKey *newnode = makeNode(UniqueKey); + + COPY_NODE_FIELD(exprs); + COPY_SCALAR_FIELD(multi_nullvals); + + return newnode; +} /* * _copyRestrictInfo */ @@ -5152,6 +5162,9 @@ copyObjectImpl(const void *from) case T_PathKey: retval = _copyPathKey(from); break; + case T_UniqueKey: + retval = _copyUniqueKey(from); + break; case T_RestrictInfo: retval = _copyRestrictInfo(from); break; diff --git a/src/backend/nodes/list.c b/src/backend/nodes/list.c index 80fa8c84e4..a7a99b70f2 100644 --- a/src/backend/nodes/list.c +++ b/src/backend/nodes/list.c @@ -687,6 +687,37 @@ list_member_oid(const List *list, Oid datum) return false; } +/* + * return true iff every entry in "members" list is also present + * in the "target" list. + */ +bool +list_is_subset(const List *members, const List *target) +{ + const ListCell *lc1, *lc2; + + Assert(IsPointerList(members)); + Assert(IsPointerList(target)); + check_list_invariants(members); + check_list_invariants(target); + + foreach(lc1, members) + { + bool found = false; + foreach(lc2, target) + { + if (equal(lfirst(lc1), lfirst(lc2))) + { + found = true; + break; + } + } + if (!found) + return false; + } + return true; +} + /* * Delete the n'th cell (counting from 0) in list. * diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 49de285f01..646cf7c9a1 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -814,3 +814,16 @@ makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols) v->va_cols = va_cols; return v; } + + +/* + * makeUniqueKey + */ +UniqueKey* +makeUniqueKey(List *exprs, bool multi_nullvals) +{ + UniqueKey * ukey = makeNode(UniqueKey); + ukey->exprs = exprs; + ukey->multi_nullvals = multi_nullvals; + return ukey; +} diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index e2f177515d..c3a9632992 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2428,6 +2428,14 @@ _outPathKey(StringInfo str, const PathKey *node) WRITE_BOOL_FIELD(pk_nulls_first); } +static void +_outUniqueKey(StringInfo str, const UniqueKey *node) +{ + WRITE_NODE_TYPE("UNIQUEKEY"); + WRITE_NODE_FIELD(exprs); + WRITE_BOOL_FIELD(multi_nullvals); +} + static void _outPathTarget(StringInfo str, const PathTarget *node) { @@ -4127,6 +4135,9 @@ outNode(StringInfo str, const void *obj) case T_PathKey: _outPathKey(str, obj); break; + case T_UniqueKey: + _outUniqueKey(str, obj); + break; case T_PathTarget: _outPathTarget(str, obj); break; diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index 42050ab719..3a18571d0c 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -452,6 +452,14 @@ _readSetOperationStmt(void) READ_DONE(); } +static UniqueKey * +_readUniqueKey(void) +{ + READ_LOCALS(UniqueKey); + READ_NODE_FIELD(exprs); + READ_BOOL_FIELD(multi_nullvals); + READ_DONE(); +} /* * Stuff from primnodes.h. @@ -2656,6 +2664,8 @@ parseNodeString(void) return_value = _readCommonTableExpr(); else if (MATCH("SETOPERATIONSTMT", 16)) return_value = _readSetOperationStmt(); + else if (MATCH("UNIQUEKEY", 9)) + return_value = _readUniqueKey(); else if (MATCH("ALIAS", 5)) return_value = _readAlias(); else if (MATCH("RANGEVAR", 8)) diff --git a/src/backend/optimizer/path/Makefile b/src/backend/optimizer/path/Makefile index 1e199ff66f..7b9820c25f 100644 --- a/src/backend/optimizer/path/Makefile +++ b/src/backend/optimizer/path/Makefile @@ -21,6 +21,7 @@ OBJS = \ joinpath.o \ joinrels.o \ pathkeys.o \ - tidpath.o + tidpath.o \ + uniquekeys.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/optimizer/path/README.uniquekey b/src/backend/optimizer/path/README.uniquekey new file mode 100644 index 0000000000..5eac761995 --- /dev/null +++ b/src/backend/optimizer/path/README.uniquekey @@ -0,0 +1,131 @@ +1. What is UniqueKey? +We can think UniqueKey is a set of exprs for a RelOptInfo, which we are insure +that doesn't yields same result among all the rows. The simplest UniqueKey +format is primary key. + +However we define the UnqiueKey as below. + +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + +exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL +is a special case of UniqueKey, which means there is only one row in that +relation.it has a stronger semantic than others. like SELECT uk FROM t; uk is +normal unique key and may have different values. SELECT colx FROM t WHERE uk = +const. colx is unique AND we have only 1 value. This field can used for +innerrel_is_unique. this logic is handled specially in add_uniquekey_for_onerow +function. + +multi_nullvals: true means multi null values may exist in these exprs, so the +uniqueness is not guaranteed in this case. This field is necessary for +remove_useless_join & reduce_unique_semijoins where we don't mind these +duplicated NULL values. It is set to true for 2 cases. One is a unique key +from a unique index but the related column is nullable. The other one is for +outer join. see populate_joinrel_uniquekeys for detail. + + +The UniqueKey can be used at the following cases at least: +1. remove_useless_joins. +2. reduce_semianti_joins +3. remove distinct node if distinct clause is unique. +4. remove aggnode if group by clause is unique. +5. Index Skip Scan (WIP) +6. Aggregation Push Down without 2 phase aggregation if the join can't + duplicated the aggregated rows. (work in progress feature) + +2. How is it maintained? + +We have a set of populate_xxx_unqiuekeys functions to maintain the uniquekey on +various cases. xxx includes baserel, joinrel, partitionedrel, distinctrel, +groupedrel, unionrel. and we also need to convert the uniquekey from subquery +to outer relation, which is what convert_subquery_uniquekeys does. + +1. The first part is about baserel. We handled 3 cases. suppose we have Unique +Index on (a, b). + +1. SELECT a, b FROM t. UniqueKey (a, b) +2. SELECT a FROM t WHERE b = 1; UniqueKey (a) +3. SELECT .. FROM t WHERE a = 1 AND b = 1; UniqueKey (NIL). onerow case, every + column is Unique. + +2. The next part is joinrel, this part is most error-prone, we simplified the rules +like below: +1. If the relation's UniqueKey can't be duplicated after join, then is will be + still valid for the join rel. The function we used here is + innerrel_keeps_unique. The basic idea is innerrel.any_col = outer.uk. + +2. If the UnqiueKey can't keep valid via the rule 1, the combination of the + UniqueKey from both sides are valid for sure. We can prove this as: if the + unique exprs from rel1 is duplicated by rel2, the duplicated rows must + contains different unique exprs from rel2. + +More considerations about onerow: +1. If relation with one row and it can't be duplicated, it is still possible + contains mulit_nullvas after outer join. +2. If the either UniqueKey can be duplicated after join, the can get one row + only when both side is one row AND there is no outer join. +3. Whenever the onerow UniqueKey is not a valid any more, we need to convert one + row UniqueKey to normal unique key since we don't store exprs for one-row + relation. get_exprs_from_uniquekeys will be used here. + + +More considerations about multi_nullvals after join: +1. If the original UnqiueKey has multi_nullvals, the final UniqueKey will have + mulit_nullvals in any case. +2. If a unique key doesn't allow mulit_nullvals, after some outer join, it + allows some outer join. + + +3. When we comes to subquery, we need to convert_subquery_unqiuekeys just like +convert_subquery_pathkeys. Only the UniqueKey insides subquery is referenced as +a Var in outer relation will be reused. The relationship between the outerrel.Var +and subquery.exprs is built with outerel->subroot->processed_tlist. + + +4. As for the SRF functions, it will break the uniqueness of uniquekey, However it +is handled in adjust_paths_for_srfs, which happens after the query_planner. so +we will maintain the UniqueKey until there and reset it to NIL at that +places. This can't help on distinct/group by elimination cases but probably help +in some other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is +semantic correctly. + + +5. As for inherit table, we first main the UnqiueKey on childrel as well. But for +partitioned table we need to maintain 2 different kinds of +UnqiueKey. 1). UniqueKey on the parent relation 2). UniqueKey on child +relation for partition wise query. + +Example: +CREATE TABLE p (a int not null, b int not null) partition by list (a); +CREATE TABLE p0 partition of p for values in (1); +CREATE TABLE p1 partition of p for values in (2); + +create unique index p0_b on p0(b); +create unique index p1_b on p1(b); + +Now b is only unique on partition level, so the distinct can't be removed on +the following cases. SELECT DISTINCT b FROM p; + +Another example is SELECT DISTINCT a, b FROM p WHERE a = 1; Since only one +partition is chosen, the UniqueKey on child relation is same as the UniqueKey on +parent relation. + +Another usage of UniqueKey on partition level is it be helpful for +partition-wise join. + +As for the UniqueKey on parent table level, it comes with 2 different ways, +1). the UniqueKey is also derived in UniqueKey index, but the index must be same +in all the related children relations and the unique index must contains +Partition Key in it. Example: + +CREATE UNIQUE INDEX p_ab ON p(a, b); -- where a is the partition key. + +-- Query +SELECT a, b FROM p; the (a, b) is a UniqueKey of p. + +2). If the parent relation has only one childrel, the UniqueKey on childrel is + the UniqueKey on parent as well. diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index 484dab0a1a..2ad9d06d7a 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -579,6 +579,12 @@ set_plain_rel_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) */ check_index_predicates(root, rel); + /* + * Now that we've marked which partial indexes are suitable, we can now + * build the relation's unique keys. + */ + populate_baserel_uniquekeys(root, rel, rel->indexlist); + /* Mark rel with estimated output rows, width, etc */ set_baserel_size_estimates(root, rel); } @@ -1310,6 +1316,8 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, /* Add paths to the append relation. */ add_paths_to_append_rel(root, rel, live_childrels); + if (IS_PARTITIONED_REL(rel)) + populate_partitionedrel_uniquekeys(root, rel, live_childrels); } @@ -2383,6 +2391,8 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, pathkeys, required_outer)); } + convert_subquery_uniquekeys(root, rel, sub_final_rel); + /* If outer rel allows parallelism, do same for partial paths. */ if (rel->consider_parallel && bms_is_empty(required_outer)) { diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c index db54a6ba2e..ef0fd2fb0b 100644 --- a/src/backend/optimizer/path/joinpath.c +++ b/src/backend/optimizer/path/joinpath.c @@ -71,13 +71,6 @@ static void consider_parallel_mergejoin(PlannerInfo *root, static void hash_inner_and_outer(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, RelOptInfo *innerrel, JoinType jointype, JoinPathExtraData *extra); -static List *select_mergejoin_clauses(PlannerInfo *root, - RelOptInfo *joinrel, - RelOptInfo *outerrel, - RelOptInfo *innerrel, - List *restrictlist, - JoinType jointype, - bool *mergejoin_allowed); static void generate_mergejoin_paths(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *innerrel, @@ -1927,7 +1920,7 @@ hash_inner_and_outer(PlannerInfo *root, * if it is mergejoinable and involves vars from the two sub-relations * currently of interest. */ -static List * +List * select_mergejoin_clauses(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c index 2d343cd293..b9163ee8ff 100644 --- a/src/backend/optimizer/path/joinrels.c +++ b/src/backend/optimizer/path/joinrels.c @@ -924,6 +924,8 @@ populate_joinrel_with_paths(PlannerInfo *root, RelOptInfo *rel1, /* Apply partitionwise join technique, if possible. */ try_partitionwise_join(root, rel1, rel2, joinrel, sjinfo, restrictlist); + + populate_joinrel_uniquekeys(root, joinrel, rel1, rel2, restrictlist, sjinfo->jointype); } diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index ce9bf87e9b..7e596d4194 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -33,7 +33,6 @@ static bool pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys); static bool matches_boolean_partition_clause(RestrictInfo *rinfo, RelOptInfo *partrel, int partkeycol); -static Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); static bool right_merge_direction(PlannerInfo *root, PathKey *pathkey); @@ -1035,7 +1034,7 @@ convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel, * We need this to ensure that we don't return pathkeys describing values * that are unavailable above the level of the subquery scan. */ -static Var * +Var * find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle) { ListCell *lc; diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c new file mode 100644 index 0000000000..b33bcd2f32 --- /dev/null +++ b/src/backend/optimizer/path/uniquekeys.c @@ -0,0 +1,1131 @@ +/*------------------------------------------------------------------------- + * + * uniquekeys.c + * Utilities for matching and building unique keys + * + * Portions Copyright (c) 2020, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/optimizer/path/uniquekeys.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "nodes/makefuncs.h" +#include "nodes/nodeFuncs.h" +#include "optimizer/pathnode.h" +#include "optimizer/paths.h" +#include "optimizer/appendinfo.h" +#include "optimizer/optimizer.h" +#include "optimizer/tlist.h" +#include "rewrite/rewriteManip.h" + + +/* + * This struct is used to help populate_joinrel_uniquekeys. + * + * added_to_joinrel is true if a uniquekey (from outerrel or innerrel) + * has been added to joinrel. + * useful is true if the exprs of the uniquekey still appears in joinrel. + */ +typedef struct UniqueKeyContextData +{ + UniqueKey *uniquekey; + bool added_to_joinrel; + bool useful; +} *UniqueKeyContext; + +static List *initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel); +static bool innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + bool reverse); + +static List *get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals); +static List *get_exprs_from_uniquekey(RelOptInfo *joinrel, + RelOptInfo *rel1, + UniqueKey *ukey); +static void add_uniquekey_for_onerow(RelOptInfo *rel); +static bool add_combined_uniquekey(RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype); + +/* Used for unique indexes checking for partitioned table */ +static bool index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind); +static IndexOptInfo *simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from); +static bool simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2); +static void adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_index); + +/* Helper function for grouped relation and distinct relation. */ +static void add_uniquekey_from_sortgroups(PlannerInfo *root, + RelOptInfo *rel, + List *sortgroups); + +/* + * populate_baserel_uniquekeys + * Populate 'baserel' uniquekeys list by looking at the rel's unique index + * and baserestrictinfo + */ +void +populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List *indexlist) +{ + ListCell *lc; + List *matched_uniq_indexes = NIL; + + /* Attrs appears in rel->reltarget->exprs. */ + Bitmapset *used_attrs = NULL; + + List *const_exprs = NIL; + List *expr_opfamilies = NIL; + + Assert(baserel->rtekind == RTE_RELATION); + + foreach(lc, indexlist) + { + IndexOptInfo *ind = (IndexOptInfo *) lfirst(lc); + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + matched_uniq_indexes = lappend(matched_uniq_indexes, ind); + } + + if (matched_uniq_indexes == NIL) + return; + + /* Check which attrs is used in baserel->reltarget */ + pull_varattnos((Node *)baserel->reltarget->exprs, baserel->relid, &used_attrs); + + /* Check which attrno is used at a mergeable const filter */ + foreach(lc, baserel->baserestrictinfo) + { + RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); + + if (rinfo->mergeopfamilies == NIL) + continue; + + if (bms_is_empty(rinfo->left_relids)) + { + const_exprs = lappend(const_exprs, get_rightop(rinfo->clause)); + } + else if (bms_is_empty(rinfo->right_relids)) + { + const_exprs = lappend(const_exprs, get_leftop(rinfo->clause)); + } + else + continue; + + expr_opfamilies = lappend(expr_opfamilies, rinfo->mergeopfamilies); + } + + foreach(lc, matched_uniq_indexes) + { + bool multi_nullvals, useful; + List *exprs = get_exprs_from_uniqueindex(lfirst_node(IndexOptInfo, lc), + const_exprs, + expr_opfamilies, + used_attrs, + &useful, + &multi_nullvals); + if (useful) + { + if (exprs == NIL) + { + /* All the columns in Unique Index matched with a restrictinfo */ + add_uniquekey_for_onerow(baserel); + return; + } + baserel->uniquekeys = lappend(baserel->uniquekeys, + makeUniqueKey(exprs, multi_nullvals)); + } + } +} + + +/* + * populate_partitionedrel_uniquekeys + * The UniqueKey on partitionrel comes from 2 cases: + * 1). Only one partition is involved in this query, the unique key can be + * copied to parent rel from childrel. + * 2). There are some unique index which includes partition key and exists + * in all the related partitions. + * We never mind rule 2 if we hit rule 1. + */ + +void +populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels) +{ + ListCell *lc; + List *global_uniq_indexlist = NIL; + RelOptInfo *childrel; + bool is_first = true; + + Assert(IS_PARTITIONED_REL(rel)); + + if (childrels == NIL) + return; + + /* + * If there is only one partition used in this query, the UniqueKey in childrel is + * still valid in parent level, but we need convert the format from child expr to + * parent expr. + */ + if (list_length(childrels) == 1) + { + /* Check for Rule 1 */ + RelOptInfo *childrel = linitial_node(RelOptInfo, childrels); + ListCell *lc; + Assert(childrel->reloptkind == RELOPT_OTHER_MEMBER_REL); + if (relation_is_onerow(childrel)) + { + add_uniquekey_for_onerow(rel); + return; + } + + foreach(lc, childrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + AppendRelInfo *appinfo = find_appinfo_by_child(root, childrel->relid); + List *parent_exprs = NIL; + bool can_reuse = true; + ListCell *lc2; + foreach(lc2, ukey->exprs) + { + Var *var = (Var *)lfirst(lc2); + /* + * If the expr comes from a expression, it is hard to build the expression + * in parent so ignore that case for now. + */ + if(!IsA(var, Var)) + { + can_reuse = false; + break; + } + /* Convert it to parent var */ + parent_exprs = lappend(parent_exprs, find_parent_var(appinfo, var)); + } + if (can_reuse) + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(parent_exprs, + ukey->multi_nullvals)); + } + } + else + { + /* Check for rule 2 */ + childrel = linitial_node(RelOptInfo, childrels); + foreach(lc, childrel->indexlist) + { + IndexOptInfo *ind = lfirst(lc); + IndexOptInfo *modified_index; + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + + /* + * During simple_copy_indexinfo_to_parent, we need to convert var from + * child var to parent var, index on expression is too complex to handle. + * so ignore it for now. + */ + if (ind->indexprs != NIL) + continue; + + modified_index = simple_copy_indexinfo_to_parent(root, rel, ind); + /* + * If the unique index doesn't contain partkey, then it is unique + * on this partition only, so it is useless for us. + */ + if (!index_constains_partkey(rel, modified_index)) + continue; + + global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index); + } + + if (global_uniq_indexlist != NIL) + { + foreach(lc, childrels) + { + RelOptInfo *child = lfirst(lc); + if (is_first) + { + is_first = false; + continue; + } + adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist); + } + /* Now we have a list of unique index which are exactly same on all childrels, + * Set the UniqueKey just like it is non-partition table + */ + populate_baserel_uniquekeys(root, rel, global_uniq_indexlist); + } + } +} + + +/* + * populate_distinctrel_uniquekeys + */ +void +populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel) +{ + /* The unique key before the distinct is still valid. */ + distinctrel->uniquekeys = list_copy(inputrel->uniquekeys); + add_uniquekey_from_sortgroups(root, distinctrel, root->parse->distinctClause); +} + +/* + * populate_grouprel_uniquekeys + */ +void +populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel) + +{ + Query *parse = root->parse; + bool input_ukey_added = false; + ListCell *lc; + + if (relation_is_onerow(inputrel)) + { + add_uniquekey_for_onerow(grouprel); + return; + } + if (parse->groupingSets) + return; + + /* A Normal group by without grouping set. */ + if (parse->groupClause) + { + /* + * Current even the groupby clause is Unique already, but if query has aggref + * We have to create grouprel still. To keep the UnqiueKey short, we will check + * the UniqueKey of input_rel still valid, if so we reuse it. + */ + foreach(lc, inputrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (list_is_subset(ukey->exprs, grouprel->reltarget->exprs)) + { + grouprel->uniquekeys = lappend(grouprel->uniquekeys, + ukey); + input_ukey_added = true; + } + } + if (!input_ukey_added) + /* + * group by clause must be a super-set of grouprel->reltarget->exprs except the + * aggregation expr, so if such exprs is unique already, no bother to generate + * new uniquekey for group by exprs. + */ + add_uniquekey_from_sortgroups(root, + grouprel, + root->parse->groupClause); + } + else + /* It has aggregation but without a group by, so only one row returned */ + add_uniquekey_for_onerow(grouprel); +} + +/* + * simple_copy_uniquekeys + * Using a function for the one-line code makes us easy to check where we simply + * copied the uniquekey. + */ +void +simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel) +{ + newrel->uniquekeys = oldrel->uniquekeys; +} + +/* + * populate_unionrel_uniquekeys + */ +void +populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel) +{ + ListCell *lc; + List *exprs = NIL; + + Assert(unionrel->uniquekeys == NIL); + + foreach(lc, unionrel->reltarget->exprs) + { + exprs = lappend(exprs, lfirst(lc)); + } + + if (exprs == NIL) + /* SQL: select union select; is valid, we need to handle it here. */ + add_uniquekey_for_onerow(unionrel); + else + unionrel->uniquekeys = lappend(unionrel->uniquekeys, + makeUniqueKey(exprs,false)); + +} + +/* + * populate_joinrel_uniquekeys + * + * populate uniquekeys for joinrel. We will check each relation to see if its + * UniqueKey is still valid via innerrel_keeps_unique, if so, we add it to + * joinrel. The multi_nullvals field will be changed to true for some outer + * join cases and one-row UniqueKey needs to be converted to normal UniqueKey + * for the same case as well. + * For the uniquekey in either baserel which can't be unique after join, we still + * check to see if combination of UniqueKeys from both side is still useful for us. + * if yes, we add it to joinrel as well. + */ +void +populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *outerrel, RelOptInfo *innerrel, + List *restrictlist, JoinType jointype) +{ + ListCell *lc, *lc2; + List *clause_list = NIL; + List *outerrel_ukey_ctx; + List *innerrel_ukey_ctx; + bool inner_onerow, outer_onerow; + bool mergejoin_allowed; + + /* Care about the outerrel relation only for SEMI/ANTI join */ + if (jointype == JOIN_SEMI || jointype == JOIN_ANTI) + { + foreach(lc, outerrel->uniquekeys) + { + UniqueKey *uniquekey = lfirst_node(UniqueKey, lc); + if (list_is_subset(uniquekey->exprs, joinrel->reltarget->exprs)) + joinrel->uniquekeys = lappend(joinrel->uniquekeys, uniquekey); + } + return; + } + + Assert(jointype == JOIN_LEFT || jointype == JOIN_FULL || jointype == JOIN_INNER); + + /* Fast path */ + if (innerrel->uniquekeys == NIL || outerrel->uniquekeys == NIL) + return; + + inner_onerow = relation_is_onerow(innerrel); + outer_onerow = relation_is_onerow(outerrel); + + outerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(outerrel); + innerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(innerrel); + + clause_list = select_mergejoin_clauses(root, joinrel, outerrel, innerrel, + restrictlist, jointype, + &mergejoin_allowed); + + if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true /* reverse */)) + { + bool outer_impact = jointype == JOIN_FULL; + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + /* Outer relation has one row, and the unique key is not duplicated after join, + * the joinrel will still has one row unless the jointype == JOIN_FULL. + */ + if (outer_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (outer_onerow) + { + /* + * The onerow outerrel becomes multi rows and multi_nullvals + * will be changed to true. We also need to set the exprs correctly since it + * can't be NIL any more. + */ + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(joinrel, outerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Change multi_nullvals to true due to the full join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, true)); + else + /* Just reuse it */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + } + ctx->added_to_joinrel = true; + } + } + + if (innerrel_keeps_unique(root, outerrel, innerrel, clause_list, false)) + { + bool outer_impact = jointype == JOIN_FULL || jointype == JOIN_LEFT;; + + foreach(lc, innerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + if (inner_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (inner_onerow) + { + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(joinrel, innerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Need to change multi_nullvals to true due to the outer join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, + true)); + else + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + + } + ctx->added_to_joinrel = true; + } + } + + /* + * The combination of the UniqueKey from both sides is unique as well regardless + * of join type, but no bother to add it if its subset has been added to joinrel + * already or it is not useful for the joinrel. + */ + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx1 = (UniqueKeyContext) lfirst(lc); + if (ctx1->added_to_joinrel || !ctx1->useful) + continue; + foreach(lc2, innerrel_ukey_ctx) + { + UniqueKeyContext ctx2 = (UniqueKeyContext) lfirst(lc2); + if (ctx2->added_to_joinrel || !ctx2->useful) + continue; + if (add_combined_uniquekey(joinrel, outerrel, innerrel, + ctx1->uniquekey, ctx2->uniquekey, + jointype)) + /* If we set a onerow UniqueKey to joinrel, we don't need other. */ + return; + } + } +} + + +/* + * convert_subquery_uniquekeys + * + * Covert the UniqueKey in subquery to outer relation. + */ +void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel) +{ + ListCell *lc; + + if (sub_final_rel->uniquekeys == NIL) + return; + + if (relation_is_onerow(sub_final_rel)) + { + add_uniquekey_for_onerow(currel); + return; + } + + Assert(currel->subroot != NULL); + + foreach(lc, sub_final_rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + ListCell *lc; + List *exprs = NIL; + bool ukey_useful = true; + + /* One row case is handled above */ + Assert(ukey->exprs != NIL); + foreach(lc, ukey->exprs) + { + Var *var; + TargetEntry *tle = tlist_member(lfirst(lc), + currel->subroot->processed_tlist); + if (tle == NULL) + { + ukey_useful = false; + break; + } + var = find_var_for_subquery_tle(currel, tle); + if (var == NULL) + { + ukey_useful = false; + break; + } + exprs = lappend(exprs, var); + } + + if (ukey_useful) + currel->uniquekeys = lappend(currel->uniquekeys, + makeUniqueKey(exprs, + ukey->multi_nullvals)); + + } +} + +/* + * innerrel_keeps_unique + * + * Check if Unique key of the innerrel is valid after join. innerrel's UniqueKey + * will be still valid if innerrel's any-column mergeop outrerel's uniquekey + * exists in clause_list. + * + * Note: the clause_list must be a list of mergeable restrictinfo already. + */ +static bool +innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *clause_list, + bool reverse) +{ + ListCell *lc, *lc2, *lc3; + + if (outerrel->uniquekeys == NIL || innerrel->uniquekeys == NIL) + return false; + + /* Check if there is outerrel's uniquekey in mergeable clause. */ + foreach(lc, outerrel->uniquekeys) + { + List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs; + bool clauselist_matchs_all_exprs = true; + foreach(lc2, outer_uq_exprs) + { + Node *outer_uq_expr = lfirst(lc2); + bool find_uq_expr_in_clauselist = false; + foreach(lc3, clause_list) + { + RestrictInfo *rinfo = lfirst_node(RestrictInfo, lc3); + Node *outer_expr; + if (reverse) + outer_expr = rinfo->outer_is_left ? get_rightop(rinfo->clause) : get_leftop(rinfo->clause); + else + outer_expr = rinfo->outer_is_left ? get_leftop(rinfo->clause) : get_rightop(rinfo->clause); + if (equal(outer_expr, outer_uq_expr)) + { + find_uq_expr_in_clauselist = true; + break; + } + } + if (!find_uq_expr_in_clauselist) + { + /* No need to check the next exprs in the current uniquekey */ + clauselist_matchs_all_exprs = false; + break; + } + } + + if (clauselist_matchs_all_exprs) + return true; + } + return false; +} + + +/* + * relation_is_onerow + * Check if it is a one-row relation by checking UniqueKey. + */ +bool +relation_is_onerow(RelOptInfo *rel) +{ + UniqueKey *ukey; + if (rel->uniquekeys == NIL) + return false; + ukey = linitial_node(UniqueKey, rel->uniquekeys); + return ukey->exprs == NIL && list_length(rel->uniquekeys) == 1; +} + +/* + * relation_has_uniquekeys_for + * Returns true if we have proofs that 'rel' cannot return multiple rows with + * the same values in each of 'exprs'. Otherwise returns false. + */ +bool +relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel, + List *exprs, bool allow_multinulls) +{ + ListCell *lc; + + /* + * For UniqueKey->onerow case, the uniquekey->exprs is empty as well + * so we can't rely on list_is_subset to handle this special cases + */ + if (exprs == NIL) + return false; + + foreach(lc, rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (ukey->multi_nullvals && !allow_multinulls) + continue; + if (list_is_subset(ukey->exprs, exprs)) + return true; + } + return false; +} + + +/* + * get_exprs_from_uniqueindex + * + * Return a list of exprs which is unique. set useful to false if this + * unique index is not useful for us. + */ +static List * +get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals) +{ + List *exprs = NIL; + ListCell *indexpr_item; + int c = 0; + + *useful = true; + *multi_nullvals = false; + + indexpr_item = list_head(unique_index->indexprs); + for(c = 0; c < unique_index->ncolumns; c++) + { + int attr = unique_index->indexkeys[c]; + Expr *expr; + bool matched_const = false; + ListCell *lc1, *lc2; + + if(attr > 0) + { + expr = list_nth_node(TargetEntry, unique_index->indextlist, c)->expr; + } + else if (attr == 0) + { + /* Expression index */ + expr = lfirst(indexpr_item); + indexpr_item = lnext(unique_index->indexprs, indexpr_item); + } + else /* attr < 0 */ + { + /* Index on system column is not supported */ + Assert(false); + } + + /* + * Check index_col = Const case with regarding to opfamily checking + * If we can remove the index_col from the final UniqueKey->exprs. + */ + forboth(lc1, const_exprs, lc2, const_expr_opfamilies) + { + if (list_member_oid((List *)lfirst(lc2), unique_index->opfamily[c]) + && match_index_to_operand((Node *) lfirst(lc1), c, unique_index)) + { + matched_const = true; + break; + } + } + + if (matched_const) + continue; + + /* Check if the indexed expr is used in rel */ + if (attr > 0) + { + /* + * Normal Indexed column, if the col is not used, then the index is useless + * for uniquekey. + */ + attr -= FirstLowInvalidHeapAttributeNumber; + + if (!bms_is_member(attr, used_varattrs)) + { + *useful = false; + break; + } + } + else if (!list_member(unique_index->rel->reltarget->exprs, expr)) + { + /* Expression index but the expression is not used in rel */ + *useful = false; + break; + } + + /* check not null property. */ + if (attr == 0) + { + /* We never know if a expression yields null or not */ + *multi_nullvals = true; + } + else if (!bms_is_member(attr, unique_index->rel->notnullattrs) + && !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber, + unique_index->rel->notnullattrs)) + { + *multi_nullvals = true; + } + + exprs = lappend(exprs, expr); + } + return exprs; +} + + +/* + * add_uniquekey_for_onerow + * If we are sure that the relation only returns one row, then all the columns + * are unique. However we don't need to create UniqueKey for every column, we + * just set exprs = NIL and overwrites all the other UniqueKey on this RelOptInfo + * since this one has strongest semantics. + */ +void +add_uniquekey_for_onerow(RelOptInfo *rel) +{ + /* + * We overwrite the previous UniqueKey on purpose since this one has the + * strongest semantic. + */ + rel->uniquekeys = list_make1(makeUniqueKey(NIL, false)); +} + + +/* + * initililze_uniquecontext_for_joinrel + * Return a List of UniqueKeyContext for an inputrel + */ +static List * +initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel) +{ + List *res = NIL; + ListCell *lc; + foreach(lc, inputrel->uniquekeys) + { + UniqueKeyContext context; + context = palloc(sizeof(struct UniqueKeyContextData)); + context->uniquekey = lfirst_node(UniqueKey, lc); + context->added_to_joinrel = false; + context->useful = true; + res = lappend(res, context); + } + return res; +} + + +/* + * get_exprs_from_uniquekey + * Unify the way of get List of exprs from a one-row UniqueKey or + * normal UniqueKey. for the onerow case, every expr in rel1 is a valid + * UniqueKey. Return a List of exprs. + * + * rel1: The relation which you want to get the exprs. + * ukey: The UniqueKey you want to get the exprs. + */ +static List * +get_exprs_from_uniquekey(RelOptInfo *joinrel, RelOptInfo *rel1, UniqueKey *ukey) +{ + ListCell *lc; + bool onerow = rel1 != NULL && relation_is_onerow(rel1); + + List *res = NIL; + Assert(onerow || ukey); + if (onerow) + { + /* Only cares about the exprs still exist in joinrel */ + foreach(lc, joinrel->reltarget->exprs) + { + Bitmapset *relids = pull_varnos(lfirst(lc)); + if (bms_is_subset(relids, rel1->relids)) + { + res = lappend(res, list_make1(lfirst(lc))); + } + } + } + else + { + res = list_make1(ukey->exprs); + } + return res; +} + +/* + * Partitioned table Unique Keys. + * The partition table unique key is maintained as: + * 1. The index must be unique as usual. + * 2. The index must contains partition key. + * 3. The index must exist on all the child rel. see simple_indexinfo_equal for + * how we compare it. + */ + +/* + * index_constains_partkey + * return true if the index contains the partiton key. + */ +static bool +index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind) +{ + ListCell *lc; + int i; + Assert(IS_PARTITIONED_REL(partrel)); + Assert(partrel->part_scheme->partnatts > 0); + + for(i = 0; i < partrel->part_scheme->partnatts; i++) + { + Node *part_expr = linitial(partrel->partexprs[i]); + bool found_in_index = false; + foreach(lc, ind->indextlist) + { + Expr *index_expr = lfirst_node(TargetEntry, lc)->expr; + if (equal(index_expr, part_expr)) + { + found_in_index = true; + break; + } + } + if (!found_in_index) + return false; + } + return true; +} + +/* + * simple_indexinfo_equal + * + * Used to check if the 2 index is same as each other. The index here + * is COPIED from childrel and did some tiny changes(see + * simple_copy_indexinfo_to_parent) + */ +static bool +simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2) +{ + Size oid_cmp_len = sizeof(Oid) * ind1->ncolumns; + + return ind1->ncolumns == ind2->ncolumns && + ind1->unique == ind2->unique && + memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0 && + memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0 && + memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0 && + memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0 && + equal(get_tlist_exprs(ind1->indextlist, true), + get_tlist_exprs(ind2->indextlist, true)); +} + + +/* + * The below macros are used for simple_copy_indexinfo_to_parent which is so + * customized that I don't want to put it to copyfuncs.c. So copy it here. + */ +#define COPY_POINTER_FIELD(fldname, sz) \ + do { \ + Size _size = (sz); \ + newnode->fldname = palloc(_size); \ + memcpy(newnode->fldname, from->fldname, _size); \ + } while (0) + +#define COPY_NODE_FIELD(fldname) \ + (newnode->fldname = copyObjectImpl(from->fldname)) + +#define COPY_SCALAR_FIELD(fldname) \ + (newnode->fldname = from->fldname) + + +/* + * simple_copy_indexinfo_to_parent (from partition) + * Copy the IndexInfo from child relation to parent relation with some modification, + * which is used to test: + * 1. If the same index exists in all the childrels. + * 2. If the parentrel->reltarget/basicrestrict info matches this index. + */ +static IndexOptInfo * +simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from) +{ + IndexOptInfo *newnode = makeNode(IndexOptInfo); + AppendRelInfo *appinfo = find_appinfo_by_child(root, from->rel->relid); + ListCell *lc; + int idx = 0; + + COPY_SCALAR_FIELD(ncolumns); + COPY_SCALAR_FIELD(nkeycolumns); + COPY_SCALAR_FIELD(unique); + COPY_SCALAR_FIELD(immediate); + /* We just need to know if it is NIL or not */ + COPY_SCALAR_FIELD(indpred); + COPY_SCALAR_FIELD(predOK); + COPY_POINTER_FIELD(indexkeys, from->ncolumns * sizeof(int)); + COPY_POINTER_FIELD(indexcollations, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opfamily, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opcintype, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(sortopfamily, from->ncolumns * sizeof(Oid)); + COPY_NODE_FIELD(indextlist); + + /* Convert index exprs on child expr to expr on parent */ + foreach(lc, newnode->indextlist) + { + TargetEntry *tle = lfirst_node(TargetEntry, lc); + /* Index on expression is ignored */ + Assert(IsA(tle->expr, Var)); + tle->expr = (Expr *) find_parent_var(appinfo, (Var *) tle->expr); + newnode->indexkeys[idx] = castNode(Var, tle->expr)->varattno; + idx++; + } + newnode->rel = parentrel; + return newnode; +} + +/* + * adjust_partition_unique_indexlist + * + * global_unique_indexes: At the beginning, it contains the copy & modified + * unique index from the first partition. And then check if each index in it still + * exists in the following partitions. If no, remove it. at last, it has an + * index list which exists in all the partitions. + */ +static void +adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_indexes) +{ + ListCell *lc, *lc2; + foreach(lc, *global_unique_indexes) + { + IndexOptInfo *g_ind = lfirst_node(IndexOptInfo, lc); + bool found_in_child = false; + + foreach(lc2, childrel->indexlist) + { + IndexOptInfo *p_ind = lfirst_node(IndexOptInfo, lc2); + IndexOptInfo *p_ind_copy; + if (!p_ind->unique || !p_ind->immediate || + (p_ind->indpred != NIL && !p_ind->predOK)) + continue; + p_ind_copy = simple_copy_indexinfo_to_parent(root, parentrel, p_ind); + if (simple_indexinfo_equal(p_ind_copy, g_ind)) + { + found_in_child = true; + break; + } + } + if (!found_in_child) + /* The index doesn't exist in childrel, remove it from global_unique_indexes */ + *global_unique_indexes = foreach_delete_current(*global_unique_indexes, lc); + } +} + +/* Helper function for groupres/distinctrel */ +static void +add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgroups) +{ + Query *parse = root->parse; + List *exprs; + + /* + * XXX: If there are some vars which is not in current levelsup, the semantic is + * imprecise, should we avoid it or not? levelsup = 1 is just a demo, maybe we need to + * check every level other than 0, if so, looks we have to write another + * pull_var_walker. + */ + List *upper_vars = pull_vars_of_level((Node*)sortgroups, 1); + + if (upper_vars != NIL) + return; + + exprs = get_sortgrouplist_exprs(sortgroups, parse->targetList); + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(exprs, + false /* sortgroupclause can't be multi_nullvals */)); +} + + +/* + * add_combined_uniquekey + * The combination of both UniqueKeys is a valid UniqueKey for joinrel no matter + * the jointype. + */ +bool +add_combined_uniquekey(RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype) +{ + + ListCell *lc1, *lc2; + + /* Either side has multi_nullvals or we have outer join, + * the combined UniqueKey has multi_nullvals */ + bool multi_nullvals = outer_ukey->multi_nullvals || + inner_ukey->multi_nullvals || IS_OUTER_JOIN(jointype); + + /* The only case we can get onerow joinrel after join */ + if (relation_is_onerow(outer_rel) + && relation_is_onerow(inner_rel) + && jointype == JOIN_INNER) + { + add_uniquekey_for_onerow(joinrel); + return true; + } + + foreach(lc1, get_exprs_from_uniquekey(joinrel, outer_rel, outer_ukey)) + { + foreach(lc2, get_exprs_from_uniquekey(joinrel, inner_rel, inner_ukey)) + { + List *exprs = list_concat_copy(lfirst_node(List, lc1), lfirst_node(List, lc2)); + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(exprs, + multi_nullvals)); + } + } + return false; +} diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index b406d41e91..0551ae0512 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -2389,6 +2389,8 @@ grouping_planner(PlannerInfo *root, bool inheritance_update, add_path(final_rel, path); } + simple_copy_uniquekeys(current_rel, final_rel); + /* * Generate partial paths for final_rel, too, if outer query levels might * be able to make use of them. @@ -3899,6 +3901,8 @@ create_grouping_paths(PlannerInfo *root, } set_cheapest(grouped_rel); + + populate_grouprel_uniquekeys(root, grouped_rel, input_rel); return grouped_rel; } @@ -4615,7 +4619,7 @@ create_window_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(window_rel); - + simple_copy_uniquekeys(input_rel, window_rel); return window_rel; } @@ -4911,7 +4915,7 @@ create_distinct_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(distinct_rel); - + populate_distinctrel_uniquekeys(root, input_rel, distinct_rel); return distinct_rel; } @@ -5172,6 +5176,8 @@ create_ordered_paths(PlannerInfo *root, */ Assert(ordered_rel->pathlist != NIL); + simple_copy_uniquekeys(input_rel, ordered_rel); + return ordered_rel; } @@ -6049,6 +6055,9 @@ adjust_paths_for_srfs(PlannerInfo *root, RelOptInfo *rel, if (list_length(targets) == 1) return; + /* UniqueKey is not valid after handling the SRF. */ + rel->uniquekeys = NIL; + /* * Stack SRF-evaluation nodes atop each path for the rel. * diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c index 951aed80e7..e94e92937c 100644 --- a/src/backend/optimizer/prep/prepunion.c +++ b/src/backend/optimizer/prep/prepunion.c @@ -689,6 +689,8 @@ generate_union_paths(SetOperationStmt *op, PlannerInfo *root, /* Undo effects of possibly forcing tuple_fraction to 0 */ root->tuple_fraction = save_fraction; + /* Add the UniqueKeys */ + populate_unionrel_uniquekeys(root, result_rel); return result_rel; } diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c index d722063cf3..44c37ecffc 100644 --- a/src/backend/optimizer/util/appendinfo.c +++ b/src/backend/optimizer/util/appendinfo.c @@ -746,3 +746,47 @@ find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos) } return appinfos; } + +/* + * find_appinfo_by_child + * + */ +AppendRelInfo * +find_appinfo_by_child(PlannerInfo *root, Index child_index) +{ + ListCell *lc; + foreach(lc, root->append_rel_list) + { + AppendRelInfo *appinfo = lfirst_node(AppendRelInfo, lc); + if (appinfo->child_relid == child_index) + return appinfo; + } + elog(ERROR, "parent relation cant be found"); + return NULL; +} + +/* + * find_parent_var + * + */ +Var * +find_parent_var(AppendRelInfo *appinfo, Var *child_var) +{ + ListCell *lc; + Var *res = NULL; + Index attno = 1; + foreach(lc, appinfo->translated_vars) + { + Node *child_node = lfirst(lc); + if (equal(child_node, child_var)) + { + res = copyObject(child_var); + res->varattno = attno; + res->varno = appinfo->parent_relid; + } + attno++; + } + if (res == NULL) + elog(ERROR, "parent var can't be found."); + return res; +} diff --git a/src/backend/optimizer/util/inherit.c b/src/backend/optimizer/util/inherit.c index 3132fd35a5..d66b40ec50 100644 --- a/src/backend/optimizer/util/inherit.c +++ b/src/backend/optimizer/util/inherit.c @@ -736,6 +736,7 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, { Node *onecq = (Node *) lfirst(lc2); bool pseudoconstant; + RestrictInfo *child_rinfo; /* check for pseudoconstant (no Vars or volatile functions) */ pseudoconstant = @@ -747,13 +748,14 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, root->hasPseudoConstantQuals = true; } /* reconstitute RestrictInfo with appropriate properties */ - childquals = lappend(childquals, - make_restrictinfo((Expr *) onecq, - rinfo->is_pushed_down, - rinfo->outerjoin_delayed, - pseudoconstant, - rinfo->security_level, - NULL, NULL, NULL)); + child_rinfo = make_restrictinfo((Expr *) onecq, + rinfo->is_pushed_down, + rinfo->outerjoin_delayed, + pseudoconstant, + rinfo->security_level, + NULL, NULL, NULL); + child_rinfo->mergeopfamilies = rinfo->mergeopfamilies; + childquals = lappend(childquals, child_rinfo); /* track minimum security level among child quals */ cq_min_security = Min(cq_min_security, rinfo->security_level); } diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 31d9aedeeb..c83f17acb7 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -16,6 +16,7 @@ #include "nodes/execnodes.h" #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" extern A_Expr *makeA_Expr(A_Expr_Kind kind, List *name, @@ -105,4 +106,6 @@ extern GroupingSet *makeGroupingSet(GroupingSetKind kind, List *content, int loc extern VacuumRelation *makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols); +extern UniqueKey* makeUniqueKey(List *exprs, bool multi_nullvals); + #endif /* MAKEFUNC_H */ diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 381d84b4e4..41110ed888 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -264,6 +264,7 @@ typedef enum NodeTag T_EquivalenceMember, T_PathKey, T_PathTarget, + T_UniqueKey, T_RestrictInfo, T_IndexClause, T_PlaceHolderVar, diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index 9e3ebd488a..02e4458bef 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -730,6 +730,7 @@ typedef struct RelOptInfo QualCost baserestrictcost; /* cost of evaluating the above */ Index baserestrict_min_security; /* min security_level found in * baserestrictinfo */ + List *uniquekeys; /* List of UniqueKey */ List *joininfo; /* RestrictInfo structures for join clauses * involving this rel */ bool has_eclass_joins; /* T means joininfo is incomplete */ @@ -1047,6 +1048,28 @@ typedef struct PathKey } PathKey; +/* + * UniqueKey + * + * Represents the unique properties held by a RelOptInfo. + * + * exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL + * is a special case of UniqueKey, which means there is only 1 row in that + * relation. + * multi_nullvals: true means multi null values may exist in these exprs, so the + * uniqueness is not guaranteed in this case. This field is necessary for + * remove_useless_join & reduce_unique_semijoins where we don't mind these + * duplicated NULL values. It is set to true for 2 cases. One is a unique key + * from a unique index but the related column is nullable. The other one is for + * outer join. see populate_joinrel_uniquekeys for detail. + */ +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + /* * PathTarget * @@ -2473,7 +2496,7 @@ typedef enum * * flags indicating what kinds of grouping are possible. * partial_costs_set is true if the agg_partial_costs and agg_final_costs - * have been initialized. + * have been initialized. * agg_partial_costs gives partial aggregation costs. * agg_final_costs gives finalization costs. * target_parallel_safe is true if target is parallel safe. @@ -2503,8 +2526,8 @@ typedef struct * limit_tuples is an estimated bound on the number of output tuples, * or -1 if no LIMIT or couldn't estimate. * count_est and offset_est are the estimated values of the LIMIT and OFFSET - * expressions computed by preprocess_limit() (see comments for - * preprocess_limit() for more information). + * expressions computed by preprocess_limit() (see comments for + * preprocess_limit() for more information). */ typedef struct { diff --git a/src/include/nodes/pg_list.h b/src/include/nodes/pg_list.h index 14ea2766ad..621f54a9f8 100644 --- a/src/include/nodes/pg_list.h +++ b/src/include/nodes/pg_list.h @@ -528,6 +528,8 @@ extern bool list_member_ptr(const List *list, const void *datum); extern bool list_member_int(const List *list, int datum); extern bool list_member_oid(const List *list, Oid datum); +extern bool list_is_subset(const List *members, const List *target); + extern List *list_delete(List *list, void *datum); extern List *list_delete_ptr(List *list, void *datum); extern List *list_delete_int(List *list, int datum); diff --git a/src/include/optimizer/appendinfo.h b/src/include/optimizer/appendinfo.h index d6a27a60dd..e87c92a054 100644 --- a/src/include/optimizer/appendinfo.h +++ b/src/include/optimizer/appendinfo.h @@ -32,4 +32,7 @@ extern Relids adjust_child_relids_multilevel(PlannerInfo *root, Relids relids, extern AppendRelInfo **find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos); +extern AppendRelInfo *find_appinfo_by_child(PlannerInfo *root, Index child_index); +extern Var *find_parent_var(AppendRelInfo *appinfo, Var *child_var); + #endif /* APPENDINFO_H */ diff --git a/src/include/optimizer/optimizer.h b/src/include/optimizer/optimizer.h index 3e4171056e..9445141263 100644 --- a/src/include/optimizer/optimizer.h +++ b/src/include/optimizer/optimizer.h @@ -23,6 +23,7 @@ #define OPTIMIZER_H #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" /* * We don't want to include nodes/pathnodes.h here, because non-planner @@ -156,6 +157,7 @@ extern TargetEntry *get_sortgroupref_tle(Index sortref, List *targetList); extern TargetEntry *get_sortgroupclause_tle(SortGroupClause *sgClause, List *targetList); +extern Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); extern Node *get_sortgroupclause_expr(SortGroupClause *sgClause, List *targetList); extern List *get_sortgrouplist_exprs(List *sgClauses, diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h index 10b6e81079..9217a8d6c6 100644 --- a/src/include/optimizer/paths.h +++ b/src/include/optimizer/paths.h @@ -240,5 +240,48 @@ extern PathKey *make_canonical_pathkey(PlannerInfo *root, int strategy, bool nulls_first); extern void add_paths_to_append_rel(PlannerInfo *root, RelOptInfo *rel, List *live_childrels); +extern List *select_mergejoin_clauses(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + JoinType jointype, + bool *mergejoin_allowed); + +/* + * uniquekeys.c + * Utilities for matching and building unique keys + */ +extern void populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List* unique_index_list); +extern void populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels); +extern void populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel); +extern void populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel); +extern void populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel); +extern void simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel); +extern void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel); +extern void populate_joinrel_uniquekeys(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + RelOptInfo *rel2, + List *restrictlist, + JoinType jointype); + +extern bool relation_has_uniquekeys_for(PlannerInfo *root, + RelOptInfo *rel, + List *exprs, + bool allow_multinulls); +extern bool relation_is_onerow(RelOptInfo *rel); #endif /* PATHS_H */ -- 2.21.0 --op7mf2i72xk5yu7c Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v36-0003-Extend-UniqueKeys.patch" ^ permalink raw reply [nested|flat] 91+ messages in thread
* [PATCH v38 2/6] Introduce UniqueKey attributes on RelOptInfo struct. @ 2020-05-11 07:50 一挃 <[email protected]> 0 siblings, 0 replies; 91+ messages in thread From: 一挃 @ 2020-05-11 07:50 UTC (permalink / raw) UniqueKey is a set of exprs on RelOptInfo which represents the exprs will be unique on the given RelOptInfo. You can see README.uniquekey for more information. --- src/backend/nodes/copyfuncs.c | 13 + src/backend/nodes/list.c | 31 + src/backend/nodes/makefuncs.c | 13 + src/backend/nodes/outfuncs.c | 11 + src/backend/nodes/readfuncs.c | 10 + src/backend/optimizer/path/Makefile | 3 +- src/backend/optimizer/path/README.uniquekey | 131 +++ src/backend/optimizer/path/allpaths.c | 10 + src/backend/optimizer/path/joinpath.c | 9 +- src/backend/optimizer/path/joinrels.c | 2 + src/backend/optimizer/path/pathkeys.c | 3 +- src/backend/optimizer/path/uniquekeys.c | 1135 +++++++++++++++++++ src/backend/optimizer/plan/planner.c | 13 +- src/backend/optimizer/prep/prepunion.c | 2 + src/backend/optimizer/util/appendinfo.c | 44 + src/backend/optimizer/util/inherit.c | 18 +- src/include/nodes/makefuncs.h | 3 + src/include/nodes/nodes.h | 1 + src/include/nodes/pathnodes.h | 29 +- src/include/nodes/pg_list.h | 10 + src/include/optimizer/appendinfo.h | 3 + src/include/optimizer/optimizer.h | 2 + src/include/optimizer/paths.h | 43 + 23 files changed, 1515 insertions(+), 24 deletions(-) create mode 100644 src/backend/optimizer/path/README.uniquekey create mode 100644 src/backend/optimizer/path/uniquekeys.c diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..75c1c5e824 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2296,6 +2296,16 @@ _copyPathKey(const PathKey *from) return newnode; } +static UniqueKey * +_copyUniqueKey(const UniqueKey *from) +{ + UniqueKey *newnode = makeNode(UniqueKey); + + COPY_NODE_FIELD(exprs); + COPY_SCALAR_FIELD(multi_nullvals); + + return newnode; +} /* * _copyRestrictInfo */ @@ -5220,6 +5230,9 @@ copyObjectImpl(const void *from) case T_PathKey: retval = _copyPathKey(from); break; + case T_UniqueKey: + retval = _copyUniqueKey(from); + break; case T_RestrictInfo: retval = _copyRestrictInfo(from); break; diff --git a/src/backend/nodes/list.c b/src/backend/nodes/list.c index dbf6b30233..ca099495a1 100644 --- a/src/backend/nodes/list.c +++ b/src/backend/nodes/list.c @@ -702,6 +702,37 @@ list_member_oid(const List *list, Oid datum) return false; } +/* + * return true iff every entry in "members" list is also present + * in the "target" list. + */ +bool +list_is_subset(const List *members, const List *target) +{ + const ListCell *lc1, *lc2; + + Assert(IsPointerList(members)); + Assert(IsPointerList(target)); + check_list_invariants(members); + check_list_invariants(target); + + foreach(lc1, members) + { + bool found = false; + foreach(lc2, target) + { + if (equal(lfirst(lc1), lfirst(lc2))) + { + found = true; + break; + } + } + if (!found) + return false; + } + return true; +} + /* * Delete the n'th cell (counting from 0) in list. * diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 01c110cd2f..40415d0f5b 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -815,3 +815,16 @@ makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols) v->va_cols = va_cols; return v; } + + +/* + * makeUniqueKey + */ +UniqueKey* +makeUniqueKey(List *exprs, bool multi_nullvals) +{ + UniqueKey * ukey = makeNode(UniqueKey); + ukey->exprs = exprs; + ukey->multi_nullvals = multi_nullvals; + return ukey; +} diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..44154cde6a 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2456,6 +2456,14 @@ _outPathKey(StringInfo str, const PathKey *node) WRITE_BOOL_FIELD(pk_nulls_first); } +static void +_outUniqueKey(StringInfo str, const UniqueKey *node) +{ + WRITE_NODE_TYPE("UNIQUEKEY"); + WRITE_NODE_FIELD(exprs); + WRITE_BOOL_FIELD(multi_nullvals); +} + static void _outPathTarget(StringInfo str, const PathTarget *node) { @@ -4198,6 +4206,9 @@ outNode(StringInfo str, const void *obj) case T_PathKey: _outPathKey(str, obj); break; + case T_UniqueKey: + _outUniqueKey(str, obj); + break; case T_PathTarget: _outPathTarget(str, obj); break; diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index c5e136e9c3..b3e212bf1c 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -492,6 +492,14 @@ _readSetOperationStmt(void) READ_DONE(); } +static UniqueKey * +_readUniqueKey(void) +{ + READ_LOCALS(UniqueKey); + READ_NODE_FIELD(exprs); + READ_BOOL_FIELD(multi_nullvals); + READ_DONE(); +} /* * Stuff from primnodes.h. @@ -2717,6 +2725,8 @@ parseNodeString(void) return_value = _readCommonTableExpr(); else if (MATCH("SETOPERATIONSTMT", 16)) return_value = _readSetOperationStmt(); + else if (MATCH("UNIQUEKEY", 9)) + return_value = _readUniqueKey(); else if (MATCH("ALIAS", 5)) return_value = _readAlias(); else if (MATCH("RANGEVAR", 8)) diff --git a/src/backend/optimizer/path/Makefile b/src/backend/optimizer/path/Makefile index 1e199ff66f..7b9820c25f 100644 --- a/src/backend/optimizer/path/Makefile +++ b/src/backend/optimizer/path/Makefile @@ -21,6 +21,7 @@ OBJS = \ joinpath.o \ joinrels.o \ pathkeys.o \ - tidpath.o + tidpath.o \ + uniquekeys.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/optimizer/path/README.uniquekey b/src/backend/optimizer/path/README.uniquekey new file mode 100644 index 0000000000..5eac761995 --- /dev/null +++ b/src/backend/optimizer/path/README.uniquekey @@ -0,0 +1,131 @@ +1. What is UniqueKey? +We can think UniqueKey is a set of exprs for a RelOptInfo, which we are insure +that doesn't yields same result among all the rows. The simplest UniqueKey +format is primary key. + +However we define the UnqiueKey as below. + +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + +exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL +is a special case of UniqueKey, which means there is only one row in that +relation.it has a stronger semantic than others. like SELECT uk FROM t; uk is +normal unique key and may have different values. SELECT colx FROM t WHERE uk = +const. colx is unique AND we have only 1 value. This field can used for +innerrel_is_unique. this logic is handled specially in add_uniquekey_for_onerow +function. + +multi_nullvals: true means multi null values may exist in these exprs, so the +uniqueness is not guaranteed in this case. This field is necessary for +remove_useless_join & reduce_unique_semijoins where we don't mind these +duplicated NULL values. It is set to true for 2 cases. One is a unique key +from a unique index but the related column is nullable. The other one is for +outer join. see populate_joinrel_uniquekeys for detail. + + +The UniqueKey can be used at the following cases at least: +1. remove_useless_joins. +2. reduce_semianti_joins +3. remove distinct node if distinct clause is unique. +4. remove aggnode if group by clause is unique. +5. Index Skip Scan (WIP) +6. Aggregation Push Down without 2 phase aggregation if the join can't + duplicated the aggregated rows. (work in progress feature) + +2. How is it maintained? + +We have a set of populate_xxx_unqiuekeys functions to maintain the uniquekey on +various cases. xxx includes baserel, joinrel, partitionedrel, distinctrel, +groupedrel, unionrel. and we also need to convert the uniquekey from subquery +to outer relation, which is what convert_subquery_uniquekeys does. + +1. The first part is about baserel. We handled 3 cases. suppose we have Unique +Index on (a, b). + +1. SELECT a, b FROM t. UniqueKey (a, b) +2. SELECT a FROM t WHERE b = 1; UniqueKey (a) +3. SELECT .. FROM t WHERE a = 1 AND b = 1; UniqueKey (NIL). onerow case, every + column is Unique. + +2. The next part is joinrel, this part is most error-prone, we simplified the rules +like below: +1. If the relation's UniqueKey can't be duplicated after join, then is will be + still valid for the join rel. The function we used here is + innerrel_keeps_unique. The basic idea is innerrel.any_col = outer.uk. + +2. If the UnqiueKey can't keep valid via the rule 1, the combination of the + UniqueKey from both sides are valid for sure. We can prove this as: if the + unique exprs from rel1 is duplicated by rel2, the duplicated rows must + contains different unique exprs from rel2. + +More considerations about onerow: +1. If relation with one row and it can't be duplicated, it is still possible + contains mulit_nullvas after outer join. +2. If the either UniqueKey can be duplicated after join, the can get one row + only when both side is one row AND there is no outer join. +3. Whenever the onerow UniqueKey is not a valid any more, we need to convert one + row UniqueKey to normal unique key since we don't store exprs for one-row + relation. get_exprs_from_uniquekeys will be used here. + + +More considerations about multi_nullvals after join: +1. If the original UnqiueKey has multi_nullvals, the final UniqueKey will have + mulit_nullvals in any case. +2. If a unique key doesn't allow mulit_nullvals, after some outer join, it + allows some outer join. + + +3. When we comes to subquery, we need to convert_subquery_unqiuekeys just like +convert_subquery_pathkeys. Only the UniqueKey insides subquery is referenced as +a Var in outer relation will be reused. The relationship between the outerrel.Var +and subquery.exprs is built with outerel->subroot->processed_tlist. + + +4. As for the SRF functions, it will break the uniqueness of uniquekey, However it +is handled in adjust_paths_for_srfs, which happens after the query_planner. so +we will maintain the UniqueKey until there and reset it to NIL at that +places. This can't help on distinct/group by elimination cases but probably help +in some other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is +semantic correctly. + + +5. As for inherit table, we first main the UnqiueKey on childrel as well. But for +partitioned table we need to maintain 2 different kinds of +UnqiueKey. 1). UniqueKey on the parent relation 2). UniqueKey on child +relation for partition wise query. + +Example: +CREATE TABLE p (a int not null, b int not null) partition by list (a); +CREATE TABLE p0 partition of p for values in (1); +CREATE TABLE p1 partition of p for values in (2); + +create unique index p0_b on p0(b); +create unique index p1_b on p1(b); + +Now b is only unique on partition level, so the distinct can't be removed on +the following cases. SELECT DISTINCT b FROM p; + +Another example is SELECT DISTINCT a, b FROM p WHERE a = 1; Since only one +partition is chosen, the UniqueKey on child relation is same as the UniqueKey on +parent relation. + +Another usage of UniqueKey on partition level is it be helpful for +partition-wise join. + +As for the UniqueKey on parent table level, it comes with 2 different ways, +1). the UniqueKey is also derived in UniqueKey index, but the index must be same +in all the related children relations and the unique index must contains +Partition Key in it. Example: + +CREATE UNIQUE INDEX p_ab ON p(a, b); -- where a is the partition key. + +-- Query +SELECT a, b FROM p; the (a, b) is a UniqueKey of p. + +2). If the parent relation has only one childrel, the UniqueKey on childrel is + the UniqueKey on parent as well. diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index 37b4223adb..d52ad59f89 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -579,6 +579,12 @@ set_plain_rel_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) */ check_index_predicates(root, rel); + /* + * Now that we've marked which partial indexes are suitable, we can now + * build the relation's unique keys. + */ + populate_baserel_uniquekeys(root, rel, rel->indexlist); + /* Mark rel with estimated output rows, width, etc */ set_baserel_size_estimates(root, rel); } @@ -1297,6 +1303,8 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, /* Add paths to the append relation. */ add_paths_to_append_rel(root, rel, live_childrels); + if (IS_PARTITIONED_REL(rel)) + populate_partitionedrel_uniquekeys(root, rel, live_childrels); } @@ -2304,6 +2312,8 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, pathkeys, required_outer)); } + convert_subquery_uniquekeys(root, rel, sub_final_rel); + /* If outer rel allows parallelism, do same for partial paths. */ if (rel->consider_parallel && bms_is_empty(required_outer)) { diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c index 57ce97fd53..697ac047fb 100644 --- a/src/backend/optimizer/path/joinpath.c +++ b/src/backend/optimizer/path/joinpath.c @@ -71,13 +71,6 @@ static void consider_parallel_mergejoin(PlannerInfo *root, static void hash_inner_and_outer(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, RelOptInfo *innerrel, JoinType jointype, JoinPathExtraData *extra); -static List *select_mergejoin_clauses(PlannerInfo *root, - RelOptInfo *joinrel, - RelOptInfo *outerrel, - RelOptInfo *innerrel, - List *restrictlist, - JoinType jointype, - bool *mergejoin_allowed); static void generate_mergejoin_paths(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *innerrel, @@ -1927,7 +1920,7 @@ hash_inner_and_outer(PlannerInfo *root, * if it is mergejoinable and involves vars from the two sub-relations * currently of interest. */ -static List * +List * select_mergejoin_clauses(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c index 0dbe2ac726..7271f044ec 100644 --- a/src/backend/optimizer/path/joinrels.c +++ b/src/backend/optimizer/path/joinrels.c @@ -924,6 +924,8 @@ populate_joinrel_with_paths(PlannerInfo *root, RelOptInfo *rel1, /* Apply partitionwise join technique, if possible. */ try_partitionwise_join(root, rel1, rel2, joinrel, sjinfo, restrictlist); + + populate_joinrel_uniquekeys(root, joinrel, rel1, rel2, restrictlist, sjinfo->jointype); } diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index bd9a176d7d..139278829b 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -33,7 +33,6 @@ static bool pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys); static bool matches_boolean_partition_clause(RestrictInfo *rinfo, RelOptInfo *partrel, int partkeycol); -static Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); static bool right_merge_direction(PlannerInfo *root, PathKey *pathkey); @@ -1035,7 +1034,7 @@ convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel, * We need this to ensure that we don't return pathkeys describing values * that are unavailable above the level of the subquery scan. */ -static Var * +Var * find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle) { ListCell *lc; diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c new file mode 100644 index 0000000000..77ed2b2eff --- /dev/null +++ b/src/backend/optimizer/path/uniquekeys.c @@ -0,0 +1,1135 @@ +/*------------------------------------------------------------------------- + * + * uniquekeys.c + * Utilities for matching and building unique keys + * + * Portions Copyright (c) 2020, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/optimizer/path/uniquekeys.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "nodes/makefuncs.h" +#include "nodes/nodeFuncs.h" +#include "optimizer/pathnode.h" +#include "optimizer/paths.h" +#include "optimizer/appendinfo.h" +#include "optimizer/optimizer.h" +#include "optimizer/tlist.h" +#include "rewrite/rewriteManip.h" + + +/* + * This struct is used to help populate_joinrel_uniquekeys. + * + * added_to_joinrel is true if a uniquekey (from outerrel or innerrel) + * has been added to joinrel. + * useful is true if the exprs of the uniquekey still appears in joinrel. + */ +typedef struct UniqueKeyContextData +{ + UniqueKey *uniquekey; + bool added_to_joinrel; + bool useful; +} *UniqueKeyContext; + +static List *initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel); +static bool innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + bool reverse); + +static List *get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals); +static List *get_exprs_from_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + UniqueKey *ukey); +static void add_uniquekey_for_onerow(RelOptInfo *rel); +static bool add_combined_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype); + +/* Used for unique indexes checking for partitioned table */ +static bool index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind); +static IndexOptInfo *simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from); +static bool simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2); +static void adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_index); + +/* Helper function for grouped relation and distinct relation. */ +static void add_uniquekey_from_sortgroups(PlannerInfo *root, + RelOptInfo *rel, + List *sortgroups); + +/* + * populate_baserel_uniquekeys + * Populate 'baserel' uniquekeys list by looking at the rel's unique index + * and baserestrictinfo + */ +void +populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List *indexlist) +{ + ListCell *lc; + List *matched_uniq_indexes = NIL; + + /* Attrs appears in rel->reltarget->exprs. */ + Bitmapset *used_attrs = NULL; + + List *const_exprs = NIL; + List *expr_opfamilies = NIL; + + Assert(baserel->rtekind == RTE_RELATION); + + foreach(lc, indexlist) + { + IndexOptInfo *ind = (IndexOptInfo *) lfirst(lc); + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + matched_uniq_indexes = lappend(matched_uniq_indexes, ind); + } + + if (matched_uniq_indexes == NIL) + return; + + /* Check which attrs is used in baserel->reltarget */ + pull_varattnos((Node *)baserel->reltarget->exprs, baserel->relid, &used_attrs); + + /* Check which attrno is used at a mergeable const filter */ + foreach(lc, baserel->baserestrictinfo) + { + RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); + + if (rinfo->mergeopfamilies == NIL) + continue; + + if (bms_is_empty(rinfo->left_relids)) + { + const_exprs = lappend(const_exprs, get_rightop(rinfo->clause)); + } + else if (bms_is_empty(rinfo->right_relids)) + { + const_exprs = lappend(const_exprs, get_leftop(rinfo->clause)); + } + else + continue; + + expr_opfamilies = lappend(expr_opfamilies, rinfo->mergeopfamilies); + } + + foreach(lc, matched_uniq_indexes) + { + bool multi_nullvals, useful; + List *exprs = get_exprs_from_uniqueindex(lfirst_node(IndexOptInfo, lc), + const_exprs, + expr_opfamilies, + used_attrs, + &useful, + &multi_nullvals); + if (useful) + { + if (exprs == NIL) + { + /* All the columns in Unique Index matched with a restrictinfo */ + add_uniquekey_for_onerow(baserel); + return; + } + baserel->uniquekeys = lappend(baserel->uniquekeys, + makeUniqueKey(exprs, multi_nullvals)); + } + } +} + + +/* + * populate_partitionedrel_uniquekeys + * The UniqueKey on partitionrel comes from 2 cases: + * 1). Only one partition is involved in this query, the unique key can be + * copied to parent rel from childrel. + * 2). There are some unique index which includes partition key and exists + * in all the related partitions. + * We never mind rule 2 if we hit rule 1. + */ + +void +populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels) +{ + ListCell *lc; + List *global_uniq_indexlist = NIL; + RelOptInfo *childrel; + bool is_first = true; + + Assert(IS_PARTITIONED_REL(rel)); + + if (childrels == NIL) + return; + + /* + * If there is only one partition used in this query, the UniqueKey in childrel is + * still valid in parent level, but we need convert the format from child expr to + * parent expr. + */ + if (list_length(childrels) == 1) + { + /* Check for Rule 1 */ + RelOptInfo *childrel = linitial_node(RelOptInfo, childrels); + ListCell *lc; + Assert(childrel->reloptkind == RELOPT_OTHER_MEMBER_REL); + if (relation_is_onerow(childrel)) + { + add_uniquekey_for_onerow(rel); + return; + } + + foreach(lc, childrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + AppendRelInfo *appinfo = find_appinfo_by_child(root, childrel->relid); + List *parent_exprs = NIL; + bool can_reuse = true; + ListCell *lc2; + foreach(lc2, ukey->exprs) + { + Var *var = (Var *)lfirst(lc2); + /* + * If the expr comes from a expression, it is hard to build the expression + * in parent so ignore that case for now. + */ + if(!IsA(var, Var)) + { + can_reuse = false; + break; + } + /* Convert it to parent var */ + parent_exprs = lappend(parent_exprs, find_parent_var(appinfo, var)); + } + if (can_reuse) + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(parent_exprs, + ukey->multi_nullvals)); + } + } + else + { + /* Check for rule 2 */ + childrel = linitial_node(RelOptInfo, childrels); + foreach(lc, childrel->indexlist) + { + IndexOptInfo *ind = lfirst(lc); + IndexOptInfo *modified_index; + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + + /* + * During simple_copy_indexinfo_to_parent, we need to convert var from + * child var to parent var, index on expression is too complex to handle. + * so ignore it for now. + */ + if (ind->indexprs != NIL) + continue; + + modified_index = simple_copy_indexinfo_to_parent(root, rel, ind); + /* + * If the unique index doesn't contain partkey, then it is unique + * on this partition only, so it is useless for us. + */ + if (!index_constains_partkey(rel, modified_index)) + continue; + + global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index); + } + + if (global_uniq_indexlist != NIL) + { + foreach(lc, childrels) + { + RelOptInfo *child = lfirst(lc); + if (is_first) + { + is_first = false; + continue; + } + adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist); + } + /* Now we have a list of unique index which are exactly same on all childrels, + * Set the UniqueKey just like it is non-partition table + */ + populate_baserel_uniquekeys(root, rel, global_uniq_indexlist); + } + } +} + + +/* + * populate_distinctrel_uniquekeys + */ +void +populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel) +{ + /* The unique key before the distinct is still valid. */ + distinctrel->uniquekeys = list_copy(inputrel->uniquekeys); + add_uniquekey_from_sortgroups(root, distinctrel, root->parse->distinctClause); +} + +/* + * populate_grouprel_uniquekeys + */ +void +populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel) + +{ + Query *parse = root->parse; + bool input_ukey_added = false; + ListCell *lc; + + if (relation_is_onerow(inputrel)) + { + add_uniquekey_for_onerow(grouprel); + return; + } + if (parse->groupingSets) + return; + + /* A Normal group by without grouping set. */ + if (parse->groupClause) + { + /* + * Current even the groupby clause is Unique already, but if query has aggref + * We have to create grouprel still. To keep the UnqiueKey short, we will check + * the UniqueKey of input_rel still valid, if so we reuse it. + */ + foreach(lc, inputrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (list_is_subset(ukey->exprs, grouprel->reltarget->exprs)) + { + grouprel->uniquekeys = lappend(grouprel->uniquekeys, + ukey); + input_ukey_added = true; + } + } + if (!input_ukey_added) + /* + * group by clause must be a super-set of grouprel->reltarget->exprs except the + * aggregation expr, so if such exprs is unique already, no bother to generate + * new uniquekey for group by exprs. + */ + add_uniquekey_from_sortgroups(root, + grouprel, + root->parse->groupClause); + } + else + /* It has aggregation but without a group by, so only one row returned */ + add_uniquekey_for_onerow(grouprel); +} + +/* + * simple_copy_uniquekeys + * Using a function for the one-line code makes us easy to check where we simply + * copied the uniquekey. + */ +void +simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel) +{ + newrel->uniquekeys = oldrel->uniquekeys; +} + +/* + * populate_unionrel_uniquekeys + */ +void +populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel) +{ + ListCell *lc; + List *exprs = NIL; + + Assert(unionrel->uniquekeys == NIL); + + foreach(lc, unionrel->reltarget->exprs) + { + exprs = lappend(exprs, lfirst(lc)); + } + + if (exprs == NIL) + /* SQL: select union select; is valid, we need to handle it here. */ + add_uniquekey_for_onerow(unionrel); + else + unionrel->uniquekeys = lappend(unionrel->uniquekeys, + makeUniqueKey(exprs,false)); + +} + +/* + * populate_joinrel_uniquekeys + * + * populate uniquekeys for joinrel. We will check each relation to see if its + * UniqueKey is still valid via innerrel_keeps_unique, if so, we add it to + * joinrel. The multi_nullvals field will be changed to true for some outer + * join cases and one-row UniqueKey needs to be converted to normal UniqueKey + * for the same case as well. + * For the uniquekey in either baserel which can't be unique after join, we still + * check to see if combination of UniqueKeys from both side is still useful for us. + * if yes, we add it to joinrel as well. + */ +void +populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *outerrel, RelOptInfo *innerrel, + List *restrictlist, JoinType jointype) +{ + ListCell *lc, *lc2; + List *clause_list = NIL; + List *outerrel_ukey_ctx; + List *innerrel_ukey_ctx; + bool inner_onerow, outer_onerow; + bool mergejoin_allowed; + + /* Care about the outerrel relation only for SEMI/ANTI join */ + if (jointype == JOIN_SEMI || jointype == JOIN_ANTI) + { + foreach(lc, outerrel->uniquekeys) + { + UniqueKey *uniquekey = lfirst_node(UniqueKey, lc); + if (list_is_subset(uniquekey->exprs, joinrel->reltarget->exprs)) + joinrel->uniquekeys = lappend(joinrel->uniquekeys, uniquekey); + } + return; + } + + Assert(jointype == JOIN_LEFT || jointype == JOIN_FULL || jointype == JOIN_INNER); + + /* Fast path */ + if (innerrel->uniquekeys == NIL || outerrel->uniquekeys == NIL) + return; + + inner_onerow = relation_is_onerow(innerrel); + outer_onerow = relation_is_onerow(outerrel); + + outerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(outerrel); + innerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(innerrel); + + clause_list = select_mergejoin_clauses(root, joinrel, outerrel, innerrel, + restrictlist, jointype, + &mergejoin_allowed); + + if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true /* reverse */)) + { + bool outer_impact = jointype == JOIN_FULL; + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + /* Outer relation has one row, and the unique key is not duplicated after join, + * the joinrel will still has one row unless the jointype == JOIN_FULL. + */ + if (outer_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (outer_onerow) + { + /* + * The onerow outerrel becomes multi rows and multi_nullvals + * will be changed to true. We also need to set the exprs correctly since it + * can't be NIL any more. + */ + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, outerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Change multi_nullvals to true due to the full join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, true)); + else + /* Just reuse it */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + } + ctx->added_to_joinrel = true; + } + } + + if (innerrel_keeps_unique(root, outerrel, innerrel, clause_list, false)) + { + bool outer_impact = jointype == JOIN_FULL || jointype == JOIN_LEFT;; + + foreach(lc, innerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + if (inner_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (inner_onerow) + { + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, innerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Need to change multi_nullvals to true due to the outer join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, + true)); + else + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + + } + ctx->added_to_joinrel = true; + } + } + + /* + * The combination of the UniqueKey from both sides is unique as well regardless + * of join type, but no bother to add it if its subset has been added to joinrel + * already or it is not useful for the joinrel. + */ + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx1 = (UniqueKeyContext) lfirst(lc); + if (ctx1->added_to_joinrel || !ctx1->useful) + continue; + foreach(lc2, innerrel_ukey_ctx) + { + UniqueKeyContext ctx2 = (UniqueKeyContext) lfirst(lc2); + if (ctx2->added_to_joinrel || !ctx2->useful) + continue; + if (add_combined_uniquekey(root, joinrel, outerrel, innerrel, + ctx1->uniquekey, ctx2->uniquekey, + jointype)) + /* If we set a onerow UniqueKey to joinrel, we don't need other. */ + return; + } + } +} + + +/* + * convert_subquery_uniquekeys + * + * Covert the UniqueKey in subquery to outer relation. + */ +void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel) +{ + ListCell *lc; + + if (sub_final_rel->uniquekeys == NIL) + return; + + if (relation_is_onerow(sub_final_rel)) + { + add_uniquekey_for_onerow(currel); + return; + } + + Assert(currel->subroot != NULL); + + foreach(lc, sub_final_rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + ListCell *lc; + List *exprs = NIL; + bool ukey_useful = true; + + /* One row case is handled above */ + Assert(ukey->exprs != NIL); + foreach(lc, ukey->exprs) + { + Var *var; + TargetEntry *tle = tlist_member(lfirst(lc), + currel->subroot->processed_tlist); + if (tle == NULL) + { + ukey_useful = false; + break; + } + var = find_var_for_subquery_tle(currel, tle); + if (var == NULL) + { + ukey_useful = false; + break; + } + exprs = lappend(exprs, var); + } + + if (ukey_useful) + currel->uniquekeys = lappend(currel->uniquekeys, + makeUniqueKey(exprs, + ukey->multi_nullvals)); + + } +} + +/* + * innerrel_keeps_unique + * + * Check if Unique key of the innerrel is valid after join. innerrel's UniqueKey + * will be still valid if innerrel's any-column mergeop outrerel's uniquekey + * exists in clause_list. + * + * Note: the clause_list must be a list of mergeable restrictinfo already. + */ +static bool +innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *clause_list, + bool reverse) +{ + ListCell *lc, *lc2, *lc3; + + if (outerrel->uniquekeys == NIL || innerrel->uniquekeys == NIL) + return false; + + /* Check if there is outerrel's uniquekey in mergeable clause. */ + foreach(lc, outerrel->uniquekeys) + { + List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs; + bool clauselist_matchs_all_exprs = true; + foreach(lc2, outer_uq_exprs) + { + Node *outer_uq_expr = lfirst(lc2); + bool find_uq_expr_in_clauselist = false; + foreach(lc3, clause_list) + { + RestrictInfo *rinfo = lfirst_node(RestrictInfo, lc3); + Node *outer_expr; + if (reverse) + outer_expr = rinfo->outer_is_left ? get_rightop(rinfo->clause) : get_leftop(rinfo->clause); + else + outer_expr = rinfo->outer_is_left ? get_leftop(rinfo->clause) : get_rightop(rinfo->clause); + if (equal(outer_expr, outer_uq_expr)) + { + find_uq_expr_in_clauselist = true; + break; + } + } + if (!find_uq_expr_in_clauselist) + { + /* No need to check the next exprs in the current uniquekey */ + clauselist_matchs_all_exprs = false; + break; + } + } + + if (clauselist_matchs_all_exprs) + return true; + } + return false; +} + + +/* + * relation_is_onerow + * Check if it is a one-row relation by checking UniqueKey. + */ +bool +relation_is_onerow(RelOptInfo *rel) +{ + UniqueKey *ukey; + if (rel->uniquekeys == NIL) + return false; + ukey = linitial_node(UniqueKey, rel->uniquekeys); + return ukey->exprs == NIL && list_length(rel->uniquekeys) == 1; +} + +/* + * relation_has_uniquekeys_for + * Returns true if we have proofs that 'rel' cannot return multiple rows with + * the same values in each of 'exprs'. Otherwise returns false. + */ +bool +relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel, + List *exprs, bool allow_multinulls) +{ + ListCell *lc; + + /* + * For UniqueKey->onerow case, the uniquekey->exprs is empty as well + * so we can't rely on list_is_subset to handle this special cases + */ + if (exprs == NIL) + return false; + + foreach(lc, rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (ukey->multi_nullvals && !allow_multinulls) + continue; + if (list_is_subset(ukey->exprs, exprs)) + return true; + } + return false; +} + + +/* + * get_exprs_from_uniqueindex + * + * Return a list of exprs which is unique. set useful to false if this + * unique index is not useful for us. + */ +static List * +get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals) +{ + List *exprs = NIL; + ListCell *indexpr_item; + int c = 0; + + *useful = true; + *multi_nullvals = false; + + indexpr_item = list_head(unique_index->indexprs); + for(c = 0; c < unique_index->ncolumns; c++) + { + int attr = unique_index->indexkeys[c]; + Expr *expr; + bool matched_const = false; + ListCell *lc1, *lc2; + + if(attr > 0) + { + expr = list_nth_node(TargetEntry, unique_index->indextlist, c)->expr; + } + else if (attr == 0) + { + /* Expression index */ + expr = lfirst(indexpr_item); + indexpr_item = lnext(unique_index->indexprs, indexpr_item); + } + else /* attr < 0 */ + { + /* Index on system column is not supported */ + Assert(false); + } + + /* + * Check index_col = Const case with regarding to opfamily checking + * If we can remove the index_col from the final UniqueKey->exprs. + */ + forboth(lc1, const_exprs, lc2, const_expr_opfamilies) + { + if (list_member_oid((List *)lfirst(lc2), unique_index->opfamily[c]) + && match_index_to_operand((Node *) lfirst(lc1), c, unique_index)) + { + matched_const = true; + break; + } + } + + if (matched_const) + continue; + + /* Check if the indexed expr is used in rel */ + if (attr > 0) + { + /* + * Normal Indexed column, if the col is not used, then the index is useless + * for uniquekey. + */ + attr -= FirstLowInvalidHeapAttributeNumber; + + if (!bms_is_member(attr, used_varattrs)) + { + *useful = false; + break; + } + } + else if (!list_member(unique_index->rel->reltarget->exprs, expr)) + { + /* Expression index but the expression is not used in rel */ + *useful = false; + break; + } + + /* check not null property. */ + if (attr == 0) + { + /* We never know if a expression yields null or not */ + *multi_nullvals = true; + } + else if (!bms_is_member(attr, unique_index->rel->notnullattrs) + && !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber, + unique_index->rel->notnullattrs)) + { + *multi_nullvals = true; + } + + exprs = lappend(exprs, expr); + } + return exprs; +} + + +/* + * add_uniquekey_for_onerow + * If we are sure that the relation only returns one row, then all the columns + * are unique. However we don't need to create UniqueKey for every column, we + * just set exprs = NIL and overwrites all the other UniqueKey on this RelOptInfo + * since this one has strongest semantics. + */ +void +add_uniquekey_for_onerow(RelOptInfo *rel) +{ + /* + * We overwrite the previous UniqueKey on purpose since this one has the + * strongest semantic. + */ + rel->uniquekeys = list_make1(makeUniqueKey(NIL, false)); +} + + +/* + * initililze_uniquecontext_for_joinrel + * Return a List of UniqueKeyContext for an inputrel + */ +static List * +initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel) +{ + List *res = NIL; + ListCell *lc; + foreach(lc, inputrel->uniquekeys) + { + UniqueKeyContext context; + context = palloc(sizeof(struct UniqueKeyContextData)); + context->uniquekey = lfirst_node(UniqueKey, lc); + context->added_to_joinrel = false; + context->useful = true; + res = lappend(res, context); + } + return res; +} + + +/* + * get_exprs_from_uniquekey + * Unify the way of get List of exprs from a one-row UniqueKey or + * normal UniqueKey. for the onerow case, every expr in rel1 is a valid + * UniqueKey. Return a List of exprs. + * + * rel1: The relation which you want to get the exprs. + * ukey: The UniqueKey you want to get the exprs. + */ +static List * +get_exprs_from_uniquekey(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *rel1, UniqueKey *ukey) +{ + ListCell *lc; + bool onerow = rel1 != NULL && relation_is_onerow(rel1); + + List *res = NIL; + Assert(onerow || ukey); + if (onerow) + { + /* Only cares about the exprs still exist in joinrel */ + foreach(lc, joinrel->reltarget->exprs) + { + Bitmapset *relids = pull_varnos(root, lfirst(lc)); + if (bms_is_subset(relids, rel1->relids)) + { + res = lappend(res, list_make1(lfirst(lc))); + } + } + } + else + { + res = list_make1(ukey->exprs); + } + return res; +} + +/* + * Partitioned table Unique Keys. + * The partition table unique key is maintained as: + * 1. The index must be unique as usual. + * 2. The index must contains partition key. + * 3. The index must exist on all the child rel. see simple_indexinfo_equal for + * how we compare it. + */ + +/* + * index_constains_partkey + * return true if the index contains the partiton key. + */ +static bool +index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind) +{ + ListCell *lc; + int i; + Assert(IS_PARTITIONED_REL(partrel)); + Assert(partrel->part_scheme->partnatts > 0); + + for(i = 0; i < partrel->part_scheme->partnatts; i++) + { + Node *part_expr = linitial(partrel->partexprs[i]); + bool found_in_index = false; + foreach(lc, ind->indextlist) + { + Expr *index_expr = lfirst_node(TargetEntry, lc)->expr; + if (equal(index_expr, part_expr)) + { + found_in_index = true; + break; + } + } + if (!found_in_index) + return false; + } + return true; +} + +/* + * simple_indexinfo_equal + * + * Used to check if the 2 index is same as each other. The index here + * is COPIED from childrel and did some tiny changes(see + * simple_copy_indexinfo_to_parent) + */ +static bool +simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2) +{ + Size oid_cmp_len = sizeof(Oid) * ind1->ncolumns; + + return ind1->ncolumns == ind2->ncolumns && + ind1->unique == ind2->unique && + memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0 && + memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0 && + memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0 && + memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0 && + equal(get_tlist_exprs(ind1->indextlist, true), + get_tlist_exprs(ind2->indextlist, true)); +} + + +/* + * The below macros are used for simple_copy_indexinfo_to_parent which is so + * customized that I don't want to put it to copyfuncs.c. So copy it here. + */ +#define COPY_POINTER_FIELD(fldname, sz) \ + do { \ + Size _size = (sz); \ + newnode->fldname = palloc(_size); \ + memcpy(newnode->fldname, from->fldname, _size); \ + } while (0) + +#define COPY_NODE_FIELD(fldname) \ + (newnode->fldname = copyObjectImpl(from->fldname)) + +#define COPY_SCALAR_FIELD(fldname) \ + (newnode->fldname = from->fldname) + + +/* + * simple_copy_indexinfo_to_parent (from partition) + * Copy the IndexInfo from child relation to parent relation with some modification, + * which is used to test: + * 1. If the same index exists in all the childrels. + * 2. If the parentrel->reltarget/basicrestrict info matches this index. + */ +static IndexOptInfo * +simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from) +{ + IndexOptInfo *newnode = makeNode(IndexOptInfo); + AppendRelInfo *appinfo = find_appinfo_by_child(root, from->rel->relid); + ListCell *lc; + int idx = 0; + + COPY_SCALAR_FIELD(ncolumns); + COPY_SCALAR_FIELD(nkeycolumns); + COPY_SCALAR_FIELD(unique); + COPY_SCALAR_FIELD(immediate); + /* We just need to know if it is NIL or not */ + COPY_SCALAR_FIELD(indpred); + COPY_SCALAR_FIELD(predOK); + COPY_POINTER_FIELD(indexkeys, from->ncolumns * sizeof(int)); + COPY_POINTER_FIELD(indexcollations, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opfamily, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opcintype, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(sortopfamily, from->ncolumns * sizeof(Oid)); + COPY_NODE_FIELD(indextlist); + + /* Convert index exprs on child expr to expr on parent */ + foreach(lc, newnode->indextlist) + { + TargetEntry *tle = lfirst_node(TargetEntry, lc); + /* Index on expression is ignored */ + Assert(IsA(tle->expr, Var)); + tle->expr = (Expr *) find_parent_var(appinfo, (Var *) tle->expr); + newnode->indexkeys[idx] = castNode(Var, tle->expr)->varattno; + idx++; + } + newnode->rel = parentrel; + return newnode; +} + +/* + * adjust_partition_unique_indexlist + * + * global_unique_indexes: At the beginning, it contains the copy & modified + * unique index from the first partition. And then check if each index in it still + * exists in the following partitions. If no, remove it. at last, it has an + * index list which exists in all the partitions. + */ +static void +adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_indexes) +{ + ListCell *lc, *lc2; + foreach(lc, *global_unique_indexes) + { + IndexOptInfo *g_ind = lfirst_node(IndexOptInfo, lc); + bool found_in_child = false; + + foreach(lc2, childrel->indexlist) + { + IndexOptInfo *p_ind = lfirst_node(IndexOptInfo, lc2); + IndexOptInfo *p_ind_copy; + if (!p_ind->unique || !p_ind->immediate || + (p_ind->indpred != NIL && !p_ind->predOK)) + continue; + p_ind_copy = simple_copy_indexinfo_to_parent(root, parentrel, p_ind); + if (simple_indexinfo_equal(p_ind_copy, g_ind)) + { + found_in_child = true; + break; + } + } + if (!found_in_child) + /* The index doesn't exist in childrel, remove it from global_unique_indexes */ + *global_unique_indexes = foreach_delete_current(*global_unique_indexes, lc); + } +} + +/* Helper function for groupres/distinctrel */ +static void +add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgroups) +{ + Query *parse = root->parse; + List *exprs; + + /* + * XXX: If there are some vars which is not in current levelsup, the semantic is + * imprecise, should we avoid it or not? levelsup = 1 is just a demo, maybe we need to + * check every level other than 0, if so, looks we have to write another + * pull_var_walker. + */ + List *upper_vars = pull_vars_of_level((Node*)sortgroups, 1); + + if (upper_vars != NIL) + return; + + exprs = get_sortgrouplist_exprs(sortgroups, parse->targetList); + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(exprs, + false /* sortgroupclause can't be multi_nullvals */)); +} + + +/* + * add_combined_uniquekey + * The combination of both UniqueKeys is a valid UniqueKey for joinrel no matter + * the jointype. + */ +bool +add_combined_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype) +{ + + ListCell *lc1, *lc2; + + /* Either side has multi_nullvals or we have outer join, + * the combined UniqueKey has multi_nullvals */ + bool multi_nullvals = outer_ukey->multi_nullvals || + inner_ukey->multi_nullvals || IS_OUTER_JOIN(jointype); + + /* The only case we can get onerow joinrel after join */ + if (relation_is_onerow(outer_rel) + && relation_is_onerow(inner_rel) + && jointype == JOIN_INNER) + { + add_uniquekey_for_onerow(joinrel); + return true; + } + + foreach(lc1, get_exprs_from_uniquekey(root, joinrel, outer_rel, outer_ukey)) + { + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, inner_rel, inner_ukey)) + { + List *exprs = list_concat_copy(lfirst_node(List, lc1), lfirst_node(List, lc2)); + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(exprs, + multi_nullvals)); + } + } + return false; +} diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index 424d25cbd5..8d8e493f5c 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -2387,6 +2387,8 @@ grouping_planner(PlannerInfo *root, bool inheritance_update, add_path(final_rel, path); } + simple_copy_uniquekeys(current_rel, final_rel); + /* * Generate partial paths for final_rel, too, if outer query levels might * be able to make use of them. @@ -3899,6 +3901,8 @@ create_grouping_paths(PlannerInfo *root, } set_cheapest(grouped_rel); + + populate_grouprel_uniquekeys(root, grouped_rel, input_rel); return grouped_rel; } @@ -4621,7 +4625,7 @@ create_window_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(window_rel); - + simple_copy_uniquekeys(input_rel, window_rel); return window_rel; } @@ -4935,7 +4939,7 @@ create_distinct_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(distinct_rel); - + populate_distinctrel_uniquekeys(root, input_rel, distinct_rel); return distinct_rel; } @@ -5196,6 +5200,8 @@ create_ordered_paths(PlannerInfo *root, */ Assert(ordered_rel->pathlist != NIL); + simple_copy_uniquekeys(input_rel, ordered_rel); + return ordered_rel; } @@ -6073,6 +6079,9 @@ adjust_paths_for_srfs(PlannerInfo *root, RelOptInfo *rel, if (list_length(targets) == 1) return; + /* UniqueKey is not valid after handling the SRF. */ + rel->uniquekeys = NIL; + /* * Stack SRF-evaluation nodes atop each path for the rel. * diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c index becdcbb872..b7626545bf 100644 --- a/src/backend/optimizer/prep/prepunion.c +++ b/src/backend/optimizer/prep/prepunion.c @@ -689,6 +689,8 @@ generate_union_paths(SetOperationStmt *op, PlannerInfo *root, /* Undo effects of possibly forcing tuple_fraction to 0 */ root->tuple_fraction = save_fraction; + /* Add the UniqueKeys */ + populate_unionrel_uniquekeys(root, result_rel); return result_rel; } diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c index 86922a273c..6817c9c787 100644 --- a/src/backend/optimizer/util/appendinfo.c +++ b/src/backend/optimizer/util/appendinfo.c @@ -746,3 +746,47 @@ find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos) } return appinfos; } + +/* + * find_appinfo_by_child + * + */ +AppendRelInfo * +find_appinfo_by_child(PlannerInfo *root, Index child_index) +{ + ListCell *lc; + foreach(lc, root->append_rel_list) + { + AppendRelInfo *appinfo = lfirst_node(AppendRelInfo, lc); + if (appinfo->child_relid == child_index) + return appinfo; + } + elog(ERROR, "parent relation cant be found"); + return NULL; +} + +/* + * find_parent_var + * + */ +Var * +find_parent_var(AppendRelInfo *appinfo, Var *child_var) +{ + ListCell *lc; + Var *res = NULL; + Index attno = 1; + foreach(lc, appinfo->translated_vars) + { + Node *child_node = lfirst(lc); + if (equal(child_node, child_var)) + { + res = copyObject(child_var); + res->varattno = attno; + res->varno = appinfo->parent_relid; + } + attno++; + } + if (res == NULL) + elog(ERROR, "parent var can't be found."); + return res; +} diff --git a/src/backend/optimizer/util/inherit.c b/src/backend/optimizer/util/inherit.c index be1c9ddd96..3eec1f4d74 100644 --- a/src/backend/optimizer/util/inherit.c +++ b/src/backend/optimizer/util/inherit.c @@ -736,6 +736,7 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, { Node *onecq = (Node *) lfirst(lc2); bool pseudoconstant; + RestrictInfo *child_rinfo; /* check for pseudoconstant (no Vars or volatile functions) */ pseudoconstant = @@ -747,14 +748,15 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, root->hasPseudoConstantQuals = true; } /* reconstitute RestrictInfo with appropriate properties */ - childquals = lappend(childquals, - make_restrictinfo(root, - (Expr *) onecq, - rinfo->is_pushed_down, - rinfo->outerjoin_delayed, - pseudoconstant, - rinfo->security_level, - NULL, NULL, NULL)); + child_rinfo = make_restrictinfo(root, + (Expr *) onecq, + rinfo->is_pushed_down, + rinfo->outerjoin_delayed, + pseudoconstant, + rinfo->security_level, + NULL, NULL, NULL); + child_rinfo->mergeopfamilies = rinfo->mergeopfamilies; + childquals = lappend(childquals, child_rinfo); /* track minimum security level among child quals */ cq_min_security = Min(cq_min_security, rinfo->security_level); } diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 48a7ebfe45..4fe1824eb0 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -16,6 +16,7 @@ #include "nodes/execnodes.h" #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" extern A_Expr *makeA_Expr(A_Expr_Kind kind, List *name, @@ -106,4 +107,6 @@ extern GroupingSet *makeGroupingSet(GroupingSetKind kind, List *content, int loc extern VacuumRelation *makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols); +extern UniqueKey* makeUniqueKey(List *exprs, bool multi_nullvals); + #endif /* MAKEFUNC_H */ diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index e22df890ef..3320273ac1 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -265,6 +265,7 @@ typedef enum NodeTag T_EquivalenceMember, T_PathKey, T_PathTarget, + T_UniqueKey, T_RestrictInfo, T_IndexClause, T_PlaceHolderVar, diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index 0d61f04d27..80561b1003 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -739,6 +739,7 @@ typedef struct RelOptInfo QualCost baserestrictcost; /* cost of evaluating the above */ Index baserestrict_min_security; /* min security_level found in * baserestrictinfo */ + List *uniquekeys; /* List of UniqueKey */ List *joininfo; /* RestrictInfo structures for join clauses * involving this rel */ bool has_eclass_joins; /* T means joininfo is incomplete */ @@ -1059,6 +1060,28 @@ typedef struct PathKey } PathKey; +/* + * UniqueKey + * + * Represents the unique properties held by a RelOptInfo. + * + * exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL + * is a special case of UniqueKey, which means there is only 1 row in that + * relation. + * multi_nullvals: true means multi null values may exist in these exprs, so the + * uniqueness is not guaranteed in this case. This field is necessary for + * remove_useless_join & reduce_unique_semijoins where we don't mind these + * duplicated NULL values. It is set to true for 2 cases. One is a unique key + * from a unique index but the related column is nullable. The other one is for + * outer join. see populate_joinrel_uniquekeys for detail. + */ +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + /* * PathTarget * @@ -2496,7 +2519,7 @@ typedef enum * * flags indicating what kinds of grouping are possible. * partial_costs_set is true if the agg_partial_costs and agg_final_costs - * have been initialized. + * have been initialized. * agg_partial_costs gives partial aggregation costs. * agg_final_costs gives finalization costs. * target_parallel_safe is true if target is parallel safe. @@ -2526,8 +2549,8 @@ typedef struct * limit_tuples is an estimated bound on the number of output tuples, * or -1 if no LIMIT or couldn't estimate. * count_est and offset_est are the estimated values of the LIMIT and OFFSET - * expressions computed by preprocess_limit() (see comments for - * preprocess_limit() for more information). + * expressions computed by preprocess_limit() (see comments for + * preprocess_limit() for more information). */ typedef struct { diff --git a/src/include/nodes/pg_list.h b/src/include/nodes/pg_list.h index 404e03f132..85c65e910c 100644 --- a/src/include/nodes/pg_list.h +++ b/src/include/nodes/pg_list.h @@ -566,6 +566,16 @@ extern pg_nodiscard List *list_delete_first(List *list); extern pg_nodiscard List *list_delete_last(List *list); extern pg_nodiscard List *list_delete_nth_cell(List *list, int n); extern pg_nodiscard List *list_delete_cell(List *list, ListCell *cell); +extern bool list_is_subset(const List *members, const List *target); + +extern List *list_delete(List *list, void *datum); +extern List *list_delete_ptr(List *list, void *datum); +extern List *list_delete_int(List *list, int datum); +extern List *list_delete_oid(List *list, Oid datum); +extern List *list_delete_first(List *list); +extern List *list_delete_last(List *list); +extern List *list_delete_nth_cell(List *list, int n); +extern List *list_delete_cell(List *list, ListCell *cell); extern List *list_union(const List *list1, const List *list2); extern List *list_union_ptr(const List *list1, const List *list2); diff --git a/src/include/optimizer/appendinfo.h b/src/include/optimizer/appendinfo.h index 4cbf8c26cc..43b8d05f69 100644 --- a/src/include/optimizer/appendinfo.h +++ b/src/include/optimizer/appendinfo.h @@ -32,4 +32,7 @@ extern Relids adjust_child_relids_multilevel(PlannerInfo *root, Relids relids, extern AppendRelInfo **find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos); +extern AppendRelInfo *find_appinfo_by_child(PlannerInfo *root, Index child_index); +extern Var *find_parent_var(AppendRelInfo *appinfo, Var *child_var); + #endif /* APPENDINFO_H */ diff --git a/src/include/optimizer/optimizer.h b/src/include/optimizer/optimizer.h index d587952b7d..843aafa51e 100644 --- a/src/include/optimizer/optimizer.h +++ b/src/include/optimizer/optimizer.h @@ -23,6 +23,7 @@ #define OPTIMIZER_H #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" /* Test if an expression node represents a SRF call. Beware multiple eval! */ #define IS_SRF_CALL(node) \ @@ -168,6 +169,7 @@ extern TargetEntry *get_sortgroupref_tle(Index sortref, List *targetList); extern TargetEntry *get_sortgroupclause_tle(SortGroupClause *sgClause, List *targetList); +extern Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); extern Node *get_sortgroupclause_expr(SortGroupClause *sgClause, List *targetList); extern List *get_sortgrouplist_exprs(List *sgClauses, diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h index 035d3e1206..1adf99c9ee 100644 --- a/src/include/optimizer/paths.h +++ b/src/include/optimizer/paths.h @@ -247,5 +247,48 @@ extern PathKey *make_canonical_pathkey(PlannerInfo *root, int strategy, bool nulls_first); extern void add_paths_to_append_rel(PlannerInfo *root, RelOptInfo *rel, List *live_childrels); +extern List *select_mergejoin_clauses(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + JoinType jointype, + bool *mergejoin_allowed); + +/* + * uniquekeys.c + * Utilities for matching and building unique keys + */ +extern void populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List* unique_index_list); +extern void populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels); +extern void populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel); +extern void populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel); +extern void populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel); +extern void simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel); +extern void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel); +extern void populate_joinrel_uniquekeys(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + RelOptInfo *rel2, + List *restrictlist, + JoinType jointype); + +extern bool relation_has_uniquekeys_for(PlannerInfo *root, + RelOptInfo *rel, + List *exprs, + bool allow_multinulls); +extern bool relation_is_onerow(RelOptInfo *rel); #endif /* PATHS_H */ -- 2.26.2 --yh7cw345mbttadjz Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v38-0003-Extend-UniqueKeys.patch" ^ permalink raw reply [nested|flat] 91+ messages in thread
* [PATCH v36 2/6] Introduce UniqueKey attributes on RelOptInfo struct. @ 2020-05-11 07:50 一挃 <[email protected]> 0 siblings, 0 replies; 91+ messages in thread From: 一挃 @ 2020-05-11 07:50 UTC (permalink / raw) UniqueKey is a set of exprs on RelOptInfo which represents the exprs will be unique on the given RelOptInfo. You can see README.uniquekey for more information. --- src/backend/nodes/copyfuncs.c | 13 + src/backend/nodes/list.c | 31 + src/backend/nodes/makefuncs.c | 13 + src/backend/nodes/outfuncs.c | 11 + src/backend/nodes/readfuncs.c | 10 + src/backend/optimizer/path/Makefile | 3 +- src/backend/optimizer/path/README.uniquekey | 131 +++ src/backend/optimizer/path/allpaths.c | 10 + src/backend/optimizer/path/joinpath.c | 9 +- src/backend/optimizer/path/joinrels.c | 2 + src/backend/optimizer/path/pathkeys.c | 3 +- src/backend/optimizer/path/uniquekeys.c | 1131 +++++++++++++++++++ src/backend/optimizer/plan/planner.c | 13 +- src/backend/optimizer/prep/prepunion.c | 2 + src/backend/optimizer/util/appendinfo.c | 44 + src/backend/optimizer/util/inherit.c | 16 +- src/include/nodes/makefuncs.h | 3 + src/include/nodes/nodes.h | 1 + src/include/nodes/pathnodes.h | 29 +- src/include/nodes/pg_list.h | 2 + src/include/optimizer/appendinfo.h | 3 + src/include/optimizer/optimizer.h | 2 + src/include/optimizer/paths.h | 43 + 23 files changed, 1502 insertions(+), 23 deletions(-) create mode 100644 src/backend/optimizer/path/README.uniquekey create mode 100644 src/backend/optimizer/path/uniquekeys.c diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 89c409de66..1f50400fd2 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2273,6 +2273,16 @@ _copyPathKey(const PathKey *from) return newnode; } +static UniqueKey * +_copyUniqueKey(const UniqueKey *from) +{ + UniqueKey *newnode = makeNode(UniqueKey); + + COPY_NODE_FIELD(exprs); + COPY_SCALAR_FIELD(multi_nullvals); + + return newnode; +} /* * _copyRestrictInfo */ @@ -5152,6 +5162,9 @@ copyObjectImpl(const void *from) case T_PathKey: retval = _copyPathKey(from); break; + case T_UniqueKey: + retval = _copyUniqueKey(from); + break; case T_RestrictInfo: retval = _copyRestrictInfo(from); break; diff --git a/src/backend/nodes/list.c b/src/backend/nodes/list.c index 80fa8c84e4..a7a99b70f2 100644 --- a/src/backend/nodes/list.c +++ b/src/backend/nodes/list.c @@ -687,6 +687,37 @@ list_member_oid(const List *list, Oid datum) return false; } +/* + * return true iff every entry in "members" list is also present + * in the "target" list. + */ +bool +list_is_subset(const List *members, const List *target) +{ + const ListCell *lc1, *lc2; + + Assert(IsPointerList(members)); + Assert(IsPointerList(target)); + check_list_invariants(members); + check_list_invariants(target); + + foreach(lc1, members) + { + bool found = false; + foreach(lc2, target) + { + if (equal(lfirst(lc1), lfirst(lc2))) + { + found = true; + break; + } + } + if (!found) + return false; + } + return true; +} + /* * Delete the n'th cell (counting from 0) in list. * diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 49de285f01..646cf7c9a1 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -814,3 +814,16 @@ makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols) v->va_cols = va_cols; return v; } + + +/* + * makeUniqueKey + */ +UniqueKey* +makeUniqueKey(List *exprs, bool multi_nullvals) +{ + UniqueKey * ukey = makeNode(UniqueKey); + ukey->exprs = exprs; + ukey->multi_nullvals = multi_nullvals; + return ukey; +} diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index e2f177515d..c3a9632992 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2428,6 +2428,14 @@ _outPathKey(StringInfo str, const PathKey *node) WRITE_BOOL_FIELD(pk_nulls_first); } +static void +_outUniqueKey(StringInfo str, const UniqueKey *node) +{ + WRITE_NODE_TYPE("UNIQUEKEY"); + WRITE_NODE_FIELD(exprs); + WRITE_BOOL_FIELD(multi_nullvals); +} + static void _outPathTarget(StringInfo str, const PathTarget *node) { @@ -4127,6 +4135,9 @@ outNode(StringInfo str, const void *obj) case T_PathKey: _outPathKey(str, obj); break; + case T_UniqueKey: + _outUniqueKey(str, obj); + break; case T_PathTarget: _outPathTarget(str, obj); break; diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index 42050ab719..3a18571d0c 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -452,6 +452,14 @@ _readSetOperationStmt(void) READ_DONE(); } +static UniqueKey * +_readUniqueKey(void) +{ + READ_LOCALS(UniqueKey); + READ_NODE_FIELD(exprs); + READ_BOOL_FIELD(multi_nullvals); + READ_DONE(); +} /* * Stuff from primnodes.h. @@ -2656,6 +2664,8 @@ parseNodeString(void) return_value = _readCommonTableExpr(); else if (MATCH("SETOPERATIONSTMT", 16)) return_value = _readSetOperationStmt(); + else if (MATCH("UNIQUEKEY", 9)) + return_value = _readUniqueKey(); else if (MATCH("ALIAS", 5)) return_value = _readAlias(); else if (MATCH("RANGEVAR", 8)) diff --git a/src/backend/optimizer/path/Makefile b/src/backend/optimizer/path/Makefile index 1e199ff66f..7b9820c25f 100644 --- a/src/backend/optimizer/path/Makefile +++ b/src/backend/optimizer/path/Makefile @@ -21,6 +21,7 @@ OBJS = \ joinpath.o \ joinrels.o \ pathkeys.o \ - tidpath.o + tidpath.o \ + uniquekeys.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/optimizer/path/README.uniquekey b/src/backend/optimizer/path/README.uniquekey new file mode 100644 index 0000000000..5eac761995 --- /dev/null +++ b/src/backend/optimizer/path/README.uniquekey @@ -0,0 +1,131 @@ +1. What is UniqueKey? +We can think UniqueKey is a set of exprs for a RelOptInfo, which we are insure +that doesn't yields same result among all the rows. The simplest UniqueKey +format is primary key. + +However we define the UnqiueKey as below. + +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + +exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL +is a special case of UniqueKey, which means there is only one row in that +relation.it has a stronger semantic than others. like SELECT uk FROM t; uk is +normal unique key and may have different values. SELECT colx FROM t WHERE uk = +const. colx is unique AND we have only 1 value. This field can used for +innerrel_is_unique. this logic is handled specially in add_uniquekey_for_onerow +function. + +multi_nullvals: true means multi null values may exist in these exprs, so the +uniqueness is not guaranteed in this case. This field is necessary for +remove_useless_join & reduce_unique_semijoins where we don't mind these +duplicated NULL values. It is set to true for 2 cases. One is a unique key +from a unique index but the related column is nullable. The other one is for +outer join. see populate_joinrel_uniquekeys for detail. + + +The UniqueKey can be used at the following cases at least: +1. remove_useless_joins. +2. reduce_semianti_joins +3. remove distinct node if distinct clause is unique. +4. remove aggnode if group by clause is unique. +5. Index Skip Scan (WIP) +6. Aggregation Push Down without 2 phase aggregation if the join can't + duplicated the aggregated rows. (work in progress feature) + +2. How is it maintained? + +We have a set of populate_xxx_unqiuekeys functions to maintain the uniquekey on +various cases. xxx includes baserel, joinrel, partitionedrel, distinctrel, +groupedrel, unionrel. and we also need to convert the uniquekey from subquery +to outer relation, which is what convert_subquery_uniquekeys does. + +1. The first part is about baserel. We handled 3 cases. suppose we have Unique +Index on (a, b). + +1. SELECT a, b FROM t. UniqueKey (a, b) +2. SELECT a FROM t WHERE b = 1; UniqueKey (a) +3. SELECT .. FROM t WHERE a = 1 AND b = 1; UniqueKey (NIL). onerow case, every + column is Unique. + +2. The next part is joinrel, this part is most error-prone, we simplified the rules +like below: +1. If the relation's UniqueKey can't be duplicated after join, then is will be + still valid for the join rel. The function we used here is + innerrel_keeps_unique. The basic idea is innerrel.any_col = outer.uk. + +2. If the UnqiueKey can't keep valid via the rule 1, the combination of the + UniqueKey from both sides are valid for sure. We can prove this as: if the + unique exprs from rel1 is duplicated by rel2, the duplicated rows must + contains different unique exprs from rel2. + +More considerations about onerow: +1. If relation with one row and it can't be duplicated, it is still possible + contains mulit_nullvas after outer join. +2. If the either UniqueKey can be duplicated after join, the can get one row + only when both side is one row AND there is no outer join. +3. Whenever the onerow UniqueKey is not a valid any more, we need to convert one + row UniqueKey to normal unique key since we don't store exprs for one-row + relation. get_exprs_from_uniquekeys will be used here. + + +More considerations about multi_nullvals after join: +1. If the original UnqiueKey has multi_nullvals, the final UniqueKey will have + mulit_nullvals in any case. +2. If a unique key doesn't allow mulit_nullvals, after some outer join, it + allows some outer join. + + +3. When we comes to subquery, we need to convert_subquery_unqiuekeys just like +convert_subquery_pathkeys. Only the UniqueKey insides subquery is referenced as +a Var in outer relation will be reused. The relationship between the outerrel.Var +and subquery.exprs is built with outerel->subroot->processed_tlist. + + +4. As for the SRF functions, it will break the uniqueness of uniquekey, However it +is handled in adjust_paths_for_srfs, which happens after the query_planner. so +we will maintain the UniqueKey until there and reset it to NIL at that +places. This can't help on distinct/group by elimination cases but probably help +in some other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is +semantic correctly. + + +5. As for inherit table, we first main the UnqiueKey on childrel as well. But for +partitioned table we need to maintain 2 different kinds of +UnqiueKey. 1). UniqueKey on the parent relation 2). UniqueKey on child +relation for partition wise query. + +Example: +CREATE TABLE p (a int not null, b int not null) partition by list (a); +CREATE TABLE p0 partition of p for values in (1); +CREATE TABLE p1 partition of p for values in (2); + +create unique index p0_b on p0(b); +create unique index p1_b on p1(b); + +Now b is only unique on partition level, so the distinct can't be removed on +the following cases. SELECT DISTINCT b FROM p; + +Another example is SELECT DISTINCT a, b FROM p WHERE a = 1; Since only one +partition is chosen, the UniqueKey on child relation is same as the UniqueKey on +parent relation. + +Another usage of UniqueKey on partition level is it be helpful for +partition-wise join. + +As for the UniqueKey on parent table level, it comes with 2 different ways, +1). the UniqueKey is also derived in UniqueKey index, but the index must be same +in all the related children relations and the unique index must contains +Partition Key in it. Example: + +CREATE UNIQUE INDEX p_ab ON p(a, b); -- where a is the partition key. + +-- Query +SELECT a, b FROM p; the (a, b) is a UniqueKey of p. + +2). If the parent relation has only one childrel, the UniqueKey on childrel is + the UniqueKey on parent as well. diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index 484dab0a1a..2ad9d06d7a 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -579,6 +579,12 @@ set_plain_rel_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) */ check_index_predicates(root, rel); + /* + * Now that we've marked which partial indexes are suitable, we can now + * build the relation's unique keys. + */ + populate_baserel_uniquekeys(root, rel, rel->indexlist); + /* Mark rel with estimated output rows, width, etc */ set_baserel_size_estimates(root, rel); } @@ -1310,6 +1316,8 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, /* Add paths to the append relation. */ add_paths_to_append_rel(root, rel, live_childrels); + if (IS_PARTITIONED_REL(rel)) + populate_partitionedrel_uniquekeys(root, rel, live_childrels); } @@ -2383,6 +2391,8 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, pathkeys, required_outer)); } + convert_subquery_uniquekeys(root, rel, sub_final_rel); + /* If outer rel allows parallelism, do same for partial paths. */ if (rel->consider_parallel && bms_is_empty(required_outer)) { diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c index db54a6ba2e..ef0fd2fb0b 100644 --- a/src/backend/optimizer/path/joinpath.c +++ b/src/backend/optimizer/path/joinpath.c @@ -71,13 +71,6 @@ static void consider_parallel_mergejoin(PlannerInfo *root, static void hash_inner_and_outer(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, RelOptInfo *innerrel, JoinType jointype, JoinPathExtraData *extra); -static List *select_mergejoin_clauses(PlannerInfo *root, - RelOptInfo *joinrel, - RelOptInfo *outerrel, - RelOptInfo *innerrel, - List *restrictlist, - JoinType jointype, - bool *mergejoin_allowed); static void generate_mergejoin_paths(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *innerrel, @@ -1927,7 +1920,7 @@ hash_inner_and_outer(PlannerInfo *root, * if it is mergejoinable and involves vars from the two sub-relations * currently of interest. */ -static List * +List * select_mergejoin_clauses(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c index 2d343cd293..b9163ee8ff 100644 --- a/src/backend/optimizer/path/joinrels.c +++ b/src/backend/optimizer/path/joinrels.c @@ -924,6 +924,8 @@ populate_joinrel_with_paths(PlannerInfo *root, RelOptInfo *rel1, /* Apply partitionwise join technique, if possible. */ try_partitionwise_join(root, rel1, rel2, joinrel, sjinfo, restrictlist); + + populate_joinrel_uniquekeys(root, joinrel, rel1, rel2, restrictlist, sjinfo->jointype); } diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index ce9bf87e9b..7e596d4194 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -33,7 +33,6 @@ static bool pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys); static bool matches_boolean_partition_clause(RestrictInfo *rinfo, RelOptInfo *partrel, int partkeycol); -static Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); static bool right_merge_direction(PlannerInfo *root, PathKey *pathkey); @@ -1035,7 +1034,7 @@ convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel, * We need this to ensure that we don't return pathkeys describing values * that are unavailable above the level of the subquery scan. */ -static Var * +Var * find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle) { ListCell *lc; diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c new file mode 100644 index 0000000000..b33bcd2f32 --- /dev/null +++ b/src/backend/optimizer/path/uniquekeys.c @@ -0,0 +1,1131 @@ +/*------------------------------------------------------------------------- + * + * uniquekeys.c + * Utilities for matching and building unique keys + * + * Portions Copyright (c) 2020, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/optimizer/path/uniquekeys.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "nodes/makefuncs.h" +#include "nodes/nodeFuncs.h" +#include "optimizer/pathnode.h" +#include "optimizer/paths.h" +#include "optimizer/appendinfo.h" +#include "optimizer/optimizer.h" +#include "optimizer/tlist.h" +#include "rewrite/rewriteManip.h" + + +/* + * This struct is used to help populate_joinrel_uniquekeys. + * + * added_to_joinrel is true if a uniquekey (from outerrel or innerrel) + * has been added to joinrel. + * useful is true if the exprs of the uniquekey still appears in joinrel. + */ +typedef struct UniqueKeyContextData +{ + UniqueKey *uniquekey; + bool added_to_joinrel; + bool useful; +} *UniqueKeyContext; + +static List *initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel); +static bool innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + bool reverse); + +static List *get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals); +static List *get_exprs_from_uniquekey(RelOptInfo *joinrel, + RelOptInfo *rel1, + UniqueKey *ukey); +static void add_uniquekey_for_onerow(RelOptInfo *rel); +static bool add_combined_uniquekey(RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype); + +/* Used for unique indexes checking for partitioned table */ +static bool index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind); +static IndexOptInfo *simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from); +static bool simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2); +static void adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_index); + +/* Helper function for grouped relation and distinct relation. */ +static void add_uniquekey_from_sortgroups(PlannerInfo *root, + RelOptInfo *rel, + List *sortgroups); + +/* + * populate_baserel_uniquekeys + * Populate 'baserel' uniquekeys list by looking at the rel's unique index + * and baserestrictinfo + */ +void +populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List *indexlist) +{ + ListCell *lc; + List *matched_uniq_indexes = NIL; + + /* Attrs appears in rel->reltarget->exprs. */ + Bitmapset *used_attrs = NULL; + + List *const_exprs = NIL; + List *expr_opfamilies = NIL; + + Assert(baserel->rtekind == RTE_RELATION); + + foreach(lc, indexlist) + { + IndexOptInfo *ind = (IndexOptInfo *) lfirst(lc); + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + matched_uniq_indexes = lappend(matched_uniq_indexes, ind); + } + + if (matched_uniq_indexes == NIL) + return; + + /* Check which attrs is used in baserel->reltarget */ + pull_varattnos((Node *)baserel->reltarget->exprs, baserel->relid, &used_attrs); + + /* Check which attrno is used at a mergeable const filter */ + foreach(lc, baserel->baserestrictinfo) + { + RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); + + if (rinfo->mergeopfamilies == NIL) + continue; + + if (bms_is_empty(rinfo->left_relids)) + { + const_exprs = lappend(const_exprs, get_rightop(rinfo->clause)); + } + else if (bms_is_empty(rinfo->right_relids)) + { + const_exprs = lappend(const_exprs, get_leftop(rinfo->clause)); + } + else + continue; + + expr_opfamilies = lappend(expr_opfamilies, rinfo->mergeopfamilies); + } + + foreach(lc, matched_uniq_indexes) + { + bool multi_nullvals, useful; + List *exprs = get_exprs_from_uniqueindex(lfirst_node(IndexOptInfo, lc), + const_exprs, + expr_opfamilies, + used_attrs, + &useful, + &multi_nullvals); + if (useful) + { + if (exprs == NIL) + { + /* All the columns in Unique Index matched with a restrictinfo */ + add_uniquekey_for_onerow(baserel); + return; + } + baserel->uniquekeys = lappend(baserel->uniquekeys, + makeUniqueKey(exprs, multi_nullvals)); + } + } +} + + +/* + * populate_partitionedrel_uniquekeys + * The UniqueKey on partitionrel comes from 2 cases: + * 1). Only one partition is involved in this query, the unique key can be + * copied to parent rel from childrel. + * 2). There are some unique index which includes partition key and exists + * in all the related partitions. + * We never mind rule 2 if we hit rule 1. + */ + +void +populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels) +{ + ListCell *lc; + List *global_uniq_indexlist = NIL; + RelOptInfo *childrel; + bool is_first = true; + + Assert(IS_PARTITIONED_REL(rel)); + + if (childrels == NIL) + return; + + /* + * If there is only one partition used in this query, the UniqueKey in childrel is + * still valid in parent level, but we need convert the format from child expr to + * parent expr. + */ + if (list_length(childrels) == 1) + { + /* Check for Rule 1 */ + RelOptInfo *childrel = linitial_node(RelOptInfo, childrels); + ListCell *lc; + Assert(childrel->reloptkind == RELOPT_OTHER_MEMBER_REL); + if (relation_is_onerow(childrel)) + { + add_uniquekey_for_onerow(rel); + return; + } + + foreach(lc, childrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + AppendRelInfo *appinfo = find_appinfo_by_child(root, childrel->relid); + List *parent_exprs = NIL; + bool can_reuse = true; + ListCell *lc2; + foreach(lc2, ukey->exprs) + { + Var *var = (Var *)lfirst(lc2); + /* + * If the expr comes from a expression, it is hard to build the expression + * in parent so ignore that case for now. + */ + if(!IsA(var, Var)) + { + can_reuse = false; + break; + } + /* Convert it to parent var */ + parent_exprs = lappend(parent_exprs, find_parent_var(appinfo, var)); + } + if (can_reuse) + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(parent_exprs, + ukey->multi_nullvals)); + } + } + else + { + /* Check for rule 2 */ + childrel = linitial_node(RelOptInfo, childrels); + foreach(lc, childrel->indexlist) + { + IndexOptInfo *ind = lfirst(lc); + IndexOptInfo *modified_index; + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + + /* + * During simple_copy_indexinfo_to_parent, we need to convert var from + * child var to parent var, index on expression is too complex to handle. + * so ignore it for now. + */ + if (ind->indexprs != NIL) + continue; + + modified_index = simple_copy_indexinfo_to_parent(root, rel, ind); + /* + * If the unique index doesn't contain partkey, then it is unique + * on this partition only, so it is useless for us. + */ + if (!index_constains_partkey(rel, modified_index)) + continue; + + global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index); + } + + if (global_uniq_indexlist != NIL) + { + foreach(lc, childrels) + { + RelOptInfo *child = lfirst(lc); + if (is_first) + { + is_first = false; + continue; + } + adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist); + } + /* Now we have a list of unique index which are exactly same on all childrels, + * Set the UniqueKey just like it is non-partition table + */ + populate_baserel_uniquekeys(root, rel, global_uniq_indexlist); + } + } +} + + +/* + * populate_distinctrel_uniquekeys + */ +void +populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel) +{ + /* The unique key before the distinct is still valid. */ + distinctrel->uniquekeys = list_copy(inputrel->uniquekeys); + add_uniquekey_from_sortgroups(root, distinctrel, root->parse->distinctClause); +} + +/* + * populate_grouprel_uniquekeys + */ +void +populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel) + +{ + Query *parse = root->parse; + bool input_ukey_added = false; + ListCell *lc; + + if (relation_is_onerow(inputrel)) + { + add_uniquekey_for_onerow(grouprel); + return; + } + if (parse->groupingSets) + return; + + /* A Normal group by without grouping set. */ + if (parse->groupClause) + { + /* + * Current even the groupby clause is Unique already, but if query has aggref + * We have to create grouprel still. To keep the UnqiueKey short, we will check + * the UniqueKey of input_rel still valid, if so we reuse it. + */ + foreach(lc, inputrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (list_is_subset(ukey->exprs, grouprel->reltarget->exprs)) + { + grouprel->uniquekeys = lappend(grouprel->uniquekeys, + ukey); + input_ukey_added = true; + } + } + if (!input_ukey_added) + /* + * group by clause must be a super-set of grouprel->reltarget->exprs except the + * aggregation expr, so if such exprs is unique already, no bother to generate + * new uniquekey for group by exprs. + */ + add_uniquekey_from_sortgroups(root, + grouprel, + root->parse->groupClause); + } + else + /* It has aggregation but without a group by, so only one row returned */ + add_uniquekey_for_onerow(grouprel); +} + +/* + * simple_copy_uniquekeys + * Using a function for the one-line code makes us easy to check where we simply + * copied the uniquekey. + */ +void +simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel) +{ + newrel->uniquekeys = oldrel->uniquekeys; +} + +/* + * populate_unionrel_uniquekeys + */ +void +populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel) +{ + ListCell *lc; + List *exprs = NIL; + + Assert(unionrel->uniquekeys == NIL); + + foreach(lc, unionrel->reltarget->exprs) + { + exprs = lappend(exprs, lfirst(lc)); + } + + if (exprs == NIL) + /* SQL: select union select; is valid, we need to handle it here. */ + add_uniquekey_for_onerow(unionrel); + else + unionrel->uniquekeys = lappend(unionrel->uniquekeys, + makeUniqueKey(exprs,false)); + +} + +/* + * populate_joinrel_uniquekeys + * + * populate uniquekeys for joinrel. We will check each relation to see if its + * UniqueKey is still valid via innerrel_keeps_unique, if so, we add it to + * joinrel. The multi_nullvals field will be changed to true for some outer + * join cases and one-row UniqueKey needs to be converted to normal UniqueKey + * for the same case as well. + * For the uniquekey in either baserel which can't be unique after join, we still + * check to see if combination of UniqueKeys from both side is still useful for us. + * if yes, we add it to joinrel as well. + */ +void +populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *outerrel, RelOptInfo *innerrel, + List *restrictlist, JoinType jointype) +{ + ListCell *lc, *lc2; + List *clause_list = NIL; + List *outerrel_ukey_ctx; + List *innerrel_ukey_ctx; + bool inner_onerow, outer_onerow; + bool mergejoin_allowed; + + /* Care about the outerrel relation only for SEMI/ANTI join */ + if (jointype == JOIN_SEMI || jointype == JOIN_ANTI) + { + foreach(lc, outerrel->uniquekeys) + { + UniqueKey *uniquekey = lfirst_node(UniqueKey, lc); + if (list_is_subset(uniquekey->exprs, joinrel->reltarget->exprs)) + joinrel->uniquekeys = lappend(joinrel->uniquekeys, uniquekey); + } + return; + } + + Assert(jointype == JOIN_LEFT || jointype == JOIN_FULL || jointype == JOIN_INNER); + + /* Fast path */ + if (innerrel->uniquekeys == NIL || outerrel->uniquekeys == NIL) + return; + + inner_onerow = relation_is_onerow(innerrel); + outer_onerow = relation_is_onerow(outerrel); + + outerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(outerrel); + innerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(innerrel); + + clause_list = select_mergejoin_clauses(root, joinrel, outerrel, innerrel, + restrictlist, jointype, + &mergejoin_allowed); + + if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true /* reverse */)) + { + bool outer_impact = jointype == JOIN_FULL; + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + /* Outer relation has one row, and the unique key is not duplicated after join, + * the joinrel will still has one row unless the jointype == JOIN_FULL. + */ + if (outer_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (outer_onerow) + { + /* + * The onerow outerrel becomes multi rows and multi_nullvals + * will be changed to true. We also need to set the exprs correctly since it + * can't be NIL any more. + */ + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(joinrel, outerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Change multi_nullvals to true due to the full join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, true)); + else + /* Just reuse it */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + } + ctx->added_to_joinrel = true; + } + } + + if (innerrel_keeps_unique(root, outerrel, innerrel, clause_list, false)) + { + bool outer_impact = jointype == JOIN_FULL || jointype == JOIN_LEFT;; + + foreach(lc, innerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + if (inner_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (inner_onerow) + { + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(joinrel, innerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Need to change multi_nullvals to true due to the outer join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, + true)); + else + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + + } + ctx->added_to_joinrel = true; + } + } + + /* + * The combination of the UniqueKey from both sides is unique as well regardless + * of join type, but no bother to add it if its subset has been added to joinrel + * already or it is not useful for the joinrel. + */ + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx1 = (UniqueKeyContext) lfirst(lc); + if (ctx1->added_to_joinrel || !ctx1->useful) + continue; + foreach(lc2, innerrel_ukey_ctx) + { + UniqueKeyContext ctx2 = (UniqueKeyContext) lfirst(lc2); + if (ctx2->added_to_joinrel || !ctx2->useful) + continue; + if (add_combined_uniquekey(joinrel, outerrel, innerrel, + ctx1->uniquekey, ctx2->uniquekey, + jointype)) + /* If we set a onerow UniqueKey to joinrel, we don't need other. */ + return; + } + } +} + + +/* + * convert_subquery_uniquekeys + * + * Covert the UniqueKey in subquery to outer relation. + */ +void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel) +{ + ListCell *lc; + + if (sub_final_rel->uniquekeys == NIL) + return; + + if (relation_is_onerow(sub_final_rel)) + { + add_uniquekey_for_onerow(currel); + return; + } + + Assert(currel->subroot != NULL); + + foreach(lc, sub_final_rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + ListCell *lc; + List *exprs = NIL; + bool ukey_useful = true; + + /* One row case is handled above */ + Assert(ukey->exprs != NIL); + foreach(lc, ukey->exprs) + { + Var *var; + TargetEntry *tle = tlist_member(lfirst(lc), + currel->subroot->processed_tlist); + if (tle == NULL) + { + ukey_useful = false; + break; + } + var = find_var_for_subquery_tle(currel, tle); + if (var == NULL) + { + ukey_useful = false; + break; + } + exprs = lappend(exprs, var); + } + + if (ukey_useful) + currel->uniquekeys = lappend(currel->uniquekeys, + makeUniqueKey(exprs, + ukey->multi_nullvals)); + + } +} + +/* + * innerrel_keeps_unique + * + * Check if Unique key of the innerrel is valid after join. innerrel's UniqueKey + * will be still valid if innerrel's any-column mergeop outrerel's uniquekey + * exists in clause_list. + * + * Note: the clause_list must be a list of mergeable restrictinfo already. + */ +static bool +innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *clause_list, + bool reverse) +{ + ListCell *lc, *lc2, *lc3; + + if (outerrel->uniquekeys == NIL || innerrel->uniquekeys == NIL) + return false; + + /* Check if there is outerrel's uniquekey in mergeable clause. */ + foreach(lc, outerrel->uniquekeys) + { + List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs; + bool clauselist_matchs_all_exprs = true; + foreach(lc2, outer_uq_exprs) + { + Node *outer_uq_expr = lfirst(lc2); + bool find_uq_expr_in_clauselist = false; + foreach(lc3, clause_list) + { + RestrictInfo *rinfo = lfirst_node(RestrictInfo, lc3); + Node *outer_expr; + if (reverse) + outer_expr = rinfo->outer_is_left ? get_rightop(rinfo->clause) : get_leftop(rinfo->clause); + else + outer_expr = rinfo->outer_is_left ? get_leftop(rinfo->clause) : get_rightop(rinfo->clause); + if (equal(outer_expr, outer_uq_expr)) + { + find_uq_expr_in_clauselist = true; + break; + } + } + if (!find_uq_expr_in_clauselist) + { + /* No need to check the next exprs in the current uniquekey */ + clauselist_matchs_all_exprs = false; + break; + } + } + + if (clauselist_matchs_all_exprs) + return true; + } + return false; +} + + +/* + * relation_is_onerow + * Check if it is a one-row relation by checking UniqueKey. + */ +bool +relation_is_onerow(RelOptInfo *rel) +{ + UniqueKey *ukey; + if (rel->uniquekeys == NIL) + return false; + ukey = linitial_node(UniqueKey, rel->uniquekeys); + return ukey->exprs == NIL && list_length(rel->uniquekeys) == 1; +} + +/* + * relation_has_uniquekeys_for + * Returns true if we have proofs that 'rel' cannot return multiple rows with + * the same values in each of 'exprs'. Otherwise returns false. + */ +bool +relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel, + List *exprs, bool allow_multinulls) +{ + ListCell *lc; + + /* + * For UniqueKey->onerow case, the uniquekey->exprs is empty as well + * so we can't rely on list_is_subset to handle this special cases + */ + if (exprs == NIL) + return false; + + foreach(lc, rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (ukey->multi_nullvals && !allow_multinulls) + continue; + if (list_is_subset(ukey->exprs, exprs)) + return true; + } + return false; +} + + +/* + * get_exprs_from_uniqueindex + * + * Return a list of exprs which is unique. set useful to false if this + * unique index is not useful for us. + */ +static List * +get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals) +{ + List *exprs = NIL; + ListCell *indexpr_item; + int c = 0; + + *useful = true; + *multi_nullvals = false; + + indexpr_item = list_head(unique_index->indexprs); + for(c = 0; c < unique_index->ncolumns; c++) + { + int attr = unique_index->indexkeys[c]; + Expr *expr; + bool matched_const = false; + ListCell *lc1, *lc2; + + if(attr > 0) + { + expr = list_nth_node(TargetEntry, unique_index->indextlist, c)->expr; + } + else if (attr == 0) + { + /* Expression index */ + expr = lfirst(indexpr_item); + indexpr_item = lnext(unique_index->indexprs, indexpr_item); + } + else /* attr < 0 */ + { + /* Index on system column is not supported */ + Assert(false); + } + + /* + * Check index_col = Const case with regarding to opfamily checking + * If we can remove the index_col from the final UniqueKey->exprs. + */ + forboth(lc1, const_exprs, lc2, const_expr_opfamilies) + { + if (list_member_oid((List *)lfirst(lc2), unique_index->opfamily[c]) + && match_index_to_operand((Node *) lfirst(lc1), c, unique_index)) + { + matched_const = true; + break; + } + } + + if (matched_const) + continue; + + /* Check if the indexed expr is used in rel */ + if (attr > 0) + { + /* + * Normal Indexed column, if the col is not used, then the index is useless + * for uniquekey. + */ + attr -= FirstLowInvalidHeapAttributeNumber; + + if (!bms_is_member(attr, used_varattrs)) + { + *useful = false; + break; + } + } + else if (!list_member(unique_index->rel->reltarget->exprs, expr)) + { + /* Expression index but the expression is not used in rel */ + *useful = false; + break; + } + + /* check not null property. */ + if (attr == 0) + { + /* We never know if a expression yields null or not */ + *multi_nullvals = true; + } + else if (!bms_is_member(attr, unique_index->rel->notnullattrs) + && !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber, + unique_index->rel->notnullattrs)) + { + *multi_nullvals = true; + } + + exprs = lappend(exprs, expr); + } + return exprs; +} + + +/* + * add_uniquekey_for_onerow + * If we are sure that the relation only returns one row, then all the columns + * are unique. However we don't need to create UniqueKey for every column, we + * just set exprs = NIL and overwrites all the other UniqueKey on this RelOptInfo + * since this one has strongest semantics. + */ +void +add_uniquekey_for_onerow(RelOptInfo *rel) +{ + /* + * We overwrite the previous UniqueKey on purpose since this one has the + * strongest semantic. + */ + rel->uniquekeys = list_make1(makeUniqueKey(NIL, false)); +} + + +/* + * initililze_uniquecontext_for_joinrel + * Return a List of UniqueKeyContext for an inputrel + */ +static List * +initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel) +{ + List *res = NIL; + ListCell *lc; + foreach(lc, inputrel->uniquekeys) + { + UniqueKeyContext context; + context = palloc(sizeof(struct UniqueKeyContextData)); + context->uniquekey = lfirst_node(UniqueKey, lc); + context->added_to_joinrel = false; + context->useful = true; + res = lappend(res, context); + } + return res; +} + + +/* + * get_exprs_from_uniquekey + * Unify the way of get List of exprs from a one-row UniqueKey or + * normal UniqueKey. for the onerow case, every expr in rel1 is a valid + * UniqueKey. Return a List of exprs. + * + * rel1: The relation which you want to get the exprs. + * ukey: The UniqueKey you want to get the exprs. + */ +static List * +get_exprs_from_uniquekey(RelOptInfo *joinrel, RelOptInfo *rel1, UniqueKey *ukey) +{ + ListCell *lc; + bool onerow = rel1 != NULL && relation_is_onerow(rel1); + + List *res = NIL; + Assert(onerow || ukey); + if (onerow) + { + /* Only cares about the exprs still exist in joinrel */ + foreach(lc, joinrel->reltarget->exprs) + { + Bitmapset *relids = pull_varnos(lfirst(lc)); + if (bms_is_subset(relids, rel1->relids)) + { + res = lappend(res, list_make1(lfirst(lc))); + } + } + } + else + { + res = list_make1(ukey->exprs); + } + return res; +} + +/* + * Partitioned table Unique Keys. + * The partition table unique key is maintained as: + * 1. The index must be unique as usual. + * 2. The index must contains partition key. + * 3. The index must exist on all the child rel. see simple_indexinfo_equal for + * how we compare it. + */ + +/* + * index_constains_partkey + * return true if the index contains the partiton key. + */ +static bool +index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind) +{ + ListCell *lc; + int i; + Assert(IS_PARTITIONED_REL(partrel)); + Assert(partrel->part_scheme->partnatts > 0); + + for(i = 0; i < partrel->part_scheme->partnatts; i++) + { + Node *part_expr = linitial(partrel->partexprs[i]); + bool found_in_index = false; + foreach(lc, ind->indextlist) + { + Expr *index_expr = lfirst_node(TargetEntry, lc)->expr; + if (equal(index_expr, part_expr)) + { + found_in_index = true; + break; + } + } + if (!found_in_index) + return false; + } + return true; +} + +/* + * simple_indexinfo_equal + * + * Used to check if the 2 index is same as each other. The index here + * is COPIED from childrel and did some tiny changes(see + * simple_copy_indexinfo_to_parent) + */ +static bool +simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2) +{ + Size oid_cmp_len = sizeof(Oid) * ind1->ncolumns; + + return ind1->ncolumns == ind2->ncolumns && + ind1->unique == ind2->unique && + memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0 && + memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0 && + memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0 && + memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0 && + equal(get_tlist_exprs(ind1->indextlist, true), + get_tlist_exprs(ind2->indextlist, true)); +} + + +/* + * The below macros are used for simple_copy_indexinfo_to_parent which is so + * customized that I don't want to put it to copyfuncs.c. So copy it here. + */ +#define COPY_POINTER_FIELD(fldname, sz) \ + do { \ + Size _size = (sz); \ + newnode->fldname = palloc(_size); \ + memcpy(newnode->fldname, from->fldname, _size); \ + } while (0) + +#define COPY_NODE_FIELD(fldname) \ + (newnode->fldname = copyObjectImpl(from->fldname)) + +#define COPY_SCALAR_FIELD(fldname) \ + (newnode->fldname = from->fldname) + + +/* + * simple_copy_indexinfo_to_parent (from partition) + * Copy the IndexInfo from child relation to parent relation with some modification, + * which is used to test: + * 1. If the same index exists in all the childrels. + * 2. If the parentrel->reltarget/basicrestrict info matches this index. + */ +static IndexOptInfo * +simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from) +{ + IndexOptInfo *newnode = makeNode(IndexOptInfo); + AppendRelInfo *appinfo = find_appinfo_by_child(root, from->rel->relid); + ListCell *lc; + int idx = 0; + + COPY_SCALAR_FIELD(ncolumns); + COPY_SCALAR_FIELD(nkeycolumns); + COPY_SCALAR_FIELD(unique); + COPY_SCALAR_FIELD(immediate); + /* We just need to know if it is NIL or not */ + COPY_SCALAR_FIELD(indpred); + COPY_SCALAR_FIELD(predOK); + COPY_POINTER_FIELD(indexkeys, from->ncolumns * sizeof(int)); + COPY_POINTER_FIELD(indexcollations, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opfamily, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opcintype, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(sortopfamily, from->ncolumns * sizeof(Oid)); + COPY_NODE_FIELD(indextlist); + + /* Convert index exprs on child expr to expr on parent */ + foreach(lc, newnode->indextlist) + { + TargetEntry *tle = lfirst_node(TargetEntry, lc); + /* Index on expression is ignored */ + Assert(IsA(tle->expr, Var)); + tle->expr = (Expr *) find_parent_var(appinfo, (Var *) tle->expr); + newnode->indexkeys[idx] = castNode(Var, tle->expr)->varattno; + idx++; + } + newnode->rel = parentrel; + return newnode; +} + +/* + * adjust_partition_unique_indexlist + * + * global_unique_indexes: At the beginning, it contains the copy & modified + * unique index from the first partition. And then check if each index in it still + * exists in the following partitions. If no, remove it. at last, it has an + * index list which exists in all the partitions. + */ +static void +adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_indexes) +{ + ListCell *lc, *lc2; + foreach(lc, *global_unique_indexes) + { + IndexOptInfo *g_ind = lfirst_node(IndexOptInfo, lc); + bool found_in_child = false; + + foreach(lc2, childrel->indexlist) + { + IndexOptInfo *p_ind = lfirst_node(IndexOptInfo, lc2); + IndexOptInfo *p_ind_copy; + if (!p_ind->unique || !p_ind->immediate || + (p_ind->indpred != NIL && !p_ind->predOK)) + continue; + p_ind_copy = simple_copy_indexinfo_to_parent(root, parentrel, p_ind); + if (simple_indexinfo_equal(p_ind_copy, g_ind)) + { + found_in_child = true; + break; + } + } + if (!found_in_child) + /* The index doesn't exist in childrel, remove it from global_unique_indexes */ + *global_unique_indexes = foreach_delete_current(*global_unique_indexes, lc); + } +} + +/* Helper function for groupres/distinctrel */ +static void +add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgroups) +{ + Query *parse = root->parse; + List *exprs; + + /* + * XXX: If there are some vars which is not in current levelsup, the semantic is + * imprecise, should we avoid it or not? levelsup = 1 is just a demo, maybe we need to + * check every level other than 0, if so, looks we have to write another + * pull_var_walker. + */ + List *upper_vars = pull_vars_of_level((Node*)sortgroups, 1); + + if (upper_vars != NIL) + return; + + exprs = get_sortgrouplist_exprs(sortgroups, parse->targetList); + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(exprs, + false /* sortgroupclause can't be multi_nullvals */)); +} + + +/* + * add_combined_uniquekey + * The combination of both UniqueKeys is a valid UniqueKey for joinrel no matter + * the jointype. + */ +bool +add_combined_uniquekey(RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype) +{ + + ListCell *lc1, *lc2; + + /* Either side has multi_nullvals or we have outer join, + * the combined UniqueKey has multi_nullvals */ + bool multi_nullvals = outer_ukey->multi_nullvals || + inner_ukey->multi_nullvals || IS_OUTER_JOIN(jointype); + + /* The only case we can get onerow joinrel after join */ + if (relation_is_onerow(outer_rel) + && relation_is_onerow(inner_rel) + && jointype == JOIN_INNER) + { + add_uniquekey_for_onerow(joinrel); + return true; + } + + foreach(lc1, get_exprs_from_uniquekey(joinrel, outer_rel, outer_ukey)) + { + foreach(lc2, get_exprs_from_uniquekey(joinrel, inner_rel, inner_ukey)) + { + List *exprs = list_concat_copy(lfirst_node(List, lc1), lfirst_node(List, lc2)); + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(exprs, + multi_nullvals)); + } + } + return false; +} diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index b406d41e91..0551ae0512 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -2389,6 +2389,8 @@ grouping_planner(PlannerInfo *root, bool inheritance_update, add_path(final_rel, path); } + simple_copy_uniquekeys(current_rel, final_rel); + /* * Generate partial paths for final_rel, too, if outer query levels might * be able to make use of them. @@ -3899,6 +3901,8 @@ create_grouping_paths(PlannerInfo *root, } set_cheapest(grouped_rel); + + populate_grouprel_uniquekeys(root, grouped_rel, input_rel); return grouped_rel; } @@ -4615,7 +4619,7 @@ create_window_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(window_rel); - + simple_copy_uniquekeys(input_rel, window_rel); return window_rel; } @@ -4911,7 +4915,7 @@ create_distinct_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(distinct_rel); - + populate_distinctrel_uniquekeys(root, input_rel, distinct_rel); return distinct_rel; } @@ -5172,6 +5176,8 @@ create_ordered_paths(PlannerInfo *root, */ Assert(ordered_rel->pathlist != NIL); + simple_copy_uniquekeys(input_rel, ordered_rel); + return ordered_rel; } @@ -6049,6 +6055,9 @@ adjust_paths_for_srfs(PlannerInfo *root, RelOptInfo *rel, if (list_length(targets) == 1) return; + /* UniqueKey is not valid after handling the SRF. */ + rel->uniquekeys = NIL; + /* * Stack SRF-evaluation nodes atop each path for the rel. * diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c index 951aed80e7..e94e92937c 100644 --- a/src/backend/optimizer/prep/prepunion.c +++ b/src/backend/optimizer/prep/prepunion.c @@ -689,6 +689,8 @@ generate_union_paths(SetOperationStmt *op, PlannerInfo *root, /* Undo effects of possibly forcing tuple_fraction to 0 */ root->tuple_fraction = save_fraction; + /* Add the UniqueKeys */ + populate_unionrel_uniquekeys(root, result_rel); return result_rel; } diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c index d722063cf3..44c37ecffc 100644 --- a/src/backend/optimizer/util/appendinfo.c +++ b/src/backend/optimizer/util/appendinfo.c @@ -746,3 +746,47 @@ find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos) } return appinfos; } + +/* + * find_appinfo_by_child + * + */ +AppendRelInfo * +find_appinfo_by_child(PlannerInfo *root, Index child_index) +{ + ListCell *lc; + foreach(lc, root->append_rel_list) + { + AppendRelInfo *appinfo = lfirst_node(AppendRelInfo, lc); + if (appinfo->child_relid == child_index) + return appinfo; + } + elog(ERROR, "parent relation cant be found"); + return NULL; +} + +/* + * find_parent_var + * + */ +Var * +find_parent_var(AppendRelInfo *appinfo, Var *child_var) +{ + ListCell *lc; + Var *res = NULL; + Index attno = 1; + foreach(lc, appinfo->translated_vars) + { + Node *child_node = lfirst(lc); + if (equal(child_node, child_var)) + { + res = copyObject(child_var); + res->varattno = attno; + res->varno = appinfo->parent_relid; + } + attno++; + } + if (res == NULL) + elog(ERROR, "parent var can't be found."); + return res; +} diff --git a/src/backend/optimizer/util/inherit.c b/src/backend/optimizer/util/inherit.c index 3132fd35a5..d66b40ec50 100644 --- a/src/backend/optimizer/util/inherit.c +++ b/src/backend/optimizer/util/inherit.c @@ -736,6 +736,7 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, { Node *onecq = (Node *) lfirst(lc2); bool pseudoconstant; + RestrictInfo *child_rinfo; /* check for pseudoconstant (no Vars or volatile functions) */ pseudoconstant = @@ -747,13 +748,14 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, root->hasPseudoConstantQuals = true; } /* reconstitute RestrictInfo with appropriate properties */ - childquals = lappend(childquals, - make_restrictinfo((Expr *) onecq, - rinfo->is_pushed_down, - rinfo->outerjoin_delayed, - pseudoconstant, - rinfo->security_level, - NULL, NULL, NULL)); + child_rinfo = make_restrictinfo((Expr *) onecq, + rinfo->is_pushed_down, + rinfo->outerjoin_delayed, + pseudoconstant, + rinfo->security_level, + NULL, NULL, NULL); + child_rinfo->mergeopfamilies = rinfo->mergeopfamilies; + childquals = lappend(childquals, child_rinfo); /* track minimum security level among child quals */ cq_min_security = Min(cq_min_security, rinfo->security_level); } diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 31d9aedeeb..c83f17acb7 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -16,6 +16,7 @@ #include "nodes/execnodes.h" #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" extern A_Expr *makeA_Expr(A_Expr_Kind kind, List *name, @@ -105,4 +106,6 @@ extern GroupingSet *makeGroupingSet(GroupingSetKind kind, List *content, int loc extern VacuumRelation *makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols); +extern UniqueKey* makeUniqueKey(List *exprs, bool multi_nullvals); + #endif /* MAKEFUNC_H */ diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 381d84b4e4..41110ed888 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -264,6 +264,7 @@ typedef enum NodeTag T_EquivalenceMember, T_PathKey, T_PathTarget, + T_UniqueKey, T_RestrictInfo, T_IndexClause, T_PlaceHolderVar, diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index 9e3ebd488a..02e4458bef 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -730,6 +730,7 @@ typedef struct RelOptInfo QualCost baserestrictcost; /* cost of evaluating the above */ Index baserestrict_min_security; /* min security_level found in * baserestrictinfo */ + List *uniquekeys; /* List of UniqueKey */ List *joininfo; /* RestrictInfo structures for join clauses * involving this rel */ bool has_eclass_joins; /* T means joininfo is incomplete */ @@ -1047,6 +1048,28 @@ typedef struct PathKey } PathKey; +/* + * UniqueKey + * + * Represents the unique properties held by a RelOptInfo. + * + * exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL + * is a special case of UniqueKey, which means there is only 1 row in that + * relation. + * multi_nullvals: true means multi null values may exist in these exprs, so the + * uniqueness is not guaranteed in this case. This field is necessary for + * remove_useless_join & reduce_unique_semijoins where we don't mind these + * duplicated NULL values. It is set to true for 2 cases. One is a unique key + * from a unique index but the related column is nullable. The other one is for + * outer join. see populate_joinrel_uniquekeys for detail. + */ +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + /* * PathTarget * @@ -2473,7 +2496,7 @@ typedef enum * * flags indicating what kinds of grouping are possible. * partial_costs_set is true if the agg_partial_costs and agg_final_costs - * have been initialized. + * have been initialized. * agg_partial_costs gives partial aggregation costs. * agg_final_costs gives finalization costs. * target_parallel_safe is true if target is parallel safe. @@ -2503,8 +2526,8 @@ typedef struct * limit_tuples is an estimated bound on the number of output tuples, * or -1 if no LIMIT or couldn't estimate. * count_est and offset_est are the estimated values of the LIMIT and OFFSET - * expressions computed by preprocess_limit() (see comments for - * preprocess_limit() for more information). + * expressions computed by preprocess_limit() (see comments for + * preprocess_limit() for more information). */ typedef struct { diff --git a/src/include/nodes/pg_list.h b/src/include/nodes/pg_list.h index 14ea2766ad..621f54a9f8 100644 --- a/src/include/nodes/pg_list.h +++ b/src/include/nodes/pg_list.h @@ -528,6 +528,8 @@ extern bool list_member_ptr(const List *list, const void *datum); extern bool list_member_int(const List *list, int datum); extern bool list_member_oid(const List *list, Oid datum); +extern bool list_is_subset(const List *members, const List *target); + extern List *list_delete(List *list, void *datum); extern List *list_delete_ptr(List *list, void *datum); extern List *list_delete_int(List *list, int datum); diff --git a/src/include/optimizer/appendinfo.h b/src/include/optimizer/appendinfo.h index d6a27a60dd..e87c92a054 100644 --- a/src/include/optimizer/appendinfo.h +++ b/src/include/optimizer/appendinfo.h @@ -32,4 +32,7 @@ extern Relids adjust_child_relids_multilevel(PlannerInfo *root, Relids relids, extern AppendRelInfo **find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos); +extern AppendRelInfo *find_appinfo_by_child(PlannerInfo *root, Index child_index); +extern Var *find_parent_var(AppendRelInfo *appinfo, Var *child_var); + #endif /* APPENDINFO_H */ diff --git a/src/include/optimizer/optimizer.h b/src/include/optimizer/optimizer.h index 3e4171056e..9445141263 100644 --- a/src/include/optimizer/optimizer.h +++ b/src/include/optimizer/optimizer.h @@ -23,6 +23,7 @@ #define OPTIMIZER_H #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" /* * We don't want to include nodes/pathnodes.h here, because non-planner @@ -156,6 +157,7 @@ extern TargetEntry *get_sortgroupref_tle(Index sortref, List *targetList); extern TargetEntry *get_sortgroupclause_tle(SortGroupClause *sgClause, List *targetList); +extern Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); extern Node *get_sortgroupclause_expr(SortGroupClause *sgClause, List *targetList); extern List *get_sortgrouplist_exprs(List *sgClauses, diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h index 10b6e81079..9217a8d6c6 100644 --- a/src/include/optimizer/paths.h +++ b/src/include/optimizer/paths.h @@ -240,5 +240,48 @@ extern PathKey *make_canonical_pathkey(PlannerInfo *root, int strategy, bool nulls_first); extern void add_paths_to_append_rel(PlannerInfo *root, RelOptInfo *rel, List *live_childrels); +extern List *select_mergejoin_clauses(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + JoinType jointype, + bool *mergejoin_allowed); + +/* + * uniquekeys.c + * Utilities for matching and building unique keys + */ +extern void populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List* unique_index_list); +extern void populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels); +extern void populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel); +extern void populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel); +extern void populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel); +extern void simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel); +extern void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel); +extern void populate_joinrel_uniquekeys(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + RelOptInfo *rel2, + List *restrictlist, + JoinType jointype); + +extern bool relation_has_uniquekeys_for(PlannerInfo *root, + RelOptInfo *rel, + List *exprs, + bool allow_multinulls); +extern bool relation_is_onerow(RelOptInfo *rel); #endif /* PATHS_H */ -- 2.21.0 --op7mf2i72xk5yu7c Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v36-0003-Extend-UniqueKeys.patch" ^ permalink raw reply [nested|flat] 91+ messages in thread
* [PATCH v38 2/6] Introduce UniqueKey attributes on RelOptInfo struct. @ 2020-05-11 07:50 一挃 <[email protected]> 0 siblings, 0 replies; 91+ messages in thread From: 一挃 @ 2020-05-11 07:50 UTC (permalink / raw) UniqueKey is a set of exprs on RelOptInfo which represents the exprs will be unique on the given RelOptInfo. You can see README.uniquekey for more information. --- src/backend/nodes/copyfuncs.c | 13 + src/backend/nodes/list.c | 31 + src/backend/nodes/makefuncs.c | 13 + src/backend/nodes/outfuncs.c | 11 + src/backend/nodes/readfuncs.c | 10 + src/backend/optimizer/path/Makefile | 3 +- src/backend/optimizer/path/README.uniquekey | 131 +++ src/backend/optimizer/path/allpaths.c | 10 + src/backend/optimizer/path/joinpath.c | 9 +- src/backend/optimizer/path/joinrels.c | 2 + src/backend/optimizer/path/pathkeys.c | 3 +- src/backend/optimizer/path/uniquekeys.c | 1135 +++++++++++++++++++ src/backend/optimizer/plan/planner.c | 13 +- src/backend/optimizer/prep/prepunion.c | 2 + src/backend/optimizer/util/appendinfo.c | 44 + src/backend/optimizer/util/inherit.c | 18 +- src/include/nodes/makefuncs.h | 3 + src/include/nodes/nodes.h | 1 + src/include/nodes/pathnodes.h | 29 +- src/include/nodes/pg_list.h | 10 + src/include/optimizer/appendinfo.h | 3 + src/include/optimizer/optimizer.h | 2 + src/include/optimizer/paths.h | 43 + 23 files changed, 1515 insertions(+), 24 deletions(-) create mode 100644 src/backend/optimizer/path/README.uniquekey create mode 100644 src/backend/optimizer/path/uniquekeys.c diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..75c1c5e824 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2296,6 +2296,16 @@ _copyPathKey(const PathKey *from) return newnode; } +static UniqueKey * +_copyUniqueKey(const UniqueKey *from) +{ + UniqueKey *newnode = makeNode(UniqueKey); + + COPY_NODE_FIELD(exprs); + COPY_SCALAR_FIELD(multi_nullvals); + + return newnode; +} /* * _copyRestrictInfo */ @@ -5220,6 +5230,9 @@ copyObjectImpl(const void *from) case T_PathKey: retval = _copyPathKey(from); break; + case T_UniqueKey: + retval = _copyUniqueKey(from); + break; case T_RestrictInfo: retval = _copyRestrictInfo(from); break; diff --git a/src/backend/nodes/list.c b/src/backend/nodes/list.c index dbf6b30233..ca099495a1 100644 --- a/src/backend/nodes/list.c +++ b/src/backend/nodes/list.c @@ -702,6 +702,37 @@ list_member_oid(const List *list, Oid datum) return false; } +/* + * return true iff every entry in "members" list is also present + * in the "target" list. + */ +bool +list_is_subset(const List *members, const List *target) +{ + const ListCell *lc1, *lc2; + + Assert(IsPointerList(members)); + Assert(IsPointerList(target)); + check_list_invariants(members); + check_list_invariants(target); + + foreach(lc1, members) + { + bool found = false; + foreach(lc2, target) + { + if (equal(lfirst(lc1), lfirst(lc2))) + { + found = true; + break; + } + } + if (!found) + return false; + } + return true; +} + /* * Delete the n'th cell (counting from 0) in list. * diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 01c110cd2f..40415d0f5b 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -815,3 +815,16 @@ makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols) v->va_cols = va_cols; return v; } + + +/* + * makeUniqueKey + */ +UniqueKey* +makeUniqueKey(List *exprs, bool multi_nullvals) +{ + UniqueKey * ukey = makeNode(UniqueKey); + ukey->exprs = exprs; + ukey->multi_nullvals = multi_nullvals; + return ukey; +} diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..44154cde6a 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2456,6 +2456,14 @@ _outPathKey(StringInfo str, const PathKey *node) WRITE_BOOL_FIELD(pk_nulls_first); } +static void +_outUniqueKey(StringInfo str, const UniqueKey *node) +{ + WRITE_NODE_TYPE("UNIQUEKEY"); + WRITE_NODE_FIELD(exprs); + WRITE_BOOL_FIELD(multi_nullvals); +} + static void _outPathTarget(StringInfo str, const PathTarget *node) { @@ -4198,6 +4206,9 @@ outNode(StringInfo str, const void *obj) case T_PathKey: _outPathKey(str, obj); break; + case T_UniqueKey: + _outUniqueKey(str, obj); + break; case T_PathTarget: _outPathTarget(str, obj); break; diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index c5e136e9c3..b3e212bf1c 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -492,6 +492,14 @@ _readSetOperationStmt(void) READ_DONE(); } +static UniqueKey * +_readUniqueKey(void) +{ + READ_LOCALS(UniqueKey); + READ_NODE_FIELD(exprs); + READ_BOOL_FIELD(multi_nullvals); + READ_DONE(); +} /* * Stuff from primnodes.h. @@ -2717,6 +2725,8 @@ parseNodeString(void) return_value = _readCommonTableExpr(); else if (MATCH("SETOPERATIONSTMT", 16)) return_value = _readSetOperationStmt(); + else if (MATCH("UNIQUEKEY", 9)) + return_value = _readUniqueKey(); else if (MATCH("ALIAS", 5)) return_value = _readAlias(); else if (MATCH("RANGEVAR", 8)) diff --git a/src/backend/optimizer/path/Makefile b/src/backend/optimizer/path/Makefile index 1e199ff66f..7b9820c25f 100644 --- a/src/backend/optimizer/path/Makefile +++ b/src/backend/optimizer/path/Makefile @@ -21,6 +21,7 @@ OBJS = \ joinpath.o \ joinrels.o \ pathkeys.o \ - tidpath.o + tidpath.o \ + uniquekeys.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/optimizer/path/README.uniquekey b/src/backend/optimizer/path/README.uniquekey new file mode 100644 index 0000000000..5eac761995 --- /dev/null +++ b/src/backend/optimizer/path/README.uniquekey @@ -0,0 +1,131 @@ +1. What is UniqueKey? +We can think UniqueKey is a set of exprs for a RelOptInfo, which we are insure +that doesn't yields same result among all the rows. The simplest UniqueKey +format is primary key. + +However we define the UnqiueKey as below. + +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + +exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL +is a special case of UniqueKey, which means there is only one row in that +relation.it has a stronger semantic than others. like SELECT uk FROM t; uk is +normal unique key and may have different values. SELECT colx FROM t WHERE uk = +const. colx is unique AND we have only 1 value. This field can used for +innerrel_is_unique. this logic is handled specially in add_uniquekey_for_onerow +function. + +multi_nullvals: true means multi null values may exist in these exprs, so the +uniqueness is not guaranteed in this case. This field is necessary for +remove_useless_join & reduce_unique_semijoins where we don't mind these +duplicated NULL values. It is set to true for 2 cases. One is a unique key +from a unique index but the related column is nullable. The other one is for +outer join. see populate_joinrel_uniquekeys for detail. + + +The UniqueKey can be used at the following cases at least: +1. remove_useless_joins. +2. reduce_semianti_joins +3. remove distinct node if distinct clause is unique. +4. remove aggnode if group by clause is unique. +5. Index Skip Scan (WIP) +6. Aggregation Push Down without 2 phase aggregation if the join can't + duplicated the aggregated rows. (work in progress feature) + +2. How is it maintained? + +We have a set of populate_xxx_unqiuekeys functions to maintain the uniquekey on +various cases. xxx includes baserel, joinrel, partitionedrel, distinctrel, +groupedrel, unionrel. and we also need to convert the uniquekey from subquery +to outer relation, which is what convert_subquery_uniquekeys does. + +1. The first part is about baserel. We handled 3 cases. suppose we have Unique +Index on (a, b). + +1. SELECT a, b FROM t. UniqueKey (a, b) +2. SELECT a FROM t WHERE b = 1; UniqueKey (a) +3. SELECT .. FROM t WHERE a = 1 AND b = 1; UniqueKey (NIL). onerow case, every + column is Unique. + +2. The next part is joinrel, this part is most error-prone, we simplified the rules +like below: +1. If the relation's UniqueKey can't be duplicated after join, then is will be + still valid for the join rel. The function we used here is + innerrel_keeps_unique. The basic idea is innerrel.any_col = outer.uk. + +2. If the UnqiueKey can't keep valid via the rule 1, the combination of the + UniqueKey from both sides are valid for sure. We can prove this as: if the + unique exprs from rel1 is duplicated by rel2, the duplicated rows must + contains different unique exprs from rel2. + +More considerations about onerow: +1. If relation with one row and it can't be duplicated, it is still possible + contains mulit_nullvas after outer join. +2. If the either UniqueKey can be duplicated after join, the can get one row + only when both side is one row AND there is no outer join. +3. Whenever the onerow UniqueKey is not a valid any more, we need to convert one + row UniqueKey to normal unique key since we don't store exprs for one-row + relation. get_exprs_from_uniquekeys will be used here. + + +More considerations about multi_nullvals after join: +1. If the original UnqiueKey has multi_nullvals, the final UniqueKey will have + mulit_nullvals in any case. +2. If a unique key doesn't allow mulit_nullvals, after some outer join, it + allows some outer join. + + +3. When we comes to subquery, we need to convert_subquery_unqiuekeys just like +convert_subquery_pathkeys. Only the UniqueKey insides subquery is referenced as +a Var in outer relation will be reused. The relationship between the outerrel.Var +and subquery.exprs is built with outerel->subroot->processed_tlist. + + +4. As for the SRF functions, it will break the uniqueness of uniquekey, However it +is handled in adjust_paths_for_srfs, which happens after the query_planner. so +we will maintain the UniqueKey until there and reset it to NIL at that +places. This can't help on distinct/group by elimination cases but probably help +in some other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is +semantic correctly. + + +5. As for inherit table, we first main the UnqiueKey on childrel as well. But for +partitioned table we need to maintain 2 different kinds of +UnqiueKey. 1). UniqueKey on the parent relation 2). UniqueKey on child +relation for partition wise query. + +Example: +CREATE TABLE p (a int not null, b int not null) partition by list (a); +CREATE TABLE p0 partition of p for values in (1); +CREATE TABLE p1 partition of p for values in (2); + +create unique index p0_b on p0(b); +create unique index p1_b on p1(b); + +Now b is only unique on partition level, so the distinct can't be removed on +the following cases. SELECT DISTINCT b FROM p; + +Another example is SELECT DISTINCT a, b FROM p WHERE a = 1; Since only one +partition is chosen, the UniqueKey on child relation is same as the UniqueKey on +parent relation. + +Another usage of UniqueKey on partition level is it be helpful for +partition-wise join. + +As for the UniqueKey on parent table level, it comes with 2 different ways, +1). the UniqueKey is also derived in UniqueKey index, but the index must be same +in all the related children relations and the unique index must contains +Partition Key in it. Example: + +CREATE UNIQUE INDEX p_ab ON p(a, b); -- where a is the partition key. + +-- Query +SELECT a, b FROM p; the (a, b) is a UniqueKey of p. + +2). If the parent relation has only one childrel, the UniqueKey on childrel is + the UniqueKey on parent as well. diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index 37b4223adb..d52ad59f89 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -579,6 +579,12 @@ set_plain_rel_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) */ check_index_predicates(root, rel); + /* + * Now that we've marked which partial indexes are suitable, we can now + * build the relation's unique keys. + */ + populate_baserel_uniquekeys(root, rel, rel->indexlist); + /* Mark rel with estimated output rows, width, etc */ set_baserel_size_estimates(root, rel); } @@ -1297,6 +1303,8 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, /* Add paths to the append relation. */ add_paths_to_append_rel(root, rel, live_childrels); + if (IS_PARTITIONED_REL(rel)) + populate_partitionedrel_uniquekeys(root, rel, live_childrels); } @@ -2304,6 +2312,8 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, pathkeys, required_outer)); } + convert_subquery_uniquekeys(root, rel, sub_final_rel); + /* If outer rel allows parallelism, do same for partial paths. */ if (rel->consider_parallel && bms_is_empty(required_outer)) { diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c index 57ce97fd53..697ac047fb 100644 --- a/src/backend/optimizer/path/joinpath.c +++ b/src/backend/optimizer/path/joinpath.c @@ -71,13 +71,6 @@ static void consider_parallel_mergejoin(PlannerInfo *root, static void hash_inner_and_outer(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, RelOptInfo *innerrel, JoinType jointype, JoinPathExtraData *extra); -static List *select_mergejoin_clauses(PlannerInfo *root, - RelOptInfo *joinrel, - RelOptInfo *outerrel, - RelOptInfo *innerrel, - List *restrictlist, - JoinType jointype, - bool *mergejoin_allowed); static void generate_mergejoin_paths(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *innerrel, @@ -1927,7 +1920,7 @@ hash_inner_and_outer(PlannerInfo *root, * if it is mergejoinable and involves vars from the two sub-relations * currently of interest. */ -static List * +List * select_mergejoin_clauses(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c index 0dbe2ac726..7271f044ec 100644 --- a/src/backend/optimizer/path/joinrels.c +++ b/src/backend/optimizer/path/joinrels.c @@ -924,6 +924,8 @@ populate_joinrel_with_paths(PlannerInfo *root, RelOptInfo *rel1, /* Apply partitionwise join technique, if possible. */ try_partitionwise_join(root, rel1, rel2, joinrel, sjinfo, restrictlist); + + populate_joinrel_uniquekeys(root, joinrel, rel1, rel2, restrictlist, sjinfo->jointype); } diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index bd9a176d7d..139278829b 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -33,7 +33,6 @@ static bool pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys); static bool matches_boolean_partition_clause(RestrictInfo *rinfo, RelOptInfo *partrel, int partkeycol); -static Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); static bool right_merge_direction(PlannerInfo *root, PathKey *pathkey); @@ -1035,7 +1034,7 @@ convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel, * We need this to ensure that we don't return pathkeys describing values * that are unavailable above the level of the subquery scan. */ -static Var * +Var * find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle) { ListCell *lc; diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c new file mode 100644 index 0000000000..77ed2b2eff --- /dev/null +++ b/src/backend/optimizer/path/uniquekeys.c @@ -0,0 +1,1135 @@ +/*------------------------------------------------------------------------- + * + * uniquekeys.c + * Utilities for matching and building unique keys + * + * Portions Copyright (c) 2020, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/optimizer/path/uniquekeys.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "nodes/makefuncs.h" +#include "nodes/nodeFuncs.h" +#include "optimizer/pathnode.h" +#include "optimizer/paths.h" +#include "optimizer/appendinfo.h" +#include "optimizer/optimizer.h" +#include "optimizer/tlist.h" +#include "rewrite/rewriteManip.h" + + +/* + * This struct is used to help populate_joinrel_uniquekeys. + * + * added_to_joinrel is true if a uniquekey (from outerrel or innerrel) + * has been added to joinrel. + * useful is true if the exprs of the uniquekey still appears in joinrel. + */ +typedef struct UniqueKeyContextData +{ + UniqueKey *uniquekey; + bool added_to_joinrel; + bool useful; +} *UniqueKeyContext; + +static List *initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel); +static bool innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + bool reverse); + +static List *get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals); +static List *get_exprs_from_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + UniqueKey *ukey); +static void add_uniquekey_for_onerow(RelOptInfo *rel); +static bool add_combined_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype); + +/* Used for unique indexes checking for partitioned table */ +static bool index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind); +static IndexOptInfo *simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from); +static bool simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2); +static void adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_index); + +/* Helper function for grouped relation and distinct relation. */ +static void add_uniquekey_from_sortgroups(PlannerInfo *root, + RelOptInfo *rel, + List *sortgroups); + +/* + * populate_baserel_uniquekeys + * Populate 'baserel' uniquekeys list by looking at the rel's unique index + * and baserestrictinfo + */ +void +populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List *indexlist) +{ + ListCell *lc; + List *matched_uniq_indexes = NIL; + + /* Attrs appears in rel->reltarget->exprs. */ + Bitmapset *used_attrs = NULL; + + List *const_exprs = NIL; + List *expr_opfamilies = NIL; + + Assert(baserel->rtekind == RTE_RELATION); + + foreach(lc, indexlist) + { + IndexOptInfo *ind = (IndexOptInfo *) lfirst(lc); + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + matched_uniq_indexes = lappend(matched_uniq_indexes, ind); + } + + if (matched_uniq_indexes == NIL) + return; + + /* Check which attrs is used in baserel->reltarget */ + pull_varattnos((Node *)baserel->reltarget->exprs, baserel->relid, &used_attrs); + + /* Check which attrno is used at a mergeable const filter */ + foreach(lc, baserel->baserestrictinfo) + { + RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); + + if (rinfo->mergeopfamilies == NIL) + continue; + + if (bms_is_empty(rinfo->left_relids)) + { + const_exprs = lappend(const_exprs, get_rightop(rinfo->clause)); + } + else if (bms_is_empty(rinfo->right_relids)) + { + const_exprs = lappend(const_exprs, get_leftop(rinfo->clause)); + } + else + continue; + + expr_opfamilies = lappend(expr_opfamilies, rinfo->mergeopfamilies); + } + + foreach(lc, matched_uniq_indexes) + { + bool multi_nullvals, useful; + List *exprs = get_exprs_from_uniqueindex(lfirst_node(IndexOptInfo, lc), + const_exprs, + expr_opfamilies, + used_attrs, + &useful, + &multi_nullvals); + if (useful) + { + if (exprs == NIL) + { + /* All the columns in Unique Index matched with a restrictinfo */ + add_uniquekey_for_onerow(baserel); + return; + } + baserel->uniquekeys = lappend(baserel->uniquekeys, + makeUniqueKey(exprs, multi_nullvals)); + } + } +} + + +/* + * populate_partitionedrel_uniquekeys + * The UniqueKey on partitionrel comes from 2 cases: + * 1). Only one partition is involved in this query, the unique key can be + * copied to parent rel from childrel. + * 2). There are some unique index which includes partition key and exists + * in all the related partitions. + * We never mind rule 2 if we hit rule 1. + */ + +void +populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels) +{ + ListCell *lc; + List *global_uniq_indexlist = NIL; + RelOptInfo *childrel; + bool is_first = true; + + Assert(IS_PARTITIONED_REL(rel)); + + if (childrels == NIL) + return; + + /* + * If there is only one partition used in this query, the UniqueKey in childrel is + * still valid in parent level, but we need convert the format from child expr to + * parent expr. + */ + if (list_length(childrels) == 1) + { + /* Check for Rule 1 */ + RelOptInfo *childrel = linitial_node(RelOptInfo, childrels); + ListCell *lc; + Assert(childrel->reloptkind == RELOPT_OTHER_MEMBER_REL); + if (relation_is_onerow(childrel)) + { + add_uniquekey_for_onerow(rel); + return; + } + + foreach(lc, childrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + AppendRelInfo *appinfo = find_appinfo_by_child(root, childrel->relid); + List *parent_exprs = NIL; + bool can_reuse = true; + ListCell *lc2; + foreach(lc2, ukey->exprs) + { + Var *var = (Var *)lfirst(lc2); + /* + * If the expr comes from a expression, it is hard to build the expression + * in parent so ignore that case for now. + */ + if(!IsA(var, Var)) + { + can_reuse = false; + break; + } + /* Convert it to parent var */ + parent_exprs = lappend(parent_exprs, find_parent_var(appinfo, var)); + } + if (can_reuse) + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(parent_exprs, + ukey->multi_nullvals)); + } + } + else + { + /* Check for rule 2 */ + childrel = linitial_node(RelOptInfo, childrels); + foreach(lc, childrel->indexlist) + { + IndexOptInfo *ind = lfirst(lc); + IndexOptInfo *modified_index; + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + + /* + * During simple_copy_indexinfo_to_parent, we need to convert var from + * child var to parent var, index on expression is too complex to handle. + * so ignore it for now. + */ + if (ind->indexprs != NIL) + continue; + + modified_index = simple_copy_indexinfo_to_parent(root, rel, ind); + /* + * If the unique index doesn't contain partkey, then it is unique + * on this partition only, so it is useless for us. + */ + if (!index_constains_partkey(rel, modified_index)) + continue; + + global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index); + } + + if (global_uniq_indexlist != NIL) + { + foreach(lc, childrels) + { + RelOptInfo *child = lfirst(lc); + if (is_first) + { + is_first = false; + continue; + } + adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist); + } + /* Now we have a list of unique index which are exactly same on all childrels, + * Set the UniqueKey just like it is non-partition table + */ + populate_baserel_uniquekeys(root, rel, global_uniq_indexlist); + } + } +} + + +/* + * populate_distinctrel_uniquekeys + */ +void +populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel) +{ + /* The unique key before the distinct is still valid. */ + distinctrel->uniquekeys = list_copy(inputrel->uniquekeys); + add_uniquekey_from_sortgroups(root, distinctrel, root->parse->distinctClause); +} + +/* + * populate_grouprel_uniquekeys + */ +void +populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel) + +{ + Query *parse = root->parse; + bool input_ukey_added = false; + ListCell *lc; + + if (relation_is_onerow(inputrel)) + { + add_uniquekey_for_onerow(grouprel); + return; + } + if (parse->groupingSets) + return; + + /* A Normal group by without grouping set. */ + if (parse->groupClause) + { + /* + * Current even the groupby clause is Unique already, but if query has aggref + * We have to create grouprel still. To keep the UnqiueKey short, we will check + * the UniqueKey of input_rel still valid, if so we reuse it. + */ + foreach(lc, inputrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (list_is_subset(ukey->exprs, grouprel->reltarget->exprs)) + { + grouprel->uniquekeys = lappend(grouprel->uniquekeys, + ukey); + input_ukey_added = true; + } + } + if (!input_ukey_added) + /* + * group by clause must be a super-set of grouprel->reltarget->exprs except the + * aggregation expr, so if such exprs is unique already, no bother to generate + * new uniquekey for group by exprs. + */ + add_uniquekey_from_sortgroups(root, + grouprel, + root->parse->groupClause); + } + else + /* It has aggregation but without a group by, so only one row returned */ + add_uniquekey_for_onerow(grouprel); +} + +/* + * simple_copy_uniquekeys + * Using a function for the one-line code makes us easy to check where we simply + * copied the uniquekey. + */ +void +simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel) +{ + newrel->uniquekeys = oldrel->uniquekeys; +} + +/* + * populate_unionrel_uniquekeys + */ +void +populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel) +{ + ListCell *lc; + List *exprs = NIL; + + Assert(unionrel->uniquekeys == NIL); + + foreach(lc, unionrel->reltarget->exprs) + { + exprs = lappend(exprs, lfirst(lc)); + } + + if (exprs == NIL) + /* SQL: select union select; is valid, we need to handle it here. */ + add_uniquekey_for_onerow(unionrel); + else + unionrel->uniquekeys = lappend(unionrel->uniquekeys, + makeUniqueKey(exprs,false)); + +} + +/* + * populate_joinrel_uniquekeys + * + * populate uniquekeys for joinrel. We will check each relation to see if its + * UniqueKey is still valid via innerrel_keeps_unique, if so, we add it to + * joinrel. The multi_nullvals field will be changed to true for some outer + * join cases and one-row UniqueKey needs to be converted to normal UniqueKey + * for the same case as well. + * For the uniquekey in either baserel which can't be unique after join, we still + * check to see if combination of UniqueKeys from both side is still useful for us. + * if yes, we add it to joinrel as well. + */ +void +populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *outerrel, RelOptInfo *innerrel, + List *restrictlist, JoinType jointype) +{ + ListCell *lc, *lc2; + List *clause_list = NIL; + List *outerrel_ukey_ctx; + List *innerrel_ukey_ctx; + bool inner_onerow, outer_onerow; + bool mergejoin_allowed; + + /* Care about the outerrel relation only for SEMI/ANTI join */ + if (jointype == JOIN_SEMI || jointype == JOIN_ANTI) + { + foreach(lc, outerrel->uniquekeys) + { + UniqueKey *uniquekey = lfirst_node(UniqueKey, lc); + if (list_is_subset(uniquekey->exprs, joinrel->reltarget->exprs)) + joinrel->uniquekeys = lappend(joinrel->uniquekeys, uniquekey); + } + return; + } + + Assert(jointype == JOIN_LEFT || jointype == JOIN_FULL || jointype == JOIN_INNER); + + /* Fast path */ + if (innerrel->uniquekeys == NIL || outerrel->uniquekeys == NIL) + return; + + inner_onerow = relation_is_onerow(innerrel); + outer_onerow = relation_is_onerow(outerrel); + + outerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(outerrel); + innerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(innerrel); + + clause_list = select_mergejoin_clauses(root, joinrel, outerrel, innerrel, + restrictlist, jointype, + &mergejoin_allowed); + + if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true /* reverse */)) + { + bool outer_impact = jointype == JOIN_FULL; + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + /* Outer relation has one row, and the unique key is not duplicated after join, + * the joinrel will still has one row unless the jointype == JOIN_FULL. + */ + if (outer_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (outer_onerow) + { + /* + * The onerow outerrel becomes multi rows and multi_nullvals + * will be changed to true. We also need to set the exprs correctly since it + * can't be NIL any more. + */ + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, outerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Change multi_nullvals to true due to the full join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, true)); + else + /* Just reuse it */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + } + ctx->added_to_joinrel = true; + } + } + + if (innerrel_keeps_unique(root, outerrel, innerrel, clause_list, false)) + { + bool outer_impact = jointype == JOIN_FULL || jointype == JOIN_LEFT;; + + foreach(lc, innerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + if (inner_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (inner_onerow) + { + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, innerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Need to change multi_nullvals to true due to the outer join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, + true)); + else + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + + } + ctx->added_to_joinrel = true; + } + } + + /* + * The combination of the UniqueKey from both sides is unique as well regardless + * of join type, but no bother to add it if its subset has been added to joinrel + * already or it is not useful for the joinrel. + */ + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx1 = (UniqueKeyContext) lfirst(lc); + if (ctx1->added_to_joinrel || !ctx1->useful) + continue; + foreach(lc2, innerrel_ukey_ctx) + { + UniqueKeyContext ctx2 = (UniqueKeyContext) lfirst(lc2); + if (ctx2->added_to_joinrel || !ctx2->useful) + continue; + if (add_combined_uniquekey(root, joinrel, outerrel, innerrel, + ctx1->uniquekey, ctx2->uniquekey, + jointype)) + /* If we set a onerow UniqueKey to joinrel, we don't need other. */ + return; + } + } +} + + +/* + * convert_subquery_uniquekeys + * + * Covert the UniqueKey in subquery to outer relation. + */ +void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel) +{ + ListCell *lc; + + if (sub_final_rel->uniquekeys == NIL) + return; + + if (relation_is_onerow(sub_final_rel)) + { + add_uniquekey_for_onerow(currel); + return; + } + + Assert(currel->subroot != NULL); + + foreach(lc, sub_final_rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + ListCell *lc; + List *exprs = NIL; + bool ukey_useful = true; + + /* One row case is handled above */ + Assert(ukey->exprs != NIL); + foreach(lc, ukey->exprs) + { + Var *var; + TargetEntry *tle = tlist_member(lfirst(lc), + currel->subroot->processed_tlist); + if (tle == NULL) + { + ukey_useful = false; + break; + } + var = find_var_for_subquery_tle(currel, tle); + if (var == NULL) + { + ukey_useful = false; + break; + } + exprs = lappend(exprs, var); + } + + if (ukey_useful) + currel->uniquekeys = lappend(currel->uniquekeys, + makeUniqueKey(exprs, + ukey->multi_nullvals)); + + } +} + +/* + * innerrel_keeps_unique + * + * Check if Unique key of the innerrel is valid after join. innerrel's UniqueKey + * will be still valid if innerrel's any-column mergeop outrerel's uniquekey + * exists in clause_list. + * + * Note: the clause_list must be a list of mergeable restrictinfo already. + */ +static bool +innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *clause_list, + bool reverse) +{ + ListCell *lc, *lc2, *lc3; + + if (outerrel->uniquekeys == NIL || innerrel->uniquekeys == NIL) + return false; + + /* Check if there is outerrel's uniquekey in mergeable clause. */ + foreach(lc, outerrel->uniquekeys) + { + List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs; + bool clauselist_matchs_all_exprs = true; + foreach(lc2, outer_uq_exprs) + { + Node *outer_uq_expr = lfirst(lc2); + bool find_uq_expr_in_clauselist = false; + foreach(lc3, clause_list) + { + RestrictInfo *rinfo = lfirst_node(RestrictInfo, lc3); + Node *outer_expr; + if (reverse) + outer_expr = rinfo->outer_is_left ? get_rightop(rinfo->clause) : get_leftop(rinfo->clause); + else + outer_expr = rinfo->outer_is_left ? get_leftop(rinfo->clause) : get_rightop(rinfo->clause); + if (equal(outer_expr, outer_uq_expr)) + { + find_uq_expr_in_clauselist = true; + break; + } + } + if (!find_uq_expr_in_clauselist) + { + /* No need to check the next exprs in the current uniquekey */ + clauselist_matchs_all_exprs = false; + break; + } + } + + if (clauselist_matchs_all_exprs) + return true; + } + return false; +} + + +/* + * relation_is_onerow + * Check if it is a one-row relation by checking UniqueKey. + */ +bool +relation_is_onerow(RelOptInfo *rel) +{ + UniqueKey *ukey; + if (rel->uniquekeys == NIL) + return false; + ukey = linitial_node(UniqueKey, rel->uniquekeys); + return ukey->exprs == NIL && list_length(rel->uniquekeys) == 1; +} + +/* + * relation_has_uniquekeys_for + * Returns true if we have proofs that 'rel' cannot return multiple rows with + * the same values in each of 'exprs'. Otherwise returns false. + */ +bool +relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel, + List *exprs, bool allow_multinulls) +{ + ListCell *lc; + + /* + * For UniqueKey->onerow case, the uniquekey->exprs is empty as well + * so we can't rely on list_is_subset to handle this special cases + */ + if (exprs == NIL) + return false; + + foreach(lc, rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (ukey->multi_nullvals && !allow_multinulls) + continue; + if (list_is_subset(ukey->exprs, exprs)) + return true; + } + return false; +} + + +/* + * get_exprs_from_uniqueindex + * + * Return a list of exprs which is unique. set useful to false if this + * unique index is not useful for us. + */ +static List * +get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals) +{ + List *exprs = NIL; + ListCell *indexpr_item; + int c = 0; + + *useful = true; + *multi_nullvals = false; + + indexpr_item = list_head(unique_index->indexprs); + for(c = 0; c < unique_index->ncolumns; c++) + { + int attr = unique_index->indexkeys[c]; + Expr *expr; + bool matched_const = false; + ListCell *lc1, *lc2; + + if(attr > 0) + { + expr = list_nth_node(TargetEntry, unique_index->indextlist, c)->expr; + } + else if (attr == 0) + { + /* Expression index */ + expr = lfirst(indexpr_item); + indexpr_item = lnext(unique_index->indexprs, indexpr_item); + } + else /* attr < 0 */ + { + /* Index on system column is not supported */ + Assert(false); + } + + /* + * Check index_col = Const case with regarding to opfamily checking + * If we can remove the index_col from the final UniqueKey->exprs. + */ + forboth(lc1, const_exprs, lc2, const_expr_opfamilies) + { + if (list_member_oid((List *)lfirst(lc2), unique_index->opfamily[c]) + && match_index_to_operand((Node *) lfirst(lc1), c, unique_index)) + { + matched_const = true; + break; + } + } + + if (matched_const) + continue; + + /* Check if the indexed expr is used in rel */ + if (attr > 0) + { + /* + * Normal Indexed column, if the col is not used, then the index is useless + * for uniquekey. + */ + attr -= FirstLowInvalidHeapAttributeNumber; + + if (!bms_is_member(attr, used_varattrs)) + { + *useful = false; + break; + } + } + else if (!list_member(unique_index->rel->reltarget->exprs, expr)) + { + /* Expression index but the expression is not used in rel */ + *useful = false; + break; + } + + /* check not null property. */ + if (attr == 0) + { + /* We never know if a expression yields null or not */ + *multi_nullvals = true; + } + else if (!bms_is_member(attr, unique_index->rel->notnullattrs) + && !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber, + unique_index->rel->notnullattrs)) + { + *multi_nullvals = true; + } + + exprs = lappend(exprs, expr); + } + return exprs; +} + + +/* + * add_uniquekey_for_onerow + * If we are sure that the relation only returns one row, then all the columns + * are unique. However we don't need to create UniqueKey for every column, we + * just set exprs = NIL and overwrites all the other UniqueKey on this RelOptInfo + * since this one has strongest semantics. + */ +void +add_uniquekey_for_onerow(RelOptInfo *rel) +{ + /* + * We overwrite the previous UniqueKey on purpose since this one has the + * strongest semantic. + */ + rel->uniquekeys = list_make1(makeUniqueKey(NIL, false)); +} + + +/* + * initililze_uniquecontext_for_joinrel + * Return a List of UniqueKeyContext for an inputrel + */ +static List * +initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel) +{ + List *res = NIL; + ListCell *lc; + foreach(lc, inputrel->uniquekeys) + { + UniqueKeyContext context; + context = palloc(sizeof(struct UniqueKeyContextData)); + context->uniquekey = lfirst_node(UniqueKey, lc); + context->added_to_joinrel = false; + context->useful = true; + res = lappend(res, context); + } + return res; +} + + +/* + * get_exprs_from_uniquekey + * Unify the way of get List of exprs from a one-row UniqueKey or + * normal UniqueKey. for the onerow case, every expr in rel1 is a valid + * UniqueKey. Return a List of exprs. + * + * rel1: The relation which you want to get the exprs. + * ukey: The UniqueKey you want to get the exprs. + */ +static List * +get_exprs_from_uniquekey(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *rel1, UniqueKey *ukey) +{ + ListCell *lc; + bool onerow = rel1 != NULL && relation_is_onerow(rel1); + + List *res = NIL; + Assert(onerow || ukey); + if (onerow) + { + /* Only cares about the exprs still exist in joinrel */ + foreach(lc, joinrel->reltarget->exprs) + { + Bitmapset *relids = pull_varnos(root, lfirst(lc)); + if (bms_is_subset(relids, rel1->relids)) + { + res = lappend(res, list_make1(lfirst(lc))); + } + } + } + else + { + res = list_make1(ukey->exprs); + } + return res; +} + +/* + * Partitioned table Unique Keys. + * The partition table unique key is maintained as: + * 1. The index must be unique as usual. + * 2. The index must contains partition key. + * 3. The index must exist on all the child rel. see simple_indexinfo_equal for + * how we compare it. + */ + +/* + * index_constains_partkey + * return true if the index contains the partiton key. + */ +static bool +index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind) +{ + ListCell *lc; + int i; + Assert(IS_PARTITIONED_REL(partrel)); + Assert(partrel->part_scheme->partnatts > 0); + + for(i = 0; i < partrel->part_scheme->partnatts; i++) + { + Node *part_expr = linitial(partrel->partexprs[i]); + bool found_in_index = false; + foreach(lc, ind->indextlist) + { + Expr *index_expr = lfirst_node(TargetEntry, lc)->expr; + if (equal(index_expr, part_expr)) + { + found_in_index = true; + break; + } + } + if (!found_in_index) + return false; + } + return true; +} + +/* + * simple_indexinfo_equal + * + * Used to check if the 2 index is same as each other. The index here + * is COPIED from childrel and did some tiny changes(see + * simple_copy_indexinfo_to_parent) + */ +static bool +simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2) +{ + Size oid_cmp_len = sizeof(Oid) * ind1->ncolumns; + + return ind1->ncolumns == ind2->ncolumns && + ind1->unique == ind2->unique && + memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0 && + memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0 && + memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0 && + memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0 && + equal(get_tlist_exprs(ind1->indextlist, true), + get_tlist_exprs(ind2->indextlist, true)); +} + + +/* + * The below macros are used for simple_copy_indexinfo_to_parent which is so + * customized that I don't want to put it to copyfuncs.c. So copy it here. + */ +#define COPY_POINTER_FIELD(fldname, sz) \ + do { \ + Size _size = (sz); \ + newnode->fldname = palloc(_size); \ + memcpy(newnode->fldname, from->fldname, _size); \ + } while (0) + +#define COPY_NODE_FIELD(fldname) \ + (newnode->fldname = copyObjectImpl(from->fldname)) + +#define COPY_SCALAR_FIELD(fldname) \ + (newnode->fldname = from->fldname) + + +/* + * simple_copy_indexinfo_to_parent (from partition) + * Copy the IndexInfo from child relation to parent relation with some modification, + * which is used to test: + * 1. If the same index exists in all the childrels. + * 2. If the parentrel->reltarget/basicrestrict info matches this index. + */ +static IndexOptInfo * +simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from) +{ + IndexOptInfo *newnode = makeNode(IndexOptInfo); + AppendRelInfo *appinfo = find_appinfo_by_child(root, from->rel->relid); + ListCell *lc; + int idx = 0; + + COPY_SCALAR_FIELD(ncolumns); + COPY_SCALAR_FIELD(nkeycolumns); + COPY_SCALAR_FIELD(unique); + COPY_SCALAR_FIELD(immediate); + /* We just need to know if it is NIL or not */ + COPY_SCALAR_FIELD(indpred); + COPY_SCALAR_FIELD(predOK); + COPY_POINTER_FIELD(indexkeys, from->ncolumns * sizeof(int)); + COPY_POINTER_FIELD(indexcollations, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opfamily, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opcintype, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(sortopfamily, from->ncolumns * sizeof(Oid)); + COPY_NODE_FIELD(indextlist); + + /* Convert index exprs on child expr to expr on parent */ + foreach(lc, newnode->indextlist) + { + TargetEntry *tle = lfirst_node(TargetEntry, lc); + /* Index on expression is ignored */ + Assert(IsA(tle->expr, Var)); + tle->expr = (Expr *) find_parent_var(appinfo, (Var *) tle->expr); + newnode->indexkeys[idx] = castNode(Var, tle->expr)->varattno; + idx++; + } + newnode->rel = parentrel; + return newnode; +} + +/* + * adjust_partition_unique_indexlist + * + * global_unique_indexes: At the beginning, it contains the copy & modified + * unique index from the first partition. And then check if each index in it still + * exists in the following partitions. If no, remove it. at last, it has an + * index list which exists in all the partitions. + */ +static void +adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_indexes) +{ + ListCell *lc, *lc2; + foreach(lc, *global_unique_indexes) + { + IndexOptInfo *g_ind = lfirst_node(IndexOptInfo, lc); + bool found_in_child = false; + + foreach(lc2, childrel->indexlist) + { + IndexOptInfo *p_ind = lfirst_node(IndexOptInfo, lc2); + IndexOptInfo *p_ind_copy; + if (!p_ind->unique || !p_ind->immediate || + (p_ind->indpred != NIL && !p_ind->predOK)) + continue; + p_ind_copy = simple_copy_indexinfo_to_parent(root, parentrel, p_ind); + if (simple_indexinfo_equal(p_ind_copy, g_ind)) + { + found_in_child = true; + break; + } + } + if (!found_in_child) + /* The index doesn't exist in childrel, remove it from global_unique_indexes */ + *global_unique_indexes = foreach_delete_current(*global_unique_indexes, lc); + } +} + +/* Helper function for groupres/distinctrel */ +static void +add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgroups) +{ + Query *parse = root->parse; + List *exprs; + + /* + * XXX: If there are some vars which is not in current levelsup, the semantic is + * imprecise, should we avoid it or not? levelsup = 1 is just a demo, maybe we need to + * check every level other than 0, if so, looks we have to write another + * pull_var_walker. + */ + List *upper_vars = pull_vars_of_level((Node*)sortgroups, 1); + + if (upper_vars != NIL) + return; + + exprs = get_sortgrouplist_exprs(sortgroups, parse->targetList); + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(exprs, + false /* sortgroupclause can't be multi_nullvals */)); +} + + +/* + * add_combined_uniquekey + * The combination of both UniqueKeys is a valid UniqueKey for joinrel no matter + * the jointype. + */ +bool +add_combined_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype) +{ + + ListCell *lc1, *lc2; + + /* Either side has multi_nullvals or we have outer join, + * the combined UniqueKey has multi_nullvals */ + bool multi_nullvals = outer_ukey->multi_nullvals || + inner_ukey->multi_nullvals || IS_OUTER_JOIN(jointype); + + /* The only case we can get onerow joinrel after join */ + if (relation_is_onerow(outer_rel) + && relation_is_onerow(inner_rel) + && jointype == JOIN_INNER) + { + add_uniquekey_for_onerow(joinrel); + return true; + } + + foreach(lc1, get_exprs_from_uniquekey(root, joinrel, outer_rel, outer_ukey)) + { + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, inner_rel, inner_ukey)) + { + List *exprs = list_concat_copy(lfirst_node(List, lc1), lfirst_node(List, lc2)); + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(exprs, + multi_nullvals)); + } + } + return false; +} diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index 424d25cbd5..8d8e493f5c 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -2387,6 +2387,8 @@ grouping_planner(PlannerInfo *root, bool inheritance_update, add_path(final_rel, path); } + simple_copy_uniquekeys(current_rel, final_rel); + /* * Generate partial paths for final_rel, too, if outer query levels might * be able to make use of them. @@ -3899,6 +3901,8 @@ create_grouping_paths(PlannerInfo *root, } set_cheapest(grouped_rel); + + populate_grouprel_uniquekeys(root, grouped_rel, input_rel); return grouped_rel; } @@ -4621,7 +4625,7 @@ create_window_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(window_rel); - + simple_copy_uniquekeys(input_rel, window_rel); return window_rel; } @@ -4935,7 +4939,7 @@ create_distinct_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(distinct_rel); - + populate_distinctrel_uniquekeys(root, input_rel, distinct_rel); return distinct_rel; } @@ -5196,6 +5200,8 @@ create_ordered_paths(PlannerInfo *root, */ Assert(ordered_rel->pathlist != NIL); + simple_copy_uniquekeys(input_rel, ordered_rel); + return ordered_rel; } @@ -6073,6 +6079,9 @@ adjust_paths_for_srfs(PlannerInfo *root, RelOptInfo *rel, if (list_length(targets) == 1) return; + /* UniqueKey is not valid after handling the SRF. */ + rel->uniquekeys = NIL; + /* * Stack SRF-evaluation nodes atop each path for the rel. * diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c index becdcbb872..b7626545bf 100644 --- a/src/backend/optimizer/prep/prepunion.c +++ b/src/backend/optimizer/prep/prepunion.c @@ -689,6 +689,8 @@ generate_union_paths(SetOperationStmt *op, PlannerInfo *root, /* Undo effects of possibly forcing tuple_fraction to 0 */ root->tuple_fraction = save_fraction; + /* Add the UniqueKeys */ + populate_unionrel_uniquekeys(root, result_rel); return result_rel; } diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c index 86922a273c..6817c9c787 100644 --- a/src/backend/optimizer/util/appendinfo.c +++ b/src/backend/optimizer/util/appendinfo.c @@ -746,3 +746,47 @@ find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos) } return appinfos; } + +/* + * find_appinfo_by_child + * + */ +AppendRelInfo * +find_appinfo_by_child(PlannerInfo *root, Index child_index) +{ + ListCell *lc; + foreach(lc, root->append_rel_list) + { + AppendRelInfo *appinfo = lfirst_node(AppendRelInfo, lc); + if (appinfo->child_relid == child_index) + return appinfo; + } + elog(ERROR, "parent relation cant be found"); + return NULL; +} + +/* + * find_parent_var + * + */ +Var * +find_parent_var(AppendRelInfo *appinfo, Var *child_var) +{ + ListCell *lc; + Var *res = NULL; + Index attno = 1; + foreach(lc, appinfo->translated_vars) + { + Node *child_node = lfirst(lc); + if (equal(child_node, child_var)) + { + res = copyObject(child_var); + res->varattno = attno; + res->varno = appinfo->parent_relid; + } + attno++; + } + if (res == NULL) + elog(ERROR, "parent var can't be found."); + return res; +} diff --git a/src/backend/optimizer/util/inherit.c b/src/backend/optimizer/util/inherit.c index be1c9ddd96..3eec1f4d74 100644 --- a/src/backend/optimizer/util/inherit.c +++ b/src/backend/optimizer/util/inherit.c @@ -736,6 +736,7 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, { Node *onecq = (Node *) lfirst(lc2); bool pseudoconstant; + RestrictInfo *child_rinfo; /* check for pseudoconstant (no Vars or volatile functions) */ pseudoconstant = @@ -747,14 +748,15 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, root->hasPseudoConstantQuals = true; } /* reconstitute RestrictInfo with appropriate properties */ - childquals = lappend(childquals, - make_restrictinfo(root, - (Expr *) onecq, - rinfo->is_pushed_down, - rinfo->outerjoin_delayed, - pseudoconstant, - rinfo->security_level, - NULL, NULL, NULL)); + child_rinfo = make_restrictinfo(root, + (Expr *) onecq, + rinfo->is_pushed_down, + rinfo->outerjoin_delayed, + pseudoconstant, + rinfo->security_level, + NULL, NULL, NULL); + child_rinfo->mergeopfamilies = rinfo->mergeopfamilies; + childquals = lappend(childquals, child_rinfo); /* track minimum security level among child quals */ cq_min_security = Min(cq_min_security, rinfo->security_level); } diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 48a7ebfe45..4fe1824eb0 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -16,6 +16,7 @@ #include "nodes/execnodes.h" #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" extern A_Expr *makeA_Expr(A_Expr_Kind kind, List *name, @@ -106,4 +107,6 @@ extern GroupingSet *makeGroupingSet(GroupingSetKind kind, List *content, int loc extern VacuumRelation *makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols); +extern UniqueKey* makeUniqueKey(List *exprs, bool multi_nullvals); + #endif /* MAKEFUNC_H */ diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index e22df890ef..3320273ac1 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -265,6 +265,7 @@ typedef enum NodeTag T_EquivalenceMember, T_PathKey, T_PathTarget, + T_UniqueKey, T_RestrictInfo, T_IndexClause, T_PlaceHolderVar, diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index 0d61f04d27..80561b1003 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -739,6 +739,7 @@ typedef struct RelOptInfo QualCost baserestrictcost; /* cost of evaluating the above */ Index baserestrict_min_security; /* min security_level found in * baserestrictinfo */ + List *uniquekeys; /* List of UniqueKey */ List *joininfo; /* RestrictInfo structures for join clauses * involving this rel */ bool has_eclass_joins; /* T means joininfo is incomplete */ @@ -1059,6 +1060,28 @@ typedef struct PathKey } PathKey; +/* + * UniqueKey + * + * Represents the unique properties held by a RelOptInfo. + * + * exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL + * is a special case of UniqueKey, which means there is only 1 row in that + * relation. + * multi_nullvals: true means multi null values may exist in these exprs, so the + * uniqueness is not guaranteed in this case. This field is necessary for + * remove_useless_join & reduce_unique_semijoins where we don't mind these + * duplicated NULL values. It is set to true for 2 cases. One is a unique key + * from a unique index but the related column is nullable. The other one is for + * outer join. see populate_joinrel_uniquekeys for detail. + */ +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + /* * PathTarget * @@ -2496,7 +2519,7 @@ typedef enum * * flags indicating what kinds of grouping are possible. * partial_costs_set is true if the agg_partial_costs and agg_final_costs - * have been initialized. + * have been initialized. * agg_partial_costs gives partial aggregation costs. * agg_final_costs gives finalization costs. * target_parallel_safe is true if target is parallel safe. @@ -2526,8 +2549,8 @@ typedef struct * limit_tuples is an estimated bound on the number of output tuples, * or -1 if no LIMIT or couldn't estimate. * count_est and offset_est are the estimated values of the LIMIT and OFFSET - * expressions computed by preprocess_limit() (see comments for - * preprocess_limit() for more information). + * expressions computed by preprocess_limit() (see comments for + * preprocess_limit() for more information). */ typedef struct { diff --git a/src/include/nodes/pg_list.h b/src/include/nodes/pg_list.h index 404e03f132..85c65e910c 100644 --- a/src/include/nodes/pg_list.h +++ b/src/include/nodes/pg_list.h @@ -566,6 +566,16 @@ extern pg_nodiscard List *list_delete_first(List *list); extern pg_nodiscard List *list_delete_last(List *list); extern pg_nodiscard List *list_delete_nth_cell(List *list, int n); extern pg_nodiscard List *list_delete_cell(List *list, ListCell *cell); +extern bool list_is_subset(const List *members, const List *target); + +extern List *list_delete(List *list, void *datum); +extern List *list_delete_ptr(List *list, void *datum); +extern List *list_delete_int(List *list, int datum); +extern List *list_delete_oid(List *list, Oid datum); +extern List *list_delete_first(List *list); +extern List *list_delete_last(List *list); +extern List *list_delete_nth_cell(List *list, int n); +extern List *list_delete_cell(List *list, ListCell *cell); extern List *list_union(const List *list1, const List *list2); extern List *list_union_ptr(const List *list1, const List *list2); diff --git a/src/include/optimizer/appendinfo.h b/src/include/optimizer/appendinfo.h index 4cbf8c26cc..43b8d05f69 100644 --- a/src/include/optimizer/appendinfo.h +++ b/src/include/optimizer/appendinfo.h @@ -32,4 +32,7 @@ extern Relids adjust_child_relids_multilevel(PlannerInfo *root, Relids relids, extern AppendRelInfo **find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos); +extern AppendRelInfo *find_appinfo_by_child(PlannerInfo *root, Index child_index); +extern Var *find_parent_var(AppendRelInfo *appinfo, Var *child_var); + #endif /* APPENDINFO_H */ diff --git a/src/include/optimizer/optimizer.h b/src/include/optimizer/optimizer.h index d587952b7d..843aafa51e 100644 --- a/src/include/optimizer/optimizer.h +++ b/src/include/optimizer/optimizer.h @@ -23,6 +23,7 @@ #define OPTIMIZER_H #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" /* Test if an expression node represents a SRF call. Beware multiple eval! */ #define IS_SRF_CALL(node) \ @@ -168,6 +169,7 @@ extern TargetEntry *get_sortgroupref_tle(Index sortref, List *targetList); extern TargetEntry *get_sortgroupclause_tle(SortGroupClause *sgClause, List *targetList); +extern Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); extern Node *get_sortgroupclause_expr(SortGroupClause *sgClause, List *targetList); extern List *get_sortgrouplist_exprs(List *sgClauses, diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h index 035d3e1206..1adf99c9ee 100644 --- a/src/include/optimizer/paths.h +++ b/src/include/optimizer/paths.h @@ -247,5 +247,48 @@ extern PathKey *make_canonical_pathkey(PlannerInfo *root, int strategy, bool nulls_first); extern void add_paths_to_append_rel(PlannerInfo *root, RelOptInfo *rel, List *live_childrels); +extern List *select_mergejoin_clauses(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + JoinType jointype, + bool *mergejoin_allowed); + +/* + * uniquekeys.c + * Utilities for matching and building unique keys + */ +extern void populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List* unique_index_list); +extern void populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels); +extern void populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel); +extern void populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel); +extern void populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel); +extern void simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel); +extern void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel); +extern void populate_joinrel_uniquekeys(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + RelOptInfo *rel2, + List *restrictlist, + JoinType jointype); + +extern bool relation_has_uniquekeys_for(PlannerInfo *root, + RelOptInfo *rel, + List *exprs, + bool allow_multinulls); +extern bool relation_is_onerow(RelOptInfo *rel); #endif /* PATHS_H */ -- 2.26.2 --yh7cw345mbttadjz Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v38-0003-Extend-UniqueKeys.patch" ^ permalink raw reply [nested|flat] 91+ messages in thread
* [PATCH v36 2/6] Introduce UniqueKey attributes on RelOptInfo struct. @ 2020-05-11 07:50 一挃 <[email protected]> 0 siblings, 0 replies; 91+ messages in thread From: 一挃 @ 2020-05-11 07:50 UTC (permalink / raw) UniqueKey is a set of exprs on RelOptInfo which represents the exprs will be unique on the given RelOptInfo. You can see README.uniquekey for more information. --- src/backend/nodes/copyfuncs.c | 13 + src/backend/nodes/list.c | 31 + src/backend/nodes/makefuncs.c | 13 + src/backend/nodes/outfuncs.c | 11 + src/backend/nodes/readfuncs.c | 10 + src/backend/optimizer/path/Makefile | 3 +- src/backend/optimizer/path/README.uniquekey | 131 +++ src/backend/optimizer/path/allpaths.c | 10 + src/backend/optimizer/path/joinpath.c | 9 +- src/backend/optimizer/path/joinrels.c | 2 + src/backend/optimizer/path/pathkeys.c | 3 +- src/backend/optimizer/path/uniquekeys.c | 1131 +++++++++++++++++++ src/backend/optimizer/plan/planner.c | 13 +- src/backend/optimizer/prep/prepunion.c | 2 + src/backend/optimizer/util/appendinfo.c | 44 + src/backend/optimizer/util/inherit.c | 16 +- src/include/nodes/makefuncs.h | 3 + src/include/nodes/nodes.h | 1 + src/include/nodes/pathnodes.h | 29 +- src/include/nodes/pg_list.h | 2 + src/include/optimizer/appendinfo.h | 3 + src/include/optimizer/optimizer.h | 2 + src/include/optimizer/paths.h | 43 + 23 files changed, 1502 insertions(+), 23 deletions(-) create mode 100644 src/backend/optimizer/path/README.uniquekey create mode 100644 src/backend/optimizer/path/uniquekeys.c diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 89c409de66..1f50400fd2 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2273,6 +2273,16 @@ _copyPathKey(const PathKey *from) return newnode; } +static UniqueKey * +_copyUniqueKey(const UniqueKey *from) +{ + UniqueKey *newnode = makeNode(UniqueKey); + + COPY_NODE_FIELD(exprs); + COPY_SCALAR_FIELD(multi_nullvals); + + return newnode; +} /* * _copyRestrictInfo */ @@ -5152,6 +5162,9 @@ copyObjectImpl(const void *from) case T_PathKey: retval = _copyPathKey(from); break; + case T_UniqueKey: + retval = _copyUniqueKey(from); + break; case T_RestrictInfo: retval = _copyRestrictInfo(from); break; diff --git a/src/backend/nodes/list.c b/src/backend/nodes/list.c index 80fa8c84e4..a7a99b70f2 100644 --- a/src/backend/nodes/list.c +++ b/src/backend/nodes/list.c @@ -687,6 +687,37 @@ list_member_oid(const List *list, Oid datum) return false; } +/* + * return true iff every entry in "members" list is also present + * in the "target" list. + */ +bool +list_is_subset(const List *members, const List *target) +{ + const ListCell *lc1, *lc2; + + Assert(IsPointerList(members)); + Assert(IsPointerList(target)); + check_list_invariants(members); + check_list_invariants(target); + + foreach(lc1, members) + { + bool found = false; + foreach(lc2, target) + { + if (equal(lfirst(lc1), lfirst(lc2))) + { + found = true; + break; + } + } + if (!found) + return false; + } + return true; +} + /* * Delete the n'th cell (counting from 0) in list. * diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 49de285f01..646cf7c9a1 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -814,3 +814,16 @@ makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols) v->va_cols = va_cols; return v; } + + +/* + * makeUniqueKey + */ +UniqueKey* +makeUniqueKey(List *exprs, bool multi_nullvals) +{ + UniqueKey * ukey = makeNode(UniqueKey); + ukey->exprs = exprs; + ukey->multi_nullvals = multi_nullvals; + return ukey; +} diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index e2f177515d..c3a9632992 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2428,6 +2428,14 @@ _outPathKey(StringInfo str, const PathKey *node) WRITE_BOOL_FIELD(pk_nulls_first); } +static void +_outUniqueKey(StringInfo str, const UniqueKey *node) +{ + WRITE_NODE_TYPE("UNIQUEKEY"); + WRITE_NODE_FIELD(exprs); + WRITE_BOOL_FIELD(multi_nullvals); +} + static void _outPathTarget(StringInfo str, const PathTarget *node) { @@ -4127,6 +4135,9 @@ outNode(StringInfo str, const void *obj) case T_PathKey: _outPathKey(str, obj); break; + case T_UniqueKey: + _outUniqueKey(str, obj); + break; case T_PathTarget: _outPathTarget(str, obj); break; diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index 42050ab719..3a18571d0c 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -452,6 +452,14 @@ _readSetOperationStmt(void) READ_DONE(); } +static UniqueKey * +_readUniqueKey(void) +{ + READ_LOCALS(UniqueKey); + READ_NODE_FIELD(exprs); + READ_BOOL_FIELD(multi_nullvals); + READ_DONE(); +} /* * Stuff from primnodes.h. @@ -2656,6 +2664,8 @@ parseNodeString(void) return_value = _readCommonTableExpr(); else if (MATCH("SETOPERATIONSTMT", 16)) return_value = _readSetOperationStmt(); + else if (MATCH("UNIQUEKEY", 9)) + return_value = _readUniqueKey(); else if (MATCH("ALIAS", 5)) return_value = _readAlias(); else if (MATCH("RANGEVAR", 8)) diff --git a/src/backend/optimizer/path/Makefile b/src/backend/optimizer/path/Makefile index 1e199ff66f..7b9820c25f 100644 --- a/src/backend/optimizer/path/Makefile +++ b/src/backend/optimizer/path/Makefile @@ -21,6 +21,7 @@ OBJS = \ joinpath.o \ joinrels.o \ pathkeys.o \ - tidpath.o + tidpath.o \ + uniquekeys.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/optimizer/path/README.uniquekey b/src/backend/optimizer/path/README.uniquekey new file mode 100644 index 0000000000..5eac761995 --- /dev/null +++ b/src/backend/optimizer/path/README.uniquekey @@ -0,0 +1,131 @@ +1. What is UniqueKey? +We can think UniqueKey is a set of exprs for a RelOptInfo, which we are insure +that doesn't yields same result among all the rows. The simplest UniqueKey +format is primary key. + +However we define the UnqiueKey as below. + +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + +exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL +is a special case of UniqueKey, which means there is only one row in that +relation.it has a stronger semantic than others. like SELECT uk FROM t; uk is +normal unique key and may have different values. SELECT colx FROM t WHERE uk = +const. colx is unique AND we have only 1 value. This field can used for +innerrel_is_unique. this logic is handled specially in add_uniquekey_for_onerow +function. + +multi_nullvals: true means multi null values may exist in these exprs, so the +uniqueness is not guaranteed in this case. This field is necessary for +remove_useless_join & reduce_unique_semijoins where we don't mind these +duplicated NULL values. It is set to true for 2 cases. One is a unique key +from a unique index but the related column is nullable. The other one is for +outer join. see populate_joinrel_uniquekeys for detail. + + +The UniqueKey can be used at the following cases at least: +1. remove_useless_joins. +2. reduce_semianti_joins +3. remove distinct node if distinct clause is unique. +4. remove aggnode if group by clause is unique. +5. Index Skip Scan (WIP) +6. Aggregation Push Down without 2 phase aggregation if the join can't + duplicated the aggregated rows. (work in progress feature) + +2. How is it maintained? + +We have a set of populate_xxx_unqiuekeys functions to maintain the uniquekey on +various cases. xxx includes baserel, joinrel, partitionedrel, distinctrel, +groupedrel, unionrel. and we also need to convert the uniquekey from subquery +to outer relation, which is what convert_subquery_uniquekeys does. + +1. The first part is about baserel. We handled 3 cases. suppose we have Unique +Index on (a, b). + +1. SELECT a, b FROM t. UniqueKey (a, b) +2. SELECT a FROM t WHERE b = 1; UniqueKey (a) +3. SELECT .. FROM t WHERE a = 1 AND b = 1; UniqueKey (NIL). onerow case, every + column is Unique. + +2. The next part is joinrel, this part is most error-prone, we simplified the rules +like below: +1. If the relation's UniqueKey can't be duplicated after join, then is will be + still valid for the join rel. The function we used here is + innerrel_keeps_unique. The basic idea is innerrel.any_col = outer.uk. + +2. If the UnqiueKey can't keep valid via the rule 1, the combination of the + UniqueKey from both sides are valid for sure. We can prove this as: if the + unique exprs from rel1 is duplicated by rel2, the duplicated rows must + contains different unique exprs from rel2. + +More considerations about onerow: +1. If relation with one row and it can't be duplicated, it is still possible + contains mulit_nullvas after outer join. +2. If the either UniqueKey can be duplicated after join, the can get one row + only when both side is one row AND there is no outer join. +3. Whenever the onerow UniqueKey is not a valid any more, we need to convert one + row UniqueKey to normal unique key since we don't store exprs for one-row + relation. get_exprs_from_uniquekeys will be used here. + + +More considerations about multi_nullvals after join: +1. If the original UnqiueKey has multi_nullvals, the final UniqueKey will have + mulit_nullvals in any case. +2. If a unique key doesn't allow mulit_nullvals, after some outer join, it + allows some outer join. + + +3. When we comes to subquery, we need to convert_subquery_unqiuekeys just like +convert_subquery_pathkeys. Only the UniqueKey insides subquery is referenced as +a Var in outer relation will be reused. The relationship between the outerrel.Var +and subquery.exprs is built with outerel->subroot->processed_tlist. + + +4. As for the SRF functions, it will break the uniqueness of uniquekey, However it +is handled in adjust_paths_for_srfs, which happens after the query_planner. so +we will maintain the UniqueKey until there and reset it to NIL at that +places. This can't help on distinct/group by elimination cases but probably help +in some other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is +semantic correctly. + + +5. As for inherit table, we first main the UnqiueKey on childrel as well. But for +partitioned table we need to maintain 2 different kinds of +UnqiueKey. 1). UniqueKey on the parent relation 2). UniqueKey on child +relation for partition wise query. + +Example: +CREATE TABLE p (a int not null, b int not null) partition by list (a); +CREATE TABLE p0 partition of p for values in (1); +CREATE TABLE p1 partition of p for values in (2); + +create unique index p0_b on p0(b); +create unique index p1_b on p1(b); + +Now b is only unique on partition level, so the distinct can't be removed on +the following cases. SELECT DISTINCT b FROM p; + +Another example is SELECT DISTINCT a, b FROM p WHERE a = 1; Since only one +partition is chosen, the UniqueKey on child relation is same as the UniqueKey on +parent relation. + +Another usage of UniqueKey on partition level is it be helpful for +partition-wise join. + +As for the UniqueKey on parent table level, it comes with 2 different ways, +1). the UniqueKey is also derived in UniqueKey index, but the index must be same +in all the related children relations and the unique index must contains +Partition Key in it. Example: + +CREATE UNIQUE INDEX p_ab ON p(a, b); -- where a is the partition key. + +-- Query +SELECT a, b FROM p; the (a, b) is a UniqueKey of p. + +2). If the parent relation has only one childrel, the UniqueKey on childrel is + the UniqueKey on parent as well. diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index 484dab0a1a..2ad9d06d7a 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -579,6 +579,12 @@ set_plain_rel_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) */ check_index_predicates(root, rel); + /* + * Now that we've marked which partial indexes are suitable, we can now + * build the relation's unique keys. + */ + populate_baserel_uniquekeys(root, rel, rel->indexlist); + /* Mark rel with estimated output rows, width, etc */ set_baserel_size_estimates(root, rel); } @@ -1310,6 +1316,8 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, /* Add paths to the append relation. */ add_paths_to_append_rel(root, rel, live_childrels); + if (IS_PARTITIONED_REL(rel)) + populate_partitionedrel_uniquekeys(root, rel, live_childrels); } @@ -2383,6 +2391,8 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, pathkeys, required_outer)); } + convert_subquery_uniquekeys(root, rel, sub_final_rel); + /* If outer rel allows parallelism, do same for partial paths. */ if (rel->consider_parallel && bms_is_empty(required_outer)) { diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c index db54a6ba2e..ef0fd2fb0b 100644 --- a/src/backend/optimizer/path/joinpath.c +++ b/src/backend/optimizer/path/joinpath.c @@ -71,13 +71,6 @@ static void consider_parallel_mergejoin(PlannerInfo *root, static void hash_inner_and_outer(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, RelOptInfo *innerrel, JoinType jointype, JoinPathExtraData *extra); -static List *select_mergejoin_clauses(PlannerInfo *root, - RelOptInfo *joinrel, - RelOptInfo *outerrel, - RelOptInfo *innerrel, - List *restrictlist, - JoinType jointype, - bool *mergejoin_allowed); static void generate_mergejoin_paths(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *innerrel, @@ -1927,7 +1920,7 @@ hash_inner_and_outer(PlannerInfo *root, * if it is mergejoinable and involves vars from the two sub-relations * currently of interest. */ -static List * +List * select_mergejoin_clauses(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c index 2d343cd293..b9163ee8ff 100644 --- a/src/backend/optimizer/path/joinrels.c +++ b/src/backend/optimizer/path/joinrels.c @@ -924,6 +924,8 @@ populate_joinrel_with_paths(PlannerInfo *root, RelOptInfo *rel1, /* Apply partitionwise join technique, if possible. */ try_partitionwise_join(root, rel1, rel2, joinrel, sjinfo, restrictlist); + + populate_joinrel_uniquekeys(root, joinrel, rel1, rel2, restrictlist, sjinfo->jointype); } diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index ce9bf87e9b..7e596d4194 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -33,7 +33,6 @@ static bool pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys); static bool matches_boolean_partition_clause(RestrictInfo *rinfo, RelOptInfo *partrel, int partkeycol); -static Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); static bool right_merge_direction(PlannerInfo *root, PathKey *pathkey); @@ -1035,7 +1034,7 @@ convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel, * We need this to ensure that we don't return pathkeys describing values * that are unavailable above the level of the subquery scan. */ -static Var * +Var * find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle) { ListCell *lc; diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c new file mode 100644 index 0000000000..b33bcd2f32 --- /dev/null +++ b/src/backend/optimizer/path/uniquekeys.c @@ -0,0 +1,1131 @@ +/*------------------------------------------------------------------------- + * + * uniquekeys.c + * Utilities for matching and building unique keys + * + * Portions Copyright (c) 2020, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/optimizer/path/uniquekeys.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "nodes/makefuncs.h" +#include "nodes/nodeFuncs.h" +#include "optimizer/pathnode.h" +#include "optimizer/paths.h" +#include "optimizer/appendinfo.h" +#include "optimizer/optimizer.h" +#include "optimizer/tlist.h" +#include "rewrite/rewriteManip.h" + + +/* + * This struct is used to help populate_joinrel_uniquekeys. + * + * added_to_joinrel is true if a uniquekey (from outerrel or innerrel) + * has been added to joinrel. + * useful is true if the exprs of the uniquekey still appears in joinrel. + */ +typedef struct UniqueKeyContextData +{ + UniqueKey *uniquekey; + bool added_to_joinrel; + bool useful; +} *UniqueKeyContext; + +static List *initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel); +static bool innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + bool reverse); + +static List *get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals); +static List *get_exprs_from_uniquekey(RelOptInfo *joinrel, + RelOptInfo *rel1, + UniqueKey *ukey); +static void add_uniquekey_for_onerow(RelOptInfo *rel); +static bool add_combined_uniquekey(RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype); + +/* Used for unique indexes checking for partitioned table */ +static bool index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind); +static IndexOptInfo *simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from); +static bool simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2); +static void adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_index); + +/* Helper function for grouped relation and distinct relation. */ +static void add_uniquekey_from_sortgroups(PlannerInfo *root, + RelOptInfo *rel, + List *sortgroups); + +/* + * populate_baserel_uniquekeys + * Populate 'baserel' uniquekeys list by looking at the rel's unique index + * and baserestrictinfo + */ +void +populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List *indexlist) +{ + ListCell *lc; + List *matched_uniq_indexes = NIL; + + /* Attrs appears in rel->reltarget->exprs. */ + Bitmapset *used_attrs = NULL; + + List *const_exprs = NIL; + List *expr_opfamilies = NIL; + + Assert(baserel->rtekind == RTE_RELATION); + + foreach(lc, indexlist) + { + IndexOptInfo *ind = (IndexOptInfo *) lfirst(lc); + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + matched_uniq_indexes = lappend(matched_uniq_indexes, ind); + } + + if (matched_uniq_indexes == NIL) + return; + + /* Check which attrs is used in baserel->reltarget */ + pull_varattnos((Node *)baserel->reltarget->exprs, baserel->relid, &used_attrs); + + /* Check which attrno is used at a mergeable const filter */ + foreach(lc, baserel->baserestrictinfo) + { + RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); + + if (rinfo->mergeopfamilies == NIL) + continue; + + if (bms_is_empty(rinfo->left_relids)) + { + const_exprs = lappend(const_exprs, get_rightop(rinfo->clause)); + } + else if (bms_is_empty(rinfo->right_relids)) + { + const_exprs = lappend(const_exprs, get_leftop(rinfo->clause)); + } + else + continue; + + expr_opfamilies = lappend(expr_opfamilies, rinfo->mergeopfamilies); + } + + foreach(lc, matched_uniq_indexes) + { + bool multi_nullvals, useful; + List *exprs = get_exprs_from_uniqueindex(lfirst_node(IndexOptInfo, lc), + const_exprs, + expr_opfamilies, + used_attrs, + &useful, + &multi_nullvals); + if (useful) + { + if (exprs == NIL) + { + /* All the columns in Unique Index matched with a restrictinfo */ + add_uniquekey_for_onerow(baserel); + return; + } + baserel->uniquekeys = lappend(baserel->uniquekeys, + makeUniqueKey(exprs, multi_nullvals)); + } + } +} + + +/* + * populate_partitionedrel_uniquekeys + * The UniqueKey on partitionrel comes from 2 cases: + * 1). Only one partition is involved in this query, the unique key can be + * copied to parent rel from childrel. + * 2). There are some unique index which includes partition key and exists + * in all the related partitions. + * We never mind rule 2 if we hit rule 1. + */ + +void +populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels) +{ + ListCell *lc; + List *global_uniq_indexlist = NIL; + RelOptInfo *childrel; + bool is_first = true; + + Assert(IS_PARTITIONED_REL(rel)); + + if (childrels == NIL) + return; + + /* + * If there is only one partition used in this query, the UniqueKey in childrel is + * still valid in parent level, but we need convert the format from child expr to + * parent expr. + */ + if (list_length(childrels) == 1) + { + /* Check for Rule 1 */ + RelOptInfo *childrel = linitial_node(RelOptInfo, childrels); + ListCell *lc; + Assert(childrel->reloptkind == RELOPT_OTHER_MEMBER_REL); + if (relation_is_onerow(childrel)) + { + add_uniquekey_for_onerow(rel); + return; + } + + foreach(lc, childrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + AppendRelInfo *appinfo = find_appinfo_by_child(root, childrel->relid); + List *parent_exprs = NIL; + bool can_reuse = true; + ListCell *lc2; + foreach(lc2, ukey->exprs) + { + Var *var = (Var *)lfirst(lc2); + /* + * If the expr comes from a expression, it is hard to build the expression + * in parent so ignore that case for now. + */ + if(!IsA(var, Var)) + { + can_reuse = false; + break; + } + /* Convert it to parent var */ + parent_exprs = lappend(parent_exprs, find_parent_var(appinfo, var)); + } + if (can_reuse) + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(parent_exprs, + ukey->multi_nullvals)); + } + } + else + { + /* Check for rule 2 */ + childrel = linitial_node(RelOptInfo, childrels); + foreach(lc, childrel->indexlist) + { + IndexOptInfo *ind = lfirst(lc); + IndexOptInfo *modified_index; + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + + /* + * During simple_copy_indexinfo_to_parent, we need to convert var from + * child var to parent var, index on expression is too complex to handle. + * so ignore it for now. + */ + if (ind->indexprs != NIL) + continue; + + modified_index = simple_copy_indexinfo_to_parent(root, rel, ind); + /* + * If the unique index doesn't contain partkey, then it is unique + * on this partition only, so it is useless for us. + */ + if (!index_constains_partkey(rel, modified_index)) + continue; + + global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index); + } + + if (global_uniq_indexlist != NIL) + { + foreach(lc, childrels) + { + RelOptInfo *child = lfirst(lc); + if (is_first) + { + is_first = false; + continue; + } + adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist); + } + /* Now we have a list of unique index which are exactly same on all childrels, + * Set the UniqueKey just like it is non-partition table + */ + populate_baserel_uniquekeys(root, rel, global_uniq_indexlist); + } + } +} + + +/* + * populate_distinctrel_uniquekeys + */ +void +populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel) +{ + /* The unique key before the distinct is still valid. */ + distinctrel->uniquekeys = list_copy(inputrel->uniquekeys); + add_uniquekey_from_sortgroups(root, distinctrel, root->parse->distinctClause); +} + +/* + * populate_grouprel_uniquekeys + */ +void +populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel) + +{ + Query *parse = root->parse; + bool input_ukey_added = false; + ListCell *lc; + + if (relation_is_onerow(inputrel)) + { + add_uniquekey_for_onerow(grouprel); + return; + } + if (parse->groupingSets) + return; + + /* A Normal group by without grouping set. */ + if (parse->groupClause) + { + /* + * Current even the groupby clause is Unique already, but if query has aggref + * We have to create grouprel still. To keep the UnqiueKey short, we will check + * the UniqueKey of input_rel still valid, if so we reuse it. + */ + foreach(lc, inputrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (list_is_subset(ukey->exprs, grouprel->reltarget->exprs)) + { + grouprel->uniquekeys = lappend(grouprel->uniquekeys, + ukey); + input_ukey_added = true; + } + } + if (!input_ukey_added) + /* + * group by clause must be a super-set of grouprel->reltarget->exprs except the + * aggregation expr, so if such exprs is unique already, no bother to generate + * new uniquekey for group by exprs. + */ + add_uniquekey_from_sortgroups(root, + grouprel, + root->parse->groupClause); + } + else + /* It has aggregation but without a group by, so only one row returned */ + add_uniquekey_for_onerow(grouprel); +} + +/* + * simple_copy_uniquekeys + * Using a function for the one-line code makes us easy to check where we simply + * copied the uniquekey. + */ +void +simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel) +{ + newrel->uniquekeys = oldrel->uniquekeys; +} + +/* + * populate_unionrel_uniquekeys + */ +void +populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel) +{ + ListCell *lc; + List *exprs = NIL; + + Assert(unionrel->uniquekeys == NIL); + + foreach(lc, unionrel->reltarget->exprs) + { + exprs = lappend(exprs, lfirst(lc)); + } + + if (exprs == NIL) + /* SQL: select union select; is valid, we need to handle it here. */ + add_uniquekey_for_onerow(unionrel); + else + unionrel->uniquekeys = lappend(unionrel->uniquekeys, + makeUniqueKey(exprs,false)); + +} + +/* + * populate_joinrel_uniquekeys + * + * populate uniquekeys for joinrel. We will check each relation to see if its + * UniqueKey is still valid via innerrel_keeps_unique, if so, we add it to + * joinrel. The multi_nullvals field will be changed to true for some outer + * join cases and one-row UniqueKey needs to be converted to normal UniqueKey + * for the same case as well. + * For the uniquekey in either baserel which can't be unique after join, we still + * check to see if combination of UniqueKeys from both side is still useful for us. + * if yes, we add it to joinrel as well. + */ +void +populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *outerrel, RelOptInfo *innerrel, + List *restrictlist, JoinType jointype) +{ + ListCell *lc, *lc2; + List *clause_list = NIL; + List *outerrel_ukey_ctx; + List *innerrel_ukey_ctx; + bool inner_onerow, outer_onerow; + bool mergejoin_allowed; + + /* Care about the outerrel relation only for SEMI/ANTI join */ + if (jointype == JOIN_SEMI || jointype == JOIN_ANTI) + { + foreach(lc, outerrel->uniquekeys) + { + UniqueKey *uniquekey = lfirst_node(UniqueKey, lc); + if (list_is_subset(uniquekey->exprs, joinrel->reltarget->exprs)) + joinrel->uniquekeys = lappend(joinrel->uniquekeys, uniquekey); + } + return; + } + + Assert(jointype == JOIN_LEFT || jointype == JOIN_FULL || jointype == JOIN_INNER); + + /* Fast path */ + if (innerrel->uniquekeys == NIL || outerrel->uniquekeys == NIL) + return; + + inner_onerow = relation_is_onerow(innerrel); + outer_onerow = relation_is_onerow(outerrel); + + outerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(outerrel); + innerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(innerrel); + + clause_list = select_mergejoin_clauses(root, joinrel, outerrel, innerrel, + restrictlist, jointype, + &mergejoin_allowed); + + if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true /* reverse */)) + { + bool outer_impact = jointype == JOIN_FULL; + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + /* Outer relation has one row, and the unique key is not duplicated after join, + * the joinrel will still has one row unless the jointype == JOIN_FULL. + */ + if (outer_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (outer_onerow) + { + /* + * The onerow outerrel becomes multi rows and multi_nullvals + * will be changed to true. We also need to set the exprs correctly since it + * can't be NIL any more. + */ + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(joinrel, outerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Change multi_nullvals to true due to the full join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, true)); + else + /* Just reuse it */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + } + ctx->added_to_joinrel = true; + } + } + + if (innerrel_keeps_unique(root, outerrel, innerrel, clause_list, false)) + { + bool outer_impact = jointype == JOIN_FULL || jointype == JOIN_LEFT;; + + foreach(lc, innerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + if (inner_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (inner_onerow) + { + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(joinrel, innerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Need to change multi_nullvals to true due to the outer join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, + true)); + else + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + + } + ctx->added_to_joinrel = true; + } + } + + /* + * The combination of the UniqueKey from both sides is unique as well regardless + * of join type, but no bother to add it if its subset has been added to joinrel + * already or it is not useful for the joinrel. + */ + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx1 = (UniqueKeyContext) lfirst(lc); + if (ctx1->added_to_joinrel || !ctx1->useful) + continue; + foreach(lc2, innerrel_ukey_ctx) + { + UniqueKeyContext ctx2 = (UniqueKeyContext) lfirst(lc2); + if (ctx2->added_to_joinrel || !ctx2->useful) + continue; + if (add_combined_uniquekey(joinrel, outerrel, innerrel, + ctx1->uniquekey, ctx2->uniquekey, + jointype)) + /* If we set a onerow UniqueKey to joinrel, we don't need other. */ + return; + } + } +} + + +/* + * convert_subquery_uniquekeys + * + * Covert the UniqueKey in subquery to outer relation. + */ +void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel) +{ + ListCell *lc; + + if (sub_final_rel->uniquekeys == NIL) + return; + + if (relation_is_onerow(sub_final_rel)) + { + add_uniquekey_for_onerow(currel); + return; + } + + Assert(currel->subroot != NULL); + + foreach(lc, sub_final_rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + ListCell *lc; + List *exprs = NIL; + bool ukey_useful = true; + + /* One row case is handled above */ + Assert(ukey->exprs != NIL); + foreach(lc, ukey->exprs) + { + Var *var; + TargetEntry *tle = tlist_member(lfirst(lc), + currel->subroot->processed_tlist); + if (tle == NULL) + { + ukey_useful = false; + break; + } + var = find_var_for_subquery_tle(currel, tle); + if (var == NULL) + { + ukey_useful = false; + break; + } + exprs = lappend(exprs, var); + } + + if (ukey_useful) + currel->uniquekeys = lappend(currel->uniquekeys, + makeUniqueKey(exprs, + ukey->multi_nullvals)); + + } +} + +/* + * innerrel_keeps_unique + * + * Check if Unique key of the innerrel is valid after join. innerrel's UniqueKey + * will be still valid if innerrel's any-column mergeop outrerel's uniquekey + * exists in clause_list. + * + * Note: the clause_list must be a list of mergeable restrictinfo already. + */ +static bool +innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *clause_list, + bool reverse) +{ + ListCell *lc, *lc2, *lc3; + + if (outerrel->uniquekeys == NIL || innerrel->uniquekeys == NIL) + return false; + + /* Check if there is outerrel's uniquekey in mergeable clause. */ + foreach(lc, outerrel->uniquekeys) + { + List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs; + bool clauselist_matchs_all_exprs = true; + foreach(lc2, outer_uq_exprs) + { + Node *outer_uq_expr = lfirst(lc2); + bool find_uq_expr_in_clauselist = false; + foreach(lc3, clause_list) + { + RestrictInfo *rinfo = lfirst_node(RestrictInfo, lc3); + Node *outer_expr; + if (reverse) + outer_expr = rinfo->outer_is_left ? get_rightop(rinfo->clause) : get_leftop(rinfo->clause); + else + outer_expr = rinfo->outer_is_left ? get_leftop(rinfo->clause) : get_rightop(rinfo->clause); + if (equal(outer_expr, outer_uq_expr)) + { + find_uq_expr_in_clauselist = true; + break; + } + } + if (!find_uq_expr_in_clauselist) + { + /* No need to check the next exprs in the current uniquekey */ + clauselist_matchs_all_exprs = false; + break; + } + } + + if (clauselist_matchs_all_exprs) + return true; + } + return false; +} + + +/* + * relation_is_onerow + * Check if it is a one-row relation by checking UniqueKey. + */ +bool +relation_is_onerow(RelOptInfo *rel) +{ + UniqueKey *ukey; + if (rel->uniquekeys == NIL) + return false; + ukey = linitial_node(UniqueKey, rel->uniquekeys); + return ukey->exprs == NIL && list_length(rel->uniquekeys) == 1; +} + +/* + * relation_has_uniquekeys_for + * Returns true if we have proofs that 'rel' cannot return multiple rows with + * the same values in each of 'exprs'. Otherwise returns false. + */ +bool +relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel, + List *exprs, bool allow_multinulls) +{ + ListCell *lc; + + /* + * For UniqueKey->onerow case, the uniquekey->exprs is empty as well + * so we can't rely on list_is_subset to handle this special cases + */ + if (exprs == NIL) + return false; + + foreach(lc, rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (ukey->multi_nullvals && !allow_multinulls) + continue; + if (list_is_subset(ukey->exprs, exprs)) + return true; + } + return false; +} + + +/* + * get_exprs_from_uniqueindex + * + * Return a list of exprs which is unique. set useful to false if this + * unique index is not useful for us. + */ +static List * +get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals) +{ + List *exprs = NIL; + ListCell *indexpr_item; + int c = 0; + + *useful = true; + *multi_nullvals = false; + + indexpr_item = list_head(unique_index->indexprs); + for(c = 0; c < unique_index->ncolumns; c++) + { + int attr = unique_index->indexkeys[c]; + Expr *expr; + bool matched_const = false; + ListCell *lc1, *lc2; + + if(attr > 0) + { + expr = list_nth_node(TargetEntry, unique_index->indextlist, c)->expr; + } + else if (attr == 0) + { + /* Expression index */ + expr = lfirst(indexpr_item); + indexpr_item = lnext(unique_index->indexprs, indexpr_item); + } + else /* attr < 0 */ + { + /* Index on system column is not supported */ + Assert(false); + } + + /* + * Check index_col = Const case with regarding to opfamily checking + * If we can remove the index_col from the final UniqueKey->exprs. + */ + forboth(lc1, const_exprs, lc2, const_expr_opfamilies) + { + if (list_member_oid((List *)lfirst(lc2), unique_index->opfamily[c]) + && match_index_to_operand((Node *) lfirst(lc1), c, unique_index)) + { + matched_const = true; + break; + } + } + + if (matched_const) + continue; + + /* Check if the indexed expr is used in rel */ + if (attr > 0) + { + /* + * Normal Indexed column, if the col is not used, then the index is useless + * for uniquekey. + */ + attr -= FirstLowInvalidHeapAttributeNumber; + + if (!bms_is_member(attr, used_varattrs)) + { + *useful = false; + break; + } + } + else if (!list_member(unique_index->rel->reltarget->exprs, expr)) + { + /* Expression index but the expression is not used in rel */ + *useful = false; + break; + } + + /* check not null property. */ + if (attr == 0) + { + /* We never know if a expression yields null or not */ + *multi_nullvals = true; + } + else if (!bms_is_member(attr, unique_index->rel->notnullattrs) + && !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber, + unique_index->rel->notnullattrs)) + { + *multi_nullvals = true; + } + + exprs = lappend(exprs, expr); + } + return exprs; +} + + +/* + * add_uniquekey_for_onerow + * If we are sure that the relation only returns one row, then all the columns + * are unique. However we don't need to create UniqueKey for every column, we + * just set exprs = NIL and overwrites all the other UniqueKey on this RelOptInfo + * since this one has strongest semantics. + */ +void +add_uniquekey_for_onerow(RelOptInfo *rel) +{ + /* + * We overwrite the previous UniqueKey on purpose since this one has the + * strongest semantic. + */ + rel->uniquekeys = list_make1(makeUniqueKey(NIL, false)); +} + + +/* + * initililze_uniquecontext_for_joinrel + * Return a List of UniqueKeyContext for an inputrel + */ +static List * +initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel) +{ + List *res = NIL; + ListCell *lc; + foreach(lc, inputrel->uniquekeys) + { + UniqueKeyContext context; + context = palloc(sizeof(struct UniqueKeyContextData)); + context->uniquekey = lfirst_node(UniqueKey, lc); + context->added_to_joinrel = false; + context->useful = true; + res = lappend(res, context); + } + return res; +} + + +/* + * get_exprs_from_uniquekey + * Unify the way of get List of exprs from a one-row UniqueKey or + * normal UniqueKey. for the onerow case, every expr in rel1 is a valid + * UniqueKey. Return a List of exprs. + * + * rel1: The relation which you want to get the exprs. + * ukey: The UniqueKey you want to get the exprs. + */ +static List * +get_exprs_from_uniquekey(RelOptInfo *joinrel, RelOptInfo *rel1, UniqueKey *ukey) +{ + ListCell *lc; + bool onerow = rel1 != NULL && relation_is_onerow(rel1); + + List *res = NIL; + Assert(onerow || ukey); + if (onerow) + { + /* Only cares about the exprs still exist in joinrel */ + foreach(lc, joinrel->reltarget->exprs) + { + Bitmapset *relids = pull_varnos(lfirst(lc)); + if (bms_is_subset(relids, rel1->relids)) + { + res = lappend(res, list_make1(lfirst(lc))); + } + } + } + else + { + res = list_make1(ukey->exprs); + } + return res; +} + +/* + * Partitioned table Unique Keys. + * The partition table unique key is maintained as: + * 1. The index must be unique as usual. + * 2. The index must contains partition key. + * 3. The index must exist on all the child rel. see simple_indexinfo_equal for + * how we compare it. + */ + +/* + * index_constains_partkey + * return true if the index contains the partiton key. + */ +static bool +index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind) +{ + ListCell *lc; + int i; + Assert(IS_PARTITIONED_REL(partrel)); + Assert(partrel->part_scheme->partnatts > 0); + + for(i = 0; i < partrel->part_scheme->partnatts; i++) + { + Node *part_expr = linitial(partrel->partexprs[i]); + bool found_in_index = false; + foreach(lc, ind->indextlist) + { + Expr *index_expr = lfirst_node(TargetEntry, lc)->expr; + if (equal(index_expr, part_expr)) + { + found_in_index = true; + break; + } + } + if (!found_in_index) + return false; + } + return true; +} + +/* + * simple_indexinfo_equal + * + * Used to check if the 2 index is same as each other. The index here + * is COPIED from childrel and did some tiny changes(see + * simple_copy_indexinfo_to_parent) + */ +static bool +simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2) +{ + Size oid_cmp_len = sizeof(Oid) * ind1->ncolumns; + + return ind1->ncolumns == ind2->ncolumns && + ind1->unique == ind2->unique && + memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0 && + memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0 && + memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0 && + memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0 && + equal(get_tlist_exprs(ind1->indextlist, true), + get_tlist_exprs(ind2->indextlist, true)); +} + + +/* + * The below macros are used for simple_copy_indexinfo_to_parent which is so + * customized that I don't want to put it to copyfuncs.c. So copy it here. + */ +#define COPY_POINTER_FIELD(fldname, sz) \ + do { \ + Size _size = (sz); \ + newnode->fldname = palloc(_size); \ + memcpy(newnode->fldname, from->fldname, _size); \ + } while (0) + +#define COPY_NODE_FIELD(fldname) \ + (newnode->fldname = copyObjectImpl(from->fldname)) + +#define COPY_SCALAR_FIELD(fldname) \ + (newnode->fldname = from->fldname) + + +/* + * simple_copy_indexinfo_to_parent (from partition) + * Copy the IndexInfo from child relation to parent relation with some modification, + * which is used to test: + * 1. If the same index exists in all the childrels. + * 2. If the parentrel->reltarget/basicrestrict info matches this index. + */ +static IndexOptInfo * +simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from) +{ + IndexOptInfo *newnode = makeNode(IndexOptInfo); + AppendRelInfo *appinfo = find_appinfo_by_child(root, from->rel->relid); + ListCell *lc; + int idx = 0; + + COPY_SCALAR_FIELD(ncolumns); + COPY_SCALAR_FIELD(nkeycolumns); + COPY_SCALAR_FIELD(unique); + COPY_SCALAR_FIELD(immediate); + /* We just need to know if it is NIL or not */ + COPY_SCALAR_FIELD(indpred); + COPY_SCALAR_FIELD(predOK); + COPY_POINTER_FIELD(indexkeys, from->ncolumns * sizeof(int)); + COPY_POINTER_FIELD(indexcollations, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opfamily, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opcintype, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(sortopfamily, from->ncolumns * sizeof(Oid)); + COPY_NODE_FIELD(indextlist); + + /* Convert index exprs on child expr to expr on parent */ + foreach(lc, newnode->indextlist) + { + TargetEntry *tle = lfirst_node(TargetEntry, lc); + /* Index on expression is ignored */ + Assert(IsA(tle->expr, Var)); + tle->expr = (Expr *) find_parent_var(appinfo, (Var *) tle->expr); + newnode->indexkeys[idx] = castNode(Var, tle->expr)->varattno; + idx++; + } + newnode->rel = parentrel; + return newnode; +} + +/* + * adjust_partition_unique_indexlist + * + * global_unique_indexes: At the beginning, it contains the copy & modified + * unique index from the first partition. And then check if each index in it still + * exists in the following partitions. If no, remove it. at last, it has an + * index list which exists in all the partitions. + */ +static void +adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_indexes) +{ + ListCell *lc, *lc2; + foreach(lc, *global_unique_indexes) + { + IndexOptInfo *g_ind = lfirst_node(IndexOptInfo, lc); + bool found_in_child = false; + + foreach(lc2, childrel->indexlist) + { + IndexOptInfo *p_ind = lfirst_node(IndexOptInfo, lc2); + IndexOptInfo *p_ind_copy; + if (!p_ind->unique || !p_ind->immediate || + (p_ind->indpred != NIL && !p_ind->predOK)) + continue; + p_ind_copy = simple_copy_indexinfo_to_parent(root, parentrel, p_ind); + if (simple_indexinfo_equal(p_ind_copy, g_ind)) + { + found_in_child = true; + break; + } + } + if (!found_in_child) + /* The index doesn't exist in childrel, remove it from global_unique_indexes */ + *global_unique_indexes = foreach_delete_current(*global_unique_indexes, lc); + } +} + +/* Helper function for groupres/distinctrel */ +static void +add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgroups) +{ + Query *parse = root->parse; + List *exprs; + + /* + * XXX: If there are some vars which is not in current levelsup, the semantic is + * imprecise, should we avoid it or not? levelsup = 1 is just a demo, maybe we need to + * check every level other than 0, if so, looks we have to write another + * pull_var_walker. + */ + List *upper_vars = pull_vars_of_level((Node*)sortgroups, 1); + + if (upper_vars != NIL) + return; + + exprs = get_sortgrouplist_exprs(sortgroups, parse->targetList); + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(exprs, + false /* sortgroupclause can't be multi_nullvals */)); +} + + +/* + * add_combined_uniquekey + * The combination of both UniqueKeys is a valid UniqueKey for joinrel no matter + * the jointype. + */ +bool +add_combined_uniquekey(RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype) +{ + + ListCell *lc1, *lc2; + + /* Either side has multi_nullvals or we have outer join, + * the combined UniqueKey has multi_nullvals */ + bool multi_nullvals = outer_ukey->multi_nullvals || + inner_ukey->multi_nullvals || IS_OUTER_JOIN(jointype); + + /* The only case we can get onerow joinrel after join */ + if (relation_is_onerow(outer_rel) + && relation_is_onerow(inner_rel) + && jointype == JOIN_INNER) + { + add_uniquekey_for_onerow(joinrel); + return true; + } + + foreach(lc1, get_exprs_from_uniquekey(joinrel, outer_rel, outer_ukey)) + { + foreach(lc2, get_exprs_from_uniquekey(joinrel, inner_rel, inner_ukey)) + { + List *exprs = list_concat_copy(lfirst_node(List, lc1), lfirst_node(List, lc2)); + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(exprs, + multi_nullvals)); + } + } + return false; +} diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index b406d41e91..0551ae0512 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -2389,6 +2389,8 @@ grouping_planner(PlannerInfo *root, bool inheritance_update, add_path(final_rel, path); } + simple_copy_uniquekeys(current_rel, final_rel); + /* * Generate partial paths for final_rel, too, if outer query levels might * be able to make use of them. @@ -3899,6 +3901,8 @@ create_grouping_paths(PlannerInfo *root, } set_cheapest(grouped_rel); + + populate_grouprel_uniquekeys(root, grouped_rel, input_rel); return grouped_rel; } @@ -4615,7 +4619,7 @@ create_window_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(window_rel); - + simple_copy_uniquekeys(input_rel, window_rel); return window_rel; } @@ -4911,7 +4915,7 @@ create_distinct_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(distinct_rel); - + populate_distinctrel_uniquekeys(root, input_rel, distinct_rel); return distinct_rel; } @@ -5172,6 +5176,8 @@ create_ordered_paths(PlannerInfo *root, */ Assert(ordered_rel->pathlist != NIL); + simple_copy_uniquekeys(input_rel, ordered_rel); + return ordered_rel; } @@ -6049,6 +6055,9 @@ adjust_paths_for_srfs(PlannerInfo *root, RelOptInfo *rel, if (list_length(targets) == 1) return; + /* UniqueKey is not valid after handling the SRF. */ + rel->uniquekeys = NIL; + /* * Stack SRF-evaluation nodes atop each path for the rel. * diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c index 951aed80e7..e94e92937c 100644 --- a/src/backend/optimizer/prep/prepunion.c +++ b/src/backend/optimizer/prep/prepunion.c @@ -689,6 +689,8 @@ generate_union_paths(SetOperationStmt *op, PlannerInfo *root, /* Undo effects of possibly forcing tuple_fraction to 0 */ root->tuple_fraction = save_fraction; + /* Add the UniqueKeys */ + populate_unionrel_uniquekeys(root, result_rel); return result_rel; } diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c index d722063cf3..44c37ecffc 100644 --- a/src/backend/optimizer/util/appendinfo.c +++ b/src/backend/optimizer/util/appendinfo.c @@ -746,3 +746,47 @@ find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos) } return appinfos; } + +/* + * find_appinfo_by_child + * + */ +AppendRelInfo * +find_appinfo_by_child(PlannerInfo *root, Index child_index) +{ + ListCell *lc; + foreach(lc, root->append_rel_list) + { + AppendRelInfo *appinfo = lfirst_node(AppendRelInfo, lc); + if (appinfo->child_relid == child_index) + return appinfo; + } + elog(ERROR, "parent relation cant be found"); + return NULL; +} + +/* + * find_parent_var + * + */ +Var * +find_parent_var(AppendRelInfo *appinfo, Var *child_var) +{ + ListCell *lc; + Var *res = NULL; + Index attno = 1; + foreach(lc, appinfo->translated_vars) + { + Node *child_node = lfirst(lc); + if (equal(child_node, child_var)) + { + res = copyObject(child_var); + res->varattno = attno; + res->varno = appinfo->parent_relid; + } + attno++; + } + if (res == NULL) + elog(ERROR, "parent var can't be found."); + return res; +} diff --git a/src/backend/optimizer/util/inherit.c b/src/backend/optimizer/util/inherit.c index 3132fd35a5..d66b40ec50 100644 --- a/src/backend/optimizer/util/inherit.c +++ b/src/backend/optimizer/util/inherit.c @@ -736,6 +736,7 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, { Node *onecq = (Node *) lfirst(lc2); bool pseudoconstant; + RestrictInfo *child_rinfo; /* check for pseudoconstant (no Vars or volatile functions) */ pseudoconstant = @@ -747,13 +748,14 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, root->hasPseudoConstantQuals = true; } /* reconstitute RestrictInfo with appropriate properties */ - childquals = lappend(childquals, - make_restrictinfo((Expr *) onecq, - rinfo->is_pushed_down, - rinfo->outerjoin_delayed, - pseudoconstant, - rinfo->security_level, - NULL, NULL, NULL)); + child_rinfo = make_restrictinfo((Expr *) onecq, + rinfo->is_pushed_down, + rinfo->outerjoin_delayed, + pseudoconstant, + rinfo->security_level, + NULL, NULL, NULL); + child_rinfo->mergeopfamilies = rinfo->mergeopfamilies; + childquals = lappend(childquals, child_rinfo); /* track minimum security level among child quals */ cq_min_security = Min(cq_min_security, rinfo->security_level); } diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 31d9aedeeb..c83f17acb7 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -16,6 +16,7 @@ #include "nodes/execnodes.h" #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" extern A_Expr *makeA_Expr(A_Expr_Kind kind, List *name, @@ -105,4 +106,6 @@ extern GroupingSet *makeGroupingSet(GroupingSetKind kind, List *content, int loc extern VacuumRelation *makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols); +extern UniqueKey* makeUniqueKey(List *exprs, bool multi_nullvals); + #endif /* MAKEFUNC_H */ diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 381d84b4e4..41110ed888 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -264,6 +264,7 @@ typedef enum NodeTag T_EquivalenceMember, T_PathKey, T_PathTarget, + T_UniqueKey, T_RestrictInfo, T_IndexClause, T_PlaceHolderVar, diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index 9e3ebd488a..02e4458bef 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -730,6 +730,7 @@ typedef struct RelOptInfo QualCost baserestrictcost; /* cost of evaluating the above */ Index baserestrict_min_security; /* min security_level found in * baserestrictinfo */ + List *uniquekeys; /* List of UniqueKey */ List *joininfo; /* RestrictInfo structures for join clauses * involving this rel */ bool has_eclass_joins; /* T means joininfo is incomplete */ @@ -1047,6 +1048,28 @@ typedef struct PathKey } PathKey; +/* + * UniqueKey + * + * Represents the unique properties held by a RelOptInfo. + * + * exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL + * is a special case of UniqueKey, which means there is only 1 row in that + * relation. + * multi_nullvals: true means multi null values may exist in these exprs, so the + * uniqueness is not guaranteed in this case. This field is necessary for + * remove_useless_join & reduce_unique_semijoins where we don't mind these + * duplicated NULL values. It is set to true for 2 cases. One is a unique key + * from a unique index but the related column is nullable. The other one is for + * outer join. see populate_joinrel_uniquekeys for detail. + */ +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + /* * PathTarget * @@ -2473,7 +2496,7 @@ typedef enum * * flags indicating what kinds of grouping are possible. * partial_costs_set is true if the agg_partial_costs and agg_final_costs - * have been initialized. + * have been initialized. * agg_partial_costs gives partial aggregation costs. * agg_final_costs gives finalization costs. * target_parallel_safe is true if target is parallel safe. @@ -2503,8 +2526,8 @@ typedef struct * limit_tuples is an estimated bound on the number of output tuples, * or -1 if no LIMIT or couldn't estimate. * count_est and offset_est are the estimated values of the LIMIT and OFFSET - * expressions computed by preprocess_limit() (see comments for - * preprocess_limit() for more information). + * expressions computed by preprocess_limit() (see comments for + * preprocess_limit() for more information). */ typedef struct { diff --git a/src/include/nodes/pg_list.h b/src/include/nodes/pg_list.h index 14ea2766ad..621f54a9f8 100644 --- a/src/include/nodes/pg_list.h +++ b/src/include/nodes/pg_list.h @@ -528,6 +528,8 @@ extern bool list_member_ptr(const List *list, const void *datum); extern bool list_member_int(const List *list, int datum); extern bool list_member_oid(const List *list, Oid datum); +extern bool list_is_subset(const List *members, const List *target); + extern List *list_delete(List *list, void *datum); extern List *list_delete_ptr(List *list, void *datum); extern List *list_delete_int(List *list, int datum); diff --git a/src/include/optimizer/appendinfo.h b/src/include/optimizer/appendinfo.h index d6a27a60dd..e87c92a054 100644 --- a/src/include/optimizer/appendinfo.h +++ b/src/include/optimizer/appendinfo.h @@ -32,4 +32,7 @@ extern Relids adjust_child_relids_multilevel(PlannerInfo *root, Relids relids, extern AppendRelInfo **find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos); +extern AppendRelInfo *find_appinfo_by_child(PlannerInfo *root, Index child_index); +extern Var *find_parent_var(AppendRelInfo *appinfo, Var *child_var); + #endif /* APPENDINFO_H */ diff --git a/src/include/optimizer/optimizer.h b/src/include/optimizer/optimizer.h index 3e4171056e..9445141263 100644 --- a/src/include/optimizer/optimizer.h +++ b/src/include/optimizer/optimizer.h @@ -23,6 +23,7 @@ #define OPTIMIZER_H #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" /* * We don't want to include nodes/pathnodes.h here, because non-planner @@ -156,6 +157,7 @@ extern TargetEntry *get_sortgroupref_tle(Index sortref, List *targetList); extern TargetEntry *get_sortgroupclause_tle(SortGroupClause *sgClause, List *targetList); +extern Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); extern Node *get_sortgroupclause_expr(SortGroupClause *sgClause, List *targetList); extern List *get_sortgrouplist_exprs(List *sgClauses, diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h index 10b6e81079..9217a8d6c6 100644 --- a/src/include/optimizer/paths.h +++ b/src/include/optimizer/paths.h @@ -240,5 +240,48 @@ extern PathKey *make_canonical_pathkey(PlannerInfo *root, int strategy, bool nulls_first); extern void add_paths_to_append_rel(PlannerInfo *root, RelOptInfo *rel, List *live_childrels); +extern List *select_mergejoin_clauses(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + JoinType jointype, + bool *mergejoin_allowed); + +/* + * uniquekeys.c + * Utilities for matching and building unique keys + */ +extern void populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List* unique_index_list); +extern void populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels); +extern void populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel); +extern void populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel); +extern void populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel); +extern void simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel); +extern void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel); +extern void populate_joinrel_uniquekeys(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + RelOptInfo *rel2, + List *restrictlist, + JoinType jointype); + +extern bool relation_has_uniquekeys_for(PlannerInfo *root, + RelOptInfo *rel, + List *exprs, + bool allow_multinulls); +extern bool relation_is_onerow(RelOptInfo *rel); #endif /* PATHS_H */ -- 2.21.0 --op7mf2i72xk5yu7c Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v36-0003-Extend-UniqueKeys.patch" ^ permalink raw reply [nested|flat] 91+ messages in thread
* [PATCH v38 2/6] Introduce UniqueKey attributes on RelOptInfo struct. @ 2020-05-11 07:50 一挃 <[email protected]> 0 siblings, 0 replies; 91+ messages in thread From: 一挃 @ 2020-05-11 07:50 UTC (permalink / raw) UniqueKey is a set of exprs on RelOptInfo which represents the exprs will be unique on the given RelOptInfo. You can see README.uniquekey for more information. --- src/backend/nodes/copyfuncs.c | 13 + src/backend/nodes/list.c | 31 + src/backend/nodes/makefuncs.c | 13 + src/backend/nodes/outfuncs.c | 11 + src/backend/nodes/readfuncs.c | 10 + src/backend/optimizer/path/Makefile | 3 +- src/backend/optimizer/path/README.uniquekey | 131 +++ src/backend/optimizer/path/allpaths.c | 10 + src/backend/optimizer/path/joinpath.c | 9 +- src/backend/optimizer/path/joinrels.c | 2 + src/backend/optimizer/path/pathkeys.c | 3 +- src/backend/optimizer/path/uniquekeys.c | 1135 +++++++++++++++++++ src/backend/optimizer/plan/planner.c | 13 +- src/backend/optimizer/prep/prepunion.c | 2 + src/backend/optimizer/util/appendinfo.c | 44 + src/backend/optimizer/util/inherit.c | 18 +- src/include/nodes/makefuncs.h | 3 + src/include/nodes/nodes.h | 1 + src/include/nodes/pathnodes.h | 29 +- src/include/nodes/pg_list.h | 10 + src/include/optimizer/appendinfo.h | 3 + src/include/optimizer/optimizer.h | 2 + src/include/optimizer/paths.h | 43 + 23 files changed, 1515 insertions(+), 24 deletions(-) create mode 100644 src/backend/optimizer/path/README.uniquekey create mode 100644 src/backend/optimizer/path/uniquekeys.c diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..75c1c5e824 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2296,6 +2296,16 @@ _copyPathKey(const PathKey *from) return newnode; } +static UniqueKey * +_copyUniqueKey(const UniqueKey *from) +{ + UniqueKey *newnode = makeNode(UniqueKey); + + COPY_NODE_FIELD(exprs); + COPY_SCALAR_FIELD(multi_nullvals); + + return newnode; +} /* * _copyRestrictInfo */ @@ -5220,6 +5230,9 @@ copyObjectImpl(const void *from) case T_PathKey: retval = _copyPathKey(from); break; + case T_UniqueKey: + retval = _copyUniqueKey(from); + break; case T_RestrictInfo: retval = _copyRestrictInfo(from); break; diff --git a/src/backend/nodes/list.c b/src/backend/nodes/list.c index dbf6b30233..ca099495a1 100644 --- a/src/backend/nodes/list.c +++ b/src/backend/nodes/list.c @@ -702,6 +702,37 @@ list_member_oid(const List *list, Oid datum) return false; } +/* + * return true iff every entry in "members" list is also present + * in the "target" list. + */ +bool +list_is_subset(const List *members, const List *target) +{ + const ListCell *lc1, *lc2; + + Assert(IsPointerList(members)); + Assert(IsPointerList(target)); + check_list_invariants(members); + check_list_invariants(target); + + foreach(lc1, members) + { + bool found = false; + foreach(lc2, target) + { + if (equal(lfirst(lc1), lfirst(lc2))) + { + found = true; + break; + } + } + if (!found) + return false; + } + return true; +} + /* * Delete the n'th cell (counting from 0) in list. * diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 01c110cd2f..40415d0f5b 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -815,3 +815,16 @@ makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols) v->va_cols = va_cols; return v; } + + +/* + * makeUniqueKey + */ +UniqueKey* +makeUniqueKey(List *exprs, bool multi_nullvals) +{ + UniqueKey * ukey = makeNode(UniqueKey); + ukey->exprs = exprs; + ukey->multi_nullvals = multi_nullvals; + return ukey; +} diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..44154cde6a 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2456,6 +2456,14 @@ _outPathKey(StringInfo str, const PathKey *node) WRITE_BOOL_FIELD(pk_nulls_first); } +static void +_outUniqueKey(StringInfo str, const UniqueKey *node) +{ + WRITE_NODE_TYPE("UNIQUEKEY"); + WRITE_NODE_FIELD(exprs); + WRITE_BOOL_FIELD(multi_nullvals); +} + static void _outPathTarget(StringInfo str, const PathTarget *node) { @@ -4198,6 +4206,9 @@ outNode(StringInfo str, const void *obj) case T_PathKey: _outPathKey(str, obj); break; + case T_UniqueKey: + _outUniqueKey(str, obj); + break; case T_PathTarget: _outPathTarget(str, obj); break; diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index c5e136e9c3..b3e212bf1c 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -492,6 +492,14 @@ _readSetOperationStmt(void) READ_DONE(); } +static UniqueKey * +_readUniqueKey(void) +{ + READ_LOCALS(UniqueKey); + READ_NODE_FIELD(exprs); + READ_BOOL_FIELD(multi_nullvals); + READ_DONE(); +} /* * Stuff from primnodes.h. @@ -2717,6 +2725,8 @@ parseNodeString(void) return_value = _readCommonTableExpr(); else if (MATCH("SETOPERATIONSTMT", 16)) return_value = _readSetOperationStmt(); + else if (MATCH("UNIQUEKEY", 9)) + return_value = _readUniqueKey(); else if (MATCH("ALIAS", 5)) return_value = _readAlias(); else if (MATCH("RANGEVAR", 8)) diff --git a/src/backend/optimizer/path/Makefile b/src/backend/optimizer/path/Makefile index 1e199ff66f..7b9820c25f 100644 --- a/src/backend/optimizer/path/Makefile +++ b/src/backend/optimizer/path/Makefile @@ -21,6 +21,7 @@ OBJS = \ joinpath.o \ joinrels.o \ pathkeys.o \ - tidpath.o + tidpath.o \ + uniquekeys.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/optimizer/path/README.uniquekey b/src/backend/optimizer/path/README.uniquekey new file mode 100644 index 0000000000..5eac761995 --- /dev/null +++ b/src/backend/optimizer/path/README.uniquekey @@ -0,0 +1,131 @@ +1. What is UniqueKey? +We can think UniqueKey is a set of exprs for a RelOptInfo, which we are insure +that doesn't yields same result among all the rows. The simplest UniqueKey +format is primary key. + +However we define the UnqiueKey as below. + +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + +exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL +is a special case of UniqueKey, which means there is only one row in that +relation.it has a stronger semantic than others. like SELECT uk FROM t; uk is +normal unique key and may have different values. SELECT colx FROM t WHERE uk = +const. colx is unique AND we have only 1 value. This field can used for +innerrel_is_unique. this logic is handled specially in add_uniquekey_for_onerow +function. + +multi_nullvals: true means multi null values may exist in these exprs, so the +uniqueness is not guaranteed in this case. This field is necessary for +remove_useless_join & reduce_unique_semijoins where we don't mind these +duplicated NULL values. It is set to true for 2 cases. One is a unique key +from a unique index but the related column is nullable. The other one is for +outer join. see populate_joinrel_uniquekeys for detail. + + +The UniqueKey can be used at the following cases at least: +1. remove_useless_joins. +2. reduce_semianti_joins +3. remove distinct node if distinct clause is unique. +4. remove aggnode if group by clause is unique. +5. Index Skip Scan (WIP) +6. Aggregation Push Down without 2 phase aggregation if the join can't + duplicated the aggregated rows. (work in progress feature) + +2. How is it maintained? + +We have a set of populate_xxx_unqiuekeys functions to maintain the uniquekey on +various cases. xxx includes baserel, joinrel, partitionedrel, distinctrel, +groupedrel, unionrel. and we also need to convert the uniquekey from subquery +to outer relation, which is what convert_subquery_uniquekeys does. + +1. The first part is about baserel. We handled 3 cases. suppose we have Unique +Index on (a, b). + +1. SELECT a, b FROM t. UniqueKey (a, b) +2. SELECT a FROM t WHERE b = 1; UniqueKey (a) +3. SELECT .. FROM t WHERE a = 1 AND b = 1; UniqueKey (NIL). onerow case, every + column is Unique. + +2. The next part is joinrel, this part is most error-prone, we simplified the rules +like below: +1. If the relation's UniqueKey can't be duplicated after join, then is will be + still valid for the join rel. The function we used here is + innerrel_keeps_unique. The basic idea is innerrel.any_col = outer.uk. + +2. If the UnqiueKey can't keep valid via the rule 1, the combination of the + UniqueKey from both sides are valid for sure. We can prove this as: if the + unique exprs from rel1 is duplicated by rel2, the duplicated rows must + contains different unique exprs from rel2. + +More considerations about onerow: +1. If relation with one row and it can't be duplicated, it is still possible + contains mulit_nullvas after outer join. +2. If the either UniqueKey can be duplicated after join, the can get one row + only when both side is one row AND there is no outer join. +3. Whenever the onerow UniqueKey is not a valid any more, we need to convert one + row UniqueKey to normal unique key since we don't store exprs for one-row + relation. get_exprs_from_uniquekeys will be used here. + + +More considerations about multi_nullvals after join: +1. If the original UnqiueKey has multi_nullvals, the final UniqueKey will have + mulit_nullvals in any case. +2. If a unique key doesn't allow mulit_nullvals, after some outer join, it + allows some outer join. + + +3. When we comes to subquery, we need to convert_subquery_unqiuekeys just like +convert_subquery_pathkeys. Only the UniqueKey insides subquery is referenced as +a Var in outer relation will be reused. The relationship between the outerrel.Var +and subquery.exprs is built with outerel->subroot->processed_tlist. + + +4. As for the SRF functions, it will break the uniqueness of uniquekey, However it +is handled in adjust_paths_for_srfs, which happens after the query_planner. so +we will maintain the UniqueKey until there and reset it to NIL at that +places. This can't help on distinct/group by elimination cases but probably help +in some other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is +semantic correctly. + + +5. As for inherit table, we first main the UnqiueKey on childrel as well. But for +partitioned table we need to maintain 2 different kinds of +UnqiueKey. 1). UniqueKey on the parent relation 2). UniqueKey on child +relation for partition wise query. + +Example: +CREATE TABLE p (a int not null, b int not null) partition by list (a); +CREATE TABLE p0 partition of p for values in (1); +CREATE TABLE p1 partition of p for values in (2); + +create unique index p0_b on p0(b); +create unique index p1_b on p1(b); + +Now b is only unique on partition level, so the distinct can't be removed on +the following cases. SELECT DISTINCT b FROM p; + +Another example is SELECT DISTINCT a, b FROM p WHERE a = 1; Since only one +partition is chosen, the UniqueKey on child relation is same as the UniqueKey on +parent relation. + +Another usage of UniqueKey on partition level is it be helpful for +partition-wise join. + +As for the UniqueKey on parent table level, it comes with 2 different ways, +1). the UniqueKey is also derived in UniqueKey index, but the index must be same +in all the related children relations and the unique index must contains +Partition Key in it. Example: + +CREATE UNIQUE INDEX p_ab ON p(a, b); -- where a is the partition key. + +-- Query +SELECT a, b FROM p; the (a, b) is a UniqueKey of p. + +2). If the parent relation has only one childrel, the UniqueKey on childrel is + the UniqueKey on parent as well. diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index 37b4223adb..d52ad59f89 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -579,6 +579,12 @@ set_plain_rel_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) */ check_index_predicates(root, rel); + /* + * Now that we've marked which partial indexes are suitable, we can now + * build the relation's unique keys. + */ + populate_baserel_uniquekeys(root, rel, rel->indexlist); + /* Mark rel with estimated output rows, width, etc */ set_baserel_size_estimates(root, rel); } @@ -1297,6 +1303,8 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, /* Add paths to the append relation. */ add_paths_to_append_rel(root, rel, live_childrels); + if (IS_PARTITIONED_REL(rel)) + populate_partitionedrel_uniquekeys(root, rel, live_childrels); } @@ -2304,6 +2312,8 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, pathkeys, required_outer)); } + convert_subquery_uniquekeys(root, rel, sub_final_rel); + /* If outer rel allows parallelism, do same for partial paths. */ if (rel->consider_parallel && bms_is_empty(required_outer)) { diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c index 57ce97fd53..697ac047fb 100644 --- a/src/backend/optimizer/path/joinpath.c +++ b/src/backend/optimizer/path/joinpath.c @@ -71,13 +71,6 @@ static void consider_parallel_mergejoin(PlannerInfo *root, static void hash_inner_and_outer(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, RelOptInfo *innerrel, JoinType jointype, JoinPathExtraData *extra); -static List *select_mergejoin_clauses(PlannerInfo *root, - RelOptInfo *joinrel, - RelOptInfo *outerrel, - RelOptInfo *innerrel, - List *restrictlist, - JoinType jointype, - bool *mergejoin_allowed); static void generate_mergejoin_paths(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *innerrel, @@ -1927,7 +1920,7 @@ hash_inner_and_outer(PlannerInfo *root, * if it is mergejoinable and involves vars from the two sub-relations * currently of interest. */ -static List * +List * select_mergejoin_clauses(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c index 0dbe2ac726..7271f044ec 100644 --- a/src/backend/optimizer/path/joinrels.c +++ b/src/backend/optimizer/path/joinrels.c @@ -924,6 +924,8 @@ populate_joinrel_with_paths(PlannerInfo *root, RelOptInfo *rel1, /* Apply partitionwise join technique, if possible. */ try_partitionwise_join(root, rel1, rel2, joinrel, sjinfo, restrictlist); + + populate_joinrel_uniquekeys(root, joinrel, rel1, rel2, restrictlist, sjinfo->jointype); } diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index bd9a176d7d..139278829b 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -33,7 +33,6 @@ static bool pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys); static bool matches_boolean_partition_clause(RestrictInfo *rinfo, RelOptInfo *partrel, int partkeycol); -static Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); static bool right_merge_direction(PlannerInfo *root, PathKey *pathkey); @@ -1035,7 +1034,7 @@ convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel, * We need this to ensure that we don't return pathkeys describing values * that are unavailable above the level of the subquery scan. */ -static Var * +Var * find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle) { ListCell *lc; diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c new file mode 100644 index 0000000000..77ed2b2eff --- /dev/null +++ b/src/backend/optimizer/path/uniquekeys.c @@ -0,0 +1,1135 @@ +/*------------------------------------------------------------------------- + * + * uniquekeys.c + * Utilities for matching and building unique keys + * + * Portions Copyright (c) 2020, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/optimizer/path/uniquekeys.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "nodes/makefuncs.h" +#include "nodes/nodeFuncs.h" +#include "optimizer/pathnode.h" +#include "optimizer/paths.h" +#include "optimizer/appendinfo.h" +#include "optimizer/optimizer.h" +#include "optimizer/tlist.h" +#include "rewrite/rewriteManip.h" + + +/* + * This struct is used to help populate_joinrel_uniquekeys. + * + * added_to_joinrel is true if a uniquekey (from outerrel or innerrel) + * has been added to joinrel. + * useful is true if the exprs of the uniquekey still appears in joinrel. + */ +typedef struct UniqueKeyContextData +{ + UniqueKey *uniquekey; + bool added_to_joinrel; + bool useful; +} *UniqueKeyContext; + +static List *initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel); +static bool innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + bool reverse); + +static List *get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals); +static List *get_exprs_from_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + UniqueKey *ukey); +static void add_uniquekey_for_onerow(RelOptInfo *rel); +static bool add_combined_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype); + +/* Used for unique indexes checking for partitioned table */ +static bool index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind); +static IndexOptInfo *simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from); +static bool simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2); +static void adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_index); + +/* Helper function for grouped relation and distinct relation. */ +static void add_uniquekey_from_sortgroups(PlannerInfo *root, + RelOptInfo *rel, + List *sortgroups); + +/* + * populate_baserel_uniquekeys + * Populate 'baserel' uniquekeys list by looking at the rel's unique index + * and baserestrictinfo + */ +void +populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List *indexlist) +{ + ListCell *lc; + List *matched_uniq_indexes = NIL; + + /* Attrs appears in rel->reltarget->exprs. */ + Bitmapset *used_attrs = NULL; + + List *const_exprs = NIL; + List *expr_opfamilies = NIL; + + Assert(baserel->rtekind == RTE_RELATION); + + foreach(lc, indexlist) + { + IndexOptInfo *ind = (IndexOptInfo *) lfirst(lc); + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + matched_uniq_indexes = lappend(matched_uniq_indexes, ind); + } + + if (matched_uniq_indexes == NIL) + return; + + /* Check which attrs is used in baserel->reltarget */ + pull_varattnos((Node *)baserel->reltarget->exprs, baserel->relid, &used_attrs); + + /* Check which attrno is used at a mergeable const filter */ + foreach(lc, baserel->baserestrictinfo) + { + RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); + + if (rinfo->mergeopfamilies == NIL) + continue; + + if (bms_is_empty(rinfo->left_relids)) + { + const_exprs = lappend(const_exprs, get_rightop(rinfo->clause)); + } + else if (bms_is_empty(rinfo->right_relids)) + { + const_exprs = lappend(const_exprs, get_leftop(rinfo->clause)); + } + else + continue; + + expr_opfamilies = lappend(expr_opfamilies, rinfo->mergeopfamilies); + } + + foreach(lc, matched_uniq_indexes) + { + bool multi_nullvals, useful; + List *exprs = get_exprs_from_uniqueindex(lfirst_node(IndexOptInfo, lc), + const_exprs, + expr_opfamilies, + used_attrs, + &useful, + &multi_nullvals); + if (useful) + { + if (exprs == NIL) + { + /* All the columns in Unique Index matched with a restrictinfo */ + add_uniquekey_for_onerow(baserel); + return; + } + baserel->uniquekeys = lappend(baserel->uniquekeys, + makeUniqueKey(exprs, multi_nullvals)); + } + } +} + + +/* + * populate_partitionedrel_uniquekeys + * The UniqueKey on partitionrel comes from 2 cases: + * 1). Only one partition is involved in this query, the unique key can be + * copied to parent rel from childrel. + * 2). There are some unique index which includes partition key and exists + * in all the related partitions. + * We never mind rule 2 if we hit rule 1. + */ + +void +populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels) +{ + ListCell *lc; + List *global_uniq_indexlist = NIL; + RelOptInfo *childrel; + bool is_first = true; + + Assert(IS_PARTITIONED_REL(rel)); + + if (childrels == NIL) + return; + + /* + * If there is only one partition used in this query, the UniqueKey in childrel is + * still valid in parent level, but we need convert the format from child expr to + * parent expr. + */ + if (list_length(childrels) == 1) + { + /* Check for Rule 1 */ + RelOptInfo *childrel = linitial_node(RelOptInfo, childrels); + ListCell *lc; + Assert(childrel->reloptkind == RELOPT_OTHER_MEMBER_REL); + if (relation_is_onerow(childrel)) + { + add_uniquekey_for_onerow(rel); + return; + } + + foreach(lc, childrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + AppendRelInfo *appinfo = find_appinfo_by_child(root, childrel->relid); + List *parent_exprs = NIL; + bool can_reuse = true; + ListCell *lc2; + foreach(lc2, ukey->exprs) + { + Var *var = (Var *)lfirst(lc2); + /* + * If the expr comes from a expression, it is hard to build the expression + * in parent so ignore that case for now. + */ + if(!IsA(var, Var)) + { + can_reuse = false; + break; + } + /* Convert it to parent var */ + parent_exprs = lappend(parent_exprs, find_parent_var(appinfo, var)); + } + if (can_reuse) + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(parent_exprs, + ukey->multi_nullvals)); + } + } + else + { + /* Check for rule 2 */ + childrel = linitial_node(RelOptInfo, childrels); + foreach(lc, childrel->indexlist) + { + IndexOptInfo *ind = lfirst(lc); + IndexOptInfo *modified_index; + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + + /* + * During simple_copy_indexinfo_to_parent, we need to convert var from + * child var to parent var, index on expression is too complex to handle. + * so ignore it for now. + */ + if (ind->indexprs != NIL) + continue; + + modified_index = simple_copy_indexinfo_to_parent(root, rel, ind); + /* + * If the unique index doesn't contain partkey, then it is unique + * on this partition only, so it is useless for us. + */ + if (!index_constains_partkey(rel, modified_index)) + continue; + + global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index); + } + + if (global_uniq_indexlist != NIL) + { + foreach(lc, childrels) + { + RelOptInfo *child = lfirst(lc); + if (is_first) + { + is_first = false; + continue; + } + adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist); + } + /* Now we have a list of unique index which are exactly same on all childrels, + * Set the UniqueKey just like it is non-partition table + */ + populate_baserel_uniquekeys(root, rel, global_uniq_indexlist); + } + } +} + + +/* + * populate_distinctrel_uniquekeys + */ +void +populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel) +{ + /* The unique key before the distinct is still valid. */ + distinctrel->uniquekeys = list_copy(inputrel->uniquekeys); + add_uniquekey_from_sortgroups(root, distinctrel, root->parse->distinctClause); +} + +/* + * populate_grouprel_uniquekeys + */ +void +populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel) + +{ + Query *parse = root->parse; + bool input_ukey_added = false; + ListCell *lc; + + if (relation_is_onerow(inputrel)) + { + add_uniquekey_for_onerow(grouprel); + return; + } + if (parse->groupingSets) + return; + + /* A Normal group by without grouping set. */ + if (parse->groupClause) + { + /* + * Current even the groupby clause is Unique already, but if query has aggref + * We have to create grouprel still. To keep the UnqiueKey short, we will check + * the UniqueKey of input_rel still valid, if so we reuse it. + */ + foreach(lc, inputrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (list_is_subset(ukey->exprs, grouprel->reltarget->exprs)) + { + grouprel->uniquekeys = lappend(grouprel->uniquekeys, + ukey); + input_ukey_added = true; + } + } + if (!input_ukey_added) + /* + * group by clause must be a super-set of grouprel->reltarget->exprs except the + * aggregation expr, so if such exprs is unique already, no bother to generate + * new uniquekey for group by exprs. + */ + add_uniquekey_from_sortgroups(root, + grouprel, + root->parse->groupClause); + } + else + /* It has aggregation but without a group by, so only one row returned */ + add_uniquekey_for_onerow(grouprel); +} + +/* + * simple_copy_uniquekeys + * Using a function for the one-line code makes us easy to check where we simply + * copied the uniquekey. + */ +void +simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel) +{ + newrel->uniquekeys = oldrel->uniquekeys; +} + +/* + * populate_unionrel_uniquekeys + */ +void +populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel) +{ + ListCell *lc; + List *exprs = NIL; + + Assert(unionrel->uniquekeys == NIL); + + foreach(lc, unionrel->reltarget->exprs) + { + exprs = lappend(exprs, lfirst(lc)); + } + + if (exprs == NIL) + /* SQL: select union select; is valid, we need to handle it here. */ + add_uniquekey_for_onerow(unionrel); + else + unionrel->uniquekeys = lappend(unionrel->uniquekeys, + makeUniqueKey(exprs,false)); + +} + +/* + * populate_joinrel_uniquekeys + * + * populate uniquekeys for joinrel. We will check each relation to see if its + * UniqueKey is still valid via innerrel_keeps_unique, if so, we add it to + * joinrel. The multi_nullvals field will be changed to true for some outer + * join cases and one-row UniqueKey needs to be converted to normal UniqueKey + * for the same case as well. + * For the uniquekey in either baserel which can't be unique after join, we still + * check to see if combination of UniqueKeys from both side is still useful for us. + * if yes, we add it to joinrel as well. + */ +void +populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *outerrel, RelOptInfo *innerrel, + List *restrictlist, JoinType jointype) +{ + ListCell *lc, *lc2; + List *clause_list = NIL; + List *outerrel_ukey_ctx; + List *innerrel_ukey_ctx; + bool inner_onerow, outer_onerow; + bool mergejoin_allowed; + + /* Care about the outerrel relation only for SEMI/ANTI join */ + if (jointype == JOIN_SEMI || jointype == JOIN_ANTI) + { + foreach(lc, outerrel->uniquekeys) + { + UniqueKey *uniquekey = lfirst_node(UniqueKey, lc); + if (list_is_subset(uniquekey->exprs, joinrel->reltarget->exprs)) + joinrel->uniquekeys = lappend(joinrel->uniquekeys, uniquekey); + } + return; + } + + Assert(jointype == JOIN_LEFT || jointype == JOIN_FULL || jointype == JOIN_INNER); + + /* Fast path */ + if (innerrel->uniquekeys == NIL || outerrel->uniquekeys == NIL) + return; + + inner_onerow = relation_is_onerow(innerrel); + outer_onerow = relation_is_onerow(outerrel); + + outerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(outerrel); + innerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(innerrel); + + clause_list = select_mergejoin_clauses(root, joinrel, outerrel, innerrel, + restrictlist, jointype, + &mergejoin_allowed); + + if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true /* reverse */)) + { + bool outer_impact = jointype == JOIN_FULL; + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + /* Outer relation has one row, and the unique key is not duplicated after join, + * the joinrel will still has one row unless the jointype == JOIN_FULL. + */ + if (outer_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (outer_onerow) + { + /* + * The onerow outerrel becomes multi rows and multi_nullvals + * will be changed to true. We also need to set the exprs correctly since it + * can't be NIL any more. + */ + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, outerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Change multi_nullvals to true due to the full join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, true)); + else + /* Just reuse it */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + } + ctx->added_to_joinrel = true; + } + } + + if (innerrel_keeps_unique(root, outerrel, innerrel, clause_list, false)) + { + bool outer_impact = jointype == JOIN_FULL || jointype == JOIN_LEFT;; + + foreach(lc, innerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + if (inner_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (inner_onerow) + { + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, innerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Need to change multi_nullvals to true due to the outer join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, + true)); + else + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + + } + ctx->added_to_joinrel = true; + } + } + + /* + * The combination of the UniqueKey from both sides is unique as well regardless + * of join type, but no bother to add it if its subset has been added to joinrel + * already or it is not useful for the joinrel. + */ + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx1 = (UniqueKeyContext) lfirst(lc); + if (ctx1->added_to_joinrel || !ctx1->useful) + continue; + foreach(lc2, innerrel_ukey_ctx) + { + UniqueKeyContext ctx2 = (UniqueKeyContext) lfirst(lc2); + if (ctx2->added_to_joinrel || !ctx2->useful) + continue; + if (add_combined_uniquekey(root, joinrel, outerrel, innerrel, + ctx1->uniquekey, ctx2->uniquekey, + jointype)) + /* If we set a onerow UniqueKey to joinrel, we don't need other. */ + return; + } + } +} + + +/* + * convert_subquery_uniquekeys + * + * Covert the UniqueKey in subquery to outer relation. + */ +void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel) +{ + ListCell *lc; + + if (sub_final_rel->uniquekeys == NIL) + return; + + if (relation_is_onerow(sub_final_rel)) + { + add_uniquekey_for_onerow(currel); + return; + } + + Assert(currel->subroot != NULL); + + foreach(lc, sub_final_rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + ListCell *lc; + List *exprs = NIL; + bool ukey_useful = true; + + /* One row case is handled above */ + Assert(ukey->exprs != NIL); + foreach(lc, ukey->exprs) + { + Var *var; + TargetEntry *tle = tlist_member(lfirst(lc), + currel->subroot->processed_tlist); + if (tle == NULL) + { + ukey_useful = false; + break; + } + var = find_var_for_subquery_tle(currel, tle); + if (var == NULL) + { + ukey_useful = false; + break; + } + exprs = lappend(exprs, var); + } + + if (ukey_useful) + currel->uniquekeys = lappend(currel->uniquekeys, + makeUniqueKey(exprs, + ukey->multi_nullvals)); + + } +} + +/* + * innerrel_keeps_unique + * + * Check if Unique key of the innerrel is valid after join. innerrel's UniqueKey + * will be still valid if innerrel's any-column mergeop outrerel's uniquekey + * exists in clause_list. + * + * Note: the clause_list must be a list of mergeable restrictinfo already. + */ +static bool +innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *clause_list, + bool reverse) +{ + ListCell *lc, *lc2, *lc3; + + if (outerrel->uniquekeys == NIL || innerrel->uniquekeys == NIL) + return false; + + /* Check if there is outerrel's uniquekey in mergeable clause. */ + foreach(lc, outerrel->uniquekeys) + { + List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs; + bool clauselist_matchs_all_exprs = true; + foreach(lc2, outer_uq_exprs) + { + Node *outer_uq_expr = lfirst(lc2); + bool find_uq_expr_in_clauselist = false; + foreach(lc3, clause_list) + { + RestrictInfo *rinfo = lfirst_node(RestrictInfo, lc3); + Node *outer_expr; + if (reverse) + outer_expr = rinfo->outer_is_left ? get_rightop(rinfo->clause) : get_leftop(rinfo->clause); + else + outer_expr = rinfo->outer_is_left ? get_leftop(rinfo->clause) : get_rightop(rinfo->clause); + if (equal(outer_expr, outer_uq_expr)) + { + find_uq_expr_in_clauselist = true; + break; + } + } + if (!find_uq_expr_in_clauselist) + { + /* No need to check the next exprs in the current uniquekey */ + clauselist_matchs_all_exprs = false; + break; + } + } + + if (clauselist_matchs_all_exprs) + return true; + } + return false; +} + + +/* + * relation_is_onerow + * Check if it is a one-row relation by checking UniqueKey. + */ +bool +relation_is_onerow(RelOptInfo *rel) +{ + UniqueKey *ukey; + if (rel->uniquekeys == NIL) + return false; + ukey = linitial_node(UniqueKey, rel->uniquekeys); + return ukey->exprs == NIL && list_length(rel->uniquekeys) == 1; +} + +/* + * relation_has_uniquekeys_for + * Returns true if we have proofs that 'rel' cannot return multiple rows with + * the same values in each of 'exprs'. Otherwise returns false. + */ +bool +relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel, + List *exprs, bool allow_multinulls) +{ + ListCell *lc; + + /* + * For UniqueKey->onerow case, the uniquekey->exprs is empty as well + * so we can't rely on list_is_subset to handle this special cases + */ + if (exprs == NIL) + return false; + + foreach(lc, rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (ukey->multi_nullvals && !allow_multinulls) + continue; + if (list_is_subset(ukey->exprs, exprs)) + return true; + } + return false; +} + + +/* + * get_exprs_from_uniqueindex + * + * Return a list of exprs which is unique. set useful to false if this + * unique index is not useful for us. + */ +static List * +get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals) +{ + List *exprs = NIL; + ListCell *indexpr_item; + int c = 0; + + *useful = true; + *multi_nullvals = false; + + indexpr_item = list_head(unique_index->indexprs); + for(c = 0; c < unique_index->ncolumns; c++) + { + int attr = unique_index->indexkeys[c]; + Expr *expr; + bool matched_const = false; + ListCell *lc1, *lc2; + + if(attr > 0) + { + expr = list_nth_node(TargetEntry, unique_index->indextlist, c)->expr; + } + else if (attr == 0) + { + /* Expression index */ + expr = lfirst(indexpr_item); + indexpr_item = lnext(unique_index->indexprs, indexpr_item); + } + else /* attr < 0 */ + { + /* Index on system column is not supported */ + Assert(false); + } + + /* + * Check index_col = Const case with regarding to opfamily checking + * If we can remove the index_col from the final UniqueKey->exprs. + */ + forboth(lc1, const_exprs, lc2, const_expr_opfamilies) + { + if (list_member_oid((List *)lfirst(lc2), unique_index->opfamily[c]) + && match_index_to_operand((Node *) lfirst(lc1), c, unique_index)) + { + matched_const = true; + break; + } + } + + if (matched_const) + continue; + + /* Check if the indexed expr is used in rel */ + if (attr > 0) + { + /* + * Normal Indexed column, if the col is not used, then the index is useless + * for uniquekey. + */ + attr -= FirstLowInvalidHeapAttributeNumber; + + if (!bms_is_member(attr, used_varattrs)) + { + *useful = false; + break; + } + } + else if (!list_member(unique_index->rel->reltarget->exprs, expr)) + { + /* Expression index but the expression is not used in rel */ + *useful = false; + break; + } + + /* check not null property. */ + if (attr == 0) + { + /* We never know if a expression yields null or not */ + *multi_nullvals = true; + } + else if (!bms_is_member(attr, unique_index->rel->notnullattrs) + && !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber, + unique_index->rel->notnullattrs)) + { + *multi_nullvals = true; + } + + exprs = lappend(exprs, expr); + } + return exprs; +} + + +/* + * add_uniquekey_for_onerow + * If we are sure that the relation only returns one row, then all the columns + * are unique. However we don't need to create UniqueKey for every column, we + * just set exprs = NIL and overwrites all the other UniqueKey on this RelOptInfo + * since this one has strongest semantics. + */ +void +add_uniquekey_for_onerow(RelOptInfo *rel) +{ + /* + * We overwrite the previous UniqueKey on purpose since this one has the + * strongest semantic. + */ + rel->uniquekeys = list_make1(makeUniqueKey(NIL, false)); +} + + +/* + * initililze_uniquecontext_for_joinrel + * Return a List of UniqueKeyContext for an inputrel + */ +static List * +initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel) +{ + List *res = NIL; + ListCell *lc; + foreach(lc, inputrel->uniquekeys) + { + UniqueKeyContext context; + context = palloc(sizeof(struct UniqueKeyContextData)); + context->uniquekey = lfirst_node(UniqueKey, lc); + context->added_to_joinrel = false; + context->useful = true; + res = lappend(res, context); + } + return res; +} + + +/* + * get_exprs_from_uniquekey + * Unify the way of get List of exprs from a one-row UniqueKey or + * normal UniqueKey. for the onerow case, every expr in rel1 is a valid + * UniqueKey. Return a List of exprs. + * + * rel1: The relation which you want to get the exprs. + * ukey: The UniqueKey you want to get the exprs. + */ +static List * +get_exprs_from_uniquekey(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *rel1, UniqueKey *ukey) +{ + ListCell *lc; + bool onerow = rel1 != NULL && relation_is_onerow(rel1); + + List *res = NIL; + Assert(onerow || ukey); + if (onerow) + { + /* Only cares about the exprs still exist in joinrel */ + foreach(lc, joinrel->reltarget->exprs) + { + Bitmapset *relids = pull_varnos(root, lfirst(lc)); + if (bms_is_subset(relids, rel1->relids)) + { + res = lappend(res, list_make1(lfirst(lc))); + } + } + } + else + { + res = list_make1(ukey->exprs); + } + return res; +} + +/* + * Partitioned table Unique Keys. + * The partition table unique key is maintained as: + * 1. The index must be unique as usual. + * 2. The index must contains partition key. + * 3. The index must exist on all the child rel. see simple_indexinfo_equal for + * how we compare it. + */ + +/* + * index_constains_partkey + * return true if the index contains the partiton key. + */ +static bool +index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind) +{ + ListCell *lc; + int i; + Assert(IS_PARTITIONED_REL(partrel)); + Assert(partrel->part_scheme->partnatts > 0); + + for(i = 0; i < partrel->part_scheme->partnatts; i++) + { + Node *part_expr = linitial(partrel->partexprs[i]); + bool found_in_index = false; + foreach(lc, ind->indextlist) + { + Expr *index_expr = lfirst_node(TargetEntry, lc)->expr; + if (equal(index_expr, part_expr)) + { + found_in_index = true; + break; + } + } + if (!found_in_index) + return false; + } + return true; +} + +/* + * simple_indexinfo_equal + * + * Used to check if the 2 index is same as each other. The index here + * is COPIED from childrel and did some tiny changes(see + * simple_copy_indexinfo_to_parent) + */ +static bool +simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2) +{ + Size oid_cmp_len = sizeof(Oid) * ind1->ncolumns; + + return ind1->ncolumns == ind2->ncolumns && + ind1->unique == ind2->unique && + memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0 && + memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0 && + memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0 && + memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0 && + equal(get_tlist_exprs(ind1->indextlist, true), + get_tlist_exprs(ind2->indextlist, true)); +} + + +/* + * The below macros are used for simple_copy_indexinfo_to_parent which is so + * customized that I don't want to put it to copyfuncs.c. So copy it here. + */ +#define COPY_POINTER_FIELD(fldname, sz) \ + do { \ + Size _size = (sz); \ + newnode->fldname = palloc(_size); \ + memcpy(newnode->fldname, from->fldname, _size); \ + } while (0) + +#define COPY_NODE_FIELD(fldname) \ + (newnode->fldname = copyObjectImpl(from->fldname)) + +#define COPY_SCALAR_FIELD(fldname) \ + (newnode->fldname = from->fldname) + + +/* + * simple_copy_indexinfo_to_parent (from partition) + * Copy the IndexInfo from child relation to parent relation with some modification, + * which is used to test: + * 1. If the same index exists in all the childrels. + * 2. If the parentrel->reltarget/basicrestrict info matches this index. + */ +static IndexOptInfo * +simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from) +{ + IndexOptInfo *newnode = makeNode(IndexOptInfo); + AppendRelInfo *appinfo = find_appinfo_by_child(root, from->rel->relid); + ListCell *lc; + int idx = 0; + + COPY_SCALAR_FIELD(ncolumns); + COPY_SCALAR_FIELD(nkeycolumns); + COPY_SCALAR_FIELD(unique); + COPY_SCALAR_FIELD(immediate); + /* We just need to know if it is NIL or not */ + COPY_SCALAR_FIELD(indpred); + COPY_SCALAR_FIELD(predOK); + COPY_POINTER_FIELD(indexkeys, from->ncolumns * sizeof(int)); + COPY_POINTER_FIELD(indexcollations, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opfamily, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opcintype, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(sortopfamily, from->ncolumns * sizeof(Oid)); + COPY_NODE_FIELD(indextlist); + + /* Convert index exprs on child expr to expr on parent */ + foreach(lc, newnode->indextlist) + { + TargetEntry *tle = lfirst_node(TargetEntry, lc); + /* Index on expression is ignored */ + Assert(IsA(tle->expr, Var)); + tle->expr = (Expr *) find_parent_var(appinfo, (Var *) tle->expr); + newnode->indexkeys[idx] = castNode(Var, tle->expr)->varattno; + idx++; + } + newnode->rel = parentrel; + return newnode; +} + +/* + * adjust_partition_unique_indexlist + * + * global_unique_indexes: At the beginning, it contains the copy & modified + * unique index from the first partition. And then check if each index in it still + * exists in the following partitions. If no, remove it. at last, it has an + * index list which exists in all the partitions. + */ +static void +adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_indexes) +{ + ListCell *lc, *lc2; + foreach(lc, *global_unique_indexes) + { + IndexOptInfo *g_ind = lfirst_node(IndexOptInfo, lc); + bool found_in_child = false; + + foreach(lc2, childrel->indexlist) + { + IndexOptInfo *p_ind = lfirst_node(IndexOptInfo, lc2); + IndexOptInfo *p_ind_copy; + if (!p_ind->unique || !p_ind->immediate || + (p_ind->indpred != NIL && !p_ind->predOK)) + continue; + p_ind_copy = simple_copy_indexinfo_to_parent(root, parentrel, p_ind); + if (simple_indexinfo_equal(p_ind_copy, g_ind)) + { + found_in_child = true; + break; + } + } + if (!found_in_child) + /* The index doesn't exist in childrel, remove it from global_unique_indexes */ + *global_unique_indexes = foreach_delete_current(*global_unique_indexes, lc); + } +} + +/* Helper function for groupres/distinctrel */ +static void +add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgroups) +{ + Query *parse = root->parse; + List *exprs; + + /* + * XXX: If there are some vars which is not in current levelsup, the semantic is + * imprecise, should we avoid it or not? levelsup = 1 is just a demo, maybe we need to + * check every level other than 0, if so, looks we have to write another + * pull_var_walker. + */ + List *upper_vars = pull_vars_of_level((Node*)sortgroups, 1); + + if (upper_vars != NIL) + return; + + exprs = get_sortgrouplist_exprs(sortgroups, parse->targetList); + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(exprs, + false /* sortgroupclause can't be multi_nullvals */)); +} + + +/* + * add_combined_uniquekey + * The combination of both UniqueKeys is a valid UniqueKey for joinrel no matter + * the jointype. + */ +bool +add_combined_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype) +{ + + ListCell *lc1, *lc2; + + /* Either side has multi_nullvals or we have outer join, + * the combined UniqueKey has multi_nullvals */ + bool multi_nullvals = outer_ukey->multi_nullvals || + inner_ukey->multi_nullvals || IS_OUTER_JOIN(jointype); + + /* The only case we can get onerow joinrel after join */ + if (relation_is_onerow(outer_rel) + && relation_is_onerow(inner_rel) + && jointype == JOIN_INNER) + { + add_uniquekey_for_onerow(joinrel); + return true; + } + + foreach(lc1, get_exprs_from_uniquekey(root, joinrel, outer_rel, outer_ukey)) + { + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, inner_rel, inner_ukey)) + { + List *exprs = list_concat_copy(lfirst_node(List, lc1), lfirst_node(List, lc2)); + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(exprs, + multi_nullvals)); + } + } + return false; +} diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index 424d25cbd5..8d8e493f5c 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -2387,6 +2387,8 @@ grouping_planner(PlannerInfo *root, bool inheritance_update, add_path(final_rel, path); } + simple_copy_uniquekeys(current_rel, final_rel); + /* * Generate partial paths for final_rel, too, if outer query levels might * be able to make use of them. @@ -3899,6 +3901,8 @@ create_grouping_paths(PlannerInfo *root, } set_cheapest(grouped_rel); + + populate_grouprel_uniquekeys(root, grouped_rel, input_rel); return grouped_rel; } @@ -4621,7 +4625,7 @@ create_window_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(window_rel); - + simple_copy_uniquekeys(input_rel, window_rel); return window_rel; } @@ -4935,7 +4939,7 @@ create_distinct_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(distinct_rel); - + populate_distinctrel_uniquekeys(root, input_rel, distinct_rel); return distinct_rel; } @@ -5196,6 +5200,8 @@ create_ordered_paths(PlannerInfo *root, */ Assert(ordered_rel->pathlist != NIL); + simple_copy_uniquekeys(input_rel, ordered_rel); + return ordered_rel; } @@ -6073,6 +6079,9 @@ adjust_paths_for_srfs(PlannerInfo *root, RelOptInfo *rel, if (list_length(targets) == 1) return; + /* UniqueKey is not valid after handling the SRF. */ + rel->uniquekeys = NIL; + /* * Stack SRF-evaluation nodes atop each path for the rel. * diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c index becdcbb872..b7626545bf 100644 --- a/src/backend/optimizer/prep/prepunion.c +++ b/src/backend/optimizer/prep/prepunion.c @@ -689,6 +689,8 @@ generate_union_paths(SetOperationStmt *op, PlannerInfo *root, /* Undo effects of possibly forcing tuple_fraction to 0 */ root->tuple_fraction = save_fraction; + /* Add the UniqueKeys */ + populate_unionrel_uniquekeys(root, result_rel); return result_rel; } diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c index 86922a273c..6817c9c787 100644 --- a/src/backend/optimizer/util/appendinfo.c +++ b/src/backend/optimizer/util/appendinfo.c @@ -746,3 +746,47 @@ find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos) } return appinfos; } + +/* + * find_appinfo_by_child + * + */ +AppendRelInfo * +find_appinfo_by_child(PlannerInfo *root, Index child_index) +{ + ListCell *lc; + foreach(lc, root->append_rel_list) + { + AppendRelInfo *appinfo = lfirst_node(AppendRelInfo, lc); + if (appinfo->child_relid == child_index) + return appinfo; + } + elog(ERROR, "parent relation cant be found"); + return NULL; +} + +/* + * find_parent_var + * + */ +Var * +find_parent_var(AppendRelInfo *appinfo, Var *child_var) +{ + ListCell *lc; + Var *res = NULL; + Index attno = 1; + foreach(lc, appinfo->translated_vars) + { + Node *child_node = lfirst(lc); + if (equal(child_node, child_var)) + { + res = copyObject(child_var); + res->varattno = attno; + res->varno = appinfo->parent_relid; + } + attno++; + } + if (res == NULL) + elog(ERROR, "parent var can't be found."); + return res; +} diff --git a/src/backend/optimizer/util/inherit.c b/src/backend/optimizer/util/inherit.c index be1c9ddd96..3eec1f4d74 100644 --- a/src/backend/optimizer/util/inherit.c +++ b/src/backend/optimizer/util/inherit.c @@ -736,6 +736,7 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, { Node *onecq = (Node *) lfirst(lc2); bool pseudoconstant; + RestrictInfo *child_rinfo; /* check for pseudoconstant (no Vars or volatile functions) */ pseudoconstant = @@ -747,14 +748,15 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, root->hasPseudoConstantQuals = true; } /* reconstitute RestrictInfo with appropriate properties */ - childquals = lappend(childquals, - make_restrictinfo(root, - (Expr *) onecq, - rinfo->is_pushed_down, - rinfo->outerjoin_delayed, - pseudoconstant, - rinfo->security_level, - NULL, NULL, NULL)); + child_rinfo = make_restrictinfo(root, + (Expr *) onecq, + rinfo->is_pushed_down, + rinfo->outerjoin_delayed, + pseudoconstant, + rinfo->security_level, + NULL, NULL, NULL); + child_rinfo->mergeopfamilies = rinfo->mergeopfamilies; + childquals = lappend(childquals, child_rinfo); /* track minimum security level among child quals */ cq_min_security = Min(cq_min_security, rinfo->security_level); } diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 48a7ebfe45..4fe1824eb0 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -16,6 +16,7 @@ #include "nodes/execnodes.h" #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" extern A_Expr *makeA_Expr(A_Expr_Kind kind, List *name, @@ -106,4 +107,6 @@ extern GroupingSet *makeGroupingSet(GroupingSetKind kind, List *content, int loc extern VacuumRelation *makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols); +extern UniqueKey* makeUniqueKey(List *exprs, bool multi_nullvals); + #endif /* MAKEFUNC_H */ diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index e22df890ef..3320273ac1 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -265,6 +265,7 @@ typedef enum NodeTag T_EquivalenceMember, T_PathKey, T_PathTarget, + T_UniqueKey, T_RestrictInfo, T_IndexClause, T_PlaceHolderVar, diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index 0d61f04d27..80561b1003 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -739,6 +739,7 @@ typedef struct RelOptInfo QualCost baserestrictcost; /* cost of evaluating the above */ Index baserestrict_min_security; /* min security_level found in * baserestrictinfo */ + List *uniquekeys; /* List of UniqueKey */ List *joininfo; /* RestrictInfo structures for join clauses * involving this rel */ bool has_eclass_joins; /* T means joininfo is incomplete */ @@ -1059,6 +1060,28 @@ typedef struct PathKey } PathKey; +/* + * UniqueKey + * + * Represents the unique properties held by a RelOptInfo. + * + * exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL + * is a special case of UniqueKey, which means there is only 1 row in that + * relation. + * multi_nullvals: true means multi null values may exist in these exprs, so the + * uniqueness is not guaranteed in this case. This field is necessary for + * remove_useless_join & reduce_unique_semijoins where we don't mind these + * duplicated NULL values. It is set to true for 2 cases. One is a unique key + * from a unique index but the related column is nullable. The other one is for + * outer join. see populate_joinrel_uniquekeys for detail. + */ +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + /* * PathTarget * @@ -2496,7 +2519,7 @@ typedef enum * * flags indicating what kinds of grouping are possible. * partial_costs_set is true if the agg_partial_costs and agg_final_costs - * have been initialized. + * have been initialized. * agg_partial_costs gives partial aggregation costs. * agg_final_costs gives finalization costs. * target_parallel_safe is true if target is parallel safe. @@ -2526,8 +2549,8 @@ typedef struct * limit_tuples is an estimated bound on the number of output tuples, * or -1 if no LIMIT or couldn't estimate. * count_est and offset_est are the estimated values of the LIMIT and OFFSET - * expressions computed by preprocess_limit() (see comments for - * preprocess_limit() for more information). + * expressions computed by preprocess_limit() (see comments for + * preprocess_limit() for more information). */ typedef struct { diff --git a/src/include/nodes/pg_list.h b/src/include/nodes/pg_list.h index 404e03f132..85c65e910c 100644 --- a/src/include/nodes/pg_list.h +++ b/src/include/nodes/pg_list.h @@ -566,6 +566,16 @@ extern pg_nodiscard List *list_delete_first(List *list); extern pg_nodiscard List *list_delete_last(List *list); extern pg_nodiscard List *list_delete_nth_cell(List *list, int n); extern pg_nodiscard List *list_delete_cell(List *list, ListCell *cell); +extern bool list_is_subset(const List *members, const List *target); + +extern List *list_delete(List *list, void *datum); +extern List *list_delete_ptr(List *list, void *datum); +extern List *list_delete_int(List *list, int datum); +extern List *list_delete_oid(List *list, Oid datum); +extern List *list_delete_first(List *list); +extern List *list_delete_last(List *list); +extern List *list_delete_nth_cell(List *list, int n); +extern List *list_delete_cell(List *list, ListCell *cell); extern List *list_union(const List *list1, const List *list2); extern List *list_union_ptr(const List *list1, const List *list2); diff --git a/src/include/optimizer/appendinfo.h b/src/include/optimizer/appendinfo.h index 4cbf8c26cc..43b8d05f69 100644 --- a/src/include/optimizer/appendinfo.h +++ b/src/include/optimizer/appendinfo.h @@ -32,4 +32,7 @@ extern Relids adjust_child_relids_multilevel(PlannerInfo *root, Relids relids, extern AppendRelInfo **find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos); +extern AppendRelInfo *find_appinfo_by_child(PlannerInfo *root, Index child_index); +extern Var *find_parent_var(AppendRelInfo *appinfo, Var *child_var); + #endif /* APPENDINFO_H */ diff --git a/src/include/optimizer/optimizer.h b/src/include/optimizer/optimizer.h index d587952b7d..843aafa51e 100644 --- a/src/include/optimizer/optimizer.h +++ b/src/include/optimizer/optimizer.h @@ -23,6 +23,7 @@ #define OPTIMIZER_H #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" /* Test if an expression node represents a SRF call. Beware multiple eval! */ #define IS_SRF_CALL(node) \ @@ -168,6 +169,7 @@ extern TargetEntry *get_sortgroupref_tle(Index sortref, List *targetList); extern TargetEntry *get_sortgroupclause_tle(SortGroupClause *sgClause, List *targetList); +extern Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); extern Node *get_sortgroupclause_expr(SortGroupClause *sgClause, List *targetList); extern List *get_sortgrouplist_exprs(List *sgClauses, diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h index 035d3e1206..1adf99c9ee 100644 --- a/src/include/optimizer/paths.h +++ b/src/include/optimizer/paths.h @@ -247,5 +247,48 @@ extern PathKey *make_canonical_pathkey(PlannerInfo *root, int strategy, bool nulls_first); extern void add_paths_to_append_rel(PlannerInfo *root, RelOptInfo *rel, List *live_childrels); +extern List *select_mergejoin_clauses(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + JoinType jointype, + bool *mergejoin_allowed); + +/* + * uniquekeys.c + * Utilities for matching and building unique keys + */ +extern void populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List* unique_index_list); +extern void populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels); +extern void populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel); +extern void populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel); +extern void populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel); +extern void simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel); +extern void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel); +extern void populate_joinrel_uniquekeys(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + RelOptInfo *rel2, + List *restrictlist, + JoinType jointype); + +extern bool relation_has_uniquekeys_for(PlannerInfo *root, + RelOptInfo *rel, + List *exprs, + bool allow_multinulls); +extern bool relation_is_onerow(RelOptInfo *rel); #endif /* PATHS_H */ -- 2.26.2 --yh7cw345mbttadjz Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v38-0003-Extend-UniqueKeys.patch" ^ permalink raw reply [nested|flat] 91+ messages in thread
* [PATCH 03/10] Introduce UniqueKey attributes on RelOptInfo struct. @ 2020-05-11 07:50 一挃 <[email protected]> 0 siblings, 0 replies; 91+ messages in thread From: 一挃 @ 2020-05-11 07:50 UTC (permalink / raw) UniqueKey is a set of exprs on RelOptInfo which represents the exprs will be unique on the given RelOptInfo. You can see README.uniquekey for more information. --- src/backend/nodes/copyfuncs.c | 13 + src/backend/nodes/list.c | 31 + src/backend/nodes/makefuncs.c | 13 + src/backend/nodes/outfuncs.c | 11 + src/backend/nodes/readfuncs.c | 10 + src/backend/optimizer/path/Makefile | 3 +- src/backend/optimizer/path/README.uniquekey | 131 +++ src/backend/optimizer/path/allpaths.c | 10 + src/backend/optimizer/path/joinpath.c | 9 +- src/backend/optimizer/path/joinrels.c | 2 + src/backend/optimizer/path/pathkeys.c | 3 +- src/backend/optimizer/path/uniquekeys.c | 1135 +++++++++++++++++++ src/backend/optimizer/plan/planner.c | 13 +- src/backend/optimizer/prep/prepunion.c | 2 + src/backend/optimizer/util/appendinfo.c | 44 + src/backend/optimizer/util/inherit.c | 18 +- src/include/nodes/makefuncs.h | 3 + src/include/nodes/nodes.h | 1 + src/include/nodes/pathnodes.h | 29 +- src/include/nodes/pg_list.h | 10 + src/include/optimizer/appendinfo.h | 3 + src/include/optimizer/optimizer.h | 2 + src/include/optimizer/paths.h | 43 + 23 files changed, 1515 insertions(+), 24 deletions(-) create mode 100644 src/backend/optimizer/path/README.uniquekey create mode 100644 src/backend/optimizer/path/uniquekeys.c diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..75c1c5e824 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2296,6 +2296,16 @@ _copyPathKey(const PathKey *from) return newnode; } +static UniqueKey * +_copyUniqueKey(const UniqueKey *from) +{ + UniqueKey *newnode = makeNode(UniqueKey); + + COPY_NODE_FIELD(exprs); + COPY_SCALAR_FIELD(multi_nullvals); + + return newnode; +} /* * _copyRestrictInfo */ @@ -5220,6 +5230,9 @@ copyObjectImpl(const void *from) case T_PathKey: retval = _copyPathKey(from); break; + case T_UniqueKey: + retval = _copyUniqueKey(from); + break; case T_RestrictInfo: retval = _copyRestrictInfo(from); break; diff --git a/src/backend/nodes/list.c b/src/backend/nodes/list.c index dbf6b30233..ca099495a1 100644 --- a/src/backend/nodes/list.c +++ b/src/backend/nodes/list.c @@ -702,6 +702,37 @@ list_member_oid(const List *list, Oid datum) return false; } +/* + * return true iff every entry in "members" list is also present + * in the "target" list. + */ +bool +list_is_subset(const List *members, const List *target) +{ + const ListCell *lc1, *lc2; + + Assert(IsPointerList(members)); + Assert(IsPointerList(target)); + check_list_invariants(members); + check_list_invariants(target); + + foreach(lc1, members) + { + bool found = false; + foreach(lc2, target) + { + if (equal(lfirst(lc1), lfirst(lc2))) + { + found = true; + break; + } + } + if (!found) + return false; + } + return true; +} + /* * Delete the n'th cell (counting from 0) in list. * diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 01c110cd2f..40415d0f5b 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -815,3 +815,16 @@ makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols) v->va_cols = va_cols; return v; } + + +/* + * makeUniqueKey + */ +UniqueKey* +makeUniqueKey(List *exprs, bool multi_nullvals) +{ + UniqueKey * ukey = makeNode(UniqueKey); + ukey->exprs = exprs; + ukey->multi_nullvals = multi_nullvals; + return ukey; +} diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..44154cde6a 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2456,6 +2456,14 @@ _outPathKey(StringInfo str, const PathKey *node) WRITE_BOOL_FIELD(pk_nulls_first); } +static void +_outUniqueKey(StringInfo str, const UniqueKey *node) +{ + WRITE_NODE_TYPE("UNIQUEKEY"); + WRITE_NODE_FIELD(exprs); + WRITE_BOOL_FIELD(multi_nullvals); +} + static void _outPathTarget(StringInfo str, const PathTarget *node) { @@ -4198,6 +4206,9 @@ outNode(StringInfo str, const void *obj) case T_PathKey: _outPathKey(str, obj); break; + case T_UniqueKey: + _outUniqueKey(str, obj); + break; case T_PathTarget: _outPathTarget(str, obj); break; diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index c5e136e9c3..b3e212bf1c 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -492,6 +492,14 @@ _readSetOperationStmt(void) READ_DONE(); } +static UniqueKey * +_readUniqueKey(void) +{ + READ_LOCALS(UniqueKey); + READ_NODE_FIELD(exprs); + READ_BOOL_FIELD(multi_nullvals); + READ_DONE(); +} /* * Stuff from primnodes.h. @@ -2717,6 +2725,8 @@ parseNodeString(void) return_value = _readCommonTableExpr(); else if (MATCH("SETOPERATIONSTMT", 16)) return_value = _readSetOperationStmt(); + else if (MATCH("UNIQUEKEY", 9)) + return_value = _readUniqueKey(); else if (MATCH("ALIAS", 5)) return_value = _readAlias(); else if (MATCH("RANGEVAR", 8)) diff --git a/src/backend/optimizer/path/Makefile b/src/backend/optimizer/path/Makefile index 1e199ff66f..7b9820c25f 100644 --- a/src/backend/optimizer/path/Makefile +++ b/src/backend/optimizer/path/Makefile @@ -21,6 +21,7 @@ OBJS = \ joinpath.o \ joinrels.o \ pathkeys.o \ - tidpath.o + tidpath.o \ + uniquekeys.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/optimizer/path/README.uniquekey b/src/backend/optimizer/path/README.uniquekey new file mode 100644 index 0000000000..5eac761995 --- /dev/null +++ b/src/backend/optimizer/path/README.uniquekey @@ -0,0 +1,131 @@ +1. What is UniqueKey? +We can think UniqueKey is a set of exprs for a RelOptInfo, which we are insure +that doesn't yields same result among all the rows. The simplest UniqueKey +format is primary key. + +However we define the UnqiueKey as below. + +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + +exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL +is a special case of UniqueKey, which means there is only one row in that +relation.it has a stronger semantic than others. like SELECT uk FROM t; uk is +normal unique key and may have different values. SELECT colx FROM t WHERE uk = +const. colx is unique AND we have only 1 value. This field can used for +innerrel_is_unique. this logic is handled specially in add_uniquekey_for_onerow +function. + +multi_nullvals: true means multi null values may exist in these exprs, so the +uniqueness is not guaranteed in this case. This field is necessary for +remove_useless_join & reduce_unique_semijoins where we don't mind these +duplicated NULL values. It is set to true for 2 cases. One is a unique key +from a unique index but the related column is nullable. The other one is for +outer join. see populate_joinrel_uniquekeys for detail. + + +The UniqueKey can be used at the following cases at least: +1. remove_useless_joins. +2. reduce_semianti_joins +3. remove distinct node if distinct clause is unique. +4. remove aggnode if group by clause is unique. +5. Index Skip Scan (WIP) +6. Aggregation Push Down without 2 phase aggregation if the join can't + duplicated the aggregated rows. (work in progress feature) + +2. How is it maintained? + +We have a set of populate_xxx_unqiuekeys functions to maintain the uniquekey on +various cases. xxx includes baserel, joinrel, partitionedrel, distinctrel, +groupedrel, unionrel. and we also need to convert the uniquekey from subquery +to outer relation, which is what convert_subquery_uniquekeys does. + +1. The first part is about baserel. We handled 3 cases. suppose we have Unique +Index on (a, b). + +1. SELECT a, b FROM t. UniqueKey (a, b) +2. SELECT a FROM t WHERE b = 1; UniqueKey (a) +3. SELECT .. FROM t WHERE a = 1 AND b = 1; UniqueKey (NIL). onerow case, every + column is Unique. + +2. The next part is joinrel, this part is most error-prone, we simplified the rules +like below: +1. If the relation's UniqueKey can't be duplicated after join, then is will be + still valid for the join rel. The function we used here is + innerrel_keeps_unique. The basic idea is innerrel.any_col = outer.uk. + +2. If the UnqiueKey can't keep valid via the rule 1, the combination of the + UniqueKey from both sides are valid for sure. We can prove this as: if the + unique exprs from rel1 is duplicated by rel2, the duplicated rows must + contains different unique exprs from rel2. + +More considerations about onerow: +1. If relation with one row and it can't be duplicated, it is still possible + contains mulit_nullvas after outer join. +2. If the either UniqueKey can be duplicated after join, the can get one row + only when both side is one row AND there is no outer join. +3. Whenever the onerow UniqueKey is not a valid any more, we need to convert one + row UniqueKey to normal unique key since we don't store exprs for one-row + relation. get_exprs_from_uniquekeys will be used here. + + +More considerations about multi_nullvals after join: +1. If the original UnqiueKey has multi_nullvals, the final UniqueKey will have + mulit_nullvals in any case. +2. If a unique key doesn't allow mulit_nullvals, after some outer join, it + allows some outer join. + + +3. When we comes to subquery, we need to convert_subquery_unqiuekeys just like +convert_subquery_pathkeys. Only the UniqueKey insides subquery is referenced as +a Var in outer relation will be reused. The relationship between the outerrel.Var +and subquery.exprs is built with outerel->subroot->processed_tlist. + + +4. As for the SRF functions, it will break the uniqueness of uniquekey, However it +is handled in adjust_paths_for_srfs, which happens after the query_planner. so +we will maintain the UniqueKey until there and reset it to NIL at that +places. This can't help on distinct/group by elimination cases but probably help +in some other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is +semantic correctly. + + +5. As for inherit table, we first main the UnqiueKey on childrel as well. But for +partitioned table we need to maintain 2 different kinds of +UnqiueKey. 1). UniqueKey on the parent relation 2). UniqueKey on child +relation for partition wise query. + +Example: +CREATE TABLE p (a int not null, b int not null) partition by list (a); +CREATE TABLE p0 partition of p for values in (1); +CREATE TABLE p1 partition of p for values in (2); + +create unique index p0_b on p0(b); +create unique index p1_b on p1(b); + +Now b is only unique on partition level, so the distinct can't be removed on +the following cases. SELECT DISTINCT b FROM p; + +Another example is SELECT DISTINCT a, b FROM p WHERE a = 1; Since only one +partition is chosen, the UniqueKey on child relation is same as the UniqueKey on +parent relation. + +Another usage of UniqueKey on partition level is it be helpful for +partition-wise join. + +As for the UniqueKey on parent table level, it comes with 2 different ways, +1). the UniqueKey is also derived in UniqueKey index, but the index must be same +in all the related children relations and the unique index must contains +Partition Key in it. Example: + +CREATE UNIQUE INDEX p_ab ON p(a, b); -- where a is the partition key. + +-- Query +SELECT a, b FROM p; the (a, b) is a UniqueKey of p. + +2). If the parent relation has only one childrel, the UniqueKey on childrel is + the UniqueKey on parent as well. diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index fc1a3a68a2..66bf6f19f7 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -579,6 +579,12 @@ set_plain_rel_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) */ check_index_predicates(root, rel); + /* + * Now that we've marked which partial indexes are suitable, we can now + * build the relation's unique keys. + */ + populate_baserel_uniquekeys(root, rel, rel->indexlist); + /* Mark rel with estimated output rows, width, etc */ set_baserel_size_estimates(root, rel); } @@ -1299,6 +1305,8 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, /* Add paths to the append relation. */ add_paths_to_append_rel(root, rel, live_childrels); + if (IS_PARTITIONED_REL(rel)) + populate_partitionedrel_uniquekeys(root, rel, live_childrels); } @@ -2306,6 +2314,8 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, pathkeys, required_outer)); } + convert_subquery_uniquekeys(root, rel, sub_final_rel); + /* If outer rel allows parallelism, do same for partial paths. */ if (rel->consider_parallel && bms_is_empty(required_outer)) { diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c index 57ce97fd53..697ac047fb 100644 --- a/src/backend/optimizer/path/joinpath.c +++ b/src/backend/optimizer/path/joinpath.c @@ -71,13 +71,6 @@ static void consider_parallel_mergejoin(PlannerInfo *root, static void hash_inner_and_outer(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, RelOptInfo *innerrel, JoinType jointype, JoinPathExtraData *extra); -static List *select_mergejoin_clauses(PlannerInfo *root, - RelOptInfo *joinrel, - RelOptInfo *outerrel, - RelOptInfo *innerrel, - List *restrictlist, - JoinType jointype, - bool *mergejoin_allowed); static void generate_mergejoin_paths(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *innerrel, @@ -1927,7 +1920,7 @@ hash_inner_and_outer(PlannerInfo *root, * if it is mergejoinable and involves vars from the two sub-relations * currently of interest. */ -static List * +List * select_mergejoin_clauses(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c index 0dbe2ac726..7271f044ec 100644 --- a/src/backend/optimizer/path/joinrels.c +++ b/src/backend/optimizer/path/joinrels.c @@ -924,6 +924,8 @@ populate_joinrel_with_paths(PlannerInfo *root, RelOptInfo *rel1, /* Apply partitionwise join technique, if possible. */ try_partitionwise_join(root, rel1, rel2, joinrel, sjinfo, restrictlist); + + populate_joinrel_uniquekeys(root, joinrel, rel1, rel2, restrictlist, sjinfo->jointype); } diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index bd9a176d7d..139278829b 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -33,7 +33,6 @@ static bool pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys); static bool matches_boolean_partition_clause(RestrictInfo *rinfo, RelOptInfo *partrel, int partkeycol); -static Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); static bool right_merge_direction(PlannerInfo *root, PathKey *pathkey); @@ -1035,7 +1034,7 @@ convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel, * We need this to ensure that we don't return pathkeys describing values * that are unavailable above the level of the subquery scan. */ -static Var * +Var * find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle) { ListCell *lc; diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c new file mode 100644 index 0000000000..77ed2b2eff --- /dev/null +++ b/src/backend/optimizer/path/uniquekeys.c @@ -0,0 +1,1135 @@ +/*------------------------------------------------------------------------- + * + * uniquekeys.c + * Utilities for matching and building unique keys + * + * Portions Copyright (c) 2020, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/optimizer/path/uniquekeys.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "nodes/makefuncs.h" +#include "nodes/nodeFuncs.h" +#include "optimizer/pathnode.h" +#include "optimizer/paths.h" +#include "optimizer/appendinfo.h" +#include "optimizer/optimizer.h" +#include "optimizer/tlist.h" +#include "rewrite/rewriteManip.h" + + +/* + * This struct is used to help populate_joinrel_uniquekeys. + * + * added_to_joinrel is true if a uniquekey (from outerrel or innerrel) + * has been added to joinrel. + * useful is true if the exprs of the uniquekey still appears in joinrel. + */ +typedef struct UniqueKeyContextData +{ + UniqueKey *uniquekey; + bool added_to_joinrel; + bool useful; +} *UniqueKeyContext; + +static List *initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel); +static bool innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + bool reverse); + +static List *get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals); +static List *get_exprs_from_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + UniqueKey *ukey); +static void add_uniquekey_for_onerow(RelOptInfo *rel); +static bool add_combined_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype); + +/* Used for unique indexes checking for partitioned table */ +static bool index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind); +static IndexOptInfo *simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from); +static bool simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2); +static void adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_index); + +/* Helper function for grouped relation and distinct relation. */ +static void add_uniquekey_from_sortgroups(PlannerInfo *root, + RelOptInfo *rel, + List *sortgroups); + +/* + * populate_baserel_uniquekeys + * Populate 'baserel' uniquekeys list by looking at the rel's unique index + * and baserestrictinfo + */ +void +populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List *indexlist) +{ + ListCell *lc; + List *matched_uniq_indexes = NIL; + + /* Attrs appears in rel->reltarget->exprs. */ + Bitmapset *used_attrs = NULL; + + List *const_exprs = NIL; + List *expr_opfamilies = NIL; + + Assert(baserel->rtekind == RTE_RELATION); + + foreach(lc, indexlist) + { + IndexOptInfo *ind = (IndexOptInfo *) lfirst(lc); + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + matched_uniq_indexes = lappend(matched_uniq_indexes, ind); + } + + if (matched_uniq_indexes == NIL) + return; + + /* Check which attrs is used in baserel->reltarget */ + pull_varattnos((Node *)baserel->reltarget->exprs, baserel->relid, &used_attrs); + + /* Check which attrno is used at a mergeable const filter */ + foreach(lc, baserel->baserestrictinfo) + { + RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); + + if (rinfo->mergeopfamilies == NIL) + continue; + + if (bms_is_empty(rinfo->left_relids)) + { + const_exprs = lappend(const_exprs, get_rightop(rinfo->clause)); + } + else if (bms_is_empty(rinfo->right_relids)) + { + const_exprs = lappend(const_exprs, get_leftop(rinfo->clause)); + } + else + continue; + + expr_opfamilies = lappend(expr_opfamilies, rinfo->mergeopfamilies); + } + + foreach(lc, matched_uniq_indexes) + { + bool multi_nullvals, useful; + List *exprs = get_exprs_from_uniqueindex(lfirst_node(IndexOptInfo, lc), + const_exprs, + expr_opfamilies, + used_attrs, + &useful, + &multi_nullvals); + if (useful) + { + if (exprs == NIL) + { + /* All the columns in Unique Index matched with a restrictinfo */ + add_uniquekey_for_onerow(baserel); + return; + } + baserel->uniquekeys = lappend(baserel->uniquekeys, + makeUniqueKey(exprs, multi_nullvals)); + } + } +} + + +/* + * populate_partitionedrel_uniquekeys + * The UniqueKey on partitionrel comes from 2 cases: + * 1). Only one partition is involved in this query, the unique key can be + * copied to parent rel from childrel. + * 2). There are some unique index which includes partition key and exists + * in all the related partitions. + * We never mind rule 2 if we hit rule 1. + */ + +void +populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels) +{ + ListCell *lc; + List *global_uniq_indexlist = NIL; + RelOptInfo *childrel; + bool is_first = true; + + Assert(IS_PARTITIONED_REL(rel)); + + if (childrels == NIL) + return; + + /* + * If there is only one partition used in this query, the UniqueKey in childrel is + * still valid in parent level, but we need convert the format from child expr to + * parent expr. + */ + if (list_length(childrels) == 1) + { + /* Check for Rule 1 */ + RelOptInfo *childrel = linitial_node(RelOptInfo, childrels); + ListCell *lc; + Assert(childrel->reloptkind == RELOPT_OTHER_MEMBER_REL); + if (relation_is_onerow(childrel)) + { + add_uniquekey_for_onerow(rel); + return; + } + + foreach(lc, childrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + AppendRelInfo *appinfo = find_appinfo_by_child(root, childrel->relid); + List *parent_exprs = NIL; + bool can_reuse = true; + ListCell *lc2; + foreach(lc2, ukey->exprs) + { + Var *var = (Var *)lfirst(lc2); + /* + * If the expr comes from a expression, it is hard to build the expression + * in parent so ignore that case for now. + */ + if(!IsA(var, Var)) + { + can_reuse = false; + break; + } + /* Convert it to parent var */ + parent_exprs = lappend(parent_exprs, find_parent_var(appinfo, var)); + } + if (can_reuse) + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(parent_exprs, + ukey->multi_nullvals)); + } + } + else + { + /* Check for rule 2 */ + childrel = linitial_node(RelOptInfo, childrels); + foreach(lc, childrel->indexlist) + { + IndexOptInfo *ind = lfirst(lc); + IndexOptInfo *modified_index; + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + + /* + * During simple_copy_indexinfo_to_parent, we need to convert var from + * child var to parent var, index on expression is too complex to handle. + * so ignore it for now. + */ + if (ind->indexprs != NIL) + continue; + + modified_index = simple_copy_indexinfo_to_parent(root, rel, ind); + /* + * If the unique index doesn't contain partkey, then it is unique + * on this partition only, so it is useless for us. + */ + if (!index_constains_partkey(rel, modified_index)) + continue; + + global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index); + } + + if (global_uniq_indexlist != NIL) + { + foreach(lc, childrels) + { + RelOptInfo *child = lfirst(lc); + if (is_first) + { + is_first = false; + continue; + } + adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist); + } + /* Now we have a list of unique index which are exactly same on all childrels, + * Set the UniqueKey just like it is non-partition table + */ + populate_baserel_uniquekeys(root, rel, global_uniq_indexlist); + } + } +} + + +/* + * populate_distinctrel_uniquekeys + */ +void +populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel) +{ + /* The unique key before the distinct is still valid. */ + distinctrel->uniquekeys = list_copy(inputrel->uniquekeys); + add_uniquekey_from_sortgroups(root, distinctrel, root->parse->distinctClause); +} + +/* + * populate_grouprel_uniquekeys + */ +void +populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel) + +{ + Query *parse = root->parse; + bool input_ukey_added = false; + ListCell *lc; + + if (relation_is_onerow(inputrel)) + { + add_uniquekey_for_onerow(grouprel); + return; + } + if (parse->groupingSets) + return; + + /* A Normal group by without grouping set. */ + if (parse->groupClause) + { + /* + * Current even the groupby clause is Unique already, but if query has aggref + * We have to create grouprel still. To keep the UnqiueKey short, we will check + * the UniqueKey of input_rel still valid, if so we reuse it. + */ + foreach(lc, inputrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (list_is_subset(ukey->exprs, grouprel->reltarget->exprs)) + { + grouprel->uniquekeys = lappend(grouprel->uniquekeys, + ukey); + input_ukey_added = true; + } + } + if (!input_ukey_added) + /* + * group by clause must be a super-set of grouprel->reltarget->exprs except the + * aggregation expr, so if such exprs is unique already, no bother to generate + * new uniquekey for group by exprs. + */ + add_uniquekey_from_sortgroups(root, + grouprel, + root->parse->groupClause); + } + else + /* It has aggregation but without a group by, so only one row returned */ + add_uniquekey_for_onerow(grouprel); +} + +/* + * simple_copy_uniquekeys + * Using a function for the one-line code makes us easy to check where we simply + * copied the uniquekey. + */ +void +simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel) +{ + newrel->uniquekeys = oldrel->uniquekeys; +} + +/* + * populate_unionrel_uniquekeys + */ +void +populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel) +{ + ListCell *lc; + List *exprs = NIL; + + Assert(unionrel->uniquekeys == NIL); + + foreach(lc, unionrel->reltarget->exprs) + { + exprs = lappend(exprs, lfirst(lc)); + } + + if (exprs == NIL) + /* SQL: select union select; is valid, we need to handle it here. */ + add_uniquekey_for_onerow(unionrel); + else + unionrel->uniquekeys = lappend(unionrel->uniquekeys, + makeUniqueKey(exprs,false)); + +} + +/* + * populate_joinrel_uniquekeys + * + * populate uniquekeys for joinrel. We will check each relation to see if its + * UniqueKey is still valid via innerrel_keeps_unique, if so, we add it to + * joinrel. The multi_nullvals field will be changed to true for some outer + * join cases and one-row UniqueKey needs to be converted to normal UniqueKey + * for the same case as well. + * For the uniquekey in either baserel which can't be unique after join, we still + * check to see if combination of UniqueKeys from both side is still useful for us. + * if yes, we add it to joinrel as well. + */ +void +populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *outerrel, RelOptInfo *innerrel, + List *restrictlist, JoinType jointype) +{ + ListCell *lc, *lc2; + List *clause_list = NIL; + List *outerrel_ukey_ctx; + List *innerrel_ukey_ctx; + bool inner_onerow, outer_onerow; + bool mergejoin_allowed; + + /* Care about the outerrel relation only for SEMI/ANTI join */ + if (jointype == JOIN_SEMI || jointype == JOIN_ANTI) + { + foreach(lc, outerrel->uniquekeys) + { + UniqueKey *uniquekey = lfirst_node(UniqueKey, lc); + if (list_is_subset(uniquekey->exprs, joinrel->reltarget->exprs)) + joinrel->uniquekeys = lappend(joinrel->uniquekeys, uniquekey); + } + return; + } + + Assert(jointype == JOIN_LEFT || jointype == JOIN_FULL || jointype == JOIN_INNER); + + /* Fast path */ + if (innerrel->uniquekeys == NIL || outerrel->uniquekeys == NIL) + return; + + inner_onerow = relation_is_onerow(innerrel); + outer_onerow = relation_is_onerow(outerrel); + + outerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(outerrel); + innerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(innerrel); + + clause_list = select_mergejoin_clauses(root, joinrel, outerrel, innerrel, + restrictlist, jointype, + &mergejoin_allowed); + + if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true /* reverse */)) + { + bool outer_impact = jointype == JOIN_FULL; + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + /* Outer relation has one row, and the unique key is not duplicated after join, + * the joinrel will still has one row unless the jointype == JOIN_FULL. + */ + if (outer_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (outer_onerow) + { + /* + * The onerow outerrel becomes multi rows and multi_nullvals + * will be changed to true. We also need to set the exprs correctly since it + * can't be NIL any more. + */ + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, outerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Change multi_nullvals to true due to the full join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, true)); + else + /* Just reuse it */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + } + ctx->added_to_joinrel = true; + } + } + + if (innerrel_keeps_unique(root, outerrel, innerrel, clause_list, false)) + { + bool outer_impact = jointype == JOIN_FULL || jointype == JOIN_LEFT;; + + foreach(lc, innerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + if (inner_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (inner_onerow) + { + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, innerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Need to change multi_nullvals to true due to the outer join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, + true)); + else + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + + } + ctx->added_to_joinrel = true; + } + } + + /* + * The combination of the UniqueKey from both sides is unique as well regardless + * of join type, but no bother to add it if its subset has been added to joinrel + * already or it is not useful for the joinrel. + */ + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx1 = (UniqueKeyContext) lfirst(lc); + if (ctx1->added_to_joinrel || !ctx1->useful) + continue; + foreach(lc2, innerrel_ukey_ctx) + { + UniqueKeyContext ctx2 = (UniqueKeyContext) lfirst(lc2); + if (ctx2->added_to_joinrel || !ctx2->useful) + continue; + if (add_combined_uniquekey(root, joinrel, outerrel, innerrel, + ctx1->uniquekey, ctx2->uniquekey, + jointype)) + /* If we set a onerow UniqueKey to joinrel, we don't need other. */ + return; + } + } +} + + +/* + * convert_subquery_uniquekeys + * + * Covert the UniqueKey in subquery to outer relation. + */ +void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel) +{ + ListCell *lc; + + if (sub_final_rel->uniquekeys == NIL) + return; + + if (relation_is_onerow(sub_final_rel)) + { + add_uniquekey_for_onerow(currel); + return; + } + + Assert(currel->subroot != NULL); + + foreach(lc, sub_final_rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + ListCell *lc; + List *exprs = NIL; + bool ukey_useful = true; + + /* One row case is handled above */ + Assert(ukey->exprs != NIL); + foreach(lc, ukey->exprs) + { + Var *var; + TargetEntry *tle = tlist_member(lfirst(lc), + currel->subroot->processed_tlist); + if (tle == NULL) + { + ukey_useful = false; + break; + } + var = find_var_for_subquery_tle(currel, tle); + if (var == NULL) + { + ukey_useful = false; + break; + } + exprs = lappend(exprs, var); + } + + if (ukey_useful) + currel->uniquekeys = lappend(currel->uniquekeys, + makeUniqueKey(exprs, + ukey->multi_nullvals)); + + } +} + +/* + * innerrel_keeps_unique + * + * Check if Unique key of the innerrel is valid after join. innerrel's UniqueKey + * will be still valid if innerrel's any-column mergeop outrerel's uniquekey + * exists in clause_list. + * + * Note: the clause_list must be a list of mergeable restrictinfo already. + */ +static bool +innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *clause_list, + bool reverse) +{ + ListCell *lc, *lc2, *lc3; + + if (outerrel->uniquekeys == NIL || innerrel->uniquekeys == NIL) + return false; + + /* Check if there is outerrel's uniquekey in mergeable clause. */ + foreach(lc, outerrel->uniquekeys) + { + List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs; + bool clauselist_matchs_all_exprs = true; + foreach(lc2, outer_uq_exprs) + { + Node *outer_uq_expr = lfirst(lc2); + bool find_uq_expr_in_clauselist = false; + foreach(lc3, clause_list) + { + RestrictInfo *rinfo = lfirst_node(RestrictInfo, lc3); + Node *outer_expr; + if (reverse) + outer_expr = rinfo->outer_is_left ? get_rightop(rinfo->clause) : get_leftop(rinfo->clause); + else + outer_expr = rinfo->outer_is_left ? get_leftop(rinfo->clause) : get_rightop(rinfo->clause); + if (equal(outer_expr, outer_uq_expr)) + { + find_uq_expr_in_clauselist = true; + break; + } + } + if (!find_uq_expr_in_clauselist) + { + /* No need to check the next exprs in the current uniquekey */ + clauselist_matchs_all_exprs = false; + break; + } + } + + if (clauselist_matchs_all_exprs) + return true; + } + return false; +} + + +/* + * relation_is_onerow + * Check if it is a one-row relation by checking UniqueKey. + */ +bool +relation_is_onerow(RelOptInfo *rel) +{ + UniqueKey *ukey; + if (rel->uniquekeys == NIL) + return false; + ukey = linitial_node(UniqueKey, rel->uniquekeys); + return ukey->exprs == NIL && list_length(rel->uniquekeys) == 1; +} + +/* + * relation_has_uniquekeys_for + * Returns true if we have proofs that 'rel' cannot return multiple rows with + * the same values in each of 'exprs'. Otherwise returns false. + */ +bool +relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel, + List *exprs, bool allow_multinulls) +{ + ListCell *lc; + + /* + * For UniqueKey->onerow case, the uniquekey->exprs is empty as well + * so we can't rely on list_is_subset to handle this special cases + */ + if (exprs == NIL) + return false; + + foreach(lc, rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (ukey->multi_nullvals && !allow_multinulls) + continue; + if (list_is_subset(ukey->exprs, exprs)) + return true; + } + return false; +} + + +/* + * get_exprs_from_uniqueindex + * + * Return a list of exprs which is unique. set useful to false if this + * unique index is not useful for us. + */ +static List * +get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals) +{ + List *exprs = NIL; + ListCell *indexpr_item; + int c = 0; + + *useful = true; + *multi_nullvals = false; + + indexpr_item = list_head(unique_index->indexprs); + for(c = 0; c < unique_index->ncolumns; c++) + { + int attr = unique_index->indexkeys[c]; + Expr *expr; + bool matched_const = false; + ListCell *lc1, *lc2; + + if(attr > 0) + { + expr = list_nth_node(TargetEntry, unique_index->indextlist, c)->expr; + } + else if (attr == 0) + { + /* Expression index */ + expr = lfirst(indexpr_item); + indexpr_item = lnext(unique_index->indexprs, indexpr_item); + } + else /* attr < 0 */ + { + /* Index on system column is not supported */ + Assert(false); + } + + /* + * Check index_col = Const case with regarding to opfamily checking + * If we can remove the index_col from the final UniqueKey->exprs. + */ + forboth(lc1, const_exprs, lc2, const_expr_opfamilies) + { + if (list_member_oid((List *)lfirst(lc2), unique_index->opfamily[c]) + && match_index_to_operand((Node *) lfirst(lc1), c, unique_index)) + { + matched_const = true; + break; + } + } + + if (matched_const) + continue; + + /* Check if the indexed expr is used in rel */ + if (attr > 0) + { + /* + * Normal Indexed column, if the col is not used, then the index is useless + * for uniquekey. + */ + attr -= FirstLowInvalidHeapAttributeNumber; + + if (!bms_is_member(attr, used_varattrs)) + { + *useful = false; + break; + } + } + else if (!list_member(unique_index->rel->reltarget->exprs, expr)) + { + /* Expression index but the expression is not used in rel */ + *useful = false; + break; + } + + /* check not null property. */ + if (attr == 0) + { + /* We never know if a expression yields null or not */ + *multi_nullvals = true; + } + else if (!bms_is_member(attr, unique_index->rel->notnullattrs) + && !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber, + unique_index->rel->notnullattrs)) + { + *multi_nullvals = true; + } + + exprs = lappend(exprs, expr); + } + return exprs; +} + + +/* + * add_uniquekey_for_onerow + * If we are sure that the relation only returns one row, then all the columns + * are unique. However we don't need to create UniqueKey for every column, we + * just set exprs = NIL and overwrites all the other UniqueKey on this RelOptInfo + * since this one has strongest semantics. + */ +void +add_uniquekey_for_onerow(RelOptInfo *rel) +{ + /* + * We overwrite the previous UniqueKey on purpose since this one has the + * strongest semantic. + */ + rel->uniquekeys = list_make1(makeUniqueKey(NIL, false)); +} + + +/* + * initililze_uniquecontext_for_joinrel + * Return a List of UniqueKeyContext for an inputrel + */ +static List * +initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel) +{ + List *res = NIL; + ListCell *lc; + foreach(lc, inputrel->uniquekeys) + { + UniqueKeyContext context; + context = palloc(sizeof(struct UniqueKeyContextData)); + context->uniquekey = lfirst_node(UniqueKey, lc); + context->added_to_joinrel = false; + context->useful = true; + res = lappend(res, context); + } + return res; +} + + +/* + * get_exprs_from_uniquekey + * Unify the way of get List of exprs from a one-row UniqueKey or + * normal UniqueKey. for the onerow case, every expr in rel1 is a valid + * UniqueKey. Return a List of exprs. + * + * rel1: The relation which you want to get the exprs. + * ukey: The UniqueKey you want to get the exprs. + */ +static List * +get_exprs_from_uniquekey(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *rel1, UniqueKey *ukey) +{ + ListCell *lc; + bool onerow = rel1 != NULL && relation_is_onerow(rel1); + + List *res = NIL; + Assert(onerow || ukey); + if (onerow) + { + /* Only cares about the exprs still exist in joinrel */ + foreach(lc, joinrel->reltarget->exprs) + { + Bitmapset *relids = pull_varnos(root, lfirst(lc)); + if (bms_is_subset(relids, rel1->relids)) + { + res = lappend(res, list_make1(lfirst(lc))); + } + } + } + else + { + res = list_make1(ukey->exprs); + } + return res; +} + +/* + * Partitioned table Unique Keys. + * The partition table unique key is maintained as: + * 1. The index must be unique as usual. + * 2. The index must contains partition key. + * 3. The index must exist on all the child rel. see simple_indexinfo_equal for + * how we compare it. + */ + +/* + * index_constains_partkey + * return true if the index contains the partiton key. + */ +static bool +index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind) +{ + ListCell *lc; + int i; + Assert(IS_PARTITIONED_REL(partrel)); + Assert(partrel->part_scheme->partnatts > 0); + + for(i = 0; i < partrel->part_scheme->partnatts; i++) + { + Node *part_expr = linitial(partrel->partexprs[i]); + bool found_in_index = false; + foreach(lc, ind->indextlist) + { + Expr *index_expr = lfirst_node(TargetEntry, lc)->expr; + if (equal(index_expr, part_expr)) + { + found_in_index = true; + break; + } + } + if (!found_in_index) + return false; + } + return true; +} + +/* + * simple_indexinfo_equal + * + * Used to check if the 2 index is same as each other. The index here + * is COPIED from childrel and did some tiny changes(see + * simple_copy_indexinfo_to_parent) + */ +static bool +simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2) +{ + Size oid_cmp_len = sizeof(Oid) * ind1->ncolumns; + + return ind1->ncolumns == ind2->ncolumns && + ind1->unique == ind2->unique && + memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0 && + memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0 && + memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0 && + memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0 && + equal(get_tlist_exprs(ind1->indextlist, true), + get_tlist_exprs(ind2->indextlist, true)); +} + + +/* + * The below macros are used for simple_copy_indexinfo_to_parent which is so + * customized that I don't want to put it to copyfuncs.c. So copy it here. + */ +#define COPY_POINTER_FIELD(fldname, sz) \ + do { \ + Size _size = (sz); \ + newnode->fldname = palloc(_size); \ + memcpy(newnode->fldname, from->fldname, _size); \ + } while (0) + +#define COPY_NODE_FIELD(fldname) \ + (newnode->fldname = copyObjectImpl(from->fldname)) + +#define COPY_SCALAR_FIELD(fldname) \ + (newnode->fldname = from->fldname) + + +/* + * simple_copy_indexinfo_to_parent (from partition) + * Copy the IndexInfo from child relation to parent relation with some modification, + * which is used to test: + * 1. If the same index exists in all the childrels. + * 2. If the parentrel->reltarget/basicrestrict info matches this index. + */ +static IndexOptInfo * +simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from) +{ + IndexOptInfo *newnode = makeNode(IndexOptInfo); + AppendRelInfo *appinfo = find_appinfo_by_child(root, from->rel->relid); + ListCell *lc; + int idx = 0; + + COPY_SCALAR_FIELD(ncolumns); + COPY_SCALAR_FIELD(nkeycolumns); + COPY_SCALAR_FIELD(unique); + COPY_SCALAR_FIELD(immediate); + /* We just need to know if it is NIL or not */ + COPY_SCALAR_FIELD(indpred); + COPY_SCALAR_FIELD(predOK); + COPY_POINTER_FIELD(indexkeys, from->ncolumns * sizeof(int)); + COPY_POINTER_FIELD(indexcollations, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opfamily, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opcintype, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(sortopfamily, from->ncolumns * sizeof(Oid)); + COPY_NODE_FIELD(indextlist); + + /* Convert index exprs on child expr to expr on parent */ + foreach(lc, newnode->indextlist) + { + TargetEntry *tle = lfirst_node(TargetEntry, lc); + /* Index on expression is ignored */ + Assert(IsA(tle->expr, Var)); + tle->expr = (Expr *) find_parent_var(appinfo, (Var *) tle->expr); + newnode->indexkeys[idx] = castNode(Var, tle->expr)->varattno; + idx++; + } + newnode->rel = parentrel; + return newnode; +} + +/* + * adjust_partition_unique_indexlist + * + * global_unique_indexes: At the beginning, it contains the copy & modified + * unique index from the first partition. And then check if each index in it still + * exists in the following partitions. If no, remove it. at last, it has an + * index list which exists in all the partitions. + */ +static void +adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_indexes) +{ + ListCell *lc, *lc2; + foreach(lc, *global_unique_indexes) + { + IndexOptInfo *g_ind = lfirst_node(IndexOptInfo, lc); + bool found_in_child = false; + + foreach(lc2, childrel->indexlist) + { + IndexOptInfo *p_ind = lfirst_node(IndexOptInfo, lc2); + IndexOptInfo *p_ind_copy; + if (!p_ind->unique || !p_ind->immediate || + (p_ind->indpred != NIL && !p_ind->predOK)) + continue; + p_ind_copy = simple_copy_indexinfo_to_parent(root, parentrel, p_ind); + if (simple_indexinfo_equal(p_ind_copy, g_ind)) + { + found_in_child = true; + break; + } + } + if (!found_in_child) + /* The index doesn't exist in childrel, remove it from global_unique_indexes */ + *global_unique_indexes = foreach_delete_current(*global_unique_indexes, lc); + } +} + +/* Helper function for groupres/distinctrel */ +static void +add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgroups) +{ + Query *parse = root->parse; + List *exprs; + + /* + * XXX: If there are some vars which is not in current levelsup, the semantic is + * imprecise, should we avoid it or not? levelsup = 1 is just a demo, maybe we need to + * check every level other than 0, if so, looks we have to write another + * pull_var_walker. + */ + List *upper_vars = pull_vars_of_level((Node*)sortgroups, 1); + + if (upper_vars != NIL) + return; + + exprs = get_sortgrouplist_exprs(sortgroups, parse->targetList); + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(exprs, + false /* sortgroupclause can't be multi_nullvals */)); +} + + +/* + * add_combined_uniquekey + * The combination of both UniqueKeys is a valid UniqueKey for joinrel no matter + * the jointype. + */ +bool +add_combined_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype) +{ + + ListCell *lc1, *lc2; + + /* Either side has multi_nullvals or we have outer join, + * the combined UniqueKey has multi_nullvals */ + bool multi_nullvals = outer_ukey->multi_nullvals || + inner_ukey->multi_nullvals || IS_OUTER_JOIN(jointype); + + /* The only case we can get onerow joinrel after join */ + if (relation_is_onerow(outer_rel) + && relation_is_onerow(inner_rel) + && jointype == JOIN_INNER) + { + add_uniquekey_for_onerow(joinrel); + return true; + } + + foreach(lc1, get_exprs_from_uniquekey(root, joinrel, outer_rel, outer_ukey)) + { + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, inner_rel, inner_ukey)) + { + List *exprs = list_concat_copy(lfirst_node(List, lc1), lfirst_node(List, lc2)); + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(exprs, + multi_nullvals)); + } + } + return false; +} diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index 424d25cbd5..8d8e493f5c 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -2387,6 +2387,8 @@ grouping_planner(PlannerInfo *root, bool inheritance_update, add_path(final_rel, path); } + simple_copy_uniquekeys(current_rel, final_rel); + /* * Generate partial paths for final_rel, too, if outer query levels might * be able to make use of them. @@ -3899,6 +3901,8 @@ create_grouping_paths(PlannerInfo *root, } set_cheapest(grouped_rel); + + populate_grouprel_uniquekeys(root, grouped_rel, input_rel); return grouped_rel; } @@ -4621,7 +4625,7 @@ create_window_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(window_rel); - + simple_copy_uniquekeys(input_rel, window_rel); return window_rel; } @@ -4935,7 +4939,7 @@ create_distinct_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(distinct_rel); - + populate_distinctrel_uniquekeys(root, input_rel, distinct_rel); return distinct_rel; } @@ -5196,6 +5200,8 @@ create_ordered_paths(PlannerInfo *root, */ Assert(ordered_rel->pathlist != NIL); + simple_copy_uniquekeys(input_rel, ordered_rel); + return ordered_rel; } @@ -6073,6 +6079,9 @@ adjust_paths_for_srfs(PlannerInfo *root, RelOptInfo *rel, if (list_length(targets) == 1) return; + /* UniqueKey is not valid after handling the SRF. */ + rel->uniquekeys = NIL; + /* * Stack SRF-evaluation nodes atop each path for the rel. * diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c index becdcbb872..b7626545bf 100644 --- a/src/backend/optimizer/prep/prepunion.c +++ b/src/backend/optimizer/prep/prepunion.c @@ -689,6 +689,8 @@ generate_union_paths(SetOperationStmt *op, PlannerInfo *root, /* Undo effects of possibly forcing tuple_fraction to 0 */ root->tuple_fraction = save_fraction; + /* Add the UniqueKeys */ + populate_unionrel_uniquekeys(root, result_rel); return result_rel; } diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c index 86922a273c..6817c9c787 100644 --- a/src/backend/optimizer/util/appendinfo.c +++ b/src/backend/optimizer/util/appendinfo.c @@ -746,3 +746,47 @@ find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos) } return appinfos; } + +/* + * find_appinfo_by_child + * + */ +AppendRelInfo * +find_appinfo_by_child(PlannerInfo *root, Index child_index) +{ + ListCell *lc; + foreach(lc, root->append_rel_list) + { + AppendRelInfo *appinfo = lfirst_node(AppendRelInfo, lc); + if (appinfo->child_relid == child_index) + return appinfo; + } + elog(ERROR, "parent relation cant be found"); + return NULL; +} + +/* + * find_parent_var + * + */ +Var * +find_parent_var(AppendRelInfo *appinfo, Var *child_var) +{ + ListCell *lc; + Var *res = NULL; + Index attno = 1; + foreach(lc, appinfo->translated_vars) + { + Node *child_node = lfirst(lc); + if (equal(child_node, child_var)) + { + res = copyObject(child_var); + res->varattno = attno; + res->varno = appinfo->parent_relid; + } + attno++; + } + if (res == NULL) + elog(ERROR, "parent var can't be found."); + return res; +} diff --git a/src/backend/optimizer/util/inherit.c b/src/backend/optimizer/util/inherit.c index be1c9ddd96..3eec1f4d74 100644 --- a/src/backend/optimizer/util/inherit.c +++ b/src/backend/optimizer/util/inherit.c @@ -736,6 +736,7 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, { Node *onecq = (Node *) lfirst(lc2); bool pseudoconstant; + RestrictInfo *child_rinfo; /* check for pseudoconstant (no Vars or volatile functions) */ pseudoconstant = @@ -747,14 +748,15 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, root->hasPseudoConstantQuals = true; } /* reconstitute RestrictInfo with appropriate properties */ - childquals = lappend(childquals, - make_restrictinfo(root, - (Expr *) onecq, - rinfo->is_pushed_down, - rinfo->outerjoin_delayed, - pseudoconstant, - rinfo->security_level, - NULL, NULL, NULL)); + child_rinfo = make_restrictinfo(root, + (Expr *) onecq, + rinfo->is_pushed_down, + rinfo->outerjoin_delayed, + pseudoconstant, + rinfo->security_level, + NULL, NULL, NULL); + child_rinfo->mergeopfamilies = rinfo->mergeopfamilies; + childquals = lappend(childquals, child_rinfo); /* track minimum security level among child quals */ cq_min_security = Min(cq_min_security, rinfo->security_level); } diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 48a7ebfe45..4fe1824eb0 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -16,6 +16,7 @@ #include "nodes/execnodes.h" #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" extern A_Expr *makeA_Expr(A_Expr_Kind kind, List *name, @@ -106,4 +107,6 @@ extern GroupingSet *makeGroupingSet(GroupingSetKind kind, List *content, int loc extern VacuumRelation *makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols); +extern UniqueKey* makeUniqueKey(List *exprs, bool multi_nullvals); + #endif /* MAKEFUNC_H */ diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index e22df890ef..3320273ac1 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -265,6 +265,7 @@ typedef enum NodeTag T_EquivalenceMember, T_PathKey, T_PathTarget, + T_UniqueKey, T_RestrictInfo, T_IndexClause, T_PlaceHolderVar, diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index 0d61f04d27..80561b1003 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -739,6 +739,7 @@ typedef struct RelOptInfo QualCost baserestrictcost; /* cost of evaluating the above */ Index baserestrict_min_security; /* min security_level found in * baserestrictinfo */ + List *uniquekeys; /* List of UniqueKey */ List *joininfo; /* RestrictInfo structures for join clauses * involving this rel */ bool has_eclass_joins; /* T means joininfo is incomplete */ @@ -1059,6 +1060,28 @@ typedef struct PathKey } PathKey; +/* + * UniqueKey + * + * Represents the unique properties held by a RelOptInfo. + * + * exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL + * is a special case of UniqueKey, which means there is only 1 row in that + * relation. + * multi_nullvals: true means multi null values may exist in these exprs, so the + * uniqueness is not guaranteed in this case. This field is necessary for + * remove_useless_join & reduce_unique_semijoins where we don't mind these + * duplicated NULL values. It is set to true for 2 cases. One is a unique key + * from a unique index but the related column is nullable. The other one is for + * outer join. see populate_joinrel_uniquekeys for detail. + */ +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + /* * PathTarget * @@ -2496,7 +2519,7 @@ typedef enum * * flags indicating what kinds of grouping are possible. * partial_costs_set is true if the agg_partial_costs and agg_final_costs - * have been initialized. + * have been initialized. * agg_partial_costs gives partial aggregation costs. * agg_final_costs gives finalization costs. * target_parallel_safe is true if target is parallel safe. @@ -2526,8 +2549,8 @@ typedef struct * limit_tuples is an estimated bound on the number of output tuples, * or -1 if no LIMIT or couldn't estimate. * count_est and offset_est are the estimated values of the LIMIT and OFFSET - * expressions computed by preprocess_limit() (see comments for - * preprocess_limit() for more information). + * expressions computed by preprocess_limit() (see comments for + * preprocess_limit() for more information). */ typedef struct { diff --git a/src/include/nodes/pg_list.h b/src/include/nodes/pg_list.h index 404e03f132..85c65e910c 100644 --- a/src/include/nodes/pg_list.h +++ b/src/include/nodes/pg_list.h @@ -566,6 +566,16 @@ extern pg_nodiscard List *list_delete_first(List *list); extern pg_nodiscard List *list_delete_last(List *list); extern pg_nodiscard List *list_delete_nth_cell(List *list, int n); extern pg_nodiscard List *list_delete_cell(List *list, ListCell *cell); +extern bool list_is_subset(const List *members, const List *target); + +extern List *list_delete(List *list, void *datum); +extern List *list_delete_ptr(List *list, void *datum); +extern List *list_delete_int(List *list, int datum); +extern List *list_delete_oid(List *list, Oid datum); +extern List *list_delete_first(List *list); +extern List *list_delete_last(List *list); +extern List *list_delete_nth_cell(List *list, int n); +extern List *list_delete_cell(List *list, ListCell *cell); extern List *list_union(const List *list1, const List *list2); extern List *list_union_ptr(const List *list1, const List *list2); diff --git a/src/include/optimizer/appendinfo.h b/src/include/optimizer/appendinfo.h index 4cbf8c26cc..43b8d05f69 100644 --- a/src/include/optimizer/appendinfo.h +++ b/src/include/optimizer/appendinfo.h @@ -32,4 +32,7 @@ extern Relids adjust_child_relids_multilevel(PlannerInfo *root, Relids relids, extern AppendRelInfo **find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos); +extern AppendRelInfo *find_appinfo_by_child(PlannerInfo *root, Index child_index); +extern Var *find_parent_var(AppendRelInfo *appinfo, Var *child_var); + #endif /* APPENDINFO_H */ diff --git a/src/include/optimizer/optimizer.h b/src/include/optimizer/optimizer.h index d587952b7d..843aafa51e 100644 --- a/src/include/optimizer/optimizer.h +++ b/src/include/optimizer/optimizer.h @@ -23,6 +23,7 @@ #define OPTIMIZER_H #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" /* Test if an expression node represents a SRF call. Beware multiple eval! */ #define IS_SRF_CALL(node) \ @@ -168,6 +169,7 @@ extern TargetEntry *get_sortgroupref_tle(Index sortref, List *targetList); extern TargetEntry *get_sortgroupclause_tle(SortGroupClause *sgClause, List *targetList); +extern Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); extern Node *get_sortgroupclause_expr(SortGroupClause *sgClause, List *targetList); extern List *get_sortgrouplist_exprs(List *sgClauses, diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h index 035d3e1206..1adf99c9ee 100644 --- a/src/include/optimizer/paths.h +++ b/src/include/optimizer/paths.h @@ -247,5 +247,48 @@ extern PathKey *make_canonical_pathkey(PlannerInfo *root, int strategy, bool nulls_first); extern void add_paths_to_append_rel(PlannerInfo *root, RelOptInfo *rel, List *live_childrels); +extern List *select_mergejoin_clauses(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + JoinType jointype, + bool *mergejoin_allowed); + +/* + * uniquekeys.c + * Utilities for matching and building unique keys + */ +extern void populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List* unique_index_list); +extern void populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels); +extern void populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel); +extern void populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel); +extern void populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel); +extern void simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel); +extern void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel); +extern void populate_joinrel_uniquekeys(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + RelOptInfo *rel2, + List *restrictlist, + JoinType jointype); + +extern bool relation_has_uniquekeys_for(PlannerInfo *root, + RelOptInfo *rel, + List *exprs, + bool allow_multinulls); +extern bool relation_is_onerow(RelOptInfo *rel); #endif /* PATHS_H */ -- 2.30.2 --------------F495F6B18672582269F00FF9 Content-Type: text/x-patch; charset=UTF-8; name="0004-review-20210317.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0004-review-20210317.patch" ^ permalink raw reply [nested|flat] 91+ messages in thread
* [PATCH v38 2/6] Introduce UniqueKey attributes on RelOptInfo struct. @ 2020-05-11 07:50 一挃 <[email protected]> 0 siblings, 0 replies; 91+ messages in thread From: 一挃 @ 2020-05-11 07:50 UTC (permalink / raw) UniqueKey is a set of exprs on RelOptInfo which represents the exprs will be unique on the given RelOptInfo. You can see README.uniquekey for more information. --- src/backend/nodes/copyfuncs.c | 13 + src/backend/nodes/list.c | 31 + src/backend/nodes/makefuncs.c | 13 + src/backend/nodes/outfuncs.c | 11 + src/backend/nodes/readfuncs.c | 10 + src/backend/optimizer/path/Makefile | 3 +- src/backend/optimizer/path/README.uniquekey | 131 +++ src/backend/optimizer/path/allpaths.c | 10 + src/backend/optimizer/path/joinpath.c | 9 +- src/backend/optimizer/path/joinrels.c | 2 + src/backend/optimizer/path/pathkeys.c | 3 +- src/backend/optimizer/path/uniquekeys.c | 1135 +++++++++++++++++++ src/backend/optimizer/plan/planner.c | 13 +- src/backend/optimizer/prep/prepunion.c | 2 + src/backend/optimizer/util/appendinfo.c | 44 + src/backend/optimizer/util/inherit.c | 18 +- src/include/nodes/makefuncs.h | 3 + src/include/nodes/nodes.h | 1 + src/include/nodes/pathnodes.h | 29 +- src/include/nodes/pg_list.h | 10 + src/include/optimizer/appendinfo.h | 3 + src/include/optimizer/optimizer.h | 2 + src/include/optimizer/paths.h | 43 + 23 files changed, 1515 insertions(+), 24 deletions(-) create mode 100644 src/backend/optimizer/path/README.uniquekey create mode 100644 src/backend/optimizer/path/uniquekeys.c diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..75c1c5e824 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2296,6 +2296,16 @@ _copyPathKey(const PathKey *from) return newnode; } +static UniqueKey * +_copyUniqueKey(const UniqueKey *from) +{ + UniqueKey *newnode = makeNode(UniqueKey); + + COPY_NODE_FIELD(exprs); + COPY_SCALAR_FIELD(multi_nullvals); + + return newnode; +} /* * _copyRestrictInfo */ @@ -5220,6 +5230,9 @@ copyObjectImpl(const void *from) case T_PathKey: retval = _copyPathKey(from); break; + case T_UniqueKey: + retval = _copyUniqueKey(from); + break; case T_RestrictInfo: retval = _copyRestrictInfo(from); break; diff --git a/src/backend/nodes/list.c b/src/backend/nodes/list.c index dbf6b30233..ca099495a1 100644 --- a/src/backend/nodes/list.c +++ b/src/backend/nodes/list.c @@ -702,6 +702,37 @@ list_member_oid(const List *list, Oid datum) return false; } +/* + * return true iff every entry in "members" list is also present + * in the "target" list. + */ +bool +list_is_subset(const List *members, const List *target) +{ + const ListCell *lc1, *lc2; + + Assert(IsPointerList(members)); + Assert(IsPointerList(target)); + check_list_invariants(members); + check_list_invariants(target); + + foreach(lc1, members) + { + bool found = false; + foreach(lc2, target) + { + if (equal(lfirst(lc1), lfirst(lc2))) + { + found = true; + break; + } + } + if (!found) + return false; + } + return true; +} + /* * Delete the n'th cell (counting from 0) in list. * diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 01c110cd2f..40415d0f5b 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -815,3 +815,16 @@ makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols) v->va_cols = va_cols; return v; } + + +/* + * makeUniqueKey + */ +UniqueKey* +makeUniqueKey(List *exprs, bool multi_nullvals) +{ + UniqueKey * ukey = makeNode(UniqueKey); + ukey->exprs = exprs; + ukey->multi_nullvals = multi_nullvals; + return ukey; +} diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..44154cde6a 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2456,6 +2456,14 @@ _outPathKey(StringInfo str, const PathKey *node) WRITE_BOOL_FIELD(pk_nulls_first); } +static void +_outUniqueKey(StringInfo str, const UniqueKey *node) +{ + WRITE_NODE_TYPE("UNIQUEKEY"); + WRITE_NODE_FIELD(exprs); + WRITE_BOOL_FIELD(multi_nullvals); +} + static void _outPathTarget(StringInfo str, const PathTarget *node) { @@ -4198,6 +4206,9 @@ outNode(StringInfo str, const void *obj) case T_PathKey: _outPathKey(str, obj); break; + case T_UniqueKey: + _outUniqueKey(str, obj); + break; case T_PathTarget: _outPathTarget(str, obj); break; diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index c5e136e9c3..b3e212bf1c 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -492,6 +492,14 @@ _readSetOperationStmt(void) READ_DONE(); } +static UniqueKey * +_readUniqueKey(void) +{ + READ_LOCALS(UniqueKey); + READ_NODE_FIELD(exprs); + READ_BOOL_FIELD(multi_nullvals); + READ_DONE(); +} /* * Stuff from primnodes.h. @@ -2717,6 +2725,8 @@ parseNodeString(void) return_value = _readCommonTableExpr(); else if (MATCH("SETOPERATIONSTMT", 16)) return_value = _readSetOperationStmt(); + else if (MATCH("UNIQUEKEY", 9)) + return_value = _readUniqueKey(); else if (MATCH("ALIAS", 5)) return_value = _readAlias(); else if (MATCH("RANGEVAR", 8)) diff --git a/src/backend/optimizer/path/Makefile b/src/backend/optimizer/path/Makefile index 1e199ff66f..7b9820c25f 100644 --- a/src/backend/optimizer/path/Makefile +++ b/src/backend/optimizer/path/Makefile @@ -21,6 +21,7 @@ OBJS = \ joinpath.o \ joinrels.o \ pathkeys.o \ - tidpath.o + tidpath.o \ + uniquekeys.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/optimizer/path/README.uniquekey b/src/backend/optimizer/path/README.uniquekey new file mode 100644 index 0000000000..5eac761995 --- /dev/null +++ b/src/backend/optimizer/path/README.uniquekey @@ -0,0 +1,131 @@ +1. What is UniqueKey? +We can think UniqueKey is a set of exprs for a RelOptInfo, which we are insure +that doesn't yields same result among all the rows. The simplest UniqueKey +format is primary key. + +However we define the UnqiueKey as below. + +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + +exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL +is a special case of UniqueKey, which means there is only one row in that +relation.it has a stronger semantic than others. like SELECT uk FROM t; uk is +normal unique key and may have different values. SELECT colx FROM t WHERE uk = +const. colx is unique AND we have only 1 value. This field can used for +innerrel_is_unique. this logic is handled specially in add_uniquekey_for_onerow +function. + +multi_nullvals: true means multi null values may exist in these exprs, so the +uniqueness is not guaranteed in this case. This field is necessary for +remove_useless_join & reduce_unique_semijoins where we don't mind these +duplicated NULL values. It is set to true for 2 cases. One is a unique key +from a unique index but the related column is nullable. The other one is for +outer join. see populate_joinrel_uniquekeys for detail. + + +The UniqueKey can be used at the following cases at least: +1. remove_useless_joins. +2. reduce_semianti_joins +3. remove distinct node if distinct clause is unique. +4. remove aggnode if group by clause is unique. +5. Index Skip Scan (WIP) +6. Aggregation Push Down without 2 phase aggregation if the join can't + duplicated the aggregated rows. (work in progress feature) + +2. How is it maintained? + +We have a set of populate_xxx_unqiuekeys functions to maintain the uniquekey on +various cases. xxx includes baserel, joinrel, partitionedrel, distinctrel, +groupedrel, unionrel. and we also need to convert the uniquekey from subquery +to outer relation, which is what convert_subquery_uniquekeys does. + +1. The first part is about baserel. We handled 3 cases. suppose we have Unique +Index on (a, b). + +1. SELECT a, b FROM t. UniqueKey (a, b) +2. SELECT a FROM t WHERE b = 1; UniqueKey (a) +3. SELECT .. FROM t WHERE a = 1 AND b = 1; UniqueKey (NIL). onerow case, every + column is Unique. + +2. The next part is joinrel, this part is most error-prone, we simplified the rules +like below: +1. If the relation's UniqueKey can't be duplicated after join, then is will be + still valid for the join rel. The function we used here is + innerrel_keeps_unique. The basic idea is innerrel.any_col = outer.uk. + +2. If the UnqiueKey can't keep valid via the rule 1, the combination of the + UniqueKey from both sides are valid for sure. We can prove this as: if the + unique exprs from rel1 is duplicated by rel2, the duplicated rows must + contains different unique exprs from rel2. + +More considerations about onerow: +1. If relation with one row and it can't be duplicated, it is still possible + contains mulit_nullvas after outer join. +2. If the either UniqueKey can be duplicated after join, the can get one row + only when both side is one row AND there is no outer join. +3. Whenever the onerow UniqueKey is not a valid any more, we need to convert one + row UniqueKey to normal unique key since we don't store exprs for one-row + relation. get_exprs_from_uniquekeys will be used here. + + +More considerations about multi_nullvals after join: +1. If the original UnqiueKey has multi_nullvals, the final UniqueKey will have + mulit_nullvals in any case. +2. If a unique key doesn't allow mulit_nullvals, after some outer join, it + allows some outer join. + + +3. When we comes to subquery, we need to convert_subquery_unqiuekeys just like +convert_subquery_pathkeys. Only the UniqueKey insides subquery is referenced as +a Var in outer relation will be reused. The relationship between the outerrel.Var +and subquery.exprs is built with outerel->subroot->processed_tlist. + + +4. As for the SRF functions, it will break the uniqueness of uniquekey, However it +is handled in adjust_paths_for_srfs, which happens after the query_planner. so +we will maintain the UniqueKey until there and reset it to NIL at that +places. This can't help on distinct/group by elimination cases but probably help +in some other cases, like reduce_unqiue_semijoins/remove_useless_joins and it is +semantic correctly. + + +5. As for inherit table, we first main the UnqiueKey on childrel as well. But for +partitioned table we need to maintain 2 different kinds of +UnqiueKey. 1). UniqueKey on the parent relation 2). UniqueKey on child +relation for partition wise query. + +Example: +CREATE TABLE p (a int not null, b int not null) partition by list (a); +CREATE TABLE p0 partition of p for values in (1); +CREATE TABLE p1 partition of p for values in (2); + +create unique index p0_b on p0(b); +create unique index p1_b on p1(b); + +Now b is only unique on partition level, so the distinct can't be removed on +the following cases. SELECT DISTINCT b FROM p; + +Another example is SELECT DISTINCT a, b FROM p WHERE a = 1; Since only one +partition is chosen, the UniqueKey on child relation is same as the UniqueKey on +parent relation. + +Another usage of UniqueKey on partition level is it be helpful for +partition-wise join. + +As for the UniqueKey on parent table level, it comes with 2 different ways, +1). the UniqueKey is also derived in UniqueKey index, but the index must be same +in all the related children relations and the unique index must contains +Partition Key in it. Example: + +CREATE UNIQUE INDEX p_ab ON p(a, b); -- where a is the partition key. + +-- Query +SELECT a, b FROM p; the (a, b) is a UniqueKey of p. + +2). If the parent relation has only one childrel, the UniqueKey on childrel is + the UniqueKey on parent as well. diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index 37b4223adb..d52ad59f89 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -579,6 +579,12 @@ set_plain_rel_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) */ check_index_predicates(root, rel); + /* + * Now that we've marked which partial indexes are suitable, we can now + * build the relation's unique keys. + */ + populate_baserel_uniquekeys(root, rel, rel->indexlist); + /* Mark rel with estimated output rows, width, etc */ set_baserel_size_estimates(root, rel); } @@ -1297,6 +1303,8 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, /* Add paths to the append relation. */ add_paths_to_append_rel(root, rel, live_childrels); + if (IS_PARTITIONED_REL(rel)) + populate_partitionedrel_uniquekeys(root, rel, live_childrels); } @@ -2304,6 +2312,8 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, pathkeys, required_outer)); } + convert_subquery_uniquekeys(root, rel, sub_final_rel); + /* If outer rel allows parallelism, do same for partial paths. */ if (rel->consider_parallel && bms_is_empty(required_outer)) { diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c index 57ce97fd53..697ac047fb 100644 --- a/src/backend/optimizer/path/joinpath.c +++ b/src/backend/optimizer/path/joinpath.c @@ -71,13 +71,6 @@ static void consider_parallel_mergejoin(PlannerInfo *root, static void hash_inner_and_outer(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, RelOptInfo *innerrel, JoinType jointype, JoinPathExtraData *extra); -static List *select_mergejoin_clauses(PlannerInfo *root, - RelOptInfo *joinrel, - RelOptInfo *outerrel, - RelOptInfo *innerrel, - List *restrictlist, - JoinType jointype, - bool *mergejoin_allowed); static void generate_mergejoin_paths(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *innerrel, @@ -1927,7 +1920,7 @@ hash_inner_and_outer(PlannerInfo *root, * if it is mergejoinable and involves vars from the two sub-relations * currently of interest. */ -static List * +List * select_mergejoin_clauses(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c index 0dbe2ac726..7271f044ec 100644 --- a/src/backend/optimizer/path/joinrels.c +++ b/src/backend/optimizer/path/joinrels.c @@ -924,6 +924,8 @@ populate_joinrel_with_paths(PlannerInfo *root, RelOptInfo *rel1, /* Apply partitionwise join technique, if possible. */ try_partitionwise_join(root, rel1, rel2, joinrel, sjinfo, restrictlist); + + populate_joinrel_uniquekeys(root, joinrel, rel1, rel2, restrictlist, sjinfo->jointype); } diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index bd9a176d7d..139278829b 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -33,7 +33,6 @@ static bool pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys); static bool matches_boolean_partition_clause(RestrictInfo *rinfo, RelOptInfo *partrel, int partkeycol); -static Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); static bool right_merge_direction(PlannerInfo *root, PathKey *pathkey); @@ -1035,7 +1034,7 @@ convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel, * We need this to ensure that we don't return pathkeys describing values * that are unavailable above the level of the subquery scan. */ -static Var * +Var * find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle) { ListCell *lc; diff --git a/src/backend/optimizer/path/uniquekeys.c b/src/backend/optimizer/path/uniquekeys.c new file mode 100644 index 0000000000..77ed2b2eff --- /dev/null +++ b/src/backend/optimizer/path/uniquekeys.c @@ -0,0 +1,1135 @@ +/*------------------------------------------------------------------------- + * + * uniquekeys.c + * Utilities for matching and building unique keys + * + * Portions Copyright (c) 2020, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/optimizer/path/uniquekeys.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "nodes/makefuncs.h" +#include "nodes/nodeFuncs.h" +#include "optimizer/pathnode.h" +#include "optimizer/paths.h" +#include "optimizer/appendinfo.h" +#include "optimizer/optimizer.h" +#include "optimizer/tlist.h" +#include "rewrite/rewriteManip.h" + + +/* + * This struct is used to help populate_joinrel_uniquekeys. + * + * added_to_joinrel is true if a uniquekey (from outerrel or innerrel) + * has been added to joinrel. + * useful is true if the exprs of the uniquekey still appears in joinrel. + */ +typedef struct UniqueKeyContextData +{ + UniqueKey *uniquekey; + bool added_to_joinrel; + bool useful; +} *UniqueKeyContext; + +static List *initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel); +static bool innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + bool reverse); + +static List *get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals); +static List *get_exprs_from_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + UniqueKey *ukey); +static void add_uniquekey_for_onerow(RelOptInfo *rel); +static bool add_combined_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype); + +/* Used for unique indexes checking for partitioned table */ +static bool index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind); +static IndexOptInfo *simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from); +static bool simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2); +static void adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_index); + +/* Helper function for grouped relation and distinct relation. */ +static void add_uniquekey_from_sortgroups(PlannerInfo *root, + RelOptInfo *rel, + List *sortgroups); + +/* + * populate_baserel_uniquekeys + * Populate 'baserel' uniquekeys list by looking at the rel's unique index + * and baserestrictinfo + */ +void +populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List *indexlist) +{ + ListCell *lc; + List *matched_uniq_indexes = NIL; + + /* Attrs appears in rel->reltarget->exprs. */ + Bitmapset *used_attrs = NULL; + + List *const_exprs = NIL; + List *expr_opfamilies = NIL; + + Assert(baserel->rtekind == RTE_RELATION); + + foreach(lc, indexlist) + { + IndexOptInfo *ind = (IndexOptInfo *) lfirst(lc); + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + matched_uniq_indexes = lappend(matched_uniq_indexes, ind); + } + + if (matched_uniq_indexes == NIL) + return; + + /* Check which attrs is used in baserel->reltarget */ + pull_varattnos((Node *)baserel->reltarget->exprs, baserel->relid, &used_attrs); + + /* Check which attrno is used at a mergeable const filter */ + foreach(lc, baserel->baserestrictinfo) + { + RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); + + if (rinfo->mergeopfamilies == NIL) + continue; + + if (bms_is_empty(rinfo->left_relids)) + { + const_exprs = lappend(const_exprs, get_rightop(rinfo->clause)); + } + else if (bms_is_empty(rinfo->right_relids)) + { + const_exprs = lappend(const_exprs, get_leftop(rinfo->clause)); + } + else + continue; + + expr_opfamilies = lappend(expr_opfamilies, rinfo->mergeopfamilies); + } + + foreach(lc, matched_uniq_indexes) + { + bool multi_nullvals, useful; + List *exprs = get_exprs_from_uniqueindex(lfirst_node(IndexOptInfo, lc), + const_exprs, + expr_opfamilies, + used_attrs, + &useful, + &multi_nullvals); + if (useful) + { + if (exprs == NIL) + { + /* All the columns in Unique Index matched with a restrictinfo */ + add_uniquekey_for_onerow(baserel); + return; + } + baserel->uniquekeys = lappend(baserel->uniquekeys, + makeUniqueKey(exprs, multi_nullvals)); + } + } +} + + +/* + * populate_partitionedrel_uniquekeys + * The UniqueKey on partitionrel comes from 2 cases: + * 1). Only one partition is involved in this query, the unique key can be + * copied to parent rel from childrel. + * 2). There are some unique index which includes partition key and exists + * in all the related partitions. + * We never mind rule 2 if we hit rule 1. + */ + +void +populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels) +{ + ListCell *lc; + List *global_uniq_indexlist = NIL; + RelOptInfo *childrel; + bool is_first = true; + + Assert(IS_PARTITIONED_REL(rel)); + + if (childrels == NIL) + return; + + /* + * If there is only one partition used in this query, the UniqueKey in childrel is + * still valid in parent level, but we need convert the format from child expr to + * parent expr. + */ + if (list_length(childrels) == 1) + { + /* Check for Rule 1 */ + RelOptInfo *childrel = linitial_node(RelOptInfo, childrels); + ListCell *lc; + Assert(childrel->reloptkind == RELOPT_OTHER_MEMBER_REL); + if (relation_is_onerow(childrel)) + { + add_uniquekey_for_onerow(rel); + return; + } + + foreach(lc, childrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + AppendRelInfo *appinfo = find_appinfo_by_child(root, childrel->relid); + List *parent_exprs = NIL; + bool can_reuse = true; + ListCell *lc2; + foreach(lc2, ukey->exprs) + { + Var *var = (Var *)lfirst(lc2); + /* + * If the expr comes from a expression, it is hard to build the expression + * in parent so ignore that case for now. + */ + if(!IsA(var, Var)) + { + can_reuse = false; + break; + } + /* Convert it to parent var */ + parent_exprs = lappend(parent_exprs, find_parent_var(appinfo, var)); + } + if (can_reuse) + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(parent_exprs, + ukey->multi_nullvals)); + } + } + else + { + /* Check for rule 2 */ + childrel = linitial_node(RelOptInfo, childrels); + foreach(lc, childrel->indexlist) + { + IndexOptInfo *ind = lfirst(lc); + IndexOptInfo *modified_index; + if (!ind->unique || !ind->immediate || + (ind->indpred != NIL && !ind->predOK)) + continue; + + /* + * During simple_copy_indexinfo_to_parent, we need to convert var from + * child var to parent var, index on expression is too complex to handle. + * so ignore it for now. + */ + if (ind->indexprs != NIL) + continue; + + modified_index = simple_copy_indexinfo_to_parent(root, rel, ind); + /* + * If the unique index doesn't contain partkey, then it is unique + * on this partition only, so it is useless for us. + */ + if (!index_constains_partkey(rel, modified_index)) + continue; + + global_uniq_indexlist = lappend(global_uniq_indexlist, modified_index); + } + + if (global_uniq_indexlist != NIL) + { + foreach(lc, childrels) + { + RelOptInfo *child = lfirst(lc); + if (is_first) + { + is_first = false; + continue; + } + adjust_partition_unique_indexlist(root, rel, child, &global_uniq_indexlist); + } + /* Now we have a list of unique index which are exactly same on all childrels, + * Set the UniqueKey just like it is non-partition table + */ + populate_baserel_uniquekeys(root, rel, global_uniq_indexlist); + } + } +} + + +/* + * populate_distinctrel_uniquekeys + */ +void +populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel) +{ + /* The unique key before the distinct is still valid. */ + distinctrel->uniquekeys = list_copy(inputrel->uniquekeys); + add_uniquekey_from_sortgroups(root, distinctrel, root->parse->distinctClause); +} + +/* + * populate_grouprel_uniquekeys + */ +void +populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel) + +{ + Query *parse = root->parse; + bool input_ukey_added = false; + ListCell *lc; + + if (relation_is_onerow(inputrel)) + { + add_uniquekey_for_onerow(grouprel); + return; + } + if (parse->groupingSets) + return; + + /* A Normal group by without grouping set. */ + if (parse->groupClause) + { + /* + * Current even the groupby clause is Unique already, but if query has aggref + * We have to create grouprel still. To keep the UnqiueKey short, we will check + * the UniqueKey of input_rel still valid, if so we reuse it. + */ + foreach(lc, inputrel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (list_is_subset(ukey->exprs, grouprel->reltarget->exprs)) + { + grouprel->uniquekeys = lappend(grouprel->uniquekeys, + ukey); + input_ukey_added = true; + } + } + if (!input_ukey_added) + /* + * group by clause must be a super-set of grouprel->reltarget->exprs except the + * aggregation expr, so if such exprs is unique already, no bother to generate + * new uniquekey for group by exprs. + */ + add_uniquekey_from_sortgroups(root, + grouprel, + root->parse->groupClause); + } + else + /* It has aggregation but without a group by, so only one row returned */ + add_uniquekey_for_onerow(grouprel); +} + +/* + * simple_copy_uniquekeys + * Using a function for the one-line code makes us easy to check where we simply + * copied the uniquekey. + */ +void +simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel) +{ + newrel->uniquekeys = oldrel->uniquekeys; +} + +/* + * populate_unionrel_uniquekeys + */ +void +populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel) +{ + ListCell *lc; + List *exprs = NIL; + + Assert(unionrel->uniquekeys == NIL); + + foreach(lc, unionrel->reltarget->exprs) + { + exprs = lappend(exprs, lfirst(lc)); + } + + if (exprs == NIL) + /* SQL: select union select; is valid, we need to handle it here. */ + add_uniquekey_for_onerow(unionrel); + else + unionrel->uniquekeys = lappend(unionrel->uniquekeys, + makeUniqueKey(exprs,false)); + +} + +/* + * populate_joinrel_uniquekeys + * + * populate uniquekeys for joinrel. We will check each relation to see if its + * UniqueKey is still valid via innerrel_keeps_unique, if so, we add it to + * joinrel. The multi_nullvals field will be changed to true for some outer + * join cases and one-row UniqueKey needs to be converted to normal UniqueKey + * for the same case as well. + * For the uniquekey in either baserel which can't be unique after join, we still + * check to see if combination of UniqueKeys from both side is still useful for us. + * if yes, we add it to joinrel as well. + */ +void +populate_joinrel_uniquekeys(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *outerrel, RelOptInfo *innerrel, + List *restrictlist, JoinType jointype) +{ + ListCell *lc, *lc2; + List *clause_list = NIL; + List *outerrel_ukey_ctx; + List *innerrel_ukey_ctx; + bool inner_onerow, outer_onerow; + bool mergejoin_allowed; + + /* Care about the outerrel relation only for SEMI/ANTI join */ + if (jointype == JOIN_SEMI || jointype == JOIN_ANTI) + { + foreach(lc, outerrel->uniquekeys) + { + UniqueKey *uniquekey = lfirst_node(UniqueKey, lc); + if (list_is_subset(uniquekey->exprs, joinrel->reltarget->exprs)) + joinrel->uniquekeys = lappend(joinrel->uniquekeys, uniquekey); + } + return; + } + + Assert(jointype == JOIN_LEFT || jointype == JOIN_FULL || jointype == JOIN_INNER); + + /* Fast path */ + if (innerrel->uniquekeys == NIL || outerrel->uniquekeys == NIL) + return; + + inner_onerow = relation_is_onerow(innerrel); + outer_onerow = relation_is_onerow(outerrel); + + outerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(outerrel); + innerrel_ukey_ctx = initililze_uniquecontext_for_joinrel(innerrel); + + clause_list = select_mergejoin_clauses(root, joinrel, outerrel, innerrel, + restrictlist, jointype, + &mergejoin_allowed); + + if (innerrel_keeps_unique(root, innerrel, outerrel, clause_list, true /* reverse */)) + { + bool outer_impact = jointype == JOIN_FULL; + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + /* Outer relation has one row, and the unique key is not duplicated after join, + * the joinrel will still has one row unless the jointype == JOIN_FULL. + */ + if (outer_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (outer_onerow) + { + /* + * The onerow outerrel becomes multi rows and multi_nullvals + * will be changed to true. We also need to set the exprs correctly since it + * can't be NIL any more. + */ + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, outerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Change multi_nullvals to true due to the full join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, true)); + else + /* Just reuse it */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + } + ctx->added_to_joinrel = true; + } + } + + if (innerrel_keeps_unique(root, outerrel, innerrel, clause_list, false)) + { + bool outer_impact = jointype == JOIN_FULL || jointype == JOIN_LEFT;; + + foreach(lc, innerrel_ukey_ctx) + { + UniqueKeyContext ctx = (UniqueKeyContext)lfirst(lc); + + if (!list_is_subset(ctx->uniquekey->exprs, joinrel->reltarget->exprs)) + { + ctx->useful = false; + continue; + } + + if (inner_onerow && !outer_impact) + { + add_uniquekey_for_onerow(joinrel); + return; + } + else if (inner_onerow) + { + ListCell *lc2; + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, innerrel, NULL)) + { + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(lfirst(lc2), true)); + } + } + else + { + if (!ctx->uniquekey->multi_nullvals && outer_impact) + /* Need to change multi_nullvals to true due to the outer join. */ + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(ctx->uniquekey->exprs, + true)); + else + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + ctx->uniquekey); + + } + ctx->added_to_joinrel = true; + } + } + + /* + * The combination of the UniqueKey from both sides is unique as well regardless + * of join type, but no bother to add it if its subset has been added to joinrel + * already or it is not useful for the joinrel. + */ + foreach(lc, outerrel_ukey_ctx) + { + UniqueKeyContext ctx1 = (UniqueKeyContext) lfirst(lc); + if (ctx1->added_to_joinrel || !ctx1->useful) + continue; + foreach(lc2, innerrel_ukey_ctx) + { + UniqueKeyContext ctx2 = (UniqueKeyContext) lfirst(lc2); + if (ctx2->added_to_joinrel || !ctx2->useful) + continue; + if (add_combined_uniquekey(root, joinrel, outerrel, innerrel, + ctx1->uniquekey, ctx2->uniquekey, + jointype)) + /* If we set a onerow UniqueKey to joinrel, we don't need other. */ + return; + } + } +} + + +/* + * convert_subquery_uniquekeys + * + * Covert the UniqueKey in subquery to outer relation. + */ +void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel) +{ + ListCell *lc; + + if (sub_final_rel->uniquekeys == NIL) + return; + + if (relation_is_onerow(sub_final_rel)) + { + add_uniquekey_for_onerow(currel); + return; + } + + Assert(currel->subroot != NULL); + + foreach(lc, sub_final_rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + ListCell *lc; + List *exprs = NIL; + bool ukey_useful = true; + + /* One row case is handled above */ + Assert(ukey->exprs != NIL); + foreach(lc, ukey->exprs) + { + Var *var; + TargetEntry *tle = tlist_member(lfirst(lc), + currel->subroot->processed_tlist); + if (tle == NULL) + { + ukey_useful = false; + break; + } + var = find_var_for_subquery_tle(currel, tle); + if (var == NULL) + { + ukey_useful = false; + break; + } + exprs = lappend(exprs, var); + } + + if (ukey_useful) + currel->uniquekeys = lappend(currel->uniquekeys, + makeUniqueKey(exprs, + ukey->multi_nullvals)); + + } +} + +/* + * innerrel_keeps_unique + * + * Check if Unique key of the innerrel is valid after join. innerrel's UniqueKey + * will be still valid if innerrel's any-column mergeop outrerel's uniquekey + * exists in clause_list. + * + * Note: the clause_list must be a list of mergeable restrictinfo already. + */ +static bool +innerrel_keeps_unique(PlannerInfo *root, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *clause_list, + bool reverse) +{ + ListCell *lc, *lc2, *lc3; + + if (outerrel->uniquekeys == NIL || innerrel->uniquekeys == NIL) + return false; + + /* Check if there is outerrel's uniquekey in mergeable clause. */ + foreach(lc, outerrel->uniquekeys) + { + List *outer_uq_exprs = lfirst_node(UniqueKey, lc)->exprs; + bool clauselist_matchs_all_exprs = true; + foreach(lc2, outer_uq_exprs) + { + Node *outer_uq_expr = lfirst(lc2); + bool find_uq_expr_in_clauselist = false; + foreach(lc3, clause_list) + { + RestrictInfo *rinfo = lfirst_node(RestrictInfo, lc3); + Node *outer_expr; + if (reverse) + outer_expr = rinfo->outer_is_left ? get_rightop(rinfo->clause) : get_leftop(rinfo->clause); + else + outer_expr = rinfo->outer_is_left ? get_leftop(rinfo->clause) : get_rightop(rinfo->clause); + if (equal(outer_expr, outer_uq_expr)) + { + find_uq_expr_in_clauselist = true; + break; + } + } + if (!find_uq_expr_in_clauselist) + { + /* No need to check the next exprs in the current uniquekey */ + clauselist_matchs_all_exprs = false; + break; + } + } + + if (clauselist_matchs_all_exprs) + return true; + } + return false; +} + + +/* + * relation_is_onerow + * Check if it is a one-row relation by checking UniqueKey. + */ +bool +relation_is_onerow(RelOptInfo *rel) +{ + UniqueKey *ukey; + if (rel->uniquekeys == NIL) + return false; + ukey = linitial_node(UniqueKey, rel->uniquekeys); + return ukey->exprs == NIL && list_length(rel->uniquekeys) == 1; +} + +/* + * relation_has_uniquekeys_for + * Returns true if we have proofs that 'rel' cannot return multiple rows with + * the same values in each of 'exprs'. Otherwise returns false. + */ +bool +relation_has_uniquekeys_for(PlannerInfo *root, RelOptInfo *rel, + List *exprs, bool allow_multinulls) +{ + ListCell *lc; + + /* + * For UniqueKey->onerow case, the uniquekey->exprs is empty as well + * so we can't rely on list_is_subset to handle this special cases + */ + if (exprs == NIL) + return false; + + foreach(lc, rel->uniquekeys) + { + UniqueKey *ukey = lfirst_node(UniqueKey, lc); + if (ukey->multi_nullvals && !allow_multinulls) + continue; + if (list_is_subset(ukey->exprs, exprs)) + return true; + } + return false; +} + + +/* + * get_exprs_from_uniqueindex + * + * Return a list of exprs which is unique. set useful to false if this + * unique index is not useful for us. + */ +static List * +get_exprs_from_uniqueindex(IndexOptInfo *unique_index, + List *const_exprs, + List *const_expr_opfamilies, + Bitmapset *used_varattrs, + bool *useful, + bool *multi_nullvals) +{ + List *exprs = NIL; + ListCell *indexpr_item; + int c = 0; + + *useful = true; + *multi_nullvals = false; + + indexpr_item = list_head(unique_index->indexprs); + for(c = 0; c < unique_index->ncolumns; c++) + { + int attr = unique_index->indexkeys[c]; + Expr *expr; + bool matched_const = false; + ListCell *lc1, *lc2; + + if(attr > 0) + { + expr = list_nth_node(TargetEntry, unique_index->indextlist, c)->expr; + } + else if (attr == 0) + { + /* Expression index */ + expr = lfirst(indexpr_item); + indexpr_item = lnext(unique_index->indexprs, indexpr_item); + } + else /* attr < 0 */ + { + /* Index on system column is not supported */ + Assert(false); + } + + /* + * Check index_col = Const case with regarding to opfamily checking + * If we can remove the index_col from the final UniqueKey->exprs. + */ + forboth(lc1, const_exprs, lc2, const_expr_opfamilies) + { + if (list_member_oid((List *)lfirst(lc2), unique_index->opfamily[c]) + && match_index_to_operand((Node *) lfirst(lc1), c, unique_index)) + { + matched_const = true; + break; + } + } + + if (matched_const) + continue; + + /* Check if the indexed expr is used in rel */ + if (attr > 0) + { + /* + * Normal Indexed column, if the col is not used, then the index is useless + * for uniquekey. + */ + attr -= FirstLowInvalidHeapAttributeNumber; + + if (!bms_is_member(attr, used_varattrs)) + { + *useful = false; + break; + } + } + else if (!list_member(unique_index->rel->reltarget->exprs, expr)) + { + /* Expression index but the expression is not used in rel */ + *useful = false; + break; + } + + /* check not null property. */ + if (attr == 0) + { + /* We never know if a expression yields null or not */ + *multi_nullvals = true; + } + else if (!bms_is_member(attr, unique_index->rel->notnullattrs) + && !bms_is_member(0 - FirstLowInvalidHeapAttributeNumber, + unique_index->rel->notnullattrs)) + { + *multi_nullvals = true; + } + + exprs = lappend(exprs, expr); + } + return exprs; +} + + +/* + * add_uniquekey_for_onerow + * If we are sure that the relation only returns one row, then all the columns + * are unique. However we don't need to create UniqueKey for every column, we + * just set exprs = NIL and overwrites all the other UniqueKey on this RelOptInfo + * since this one has strongest semantics. + */ +void +add_uniquekey_for_onerow(RelOptInfo *rel) +{ + /* + * We overwrite the previous UniqueKey on purpose since this one has the + * strongest semantic. + */ + rel->uniquekeys = list_make1(makeUniqueKey(NIL, false)); +} + + +/* + * initililze_uniquecontext_for_joinrel + * Return a List of UniqueKeyContext for an inputrel + */ +static List * +initililze_uniquecontext_for_joinrel(RelOptInfo *inputrel) +{ + List *res = NIL; + ListCell *lc; + foreach(lc, inputrel->uniquekeys) + { + UniqueKeyContext context; + context = palloc(sizeof(struct UniqueKeyContextData)); + context->uniquekey = lfirst_node(UniqueKey, lc); + context->added_to_joinrel = false; + context->useful = true; + res = lappend(res, context); + } + return res; +} + + +/* + * get_exprs_from_uniquekey + * Unify the way of get List of exprs from a one-row UniqueKey or + * normal UniqueKey. for the onerow case, every expr in rel1 is a valid + * UniqueKey. Return a List of exprs. + * + * rel1: The relation which you want to get the exprs. + * ukey: The UniqueKey you want to get the exprs. + */ +static List * +get_exprs_from_uniquekey(PlannerInfo *root, RelOptInfo *joinrel, + RelOptInfo *rel1, UniqueKey *ukey) +{ + ListCell *lc; + bool onerow = rel1 != NULL && relation_is_onerow(rel1); + + List *res = NIL; + Assert(onerow || ukey); + if (onerow) + { + /* Only cares about the exprs still exist in joinrel */ + foreach(lc, joinrel->reltarget->exprs) + { + Bitmapset *relids = pull_varnos(root, lfirst(lc)); + if (bms_is_subset(relids, rel1->relids)) + { + res = lappend(res, list_make1(lfirst(lc))); + } + } + } + else + { + res = list_make1(ukey->exprs); + } + return res; +} + +/* + * Partitioned table Unique Keys. + * The partition table unique key is maintained as: + * 1. The index must be unique as usual. + * 2. The index must contains partition key. + * 3. The index must exist on all the child rel. see simple_indexinfo_equal for + * how we compare it. + */ + +/* + * index_constains_partkey + * return true if the index contains the partiton key. + */ +static bool +index_constains_partkey(RelOptInfo *partrel, IndexOptInfo *ind) +{ + ListCell *lc; + int i; + Assert(IS_PARTITIONED_REL(partrel)); + Assert(partrel->part_scheme->partnatts > 0); + + for(i = 0; i < partrel->part_scheme->partnatts; i++) + { + Node *part_expr = linitial(partrel->partexprs[i]); + bool found_in_index = false; + foreach(lc, ind->indextlist) + { + Expr *index_expr = lfirst_node(TargetEntry, lc)->expr; + if (equal(index_expr, part_expr)) + { + found_in_index = true; + break; + } + } + if (!found_in_index) + return false; + } + return true; +} + +/* + * simple_indexinfo_equal + * + * Used to check if the 2 index is same as each other. The index here + * is COPIED from childrel and did some tiny changes(see + * simple_copy_indexinfo_to_parent) + */ +static bool +simple_indexinfo_equal(IndexOptInfo *ind1, IndexOptInfo *ind2) +{ + Size oid_cmp_len = sizeof(Oid) * ind1->ncolumns; + + return ind1->ncolumns == ind2->ncolumns && + ind1->unique == ind2->unique && + memcmp(ind1->indexkeys, ind2->indexkeys, sizeof(int) * ind1->ncolumns) == 0 && + memcmp(ind1->opfamily, ind2->opfamily, oid_cmp_len) == 0 && + memcmp(ind1->opcintype, ind2->opcintype, oid_cmp_len) == 0 && + memcmp(ind1->sortopfamily, ind2->sortopfamily, oid_cmp_len) == 0 && + equal(get_tlist_exprs(ind1->indextlist, true), + get_tlist_exprs(ind2->indextlist, true)); +} + + +/* + * The below macros are used for simple_copy_indexinfo_to_parent which is so + * customized that I don't want to put it to copyfuncs.c. So copy it here. + */ +#define COPY_POINTER_FIELD(fldname, sz) \ + do { \ + Size _size = (sz); \ + newnode->fldname = palloc(_size); \ + memcpy(newnode->fldname, from->fldname, _size); \ + } while (0) + +#define COPY_NODE_FIELD(fldname) \ + (newnode->fldname = copyObjectImpl(from->fldname)) + +#define COPY_SCALAR_FIELD(fldname) \ + (newnode->fldname = from->fldname) + + +/* + * simple_copy_indexinfo_to_parent (from partition) + * Copy the IndexInfo from child relation to parent relation with some modification, + * which is used to test: + * 1. If the same index exists in all the childrels. + * 2. If the parentrel->reltarget/basicrestrict info matches this index. + */ +static IndexOptInfo * +simple_copy_indexinfo_to_parent(PlannerInfo *root, + RelOptInfo *parentrel, + IndexOptInfo *from) +{ + IndexOptInfo *newnode = makeNode(IndexOptInfo); + AppendRelInfo *appinfo = find_appinfo_by_child(root, from->rel->relid); + ListCell *lc; + int idx = 0; + + COPY_SCALAR_FIELD(ncolumns); + COPY_SCALAR_FIELD(nkeycolumns); + COPY_SCALAR_FIELD(unique); + COPY_SCALAR_FIELD(immediate); + /* We just need to know if it is NIL or not */ + COPY_SCALAR_FIELD(indpred); + COPY_SCALAR_FIELD(predOK); + COPY_POINTER_FIELD(indexkeys, from->ncolumns * sizeof(int)); + COPY_POINTER_FIELD(indexcollations, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opfamily, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(opcintype, from->ncolumns * sizeof(Oid)); + COPY_POINTER_FIELD(sortopfamily, from->ncolumns * sizeof(Oid)); + COPY_NODE_FIELD(indextlist); + + /* Convert index exprs on child expr to expr on parent */ + foreach(lc, newnode->indextlist) + { + TargetEntry *tle = lfirst_node(TargetEntry, lc); + /* Index on expression is ignored */ + Assert(IsA(tle->expr, Var)); + tle->expr = (Expr *) find_parent_var(appinfo, (Var *) tle->expr); + newnode->indexkeys[idx] = castNode(Var, tle->expr)->varattno; + idx++; + } + newnode->rel = parentrel; + return newnode; +} + +/* + * adjust_partition_unique_indexlist + * + * global_unique_indexes: At the beginning, it contains the copy & modified + * unique index from the first partition. And then check if each index in it still + * exists in the following partitions. If no, remove it. at last, it has an + * index list which exists in all the partitions. + */ +static void +adjust_partition_unique_indexlist(PlannerInfo *root, + RelOptInfo *parentrel, + RelOptInfo *childrel, + List **global_unique_indexes) +{ + ListCell *lc, *lc2; + foreach(lc, *global_unique_indexes) + { + IndexOptInfo *g_ind = lfirst_node(IndexOptInfo, lc); + bool found_in_child = false; + + foreach(lc2, childrel->indexlist) + { + IndexOptInfo *p_ind = lfirst_node(IndexOptInfo, lc2); + IndexOptInfo *p_ind_copy; + if (!p_ind->unique || !p_ind->immediate || + (p_ind->indpred != NIL && !p_ind->predOK)) + continue; + p_ind_copy = simple_copy_indexinfo_to_parent(root, parentrel, p_ind); + if (simple_indexinfo_equal(p_ind_copy, g_ind)) + { + found_in_child = true; + break; + } + } + if (!found_in_child) + /* The index doesn't exist in childrel, remove it from global_unique_indexes */ + *global_unique_indexes = foreach_delete_current(*global_unique_indexes, lc); + } +} + +/* Helper function for groupres/distinctrel */ +static void +add_uniquekey_from_sortgroups(PlannerInfo *root, RelOptInfo *rel, List *sortgroups) +{ + Query *parse = root->parse; + List *exprs; + + /* + * XXX: If there are some vars which is not in current levelsup, the semantic is + * imprecise, should we avoid it or not? levelsup = 1 is just a demo, maybe we need to + * check every level other than 0, if so, looks we have to write another + * pull_var_walker. + */ + List *upper_vars = pull_vars_of_level((Node*)sortgroups, 1); + + if (upper_vars != NIL) + return; + + exprs = get_sortgrouplist_exprs(sortgroups, parse->targetList); + rel->uniquekeys = lappend(rel->uniquekeys, + makeUniqueKey(exprs, + false /* sortgroupclause can't be multi_nullvals */)); +} + + +/* + * add_combined_uniquekey + * The combination of both UniqueKeys is a valid UniqueKey for joinrel no matter + * the jointype. + */ +bool +add_combined_uniquekey(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outer_rel, + RelOptInfo *inner_rel, + UniqueKey *outer_ukey, + UniqueKey *inner_ukey, + JoinType jointype) +{ + + ListCell *lc1, *lc2; + + /* Either side has multi_nullvals or we have outer join, + * the combined UniqueKey has multi_nullvals */ + bool multi_nullvals = outer_ukey->multi_nullvals || + inner_ukey->multi_nullvals || IS_OUTER_JOIN(jointype); + + /* The only case we can get onerow joinrel after join */ + if (relation_is_onerow(outer_rel) + && relation_is_onerow(inner_rel) + && jointype == JOIN_INNER) + { + add_uniquekey_for_onerow(joinrel); + return true; + } + + foreach(lc1, get_exprs_from_uniquekey(root, joinrel, outer_rel, outer_ukey)) + { + foreach(lc2, get_exprs_from_uniquekey(root, joinrel, inner_rel, inner_ukey)) + { + List *exprs = list_concat_copy(lfirst_node(List, lc1), lfirst_node(List, lc2)); + joinrel->uniquekeys = lappend(joinrel->uniquekeys, + makeUniqueKey(exprs, + multi_nullvals)); + } + } + return false; +} diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index 424d25cbd5..8d8e493f5c 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -2387,6 +2387,8 @@ grouping_planner(PlannerInfo *root, bool inheritance_update, add_path(final_rel, path); } + simple_copy_uniquekeys(current_rel, final_rel); + /* * Generate partial paths for final_rel, too, if outer query levels might * be able to make use of them. @@ -3899,6 +3901,8 @@ create_grouping_paths(PlannerInfo *root, } set_cheapest(grouped_rel); + + populate_grouprel_uniquekeys(root, grouped_rel, input_rel); return grouped_rel; } @@ -4621,7 +4625,7 @@ create_window_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(window_rel); - + simple_copy_uniquekeys(input_rel, window_rel); return window_rel; } @@ -4935,7 +4939,7 @@ create_distinct_paths(PlannerInfo *root, /* Now choose the best path(s) */ set_cheapest(distinct_rel); - + populate_distinctrel_uniquekeys(root, input_rel, distinct_rel); return distinct_rel; } @@ -5196,6 +5200,8 @@ create_ordered_paths(PlannerInfo *root, */ Assert(ordered_rel->pathlist != NIL); + simple_copy_uniquekeys(input_rel, ordered_rel); + return ordered_rel; } @@ -6073,6 +6079,9 @@ adjust_paths_for_srfs(PlannerInfo *root, RelOptInfo *rel, if (list_length(targets) == 1) return; + /* UniqueKey is not valid after handling the SRF. */ + rel->uniquekeys = NIL; + /* * Stack SRF-evaluation nodes atop each path for the rel. * diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c index becdcbb872..b7626545bf 100644 --- a/src/backend/optimizer/prep/prepunion.c +++ b/src/backend/optimizer/prep/prepunion.c @@ -689,6 +689,8 @@ generate_union_paths(SetOperationStmt *op, PlannerInfo *root, /* Undo effects of possibly forcing tuple_fraction to 0 */ root->tuple_fraction = save_fraction; + /* Add the UniqueKeys */ + populate_unionrel_uniquekeys(root, result_rel); return result_rel; } diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c index 86922a273c..6817c9c787 100644 --- a/src/backend/optimizer/util/appendinfo.c +++ b/src/backend/optimizer/util/appendinfo.c @@ -746,3 +746,47 @@ find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos) } return appinfos; } + +/* + * find_appinfo_by_child + * + */ +AppendRelInfo * +find_appinfo_by_child(PlannerInfo *root, Index child_index) +{ + ListCell *lc; + foreach(lc, root->append_rel_list) + { + AppendRelInfo *appinfo = lfirst_node(AppendRelInfo, lc); + if (appinfo->child_relid == child_index) + return appinfo; + } + elog(ERROR, "parent relation cant be found"); + return NULL; +} + +/* + * find_parent_var + * + */ +Var * +find_parent_var(AppendRelInfo *appinfo, Var *child_var) +{ + ListCell *lc; + Var *res = NULL; + Index attno = 1; + foreach(lc, appinfo->translated_vars) + { + Node *child_node = lfirst(lc); + if (equal(child_node, child_var)) + { + res = copyObject(child_var); + res->varattno = attno; + res->varno = appinfo->parent_relid; + } + attno++; + } + if (res == NULL) + elog(ERROR, "parent var can't be found."); + return res; +} diff --git a/src/backend/optimizer/util/inherit.c b/src/backend/optimizer/util/inherit.c index be1c9ddd96..3eec1f4d74 100644 --- a/src/backend/optimizer/util/inherit.c +++ b/src/backend/optimizer/util/inherit.c @@ -736,6 +736,7 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, { Node *onecq = (Node *) lfirst(lc2); bool pseudoconstant; + RestrictInfo *child_rinfo; /* check for pseudoconstant (no Vars or volatile functions) */ pseudoconstant = @@ -747,14 +748,15 @@ apply_child_basequals(PlannerInfo *root, RelOptInfo *parentrel, root->hasPseudoConstantQuals = true; } /* reconstitute RestrictInfo with appropriate properties */ - childquals = lappend(childquals, - make_restrictinfo(root, - (Expr *) onecq, - rinfo->is_pushed_down, - rinfo->outerjoin_delayed, - pseudoconstant, - rinfo->security_level, - NULL, NULL, NULL)); + child_rinfo = make_restrictinfo(root, + (Expr *) onecq, + rinfo->is_pushed_down, + rinfo->outerjoin_delayed, + pseudoconstant, + rinfo->security_level, + NULL, NULL, NULL); + child_rinfo->mergeopfamilies = rinfo->mergeopfamilies; + childquals = lappend(childquals, child_rinfo); /* track minimum security level among child quals */ cq_min_security = Min(cq_min_security, rinfo->security_level); } diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index 48a7ebfe45..4fe1824eb0 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -16,6 +16,7 @@ #include "nodes/execnodes.h" #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" extern A_Expr *makeA_Expr(A_Expr_Kind kind, List *name, @@ -106,4 +107,6 @@ extern GroupingSet *makeGroupingSet(GroupingSetKind kind, List *content, int loc extern VacuumRelation *makeVacuumRelation(RangeVar *relation, Oid oid, List *va_cols); +extern UniqueKey* makeUniqueKey(List *exprs, bool multi_nullvals); + #endif /* MAKEFUNC_H */ diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index e22df890ef..3320273ac1 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -265,6 +265,7 @@ typedef enum NodeTag T_EquivalenceMember, T_PathKey, T_PathTarget, + T_UniqueKey, T_RestrictInfo, T_IndexClause, T_PlaceHolderVar, diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index 0d61f04d27..80561b1003 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -739,6 +739,7 @@ typedef struct RelOptInfo QualCost baserestrictcost; /* cost of evaluating the above */ Index baserestrict_min_security; /* min security_level found in * baserestrictinfo */ + List *uniquekeys; /* List of UniqueKey */ List *joininfo; /* RestrictInfo structures for join clauses * involving this rel */ bool has_eclass_joins; /* T means joininfo is incomplete */ @@ -1059,6 +1060,28 @@ typedef struct PathKey } PathKey; +/* + * UniqueKey + * + * Represents the unique properties held by a RelOptInfo. + * + * exprs is a list of exprs which is unique on current RelOptInfo. exprs = NIL + * is a special case of UniqueKey, which means there is only 1 row in that + * relation. + * multi_nullvals: true means multi null values may exist in these exprs, so the + * uniqueness is not guaranteed in this case. This field is necessary for + * remove_useless_join & reduce_unique_semijoins where we don't mind these + * duplicated NULL values. It is set to true for 2 cases. One is a unique key + * from a unique index but the related column is nullable. The other one is for + * outer join. see populate_joinrel_uniquekeys for detail. + */ +typedef struct UniqueKey +{ + NodeTag type; + List *exprs; + bool multi_nullvals; +} UniqueKey; + /* * PathTarget * @@ -2496,7 +2519,7 @@ typedef enum * * flags indicating what kinds of grouping are possible. * partial_costs_set is true if the agg_partial_costs and agg_final_costs - * have been initialized. + * have been initialized. * agg_partial_costs gives partial aggregation costs. * agg_final_costs gives finalization costs. * target_parallel_safe is true if target is parallel safe. @@ -2526,8 +2549,8 @@ typedef struct * limit_tuples is an estimated bound on the number of output tuples, * or -1 if no LIMIT or couldn't estimate. * count_est and offset_est are the estimated values of the LIMIT and OFFSET - * expressions computed by preprocess_limit() (see comments for - * preprocess_limit() for more information). + * expressions computed by preprocess_limit() (see comments for + * preprocess_limit() for more information). */ typedef struct { diff --git a/src/include/nodes/pg_list.h b/src/include/nodes/pg_list.h index 404e03f132..85c65e910c 100644 --- a/src/include/nodes/pg_list.h +++ b/src/include/nodes/pg_list.h @@ -566,6 +566,16 @@ extern pg_nodiscard List *list_delete_first(List *list); extern pg_nodiscard List *list_delete_last(List *list); extern pg_nodiscard List *list_delete_nth_cell(List *list, int n); extern pg_nodiscard List *list_delete_cell(List *list, ListCell *cell); +extern bool list_is_subset(const List *members, const List *target); + +extern List *list_delete(List *list, void *datum); +extern List *list_delete_ptr(List *list, void *datum); +extern List *list_delete_int(List *list, int datum); +extern List *list_delete_oid(List *list, Oid datum); +extern List *list_delete_first(List *list); +extern List *list_delete_last(List *list); +extern List *list_delete_nth_cell(List *list, int n); +extern List *list_delete_cell(List *list, ListCell *cell); extern List *list_union(const List *list1, const List *list2); extern List *list_union_ptr(const List *list1, const List *list2); diff --git a/src/include/optimizer/appendinfo.h b/src/include/optimizer/appendinfo.h index 4cbf8c26cc..43b8d05f69 100644 --- a/src/include/optimizer/appendinfo.h +++ b/src/include/optimizer/appendinfo.h @@ -32,4 +32,7 @@ extern Relids adjust_child_relids_multilevel(PlannerInfo *root, Relids relids, extern AppendRelInfo **find_appinfos_by_relids(PlannerInfo *root, Relids relids, int *nappinfos); +extern AppendRelInfo *find_appinfo_by_child(PlannerInfo *root, Index child_index); +extern Var *find_parent_var(AppendRelInfo *appinfo, Var *child_var); + #endif /* APPENDINFO_H */ diff --git a/src/include/optimizer/optimizer.h b/src/include/optimizer/optimizer.h index d587952b7d..843aafa51e 100644 --- a/src/include/optimizer/optimizer.h +++ b/src/include/optimizer/optimizer.h @@ -23,6 +23,7 @@ #define OPTIMIZER_H #include "nodes/parsenodes.h" +#include "nodes/pathnodes.h" /* Test if an expression node represents a SRF call. Beware multiple eval! */ #define IS_SRF_CALL(node) \ @@ -168,6 +169,7 @@ extern TargetEntry *get_sortgroupref_tle(Index sortref, List *targetList); extern TargetEntry *get_sortgroupclause_tle(SortGroupClause *sgClause, List *targetList); +extern Var *find_var_for_subquery_tle(RelOptInfo *rel, TargetEntry *tle); extern Node *get_sortgroupclause_expr(SortGroupClause *sgClause, List *targetList); extern List *get_sortgrouplist_exprs(List *sgClauses, diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h index 035d3e1206..1adf99c9ee 100644 --- a/src/include/optimizer/paths.h +++ b/src/include/optimizer/paths.h @@ -247,5 +247,48 @@ extern PathKey *make_canonical_pathkey(PlannerInfo *root, int strategy, bool nulls_first); extern void add_paths_to_append_rel(PlannerInfo *root, RelOptInfo *rel, List *live_childrels); +extern List *select_mergejoin_clauses(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + List *restrictlist, + JoinType jointype, + bool *mergejoin_allowed); + +/* + * uniquekeys.c + * Utilities for matching and building unique keys + */ +extern void populate_baserel_uniquekeys(PlannerInfo *root, + RelOptInfo *baserel, + List* unique_index_list); +extern void populate_partitionedrel_uniquekeys(PlannerInfo *root, + RelOptInfo *rel, + List *childrels); +extern void populate_distinctrel_uniquekeys(PlannerInfo *root, + RelOptInfo *inputrel, + RelOptInfo *distinctrel); +extern void populate_grouprel_uniquekeys(PlannerInfo *root, + RelOptInfo *grouprel, + RelOptInfo *inputrel); +extern void populate_unionrel_uniquekeys(PlannerInfo *root, + RelOptInfo *unionrel); +extern void simple_copy_uniquekeys(RelOptInfo *oldrel, + RelOptInfo *newrel); +extern void convert_subquery_uniquekeys(PlannerInfo *root, + RelOptInfo *currel, + RelOptInfo *sub_final_rel); +extern void populate_joinrel_uniquekeys(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *rel1, + RelOptInfo *rel2, + List *restrictlist, + JoinType jointype); + +extern bool relation_has_uniquekeys_for(PlannerInfo *root, + RelOptInfo *rel, + List *exprs, + bool allow_multinulls); +extern bool relation_is_onerow(RelOptInfo *rel); #endif /* PATHS_H */ -- 2.26.2 --yh7cw345mbttadjz Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v38-0003-Extend-UniqueKeys.patch" ^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add sub-transaction overflow status in pg_stat_activity @ 2022-01-14 16:25 Tom Lane <[email protected]> 0 siblings, 2 replies; 91+ messages in thread From: Tom Lane @ 2022-01-14 16:25 UTC (permalink / raw) To: Julien Rouhaud <[email protected]>; +Cc: Bossart, Nathan <[email protected]>; Dilip Kumar <[email protected]>; Ashutosh Sharma <[email protected]>; Justin Pryzby <[email protected]>; PostgreSQL Hackers <[email protected]> Julien Rouhaud <[email protected]> writes: > Like many I previously had to investigate a slowdown due to sub-transaction > overflow, and even with the information available in a monitoring view (I had > to rely on a quick hackish extension as I couldn't patch postgres) it's not > terribly fun to do this way. On top of that log analyzers like pgBadger could > help to highlight such a problem. It feels to me like far too much effort is being invested in fundamentally the wrong direction here. If the subxact overflow business is causing real-world performance problems, let's find a way to fix that, not put effort into monitoring tools that do little to actually alleviate anyone's pain. regards, tom lane ^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add sub-transaction overflow status in pg_stat_activity @ 2022-01-14 19:42 Bossart, Nathan <[email protected]> parent: Tom Lane <[email protected]> 1 sibling, 1 reply; 91+ messages in thread From: Bossart, Nathan @ 2022-01-14 19:42 UTC (permalink / raw) To: Tom Lane <[email protected]>; Julien Rouhaud <[email protected]>; +Cc: Dilip Kumar <[email protected]>; Ashutosh Sharma <[email protected]>; Justin Pryzby <[email protected]>; PostgreSQL Hackers <[email protected]> On 1/14/22, 8:26 AM, "Tom Lane" <[email protected]> wrote: > Julien Rouhaud <[email protected]> writes: >> Like many I previously had to investigate a slowdown due to sub-transaction >> overflow, and even with the information available in a monitoring view (I had >> to rely on a quick hackish extension as I couldn't patch postgres) it's not >> terribly fun to do this way. On top of that log analyzers like pgBadger could >> help to highlight such a problem. > > It feels to me like far too much effort is being invested in fundamentally > the wrong direction here. If the subxact overflow business is causing > real-world performance problems, let's find a way to fix that, not put > effort into monitoring tools that do little to actually alleviate anyone's > pain. +1 An easy first step might be to increase PGPROC_MAX_CACHED_SUBXIDS and NUM_SUBTRANS_BUFFERS. This wouldn't be a long-term solution to all such performance problems, and we'd still probably want the proposed monitoring tools, but maybe it'd kick the can down the road a bit. Perhaps another improvement could be to store the topmost transaction along with the parent transaction in the subtransaction log to avoid the loop in SubTransGetTopmostTransaction(). Nathan ^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add sub-transaction overflow status in pg_stat_activity @ 2022-01-15 04:15 Julien Rouhaud <[email protected]> parent: Bossart, Nathan <[email protected]> 0 siblings, 1 reply; 91+ messages in thread From: Julien Rouhaud @ 2022-01-15 04:15 UTC (permalink / raw) To: Bossart, Nathan <[email protected]>; +Cc: Tom Lane <[email protected]>; Dilip Kumar <[email protected]>; Ashutosh Sharma <[email protected]>; Justin Pryzby <[email protected]>; PostgreSQL Hackers <[email protected]> On Fri, Jan 14, 2022 at 07:42:29PM +0000, Bossart, Nathan wrote: > On 1/14/22, 8:26 AM, "Tom Lane" <[email protected]> wrote: > > > > It feels to me like far too much effort is being invested in fundamentally > > the wrong direction here. If the subxact overflow business is causing > > real-world performance problems, let's find a way to fix that, not put > > effort into monitoring tools that do little to actually alleviate anyone's > > pain. > > +1 Agreed, it would be better but if that leads to significant work that doesn't land in pg15, it would be nice to at least get more monitoring possibilities in pg15 to help locate problems in application. > An easy first step might be to increase PGPROC_MAX_CACHED_SUBXIDS and > NUM_SUBTRANS_BUFFERS. There's already something proposed for slru sizing: https://commitfest.postgresql.org/36/2627/. Unfortunately it hasn't been committed yet despite some popularity. I also don't know how much it improves workloads that hit the overflow issue. > This wouldn't be a long-term solution to all > such performance problems, and we'd still probably want the proposed > monitoring tools, but maybe it'd kick the can down the road a bit. Yeah simply increasing PGPROC_MAX_CACHED_SUBXIDS won't really solve the problem. Also the xid cache is already ~30% of the PGPROC size, increasing it any further is likely to end up being a loss for everyone that doesn't heavily rely on needing more than 64 subtransactions. There's also something proposed at https://www.postgresql.org/message-id/[email protected], which seems to reach some nice improvement without a major redesign of the subtransaction system, but I now realize that apparently no one added it to the commitfest :( ^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add sub-transaction overflow status in pg_stat_activity @ 2022-01-15 05:13 Tom Lane <[email protected]> parent: Julien Rouhaud <[email protected]> 0 siblings, 1 reply; 91+ messages in thread From: Tom Lane @ 2022-01-15 05:13 UTC (permalink / raw) To: Julien Rouhaud <[email protected]>; +Cc: Bossart, Nathan <[email protected]>; Dilip Kumar <[email protected]>; Ashutosh Sharma <[email protected]>; Justin Pryzby <[email protected]>; PostgreSQL Hackers <[email protected]> Julien Rouhaud <[email protected]> writes: > On Fri, Jan 14, 2022 at 07:42:29PM +0000, Bossart, Nathan wrote: >> On 1/14/22, 8:26 AM, "Tom Lane" <[email protected]> wrote: >>> It feels to me like far too much effort is being invested in fundamentally >>> the wrong direction here. > Agreed, it would be better but if that leads to significant work that doesn't > land in pg15, it would be nice to at least get more monitoring possibilities > in pg15 to help locate problems in application. The discussion just upthread was envisioning not only that we'd add infrastructure for this, but then that other projects would build more infrastructure on top of that. That's an awful lot of work that will become useless --- indeed maybe counterproductive --- once we find an actual fix. I say "counterproductive" because I wonder what compatibility problems we'd have if the eventual fix results in fundamental changes in the way things work in this area. Since it's worked the same way for a lot of years, I'm not impressed by arguments that we need to push something into v15. >> An easy first step might be to increase PGPROC_MAX_CACHED_SUBXIDS and >> NUM_SUBTRANS_BUFFERS. I don't think that's an avenue to a fix. We need some more-fundamental rethinking about how this should work. (No, I don't have any ideas at the moment.) > There's also something proposed at > https://www.postgresql.org/message-id/[email protected], > which seems to reach some nice improvement without a major redesign of the > subtransaction system, but I now realize that apparently no one added it to the > commitfest :( Hmm ... that could win if we're looking up the same subtransaction's parent over and over, but I wonder if it wouldn't degrade a lot of workloads too. regards, tom lane ^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add sub-transaction overflow status in pg_stat_activity @ 2022-01-15 05:29 Julien Rouhaud <[email protected]> parent: Tom Lane <[email protected]> 0 siblings, 0 replies; 91+ messages in thread From: Julien Rouhaud @ 2022-01-15 05:29 UTC (permalink / raw) To: Tom Lane <[email protected]>; +Cc: Bossart, Nathan <[email protected]>; Dilip Kumar <[email protected]>; Ashutosh Sharma <[email protected]>; Justin Pryzby <[email protected]>; PostgreSQL Hackers <[email protected]> On Sat, Jan 15, 2022 at 12:13:39AM -0500, Tom Lane wrote: > > The discussion just upthread was envisioning not only that we'd add > infrastructure for this, but then that other projects would build > more infrastructure on top of that. That's an awful lot of work > that will become useless --- indeed maybe counterproductive --- once > we find an actual fix. I say "counterproductive" because I wonder > what compatibility problems we'd have if the eventual fix results in > fundamental changes in the way things work in this area. I'm not sure what you're referring to. If that's the hackish extension I mentioned, its goal was to provide exactly what this thread is about so I wasn't advocating for additional tooling. If that's about pgBagder, no extra work would be needed: there's already a report about any WARNING/ERROR and such found in the logs so the information would be immediately visible. > Since it's worked the same way for a lot of years, I'm not impressed > by arguments that we need to push something into v15. Well, people have also been struggling with it for a lot of years, even if they don't always come here to complain about it. And apparently at least 2 people already had to code something similar to be able to find the problematic transactions, so I still think that at least some monitoring improvement would be welcome in v15 if none of the other approach get committed. ^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add sub-transaction overflow status in pg_stat_activity @ 2022-03-21 23:45 Andres Freund <[email protected]> parent: Tom Lane <[email protected]> 1 sibling, 1 reply; 91+ messages in thread From: Andres Freund @ 2022-03-21 23:45 UTC (permalink / raw) To: Tom Lane <[email protected]>; +Cc: Julien Rouhaud <[email protected]>; Bossart, Nathan <[email protected]>; Dilip Kumar <[email protected]>; Ashutosh Sharma <[email protected]>; Justin Pryzby <[email protected]>; PostgreSQL Hackers <[email protected]> On 2022-01-14 11:25:45 -0500, Tom Lane wrote: > Julien Rouhaud <[email protected]> writes: > > Like many I previously had to investigate a slowdown due to sub-transaction > > overflow, and even with the information available in a monitoring view (I had > > to rely on a quick hackish extension as I couldn't patch postgres) it's not > > terribly fun to do this way. On top of that log analyzers like pgBadger could > > help to highlight such a problem. > > It feels to me like far too much effort is being invested in fundamentally > the wrong direction here. If the subxact overflow business is causing > real-world performance problems, let's find a way to fix that, not put > effort into monitoring tools that do little to actually alleviate anyone's > pain. There seems to be some agreement on this (I certainly do agree). Thus it seems we should mark the CF entry as rejected? It's been failing on cfbot for weeks... https://cirrus-ci.com/task/5289336424890368?logs=docs_build#L347 ^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add sub-transaction overflow status in pg_stat_activity @ 2022-11-14 15:09 Robert Haas <[email protected]> parent: Andres Freund <[email protected]> 0 siblings, 4 replies; 91+ messages in thread From: Robert Haas @ 2022-11-14 15:09 UTC (permalink / raw) To: Andres Freund <[email protected]>; +Cc: Tom Lane <[email protected]>; Julien Rouhaud <[email protected]>; Bossart, Nathan <[email protected]>; Dilip Kumar <[email protected]>; Ashutosh Sharma <[email protected]>; Justin Pryzby <[email protected]>; PostgreSQL Hackers <[email protected]> On Mon, Mar 21, 2022 at 7:45 PM Andres Freund <[email protected]> wrote: > > It feels to me like far too much effort is being invested in fundamentally > > the wrong direction here. If the subxact overflow business is causing > > real-world performance problems, let's find a way to fix that, not put > > effort into monitoring tools that do little to actually alleviate anyone's > > pain. > > There seems to be some agreement on this (I certainly do agree). Thus it seems > we should mark the CF entry as rejected? I don't think I agree with this outcome, for two reasons. First, we're just talking about an extra couple of columns in pg_stat_activity here, which does not seem like a heavy price to pay. I'm not even sure we need two columns; I think we could get down to one pretty easily. Rough idea: number of cached subtransaction XIDs if not overflowed, else NULL. Or if that's likely to create 0/NULL confusion, then maybe just a Boolean, overflowed or not. Second, the problem seems pretty fundamental to me. Shared memory is fixed size, so we cannot use it to store an unbounded number of subtransaction IDs. We could perhaps rejigger things to be more memory-efficient in some way, but no matter how many subtransaction XIDs you can keep in shared memory, the user can always consume that number plus one -- unless you allow for ~2^31 in shared memory, which seems unrealistic. To me, that means that overflowed snapshots are not going away. We could make them less painful by rewriting the SLRU stuff to be more efficient, and I bet that's possible, but I think it's probably hard, or someone would have gotten it done by now. This has been sucking for a long time and I see no evidence that progress is imminent. Even if it happens, it is unlikely that it will be a full solution. If it were possible to make SLRU lookups fast enough not to matter, we wouldn't need to have hint bits, but in reality we do have them and attempts to get rid of them have not gone well up until now, and in my opinion probably never will. The way that I view this problem is that it is relatively rare but hard for some users to troubleshoot. I think I've seen it come up multiple times, and judging from the earlier responses on this thread, several other people here have, too. In my experience, the problem is inevitably that someone has a DML statement inside a plpgsql EXCEPTION block inside a plpgsql loop. Concurrently with that, they are running a lot of queries that look at recently modified data, so that the overflowed snapshot trigger SLRU lookups often enough to matter. How is a user supposed to identify which backend is causing the problem, as things stand today? I have generally given people the advice to go find the DML inside of a plpgsql EXCEPTION block inside of a loop, but some users have trouble doing that. The DBA who is observing the performance problem is not necessarily the developer who wrote all of the PL code, and the PL code may be large and badly formatted and there could be a bunch of EXCEPTION blocks and it might not be clear which one is the problem. The exception block could be calling another function or procedure that does the actual DML rather than doing it directly, and the loop surrounding it might not be in the same function or procedure but in some other one that calls it, or it could be called repeatedly from the SQL level. I think I fundamentally disagree with the idea that we should refuse to expose instrumentation data because some day the internals might change. If we accepted that argument categorically, we wouldn't have things like backend_xmin or backend_xid in pg_stat_activity, or wait events either, but we do have those things and users find them useful. They suck in the sense that you need to know quite a bit about how the internals work in order to use them to find problems, but people who want to support production PostgreSQL instances have to learn about how those internals work one way or the other because they demonstrably matter. It is absolutely stellar when we can say "hey, we don't need to have a way for users to see what's going on here internally because they don't ever need to care," but once it is established that they do need to care, we should let them see directly the data they need to care about rather than forcing them to troubleshoot the problem in some more roundabout way like auditing all of the code and guessing which part is the problem, or writing custom dtrace scripts to run on their production instances. If and when it happens that a field like backend_xmin or the new ones proposed here are no longer relevant, we can just remove them from the monitoring views. Yeah, that's a backward compatibility break, and there's some pain associated with that. But we have demonstrated that we are perfectly willing to incur the pain associated with adding new columns when there is new and valuable information to display, and that is equally a compatibility break, in the sense that it has about the same chance of making pg_upgrade fail. In short, I think this is a good idea, and if somebody thinks that we should solve the underlying problem instead, I'd like to hear what people think a realistic solution might be. Because to me, it looks like we're refusing to commit a patch that probably took an hour to write because with 10 years of engineering effort we could *maybe* fix the root cause. -- Robert Haas EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add sub-transaction overflow status in pg_stat_activity @ 2022-11-14 15:41 Tom Lane <[email protected]> parent: Robert Haas <[email protected]> 3 siblings, 1 reply; 91+ messages in thread From: Tom Lane @ 2022-11-14 15:41 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Andres Freund <[email protected]>; Julien Rouhaud <[email protected]>; Bossart, Nathan <[email protected]>; Dilip Kumar <[email protected]>; Ashutosh Sharma <[email protected]>; Justin Pryzby <[email protected]>; PostgreSQL Hackers <[email protected]> Robert Haas <[email protected]> writes: > In short, I think this is a good idea, and if somebody thinks that we > should solve the underlying problem instead, I'd like to hear what > people think a realistic solution might be. Because to me, it looks > like we're refusing to commit a patch that probably took an hour to > write because with 10 years of engineering effort we could *maybe* fix > the root cause. Maybe the original patch took an hour to write, but it's sure been bikeshedded to death :-(. I was complaining about the total amount of attention spent more than the patch itself. The patch of record seems to be v4 from 2022-01-13, which was failing in cfbot at last report but presumably could be fixed easily. The proposed documentation's grammar is pretty shaky, but I don't see much else wrong in a quick eyeball scan. regards, tom lane ^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add sub-transaction overflow status in pg_stat_activity @ 2022-11-14 15:52 Robert Haas <[email protected]> parent: Tom Lane <[email protected]> 0 siblings, 0 replies; 91+ messages in thread From: Robert Haas @ 2022-11-14 15:52 UTC (permalink / raw) To: Tom Lane <[email protected]>; +Cc: Andres Freund <[email protected]>; Julien Rouhaud <[email protected]>; Bossart, Nathan <[email protected]>; Dilip Kumar <[email protected]>; Ashutosh Sharma <[email protected]>; Justin Pryzby <[email protected]>; PostgreSQL Hackers <[email protected]> On Mon, Nov 14, 2022 at 10:41 AM Tom Lane <[email protected]> wrote: > Maybe the original patch took an hour to write, but it's sure been > bikeshedded to death :-(. I was complaining about the total amount > of attention spent more than the patch itself. Unfortunately, that problem is not unique to this patch, and even more unfortunately, despite all the bikeshedding, we still often get it wrong. Catching up from my week off I see that you've fixed not one but two bugs in a patch I thought I'd reviewed half to death. :-( > The patch of record seems to be v4 from 2022-01-13, which was failing > in cfbot at last report but presumably could be fixed easily. The > proposed documentation's grammar is pretty shaky, but I don't see > much else wrong in a quick eyeball scan. I can take a crack at improving the documentation. Do you have a view on the best way to cut this down to a single new column, or the desirability of doing so? -- Robert Haas EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add sub-transaction overflow status in pg_stat_activity @ 2022-11-14 15:57 Justin Pryzby <[email protected]> parent: Robert Haas <[email protected]> 3 siblings, 1 reply; 91+ messages in thread From: Justin Pryzby @ 2022-11-14 15:57 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Andres Freund <[email protected]>; Tom Lane <[email protected]>; Julien Rouhaud <[email protected]>; Bossart, Nathan <[email protected]>; Dilip Kumar <[email protected]>; Ashutosh Sharma <[email protected]>; PostgreSQL Hackers <[email protected]> On Mon, Nov 14, 2022 at 10:09:57AM -0500, Robert Haas wrote: > On Mon, Mar 21, 2022 at 7:45 PM Andres Freund <[email protected]> wrote: > > > It feels to me like far too much effort is being invested in fundamentally > > > the wrong direction here. If the subxact overflow business is causing > > > real-world performance problems, let's find a way to fix that, not put > > > effort into monitoring tools that do little to actually alleviate anyone's > > > pain. > > > > There seems to be some agreement on this (I certainly do agree). Thus it seems > > we should mark the CF entry as rejected? > > I don't think I agree with this outcome, for two reasons. > > First, we're just talking about an extra couple of columns in > pg_stat_activity here, which does not seem like a heavy price to pay. The most recent patch adds a separate function rather than adding more columns to pg_stat_activity. I think the complaint about making that view wider for infrequently-used columns is entirely valid. > If and when it happens that a field like backend_xmin or the new ones > proposed here are no longer relevant, we can just remove them from the > monitoring views. Yeah, that's a backward compatibility break, and > there's some pain associated with that. But we have demonstrated that > we are perfectly willing to incur the pain associated with adding new > columns when there is new and valuable information to display, and > that is equally a compatibility break, in the sense that it has about > the same chance of making pg_upgrade fail. Why would pg_upgrade fail due to new/removed columns in pg_stat_activity? Do you mean if a user creates a view on top of it? -- Justin ^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add sub-transaction overflow status in pg_stat_activity @ 2022-11-14 16:04 Robert Haas <[email protected]> parent: Justin Pryzby <[email protected]> 0 siblings, 1 reply; 91+ messages in thread From: Robert Haas @ 2022-11-14 16:04 UTC (permalink / raw) To: Justin Pryzby <[email protected]>; +Cc: Andres Freund <[email protected]>; Tom Lane <[email protected]>; Julien Rouhaud <[email protected]>; Bossart, Nathan <[email protected]>; Dilip Kumar <[email protected]>; Ashutosh Sharma <[email protected]>; PostgreSQL Hackers <[email protected]> On Mon, Nov 14, 2022 at 10:57 AM Justin Pryzby <[email protected]> wrote: > > First, we're just talking about an extra couple of columns in > > pg_stat_activity here, which does not seem like a heavy price to pay. > > The most recent patch adds a separate function rather than adding more > columns to pg_stat_activity. I think the complaint about making that > view wider for infrequently-used columns is entirely valid. I guess that's OK. I don't particularly favor that approach here but I can live with it. I agree that too-wide views are annoying, but as far as pg_stat_activity goes, that ship has pretty much sailed already, and the same is true for a lot of other views. Inventing a one-off solution for this particular case doesn't seem particularly warranted to me but, again, I can live with it. > Why would pg_upgrade fail due to new/removed columns in > pg_stat_activity? Do you mean if a user creates a view on top of it? Yes, that is a thing that some people do, and I think it is the most likely way for any changes to the view definition to cause compatibility problems. I could be wrong, though. -- Robert Haas EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add sub-transaction overflow status in pg_stat_activity @ 2022-11-14 16:17 David G. Johnston <[email protected]> parent: Robert Haas <[email protected]> 0 siblings, 1 reply; 91+ messages in thread From: David G. Johnston @ 2022-11-14 16:17 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Justin Pryzby <[email protected]>; Andres Freund <[email protected]>; Tom Lane <[email protected]>; Julien Rouhaud <[email protected]>; Bossart, Nathan <[email protected]>; Dilip Kumar <[email protected]>; Ashutosh Sharma <[email protected]>; PostgreSQL Hackers <[email protected]> On Mon, Nov 14, 2022 at 9:04 AM Robert Haas <[email protected]> wrote: > On Mon, Nov 14, 2022 at 10:57 AM Justin Pryzby <[email protected]> > wrote: > > > First, we're just talking about an extra couple of columns in > > > pg_stat_activity here, which does not seem like a heavy price to pay. > > > > The most recent patch adds a separate function rather than adding more > > columns to pg_stat_activity. I think the complaint about making that > > view wider for infrequently-used columns is entirely valid. > > I guess that's OK. I don't particularly favor that approach here but I > can live with it. I agree that too-wide views are annoying, but as far > as pg_stat_activity goes, that ship has pretty much sailed already, > and the same is true for a lot of other views. Inventing a one-off > solution for this particular case doesn't seem particularly warranted > to me but, again, I can live with it. > > I can see putting counts that people would want to use for statistics elsewhere but IIUC the whole purpose of "overflowed" is to inform someone that their session presently has degraded performance because it has created too many subtransactions. Just because the "degraded" condition itself is rare doesn't mean the field "is my session degraded" is going to be seldom consulted. In fact, I would rather think it is always briefly consulted to confirm it has the expected value of "false" (blank, IMO, don't show anything in that column unless it is exceptional) and the presence of a value there would draw attention to the desired fact that something is wrong and warrants further investigation. The pg_stat_activity view seems like the perfect place to at least display that exception flag. David J. ^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add sub-transaction overflow status in pg_stat_activity @ 2022-11-14 16:28 Robert Haas <[email protected]> parent: David G. Johnston <[email protected]> 0 siblings, 0 replies; 91+ messages in thread From: Robert Haas @ 2022-11-14 16:28 UTC (permalink / raw) To: David G. Johnston <[email protected]>; +Cc: Justin Pryzby <[email protected]>; Andres Freund <[email protected]>; Tom Lane <[email protected]>; Julien Rouhaud <[email protected]>; Bossart, Nathan <[email protected]>; Dilip Kumar <[email protected]>; Ashutosh Sharma <[email protected]>; PostgreSQL Hackers <[email protected]> On Mon, Nov 14, 2022 at 11:18 AM David G. Johnston <[email protected]> wrote: >> I guess that's OK. I don't particularly favor that approach here but I >> can live with it. I agree that too-wide views are annoying, but as far >> as pg_stat_activity goes, that ship has pretty much sailed already, >> and the same is true for a lot of other views. Inventing a one-off >> solution for this particular case doesn't seem particularly warranted >> to me but, again, I can live with it. > > I can see putting counts that people would want to use for statistics elsewhere but IIUC the whole purpose of "overflowed" is to inform someone that their session presently has degraded performance because it has created too many subtransactions. Just because the "degraded" condition itself is rare doesn't mean the field "is my session degraded" is going to be seldom consulted. In fact, I would rather think it is always briefly consulted to confirm it has the expected value of "false" (blank, IMO, don't show anything in that column unless it is exceptional) and the presence of a value there would draw attention to the desired fact that something is wrong and warrants further investigation. The pg_stat_activity view seems like the perfect place to at least display that exception flag. OK, thanks for voting. I take that as +1 for putting it in pg_stat_activity proper, which is also my preferred approach. However, a slight correction: it doesn't inform you that your session has degraded performance. It informs you that your session may be degrading everyone else's performance. -- Robert Haas EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add sub-transaction overflow status in pg_stat_activity @ 2022-11-14 16:34 Amit Singh <[email protected]> parent: Robert Haas <[email protected]> 3 siblings, 1 reply; 91+ messages in thread From: Amit Singh @ 2022-11-14 16:34 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Andres Freund <[email protected]>; Tom Lane <[email protected]>; Julien Rouhaud <[email protected]>; Bossart, Nathan <[email protected]>; Dilip Kumar <[email protected]>; Ashutosh Sharma <[email protected]>; Justin Pryzby <[email protected]>; PostgreSQL Hackers <[email protected]> Making the information available in pg_stat_activity makes it a lot easier to identify the pid which has caused the subtran overflow. Debugging through the app code can be an endless exercise and logging every statement in postgresql logs is not practical either. If the overhead of fetching the information isn't too big, I think we should consider the subtransaction_count and is_overflowed field as potential candidates for the enhancement of pg_stat_activity. Regards Amit Singh On Mon, Nov 14, 2022 at 11:10 PM Robert Haas <[email protected]> wrote: > On Mon, Mar 21, 2022 at 7:45 PM Andres Freund <[email protected]> wrote: > > > It feels to me like far too much effort is being invested in > fundamentally > > > the wrong direction here. If the subxact overflow business is causing > > > real-world performance problems, let's find a way to fix that, not put > > > effort into monitoring tools that do little to actually alleviate > anyone's > > > pain. > > > > There seems to be some agreement on this (I certainly do agree). Thus it > seems > > we should mark the CF entry as rejected? > > I don't think I agree with this outcome, for two reasons. > > First, we're just talking about an extra couple of columns in > pg_stat_activity here, which does not seem like a heavy price to pay. > I'm not even sure we need two columns; I think we could get down to > one pretty easily. Rough idea: number of cached subtransaction XIDs if > not overflowed, else NULL. Or if that's likely to create 0/NULL > confusion, then maybe just a Boolean, overflowed or not. > > Second, the problem seems pretty fundamental to me. Shared memory is > fixed size, so we cannot use it to store an unbounded number of > subtransaction IDs. We could perhaps rejigger things to be more > memory-efficient in some way, but no matter how many subtransaction > XIDs you can keep in shared memory, the user can always consume that > number plus one -- unless you allow for ~2^31 in shared memory, which > seems unrealistic. To me, that means that overflowed snapshots are not > going away. We could make them less painful by rewriting the SLRU > stuff to be more efficient, and I bet that's possible, but I think > it's probably hard, or someone would have gotten it done by now. This > has been sucking for a long time and I see no evidence that progress > is imminent. Even if it happens, it is unlikely that it will be a full > solution. If it were possible to make SLRU lookups fast enough not to > matter, we wouldn't need to have hint bits, but in reality we do have > them and attempts to get rid of them have not gone well up until now, > and in my opinion probably never will. > > The way that I view this problem is that it is relatively rare but > hard for some users to troubleshoot. I think I've seen it come up > multiple times, and judging from the earlier responses on this thread, > several other people here have, too. In my experience, the problem is > inevitably that someone has a DML statement inside a plpgsql EXCEPTION > block inside a plpgsql loop. Concurrently with that, they are running > a lot of queries that look at recently modified data, so that the > overflowed snapshot trigger SLRU lookups often enough to matter. How > is a user supposed to identify which backend is causing the problem, > as things stand today? I have generally given people the advice to go > find the DML inside of a plpgsql EXCEPTION block inside of a loop, but > some users have trouble doing that. The DBA who is observing the > performance problem is not necessarily the developer who wrote all of > the PL code, and the PL code may be large and badly formatted and > there could be a bunch of EXCEPTION blocks and it might not be clear > which one is the problem. The exception block could be calling another > function or procedure that does the actual DML rather than doing it > directly, and the loop surrounding it might not be in the same > function or procedure but in some other one that calls it, or it could > be called repeatedly from the SQL level. > > I think I fundamentally disagree with the idea that we should refuse > to expose instrumentation data because some day the internals might > change. If we accepted that argument categorically, we wouldn't have > things like backend_xmin or backend_xid in pg_stat_activity, or wait > events either, but we do have those things and users find them useful. > They suck in the sense that you need to know quite a bit about how the > internals work in order to use them to find problems, but people who > want to support production PostgreSQL instances have to learn about > how those internals work one way or the other because they > demonstrably matter. It is absolutely stellar when we can say "hey, we > don't need to have a way for users to see what's going on here > internally because they don't ever need to care," but once it is > established that they do need to care, we should let them see directly > the data they need to care about rather than forcing them to > troubleshoot the problem in some more roundabout way like auditing all > of the code and guessing which part is the problem, or writing custom > dtrace scripts to run on their production instances. > > If and when it happens that a field like backend_xmin or the new ones > proposed here are no longer relevant, we can just remove them from the > monitoring views. Yeah, that's a backward compatibility break, and > there's some pain associated with that. But we have demonstrated that > we are perfectly willing to incur the pain associated with adding new > columns when there is new and valuable information to display, and > that is equally a compatibility break, in the sense that it has about > the same chance of making pg_upgrade fail. > > In short, I think this is a good idea, and if somebody thinks that we > should solve the underlying problem instead, I'd like to hear what > people think a realistic solution might be. Because to me, it looks > like we're refusing to commit a patch that probably took an hour to > write because with 10 years of engineering effort we could *maybe* fix > the root cause. > -- > Robert Haas > EDB: http://www.enterprisedb.com > > > ^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add sub-transaction overflow status in pg_stat_activity @ 2022-11-14 16:41 Robert Haas <[email protected]> parent: Amit Singh <[email protected]> 0 siblings, 1 reply; 91+ messages in thread From: Robert Haas @ 2022-11-14 16:41 UTC (permalink / raw) To: Amit Singh <[email protected]>; +Cc: Andres Freund <[email protected]>; Tom Lane <[email protected]>; Julien Rouhaud <[email protected]>; Bossart, Nathan <[email protected]>; Dilip Kumar <[email protected]>; Ashutosh Sharma <[email protected]>; Justin Pryzby <[email protected]>; PostgreSQL Hackers <[email protected]> On Mon, Nov 14, 2022 at 11:35 AM Amit Singh <[email protected]> wrote: > Making the information available in pg_stat_activity makes it a lot easier to identify the pid which has caused the subtran overflow. Debugging through the app code can be an endless exercise and logging every statement in postgresql logs is not practical either. If the overhead of fetching the information isn't too big, I think we should consider the subtransaction_count and is_overflowed field as potential candidates for the enhancement of pg_stat_activity. The overhead of fetching the information is not large, but Justin is concerned about the effect on the display width. I feel that's kind of a lost cause because it's so wide already anyway, but I don't see a reason why we need *two* new columns. Can't we get by with just one? It could be overflowed true/false, or it could be the number of subtransaction XIDs but with NULL instead if overflowed. Do you have a view on this point? -- Robert Haas EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add sub-transaction overflow status in pg_stat_activity @ 2022-11-14 16:48 David G. Johnston <[email protected]> parent: Robert Haas <[email protected]> 0 siblings, 2 replies; 91+ messages in thread From: David G. Johnston @ 2022-11-14 16:48 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Amit Singh <[email protected]>; Andres Freund <[email protected]>; Tom Lane <[email protected]>; Julien Rouhaud <[email protected]>; Bossart, Nathan <[email protected]>; Dilip Kumar <[email protected]>; Ashutosh Sharma <[email protected]>; Justin Pryzby <[email protected]>; PostgreSQL Hackers <[email protected]> On Mon, Nov 14, 2022 at 9:41 AM Robert Haas <[email protected]> wrote: > On Mon, Nov 14, 2022 at 11:35 AM Amit Singh <[email protected]> > wrote: > > Making the information available in pg_stat_activity makes it a lot > easier to identify the pid which has caused the subtran overflow. Debugging > through the app code can be an endless exercise and logging every statement > in postgresql logs is not practical either. If the overhead of fetching the > information isn't too big, I think we should consider the > subtransaction_count and is_overflowed field as potential candidates for > the enhancement of pg_stat_activity. > > The overhead of fetching the information is not large, but Justin is > concerned about the effect on the display width. I feel that's kind of > a lost cause because it's so wide already anyway, but I don't see a > reason why we need *two* new columns. Can't we get by with just one? > It could be overflowed true/false, or it could be the number of > subtransaction XIDs but with NULL instead if overflowed. > > Do you have a view on this point? > > NULL when overflowed seems like the opposite of the desired effect, calling attention to the exceptional status. Make it a text column and write "overflow" or "###" as appropriate. Anyone using the column is going to end up wanting to special-case overflow anyway and number-to-text conversion aside from overflow is simple enough if a number, and not just a display label, is needed. David J. ^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add sub-transaction overflow status in pg_stat_activity @ 2022-11-14 17:29 Tom Lane <[email protected]> parent: David G. Johnston <[email protected]> 1 sibling, 1 reply; 91+ messages in thread From: Tom Lane @ 2022-11-14 17:29 UTC (permalink / raw) To: David G. Johnston <[email protected]>; +Cc: Robert Haas <[email protected]>; Amit Singh <[email protected]>; Andres Freund <[email protected]>; Julien Rouhaud <[email protected]>; Bossart, Nathan <[email protected]>; Dilip Kumar <[email protected]>; Ashutosh Sharma <[email protected]>; Justin Pryzby <[email protected]>; PostgreSQL Hackers <[email protected]> "David G. Johnston" <[email protected]> writes: > On Mon, Nov 14, 2022 at 9:41 AM Robert Haas <[email protected]> wrote: >> The overhead of fetching the information is not large, but Justin is >> concerned about the effect on the display width. I feel that's kind of >> a lost cause because it's so wide already anyway, but I don't see a >> reason why we need *two* new columns. Can't we get by with just one? >> It could be overflowed true/false, or it could be the number of >> subtransaction XIDs but with NULL instead if overflowed. > NULL when overflowed seems like the opposite of the desired effect, calling > attention to the exceptional status. Make it a text column and write > "overflow" or "###" as appropriate. Anyone using the column is going to > end up wanting to special-case overflow anyway and number-to-text > conversion aside from overflow is simple enough if a number, and not just a > display label, is needed. I'd vote for just overflowed true/false. Why do people need to know the exact number of subtransactions? (If there is a use-case, that would definitely be material for an auxiliary function instead of a view column.) regards, tom lane ^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add sub-transaction overflow status in pg_stat_activity @ 2022-11-14 17:47 Andres Freund <[email protected]> parent: Tom Lane <[email protected]> 0 siblings, 1 reply; 91+ messages in thread From: Andres Freund @ 2022-11-14 17:47 UTC (permalink / raw) To: Tom Lane <[email protected]>; +Cc: David G. Johnston <[email protected]>; Robert Haas <[email protected]>; Amit Singh <[email protected]>; Julien Rouhaud <[email protected]>; Bossart, Nathan <[email protected]>; Dilip Kumar <[email protected]>; Ashutosh Sharma <[email protected]>; Justin Pryzby <[email protected]>; PostgreSQL Hackers <[email protected]> Hi, On 2022-11-14 12:29:58 -0500, Tom Lane wrote: > I'd vote for just overflowed true/false. Why do people need to know > the exact number of subtransactions? (If there is a use-case, that > would definitely be material for an auxiliary function instead of a > view column.) I'd go the other way. It's pretty unimportant whether it overflowed, it's important how many subtxns there are. The cases where overflowing causes real problems are when there's many thousand subtxns - which one can't judge just from suboverflowed alone. Nor can monitoring a boolean tell you whether you're creeping closer to the danger zone. Monitoring the number also has the advantage that we'd not embed an implementation detail ("suboverflowed") in a view. The number of subtransactions is far less prone to changing than the way we implement subtransactions in the procarray. But TBH, to me this still is something that'd be better addressed with a tracepoint. I don't buy the argument that the ship of pg_stat_activity width has entirely sailed. A session still fits onto a reasonably sized terminal in \x output - but not much longer. Greetings, Andres Freund ^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add sub-transaction overflow status in pg_stat_activity @ 2022-11-14 18:43 Robert Haas <[email protected]> parent: Andres Freund <[email protected]> 0 siblings, 3 replies; 91+ messages in thread From: Robert Haas @ 2022-11-14 18:43 UTC (permalink / raw) To: Andres Freund <[email protected]>; +Cc: Tom Lane <[email protected]>; David G. Johnston <[email protected]>; Amit Singh <[email protected]>; Julien Rouhaud <[email protected]>; Bossart, Nathan <[email protected]>; Dilip Kumar <[email protected]>; Ashutosh Sharma <[email protected]>; Justin Pryzby <[email protected]>; PostgreSQL Hackers <[email protected]> On Mon, Nov 14, 2022 at 12:47 PM Andres Freund <[email protected]> wrote: > I'd go the other way. It's pretty unimportant whether it overflowed, it's > important how many subtxns there are. The cases where overflowing causes real > problems are when there's many thousand subtxns - which one can't judge just > from suboverflowed alone. Nor can monitoring a boolean tell you whether you're > creeping closer to the danger zone. This is the opposite of what I believe to be true. I thought the problem is that once a single backend overflows the subxid array, all snapshots have to be created suboverflowed, and this makes visibility checking more expensive. It's my impression that for some users this creates and extremely steep performance cliff: the difference between no backends overflowing and 1 backend overflowing is large, but whether you are close to the limit makes no difference as long as you don't reach it, and once you've passed it it makes little difference how far past it you go. > But TBH, to me this still is something that'd be better addressed with a > tracepoint. I think that makes it far, far less accessible to the typical user. > I don't buy the argument that the ship of pg_stat_activity width has entirely > sailed. A session still fits onto a reasonably sized terminal in \x output - > but not much longer. I guess it depends on what you mean by reasonable. For me, without \x, it wraps across five times on an idle system with the 24x80 window that I normally use, and even if I full screen my terminal window, it still wraps around. With \x, sure, it fits, both only if the query is shorter than the width of my window minus ~25 characters, which isn't that likely to be the case IME because users write long queries. I don't even try to use \x most of the time because the queries are likely to be long enough to destroy any benefit, but it all depends on how big your terminal is and how long your queries are. -- Robert Haas EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add sub-transaction overflow status in pg_stat_activity @ 2022-11-14 19:16 David G. Johnston <[email protected]> parent: Robert Haas <[email protected]> 2 siblings, 1 reply; 91+ messages in thread From: David G. Johnston @ 2022-11-14 19:16 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Andres Freund <[email protected]>; Tom Lane <[email protected]>; Amit Singh <[email protected]>; Julien Rouhaud <[email protected]>; Bossart, Nathan <[email protected]>; Dilip Kumar <[email protected]>; Ashutosh Sharma <[email protected]>; Justin Pryzby <[email protected]>; PostgreSQL Hackers <[email protected]> On Mon, Nov 14, 2022 at 11:43 AM Robert Haas <[email protected]> wrote: > On Mon, Nov 14, 2022 at 12:47 PM Andres Freund <[email protected]> wrote: > > I'd go the other way. It's pretty unimportant whether it overflowed, it's > > important how many subtxns there are. The cases where overflowing causes > real > > problems are when there's many thousand subtxns - which one can't judge > just > > from suboverflowed alone. Nor can monitoring a boolean tell you whether > you're > > creeping closer to the danger zone. > > This is the opposite of what I believe to be true. I thought the > problem is that once a single backend overflows the subxid array, all > snapshots have to be created suboverflowed, and this makes visibility > checking more expensive. It's my impression that for some users this > creates and extremely steep performance cliff: the difference between > no backends overflowing and 1 backend overflowing is large, but > whether you are close to the limit makes no difference as long as you > don't reach it, and once you've passed it it makes little difference > how far past it you go. > > Assuming getting an actual count value to print is fairly cheap, or even a sunk cost if you are going to report overflow, I don't see why we wouldn't want to provide the more detailed data. My concern, through ignorance, with reporting a number is that it would have no context in the query result itself. If I have two rows with numbers, one with 10 and one with 1,000, is the two orders of magnitude of the second number important or does overflow happen at, say, 65,000 and so both numbers are exceedingly small and thus not worth worrying about? That can be handled by documentation just fine, so long as the reference number in question isn't a per-session variable. Otherwise, showing some kind of "percent of max" computation seems warranted. In which case maybe the two presentation outputs would be: 1,000 (13%) Overflowed David J. ^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add sub-transaction overflow status in pg_stat_activity @ 2022-11-14 19:32 Robert Haas <[email protected]> parent: David G. Johnston <[email protected]> 0 siblings, 0 replies; 91+ messages in thread From: Robert Haas @ 2022-11-14 19:32 UTC (permalink / raw) To: David G. Johnston <[email protected]>; +Cc: Andres Freund <[email protected]>; Tom Lane <[email protected]>; Amit Singh <[email protected]>; Julien Rouhaud <[email protected]>; Bossart, Nathan <[email protected]>; Dilip Kumar <[email protected]>; Ashutosh Sharma <[email protected]>; Justin Pryzby <[email protected]>; PostgreSQL Hackers <[email protected]> On Mon, Nov 14, 2022 at 2:17 PM David G. Johnston <[email protected]> wrote: > Assuming getting an actual count value to print is fairly cheap, or even a sunk cost if you are going to report overflow, I don't see why we wouldn't want to provide the more detailed data. > > My concern, through ignorance, with reporting a number is that it would have no context in the query result itself. If I have two rows with numbers, one with 10 and one with 1,000, is the two orders of magnitude of the second number important or does overflow happen at, say, 65,000 and so both numbers are exceedingly small and thus not worth worrying about? That can be handled by documentation just fine, so long as the reference number in question isn't a per-session variable. Otherwise, showing some kind of "percent of max" computation seems warranted. In which case maybe the two presentation outputs would be: > > 1,000 (13%) > Overflowed I think the idea of cramming a bunch of stuff into a text field is dead on arrival. Data types are a wonderful invention because they let people write queries, say looking for backends where overflowed = true, or backends where subxids > 64. that gets much harder if the query has to try to make sense of some random text representation. If both values are separately important, then we need to report them both, and the only question is whether to do that in pg_stat_activity or via a side mechanism. What I don't yet understand is why that's true. I think the important question is whether there are overflowed backends, and Andres thinks it's how many subtransaction XIDs there are, so there is a reasonable chance that both things actually matter in separate scenarios. But I only know the scenario in which overflowed matters, not the one in which subtransaction XID count matters. -- Robert Haas EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add sub-transaction overflow status in pg_stat_activity @ 2022-11-14 21:03 Tom Lane <[email protected]> parent: Robert Haas <[email protected]> 2 siblings, 0 replies; 91+ messages in thread From: Tom Lane @ 2022-11-14 21:03 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Andres Freund <[email protected]>; David G. Johnston <[email protected]>; Amit Singh <[email protected]>; Julien Rouhaud <[email protected]>; Bossart, Nathan <[email protected]>; Dilip Kumar <[email protected]>; Ashutosh Sharma <[email protected]>; Justin Pryzby <[email protected]>; PostgreSQL Hackers <[email protected]> Robert Haas <[email protected]> writes: > On Mon, Nov 14, 2022 at 12:47 PM Andres Freund <[email protected]> wrote: >> I'd go the other way. It's pretty unimportant whether it overflowed, it's >> important how many subtxns there are. The cases where overflowing causes real >> problems are when there's many thousand subtxns - which one can't judge just >> from suboverflowed alone. Nor can monitoring a boolean tell you whether you're >> creeping closer to the danger zone. > This is the opposite of what I believe to be true. I thought the > problem is that once a single backend overflows the subxid array, all > snapshots have to be created suboverflowed, and this makes visibility > checking more expensive. Yeah, that's what I thought too. Andres, please enlarge ... regards, tom lane ^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add sub-transaction overflow status in pg_stat_activity @ 2022-11-14 21:17 Andres Freund <[email protected]> parent: Robert Haas <[email protected]> 2 siblings, 2 replies; 91+ messages in thread From: Andres Freund @ 2022-11-14 21:17 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Tom Lane <[email protected]>; David G. Johnston <[email protected]>; Amit Singh <[email protected]>; Julien Rouhaud <[email protected]>; Bossart, Nathan <[email protected]>; Dilip Kumar <[email protected]>; Ashutosh Sharma <[email protected]>; Justin Pryzby <[email protected]>; PostgreSQL Hackers <[email protected]> Hi, On 2022-11-14 13:43:41 -0500, Robert Haas wrote: > On Mon, Nov 14, 2022 at 12:47 PM Andres Freund <[email protected]> wrote: > > I'd go the other way. It's pretty unimportant whether it overflowed, it's > > important how many subtxns there are. The cases where overflowing causes real > > problems are when there's many thousand subtxns - which one can't judge just > > from suboverflowed alone. Nor can monitoring a boolean tell you whether you're > > creeping closer to the danger zone. > > This is the opposite of what I believe to be true. I thought the > problem is that once a single backend overflows the subxid array, all > snapshots have to be created suboverflowed, and this makes visibility > checking more expensive. It's my impression that for some users this > creates and extremely steep performance cliff: the difference between > no backends overflowing and 1 backend overflowing is large, but > whether you are close to the limit makes no difference as long as you > don't reach it, and once you've passed it it makes little difference > how far past it you go. First, it's not good to have a cliff that you can't see coming - presumbly you'd want to warn *before* you regularly reach PGPROC_MAX_CACHED_SUBXIDS subxids, rather when the shit has hit the fan already. IMO the number matters a lot when analyzing why this is happening / how to react. A session occasionally reaching 65 subxids might be tolerable and not necessarily indicative of a bug. But 100k subxids is something that one just can't accept. Perhaps this would better be tackled by a new "visibility" view. It could show - number of sessions with a snapshot - max age of backend xmin - pid with max backend xmin - number of sessions that suboverflowed - pid of the session with the most subxids - age of the oldest prepared xact - age of the oldest slot - age of the oldest walsender - ... Perhaps implemented in SQL, with new functions for accessing the properties we don't expose today. That'd address the pg_stat_activity width, while still allowing very granular access when necessary. And provide insight into something that's way to hard to query right now. > > I don't buy the argument that the ship of pg_stat_activity width has entirely > > sailed. A session still fits onto a reasonably sized terminal in \x output - > > but not much longer. > > I guess it depends on what you mean by reasonable. For me, without \x, > it wraps across five times on an idle system with the 24x80 window > that I normally use, and even if I full screen my terminal window, it > still wraps around. With \x, sure, it fits, both only if the query is > shorter than the width of my window minus ~25 characters, which isn't > that likely to be the case IME because users write long queries. > > I don't even try to use \x most of the time because the queries are likely > to be long enough to destroy any benefit, but it all depends on how big your > terminal is and how long your queries are. I pretty much always use less with -S/--chop-long-lines (via $LESS), otherwise I find psql to be pretty hard to use. Greetings, Andres Freund ^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add sub-transaction overflow status in pg_stat_activity @ 2022-11-15 05:23 Dilip Kumar <[email protected]> parent: Andres Freund <[email protected]> 1 sibling, 0 replies; 91+ messages in thread From: Dilip Kumar @ 2022-11-15 05:23 UTC (permalink / raw) To: Andres Freund <[email protected]>; +Cc: Robert Haas <[email protected]>; Tom Lane <[email protected]>; David G. Johnston <[email protected]>; Amit Singh <[email protected]>; Julien Rouhaud <[email protected]>; Bossart, Nathan <[email protected]>; Ashutosh Sharma <[email protected]>; Justin Pryzby <[email protected]>; PostgreSQL Hackers <[email protected]> On Tue, Nov 15, 2022 at 2:47 AM Andres Freund <[email protected]> wrote: > > First, it's not good to have a cliff that you can't see coming - presumbly > you'd want to warn *before* you regularly reach PGPROC_MAX_CACHED_SUBXIDS > subxids, rather when the shit has hit the fan already. I agree with the point that it is good to have a way to know that the problem is about to happen. So for that reason, we should show the subtransaction count. With showing count user can exactly know if there are some sessions that could create problems in near future and may take some action before the problem actually happens. > IMO the number matters a lot when analyzing why this is happening / how to > react. A session occasionally reaching 65 subxids might be tolerable and not > necessarily indicative of a bug. But 100k subxids is something that one just > can't accept. Actually, we will see the problem as soon as it has crossed 64 because after that for any visibility checking we need to check the SLRU. So I feel both count and overflow are important. Count to know that we are heading towards overflow and overflow to know that it has already happened. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add sub-transaction overflow status in pg_stat_activity @ 2022-11-15 05:25 Dilip Kumar <[email protected]> parent: David G. Johnston <[email protected]> 1 sibling, 0 replies; 91+ messages in thread From: Dilip Kumar @ 2022-11-15 05:25 UTC (permalink / raw) To: David G. Johnston <[email protected]>; +Cc: Robert Haas <[email protected]>; Amit Singh <[email protected]>; Andres Freund <[email protected]>; Tom Lane <[email protected]>; Julien Rouhaud <[email protected]>; Bossart, Nathan <[email protected]>; Ashutosh Sharma <[email protected]>; Justin Pryzby <[email protected]>; PostgreSQL Hackers <[email protected]> On Mon, Nov 14, 2022 at 10:18 PM David G. Johnston <[email protected]> wrote: >> Do you have a view on this point? >> > > NULL when overflowed seems like the opposite of the desired effect, calling attention to the exceptional status. Make it a text column and write "overflow" or "###" as appropriate. Anyone using the column is going to end up wanting to special-case overflow anyway and number-to-text conversion aside from overflow is simple enough if a number, and not just a display label, is needed. +1, if we are interested to add only one column then this could be the best way to show. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add sub-transaction overflow status in pg_stat_activity @ 2022-11-15 14:04 Robert Haas <[email protected]> parent: Andres Freund <[email protected]> 1 sibling, 2 replies; 91+ messages in thread From: Robert Haas @ 2022-11-15 14:04 UTC (permalink / raw) To: Andres Freund <[email protected]>; +Cc: Tom Lane <[email protected]>; David G. Johnston <[email protected]>; Amit Singh <[email protected]>; Julien Rouhaud <[email protected]>; Bossart, Nathan <[email protected]>; Dilip Kumar <[email protected]>; Ashutosh Sharma <[email protected]>; Justin Pryzby <[email protected]>; PostgreSQL Hackers <[email protected]> On Mon, Nov 14, 2022 at 4:17 PM Andres Freund <[email protected]> wrote: > Perhaps this would better be tackled by a new "visibility" view. It could show > - number of sessions with a snapshot > - max age of backend xmin > - pid with max backend xmin > - number of sessions that suboverflowed > - pid of the session with the most subxids > - age of the oldest prepared xact > - age of the oldest slot > - age of the oldest walsender > - ... > > Perhaps implemented in SQL, with new functions for accessing the properties we > don't expose today. That'd address the pg_stat_activity width, while still > allowing very granular access when necessary. And provide insight into > something that's way to hard to query right now. I wouldn't be against a pg_stat_visibility view, but I don't think I'd want it to just output a single summary row. I think we really need to give people an easy way to track down which session is the problem; the existence of the problem is already obvious from the SLRU-related wait events. If we moved backend_xid and backend_xmin out to this new view, added these subtransaction-related things, and allowed for a join on pid, I could get behind that, but it's probably a bit more painful for users than just accepting that the view is going to further outgrow the terminal window. It might be better in the long term because perhaps we're going to find more things that would fit into this new view, but I don't know. -- Robert Haas EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add sub-transaction overflow status in pg_stat_activity @ 2022-11-15 19:29 Andres Freund <[email protected]> parent: Robert Haas <[email protected]> 1 sibling, 1 reply; 91+ messages in thread From: Andres Freund @ 2022-11-15 19:29 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Tom Lane <[email protected]>; David G. Johnston <[email protected]>; Amit Singh <[email protected]>; Julien Rouhaud <[email protected]>; Bossart, Nathan <[email protected]>; Dilip Kumar <[email protected]>; Ashutosh Sharma <[email protected]>; Justin Pryzby <[email protected]>; PostgreSQL Hackers <[email protected]> Hi, On 2022-11-15 09:04:25 -0500, Robert Haas wrote: > On Mon, Nov 14, 2022 at 4:17 PM Andres Freund <[email protected]> wrote: > > Perhaps this would better be tackled by a new "visibility" view. It could show > > - number of sessions with a snapshot > > - max age of backend xmin > > - pid with max backend xmin > > - number of sessions that suboverflowed > > - pid of the session with the most subxids > > - age of the oldest prepared xact > > - age of the oldest slot > > - age of the oldest walsender > > - ... > > > > Perhaps implemented in SQL, with new functions for accessing the properties we > > don't expose today. That'd address the pg_stat_activity width, while still > > allowing very granular access when necessary. And provide insight into > > something that's way to hard to query right now. > > I wouldn't be against a pg_stat_visibility view, but I don't think I'd > want it to just output a single summary row. I think it'd be more helpful to just have a single row (or maybe a fixed number of rows) - from what I've observed the main problem people have is condensing the available information, rather than not having information available at all. > I think we really need to > give people an easy way to track down which session is the problem; > the existence of the problem is already obvious from the SLRU-related > wait events. Hence the suggestion to show the pid of the session with the most subxacts. We probably also should add a bunch of accessor functions for people that want more detail... But just seeing in one place what's problematic would be the big get, the rest will be a small percentage of users IME. Greetings, Andres Freund ^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add sub-transaction overflow status in pg_stat_activity @ 2022-11-16 10:31 Dilip Kumar <[email protected]> parent: Robert Haas <[email protected]> 1 sibling, 0 replies; 91+ messages in thread From: Dilip Kumar @ 2022-11-16 10:31 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Andres Freund <[email protected]>; Tom Lane <[email protected]>; David G. Johnston <[email protected]>; Amit Singh <[email protected]>; Julien Rouhaud <[email protected]>; Bossart, Nathan <[email protected]>; Ashutosh Sharma <[email protected]>; Justin Pryzby <[email protected]>; PostgreSQL Hackers <[email protected]> On Tue, Nov 15, 2022 at 7:34 PM Robert Haas <[email protected]> wrote: > > On Mon, Nov 14, 2022 at 4:17 PM Andres Freund <[email protected]> wrote: > > Perhaps this would better be tackled by a new "visibility" view. It could show > > - number of sessions with a snapshot > > - max age of backend xmin > > - pid with max backend xmin > > - number of sessions that suboverflowed > > - pid of the session with the most subxids > > - age of the oldest prepared xact > > - age of the oldest slot > > - age of the oldest walsender > > - ... > > > > Perhaps implemented in SQL, with new functions for accessing the properties we > > don't expose today. That'd address the pg_stat_activity width, while still > > allowing very granular access when necessary. And provide insight into > > something that's way to hard to query right now. > > I wouldn't be against a pg_stat_visibility view, but I don't think I'd > want it to just output a single summary row. I think we really need to > give people an easy way to track down which session is the problem; > the existence of the problem is already obvious from the SLRU-related > wait events. > Even I feel per backend-wise information would be more useful and easy to use instead of a single summary row. I think It's fine to create a new view if we do not want to add more members to the existing view. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add sub-transaction overflow status in pg_stat_activity @ 2022-11-16 17:59 Robert Haas <[email protected]> parent: Andres Freund <[email protected]> 0 siblings, 0 replies; 91+ messages in thread From: Robert Haas @ 2022-11-16 17:59 UTC (permalink / raw) To: Andres Freund <[email protected]>; +Cc: Tom Lane <[email protected]>; David G. Johnston <[email protected]>; Amit Singh <[email protected]>; Julien Rouhaud <[email protected]>; Bossart, Nathan <[email protected]>; Dilip Kumar <[email protected]>; Ashutosh Sharma <[email protected]>; Justin Pryzby <[email protected]>; PostgreSQL Hackers <[email protected]> On Tue, Nov 15, 2022 at 2:29 PM Andres Freund <[email protected]> wrote: > Hence the suggestion to show the pid of the session with the most subxacts. We > probably also should add a bunch of accessor functions for people that want > more detail... But just seeing in one place what's problematic would be the > big get, the rest will be a small percentage of users IME. I guess all I can say here is that my experience differs. -- Robert Haas EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add sub-transaction overflow status in pg_stat_activity @ 2022-11-23 19:01 Bruce Momjian <[email protected]> parent: Robert Haas <[email protected]> 3 siblings, 1 reply; 91+ messages in thread From: Bruce Momjian @ 2022-11-23 19:01 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Andres Freund <[email protected]>; Tom Lane <[email protected]>; Julien Rouhaud <[email protected]>; Bossart, Nathan <[email protected]>; Dilip Kumar <[email protected]>; Ashutosh Sharma <[email protected]>; Justin Pryzby <[email protected]>; PostgreSQL Hackers <[email protected]> On Mon, Nov 14, 2022 at 10:09:57AM -0500, Robert Haas wrote: > I think I fundamentally disagree with the idea that we should refuse > to expose instrumentation data because some day the internals might > change. If we accepted that argument categorically, we wouldn't have > things like backend_xmin or backend_xid in pg_stat_activity, or wait > events either, but we do have those things and users find them useful. > They suck in the sense that you need to know quite a bit about how the > internals work in order to use them to find problems, but people who > want to support production PostgreSQL instances have to learn about > how those internals work one way or the other because they > demonstrably matter. It is absolutely stellar when we can say "hey, we I originally thought having this value in pg_stat_activity was overkill, but seeing the other internal/warning columns in that view, I think it makes sense. Oddly, is our 64 snapshot performance limit even documented anywhere? I know it is in Simon's patch I am working on. -- Bruce Momjian <[email protected]> https://momjian.us EDB https://enterprisedb.com Indecision is a decision. Inaction is an action. Mark Batterson ^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add sub-transaction overflow status in pg_stat_activity @ 2022-11-23 20:25 Robert Haas <[email protected]> parent: Bruce Momjian <[email protected]> 0 siblings, 1 reply; 91+ messages in thread From: Robert Haas @ 2022-11-23 20:25 UTC (permalink / raw) To: Bruce Momjian <[email protected]>; +Cc: Andres Freund <[email protected]>; Tom Lane <[email protected]>; Julien Rouhaud <[email protected]>; Bossart, Nathan <[email protected]>; Dilip Kumar <[email protected]>; Ashutosh Sharma <[email protected]>; Justin Pryzby <[email protected]>; PostgreSQL Hackers <[email protected]> On Wed, Nov 23, 2022 at 2:01 PM Bruce Momjian <[email protected]> wrote: > I originally thought having this value in pg_stat_activity was overkill, > but seeing the other internal/warning columns in that view, I think it > makes sense. Oddly, is our 64 snapshot performance limit even > documented anywhere? I know it is in Simon's patch I am working on. If it is, I'm not aware of it. We often don't document things that are as internal as that. One thing that I'd really like to see better documented is exactly what it is that causes a problem. But first we'd have to understand it ourselves. It's not as simple as "if you have more than 64 subxacts in any top-level xact, kiss performance good-bye!" because for there to be a problem, at least one backend (and probably many) have to take snapshots that include that see that overflowed subxact cache and thus get marked suboverflowed. Then after that, those snapshots have to be used often enough that the additional visibility-checking cost becomes a problem. But it's also not good enough to just use those snapshots against any old tuples, because tuples that are older than the snapshot's xmin aren't going to cause additional lookups, nor are tuples newer than the snapshot's xmax. So it feels a bit complicated to me to think through the workload where this really hurts. What I'm imagining is that you need a relatively long-running transaction that overflows its subxact limitation but then doesn't commit, so that lots of other backends get overflowed snapshots, and also so that the xmin and xmax of the snapshots being taken get further apart. Or maybe you can have a series short-running transactions that each overflow their subxact cache briefly, but they overlap, so that there's usually at least 1 around in that state, but in that case I think you need a separate long-running transaction to push xmin and xmax further apart. Either way, the backends that get the overflowed snapshots then need to go look at some table data that's been recently modified, so that there are xmin and xmax values newer than the snapshot's xmin. Intuitively, I feel like this should be pretty rare, and largely avoidable if you just don't use long-running transactions, which is a good thing to avoid for other reasons anyway. But there may be more to it than I'm realizing, because I've seen customers hit this issue multiple times. I wonder whether there's some subtlety to the triggering conditions that I'm not fully understanding. -- Robert Haas EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add sub-transaction overflow status in pg_stat_activity @ 2022-11-23 20:56 Andres Freund <[email protected]> parent: Robert Haas <[email protected]> 0 siblings, 2 replies; 91+ messages in thread From: Andres Freund @ 2022-11-23 20:56 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Bruce Momjian <[email protected]>; Tom Lane <[email protected]>; Julien Rouhaud <[email protected]>; Bossart, Nathan <[email protected]>; Dilip Kumar <[email protected]>; Ashutosh Sharma <[email protected]>; Justin Pryzby <[email protected]>; PostgreSQL Hackers <[email protected]> Hi, On 2022-11-23 15:25:39 -0500, Robert Haas wrote: > One thing that I'd really like to see better documented is exactly > what it is that causes a problem. But first we'd have to understand it > ourselves. It's not as simple as "if you have more than 64 subxacts in > any top-level xact, kiss performance good-bye!" because for there to > be a problem, at least one backend (and probably many) have to take > snapshots that include that see that overflowed subxact cache and thus > get marked suboverflowed. Then after that, those snapshots have to be > used often enough that the additional visibility-checking cost becomes > a problem. But it's also not good enough to just use those snapshots > against any old tuples, because tuples that are older than the > snapshot's xmin aren't going to cause additional lookups, nor are > tuples newer than the snapshot's xmax. Indeed. This is why I was thinking that just alerting for overflowed xact isn't particularly helpful. You really want to see how much they overflow and how often. But even that might not be that helpful. Perhaps what we actually need is an aggregate measure showing the time spent doing subxact lookups due to overflowed snapshots? Seeing a substantial amount of time spent doing subxact lookups would be much more accurate call to action than seeing a that some sessions have a lot of subxacts. > Intuitively, I feel like this should be pretty rare, and largely > avoidable if you just don't use long-running transactions, which is a > good thing to avoid for other reasons anyway. I think they're just not always avoidable, even in a very well operated system. I wonder if we could lower the impact of suboverflowed snapshots by improving the representation in PGPROC and SnapshotData. What if we a) Recorded the min and max assigned subxid in PGPROC b) Instead of giving up in GetSnapshotData() once we see a suboverflowed PGPROC, store the min/max subxid of the proc in SnapshotData. We could reliably "steal" space for that from ->subxip, as we won't need to store subxids for that proc. c) When determining visibility with a suboverflowed snapshot we use the ranges from b) to check whether we need to do a subtrans lookup. I think that'll often prevent subtrans lookups. d) If we encounter a subxid whose parent is in progress and not in ->subxid, and subxcnt isn't the max, add that subxid to subxip. That's not free because we'd basically need to do an insertion sort, but likely still a lot cheaper than doing repeated subtrans lookups. I think we'd just need a one or two additional fields in SnapshotData. Greetings, Andres Freund ^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add sub-transaction overflow status in pg_stat_activity @ 2022-11-25 04:01 Dilip Kumar <[email protected]> parent: Andres Freund <[email protected]> 1 sibling, 0 replies; 91+ messages in thread From: Dilip Kumar @ 2022-11-25 04:01 UTC (permalink / raw) To: Andres Freund <[email protected]>; +Cc: Robert Haas <[email protected]>; Bruce Momjian <[email protected]>; Tom Lane <[email protected]>; Julien Rouhaud <[email protected]>; Bossart, Nathan <[email protected]>; Ashutosh Sharma <[email protected]>; Justin Pryzby <[email protected]>; PostgreSQL Hackers <[email protected]> On Thu, Nov 24, 2022 at 2:26 AM Andres Freund <[email protected]> wrote: > > Indeed. This is why I was thinking that just alerting for overflowed xact > isn't particularly helpful. You really want to see how much they overflow and > how often. I think the way of monitoring the subtransaction count and overflow status is helpful at least for troubleshooting purposes. By regularly monitoring user will know which backend(pid) is particularly using more subtransactions and prone to overflow and which backends are actually frequently causing sub-overflow. > I think they're just not always avoidable, even in a very well operated > system. > > > I wonder if we could lower the impact of suboverflowed snapshots by improving > the representation in PGPROC and SnapshotData. What if we > > a) Recorded the min and max assigned subxid in PGPROC > > b) Instead of giving up in GetSnapshotData() once we see a suboverflowed > PGPROC, store the min/max subxid of the proc in SnapshotData. We could > reliably "steal" space for that from ->subxip, as we won't need to store > subxids for that proc. > > c) When determining visibility with a suboverflowed snapshot we use the > ranges from b) to check whether we need to do a subtrans lookup. I think > that'll often prevent subtrans lookups. > > d) If we encounter a subxid whose parent is in progress and not in ->subxid, > and subxcnt isn't the max, add that subxid to subxip. That's not free > because we'd basically need to do an insertion sort, but likely still a lot > cheaper than doing repeated subtrans lookups. > > I think we'd just need a one or two additional fields in SnapshotData. +1 I think this approach will be helpful in many cases, especially when only some of the backend is creating sub-overflow and impacting overall system performance. Now, most of the xids especially the top xid will not fall in that range (unless that sub-overflowing backend is constantly generating subxids and increasing its range) and the lookups for that xids can be done directly in the snapshot's xip array. On another thought, in XidInMVCCSnapshot() in case of sub-overflow why don't we look into the snapshot's xip array first and see if the xid exists there? if not then we can look into the pg_subtrans SLRU and fetch the top xid and relook again into the xip array. It will be more costly in cases where we do not find xid in the xip array because then we will have to search this array twice but I think looking into this array is much cheaper than directly accessing SLRU. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add sub-transaction overflow status in pg_stat_activity @ 2022-11-30 16:01 Robert Haas <[email protected]> parent: Andres Freund <[email protected]> 1 sibling, 1 reply; 91+ messages in thread From: Robert Haas @ 2022-11-30 16:01 UTC (permalink / raw) To: Andres Freund <[email protected]>; +Cc: Bruce Momjian <[email protected]>; Tom Lane <[email protected]>; Julien Rouhaud <[email protected]>; Bossart, Nathan <[email protected]>; Dilip Kumar <[email protected]>; Ashutosh Sharma <[email protected]>; Justin Pryzby <[email protected]>; PostgreSQL Hackers <[email protected]> On Wed, Nov 23, 2022 at 3:56 PM Andres Freund <[email protected]> wrote: > Indeed. This is why I was thinking that just alerting for overflowed xact > isn't particularly helpful. You really want to see how much they overflow and > how often. I think if we just expose the is-overflowed feld and the count, people can poll. It works fine for wait events and I think it's fine here, too. > But even that might not be that helpful. Perhaps what we actually need is an > aggregate measure showing the time spent doing subxact lookups due to > overflowed snapshots? Seeing a substantial amount of time spent doing subxact > lookups would be much more accurate call to action than seeing a that some > sessions have a lot of subxacts. That's not responsive to the need that I have. I need users to be able to figure out which backend(s) are overflowing their snapshots -- and perhaps how badly and how often --- not which backends are incurring an expense as a result. There may well be a use case for the latter thing but it's a different problem. > I wonder if we could lower the impact of suboverflowed snapshots by improving > the representation in PGPROC and SnapshotData. What if we > > a) Recorded the min and max assigned subxid in PGPROC > > b) Instead of giving up in GetSnapshotData() once we see a suboverflowed > PGPROC, store the min/max subxid of the proc in SnapshotData. We could > reliably "steal" space for that from ->subxip, as we won't need to store > subxids for that proc. > > c) When determining visibility with a suboverflowed snapshot we use the > ranges from b) to check whether we need to do a subtrans lookup. I think > that'll often prevent subtrans lookups. Wouldn't you basically need to take the union of all the ranges, probably by keeping the lowest min and the highest max? I'm not sure how much that would really help at that point. -- Robert Haas EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add sub-transaction overflow status in pg_stat_activity @ 2022-12-12 16:15 Robert Haas <[email protected]> parent: Robert Haas <[email protected]> 0 siblings, 1 reply; 91+ messages in thread From: Robert Haas @ 2022-12-12 16:15 UTC (permalink / raw) To: Andres Freund <[email protected]>; +Cc: Bruce Momjian <[email protected]>; Tom Lane <[email protected]>; Julien Rouhaud <[email protected]>; Bossart, Nathan <[email protected]>; Dilip Kumar <[email protected]>; Ashutosh Sharma <[email protected]>; Justin Pryzby <[email protected]>; PostgreSQL Hackers <[email protected]> On Wed, Nov 30, 2022 at 11:01 AM Robert Haas <[email protected]> wrote: > That's not responsive to the need that I have. I need users to be able > to figure out which backend(s) are overflowing their snapshots -- and > perhaps how badly and how often --- not which backends are incurring > an expense as a result. There may well be a use case for the latter > thing but it's a different problem. So ... I want to go ahead and commit Dilip's v4 patch, or something very like it. Most people were initially supportive. Tom expressed some opposition, but it sounds like that was mostly to the discussion going on and on rather than the idea per se. Andres also expressed some concerns, but I really think the problem he's worried about is something slightly different and need not block this work. I note also that the v4 patch is designed in such a way that it does not change any view definitions, so the compatibility impact of committing it is basically nil. Any strenuous objections? -- Robert Haas EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add sub-transaction overflow status in pg_stat_activity @ 2022-12-12 17:33 Nathan Bossart <[email protected]> parent: Robert Haas <[email protected]> 0 siblings, 1 reply; 91+ messages in thread From: Nathan Bossart @ 2022-12-12 17:33 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Andres Freund <[email protected]>; Bruce Momjian <[email protected]>; Tom Lane <[email protected]>; Julien Rouhaud <[email protected]>; Bossart, Nathan <[email protected]>; Dilip Kumar <[email protected]>; Ashutosh Sharma <[email protected]>; Justin Pryzby <[email protected]>; PostgreSQL Hackers <[email protected]> On Mon, Dec 12, 2022 at 11:15:43AM -0500, Robert Haas wrote: > Any strenuous objections? Nope. In fact, +1. Until more work is done to alleviate the performance issues, this information will likely prove useful. -- Nathan Bossart Amazon Web Services: https://aws.amazon.com ^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add sub-transaction overflow status in pg_stat_activity @ 2022-12-12 17:42 Justin Pryzby <[email protected]> parent: Nathan Bossart <[email protected]> 0 siblings, 1 reply; 91+ messages in thread From: Justin Pryzby @ 2022-12-12 17:42 UTC (permalink / raw) To: Nathan Bossart <[email protected]>; +Cc: Robert Haas <[email protected]>; Andres Freund <[email protected]>; Bruce Momjian <[email protected]>; Tom Lane <[email protected]>; Julien Rouhaud <[email protected]>; Bossart, Nathan <[email protected]>; Dilip Kumar <[email protected]>; Ashutosh Sharma <[email protected]>; [email protected] On Mon, Dec 12, 2022 at 09:33:51AM -0800, Nathan Bossart wrote: > On Mon, Dec 12, 2022 at 11:15:43AM -0500, Robert Haas wrote: > > Any strenuous objections? > > Nope. In fact, +1. Until more work is done to alleviate the performance > issues, this information will likely prove useful. The docs could use a bit of attention. Otherwise +1. diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml index 4efa1d5fca0..ac15e2ce789 100644 --- a/doc/src/sgml/monitoring.sgml +++ b/doc/src/sgml/monitoring.sgml @@ -5680,12 +5680,12 @@ FROM pg_stat_get_backend_idset() AS backendid; <returnvalue>record</returnvalue> </para> <para> - Returns a record of information about the backend's subtransactions. - The fields returned are <parameter>subxact_count</parameter> identifies - number of active subtransaction and <parameter>subxact_overflow - </parameter> shows whether the backend's subtransaction cache is - overflowed or not. - </para></entry> + Returns a record of information about the subtransactions of the backend + with the specified ID. + The fields returned are <parameter>subxact_count</parameter>, which + identifies the number of active subtransaction and + <parameter>subxact_overflow</parameter>, which shows whether the + backend's subtransaction cache is overflowed or not. </para></entry> </row> ^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add sub-transaction overflow status in pg_stat_activity @ 2022-12-12 17:51 Robert Haas <[email protected]> parent: Justin Pryzby <[email protected]> 0 siblings, 1 reply; 91+ messages in thread From: Robert Haas @ 2022-12-12 17:51 UTC (permalink / raw) To: Justin Pryzby <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Bruce Momjian <[email protected]>; Tom Lane <[email protected]>; Julien Rouhaud <[email protected]>; Bossart, Nathan <[email protected]>; Dilip Kumar <[email protected]>; Ashutosh Sharma <[email protected]>; [email protected] On Mon, Dec 12, 2022 at 12:42 PM Justin Pryzby <[email protected]> wrote: > diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml > index 4efa1d5fca0..ac15e2ce789 100644 > --- a/doc/src/sgml/monitoring.sgml > +++ b/doc/src/sgml/monitoring.sgml > @@ -5680,12 +5680,12 @@ FROM pg_stat_get_backend_idset() AS backendid; > <returnvalue>record</returnvalue> > </para> > <para> > - Returns a record of information about the backend's subtransactions. > - The fields returned are <parameter>subxact_count</parameter> identifies > - number of active subtransaction and <parameter>subxact_overflow > - </parameter> shows whether the backend's subtransaction cache is > - overflowed or not. > - </para></entry> > + Returns a record of information about the subtransactions of the backend > + with the specified ID. > + The fields returned are <parameter>subxact_count</parameter>, which > + identifies the number of active subtransaction and > + <parameter>subxact_overflow</parameter>, which shows whether the > + backend's subtransaction cache is overflowed or not. > </para></entry> > </row> Makes sense. -- Robert Haas EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add sub-transaction overflow status in pg_stat_activity @ 2022-12-13 04:09 Dilip Kumar <[email protected]> parent: Robert Haas <[email protected]> 0 siblings, 1 reply; 91+ messages in thread From: Dilip Kumar @ 2022-12-13 04:09 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Justin Pryzby <[email protected]>; Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Bruce Momjian <[email protected]>; Tom Lane <[email protected]>; Julien Rouhaud <[email protected]>; Bossart, Nathan <[email protected]>; Ashutosh Sharma <[email protected]>; [email protected] On Mon, Dec 12, 2022 at 11:21 PM Robert Haas <[email protected]> wrote: > > On Mon, Dec 12, 2022 at 12:42 PM Justin Pryzby <[email protected]> wrote: > > diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml > > index 4efa1d5fca0..ac15e2ce789 100644 > > --- a/doc/src/sgml/monitoring.sgml > > +++ b/doc/src/sgml/monitoring.sgml > > @@ -5680,12 +5680,12 @@ FROM pg_stat_get_backend_idset() AS backendid; > > <returnvalue>record</returnvalue> > > </para> > > <para> > > - Returns a record of information about the backend's subtransactions. > > - The fields returned are <parameter>subxact_count</parameter> identifies > > - number of active subtransaction and <parameter>subxact_overflow > > - </parameter> shows whether the backend's subtransaction cache is > > - overflowed or not. > > - </para></entry> > > + Returns a record of information about the subtransactions of the backend > > + with the specified ID. > > + The fields returned are <parameter>subxact_count</parameter>, which > > + identifies the number of active subtransaction and > > + <parameter>subxact_overflow</parameter>, which shows whether the > > + backend's subtransaction cache is overflowed or not. > > </para></entry> > > </row> > > Makes sense. +1 -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add sub-transaction overflow status in pg_stat_activity @ 2022-12-13 07:29 Julien Rouhaud <[email protected]> parent: Dilip Kumar <[email protected]> 0 siblings, 1 reply; 91+ messages in thread From: Julien Rouhaud @ 2022-12-13 07:29 UTC (permalink / raw) To: Dilip Kumar <[email protected]>; +Cc: Robert Haas <[email protected]>; Justin Pryzby <[email protected]>; Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Bruce Momjian <[email protected]>; Tom Lane <[email protected]>; Bossart, Nathan <[email protected]>; Ashutosh Sharma <[email protected]>; [email protected] On Tue, Dec 13, 2022 at 5:09 AM Dilip Kumar <[email protected]> wrote: > > On Mon, Dec 12, 2022 at 11:21 PM Robert Haas <[email protected]> wrote: > > > > On Mon, Dec 12, 2022 at 12:42 PM Justin Pryzby <[email protected]> wrote: > > > diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml > > > index 4efa1d5fca0..ac15e2ce789 100644 > > > --- a/doc/src/sgml/monitoring.sgml > > > +++ b/doc/src/sgml/monitoring.sgml > > > @@ -5680,12 +5680,12 @@ FROM pg_stat_get_backend_idset() AS backendid; > > > <returnvalue>record</returnvalue> > > > </para> > > > <para> > > > - Returns a record of information about the backend's subtransactions. > > > - The fields returned are <parameter>subxact_count</parameter> identifies > > > - number of active subtransaction and <parameter>subxact_overflow > > > - </parameter> shows whether the backend's subtransaction cache is > > > - overflowed or not. > > > - </para></entry> > > > + Returns a record of information about the subtransactions of the backend > > > + with the specified ID. > > > + The fields returned are <parameter>subxact_count</parameter>, which > > > + identifies the number of active subtransaction and > > > + <parameter>subxact_overflow</parameter>, which shows whether the > > > + backend's subtransaction cache is overflowed or not. > > > </para></entry> > > > </row> > > > > Makes sense. > > +1 +1 ^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add sub-transaction overflow status in pg_stat_activity @ 2022-12-19 19:56 Robert Haas <[email protected]> parent: Julien Rouhaud <[email protected]> 0 siblings, 1 reply; 91+ messages in thread From: Robert Haas @ 2022-12-19 19:56 UTC (permalink / raw) To: Julien Rouhaud <[email protected]>; +Cc: Dilip Kumar <[email protected]>; Justin Pryzby <[email protected]>; Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Bruce Momjian <[email protected]>; Tom Lane <[email protected]>; Bossart, Nathan <[email protected]>; Ashutosh Sharma <[email protected]>; [email protected] On Tue, Dec 13, 2022 at 2:29 AM Julien Rouhaud <[email protected]> wrote: > > > Makes sense. > > > > +1 > > +1 Committed with a bit more word-smithing on the documentation. -- Robert Haas EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add sub-transaction overflow status in pg_stat_activity @ 2022-12-19 20:48 Ted Yu <[email protected]> parent: Robert Haas <[email protected]> 0 siblings, 1 reply; 91+ messages in thread From: Ted Yu @ 2022-12-19 20:48 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Julien Rouhaud <[email protected]>; Dilip Kumar <[email protected]>; Justin Pryzby <[email protected]>; Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Bruce Momjian <[email protected]>; Tom Lane <[email protected]>; Bossart, Nathan <[email protected]>; Ashutosh Sharma <[email protected]>; [email protected] On Mon, Dec 19, 2022 at 11:57 AM Robert Haas <[email protected]> wrote: > On Tue, Dec 13, 2022 at 2:29 AM Julien Rouhaud <[email protected]> wrote: > > > > Makes sense. > > > > > > +1 > > > > +1 > > Committed with a bit more word-smithing on the documentation. > > -- > Robert Haas > EDB: http://www.enterprisedb.com > > Hi, It seems the comment for `backend_subxact_overflowed` missed a word. Please see the patch. Thanks Attachments: [application/octet-stream] subtxn-number-comment.patch (577B, ../../CALte62wkFB05=RTWf7BL_6MfWs2=DY=ai-K7LWn_+0TJUuPJ2w@mail.gmail.com/3-subtxn-number-comment.patch) download | inline diff: diff --git a/src/include/utils/backend_status.h b/src/include/utils/backend_status.h index e076adcaa8..8a7cce7ef5 100644 --- a/src/include/utils/backend_status.h +++ b/src/include/utils/backend_status.h @@ -273,8 +273,8 @@ typedef struct LocalPgBackendStatus int backend_subxact_count; /* - * The number of subtransactions in the current session exceeded the cached - * subtransaction limit. + * The number of subtransactions in the current session which exceeded the + * cached subtransaction limit. */ bool backend_subxact_overflowed; } LocalPgBackendStatus; ^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add sub-transaction overflow status in pg_stat_activity @ 2022-12-19 21:02 Robert Haas <[email protected]> parent: Ted Yu <[email protected]> 0 siblings, 1 reply; 91+ messages in thread From: Robert Haas @ 2022-12-19 21:02 UTC (permalink / raw) To: Ted Yu <[email protected]>; +Cc: Julien Rouhaud <[email protected]>; Dilip Kumar <[email protected]>; Justin Pryzby <[email protected]>; Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Bruce Momjian <[email protected]>; Tom Lane <[email protected]>; Bossart, Nathan <[email protected]>; Ashutosh Sharma <[email protected]>; [email protected] On Mon, Dec 19, 2022 at 3:48 PM Ted Yu <[email protected]> wrote: > It seems the comment for `backend_subxact_overflowed` missed a word. > > Please see the patch. Committed this fix, thanks. -- Robert Haas EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add sub-transaction overflow status in pg_stat_activity @ 2022-12-20 04:23 Dilip Kumar <[email protected]> parent: Robert Haas <[email protected]> 0 siblings, 1 reply; 91+ messages in thread From: Dilip Kumar @ 2022-12-20 04:23 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Ted Yu <[email protected]>; Julien Rouhaud <[email protected]>; Justin Pryzby <[email protected]>; Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Bruce Momjian <[email protected]>; Tom Lane <[email protected]>; Bossart, Nathan <[email protected]>; Ashutosh Sharma <[email protected]>; [email protected] On Tue, Dec 20, 2022 at 2:32 AM Robert Haas <[email protected]> wrote: > > On Mon, Dec 19, 2022 at 3:48 PM Ted Yu <[email protected]> wrote: > > It seems the comment for `backend_subxact_overflowed` missed a word. > > > > Please see the patch. > > Committed this fix, thanks. Thanks, Robert! -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 91+ messages in thread
* Re: Add sub-transaction overflow status in pg_stat_activity @ 2023-02-08 09:10 Kirill Reshke <[email protected]> parent: Dilip Kumar <[email protected]> 0 siblings, 0 replies; 91+ messages in thread From: Kirill Reshke @ 2023-02-08 09:10 UTC (permalink / raw) To: Dilip Kumar <[email protected]>; +Cc: Robert Haas <[email protected]>; Ted Yu <[email protected]>; Julien Rouhaud <[email protected]>; Justin Pryzby <[email protected]>; Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Bruce Momjian <[email protected]>; Tom Lane <[email protected]>; Bossart, Nathan <[email protected]>; Ashutosh Sharma <[email protected]>; [email protected] On Tue, 20 Dec 2022 at 09:23, Dilip Kumar <[email protected]> wrote: > > On Tue, Dec 20, 2022 at 2:32 AM Robert Haas <[email protected]> wrote: > > > > On Mon, Dec 19, 2022 at 3:48 PM Ted Yu <[email protected]> wrote: > > > It seems the comment for `backend_subxact_overflowed` missed a word. > > > > > > Please see the patch. > > > > Committed this fix, thanks. > > Thanks, Robert! > > -- > Regards, > Dilip Kumar > EnterpriseDB: http://www.enterprisedb.com > > Hi hackers! Nice patch, seems it may be useful in cases like alerting that subxid overflow happened is database or whatever. But I'm curious, what is the following work on this? I think it may be way more helpful to, for example, log queries, causing sub-tx overflow, or even kill the backend, causing sub-tx overflow with GUC variables, setting server behaviour. For example, in Greenplum there is gp_subtransaction_overflow extension and GUC for simply logging problematic queries[1]. Can we have something similar in PostgreSQL on the server-side? [1] https://github.com/greenplum-db/gpdb/blob/6X_STABLE/gpcontrib/gp_subtransaction_overflow/gp_subtrans... ^ permalink raw reply [nested|flat] 91+ messages in thread
end of thread, other threads:[~2023-02-08 09:10 UTC | newest] Thread overview: 91+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2020-05-11 07:50 [PATCH v38 2/6] Introduce UniqueKey attributes on RelOptInfo struct. 一挃 <[email protected]> 2020-05-11 07:50 [PATCH v38 2/6] Introduce UniqueKey attributes on RelOptInfo struct. 一挃 <[email protected]> 2020-05-11 07:50 [PATCH v38 2/6] Introduce UniqueKey attributes on RelOptInfo struct. 一挃 <[email protected]> 2020-05-11 07:50 [PATCH v36 2/6] Introduce UniqueKey attributes on RelOptInfo struct. 一挃 <[email protected]> 2020-05-11 07:50 [PATCH v36 2/6] Introduce UniqueKey attributes on RelOptInfo struct. 一挃 <[email protected]> 2020-05-11 07:50 [PATCH v38 2/6] Introduce UniqueKey attributes on RelOptInfo struct. 一挃 <[email protected]> 2020-05-11 07:50 [PATCH v38 2/6] Introduce UniqueKey attributes on RelOptInfo struct. 一挃 <[email protected]> 2020-05-11 07:50 [PATCH v36 2/6] Introduce UniqueKey attributes on RelOptInfo struct. 一挃 <[email protected]> 2020-05-11 07:50 [PATCH v36 2/6] Introduce UniqueKey attributes on RelOptInfo struct. 一挃 <[email protected]> 2020-05-11 07:50 [PATCH v36 2/6] Introduce UniqueKey attributes on RelOptInfo struct. 一挃 <[email protected]> 2020-05-11 07:50 [PATCH 03/10] Introduce UniqueKey attributes on RelOptInfo struct. 一挃 <[email protected]> 2020-05-11 07:50 [PATCH v36 2/6] Introduce UniqueKey attributes on RelOptInfo struct. 一挃 <[email protected]> 2020-05-11 07:50 [PATCH v38 2/6] Introduce UniqueKey attributes on RelOptInfo struct. 一挃 <[email protected]> 2020-05-11 07:50 [PATCH v38 2/6] Introduce UniqueKey attributes on RelOptInfo struct. 一挃 <[email protected]> 2020-05-11 07:50 [PATCH v36 2/6] Introduce UniqueKey attributes on RelOptInfo struct. 一挃 <[email protected]> 2020-05-11 07:50 [PATCH v38 2/6] Introduce UniqueKey attributes on RelOptInfo struct. 一挃 <[email protected]> 2020-05-11 07:50 [PATCH v38 2/6] Introduce UniqueKey attributes on RelOptInfo struct. 一挃 <[email protected]> 2020-05-11 07:50 [PATCH v38 2/6] Introduce UniqueKey attributes on RelOptInfo struct. 一挃 <[email protected]> 2020-05-11 07:50 [PATCH v38 2/6] Introduce UniqueKey attributes on RelOptInfo struct. 一挃 <[email protected]> 2020-05-11 07:50 [PATCH v36 2/6] Introduce UniqueKey attributes on RelOptInfo struct. 一挃 <[email protected]> 2020-05-11 07:50 [PATCH v36 2/6] Introduce UniqueKey attributes on RelOptInfo struct. 一挃 <[email protected]> 2020-05-11 07:50 [PATCH v38 2/6] Introduce UniqueKey attributes on RelOptInfo struct. 一挃 <[email protected]> 2020-05-11 07:50 [PATCH v38 2/6] Introduce UniqueKey attributes on RelOptInfo struct. 一挃 <[email protected]> 2020-05-11 07:50 [PATCH v38 2/6] Introduce UniqueKey attributes on RelOptInfo struct. 一挃 <[email protected]> 2020-05-11 07:50 [PATCH v38 2/6] Introduce UniqueKey attributes on RelOptInfo struct. 一挃 <[email protected]> 2020-05-11 07:50 [PATCH v36 2/6] Introduce UniqueKey attributes on RelOptInfo struct. 一挃 <[email protected]> 2020-05-11 07:50 [PATCH v38 2/6] Introduce UniqueKey attributes on RelOptInfo struct. 一挃 <[email protected]> 2020-05-11 07:50 [PATCH v38 2/6] Introduce UniqueKey attributes on RelOptInfo struct. 一挃 <[email protected]> 2020-05-11 07:50 [PATCH v36 2/6] Introduce UniqueKey attributes on RelOptInfo struct. 一挃 <[email protected]> 2020-05-11 07:50 [PATCH 03/10] Introduce UniqueKey attributes on RelOptInfo struct. 一挃 <[email protected]> 2020-05-11 07:50 [PATCH v36 2/6] Introduce UniqueKey attributes on RelOptInfo struct. 一挃 <[email protected]> 2020-05-11 07:50 [PATCH v38 2/6] Introduce UniqueKey attributes on RelOptInfo struct. 一挃 <[email protected]> 2020-05-11 07:50 [PATCH v36 2/6] Introduce UniqueKey attributes on RelOptInfo struct. 一挃 <[email protected]> 2020-05-11 07:50 [PATCH v36 2/6] Introduce UniqueKey attributes on RelOptInfo struct. 一挃 <[email protected]> 2020-05-11 07:50 [PATCH v36 2/6] Introduce UniqueKey attributes on RelOptInfo struct. 一挃 <[email protected]> 2020-05-11 07:50 [PATCH v37 2/6] Introduce UniqueKey attributes on RelOptInfo struct. 一挃 <[email protected]> 2020-05-11 07:50 [PATCH v36 2/6] Introduce UniqueKey attributes on RelOptInfo struct. 一挃 <[email protected]> 2020-05-11 07:50 [PATCH v38 2/6] Introduce UniqueKey attributes on RelOptInfo struct. 一挃 <[email protected]> 2020-05-11 07:50 [PATCH v38 2/6] Introduce UniqueKey attributes on RelOptInfo struct. 一挃 <[email protected]> 2020-05-11 07:50 [PATCH v36 2/6] Introduce UniqueKey attributes on RelOptInfo struct. 一挃 <[email protected]> 2020-05-11 07:50 [PATCH v38 2/6] Introduce UniqueKey attributes on RelOptInfo struct. 一挃 <[email protected]> 2020-05-11 07:50 [PATCH v38 2/6] Introduce UniqueKey attributes on RelOptInfo struct. 一挃 <[email protected]> 2020-05-11 07:50 [PATCH v36 2/6] Introduce UniqueKey attributes on RelOptInfo struct. 一挃 <[email protected]> 2020-05-11 07:50 [PATCH v38 2/6] Introduce UniqueKey attributes on RelOptInfo struct. 一挃 <[email protected]> 2020-05-11 07:50 [PATCH v38 2/6] Introduce UniqueKey attributes on RelOptInfo struct. 一挃 <[email protected]> 2020-05-11 07:50 [PATCH v36 2/6] Introduce UniqueKey attributes on RelOptInfo struct. 一挃 <[email protected]> 2022-01-14 16:25 Re: Add sub-transaction overflow status in pg_stat_activity Tom Lane <[email protected]> 2022-01-14 19:42 ` Re: Add sub-transaction overflow status in pg_stat_activity Bossart, Nathan <[email protected]> 2022-01-15 04:15 ` Re: Add sub-transaction overflow status in pg_stat_activity Julien Rouhaud <[email protected]> 2022-01-15 05:13 ` Re: Add sub-transaction overflow status in pg_stat_activity Tom Lane <[email protected]> 2022-01-15 05:29 ` Re: Add sub-transaction overflow status in pg_stat_activity Julien Rouhaud <[email protected]> 2022-03-21 23:45 ` Re: Add sub-transaction overflow status in pg_stat_activity Andres Freund <[email protected]> 2022-11-14 15:09 ` Re: Add sub-transaction overflow status in pg_stat_activity Robert Haas <[email protected]> 2022-11-14 15:41 ` Re: Add sub-transaction overflow status in pg_stat_activity Tom Lane <[email protected]> 2022-11-14 15:52 ` Re: Add sub-transaction overflow status in pg_stat_activity Robert Haas <[email protected]> 2022-11-14 15:57 ` Re: Add sub-transaction overflow status in pg_stat_activity Justin Pryzby <[email protected]> 2022-11-14 16:04 ` Re: Add sub-transaction overflow status in pg_stat_activity Robert Haas <[email protected]> 2022-11-14 16:17 ` Re: Add sub-transaction overflow status in pg_stat_activity David G. Johnston <[email protected]> 2022-11-14 16:28 ` Re: Add sub-transaction overflow status in pg_stat_activity Robert Haas <[email protected]> 2022-11-14 16:34 ` Re: Add sub-transaction overflow status in pg_stat_activity Amit Singh <[email protected]> 2022-11-14 16:41 ` Re: Add sub-transaction overflow status in pg_stat_activity Robert Haas <[email protected]> 2022-11-14 16:48 ` Re: Add sub-transaction overflow status in pg_stat_activity David G. Johnston <[email protected]> 2022-11-14 17:29 ` Re: Add sub-transaction overflow status in pg_stat_activity Tom Lane <[email protected]> 2022-11-14 17:47 ` Re: Add sub-transaction overflow status in pg_stat_activity Andres Freund <[email protected]> 2022-11-14 18:43 ` Re: Add sub-transaction overflow status in pg_stat_activity Robert Haas <[email protected]> 2022-11-14 19:16 ` Re: Add sub-transaction overflow status in pg_stat_activity David G. Johnston <[email protected]> 2022-11-14 19:32 ` Re: Add sub-transaction overflow status in pg_stat_activity Robert Haas <[email protected]> 2022-11-14 21:03 ` Re: Add sub-transaction overflow status in pg_stat_activity Tom Lane <[email protected]> 2022-11-14 21:17 ` Re: Add sub-transaction overflow status in pg_stat_activity Andres Freund <[email protected]> 2022-11-15 05:23 ` Re: Add sub-transaction overflow status in pg_stat_activity Dilip Kumar <[email protected]> 2022-11-15 14:04 ` Re: Add sub-transaction overflow status in pg_stat_activity Robert Haas <[email protected]> 2022-11-15 19:29 ` Re: Add sub-transaction overflow status in pg_stat_activity Andres Freund <[email protected]> 2022-11-16 17:59 ` Re: Add sub-transaction overflow status in pg_stat_activity Robert Haas <[email protected]> 2022-11-16 10:31 ` Re: Add sub-transaction overflow status in pg_stat_activity Dilip Kumar <[email protected]> 2022-11-15 05:25 ` Re: Add sub-transaction overflow status in pg_stat_activity Dilip Kumar <[email protected]> 2022-11-23 19:01 ` Re: Add sub-transaction overflow status in pg_stat_activity Bruce Momjian <[email protected]> 2022-11-23 20:25 ` Re: Add sub-transaction overflow status in pg_stat_activity Robert Haas <[email protected]> 2022-11-23 20:56 ` Re: Add sub-transaction overflow status in pg_stat_activity Andres Freund <[email protected]> 2022-11-25 04:01 ` Re: Add sub-transaction overflow status in pg_stat_activity Dilip Kumar <[email protected]> 2022-11-30 16:01 ` Re: Add sub-transaction overflow status in pg_stat_activity Robert Haas <[email protected]> 2022-12-12 16:15 ` Re: Add sub-transaction overflow status in pg_stat_activity Robert Haas <[email protected]> 2022-12-12 17:33 ` Re: Add sub-transaction overflow status in pg_stat_activity Nathan Bossart <[email protected]> 2022-12-12 17:42 ` Re: Add sub-transaction overflow status in pg_stat_activity Justin Pryzby <[email protected]> 2022-12-12 17:51 ` Re: Add sub-transaction overflow status in pg_stat_activity Robert Haas <[email protected]> 2022-12-13 04:09 ` Re: Add sub-transaction overflow status in pg_stat_activity Dilip Kumar <[email protected]> 2022-12-13 07:29 ` Re: Add sub-transaction overflow status in pg_stat_activity Julien Rouhaud <[email protected]> 2022-12-19 19:56 ` Re: Add sub-transaction overflow status in pg_stat_activity Robert Haas <[email protected]> 2022-12-19 20:48 ` Re: Add sub-transaction overflow status in pg_stat_activity Ted Yu <[email protected]> 2022-12-19 21:02 ` Re: Add sub-transaction overflow status in pg_stat_activity Robert Haas <[email protected]> 2022-12-20 04:23 ` Re: Add sub-transaction overflow status in pg_stat_activity Dilip Kumar <[email protected]> 2023-02-08 09:10 ` Re: Add sub-transaction overflow status in pg_stat_activity Kirill Reshke <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox