Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtp (Exim 4.80) (envelope-from ) id 1YwzEb-0002JX-KE for pgsql-hackers@arkaria.postgresql.org; Mon, 25 May 2015 20:44:31 +0000 Received: from localhost ([127.0.0.1] helo=postgresql.org) by malur.postgresql.org with smtp (Exim 4.80) (envelope-from ) id 1YwzEa-00017g-EZ for pgsql-hackers@arkaria.postgresql.org; Mon, 25 May 2015 20:44:28 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.2:RSA_AES_256_CBC_SHA256:256) (Exim 4.80) (envelope-from ) id 1YwzEN-0000wm-Uy for pgsql-hackers@postgresql.org; Mon, 25 May 2015 20:44:16 +0000 Received: from mail-wg0-f45.google.com ([74.125.82.45]) by magus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.84) (envelope-from ) id 1YwzE2-0004CC-Np for pgsql-hackers@postgresql.org; Mon, 25 May 2015 20:44:15 +0000 Received: by wgez8 with SMTP id z8so80670640wge.0 for ; Mon, 25 May 2015 13:43:51 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-type; bh=WZjpCktCp3V1TSSf3CPVguegjOLCvcQUi3OSnY6CEdI=; b=Bd1ZSl5eGtcapxVuPqQ6xucraLwDbuZ77EknBekmqLR1rL2NffsKKL5ID2C68AlqTf 8HoZEbFmKgynHD98hy3PY9OxtW2M4xefOcsrX8tohBl0C+6HTpUOulAdRWx28W2n8S+C oIprLRCfFwfG0sIMbdwIEBr7iWTMSX5Hk/9KlcZeer7D4JJCUUDvFVGCjWOHdz3CXc71 EHiVsYV+31v6GFATmt31fm0aIbLRw6LHKkDMFRDiYb8nX+dZrcSHraemzh8D4YR26Zd4 0yzGSod9tiwThycSpMg3oIhhH6HGOmeoofyFZPBo9A0k30eaLqHIAjp7bzp38/grYz7w YqMw== X-Gm-Message-State: ALoCoQkwUStLFOmP7Vjfu9MbqRrj8MnbW3n6Z3xeCOwAuCKvD8pUSLvMA+uoNr9Kor1vZNRYe0ia X-Received: by 10.180.85.9 with SMTP id d9mr35143480wiz.28.1432586631503; Mon, 25 May 2015 13:43:51 -0700 (PDT) Received: from [10.137.2.12] (ip-78-45-136-74.net.upcbroadband.cz. [78.45.136.74]) by mx.google.com with ESMTPSA id l3sm13554616wik.16.2015.05.25.13.43.48 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 25 May 2015 13:43:49 -0700 (PDT) Message-ID: <55638982.40904@2ndquadrant.com> Date: Mon, 25 May 2015 22:43:46 +0200 From: Tomas Vondra User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: pgsql-hackers CC: Jeff Janes , Stephen Frost , Kyotaro HORIGUCHI Subject: Re: multivariate statistics / patch v7 References: <54C3FED3.1060600@2ndquadrant.com> <20150320.173307.249113775.horiguchi.kyotaro@lab.ntt.co.jp> <550C8E50.2010600@2ndquadrant.com> <20150324.143427.34552953.horiguchi.kyotaro@lab.ntt.co.jp> <5519E9B2.5080105@2ndquadrant.com> <20150428161310.GR30322@tamriel.snowman.net> <553FCE56.2000008@2ndquadrant.com> <554A73A6.2060603@2ndquadrant.com> In-Reply-To: <554A73A6.2060603@2ndquadrant.com> Content-Type: multipart/mixed; boundary="------------080401000304040708010509" X-Pg-Spam-Score: -1.9 (-) List-Archive: List-Help: List-ID: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: X-Mailing-List: pgsql-hackers Precedence: bulk Sender: pgsql-hackers-owner@postgresql.org This is a multi-part message in MIME format. --------------080401000304040708010509 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Hello, attached is v7 of the multivariate stats patch. The main improvement is major refactoring of the clausesel.c portion - splitting the awfully long spaghetti-style functions into smaller pieces, making it much more understandable etc. I do assume some of those pieces are unnecessary because there already is a helper function with the same purpose (but I'm not aware of that). But IMHO this piece of code begins to look reasonable (especially when compared to the previous state). The other major improvement it review of the comments (including FIXMEs and TODOs), and removal of the obsolete / misplaced ones. And there was plenty of those ... These changes made this version ~20k smaller than v6. The patch also rebases to current master, which I assume shall be quite stable - so hopefully no more duplicate OIDs for a while. There are 6 files attached, but only 0002-0006 are actually part of the multivariate statistics patch itself. The first part makes it possible to use pull_varnos() with expression trees containing RestrictInfo nodes, but maybe this is not the right way to fix this (there's another thread where this was discussed). Also, the regression tests testing plan choice with multivariate stats (e.g. that a bitmap index scan is chosen instead of index scan) fail from time to time. I suppose this happens because the invalidation after ANALYZE is not processed before executing the query, so the optimizer does not see the stats, or something like that. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services --------------080401000304040708010509 Content-Type: text/x-patch; name="0001-teach-expression-walker-about-RestrictInfo-v7.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0001-teach-expression-walker-about-RestrictInfo-v7.patch" From 886edce86cbe571283ebe49177288e9978b10c81 Mon Sep 17 00:00:00 2001 From: Tomas Vondra Date: Tue, 28 Apr 2015 19:56:33 +0200 Subject: [PATCH 1/6] teach expression walker about RestrictInfo otherwise pull_varnos fails when processing OR clauses --- src/backend/nodes/nodeFuncs.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index a2bcca5..7dcc1c1 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -1995,6 +1995,8 @@ expression_tree_walker(Node *node, return walker(((PlaceHolderInfo *) node)->ph_var, context); case T_RangeTblFunction: return walker(((RangeTblFunction *) node)->funcexpr, context); + case T_RestrictInfo: + return walker(((RestrictInfo *) node)->clause, context); default: elog(ERROR, "unrecognized node type: %d", (int) nodeTag(node)); -- 1.9.3 --------------080401000304040708010509 Content-Type: text/x-patch; name="0002-shared-infrastructure-and-functional-dependencies-v7.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename*0="0002-shared-infrastructure-and-functional-dependencies-v7.pa"; filename*1="tch" From 9e4e4141af44c03ccec77490c84f6c70e68e4449 Mon Sep 17 00:00:00 2001 From: Tomas Vondra Date: Sun, 11 Jan 2015 19:51:48 +0100 Subject: [PATCH 2/6] shared infrastructure and functional dependencies Basic infrastructure shared by all kinds of multivariate stats, most importantly: - adds a new system catalog (pg_mv_statistic) - ALTER TABLE ... ADD STATISTICS - ALTER TABLE ... DROP STATISTICS - implementation of functional dependencies (the simplest type of multivariate statistics) - building functional dependencies in ANALYZE - updates regression tests (new catalog etc.) This does not include any changes to the optimizer, i.e. it does not influence the query planning (subject to follow-up patches). The current implementation requires a valid 'ltopr' for the columns, so that we can sort the sample rows in various ways, both in this patch and other kinds of statistics. Maybe this restriction could be relaxed in the future, requiring just 'eqopr' in case of stats not sorting the data (e.g. functional dependencies and MCV lists). Maybe some of the stats (functional dependencies and MCV list with limited functionality) might be made to work with hashes of the values, which is sufficient for equality comparisons. But the queries would require the equality operator anyway, so it's not really a weaker requirement. The hashes might reduce space requirements, though. The algorithm detecting the dependencies is rather simple and probably needs improvements, so that it detects more complicated dependencies, and also validation of the math. The name 'functional dependencies' is more correct (than 'association rules') as it's exactly the name used in relational theory (esp. Normal Forms) for tracking column-level dependencies. The multivariate statistics are automatically removed in two situations (a) after a DROP TABLE (obviously) (b) after ALTER TABLE ... DROP COLUMN, if the statistics would be defined on less than 2 columns (remaining) If there are more at least 2 columns remaining, we keep the statistics but perform cleanup on the next ANALYZE. The dropped columns are removed from stakeys, and the new statistics is built on the smaller set. We can't do this at DROP COLUMN, because that'd leave us with invalid statistics, or we'd have to throw it away although we can still use it. This lazy approach lets us use the statistics although some of the columns are dead. Dropping the statistics is done using DROP STATISTICS ALTER TABLE ... DROP STATISTICS ALL; ALTER TABLE ... DROP STATISTICS (opts) ON (cols); The bad consequence of this is that 'statistics' becomes a reserved keyword (was unreserved before), otherwise it conflicts with DROP in the grammar. Not sure if there's a workaround to this. This also adds a simple list of statistics to \d in psql. --- src/backend/catalog/Makefile | 1 + src/backend/catalog/heap.c | 102 +++++ src/backend/catalog/system_views.sql | 10 + src/backend/commands/analyze.c | 21 + src/backend/commands/tablecmds.c | 342 +++++++++++++++- src/backend/nodes/copyfuncs.c | 13 + src/backend/nodes/outfuncs.c | 18 + src/backend/optimizer/util/plancat.c | 63 +++ src/backend/parser/gram.y | 83 +++- src/backend/utils/Makefile | 2 +- src/backend/utils/cache/relcache.c | 59 +++ src/backend/utils/cache/syscache.c | 12 + src/backend/utils/mvstats/Makefile | 17 + src/backend/utils/mvstats/common.c | 356 ++++++++++++++++ src/backend/utils/mvstats/common.h | 75 ++++ src/backend/utils/mvstats/dependencies.c | 638 +++++++++++++++++++++++++++++ src/bin/psql/describe.c | 40 ++ src/include/catalog/heap.h | 1 + src/include/catalog/indexing.h | 5 + src/include/catalog/pg_mv_statistic.h | 69 ++++ src/include/catalog/pg_proc.h | 5 + src/include/catalog/toasting.h | 1 + src/include/nodes/nodes.h | 2 + src/include/nodes/parsenodes.h | 12 +- src/include/nodes/relation.h | 28 ++ src/include/parser/kwlist.h | 2 +- src/include/utils/mvstats.h | 69 ++++ src/include/utils/rel.h | 4 + src/include/utils/relcache.h | 1 + src/include/utils/syscache.h | 1 + src/test/regress/expected/rules.out | 8 + src/test/regress/expected/sanity_check.out | 1 + 32 files changed, 2053 insertions(+), 8 deletions(-) create mode 100644 src/backend/utils/mvstats/Makefile create mode 100644 src/backend/utils/mvstats/common.c create mode 100644 src/backend/utils/mvstats/common.h create mode 100644 src/backend/utils/mvstats/dependencies.c create mode 100644 src/include/catalog/pg_mv_statistic.h create mode 100644 src/include/utils/mvstats.h diff --git a/src/backend/catalog/Makefile b/src/backend/catalog/Makefile index 3d1139b..c6de23c 100644 --- a/src/backend/catalog/Makefile +++ b/src/backend/catalog/Makefile @@ -32,6 +32,7 @@ POSTGRES_BKI_SRCS = $(addprefix $(top_srcdir)/src/include/catalog/,\ pg_attrdef.h pg_constraint.h pg_inherits.h pg_index.h pg_operator.h \ pg_opfamily.h pg_opclass.h pg_am.h pg_amop.h pg_amproc.h \ pg_language.h pg_largeobject_metadata.h pg_largeobject.h pg_aggregate.h \ + pg_mv_statistic.h \ pg_statistic.h pg_rewrite.h pg_trigger.h pg_event_trigger.h pg_description.h \ pg_cast.h pg_enum.h pg_namespace.h pg_conversion.h pg_depend.h \ pg_database.h pg_db_role_setting.h pg_tablespace.h pg_pltemplate.h \ diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index d04e94d..1c28ca3 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -46,6 +46,7 @@ #include "catalog/pg_constraint.h" #include "catalog/pg_foreign_table.h" #include "catalog/pg_inherits.h" +#include "catalog/pg_mv_statistic.h" #include "catalog/pg_namespace.h" #include "catalog/pg_statistic.h" #include "catalog/pg_tablespace.h" @@ -1611,7 +1612,10 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) heap_close(attr_rel, RowExclusiveLock); if (attnum > 0) + { RemoveStatistics(relid, attnum); + RemoveMVStatistics(relid, attnum); + } relation_close(rel, NoLock); } @@ -1839,6 +1843,11 @@ heap_drop_with_catalog(Oid relid) RemoveStatistics(relid, 0); /* + * delete multi-variate statistics + */ + RemoveMVStatistics(relid, 0); + + /* * delete attribute tuples */ DeleteAttributeTuples(relid); @@ -2694,6 +2703,99 @@ RemoveStatistics(Oid relid, AttrNumber attnum) /* + * RemoveMVStatistics --- remove entries in pg_mv_statistic for a rel + * + * If attnum is zero, remove all entries for rel; else remove only the one(s) + * for that column. + */ +void +RemoveMVStatistics(Oid relid, AttrNumber attnum) +{ + Relation pgmvstatistic; + TupleDesc tupdesc = NULL; + SysScanDesc scan; + ScanKeyData key; + HeapTuple tuple; + + /* + * When dropping a column, we'll drop statistics with a single + * remaining (undropped column). To do that, we need the tuple + * descriptor. + * + * We already have the relation locked (as we're running ALTER + * TABLE ... DROP COLUMN), so we'll just get the descriptor here. + */ + if (attnum != 0) + { + Relation rel = relation_open(relid, NoLock); + + /* multivariate stats are supported on tables and matviews */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_MATVIEW) + tupdesc = RelationGetDescr(rel); + + relation_close(rel, NoLock); + } + + if (tupdesc == NULL) + return; + + pgmvstatistic = heap_open(MvStatisticRelationId, RowExclusiveLock); + + ScanKeyInit(&key, + Anum_pg_mv_statistic_starelid, + BTEqualStrategyNumber, F_OIDEQ, + ObjectIdGetDatum(relid)); + + scan = systable_beginscan(pgmvstatistic, + MvStatisticRelidIndexId, + true, NULL, 1, &key); + + /* we must loop even when attnum != 0, in case of inherited stats */ + while (HeapTupleIsValid(tuple = systable_getnext(scan))) + { + bool delete = true; + + if (attnum != 0) + { + Datum adatum; + bool isnull; + int i; + int ncolumns = 0; + ArrayType *arr; + int16 *attnums; + + /* get the columns */ + adatum = SysCacheGetAttr(MVSTATOID, tuple, + Anum_pg_mv_statistic_stakeys, &isnull); + Assert(!isnull); + + arr = DatumGetArrayTypeP(adatum); + attnums = (int16*)ARR_DATA_PTR(arr); + + for (i = 0; i < ARR_DIMS(arr)[0]; i++) + { + /* count the column unless it's has been / is being dropped */ + if ((! tupdesc->attrs[attnums[i]-1]->attisdropped) && + (attnums[i] != attnum)) + ncolumns += 1; + } + + /* delete if there are less than two attributes */ + delete = (ncolumns < 2); + } + + if (delete) + simple_heap_delete(pgmvstatistic, &tuple->t_self); + } + + systable_endscan(scan); + + heap_close(pgmvstatistic, RowExclusiveLock); +} + + +/* * RelationTruncateIndexes - truncate all indexes associated * with the heap relation to zero tuples. * diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql index 18921c4..0dedaba 100644 --- a/src/backend/catalog/system_views.sql +++ b/src/backend/catalog/system_views.sql @@ -150,6 +150,16 @@ CREATE VIEW pg_indexes AS LEFT JOIN pg_tablespace T ON (T.oid = I.reltablespace) WHERE C.relkind IN ('r', 'm') AND I.relkind = 'i'; +CREATE VIEW pg_mv_stats AS + SELECT + N.nspname AS schemaname, + C.relname AS tablename, + S.stakeys AS attnums, + length(S.stadeps) as depsbytes, + pg_mv_stats_dependencies_info(S.stadeps) as depsinfo + FROM (pg_mv_statistic S JOIN pg_class C ON (C.oid = S.starelid)) + LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace); + CREATE VIEW pg_stats AS SELECT nspname AS schemaname, diff --git a/src/backend/commands/analyze.c b/src/backend/commands/analyze.c index 861048f..1f50036 100644 --- a/src/backend/commands/analyze.c +++ b/src/backend/commands/analyze.c @@ -27,6 +27,7 @@ #include "catalog/indexing.h" #include "catalog/pg_collation.h" #include "catalog/pg_inherits_fn.h" +#include "catalog/pg_mv_statistic.h" #include "catalog/pg_namespace.h" #include "commands/dbcommands.h" #include "commands/tablecmds.h" @@ -55,7 +56,11 @@ #include "utils/syscache.h" #include "utils/timestamp.h" #include "utils/tqual.h" +#include "utils/fmgroids.h" +#include "utils/builtins.h" +#include "utils/mvstats.h" +#include "access/sysattr.h" /* Per-index data for ANALYZE */ typedef struct AnlIndexData @@ -460,6 +465,19 @@ do_analyze_rel(Relation onerel, int options, VacuumParams *params, * all analyzable columns. We use a lower bound of 100 rows to avoid * possible overflow in Vitter's algorithm. (Note: that will also be the * target in the corner case where there are no analyzable columns.) + * + * FIXME This sample sizing is mostly OK when computing stats for + * individual columns, but when computing multi-variate stats + * for multivariate stats (histograms, mcv, ...) it's rather + * insufficient. For stats on multiple columns / complex stats + * we need larger sample sizes, because we need to build more + * detailed stats (more MCV items / histogram buckets) to get + * good accuracy. Maybe it'd be appropriate to use samples + * proportional to the table (say, 0.5% - 1%) instead of a + * fixed size might be more appropriate. Also, this should be + * bound to the requested statistics size - e.g. number of MCV + * items or histogram buckets should require several sample + * rows per item/bucket (so the sample should be k*size). */ targrows = 100; for (i = 0; i < attr_cnt; i++) @@ -562,6 +580,9 @@ do_analyze_rel(Relation onerel, int options, VacuumParams *params, update_attstats(RelationGetRelid(Irel[ind]), false, thisdata->attr_cnt, thisdata->vacattrstats); } + + /* Build multivariate stats (if there are any). */ + build_mv_stats(onerel, numrows, rows, attr_cnt, vacattrstats); } /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 84dbee0..d6c6f8e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -35,6 +35,7 @@ #include "catalog/pg_foreign_table.h" #include "catalog/pg_inherits.h" #include "catalog/pg_inherits_fn.h" +#include "catalog/pg_mv_statistic.h" #include "catalog/pg_namespace.h" #include "catalog/pg_opclass.h" #include "catalog/pg_tablespace.h" @@ -92,7 +93,7 @@ #include "utils/syscache.h" #include "utils/tqual.h" #include "utils/typcache.h" - +#include "utils/mvstats.h" /* * ON COMMIT action list @@ -140,8 +141,9 @@ static List *on_commits = NIL; #define AT_PASS_ADD_COL 5 /* ADD COLUMN */ #define AT_PASS_ADD_INDEX 6 /* ADD indexes */ #define AT_PASS_ADD_CONSTR 7 /* ADD constraints, defaults */ -#define AT_PASS_MISC 8 /* other stuff */ -#define AT_NUM_PASSES 9 +#define AT_PASS_ADD_STATS 8 /* ADD statistics */ +#define AT_PASS_MISC 9 /* other stuff */ +#define AT_NUM_PASSES 10 typedef struct AlteredTableInfo { @@ -416,6 +418,10 @@ static void ATExecReplicaIdentity(Relation rel, ReplicaIdentityStmt *stmt, LOCKM static void ATExecGenericOptions(Relation rel, List *options); static void ATExecEnableRowSecurity(Relation rel); static void ATExecDisableRowSecurity(Relation rel); +static void ATExecAddStatistics(AlteredTableInfo *tab, Relation rel, + StatisticsDef *def, LOCKMODE lockmode); +static void ATExecDropStatistics(AlteredTableInfo *tab, Relation rel, + StatisticsDef *def, LOCKMODE lockmode); static void copy_relation_data(SMgrRelation rel, SMgrRelation dst, ForkNumber forkNum, char relpersistence); @@ -3013,6 +3019,8 @@ AlterTableGetLockLevel(List *cmds) * updates. */ case AT_SetStatistics: /* Uses MVCC in getTableAttrs() */ + case AT_AddStatistics: /* XXX not sure if the right level */ + case AT_DropStatistics: /* XXX not sure if the right level */ case AT_ClusterOn: /* Uses MVCC in getIndexes() */ case AT_DropCluster: /* Uses MVCC in getIndexes() */ case AT_SetOptions: /* Uses MVCC in getTableAttrs() */ @@ -3169,6 +3177,8 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd, pass = AT_PASS_ADD_CONSTR; break; case AT_SetStatistics: /* ALTER COLUMN SET STATISTICS */ + case AT_AddStatistics: /* XXX maybe not the right place */ + case AT_DropStatistics: /* XXX maybe not the right place */ ATSimpleRecursion(wqueue, rel, cmd, recurse, lockmode); /* Performs own permission checks */ ATPrepSetStatistics(rel, cmd->name, cmd->def, lockmode); @@ -3471,6 +3481,12 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel, case AT_SetStatistics: /* ALTER COLUMN SET STATISTICS */ address = ATExecSetStatistics(rel, cmd->name, cmd->def, lockmode); break; + case AT_AddStatistics: /* ADD STATISTICS */ + ATExecAddStatistics(tab, rel, (StatisticsDef *) cmd->def, lockmode); + break; + case AT_DropStatistics: /* DROP STATISTICS */ + ATExecDropStatistics(tab, rel, (StatisticsDef *) cmd->def, lockmode); + break; case AT_SetOptions: /* ALTER COLUMN SET ( options ) */ address = ATExecSetOptions(rel, cmd->name, cmd->def, false, lockmode); break; @@ -11868,3 +11884,323 @@ RangeVarCallbackForAlterRelation(const RangeVar *rv, Oid relid, Oid oldrelid, ReleaseSysCache(tuple); } + +/* used for sorting the attnums in ATExecAddStatistics */ +static int compare_int16(const void *a, const void *b) +{ + return memcmp(a, b, sizeof(int16)); +} + +/* + * Implements the ALTER TABLE ... ADD STATISTICS (options) ON (columns). + * + * TODO Check that the types support sort, although maybe we can live + * without it (and only build MCV list / association rules). + * + * TODO This should probably check for duplicate stats (i.e. same + * keys, same options). Although maybe it's useful to have + * multiple stats on the same columns with different options + * (say, a detailed MCV-only stats for some queries, histogram + * for others, etc.) + */ +static void ATExecAddStatistics(AlteredTableInfo *tab, Relation rel, + StatisticsDef *def, LOCKMODE lockmode) +{ + int i, j; + ListCell *l; + int16 attnums[INDEX_MAX_KEYS]; + int numcols = 0; + + HeapTuple htup; + Datum values[Natts_pg_mv_statistic]; + bool nulls[Natts_pg_mv_statistic]; + int2vector *stakeys; + Relation mvstatrel; + + /* by default build nothing */ + bool build_dependencies = false; + + Assert(IsA(def, StatisticsDef)); + + /* transform the column names to attnum values */ + + foreach(l, def->keys) + { + char *attname = strVal(lfirst(l)); + HeapTuple atttuple; + + atttuple = SearchSysCacheAttName(RelationGetRelid(rel), attname); + + if (!HeapTupleIsValid(atttuple)) + ereport(ERROR, + (errcode(ERRCODE_UNDEFINED_COLUMN), + errmsg("column \"%s\" referenced in statistics does not exist", + attname))); + + /* more than MVHIST_MAX_DIMENSIONS columns not allowed */ + if (numcols >= MVSTATS_MAX_DIMENSIONS) + ereport(ERROR, + (errcode(ERRCODE_TOO_MANY_COLUMNS), + errmsg("cannot have more than %d keys in a statistics", + MVSTATS_MAX_DIMENSIONS))); + + attnums[numcols] = ((Form_pg_attribute) GETSTRUCT(atttuple))->attnum; + ReleaseSysCache(atttuple); + numcols++; + } + + /* + * Check the lower bound (at least 2 columns), the upper bound was + * already checked in the loop. + */ + if (numcols < 2) + ereport(ERROR, + (errcode(ERRCODE_TOO_MANY_COLUMNS), + errmsg("multivariate stats require 2 or more columns"))); + + /* look for duplicities */ + for (i = 0; i < numcols; i++) + for (j = 0; j < numcols; j++) + if ((i != j) && (attnums[i] == attnums[j])) + ereport(ERROR, + (errcode(ERRCODE_UNDEFINED_COLUMN), + errmsg("duplicate column name in statistics definition"))); + + /* parse the statistics options */ + foreach (l, def->options) + { + DefElem *opt = (DefElem*)lfirst(l); + + if (strcmp(opt->defname, "dependencies") == 0) + build_dependencies = defGetBoolean(opt); + else + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("unrecognized STATISTICS option \"%s\"", + opt->defname))); + } + + /* sort the attnums and build int2vector */ + qsort(attnums, numcols, sizeof(int16), compare_int16); + stakeys = buildint2vector(attnums, numcols); + + /* + * Okay, let's create the pg_mv_statistic entry. + */ + memset(values, 0, sizeof(values)); + memset(nulls, false, sizeof(nulls)); + + /* no stats collected yet, so just the keys */ + values[Anum_pg_mv_statistic_starelid-1] = ObjectIdGetDatum(RelationGetRelid(rel)); + + values[Anum_pg_mv_statistic_stakeys -1] = PointerGetDatum(stakeys); + values[Anum_pg_mv_statistic_deps_enabled -1] = BoolGetDatum(build_dependencies); + + nulls[Anum_pg_mv_statistic_stadeps -1] = true; + + /* insert the tuple into pg_mv_statistic */ + mvstatrel = heap_open(MvStatisticRelationId, RowExclusiveLock); + + htup = heap_form_tuple(mvstatrel->rd_att, values, nulls); + + simple_heap_insert(mvstatrel, htup); + + CatalogUpdateIndexes(mvstatrel, htup); + + heap_freetuple(htup); + + heap_close(mvstatrel, RowExclusiveLock); + + /* + * Invalidate relcache so that others see the new statistics. + */ + CacheInvalidateRelcache(rel); + + return; +} + +/* + * Implements the ALTER TABLE ... DROP STATISTICS in two forms: + * + * ALTER TABLE ... DROP STATISTICS (options) ON (columns) + * ALTER TABLE ... DROP STATISTICS ALL; + * + * The first one requires an exact match, the second one just drops + * all the statistics on a table. + */ +static void ATExecDropStatistics(AlteredTableInfo *tab, Relation rel, + StatisticsDef *def, LOCKMODE lockmode) +{ + Relation statrel; + SysScanDesc scan; + ScanKeyData key; + HeapTuple tuple; + + ListCell *l; + + int16 attnums[INDEX_MAX_KEYS]; + int numcols = 0; + + /* checking whether the statistics matches / should be dropped */ + bool build_dependencies = false; + bool check_dependencies = false; + + if (def != NULL) + { + Assert(IsA(def, StatisticsDef)); + + /* collect attribute numbers */ + foreach(l, def->keys) + { + char *attname = strVal(lfirst(l)); + HeapTuple atttuple; + + atttuple = SearchSysCacheAttName(RelationGetRelid(rel), attname); + + if (!HeapTupleIsValid(atttuple)) + ereport(ERROR, + (errcode(ERRCODE_UNDEFINED_COLUMN), + errmsg("column \"%s\" referenced in statistics does not exist", + attname))); + + /* more than MVHIST_MAX_DIMENSIONS columns not allowed */ + if (numcols >= MVSTATS_MAX_DIMENSIONS) + ereport(ERROR, + (errcode(ERRCODE_TOO_MANY_COLUMNS), + errmsg("cannot have more than %d keys in a statistics", + MVSTATS_MAX_DIMENSIONS))); + + attnums[numcols] = ((Form_pg_attribute) GETSTRUCT(atttuple))->attnum; + ReleaseSysCache(atttuple); + numcols++; + } + + /* parse the statistics options */ + foreach (l, def->options) + { + DefElem *opt = (DefElem*)lfirst(l); + + if (strcmp(opt->defname, "dependencies") == 0) + { + check_dependencies = true; + build_dependencies = defGetBoolean(opt); + } + else + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("unrecognized STATISTICS option \"%s\"", + opt->defname))); + } + + } + + statrel = heap_open(MvStatisticRelationId, RowExclusiveLock); + + ScanKeyInit(&key, + Anum_pg_mv_statistic_starelid, + BTEqualStrategyNumber, F_OIDEQ, + ObjectIdGetDatum(RelationGetRelid(rel))); + + scan = systable_beginscan(statrel, + MvStatisticRelidIndexId, + true, NULL, 1, &key); + + /* we must loop even when attnum != 0, in case of inherited stats */ + while (HeapTupleIsValid(tuple = systable_getnext(scan))) + { + /* by default we delete everything */ + bool delete = true; + + /* check that the options match (dependencies, mcv, histogram) */ + if (delete && check_dependencies) + { + bool isnull; + Datum adatum = heap_getattr(tuple, + Anum_pg_mv_statistic_deps_enabled, + RelationGetDescr(statrel), + &isnull); + + delete = (! isnull) && + (DatumGetBool(adatum) == build_dependencies); + } + + /* check that the columns match the statistics definition */ + if (delete && (numcols > 0)) + { + int i, j; + ArrayType *arr; + bool isnull; + + int16 *stakeys; + int nstakeys; + + Datum adatum = SysCacheGetAttr(MVSTATOID, tuple, + Anum_pg_mv_statistic_stakeys, &isnull); + Assert(!isnull); + + arr = DatumGetArrayTypeP(adatum); + + nstakeys = ARR_DIMS(arr)[0]; + stakeys = (int16 *) ARR_DATA_PTR(arr); + + /* assume match */ + delete = true; + + /* check that for each column we find a match in stakeys */ + for (i = 0; i < numcols; i++) + { + bool found = false; + for (j = 0; j < nstakeys; j++) + { + if (attnums[i] == stakeys[j]) + { + found = true; + break; + } + } + + if (! found) + { + delete = false; + break; + } + } + + /* check that for each stakeys we find a match in columns */ + for (j = 0; j < nstakeys; j++) + { + bool found = false; + + for (i = 0; i < numcols; i++) + { + if (attnums[i] == stakeys[j]) + { + found = true; + break; + } + } + + if (! found) + { + delete = false; + break; + } + } + } + + /* don't delete, if we've found mismatches */ + if (delete) + simple_heap_delete(statrel, &tuple->t_self); + } + + systable_endscan(scan); + + heap_close(statrel, RowExclusiveLock); + + /* + * Invalidate relcache so that others forget the dropped statistics. + */ + CacheInvalidateRelcache(rel); + + return; +} diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 4c363d3..e5a3d96 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -4095,6 +4095,17 @@ _copyAlterPolicyStmt(const AlterPolicyStmt *from) return newnode; } +static StatisticsDef * +_copyStatisticsDef(const StatisticsDef *from) +{ + StatisticsDef *newnode = makeNode(StatisticsDef); + + COPY_NODE_FIELD(keys); + COPY_NODE_FIELD(options); + + return newnode; +} + /* **************************************************************** * pg_list.h copy functions * **************************************************************** @@ -4938,6 +4949,8 @@ copyObject(const void *from) break; case T_TableSampleClause: retval = _copyTableSampleClause(from); + case T_StatisticsDef: + retval = _copyStatisticsDef(from); break; case T_FuncWithArgs: retval = _copyFuncWithArgs(from); diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 4775acf..93a6f04 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -1898,6 +1898,21 @@ _outIndexOptInfo(StringInfo str, const IndexOptInfo *node) } static void +_outMVStatisticInfo(StringInfo str, const MVStatisticInfo *node) +{ + WRITE_NODE_TYPE("MVSTATISTICINFO"); + + /* NB: this isn't a complete set of fields */ + WRITE_OID_FIELD(mvoid); + + /* enabled statistics */ + WRITE_BOOL_FIELD(deps_enabled); + + /* built/available statistics */ + WRITE_BOOL_FIELD(deps_built); +} + +static void _outEquivalenceClass(StringInfo str, const EquivalenceClass *node) { /* @@ -3331,6 +3346,9 @@ _outNode(StringInfo str, const void *obj) case T_PlannerParamItem: _outPlannerParamItem(str, obj); break; + case T_MVStatisticInfo: + _outMVStatisticInfo(str, obj); + break; case T_CreateStmt: _outCreateStmt(str, obj); diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index b04dc2e..c397773 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -27,6 +27,7 @@ #include "catalog/catalog.h" #include "catalog/dependency.h" #include "catalog/heap.h" +#include "catalog/pg_mv_statistic.h" #include "foreign/fdwapi.h" #include "miscadmin.h" #include "nodes/makefuncs.h" @@ -39,7 +40,9 @@ #include "parser/parsetree.h" #include "rewrite/rewriteManip.h" #include "storage/bufmgr.h" +#include "utils/builtins.h" #include "utils/lsyscache.h" +#include "utils/syscache.h" #include "utils/rel.h" #include "utils/snapmgr.h" @@ -92,6 +95,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, Relation relation; bool hasindex; List *indexinfos = NIL; + List *stainfos = NIL; /* * We need not lock the relation since it was already locked, either by @@ -380,6 +384,65 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, rel->indexlist = indexinfos; + if (true) + { + List *mvstatoidlist; + ListCell *l; + + mvstatoidlist = RelationGetMVStatList(relation); + + foreach(l, mvstatoidlist) + { + ArrayType *arr; + Datum adatum; + bool isnull; + Oid mvoid = lfirst_oid(l); + Form_pg_mv_statistic mvstat; + MVStatisticInfo *info; + + HeapTuple htup = SearchSysCache1(MVSTATOID, ObjectIdGetDatum(mvoid)); + + /* XXX syscache contains OIDs of deleted stats (not invalidated) */ + if (! HeapTupleIsValid(htup)) + continue; + + mvstat = (Form_pg_mv_statistic) GETSTRUCT(htup); + + /* unavailable stats are not interesting for the planner */ + if (mvstat->deps_built) + { + info = makeNode(MVStatisticInfo); + + info->mvoid = mvoid; + info->rel = rel; + + /* enabled statistics */ + info->deps_enabled = mvstat->deps_enabled; + + /* built/available statistics */ + info->deps_built = mvstat->deps_built; + + /* stakeys */ + adatum = SysCacheGetAttr(MVSTATOID, htup, + Anum_pg_mv_statistic_stakeys, &isnull); + Assert(!isnull); + + arr = DatumGetArrayTypeP(adatum); + + info->stakeys = buildint2vector((int16 *) ARR_DATA_PTR(arr), + ARR_DIMS(arr)[0]); + + stainfos = lcons(info, stainfos); + } + + ReleaseSysCache(htup); + } + + list_free(mvstatoidlist); + } + + rel->mvstatlist = stainfos; + /* Grab foreign-table info using the relcache, while we have it */ if (relation->rd_rel->relkind == RELKIND_FOREIGN_TABLE) { diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index e0ff6f1..d81bab6 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -375,6 +375,12 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type group_by_item empty_grouping_set rollup_clause cube_clause %type grouping_sets_clause +%type OptStatsOptions +%type stats_options_name +%type stats_options_arg +%type stats_options_elem +%type stats_options_list + %type opt_fdw_options fdw_options %type fdw_option @@ -501,7 +507,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type unreserved_keyword type_func_name_keyword %type col_name_keyword reserved_keyword -%type TableConstraint TableLikeClause +%type TableConstraint TableLikeClause TableStatistics %type TableLikeOptionList TableLikeOption %type ColQualList %type ColConstraint ColConstraintElem ConstraintAttr @@ -2333,6 +2339,29 @@ alter_table_cmd: n->subtype = AT_DisableRowSecurity; $$ = (Node *)n; } + /* ALTER TABLE ADD STATISTICS (options) ON (columns) */ + | ADD_P TableStatistics + { + AlterTableCmd *n = makeNode(AlterTableCmd); + n->subtype = AT_AddStatistics; + n->def = $2; + $$ = (Node *)n; + } + /* ALTER TABLE DROP STATISTICS (options) ON (columns) */ + | DROP TableStatistics + { + AlterTableCmd *n = makeNode(AlterTableCmd); + n->subtype = AT_DropStatistics; + n->def = $2; + $$ = (Node *)n; + } + /* ALTER TABLE DROP STATISTICS ALL */ + | DROP STATISTICS ALL + { + AlterTableCmd *n = makeNode(AlterTableCmd); + n->subtype = AT_DropStatistics; + $$ = (Node *)n; + } | alter_generic_options { AlterTableCmd *n = makeNode(AlterTableCmd); @@ -3407,6 +3436,56 @@ OptConsTableSpace: USING INDEX TABLESPACE name { $$ = $4; } ExistingIndex: USING INDEX index_name { $$ = $3; } ; +/***************************************************************************** + * + * QUERY : + * ALTER TABLE relname ADD STATISTICS (columns) WITH (options) + * + *****************************************************************************/ + +TableStatistics: + STATISTICS OptStatsOptions ON '(' columnList ')' + { + StatisticsDef *n = makeNode(StatisticsDef); + n->keys = $5; + n->options = $2; + $$ = (Node *) n; + } + ; + +OptStatsOptions: + '(' stats_options_list ')' { $$ = $2; } + | /*EMPTY*/ { $$ = NIL; } + ; + +stats_options_list: + stats_options_elem + { + $$ = list_make1($1); + } + | stats_options_list ',' stats_options_elem + { + $$ = lappend($1, $3); + } + ; + +stats_options_elem: + stats_options_name stats_options_arg + { + $$ = makeDefElem($1, $2); + } + ; + +stats_options_name: + NonReservedWord { $$ = $1; } + ; + +stats_options_arg: + opt_boolean_or_string { $$ = (Node *) makeString($1); } + | NumericOnly { $$ = (Node *) $1; } + | /* EMPTY */ { $$ = NULL; } + ; + /***************************************************************************** * @@ -13796,7 +13875,6 @@ unreserved_keyword: | STANDALONE_P | START | STATEMENT - | STATISTICS | STDIN | STDOUT | STORAGE @@ -14013,6 +14091,7 @@ reserved_keyword: | SELECT | SESSION_USER | SOME + | STATISTICS | SYMMETRIC | TABLE | THEN diff --git a/src/backend/utils/Makefile b/src/backend/utils/Makefile index 8374533..eba0352 100644 --- a/src/backend/utils/Makefile +++ b/src/backend/utils/Makefile @@ -9,7 +9,7 @@ top_builddir = ../../.. include $(top_builddir)/src/Makefile.global OBJS = fmgrtab.o -SUBDIRS = adt cache error fmgr hash init mb misc mmgr resowner sort time +SUBDIRS = adt cache error fmgr hash init mb misc mmgr mvstats resowner sort time # location of Catalog.pm catalogdir = $(top_srcdir)/src/backend/catalog diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c index f60f3cb..8e17872 100644 --- a/src/backend/utils/cache/relcache.c +++ b/src/backend/utils/cache/relcache.c @@ -47,6 +47,7 @@ #include "catalog/pg_auth_members.h" #include "catalog/pg_constraint.h" #include "catalog/pg_database.h" +#include "catalog/pg_mv_statistic.h" #include "catalog/pg_namespace.h" #include "catalog/pg_opclass.h" #include "catalog/pg_proc.h" @@ -3906,6 +3907,62 @@ RelationGetIndexList(Relation relation) return result; } + +List * +RelationGetMVStatList(Relation relation) +{ + Relation indrel; + SysScanDesc indscan; + ScanKeyData skey; + HeapTuple htup; + List *result; + List *oldlist; + MemoryContext oldcxt; + + /* Quick exit if we already computed the list. */ + if (relation->rd_mvstatvalid != 0) + return list_copy(relation->rd_mvstatlist); + + /* + * We build the list we intend to return (in the caller's context) while + * doing the scan. After successfully completing the scan, we copy that + * list into the relcache entry. This avoids cache-context memory leakage + * if we get some sort of error partway through. + */ + result = NIL; + + /* Prepare to scan pg_index for entries having indrelid = this rel. */ + ScanKeyInit(&skey, + Anum_pg_mv_statistic_starelid, + BTEqualStrategyNumber, F_OIDEQ, + ObjectIdGetDatum(RelationGetRelid(relation))); + + indrel = heap_open(MvStatisticRelationId, AccessShareLock); + indscan = systable_beginscan(indrel, MvStatisticRelidIndexId, true, + NULL, 1, &skey); + + while (HeapTupleIsValid(htup = systable_getnext(indscan))) + /* TODO maybe include only already built statistics? */ + result = insert_ordered_oid(result, HeapTupleGetOid(htup)); + + systable_endscan(indscan); + + heap_close(indrel, AccessShareLock); + + /* Now save a copy of the completed list in the relcache entry. */ + oldcxt = MemoryContextSwitchTo(CacheMemoryContext); + oldlist = relation->rd_mvstatlist; + relation->rd_mvstatlist = list_copy(result); + + relation->rd_mvstatvalid = true; + MemoryContextSwitchTo(oldcxt); + + /* Don't leak the old list, if there is one */ + list_free(oldlist); + + return result; +} + /* * insert_ordered_oid * Insert a new Oid into a sorted list of Oids, preserving ordering @@ -4875,6 +4932,8 @@ load_relcache_init_file(bool shared) rel->rd_indexattr = NULL; rel->rd_keyattr = NULL; rel->rd_idattr = NULL; + rel->rd_mvstatvalid = false; + rel->rd_mvstatlist = NIL; rel->rd_createSubid = InvalidSubTransactionId; rel->rd_newRelfilenodeSubid = InvalidSubTransactionId; rel->rd_amcache = NULL; diff --git a/src/backend/utils/cache/syscache.c b/src/backend/utils/cache/syscache.c index 58f90f6..89173d6 100644 --- a/src/backend/utils/cache/syscache.c +++ b/src/backend/utils/cache/syscache.c @@ -43,6 +43,7 @@ #include "catalog/pg_foreign_server.h" #include "catalog/pg_foreign_table.h" #include "catalog/pg_language.h" +#include "catalog/pg_mv_statistic.h" #include "catalog/pg_namespace.h" #include "catalog/pg_opclass.h" #include "catalog/pg_operator.h" @@ -502,6 +503,17 @@ static const struct cachedesc cacheinfo[] = { }, 4 }, + {MvStatisticRelationId, /* MVSTATOID */ + MvStatisticOidIndexId, + 1, + { + ObjectIdAttributeNumber, + 0, + 0, + 0 + }, + 128 + }, {NamespaceRelationId, /* NAMESPACENAME */ NamespaceNameIndexId, 1, diff --git a/src/backend/utils/mvstats/Makefile b/src/backend/utils/mvstats/Makefile new file mode 100644 index 0000000..099f1ed --- /dev/null +++ b/src/backend/utils/mvstats/Makefile @@ -0,0 +1,17 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for utils/mvstats +# +# IDENTIFICATION +# src/backend/utils/mvstats/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/utils/mvstats +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = common.o dependencies.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/utils/mvstats/common.c b/src/backend/utils/mvstats/common.c new file mode 100644 index 0000000..a755c49 --- /dev/null +++ b/src/backend/utils/mvstats/common.c @@ -0,0 +1,356 @@ +/*------------------------------------------------------------------------- + * + * common.c + * POSTGRES multivariate statistics + * + * + * Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * + * IDENTIFICATION + * src/backend/utils/mvstats/common.c + * + *------------------------------------------------------------------------- + */ + +#include "common.h" + +static VacAttrStats ** lookup_var_attr_stats(int2vector *attrs, + int natts, VacAttrStats **vacattrstats); + +static List* list_mv_stats(Oid relid); + + +/* + * Compute requested multivariate stats, using the rows sampled for the + * plain (single-column) stats. + * + * This fetches a list of stats from pg_mv_statistic, computes the stats + * and serializes them back into the catalog (as bytea values). + */ +void +build_mv_stats(Relation onerel, int numrows, HeapTuple *rows, + int natts, VacAttrStats **vacattrstats) +{ + ListCell *lc; + List *mvstats; + + TupleDesc tupdesc = RelationGetDescr(onerel); + + /* + * Fetch defined MV groups from pg_mv_statistic, and then compute + * the MV statistics (histograms for now). + */ + mvstats = list_mv_stats(RelationGetRelid(onerel)); + + foreach (lc, mvstats) + { + int j; + MVStatisticInfo *stat = (MVStatisticInfo *)lfirst(lc); + MVDependencies deps = NULL; + + VacAttrStats **stats = NULL; + int numatts = 0; + + /* int2 vector of attnums the stats should be computed on */ + int2vector * attrs = stat->stakeys; + + /* see how many of the columns are not dropped */ + for (j = 0; j < attrs->dim1; j++) + if (! tupdesc->attrs[attrs->values[j]-1]->attisdropped) + numatts += 1; + + /* if there are dropped attributes, build a filtered int2vector */ + if (numatts != attrs->dim1) + { + int16 *tmp = palloc0(numatts * sizeof(int16)); + int attnum = 0; + + for (j = 0; j < attrs->dim1; j++) + if (! tupdesc->attrs[attrs->values[j]-1]->attisdropped) + tmp[attnum++] = attrs->values[j]; + + pfree(attrs); + attrs = buildint2vector(tmp, numatts); + } + + /* filter only the interesting vacattrstats records */ + stats = lookup_var_attr_stats(attrs, natts, vacattrstats); + + /* check allowed number of dimensions */ + Assert((attrs->dim1 >= 2) && (attrs->dim1 <= MVSTATS_MAX_DIMENSIONS)); + + /* + * Analyze functional dependencies of columns. + */ + deps = build_mv_dependencies(numrows, rows, attrs, stats); + + /* store the histogram / MCV list in the catalog */ + update_mv_stats(stat->mvoid, deps, attrs); + } +} + +/* + * Lookup the VacAttrStats info for the selected columns, with indexes + * matching the attrs vector (to make it easy to work with when + * computing multivariate stats). + */ +static VacAttrStats ** +lookup_var_attr_stats(int2vector *attrs, int natts, VacAttrStats **vacattrstats) +{ + int i, j; + int numattrs = attrs->dim1; + VacAttrStats **stats = (VacAttrStats**)palloc0(numattrs * sizeof(VacAttrStats*)); + + /* lookup VacAttrStats info for the requested columns (same attnum) */ + for (i = 0; i < numattrs; i++) + { + stats[i] = NULL; + for (j = 0; j < natts; j++) + { + if (attrs->values[i] == vacattrstats[j]->tupattnum) + { + stats[i] = vacattrstats[j]; + break; + } + } + + /* + * Check that we found the info, that the attnum matches and + * that there's the requested 'lt' operator and that the type + * is 'passed-by-value'. + */ + Assert(stats[i] != NULL); + Assert(stats[i]->tupattnum == attrs->values[i]); + + /* FIXME This is rather ugly way to check for 'ltopr' (which + * is defined for 'scalar' attributes). + */ + Assert(((StdAnalyzeData *)stats[i]->extra_data)->ltopr != InvalidOid); + } + + return stats; +} + +/* + * Fetch list of MV stats defined on a table, without the actual data + * for histograms, MCV lists etc. + */ +static List* +list_mv_stats(Oid relid) +{ + Relation indrel; + SysScanDesc indscan; + ScanKeyData skey; + HeapTuple htup; + List *result = NIL; + + /* Prepare to scan pg_mv_statistic for entries having indrelid = this rel. */ + ScanKeyInit(&skey, + Anum_pg_mv_statistic_starelid, + BTEqualStrategyNumber, F_OIDEQ, + ObjectIdGetDatum(relid)); + + indrel = heap_open(MvStatisticRelationId, AccessShareLock); + indscan = systable_beginscan(indrel, MvStatisticRelidIndexId, true, + NULL, 1, &skey); + + while (HeapTupleIsValid(htup = systable_getnext(indscan))) + { + MVStatisticInfo *info = makeNode(MVStatisticInfo); + Form_pg_mv_statistic stats = (Form_pg_mv_statistic) GETSTRUCT(htup); + + info->mvoid = HeapTupleGetOid(htup); + info->stakeys = buildint2vector(stats->stakeys.values, stats->stakeys.dim1); + info->deps_built = stats->deps_built; + + result = lappend(result, info); + } + + systable_endscan(indscan); + + heap_close(indrel, AccessShareLock); + + /* TODO maybe save the list into relcache, as in RelationGetIndexList + * (which was used as an inspiration of this one)?. */ + + return result; +} + +void +update_mv_stats(Oid mvoid, MVDependencies dependencies, int2vector *attrs) +{ + HeapTuple stup, + oldtup; + Datum values[Natts_pg_mv_statistic]; + bool nulls[Natts_pg_mv_statistic]; + bool replaces[Natts_pg_mv_statistic]; + + Relation sd = heap_open(MvStatisticRelationId, RowExclusiveLock); + + memset(nulls, 1, Natts_pg_mv_statistic * sizeof(bool)); + memset(replaces, 0, Natts_pg_mv_statistic * sizeof(bool)); + memset(values, 0, Natts_pg_mv_statistic * sizeof(Datum)); + + /* + * Construct a new pg_mv_statistic tuple - replace only the histogram + * and MCV list, depending whether it actually was computed. + */ + if (dependencies != NULL) + { + nulls[Anum_pg_mv_statistic_stadeps -1] = false; + values[Anum_pg_mv_statistic_stadeps - 1] + = PointerGetDatum(serialize_mv_dependencies(dependencies)); + } + + /* always replace the value (either by bytea or NULL) */ + replaces[Anum_pg_mv_statistic_stadeps -1] = true; + + /* always change the availability flags */ + nulls[Anum_pg_mv_statistic_deps_built -1] = false; + nulls[Anum_pg_mv_statistic_stakeys-1] = false; + + /* use the new attnums, in case we removed some dropped ones */ + replaces[Anum_pg_mv_statistic_deps_built-1] = true; + replaces[Anum_pg_mv_statistic_stakeys -1] = true; + + values[Anum_pg_mv_statistic_deps_built-1] = BoolGetDatum(dependencies != NULL); + values[Anum_pg_mv_statistic_stakeys -1] = PointerGetDatum(attrs); + + /* Is there already a pg_mv_statistic tuple for this attribute? */ + oldtup = SearchSysCache1(MVSTATOID, + ObjectIdGetDatum(mvoid)); + + if (HeapTupleIsValid(oldtup)) + { + /* Yes, replace it */ + stup = heap_modify_tuple(oldtup, + RelationGetDescr(sd), + values, + nulls, + replaces); + ReleaseSysCache(oldtup); + simple_heap_update(sd, &stup->t_self, stup); + } + else + elog(ERROR, "invalid pg_mv_statistic record (oid=%d)", mvoid); + + /* update indexes too */ + CatalogUpdateIndexes(sd, stup); + + heap_freetuple(stup); + + heap_close(sd, RowExclusiveLock); +} + +/* multi-variate stats comparator */ + +/* + * qsort_arg comparator for sorting Datums (MV stats) + * + * This does not maintain the tupnoLink array. + */ +int +compare_scalars_simple(const void *a, const void *b, void *arg) +{ + Datum da = *(Datum*)a; + Datum db = *(Datum*)b; + SortSupport ssup= (SortSupport) arg; + + return ApplySortComparator(da, false, db, false, ssup); +} + +/* + * qsort_arg comparator for sorting data when partitioning a MV bucket + */ +int +compare_scalars_partition(const void *a, const void *b, void *arg) +{ + Datum da = ((ScalarItem*)a)->value; + Datum db = ((ScalarItem*)b)->value; + SortSupport ssup= (SortSupport) arg; + + return ApplySortComparator(da, false, db, false, ssup); +} + +/* initialize multi-dimensional sort */ +MultiSortSupport +multi_sort_init(int ndims) +{ + MultiSortSupport mss; + + Assert(ndims >= 2); + + mss = (MultiSortSupport)palloc0(offsetof(MultiSortSupportData, ssup) + + sizeof(SortSupportData)*ndims); + + mss->ndims = ndims; + + return mss; +} + +/* + * add sort into for dimension 'dim' (index into vacattrstats) to mss, + * at the position 'sortattr' + */ +void +multi_sort_add_dimension(MultiSortSupport mss, int sortdim, + int dim, VacAttrStats **vacattrstats) +{ + /* first, lookup StdAnalyzeData for the dimension (attribute) */ + SortSupportData ssup; + StdAnalyzeData *tmp = (StdAnalyzeData *)vacattrstats[dim]->extra_data; + + Assert(mss != NULL); + Assert(sortdim < mss->ndims); + + /* initialize sort support, etc. */ + memset(&ssup, 0, sizeof(ssup)); + ssup.ssup_cxt = CurrentMemoryContext; + + /* We always use the default collation for statistics */ + ssup.ssup_collation = DEFAULT_COLLATION_OID; + ssup.ssup_nulls_first = false; + + PrepareSortSupportFromOrderingOp(tmp->ltopr, &ssup); + + mss->ssup[sortdim] = ssup; +} + +/* compare all the dimensions in the selected order */ +int +multi_sort_compare(const void *a, const void *b, void *arg) +{ + int i; + SortItem *ia = (SortItem*)a; + SortItem *ib = (SortItem*)b; + + MultiSortSupport mss = (MultiSortSupport)arg; + + for (i = 0; i < mss->ndims; i++) + { + int compare; + + compare = ApplySortComparator(ia->values[i], ia->isnull[i], + ib->values[i], ib->isnull[i], + &mss->ssup[i]); + + if (compare != 0) + return compare; + + } + + /* equal by default */ + return 0; +} + +/* compare selected dimension */ +int +multi_sort_compare_dim(int dim, const SortItem *a, const SortItem *b, + MultiSortSupport mss) +{ + return ApplySortComparator(a->values[dim], a->isnull[dim], + b->values[dim], b->isnull[dim], + &mss->ssup[dim]); +} diff --git a/src/backend/utils/mvstats/common.h b/src/backend/utils/mvstats/common.h new file mode 100644 index 0000000..6d5465b --- /dev/null +++ b/src/backend/utils/mvstats/common.h @@ -0,0 +1,75 @@ +/*------------------------------------------------------------------------- + * + * common.h + * POSTGRES multivariate statistics + * + * + * Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * + * IDENTIFICATION + * src/backend/utils/mvstats/common.h + * + *------------------------------------------------------------------------- + */ + +#include "postgres.h" + +#include "access/tuptoaster.h" +#include "catalog/indexing.h" +#include "catalog/pg_collation.h" +#include "catalog/pg_mv_statistic.h" +#include "foreign/fdwapi.h" +#include "postmaster/autovacuum.h" +#include "storage/lmgr.h" +#include "utils/datum.h" +#include "utils/sortsupport.h" +#include "utils/syscache.h" +#include "utils/fmgroids.h" +#include "utils/builtins.h" +#include "access/sysattr.h" + +#include "utils/mvstats.h" + +/* FIXME private structure copied from analyze.c */ + +typedef struct +{ + Oid eqopr; /* '=' operator for datatype, if any */ + Oid eqfunc; /* and associated function */ + Oid ltopr; /* '<' operator for datatype, if any */ +} StdAnalyzeData; + +typedef struct +{ + Datum value; /* a data value */ + int tupno; /* position index for tuple it came from */ +} ScalarItem; + +/* multi-sort */ +typedef struct MultiSortSupportData { + int ndims; /* number of dimensions supported by the */ + SortSupportData ssup[1]; /* sort support data for each dimension */ +} MultiSortSupportData; + +typedef MultiSortSupportData* MultiSortSupport; + +typedef struct SortItem { + Datum *values; + bool *isnull; +} SortItem; + +MultiSortSupport multi_sort_init(int ndims); + +void multi_sort_add_dimension(MultiSortSupport mss, int sortdim, + int dim, VacAttrStats **vacattrstats); + +int multi_sort_compare(const void *a, const void *b, void *arg); + +int multi_sort_compare_dim(int dim, const SortItem *a, + const SortItem *b, MultiSortSupport mss); + +/* comparators, used when constructing multivariate stats */ +int compare_scalars_simple(const void *a, const void *b, void *arg); +int compare_scalars_partition(const void *a, const void *b, void *arg); diff --git a/src/backend/utils/mvstats/dependencies.c b/src/backend/utils/mvstats/dependencies.c new file mode 100644 index 0000000..84b6561 --- /dev/null +++ b/src/backend/utils/mvstats/dependencies.c @@ -0,0 +1,638 @@ +/*------------------------------------------------------------------------- + * + * dependencies.c + * POSTGRES multivariate functional dependencies + * + * + * Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * + * IDENTIFICATION + * src/backend/utils/mvstats/dependencies.c + * + *------------------------------------------------------------------------- + */ + +#include "common.h" +#include "utils/lsyscache.h" + +/* + * Mine functional dependencies between columns, in the form (A => B), + * meaning that a value in column 'A' determines value in 'B'. A simple + * artificial example may be a table created like this + * + * CREATE TABLE deptest (a INT, b INT) + * AS SELECT i, i/10 FROM generate_series(1,100000) s(i); + * + * Clearly, once we know the value for 'A' we can easily determine the + * value of 'B' by dividing (A/10). A more practical example may be + * addresses, where (ZIP code => city name), i.e. once we know the ZIP, + * we probably know which city it belongs to. Larger cities usually have + * multiple ZIP codes, so the dependency can't be reversed. + * + * Functional dependencies are a concept well described in relational + * theory, especially in definition of normalization and "normal forms". + * Wikipedia has a nice definition of a functional dependency [1]: + * + * In a given table, an attribute Y is said to have a functional + * dependency on a set of attributes X (written X -> Y) if and only + * if each X value is associated with precisely one Y value. For + * example, in an "Employee" table that includes the attributes + * "Employee ID" and "Employee Date of Birth", the functional + * dependency {Employee ID} -> {Employee Date of Birth} would hold. + * It follows from the previous two sentences that each {Employee ID} + * is associated with precisely one {Employee Date of Birth}. + * + * [1] http://en.wikipedia.org/wiki/Database_normalization + * + * Most datasets might be normalized not to contain any such functional + * dependencies, but sometimes it's not practical. In some cases it's + * actually a conscious choice to model the dataset in denormalized way, + * either because of performance or to make querying easier. + * + * The current implementation supports only dependencies between two + * columns, but this is merely a simplification of the initial patch. + * It's certainly useful to mine for dependencies involving multiple + * columns on the 'left' side, i.e. a condition for the dependency. + * That is dependencies [A,B] => C and so on. + * + * TODO The implementation may/should be smart enough not to mine both + * [A => B] and [A,C => B], because the second dependency is a + * consequence of the first one (if values of A determine values + * of B, adding another column won't change that). The ANALYZE + * should first analyze 1:1 dependencies, then 2:1 dependencies + * (and skip the already identified ones), etc. + * + * For example the dependency [city name => zip code] is much weaker + * than [city name, state name => zip code], because there may be + * multiple cities with the same name in various states. It's not + * perfect though - there are probably cities with the same name within + * the same state, but this is relatively rare occurence hopefully. + * More about this in the section about dependency mining. + * + * Handling multiple columns on the right side is not necessary, as such + * dependencies may be decomposed into a set of dependencies with + * the same meaning, one for each column on the right side. For example + * + * A => [B,C] + * + * is exactly the same as + * + * (A => B) & (A => C). + * + * Of course, storing (A => [B, C]) may be more efficient thant storing + * the two dependencies (A => B) and (A => C) separately. + * + * + * Dependency mining (ANALYZE) + * --------------------------- + * + * The current build algorithm is rather simple - for each pair [A,B] of + * columns, the data are sorted lexicographically (first by A, then B), + * and then a number of metrics is computed by walking the sorted data. + * + * In general the algorithm counts distict values of A (forming groups + * thanks to the sorting), supporting or contradicting the hypothesis + * that A => B (i.e. that values of B are predetermined by A). If there + * are multiple values of B for a single value of A, it's counted as + * contradicting. + * + * A group may be neither supporting nor contradicting. To be counted as + * supporting, the group has to have at least min_group_size(=3) rows. + * Smaller 'supporting' groups are counted as neutral. + * + * Finally, the number of rows in supporting and contradicting groups is + * compared, and if there is at least 10x more supporting rows, the + * dependency is considered valid. + * + * + * Real-world datasets are imperfect - there may be errors (e.g. due to + * data-entry mistakes), or factually correct records, yet contradicting + * the dependency (e.g. when a city splits into two, but both keep the + * same ZIP code). A strict ANALYZE implementation (where the functional + * dependencies are identified) would ignore dependencies on such noisy + * data, making the approach unusable in practice. + * + * The proposed implementation attempts to handle such noisy cases + * gracefully, by tolerating small number of contradicting cases. + * + * In the future this might also perform some sort of test and decide + * whether it's worth building any other kind of multivariate stats, + * or whether the dependencies sufficiently describe the data. Or at + * least not build the MCV list / histogram on the implied columns. + * Such reduction would however make the 'verification' (see the next + * section) impossible. + * + * + * Clause reduction (planner/optimizer) + * ------------------------------------ + * + * Apllying the dependencies is quite simple - given a list of clauses, + * try to apply all the dependencies. For example given clause list + * + * (a = 1) AND (b = 1) AND (c = 1) AND (d < 100) + * + * and dependencies [a=>b] and [a=>d], this may be reduced to + * + * (a = 1) AND (c = 1) AND (d < 100) + * + * The (d<100) can't be reduced as it's not an equality clause, so the + * dependency [a=>d] can't be applied. + * + * See clauselist_apply_dependencies() for more details. + * + * The problem with the reduction is that the query may use conditions + * that are not redundant, but in fact contradictory - e.g. the user + * may search for a ZIP code and a city name not matching the ZIP code. + * + * In such cases, the condition on the city name is not actually + * redundant, but actually contradictory (making the result empty), and + * removing it while estimating the cardinality will make the estimate + * worse. + * + * The current estimation assuming independence (and multiplying the + * selectivities) works better in this case, but only by utter luck. + * + * In some cases this might be verified using the other multivariate + * statistics - MCV lists and histograms. For MCV lists the verification + * might be very simple - peek into the list if there are any items + * matching the clause on the 'A' column (e.g. ZIP code), and if such + * item is found, check that the 'B' column matches the other clause. + * If it does not, the clauses are contradictory. We can't really say + * if such item was not found, except maybe restricting the selectivity + * using the MCV data (e.g. using min/max selectivity, or something). + * + * With histograms, it might work similarly - we can't check the values + * directly (because histograms use buckets, unlike MCV lists, storing + * the actual values). So we can only observe the buckets matching the + * clauses - if those buckets have very low frequency, it probably means + * the two clauses are incompatible. + * + * It's unclear what 'low frequency' is, but if one of the clauses is + * implied (automatically true because of the other clause), then + * + * selectivity[clause(A)] = selectivity[clause(A) & clause(B)] + * + * So we might compute selectivity of the first clause (on the column + * A in dependency [A=>B]) - for example using regular statistics. + * And then check if the selectivity computed from the histogram is + * about the same (or significantly lower). + * + * The problem is that histograms work well only when the data ordering + * matches the natural meaning. For values that serve as labels - like + * city names or ZIP codes, or even generated IDs, histograms really + * don't work all that well. For example sorting cities by name won't + * match the sorting of ZIP codes, rendering the histogram unusable. + * + * The MCV are probably going to work much better, because they don't + * really assume any sort of ordering. And it's probably more appropriate + * for the label-like data. + * + * TODO Support dependencies with multiple columns on left/right. + * + * TODO Investigate using histogram and MCV list to confirm the + * functional dependencies. + * + * TODO Investigate statistical testing of the distribution (to decide + * whether it makes sense to build the histogram/MCV list). + * + * TODO Using a min/max of selectivities would probably make more sense + * for the associated columns. + * + * TODO Consider eliminating the implied columns from the histogram and + * MCV lists (but maybe that's not a good idea, because that'd make + * it impossible to use these stats for non-equality clauses and + * also it wouldn't be possible to use the stats for verification + * of the dependencies as proposed in another TODO). + * + * TODO This builds a complete set of dependencies, i.e. including + * transitive dependencies - if we identify [A => B] and [B => C], + * we're likely to identify [A => C] too. It might be better to + * keep only the minimal set of dependencies, i.e. prune all the + * dependencies that we can recreate by transivitity. + * + * There are two conceptual ways to do that: + * + * (a) generate all the rules, and then prune the rules that may + * be recteated by combining other dependencies, or + * + * (b) performing the 'is combination of other dependencies' check + * before actually doing the work + * + * The second option has the advantage that we don't really need + * to perform the sort/count. It's not sufficient alone, though, + * because we may discover the dependencies in the wrong order. + * For example [A => B], [A => C] and then [B => C]. None of those + * dependencies is a combination of the already known ones, yet + * [A => C] is a combination of [A => B] and [B => C]. + * + * FIXME Not sure the current NULL handling makes much sense. We assume + * that NULL is 0, so it's handled like a regular value + * (NULL == NULL), so all NULLs in a single column form a single + * group. Maybe that's not the right thing to do, especially with + * equality conditions - in that case NULLs are irrelevant. So + * maybe the right solution would be to just ignore NULL values? + * + * However simply "ignoring" the NULL values does not seem like + * a good idea - imagine columns A and B, where for each value of + * A, values in B are constant (same for the whole group) or NULL. + * Let's say only 10% of B values in each group is not NULL. Then + * ignoring the NULL values will result in 10x misestimate (and + * it's trivial to construct arbitrary errors). So maybe handling + * NULL values just like a regular value is the right thing here. + * + * Or maybe NULL values should be treated differently on each side + * of the dependency? E.g. as ignored on the left (condition) and + * as regular values on the right - this seems consistent with how + * equality clauses work, as equality clause means 'NOT NULL'. + * So if we say [A => B] then it may also imply "NOT NULL" on the + * right side. + */ +MVDependencies +build_mv_dependencies(int numrows, HeapTuple *rows, int2vector *attrs, + VacAttrStats **stats) +{ + int i; + int numattrs = attrs->dim1; + + /* result */ + int ndeps = 0; + MVDependencies dependencies = NULL; + MultiSortSupport mss = multi_sort_init(2); /* 2 dimensions for now */ + + /* TODO Maybe this should be somehow related to the number of + * distinct values in the two columns we're currently analyzing. + * Assuming the distribution is uniform, we can estimate the + * average group size and use it as a threshold. Or something + * like that. Seems better than a static approach. + */ + int min_group_size = 3; + + /* dimension indexes we'll check for associations [a => b] */ + int dima, dimb; + + /* + * We'll reuse the same array for all the 2-column combinations. + * + * It's possible to sort the sample rows directly, but this seemed + * somehow simples / less error prone. Another option would be to + * allocate the arrays for each SortItem separately, but that'd be + * significant overhead (not just CPU, but especially memory bloat). + */ + SortItem * items = (SortItem*)palloc0(numrows * sizeof(SortItem)); + + Datum *values = (Datum*)palloc0(sizeof(Datum) * numrows * 2); + bool *isnull = (bool*)palloc0(sizeof(bool) * numrows * 2); + + for (i = 0; i < numrows; i++) + { + items[i].values = &values[i * 2]; + items[i].isnull = &isnull[i * 2]; + } + + Assert(numattrs >= 2); + + /* + * Evaluate all possible combinations of [A => B], using a simple algorithm: + * + * (a) sort the data by [A,B] + * (b) split the data into groups by A (new group whenever a value changes) + * (c) count different values in the B column (again, value changes) + * + * TODO It should be rather simple to merge [A => B] and [A => C] into + * [A => B,C]. Just keep A constant, collect all the "implied" columns + * and you're done. + */ + for (dima = 0; dima < numattrs; dima++) + { + /* prepare the sort function for the first dimension */ + multi_sort_add_dimension(mss, 0, dima, stats); + + for (dimb = 0; dimb < numattrs; dimb++) + { + SortItem current; + + /* number of groups supporting / contradicting the dependency */ + int n_supporting = 0; + int n_contradicting = 0; + + /* counters valid within a group */ + int group_size = 0; + int n_violations = 0; + + int n_supporting_rows = 0; + int n_contradicting_rows = 0; + + /* make sure the columns are different (A => A) */ + if (dima == dimb) + continue; + + /* prepare the sort function for the second dimension */ + multi_sort_add_dimension(mss, 1, dimb, stats); + + /* reset the values and isnull flags */ + memset(values, 0, sizeof(Datum) * numrows * 2); + memset(isnull, 0, sizeof(bool) * numrows * 2); + + /* accumulate all the data for both columns into an array and sort it */ + for (i = 0; i < numrows; i++) + { + items[i].values[0] + = heap_getattr(rows[i], attrs->values[dima], + stats[dima]->tupDesc, &items[i].isnull[0]); + + items[i].values[1] + = heap_getattr(rows[i], attrs->values[dimb], + stats[dimb]->tupDesc, &items[i].isnull[1]); + } + + qsort_arg((void *) items, numrows, sizeof(SortItem), + multi_sort_compare, mss); + + /* + * Walk through the array, split it into rows according to + * the A value, and count distinct values in the other one. + * If there's a single B value for the whole group, we count + * it as supporting the association, otherwise we count it + * as contradicting. + * + * Furthermore we require a group to have at least a certain + * number of rows to be considered useful for supporting the + * dependency. But when it's contradicting, use it always useful. + */ + + /* start with values from the first row */ + current = items[0]; + group_size = 1; + + for (i = 1; i < numrows; i++) + { + /* end of the group */ + if (multi_sort_compare_dim(0, &items[i], ¤t, mss) != 0) + { + /* + * If there are no contradicting rows, count it as + * supporting (otherwise contradicting), but only if + * the group is large enough. + * + * The requirement of a minimum group size makes it + * impossible to identify [unique,unique] cases, but + * that's probably a different case. This is more + * about [zip => city] associations etc. + * + * If there are violations, count the group/rows as + * a violation. + * + * It may ne neither, if the group is too small (does + * not contain at least min_group_size rows). + */ + if ((n_violations == 0) && (group_size >= min_group_size)) + { + n_supporting += 1; + n_supporting_rows += group_size; + } + else if (n_violations > 0) + { + n_contradicting += 1; + n_contradicting_rows += group_size; + } + + /* current values start a new group */ + n_violations = 0; + group_size = 0; + } + /* mismatch of a B value is contradicting */ + else if (multi_sort_compare_dim(1, &items[i], ¤t, mss) != 0) + { + n_violations += 1; + } + + current = items[i]; + group_size += 1; + } + + /* handle the last group (just like above) */ + if ((n_violations == 0) && (group_size >= min_group_size)) + { + n_supporting += 1; + n_supporting_rows += group_size; + } + else if (n_violations) + { + n_contradicting += 1; + n_contradicting_rows += group_size; + } + + /* + * See if the number of rows supporting the association is at least + * 10x the number of rows violating the hypothetical dependency. + * + * TODO This is rather arbitrary limit - I guess it's possible to do + * some math to come up with a better rule (e.g. testing a hypothesis + * 'this is due to randomness'). We can create a contingency table + * from the values and use it for testing. Possibly only when + * there are no contradicting rows? + * + * TODO Also, if (a => b) and (b => a) at the same time, it pretty much + * means there's a 1:1 relation (or one is a 'label'), making the + * conditions rather redundant. Although it's possible that the + * query uses incompatible combination of values. + */ + if (n_supporting_rows > (n_contradicting_rows * 10)) + { + if (dependencies == NULL) + { + dependencies = (MVDependencies)palloc0(sizeof(MVDependenciesData)); + dependencies->magic = MVSTAT_DEPS_MAGIC; + } + else + dependencies = repalloc(dependencies, offsetof(MVDependenciesData, deps) + + sizeof(MVDependency) * (dependencies->ndeps + 1)); + + /* update the */ + dependencies->deps[ndeps] = (MVDependency)palloc0(sizeof(MVDependencyData)); + dependencies->deps[ndeps]->a = attrs->values[dima]; + dependencies->deps[ndeps]->b = attrs->values[dimb]; + + dependencies->ndeps = (++ndeps); + } + } + } + + pfree(items); + pfree(values); + pfree(isnull); + pfree(stats); + pfree(mss); + + return dependencies; +} + +/* + * Store the dependencies into a bytea, so that it can be stored in the + * pg_mv_statistic catalog. + * + * Currently this only supports simple two-column rules, and stores them + * as a sequence of attnum pairs. In the future, this needs to be made + * more complex to support multiple columns on both sides of the + * implication (using AND on left, OR on right). + */ +bytea * +serialize_mv_dependencies(MVDependencies dependencies) +{ + int i; + + /* we need to store ndeps, and each needs 2 * int16 */ + Size len = VARHDRSZ + offsetof(MVDependenciesData, deps) + + dependencies->ndeps * (sizeof(int16) * 2); + + bytea * output = (bytea*)palloc0(len); + + char * tmp = VARDATA(output); + + SET_VARSIZE(output, len); + + /* first, store the number of dimensions / items */ + memcpy(tmp, dependencies, offsetof(MVDependenciesData, deps)); + tmp += offsetof(MVDependenciesData, deps); + + /* walk through the dependencies and copy both columns into the bytea */ + for (i = 0; i < dependencies->ndeps; i++) + { + memcpy(tmp, &(dependencies->deps[i]->a), sizeof(int16)); + tmp += sizeof(int16); + + memcpy(tmp, &(dependencies->deps[i]->b), sizeof(int16)); + tmp += sizeof(int16); + } + + return output; +} + +/* + * Reads serialized dependencies into MVDependencies structure. + */ +MVDependencies +deserialize_mv_dependencies(bytea * data) +{ + int i; + Size expected_size; + MVDependencies dependencies; + char *tmp; + + if (data == NULL) + return NULL; + + if (VARSIZE_ANY_EXHDR(data) < offsetof(MVDependenciesData,deps)) + elog(ERROR, "invalid MVDependencies size %ld (expected at least %ld)", + VARSIZE_ANY_EXHDR(data), offsetof(MVDependenciesData,deps)); + + /* read the MVDependencies header */ + dependencies = (MVDependencies)palloc0(sizeof(MVDependenciesData)); + + /* initialize pointer to the data part (skip the varlena header) */ + tmp = VARDATA(data); + + /* get the header and perform basic sanity checks */ + memcpy(dependencies, tmp, offsetof(MVDependenciesData, deps)); + tmp += offsetof(MVDependenciesData, deps); + + if (dependencies->magic != MVSTAT_DEPS_MAGIC) + { + pfree(dependencies); + elog(WARNING, "not a MV Dependencies (magic number mismatch)"); + return NULL; + } + + Assert(dependencies->ndeps > 0); + + /* what bytea size do we expect for those parameters */ + expected_size = offsetof(MVDependenciesData,deps) + + dependencies->ndeps * sizeof(int16) * 2; + + if (VARSIZE_ANY_EXHDR(data) != expected_size) + elog(ERROR, "invalid dependencies size %ld (expected %ld)", + VARSIZE_ANY_EXHDR(data), expected_size); + + /* allocate space for the MCV items */ + dependencies = repalloc(dependencies, offsetof(MVDependenciesData,deps) + + (dependencies->ndeps * sizeof(MVDependency))); + + for (i = 0; i < dependencies->ndeps; i++) + { + dependencies->deps[i] = (MVDependency)palloc0(sizeof(MVDependencyData)); + + memcpy(&(dependencies->deps[i]->a), tmp, sizeof(int16)); + tmp += sizeof(int16); + + memcpy(&(dependencies->deps[i]->b), tmp, sizeof(int16)); + tmp += sizeof(int16); + } + + return dependencies; +} + +/* print some basic info about dependencies (number of dependencies) */ +Datum +pg_mv_stats_dependencies_info(PG_FUNCTION_ARGS) +{ + bytea *data = PG_GETARG_BYTEA_P(0); + char *result; + + MVDependencies dependencies = deserialize_mv_dependencies(data); + + if (dependencies == NULL) + PG_RETURN_NULL(); + + result = palloc0(128); + snprintf(result, 128, "dependencies=%d", dependencies->ndeps); + + /* FIXME free the deserialized data (pfree is not enough) */ + + PG_RETURN_TEXT_P(cstring_to_text(result)); +} + +/* print the dependencies + * + * TODO Would be nice if this knew the actual column names (instead of + * the attnums). + * + * FIXME This is really ugly and does not really check the lengths and + * strcpy/snprintf return values properly. Needs to be fixed. + */ +Datum +pg_mv_stats_dependencies_show(PG_FUNCTION_ARGS) +{ + int i = 0; + bytea *data = PG_GETARG_BYTEA_P(0); + char *result = NULL; + int len = 0; + + MVDependencies dependencies = deserialize_mv_dependencies(data); + + if (dependencies == NULL) + PG_RETURN_NULL(); + + for (i = 0; i < dependencies->ndeps; i++) + { + MVDependency dependency = dependencies->deps[i]; + char buffer[128]; + + int tmp = snprintf(buffer, 128, "%s%d => %d", + ((i == 0) ? "" : ", "), dependency->a, dependency->b); + + if (tmp < 127) + { + if (result == NULL) + result = palloc0(len + tmp + 1); + else + result = repalloc(result, len + tmp + 1); + + strcpy(result + len, buffer); + len += tmp; + } + } + + PG_RETURN_TEXT_P(cstring_to_text(result)); +} diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index db56809..912b4f3 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -2096,6 +2096,46 @@ describeOneTableDetails(const char *schemaname, PQclear(result); } + /* print any multivariate statistics */ + if (pset.sversion >= 90500) + { + printfPQExpBuffer(&buf, + "SELECT oid, stakeys,\n" + " deps_enabled,\n" + " deps_built,\n" + " mcv_max_items, hist_max_buckets,\n" + " (SELECT string_agg(attname::text,', ')\n" + " FROM ((SELECT unnest(stakeys) AS attnum) s\n" + " JOIN pg_attribute a ON (starelid = a.attrelid and a.attnum = s.attnum))) AS attnums\n" + "FROM pg_mv_statistic stat WHERE starelid = '%s' ORDER BY 1;", + oid); + + result = PSQLexec(buf.data); + if (!result) + goto error_return; + else + tuples = PQntuples(result); + + if (tuples > 0) + { + printTableAddFooter(&cont, _("Statistics:")); + for (i = 0; i < tuples; i++) + { + printfPQExpBuffer(&buf, " "); + + /* options */ + if (!strcmp(PQgetvalue(result, i, 2), "t")) + appendPQExpBuffer(&buf, "(dependencies)"); + + appendPQExpBuffer(&buf, " ON (%s)", + PQgetvalue(result, i, 6)); + + printTableAddFooter(&cont, buf.data); + } + } + PQclear(result); + } + /* print rules */ if (tableinfo.hasrules && tableinfo.relkind != 'm') { diff --git a/src/include/catalog/heap.h b/src/include/catalog/heap.h index e6ac394..36debeb 100644 --- a/src/include/catalog/heap.h +++ b/src/include/catalog/heap.h @@ -119,6 +119,7 @@ extern void RemoveAttrDefault(Oid relid, AttrNumber attnum, DropBehavior behavior, bool complain, bool internal); extern void RemoveAttrDefaultById(Oid attrdefId); extern void RemoveStatistics(Oid relid, AttrNumber attnum); +extern void RemoveMVStatistics(Oid relid, AttrNumber attnum); extern Form_pg_attribute SystemAttributeDefinition(AttrNumber attno, bool relhasoids); diff --git a/src/include/catalog/indexing.h b/src/include/catalog/indexing.h index 748aadd..03ada1b 100644 --- a/src/include/catalog/indexing.h +++ b/src/include/catalog/indexing.h @@ -173,6 +173,11 @@ DECLARE_UNIQUE_INDEX(pg_largeobject_loid_pn_index, 2683, on pg_largeobject using DECLARE_UNIQUE_INDEX(pg_largeobject_metadata_oid_index, 2996, on pg_largeobject_metadata using btree(oid oid_ops)); #define LargeObjectMetadataOidIndexId 2996 +DECLARE_UNIQUE_INDEX(pg_mv_statistic_oid_index, 3380, on pg_mv_statistic using btree(oid oid_ops)); +#define MvStatisticOidIndexId 3380 +DECLARE_INDEX(pg_mv_statistic_relid_index, 3379, on pg_mv_statistic using btree(starelid oid_ops)); +#define MvStatisticRelidIndexId 3379 + DECLARE_UNIQUE_INDEX(pg_namespace_nspname_index, 2684, on pg_namespace using btree(nspname name_ops)); #define NamespaceNameIndexId 2684 DECLARE_UNIQUE_INDEX(pg_namespace_oid_index, 2685, on pg_namespace using btree(oid oid_ops)); diff --git a/src/include/catalog/pg_mv_statistic.h b/src/include/catalog/pg_mv_statistic.h new file mode 100644 index 0000000..81ec23b --- /dev/null +++ b/src/include/catalog/pg_mv_statistic.h @@ -0,0 +1,69 @@ +/*------------------------------------------------------------------------- + * + * pg_mv_statistic.h + * definition of the system "multivariate statistic" relation (pg_mv_statistic) + * along with the relation's initial contents. + * + * + * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * src/include/catalog/pg_mv_statistic.h + * + * NOTES + * the genbki.pl script reads this file and generates .bki + * information from the DATA() statements. + * + *------------------------------------------------------------------------- + */ +#ifndef PG_MV_STATISTIC_H +#define PG_MV_STATISTIC_H + +#include "catalog/genbki.h" + +/* ---------------- + * pg_mv_statistic definition. cpp turns this into + * typedef struct FormData_pg_mv_statistic + * ---------------- + */ +#define MvStatisticRelationId 3381 + +CATALOG(pg_mv_statistic,3381) +{ + /* These fields form the unique key for the entry: */ + Oid starelid; /* relation containing attributes */ + + /* statistics requested to build */ + bool deps_enabled; /* analyze dependencies? */ + + /* statistics that are available (if requested) */ + bool deps_built; /* dependencies were built */ + + /* variable-length fields start here, but we allow direct access to stakeys */ + int2vector stakeys; /* array of column keys */ + +#ifdef CATALOG_VARLEN + bytea stadeps; /* dependencies (serialized) */ +#endif + +} FormData_pg_mv_statistic; + +/* ---------------- + * Form_pg_mv_statistic corresponds to a pointer to a tuple with + * the format of pg_mv_statistic relation. + * ---------------- + */ +typedef FormData_pg_mv_statistic *Form_pg_mv_statistic; + +/* ---------------- + * compiler constants for pg_attrdef + * ---------------- + */ +#define Natts_pg_mv_statistic 5 +#define Anum_pg_mv_statistic_starelid 1 +#define Anum_pg_mv_statistic_deps_enabled 2 +#define Anum_pg_mv_statistic_deps_built 3 +#define Anum_pg_mv_statistic_stakeys 4 +#define Anum_pg_mv_statistic_stadeps 5 + +#endif /* PG_MV_STATISTIC_H */ diff --git a/src/include/catalog/pg_proc.h b/src/include/catalog/pg_proc.h index c0aab38..69fc482 100644 --- a/src/include/catalog/pg_proc.h +++ b/src/include/catalog/pg_proc.h @@ -2735,6 +2735,11 @@ DESCR("current user privilege on any column by rel name"); DATA(insert OID = 3029 ( has_any_column_privilege PGNSP PGUID 12 10 0 0 0 f f f f t f s 2 0 16 "26 25" _null_ _null_ _null_ _null_ _null_ has_any_column_privilege_id _null_ _null_ _null_ )); DESCR("current user privilege on any column by rel oid"); +DATA(insert OID = 3307 ( pg_mv_stats_dependencies_info PGNSP PGUID 12 1 0 0 0 f f f f t f i 1 0 25 "17" _null_ _null_ _null_ _null_ _null_ pg_mv_stats_dependencies_info _null_ _null_ _null_ )); +DESCR("multivariate stats: functional dependencies info"); +DATA(insert OID = 3308 ( pg_mv_stats_dependencies_show PGNSP PGUID 12 1 0 0 0 f f f f t f i 1 0 25 "17" _null_ _null_ _null_ _null_ _null_ pg_mv_stats_dependencies_show _null_ _null_ _null_ )); +DESCR("multivariate stats: functional dependencies show"); + DATA(insert OID = 1928 ( pg_stat_get_numscans PGNSP PGUID 12 1 0 0 0 f f f f t f s 1 0 20 "26" _null_ _null_ _null_ _null_ _null_ pg_stat_get_numscans _null_ _null_ _null_ )); DESCR("statistics: number of scans done for table/index"); DATA(insert OID = 1929 ( pg_stat_get_tuples_returned PGNSP PGUID 12 1 0 0 0 f f f f t f s 1 0 20 "26" _null_ _null_ _null_ _null_ _null_ pg_stat_get_tuples_returned _null_ _null_ _null_ )); diff --git a/src/include/catalog/toasting.h b/src/include/catalog/toasting.h index fb2f035..55f6079 100644 --- a/src/include/catalog/toasting.h +++ b/src/include/catalog/toasting.h @@ -49,6 +49,7 @@ extern void BootstrapToastTable(char *relName, DECLARE_TOAST(pg_attrdef, 2830, 2831); DECLARE_TOAST(pg_constraint, 2832, 2833); DECLARE_TOAST(pg_description, 2834, 2835); +DECLARE_TOAST(pg_mv_statistic, 3309, 3310); DECLARE_TOAST(pg_proc, 2836, 2837); DECLARE_TOAST(pg_rewrite, 2838, 2839); DECLARE_TOAST(pg_seclabel, 3598, 3599); diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 290cdb3..9254f85 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -249,6 +249,7 @@ typedef enum NodeTag T_PlaceHolderInfo, T_MinMaxAggInfo, T_PlannerParamItem, + T_MVStatisticInfo, /* * TAGS FOR MEMORY NODES (memnodes.h) @@ -426,6 +427,7 @@ typedef enum NodeTag T_RoleSpec, T_RangeTableSample, T_TableSampleClause, + T_StatisticsDef, /* * TAGS FOR REPLICATION GRAMMAR PARSE NODES (replnodes.h) diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 868905b..d81537c 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -610,6 +610,14 @@ typedef struct ColumnDef int location; /* parse location, or -1 if none/unknown */ } ColumnDef; +typedef struct StatisticsDef +{ + NodeTag type; + List *keys; /* String nodes naming referenced column(s) */ + List *options; /* list of DefElem nodes */ +} StatisticsDef; + + /* * TableLikeClause - CREATE TABLE ( ... LIKE ... ) clause */ @@ -1515,7 +1523,9 @@ typedef enum AlterTableType AT_ReplicaIdentity, /* REPLICA IDENTITY */ AT_EnableRowSecurity, /* ENABLE ROW SECURITY */ AT_DisableRowSecurity, /* DISABLE ROW SECURITY */ - AT_GenericOptions /* OPTIONS (...) */ + AT_GenericOptions, /* OPTIONS (...) */ + AT_AddStatistics, /* ADD STATISTICS */ + AT_DropStatistics /* DROP STATISTICS */ } AlterTableType; typedef struct ReplicaIdentityStmt diff --git a/src/include/nodes/relation.h b/src/include/nodes/relation.h index 279051e..10f7425 100644 --- a/src/include/nodes/relation.h +++ b/src/include/nodes/relation.h @@ -459,6 +459,7 @@ typedef struct RelOptInfo Relids lateral_relids; /* minimum parameterization of rel */ Relids lateral_referencers; /* rels that reference me laterally */ List *indexlist; /* list of IndexOptInfo */ + List *mvstatlist; /* list of MVStatisticInfo */ BlockNumber pages; /* size estimates derived from pg_class */ double tuples; double allvisfrac; @@ -553,6 +554,33 @@ typedef struct IndexOptInfo bool amhasgetbitmap; /* does AM have amgetbitmap interface? */ } IndexOptInfo; +/* + * MVStatisticInfo + * Information about multivariate stats for planning/optimization + * + * This contains information about which columns are covered by the + * statistics (stakeys), which options were requested while adding the + * statistics (*_enabled), and which kinds of statistics were actually + * built and are available for the optimizer (*_built). + */ +typedef struct MVStatisticInfo +{ + NodeTag type; + + Oid mvoid; /* OID of the statistics row */ + RelOptInfo *rel; /* back-link to index's table */ + + /* enabled statistics */ + bool deps_enabled; /* functional dependencies enabled */ + + /* built/available statistics */ + bool deps_built; /* functional dependencies built */ + + /* columns in the statistics (attnums) */ + int2vector *stakeys; /* attnums of the columns covered */ + +} MVStatisticInfo; + /* * EquivalenceClasses diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 2414069..f69480b 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -360,7 +360,7 @@ PG_KEYWORD("stable", STABLE, UNRESERVED_KEYWORD) PG_KEYWORD("standalone", STANDALONE_P, UNRESERVED_KEYWORD) PG_KEYWORD("start", START, UNRESERVED_KEYWORD) PG_KEYWORD("statement", STATEMENT, UNRESERVED_KEYWORD) -PG_KEYWORD("statistics", STATISTICS, UNRESERVED_KEYWORD) +PG_KEYWORD("statistics", STATISTICS, RESERVED_KEYWORD) PG_KEYWORD("stdin", STDIN, UNRESERVED_KEYWORD) PG_KEYWORD("stdout", STDOUT, UNRESERVED_KEYWORD) PG_KEYWORD("storage", STORAGE, UNRESERVED_KEYWORD) diff --git a/src/include/utils/mvstats.h b/src/include/utils/mvstats.h new file mode 100644 index 0000000..411cd16 --- /dev/null +++ b/src/include/utils/mvstats.h @@ -0,0 +1,69 @@ +/*------------------------------------------------------------------------- + * + * mvstats.h + * Multivariate statistics and selectivity estimation functions. + * + * + * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * src/include/utils/mvstats.h + * + *------------------------------------------------------------------------- + */ +#ifndef MVSTATS_H +#define MVSTATS_H + +#include "commands/vacuum.h" + + +#define MVSTATS_MAX_DIMENSIONS 8 /* max number of attributes */ + +/* An associative rule, tracking [a => b] dependency. + * + * TODO Make this work with multiple columns on both sides. + */ +typedef struct MVDependencyData { + int16 a; + int16 b; +} MVDependencyData; + +typedef MVDependencyData* MVDependency; + +typedef struct MVDependenciesData { + uint32 magic; /* magic constant marker */ + int32 ndeps; /* number of dependencies */ + MVDependency deps[1]; /* XXX why not a pointer? */ +} MVDependenciesData; + +typedef MVDependenciesData* MVDependencies; + +#define MVSTAT_DEPS_MAGIC 0xB4549A2C /* marks serialized bytea */ +#define MVSTAT_DEPS_TYPE_BASIC 1 /* basic dependencies type */ + +/* + * TODO Maybe fetching the histogram/MCV list separately is inefficient? + * Consider adding a single `fetch_stats` method, fetching all + * stats specified using flags (or something like that). + */ + +bytea * serialize_mv_dependencies(MVDependencies dependencies); + +/* deserialization of stats (serialization is private to analyze) */ +MVDependencies deserialize_mv_dependencies(bytea * data); + +/* FIXME this probably belongs somewhere else (not to operations stats) */ +extern Datum pg_mv_stats_dependencies_info(PG_FUNCTION_ARGS); +extern Datum pg_mv_stats_dependencies_show(PG_FUNCTION_ARGS); + +MVDependencies +build_mv_dependencies(int numrows, HeapTuple *rows, + int2vector *attrs, + VacAttrStats **stats); + +void build_mv_stats(Relation onerel, int numrows, HeapTuple *rows, + int natts, VacAttrStats **vacattrstats); + +void update_mv_stats(Oid relid, MVDependencies dependencies, int2vector *attrs); + +#endif diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 8a55a09..4d6edb6 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -79,6 +79,7 @@ typedef struct RelationData bool rd_isvalid; /* relcache entry is valid */ char rd_indexvalid; /* state of rd_indexlist: 0 = not valid, 1 = * valid, 2 = temporarily forced */ + bool rd_mvstatvalid; /* state of rd_mvstatlist: true/false */ /* * rd_createSubid is the ID of the highest subtransaction the rel has @@ -111,6 +112,9 @@ typedef struct RelationData List *rd_indexlist; /* list of OIDs of indexes on relation */ Oid rd_oidindex; /* OID of unique index on OID, if any */ Oid rd_replidindex; /* OID of replica identity index, if any */ + + /* data managed by RelationGetMVStatList: */ + List *rd_mvstatlist; /* list of OIDs of multivariate stats */ /* data managed by RelationGetIndexAttrBitmap: */ Bitmapset *rd_indexattr; /* identifies columns used in indexes */ diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h index 6953281..77efeff 100644 --- a/src/include/utils/relcache.h +++ b/src/include/utils/relcache.h @@ -38,6 +38,7 @@ extern void RelationClose(Relation relation); * Routines to compute/retrieve additional cached information */ extern List *RelationGetIndexList(Relation relation); +extern List *RelationGetMVStatList(Relation relation); extern Oid RelationGetOidIndex(Relation relation); extern Oid RelationGetReplicaIndex(Relation relation); extern List *RelationGetIndexExpressions(Relation relation); diff --git a/src/include/utils/syscache.h b/src/include/utils/syscache.h index 2dbd384..814269b 100644 --- a/src/include/utils/syscache.h +++ b/src/include/utils/syscache.h @@ -66,6 +66,7 @@ enum SysCacheIdentifier INDEXRELID, LANGNAME, LANGOID, + MVSTATOID, NAMESPACENAME, NAMESPACEOID, OPERNAMENSP, diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out index 60c1f40..a12ad30 100644 --- a/src/test/regress/expected/rules.out +++ b/src/test/regress/expected/rules.out @@ -1363,6 +1363,14 @@ pg_matviews| SELECT n.nspname AS schemaname, LEFT JOIN pg_namespace n ON ((n.oid = c.relnamespace))) LEFT JOIN pg_tablespace t ON ((t.oid = c.reltablespace))) WHERE (c.relkind = 'm'::"char"); +pg_mv_stats| SELECT n.nspname AS schemaname, + c.relname AS tablename, + s.stakeys AS attnums, + length(s.stadeps) AS depsbytes, + pg_mv_stats_dependencies_info(s.stadeps) AS depsinfo + FROM ((pg_mv_statistic s + JOIN pg_class c ON ((c.oid = s.starelid))) + LEFT JOIN pg_namespace n ON ((n.oid = c.relnamespace))); pg_policies| SELECT n.nspname AS schemaname, c.relname AS tablename, pol.polname AS policyname, diff --git a/src/test/regress/expected/sanity_check.out b/src/test/regress/expected/sanity_check.out index 14acd16..d740241 100644 --- a/src/test/regress/expected/sanity_check.out +++ b/src/test/regress/expected/sanity_check.out @@ -113,6 +113,7 @@ pg_inherits|t pg_language|t pg_largeobject|t pg_largeobject_metadata|t +pg_mv_statistic|t pg_namespace|t pg_opclass|t pg_operator|t -- 1.9.3 --------------080401000304040708010509 Content-Type: text/x-patch; name="0003-clause-reduction-using-functional-dependencies-v7.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename*0="0003-clause-reduction-using-functional-dependencies-v7.patch" From 1bc8e278cf96a33bdb5716023ae9929e4c625893 Mon Sep 17 00:00:00 2001 From: Tomas Vondra Date: Mon, 6 Apr 2015 19:42:18 +0200 Subject: [PATCH 3/6] clause reduction using functional dependencies During planning, use functional dependencies to decide which clauses to skip during cardinality estimation. Initial and rather simplistic implementation. This only works with regular WHERE clauses, not clauses used for join clauses. Note: The clause_is_mv_compatible() needs to identify the relation (so that we can fetch the list of multivariate stats by OID). planner_rt_fetch() seems like the appropriate way to get the relation OID, but apparently it only works with simple vars. Maybe examine_variable() would make this work with more complex vars too? Includes regression tests analyzing functional dependencies (part of ANALYZE) on several datasets (no dependencies, no transitive dependencies, ...). Checks that a query with conditions on two columns, where one (B) is functionally dependent on the other one (A), correctly ignores the clause on (B) and chooses bitmap index scan instead of plain index scan (which is what happens otherwise, thanks to assumption of independence). Note: Functional dependencies only work with equality clauses, no inequalities etc. --- src/backend/commands/tablecmds.c | 6 + src/backend/nodes/copyfuncs.c | 1 + src/backend/optimizer/path/clausesel.c | 911 +++++++++++++++++++++++++- src/backend/utils/mvstats/common.c | 5 +- src/backend/utils/mvstats/dependencies.c | 24 + src/bin/psql/describe.c | 1 - src/include/utils/mvstats.h | 16 +- src/test/regress/expected/mv_dependencies.out | 172 +++++ src/test/regress/parallel_schedule | 3 + src/test/regress/regression.diffs | 30 + src/test/regress/regression.out | 156 +++++ src/test/regress/serial_schedule | 1 + src/test/regress/sql/mv_dependencies.sql | 150 +++++ 13 files changed, 1470 insertions(+), 6 deletions(-) create mode 100644 src/test/regress/expected/mv_dependencies.out create mode 100644 src/test/regress/regression.diffs create mode 100644 src/test/regress/regression.out create mode 100644 src/test/regress/sql/mv_dependencies.sql diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index d6c6f8e..107e9fc 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -11980,6 +11980,12 @@ static void ATExecAddStatistics(AlteredTableInfo *tab, Relation rel, opt->defname))); } + /* check that at least some statistics were requested */ + if (! build_dependencies) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("no statistics type (dependencies) was requested"))); + /* sort the attnums and build int2vector */ qsort(attnums, numcols, sizeof(int16), compare_int16); stakeys = buildint2vector(attnums, numcols); diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index e5a3d96..36094c0 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -4949,6 +4949,7 @@ copyObject(const void *from) break; case T_TableSampleClause: retval = _copyTableSampleClause(from); + break; case T_StatisticsDef: retval = _copyStatisticsDef(from); break; diff --git a/src/backend/optimizer/path/clausesel.c b/src/backend/optimizer/path/clausesel.c index dcac1c1..6365425 100644 --- a/src/backend/optimizer/path/clausesel.c +++ b/src/backend/optimizer/path/clausesel.c @@ -14,15 +14,19 @@ */ #include "postgres.h" +#include "access/sysattr.h" #include "catalog/pg_operator.h" #include "nodes/makefuncs.h" #include "optimizer/clauses.h" #include "optimizer/cost.h" #include "optimizer/pathnode.h" #include "optimizer/plancat.h" +#include "optimizer/var.h" #include "utils/fmgroids.h" #include "utils/lsyscache.h" +#include "utils/mvstats.h" #include "utils/selfuncs.h" +#include "utils/typcache.h" /* @@ -42,6 +46,44 @@ typedef struct RangeQueryClause static void addRangeClause(RangeQueryClause **rqlist, Node *clause, bool varonleft, bool isLTsel, Selectivity s2); +#define MV_CLAUSE_TYPE_FDEP 0x01 + +static bool clause_is_mv_compatible(PlannerInfo *root, Node *clause, Oid varRelid, + Index *relid, AttrNumber *attnum, SpecialJoinInfo *sjinfo); + +static Bitmapset *collect_mv_attnums(PlannerInfo *root, List *clauses, + Oid varRelid, Index *relid, SpecialJoinInfo *sjinfo); + +static List *clauselist_apply_dependencies(PlannerInfo *root, List *clauses, + Oid varRelid, List *stats, + SpecialJoinInfo *sjinfo); + +static bool has_stats(List *stats, int type); + +static List * find_stats(PlannerInfo *root, List *clauses, + Oid varRelid, Index *relid); + +static Bitmapset* fdeps_collect_attnums(List *stats); + +static int *make_idx_to_attnum_mapping(Bitmapset *attnums); +static int *make_attnum_to_idx_mapping(Bitmapset *attnums); + +static bool *build_adjacency_matrix(List *stats, Bitmapset *attnums, + int *idx_to_attnum, int *attnum_to_idx); + +static void multiply_adjacency_matrix(bool *matrix, int natts); + +static List* fdeps_reduce_clauses(List *clauses, + Bitmapset *attnums, bool *matrix, + int *idx_to_attnum, int *attnum_to_idx, + Index relid); + +static Bitmapset *fdeps_filter_clauses(PlannerInfo *root, + List *clauses, Bitmapset *deps_attnums, + List **reduced_clauses, List **deps_clauses, + Oid varRelid, Index *relid, SpecialJoinInfo *sjinfo); + +static Bitmapset * get_varattnos(Node * node, Index relid); /**************************************************************************** * ROUTINES TO COMPUTE SELECTIVITIES @@ -61,7 +103,7 @@ static void addRangeClause(RangeQueryClause **rqlist, Node *clause, * subclauses. However, that's only right if the subclauses have independent * probabilities, and in reality they are often NOT independent. So, * we want to be smarter where we can. - + * * Currently, the only extra smarts we have is to recognize "range queries", * such as "x > 34 AND x < 42". Clauses are recognized as possible range * query components if they are restriction opclauses whose operators have @@ -88,6 +130,88 @@ static void addRangeClause(RangeQueryClause **rqlist, Node *clause, * * Of course this is all very dependent on the behavior of * scalarltsel/scalargtsel; perhaps some day we can generalize the approach. + * + * + * Multivariate statististics + * -------------------------- + * This also uses multivariate stats to estimate combinations of + * conditions, in a way (a) maximizing the estimate accuracy by using + * as many stats as possible, and (b) minimizing the overhead, + * especially when there are no suitable multivariate stats (so if you + * are not using multivariate stats, there's no additional overhead). + * + * The following checks are performed (in this order), and the optimizer + * falls back to regular stats on the first 'false'. + * + * NOTE: This explains how this works with all the patches applied, not + * just the functional dependencies. + * + * (0) check if there are multivariate stats on the relation + * + * If no, just skip all the following steps (directly to the + * original code). + * + * (1) check how many attributes are there in conditions compatible + * with functional dependencies + * + * Only simple equality clauses are considered compatible with + * functional dependencies (and that's unlikely to change, because + * that's the only case when functional dependencies are useful). + * + * If there are no conditions that might be handled by multivariate + * stats, or if the conditions reference just a single column, it + * makes no sense to use functional dependencies, so skip to (4). + * + * (2) reduce the clauses using functional dependencies + * + * This simply attempts to 'reduce' the clauses by applying functional + * dependencies. For example if there are two clauses: + * + * WHERE (a = 1) AND (b = 2) + * + * and we know that 'a' determines the value of 'b', we may remove + * the second condition (b = 2) when computing the selectivity. + * This is of course tricky - see mvstats/dependencies.c for details. + * + * After the reduction, step (1) is to be repeated. + * + * (3) check how many attributes are there in conditions compatible + * with MCV lists and histograms + * + * What conditions are compatible with multivariate stats is decided + * by clause_is_mv_compatible(). At this moment, only conditions + * of the form "column operator constant" (for simple comparison + * operators), IS [NOT] NULL and some AND/OR clauses are considered + * compatible with multivariate statistics. + * + * Again, see clause_is_mv_compatible() for details. + * + * (4) check how many attributes are there in conditions compatible + * with MCV lists and histograms + * + * If there are no conditions that might be handled by MCV lists + * or histograms, or if the conditions reference just a single + * column, it makes no sense to continue, so just skip to (7). + * + * (5) choose the stats matching the most columns + * + * If there are multiple instances of multivariate statistics (e.g. + * built on different sets of columns), we choose the stats covering + * the most columns from step (1). It may happen that all available + * stats match just a single column - for example with conditions + * + * WHERE a = 1 AND b = 2 + * + * and statistics built on (a,c) and (b,c). In such case just fall + * back to the regular stats because it makes no sense to use the + * multivariate statistics. + * + * For more details about how exactly we choose the stats, see + * choose_mv_statistics(). + * + * (6) use the multivariate stats to estimate matching clauses + * + * (7) estimate the remaining clauses using the regular statistics */ Selectivity clauselist_selectivity(PlannerInfo *root, @@ -100,6 +224,16 @@ clauselist_selectivity(PlannerInfo *root, RangeQueryClause *rqlist = NULL; ListCell *l; + /* processing mv stats */ + Oid relid = InvalidOid; + + /* attributes in mv-compatible clauses */ + Bitmapset *mvattnums = NULL; + List *stats = NIL; + + /* use clauses (not conditions), because those are always non-empty */ + stats = find_stats(root, clauses, varRelid, &relid); + /* * If there's exactly one clause, then no use in trying to match up pairs, * so just go directly to clause_selectivity(). @@ -109,6 +243,31 @@ clauselist_selectivity(PlannerInfo *root, varRelid, jointype, sjinfo); /* + * Check that there are some stats with functional dependencies + * built (by walking the stats list). We're going to find that + * anyway when trying to apply the functional dependencies, but + * this is probably a tad faster. + */ + if (has_stats(stats, MV_CLAUSE_TYPE_FDEP)) + { + /* collect attributes referenced by mv-compatible clauses */ + mvattnums = collect_mv_attnums(root, clauses, varRelid, &relid, sjinfo); + + /* + * If there are mv-compatible clauses, referencing at least two + * different columns (otherwise it makes no sense to use mv stats), + * try to reduce the clauses using functional dependencies, and + * recollect the attributes from the reduced list. + * + * We don't need to select a single statistics for this - we can + * apply all the functional dependencies we have. + */ + if (bms_num_members(mvattnums) >= 2) + clauses = clauselist_apply_dependencies(root, clauses, varRelid, + stats, sjinfo); + } + + /* * Initial scan over clauses. Anything that doesn't look like a potential * rangequery clause gets multiplied into s1 and forgotten. Anything that * does gets inserted into an rqlist entry. @@ -782,3 +941,753 @@ clause_selectivity(PlannerInfo *root, return s1; } + +/* + * Collect attributes from mv-compatible clauses. + */ +static Bitmapset * +collect_mv_attnums(PlannerInfo *root, List *clauses, Oid varRelid, + Index *relid, SpecialJoinInfo *sjinfo) +{ + Bitmapset *attnums = NULL; + ListCell *l; + + /* + * Walk through the clauses and identify the ones we can estimate + * using multivariate stats, and remember the relid/columns. We'll + * then cross-check if we have suitable stats, and only if needed + * we'll split the clauses into multivariate and regular lists. + * + * For now we're only interested in RestrictInfo nodes with nested + * OpExpr, using either a range or equality. + */ + foreach (l, clauses) + { + AttrNumber attnum; + Node *clause = (Node *) lfirst(l); + + /* ignore the result for now - we only need the info */ + if (clause_is_mv_compatible(root, clause, varRelid, relid, &attnum, sjinfo)) + attnums = bms_add_member(attnums, attnum); + } + + /* + * If there are not at least two attributes referenced by the clause(s), + * we can throw everything out (as we'll revert to simple stats). + */ + if (bms_num_members(attnums) <= 1) + { + if (attnums != NULL) + pfree(attnums); + attnums = NULL; + *relid = InvalidOid; + } + + return attnums; +} + +/* + * Determines whether the clause is compatible with multivariate stats, + * and if it is, returns some additional information - varno (index + * into simple_rte_array) and a bitmap of attributes. This is then + * used to fetch related multivariate statistics. + * + * At this moment we only support basic conditions of the form + * + * variable OP constant + * + * where OP is one of [=,<,<=,>=,>] (which is however determined by + * looking at the associated function for estimating selectivity, just + * like with the single-dimensional case). + * + * TODO Support 'OR clauses' - shouldn't be all that difficult to + * evaluate them using multivariate stats. + */ +static bool +clause_is_mv_compatible(PlannerInfo *root, Node *clause, Oid varRelid, + Index *relid, AttrNumber *attnum, SpecialJoinInfo *sjinfo) +{ + + if (IsA(clause, RestrictInfo)) + { + RestrictInfo *rinfo = (RestrictInfo *) clause; + + /* Pseudoconstants are not really interesting here. */ + if (rinfo->pseudoconstant) + return false; + + /* no support for OR clauses at this point */ + if (rinfo->orclause) + return false; + + /* get the actual clause from the RestrictInfo (it's not an OR clause) */ + clause = (Node*)rinfo->clause; + + /* only simple opclauses are compatible with multivariate stats */ + if (! is_opclause(clause)) + return false; + + /* we don't support join conditions at this moment */ + if (treat_as_join_clause(clause, rinfo, varRelid, sjinfo)) + return false; + + /* is it 'variable op constant' ? */ + if (list_length(((OpExpr *) clause)->args) == 2) + { + OpExpr *expr = (OpExpr *) clause; + bool varonleft = true; + bool ok; + + ok = (bms_membership(rinfo->clause_relids) == BMS_SINGLETON) && + (is_pseudo_constant_clause_relids(lsecond(expr->args), + rinfo->right_relids) || + (varonleft = false, + is_pseudo_constant_clause_relids(linitial(expr->args), + rinfo->left_relids))); + + if (ok) + { + Var * var = (varonleft) ? linitial(expr->args) : lsecond(expr->args); + + /* + * Simple variables only - otherwise the planner_rt_fetch seems to fail + * (return NULL). + * + * TODO Maybe use examine_variable() would fix that? + */ + if (! (IsA(var, Var) && (varRelid == 0 || varRelid == var->varno))) + return false; + + /* + * Only consider this variable if (varRelid == 0) or when the varno + * matches varRelid (see explanation at clause_selectivity). + * + * FIXME I suspect this may not be really necessary. The (varRelid == 0) + * part seems to be enforced by treat_as_join_clause(). + */ + if (! ((varRelid == 0) || (varRelid == var->varno))) + return false; + + /* Also skip special varno values, and system attributes ... */ + if ((IS_SPECIAL_VARNO(var->varno)) || (! AttrNumberIsForUserDefinedAttr(var->varattno))) + return false; + + *relid = var->varno; + + /* + * If it's not a "<" or ">" or "=" operator, just ignore the + * clause. Otherwise note the relid and attnum for the variable. + * This uses the function for estimating selectivity, ont the + * operator directly (a bit awkward, but well ...). + */ + switch (get_oprrest(expr->opno)) + { + case F_EQSEL: + *attnum = var->varattno; + return true; + } + } + } + } + + return false; + +} + +/* + * Performs reduction of clauses using functional dependencies, i.e. + * removes clauses that are considered redundant. It simply walks + * through dependencies, and checks whether the dependency 'matches' + * the clauses, i.e. if there's a clause matching the condition. If yes, + * all clauses matching the implied part of the dependency are removed + * from the list. + * + * This simply looks at attnums references by the clauses, not at the + * type of the operator (equality, inequality, ...). This may not be the + * right way to do - it certainly works best for equalities, which is + * naturally consistent with functional dependencies (implications). + * It's not clear that other operators are handled sensibly - for + * example for inequalities, like + * + * WHERE (A >= 10) AND (B <= 20) + * + * and a trivial case where [A == B], resulting in symmetric pair of + * rules [A => B], [B => A], it's rather clear we can't remove either of + * those clauses. + * + * That only highlights that functional dependencies are most suitable + * for label-like data, where using non-equality operators is very rare. + * Using the common city/zipcode example, clauses like + * + * (zipcode <= 12345) + * + * or + * + * (cityname >= 'Washington') + * + * are rare. So restricting the reduction to equality should not harm + * the usefulness / applicability. + * + * The other assumption is that this assumes 'compatible' clauses. For + * example by using mismatching zip code and city name, this is unable + * to identify the discrepancy and eliminates one of the clauses. The + * usual approach (multiplying both selectivities) thus produces a more + * accurate estimate, although mostly by luck - the multiplication + * comes from assumption of statistical independence of the two + * conditions (which is not not valid in this case), but moves the + * estimate in the right direction (towards 0%). + * + * This might be somewhat improved by cross-checking the selectivities + * against MCV and/or histogram. + * + * The implementation needs to be careful about cyclic rules, i.e. rules + * like [A => B] and [B => A] at the same time. This must not reduce + * clauses on both attributes at the same time. + * + * Technically we might consider selectivities here too, somehow. E.g. + * when (A => B) and (B => A), we might use the clauses with minimum + * selectivity. + * + * TODO Consider restricting the reduction to equality clauses. Or maybe + * use equality classes somehow? + * + * TODO Merge this docs to dependencies.c, as it's saying mostly the + * same things as the comments there. + * + * TODO Currently this is applied only to the top-level clauses, but + * maybe we could apply it to lists at subtrees too, e.g. to the + * two AND-clauses in + * + * (x=1 AND y=2) OR (z=3 AND q=10) + * + */ +static List * +clauselist_apply_dependencies(PlannerInfo *root, List *clauses, + Oid varRelid, List *stats, + SpecialJoinInfo *sjinfo) +{ + List *reduced_clauses = NIL; + Index relid; + + /* + * matrix of (natts x natts), 1 means x=>y + * + * This serves two purposes - first, it merges dependencies from all + * the statistics, second it makes generating all the transitive + * dependencies easier. + * + * We need to build this only for attributes from the dependencies, + * not for all attributes in the table. + * + * We can't do that only for attributes from the clauses, because we + * want to build transitive dependencies (including those going + * through attributes not listed in the stats). + * + * This only works for A=>B dependencies, not sure how to do that + * for complex dependencies. + */ + bool *deps_matrix; + int deps_natts; /* size of the matric */ + + /* mapping attnum <=> matrix index */ + int *deps_idx_to_attnum; + int *deps_attnum_to_idx; + + /* attnums in dependencies and clauses (and intersection) */ + List *deps_clauses = NIL; + Bitmapset *deps_attnums = NULL; + Bitmapset *clause_attnums = NULL; + Bitmapset *intersect_attnums = NULL; + + /* + * Is there at least one statistics with functional dependencies? + * If not, return the original clauses right away. + * + * XXX Isn't this pointless, thanks to exactly the same check in + * clauselist_selectivity()? Can we trigger the condition here? + */ + if (! has_stats(stats, MV_CLAUSE_TYPE_FDEP)) + return clauses; + + /* + * Build the dependency matrix, i.e. attribute adjacency matrix, + * where 1 means (a=>b). Once we have the adjacency matrix, we'll + * multiply it by itself, to get transitive dependencies. + * + * Note: This is pretty much transitive closure from graph theory. + * + * First, let's see what attributes are covered by functional + * dependencies (sides of the adjacency matrix), and also a maximum + * attribute (size of mapping to simple integer indexes); + */ + deps_attnums = fdeps_collect_attnums(stats); + + /* + * Walk through the clauses - clauses that are (one of) + * + * (a) not mv-compatible + * (b) are using more than a single attnum + * (c) using attnum not covered by functional depencencies + * + * may be copied directly to the result. The interesting clauses are + * kept in 'deps_clauses' and will be processed later. + */ + clause_attnums = fdeps_filter_clauses(root, clauses, deps_attnums, + &reduced_clauses, &deps_clauses, + varRelid, &relid, sjinfo); + + /* + * we need at least two clauses referencing two different attributes + * referencing to do the reduction + */ + if ((list_length(deps_clauses) < 2) || (bms_num_members(clause_attnums) < 2)) + { + bms_free(clause_attnums); + list_free(reduced_clauses); + list_free(deps_clauses); + + return clauses; + } + + + /* + * We need at least two matching attributes in the clauses and + * dependencies, otherwise we can't really reduce anything. + */ + intersect_attnums = bms_intersect(clause_attnums, deps_attnums); + if (bms_num_members(intersect_attnums) < 2) + { + bms_free(clause_attnums); + bms_free(deps_attnums); + bms_free(intersect_attnums); + + list_free(deps_clauses); + list_free(reduced_clauses); + + return clauses; + } + + /* + * Build mapping between matrix indexes and attnums, and then the + * adjacency matrix itself. + */ + deps_idx_to_attnum = make_idx_to_attnum_mapping(deps_attnums); + deps_attnum_to_idx = make_attnum_to_idx_mapping(deps_attnums); + + /* build the adjacency matrix */ + deps_matrix = build_adjacency_matrix(stats, deps_attnums, + deps_idx_to_attnum, + deps_attnum_to_idx); + + deps_natts = bms_num_members(deps_attnums); + + /* + * Multiply the matrix N-times (N = size of the matrix), so that we + * get all the transitive dependencies. That makes the next step + * much easier and faster. + * + * This is essentially an adjacency matrix from graph theory, and + * by multiplying it we get transitive edges. We don't really care + * about the exact number (number of paths between vertices) though, + * so we can do the multiplication in-place (we don't care whether + * we found the dependency in this round or in the previous one). + * + * Track how many new dependencies were added, and stop when 0, but + * we can't multiply more than N-times (longest path in the graph). + */ + multiply_adjacency_matrix(deps_matrix, deps_natts); + + /* + * Walk through the clauses, and see which other clauses we may + * reduce. The matrix contains all transitive dependencies, which + * makes this very fast. + * + * We have to be careful not to reduce the clause using itself, or + * reducing all clauses forming a cycle (so we have to skip already + * eliminated clauses). + * + * I'm not sure whether this guarantees finding the best solution, + * i.e. reducing the most clauses, but it probably does (thanks to + * having all the transitive dependencies). + */ + deps_clauses = fdeps_reduce_clauses(deps_clauses, + deps_attnums, deps_matrix, + deps_idx_to_attnum, + deps_attnum_to_idx, relid); + + /* join the two lists of clauses */ + reduced_clauses = list_union(reduced_clauses, deps_clauses); + + pfree(deps_matrix); + pfree(deps_idx_to_attnum); + pfree(deps_attnum_to_idx); + + bms_free(deps_attnums); + bms_free(clause_attnums); + bms_free(intersect_attnums); + + return reduced_clauses; +} + +static bool +has_stats(List *stats, int type) +{ + ListCell *s; + + foreach (s, stats) + { + MVStatisticInfo *stat = (MVStatisticInfo *)lfirst(s); + + if ((type & MV_CLAUSE_TYPE_FDEP) && stat->deps_built) + return true; + } + + return false; +} + +/* + * Determing relid (either from varRelid or from clauses) and then + * lookup stats using the relid. + */ +static List * +find_stats(PlannerInfo *root, List *clauses, Oid varRelid, Index *relid) +{ + /* unknown relid by default */ + *relid = InvalidOid; + + /* + * First we need to find the relid (index info simple_rel_array). + * If varRelid is not 0, we already have it, otherwise we have to + * look it up from the clauses. + */ + if (varRelid != 0) + *relid = varRelid; + else + { + Relids relids = pull_varnos((Node*)clauses); + + /* + * We only expect 0 or 1 members in the bitmapset. If there are + * no vars, we'll get empty bitmapset, otherwise we'll get the + * relid as the single member. + * + * FIXME For some reason we can get 2 relids here (e.g. \d in + * psql does that). + */ + if (bms_num_members(relids) == 1) + *relid = bms_singleton_member(relids); + + bms_free(relids); + } + + /* + * if we found the relid, we can get the stats from simple_rel_array + * + * This only gets stats that are already built, because that's how + * we load it into RelOptInfo (see get_relation_info), but we don't + * detoast the whole stats yet. That'll be done later, after we + * decide which stats to use. + */ + if (*relid != InvalidOid) + return root->simple_rel_array[*relid]->mvstatlist; + + return NIL; +} + +static Bitmapset* +fdeps_collect_attnums(List *stats) +{ + ListCell *lc; + Bitmapset *attnums = NULL; + + foreach (lc, stats) + { + int j; + MVStatisticInfo *info = (MVStatisticInfo *)lfirst(lc); + + int2vector *stakeys = info->stakeys; + + /* skip stats without functional dependencies built */ + if (! info->deps_built) + continue; + + for (j = 0; j < stakeys->dim1; j++) + attnums = bms_add_member(attnums, stakeys->values[j]); + } + + return attnums; +} + + +static int* +make_idx_to_attnum_mapping(Bitmapset *attnums) +{ + int attidx = 0; + int attnum = -1; + + int *mapping = (int*)palloc0(bms_num_members(attnums) * sizeof(int)); + + while ((attnum = bms_next_member(attnums, attnum)) >= 0) + mapping[attidx++] = attnum; + + Assert(attidx == bms_num_members(attnums)); + + return mapping; +} + +static int* +make_attnum_to_idx_mapping(Bitmapset *attnums) +{ + int attidx = 0; + int attnum = -1; + int maxattnum = -1; + int *mapping; + + while ((attnum = bms_next_member(attnums, attnum)) >= 0) + maxattnum = attnum; + + mapping = (int*)palloc0((maxattnum+1) * sizeof(int)); + + attnum = -1; + while ((attnum = bms_next_member(attnums, attnum)) >= 0) + mapping[attnum] = attidx++; + + Assert(attidx == bms_num_members(attnums)); + + return mapping; +} + +static bool* +build_adjacency_matrix(List *stats, Bitmapset *attnums, + int *idx_to_attnum, int *attnum_to_idx) +{ + ListCell *lc; + int natts = bms_num_members(attnums); + bool *matrix = (bool*)palloc0(natts * natts * sizeof(bool)); + + foreach (lc, stats) + { + int j; + MVStatisticInfo *stat = (MVStatisticInfo *)lfirst(lc); + MVDependencies dependencies = NULL; + + /* skip stats without functional dependencies built */ + if (! stat->deps_built) + continue; + + /* fetch and deserialize dependencies */ + dependencies = load_mv_dependencies(stat->mvoid); + if (dependencies == NULL) + { + elog(WARNING, "failed to deserialize func deps %d", stat->mvoid); + continue; + } + + /* set matrix[a,b] to 'true' if 'a=>b' */ + for (j = 0; j < dependencies->ndeps; j++) + { + int aidx = attnum_to_idx[dependencies->deps[j]->a]; + int bidx = attnum_to_idx[dependencies->deps[j]->b]; + + /* a=> b */ + matrix[aidx * natts + bidx] = true; + } + } + + return matrix; +} + +static void +multiply_adjacency_matrix(bool *matrix, int natts) +{ + int i; + + for (i = 0; i < natts; i++) + { + int k, l, m; + int nchanges = 0; + + /* k => l */ + for (k = 0; k < natts; k++) + { + for (l = 0; l < natts; l++) + { + /* we already have this dependency */ + if (matrix[k * natts + l]) + continue; + + /* we don't really care about the exact value, just 0/1 */ + for (m = 0; m < natts; m++) + { + if (matrix[k * natts + m] * matrix[m * natts + l]) + { + matrix[k * natts + l] = true; + nchanges += 1; + break; + } + } + } + } + + /* no transitive dependency added here, so terminate */ + if (nchanges == 0) + break; + } +} + +static List* +fdeps_reduce_clauses(List *clauses, Bitmapset *attnums, bool *matrix, + int *idx_to_attnum, int *attnum_to_idx, Index relid) +{ + int i; + ListCell *lc; + List *reduced_clauses = NIL; + + int nmvclauses; /* size of the arrays */ + bool *reduced; + AttrNumber *mvattnums; + Node **mvclauses; + + int natts = bms_num_members(attnums); + + /* + * Preallocate space for all clauses (the list only containst + * compatible clauses at this point). This makes it somewhat easier + * to access the stats / attnums randomly. + * + * XXX This assumes each clause references exactly one Var, so the + * arrays are sized accordingly - for functional dependencies + * this is safe, because it only works with Var=Const. + */ + mvclauses = (Node**)palloc0(list_length(clauses) * sizeof(Node*)); + mvattnums = (AttrNumber*)palloc0(list_length(clauses) * sizeof(AttrNumber)); + reduced = (bool*)palloc0(list_length(clauses) * sizeof(bool)); + + /* fill the arrays */ + nmvclauses = 0; + foreach (lc, clauses) + { + Node * clause = (Node*)lfirst(lc); + Bitmapset * attnums = get_varattnos(clause, relid); + + mvclauses[nmvclauses] = clause; + mvattnums[nmvclauses] = bms_singleton_member(attnums); + nmvclauses++; + } + + Assert(nmvclauses == list_length(clauses)); + + /* now try to reduce the clauses (using the dependencies) */ + for (i = 0; i < nmvclauses; i++) + { + int j; + + /* not covered by dependencies */ + if (! bms_is_member(mvattnums[i], attnums)) + continue; + + /* this clause was already reduced, so let's skip it */ + if (reduced[i]) + continue; + + /* walk the potentially 'implied' clauses */ + for (j = 0; j < nmvclauses; j++) + { + int aidx, bidx; + + /* not covered by dependencies */ + if (! bms_is_member(mvattnums[j], attnums)) + continue; + + aidx = attnum_to_idx[mvattnums[i]]; + bidx = attnum_to_idx[mvattnums[j]]; + + /* can't reduce the clause by itself, or if already reduced */ + if ((i == j) || reduced[j]) + continue; + + /* mark the clause as reduced (if aidx => bidx) */ + reduced[j] = matrix[aidx * natts + bidx]; + } + } + + /* now walk through the clauses, and keep only those not reduced */ + for (i = 0; i < nmvclauses; i++) + if (! reduced[i]) + reduced_clauses = lappend(reduced_clauses, mvclauses[i]); + + pfree(reduced); + pfree(mvclauses); + pfree(mvattnums); + + return reduced_clauses; +} + + +static Bitmapset * +fdeps_filter_clauses(PlannerInfo *root, + List *clauses, Bitmapset *deps_attnums, + List **reduced_clauses, List **deps_clauses, + Oid varRelid, Index *relid, SpecialJoinInfo *sjinfo) +{ + ListCell *lc; + Bitmapset *clause_attnums = NULL; + + foreach (lc, clauses) + { + AttrNumber attnum; + Node *clause = (Node *) lfirst(lc); + + if (! clause_is_mv_compatible(root, clause, varRelid, relid, + &attnum, sjinfo)) + + /* clause incompatible with functional dependencies */ + *reduced_clauses = lappend(*reduced_clauses, clause); + + else if (! bms_is_member(attnum, deps_attnums)) + + /* clause not covered by the dependencies */ + *reduced_clauses = lappend(*reduced_clauses, clause); + + else + { + *deps_clauses = lappend(*deps_clauses, clause); + clause_attnums = bms_add_member(clause_attnums, attnum); + } + } + + return clause_attnums; +} + +/* + * Pull varattnos from the clauses, similarly to pull_varattnos() but: + * + * (a) only get attributes for a particular relation (relid) + * (b) ignore system attributes (we can't build stats on them anyway) + * + * This makes it possible to directly compare the result with attnum + * values from pg_attribute etc. + */ +static Bitmapset * +get_varattnos(Node * node, Index relid) +{ + int k; + Bitmapset *varattnos = NULL; + Bitmapset *result = NULL; + + /* get the varattnos */ + pull_varattnos(node, relid, &varattnos); + + k = -1; + while ((k = bms_next_member(varattnos, k)) >= 0) + { + if (k + FirstLowInvalidHeapAttributeNumber > 0) + result + = bms_add_member(result, + k + FirstLowInvalidHeapAttributeNumber); + } + + bms_free(varattnos); + + return result; +} diff --git a/src/backend/utils/mvstats/common.c b/src/backend/utils/mvstats/common.c index a755c49..bd200bc 100644 --- a/src/backend/utils/mvstats/common.c +++ b/src/backend/utils/mvstats/common.c @@ -84,7 +84,8 @@ build_mv_stats(Relation onerel, int numrows, HeapTuple *rows, /* * Analyze functional dependencies of columns. */ - deps = build_mv_dependencies(numrows, rows, attrs, stats); + if (stat->deps_enabled) + deps = build_mv_dependencies(numrows, rows, attrs, stats); /* store the histogram / MCV list in the catalog */ update_mv_stats(stat->mvoid, deps, attrs); @@ -163,6 +164,7 @@ list_mv_stats(Oid relid) info->mvoid = HeapTupleGetOid(htup); info->stakeys = buildint2vector(stats->stakeys.values, stats->stakeys.dim1); + info->deps_enabled = stats->deps_enabled; info->deps_built = stats->deps_built; result = lappend(result, info); @@ -274,6 +276,7 @@ compare_scalars_partition(const void *a, const void *b, void *arg) return ApplySortComparator(da, false, db, false, ssup); } + /* initialize multi-dimensional sort */ MultiSortSupport multi_sort_init(int ndims) diff --git a/src/backend/utils/mvstats/dependencies.c b/src/backend/utils/mvstats/dependencies.c index 84b6561..0a08d12 100644 --- a/src/backend/utils/mvstats/dependencies.c +++ b/src/backend/utils/mvstats/dependencies.c @@ -636,3 +636,27 @@ pg_mv_stats_dependencies_show(PG_FUNCTION_ARGS) PG_RETURN_TEXT_P(cstring_to_text(result)); } + +MVDependencies +load_mv_dependencies(Oid mvoid) +{ + bool isnull = false; + Datum deps; + + /* Prepare to scan pg_mv_statistic for entries having indrelid = this rel. */ + HeapTuple htup = SearchSysCache1(MVSTATOID, ObjectIdGetDatum(mvoid)); + +#ifdef USE_ASSERT_CHECKING + Form_pg_mv_statistic mvstat = (Form_pg_mv_statistic) GETSTRUCT(htup); + Assert(mvstat->deps_enabled && mvstat->deps_built); +#endif + + deps = SysCacheGetAttr(MVSTATOID, htup, + Anum_pg_mv_statistic_stadeps, &isnull); + + Assert(!isnull); + + ReleaseSysCache(htup); + + return deserialize_mv_dependencies(DatumGetByteaP(deps)); +} diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 912b4f3..5f89604 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -2103,7 +2103,6 @@ describeOneTableDetails(const char *schemaname, "SELECT oid, stakeys,\n" " deps_enabled,\n" " deps_built,\n" - " mcv_max_items, hist_max_buckets,\n" " (SELECT string_agg(attname::text,', ')\n" " FROM ((SELECT unnest(stakeys) AS attnum) s\n" " JOIN pg_attribute a ON (starelid = a.attrelid and a.attnum = s.attnum))) AS attnums\n" diff --git a/src/include/utils/mvstats.h b/src/include/utils/mvstats.h index 411cd16..02a7dda 100644 --- a/src/include/utils/mvstats.h +++ b/src/include/utils/mvstats.h @@ -16,12 +16,20 @@ #include "commands/vacuum.h" +/* + * Degree of how much MCV item / histogram bucket matches a clause. + * This is then considered when computing the selectivity. + */ +#define MVSTATS_MATCH_NONE 0 /* no match at all */ +#define MVSTATS_MATCH_PARTIAL 1 /* partial match */ +#define MVSTATS_MATCH_FULL 2 /* full match */ #define MVSTATS_MAX_DIMENSIONS 8 /* max number of attributes */ -/* An associative rule, tracking [a => b] dependency. - * - * TODO Make this work with multiple columns on both sides. + +/* + * Functional dependencies, tracking column-level relationships (values + * in one column determine values in another one). */ typedef struct MVDependencyData { int16 a; @@ -47,6 +55,8 @@ typedef MVDependenciesData* MVDependencies; * stats specified using flags (or something like that). */ +MVDependencies load_mv_dependencies(Oid mvoid); + bytea * serialize_mv_dependencies(MVDependencies dependencies); /* deserialization of stats (serialization is private to analyze) */ diff --git a/src/test/regress/expected/mv_dependencies.out b/src/test/regress/expected/mv_dependencies.out new file mode 100644 index 0000000..cf986e8 --- /dev/null +++ b/src/test/regress/expected/mv_dependencies.out @@ -0,0 +1,172 @@ +-- data type passed by value +CREATE TABLE functional_dependencies ( + a INT, + b INT, + c INT +); +-- unknown column +ALTER TABLE functional_dependencies ADD STATISTICS (dependencies) ON (unknown_column); +ERROR: column "unknown_column" referenced in statistics does not exist +-- single column +ALTER TABLE functional_dependencies ADD STATISTICS (dependencies) ON (a); +ERROR: multivariate stats require 2 or more columns +-- single column, duplicated +ALTER TABLE functional_dependencies ADD STATISTICS (dependencies) ON (a, a); +ERROR: duplicate column name in statistics definition +-- two columns, one duplicated +ALTER TABLE functional_dependencies ADD STATISTICS (dependencies) ON (a, a, b); +ERROR: duplicate column name in statistics definition +-- unknown option +ALTER TABLE functional_dependencies ADD STATISTICS (unknown_option) ON (a, b, c); +ERROR: unrecognized STATISTICS option "unknown_option" +-- correct command +ALTER TABLE functional_dependencies ADD STATISTICS (dependencies) ON (a, b, c); +-- random data (no functional dependencies) +INSERT INTO functional_dependencies + SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i); +ANALYZE functional_dependencies; +SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps) + FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass; + deps_enabled | deps_built | pg_mv_stats_dependencies_show +--------------+------------+------------------------------- + t | f | +(1 row) + +TRUNCATE functional_dependencies; +-- a => b, a => c, b => c +INSERT INTO functional_dependencies + SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i); +ANALYZE functional_dependencies; +SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps) + FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass; + deps_enabled | deps_built | pg_mv_stats_dependencies_show +--------------+------------+------------------------------- + t | t | 1 => 2, 1 => 3, 2 => 3 +(1 row) + +TRUNCATE functional_dependencies; +-- a => b, a => c +INSERT INTO functional_dependencies + SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i); +ANALYZE functional_dependencies; +SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps) + FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass; + deps_enabled | deps_built | pg_mv_stats_dependencies_show +--------------+------------+------------------------------- + t | t | 1 => 2, 1 => 3 +(1 row) + +TRUNCATE functional_dependencies; +-- check explain (expect bitmap index scan, not plain index scan) +INSERT INTO functional_dependencies + SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i); +CREATE INDEX fdeps_idx ON functional_dependencies (a, b); +ANALYZE functional_dependencies; +SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps) + FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass; + deps_enabled | deps_built | pg_mv_stats_dependencies_show +--------------+------------+------------------------------- + t | t | 1 => 2, 1 => 3, 2 => 3 +(1 row) + +EXPLAIN (COSTS off) + SELECT * FROM functional_dependencies WHERE a = 10 AND b = 5; + QUERY PLAN +--------------------------------------------- + Bitmap Heap Scan on functional_dependencies + Recheck Cond: ((a = 10) AND (b = 5)) + -> Bitmap Index Scan on fdeps_idx + Index Cond: ((a = 10) AND (b = 5)) +(4 rows) + +DROP TABLE functional_dependencies; +-- varlena type (text) +CREATE TABLE functional_dependencies ( + a TEXT, + b TEXT, + c TEXT +); +ALTER TABLE functional_dependencies ADD STATISTICS (dependencies) ON (a, b, c); +-- random data (no functional dependencies) +INSERT INTO functional_dependencies + SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i); +ANALYZE functional_dependencies; +SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps) + FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass; + deps_enabled | deps_built | pg_mv_stats_dependencies_show +--------------+------------+------------------------------- + t | f | +(1 row) + +TRUNCATE functional_dependencies; +-- a => b, a => c, b => c +INSERT INTO functional_dependencies + SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i); +ANALYZE functional_dependencies; +SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps) + FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass; + deps_enabled | deps_built | pg_mv_stats_dependencies_show +--------------+------------+------------------------------- + t | t | 1 => 2, 1 => 3, 2 => 3 +(1 row) + +TRUNCATE functional_dependencies; +-- a => b, a => c +INSERT INTO functional_dependencies + SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i); +ANALYZE functional_dependencies; +SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps) + FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass; + deps_enabled | deps_built | pg_mv_stats_dependencies_show +--------------+------------+------------------------------- + t | t | 1 => 2, 1 => 3 +(1 row) + +TRUNCATE functional_dependencies; +-- check explain (expect bitmap index scan, not plain index scan) +INSERT INTO functional_dependencies + SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i); +CREATE INDEX fdeps_idx ON functional_dependencies (a, b); +ANALYZE functional_dependencies; +SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps) + FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass; + deps_enabled | deps_built | pg_mv_stats_dependencies_show +--------------+------------+------------------------------- + t | t | 1 => 2, 1 => 3, 2 => 3 +(1 row) + +EXPLAIN (COSTS off) + SELECT * FROM functional_dependencies WHERE a = '10' AND b = '5'; + QUERY PLAN +------------------------------------------------------------ + Bitmap Heap Scan on functional_dependencies + Recheck Cond: ((a = '10'::text) AND (b = '5'::text)) + -> Bitmap Index Scan on fdeps_idx + Index Cond: ((a = '10'::text) AND (b = '5'::text)) +(4 rows) + +DROP TABLE functional_dependencies; +-- NULL values (mix of int and text columns) +CREATE TABLE functional_dependencies ( + a INT, + b TEXT, + c INT, + d TEXT +); +ALTER TABLE functional_dependencies ADD STATISTICS (dependencies) ON (a, b, c, d); +INSERT INTO functional_dependencies + SELECT + mod(i, 100), + (CASE WHEN mod(i, 200) = 0 THEN NULL ELSE mod(i,200) END), + mod(i, 400), + (CASE WHEN mod(i, 300) = 0 THEN NULL ELSE mod(i,600) END) + FROM generate_series(1,10000) s(i); +ANALYZE functional_dependencies; +SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps) + FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass; + deps_enabled | deps_built | pg_mv_stats_dependencies_show +--------------+------------+---------------------------------------- + t | t | 2 => 1, 3 => 1, 3 => 2, 4 => 1, 4 => 2 +(1 row) + +DROP TABLE functional_dependencies; diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 91780cd..11d9d38 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -110,3 +110,6 @@ test: event_trigger # run stats by itself because its delay may be insufficient under heavy load test: stats + +# run tests of multivariate stats +test: mv_dependencies diff --git a/src/test/regress/regression.diffs b/src/test/regress/regression.diffs new file mode 100644 index 0000000..95b9cc5 --- /dev/null +++ b/src/test/regress/regression.diffs @@ -0,0 +1,30 @@ +*** /home/user/work/tvondra_postgres/src/test/regress/expected/rolenames.out Wed May 6 21:31:06 2015 +--- /home/user/work/tvondra_postgres/src/test/regress/results/rolenames.out Mon May 25 22:24:21 2015 +*************** +*** 38,47 **** +--- 38,52 ---- + ORDER BY 2; + $$ LANGUAGE SQL; + CREATE ROLE "Public"; ++ ERROR: role "Public" already exists + CREATE ROLE "None"; ++ ERROR: role "None" already exists + CREATE ROLE "current_user"; ++ ERROR: role "current_user" already exists + CREATE ROLE "session_user"; ++ ERROR: role "session_user" already exists + CREATE ROLE "user"; ++ ERROR: role "user" already exists + CREATE ROLE current_user; -- error + ERROR: CURRENT_USER cannot be used as a role name here + LINE 1: CREATE ROLE current_user; +*************** +*** 938,940 **** +--- 943,946 ---- + DROP OWNED BY testrol0, "Public", "current_user", testrol1, testrol2, testrolx CASCADE; + DROP ROLE testrol0, testrol1, testrol2, testrolx; + DROP ROLE "Public", "None", "current_user", "session_user", "user"; ++ ERROR: current user cannot be dropped + +====================================================================== + diff --git a/src/test/regress/regression.out b/src/test/regress/regression.out new file mode 100644 index 0000000..bd81385 --- /dev/null +++ b/src/test/regress/regression.out @@ -0,0 +1,156 @@ +test tablespace ... ok +test boolean ... ok +test char ... ok +test name ... ok +test varchar ... ok +test text ... ok +test int2 ... ok +test int4 ... ok +test int8 ... ok +test oid ... ok +test float4 ... ok +test float8 ... ok +test bit ... ok +test numeric ... ok +test txid ... ok +test uuid ... ok +test enum ... ok +test money ... ok +test rangetypes ... ok +test pg_lsn ... ok +test regproc ... ok +test strings ... ok +test numerology ... ok +test point ... ok +test lseg ... ok +test line ... ok +test box ... ok +test path ... ok +test polygon ... ok +test circle ... ok +test date ... ok +test time ... ok +test timetz ... ok +test timestamp ... ok +test timestamptz ... ok +test interval ... ok +test abstime ... ok +test reltime ... ok +test tinterval ... ok +test inet ... ok +test macaddr ... ok +test tstypes ... ok +test comments ... ok +test geometry ... ok +test horology ... ok +test regex ... ok +test oidjoins ... ok +test type_sanity ... ok +test opr_sanity ... ok +test insert ... ok +test insert_conflict ... ok +test create_function_1 ... ok +test create_type ... ok +test create_table ... ok +test create_function_2 ... ok +test copy ... ok +test copyselect ... ok +test create_misc ... ok +test create_operator ... ok +test create_index ... ok +test create_view ... ok +test create_aggregate ... ok +test create_function_3 ... ok +test create_cast ... ok +test constraints ... ok +test triggers ... ok +test inherit ... ok +test create_table_like ... ok +test typed_table ... ok +test vacuum ... ok +test drop_if_exists ... ok +test updatable_views ... ok +test rolenames ... FAILED +test sanity_check ... ok +test errors ... ok +test select ... ok +test select_into ... ok +test select_distinct ... ok +test select_distinct_on ... ok +test select_implicit ... ok +test select_having ... ok +test subselect ... ok +test union ... ok +test case ... ok +test join ... ok +test aggregates ... ok +test groupingsets ... ok +test transactions ... ok +test random ... ok +test portals ... ok +test arrays ... ok +test btree_index ... ok +test hash_index ... ok +test update ... ok +test delete ... ok +test namespace ... ok +test prepared_xacts ... ok +test brin ... ok +test gin ... ok +test gist ... ok +test spgist ... ok +test privileges ... ok +test security_label ... ok +test collate ... ok +test matview ... ok +test lock ... ok +test replica_identity ... ok +test rowsecurity ... ok +test object_address ... ok +test alter_generic ... ok +test misc ... ok +test psql ... ok +test async ... ok +test rules ... ok +test select_views ... ok +test portals_p2 ... ok +test foreign_key ... ok +test cluster ... ok +test dependency ... ok +test guc ... ok +test bitmapops ... ok +test combocid ... ok +test tsearch ... ok +test tsdicts ... ok +test foreign_data ... ok +test window ... ok +test xmlmap ... ok +test functional_deps ... ok +test advisory_lock ... ok +test json ... ok +test jsonb ... ok +test indirect_toast ... ok +test equivclass ... ok +test plancache ... ok +test limit ... ok +test plpgsql ... ok +test copy2 ... ok +test temp ... ok +test domain ... ok +test rangefuncs ... ok +test prepare ... ok +test without_oid ... ok +test conversion ... ok +test truncate ... ok +test alter_table ... ok +test sequence ... ok +test polymorphism ... ok +test rowtypes ... ok +test returning ... ok +test largeobject ... ok +test with ... ok +test xml ... ok +test event_trigger ... ok +test stats ... ok +test tablesample ... ok +test mv_dependencies ... ok diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index a2e0ceb..66925b3 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -156,3 +156,4 @@ test: xml test: event_trigger test: stats test: tablesample +test: mv_dependencies diff --git a/src/test/regress/sql/mv_dependencies.sql b/src/test/regress/sql/mv_dependencies.sql new file mode 100644 index 0000000..2491aca --- /dev/null +++ b/src/test/regress/sql/mv_dependencies.sql @@ -0,0 +1,150 @@ +-- data type passed by value +CREATE TABLE functional_dependencies ( + a INT, + b INT, + c INT +); + +-- unknown column +ALTER TABLE functional_dependencies ADD STATISTICS (dependencies) ON (unknown_column); + +-- single column +ALTER TABLE functional_dependencies ADD STATISTICS (dependencies) ON (a); + +-- single column, duplicated +ALTER TABLE functional_dependencies ADD STATISTICS (dependencies) ON (a, a); + +-- two columns, one duplicated +ALTER TABLE functional_dependencies ADD STATISTICS (dependencies) ON (a, a, b); + +-- unknown option +ALTER TABLE functional_dependencies ADD STATISTICS (unknown_option) ON (a, b, c); + +-- correct command +ALTER TABLE functional_dependencies ADD STATISTICS (dependencies) ON (a, b, c); + +-- random data (no functional dependencies) +INSERT INTO functional_dependencies + SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i); + +ANALYZE functional_dependencies; + +SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps) + FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass; + +TRUNCATE functional_dependencies; + +-- a => b, a => c, b => c +INSERT INTO functional_dependencies + SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i); + +ANALYZE functional_dependencies; + +SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps) + FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass; + +TRUNCATE functional_dependencies; + +-- a => b, a => c +INSERT INTO functional_dependencies + SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i); +ANALYZE functional_dependencies; + +SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps) + FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass; + +TRUNCATE functional_dependencies; + +-- check explain (expect bitmap index scan, not plain index scan) +INSERT INTO functional_dependencies + SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i); +CREATE INDEX fdeps_idx ON functional_dependencies (a, b); +ANALYZE functional_dependencies; + +SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps) + FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass; + +EXPLAIN (COSTS off) + SELECT * FROM functional_dependencies WHERE a = 10 AND b = 5; + +DROP TABLE functional_dependencies; + +-- varlena type (text) +CREATE TABLE functional_dependencies ( + a TEXT, + b TEXT, + c TEXT +); + +ALTER TABLE functional_dependencies ADD STATISTICS (dependencies) ON (a, b, c); + +-- random data (no functional dependencies) +INSERT INTO functional_dependencies + SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i); + +ANALYZE functional_dependencies; + +SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps) + FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass; + +TRUNCATE functional_dependencies; + +-- a => b, a => c, b => c +INSERT INTO functional_dependencies + SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i); + +ANALYZE functional_dependencies; + +SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps) + FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass; + +TRUNCATE functional_dependencies; + +-- a => b, a => c +INSERT INTO functional_dependencies + SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i); +ANALYZE functional_dependencies; + +SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps) + FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass; + +TRUNCATE functional_dependencies; + +-- check explain (expect bitmap index scan, not plain index scan) +INSERT INTO functional_dependencies + SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i); +CREATE INDEX fdeps_idx ON functional_dependencies (a, b); +ANALYZE functional_dependencies; + +SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps) + FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass; + +EXPLAIN (COSTS off) + SELECT * FROM functional_dependencies WHERE a = '10' AND b = '5'; + +DROP TABLE functional_dependencies; + +-- NULL values (mix of int and text columns) +CREATE TABLE functional_dependencies ( + a INT, + b TEXT, + c INT, + d TEXT +); + +ALTER TABLE functional_dependencies ADD STATISTICS (dependencies) ON (a, b, c, d); + +INSERT INTO functional_dependencies + SELECT + mod(i, 100), + (CASE WHEN mod(i, 200) = 0 THEN NULL ELSE mod(i,200) END), + mod(i, 400), + (CASE WHEN mod(i, 300) = 0 THEN NULL ELSE mod(i,600) END) + FROM generate_series(1,10000) s(i); + +ANALYZE functional_dependencies; + +SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps) + FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass; + +DROP TABLE functional_dependencies; -- 1.9.3 --------------080401000304040708010509 Content-Type: text/x-patch; name="0004-multivariate-MCV-lists-v7.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0004-multivariate-MCV-lists-v7.patch" From 3114c82ae310d840f613583b169ac1cc79520f81 Mon Sep 17 00:00:00 2001 From: Tomas Vondra Date: Mon, 6 Apr 2015 16:52:15 +0200 Subject: [PATCH 4/6] multivariate MCV lists - extends the pg_mv_statistic catalog (add 'mcv' fields) - building the MCV lists during ANALYZE - simple estimation while planning the queries Includes regression tests, mostly equal to regression tests for functional dependencies. --- src/backend/catalog/system_views.sql | 4 +- src/backend/commands/tablecmds.c | 89 ++- src/backend/nodes/outfuncs.c | 2 + src/backend/optimizer/path/clausesel.c | 1079 ++++++++++++++++++++++++++-- src/backend/optimizer/util/plancat.c | 4 +- src/backend/utils/mvstats/Makefile | 2 +- src/backend/utils/mvstats/common.c | 104 ++- src/backend/utils/mvstats/common.h | 11 +- src/backend/utils/mvstats/mcv.c | 1237 ++++++++++++++++++++++++++++++++ src/bin/psql/describe.c | 25 +- src/include/catalog/pg_mv_statistic.h | 18 +- src/include/catalog/pg_proc.h | 4 + src/include/nodes/relation.h | 2 + src/include/utils/mvstats.h | 69 +- src/test/regress/expected/mv_mcv.out | 207 ++++++ src/test/regress/expected/rules.out | 4 +- src/test/regress/parallel_schedule | 2 +- src/test/regress/regression.diffs | 30 - src/test/regress/regression.out | 156 ---- src/test/regress/serial_schedule | 1 + src/test/regress/sql/mv_mcv.sql | 178 +++++ 21 files changed, 2940 insertions(+), 288 deletions(-) create mode 100644 src/backend/utils/mvstats/mcv.c create mode 100644 src/test/regress/expected/mv_mcv.out delete mode 100644 src/test/regress/regression.diffs delete mode 100644 src/test/regress/regression.out create mode 100644 src/test/regress/sql/mv_mcv.sql diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql index 0dedaba..3144a29 100644 --- a/src/backend/catalog/system_views.sql +++ b/src/backend/catalog/system_views.sql @@ -156,7 +156,9 @@ CREATE VIEW pg_mv_stats AS C.relname AS tablename, S.stakeys AS attnums, length(S.stadeps) as depsbytes, - pg_mv_stats_dependencies_info(S.stadeps) as depsinfo + pg_mv_stats_dependencies_info(S.stadeps) as depsinfo, + length(S.stamcv) AS mcvbytes, + pg_mv_stats_mcvlist_info(S.stamcv) AS mcvinfo FROM (pg_mv_statistic S JOIN pg_class C ON (C.oid = S.starelid)) LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace); diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 107e9fc..0d72aec 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -11918,7 +11918,13 @@ static void ATExecAddStatistics(AlteredTableInfo *tab, Relation rel, Relation mvstatrel; /* by default build nothing */ - bool build_dependencies = false; + bool build_dependencies = false, + build_mcv = false; + + int32 max_mcv_items = -1; + + /* options required because of other options */ + bool require_mcv = false; Assert(IsA(def, StatisticsDef)); @@ -11973,6 +11979,29 @@ static void ATExecAddStatistics(AlteredTableInfo *tab, Relation rel, if (strcmp(opt->defname, "dependencies") == 0) build_dependencies = defGetBoolean(opt); + else if (strcmp(opt->defname, "mcv") == 0) + build_mcv = defGetBoolean(opt); + else if (strcmp(opt->defname, "max_mcv_items") == 0) + { + max_mcv_items = defGetInt32(opt); + + /* this option requires 'mcv' to be enabled */ + require_mcv = true; + + /* sanity check */ + if (max_mcv_items < MVSTAT_MCVLIST_MIN_ITEMS) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("max number of MCV items must be at least %d", + MVSTAT_MCVLIST_MIN_ITEMS))); + + else if (max_mcv_items > MVSTAT_MCVLIST_MAX_ITEMS) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("max number of MCV items is %d", + MVSTAT_MCVLIST_MAX_ITEMS))); + + } else ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), @@ -11981,10 +12010,16 @@ static void ATExecAddStatistics(AlteredTableInfo *tab, Relation rel, } /* check that at least some statistics were requested */ - if (! build_dependencies) + if (! (build_dependencies || build_mcv)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("no statistics type (dependencies, mcv) was requested"))); + + /* now do some checking of the options */ + if (require_mcv && (! build_mcv)) ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), - errmsg("no statistics type (dependencies) was requested"))); + errmsg("option 'mcv' is required by other options(s)"))); /* sort the attnums and build int2vector */ qsort(attnums, numcols, sizeof(int16), compare_int16); @@ -12000,9 +12035,13 @@ static void ATExecAddStatistics(AlteredTableInfo *tab, Relation rel, values[Anum_pg_mv_statistic_starelid-1] = ObjectIdGetDatum(RelationGetRelid(rel)); values[Anum_pg_mv_statistic_stakeys -1] = PointerGetDatum(stakeys); + values[Anum_pg_mv_statistic_deps_enabled -1] = BoolGetDatum(build_dependencies); + values[Anum_pg_mv_statistic_mcv_enabled -1] = BoolGetDatum(build_mcv); + values[Anum_pg_mv_statistic_mcv_max_items -1] = Int32GetDatum(max_mcv_items); - nulls[Anum_pg_mv_statistic_stadeps -1] = true; + nulls[Anum_pg_mv_statistic_stadeps -1] = true; + nulls[Anum_pg_mv_statistic_stamcv -1] = true; /* insert the tuple into pg_mv_statistic */ mvstatrel = heap_open(MvStatisticRelationId, RowExclusiveLock); @@ -12049,7 +12088,13 @@ static void ATExecDropStatistics(AlteredTableInfo *tab, Relation rel, /* checking whether the statistics matches / should be dropped */ bool build_dependencies = false; + bool build_mcv = false; + + bool max_mcv_items = 0; + bool check_dependencies = false; + bool check_mcv = false; + bool check_mcv_items = false; if (def != NULL) { @@ -12091,6 +12136,18 @@ static void ATExecDropStatistics(AlteredTableInfo *tab, Relation rel, check_dependencies = true; build_dependencies = defGetBoolean(opt); } + else if (strcmp(opt->defname, "mcv") == 0) + { + check_mcv = true; + build_mcv = defGetBoolean(opt); + } + else if (strcmp(opt->defname, "max_mcv_items") == 0) + { + check_mcv = true; + check_mcv_items = true; + build_mcv = true; + max_mcv_items = defGetInt32(opt); + } else ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), @@ -12130,6 +12187,30 @@ static void ATExecDropStatistics(AlteredTableInfo *tab, Relation rel, (DatumGetBool(adatum) == build_dependencies); } + if (delete && check_mcv) + { + bool isnull; + Datum adatum = heap_getattr(tuple, + Anum_pg_mv_statistic_mcv_enabled, + RelationGetDescr(statrel), + &isnull); + + delete = (! isnull) && + (DatumGetBool(adatum) == build_mcv); + } + + if (delete && check_mcv_items) + { + bool isnull; + Datum adatum = heap_getattr(tuple, + Anum_pg_mv_statistic_mcv_max_items, + RelationGetDescr(statrel), + &isnull); + + delete = (! isnull) && + (DatumGetInt32(adatum) == max_mcv_items); + } + /* check that the columns match the statistics definition */ if (delete && (numcols > 0)) { diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 93a6f04..1867ab7 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -1907,9 +1907,11 @@ _outMVStatisticInfo(StringInfo str, const MVStatisticInfo *node) /* enabled statistics */ WRITE_BOOL_FIELD(deps_enabled); + WRITE_BOOL_FIELD(mcv_enabled); /* built/available statistics */ WRITE_BOOL_FIELD(deps_built); + WRITE_BOOL_FIELD(mcv_built); } static void diff --git a/src/backend/optimizer/path/clausesel.c b/src/backend/optimizer/path/clausesel.c index 6365425..95872de 100644 --- a/src/backend/optimizer/path/clausesel.c +++ b/src/backend/optimizer/path/clausesel.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/sysattr.h" +#include "catalog/pg_collation.h" #include "catalog/pg_operator.h" #include "nodes/makefuncs.h" #include "optimizer/clauses.h" @@ -47,17 +48,38 @@ static void addRangeClause(RangeQueryClause **rqlist, Node *clause, bool varonleft, bool isLTsel, Selectivity s2); #define MV_CLAUSE_TYPE_FDEP 0x01 +#define MV_CLAUSE_TYPE_MCV 0x02 static bool clause_is_mv_compatible(PlannerInfo *root, Node *clause, Oid varRelid, - Index *relid, AttrNumber *attnum, SpecialJoinInfo *sjinfo); + Index *relid, Bitmapset **attnums, SpecialJoinInfo *sjinfo, + int type); static Bitmapset *collect_mv_attnums(PlannerInfo *root, List *clauses, - Oid varRelid, Index *relid, SpecialJoinInfo *sjinfo); + Oid varRelid, Index *relid, SpecialJoinInfo *sjinfo, + int type); static List *clauselist_apply_dependencies(PlannerInfo *root, List *clauses, Oid varRelid, List *stats, SpecialJoinInfo *sjinfo); +static MVStatisticInfo *choose_mv_statistics(List *mvstats, Bitmapset *attnums); + +static List *clauselist_mv_split(PlannerInfo *root, SpecialJoinInfo *sjinfo, + List *clauses, Oid varRelid, + List **mvclauses, MVStatisticInfo *mvstats, int types); + +static Selectivity clauselist_mv_selectivity(PlannerInfo *root, + List *clauses, MVStatisticInfo *mvstats); +static Selectivity clauselist_mv_selectivity_mcvlist(PlannerInfo *root, + List *clauses, MVStatisticInfo *mvstats, + bool *fullmatch, Selectivity *lowsel); + +static int update_match_bitmap_mcvlist(PlannerInfo *root, List *clauses, + int2vector *stakeys, MCVList mcvlist, + int nmatches, char * matches, + Selectivity *lowsel, bool *fullmatch, + bool is_or); + static bool has_stats(List *stats, int type); static List * find_stats(PlannerInfo *root, List *clauses, @@ -85,6 +107,13 @@ static Bitmapset *fdeps_filter_clauses(PlannerInfo *root, static Bitmapset * get_varattnos(Node * node, Index relid); +/* used for merging bitmaps - AND (min), OR (max) */ +#define MAX(x, y) (((x) > (y)) ? (x) : (y)) +#define MIN(x, y) (((x) < (y)) ? (x) : (y)) + +#define UPDATE_RESULT(m,r,isor) \ + (m) = (isor) ? (MAX(m,r)) : (MIN(m,r)) + /**************************************************************************** * ROUTINES TO COMPUTE SELECTIVITIES ****************************************************************************/ @@ -250,8 +279,12 @@ clauselist_selectivity(PlannerInfo *root, */ if (has_stats(stats, MV_CLAUSE_TYPE_FDEP)) { - /* collect attributes referenced by mv-compatible clauses */ - mvattnums = collect_mv_attnums(root, clauses, varRelid, &relid, sjinfo); + /* + * Collect attributes referenced by mv-compatible clauses (looking + * for clauses compatible with functional dependencies for now). + */ + mvattnums = collect_mv_attnums(root, clauses, varRelid, &relid, sjinfo, + MV_CLAUSE_TYPE_FDEP); /* * If there are mv-compatible clauses, referencing at least two @@ -268,6 +301,48 @@ clauselist_selectivity(PlannerInfo *root, } /* + * Check that there are statistics with MCV list. If not, we don't + * need to waste time with the optimization. + */ + if (has_stats(stats, MV_CLAUSE_TYPE_MCV)) + { + /* + * Recollect attributes from mv-compatible clauses (maybe we've + * removed so many clauses we have a single mv-compatible attnum). + * From now on we're only interested in MCV-compatible clauses. + */ + mvattnums = collect_mv_attnums(root, clauses, varRelid, &relid, sjinfo, + MV_CLAUSE_TYPE_MCV); + + /* + * If there still are at least two columns, we'll try to select + * a suitable multivariate stats. + */ + if (bms_num_members(mvattnums) >= 2) + { + /* see choose_mv_statistics() for details */ + MVStatisticInfo *mvstat = choose_mv_statistics(stats, mvattnums); + + if (mvstat != NULL) /* we have a matching stats */ + { + /* clauses compatible with multi-variate stats */ + List *mvclauses = NIL; + + /* split the clauselist into regular and mv-clauses */ + clauses = clauselist_mv_split(root, sjinfo, clauses, + varRelid, &mvclauses, mvstat, + MV_CLAUSE_TYPE_MCV); + + /* we've chosen the histogram to match the clauses */ + Assert(mvclauses != NIL); + + /* compute the multivariate stats */ + s1 *= clauselist_mv_selectivity(root, mvclauses, mvstat); + } + } + } + + /* * Initial scan over clauses. Anything that doesn't look like a potential * rangequery clause gets multiplied into s1 and forgotten. Anything that * does gets inserted into an rqlist entry. @@ -942,12 +1017,129 @@ clause_selectivity(PlannerInfo *root, return s1; } + +/* + * Estimate selectivity for the list of MV-compatible clauses, using + * using a MV statistics (combining a histogram and MCV list). + * + * This simply passes the estimation to the MCV list and then to the + * histogram, if available. + * + * TODO Clamp the selectivity by min of the per-clause selectivities + * (i.e. the selectivity of the most restrictive clause), because + * that's the maximum we can ever get from ANDed list of clauses. + * This may probably prevent issues with hitting too many buckets + * and low precision histograms. + * + * TODO We may support some additional conditions, most importantly + * those matching multiple columns (e.g. "a = b" or "a < b"). + * Ultimately we could track multi-table histograms for join + * cardinality estimation. + * + * TODO Further thoughts on processing equality clauses: Maybe it'd be + * better to look for stats (with MCV) covered by the equality + * clauses, because then we have a chance to find an exact match + * in the MCV list, which is pretty much the best we can do. We may + * also look at the least frequent MCV item, and use it as a upper + * boundary for the selectivity (had there been a more frequent + * item, it'd be in the MCV list). + * + * TODO There are several options for 'sanity clamping' the estimates. + * + * First, if we have selectivities for each condition, then + * + * P(A,B) <= MIN(P(A), P(B)) + * + * Because additional conditions (connected by AND) can only lower + * the probability. + * + * So we can do some basic sanity checks using the single-variate + * stats (the ones we have right now). + * + * Second, when we have multivariate stats with a MCV list, then + * + * (a) if we have a full equality condition (one equality condition + * on each column) and we found a match in the MCV list, this is + * the selectivity (and it's supposed to be exact) + * + * (b) if we have a full equality condition and we haven't found a + * match in the MCV list, then the selectivity is below the + * lowest selectivity in the MCV list + * + * (c) if we have a equality condition (not full), we can still + * search the MCV for matches and use the sum of probabilities + * as a lower boundary for the histogram (if there are no + * matches in the MCV list, then we have no boundary) + * + * Third, if there are multiple (combinations of) multivariate + * stats for a set of clauses, we may compute all of them and then + * somehow aggregate them - e.g. by choosing the minimum, median or + * average. The stats are susceptible to overestimation (because + * we take 50% of the bucket for partial matches). Some stats may + * give better estimates than others, but it's very difficult to + * say that in advance which one is the best (it depends on the + * number of buckets, number of additional columns not referenced + * in the clauses, type of condition etc.). + * + * So we may compute them all and then choose a sane aggregation + * (minimum seems like a good approach). Of course, this may result + * in longer / more expensive estimation (CPU-wise), but it may be + * worth it. + * + * It's possible to add a GUC choosing whether to do a 'simple' + * (using a single stats expected to give the best estimate) and + * 'complex' (combining the multiple estimates). + * + * multivariate_estimates = (simple|full) + * + * Also, this might be enabled at a table level, by something like + * + * ALTER TABLE ... SET STATISTICS (simple|full) + * + * Which would make it possible to use this only for the tables + * where the simple approach does not work. + * + * Also, there are ways to optimize this algorithmically. E.g. we + * may try to get an estimate from a matching MCV list first, and + * if we happen to get a "full equality match" we may stop computing + * the estimates from other stats (for this condition) because + * that's probably the best estimate we can really get. + * + * TODO When applying the clauses to the histogram/MCV list, we can do + * that from the most selective clauses first, because that'll + * eliminate the buckets/items sooner (so we'll be able to skip + * them without inspection, which is more expensive). But this + * requires really knowing the per-clause selectivities in advance, + * and that's not what we do now. + */ +static Selectivity +clauselist_mv_selectivity(PlannerInfo *root, List *clauses, MVStatisticInfo *mvstats) +{ + bool fullmatch = false; + + /* + * Lowest frequency in the MCV list (may be used as an upper bound + * for full equality conditions that did not match any MCV item). + */ + Selectivity mcv_low = 0.0; + + /* TODO Evaluate simple 1D selectivities, use the smallest one as + * an upper bound, product as lower bound, and sort the + * clauses in ascending order by selectivity (to optimize the + * MCV/histogram evaluation). + */ + + /* Evaluate the MCV selectivity */ + return clauselist_mv_selectivity_mcvlist(root, clauses, mvstats, + &fullmatch, &mcv_low); +} + /* * Collect attributes from mv-compatible clauses. */ static Bitmapset * collect_mv_attnums(PlannerInfo *root, List *clauses, Oid varRelid, - Index *relid, SpecialJoinInfo *sjinfo) + Index *relid, SpecialJoinInfo *sjinfo, int types) { Bitmapset *attnums = NULL; ListCell *l; @@ -963,12 +1155,11 @@ collect_mv_attnums(PlannerInfo *root, List *clauses, Oid varRelid, */ foreach (l, clauses) { - AttrNumber attnum; Node *clause = (Node *) lfirst(l); - /* ignore the result for now - we only need the info */ - if (clause_is_mv_compatible(root, clause, varRelid, relid, &attnum, sjinfo)) - attnums = bms_add_member(attnums, attnum); + /* ignore the result here - we only need the attnums */ + clause_is_mv_compatible(root, clause, varRelid, relid, &attnums, + sjinfo, types); } /* @@ -987,6 +1178,188 @@ collect_mv_attnums(PlannerInfo *root, List *clauses, Oid varRelid, } /* + * We're looking for statistics matching at least 2 attributes, + * referenced in the clauses compatible with multivariate statistics. + * The current selection criteria is very simple - we choose the + * statistics referencing the most attributes. + * + * If there are multiple statistics referencing the same number of + * columns (from the clauses), the one with less source columns + * (as listed in the ADD STATISTICS when creating the statistics) wins. + * Other wise the first one wins. + * + * This is a very simple criteria, and has several weaknesses: + * + * (a) does not consider the accuracy of the statistics + * + * If there are two histograms built on the same set of columns, + * but one has 100 buckets and the other one has 1000 buckets (thus + * likely providing better estimates), this is not currently + * considered. + * + * (b) does not consider the type of statistics + * + * If there are three statistics - one containing just a MCV list, + * another one with just a histogram and a third one with both, + * this is not considered. + * + * (c) does not consider the number of clauses + * + * As explained, only the number of referenced attributes counts, + * so if there are multiple clauses on a single attribute, this + * still counts as a single attribute. + * + * (d) does not consider type of condition + * + * Some clauses may work better with some statistics - for example + * equality clauses probably work better with MCV lists than with + * histograms. But IS [NOT] NULL conditions may often work better + * with histograms (thanks to NULL-buckets). + * + * So for example with five WHERE conditions + * + * WHERE (a = 1) AND (b = 1) AND (c = 1) AND (d = 1) AND (e = 1) + * + * and statistics on (a,b), (a,b,e) and (a,b,c,d), the last one will be + * selected as it references the most columns. + * + * Once we have selected the multivariate statistics, we split the list + * of clauses into two parts - conditions that are compatible with the + * selected stats, and conditions are estimated using simple statistics. + * + * From the example above, conditions + * + * (a = 1) AND (b = 1) AND (c = 1) AND (d = 1) + * + * will be estimated using the multivariate statistics (a,b,c,d) while + * the last condition (e = 1) will get estimated using the regular ones. + * + * There are various alternative selection criteria (e.g. counting + * conditions instead of just referenced attributes), but eventually + * the best option should be to combine multiple statistics. But that's + * much harder to do correctly. + * + * TODO Select multiple statistics and combine them when computing + * the estimate. + * + * TODO This will probably have to consider compatibility of clauses, + * because 'dependencies' will probably work only with equality + * clauses. + */ +static MVStatisticInfo * +choose_mv_statistics(List *stats, Bitmapset *attnums) +{ + int i; + ListCell *lc; + + MVStatisticInfo *choice = NULL; + + int current_matches = 1; /* goal #1: maximize */ + int current_dims = (MVSTATS_MAX_DIMENSIONS+1); /* goal #2: minimize */ + + /* + * Walk through the statistics (simple array with nmvstats elements) + * and for each one count the referenced attributes (encoded in + * the 'attnums' bitmap). + */ + foreach (lc, stats) + { + MVStatisticInfo *info = (MVStatisticInfo *)lfirst(lc); + + /* columns matching this statistics */ + int matches = 0; + + int2vector * attrs = info->stakeys; + int numattrs = attrs->dim1; + + /* skip dependencies-only stats */ + if (! info->mcv_built) + continue; + + /* count columns covered by the histogram */ + for (i = 0; i < numattrs; i++) + if (bms_is_member(attrs->values[i], attnums)) + matches++; + + /* + * Use this statistics when it improves the number of matches or + * when it matches the same number of attributes but is smaller. + */ + if ((matches > current_matches) || + ((matches == current_matches) && (current_dims > numattrs))) + { + choice = info; + current_matches = matches; + current_dims = numattrs; + } + } + + return choice; +} + + +/* + * This splits the clauses list into two parts - one containing clauses + * that will be evaluated using the chosen statistics, and the remaining + * clauses (either non-mvcompatible, or not related to the histogram). + */ +static List * +clauselist_mv_split(PlannerInfo *root, SpecialJoinInfo *sjinfo, + List *clauses, Oid varRelid, List **mvclauses, + MVStatisticInfo *mvstats, int types) +{ + int i; + ListCell *l; + List *non_mvclauses = NIL; + + /* FIXME is there a better way to get info on int2vector? */ + int2vector * attrs = mvstats->stakeys; + int numattrs = mvstats->stakeys->dim1; + + Bitmapset *mvattnums = NULL; + + /* build bitmap of attributes covered by the stats, so we can + * do bms_is_subset later */ + for (i = 0; i < numattrs; i++) + mvattnums = bms_add_member(mvattnums, attrs->values[i]); + + /* erase the list of mv-compatible clauses */ + *mvclauses = NIL; + + foreach (l, clauses) + { + bool match = false; /* by default not mv-compatible */ + Bitmapset *attnums = NULL; + Node *clause = (Node *) lfirst(l); + + if (clause_is_mv_compatible(root, clause, varRelid, NULL, + &attnums, sjinfo, types)) + { + /* are all the attributes part of the selected stats? */ + if (bms_is_subset(attnums, mvattnums)) + match = true; + } + + /* + * The clause matches the selected stats, so put it to the list + * of mv-compatible clauses. Otherwise, keep it in the list of + * 'regular' clauses (that may be selected later). + */ + if (match) + *mvclauses = lappend(*mvclauses, clause); + else + non_mvclauses = lappend(non_mvclauses, clause); + } + + /* + * Perform regular estimation using the clauses incompatible + * with the chosen histogram (or MV stats in general). + */ + return non_mvclauses; + +} + +/* * Determines whether the clause is compatible with multivariate stats, * and if it is, returns some additional information - varno (index * into simple_rte_array) and a bitmap of attributes. This is then @@ -1005,8 +1378,12 @@ collect_mv_attnums(PlannerInfo *root, List *clauses, Oid varRelid, */ static bool clause_is_mv_compatible(PlannerInfo *root, Node *clause, Oid varRelid, - Index *relid, AttrNumber *attnum, SpecialJoinInfo *sjinfo) + Index *relid, Bitmapset **attnums, SpecialJoinInfo *sjinfo, + int types) { + Relids clause_relids; + Relids left_relids; + Relids right_relids; if (IsA(clause, RestrictInfo)) { @@ -1016,82 +1393,176 @@ clause_is_mv_compatible(PlannerInfo *root, Node *clause, Oid varRelid, if (rinfo->pseudoconstant) return false; - /* no support for OR clauses at this point */ - if (rinfo->orclause) - return false; - /* get the actual clause from the RestrictInfo (it's not an OR clause) */ clause = (Node*)rinfo->clause; - /* only simple opclauses are compatible with multivariate stats */ - if (! is_opclause(clause)) - return false; - /* we don't support join conditions at this moment */ if (treat_as_join_clause(clause, rinfo, varRelid, sjinfo)) return false; + clause_relids = rinfo->clause_relids; + left_relids = rinfo->left_relids; + right_relids = rinfo->right_relids; + } + else if (is_opclause(clause) && list_length(((OpExpr *) clause)->args) == 2) + { + left_relids = pull_varnos(get_leftop((Expr*)clause)); + right_relids = pull_varnos(get_rightop((Expr*)clause)); + + clause_relids = bms_union(left_relids, + right_relids); + } + else + { + /* Not a binary opclause, so mark left/right relid sets as empty */ + left_relids = NULL; + right_relids = NULL; + /* and get the total relid set the hard way */ + clause_relids = pull_varnos((Node *) clause); + } + + /* + * Only simple opclauses and IS NULL tests are compatible with + * multivariate stats at this point. + */ + if ((is_opclause(clause)) + && (list_length(((OpExpr *) clause)->args) == 2)) + { + OpExpr *expr = (OpExpr *) clause; + bool varonleft = true; + bool ok; + /* is it 'variable op constant' ? */ - if (list_length(((OpExpr *) clause)->args) == 2) + + ok = (bms_membership(clause_relids) == BMS_SINGLETON) && + (is_pseudo_constant_clause_relids(lsecond(expr->args), + right_relids) || + (varonleft = false, + is_pseudo_constant_clause_relids(linitial(expr->args), + left_relids))); + + if (ok) { - OpExpr *expr = (OpExpr *) clause; - bool varonleft = true; - bool ok; + Var * var = (varonleft) ? linitial(expr->args) : lsecond(expr->args); - ok = (bms_membership(rinfo->clause_relids) == BMS_SINGLETON) && - (is_pseudo_constant_clause_relids(lsecond(expr->args), - rinfo->right_relids) || - (varonleft = false, - is_pseudo_constant_clause_relids(linitial(expr->args), - rinfo->left_relids))); + /* + * Simple variables only - otherwise the planner_rt_fetch seems to fail + * (return NULL). + * + * TODO Maybe use examine_variable() would fix that? + */ + if (! (IsA(var, Var) && (varRelid == 0 || varRelid == var->varno))) + return false; - if (ok) - { - Var * var = (varonleft) ? linitial(expr->args) : lsecond(expr->args); + /* + * Only consider this variable if (varRelid == 0) or when the varno + * matches varRelid (see explanation at clause_selectivity). + * + * FIXME I suspect this may not be really necessary. The (varRelid == 0) + * part seems to be enforced by treat_as_join_clause(). + */ + if (! ((varRelid == 0) || (varRelid == var->varno))) + return false; - /* - * Simple variables only - otherwise the planner_rt_fetch seems to fail - * (return NULL). - * - * TODO Maybe use examine_variable() would fix that? - */ - if (! (IsA(var, Var) && (varRelid == 0 || varRelid == var->varno))) - return false; + /* Also skip special varno values, and system attributes ... */ + if ((IS_SPECIAL_VARNO(var->varno)) || (! AttrNumberIsForUserDefinedAttr(var->varattno))) + return false; - /* - * Only consider this variable if (varRelid == 0) or when the varno - * matches varRelid (see explanation at clause_selectivity). - * - * FIXME I suspect this may not be really necessary. The (varRelid == 0) - * part seems to be enforced by treat_as_join_clause(). - */ - if (! ((varRelid == 0) || (varRelid == var->varno))) - return false; + /* Lookup info about the base relation (we need to pass the OID out) */ + if (relid != NULL) + *relid = var->varno; + + /* + * If it's not a "<" or ">" or "=" operator, just ignore the + * clause. Otherwise note the relid and attnum for the variable. + * This uses the function for estimating selectivity, ont the + * operator directly (a bit awkward, but well ...). + */ + switch (get_oprrest(expr->opno)) + { + case F_SCALARLTSEL: + case F_SCALARGTSEL: + /* not compatible with functional dependencies */ + if (types & MV_CLAUSE_TYPE_MCV) + { + *attnums = bms_add_member(*attnums, var->varattno); + return (types & MV_CLAUSE_TYPE_MCV); + } + return false; + + case F_EQSEL: + *attnums = bms_add_member(*attnums, var->varattno); + return true; + } + } + } + else if (IsA(clause, NullTest) + && IsA(((NullTest*)clause)->arg, Var)) + { + Var * var = (Var*)((NullTest*)clause)->arg; + + /* + * Simple variables only - otherwise the planner_rt_fetch seems to fail + * (return NULL). + * + * TODO Maybe use examine_variable() would fix that? + */ + if (! (IsA(var, Var) && (varRelid == 0 || varRelid == var->varno))) + return false; + + /* + * Only consider this variable if (varRelid == 0) or when the varno + * matches varRelid (see explanation at clause_selectivity). + * + * FIXME I suspect this may not be really necessary. The (varRelid == 0) + * part seems to be enforced by treat_as_join_clause(). + */ + if (! ((varRelid == 0) || (varRelid == var->varno))) + return false; - /* Also skip special varno values, and system attributes ... */ - if ((IS_SPECIAL_VARNO(var->varno)) || (! AttrNumberIsForUserDefinedAttr(var->varattno))) - return false; + /* Also skip special varno values, and system attributes ... */ + if ((IS_SPECIAL_VARNO(var->varno)) || (! AttrNumberIsForUserDefinedAttr(var->varattno))) + return false; + /* Lookup info about the base relation (we need to pass the OID out) */ + if (relid != NULL) *relid = var->varno; - /* - * If it's not a "<" or ">" or "=" operator, just ignore the - * clause. Otherwise note the relid and attnum for the variable. - * This uses the function for estimating selectivity, ont the - * operator directly (a bit awkward, but well ...). - */ - switch (get_oprrest(expr->opno)) - { - case F_EQSEL: - *attnum = var->varattno; - return true; - } - } + *attnums = bms_add_member(*attnums, var->varattno); + + return true; + } + else if (or_clause(clause) || and_clause(clause)) + { + /* + * AND/OR-clauses are supported if all sub-clauses are supported + * + * TODO We might support mixed case, where some of the clauses + * are supported and some are not, and treat all supported + * subclauses as a single clause, compute it's selectivity + * using mv stats, and compute the total selectivity using + * the current algorithm. + * + * TODO For RestrictInfo above an OR-clause, we might use the + * orclause with nested RestrictInfo - we won't have to + * call pull_varnos() for each clause, saving time. + */ + Bitmapset *tmp = NULL; + ListCell *l; + foreach (l, ((BoolExpr*)clause)->args) + { + if (! clause_is_mv_compatible(root, (Node*)lfirst(l), + varRelid, relid, &tmp, sjinfo, types)) + return false; } + + /* add the attnums from the OR-clause to the set of attnums */ + *attnums = bms_join(*attnums, tmp); + + return true; } return false; - } /* @@ -1340,6 +1811,9 @@ has_stats(List *stats, int type) if ((type & MV_CLAUSE_TYPE_FDEP) && stat->deps_built) return true; + + if ((type & MV_CLAUSE_TYPE_MCV) && stat->mcv_built) + return true; } return false; @@ -1635,25 +2109,39 @@ fdeps_filter_clauses(PlannerInfo *root, foreach (lc, clauses) { - AttrNumber attnum; + Bitmapset *attnums = NULL; Node *clause = (Node *) lfirst(lc); - if (! clause_is_mv_compatible(root, clause, varRelid, relid, - &attnum, sjinfo)) + if (! clause_is_mv_compatible(root, clause, varRelid, relid, &attnums, + sjinfo, MV_CLAUSE_TYPE_FDEP)) /* clause incompatible with functional dependencies */ *reduced_clauses = lappend(*reduced_clauses, clause); - else if (! bms_is_member(attnum, deps_attnums)) + else if (bms_num_members(attnums) > 1) + + /* + * clause referencing multiple attributes (strange, should + * this be handled by clause_is_mv_compatible directly) + */ + *reduced_clauses = lappend(*reduced_clauses, clause); + + else if (! bms_is_member(bms_singleton_member(attnums), deps_attnums)) /* clause not covered by the dependencies */ *reduced_clauses = lappend(*reduced_clauses, clause); else { + /* ok, clause compatible with existing dependencies */ + Assert(bms_num_members(attnums) == 1); + *deps_clauses = lappend(*deps_clauses, clause); - clause_attnums = bms_add_member(clause_attnums, attnum); + clause_attnums = bms_add_member(clause_attnums, + bms_singleton_member(attnums)); } + + bms_free(attnums); } return clause_attnums; @@ -1691,3 +2179,454 @@ get_varattnos(Node * node, Index relid) return result; } + +/* + * Estimate selectivity of clauses using a MCV list. + * + * If there's no MCV list for the stats, the function returns 0.0. + * + * While computing the estimate, the function checks whether all the + * columns were matched with an equality condition. If that's the case, + * we can skip processing the histogram, as there can be no rows in + * it with the same values - all the rows matching the condition are + * represented by the MCV item. This can only happen with equality + * on all the attributes. + * + * The algorithm works like this: + * + * 1) mark all items as 'match' + * 2) walk through all the clauses + * 3) for a particular clause, walk through all the items + * 4) skip items that are already 'no match' + * 5) check clause for items that still match + * 6) sum frequencies for items to get selectivity + * + * The function also returns the frequency of the least frequent item + * on the MCV list, which may be useful for clamping estimate from the + * histogram (all items not present in the MCV list are less frequent). + * This however seems useful only for cases with conditions on all + * attributes. + * + * TODO This only handles AND-ed clauses, but it might work for OR-ed + * lists too - it just needs to reverse the logic a bit. I.e. start + * with 'no match' for all items, and mark the items as a match + * as the clauses are processed (and skip items that are 'match'). + */ +static Selectivity +clauselist_mv_selectivity_mcvlist(PlannerInfo *root, List *clauses, + MVStatisticInfo *mvstats, bool *fullmatch, + Selectivity *lowsel) +{ + int i; + Selectivity s = 0.0; + Selectivity u = 0.0; + + MCVList mcvlist = NULL; + int nmatches = 0; + + /* match/mismatch bitmap for each MCV item */ + char * matches = NULL; + + Assert(clauses != NIL); + Assert(list_length(clauses) >= 2); + + /* there's no MCV list built yet */ + if (! mvstats->mcv_built) + return 0.0; + + mcvlist = load_mv_mcvlist(mvstats->mvoid); + + Assert(mcvlist != NULL); + Assert(mcvlist->nitems > 0); + + /* by default all the MCV items match the clauses fully */ + matches = palloc0(sizeof(char) * mcvlist->nitems); + memset(matches, MVSTATS_MATCH_FULL, sizeof(char)*mcvlist->nitems); + + /* number of matching MCV items */ + nmatches = mcvlist->nitems; + + nmatches = update_match_bitmap_mcvlist(root, clauses, + mvstats->stakeys, mcvlist, + nmatches, matches, + lowsel, fullmatch, false); + + /* sum frequencies for all the matching MCV items */ + for (i = 0; i < mcvlist->nitems; i++) + { + /* used to 'scale' for MCV lists not covering all tuples */ + u += mcvlist->items[i]->frequency; + + if (matches[i] != MVSTATS_MATCH_NONE) + s += mcvlist->items[i]->frequency; + } + + pfree(matches); + pfree(mcvlist); + + return s*u; +} + +/* + * Evaluate clauses using the MCV list, and update the match bitmap. + * + * The bitmap may be already partially set, so this is really a way to + * combine results of several clause lists - either when computing + * conditional probability P(A|B) or a combination of AND/OR clauses. + * + * TODO This works with 'bitmap' where each bit is represented as a char, + * which is slightly wasteful. Instead, we could use a regular + * bitmap, reducing the size to ~1/8. Another thing is merging the + * bitmaps using & and |, which might be faster than min/max. + */ +static int +update_match_bitmap_mcvlist(PlannerInfo *root, List *clauses, + int2vector *stakeys, MCVList mcvlist, + int nmatches, char * matches, + Selectivity *lowsel, bool *fullmatch, + bool is_or) +{ + int i; + ListCell * l; + + Bitmapset *eqmatches = NULL; /* attributes with equality matches */ + + /* The bitmap may be partially built. */ + Assert(nmatches >= 0); + Assert(nmatches <= mcvlist->nitems); + Assert(clauses != NIL); + Assert(list_length(clauses) >= 1); + Assert(mcvlist != NULL); + Assert(mcvlist->nitems > 0); + + /* No possible matches (only works for AND-ded clauses) */ + if (((nmatches == 0) && (! is_or)) || + ((nmatches == mcvlist->nitems) && is_or)) + return nmatches; + + /* frequency of the lowest MCV item */ + *lowsel = 1.0; + + /* + * Loop through the list of clauses, and for each of them evaluate + * all the MCV items not yet eliminated by the preceding clauses. + * + * FIXME This would probably deserve a refactoring, I guess. Unify + * the two loops and put the checks inside, or something like + * that. + */ + foreach (l, clauses) + { + Node * clause = (Node*)lfirst(l); + + /* if it's a RestrictInfo, then extract the clause */ + if (IsA(clause, RestrictInfo)) + clause = (Node*)((RestrictInfo*)clause)->clause; + + /* if there are no remaining matches possible, we can stop */ + if (((nmatches == 0) && (! is_or)) || + ((nmatches == mcvlist->nitems) && is_or)) + break; + + /* it's either OpClause, or NullTest */ + if (is_opclause(clause)) + { + OpExpr * expr = (OpExpr*)clause; + bool varonleft = true; + bool ok; + + /* operator */ + FmgrInfo opproc; + + /* get procedure computing operator selectivity */ + RegProcedure oprrest = get_oprrest(expr->opno); + + fmgr_info(get_opcode(expr->opno), &opproc); + + ok = (NumRelids(clause) == 1) && + (is_pseudo_constant_clause(lsecond(expr->args)) || + (varonleft = false, + is_pseudo_constant_clause(linitial(expr->args)))); + + if (ok) + { + + FmgrInfo ltproc, gtproc; + Var * var = (varonleft) ? linitial(expr->args) : lsecond(expr->args); + Const * cst = (varonleft) ? lsecond(expr->args) : linitial(expr->args); + bool isgt = (! varonleft); + + /* + * TODO Fetch only when really needed (probably for equality only) + * TODO Technically either lt/gt is sufficient. + * + * FIXME The code in analyze.c creates histograms only for types + * with enough ordering (by calling get_sort_group_operators). + * Is this the same assumption, i.e. are we certain that we + * get the ltproc/gtproc every time we ask? Or are there types + * where get_sort_group_operators returns ltopr and here we + * get nothing? + */ + TypeCacheEntry *typecache + = lookup_type_cache(var->vartype, + TYPECACHE_EQ_OPR | TYPECACHE_LT_OPR | TYPECACHE_GT_OPR); + + /* FIXME proper matching attribute to dimension */ + int idx = mv_get_index(var->varattno, stakeys); + + fmgr_info(get_opcode(typecache->lt_opr), <proc); + fmgr_info(get_opcode(typecache->gt_opr), >proc); + + /* + * Walk through the MCV items and evaluate the current clause. We can + * skip items that were already ruled out, and terminate if there are + * no remaining MCV items that might possibly match. + */ + for (i = 0; i < mcvlist->nitems; i++) + { + bool mismatch = false; + MCVItem item = mcvlist->items[i]; + + /* + * find the lowest selectivity in the MCV + * FIXME Maybe not the best place do do this (in for all clauses). + */ + if (item->frequency < *lowsel) + *lowsel = item->frequency; + + /* + * If there are no more matches (AND) or no remaining unmatched + * items (OR), we can stop processing this clause. + */ + if (((nmatches == 0) && (! is_or)) || + ((nmatches == mcvlist->nitems) && is_or)) + break; + + /* + * For AND-lists, we can also mark NULL items as 'no match' (and + * then skip them). For OR-lists this is not possible. + */ + if ((! is_or) && item->isnull[idx]) + matches[i] = MVSTATS_MATCH_NONE; + + /* skip MCV items that were already ruled out */ + if ((! is_or) && (matches[i] == MVSTATS_MATCH_NONE)) + continue; + else if (is_or && (matches[i] == MVSTATS_MATCH_FULL)) + continue; + + /* TODO consider bsearch here (list is sorted by values) + * TODO handle other operators too (LT, GT) + * TODO identify "full match" when the clauses fully + * match the whole MCV list (so that checking the + * histogram is not needed) + */ + if (oprrest == F_EQSEL) + { + /* + * We don't care about isgt in equality, because it does not + * matter whether it's (var = const) or (const = var). + */ + bool match = DatumGetBool(FunctionCall2Coll(&opproc, + DEFAULT_COLLATION_OID, + cst->constvalue, + item->values[idx])); + + if (match) + eqmatches = bms_add_member(eqmatches, idx); + + mismatch = (! match); + } + else if (oprrest == F_SCALARLTSEL) /* column < constant */ + { + + if (! isgt) /* (var < const) */ + { + /* + * First check whether the constant is below the lower boundary (in that + * case we can skip the bucket, because there's no overlap). + */ + mismatch = DatumGetBool(FunctionCall2Coll(&opproc, + DEFAULT_COLLATION_OID, + cst->constvalue, + item->values[idx])); + + } /* (get_oprrest(expr->opno) == F_SCALARLTSEL) */ + else /* (const < var) */ + { + /* + * First check whether the constant is above the upper boundary (in that + * case we can skip the bucket, because there's no overlap). + */ + mismatch = DatumGetBool(FunctionCall2Coll(&opproc, + DEFAULT_COLLATION_OID, + item->values[idx], + cst->constvalue)); + } + } + else if (oprrest == F_SCALARGTSEL) /* column > constant */ + { + + if (! isgt) /* (var > const) */ + { + /* + * First check whether the constant is above the upper boundary (in that + * case we can skip the bucket, because there's no overlap). + */ + mismatch = DatumGetBool(FunctionCall2Coll(&opproc, + DEFAULT_COLLATION_OID, + cst->constvalue, + item->values[idx])); + } + else /* (const > var) */ + { + /* + * First check whether the constant is below the lower boundary (in + * that case we can skip the bucket, because there's no overlap). + */ + mismatch = DatumGetBool(FunctionCall2Coll(&opproc, + DEFAULT_COLLATION_OID, + item->values[idx], + cst->constvalue)); + } + + } /* (get_oprrest(expr->opno) == F_SCALARGTSEL) */ + + /* XXX The conditions on matches[i] are not needed, as we + * skip MCV items that can't become true/false, depending + * on the current flag. See beginning of the loop over + * MCV items. + */ + + if ((is_or) && (matches[i] == MVSTATS_MATCH_NONE) && (! mismatch)) + { + /* OR - was MATCH_NONE, but will be MATCH_FULL */ + matches[i] = MVSTATS_MATCH_FULL; + ++nmatches; + continue; + } + else if ((! is_or) && (matches[i] == MVSTATS_MATCH_FULL) && mismatch) + { + /* AND - was MATC_FULL, but will be MATCH_NONE */ + matches[i] = MVSTATS_MATCH_NONE; + --nmatches; + continue; + } + + } + } + } + else if (IsA(clause, NullTest)) + { + NullTest * expr = (NullTest*)clause; + Var * var = (Var*)(expr->arg); + + /* FIXME proper matching attribute to dimension */ + int idx = mv_get_index(var->varattno, stakeys); + + /* + * Walk through the MCV items and evaluate the current clause. We can + * skip items that were already ruled out, and terminate if there are + * no remaining MCV items that might possibly match. + */ + for (i = 0; i < mcvlist->nitems; i++) + { + MCVItem item = mcvlist->items[i]; + + /* + * find the lowest selectivity in the MCV + * FIXME Maybe not the best place do do this (in for all clauses). + */ + if (item->frequency < *lowsel) + *lowsel = item->frequency; + + /* if there are no more matches, we can stop processing this clause */ + if (nmatches == 0) + break; + + /* skip MCV items that were already ruled out */ + if (matches[i] == MVSTATS_MATCH_NONE) + continue; + + /* if the clause mismatches the MCV item, set it as MATCH_NONE */ + if (((expr->nulltesttype == IS_NULL) && (! mcvlist->items[i]->isnull[idx])) || + ((expr->nulltesttype == IS_NOT_NULL) && (mcvlist->items[i]->isnull[idx]))) + { + matches[i] = MVSTATS_MATCH_NONE; + --nmatches; + } + } + } + else if (or_clause(clause) || and_clause(clause)) + { + /* AND/OR clause, with all clauses compatible with the selected MV stat */ + + int i; + BoolExpr *orclause = ((BoolExpr*)clause); + List *orclauses = orclause->args; + + /* match/mismatch bitmap for each MCV item */ + int or_nmatches = 0; + char * or_matches = NULL; + + Assert(orclauses != NIL); + Assert(list_length(orclauses) >= 2); + + /* number of matching MCV items */ + or_nmatches = mcvlist->nitems; + + /* by default none of the MCV items matches the clauses */ + or_matches = palloc0(sizeof(char) * or_nmatches); + + if (or_clause(clause)) + { + /* OR clauses assume nothing matches, initially */ + memset(or_matches, MVSTATS_MATCH_NONE, sizeof(char)*or_nmatches); + or_nmatches = 0; + } + else + { + /* AND clauses assume nothing matches, initially */ + memset(or_matches, MVSTATS_MATCH_FULL, sizeof(char)*or_nmatches); + } + + /* build the match bitmap for the OR-clauses */ + or_nmatches = update_match_bitmap_mcvlist(root, orclauses, + stakeys, mcvlist, + or_nmatches, or_matches, + lowsel, fullmatch, or_clause(clause)); + + /* merge the bitmap into the existing one*/ + for (i = 0; i < mcvlist->nitems; i++) + { + /* + * To AND-merge the bitmaps, a MIN() semantics is used. + * For OR-merge, use MAX(). + * + * FIXME this does not decrease the number of matches + */ + UPDATE_RESULT(matches[i], or_matches[i], is_or); + } + + pfree(or_matches); + + } + else + { + elog(ERROR, "unknown clause type: %d", clause->type); + } + } + + /* + * If all the columns were matched by equality, it's a full match. + * In this case there can be just a single MCV item, matching the + * clause (if there were two, both would match the other one). + */ + *fullmatch = (bms_num_members(eqmatches) == mcvlist->ndimensions); + + /* free the allocated pieces */ + if (eqmatches) + pfree(eqmatches); + + return nmatches; +} diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index c397773..8c4396a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -409,7 +409,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, mvstat = (Form_pg_mv_statistic) GETSTRUCT(htup); /* unavailable stats are not interesting for the planner */ - if (mvstat->deps_built) + if (mvstat->deps_built || mvstat->mcv_built) { info = makeNode(MVStatisticInfo); @@ -418,9 +418,11 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, /* enabled statistics */ info->deps_enabled = mvstat->deps_enabled; + info->mcv_enabled = mvstat->mcv_enabled; /* built/available statistics */ info->deps_built = mvstat->deps_built; + info->mcv_built = mvstat->mcv_built; /* stakeys */ adatum = SysCacheGetAttr(MVSTATOID, htup, diff --git a/src/backend/utils/mvstats/Makefile b/src/backend/utils/mvstats/Makefile index 099f1ed..f9bf10c 100644 --- a/src/backend/utils/mvstats/Makefile +++ b/src/backend/utils/mvstats/Makefile @@ -12,6 +12,6 @@ subdir = src/backend/utils/mvstats top_builddir = ../../../.. include $(top_builddir)/src/Makefile.global -OBJS = common.o dependencies.o +OBJS = common.o dependencies.o mcv.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/utils/mvstats/common.c b/src/backend/utils/mvstats/common.c index bd200bc..d1da714 100644 --- a/src/backend/utils/mvstats/common.c +++ b/src/backend/utils/mvstats/common.c @@ -16,12 +16,14 @@ #include "common.h" +#include "utils/array.h" + static VacAttrStats ** lookup_var_attr_stats(int2vector *attrs, - int natts, VacAttrStats **vacattrstats); + int natts, + VacAttrStats **vacattrstats); static List* list_mv_stats(Oid relid); - /* * Compute requested multivariate stats, using the rows sampled for the * plain (single-column) stats. @@ -49,6 +51,8 @@ build_mv_stats(Relation onerel, int numrows, HeapTuple *rows, int j; MVStatisticInfo *stat = (MVStatisticInfo *)lfirst(lc); MVDependencies deps = NULL; + MCVList mcvlist = NULL; + int numrows_filtered = 0; VacAttrStats **stats = NULL; int numatts = 0; @@ -87,8 +91,12 @@ build_mv_stats(Relation onerel, int numrows, HeapTuple *rows, if (stat->deps_enabled) deps = build_mv_dependencies(numrows, rows, attrs, stats); + /* build the MCV list */ + if (stat->mcv_enabled) + mcvlist = build_mv_mcvlist(numrows, rows, attrs, stats, &numrows_filtered); + /* store the histogram / MCV list in the catalog */ - update_mv_stats(stat->mvoid, deps, attrs); + update_mv_stats(stat->mvoid, deps, mcvlist, attrs, stats); } } @@ -166,6 +174,8 @@ list_mv_stats(Oid relid) info->stakeys = buildint2vector(stats->stakeys.values, stats->stakeys.dim1); info->deps_enabled = stats->deps_enabled; info->deps_built = stats->deps_built; + info->mcv_enabled = stats->mcv_enabled; + info->mcv_built = stats->mcv_built; result = lappend(result, info); } @@ -180,8 +190,56 @@ list_mv_stats(Oid relid) return result; } + +/* + * Find attnims of MV stats using the mvoid. + */ +int2vector* +find_mv_attnums(Oid mvoid, Oid *relid) +{ + ArrayType *arr; + Datum adatum; + bool isnull; + HeapTuple htup; + int2vector *keys; + + /* Prepare to scan pg_mv_statistic for entries having indrelid = this rel. */ + htup = SearchSysCache1(MVSTATOID, + ObjectIdGetDatum(mvoid)); + + /* XXX syscache contains OIDs of deleted stats (not invalidated) */ + if (! HeapTupleIsValid(htup)) + return NULL; + + /* starelid */ + adatum = SysCacheGetAttr(MVSTATOID, htup, + Anum_pg_mv_statistic_starelid, &isnull); + Assert(!isnull); + + *relid = DatumGetObjectId(adatum); + + /* stakeys */ + adatum = SysCacheGetAttr(MVSTATOID, htup, + Anum_pg_mv_statistic_stakeys, &isnull); + Assert(!isnull); + + arr = DatumGetArrayTypeP(adatum); + + keys = buildint2vector((int16 *) ARR_DATA_PTR(arr), + ARR_DIMS(arr)[0]); + ReleaseSysCache(htup); + + /* TODO maybe save the list into relcache, as in RelationGetIndexList + * (which was used as an inspiration of this one)?. */ + + return keys; +} + + void -update_mv_stats(Oid mvoid, MVDependencies dependencies, int2vector *attrs) +update_mv_stats(Oid mvoid, + MVDependencies dependencies, MCVList mcvlist, + int2vector *attrs, VacAttrStats **stats) { HeapTuple stup, oldtup; @@ -206,18 +264,29 @@ update_mv_stats(Oid mvoid, MVDependencies dependencies, int2vector *attrs) = PointerGetDatum(serialize_mv_dependencies(dependencies)); } + if (mcvlist != NULL) + { + bytea * data = serialize_mv_mcvlist(mcvlist, attrs, stats); + nulls[Anum_pg_mv_statistic_stamcv -1] = (data == NULL); + values[Anum_pg_mv_statistic_stamcv - 1] = PointerGetDatum(data); + } + /* always replace the value (either by bytea or NULL) */ replaces[Anum_pg_mv_statistic_stadeps -1] = true; + replaces[Anum_pg_mv_statistic_stamcv -1] = true; /* always change the availability flags */ nulls[Anum_pg_mv_statistic_deps_built -1] = false; + nulls[Anum_pg_mv_statistic_mcv_built -1] = false; nulls[Anum_pg_mv_statistic_stakeys-1] = false; /* use the new attnums, in case we removed some dropped ones */ replaces[Anum_pg_mv_statistic_deps_built-1] = true; + replaces[Anum_pg_mv_statistic_mcv_built -1] = true; replaces[Anum_pg_mv_statistic_stakeys -1] = true; values[Anum_pg_mv_statistic_deps_built-1] = BoolGetDatum(dependencies != NULL); + values[Anum_pg_mv_statistic_mcv_built -1] = BoolGetDatum(mcvlist != NULL); values[Anum_pg_mv_statistic_stakeys -1] = PointerGetDatum(attrs); /* Is there already a pg_mv_statistic tuple for this attribute? */ @@ -246,6 +315,21 @@ update_mv_stats(Oid mvoid, MVDependencies dependencies, int2vector *attrs) heap_close(sd, RowExclusiveLock); } + +int +mv_get_index(AttrNumber varattno, int2vector * stakeys) +{ + int i, idx = 0; + for (i = 0; i < stakeys->dim1; i++) + { + if (stakeys->values[i] < varattno) + idx += 1; + else + break; + } + return idx; +} + /* multi-variate stats comparator */ /* @@ -256,11 +340,15 @@ update_mv_stats(Oid mvoid, MVDependencies dependencies, int2vector *attrs) int compare_scalars_simple(const void *a, const void *b, void *arg) { - Datum da = *(Datum*)a; - Datum db = *(Datum*)b; - SortSupport ssup= (SortSupport) arg; + return compare_datums_simple(*(Datum*)a, + *(Datum*)b, + (SortSupport)arg); +} - return ApplySortComparator(da, false, db, false, ssup); +int +compare_datums_simple(Datum a, Datum b, SortSupport ssup) +{ + return ApplySortComparator(a, false, b, false, ssup); } /* diff --git a/src/backend/utils/mvstats/common.h b/src/backend/utils/mvstats/common.h index 6d5465b..f4309f7 100644 --- a/src/backend/utils/mvstats/common.h +++ b/src/backend/utils/mvstats/common.h @@ -46,7 +46,15 @@ typedef struct Datum value; /* a data value */ int tupno; /* position index for tuple it came from */ } ScalarItem; - + +/* (de)serialization info */ +typedef struct DimensionInfo { + int nvalues; /* number of deduplicated values */ + int nbytes; /* number of bytes (serialized) */ + int typlen; /* pg_type.typlen */ + bool typbyval; /* pg_type.typbyval */ +} DimensionInfo; + /* multi-sort */ typedef struct MultiSortSupportData { int ndims; /* number of dimensions supported by the */ @@ -71,5 +79,6 @@ int multi_sort_compare_dim(int dim, const SortItem *a, const SortItem *b, MultiSortSupport mss); /* comparators, used when constructing multivariate stats */ +int compare_datums_simple(Datum a, Datum b, SortSupport ssup); int compare_scalars_simple(const void *a, const void *b, void *arg); int compare_scalars_partition(const void *a, const void *b, void *arg); diff --git a/src/backend/utils/mvstats/mcv.c b/src/backend/utils/mvstats/mcv.c new file mode 100644 index 0000000..670dbda --- /dev/null +++ b/src/backend/utils/mvstats/mcv.c @@ -0,0 +1,1237 @@ +/*------------------------------------------------------------------------- + * + * mcv.c + * POSTGRES multivariate MCV lists + * + * + * Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * + * IDENTIFICATION + * src/backend/utils/mvstats/mcv.c + * + *------------------------------------------------------------------------- + */ + +#include "postgres.h" + +#include "fmgr.h" +#include "funcapi.h" + +#include "utils/lsyscache.h" + +#include "common.h" + +/* + * Multivariate MCVs (most-common values lists) are a straightforward + * extension of regular MCV list, tracking combinations of values for + * several attributes (columns), including NULL flags, and frequency + * of the combination. + * + * For columns with small number of distinct values, this works quite + * well and may represent the distribution very accurately. For columns + * with large number of distinct values (e.g. stored as FLOAT), this + * does not work that well. Especially if the distribution is mostly + * uniform, with no very common combinations. + * + * If we can represent the distribution as a MCV list, we can estimate + * some clauses (e.g. equality clauses) much accurately than using + * histograms for example. + * + * Another benefit of MCV lists (compared to histograms) is that they + * don't require sorting of the values, so that they work better for + * data types that either don't support sorting at all, or when the + * sorting does not really match the meaning. For example we know how to + * sort strings, but it's unlikely to make much sense for city names. + * + * + * Hashed MCV (not yet implemented) + * -------------------------------- + * By restricting to MCV list and equality conditions, we may use hash + * values instead of the long varlena values. This significantly reduces + * the storage requirements, and we can still use it to estimate the + * equality conditions (assuming the collisions are rare enough). + * + * This however complicates matching the columns to available stats, as + * it requires matching clauses (not columns) to stats. And it may get + * quite complex - e.g. what if there are multiple clauses, each + * compatible with different stats subset? + * + * + * Selectivity estimation + * ---------------------- + * The estimation, implemented in clauselist_mv_selectivity_mcvlist(), + * is quite simple in principle - walk through the MCV items and sum + * frequencies of all the items that match all the clauses. + * + * The current implementation uses MCV lists to estimates those types + * of clauses (think of WHERE conditions): + * + * (a) equality clauses WHERE (a = 1) AND (b = 2) + * (b) inequality clauses WHERE (a < 1) AND (b >= 2) + * (c) NULL clauses WHERE (a IS NULL) AND (b IS NOT NULL) + * (d) OR clauses WHERE (a < 1) OR (b >= 2) + * + * It's possible to add more clauses, for example: + * + * (e) multi-var clauses WHERE (a > b) + * + * and so on. These are tasks for the future, not yet implemented. + * + * + * Estimating equality clauses + * --------------------------- + * When computing selectivity estimate for equality clauses + * + * (a = 1) AND (b = 2) + * + * we can do this estimate pretty exactly assuming that two conditions + * are met: + * + * (1) there's an equality condition on each attribute + * + * (2) we find a matching item in the MCV list + * + * In that case we know the MCV item represents all the tuples matching + * the clauses, and the selectivity estimate is complete. This is what + * we call 'full match'. + * + * When only (1) holds, but there's no matching MCV item, we don't know + * whether there are no such rows or just are not very frequent. We can + * however use the frequency of the least frequent MCV item as an upper + * bound for the selectivity. + * + * If the equality conditions match only a subset of the attributes + * the MCV list is built on (i.e. we can't get a full match - we may get + * multiple MCV items matching the clauses, but even if we get a single + * match there may be items that did not get into the MCV list. But in + * this case we can still use the frequency of the last MCV item to clam + * the 'additional' selectivity not accounted for by the matching items. + * + * If there's no histogram, because the MCV list approximates the + * distribution accurately (not because the histogram was disabled), + * it does not really matter whether there are equality conditions on + * all the columns - we can do pretty accurate estimation using the MCV. + * + * TODO For a combination of equality conditions (not full-match case) + * we probably can clamp the selectivity by the minimum of + * selectivities for each condition. For example if we know the + * number of distinct values for each column, we can use 1/ndistinct + * as a per-column estimate. Or rather 1/ndistinct + selectivity + * derived from the MCV list. + * + * If we know the estimate of number of combinations of the columns + * (i.e. ndistinct(A,B)), we may estimate the average frequency of + * items in the remaining 10% as [10% / ndistinct(A,B)]. + * + * + * Bounding estimates + * ------------------ + * In general the MCV lists may not provide estimates as accurate as + * for the full-match equality case, but may provide some useful + * lower/upper boundaries for the estimation error. + * + * With equality clauses we can do a few more tricks to narrow this + * error range (see the previous section and TODO), but with inequality + * clauses (or generally non-equality clauses), it's rather dificult. + * There's nothing like a 'full match' - we have to consider both the + * MCV items and the remaining part every time. We can't use the minimum + * selectivity of MCV items, as the clauses may match multiple items. + * + * For example with a MCV list on columns (A, B), covering 90% of the + * table (computed while building the MCV list), about ~10% of the table + * is not represented by the MCV list. So even if the conditions match + * all the remaining rows (not represented by the MCV items), we can't + * get selectivity higher than those 10%. We may use 1/2 the remaining + * selectivity as an estimate (minimizing average error). + * + * TODO Most of these ideas (error limiting) are not yet implemented. + * + * + * General TODO + * ------------ + * + * FIXME Use max_mcv_items from ALTER TABLE ADD STATISTICS command. + * + * TODO Add support for clauses referencing multiple columns (a < b). + * + * TODO It's possible to build a special case of MCV list, storing not + * the actual values but only 32/64-bit hash. This is only useful + * for estimating equality clauses and for large varlena types, + * which are very impractical for plain MCV list because of size. + * But for those data types we really want just the equality + * clauses, so it's actually a good solution. + * + * TODO Currently there's no logic to consider building only a MCV list + * (and not building the histogram at all), except for doing this + * decision manually in ADD STATISTICS. + */ + +/* + * Each serialized item needs to store (in this order): + * + * - indexes (ndim * sizeof(int32)) + * - null flags (ndim * sizeof(bool)) + * - frequency (sizeof(double)) + * + * So in total: + * + * ndim * (sizeof(int32) + sizeof(bool)) + sizeof(double) + */ +#define ITEM_SIZE(ndims) \ + (ndims * (sizeof(uint16) + sizeof(bool)) + sizeof(double)) + +/* pointers into a flat serialized item of ITEM_SIZE(n) bytes */ +#define ITEM_INDEXES(item) ((uint16*)item) +#define ITEM_NULLS(item,ndims) ((bool*)(ITEM_INDEXES(item) + ndims)) +#define ITEM_FREQUENCY(item,ndims) ((double*)(ITEM_NULLS(item,ndims) + ndims)) + +/* + * Builds MCV list from sample rows, and removes rows represented by + * the MCV list from the sample (the number of remaining sample rows is + * returned by the numrows_filtered parameter). + * + * The method is quite simple - in short it does about these steps: + * + * (1) sort the data (default collation, '<' for the data type) + * + * (2) count distinct groups, decide how many to keep + * + * (3) build the MCV list using the threshold determined in (2) + * + * (4) remove rows represented by the MCV from the sample + * + * For more details, see the comments in the code. + * + * FIXME Single-dimensional MCV is sorted by frequency (descending). We + * should do that too, because when walking through the list we + * want to check the most frequent items first. + * + * TODO We're using Datum (8B), even for data types (e.g. int4 or + * float4). Maybe we could save some space here, but the bytea + * compression should handle it just fine. + * + * TODO This probably should not use the ndistinct directly (as computed + * from the table, but rather estimate the number of distinct + * values in the table), no? + */ +MCVList +build_mv_mcvlist(int numrows, HeapTuple *rows, int2vector *attrs, + VacAttrStats **stats, int *numrows_filtered) +{ + int i, j; + int numattrs = attrs->dim1; + int ndistinct = 0; + int mcv_threshold = 0; + int count = 0; + int nitems = 0; + + MCVList mcvlist = NULL; + + /* Sort by multiple columns (using array of SortSupport) */ + MultiSortSupport mss = multi_sort_init(numattrs); + + /* + * Preallocate space for all the items as a single chunk, and point + * the items to the appropriate parts of the array. + */ + SortItem *items = (SortItem*)palloc0(numrows * sizeof(SortItem)); + Datum *values = (Datum*)palloc0(sizeof(Datum) * numrows * numattrs); + bool *isnull = (bool*)palloc0(sizeof(bool) * numrows * numattrs); + + /* keep all the rows by default (as if there was no MCV list) */ + *numrows_filtered = numrows; + + for (i = 0; i < numrows; i++) + { + items[i].values = &values[i * numattrs]; + items[i].isnull = &isnull[i * numattrs]; + } + + /* load the values/null flags from sample rows */ + for (j = 0; j < numrows; j++) + for (i = 0; i < numattrs; i++) + items[j].values[i] = heap_getattr(rows[j], attrs->values[i], + stats[i]->tupDesc, &items[j].isnull[i]); + + /* prepare the sort functions for all the attributes */ + for (i = 0; i < numattrs; i++) + multi_sort_add_dimension(mss, i, i, stats); + + /* do the sort, using the multi-sort */ + qsort_arg((void *) items, numrows, sizeof(SortItem), + multi_sort_compare, mss); + + /* + * Count the number of distinct groups - just walk through the + * sorted list and count the number of key changes. We use this to + * determine the threshold (125% of the average frequency). + */ + ndistinct = 1; + for (i = 1; i < numrows; i++) + if (multi_sort_compare(&items[i], &items[i-1], mss) != 0) + ndistinct += 1; + + /* + * Determine how many groups actually exceed the threshold, and then + * walk the array again and collect them into an array. We'll always + * require at least 4 rows per group. + * + * But if we can fit all the distinct values in the MCV list (i.e. + * if there are less distinct groups than MVSTAT_MCVLIST_MAX_ITEMS), + * we'll require only 2 rows per group. + * + * TODO For now the threshold is the same as in the single-column + * case (average + 25%), but maybe that's worth revisiting + * for the multivariate case. + * + * TODO We can do this only if we believe we got all the distinct + * values of the table. + * + * FIXME This should really reference mcv_max_items (from catalog) + * instead of the constant MVSTAT_MCVLIST_MAX_ITEMS. + */ + mcv_threshold = 1.25 * numrows / ndistinct; + mcv_threshold = (mcv_threshold < 4) ? 4 : mcv_threshold; + + if (ndistinct <= MVSTAT_MCVLIST_MAX_ITEMS) + mcv_threshold = 2; + + /* + * Walk through the sorted data again, and see how many groups + * reach the mcv_threshold (and become an item in the MCV list). + */ + count = 1; + for (i = 1; i <= numrows; i++) + { + /* last row or new group, so check if we exceed mcv_threshold */ + if ((i == numrows) || (multi_sort_compare(&items[i], &items[i-1], mss) != 0)) + { + /* group hits the threshold, count the group as MCV item */ + if (count >= mcv_threshold) + nitems += 1; + + count = 1; + } + else /* within group, so increase the number of items */ + count += 1; + } + + /* we know the number of MCV list items, so let's build the list */ + if (nitems > 0) + { + /* allocate the MCV list structure, set parameters we know */ + mcvlist = (MCVList)palloc0(sizeof(MCVListData)); + + mcvlist->magic = MVSTAT_MCV_MAGIC; + mcvlist->type = MVSTAT_MCV_TYPE_BASIC; + mcvlist->ndimensions = numattrs; + mcvlist->nitems = nitems; + + /* + * Preallocate Datum/isnull arrays (not as a single chunk, as + * we'll pass this outside this method and thus it needs to be + * easy to pfree() the data (and we wouldn't know where the + * arrays start). + * + * TODO Maybe the reasoning that we can't allocate a single + * piece because we're passing it out is bogus? Who'd + * free a single item of the MCV list, anyway? + * + * TODO Maybe with a proper encoding (stuffing all the values + * into a list-level array, this will be untrue)? + */ + mcvlist->items = (MCVItem*)palloc0(sizeof(MCVItem)*nitems); + + for (i = 0; i < nitems; i++) + { + mcvlist->items[i] = (MCVItem)palloc0(sizeof(MCVItemData)); + mcvlist->items[i]->values = (Datum*)palloc0(sizeof(Datum)*numattrs); + mcvlist->items[i]->isnull = (bool*)palloc0(sizeof(bool)*numattrs); + } + + /* + * Repeat the same loop as above, but this time copy the data + * into the MCV list (for items exceeding the threshold). + * + * TODO Maybe we could simply remember indexes of the last item + * in each group (from the previous loop)? + */ + count = 1; + nitems = 0; + for (i = 1; i <= numrows; i++) + { + /* last row or a new group */ + if ((i == numrows) || (multi_sort_compare(&items[i], &items[i-1], mss) != 0)) + { + /* count the MCV item if exceeding the threshold (and copy into the array) */ + if (count >= mcv_threshold) + { + /* just pointer to the proper place in the list */ + MCVItem item = mcvlist->items[nitems]; + + /* copy values from the _previous_ group (last item of) */ + memcpy(item->values, items[(i-1)].values, sizeof(Datum) * numattrs); + memcpy(item->isnull, items[(i-1)].isnull, sizeof(bool) * numattrs); + + + /* and finally the group frequency */ + item->frequency = (double)count / numrows; + + /* next item */ + nitems += 1; + } + + count = 1; + } + else /* same group, just increase the number of items */ + count += 1; + } + + /* make sure the loops are consistent */ + Assert(nitems == mcvlist->nitems); + + /* + * Remove the rows matching the MCV list (i.e. keep only rows + * that are not represented by the MCV list). + * + * FIXME This implementation is rather naive, effectively O(N^2). + * As the MCV list grows, the check will take longer and + * longer. And as the number of sampled rows increases (by + * increasing statistics target), it will take longer and + * longer. One option is to sort the MCV items first and + * then perform a binary search. + * + * A better option would be keeping the ID of the row in + * the sort item, and then just walk through the items and + * mark rows to remove (in a bitmap of the same size). + * There's not space for that in SortItem at this moment, + * but it's trivial to add 'private' pointer, or just + * using another structure with extra field (starting with + * SortItem, so that the comparators etc. still work). + * + * Another option is to use the sorted array of items + * (because that's how we sorted the source data), and + * simply do a bsearch() into it. If we find a matching + * item, the row belongs to the MCV list. + */ + if (nitems == ndistinct) /* all rows are covered by MCV items */ + *numrows_filtered = 0; + else /* (nitems < ndistinct) && (nitems > 0) */ + { + int nfiltered = 0; + HeapTuple *rows_filtered = (HeapTuple*)palloc0(sizeof(HeapTuple) * numrows); + + /* used for the searches */ + SortItem item, mcvitem;; + + item.values = (Datum*)palloc0(numattrs * sizeof(Datum)); + item.isnull = (bool*)palloc0(numattrs * sizeof(bool)); + + /* + * FIXME we don't need to allocate this, we can reference + * the MCV item directly ... + */ + mcvitem.values = (Datum*)palloc0(numattrs * sizeof(Datum)); + mcvitem.isnull = (bool*)palloc0(numattrs * sizeof(bool)); + + /* walk through the tuples, compare the values to MCV items */ + for (i = 0; i < numrows; i++) + { + bool match = false; + + /* collect the key values from the row */ + for (j = 0; j < numattrs; j++) + item.values[j] = heap_getattr(rows[i], attrs->values[j], + stats[j]->tupDesc, &item.isnull[j]); + + /* scan through the MCV list for matches */ + for (j = 0; j < mcvlist->nitems; j++) + { + /* + * TODO Create a SortItem/MCVItem comparator so that + * we don't need to do memcpy() like crazy. + */ + memcpy(mcvitem.values, mcvlist->items[j]->values, + numattrs * sizeof(Datum)); + memcpy(mcvitem.isnull, mcvlist->items[j]->isnull, + numattrs * sizeof(bool)); + + if (multi_sort_compare(&item, &mcvitem, mss) == 0) + { + match = true; + break; + } + } + + /* if no match in the MCV list, copy the row into the filtered ones */ + if (! match) + memcpy(&rows_filtered[nfiltered++], &rows[i], sizeof(HeapTuple)); + } + + /* replace the rows and remember how many rows we kept */ + memcpy(rows, rows_filtered, sizeof(HeapTuple) * nfiltered); + *numrows_filtered = nfiltered; + + /* free all the data used here */ + pfree(rows_filtered); + pfree(item.values); + pfree(item.isnull); + pfree(mcvitem.values); + pfree(mcvitem.isnull); + } + } + + pfree(values); + pfree(items); + pfree(isnull); + + return mcvlist; +} + + +/* fetch the MCV list (as a bytea) from the pg_mv_statistic catalog */ +MCVList +load_mv_mcvlist(Oid mvoid) +{ + bool isnull = false; + Datum mcvlist; + +#ifdef USE_ASSERT_CHECKING + Form_pg_mv_statistic mvstat; +#endif + + /* Prepare to scan pg_mv_statistic for entries having indrelid = this rel. */ + HeapTuple htup = SearchSysCache1(MVSTATOID, ObjectIdGetDatum(mvoid)); + + if (! HeapTupleIsValid(htup)) + return NULL; + +#ifdef USE_ASSERT_CHECKING + mvstat = (Form_pg_mv_statistic) GETSTRUCT(htup); + Assert(mvstat->mcv_enabled && mvstat->mcv_built); +#endif + + mcvlist = SysCacheGetAttr(MVSTATOID, htup, + Anum_pg_mv_statistic_stamcv, &isnull); + + Assert(!isnull); + + ReleaseSysCache(htup); + + return deserialize_mv_mcvlist(DatumGetByteaP(mcvlist)); +} + +/* print some basic info about the MCV list + * + * TODO Add info about what part of the table this covers. + */ +Datum +pg_mv_stats_mcvlist_info(PG_FUNCTION_ARGS) +{ + bytea *data = PG_GETARG_BYTEA_P(0); + char *result; + + MCVList mcvlist = deserialize_mv_mcvlist(data); + + result = palloc0(128); + snprintf(result, 128, "nitems=%d", mcvlist->nitems); + + pfree(mcvlist); + + PG_RETURN_TEXT_P(cstring_to_text(result)); +} + +/* used to pass context into bsearch() */ +static SortSupport ssup_private = NULL; + +static int bsearch_comparator(const void * a, const void * b); + +/* + * Serialize MCV list into a bytea value. The basic algorithm is simple: + * + * (1) perform deduplication for each attribute (separately) + * (a) collect all (non-NULL) attribute values from all MCV items + * (b) sort the data (using 'lt' from VacAttrStats) + * (c) remove duplicate values from the array + * + * (2) serialize the arrays into a bytea value + * + * (3) process all MCV list items + * (a) replace values with indexes into the arrays + * + * Each attribute has to be processed separately, because we're mixing + * different datatypes, and we don't know what equality means for them. + * We're also mixing pass-by-value and pass-by-ref types, and so on. + * + * We'll use uint16 values for the indexes in step (3), as we don't + * allow more than 8k MCV items (see list max_mcv_items). We might + * increase this to 65k and still fit into uint16. + * + * We don't really expect the high compression as with histograms, + * because we're not doing any bucket splits etc. (which is the source + * of high redundancy there), but we need to do it anyway as we need + * to serialize varlena values etc. We might invent another way to + * serialize MCV lists, but let's keep it consistent. + * + * FIXME This probably leaks memory, or at least uses it inefficiently + * (many small palloc() calls instead of a large one). + * + * TODO Consider using 16-bit values for the indexes in step (3). + * + * TODO Consider packing boolean flags (NULL) for each item into 'char' + * or a longer type (instead of using an array of bool items). + */ +bytea * +serialize_mv_mcvlist(MCVList mcvlist, int2vector *attrs, + VacAttrStats **stats) +{ + int i, j; + int ndims = mcvlist->ndimensions; + int itemsize = ITEM_SIZE(ndims); + + Size total_length = 0; + + char *item = palloc0(itemsize); + + /* serialized items (indexes into arrays, etc.) */ + bytea *output; + char *data = NULL; + + /* values per dimension (and number of non-NULL values) */ + Datum **values = (Datum**)palloc0(sizeof(Datum*) * ndims); + int *counts = (int*)palloc0(sizeof(int) * ndims); + + /* info about dimensions (for deserialize) */ + DimensionInfo * info + = (DimensionInfo *)palloc0(sizeof(DimensionInfo)*ndims); + + /* sort support data */ + SortSupport ssup = (SortSupport)palloc0(sizeof(SortSupportData)*ndims); + + /* collect and deduplicate values for each dimension */ + for (i = 0; i < ndims; i++) + { + int count; + StdAnalyzeData *tmp = (StdAnalyzeData *)stats[i]->extra_data; + + /* keep important info about the data type */ + info[i].typlen = stats[i]->attrtype->typlen; + info[i].typbyval = stats[i]->attrtype->typbyval; + + /* allocate space for all values, including NULLs (won't use them) */ + values[i] = (Datum*)palloc0(sizeof(Datum) * mcvlist->nitems); + + for (j = 0; j < mcvlist->nitems; j++) + { + if (! mcvlist->items[j]->isnull[i]) /* skip NULL values */ + { + values[i][counts[i]] = mcvlist->items[j]->values[i]; + counts[i] += 1; + } + } + + /* there are just NULL values in this dimension */ + if (counts[i] == 0) + continue; + + /* sort and deduplicate */ + ssup[i].ssup_cxt = CurrentMemoryContext; + ssup[i].ssup_collation = DEFAULT_COLLATION_OID; + ssup[i].ssup_nulls_first = false; + + PrepareSortSupportFromOrderingOp(tmp->ltopr, &ssup[i]); + + qsort_arg(values[i], counts[i], sizeof(Datum), + compare_scalars_simple, &ssup[i]); + + /* + * Walk through the array and eliminate duplicitate values, but + * keep the ordering (so that we can do bsearch later). We know + * there's at least 1 item, so we can skip the first element. + */ + count = 1; /* number of deduplicated items */ + for (j = 1; j < counts[i]; j++) + { + /* if it's different from the previous value, we need to keep it */ + if (compare_datums_simple(values[i][j-1], values[i][j], &ssup[i]) != 0) + { + /* XXX: not needed if (count == j) */ + values[i][count] = values[i][j]; + count += 1; + } + } + + /* do not exceed UINT16_MAX */ + Assert(count <= UINT16_MAX); + + /* keep info about the deduplicated count */ + info[i].nvalues = count; + + /* compute size of the serialized data */ + if (info[i].typbyval || (info[i].typlen > 0)) + /* by value pased by reference, but fixed length */ + info[i].nbytes = info[i].nvalues * info[i].typlen; + else if (info[i].typlen == -1) + /* varlena, so just use VARSIZE_ANY */ + for (j = 0; j < info[i].nvalues; j++) + info[i].nbytes += VARSIZE_ANY(values[i][j]); + else if (info[i].typlen == -2) + /* cstring, so simply strlen */ + for (j = 0; j < info[i].nvalues; j++) + info[i].nbytes += strlen(DatumGetPointer(values[i][j])); + else + elog(ERROR, "unknown data type typbyval=%d typlen=%d", + info[i].typbyval, info[i].typlen); + } + + /* + * Now we finally know how much space we'll need for the serialized + * MCV list, as it contains these fields: + * + * - length (4B) for varlena + * - magic (4B) + * - type (4B) + * - ndimensions (4B) + * - nitems (4B) + * - info (ndim * sizeof(DimensionInfo) + * - arrays of values for each dimension + * - serialized items (nitems * itemsize) + * + * So the 'header' size is 20B + ndim * sizeof(DimensionInfo) and + * then we'll place the data. + */ + total_length = (sizeof(int32) + offsetof(MCVListData, items) + + ndims * sizeof(DimensionInfo) + + mcvlist->nitems * itemsize); + + for (i = 0; i < ndims; i++) + total_length += info[i].nbytes; + + /* enforce arbitrary limit of 1MB */ + if (total_length > 1024 * 1024) + elog(ERROR, "serialized MCV exceeds 1MB (%ld)", total_length); + + /* allocate space for the serialized MCV list, set header fields */ + output = (bytea*)palloc0(total_length); + SET_VARSIZE(output, total_length); + + /* we'll use 'ptr' to keep track of the place to write data */ + data = VARDATA(output); + + memcpy(data, mcvlist, offsetof(MCVListData, items)); + data += offsetof(MCVListData, items); + + memcpy(data, info, sizeof(DimensionInfo) * ndims); + data += sizeof(DimensionInfo) * ndims; + + /* value array for each dimension */ + for (i = 0; i < ndims; i++) + { +#ifdef USE_ASSERT_CHECKING + char *tmp = data; +#endif + for (j = 0; j < info[i].nvalues; j++) + { + if (info[i].typbyval) + { + /* passed by value / Datum */ + memcpy(data, &values[i][j], info[i].typlen); + data += info[i].typlen; + } + else if (info[i].typlen > 0) + { + /* pased by reference, but fixed length (name, tid, ...) */ + memcpy(data, &values[i][j], info[i].typlen); + data += info[i].typlen; + } + else if (info[i].typlen == -1) + { + /* varlena */ + memcpy(data, DatumGetPointer(values[i][j]), + VARSIZE_ANY(values[i][j])); + data += VARSIZE_ANY(values[i][j]); + } + else if (info[i].typlen == -2) + { + /* cstring (don't forget the \0 terminator!) */ + memcpy(data, DatumGetPointer(values[i][j]), + strlen(DatumGetPointer(values[i][j])) + 1); + data += strlen(DatumGetPointer(values[i][j])) + 1; + } + } + Assert((data - tmp) == info[i].nbytes); + } + + /* and finally, the MCV items */ + for (i = 0; i < mcvlist->nitems; i++) + { + /* don't write beyond the allocated space */ + Assert(data <= (char*)output + total_length - itemsize); + + /* reset the values for each item */ + memset(item, 0, itemsize); + + for (j = 0; j < ndims; j++) + { + /* do the lookup only for non-NULL values */ + if (! mcvlist->items[i]->isnull[j]) + { + Datum * v = NULL; + ssup_private = &ssup[j]; + + v = (Datum*)bsearch(&mcvlist->items[i]->values[j], + values[j], info[j].nvalues, sizeof(Datum), + bsearch_comparator); + + if (v == NULL) + elog(ERROR, "value for dim %d not found in array", j); + + /* compute index within the array */ + ITEM_INDEXES(item)[j] = (v - values[j]); + + /* check the index is within expected bounds */ + Assert(ITEM_INDEXES(item)[j] >= 0); + Assert(ITEM_INDEXES(item)[j] < info[j].nvalues); + } + } + + /* copy NULL and frequency flags into the item */ + memcpy(ITEM_NULLS(item, ndims), + mcvlist->items[i]->isnull, sizeof(bool) * ndims); + memcpy(ITEM_FREQUENCY(item, ndims), + &mcvlist->items[i]->frequency, sizeof(double)); + + /* copy the item into the array */ + memcpy(data, item, itemsize); + + data += itemsize; + } + + /* at this point we expect to match the total_length exactly */ + Assert((data - (char*)output) == total_length); + + return output; +} + +/* + * Inverse to serialize_mv_mcvlist() - see the comment there. + * + * We'll do full deserialization, because we don't really expect high + * duplication of values so the caching may not be as efficient as with + * histograms. + */ +MCVList deserialize_mv_mcvlist(bytea * data) +{ + int i, j; + Size expected_size; + MCVList mcvlist; + char *tmp; + + int ndims, nitems, itemsize; + DimensionInfo *info = NULL; + + uint16 *indexes = NULL; + Datum **values = NULL; + + /* local allocation buffer (used only for deserialization) */ + int bufflen; + char *buff; + char *ptr; + + /* buffer used for the result */ + int rbufflen; + char *rbuff; + char *rptr; + + if (data == NULL) + return NULL; + + if (VARSIZE_ANY_EXHDR(data) < offsetof(MCVListData,items)) + elog(ERROR, "invalid MCV Size %ld (expected at least %ld)", + VARSIZE_ANY_EXHDR(data), offsetof(MCVListData,items)); + + /* read the MCV list header */ + mcvlist = (MCVList)palloc0(sizeof(MCVListData)); + + /* initialize pointer to the data part (skip the varlena header) */ + tmp = VARDATA(data); + + /* get the header and perform basic sanity checks */ + memcpy(mcvlist, tmp, offsetof(MCVListData,items)); + tmp += offsetof(MCVListData,items); + + if (mcvlist->magic != MVSTAT_MCV_MAGIC) + elog(ERROR, "invalid MCV magic %d (expected %dd)", + mcvlist->magic, MVSTAT_MCV_MAGIC); + + if (mcvlist->type != MVSTAT_MCV_TYPE_BASIC) + elog(ERROR, "invalid MCV type %d (expected %dd)", + mcvlist->type, MVSTAT_MCV_TYPE_BASIC); + + nitems = mcvlist->nitems; + ndims = mcvlist->ndimensions; + itemsize = ITEM_SIZE(ndims); + + Assert(nitems > 0); + Assert((ndims >= 2) && (ndims <= MVSTATS_MAX_DIMENSIONS)); + + /* + * What size do we expect with those parameters (it's incomplete, + * as we yet have to count the array sizes (from DimensionInfo + * records). + */ + expected_size = offsetof(MCVListData,items) + + ndims * sizeof(DimensionInfo) + + (nitems * itemsize); + + /* check that we have at least the DimensionInfo records */ + if (VARSIZE_ANY_EXHDR(data) < expected_size) + elog(ERROR, "invalid MCV Size %ld (expected %ld)", + VARSIZE_ANY_EXHDR(data), expected_size); + + info = (DimensionInfo*)(tmp); + tmp += ndims * sizeof(DimensionInfo); + + /* account for the value arrays */ + for (i = 0; i < ndims; i++) + expected_size += info[i].nbytes; + + if (VARSIZE_ANY_EXHDR(data) != expected_size) + elog(ERROR, "invalid MCV Size %ld (expected %ld)", + VARSIZE_ANY_EXHDR(data), expected_size); + + /* looks OK - not corrupted or something */ + + /* + * We'll allocate one large chunk of memory for the intermediate + * data, needed only for deserializing the MCV list, and we'll pack + * use a local dense allocation to minimize the palloc overhead. + * + * Let's see how much space we'll actually need, and also include + * space for the array with pointers. + */ + bufflen = sizeof(Datum*) * ndims; /* space for pointers */ + + for (i = 0; i < ndims; i++) + /* for full-size byval types, we reuse the serialized value */ + if (! (info[i].typbyval && info[i].typlen == sizeof(Datum))) + bufflen += (sizeof(Datum) * info[i].nvalues); + + buff = palloc(bufflen); + ptr = buff; + + values = (Datum**)buff; + ptr += (sizeof(Datum*) * ndims); + + /* + * FIXME This uses pointers to the original data array (the types + * not passed by value), so when someone frees the memory, + * e.g. by doing something like this: + * + * bytea * data = ... fetch the data from catalog ... + * MCVList mcvlist = deserialize_mcv_list(data); + * pfree(data); + * + * then 'mcvlist' references the freed memory. This needs to + * copy the pieces. + */ + for (i = 0; i < ndims; i++) + { + if (info[i].typbyval) + { + /* passed by value / Datum - simply reuse the array */ + if (info[i].typlen == sizeof(Datum)) + { + values[i] = (Datum*)tmp; + tmp += info[i].nbytes; + } + else + { + values[i] = (Datum*)ptr; + ptr += (sizeof(Datum) * info[i].nvalues); + + for (j = 0; j < info[i].nvalues; j++) + { + /* just point into the array */ + memcpy(&values[i][j], tmp, info[i].typlen); + tmp += info[i].typlen; + } + } + } + else + { + /* all the varlena data need a chunk from the buffer */ + values[i] = (Datum*)ptr; + ptr += (sizeof(Datum) * info[i].nvalues); + + /* pased by reference, but fixed length (name, tid, ...) */ + if (info[i].typlen > 0) + { + for (j = 0; j < info[i].nvalues; j++) + { + /* just point into the array */ + values[i][j] = PointerGetDatum(tmp); + tmp += info[i].typlen; + } + } + else if (info[i].typlen == -1) + { + /* varlena */ + for (j = 0; j < info[i].nvalues; j++) + { + /* just point into the array */ + values[i][j] = PointerGetDatum(tmp); + tmp += VARSIZE_ANY(tmp); + } + } + else if (info[i].typlen == -2) + { + /* cstring */ + for (j = 0; j < info[i].nvalues; j++) + { + /* just point into the array */ + values[i][j] = PointerGetDatum(tmp); + tmp += (strlen(tmp) + 1); /* don't forget the \0 */ + } + } + } + } + + /* we should exhaust the buffer exactly */ + Assert((ptr - buff) == bufflen); + + /* allocate space for the MCV items in a single piece */ + rbufflen = (sizeof(MCVItem) + sizeof(MCVItemData) + + sizeof(Datum)*ndims + sizeof(bool)*ndims) * nitems; + + rbuff = palloc(rbufflen); + rptr = rbuff; + + mcvlist->items = (MCVItem*)rbuff; + rptr += (sizeof(MCVItem) * nitems); + + for (i = 0; i < nitems; i++) + { + MCVItem item = (MCVItem)rptr; + rptr += (sizeof(MCVItemData)); + + item->values = (Datum*)rptr; + rptr += (sizeof(Datum)*ndims); + + item->isnull = (bool*)rptr; + rptr += (sizeof(bool) *ndims); + + /* just point to the right place */ + indexes = ITEM_INDEXES(tmp); + + memcpy(item->isnull, ITEM_NULLS(tmp, ndims), sizeof(bool) * ndims); + memcpy(&item->frequency, ITEM_FREQUENCY(tmp, ndims), sizeof(double)); + +#ifdef ASSERT_CHECKING + for (j = 0; j < ndims; j++) + Assert(indexes[j] <= UINT16_MAX); +#endif + + /* translate the values */ + for (j = 0; j < ndims; j++) + if (! item->isnull[j]) + item->values[j] = values[j][indexes[j]]; + + mcvlist->items[i] = item; + + tmp += ITEM_SIZE(ndims); + + Assert(tmp <= (char*)data + VARSIZE_ANY(data)); + } + + /* check that we processed all the data */ + Assert(tmp == (char*)data + VARSIZE_ANY(data)); + + /* release the temporary buffer */ + pfree(buff); + + return mcvlist; +} + +/* + * We need to pass the SortSupport to the comparator, but bsearch() + * has no 'context' parameter, so we use a global variable (ugly). + */ +static int +bsearch_comparator(const void * a, const void * b) +{ + Assert(ssup_private != NULL); + return compare_scalars_simple(a, b, (void*)ssup_private); +} +/* + * SRF with details about buckets of a histogram: + * + * - item ID (0...nitems) + * - values (string array) + * - nulls only (boolean array) + * - frequency (double precision) + * + * The input is the OID of the statistics, and there are no rows + * returned if the statistics contains no histogram. + */ +PG_FUNCTION_INFO_V1(pg_mv_mcv_items); + +Datum +pg_mv_mcv_items(PG_FUNCTION_ARGS) +{ + FuncCallContext *funcctx; + int call_cntr; + int max_calls; + TupleDesc tupdesc; + AttInMetadata *attinmeta; + + /* stuff done only on the first call of the function */ + if (SRF_IS_FIRSTCALL()) + { + MemoryContext oldcontext; + MCVList mcvlist; + + /* create a function context for cross-call persistence */ + funcctx = SRF_FIRSTCALL_INIT(); + + /* switch to memory context appropriate for multiple function calls */ + oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx); + + mcvlist = load_mv_mcvlist(PG_GETARG_OID(0)); + + funcctx->user_fctx = mcvlist; + + /* total number of tuples to be returned */ + funcctx->max_calls = 0; + if (funcctx->user_fctx != NULL) + funcctx->max_calls = mcvlist->nitems; + + /* Build a tuple descriptor for our result type */ + if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("function returning record called in context " + "that cannot accept type record"))); + + /* + * generate attribute metadata needed later to produce tuples + * from raw C strings + */ + attinmeta = TupleDescGetAttInMetadata(tupdesc); + funcctx->attinmeta = attinmeta; + + MemoryContextSwitchTo(oldcontext); + } + + /* stuff done on every call of the function */ + funcctx = SRF_PERCALL_SETUP(); + + call_cntr = funcctx->call_cntr; + max_calls = funcctx->max_calls; + attinmeta = funcctx->attinmeta; + + if (call_cntr < max_calls) /* do when there is more left to send */ + { + char **values; + HeapTuple tuple; + Datum result; + int2vector *stakeys; + Oid relid; + + char *buff = palloc0(1024); + char *format; + + int i; + + Oid *outfuncs; + FmgrInfo *fmgrinfo; + + MCVList mcvlist; + MCVItem item; + + mcvlist = (MCVList)funcctx->user_fctx; + + Assert(call_cntr < mcvlist->nitems); + + item = mcvlist->items[call_cntr]; + + stakeys = find_mv_attnums(PG_GETARG_OID(0), &relid); + + /* + * Prepare a values array for building the returned tuple. + * This should be an array of C strings which will + * be processed later by the type input functions. + */ + values = (char **) palloc(4 * sizeof(char *)); + + values[0] = (char *) palloc(64 * sizeof(char)); + + /* arrays */ + values[1] = (char *) palloc0(1024 * sizeof(char)); + values[2] = (char *) palloc0(1024 * sizeof(char)); + + /* frequency */ + values[3] = (char *) palloc(64 * sizeof(char)); + + outfuncs = (Oid*)palloc0(sizeof(Oid) * mcvlist->ndimensions); + fmgrinfo = (FmgrInfo*)palloc0(sizeof(FmgrInfo) * mcvlist->ndimensions); + + for (i = 0; i < mcvlist->ndimensions; i++) + { + bool isvarlena; + + getTypeOutputInfo(get_atttype(relid, stakeys->values[i]), + &outfuncs[i], &isvarlena); + + fmgr_info(outfuncs[i], &fmgrinfo[i]); + } + + snprintf(values[0], 64, "%d", call_cntr); /* item ID */ + + for (i = 0; i < mcvlist->ndimensions; i++) + { + Datum val, valout; + + format = "%s, %s"; + if (i == 0) + format = "{%s%s"; + else if (i == mcvlist->ndimensions-1) + format = "%s, %s}"; + + val = item->values[i]; + valout = FunctionCall1(&fmgrinfo[i], val); + + snprintf(buff, 1024, format, values[1], DatumGetPointer(valout)); + strncpy(values[1], buff, 1023); + buff[0] = '\0'; + + snprintf(buff, 1024, format, values[2], item->isnull[i] ? "t" : "f"); + strncpy(values[2], buff, 1023); + buff[0] = '\0'; + } + + snprintf(values[3], 64, "%f", item->frequency); /* frequency */ + + /* build a tuple */ + tuple = BuildTupleFromCStrings(attinmeta, values); + + /* make the tuple into a datum */ + result = HeapTupleGetDatum(tuple); + + /* clean up (this is not really necessary) */ + pfree(values[0]); + pfree(values[1]); + pfree(values[2]); + pfree(values[3]); + + pfree(values); + + SRF_RETURN_NEXT(funcctx, result); + } + else /* do when there is no more left */ + { + SRF_RETURN_DONE(funcctx); + } +} diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 5f89604..01d29db 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -2101,8 +2101,9 @@ describeOneTableDetails(const char *schemaname, { printfPQExpBuffer(&buf, "SELECT oid, stakeys,\n" - " deps_enabled,\n" - " deps_built,\n" + " deps_enabled, mcv_enabled,\n" + " deps_built, mcv_built,\n" + " mcv_max_items,\n" " (SELECT string_agg(attname::text,', ')\n" " FROM ((SELECT unnest(stakeys) AS attnum) s\n" " JOIN pg_attribute a ON (starelid = a.attrelid and a.attnum = s.attnum))) AS attnums\n" @@ -2120,14 +2121,28 @@ describeOneTableDetails(const char *schemaname, printTableAddFooter(&cont, _("Statistics:")); for (i = 0; i < tuples; i++) { + bool first = true; + printfPQExpBuffer(&buf, " "); /* options */ if (!strcmp(PQgetvalue(result, i, 2), "t")) - appendPQExpBuffer(&buf, "(dependencies)"); + { + appendPQExpBuffer(&buf, "(dependencies"); + first = false; + } + + if (!strcmp(PQgetvalue(result, i, 3), "t")) + { + if (! first) + appendPQExpBuffer(&buf, ", mcv"); + else + appendPQExpBuffer(&buf, "(mcv"); + first = false; + } - appendPQExpBuffer(&buf, " ON (%s)", - PQgetvalue(result, i, 6)); + appendPQExpBuffer(&buf, ") ON (%s)", + PQgetvalue(result, i, 8)); printTableAddFooter(&cont, buf.data); } diff --git a/src/include/catalog/pg_mv_statistic.h b/src/include/catalog/pg_mv_statistic.h index 81ec23b..c6e7d74 100644 --- a/src/include/catalog/pg_mv_statistic.h +++ b/src/include/catalog/pg_mv_statistic.h @@ -35,15 +35,21 @@ CATALOG(pg_mv_statistic,3381) /* statistics requested to build */ bool deps_enabled; /* analyze dependencies? */ + bool mcv_enabled; /* build MCV list? */ + + /* MCV size */ + int32 mcv_max_items; /* max MCV items */ /* statistics that are available (if requested) */ bool deps_built; /* dependencies were built */ + bool mcv_built; /* MCV list was built */ /* variable-length fields start here, but we allow direct access to stakeys */ int2vector stakeys; /* array of column keys */ #ifdef CATALOG_VARLEN bytea stadeps; /* dependencies (serialized) */ + bytea stamcv; /* MCV list (serialized) */ #endif } FormData_pg_mv_statistic; @@ -59,11 +65,15 @@ typedef FormData_pg_mv_statistic *Form_pg_mv_statistic; * compiler constants for pg_attrdef * ---------------- */ -#define Natts_pg_mv_statistic 5 +#define Natts_pg_mv_statistic 9 #define Anum_pg_mv_statistic_starelid 1 #define Anum_pg_mv_statistic_deps_enabled 2 -#define Anum_pg_mv_statistic_deps_built 3 -#define Anum_pg_mv_statistic_stakeys 4 -#define Anum_pg_mv_statistic_stadeps 5 +#define Anum_pg_mv_statistic_mcv_enabled 3 +#define Anum_pg_mv_statistic_mcv_max_items 4 +#define Anum_pg_mv_statistic_deps_built 5 +#define Anum_pg_mv_statistic_mcv_built 6 +#define Anum_pg_mv_statistic_stakeys 7 +#define Anum_pg_mv_statistic_stadeps 8 +#define Anum_pg_mv_statistic_stamcv 9 #endif /* PG_MV_STATISTIC_H */ diff --git a/src/include/catalog/pg_proc.h b/src/include/catalog/pg_proc.h index 69fc482..890c763 100644 --- a/src/include/catalog/pg_proc.h +++ b/src/include/catalog/pg_proc.h @@ -2739,6 +2739,10 @@ DATA(insert OID = 3307 ( pg_mv_stats_dependencies_info PGNSP PGUID 12 1 0 0 DESCR("multivariate stats: functional dependencies info"); DATA(insert OID = 3308 ( pg_mv_stats_dependencies_show PGNSP PGUID 12 1 0 0 0 f f f f t f i 1 0 25 "17" _null_ _null_ _null_ _null_ _null_ pg_mv_stats_dependencies_show _null_ _null_ _null_ )); DESCR("multivariate stats: functional dependencies show"); +DATA(insert OID = 3376 ( pg_mv_stats_mcvlist_info PGNSP PGUID 12 1 0 0 0 f f f f t f i 1 0 25 "17" _null_ _null_ _null_ _null_ _null_ pg_mv_stats_mcvlist_info _null_ _null_ _null_ )); +DESCR("multi-variate statistics: MCV list info"); +DATA(insert OID = 3373 ( pg_mv_mcv_items PGNSP PGUID 12 1 1000 0 0 f f f f t t i 1 0 2249 "26" "{26,23,1009,1000,701}" "{i,o,o,o,o}" "{oid,index,values,nulls,frequency}" _null_ _null_ pg_mv_mcv_items _null_ _null_ _null_ )); +DESCR("details about MCV list items"); DATA(insert OID = 1928 ( pg_stat_get_numscans PGNSP PGUID 12 1 0 0 0 f f f f t f s 1 0 20 "26" _null_ _null_ _null_ _null_ _null_ pg_stat_get_numscans _null_ _null_ _null_ )); DESCR("statistics: number of scans done for table/index"); diff --git a/src/include/nodes/relation.h b/src/include/nodes/relation.h index 10f7425..917ae8d 100644 --- a/src/include/nodes/relation.h +++ b/src/include/nodes/relation.h @@ -572,9 +572,11 @@ typedef struct MVStatisticInfo /* enabled statistics */ bool deps_enabled; /* functional dependencies enabled */ + bool mcv_enabled; /* MCV list enabled */ /* built/available statistics */ bool deps_built; /* functional dependencies built */ + bool mcv_built; /* MCV list built */ /* columns in the statistics (attnums) */ int2vector *stakeys; /* attnums of the columns covered */ diff --git a/src/include/utils/mvstats.h b/src/include/utils/mvstats.h index 02a7dda..b028192 100644 --- a/src/include/utils/mvstats.h +++ b/src/include/utils/mvstats.h @@ -50,30 +50,89 @@ typedef MVDependenciesData* MVDependencies; #define MVSTAT_DEPS_TYPE_BASIC 1 /* basic dependencies type */ /* + * Multivariate MCV (most-common value) lists + * + * A straight-forward extension of MCV items - i.e. a list (array) of + * combinations of attribute values, together with a frequency and + * null flags. + */ +typedef struct MCVItemData { + double frequency; /* frequency of this combination */ + bool *isnull; /* lags of NULL values (up to 32 columns) */ + Datum *values; /* variable-length (ndimensions) */ +} MCVItemData; + +typedef MCVItemData *MCVItem; + +/* multivariate MCV list - essentally an array of MCV items */ +typedef struct MCVListData { + uint32 magic; /* magic constant marker */ + uint32 type; /* type of MCV list (BASIC) */ + uint32 ndimensions; /* number of dimensions */ + uint32 nitems; /* number of MCV items in the array */ + MCVItem *items; /* array of MCV items */ +} MCVListData; + +typedef MCVListData *MCVList; + +/* used to flag stats serialized to bytea */ +#define MVSTAT_MCV_MAGIC 0xE1A651C2 /* marks serialized bytea */ +#define MVSTAT_MCV_TYPE_BASIC 1 /* basic MCV list type */ + +/* + * Limits used for mcv_max_items option, i.e. we're always guaranteed + * to have space for at least MVSTAT_MCVLIST_MIN_ITEMS, and we cannot + * have more than MVSTAT_MCVLIST_MAX_ITEMS items. + * + * This is just a boundary for the 'max' threshold - the actual list + * may of course contain less items than MVSTAT_MCVLIST_MIN_ITEMS. + */ +#define MVSTAT_MCVLIST_MIN_ITEMS 128 /* min items in MCV list */ +#define MVSTAT_MCVLIST_MAX_ITEMS 8192 /* max items in MCV list */ + +/* * TODO Maybe fetching the histogram/MCV list separately is inefficient? * Consider adding a single `fetch_stats` method, fetching all * stats specified using flags (or something like that). */ MVDependencies load_mv_dependencies(Oid mvoid); +MCVList load_mv_mcvlist(Oid mvoid); bytea * serialize_mv_dependencies(MVDependencies dependencies); +bytea * serialize_mv_mcvlist(MCVList mcvlist, int2vector *attrs, + VacAttrStats **stats); /* deserialization of stats (serialization is private to analyze) */ MVDependencies deserialize_mv_dependencies(bytea * data); +MCVList deserialize_mv_mcvlist(bytea * data); + +/* + * Returns index of the attribute number within the vector (i.e. a + * dimension within the stats). + */ +int mv_get_index(AttrNumber varattno, int2vector * stakeys); + +int2vector* find_mv_attnums(Oid mvoid, Oid *relid); /* FIXME this probably belongs somewhere else (not to operations stats) */ extern Datum pg_mv_stats_dependencies_info(PG_FUNCTION_ARGS); extern Datum pg_mv_stats_dependencies_show(PG_FUNCTION_ARGS); +extern Datum pg_mv_stats_mcvlist_info(PG_FUNCTION_ARGS); +extern Datum pg_mv_mcvlist_items(PG_FUNCTION_ARGS); MVDependencies -build_mv_dependencies(int numrows, HeapTuple *rows, - int2vector *attrs, - VacAttrStats **stats); +build_mv_dependencies(int numrows, HeapTuple *rows, int2vector *attrs, + VacAttrStats **stats); + +MCVList +build_mv_mcvlist(int numrows, HeapTuple *rows, int2vector *attrs, + VacAttrStats **stats, int *numrows_filtered); void build_mv_stats(Relation onerel, int numrows, HeapTuple *rows, - int natts, VacAttrStats **vacattrstats); + int natts, VacAttrStats **vacattrstats); -void update_mv_stats(Oid relid, MVDependencies dependencies, int2vector *attrs); +void update_mv_stats(Oid relid, MVDependencies dependencies, MCVList mcvlist, + int2vector *attrs, VacAttrStats **stats); #endif diff --git a/src/test/regress/expected/mv_mcv.out b/src/test/regress/expected/mv_mcv.out new file mode 100644 index 0000000..85e8499 --- /dev/null +++ b/src/test/regress/expected/mv_mcv.out @@ -0,0 +1,207 @@ +-- data type passed by value +CREATE TABLE mcv_list ( + a INT, + b INT, + c INT +); +-- unknown column +ALTER TABLE mcv_list ADD STATISTICS (mcv) ON (unknown_column); +ERROR: column "unknown_column" referenced in statistics does not exist +-- single column +ALTER TABLE mcv_list ADD STATISTICS (mcv) ON (a); +ERROR: multivariate stats require 2 or more columns +-- single column, duplicated +ALTER TABLE mcv_list ADD STATISTICS (mcv) ON (a, a); +ERROR: duplicate column name in statistics definition +-- two columns, one duplicated +ALTER TABLE mcv_list ADD STATISTICS (mcv) ON (a, a, b); +ERROR: duplicate column name in statistics definition +-- unknown option +ALTER TABLE mcv_list ADD STATISTICS (unknown_option) ON (a, b, c); +ERROR: unrecognized STATISTICS option "unknown_option" +-- missing MCV statistics +ALTER TABLE mcv_list ADD STATISTICS (dependencies, max_mcv_items 200) ON (a, b, c); +ERROR: option 'mcv' is required by other options(s) +-- invalid mcv_max_items value / too low +ALTER TABLE mcv_list ADD STATISTICS (mcv, max_mcv_items 10) ON (a, b, c); +ERROR: max number of MCV items must be at least 128 +-- invalid mcv_max_items value / too high +ALTER TABLE mcv_list ADD STATISTICS (mcv, max_mcv_items 10000) ON (a, b, c); +ERROR: max number of MCV items is 8192 +-- correct command +ALTER TABLE mcv_list ADD STATISTICS (mcv) ON (a, b, c); +-- random data +INSERT INTO mcv_list + SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i); +ANALYZE mcv_list; +SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv) + FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass; + mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info +-------------+-----------+-------------------------- + t | f | +(1 row) + +TRUNCATE mcv_list; +-- a => b, a => c, b => c +INSERT INTO mcv_list + SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i); +ANALYZE mcv_list; +SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv) + FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass; + mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info +-------------+-----------+-------------------------- + t | t | nitems=1000 +(1 row) + +TRUNCATE mcv_list; +-- a => b, a => c +INSERT INTO mcv_list + SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i); +ANALYZE mcv_list; +SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv) + FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass; + mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info +-------------+-----------+-------------------------- + t | t | nitems=1000 +(1 row) + +TRUNCATE mcv_list; +-- check explain (expect bitmap index scan, not plain index scan) +INSERT INTO mcv_list + SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i); +CREATE INDEX mcv_idx ON mcv_list (a, b); +ANALYZE mcv_list; +SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv) + FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass; + mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info +-------------+-----------+-------------------------- + t | t | nitems=100 +(1 row) + +EXPLAIN (COSTS off) + SELECT * FROM mcv_list WHERE a = 10 AND b = 5; + QUERY PLAN +-------------------------------------------- + Bitmap Heap Scan on mcv_list + Recheck Cond: ((a = 10) AND (b = 5)) + -> Bitmap Index Scan on mcv_idx + Index Cond: ((a = 10) AND (b = 5)) +(4 rows) + +DROP TABLE mcv_list; +-- varlena type (text) +CREATE TABLE mcv_list ( + a TEXT, + b TEXT, + c TEXT +); +ALTER TABLE mcv_list ADD STATISTICS (mcv) ON (a, b, c); +-- random data +INSERT INTO mcv_list + SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i); +ANALYZE mcv_list; +SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv) + FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass; + mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info +-------------+-----------+-------------------------- + t | f | +(1 row) + +TRUNCATE mcv_list; +-- a => b, a => c, b => c +INSERT INTO mcv_list + SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i); +ANALYZE mcv_list; +SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv) + FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass; + mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info +-------------+-----------+-------------------------- + t | t | nitems=1000 +(1 row) + +TRUNCATE mcv_list; +-- a => b, a => c +INSERT INTO mcv_list + SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i); +ANALYZE mcv_list; +SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv) + FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass; + mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info +-------------+-----------+-------------------------- + t | t | nitems=1000 +(1 row) + +TRUNCATE mcv_list; +-- check explain (expect bitmap index scan, not plain index scan) +INSERT INTO mcv_list + SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i); +CREATE INDEX mcv_idx ON mcv_list (a, b); +ANALYZE mcv_list; +SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv) + FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass; + mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info +-------------+-----------+-------------------------- + t | t | nitems=100 +(1 row) + +EXPLAIN (COSTS off) + SELECT * FROM mcv_list WHERE a = '10' AND b = '5'; + QUERY PLAN +------------------------------------------------------------ + Bitmap Heap Scan on mcv_list + Recheck Cond: ((a = '10'::text) AND (b = '5'::text)) + -> Bitmap Index Scan on mcv_idx + Index Cond: ((a = '10'::text) AND (b = '5'::text)) +(4 rows) + +TRUNCATE mcv_list; +-- check explain (expect bitmap index scan, not plain index scan) with NULLs +INSERT INTO mcv_list + SELECT + (CASE WHEN i/10000 = 0 THEN NULL ELSE i/10000 END), + (CASE WHEN i/20000 = 0 THEN NULL ELSE i/20000 END), + (CASE WHEN i/40000 = 0 THEN NULL ELSE i/40000 END) + FROM generate_series(1,1000000) s(i); +ANALYZE mcv_list; +SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv) + FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass; + mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info +-------------+-----------+-------------------------- + t | t | nitems=100 +(1 row) + +EXPLAIN (COSTS off) + SELECT * FROM mcv_list WHERE a IS NULL AND b IS NULL; + QUERY PLAN +--------------------------------------------------- + Bitmap Heap Scan on mcv_list + Recheck Cond: ((a IS NULL) AND (b IS NULL)) + -> Bitmap Index Scan on mcv_idx + Index Cond: ((a IS NULL) AND (b IS NULL)) +(4 rows) + +DROP TABLE mcv_list; +-- NULL values (mix of int and text columns) +CREATE TABLE mcv_list ( + a INT, + b TEXT, + c INT, + d TEXT +); +ALTER TABLE mcv_list ADD STATISTICS (mcv) ON (a, b, c, d); +INSERT INTO mcv_list + SELECT + mod(i, 100), + (CASE WHEN mod(i, 200) = 0 THEN NULL ELSE mod(i,200) END), + mod(i, 400), + (CASE WHEN mod(i, 300) = 0 THEN NULL ELSE mod(i,600) END) + FROM generate_series(1,10000) s(i); +ANALYZE mcv_list; +SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv) + FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass; + mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info +-------------+-----------+-------------------------- + t | t | nitems=1200 +(1 row) + +DROP TABLE mcv_list; diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out index a12ad30..faa41c7 100644 --- a/src/test/regress/expected/rules.out +++ b/src/test/regress/expected/rules.out @@ -1367,7 +1367,9 @@ pg_mv_stats| SELECT n.nspname AS schemaname, c.relname AS tablename, s.stakeys AS attnums, length(s.stadeps) AS depsbytes, - pg_mv_stats_dependencies_info(s.stadeps) AS depsinfo + pg_mv_stats_dependencies_info(s.stadeps) AS depsinfo, + length(s.stamcv) AS mcvbytes, + pg_mv_stats_mcvlist_info(s.stamcv) AS mcvinfo FROM ((pg_mv_statistic s JOIN pg_class c ON ((c.oid = s.starelid))) LEFT JOIN pg_namespace n ON ((n.oid = c.relnamespace))); diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 11d9d38..d083442 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -112,4 +112,4 @@ test: event_trigger test: stats # run tests of multivariate stats -test: mv_dependencies +test: mv_dependencies mv_mcv diff --git a/src/test/regress/regression.diffs b/src/test/regress/regression.diffs deleted file mode 100644 index 95b9cc5..0000000 --- a/src/test/regress/regression.diffs +++ /dev/null @@ -1,30 +0,0 @@ -*** /home/user/work/tvondra_postgres/src/test/regress/expected/rolenames.out Wed May 6 21:31:06 2015 ---- /home/user/work/tvondra_postgres/src/test/regress/results/rolenames.out Mon May 25 22:24:21 2015 -*************** -*** 38,47 **** ---- 38,52 ---- - ORDER BY 2; - $$ LANGUAGE SQL; - CREATE ROLE "Public"; -+ ERROR: role "Public" already exists - CREATE ROLE "None"; -+ ERROR: role "None" already exists - CREATE ROLE "current_user"; -+ ERROR: role "current_user" already exists - CREATE ROLE "session_user"; -+ ERROR: role "session_user" already exists - CREATE ROLE "user"; -+ ERROR: role "user" already exists - CREATE ROLE current_user; -- error - ERROR: CURRENT_USER cannot be used as a role name here - LINE 1: CREATE ROLE current_user; -*************** -*** 938,940 **** ---- 943,946 ---- - DROP OWNED BY testrol0, "Public", "current_user", testrol1, testrol2, testrolx CASCADE; - DROP ROLE testrol0, testrol1, testrol2, testrolx; - DROP ROLE "Public", "None", "current_user", "session_user", "user"; -+ ERROR: current user cannot be dropped - -====================================================================== - diff --git a/src/test/regress/regression.out b/src/test/regress/regression.out deleted file mode 100644 index bd81385..0000000 --- a/src/test/regress/regression.out +++ /dev/null @@ -1,156 +0,0 @@ -test tablespace ... ok -test boolean ... ok -test char ... ok -test name ... ok -test varchar ... ok -test text ... ok -test int2 ... ok -test int4 ... ok -test int8 ... ok -test oid ... ok -test float4 ... ok -test float8 ... ok -test bit ... ok -test numeric ... ok -test txid ... ok -test uuid ... ok -test enum ... ok -test money ... ok -test rangetypes ... ok -test pg_lsn ... ok -test regproc ... ok -test strings ... ok -test numerology ... ok -test point ... ok -test lseg ... ok -test line ... ok -test box ... ok -test path ... ok -test polygon ... ok -test circle ... ok -test date ... ok -test time ... ok -test timetz ... ok -test timestamp ... ok -test timestamptz ... ok -test interval ... ok -test abstime ... ok -test reltime ... ok -test tinterval ... ok -test inet ... ok -test macaddr ... ok -test tstypes ... ok -test comments ... ok -test geometry ... ok -test horology ... ok -test regex ... ok -test oidjoins ... ok -test type_sanity ... ok -test opr_sanity ... ok -test insert ... ok -test insert_conflict ... ok -test create_function_1 ... ok -test create_type ... ok -test create_table ... ok -test create_function_2 ... ok -test copy ... ok -test copyselect ... ok -test create_misc ... ok -test create_operator ... ok -test create_index ... ok -test create_view ... ok -test create_aggregate ... ok -test create_function_3 ... ok -test create_cast ... ok -test constraints ... ok -test triggers ... ok -test inherit ... ok -test create_table_like ... ok -test typed_table ... ok -test vacuum ... ok -test drop_if_exists ... ok -test updatable_views ... ok -test rolenames ... FAILED -test sanity_check ... ok -test errors ... ok -test select ... ok -test select_into ... ok -test select_distinct ... ok -test select_distinct_on ... ok -test select_implicit ... ok -test select_having ... ok -test subselect ... ok -test union ... ok -test case ... ok -test join ... ok -test aggregates ... ok -test groupingsets ... ok -test transactions ... ok -test random ... ok -test portals ... ok -test arrays ... ok -test btree_index ... ok -test hash_index ... ok -test update ... ok -test delete ... ok -test namespace ... ok -test prepared_xacts ... ok -test brin ... ok -test gin ... ok -test gist ... ok -test spgist ... ok -test privileges ... ok -test security_label ... ok -test collate ... ok -test matview ... ok -test lock ... ok -test replica_identity ... ok -test rowsecurity ... ok -test object_address ... ok -test alter_generic ... ok -test misc ... ok -test psql ... ok -test async ... ok -test rules ... ok -test select_views ... ok -test portals_p2 ... ok -test foreign_key ... ok -test cluster ... ok -test dependency ... ok -test guc ... ok -test bitmapops ... ok -test combocid ... ok -test tsearch ... ok -test tsdicts ... ok -test foreign_data ... ok -test window ... ok -test xmlmap ... ok -test functional_deps ... ok -test advisory_lock ... ok -test json ... ok -test jsonb ... ok -test indirect_toast ... ok -test equivclass ... ok -test plancache ... ok -test limit ... ok -test plpgsql ... ok -test copy2 ... ok -test temp ... ok -test domain ... ok -test rangefuncs ... ok -test prepare ... ok -test without_oid ... ok -test conversion ... ok -test truncate ... ok -test alter_table ... ok -test sequence ... ok -test polymorphism ... ok -test rowtypes ... ok -test returning ... ok -test largeobject ... ok -test with ... ok -test xml ... ok -test event_trigger ... ok -test stats ... ok -test tablesample ... ok -test mv_dependencies ... ok diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 66925b3..e63b7aa 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -157,3 +157,4 @@ test: event_trigger test: stats test: tablesample test: mv_dependencies +test: mv_mcv diff --git a/src/test/regress/sql/mv_mcv.sql b/src/test/regress/sql/mv_mcv.sql new file mode 100644 index 0000000..5de3d29 --- /dev/null +++ b/src/test/regress/sql/mv_mcv.sql @@ -0,0 +1,178 @@ +-- data type passed by value +CREATE TABLE mcv_list ( + a INT, + b INT, + c INT +); + +-- unknown column +ALTER TABLE mcv_list ADD STATISTICS (mcv) ON (unknown_column); + +-- single column +ALTER TABLE mcv_list ADD STATISTICS (mcv) ON (a); + +-- single column, duplicated +ALTER TABLE mcv_list ADD STATISTICS (mcv) ON (a, a); + +-- two columns, one duplicated +ALTER TABLE mcv_list ADD STATISTICS (mcv) ON (a, a, b); + +-- unknown option +ALTER TABLE mcv_list ADD STATISTICS (unknown_option) ON (a, b, c); + +-- missing MCV statistics +ALTER TABLE mcv_list ADD STATISTICS (dependencies, max_mcv_items 200) ON (a, b, c); + +-- invalid mcv_max_items value / too low +ALTER TABLE mcv_list ADD STATISTICS (mcv, max_mcv_items 10) ON (a, b, c); + +-- invalid mcv_max_items value / too high +ALTER TABLE mcv_list ADD STATISTICS (mcv, max_mcv_items 10000) ON (a, b, c); + +-- correct command +ALTER TABLE mcv_list ADD STATISTICS (mcv) ON (a, b, c); + +-- random data +INSERT INTO mcv_list + SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i); + +ANALYZE mcv_list; + +SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv) + FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass; + +TRUNCATE mcv_list; + +-- a => b, a => c, b => c +INSERT INTO mcv_list + SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i); + +ANALYZE mcv_list; + +SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv) + FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass; + +TRUNCATE mcv_list; + +-- a => b, a => c +INSERT INTO mcv_list + SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i); + +ANALYZE mcv_list; + +SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv) + FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass; + +TRUNCATE mcv_list; + +-- check explain (expect bitmap index scan, not plain index scan) +INSERT INTO mcv_list + SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i); +CREATE INDEX mcv_idx ON mcv_list (a, b); + +ANALYZE mcv_list; + +SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv) + FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass; + +EXPLAIN (COSTS off) + SELECT * FROM mcv_list WHERE a = 10 AND b = 5; + +DROP TABLE mcv_list; + +-- varlena type (text) +CREATE TABLE mcv_list ( + a TEXT, + b TEXT, + c TEXT +); + +ALTER TABLE mcv_list ADD STATISTICS (mcv) ON (a, b, c); + +-- random data +INSERT INTO mcv_list + SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i); + +ANALYZE mcv_list; + +SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv) + FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass; + +TRUNCATE mcv_list; + +-- a => b, a => c, b => c +INSERT INTO mcv_list + SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i); + +ANALYZE mcv_list; + +SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv) + FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass; + +TRUNCATE mcv_list; + +-- a => b, a => c +INSERT INTO mcv_list + SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i); +ANALYZE mcv_list; + +SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv) + FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass; + +TRUNCATE mcv_list; + +-- check explain (expect bitmap index scan, not plain index scan) +INSERT INTO mcv_list + SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i); +CREATE INDEX mcv_idx ON mcv_list (a, b); +ANALYZE mcv_list; + +SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv) + FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass; + +EXPLAIN (COSTS off) + SELECT * FROM mcv_list WHERE a = '10' AND b = '5'; + +TRUNCATE mcv_list; + +-- check explain (expect bitmap index scan, not plain index scan) with NULLs +INSERT INTO mcv_list + SELECT + (CASE WHEN i/10000 = 0 THEN NULL ELSE i/10000 END), + (CASE WHEN i/20000 = 0 THEN NULL ELSE i/20000 END), + (CASE WHEN i/40000 = 0 THEN NULL ELSE i/40000 END) + FROM generate_series(1,1000000) s(i); +ANALYZE mcv_list; + +SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv) + FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass; + +EXPLAIN (COSTS off) + SELECT * FROM mcv_list WHERE a IS NULL AND b IS NULL; + +DROP TABLE mcv_list; + +-- NULL values (mix of int and text columns) +CREATE TABLE mcv_list ( + a INT, + b TEXT, + c INT, + d TEXT +); + +ALTER TABLE mcv_list ADD STATISTICS (mcv) ON (a, b, c, d); + +INSERT INTO mcv_list + SELECT + mod(i, 100), + (CASE WHEN mod(i, 200) = 0 THEN NULL ELSE mod(i,200) END), + mod(i, 400), + (CASE WHEN mod(i, 300) = 0 THEN NULL ELSE mod(i,600) END) + FROM generate_series(1,10000) s(i); + +ANALYZE mcv_list; + +SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv) + FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass; + +DROP TABLE mcv_list; -- 1.9.3 --------------080401000304040708010509 Content-Type: text/x-patch; name="0005-multivariate-histograms-v7.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0005-multivariate-histograms-v7.patch" From 89db32a7015e92bb5642604b822e9d3a41db2701 Mon Sep 17 00:00:00 2001 From: Tomas Vondra Date: Sun, 11 Jan 2015 20:18:24 +0100 Subject: [PATCH 5/6] multivariate histograms - extends the pg_mv_statistic catalog (add 'hist' fields) - building the histograms during ANALYZE - simple estimation while planning the queries Includes regression tests mostly equal to those for functional dependencies / MCV lists. --- src/backend/catalog/system_views.sql | 4 +- src/backend/commands/tablecmds.c | 86 +- src/backend/nodes/outfuncs.c | 2 + src/backend/optimizer/path/clausesel.c | 713 ++++++++- src/backend/optimizer/util/plancat.c | 4 +- src/backend/utils/mvstats/Makefile | 2 +- src/backend/utils/mvstats/common.c | 37 +- src/backend/utils/mvstats/histogram.c | 2188 ++++++++++++++++++++++++++++ src/bin/psql/describe.c | 17 +- src/include/catalog/pg_mv_statistic.h | 24 +- src/include/catalog/pg_proc.h | 4 + src/include/nodes/relation.h | 2 + src/include/utils/mvstats.h | 131 +- src/test/regress/expected/mv_histogram.out | 207 +++ src/test/regress/expected/rules.out | 4 +- src/test/regress/parallel_schedule | 2 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/mv_histogram.sql | 176 +++ 18 files changed, 3566 insertions(+), 38 deletions(-) create mode 100644 src/backend/utils/mvstats/histogram.c create mode 100644 src/test/regress/expected/mv_histogram.out create mode 100644 src/test/regress/sql/mv_histogram.sql diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql index 3144a29..0a1c25b 100644 --- a/src/backend/catalog/system_views.sql +++ b/src/backend/catalog/system_views.sql @@ -158,7 +158,9 @@ CREATE VIEW pg_mv_stats AS length(S.stadeps) as depsbytes, pg_mv_stats_dependencies_info(S.stadeps) as depsinfo, length(S.stamcv) AS mcvbytes, - pg_mv_stats_mcvlist_info(S.stamcv) AS mcvinfo + pg_mv_stats_mcvlist_info(S.stamcv) AS mcvinfo, + length(S.stahist) AS histbytes, + pg_mv_stats_histogram_info(S.stahist) AS histinfo FROM (pg_mv_statistic S JOIN pg_class C ON (C.oid = S.starelid)) LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace); diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 0d72aec..4c2da51 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -11919,12 +11919,15 @@ static void ATExecAddStatistics(AlteredTableInfo *tab, Relation rel, /* by default build nothing */ bool build_dependencies = false, - build_mcv = false; + build_mcv = false, + build_histogram = false; - int32 max_mcv_items = -1; + int32 max_buckets = -1, + max_mcv_items = -1; /* options required because of other options */ - bool require_mcv = false; + bool require_mcv = false, + require_histogram = false; Assert(IsA(def, StatisticsDef)); @@ -12002,6 +12005,29 @@ static void ATExecAddStatistics(AlteredTableInfo *tab, Relation rel, MVSTAT_MCVLIST_MAX_ITEMS))); } + else if (strcmp(opt->defname, "histogram") == 0) + build_histogram = defGetBoolean(opt); + else if (strcmp(opt->defname, "max_buckets") == 0) + { + max_buckets = defGetInt32(opt); + + /* this option requires 'histogram' to be enabled */ + require_histogram = true; + + /* sanity check */ + if (max_buckets < MVSTAT_HIST_MIN_BUCKETS) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("minimum number of buckets is %d", + MVSTAT_HIST_MIN_BUCKETS))); + + else if (max_buckets > MVSTAT_HIST_MAX_BUCKETS) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("minimum number of buckets is %d", + MVSTAT_HIST_MAX_BUCKETS))); + + } else ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), @@ -12010,10 +12036,10 @@ static void ATExecAddStatistics(AlteredTableInfo *tab, Relation rel, } /* check that at least some statistics were requested */ - if (! (build_dependencies || build_mcv)) + if (! (build_dependencies || build_mcv || build_histogram)) ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), - errmsg("no statistics type (dependencies, mcv) was requested"))); + errmsg("no statistics type (dependencies, mcv, histogram) was requested"))); /* now do some checking of the options */ if (require_mcv && (! build_mcv)) @@ -12021,6 +12047,11 @@ static void ATExecAddStatistics(AlteredTableInfo *tab, Relation rel, (errcode(ERRCODE_SYNTAX_ERROR), errmsg("option 'mcv' is required by other options(s)"))); + if (require_histogram && (! build_histogram)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("option 'histogram' is required by other options(s)"))); + /* sort the attnums and build int2vector */ qsort(attnums, numcols, sizeof(int16), compare_int16); stakeys = buildint2vector(attnums, numcols); @@ -12038,10 +12069,14 @@ static void ATExecAddStatistics(AlteredTableInfo *tab, Relation rel, values[Anum_pg_mv_statistic_deps_enabled -1] = BoolGetDatum(build_dependencies); values[Anum_pg_mv_statistic_mcv_enabled -1] = BoolGetDatum(build_mcv); + values[Anum_pg_mv_statistic_hist_enabled -1] = BoolGetDatum(build_histogram); + values[Anum_pg_mv_statistic_mcv_max_items -1] = Int32GetDatum(max_mcv_items); + values[Anum_pg_mv_statistic_hist_max_buckets -1] = Int32GetDatum(max_buckets); nulls[Anum_pg_mv_statistic_stadeps -1] = true; nulls[Anum_pg_mv_statistic_stamcv -1] = true; + nulls[Anum_pg_mv_statistic_stahist -1] = true; /* insert the tuple into pg_mv_statistic */ mvstatrel = heap_open(MvStatisticRelationId, RowExclusiveLock); @@ -12064,6 +12099,7 @@ static void ATExecAddStatistics(AlteredTableInfo *tab, Relation rel, return; } + /* * Implements the ALTER TABLE ... DROP STATISTICS in two forms: * @@ -12089,12 +12125,16 @@ static void ATExecDropStatistics(AlteredTableInfo *tab, Relation rel, /* checking whether the statistics matches / should be dropped */ bool build_dependencies = false; bool build_mcv = false; + bool build_histogram = false; bool max_mcv_items = 0; + bool max_buckets = 0; bool check_dependencies = false; bool check_mcv = false; bool check_mcv_items = false; + bool check_histogram = false; + bool check_buckets = false; if (def != NULL) { @@ -12148,6 +12188,18 @@ static void ATExecDropStatistics(AlteredTableInfo *tab, Relation rel, build_mcv = true; max_mcv_items = defGetInt32(opt); } + else if (strcmp(opt->defname, "histogram") == 0) + { + check_histogram = true; + build_histogram = defGetBoolean(opt); + } + else if (strcmp(opt->defname, "max_buckets") == 0) + { + check_histogram = true; + check_buckets = true; + max_buckets = defGetInt32(opt); + build_histogram = true; + } else ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), @@ -12211,6 +12263,30 @@ static void ATExecDropStatistics(AlteredTableInfo *tab, Relation rel, (DatumGetInt32(adatum) == max_mcv_items); } + if (delete && check_histogram) + { + bool isnull; + Datum adatum = heap_getattr(tuple, + Anum_pg_mv_statistic_hist_enabled, + RelationGetDescr(statrel), + &isnull); + + delete = (! isnull) && + (DatumGetBool(adatum) == build_histogram); + } + + if (delete && check_buckets) + { + bool isnull; + Datum adatum = heap_getattr(tuple, + Anum_pg_mv_statistic_hist_max_buckets, + RelationGetDescr(statrel), + &isnull); + + delete = (! isnull) && + (DatumGetInt32(adatum) == max_buckets); + } + /* check that the columns match the statistics definition */ if (delete && (numcols > 0)) { diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 1867ab7..19d672f 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -1908,10 +1908,12 @@ _outMVStatisticInfo(StringInfo str, const MVStatisticInfo *node) /* enabled statistics */ WRITE_BOOL_FIELD(deps_enabled); WRITE_BOOL_FIELD(mcv_enabled); + WRITE_BOOL_FIELD(hist_enabled); /* built/available statistics */ WRITE_BOOL_FIELD(deps_built); WRITE_BOOL_FIELD(mcv_built); + WRITE_BOOL_FIELD(hist_built); } static void diff --git a/src/backend/optimizer/path/clausesel.c b/src/backend/optimizer/path/clausesel.c index 95872de..bc02e92 100644 --- a/src/backend/optimizer/path/clausesel.c +++ b/src/backend/optimizer/path/clausesel.c @@ -49,6 +49,7 @@ static void addRangeClause(RangeQueryClause **rqlist, Node *clause, #define MV_CLAUSE_TYPE_FDEP 0x01 #define MV_CLAUSE_TYPE_MCV 0x02 +#define MV_CLAUSE_TYPE_HIST 0x04 static bool clause_is_mv_compatible(PlannerInfo *root, Node *clause, Oid varRelid, Index *relid, Bitmapset **attnums, SpecialJoinInfo *sjinfo, @@ -73,6 +74,8 @@ static Selectivity clauselist_mv_selectivity(PlannerInfo *root, static Selectivity clauselist_mv_selectivity_mcvlist(PlannerInfo *root, List *clauses, MVStatisticInfo *mvstats, bool *fullmatch, Selectivity *lowsel); +static Selectivity clauselist_mv_selectivity_histogram(PlannerInfo *root, + List *clauses, MVStatisticInfo *mvstats); static int update_match_bitmap_mcvlist(PlannerInfo *root, List *clauses, int2vector *stakeys, MCVList mcvlist, @@ -80,6 +83,12 @@ static int update_match_bitmap_mcvlist(PlannerInfo *root, List *clauses, Selectivity *lowsel, bool *fullmatch, bool is_or); +static int update_match_bitmap_histogram(PlannerInfo *root, List *clauses, + int2vector *stakeys, + MVSerializedHistogram mvhist, + int nmatches, char * matches, + bool is_or); + static bool has_stats(List *stats, int type); static List * find_stats(PlannerInfo *root, List *clauses, @@ -304,7 +313,7 @@ clauselist_selectivity(PlannerInfo *root, * Check that there are statistics with MCV list. If not, we don't * need to waste time with the optimization. */ - if (has_stats(stats, MV_CLAUSE_TYPE_MCV)) + if (has_stats(stats, MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST)) { /* * Recollect attributes from mv-compatible clauses (maybe we've @@ -312,7 +321,7 @@ clauselist_selectivity(PlannerInfo *root, * From now on we're only interested in MCV-compatible clauses. */ mvattnums = collect_mv_attnums(root, clauses, varRelid, &relid, sjinfo, - MV_CLAUSE_TYPE_MCV); + (MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST)); /* * If there still are at least two columns, we'll try to select @@ -331,7 +340,7 @@ clauselist_selectivity(PlannerInfo *root, /* split the clauselist into regular and mv-clauses */ clauses = clauselist_mv_split(root, sjinfo, clauses, varRelid, &mvclauses, mvstat, - MV_CLAUSE_TYPE_MCV); + (MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST)); /* we've chosen the histogram to match the clauses */ Assert(mvclauses != NIL); @@ -1116,6 +1125,7 @@ static Selectivity clauselist_mv_selectivity(PlannerInfo *root, List *clauses, MVStatisticInfo *mvstats) { bool fullmatch = false; + Selectivity s1 = 0.0, s2 = 0.0; /* * Lowest frequency in the MCV list (may be used as an upper bound @@ -1129,9 +1139,24 @@ clauselist_mv_selectivity(PlannerInfo *root, List *clauses, MVStatisticInfo *mvs * MCV/histogram evaluation). */ - /* Evaluate the MCV selectivity */ - return clauselist_mv_selectivity_mcvlist(root, clauses, mvstats, + /* Evaluate the MCV first. */ + s1 = clauselist_mv_selectivity_mcvlist(root, clauses, mvstats, &fullmatch, &mcv_low); + + /* + * If we got a full equality match on the MCV list, we're done (and + * the estimate is pretty good). + */ + if (fullmatch && (s1 > 0.0)) + return s1; + + /* FIXME if (fullmatch) without matching MCV item, use the mcv_low + * selectivity as upper bound */ + + s2 = clauselist_mv_selectivity_histogram(root, clauses, mvstats); + + /* TODO clamp to <= 1.0 (or more strictly, when possible) */ + return s1 + s2; } /* @@ -1273,7 +1298,7 @@ choose_mv_statistics(List *stats, Bitmapset *attnums) int numattrs = attrs->dim1; /* skip dependencies-only stats */ - if (! info->mcv_built) + if (! (info->mcv_built || info->hist_built)) continue; /* count columns covered by the histogram */ @@ -1433,7 +1458,6 @@ clause_is_mv_compatible(PlannerInfo *root, Node *clause, Oid varRelid, bool ok; /* is it 'variable op constant' ? */ - ok = (bms_membership(clause_relids) == BMS_SINGLETON) && (is_pseudo_constant_clause_relids(lsecond(expr->args), right_relids) || @@ -1483,10 +1507,10 @@ clause_is_mv_compatible(PlannerInfo *root, Node *clause, Oid varRelid, case F_SCALARLTSEL: case F_SCALARGTSEL: /* not compatible with functional dependencies */ - if (types & MV_CLAUSE_TYPE_MCV) + if (types & (MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST)) { *attnums = bms_add_member(*attnums, var->varattno); - return (types & MV_CLAUSE_TYPE_MCV); + return (types & (MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST)); } return false; @@ -1814,6 +1838,9 @@ has_stats(List *stats, int type) if ((type & MV_CLAUSE_TYPE_MCV) && stat->mcv_built) return true; + + if ((type & MV_CLAUSE_TYPE_HIST) && stat->hist_built) + return true; } return false; @@ -2630,3 +2657,671 @@ update_match_bitmap_mcvlist(PlannerInfo *root, List *clauses, return nmatches; } + +/* + * Estimate selectivity of clauses using a histogram. + * + * If there's no histogram for the stats, the function returns 0.0. + * + * The general idea of this method is similar to how MCV lists are + * processed, except that this introduces the concept of a partial + * match (MCV only works with full match / mismatch). + * + * The algorithm works like this: + * + * 1) mark all buckets as 'full match' + * 2) walk through all the clauses + * 3) for a particular clause, walk through all the buckets + * 4) skip buckets that are already 'no match' + * 5) check clause for buckets that still match (at least partially) + * 6) sum frequencies for buckets to get selectivity + * + * Unlike MCV lists, histograms have a concept of a partial match. In + * that case we use 1/2 the bucket, to minimize the average error. The + * MV histograms are usually less detailed than the per-column ones, + * meaning the sum is often quite high (thanks to combining a lot of + * "partially hit" buckets). + * + * Maybe we could use per-bucket information with number of distinct + * values it contains (for each dimension), and then use that to correct + * the estimate (so with 10 distinct values, we'd use 1/10 of the bucket + * frequency). We might also scale the value depending on the actual + * ndistinct estimate (not just the values observed in the sample). + * + * Another option would be to multiply the selectivities, i.e. if we get + * 'partial match' for a bucket for multiple conditions, we might use + * 0.5^k (where k is the number of conditions), instead of 0.5. This + * probably does not minimize the average error, though. + * + * TODO This might use a similar shortcut to MCV lists - count buckets + * marked as partial/full match, and terminate once this drop to 0. + * Not sure if it's really worth it - for MCV lists a situation like + * this is not uncommon, but for histograms it's not that clear. + */ +static Selectivity +clauselist_mv_selectivity_histogram(PlannerInfo *root, List *clauses, + MVStatisticInfo *mvstats) +{ + int i; + Selectivity s = 0.0; + Selectivity u = 0.0; + + int nmatches = 0; + char *matches = NULL; + + MVSerializedHistogram mvhist = NULL; + + /* there's no histogram */ + if (! mvstats->hist_built) + return 0.0; + + /* There may be no histogram in the stats (check hist_built flag) */ + mvhist = load_mv_histogram(mvstats->mvoid); + + Assert (mvhist != NULL); + Assert (clauses != NIL); + Assert (list_length(clauses) >= 2); + + /* + * Bitmap of bucket matches (mismatch, partial, full). by default + * all buckets fully match (and we'll eliminate them). + */ + matches = palloc0(sizeof(char) * mvhist->nbuckets); + memset(matches, MVSTATS_MATCH_FULL, sizeof(char)*mvhist->nbuckets); + + nmatches = mvhist->nbuckets; + + /* build the match bitmap */ + update_match_bitmap_histogram(root, clauses, + mvstats->stakeys, mvhist, + nmatches, matches, false); + + /* now, walk through the buckets and sum the selectivities */ + for (i = 0; i < mvhist->nbuckets; i++) + { + /* + * Find out what part of the data is covered by the histogram, + * so that we can 'scale' the selectivity properly (e.g. when + * only 50% of the sample got into the histogram, and the rest + * is in a MCV list). + * + * TODO This might be handled by keeping a global "frequency" + * for the whole histogram, which might save us some time + * spent accessing the not-matching part of the histogram. + * Although it's likely in a cache, so it's very fast. + */ + u += mvhist->buckets[i]->ntuples; + + if (matches[i] == MVSTATS_MATCH_FULL) + s += mvhist->buckets[i]->ntuples; + else if (matches[i] == MVSTATS_MATCH_PARTIAL) + s += 0.5 * mvhist->buckets[i]->ntuples; + } + + /* release the allocated bitmap and deserialized histogram */ + pfree(matches); + pfree(mvhist); + + return s * u; +} + +/* + * Evaluate clauses using the histogram, and update the match bitmap. + * + * The bitmap may be already partially set, so this is really a way to + * combine results of several clause lists - either when computing + * conditional probability P(A|B) or a combination of AND/OR clauses. + * + * Note: This is not a simple bitmap in the sense that there are more + * than two possible values for each item - no match, partial + * match and full match. So we need 2 bits per item. + * + * TODO This works with 'bitmap' where each item is represented as a + * char, which is slightly wasteful. Instead, we could use a bitmap + * with 2 bits per item, reducing the size to ~1/4. By using values + * 0, 1 and 3 (instead of 0, 1 and 2), the operations (merging etc.) + * might be performed just like for simple bitmap by using & and |, + * which might be faster than min/max. + */ +static int +update_match_bitmap_histogram(PlannerInfo *root, List *clauses, + int2vector *stakeys, + MVSerializedHistogram mvhist, + int nmatches, char * matches, + bool is_or) +{ + int i; + ListCell * l; + + /* + * Used for caching function calls, only once per deduplicated value. + * + * We know may have up to (2 * nbuckets) values per dimension. It's + * probably overkill, but let's allocate that once for all clauses, + * to minimize overhead. + * + * Also, we only need two bits per value, but this allocates byte + * per value. Might be worth optimizing. + * + * 0x00 - not yet called + * 0x01 - called, result is 'false' + * 0x03 - called, result is 'true' + */ + char *callcache = palloc(mvhist->nbuckets); + + Assert(mvhist != NULL); + Assert(mvhist->nbuckets > 0); + Assert(nmatches >= 0); + Assert(nmatches <= mvhist->nbuckets); + + Assert(clauses != NIL); + Assert(list_length(clauses) >= 1); + + /* loop through the clauses and do the estimation */ + foreach (l, clauses) + { + Node * clause = (Node*)lfirst(l); + + /* if it's a RestrictInfo, then extract the clause */ + if (IsA(clause, RestrictInfo)) + clause = (Node*)((RestrictInfo*)clause)->clause; + + /* it's either OpClause, or NullTest */ + if (is_opclause(clause)) + { + OpExpr * expr = (OpExpr*)clause; + bool varonleft = true; + bool ok; + + FmgrInfo opproc; /* operator */ + fmgr_info(get_opcode(expr->opno), &opproc); + + /* reset the cache (per clause) */ + memset(callcache, 0, mvhist->nbuckets); + + ok = (NumRelids(clause) == 1) && + (is_pseudo_constant_clause(lsecond(expr->args)) || + (varonleft = false, + is_pseudo_constant_clause(linitial(expr->args)))); + + if (ok) + { + FmgrInfo ltproc; + RegProcedure oprrest = get_oprrest(expr->opno); + + Var * var = (varonleft) ? linitial(expr->args) : lsecond(expr->args); + Const * cst = (varonleft) ? lsecond(expr->args) : linitial(expr->args); + bool isgt = (! varonleft); + + /* + * TODO Fetch only when really needed (probably for equality only) + * + * TODO Technically either lt/gt is sufficient. + * + * FIXME The code in analyze.c creates histograms only for types + * with enough ordering (by calling get_sort_group_operators). + * Is this the same assumption, i.e. are we certain that we + * get the ltproc/gtproc every time we ask? Or are there types + * where get_sort_group_operators returns ltopr and here we + * get nothing? + */ + TypeCacheEntry *typecache + = lookup_type_cache(var->vartype, TYPECACHE_EQ_OPR | TYPECACHE_LT_OPR + | TYPECACHE_GT_OPR); + + /* lookup dimension for the attribute */ + int idx = mv_get_index(var->varattno, stakeys); + + fmgr_info(get_opcode(typecache->lt_opr), <proc); + + /* + * Check this for all buckets that still have "true" in the bitmap + * + * We already know the clauses use suitable operators (because that's + * how we filtered them). + */ + for (i = 0; i < mvhist->nbuckets; i++) + { + bool tmp; + MVSerializedBucket bucket = mvhist->buckets[i]; + + /* histogram boundaries */ + Datum minval, maxval; + + /* values from the call cache */ + char mincached, maxcached; + + /* + * For AND-lists, we can also mark NULL buckets as 'no match' + * (and then skip them). For OR-lists this is not possible. + */ + if ((! is_or) && bucket->nullsonly[idx]) + matches[i] = MVSTATS_MATCH_NONE; + + /* + * Skip buckets that were already eliminated - this is impotant + * considering how we update the info (we only lower the match). + * We can't really do anything about the MATCH_PARTIAL buckets. + */ + if ((! is_or) && (matches[i] == MVSTATS_MATCH_NONE)) + continue; + else if (is_or && (matches[i] == MVSTATS_MATCH_FULL)) + continue; + + /* lookup the values and cache of function calls */ + minval = mvhist->values[idx][bucket->min[idx]]; + maxval = mvhist->values[idx][bucket->max[idx]]; + + mincached = callcache[bucket->min[idx]]; + maxcached = callcache[bucket->max[idx]]; + + /* + * TODO Maybe it's possible to add here a similar optimization + * as for the MCV lists: + * + * (nmatches == 0) && AND-list => all eliminated (FALSE) + * (nmatches == N) && OR-list => all eliminated (TRUE) + * + * But it's more complex because of the partial matches. + */ + + /* + * If it's not a "<" or ">" or "=" operator, just ignore the + * clause. Otherwise note the relid and attnum for the variable. + * + * TODO I'm really unsure the handling of 'isgt' flag (that is, clauses + * with reverse order of variable/constant) is correct. I wouldn't + * be surprised if there was some mixup. Using the lt/gt operators + * instead of messing with the opproc could make it simpler. + * It would however be using a different operator than the query, + * although it's not any shadier than using the selectivity function + * as is done currently. + * + * FIXME Once the min/max values are deduplicated, we can easily minimize + * the number of calls to the comparator (assuming we keep the + * deduplicated structure). See the note on compression at MVBucket + * serialize/deserialize methods. + */ + switch (oprrest) + { + case F_SCALARLTSEL: /* column < constant */ + + if (! isgt) /* (var < const) */ + { + /* + * First check whether the constant is below the lower boundary (in that + * case we can skip the bucket, because there's no overlap). + */ + if (! mincached) + { + tmp = DatumGetBool(FunctionCall2Coll(&opproc, + DEFAULT_COLLATION_OID, + cst->constvalue, + minval)); + + /* + * Update the cache, but with the inverse value, as we keep the + * cache for calls with (minval, constvalue). + */ + callcache[bucket->min[idx]] = (tmp) ? 0x01 : 0x03; + } + else + tmp = !(mincached & 0x02); /* get call result from the cache (inverse) */ + + if (tmp) + { + /* no match */ + UPDATE_RESULT(matches[i], MVSTATS_MATCH_NONE, is_or); + continue; + } + + /* + * Now check whether the upper boundary is below the constant (in that + * case it's a partial match). + */ + if (! maxcached) + { + tmp = DatumGetBool(FunctionCall2Coll(&opproc, + DEFAULT_COLLATION_OID, + cst->constvalue, + maxval)); + + /* + * Update the cache, but with the inverse value, as we keep the + * cache for calls with (minval, constvalue). + */ + callcache[bucket->max[idx]] = (tmp) ? 0x01 : 0x03; + } + else + tmp = !(maxcached & 0x02); /* extract the result (reverse) */ + + if (tmp) /* partial match */ + UPDATE_RESULT(matches[i], MVSTATS_MATCH_PARTIAL, is_or); + + } + else /* (const < var) */ + { + /* + * First check whether the constant is above the upper boundary (in that + * case we can skip the bucket, because there's no overlap). + */ + if (! maxcached) + { + tmp = DatumGetBool(FunctionCall2Coll(&opproc, + DEFAULT_COLLATION_OID, + maxval, + cst->constvalue)); + + /* Update the cache. */ + callcache[bucket->max[idx]] = (tmp) ? 0x03 : 0x01; + } + else + tmp = (maxcached & 0x02); /* extract the result */ + + if (tmp) + { + /* no match */ + UPDATE_RESULT(matches[i], MVSTATS_MATCH_NONE, is_or); + continue; + } + + /* + * Now check whether the lower boundary is below the constant (in that + * case it's a partial match). + */ + if (! mincached) + { + tmp = DatumGetBool(FunctionCall2Coll(&opproc, + DEFAULT_COLLATION_OID, + minval, + cst->constvalue)); + + /* Update the cache. */ + callcache[bucket->min[idx]] = (tmp) ? 0x03 : 0x01; + } + else + tmp = (mincached & 0x02); /* extract the result */ + + if (tmp) /* partial match */ + UPDATE_RESULT(matches[i], MVSTATS_MATCH_PARTIAL, is_or); + } + break; + + case F_SCALARGTSEL: /* column > constant */ + + if (! isgt) /* (var > const) */ + { + /* + * First check whether the constant is above the upper boundary (in that + * case we can skip the bucket, because there's no overlap). + */ + if (! maxcached) + { + tmp = DatumGetBool(FunctionCall2Coll(&opproc, + DEFAULT_COLLATION_OID, + cst->constvalue, + maxval)); + + /* + * Update the cache, but with the inverse value, as we keep the + * cache for calls with (val, constvalue). + */ + callcache[bucket->max[idx]] = (tmp) ? 0x01 : 0x03; + } + else + tmp = !(maxcached & 0x02); /* extract the result */ + + if (tmp) + { + /* no match */ + UPDATE_RESULT(matches[i], MVSTATS_MATCH_NONE, is_or); + continue; + } + + /* + * Now check whether the lower boundary is below the constant (in that + * case it's a partial match). + */ + if (! mincached) + { + tmp = DatumGetBool(FunctionCall2Coll(&opproc, + DEFAULT_COLLATION_OID, + cst->constvalue, + minval)); + + /* + * Update the cache, but with the inverse value, as we keep the + * cache for calls with (val, constvalue). + */ + callcache[bucket->min[idx]] = (tmp) ? 0x01 : 0x03; + } + else + tmp = !(mincached & 0x02); /* extract the result */ + + if (tmp) + /* partial match */ + UPDATE_RESULT(matches[i], MVSTATS_MATCH_PARTIAL, is_or); + } + else /* (const > var) */ + { + /* + * First check whether the constant is below the lower boundary (in + * that case we can skip the bucket, because there's no overlap). + */ + if (! mincached) + { + tmp = DatumGetBool(FunctionCall2Coll(&opproc, + DEFAULT_COLLATION_OID, + minval, + cst->constvalue)); + + /* Update the cache. */ + callcache[bucket->min[idx]] = (tmp) ? 0x03 : 0x01; + } + else + tmp = (mincached & 0x02); /* extract the result */ + + if (tmp) + { + /* no match */ + UPDATE_RESULT(matches[i], MVSTATS_MATCH_NONE, is_or); + continue; + } + + /* + * Now check whether the upper boundary is below the constant (in that + * case it's a partial match). + */ + if (! maxcached) + { + tmp = DatumGetBool(FunctionCall2Coll(&opproc, + DEFAULT_COLLATION_OID, + maxval, + cst->constvalue)); + + /* Update the cache. */ + callcache[bucket->max[idx]] = (tmp) ? 0x03 : 0x01; + } + else + tmp = (maxcached & 0x02); /* extract the result */ + + if (tmp) + /* partial match */ + UPDATE_RESULT(matches[i], MVSTATS_MATCH_PARTIAL, is_or); + } + break; + + case F_EQSEL: + + /* + * We only check whether the value is within the bucket, using the lt/gt + * operators fetched from type cache. + * + * TODO We'll use the default 50% estimate, but that's probably way off + * if there are multiple distinct values. Consider tweaking this a + * somehow, e.g. using only a part inversely proportional to the + * estimated number of distinct values in the bucket. + * + * TODO This does not handle inclusion flags at the moment, thus counting + * some buckets twice (when hitting the boundary). + * + * TODO Optimization is that if max[i] == min[i], it's effectively a MCV + * item and we can count the whole bucket as a complete match (thus + * using 100% bucket selectivity and not just 50%). + * + * TODO Technically some buckets may "degenerate" into single-value + * buckets (not necessarily for all the dimensions) - maybe this + * is better than keeping a separate MCV list (multi-dimensional). + * Update: Actually, that's unlikely to be better than a separate + * MCV list for two reasons - first, it requires ~2x the space + * (because of storing lower/upper boundaries) and second because + * the buckets are ranges - depending on the partitioning algorithm + * it may not even degenerate into (min=max) bucket. For example the + * the current partitioning algorithm never does that. + */ + if (! mincached) + { + tmp = DatumGetBool(FunctionCall2Coll(<proc, + DEFAULT_COLLATION_OID, + cst->constvalue, + minval)); + + /* Update the cache. */ + callcache[bucket->min[idx]] = (tmp) ? 0x03 : 0x01; + } + else + tmp = (mincached & 0x02); /* extract the result */ + + if (tmp) + { + /* no match */ + UPDATE_RESULT(matches[i], MVSTATS_MATCH_NONE, is_or); + continue; + } + + if (! maxcached) + { + tmp = DatumGetBool(FunctionCall2Coll(<proc, + DEFAULT_COLLATION_OID, + maxval, + cst->constvalue)); + + /* Update the cache. */ + callcache[bucket->max[idx]] = (tmp) ? 0x03 : 0x01; + } + else + tmp = (maxcached & 0x02); /* extract the result */ + + if (tmp) + { + /* no match */ + UPDATE_RESULT(matches[i], MVSTATS_MATCH_NONE, is_or); + continue; + } + + /* partial match */ + UPDATE_RESULT(matches[i], MVSTATS_MATCH_PARTIAL, is_or); + + break; + } + } + } + } + else if (IsA(clause, NullTest)) + { + NullTest * expr = (NullTest*)clause; + Var * var = (Var*)(expr->arg); + + /* FIXME proper matching attribute to dimension */ + int idx = mv_get_index(var->varattno, stakeys); + + /* + * Walk through the buckets and evaluate the current clause. We can + * skip items that were already ruled out, and terminate if there are + * no remaining buckets that might possibly match. + */ + for (i = 0; i < mvhist->nbuckets; i++) + { + MVSerializedBucket bucket = mvhist->buckets[i]; + + /* + * Skip buckets that were already eliminated - this is impotant + * considering how we update the info (we only lower the match) + */ + if ((! is_or) && (matches[i] == MVSTATS_MATCH_NONE)) + continue; + else if (is_or && (matches[i] == MVSTATS_MATCH_FULL)) + continue; + + /* if the clause mismatches the MCV item, set it as MATCH_NONE */ + if ((expr->nulltesttype == IS_NULL) + && (! bucket->nullsonly[idx])) + UPDATE_RESULT(matches[i], MVSTATS_MATCH_NONE, is_or); + + else if ((expr->nulltesttype == IS_NOT_NULL) && + (bucket->nullsonly[idx])) + UPDATE_RESULT(matches[i], MVSTATS_MATCH_NONE, is_or); + } + } + else if (or_clause(clause) || and_clause(clause)) + { + /* AND/OR clause, with all clauses compatible with the selected MV stat */ + + int i; + BoolExpr *orclause = ((BoolExpr*)clause); + List *orclauses = orclause->args; + + /* match/mismatch bitmap for each bucket */ + int or_nmatches = 0; + char * or_matches = NULL; + + Assert(orclauses != NIL); + Assert(list_length(orclauses) >= 2); + + /* number of matching buckets */ + or_nmatches = mvhist->nbuckets; + + /* by default none of the buckets matches the clauses */ + or_matches = palloc0(sizeof(char) * or_nmatches); + + if (or_clause(clause)) + { + /* OR clauses assume nothing matches, initially */ + memset(or_matches, MVSTATS_MATCH_NONE, sizeof(char)*or_nmatches); + or_nmatches = 0; + } + else + { + /* AND clauses assume nothing matches, initially */ + memset(or_matches, MVSTATS_MATCH_FULL, sizeof(char)*or_nmatches); + } + + /* build the match bitmap for the OR-clauses */ + or_nmatches = update_match_bitmap_histogram(root, orclauses, + stakeys, mvhist, + or_nmatches, or_matches, or_clause(clause)); + + /* merge the bitmap into the existing one*/ + for (i = 0; i < mvhist->nbuckets; i++) + { + /* + * To AND-merge the bitmaps, a MIN() semantics is used. + * For OR-merge, use MAX(). + * + * FIXME this does not decrease the number of matches + */ + UPDATE_RESULT(matches[i], or_matches[i], is_or); + } + + pfree(or_matches); + + } + else + elog(ERROR, "unknown clause type: %d", clause->type); + } + + /* free the call cache */ + pfree(callcache); + + return nmatches; +} diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index 8c4396a..0dc575a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -409,7 +409,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, mvstat = (Form_pg_mv_statistic) GETSTRUCT(htup); /* unavailable stats are not interesting for the planner */ - if (mvstat->deps_built || mvstat->mcv_built) + if (mvstat->deps_built || mvstat->mcv_built || mvstat->hist_built) { info = makeNode(MVStatisticInfo); @@ -419,10 +419,12 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, /* enabled statistics */ info->deps_enabled = mvstat->deps_enabled; info->mcv_enabled = mvstat->mcv_enabled; + info->hist_enabled = mvstat->hist_enabled; /* built/available statistics */ info->deps_built = mvstat->deps_built; info->mcv_built = mvstat->mcv_built; + info->hist_built = mvstat->hist_built; /* stakeys */ adatum = SysCacheGetAttr(MVSTATOID, htup, diff --git a/src/backend/utils/mvstats/Makefile b/src/backend/utils/mvstats/Makefile index f9bf10c..9dbb3b6 100644 --- a/src/backend/utils/mvstats/Makefile +++ b/src/backend/utils/mvstats/Makefile @@ -12,6 +12,6 @@ subdir = src/backend/utils/mvstats top_builddir = ../../../.. include $(top_builddir)/src/Makefile.global -OBJS = common.o dependencies.o mcv.o +OBJS = common.o dependencies.o histogram.o mcv.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/utils/mvstats/common.c b/src/backend/utils/mvstats/common.c index d1da714..ffb76f4 100644 --- a/src/backend/utils/mvstats/common.c +++ b/src/backend/utils/mvstats/common.c @@ -13,11 +13,11 @@ * *------------------------------------------------------------------------- */ +#include "postgres.h" +#include "utils/array.h" #include "common.h" -#include "utils/array.h" - static VacAttrStats ** lookup_var_attr_stats(int2vector *attrs, int natts, VacAttrStats **vacattrstats); @@ -52,7 +52,8 @@ build_mv_stats(Relation onerel, int numrows, HeapTuple *rows, MVStatisticInfo *stat = (MVStatisticInfo *)lfirst(lc); MVDependencies deps = NULL; MCVList mcvlist = NULL; - int numrows_filtered = 0; + MVHistogram histogram = NULL; + int numrows_filtered = numrows; VacAttrStats **stats = NULL; int numatts = 0; @@ -95,8 +96,12 @@ build_mv_stats(Relation onerel, int numrows, HeapTuple *rows, if (stat->mcv_enabled) mcvlist = build_mv_mcvlist(numrows, rows, attrs, stats, &numrows_filtered); + /* build a multivariate histogram on the columns */ + if ((numrows_filtered > 0) && (stat->hist_enabled)) + histogram = build_mv_histogram(numrows_filtered, rows, attrs, stats, numrows); + /* store the histogram / MCV list in the catalog */ - update_mv_stats(stat->mvoid, deps, mcvlist, attrs, stats); + update_mv_stats(stat->mvoid, deps, mcvlist, histogram, attrs, stats); } } @@ -176,6 +181,8 @@ list_mv_stats(Oid relid) info->deps_built = stats->deps_built; info->mcv_enabled = stats->mcv_enabled; info->mcv_built = stats->mcv_built; + info->hist_enabled = stats->hist_enabled; + info->hist_built = stats->hist_built; result = lappend(result, info); } @@ -190,7 +197,6 @@ list_mv_stats(Oid relid) return result; } - /* * Find attnims of MV stats using the mvoid. */ @@ -236,9 +242,16 @@ find_mv_attnums(Oid mvoid, Oid *relid) } +/* + * FIXME This adds statistics, but we need to drop statistics when the + * table is dropped. Not sure what to do when a column is dropped. + * Either we can (a) remove all stats on that column, (b) remove + * the column from defined stats and force rebuild, (c) remove the + * column on next ANALYZE. Or maybe something else? + */ void update_mv_stats(Oid mvoid, - MVDependencies dependencies, MCVList mcvlist, + MVDependencies dependencies, MCVList mcvlist, MVHistogram histogram, int2vector *attrs, VacAttrStats **stats) { HeapTuple stup, @@ -271,22 +284,34 @@ update_mv_stats(Oid mvoid, values[Anum_pg_mv_statistic_stamcv - 1] = PointerGetDatum(data); } + if (histogram != NULL) + { + bytea * data = serialize_mv_histogram(histogram, attrs, stats); + nulls[Anum_pg_mv_statistic_stahist-1] = (data == NULL); + values[Anum_pg_mv_statistic_stahist - 1] + = PointerGetDatum(data); + } + /* always replace the value (either by bytea or NULL) */ replaces[Anum_pg_mv_statistic_stadeps -1] = true; replaces[Anum_pg_mv_statistic_stamcv -1] = true; + replaces[Anum_pg_mv_statistic_stahist-1] = true; /* always change the availability flags */ nulls[Anum_pg_mv_statistic_deps_built -1] = false; nulls[Anum_pg_mv_statistic_mcv_built -1] = false; + nulls[Anum_pg_mv_statistic_hist_built-1] = false; nulls[Anum_pg_mv_statistic_stakeys-1] = false; /* use the new attnums, in case we removed some dropped ones */ replaces[Anum_pg_mv_statistic_deps_built-1] = true; replaces[Anum_pg_mv_statistic_mcv_built -1] = true; + replaces[Anum_pg_mv_statistic_hist_built -1] = true; replaces[Anum_pg_mv_statistic_stakeys -1] = true; values[Anum_pg_mv_statistic_deps_built-1] = BoolGetDatum(dependencies != NULL); values[Anum_pg_mv_statistic_mcv_built -1] = BoolGetDatum(mcvlist != NULL); + values[Anum_pg_mv_statistic_hist_built -1] = BoolGetDatum(histogram != NULL); values[Anum_pg_mv_statistic_stakeys -1] = PointerGetDatum(attrs); /* Is there already a pg_mv_statistic tuple for this attribute? */ diff --git a/src/backend/utils/mvstats/histogram.c b/src/backend/utils/mvstats/histogram.c new file mode 100644 index 0000000..6290d2f --- /dev/null +++ b/src/backend/utils/mvstats/histogram.c @@ -0,0 +1,2188 @@ +/*------------------------------------------------------------------------- + * + * histogram.c + * POSTGRES multivariate histograms + * + * + * Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * + * IDENTIFICATION + * src/backend/utils/mvstats/histogram.c + * + *------------------------------------------------------------------------- + */ + +#include "postgres.h" + +#include "fmgr.h" +#include "funcapi.h" + +#include "utils/lsyscache.h" + +#include "common.h" +#include + +/* + * Multivariate histograms + * ----------------------- + * + * Histograms are a collection of buckets, represented by n-dimensional + * rectangles. Each rectangle is delimited by a min/max value in each + * dimension, stored in an array, so that the bucket includes values + * fulfilling condition + * + * min[i] <= value[i] <= max[i] + * + * where 'i' is the dimension. In 1D this corresponds to a simple + * interval, in 2D to a rectangle, and in 3D to a block. If you can + * imagine this in 4D, congrats! + * + * In addition to the bounaries, each bucket tracks additional details: + * + * * frequency (fraction of tuples it matches) + * * whether the boundaries are inclusive or exclusive + * * whether the dimension contains only NULL values + * * number of distinct values in each dimension (for building) + * + * and possibly some additional information. + * + * We do expect to support multiple histogram types, with different + * features etc. The 'type' field is used to identify those types. + * Technically some histogram types might use completely different + * bucket representation, but that's not expected at the moment. + * + * Although the current implementation builds non-overlapping buckets, + * the code does not (and should not) rely on the non-overlapping + * nature - there are interesting types of histograms / histogram + * building algorithms producing overlapping buckets. + * + * + * NULL handling (create_null_buckets) + * ----------------------------------- + * Another thing worth mentioning is handling of NULL values. It would + * be quite difficult to work with buckets containing NULL and non-NULL + * values for a single dimension. To work around this, the initial step + * in building a histogram is building a set of 'NULL-buckets', i.e. + * buckets with one or more NULL-only dimensions. + * + * After that, no buckets are mixing NULL and non-NULL values in one + * dimension, and the actual histogram building starts. As that only + * splits the buckets into smaller ones, the resulting buckets can't + * mix NULL and non-NULL values either. + * + * The maximum number of NULL-buckets is determined by the number of + * attributes the histogram is built on. For N-dimensional histogram, + * the maximum number of NULL-buckets is 2^N. So for 8 attributes + * (which is the current value of MVSTATS_MAX_DIMENSIONS), there may be + * up to 256 NULL-buckets. + * + * Those buckets are only built if needed - if there are no NULL values + * in the data, no such buckets are built. + * + * + * Estimating selectivity + * ---------------------- + * With histograms, we always "match" a whole bucket, not indivitual + * rows (or values), irrespectedly of the type of clause. Therefore we + * can't use the optimizations for equality clauses, as in MCV lists. + * + * The current implementation uses histograms to estimates those types + * of clauses (think of WHERE conditions): + * + * (a) equality clauses WHERE (a = 1) AND (b = 2) + * (b) inequality clauses WHERE (a < 1) AND (b >= 2) + * (c) NULL clauses WHERE (a IS NULL) AND (b IS NOT NULL) + * (d) OR-clauses WHERE (a = 1) OR (b = 2) + * + * It's possible to add more clauses, for example: + * + * (e) multi-var clauses WHERE (a > b) + * + * and so on. These are tasks for the future, not yet implemented. + * + * When used on low-cardinality data, histograms usually perform + * considerably worse than MCV lists (which are a good fit for this + * kind of data). This is especially true on categorical data, where + * ordering of the values is mostly unrelated to meaning of the data, + * as proper ordering is crucial for histograms. + * + * On high-cardinality data the histograms are usually a better choice, + * because MCV lists can't represent the distribution accurately enough. + * + * By evaluating a clause on a bucket, we may get one of three results: + * + * (a) FULL_MATCH - The bucket definitely matches the clause. + * + * (b) PARTIAL_MATCH - The bucket matches the clause, but not + * necessarily all the tuples it represents. + * + * (c) NO_MATCH - The bucket definitely does not match the clause. + * + * This may be illustrated using a range [1, 5], which is essentially + * a 1-D bucket. With clause + * + * WHERE (a < 10) => FULL_MATCH (all range values are below + * 10, so the whole bucket matches) + * + * WHERE (a < 3) => PARTIAL_MATCH (there may be values matching + * the clause, but we don't know how many) + * + * WHERE (a < 0) => NO_MATCH (the whole range is above 1, so + * no values from the bucket can match) + * + * Some clauses may produce only some of those results - for example + * equality clauses may never produce FULL_MATCH as we always hit only + * part of the bucket (we can't match both boundaries at the same time). + * This results in less accurate estimates compared to MCV lists, where + * we can hit a MCV items exactly (there's no PARTIAL match in MCV). + * + * There are clauses that may not produce any PARTIAL_MATCH results. + * A nice example of that is 'IS [NOT] NULL' clause, which either + * matches the bucket completely (FULL_MATCH) or not at all (NO_MATCH), + * thanks to how the NULL-buckets are constructed. + * + * Computing the total selectivity estimate is trivial - simply sum + * selectivities from all the FULL_MATCH and PARTIAL_MATCH buckets (but + * multiply the PARTIAL_MATCH buckets by 0.5 to minimize average error). + * + * + * Serialization + * ------------- + * After building, the histogram is serialized into a more efficient + * form (dedup boundary values etc.). See serialize_mv_histogram() for + * more details about how it's done. + * + * Serialized histograms are marked with 'magic' constant, to make it + * easier to check the bytea value really is a serialized histogram. + * + * In the serialized form, values for each dimension are deduplicated, + * and referenced using an uint16 index. This saves a lot of space, + * because every time we split a bucket, we introduce a single new + * boundary value (to split the bucket by the selected dimension), but + * we actually copy all the boundary values for all dimensions. So for + * a histogram with 4 dimensions and 1000 buckets, we do have + * + * 1000 * 4 * 2 = 8000 + * + * boundary values, but many of them are actually duplicated because + * the histogram started with a single bucket (8 boundary values) and + * then there were 999 splits (each introducing 1 new value): + * + * 8 + 999 = 1007 + * + * So that's quite large diffence. Let's assume the Datum values are + * 8 bytes each. Storing the raw histogram would take ~ 64 kB, while + * with deduplication it's only ~18 kB. + * + * The difference may be removed by the transparent bytea compression, + * but the deduplication is also used to optimize the estimation. It's + * possible to process the deduplicated values, and then use this as + * a cache to minimize the actual function calls while checking the + * buckets. This significantly reduces the number of calls to the + * (often quite expensive) operator functions etc. + * + * + * The current limit on number of buckets (16384) is mostly arbitrary, + * but set so that it makes sure we don't exceed the number of distinct + * values indexable by uint16. In practice we could handle more buckets, + * because we index each dimension independently, and we do the splits + * over multiple dimensions. + * + * Histograms with more than 16k buckets are quite expensive to build + * and process, so the current limit is somewhat reasonable. + * + * The actual number of buckets is also related to statistics target, + * because we require MIN_BUCKET_ROWS (10) tuples per bucket before + * a split, so we can't have more than (2 * 300 * target / 10) buckets. + * + * + * TODO Maybe the distinct stats (both for combination of all columns + * and for combinations of various subsets of columns) should be + * moved to a separate structure (next to histogram/MCV/...) to + * make it useful even without a histogram computed etc. + * + * This would actually make mvcoeff (proposed by Kyotaro Horiguchi + * in [1]) possible. Seems like a good way to estimate GROUP BY + * cardinality, and also some other cases, pointed out by Kyotaro: + * + * [1] http://www.postgresql.org/message-id/20150515.152936.83796179.horiguchi.kyotaro@lab.ntt.co.jp + * + * This is not implemented at the moment, though. Also, Kyotaro's + * patch only works with pairs of columns, but maybe tracking all + * the combinations would be useful to handle more complex + * conditions. It only seems to handle equalities, though (but for + * GROUP BY estimation that's not a big deal). + */ + +static MVBucket create_initial_mv_bucket(int numrows, HeapTuple *rows, + int2vector *attrs, + VacAttrStats **stats); + +static MVBucket select_bucket_to_partition(int nbuckets, MVBucket * buckets); + +static MVBucket partition_bucket(MVBucket bucket, int2vector *attrs, + VacAttrStats **stats, + int *ndistvalues, Datum **distvalues); + +static MVBucket copy_mv_bucket(MVBucket bucket, uint32 ndimensions); + +static void update_bucket_ndistinct(MVBucket bucket, int2vector *attrs, + VacAttrStats ** stats); + +static void update_dimension_ndistinct(MVBucket bucket, int dimension, + int2vector *attrs, + VacAttrStats ** stats, + bool update_boundaries); + +static void create_null_buckets(MVHistogram histogram, int bucket_idx, + int2vector *attrs, VacAttrStats ** stats); + +static int bsearch_comparator(const void * a, const void * b); + +/* + * Each serialized bucket needs to store (in this order): + * + * - number of tuples (float) + * - number of distinct (float) + * - min inclusive flags (ndim * sizeof(bool)) + * - max inclusive flags (ndim * sizeof(bool)) + * - null dimension flags (ndim * sizeof(bool)) + * - min boundary indexes (2 * ndim * sizeof(int32)) + * - max boundary indexes (2 * ndim * sizeof(int32)) + * + * So in total: + * + * ndim * (4 * sizeof(int32) + 3 * sizeof(bool)) + + * 2 * sizeof(float) + */ +#define BUCKET_SIZE(ndims) \ + (ndims * (4 * sizeof(uint16) + 3 * sizeof(bool)) + sizeof(float)) + +/* pointers into a flat serialized bucket of BUCKET_SIZE(n) bytes */ +#define BUCKET_NTUPLES(b) ((float*)b) +#define BUCKET_MIN_INCL(b,n) ((bool*)(b + sizeof(float))) +#define BUCKET_MAX_INCL(b,n) (BUCKET_MIN_INCL(b,n) + n) +#define BUCKET_NULLS_ONLY(b,n) (BUCKET_MAX_INCL(b,n) + n) +#define BUCKET_MIN_INDEXES(b,n) ((uint16*)(BUCKET_NULLS_ONLY(b,n) + n)) +#define BUCKET_MAX_INDEXES(b,n) ((BUCKET_MIN_INDEXES(b,n) + n)) + +/* can't split bucket with less than 10 rows */ +#define MIN_BUCKET_ROWS 10 + +/* + * Data used while building the histogram. + */ +typedef struct HistogramBuildData { + + float ndistinct; /* frequency of distinct values */ + + HeapTuple *rows; /* aray of sample rows */ + uint32 numrows; /* number of sample rows (array size) */ + + /* + * Number of distinct values in each dimension. This is used when + * building the histogram (and is not serialized/deserialized). + */ + uint32 *ndistincts; + +} HistogramBuildData; + +typedef HistogramBuildData *HistogramBuild; + +/* + * Building a multivariate algorithm. In short it first creates a single + * bucket containing all the rows, and then repeatedly split is by first + * searching for the bucket / dimension most in need of a split. + * + * The current criteria is rather simple, chosen so that the algorithm + * produces buckets with about equal frequency and regular size. + * + * See the discussion at select_bucket_to_partition and partition_bucket + * for more details about the algorithm. + * + * The current algorithm works like this: + * + * build NULL-buckets (create_null_buckets) + * + * while [not reaching maximum number of buckets] + * + * choose bucket to partition (largest bucket) + * if no bucket to partition + * terminate the algorithm + * + * choose bucket dimension to partition (largest dimension) + * split the bucket into two buckets + */ +MVHistogram +build_mv_histogram(int numrows, HeapTuple *rows, int2vector *attrs, + VacAttrStats **stats, int numrows_total) +{ + int i; + int numattrs = attrs->dim1; + + int *ndistvalues; + Datum **distvalues; + + MVHistogram histogram = (MVHistogram)palloc0(sizeof(MVHistogramData)); + + HeapTuple * rows_copy = (HeapTuple*)palloc0(numrows * sizeof(HeapTuple)); + memcpy(rows_copy, rows, sizeof(HeapTuple) * numrows); + + Assert((numattrs >= 2) && (numattrs <= MVSTATS_MAX_DIMENSIONS)); + + histogram->ndimensions = numattrs; + + histogram->magic = MVSTAT_HIST_MAGIC; + histogram->type = MVSTAT_HIST_TYPE_BASIC; + histogram->nbuckets = 1; + + /* create max buckets (better than repalloc for short-lived objects) */ + histogram->buckets + = (MVBucket*)palloc0(MVSTAT_HIST_MAX_BUCKETS * sizeof(MVBucket)); + + /* create the initial bucket, covering the whole sample set */ + histogram->buckets[0] + = create_initial_mv_bucket(numrows, rows_copy, attrs, stats); + + /* + * Collect info on distinct values in each dimension (used later + * to select dimension to partition). + */ + ndistvalues = (int*)palloc0(sizeof(int) * numattrs); + distvalues = (Datum**)palloc0(sizeof(Datum*) * numattrs); + + for (i = 0; i < numattrs; i++) + { + int j; + int nvals; + Datum *tmp; + + SortSupportData ssup; + StdAnalyzeData *mystats = (StdAnalyzeData *) stats[i]->extra_data; + + /* initialize sort support, etc. */ + memset(&ssup, 0, sizeof(ssup)); + ssup.ssup_cxt = CurrentMemoryContext; + + /* We always use the default collation for statistics */ + ssup.ssup_collation = DEFAULT_COLLATION_OID; + ssup.ssup_nulls_first = false; + + PrepareSortSupportFromOrderingOp(mystats->ltopr, &ssup); + + nvals = 0; + tmp = (Datum*)palloc0(sizeof(Datum) * numrows); + + for (j = 0; j < numrows; j++) + { + bool isnull; + + /* remember the index of the sample row, to make the partitioning simpler */ + Datum value = heap_getattr(rows[j], attrs->values[i], + stats[i]->tupDesc, &isnull); + + if (isnull) + continue; + + tmp[nvals++] = value; + } + + /* do the sort and stuff only if there are non-NULL values */ + if (nvals > 0) + { + /* sort the array of values */ + qsort_arg((void *) tmp, nvals, sizeof(Datum), + compare_scalars_simple, (void *) &ssup); + + /* count distinct values */ + ndistvalues[i] = 1; + for (j = 1; j < nvals; j++) + if (compare_scalars_simple(&tmp[j], &tmp[j-1], &ssup) != 0) + ndistvalues[i] += 1; + + /* FIXME allocate only needed space (count ndistinct first) */ + distvalues[i] = (Datum*)palloc0(sizeof(Datum) * ndistvalues[i]); + + /* now collect distinct values into the array */ + distvalues[i][0] = tmp[0]; + ndistvalues[i] = 1; + + for (j = 1; j < nvals; j++) + { + if (compare_scalars_simple(&tmp[j], &tmp[j-1], &ssup) != 0) + { + distvalues[i][ndistvalues[i]] = tmp[j]; + ndistvalues[i] += 1; + } + } + } + + pfree(tmp); + } + + /* + * The initial bucket may contain NULL values, so we have to create + * buckets with NULL-only dimensions. + * + * FIXME We may need up to 2^ndims buckets - check that there are + * enough buckets (MVSTAT_HIST_MAX_BUCKETS >= 2^ndims). + */ + create_null_buckets(histogram, 0, attrs, stats); + + while (histogram->nbuckets < MVSTAT_HIST_MAX_BUCKETS) + { + MVBucket bucket = select_bucket_to_partition(histogram->nbuckets, + histogram->buckets); + + /* no more buckets to partition */ + if (bucket == NULL) + break; + + histogram->buckets[histogram->nbuckets] + = partition_bucket(bucket, attrs, stats, + ndistvalues, distvalues); + + histogram->nbuckets += 1; + } + + /* finalize the frequencies etc. */ + for (i = 0; i < histogram->nbuckets; i++) + { + HistogramBuild build_data + = ((HistogramBuild)histogram->buckets[i]->build_data); + + /* + * The frequency has to be computed from the whole sample, in + * case some of the rows were used for MCV (and thus are missing + * from the histogram). + */ + histogram->buckets[i]->ntuples + = (build_data->numrows * 1.0) / numrows_total; + } + + return histogram; +} + +/* fetch the histogram (as a bytea) from the pg_mv_statistic catalog */ +MVSerializedHistogram +load_mv_histogram(Oid mvoid) +{ + bool isnull = false; + Datum histogram; + +#ifdef USE_ASSERT_CHECKING + Form_pg_mv_statistic mvstat; +#endif + + /* Prepare to scan pg_mv_statistic for entries having indrelid = this rel. */ + HeapTuple htup = SearchSysCache1(MVSTATOID, ObjectIdGetDatum(mvoid)); + + if (! HeapTupleIsValid(htup)) + return NULL; + +#ifdef USE_ASSERT_CHECKING + mvstat = (Form_pg_mv_statistic) GETSTRUCT(htup); + Assert(mvstat->hist_enabled && mvstat->hist_built); +#endif + + histogram = SysCacheGetAttr(MVSTATOID, htup, + Anum_pg_mv_statistic_stahist, &isnull); + + Assert(!isnull); + + ReleaseSysCache(htup); + + return deserialize_mv_histogram(DatumGetByteaP(histogram)); +} + +/* print some basic info about the histogram */ +Datum +pg_mv_stats_histogram_info(PG_FUNCTION_ARGS) +{ + bytea *data = PG_GETARG_BYTEA_P(0); + char *result; + + MVSerializedHistogram hist = deserialize_mv_histogram(data); + + result = palloc0(128); + snprintf(result, 128, "nbuckets=%d", hist->nbuckets); + + PG_RETURN_TEXT_P(cstring_to_text(result)); +} + + +/* used to pass context into bsearch() */ +static SortSupport ssup_private = NULL; + +/* + * Serialize the MV histogram into a bytea value. The basic algorithm + * is simple, and mostly mimincs the MCV serialization: + * + * (1) perform deduplication for each attribute (separately) + * (a) collect all (non-NULL) attribute values from all buckets + * (b) sort the data (using 'lt' from VacAttrStats) + * (c) remove duplicate values from the array + * + * (2) serialize the arrays into a bytea value + * + * (3) process all buckets + * (a) replace min/max values with indexes into the arrays + * + * Each attribute has to be processed separately, because we're mixing + * different datatypes, and we don't know what equality means for them. + * We're also mixing pass-by-value and pass-by-ref types, and so on. + * + * We'll use 32-bit values for the indexes in step (3), although we + * could probably use just 16 bits as we don't allow more than 8k + * buckets in the histogram max_buckets (well, we might increase this + * to 16k and still fit into signed 16-bits). But let's be lazy and rely + * on the varlena compression to kick in. If most bytes will be 0x00 + * so it should work nicely. + * + * + * Deduplication in serialization + * ------------------------------ + * The deduplication is very effective and important here, because every + * time we split a bucket, we keep all the boundary values, except for + * the dimension that was used for the split. Another way to look at + * this is that each split introduces 1 new value (the value used to do + * the split). A histogram with M buckets was created by (M-1) splits + * of the initial bucket, and each bucket has 2*N boundary values. So + * assuming the initial bucket does not have any 'collapsed' dimensions, + * the number of distinct values is + * + * (2*N + (M-1)) + * + * but the total number of boundary values is + * + * 2*N*M + * + * which is clearly much higher. For a histogram on two columns, with + * 1024 buckets, it's 1027 vs. 4096. Of course, we're not saving all + * the difference (because we'll use 32-bit indexes into the values). + * But with large values (e.g. stored as varlena), this saves a lot. + * + * An interesting feature is that the total number of distinct values + * does not really grow with the number of dimensions, except for the + * size of the initial bucket. After that it only depends on number of + * buckets (i.e. number of splits). + * + * XXX Of course this only holds for the current histogram building + * algorithm. Algorithms doing the splits differently (e.g. + * producing overlapping buckets) may behave differently. + * + * TODO This only confirms we can use the uint16 indexes. The worst + * that could happen is if all the splits happened by a single + * dimension. To exhaust the uint16 this would require ~64k + * splits (needs to be reflected in MVSTAT_HIST_MAX_BUCKETS). + * + * TODO We don't need to use a separate boolean for each flag, instead + * use a single char and set bits. + * + * TODO We might get a bit better compression by considering the actual + * data type length. The current implementation treats all data + * types passed by value as requiring 8B, but for INT it's actually + * just 4B etc. + * + * OTOH this is only related to the lookup table, and most of the + * space is occupied by the buckets (with int16 indexes). + * + * + * Varlena compression + * ------------------- + * This encoding may prevent automatic varlena compression (similarly + * to JSONB), because first part of the serialized bytea will be an + * array of unique values (although sorted), and pglz decides whether + * to compress by trying to compress the first part (~1kB or so). Which + * is likely to be poor, due to the lack of repetition. + * + * One possible cure to that might be storing the buckets first, and + * then the deduplicated arrays. The buckets might be better suited + * for compression. + * + * On the other hand the encoding scheme is a context-aware compression, + * usually compressing to ~30% (or less, with large data types). So the + * lack of pglz compression may be OK. + * + * XXX But maybe we don't really want to compress this, to save on + * planning time? + * + * TODO Try storing the buckets / deduplicated arrays in reverse order, + * measure impact on compression. + * + * + * Deserialization + * --------------- + * The deserialization is currently implemented so that it reconstructs + * the histogram back into the same structures - this involves quite + * a few of memcpy() and palloc(), but maybe we could create a special + * structure for the serialized histogram, and access the data directly, + * without the unpacking. + * + * Not only it would save some memory and CPU time, but might actually + * work better with CPU caches (not polluting the caches). + * + * TODO Try to keep the compressed form, instead of deserializing it to + * MVHistogram/MVBucket. + * + * + * General TODOs + * ------------- + * FIXME This probably leaks memory, or at least uses it inefficiently + * (many small palloc() calls instead of a large one). + * + * FIXME This probably leaks memory, or at least uses it inefficiently + * (many small palloc() calls instead of a large one). + * + * TODO Consider packing boolean flags (NULL) for each item into 'char' + * or a longer type (instead of using an array of bool items). + */ +bytea * +serialize_mv_histogram(MVHistogram histogram, int2vector *attrs, + VacAttrStats **stats) +{ + int i = 0, j = 0; + Size total_length = 0; + + bytea *output = NULL; + char *data = NULL; + + int nbuckets = histogram->nbuckets; + int ndims = histogram->ndimensions; + + /* allocated for serialized bucket data */ + int bucketsize = BUCKET_SIZE(ndims); + char *bucket = palloc0(bucketsize); + + /* values per dimension (and number of non-NULL values) */ + Datum **values = (Datum**)palloc0(sizeof(Datum*) * ndims); + int *counts = (int*)palloc0(sizeof(int) * ndims); + + /* info about dimensions (for deserialize) */ + DimensionInfo * info + = (DimensionInfo *)palloc0(sizeof(DimensionInfo)*ndims); + + /* sort support data */ + SortSupport ssup = (SortSupport)palloc0(sizeof(SortSupportData)*ndims); + + /* collect and deduplicate values for each dimension separately */ + for (i = 0; i < ndims; i++) + { + int count; + StdAnalyzeData *tmp = (StdAnalyzeData *)stats[i]->extra_data; + + /* keep important info about the data type */ + info[i].typlen = stats[i]->attrtype->typlen; + info[i].typbyval = stats[i]->attrtype->typbyval; + + /* + * Allocate space for all min/max values, including NULLs + * (we won't use them, but we don't know how many are there), + * and then collect all non-NULL values. + */ + values[i] = (Datum*)palloc0(sizeof(Datum) * nbuckets * 2); + + for (j = 0; j < histogram->nbuckets; j++) + { + /* skip buckets where this dimension is NULL-only */ + if (! histogram->buckets[j]->nullsonly[i]) + { + values[i][counts[i]] = histogram->buckets[j]->min[i]; + counts[i] += 1; + + values[i][counts[i]] = histogram->buckets[j]->max[i]; + counts[i] += 1; + } + } + + /* there are just NULL values in this dimension */ + if (counts[i] == 0) + continue; + + /* sort and deduplicate */ + ssup[i].ssup_cxt = CurrentMemoryContext; + ssup[i].ssup_collation = DEFAULT_COLLATION_OID; + ssup[i].ssup_nulls_first = false; + + PrepareSortSupportFromOrderingOp(tmp->ltopr, &ssup[i]); + + qsort_arg(values[i], counts[i], sizeof(Datum), + compare_scalars_simple, &ssup[i]); + + /* + * Walk through the array and eliminate duplicitate values, but + * keep the ordering (so that we can do bsearch later). We know + * there's at least 1 item, so we can skip the first element. + */ + count = 1; /* number of deduplicated items */ + for (j = 1; j < counts[i]; j++) + { + /* if it's different from the previous value, we need to keep it */ + if (compare_datums_simple(values[i][j-1], values[i][j], &ssup[i]) != 0) + { + /* XXX: not needed if (count == j) */ + values[i][count] = values[i][j]; + count += 1; + } + } + + /* make sure we fit into uint16 */ + Assert(count <= UINT16_MAX); + + /* keep info about the deduplicated count */ + info[i].nvalues = count; + + /* compute size of the serialized data */ + if (info[i].typlen > 0) + /* byval or byref, but with fixed length (name, tid, ...) */ + info[i].nbytes = info[i].nvalues * info[i].typlen; + else if (info[i].typlen == -1) + /* varlena, so just use VARSIZE_ANY */ + for (j = 0; j < info[i].nvalues; j++) + info[i].nbytes += VARSIZE_ANY(values[i][j]); + else if (info[i].typlen == -2) + /* cstring, so simply strlen */ + for (j = 0; j < info[i].nvalues; j++) + info[i].nbytes += strlen(DatumGetPointer(values[i][j])); + else + elog(ERROR, "unknown data type typbyval=%d typlen=%d", + info[i].typbyval, info[i].typlen); + } + + /* + * Now we finally know how much space we'll need for the serialized + * histogram, as it contains these fields: + * + * - length (4B) for varlena + * - magic (4B) + * - type (4B) + * - ndimensions (4B) + * - nbuckets (4B) + * - info (ndim * sizeof(DimensionInfo) + * - arrays of values for each dimension + * - serialized buckets (nbuckets * bucketsize) + * + * So the 'header' size is 20B + ndim * sizeof(DimensionInfo) and + * then we'll place the data (and buckets). + */ + total_length = (sizeof(int32) + offsetof(MVHistogramData, buckets) + + ndims * sizeof(DimensionInfo) + + nbuckets * bucketsize); + + /* account for the deduplicated data */ + for (i = 0; i < ndims; i++) + total_length += info[i].nbytes; + + /* enforce arbitrary limit of 1MB */ + if (total_length > (10 * 1024 * 1024)) + elog(ERROR, "serialized histogram exceeds 10MB (%ld > %d)", + total_length, (10 * 1024 * 1024)); + + /* allocate space for the serialized histogram list, set header */ + output = (bytea*)palloc0(total_length); + SET_VARSIZE(output, total_length); + + /* we'll use 'data' to keep track of the place to write data */ + data = VARDATA(output); + + memcpy(data, histogram, offsetof(MVHistogramData, buckets)); + data += offsetof(MVHistogramData, buckets); + + memcpy(data, info, sizeof(DimensionInfo) * ndims); + data += sizeof(DimensionInfo) * ndims; + + /* value array for each dimension */ + for (i = 0; i < ndims; i++) + { +#ifdef USE_ASSERT_CHECKING + char *tmp = data; +#endif + for (j = 0; j < info[i].nvalues; j++) + { + if (info[i].typlen > 0) + { + /* pased by value or reference, but fixed length */ + memcpy(data, &values[i][j], info[i].typlen); + data += info[i].typlen; + } + else if (info[i].typlen == -1) + { + /* varlena */ + memcpy(data, DatumGetPointer(values[i][j]), + VARSIZE_ANY(values[i][j])); + data += VARSIZE_ANY(values[i][j]); + } + else if (info[i].typlen == -2) + { + /* cstring (don't forget the \0 terminator!) */ + memcpy(data, DatumGetPointer(values[i][j]), + strlen(DatumGetPointer(values[i][j])) + 1); + data += strlen(DatumGetPointer(values[i][j])) + 1; + } + } + Assert((data - tmp) == info[i].nbytes); + } + + /* and finally, the histogram buckets */ + for (i = 0; i < nbuckets; i++) + { + /* don't write beyond the allocated space */ + Assert(data <= (char*)output + total_length - bucketsize); + + /* reset the values for each item */ + memset(bucket, 0, bucketsize); + + *BUCKET_NTUPLES(bucket) = histogram->buckets[i]->ntuples; + + for (j = 0; j < ndims; j++) + { + /* do the lookup only for non-NULL values */ + if (! histogram->buckets[i]->nullsonly[j]) + { + uint16 idx; + Datum * v = NULL; + ssup_private = &ssup[j]; + + /* min boundary */ + v = (Datum*)bsearch(&histogram->buckets[i]->min[j], + values[j], info[j].nvalues, sizeof(Datum), + bsearch_comparator); + + if (v == NULL) + elog(ERROR, "value for dim %d not found in array", j); + + /* compute index within the array */ + idx = (v - values[j]); + + Assert((idx >= 0) && (idx < info[j].nvalues)); + + BUCKET_MIN_INDEXES(bucket, ndims)[j] = idx; + + /* max boundary */ + v = (Datum*)bsearch(&histogram->buckets[i]->max[j], + values[j], info[j].nvalues, sizeof(Datum), + bsearch_comparator); + + if (v == NULL) + elog(ERROR, "value for dim %d not found in array", j); + + /* compute index within the array */ + idx = (v - values[j]); + + Assert((idx >= 0) && (idx < info[j].nvalues)); + + BUCKET_MAX_INDEXES(bucket, ndims)[j] = idx; + } + } + + /* copy flags (nulls, min/max inclusive) */ + memcpy(BUCKET_NULLS_ONLY(bucket, ndims), + histogram->buckets[i]->nullsonly, sizeof(bool) * ndims); + + memcpy(BUCKET_MIN_INCL(bucket, ndims), + histogram->buckets[i]->min_inclusive, sizeof(bool) * ndims); + + memcpy(BUCKET_MAX_INCL(bucket, ndims), + histogram->buckets[i]->max_inclusive, sizeof(bool) * ndims); + + /* copy the item into the array */ + memcpy(data, bucket, bucketsize); + + data += bucketsize; + } + + /* at this point we expect to match the total_length exactly */ + Assert((data - (char*)output) == total_length); + + /* FIXME free the values/counts arrays here */ + + return output; +} + +/* + * Returns histogram in a partially-serialized form (keeps the boundary + * values deduplicated, so that it's possible to optimize the estimation + * part by caching function call results between buckets etc.). + */ +MVSerializedHistogram +deserialize_mv_histogram(bytea * data) +{ + int i = 0, j = 0; + + Size expected_size; + char *tmp = NULL; + + MVSerializedHistogram histogram; + DimensionInfo *info; + + int nbuckets; + int ndims; + int bucketsize; + + /* temporary deserialization buffer */ + int bufflen; + char *buff; + char *ptr; + + if (data == NULL) + return NULL; + + if (VARSIZE_ANY_EXHDR(data) < offsetof(MVSerializedHistogramData,buckets)) + elog(ERROR, "invalid histogram size %ld (expected at least %ld)", + VARSIZE_ANY_EXHDR(data), offsetof(MVSerializedHistogramData,buckets)); + + /* read the histogram header */ + histogram + = (MVSerializedHistogram)palloc(sizeof(MVSerializedHistogramData)); + + /* initialize pointer to the data part (skip the varlena header) */ + tmp = VARDATA(data); + + /* get the header and perform basic sanity checks */ + memcpy(histogram, tmp, offsetof(MVSerializedHistogramData, buckets)); + tmp += offsetof(MVSerializedHistogramData, buckets); + + if (histogram->magic != MVSTAT_HIST_MAGIC) + elog(ERROR, "invalid histogram magic %d (expected %dd)", + histogram->magic, MVSTAT_HIST_MAGIC); + + if (histogram->type != MVSTAT_HIST_TYPE_BASIC) + elog(ERROR, "invalid histogram type %d (expected %dd)", + histogram->type, MVSTAT_HIST_TYPE_BASIC); + + nbuckets = histogram->nbuckets; + ndims = histogram->ndimensions; + bucketsize = BUCKET_SIZE(ndims); + + Assert((nbuckets > 0) && (nbuckets <= MVSTAT_HIST_MAX_BUCKETS)); + Assert((ndims >= 2) && (ndims <= MVSTATS_MAX_DIMENSIONS)); + + /* + * What size do we expect with those parameters (it's incomplete, + * as we yet have to count the array sizes (from DimensionInfo + * records). + */ + expected_size = offsetof(MVSerializedHistogramData,buckets) + + ndims * sizeof(DimensionInfo) + + (nbuckets * bucketsize); + + /* check that we have at least the DimensionInfo records */ + if (VARSIZE_ANY_EXHDR(data) < expected_size) + elog(ERROR, "invalid histogram size %ld (expected %ld)", + VARSIZE_ANY_EXHDR(data), expected_size); + + info = (DimensionInfo*)(tmp); + tmp += ndims * sizeof(DimensionInfo); + + /* account for the value arrays */ + for (i = 0; i < ndims; i++) + expected_size += info[i].nbytes; + + if (VARSIZE_ANY_EXHDR(data) != expected_size) + elog(ERROR, "invalid histogram size %ld (expected %ld)", + VARSIZE_ANY_EXHDR(data), expected_size); + + /* looks OK - not corrupted or something */ + + /* now let's allocate a single buffer for all the values and counts */ + + bufflen = (sizeof(int) + sizeof(Datum*)) * ndims; + for (i = 0; i < ndims; i++) + { + /* don't allocate space for byval types, matching Datum */ + if (! (info[i].typbyval && (info[i].typlen == sizeof(Datum)))) + bufflen += (sizeof(Datum) * info[i].nvalues); + } + + /* also, include space for the result, tracking the buckets */ + bufflen += nbuckets * ( + sizeof(MVSerializedBucket) + /* bucket pointer */ + sizeof(MVSerializedBucketData)); /* bucket data */ + + buff = palloc(bufflen); + ptr = buff; + + histogram->nvalues = (int*)ptr; + ptr += (sizeof(int) * ndims); + + histogram->values = (Datum**)ptr; + ptr += (sizeof(Datum*) * ndims); + + /* + * FIXME This uses pointers to the original data array (the types + * not passed by value), so when someone frees the memory, + * e.g. by doing something like this: + * + * bytea * data = ... fetch the data from catalog ... + * MCVList mcvlist = deserialize_mcv_list(data); + * pfree(data); + * + * then 'mcvlist' references the freed memory. This needs to + * copy the pieces. + * + * TODO same as in MCV deserialization / consider moving to common.c + */ + for (i = 0; i < ndims; i++) + { + histogram->nvalues[i] = info[i].nvalues; + + if (info[i].typbyval && info[i].typlen == sizeof(Datum)) + { + /* passed by value / Datum - simply reuse the array */ + histogram->values[i] = (Datum*)tmp; + tmp += info[i].nbytes; + } + else + { + /* all the varlena data need a chunk from the buffer */ + histogram->values[i] = (Datum*)ptr; + ptr += (sizeof(Datum) * info[i].nvalues); + + if (info[i].typbyval) + { + /* pased by value, but smaller than Datum */ + for (j = 0; j < info[i].nvalues; j++) + { + /* just point into the array */ + memcpy(&histogram->values[i][j], tmp, info[i].typlen); + tmp += info[i].typlen; + } + } + else if (info[i].typlen > 0) + { + /* pased by reference, but fixed length (name, tid, ...) */ + for (j = 0; j < info[i].nvalues; j++) + { + /* just point into the array */ + histogram->values[i][j] = PointerGetDatum(tmp); + tmp += info[i].typlen; + } + } + else if (info[i].typlen == -1) + { + /* varlena */ + for (j = 0; j < info[i].nvalues; j++) + { + /* just point into the array */ + histogram->values[i][j] = PointerGetDatum(tmp); + tmp += VARSIZE_ANY(tmp); + } + } + else if (info[i].typlen == -2) + { + /* cstring */ + for (j = 0; j < info[i].nvalues; j++) + { + /* just point into the array */ + histogram->values[i][j] = PointerGetDatum(tmp); + tmp += (strlen(tmp) + 1); /* don't forget the \0 */ + } + } + } + } + + histogram->buckets = (MVSerializedBucket*)ptr; + ptr += (sizeof(MVSerializedBucket) * nbuckets); + + for (i = 0; i < nbuckets; i++) + { + MVSerializedBucket bucket = (MVSerializedBucket)ptr; + ptr += sizeof(MVSerializedBucketData); + + bucket->ntuples = *BUCKET_NTUPLES(tmp); + bucket->nullsonly = BUCKET_NULLS_ONLY(tmp, ndims); + bucket->min_inclusive = BUCKET_MIN_INCL(tmp, ndims); + bucket->max_inclusive = BUCKET_MAX_INCL(tmp, ndims); + + bucket->min = BUCKET_MIN_INDEXES(tmp, ndims); + bucket->max = BUCKET_MAX_INDEXES(tmp, ndims); + + histogram->buckets[i] = bucket; + + Assert(tmp <= (char*)data + VARSIZE_ANY(data)); + + tmp += bucketsize; + } + + /* at this point we expect to match the total_length exactly */ + Assert((tmp - VARDATA(data)) == expected_size); + + /* we should exhaust the output buffer exactly */ + Assert((ptr - buff) == bufflen); + + return histogram; +} + +/* + * Build the initial bucket, which will be then split into smaller ones. + */ +static MVBucket +create_initial_mv_bucket(int numrows, HeapTuple *rows, int2vector *attrs, + VacAttrStats **stats) +{ + int i; + int numattrs = attrs->dim1; + HistogramBuild data = NULL; + + /* TODO allocate bucket as a single piece, including all the fields. */ + MVBucket bucket = (MVBucket)palloc0(sizeof(MVBucketData)); + + Assert(numrows > 0); + Assert(rows != NULL); + Assert((numattrs >= 2) && (numattrs <= MVSTATS_MAX_DIMENSIONS)); + + /* allocate the per-dimension arrays */ + + /* flags for null-only dimensions */ + bucket->nullsonly = (bool*)palloc0(numattrs * sizeof(bool)); + + /* inclusiveness boundaries - lower/upper bounds */ + bucket->min_inclusive = (bool*)palloc0(numattrs * sizeof(bool)); + bucket->max_inclusive = (bool*)palloc0(numattrs * sizeof(bool)); + + /* lower/upper boundaries */ + bucket->min = (Datum*)palloc0(numattrs * sizeof(Datum)); + bucket->max = (Datum*)palloc0(numattrs * sizeof(Datum)); + + /* build-data */ + data = (HistogramBuild)palloc0(sizeof(HistogramBuildData)); + + /* number of distinct values (per dimension) */ + data->ndistincts = (uint32*)palloc0(numattrs * sizeof(uint32)); + + /* all the sample rows fall into the initial bucket */ + data->numrows = numrows; + data->rows = rows; + + bucket->build_data = data; + + /* + * Update the number of ndistinct combinations in the bucket (which + * we use when selecting bucket to partition), and then number of + * distinct values for each partition (which we use when choosing + * which dimension to split). + */ + update_bucket_ndistinct(bucket, attrs, stats); + + /* Update ndistinct (and also set min/max) for all dimensions. */ + for (i = 0; i < numattrs; i++) + update_dimension_ndistinct(bucket, i, attrs, stats, true); + + return bucket; +} + +/* + * Choose the bucket to partition next. + * + * The current criteria is rather simple, chosen so that the algorithm + * produces buckets with about equal frequency and regular size. We + * select the bucket with the highest number of distinct values, and + * then split it by the longest dimension. + * + * The distinct values are uniformly mapped to [0,1] interval, and this + * is used to compute length of the value range. + * + * NOTE: This is not the same array used for deduplication, as this + * contains values for all the tuples from the sample, not just + * the boundary values. + * + * Returns either pointer to the bucket selected to be partitioned, + * or NULL if there are no buckets that may be split (i.e. all buckets + * contain a single distinct value). + * + * TODO Consider other partitioning criteria (v-optimal, maxdiff etc.). + * For example use the "bucket volume" (product of dimension + * lengths) to select the bucket. + * + * TODO Allowing the bucket to degenerate to a single combination of + * values makes it rather strange MCV list. Maybe we should use + * higher lower boundary, or maybe make the selection criteria + * more complex (e.g. consider number of rows in the bucket, etc.). + * + * That however is different from buckets 'degenerated' only for + * some dimensions (e.g. half of them), which is perfectly + * appropriate for statistics on a combination of low and high + * cardinality columns. + */ +static MVBucket +select_bucket_to_partition(int nbuckets, MVBucket * buckets) +{ + int i; + int numrows = 0; + MVBucket bucket = NULL; + + for (i = 0; i < nbuckets; i++) + { + HistogramBuild data = (HistogramBuild)buckets[i]->build_data; + /* if the number of rows is higher, use this bucket */ + if ((data->ndistinct > 2) && + (data->numrows > numrows) && + (data->numrows >= MIN_BUCKET_ROWS)) { + bucket = buckets[i]; + numrows = data->numrows; + } + } + + /* may be NULL if there are not buckets with (ndistinct>1) */ + return bucket; +} + +/* + * A simple bucket partitioning implementation - we choose the longest + * bucket dimension, measured using the array of distinct values built + * at the very beginning of the build. + * + * We map all the distinct values to a [0,1] interval, uniformly + * distributed, and then use this to measure length. It's essentially + * a number of distinct values within the range, normalized to [0,1]. + * + * Then we choose a 'middle' value splitting the bucket into two parts + * with roughly the same frequency. + * + * This splits the bucket by tweaking the existing one, and returning + * the new bucket (essentially shrinking the existing one in-place and + * returning the other "half" as a new bucket). The caller is responsible + * for adding the new bucket into the list of buckets. + * + * There are multiple histogram options, centered around the partitioning + * criteria, specifying both how to choose a bucket and the dimension + * most in need of a split. For a nice summary and general overview, see + * "rK-Hist : an R-Tree based histogram for multi-dimensional selectivity + * estimation" thesis by J. A. Lopez, Concordia University, p.34-37 (and + * possibly p. 32-34 for explanation of the terms). + * + * TODO It requires care to prevent splitting only one dimension and not + * splitting another one at all (which might happen easily in case + * of strongly dependent columns - e.g. y=x). The current algorithm + * minimizes this, but may still happen for perfectly dependent + * examples (when all the dimensions have equal length, the first + * one will be selected). + * + * TODO Should probably consider statistics target for the columns (e.g. + * to split dimensions with higher statistics target more frequently). + */ +static MVBucket +partition_bucket(MVBucket bucket, int2vector *attrs, + VacAttrStats **stats, + int *ndistvalues, Datum **distvalues) +{ + int i; + int dimension; + int numattrs = attrs->dim1; + + Datum split_value; + MVBucket new_bucket; + HistogramBuild new_data; + + /* needed for sort, when looking for the split value */ + bool isNull; + int nvalues = 0; + HistogramBuild data = (HistogramBuild)bucket->build_data; + StdAnalyzeData * mystats = NULL; + ScalarItem * values = (ScalarItem*)palloc0(data->numrows * sizeof(ScalarItem)); + SortSupportData ssup; + + /* looking for the split value */ + // int ndistinct = 1; /* number of distinct values below current value */ + int nrows = 1; /* number of rows below current value */ + double delta; + + /* needed when splitting the values */ + HeapTuple * oldrows = data->rows; + int oldnrows = data->numrows; + + /* + * We can't split buckets with a single distinct value (this also + * disqualifies NULL-only dimensions). Also, there has to be multiple + * sample rows (otherwise, how could there be more distinct values). + */ + Assert(data->ndistinct > 1); + Assert(data->numrows > 1); + Assert((numattrs >= 2) && (numattrs <= MVSTATS_MAX_DIMENSIONS)); + + /* + * Look for the next dimension to split. + */ + delta = 0.0; + dimension = -1; + + for (i = 0; i < numattrs; i++) + { + Datum *a, *b; + + mystats = (StdAnalyzeData *) stats[i]->extra_data; + + /* initialize sort support, etc. */ + memset(&ssup, 0, sizeof(ssup)); + ssup.ssup_cxt = CurrentMemoryContext; + + /* We always use the default collation for statistics */ + ssup.ssup_collation = DEFAULT_COLLATION_OID; + ssup.ssup_nulls_first = false; + + PrepareSortSupportFromOrderingOp(mystats->ltopr, &ssup); + + /* can't split NULL-only dimension */ + if (bucket->nullsonly[i]) + continue; + + /* can't split dimension with a single ndistinct value */ + if (data->ndistincts[i] <= 1) + continue; + + /* sort support for the bsearch_comparator */ + ssup_private = &ssup; + + /* search for min boundary in the distinct list */ + a = (Datum*)bsearch(&bucket->min[i], + distvalues[i], ndistvalues[i], + sizeof(Datum), bsearch_comparator); + + b = (Datum*)bsearch(&bucket->max[i], + distvalues[i], ndistvalues[i], + sizeof(Datum), bsearch_comparator); + + /* if this dimension is 'larger' then partition by it */ + if (((b-a)*1.0 / ndistvalues[i]) > delta) + { + delta = ((b-a)*1.0 / ndistvalues[i]); + dimension = i; + } + } + + /* + * If we haven't found a dimension here, we've done something + * wrong in select_bucket_to_partition. + */ + Assert(dimension != -1); + + /* + * Walk through the selected dimension, collect and sort the values + * and then choose the value to use as the new boundary. + */ + mystats = (StdAnalyzeData *) stats[dimension]->extra_data; + + /* initialize sort support, etc. */ + memset(&ssup, 0, sizeof(ssup)); + ssup.ssup_cxt = CurrentMemoryContext; + + /* We always use the default collation for statistics */ + ssup.ssup_collation = DEFAULT_COLLATION_OID; + ssup.ssup_nulls_first = false; + + PrepareSortSupportFromOrderingOp(mystats->ltopr, &ssup); + + for (i = 0; i < data->numrows; i++) + { + /* remember the index of the sample row, to make the partitioning simpler */ + values[nvalues].value = heap_getattr(data->rows[i], attrs->values[dimension], + stats[dimension]->tupDesc, &isNull); + values[nvalues].tupno = i; + + /* no NULL values allowed here (we don't do splits by null-only dimensions) */ + Assert(!isNull); + + nvalues++; + } + + /* sort the array of values */ + qsort_arg((void *) values, nvalues, sizeof(ScalarItem), + compare_scalars_partition, (void *) &ssup); + + /* + * We know there are bucket->ndistincts[dimension] distinct values + * in this dimension, and we want to split this into half, so walk + * through the array and stop once we see (ndistinct/2) values. + * + * We always choose the "next" value, i.e. (n/2+1)-th distinct value, + * and use it as an exclusive upper boundary (and inclusive lower + * boundary). + * + * TODO Maybe we should use "average" of the two middle distinct + * values (at least for even distinct counts), but that would + * require being able to do an average (which does not work + * for non-arithmetic types). + * + * TODO Another option is to look for a split that'd give about + * 50% tuples (not distinct values) in each partition. That + * might work better when there are a few very frequent + * values, and many rare ones. + */ + delta = fabs(data->numrows); + split_value = values[0].value; + + for (i = 1; i < data->numrows; i++) + { + if (values[i].value != values[i-1].value) + { + /* are we closer to splitting the bucket in half? */ + if (fabs(i - data->numrows/2.0) < delta) + { + /* let's assume we'll use this value for the split */ + split_value = values[i].value; + delta = fabs(i - data->numrows/2.0); + nrows = i; + } + } + } + + Assert(nrows > 0); + Assert(nrows < data->numrows); + + /* create the new bucket as a (incomplete) copy of the one being partitioned. */ + new_bucket = copy_mv_bucket(bucket, numattrs); + new_data = (HistogramBuild)new_bucket->build_data; + + /* + * Do the actual split of the chosen dimension, using the split value as the + * upper bound for the existing bucket, and lower bound for the new one. + */ + bucket->max[dimension] = split_value; + new_bucket->min[dimension] = split_value; + + bucket->max_inclusive[dimension] = false; + new_bucket->max_inclusive[dimension] = true; + + /* + * Redistribute the sample tuples using the 'ScalarItem->tupno' + * index. We know 'nrows' rows should remain in the original + * bucket and the rest goes to the new one. + */ + + data->rows = (HeapTuple*)palloc0(nrows * sizeof(HeapTuple)); + new_data->rows = (HeapTuple*)palloc0((oldnrows - nrows) * sizeof(HeapTuple)); + + data->numrows = nrows; + new_data->numrows = (oldnrows - nrows); + + /* + * The first nrows should go to the first bucket, the rest should + * go to the new one. Use the tupno field to get the actual HeapTuple + * row from the original array of sample rows. + */ + for (i = 0; i < nrows; i++) + memcpy(&data->rows[i], &oldrows[values[i].tupno], sizeof(HeapTuple)); + + for (i = nrows; i < oldnrows; i++) + memcpy(&new_data->rows[i-nrows], &oldrows[values[i].tupno], sizeof(HeapTuple)); + + /* update ndistinct values for the buckets (total and per dimension) */ + update_bucket_ndistinct(bucket, attrs, stats); + update_bucket_ndistinct(new_bucket, attrs, stats); + + /* + * TODO We don't need to do this for the dimension we used for split, + * because we know how many distinct values went to each partition. + */ + for (i = 0; i < numattrs; i++) + { + update_dimension_ndistinct(bucket, i, attrs, stats, false); + update_dimension_ndistinct(new_bucket, i, attrs, stats, false); + } + + pfree(oldrows); + pfree(values); + + return new_bucket; +} + +/* + * Copy a histogram bucket. The copy does not include the build-time + * data, i.e. sampled rows etc. + */ +static MVBucket +copy_mv_bucket(MVBucket bucket, uint32 ndimensions) +{ + /* TODO allocate as a single piece (including all the fields) */ + MVBucket new_bucket = (MVBucket)palloc0(sizeof(MVBucketData)); + HistogramBuild data = (HistogramBuild)palloc0(sizeof(HistogramBuildData)); + + /* Copy only the attributes that will stay the same after the split, and + * we'll recompute the rest after the split. */ + + /* allocate the per-dimension arrays */ + new_bucket->nullsonly = (bool*)palloc0(ndimensions * sizeof(bool)); + + /* inclusiveness boundaries - lower/upper bounds */ + new_bucket->min_inclusive = (bool*)palloc0(ndimensions * sizeof(bool)); + new_bucket->max_inclusive = (bool*)palloc0(ndimensions * sizeof(bool)); + + /* lower/upper boundaries */ + new_bucket->min = (Datum*)palloc0(ndimensions * sizeof(Datum)); + new_bucket->max = (Datum*)palloc0(ndimensions * sizeof(Datum)); + + /* copy data */ + memcpy(new_bucket->nullsonly, bucket->nullsonly, ndimensions * sizeof(bool)); + + memcpy(new_bucket->min_inclusive, bucket->min_inclusive, ndimensions*sizeof(bool)); + memcpy(new_bucket->min, bucket->min, ndimensions*sizeof(Datum)); + + memcpy(new_bucket->max_inclusive, bucket->max_inclusive, ndimensions*sizeof(bool)); + memcpy(new_bucket->max, bucket->max, ndimensions*sizeof(Datum)); + + /* allocate and copy the interesting part of the build data */ + data->ndistincts = (uint32*)palloc0(ndimensions * sizeof(uint32)); + + new_bucket->build_data = data; + + return new_bucket; +} + +/* + * Counts the number of distinct values in the bucket. This just copies + * the Datum values into a simple array, and sorts them using memcmp-based + * comparator. That means it only works for pass-by-value data types + * (assuming they don't use collations etc.) + * + * TODO This might evaluate and store the distinct counts for all + * possible attribute combinations. The assumption is this might be + * useful for estimating things like GROUP BY cardinalities (e.g. + * in cases when some buckets contain a lot of low-frequency + * combinations, and other buckets contain few high-frequency ones). + * + * But it's unclear whether it's worth the price. Computing this + * is actually quite cheap, because it may be evaluated at the very + * end, when the buckets are rather small (so sorting it in 2^N ways + * is not a big deal). Assuming the partitioning algorithm does not + * use these values to do the decisions, of course (the current + * algorithm does not). + * + * The overhead with storing, fetching and parsing the data is more + * concerning - adding 2^N values per bucket (even if it's just + * a 1B or 2B value) would significantly bloat the histogram, and + * thus the impact on optimizer. Which is not really desirable. + * + * TODO This only updates the ndistinct for the sample (or bucket), but + * we eventually need an estimate of the total number of distinct + * values in the dataset. It's possible to either use the current + * 1D approach (i.e., if it's more than 10% of the sample, assume + * it's proportional to the number of rows). Or it's possible to + * implement the estimator suggested in the article, supposedly + * giving 'optimal' estimates (w.r.t. probability of error). + */ +static void +update_bucket_ndistinct(MVBucket bucket, int2vector *attrs, VacAttrStats ** stats) +{ + int i, j; + int numattrs = attrs->dim1; + + HistogramBuild data = (HistogramBuild)bucket->build_data; + int numrows = data->numrows; + + MultiSortSupport mss = multi_sort_init(numattrs); + + /* + * We could collect this while walking through all the attributes + * above (this way we have to call heap_getattr twice). + */ + SortItem *items = (SortItem*)palloc0(numrows * sizeof(SortItem)); + Datum *values = (Datum*)palloc0(numrows * sizeof(Datum) * numattrs); + bool *isnull = (bool*)palloc0(numrows * sizeof(bool) * numattrs); + + for (i = 0; i < numrows; i++) + { + items[i].values = &values[i * numattrs]; + items[i].isnull = &isnull[i * numattrs]; + } + + /* prepare the sort function for the first dimension */ + for (i = 0; i < numattrs; i++) + multi_sort_add_dimension(mss, i, i, stats); + + /* collect the values */ + for (i = 0; i < numrows; i++) + for (j = 0; j < numattrs; j++) + items[i].values[j] + = heap_getattr(data->rows[i], attrs->values[j], + stats[j]->tupDesc, &items[i].isnull[j]); + + qsort_arg((void *) items, numrows, sizeof(SortItem), + multi_sort_compare, mss); + + data->ndistinct = 1; + + for (i = 1; i < numrows; i++) + if (multi_sort_compare(&items[i], &items[i-1], mss) != 0) + data->ndistinct += 1; + + pfree(items); + pfree(values); + pfree(isnull); +} + +/* + * Count distinct values per bucket dimension. + */ +static void +update_dimension_ndistinct(MVBucket bucket, int dimension, int2vector *attrs, + VacAttrStats ** stats, bool update_boundaries) +{ + int j; + int nvalues = 0; + bool isNull; + HistogramBuild data = (HistogramBuild)bucket->build_data; + Datum * values = (Datum*)palloc0(data->numrows * sizeof(Datum)); + SortSupportData ssup; + + StdAnalyzeData * mystats = (StdAnalyzeData *) stats[dimension]->extra_data; + + /* we may already know this is a NULL-only dimension */ + if (bucket->nullsonly[dimension]) + data->ndistincts[dimension] = 1; + + memset(&ssup, 0, sizeof(ssup)); + ssup.ssup_cxt = CurrentMemoryContext; + + /* We always use the default collation for statistics */ + ssup.ssup_collation = DEFAULT_COLLATION_OID; + ssup.ssup_nulls_first = false; + + PrepareSortSupportFromOrderingOp(mystats->ltopr, &ssup); + + for (j = 0; j < data->numrows; j++) + { + values[nvalues] = heap_getattr(data->rows[j], attrs->values[dimension], + stats[dimension]->tupDesc, &isNull); + + /* ignore NULL values */ + if (! isNull) + nvalues++; + } + + /* there's always at least 1 distinct value (may be NULL) */ + data->ndistincts[dimension] = 1; + + /* if there are only NULL values in the column, mark it so and continue + * with the next one */ + if (nvalues == 0) + { + pfree(values); + bucket->nullsonly[dimension] = true; + return; + } + + /* sort the array (pass-by-value datum */ + qsort_arg((void *) values, nvalues, sizeof(Datum), + compare_scalars_simple, (void *) &ssup); + + /* + * Update min/max boundaries to the smallest bounding box. Generally, this + * needs to be done only when constructing the initial bucket. + */ + if (update_boundaries) + { + /* store the min/max values */ + bucket->min[dimension] = values[0]; + bucket->min_inclusive[dimension] = true; + + bucket->max[dimension] = values[nvalues-1]; + bucket->max_inclusive[dimension] = true; + } + + /* + * Walk through the array and count distinct values by comparing + * succeeding values. + * + * FIXME This only works for pass-by-value types (i.e. not VARCHARs + * etc.). Although thanks to the deduplication it might work + * even for those types (equal values will get the same item + * in the deduplicated array). + */ + for (j = 1; j < nvalues; j++) { + if (values[j] != values[j-1]) + data->ndistincts[dimension] += 1; + } + + pfree(values); +} + +/* + * A properly built histogram must not contain buckets mixing NULL and + * non-NULL values in a single dimension. Each dimension may either be + * marked as 'nulls only', and thus containing only NULL values, or + * it must not contain any NULL values. + * + * Therefore, if the sample contains NULL values in any of the columns, + * it's necessary to build those NULL-buckets. This is done in an + * iterative way using this algorithm, operating on a single bucket: + * + * (1) Check that all dimensions are well-formed (not mixing NULL + * and non-NULL values). + * + * (2) If all dimensions are well-formed, terminate. + * + * (3) If the dimension contains only NULL values, but is not + * marked as NULL-only, mark it as NULL-only and run the + * algorithm again (on this bucket). + * + * (4) If the dimension mixes NULL and non-NULL values, split the + * bucket into two parts - one with NULL values, one with + * non-NULL values (replacing the current one). Then run + * the algorithm on both buckets. + * + * This is executed in a recursive manner, but the number of executions + * should be quite low - limited by the number of NULL-buckets. Also, + * in each branch the number of nested calls is limited by the number + * of dimensions (attributes) of the histogram. + * + * At the end, there should be buckets with no mixed dimensions. The + * number of buckets produced by this algorithm is rather limited - with + * N dimensions, there may be only 2^N such buckets (each dimension may + * be either NULL or non-NULL). So with 8 dimensions (current value of + * MVSTATS_MAX_DIMENSIONS) there may be only 256 such buckets. + * + * After this, a 'regular' bucket-split algorithm shall run, further + * optimizing the histogram. + */ +static void +create_null_buckets(MVHistogram histogram, int bucket_idx, + int2vector *attrs, VacAttrStats ** stats) +{ + int i, j; + int null_dim = -1; + int null_count = 0; + bool null_found = false; + MVBucket bucket, null_bucket; + int null_idx, curr_idx; + HistogramBuild data, null_data; + + /* remember original values from the bucket */ + int numrows; + HeapTuple *oldrows = NULL; + + Assert(bucket_idx < histogram->nbuckets); + Assert(histogram->ndimensions == attrs->dim1); + + bucket = histogram->buckets[bucket_idx]; + data = (HistogramBuild)bucket->build_data; + + numrows = data->numrows; + oldrows = data->rows; + + /* + * Walk through all rows / dimensions, and stop once we find NULL + * in a dimension not yet marked as NULL-only. + */ + for (i = 0; i < data->numrows; i++) + { + for (j = 0; j < histogram->ndimensions; j++) + { + /* Is this a NULL-only dimension? If yes, skip. */ + if (bucket->nullsonly[j]) + continue; + + /* found a NULL in that dimension? */ + if (heap_attisnull(data->rows[i], attrs->values[j])) + { + null_found = true; + null_dim = j; + break; + } + } + + /* terminate if we found attribute with NULL values */ + if (null_found) + break; + } + + /* no regular dimension contains NULL values => we're done */ + if (! null_found) + return; + + /* walk through the rows again, count NULL values in 'null_dim' */ + for (i = 0; i < data->numrows; i++) + { + if (heap_attisnull(data->rows[i], attrs->values[null_dim])) + null_count += 1; + } + + Assert(null_count <= data->numrows); + + /* + * If (null_count == numrows) the dimension already is NULL-only, + * but is not yet marked like that. It's enough to mark it and + * repeat the process recursively (until we run out of dimensions). + */ + if (null_count == data->numrows) + { + bucket->nullsonly[null_dim] = true; + create_null_buckets(histogram, bucket_idx, attrs, stats); + return; + } + + /* + * We have to split the bucket into two - one with NULL values in + * the dimension, one with non-NULL values. We don't need to sort + * the data or anything, but otherwise it's similar to what's done + * in partition_bucket(). + */ + + /* create bucket with NULL-only dimension 'dim' */ + null_bucket = copy_mv_bucket(bucket, histogram->ndimensions); + null_data = (HistogramBuild)null_bucket->build_data; + + /* remember the current array info */ + oldrows = data->rows; + numrows = data->numrows; + + /* we'll keep non-NULL values in the current bucket */ + data->numrows = (numrows - null_count); + data->rows + = (HeapTuple*)palloc0(data->numrows * sizeof(HeapTuple)); + + /* and the NULL values will go to the new one */ + null_data->numrows = null_count; + null_data->rows + = (HeapTuple*)palloc0(null_data->numrows * sizeof(HeapTuple)); + + /* mark the dimension as NULL-only (in the new bucket) */ + null_bucket->nullsonly[null_dim] = true; + + /* walk through the sample rows and distribute them accordingly */ + null_idx = 0; + curr_idx = 0; + for (i = 0; i < numrows; i++) + { + if (heap_attisnull(oldrows[i], attrs->values[null_dim])) + /* NULL => copy to the new bucket */ + memcpy(&null_data->rows[null_idx++], &oldrows[i], + sizeof(HeapTuple)); + else + memcpy(&data->rows[curr_idx++], &oldrows[i], + sizeof(HeapTuple)); + } + + /* update ndistinct values for the buckets (total and per dimension) */ + update_bucket_ndistinct(bucket, attrs, stats); + update_bucket_ndistinct(null_bucket, attrs, stats); + + /* + * TODO We don't need to do this for the dimension we used for split, + * because we know how many distinct values went to each + * bucket (NULL is not a value, so 0, and the other bucket got + * all the ndistinct values). + */ + for (i = 0; i < histogram->ndimensions; i++) + { + update_dimension_ndistinct(bucket, i, attrs, stats, false); + update_dimension_ndistinct(null_bucket, i, attrs, stats, false); + } + + pfree(oldrows); + + /* add the NULL bucket to the histogram */ + histogram->buckets[histogram->nbuckets++] = null_bucket; + + /* + * And now run the function recursively on both buckets (the new + * one first, because the call may change number of buckets, and + * it's used as an index). + */ + create_null_buckets(histogram, (histogram->nbuckets-1), attrs, stats); + create_null_buckets(histogram, bucket_idx, attrs, stats); + +} + +/* + * We need to pass the SortSupport to the comparator, but bsearch() + * has no 'context' parameter, so we use a global variable (ugly). + */ +static int +bsearch_comparator(const void * a, const void * b) +{ + Assert(ssup_private != NULL); + return compare_scalars_simple(a, b, (void*)ssup_private); +} + +/* + * SRF with details about buckets of a histogram: + * + * - bucket ID (0...nbuckets) + * - min values (string array) + * - max values (string array) + * - nulls only (boolean array) + * - min inclusive flags (boolean array) + * - max inclusive flags (boolean array) + * - frequency (double precision) + * + * The input is the OID of the statistics, and there are no rows + * returned if the statistics contains no histogram (or if there's no + * statistics for the OID). + * + * The second parameter (type) determines what values will be returned + * in the (minvals,maxvals). There are three possible values: + * + * 0 (actual values) + * ----------------- + * - prints actual values + * - using the output function of the data type (as string) + * - handy for investigating the histogram + * + * 1 (distinct index) + * ------------------ + * - prints index of the distinct value (into the serialized array) + * - makes it easier to spot neighbor buckets, etc. + * - handy for plotting the histogram + * + * 2 (normalized distinct index) + * ----------------------------- + * - prints index of the distinct value, but normalized into [0,1] + * - similar to 1, but shows how 'long' the bucket range is + * - handy for plotting the histogram + * + * When plotting the histogram, be careful as the (1) and (2) options + * skew the lengths by distributing the distinct values uniformly. For + * data types without a clear meaning of 'distance' (e.g. strings) that + * is not a big deal, but for numbers it may be confusing. + */ +PG_FUNCTION_INFO_V1(pg_mv_histogram_buckets); + +Datum +pg_mv_histogram_buckets(PG_FUNCTION_ARGS) +{ + FuncCallContext *funcctx; + int call_cntr; + int max_calls; + TupleDesc tupdesc; + AttInMetadata *attinmeta; + + Oid mvoid = PG_GETARG_OID(0); + int otype = PG_GETARG_INT32(1); + + if ((otype < 0) || (otype > 2)) + elog(ERROR, "invalid output type specified"); + + /* stuff done only on the first call of the function */ + if (SRF_IS_FIRSTCALL()) + { + MemoryContext oldcontext; + MVSerializedHistogram histogram; + + /* create a function context for cross-call persistence */ + funcctx = SRF_FIRSTCALL_INIT(); + + /* switch to memory context appropriate for multiple function calls */ + oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx); + + histogram = load_mv_histogram(mvoid); + + funcctx->user_fctx = histogram; + + /* total number of tuples to be returned */ + funcctx->max_calls = 0; + if (funcctx->user_fctx != NULL) + funcctx->max_calls = histogram->nbuckets; + + /* Build a tuple descriptor for our result type */ + if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("function returning record called in context " + "that cannot accept type record"))); + + /* + * generate attribute metadata needed later to produce tuples + * from raw C strings + */ + attinmeta = TupleDescGetAttInMetadata(tupdesc); + funcctx->attinmeta = attinmeta; + + MemoryContextSwitchTo(oldcontext); + } + + /* stuff done on every call of the function */ + funcctx = SRF_PERCALL_SETUP(); + + call_cntr = funcctx->call_cntr; + max_calls = funcctx->max_calls; + attinmeta = funcctx->attinmeta; + + if (call_cntr < max_calls) /* do when there is more left to send */ + { + char **values; + HeapTuple tuple; + Datum result; + int2vector *stakeys; + Oid relid; + double bucket_size = 1.0; + + char *buff = palloc0(1024); + char *format; + + int i; + + Oid *outfuncs; + FmgrInfo *fmgrinfo; + + MVSerializedHistogram histogram; + MVSerializedBucket bucket; + + histogram = (MVSerializedHistogram)funcctx->user_fctx; + + Assert(call_cntr < histogram->nbuckets); + + bucket = histogram->buckets[call_cntr]; + + stakeys = find_mv_attnums(mvoid, &relid); + + /* + * Prepare a values array for building the returned tuple. + * This should be an array of C strings which will + * be processed later by the type input functions. + */ + values = (char **) palloc(9 * sizeof(char *)); + + values[0] = (char *) palloc(64 * sizeof(char)); + + /* arrays */ + values[1] = (char *) palloc0(1024 * sizeof(char)); + values[2] = (char *) palloc0(1024 * sizeof(char)); + values[3] = (char *) palloc0(1024 * sizeof(char)); + values[4] = (char *) palloc0(1024 * sizeof(char)); + values[5] = (char *) palloc0(1024 * sizeof(char)); + + values[6] = (char *) palloc(64 * sizeof(char)); + values[7] = (char *) palloc(64 * sizeof(char)); + values[8] = (char *) palloc(64 * sizeof(char)); + + /* we need to do this only when printing the actual values */ + outfuncs = (Oid*)palloc0(sizeof(Oid) * histogram->ndimensions); + fmgrinfo = (FmgrInfo*)palloc0(sizeof(FmgrInfo) * histogram->ndimensions); + + for (i = 0; i < histogram->ndimensions; i++) + { + bool isvarlena; + + getTypeOutputInfo(get_atttype(relid, stakeys->values[i]), + &outfuncs[i], &isvarlena); + + fmgr_info(outfuncs[i], &fmgrinfo[i]); + } + + snprintf(values[0], 64, "%d", call_cntr); /* bucket ID */ + + /* + * currently we only print array of indexes, but the deduplicated + * values should be sorted, so this is actually quite useful + * + * TODO print the actual min/max values, using the output + * function of the attribute type + */ + + for (i = 0; i < histogram->ndimensions; i++) + { + bucket_size *= (bucket->max[i] - bucket->min[i]) * 1.0 + / (histogram->nvalues[i]-1); + + /* print the actual values, i.e. use output function etc. */ + if (otype == 0) + { + Datum minval, maxval; + Datum minout, maxout; + + format = "%s, %s"; + if (i == 0) + format = "{%s%s"; + else if (i == histogram->ndimensions-1) + format = "%s, %s}"; + + minval = histogram->values[i][bucket->min[i]]; + minout = FunctionCall1(&fmgrinfo[i], minval); + + maxval = histogram->values[i][bucket->max[i]]; + maxout = FunctionCall1(&fmgrinfo[i], maxval); + + // snprintf(buff, 1024, format, values[1], bucket->min[i]); + snprintf(buff, 1024, format, values[1], DatumGetPointer(minout)); + strncpy(values[1], buff, 1023); + buff[0] = '\0'; + + // snprintf(buff, 1024, format, values[2], bucket->max[i]); + snprintf(buff, 1024, format, values[2], DatumGetPointer(maxout)); + strncpy(values[2], buff, 1023); + buff[0] = '\0'; + } + else if (otype == 1) + { + format = "%s, %d"; + if (i == 0) + format = "{%s%d"; + else if (i == histogram->ndimensions-1) + format = "%s, %d}"; + + snprintf(buff, 1024, format, values[1], bucket->min[i]); + strncpy(values[1], buff, 1023); + buff[0] = '\0'; + + snprintf(buff, 1024, format, values[2], bucket->max[i]); + strncpy(values[2], buff, 1023); + buff[0] = '\0'; + } + else + { + format = "%s, %f"; + if (i == 0) + format = "{%s%f"; + else if (i == histogram->ndimensions-1) + format = "%s, %f}"; + + snprintf(buff, 1024, format, values[1], + bucket->min[i] * 1.0 / (histogram->nvalues[i]-1)); + strncpy(values[1], buff, 1023); + buff[0] = '\0'; + + snprintf(buff, 1024, format, values[2], + bucket->max[i] * 1.0 / (histogram->nvalues[i]-1)); + strncpy(values[2], buff, 1023); + buff[0] = '\0'; + } + + format = "%s, %s"; + if (i == 0) + format = "{%s%s"; + else if (i == histogram->ndimensions-1) + format = "%s, %s}"; + + snprintf(buff, 1024, format, values[3], bucket->nullsonly[i] ? "t" : "f"); + strncpy(values[3], buff, 1023); + buff[0] = '\0'; + + snprintf(buff, 1024, format, values[4], bucket->min_inclusive[i] ? "t" : "f"); + strncpy(values[4], buff, 1023); + buff[0] = '\0'; + + snprintf(buff, 1024, format, values[5], bucket->max_inclusive[i] ? "t" : "f"); + strncpy(values[5], buff, 1023); + buff[0] = '\0'; + } + + snprintf(values[6], 64, "%f", bucket->ntuples); /* frequency */ + snprintf(values[7], 64, "%f", bucket->ntuples / bucket_size); /* density */ + snprintf(values[8], 64, "%f", bucket_size); /* bucket_size */ + + /* build a tuple */ + tuple = BuildTupleFromCStrings(attinmeta, values); + + /* make the tuple into a datum */ + result = HeapTupleGetDatum(tuple); + + /* clean up (this is not really necessary) */ + pfree(values[0]); + pfree(values[1]); + pfree(values[2]); + pfree(values[3]); + pfree(values[4]); + pfree(values[5]); + pfree(values[6]); + + pfree(values); + + SRF_RETURN_NEXT(funcctx, result); + } + else /* do when there is no more left */ + { + SRF_RETURN_DONE(funcctx); + } +} diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 01d29db..af3bd62 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -2101,9 +2101,9 @@ describeOneTableDetails(const char *schemaname, { printfPQExpBuffer(&buf, "SELECT oid, stakeys,\n" - " deps_enabled, mcv_enabled,\n" - " deps_built, mcv_built,\n" - " mcv_max_items,\n" + " deps_enabled, mcv_enabled, hist_enabled,\n" + " deps_built, mcv_built, hist_built,\n" + " mcv_max_items, hist_max_buckets,\n" " (SELECT string_agg(attname::text,', ')\n" " FROM ((SELECT unnest(stakeys) AS attnum) s\n" " JOIN pg_attribute a ON (starelid = a.attrelid and a.attnum = s.attnum))) AS attnums\n" @@ -2141,8 +2141,17 @@ describeOneTableDetails(const char *schemaname, first = false; } + if (!strcmp(PQgetvalue(result, i, 4), "t")) + { + if (! first) + appendPQExpBuffer(&buf, ", histogram"); + else + appendPQExpBuffer(&buf, "(histogram"); + first = false; + } + appendPQExpBuffer(&buf, ") ON (%s)", - PQgetvalue(result, i, 8)); + PQgetvalue(result, i, 10)); printTableAddFooter(&cont, buf.data); } diff --git a/src/include/catalog/pg_mv_statistic.h b/src/include/catalog/pg_mv_statistic.h index c6e7d74..84579da 100644 --- a/src/include/catalog/pg_mv_statistic.h +++ b/src/include/catalog/pg_mv_statistic.h @@ -36,13 +36,16 @@ CATALOG(pg_mv_statistic,3381) /* statistics requested to build */ bool deps_enabled; /* analyze dependencies? */ bool mcv_enabled; /* build MCV list? */ + bool hist_enabled; /* build histogram? */ - /* MCV size */ + /* histogram / MCV size */ int32 mcv_max_items; /* max MCV items */ + int32 hist_max_buckets; /* max histogram buckets */ /* statistics that are available (if requested) */ bool deps_built; /* dependencies were built */ bool mcv_built; /* MCV list was built */ + bool hist_built; /* histogram was built */ /* variable-length fields start here, but we allow direct access to stakeys */ int2vector stakeys; /* array of column keys */ @@ -50,6 +53,7 @@ CATALOG(pg_mv_statistic,3381) #ifdef CATALOG_VARLEN bytea stadeps; /* dependencies (serialized) */ bytea stamcv; /* MCV list (serialized) */ + bytea stahist; /* MV histogram (serialized) */ #endif } FormData_pg_mv_statistic; @@ -65,15 +69,19 @@ typedef FormData_pg_mv_statistic *Form_pg_mv_statistic; * compiler constants for pg_attrdef * ---------------- */ -#define Natts_pg_mv_statistic 9 +#define Natts_pg_mv_statistic 13 #define Anum_pg_mv_statistic_starelid 1 #define Anum_pg_mv_statistic_deps_enabled 2 #define Anum_pg_mv_statistic_mcv_enabled 3 -#define Anum_pg_mv_statistic_mcv_max_items 4 -#define Anum_pg_mv_statistic_deps_built 5 -#define Anum_pg_mv_statistic_mcv_built 6 -#define Anum_pg_mv_statistic_stakeys 7 -#define Anum_pg_mv_statistic_stadeps 8 -#define Anum_pg_mv_statistic_stamcv 9 +#define Anum_pg_mv_statistic_hist_enabled 4 +#define Anum_pg_mv_statistic_mcv_max_items 5 +#define Anum_pg_mv_statistic_hist_max_buckets 6 +#define Anum_pg_mv_statistic_deps_built 7 +#define Anum_pg_mv_statistic_mcv_built 8 +#define Anum_pg_mv_statistic_hist_built 9 +#define Anum_pg_mv_statistic_stakeys 10 +#define Anum_pg_mv_statistic_stadeps 11 +#define Anum_pg_mv_statistic_stamcv 12 +#define Anum_pg_mv_statistic_stahist 13 #endif /* PG_MV_STATISTIC_H */ diff --git a/src/include/catalog/pg_proc.h b/src/include/catalog/pg_proc.h index 890c763..1d451f6 100644 --- a/src/include/catalog/pg_proc.h +++ b/src/include/catalog/pg_proc.h @@ -2743,6 +2743,10 @@ DATA(insert OID = 3376 ( pg_mv_stats_mcvlist_info PGNSP PGUID 12 1 0 0 0 f f f DESCR("multi-variate statistics: MCV list info"); DATA(insert OID = 3373 ( pg_mv_mcv_items PGNSP PGUID 12 1 1000 0 0 f f f f t t i 1 0 2249 "26" "{26,23,1009,1000,701}" "{i,o,o,o,o}" "{oid,index,values,nulls,frequency}" _null_ _null_ pg_mv_mcv_items _null_ _null_ _null_ )); DESCR("details about MCV list items"); +DATA(insert OID = 3375 ( pg_mv_stats_histogram_info PGNSP PGUID 12 1 0 0 0 f f f f t f i 1 0 25 "17" _null_ _null_ _null_ _null_ _null_ pg_mv_stats_histogram_info _null_ _null_ _null_ )); +DESCR("multi-variate statistics: histogram info"); +DATA(insert OID = 3374 ( pg_mv_histogram_buckets PGNSP PGUID 12 1 1000 0 0 f f f f t t i 2 0 2249 "26 23" "{26,23,23,1009,1009,1000,1000,1000,701,701,701}" "{i,i,o,o,o,o,o,o,o,o,o}" "{oid,otype,index,minvals,maxvals,nullsonly,mininclusive,maxinclusive,frequency,density,bucket_size}" _null_ _null_ pg_mv_histogram_buckets _null_ _null_ _null_ )); +DESCR("details about histogram buckets"); DATA(insert OID = 1928 ( pg_stat_get_numscans PGNSP PGUID 12 1 0 0 0 f f f f t f s 1 0 20 "26" _null_ _null_ _null_ _null_ _null_ pg_stat_get_numscans _null_ _null_ _null_ )); DESCR("statistics: number of scans done for table/index"); diff --git a/src/include/nodes/relation.h b/src/include/nodes/relation.h index 917ae8d..abf5815 100644 --- a/src/include/nodes/relation.h +++ b/src/include/nodes/relation.h @@ -573,10 +573,12 @@ typedef struct MVStatisticInfo /* enabled statistics */ bool deps_enabled; /* functional dependencies enabled */ bool mcv_enabled; /* MCV list enabled */ + bool hist_enabled; /* histogram enabled */ /* built/available statistics */ bool deps_built; /* functional dependencies built */ bool mcv_built; /* MCV list built */ + bool hist_built; /* histogram built */ /* columns in the statistics (attnums) */ int2vector *stakeys; /* attnums of the columns covered */ diff --git a/src/include/utils/mvstats.h b/src/include/utils/mvstats.h index b028192..70f79ed 100644 --- a/src/include/utils/mvstats.h +++ b/src/include/utils/mvstats.h @@ -91,6 +91,123 @@ typedef MCVListData *MCVList; #define MVSTAT_MCVLIST_MAX_ITEMS 8192 /* max items in MCV list */ /* + * Multivariate histograms + */ +typedef struct MVBucketData { + + /* Frequencies of this bucket. */ + float ntuples; /* frequency of tuples tuples */ + + /* + * Information about dimensions being NULL-only. Not yet used. + */ + bool *nullsonly; + + /* lower boundaries - values and information about the inequalities */ + Datum *min; + bool *min_inclusive; + + /* upper boundaries - values and information about the inequalities */ + Datum *max; + bool *max_inclusive; + + /* used when building the histogram (not serialized/deserialized) */ + void *build_data; + +} MVBucketData; + +typedef MVBucketData *MVBucket; + + +typedef struct MVHistogramData { + + uint32 magic; /* magic constant marker */ + uint32 type; /* type of histogram (BASIC) */ + uint32 nbuckets; /* number of buckets (buckets array) */ + uint32 ndimensions; /* number of dimensions */ + + MVBucket *buckets; /* array of buckets */ + +} MVHistogramData; + +typedef MVHistogramData *MVHistogram; + +/* + * Histogram in a partially serialized form, with deduplicated boundary + * values etc. + * + * TODO add more detailed description here + */ + +typedef struct MVSerializedBucketData { + + /* Frequencies of this bucket. */ + float ntuples; /* frequency of tuples tuples */ + + /* + * Information about dimensions being NULL-only. Not yet used. + */ + bool *nullsonly; + + /* lower boundaries - values and information about the inequalities */ + uint16 *min; + bool *min_inclusive; + + /* indexes of upper boundaries - values and information about the + * inequalities (exclusive vs. inclusive) */ + uint16 *max; + bool *max_inclusive; + +} MVSerializedBucketData; + +typedef MVSerializedBucketData *MVSerializedBucket; + +typedef struct MVSerializedHistogramData { + + uint32 magic; /* magic constant marker */ + uint32 type; /* type of histogram (BASIC) */ + uint32 nbuckets; /* number of buckets (buckets array) */ + uint32 ndimensions; /* number of dimensions */ + + /* + * keep this the same with MVHistogramData, because of + * deserialization (same offset) + */ + MVSerializedBucket *buckets; /* array of buckets */ + + /* + * serialized boundary values, one array per dimension, deduplicated + * (the min/max indexes point into these arrays) + */ + int *nvalues; + Datum **values; + +} MVSerializedHistogramData; + +typedef MVSerializedHistogramData *MVSerializedHistogram; + + +/* used to flag stats serialized to bytea */ +#define MVSTAT_HIST_MAGIC 0x7F8C5670 /* marks serialized bytea */ +#define MVSTAT_HIST_TYPE_BASIC 1 /* basic histogram type */ + +/* + * Limits used for max_buckets option, i.e. we're always guaranteed + * to have space for at least MVSTAT_HIST_MIN_BUCKETS, and we cannot + * have more than MVSTAT_HIST_MAX_BUCKETS buckets. + * + * This is just a boundary for the 'max' threshold - the actual + * histogram may use less buckets than MVSTAT_HIST_MAX_BUCKETS. + * + * TODO The MVSTAT_HIST_MIN_BUCKETS should be related to the number of + * attributes (MVSTATS_MAX_DIMENSIONS) because of NULL-buckets. + * There should be at least 2^N buckets, otherwise we may be unable + * to build the NULL buckets. + */ +#define MVSTAT_HIST_MIN_BUCKETS 128 /* min number of buckets */ +#define MVSTAT_HIST_MAX_BUCKETS 16384 /* max number of buckets */ + +/* * TODO Maybe fetching the histogram/MCV list separately is inefficient? * Consider adding a single `fetch_stats` method, fetching all * stats specified using flags (or something like that). @@ -98,20 +215,25 @@ typedef MCVListData *MCVList; MVDependencies load_mv_dependencies(Oid mvoid); MCVList load_mv_mcvlist(Oid mvoid); +MVSerializedHistogram load_mv_histogram(Oid mvoid); bytea * serialize_mv_dependencies(MVDependencies dependencies); bytea * serialize_mv_mcvlist(MCVList mcvlist, int2vector *attrs, VacAttrStats **stats); +bytea * serialize_mv_histogram(MVHistogram histogram, int2vector *attrs, + VacAttrStats **stats); /* deserialization of stats (serialization is private to analyze) */ MVDependencies deserialize_mv_dependencies(bytea * data); MCVList deserialize_mv_mcvlist(bytea * data); +MVSerializedHistogram deserialize_mv_histogram(bytea * data); /* * Returns index of the attribute number within the vector (i.e. a * dimension within the stats). */ int mv_get_index(AttrNumber varattno, int2vector * stakeys); +int2vector* find_mv_attnums(Oid mvoid, Oid *relid); int2vector* find_mv_attnums(Oid mvoid, Oid *relid); @@ -120,6 +242,8 @@ extern Datum pg_mv_stats_dependencies_info(PG_FUNCTION_ARGS); extern Datum pg_mv_stats_dependencies_show(PG_FUNCTION_ARGS); extern Datum pg_mv_stats_mcvlist_info(PG_FUNCTION_ARGS); extern Datum pg_mv_mcvlist_items(PG_FUNCTION_ARGS); +extern Datum pg_mv_stats_histogram_info(PG_FUNCTION_ARGS); +extern Datum pg_mv_histogram_buckets(PG_FUNCTION_ARGS); MVDependencies build_mv_dependencies(int numrows, HeapTuple *rows, int2vector *attrs, @@ -129,10 +253,15 @@ MCVList build_mv_mcvlist(int numrows, HeapTuple *rows, int2vector *attrs, VacAttrStats **stats, int *numrows_filtered); +MVHistogram +build_mv_histogram(int numrows, HeapTuple *rows, int2vector *attrs, + VacAttrStats **stats, int numrows_total); + void build_mv_stats(Relation onerel, int numrows, HeapTuple *rows, int natts, VacAttrStats **vacattrstats); -void update_mv_stats(Oid relid, MVDependencies dependencies, MCVList mcvlist, +void update_mv_stats(Oid relid, MVDependencies dependencies, + MCVList mcvlist, MVHistogram histogram, int2vector *attrs, VacAttrStats **stats); #endif diff --git a/src/test/regress/expected/mv_histogram.out b/src/test/regress/expected/mv_histogram.out new file mode 100644 index 0000000..a3d3fd8 --- /dev/null +++ b/src/test/regress/expected/mv_histogram.out @@ -0,0 +1,207 @@ +-- data type passed by value +CREATE TABLE mv_histogram ( + a INT, + b INT, + c INT +); +-- unknown column +ALTER TABLE mv_histogram ADD STATISTICS (histogram) ON (unknown_column); +ERROR: column "unknown_column" referenced in statistics does not exist +-- single column +ALTER TABLE mv_histogram ADD STATISTICS (histogram) ON (a); +ERROR: multivariate stats require 2 or more columns +-- single column, duplicated +ALTER TABLE mv_histogram ADD STATISTICS (histogram) ON (a, a); +ERROR: duplicate column name in statistics definition +-- two columns, one duplicated +ALTER TABLE mv_histogram ADD STATISTICS (histogram) ON (a, a, b); +ERROR: duplicate column name in statistics definition +-- unknown option +ALTER TABLE mv_histogram ADD STATISTICS (unknown_option) ON (a, b, c); +ERROR: unrecognized STATISTICS option "unknown_option" +-- missing histogram statistics +ALTER TABLE mv_histogram ADD STATISTICS (dependencies, max_buckets 200) ON (a, b, c); +ERROR: option 'histogram' is required by other options(s) +-- invalid max_buckets value / too low +ALTER TABLE mv_histogram ADD STATISTICS (mcv, max_buckets 10) ON (a, b, c); +ERROR: minimum number of buckets is 128 +-- invalid max_buckets value / too high +ALTER TABLE mv_histogram ADD STATISTICS (mcv, max_buckets 100000) ON (a, b, c); +ERROR: minimum number of buckets is 16384 +-- correct command +ALTER TABLE mv_histogram ADD STATISTICS (histogram) ON (a, b, c); +-- random data (no functional dependencies) +INSERT INTO mv_histogram + SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i); +ANALYZE mv_histogram; +SELECT hist_enabled, hist_built + FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass; + hist_enabled | hist_built +--------------+------------ + t | t +(1 row) + +TRUNCATE mv_histogram; +-- a => b, a => c, b => c +INSERT INTO mv_histogram + SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i); +ANALYZE mv_histogram; +SELECT hist_enabled, hist_built + FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass; + hist_enabled | hist_built +--------------+------------ + t | t +(1 row) + +TRUNCATE mv_histogram; +-- a => b, a => c +INSERT INTO mv_histogram + SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i); +ANALYZE mv_histogram; +SELECT hist_enabled, hist_built + FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass; + hist_enabled | hist_built +--------------+------------ + t | t +(1 row) + +TRUNCATE mv_histogram; +-- check explain (expect bitmap index scan, not plain index scan) +INSERT INTO mv_histogram + SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i); +CREATE INDEX hist_idx ON mv_histogram (a, b); +ANALYZE mv_histogram; +SELECT hist_enabled, hist_built + FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass; + hist_enabled | hist_built +--------------+------------ + t | t +(1 row) + +EXPLAIN (COSTS off) + SELECT * FROM mv_histogram WHERE a = 10 AND b = 5; + QUERY PLAN +-------------------------------------------- + Bitmap Heap Scan on mv_histogram + Recheck Cond: ((a = 10) AND (b = 5)) + -> Bitmap Index Scan on hist_idx + Index Cond: ((a = 10) AND (b = 5)) +(4 rows) + +DROP TABLE mv_histogram; +-- varlena type (text) +CREATE TABLE mv_histogram ( + a TEXT, + b TEXT, + c TEXT +); +ALTER TABLE mv_histogram ADD STATISTICS (histogram) ON (a, b, c); +-- random data (no functional dependencies) +INSERT INTO mv_histogram + SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i); +ANALYZE mv_histogram; +SELECT hist_enabled, hist_built + FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass; + hist_enabled | hist_built +--------------+------------ + t | t +(1 row) + +TRUNCATE mv_histogram; +-- a => b, a => c, b => c +INSERT INTO mv_histogram + SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i); +ANALYZE mv_histogram; +SELECT hist_enabled, hist_built + FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass; + hist_enabled | hist_built +--------------+------------ + t | t +(1 row) + +TRUNCATE mv_histogram; +-- a => b, a => c +INSERT INTO mv_histogram + SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i); +ANALYZE mv_histogram; +SELECT hist_enabled, hist_built + FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass; + hist_enabled | hist_built +--------------+------------ + t | t +(1 row) + +TRUNCATE mv_histogram; +-- check explain (expect bitmap index scan, not plain index scan) +INSERT INTO mv_histogram + SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i); +CREATE INDEX hist_idx ON mv_histogram (a, b); +ANALYZE mv_histogram; +SELECT hist_enabled, hist_built + FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass; + hist_enabled | hist_built +--------------+------------ + t | t +(1 row) + +EXPLAIN (COSTS off) + SELECT * FROM mv_histogram WHERE a = '10' AND b = '5'; + QUERY PLAN +------------------------------------------------------------ + Bitmap Heap Scan on mv_histogram + Recheck Cond: ((a = '10'::text) AND (b = '5'::text)) + -> Bitmap Index Scan on hist_idx + Index Cond: ((a = '10'::text) AND (b = '5'::text)) +(4 rows) + +TRUNCATE mv_histogram; +-- check explain (expect bitmap index scan, not plain index scan) with NULLs +INSERT INTO mv_histogram + SELECT + (CASE WHEN i/10000 = 0 THEN NULL ELSE i/10000 END), + (CASE WHEN i/20000 = 0 THEN NULL ELSE i/20000 END), + (CASE WHEN i/40000 = 0 THEN NULL ELSE i/40000 END) + FROM generate_series(1,1000000) s(i); +ANALYZE mv_histogram; +SELECT hist_enabled, hist_built + FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass; + hist_enabled | hist_built +--------------+------------ + t | t +(1 row) + +EXPLAIN (COSTS off) + SELECT * FROM mv_histogram WHERE a IS NULL AND b IS NULL; + QUERY PLAN +--------------------------------------------------- + Bitmap Heap Scan on mv_histogram + Recheck Cond: ((a IS NULL) AND (b IS NULL)) + -> Bitmap Index Scan on hist_idx + Index Cond: ((a IS NULL) AND (b IS NULL)) +(4 rows) + +DROP TABLE mv_histogram; +-- NULL values (mix of int and text columns) +CREATE TABLE mv_histogram ( + a INT, + b TEXT, + c INT, + d TEXT +); +ALTER TABLE mv_histogram ADD STATISTICS (histogram) ON (a, b, c, d); +INSERT INTO mv_histogram + SELECT + mod(i, 100), + (CASE WHEN mod(i, 200) = 0 THEN NULL ELSE mod(i,200) END), + mod(i, 400), + (CASE WHEN mod(i, 300) = 0 THEN NULL ELSE mod(i,600) END) + FROM generate_series(1,10000) s(i); +ANALYZE mv_histogram; +SELECT hist_enabled, hist_built + FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass; + hist_enabled | hist_built +--------------+------------ + t | t +(1 row) + +DROP TABLE mv_histogram; diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out index faa41c7..e230e58 100644 --- a/src/test/regress/expected/rules.out +++ b/src/test/regress/expected/rules.out @@ -1369,7 +1369,9 @@ pg_mv_stats| SELECT n.nspname AS schemaname, length(s.stadeps) AS depsbytes, pg_mv_stats_dependencies_info(s.stadeps) AS depsinfo, length(s.stamcv) AS mcvbytes, - pg_mv_stats_mcvlist_info(s.stamcv) AS mcvinfo + pg_mv_stats_mcvlist_info(s.stamcv) AS mcvinfo, + length(s.stahist) AS histbytes, + pg_mv_stats_histogram_info(s.stahist) AS histinfo FROM ((pg_mv_statistic s JOIN pg_class c ON ((c.oid = s.starelid))) LEFT JOIN pg_namespace n ON ((n.oid = c.relnamespace))); diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index d083442..8715d17 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -112,4 +112,4 @@ test: event_trigger test: stats # run tests of multivariate stats -test: mv_dependencies mv_mcv +test: mv_dependencies mv_mcv mv_histogram diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index e63b7aa..6b9ed27 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -158,3 +158,4 @@ test: stats test: tablesample test: mv_dependencies test: mv_mcv +test: mv_histogram diff --git a/src/test/regress/sql/mv_histogram.sql b/src/test/regress/sql/mv_histogram.sql new file mode 100644 index 0000000..31c627a --- /dev/null +++ b/src/test/regress/sql/mv_histogram.sql @@ -0,0 +1,176 @@ +-- data type passed by value +CREATE TABLE mv_histogram ( + a INT, + b INT, + c INT +); + +-- unknown column +ALTER TABLE mv_histogram ADD STATISTICS (histogram) ON (unknown_column); + +-- single column +ALTER TABLE mv_histogram ADD STATISTICS (histogram) ON (a); + +-- single column, duplicated +ALTER TABLE mv_histogram ADD STATISTICS (histogram) ON (a, a); + +-- two columns, one duplicated +ALTER TABLE mv_histogram ADD STATISTICS (histogram) ON (a, a, b); + +-- unknown option +ALTER TABLE mv_histogram ADD STATISTICS (unknown_option) ON (a, b, c); + +-- missing histogram statistics +ALTER TABLE mv_histogram ADD STATISTICS (dependencies, max_buckets 200) ON (a, b, c); + +-- invalid max_buckets value / too low +ALTER TABLE mv_histogram ADD STATISTICS (mcv, max_buckets 10) ON (a, b, c); + +-- invalid max_buckets value / too high +ALTER TABLE mv_histogram ADD STATISTICS (mcv, max_buckets 100000) ON (a, b, c); + +-- correct command +ALTER TABLE mv_histogram ADD STATISTICS (histogram) ON (a, b, c); + +-- random data (no functional dependencies) +INSERT INTO mv_histogram + SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i); + +ANALYZE mv_histogram; + +SELECT hist_enabled, hist_built + FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass; + +TRUNCATE mv_histogram; + +-- a => b, a => c, b => c +INSERT INTO mv_histogram + SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i); + +ANALYZE mv_histogram; + +SELECT hist_enabled, hist_built + FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass; + +TRUNCATE mv_histogram; + +-- a => b, a => c +INSERT INTO mv_histogram + SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i); +ANALYZE mv_histogram; + +SELECT hist_enabled, hist_built + FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass; + +TRUNCATE mv_histogram; + +-- check explain (expect bitmap index scan, not plain index scan) +INSERT INTO mv_histogram + SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i); +CREATE INDEX hist_idx ON mv_histogram (a, b); +ANALYZE mv_histogram; + +SELECT hist_enabled, hist_built + FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass; + +EXPLAIN (COSTS off) + SELECT * FROM mv_histogram WHERE a = 10 AND b = 5; + +DROP TABLE mv_histogram; + +-- varlena type (text) +CREATE TABLE mv_histogram ( + a TEXT, + b TEXT, + c TEXT +); + +ALTER TABLE mv_histogram ADD STATISTICS (histogram) ON (a, b, c); + +-- random data (no functional dependencies) +INSERT INTO mv_histogram + SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i); + +ANALYZE mv_histogram; + +SELECT hist_enabled, hist_built + FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass; + +TRUNCATE mv_histogram; + +-- a => b, a => c, b => c +INSERT INTO mv_histogram + SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i); + +ANALYZE mv_histogram; + +SELECT hist_enabled, hist_built + FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass; + +TRUNCATE mv_histogram; + +-- a => b, a => c +INSERT INTO mv_histogram + SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i); +ANALYZE mv_histogram; + +SELECT hist_enabled, hist_built + FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass; + +TRUNCATE mv_histogram; + +-- check explain (expect bitmap index scan, not plain index scan) +INSERT INTO mv_histogram + SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i); +CREATE INDEX hist_idx ON mv_histogram (a, b); +ANALYZE mv_histogram; + +SELECT hist_enabled, hist_built + FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass; + +EXPLAIN (COSTS off) + SELECT * FROM mv_histogram WHERE a = '10' AND b = '5'; + +TRUNCATE mv_histogram; + +-- check explain (expect bitmap index scan, not plain index scan) with NULLs +INSERT INTO mv_histogram + SELECT + (CASE WHEN i/10000 = 0 THEN NULL ELSE i/10000 END), + (CASE WHEN i/20000 = 0 THEN NULL ELSE i/20000 END), + (CASE WHEN i/40000 = 0 THEN NULL ELSE i/40000 END) + FROM generate_series(1,1000000) s(i); +ANALYZE mv_histogram; + +SELECT hist_enabled, hist_built + FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass; + +EXPLAIN (COSTS off) + SELECT * FROM mv_histogram WHERE a IS NULL AND b IS NULL; + +DROP TABLE mv_histogram; + +-- NULL values (mix of int and text columns) +CREATE TABLE mv_histogram ( + a INT, + b TEXT, + c INT, + d TEXT +); + +ALTER TABLE mv_histogram ADD STATISTICS (histogram) ON (a, b, c, d); + +INSERT INTO mv_histogram + SELECT + mod(i, 100), + (CASE WHEN mod(i, 200) = 0 THEN NULL ELSE mod(i,200) END), + mod(i, 400), + (CASE WHEN mod(i, 300) = 0 THEN NULL ELSE mod(i,600) END) + FROM generate_series(1,10000) s(i); + +ANALYZE mv_histogram; + +SELECT hist_enabled, hist_built + FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass; + +DROP TABLE mv_histogram; -- 1.9.3 --------------080401000304040708010509 Content-Type: text/x-patch; name="0006-multi-statistics-estimation-v7.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0006-multi-statistics-estimation-v7.patch" From a9df974e90067f68ea106e89a08ebc887412b5b5 Mon Sep 17 00:00:00 2001 From: Tomas Vondra Date: Fri, 6 Feb 2015 01:42:38 +0100 Subject: [PATCH 6/6] multi-statistics estimation The general idea is that a probability (which is what selectivity is) can be split into a product of conditional probabilities like this: P(A & B & C) = P(A & B) * P(C|A & B) If we assume that C and B are independent, the last part may be simplified like this P(A & B & C) = P(A & B) * P(C|A) we only need probabilities on [A,B] and [C,A] to compute the original probability. The implementation works in the other direction, though. We know what probability P(A & B & C) we need to compute, and also what statistics are available. So we search for a combinations of statistics, covering the clauses in an optimal way (most clauses covered, most dependencies exploited). There are two possible approaches - exhaustive and greedy. The exhaustive one walks through all permutations of stats using dynamic programming, so it's guaranteed to find the optimal solution, but it soon gets very slow as it's roughly O(N!). The dynamic programming may improve that a bit, but it's still far too expensive for large numbers of statistics (on a single table). The greedy algorithm is very simple - in every step choose the best solution. That may not guarantee the best solution globally (but maybe it does?), but it only needs N steps to find the solution, so it's very fast (processing the selected stats is usually way more expensive). There's a GUC for selecting the search algorithm mvstat_search = {'greedy', 'exhaustive'} The default value is 'greedy' as that's much safer (with respect to runtime). See choose_mv_statistics(). Once we have found a sequence of statistics, we apply them to the clauses using the conditional probabilities. We process the selected stats one by one, and for each we select the estimated clauses and conditions. See clauselist_selectivity() for more details. Limitations ----------- It's still true that each clause at a given level has to be covered by a single MV statistics. So with this query WHERE (clause1) AND (clause2) AND (clause3 OR clause4) each parenthesized clause has to be covered by a single multivariate statistics. Clauses not covered by a single statistics at this level will be passed to clause_selectivity() but this will treat them as a collection of simpler clauses (connected by AND or OR), and the clauses from the previous level will be used as conditions. So using the same example, the last clause will be passed to clause_selectivity() with 'clause1' and 'clause2' as conditions, and it will be processed using multivariate stats if possible. The other limitation is that all the expressions have to be mv-compatible, i.e. there can't be a mix of expressions. If this is violated, the clause may be passed to the next level (just like with list of clauses not covered by a single statistics), which splits that into clauses handled by multivariate stats and clauses handler by regular statistics. --- contrib/file_fdw/file_fdw.c | 3 +- contrib/postgres_fdw/postgres_fdw.c | 6 +- src/backend/optimizer/path/clausesel.c | 2151 +++++++++++++++++++++++++++++--- src/backend/optimizer/path/costsize.c | 23 +- src/backend/optimizer/util/orclauses.c | 4 +- src/backend/utils/adt/selfuncs.c | 17 +- src/backend/utils/misc/guc.c | 20 + src/include/optimizer/cost.h | 6 +- src/include/utils/mvstats.h | 8 + 9 files changed, 2016 insertions(+), 222 deletions(-) diff --git a/contrib/file_fdw/file_fdw.c b/contrib/file_fdw/file_fdw.c index 499f24f..0d7d2e7 100644 --- a/contrib/file_fdw/file_fdw.c +++ b/contrib/file_fdw/file_fdw.c @@ -949,7 +949,8 @@ estimate_size(PlannerInfo *root, RelOptInfo *baserel, baserel->baserestrictinfo, 0, JOIN_INNER, - NULL); + NULL, + NIL); nrows = clamp_row_est(nrows); diff --git a/contrib/postgres_fdw/postgres_fdw.c b/contrib/postgres_fdw/postgres_fdw.c index 6da01e1..bd487c5 100644 --- a/contrib/postgres_fdw/postgres_fdw.c +++ b/contrib/postgres_fdw/postgres_fdw.c @@ -479,7 +479,8 @@ postgresGetForeignRelSize(PlannerInfo *root, fpinfo->local_conds, baserel->relid, JOIN_INNER, - NULL); + NULL, + NIL); cost_qual_eval(&fpinfo->local_conds_cost, fpinfo->local_conds, root); @@ -1785,7 +1786,8 @@ estimate_path_cost_size(PlannerInfo *root, local_join_conds, baserel->relid, JOIN_INNER, - NULL); + NULL, + NIL); local_sel *= fpinfo->local_conds_sel; rows = clamp_row_est(rows * local_sel); diff --git a/src/backend/optimizer/path/clausesel.c b/src/backend/optimizer/path/clausesel.c index bc02e92..fce77ec 100644 --- a/src/backend/optimizer/path/clausesel.c +++ b/src/backend/optimizer/path/clausesel.c @@ -29,6 +29,8 @@ #include "utils/selfuncs.h" #include "utils/typcache.h" +#include "miscadmin.h" + /* * Data structure for accumulating info about possible range-query @@ -44,6 +46,13 @@ typedef struct RangeQueryClause Selectivity hibound; /* Selectivity of a var < something clause */ } RangeQueryClause; +static Selectivity clauselist_selectivity_or(PlannerInfo *root, + List *clauses, + int varRelid, + JoinType jointype, + SpecialJoinInfo *sjinfo, + List *conditions); + static void addRangeClause(RangeQueryClause **rqlist, Node *clause, bool varonleft, bool isLTsel, Selectivity s2); @@ -59,23 +68,29 @@ static Bitmapset *collect_mv_attnums(PlannerInfo *root, List *clauses, Oid varRelid, Index *relid, SpecialJoinInfo *sjinfo, int type); +static Bitmapset *clause_mv_get_attnums(PlannerInfo *root, Node *clause); + static List *clauselist_apply_dependencies(PlannerInfo *root, List *clauses, Oid varRelid, List *stats, SpecialJoinInfo *sjinfo); -static MVStatisticInfo *choose_mv_statistics(List *mvstats, Bitmapset *attnums); - static List *clauselist_mv_split(PlannerInfo *root, SpecialJoinInfo *sjinfo, List *clauses, Oid varRelid, List **mvclauses, MVStatisticInfo *mvstats, int types); static Selectivity clauselist_mv_selectivity(PlannerInfo *root, - List *clauses, MVStatisticInfo *mvstats); + MVStatisticInfo *mvstats, List *clauses, + List *conditions, bool is_or); + static Selectivity clauselist_mv_selectivity_mcvlist(PlannerInfo *root, - List *clauses, MVStatisticInfo *mvstats, - bool *fullmatch, Selectivity *lowsel); + MVStatisticInfo *mvstats, + List *clauses, List *conditions, + bool is_or, bool *fullmatch, + Selectivity *lowsel); static Selectivity clauselist_mv_selectivity_histogram(PlannerInfo *root, - List *clauses, MVStatisticInfo *mvstats); + MVStatisticInfo *mvstats, + List *clauses, List *conditions, + bool is_or); static int update_match_bitmap_mcvlist(PlannerInfo *root, List *clauses, int2vector *stakeys, MCVList mcvlist, @@ -89,11 +104,59 @@ static int update_match_bitmap_histogram(PlannerInfo *root, List *clauses, int nmatches, char * matches, bool is_or); +/* + * Describes a combination of multiple statistics to cover attributes + * referenced by the clauses. The array 'stats' (with nstats elements) + * lists attributes (in the order as they are applied), and number of + * clause attributes covered by this solution. + * + * choose_mv_statistics_exhaustive() uses this to track both the current + * and the best solutions, while walking through the state of possible + * combination. + */ +typedef struct mv_solution_t { + int nclauses; /* number of clauses covered */ + int nconditions; /* number of conditions covered */ + int nstats; /* number of stats applied */ + int *stats; /* stats (in the apply order) */ +} mv_solution_t; + +static List *choose_mv_statistics(PlannerInfo *root, + List *mvstats, + List *clauses, List *conditions, + Oid varRelid, + SpecialJoinInfo *sjinfo); + +static List *filter_clauses(PlannerInfo *root, Oid varRelid, + SpecialJoinInfo *sjinfo, int type, + List *stats, List *clauses, + Bitmapset **attnums); + +static List *filter_stats(List *stats, Bitmapset *new_attnums, + Bitmapset *all_attnums); + +static Bitmapset **make_stats_attnums(MVStatisticInfo *mvstats, + int nmvstats); + +static MVStatisticInfo *make_stats_array(List *stats, int *nmvstats); + +static List* filter_redundant_stats(List *stats, + List *clauses, List *conditions); + +static Node** make_clauses_array(List *clauses, int *nclauses); + +static Bitmapset ** make_clauses_attnums(PlannerInfo *root, Oid varRelid, + SpecialJoinInfo *sjinfo, int type, + Node **clauses, int nclauses); + +static bool* make_cover_map(Bitmapset **stats_attnums, int nmvstats, + Bitmapset **clauses_attnums, int nclauses); + static bool has_stats(List *stats, int type); static List * find_stats(PlannerInfo *root, List *clauses, Oid varRelid, Index *relid); - + static Bitmapset* fdeps_collect_attnums(List *stats); static int *make_idx_to_attnum_mapping(Bitmapset *attnums); @@ -116,6 +179,8 @@ static Bitmapset *fdeps_filter_clauses(PlannerInfo *root, static Bitmapset * get_varattnos(Node * node, Index relid); +int mvstat_search_type = MVSTAT_SEARCH_GREEDY; + /* used for merging bitmaps - AND (min), OR (max) */ #define MAX(x, y) (((x) > (y)) ? (x) : (y)) #define MIN(x, y) (((x) < (y)) ? (x) : (y)) @@ -256,14 +321,15 @@ clauselist_selectivity(PlannerInfo *root, List *clauses, int varRelid, JoinType jointype, - SpecialJoinInfo *sjinfo) + SpecialJoinInfo *sjinfo, + List *conditions) { Selectivity s1 = 1.0; RangeQueryClause *rqlist = NULL; ListCell *l; /* processing mv stats */ - Oid relid = InvalidOid; + Index relid = InvalidOid; /* attributes in mv-compatible clauses */ Bitmapset *mvattnums = NULL; @@ -273,12 +339,13 @@ clauselist_selectivity(PlannerInfo *root, stats = find_stats(root, clauses, varRelid, &relid); /* - * If there's exactly one clause, then no use in trying to match up pairs, - * so just go directly to clause_selectivity(). + * If there's exactly one clause, then no use in trying to match up + * pairs, or matching multivariate statistics, so just go directly + * to clause_selectivity(). */ if (list_length(clauses) == 1) return clause_selectivity(root, (Node *) linitial(clauses), - varRelid, jointype, sjinfo); + varRelid, jointype, sjinfo, conditions); /* * Check that there are some stats with functional dependencies @@ -310,8 +377,8 @@ clauselist_selectivity(PlannerInfo *root, } /* - * Check that there are statistics with MCV list. If not, we don't - * need to waste time with the optimization. + * Check that there are statistics with MCV list or histogram. + * If not, we don't need to waste time with the optimization. */ if (has_stats(stats, MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST)) { @@ -325,33 +392,194 @@ clauselist_selectivity(PlannerInfo *root, /* * If there still are at least two columns, we'll try to select - * a suitable multivariate stats. + * a suitable combination of multivariate stats. If there are + * multiple combinations, we'll try to choose the best one. + * See choose_mv_statistics for more details. */ if (bms_num_members(mvattnums) >= 2) { - /* see choose_mv_statistics() for details */ - MVStatisticInfo *mvstat = choose_mv_statistics(stats, mvattnums); + int k; + ListCell *s; - if (mvstat != NULL) /* we have a matching stats */ + /* + * Copy the list of conditions, so that we can build a list + * of local conditions (and keep the original intact, for + * the other clauses at the same level). + */ + List *conditions_local = list_copy(conditions); + + /* find the best combination of statistics */ + List *solution = choose_mv_statistics(root, stats, + clauses, conditions, + varRelid, sjinfo); + + /* we have a good solution (list of stats) */ + foreach (s, solution) { + MVStatisticInfo *mvstat = (MVStatisticInfo *)lfirst(s); + /* clauses compatible with multi-variate stats */ List *mvclauses = NIL; + List *mvclauses_new = NIL; + List *mvclauses_conditions = NIL; + Bitmapset *stat_attnums = NULL; - /* split the clauselist into regular and mv-clauses */ - clauses = clauselist_mv_split(root, sjinfo, clauses, + /* build attnum bitmapset for this statistics */ + for (k = 0; k < mvstat->stakeys->dim1; k++) + stat_attnums = bms_add_member(stat_attnums, + mvstat->stakeys->values[k]); + + /* + * Append the compatible conditions (passed from above) + * to mvclauses_conditions. + */ + foreach (l, conditions) + { + Node *c = (Node*)lfirst(l); + Bitmapset *tmp = clause_mv_get_attnums(root, c); + + if (bms_is_subset(tmp, stat_attnums)) + mvclauses_conditions + = lappend(mvclauses_conditions, c); + + bms_free(tmp); + } + + /* split the clauselist into regular and mv-clauses + * + * We keep the list of clauses (we don't remove the + * clauses yet, because we want to use the clauses + * as conditions of other clauses). + * + * FIXME Do this only once, i.e. filter the clauses + * once (selecting clauses covered by at least + * one statistics) and then convert them into + * smaller per-statistics lists of conditions + * and estimated clauses. + */ + clauselist_mv_split(root, sjinfo, clauses, varRelid, &mvclauses, mvstat, (MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST)); - /* we've chosen the histogram to match the clauses */ + /* + * We've chosen the statistics to match the clauses, so + * each statistics from the solution should have at least + * one new clause (not covered by the previous stats). + */ Assert(mvclauses != NIL); + /* + * Mvclauses now contains only clauses compatible + * with the currently selected stats, but we have to + * split that into conditions (already matched by + * the previous stats), and the new clauses we need + * to estimate using this stats. + */ + foreach (l, mvclauses) + { + ListCell *p; + bool covered = false; + Node *clause = (Node *) lfirst(l); + Bitmapset *clause_attnums = clause_mv_get_attnums(root, clause); + + /* + * If already covered by previous stats, add it to + * conditions. + * + * TODO Maybe this could be relaxed a bit? Because + * with complex and/or clauses, this might + * mean no statistics actually covers such + * complex clause. + */ + foreach (p, solution) + { + int k; + Bitmapset *stat_attnums = NULL; + + MVStatisticInfo *prev_stat + = (MVStatisticInfo *)lfirst(p); + + /* break if we've ran into current statistic */ + if (prev_stat == mvstat) + break; + + for (k = 0; k < prev_stat->stakeys->dim1; k++) + stat_attnums = bms_add_member(stat_attnums, + prev_stat->stakeys->values[k]); + + covered = bms_is_subset(clause_attnums, stat_attnums); + + bms_free(stat_attnums); + + if (covered) + break; + } + + if (covered) + mvclauses_conditions + = lappend(mvclauses_conditions, clause); + else + mvclauses_new + = lappend(mvclauses_new, clause); + } + + /* + * We need at least one new clause (not just conditions). + */ + Assert(mvclauses_new != NIL); + /* compute the multivariate stats */ - s1 *= clauselist_mv_selectivity(root, mvclauses, mvstat); + s1 *= clauselist_mv_selectivity(root, mvstat, + mvclauses_new, + mvclauses_conditions, + false); /* AND */ + } + + /* + * And now finally remove all the mv-compatible clauses. + * + * This only repeats the same split as above, but this + * time we actually use the result list (and feed it to + * the next call). + */ + foreach (s, solution) + { + /* clauses compatible with multi-variate stats */ + List *mvclauses = NIL; + + MVStatisticInfo *mvstat = (MVStatisticInfo *)lfirst(s); + + /* split the list into regular and mv-clauses */ + clauses = clauselist_mv_split(root, sjinfo, clauses, + varRelid, &mvclauses, mvstat, + (MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST)); + + /* + * Add the clauses to the conditions (to be passed + * to regular clauses), irrespectedly whether it + * will be used as a condition or a clause here. + * + * We only keep the remaining conditions in the + * clauses (we keep what clauselist_mv_split returns) + * so we add each MV condition exactly once. + */ + conditions_local = list_concat(conditions_local, mvclauses); } + + /* from now on, work with the 'local' list of conditions */ + conditions = conditions_local; } } /* + * If there's exactly one clause, then no use in trying to match up + * pairs, so just go directly to clause_selectivity(). + */ + if (list_length(clauses) == 1) + return clause_selectivity(root, (Node *) linitial(clauses), + varRelid, jointype, sjinfo, conditions); + + /* * Initial scan over clauses. Anything that doesn't look like a potential * rangequery clause gets multiplied into s1 and forgotten. Anything that * does gets inserted into an rqlist entry. @@ -363,7 +591,8 @@ clauselist_selectivity(PlannerInfo *root, Selectivity s2; /* Always compute the selectivity using clause_selectivity */ - s2 = clause_selectivity(root, clause, varRelid, jointype, sjinfo); + s2 = clause_selectivity(root, clause, varRelid, jointype, sjinfo, + conditions); /* * Check for being passed a RestrictInfo. @@ -522,6 +751,253 @@ clauselist_selectivity(PlannerInfo *root, } /* + * Similar to clauselist_selectivity(), but for clauses connected by OR. + * + * That means a few differences: + * + * - functional dependencies don't apply to OR-clauses + * + * - we can't add the previous clauses to conditions + * + * - combined selectivities are combined using (s1+s2 - s1*s2) + * and not as a multiplication (s1*s2) + * + * Another way to evaluate this might be turning + * + * (a OR b OR c) + * + * into + * + * NOT ((NOT a) AND (NOT b) AND (NOT c)) + * + * and computing selectivity of that using clauselist_selectivity(). + * That would allow (a) using the clauselist_selectivity directly and + * (b) using the previous clauses as conditions. Not sure if it's + * worth the additional complexity, though. + */ +static Selectivity +clauselist_selectivity_or(PlannerInfo *root, + List *clauses, + int varRelid, + JoinType jointype, + SpecialJoinInfo *sjinfo, + List *conditions) +{ + Selectivity s1 = 0.0; + ListCell *l; + + /* processing mv stats */ + Index relid = InvalidOid; + + /* attributes in mv-compatible clauses */ + Bitmapset *mvattnums = NULL; + List *stats = NIL; + + /* use clauses (not conditions), because those are always non-empty */ + stats = find_stats(root, clauses, varRelid, &relid); + + /* OR-clauses do not work with functional dependencies */ + if (has_stats(stats, MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST)) + { + /* + * Recollect attributes from mv-compatible clauses (maybe we've + * removed so many clauses we have a single mv-compatible attnum). + * From now on we're only interested in MCV-compatible clauses. + */ + mvattnums = collect_mv_attnums(root, clauses, varRelid, &relid, sjinfo, + (MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST)); + + /* + * If there still are at least two columns, we'll try to select + * a suitable multivariate stats. + */ + if (bms_num_members(mvattnums) >= 2) + { + int k; + ListCell *s; + + List *solution + = choose_mv_statistics(root, stats, + clauses, conditions, + varRelid, sjinfo); + + /* we have a good solution stats */ + foreach (s, solution) + { + Selectivity s2; + MVStatisticInfo *mvstat = (MVStatisticInfo *)lfirst(s); + + /* clauses compatible with multi-variate stats */ + List *mvclauses = NIL; + List *mvclauses_new = NIL; + List *mvclauses_conditions = NIL; + Bitmapset *stat_attnums = NULL; + + /* build attnum bitmapset for this statistics */ + for (k = 0; k < mvstat->stakeys->dim1; k++) + stat_attnums = bms_add_member(stat_attnums, + mvstat->stakeys->values[k]); + + /* + * Append the compatible conditions (passed from above) + * to mvclauses_conditions. + */ + foreach (l, conditions) + { + Node *c = (Node*)lfirst(l); + Bitmapset *tmp = clause_mv_get_attnums(root, c); + + if (bms_is_subset(tmp, stat_attnums)) + mvclauses_conditions + = lappend(mvclauses_conditions, c); + + bms_free(tmp); + } + + /* split the clauselist into regular and mv-clauses + * + * We keep the list of clauses (we don't remove the + * clauses yet, because we want to use the clauses + * as conditions of other clauses). + * + * FIXME Do this only once, i.e. filter the clauses + * once (selecting clauses covered by at least + * one statistics) and then convert them into + * smaller per-statistics lists of conditions + * and estimated clauses. + */ + clauselist_mv_split(root, sjinfo, clauses, + varRelid, &mvclauses, mvstat, + (MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST)); + + /* + * We've chosen the statistics to match the clauses, so + * each statistics from the solution should have at least + * one new clause (not covered by the previous stats). + */ + Assert(mvclauses != NIL); + + /* + * Mvclauses now contains only clauses compatible + * with the currently selected stats, but we have to + * split that into conditions (already matched by + * the previous stats), and the new clauses we need + * to estimate using this stats. + * + * XXX We'll only use the new clauses, but maybe we + * should use the conditions too, somehow. We can't + * use that directly in conditional probability, but + * maybe we might use them in a different way? + * + * If we have a clause (a OR b OR c), then knowing + * that 'a' is TRUE means (b OR c) can't make the + * whole clause FALSE. + * + * This is pretty much what + * + * (a OR b) == NOT ((NOT a) AND (NOT b)) + * + * implies. + */ + foreach (l, mvclauses) + { + ListCell *p; + bool covered = false; + Node *clause = (Node *) lfirst(l); + Bitmapset *clause_attnums = clause_mv_get_attnums(root, clause); + + /* + * If already covered by previous stats, add it to + * conditions. + * + * TODO Maybe this could be relaxed a bit? Because + * with complex and/or clauses, this might + * mean no statistics actually covers such + * complex clause. + */ + foreach (p, solution) + { + int k; + Bitmapset *stat_attnums = NULL; + + MVStatisticInfo *prev_stat + = (MVStatisticInfo *)lfirst(p); + + /* break if we've ran into current statistic */ + if (prev_stat == mvstat) + break; + + for (k = 0; k < prev_stat->stakeys->dim1; k++) + stat_attnums = bms_add_member(stat_attnums, + prev_stat->stakeys->values[k]); + + covered = bms_is_subset(clause_attnums, stat_attnums); + + bms_free(stat_attnums); + + if (covered) + break; + } + + if (! covered) + mvclauses_new = lappend(mvclauses_new, clause); + } + + /* + * We need at least one new clause (not just conditions). + */ + Assert(mvclauses_new != NIL); + + /* compute the multivariate stats */ + s2 = clauselist_mv_selectivity(root, mvstat, + mvclauses_new, + mvclauses_conditions, + true); /* OR */ + + s1 = s1 + s2 - s1 * s2; + } + + /* + * And now finally remove all the mv-compatible clauses. + * + * This only repeats the same split as above, but this + * time we actually use the result list (and feed it to + * the next call). + */ + foreach (s, solution) + { + /* clauses compatible with multi-variate stats */ + List *mvclauses = NIL; + + MVStatisticInfo *mvstat = (MVStatisticInfo *)lfirst(s); + + /* split the list into regular and mv-clauses */ + clauses = clauselist_mv_split(root, sjinfo, clauses, + varRelid, &mvclauses, mvstat, + (MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST)); + } + } + } + + /* + * Handle the remaining clauses (either using regular statistics, + * or by multivariate stats at the next level). + */ + foreach(l, clauses) + { + Selectivity s2 = clause_selectivity(root, + (Node *) lfirst(l), + varRelid, + jointype, + sjinfo, + conditions); + s1 = s1 + s2 - s1 * s2; + } + + return s1; +} + +/* * addRangeClause --- add a new range clause for clauselist_selectivity * * Here is where we try to match up pairs of range-query clauses @@ -728,7 +1204,8 @@ clause_selectivity(PlannerInfo *root, Node *clause, int varRelid, JoinType jointype, - SpecialJoinInfo *sjinfo) + SpecialJoinInfo *sjinfo, + List *conditions) { Selectivity s1 = 0.5; /* default for any unhandled clause type */ RestrictInfo *rinfo = NULL; @@ -858,7 +1335,8 @@ clause_selectivity(PlannerInfo *root, (Node *) get_notclausearg((Expr *) clause), varRelid, jointype, - sjinfo); + sjinfo, + conditions); } else if (and_clause(clause)) { @@ -867,29 +1345,18 @@ clause_selectivity(PlannerInfo *root, ((BoolExpr *) clause)->args, varRelid, jointype, - sjinfo); + sjinfo, + conditions); } else if (or_clause(clause)) { - /* - * Selectivities for an OR clause are computed as s1+s2 - s1*s2 to - * account for the probable overlap of selected tuple sets. - * - * XXX is this too conservative? - */ - ListCell *arg; - - s1 = 0.0; - foreach(arg, ((BoolExpr *) clause)->args) - { - Selectivity s2 = clause_selectivity(root, - (Node *) lfirst(arg), - varRelid, - jointype, - sjinfo); - - s1 = s1 + s2 - s1 * s2; - } + /* just call to clauselist_selectivity_or() */ + s1 = clauselist_selectivity_or(root, + ((BoolExpr *) clause)->args, + varRelid, + jointype, + sjinfo, + conditions); } else if (is_opclause(clause) || IsA(clause, DistinctExpr)) { @@ -998,7 +1465,8 @@ clause_selectivity(PlannerInfo *root, (Node *) ((RelabelType *) clause)->arg, varRelid, jointype, - sjinfo); + sjinfo, + conditions); } else if (IsA(clause, CoerceToDomain)) { @@ -1007,7 +1475,8 @@ clause_selectivity(PlannerInfo *root, (Node *) ((CoerceToDomain *) clause)->arg, varRelid, jointype, - sjinfo); + sjinfo, + conditions); } /* Cache the result if possible */ @@ -1120,9 +1589,67 @@ clause_selectivity(PlannerInfo *root, * them without inspection, which is more expensive). But this * requires really knowing the per-clause selectivities in advance, * and that's not what we do now. + * + * TODO All this is based on the assumption that the statistics represent + * the necessary dependencies, i.e. that if two colunms are not in + * the same statistics, there's no dependency. If that's not the + * case, we may get misestimates, just like before. For example + * assume we have a table with three columns [a,b,c] with exactly + * the same values, and statistics on [a,b] and [b,c]. So somthing + * like this: + * + * CREATE TABLE test AS SELECT i, i, i + FROM generate_series(1,1000); + * + * ALTER TABLE test ADD STATISTICS (mcv) ON (a,b); + * ALTER TABLE test ADD STATISTICS (mcv) ON (b,c); + * + * ANALYZE test; + * + * EXPLAIN ANALYZE SELECT * FROM test + * WHERE (a < 10) AND (b < 20) AND (c < 10); + * + * The problem here is that the only shared column between the two + * statistics is 'b' so the probability will be computed like this + * + * P[(a < 10) & (b < 20) & (c < 10)] + * = P[(a < 10) & (b < 20)] * P[(c < 10) | (a < 10) & (b < 20)] + * = P[(a < 10) & (b < 20)] * P[(c < 10) | (b < 20)] + * + * or like this + * + * P[(a < 10) & (b < 20) & (c < 10)] + * = P[(b < 20) & (c < 10)] * P[(a < 10) | (b < 20) & (c < 10)] + * = P[(b < 20) & (c < 10)] * P[(a < 10) | (b < 20)] + * + * In both cases the conditional probabilities will be evaluated as + * 0.5, because they lack the other column (which would make it 1.0). + * + * Theoretically it might be possible to transfer the dependency, + * e.g. by building bitmap for [a,b] and then combine it with [b,c] + * by doing something like this: + * + * 1) build bitmap on [a,b] using [(a<10) & (b < 20)] + * 2) for each element in [b,c] check the bitmap + * + * But that's certainly nontrivial - for example the statistics may + * be different (MCV list vs. histogram) and/or the items may not + * match (e.g. MCV items or histogram buckets will be built + * differently). Also, for one value of 'b' there might be multiple + * MCV items (because of the other column values) with different + * bitmap values (some will match, some won't) - so it's not exactly + * bitmap but a partial match. + * + * Maybe a hash table with number of matches and mismatches (or + * maybe sums of frequencies) would work? The step (2) would then + * lookup the values and use that to weight the item somehow. + * + * Currently the only solution is to build statistics on all three + * columns. */ static Selectivity -clauselist_mv_selectivity(PlannerInfo *root, List *clauses, MVStatisticInfo *mvstats) +clauselist_mv_selectivity(PlannerInfo *root, MVStatisticInfo *mvstats, + List *clauses, List *conditions, bool is_or) { bool fullmatch = false; Selectivity s1 = 0.0, s2 = 0.0; @@ -1140,7 +1667,8 @@ clauselist_mv_selectivity(PlannerInfo *root, List *clauses, MVStatisticInfo *mvs */ /* Evaluate the MCV first. */ - s1 = clauselist_mv_selectivity_mcvlist(root, clauses, mvstats, + s1 = clauselist_mv_selectivity_mcvlist(root, mvstats, + clauses, conditions, is_or, &fullmatch, &mcv_low); /* @@ -1153,7 +1681,8 @@ clauselist_mv_selectivity(PlannerInfo *root, List *clauses, MVStatisticInfo *mvs /* FIXME if (fullmatch) without matching MCV item, use the mcv_low * selectivity as upper bound */ - s2 = clauselist_mv_selectivity_histogram(root, clauses, mvstats); + s2 = clauselist_mv_selectivity_histogram(root, mvstats, + clauses, conditions, is_or); /* TODO clamp to <= 1.0 (or more strictly, when possible) */ return s1 + s2; @@ -1193,8 +1722,7 @@ collect_mv_attnums(PlannerInfo *root, List *clauses, Oid varRelid, */ if (bms_num_members(attnums) <= 1) { - if (attnums != NULL) - pfree(attnums); + bms_free(attnums); attnums = NULL; *relid = InvalidOid; } @@ -1203,123 +1731,852 @@ collect_mv_attnums(PlannerInfo *root, List *clauses, Oid varRelid, } /* - * We're looking for statistics matching at least 2 attributes, - * referenced in the clauses compatible with multivariate statistics. - * The current selection criteria is very simple - we choose the - * statistics referencing the most attributes. + * Selects the best combination of multivariate statistics, in an + * exhaustive way, where 'best' means: + * + * (a) covering the most attributes (referenced by clauses) + * (b) using the least number of multivariate stats + * (c) using the most conditions to exploit dependency + * + * There may be other optimality criteria, not considered in the initial + * implementation (more on that 'weaknesses' section). + * + * This pretty much splits the probability of clauses (aka selectivity) + * into a sequence of conditional probabilities, like this + * + * P(A,B,C,D) = P(A,B) * P(C|A,B) * P(D|A,B,C) + * + * and removing the attributes not referenced by the existing stats, + * under the assumption that there's no dependency (otherwise the DBA + * would create the stats). + * + * The last criteria means that when we have the choice to compute like + * this + * + * P(A,B,C,D) = P(A,B,C) * P(D|B,C) * - * If there are multiple statistics referencing the same number of - * columns (from the clauses), the one with less source columns - * (as listed in the ADD STATISTICS when creating the statistics) wins. - * Other wise the first one wins. + * or like this * - * This is a very simple criteria, and has several weaknesses: + * P(A,B,C,D) = P(A,B,C) * P(D|C) * - * (a) does not consider the accuracy of the statistics + * we should use the first option, as that exploits more dependencies. * - * If there are two histograms built on the same set of columns, - * but one has 100 buckets and the other one has 1000 buckets (thus - * likely providing better estimates), this is not currently - * considered. + * The order of statistics in the solution implicitly determines the + * order of estimation of clauses, because as we apply a statistics, + * we always use it to estimate all the clauses covered by it (and + * then we use those clauses as conditions for the next statistics). * - * (b) does not consider the type of statistics + * Don't call this directly but through choose_mv_statistics(). * - * If there are three statistics - one containing just a MCV list, - * another one with just a histogram and a third one with both, - * this is not considered. * - * (c) does not consider the number of clauses + * Algorithm + * --------- + * The algorithm is a recursive implementation of backtracking, with + * maximum 'depth' equal to the number of multi-variate statistics + * available on the table. * - * As explained, only the number of referenced attributes counts, - * so if there are multiple clauses on a single attribute, this - * still counts as a single attribute. + * It explores all the possible permutations of the stats. + * + * Whenever it considers adding the next statistics, the clauses it + * matches are divided into 'conditions' (clauses already matched by at + * least one previous statistics) and clauses that are estimated. * - * (d) does not consider type of condition + * Then several checks are performed: * - * Some clauses may work better with some statistics - for example - * equality clauses probably work better with MCV lists than with - * histograms. But IS [NOT] NULL conditions may often work better - * with histograms (thanks to NULL-buckets). + * (a) The statistics covers at least 2 columns, referenced in the + * estimated clauses (otherwise multi-variate stats are useless). * - * So for example with five WHERE conditions + * (b) The statistics covers at least 1 new column, i.e. column not + * refefenced by the already used stats (and the new column has + * to be referenced by the clauses, of couse). Otherwise the + * statistics would not add any new information. * - * WHERE (a = 1) AND (b = 1) AND (c = 1) AND (d = 1) AND (e = 1) + * There are some other sanity checks (e.g. that the stats must not be + * used twice etc.). * - * and statistics on (a,b), (a,b,e) and (a,b,c,d), the last one will be - * selected as it references the most columns. + * Finally the new solution is compared to the currently best one, and + * if it's considered better, it's used instead. * - * Once we have selected the multivariate statistics, we split the list - * of clauses into two parts - conditions that are compatible with the - * selected stats, and conditions are estimated using simple statistics. * - * From the example above, conditions + * Weaknesses + * ---------- + * The current implemetation uses a somewhat simple optimality criteria, + * suffering by the following weaknesses. * - * (a = 1) AND (b = 1) AND (c = 1) AND (d = 1) + * (a) There may be multiple solutions with the same number of covered + * attributes and number of statistics (e.g. the same solution but + * with statistics in a different order). It's unclear which solution + * is the best one - in a sense all of them are equal. * - * will be estimated using the multivariate statistics (a,b,c,d) while - * the last condition (e = 1) will get estimated using the regular ones. + * TODO It might be possible to compute estimate for each of those + * solutions, and then combine them to get the final estimate + * (e.g. by using average or median). * - * There are various alternative selection criteria (e.g. counting - * conditions instead of just referenced attributes), but eventually - * the best option should be to combine multiple statistics. But that's - * much harder to do correctly. + * (b) Does not consider that some types of stats are a better match for + * some types of clauses (e.g. MCV list is a good match for equality + * than a histogram). * - * TODO Select multiple statistics and combine them when computing - * the estimate. + * XXX Maybe MCV is almost always better / more accurate? + * + * But maybe this is pointless - generally, each column is either + * a label (it's not important whether because of the data type or + * how it's used), or a value with ordering that makes sense. So + * either a MCV list is more appropriate (labels) or a histogram + * (values with orderings). + * + * Now sure what to do with statistics on columns mixing columns of + * both types - maybe it'd be beeter to invent a new type of stats + * combining MCV list and histogram (keeping a small histogram for + * each MCV item, and a separate histogram for values not on the + * MCV list). But that's not implemented at this moment. + * + * TODO The algorithm should probably count number of Vars (not just + * attnums) when computing the 'score' of each solution. Computing + * the ratio of (num of all vars) / (num of condition vars) as a + * measure of how well the solution uses conditions might be + * useful. + */ +static void +choose_mv_statistics_exhaustive(PlannerInfo *root, int step, + int nmvstats, MVStatisticInfo *mvstats, Bitmapset ** stats_attnums, + int nclauses, Node ** clauses, Bitmapset ** clauses_attnums, + int nconditions, Node ** conditions, Bitmapset ** conditions_attnums, + bool *cover_map, bool *condition_map, int *ruled_out, + mv_solution_t *current, mv_solution_t **best) +{ + int i, j; + + Assert(best != NULL); + Assert((step == 0 && current == NULL) || (step > 0 && current != NULL)); + + CHECK_FOR_INTERRUPTS(); + + if (current == NULL) + { + current = (mv_solution_t*)palloc0(sizeof(mv_solution_t)); + current->stats = (int*)palloc0(sizeof(int)*nmvstats); + current->nstats = 0; + current->nclauses = 0; + current->nconditions = 0; + } + + /* + * Now try to apply each statistics, matching at least two attributes, + * unless it's already used in one of the previous steps. + */ + for (i = 0; i < nmvstats; i++) + { + int c; + + int ncovered_clauses = 0; /* number of covered clauses */ + int ncovered_conditions = 0; /* number of covered conditions */ + int nattnums = 0; /* number of covered attributes */ + + Bitmapset *all_attnums = NULL; + Bitmapset *new_attnums = NULL; + + /* skip statistics that were already used or eliminated */ + if (ruled_out[i] != -1) + continue; + + /* + * See if we have clauses covered by this statistics, but not + * yet covered by any of the preceding onces. + */ + for (c = 0; c < nclauses; c++) + { + bool covered = false; + Bitmapset *clause_attnums = clauses_attnums[c]; + Bitmapset *tmp = NULL; + + /* + * If this clause is not covered by this stats, we can't + * use the stats to estimate that at all. + */ + if (! cover_map[i * nclauses + c]) + continue; + + /* + * Now we know we'll use this clause - either as a condition + * or as a new clause (the estimated one). So let's add the + * attributes to the attnums from all the clauses usable with + * this statistics. + */ + tmp = bms_union(all_attnums, clause_attnums); + + /* free the old bitmap */ + bms_free(all_attnums); + all_attnums = tmp; + + /* let's see if it's covered by any of the previous stats */ + for (j = 0; j < step; j++) + { + /* already covered by the previous stats */ + if (cover_map[current->stats[j] * nclauses + c]) + covered = true; + + if (covered) + break; + } + + /* if already covered, continue with the next clause */ + if (covered) + { + ncovered_conditions += 1; + continue; + } + + /* + * OK, this clause is covered by this statistics (and not by + * any of the previous ones) + */ + ncovered_clauses += 1; + + /* add the attnums into attnums from 'new clauses' */ + // new_attnums = bms_union(new_attnums, clause_attnums); + } + + /* can't have more new clauses than original clauses */ + Assert(nclauses >= ncovered_clauses); + Assert(ncovered_clauses >= 0); /* mostly paranoia */ + + nattnums = bms_num_members(all_attnums); + + /* free all the bitmapsets - we don't need them anymore */ + bms_free(all_attnums); + bms_free(new_attnums); + + all_attnums = NULL; + new_attnums = NULL; + + /* + * See if we have clauses covered by this statistics, but not + * yet covered by any of the preceding onces. + */ + for (c = 0; c < nconditions; c++) + { + Bitmapset *clause_attnums = conditions_attnums[c]; + Bitmapset *tmp = NULL; + + /* + * If this clause is not covered by this stats, we can't + * use the stats to estimate that at all. + */ + if (! condition_map[i * nconditions + c]) + continue; + + /* count this as a condition */ + ncovered_conditions += 1; + + /* + * Now we know we'll use this clause - either as a condition + * or as a new clause (the estimated one). So let's add the + * attributes to the attnums from all the clauses usable with + * this statistics. + */ + tmp = bms_union(all_attnums, clause_attnums); + + /* free the old bitmap */ + bms_free(all_attnums); + all_attnums = tmp; + } + + /* + * Let's mark the statistics as 'ruled out' - either we'll use + * it (and proceed to the next step), or it's incompatible. + */ + ruled_out[i] = step; + + /* + * There are no clauses usable with this statistics (not already + * covered by aome of the previous stats). + * + * Similarly, if the clauses only use a single attribute, we + * can't really use that. + */ + if ((ncovered_clauses == 0) || (nattnums < 2)) + continue; + + /* + * TODO Not sure if it's possible to add a clause referencing + * only attributes already covered by previous stats? + * Introducing only some new dependency, not a new + * attribute. Couldn't come up with an example, though. + * Might be worth adding some assert. + */ + + /* + * got a suitable statistics - let's update the current solution, + * maybe use it as the best solution + */ + current->nclauses += ncovered_clauses; + current->nconditions += ncovered_conditions; + current->nstats += 1; + current->stats[step] = i; + + /* + * We can never cover more clauses, or use more stats that we + * actually have at the beginning. + */ + Assert(nclauses >= current->nclauses); + Assert(nmvstats >= current->nstats); + Assert(step < nmvstats); + + /* we can't get more conditions that clauses and conditions combined + * + * FIXME This assert does not work because we count the conditions + * repeatedly (once for each statistics covering it). + */ + /* Assert((nconditions + nclauses) >= current->nconditions); */ + + if (*best == NULL) + { + *best = (mv_solution_t*)palloc0(sizeof(mv_solution_t)); + (*best)->stats = (int*)palloc0(sizeof(int)*nmvstats); + (*best)->nstats = 0; + (*best)->nclauses = 0; + (*best)->nconditions = 0; + } + + /* see if it's better than the current 'best' solution */ + if ((current->nclauses > (*best)->nclauses) || + ((current->nclauses == (*best)->nclauses) && + ((current->nstats > (*best)->nstats)))) + { + (*best)->nstats = current->nstats; + (*best)->nclauses = current->nclauses; + (*best)->nconditions = current->nconditions; + memcpy((*best)->stats, current->stats, nmvstats * sizeof(int)); + } + + /* + * The recursion only makes sense if we haven't covered all the + * attributes (then adding stats is not really possible). + */ + if ((step + 1) < nmvstats) + choose_mv_statistics_exhaustive(root, step+1, + nmvstats, mvstats, stats_attnums, + nclauses, clauses, clauses_attnums, + nconditions, conditions, conditions_attnums, + cover_map, condition_map, ruled_out, + current, best); + + /* reset the last step */ + current->nclauses -= ncovered_clauses; + current->nconditions -= ncovered_conditions; + current->nstats -= 1; + current->stats[step] = 0; + + /* mark the statistics as usable again */ + ruled_out[i] = -1; + + Assert(current->nclauses >= 0); + Assert(current->nstats >= 0); + } + + /* reset all statistics as 'incompatible' in this step */ + for (i = 0; i < nmvstats; i++) + if (ruled_out[i] == step) + ruled_out[i] = -1; + +} + +/* + * Greedy search for a multivariate solution - a sequence of statistics + * covering the clauses. This chooses the "best" statistics at each step, + * so the resulting solution may not be the best solution globally, but + * this produces the solution in only N steps (where N is the number of + * statistics), while the exhaustive approach may have to walk through + * ~N! combinations (although some of those are terminated early). + * + * See the comments at choose_mv_statistics_exhaustive() as this does + * the same thing (but in a different way). + * + * Don't call this directly, but through choose_mv_statistics(). + * + * TODO There are probably other metrics we might use - e.g. using + * number of columns (num_cond_columns / num_cov_columns), which + * might work better with a mix of simple and complex clauses. + * + * TODO Also the choice at the very first step should be handled + * in a special way, because there will be 0 conditions at that + * moment, so there needs to be some other criteria - e.g. using + * the simplest (or most complex?) clause might be a good idea. + * + * TODO We might also select multiple stats using different criteria, + * and branch the search. This is however tricky, because if we + * choose k statistics at each step, we get k^N branches to + * walk through (with N steps). That's not really good with + * large number of stats (yet better than exhaustive search). + */ +static void +choose_mv_statistics_greedy(PlannerInfo *root, int step, + int nmvstats, MVStatisticInfo *mvstats, Bitmapset ** stats_attnums, + int nclauses, Node ** clauses, Bitmapset ** clauses_attnums, + int nconditions, Node ** conditions, Bitmapset ** conditions_attnums, + bool *cover_map, bool *condition_map, int *ruled_out, + mv_solution_t *current, mv_solution_t **best) +{ + int i, j; + int best_stat = -1; + double gain, max_gain = -1.0; + + /* + * Bitmap tracking which clauses are already covered (by the previous + * statistics) and may thus serve only as a condition in this step. + */ + bool *covered_clauses = (bool*)palloc0(nclauses); + + /* + * Number of clauses and columns covered by each statistics - this + * includes both conditions and clauses covered by the statistics for + * the first time. The number of columns may count some columns + * repeatedly - if a column is shared by multiple clauses, it will + * be counted once for each clause (covered by the statistics). + * So with two clauses [(a=1 OR b=2),(a<2 OR c>1)] the column "a" + * will be counted twice (if both clauses are covered). + * + * The values for reduded statistics (that can't be applied) are + * not computed, because that'd be pointless. + */ + int *num_cov_clauses = (int*)palloc0(sizeof(int) * nmvstats); + int *num_cov_columns = (int*)palloc0(sizeof(int) * nmvstats); + + /* + * Same as above, but this only includes clauses that are already + * covered by the previous stats (and the current one). + */ + int *num_cond_clauses = (int*)palloc0(sizeof(int) * nmvstats); + int *num_cond_columns = (int*)palloc0(sizeof(int) * nmvstats); + + /* + * Number of attributes for each clause. + * + * TODO Might be computed in choose_mv_statistics() and then passed + * here, but then the function would not have the same signature + * as _exhaustive(). + */ + int *attnum_counts = (int*)palloc0(sizeof(int) * nclauses); + int *attnum_cond_counts = (int*)palloc0(sizeof(int) * nconditions); + + CHECK_FOR_INTERRUPTS(); + + Assert(best != NULL); + Assert((step == 0 && current == NULL) || (step > 0 && current != NULL)); + + /* compute attributes (columns) for each clause */ + for (i = 0; i < nclauses; i++) + attnum_counts[i] = bms_num_members(clauses_attnums[i]); + + /* compute attributes (columns) for each condition */ + for (i = 0; i < nconditions; i++) + attnum_cond_counts[i] = bms_num_members(conditions_attnums[i]); + + /* see which clauses are already covered at this point (by previous stats) */ + for (i = 0; i < step; i++) + for (j = 0; j < nclauses; j++) + covered_clauses[j] |= (cover_map[current->stats[i] * nclauses + j]); + + /* which remaining statistics covers most clauses / uses most conditions? */ + for (i = 0; i < nmvstats; i++) + { + Bitmapset *attnums_covered = NULL; + Bitmapset *attnums_conditions = NULL; + + /* skip stats that are already ruled out (either used or inapplicable) */ + if (ruled_out[i] != -1) + continue; + + /* count covered clauses and conditions (for the statistics) */ + for (j = 0; j < nclauses; j++) + { + if (cover_map[i * nclauses + j]) + { + Bitmapset *attnums_new + = bms_union(attnums_covered, clauses_attnums[j]); + + /* get rid of the old bitmap and keep the unified result */ + bms_free(attnums_covered); + attnums_covered = attnums_new; + + num_cov_clauses[i] += 1; + num_cov_columns[i] += attnum_counts[j]; + + /* is the clause already covered (i.e. a condition)? */ + if (covered_clauses[j]) + { + num_cond_clauses[i] += 1; + num_cond_columns[i] += attnum_counts[j]; + attnums_new = bms_union(attnums_conditions, + clauses_attnums[j]); + + bms_free(attnums_conditions); + attnums_conditions = attnums_new; + } + } + } + + /* if all covered clauses are covered by prev stats (thus conditions) */ + if (num_cov_clauses[i] == num_cond_clauses[i]) + ruled_out[i] = step; + + /* same if there are no new attributes */ + else if (bms_num_members(attnums_conditions) == bms_num_members(attnums_covered)) + ruled_out[i] = step; + + bms_free(attnums_covered); + bms_free(attnums_conditions); + + /* if the statistics is inapplicable, try the next one */ + if (ruled_out[i] != -1) + continue; + + /* now let's walk through conditions and count the covered */ + for (j = 0; j < nconditions; j++) + { + if (condition_map[i * nconditions + j]) + { + num_cond_clauses[i] += 1; + num_cond_columns[i] += attnum_cond_counts[j]; + } + } + + /* otherwise see if this improves the interesting metrics */ + gain = num_cond_columns[i] / (double)num_cov_columns[i]; + + if (gain > max_gain) + { + max_gain = gain; + best_stat = i; + } + } + + /* + * Have we found a suitable statistics? Add it to the solution and + * try next step. + */ + if (best_stat != -1) + { + /* mark the statistics, so that we skip it in next steps */ + ruled_out[best_stat] = step; + + /* allocate current solution if necessary */ + if (current == NULL) + { + current = (mv_solution_t*)palloc0(sizeof(mv_solution_t)); + current->stats = (int*)palloc0(sizeof(int)*nmvstats); + current->nstats = 0; + current->nclauses = 0; + current->nconditions = 0; + } + + current->nclauses += num_cov_clauses[best_stat]; + current->nconditions += num_cond_clauses[best_stat]; + current->stats[step] = best_stat; + current->nstats++; + + if (*best == NULL) + { + (*best) = (mv_solution_t*)palloc0(sizeof(mv_solution_t)); + (*best)->nstats = current->nstats; + (*best)->nclauses = current->nclauses; + (*best)->nconditions = current->nconditions; + + (*best)->stats = (int*)palloc0(sizeof(int)*nmvstats); + memcpy((*best)->stats, current->stats, nmvstats * sizeof(int)); + } + else + { + /* see if this is a better solution */ + double current_gain = (double)current->nconditions / current->nclauses; + double best_gain = (double)(*best)->nconditions / (*best)->nclauses; + + if ((current_gain > best_gain) || + ((current_gain == best_gain) && (current->nstats < (*best)->nstats))) + { + (*best)->nstats = current->nstats; + (*best)->nclauses = current->nclauses; + (*best)->nconditions = current->nconditions; + memcpy((*best)->stats, current->stats, nmvstats * sizeof(int)); + } + } + + /* + * The recursion only makes sense if we haven't covered all the + * attributes (then adding stats is not really possible). + */ + if ((step + 1) < nmvstats) + choose_mv_statistics_greedy(root, step+1, + nmvstats, mvstats, stats_attnums, + nclauses, clauses, clauses_attnums, + nconditions, conditions, conditions_attnums, + cover_map, condition_map, ruled_out, + current, best); + + /* reset the last step */ + current->nclauses -= num_cov_clauses[best_stat]; + current->nconditions -= num_cond_clauses[best_stat]; + current->nstats -= 1; + current->stats[step] = 0; + + /* mark the statistics as usable again */ + ruled_out[best_stat] = -1; + } + + /* reset all statistics eliminated in this step */ + for (i = 0; i < nmvstats; i++) + if (ruled_out[i] == step) + ruled_out[i] = -1; + + /* free everything allocated in this step */ + pfree(covered_clauses); + pfree(attnum_counts); + pfree(num_cov_clauses); + pfree(num_cov_columns); + pfree(num_cond_clauses); + pfree(num_cond_columns); +} + +/* + * Chooses the combination of statistics, optimal for estimation of + * a particular clause list. + * + * This only handles a 'preparation' shared by the exhaustive and greedy + * implementations (see the previous methods), mostly trying to reduce + * the size of the problem (eliminate clauses/statistics that can't be + * really used in the solution). + * + * It also precomputes bitmaps for attributes covered by clauses and + * statistics, so that we don't need to do that over and over in the + * actual optimizations (as it's both CPU and memory intensive). * * TODO This will probably have to consider compatibility of clauses, * because 'dependencies' will probably work only with equality * clauses. + * + * TODO Another way to make the optimization problems smaller might + * be splitting the statistics into several disjoint subsets, i.e. + * if we can split the graph of statistics (after the elimination) + * into multiple components (so that stats in different components + * share no attributes), we can do the optimization for each + * component separately. + * + * TODO If we could compute what is a "perfect solution" maybe we could + * terminate the search after reaching ~90% of it? Say, if we knew + * that we can cover 10 clauses and reuse 8 dependencies, maybe + * covering 9 clauses and 7 dependencies would be OK? */ -static MVStatisticInfo * -choose_mv_statistics(List *stats, Bitmapset *attnums) +static List* +choose_mv_statistics(PlannerInfo *root, List *stats, + List *clauses, List *conditions, + Oid varRelid, SpecialJoinInfo *sjinfo) { int i; - ListCell *lc; + mv_solution_t *best = NULL; + List *result = NIL; + + int nmvstats; + MVStatisticInfo *mvstats; + + /* we only work with MCV lists and histograms here */ + int type = (MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST); + + bool *clause_cover_map = NULL, + *condition_cover_map = NULL; + int *ruled_out = NULL; + + /* build bitmapsets for all stats and clauses */ + Bitmapset **stats_attnums; + Bitmapset **clauses_attnums; + Bitmapset **conditions_attnums; - MVStatisticInfo *choice = NULL; + int nclauses, nconditions; + Node ** clauses_array; + Node ** conditions_array; - int current_matches = 1; /* goal #1: maximize */ - int current_dims = (MVSTATS_MAX_DIMENSIONS+1); /* goal #2: minimize */ + /* copy lists, so that we can free them during elimination easily */ + clauses = list_copy(clauses); + conditions = list_copy(conditions); + stats = list_copy(stats); /* - * Walk through the statistics (simple array with nmvstats elements) - * and for each one count the referenced attributes (encoded in - * the 'attnums' bitmap). + * Reduce the optimization problem size as much as possible. + * + * Eliminate clauses and conditions not covered by any statistics, + * or statistics not matching at least two attributes (one of them + * has to be in a regular clause). + * + * It's possible that removing a statistics in one iteration + * eliminates clause in the next one, so we'll repeat this until we + * eliminate no clauses/stats in that iteration. + * + * This can only happen after eliminating a statistics - clauses are + * eliminated first, so statistics always reflect that. */ - foreach (lc, stats) + while (true) { - MVStatisticInfo *info = (MVStatisticInfo *)lfirst(lc); + List *tmp; + + Bitmapset *compatible_attnums = NULL; + Bitmapset *condition_attnums = NULL; + Bitmapset *all_attnums = NULL; + + /* + * Clauses + * + * Walk through clauses and keep only those covered by at least + * one of the statistics we still have. We'll also keep info + * about attnums in clauses (without conditions) so that we can + * ignore stats covering just conditions (which is pointless). + */ + tmp = filter_clauses(root, varRelid, sjinfo, type, + stats, clauses, &compatible_attnums); + + /* discard the original list */ + list_free(clauses); + clauses = tmp; + + /* + * Conditions + * + * Walk through clauses and keep only those covered by at least + * one of the statistics we still have. Also, collect bitmap of + * attributes so that we can make sure we add at least one new + * attribute (by comparing with clauses). + */ + if (conditions != NIL) + { + tmp = filter_clauses(root, varRelid, sjinfo, type, + stats, conditions, &condition_attnums); + + /* discard the original list */ + list_free(conditions); + conditions = tmp; + } + + /* get a union of attnums (from conditions and new clauses) */ + all_attnums = bms_union(compatible_attnums, condition_attnums); + + /* + * Statisitics + * + * Walk through statistics and only keep those covering at least + * one new attribute (excluding conditions) and at two attributes + * in both clauses and conditions. + */ + tmp = filter_stats(stats, compatible_attnums, all_attnums); + + /* if we've not eliminated anything, terminate */ + if (list_length(stats) == list_length(tmp)) + break; + + /* work only with filtered statistics from now */ + list_free(stats); + stats = tmp; + } + + /* only do the optimization if we have clauses/statistics */ + if ((list_length(stats) == 0) || (list_length(clauses) == 0)) + return NULL; + + /* remove redundant stats (stats covered by another stats) */ + stats = filter_redundant_stats(stats, clauses, conditions); + + /* + * TODO We should sort the stats to make the order deterministic, + * otherwise we may get different estimates on different + * executions - if there are multiple "equally good" solutions, + * we'll keep the first solution we see. + * + * Sorting by OID probably is not the right solution though, + * because we'd like it to be somehow reproducible, + * irrespectedly of the order of ADD STATISTICS commands. + * So maybe statkeys? + */ + mvstats = make_stats_array(stats, &nmvstats); + stats_attnums = make_stats_attnums(mvstats, nmvstats); + + /* collect clauses an bitmap of attnums */ + clauses_array = make_clauses_array(clauses, &nclauses); + clauses_attnums = make_clauses_attnums(root, varRelid, sjinfo, type, + clauses_array, nclauses); + + /* collect conditions and bitmap of attnums */ + conditions_array = make_clauses_array(conditions, &nconditions); + conditions_attnums = make_clauses_attnums(root, varRelid, sjinfo, type, + conditions_array, nconditions); + + /* + * Build bitmaps with info about which clauses/conditions are + * covered by each statistics (so that we don't need to call the + * bms_is_subset over and over again). + */ + clause_cover_map = make_cover_map(stats_attnums, nmvstats, + clauses_attnums, nclauses); + + condition_cover_map = make_cover_map(stats_attnums, nmvstats, + conditions_attnums, nconditions); + + ruled_out = (int*)palloc0(nmvstats * sizeof(int)); + + /* no stats are ruled out by default */ + for (i = 0; i < nmvstats; i++) + ruled_out[i] = -1; + + /* do the optimization itself */ + if (mvstat_search_type == MVSTAT_SEARCH_EXHAUSTIVE) + choose_mv_statistics_exhaustive(root, 0, + nmvstats, mvstats, stats_attnums, + nclauses, clauses_array, clauses_attnums, + nconditions, conditions_array, conditions_attnums, + clause_cover_map, condition_cover_map, + ruled_out, NULL, &best); + else + choose_mv_statistics_greedy(root, 0, + nmvstats, mvstats, stats_attnums, + nclauses, clauses_array, clauses_attnums, + nconditions, conditions_array, conditions_attnums, + clause_cover_map, condition_cover_map, + ruled_out, NULL, &best); + + /* create a list of statistics from the array */ + if (best != NULL) + { + for (i = 0; i < best->nstats; i++) + { + MVStatisticInfo *info = makeNode(MVStatisticInfo); + memcpy(info, &mvstats[best->stats[i]], sizeof(MVStatisticInfo)); + result = lappend(result, info); + } + pfree(best); + } - /* columns matching this statistics */ - int matches = 0; + /* cleanup (maybe leave it up to the memory context?) */ + for (i = 0; i < nmvstats; i++) + bms_free(stats_attnums[i]); - int2vector * attrs = info->stakeys; - int numattrs = attrs->dim1; + for (i = 0; i < nclauses; i++) + bms_free(clauses_attnums[i]); - /* skip dependencies-only stats */ - if (! (info->mcv_built || info->hist_built)) - continue; + for (i = 0; i < nconditions; i++) + bms_free(conditions_attnums[i]); - /* count columns covered by the histogram */ - for (i = 0; i < numattrs; i++) - if (bms_is_member(attrs->values[i], attnums)) - matches++; + pfree(stats_attnums); + pfree(clauses_attnums); + pfree(conditions_attnums); - /* - * Use this statistics when it improves the number of matches or - * when it matches the same number of attributes but is smaller. - */ - if ((matches > current_matches) || - ((matches == current_matches) && (current_dims > numattrs))) - { - choice = info; - current_matches = matches; - current_dims = numattrs; - } - } + pfree(clauses_array); + pfree(conditions_array); + pfree(clause_cover_map); + pfree(condition_cover_map); + pfree(ruled_out); + pfree(mvstats); - return choice; + list_free(clauses); + list_free(conditions); + list_free(stats); + + return result; } @@ -1589,6 +2846,51 @@ clause_is_mv_compatible(PlannerInfo *root, Node *clause, Oid varRelid, return false; } + +static Bitmapset * +clause_mv_get_attnums(PlannerInfo *root, Node *clause) +{ + Bitmapset * attnums = NULL; + + /* Extract clause from restrict info, if needed. */ + if (IsA(clause, RestrictInfo)) + clause = (Node*)((RestrictInfo*)clause)->clause; + + /* + * Only simple opclauses and IS NULL tests are compatible with + * multivariate stats at this point. + */ + if ((is_opclause(clause)) + && (list_length(((OpExpr *) clause)->args) == 2)) + { + OpExpr *expr = (OpExpr *) clause; + + if (IsA(linitial(expr->args), Var)) + attnums = bms_add_member(attnums, + ((Var*)linitial(expr->args))->varattno); + else + attnums = bms_add_member(attnums, + ((Var*)lsecond(expr->args))->varattno); + } + else if (IsA(clause, NullTest) + && IsA(((NullTest*)clause)->arg, Var)) + { + attnums = bms_add_member(attnums, + ((Var*)((NullTest*)clause)->arg)->varattno); + } + else if (or_clause(clause) || and_clause(clause)) + { + ListCell *l; + foreach (l, ((BoolExpr*)clause)->args) + { + attnums = bms_join(attnums, + clause_mv_get_attnums(root, (Node*)lfirst(l))); + } + } + + return attnums; +} + /* * Performs reduction of clauses using functional dependencies, i.e. * removes clauses that are considered redundant. It simply walks @@ -2240,22 +3542,26 @@ get_varattnos(Node * node, Index relid) * as the clauses are processed (and skip items that are 'match'). */ static Selectivity -clauselist_mv_selectivity_mcvlist(PlannerInfo *root, List *clauses, - MVStatisticInfo *mvstats, bool *fullmatch, - Selectivity *lowsel) +clauselist_mv_selectivity_mcvlist(PlannerInfo *root, MVStatisticInfo *mvstats, + List *clauses, List *conditions, bool is_or, + bool *fullmatch, Selectivity *lowsel) { int i; Selectivity s = 0.0; + Selectivity t = 0.0; Selectivity u = 0.0; MCVList mcvlist = NULL; + int nmatches = 0; + int nconditions = 0; /* match/mismatch bitmap for each MCV item */ char * matches = NULL; + char * condition_matches = NULL; Assert(clauses != NIL); - Assert(list_length(clauses) >= 2); + Assert(list_length(clauses) >= 1); /* there's no MCV list built yet */ if (! mvstats->mcv_built) @@ -2266,32 +3572,85 @@ clauselist_mv_selectivity_mcvlist(PlannerInfo *root, List *clauses, Assert(mcvlist != NULL); Assert(mcvlist->nitems > 0); - /* by default all the MCV items match the clauses fully */ - matches = palloc0(sizeof(char) * mcvlist->nitems); - memset(matches, MVSTATS_MATCH_FULL, sizeof(char)*mcvlist->nitems); - /* number of matching MCV items */ nmatches = mcvlist->nitems; + nconditions = mcvlist->nitems; + + /* + * Bitmap of bucket matches (mismatch, partial, full). + * + * For AND clauses all buckets match (and we'll eliminate them). + * For OR clauses no buckets match (and we'll add them). + * + * We only need to do the memset for AND clauses (for OR clauses + * it's already set correctly by the palloc0). + */ + matches = palloc0(sizeof(char) * nmatches); + if (! is_or) /* AND-clause */ + memset(matches, MVSTATS_MATCH_FULL, sizeof(char)*nmatches); + + /* Conditions are treated as AND clause, so match by default. */ + condition_matches = palloc0(sizeof(char) * nconditions); + memset(condition_matches, MVSTATS_MATCH_FULL, sizeof(char)*nconditions); + + /* + * build the match bitmap for the conditions (conditions are always + * connected by AND) + */ + if (conditions != NIL) + nconditions = update_match_bitmap_mcvlist(root, conditions, + mvstats->stakeys, mcvlist, + nconditions, condition_matches, + lowsel, fullmatch, false); + + /* + * build the match bitmap for the estimated clauses + * + * TODO This evaluates the clauses for all MCV items, even those + * ruled out by the conditions. The final result should be the + * same, but it might be faster. + */ nmatches = update_match_bitmap_mcvlist(root, clauses, mvstats->stakeys, mcvlist, - nmatches, matches, - lowsel, fullmatch, false); + ((is_or) ? 0 : nmatches), matches, + lowsel, fullmatch, is_or); /* sum frequencies for all the matching MCV items */ for (i = 0; i < mcvlist->nitems; i++) { - /* used to 'scale' for MCV lists not covering all tuples */ + /* + * Find out what part of the data is covered by the MCV list, + * so that we can 'scale' the selectivity properly (e.g. when + * only 50% of the sample items got into the MCV, and the rest + * is either in a histogram, or not covered by stats). + * + * TODO This might be handled by keeping a global "frequency" + * for the whole list, which might save us a bit of time + * spent on accessing the not-matching part of the MCV list. + * Although it's likely in a cache, so it's very fast. + */ u += mcvlist->items[i]->frequency; + /* skit MCV items not matching the conditions */ + if (condition_matches[i] == MVSTATS_MATCH_NONE) + continue; + if (matches[i] != MVSTATS_MATCH_NONE) s += mcvlist->items[i]->frequency; + + t += mcvlist->items[i]->frequency; } pfree(matches); + pfree(condition_matches); pfree(mcvlist); - return s*u; + /* no condition matches */ + if (t == 0.0) + return (Selectivity)0.0; + + return (s / t) * u; } /* @@ -2589,38 +3948,29 @@ update_match_bitmap_mcvlist(PlannerInfo *root, List *clauses, /* AND/OR clause, with all clauses compatible with the selected MV stat */ int i; - BoolExpr *orclause = ((BoolExpr*)clause); - List *orclauses = orclause->args; + List *tmp_clauses = ((BoolExpr*)clause)->args; /* match/mismatch bitmap for each MCV item */ - int or_nmatches = 0; - char * or_matches = NULL; + int tmp_nmatches = 0; + char * tmp_matches = NULL; - Assert(orclauses != NIL); - Assert(list_length(orclauses) >= 2); + Assert(tmp_clauses != NIL); + Assert(list_length(tmp_clauses) >= 2); /* number of matching MCV items */ - or_nmatches = mcvlist->nitems; + tmp_nmatches = (or_clause(clause)) ? 0 : mcvlist->nitems; /* by default none of the MCV items matches the clauses */ - or_matches = palloc0(sizeof(char) * or_nmatches); + tmp_matches = palloc0(sizeof(char) * mcvlist->nitems); - if (or_clause(clause)) - { - /* OR clauses assume nothing matches, initially */ - memset(or_matches, MVSTATS_MATCH_NONE, sizeof(char)*or_nmatches); - or_nmatches = 0; - } - else - { - /* AND clauses assume nothing matches, initially */ - memset(or_matches, MVSTATS_MATCH_FULL, sizeof(char)*or_nmatches); - } + /* AND clauses assume everything matches, initially */ + if (! or_clause(clause)) + memset(tmp_matches, MVSTATS_MATCH_FULL, sizeof(char)*mcvlist->nitems); /* build the match bitmap for the OR-clauses */ - or_nmatches = update_match_bitmap_mcvlist(root, orclauses, + tmp_nmatches = update_match_bitmap_mcvlist(root, tmp_clauses, stakeys, mcvlist, - or_nmatches, or_matches, + tmp_nmatches, tmp_matches, lowsel, fullmatch, or_clause(clause)); /* merge the bitmap into the existing one*/ @@ -2632,16 +3982,14 @@ update_match_bitmap_mcvlist(PlannerInfo *root, List *clauses, * * FIXME this does not decrease the number of matches */ - UPDATE_RESULT(matches[i], or_matches[i], is_or); + UPDATE_RESULT(matches[i], tmp_matches[i], is_or); } - pfree(or_matches); + pfree(tmp_matches); } else - { elog(ERROR, "unknown clause type: %d", clause->type); - } } /* @@ -2699,15 +4047,18 @@ update_match_bitmap_mcvlist(PlannerInfo *root, List *clauses, * this is not uncommon, but for histograms it's not that clear. */ static Selectivity -clauselist_mv_selectivity_histogram(PlannerInfo *root, List *clauses, - MVStatisticInfo *mvstats) +clauselist_mv_selectivity_histogram(PlannerInfo *root, MVStatisticInfo *mvstats, + List *clauses, List *conditions, bool is_or) { int i; Selectivity s = 0.0; + Selectivity t = 0.0; Selectivity u = 0.0; int nmatches = 0; + int nconditions = 0; char *matches = NULL; + char *condition_matches = NULL; MVSerializedHistogram mvhist = NULL; @@ -2720,25 +4071,52 @@ clauselist_mv_selectivity_histogram(PlannerInfo *root, List *clauses, Assert (mvhist != NULL); Assert (clauses != NIL); - Assert (list_length(clauses) >= 2); + Assert (list_length(clauses) >= 1); + + nmatches = mvhist->nbuckets; + nconditions = mvhist->nbuckets; /* - * Bitmap of bucket matches (mismatch, partial, full). by default - * all buckets fully match (and we'll eliminate them). + * Bitmap of bucket matches (mismatch, partial, full). + * + * For AND clauses all buckets match (and we'll eliminate them). + * For OR clauses no buckets match (and we'll add them). + * + * We only need to do the memset for AND clauses (for OR clauses + * it's already set correctly by the palloc0). */ - matches = palloc0(sizeof(char) * mvhist->nbuckets); - memset(matches, MVSTATS_MATCH_FULL, sizeof(char)*mvhist->nbuckets); + matches = palloc0(sizeof(char) * nmatches); - nmatches = mvhist->nbuckets; + if (! is_or) /* AND-clause */ + memset(matches, MVSTATS_MATCH_FULL, sizeof(char)*nmatches); + + /* Conditions are treated as AND clause, so match by default. */ + condition_matches = palloc0(sizeof(char)*nconditions); + memset(condition_matches, MVSTATS_MATCH_FULL, sizeof(char)*nconditions); - /* build the match bitmap */ + /* build the match bitmap for the conditions */ + if (conditions != NIL) + update_match_bitmap_histogram(root, conditions, + mvstats->stakeys, mvhist, + nconditions, condition_matches, is_or); + + /* + * build the match bitmap for the estimated clauses + * + * TODO This evaluates the clauses for all buckets, even those + * ruled out by the conditions. The final result should be + * the same, but it might be faster. + */ update_match_bitmap_histogram(root, clauses, mvstats->stakeys, mvhist, - nmatches, matches, false); + ((is_or) ? 0 : nmatches), matches, + is_or); /* now, walk through the buckets and sum the selectivities */ for (i = 0; i < mvhist->nbuckets; i++) { + float coeff = 1.0; + /* * Find out what part of the data is covered by the histogram, * so that we can 'scale' the selectivity properly (e.g. when @@ -2752,17 +4130,35 @@ clauselist_mv_selectivity_histogram(PlannerInfo *root, List *clauses, */ u += mvhist->buckets[i]->ntuples; + /* skip buckets not matching the conditions */ + if (condition_matches[i] == MVSTATS_MATCH_NONE) + continue; + else if (condition_matches[i] == MVSTATS_MATCH_PARTIAL) + coeff = 0.5; + + t += coeff * mvhist->buckets[i]->ntuples; + if (matches[i] == MVSTATS_MATCH_FULL) - s += mvhist->buckets[i]->ntuples; + s += coeff * mvhist->buckets[i]->ntuples; else if (matches[i] == MVSTATS_MATCH_PARTIAL) - s += 0.5 * mvhist->buckets[i]->ntuples; + /* + * TODO If both conditions and clauses match partially, this + * will use 0.25 match - not sure if that's the right + * thing solution, but seems about right. + */ + s += coeff * 0.5 * mvhist->buckets[i]->ntuples; } /* release the allocated bitmap and deserialized histogram */ pfree(matches); + pfree(condition_matches); pfree(mvhist); - return s * u; + /* no condition matches */ + if (t == 0.0) + return (Selectivity)0.0; + + return (s / t) * u; } /* @@ -3268,38 +4664,31 @@ update_match_bitmap_histogram(PlannerInfo *root, List *clauses, /* AND/OR clause, with all clauses compatible with the selected MV stat */ int i; - BoolExpr *orclause = ((BoolExpr*)clause); - List *orclauses = orclause->args; + List *tmp_clauses = ((BoolExpr*)clause)->args; /* match/mismatch bitmap for each bucket */ - int or_nmatches = 0; - char * or_matches = NULL; + int tmp_nmatches = 0; + char * tmp_matches = NULL; - Assert(orclauses != NIL); - Assert(list_length(orclauses) >= 2); + Assert(tmp_clauses != NIL); + Assert(list_length(tmp_clauses) >= 2); /* number of matching buckets */ - or_nmatches = mvhist->nbuckets; + tmp_nmatches = (or_clause(clause)) ? 0 : mvhist->nbuckets; /* by default none of the buckets matches the clauses */ - or_matches = palloc0(sizeof(char) * or_nmatches); + tmp_matches = palloc0(sizeof(char) * mvhist->nbuckets); if (or_clause(clause)) { - /* OR clauses assume nothing matches, initially */ - memset(or_matches, MVSTATS_MATCH_NONE, sizeof(char)*or_nmatches); - or_nmatches = 0; - } - else - { - /* AND clauses assume nothing matches, initially */ - memset(or_matches, MVSTATS_MATCH_FULL, sizeof(char)*or_nmatches); + /* AND clauses assume everything matches, initially */ + memset(tmp_matches, MVSTATS_MATCH_FULL, sizeof(char)*mvhist->nbuckets); } /* build the match bitmap for the OR-clauses */ - or_nmatches = update_match_bitmap_histogram(root, orclauses, + tmp_nmatches = update_match_bitmap_histogram(root, tmp_clauses, stakeys, mvhist, - or_nmatches, or_matches, or_clause(clause)); + tmp_nmatches, tmp_matches, or_clause(clause)); /* merge the bitmap into the existing one*/ for (i = 0; i < mvhist->nbuckets; i++) @@ -3310,10 +4699,10 @@ update_match_bitmap_histogram(PlannerInfo *root, List *clauses, * * FIXME this does not decrease the number of matches */ - UPDATE_RESULT(matches[i], or_matches[i], is_or); + UPDATE_RESULT(matches[i], tmp_matches[i], is_or); } - pfree(or_matches); + pfree(tmp_matches); } else @@ -3325,3 +4714,363 @@ update_match_bitmap_histogram(PlannerInfo *root, List *clauses, return nmatches; } + +/* + * Walk through clauses and keep only those covered by at least + * one of the statistics. + */ +static List * +filter_clauses(PlannerInfo *root, Oid varRelid, SpecialJoinInfo *sjinfo, + int type, List *stats, List *clauses, Bitmapset **attnums) +{ + ListCell *c; + ListCell *s; + + /* results (list of compatible clauses, attnums) */ + List *rclauses = NIL; + + foreach (c, clauses) + { + Node *clause = (Node*)lfirst(c); + Bitmapset *clause_attnums = NULL; + Index relid; + + /* + * The clause has to be mv-compatible (suitable operators etc.). + */ + if (! clause_is_mv_compatible(root, clause, varRelid, + &relid, &clause_attnums, sjinfo, type)) + elog(ERROR, "should not get non-mv-compatible cluase"); + + /* is there a statistics covering this clause? */ + foreach (s, stats) + { + int k, matches = 0; + MVStatisticInfo *stat = (MVStatisticInfo *)lfirst(s); + + for (k = 0; k < stat->stakeys->dim1; k++) + { + if (bms_is_member(stat->stakeys->values[k], + clause_attnums)) + matches += 1; + } + + /* + * The clause is compatible if all attributes it references + * are covered by the statistics. + */ + if (bms_num_members(clause_attnums) == matches) + { + *attnums = bms_union(*attnums, clause_attnums); + rclauses = lappend(rclauses, clause); + break; + } + } + + bms_free(clause_attnums); + } + + /* we can't have more compatible conditions than source conditions */ + Assert(list_length(clauses) >= list_length(rclauses)); + + return rclauses; +} + + +/* + * Walk through statistics and only keep those covering at least + * one new attribute (excluding conditions) and at two attributes + * in both clauses and conditions. + * + * This check might be made more strict by checking against individual + * clauses, because by using the bitmapsets of all attnums we may + * actually use attnums from clauses that are not covered by the + * statistics. For example, we may have a condition + * + * (a=1 AND b=2) + * + * and a new clause + * + * (c=1 AND d=1) + * + * With only bitmapsets, statistics on [b,c] will pass through this + * (assuming there are some statistics covering both clases). + * + * TODO Do the more strict check. + */ +static List * +filter_stats(List *stats, Bitmapset *new_attnums, Bitmapset *all_attnums) +{ + ListCell *s; + List *stats_filtered = NIL; + + foreach (s, stats) + { + int k; + int matches_new = 0, + matches_all = 0; + + MVStatisticInfo *stat = (MVStatisticInfo *)lfirst(s); + + /* see how many attributes the statistics covers */ + for (k = 0; k < stat->stakeys->dim1; k++) + { + /* attributes from new clauses */ + if (bms_is_member(stat->stakeys->values[k], new_attnums)) + matches_new += 1; + + /* attributes from onditions */ + if (bms_is_member(stat->stakeys->values[k], all_attnums)) + matches_all += 1; + } + + /* check we have enough attributes for this statistics */ + if ((matches_new >= 1) && (matches_all >= 2)) + stats_filtered = lappend(stats_filtered, stat); + } + + /* we can't have more useful stats than we had originally */ + Assert(list_length(stats) >= list_length(stats_filtered)); + + return stats_filtered; +} + +static MVStatisticInfo * +make_stats_array(List *stats, int *nmvstats) +{ + int i; + ListCell *l; + + MVStatisticInfo *mvstats = NULL; + *nmvstats = list_length(stats); + + mvstats + = (MVStatisticInfo*)palloc0((*nmvstats) * sizeof(MVStatisticInfo)); + + i = 0; + foreach (l, stats) + { + MVStatisticInfo *stat = (MVStatisticInfo *)lfirst(l); + memcpy(&mvstats[i++], stat, sizeof(MVStatisticInfo)); + } + + return mvstats; +} + +static Bitmapset ** +make_stats_attnums(MVStatisticInfo *mvstats, int nmvstats) +{ + int i, j; + Bitmapset **stats_attnums = NULL; + + Assert(nmvstats > 0); + + /* build bitmaps of attnums for the stats (easier to compare) */ + stats_attnums = (Bitmapset **)palloc0(nmvstats * sizeof(Bitmapset*)); + + for (i = 0; i < nmvstats; i++) + for (j = 0; j < mvstats[i].stakeys->dim1; j++) + stats_attnums[i] + = bms_add_member(stats_attnums[i], + mvstats[i].stakeys->values[j]); + + return stats_attnums; +} + + +/* + * Now let's remove redundant statistics, covering the same columns + * as some other stats, when restricted to the attributes from + * remaining clauses. + * + * If statistics S1 covers S2 (covers S2 attributes and possibly + * some more), we can probably remove S2. What actually matters are + * attributes from covered clauses (not all the attributes). This + * might however prefer larger, and thus less accurate, statistics. + * + * When a redundancy is detected, we simply keep the smaller + * statistics (less number of columns), on the assumption that it's + * more accurate and faster to process. That might be incorrect for + * two reasons - first, the accuracy really depends on number of + * buckets/MCV items, not the number of columns. Second, we might + * prefer MCV lists over histograms or something like that. + */ +static List* +filter_redundant_stats(List *stats, List *clauses, List *conditions) +{ + int i, j, nmvstats; + + MVStatisticInfo *mvstats; + bool *redundant; + Bitmapset **stats_attnums; + Bitmapset *varattnos; + Index relid; + + Assert(list_length(stats) > 0); + Assert(list_length(clauses) > 0); + + /* + * We'll convert the list of statistics into an array now, because + * the reduction of redundant statistics is easier to do that way + * (we can mark previous stats as redundant, etc.). + */ + mvstats = make_stats_array(stats, &nmvstats); + stats_attnums = make_stats_attnums(mvstats, nmvstats); + + /* by default, none of the stats is redundant (so palloc0) */ + redundant = palloc0(nmvstats * sizeof(bool)); + + /* + * We only expect a single relid here, and also we should get the + * same relid from clauses and conditions (but we get it from + * clauses, because those are certainly non-empty). + */ + relid = bms_singleton_member(pull_varnos((Node*)clauses)); + + /* + * Get the varattnos from both conditions and clauses. + * + * This skips system attributes, although that should be impossible + * thanks to previous filtering out of incompatible clauses. + * + * XXX Is that really true? + */ + varattnos = bms_union(get_varattnos((Node*)clauses, relid), + get_varattnos((Node*)conditions, relid)); + + for (i = 1; i < nmvstats; i++) + { + /* intersect with current statistics */ + Bitmapset *curr = bms_intersect(stats_attnums[i], varattnos); + + /* walk through 'previous' stats and check redundancy */ + for (j = 0; j < i; j++) + { + /* intersect with current statistics */ + Bitmapset *prev; + + /* skip stats already identified as redundant */ + if (redundant[j]) + continue; + + prev = bms_intersect(stats_attnums[j], varattnos); + + switch (bms_subset_compare(curr, prev)) + { + case BMS_EQUAL: + /* + * Use the smaller one (hopefully more accurate). + * If both have the same size, use the first one. + */ + if (mvstats[i].stakeys->dim1 >= mvstats[j].stakeys->dim1) + redundant[i] = TRUE; + else + redundant[j] = TRUE; + + break; + + case BMS_SUBSET1: /* curr is subset of prev */ + redundant[i] = TRUE; + break; + + case BMS_SUBSET2: /* prev is subset of curr */ + redundant[j] = TRUE; + break; + + case BMS_DIFFERENT: + /* do nothing - keep both stats */ + break; + } + + bms_free(prev); + } + + bms_free(curr); + } + + /* can't reduce all statistics (at least one has to remain) */ + Assert(nmvstats > 0); + + /* now, let's remove the reduced statistics from the arrays */ + list_free(stats); + stats = NIL; + + for (i = 0; i < nmvstats; i++) + { + MVStatisticInfo *info; + + pfree(stats_attnums[i]); + + if (redundant[i]) + continue; + + info = makeNode(MVStatisticInfo); + memcpy(info, &mvstats[i], sizeof(MVStatisticInfo)); + + stats = lappend(stats, info); + } + + pfree(mvstats); + pfree(stats_attnums); + pfree(redundant); + + return stats; +} + +static Node** +make_clauses_array(List *clauses, int *nclauses) +{ + int i; + ListCell *l; + + Node** clauses_array; + + *nclauses = list_length(clauses); + clauses_array = (Node **)palloc0((*nclauses) * sizeof(Node *)); + + i = 0; + foreach (l, clauses) + clauses_array[i++] = (Node *)lfirst(l); + + *nclauses = i; + + return clauses_array; +} + +static Bitmapset ** +make_clauses_attnums(PlannerInfo *root, Oid varRelid, SpecialJoinInfo *sjinfo, + int type, Node **clauses, int nclauses) +{ + int i; + Index relid; + Bitmapset **clauses_attnums + = (Bitmapset **)palloc0(nclauses * sizeof(Bitmapset *)); + + for (i = 0; i < nclauses; i++) + { + Bitmapset * attnums = NULL; + + if (! clause_is_mv_compatible(root, clauses[i], varRelid, + &relid, &attnums, sjinfo, type)) + elog(ERROR, "should not get non-mv-compatible cluase"); + + clauses_attnums[i] = attnums; + } + + return clauses_attnums; +} + +static bool* +make_cover_map(Bitmapset **stats_attnums, int nmvstats, + Bitmapset **clauses_attnums, int nclauses) +{ + int i, j; + bool *cover_map = (bool*)palloc0(nclauses * nmvstats); + + for (i = 0; i < nmvstats; i++) + for (j = 0; j < nclauses; j++) + cover_map[i * nclauses + j] + = bms_is_subset(clauses_attnums[j], stats_attnums[i]); + + return cover_map; +} diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index ac865be..8f625e6 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -3347,7 +3347,8 @@ compute_semi_anti_join_factors(PlannerInfo *root, joinquals, 0, jointype, - sjinfo); + sjinfo, + NIL); /* * Also get the normal inner-join selectivity of the join clauses. @@ -3370,7 +3371,8 @@ compute_semi_anti_join_factors(PlannerInfo *root, joinquals, 0, JOIN_INNER, - &norm_sjinfo); + &norm_sjinfo, + NIL); /* Avoid leaking a lot of ListCells */ if (jointype == JOIN_ANTI) @@ -3537,7 +3539,7 @@ approx_tuple_count(PlannerInfo *root, JoinPath *path, List *quals) Node *qual = (Node *) lfirst(l); /* Note that clause_selectivity will be able to cache its result */ - selec *= clause_selectivity(root, qual, 0, JOIN_INNER, &sjinfo); + selec *= clause_selectivity(root, qual, 0, JOIN_INNER, &sjinfo, NIL); } /* Apply it to the input relation sizes */ @@ -3573,7 +3575,8 @@ set_baserel_size_estimates(PlannerInfo *root, RelOptInfo *rel) rel->baserestrictinfo, 0, JOIN_INNER, - NULL); + NULL, + NIL); rel->rows = clamp_row_est(nrows); @@ -3610,7 +3613,8 @@ get_parameterized_baserel_size(PlannerInfo *root, RelOptInfo *rel, allclauses, rel->relid, /* do not use 0! */ JOIN_INNER, - NULL); + NULL, + NIL); nrows = clamp_row_est(nrows); /* For safety, make sure result is not more than the base estimate */ if (nrows > rel->rows) @@ -3748,12 +3752,14 @@ calc_joinrel_size_estimate(PlannerInfo *root, joinquals, 0, jointype, - sjinfo); + sjinfo, + NIL); pselec = clauselist_selectivity(root, pushedquals, 0, jointype, - sjinfo); + sjinfo, + NIL); /* Avoid leaking a lot of ListCells */ list_free(joinquals); @@ -3765,7 +3771,8 @@ calc_joinrel_size_estimate(PlannerInfo *root, restrictlist, 0, jointype, - sjinfo); + sjinfo, + NIL); pselec = 0.0; /* not used, keep compiler quiet */ } diff --git a/src/backend/optimizer/util/orclauses.c b/src/backend/optimizer/util/orclauses.c index f0acc14..e41508b 100644 --- a/src/backend/optimizer/util/orclauses.c +++ b/src/backend/optimizer/util/orclauses.c @@ -280,7 +280,7 @@ consider_new_or_clause(PlannerInfo *root, RelOptInfo *rel, * saving work later.) */ or_selec = clause_selectivity(root, (Node *) or_rinfo, - 0, JOIN_INNER, NULL); + 0, JOIN_INNER, NULL, NIL); /* * The clause is only worth adding to the query if it rejects a useful @@ -342,7 +342,7 @@ consider_new_or_clause(PlannerInfo *root, RelOptInfo *rel, /* Compute inner-join size */ orig_selec = clause_selectivity(root, (Node *) join_or_rinfo, - 0, JOIN_INNER, &sjinfo); + 0, JOIN_INNER, &sjinfo, NIL); /* And hack cached selectivity so join size remains the same */ join_or_rinfo->norm_selec = orig_selec / or_selec; diff --git a/src/backend/utils/adt/selfuncs.c b/src/backend/utils/adt/selfuncs.c index 04ed07b..3e2f7a4 100644 --- a/src/backend/utils/adt/selfuncs.c +++ b/src/backend/utils/adt/selfuncs.c @@ -1580,13 +1580,15 @@ booltestsel(PlannerInfo *root, BoolTestType booltesttype, Node *arg, case IS_NOT_FALSE: selec = (double) clause_selectivity(root, arg, varRelid, - jointype, sjinfo); + jointype, sjinfo, + NIL); break; case IS_FALSE: case IS_NOT_TRUE: selec = 1.0 - (double) clause_selectivity(root, arg, varRelid, - jointype, sjinfo); + jointype, sjinfo, + NIL); break; default: elog(ERROR, "unrecognized booltesttype: %d", @@ -6209,7 +6211,8 @@ genericcostestimate(PlannerInfo *root, indexSelectivity = clauselist_selectivity(root, selectivityQuals, index->rel->relid, JOIN_INNER, - NULL); + NULL, + NIL); /* * If caller didn't give us an estimate, estimate the number of index @@ -6534,7 +6537,8 @@ btcostestimate(PG_FUNCTION_ARGS) btreeSelectivity = clauselist_selectivity(root, selectivityQuals, index->rel->relid, JOIN_INNER, - NULL); + NULL, + NIL); numIndexTuples = btreeSelectivity * index->rel->tuples; /* @@ -7277,7 +7281,8 @@ gincostestimate(PG_FUNCTION_ARGS) *indexSelectivity = clauselist_selectivity(root, selectivityQuals, index->rel->relid, JOIN_INNER, - NULL); + NULL, + NIL); /* fetch estimated page cost for tablespace containing index */ get_tablespace_page_costs(index->reltablespace, @@ -7509,7 +7514,7 @@ brincostestimate(PG_FUNCTION_ARGS) *indexSelectivity = clauselist_selectivity(root, indexQuals, path->indexinfo->rel->relid, - JOIN_INNER, NULL); + JOIN_INNER, NULL, NIL); *indexCorrelation = 1; /* diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c index be7ba4f..982b66a 100644 --- a/src/backend/utils/misc/guc.c +++ b/src/backend/utils/misc/guc.c @@ -75,6 +75,7 @@ #include "utils/bytea.h" #include "utils/guc_tables.h" #include "utils/memutils.h" +#include "utils/mvstats.h" #include "utils/pg_locale.h" #include "utils/plancache.h" #include "utils/portal.h" @@ -393,6 +394,15 @@ static const struct config_enum_entry row_security_options[] = { }; /* + * Search algorithm for multivariate stats. + */ +static const struct config_enum_entry mvstat_search_options[] = { + {"greedy", MVSTAT_SEARCH_GREEDY, false}, + {"exhaustive", MVSTAT_SEARCH_EXHAUSTIVE, false}, + {NULL, 0, false} +}; + +/* * Options for enum values stored in other modules */ extern const struct config_enum_entry wal_level_options[]; @@ -3648,6 +3658,16 @@ static struct config_enum ConfigureNamesEnum[] = NULL, NULL, NULL }, + { + {"mvstat_search", PGC_USERSET, QUERY_TUNING_OTHER, + gettext_noop("Sets the algorithm used for combining multivariate stats."), + NULL + }, + &mvstat_search_type, + MVSTAT_SEARCH_GREEDY, mvstat_search_options, + NULL, NULL, NULL + }, + /* End-of-list marker */ { {NULL, 0, 0, NULL, NULL}, NULL, 0, NULL, NULL, NULL, NULL diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h index 24003ae..6bfd338 100644 --- a/src/include/optimizer/cost.h +++ b/src/include/optimizer/cost.h @@ -183,11 +183,13 @@ extern Selectivity clauselist_selectivity(PlannerInfo *root, List *clauses, int varRelid, JoinType jointype, - SpecialJoinInfo *sjinfo); + SpecialJoinInfo *sjinfo, + List *conditions); extern Selectivity clause_selectivity(PlannerInfo *root, Node *clause, int varRelid, JoinType jointype, - SpecialJoinInfo *sjinfo); + SpecialJoinInfo *sjinfo, + List *conditions); #endif /* COST_H */ diff --git a/src/include/utils/mvstats.h b/src/include/utils/mvstats.h index 70f79ed..f2fbc11 100644 --- a/src/include/utils/mvstats.h +++ b/src/include/utils/mvstats.h @@ -16,6 +16,14 @@ #include "commands/vacuum.h" +typedef enum MVStatSearchType +{ + MVSTAT_SEARCH_EXHAUSTIVE, /* exhaustive search */ + MVSTAT_SEARCH_GREEDY /* greedy search */ +} MVStatSearchType; + +extern int mvstat_search_type; + /* * Degree of how much MCV item / histogram bucket matches a clause. * This is then considered when computing the selectivity. -- 1.9.3 --------------080401000304040708010509 Content-Type: text/plain Content-Disposition: inline Content-Transfer-Encoding: 8bit MIME-Version: 1.0 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers --------------080401000304040708010509--