pg.ddx.io pgsql-hackers@postgresql.org mailing list archive
help / color / mirror / Atom feedWIP: multivariate statistics / proof of concept
37+ messages / 16 participants
[nested] [flat]
* WIP: multivariate statistics / proof of concept
@ 2014-10-12 22:00 Tomas Vondra <tv@fuzzy.cz>
2014-10-13 07:36 ` Re: WIP: multivariate statistics / proof of concept Albe Laurenz <laurenz.albe@wien.gv.at>
2014-10-29 09:41 ` Re: WIP: multivariate statistics / proof of concept David Rowley <dgrowleyml@gmail.com>
2014-11-13 11:31 ` Re: WIP: multivariate statistics / proof of concept Simon Riggs <simon@2ndQuadrant.com>
2014-12-11 16:53 ` Re: WIP: multivariate statistics / proof of concept Heikki Linnakangas <hlinnakangas@vmware.com>
0 siblings, 4 replies; 37+ messages in thread
From: Tomas Vondra @ 2014-10-12 22:00 UTC (permalink / raw)
To: pgsql-hackers
Hi,
attached is a WIP patch implementing multivariate statistics. The code
certainly is not "ready" - parts of it look as if written by a rogue
chimp who got bored of attempts to type the complete works of William
Shakespeare, and decided to try something different.
I also cut some corners to make it work, and those limitations need to
be fixed before the eventual commit (those are not difficult problems,
but were not necessary for a proof-of-concept patch).
It however seems to be working sufficiently well at this point, enough
to get some useful feedback. So here we go.
I expect to be busy over the next two weeks because of travel, so sorry
for somehow delayed responses. If you happen to attend pgconf.eu next
week (Oct 20-24), we can of course discuss this patch in person.
Goals and basics
----------------
The goal of this patch is allowing users to define multivariate
statistics (i.e. statistics on multiple columns), and improving
estimation when the columns are correlated.
Take for example a table like this:
CREATE TABLE test (a INT, b INT, c INT);
INSERT INTO test SELECT i/10000, i/10000, i/10000
FROM generate_series(1,1000000) s(i);
ANALYZE test;
and do a query like this:
SELECT * FROM test WHERE (a = 10) AND (b = 10) AND (c = 10);
which is estimated like this:
QUERY PLAN
---------------------------------------------------------
Seq Scan on test (cost=0.00..22906.00 rows=1 width=12)
Filter: ((a = 10) AND (b = 10) AND (c = 10))
Planning time: 0.142 ms
(3 rows)
The query of course returns 10.000 rows, but the planner assumes the
columns are independent and thus multiplies the selectivities. And 1/100
for each column means 1/1000000 in total, which is 1 row.
This example is of course somehow artificial, but the problem is far
from uncommon, especially in denormalized datasets (e.g. star schema).
If you ever got an index scan instead of a sequential scan due to poor
estimate, resulting in a query running for hours instead of seconds, you
know the pain.
The patch allows you to do this:
ALTER TABLE test ADD STATISTICS ON (a, b, c);
ANALYZE test;
which then results in this estimate:
QUERY PLAN
------------------------------------------------------------
Seq Scan on test (cost=0.00..22906.00 rows=9667 width=12)
Filter: ((a = 10) AND (b = 10) AND (c = 10))
Planning time: 0.110 ms
(3 rows)
This however is not free - both building such statistics (during
ANALYZE) and using it (during planning) costs some cycles. Even if we
optimize the hell out of it, it won't be entirely free.
One of the design goals in this patch is not to make the ANALYZE or
planning more expensive unless you add such statistics.
Those who add such statistics probably decided that the price is worth
the improved estimates, and lower risk of inefficient plans. If the
planning takes a few more miliseconds, it's probably worth it if you
risk queries running for minutes or hours because of misestimates.
It also does not guarantee the estimates to be always better. There will
be misestimates, although rather in the other direction (independence
assumption usually leads to underestimates, this may lead to
overestimates). However based on my experience from writing the patch I
be I believe it's possible to reasonably limit the extent of such errors
(just like in the single-column histograms, it's related to the bucket
size).
Of course, there will be cases when the old approach is lucky by
accident - there's not much we can do to beat luck. And we can't rely on
it either.
Design overview
---------------
The patch adds a new system catalog, called pg_mv_statistic, which is
used to keep track of requested statistics. There's also a pg_mv_stats
view, showing some basic info about the stats (not all the data).
There are three kinds of statistics
- list of most common combinations of values (MCV list)
- multi-dimensional histogram
- associative rules
The first two are extensions of the single-column stats we already have.
The MCV list is a trivial extension to multiple dimensions, just
tracking combinations and frequencies. The histogram is more complex -
the structure is quite simple (multi-dimensional rectangles) but there's
a lot of ways to build it. But even the current naive and simple
implementation seems to work quite well.
The last kind (associative rules) is an attempt to track "implications"
between columns. It is however an experiment and it's not really used in
the patch so I'll ignore it for now.
I'm not going to explain all the implementation details here - if you
want to learn more, the best way is probably by reading the changes in
those files (probably in this order):
src/include/utils/mvstats.h
src/backend/commands/analyze.c
src/backend/optimizer/path/clausesel.c
I tried to explain the ideas thoroughly in the comments, along with a
lot of TODO/FIXME items related to limitations, explained in the next
section.
Limitations
-----------
As I mentioned, the current patch has a number of practical limitations,
most importantly:
(a) only data types passed by value (no varlena types)
(b) only data types with sort (to be able to build histogram)
(c) no NULL values supported
(d) not handling DROP COLUMN or DROP TABLE and such
(e) limited to stats on 8 columns (max)
(f) optimizer uses single stats per table
(g) limited list of compatible WHERE clauses
(h) incomplete ADD STATISTICS syntax
The first three conditions are really a shortcut to a working patch, and
fixing them should not be difficult.
The limited number of columns is really just a sanity check. It's
possible to increase it, but I doubt stats on more columns will be
practical because of excessive size or poor accuracy.
A better approach is to support combining multiple stats, defined on
various subsets of columns. This is not implemented at the memoment, but
it's certainly on the roadmap. Currently the "smallest" stats covering
the most columns is selected.
Regarding the compatible WHERE clauses, the patch currently handles
conditions of the form
column OPERATOR constant
where operator is one of the comparison operators (=, <, >, =<, >=). In
the future it's possible to add support for more conditions, e.g.
"column IS NULL" or "column OPERATOR column".
The last point is really just "unfinished implementation" - the syntax I
propose is this:
ALTER TABLE ... ADD STATISTICS (options) ON (columns)
where the options influence the MCV list and histogram size, etc. The
options are recognized and may give you an idea of what it might do, but
it's not really used at the moment (except for storing in the
pg_mv_statistic catalog).
Examples
--------
Let's see a few examples of how to define the stats, and what difference
in estimates it makes:
CREATE TABLE test (a INT, b INT c INT);
-- same value in all columns
INSERT INTO test SELECT mod(i,100), mod(i,100), mod(i,100)
FROM generate_series(1,1000000) s(i);
ANALYZE test;
=============== no multivariate stats ============================
SELECT * FROM test WHERE a = 10 AND b = 10;
QUERY PLAN
-------------------------------------------------------------------
Seq Scan on test (cost=0.00..20406.00 rows=101 width=12)
(actual time=0.007..60.902 rows=10000 loops=1)
Filter: ((a = 10) AND (b = 10))
Rows Removed by Filter: 990000
Planning time: 0.119 ms
Execution time: 61.164 ms
(5 rows)
SELECT * FROM test WHERE a = 10 AND b = 10 AND c = 10;
QUERY PLAN
-------------------------------------------------------------------
Seq Scan on test (cost=0.00..22906.00 rows=1 width=12)
(actual time=0.010..56.780 rows=10000 loops=1)
Filter: ((a = 10) AND (b = 10) AND (c = 10))
Rows Removed by Filter: 990000
Planning time: 0.061 ms
Execution time: 56.994 ms
(5 rows)
=============== with multivariate stats ===========================
ALTER TABLE test ADD STATISTICS ON (a, b, c);
ANALYZE test;
SELECT * FROM test WHERE a = 10 AND b = 10;
QUERY PLAN
-------------------------------------------------------------------
Seq Scan on test (cost=0.00..20406.00 rows=10767 width=12)
(actual time=0.007..58.981 rows=10000 loops=1)
Filter: ((a = 10) AND (b = 10))
Rows Removed by Filter: 990000
Planning time: 0.114 ms
Execution time: 59.214 ms
(5 rows)
SELECT * FROM test WHERE a = 10 AND b = 10 AND c = 10;
QUERY PLAN
-------------------------------------------------------------------
Seq Scan on test (cost=0.00..22906.00 rows=10767 width=12)
(actual time=0.008..61.838 rows=10000 loops=1)
Filter: ((a = 10) AND (b = 10) AND (c = 10))
Rows Removed by Filter: 990000
Planning time: 0.088 ms
Execution time: 62.057 ms
(5 rows)
OK, that was rather significant improvement, but it's also trivial
dataset. Let's see something more complicated - the following table has
correlated columns with distributions skewed to 0.
CREATE TABLE test (a INT, b INT, c INT);
INSERT INTO test SELECT r*MOD(i,50),
pow(r,2)*MOD(i,100),
pow(r,4)*MOD(i,500)
FROM (SELECT random() AS r, i
FROM generate_series(1,1000000) s(i)) foo;
ANALYZE test;
SELECT * FROM test WHERE a = 0 AND b = 0;
=============== no multivariate stats ============================
QUERY PLAN
-------------------------------------------------------------------
Seq Scan on test (cost=0.00..20406.00 rows=9024 width=12)
(actual time=0.007..62.969 rows=49503 loops=1)
Filter: ((a = 0) AND (b = 0))
Rows Removed by Filter: 950497
Planning time: 0.057 ms
Execution time: 64.098 ms
(5 rows)
SELECT * FROM test WHERE a = 0 AND b = 0 AND c = 0;
QUERY PLAN
-------------------------------------------------------------------
Seq Scan on test (cost=0.00..22906.00 rows=2126 width=12)
(actual time=0.008..63.862 rows=40770 loops=1)
Filter: ((a = 0) AND (b = 0) AND (c = 0))
Rows Removed by Filter: 959230
Planning time: 0.060 ms
Execution time: 64.794 ms
(5 rows)
=============== with multivariate stats ============================
ALTER TABLE test ADD STATISTICS ON (a, b, c);
ANALYZE test;
db=> SELECT * FROM pg_mv_stats;
schemaname | public
tablename | test
attnums | 1 2 3
mcvbytes | 25904
mcvinfo | nitems=809
histbytes | 568240
histinfo | nbuckets=13772
SELECT * FROM test WHERE a = 0 AND b = 0;
QUERY PLAN
-------------------------------------------------------------------
Seq Scan on test (cost=0.00..20406.00 rows=47717 width=12)
(actual time=0.007..61.782 rows=49503 loops=1)
Filter: ((a = 0) AND (b = 0))
Rows Removed by Filter: 950497
Planning time: 3.181 ms
Execution time: 62.859 ms
(5 rows)
SELECT * FROM test WHERE a = 0 AND b = 0 AND c = 0;
QUERY PLAN
-------------------------------------------------------------------
Seq Scan on test (cost=0.00..22906.00 rows=40567 width=12)
(actual time=0.009..66.685 rows=40770 loops=1)
Filter: ((a = 0) AND (b = 0) AND (c = 0))
Rows Removed by Filter: 959230
Planning time: 0.188 ms
Execution time: 67.593 ms
(5 rows)
regards
Tomas
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[text/x-diff] multivar-stats-v1.patch (139.7K, ../../543AFA15.4080608@fuzzy.cz/2-multivar-stats-v1.patch)
download | inline diff:
diff --git a/src/backend/catalog/Makefile b/src/backend/catalog/Makefile
index b257b02..6e63afe 100644
--- a/src/backend/catalog/Makefile
+++ b/src/backend/catalog/Makefile
@@ -32,6 +32,7 @@ POSTGRES_BKI_SRCS = $(addprefix $(top_srcdir)/src/include/catalog/,\
pg_attrdef.h pg_constraint.h pg_inherits.h pg_index.h pg_operator.h \
pg_opfamily.h pg_opclass.h pg_am.h pg_amop.h pg_amproc.h \
pg_language.h pg_largeobject_metadata.h pg_largeobject.h pg_aggregate.h \
+ pg_mv_statistic.h \
pg_statistic.h pg_rewrite.h pg_trigger.h pg_event_trigger.h pg_description.h \
pg_cast.h pg_enum.h pg_namespace.h pg_conversion.h pg_depend.h \
pg_database.h pg_db_role_setting.h pg_tablespace.h pg_pltemplate.h \
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 9d9d239..68ec1aa 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -150,6 +150,18 @@ CREATE VIEW pg_indexes AS
LEFT JOIN pg_tablespace T ON (T.oid = I.reltablespace)
WHERE C.relkind IN ('r', 'm') AND I.relkind = 'i';
+CREATE VIEW pg_mv_stats AS
+ SELECT
+ N.nspname AS schemaname,
+ C.relname AS tablename,
+ S.stakeys AS attnums,
+ length(S.stamcv) AS mcvbytes,
+ pg_mv_stats_mvclist_info(S.stamcv) AS mcvinfo,
+ length(S.stahist) AS histbytes,
+ pg_mv_stats_histogram_info(S.stahist) AS histinfo
+ FROM (pg_mv_statistic S JOIN pg_class C ON (C.oid = S.starelid))
+ LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace);
+
CREATE VIEW pg_stats AS
SELECT
nspname AS schemaname,
diff --git a/src/backend/commands/analyze.c b/src/backend/commands/analyze.c
index c09ca7e..df51805 100644
--- a/src/backend/commands/analyze.c
+++ b/src/backend/commands/analyze.c
@@ -27,6 +27,7 @@
#include "catalog/indexing.h"
#include "catalog/pg_collation.h"
#include "catalog/pg_inherits_fn.h"
+#include "catalog/pg_mv_statistic.h"
#include "catalog/pg_namespace.h"
#include "commands/dbcommands.h"
#include "commands/tablecmds.h"
@@ -54,7 +55,11 @@
#include "utils/syscache.h"
#include "utils/timestamp.h"
#include "utils/tqual.h"
+#include "utils/fmgroids.h"
+#include "utils/builtins.h"
+#include "utils/mvstats.h"
+#include "access/sysattr.h"
/* Data structure for Algorithm S from Knuth 3.4.2 */
typedef struct
@@ -111,6 +116,62 @@ static Datum std_fetch_func(VacAttrStatsP stats, int rownum, bool *isNull);
static Datum ind_fetch_func(VacAttrStatsP stats, int rownum, bool *isNull);
+/* multivariate statistics (histogram, MCV list, associative rules) */
+
+static void build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
+ int natts, VacAttrStats **vacattrstats);
+static void update_mv_stats(Oid relid,
+ MVHistogram histogram, MCVList mcvlist);
+
+/* multivariate histograms */
+static MVHistogram build_mv_histogram(int numrows, HeapTuple *rows,
+ int2vector *attrs,
+ int attr_cnt, VacAttrStats **vacattrstats,
+ int numrows_total);
+static MVBucket create_initial_mv_bucket(int numrows, HeapTuple *rows,
+ int2vector *attrs, int natts,
+ VacAttrStats **vacattrstats);
+static MVBucket select_bucket_to_partition(int nbuckets, MVBucket * buckets);
+static MVBucket partition_bucket(MVBucket bucket, int2vector *attrs,
+ int natts, VacAttrStats **vacattrstats);
+static MVBucket copy_mv_bucket(MVBucket bucket, uint32 ndimensions);
+
+static void update_bucket_ndistinct(MVBucket bucket, int2vector *attrs,
+ VacAttrStats ** stats);
+static void update_dimension_ndistinct(MVBucket bucket, int dimension,
+ int2vector *attrs,
+ VacAttrStats ** stats,
+ bool update_boundaries);
+/* multivariate MCV list */
+static MCVList build_mv_mcvlist(int numrows, HeapTuple *rows,
+ int2vector *attrs,
+ int natts, VacAttrStats **vacattrstats,
+ int *numrows_filtered);
+
+/* multivariate associative rules */
+static void build_mv_associations(int numrows, HeapTuple *rows,
+ int2vector *attrs,
+ int natts, VacAttrStats **vacattrstats);
+
+/* serialization */
+static bytea * serialize_mv_histogram(MVHistogram histogram);
+static bytea * serialize_mv_mcvlist(MCVList mcvlist);
+
+/* comparators, used when constructing multivariate stats */
+static int compare_scalars_simple(const void *a, const void *b, void *arg);
+static int compare_scalars_partition(const void *a, const void *b, void *arg);
+static int compare_scalars_memcmp(const void *a, const void *b, void *arg);
+static int compare_scalars_memcmp_2(const void *a, const void *b);
+
+static VacAttrStats ** lookup_var_attr_stats(int2vector *attrs,
+ int natts, VacAttrStats **vacattrstats);
+
+/* some debugging methods */
+#ifdef MVSTATS_DEBUG
+static void print_mv_histogram_info(MVHistogram histogram);
+#endif
+
+
/*
* analyze_rel() -- analyze one relation
*/
@@ -469,6 +530,13 @@ do_analyze_rel(Relation onerel, VacuumStmt *vacstmt,
* all analyzable columns. We use a lower bound of 100 rows to avoid
* possible overflow in Vitter's algorithm. (Note: that will also be the
* target in the corner case where there are no analyzable columns.)
+ *
+ * FIXME This sample sizing is mostly OK when computing stats for
+ * individual columns, but when computing multi-variate stats
+ * for multivariate stats (histograms, mcv, ...) it's rather
+ * insufficient. For small number of dimensions it works, but
+ * for complex stats it'd be nice use sample proportional to
+ * the table (say, 0.5% - 1%) instead of a fixed size.
*/
targrows = 100;
for (i = 0; i < attr_cnt; i++)
@@ -571,6 +639,9 @@ do_analyze_rel(Relation onerel, VacuumStmt *vacstmt,
update_attstats(RelationGetRelid(Irel[ind]), false,
thisdata->attr_cnt, thisdata->vacattrstats);
}
+
+ /* Build multivariate stats (if there are any). */
+ build_mv_stats(onerel, numrows, rows, attr_cnt, vacattrstats);
}
/*
@@ -2810,3 +2881,1979 @@ compare_mcvs(const void *a, const void *b)
return da - db;
}
+
+/*
+ * Compute requested multivariate stats, using the rows sampled for the
+ * plain (single-column) stats.
+ *
+ * This fetches a list of stats from pg_mv_statistic, computes the stats
+ * and serializes them back into the catalog (as bytea values).
+ */
+static void
+build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
+ int natts, VacAttrStats **vacattrstats)
+{
+ int i;
+ MVStats mvstats;
+ int nmvstats;
+
+ /*
+ * Fetch defined MV groups from pg_mv_statistic, and then compute
+ * the MV statistics (histograms for now).
+ *
+ * TODO move this to a separate method or something ...
+ */
+ mvstats = list_mv_stats(RelationGetRelid(onerel), &nmvstats, false);
+
+ for (i = 0; i < nmvstats; i++)
+ {
+ MCVList mcvlist = NULL;
+ MVHistogram histogram = NULL;
+ int numrows_filtered = 0;
+
+ /* int2 vector of attnums the stats should be computed on */
+ int2vector * attrs = mvstats[i].stakeys;
+
+ /* check allowed number of dimensions */
+ Assert((attrs->dim1 >= 2) && (attrs->dim1 <= MVSTATS_MAX_DIMENSIONS));
+
+ /*
+ * Analyze associations between pairs of columns.
+ *
+ * FIXME store the identified associations back to pg_mv_statistic
+ */
+ build_mv_associations(numrows, rows, attrs, natts, vacattrstats);
+
+ /* build the MCV list */
+ mcvlist = build_mv_mcvlist(numrows, rows, attrs, natts, vacattrstats, &numrows_filtered);
+
+ /*
+ * Build a multivariate histogram on the columns.
+ *
+ * FIXME remove the rows used to build the MCV from the histogram.
+ * Another option might be subtracting the MCV selectivities
+ * from the histogram, but I'm not sure whether that works
+ * accurately (maybe it introduces additional errors).
+ */
+ if (numrows_filtered > 0)
+ histogram = build_mv_histogram(numrows_filtered, rows, attrs, natts, vacattrstats, numrows);
+
+ /* store the histogram / MCV list in the catalog */
+ update_mv_stats(mvstats[i].mvoid, histogram, mcvlist);
+
+#ifdef MVSTATS_DEBUG
+ print_mv_histogram_info(histogram);
+#endif
+
+ }
+}
+
+/*
+ * Lookup the VacAttrStats info for the selected columns, with indexes
+ * matching the attrs vector (to make it easy to work with when
+ * computing multivariate stats).
+ */
+static VacAttrStats **
+lookup_var_attr_stats(int2vector *attrs, int natts, VacAttrStats **vacattrstats)
+{
+ int i, j;
+ int numattrs = attrs->dim1;
+ VacAttrStats **stats = (VacAttrStats**)palloc0(numattrs * sizeof(VacAttrStats*));
+
+ /* lookup VacAttrStats info for the requested columns (same attnum) */
+ for (i = 0; i < numattrs; i++)
+ {
+ stats[i] = NULL;
+ for (j = 0; j < natts; j++)
+ {
+ if (attrs->values[i] == vacattrstats[j]->tupattnum)
+ {
+ stats[i] = vacattrstats[j];
+ break;
+ }
+ }
+
+ /*
+ * Check that we found the info, that the attnum matches and
+ * that there's the requested 'lt' operator and that the type
+ * is 'passed-by-value'.
+ */
+ Assert(stats[i] != NULL);
+ Assert(stats[i]->tupattnum == attrs->values[i]);
+
+ /* FIXME This is rather ugly way to check for 'ltopr' (which
+ * is defined for 'scalar' attributes).
+ */
+ Assert(stats[i]->compute_stats == compute_scalar_stats);
+
+ /* TODO remove the 'pass by value' requirement */
+ Assert(stats[i]->attrtype->typbyval);
+ }
+
+ return stats;
+}
+
+/*
+ * TODO Add ndistinct estimation, probably the one described in "Towards
+ * Estimation Error Guarantees for Distinct Values, PODS 2000,
+ * p. 268-279" (the ones called GEE, or maybe AE).
+ *
+ * TODO The "combined" ndistinct is more likely to scale with the number
+ * of rows (in the table), because a single column behaving this
+ * way is sufficient for such behavior.
+ */
+static MVBucket
+create_initial_mv_bucket(int numrows, HeapTuple *rows, int2vector *attrs,
+ int natts, VacAttrStats **vacattrstats)
+{
+ int i;
+ int numattrs = attrs->dim1;
+
+ /* info for the interesting attributes only */
+ VacAttrStats **stats = lookup_var_attr_stats(attrs, natts, vacattrstats);
+
+ /* resulting bucket */
+ MVBucket bucket = (MVBucket)palloc0(sizeof(MVBucketData));
+
+ Assert(numrows > 0);
+ Assert(rows != NULL);
+ Assert((numattrs >= 2) && (numattrs <= MVSTATS_MAX_DIMENSIONS));
+
+ /* allocate the per-dimension arrays */
+ bucket->ndistincts = (uint32*)palloc0(numattrs * sizeof(uint32));
+ bucket->nullsonly = (bool*)palloc0(numattrs * sizeof(bool));
+
+ /* inclusiveness boundaries - lower/upper bounds */
+ bucket->min_inclusive = (bool*)palloc0(numattrs * sizeof(bool));
+ bucket->max_inclusive = (bool*)palloc0(numattrs * sizeof(bool));
+
+ /* lower/upper boundaries */
+ bucket->min = (Datum*)palloc0(numattrs * sizeof(Datum));
+ bucket->max = (Datum*)palloc0(numattrs * sizeof(Datum));
+
+ /*
+ * All the sample rows fall into the initial bucket.
+ *
+ * FIXME This is wrong (unless all columns are NOT NULL), because we
+ * skipped the NULL values.
+ */
+ bucket->numrows = numrows;
+ bucket->ntuples = numrows;
+ bucket->rows = rows;
+
+ /*
+ * Update the number of ndistinct combinations in the bucket (which
+ * we use when selecting bucket to partition), and then number of
+ * distinct values for each partition (which we use when choosing
+ * which dimension to split).
+ */
+ update_bucket_ndistinct(bucket, attrs, stats);
+
+ for (i = 0; i < numattrs; i++)
+ update_dimension_ndistinct(bucket, i, attrs, stats, true);
+
+ /*
+ * The initial bucket was not split at all, so we'll start with the
+ * first dimension in the next round (index = 0).
+ */
+ bucket->last_split_dimension = -1;
+
+ return bucket;
+}
+
+/*
+ * TODO Fix to handle arbitrarily-sized histograms (not just 2D ones)
+ * and call the right output procedures (for the particular type).
+ *
+ * TODO This should somehow fetch info about the data types, and use
+ * the appropriate output functions to print the boundary values.
+ * Right now this prints the 8B value as an integer.
+ *
+ * TODO Also, provide a special function for 2D histogram, printing
+ * a gnuplot script (with rectangles).
+ *
+ * TODO For string types (once supported) we can sort the strings first,
+ * assign them a sequence of integers and use the original values
+ * as labels.
+ */
+#ifdef MVSTATS_DEBUG
+static void
+print_mv_histogram_info(MVHistogram histogram)
+{
+ int i = 0;
+
+ elog(WARNING, "histogram nbuckets=%d", histogram->nbuckets);
+
+ for (i = 0; i < histogram->nbuckets; i++)
+ {
+ MVBucket bucket = histogram->buckets[i];
+ elog(WARNING, " bucket %d : ndistinct=%f ntuples=%d min=[%ld, %ld], max=[%ld, %ld] distinct=[%d,%d]",
+ i, bucket->ndistinct, bucket->numrows,
+ bucket->min[0], bucket->min[1], bucket->max[0], bucket->max[1],
+ bucket->ndistincts[0], bucket->ndistincts[1]);
+ }
+}
+#endif
+
+/*
+ * A very simple partitioning selection criteria - choose the bucket
+ * with the highest number of distinct values.
+ *
+ * Returns either pointer to the bucket selected to be partitioned,
+ * or NULL if there are no buckets that may be split (i.e. all buckets
+ * contain a single distinct value).
+ *
+ * TODO Consider other partitioning criteria (v-optimal, maxdiff etc.).
+ *
+ * TODO Allowing the bucket to degenerate to a single combination of
+ * values makes it rather strange MCV list. Maybe we should use
+ * higher lower boundary, or maybe make the selection criteria
+ * more complex (e.g. consider number of rows in the bucket, etc.).
+ *
+ * That however is different from buckets 'degenerated' only for
+ * some dimensions (e.g. half of them), which is perfectly
+ * appropriate for statistics on a combination of low and high
+ * cardinality columns.
+ */
+static MVBucket
+select_bucket_to_partition(int nbuckets, MVBucket * buckets)
+{
+ int i;
+ int ndistinct = 1; /* if ndistinct=1, we can't split the bucket */
+ MVBucket bucket = NULL;
+
+ for (i = 0; i < nbuckets; i++)
+ {
+ /* if the ndistinct count is higher, use this bucket */
+ if (buckets[i]->ndistinct > ndistinct) {
+ bucket = buckets[i];
+ ndistinct = buckets[i]->ndistinct;
+ }
+ }
+
+ /* may be NULL if there are not buckets with (ndistinct>1) */
+ return bucket;
+}
+
+/*
+ * A simple bucket partitioning implementation - splits the dimensions in
+ * a round-robin manner (considering only those with ndistinct>1). That
+ * is first a dimension 0 is split, then 1, 2, ... until reaching the
+ * end of attribute list, and then wrapping back to 0. Of course,
+ * dimensions with a single distinct value are skipped.
+ *
+ * This is essentially what Muralikrishna/DeWitt described in their SIGMOD
+ * article (M. Muralikrishna, David J. DeWitt: Equi-Depth Histograms For
+ * Estimating Selectivity Factors For Multi-Dimensional Queries. SIGMOD
+ * Conference 1988: 28-36).
+ *
+ * There are multiple histogram options, centered around the partitioning
+ * criteria, specifying both how to choose a bucket and the dimension
+ * most in need of a split. For a nice summary and general overview, see
+ * "rK-Hist : an R-Tree based histogram for multi-dimensional selectivity
+ * estimation" thesis by J. A. Lopez, Concordia University, p.34-37 (and
+ * possibly p. 32-34 for explanation of the terms).
+ *
+ * This splits the bucket by tweaking the existing one, and returning the
+ * new bucket (essentially shrinking the existing one in-place and returning
+ * the other "half" as a new bucket). The caller is responsible for adding
+ * the new bucket into the list of buckets.
+ *
+ * TODO It requires care to prevent splitting only one dimension and not
+ * splitting another one at all (which might happen easily in case of
+ * strongly dependent columns - e.g. y=x).
+ *
+ * TODO Should probably consider statistics target for the columns (e.g. to
+ * split dimensions with higher statistics target more frequently).
+ */
+static MVBucket
+partition_bucket(MVBucket bucket, int2vector *attrs,
+ int natts, VacAttrStats **vacattrstats)
+{
+ int i;
+ int dimension;
+ int numattrs = attrs->dim1;
+
+ Datum split_value;
+ MVBucket new_bucket;
+
+ /* needed for sort, when looking for the split value */
+ bool isNull;
+ int nvalues = 0;
+ StdAnalyzeData * mystats = NULL;
+ ScalarItem * values = (ScalarItem*)palloc0(bucket->numrows * sizeof(ScalarItem));
+ SortSupportData ssup;
+
+ /* looking for the split value */
+ int ndistinct = 1; /* number of distinct values below current value */
+ int nrows = 1; /* number of rows below current value */
+
+ /* needed when splitting the values */
+ HeapTuple * oldrows = bucket->rows;
+ int oldnrows = bucket->numrows;
+
+ /* info for the interesting attributes only */
+ VacAttrStats **stats = lookup_var_attr_stats(attrs, natts, vacattrstats);
+
+ /*
+ * We can't split buckets with a single distinct value (this also
+ * disqualifies NULL-only dimensions). Also, there has to be multiple
+ * sample rows (otherwise, how could there be more distinct values).
+ */
+ Assert(bucket->ndistinct > 1);
+ Assert(bucket->numrows > 1);
+ Assert((numattrs >= 2) && (numattrs <= MVSTATS_MAX_DIMENSIONS));
+
+ /*
+ * Look for the next dimension to split, in a round robin manner.
+ * We'll use the first one with (ndistinc > 1).
+ *
+ * If we happen to wrap around, something clearly went wrong (we
+ * can't mess with the last_split_dimension directly, because we
+ * couldn't do this check).
+ */
+ dimension = bucket->last_split_dimension;
+ while (true)
+ {
+ dimension = (dimension + 1) % numattrs;
+
+ if (bucket->ndistincts[dimension] > 1)
+ break;
+
+ /* if we ran the previous split dimension, it's infinite loop */
+ Assert(dimension != bucket->last_split_dimension);
+ }
+
+ /* Remember the dimension for the next split of this bucket. */
+ bucket->last_split_dimension = dimension;
+
+ /*
+ * Walk through the selected dimension, collect and sort the values
+ * and then choose the value to use as the new boundary.
+ */
+ mystats = (StdAnalyzeData *) stats[dimension]->extra_data;
+
+ /* initialize sort support, etc. */
+ memset(&ssup, 0, sizeof(ssup));
+ ssup.ssup_cxt = CurrentMemoryContext;
+
+ /* We always use the default collation for statistics */
+ ssup.ssup_collation = DEFAULT_COLLATION_OID;
+ ssup.ssup_nulls_first = false;
+
+ PrepareSortSupportFromOrderingOp(mystats->ltopr, &ssup);
+
+ for (i = 0; i < bucket->numrows; i++)
+ {
+ /* remember the index of the sample row, to make the partitioning simpler */
+ values[nvalues].value = heap_getattr(bucket->rows[i], attrs->values[dimension],
+ stats[dimension]->tupDesc, &isNull);
+ values[nvalues].tupno = i;
+
+ /* no NULL values allowed here (we don't do splits by null-only dimensions) */
+ Assert(!isNull);
+
+ nvalues++;
+ }
+
+ /* sort the array (pass-by-value datum */
+ qsort_arg((void *) values, nvalues, sizeof(ScalarItem),
+ compare_scalars_partition, (void *) &ssup);
+
+ /*
+ * We know there are bucket->ndistincts[dimension] distinct values
+ * in this dimension, and we want to split this into half, so walk
+ * through the array and stop once we see (ndistinct/2) values.
+ *
+ * We always choose the "next" value, i.e. (n/2+1)-th distinct value,
+ * and use it as an exclusive upper boundary (and inclusive lower
+ * boundary).
+ *
+ * TODO Maybe we should use "average" of the two middle distinct
+ * values (at least for even distinct counts), but that would
+ * require being able to do an average (which does not work
+ * for non-arithmetic types).
+ *
+ * TODO Another option is to look for a split that'd give about
+ * 50% tuples (not distinct values) in each partition. That
+ * might work better when there are a few very frequent
+ * values, and many rare ones.
+ */
+ split_value = values[0].value;
+ for (i = 1; i < bucket->numrows; i++)
+ {
+ /* count distinct values */
+ if (values[i].value != values[i-1].value)
+ ndistinct += 1;
+
+ /* once we've seen 1/2 distinct values (and use the value) */
+ if (ndistinct > bucket->ndistincts[dimension] / 2)
+ {
+ split_value = values[i].value;
+ break;
+ }
+
+ /* keep track how many rows belong to the first bucket */
+ nrows += 1;
+ }
+
+ Assert(nrows > 0);
+ Assert(nrows < bucket->numrows);
+
+ /* create the new bucket as a (incomplete) copy of the one being partitioned. */
+ new_bucket = copy_mv_bucket(bucket, numattrs);
+
+ /*
+ * Do the actual split of the chosen dimension, using the split value as the
+ * upper bound for the existing bucket, and lower bound for the new one.
+ */
+ bucket->max[dimension] = split_value;
+ new_bucket->min[dimension] = split_value;
+
+ bucket->max_inclusive[dimension] = false;
+ new_bucket->max_inclusive[dimension] = true;
+
+ /*
+ * Redistribute the sample tuples using the 'ScalarItem->tupno'
+ * index. We know 'nrows' rows should remain in the original
+ * bucket and the rest goes to the new one.
+ */
+
+ bucket->rows = (HeapTuple*)palloc0(nrows * sizeof(HeapTuple));
+ new_bucket->rows = (HeapTuple*)palloc0((oldnrows - nrows) * sizeof(HeapTuple));
+
+ bucket->numrows = nrows;
+ new_bucket->numrows = (oldnrows - nrows);
+
+ /*
+ * The first nrows should go to the first bucket, the rest should
+ * go to the new one. Use the tupno field to get the actual HeapTuple
+ * row from the original array of sample rows.
+ */
+ for (i = 0; i < nrows; i++)
+ memcpy(&bucket->rows[i], &oldrows[values[i].tupno], sizeof(HeapTuple));
+
+ for (i = nrows; i < oldnrows; i++)
+ memcpy(&new_bucket->rows[i-nrows], &oldrows[values[i].tupno], sizeof(HeapTuple));
+
+ /* update ndistinct values for the buckets (total and per dimension) */
+ update_bucket_ndistinct(bucket, attrs, stats);
+ update_bucket_ndistinct(new_bucket, attrs, stats);
+
+ /*
+ * TODO We don't need to do this for the dimension we used for split,
+ * because we know how many distinct values went to each partition.
+ */
+ for (i = 0; i < numattrs; i++)
+ {
+ update_dimension_ndistinct(bucket, i, attrs, stats, false);
+ update_dimension_ndistinct(new_bucket, i, attrs, stats, false);
+ }
+
+ pfree(oldrows);
+ pfree(values);
+
+ return new_bucket;
+}
+
+/*
+ * Copy a histogram bucket. The copy does not include the build-time
+ * data, i.e. sampled rows etc.
+ */
+static MVBucket
+copy_mv_bucket(MVBucket bucket, uint32 ndimensions)
+{
+ MVBucket new_bucket = (MVBucket)palloc0(sizeof(MVBucketData));
+
+ /* Copy only the attributes that will stay the same after the split, and
+ * we'll recompute the rest after the split. */
+
+ new_bucket->last_split_dimension = bucket->last_split_dimension;
+
+ /* allocate the per-dimension arrays */
+ new_bucket->ndistincts = (uint32*)palloc0(ndimensions * sizeof(uint32));
+ new_bucket->nullsonly = (bool*)palloc0(ndimensions * sizeof(bool));
+
+ /* inclusiveness boundaries - lower/upper bounds */
+ new_bucket->min_inclusive = (bool*)palloc0(ndimensions * sizeof(bool));
+ new_bucket->max_inclusive = (bool*)palloc0(ndimensions * sizeof(bool));
+
+ /* lower/upper boundaries */
+ new_bucket->min = (Datum*)palloc0(ndimensions * sizeof(Datum));
+ new_bucket->max = (Datum*)palloc0(ndimensions * sizeof(Datum));
+
+ /* copy data */
+ memcpy(new_bucket->nullsonly, bucket->nullsonly, ndimensions * sizeof(bool));
+
+ memcpy(new_bucket->min_inclusive, bucket->min_inclusive, ndimensions*sizeof(bool));
+ memcpy(new_bucket->min, bucket->min, ndimensions*sizeof(Datum));
+
+ memcpy(new_bucket->max_inclusive, bucket->max_inclusive, ndimensions*sizeof(bool));
+ memcpy(new_bucket->max, bucket->max, ndimensions*sizeof(Datum));
+
+ return new_bucket;
+}
+
+/*
+ * Counts the number of distinct values in the bucket. This just copies
+ * the Datum values into a simple array, and sorts them using memcmp-based
+ * comparator. That means it only works for pass-by-value data types
+ * (assuming they don't use collations etc.)
+ *
+ * FIXME Make this work with all types (not just pass-by-value ones).
+ *
+ * TODO This might evaluate and store the distinct counts for all
+ * possible attribute combinations. The assumption is this might be
+ * useful for estimating things like GROUP BY cardinalities (e.g.
+ * in cases when some buckets contain a lot of low-frequency
+ * combinations, and other buckets contain few high-frequency ones).
+ *
+ * But it's unclear whether it's worth the price. Computing this
+ * is actually quite cheap, because it may be evaluated at the very
+ * end, when the buckets are rather small (so sorting it in 2^N ways
+ * is not a big deal). Assuming the partitioning algorithm does not
+ * use these values to do the decisions, of course (the current
+ * algorithm does not).
+ *
+ * The overhead with storing, fetching and parsing the data is more
+ * concerning - adding 2^N values per bucket (even if it's just
+ * a 1B or 2B value) would significantly bloat the histogram, and
+ * thus the impact on optimizer. Which is not really desirable.
+ *
+ * TODO This only updates the ndistinct for the sample (or bucket), but
+ * we eventually need an estimate of the total number of distinct
+ * values in the dataset. It's possible to either use the current
+ * 1D approach (i.e., if it's more than 10% of the sample, assume
+ * it's proportional to the number of rows). Or it's possible to
+ * implement the estimator suggested in the article, supposedly
+ * giving 'optimal' estimates (w.r.t. probability of error).
+ */
+static void
+update_bucket_ndistinct(MVBucket bucket, int2vector *attrs, VacAttrStats ** stats)
+{
+ int i, j, idx = 0;
+ int numattrs = attrs->dim1;
+ Size len = sizeof(Datum) * numattrs;
+ bool isNull;
+
+ /*
+ * We could collect this while walking through all the attributes
+ * above (this way we have to call heap_getattr twice).
+ */
+ Datum * values = palloc0(bucket->numrows * numattrs * sizeof(Datum));
+
+ for (j = 0; j < bucket->numrows; j++)
+ for (i = 0; i < numattrs; i++)
+ values[idx++] = heap_getattr(bucket->rows[j], attrs->values[i],
+ stats[i]->tupDesc, &isNull);
+
+ qsort_arg((void *) values, bucket->numrows, sizeof(Datum) * numattrs,
+ compare_scalars_memcmp, &len);
+
+ bucket->ndistinct = 1;
+
+ for (i = 1; i < bucket->numrows; i++)
+ if (memcmp(&values[i * numattrs], &values[(i-1) * numattrs], len) != 0)
+ bucket->ndistinct += 1;
+
+ pfree(values);
+
+}
+
+/*
+ * Count distinct values per bucket dimension.
+ *
+ * TODO Remove unnecessary parameters - don't pass in the whole arrays,
+ * just the proper elements.
+ */
+static void
+update_dimension_ndistinct(MVBucket bucket, int dimension, int2vector *attrs,
+ VacAttrStats ** stats, bool update_boundaries)
+{
+ int j;
+ int nvalues = 0;
+ bool isNull;
+ Datum * values = (Datum*)palloc0(bucket->numrows * sizeof(Datum));
+ SortSupportData ssup;
+
+ StdAnalyzeData * mystats = (StdAnalyzeData *) stats[dimension]->extra_data;
+
+ memset(&ssup, 0, sizeof(ssup));
+ ssup.ssup_cxt = CurrentMemoryContext;
+
+ /* We always use the default collation for statistics */
+ ssup.ssup_collation = DEFAULT_COLLATION_OID;
+ ssup.ssup_nulls_first = false;
+
+ PrepareSortSupportFromOrderingOp(mystats->ltopr, &ssup);
+
+ for (j = 0; j < bucket->numrows; j++)
+ {
+ values[nvalues] = heap_getattr(bucket->rows[j], attrs->values[dimension],
+ stats[dimension]->tupDesc, &isNull);
+
+ /* ignore NULL values */
+ if (! isNull)
+ nvalues++;
+ }
+
+ /* there's always at least 1 distinct value (may be NULL) */
+ bucket->ndistincts[dimension] = 1;
+
+ /* if there are only NULL values in the column, mark it so and continue
+ * with the next one */
+ if (nvalues == 0)
+ {
+ pfree(values);
+ bucket->nullsonly[dimension] = true;
+ return;
+ }
+
+ /* sort the array (pass-by-value datum */
+ qsort_arg((void *) values, nvalues, sizeof(Datum),
+ compare_scalars_simple, (void *) &ssup);
+
+ /*
+ * Update min/max boundaries to the smallest bounding box. Generally, this
+ * needs to be done only when constructing the initial bucket.
+ */
+ if (update_boundaries)
+ {
+ /* store the min/max values */
+ bucket->min[dimension] = values[0];
+ bucket->min_inclusive[dimension] = true;
+
+ bucket->max[dimension] = values[nvalues-1];
+ bucket->max_inclusive[dimension] = true;
+ }
+
+ /*
+ * Walk through the array and count distinct values by comparing
+ * succeeding values.
+ *
+ * FIXME This only works for pass-by-value types (i.e. not VARCHARs etc.).
+ */
+ for (j = 1; j < nvalues; j++) {
+ if (values[j] != values[j-1])
+ bucket->ndistincts[dimension] += 1;
+ }
+
+ pfree(values);
+}
+
+/*
+ * Fetch list of MV stats defined on a table, without the actual data
+ * for histograms, MCV lists etc.
+ */
+MVStats
+list_mv_stats(Oid relid, int *nstats, bool built_only)
+{
+ Relation indrel;
+ SysScanDesc indscan;
+ ScanKeyData skey;
+ HeapTuple htup;
+ MVStats result;
+
+ /* start with 16 items, that should be enough for most cases */
+ int maxitems = 16;
+ result = (MVStats)palloc0(sizeof(MVStatsData) * maxitems);
+ *nstats = 0;
+
+ /* Prepare to scan pg_mv_statistic for entries having indrelid = this rel. */
+ ScanKeyInit(&skey,
+ Anum_pg_mv_statistic_starelid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(relid));
+
+ indrel = heap_open(MvStatisticRelationId, AccessShareLock);
+ indscan = systable_beginscan(indrel, MvStatisticRelidIndexId, true,
+ NULL, 1, &skey);
+
+ while (HeapTupleIsValid(htup = systable_getnext(indscan)))
+ {
+ Form_pg_mv_statistic stats = (Form_pg_mv_statistic) GETSTRUCT(htup);
+
+ /*
+ * Skip statistics that were not computed yet (if only stats
+ * that were already built were requested)
+ */
+ if (built_only && (! (stats->hist_built || stats->mcv_built || stats->assoc_built)))
+ continue;
+
+ /* double the array size if needed */
+ if (*nstats == maxitems)
+ {
+ maxitems *= 2;
+ result = (MVStats)repalloc(result, sizeof(MVStatsData) * maxitems);
+ }
+
+ result[*nstats].mvoid = HeapTupleGetOid(htup);
+ result[*nstats].stakeys = buildint2vector(stats->stakeys.values, stats->stakeys.dim1);
+ result[*nstats].hist_built = stats->hist_built;
+ result[*nstats].mcv_built = stats->mcv_built;
+ result[*nstats].assoc_built = stats->assoc_built;
+ *nstats += 1;
+ }
+
+ systable_endscan(indscan);
+
+ heap_close(indrel, AccessShareLock);
+
+ /* TODO maybe save the list into relcache, as in RelationGetIndexList
+ * (which was used as an inspiration of this one)?. */
+
+ return result;
+}
+
+
+/*
+ * Serialize the MV histogram into a bytea value.
+ *
+ * The serialized first deduplicates the boundary values into a separate
+ * array, and uses 2B indexes when serializing the buckets. This stores
+ * a significant amount of space because each bucket split adds a single
+ * new boundary value, so e.g. with 4 attributes and 8191 splits (thus
+ * 8192 buckets), there are only ~8200 distinct boundary values.
+ *
+ * But as each bucket has 8 boundary values (4+4), that's ~64k Datums.
+ * That's roughly 65kB vs. 512kB, but we haven't included the indexes
+ * used to reference the boundary values. By using int16 indexes (which
+ * should be more than enough for all reasonable histogram sizes),
+ * this amounts to ~128kB (8192*8*2). So in total it's ~196kB vs. 512kB,
+ * i.e. more than 2x compression, which is nice.
+ *
+ * The implementation is simple - walk through the buckets, collect all
+ * the boundary values, keep only distinct values (in a sorted array)
+ * and then replace the values with indexes (using binary search).
+ *
+ * It's possible to either serialize/deserialize the histogram into
+ * a MVHistogram, or create a special structure working with this
+ * compressed structure (and keep MVBucket/MVHistogram only for the
+ * building phase). This might actually work better thanks to better
+ * CPU cache hit ratio, and simpler deserialization.
+ *
+ * This encoding will probably prevent automatic varlena compression,
+ * because first part of the serialized bytea will be an array of unique
+ * values (although sorted), and pglz decides whether to compress by
+ * trying to compress the first part (~1kB or so). Which will be poor,
+ * due to the lack of repetition.
+ *
+ * But in this case this is probably desirable - the data in general
+ * won't be really compressible (in addition to the 2x compression we
+ * got thanks to the encoding). In a sense the encoding scheme is
+ * actually a context-aware compression (usually compressing to ~30%).
+ * So this seems appropriate in this case.
+ *
+ * FIXME Make this work with arbitrary types.
+ *
+ * TODO Try to keep the compressed form, instead of deserializing it to
+ * MVHistogram/MVBucket.
+ *
+ * TODO We might get a bit better compression by considering the actual
+ * data type length. The current implementation treats all data as
+ * 8B values, but for INT it's actually 4B etc. OTOH this is only
+ * related to the lookup table, and most of the space is occupied
+ * by the buckets (with int16 indexes). And we don't have type info
+ * at the moment, so it would be difficult (but we'll nedd it to
+ * support all types, so maybe then).
+ */
+static bytea *
+serialize_mv_histogram(MVHistogram histogram)
+{
+ int i = 0, j = 0;
+
+ /* total size (histogram header + all buckets) */
+ Size total_len;
+ char *tmp = NULL;
+ bytea *result = NULL;
+
+ /* we need to accumulate all boundary values (min/max) */
+ int idx = 0;
+ int max_values = histogram->nbuckets * histogram->ndimensions * 2;
+ Datum * values = (Datum*)palloc0(max_values * sizeof(Datum));
+ Size len = sizeof(Datum);
+
+ /* we'll collect unique boundary values into this */
+ int ndistinct = 0;
+ Datum *lookup = NULL;
+
+ /*
+ * Collect the boundary values first, sort them and generate a small
+ * array with only distinct values.
+ */
+ for (i = 0; i < histogram->nbuckets; i++)
+ {
+ for (j = 0; j < histogram->ndimensions; j++)
+ {
+ values[idx++] = histogram->buckets[i]->min[j];
+ values[idx++] = histogram->buckets[i]->max[j];
+ }
+ }
+
+ /*
+ * We've allocated just enough space for all boundary values, but
+ * this may change once we start handling NULL values (as we'll
+ * probably skip those).
+ *
+ * Also, we expect at least one boundary value at this moment.
+ */
+ Assert(max_values == idx);
+ Assert(idx > 1);
+
+ /*
+ * Sort the collected boundary values using a simple memcmp-based
+ * comparator (this won't work for pass-by-reference types), and
+ * then walk the data and count the distinct values.
+ */
+ qsort((void *) values, idx, len, compare_scalars_memcmp_2);
+
+ ndistinct = 1;
+ for (i = 1; i < max_values; i++)
+ ndistinct += (values[i-1] != values[i]) ? 1 : 0;
+
+ /*
+ * At this moment we can allocate the bytea value (and we'll collect
+ * the boundary values directly into it).
+ *
+ * The bytea will be structured like this:
+ *
+ * - varlena header : VARHDRSZ
+ * - histogram header : offsetof(MVHistogram,buckets)
+ * - number of boundary values : sizeof(uint32)
+ * - boundary values : ndistinct * sizeof(Datum)
+ * - buckets : nbuckets * BUCKET_SIZE_SERIALIZED
+ *
+ * We'll assume 2B indexes into the boundary values, because each
+ * bucket 'split' introduces one boundary value. Moreover, multiple
+ * splits may introduce the same value, so this should be enough for
+ * at least 65k buckets (and likely more). That's more than enough
+ * for reasonable histogram sizes.
+ */
+
+ Assert(ndistinct <= 65536);
+
+ total_len = VARHDRSZ + offsetof(MVHistogramData, buckets) +
+ (sizeof(uint32) + ndistinct * sizeof(Datum)) +
+ histogram->nbuckets * BUCKET_SIZE_SERIALIZED(histogram->ndimensions);
+
+ result = (bytea*)palloc0(total_len);
+ tmp = VARDATA(result);
+
+ SET_VARSIZE(result, total_len);
+
+ /* copy the global histogram header */
+ memcpy(tmp, histogram, offsetof(MVHistogramData, buckets));
+ tmp += offsetof(MVHistogramData, buckets);
+
+ /*
+ * Copy the number of distinct values, and then all the distinct
+ * values currently stored in the 'values' array (sorted).
+ */
+ memcpy(tmp, &ndistinct, sizeof(uint32));
+ tmp += sizeof(uint32);
+
+ lookup = (Datum*)tmp;
+
+ for (i = 0; i < max_values; i++)
+ {
+ /* skip values that are equal to the previous one */
+ if ((i > 0) && (values[i-1] == values[i]))
+ continue;
+
+ memcpy(tmp, &values[i], sizeof(Datum));
+ tmp += sizeof(Datum);
+ }
+
+ Assert(tmp - (char*)lookup == ndistinct * sizeof(Datum));
+
+ /* now serialize all the buckets - first the header, without the
+ * variable-length part, then all the variable length parts */
+ for (i = 0; i < histogram->nbuckets; i++)
+ {
+ MVBucket bucket = histogram->buckets[i];
+ uint16 indexes[histogram->ndimensions];
+
+ /* write the common bucket header */
+ memcpy(tmp, bucket, offsetof(MVBucketData, ndistincts));
+ tmp += offsetof(MVBucketData, ndistincts);
+
+ /* per-dimension ndistincts / nullsonly */
+ memcpy(tmp, bucket->ndistincts, sizeof(uint32)*histogram->ndimensions);
+ tmp += sizeof(uint32)*histogram->ndimensions;
+
+ memcpy(tmp, bucket->nullsonly, sizeof(bool)*histogram->ndimensions);
+ tmp += sizeof(bool)*histogram->ndimensions;
+
+ memcpy(tmp, bucket->min_inclusive, sizeof(bool)*histogram->ndimensions);
+ tmp += sizeof(bool)*histogram->ndimensions;
+
+ memcpy(tmp, bucket->max_inclusive, sizeof(bool)*histogram->ndimensions);
+ tmp += sizeof(bool)*histogram->ndimensions;
+
+ /* and now translate the min (and then max) boundaries to indexes */
+ for (j = 0; j < histogram->ndimensions; j++)
+ {
+ Datum *v = (Datum*)bsearch(&bucket->min[j], lookup, ndistinct,
+ sizeof(Datum), compare_scalars_memcmp_2);
+
+ Assert(v != NULL);
+ indexes[j] = (v - lookup); /* Datum arithmetics (not char) */
+ Assert(indexes[j] < ndistinct); /* we have to be within the array */
+ }
+
+ memcpy(tmp, indexes, sizeof(uint16)*histogram->ndimensions);
+ tmp += sizeof(uint16)*histogram->ndimensions;
+
+ for (j = 0; j < histogram->ndimensions; j++)
+ {
+ Datum *v = (Datum*)bsearch(&bucket->max[j], lookup, ndistinct,
+ sizeof(Datum), compare_scalars_memcmp_2);
+ Assert(v != NULL);
+ indexes[j] = (v - lookup); /* Datum arithmetics (not char) */
+ Assert(indexes[j] < ndistinct); /* we have to be within the array */
+ }
+
+ memcpy(tmp, indexes, sizeof(uint16)*histogram->ndimensions);
+ tmp += sizeof(uint16)*histogram->ndimensions;
+ }
+
+ return result;
+}
+
+/*
+ * Reverse to serialize histogram. This essentially expands the serialized
+ * form back to MVHistogram / MVBucket.
+ */
+MVHistogram
+deserialize_mv_histogram(bytea * data)
+{
+ int i = 0, j = 0;
+
+ Size expected_length;
+ char *tmp = NULL;
+ MVHistogram histogram;
+
+ uint32 nlookup; /* Datum lookup table */
+ Datum *lookup = NULL;
+
+ if (data == NULL)
+ return NULL;
+
+ /* get pointer to the data part of the varlena */
+ tmp = VARDATA(data);
+
+ histogram = (MVHistogram)palloc0(sizeof(MVHistogramData));
+
+ /* copy the histogram header in place */
+ memcpy(histogram, tmp, offsetof(MVHistogramData, buckets));
+ tmp += offsetof(MVHistogramData, buckets);
+
+ if (histogram->magic != MVHIST_MAGIC)
+ {
+ pfree(histogram);
+ elog(WARNING, "not a MV Histogram (magic number mismatch)");
+ return NULL;
+ }
+
+ Assert(histogram->type == MVHIST_TYPE_BASIC);
+ Assert(histogram->nbuckets > 0);
+ Assert(histogram->nbuckets <= MVHIST_MAX_BUCKETS);
+ Assert(histogram->ndimensions > 0);
+ Assert(histogram->ndimensions <= MVSTATS_MAX_DIMENSIONS);
+
+ /* now, get the size of the lookup table */
+ memcpy(&nlookup, tmp, sizeof(uint32));
+ tmp += sizeof(uint32);
+ lookup = (Datum*)tmp;
+
+ /* skip to the first bucket */
+ tmp += sizeof(Datum) * nlookup;
+
+ /* check the total serialized length */
+ expected_length = offsetof(MVHistogramData, buckets) +
+ sizeof(uint32) + nlookup * sizeof(Datum) +
+ histogram->nbuckets * BUCKET_SIZE_SERIALIZED(histogram->ndimensions);
+
+ /* check serialized length */
+ if (VARSIZE_ANY_EXHDR(data) != expected_length)
+ {
+ elog(ERROR, "invalid MV histogram serialized size (expected %ld, got %ld)",
+ VARSIZE_ANY_EXHDR(data), expected_length);
+ return NULL;
+ }
+
+ /* allocate bucket pointers */
+ histogram->buckets = (MVBucket*)palloc0(histogram->nbuckets * sizeof(MVBucket));
+
+ /* deserialize the buckets, one by one */
+ for (i = 0; i < histogram->nbuckets; i++)
+ {
+ /* don't allocate space for the build-only fields */
+ MVBucket bucket = (MVBucket)palloc0(offsetof(MVBucketData, rows));
+ uint16 *indexes = NULL;
+
+ /* write the common bucket header */
+ memcpy(bucket, tmp, offsetof(MVBucketData, ndistincts));
+ tmp += offsetof(MVBucketData, ndistincts);
+
+ /* per-dimension ndistincts / nullsonly */
+ bucket->ndistincts = (uint32*)palloc0(sizeof(uint32)*histogram->ndimensions);
+ memcpy(bucket->ndistincts, tmp, sizeof(uint32)*histogram->ndimensions);
+ tmp += sizeof(uint32)*histogram->ndimensions;
+
+ bucket->nullsonly = (bool*)palloc0(sizeof(bool)*histogram->ndimensions);
+ memcpy(bucket->nullsonly, tmp, sizeof(bool)*histogram->ndimensions);
+ tmp += sizeof(bool)*histogram->ndimensions;
+
+ bucket->min_inclusive = (bool*)palloc0(sizeof(bool)*histogram->ndimensions);
+ memcpy(bucket->min_inclusive, tmp, sizeof(bool)*histogram->ndimensions);
+ tmp += sizeof(bool)*histogram->ndimensions;
+
+ bucket->max_inclusive = (bool*)palloc0(sizeof(bool)*histogram->ndimensions);
+ memcpy(bucket->max_inclusive, tmp, sizeof(bool)*histogram->ndimensions);
+ tmp += sizeof(bool)*histogram->ndimensions;
+
+ /* translate the indexes back to Datum values */
+ bucket->min = (Datum*)palloc0(sizeof(Datum)*histogram->ndimensions);
+ bucket->max = (Datum*)palloc0(sizeof(Datum)*histogram->ndimensions);
+
+ indexes = (uint16*)tmp;
+ tmp += sizeof(uint16) * histogram->ndimensions;
+ for (j = 0; j < histogram->ndimensions; j++)
+ memcpy(&bucket->min[j], &lookup[indexes[j]], sizeof(Datum));
+
+ indexes = (uint16*)tmp;
+ tmp += sizeof(uint16) * histogram->ndimensions;
+ for (j = 0; j < histogram->ndimensions; j++)
+ memcpy(&bucket->max[j], &lookup[indexes[j]], sizeof(Datum));
+
+ histogram->buckets[i] = bucket;
+ }
+
+ return histogram;
+}
+
+/*
+ * Serialize MCV list into a bytea value.
+ *
+ * This does not use any kind of deduplication (compared to histogram
+ * serialization), as we don't expect the same efficiency here.
+ *
+ * This simply writes a MCV header (number of items, ...) and then Datum
+ * values for all attribute of a item, followed by the item frequency
+ * (as a double).
+ */
+static bytea *
+serialize_mv_mcvlist(MCVList mcvlist)
+{
+ int i;
+
+ /* we need to store nitems, and each needs ndimension * Datum, plus a double */
+ Size len = VARHDRSZ + offsetof(MCVListData, items) + mcvlist->nitems * (sizeof(Datum) * mcvlist->ndimensions + sizeof(double));
+
+ bytea * output = (bytea*)palloc0(len);
+
+ char * tmp = VARDATA(output);
+
+ SET_VARSIZE(output, len);
+
+ /* first, store the number of dimensions / items */
+ memcpy(tmp, mcvlist, offsetof(MCVListData, items));
+ tmp += offsetof(MCVListData, items);
+
+ /* now, walk through the items and store values + frequency for each MCV item */
+ for (i = 0; i < mcvlist->nitems; i++)
+ {
+ memcpy(tmp, mcvlist->items[i]->values, mcvlist->ndimensions * sizeof(Datum));
+ tmp += mcvlist->ndimensions * sizeof(Datum);
+
+ memcpy(tmp, &mcvlist->items[i]->frequency, sizeof(double));
+ tmp += sizeof(double);
+ }
+
+ return output;
+
+}
+
+MCVList deserialize_mv_mcvlist(bytea * data)
+{
+ int i;
+ Size expected_size;
+ MCVList mcvlist;
+ char *tmp;
+
+ if (data == NULL)
+ return NULL;
+
+ if (VARSIZE_ANY_EXHDR(data) < offsetof(MCVListData,items))
+ elog(ERROR, "invalid MCV Size %ld (expected at least %ld)",
+ VARSIZE_ANY_EXHDR(data), offsetof(MCVListData,items));
+
+ /* read the MCV list header */
+ mcvlist = (MCVList)palloc0(sizeof(MCVListData));
+
+ /* initialize pointer to the data part (skip the varlena header) */
+ tmp = VARDATA(data);
+
+ /* get the header and perform basic sanity checks */
+ memcpy(mcvlist, tmp, offsetof(MCVListData,items));
+ tmp += offsetof(MCVListData,items);
+
+ if (mcvlist->magic != MVSTAT_MCV_MAGIC)
+ elog(ERROR, "invalid MCV magic %d (expected %dd)",
+ mcvlist->magic, MVSTAT_MCV_MAGIC);
+
+ if (mcvlist->type != MVSTAT_MCV_TYPE_BASIC)
+ elog(ERROR, "invalid MCV type %d (expected %dd)",
+ mcvlist->type, MVSTAT_MCV_TYPE_BASIC);
+
+ Assert(mcvlist->nitems > 0);
+ Assert((mcvlist->ndimensions >= 2) && (mcvlist->ndimensions <= MVSTATS_MAX_DIMENSIONS));
+
+ /* what bytea size do we expect for those parameters */
+ expected_size = offsetof(MCVListData,items) +
+ mcvlist->nitems * (sizeof(Datum) * mcvlist->ndimensions + sizeof(double));
+
+ if (VARSIZE_ANY_EXHDR(data) != expected_size)
+ elog(ERROR, "invalid MCV Size %ld (expected %ld)",
+ VARSIZE_ANY_EXHDR(data), expected_size);
+
+ /* allocate space for the MCV items */
+ mcvlist->items = (MCVItem*)palloc0(sizeof(MCVItem) * mcvlist->nitems);
+
+ for (i = 0; i < mcvlist->nitems; i++)
+ {
+ MCVItem item = (MCVItem)palloc0(offsetof(MCVItemData, values) +
+ mcvlist->ndimensions * sizeof(Datum));
+
+ memcpy(item->values, tmp, mcvlist->ndimensions * sizeof(Datum));
+ tmp += mcvlist->ndimensions * sizeof(Datum);
+
+ memcpy(&item->frequency, tmp, sizeof(double));
+ tmp += sizeof(double);
+
+ mcvlist->items[i] = item;
+ }
+
+ return mcvlist;
+}
+
+static void
+update_mv_stats(Oid mvoid, MVHistogram histogram, MCVList mcvlist)
+{
+ HeapTuple stup,
+ oldtup;
+ Datum values[Natts_pg_mv_statistic];
+ bool nulls[Natts_pg_mv_statistic];
+ bool replaces[Natts_pg_mv_statistic];
+
+ Relation sd = heap_open(MvStatisticRelationId, RowExclusiveLock);
+
+ memset(nulls, 1, Natts_pg_mv_statistic * sizeof(bool));
+ memset(replaces, 0, Natts_pg_mv_statistic * sizeof(bool));
+ memset(values, 0, Natts_pg_mv_statistic * sizeof(Datum));
+
+ /*
+ * Construct a new pg_mv_statistic tuple - replace only the histogram
+ * and MCV list, depending whether it actually was computed.
+ */
+ if (histogram != NULL)
+ {
+ nulls[Anum_pg_mv_statistic_stahist-1] = false;
+ values[Anum_pg_mv_statistic_stahist - 1]
+ = PointerGetDatum(serialize_mv_histogram(histogram));
+ }
+
+ if (mcvlist != NULL)
+ {
+ nulls[Anum_pg_mv_statistic_stamcv -1] = false;
+ values[Anum_pg_mv_statistic_stamcv - 1]
+ = PointerGetDatum(serialize_mv_mcvlist(mcvlist));
+ }
+
+ /* always replace the value (either by bytea or NULL) */
+ replaces[Anum_pg_mv_statistic_stahist-1] = true;
+ replaces[Anum_pg_mv_statistic_stamcv -1] = true;
+
+ /* always change the availability flags */
+ nulls[Anum_pg_mv_statistic_hist_built-1] = false;
+ nulls[Anum_pg_mv_statistic_mcv_built -1] = false;
+
+ replaces[Anum_pg_mv_statistic_hist_built -1] = true;
+ replaces[Anum_pg_mv_statistic_mcv_built -1] = true;
+
+ values[Anum_pg_mv_statistic_hist_built -1] = BoolGetDatum(histogram != NULL);
+ values[Anum_pg_mv_statistic_mcv_built -1] = BoolGetDatum(mcvlist != NULL);
+
+ /* Is there already a pg_mv_statistic tuple for this attribute? */
+ oldtup = SearchSysCache1(MVSTATOID,
+ ObjectIdGetDatum(mvoid));
+
+ if (HeapTupleIsValid(oldtup))
+ {
+ /* Yes, replace it */
+ stup = heap_modify_tuple(oldtup,
+ RelationGetDescr(sd),
+ values,
+ nulls,
+ replaces);
+ ReleaseSysCache(oldtup);
+ simple_heap_update(sd, &stup->t_self, stup);
+ }
+ else
+ elog(ERROR, "invalid pg_mv_statistic record (oid=%d)", mvoid);
+
+ /* update indexes too */
+ CatalogUpdateIndexes(sd, stup);
+
+ heap_freetuple(stup);
+
+ heap_close(sd, RowExclusiveLock);
+}
+
+
+/* MV stats */
+
+Datum
+pg_mv_stats_histogram_info(PG_FUNCTION_ARGS)
+{
+ bytea *data = PG_GETARG_BYTEA_P(0);
+ char *result;
+
+ MVHistogram hist = deserialize_mv_histogram(data);
+
+ result = palloc0(128);
+ snprintf(result, 128, "nbuckets=%d", hist->nbuckets);
+
+ PG_RETURN_TEXT_P(cstring_to_text(result));
+}
+
+Datum
+pg_mv_stats_mvclist_info(PG_FUNCTION_ARGS)
+{
+ bytea *data = PG_GETARG_BYTEA_P(0);
+ char *result;
+
+ MCVList mcvlist = deserialize_mv_mcvlist(data);
+
+ result = palloc0(128);
+ snprintf(result, 128, "nitems=%d", mcvlist->nitems);
+
+ pfree(mcvlist);
+
+ PG_RETURN_TEXT_P(cstring_to_text(result));
+}
+
+Datum
+pg_mv_stats_histogram_gnuplot(PG_FUNCTION_ARGS)
+{
+ int i = 0;
+
+ /* FIXME (handle the length properly using StringBuilder */
+ Size len = 1024*1024;
+ char *buffer = palloc0(len);
+ char *str = buffer;
+ bytea *data = PG_GETARG_BYTEA_P(0);
+
+ MVHistogram hist = deserialize_mv_histogram(data);
+
+ for (i = 0; i < hist->nbuckets; i++)
+ {
+ str += snprintf(str, len - (str - buffer),
+ "set object %d rect from %ld,%ld to %ld,%ld lw 1\n",
+ (i+1),
+ hist->buckets[i]->min[0], hist->buckets[i]->min[1],
+ hist->buckets[i]->max[0], hist->buckets[i]->max[1]);
+ }
+
+ PG_RETURN_TEXT_P(cstring_to_text(buffer));
+
+}
+
+bytea *
+fetch_mv_histogram(Oid mvoid)
+{
+ Relation indrel;
+ SysScanDesc indscan;
+ ScanKeyData skey;
+ HeapTuple htup;
+ bytea *stahist = NULL;
+
+ /* Prepare to scan pg_mv_statistic for entries having indrelid = this rel. */
+ ScanKeyInit(&skey,
+ ObjectIdAttributeNumber,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(mvoid));
+
+ indrel = heap_open(MvStatisticRelationId, AccessShareLock);
+ indscan = systable_beginscan(indrel, MvStatisticOidIndexId, true,
+ NULL, 1, &skey);
+
+ while (HeapTupleIsValid(htup = systable_getnext(indscan)))
+ {
+ bool isnull = false;
+ Datum hist = SysCacheGetAttr(MVSTATOID, htup,
+ Anum_pg_mv_statistic_stahist, &isnull);
+
+ Assert(!isnull);
+
+ stahist = DatumGetByteaP(hist);
+
+ break;
+ }
+
+ systable_endscan(indscan);
+
+ heap_close(indrel, AccessShareLock);
+
+ /* TODO maybe save the list into relcache, as in RelationGetIndexList
+ * (which was used as an inspiration of this one)?. */
+
+ return stahist;
+}
+
+bytea *
+fetch_mv_mcvlist(Oid mvoid)
+{
+ Relation indrel;
+ SysScanDesc indscan;
+ ScanKeyData skey;
+ HeapTuple htup;
+ bytea *mcvlist = NULL;
+
+ /* Prepare to scan pg_mv_statistic for entries having indrelid = this rel. */
+ ScanKeyInit(&skey,
+ ObjectIdAttributeNumber,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(mvoid));
+
+ indrel = heap_open(MvStatisticRelationId, AccessShareLock);
+ indscan = systable_beginscan(indrel, MvStatisticOidIndexId, true,
+ NULL, 1, &skey);
+
+ while (HeapTupleIsValid(htup = systable_getnext(indscan)))
+ {
+ bool isnull = false;
+ Datum tmp = SysCacheGetAttr(MVSTATOID, htup,
+ Anum_pg_mv_statistic_stamcv, &isnull);
+
+ Assert(!isnull);
+
+ mcvlist = DatumGetByteaP(tmp);
+
+ break;
+ }
+
+ systable_endscan(indscan);
+
+ heap_close(indrel, AccessShareLock);
+
+ /* TODO maybe save the list into relcache, as in RelationGetIndexList
+ * (which was used as an inspiration of this one)?. */
+
+ return mcvlist;
+}
+
+int
+mv_get_index(AttrNumber varattno, int2vector * stakeys)
+{
+ int i, idx = 0;
+ for (i = 0; i < stakeys->dim1; i++)
+ {
+ if (stakeys->values[i] < varattno)
+ idx += 1;
+ else
+ break;
+ }
+ return idx;
+}
+
+/*
+ * Building a multivariate algorithm. In short it first creates a single
+ * bucket containing all the rows, and then repeatedly split is by first
+ * searching for the bucket / dimension most in need of a split.
+ *
+ * The current criteria is rather simple, by looking at the number of
+ * distinct values (combination of column values for bucket, column
+ * values for a dimension). This is somehow naive, but seems to work
+ * quite well. See the discussion at select_bucket_to_partition and
+ * partition_bucket for more details about alternative algorithms.
+ *
+ * So the current algorithm looks like this:
+ *
+ * while [not reaching maximum number of buckets]
+ *
+ * choose bucket to partition (max distinct combinations)
+ * if no bucket to partition
+ * terminate the algorithm
+ *
+ * choose bucket dimension to partition (max distinct values)
+ * split the bucket into two buckets
+ *
+ */
+static MVHistogram
+build_mv_histogram(int numrows, HeapTuple *rows, int2vector *attrs,
+ int attr_cnt, VacAttrStats **vacattrstats,
+ int numrows_total)
+{
+ int i;
+ int ndistinct;
+ int numattrs = attrs->dim1;
+ int ndistincts[numattrs];
+
+ MVHistogram histogram = (MVHistogram)palloc0(sizeof(MVHistogramData));
+
+ HeapTuple * rows_copy = (HeapTuple*)palloc0(numrows * sizeof(HeapTuple));
+ memcpy(rows_copy, rows, sizeof(HeapTuple) * numrows);
+
+ Assert((numattrs >= 2) && (numattrs <= MVSTATS_MAX_DIMENSIONS));
+
+ histogram->ndimensions = numattrs;
+
+ histogram->magic = MVHIST_MAGIC;
+ histogram->type = MVHIST_TYPE_BASIC;
+ histogram->nbuckets = 1;
+
+ /* create max buckets (better than repalloc for short-lived objects) */
+ histogram->buckets = (MVBucket*)palloc0(MVHIST_MAX_BUCKETS * sizeof(MVBucket));
+
+ /* create the initial bucket, covering the whole sample set */
+ histogram->buckets[0] = create_initial_mv_bucket(numrows, rows_copy, attrs,
+ attr_cnt, vacattrstats);
+
+ ndistinct = histogram->buckets[0]->ndistinct;
+
+ /* keep the global ndistinct values */
+ for (i = 0; i < numattrs; i++)
+ ndistincts[i] = histogram->buckets[0]->ndistincts[i];
+
+ while (histogram->nbuckets < MVHIST_MAX_BUCKETS)
+ {
+ MVBucket bucket = select_bucket_to_partition(histogram->nbuckets, histogram->buckets);
+
+ /* no more buckets to partition */
+ if (bucket == NULL)
+ break;
+
+ histogram->buckets[histogram->nbuckets] = partition_bucket(bucket, attrs,
+ attr_cnt, vacattrstats);
+
+ histogram->nbuckets += 1;
+ }
+
+ /*
+ * FIXME store the histogram in a catalog in a serialized form (simple for
+ * pass-by-value, more complicated for buckets on varlena types)
+ */
+ for (i = 0; i < histogram->nbuckets; i++)
+ {
+ int d;
+ histogram->buckets[i]->ntuples = (histogram->buckets[i]->numrows * 1.0) / numrows_total;
+ histogram->buckets[i]->ndistinct = (histogram->buckets[i]->ndistinct * 1.0) / ndistinct;
+
+ for (d = 0; d < numattrs; d++)
+ histogram->buckets[i]->ndistincts[d] = (histogram->buckets[i]->ndistincts[d] * 1.0) / ndistincts[d];
+ }
+
+ return histogram;
+
+}
+
+/*
+ * Mine associations between the columns, in the form (A => B).
+ *
+ * At the moment this only works for associations between two columns,
+ * but it might be useful to mine for rules involving multiple columns
+ * on the left side. That is rules [A,B] => C and so on. Handling
+ * multiple columns on the right side is not necessary, because such
+ * rules may be decomposed into a set of rules, one for each column.
+ * I.e. A => [B,C] is exactly the same as (A => B) & (A => C).
+ *
+ * Those rules don't immediately identify redundant clauses, because the
+ * user may choose "incompatible conditions" (e.g. by using a zip code
+ * and a mismatching city) and so on. This should however be easy to
+ * identify from a histogram, because the conditions will match a bucket
+ * with low frequencies.
+ *
+ * The question is whether this can be useful when we have a histogram,
+ * because such incompatible conditions should result in not matching
+ * any buckets (or matching only buckets with low frequencies).
+ *
+ * The problem is that histograms work like this when the sorting is
+ * compatible with the meaning of the data. We're often using data types
+ * that support sorting (e.g. INT, BIGING) as a kind of labels where
+ * the sorting really does not make much sense. Sorting by ZIP code will
+ * result in sorting the cities quite randomly, and similarly for most
+ * surrogate primary / foreign keys. In such cases the histograms are
+ * pretty useless.
+ *
+ * So, a good approach might be testing the independence of the data
+ * (by building a contingency table) and buildint the MV histogram only
+ * when there's a dependency. For the 'label' data this should notice
+ * the histogram is useless. So we won't build it (and we may use that
+ * as a sign supporting the association rule).
+ *
+ * Another option is to look at selectivity of A and B separately, and
+ * then use the minimum of those.
+ *
+ * TODO investigate using histogram and MCV list to confirm the
+ * associative rule
+ *
+ * TODO investigate statistical testing of the distribution (to decide
+ * whether it makes sense to build the histogram)
+ *
+ * TODO Using a min/max of selectivities would probably make more sense
+ * for the associated columns.
+ */
+static void
+build_mv_associations(int numrows, HeapTuple *rows, int2vector *attrs,
+ int natts, VacAttrStats **vacattrstats)
+{
+ int i;
+ bool isNull;
+ Size len = 2 * sizeof(Datum); /* only simple associations a => b */
+ int numattrs = attrs->dim1;
+
+ /* TODO Maybe this should be somehow related to the number of
+ * distinct columns in the two columns we're currently analyzing.
+ * Assuming the distribution is uniform, we should expected to
+ * observe in the sample - we can then use the average group
+ * size as a threshold. That seems better than a static approach.
+ */
+ int min_group_size = 10;
+
+ /* dimension indexes we'll check for associations [a => b] */
+ int dima, dimb;
+
+ /* info for the interesting attributes only
+ *
+ * TODO Compute this only once and pass it to all the methods
+ * that need it.
+ */
+ VacAttrStats **stats = lookup_var_attr_stats(attrs, natts, vacattrstats);
+
+ /* We'll reuse the same array for all the combinations */
+ Datum * values = (Datum*)palloc0(numrows * 2 * sizeof(Datum));
+
+ Assert(numattrs >= 2);
+
+ for (dima = 0; dima < numattrs; dima++)
+ {
+
+ for (dimb = 0; dimb < numattrs; dimb++)
+ {
+
+ int supporting = 0;
+ int contradicting = 0;
+
+ Datum val_a, val_b;
+ int violations = 0;
+ int group_size = 0;
+
+ int supporting_rows = 0;
+
+ /* skip (dima==dimb) */
+ if (dima == dimb)
+ continue;
+
+ /*
+ * FIXME Not sure if this handles NULL values properly (not sure
+ * how to do that). We assume that NULL means 0 for now,
+ * handling it just like any other value.
+ */
+ for (i = 0; i < numrows; i++)
+ {
+ values[i*2] = heap_getattr(rows[i], attrs->values[dima], stats[dima]->tupDesc, &isNull);
+ values[i*2+1] = heap_getattr(rows[i], attrs->values[dimb], stats[dimb]->tupDesc, &isNull);
+ }
+
+ qsort_arg((void *) values, numrows, sizeof(Datum) * 2, compare_scalars_memcmp, &len);
+
+ /*
+ * Walk through the array, split it into rows according to
+ * the A value, and count distinct values in the other one.
+ * If there's a single B value for the whole group, we count
+ * it as supporting the association, otherwise we count it
+ * as contradicting.
+ *
+ * Furthermore we require a group to have at least a certain
+ * number of rows to be considered useful. When contradicting,
+ * use it always.
+ */
+
+ /* start with values from the first row */
+ val_a = values[0];
+ val_b = values[1];
+ group_size = 1;
+
+ for (i = 1; i < numrows; i++)
+ {
+ if (values[2*i] != val_a) /* end of the group */
+ {
+ /*
+ * If there are no contradicting rows, count it as
+ * supporting (otherwise contradicting), but only if
+ * the group is large enough.
+ *
+ * The requirement of a minimum group size makes it
+ * impossible to identify [unique,unique] cases, but
+ * that's probably a different case. This is more
+ * about [zip => city] associations etc.
+ */
+ supporting += ((violations == 0) && (group_size >= min_group_size)) ? 1 : 0;
+ contradicting += (violations != 0) ? 1 : 0;
+
+ supporting_rows += ((violations == 0) && (group_size >= min_group_size)) ? group_size : 0;
+
+ /* current values start a new group */
+ val_a = values[2*i];
+ val_b = values[2*i+1];
+ violations = 0;
+ group_size = 1;
+ }
+ else
+ {
+ if (values[2*i+1] != val_b) /* mismatch of a B value */
+ {
+ val_b = values[2*i+1];
+ violations += 1;
+ }
+
+ group_size += 1;
+ }
+ }
+
+ /* FIXME handle the last group */
+ supporting += ((violations == 0) && (group_size >= min_group_size)) ? 1 : 0;
+ contradicting += (violations != 0) ? 1 : 0;
+ supporting_rows += ((violations == 0) && (group_size >= min_group_size)) ? group_size : 0;
+
+ /*
+ * See if the number of rows supporting the association is at least
+ * 10x the number of rows violating the hypothetical rule.
+ *
+ * TODO This is rather arbitrary limit - I guess it's possible to do
+ * some math to come up with a better rule (e.g. testing a hypothesis
+ * 'this is due to randomness'). We can create a contingency table
+ * from the values and use it for testing. Possibly only when
+ * there are no contradicting rows?
+ *
+ * TODO Also, if (a => b) and (b => a) at the same time, it pretty much
+ * means the columns have the same values (or one is a 'label'),
+ * making the conditions rather redundant. Although it's possible
+ * that the query uses incompatible combination of values.
+ */
+ if (supporting_rows > (numrows - supporting_rows) * 10)
+ {
+ // elog(WARNING, "%d => %d : supporting=%d contradicting=%d", dima, dimb, supporting, contradicting);
+ }
+
+ }
+ }
+
+ pfree(values);
+
+}
+
+/*
+ * Compute the list of most common items, where item is a combination of
+ * values for all the columns. For small number of distinct values, we
+ * may be able to represent the distribution pretty exactly, with
+ * per-item statistics.
+ *
+ * If we can represent the distribution using a MCV list only, it's great
+ * because that allows much better estimates (especially for equality).
+ * Such discrete distributions are also easier to combine (more
+ * efficient and more accurate) than when using histograms.
+ *
+ * FIXME This does not handle NULL values at the moment.
+ *
+ * TODO When computing equality selectivity (a=1 AND b=2), we can do that
+ * pretty exactly assuming (a) we hit a MCV item and (b) the
+ * histogram is built on those two columns only (i.e. there are no
+ * other columns). In that case we can estimate the selectivity
+ * using only the MCV.
+ *
+ * When we don't hit a MCV item, we can use the frequency of the
+ * least probable MCV item as upper bound of the selectivity
+ * (otherwise it'd get into the MCV list). Again, this only works
+ * when the histogram size matches the restricted columns.
+ *
+ * When the histogram is larger (i.e. there are additional columns),
+ * we can't be sure how is the selectivity distributed among the MCV
+ * list and the histogram (we may get several MCV items matching
+ * the conditions and several histogram buckets at the same time).
+ *
+ * In this case we can probably clamp the selectivity by minimum of
+ * selectivities for each condition. For example if we know the
+ * number of distinct values for each column, we can use 1/ndistinct
+ * as a per-column estimate. Or rather 1/ndistinct + selectivity
+ * derived from the MCV list.
+ *
+ * If there's no histogram (thus the distribution is approximated
+ * only by the MCV list), the size of the stats (whether there are
+ * some other columns, not referenced in the conditions) does not
+ * matter. We can do pretty accurate estimation using the MCV.
+ *
+ * TODO Currently there's no logic to consider building only a MCV list
+ * (and not building the histogram at all).
+ *
+ * TODO For types that don't reasonably support ordering (either because
+ * the type does not support that or when the user adds some option
+ * to the ADD STATISTICS command - e.g. UNSORTED_STATS), building
+ * the histogram may be pointless and inefficient. This is esp.
+ * true for varlena types that may be quite large and a large MCV
+ * list may be a better choice, because it makes equality estimates
+ * more accurate. Due to the unsorted nature, range queries on those
+ * attributes are rather useless anyway.
+ *
+ * Another thing is that by restricting to MCV list and equality
+ * conditions, we can use hash values instead of long varlena values.
+ * The equality estimation will be very accurate.
+ *
+ * This however complicates matching the columns to available
+ * statistics, as it will require matching clauses (not columns) to
+ * stats. And it may get quite complex - e.g. what if there are
+ * multiple clauses, each compatible with different stats subset?
+ *
+ * FIXME Create a special-purpose type for MCV items (instead of a plain
+ * Datum array, which is very difficult to work with).
+ */
+static MCVList
+build_mv_mcvlist(int numrows, HeapTuple *rows, int2vector *attrs,
+ int natts, VacAttrStats **vacattrstats,
+ int *numrows_filtered)
+{
+ int i, j, idx = 0;
+ int numattrs = attrs->dim1;
+ Size len = sizeof(Datum) * numattrs;
+ bool isNull;
+ int ndistinct = 0;
+ int mcv_threshold = 0;
+ int count = 0;
+ int nitems = 0;
+
+ MCVList mcvlist = NULL;
+
+ VacAttrStats **stats = lookup_var_attr_stats(attrs, natts, vacattrstats);
+
+ /*
+ * We could collect this while walking through all the attributes
+ * above (this way we have to call heap_getattr twice).
+ *
+ * TODO We're using Datum (8B), even for data types smaller than this
+ * (notably int4 and float4). Maybe we could save some space here,
+ * although it seems the bytea compression will handle it just fine.
+ */
+ Datum * values = palloc0(numrows * numattrs * sizeof(Datum));
+
+ for (j = 0; j < numrows; j++)
+ for (i = 0; i < numattrs; i++)
+ values[idx++] = heap_getattr(rows[j], attrs->values[i], stats[i]->tupDesc, &isNull);
+
+ qsort_arg((void *) values, numrows, sizeof(Datum) * numattrs, compare_scalars_memcmp, &len);
+
+ /*
+ * Count the number of distinct values - we need this to determine
+ * the threshold (125% of the average frequency).
+ */
+ ndistinct = 1;
+ for (i = 1; i < numrows; i++)
+ if (memcmp(&values[i * numattrs], &values[(i-1) * numattrs], len) != 0)
+ ndistinct += 1;
+
+ /*
+ * Determine how many groups actually exceed the threshold, and then
+ * walk the array again and collect them into an array.
+ *
+ * TODO for now the threshold is the same as in the single-column
+ * case (average + 25%), but maybe that's worth revisiting
+ *
+ * TODO see if we can fit all the distinct values in the MCV list
+ */
+ mcv_threshold = 1.25 * numrows / ndistinct;
+ mcv_threshold = (mcv_threshold < 4) ? 4 : mcv_threshold;
+
+ /*
+ * If there are less than some number of items, store all with at
+ * least two rows in the sample.
+ *
+ * FIXME We can do this only if we believe we got all the distinct
+ * values of the table.
+ */
+ if (ndistinct <= MVSTAT_MCVLIST_MAX_ITEMS)
+ mcv_threshold = 2;
+
+ count = 1;
+ for (i = 1; i <= numrows; i++)
+ {
+ /* last row or a new group */
+ if ((i == numrows) || (memcmp(&values[i * numattrs], &values[(i-1) * numattrs], len) != 0))
+ {
+ /* count the MCV item if exceeding the threshold */
+ if (count >= mcv_threshold)
+ nitems += 1;
+
+ count = 1;
+ }
+ else /* same group, just increase the number of items */
+ count += 1;
+ }
+
+ /* by default we keep all the rows (even if there's no MCV list) */
+ *numrows_filtered = numrows;
+
+ /* we know the number of mcvitems, now collect them in a 2nd pass */
+ if (nitems > 0)
+ {
+ /* we need to store the frequency for each group, so (numattrs + 1) */
+ mcvlist = (MCVList)palloc0(sizeof(MCVListData));
+
+ mcvlist->magic = MVSTAT_MCV_MAGIC;
+ mcvlist->type = MVSTAT_MCV_TYPE_BASIC;
+ mcvlist->ndimensions = numattrs;
+ mcvlist->nitems = nitems;
+ mcvlist->items = (MCVItem*)palloc0(sizeof(MCVItem)*nitems);
+
+ /* now repeat the same loop as above, but this time copy the data
+ * for items exceeding the threshold */
+ count = 1;
+ nitems = 0;
+ for (i = 1; i <= numrows; i++)
+ {
+
+ /* last row or a new group */
+ if ((i == numrows) || (memcmp(&values[i * numattrs], &values[(i-1) * numattrs], len) != 0))
+ {
+ /* count the MCV item if exceeding the threshold (and copy into the array) */
+ if (count >= mcv_threshold)
+ {
+ /* first, allocate the item (with the proper size of values) */
+ MCVItem item = (MCVItem)palloc0(offsetof(MCVItemData, values) +
+ sizeof(Datum)*mcvlist->ndimensions);
+
+ /* then copy values from the _previous_ group */
+ memcpy(item->values, &values[(i-1)*numattrs], len);
+
+ /* and finally the group frequency */
+ item->frequency = (double)count / numrows;
+
+ mcvlist->items[nitems] = item;
+ nitems += 1;
+ }
+
+ count = 1;
+ }
+ else /* same group, just increase the number of items */
+ count += 1;
+ }
+
+ /* make sure the loops are consistent */
+ Assert(nitems == mcvlist->nitems);
+
+ /*
+ * Remove the rows matching the MCV items.
+ *
+ * FIXME This implementation is rather naive, effectively O(N^2).
+ * As the MCV list grows, the check will take longer and
+ * longer. And as the number of sampled rows increases (by
+ * increasing statistics target), it will take longer and
+ * longer. One option is to sort the MCV items first and
+ * then perform a binary search.
+ */
+ if (nitems == ndistinct) /* all rows are covered by MCV items */
+ *numrows_filtered = 0;
+ else /* (nitems < ndistinct) && (nitems > 0) */
+ {
+ int nfiltered = 0;
+ HeapTuple *rows_filtered = (HeapTuple*)palloc0(sizeof(HeapTuple) * numrows);
+
+ /* walk through the tuples, compare the values to MCV items */
+ for (i = 0; i < numrows; i++)
+ {
+ bool match = false;
+ Datum keys[numattrs];
+
+ /* collect the key values */
+ for (j = 0; j < numattrs; j++)
+ keys[j] = heap_getattr(rows[i], attrs->values[j], stats[j]->tupDesc, &isNull);
+
+ /* scan through the MCV list for matches */
+ for (j = 0; j < mcvlist->nitems; j++)
+ if (memcmp(keys, mcvlist->items[j]->values, sizeof(Datum)*numattrs) == 0)
+ {
+ match = true;
+ break;
+ }
+
+ /* if no match in the MCV list, copy the row into the filtered ones */
+ if (! match)
+ memcpy(&rows_filtered[nfiltered++], &rows[i], sizeof(HeapTuple));
+ }
+
+ /* replace the first part */
+ memcpy(rows, rows_filtered, sizeof(HeapTuple) * nfiltered);
+ *numrows_filtered = nfiltered;
+
+ pfree(rows_filtered);
+
+ }
+ }
+
+ pfree(values);
+
+ /*
+ * TODO Single-dimensional MCV is stored sorted by frequency (descending).
+ * Maybe this should be stored like that too?
+ */
+
+ return mcvlist;
+}
+
+/* multi-variate stats comparator */
+
+/*
+ * qsort_arg comparator for sorting Datums (MV stats)
+ *
+ * This does not maintain the tupnoLink array.
+ */
+static int
+compare_scalars_simple(const void *a, const void *b, void *arg)
+{
+ Datum da = *(Datum*)a;
+ Datum db = *(Datum*)b;
+ SortSupport ssup= (SortSupport) arg;
+
+ return ApplySortComparator(da, false, db, false, ssup);
+}
+
+/*
+ * qsort_arg comparator for sorting data when partitioning a MV bucket
+ */
+static int
+compare_scalars_partition(const void *a, const void *b, void *arg)
+{
+ Datum da = ((ScalarItem*)a)->value;
+ Datum db = ((ScalarItem*)b)->value;
+ SortSupport ssup= (SortSupport) arg;
+
+ return ApplySortComparator(da, false, db, false, ssup);
+}
+
+/*
+ * qsort_arg comparator for sorting Datum[] (row of Datums) when
+ * counting distinct values.
+ */
+static int
+compare_scalars_memcmp(const void *a, const void *b, void *arg)
+{
+ Size len = *(Size*)arg;
+
+ return memcmp(a, b, len);
+}
+
+static int
+compare_scalars_memcmp_2(const void *a, const void *b)
+{
+ return memcmp(a, b, sizeof(Datum));
+}
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index cb16c53..28bad78 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -34,6 +34,7 @@
#include "catalog/pg_foreign_table.h"
#include "catalog/pg_inherits.h"
#include "catalog/pg_inherits_fn.h"
+#include "catalog/pg_mv_statistic.h"
#include "catalog/pg_namespace.h"
#include "catalog/pg_opclass.h"
#include "catalog/pg_rowsecurity.h"
@@ -89,7 +90,7 @@
#include "utils/syscache.h"
#include "utils/tqual.h"
#include "utils/typcache.h"
-
+#include "utils/mvstats.h"
/*
* ON COMMIT action list
@@ -137,8 +138,9 @@ static List *on_commits = NIL;
#define AT_PASS_ADD_COL 5 /* ADD COLUMN */
#define AT_PASS_ADD_INDEX 6 /* ADD indexes */
#define AT_PASS_ADD_CONSTR 7 /* ADD constraints, defaults */
-#define AT_PASS_MISC 8 /* other stuff */
-#define AT_NUM_PASSES 9
+#define AT_PASS_ADD_STATS 8 /* ADD statistics */
+#define AT_PASS_MISC 9 /* other stuff */
+#define AT_NUM_PASSES 10
typedef struct AlteredTableInfo
{
@@ -412,7 +414,8 @@ static void ATExecReplicaIdentity(Relation rel, ReplicaIdentityStmt *stmt, LOCKM
static void ATExecGenericOptions(Relation rel, List *options);
static void ATExecEnableRowSecurity(Relation rel);
static void ATExecDisableRowSecurity(Relation rel);
-
+static void ATExecAddStatistics(AlteredTableInfo *tab, Relation rel,
+ StatisticsDef *def, LOCKMODE lockmode);
static void copy_relation_data(SMgrRelation rel, SMgrRelation dst,
ForkNumber forkNum, char relpersistence);
static const char *storage_name(char c);
@@ -2963,6 +2966,7 @@ AlterTableGetLockLevel(List *cmds)
* updates.
*/
case AT_SetStatistics: /* Uses MVCC in getTableAttrs() */
+ case AT_AddStatistics: /* XXX not sure if the right level */
case AT_ClusterOn: /* Uses MVCC in getIndexes() */
case AT_DropCluster: /* Uses MVCC in getIndexes() */
case AT_SetOptions: /* Uses MVCC in getTableAttrs() */
@@ -3110,6 +3114,7 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd,
pass = AT_PASS_ADD_CONSTR;
break;
case AT_SetStatistics: /* ALTER COLUMN SET STATISTICS */
+ case AT_AddStatistics: /* XXX maybe not the right place */
ATSimpleRecursion(wqueue, rel, cmd, recurse, lockmode);
/* Performs own permission checks */
ATPrepSetStatistics(rel, cmd->name, cmd->def, lockmode);
@@ -3405,6 +3410,9 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,
case AT_SetStatistics: /* ALTER COLUMN SET STATISTICS */
ATExecSetStatistics(rel, cmd->name, cmd->def, lockmode);
break;
+ case AT_AddStatistics: /* ADD STATISTICS */
+ ATExecAddStatistics(tab, rel, (StatisticsDef *) cmd->def, lockmode);
+ break;
case AT_SetOptions: /* ALTER COLUMN SET ( options ) */
ATExecSetOptions(rel, cmd->name, cmd->def, false, lockmode);
break;
@@ -11614,3 +11622,197 @@ RangeVarCallbackForAlterRelation(const RangeVar *rv, Oid relid, Oid oldrelid,
ReleaseSysCache(tuple);
}
+
+/* used for sorting the attnums in ATExecAddStatistics */
+static int compare_int16(const void *a, const void *b)
+{
+ return memcmp(a, b, sizeof(int16));
+}
+
+/*
+ * Implements the ALTER TABLE ... ADD STATISTICS (options) ON (columns).
+ *
+ * The code is an unholy mix of pieces that really belong to other parts
+ * of the source tree.
+ *
+ * FIXME Check that the types are pass-by-value and support sort,
+ * although maybe we can live without the sort (and only build
+ * MCV list / association rules).
+ *
+ * FIXME This should probably check for duplicate stats (i.e. same
+ * keys, same options). Although maybe it's useful to have
+ * multiple stats on the same columns with different options
+ * (say, a detailed MCV-only stats for some queries, histogram
+ * for others, etc.)
+ */
+static void ATExecAddStatistics(AlteredTableInfo *tab, Relation rel,
+ StatisticsDef *def, LOCKMODE lockmode)
+{
+ int i, j;
+ ListCell *l;
+ int16 attnums[INDEX_MAX_KEYS];
+ Oid atttypids[INDEX_MAX_KEYS];
+ int numcols = 0;
+
+ Oid mvstatoid;
+ HeapTuple htup;
+ Datum values[Natts_pg_mv_statistic];
+ bool nulls[Natts_pg_mv_statistic];
+ int2vector *stakeys;
+ Relation mvstatrel;
+
+ /* by default build everything */
+ bool build_histogram = true,
+ build_mcv = true,
+ build_associations = true;
+
+ /* build regular MCV (not hashed by default) */
+ bool mcv_hashed = false;
+
+ int32 max_buckets = -1,
+ max_mcv_items = -1;
+
+ Assert(IsA(def, StatisticsDef));
+
+ /* transform the column names to attnum values */
+
+ foreach(l, def->keys)
+ {
+ char *attname = strVal(lfirst(l));
+ HeapTuple atttuple;
+
+ atttuple = SearchSysCacheAttName(RelationGetRelid(rel), attname);
+
+ if (!HeapTupleIsValid(atttuple))
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_COLUMN),
+ errmsg("column \"%s\" referenced in statistics does not exist",
+ attname)));
+
+ /* more than MVHIST_MAX_DIMENSIONS columns not allowed */
+ if (numcols >= MVSTATS_MAX_DIMENSIONS)
+ ereport(ERROR,
+ (errcode(ERRCODE_TOO_MANY_COLUMNS),
+ errmsg("cannot have more than %d keys in a statistics",
+ MVSTATS_MAX_DIMENSIONS)));
+
+ attnums[numcols] = ((Form_pg_attribute) GETSTRUCT(atttuple))->attnum;
+ atttypids[numcols] = ((Form_pg_attribute) GETSTRUCT(atttuple))->atttypid;
+ ReleaseSysCache(atttuple);
+ numcols++;
+ }
+
+ /*
+ * Check the lower bound (at least 2 columns), the upper bound was
+ * already checked in the loop.
+ */
+ if (numcols < 2)
+ ereport(ERROR,
+ (errcode(ERRCODE_TOO_MANY_COLUMNS),
+ errmsg("multivariate stats require 2 or more columns")));
+
+ /* look for duplicities */
+ for (i = 0; i < numcols; i++)
+ for (j = 0; j < numcols; j++)
+ if ((i != j) && (attnums[i] == attnums[j]))
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_COLUMN),
+ errmsg("duplicate column name in statistics definition")));
+
+ /* parse the statistics options */
+ foreach (l, def->options)
+ {
+ DefElem *opt = (DefElem*)lfirst(l);
+
+ if (strcmp(opt->defname, "histogram") == 0)
+ build_histogram = defGetBoolean(opt);
+ else if (strcmp(opt->defname, "mcv") == 0)
+ build_mcv = defGetBoolean(opt);
+ else if (strcmp(opt->defname, "mcv_hashed") == 0)
+ mcv_hashed = defGetBoolean(opt);
+ else if (strcmp(opt->defname, "associations") == 0)
+ build_associations = defGetBoolean(opt);
+ else if (strcmp(opt->defname, "max_buckets") == 0)
+ {
+ max_buckets = defGetInt32(opt);
+
+ /* TODO check that this is not used with 'histogram off' */
+
+ /* sanity check */
+ if (max_buckets < 1024)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("minimum number of buckets is 1024")));
+
+ else if (max_buckets > 32768) /* FIXME use the proper constant */
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("minimum number of buckets is 1024")));
+
+ }
+ else if (strcmp(opt->defname, "max_mcv_items") == 0)
+ {
+ max_mcv_items = defGetInt32(opt);
+
+ /* TODO check that this is not used with 'mcv off' */
+
+ /* sanity check */
+ if (max_mcv_items < 0)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("max number of MCV items must be non-negative")));
+
+ else if (max_mcv_items > 8192) /* FIXME use the proper constant */
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("max number of MCV items is 8192")));
+
+ }
+ else
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("unrecognized STATISTICS option \"%s\"",
+ opt->defname)));
+ }
+
+ /* sort the attnums and build int2vector */
+ qsort(attnums, numcols, sizeof(int16), compare_int16);
+ stakeys = buildint2vector(attnums, numcols);
+
+ /*
+ * Okay, let's create the pg_mv_statistic entry.
+ */
+ memset(values, 0, sizeof(values));
+ memset(nulls, false, sizeof(nulls));
+
+ /* no stats collected yet, so just the keys */
+ values[Anum_pg_mv_statistic_starelid-1] = ObjectIdGetDatum(RelationGetRelid(rel));
+
+ values[Anum_pg_mv_statistic_stakeys -1] = PointerGetDatum(stakeys);
+ values[Anum_pg_mv_statistic_hist_enabled -1] = BoolGetDatum(build_histogram);
+ values[Anum_pg_mv_statistic_mcv_enabled -1] = BoolGetDatum(build_mcv);
+ values[Anum_pg_mv_statistic_mcv_hashed -1] = BoolGetDatum(mcv_hashed);
+ values[Anum_pg_mv_statistic_assoc_enabled -1] = BoolGetDatum(build_associations);
+
+ values[Anum_pg_mv_statistic_hist_max_buckets -1] = Int32GetDatum(max_buckets);
+ values[Anum_pg_mv_statistic_mcv_max_items -1] = Int32GetDatum(max_mcv_items);
+
+ nulls[Anum_pg_mv_statistic_staassoc -1] = true;
+ nulls[Anum_pg_mv_statistic_stamcv -1] = true;
+ nulls[Anum_pg_mv_statistic_stahist -1] = true;
+
+ /* insert the tuple into pg_mv_statistic */
+ mvstatrel = heap_open(MvStatisticRelationId, RowExclusiveLock);
+
+ htup = heap_form_tuple(mvstatrel->rd_att, values, nulls);
+
+ mvstatoid = simple_heap_insert(mvstatrel, htup);
+
+ CatalogUpdateIndexes(mvstatrel, htup);
+
+ heap_freetuple(htup);
+
+ heap_close(mvstatrel, RowExclusiveLock);
+
+ return;
+}
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 225756c..18464b9 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -3879,6 +3879,17 @@ _copyAlterPolicyStmt(const AlterPolicyStmt *from)
return newnode;
}
+static StatisticsDef *
+_copyStatisticsDef(const StatisticsDef *from)
+{
+ StatisticsDef *newnode = makeNode(StatisticsDef);
+
+ COPY_NODE_FIELD(keys);
+ COPY_NODE_FIELD(options);
+
+ return newnode;
+}
+
/* ****************************************************************
* pg_list.h copy functions
* ****************************************************************
@@ -4690,6 +4701,9 @@ copyObject(const void *from)
case T_CommonTableExpr:
retval = _copyCommonTableExpr(from);
break;
+ case T_StatisticsDef:
+ retval = _copyStatisticsDef(from);
+ break;
case T_PrivGrantee:
retval = _copyPrivGrantee(from);
break;
@@ -4702,7 +4716,6 @@ copyObject(const void *from)
case T_XmlSerialize:
retval = _copyXmlSerialize(from);
break;
-
default:
elog(ERROR, "unrecognized node type: %d", (int) nodeTag(from));
retval = 0; /* keep compiler quiet */
diff --git a/src/backend/optimizer/path/clausesel.c b/src/backend/optimizer/path/clausesel.c
index 9b657fb..9c32735 100644
--- a/src/backend/optimizer/path/clausesel.c
+++ b/src/backend/optimizer/path/clausesel.c
@@ -24,6 +24,9 @@
#include "utils/lsyscache.h"
#include "utils/selfuncs.h"
+#include "utils/mvstats.h"
+#include "catalog/pg_collation.h"
+#include "utils/typcache.h"
/*
* Data structure for accumulating info about possible range-query
@@ -43,6 +46,23 @@ static void addRangeClause(RangeQueryClause **rqlist, Node *clause,
bool varonleft, bool isLTsel, Selectivity s2);
+static bool is_mv_compatible(Node *clause, Oid varRelid, Index *varno,
+ Bitmapset **attnums);
+static Bitmapset *collect_mv_attnums(PlannerInfo *root, List *clauses,
+ Oid varRelid, Oid *relid);
+static int choose_mv_histogram(int nmvstats, MVStats mvstats,
+ Bitmapset *attnums);
+static List *clauselist_mv_split(List *clauses, Oid varRelid,
+ List **mvclauses, MVStats mvstats);
+
+static Selectivity clauselist_mv_selectivity(PlannerInfo *root,
+ List *clauses, MVStats mvstats);
+static Selectivity clauselist_mv_selectivity_mcvlist(PlannerInfo *root,
+ List *clauses, MVStats mvstats,
+ bool *fullmatch, Selectivity *lowsel);
+static Selectivity clauselist_mv_selectivity_histogram(PlannerInfo *root,
+ List *clauses, MVStats mvstats);
+
/****************************************************************************
* ROUTINES TO COMPUTE SELECTIVITIES
****************************************************************************/
@@ -100,14 +120,74 @@ clauselist_selectivity(PlannerInfo *root,
RangeQueryClause *rqlist = NULL;
ListCell *l;
+ /* processing mv stats */
+ Oid relid = InvalidOid;
+ int nmvstats = 0;
+ MVStats mvstats = NULL;
+
+ /* attributes in mv-compatible clauses */
+ Bitmapset *mvattnums = NULL;
+
/*
- * If there's exactly one clause, then no use in trying to match up pairs,
- * so just go directly to clause_selectivity().
+ * If there's exactly one clause, then no use in trying to match up
+ * pairs, so just go directly to clause_selectivity().
*/
if (list_length(clauses) == 1)
return clause_selectivity(root, (Node *) linitial(clauses),
varRelid, jointype, sjinfo);
+ /* collect attributes from mv-compatible clauses */
+ mvattnums = collect_mv_attnums(root, clauses, varRelid, &relid);
+
+ /*
+ * If there are mv-compatible clauses, referencing at least two
+ * columns (otherwise it makes no sense to use mv stats), fetch the
+ * MV histograms for the relation (only the column keys, not the
+ * histograms yet - we'll decide which histogram to use first).
+ */
+ if (bms_num_members(mvattnums) >= 2)
+ {
+ /* clauses compatible with multi-variate stats */
+ List *mvclauses = NIL;
+
+ /* fetch info from the catalog (not the serialized stats yet) */
+ mvstats = list_mv_stats(relid, &nmvstats, true);
+
+ /*
+ * If there are candidate statistics, choose the histogram first.
+ * At the moment we only use a single statistics, covering the
+ * most columns (using info from the previous step). If there
+ * are multiple such histograms, we'll use the smallest one
+ * (with the lowest number of dimensions).
+ *
+ * This may not be optimal choice, if the 'smaller' stats has
+ * much less buckets than the rejected one (making it less
+ * accurate).
+ *
+ * We may end up without multivariate statistics, if none of the
+ * stats matches at least two columns from the clauses (in that
+ * case we may just use the single dimensional stats).
+ */
+ if (nmvstats > 0)
+ {
+ int idx = choose_mv_histogram(nmvstats, mvstats, mvattnums);
+
+ if (idx >= 0) /* we have a matching stats */
+ {
+ MVStats mvstat = &mvstats[idx];
+
+ /* split the clauselist into regular and mv-clauses */
+ clauses = clauselist_mv_split(clauses, varRelid, &mvclauses, mvstat);
+
+ /* we've chosen the histogram to match the clauses */
+ Assert(mvclauses != NIL);
+
+ /* compute the multivariate stats */
+ s1 *= clauselist_mv_selectivity(root, mvclauses, mvstat);
+ }
+ }
+ }
+
/*
* Initial scan over clauses. Anything that doesn't look like a potential
* rangequery clause gets multiplied into s1 and forgotten. Anything that
@@ -782,3 +862,1010 @@ clause_selectivity(PlannerInfo *root,
return s1;
}
+
+
+
+/*
+ * Estimate selectivity for the list of MV-compatible clauses, using that
+ * particular histogram.
+ *
+ * When we hit a single bucket, we don't know what portion of it actually
+ * matches the clauses (e.g. equality), and we use 1/2 the bucket by
+ * default. However, the MV histograms are usually less detailed than
+ * the per-column ones, meaning the sum of buckets is often quite high
+ * (thanks to combining a lot of "partially hit" buckets).
+ *
+ * There are several ways to improve this, usually with cases when it
+ * won't really help. Also, the more complex the process, the worse
+ * the failures (i.e. misestimates).
+ *
+ * (1) Use the MV histogram only as a way to combine multiple
+ * per-column histograms, essentially rewriting
+ *
+ * P(A & B) = P(A) * P(B|A)
+ *
+ * where P(B|A) may be computed using a proper "slice" of the
+ * histogram, by first selecting only buckets where A is true, and
+ * then using the boundaries to 'restrict' the per-colunm histogram.
+ *
+ * With more clauses, it gets more complicated, of course
+ *
+ * P(A & B & C) = P(A & C) * P(B|A & C)
+ * = P(A) * P(C|A) * P(B|A & C)
+ *
+ * and so on.
+ *
+ * Of course, the question is how well and efficiently we can
+ * compute the conditional probabilities - whether this approach
+ * can improve the estimates (instead of amplifying the errors).
+ *
+ * Also, this does not eliminate the need for histogram on [A,B,C].
+ *
+ * (2) Use multiple smaller (and more accurate) histograms, and combine
+ * them using a process similar to the above. E.g. by assuming that
+ * B and C are independent, we can rewrite
+ *
+ * P(B|A & C) = P(B|A)
+ *
+ * so we can rewrite the whole formula to
+ *
+ * P(A & B & C) = P(A) * P(C|A) * P(B|A)
+ *
+ * and we're OK with two 2D histograms [A,C] and [A,B].
+ *
+ * It'd be nice to perform some sort of statistical test (Fisher
+ * or another chi-squared test) to identify independent components
+ * and automatically separate them into smaller histograms.
+ *
+ * (3) Using the estimated number of distinct values in a bucket to
+ * decide the selectivity of equality in the bucket (instead of
+ * blindly using 1/2 of the bucket, we may use 1/ndistinct).
+ * Of course, if the ndistinct estimate is way off, or when the
+ * distribution is not uniform (one distict items get much more
+ * items), this will fail. Also, we currently don't have ndistinct
+ * estimate available at this moment (but it shouldn't be that
+ * difficult to compute as ndistinct and ntuples should be available).
+ *
+ * TODO Clamp the selectivity by min of the per-clause selectivities
+ * (i.e. the selectivity of the most restrictive clause), because
+ * that's the maximum we can ever get from ANDed list of clauses.
+ * This may probably prevent issues with hitting too many buckets
+ * and low precision histograms.
+ *
+ * TODO We may support some additional conditions, most importantly
+ * those matching multiple columns (e.g. "a = b" or "a < b").
+ * Ultimately we could track multi-table histograms for join
+ * cardinality estimation.
+ *
+ * TODO Currently this is only estimating all clauses, or clauses
+ * matching varRelid (when it's not 0). I'm not sure what's the
+ * purpose of varRelid, but my assumption is this is used for
+ * join conditions and such. In that case we can use those clauses
+ * to restrict the other (i.e. filter the histogram buckets first,
+ * before estimating the other clauses). This is essentially equal
+ * to computing P(A|B) where "B" are the clauses not matching the
+ * varRelid.
+ *
+ * TODO Further thoughts on processing equality clauses - maybe it'd be
+ * better to look for stats (with MCV) covered by the equality
+ * clauses, because then we have a chance to find an exact match
+ * in the MCV list, which is pretty much the best we can do. We may
+ * also look at the least frequent MCV item, and use it as a upper
+ * boundary for the selectivity (had there been a more frequent
+ * item, it'd be in the MCV list).
+ *
+ * These conditions may then be used as a condition for the other
+ * selectivities, i.e. we may estimate P(A,B) first, and then
+ * compute P(C|A,B) from another histogram. This may be useful when
+ * we can estimate P(A,B) accurately (e.g. because it's a complete
+ * equality match evaluated on MCV list), and then compute the
+ * conditional probability P(C|A,B), giving us the requested stats
+ *
+ * P(A,B,C) = P(A,B) * P(C|A,B)
+ *
+ * TODO There are several options for 'sanity clamping' the estimates.
+ *
+ * First, if we have selectivities for each condition, then
+ *
+ * P(A,B) <= MIN(P(A), P(B))
+ *
+ * Because additional conditions (connected by AND) can only lower
+ * the probability.
+ *
+ * So we can do some basic sanity checks using the single-variate
+ * stats (the ones we have right now).
+ *
+ * Second, when we have multivariate stats with a MCV list, then
+ *
+ * (a) if we have a full equality condition (one equality condition
+ * on each column) and we found a match in the MCV list, this is
+ * the selectivity (and it's supposed to be exact)
+ *
+ * (b) if we have a full equality condition and we haven't found a
+ * match in the MCV list, then the selectivity is below the
+ * lowest selectivity in the MCV list
+ *
+ * (c) if we have a equality condition (not full), we can still
+ * search the MCV for matches and use the sum of probabilities
+ * as a lower boundary for the histogram (if there are no
+ * matches in the MCV list, then we have no boundary)
+ *
+ * Third, if there are multiple multivariate stats for a set of
+ * clauses, we may compute all of them and then somehow aggregate
+ * them - e.g. by choosing the minimum, median or average. The
+ * multi-variate stats are susceptible to overestimation (because
+ * we take 50% of the bucket for partial matches). Some stats may
+ * give better estimates than others, but it's very difficult to
+ * say determine that in advance which one is the best (it depends
+ * on the number of buckets, number of additional columns not
+ * referenced in the clauses etc.) so we may compute all and then
+ * choose a sane aggregation (minimum seems like a good approach).
+ * Of course, this may result in longer / more expensive estimation
+ * (CPU-wise), but it may be worth it.
+ *
+ * There are ways to address this, though. First, it's possible to
+ * add a GUC choosing whether to do a 'simple' (using a single
+ * stats expected to give the best estimate) and 'complex' (combining
+ * the multiple estimates).
+ *
+ * multivariate_estimates = (simple|full)
+ *
+ * Also, this might be enabled at a table level, by something like
+ *
+ * ALTER TABLE ... SET STATISTICS (simple|full)
+ *
+ * Which would make it possible to use this only for the tables
+ * where the simple approach does not work.
+ *
+ * Also, there are ways to optimize this algorithmically. E.g. we
+ * may try to get an estimate from a matching MCV list first, and
+ * if we happen to get a "full equality match" we may stop computing
+ * the estimates from other stats (for this condition) because
+ * that's probably the best estimate we can really get.
+ *
+ * TODO When applying the clauses to the histogram/MCV list, we can do
+ * that from the most selective clauses first, because that'll
+ * eliminate the buckets/items sooner (so we'll be able to skip
+ * them without inspection, which is more expensive).
+ */
+static Selectivity
+clauselist_mv_selectivity(PlannerInfo *root, List *clauses, MVStats mvstats)
+{
+ bool fullmatch = false;
+ Selectivity s1 = 0.0, s2 = 0.0;
+
+ /*
+ * Lowest frequency in the MCV list (may be used as an upper bound
+ * for full equality conditions that did not match any MCV item).
+ */
+ Selectivity mcv_low = 0.0;
+
+ /* TODO Evaluate simple 1D selectivities, use the smallest one as
+ * an upper bound, product as lower bound, and sort the
+ * clauses in ascending order by selectivity (to optimize the
+ * MCV/histogram evaluation).
+ */
+
+ /* Evaluate the MCV first. */
+ s1 = clauselist_mv_selectivity_mcvlist(root, clauses, mvstats,
+ &fullmatch, &mcv_low);
+
+ /*
+ * If we got a full equality match on the MCV list, we're done (and
+ * the estimate is pretty good).
+ */
+ if (fullmatch && (s1 > 0.0))
+ return s1;
+
+ /* FIXME if (fullmatch) without matching MCV item, use the mcv_low
+ * selectivity as upper bound */
+
+ s2 = clauselist_mv_selectivity_histogram(root, clauses, mvstats);
+
+ /* TODO clamp to <= 1.0 (or more strictly, when possible) */
+ return s1 + s2;
+}
+
+/*
+ * Collect attributes from mv-compatible clauses.
+ *
+ */
+static Bitmapset *
+collect_mv_attnums(PlannerInfo *root, List *clauses, Oid varRelid, Oid *relid)
+{
+ Index varno = 0;
+ Bitmapset *attnums = NULL;
+ ListCell *l;
+
+ /*
+ * Walk through the clauses and identify the ones we can estimate
+ * using multivariate stats, and remember the relid/columns. We'll
+ * then cross-check if we have suitable stats, and only if needed
+ * we'll split the clauses into multivariate and regular lists.
+ *
+ * For now we're only interested in RestrictInfo nodes with nested
+ * OpExpr, using either a range or equality.
+ */
+ foreach (l, clauses)
+ {
+ Node *clause = (Node *) lfirst(l);
+
+ /* ignore the result for now - we only need the info */
+ is_mv_compatible(clause, varRelid, &varno, &attnums);
+ }
+
+ /*
+ * If there are at least two attributes referenced by the clause(s),
+ * fetch the relation info (and pass back the Oid of the relation).
+ */
+ if (bms_num_members(attnums) > 1)
+ {
+ RelOptInfo *rel = find_base_rel(root, varno);
+ *relid = root->simple_rte_array[bms_singleton_member(rel->relids)]->relid;
+ }
+ else
+ {
+ if (attnums != NULL)
+ pfree(attnums);
+ attnums = NULL;
+ *relid = InvalidOid;
+ }
+
+ return attnums;
+}
+
+/*
+ * We're looking for a histogram matching at least 2 attributes, and we
+ * want the smallest histogram available wrt. to number of buckets (to
+ * get efficient estimation and likely better precision. The precision
+ * depends on the total number of buckets too, but the lower the number
+ * of dimensions the smaller (and more precise) the buckets can get.
+ */
+static int
+choose_mv_histogram(int nmvstats, MVStats mvstats, Bitmapset *attnums)
+{
+ int i, j;
+
+ int choice = -1;
+ int current_matches = 1; /* goal #1: maximize */
+ int current_dims = (MVSTATS_MAX_DIMENSIONS+1); /* goal #2: minimize */
+
+ for (i = 0; i < nmvstats; i++)
+ {
+ int matches = 0; /* columns matching this histogram */
+
+ int2vector * attrs = mvstats[i].stakeys;
+ int numattrs = mvstats[i].stakeys->dim1;
+
+ /* count columns covered by the histogram */
+ for (j = 0; j < numattrs; j++)
+ if (bms_is_member(attrs->values[j], attnums))
+ matches++;
+
+ /*
+ * Use this histogram when it improves the number of matches or
+ * when it keeps the number of matches and is smaller.
+ */
+ if ((matches > current_matches) ||
+ ((matches == current_matches) && (current_dims > numattrs)))
+ {
+ choice = i;
+ current_matches = matches;
+ current_dims = numattrs;
+ }
+ }
+
+ return choice;
+}
+
+/*
+ * This splits the clauses list into two parts - one containing clauses
+ * that will be evaluated using the chosen histogram, and the remaining
+ * clauses (either non-mvcompatible, or not related to the histogram).
+ */
+static List *
+clauselist_mv_split(List *clauses, Oid varRelid, List **mvclauses, MVStats mvstats)
+{
+ int i;
+ ListCell *l;
+ List *non_mvclauses = NIL;
+
+ /* FIXME is there a better way to get info on int2vector? */
+ int2vector * attrs = mvstats->stakeys;
+ int numattrs = mvstats->stakeys->dim1;
+
+ /* erase the list of mv-compatible clauses */
+ *mvclauses = NIL;
+
+ foreach (l, clauses)
+ {
+ RestrictInfo *rinfo;
+ Node *clause = (Node *) lfirst(l);
+
+ /*
+ * Only restrictinfo may be mv-compatible, so everything else
+ * goes to the non-mv list directly
+ *
+ * TODO create a macro/function to decide mv-compatible clauses
+ * (along the is_opclause for example)
+ */
+ if (! IsA(clause, RestrictInfo))
+ {
+ non_mvclauses = lappend(non_mvclauses, clause);
+ continue;
+ }
+
+ rinfo = (RestrictInfo *) clause;
+ clause = (Node*)rinfo->clause;
+
+ /* Pseudoconstants go directly to the non-mv list too. */
+ if (rinfo->pseudoconstant)
+ {
+ non_mvclauses = lappend(non_mvclauses, rinfo);
+ continue;
+ }
+
+ if (is_opclause(clause) && list_length(((OpExpr *) clause)->args) == 2)
+ {
+ OpExpr *expr = (OpExpr *) clause;
+ bool varonleft = true;
+ bool ok;
+
+ ok = (bms_membership(rinfo->clause_relids) == BMS_SINGLETON) &&
+ (is_pseudo_constant_clause_relids(lsecond(expr->args),
+ rinfo->right_relids) ||
+ (varonleft = false,
+ is_pseudo_constant_clause_relids(linitial(expr->args),
+ rinfo->left_relids)));
+
+ if (ok)
+ {
+
+ Var * var = (varonleft) ? linitial(expr->args) : lsecond(expr->args);
+
+ /*
+ * Only consider this variable if (varRelid == 0) or when the varno
+ * matches varRelid (see explanation at clause_selectivity).
+ */
+ if (! ((varRelid == 0) || (varRelid == var->varno)))
+ {
+ non_mvclauses = lappend(non_mvclauses, rinfo);
+ continue;
+ }
+
+ /*
+ * If it's not a "<" or ">" or "=" operator, just ignore the
+ * clause. Otherwise note the relid and attnum for the variable.
+ */
+ switch (get_oprrest(expr->opno))
+ {
+ case F_SCALARLTSEL:
+ case F_SCALARGTSEL:
+ case F_EQSEL:
+ if (! IS_SPECIAL_VARNO(var->varno)) /* FIXME necessary here? */
+ {
+ bool match = false;
+ for (i = 0; i < numattrs; i++)
+ if (attrs->values[i] == var->varattno)
+ match = true;
+
+ if (match)
+ *mvclauses = lappend(*mvclauses, clause);
+ else
+ non_mvclauses = lappend(non_mvclauses, rinfo);
+ }
+ }
+ }
+ }
+ }
+
+ /*
+ * Perform regular estimation using the clauses incompatible
+ * with the chosen histogram (or MV stats in general).
+ */
+ return non_mvclauses;
+
+}
+
+/*
+ * Determines whether the clause is compatible with multivariate stats,
+ * and if it is, returns some additional information - varno (index
+ * into simple_rte_array) and a bitmap of attributes. This is then
+ * used to fetch related multivariate statistics.
+ *
+ * At this moment we only support basic conditions of the form
+ *
+ * variable OP constant
+ *
+ * where OP is one of [=,<,<=,>=,>] (which is however determined by
+ * looking at the associated function for estimating selectivity, just
+ * like with the single-dimensional case).
+ */
+static bool
+is_mv_compatible(Node *clause, Oid varRelid, Index *varno, Bitmapset **attnums)
+{
+
+ if (IsA(clause, RestrictInfo))
+ {
+ RestrictInfo *rinfo = (RestrictInfo *) clause;
+
+ /* Pseudoconstants are not really interesting here. */
+ if (rinfo->pseudoconstant)
+ return false;
+
+ /* get the actual clause from the RestrictInfo ... */
+ clause = (Node*)rinfo->clause;
+
+ /* is it 'variable op constant' ? */
+ if (is_opclause(clause) && list_length(((OpExpr *) clause)->args) == 2)
+ {
+ OpExpr *expr = (OpExpr *) clause;
+ bool varonleft = true;
+ bool ok;
+
+ ok = (bms_membership(rinfo->clause_relids) == BMS_SINGLETON) &&
+ (is_pseudo_constant_clause_relids(lsecond(expr->args),
+ rinfo->right_relids) ||
+ (varonleft = false,
+ is_pseudo_constant_clause_relids(linitial(expr->args),
+ rinfo->left_relids)));
+
+ if (ok)
+ {
+
+ Var * var = (varonleft) ? linitial(expr->args) : lsecond(expr->args);
+
+ /*
+ * Only consider this variable if (varRelid == 0) or when the varno
+ * matches varRelid (see explanation at clause_selectivity).
+ */
+ if (! ((varRelid == 0) || (varRelid == var->varno)))
+ return false;
+
+ /* Also skip special varno values, and system attributes ... */
+ if ((IS_SPECIAL_VARNO(var->varno)) || (! AttrNumberIsForUserDefinedAttr(var->varattno)))
+ return false;
+
+ /*
+ * If it's not a "<" or ">" or "=" operator, just ignore the
+ * clause. Otherwise note the relid and attnum for the variable.
+ * This uses the function for estimating selectivity, ont the
+ * operator directly (a bit awkward, but well ...).
+ */
+ switch (get_oprrest(expr->opno))
+ {
+ case F_SCALARLTSEL:
+ case F_SCALARGTSEL:
+ case F_EQSEL:
+ *varno = var->varno;
+ *attnums = bms_add_member(*attnums, var->varattno);
+ return true;
+ }
+ }
+ }
+ }
+
+ return false;
+
+}
+
+/*
+ * Estimate selectivity of clauses using a MCV list.
+ *
+ * If there's no MCV list for the stats, the function returns 0.0.
+ *
+ * While computing the estimate, the function checks whether all the
+ * columns were matched with an equality condition. If that's the case,
+ * it's assumed we can skip computing the estimate from histogram,
+ * because all the rows matching the condition are represented by the
+ * MCV item.
+ *
+ * The function also returns the frequency of the least frequent item
+ * on the MCV list, which may be useful for clamping estimate from the
+ * histogram.
+ */
+static Selectivity
+clauselist_mv_selectivity_mcvlist(PlannerInfo *root, List *clauses,
+ MVStats mvstats, bool *fullmatch,
+ Selectivity *lowsel)
+{
+ int i;
+ Selectivity s = 0.0;
+ ListCell * l;
+ char * mcvitems = NULL;
+ MCVList mcvlist = NULL;
+
+ Bitmapset *matches = NULL; /* attributes with equality matches */
+
+ /* there's no MCV list yet */
+ if (! mvstats->mcv_built)
+ return 0.0;
+
+ mcvlist = deserialize_mv_mcvlist(fetch_mv_mcvlist(mvstats->mvoid));
+
+ Assert(mcvlist != NULL);
+ Assert (clauses != NIL);
+ Assert (list_length(clauses) >= 2);
+
+ mcvitems = palloc0(sizeof(char) * mcvlist->nitems);
+ memset(mcvitems, MVSTATS_MATCH_FULL, sizeof(char)*mcvlist->nitems);
+
+ /* no match here */
+ *lowsel = 1.0;
+
+ /* loop through the list of MV-compatible clauses and do the estimation */
+ foreach (l, clauses)
+ {
+ Node * clause = (Node*)lfirst(l);
+ OpExpr * expr = (OpExpr*)clause;
+ bool varonleft = true;
+ bool ok;
+
+ /* operator */
+ FmgrInfo opproc;
+
+ fmgr_info(get_opcode(expr->opno), &opproc);
+
+ ok = (NumRelids(clause) == 1) &&
+ (is_pseudo_constant_clause(lsecond(expr->args)) ||
+ (varonleft = false,
+ is_pseudo_constant_clause(linitial(expr->args))));
+
+ if (ok)
+ {
+
+ FmgrInfo ltproc, gtproc;
+ Var * var = (varonleft) ? linitial(expr->args) : lsecond(expr->args);
+ Const * cst = (varonleft) ? lsecond(expr->args) : linitial(expr->args);
+ bool isgt = (! varonleft);
+
+ /*
+ * TODO Fetch only when really needed (probably for equality only)
+ * TODO Technically either lt/gt is sufficient.
+ *
+ * FIXME The code in analyze.c creates histograms only for types
+ * with enough ordering (by calling get_sort_group_operators).
+ * Is this the same assumption, i.e. are we certain that we
+ * get the ltproc/gtproc every time we ask? Or are there types
+ * where get_sort_group_operators returns ltopr and here we
+ * get nothing?
+ */
+ TypeCacheEntry *typecache = lookup_type_cache(var->vartype, TYPECACHE_EQ_OPR | TYPECACHE_LT_OPR | TYPECACHE_GT_OPR);
+
+ /* FIXME proper matching attribute to dimension */
+ int idx = mv_get_index(var->varattno, mvstats->stakeys);
+
+ fmgr_info(get_opcode(typecache->lt_opr), <proc);
+ fmgr_info(get_opcode(typecache->gt_opr), >proc);
+
+ /* process the MCV list first */
+ for (i = 0; i < mcvlist->nitems; i++)
+ {
+ bool tmp;
+ MCVItem item = mcvlist->items[i];
+
+ /* find the lowest selectivity in the MCV */
+ if (item->frequency < *lowsel)
+ *lowsel = item->frequency;
+
+ /* skip MCV items already ruled out */
+ if (mcvitems[i] == MVSTATS_MATCH_NONE)
+ continue;
+
+ /* TODO consider bsearch here (list is sorted by values)
+ * TODO handle other operators too (LT, GT)
+ * TODO identify "full match" when the clauses fully
+ * match the whole MCV list (so that checking the
+ * histogram is not needed)
+ */
+ if (get_oprrest(expr->opno) == F_EQSEL)
+ {
+ /*
+ * We don't care about isgt in equality, because it does not matter
+ * whether it's (var = const) or (const = var).
+ */
+ if (memcmp(&cst->constvalue, &item->values[idx], sizeof(Datum)) != 0)
+ mcvitems[i] = MVSTATS_MATCH_NONE;
+ else
+ matches = bms_add_member(matches, idx);
+ }
+ else if (get_oprrest(expr->opno) == F_SCALARLTSEL) /* column < constant */
+ {
+
+ if (! isgt) /* (var < const) */
+ {
+ /*
+ * First check whether the constant is below the lower boundary (in that
+ * case we can skip the bucket, because there's no overlap).
+ */
+ tmp = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ cst->constvalue,
+ item->values[idx]));
+
+ if (tmp)
+ {
+ mcvitems[i] = MVSTATS_MATCH_NONE; /* no match */
+ continue;
+ }
+
+ } /* (get_oprrest(expr->opno) == F_SCALARLTSEL) */
+ else /* (const < var) */
+ {
+ /*
+ * First check whether the constant is above the upper boundary (in that
+ * case we can skip the bucket, because there's no overlap).
+ */
+ tmp = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ item->values[idx],
+ cst->constvalue));
+ if (tmp)
+ {
+ mcvitems[i] = MVSTATS_MATCH_NONE; /* no match */
+ continue;
+ }
+ }
+ }
+ else if (get_oprrest(expr->opno) == F_SCALARGTSEL) /* column > constant */
+ {
+
+ if (! isgt) /* (var > const) */
+ {
+ /*
+ * First check whether the constant is above the upper boundary (in that
+ * case we can skip the bucket, because there's no overlap).
+ */
+ tmp = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ cst->constvalue,
+ item->values[idx]));
+ if (tmp)
+ {
+ mcvitems[i] = MVSTATS_MATCH_NONE; /* no match */
+ continue;
+ }
+
+ }
+ else /* (const > var) */
+ {
+ /*
+ * First check whether the constant is below the lower boundary (in
+ * that case we can skip the bucket, because there's no overlap).
+ */
+ tmp = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ item->values[idx],
+ cst->constvalue));
+ if (tmp)
+ {
+ mcvitems[i] = MVSTATS_MATCH_NONE; /* no match */
+ continue;
+ }
+ }
+
+ } /* (get_oprrest(expr->opno) == F_SCALARGTSEL) */
+
+ }
+ }
+ }
+
+ for (i = 0; i < mcvlist->nitems; i++)
+ {
+ if (mcvitems[i] != MVSTATS_MATCH_NONE)
+ s += mcvlist->items[i]->frequency;
+ }
+
+ *fullmatch = (bms_num_members(matches) == mcvlist->ndimensions);
+
+ pfree(mcvitems);
+ pfree(mcvlist);
+
+ return s;
+}
+
+/*
+ * Estimate selectivity of clauses using a histogram.
+ *
+ * If there's no histogram list for the stats, the function returns 0.0.
+ */
+static Selectivity
+clauselist_mv_selectivity_histogram(PlannerInfo *root, List *clauses,
+ MVStats mvstats)
+{
+ int i;
+ Selectivity s = 0.0;
+ ListCell * l;
+ char *buckets = NULL;
+ MVHistogram mvhist = NULL;
+
+ /* there's no histogram */
+ if (! mvstats->hist_built)
+ return 0.0;
+
+ /* There may be no histogram in the stats (check hist_built flag) */
+ mvhist = deserialize_mv_histogram(fetch_mv_histogram(mvstats->mvoid));
+
+ Assert (mvhist != NULL);
+ Assert (clauses != NIL);
+ Assert (list_length(clauses) >= 2);
+
+ /*
+ * Bitmap of bucket matches (mismatch, partial, full). by default
+ * all buckets fully match (and we'll eliminate them).
+ */
+ buckets = palloc0(sizeof(char) * mvhist->nbuckets);
+ memset(buckets, MVSTATS_MATCH_FULL, sizeof(char)*mvhist->nbuckets);
+
+ /* loop through the clauses and do the estimation */
+ foreach (l, clauses)
+ {
+ Node * clause = (Node*)lfirst(l);
+ OpExpr * expr = (OpExpr*)clause;
+ bool varonleft = true;
+ bool ok;
+
+ FmgrInfo opproc; /* operator */
+ fmgr_info(get_opcode(expr->opno), &opproc);
+
+ ok = (NumRelids(clause) == 1) &&
+ (is_pseudo_constant_clause(lsecond(expr->args)) ||
+ (varonleft = false,
+ is_pseudo_constant_clause(linitial(expr->args))));
+
+ if (ok)
+ {
+ FmgrInfo ltproc;
+ Var * var = (varonleft) ? linitial(expr->args) : lsecond(expr->args);
+ Const * cst = (varonleft) ? lsecond(expr->args) : linitial(expr->args);
+ bool isgt = (! varonleft);
+
+ /*
+ * TODO Fetch only when really needed (probably for equality only)
+ *
+ * TODO Technically either lt/gt is sufficient.
+ *
+ * FIXME The code in analyze.c creates histograms only for types
+ * with enough ordering (by calling get_sort_group_operators).
+ * Is this the same assumption, i.e. are we certain that we
+ * get the ltproc/gtproc every time we ask? Or are there types
+ * where get_sort_group_operators returns ltopr and here we
+ * get nothing?
+ */
+ TypeCacheEntry *typecache
+ = lookup_type_cache(var->vartype, TYPECACHE_EQ_OPR | TYPECACHE_LT_OPR
+ | TYPECACHE_GT_OPR);
+
+ /* lookup dimension for the attribute */
+ int idx = mv_get_index(var->varattno, mvstats->stakeys);
+
+ fmgr_info(get_opcode(typecache->lt_opr), <proc);
+
+ /*
+ * Check this for all buckets that still have "true" in the bitmap
+ *
+ * We already know the clauses use suitable operators (because that's
+ * how we filtered them).
+ */
+ for (i = 0; i < mvhist->nbuckets; i++)
+ {
+ bool tmp;
+ MVBucket bucket = mvhist->buckets[i];
+
+ /*
+ * Skip buckets that were already eliminated - this is impotant
+ * considering how we update the info (we only lower the match)
+ */
+ if (buckets[i] == MVSTATS_MATCH_NONE)
+ continue;
+
+ /*
+ * If it's not a "<" or ">" or "=" operator, just ignore the
+ * clause. Otherwise note the relid and attnum for the variable.
+ *
+ * TODO I'm really unsure the handling of 'isgt' flag (that is, clauses
+ * with reverse order of variable/constant) is correct. I wouldn't
+ * be surprised if there was some mixup. Using the lt/gt operators
+ * instead of messing with the opproc could make it simpler.
+ * It would however be using a different operator than the query,
+ * although it's not any shadier than using the selectivity function
+ * as is done currently.
+ *
+ * FIXME Once the min/max values are deduplicated, we can easily minimize
+ * the number of calls to the comparator (assuming we keep the
+ * deduplicated structure). See the note on compression at MVBucket
+ * serialize/deserialize methods.
+ */
+ switch (get_oprrest(expr->opno))
+ {
+ case F_SCALARLTSEL: /* column < constant */
+
+ if (! isgt) /* (var < const) */
+ {
+ /*
+ * First check whether the constant is below the lower boundary (in that
+ * case we can skip the bucket, because there's no overlap).
+ */
+ tmp = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ cst->constvalue,
+ bucket->min[idx]));
+ if (tmp)
+ {
+ buckets[i] = MVSTATS_MATCH_NONE; /* no match */
+ continue;
+ }
+
+ /*
+ * Now check whether the upper boundary is below the constant (in that
+ * case it's a partial match).
+ */
+ tmp = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ cst->constvalue,
+ bucket->max[idx]));
+
+ if (tmp)
+ buckets[i] = MVSTATS_MATCH_PARTIAL; /* partial match */
+ }
+ else /* (const < var) */
+ {
+ /*
+ * First check whether the constant is above the upper boundary (in that
+ * case we can skip the bucket, because there's no overlap).
+ */
+ tmp = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ bucket->max[idx],
+ cst->constvalue));
+ if (tmp)
+ {
+ buckets[i] = MVSTATS_MATCH_NONE; /* no match */
+ continue;
+ }
+
+ /*
+ * Now check whether the lower boundary is below the constant (in that
+ * case it's a partial match).
+ */
+ tmp = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ bucket->min[idx],
+ cst->constvalue));
+
+ if (tmp)
+ buckets[i] = MVSTATS_MATCH_PARTIAL; /* partial match */
+ }
+ break;
+
+ case F_SCALARGTSEL: /* column > constant */
+
+ if (! isgt) /* (var > const) */
+ {
+ /*
+ * First check whether the constant is above the upper boundary (in that
+ * case we can skip the bucket, because there's no overlap).
+ */
+ tmp = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ cst->constvalue,
+ bucket->max[idx]));
+ if (tmp)
+ {
+ buckets[i] = MVSTATS_MATCH_NONE; /* no match */
+ continue;
+ }
+
+ /*
+ * Now check whether the lower boundary is below the constant (in that
+ * case it's a partial match).
+ */
+ tmp = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ cst->constvalue,
+ bucket->min[idx]));
+
+ if (tmp)
+ buckets[i] = MVSTATS_MATCH_PARTIAL; /* partial match */
+ }
+ else /* (const > var) */
+ {
+ /*
+ * First check whether the constant is below the lower boundary (in
+ * that case we can skip the bucket, because there's no overlap).
+ */
+ tmp = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ bucket->min[idx],
+ cst->constvalue));
+ if (tmp)
+ {
+ buckets[i] = MVSTATS_MATCH_NONE; /* no match */
+ continue;
+ }
+
+ /*
+ * Now check whether the upper boundary is below the constant (in that
+ * case it's a partial match).
+ */
+ tmp = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ bucket->max[idx],
+ cst->constvalue));
+
+ if (tmp)
+ buckets[i] = MVSTATS_MATCH_PARTIAL; /* partial match */
+ }
+
+ break;
+
+ case F_EQSEL:
+
+ /*
+ * We only check whether the value is within the bucket, using the lt/gt
+ * operators fetched from type cache.
+ *
+ * TODO We'll use the default 50% estimate, but that's probably way off
+ * if there are multiple distinct values. Consider tweaking this a
+ * somehow, e.g. using only a part inversely proportional to the
+ * estimated number of distinct values in the bucket.
+ *
+ * TODO This does not handle inclusion flags at the moment, thus counting
+ * some buckets twice (when hitting the boundary).
+ *
+ * TODO Optimization is that if max[i] == min[i], it's effectively a MCV
+ * item and we can count the whole bucket as a complete match (thus
+ * using 100% bucket selectivity and not just 50%).
+ *
+ * TODO Technically some buckets may "degenerate" into single-value
+ * buckets (not necessarily for all the dimensions) - maybe this
+ * is better than keeping a separate MCV list (multi-dimensional).
+ * Update: Actually, that's unlikely to be better than a separate
+ * MCV list for two reasons - first, it requires ~2x the space
+ * (because of storing lower/upper boundaries) and second because
+ * the buckets are ranges - depending on the partitioning algorithm
+ * it may not even degenerate into (min=max) bucket. For example the
+ * the current partitioning algorithm never does that.
+ */
+ tmp = DatumGetBool(FunctionCall2Coll(<proc,
+ DEFAULT_COLLATION_OID,
+ cst->constvalue,
+ bucket->min[idx]));
+
+ if (tmp)
+ {
+ buckets[i] = MVSTATS_MATCH_NONE; /* constvalue < min */
+ continue;
+ }
+
+ tmp = DatumGetBool(FunctionCall2Coll(<proc,
+ DEFAULT_COLLATION_OID,
+ bucket->max[idx],
+ cst->constvalue));
+
+ if (tmp)
+ {
+ buckets[i] = MVSTATS_MATCH_NONE; /* constvalue > max */
+ continue;
+ }
+
+ /* partial match */
+ buckets[i] = MVSTATS_MATCH_PARTIAL;
+
+ break;
+ }
+ }
+ }
+ }
+
+ /* now, walk through the buckets and sum the selectivities */
+ for (i = 0; i < mvhist->nbuckets; i++)
+ {
+ if (buckets[i] == MVSTATS_MATCH_FULL)
+ s += mvhist->buckets[i]->ntuples;
+ else if (buckets[i] == MVSTATS_MATCH_PARTIAL)
+ s += 0.5 * mvhist->buckets[i]->ntuples;
+ }
+
+ return s;
+}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 77d2f29..038c878 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -365,6 +365,13 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
create_generic_options alter_generic_options
relation_expr_list dostmt_opt_list
+%type <list> OptStatsOptions
+%type <str> stats_options_name
+%type <node> stats_options_arg
+%type <defelt> stats_options_elem
+%type <list> stats_options_list
+
+
%type <list> opt_fdw_options fdw_options
%type <defelt> fdw_option
@@ -483,7 +490,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <keyword> unreserved_keyword type_func_name_keyword
%type <keyword> col_name_keyword reserved_keyword
-%type <node> TableConstraint TableLikeClause
+%type <node> TableConstraint TableLikeClause TableStatistics
%type <ival> TableLikeOptionList TableLikeOption
%type <list> ColQualList
%type <node> ColConstraint ColConstraintElem ConstraintAttr
@@ -2327,6 +2334,14 @@ alter_table_cmd:
n->subtype = AT_DisableRowSecurity;
$$ = (Node *)n;
}
+ /* ALTER TABLE <name> ADD STATISTICS (options) ON (columns) ... */
+ | ADD_P TableStatistics
+ {
+ AlterTableCmd *n = makeNode(AlterTableCmd);
+ n->subtype = AT_AddStatistics;
+ n->def = $2;
+ $$ = (Node *)n;
+ }
| alter_generic_options
{
AlterTableCmd *n = makeNode(AlterTableCmd);
@@ -3397,6 +3412,56 @@ OptConsTableSpace: USING INDEX TABLESPACE name { $$ = $4; }
ExistingIndex: USING INDEX index_name { $$ = $3; }
;
+/*****************************************************************************
+ *
+ * QUERY :
+ * ALTER TABLE relname ADD STATISTICS (columns) WITH (options)
+ *
+ *****************************************************************************/
+
+TableStatistics:
+ STATISTICS OptStatsOptions ON '(' columnList ')'
+ {
+ StatisticsDef *n = makeNode(StatisticsDef);
+ n->keys = $5;
+ n->options = $2;
+ $$ = (Node *) n;
+ }
+ ;
+
+OptStatsOptions:
+ '(' stats_options_list ')' { $$ = $2; }
+ | /*EMPTY*/ { $$ = NIL; }
+ ;
+
+stats_options_list:
+ stats_options_elem
+ {
+ $$ = list_make1($1);
+ }
+ | stats_options_list ',' stats_options_elem
+ {
+ $$ = lappend($1, $3);
+ }
+ ;
+
+stats_options_elem:
+ stats_options_name stats_options_arg
+ {
+ $$ = makeDefElem($1, $2);
+ }
+ ;
+
+stats_options_name:
+ NonReservedWord { $$ = $1; }
+ ;
+
+stats_options_arg:
+ opt_boolean_or_string { $$ = (Node *) makeString($1); }
+ | NumericOnly { $$ = (Node *) $1; }
+ | /* EMPTY */ { $$ = NULL; }
+ ;
+
/*****************************************************************************
*
diff --git a/src/backend/utils/cache/syscache.c b/src/backend/utils/cache/syscache.c
index 94d951c..ec90773 100644
--- a/src/backend/utils/cache/syscache.c
+++ b/src/backend/utils/cache/syscache.c
@@ -43,6 +43,7 @@
#include "catalog/pg_foreign_server.h"
#include "catalog/pg_foreign_table.h"
#include "catalog/pg_language.h"
+#include "catalog/pg_mv_statistic.h"
#include "catalog/pg_namespace.h"
#include "catalog/pg_opclass.h"
#include "catalog/pg_operator.h"
@@ -499,6 +500,17 @@ static const struct cachedesc cacheinfo[] = {
},
4
},
+ {MvStatisticRelationId, /* MVSTATOID */
+ MvStatisticOidIndexId,
+ 1,
+ {
+ ObjectIdAttributeNumber,
+ 0,
+ 0,
+ 0
+ },
+ 128
+ },
{NamespaceRelationId, /* NAMESPACENAME */
NamespaceNameIndexId,
1,
diff --git a/src/include/catalog/indexing.h b/src/include/catalog/indexing.h
index 870692c..d57cdbe 100644
--- a/src/include/catalog/indexing.h
+++ b/src/include/catalog/indexing.h
@@ -173,6 +173,11 @@ DECLARE_UNIQUE_INDEX(pg_largeobject_loid_pn_index, 2683, on pg_largeobject using
DECLARE_UNIQUE_INDEX(pg_largeobject_metadata_oid_index, 2996, on pg_largeobject_metadata using btree(oid oid_ops));
#define LargeObjectMetadataOidIndexId 2996
+DECLARE_UNIQUE_INDEX(pg_mv_statistic_oid_index, 3259, on pg_mv_statistic using btree(oid oid_ops));
+#define MvStatisticOidIndexId 3259
+DECLARE_INDEX(pg_mv_statistic_relid_index, 3264, on pg_mv_statistic using btree(starelid oid_ops));
+#define MvStatisticRelidIndexId 3264
+
DECLARE_UNIQUE_INDEX(pg_namespace_nspname_index, 2684, on pg_namespace using btree(nspname name_ops));
#define NamespaceNameIndexId 2684
DECLARE_UNIQUE_INDEX(pg_namespace_oid_index, 2685, on pg_namespace using btree(oid oid_ops));
diff --git a/src/include/catalog/pg_mv_statistic.h b/src/include/catalog/pg_mv_statistic.h
new file mode 100644
index 0000000..703931e
--- /dev/null
+++ b/src/include/catalog/pg_mv_statistic.h
@@ -0,0 +1,89 @@
+/*-------------------------------------------------------------------------
+ *
+ * pg_mv_statistic.h
+ * definition of the system "multivariate statistic" relation (pg_mv_statistic)
+ * along with the relation's initial contents.
+ *
+ *
+ * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/catalog/pg_mv_statistic.h
+ *
+ * NOTES
+ * the genbki.pl script reads this file and generates .bki
+ * information from the DATA() statements.
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef PG_MV_STATISTIC_H
+#define PG_MV_STATISTIC_H
+
+#include "catalog/genbki.h"
+
+/* ----------------
+ * pg_mv_statistic definition. cpp turns this into
+ * typedef struct FormData_pg_mv_statistic
+ * ----------------
+ */
+#define MvStatisticRelationId 3260
+
+CATALOG(pg_mv_statistic,3260)
+{
+ /* These fields form the unique key for the entry: */
+ Oid starelid; /* relation containing attributes */
+
+ /* statistics requested to build */
+ bool hist_enabled; /* build histogram? */
+ bool mcv_enabled; /* build MCV list? */
+ bool mcv_hashed; /* build hashed MCV? */
+ bool assoc_enabled; /* analyze associations? */
+
+ /* histogram / MCV size */
+ int32 hist_max_buckets; /* max buckets */
+ int32 mcv_max_items; /* max MCV items */
+
+ /* statistics that are available (if requested) */
+ bool hist_built; /* histogram was built */
+ bool mcv_built; /* MCV list was built */
+ bool assoc_built; /* associations were built */
+
+ /* variable-length fields start here, but we allow direct access to stakeys */
+ int2vector stakeys; /* array of column keys */
+
+#ifdef CATALOG_VARLEN
+ bytea staassoc; /* association rules (serialized) */
+ bytea stamcv; /* MCV list (serialized) */
+ bytea stahist; /* MV histogram (serialized) */
+#endif
+
+} FormData_pg_mv_statistic;
+
+/* ----------------
+ * Form_pg_mv_statistic corresponds to a pointer to a tuple with
+ * the format of pg_mv_statistic relation.
+ * ----------------
+ */
+typedef FormData_pg_mv_statistic *Form_pg_mv_statistic;
+
+/* ----------------
+ * compiler constants for pg_attrdef
+ * ----------------
+ */
+#define Natts_pg_mv_statistic 14
+#define Anum_pg_mv_statistic_starelid 1
+#define Anum_pg_mv_statistic_hist_enabled 2
+#define Anum_pg_mv_statistic_mcv_enabled 3
+#define Anum_pg_mv_statistic_mcv_hashed 4
+#define Anum_pg_mv_statistic_assoc_enabled 5
+#define Anum_pg_mv_statistic_hist_max_buckets 6
+#define Anum_pg_mv_statistic_mcv_max_items 7
+#define Anum_pg_mv_statistic_hist_built 8
+#define Anum_pg_mv_statistic_mcv_built 9
+#define Anum_pg_mv_statistic_assoc_built 10
+#define Anum_pg_mv_statistic_stakeys 11
+#define Anum_pg_mv_statistic_staassoc 12
+#define Anum_pg_mv_statistic_stamcv 13
+#define Anum_pg_mv_statistic_stahist 14
+
+#endif /* PG_MV_STATISTIC_H */
diff --git a/src/include/catalog/pg_proc.h b/src/include/catalog/pg_proc.h
index 3ce9849..6961b7c 100644
--- a/src/include/catalog/pg_proc.h
+++ b/src/include/catalog/pg_proc.h
@@ -2647,6 +2647,13 @@ DESCR("current user privilege on any column by rel name");
DATA(insert OID = 3029 ( has_any_column_privilege PGNSP PGUID 12 10 0 0 0 f f f f t f s 2 0 16 "26 25" _null_ _null_ _null_ _null_ has_any_column_privilege_id _null_ _null_ _null_ ));
DESCR("current user privilege on any column by rel oid");
+DATA(insert OID = 3261 ( pg_mv_stats_histogram_info PGNSP PGUID 12 1 0 0 0 f f f f t f i 1 0 25 "17" _null_ _null_ _null_ _null_ pg_mv_stats_histogram_info _null_ _null_ _null_ ));
+DESCR("multi-variate statistics: histogram info");
+DATA(insert OID = 3262 ( pg_mv_stats_mvclist_info PGNSP PGUID 12 1 0 0 0 f f f f t f i 1 0 25 "17" _null_ _null_ _null_ _null_ pg_mv_stats_mvclist_info _null_ _null_ _null_ ));
+DESCR("multi-variate statistics: MCV list info");
+DATA(insert OID = 3263 ( pg_mv_stats_histogram_gnuplot PGNSP PGUID 12 1 0 0 0 f f f f t f i 1 0 25 "17" _null_ _null_ _null_ _null_ pg_mv_stats_histogram_gnuplot _null_ _null_ _null_ ));
+DESCR("multi-variate statistics: 2D histogram gnuplot");
+
DATA(insert OID = 1928 ( pg_stat_get_numscans PGNSP PGUID 12 1 0 0 0 f f f f t f s 1 0 20 "26" _null_ _null_ _null_ _null_ pg_stat_get_numscans _null_ _null_ _null_ ));
DESCR("statistics: number of scans done for table/index");
DATA(insert OID = 1929 ( pg_stat_get_tuples_returned PGNSP PGUID 12 1 0 0 0 f f f f t f s 1 0 20 "26" _null_ _null_ _null_ _null_ pg_stat_get_tuples_returned _null_ _null_ _null_ ));
diff --git a/src/include/catalog/toasting.h b/src/include/catalog/toasting.h
index a4af551..02b9aa3 100644
--- a/src/include/catalog/toasting.h
+++ b/src/include/catalog/toasting.h
@@ -49,6 +49,7 @@ extern void BootstrapToastTable(char *relName,
DECLARE_TOAST(pg_attrdef, 2830, 2831);
DECLARE_TOAST(pg_constraint, 2832, 2833);
DECLARE_TOAST(pg_description, 2834, 2835);
+DECLARE_TOAST(pg_mv_statistic, 3952, 3954);
DECLARE_TOAST(pg_proc, 2836, 2837);
DECLARE_TOAST(pg_rewrite, 2838, 2839);
DECLARE_TOAST(pg_seclabel, 3598, 3599);
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 154d943..36e675b 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -410,6 +410,7 @@ typedef enum NodeTag
T_XmlSerialize,
T_WithClause,
T_CommonTableExpr,
+ T_StatisticsDef,
/*
* TAGS FOR REPLICATION GRAMMAR PARSE NODES (replnodes.h)
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index f3aa69e..e7ed773 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -542,6 +542,14 @@ typedef struct ColumnDef
int location; /* parse location, or -1 if none/unknown */
} ColumnDef;
+typedef struct StatisticsDef
+{
+ NodeTag type;
+ List *keys; /* String nodes naming referenced column(s) */
+ List *options; /* list of DefElem nodes */
+} StatisticsDef;
+
+
/*
* TableLikeClause - CREATE TABLE ( ... LIKE ... ) clause
*/
@@ -1337,7 +1345,8 @@ typedef enum AlterTableType
AT_ReplicaIdentity, /* REPLICA IDENTITY */
AT_EnableRowSecurity, /* ENABLE ROW SECURITY */
AT_DisableRowSecurity, /* DISABLE ROW SECURITY */
- AT_GenericOptions /* OPTIONS (...) */
+ AT_GenericOptions, /* OPTIONS (...) */
+ AT_AddStatistics /* add statistics */
} AlterTableType;
typedef struct ReplicaIdentityStmt
diff --git a/src/include/utils/mvstats.h b/src/include/utils/mvstats.h
new file mode 100644
index 0000000..157891a
--- /dev/null
+++ b/src/include/utils/mvstats.h
@@ -0,0 +1,283 @@
+/*-------------------------------------------------------------------------
+ *
+ * mvstats.h
+ * Multivariate statistics and selectivity estimation functions.
+ *
+ *
+ * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/utils/mvstats.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef MVSTATS_H
+#define MVSTATS_H
+
+/*
+ * Multivariate statistics for planner/optimizer, implementing extensions
+ * of the single-column statistics:
+ *
+ * - multivariate MCV list
+ * - multivariate histograms
+ *
+ * There's also an experimental support for associative rules (values in
+ * one column implying values in other columns - e.g. ZIP code implies
+ * name of a city, etc.).
+ *
+ * The current implementation has various limitations:
+ *
+ * (a) it supports only data types passed by value
+ *
+ * (b) no support for NULL values
+ *
+ * Both (a) and (b) should be straightforward to fix (and usually
+ * described in comments at related data structures or functions).
+ *
+ * The stats may be built only directly on columns, not on expressions.
+ * And there are usually some additional technical limits (e.g. number
+ * of columns in a histogram, etc.).
+ *
+ * Those limits serve mostly as sanity checks and while increasing them
+ * is possible (the implementation should not break), it's expected to
+ * lead either to very bad precision or expensive planning.
+ */
+
+/*
+ * Multivariate histograms
+ *
+ * Histograms are a collection of buckets, represented by n-dimensional
+ * rectangles. Each rectangle is delimited by an array of lower and
+ * upper boundaries, so that for for the i-th attribute
+ *
+ * min[i] <= value[i] <= max[i]
+ *
+ * Each bucket tracks frequency (fraction of tuples it contains),
+ * information about the inequalities, number of distinct values in
+ * each dimension (which is used when building the histogram) etc.
+ *
+ * The boundaries may be either inclusive or exclusive, or the whole
+ * dimension may be NULL.
+ *
+ * The buckets may overlap (assuming the build algorithm keeps the
+ * frequencies additive) or may not cover the whole space (i.e. allow
+ * gaps). This entirely depends on the algorithm used to build the
+ * histogram.
+ *
+ * The histograms are marked with a 'magic' constant, mostly to make
+ * sure the bytea really is a histogram in serialized form.
+ *
+ * We do expect to support multiple histogram types, with different
+ * features etc. The 'type' field is used to identify those types.
+ * Technically some histogram types might use completely different
+ * bucket representation, but that's not expected at the moment.
+ *
+ * TODO Add pointer to 'private' data, meant for private data for
+ * other algorithms for building the histogram.
+ *
+ * TODO The current implementation does not handle NULL values (it's
+ * somehow prepared for that, but the algorithm building the
+ * histogram ignores them). The idea is to build buckets with one
+ * or more NULL-only dimensions - there'll be at most 2^ndimensions
+ * such buckets, which for 8 atttributes (current limit) is 256.
+ * That's quite reasonable, considering we expect thousands of
+ * buckets in total.
+ *
+ * TODO This structure is used both when building the histogram, and
+ * then when using it to compute estimates. That's why the last
+ * few elements are not used once the histogram is built.
+ *
+ * TODO The limit on number of buckets is quite arbitrary, aiming for
+ * sufficient accuracy while still being fast. Probably should be
+ * replaced with a dynamic limit dependent on statistics target,
+ * number of attributes (dimensions) and statistics target
+ * associated with the attributes. Also, this needs to be related
+ * to the number of sampled rows, by either clamping it to a
+ * reasonable number (after seeing the number of rows) or using
+ * it when computing the number of rows to sample. Something like
+ * 10 rows per bucket seems reasonable.
+ *
+ * TODO We may replace the bool arrays with a suitably large data type
+ * (say, uint16 or uint32) and get rid of the allocations. It's
+ * unlikely we'll ever support more than 32 columns as that'd
+ * result in poor precision, huge histograms (splitting each
+ * dimension once would mean 2^32 buckets), and very expensive
+ * estimation. MCVItem already does it this way.
+ *
+ * TODO Actually the distinct stats (both for combination of all columns
+ * and for combinations of various subsets of columns) should be
+ * moved to a separate structure (next to histogram/MCV/...) to
+ * make it useful even without a histogram computed etc.
+ */
+typedef struct MVBucketData {
+
+ /* Frequencies of this bucket. */
+ float ntuples; /* frequency of tuples tuples */
+ float ndistinct; /* frequency of distinct values */
+
+ /*
+ * Number of distinct values in each dimension. This is used when
+ * building the histogram (and is not serialized/deserialized), but
+ * it could be useful for estimating ndistinct for combinations of
+ * columns.
+ *
+ * It would mean tracking 2^N values for each bucket, and even if
+ * those values might be stores in 1B it's still a lot of space
+ * (considering the expected number of buckets).
+ *
+ * TODO Consider tracking ndistincts for all attribute combinations.
+ */
+ uint32 *ndistincts;
+
+ /*
+ * Information about dimensions being NULL-only. Not yet used.
+ */
+ bool *nullsonly;
+
+ /* lower boundaries - values and information about the inequalities */
+ Datum *min;
+ bool *min_inclusive;
+
+ /* upper boundaries - values and information about the inequalities */
+ Datum *max;
+ bool *max_inclusive;
+
+ /*
+ * Sample tuples falling into this bucket, index of the dimension
+ * the bucket was split by in the last step.
+ *
+ * XXX These fields are needed only while building the histogram,
+ * and are not serialized at all.
+ */
+ HeapTuple *rows;
+ uint32 numrows;
+ int last_split_dimension;
+
+} MVBucketData;
+
+typedef MVBucketData *MVBucket;
+
+
+typedef struct MVHistogramData {
+
+ uint32 magic; /* magic constant marker */
+ uint32 type; /* type of histogram (BASIC) */
+ uint32 nbuckets; /* number of buckets (buckets array) */
+ uint32 ndimensions; /* number of dimensions */
+
+ MVBucket *buckets; /* array of buckets */
+
+} MVHistogramData;
+
+typedef MVHistogramData *MVHistogram;
+
+
+/* used to flag stats serialized to bytea */
+#define MVHIST_MAGIC 0x7F8C5670 /* marks serialized bytea */
+#define MVHIST_TYPE_BASIC 1 /* basic histogram type */
+
+/* limits (mostly sanity check, may be relaxed in the future) */
+#define MVHIST_MAX_BUCKETS 16384 /* max number of buckets */
+
+/* bucket size in a serialized form */
+#define BUCKET_SIZE_SERIALIZED(ndims) \
+ (offsetof(MVBucketData, ndistincts) + \
+ (ndims) * (2 * sizeof(uint16) + sizeof(uint32) + 3 * sizeof(bool)))
+
+
+/*
+ * Multivariate MCV (most-common value) lists
+ *
+ * A straight-forward extension of MCV items - i.e. a list (array) of
+ * combinations of attribute values, together with a frequency and
+ * null flags.
+ *
+ * This already uses the trick with using uint32 as a null bitmap.
+ *
+ * TODO Shouldn't the MCVItemData use plain pointer for values, instead
+ * of the single-item array trick?
+ *
+ * TODO It's possible to build a special case of MCV list, storing not
+ * the actual values but only 32/64-bit hash. This is only useful
+ * for estimating equality clauses and for large varlena types.
+ */
+typedef struct MCVItemData {
+ double frequency; /* frequency of this combination */
+ uint32 nulls; /* lags of NULL values (up to 32 columns) */
+ Datum values[1]; /* variable-length (ndimensions) */
+} MCVItemData;
+
+typedef MCVItemData *MCVItem;
+
+/* multivariate MCV list - essentally an array of MCV items */
+typedef struct MCVListData {
+ uint32 magic; /* magic constant marker */
+ uint32 type; /* type of MCV list (BASIC) */
+ uint32 ndimensions; /* number of dimensions */
+ uint32 nitems; /* number of MCV items in the array */
+ MCVItem *items; /* array of MCV items */
+} MCVListData;
+
+typedef MCVListData *MCVList;
+
+/* used to flag stats serialized to bytea */
+#define MVSTAT_MCV_MAGIC 0xE1A651C2 /* marks serialized bytea */
+#define MVSTAT_MCV_TYPE_BASIC 1 /* basic MCV list type */
+
+/* TODO consider increasing the limit, and/or using statistics target */
+#define MVSTAT_MCVLIST_MAX_ITEMS 1024 /* max items in MCV list */
+
+
+/*
+ * Basic info about the stats, used when choosing what to use
+ *
+ * TODO Add info about what statistics is available (histogram, MCV,
+ * hashed MCV, assciative rules).
+ */
+typedef struct MVStatsData {
+ Oid mvoid; /* OID of the stats in pg_mv_statistic */
+ int2vector *stakeys; /* attnums for columns in the stats */
+ bool hist_built; /* histogram is already available */
+ bool mcv_built; /* MCV list is already available */
+ bool assoc_built; /* associative rules available */
+} MVStatsData;
+
+typedef struct MVStatsData *MVStats;
+
+
+/*
+ * Degree of how much MCV item / histogram bucket matches a clause.
+ * This is then considered when computing the selectivity.
+ */
+#define MVSTATS_MATCH_NONE 0 /* no match at all */
+#define MVSTATS_MATCH_PARTIAL 1 /* partial match */
+#define MVSTATS_MATCH_FULL 2 /* full match */
+
+
+#define MVSTATS_MAX_DIMENSIONS 8 /* max number of attributes */
+
+/*
+ * TODO Maybe fetching the histogram/MCV list separately is inefficient?
+ * Consider adding a single `fetch_stats` method, fetching all
+ * stats specified using flags (or something like that).
+ */
+MVStats list_mv_stats(Oid relid, int *nstats, bool built_only);
+bytea * fetch_mv_histogram(Oid mvoid);
+bytea * fetch_mv_mcvlist(Oid mvoid);
+
+/* deserialization of stats (serialization is private to analyze) */
+MVHistogram deserialize_mv_histogram(bytea * data);
+MCVList deserialize_mv_mcvlist(bytea * data);
+
+/*
+ * Returns index of the attribute number within the vector (i.e. a
+ * dimension within the stats).
+ */
+int mv_get_index(AttrNumber varattno, int2vector * stakeys);
+
+/* FIXME this probably belongs somewhere else (not to operations stats) */
+extern Datum pg_mv_stats_histogram_info(PG_FUNCTION_ARGS);
+extern Datum pg_mv_stats_histogram_gnuplot(PG_FUNCTION_ARGS);
+extern Datum pg_mv_stats_mvclist_info(PG_FUNCTION_ARGS);
+
+#endif
diff --git a/src/include/utils/syscache.h b/src/include/utils/syscache.h
index f97229f..a275bd5 100644
--- a/src/include/utils/syscache.h
+++ b/src/include/utils/syscache.h
@@ -66,6 +66,7 @@ enum SysCacheIdentifier
INDEXRELID,
LANGNAME,
LANGOID,
+ MVSTATOID,
NAMESPACENAME,
NAMESPACEOID,
OPERNAMENSP,
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: WIP: multivariate statistics / proof of concept
2014-10-12 22:00 WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
@ 2014-10-13 07:36 ` Albe Laurenz <laurenz.albe@wien.gv.at>
2014-10-13 19:47 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
3 siblings, 1 reply; 37+ messages in thread
From: Albe Laurenz @ 2014-10-13 07:36 UTC (permalink / raw)
To: Tomas Vondra *EXTERN* <tv@fuzzy.cz>; pgsql-hackers
Tomas Vondra wrote:
> attached is a WIP patch implementing multivariate statistics.
I think that is pretty useful.
Oracle has an identical feature called "extended statistics".
That's probably an entirely different thing, but it would be very
nice to have statistics to estimate the correlation between columns
of different tables, to improve the estimate for the number of rows
in a join.
Yours,
Laurenz Albe
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: WIP: multivariate statistics / proof of concept
2014-10-12 22:00 WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-10-13 07:36 ` Re: WIP: multivariate statistics / proof of concept Albe Laurenz <laurenz.albe@wien.gv.at>
@ 2014-10-13 19:47 ` Tomas Vondra <tv@fuzzy.cz>
0 siblings, 0 replies; 37+ messages in thread
From: Tomas Vondra @ 2014-10-13 19:47 UTC (permalink / raw)
To: pgsql-hackers
Hi!
On 13.10.2014 09:36, Albe Laurenz wrote:
> Tomas Vondra wrote:
>> attached is a WIP patch implementing multivariate statistics.
>
> I think that is pretty useful.
> Oracle has an identical feature called "extended statistics".
>
> That's probably an entirely different thing, but it would be very
> nice to have statistics to estimate the correlation between columns
> of different tables, to improve the estimate for the number of rows
> in a join.
I don't have a clear idea of how that should work, but from the quick
look at how join selectivity estimation is implemented, I believe two
things might be possible:
(a) using conditional probabilities
Say we have a join "ta JOIN tb ON (ta.x = tb.y)"
Currently, the selectivity is derived from stats on the two keys.
Essentially probabilities P(x), P(y), represented by the MCV lists.
But if there are additional WHERE conditions on the tables, and we
have suitable multivariate stats, it's possible to use conditional
probabilities.
E.g. if the query actually uses
... ta JOIN tb ON (ta.x = tb.y) WHERE ta.z = 10
and we have stats on (ta.x, ta.z), we can use P(x|z=10) instead.
If the two columns are correlated, this might be much different.
(b) using this for multi-column conditions
If the join condition involves multiple columns, e.g.
ON (ta.x = tb.y AND ta.p = tb.q)
and we happen to have stats on (ta.x,ta.p) and (tb.y,tb.q), we may
use this to compute the cardinality (pretty much as we do today).
But I haven't really worked on this so far, I suspect there are various
subtle issues and I certainly don't plan to address this in the first
phase of the patch.
Tomas
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: WIP: multivariate statistics / proof of concept
2014-10-12 22:00 WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
@ 2014-10-29 09:41 ` David Rowley <dgrowleyml@gmail.com>
2014-10-29 11:21 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-10-29 11:31 ` Re: WIP: multivariate statistics / proof of concept Petr Jelinek <petr@2ndquadrant.com>
3 siblings, 2 replies; 37+ messages in thread
From: David Rowley @ 2014-10-29 09:41 UTC (permalink / raw)
To: Tomas Vondra <tv@fuzzy.cz>; +Cc: pgsql-hackers
On Mon, Oct 13, 2014 at 11:00 AM, Tomas Vondra <tv@fuzzy.cz> wrote:
> Hi,
>
> attached is a WIP patch implementing multivariate statistics. The code
> certainly is not "ready" - parts of it look as if written by a rogue
> chimp who got bored of attempts to type the complete works of William
> Shakespeare, and decided to try something different.
>
>
I'm really glad you're working on this. I had been thinking of looking into
doing this myself.
> The last point is really just "unfinished implementation" - the syntax I
> propose is this:
>
> ALTER TABLE ... ADD STATISTICS (options) ON (columns)
>
> where the options influence the MCV list and histogram size, etc. The
> options are recognized and may give you an idea of what it might do, but
> it's not really used at the moment (except for storing in the
> pg_mv_statistic catalog).
>
>
>
I've not really gotten around to looking at the patch yet, but I'm also
wondering if it would be simple include allowing functional statistics too.
The pg_mv_statistic name seems to indicate multi columns, but how about
stats on date(datetime_column), or perhaps any non-volatile function. This
would help to solve the problem highlighted here
http://www.postgresql.org/message-id/CAApHDvp2vH=7O-gp-zAf7aWy+A-WHWVg7h3Vc6=5pf9Uf34DhQ@mail.gmail....
. Without giving it too much thought, perhaps any expression that can be
indexed should be allowed to have stats? Would that be really difficult to
implement in comparison to what you've already done with the patch so far?
I'm quite interested in reviewing your work on this, but it appears that
some of your changes are not C89:
src\backend\commands\analyze.c(3774): error C2057: expected constant
expression [D:\Postgres\a\postgres.vcxproj]
src\backend\commands\analyze.c(3774): error C2466: cannot allocate an
array of constant size 0 [D:\Postgres\a\postgres.vcxproj]
src\backend\commands\analyze.c(3774): error C2133: 'indexes' : unknown
size [D:\Postgres\a\postgres.vcxproj]
src\backend\commands\analyze.c(4302): error C2057: expected constant
expression [D:\Postgres\a\postgres.vcxproj]
src\backend\commands\analyze.c(4302): error C2466: cannot allocate an
array of constant size 0 [D:\Postgres\a\postgres.vcxproj]
src\backend\commands\analyze.c(4302): error C2133: 'ndistincts' : unknown
size [D:\Postgres\a\postgres.vcxproj]
src\backend\commands\analyze.c(4775): error C2057: expected constant
expression [D:\Postgres\a\postgres.vcxproj]
src\backend\commands\analyze.c(4775): error C2466: cannot allocate an
array of constant size 0 [D:\Postgres\a\postgres.vcxproj]
src\backend\commands\analyze.c(4775): error C2133: 'keys' : unknown size
[D:\Postgres\a\postgres.vcxproj]
The compiler I'm using is a bit too stupid to understand the C99 syntax.
I guess you'd need to palloc() these arrays instead in order to comply with
the project standards.
http://www.postgresql.org/docs/devel/static/install-requirements.html
I'm going to sign myself up to review this, so probably my first feedback
would be the compiling problem.
Regards
David Rowley
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: WIP: multivariate statistics / proof of concept
2014-10-12 22:00 WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-10-29 09:41 ` Re: WIP: multivariate statistics / proof of concept David Rowley <dgrowleyml@gmail.com>
@ 2014-10-29 11:21 ` Tomas Vondra <tv@fuzzy.cz>
2014-10-30 09:23 ` Re: WIP: multivariate statistics / proof of concept David Rowley <dgrowleyml@gmail.com>
1 sibling, 1 reply; 37+ messages in thread
From: Tomas Vondra @ 2014-10-29 11:21 UTC (permalink / raw)
To: David Rowley <dgrowleyml@gmail.com>; +Cc: Tomas Vondra <tv@fuzzy.cz>; pgsql-hackers
Dne 29 Říjen 2014, 10:41, David Rowley napsal(a):
>
> I've not really gotten around to looking at the patch yet, but I'm also
> wondering if it would be simple include allowing functional statistics
> too.
> The pg_mv_statistic name seems to indicate multi columns, but how about
> stats on date(datetime_column), or perhaps any non-volatile function. This
> would help to solve the problem highlighted here
> http://www.postgresql.org/message-id/CAApHDvp2vH=7O-gp-zAf7aWy+A-WHWVg7h3Vc6=5pf9Uf34DhQ@mail.gmail....
> . Without giving it too much thought, perhaps any expression that can be
> indexed should be allowed to have stats? Would that be really difficult to
> implement in comparison to what you've already done with the patch so far?
I don't know, but it seems mostly orthogonal to what the patch aims to do.
If we add collecting statistics on expressions (on a single column), then I'd
expect it to be reasonably simple to add this to the multi-column case.
There are features like join stats or range type stats, that are probably
more directly related to the patch (but out of scope for the initial
version).
> I'm quite interested in reviewing your work on this, but it appears that
> some of your changes are not C89:
>
> src\backend\commands\analyze.c(3774): error C2057: expected constant
> expression [D:\Postgres\a\postgres.vcxproj]
> src\backend\commands\analyze.c(3774): error C2466: cannot allocate an
> array of constant size 0 [D:\Postgres\a\postgres.vcxproj]
> src\backend\commands\analyze.c(3774): error C2133: 'indexes' : unknown
> size [D:\Postgres\a\postgres.vcxproj]
> src\backend\commands\analyze.c(4302): error C2057: expected constant
> expression [D:\Postgres\a\postgres.vcxproj]
> src\backend\commands\analyze.c(4302): error C2466: cannot allocate an
> array of constant size 0 [D:\Postgres\a\postgres.vcxproj]
> src\backend\commands\analyze.c(4302): error C2133: 'ndistincts' : unknown
> size [D:\Postgres\a\postgres.vcxproj]
> src\backend\commands\analyze.c(4775): error C2057: expected constant
> expression [D:\Postgres\a\postgres.vcxproj]
> src\backend\commands\analyze.c(4775): error C2466: cannot allocate an
> array of constant size 0 [D:\Postgres\a\postgres.vcxproj]
> src\backend\commands\analyze.c(4775): error C2133: 'keys' : unknown size
> [D:\Postgres\a\postgres.vcxproj]
>
> The compiler I'm using is a bit too stupid to understand the C99 syntax.
>
> I guess you'd need to palloc() these arrays instead in order to comply
> with
> the project standards.
>
> http://www.postgresql.org/docs/devel/static/install-requirements.html
>
> I'm going to sign myself up to review this, so probably my first feedback
> would be the compiling problem.
I'll look into that. The thing is I don't have access to MSVC, so it's a bit
difficult to spot / fix those issues :-(
regards
Tomas
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: WIP: multivariate statistics / proof of concept
2014-10-12 22:00 WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-10-29 09:41 ` Re: WIP: multivariate statistics / proof of concept David Rowley <dgrowleyml@gmail.com>
2014-10-29 11:21 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
@ 2014-10-30 09:23 ` David Rowley <dgrowleyml@gmail.com>
2014-11-10 02:34 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
0 siblings, 1 reply; 37+ messages in thread
From: David Rowley @ 2014-10-30 09:23 UTC (permalink / raw)
To: Tomas Vondra <tv@fuzzy.cz>; +Cc: pgsql-hackers
On Thu, Oct 30, 2014 at 12:21 AM, Tomas Vondra <tv@fuzzy.cz> wrote:
> Dne 29 Říjen 2014, 10:41, David Rowley napsal(a):
> > I'm quite interested in reviewing your work on this, but it appears that
> > some of your changes are not C89:
> >
> > src\backend\commands\analyze.c(3774): error C2057: expected constant
> > expression [D:\Postgres\a\postgres.vcxproj]
> > src\backend\commands\analyze.c(3774): error C2466: cannot allocate an
> > array of constant size 0 [D:\Postgres\a\postgres.vcxproj]
> > src\backend\commands\analyze.c(3774): error C2133: 'indexes' : unknown
> > size [D:\Postgres\a\postgres.vcxproj]
> > src\backend\commands\analyze.c(4302): error C2057: expected constant
> > expression [D:\Postgres\a\postgres.vcxproj]
> > src\backend\commands\analyze.c(4302): error C2466: cannot allocate an
> > array of constant size 0 [D:\Postgres\a\postgres.vcxproj]
> > src\backend\commands\analyze.c(4302): error C2133: 'ndistincts' :
> unknown
> > size [D:\Postgres\a\postgres.vcxproj]
> > src\backend\commands\analyze.c(4775): error C2057: expected constant
> > expression [D:\Postgres\a\postgres.vcxproj]
> > src\backend\commands\analyze.c(4775): error C2466: cannot allocate an
> > array of constant size 0 [D:\Postgres\a\postgres.vcxproj]
> > src\backend\commands\analyze.c(4775): error C2133: 'keys' : unknown size
> > [D:\Postgres\a\postgres.vcxproj]
> >
>
> I'll look into that. The thing is I don't have access to MSVC, so it's a
> bit
> difficult to spot / fix those issues :-(
>
>
It should be a pretty simple fix, just use the files and line numbers from
the above. It's just a problem that in those 3 places you're declaring an
array of a variable size, which is not allowed in C89. The thing to do
instead would just be to palloc() the size you need and the pfree() it when
you're done.
Regards
David Rowley
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: WIP: multivariate statistics / proof of concept
2014-10-12 22:00 WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-10-29 09:41 ` Re: WIP: multivariate statistics / proof of concept David Rowley <dgrowleyml@gmail.com>
2014-10-29 11:21 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-10-30 09:23 ` Re: WIP: multivariate statistics / proof of concept David Rowley <dgrowleyml@gmail.com>
@ 2014-11-10 02:34 ` Tomas Vondra <tv@fuzzy.cz>
0 siblings, 0 replies; 37+ messages in thread
From: Tomas Vondra @ 2014-11-10 02:34 UTC (permalink / raw)
To: pgsql-hackers
On 30.10.2014 10:23, David Rowley wrote:
> On Thu, Oct 30, 2014 at 12:21 AM, Tomas Vondra <tv@fuzzy.cz
> <mailto:tv@fuzzy.cz>> wrote:
>
> Dne 29 Říjen 2014, 10:41, David Rowley napsal(a):
> > I'm quite interested in reviewing your work on this, but it
> appears that
> > some of your changes are not C89:
> >
> > src\backend\commands\analyze.c(3774): error C2057: expected constant
> > expression [D:\Postgres\a\postgres.vcxproj]
> > src\backend\commands\analyze.c(3774): error C2466: cannot allocate an
> > array of constant size 0 [D:\Postgres\a\postgres.vcxproj]
> > src\backend\commands\analyze.c(3774): error C2133: 'indexes' :
> unknown
> > size [D:\Postgres\a\postgres.vcxproj]
> > src\backend\commands\analyze.c(4302): error C2057: expected constant
> > expression [D:\Postgres\a\postgres.vcxproj]
> > src\backend\commands\analyze.c(4302): error C2466: cannot allocate an
> > array of constant size 0 [D:\Postgres\a\postgres.vcxproj]
> > src\backend\commands\analyze.c(4302): error C2133: 'ndistincts' :
> unknown
> > size [D:\Postgres\a\postgres.vcxproj]
> > src\backend\commands\analyze.c(4775): error C2057: expected constant
> > expression [D:\Postgres\a\postgres.vcxproj]
> > src\backend\commands\analyze.c(4775): error C2466: cannot allocate an
> > array of constant size 0 [D:\Postgres\a\postgres.vcxproj]
> > src\backend\commands\analyze.c(4775): error C2133: 'keys' :
> unknown size
> > [D:\Postgres\a\postgres.vcxproj]
> >
>
> I'll look into that. The thing is I don't have access to MSVC, so
> it's a bit difficult to spot / fix those issues :-(
>
>
> It should be a pretty simple fix, just use the files and line
> numbers from the above. It's just a problem that in those 3 places
> you're declaring an array of a variable size, which is not allowed in
> C89. The thing to do instead would just be to palloc() the size you
> need and the pfree() it when you're done.
Attached is a patch that should fix these issues.
The bad news is there are a few installcheck failures (and were in the
previous patch, but I haven't noticed for some reason). Apparently,
there's some mixup in how the patch handles Var->varno in some causes,
causing issues with a handful of regression tests.
The problem is that is_mv_compatible (checking whether the condition is
compatible with multivariate stats) does this
if (! ((varRelid == 0) || (varRelid == var->varno)))
return false;
/* Also skip special varno values, and system attributes ... */
if ((IS_SPECIAL_VARNO(var->varno)) ||
(! AttrNumberIsForUserDefinedAttr(var->varattno)))
return false;
assuming that after this, varno represents an index into the range
table, and passes it out to the caller.
And the caller (collect_mv_attnums) does this:
RelOptInfo *rel = find_base_rel(root, varno);
which fails with errors like these:
ERROR: no relation entry for relid 0
ERROR: no relation entry for relid 1880
or whatever. What's even stranger is this:
regression=# SELECT table_name, is_updatable, is_insertable_into
regression-# FROM information_schema.views
regression-# WHERE table_name = 'rw_view1';
ERROR: no relation entry for relid 0
regression=# SELECT table_name, is_updatable, is_insertable_into
regression-# FROM information_schema.views
regression-# ;
regression=# SELECT table_name, is_updatable, is_insertable_into
regression-# FROM information_schema.views
regression-# WHERE table_name = 'rw_view1';
table_name | is_updatable | is_insertable_into
------------+--------------+--------------------
(0 rows)
regression=# explain SELECT table_name, is_updatable, is_insertable_into
FROM information_schema.views
WHERE table_name = 'rw_view1';
ERROR: no relation entry for relid 0
So, the query fails. After removing the WHERE clause it works, and this
somehow fixes the original query (with the WHERE clause). Nevertheless,
I still can't do explain on the query.
Clearly, I'm doing something wrong. I suspect it's caused either by
conditions involving function calls, or the fact that the view is a join
of multiple tables. But what?
For simple queries (single table, ...) it seems to be working fine.
regards
Tomas
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[text/x-diff] multivar-stats-v2.patch (161.6K, ../../5460244C.8080109@fuzzy.cz/2-multivar-stats-v2.patch)
download | inline diff:
diff --git a/src/backend/catalog/Makefile b/src/backend/catalog/Makefile
index b257b02..6e63afe 100644
--- a/src/backend/catalog/Makefile
+++ b/src/backend/catalog/Makefile
@@ -32,6 +32,7 @@ POSTGRES_BKI_SRCS = $(addprefix $(top_srcdir)/src/include/catalog/,\
pg_attrdef.h pg_constraint.h pg_inherits.h pg_index.h pg_operator.h \
pg_opfamily.h pg_opclass.h pg_am.h pg_amop.h pg_amproc.h \
pg_language.h pg_largeobject_metadata.h pg_largeobject.h pg_aggregate.h \
+ pg_mv_statistic.h \
pg_statistic.h pg_rewrite.h pg_trigger.h pg_event_trigger.h pg_description.h \
pg_cast.h pg_enum.h pg_namespace.h pg_conversion.h pg_depend.h \
pg_database.h pg_db_role_setting.h pg_tablespace.h pg_pltemplate.h \
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index a819952..bb82fe8 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -152,6 +152,18 @@ CREATE VIEW pg_indexes AS
LEFT JOIN pg_tablespace T ON (T.oid = I.reltablespace)
WHERE C.relkind IN ('r', 'm') AND I.relkind = 'i';
+CREATE VIEW pg_mv_stats AS
+ SELECT
+ N.nspname AS schemaname,
+ C.relname AS tablename,
+ S.stakeys AS attnums,
+ length(S.stamcv) AS mcvbytes,
+ pg_mv_stats_mvclist_info(S.stamcv) AS mcvinfo,
+ length(S.stahist) AS histbytes,
+ pg_mv_stats_histogram_info(S.stahist) AS histinfo
+ FROM (pg_mv_statistic S JOIN pg_class C ON (C.oid = S.starelid))
+ LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace);
+
CREATE VIEW pg_stats AS
SELECT
nspname AS schemaname,
diff --git a/src/backend/commands/analyze.c b/src/backend/commands/analyze.c
index 954e5a6..32e0d07 100644
--- a/src/backend/commands/analyze.c
+++ b/src/backend/commands/analyze.c
@@ -27,6 +27,7 @@
#include "catalog/indexing.h"
#include "catalog/pg_collation.h"
#include "catalog/pg_inherits_fn.h"
+#include "catalog/pg_mv_statistic.h"
#include "catalog/pg_namespace.h"
#include "commands/dbcommands.h"
#include "commands/tablecmds.h"
@@ -54,7 +55,11 @@
#include "utils/syscache.h"
#include "utils/timestamp.h"
#include "utils/tqual.h"
+#include "utils/fmgroids.h"
+#include "utils/builtins.h"
+#include "utils/mvstats.h"
+#include "access/sysattr.h"
/* Data structure for Algorithm S from Knuth 3.4.2 */
typedef struct
@@ -111,6 +116,62 @@ static Datum std_fetch_func(VacAttrStatsP stats, int rownum, bool *isNull);
static Datum ind_fetch_func(VacAttrStatsP stats, int rownum, bool *isNull);
+/* multivariate statistics (histogram, MCV list, associative rules) */
+
+static void build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
+ int natts, VacAttrStats **vacattrstats);
+static void update_mv_stats(Oid relid,
+ MVHistogram histogram, MCVList mcvlist);
+
+/* multivariate histograms */
+static MVHistogram build_mv_histogram(int numrows, HeapTuple *rows,
+ int2vector *attrs,
+ int attr_cnt, VacAttrStats **vacattrstats,
+ int numrows_total);
+static MVBucket create_initial_mv_bucket(int numrows, HeapTuple *rows,
+ int2vector *attrs, int natts,
+ VacAttrStats **vacattrstats);
+static MVBucket select_bucket_to_partition(int nbuckets, MVBucket * buckets);
+static MVBucket partition_bucket(MVBucket bucket, int2vector *attrs,
+ int natts, VacAttrStats **vacattrstats);
+static MVBucket copy_mv_bucket(MVBucket bucket, uint32 ndimensions);
+
+static void update_bucket_ndistinct(MVBucket bucket, int2vector *attrs,
+ VacAttrStats ** stats);
+static void update_dimension_ndistinct(MVBucket bucket, int dimension,
+ int2vector *attrs,
+ VacAttrStats ** stats,
+ bool update_boundaries);
+/* multivariate MCV list */
+static MCVList build_mv_mcvlist(int numrows, HeapTuple *rows,
+ int2vector *attrs,
+ int natts, VacAttrStats **vacattrstats,
+ int *numrows_filtered);
+
+/* multivariate associative rules */
+static void build_mv_associations(int numrows, HeapTuple *rows,
+ int2vector *attrs,
+ int natts, VacAttrStats **vacattrstats);
+
+/* serialization */
+static bytea * serialize_mv_histogram(MVHistogram histogram);
+static bytea * serialize_mv_mcvlist(MCVList mcvlist);
+
+/* comparators, used when constructing multivariate stats */
+static int compare_scalars_simple(const void *a, const void *b, void *arg);
+static int compare_scalars_partition(const void *a, const void *b, void *arg);
+static int compare_scalars_memcmp(const void *a, const void *b, void *arg);
+static int compare_scalars_memcmp_2(const void *a, const void *b);
+
+static VacAttrStats ** lookup_var_attr_stats(int2vector *attrs,
+ int natts, VacAttrStats **vacattrstats);
+
+/* some debugging methods */
+#ifdef MVSTATS_DEBUG
+static void print_mv_histogram_info(MVHistogram histogram);
+#endif
+
+
/*
* analyze_rel() -- analyze one relation
*/
@@ -472,6 +533,13 @@ do_analyze_rel(Relation onerel, VacuumStmt *vacstmt,
* all analyzable columns. We use a lower bound of 100 rows to avoid
* possible overflow in Vitter's algorithm. (Note: that will also be the
* target in the corner case where there are no analyzable columns.)
+ *
+ * FIXME This sample sizing is mostly OK when computing stats for
+ * individual columns, but when computing multi-variate stats
+ * for multivariate stats (histograms, mcv, ...) it's rather
+ * insufficient. For small number of dimensions it works, but
+ * for complex stats it'd be nice use sample proportional to
+ * the table (say, 0.5% - 1%) instead of a fixed size.
*/
targrows = 100;
for (i = 0; i < attr_cnt; i++)
@@ -574,6 +642,9 @@ do_analyze_rel(Relation onerel, VacuumStmt *vacstmt,
update_attstats(RelationGetRelid(Irel[ind]), false,
thisdata->attr_cnt, thisdata->vacattrstats);
}
+
+ /* Build multivariate stats (if there are any). */
+ build_mv_stats(onerel, numrows, rows, attr_cnt, vacattrstats);
}
/*
@@ -2815,3 +2886,1985 @@ compare_mcvs(const void *a, const void *b)
return da - db;
}
+
+/*
+ * Compute requested multivariate stats, using the rows sampled for the
+ * plain (single-column) stats.
+ *
+ * This fetches a list of stats from pg_mv_statistic, computes the stats
+ * and serializes them back into the catalog (as bytea values).
+ */
+static void
+build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
+ int natts, VacAttrStats **vacattrstats)
+{
+ int i;
+ MVStats mvstats;
+ int nmvstats;
+
+ /*
+ * Fetch defined MV groups from pg_mv_statistic, and then compute
+ * the MV statistics (histograms for now).
+ *
+ * TODO move this to a separate method or something ...
+ */
+ mvstats = list_mv_stats(RelationGetRelid(onerel), &nmvstats, false);
+
+ for (i = 0; i < nmvstats; i++)
+ {
+ MCVList mcvlist = NULL;
+ MVHistogram histogram = NULL;
+ int numrows_filtered = 0;
+
+ /* int2 vector of attnums the stats should be computed on */
+ int2vector * attrs = mvstats[i].stakeys;
+
+ /* check allowed number of dimensions */
+ Assert((attrs->dim1 >= 2) && (attrs->dim1 <= MVSTATS_MAX_DIMENSIONS));
+
+ /*
+ * Analyze associations between pairs of columns.
+ *
+ * FIXME store the identified associations back to pg_mv_statistic
+ */
+ build_mv_associations(numrows, rows, attrs, natts, vacattrstats);
+
+ /* build the MCV list */
+ mcvlist = build_mv_mcvlist(numrows, rows, attrs, natts, vacattrstats, &numrows_filtered);
+
+ /*
+ * Build a multivariate histogram on the columns.
+ *
+ * FIXME remove the rows used to build the MCV from the histogram.
+ * Another option might be subtracting the MCV selectivities
+ * from the histogram, but I'm not sure whether that works
+ * accurately (maybe it introduces additional errors).
+ */
+ if (numrows_filtered > 0)
+ histogram = build_mv_histogram(numrows_filtered, rows, attrs, natts, vacattrstats, numrows);
+
+ /* store the histogram / MCV list in the catalog */
+ update_mv_stats(mvstats[i].mvoid, histogram, mcvlist);
+
+#ifdef MVSTATS_DEBUG
+ print_mv_histogram_info(histogram);
+#endif
+
+ }
+}
+
+/*
+ * Lookup the VacAttrStats info for the selected columns, with indexes
+ * matching the attrs vector (to make it easy to work with when
+ * computing multivariate stats).
+ */
+static VacAttrStats **
+lookup_var_attr_stats(int2vector *attrs, int natts, VacAttrStats **vacattrstats)
+{
+ int i, j;
+ int numattrs = attrs->dim1;
+ VacAttrStats **stats = (VacAttrStats**)palloc0(numattrs * sizeof(VacAttrStats*));
+
+ /* lookup VacAttrStats info for the requested columns (same attnum) */
+ for (i = 0; i < numattrs; i++)
+ {
+ stats[i] = NULL;
+ for (j = 0; j < natts; j++)
+ {
+ if (attrs->values[i] == vacattrstats[j]->tupattnum)
+ {
+ stats[i] = vacattrstats[j];
+ break;
+ }
+ }
+
+ /*
+ * Check that we found the info, that the attnum matches and
+ * that there's the requested 'lt' operator and that the type
+ * is 'passed-by-value'.
+ */
+ Assert(stats[i] != NULL);
+ Assert(stats[i]->tupattnum == attrs->values[i]);
+
+ /* FIXME This is rather ugly way to check for 'ltopr' (which
+ * is defined for 'scalar' attributes).
+ */
+ Assert(stats[i]->compute_stats == compute_scalar_stats);
+
+ /* TODO remove the 'pass by value' requirement */
+ Assert(stats[i]->attrtype->typbyval);
+ }
+
+ return stats;
+}
+
+/*
+ * TODO Add ndistinct estimation, probably the one described in "Towards
+ * Estimation Error Guarantees for Distinct Values, PODS 2000,
+ * p. 268-279" (the ones called GEE, or maybe AE).
+ *
+ * TODO The "combined" ndistinct is more likely to scale with the number
+ * of rows (in the table), because a single column behaving this
+ * way is sufficient for such behavior.
+ */
+static MVBucket
+create_initial_mv_bucket(int numrows, HeapTuple *rows, int2vector *attrs,
+ int natts, VacAttrStats **vacattrstats)
+{
+ int i;
+ int numattrs = attrs->dim1;
+
+ /* info for the interesting attributes only */
+ VacAttrStats **stats = lookup_var_attr_stats(attrs, natts, vacattrstats);
+
+ /* resulting bucket */
+ MVBucket bucket = (MVBucket)palloc0(sizeof(MVBucketData));
+
+ Assert(numrows > 0);
+ Assert(rows != NULL);
+ Assert((numattrs >= 2) && (numattrs <= MVSTATS_MAX_DIMENSIONS));
+
+ /* allocate the per-dimension arrays */
+ bucket->ndistincts = (uint32*)palloc0(numattrs * sizeof(uint32));
+ bucket->nullsonly = (bool*)palloc0(numattrs * sizeof(bool));
+
+ /* inclusiveness boundaries - lower/upper bounds */
+ bucket->min_inclusive = (bool*)palloc0(numattrs * sizeof(bool));
+ bucket->max_inclusive = (bool*)palloc0(numattrs * sizeof(bool));
+
+ /* lower/upper boundaries */
+ bucket->min = (Datum*)palloc0(numattrs * sizeof(Datum));
+ bucket->max = (Datum*)palloc0(numattrs * sizeof(Datum));
+
+ /*
+ * All the sample rows fall into the initial bucket.
+ *
+ * FIXME This is wrong (unless all columns are NOT NULL), because we
+ * skipped the NULL values.
+ */
+ bucket->numrows = numrows;
+ bucket->ntuples = numrows;
+ bucket->rows = rows;
+
+ /*
+ * Update the number of ndistinct combinations in the bucket (which
+ * we use when selecting bucket to partition), and then number of
+ * distinct values for each partition (which we use when choosing
+ * which dimension to split).
+ */
+ update_bucket_ndistinct(bucket, attrs, stats);
+
+ for (i = 0; i < numattrs; i++)
+ update_dimension_ndistinct(bucket, i, attrs, stats, true);
+
+ /*
+ * The initial bucket was not split at all, so we'll start with the
+ * first dimension in the next round (index = 0).
+ */
+ bucket->last_split_dimension = -1;
+
+ return bucket;
+}
+
+/*
+ * TODO Fix to handle arbitrarily-sized histograms (not just 2D ones)
+ * and call the right output procedures (for the particular type).
+ *
+ * TODO This should somehow fetch info about the data types, and use
+ * the appropriate output functions to print the boundary values.
+ * Right now this prints the 8B value as an integer.
+ *
+ * TODO Also, provide a special function for 2D histogram, printing
+ * a gnuplot script (with rectangles).
+ *
+ * TODO For string types (once supported) we can sort the strings first,
+ * assign them a sequence of integers and use the original values
+ * as labels.
+ */
+#ifdef MVSTATS_DEBUG
+static void
+print_mv_histogram_info(MVHistogram histogram)
+{
+ int i = 0;
+
+ elog(WARNING, "histogram nbuckets=%d", histogram->nbuckets);
+
+ for (i = 0; i < histogram->nbuckets; i++)
+ {
+ MVBucket bucket = histogram->buckets[i];
+ elog(WARNING, " bucket %d : ndistinct=%f ntuples=%d min=[%ld, %ld], max=[%ld, %ld] distinct=[%d,%d]",
+ i, bucket->ndistinct, bucket->numrows,
+ bucket->min[0], bucket->min[1], bucket->max[0], bucket->max[1],
+ bucket->ndistincts[0], bucket->ndistincts[1]);
+ }
+}
+#endif
+
+/*
+ * A very simple partitioning selection criteria - choose the bucket
+ * with the highest number of distinct values.
+ *
+ * Returns either pointer to the bucket selected to be partitioned,
+ * or NULL if there are no buckets that may be split (i.e. all buckets
+ * contain a single distinct value).
+ *
+ * TODO Consider other partitioning criteria (v-optimal, maxdiff etc.).
+ *
+ * TODO Allowing the bucket to degenerate to a single combination of
+ * values makes it rather strange MCV list. Maybe we should use
+ * higher lower boundary, or maybe make the selection criteria
+ * more complex (e.g. consider number of rows in the bucket, etc.).
+ *
+ * That however is different from buckets 'degenerated' only for
+ * some dimensions (e.g. half of them), which is perfectly
+ * appropriate for statistics on a combination of low and high
+ * cardinality columns.
+ */
+static MVBucket
+select_bucket_to_partition(int nbuckets, MVBucket * buckets)
+{
+ int i;
+ int ndistinct = 1; /* if ndistinct=1, we can't split the bucket */
+ MVBucket bucket = NULL;
+
+ for (i = 0; i < nbuckets; i++)
+ {
+ /* if the ndistinct count is higher, use this bucket */
+ if (buckets[i]->ndistinct > ndistinct) {
+ bucket = buckets[i];
+ ndistinct = buckets[i]->ndistinct;
+ }
+ }
+
+ /* may be NULL if there are not buckets with (ndistinct>1) */
+ return bucket;
+}
+
+/*
+ * A simple bucket partitioning implementation - splits the dimensions in
+ * a round-robin manner (considering only those with ndistinct>1). That
+ * is first a dimension 0 is split, then 1, 2, ... until reaching the
+ * end of attribute list, and then wrapping back to 0. Of course,
+ * dimensions with a single distinct value are skipped.
+ *
+ * This is essentially what Muralikrishna/DeWitt described in their SIGMOD
+ * article (M. Muralikrishna, David J. DeWitt: Equi-Depth Histograms For
+ * Estimating Selectivity Factors For Multi-Dimensional Queries. SIGMOD
+ * Conference 1988: 28-36).
+ *
+ * There are multiple histogram options, centered around the partitioning
+ * criteria, specifying both how to choose a bucket and the dimension
+ * most in need of a split. For a nice summary and general overview, see
+ * "rK-Hist : an R-Tree based histogram for multi-dimensional selectivity
+ * estimation" thesis by J. A. Lopez, Concordia University, p.34-37 (and
+ * possibly p. 32-34 for explanation of the terms).
+ *
+ * This splits the bucket by tweaking the existing one, and returning the
+ * new bucket (essentially shrinking the existing one in-place and returning
+ * the other "half" as a new bucket). The caller is responsible for adding
+ * the new bucket into the list of buckets.
+ *
+ * TODO It requires care to prevent splitting only one dimension and not
+ * splitting another one at all (which might happen easily in case of
+ * strongly dependent columns - e.g. y=x).
+ *
+ * TODO Should probably consider statistics target for the columns (e.g. to
+ * split dimensions with higher statistics target more frequently).
+ */
+static MVBucket
+partition_bucket(MVBucket bucket, int2vector *attrs,
+ int natts, VacAttrStats **vacattrstats)
+{
+ int i;
+ int dimension;
+ int numattrs = attrs->dim1;
+
+ Datum split_value;
+ MVBucket new_bucket;
+
+ /* needed for sort, when looking for the split value */
+ bool isNull;
+ int nvalues = 0;
+ StdAnalyzeData * mystats = NULL;
+ ScalarItem * values = (ScalarItem*)palloc0(bucket->numrows * sizeof(ScalarItem));
+ SortSupportData ssup;
+
+ /* looking for the split value */
+ int ndistinct = 1; /* number of distinct values below current value */
+ int nrows = 1; /* number of rows below current value */
+
+ /* needed when splitting the values */
+ HeapTuple * oldrows = bucket->rows;
+ int oldnrows = bucket->numrows;
+
+ /* info for the interesting attributes only */
+ VacAttrStats **stats = lookup_var_attr_stats(attrs, natts, vacattrstats);
+
+ /*
+ * We can't split buckets with a single distinct value (this also
+ * disqualifies NULL-only dimensions). Also, there has to be multiple
+ * sample rows (otherwise, how could there be more distinct values).
+ */
+ Assert(bucket->ndistinct > 1);
+ Assert(bucket->numrows > 1);
+ Assert((numattrs >= 2) && (numattrs <= MVSTATS_MAX_DIMENSIONS));
+
+ /*
+ * Look for the next dimension to split, in a round robin manner.
+ * We'll use the first one with (ndistinct > 1).
+ *
+ * If we happen to wrap around, something clearly went wrong (we
+ * can't mess with the last_split_dimension directly, because we
+ * couldn't do this check).
+ */
+ dimension = bucket->last_split_dimension;
+ while (true)
+ {
+ dimension = (dimension + 1) % numattrs;
+
+ if (bucket->ndistincts[dimension] > 1)
+ break;
+
+ /* if we ran the previous split dimension, it's infinite loop */
+ Assert(dimension != bucket->last_split_dimension);
+ }
+
+ /* Remember the dimension for the next split of this bucket. */
+ bucket->last_split_dimension = dimension;
+
+ /*
+ * Walk through the selected dimension, collect and sort the values
+ * and then choose the value to use as the new boundary.
+ */
+ mystats = (StdAnalyzeData *) stats[dimension]->extra_data;
+
+ /* initialize sort support, etc. */
+ memset(&ssup, 0, sizeof(ssup));
+ ssup.ssup_cxt = CurrentMemoryContext;
+
+ /* We always use the default collation for statistics */
+ ssup.ssup_collation = DEFAULT_COLLATION_OID;
+ ssup.ssup_nulls_first = false;
+
+ PrepareSortSupportFromOrderingOp(mystats->ltopr, &ssup);
+
+ for (i = 0; i < bucket->numrows; i++)
+ {
+ /* remember the index of the sample row, to make the partitioning simpler */
+ values[nvalues].value = heap_getattr(bucket->rows[i], attrs->values[dimension],
+ stats[dimension]->tupDesc, &isNull);
+ values[nvalues].tupno = i;
+
+ /* no NULL values allowed here (we don't do splits by null-only dimensions) */
+ Assert(!isNull);
+
+ nvalues++;
+ }
+
+ /* sort the array (pass-by-value datum */
+ qsort_arg((void *) values, nvalues, sizeof(ScalarItem),
+ compare_scalars_partition, (void *) &ssup);
+
+ /*
+ * We know there are bucket->ndistincts[dimension] distinct values
+ * in this dimension, and we want to split this into half, so walk
+ * through the array and stop once we see (ndistinct/2) values.
+ *
+ * We always choose the "next" value, i.e. (n/2+1)-th distinct value,
+ * and use it as an exclusive upper boundary (and inclusive lower
+ * boundary).
+ *
+ * TODO Maybe we should use "average" of the two middle distinct
+ * values (at least for even distinct counts), but that would
+ * require being able to do an average (which does not work
+ * for non-arithmetic types).
+ *
+ * TODO Another option is to look for a split that'd give about
+ * 50% tuples (not distinct values) in each partition. That
+ * might work better when there are a few very frequent
+ * values, and many rare ones.
+ */
+ split_value = values[0].value;
+ for (i = 1; i < bucket->numrows; i++)
+ {
+ /* count distinct values */
+ if (values[i].value != values[i-1].value)
+ ndistinct += 1;
+
+ /* once we've seen 1/2 distinct values (and use the value) */
+ if (ndistinct > bucket->ndistincts[dimension] / 2)
+ {
+ split_value = values[i].value;
+ break;
+ }
+
+ /* keep track how many rows belong to the first bucket */
+ nrows += 1;
+ }
+
+ Assert(nrows > 0);
+ Assert(nrows < bucket->numrows);
+
+ /* create the new bucket as a (incomplete) copy of the one being partitioned. */
+ new_bucket = copy_mv_bucket(bucket, numattrs);
+
+ /*
+ * Do the actual split of the chosen dimension, using the split value as the
+ * upper bound for the existing bucket, and lower bound for the new one.
+ */
+ bucket->max[dimension] = split_value;
+ new_bucket->min[dimension] = split_value;
+
+ bucket->max_inclusive[dimension] = false;
+ new_bucket->max_inclusive[dimension] = true;
+
+ /*
+ * Redistribute the sample tuples using the 'ScalarItem->tupno'
+ * index. We know 'nrows' rows should remain in the original
+ * bucket and the rest goes to the new one.
+ */
+
+ bucket->rows = (HeapTuple*)palloc0(nrows * sizeof(HeapTuple));
+ new_bucket->rows = (HeapTuple*)palloc0((oldnrows - nrows) * sizeof(HeapTuple));
+
+ bucket->numrows = nrows;
+ new_bucket->numrows = (oldnrows - nrows);
+
+ /*
+ * The first nrows should go to the first bucket, the rest should
+ * go to the new one. Use the tupno field to get the actual HeapTuple
+ * row from the original array of sample rows.
+ */
+ for (i = 0; i < nrows; i++)
+ memcpy(&bucket->rows[i], &oldrows[values[i].tupno], sizeof(HeapTuple));
+
+ for (i = nrows; i < oldnrows; i++)
+ memcpy(&new_bucket->rows[i-nrows], &oldrows[values[i].tupno], sizeof(HeapTuple));
+
+ /* update ndistinct values for the buckets (total and per dimension) */
+ update_bucket_ndistinct(bucket, attrs, stats);
+ update_bucket_ndistinct(new_bucket, attrs, stats);
+
+ /*
+ * TODO We don't need to do this for the dimension we used for split,
+ * because we know how many distinct values went to each partition.
+ */
+ for (i = 0; i < numattrs; i++)
+ {
+ update_dimension_ndistinct(bucket, i, attrs, stats, false);
+ update_dimension_ndistinct(new_bucket, i, attrs, stats, false);
+ }
+
+ pfree(oldrows);
+ pfree(values);
+
+ return new_bucket;
+}
+
+/*
+ * Copy a histogram bucket. The copy does not include the build-time
+ * data, i.e. sampled rows etc.
+ */
+static MVBucket
+copy_mv_bucket(MVBucket bucket, uint32 ndimensions)
+{
+ MVBucket new_bucket = (MVBucket)palloc0(sizeof(MVBucketData));
+
+ /* Copy only the attributes that will stay the same after the split, and
+ * we'll recompute the rest after the split. */
+
+ new_bucket->last_split_dimension = bucket->last_split_dimension;
+
+ /* allocate the per-dimension arrays */
+ new_bucket->ndistincts = (uint32*)palloc0(ndimensions * sizeof(uint32));
+ new_bucket->nullsonly = (bool*)palloc0(ndimensions * sizeof(bool));
+
+ /* inclusiveness boundaries - lower/upper bounds */
+ new_bucket->min_inclusive = (bool*)palloc0(ndimensions * sizeof(bool));
+ new_bucket->max_inclusive = (bool*)palloc0(ndimensions * sizeof(bool));
+
+ /* lower/upper boundaries */
+ new_bucket->min = (Datum*)palloc0(ndimensions * sizeof(Datum));
+ new_bucket->max = (Datum*)palloc0(ndimensions * sizeof(Datum));
+
+ /* copy data */
+ memcpy(new_bucket->nullsonly, bucket->nullsonly, ndimensions * sizeof(bool));
+
+ memcpy(new_bucket->min_inclusive, bucket->min_inclusive, ndimensions*sizeof(bool));
+ memcpy(new_bucket->min, bucket->min, ndimensions*sizeof(Datum));
+
+ memcpy(new_bucket->max_inclusive, bucket->max_inclusive, ndimensions*sizeof(bool));
+ memcpy(new_bucket->max, bucket->max, ndimensions*sizeof(Datum));
+
+ return new_bucket;
+}
+
+/*
+ * Counts the number of distinct values in the bucket. This just copies
+ * the Datum values into a simple array, and sorts them using memcmp-based
+ * comparator. That means it only works for pass-by-value data types
+ * (assuming they don't use collations etc.)
+ *
+ * FIXME Make this work with all types (not just pass-by-value ones).
+ *
+ * TODO This might evaluate and store the distinct counts for all
+ * possible attribute combinations. The assumption is this might be
+ * useful for estimating things like GROUP BY cardinalities (e.g.
+ * in cases when some buckets contain a lot of low-frequency
+ * combinations, and other buckets contain few high-frequency ones).
+ *
+ * But it's unclear whether it's worth the price. Computing this
+ * is actually quite cheap, because it may be evaluated at the very
+ * end, when the buckets are rather small (so sorting it in 2^N ways
+ * is not a big deal). Assuming the partitioning algorithm does not
+ * use these values to do the decisions, of course (the current
+ * algorithm does not).
+ *
+ * The overhead with storing, fetching and parsing the data is more
+ * concerning - adding 2^N values per bucket (even if it's just
+ * a 1B or 2B value) would significantly bloat the histogram, and
+ * thus the impact on optimizer. Which is not really desirable.
+ *
+ * TODO This only updates the ndistinct for the sample (or bucket), but
+ * we eventually need an estimate of the total number of distinct
+ * values in the dataset. It's possible to either use the current
+ * 1D approach (i.e., if it's more than 10% of the sample, assume
+ * it's proportional to the number of rows). Or it's possible to
+ * implement the estimator suggested in the article, supposedly
+ * giving 'optimal' estimates (w.r.t. probability of error).
+ */
+static void
+update_bucket_ndistinct(MVBucket bucket, int2vector *attrs, VacAttrStats ** stats)
+{
+ int i, j, idx = 0;
+ int numattrs = attrs->dim1;
+ Size len = sizeof(Datum) * numattrs;
+ bool isNull;
+
+ /*
+ * We could collect this while walking through all the attributes
+ * above (this way we have to call heap_getattr twice).
+ */
+ Datum * values = palloc0(bucket->numrows * numattrs * sizeof(Datum));
+
+ for (j = 0; j < bucket->numrows; j++)
+ for (i = 0; i < numattrs; i++)
+ values[idx++] = heap_getattr(bucket->rows[j], attrs->values[i],
+ stats[i]->tupDesc, &isNull);
+
+ qsort_arg((void *) values, bucket->numrows, sizeof(Datum) * numattrs,
+ compare_scalars_memcmp, &len);
+
+ bucket->ndistinct = 1;
+
+ for (i = 1; i < bucket->numrows; i++)
+ if (memcmp(&values[i * numattrs], &values[(i-1) * numattrs], len) != 0)
+ bucket->ndistinct += 1;
+
+ pfree(values);
+
+}
+
+/*
+ * Count distinct values per bucket dimension.
+ *
+ * TODO Remove unnecessary parameters - don't pass in the whole arrays,
+ * just the proper elements.
+ */
+static void
+update_dimension_ndistinct(MVBucket bucket, int dimension, int2vector *attrs,
+ VacAttrStats ** stats, bool update_boundaries)
+{
+ int j;
+ int nvalues = 0;
+ bool isNull;
+ Datum * values = (Datum*)palloc0(bucket->numrows * sizeof(Datum));
+ SortSupportData ssup;
+
+ StdAnalyzeData * mystats = (StdAnalyzeData *) stats[dimension]->extra_data;
+
+ memset(&ssup, 0, sizeof(ssup));
+ ssup.ssup_cxt = CurrentMemoryContext;
+
+ /* We always use the default collation for statistics */
+ ssup.ssup_collation = DEFAULT_COLLATION_OID;
+ ssup.ssup_nulls_first = false;
+
+ PrepareSortSupportFromOrderingOp(mystats->ltopr, &ssup);
+
+ for (j = 0; j < bucket->numrows; j++)
+ {
+ values[nvalues] = heap_getattr(bucket->rows[j], attrs->values[dimension],
+ stats[dimension]->tupDesc, &isNull);
+
+ /* ignore NULL values */
+ if (! isNull)
+ nvalues++;
+ }
+
+ /* there's always at least 1 distinct value (may be NULL) */
+ bucket->ndistincts[dimension] = 1;
+
+ /* if there are only NULL values in the column, mark it so and continue
+ * with the next one */
+ if (nvalues == 0)
+ {
+ pfree(values);
+ bucket->nullsonly[dimension] = true;
+ return;
+ }
+
+ /* sort the array (pass-by-value datum */
+ qsort_arg((void *) values, nvalues, sizeof(Datum),
+ compare_scalars_simple, (void *) &ssup);
+
+ /*
+ * Update min/max boundaries to the smallest bounding box. Generally, this
+ * needs to be done only when constructing the initial bucket.
+ */
+ if (update_boundaries)
+ {
+ /* store the min/max values */
+ bucket->min[dimension] = values[0];
+ bucket->min_inclusive[dimension] = true;
+
+ bucket->max[dimension] = values[nvalues-1];
+ bucket->max_inclusive[dimension] = true;
+ }
+
+ /*
+ * Walk through the array and count distinct values by comparing
+ * succeeding values.
+ *
+ * FIXME This only works for pass-by-value types (i.e. not VARCHARs etc.).
+ */
+ for (j = 1; j < nvalues; j++) {
+ if (values[j] != values[j-1])
+ bucket->ndistincts[dimension] += 1;
+ }
+
+ pfree(values);
+}
+
+/*
+ * Fetch list of MV stats defined on a table, without the actual data
+ * for histograms, MCV lists etc.
+ */
+MVStats
+list_mv_stats(Oid relid, int *nstats, bool built_only)
+{
+ Relation indrel;
+ SysScanDesc indscan;
+ ScanKeyData skey;
+ HeapTuple htup;
+ MVStats result;
+
+ /* start with 16 items, that should be enough for most cases */
+ int maxitems = 16;
+ result = (MVStats)palloc0(sizeof(MVStatsData) * maxitems);
+ *nstats = 0;
+
+ /* Prepare to scan pg_mv_statistic for entries having indrelid = this rel. */
+ ScanKeyInit(&skey,
+ Anum_pg_mv_statistic_starelid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(relid));
+
+ indrel = heap_open(MvStatisticRelationId, AccessShareLock);
+ indscan = systable_beginscan(indrel, MvStatisticRelidIndexId, true,
+ NULL, 1, &skey);
+
+ while (HeapTupleIsValid(htup = systable_getnext(indscan)))
+ {
+ Form_pg_mv_statistic stats = (Form_pg_mv_statistic) GETSTRUCT(htup);
+
+ /*
+ * Skip statistics that were not computed yet (if only stats
+ * that were already built were requested)
+ */
+ if (built_only && (! (stats->hist_built || stats->mcv_built || stats->assoc_built)))
+ continue;
+
+ /* double the array size if needed */
+ if (*nstats == maxitems)
+ {
+ maxitems *= 2;
+ result = (MVStats)repalloc(result, sizeof(MVStatsData) * maxitems);
+ }
+
+ result[*nstats].mvoid = HeapTupleGetOid(htup);
+ result[*nstats].stakeys = buildint2vector(stats->stakeys.values, stats->stakeys.dim1);
+ result[*nstats].hist_built = stats->hist_built;
+ result[*nstats].mcv_built = stats->mcv_built;
+ result[*nstats].assoc_built = stats->assoc_built;
+ *nstats += 1;
+ }
+
+ systable_endscan(indscan);
+
+ heap_close(indrel, AccessShareLock);
+
+ /* TODO maybe save the list into relcache, as in RelationGetIndexList
+ * (which was used as an inspiration of this one)?. */
+
+ return result;
+}
+
+
+/*
+ * Serialize the MV histogram into a bytea value.
+ *
+ * The serialized first deduplicates the boundary values into a separate
+ * array, and uses 2B indexes when serializing the buckets. This stores
+ * a significant amount of space because each bucket split adds a single
+ * new boundary value, so e.g. with 4 attributes and 8191 splits (thus
+ * 8192 buckets), there are only ~8200 distinct boundary values.
+ *
+ * But as each bucket has 8 boundary values (4+4), that's ~64k Datums.
+ * That's roughly 65kB vs. 512kB, but we haven't included the indexes
+ * used to reference the boundary values. By using int16 indexes (which
+ * should be more than enough for all reasonable histogram sizes),
+ * this amounts to ~128kB (8192*8*2). So in total it's ~196kB vs. 512kB,
+ * i.e. more than 2x compression, which is nice.
+ *
+ * The implementation is simple - walk through the buckets, collect all
+ * the boundary values, keep only distinct values (in a sorted array)
+ * and then replace the values with indexes (using binary search).
+ *
+ * It's possible to either serialize/deserialize the histogram into
+ * a MVHistogram, or create a special structure working with this
+ * compressed structure (and keep MVBucket/MVHistogram only for the
+ * building phase). This might actually work better thanks to better
+ * CPU cache hit ratio, and simpler deserialization.
+ *
+ * This encoding will probably prevent automatic varlena compression,
+ * because first part of the serialized bytea will be an array of unique
+ * values (although sorted), and pglz decides whether to compress by
+ * trying to compress the first part (~1kB or so). Which will be poor,
+ * due to the lack of repetition.
+ *
+ * But in this case this is probably desirable - the data in general
+ * won't be really compressible (in addition to the 2x compression we
+ * got thanks to the encoding). In a sense the encoding scheme is
+ * actually a context-aware compression (usually compressing to ~30%).
+ * So this seems appropriate in this case.
+ *
+ * FIXME Make this work with arbitrary types.
+ *
+ * TODO Try to keep the compressed form, instead of deserializing it to
+ * MVHistogram/MVBucket.
+ *
+ * TODO We might get a bit better compression by considering the actual
+ * data type length. The current implementation treats all data as
+ * 8B values, but for INT it's actually 4B etc. OTOH this is only
+ * related to the lookup table, and most of the space is occupied
+ * by the buckets (with int16 indexes). And we don't have type info
+ * at the moment, so it would be difficult (but we'll nedd it to
+ * support all types, so maybe then).
+ */
+static bytea *
+serialize_mv_histogram(MVHistogram histogram)
+{
+ int i = 0, j = 0;
+
+ /* total size (histogram header + all buckets) */
+ Size total_len;
+ char *tmp = NULL;
+ bytea *result = NULL;
+
+ /* we need to accumulate all boundary values (min/max) */
+ int idx = 0;
+ int max_values = histogram->nbuckets * histogram->ndimensions * 2;
+ Datum * values = (Datum*)palloc0(max_values * sizeof(Datum));
+ Size len = sizeof(Datum);
+
+ /* we'll collect unique boundary values into this */
+ int ndistinct = 0;
+ Datum *lookup = NULL;
+ uint16 *indexes = (uint16*)palloc0(sizeof(uint16) * histogram->ndimensions);
+
+ /*
+ * Collect the boundary values first, sort them and generate a small
+ * array with only distinct values.
+ */
+ for (i = 0; i < histogram->nbuckets; i++)
+ {
+ for (j = 0; j < histogram->ndimensions; j++)
+ {
+ values[idx++] = histogram->buckets[i]->min[j];
+ values[idx++] = histogram->buckets[i]->max[j];
+ }
+ }
+
+ /*
+ * We've allocated just enough space for all boundary values, but
+ * this may change once we start handling NULL values (as we'll
+ * probably skip those).
+ *
+ * Also, we expect at least one boundary value at this moment.
+ */
+ Assert(max_values == idx);
+ Assert(idx > 1);
+
+ /*
+ * Sort the collected boundary values using a simple memcmp-based
+ * comparator (this won't work for pass-by-reference types), and
+ * then walk the data and count the distinct values.
+ */
+ qsort((void *) values, idx, len, compare_scalars_memcmp_2);
+
+ ndistinct = 1;
+ for (i = 1; i < max_values; i++)
+ ndistinct += (values[i-1] != values[i]) ? 1 : 0;
+
+ /*
+ * At this moment we can allocate the bytea value (and we'll collect
+ * the boundary values directly into it).
+ *
+ * The bytea will be structured like this:
+ *
+ * - varlena header : VARHDRSZ
+ * - histogram header : offsetof(MVHistogram,buckets)
+ * - number of boundary values : sizeof(uint32)
+ * - boundary values : ndistinct * sizeof(Datum)
+ * - buckets : nbuckets * BUCKET_SIZE_SERIALIZED
+ *
+ * We'll assume 2B indexes into the boundary values, because each
+ * bucket 'split' introduces one boundary value. Moreover, multiple
+ * splits may introduce the same value, so this should be enough for
+ * at least 65k buckets (and likely more). That's more than enough
+ * for reasonable histogram sizes.
+ */
+
+ Assert(ndistinct <= 65536);
+
+ total_len = VARHDRSZ + offsetof(MVHistogramData, buckets) +
+ (sizeof(uint32) + ndistinct * sizeof(Datum)) +
+ histogram->nbuckets * BUCKET_SIZE_SERIALIZED(histogram->ndimensions);
+
+ result = (bytea*)palloc0(total_len);
+ tmp = VARDATA(result);
+
+ SET_VARSIZE(result, total_len);
+
+ /* copy the global histogram header */
+ memcpy(tmp, histogram, offsetof(MVHistogramData, buckets));
+ tmp += offsetof(MVHistogramData, buckets);
+
+ /*
+ * Copy the number of distinct values, and then all the distinct
+ * values currently stored in the 'values' array (sorted).
+ */
+ memcpy(tmp, &ndistinct, sizeof(uint32));
+ tmp += sizeof(uint32);
+
+ lookup = (Datum*)tmp;
+
+ for (i = 0; i < max_values; i++)
+ {
+ /* skip values that are equal to the previous one */
+ if ((i > 0) && (values[i-1] == values[i]))
+ continue;
+
+ memcpy(tmp, &values[i], sizeof(Datum));
+ tmp += sizeof(Datum);
+ }
+
+ Assert(tmp - (char*)lookup == ndistinct * sizeof(Datum));
+
+ /* now serialize all the buckets - first the header, without the
+ * variable-length part, then all the variable length parts */
+ for (i = 0; i < histogram->nbuckets; i++)
+ {
+ MVBucket bucket = histogram->buckets[i];
+
+ /* write the common bucket header */
+ memcpy(tmp, bucket, offsetof(MVBucketData, ndistincts));
+ tmp += offsetof(MVBucketData, ndistincts);
+
+ /* per-dimension ndistincts / nullsonly */
+ memcpy(tmp, bucket->ndistincts, sizeof(uint32)*histogram->ndimensions);
+ tmp += sizeof(uint32)*histogram->ndimensions;
+
+ memcpy(tmp, bucket->nullsonly, sizeof(bool)*histogram->ndimensions);
+ tmp += sizeof(bool)*histogram->ndimensions;
+
+ memcpy(tmp, bucket->min_inclusive, sizeof(bool)*histogram->ndimensions);
+ tmp += sizeof(bool)*histogram->ndimensions;
+
+ memcpy(tmp, bucket->max_inclusive, sizeof(bool)*histogram->ndimensions);
+ tmp += sizeof(bool)*histogram->ndimensions;
+
+ /* and now translate the min (and then max) boundaries to indexes */
+ for (j = 0; j < histogram->ndimensions; j++)
+ {
+ Datum *v = (Datum*)bsearch(&bucket->min[j], lookup, ndistinct,
+ sizeof(Datum), compare_scalars_memcmp_2);
+
+ Assert(v != NULL);
+ indexes[j] = (v - lookup); /* Datum arithmetics (not char) */
+ Assert(indexes[j] < ndistinct); /* we have to be within the array */
+ }
+
+ memcpy(tmp, indexes, sizeof(uint16)*histogram->ndimensions);
+ tmp += sizeof(uint16)*histogram->ndimensions;
+
+ for (j = 0; j < histogram->ndimensions; j++)
+ {
+ Datum *v = (Datum*)bsearch(&bucket->max[j], lookup, ndistinct,
+ sizeof(Datum), compare_scalars_memcmp_2);
+ Assert(v != NULL);
+ indexes[j] = (v - lookup); /* Datum arithmetics (not char) */
+ Assert(indexes[j] < ndistinct); /* we have to be within the array */
+ }
+
+ memcpy(tmp, indexes, sizeof(uint16)*histogram->ndimensions);
+ tmp += sizeof(uint16)*histogram->ndimensions;
+ }
+
+ pfree(indexes);
+
+ return result;
+}
+
+/*
+ * Reverse to serialize histogram. This essentially expands the serialized
+ * form back to MVHistogram / MVBucket.
+ */
+MVHistogram
+deserialize_mv_histogram(bytea * data)
+{
+ int i = 0, j = 0;
+
+ Size expected_length;
+ char *tmp = NULL;
+ MVHistogram histogram;
+
+ uint32 nlookup; /* Datum lookup table */
+ Datum *lookup = NULL;
+
+ if (data == NULL)
+ return NULL;
+
+ /* get pointer to the data part of the varlena */
+ tmp = VARDATA(data);
+
+ histogram = (MVHistogram)palloc0(sizeof(MVHistogramData));
+
+ /* copy the histogram header in place */
+ memcpy(histogram, tmp, offsetof(MVHistogramData, buckets));
+ tmp += offsetof(MVHistogramData, buckets);
+
+ if (histogram->magic != MVHIST_MAGIC)
+ {
+ pfree(histogram);
+ elog(WARNING, "not a MV Histogram (magic number mismatch)");
+ return NULL;
+ }
+
+ Assert(histogram->type == MVHIST_TYPE_BASIC);
+ Assert(histogram->nbuckets > 0);
+ Assert(histogram->nbuckets <= MVHIST_MAX_BUCKETS);
+ Assert(histogram->ndimensions > 0);
+ Assert(histogram->ndimensions <= MVSTATS_MAX_DIMENSIONS);
+
+ /* now, get the size of the lookup table */
+ memcpy(&nlookup, tmp, sizeof(uint32));
+ tmp += sizeof(uint32);
+ lookup = (Datum*)tmp;
+
+ /* skip to the first bucket */
+ tmp += sizeof(Datum) * nlookup;
+
+ /* check the total serialized length */
+ expected_length = offsetof(MVHistogramData, buckets) +
+ sizeof(uint32) + nlookup * sizeof(Datum) +
+ histogram->nbuckets * BUCKET_SIZE_SERIALIZED(histogram->ndimensions);
+
+ /* check serialized length */
+ if (VARSIZE_ANY_EXHDR(data) != expected_length)
+ {
+ elog(ERROR, "invalid MV histogram serialized size (expected %ld, got %ld)",
+ VARSIZE_ANY_EXHDR(data), expected_length);
+ return NULL;
+ }
+
+ /* allocate bucket pointers */
+ histogram->buckets = (MVBucket*)palloc0(histogram->nbuckets * sizeof(MVBucket));
+
+ /* deserialize the buckets, one by one */
+ for (i = 0; i < histogram->nbuckets; i++)
+ {
+ /* don't allocate space for the build-only fields */
+ MVBucket bucket = (MVBucket)palloc0(offsetof(MVBucketData, rows));
+ uint16 *indexes = NULL;
+
+ /* write the common bucket header */
+ memcpy(bucket, tmp, offsetof(MVBucketData, ndistincts));
+ tmp += offsetof(MVBucketData, ndistincts);
+
+ /* per-dimension ndistincts / nullsonly */
+ bucket->ndistincts = (uint32*)palloc0(sizeof(uint32)*histogram->ndimensions);
+ memcpy(bucket->ndistincts, tmp, sizeof(uint32)*histogram->ndimensions);
+ tmp += sizeof(uint32)*histogram->ndimensions;
+
+ bucket->nullsonly = (bool*)palloc0(sizeof(bool)*histogram->ndimensions);
+ memcpy(bucket->nullsonly, tmp, sizeof(bool)*histogram->ndimensions);
+ tmp += sizeof(bool)*histogram->ndimensions;
+
+ bucket->min_inclusive = (bool*)palloc0(sizeof(bool)*histogram->ndimensions);
+ memcpy(bucket->min_inclusive, tmp, sizeof(bool)*histogram->ndimensions);
+ tmp += sizeof(bool)*histogram->ndimensions;
+
+ bucket->max_inclusive = (bool*)palloc0(sizeof(bool)*histogram->ndimensions);
+ memcpy(bucket->max_inclusive, tmp, sizeof(bool)*histogram->ndimensions);
+ tmp += sizeof(bool)*histogram->ndimensions;
+
+ /* translate the indexes back to Datum values */
+ bucket->min = (Datum*)palloc0(sizeof(Datum)*histogram->ndimensions);
+ bucket->max = (Datum*)palloc0(sizeof(Datum)*histogram->ndimensions);
+
+ indexes = (uint16*)tmp;
+ tmp += sizeof(uint16) * histogram->ndimensions;
+ for (j = 0; j < histogram->ndimensions; j++)
+ memcpy(&bucket->min[j], &lookup[indexes[j]], sizeof(Datum));
+
+ indexes = (uint16*)tmp;
+ tmp += sizeof(uint16) * histogram->ndimensions;
+ for (j = 0; j < histogram->ndimensions; j++)
+ memcpy(&bucket->max[j], &lookup[indexes[j]], sizeof(Datum));
+
+ histogram->buckets[i] = bucket;
+ }
+
+ return histogram;
+}
+
+/*
+ * Serialize MCV list into a bytea value.
+ *
+ * This does not use any kind of deduplication (compared to histogram
+ * serialization), as we don't expect the same efficiency here.
+ *
+ * This simply writes a MCV header (number of items, ...) and then Datum
+ * values for all attribute of a item, followed by the item frequency
+ * (as a double).
+ */
+static bytea *
+serialize_mv_mcvlist(MCVList mcvlist)
+{
+ int i;
+
+ /* we need to store nitems, and each needs ndimension * Datum, plus a double */
+ Size len = VARHDRSZ + offsetof(MCVListData, items) + mcvlist->nitems * (sizeof(Datum) * mcvlist->ndimensions + sizeof(double));
+
+ bytea * output = (bytea*)palloc0(len);
+
+ char * tmp = VARDATA(output);
+
+ SET_VARSIZE(output, len);
+
+ /* first, store the number of dimensions / items */
+ memcpy(tmp, mcvlist, offsetof(MCVListData, items));
+ tmp += offsetof(MCVListData, items);
+
+ /* now, walk through the items and store values + frequency for each MCV item */
+ for (i = 0; i < mcvlist->nitems; i++)
+ {
+ memcpy(tmp, mcvlist->items[i]->values, mcvlist->ndimensions * sizeof(Datum));
+ tmp += mcvlist->ndimensions * sizeof(Datum);
+
+ memcpy(tmp, &mcvlist->items[i]->frequency, sizeof(double));
+ tmp += sizeof(double);
+ }
+
+ return output;
+
+}
+
+MCVList deserialize_mv_mcvlist(bytea * data)
+{
+ int i;
+ Size expected_size;
+ MCVList mcvlist;
+ char *tmp;
+
+ if (data == NULL)
+ return NULL;
+
+ if (VARSIZE_ANY_EXHDR(data) < offsetof(MCVListData,items))
+ elog(ERROR, "invalid MCV Size %ld (expected at least %ld)",
+ VARSIZE_ANY_EXHDR(data), offsetof(MCVListData,items));
+
+ /* read the MCV list header */
+ mcvlist = (MCVList)palloc0(sizeof(MCVListData));
+
+ /* initialize pointer to the data part (skip the varlena header) */
+ tmp = VARDATA(data);
+
+ /* get the header and perform basic sanity checks */
+ memcpy(mcvlist, tmp, offsetof(MCVListData,items));
+ tmp += offsetof(MCVListData,items);
+
+ if (mcvlist->magic != MVSTAT_MCV_MAGIC)
+ elog(ERROR, "invalid MCV magic %d (expected %dd)",
+ mcvlist->magic, MVSTAT_MCV_MAGIC);
+
+ if (mcvlist->type != MVSTAT_MCV_TYPE_BASIC)
+ elog(ERROR, "invalid MCV type %d (expected %dd)",
+ mcvlist->type, MVSTAT_MCV_TYPE_BASIC);
+
+ Assert(mcvlist->nitems > 0);
+ Assert((mcvlist->ndimensions >= 2) && (mcvlist->ndimensions <= MVSTATS_MAX_DIMENSIONS));
+
+ /* what bytea size do we expect for those parameters */
+ expected_size = offsetof(MCVListData,items) +
+ mcvlist->nitems * (sizeof(Datum) * mcvlist->ndimensions + sizeof(double));
+
+ if (VARSIZE_ANY_EXHDR(data) != expected_size)
+ elog(ERROR, "invalid MCV Size %ld (expected %ld)",
+ VARSIZE_ANY_EXHDR(data), expected_size);
+
+ /* allocate space for the MCV items */
+ mcvlist->items = (MCVItem*)palloc0(sizeof(MCVItem) * mcvlist->nitems);
+
+ for (i = 0; i < mcvlist->nitems; i++)
+ {
+ MCVItem item = (MCVItem)palloc0(offsetof(MCVItemData, values) +
+ mcvlist->ndimensions * sizeof(Datum));
+
+ memcpy(item->values, tmp, mcvlist->ndimensions * sizeof(Datum));
+ tmp += mcvlist->ndimensions * sizeof(Datum);
+
+ memcpy(&item->frequency, tmp, sizeof(double));
+ tmp += sizeof(double);
+
+ mcvlist->items[i] = item;
+ }
+
+ return mcvlist;
+}
+
+static void
+update_mv_stats(Oid mvoid, MVHistogram histogram, MCVList mcvlist)
+{
+ HeapTuple stup,
+ oldtup;
+ Datum values[Natts_pg_mv_statistic];
+ bool nulls[Natts_pg_mv_statistic];
+ bool replaces[Natts_pg_mv_statistic];
+
+ Relation sd = heap_open(MvStatisticRelationId, RowExclusiveLock);
+
+ memset(nulls, 1, Natts_pg_mv_statistic * sizeof(bool));
+ memset(replaces, 0, Natts_pg_mv_statistic * sizeof(bool));
+ memset(values, 0, Natts_pg_mv_statistic * sizeof(Datum));
+
+ /*
+ * Construct a new pg_mv_statistic tuple - replace only the histogram
+ * and MCV list, depending whether it actually was computed.
+ */
+ if (histogram != NULL)
+ {
+ nulls[Anum_pg_mv_statistic_stahist-1] = false;
+ values[Anum_pg_mv_statistic_stahist - 1]
+ = PointerGetDatum(serialize_mv_histogram(histogram));
+ }
+
+ if (mcvlist != NULL)
+ {
+ nulls[Anum_pg_mv_statistic_stamcv -1] = false;
+ values[Anum_pg_mv_statistic_stamcv - 1]
+ = PointerGetDatum(serialize_mv_mcvlist(mcvlist));
+ }
+
+ /* always replace the value (either by bytea or NULL) */
+ replaces[Anum_pg_mv_statistic_stahist-1] = true;
+ replaces[Anum_pg_mv_statistic_stamcv -1] = true;
+
+ /* always change the availability flags */
+ nulls[Anum_pg_mv_statistic_hist_built-1] = false;
+ nulls[Anum_pg_mv_statistic_mcv_built -1] = false;
+
+ replaces[Anum_pg_mv_statistic_hist_built -1] = true;
+ replaces[Anum_pg_mv_statistic_mcv_built -1] = true;
+
+ values[Anum_pg_mv_statistic_hist_built -1] = BoolGetDatum(histogram != NULL);
+ values[Anum_pg_mv_statistic_mcv_built -1] = BoolGetDatum(mcvlist != NULL);
+
+ /* Is there already a pg_mv_statistic tuple for this attribute? */
+ oldtup = SearchSysCache1(MVSTATOID,
+ ObjectIdGetDatum(mvoid));
+
+ if (HeapTupleIsValid(oldtup))
+ {
+ /* Yes, replace it */
+ stup = heap_modify_tuple(oldtup,
+ RelationGetDescr(sd),
+ values,
+ nulls,
+ replaces);
+ ReleaseSysCache(oldtup);
+ simple_heap_update(sd, &stup->t_self, stup);
+ }
+ else
+ elog(ERROR, "invalid pg_mv_statistic record (oid=%d)", mvoid);
+
+ /* update indexes too */
+ CatalogUpdateIndexes(sd, stup);
+
+ heap_freetuple(stup);
+
+ heap_close(sd, RowExclusiveLock);
+}
+
+
+/* MV stats */
+
+Datum
+pg_mv_stats_histogram_info(PG_FUNCTION_ARGS)
+{
+ bytea *data = PG_GETARG_BYTEA_P(0);
+ char *result;
+
+ MVHistogram hist = deserialize_mv_histogram(data);
+
+ result = palloc0(128);
+ snprintf(result, 128, "nbuckets=%d", hist->nbuckets);
+
+ PG_RETURN_TEXT_P(cstring_to_text(result));
+}
+
+Datum
+pg_mv_stats_mvclist_info(PG_FUNCTION_ARGS)
+{
+ bytea *data = PG_GETARG_BYTEA_P(0);
+ char *result;
+
+ MCVList mcvlist = deserialize_mv_mcvlist(data);
+
+ result = palloc0(128);
+ snprintf(result, 128, "nitems=%d", mcvlist->nitems);
+
+ pfree(mcvlist);
+
+ PG_RETURN_TEXT_P(cstring_to_text(result));
+}
+
+Datum
+pg_mv_stats_histogram_gnuplot(PG_FUNCTION_ARGS)
+{
+ int i = 0;
+
+ /* FIXME (handle the length properly using StringBuilder */
+ Size len = 1024*1024;
+ char *buffer = palloc0(len);
+ char *str = buffer;
+ bytea *data = PG_GETARG_BYTEA_P(0);
+
+ MVHistogram hist = deserialize_mv_histogram(data);
+
+ for (i = 0; i < hist->nbuckets; i++)
+ {
+ str += snprintf(str, len - (str - buffer),
+ "set object %d rect from %ld,%ld to %ld,%ld lw 1\n",
+ (i+1),
+ hist->buckets[i]->min[0], hist->buckets[i]->min[1],
+ hist->buckets[i]->max[0], hist->buckets[i]->max[1]);
+ }
+
+ PG_RETURN_TEXT_P(cstring_to_text(buffer));
+
+}
+
+bytea *
+fetch_mv_histogram(Oid mvoid)
+{
+ Relation indrel;
+ SysScanDesc indscan;
+ ScanKeyData skey;
+ HeapTuple htup;
+ bytea *stahist = NULL;
+
+ /* Prepare to scan pg_mv_statistic for entries having indrelid = this rel. */
+ ScanKeyInit(&skey,
+ ObjectIdAttributeNumber,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(mvoid));
+
+ indrel = heap_open(MvStatisticRelationId, AccessShareLock);
+ indscan = systable_beginscan(indrel, MvStatisticOidIndexId, true,
+ NULL, 1, &skey);
+
+ while (HeapTupleIsValid(htup = systable_getnext(indscan)))
+ {
+ bool isnull = false;
+ Datum hist = SysCacheGetAttr(MVSTATOID, htup,
+ Anum_pg_mv_statistic_stahist, &isnull);
+
+ Assert(!isnull);
+
+ stahist = DatumGetByteaP(hist);
+
+ break;
+ }
+
+ systable_endscan(indscan);
+
+ heap_close(indrel, AccessShareLock);
+
+ /* TODO maybe save the list into relcache, as in RelationGetIndexList
+ * (which was used as an inspiration of this one)?. */
+
+ return stahist;
+}
+
+bytea *
+fetch_mv_mcvlist(Oid mvoid)
+{
+ Relation indrel;
+ SysScanDesc indscan;
+ ScanKeyData skey;
+ HeapTuple htup;
+ bytea *mcvlist = NULL;
+
+ /* Prepare to scan pg_mv_statistic for entries having indrelid = this rel. */
+ ScanKeyInit(&skey,
+ ObjectIdAttributeNumber,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(mvoid));
+
+ indrel = heap_open(MvStatisticRelationId, AccessShareLock);
+ indscan = systable_beginscan(indrel, MvStatisticOidIndexId, true,
+ NULL, 1, &skey);
+
+ while (HeapTupleIsValid(htup = systable_getnext(indscan)))
+ {
+ bool isnull = false;
+ Datum tmp = SysCacheGetAttr(MVSTATOID, htup,
+ Anum_pg_mv_statistic_stamcv, &isnull);
+
+ Assert(!isnull);
+
+ mcvlist = DatumGetByteaP(tmp);
+
+ break;
+ }
+
+ systable_endscan(indscan);
+
+ heap_close(indrel, AccessShareLock);
+
+ /* TODO maybe save the list into relcache, as in RelationGetIndexList
+ * (which was used as an inspiration of this one)?. */
+
+ return mcvlist;
+}
+
+int
+mv_get_index(AttrNumber varattno, int2vector * stakeys)
+{
+ int i, idx = 0;
+ for (i = 0; i < stakeys->dim1; i++)
+ {
+ if (stakeys->values[i] < varattno)
+ idx += 1;
+ else
+ break;
+ }
+ return idx;
+}
+
+/*
+ * Building a multivariate algorithm. In short it first creates a single
+ * bucket containing all the rows, and then repeatedly split is by first
+ * searching for the bucket / dimension most in need of a split.
+ *
+ * The current criteria is rather simple, by looking at the number of
+ * distinct values (combination of column values for bucket, column
+ * values for a dimension). This is somehow naive, but seems to work
+ * quite well. See the discussion at select_bucket_to_partition and
+ * partition_bucket for more details about alternative algorithms.
+ *
+ * So the current algorithm looks like this:
+ *
+ * while [not reaching maximum number of buckets]
+ *
+ * choose bucket to partition (max distinct combinations)
+ * if no bucket to partition
+ * terminate the algorithm
+ *
+ * choose bucket dimension to partition (max distinct values)
+ * split the bucket into two buckets
+ *
+ */
+static MVHistogram
+build_mv_histogram(int numrows, HeapTuple *rows, int2vector *attrs,
+ int attr_cnt, VacAttrStats **vacattrstats,
+ int numrows_total)
+{
+ int i;
+ int ndistinct;
+ int numattrs = attrs->dim1;
+ int *ndistincts = (int*)palloc0(sizeof(int) * numattrs);
+
+ MVHistogram histogram = (MVHistogram)palloc0(sizeof(MVHistogramData));
+
+ HeapTuple * rows_copy = (HeapTuple*)palloc0(numrows * sizeof(HeapTuple));
+ memcpy(rows_copy, rows, sizeof(HeapTuple) * numrows);
+
+ Assert((numattrs >= 2) && (numattrs <= MVSTATS_MAX_DIMENSIONS));
+
+ histogram->ndimensions = numattrs;
+
+ histogram->magic = MVHIST_MAGIC;
+ histogram->type = MVHIST_TYPE_BASIC;
+ histogram->nbuckets = 1;
+
+ /* create max buckets (better than repalloc for short-lived objects) */
+ histogram->buckets = (MVBucket*)palloc0(MVHIST_MAX_BUCKETS * sizeof(MVBucket));
+
+ /* create the initial bucket, covering the whole sample set */
+ histogram->buckets[0] = create_initial_mv_bucket(numrows, rows_copy, attrs,
+ attr_cnt, vacattrstats);
+
+ ndistinct = histogram->buckets[0]->ndistinct;
+
+ /* keep the global ndistinct values */
+ for (i = 0; i < numattrs; i++)
+ ndistincts[i] = histogram->buckets[0]->ndistincts[i];
+
+ while (histogram->nbuckets < MVHIST_MAX_BUCKETS)
+ {
+ MVBucket bucket = select_bucket_to_partition(histogram->nbuckets, histogram->buckets);
+
+ /* no more buckets to partition */
+ if (bucket == NULL)
+ break;
+
+ histogram->buckets[histogram->nbuckets] = partition_bucket(bucket, attrs,
+ attr_cnt, vacattrstats);
+
+ histogram->nbuckets += 1;
+ }
+
+ /*
+ * FIXME store the histogram in a catalog in a serialized form (simple for
+ * pass-by-value, more complicated for buckets on varlena types)
+ */
+ for (i = 0; i < histogram->nbuckets; i++)
+ {
+ int d;
+ histogram->buckets[i]->ntuples = (histogram->buckets[i]->numrows * 1.0) / numrows_total;
+ histogram->buckets[i]->ndistinct = (histogram->buckets[i]->ndistinct * 1.0) / ndistinct;
+
+ for (d = 0; d < numattrs; d++)
+ histogram->buckets[i]->ndistincts[d] = (histogram->buckets[i]->ndistincts[d] * 1.0) / ndistincts[d];
+ }
+
+ pfree(ndistincts);
+
+ return histogram;
+
+}
+
+/*
+ * Mine associations between the columns, in the form (A => B).
+ *
+ * At the moment this only works for associations between two columns,
+ * but it might be useful to mine for rules involving multiple columns
+ * on the left side. That is rules [A,B] => C and so on. Handling
+ * multiple columns on the right side is not necessary, because such
+ * rules may be decomposed into a set of rules, one for each column.
+ * I.e. A => [B,C] is exactly the same as (A => B) & (A => C).
+ *
+ * Those rules don't immediately identify redundant clauses, because the
+ * user may choose "incompatible conditions" (e.g. by using a zip code
+ * and a mismatching city) and so on. This should however be easy to
+ * identify from a histogram, because the conditions will match a bucket
+ * with low frequencies.
+ *
+ * The question is whether this can be useful when we have a histogram,
+ * because such incompatible conditions should result in not matching
+ * any buckets (or matching only buckets with low frequencies).
+ *
+ * The problem is that histograms work like this when the sorting is
+ * compatible with the meaning of the data. We're often using data types
+ * that support sorting (e.g. INT, BIGING) as a kind of labels where
+ * the sorting really does not make much sense. Sorting by ZIP code will
+ * result in sorting the cities quite randomly, and similarly for most
+ * surrogate primary / foreign keys. In such cases the histograms are
+ * pretty useless.
+ *
+ * So, a good approach might be testing the independence of the data
+ * (by building a contingency table) and buildint the MV histogram only
+ * when there's a dependency. For the 'label' data this should notice
+ * the histogram is useless. So we won't build it (and we may use that
+ * as a sign supporting the association rule).
+ *
+ * Another option is to look at selectivity of A and B separately, and
+ * then use the minimum of those.
+ *
+ * TODO investigate using histogram and MCV list to confirm the
+ * associative rule
+ *
+ * TODO investigate statistical testing of the distribution (to decide
+ * whether it makes sense to build the histogram)
+ *
+ * TODO Using a min/max of selectivities would probably make more sense
+ * for the associated columns.
+ */
+static void
+build_mv_associations(int numrows, HeapTuple *rows, int2vector *attrs,
+ int natts, VacAttrStats **vacattrstats)
+{
+ int i;
+ bool isNull;
+ Size len = 2 * sizeof(Datum); /* only simple associations a => b */
+ int numattrs = attrs->dim1;
+
+ /* TODO Maybe this should be somehow related to the number of
+ * distinct columns in the two columns we're currently analyzing.
+ * Assuming the distribution is uniform, we should expected to
+ * observe in the sample - we can then use the average group
+ * size as a threshold. That seems better than a static approach.
+ */
+ int min_group_size = 10;
+
+ /* dimension indexes we'll check for associations [a => b] */
+ int dima, dimb;
+
+ /* info for the interesting attributes only
+ *
+ * TODO Compute this only once and pass it to all the methods
+ * that need it.
+ */
+ VacAttrStats **stats = lookup_var_attr_stats(attrs, natts, vacattrstats);
+
+ /* We'll reuse the same array for all the combinations */
+ Datum * values = (Datum*)palloc0(numrows * 2 * sizeof(Datum));
+
+ Assert(numattrs >= 2);
+
+ for (dima = 0; dima < numattrs; dima++)
+ {
+
+ for (dimb = 0; dimb < numattrs; dimb++)
+ {
+
+ int supporting = 0;
+ int contradicting = 0;
+
+ Datum val_a, val_b;
+ int violations = 0;
+ int group_size = 0;
+
+ int supporting_rows = 0;
+
+ /* skip (dima==dimb) */
+ if (dima == dimb)
+ continue;
+
+ /*
+ * FIXME Not sure if this handles NULL values properly (not sure
+ * how to do that). We assume that NULL means 0 for now,
+ * handling it just like any other value.
+ */
+ for (i = 0; i < numrows; i++)
+ {
+ values[i*2] = heap_getattr(rows[i], attrs->values[dima], stats[dima]->tupDesc, &isNull);
+ values[i*2+1] = heap_getattr(rows[i], attrs->values[dimb], stats[dimb]->tupDesc, &isNull);
+ }
+
+ qsort_arg((void *) values, numrows, sizeof(Datum) * 2, compare_scalars_memcmp, &len);
+
+ /*
+ * Walk through the array, split it into rows according to
+ * the A value, and count distinct values in the other one.
+ * If there's a single B value for the whole group, we count
+ * it as supporting the association, otherwise we count it
+ * as contradicting.
+ *
+ * Furthermore we require a group to have at least a certain
+ * number of rows to be considered useful. When contradicting,
+ * use it always.
+ */
+
+ /* start with values from the first row */
+ val_a = values[0];
+ val_b = values[1];
+ group_size = 1;
+
+ for (i = 1; i < numrows; i++)
+ {
+ if (values[2*i] != val_a) /* end of the group */
+ {
+ /*
+ * If there are no contradicting rows, count it as
+ * supporting (otherwise contradicting), but only if
+ * the group is large enough.
+ *
+ * The requirement of a minimum group size makes it
+ * impossible to identify [unique,unique] cases, but
+ * that's probably a different case. This is more
+ * about [zip => city] associations etc.
+ */
+ supporting += ((violations == 0) && (group_size >= min_group_size)) ? 1 : 0;
+ contradicting += (violations != 0) ? 1 : 0;
+
+ supporting_rows += ((violations == 0) && (group_size >= min_group_size)) ? group_size : 0;
+
+ /* current values start a new group */
+ val_a = values[2*i];
+ val_b = values[2*i+1];
+ violations = 0;
+ group_size = 1;
+ }
+ else
+ {
+ if (values[2*i+1] != val_b) /* mismatch of a B value */
+ {
+ val_b = values[2*i+1];
+ violations += 1;
+ }
+
+ group_size += 1;
+ }
+ }
+
+ /* FIXME handle the last group */
+ supporting += ((violations == 0) && (group_size >= min_group_size)) ? 1 : 0;
+ contradicting += (violations != 0) ? 1 : 0;
+ supporting_rows += ((violations == 0) && (group_size >= min_group_size)) ? group_size : 0;
+
+ /*
+ * See if the number of rows supporting the association is at least
+ * 10x the number of rows violating the hypothetical rule.
+ *
+ * TODO This is rather arbitrary limit - I guess it's possible to do
+ * some math to come up with a better rule (e.g. testing a hypothesis
+ * 'this is due to randomness'). We can create a contingency table
+ * from the values and use it for testing. Possibly only when
+ * there are no contradicting rows?
+ *
+ * TODO Also, if (a => b) and (b => a) at the same time, it pretty much
+ * means the columns have the same values (or one is a 'label'),
+ * making the conditions rather redundant. Although it's possible
+ * that the query uses incompatible combination of values.
+ */
+ if (supporting_rows > (numrows - supporting_rows) * 10)
+ {
+ // elog(WARNING, "%d => %d : supporting=%d contradicting=%d", dima, dimb, supporting, contradicting);
+ }
+
+ }
+ }
+
+ pfree(values);
+
+}
+
+/*
+ * Compute the list of most common items, where item is a combination of
+ * values for all the columns. For small number of distinct values, we
+ * may be able to represent the distribution pretty exactly, with
+ * per-item statistics.
+ *
+ * If we can represent the distribution using a MCV list only, it's great
+ * because that allows much better estimates (especially for equality).
+ * Such discrete distributions are also easier to combine (more
+ * efficient and more accurate) than when using histograms.
+ *
+ * FIXME This does not handle NULL values at the moment.
+ *
+ * TODO When computing equality selectivity (a=1 AND b=2), we can do that
+ * pretty exactly assuming (a) we hit a MCV item and (b) the
+ * histogram is built on those two columns only (i.e. there are no
+ * other columns). In that case we can estimate the selectivity
+ * using only the MCV.
+ *
+ * When we don't hit a MCV item, we can use the frequency of the
+ * least probable MCV item as upper bound of the selectivity
+ * (otherwise it'd get into the MCV list). Again, this only works
+ * when the histogram size matches the restricted columns.
+ *
+ * When the histogram is larger (i.e. there are additional columns),
+ * we can't be sure how is the selectivity distributed among the MCV
+ * list and the histogram (we may get several MCV items matching
+ * the conditions and several histogram buckets at the same time).
+ *
+ * In this case we can probably clamp the selectivity by minimum of
+ * selectivities for each condition. For example if we know the
+ * number of distinct values for each column, we can use 1/ndistinct
+ * as a per-column estimate. Or rather 1/ndistinct + selectivity
+ * derived from the MCV list.
+ *
+ * If there's no histogram (thus the distribution is approximated
+ * only by the MCV list), the size of the stats (whether there are
+ * some other columns, not referenced in the conditions) does not
+ * matter. We can do pretty accurate estimation using the MCV.
+ *
+ * TODO Currently there's no logic to consider building only a MCV list
+ * (and not building the histogram at all).
+ *
+ * TODO For types that don't reasonably support ordering (either because
+ * the type does not support that or when the user adds some option
+ * to the ADD STATISTICS command - e.g. UNSORTED_STATS), building
+ * the histogram may be pointless and inefficient. This is esp.
+ * true for varlena types that may be quite large and a large MCV
+ * list may be a better choice, because it makes equality estimates
+ * more accurate. Due to the unsorted nature, range queries on those
+ * attributes are rather useless anyway.
+ *
+ * Another thing is that by restricting to MCV list and equality
+ * conditions, we can use hash values instead of long varlena values.
+ * The equality estimation will be very accurate.
+ *
+ * This however complicates matching the columns to available
+ * statistics, as it will require matching clauses (not columns) to
+ * stats. And it may get quite complex - e.g. what if there are
+ * multiple clauses, each compatible with different stats subset?
+ *
+ * FIXME Create a special-purpose type for MCV items (instead of a plain
+ * Datum array, which is very difficult to work with).
+ */
+static MCVList
+build_mv_mcvlist(int numrows, HeapTuple *rows, int2vector *attrs,
+ int natts, VacAttrStats **vacattrstats,
+ int *numrows_filtered)
+{
+ int i, j, idx = 0;
+ int numattrs = attrs->dim1;
+ Size len = sizeof(Datum) * numattrs;
+ bool isNull;
+ int ndistinct = 0;
+ int mcv_threshold = 0;
+ int count = 0;
+ int nitems = 0;
+
+ MCVList mcvlist = NULL;
+
+ VacAttrStats **stats = lookup_var_attr_stats(attrs, natts, vacattrstats);
+
+ /*
+ * We could collect this while walking through all the attributes
+ * above (this way we have to call heap_getattr twice).
+ *
+ * TODO We're using Datum (8B), even for data types smaller than this
+ * (notably int4 and float4). Maybe we could save some space here,
+ * although it seems the bytea compression will handle it just fine.
+ */
+ Datum * values = palloc0(numrows * numattrs * sizeof(Datum));
+
+ for (j = 0; j < numrows; j++)
+ for (i = 0; i < numattrs; i++)
+ values[idx++] = heap_getattr(rows[j], attrs->values[i], stats[i]->tupDesc, &isNull);
+
+ qsort_arg((void *) values, numrows, sizeof(Datum) * numattrs, compare_scalars_memcmp, &len);
+
+ /*
+ * Count the number of distinct values - we need this to determine
+ * the threshold (125% of the average frequency).
+ */
+ ndistinct = 1;
+ for (i = 1; i < numrows; i++)
+ if (memcmp(&values[i * numattrs], &values[(i-1) * numattrs], len) != 0)
+ ndistinct += 1;
+
+ /*
+ * Determine how many groups actually exceed the threshold, and then
+ * walk the array again and collect them into an array.
+ *
+ * TODO for now the threshold is the same as in the single-column
+ * case (average + 25%), but maybe that's worth revisiting
+ *
+ * TODO see if we can fit all the distinct values in the MCV list
+ */
+ mcv_threshold = 1.25 * numrows / ndistinct;
+ mcv_threshold = (mcv_threshold < 4) ? 4 : mcv_threshold;
+
+ /*
+ * If there are less than some number of items, store all with at
+ * least two rows in the sample.
+ *
+ * FIXME We can do this only if we believe we got all the distinct
+ * values of the table.
+ */
+ if (ndistinct <= MVSTAT_MCVLIST_MAX_ITEMS)
+ mcv_threshold = 2;
+
+ count = 1;
+ for (i = 1; i <= numrows; i++)
+ {
+ /* last row or a new group */
+ if ((i == numrows) || (memcmp(&values[i * numattrs], &values[(i-1) * numattrs], len) != 0))
+ {
+ /* count the MCV item if exceeding the threshold */
+ if (count >= mcv_threshold)
+ nitems += 1;
+
+ count = 1;
+ }
+ else /* same group, just increase the number of items */
+ count += 1;
+ }
+
+ /* by default we keep all the rows (even if there's no MCV list) */
+ *numrows_filtered = numrows;
+
+ /* we know the number of mcvitems, now collect them in a 2nd pass */
+ if (nitems > 0)
+ {
+ /* we need to store the frequency for each group, so (numattrs + 1) */
+ mcvlist = (MCVList)palloc0(sizeof(MCVListData));
+
+ mcvlist->magic = MVSTAT_MCV_MAGIC;
+ mcvlist->type = MVSTAT_MCV_TYPE_BASIC;
+ mcvlist->ndimensions = numattrs;
+ mcvlist->nitems = nitems;
+ mcvlist->items = (MCVItem*)palloc0(sizeof(MCVItem)*nitems);
+
+ /* now repeat the same loop as above, but this time copy the data
+ * for items exceeding the threshold */
+ count = 1;
+ nitems = 0;
+ for (i = 1; i <= numrows; i++)
+ {
+
+ /* last row or a new group */
+ if ((i == numrows) || (memcmp(&values[i * numattrs], &values[(i-1) * numattrs], len) != 0))
+ {
+ /* count the MCV item if exceeding the threshold (and copy into the array) */
+ if (count >= mcv_threshold)
+ {
+ /* first, allocate the item (with the proper size of values) */
+ MCVItem item = (MCVItem)palloc0(offsetof(MCVItemData, values) +
+ sizeof(Datum)*mcvlist->ndimensions);
+
+ /* then copy values from the _previous_ group */
+ memcpy(item->values, &values[(i-1)*numattrs], len);
+
+ /* and finally the group frequency */
+ item->frequency = (double)count / numrows;
+
+ mcvlist->items[nitems] = item;
+ nitems += 1;
+ }
+
+ count = 1;
+ }
+ else /* same group, just increase the number of items */
+ count += 1;
+ }
+
+ /* make sure the loops are consistent */
+ Assert(nitems == mcvlist->nitems);
+
+ /*
+ * Remove the rows matching the MCV items.
+ *
+ * FIXME This implementation is rather naive, effectively O(N^2).
+ * As the MCV list grows, the check will take longer and
+ * longer. And as the number of sampled rows increases (by
+ * increasing statistics target), it will take longer and
+ * longer. One option is to sort the MCV items first and
+ * then perform a binary search.
+ */
+ if (nitems == ndistinct) /* all rows are covered by MCV items */
+ *numrows_filtered = 0;
+ else /* (nitems < ndistinct) && (nitems > 0) */
+ {
+ int nfiltered = 0;
+ HeapTuple *rows_filtered = (HeapTuple*)palloc0(sizeof(HeapTuple) * numrows);
+
+ /* walk through the tuples, compare the values to MCV items */
+ for (i = 0; i < numrows; i++)
+ {
+ bool match = false;
+ Datum *keys = (Datum*)palloc0(numattrs * sizeof(Datum));
+
+ /* collect the key values */
+ for (j = 0; j < numattrs; j++)
+ keys[j] = heap_getattr(rows[i], attrs->values[j], stats[j]->tupDesc, &isNull);
+
+ /* scan through the MCV list for matches */
+ for (j = 0; j < mcvlist->nitems; j++)
+ if (memcmp(keys, mcvlist->items[j]->values, sizeof(Datum)*numattrs) == 0)
+ {
+ match = true;
+ break;
+ }
+
+ /* if no match in the MCV list, copy the row into the filtered ones */
+ if (! match)
+ memcpy(&rows_filtered[nfiltered++], &rows[i], sizeof(HeapTuple));
+
+ pfree(keys);
+ }
+
+ /* replace the first part */
+ memcpy(rows, rows_filtered, sizeof(HeapTuple) * nfiltered);
+ *numrows_filtered = nfiltered;
+
+ pfree(rows_filtered);
+
+ }
+ }
+
+ pfree(values);
+
+ /*
+ * TODO Single-dimensional MCV is stored sorted by frequency (descending).
+ * Maybe this should be stored like that too?
+ */
+
+ return mcvlist;
+}
+
+/* multi-variate stats comparator */
+
+/*
+ * qsort_arg comparator for sorting Datums (MV stats)
+ *
+ * This does not maintain the tupnoLink array.
+ */
+static int
+compare_scalars_simple(const void *a, const void *b, void *arg)
+{
+ Datum da = *(Datum*)a;
+ Datum db = *(Datum*)b;
+ SortSupport ssup= (SortSupport) arg;
+
+ return ApplySortComparator(da, false, db, false, ssup);
+}
+
+/*
+ * qsort_arg comparator for sorting data when partitioning a MV bucket
+ */
+static int
+compare_scalars_partition(const void *a, const void *b, void *arg)
+{
+ Datum da = ((ScalarItem*)a)->value;
+ Datum db = ((ScalarItem*)b)->value;
+ SortSupport ssup= (SortSupport) arg;
+
+ return ApplySortComparator(da, false, db, false, ssup);
+}
+
+/*
+ * qsort_arg comparator for sorting Datum[] (row of Datums) when
+ * counting distinct values.
+ */
+static int
+compare_scalars_memcmp(const void *a, const void *b, void *arg)
+{
+ Size len = *(Size*)arg;
+
+ return memcmp(a, b, len);
+}
+
+static int
+compare_scalars_memcmp_2(const void *a, const void *b)
+{
+ return memcmp(a, b, sizeof(Datum));
+}
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 714a9f1..7f9e54f 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -35,6 +35,7 @@
#include "catalog/pg_foreign_table.h"
#include "catalog/pg_inherits.h"
#include "catalog/pg_inherits_fn.h"
+#include "catalog/pg_mv_statistic.h"
#include "catalog/pg_namespace.h"
#include "catalog/pg_opclass.h"
#include "catalog/pg_rowsecurity.h"
@@ -91,7 +92,7 @@
#include "utils/syscache.h"
#include "utils/tqual.h"
#include "utils/typcache.h"
-
+#include "utils/mvstats.h"
/*
* ON COMMIT action list
@@ -139,8 +140,9 @@ static List *on_commits = NIL;
#define AT_PASS_ADD_COL 5 /* ADD COLUMN */
#define AT_PASS_ADD_INDEX 6 /* ADD indexes */
#define AT_PASS_ADD_CONSTR 7 /* ADD constraints, defaults */
-#define AT_PASS_MISC 8 /* other stuff */
-#define AT_NUM_PASSES 9
+#define AT_PASS_ADD_STATS 8 /* ADD statistics */
+#define AT_PASS_MISC 9 /* other stuff */
+#define AT_NUM_PASSES 10
typedef struct AlteredTableInfo
{
@@ -414,7 +416,8 @@ static void ATExecReplicaIdentity(Relation rel, ReplicaIdentityStmt *stmt, LOCKM
static void ATExecGenericOptions(Relation rel, List *options);
static void ATExecEnableRowSecurity(Relation rel);
static void ATExecDisableRowSecurity(Relation rel);
-
+static void ATExecAddStatistics(AlteredTableInfo *tab, Relation rel,
+ StatisticsDef *def, LOCKMODE lockmode);
static void copy_relation_data(SMgrRelation rel, SMgrRelation dst,
ForkNumber forkNum, char relpersistence);
static const char *storage_name(char c);
@@ -2965,6 +2968,7 @@ AlterTableGetLockLevel(List *cmds)
* updates.
*/
case AT_SetStatistics: /* Uses MVCC in getTableAttrs() */
+ case AT_AddStatistics: /* XXX not sure if the right level */
case AT_ClusterOn: /* Uses MVCC in getIndexes() */
case AT_DropCluster: /* Uses MVCC in getIndexes() */
case AT_SetOptions: /* Uses MVCC in getTableAttrs() */
@@ -3112,6 +3116,7 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd,
pass = AT_PASS_ADD_CONSTR;
break;
case AT_SetStatistics: /* ALTER COLUMN SET STATISTICS */
+ case AT_AddStatistics: /* XXX maybe not the right place */
ATSimpleRecursion(wqueue, rel, cmd, recurse, lockmode);
/* Performs own permission checks */
ATPrepSetStatistics(rel, cmd->name, cmd->def, lockmode);
@@ -3407,6 +3412,9 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,
case AT_SetStatistics: /* ALTER COLUMN SET STATISTICS */
ATExecSetStatistics(rel, cmd->name, cmd->def, lockmode);
break;
+ case AT_AddStatistics: /* ADD STATISTICS */
+ ATExecAddStatistics(tab, rel, (StatisticsDef *) cmd->def, lockmode);
+ break;
case AT_SetOptions: /* ALTER COLUMN SET ( options ) */
ATExecSetOptions(rel, cmd->name, cmd->def, false, lockmode);
break;
@@ -11616,3 +11624,197 @@ RangeVarCallbackForAlterRelation(const RangeVar *rv, Oid relid, Oid oldrelid,
ReleaseSysCache(tuple);
}
+
+/* used for sorting the attnums in ATExecAddStatistics */
+static int compare_int16(const void *a, const void *b)
+{
+ return memcmp(a, b, sizeof(int16));
+}
+
+/*
+ * Implements the ALTER TABLE ... ADD STATISTICS (options) ON (columns).
+ *
+ * The code is an unholy mix of pieces that really belong to other parts
+ * of the source tree.
+ *
+ * FIXME Check that the types are pass-by-value and support sort,
+ * although maybe we can live without the sort (and only build
+ * MCV list / association rules).
+ *
+ * FIXME This should probably check for duplicate stats (i.e. same
+ * keys, same options). Although maybe it's useful to have
+ * multiple stats on the same columns with different options
+ * (say, a detailed MCV-only stats for some queries, histogram
+ * for others, etc.)
+ */
+static void ATExecAddStatistics(AlteredTableInfo *tab, Relation rel,
+ StatisticsDef *def, LOCKMODE lockmode)
+{
+ int i, j;
+ ListCell *l;
+ int16 attnums[INDEX_MAX_KEYS];
+ Oid atttypids[INDEX_MAX_KEYS];
+ int numcols = 0;
+
+ Oid mvstatoid;
+ HeapTuple htup;
+ Datum values[Natts_pg_mv_statistic];
+ bool nulls[Natts_pg_mv_statistic];
+ int2vector *stakeys;
+ Relation mvstatrel;
+
+ /* by default build everything */
+ bool build_histogram = true,
+ build_mcv = true,
+ build_associations = true;
+
+ /* build regular MCV (not hashed by default) */
+ bool mcv_hashed = false;
+
+ int32 max_buckets = -1,
+ max_mcv_items = -1;
+
+ Assert(IsA(def, StatisticsDef));
+
+ /* transform the column names to attnum values */
+
+ foreach(l, def->keys)
+ {
+ char *attname = strVal(lfirst(l));
+ HeapTuple atttuple;
+
+ atttuple = SearchSysCacheAttName(RelationGetRelid(rel), attname);
+
+ if (!HeapTupleIsValid(atttuple))
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_COLUMN),
+ errmsg("column \"%s\" referenced in statistics does not exist",
+ attname)));
+
+ /* more than MVHIST_MAX_DIMENSIONS columns not allowed */
+ if (numcols >= MVSTATS_MAX_DIMENSIONS)
+ ereport(ERROR,
+ (errcode(ERRCODE_TOO_MANY_COLUMNS),
+ errmsg("cannot have more than %d keys in a statistics",
+ MVSTATS_MAX_DIMENSIONS)));
+
+ attnums[numcols] = ((Form_pg_attribute) GETSTRUCT(atttuple))->attnum;
+ atttypids[numcols] = ((Form_pg_attribute) GETSTRUCT(atttuple))->atttypid;
+ ReleaseSysCache(atttuple);
+ numcols++;
+ }
+
+ /*
+ * Check the lower bound (at least 2 columns), the upper bound was
+ * already checked in the loop.
+ */
+ if (numcols < 2)
+ ereport(ERROR,
+ (errcode(ERRCODE_TOO_MANY_COLUMNS),
+ errmsg("multivariate stats require 2 or more columns")));
+
+ /* look for duplicities */
+ for (i = 0; i < numcols; i++)
+ for (j = 0; j < numcols; j++)
+ if ((i != j) && (attnums[i] == attnums[j]))
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_COLUMN),
+ errmsg("duplicate column name in statistics definition")));
+
+ /* parse the statistics options */
+ foreach (l, def->options)
+ {
+ DefElem *opt = (DefElem*)lfirst(l);
+
+ if (strcmp(opt->defname, "histogram") == 0)
+ build_histogram = defGetBoolean(opt);
+ else if (strcmp(opt->defname, "mcv") == 0)
+ build_mcv = defGetBoolean(opt);
+ else if (strcmp(opt->defname, "mcv_hashed") == 0)
+ mcv_hashed = defGetBoolean(opt);
+ else if (strcmp(opt->defname, "associations") == 0)
+ build_associations = defGetBoolean(opt);
+ else if (strcmp(opt->defname, "max_buckets") == 0)
+ {
+ max_buckets = defGetInt32(opt);
+
+ /* TODO check that this is not used with 'histogram off' */
+
+ /* sanity check */
+ if (max_buckets < 1024)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("minimum number of buckets is 1024")));
+
+ else if (max_buckets > 32768) /* FIXME use the proper constant */
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("minimum number of buckets is 1024")));
+
+ }
+ else if (strcmp(opt->defname, "max_mcv_items") == 0)
+ {
+ max_mcv_items = defGetInt32(opt);
+
+ /* TODO check that this is not used with 'mcv off' */
+
+ /* sanity check */
+ if (max_mcv_items < 0)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("max number of MCV items must be non-negative")));
+
+ else if (max_mcv_items > 8192) /* FIXME use the proper constant */
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("max number of MCV items is 8192")));
+
+ }
+ else
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("unrecognized STATISTICS option \"%s\"",
+ opt->defname)));
+ }
+
+ /* sort the attnums and build int2vector */
+ qsort(attnums, numcols, sizeof(int16), compare_int16);
+ stakeys = buildint2vector(attnums, numcols);
+
+ /*
+ * Okay, let's create the pg_mv_statistic entry.
+ */
+ memset(values, 0, sizeof(values));
+ memset(nulls, false, sizeof(nulls));
+
+ /* no stats collected yet, so just the keys */
+ values[Anum_pg_mv_statistic_starelid-1] = ObjectIdGetDatum(RelationGetRelid(rel));
+
+ values[Anum_pg_mv_statistic_stakeys -1] = PointerGetDatum(stakeys);
+ values[Anum_pg_mv_statistic_hist_enabled -1] = BoolGetDatum(build_histogram);
+ values[Anum_pg_mv_statistic_mcv_enabled -1] = BoolGetDatum(build_mcv);
+ values[Anum_pg_mv_statistic_mcv_hashed -1] = BoolGetDatum(mcv_hashed);
+ values[Anum_pg_mv_statistic_assoc_enabled -1] = BoolGetDatum(build_associations);
+
+ values[Anum_pg_mv_statistic_hist_max_buckets -1] = Int32GetDatum(max_buckets);
+ values[Anum_pg_mv_statistic_mcv_max_items -1] = Int32GetDatum(max_mcv_items);
+
+ nulls[Anum_pg_mv_statistic_staassoc -1] = true;
+ nulls[Anum_pg_mv_statistic_stamcv -1] = true;
+ nulls[Anum_pg_mv_statistic_stahist -1] = true;
+
+ /* insert the tuple into pg_mv_statistic */
+ mvstatrel = heap_open(MvStatisticRelationId, RowExclusiveLock);
+
+ htup = heap_form_tuple(mvstatrel->rd_att, values, nulls);
+
+ mvstatoid = simple_heap_insert(mvstatrel, htup);
+
+ CatalogUpdateIndexes(mvstatrel, htup);
+
+ heap_freetuple(htup);
+
+ heap_close(mvstatrel, RowExclusiveLock);
+
+ return;
+}
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index e76b5b3..da35331 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -3903,6 +3903,17 @@ _copyAlterPolicyStmt(const AlterPolicyStmt *from)
return newnode;
}
+static StatisticsDef *
+_copyStatisticsDef(const StatisticsDef *from)
+{
+ StatisticsDef *newnode = makeNode(StatisticsDef);
+
+ COPY_NODE_FIELD(keys);
+ COPY_NODE_FIELD(options);
+
+ return newnode;
+}
+
/* ****************************************************************
* pg_list.h copy functions
* ****************************************************************
@@ -4717,6 +4728,9 @@ copyObject(const void *from)
case T_CommonTableExpr:
retval = _copyCommonTableExpr(from);
break;
+ case T_StatisticsDef:
+ retval = _copyStatisticsDef(from);
+ break;
case T_PrivGrantee:
retval = _copyPrivGrantee(from);
break;
@@ -4729,7 +4743,6 @@ copyObject(const void *from)
case T_XmlSerialize:
retval = _copyXmlSerialize(from);
break;
-
default:
elog(ERROR, "unrecognized node type: %d", (int) nodeTag(from));
retval = 0; /* keep compiler quiet */
diff --git a/src/backend/optimizer/path/clausesel.c b/src/backend/optimizer/path/clausesel.c
index 9b657fb..9c32735 100644
--- a/src/backend/optimizer/path/clausesel.c
+++ b/src/backend/optimizer/path/clausesel.c
@@ -24,6 +24,9 @@
#include "utils/lsyscache.h"
#include "utils/selfuncs.h"
+#include "utils/mvstats.h"
+#include "catalog/pg_collation.h"
+#include "utils/typcache.h"
/*
* Data structure for accumulating info about possible range-query
@@ -43,6 +46,23 @@ static void addRangeClause(RangeQueryClause **rqlist, Node *clause,
bool varonleft, bool isLTsel, Selectivity s2);
+static bool is_mv_compatible(Node *clause, Oid varRelid, Index *varno,
+ Bitmapset **attnums);
+static Bitmapset *collect_mv_attnums(PlannerInfo *root, List *clauses,
+ Oid varRelid, Oid *relid);
+static int choose_mv_histogram(int nmvstats, MVStats mvstats,
+ Bitmapset *attnums);
+static List *clauselist_mv_split(List *clauses, Oid varRelid,
+ List **mvclauses, MVStats mvstats);
+
+static Selectivity clauselist_mv_selectivity(PlannerInfo *root,
+ List *clauses, MVStats mvstats);
+static Selectivity clauselist_mv_selectivity_mcvlist(PlannerInfo *root,
+ List *clauses, MVStats mvstats,
+ bool *fullmatch, Selectivity *lowsel);
+static Selectivity clauselist_mv_selectivity_histogram(PlannerInfo *root,
+ List *clauses, MVStats mvstats);
+
/****************************************************************************
* ROUTINES TO COMPUTE SELECTIVITIES
****************************************************************************/
@@ -100,14 +120,74 @@ clauselist_selectivity(PlannerInfo *root,
RangeQueryClause *rqlist = NULL;
ListCell *l;
+ /* processing mv stats */
+ Oid relid = InvalidOid;
+ int nmvstats = 0;
+ MVStats mvstats = NULL;
+
+ /* attributes in mv-compatible clauses */
+ Bitmapset *mvattnums = NULL;
+
/*
- * If there's exactly one clause, then no use in trying to match up pairs,
- * so just go directly to clause_selectivity().
+ * If there's exactly one clause, then no use in trying to match up
+ * pairs, so just go directly to clause_selectivity().
*/
if (list_length(clauses) == 1)
return clause_selectivity(root, (Node *) linitial(clauses),
varRelid, jointype, sjinfo);
+ /* collect attributes from mv-compatible clauses */
+ mvattnums = collect_mv_attnums(root, clauses, varRelid, &relid);
+
+ /*
+ * If there are mv-compatible clauses, referencing at least two
+ * columns (otherwise it makes no sense to use mv stats), fetch the
+ * MV histograms for the relation (only the column keys, not the
+ * histograms yet - we'll decide which histogram to use first).
+ */
+ if (bms_num_members(mvattnums) >= 2)
+ {
+ /* clauses compatible with multi-variate stats */
+ List *mvclauses = NIL;
+
+ /* fetch info from the catalog (not the serialized stats yet) */
+ mvstats = list_mv_stats(relid, &nmvstats, true);
+
+ /*
+ * If there are candidate statistics, choose the histogram first.
+ * At the moment we only use a single statistics, covering the
+ * most columns (using info from the previous step). If there
+ * are multiple such histograms, we'll use the smallest one
+ * (with the lowest number of dimensions).
+ *
+ * This may not be optimal choice, if the 'smaller' stats has
+ * much less buckets than the rejected one (making it less
+ * accurate).
+ *
+ * We may end up without multivariate statistics, if none of the
+ * stats matches at least two columns from the clauses (in that
+ * case we may just use the single dimensional stats).
+ */
+ if (nmvstats > 0)
+ {
+ int idx = choose_mv_histogram(nmvstats, mvstats, mvattnums);
+
+ if (idx >= 0) /* we have a matching stats */
+ {
+ MVStats mvstat = &mvstats[idx];
+
+ /* split the clauselist into regular and mv-clauses */
+ clauses = clauselist_mv_split(clauses, varRelid, &mvclauses, mvstat);
+
+ /* we've chosen the histogram to match the clauses */
+ Assert(mvclauses != NIL);
+
+ /* compute the multivariate stats */
+ s1 *= clauselist_mv_selectivity(root, mvclauses, mvstat);
+ }
+ }
+ }
+
/*
* Initial scan over clauses. Anything that doesn't look like a potential
* rangequery clause gets multiplied into s1 and forgotten. Anything that
@@ -782,3 +862,1010 @@ clause_selectivity(PlannerInfo *root,
return s1;
}
+
+
+
+/*
+ * Estimate selectivity for the list of MV-compatible clauses, using that
+ * particular histogram.
+ *
+ * When we hit a single bucket, we don't know what portion of it actually
+ * matches the clauses (e.g. equality), and we use 1/2 the bucket by
+ * default. However, the MV histograms are usually less detailed than
+ * the per-column ones, meaning the sum of buckets is often quite high
+ * (thanks to combining a lot of "partially hit" buckets).
+ *
+ * There are several ways to improve this, usually with cases when it
+ * won't really help. Also, the more complex the process, the worse
+ * the failures (i.e. misestimates).
+ *
+ * (1) Use the MV histogram only as a way to combine multiple
+ * per-column histograms, essentially rewriting
+ *
+ * P(A & B) = P(A) * P(B|A)
+ *
+ * where P(B|A) may be computed using a proper "slice" of the
+ * histogram, by first selecting only buckets where A is true, and
+ * then using the boundaries to 'restrict' the per-colunm histogram.
+ *
+ * With more clauses, it gets more complicated, of course
+ *
+ * P(A & B & C) = P(A & C) * P(B|A & C)
+ * = P(A) * P(C|A) * P(B|A & C)
+ *
+ * and so on.
+ *
+ * Of course, the question is how well and efficiently we can
+ * compute the conditional probabilities - whether this approach
+ * can improve the estimates (instead of amplifying the errors).
+ *
+ * Also, this does not eliminate the need for histogram on [A,B,C].
+ *
+ * (2) Use multiple smaller (and more accurate) histograms, and combine
+ * them using a process similar to the above. E.g. by assuming that
+ * B and C are independent, we can rewrite
+ *
+ * P(B|A & C) = P(B|A)
+ *
+ * so we can rewrite the whole formula to
+ *
+ * P(A & B & C) = P(A) * P(C|A) * P(B|A)
+ *
+ * and we're OK with two 2D histograms [A,C] and [A,B].
+ *
+ * It'd be nice to perform some sort of statistical test (Fisher
+ * or another chi-squared test) to identify independent components
+ * and automatically separate them into smaller histograms.
+ *
+ * (3) Using the estimated number of distinct values in a bucket to
+ * decide the selectivity of equality in the bucket (instead of
+ * blindly using 1/2 of the bucket, we may use 1/ndistinct).
+ * Of course, if the ndistinct estimate is way off, or when the
+ * distribution is not uniform (one distict items get much more
+ * items), this will fail. Also, we currently don't have ndistinct
+ * estimate available at this moment (but it shouldn't be that
+ * difficult to compute as ndistinct and ntuples should be available).
+ *
+ * TODO Clamp the selectivity by min of the per-clause selectivities
+ * (i.e. the selectivity of the most restrictive clause), because
+ * that's the maximum we can ever get from ANDed list of clauses.
+ * This may probably prevent issues with hitting too many buckets
+ * and low precision histograms.
+ *
+ * TODO We may support some additional conditions, most importantly
+ * those matching multiple columns (e.g. "a = b" or "a < b").
+ * Ultimately we could track multi-table histograms for join
+ * cardinality estimation.
+ *
+ * TODO Currently this is only estimating all clauses, or clauses
+ * matching varRelid (when it's not 0). I'm not sure what's the
+ * purpose of varRelid, but my assumption is this is used for
+ * join conditions and such. In that case we can use those clauses
+ * to restrict the other (i.e. filter the histogram buckets first,
+ * before estimating the other clauses). This is essentially equal
+ * to computing P(A|B) where "B" are the clauses not matching the
+ * varRelid.
+ *
+ * TODO Further thoughts on processing equality clauses - maybe it'd be
+ * better to look for stats (with MCV) covered by the equality
+ * clauses, because then we have a chance to find an exact match
+ * in the MCV list, which is pretty much the best we can do. We may
+ * also look at the least frequent MCV item, and use it as a upper
+ * boundary for the selectivity (had there been a more frequent
+ * item, it'd be in the MCV list).
+ *
+ * These conditions may then be used as a condition for the other
+ * selectivities, i.e. we may estimate P(A,B) first, and then
+ * compute P(C|A,B) from another histogram. This may be useful when
+ * we can estimate P(A,B) accurately (e.g. because it's a complete
+ * equality match evaluated on MCV list), and then compute the
+ * conditional probability P(C|A,B), giving us the requested stats
+ *
+ * P(A,B,C) = P(A,B) * P(C|A,B)
+ *
+ * TODO There are several options for 'sanity clamping' the estimates.
+ *
+ * First, if we have selectivities for each condition, then
+ *
+ * P(A,B) <= MIN(P(A), P(B))
+ *
+ * Because additional conditions (connected by AND) can only lower
+ * the probability.
+ *
+ * So we can do some basic sanity checks using the single-variate
+ * stats (the ones we have right now).
+ *
+ * Second, when we have multivariate stats with a MCV list, then
+ *
+ * (a) if we have a full equality condition (one equality condition
+ * on each column) and we found a match in the MCV list, this is
+ * the selectivity (and it's supposed to be exact)
+ *
+ * (b) if we have a full equality condition and we haven't found a
+ * match in the MCV list, then the selectivity is below the
+ * lowest selectivity in the MCV list
+ *
+ * (c) if we have a equality condition (not full), we can still
+ * search the MCV for matches and use the sum of probabilities
+ * as a lower boundary for the histogram (if there are no
+ * matches in the MCV list, then we have no boundary)
+ *
+ * Third, if there are multiple multivariate stats for a set of
+ * clauses, we may compute all of them and then somehow aggregate
+ * them - e.g. by choosing the minimum, median or average. The
+ * multi-variate stats are susceptible to overestimation (because
+ * we take 50% of the bucket for partial matches). Some stats may
+ * give better estimates than others, but it's very difficult to
+ * say determine that in advance which one is the best (it depends
+ * on the number of buckets, number of additional columns not
+ * referenced in the clauses etc.) so we may compute all and then
+ * choose a sane aggregation (minimum seems like a good approach).
+ * Of course, this may result in longer / more expensive estimation
+ * (CPU-wise), but it may be worth it.
+ *
+ * There are ways to address this, though. First, it's possible to
+ * add a GUC choosing whether to do a 'simple' (using a single
+ * stats expected to give the best estimate) and 'complex' (combining
+ * the multiple estimates).
+ *
+ * multivariate_estimates = (simple|full)
+ *
+ * Also, this might be enabled at a table level, by something like
+ *
+ * ALTER TABLE ... SET STATISTICS (simple|full)
+ *
+ * Which would make it possible to use this only for the tables
+ * where the simple approach does not work.
+ *
+ * Also, there are ways to optimize this algorithmically. E.g. we
+ * may try to get an estimate from a matching MCV list first, and
+ * if we happen to get a "full equality match" we may stop computing
+ * the estimates from other stats (for this condition) because
+ * that's probably the best estimate we can really get.
+ *
+ * TODO When applying the clauses to the histogram/MCV list, we can do
+ * that from the most selective clauses first, because that'll
+ * eliminate the buckets/items sooner (so we'll be able to skip
+ * them without inspection, which is more expensive).
+ */
+static Selectivity
+clauselist_mv_selectivity(PlannerInfo *root, List *clauses, MVStats mvstats)
+{
+ bool fullmatch = false;
+ Selectivity s1 = 0.0, s2 = 0.0;
+
+ /*
+ * Lowest frequency in the MCV list (may be used as an upper bound
+ * for full equality conditions that did not match any MCV item).
+ */
+ Selectivity mcv_low = 0.0;
+
+ /* TODO Evaluate simple 1D selectivities, use the smallest one as
+ * an upper bound, product as lower bound, and sort the
+ * clauses in ascending order by selectivity (to optimize the
+ * MCV/histogram evaluation).
+ */
+
+ /* Evaluate the MCV first. */
+ s1 = clauselist_mv_selectivity_mcvlist(root, clauses, mvstats,
+ &fullmatch, &mcv_low);
+
+ /*
+ * If we got a full equality match on the MCV list, we're done (and
+ * the estimate is pretty good).
+ */
+ if (fullmatch && (s1 > 0.0))
+ return s1;
+
+ /* FIXME if (fullmatch) without matching MCV item, use the mcv_low
+ * selectivity as upper bound */
+
+ s2 = clauselist_mv_selectivity_histogram(root, clauses, mvstats);
+
+ /* TODO clamp to <= 1.0 (or more strictly, when possible) */
+ return s1 + s2;
+}
+
+/*
+ * Collect attributes from mv-compatible clauses.
+ *
+ */
+static Bitmapset *
+collect_mv_attnums(PlannerInfo *root, List *clauses, Oid varRelid, Oid *relid)
+{
+ Index varno = 0;
+ Bitmapset *attnums = NULL;
+ ListCell *l;
+
+ /*
+ * Walk through the clauses and identify the ones we can estimate
+ * using multivariate stats, and remember the relid/columns. We'll
+ * then cross-check if we have suitable stats, and only if needed
+ * we'll split the clauses into multivariate and regular lists.
+ *
+ * For now we're only interested in RestrictInfo nodes with nested
+ * OpExpr, using either a range or equality.
+ */
+ foreach (l, clauses)
+ {
+ Node *clause = (Node *) lfirst(l);
+
+ /* ignore the result for now - we only need the info */
+ is_mv_compatible(clause, varRelid, &varno, &attnums);
+ }
+
+ /*
+ * If there are at least two attributes referenced by the clause(s),
+ * fetch the relation info (and pass back the Oid of the relation).
+ */
+ if (bms_num_members(attnums) > 1)
+ {
+ RelOptInfo *rel = find_base_rel(root, varno);
+ *relid = root->simple_rte_array[bms_singleton_member(rel->relids)]->relid;
+ }
+ else
+ {
+ if (attnums != NULL)
+ pfree(attnums);
+ attnums = NULL;
+ *relid = InvalidOid;
+ }
+
+ return attnums;
+}
+
+/*
+ * We're looking for a histogram matching at least 2 attributes, and we
+ * want the smallest histogram available wrt. to number of buckets (to
+ * get efficient estimation and likely better precision. The precision
+ * depends on the total number of buckets too, but the lower the number
+ * of dimensions the smaller (and more precise) the buckets can get.
+ */
+static int
+choose_mv_histogram(int nmvstats, MVStats mvstats, Bitmapset *attnums)
+{
+ int i, j;
+
+ int choice = -1;
+ int current_matches = 1; /* goal #1: maximize */
+ int current_dims = (MVSTATS_MAX_DIMENSIONS+1); /* goal #2: minimize */
+
+ for (i = 0; i < nmvstats; i++)
+ {
+ int matches = 0; /* columns matching this histogram */
+
+ int2vector * attrs = mvstats[i].stakeys;
+ int numattrs = mvstats[i].stakeys->dim1;
+
+ /* count columns covered by the histogram */
+ for (j = 0; j < numattrs; j++)
+ if (bms_is_member(attrs->values[j], attnums))
+ matches++;
+
+ /*
+ * Use this histogram when it improves the number of matches or
+ * when it keeps the number of matches and is smaller.
+ */
+ if ((matches > current_matches) ||
+ ((matches == current_matches) && (current_dims > numattrs)))
+ {
+ choice = i;
+ current_matches = matches;
+ current_dims = numattrs;
+ }
+ }
+
+ return choice;
+}
+
+/*
+ * This splits the clauses list into two parts - one containing clauses
+ * that will be evaluated using the chosen histogram, and the remaining
+ * clauses (either non-mvcompatible, or not related to the histogram).
+ */
+static List *
+clauselist_mv_split(List *clauses, Oid varRelid, List **mvclauses, MVStats mvstats)
+{
+ int i;
+ ListCell *l;
+ List *non_mvclauses = NIL;
+
+ /* FIXME is there a better way to get info on int2vector? */
+ int2vector * attrs = mvstats->stakeys;
+ int numattrs = mvstats->stakeys->dim1;
+
+ /* erase the list of mv-compatible clauses */
+ *mvclauses = NIL;
+
+ foreach (l, clauses)
+ {
+ RestrictInfo *rinfo;
+ Node *clause = (Node *) lfirst(l);
+
+ /*
+ * Only restrictinfo may be mv-compatible, so everything else
+ * goes to the non-mv list directly
+ *
+ * TODO create a macro/function to decide mv-compatible clauses
+ * (along the is_opclause for example)
+ */
+ if (! IsA(clause, RestrictInfo))
+ {
+ non_mvclauses = lappend(non_mvclauses, clause);
+ continue;
+ }
+
+ rinfo = (RestrictInfo *) clause;
+ clause = (Node*)rinfo->clause;
+
+ /* Pseudoconstants go directly to the non-mv list too. */
+ if (rinfo->pseudoconstant)
+ {
+ non_mvclauses = lappend(non_mvclauses, rinfo);
+ continue;
+ }
+
+ if (is_opclause(clause) && list_length(((OpExpr *) clause)->args) == 2)
+ {
+ OpExpr *expr = (OpExpr *) clause;
+ bool varonleft = true;
+ bool ok;
+
+ ok = (bms_membership(rinfo->clause_relids) == BMS_SINGLETON) &&
+ (is_pseudo_constant_clause_relids(lsecond(expr->args),
+ rinfo->right_relids) ||
+ (varonleft = false,
+ is_pseudo_constant_clause_relids(linitial(expr->args),
+ rinfo->left_relids)));
+
+ if (ok)
+ {
+
+ Var * var = (varonleft) ? linitial(expr->args) : lsecond(expr->args);
+
+ /*
+ * Only consider this variable if (varRelid == 0) or when the varno
+ * matches varRelid (see explanation at clause_selectivity).
+ */
+ if (! ((varRelid == 0) || (varRelid == var->varno)))
+ {
+ non_mvclauses = lappend(non_mvclauses, rinfo);
+ continue;
+ }
+
+ /*
+ * If it's not a "<" or ">" or "=" operator, just ignore the
+ * clause. Otherwise note the relid and attnum for the variable.
+ */
+ switch (get_oprrest(expr->opno))
+ {
+ case F_SCALARLTSEL:
+ case F_SCALARGTSEL:
+ case F_EQSEL:
+ if (! IS_SPECIAL_VARNO(var->varno)) /* FIXME necessary here? */
+ {
+ bool match = false;
+ for (i = 0; i < numattrs; i++)
+ if (attrs->values[i] == var->varattno)
+ match = true;
+
+ if (match)
+ *mvclauses = lappend(*mvclauses, clause);
+ else
+ non_mvclauses = lappend(non_mvclauses, rinfo);
+ }
+ }
+ }
+ }
+ }
+
+ /*
+ * Perform regular estimation using the clauses incompatible
+ * with the chosen histogram (or MV stats in general).
+ */
+ return non_mvclauses;
+
+}
+
+/*
+ * Determines whether the clause is compatible with multivariate stats,
+ * and if it is, returns some additional information - varno (index
+ * into simple_rte_array) and a bitmap of attributes. This is then
+ * used to fetch related multivariate statistics.
+ *
+ * At this moment we only support basic conditions of the form
+ *
+ * variable OP constant
+ *
+ * where OP is one of [=,<,<=,>=,>] (which is however determined by
+ * looking at the associated function for estimating selectivity, just
+ * like with the single-dimensional case).
+ */
+static bool
+is_mv_compatible(Node *clause, Oid varRelid, Index *varno, Bitmapset **attnums)
+{
+
+ if (IsA(clause, RestrictInfo))
+ {
+ RestrictInfo *rinfo = (RestrictInfo *) clause;
+
+ /* Pseudoconstants are not really interesting here. */
+ if (rinfo->pseudoconstant)
+ return false;
+
+ /* get the actual clause from the RestrictInfo ... */
+ clause = (Node*)rinfo->clause;
+
+ /* is it 'variable op constant' ? */
+ if (is_opclause(clause) && list_length(((OpExpr *) clause)->args) == 2)
+ {
+ OpExpr *expr = (OpExpr *) clause;
+ bool varonleft = true;
+ bool ok;
+
+ ok = (bms_membership(rinfo->clause_relids) == BMS_SINGLETON) &&
+ (is_pseudo_constant_clause_relids(lsecond(expr->args),
+ rinfo->right_relids) ||
+ (varonleft = false,
+ is_pseudo_constant_clause_relids(linitial(expr->args),
+ rinfo->left_relids)));
+
+ if (ok)
+ {
+
+ Var * var = (varonleft) ? linitial(expr->args) : lsecond(expr->args);
+
+ /*
+ * Only consider this variable if (varRelid == 0) or when the varno
+ * matches varRelid (see explanation at clause_selectivity).
+ */
+ if (! ((varRelid == 0) || (varRelid == var->varno)))
+ return false;
+
+ /* Also skip special varno values, and system attributes ... */
+ if ((IS_SPECIAL_VARNO(var->varno)) || (! AttrNumberIsForUserDefinedAttr(var->varattno)))
+ return false;
+
+ /*
+ * If it's not a "<" or ">" or "=" operator, just ignore the
+ * clause. Otherwise note the relid and attnum for the variable.
+ * This uses the function for estimating selectivity, ont the
+ * operator directly (a bit awkward, but well ...).
+ */
+ switch (get_oprrest(expr->opno))
+ {
+ case F_SCALARLTSEL:
+ case F_SCALARGTSEL:
+ case F_EQSEL:
+ *varno = var->varno;
+ *attnums = bms_add_member(*attnums, var->varattno);
+ return true;
+ }
+ }
+ }
+ }
+
+ return false;
+
+}
+
+/*
+ * Estimate selectivity of clauses using a MCV list.
+ *
+ * If there's no MCV list for the stats, the function returns 0.0.
+ *
+ * While computing the estimate, the function checks whether all the
+ * columns were matched with an equality condition. If that's the case,
+ * it's assumed we can skip computing the estimate from histogram,
+ * because all the rows matching the condition are represented by the
+ * MCV item.
+ *
+ * The function also returns the frequency of the least frequent item
+ * on the MCV list, which may be useful for clamping estimate from the
+ * histogram.
+ */
+static Selectivity
+clauselist_mv_selectivity_mcvlist(PlannerInfo *root, List *clauses,
+ MVStats mvstats, bool *fullmatch,
+ Selectivity *lowsel)
+{
+ int i;
+ Selectivity s = 0.0;
+ ListCell * l;
+ char * mcvitems = NULL;
+ MCVList mcvlist = NULL;
+
+ Bitmapset *matches = NULL; /* attributes with equality matches */
+
+ /* there's no MCV list yet */
+ if (! mvstats->mcv_built)
+ return 0.0;
+
+ mcvlist = deserialize_mv_mcvlist(fetch_mv_mcvlist(mvstats->mvoid));
+
+ Assert(mcvlist != NULL);
+ Assert (clauses != NIL);
+ Assert (list_length(clauses) >= 2);
+
+ mcvitems = palloc0(sizeof(char) * mcvlist->nitems);
+ memset(mcvitems, MVSTATS_MATCH_FULL, sizeof(char)*mcvlist->nitems);
+
+ /* no match here */
+ *lowsel = 1.0;
+
+ /* loop through the list of MV-compatible clauses and do the estimation */
+ foreach (l, clauses)
+ {
+ Node * clause = (Node*)lfirst(l);
+ OpExpr * expr = (OpExpr*)clause;
+ bool varonleft = true;
+ bool ok;
+
+ /* operator */
+ FmgrInfo opproc;
+
+ fmgr_info(get_opcode(expr->opno), &opproc);
+
+ ok = (NumRelids(clause) == 1) &&
+ (is_pseudo_constant_clause(lsecond(expr->args)) ||
+ (varonleft = false,
+ is_pseudo_constant_clause(linitial(expr->args))));
+
+ if (ok)
+ {
+
+ FmgrInfo ltproc, gtproc;
+ Var * var = (varonleft) ? linitial(expr->args) : lsecond(expr->args);
+ Const * cst = (varonleft) ? lsecond(expr->args) : linitial(expr->args);
+ bool isgt = (! varonleft);
+
+ /*
+ * TODO Fetch only when really needed (probably for equality only)
+ * TODO Technically either lt/gt is sufficient.
+ *
+ * FIXME The code in analyze.c creates histograms only for types
+ * with enough ordering (by calling get_sort_group_operators).
+ * Is this the same assumption, i.e. are we certain that we
+ * get the ltproc/gtproc every time we ask? Or are there types
+ * where get_sort_group_operators returns ltopr and here we
+ * get nothing?
+ */
+ TypeCacheEntry *typecache = lookup_type_cache(var->vartype, TYPECACHE_EQ_OPR | TYPECACHE_LT_OPR | TYPECACHE_GT_OPR);
+
+ /* FIXME proper matching attribute to dimension */
+ int idx = mv_get_index(var->varattno, mvstats->stakeys);
+
+ fmgr_info(get_opcode(typecache->lt_opr), <proc);
+ fmgr_info(get_opcode(typecache->gt_opr), >proc);
+
+ /* process the MCV list first */
+ for (i = 0; i < mcvlist->nitems; i++)
+ {
+ bool tmp;
+ MCVItem item = mcvlist->items[i];
+
+ /* find the lowest selectivity in the MCV */
+ if (item->frequency < *lowsel)
+ *lowsel = item->frequency;
+
+ /* skip MCV items already ruled out */
+ if (mcvitems[i] == MVSTATS_MATCH_NONE)
+ continue;
+
+ /* TODO consider bsearch here (list is sorted by values)
+ * TODO handle other operators too (LT, GT)
+ * TODO identify "full match" when the clauses fully
+ * match the whole MCV list (so that checking the
+ * histogram is not needed)
+ */
+ if (get_oprrest(expr->opno) == F_EQSEL)
+ {
+ /*
+ * We don't care about isgt in equality, because it does not matter
+ * whether it's (var = const) or (const = var).
+ */
+ if (memcmp(&cst->constvalue, &item->values[idx], sizeof(Datum)) != 0)
+ mcvitems[i] = MVSTATS_MATCH_NONE;
+ else
+ matches = bms_add_member(matches, idx);
+ }
+ else if (get_oprrest(expr->opno) == F_SCALARLTSEL) /* column < constant */
+ {
+
+ if (! isgt) /* (var < const) */
+ {
+ /*
+ * First check whether the constant is below the lower boundary (in that
+ * case we can skip the bucket, because there's no overlap).
+ */
+ tmp = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ cst->constvalue,
+ item->values[idx]));
+
+ if (tmp)
+ {
+ mcvitems[i] = MVSTATS_MATCH_NONE; /* no match */
+ continue;
+ }
+
+ } /* (get_oprrest(expr->opno) == F_SCALARLTSEL) */
+ else /* (const < var) */
+ {
+ /*
+ * First check whether the constant is above the upper boundary (in that
+ * case we can skip the bucket, because there's no overlap).
+ */
+ tmp = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ item->values[idx],
+ cst->constvalue));
+ if (tmp)
+ {
+ mcvitems[i] = MVSTATS_MATCH_NONE; /* no match */
+ continue;
+ }
+ }
+ }
+ else if (get_oprrest(expr->opno) == F_SCALARGTSEL) /* column > constant */
+ {
+
+ if (! isgt) /* (var > const) */
+ {
+ /*
+ * First check whether the constant is above the upper boundary (in that
+ * case we can skip the bucket, because there's no overlap).
+ */
+ tmp = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ cst->constvalue,
+ item->values[idx]));
+ if (tmp)
+ {
+ mcvitems[i] = MVSTATS_MATCH_NONE; /* no match */
+ continue;
+ }
+
+ }
+ else /* (const > var) */
+ {
+ /*
+ * First check whether the constant is below the lower boundary (in
+ * that case we can skip the bucket, because there's no overlap).
+ */
+ tmp = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ item->values[idx],
+ cst->constvalue));
+ if (tmp)
+ {
+ mcvitems[i] = MVSTATS_MATCH_NONE; /* no match */
+ continue;
+ }
+ }
+
+ } /* (get_oprrest(expr->opno) == F_SCALARGTSEL) */
+
+ }
+ }
+ }
+
+ for (i = 0; i < mcvlist->nitems; i++)
+ {
+ if (mcvitems[i] != MVSTATS_MATCH_NONE)
+ s += mcvlist->items[i]->frequency;
+ }
+
+ *fullmatch = (bms_num_members(matches) == mcvlist->ndimensions);
+
+ pfree(mcvitems);
+ pfree(mcvlist);
+
+ return s;
+}
+
+/*
+ * Estimate selectivity of clauses using a histogram.
+ *
+ * If there's no histogram list for the stats, the function returns 0.0.
+ */
+static Selectivity
+clauselist_mv_selectivity_histogram(PlannerInfo *root, List *clauses,
+ MVStats mvstats)
+{
+ int i;
+ Selectivity s = 0.0;
+ ListCell * l;
+ char *buckets = NULL;
+ MVHistogram mvhist = NULL;
+
+ /* there's no histogram */
+ if (! mvstats->hist_built)
+ return 0.0;
+
+ /* There may be no histogram in the stats (check hist_built flag) */
+ mvhist = deserialize_mv_histogram(fetch_mv_histogram(mvstats->mvoid));
+
+ Assert (mvhist != NULL);
+ Assert (clauses != NIL);
+ Assert (list_length(clauses) >= 2);
+
+ /*
+ * Bitmap of bucket matches (mismatch, partial, full). by default
+ * all buckets fully match (and we'll eliminate them).
+ */
+ buckets = palloc0(sizeof(char) * mvhist->nbuckets);
+ memset(buckets, MVSTATS_MATCH_FULL, sizeof(char)*mvhist->nbuckets);
+
+ /* loop through the clauses and do the estimation */
+ foreach (l, clauses)
+ {
+ Node * clause = (Node*)lfirst(l);
+ OpExpr * expr = (OpExpr*)clause;
+ bool varonleft = true;
+ bool ok;
+
+ FmgrInfo opproc; /* operator */
+ fmgr_info(get_opcode(expr->opno), &opproc);
+
+ ok = (NumRelids(clause) == 1) &&
+ (is_pseudo_constant_clause(lsecond(expr->args)) ||
+ (varonleft = false,
+ is_pseudo_constant_clause(linitial(expr->args))));
+
+ if (ok)
+ {
+ FmgrInfo ltproc;
+ Var * var = (varonleft) ? linitial(expr->args) : lsecond(expr->args);
+ Const * cst = (varonleft) ? lsecond(expr->args) : linitial(expr->args);
+ bool isgt = (! varonleft);
+
+ /*
+ * TODO Fetch only when really needed (probably for equality only)
+ *
+ * TODO Technically either lt/gt is sufficient.
+ *
+ * FIXME The code in analyze.c creates histograms only for types
+ * with enough ordering (by calling get_sort_group_operators).
+ * Is this the same assumption, i.e. are we certain that we
+ * get the ltproc/gtproc every time we ask? Or are there types
+ * where get_sort_group_operators returns ltopr and here we
+ * get nothing?
+ */
+ TypeCacheEntry *typecache
+ = lookup_type_cache(var->vartype, TYPECACHE_EQ_OPR | TYPECACHE_LT_OPR
+ | TYPECACHE_GT_OPR);
+
+ /* lookup dimension for the attribute */
+ int idx = mv_get_index(var->varattno, mvstats->stakeys);
+
+ fmgr_info(get_opcode(typecache->lt_opr), <proc);
+
+ /*
+ * Check this for all buckets that still have "true" in the bitmap
+ *
+ * We already know the clauses use suitable operators (because that's
+ * how we filtered them).
+ */
+ for (i = 0; i < mvhist->nbuckets; i++)
+ {
+ bool tmp;
+ MVBucket bucket = mvhist->buckets[i];
+
+ /*
+ * Skip buckets that were already eliminated - this is impotant
+ * considering how we update the info (we only lower the match)
+ */
+ if (buckets[i] == MVSTATS_MATCH_NONE)
+ continue;
+
+ /*
+ * If it's not a "<" or ">" or "=" operator, just ignore the
+ * clause. Otherwise note the relid and attnum for the variable.
+ *
+ * TODO I'm really unsure the handling of 'isgt' flag (that is, clauses
+ * with reverse order of variable/constant) is correct. I wouldn't
+ * be surprised if there was some mixup. Using the lt/gt operators
+ * instead of messing with the opproc could make it simpler.
+ * It would however be using a different operator than the query,
+ * although it's not any shadier than using the selectivity function
+ * as is done currently.
+ *
+ * FIXME Once the min/max values are deduplicated, we can easily minimize
+ * the number of calls to the comparator (assuming we keep the
+ * deduplicated structure). See the note on compression at MVBucket
+ * serialize/deserialize methods.
+ */
+ switch (get_oprrest(expr->opno))
+ {
+ case F_SCALARLTSEL: /* column < constant */
+
+ if (! isgt) /* (var < const) */
+ {
+ /*
+ * First check whether the constant is below the lower boundary (in that
+ * case we can skip the bucket, because there's no overlap).
+ */
+ tmp = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ cst->constvalue,
+ bucket->min[idx]));
+ if (tmp)
+ {
+ buckets[i] = MVSTATS_MATCH_NONE; /* no match */
+ continue;
+ }
+
+ /*
+ * Now check whether the upper boundary is below the constant (in that
+ * case it's a partial match).
+ */
+ tmp = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ cst->constvalue,
+ bucket->max[idx]));
+
+ if (tmp)
+ buckets[i] = MVSTATS_MATCH_PARTIAL; /* partial match */
+ }
+ else /* (const < var) */
+ {
+ /*
+ * First check whether the constant is above the upper boundary (in that
+ * case we can skip the bucket, because there's no overlap).
+ */
+ tmp = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ bucket->max[idx],
+ cst->constvalue));
+ if (tmp)
+ {
+ buckets[i] = MVSTATS_MATCH_NONE; /* no match */
+ continue;
+ }
+
+ /*
+ * Now check whether the lower boundary is below the constant (in that
+ * case it's a partial match).
+ */
+ tmp = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ bucket->min[idx],
+ cst->constvalue));
+
+ if (tmp)
+ buckets[i] = MVSTATS_MATCH_PARTIAL; /* partial match */
+ }
+ break;
+
+ case F_SCALARGTSEL: /* column > constant */
+
+ if (! isgt) /* (var > const) */
+ {
+ /*
+ * First check whether the constant is above the upper boundary (in that
+ * case we can skip the bucket, because there's no overlap).
+ */
+ tmp = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ cst->constvalue,
+ bucket->max[idx]));
+ if (tmp)
+ {
+ buckets[i] = MVSTATS_MATCH_NONE; /* no match */
+ continue;
+ }
+
+ /*
+ * Now check whether the lower boundary is below the constant (in that
+ * case it's a partial match).
+ */
+ tmp = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ cst->constvalue,
+ bucket->min[idx]));
+
+ if (tmp)
+ buckets[i] = MVSTATS_MATCH_PARTIAL; /* partial match */
+ }
+ else /* (const > var) */
+ {
+ /*
+ * First check whether the constant is below the lower boundary (in
+ * that case we can skip the bucket, because there's no overlap).
+ */
+ tmp = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ bucket->min[idx],
+ cst->constvalue));
+ if (tmp)
+ {
+ buckets[i] = MVSTATS_MATCH_NONE; /* no match */
+ continue;
+ }
+
+ /*
+ * Now check whether the upper boundary is below the constant (in that
+ * case it's a partial match).
+ */
+ tmp = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ bucket->max[idx],
+ cst->constvalue));
+
+ if (tmp)
+ buckets[i] = MVSTATS_MATCH_PARTIAL; /* partial match */
+ }
+
+ break;
+
+ case F_EQSEL:
+
+ /*
+ * We only check whether the value is within the bucket, using the lt/gt
+ * operators fetched from type cache.
+ *
+ * TODO We'll use the default 50% estimate, but that's probably way off
+ * if there are multiple distinct values. Consider tweaking this a
+ * somehow, e.g. using only a part inversely proportional to the
+ * estimated number of distinct values in the bucket.
+ *
+ * TODO This does not handle inclusion flags at the moment, thus counting
+ * some buckets twice (when hitting the boundary).
+ *
+ * TODO Optimization is that if max[i] == min[i], it's effectively a MCV
+ * item and we can count the whole bucket as a complete match (thus
+ * using 100% bucket selectivity and not just 50%).
+ *
+ * TODO Technically some buckets may "degenerate" into single-value
+ * buckets (not necessarily for all the dimensions) - maybe this
+ * is better than keeping a separate MCV list (multi-dimensional).
+ * Update: Actually, that's unlikely to be better than a separate
+ * MCV list for two reasons - first, it requires ~2x the space
+ * (because of storing lower/upper boundaries) and second because
+ * the buckets are ranges - depending on the partitioning algorithm
+ * it may not even degenerate into (min=max) bucket. For example the
+ * the current partitioning algorithm never does that.
+ */
+ tmp = DatumGetBool(FunctionCall2Coll(<proc,
+ DEFAULT_COLLATION_OID,
+ cst->constvalue,
+ bucket->min[idx]));
+
+ if (tmp)
+ {
+ buckets[i] = MVSTATS_MATCH_NONE; /* constvalue < min */
+ continue;
+ }
+
+ tmp = DatumGetBool(FunctionCall2Coll(<proc,
+ DEFAULT_COLLATION_OID,
+ bucket->max[idx],
+ cst->constvalue));
+
+ if (tmp)
+ {
+ buckets[i] = MVSTATS_MATCH_NONE; /* constvalue > max */
+ continue;
+ }
+
+ /* partial match */
+ buckets[i] = MVSTATS_MATCH_PARTIAL;
+
+ break;
+ }
+ }
+ }
+ }
+
+ /* now, walk through the buckets and sum the selectivities */
+ for (i = 0; i < mvhist->nbuckets; i++)
+ {
+ if (buckets[i] == MVSTATS_MATCH_FULL)
+ s += mvhist->buckets[i]->ntuples;
+ else if (buckets[i] == MVSTATS_MATCH_PARTIAL)
+ s += 0.5 * mvhist->buckets[i]->ntuples;
+ }
+
+ return s;
+}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index bd180e7..d725ae0 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -366,6 +366,13 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
create_generic_options alter_generic_options
relation_expr_list dostmt_opt_list
+%type <list> OptStatsOptions
+%type <str> stats_options_name
+%type <node> stats_options_arg
+%type <defelt> stats_options_elem
+%type <list> stats_options_list
+
+
%type <list> opt_fdw_options fdw_options
%type <defelt> fdw_option
@@ -484,7 +491,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <keyword> unreserved_keyword type_func_name_keyword
%type <keyword> col_name_keyword reserved_keyword
-%type <node> TableConstraint TableLikeClause
+%type <node> TableConstraint TableLikeClause TableStatistics
%type <ival> TableLikeOptionList TableLikeOption
%type <list> ColQualList
%type <node> ColConstraint ColConstraintElem ConstraintAttr
@@ -2312,6 +2319,14 @@ alter_table_cmd:
n->subtype = AT_DisableRowSecurity;
$$ = (Node *)n;
}
+ /* ALTER TABLE <name> ADD STATISTICS (options) ON (columns) ... */
+ | ADD_P TableStatistics
+ {
+ AlterTableCmd *n = makeNode(AlterTableCmd);
+ n->subtype = AT_AddStatistics;
+ n->def = $2;
+ $$ = (Node *)n;
+ }
| alter_generic_options
{
AlterTableCmd *n = makeNode(AlterTableCmd);
@@ -3382,6 +3397,56 @@ OptConsTableSpace: USING INDEX TABLESPACE name { $$ = $4; }
ExistingIndex: USING INDEX index_name { $$ = $3; }
;
+/*****************************************************************************
+ *
+ * QUERY :
+ * ALTER TABLE relname ADD STATISTICS (columns) WITH (options)
+ *
+ *****************************************************************************/
+
+TableStatistics:
+ STATISTICS OptStatsOptions ON '(' columnList ')'
+ {
+ StatisticsDef *n = makeNode(StatisticsDef);
+ n->keys = $5;
+ n->options = $2;
+ $$ = (Node *) n;
+ }
+ ;
+
+OptStatsOptions:
+ '(' stats_options_list ')' { $$ = $2; }
+ | /*EMPTY*/ { $$ = NIL; }
+ ;
+
+stats_options_list:
+ stats_options_elem
+ {
+ $$ = list_make1($1);
+ }
+ | stats_options_list ',' stats_options_elem
+ {
+ $$ = lappend($1, $3);
+ }
+ ;
+
+stats_options_elem:
+ stats_options_name stats_options_arg
+ {
+ $$ = makeDefElem($1, $2);
+ }
+ ;
+
+stats_options_name:
+ NonReservedWord { $$ = $1; }
+ ;
+
+stats_options_arg:
+ opt_boolean_or_string { $$ = (Node *) makeString($1); }
+ | NumericOnly { $$ = (Node *) $1; }
+ | /* EMPTY */ { $$ = NULL; }
+ ;
+
/*****************************************************************************
*
diff --git a/src/backend/utils/cache/syscache.c b/src/backend/utils/cache/syscache.c
index 94d951c..ec90773 100644
--- a/src/backend/utils/cache/syscache.c
+++ b/src/backend/utils/cache/syscache.c
@@ -43,6 +43,7 @@
#include "catalog/pg_foreign_server.h"
#include "catalog/pg_foreign_table.h"
#include "catalog/pg_language.h"
+#include "catalog/pg_mv_statistic.h"
#include "catalog/pg_namespace.h"
#include "catalog/pg_opclass.h"
#include "catalog/pg_operator.h"
@@ -499,6 +500,17 @@ static const struct cachedesc cacheinfo[] = {
},
4
},
+ {MvStatisticRelationId, /* MVSTATOID */
+ MvStatisticOidIndexId,
+ 1,
+ {
+ ObjectIdAttributeNumber,
+ 0,
+ 0,
+ 0
+ },
+ 128
+ },
{NamespaceRelationId, /* NAMESPACENAME */
NamespaceNameIndexId,
1,
diff --git a/src/include/catalog/indexing.h b/src/include/catalog/indexing.h
index 870692c..d2266c0 100644
--- a/src/include/catalog/indexing.h
+++ b/src/include/catalog/indexing.h
@@ -173,6 +173,11 @@ DECLARE_UNIQUE_INDEX(pg_largeobject_loid_pn_index, 2683, on pg_largeobject using
DECLARE_UNIQUE_INDEX(pg_largeobject_metadata_oid_index, 2996, on pg_largeobject_metadata using btree(oid oid_ops));
#define LargeObjectMetadataOidIndexId 2996
+DECLARE_UNIQUE_INDEX(pg_mv_statistic_oid_index, 3259, on pg_mv_statistic using btree(oid oid_ops));
+#define MvStatisticOidIndexId 3259
+DECLARE_INDEX(pg_mv_statistic_relid_index, 3264, on pg_mv_statistic using btree(starelid oid_ops));
+#define MvStatisticRelidIndexId 3264
+
DECLARE_UNIQUE_INDEX(pg_namespace_nspname_index, 2684, on pg_namespace using btree(nspname name_ops));
#define NamespaceNameIndexId 2684
DECLARE_UNIQUE_INDEX(pg_namespace_oid_index, 2685, on pg_namespace using btree(oid oid_ops));
diff --git a/src/include/catalog/pg_mv_statistic.h b/src/include/catalog/pg_mv_statistic.h
new file mode 100644
index 0000000..d725957
--- /dev/null
+++ b/src/include/catalog/pg_mv_statistic.h
@@ -0,0 +1,89 @@
+/*-------------------------------------------------------------------------
+ *
+ * pg_mv_statistic.h
+ * definition of the system "multivariate statistic" relation (pg_mv_statistic)
+ * along with the relation's initial contents.
+ *
+ *
+ * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/catalog/pg_mv_statistic.h
+ *
+ * NOTES
+ * the genbki.pl script reads this file and generates .bki
+ * information from the DATA() statements.
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef PG_MV_STATISTIC_H
+#define PG_MV_STATISTIC_H
+
+#include "catalog/genbki.h"
+
+/* ----------------
+ * pg_mv_statistic definition. cpp turns this into
+ * typedef struct FormData_pg_mv_statistic
+ * ----------------
+ */
+#define MvStatisticRelationId 3260
+
+CATALOG(pg_mv_statistic,3260)
+{
+ /* These fields form the unique key for the entry: */
+ Oid starelid; /* relation containing attributes */
+
+ /* statistics requested to build */
+ bool hist_enabled; /* build histogram? */
+ bool mcv_enabled; /* build MCV list? */
+ bool mcv_hashed; /* build hashed MCV? */
+ bool assoc_enabled; /* analyze associations? */
+
+ /* histogram / MCV size */
+ int32 hist_max_buckets; /* max buckets */
+ int32 mcv_max_items; /* max MCV items */
+
+ /* statistics that are available (if requested) */
+ bool hist_built; /* histogram was built */
+ bool mcv_built; /* MCV list was built */
+ bool assoc_built; /* associations were built */
+
+ /* variable-length fields start here, but we allow direct access to stakeys */
+ int2vector stakeys; /* array of column keys */
+
+#ifdef CATALOG_VARLEN
+ bytea staassoc; /* association rules (serialized) */
+ bytea stamcv; /* MCV list (serialized) */
+ bytea stahist; /* MV histogram (serialized) */
+#endif
+
+} FormData_pg_mv_statistic;
+
+/* ----------------
+ * Form_pg_mv_statistic corresponds to a pointer to a tuple with
+ * the format of pg_mv_statistic relation.
+ * ----------------
+ */
+typedef FormData_pg_mv_statistic *Form_pg_mv_statistic;
+
+/* ----------------
+ * compiler constants for pg_attrdef
+ * ----------------
+ */
+#define Natts_pg_mv_statistic 14
+#define Anum_pg_mv_statistic_starelid 1
+#define Anum_pg_mv_statistic_hist_enabled 2
+#define Anum_pg_mv_statistic_mcv_enabled 3
+#define Anum_pg_mv_statistic_mcv_hashed 4
+#define Anum_pg_mv_statistic_assoc_enabled 5
+#define Anum_pg_mv_statistic_hist_max_buckets 6
+#define Anum_pg_mv_statistic_mcv_max_items 7
+#define Anum_pg_mv_statistic_hist_built 8
+#define Anum_pg_mv_statistic_mcv_built 9
+#define Anum_pg_mv_statistic_assoc_built 10
+#define Anum_pg_mv_statistic_stakeys 11
+#define Anum_pg_mv_statistic_staassoc 12
+#define Anum_pg_mv_statistic_stamcv 13
+#define Anum_pg_mv_statistic_stahist 14
+
+#endif /* PG_MV_STATISTIC_H */
diff --git a/src/include/catalog/pg_proc.h b/src/include/catalog/pg_proc.h
index 497e652..c3c03b6 100644
--- a/src/include/catalog/pg_proc.h
+++ b/src/include/catalog/pg_proc.h
@@ -2676,6 +2676,13 @@ DESCR("current user privilege on any column by rel name");
DATA(insert OID = 3029 ( has_any_column_privilege PGNSP PGUID 12 10 0 0 0 f f f f t f s 2 0 16 "26 25" _null_ _null_ _null_ _null_ has_any_column_privilege_id _null_ _null_ _null_ ));
DESCR("current user privilege on any column by rel oid");
+DATA(insert OID = 3261 ( pg_mv_stats_histogram_info PGNSP PGUID 12 1 0 0 0 f f f f t f i 1 0 25 "17" _null_ _null_ _null_ _null_ pg_mv_stats_histogram_info _null_ _null_ _null_ ));
+DESCR("multi-variate statistics: histogram info");
+DATA(insert OID = 3262 ( pg_mv_stats_mvclist_info PGNSP PGUID 12 1 0 0 0 f f f f t f i 1 0 25 "17" _null_ _null_ _null_ _null_ pg_mv_stats_mvclist_info _null_ _null_ _null_ ));
+DESCR("multi-variate statistics: MCV list info");
+DATA(insert OID = 3263 ( pg_mv_stats_histogram_gnuplot PGNSP PGUID 12 1 0 0 0 f f f f t f i 1 0 25 "17" _null_ _null_ _null_ _null_ pg_mv_stats_histogram_gnuplot _null_ _null_ _null_ ));
+DESCR("multi-variate statistics: 2D histogram gnuplot");
+
DATA(insert OID = 1928 ( pg_stat_get_numscans PGNSP PGUID 12 1 0 0 0 f f f f t f s 1 0 20 "26" _null_ _null_ _null_ _null_ pg_stat_get_numscans _null_ _null_ _null_ ));
DESCR("statistics: number of scans done for table/index");
DATA(insert OID = 1929 ( pg_stat_get_tuples_returned PGNSP PGUID 12 1 0 0 0 f f f f t f s 1 0 20 "26" _null_ _null_ _null_ _null_ pg_stat_get_tuples_returned _null_ _null_ _null_ ));
diff --git a/src/include/catalog/toasting.h b/src/include/catalog/toasting.h
index a4af551..c7839c0 100644
--- a/src/include/catalog/toasting.h
+++ b/src/include/catalog/toasting.h
@@ -49,6 +49,7 @@ extern void BootstrapToastTable(char *relName,
DECLARE_TOAST(pg_attrdef, 2830, 2831);
DECLARE_TOAST(pg_constraint, 2832, 2833);
DECLARE_TOAST(pg_description, 2834, 2835);
+DECLARE_TOAST(pg_mv_statistic, 3265, 3954);
DECLARE_TOAST(pg_proc, 2836, 2837);
DECLARE_TOAST(pg_rewrite, 2838, 2839);
DECLARE_TOAST(pg_seclabel, 3598, 3599);
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index bc71fea..b916edd 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -413,6 +413,7 @@ typedef enum NodeTag
T_XmlSerialize,
T_WithClause,
T_CommonTableExpr,
+ T_StatisticsDef,
/*
* TAGS FOR REPLICATION GRAMMAR PARSE NODES (replnodes.h)
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 3e4f815..c3e458a 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -543,6 +543,14 @@ typedef struct ColumnDef
int location; /* parse location, or -1 if none/unknown */
} ColumnDef;
+typedef struct StatisticsDef
+{
+ NodeTag type;
+ List *keys; /* String nodes naming referenced column(s) */
+ List *options; /* list of DefElem nodes */
+} StatisticsDef;
+
+
/*
* TableLikeClause - CREATE TABLE ( ... LIKE ... ) clause
*/
@@ -1338,7 +1346,8 @@ typedef enum AlterTableType
AT_ReplicaIdentity, /* REPLICA IDENTITY */
AT_EnableRowSecurity, /* ENABLE ROW SECURITY */
AT_DisableRowSecurity, /* DISABLE ROW SECURITY */
- AT_GenericOptions /* OPTIONS (...) */
+ AT_GenericOptions, /* OPTIONS (...) */
+ AT_AddStatistics /* add statistics */
} AlterTableType;
typedef struct ReplicaIdentityStmt
diff --git a/src/include/utils/mvstats.h b/src/include/utils/mvstats.h
new file mode 100644
index 0000000..157891a
--- /dev/null
+++ b/src/include/utils/mvstats.h
@@ -0,0 +1,283 @@
+/*-------------------------------------------------------------------------
+ *
+ * mvstats.h
+ * Multivariate statistics and selectivity estimation functions.
+ *
+ *
+ * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/utils/mvstats.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef MVSTATS_H
+#define MVSTATS_H
+
+/*
+ * Multivariate statistics for planner/optimizer, implementing extensions
+ * of the single-column statistics:
+ *
+ * - multivariate MCV list
+ * - multivariate histograms
+ *
+ * There's also an experimental support for associative rules (values in
+ * one column implying values in other columns - e.g. ZIP code implies
+ * name of a city, etc.).
+ *
+ * The current implementation has various limitations:
+ *
+ * (a) it supports only data types passed by value
+ *
+ * (b) no support for NULL values
+ *
+ * Both (a) and (b) should be straightforward to fix (and usually
+ * described in comments at related data structures or functions).
+ *
+ * The stats may be built only directly on columns, not on expressions.
+ * And there are usually some additional technical limits (e.g. number
+ * of columns in a histogram, etc.).
+ *
+ * Those limits serve mostly as sanity checks and while increasing them
+ * is possible (the implementation should not break), it's expected to
+ * lead either to very bad precision or expensive planning.
+ */
+
+/*
+ * Multivariate histograms
+ *
+ * Histograms are a collection of buckets, represented by n-dimensional
+ * rectangles. Each rectangle is delimited by an array of lower and
+ * upper boundaries, so that for for the i-th attribute
+ *
+ * min[i] <= value[i] <= max[i]
+ *
+ * Each bucket tracks frequency (fraction of tuples it contains),
+ * information about the inequalities, number of distinct values in
+ * each dimension (which is used when building the histogram) etc.
+ *
+ * The boundaries may be either inclusive or exclusive, or the whole
+ * dimension may be NULL.
+ *
+ * The buckets may overlap (assuming the build algorithm keeps the
+ * frequencies additive) or may not cover the whole space (i.e. allow
+ * gaps). This entirely depends on the algorithm used to build the
+ * histogram.
+ *
+ * The histograms are marked with a 'magic' constant, mostly to make
+ * sure the bytea really is a histogram in serialized form.
+ *
+ * We do expect to support multiple histogram types, with different
+ * features etc. The 'type' field is used to identify those types.
+ * Technically some histogram types might use completely different
+ * bucket representation, but that's not expected at the moment.
+ *
+ * TODO Add pointer to 'private' data, meant for private data for
+ * other algorithms for building the histogram.
+ *
+ * TODO The current implementation does not handle NULL values (it's
+ * somehow prepared for that, but the algorithm building the
+ * histogram ignores them). The idea is to build buckets with one
+ * or more NULL-only dimensions - there'll be at most 2^ndimensions
+ * such buckets, which for 8 atttributes (current limit) is 256.
+ * That's quite reasonable, considering we expect thousands of
+ * buckets in total.
+ *
+ * TODO This structure is used both when building the histogram, and
+ * then when using it to compute estimates. That's why the last
+ * few elements are not used once the histogram is built.
+ *
+ * TODO The limit on number of buckets is quite arbitrary, aiming for
+ * sufficient accuracy while still being fast. Probably should be
+ * replaced with a dynamic limit dependent on statistics target,
+ * number of attributes (dimensions) and statistics target
+ * associated with the attributes. Also, this needs to be related
+ * to the number of sampled rows, by either clamping it to a
+ * reasonable number (after seeing the number of rows) or using
+ * it when computing the number of rows to sample. Something like
+ * 10 rows per bucket seems reasonable.
+ *
+ * TODO We may replace the bool arrays with a suitably large data type
+ * (say, uint16 or uint32) and get rid of the allocations. It's
+ * unlikely we'll ever support more than 32 columns as that'd
+ * result in poor precision, huge histograms (splitting each
+ * dimension once would mean 2^32 buckets), and very expensive
+ * estimation. MCVItem already does it this way.
+ *
+ * TODO Actually the distinct stats (both for combination of all columns
+ * and for combinations of various subsets of columns) should be
+ * moved to a separate structure (next to histogram/MCV/...) to
+ * make it useful even without a histogram computed etc.
+ */
+typedef struct MVBucketData {
+
+ /* Frequencies of this bucket. */
+ float ntuples; /* frequency of tuples tuples */
+ float ndistinct; /* frequency of distinct values */
+
+ /*
+ * Number of distinct values in each dimension. This is used when
+ * building the histogram (and is not serialized/deserialized), but
+ * it could be useful for estimating ndistinct for combinations of
+ * columns.
+ *
+ * It would mean tracking 2^N values for each bucket, and even if
+ * those values might be stores in 1B it's still a lot of space
+ * (considering the expected number of buckets).
+ *
+ * TODO Consider tracking ndistincts for all attribute combinations.
+ */
+ uint32 *ndistincts;
+
+ /*
+ * Information about dimensions being NULL-only. Not yet used.
+ */
+ bool *nullsonly;
+
+ /* lower boundaries - values and information about the inequalities */
+ Datum *min;
+ bool *min_inclusive;
+
+ /* upper boundaries - values and information about the inequalities */
+ Datum *max;
+ bool *max_inclusive;
+
+ /*
+ * Sample tuples falling into this bucket, index of the dimension
+ * the bucket was split by in the last step.
+ *
+ * XXX These fields are needed only while building the histogram,
+ * and are not serialized at all.
+ */
+ HeapTuple *rows;
+ uint32 numrows;
+ int last_split_dimension;
+
+} MVBucketData;
+
+typedef MVBucketData *MVBucket;
+
+
+typedef struct MVHistogramData {
+
+ uint32 magic; /* magic constant marker */
+ uint32 type; /* type of histogram (BASIC) */
+ uint32 nbuckets; /* number of buckets (buckets array) */
+ uint32 ndimensions; /* number of dimensions */
+
+ MVBucket *buckets; /* array of buckets */
+
+} MVHistogramData;
+
+typedef MVHistogramData *MVHistogram;
+
+
+/* used to flag stats serialized to bytea */
+#define MVHIST_MAGIC 0x7F8C5670 /* marks serialized bytea */
+#define MVHIST_TYPE_BASIC 1 /* basic histogram type */
+
+/* limits (mostly sanity check, may be relaxed in the future) */
+#define MVHIST_MAX_BUCKETS 16384 /* max number of buckets */
+
+/* bucket size in a serialized form */
+#define BUCKET_SIZE_SERIALIZED(ndims) \
+ (offsetof(MVBucketData, ndistincts) + \
+ (ndims) * (2 * sizeof(uint16) + sizeof(uint32) + 3 * sizeof(bool)))
+
+
+/*
+ * Multivariate MCV (most-common value) lists
+ *
+ * A straight-forward extension of MCV items - i.e. a list (array) of
+ * combinations of attribute values, together with a frequency and
+ * null flags.
+ *
+ * This already uses the trick with using uint32 as a null bitmap.
+ *
+ * TODO Shouldn't the MCVItemData use plain pointer for values, instead
+ * of the single-item array trick?
+ *
+ * TODO It's possible to build a special case of MCV list, storing not
+ * the actual values but only 32/64-bit hash. This is only useful
+ * for estimating equality clauses and for large varlena types.
+ */
+typedef struct MCVItemData {
+ double frequency; /* frequency of this combination */
+ uint32 nulls; /* lags of NULL values (up to 32 columns) */
+ Datum values[1]; /* variable-length (ndimensions) */
+} MCVItemData;
+
+typedef MCVItemData *MCVItem;
+
+/* multivariate MCV list - essentally an array of MCV items */
+typedef struct MCVListData {
+ uint32 magic; /* magic constant marker */
+ uint32 type; /* type of MCV list (BASIC) */
+ uint32 ndimensions; /* number of dimensions */
+ uint32 nitems; /* number of MCV items in the array */
+ MCVItem *items; /* array of MCV items */
+} MCVListData;
+
+typedef MCVListData *MCVList;
+
+/* used to flag stats serialized to bytea */
+#define MVSTAT_MCV_MAGIC 0xE1A651C2 /* marks serialized bytea */
+#define MVSTAT_MCV_TYPE_BASIC 1 /* basic MCV list type */
+
+/* TODO consider increasing the limit, and/or using statistics target */
+#define MVSTAT_MCVLIST_MAX_ITEMS 1024 /* max items in MCV list */
+
+
+/*
+ * Basic info about the stats, used when choosing what to use
+ *
+ * TODO Add info about what statistics is available (histogram, MCV,
+ * hashed MCV, assciative rules).
+ */
+typedef struct MVStatsData {
+ Oid mvoid; /* OID of the stats in pg_mv_statistic */
+ int2vector *stakeys; /* attnums for columns in the stats */
+ bool hist_built; /* histogram is already available */
+ bool mcv_built; /* MCV list is already available */
+ bool assoc_built; /* associative rules available */
+} MVStatsData;
+
+typedef struct MVStatsData *MVStats;
+
+
+/*
+ * Degree of how much MCV item / histogram bucket matches a clause.
+ * This is then considered when computing the selectivity.
+ */
+#define MVSTATS_MATCH_NONE 0 /* no match at all */
+#define MVSTATS_MATCH_PARTIAL 1 /* partial match */
+#define MVSTATS_MATCH_FULL 2 /* full match */
+
+
+#define MVSTATS_MAX_DIMENSIONS 8 /* max number of attributes */
+
+/*
+ * TODO Maybe fetching the histogram/MCV list separately is inefficient?
+ * Consider adding a single `fetch_stats` method, fetching all
+ * stats specified using flags (or something like that).
+ */
+MVStats list_mv_stats(Oid relid, int *nstats, bool built_only);
+bytea * fetch_mv_histogram(Oid mvoid);
+bytea * fetch_mv_mcvlist(Oid mvoid);
+
+/* deserialization of stats (serialization is private to analyze) */
+MVHistogram deserialize_mv_histogram(bytea * data);
+MCVList deserialize_mv_mcvlist(bytea * data);
+
+/*
+ * Returns index of the attribute number within the vector (i.e. a
+ * dimension within the stats).
+ */
+int mv_get_index(AttrNumber varattno, int2vector * stakeys);
+
+/* FIXME this probably belongs somewhere else (not to operations stats) */
+extern Datum pg_mv_stats_histogram_info(PG_FUNCTION_ARGS);
+extern Datum pg_mv_stats_histogram_gnuplot(PG_FUNCTION_ARGS);
+extern Datum pg_mv_stats_mvclist_info(PG_FUNCTION_ARGS);
+
+#endif
diff --git a/src/include/utils/syscache.h b/src/include/utils/syscache.h
index f97229f..a275bd5 100644
--- a/src/include/utils/syscache.h
+++ b/src/include/utils/syscache.h
@@ -66,6 +66,7 @@ enum SysCacheIdentifier
INDEXRELID,
LANGNAME,
LANGOID,
+ MVSTATOID,
NAMESPACENAME,
NAMESPACEOID,
OPERNAMENSP,
diff --git a/src/test/regress/regression.diffs b/src/test/regress/regression.diffs
new file mode 100644
index 0000000..179c09d
--- /dev/null
+++ b/src/test/regress/regression.diffs
@@ -0,0 +1,294 @@
+*** /home/tomas/work/postgres/src/test/regress/expected/updatable_views.out 2014-10-29 00:22:04.820171312 +0100
+--- /home/tomas/work/postgres/src/test/regress/results/updatable_views.out 2014-11-10 02:54:44.083052362 +0100
+***************
+*** 657,668 ****
+ FROM information_schema.views
+ WHERE table_name LIKE 'rw_view%'
+ ORDER BY table_name;
+! table_name | is_updatable | is_insertable_into | is_trigger_updatable | is_trigger_deletable | is_trigger_insertable_into
+! ------------+--------------+--------------------+----------------------+----------------------+----------------------------
+! rw_view1 | NO | NO | NO | NO | NO
+! rw_view2 | NO | NO | NO | NO | NO
+! (2 rows)
+!
+ SELECT table_name, column_name, is_updatable
+ FROM information_schema.columns
+ WHERE table_name LIKE 'rw_view%'
+--- 657,663 ----
+ FROM information_schema.views
+ WHERE table_name LIKE 'rw_view%'
+ ORDER BY table_name;
+! ERROR: no relation entry for relid 1880
+ SELECT table_name, column_name, is_updatable
+ FROM information_schema.columns
+ WHERE table_name LIKE 'rw_view%'
+***************
+*** 710,721 ****
+ FROM information_schema.views
+ WHERE table_name LIKE 'rw_view%'
+ ORDER BY table_name;
+! table_name | is_updatable | is_insertable_into | is_trigger_updatable | is_trigger_deletable | is_trigger_insertable_into
+! ------------+--------------+--------------------+----------------------+----------------------+----------------------------
+! rw_view1 | NO | NO | NO | NO | YES
+! rw_view2 | NO | NO | NO | NO | NO
+! (2 rows)
+!
+ SELECT table_name, column_name, is_updatable
+ FROM information_schema.columns
+ WHERE table_name LIKE 'rw_view%'
+--- 705,711 ----
+ FROM information_schema.views
+ WHERE table_name LIKE 'rw_view%'
+ ORDER BY table_name;
+! ERROR: no relation entry for relid 1880
+ SELECT table_name, column_name, is_updatable
+ FROM information_schema.columns
+ WHERE table_name LIKE 'rw_view%'
+***************
+*** 746,757 ****
+ FROM information_schema.views
+ WHERE table_name LIKE 'rw_view%'
+ ORDER BY table_name;
+! table_name | is_updatable | is_insertable_into | is_trigger_updatable | is_trigger_deletable | is_trigger_insertable_into
+! ------------+--------------+--------------------+----------------------+----------------------+----------------------------
+! rw_view1 | NO | NO | YES | NO | YES
+! rw_view2 | NO | NO | NO | NO | NO
+! (2 rows)
+!
+ SELECT table_name, column_name, is_updatable
+ FROM information_schema.columns
+ WHERE table_name LIKE 'rw_view%'
+--- 736,742 ----
+ FROM information_schema.views
+ WHERE table_name LIKE 'rw_view%'
+ ORDER BY table_name;
+! ERROR: no relation entry for relid 1880
+ SELECT table_name, column_name, is_updatable
+ FROM information_schema.columns
+ WHERE table_name LIKE 'rw_view%'
+***************
+*** 782,793 ****
+ FROM information_schema.views
+ WHERE table_name LIKE 'rw_view%'
+ ORDER BY table_name;
+! table_name | is_updatable | is_insertable_into | is_trigger_updatable | is_trigger_deletable | is_trigger_insertable_into
+! ------------+--------------+--------------------+----------------------+----------------------+----------------------------
+! rw_view1 | NO | NO | YES | YES | YES
+! rw_view2 | NO | NO | NO | NO | NO
+! (2 rows)
+!
+ SELECT table_name, column_name, is_updatable
+ FROM information_schema.columns
+ WHERE table_name LIKE 'rw_view%'
+--- 767,773 ----
+ FROM information_schema.views
+ WHERE table_name LIKE 'rw_view%'
+ ORDER BY table_name;
+! ERROR: no relation entry for relid 1880
+ SELECT table_name, column_name, is_updatable
+ FROM information_schema.columns
+ WHERE table_name LIKE 'rw_view%'
+***************
+*** 1385,1398 ****
+ Options: check_option=local
+
+ SELECT * FROM information_schema.views WHERE table_name = 'rw_view1';
+! table_catalog | table_schema | table_name | view_definition | check_option | is_updatable | is_insertable_into | is_trigger_updatable | is_trigger_deletable | is_trigger_insertable_into
+! ---------------+--------------+------------+------------------------------------+--------------+--------------+--------------------+----------------------+----------------------+----------------------------
+! regression | public | rw_view1 | SELECT base_tbl.a, +| LOCAL | YES | YES | NO | NO | NO
+! | | | base_tbl.b +| | | | | |
+! | | | FROM base_tbl +| | | | | |
+! | | | WHERE (base_tbl.a < base_tbl.b); | | | | | |
+! (1 row)
+!
+ INSERT INTO rw_view1 VALUES(3,4); -- ok
+ INSERT INTO rw_view1 VALUES(4,3); -- should fail
+ ERROR: new row violates WITH CHECK OPTION for "rw_view1"
+--- 1365,1371 ----
+ Options: check_option=local
+
+ SELECT * FROM information_schema.views WHERE table_name = 'rw_view1';
+! ERROR: no relation entry for relid 1880
+ INSERT INTO rw_view1 VALUES(3,4); -- ok
+ INSERT INTO rw_view1 VALUES(4,3); -- should fail
+ ERROR: new row violates WITH CHECK OPTION for "rw_view1"
+***************
+*** 1437,1449 ****
+ Options: check_option=cascaded
+
+ SELECT * FROM information_schema.views WHERE table_name = 'rw_view2';
+! table_catalog | table_schema | table_name | view_definition | check_option | is_updatable | is_insertable_into | is_trigger_updatable | is_trigger_deletable | is_trigger_insertable_into
+! ---------------+--------------+------------+----------------------------+--------------+--------------+--------------------+----------------------+----------------------+----------------------------
+! regression | public | rw_view2 | SELECT rw_view1.a +| CASCADED | YES | YES | NO | NO | NO
+! | | | FROM rw_view1 +| | | | | |
+! | | | WHERE (rw_view1.a < 10); | | | | | |
+! (1 row)
+!
+ INSERT INTO rw_view2 VALUES (-5); -- should fail
+ ERROR: new row violates WITH CHECK OPTION for "rw_view1"
+ DETAIL: Failing row contains (-5).
+--- 1410,1416 ----
+ Options: check_option=cascaded
+
+ SELECT * FROM information_schema.views WHERE table_name = 'rw_view2';
+! ERROR: no relation entry for relid 1880
+ INSERT INTO rw_view2 VALUES (-5); -- should fail
+ ERROR: new row violates WITH CHECK OPTION for "rw_view1"
+ DETAIL: Failing row contains (-5).
+***************
+*** 1477,1489 ****
+ Options: check_option=local
+
+ SELECT * FROM information_schema.views WHERE table_name = 'rw_view2';
+! table_catalog | table_schema | table_name | view_definition | check_option | is_updatable | is_insertable_into | is_trigger_updatable | is_trigger_deletable | is_trigger_insertable_into
+! ---------------+--------------+------------+----------------------------+--------------+--------------+--------------------+----------------------+----------------------+----------------------------
+! regression | public | rw_view2 | SELECT rw_view1.a +| LOCAL | YES | YES | NO | NO | NO
+! | | | FROM rw_view1 +| | | | | |
+! | | | WHERE (rw_view1.a < 10); | | | | | |
+! (1 row)
+!
+ INSERT INTO rw_view2 VALUES (-10); -- ok, but not in view
+ INSERT INTO rw_view2 VALUES (20); -- should fail
+ ERROR: new row violates WITH CHECK OPTION for "rw_view2"
+--- 1444,1450 ----
+ Options: check_option=local
+
+ SELECT * FROM information_schema.views WHERE table_name = 'rw_view2';
+! ERROR: no relation entry for relid 1880
+ INSERT INTO rw_view2 VALUES (-10); -- ok, but not in view
+ INSERT INTO rw_view2 VALUES (20); -- should fail
+ ERROR: new row violates WITH CHECK OPTION for "rw_view2"
+***************
+*** 1517,1529 ****
+ WHERE rw_view1.a < 10;
+
+ SELECT * FROM information_schema.views WHERE table_name = 'rw_view2';
+! table_catalog | table_schema | table_name | view_definition | check_option | is_updatable | is_insertable_into | is_trigger_updatable | is_trigger_deletable | is_trigger_insertable_into
+! ---------------+--------------+------------+----------------------------+--------------+--------------+--------------------+----------------------+----------------------+----------------------------
+! regression | public | rw_view2 | SELECT rw_view1.a +| NONE | YES | YES | NO | NO | NO
+! | | | FROM rw_view1 +| | | | | |
+! | | | WHERE (rw_view1.a < 10); | | | | | |
+! (1 row)
+!
+ INSERT INTO rw_view2 VALUES (30); -- ok, but not in view
+ SELECT * FROM base_tbl;
+ a
+--- 1478,1484 ----
+ WHERE rw_view1.a < 10;
+
+ SELECT * FROM information_schema.views WHERE table_name = 'rw_view2';
+! ERROR: no relation entry for relid 1880
+ INSERT INTO rw_view2 VALUES (30); -- ok, but not in view
+ SELECT * FROM base_tbl;
+ a
+***************
+*** 1543,1559 ****
+ CREATE VIEW rw_view2 AS SELECT * FROM rw_view1 WHERE a > 0;
+ CREATE VIEW rw_view3 AS SELECT * FROM rw_view2 WITH CHECK OPTION;
+ SELECT * FROM information_schema.views WHERE table_name LIKE E'rw\\_view_' ORDER BY table_name;
+! table_catalog | table_schema | table_name | view_definition | check_option | is_updatable | is_insertable_into | is_trigger_updatable | is_trigger_deletable | is_trigger_insertable_into
+! ---------------+--------------+------------+---------------------------+--------------+--------------+--------------------+----------------------+----------------------+----------------------------
+! regression | public | rw_view1 | SELECT base_tbl.a +| CASCADED | YES | YES | NO | NO | NO
+! | | | FROM base_tbl; | | | | | |
+! regression | public | rw_view2 | SELECT rw_view1.a +| NONE | YES | YES | NO | NO | NO
+! | | | FROM rw_view1 +| | | | | |
+! | | | WHERE (rw_view1.a > 0); | | | | | |
+! regression | public | rw_view3 | SELECT rw_view2.a +| CASCADED | YES | YES | NO | NO | NO
+! | | | FROM rw_view2; | | | | | |
+! (3 rows)
+!
+ INSERT INTO rw_view1 VALUES (-1); -- ok
+ INSERT INTO rw_view1 VALUES (1); -- ok
+ INSERT INTO rw_view2 VALUES (-2); -- ok, but not in view
+--- 1498,1504 ----
+ CREATE VIEW rw_view2 AS SELECT * FROM rw_view1 WHERE a > 0;
+ CREATE VIEW rw_view3 AS SELECT * FROM rw_view2 WITH CHECK OPTION;
+ SELECT * FROM information_schema.views WHERE table_name LIKE E'rw\\_view_' ORDER BY table_name;
+! ERROR: no relation entry for relid 1880
+ INSERT INTO rw_view1 VALUES (-1); -- ok
+ INSERT INTO rw_view1 VALUES (1); -- ok
+ INSERT INTO rw_view2 VALUES (-2); -- ok, but not in view
+
+======================================================================
+
+*** /home/tomas/work/postgres/src/test/regress/expected/sanity_check.out 2014-10-29 00:22:04.812171313 +0100
+--- /home/tomas/work/postgres/src/test/regress/results/sanity_check.out 2014-11-10 02:54:44.150052357 +0100
+***************
+*** 113,118 ****
+--- 113,119 ----
+ pg_language|t
+ pg_largeobject|t
+ pg_largeobject_metadata|t
++ pg_mv_statistic|t
+ pg_namespace|t
+ pg_opclass|t
+ pg_operator|t
+
+======================================================================
+
+*** /home/tomas/work/postgres/src/test/regress/expected/rowsecurity.out 2014-10-29 00:22:04.811171313 +0100
+--- /home/tomas/work/postgres/src/test/regress/results/rowsecurity.out 2014-11-10 02:54:45.775052238 +0100
+***************
+*** 901,925 ****
+ -- prepared statement with rls_regress_user0 privilege
+ PREPARE p1(int) AS SELECT * FROM t1 WHERE a <= $1;
+ EXECUTE p1(2);
+! a | b
+! ---+-----
+! 2 | bbb
+! 2 | bcd
+! 2 | yyy
+! (3 rows)
+!
+ EXPLAIN (COSTS OFF) EXECUTE p1(2);
+! QUERY PLAN
+! ----------------------------------------------
+! Append
+! -> Seq Scan on t1
+! Filter: ((a <= 2) AND ((a % 2) = 0))
+! -> Seq Scan on t2
+! Filter: ((a <= 2) AND ((a % 2) = 0))
+! -> Seq Scan on t3
+! Filter: ((a <= 2) AND ((a % 2) = 0))
+! (7 rows)
+!
+ -- superuser is allowed to bypass RLS checks
+ RESET SESSION AUTHORIZATION;
+ SET row_security TO OFF;
+--- 901,909 ----
+ -- prepared statement with rls_regress_user0 privilege
+ PREPARE p1(int) AS SELECT * FROM t1 WHERE a <= $1;
+ EXECUTE p1(2);
+! ERROR: no relation entry for relid 530
+ EXPLAIN (COSTS OFF) EXECUTE p1(2);
+! ERROR: no relation entry for relid 530
+ -- superuser is allowed to bypass RLS checks
+ RESET SESSION AUTHORIZATION;
+ SET row_security TO OFF;
+
+======================================================================
+
+*** /home/tomas/work/postgres/src/test/regress/expected/rules.out 2014-10-29 00:22:04.812171313 +0100
+--- /home/tomas/work/postgres/src/test/regress/results/rules.out 2014-11-10 02:54:48.329052050 +0100
+***************
+*** 1353,1358 ****
+--- 1353,1368 ----
+ LEFT JOIN pg_namespace n ON ((n.oid = c.relnamespace)))
+ LEFT JOIN pg_tablespace t ON ((t.oid = c.reltablespace)))
+ WHERE (c.relkind = 'm'::"char");
++ pg_mv_stats| SELECT n.nspname AS schemaname,
++ c.relname AS tablename,
++ s.stakeys AS attnums,
++ length(s.stamcv) AS mcvbytes,
++ pg_mv_stats_mvclist_info(s.stamcv) AS mcvinfo,
++ length(s.stahist) AS histbytes,
++ pg_mv_stats_histogram_info(s.stahist) AS histinfo
++ FROM ((pg_mv_statistic s
++ JOIN pg_class c ON ((c.oid = s.starelid)))
++ LEFT JOIN pg_namespace n ON ((n.oid = c.relnamespace)));
+ pg_policies| SELECT n.nspname AS schemaname,
+ c.relname AS tablename,
+ rs.rsecpolname AS policyname,
+
+======================================================================
+
diff --git a/src/test/regress/regression.out b/src/test/regress/regression.out
new file mode 100644
index 0000000..48a4a25
--- /dev/null
+++ b/src/test/regress/regression.out
@@ -0,0 +1,147 @@
+test tablespace ... ok
+test boolean ... ok
+test char ... ok
+test name ... ok
+test varchar ... ok
+test text ... ok
+test int2 ... ok
+test int4 ... ok
+test int8 ... ok
+test oid ... ok
+test float4 ... ok
+test float8 ... ok
+test bit ... ok
+test numeric ... ok
+test txid ... ok
+test uuid ... ok
+test enum ... ok
+test money ... ok
+test rangetypes ... ok
+test pg_lsn ... ok
+test regproc ... ok
+test strings ... ok
+test numerology ... ok
+test point ... ok
+test lseg ... ok
+test line ... ok
+test box ... ok
+test path ... ok
+test polygon ... ok
+test circle ... ok
+test date ... ok
+test time ... ok
+test timetz ... ok
+test timestamp ... ok
+test timestamptz ... ok
+test interval ... ok
+test abstime ... ok
+test reltime ... ok
+test tinterval ... ok
+test inet ... ok
+test macaddr ... ok
+test tstypes ... ok
+test comments ... ok
+test geometry ... ok
+test horology ... ok
+test regex ... ok
+test oidjoins ... ok
+test type_sanity ... ok
+test opr_sanity ... ok
+test insert ... ok
+test create_function_1 ... ok
+test create_type ... ok
+test create_table ... ok
+test create_function_2 ... ok
+test copy ... ok
+test copyselect ... ok
+test create_misc ... ok
+test create_operator ... ok
+test create_index ... ok
+test create_view ... ok
+test create_aggregate ... ok
+test create_function_3 ... ok
+test create_cast ... ok
+test constraints ... ok
+test triggers ... ok
+test inherit ... ok
+test create_table_like ... ok
+test typed_table ... ok
+test vacuum ... ok
+test drop_if_exists ... ok
+test updatable_views ... FAILED
+test sanity_check ... FAILED
+test errors ... ok
+test select ... ok
+test select_into ... ok
+test select_distinct ... ok
+test select_distinct_on ... ok
+test select_implicit ... ok
+test select_having ... ok
+test subselect ... ok
+test union ... ok
+test case ... ok
+test join ... ok
+test aggregates ... ok
+test transactions ... ok
+test random ... ok
+test portals ... ok
+test arrays ... ok
+test btree_index ... ok
+test hash_index ... ok
+test update ... ok
+test delete ... ok
+test namespace ... ok
+test prepared_xacts ... ok
+test privileges ... ok
+test security_label ... ok
+test collate ... ok
+test matview ... ok
+test lock ... ok
+test replica_identity ... ok
+test rowsecurity ... FAILED
+test alter_generic ... ok
+test brin ... ok
+test misc ... ok
+test psql ... ok
+test async ... ok
+test rules ... FAILED
+test event_trigger ... ok
+test select_views ... ok
+test portals_p2 ... ok
+test foreign_key ... ok
+test cluster ... ok
+test dependency ... ok
+test guc ... ok
+test bitmapops ... ok
+test combocid ... ok
+test tsearch ... ok
+test tsdicts ... ok
+test foreign_data ... ok
+test window ... ok
+test xmlmap ... ok
+test functional_deps ... ok
+test advisory_lock ... ok
+test json ... ok
+test jsonb ... ok
+test indirect_toast ... ok
+test equivclass ... ok
+test plancache ... ok
+test limit ... ok
+test plpgsql ... ok
+test copy2 ... ok
+test temp ... ok
+test domain ... ok
+test rangefuncs ... ok
+test prepare ... ok
+test without_oid ... ok
+test conversion ... ok
+test truncate ... ok
+test alter_table ... ok
+test sequence ... ok
+test polymorphism ... ok
+test rowtypes ... ok
+test returning ... ok
+test largeobject ... ok
+test with ... ok
+test xml ... ok
+test stats ... ok
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: WIP: multivariate statistics / proof of concept
2014-10-12 22:00 WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-10-29 09:41 ` Re: WIP: multivariate statistics / proof of concept David Rowley <dgrowleyml@gmail.com>
@ 2014-10-29 11:31 ` Petr Jelinek <petr@2ndquadrant.com>
2014-10-29 11:48 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
1 sibling, 1 reply; 37+ messages in thread
From: Petr Jelinek @ 2014-10-29 11:31 UTC (permalink / raw)
To: David Rowley <dgrowleyml@gmail.com>; Tomas Vondra <tv@fuzzy.cz>; +Cc: pgsql-hackers
On 29/10/14 10:41, David Rowley wrote:
> On Mon, Oct 13, 2014 at 11:00 AM, Tomas Vondra <tv@fuzzy.cz
>
> The last point is really just "unfinished implementation" - the syntax I
> propose is this:
>
> ALTER TABLE ... ADD STATISTICS (options) ON (columns)
>
> where the options influence the MCV list and histogram size, etc. The
> options are recognized and may give you an idea of what it might do, but
> it's not really used at the moment (except for storing in the
> pg_mv_statistic catalog).
>
>
>
> I've not really gotten around to looking at the patch yet, but I'm also
> wondering if it would be simple include allowing functional statistics
> too. The pg_mv_statistic name seems to indicate multi columns, but how
> about stats on date(datetime_column), or perhaps any non-volatile
> function. This would help to solve the problem highlighted here
> http://www.postgresql.org/message-id/CAApHDvp2vH=7O-gp-zAf7aWy+A-WHWVg7h3Vc6=5pf9Uf34DhQ@mail.gmail....
> . Without giving it too much thought, perhaps any expression that can be
> indexed should be allowed to have stats? Would that be really difficult
> to implement in comparison to what you've already done with the patch so
> far?
>
I would not over-complicate requirements for the first version of this,
I think it's already complicated enough.
Quick look at the patch suggests that it mainly needs discussion about
design and particular implementation choices, there is fair amount of
TODOs and FIXMEs. I'd like to look at it too but I doubt that I'll have
time to do in depth review in this CF.
--
Petr Jelinek http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: WIP: multivariate statistics / proof of concept
2014-10-12 22:00 WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-10-29 09:41 ` Re: WIP: multivariate statistics / proof of concept David Rowley <dgrowleyml@gmail.com>
2014-10-29 11:31 ` Re: WIP: multivariate statistics / proof of concept Petr Jelinek <petr@2ndquadrant.com>
@ 2014-10-29 11:48 ` Tomas Vondra <tv@fuzzy.cz>
2014-10-30 09:17 ` Re: WIP: multivariate statistics / proof of concept David Rowley <dgrowleyml@gmail.com>
0 siblings, 1 reply; 37+ messages in thread
From: Tomas Vondra @ 2014-10-29 11:48 UTC (permalink / raw)
To: Petr Jelinek <petr@2ndquadrant.com>; +Cc: David Rowley <dgrowleyml@gmail.com>; Tomas Vondra <tv@fuzzy.cz>; pgsql-hackers
Dne 29 Říjen 2014, 12:31, Petr Jelinek napsal(a):
> On 29/10/14 10:41, David Rowley wrote:
>> On Mon, Oct 13, 2014 at 11:00 AM, Tomas Vondra <tv@fuzzy.cz
>>
>> The last point is really just "unfinished implementation" - the
>> syntax I
>> propose is this:
>>
>> ALTER TABLE ... ADD STATISTICS (options) ON (columns)
>>
>> where the options influence the MCV list and histogram size, etc.
>> The
>> options are recognized and may give you an idea of what it might do,
>> but
>> it's not really used at the moment (except for storing in the
>> pg_mv_statistic catalog).
>>
>>
>>
>> I've not really gotten around to looking at the patch yet, but I'm also
>> wondering if it would be simple include allowing functional statistics
>> too. The pg_mv_statistic name seems to indicate multi columns, but how
>> about stats on date(datetime_column), or perhaps any non-volatile
>> function. This would help to solve the problem highlighted here
>> http://www.postgresql.org/message-id/CAApHDvp2vH=7O-gp-zAf7aWy+A-WHWVg7h3Vc6=5pf9Uf34DhQ@mail.gmail....
>> . Without giving it too much thought, perhaps any expression that can be
>> indexed should be allowed to have stats? Would that be really difficult
>> to implement in comparison to what you've already done with the patch so
>> far?
>>
>
> I would not over-complicate requirements for the first version of this,
> I think it's already complicated enough.
My thoughts, exactly. I'm not willing to put more features into the
initial version of the patch. Actually, I'm thinking about ripping out
some experimental features (particularly "hashed MCV" and "associative
rules").
> Quick look at the patch suggests that it mainly needs discussion about
> design and particular implementation choices, there is fair amount of
> TODOs and FIXMEs. I'd like to look at it too but I doubt that I'll have
> time to do in depth review in this CF.
Yes. I think it's a bit premature to discuss the code thoroughly at this
point - I'd like to discuss the general approach to the feature (i.e.
minimizing the impact on those not using it, etc.).
The most interesting part of the code are probably the comments,
explaining the design in more detail, known shortcomings and possible ways
to address them.
regards
Tomas
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: WIP: multivariate statistics / proof of concept
2014-10-12 22:00 WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-10-29 09:41 ` Re: WIP: multivariate statistics / proof of concept David Rowley <dgrowleyml@gmail.com>
2014-10-29 11:31 ` Re: WIP: multivariate statistics / proof of concept Petr Jelinek <petr@2ndquadrant.com>
2014-10-29 11:48 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
@ 2014-10-30 09:17 ` David Rowley <dgrowleyml@gmail.com>
2014-10-30 10:29 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
0 siblings, 1 reply; 37+ messages in thread
From: David Rowley @ 2014-10-30 09:17 UTC (permalink / raw)
To: Tomas Vondra <tv@fuzzy.cz>; +Cc: Petr Jelinek <petr@2ndquadrant.com>; pgsql-hackers
On Thu, Oct 30, 2014 at 12:48 AM, Tomas Vondra <tv@fuzzy.cz> wrote:
> Dne 29 Říjen 2014, 12:31, Petr Jelinek napsal(a):
> >> I've not really gotten around to looking at the patch yet, but I'm also
> >> wondering if it would be simple include allowing functional statistics
> >> too. The pg_mv_statistic name seems to indicate multi columns, but how
> >> about stats on date(datetime_column), or perhaps any non-volatile
> >> function. This would help to solve the problem highlighted here
> >>
> http://www.postgresql.org/message-id/CAApHDvp2vH=7O-gp-zAf7aWy+A-WHWVg7h3Vc6=5pf9Uf34DhQ@mail.gmail....
> >> . Without giving it too much thought, perhaps any expression that can be
> >> indexed should be allowed to have stats? Would that be really difficult
> >> to implement in comparison to what you've already done with the patch so
> >> far?
> >>
> >
> > I would not over-complicate requirements for the first version of this,
> > I think it's already complicated enough.
>
> My thoughts, exactly. I'm not willing to put more features into the
> initial version of the patch. Actually, I'm thinking about ripping out
> some experimental features (particularly "hashed MCV" and "associative
> rules").
>
>
That's fair, but I didn't really mean to imply that you should go work on
that too and that it should be part of this patch..
I was thinking more along the lines of that I don't really agree with the
table name for the new stats and that at some later date someone will want
to add expression stats and we'd probably better come up design that would
be friendly towards that. At this time I can only think that the name of
the table might not suit well to expression stats, I'd hate to see someone
have to invent a 3rd table to support these when we could likely come up
with something that could be extended later and still make sense both today
and in the future.
I was just looking at how expression indexes are stored in pg_index and I
see that if it's an expression index that the expression is stored in
the indexprs column which is of type pg_node_tree, so quite possibly at
some point in the future the new stats table could just have an extra
column added, and for today, we'd just need to come up with a future proof
name... Perhaps pg_statistic_ext or pg_statisticx, and name functions and
source files something along those lines instead?
Regards
David Rowley
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: WIP: multivariate statistics / proof of concept
2014-10-12 22:00 WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-10-29 09:41 ` Re: WIP: multivariate statistics / proof of concept David Rowley <dgrowleyml@gmail.com>
2014-10-29 11:31 ` Re: WIP: multivariate statistics / proof of concept Petr Jelinek <petr@2ndquadrant.com>
2014-10-29 11:48 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-10-30 09:17 ` Re: WIP: multivariate statistics / proof of concept David Rowley <dgrowleyml@gmail.com>
@ 2014-10-30 10:29 ` Tomas Vondra <tv@fuzzy.cz>
0 siblings, 0 replies; 37+ messages in thread
From: Tomas Vondra @ 2014-10-30 10:29 UTC (permalink / raw)
To: David Rowley <dgrowleyml@gmail.com>; +Cc: Tomas Vondra <tv@fuzzy.cz>; Petr Jelinek <petr@2ndquadrant.com>; pgsql-hackers
Dne 30 Říjen 2014, 10:17, David Rowley napsal(a):
> On Thu, Oct 30, 2014 at 12:48 AM, Tomas Vondra <tv@fuzzy.cz> wrote:
>
>> Dne 29 Říjen 2014, 12:31, Petr Jelinek napsal(a):
>> >> I've not really gotten around to looking at the patch yet, but I'm
>> also
>> >> wondering if it would be simple include allowing functional
>> statistics
>> >> too. The pg_mv_statistic name seems to indicate multi columns, but
>> how
>> >> about stats on date(datetime_column), or perhaps any non-volatile
>> >> function. This would help to solve the problem highlighted here
>> >>
>> http://www.postgresql.org/message-id/CAApHDvp2vH=7O-gp-zAf7aWy+A-WHWVg7h3Vc6=5pf9Uf34DhQ@mail.gmail....
>> >> . Without giving it too much thought, perhaps any expression that can
>> be
>> >> indexed should be allowed to have stats? Would that be really
>> difficult
>> >> to implement in comparison to what you've already done with the patch
>> so
>> >> far?
>> >>
>> >
>> > I would not over-complicate requirements for the first version of
>> this,
>> > I think it's already complicated enough.
>>
>> My thoughts, exactly. I'm not willing to put more features into the
>> initial version of the patch. Actually, I'm thinking about ripping out
>> some experimental features (particularly "hashed MCV" and "associative
>> rules").
>>
>>
> That's fair, but I didn't really mean to imply that you should go work on
> that too and that it should be part of this patch..
> I was thinking more along the lines of that I don't really agree with the
> table name for the new stats and that at some later date someone will want
> to add expression stats and we'd probably better come up design that would
> be friendly towards that. At this time I can only think that the name of
> the table might not suit well to expression stats, I'd hate to see someone
> have to invent a 3rd table to support these when we could likely come up
> with something that could be extended later and still make sense both
> today
> and in the future.
>
> I was just looking at how expression indexes are stored in pg_index and I
> see that if it's an expression index that the expression is stored in
> the indexprs column which is of type pg_node_tree, so quite possibly at
> some point in the future the new stats table could just have an extra
> column added, and for today, we'd just need to come up with a future proof
> name... Perhaps pg_statistic_ext or pg_statisticx, and name functions and
> source files something along those lines instead?
Ah, OK. I don't think the catalog name "pg_mv_statistic" is somehow
inappropriate for this purpose, though. IMHO the "multivariate" does not
mean "only columns" or "no expressions", it simply describes that the
approximated density function has multiple input variables, be it
attributes or expressions.
But maybe there's a better name.
Tomas
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: WIP: multivariate statistics / proof of concept
2014-10-12 22:00 WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
@ 2014-11-13 11:31 ` Simon Riggs <simon@2ndQuadrant.com>
2014-11-13 13:11 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2016-11-21 22:10 ` Re: WIP: multivariate statistics / proof of concept Robert Haas <robertmhaas@gmail.com>
3 siblings, 2 replies; 37+ messages in thread
From: Simon Riggs @ 2014-11-13 11:31 UTC (permalink / raw)
To: Tomas Vondra <tv@fuzzy.cz>; +Cc: pgsql-hackers
On 12 October 2014 23:00, Tomas Vondra <tv@fuzzy.cz> wrote:
> It however seems to be working sufficiently well at this point, enough
> to get some useful feedback. So here we go.
This looks interesting and useful.
What I'd like to check before a detailed review is that this has
sufficient applicability to be useful.
My understanding is that Q9 and Q18 of TPC-H have poor plans as a
result of multi-column stats errors.
Could you look at those queries and confirm that this patch can
produce better plans for them?
If so, I will work with you to review this patch.
One aspect of the patch that seems to be missing is a user declaration
of correlation, just as we have for setting n_distinct. It seems like
an even easier place to start to just let the user specify the stats
declaratively. That way we can split the patch into two parts. First,
allow multi column stats that are user declared. Then add user stats
collected by ANALYZE. The first part is possibly contentious and thus
a good initial focus. The second part will have lots of discussion, so
good to skip for a first version.
--
Simon Riggs http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: WIP: multivariate statistics / proof of concept
2014-10-12 22:00 WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-11-13 11:31 ` Re: WIP: multivariate statistics / proof of concept Simon Riggs <simon@2ndQuadrant.com>
@ 2014-11-13 13:11 ` Tomas Vondra <tv@fuzzy.cz>
2014-11-13 15:51 ` Re: WIP: multivariate statistics / proof of concept Katharina Büchse <katharina.buechse@uni-jena.de>
1 sibling, 1 reply; 37+ messages in thread
From: Tomas Vondra @ 2014-11-13 13:11 UTC (permalink / raw)
To: pgsql-hackers
Dne 13 Listopad 2014, 12:31, Simon Riggs napsal(a):
> On 12 October 2014 23:00, Tomas Vondra <tv@fuzzy.cz> wrote:
>
>> It however seems to be working sufficiently well at this point, enough
>> to get some useful feedback. So here we go.
>
> This looks interesting and useful.
>
> What I'd like to check before a detailed review is that this has
> sufficient applicability to be useful.
>
> My understanding is that Q9 and Q18 of TPC-H have poor plans as a
> result of multi-column stats errors.
>
> Could you look at those queries and confirm that this patch can
> produce better plans for them?
Sure. I planned to do such verification/demonstration anyway, after
discussing the overall approach.
I planned to give it a try on TPC-DS, but I can start with the TPC-H
queries you propose. I'm not sure whether the poor estimates in Q9 & Q18
come from column correlation though - if it's due to some other issues
(e.g. conditions that are difficult to estimate), this patch can't do
anything with them. But it's a good start.
> If so, I will work with you to review this patch.
Thanks!
> One aspect of the patch that seems to be missing is a user declaration
> of correlation, just as we have for setting n_distinct. It seems like
> an even easier place to start to just let the user specify the stats
> declaratively. That way we can split the patch into two parts. First,
> allow multi column stats that are user declared. Then add user stats
> collected by ANALYZE. The first part is possibly contentious and thus
> a good initial focus. The second part will have lots of discussion, so
> good to skip for a first version.
I'm not a big fan of this approach, for a number of reasons.
Firstly, it only works for "simple" parameters that are trivial to specify
(say, Pearson's correlation coefficient), and the patch does not work with
those at all - it only works with histograms, MCV lists (and might work
with associative rules in the future). And we certainly can't ask users to
specify multivariate histograms - because it's very difficult to do, and
also because complex stats are more susceptible to get stale after adding
new data to the table.
Secondly, even if we add such "simple" parameters to the patch, we have to
come up with a way to apply those parameters to the estimates. The
problem is that as the parameters get simpler, it's less and less useful
to compute the stats.
Another question is whether it should support more than 2 columns ...
The only place where I think this might work are the associative rules.
It's simple to specify rules like ("ZIP code" implies "city") and we could
even do some simple check against the data to see if it actually makes
sense (and 'disable' the rule if not).
But maybe I got it wrong and you have something particular in mind? Can
you give an example of how it would work?
regards
Tomas
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: WIP: multivariate statistics / proof of concept
2014-10-12 22:00 WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-11-13 11:31 ` Re: WIP: multivariate statistics / proof of concept Simon Riggs <simon@2ndQuadrant.com>
2014-11-13 13:11 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
@ 2014-11-13 15:51 ` Katharina Büchse <katharina.buechse@uni-jena.de>
2014-11-13 16:42 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
0 siblings, 1 reply; 37+ messages in thread
From: Katharina Büchse @ 2014-11-13 15:51 UTC (permalink / raw)
To: pgsql-hackers
On 13.11.2014 14:11, Tomas Vondra wrote:
> Dne 13 Listopad 2014, 12:31, Simon Riggs napsal(a):
>> On 12 October 2014 23:00, Tomas Vondra <tv@fuzzy.cz> wrote:
>>
>>> It however seems to be working sufficiently well at this point, enough
>>> to get some useful feedback. So here we go.
>> This looks interesting and useful.
>>
>> What I'd like to check before a detailed review is that this has
>> sufficient applicability to be useful.
>>
>> My understanding is that Q9 and Q18 of TPC-H have poor plans as a
>> result of multi-column stats errors.
>>
>> Could you look at those queries and confirm that this patch can
>> produce better plans for them?
> Sure. I planned to do such verification/demonstration anyway, after
> discussing the overall approach.
>
> I planned to give it a try on TPC-DS, but I can start with the TPC-H
> queries you propose. I'm not sure whether the poor estimates in Q9 & Q18
> come from column correlation though - if it's due to some other issues
> (e.g. conditions that are difficult to estimate), this patch can't do
> anything with them. But it's a good start.
>
>> If so, I will work with you to review this patch.
> Thanks!
>
>> One aspect of the patch that seems to be missing is a user declaration
>> of correlation, just as we have for setting n_distinct. It seems like
>> an even easier place to start to just let the user specify the stats
>> declaratively. That way we can split the patch into two parts. First,
>> allow multi column stats that are user declared. Then add user stats
>> collected by ANALYZE. The first part is possibly contentious and thus
>> a good initial focus. The second part will have lots of discussion, so
>> good to skip for a first version.
> I'm not a big fan of this approach, for a number of reasons.
>
> Firstly, it only works for "simple" parameters that are trivial to specify
> (say, Pearson's correlation coefficient), and the patch does not work with
> those at all - it only works with histograms, MCV lists (and might work
> with associative rules in the future). And we certainly can't ask users to
> specify multivariate histograms - because it's very difficult to do, and
> also because complex stats are more susceptible to get stale after adding
> new data to the table.
>
> Secondly, even if we add such "simple" parameters to the patch, we have to
> come up with a way to apply those parameters to the estimates. The
> problem is that as the parameters get simpler, it's less and less useful
> to compute the stats.
>
> Another question is whether it should support more than 2 columns ...
>
> The only place where I think this might work are the associative rules.
> It's simple to specify rules like ("ZIP code" implies "city") and we could
> even do some simple check against the data to see if it actually makes
> sense (and 'disable' the rule if not).
and even this simple example has its limits, at least in Germany ZIP
codes are not unique for rural areas, where several villages have the
same ZIP code.
I guess there are just a few examples where columns are completely
functional dependent without any exceptions.
But of course, if the user gives this information just for optimization
the statistics, some exceptions don't matter.
If this information should be used for creating different execution
plans (e.g. on column A is an index and column B is functional
dependent, one could think about using this index on A and the
dependency instead of running through the whole table to find all tuples
that fit the query on column B), exceptions are a very important issue.
>
> But maybe I got it wrong and you have something particular in mind? Can
> you give an example of how it would work?
>
> regards
> Tomas
>
>
>
--
Dipl.-Math. Katharina Büchse
Friedrich-Schiller-Universität Jena
Institut für Informatik
Lehrstuhl für Datenbanken und Informationssysteme
Ernst-Abbe-Platz 2
07743 Jena
Telefon 03641/946367
Webseite http://users.minet.uni-jena.de/~re89qen/
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: WIP: multivariate statistics / proof of concept
2014-10-12 22:00 WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-11-13 11:31 ` Re: WIP: multivariate statistics / proof of concept Simon Riggs <simon@2ndQuadrant.com>
2014-11-13 13:11 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-11-13 15:51 ` Re: WIP: multivariate statistics / proof of concept Katharina Büchse <katharina.buechse@uni-jena.de>
@ 2014-11-13 16:42 ` Tomas Vondra <tv@fuzzy.cz>
2014-11-15 17:49 ` Re: WIP: multivariate statistics / proof of concept Kevin Grittner <kgrittn@ymail.com>
0 siblings, 1 reply; 37+ messages in thread
From: Tomas Vondra @ 2014-11-13 16:42 UTC (permalink / raw)
To: "Katharina Büchse" <katharina.buechse@uni-jena.de>; +Cc: pgsql-hackers
Dne 13 Listopad 2014, 16:51, Katharina Büchse napsal(a):
> On 13.11.2014 14:11, Tomas Vondra wrote:
>
>> The only place where I think this might work are the associative rules.
>> It's simple to specify rules like ("ZIP code" implies "city") and we
>> could
>> even do some simple check against the data to see if it actually makes
>> sense (and 'disable' the rule if not).
>
> and even this simple example has its limits, at least in Germany ZIP
> codes are not unique for rural areas, where several villages have the
> same ZIP code.
>
> I guess there are just a few examples where columns are completely
> functional dependent without any exceptions.
> But of course, if the user gives this information just for optimization
> the statistics, some exceptions don't matter.
> If this information should be used for creating different execution
> plans (e.g. on column A is an index and column B is functional
> dependent, one could think about using this index on A and the
> dependency instead of running through the whole table to find all tuples
> that fit the query on column B), exceptions are a very important issue.
Yes, exactly. The aim of this patch is "only" improving estimates, not
removing conditions from the plan (e.g. checking only the ZIP code and not
the city name). That certainly can't be done solely based on approximate
statistics, and as you point out most real-world data either contain bugs
or are inherently imperfect (we have the same kind of ZIP/city
inconsistencies in Czech). That's not a big issue for estimates (assuming
only small fraction of rows violates the rule) though.
Tomas
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: WIP: multivariate statistics / proof of concept
2014-10-12 22:00 WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-11-13 11:31 ` Re: WIP: multivariate statistics / proof of concept Simon Riggs <simon@2ndQuadrant.com>
2014-11-13 13:11 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-11-13 15:51 ` Re: WIP: multivariate statistics / proof of concept Katharina Büchse <katharina.buechse@uni-jena.de>
2014-11-13 16:42 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
@ 2014-11-15 17:49 ` Kevin Grittner <kgrittn@ymail.com>
2014-11-15 18:35 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
0 siblings, 1 reply; 37+ messages in thread
From: Kevin Grittner @ 2014-11-15 17:49 UTC (permalink / raw)
To: Tomas Vondra <tv@fuzzy.cz>; Katharina Büchse <katharina.buechse@uni-jena.de>; +Cc: pgsql-hackers
Tomas Vondra <tv@fuzzy.cz> wrote:
> Dne 13 Listopad 2014, 16:51, Katharina Büchse napsal(a):
>> On 13.11.2014 14:11, Tomas Vondra wrote:
>>
>>> The only place where I think this might work are the associative rules.
>>> It's simple to specify rules like ("ZIP code" implies "city") and we could
>>> even do some simple check against the data to see if it actually makes
>>> sense (and 'disable' the rule if not).
>>
>> and even this simple example has its limits, at least in Germany ZIP
>> codes are not unique for rural areas, where several villages have the
>> same ZIP code.
> as you point out most real-world data either contain bugs
> or are inherently imperfect (we have the same kind of ZIP/city
> inconsistencies in Czech).
You can have lots of fun with U.S. zip code, too. Just on the
nominally "Madison, Wisconsin" zip codes (those starting with 537),
there are several exceptions:
select zipcode, city, locationtype
from zipcode
where zipcode like '537%'
and Decommisioned = 'false'
and zipcodetype = 'STANDARD'
and locationtype in ('PRIMARY', 'ACCEPTABLE')
order by zipcode, city;
zipcode | city | locationtype
---------+-----------+--------------
53703 | MADISON | PRIMARY
53704 | MADISON | PRIMARY
53705 | MADISON | PRIMARY
53706 | MADISON | PRIMARY
53711 | FITCHBURG | ACCEPTABLE
53711 | MADISON | PRIMARY
53713 | FITCHBURG | ACCEPTABLE
53713 | MADISON | PRIMARY
53713 | MONONA | ACCEPTABLE
53714 | MADISON | PRIMARY
53714 | MONONA | ACCEPTABLE
53715 | MADISON | PRIMARY
53716 | MADISON | PRIMARY
53716 | MONONA | ACCEPTABLE
53717 | MADISON | PRIMARY
53718 | MADISON | PRIMARY
53719 | FITCHBURG | ACCEPTABLE
53719 | MADISON | PRIMARY
53725 | MADISON | PRIMARY
53726 | MADISON | PRIMARY
53744 | MADISON | PRIMARY
(21 rows)
If you eliminate the quals besides the zipcode column you get 61
rows and it gets much stranger, with legal municipalities that are
completely surrounded by Madison that the postal service would
rather you didn't use in addressing your envelopes, but they have
to deliver to anyway, and organizations inside Madison receiving
enough mail to (literally) have their own zip code -- where the
postal service allows the organization name as a deliverable
"city".
If you want to have your own fun with this data, you can download
it here:
http://federalgovernmentzipcodes.us/free-zipcode-database.csv
I was able to load it into PostgreSQL with this:
create table zipcode
(
recordnumber integer not null,
zipcode text not null,
zipcodetype text not null,
city text not null,
state text not null,
locationtype text not null,
lat double precision,
long double precision,
xaxis double precision not null,
yaxis double precision not null,
zaxis double precision not null,
worldregion text not null,
country text not null,
locationtext text,
location text,
decommisioned text not null,
taxreturnsfiled bigint,
estimatedpopulation bigint,
totalwages bigint,
notes text
);
comment on column zipcode.zipcode is 'Zipcode or military postal code(FPO/APO)';
comment on column zipcode.zipcodetype is 'Standard, PO BOX Only, Unique, Military(implies APO or FPO)';
comment on column zipcode.city is 'offical city name(s)';
comment on column zipcode.state is 'offical state, territory, or quasi-state (AA, AE, AP) abbreviation code';
comment on column zipcode.locationtype is 'Primary, Acceptable,Not Acceptable';
comment on column zipcode.lat is 'Decimal Latitude, if available';
comment on column zipcode.long is 'Decimal Longitude, if available';
comment on column zipcode.location is 'Standard Display (eg Phoenix, AZ ; Pago Pago, AS ; Melbourne, AU )';
comment on column zipcode.decommisioned is 'If Primary location, Yes implies historical Zipcode, No Implies current Zipcode; If not Primary, Yes implies Historical Placename';
comment on column zipcode.taxreturnsfiled is 'Number of Individual Tax Returns Filed in 2008';
copy zipcode from 'filepath' with (format csv, header);
alter table zipcode add primary key (recordnumber);
create unique index zipcode_city on zipcode (zipcode, city);
I bet there are all sorts of correlation possibilities with, for
example, latitude and longitude and other variables. With 81831
rows and so many correlations among the columns, it might be a
useful data set to test with.
--
Kevin Grittner
EDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: WIP: multivariate statistics / proof of concept
2014-10-12 22:00 WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-11-13 11:31 ` Re: WIP: multivariate statistics / proof of concept Simon Riggs <simon@2ndQuadrant.com>
2014-11-13 13:11 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-11-13 15:51 ` Re: WIP: multivariate statistics / proof of concept Katharina Büchse <katharina.buechse@uni-jena.de>
2014-11-13 16:42 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-11-15 17:49 ` Re: WIP: multivariate statistics / proof of concept Kevin Grittner <kgrittn@ymail.com>
@ 2014-11-15 18:35 ` Tomas Vondra <tv@fuzzy.cz>
2014-12-08 01:01 ` Re: WIP: multivariate statistics / proof of concept Michael Paquier <michael.paquier@gmail.com>
0 siblings, 1 reply; 37+ messages in thread
From: Tomas Vondra @ 2014-11-15 18:35 UTC (permalink / raw)
To: pgsql-hackers
On 15.11.2014 18:49, Kevin Grittner
> If you eliminate the quals besides the zipcode column you get 61
> rows and it gets much stranger, with legal municipalities that are
> completely surrounded by Madison that the postal service would
> rather you didn't use in addressing your envelopes, but they have
> to deliver to anyway, and organizations inside Madison receiving
> enough mail to (literally) have their own zip code -- where the
> postal service allows the organization name as a deliverable
> "city".
>
> If you want to have your own fun with this data, you can download
> it here:
>
> http://federalgovernmentzipcodes.us/free-zipcode-database.csv
>
...
>
> I bet there are all sorts of correlation possibilities with, for
> example, latitude and longitude and other variables. With 81831
> rows and so many correlations among the columns, it might be a
> useful data set to test with.
Thanks for the link. I've been looking for a good dataset with such
data, and this one is by far the best one.
The current version of the patch supports only data types passed by
value (i.e. no varlena types - text, ), which means it's impossible to
build multivariate stats on some of the interesting columns (state,
city, ...).
I guess it's time to start working on removing this limitation.
Tomas
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: WIP: multivariate statistics / proof of concept
2014-10-12 22:00 WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-11-13 11:31 ` Re: WIP: multivariate statistics / proof of concept Simon Riggs <simon@2ndQuadrant.com>
2014-11-13 13:11 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-11-13 15:51 ` Re: WIP: multivariate statistics / proof of concept Katharina Büchse <katharina.buechse@uni-jena.de>
2014-11-13 16:42 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-11-15 17:49 ` Re: WIP: multivariate statistics / proof of concept Kevin Grittner <kgrittn@ymail.com>
2014-11-15 18:35 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
@ 2014-12-08 01:01 ` Michael Paquier <michael.paquier@gmail.com>
2014-12-09 20:15 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
0 siblings, 1 reply; 37+ messages in thread
From: Michael Paquier @ 2014-12-08 01:01 UTC (permalink / raw)
To: Tomas Vondra <tv@fuzzy.cz>; +Cc: pgsql-hackers
On Sun, Nov 16, 2014 at 3:35 AM, Tomas Vondra <tv@fuzzy.cz> wrote:
> Thanks for the link. I've been looking for a good dataset with such
> data, and this one is by far the best one.
>
> The current version of the patch supports only data types passed by
> value (i.e. no varlena types - text, ), which means it's impossible to
> build multivariate stats on some of the interesting columns (state,
> city, ...).
>
> I guess it's time to start working on removing this limitation.
Tomas, what's your status on this patch? Are you planning to make it
more complicated than it is? For now I have switched it to a "Needs
Review" state because even your first version did not get advanced
review (that's quite big btw). I guess that we should switch it to the
next CF.
Regards,
--
Michael
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: WIP: multivariate statistics / proof of concept
2014-10-12 22:00 WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-11-13 11:31 ` Re: WIP: multivariate statistics / proof of concept Simon Riggs <simon@2ndQuadrant.com>
2014-11-13 13:11 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-11-13 15:51 ` Re: WIP: multivariate statistics / proof of concept Katharina Büchse <katharina.buechse@uni-jena.de>
2014-11-13 16:42 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-11-15 17:49 ` Re: WIP: multivariate statistics / proof of concept Kevin Grittner <kgrittn@ymail.com>
2014-11-15 18:35 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-12-08 01:01 ` Re: WIP: multivariate statistics / proof of concept Michael Paquier <michael.paquier@gmail.com>
@ 2014-12-09 20:15 ` Tomas Vondra <tv@fuzzy.cz>
2014-12-15 02:55 ` Re: WIP: multivariate statistics / proof of concept Michael Paquier <michael.paquier@gmail.com>
0 siblings, 1 reply; 37+ messages in thread
From: Tomas Vondra @ 2014-12-09 20:15 UTC (permalink / raw)
To: pgsql-hackers
On 8.12.2014 02:01, Michael Paquier wrote:
> On Sun, Nov 16, 2014 at 3:35 AM, Tomas Vondra <tv@fuzzy.cz> wrote:
>> Thanks for the link. I've been looking for a good dataset with such
>> data, and this one is by far the best one.
>>
>> The current version of the patch supports only data types passed by
>> value (i.e. no varlena types - text, ), which means it's impossible to
>> build multivariate stats on some of the interesting columns (state,
>> city, ...).
>>
>> I guess it's time to start working on removing this limitation.
> Tomas, what's your status on this patch? Are you planning to make it
> more complicated than it is? For now I have switched it to a "Needs
> Review" state because even your first version did not get advanced
> review (that's quite big btw). I guess that we should switch it to the
> next CF.
Hello Michael,
I agree with moving the patch to the next CF - I'm working on the patch,
but I will take a bit more time to submit a new version and I can do
that in the next CF.
regards
Tomas
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: WIP: multivariate statistics / proof of concept
2014-10-12 22:00 WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-11-13 11:31 ` Re: WIP: multivariate statistics / proof of concept Simon Riggs <simon@2ndQuadrant.com>
2014-11-13 13:11 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-11-13 15:51 ` Re: WIP: multivariate statistics / proof of concept Katharina Büchse <katharina.buechse@uni-jena.de>
2014-11-13 16:42 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-11-15 17:49 ` Re: WIP: multivariate statistics / proof of concept Kevin Grittner <kgrittn@ymail.com>
2014-11-15 18:35 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-12-08 01:01 ` Re: WIP: multivariate statistics / proof of concept Michael Paquier <michael.paquier@gmail.com>
2014-12-09 20:15 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
@ 2014-12-15 02:55 ` Michael Paquier <michael.paquier@gmail.com>
2015-01-15 08:00 ` Re: WIP: multivariate statistics / proof of concept Michael Paquier <michael.paquier@gmail.com>
0 siblings, 1 reply; 37+ messages in thread
From: Michael Paquier @ 2014-12-15 02:55 UTC (permalink / raw)
To: Tomas Vondra <tv@fuzzy.cz>; +Cc: pgsql-hackers
On Wed, Dec 10, 2014 at 5:15 AM, Tomas Vondra <tv@fuzzy.cz> wrote:
> I agree with moving the patch to the next CF - I'm working on the patch,
> but I will take a bit more time to submit a new version and I can do
> that in the next CF.
OK cool. I just moved it by myself. I didn't see it yet registered in 2014-12.
Thanks,
--
Michael
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: WIP: multivariate statistics / proof of concept
2014-10-12 22:00 WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-11-13 11:31 ` Re: WIP: multivariate statistics / proof of concept Simon Riggs <simon@2ndQuadrant.com>
2014-11-13 13:11 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-11-13 15:51 ` Re: WIP: multivariate statistics / proof of concept Katharina Büchse <katharina.buechse@uni-jena.de>
2014-11-13 16:42 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-11-15 17:49 ` Re: WIP: multivariate statistics / proof of concept Kevin Grittner <kgrittn@ymail.com>
2014-11-15 18:35 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-12-08 01:01 ` Re: WIP: multivariate statistics / proof of concept Michael Paquier <michael.paquier@gmail.com>
2014-12-09 20:15 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-12-15 02:55 ` Re: WIP: multivariate statistics / proof of concept Michael Paquier <michael.paquier@gmail.com>
@ 2015-01-15 08:00 ` Michael Paquier <michael.paquier@gmail.com>
2015-01-24 20:21 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 1 reply; 37+ messages in thread
From: Michael Paquier @ 2015-01-15 08:00 UTC (permalink / raw)
To: Tomas Vondra <tv@fuzzy.cz>; +Cc: pgsql-hackers
On Mon, Dec 15, 2014 at 11:55 AM, Michael Paquier
<michael.paquier@gmail.com> wrote:
> On Wed, Dec 10, 2014 at 5:15 AM, Tomas Vondra <tv@fuzzy.cz> wrote:
>> I agree with moving the patch to the next CF - I'm working on the patch,
>> but I will take a bit more time to submit a new version and I can do
>> that in the next CF.
> OK cool. I just moved it by myself. I didn't see it yet registered in 2014-12.
Marked as returned with feedback. No new version showed up in the last
month and this patch was waiting for input from author.
--
Michael
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: WIP: multivariate statistics / proof of concept
2014-10-12 22:00 WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-11-13 11:31 ` Re: WIP: multivariate statistics / proof of concept Simon Riggs <simon@2ndQuadrant.com>
2014-11-13 13:11 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-11-13 15:51 ` Re: WIP: multivariate statistics / proof of concept Katharina Büchse <katharina.buechse@uni-jena.de>
2014-11-13 16:42 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-11-15 17:49 ` Re: WIP: multivariate statistics / proof of concept Kevin Grittner <kgrittn@ymail.com>
2014-11-15 18:35 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-12-08 01:01 ` Re: WIP: multivariate statistics / proof of concept Michael Paquier <michael.paquier@gmail.com>
2014-12-09 20:15 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-12-15 02:55 ` Re: WIP: multivariate statistics / proof of concept Michael Paquier <michael.paquier@gmail.com>
2015-01-15 08:00 ` Re: WIP: multivariate statistics / proof of concept Michael Paquier <michael.paquier@gmail.com>
@ 2015-01-24 20:21 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-20 08:33 ` Re: WIP: multivariate statistics / proof of concept Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
0 siblings, 1 reply; 37+ messages in thread
From: Tomas Vondra @ 2015-01-24 20:21 UTC (permalink / raw)
To: pgsql-hackers
Hi,
attached is an updated version of the multivariate stats patch. This is
going to be a bit longer mail, so I'll put here a small ToC ;-)
1) patch split into 4 parts
2) where to start / documentation
3) state of the code
4) main changes/improvements
5) remaining limitations
The motivation and design ideas, explained in the first message of this
thread are still valid. It might be a good idea to read it first:
http://www.postgresql.org/message-id/flat/543AFA15.4080608@fuzzy.cz
BTW if you happen to go to FOSDEM [PGDay], I'll gladly give you an intro
into the patch in person, or discuss the patch in general.
1) Patch split into 4 parts
---------------------------
Firstly, the patch got broken into the following four pieces, to make
the reviews somewhat easier:
1) 0001-shared-infrastructure-and-functional-dependencies.patch
- infrastructure, shared by all the kinds of stats added
in the following patches (catalog, ALTER TABLE, ANALYZE ...)
- implementation of a simple statistics, tracking functional
dependencies between columns (previously called "associative
rules", but that's incorrect for several reasons)
- this does not modify the optimizer in any way
2) 0002-clause-reduction-using-functional-dependencies.patch
- applies the functional dependencies to optimizer (i.e. considers
the rules in clauselist_selectivity())
3) 0003-multivariate-MCV-lists.patch
- multivariate MCV lists (both ANALYZE and optimizer parts)
4) 0004-multivariate-histograms.patch
- multivariate histograms (both ANALYZE and optimizer parts)
You may look at the patches at github here:
https://github.com/tvondra/postgres/tree/multivariate-stats-squashed
The branch is not stable, i.e. I'll rebase / squash / force-push changes
in the future. (There's also multivariate-stats development branch with
unsquashed changes, but you don't want to look at that, trust me.)
The patches are not exactly small (being in the 50-100 kB range), but
that's mostly because of the amount of comments explaining the goals and
implementation details.
2) Where to start / documentation
---------------------------------
I strived to document all the pieces properly, mostly in the form of
comments. There's no sgml documentation at this point, which should
obviously change in the future.
Anyway, I'd suggest reading the first e-mail in this thread, explaining
the ideas, and then these comments:
1) functional dependencies (patch 0001)
- src/backend/utils/mvstats/dependencies.c
2) MCV lists (patch 0003)
- src/backend/utils/mvstats/mcv.c
3) histograms (patch 0004)
- src/backend/utils/mvstats/mcv.c
- also see clauselist_mv_selectivity_mcvlist() in clausesel.c
- also see clauselist_mv_selectivity_histogram() in clausesel.c
4) selectivity estimation (patches 0002-0004)
- all in src/backend/optimizer/path/clausesel.c
- clauselist_selectivity() - overview of how the stats are applied
- clauselist_apply_dependencies() - functional dependencies reduction
- clauselist_mv_selectivity_mcvlist() - MCV list estimation
- clauselist_mv_selectivity_histogram() - histogram estimation
3) State of the code
--------------------
I've spent a fair amount of time testing the patches, and while I
believe there are no segfaults or so, I know parts of the code need a
bit more love.
The part most in need of improvements / comments is probably the code in
clausesel.c - that seems a bit quirky. Reviews / comments regarding this
part of the code are very welcome - I'm sure there are many ways to
improve this part.
There are a few FIXMEs elsewhere (e.g. about memory allocation in the
(de)serialization code), but those are mostly well-defined issues that I
know how to address (at least I believe so).
4) Main changes/improvements
----------------------------
There are many significant improvements. The previous patch version was
in the 'proof of concept' category (missing pieces, knowingly broken in
some areas), the current patch should 'mostly work'.
The patch fixes two most annoying limitations of the first version:
(a) support for all data types (not just those passed by value)
(b) handles NULL values properly
(c) adds support for IS [NOT] NULL clauses
Aside from that the code was significantly improved, there are proper
regression tests and plenty of comments explaining the details.
5) Remaining limitations
------------------------
(a) limited to stats on 8 columns
This is mostly just a 'safeguard' restriction.
(b) only data types with '<' operator
I don't think this will change anytime soon, because all the
algorithms for building the stats rely on this. I don't see
this as a serious limitation though.
(c) not handling DROP COLUMN or DROP TABLE and so on
Currently this is not handled at all (so the regression tests
do an explicit DELETE from the pg_mv_statistic catalog).
Handling the DROP TABLE won't be difficult, it's similar to the
current stats. Handling ALTER TABLE ... DROP COLUMN will be much
more tricky I guess - should we drop all the stats referencing
that column, or should we just remove it from the stats? Or
should we keep it and treat it as NULL? Not sure what's the best
solution.
(d) limited list of compatible WHERE clauses
The initial patch handled only simple operator clauses
(Var op Constant)
where operator is one of ('<', '<=', '=', '>=', '>'). Now it also
handles IS [NOT] NULL clauses. Adding more clause types should
not be overly difficult - starting with more traditional
'BooleanTest' conditions, or even multi-column conditions
(Var op Var)
which are difficult to estimate using simple-column stats.
(e) optimizer uses single stats per table
This is still true and I don't think this will change soon. i do
have some ideas on how to merge multiple stats etc. but it's
certainly complex stuff, unlikely to happen within this CF. The
patch makes a lot of sense even without this particular feature,
because you can create multiple stats, each suitable for different
queries.
(f) no JOIN conditions
Similarly to the previous point, it's on the TODO but it's not
going to happen in this CF.
kind regards
--
Tomas Vondra http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[text/x-diff] 0001-shared-infrastructure-and-functional-dependencies.patch (59.8K, ../../54C3FED3.1060600@2ndquadrant.com/2-0001-shared-infrastructure-and-functional-dependencies.patch)
download | inline diff:
From 2b8cbad288a0cd8fb5603af447c99f706ba7bbee Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Sun, 11 Jan 2015 19:51:48 +0100
Subject: [PATCH 1/4] shared infrastructure and functional dependencies
Basic infrastructure shared by all kinds of multivariate
stats, most importantly:
- adds a new system catalog (pg_mv_statistic)
- ALTER TABLE ... ADD STATISTICS syntax
- implementation of functional dependencies (the simplest
type of multivariate statistics)
- building functional dependencies in ANALYZE
- updates regression tests (new catalog etc.)
This does not include any changes to the optimizer, i.e.
it does not influence the query planning.
FIX: invalid assert in lookup_var_attr_stats()
The current implementation requires a valid 'ltopr'
so that we can sort the sample rows in various ways,
and the assert did verify this by checking that the
function is 'compute_scalar_stats'. This is however
private function in analyze.c, so the check failed
after moving the code into common.c.
Fixed by checking the 'ltopr' operator directly.
Eventually this will be removed, as ltopr is only
needed for histograms (functional dependencies and
MVC lists may be built without it).
FIX: improved comments about functional dependencies
FIX: add magic (MVSTAT_DEPS_MAGIC) into MVDependencies
FIX: improved analysis of functional dependencies
Changes:
- decreased minimum group size
- count contradicting rows ('not supporting' ones)
The algorithm is still rather simple and probably needs
other improvements.
FIX: add pg_mv_stats_dependencies_show() function
This function actually prints the rules, not just some basic
info (number of rules) as pg_mv_stats_dependencies_info().
FIX: (dependencies != NULL) in pg_mv_stats_dependencies_info()
STRICT is not a solution, because the deserialization may fail
for some reason (corrupted data, ...)
FIX: rename 'associative rules' to 'functional dependencies'
It's a more appropriate name as functional dependencies,
as defined in relational theory (esp. Normal Forms) are
tracking column-level dependencies.
Associative (or more correctly 'association') rules are
tracking dependencies between particular values, and not
necessarily in different columns (shopping bag analysis).
Also, did a bunch of comment improvements, minor fixes.
This does not include changes in clausesel.c!
FIX: remove obsolete Assert() enforcing typbyval types
---
src/backend/catalog/Makefile | 1 +
src/backend/catalog/system_views.sql | 10 +
src/backend/commands/analyze.c | 17 +-
src/backend/commands/tablecmds.c | 149 +++++++-
src/backend/nodes/copyfuncs.c | 15 +-
src/backend/parser/gram.y | 67 +++-
src/backend/utils/Makefile | 2 +-
src/backend/utils/cache/syscache.c | 12 +
src/backend/utils/mvstats/Makefile | 17 +
src/backend/utils/mvstats/common.c | 272 ++++++++++++++
src/backend/utils/mvstats/common.h | 70 ++++
src/backend/utils/mvstats/dependencies.c | 554 +++++++++++++++++++++++++++++
src/include/catalog/indexing.h | 5 +
src/include/catalog/pg_mv_statistic.h | 69 ++++
src/include/catalog/pg_proc.h | 5 +
src/include/catalog/toasting.h | 1 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 11 +-
src/include/utils/mvstats.h | 86 +++++
src/include/utils/syscache.h | 1 +
src/test/regress/expected/rules.out | 8 +
src/test/regress/expected/sanity_check.out | 1 +
22 files changed, 1365 insertions(+), 9 deletions(-)
create mode 100644 src/backend/utils/mvstats/Makefile
create mode 100644 src/backend/utils/mvstats/common.c
create mode 100644 src/backend/utils/mvstats/common.h
create mode 100644 src/backend/utils/mvstats/dependencies.c
create mode 100644 src/include/catalog/pg_mv_statistic.h
create mode 100644 src/include/utils/mvstats.h
diff --git a/src/backend/catalog/Makefile b/src/backend/catalog/Makefile
index a403c64..d6c16f8 100644
--- a/src/backend/catalog/Makefile
+++ b/src/backend/catalog/Makefile
@@ -32,6 +32,7 @@ POSTGRES_BKI_SRCS = $(addprefix $(top_srcdir)/src/include/catalog/,\
pg_attrdef.h pg_constraint.h pg_inherits.h pg_index.h pg_operator.h \
pg_opfamily.h pg_opclass.h pg_am.h pg_amop.h pg_amproc.h \
pg_language.h pg_largeobject_metadata.h pg_largeobject.h pg_aggregate.h \
+ pg_mv_statistic.h \
pg_statistic.h pg_rewrite.h pg_trigger.h pg_event_trigger.h pg_description.h \
pg_cast.h pg_enum.h pg_namespace.h pg_conversion.h pg_depend.h \
pg_database.h pg_db_role_setting.h pg_tablespace.h pg_pltemplate.h \
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 4bc874f..da957fc 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -152,6 +152,16 @@ CREATE VIEW pg_indexes AS
LEFT JOIN pg_tablespace T ON (T.oid = I.reltablespace)
WHERE C.relkind IN ('r', 'm') AND I.relkind = 'i';
+CREATE VIEW pg_mv_stats AS
+ SELECT
+ N.nspname AS schemaname,
+ C.relname AS tablename,
+ S.stakeys AS attnums,
+ length(S.stadeps) as depsbytes,
+ pg_mv_stats_dependencies_info(S.stadeps) as depsinfo
+ FROM (pg_mv_statistic S JOIN pg_class C ON (C.oid = S.starelid))
+ LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace);
+
CREATE VIEW pg_stats AS
SELECT
nspname AS schemaname,
diff --git a/src/backend/commands/analyze.c b/src/backend/commands/analyze.c
index 5de2b39..a02dcb2 100644
--- a/src/backend/commands/analyze.c
+++ b/src/backend/commands/analyze.c
@@ -27,6 +27,7 @@
#include "catalog/indexing.h"
#include "catalog/pg_collation.h"
#include "catalog/pg_inherits_fn.h"
+#include "catalog/pg_mv_statistic.h"
#include "catalog/pg_namespace.h"
#include "commands/dbcommands.h"
#include "commands/tablecmds.h"
@@ -54,7 +55,11 @@
#include "utils/syscache.h"
#include "utils/timestamp.h"
#include "utils/tqual.h"
+#include "utils/fmgroids.h"
+#include "utils/builtins.h"
+#include "utils/mvstats.h"
+#include "access/sysattr.h"
/* Data structure for Algorithm S from Knuth 3.4.2 */
typedef struct
@@ -110,7 +115,6 @@ static void update_attstats(Oid relid, bool inh,
static Datum std_fetch_func(VacAttrStatsP stats, int rownum, bool *isNull);
static Datum ind_fetch_func(VacAttrStatsP stats, int rownum, bool *isNull);
-
/*
* analyze_rel() -- analyze one relation
*/
@@ -472,6 +476,13 @@ do_analyze_rel(Relation onerel, VacuumStmt *vacstmt,
* all analyzable columns. We use a lower bound of 100 rows to avoid
* possible overflow in Vitter's algorithm. (Note: that will also be the
* target in the corner case where there are no analyzable columns.)
+ *
+ * FIXME This sample sizing is mostly OK when computing stats for
+ * individual columns, but when computing multi-variate stats
+ * for multivariate stats (histograms, mcv, ...) it's rather
+ * insufficient. For small number of dimensions it works, but
+ * for complex stats it'd be nice use sample proportional to
+ * the table (say, 0.5% - 1%) instead of a fixed size.
*/
targrows = 100;
for (i = 0; i < attr_cnt; i++)
@@ -574,6 +585,9 @@ do_analyze_rel(Relation onerel, VacuumStmt *vacstmt,
update_attstats(RelationGetRelid(Irel[ind]), false,
thisdata->attr_cnt, thisdata->vacattrstats);
}
+
+ /* Build multivariate stats (if there are any). */
+ build_mv_stats(onerel, numrows, rows, attr_cnt, vacattrstats);
}
/*
@@ -2819,3 +2833,4 @@ compare_mcvs(const void *a, const void *b)
return da - db;
}
+
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 66d5083..3ec1a5a 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -35,6 +35,7 @@
#include "catalog/pg_foreign_table.h"
#include "catalog/pg_inherits.h"
#include "catalog/pg_inherits_fn.h"
+#include "catalog/pg_mv_statistic.h"
#include "catalog/pg_namespace.h"
#include "catalog/pg_opclass.h"
#include "catalog/pg_tablespace.h"
@@ -91,7 +92,7 @@
#include "utils/syscache.h"
#include "utils/tqual.h"
#include "utils/typcache.h"
-
+#include "utils/mvstats.h"
/*
* ON COMMIT action list
@@ -139,8 +140,9 @@ static List *on_commits = NIL;
#define AT_PASS_ADD_COL 5 /* ADD COLUMN */
#define AT_PASS_ADD_INDEX 6 /* ADD indexes */
#define AT_PASS_ADD_CONSTR 7 /* ADD constraints, defaults */
-#define AT_PASS_MISC 8 /* other stuff */
-#define AT_NUM_PASSES 9
+#define AT_PASS_ADD_STATS 8 /* ADD statistics */
+#define AT_PASS_MISC 9 /* other stuff */
+#define AT_NUM_PASSES 10
typedef struct AlteredTableInfo
{
@@ -415,7 +417,8 @@ static void ATExecReplicaIdentity(Relation rel, ReplicaIdentityStmt *stmt, LOCKM
static void ATExecGenericOptions(Relation rel, List *options);
static void ATExecEnableRowSecurity(Relation rel);
static void ATExecDisableRowSecurity(Relation rel);
-
+static void ATExecAddStatistics(AlteredTableInfo *tab, Relation rel,
+ StatisticsDef *def, LOCKMODE lockmode);
static void copy_relation_data(SMgrRelation rel, SMgrRelation dst,
ForkNumber forkNum, char relpersistence);
static const char *storage_name(char c);
@@ -2965,6 +2968,7 @@ AlterTableGetLockLevel(List *cmds)
* updates.
*/
case AT_SetStatistics: /* Uses MVCC in getTableAttrs() */
+ case AT_AddStatistics: /* XXX not sure if the right level */
case AT_ClusterOn: /* Uses MVCC in getIndexes() */
case AT_DropCluster: /* Uses MVCC in getIndexes() */
case AT_SetOptions: /* Uses MVCC in getTableAttrs() */
@@ -3119,6 +3123,7 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd,
pass = AT_PASS_ADD_CONSTR;
break;
case AT_SetStatistics: /* ALTER COLUMN SET STATISTICS */
+ case AT_AddStatistics: /* XXX maybe not the right place */
ATSimpleRecursion(wqueue, rel, cmd, recurse, lockmode);
/* Performs own permission checks */
ATPrepSetStatistics(rel, cmd->name, cmd->def, lockmode);
@@ -3414,6 +3419,9 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,
case AT_SetStatistics: /* ALTER COLUMN SET STATISTICS */
ATExecSetStatistics(rel, cmd->name, cmd->def, lockmode);
break;
+ case AT_AddStatistics: /* ADD STATISTICS */
+ ATExecAddStatistics(tab, rel, (StatisticsDef *) cmd->def, lockmode);
+ break;
case AT_SetOptions: /* ALTER COLUMN SET ( options ) */
ATExecSetOptions(rel, cmd->name, cmd->def, false, lockmode);
break;
@@ -11605,3 +11613,136 @@ RangeVarCallbackForAlterRelation(const RangeVar *rv, Oid relid, Oid oldrelid,
ReleaseSysCache(tuple);
}
+
+/* used for sorting the attnums in ATExecAddStatistics */
+static int compare_int16(const void *a, const void *b)
+{
+ return memcmp(a, b, sizeof(int16));
+}
+
+/*
+ * Implements the ALTER TABLE ... ADD STATISTICS (options) ON (columns).
+ *
+ * The code is an unholy mix of pieces that really belong to other parts
+ * of the source tree.
+ *
+ * FIXME Check that the types are pass-by-value and support sort,
+ * although maybe we can live without the sort (and only build
+ * MCV list / association rules).
+ *
+ * FIXME This should probably check for duplicate stats (i.e. same
+ * keys, same options). Although maybe it's useful to have
+ * multiple stats on the same columns with different options
+ * (say, a detailed MCV-only stats for some queries, histogram
+ * for others, etc.)
+ */
+static void ATExecAddStatistics(AlteredTableInfo *tab, Relation rel,
+ StatisticsDef *def, LOCKMODE lockmode)
+{
+ int i, j;
+ ListCell *l;
+ int16 attnums[INDEX_MAX_KEYS];
+ int numcols = 0;
+
+ HeapTuple htup;
+ Datum values[Natts_pg_mv_statistic];
+ bool nulls[Natts_pg_mv_statistic];
+ int2vector *stakeys;
+ Relation mvstatrel;
+
+ /* by default build everything */
+ bool build_dependencies = true;
+
+ Assert(IsA(def, StatisticsDef));
+
+ /* transform the column names to attnum values */
+
+ foreach(l, def->keys)
+ {
+ char *attname = strVal(lfirst(l));
+ HeapTuple atttuple;
+
+ atttuple = SearchSysCacheAttName(RelationGetRelid(rel), attname);
+
+ if (!HeapTupleIsValid(atttuple))
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_COLUMN),
+ errmsg("column \"%s\" referenced in statistics does not exist",
+ attname)));
+
+ /* more than MVHIST_MAX_DIMENSIONS columns not allowed */
+ if (numcols >= MVSTATS_MAX_DIMENSIONS)
+ ereport(ERROR,
+ (errcode(ERRCODE_TOO_MANY_COLUMNS),
+ errmsg("cannot have more than %d keys in a statistics",
+ MVSTATS_MAX_DIMENSIONS)));
+
+ attnums[numcols] = ((Form_pg_attribute) GETSTRUCT(atttuple))->attnum;
+ ReleaseSysCache(atttuple);
+ numcols++;
+ }
+
+ /*
+ * Check the lower bound (at least 2 columns), the upper bound was
+ * already checked in the loop.
+ */
+ if (numcols < 2)
+ ereport(ERROR,
+ (errcode(ERRCODE_TOO_MANY_COLUMNS),
+ errmsg("multivariate stats require 2 or more columns")));
+
+ /* look for duplicities */
+ for (i = 0; i < numcols; i++)
+ for (j = 0; j < numcols; j++)
+ if ((i != j) && (attnums[i] == attnums[j]))
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_COLUMN),
+ errmsg("duplicate column name in statistics definition")));
+
+ /* parse the statistics options */
+ foreach (l, def->options)
+ {
+ DefElem *opt = (DefElem*)lfirst(l);
+
+ if (strcmp(opt->defname, "dependencies") == 0)
+ build_dependencies = defGetBoolean(opt);
+ else
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("unrecognized STATISTICS option \"%s\"",
+ opt->defname)));
+ }
+
+ /* sort the attnums and build int2vector */
+ qsort(attnums, numcols, sizeof(int16), compare_int16);
+ stakeys = buildint2vector(attnums, numcols);
+
+ /*
+ * Okay, let's create the pg_mv_statistic entry.
+ */
+ memset(values, 0, sizeof(values));
+ memset(nulls, false, sizeof(nulls));
+
+ /* no stats collected yet, so just the keys */
+ values[Anum_pg_mv_statistic_starelid-1] = ObjectIdGetDatum(RelationGetRelid(rel));
+
+ values[Anum_pg_mv_statistic_stakeys -1] = PointerGetDatum(stakeys);
+ values[Anum_pg_mv_statistic_deps_enabled -1] = BoolGetDatum(build_dependencies);
+
+ nulls[Anum_pg_mv_statistic_stadeps -1] = true;
+
+ /* insert the tuple into pg_mv_statistic */
+ mvstatrel = heap_open(MvStatisticRelationId, RowExclusiveLock);
+
+ htup = heap_form_tuple(mvstatrel->rd_att, values, nulls);
+
+ simple_heap_insert(mvstatrel, htup);
+
+ CatalogUpdateIndexes(mvstatrel, htup);
+
+ heap_freetuple(htup);
+
+ heap_close(mvstatrel, RowExclusiveLock);
+
+ return;
+}
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index f1a24f5..eb406ff 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -3909,6 +3909,17 @@ _copyAlterPolicyStmt(const AlterPolicyStmt *from)
return newnode;
}
+static StatisticsDef *
+_copyStatisticsDef(const StatisticsDef *from)
+{
+ StatisticsDef *newnode = makeNode(StatisticsDef);
+
+ COPY_NODE_FIELD(keys);
+ COPY_NODE_FIELD(options);
+
+ return newnode;
+}
+
/* ****************************************************************
* pg_list.h copy functions
* ****************************************************************
@@ -4723,6 +4734,9 @@ copyObject(const void *from)
case T_CommonTableExpr:
retval = _copyCommonTableExpr(from);
break;
+ case T_StatisticsDef:
+ retval = _copyStatisticsDef(from);
+ break;
case T_PrivGrantee:
retval = _copyPrivGrantee(from);
break;
@@ -4735,7 +4749,6 @@ copyObject(const void *from)
case T_XmlSerialize:
retval = _copyXmlSerialize(from);
break;
-
default:
elog(ERROR, "unrecognized node type: %d", (int) nodeTag(from));
retval = 0; /* keep compiler quiet */
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 679e1bb..7a89f6c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -366,6 +366,13 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
create_generic_options alter_generic_options
relation_expr_list dostmt_opt_list
+%type <list> OptStatsOptions
+%type <str> stats_options_name
+%type <node> stats_options_arg
+%type <defelt> stats_options_elem
+%type <list> stats_options_list
+
+
%type <list> opt_fdw_options fdw_options
%type <defelt> fdw_option
@@ -484,7 +491,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <keyword> unreserved_keyword type_func_name_keyword
%type <keyword> col_name_keyword reserved_keyword
-%type <node> TableConstraint TableLikeClause
+%type <node> TableConstraint TableLikeClause TableStatistics
%type <ival> TableLikeOptionList TableLikeOption
%type <list> ColQualList
%type <node> ColConstraint ColConstraintElem ConstraintAttr
@@ -2312,6 +2319,14 @@ alter_table_cmd:
n->subtype = AT_DisableRowSecurity;
$$ = (Node *)n;
}
+ /* ALTER TABLE <name> ADD STATISTICS (options) ON (columns) ... */
+ | ADD_P TableStatistics
+ {
+ AlterTableCmd *n = makeNode(AlterTableCmd);
+ n->subtype = AT_AddStatistics;
+ n->def = $2;
+ $$ = (Node *)n;
+ }
| alter_generic_options
{
AlterTableCmd *n = makeNode(AlterTableCmd);
@@ -3382,6 +3397,56 @@ OptConsTableSpace: USING INDEX TABLESPACE name { $$ = $4; }
ExistingIndex: USING INDEX index_name { $$ = $3; }
;
+/*****************************************************************************
+ *
+ * QUERY :
+ * ALTER TABLE relname ADD STATISTICS (columns) WITH (options)
+ *
+ *****************************************************************************/
+
+TableStatistics:
+ STATISTICS OptStatsOptions ON '(' columnList ')'
+ {
+ StatisticsDef *n = makeNode(StatisticsDef);
+ n->keys = $5;
+ n->options = $2;
+ $$ = (Node *) n;
+ }
+ ;
+
+OptStatsOptions:
+ '(' stats_options_list ')' { $$ = $2; }
+ | /*EMPTY*/ { $$ = NIL; }
+ ;
+
+stats_options_list:
+ stats_options_elem
+ {
+ $$ = list_make1($1);
+ }
+ | stats_options_list ',' stats_options_elem
+ {
+ $$ = lappend($1, $3);
+ }
+ ;
+
+stats_options_elem:
+ stats_options_name stats_options_arg
+ {
+ $$ = makeDefElem($1, $2);
+ }
+ ;
+
+stats_options_name:
+ NonReservedWord { $$ = $1; }
+ ;
+
+stats_options_arg:
+ opt_boolean_or_string { $$ = (Node *) makeString($1); }
+ | NumericOnly { $$ = (Node *) $1; }
+ | /* EMPTY */ { $$ = NULL; }
+ ;
+
/*****************************************************************************
*
diff --git a/src/backend/utils/Makefile b/src/backend/utils/Makefile
index 8374533..eba0352 100644
--- a/src/backend/utils/Makefile
+++ b/src/backend/utils/Makefile
@@ -9,7 +9,7 @@ top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
OBJS = fmgrtab.o
-SUBDIRS = adt cache error fmgr hash init mb misc mmgr resowner sort time
+SUBDIRS = adt cache error fmgr hash init mb misc mmgr mvstats resowner sort time
# location of Catalog.pm
catalogdir = $(top_srcdir)/src/backend/catalog
diff --git a/src/backend/utils/cache/syscache.c b/src/backend/utils/cache/syscache.c
index bd27168..f61ef7e 100644
--- a/src/backend/utils/cache/syscache.c
+++ b/src/backend/utils/cache/syscache.c
@@ -43,6 +43,7 @@
#include "catalog/pg_foreign_server.h"
#include "catalog/pg_foreign_table.h"
#include "catalog/pg_language.h"
+#include "catalog/pg_mv_statistic.h"
#include "catalog/pg_namespace.h"
#include "catalog/pg_opclass.h"
#include "catalog/pg_operator.h"
@@ -499,6 +500,17 @@ static const struct cachedesc cacheinfo[] = {
},
4
},
+ {MvStatisticRelationId, /* MVSTATOID */
+ MvStatisticOidIndexId,
+ 1,
+ {
+ ObjectIdAttributeNumber,
+ 0,
+ 0,
+ 0
+ },
+ 128
+ },
{NamespaceRelationId, /* NAMESPACENAME */
NamespaceNameIndexId,
1,
diff --git a/src/backend/utils/mvstats/Makefile b/src/backend/utils/mvstats/Makefile
new file mode 100644
index 0000000..099f1ed
--- /dev/null
+++ b/src/backend/utils/mvstats/Makefile
@@ -0,0 +1,17 @@
+#-------------------------------------------------------------------------
+#
+# Makefile--
+# Makefile for utils/mvstats
+#
+# IDENTIFICATION
+# src/backend/utils/mvstats/Makefile
+#
+#-------------------------------------------------------------------------
+
+subdir = src/backend/utils/mvstats
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+
+OBJS = common.o dependencies.o
+
+include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/utils/mvstats/common.c b/src/backend/utils/mvstats/common.c
new file mode 100644
index 0000000..36757d5
--- /dev/null
+++ b/src/backend/utils/mvstats/common.c
@@ -0,0 +1,272 @@
+/*-------------------------------------------------------------------------
+ *
+ * common.c
+ * POSTGRES multivariate statistics
+ *
+ *
+ * Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/utils/mvstats/common.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "common.h"
+
+/*
+ * Compute requested multivariate stats, using the rows sampled for the
+ * plain (single-column) stats.
+ *
+ * This fetches a list of stats from pg_mv_statistic, computes the stats
+ * and serializes them back into the catalog (as bytea values).
+ */
+void
+build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
+ int natts, VacAttrStats **vacattrstats)
+{
+ int i;
+ MVStats mvstats;
+ int nmvstats;
+
+ /*
+ * Fetch defined MV groups from pg_mv_statistic, and then compute
+ * the MV statistics (histograms for now).
+ */
+ mvstats = list_mv_stats(RelationGetRelid(onerel), &nmvstats, false);
+
+ for (i = 0; i < nmvstats; i++)
+ {
+ MVDependencies deps = NULL;
+
+ /* int2 vector of attnums the stats should be computed on */
+ int2vector * attrs = mvstats[i].stakeys;
+
+ /* check allowed number of dimensions */
+ Assert((attrs->dim1 >= 2) && (attrs->dim1 <= MVSTATS_MAX_DIMENSIONS));
+
+ /*
+ * Analyze functional dependencies of columns.
+ */
+ deps = build_mv_dependencies(numrows, rows, attrs, natts, vacattrstats);
+
+ /* store the histogram / MCV list in the catalog */
+ update_mv_stats(mvstats[i].mvoid, deps);
+ }
+}
+
+/*
+ * Lookup the VacAttrStats info for the selected columns, with indexes
+ * matching the attrs vector (to make it easy to work with when
+ * computing multivariate stats).
+ */
+VacAttrStats **
+lookup_var_attr_stats(int2vector *attrs, int natts, VacAttrStats **vacattrstats)
+{
+ int i, j;
+ int numattrs = attrs->dim1;
+ VacAttrStats **stats = (VacAttrStats**)palloc0(numattrs * sizeof(VacAttrStats*));
+
+ /* lookup VacAttrStats info for the requested columns (same attnum) */
+ for (i = 0; i < numattrs; i++)
+ {
+ stats[i] = NULL;
+ for (j = 0; j < natts; j++)
+ {
+ if (attrs->values[i] == vacattrstats[j]->tupattnum)
+ {
+ stats[i] = vacattrstats[j];
+ break;
+ }
+ }
+
+ /*
+ * Check that we found the info, that the attnum matches and
+ * that there's the requested 'lt' operator and that the type
+ * is 'passed-by-value'.
+ */
+ Assert(stats[i] != NULL);
+ Assert(stats[i]->tupattnum == attrs->values[i]);
+
+ /* FIXME This is rather ugly way to check for 'ltopr' (which
+ * is defined for 'scalar' attributes).
+ */
+ Assert(((StdAnalyzeData *)stats[i]->extra_data)->ltopr != InvalidOid);
+ }
+
+ return stats;
+}
+
+/*
+ * Fetch list of MV stats defined on a table, without the actual data
+ * for histograms, MCV lists etc.
+ */
+MVStats
+list_mv_stats(Oid relid, int *nstats, bool built_only)
+{
+ Relation indrel;
+ SysScanDesc indscan;
+ ScanKeyData skey;
+ HeapTuple htup;
+ MVStats result;
+
+ /* start with 16 items, that should be enough for most cases */
+ int maxitems = 16;
+ result = (MVStats)palloc0(sizeof(MVStatsData) * maxitems);
+ *nstats = 0;
+
+ /* Prepare to scan pg_mv_statistic for entries having indrelid = this rel. */
+ ScanKeyInit(&skey,
+ Anum_pg_mv_statistic_starelid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(relid));
+
+ indrel = heap_open(MvStatisticRelationId, AccessShareLock);
+ indscan = systable_beginscan(indrel, MvStatisticRelidIndexId, true,
+ NULL, 1, &skey);
+
+ while (HeapTupleIsValid(htup = systable_getnext(indscan)))
+ {
+ Form_pg_mv_statistic stats = (Form_pg_mv_statistic) GETSTRUCT(htup);
+
+ /*
+ * Skip statistics that were not computed yet (if only stats
+ * that were already built were requested)
+ */
+ if (built_only && (! stats->deps_built))
+ continue;
+
+ /* double the array size if needed */
+ if (*nstats == maxitems)
+ {
+ maxitems *= 2;
+ result = (MVStats)repalloc(result, sizeof(MVStatsData) * maxitems);
+ }
+
+ result[*nstats].mvoid = HeapTupleGetOid(htup);
+ result[*nstats].stakeys = buildint2vector(stats->stakeys.values, stats->stakeys.dim1);
+ result[*nstats].deps_built = stats->deps_built;
+ *nstats += 1;
+ }
+
+ systable_endscan(indscan);
+
+ heap_close(indrel, AccessShareLock);
+
+ /* TODO maybe save the list into relcache, as in RelationGetIndexList
+ * (which was used as an inspiration of this one)?. */
+
+ return result;
+}
+
+void
+update_mv_stats(Oid mvoid, MVDependencies dependencies)
+{
+ HeapTuple stup,
+ oldtup;
+ Datum values[Natts_pg_mv_statistic];
+ bool nulls[Natts_pg_mv_statistic];
+ bool replaces[Natts_pg_mv_statistic];
+
+ Relation sd = heap_open(MvStatisticRelationId, RowExclusiveLock);
+
+ memset(nulls, 1, Natts_pg_mv_statistic * sizeof(bool));
+ memset(replaces, 0, Natts_pg_mv_statistic * sizeof(bool));
+ memset(values, 0, Natts_pg_mv_statistic * sizeof(Datum));
+
+ /*
+ * Construct a new pg_mv_statistic tuple - replace only the histogram
+ * and MCV list, depending whether it actually was computed.
+ */
+ if (dependencies != NULL)
+ {
+ nulls[Anum_pg_mv_statistic_stadeps -1] = false;
+ values[Anum_pg_mv_statistic_stadeps - 1]
+ = PointerGetDatum(serialize_mv_dependencies(dependencies));
+ }
+
+ /* always replace the value (either by bytea or NULL) */
+ replaces[Anum_pg_mv_statistic_stadeps -1] = true;
+
+ /* always change the availability flags */
+ nulls[Anum_pg_mv_statistic_deps_built -1] = false;
+
+ replaces[Anum_pg_mv_statistic_deps_built-1] = true;
+
+ values[Anum_pg_mv_statistic_deps_built-1] = BoolGetDatum(dependencies != NULL);
+
+ /* Is there already a pg_mv_statistic tuple for this attribute? */
+ oldtup = SearchSysCache1(MVSTATOID,
+ ObjectIdGetDatum(mvoid));
+
+ if (HeapTupleIsValid(oldtup))
+ {
+ /* Yes, replace it */
+ stup = heap_modify_tuple(oldtup,
+ RelationGetDescr(sd),
+ values,
+ nulls,
+ replaces);
+ ReleaseSysCache(oldtup);
+ simple_heap_update(sd, &stup->t_self, stup);
+ }
+ else
+ elog(ERROR, "invalid pg_mv_statistic record (oid=%d)", mvoid);
+
+ /* update indexes too */
+ CatalogUpdateIndexes(sd, stup);
+
+ heap_freetuple(stup);
+
+ heap_close(sd, RowExclusiveLock);
+}
+
+/* multi-variate stats comparator */
+
+/*
+ * qsort_arg comparator for sorting Datums (MV stats)
+ *
+ * This does not maintain the tupnoLink array.
+ */
+int
+compare_scalars_simple(const void *a, const void *b, void *arg)
+{
+ Datum da = *(Datum*)a;
+ Datum db = *(Datum*)b;
+ SortSupport ssup= (SortSupport) arg;
+
+ return ApplySortComparator(da, false, db, false, ssup);
+}
+
+/*
+ * qsort_arg comparator for sorting data when partitioning a MV bucket
+ */
+int
+compare_scalars_partition(const void *a, const void *b, void *arg)
+{
+ Datum da = ((ScalarItem*)a)->value;
+ Datum db = ((ScalarItem*)b)->value;
+ SortSupport ssup= (SortSupport) arg;
+
+ return ApplySortComparator(da, false, db, false, ssup);
+}
+
+/*
+ * qsort_arg comparator for sorting Datum[] (row of Datums) when
+ * counting distinct values.
+ */
+int
+compare_scalars_memcmp(const void *a, const void *b, void *arg)
+{
+ Size len = *(Size*)arg;
+
+ return memcmp(a, b, len);
+}
+
+int
+compare_scalars_memcmp_2(const void *a, const void *b)
+{
+ return memcmp(a, b, sizeof(Datum));
+}
diff --git a/src/backend/utils/mvstats/common.h b/src/backend/utils/mvstats/common.h
new file mode 100644
index 0000000..f511c4e
--- /dev/null
+++ b/src/backend/utils/mvstats/common.h
@@ -0,0 +1,70 @@
+/*-------------------------------------------------------------------------
+ *
+ * common.h
+ * POSTGRES multivariate statistics
+ *
+ *
+ * Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/utils/mvstats/common.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/tuptoaster.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_collation.h"
+#include "catalog/pg_mv_statistic.h"
+#include "foreign/fdwapi.h"
+#include "postmaster/autovacuum.h"
+#include "storage/lmgr.h"
+#include "utils/datum.h"
+#include "utils/sortsupport.h"
+#include "utils/syscache.h"
+#include "utils/fmgroids.h"
+#include "utils/builtins.h"
+#include "access/sysattr.h"
+
+#include "utils/mvstats.h"
+
+/* FIXME private structure copied from analyze.c */
+
+typedef struct
+{
+ Oid eqopr; /* '=' operator for datatype, if any */
+ Oid eqfunc; /* and associated function */
+ Oid ltopr; /* '<' operator for datatype, if any */
+} StdAnalyzeData;
+
+typedef struct
+{
+ Datum value; /* a data value */
+ int tupno; /* position index for tuple it came from */
+} ScalarItem;
+
+typedef struct
+{
+ int count; /* # of duplicates */
+ int first; /* values[] index of first occurrence */
+} ScalarMCVItem;
+
+typedef struct
+{
+ SortSupport ssup;
+ int *tupnoLink;
+} CompareScalarsContext;
+
+
+VacAttrStats ** lookup_var_attr_stats(int2vector *attrs,
+ int natts, VacAttrStats **vacattrstats);
+
+/* comparators, used when constructing multivariate stats */
+int compare_scalars_simple(const void *a, const void *b, void *arg);
+int compare_scalars_partition(const void *a, const void *b, void *arg);
+int compare_scalars_memcmp(const void *a, const void *b, void *arg);
+int compare_scalars_memcmp_2(const void *a, const void *b);
diff --git a/src/backend/utils/mvstats/dependencies.c b/src/backend/utils/mvstats/dependencies.c
new file mode 100644
index 0000000..b900efd
--- /dev/null
+++ b/src/backend/utils/mvstats/dependencies.c
@@ -0,0 +1,554 @@
+/*-------------------------------------------------------------------------
+ *
+ * dependencies.c
+ * POSTGRES multivariate functional dependencies
+ *
+ *
+ * Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/utils/mvstats/dependencies.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "common.h"
+
+/*
+ * Mine functional dependencies between columns, in the form (A => B),
+ * meaning that a value in column 'A' determines value in 'B'. A simple
+ * artificial example may be a table created like this
+ *
+ * CREATE TABLE deptest (a INT, b INT)
+ * AS SELECT i, i/10 FROM generate_series(1,100000) s(i);
+ *
+ * Clearly, once we know the value for 'A' we can easily determine the
+ * value of 'B' by dividing (A/10). A more practical example may be
+ * addresses, where (ZIP code => city name), i.e. once we know the ZIP,
+ * we probably know which city it belongs to. Larger cities usually have
+ * multiple ZIP codes, so the dependency can't be reversed.
+ *
+ * Functional dependencies are a concept well described in relational
+ * theory, especially in definition of normalization and "normal forms".
+ * Wikipedia has a nice definition of a functional dependency [1]:
+ *
+ * In a given table, an attribute Y is said to have a functional
+ * dependency on a set of attributes X (written X -> Y) if and only
+ * if each X value is associated with precisely one Y value. For
+ * example, in an "Employee" table that includes the attributes
+ * "Employee ID" and "Employee Date of Birth", the functional
+ * dependency {Employee ID} -> {Employee Date of Birth} would hold.
+ * It follows from the previous two sentences that each {Employee ID}
+ * is associated with precisely one {Employee Date of Birth}.
+ *
+ * [1] http://en.wikipedia.org/wiki/Database_normalization
+ *
+ * Most datasets might be normalized not to contain any such functional
+ * dependencies, but sometimes it's not practical. In some cases it's
+ * actually a conscious choice to model the dataset in denormalized way,
+ * either because of performance or to make querying easier.
+ *
+ * The current implementation supports only dependencies between two
+ * columns, but this is merely a simplification of the initial patch.
+ * It's certainly useful to mine for dependencies involving multiple
+ * columns on the 'left' side, i.e. a condition for the dependency.
+ * That is dependencies [A,B] => C and so on.
+ *
+ * Handling multiple columns on the right side is not necessary, as such
+ * dependencies may be decomposed into a set of dependencies with
+ * the same meaning, one for each column on the right side. For example
+ *
+ * A => [B,C]
+ *
+ * is exactly the same as
+ *
+ * (A => B) & (A => C).
+ *
+ * Of course, storing (A => [B, C]) may be more efficient thant storing
+ * the two dependencies (A => B) and (A => C) separately.
+ *
+ *
+ * Dependency mining (ANALYZE)
+ * ---------------------------
+ *
+ * FIXME Add more details about how build_mv_dependencies() works
+ * (minimum group size, supporting/contradicting etc.).
+ *
+ * Real-world datasets are imperfect - there may be errors (e.g. due to
+ * data-entry mistakes), or factually correct records, yet contradicting
+ * the dependency (e.g. when a city splits into two, but both keep the
+ * same ZIP code). A strict ANALYZE implementation (where the functional
+ * dependencies are identified) would ignore dependencies on such noisy
+ * data, making the approach unusable in practice.
+ *
+ * The proposed implementation attempts to handle such noisy cases
+ * gracefully, by tolerating small number of contradicting cases.
+ *
+ * In the future this might also perform some sort of test and decide
+ * whether it's worth building any other kind of multivariate stats,
+ * or whether the dependencies sufficiently describe the data. Or at
+ * least not build the MCV list / histogram on the implied columns.
+ * Such reduction would however make the 'verification' (see the next
+ * section) impossible.
+ *
+ *
+ * Clause reduction (planner/optimizer)
+ * ------------------------------------
+ *
+ * FIXME Explain how reduction works.
+ *
+ * The problem with the reduction is that the query may use conditions
+ * that are not redundant, but in fact contradictory - e.g. the user
+ * may search for a ZIP code and a city name not matching the ZIP code.
+ *
+ * In such cases, the condition on the city name is not actually
+ * redundant, but actually contradictory (making the result empty), and
+ * removing it while estimating the cardinality will make the estimate
+ * worse.
+ *
+ * The current estimation assuming independence (and multiplying the
+ * selectivities) works better in this case, but only by utter luck.
+ *
+ * In some cases this might be verified using the other multivariate
+ * statistics - MCV lists and histograms. For MCV lists the verification
+ * might be very simple - peek into the list if there are any items
+ * matching the clause on the 'A' column (e.g. ZIP code), and if such
+ * item is found, check that the 'B' column matches the other clause.
+ * If it does not, the clauses are contradictory. We can't really say
+ * if such item was not found, except maybe restricting the selectivity
+ * using the MCV data (e.g. using min/max selectivity, or something).
+ *
+ * With histograms, it might work similarly - we can't check the values
+ * directly (because histograms use buckets, unlike MCV lists, storing
+ * the actual values). So we can only observe the buckets matching the
+ * clauses - if those buckets have very low frequency, it probably means
+ * the two clauses are incompatible.
+ *
+ * It's unclear what 'low frequency' is, but if one of the clauses is
+ * implied (automatically true because of the other clause), then
+ *
+ * selectivity[clause(A)] = selectivity[clause(A) & clause(B)]
+ *
+ * So we might compute selectivity of the first clause (on the column
+ * A in dependency [A=>B]) - for example using regular statistics.
+ * And then check if the selectivity computed from the histogram is
+ * about the same (or significantly lower).
+ *
+ * The problem is that histograms work well only when the data ordering
+ * matches the natural meaning. For values that serve as labels - like
+ * city names or ZIP codes, or even generated IDs, histograms really
+ * don't work all that well. For example sorting cities by name won't
+ * match the sorting of ZIP codes, rendering the histogram unusable.
+ *
+ * The MCV are probably going to work much better, because they don't
+ * really assume any sort of ordering. And it's probably more appropriate
+ * for the label-like data.
+ *
+ * TODO Support dependencies with multiple columns on left/right.
+ *
+ * TODO Investigate using histogram and MCV list to confirm the
+ * functional dependencies.
+ *
+ * TODO Investigate statistical testing of the distribution (to decide
+ * whether it makes sense to build the histogram/MCV list).
+ *
+ * TODO Using a min/max of selectivities would probably make more sense
+ * for the associated columns.
+ *
+ * TODO Consider eliminating the implied columns from the histogram and
+ * MCV lists (but maybe that's not a good idea).
+ *
+ * FIXME Not sure if this handles NULL values properly (not sure how to
+ * do that). We assume that NULL means 0 for now, handling it just
+ * like any other value.
+ */
+MVDependencies
+build_mv_dependencies(int numrows, HeapTuple *rows, int2vector *attrs,
+ int natts, VacAttrStats **vacattrstats)
+{
+ int i;
+ bool isNull;
+ Size len = 2 * sizeof(Datum); /* only simple associations a => b */
+ int numattrs = attrs->dim1;
+
+ /* result */
+ int ndeps = 0;
+ MVDependencies dependencies = NULL;
+
+ /* TODO Maybe this should be somehow related to the number of
+ * distinct columns in the two columns we're currently analyzing.
+ * Assuming the distribution is uniform, we should expected to
+ * observe in the sample - we can then use the average group
+ * size as a threshold. That seems better than a static approach.
+ */
+ int min_group_size = 3;
+
+ /* dimension indexes we'll check for associations [a => b] */
+ int dima, dimb;
+
+ /* info for the interesting attributes only
+ *
+ * TODO Compute this only once and pass it to all the methods
+ * that need it.
+ */
+ VacAttrStats **stats = lookup_var_attr_stats(attrs, natts, vacattrstats);
+
+ /* We'll reuse the same array for all the combinations */
+ Datum * values = (Datum*)palloc0(numrows * 2 * sizeof(Datum));
+
+ Assert(numattrs >= 2);
+
+ /*
+ * Evaluate all possible combinations of [A => B], using a simple algorithm:
+ *
+ * (a) sort the data by [A,B]
+ * (b) split the data into groups by A (new group whenever a value changes)
+ * (c) count different values in the B column (again, value changes)
+ *
+ * TODO It should be rather simple to merge [A => B] and [A => C] into
+ * [A => B,C]. Just keep A constant, collect all the "implied" columns
+ * and you're done.
+ */
+ for (dima = 0; dima < numattrs; dima++)
+ {
+ for (dimb = 0; dimb < numattrs; dimb++)
+ {
+ Datum val_a, val_b;
+
+ /* number of groups supporting / contradicting the dependency */
+ int n_supporting = 0;
+ int n_contradicting = 0;
+
+ /* counters valid within a group */
+ int group_size = 0;
+ int n_violations = 0;
+
+ int n_supporting_rows = 0;
+ int n_contradicting_rows = 0;
+
+ /* make sure the columns are different (A => A) */
+ if (dima == dimb)
+ continue;
+
+ /* accumulate all the data for both columns into an array and sort it */
+ for (i = 0; i < numrows; i++)
+ {
+ values[i*2] = heap_getattr(rows[i], attrs->values[dima], stats[dima]->tupDesc, &isNull);
+ values[i*2+1] = heap_getattr(rows[i], attrs->values[dimb], stats[dimb]->tupDesc, &isNull);
+ }
+
+ qsort_arg((void *) values, numrows, sizeof(Datum) * 2, compare_scalars_memcmp, &len);
+
+ /*
+ * Walk through the array, split it into rows according to
+ * the A value, and count distinct values in the other one.
+ * If there's a single B value for the whole group, we count
+ * it as supporting the association, otherwise we count it
+ * as contradicting.
+ *
+ * Furthermore we require a group to have at least a certain
+ * number of rows to be considered useful for supporting the
+ * dependency. But when it's contradicting, use it always useful.
+ */
+
+ /* start with values from the first row */
+ val_a = values[0];
+ val_b = values[1];
+ group_size = 1;
+
+ for (i = 1; i < numrows; i++)
+ {
+ if (values[2*i] != val_a) /* end of the group */
+ {
+ /*
+ * If there are no contradicting rows, count it as
+ * supporting (otherwise contradicting), but only if
+ * the group is large enough.
+ *
+ * The requirement of a minimum group size makes it
+ * impossible to identify [unique,unique] cases, but
+ * that's probably a different case. This is more
+ * about [zip => city] associations etc.
+ */
+ n_supporting += ((n_violations == 0) && (group_size >= min_group_size)) ? 1 : 0;
+ n_contradicting += (n_violations != 0) ? 1 : 0;
+
+ n_supporting_rows += ((n_violations == 0) && (group_size >= min_group_size)) ? group_size : 0;
+ n_contradicting_rows += (n_violations > 0) ? group_size : 0;
+
+ /* current values start a new group */
+ val_a = values[2*i];
+ val_b = values[2*i+1];
+ n_violations = 0;
+ group_size = 1;
+ }
+ else
+ {
+ if (values[2*i+1] != val_b) /* mismatch of a B value is contradicting */
+ {
+ val_b = values[2*i+1];
+ n_violations += 1;
+ }
+
+ group_size += 1;
+ }
+ }
+
+ /* handle the last group */
+ n_supporting += ((n_violations == 0) && (group_size >= min_group_size)) ? 1 : 0;
+ n_contradicting += (n_violations != 0) ? 1 : 0;
+ n_supporting_rows += ((n_violations == 0) && (group_size >= min_group_size)) ? group_size : 0;
+ n_contradicting_rows += (n_violations > 0) ? group_size : 0;
+
+ /*
+ * See if the number of rows supporting the association is at least
+ * 10x the number of rows violating the hypothetical dependency.
+ *
+ * TODO This is rather arbitrary limit - I guess it's possible to do
+ * some math to come up with a better rule (e.g. testing a hypothesis
+ * 'this is due to randomness'). We can create a contingency table
+ * from the values and use it for testing. Possibly only when
+ * there are no contradicting rows?
+ *
+ * TODO Also, if (a => b) and (b => a) at the same time, it pretty much
+ * means the columns have the same values (or one is a 'label'),
+ * making the conditions rather redundant. Although it's possible
+ * that the query uses incompatible combination of values.
+ */
+ if (n_supporting_rows > (n_contradicting_rows * 10))
+ {
+ if (dependencies == NULL)
+ {
+ dependencies = (MVDependencies)palloc0(sizeof(MVDependenciesData));
+ dependencies->magic = MVSTAT_DEPS_MAGIC;
+ }
+ else
+ dependencies = repalloc(dependencies, offsetof(MVDependenciesData, deps)
+ + sizeof(MVDependency) * (dependencies->ndeps + 1));
+
+ /* update the */
+ dependencies->deps[ndeps] = (MVDependency)palloc0(sizeof(MVDependencyData));
+ dependencies->deps[ndeps]->a = attrs->values[dima];
+ dependencies->deps[ndeps]->b = attrs->values[dimb];
+
+ dependencies->ndeps = (++ndeps);
+ }
+ }
+ }
+
+ pfree(values);
+
+ return dependencies;
+}
+
+/*
+ * Store the dependencies into a bytea, so that it can be stored in the
+ * pg_mv_statistic catalog.
+ *
+ * Currently this only supports simple two-column rules, and stores them
+ * as a sequence of attnum pairs. In the future, this needs to be made
+ * more complex to support multiple columns on both sides of the
+ * implication (using AND on left, OR on right).
+ */
+bytea *
+serialize_mv_dependencies(MVDependencies dependencies)
+{
+ int i;
+
+ /* we need to store ndeps, and each needs 2 * int16 */
+ Size len = VARHDRSZ + offsetof(MVDependenciesData, deps)
+ + dependencies->ndeps * (sizeof(int16) * 2);
+
+ bytea * output = (bytea*)palloc0(len);
+
+ char * tmp = VARDATA(output);
+
+ SET_VARSIZE(output, len);
+
+ /* first, store the number of dimensions / items */
+ memcpy(tmp, dependencies, offsetof(MVDependenciesData, deps));
+ tmp += offsetof(MVDependenciesData, deps);
+
+ /* walk through the dependencies and copy both columns into the bytea */
+ for (i = 0; i < dependencies->ndeps; i++)
+ {
+ memcpy(tmp, &(dependencies->deps[i]->a), sizeof(int16));
+ tmp += sizeof(int16);
+
+ memcpy(tmp, &(dependencies->deps[i]->b), sizeof(int16));
+ tmp += sizeof(int16);
+ }
+
+ return output;
+}
+
+/*
+ * Reads serialized dependencies into MVDependencies structure.
+ */
+MVDependencies
+deserialize_mv_dependencies(bytea * data)
+{
+ int i;
+ Size expected_size;
+ MVDependencies dependencies;
+ char *tmp;
+
+ if (data == NULL)
+ return NULL;
+
+ if (VARSIZE_ANY_EXHDR(data) < offsetof(MVDependenciesData,deps))
+ elog(ERROR, "invalid MVDependencies size %ld (expected at least %ld)",
+ VARSIZE_ANY_EXHDR(data), offsetof(MVDependenciesData,deps));
+
+ /* read the MVDependencies header */
+ dependencies = (MVDependencies)palloc0(sizeof(MVDependenciesData));
+
+ /* initialize pointer to the data part (skip the varlena header) */
+ tmp = VARDATA(data);
+
+ /* get the header and perform basic sanity checks */
+ memcpy(dependencies, tmp, offsetof(MVDependenciesData, deps));
+ tmp += offsetof(MVDependenciesData, deps);
+
+ if (dependencies->magic != MVSTAT_DEPS_MAGIC)
+ {
+ pfree(dependencies);
+ elog(WARNING, "not a MV Dependencies (magic number mismatch)");
+ return NULL;
+ }
+
+ Assert(dependencies->ndeps > 0);
+
+ /* what bytea size do we expect for those parameters */
+ expected_size = offsetof(MVDependenciesData,deps) +
+ dependencies->ndeps * sizeof(int16) * 2;
+
+ if (VARSIZE_ANY_EXHDR(data) != expected_size)
+ elog(ERROR, "invalid dependencies size %ld (expected %ld)",
+ VARSIZE_ANY_EXHDR(data), expected_size);
+
+ /* allocate space for the MCV items */
+ dependencies = repalloc(dependencies, offsetof(MVDependenciesData,deps)
+ + (dependencies->ndeps * sizeof(MVDependency)));
+
+ for (i = 0; i < dependencies->ndeps; i++)
+ {
+ dependencies->deps[i] = (MVDependency)palloc0(sizeof(MVDependencyData));
+
+ memcpy(&(dependencies->deps[i]->a), tmp, sizeof(int16));
+ tmp += sizeof(int16);
+
+ memcpy(&(dependencies->deps[i]->b), tmp, sizeof(int16));
+ tmp += sizeof(int16);
+ }
+
+ return dependencies;
+}
+
+/* print some basic info about dependencies (number of dependencies) */
+Datum
+pg_mv_stats_dependencies_info(PG_FUNCTION_ARGS)
+{
+ bytea *data = PG_GETARG_BYTEA_P(0);
+ char *result;
+
+ MVDependencies dependencies = deserialize_mv_dependencies(data);
+
+ if (dependencies == NULL)
+ PG_RETURN_NULL();
+
+ result = palloc0(128);
+ snprintf(result, 128, "dependencies=%d", dependencies->ndeps);
+
+ /* FIXME free the deserialized data (pfree is not enough) */
+
+ PG_RETURN_TEXT_P(cstring_to_text(result));
+}
+
+/* print the dependencies
+ *
+ * TODO Would be nice if this knew the actual column names (instead of
+ * the attnums).
+ *
+ * FIXME This is really ugly and does not really check the lengths and
+ * strcpy/snprintf return values properly. Needs to be fixed.
+ */
+Datum
+pg_mv_stats_dependencies_show(PG_FUNCTION_ARGS)
+{
+ int i = 0;
+ bytea *data = PG_GETARG_BYTEA_P(0);
+ char *result = NULL;
+ int len = 0;
+
+ MVDependencies dependencies = deserialize_mv_dependencies(data);
+
+ if (dependencies == NULL)
+ PG_RETURN_NULL();
+
+ for (i = 0; i < dependencies->ndeps; i++)
+ {
+ MVDependency dependency = dependencies->deps[i];
+ char buffer[128];
+
+ int tmp = snprintf(buffer, 128, "%s%d => %d",
+ ((i == 0) ? "" : ", "), dependency->a, dependency->b);
+
+ if (tmp < 127)
+ {
+ if (result == NULL)
+ result = palloc0(len + tmp + 1);
+ else
+ result = repalloc(result, len + tmp + 1);
+
+ strcpy(result + len, buffer);
+ len += tmp;
+ }
+ }
+
+ PG_RETURN_TEXT_P(cstring_to_text(result));
+}
+
+bytea *
+fetch_mv_dependencies(Oid mvoid)
+{
+ Relation indrel;
+ SysScanDesc indscan;
+ ScanKeyData skey;
+ HeapTuple htup;
+ bytea *stadeps = NULL;
+
+ /* Prepare to scan pg_mv_statistic for entries having indrelid = this rel. */
+ ScanKeyInit(&skey,
+ ObjectIdAttributeNumber,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(mvoid));
+
+ indrel = heap_open(MvStatisticRelationId, AccessShareLock);
+ indscan = systable_beginscan(indrel, MvStatisticOidIndexId, true,
+ NULL, 1, &skey);
+
+ while (HeapTupleIsValid(htup = systable_getnext(indscan)))
+ {
+ bool isnull = false;
+ Datum deps = SysCacheGetAttr(MVSTATOID, htup,
+ Anum_pg_mv_statistic_stadeps, &isnull);
+
+ Assert(!isnull);
+
+ stadeps = DatumGetByteaP(deps);
+
+ break;
+ }
+
+ systable_endscan(indscan);
+
+ heap_close(indrel, AccessShareLock);
+
+ /* TODO maybe save the list into relcache, as in RelationGetIndexList
+ * (which was used as an inspiration of this one)?. */
+
+ return stadeps;
+}
diff --git a/src/include/catalog/indexing.h b/src/include/catalog/indexing.h
index a680229..048cd7c 100644
--- a/src/include/catalog/indexing.h
+++ b/src/include/catalog/indexing.h
@@ -173,6 +173,11 @@ DECLARE_UNIQUE_INDEX(pg_largeobject_loid_pn_index, 2683, on pg_largeobject using
DECLARE_UNIQUE_INDEX(pg_largeobject_metadata_oid_index, 2996, on pg_largeobject_metadata using btree(oid oid_ops));
#define LargeObjectMetadataOidIndexId 2996
+DECLARE_UNIQUE_INDEX(pg_mv_statistic_oid_index, 3277, on pg_mv_statistic using btree(oid oid_ops));
+#define MvStatisticOidIndexId 3277
+DECLARE_INDEX(pg_mv_statistic_relid_index, 3278, on pg_mv_statistic using btree(starelid oid_ops));
+#define MvStatisticRelidIndexId 3278
+
DECLARE_UNIQUE_INDEX(pg_namespace_nspname_index, 2684, on pg_namespace using btree(nspname name_ops));
#define NamespaceNameIndexId 2684
DECLARE_UNIQUE_INDEX(pg_namespace_oid_index, 2685, on pg_namespace using btree(oid oid_ops));
diff --git a/src/include/catalog/pg_mv_statistic.h b/src/include/catalog/pg_mv_statistic.h
new file mode 100644
index 0000000..76b7db7
--- /dev/null
+++ b/src/include/catalog/pg_mv_statistic.h
@@ -0,0 +1,69 @@
+/*-------------------------------------------------------------------------
+ *
+ * pg_mv_statistic.h
+ * definition of the system "multivariate statistic" relation (pg_mv_statistic)
+ * along with the relation's initial contents.
+ *
+ *
+ * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/catalog/pg_mv_statistic.h
+ *
+ * NOTES
+ * the genbki.pl script reads this file and generates .bki
+ * information from the DATA() statements.
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef PG_MV_STATISTIC_H
+#define PG_MV_STATISTIC_H
+
+#include "catalog/genbki.h"
+
+/* ----------------
+ * pg_mv_statistic definition. cpp turns this into
+ * typedef struct FormData_pg_mv_statistic
+ * ----------------
+ */
+#define MvStatisticRelationId 3281
+
+CATALOG(pg_mv_statistic,3281)
+{
+ /* These fields form the unique key for the entry: */
+ Oid starelid; /* relation containing attributes */
+
+ /* statistics requested to build */
+ bool deps_enabled; /* analyze dependencies? */
+
+ /* statistics that are available (if requested) */
+ bool deps_built; /* dependencies were built */
+
+ /* variable-length fields start here, but we allow direct access to stakeys */
+ int2vector stakeys; /* array of column keys */
+
+#ifdef CATALOG_VARLEN
+ bytea stadeps; /* dependencies (serialized) */
+#endif
+
+} FormData_pg_mv_statistic;
+
+/* ----------------
+ * Form_pg_mv_statistic corresponds to a pointer to a tuple with
+ * the format of pg_mv_statistic relation.
+ * ----------------
+ */
+typedef FormData_pg_mv_statistic *Form_pg_mv_statistic;
+
+/* ----------------
+ * compiler constants for pg_attrdef
+ * ----------------
+ */
+#define Natts_pg_mv_statistic 5
+#define Anum_pg_mv_statistic_starelid 1
+#define Anum_pg_mv_statistic_deps_enabled 2
+#define Anum_pg_mv_statistic_deps_built 3
+#define Anum_pg_mv_statistic_stakeys 4
+#define Anum_pg_mv_statistic_stadeps 5
+
+#endif /* PG_MV_STATISTIC_H */
diff --git a/src/include/catalog/pg_proc.h b/src/include/catalog/pg_proc.h
index 9edfdb8..9fb118a 100644
--- a/src/include/catalog/pg_proc.h
+++ b/src/include/catalog/pg_proc.h
@@ -2683,6 +2683,11 @@ DESCR("current user privilege on any column by rel name");
DATA(insert OID = 3029 ( has_any_column_privilege PGNSP PGUID 12 10 0 0 0 f f f f t f s 2 0 16 "26 25" _null_ _null_ _null_ _null_ has_any_column_privilege_id _null_ _null_ _null_ ));
DESCR("current user privilege on any column by rel oid");
+DATA(insert OID = 3284 ( pg_mv_stats_dependencies_info PGNSP PGUID 12 1 0 0 0 f f f f t f i 1 0 25 "17" _null_ _null_ _null_ _null_ pg_mv_stats_dependencies_info _null_ _null_ _null_ ));
+DESCR("multivariate stats: functional dependencies info");
+DATA(insert OID = 3285 ( pg_mv_stats_dependencies_show PGNSP PGUID 12 1 0 0 0 f f f f t f i 1 0 25 "17" _null_ _null_ _null_ _null_ pg_mv_stats_dependencies_show _null_ _null_ _null_ ));
+DESCR("multivariate stats: functional dependencies show");
+
DATA(insert OID = 1928 ( pg_stat_get_numscans PGNSP PGUID 12 1 0 0 0 f f f f t f s 1 0 20 "26" _null_ _null_ _null_ _null_ pg_stat_get_numscans _null_ _null_ _null_ ));
DESCR("statistics: number of scans done for table/index");
DATA(insert OID = 1929 ( pg_stat_get_tuples_returned PGNSP PGUID 12 1 0 0 0 f f f f t f s 1 0 20 "26" _null_ _null_ _null_ _null_ pg_stat_get_tuples_returned _null_ _null_ _null_ ));
diff --git a/src/include/catalog/toasting.h b/src/include/catalog/toasting.h
index cba4ae7..bf11005 100644
--- a/src/include/catalog/toasting.h
+++ b/src/include/catalog/toasting.h
@@ -49,6 +49,7 @@ extern void BootstrapToastTable(char *relName,
DECLARE_TOAST(pg_attrdef, 2830, 2831);
DECLARE_TOAST(pg_constraint, 2832, 2833);
DECLARE_TOAST(pg_description, 2834, 2835);
+DECLARE_TOAST(pg_mv_statistic, 3279, 3280);
DECLARE_TOAST(pg_proc, 2836, 2837);
DECLARE_TOAST(pg_rewrite, 2838, 2839);
DECLARE_TOAST(pg_seclabel, 3598, 3599);
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 97ef0fc..f1d79eb 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -413,6 +413,7 @@ typedef enum NodeTag
T_XmlSerialize,
T_WithClause,
T_CommonTableExpr,
+ T_StatisticsDef,
/*
* TAGS FOR REPLICATION GRAMMAR PARSE NODES (replnodes.h)
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index b1dfa85..b8700dd 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -543,6 +543,14 @@ typedef struct ColumnDef
int location; /* parse location, or -1 if none/unknown */
} ColumnDef;
+typedef struct StatisticsDef
+{
+ NodeTag type;
+ List *keys; /* String nodes naming referenced column(s) */
+ List *options; /* list of DefElem nodes */
+} StatisticsDef;
+
+
/*
* TableLikeClause - CREATE TABLE ( ... LIKE ... ) clause
*/
@@ -1340,7 +1348,8 @@ typedef enum AlterTableType
AT_ReplicaIdentity, /* REPLICA IDENTITY */
AT_EnableRowSecurity, /* ENABLE ROW SECURITY */
AT_DisableRowSecurity, /* DISABLE ROW SECURITY */
- AT_GenericOptions /* OPTIONS (...) */
+ AT_GenericOptions, /* OPTIONS (...) */
+ AT_AddStatistics /* add statistics */
} AlterTableType;
typedef struct ReplicaIdentityStmt
diff --git a/src/include/utils/mvstats.h b/src/include/utils/mvstats.h
new file mode 100644
index 0000000..2b59c2d
--- /dev/null
+++ b/src/include/utils/mvstats.h
@@ -0,0 +1,86 @@
+/*-------------------------------------------------------------------------
+ *
+ * mvstats.h
+ * Multivariate statistics and selectivity estimation functions.
+ *
+ *
+ * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/utils/mvstats.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef MVSTATS_H
+#define MVSTATS_H
+
+#include "commands/vacuum.h"
+
+/*
+ * Basic info about the stats, used when choosing what to use
+ *
+ * TODO Add info about what statistics is available (histogram, MCV,
+ * hashed MCV, functional dependencies).
+ */
+typedef struct MVStatsData {
+ Oid mvoid; /* OID of the stats in pg_mv_statistic */
+ int2vector *stakeys; /* attnums for columns in the stats */
+ bool deps_built; /* functional dependencies available */
+} MVStatsData;
+
+typedef struct MVStatsData *MVStats;
+
+
+#define MVSTATS_MAX_DIMENSIONS 8 /* max number of attributes */
+
+/* An associative rule, tracking [a => b] dependency.
+ *
+ * TODO Make this work with multiple columns on both sides.
+ */
+typedef struct MVDependencyData {
+ int16 a;
+ int16 b;
+} MVDependencyData;
+
+typedef MVDependencyData* MVDependency;
+
+typedef struct MVDependenciesData {
+ uint32 magic; /* magic constant marker */
+ int32 ndeps; /* number of dependencies */
+ MVDependency deps[1]; /* XXX why not a pointer? */
+} MVDependenciesData;
+
+typedef MVDependenciesData* MVDependencies;
+
+#define MVSTAT_DEPS_MAGIC 0xB4549A2C /* marks serialized bytea */
+#define MVSTAT_DEPS_TYPE_BASIC 1 /* basic dependencies type */
+
+/*
+ * TODO Maybe fetching the histogram/MCV list separately is inefficient?
+ * Consider adding a single `fetch_stats` method, fetching all
+ * stats specified using flags (or something like that).
+ */
+MVStats list_mv_stats(Oid relid, int *nstats, bool built_only);
+
+bytea * fetch_mv_dependencies(Oid mvoid);
+
+bytea * serialize_mv_dependencies(MVDependencies dependencies);
+
+/* deserialization of stats (serialization is private to analyze) */
+MVDependencies deserialize_mv_dependencies(bytea * data);
+
+/* FIXME this probably belongs somewhere else (not to operations stats) */
+extern Datum pg_mv_stats_dependencies_info(PG_FUNCTION_ARGS);
+extern Datum pg_mv_stats_dependencies_show(PG_FUNCTION_ARGS);
+
+MVDependencies
+build_mv_dependencies(int numrows, HeapTuple *rows,
+ int2vector *attrs,
+ int natts, VacAttrStats **vacattrstats);
+
+void build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
+ int natts, VacAttrStats **vacattrstats);
+
+void update_mv_stats(Oid relid, MVDependencies dependencies);
+
+#endif
diff --git a/src/include/utils/syscache.h b/src/include/utils/syscache.h
index ba0b090..12147ab 100644
--- a/src/include/utils/syscache.h
+++ b/src/include/utils/syscache.h
@@ -66,6 +66,7 @@ enum SysCacheIdentifier
INDEXRELID,
LANGNAME,
LANGOID,
+ MVSTATOID,
NAMESPACENAME,
NAMESPACEOID,
OPERNAMENSP,
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 80c3351..82c2659 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1353,6 +1353,14 @@ pg_matviews| SELECT n.nspname AS schemaname,
LEFT JOIN pg_namespace n ON ((n.oid = c.relnamespace)))
LEFT JOIN pg_tablespace t ON ((t.oid = c.reltablespace)))
WHERE (c.relkind = 'm'::"char");
+pg_mv_stats| SELECT n.nspname AS schemaname,
+ c.relname AS tablename,
+ s.stakeys AS attnums,
+ length(s.stadeps) AS depsbytes,
+ pg_mv_stats_dependencies_info(s.stadeps) AS depsinfo
+ FROM ((pg_mv_statistic s
+ JOIN pg_class c ON ((c.oid = s.starelid)))
+ LEFT JOIN pg_namespace n ON ((n.oid = c.relnamespace)));
pg_policies| SELECT n.nspname AS schemaname,
c.relname AS tablename,
pol.polname AS policyname,
diff --git a/src/test/regress/expected/sanity_check.out b/src/test/regress/expected/sanity_check.out
index c7be273..00f5fe7 100644
--- a/src/test/regress/expected/sanity_check.out
+++ b/src/test/regress/expected/sanity_check.out
@@ -113,6 +113,7 @@ pg_inherits|t
pg_language|t
pg_largeobject|t
pg_largeobject_metadata|t
+pg_mv_statistic|t
pg_namespace|t
pg_opclass|t
pg_operator|t
--
2.0.5
[text/x-diff] 0002-clause-reduction-using-functional-dependencies.patch (53.4K, ../../54C3FED3.1060600@2ndquadrant.com/3-0002-clause-reduction-using-functional-dependencies.patch)
download | inline diff:
From 4881b97548ed6fa8acbef153562da617ea7e58cb Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@pgaddict.com>
Date: Fri, 16 Jan 2015 22:33:41 +0100
Subject: [PATCH 2/4] clause reduction using functional dependencies
During planning, use functional dependencies to decide
which clauses to skip during cardinality estimation.
Initial and rather simplistic implementation.
FIX: second part of the rename to functional dependencies
FIX: don't build functional dependencies by default
FIX: build deps only when requested
FIX: use treat_as_join_clause() in clause_is_mv_compatible()
We don't want to process clauses that are used for joining,
but only simple WHERE clauses.
FIX: use planner_rt_fetch() to identify relation
The clause_is_mv_compatible() needs to identify the relation
(so that we can fetch the list of multivariate stats by OID).
planner_rt_fetch() seems like the appropriate way to get the
relation OID, but apparently it only works with simple vars.
Maybe examine_variable() would make this work with more
complex vars too?
FIX: comment about functional dependencies and transitivity
FIX: comment about multi-column functional dependencies
FIX: test: functional dependencies / ANALYZE
Test analyzing functional dependencies (part of ANALYZE)
on several datasets (no dependencies, no transitive
dependencies, ...).
FIX: test: clause reduction using function dependencies / EXPLAIN
Checks that a query with conditions on two columns, where one (B)
is functionally dependent on the other one (A), correctly ignores
the clause on (B) and chooses bitmap index scan instead of plain
index scan (which is what happens otherwise, thanks to assumption
of independence).
FIX: comment about building multi-column dependencies (TODO)
FIX: support functional dependencies on all data types
Until now build_mv_dependencies() only supported data types
passed by value (i.e. not varlena types or types passed by
reference). This commit adds support for these data types
by using SortSupport to do the sorting.
This however keeps the 'typbyval' assert in common.c:
Assert(stats[i]->attrtype->typbyval);
as that method is used for all types of multivariate stats
and we don't want to make this work for all of them. If
you want to play with functional dependencies on columns
with such data types, comment this assert out.
FIX: support NULL values in functional dependencies
FIX: typo in regression test of functional dependencies
FIX: added regression test for functional dependencies with TEXT
FIX: rework build_mv_dependencies() not to fail with mixed columns
FIX: readability improvement in build_mv_dependencies()
FIX: readability fixes in build_mv_dependencies()
FIX: regression test - dependencies with mix of data types / NULLs
FIX: minor formatting fixes in build_mv_dependencies()
FIX: comment about efficient building of multi-column dependencies
FIX: comment about proper NULL handling in build_mv_dependencies()
FIX: minor comment in build_mv_dependencies()
FIX: comment about handling NULLs like regular values (dependencies)
FIX: explanation of allocations in build_mv_dependencies()
FIX: move multisort typedefs/functions to common.h/c
FIX: check that at least some statistics were requested (dependencies)
FIX: comment about handling NULL values in dependencies
FIX: minor improvements in mvstat.h (functional dependencies)
FIX: add regression test for ADD STATISTICS options (dependencies)
FIX: comment about multivariate stats at clauselist_selectivity()
FIX: updated comments in clausesel.c (dependencies)
FIX: note in clauselist_selectivity()
FIX: fixed typo in tablecmds.c (comma -> semicolon)
FIX: make regression tests parallel-happy (functional dependencies)
---
src/backend/commands/tablecmds.c | 8 +-
src/backend/optimizer/path/clausesel.c | 476 +++++++++++++++++++++++++-
src/backend/utils/mvstats/common.c | 86 ++++-
src/backend/utils/mvstats/common.h | 22 ++
src/backend/utils/mvstats/dependencies.c | 170 +++++++--
src/include/utils/mvstats.h | 23 +-
src/test/regress/expected/mv_dependencies.out | 175 ++++++++++
src/test/regress/parallel_schedule | 3 +
src/test/regress/serial_schedule | 1 +
src/test/regress/sql/mv_dependencies.sql | 153 +++++++++
10 files changed, 1076 insertions(+), 41 deletions(-)
create mode 100644 src/test/regress/expected/mv_dependencies.out
create mode 100644 src/test/regress/sql/mv_dependencies.sql
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 3ec1a5a..3c82b89 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -11651,7 +11651,7 @@ static void ATExecAddStatistics(AlteredTableInfo *tab, Relation rel,
Relation mvstatrel;
/* by default build everything */
- bool build_dependencies = true;
+ bool build_dependencies = false;
Assert(IsA(def, StatisticsDef));
@@ -11713,6 +11713,12 @@ static void ATExecAddStatistics(AlteredTableInfo *tab, Relation rel,
opt->defname)));
}
+ /* check that at least some statistics were requested */
+ if (! build_dependencies)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("no statistics type (dependencies) was requested")));
+
/* sort the attnums and build int2vector */
qsort(attnums, numcols, sizeof(int16), compare_int16);
stakeys = buildint2vector(attnums, numcols);
diff --git a/src/backend/optimizer/path/clausesel.c b/src/backend/optimizer/path/clausesel.c
index dcac1c1..36e5bce 100644
--- a/src/backend/optimizer/path/clausesel.c
+++ b/src/backend/optimizer/path/clausesel.c
@@ -24,6 +24,12 @@
#include "utils/lsyscache.h"
#include "utils/selfuncs.h"
+#include "utils/mvstats.h"
+#include "catalog/pg_collation.h"
+#include "utils/typcache.h"
+
+#include "parser/parsetree.h"
+
/*
* Data structure for accumulating info about possible range-query
@@ -43,6 +49,16 @@ static void addRangeClause(RangeQueryClause **rqlist, Node *clause,
bool varonleft, bool isLTsel, Selectivity s2);
+static bool clause_is_mv_compatible(PlannerInfo *root, Node *clause, Oid varRelid,
+ Oid *relid, AttrNumber *attnum, SpecialJoinInfo *sjinfo);
+
+static Bitmapset *collect_mv_attnums(PlannerInfo *root, List *clauses,
+ Oid varRelid, Oid *relid, SpecialJoinInfo *sjinfo);
+
+static List *clauselist_apply_dependencies(PlannerInfo *root, List *clauses,
+ Oid varRelid, int nmvstats, MVStats mvstats,
+ SpecialJoinInfo *sjinfo);
+
/****************************************************************************
* ROUTINES TO COMPUTE SELECTIVITIES
****************************************************************************/
@@ -61,7 +77,7 @@ static void addRangeClause(RangeQueryClause **rqlist, Node *clause,
* subclauses. However, that's only right if the subclauses have independent
* probabilities, and in reality they are often NOT independent. So,
* we want to be smarter where we can.
-
+ *
* Currently, the only extra smarts we have is to recognize "range queries",
* such as "x > 34 AND x < 42". Clauses are recognized as possible range
* query components if they are restriction opclauses whose operators have
@@ -88,6 +104,76 @@ static void addRangeClause(RangeQueryClause **rqlist, Node *clause,
*
* Of course this is all very dependent on the behavior of
* scalarltsel/scalargtsel; perhaps some day we can generalize the approach.
+ *
+ *
+ * Multivariate statististics
+ * --------------------------
+ * This also uses multivariate stats to estimate combinations of conditions,
+ * in a way attempting to minimize the overhead when there are no suitable
+ * multivariate stats.
+ *
+ * The following checks are performed (in this order), and the optimizer
+ * falls back to regular stats on the first 'false'.
+ *
+ * NOTE: This explains how this works with all the patches applied, not
+ * just the functional dependencies.
+ *
+ * (1) check that at least two columns are referenced from conditions
+ * compatible with multivariate stats
+ *
+ * If there are no conditions that might be handled by multivariate
+ * stats, or if the conditions reference just a single column, it
+ * makes no sense to use multivariate stats.
+ *
+ * What conditions are compatible with multivariate stats is decided
+ * by clause_is_mv_compatible(). At this moment, only simple conditions
+ * of the form "column operator constant" (for simple comparison
+ * operators), and IS NULL / IS NOT NULL are considered compatible
+ * with multivariate statistics.
+ *
+ * (2) reduce the clauses using functional dependencies
+ *
+ * This simply attempts to 'reduce' the clauses by applying functional
+ * dependencies. For example if there are two clauses:
+ *
+ * WHERE (a = 1) AND (b = 2)
+ *
+ * and we know that 'a' determines the value of 'b', we may remove
+ * the second condition (b = 2) when computing the selectivity.
+ * This is of course tricky - see mvstats/dependencies.c for details.
+ *
+ * After the reduction, step (1) is to be repeated.
+ *
+ * (3) check if there are multivariate stats built on the columns
+ *
+ * If there are no multivariate statistics, we have to fall back to
+ * the regular stats. We might perform checks (1) and (2) in reverse
+ * order, i.e. first check if there are multivariate statistics and
+ * then collect the attributes only if needed. The assumption is
+ * that checking the clauses is cheaper than querying the catalog,
+ * so this check is performed first.
+ *
+ * (4) choose the stats matching the most columns (at least two)
+ *
+ * If there are multiple instances of multivariate statistics (e.g.
+ * built on different sets of columns), we choose the stats covering
+ * the most columns from step (1). It may happen that all available
+ * stats match just a single column - for example with conditions
+ *
+ * WHERE a = 1 AND b = 2
+ *
+ * and statistics built on (a,c) and (b,c). In such case just fall
+ * back to the regular stats because it makes no sense to use the
+ * multivariate statistics.
+ *
+ * This selection criteria (the most columns) is certainly very
+ * simple and definitely not optimal - it's simple to come up with
+ * examples where other approaches work better. More about this
+ * at choose_mv_statistics().
+ *
+ * (5) use the multivariate stats to estimate matching clauses
+ *
+ * (6) estimate the remaining clauses using the regular statistics
*/
Selectivity
clauselist_selectivity(PlannerInfo *root,
@@ -100,6 +186,14 @@ clauselist_selectivity(PlannerInfo *root,
RangeQueryClause *rqlist = NULL;
ListCell *l;
+ /* processing mv stats */
+ Oid relid = InvalidOid;
+ int nmvstats = 0;
+ MVStats mvstats = NULL;
+
+ /* attributes in mv-compatible clauses */
+ Bitmapset *mvattnums = NULL;
+
/*
* If there's exactly one clause, then no use in trying to match up pairs,
* so just go directly to clause_selectivity().
@@ -108,6 +202,28 @@ clauselist_selectivity(PlannerInfo *root,
return clause_selectivity(root, (Node *) linitial(clauses),
varRelid, jointype, sjinfo);
+ /* collect attributes referenced by mv-compatible clauses */
+ mvattnums = collect_mv_attnums(root, clauses, varRelid, &relid, sjinfo);
+
+ /*
+ * If there are mv-compatible clauses, referencing at least two
+ * different columns (otherwise it makes no sense to use mv stats),
+ * try to reduce the clauses using functional dependencies, and
+ * recollect the attributes from the reduced list.
+ *
+ * We don't need to select a single statistics for this - we can
+ * apply all the functional dependencies we have.
+ */
+ if (bms_num_members(mvattnums) >= 2)
+ {
+ /* fetch info from the catalog (not the serialized stats yet) */
+ mvstats = list_mv_stats(relid, &nmvstats, true);
+
+ /* reduce clauses by applying functional dependencies rules */
+ clauses = clauselist_apply_dependencies(root, clauses, varRelid,
+ nmvstats, mvstats, sjinfo);
+ }
+
/*
* Initial scan over clauses. Anything that doesn't look like a potential
* rangequery clause gets multiplied into s1 and forgotten. Anything that
@@ -782,3 +898,361 @@ clause_selectivity(PlannerInfo *root,
return s1;
}
+
+/*
+ * Collect attributes from mv-compatible clauses.
+ *
+ */
+static Bitmapset *
+collect_mv_attnums(PlannerInfo *root, List *clauses, Oid varRelid,
+ Oid *relid, SpecialJoinInfo *sjinfo)
+{
+ Bitmapset *attnums = NULL;
+ ListCell *l;
+
+ /*
+ * Walk through the clauses and identify the ones we can estimate
+ * using multivariate stats, and remember the relid/columns. We'll
+ * then cross-check if we have suitable stats, and only if needed
+ * we'll split the clauses into multivariate and regular lists.
+ *
+ * For now we're only interested in RestrictInfo nodes with nested
+ * OpExpr, using either a range or equality.
+ */
+ foreach (l, clauses)
+ {
+ AttrNumber attnum;
+ Node *clause = (Node *) lfirst(l);
+
+ /* ignore the result for now - we only need the info */
+ if (clause_is_mv_compatible(root, clause, varRelid, relid, &attnum, sjinfo))
+ attnums = bms_add_member(attnums, attnum);
+ }
+
+ /*
+ * If there are not at least two attributes referenced by the clause(s),
+ * we can throw everything out (as we'll revert to simple stats).
+ */
+ if (bms_num_members(attnums) <= 1)
+ {
+ if (attnums != NULL)
+ pfree(attnums);
+ attnums = NULL;
+ *relid = InvalidOid;
+ }
+
+ return attnums;
+}
+
+/*
+ * Determines whether the clause is compatible with multivariate stats,
+ * and if it is, returns some additional information - varno (index
+ * into simple_rte_array) and a bitmap of attributes. This is then
+ * used to fetch related multivariate statistics.
+ *
+ * At this moment we only support basic conditions of the form
+ *
+ * variable OP constant
+ *
+ * where OP is one of [=,<,<=,>=,>] (which is however determined by
+ * looking at the associated function for estimating selectivity, just
+ * like with the single-dimensional case).
+ *
+ * TODO Support 'OR clauses' - shouldn't be all that difficult to
+ * evaluate them using multivariate stats.
+ */
+static bool
+clause_is_mv_compatible(PlannerInfo *root, Node *clause, Oid varRelid,
+ Oid *relid, AttrNumber *attnum, SpecialJoinInfo *sjinfo)
+{
+
+ if (IsA(clause, RestrictInfo))
+ {
+ RestrictInfo *rinfo = (RestrictInfo *) clause;
+
+ /* Pseudoconstants are not really interesting here. */
+ if (rinfo->pseudoconstant)
+ return false;
+
+ /* no support for OR clauses at this point */
+ if (rinfo->orclause)
+ return false;
+
+ /* get the actual clause from the RestrictInfo (it's not an OR clause) */
+ clause = (Node*)rinfo->clause;
+
+ /* only simple opclauses are compatible with multivariate stats */
+ if (! is_opclause(clause))
+ return false;
+
+ /* we don't support join conditions at this moment */
+ if (treat_as_join_clause(clause, rinfo, varRelid, sjinfo))
+ return false;
+
+ /* is it 'variable op constant' ? */
+ if (list_length(((OpExpr *) clause)->args) == 2)
+ {
+ OpExpr *expr = (OpExpr *) clause;
+ bool varonleft = true;
+ bool ok;
+
+ ok = (bms_membership(rinfo->clause_relids) == BMS_SINGLETON) &&
+ (is_pseudo_constant_clause_relids(lsecond(expr->args),
+ rinfo->right_relids) ||
+ (varonleft = false,
+ is_pseudo_constant_clause_relids(linitial(expr->args),
+ rinfo->left_relids)));
+
+ if (ok)
+ {
+ RangeTblEntry * rte;
+ Var * var = (varonleft) ? linitial(expr->args) : lsecond(expr->args);
+
+ /*
+ * Simple variables only - otherwise the planner_rt_fetch seems to fail
+ * (return NULL).
+ *
+ * TODO Maybe use examine_variable() would fix that?
+ */
+ if (! (IsA(var, Var) && (varRelid == 0 || varRelid == var->varno)))
+ return false;
+
+ /*
+ * Only consider this variable if (varRelid == 0) or when the varno
+ * matches varRelid (see explanation at clause_selectivity).
+ *
+ * FIXME I suspet this may not be really necessary. The (varRelid == 0)
+ * part seems to be enforced by treat_as_join_clause().
+ */
+ if (! ((varRelid == 0) || (varRelid == var->varno)))
+ return false;
+
+ /* Also skip special varno values, and system attributes ... */
+ if ((IS_SPECIAL_VARNO(var->varno)) || (! AttrNumberIsForUserDefinedAttr(var->varattno)))
+ return false;
+
+ /* Lookup info about the base relation (we need to pass the OID out) */
+ rte = planner_rt_fetch(var->varno, root);
+ *relid = rte->relid;
+
+ /*
+ * If it's not a "<" or ">" or "=" operator, just ignore the
+ * clause. Otherwise note the relid and attnum for the variable.
+ * This uses the function for estimating selectivity, ont the
+ * operator directly (a bit awkward, but well ...).
+ */
+ switch (get_oprrest(expr->opno))
+ {
+ case F_SCALARLTSEL:
+ case F_SCALARGTSEL:
+ case F_EQSEL:
+ *attnum = var->varattno;
+ return true;
+ }
+ }
+ }
+ }
+
+ return false;
+
+}
+
+/*
+ * Performs reduction of clauses using functional dependencies, i.e.
+ * removes clauses that are considered redundant. It simply walks
+ * through dependencies, and checks whether the dependency 'matches'
+ * the clauses, i.e. if there's a clause matching the condition. If yes,
+ * all clauses matching the implied part of the dependency are removed
+ * from the list.
+ *
+ * This simply looks at attnums references by the clauses, not at the
+ * type of the operator (equality, inequality, ...). This may not be the
+ * right way to do - it certainly works best for equalities, which is
+ * naturally consistent with functional dependencies (implications).
+ * It's not clear that other operators are handled sensibly - for
+ * example for inequalities, like
+ *
+ * WHERE (A >= 10) AND (B <= 20)
+ *
+ * and a trivial case where [A == B], resulting in symmetric pair of
+ * rules [A => B], [B => A], it's rather clear we can't remove either of
+ * those clauses.
+ *
+ * That only highlights that functional dependencies are most suitable
+ * for label-like data, where using non-equality operators is very rare.
+ * Using the common city/zipcode example, clauses like
+ *
+ * (zipcode <= 12345)
+ *
+ * or
+ *
+ * (cityname >= 'Washington')
+ *
+ * are rare. So restricting the reduction to equality should not harm
+ * the usefulness / applicability.
+ *
+ * The other assumption is that this assumes 'compatible' clauses. For
+ * example by using mismatching zip code and city name, this is unable
+ * to identify the discrepancy and eliminates one of the clauses. The
+ * usual approach (multiplying both selectivities) thus produces a more
+ * accurate estimate, although mostly by luck - the multiplication
+ * comes from assumption of statistical independence of the two
+ * conditions (which is not not valid in this case), but moves the
+ * estimate in the right direction (towards 0%).
+ *
+ * This might be somewhat improved by cross-checking the selectivities
+ * against MCV and/or histogram.
+ *
+ * The implementation needs to be careful about cyclic rules, i.e. rules
+ * like [A => B] and [B => A] at the same time. This must not reduce
+ * clauses on both attributes at the same time.
+ *
+ * Technically we might consider selectivities here too, somehow. E.g.
+ * when (A => B) and (B => A), we might use the clauses with minimum
+ * selectivity.
+ *
+ * TODO Consider restricting the reduction to equality clauses. Or maybe
+ * use equality classes somehow?
+ *
+ * TODO Merge this docs to dependencies.c, as it's saying mostly the
+ * same things as the comments there.
+ */
+static List *
+clauselist_apply_dependencies(PlannerInfo *root, List *clauses, Oid varRelid,
+ int nmvstats, MVStats mvstats, SpecialJoinInfo *sjinfo)
+{
+ int i;
+ ListCell *lc;
+ List * reduced_clauses = NIL;
+ Oid relid;
+
+ /*
+ * preallocate space for all clauses, including non-mv-compatible,
+ * so that we don't need to reallocate the arrays repeatedly
+ */
+ bool *reduced = (bool*)palloc0(list_length(clauses) * sizeof(bool));
+ AttrNumber *mvattnums = (AttrNumber*)palloc0(list_length(clauses) * sizeof(AttrNumber));
+ Node **mvclauses = (Node**)palloc0(list_length(clauses) * sizeof(Node*));
+ int nmvclauses = 0; /* number of mv-compatible clauses */
+
+ /*
+ * Walk through the clauses - clauses that are not mv-compatible copy
+ * directly into the result list, and mv-compatible ones store into
+ * an array of clauses (and remember the attnumb in another array).
+ */
+ foreach (lc, clauses)
+ {
+ AttrNumber attnum;
+ Node *clause = (Node *) lfirst(lc);
+ if (! clause_is_mv_compatible(root, clause, varRelid, &relid, &attnum, sjinfo))
+ lappend(reduced_clauses, clause);
+ else
+ {
+ mvclauses[nmvclauses] = clause;
+ mvattnums[nmvclauses] = attnum;
+ nmvclauses++;
+ }
+ }
+
+ Assert(nmvclauses >= 2);
+
+ /* walk through all the mvstats, and try to apply all the rules */
+ for (i = 0; i < nmvstats; i++)
+ {
+ int j;
+ MVDependencies dependencies = NULL;
+
+ /* skip stats without functional dependencies built */
+ if (! mvstats[i].deps_built)
+ continue;
+
+ /* fetch dependencies */
+ dependencies = deserialize_mv_dependencies(fetch_mv_dependencies(mvstats->mvoid));
+ if (dependencies == NULL)
+ continue;
+
+ /*
+ * Walk through the dependencies and eliminate all the implied
+ * clauses, i.e. when there's a rule [A => B], and if we find
+ * a clause referencing column A (not yet eliminated), eliminate
+ * all clauses referencing "B".
+ *
+ * This is imperfect for a number of reasons. First, this greedy
+ * approach does not guarantee eliminating the most clauses.
+ * For example consider dependency [A => B] and [B => A], and
+ * three clauses referencing A, A and B, i.e. something like
+ *
+ * WHERE (A >= 10) AND (A <= 20) AND (B = 20)
+ *
+ * Then by considering the dependency [A => B] a single clause
+ * on B is eliminated, while by considering [B => A], both
+ * clauses on A are eliminated.
+ *
+ * The order of the dependencies may be either due to ordering
+ * within a single pg_mv_statistics record, or due to rules
+ * placed in different records.
+ *
+ * Possible solutions:
+ *
+ * (a) backtracking/recursion, with tracking of how many clauses
+ * were eliminated
+ *
+ * (b) building adjacency matrix (where A and B are adjacent
+ * when [A => B]), and multiplying it to construct
+ * transitive implications. I.e. by having [A=>B] and [B=>C]
+ * this also results in [A=>C]. Then we can simply choose
+ * the attribute that eliminates the most clauses (and
+ * repeat).
+ *
+ * We don't expect to have many clauses enough to result in long
+ * runtimes.
+ *
+ * This may also merge all the dependencies, possibly leading to
+ * longer sequences of transitive dependencies.
+ *
+ * E.g. rule [A=>B] in one pg_mv_statistic record and [B=>C] in
+ * another one results in [A=>C], which can't be deduced if the
+ * records are considered separately.
+ */
+ for (j = 0; j < dependencies->ndeps; j++)
+ {
+ int k;
+ bool applicable = false;
+
+ /* are clauses on 'A' already eliminated */
+ for (k = 0; k < nmvclauses; k++)
+ {
+ /* clause on 'A' and not yet eliminated */
+ if ((! reduced[k]) && (mvattnums[k] == dependencies->deps[j]->a))
+ {
+ applicable = true; /* we can apply this rule */
+ break;
+ }
+ }
+
+ /* if the rule is not applicable, skip to the next one */
+ if (! applicable)
+ continue;
+
+ /* eliminate all clauses on 'B ' */
+ for (k = 0; k < nmvclauses; k++)
+ {
+ if (mvattnums[k] == dependencies->deps[j]->b)
+ reduced[k] = true;
+ }
+ }
+ }
+
+ /* now walk through the clauses, and keep those that were not reduced */
+ for (i = 0; i < nmvclauses; i++)
+ {
+ if (! reduced[i])
+ reduced_clauses = lappend(reduced_clauses, mvclauses[i]);
+ }
+
+ pfree(reduced);
+ pfree(mvclauses);
+ pfree(mvattnums);
+
+ return reduced_clauses;
+}
diff --git a/src/backend/utils/mvstats/common.c b/src/backend/utils/mvstats/common.c
index 36757d5..0edaaa6 100644
--- a/src/backend/utils/mvstats/common.c
+++ b/src/backend/utils/mvstats/common.c
@@ -50,7 +50,8 @@ build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
/*
* Analyze functional dependencies of columns.
*/
- deps = build_mv_dependencies(numrows, rows, attrs, natts, vacattrstats);
+ if (mvstats->deps_enabled)
+ deps = build_mv_dependencies(numrows, rows, attrs, natts, vacattrstats);
/* store the histogram / MCV list in the catalog */
update_mv_stats(mvstats[i].mvoid, deps);
@@ -147,6 +148,7 @@ list_mv_stats(Oid relid, int *nstats, bool built_only)
result[*nstats].mvoid = HeapTupleGetOid(htup);
result[*nstats].stakeys = buildint2vector(stats->stakeys.values, stats->stakeys.dim1);
+ result[*nstats].deps_enabled = stats->deps_enabled;
result[*nstats].deps_built = stats->deps_built;
*nstats += 1;
}
@@ -270,3 +272,85 @@ compare_scalars_memcmp_2(const void *a, const void *b)
{
return memcmp(a, b, sizeof(Datum));
}
+
+
+/* initialize multi-dimensional sort */
+MultiSortSupport
+multi_sort_init(int ndims)
+{
+ MultiSortSupport mss;
+
+ Assert(ndims >= 2);
+
+ mss = (MultiSortSupport)palloc0(offsetof(MultiSortSupportData, ssup)
+ + sizeof(SortSupportData)*ndims);
+
+ mss->ndims = ndims;
+
+ return mss;
+}
+
+/*
+ * add sort into for dimension 'dim' (index into vacattrstats) to mss,
+ * at the position 'sortattr'
+ */
+void
+multi_sort_add_dimension(MultiSortSupport mss, int sortdim,
+ int dim, VacAttrStats **vacattrstats)
+{
+ /* first, lookup StdAnalyzeData for the dimension (attribute) */
+ SortSupportData ssup;
+ StdAnalyzeData *tmp = (StdAnalyzeData *)vacattrstats[dim]->extra_data;
+
+ Assert(mss != NULL);
+ Assert(sortdim < mss->ndims);
+
+ /* initialize sort support, etc. */
+ memset(&ssup, 0, sizeof(ssup));
+ ssup.ssup_cxt = CurrentMemoryContext;
+
+ /* We always use the default collation for statistics */
+ ssup.ssup_collation = DEFAULT_COLLATION_OID;
+ ssup.ssup_nulls_first = false;
+
+ PrepareSortSupportFromOrderingOp(tmp->ltopr, &ssup);
+
+ mss->ssup[sortdim] = ssup;
+}
+
+/* compare all the dimensions in the selected order */
+int
+multi_sort_compare(const void *a, const void *b, void *arg)
+{
+ int i;
+ SortItem *ia = (SortItem*)a;
+ SortItem *ib = (SortItem*)b;
+
+ MultiSortSupport mss = (MultiSortSupport)arg;
+
+ for (i = 0; i < mss->ndims; i++)
+ {
+ int compare;
+
+ compare = ApplySortComparator(ia->values[i], ia->isnull[i],
+ ib->values[i], ib->isnull[i],
+ &mss->ssup[i]);
+
+ if (compare != 0)
+ return compare;
+
+ }
+
+ /* equal by default */
+ return 0;
+}
+
+/* compare selected dimension */
+int
+multi_sort_compare_dim(int dim, const SortItem *a, const SortItem *b,
+ MultiSortSupport mss)
+{
+ return ApplySortComparator(a->values[dim], a->isnull[dim],
+ b->values[dim], b->isnull[dim],
+ &mss->ssup[dim]);
+}
diff --git a/src/backend/utils/mvstats/common.h b/src/backend/utils/mvstats/common.h
index f511c4e..b98ceb7 100644
--- a/src/backend/utils/mvstats/common.h
+++ b/src/backend/utils/mvstats/common.h
@@ -59,6 +59,28 @@ typedef struct
int *tupnoLink;
} CompareScalarsContext;
+/* multi-sort */
+typedef struct MultiSortSupportData {
+ int ndims; /* number of dimensions supported by the */
+ SortSupportData ssup[1]; /* sort support data for each dimension */
+} MultiSortSupportData;
+
+typedef MultiSortSupportData* MultiSortSupport;
+
+typedef struct SortItem {
+ Datum *values;
+ bool *isnull;
+} SortItem;
+
+MultiSortSupport multi_sort_init(int ndims);
+
+void multi_sort_add_dimension(MultiSortSupport mss, int sortdim,
+ int dim, VacAttrStats **vacattrstats);
+
+int multi_sort_compare(const void *a, const void *b, void *arg);
+
+int multi_sort_compare_dim(int dim, const SortItem *a,
+ const SortItem *b, MultiSortSupport mss);
VacAttrStats ** lookup_var_attr_stats(int2vector *attrs,
int natts, VacAttrStats **vacattrstats);
diff --git a/src/backend/utils/mvstats/dependencies.c b/src/backend/utils/mvstats/dependencies.c
index b900efd..93a2fa6 100644
--- a/src/backend/utils/mvstats/dependencies.c
+++ b/src/backend/utils/mvstats/dependencies.c
@@ -15,6 +15,7 @@
*/
#include "common.h"
+#include "utils/lsyscache.h"
/*
* Mine functional dependencies between columns, in the form (A => B),
@@ -56,6 +57,20 @@
* columns on the 'left' side, i.e. a condition for the dependency.
* That is dependencies [A,B] => C and so on.
*
+ * TODO The implementation may/should be smart enough not to mine both
+ * [A => B] and [A,C => B], because the second dependency is a
+ * consequence of the first one (if values of A determine values
+ * of B, adding another column won't change that). The ANALYZE
+ * should first analyze 1:1 dependencies, then 2:1 dependencies
+ * (and skip the already identified ones), etc.
+ *
+ * For example the dependency [city name => zip code] is much weaker
+ * than [city name, state name => zip code], because there may be
+ * multiple cities with the same name in various states. It's not
+ * perfect though - there are probably cities with the same name within
+ * the same state, but this is relatively rare occurence hopefully.
+ * More about this in the section about dependency mining.
+ *
* Handling multiple columns on the right side is not necessary, as such
* dependencies may be decomposed into a set of dependencies with
* the same meaning, one for each column on the right side. For example
@@ -163,19 +178,61 @@
* FIXME Not sure if this handles NULL values properly (not sure how to
* do that). We assume that NULL means 0 for now, handling it just
* like any other value.
+ *
+ * FIXME This builds a complete set of dependencies, i.e. including
+ * transitive dependencies - if we identify [A => B] and [B => C],
+ * we're likely to identify [A => C] too. It might be better to
+ * keep only the minimal set of dependencies, i.e. prune all the
+ * dependencies that we can recreate by transivitity.
+ *
+ * There are two conceptual ways to do that:
+ *
+ * (a) generate all the rules, and then prune the rules that may
+ * be recteated by combining other dependencies, or
+ *
+ * (b) performing the 'is combination of other dependencies' check
+ * before actually doing the work
+ *
+ * The second option has the advantage that we don't really need
+ * to perform the sort/count. It's not sufficient alone, though,
+ * because we may discover the dependencies in the wrong order.
+ * For example [A => B], [A => C] and then [B => C]. None of those
+ * dependencies is a combination of the already known ones, yet
+ * [A => C] is a combination of [A => B] and [B => C].
+ *
+ * TODO Not sure the current NULL handling makes much sense. It's
+ * handled like a regular value (NULL == NULL), so all NULLs in
+ * a single column form a single group. Maybe that's not the right
+ * thing to do, especially with equality conditions - in that case
+ * NULLs are irrelevant. So maybe the right solution would be to
+ * just ignore NULL values here?
+ *
+ * However simply "ignoring" the NULL values does not seem like
+ * a good idea - imagine columns A and B, where for each value of
+ * A, values in B are constant (same for the whole group) or NULL.
+ * Let's say only 10% of B values in each group is not NULL. Then
+ * ignoring the NULL values will result in 10x misestimate (and
+ * it's trivial to construct arbitrary errors). So maybe handling
+ * NULL values just like a regular value is the right thing here.
+ *
+ * Or maybe NULL values should be treated differently on each side
+ * of the dependency? E.g. as ignored on the left (condition) and
+ * as regular values on the right - this seems consistent with how
+ * equality clauses work, as equality clause means 'NOT NULL'.
+ * So if we say [A => B] then it may also imply "NOT NULL" on the
+ * right side.
*/
MVDependencies
build_mv_dependencies(int numrows, HeapTuple *rows, int2vector *attrs,
int natts, VacAttrStats **vacattrstats)
{
int i;
- bool isNull;
- Size len = 2 * sizeof(Datum); /* only simple associations a => b */
int numattrs = attrs->dim1;
/* result */
int ndeps = 0;
MVDependencies dependencies = NULL;
+ MultiSortSupport mss = multi_sort_init(2); /* 2 dimensions for now */
/* TODO Maybe this should be somehow related to the number of
* distinct columns in the two columns we're currently analyzing.
@@ -195,8 +252,24 @@ build_mv_dependencies(int numrows, HeapTuple *rows, int2vector *attrs,
*/
VacAttrStats **stats = lookup_var_attr_stats(attrs, natts, vacattrstats);
- /* We'll reuse the same array for all the combinations */
- Datum * values = (Datum*)palloc0(numrows * 2 * sizeof(Datum));
+ /*
+ * We'll reuse the same array for all the 2-column combinations.
+ *
+ * It's possible to sort the sample rows directly, but this seemed
+ * somehow simples / less error prone. Another option would be to
+ * allocate the arrays for each SortItem separately, but that'd be
+ * significant overhead (not just CPU, but especially memory bloat).
+ */
+ SortItem * items = (SortItem*)palloc0(numrows * sizeof(SortItem));
+
+ Datum *values = (Datum*)palloc0(sizeof(Datum) * numrows * 2);
+ bool *isnull = (bool*)palloc0(sizeof(bool) * numrows * 2);
+
+ for (i = 0; i < numrows; i++)
+ {
+ items[i].values = &values[i * 2];
+ items[i].isnull = &isnull[i * 2];
+ }
Assert(numattrs >= 2);
@@ -213,9 +286,12 @@ build_mv_dependencies(int numrows, HeapTuple *rows, int2vector *attrs,
*/
for (dima = 0; dima < numattrs; dima++)
{
+ /* prepare the sort function for the first dimension */
+ multi_sort_add_dimension(mss, 0, dima, vacattrstats);
+
for (dimb = 0; dimb < numattrs; dimb++)
{
- Datum val_a, val_b;
+ SortItem current;
/* number of groups supporting / contradicting the dependency */
int n_supporting = 0;
@@ -232,14 +308,27 @@ build_mv_dependencies(int numrows, HeapTuple *rows, int2vector *attrs,
if (dima == dimb)
continue;
+ /* prepare the sort function for the second dimension */
+ multi_sort_add_dimension(mss, 1, dimb, vacattrstats);
+
+ /* reset the values and isnull flags */
+ memset(values, 0, sizeof(Datum) * numrows * 2);
+ memset(isnull, 0, sizeof(bool) * numrows * 2);
+
/* accumulate all the data for both columns into an array and sort it */
for (i = 0; i < numrows; i++)
{
- values[i*2] = heap_getattr(rows[i], attrs->values[dima], stats[dima]->tupDesc, &isNull);
- values[i*2+1] = heap_getattr(rows[i], attrs->values[dimb], stats[dimb]->tupDesc, &isNull);
+ items[i].values[0]
+ = heap_getattr(rows[i], attrs->values[dima],
+ stats[dima]->tupDesc, &items[i].isnull[0]);
+
+ items[i].values[1]
+ = heap_getattr(rows[i], attrs->values[dimb],
+ stats[dimb]->tupDesc, &items[i].isnull[1]);
}
- qsort_arg((void *) values, numrows, sizeof(Datum) * 2, compare_scalars_memcmp, &len);
+ qsort_arg((void *) items, numrows, sizeof(SortItem),
+ multi_sort_compare, mss);
/*
* Walk through the array, split it into rows according to
@@ -254,13 +343,13 @@ build_mv_dependencies(int numrows, HeapTuple *rows, int2vector *attrs,
*/
/* start with values from the first row */
- val_a = values[0];
- val_b = values[1];
+ current = items[0];
group_size = 1;
for (i = 1; i < numrows; i++)
{
- if (values[2*i] != val_a) /* end of the group */
+ /* end of the group */
+ if (multi_sort_compare_dim(0, &items[i], ¤t, mss) != 0)
{
/*
* If there are no contradicting rows, count it as
@@ -271,36 +360,49 @@ build_mv_dependencies(int numrows, HeapTuple *rows, int2vector *attrs,
* impossible to identify [unique,unique] cases, but
* that's probably a different case. This is more
* about [zip => city] associations etc.
+ *
+ * If there are violations, count the group/rows as
+ * a violation.
+ *
+ * It may ne neither, if the group is too small (does
+ * not contain at least min_group_size rows).
*/
- n_supporting += ((n_violations == 0) && (group_size >= min_group_size)) ? 1 : 0;
- n_contradicting += (n_violations != 0) ? 1 : 0;
-
- n_supporting_rows += ((n_violations == 0) && (group_size >= min_group_size)) ? group_size : 0;
- n_contradicting_rows += (n_violations > 0) ? group_size : 0;
+ if ((n_violations == 0) && (group_size >= min_group_size))
+ {
+ n_supporting += 1;
+ n_supporting_rows += group_size;
+ }
+ else if (n_violations > 0)
+ {
+ n_contradicting += 1;
+ n_contradicting_rows += group_size;
+ }
/* current values start a new group */
- val_a = values[2*i];
- val_b = values[2*i+1];
n_violations = 0;
- group_size = 1;
+ group_size = 0;
}
- else
+ /* mismatch of a B value is contradicting */
+ else if (multi_sort_compare_dim(1, &items[i], ¤t, mss) != 0)
{
- if (values[2*i+1] != val_b) /* mismatch of a B value is contradicting */
- {
- val_b = values[2*i+1];
- n_violations += 1;
- }
-
- group_size += 1;
+ n_violations += 1;
}
+
+ current = items[i];
+ group_size += 1;
}
- /* handle the last group */
- n_supporting += ((n_violations == 0) && (group_size >= min_group_size)) ? 1 : 0;
- n_contradicting += (n_violations != 0) ? 1 : 0;
- n_supporting_rows += ((n_violations == 0) && (group_size >= min_group_size)) ? group_size : 0;
- n_contradicting_rows += (n_violations > 0) ? group_size : 0;
+ /* handle the last group (just like above) */
+ if ((n_violations == 0) && (group_size >= min_group_size))
+ {
+ n_supporting += 1;
+ n_supporting_rows += group_size;
+ }
+ else if (n_violations)
+ {
+ n_contradicting += 1;
+ n_contradicting_rows += group_size;
+ }
/*
* See if the number of rows supporting the association is at least
@@ -338,7 +440,11 @@ build_mv_dependencies(int numrows, HeapTuple *rows, int2vector *attrs,
}
}
+ pfree(items);
pfree(values);
+ pfree(isnull);
+ pfree(stats);
+ pfree(mss);
return dependencies;
}
diff --git a/src/include/utils/mvstats.h b/src/include/utils/mvstats.h
index 2b59c2d..a074253 100644
--- a/src/include/utils/mvstats.h
+++ b/src/include/utils/mvstats.h
@@ -18,24 +18,34 @@
/*
* Basic info about the stats, used when choosing what to use
- *
- * TODO Add info about what statistics is available (histogram, MCV,
- * hashed MCV, functional dependencies).
*/
typedef struct MVStatsData {
Oid mvoid; /* OID of the stats in pg_mv_statistic */
int2vector *stakeys; /* attnums for columns in the stats */
+
+ /* statistics requested in ALTER TABLE ... ADD STATISTICS */
+ bool deps_enabled; /* analyze functional dependencies */
+
+ /* available statistics (computed by ANALYZE) */
bool deps_built; /* functional dependencies available */
} MVStatsData;
typedef struct MVStatsData *MVStats;
+/*
+ * Degree of how much MCV item / histogram bucket matches a clause.
+ * This is then considered when computing the selectivity.
+ */
+#define MVSTATS_MATCH_NONE 0 /* no match at all */
+#define MVSTATS_MATCH_PARTIAL 1 /* partial match */
+#define MVSTATS_MATCH_FULL 2 /* full match */
#define MVSTATS_MAX_DIMENSIONS 8 /* max number of attributes */
-/* An associative rule, tracking [a => b] dependency.
- *
- * TODO Make this work with multiple columns on both sides.
+
+/*
+ * Functional dependencies, tracking column-level relationships (values
+ * in one column determine values in another one).
*/
typedef struct MVDependencyData {
int16 a;
@@ -61,6 +71,7 @@ typedef MVDependenciesData* MVDependencies;
* stats specified using flags (or something like that).
*/
MVStats list_mv_stats(Oid relid, int *nstats, bool built_only);
+bytea * fetch_mv_rules(Oid mvoid);
bytea * fetch_mv_dependencies(Oid mvoid);
diff --git a/src/test/regress/expected/mv_dependencies.out b/src/test/regress/expected/mv_dependencies.out
new file mode 100644
index 0000000..dbfb5cf
--- /dev/null
+++ b/src/test/regress/expected/mv_dependencies.out
@@ -0,0 +1,175 @@
+-- data type passed by value
+CREATE TABLE functional_dependencies (
+ a INT,
+ b INT,
+ c INT
+);
+-- unknown column
+ALTER TABLE functional_dependencies ADD STATISTICS (dependencies) ON (unknown_column);
+ERROR: column "unknown_column" referenced in statistics does not exist
+-- single column
+ALTER TABLE functional_dependencies ADD STATISTICS (dependencies) ON (a);
+ERROR: multivariate stats require 2 or more columns
+-- single column, duplicated
+ALTER TABLE functional_dependencies ADD STATISTICS (dependencies) ON (a, a);
+ERROR: duplicate column name in statistics definition
+-- two columns, one duplicated
+ALTER TABLE functional_dependencies ADD STATISTICS (dependencies) ON (a, a, b);
+ERROR: duplicate column name in statistics definition
+-- unknown option
+ALTER TABLE functional_dependencies ADD STATISTICS (unknown_option) ON (a, b, c);
+ERROR: unrecognized STATISTICS option "unknown_option"
+-- correct command
+ALTER TABLE functional_dependencies ADD STATISTICS (dependencies) ON (a, b, c);
+-- random data (no functional dependencies)
+INSERT INTO functional_dependencies
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+ANALYZE functional_dependencies;
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+ deps_enabled | deps_built | pg_mv_stats_dependencies_show
+--------------+------------+-------------------------------
+ t | f |
+(1 row)
+
+TRUNCATE functional_dependencies;
+-- a => b, a => c, b => c
+INSERT INTO functional_dependencies
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE functional_dependencies;
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+ deps_enabled | deps_built | pg_mv_stats_dependencies_show
+--------------+------------+-------------------------------
+ t | t | 1 => 2, 1 => 3, 2 => 3
+(1 row)
+
+TRUNCATE functional_dependencies;
+-- a => b, a => c
+INSERT INTO functional_dependencies
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE functional_dependencies;
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+ deps_enabled | deps_built | pg_mv_stats_dependencies_show
+--------------+------------+-------------------------------
+ t | t | 1 => 2, 1 => 3
+(1 row)
+
+TRUNCATE functional_dependencies;
+-- check explain (expect bitmap index scan, not plain index scan)
+INSERT INTO functional_dependencies
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX fdeps_idx ON functional_dependencies (a, b);
+ANALYZE functional_dependencies;
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+ deps_enabled | deps_built | pg_mv_stats_dependencies_show
+--------------+------------+-------------------------------
+ t | t | 1 => 2, 1 => 3, 2 => 3
+(1 row)
+
+EXPLAIN (COSTS off)
+ SELECT * FROM functional_dependencies WHERE a = 10 AND b = 5;
+ QUERY PLAN
+---------------------------------------------
+ Bitmap Heap Scan on functional_dependencies
+ Recheck Cond: ((a = 10) AND (b = 5))
+ -> Bitmap Index Scan on fdeps_idx
+ Index Cond: ((a = 10) AND (b = 5))
+(4 rows)
+
+DELETE FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+DROP TABLE functional_dependencies;
+-- varlena type (text)
+CREATE TABLE functional_dependencies (
+ a TEXT,
+ b TEXT,
+ c TEXT
+);
+ALTER TABLE functional_dependencies ADD STATISTICS (dependencies) ON (a, b, c);
+-- random data (no functional dependencies)
+INSERT INTO functional_dependencies
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+ANALYZE functional_dependencies;
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+ deps_enabled | deps_built | pg_mv_stats_dependencies_show
+--------------+------------+-------------------------------
+ t | f |
+(1 row)
+
+TRUNCATE functional_dependencies;
+-- a => b, a => c, b => c
+INSERT INTO functional_dependencies
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE functional_dependencies;
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+ deps_enabled | deps_built | pg_mv_stats_dependencies_show
+--------------+------------+-------------------------------
+ t | t | 1 => 2, 1 => 3, 2 => 3
+(1 row)
+
+TRUNCATE functional_dependencies;
+-- a => b, a => c
+INSERT INTO functional_dependencies
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE functional_dependencies;
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+ deps_enabled | deps_built | pg_mv_stats_dependencies_show
+--------------+------------+-------------------------------
+ t | t | 1 => 2, 1 => 3
+(1 row)
+
+TRUNCATE functional_dependencies;
+-- check explain (expect bitmap index scan, not plain index scan)
+INSERT INTO functional_dependencies
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX fdeps_idx ON functional_dependencies (a, b);
+ANALYZE functional_dependencies;
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+ deps_enabled | deps_built | pg_mv_stats_dependencies_show
+--------------+------------+-------------------------------
+ t | t | 1 => 2, 1 => 3, 2 => 3
+(1 row)
+
+EXPLAIN (COSTS off)
+ SELECT * FROM functional_dependencies WHERE a = '10' AND b = '5';
+ QUERY PLAN
+------------------------------------------------------------
+ Bitmap Heap Scan on functional_dependencies
+ Recheck Cond: ((a = '10'::text) AND (b = '5'::text))
+ -> Bitmap Index Scan on fdeps_idx
+ Index Cond: ((a = '10'::text) AND (b = '5'::text))
+(4 rows)
+
+DELETE FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+DROP TABLE functional_dependencies;
+-- NULL values (mix of int and text columns)
+CREATE TABLE functional_dependencies (
+ a INT,
+ b TEXT,
+ c INT,
+ d TEXT
+);
+ALTER TABLE functional_dependencies ADD STATISTICS (dependencies) ON (a, b, c, d);
+INSERT INTO functional_dependencies
+ SELECT
+ mod(i, 100),
+ (CASE WHEN mod(i, 200) = 0 THEN NULL ELSE mod(i,200) END),
+ mod(i, 400),
+ (CASE WHEN mod(i, 300) = 0 THEN NULL ELSE mod(i,600) END)
+ FROM generate_series(1,10000) s(i);
+ANALYZE functional_dependencies;
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+ deps_enabled | deps_built | pg_mv_stats_dependencies_show
+--------------+------------+----------------------------------------
+ t | t | 2 => 1, 3 => 1, 3 => 2, 4 => 1, 4 => 2
+(1 row)
+
+DELETE FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+DROP TABLE functional_dependencies;
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index e0ae2f2..c41762c 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -109,3 +109,6 @@ test: event_trigger
# run stats by itself because its delay may be insufficient under heavy load
test: stats
+
+# run tests of multivariate stats
+test: mv_dependencies
diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule
index 7f762bd..3845b0f 100644
--- a/src/test/regress/serial_schedule
+++ b/src/test/regress/serial_schedule
@@ -152,3 +152,4 @@ test: with
test: xml
test: event_trigger
test: stats
+test: mv_dependencies
diff --git a/src/test/regress/sql/mv_dependencies.sql b/src/test/regress/sql/mv_dependencies.sql
new file mode 100644
index 0000000..5d1ad52
--- /dev/null
+++ b/src/test/regress/sql/mv_dependencies.sql
@@ -0,0 +1,153 @@
+-- data type passed by value
+CREATE TABLE functional_dependencies (
+ a INT,
+ b INT,
+ c INT
+);
+
+-- unknown column
+ALTER TABLE functional_dependencies ADD STATISTICS (dependencies) ON (unknown_column);
+
+-- single column
+ALTER TABLE functional_dependencies ADD STATISTICS (dependencies) ON (a);
+
+-- single column, duplicated
+ALTER TABLE functional_dependencies ADD STATISTICS (dependencies) ON (a, a);
+
+-- two columns, one duplicated
+ALTER TABLE functional_dependencies ADD STATISTICS (dependencies) ON (a, a, b);
+
+-- unknown option
+ALTER TABLE functional_dependencies ADD STATISTICS (unknown_option) ON (a, b, c);
+
+-- correct command
+ALTER TABLE functional_dependencies ADD STATISTICS (dependencies) ON (a, b, c);
+
+-- random data (no functional dependencies)
+INSERT INTO functional_dependencies
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+
+ANALYZE functional_dependencies;
+
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+
+TRUNCATE functional_dependencies;
+
+-- a => b, a => c, b => c
+INSERT INTO functional_dependencies
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+
+ANALYZE functional_dependencies;
+
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+
+TRUNCATE functional_dependencies;
+
+-- a => b, a => c
+INSERT INTO functional_dependencies
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE functional_dependencies;
+
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+
+TRUNCATE functional_dependencies;
+
+-- check explain (expect bitmap index scan, not plain index scan)
+INSERT INTO functional_dependencies
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX fdeps_idx ON functional_dependencies (a, b);
+ANALYZE functional_dependencies;
+
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+
+EXPLAIN (COSTS off)
+ SELECT * FROM functional_dependencies WHERE a = 10 AND b = 5;
+
+DELETE FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+DROP TABLE functional_dependencies;
+
+-- varlena type (text)
+CREATE TABLE functional_dependencies (
+ a TEXT,
+ b TEXT,
+ c TEXT
+);
+
+ALTER TABLE functional_dependencies ADD STATISTICS (dependencies) ON (a, b, c);
+
+-- random data (no functional dependencies)
+INSERT INTO functional_dependencies
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+
+ANALYZE functional_dependencies;
+
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+
+TRUNCATE functional_dependencies;
+
+-- a => b, a => c, b => c
+INSERT INTO functional_dependencies
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+
+ANALYZE functional_dependencies;
+
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+
+TRUNCATE functional_dependencies;
+
+-- a => b, a => c
+INSERT INTO functional_dependencies
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE functional_dependencies;
+
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+
+TRUNCATE functional_dependencies;
+
+-- check explain (expect bitmap index scan, not plain index scan)
+INSERT INTO functional_dependencies
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX fdeps_idx ON functional_dependencies (a, b);
+ANALYZE functional_dependencies;
+
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+
+EXPLAIN (COSTS off)
+ SELECT * FROM functional_dependencies WHERE a = '10' AND b = '5';
+
+DELETE FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+DROP TABLE functional_dependencies;
+
+-- NULL values (mix of int and text columns)
+CREATE TABLE functional_dependencies (
+ a INT,
+ b TEXT,
+ c INT,
+ d TEXT
+);
+
+ALTER TABLE functional_dependencies ADD STATISTICS (dependencies) ON (a, b, c, d);
+
+INSERT INTO functional_dependencies
+ SELECT
+ mod(i, 100),
+ (CASE WHEN mod(i, 200) = 0 THEN NULL ELSE mod(i,200) END),
+ mod(i, 400),
+ (CASE WHEN mod(i, 300) = 0 THEN NULL ELSE mod(i,600) END)
+ FROM generate_series(1,10000) s(i);
+
+ANALYZE functional_dependencies;
+
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+
+DELETE FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+DROP TABLE functional_dependencies;
--
2.0.5
[text/x-diff] 0003-multivariate-MCV-lists.patch (100.6K, ../../54C3FED3.1060600@2ndquadrant.com/4-0003-multivariate-MCV-lists.patch)
download | inline diff:
From d6d169988a8ccef1e41e9620599bdfc83a192433 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Sun, 11 Jan 2015 20:15:37 +0100
Subject: [PATCH 3/4] multivariate MCV lists
- extends the pg_mv_statistic catalog (add 'mcv' fields)
- building the MCV lists during ANALYZE
- simple estimation while planning the queries
FIX: don't build MCV list by default
FIX: analyze MCV lists only when requested
FIX: improvements in clauselist_mv_selectivity_mcvlist()
FIX: comment about upper selectivity boundaries for MCV lists
FIX: switch MCV build to multi_sort functions (from dependencies)
This adds support for proper sorting (per data type), arbitrary
data types (as long as they have '<' operator) and NULL values
to building MCV lists. It still needs a fair amount of love, and
does nothing to serializing/deserializing the MCV lists.
FIX: comment about using max_mcv_items (ADD STATISTICS option)
FIX: initial support for all data types and NULL in MCV lists
This changes the serialize_mcvlist/update_mv_stats in a bit
strange way (passing VacAttrStats all over the place). This
needs to be improved, somehow, before rebasing into the MCV
part. Otherwise it'll cause needless conflicts.
FIX: fixed MCV build / removed debugging WARNING log message
FIX: refactoring lookup_var_attr_stats() - moving to common.c, static
This only includes changes in the common part + functional dependencies.
FIX: refactoring lookup_var_attr_stats() / MCV lists
FIX: a set of regression tests for MCV lists
This is mostly equal to a combination of all the regression tests
for functional dependencies.
One of the tests (EXPLAIN with TEXT columns) currently fails, and
produces Index Scan instead of Bitmap Index Scan. Will investigate.
FIX: comment about memory corruption in deserializing MCV list
FIX: correct MCV spelling on a few places (MVC -> MCV)
FIX: get rid of the custom comparators in mcv.c
FIX: use USE_ASSERT_CHECKING for assert-only variable (MCV)
FIX: check 'mcv' and 'mcv_max_items' options in ADD STATISTICS
FIX: proper handling of 'mcv_max_items' options (constants etc.)
FIX: check that either dependencies or MCV were requested
FIX: improved comments / docs for MCV lists
FIX: move DimensionInfo to common.h
FIX: move MCV list definitions after functional dependencies
FIX: incorrect memcpy() when building MCV list, causing segfaults
FIX: replace variables by macros in MCV serialize/deserialize
FIX: rework clauselist_mv_split() to call clause_is_mv_compatible()
Mostly duplicated the code, making it difficult to add more clause
types etc.
FIX: fixed estimation of equality clauses using MCV lists
FIX: add support for 'IS [NOT] NULL' support to MCV lists
FIX: add regression test for ADD STATISTICS options (MCV list)
FIX: added regression test to test IS [NOT] NULL with MCV lists
FIX: updated comments in clausesel.c (mcv)
FIX: obsolete Assert in mcv code (indexes -> ITEM_INDEXES)
FIX: make regression tests parallel-happy (MCV lists)
---
src/backend/catalog/system_views.sql | 4 +-
src/backend/commands/tablecmds.c | 47 +-
src/backend/optimizer/path/clausesel.c | 788 ++++++++++++++++++++++-
src/backend/utils/mvstats/Makefile | 2 +-
src/backend/utils/mvstats/common.c | 65 +-
src/backend/utils/mvstats/common.h | 12 +-
src/backend/utils/mvstats/dependencies.c | 13 +-
src/backend/utils/mvstats/mcv.c | 1002 ++++++++++++++++++++++++++++++
src/include/catalog/pg_mv_statistic.h | 18 +-
src/include/catalog/pg_proc.h | 2 +
src/include/utils/mvstats.h | 68 +-
src/test/regress/expected/mv_mcv.out | 210 +++++++
src/test/regress/expected/rules.out | 4 +-
src/test/regress/parallel_schedule | 2 +-
src/test/regress/serial_schedule | 1 +
src/test/regress/sql/mv_mcv.sql | 181 ++++++
16 files changed, 2370 insertions(+), 49 deletions(-)
create mode 100644 src/backend/utils/mvstats/mcv.c
create mode 100644 src/test/regress/expected/mv_mcv.out
create mode 100644 src/test/regress/sql/mv_mcv.sql
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index da957fc..8acf160 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -158,7 +158,9 @@ CREATE VIEW pg_mv_stats AS
C.relname AS tablename,
S.stakeys AS attnums,
length(S.stadeps) as depsbytes,
- pg_mv_stats_dependencies_info(S.stadeps) as depsinfo
+ pg_mv_stats_dependencies_info(S.stadeps) as depsinfo,
+ length(S.stamcv) AS mcvbytes,
+ pg_mv_stats_mcvlist_info(S.stamcv) AS mcvinfo
FROM (pg_mv_statistic S JOIN pg_class C ON (C.oid = S.starelid))
LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace);
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 3c82b89..1f08c1c 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -11651,7 +11651,13 @@ static void ATExecAddStatistics(AlteredTableInfo *tab, Relation rel,
Relation mvstatrel;
/* by default build everything */
- bool build_dependencies = false;
+ bool build_dependencies = false,
+ build_mcv = false;
+
+ int32 max_mcv_items = -1;
+
+ /* options required because of other options */
+ bool require_mcv = false;
Assert(IsA(def, StatisticsDef));
@@ -11706,6 +11712,29 @@ static void ATExecAddStatistics(AlteredTableInfo *tab, Relation rel,
if (strcmp(opt->defname, "dependencies") == 0)
build_dependencies = defGetBoolean(opt);
+ else if (strcmp(opt->defname, "mcv") == 0)
+ build_mcv = defGetBoolean(opt);
+ else if (strcmp(opt->defname, "max_mcv_items") == 0)
+ {
+ max_mcv_items = defGetInt32(opt);
+
+ /* this option requires 'mcv' to be enabled */
+ require_mcv = true;
+
+ /* sanity check */
+ if (max_mcv_items < MVSTAT_MCVLIST_MIN_ITEMS)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("max number of MCV items must be at least %d",
+ MVSTAT_MCVLIST_MIN_ITEMS)));
+
+ else if (max_mcv_items > MVSTAT_MCVLIST_MAX_ITEMS)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("max number of MCV items is %d",
+ MVSTAT_MCVLIST_MAX_ITEMS)));
+
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -11714,10 +11743,16 @@ static void ATExecAddStatistics(AlteredTableInfo *tab, Relation rel,
}
/* check that at least some statistics were requested */
- if (! build_dependencies)
+ if (! (build_dependencies || build_mcv))
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("no statistics type (dependencies) was requested")));
+ errmsg("no statistics type (dependencies, mcv) was requested")));
+
+ /* now do some checking of the options */
+ if (require_mcv && (! build_mcv))
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("option 'mcv' is required by other options(s)")));
/* sort the attnums and build int2vector */
qsort(attnums, numcols, sizeof(int16), compare_int16);
@@ -11733,9 +11768,13 @@ static void ATExecAddStatistics(AlteredTableInfo *tab, Relation rel,
values[Anum_pg_mv_statistic_starelid-1] = ObjectIdGetDatum(RelationGetRelid(rel));
values[Anum_pg_mv_statistic_stakeys -1] = PointerGetDatum(stakeys);
+
values[Anum_pg_mv_statistic_deps_enabled -1] = BoolGetDatum(build_dependencies);
+ values[Anum_pg_mv_statistic_mcv_enabled -1] = BoolGetDatum(build_mcv);
+ values[Anum_pg_mv_statistic_mcv_max_items -1] = Int32GetDatum(max_mcv_items);
- nulls[Anum_pg_mv_statistic_stadeps -1] = true;
+ nulls[Anum_pg_mv_statistic_stadeps -1] = true;
+ nulls[Anum_pg_mv_statistic_stamcv -1] = true;
/* insert the tuple into pg_mv_statistic */
mvstatrel = heap_open(MvStatisticRelationId, RowExclusiveLock);
diff --git a/src/backend/optimizer/path/clausesel.c b/src/backend/optimizer/path/clausesel.c
index 36e5bce..1446fa0 100644
--- a/src/backend/optimizer/path/clausesel.c
+++ b/src/backend/optimizer/path/clausesel.c
@@ -59,6 +59,18 @@ static List *clauselist_apply_dependencies(PlannerInfo *root, List *clauses,
Oid varRelid, int nmvstats, MVStats mvstats,
SpecialJoinInfo *sjinfo);
+static int choose_mv_statistics(int nmvstats, MVStats mvstats,
+ Bitmapset *attnums);
+static List *clauselist_mv_split(PlannerInfo *root, SpecialJoinInfo *sjinfo,
+ List *clauses, Oid varRelid,
+ List **mvclauses, MVStats mvstats);
+
+static Selectivity clauselist_mv_selectivity(PlannerInfo *root,
+ List *clauses, MVStats mvstats);
+static Selectivity clauselist_mv_selectivity_mcvlist(PlannerInfo *root,
+ List *clauses, MVStats mvstats,
+ bool *fullmatch, Selectivity *lowsel);
+
/****************************************************************************
* ROUTINES TO COMPUTE SELECTIVITIES
****************************************************************************/
@@ -195,8 +207,8 @@ clauselist_selectivity(PlannerInfo *root,
Bitmapset *mvattnums = NULL;
/*
- * If there's exactly one clause, then no use in trying to match up pairs,
- * so just go directly to clause_selectivity().
+ * If there's exactly one clause, then no use in trying to match up
+ * pairs, so just go directly to clause_selectivity().
*/
if (list_length(clauses) == 1)
return clause_selectivity(root, (Node *) linitial(clauses),
@@ -222,6 +234,46 @@ clauselist_selectivity(PlannerInfo *root,
/* reduce clauses by applying functional dependencies rules */
clauses = clauselist_apply_dependencies(root, clauses, varRelid,
nmvstats, mvstats, sjinfo);
+
+ /*
+ * recollect attributes from mv-compatible clauses (maybe we've
+ * removed so many clauses we have a single mv-compatible attnum)
+ */
+ mvattnums = collect_mv_attnums(root, clauses, varRelid, &relid, sjinfo);
+ }
+
+ /*
+ * If there still are at least two columns, we'll try to select
+ * a suitable multivariate stats.
+ */
+ if (bms_num_members(mvattnums) >= 2)
+ {
+ /* fetch info from the catalog (not the serialized stats yet) */
+ mvstats = list_mv_stats(relid, &nmvstats, true);
+
+ /* see choose_mv_statistics() for details */
+ if (nmvstats > 0)
+ {
+ int idx = choose_mv_statistics(nmvstats, mvstats, mvattnums);
+
+ if (idx >= 0) /* we have a matching stats */
+ {
+ MVStats mvstat = &mvstats[idx];
+
+ /* clauses compatible with multi-variate stats */
+ List *mvclauses = NIL;
+
+ /* split the clauselist into regular and mv-clauses */
+ clauses = clauselist_mv_split(root, sjinfo, clauses,
+ varRelid, &mvclauses, mvstat);
+
+ /* we've chosen the histogram to match the clauses */
+ Assert(mvclauses != NIL);
+
+ /* compute the multivariate stats */
+ s1 *= clauselist_mv_selectivity(root, mvclauses, mvstat);
+ }
+ }
}
/*
@@ -899,6 +951,192 @@ clause_selectivity(PlannerInfo *root,
return s1;
}
+
+/*
+ * Estimate selectivity for the list of MV-compatible clauses, using that
+ * particular histogram.
+ *
+ * When we hit a single bucket, we don't know what portion of it actually
+ * matches the clauses (e.g. equality), and we use 1/2 the bucket by
+ * default. However, the MV histograms are usually less detailed than
+ * the per-column ones, meaning the sum of buckets is often quite high
+ * (thanks to combining a lot of "partially hit" buckets).
+ *
+ * There are several ways to improve this, usually with cases when it
+ * won't really help. Also, the more complex the process, the worse
+ * the failures (i.e. misestimates).
+ *
+ * (1) Use the MV histogram only as a way to combine multiple
+ * per-column histograms, essentially rewriting
+ *
+ * P(A & B) = P(A) * P(B|A)
+ *
+ * where P(B|A) may be computed using a proper "slice" of the
+ * histogram, by first selecting only buckets where A is true, and
+ * then using the boundaries to 'restrict' the per-colunm histogram.
+ *
+ * With more clauses, it gets more complicated, of course
+ *
+ * P(A & B & C) = P(A & C) * P(B|A & C)
+ * = P(A) * P(C|A) * P(B|A & C)
+ *
+ * and so on.
+ *
+ * Of course, the question is how well and efficiently we can
+ * compute the conditional probabilities - whether this approach
+ * can improve the estimates (instead of amplifying the errors).
+ *
+ * Also, this does not eliminate the need for histogram on [A,B,C].
+ *
+ * (2) Use multiple smaller (and more accurate) histograms, and combine
+ * them using a process similar to the above. E.g. by assuming that
+ * B and C are independent, we can rewrite
+ *
+ * P(B|A & C) = P(B|A)
+ *
+ * so we can rewrite the whole formula to
+ *
+ * P(A & B & C) = P(A) * P(C|A) * P(B|A)
+ *
+ * and we're OK with two 2D histograms [A,C] and [A,B].
+ *
+ * It'd be nice to perform some sort of statistical test (Fisher
+ * or another chi-squared test) to identify independent components
+ * and automatically separate them into smaller histograms.
+ *
+ * (3) Using the estimated number of distinct values in a bucket to
+ * decide the selectivity of equality in the bucket (instead of
+ * blindly using 1/2 of the bucket, we may use 1/ndistinct).
+ * Of course, if the ndistinct estimate is way off, or when the
+ * distribution is not uniform (one distict items get much more
+ * items), this will fail. Also, we currently don't have ndistinct
+ * estimate available at this moment (but it shouldn't be that
+ * difficult to compute as ndistinct and ntuples should be available).
+ *
+ * TODO Clamp the selectivity by min of the per-clause selectivities
+ * (i.e. the selectivity of the most restrictive clause), because
+ * that's the maximum we can ever get from ANDed list of clauses.
+ * This may probably prevent issues with hitting too many buckets
+ * and low precision histograms.
+ *
+ * TODO We may support some additional conditions, most importantly
+ * those matching multiple columns (e.g. "a = b" or "a < b").
+ * Ultimately we could track multi-table histograms for join
+ * cardinality estimation.
+ *
+ * TODO Currently this is only estimating all clauses, or clauses
+ * matching varRelid (when it's not 0). I'm not sure what's the
+ * purpose of varRelid, but my assumption is this is used for
+ * join conditions and such. In that case we can use those clauses
+ * to restrict the other (i.e. filter the histogram buckets first,
+ * before estimating the other clauses). This is essentially equal
+ * to computing P(A|B) where "B" are the clauses not matching the
+ * varRelid.
+ *
+ * TODO Further thoughts on processing equality clauses - maybe it'd be
+ * better to look for stats (with MCV) covered by the equality
+ * clauses, because then we have a chance to find an exact match
+ * in the MCV list, which is pretty much the best we can do. We may
+ * also look at the least frequent MCV item, and use it as a upper
+ * boundary for the selectivity (had there been a more frequent
+ * item, it'd be in the MCV list).
+ *
+ * These conditions may then be used as a condition for the other
+ * selectivities, i.e. we may estimate P(A,B) first, and then
+ * compute P(C|A,B) from another histogram. This may be useful when
+ * we can estimate P(A,B) accurately (e.g. because it's a complete
+ * equality match evaluated on MCV list), and then compute the
+ * conditional probability P(C|A,B), giving us the requested stats
+ *
+ * P(A,B,C) = P(A,B) * P(C|A,B)
+ *
+ * TODO There are several options for 'sanity clamping' the estimates.
+ *
+ * First, if we have selectivities for each condition, then
+ *
+ * P(A,B) <= MIN(P(A), P(B))
+ *
+ * Because additional conditions (connected by AND) can only lower
+ * the probability.
+ *
+ * So we can do some basic sanity checks using the single-variate
+ * stats (the ones we have right now).
+ *
+ * Second, when we have multivariate stats with a MCV list, then
+ *
+ * (a) if we have a full equality condition (one equality condition
+ * on each column) and we found a match in the MCV list, this is
+ * the selectivity (and it's supposed to be exact)
+ *
+ * (b) if we have a full equality condition and we haven't found a
+ * match in the MCV list, then the selectivity is below the
+ * lowest selectivity in the MCV list
+ *
+ * (c) if we have a equality condition (not full), we can still
+ * search the MCV for matches and use the sum of probabilities
+ * as a lower boundary for the histogram (if there are no
+ * matches in the MCV list, then we have no boundary)
+ *
+ * Third, if there are multiple multivariate stats for a set of
+ * clauses, we may compute all of them and then somehow aggregate
+ * them - e.g. by choosing the minimum, median or average. The
+ * multi-variate stats are susceptible to overestimation (because
+ * we take 50% of the bucket for partial matches). Some stats may
+ * give better estimates than others, but it's very difficult to
+ * say determine that in advance which one is the best (it depends
+ * on the number of buckets, number of additional columns not
+ * referenced in the clauses etc.) so we may compute all and then
+ * choose a sane aggregation (minimum seems like a good approach).
+ * Of course, this may result in longer / more expensive estimation
+ * (CPU-wise), but it may be worth it.
+ *
+ * There are ways to address this, though. First, it's possible to
+ * add a GUC choosing whether to do a 'simple' (using a single
+ * stats expected to give the best estimate) and 'complex' (combining
+ * the multiple estimates).
+ *
+ * multivariate_estimates = (simple|full)
+ *
+ * Also, this might be enabled at a table level, by something like
+ *
+ * ALTER TABLE ... SET STATISTICS (simple|full)
+ *
+ * Which would make it possible to use this only for the tables
+ * where the simple approach does not work.
+ *
+ * Also, there are ways to optimize this algorithmically. E.g. we
+ * may try to get an estimate from a matching MCV list first, and
+ * if we happen to get a "full equality match" we may stop computing
+ * the estimates from other stats (for this condition) because
+ * that's probably the best estimate we can really get.
+ *
+ * TODO When applying the clauses to the histogram/MCV list, we can do
+ * that from the most selective clauses first, because that'll
+ * eliminate the buckets/items sooner (so we'll be able to skip
+ * them without inspection, which is more expensive).
+ */
+static Selectivity
+clauselist_mv_selectivity(PlannerInfo *root, List *clauses, MVStats mvstats)
+{
+ bool fullmatch = false;
+
+ /*
+ * Lowest frequency in the MCV list (may be used as an upper bound
+ * for full equality conditions that did not match any MCV item).
+ */
+ Selectivity mcv_low = 0.0;
+
+ /* TODO Evaluate simple 1D selectivities, use the smallest one as
+ * an upper bound, product as lower bound, and sort the
+ * clauses in ascending order by selectivity (to optimize the
+ * MCV/histogram evaluation).
+ */
+
+ /* Evaluate the MCV selectivity */
+ return clauselist_mv_selectivity_mcvlist(root, clauses, mvstats,
+ &fullmatch, &mcv_low);
+}
+
/*
* Collect attributes from mv-compatible clauses.
*
@@ -945,6 +1183,175 @@ collect_mv_attnums(PlannerInfo *root, List *clauses, Oid varRelid,
}
/*
+ * We're looking for statistics matching at least 2 attributes,
+ * referenced in the clauses compatible with multivariate statistics.
+ * The current selection criteria is very simple - we choose the
+ * statistics referencing the most attributes.
+ *
+ * If there are multiple statistics referencing the same number of
+ * columns (from the clauses), the one with less source columns
+ * (as listed in the ADD STATISTICS when creating the statistics) wins.
+ * Other wise the first one wins.
+ *
+ * This is a very simple criteria, and has several weaknesses:
+ *
+ * (a) does not consider the accuracy of the statistics
+ *
+ * If there are two histograms built on the same set of columns,
+ * but one has 100 buckets and the other one has 1000 buckets (thus
+ * likely providing better estimates), this is not currently
+ * considered.
+ *
+ * (b) does not consider the type of statistics
+ *
+ * If there are three statistics - one containing just a MCV list,
+ * another one with just a histogram and a third one with both,
+ * this is not considered.
+ *
+ * (c) does not consider the number of clauses
+ *
+ * As explained, only the number of referenced attributes counts,
+ * so if there are multiple clauses on a single attribute, this
+ * still counts as a single attribute.
+ *
+ * (d) does not consider type of condition
+ *
+ * Some clauses may work better with some statistics - for example
+ * equality clauses probably work better with MCV lists than with
+ * histograms. But IS [NOT] NULL conditions may often work better
+ * with histograms (thanks to NULL-buckets).
+ *
+ * So for example with five WHERE conditions
+ *
+ * WHERE (a = 1) AND (b = 1) AND (c = 1) AND (d = 1) AND (e = 1)
+ *
+ * and statistics on (a,b), (a,b,e) and (a,b,c,d), the last one will be
+ * selected as it references the most columns.
+ *
+ * Once we have selected the multivariate statistics, we split the list
+ * of clauses into two parts - conditions that are compatible with the
+ * selected stats, and conditions are estimated using simple statistics.
+ *
+ * From the example above, conditions
+ *
+ * (a = 1) AND (b = 1) AND (c = 1) AND (d = 1)
+ *
+ * will be estimated using the multivariate statistics (a,b,c,d) while
+ * the last condition (e = 1) will get estimated using the regular ones.
+ *
+ * There are various alternative selection criteria (e.g. counting
+ * conditions instead of just referenced attributes), but eventually
+ * the best option should be to combine multiple statistics. But that's
+ * much harder to do correctly.
+ *
+ * TODO Select multiple statistics and combine them when computing
+ * the estimate.
+ *
+ * TODO This will probably have to consider compatibility of clauses,
+ * because 'dependencies' will probably work only with equality
+ * clauses.
+ */
+static int
+choose_mv_statistics(int nmvstats, MVStats mvstats, Bitmapset *attnums)
+{
+ int i, j;
+
+ int choice = -1;
+ int current_matches = 1; /* goal #1: maximize */
+ int current_dims = (MVSTATS_MAX_DIMENSIONS+1); /* goal #2: minimize */
+
+ /*
+ * Walk through the statistics (simple array with nmvstats elements)
+ * and for each one count the referenced attributes (encoded in
+ * the 'attnums' bitmap).
+ */
+ for (i = 0; i < nmvstats; i++)
+ {
+ /* columns matching this statistics */
+ int matches = 0;
+
+ int2vector * attrs = mvstats[i].stakeys;
+ int numattrs = mvstats[i].stakeys->dim1;
+
+ /* count columns covered by the histogram */
+ for (j = 0; j < numattrs; j++)
+ if (bms_is_member(attrs->values[j], attnums))
+ matches++;
+
+ /*
+ * Use this statistics when it improves the number of matches or
+ * when it matches the same number of attributes but is smaller.
+ */
+ if ((matches > current_matches) ||
+ ((matches == current_matches) && (current_dims > numattrs)))
+ {
+ choice = i;
+ current_matches = matches;
+ current_dims = numattrs;
+ }
+ }
+
+ return choice;
+}
+
+/*
+ * This splits the clauses list into two parts - one containing clauses
+ * that will be evaluated using the chosen histogram, and the remaining
+ * clauses (either non-mvcompatible, or not related to the histogram).
+ */
+static List *
+clauselist_mv_split(PlannerInfo *root, SpecialJoinInfo *sjinfo,
+ List *clauses, Oid varRelid, List **mvclauses,
+ MVStats mvstats)
+{
+ int i;
+ ListCell *l;
+ List *non_mvclauses = NIL;
+
+ /* FIXME is there a better way to get info on int2vector? */
+ int2vector * attrs = mvstats->stakeys;
+ int numattrs = mvstats->stakeys->dim1;
+
+ /* erase the list of mv-compatible clauses */
+ *mvclauses = NIL;
+
+ foreach (l, clauses)
+ {
+ bool match = false; /* by default not mv-compatible */
+ AttrNumber attnum;
+ Node *clause = (Node *) lfirst(l);
+
+ if (clause_is_mv_compatible(root, clause, varRelid, NULL, &attnum, sjinfo))
+ {
+ /* Is the attribute part of the selected stats? */
+ for (i = 0; i < numattrs; i++)
+ if (attrs->values[i] == attnum)
+ match = true;
+ }
+
+ if (match)
+ {
+ /*
+ * The clause matches the selected stats, so extract the
+ * clause from the RestrictInfo and put it to the
+ * multivariate list. We'll use it directly.
+ */
+ RestrictInfo * rinfo = (RestrictInfo *) clause;
+ *mvclauses = lappend(*mvclauses, (Node*)rinfo->clause);
+ }
+ else
+ non_mvclauses = lappend(non_mvclauses, clause);
+ }
+
+ /*
+ * Perform regular estimation using the clauses incompatible
+ * with the chosen histogram (or MV stats in general).
+ */
+ return non_mvclauses;
+
+}
+
+/*
* Determines whether the clause is compatible with multivariate stats,
* and if it is, returns some additional information - varno (index
* into simple_rte_array) and a bitmap of attributes. This is then
@@ -981,21 +1388,23 @@ clause_is_mv_compatible(PlannerInfo *root, Node *clause, Oid varRelid,
/* get the actual clause from the RestrictInfo (it's not an OR clause) */
clause = (Node*)rinfo->clause;
- /* only simple opclauses are compatible with multivariate stats */
- if (! is_opclause(clause))
- return false;
-
/* we don't support join conditions at this moment */
if (treat_as_join_clause(clause, rinfo, varRelid, sjinfo))
return false;
- /* is it 'variable op constant' ? */
- if (list_length(((OpExpr *) clause)->args) == 2)
+ /*
+ * Only simple opclauses and IS NULL tests are compatible with
+ * multivariate stats at this point.
+ */
+ if ((is_opclause(clause))
+ && (list_length(((OpExpr *) clause)->args) == 2))
{
OpExpr *expr = (OpExpr *) clause;
bool varonleft = true;
bool ok;
+ /* is it 'variable op constant' ? */
+
ok = (bms_membership(rinfo->clause_relids) == BMS_SINGLETON) &&
(is_pseudo_constant_clause_relids(lsecond(expr->args),
rinfo->right_relids) ||
@@ -1032,8 +1441,11 @@ clause_is_mv_compatible(PlannerInfo *root, Node *clause, Oid varRelid,
return false;
/* Lookup info about the base relation (we need to pass the OID out) */
- rte = planner_rt_fetch(var->varno, root);
- *relid = rte->relid;
+ if (relid != NULL)
+ {
+ rte = planner_rt_fetch(var->varno, root);
+ *relid = rte->relid;
+ }
/*
* If it's not a "<" or ">" or "=" operator, just ignore the
@@ -1051,6 +1463,45 @@ clause_is_mv_compatible(PlannerInfo *root, Node *clause, Oid varRelid,
}
}
}
+ else if (IsA(clause, NullTest)
+ && IsA(((NullTest*)clause)->arg, Var))
+ {
+ RangeTblEntry * rte;
+ Var * var = (Var*)((NullTest*)clause)->arg;
+
+ /*
+ * Simple variables only - otherwise the planner_rt_fetch seems to fail
+ * (return NULL).
+ *
+ * TODO Maybe use examine_variable() would fix that?
+ */
+ if (! (IsA(var, Var) && (varRelid == 0 || varRelid == var->varno)))
+ return false;
+
+ /*
+ * Only consider this variable if (varRelid == 0) or when the varno
+ * matches varRelid (see explanation at clause_selectivity).
+ *
+ * FIXME I suspet this may not be really necessary. The (varRelid == 0)
+ * part seems to be enforced by treat_as_join_clause().
+ */
+ if (! ((varRelid == 0) || (varRelid == var->varno)))
+ return false;
+
+ /* Also skip special varno values, and system attributes ... */
+ if ((IS_SPECIAL_VARNO(var->varno)) || (! AttrNumberIsForUserDefinedAttr(var->varattno)))
+ return false;
+
+ /* Lookup info about the base relation (we need to pass the OID out) */
+ if (relid != NULL)
+ {
+ rte = planner_rt_fetch(var->varno, root);
+ *relid = rte->relid;
+ }
+ *attnum = var->varattno;
+
+ return true;
+ }
}
return false;
@@ -1256,3 +1707,320 @@ clauselist_apply_dependencies(PlannerInfo *root, List *clauses, Oid varRelid,
return reduced_clauses;
}
+
+/*
+ * Estimate selectivity of clauses using a MCV list.
+ *
+ * If there's no MCV list for the stats, the function returns 0.0.
+ *
+ * While computing the estimate, the function checks whether all the
+ * columns were matched with an equality condition. If that's the case,
+ * we can skip processing the histogram, as there can be no rows in
+ * it with the same values - all the rows matching the condition are
+ * represented by the MCV item. This can only happen with equality
+ * on all the attributes.
+ *
+ * The algorithm works like this:
+ *
+ * 1) mark all items as 'match'
+ * 2) walk through all the clauses
+ * 3) for a particular clause, walk through all the items
+ * 4) skip items that are already 'no match'
+ * 5) check clause for items that still match
+ * 6) sum frequencies for items to get selectivity
+ *
+ * The function also returns the frequency of the least frequent item
+ * on the MCV list, which may be useful for clamping estimate from the
+ * histogram (all items not present in the MCV list are less frequent).
+ * This however seems useful only for cases with conditions on all
+ * attributes.
+ *
+ * TODO This only handles AND-ed clauses, but it might work for OR-ed
+ * lists too - it just needs to reverse the logic a bit. I.e. start
+ * with 'no match' for all items, and mark the items as a match
+ * as the clauses are processed (and skip items that are 'match').
+ */
+static Selectivity
+clauselist_mv_selectivity_mcvlist(PlannerInfo *root, List *clauses,
+ MVStats mvstats, bool *fullmatch,
+ Selectivity *lowsel)
+{
+ int i;
+ Selectivity s = 0.0;
+ ListCell * l;
+ MCVList mcvlist = NULL;
+ int nmatches = 0;
+
+ char * matches = NULL; /* match/mismatch for each MCV item */
+ Bitmapset *eqmatches = NULL; /* attributes with equality matches */
+
+ /* there's no MCV list built yet */
+ if (! mvstats->mcv_built)
+ return 0.0;
+
+ mcvlist = deserialize_mv_mcvlist(fetch_mv_mcvlist(mvstats->mvoid));
+
+ Assert(mcvlist != NULL);
+ Assert(clauses != NIL);
+ Assert(mcvlist->nitems > 0);
+ Assert(list_length(clauses) >= 2);
+
+ /* by default all the MCV items match the clauses fully */
+ matches = palloc0(sizeof(char) * mcvlist->nitems);
+ memset(matches, MVSTATS_MATCH_FULL, sizeof(char)*mcvlist->nitems);
+
+ /* frequency of the lowest MCV item */
+ *lowsel = 1.0;
+
+ /* number of matching MCV items */
+ nmatches = mcvlist->nitems;
+
+ /*
+ * Loop through the list of clauses, and for each of them evaluate
+ * all the MCV items not yet eliminated by the preceding clauses.
+ *
+ * FIXME This would probably deserve a refactoring, I guess. Unify
+ * the two loops and put the checks inside, or something like
+ * that.
+ */
+ foreach (l, clauses)
+ {
+ Node * clause = (Node*)lfirst(l);
+
+ /* if there are no remaining matches possible, we can stop */
+ if (nmatches == 0)
+ break;
+
+ /* it's either OpClause, or NullTest */
+ if (is_opclause(clause))
+ {
+ OpExpr * expr = (OpExpr*)clause;
+ bool varonleft = true;
+ bool ok;
+
+ /* operator */
+ FmgrInfo opproc;
+
+ fmgr_info(get_opcode(expr->opno), &opproc);
+
+ ok = (NumRelids(clause) == 1) &&
+ (is_pseudo_constant_clause(lsecond(expr->args)) ||
+ (varonleft = false,
+ is_pseudo_constant_clause(linitial(expr->args))));
+
+ if (ok)
+ {
+
+ FmgrInfo ltproc, gtproc;
+ Var * var = (varonleft) ? linitial(expr->args) : lsecond(expr->args);
+ Const * cst = (varonleft) ? lsecond(expr->args) : linitial(expr->args);
+ bool isgt = (! varonleft);
+
+ /*
+ * TODO Fetch only when really needed (probably for equality only)
+ * TODO Technically either lt/gt is sufficient.
+ *
+ * FIXME The code in analyze.c creates histograms only for types
+ * with enough ordering (by calling get_sort_group_operators).
+ * Is this the same assumption, i.e. are we certain that we
+ * get the ltproc/gtproc every time we ask? Or are there types
+ * where get_sort_group_operators returns ltopr and here we
+ * get nothing?
+ */
+ TypeCacheEntry *typecache
+ = lookup_type_cache(var->vartype,
+ TYPECACHE_EQ_OPR | TYPECACHE_LT_OPR | TYPECACHE_GT_OPR);
+
+ /* FIXME proper matching attribute to dimension */
+ int idx = mv_get_index(var->varattno, mvstats->stakeys);
+
+ fmgr_info(get_opcode(typecache->lt_opr), <proc);
+ fmgr_info(get_opcode(typecache->gt_opr), >proc);
+
+ /*
+ * Walk through the MCV items and evaluate the current clause. We can
+ * skip items that were already ruled out, and terminate if there are
+ * no remaining MCV items that might possibly match.
+ */
+ for (i = 0; i < mcvlist->nitems; i++)
+ {
+ bool tmp;
+ MCVItem item = mcvlist->items[i];
+
+ /*
+ * find the lowest selectivity in the MCV
+ * FIXME Maybe not the best place do do this (in for all clauses).
+ */
+ if (item->frequency < *lowsel)
+ *lowsel = item->frequency;
+
+ /* if there are no more matches, we can stop processing this clause */
+ if (nmatches == 0)
+ break;
+
+ /* skip MCV items that were already ruled out */
+ if (matches[i] == MVSTATS_MATCH_NONE)
+ continue;
+
+ /* TODO consider bsearch here (list is sorted by values)
+ * TODO handle other operators too (LT, GT)
+ * TODO identify "full match" when the clauses fully
+ * match the whole MCV list (so that checking the
+ * histogram is not needed)
+ */
+ if (get_oprrest(expr->opno) == F_EQSEL)
+ {
+ /*
+ * We don't care about isgt in equality, because it does not
+ * matter whether it's (var = const) or (const = var).
+ */
+ tmp = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ cst->constvalue,
+ item->values[idx]));
+ if (! tmp)
+ matches[i] = MVSTATS_MATCH_NONE;
+ else
+ eqmatches = bms_add_member(eqmatches, idx);
+ }
+ else if (get_oprrest(expr->opno) == F_SCALARLTSEL) /* column < constant */
+ {
+
+ if (! isgt) /* (var < const) */
+ {
+ /*
+ * First check whether the constant is below the lower boundary (in that
+ * case we can skip the bucket, because there's no overlap).
+ */
+ tmp = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ cst->constvalue,
+ item->values[idx]));
+
+ if (tmp)
+ {
+ matches[i] = MVSTATS_MATCH_NONE; /* no match */
+ continue;
+ }
+
+ } /* (get_oprrest(expr->opno) == F_SCALARLTSEL) */
+ else /* (const < var) */
+ {
+ /*
+ * First check whether the constant is above the upper boundary (in that
+ * case we can skip the bucket, because there's no overlap).
+ */
+ tmp = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ item->values[idx],
+ cst->constvalue));
+ if (tmp)
+ {
+ matches[i] = MVSTATS_MATCH_NONE; /* no match */
+ continue;
+ }
+ }
+ }
+ else if (get_oprrest(expr->opno) == F_SCALARGTSEL) /* column > constant */
+ {
+
+ if (! isgt) /* (var > const) */
+ {
+ /*
+ * First check whether the constant is above the upper boundary (in that
+ * case we can skip the bucket, because there's no overlap).
+ */
+ tmp = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ cst->constvalue,
+ item->values[idx]));
+ if (tmp)
+ {
+ matches[i] = MVSTATS_MATCH_NONE; /* no match */
+ continue;
+ }
+
+ }
+ else /* (const > var) */
+ {
+ /*
+ * First check whether the constant is below the lower boundary (in
+ * that case we can skip the bucket, because there's no overlap).
+ */
+ tmp = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ item->values[idx],
+ cst->constvalue));
+ if (tmp)
+ {
+ matches[i] = MVSTATS_MATCH_NONE; /* no match */
+ continue;
+ }
+ }
+
+ } /* (get_oprrest(expr->opno) == F_SCALARGTSEL) */
+ }
+ }
+ }
+ else if (IsA(clause, NullTest))
+ {
+ NullTest * expr = (NullTest*)clause;
+ Var * var = (Var*)(expr->arg);
+
+ /* FIXME proper matching attribute to dimension */
+ int idx = mv_get_index(var->varattno, mvstats->stakeys);
+
+ /*
+ * Walk through the MCV items and evaluate the current clause. We can
+ * skip items that were already ruled out, and terminate if there are
+ * no remaining MCV items that might possibly match.
+ */
+ for (i = 0; i < mcvlist->nitems; i++)
+ {
+ MCVItem item = mcvlist->items[i];
+
+ /*
+ * find the lowest selectivity in the MCV
+ * FIXME Maybe not the best place do do this (in for all clauses).
+ */
+ if (item->frequency < *lowsel)
+ *lowsel = item->frequency;
+
+ /* if there are no more matches, we can stop processing this clause */
+ if (nmatches == 0)
+ break;
+
+ /* skip MCV items that were already ruled out */
+ if (matches[i] == MVSTATS_MATCH_NONE)
+ continue;
+
+ /* if the clause mismatches the MCV item, set it as MATCH_NONE */
+ if ((expr->nulltesttype == IS_NULL)
+ && (! mcvlist->items[i]->isnull[idx]))
+ matches[i] = MVSTATS_MATCH_NONE;
+ else if ((expr->nulltesttype == IS_NOT_NULL) &&
+ (mcvlist->items[i]->isnull[idx]))
+ matches[i] = MVSTATS_MATCH_NONE;
+ }
+ }
+ }
+
+ /* sum frequencies for all the matching MCV items */
+ for (i = 0; i < mcvlist->nitems; i++)
+ {
+ if (matches[i] != MVSTATS_MATCH_NONE)
+ s += mcvlist->items[i]->frequency;
+ }
+
+ /*
+ * If all the columns were matched by equality, it's a full match.
+ * In this case there can be just a single MCV item, matching the
+ * clause (if there were two, both would match the other one).
+ */
+ *fullmatch = (bms_num_members(eqmatches) == mcvlist->ndimensions);
+
+ pfree(matches);
+ pfree(mcvlist);
+
+ return s;
+}
diff --git a/src/backend/utils/mvstats/Makefile b/src/backend/utils/mvstats/Makefile
index 099f1ed..3c0aff4 100644
--- a/src/backend/utils/mvstats/Makefile
+++ b/src/backend/utils/mvstats/Makefile
@@ -12,6 +12,6 @@ subdir = src/backend/utils/mvstats
top_builddir = ../../../..
include $(top_builddir)/src/Makefile.global
-OBJS = common.o dependencies.o
+OBJS = common.o mcv.o dependencies.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/utils/mvstats/common.c b/src/backend/utils/mvstats/common.c
index 0edaaa6..69ab805 100644
--- a/src/backend/utils/mvstats/common.c
+++ b/src/backend/utils/mvstats/common.c
@@ -16,6 +16,10 @@
#include "common.h"
+static VacAttrStats ** lookup_var_attr_stats(int2vector *attrs,
+ int natts,
+ VacAttrStats **vacattrstats);
+
/*
* Compute requested multivariate stats, using the rows sampled for the
* plain (single-column) stats.
@@ -40,10 +44,15 @@ build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
for (i = 0; i < nmvstats; i++)
{
MVDependencies deps = NULL;
+ MCVList mcvlist = NULL;
+ int numrows_filtered = 0;
/* int2 vector of attnums the stats should be computed on */
int2vector * attrs = mvstats[i].stakeys;
+ /* filter only the interesting vacattrstats records */
+ VacAttrStats **stats = lookup_var_attr_stats(attrs, natts, vacattrstats);
+
/* check allowed number of dimensions */
Assert((attrs->dim1 >= 2) && (attrs->dim1 <= MVSTATS_MAX_DIMENSIONS));
@@ -51,10 +60,14 @@ build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
* Analyze functional dependencies of columns.
*/
if (mvstats->deps_enabled)
- deps = build_mv_dependencies(numrows, rows, attrs, natts, vacattrstats);
+ deps = build_mv_dependencies(numrows, rows, attrs, stats);
+
+ /* build the MCV list */
+ if (mvstats->mcv_enabled)
+ mcvlist = build_mv_mcvlist(numrows, rows, attrs, stats, &numrows_filtered);
/* store the histogram / MCV list in the catalog */
- update_mv_stats(mvstats[i].mvoid, deps);
+ update_mv_stats(mvstats[i].mvoid, deps, mcvlist, attrs, stats);
}
}
@@ -63,7 +76,7 @@ build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
* matching the attrs vector (to make it easy to work with when
* computing multivariate stats).
*/
-VacAttrStats **
+static VacAttrStats **
lookup_var_attr_stats(int2vector *attrs, int natts, VacAttrStats **vacattrstats)
{
int i, j;
@@ -136,7 +149,7 @@ list_mv_stats(Oid relid, int *nstats, bool built_only)
* Skip statistics that were not computed yet (if only stats
* that were already built were requested)
*/
- if (built_only && (! stats->deps_built))
+ if (built_only && (! (stats->mcv_built || stats->deps_built)))
continue;
/* double the array size if needed */
@@ -149,7 +162,9 @@ list_mv_stats(Oid relid, int *nstats, bool built_only)
result[*nstats].mvoid = HeapTupleGetOid(htup);
result[*nstats].stakeys = buildint2vector(stats->stakeys.values, stats->stakeys.dim1);
result[*nstats].deps_enabled = stats->deps_enabled;
+ result[*nstats].mcv_enabled = stats->mcv_enabled;
result[*nstats].deps_built = stats->deps_built;
+ result[*nstats].mcv_built = stats->mcv_built;
*nstats += 1;
}
@@ -164,7 +179,9 @@ list_mv_stats(Oid relid, int *nstats, bool built_only)
}
void
-update_mv_stats(Oid mvoid, MVDependencies dependencies)
+update_mv_stats(Oid mvoid,
+ MVDependencies dependencies, MCVList mcvlist,
+ int2vector *attrs, VacAttrStats **stats)
{
HeapTuple stup,
oldtup;
@@ -189,15 +206,26 @@ update_mv_stats(Oid mvoid, MVDependencies dependencies)
= PointerGetDatum(serialize_mv_dependencies(dependencies));
}
+ if (mcvlist != NULL)
+ {
+ bytea * data = serialize_mv_mcvlist(mcvlist, attrs, stats);
+ nulls[Anum_pg_mv_statistic_stamcv -1] = (data == NULL);
+ values[Anum_pg_mv_statistic_stamcv - 1] = PointerGetDatum(data);
+ }
+
/* always replace the value (either by bytea or NULL) */
replaces[Anum_pg_mv_statistic_stadeps -1] = true;
+ replaces[Anum_pg_mv_statistic_stamcv -1] = true;
/* always change the availability flags */
nulls[Anum_pg_mv_statistic_deps_built -1] = false;
+ nulls[Anum_pg_mv_statistic_mcv_built -1] = false;
replaces[Anum_pg_mv_statistic_deps_built-1] = true;
+ replaces[Anum_pg_mv_statistic_mcv_built -1] = true;
values[Anum_pg_mv_statistic_deps_built-1] = BoolGetDatum(dependencies != NULL);
+ values[Anum_pg_mv_statistic_mcv_built -1] = BoolGetDatum(mcvlist != NULL);
/* Is there already a pg_mv_statistic tuple for this attribute? */
oldtup = SearchSysCache1(MVSTATOID,
@@ -225,6 +253,21 @@ update_mv_stats(Oid mvoid, MVDependencies dependencies)
heap_close(sd, RowExclusiveLock);
}
+
+int
+mv_get_index(AttrNumber varattno, int2vector * stakeys)
+{
+ int i, idx = 0;
+ for (i = 0; i < stakeys->dim1; i++)
+ {
+ if (stakeys->values[i] < varattno)
+ idx += 1;
+ else
+ break;
+ }
+ return idx;
+}
+
/* multi-variate stats comparator */
/*
@@ -235,11 +278,15 @@ update_mv_stats(Oid mvoid, MVDependencies dependencies)
int
compare_scalars_simple(const void *a, const void *b, void *arg)
{
- Datum da = *(Datum*)a;
- Datum db = *(Datum*)b;
- SortSupport ssup= (SortSupport) arg;
+ return compare_datums_simple(*(Datum*)a,
+ *(Datum*)b,
+ (SortSupport)arg);
+}
- return ApplySortComparator(da, false, db, false, ssup);
+int
+compare_datums_simple(Datum a, Datum b, SortSupport ssup)
+{
+ return ApplySortComparator(a, false, b, false, ssup);
}
/*
diff --git a/src/backend/utils/mvstats/common.h b/src/backend/utils/mvstats/common.h
index b98ceb7..fca2782 100644
--- a/src/backend/utils/mvstats/common.h
+++ b/src/backend/utils/mvstats/common.h
@@ -59,6 +59,14 @@ typedef struct
int *tupnoLink;
} CompareScalarsContext;
+/* (de)serialization info */
+typedef struct DimensionInfo {
+ int nvalues; /* number of deduplicated values */
+ int nbytes; /* number of bytes (serialized) */
+ int typlen; /* pg_type.typlen */
+ bool typbyval; /* pg_type.typbyval */
+} DimensionInfo;
+
/* multi-sort */
typedef struct MultiSortSupportData {
int ndims; /* number of dimensions supported by the */
@@ -82,10 +90,8 @@ int multi_sort_compare(const void *a, const void *b, void *arg);
int multi_sort_compare_dim(int dim, const SortItem *a,
const SortItem *b, MultiSortSupport mss);
-VacAttrStats ** lookup_var_attr_stats(int2vector *attrs,
- int natts, VacAttrStats **vacattrstats);
-
/* comparators, used when constructing multivariate stats */
+int compare_datums_simple(Datum a, Datum b, SortSupport ssup);
int compare_scalars_simple(const void *a, const void *b, void *arg);
int compare_scalars_partition(const void *a, const void *b, void *arg);
int compare_scalars_memcmp(const void *a, const void *b, void *arg);
diff --git a/src/backend/utils/mvstats/dependencies.c b/src/backend/utils/mvstats/dependencies.c
index 93a2fa6..0543690 100644
--- a/src/backend/utils/mvstats/dependencies.c
+++ b/src/backend/utils/mvstats/dependencies.c
@@ -224,7 +224,7 @@
*/
MVDependencies
build_mv_dependencies(int numrows, HeapTuple *rows, int2vector *attrs,
- int natts, VacAttrStats **vacattrstats)
+ VacAttrStats **stats)
{
int i;
int numattrs = attrs->dim1;
@@ -245,13 +245,6 @@ build_mv_dependencies(int numrows, HeapTuple *rows, int2vector *attrs,
/* dimension indexes we'll check for associations [a => b] */
int dima, dimb;
- /* info for the interesting attributes only
- *
- * TODO Compute this only once and pass it to all the methods
- * that need it.
- */
- VacAttrStats **stats = lookup_var_attr_stats(attrs, natts, vacattrstats);
-
/*
* We'll reuse the same array for all the 2-column combinations.
*
@@ -287,7 +280,7 @@ build_mv_dependencies(int numrows, HeapTuple *rows, int2vector *attrs,
for (dima = 0; dima < numattrs; dima++)
{
/* prepare the sort function for the first dimension */
- multi_sort_add_dimension(mss, 0, dima, vacattrstats);
+ multi_sort_add_dimension(mss, 0, dima, stats);
for (dimb = 0; dimb < numattrs; dimb++)
{
@@ -309,7 +302,7 @@ build_mv_dependencies(int numrows, HeapTuple *rows, int2vector *attrs,
continue;
/* prepare the sort function for the second dimension */
- multi_sort_add_dimension(mss, 1, dimb, vacattrstats);
+ multi_sort_add_dimension(mss, 1, dimb, stats);
/* reset the values and isnull flags */
memset(values, 0, sizeof(Datum) * numrows * 2);
diff --git a/src/backend/utils/mvstats/mcv.c b/src/backend/utils/mvstats/mcv.c
new file mode 100644
index 0000000..2b3d171
--- /dev/null
+++ b/src/backend/utils/mvstats/mcv.c
@@ -0,0 +1,1002 @@
+/*-------------------------------------------------------------------------
+ *
+ * mcv.c
+ * POSTGRES multivariate MCV lists
+ *
+ *
+ * Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/utils/mvstats/mcv.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "common.h"
+
+/*
+ * Multivariate MCVs (most-common values lists) are a straightforward
+ * extension of regular MCV list by tracking combinations of values for
+ * several attributes (columns), including NULL flags, and frequency
+ * of the combination.
+ *
+ * For columns small number of distinct values, this works quite well
+ * and may represent the distribution pretty exactly. For columns with
+ * large number of distinct values (e.g. stored as FLOAT), this does
+ * not work that well.
+ *
+ * If we can represent the distribution as a MCV list, we can estimate
+ * some clauses (e.g. equality clauses) much accurately than using
+ * histograms for example.
+ *
+ * Discrete distributions are also easier to combine into a larger
+ * distribution (but this is not yet implemented).
+ *
+ *
+ * TODO For types that don't reasonably support ordering (either because
+ * the type does not support that or when the user adds some option
+ * to the ADD STATISTICS command - e.g. UNSORTED_STATS), building
+ * the histogram may be pointless and inefficient. This is esp.
+ * true for varlena types that may be quite large and a large MCV
+ * list may be a better choice, because it makes equality estimates
+ * more accurate. Due to the unsorted nature, range queries on those
+ * attributes are rather useless anyway.
+ *
+ * Another thing is that by restricting to MCV list and equality
+ * conditions, we can use hash values instead of long varlena values.
+ * The equality estimation will be very accurate.
+ *
+ * This however complicates matching the columns to available
+ * statistics, as it will require matching clauses (not columns) to
+ * stats. And it may get quite complex - e.g. what if there are
+ * multiple clauses, each compatible with different stats subset?
+ *
+ *
+ * Selectivity estimation
+ * ----------------------
+ * The estimation, implemented in clauselist_mv_selectivity_mcvlist(),
+ * is quite simple in principle - walk through the MCV items and sum
+ * frequencies of all the items that match all the clauses.
+ *
+ * The current implementation uses MCV lists to estimates those types
+ * of clauses (think of WHERE conditions):
+ *
+ * (a) equality clauses WHERE (a = 1) AND (b = 2)
+ *
+ * (b) inequality clauses WHERE (a < 1) AND (b >= 2)
+ *
+ * It's possible to add more clauses, for example:
+ *
+ * (a) NULL clauses WHERE (a IS NULL) AND (b IS NOT NULL)
+ *
+ * (b) multi-var clauses WHERE (a > b)
+ *
+ * and so on. These are tasks for the future, not yet implemented.
+ *
+ *
+ * Estimating equality clauses
+ * ---------------------------
+ * When computing selectivity estimate for equality clauses
+ *
+ * (a = 1) AND (b = 2)
+ *
+ * we can do this estimate pretty exactly assuming that two conditions
+ * are met:
+ *
+ * (1) there's an equality condition on each attribute
+ *
+ * (2) we find a matching item in the MCV list
+ *
+ * In that case we know the MCV item represents all the tuples matching
+ * the clauses, and the selectivity estimate is complete. This is what
+ * we call 'full match'.
+ *
+ * When only (1) holds, but there's no matching MCV item, we don't know
+ * whether there are no such rows or just are not very frequent. We can
+ * however use the frequency of the least frequent MCV item as an upper
+ * bound for the selectivity.
+ *
+ * If the equality conditions match only a subset of the attributes
+ * the MCV list is built on (i.e. we can't get a full match - we may get
+ * multiple MCV items matching the clauses, but even if we get a single
+ * match there may be items that did not get into the MCV list. But in
+ * this case we can still use the frequency of the last MCV item to clam
+ * the 'additional' selectivity not accounted for by the matching items.
+ *
+ * If there's no histogram, because the MCV list approximates the
+ * distribution accurately (not because the histogram was disabled),
+ * it does not really matter whether there are equality conditions on
+ * all the columns - we can do pretty accurate estimation using the MCV.
+ *
+ * TODO For a combination of equality conditions (not full-match case)
+ * we probably can clamp the selectivity by the minimum of
+ * selectivities for each condition. For example if we know the
+ * number of distinct values for each column, we can use 1/ndistinct
+ * as a per-column estimate. Or rather 1/ndistinct + selectivity
+ * derived from the MCV list.
+ *
+ * If we know the estimate of number of combinations of the columns
+ * (i.e. ndistinct(A,B)), we may estimate the average frequency of
+ * items in the remaining 10% as [10% / ndistinct(A,B)].
+ *
+ *
+ * Bounding estimates
+ * ------------------
+ * In general the MCV lists may not provide estimates as accurate as
+ * for the full-match equality case, but may provide some useful
+ * lower/upper boundaries for the estimation error.
+ *
+ * With equality clauses we can do a few more tricks to narrow this
+ * error range (see the previous section and TODO), but with inequality
+ * clauses (or generally non-equality clauses), it's rather dificult.
+ * There's nothing like a 'full match' - we have to consider both the
+ * MCV items and the remaining part every time. We can't use the minimum
+ * selectivity of MCV items, as the clauses may match multiple items.
+ *
+ * For example with a MCV list on columns (A, B), covering 90% of the
+ * table (computed while building the MCV list), about ~10% of the table
+ * is not represented by the MCV list. So even if the conditions match
+ * all the remaining rows (not represented by the MCV items), we can't
+ * get selectivity higher than those 10%. We may use 1/2 the remaining
+ * selectivity as an estimate (minimizing average error).
+ *
+ * TODO Most of these ideas (error limiting) are not yet implemented.
+ *
+ *
+ * General TODO
+ * ------------
+ *
+ * FIXME Use max_mcv_items from ALTER TABLE ADD STATISTICS command.
+ *
+ * TODO Add support for IS [NOT] NULL clauses, and clauses referencing
+ * multiple columns (a < b).
+ *
+ * TODO It's possible to build a special case of MCV list, storing not
+ * the actual values but only 32/64-bit hash. This is only useful
+ * for estimating equality clauses and for large varlena types,
+ * which are very impractical for plain MCV list because of size.
+ * But for those data types we really want just the equality
+ * clauses, so it's actually a good solution.
+ *
+ * TODO Currently there's no logic to consider building only a MCV list
+ * (and not building the histogram at all), except for doing this
+ * decision manually in ADD STATISTICS.
+ */
+
+/*
+ * Each serialized item needs to store (in this order):
+ *
+ * - indexes (ndim * sizeof(int32))
+ * - null flags (ndim * sizeof(bool))
+ * - frequency (sizeof(double))
+ *
+ * So in total:
+ *
+ * ndim * (sizeof(int32) + sizeof(bool)) + sizeof(double)
+ */
+#define ITEM_SIZE(ndims) \
+ (ndims * (sizeof(int32) + sizeof(bool)) + sizeof(double))
+
+/* pointers into a flat serialized item of ITEM_SIZE(n) bytes */
+#define ITEM_INDEXES(item) ((int32*)item)
+#define ITEM_NULLS(item,ndims) ((bool*)(ITEM_INDEXES(item) + ndims))
+#define ITEM_FREQUENCY(item,ndims) ((double*)(ITEM_NULLS(item,ndims) + ndims))
+
+/*
+ * Builds MCV list from sample rows, and removes rows represented by
+ * the MCV list from the sample (the number of remaining sample rows is
+ * returned by the numrows_filtered parameter).
+ *
+ * The method is quite simple - in short it does about these steps:
+ *
+ * (1) sort the data (default collation, '<' for the data type)
+ *
+ * (2) count distinct groups, decide how many to keep
+ *
+ * (3) build the MCV list using the threshold determined in (2)
+ *
+ * (4) remove rows represented by the MCV from the sample
+ *
+ * For more details, see the comments in the code.
+ *
+ * FIXME Single-dimensional MCV is sorted by frequency (descending). We
+ * should do that too, because when walking through the list we
+ * want to check the most frequent items first.
+ *
+ * TODO We're using Datum (8B), even for data types (e.g. int4 or
+ * float4). Maybe we could save some space here, but the bytea
+ * compression should handle it just fine.
+ *
+ * TODO This probably should not use the ndistinct directly (as computed
+ * from the table, but rather estimate the number of distinct
+ * values in the table), no?
+ */
+MCVList
+build_mv_mcvlist(int numrows, HeapTuple *rows, int2vector *attrs,
+ VacAttrStats **stats, int *numrows_filtered)
+{
+ int i, j;
+ int numattrs = attrs->dim1;
+ int ndistinct = 0;
+ int mcv_threshold = 0;
+ int count = 0;
+ int nitems = 0;
+
+ MCVList mcvlist = NULL;
+
+ /* Sort by multiple columns (using array of SortSupport) */
+ MultiSortSupport mss = multi_sort_init(numattrs);
+
+ /*
+ * Preallocate space for all the items as a single chunk, and point
+ * the items to the appropriate parts of the array.
+ */
+ SortItem *items = (SortItem*)palloc0(numrows * sizeof(SortItem));
+ Datum *values = (Datum*)palloc0(sizeof(Datum) * numrows * numattrs);
+ bool *isnull = (bool*)palloc0(sizeof(bool) * numrows * numattrs);
+
+ /* keep all the rows by default (as if there was no MCV list) */
+ *numrows_filtered = numrows;
+
+ for (i = 0; i < numrows; i++)
+ {
+ items[i].values = &values[i * numattrs];
+ items[i].isnull = &isnull[i * numattrs];
+ }
+
+ /* load the values/null flags from sample rows */
+ for (j = 0; j < numrows; j++)
+ for (i = 0; i < numattrs; i++)
+ items[j].values[i] = heap_getattr(rows[j], attrs->values[i],
+ stats[i]->tupDesc, &items[j].isnull[i]);
+
+ /* prepare the sort functions for all the attributes */
+ for (i = 0; i < numattrs; i++)
+ multi_sort_add_dimension(mss, i, i, stats);
+
+ /* do the sort, using the multi-sort */
+ qsort_arg((void *) items, numrows, sizeof(SortItem),
+ multi_sort_compare, mss);
+
+ /*
+ * Count the number of distinct groups - just walk through the
+ * sorted list and count the number of key changes. We use this to
+ * determine the threshold (125% of the average frequency).
+ */
+ ndistinct = 1;
+ for (i = 1; i < numrows; i++)
+ if (multi_sort_compare(&items[i], &items[i-1], mss) != 0)
+ ndistinct += 1;
+
+ /*
+ * Determine how many groups actually exceed the threshold, and then
+ * walk the array again and collect them into an array. We'll always
+ * require at least 4 rows per group.
+ *
+ * But if we can fit all the distinct values in the MCV list (i.e.
+ * if there are less distinct groups than MVSTAT_MCVLIST_MAX_ITEMS),
+ * we'll require only 2 rows per group.
+ *
+ * TODO For now the threshold is the same as in the single-column
+ * case (average + 25%), but maybe that's worth revisiting
+ * for the multivariate case.
+ *
+ * TODO We can do this only if we believe we got all the distinct
+ * values of the table.
+ *
+ * FIXME This should really reference mcv_max_items (from catalog)
+ * instead of the constant MVSTAT_MCVLIST_MAX_ITEMS.
+ */
+ mcv_threshold = 1.25 * numrows / ndistinct;
+ mcv_threshold = (mcv_threshold < 4) ? 4 : mcv_threshold;
+
+ if (ndistinct <= MVSTAT_MCVLIST_MAX_ITEMS)
+ mcv_threshold = 2;
+
+ /*
+ * Walk through the sorted data again, and see how many groups
+ * reach the mcv_threshold (and become an item in the MCV list).
+ */
+ count = 1;
+ for (i = 1; i <= numrows; i++)
+ {
+ /* last row or new group, so check if we exceed mcv_threshold */
+ if ((i == numrows) || (multi_sort_compare(&items[i], &items[i-1], mss) != 0))
+ {
+ /* group hits the threshold, count the group as MCV item */
+ if (count >= mcv_threshold)
+ nitems += 1;
+
+ count = 1;
+ }
+ else /* within group, so increase the number of items */
+ count += 1;
+ }
+
+ /* we know the number of MCV list items, so let's build the list */
+ if (nitems > 0)
+ {
+ /* allocate the MCV list structure, set parameters we know */
+ mcvlist = (MCVList)palloc0(sizeof(MCVListData));
+
+ mcvlist->magic = MVSTAT_MCV_MAGIC;
+ mcvlist->type = MVSTAT_MCV_TYPE_BASIC;
+ mcvlist->ndimensions = numattrs;
+ mcvlist->nitems = nitems;
+
+ /*
+ * Preallocate Datum/isnull arrays (not as a single chunk, as
+ * we'll pass this outside this method and thus it needs to be
+ * easy to pfree() the data (and we wouldn't know where the
+ * arrays start).
+ *
+ * TODO Maybe the reasoning that we can't allocate a single
+ * piece because we're passing it out is bogus? Who'd
+ * free a single item of the MCV list, anyway?
+ *
+ * TODO Maybe with a proper encoding (stuffing all the values
+ * into a list-level array, this will be untrue)?
+ */
+ mcvlist->items = (MCVItem*)palloc0(sizeof(MCVItem)*nitems);
+
+ for (i = 0; i < nitems; i++)
+ {
+ mcvlist->items[i] = (MCVItem)palloc0(sizeof(MCVItemData));
+ mcvlist->items[i]->values = (Datum*)palloc0(sizeof(Datum)*numattrs);
+ mcvlist->items[i]->isnull = (bool*)palloc0(sizeof(bool)*numattrs);
+ }
+
+ /*
+ * Repeat the same loop as above, but this time copy the data
+ * into the MCV list (for items exceeding the threshold).
+ *
+ * TODO Maybe we could simply remember indexes of the last item
+ * in each group (from the previous loop)?
+ */
+ count = 1;
+ nitems = 0;
+ for (i = 1; i <= numrows; i++)
+ {
+ /* last row or a new group */
+ if ((i == numrows) || (multi_sort_compare(&items[i], &items[i-1], mss) != 0))
+ {
+ /* count the MCV item if exceeding the threshold (and copy into the array) */
+ if (count >= mcv_threshold)
+ {
+ /* just pointer to the proper place in the list */
+ MCVItem item = mcvlist->items[nitems];
+
+ /* copy values from the _previous_ group (last item of) */
+ memcpy(item->values, items[(i-1)].values, sizeof(Datum) * numattrs);
+ memcpy(item->isnull, items[(i-1)].isnull, sizeof(bool) * numattrs);
+
+
+ /* and finally the group frequency */
+ item->frequency = (double)count / numrows;
+
+ /* next item */
+ nitems += 1;
+ }
+
+ count = 1;
+ }
+ else /* same group, just increase the number of items */
+ count += 1;
+ }
+
+ /* make sure the loops are consistent */
+ Assert(nitems == mcvlist->nitems);
+
+ /*
+ * Remove the rows matching the MCV list (i.e. keep only rows
+ * that are not represented by the MCV list).
+ *
+ * FIXME This implementation is rather naive, effectively O(N^2).
+ * As the MCV list grows, the check will take longer and
+ * longer. And as the number of sampled rows increases (by
+ * increasing statistics target), it will take longer and
+ * longer. One option is to sort the MCV items first and
+ * then perform a binary search.
+ *
+ * A better option would be keeping the ID of the row in
+ * the sort item, and then just walk through the items and
+ * mark rows to remove (in a bitmap of the same size).
+ * There's not space for that in SortItem at this moment,
+ * but it's trivial to add 'private' pointer, or just
+ * using another structure with extra field (starting with
+ * SortItem, so that the comparators etc. still work).
+ *
+ * Another option is to use the sorted array of items
+ * (because that's how we sorted the source data), and
+ * simply do a bsearch() into it. If we find a matching
+ * item, the row belongs to the MCV list.
+ */
+ if (nitems == ndistinct) /* all rows are covered by MCV items */
+ *numrows_filtered = 0;
+ else /* (nitems < ndistinct) && (nitems > 0) */
+ {
+ int nfiltered = 0;
+ HeapTuple *rows_filtered = (HeapTuple*)palloc0(sizeof(HeapTuple) * numrows);
+
+ /* used for the searches */
+ SortItem item, mcvitem;;
+
+ item.values = (Datum*)palloc0(numattrs * sizeof(Datum));
+ item.isnull = (bool*)palloc0(numattrs * sizeof(bool));
+
+ /*
+ * FIXME we don't need to allocate this, we can reference
+ * the MCV item directly ...
+ */
+ mcvitem.values = (Datum*)palloc0(numattrs * sizeof(Datum));
+ mcvitem.isnull = (bool*)palloc0(numattrs * sizeof(bool));
+
+ /* walk through the tuples, compare the values to MCV items */
+ for (i = 0; i < numrows; i++)
+ {
+ bool match = false;
+
+ /* collect the key values from the row */
+ for (j = 0; j < numattrs; j++)
+ item.values[j] = heap_getattr(rows[i], attrs->values[j],
+ stats[j]->tupDesc, &item.isnull[j]);
+
+ /* scan through the MCV list for matches */
+ for (j = 0; j < mcvlist->nitems; j++)
+ {
+ /*
+ * TODO Create a SortItem/MCVItem comparator so that
+ * we don't need to do memcpy() like crazy.
+ */
+ memcpy(mcvitem.values, mcvlist->items[j]->values,
+ numattrs * sizeof(Datum));
+ memcpy(mcvitem.isnull, mcvlist->items[j]->isnull,
+ numattrs * sizeof(bool));
+
+ if (multi_sort_compare(&item, &mcvitem, mss) == 0)
+ {
+ match = true;
+ break;
+ }
+ }
+
+ /* if no match in the MCV list, copy the row into the filtered ones */
+ if (! match)
+ memcpy(&rows_filtered[nfiltered++], &rows[i], sizeof(HeapTuple));
+ }
+
+ /* replace the rows and remember how many rows we kept */
+ memcpy(rows, rows_filtered, sizeof(HeapTuple) * nfiltered);
+ *numrows_filtered = nfiltered;
+
+ /* free all the data used here */
+ pfree(rows_filtered);
+ pfree(item.values);
+ pfree(item.isnull);
+ pfree(mcvitem.values);
+ pfree(mcvitem.isnull);
+ }
+ }
+
+ pfree(values);
+ pfree(items);
+ pfree(isnull);
+
+ return mcvlist;
+}
+
+
+/* fetch the MCV list (as a bytea) from the pg_mv_statistic catalog */
+bytea *
+fetch_mv_mcvlist(Oid mvoid)
+{
+ Relation indrel;
+ SysScanDesc indscan;
+ ScanKeyData skey;
+ HeapTuple htup;
+ bytea *mcvlist = NULL;
+
+ /* Prepare to scan pg_mv_statistic for entries having indrelid = this rel. */
+ ScanKeyInit(&skey,
+ ObjectIdAttributeNumber,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(mvoid));
+
+ indrel = heap_open(MvStatisticRelationId, AccessShareLock);
+ indscan = systable_beginscan(indrel, MvStatisticOidIndexId, true,
+ NULL, 1, &skey);
+
+ while (HeapTupleIsValid(htup = systable_getnext(indscan)))
+ {
+ bool isnull = false;
+ Datum tmp = SysCacheGetAttr(MVSTATOID, htup,
+ Anum_pg_mv_statistic_stamcv, &isnull);
+
+ Assert(!isnull);
+
+ mcvlist = DatumGetByteaP(tmp);
+
+ break;
+ }
+
+ systable_endscan(indscan);
+
+ heap_close(indrel, AccessShareLock);
+
+ /* TODO maybe save the list into relcache, as in RelationGetIndexList
+ * (which was used as an inspiration of this one)?. */
+
+ return mcvlist;
+}
+
+/* print some basic info about the MCV list
+ *
+ * TODO Add info about what part of the table this covers.
+ */
+Datum
+pg_mv_stats_mcvlist_info(PG_FUNCTION_ARGS)
+{
+ bytea *data = PG_GETARG_BYTEA_P(0);
+ char *result;
+
+ MCVList mcvlist = deserialize_mv_mcvlist(data);
+
+ result = palloc0(128);
+ snprintf(result, 128, "nitems=%d", mcvlist->nitems);
+
+ pfree(mcvlist);
+
+ PG_RETURN_TEXT_P(cstring_to_text(result));
+}
+
+/* used to pass context into bsearch() */
+static SortSupport ssup_private = NULL;
+
+static int bsearch_comparator(const void * a, const void * b);
+
+/*
+ * Serialize MCV list into a bytea value. The basic algorithm is simple:
+ *
+ * (1) perform deduplication for each attribute (separately)
+ * (a) collect all (non-NULL) attribute values from all MCV items
+ * (b) sort the data (using 'lt' from VacAttrStats)
+ * (c) remove duplicate values from the array
+ *
+ * (2) serialize the arrays into a bytea value
+ *
+ * (3) process all MCV list items
+ * (a) replace values with indexes into the arrays
+ *
+ * Each attribute has to be processed separately, because we're mixing
+ * different datatypes, and we don't know what equality means for them.
+ * We're also mixing pass-by-value and pass-by-ref types, and so on.
+ *
+ * We'll use 32-bit values for the indexes in step (3), although we
+ * could probably use just 16 bits as we don't allow more than 8k
+ * items in the MCV list max_mcv_items (well, we might increase this to
+ * 32k and still fit into signed 16-bits). But let's be lazy and rely
+ * on the varlena compression to kick in. If most bytes will be 0x00
+ * so it should work nicely.
+ *
+ * FIXME This probably leaks memory, or at least uses it inefficiently
+ * (many small palloc() calls instead of a large one).
+ *
+ * TODO Consider using 16-bit values for the indexes in step (3).
+ *
+ * TODO Consider packing boolean flags (NULL) for each item into 'char'
+ * or a longer type (instead of using an array of bool items).
+ */
+bytea *
+serialize_mv_mcvlist(MCVList mcvlist, int2vector *attrs,
+ VacAttrStats **stats)
+{
+ int i, j;
+ int ndims = mcvlist->ndimensions;
+ int itemsize = ITEM_SIZE(ndims);
+
+ Size total_length = 0;
+
+ char *item = palloc0(itemsize);
+
+ /* serialized items (indexes into arrays, etc.) */
+ bytea *output;
+ char *data = NULL;
+
+ /* values per dimension (and number of non-NULL values) */
+ Datum **values = (Datum**)palloc0(sizeof(Datum*) * ndims);
+ int *counts = (int*)palloc0(sizeof(int) * ndims);
+
+ /* info about dimensions (for deserialize) */
+ DimensionInfo * info
+ = (DimensionInfo *)palloc0(sizeof(DimensionInfo)*ndims);
+
+ /* sort support data */
+ SortSupport ssup = (SortSupport)palloc0(sizeof(SortSupportData)*ndims);
+
+ /* collect and deduplicate values for each dimension */
+ for (i = 0; i < ndims; i++)
+ {
+ int count;
+ StdAnalyzeData *tmp = (StdAnalyzeData *)stats[i]->extra_data;
+
+ /* keep important info about the data type */
+ info[i].typlen = stats[i]->attrtype->typlen;
+ info[i].typbyval = stats[i]->attrtype->typbyval;
+
+ /* allocate space for all values, including NULLs (won't use them) */
+ values[i] = (Datum*)palloc0(sizeof(Datum) * mcvlist->nitems);
+
+ for (j = 0; j < mcvlist->nitems; j++)
+ {
+ if (! mcvlist->items[j]->isnull[i]) /* skip NULL values */
+ {
+ values[i][counts[i]] = mcvlist->items[j]->values[i];
+ counts[i] += 1;
+ }
+ }
+
+ /* there are just NULL values in this dimension */
+ if (counts[i] == 0)
+ continue;
+
+ /* sort and deduplicate */
+ ssup[i].ssup_cxt = CurrentMemoryContext;
+ ssup[i].ssup_collation = DEFAULT_COLLATION_OID;
+ ssup[i].ssup_nulls_first = false;
+
+ PrepareSortSupportFromOrderingOp(tmp->ltopr, &ssup[i]);
+
+ qsort_arg(values[i], counts[i], sizeof(Datum),
+ compare_scalars_simple, &ssup[i]);
+
+ /*
+ * Walk through the array and eliminate duplicitate values, but
+ * keep the ordering (so that we can do bsearch later). We know
+ * there's at least 1 item, so we can skip the first element.
+ */
+ count = 1; /* number of deduplicated items */
+ for (j = 1; j < counts[i]; j++)
+ {
+ /* if it's different from the previous value, we need to keep it */
+ if (compare_datums_simple(values[i][j-1], values[i][j], &ssup[i]) != 0)
+ {
+ /* XXX: not needed if (count == j) */
+ values[i][count] = values[i][j];
+ count += 1;
+ }
+ }
+
+ /* keep info about the deduplicated count */
+ info[i].nvalues = count;
+
+ /* compute size of the serialized data */
+ if (info[i].typbyval)
+ /*
+ * passed by value, so just Datum array (int4, int8, ...)
+ *
+ * TODO Might save a few bytes here, by storing just typlen
+ * bytes instead of whole Datum (8B) on 64-bits.
+ */
+ info[i].nbytes = info[i].nvalues * sizeof(Datum);
+ else if (info[i].typlen > 0)
+ /* pased by reference, but fixed length (name, tid, ...) */
+ info[i].nbytes = info[i].nvalues * info[i].typlen;
+ else if (info[i].typlen == -1)
+ /* varlena, so just use VARSIZE_ANY */
+ for (j = 0; j < info[i].nvalues; j++)
+ info[i].nbytes += VARSIZE_ANY(values[i][j]);
+ else if (info[i].typlen == -2)
+ /* cstring, so simply strlen */
+ for (j = 0; j < info[i].nvalues; j++)
+ info[i].nbytes += strlen(DatumGetPointer(values[i][j]));
+ else
+ elog(ERROR, "unknown data type typbyval=%d typlen=%d",
+ info[i].typbyval, info[i].typlen);
+ }
+
+ /*
+ * Now we finally know how much space we'll need for the serialized
+ * MCV list, as it contains these fields:
+ *
+ * - length (4B) for varlena
+ * - magic (4B)
+ * - type (4B)
+ * - ndimensions (4B)
+ * - nitems (4B)
+ * - info (ndim * sizeof(DimensionInfo)
+ * - arrays of values for each dimension
+ * - serialized items (nitems * itemsize)
+ *
+ * So the 'header' size is 20B + ndim * sizeof(DimensionInfo) and
+ * then we'll place the data.
+ */
+ total_length = (sizeof(int32) + offsetof(MCVListData, items)
+ + ndims * sizeof(DimensionInfo)
+ + mcvlist->nitems * itemsize);
+
+ for (i = 0; i < ndims; i++)
+ total_length += info[i].nbytes;
+
+ /* enforce arbitrary limit of 1MB */
+ if (total_length > 1024 * 1024)
+ elog(ERROR, "serialized MCV exceeds 1MB (%ld)", total_length);
+
+ /* allocate space for the serialized MCV list, set header fields */
+ output = (bytea*)palloc0(total_length);
+ SET_VARSIZE(output, total_length);
+
+ /* we'll use 'ptr' to keep track of the place to write data */
+ data = VARDATA(output);
+
+ memcpy(data, mcvlist, offsetof(MCVListData, items));
+ data += offsetof(MCVListData, items);
+
+ memcpy(data, info, sizeof(DimensionInfo) * ndims);
+ data += sizeof(DimensionInfo) * ndims;
+
+ /* value array for each dimension */
+ for (i = 0; i < ndims; i++)
+ {
+#ifdef USE_ASSERT_CHECKING
+ char *tmp = data;
+#endif
+ for (j = 0; j < info[i].nvalues; j++)
+ {
+ if (info[i].typbyval)
+ {
+ /* passed by value / Datum */
+ memcpy(data, &values[i][j], sizeof(Datum));
+ data += sizeof(Datum);
+ }
+ else if (info[i].typlen > 0)
+ {
+ /* pased by reference, but fixed length (name, tid, ...) */
+ memcpy(data, &values[i][j], info[i].typlen);
+ data += info[i].typlen;
+ }
+ else if (info[i].typlen == -1)
+ {
+ /* varlena */
+ memcpy(data, DatumGetPointer(values[i][j]),
+ VARSIZE_ANY(values[i][j]));
+ data += VARSIZE_ANY(values[i][j]);
+ }
+ else if (info[i].typlen == -2)
+ {
+ /* cstring (don't forget the \0 terminator!) */
+ memcpy(data, DatumGetPointer(values[i][j]),
+ strlen(DatumGetPointer(values[i][j])) + 1);
+ data += strlen(DatumGetPointer(values[i][j])) + 1;
+ }
+ }
+ Assert((data - tmp) == info[i].nbytes);
+ }
+
+ /* and finally, the MCV items */
+ for (i = 0; i < mcvlist->nitems; i++)
+ {
+ /* don't write beyond the allocated space */
+ Assert(data <= (char*)output + total_length - itemsize);
+
+ /* reset the values for each item */
+ memset(item, 0, itemsize);
+
+ for (j = 0; j < ndims; j++)
+ {
+ /* do the lookup only for non-NULL values */
+ if (! mcvlist->items[i]->isnull[j])
+ {
+ Datum * v = NULL;
+ ssup_private = &ssup[j];
+
+ v = (Datum*)bsearch(&mcvlist->items[i]->values[j],
+ values[j], info[j].nvalues, sizeof(Datum),
+ bsearch_comparator);
+
+ if (v == NULL)
+ elog(ERROR, "value for dim %d not found in array", j);
+
+ /* compute index within the array */
+ ITEM_INDEXES(item)[j] = (v - values[j]);
+
+ /* check the index is within expected bounds */
+ Assert(ITEM_INDEXES(item)[j] >= 0);
+ Assert(ITEM_INDEXES(item)[j] < info[j].nvalues);
+ }
+ }
+
+ /* copy NULL and frequency flags into the item */
+ memcpy(ITEM_NULLS(item, ndims),
+ mcvlist->items[i]->isnull, sizeof(bool) * ndims);
+ memcpy(ITEM_FREQUENCY(item, ndims),
+ &mcvlist->items[i]->frequency, sizeof(double));
+
+ /* copy the item into the array */
+ memcpy(data, item, itemsize);
+
+ data += itemsize;
+ }
+
+ /* at this point we expect to match the total_length exactly */
+ Assert((data - (char*)output) == total_length);
+
+ return output;
+}
+
+/* inverse to serialize_mv_mcvlist() - see the comment there */
+MCVList deserialize_mv_mcvlist(bytea * data)
+{
+ int i, j;
+ Size expected_size;
+ MCVList mcvlist;
+ char *tmp;
+
+ int ndims, nitems, itemsize;
+ DimensionInfo *info = NULL;
+
+ int32 *indexes = NULL;
+ Datum **values = NULL;
+
+ if (data == NULL)
+ return NULL;
+
+ if (VARSIZE_ANY_EXHDR(data) < offsetof(MCVListData,items))
+ elog(ERROR, "invalid MCV Size %ld (expected at least %ld)",
+ VARSIZE_ANY_EXHDR(data), offsetof(MCVListData,items));
+
+ /* read the MCV list header */
+ mcvlist = (MCVList)palloc0(sizeof(MCVListData));
+
+ /* initialize pointer to the data part (skip the varlena header) */
+ tmp = VARDATA(data);
+
+ /* get the header and perform basic sanity checks */
+ memcpy(mcvlist, tmp, offsetof(MCVListData,items));
+ tmp += offsetof(MCVListData,items);
+
+ if (mcvlist->magic != MVSTAT_MCV_MAGIC)
+ elog(ERROR, "invalid MCV magic %d (expected %dd)",
+ mcvlist->magic, MVSTAT_MCV_MAGIC);
+
+ if (mcvlist->type != MVSTAT_MCV_TYPE_BASIC)
+ elog(ERROR, "invalid MCV type %d (expected %dd)",
+ mcvlist->type, MVSTAT_MCV_TYPE_BASIC);
+
+ nitems = mcvlist->nitems;
+ ndims = mcvlist->ndimensions;
+ itemsize = ITEM_SIZE(ndims);
+
+ Assert(nitems > 0);
+ Assert((ndims >= 2) && (ndims <= MVSTATS_MAX_DIMENSIONS));
+
+ /*
+ * What size do we expect with those parameters (it's incomplete,
+ * as we yet have to count the array sizes (from DimensionInfo
+ * records).
+ */
+ expected_size = offsetof(MCVListData,items) +
+ ndims * sizeof(DimensionInfo) +
+ (nitems * itemsize);
+
+ /* check that we have at least the DimensionInfo records */
+ if (VARSIZE_ANY_EXHDR(data) < expected_size)
+ elog(ERROR, "invalid MCV Size %ld (expected %ld)",
+ VARSIZE_ANY_EXHDR(data), expected_size);
+
+ info = (DimensionInfo*)(tmp);
+ tmp += ndims * sizeof(DimensionInfo);
+
+ /* account for the value arrays */
+ for (i = 0; i < ndims; i++)
+ expected_size += info[i].nbytes;
+
+ if (VARSIZE_ANY_EXHDR(data) != expected_size)
+ elog(ERROR, "invalid MCV Size %ld (expected %ld)",
+ VARSIZE_ANY_EXHDR(data), expected_size);
+
+ /* looks OK - not corrupted or something */
+
+ /* let's parse the value arrays */
+ values = (Datum**)palloc0(sizeof(Datum*) * ndims);
+
+ /*
+ * FIXME This uses pointers to the original data array (the types
+ * not passed by value), so when someone frees the memory,
+ * e.g. by doing something like this:
+ *
+ * bytea * data = ... fetch the data from catalog ...
+ * MCVList mcvlist = deserialize_mcv_list(data);
+ * pfree(data);
+ *
+ * then 'mcvlist' references the freed memory. This needs to
+ * copy the pieces.
+ */
+ for (i = 0; i < ndims; i++)
+ {
+ if (info[i].typbyval)
+ {
+ /* passed by value / Datum - simply reuse the array */
+ values[i] = (Datum*)tmp;
+ tmp += info[i].nbytes;
+ }
+ else if (info[i].typlen > 0)
+ {
+ /* pased by reference, but fixed length (name, tid, ...) */
+ values[i] = (Datum*)palloc0(sizeof(Datum) * info[i].nvalues);
+ for (j = 0; j < info[i].nvalues; j++)
+ {
+ /* just point into the array */
+ values[i][j] = PointerGetDatum(tmp);
+ tmp += info[i].typlen;
+ }
+ }
+ else if (info[i].typlen == -1)
+ {
+ /* varlena */
+ values[i] = (Datum*)palloc0(sizeof(Datum) * info[i].nvalues);
+ for (j = 0; j < info[i].nvalues; j++)
+ {
+ /* just point into the array */
+ values[i][j] = PointerGetDatum(tmp);
+ tmp += VARSIZE_ANY(tmp);
+ }
+ }
+ else if (info[i].typlen == -2)
+ {
+ /* cstring */
+ values[i] = (Datum*)palloc0(sizeof(Datum) * info[i].nvalues);
+ for (j = 0; j < info[i].nvalues; j++)
+ {
+ /* just point into the array */
+ values[i][j] = PointerGetDatum(tmp);
+ tmp += (strlen(tmp) + 1); /* don't forget the \0 */
+ }
+ }
+ }
+
+ /* allocate space for the MCV items */
+ mcvlist->items = (MCVItem*)palloc0(sizeof(MCVItem) * nitems);
+
+ for (i = 0; i < nitems; i++)
+ {
+ MCVItem item = (MCVItem)palloc0(sizeof(MCVItemData));
+
+ item->values = (Datum*)palloc0(sizeof(Datum)*ndims);
+ item->isnull = (bool*) palloc0(sizeof(bool) *ndims);
+
+ /* just point to the right place */
+ indexes = ITEM_INDEXES(tmp);
+
+ memcpy(item->isnull, ITEM_NULLS(tmp, ndims), sizeof(bool) * ndims);
+ memcpy(&item->frequency, ITEM_FREQUENCY(tmp, ndims), sizeof(double));
+
+ /* translate the values */
+ for (j = 0; j < ndims; j++)
+ if (! item->isnull[j])
+ item->values[j] = values[j][indexes[j]];
+
+ mcvlist->items[i] = item;
+
+ tmp += ITEM_SIZE(ndims);
+
+ Assert(tmp <= (char*)data + VARSIZE_ANY(data));
+ }
+
+ /* check that we processed all the data */
+ Assert(tmp == (char*)data + VARSIZE_ANY(data));
+
+ return mcvlist;
+}
+
+/*
+ * We need to pass the SortSupport to the comparator, but bsearch()
+ * has no 'context' parameter, so we use a global variable (ugly).
+ */
+static int
+bsearch_comparator(const void * a, const void * b)
+{
+ Assert(ssup_private != NULL);
+ return compare_scalars_simple(a, b, (void*)ssup_private);
+}
diff --git a/src/include/catalog/pg_mv_statistic.h b/src/include/catalog/pg_mv_statistic.h
index 76b7db7..f88e200 100644
--- a/src/include/catalog/pg_mv_statistic.h
+++ b/src/include/catalog/pg_mv_statistic.h
@@ -35,15 +35,21 @@ CATALOG(pg_mv_statistic,3281)
/* statistics requested to build */
bool deps_enabled; /* analyze dependencies? */
+ bool mcv_enabled; /* build MCV list? */
+
+ /* MCV size */
+ int32 mcv_max_items; /* max MCV items */
/* statistics that are available (if requested) */
bool deps_built; /* dependencies were built */
+ bool mcv_built; /* MCV list was built */
/* variable-length fields start here, but we allow direct access to stakeys */
int2vector stakeys; /* array of column keys */
#ifdef CATALOG_VARLEN
bytea stadeps; /* dependencies (serialized) */
+ bytea stamcv; /* MCV list (serialized) */
#endif
} FormData_pg_mv_statistic;
@@ -59,11 +65,15 @@ typedef FormData_pg_mv_statistic *Form_pg_mv_statistic;
* compiler constants for pg_attrdef
* ----------------
*/
-#define Natts_pg_mv_statistic 5
+#define Natts_pg_mv_statistic 9
#define Anum_pg_mv_statistic_starelid 1
#define Anum_pg_mv_statistic_deps_enabled 2
-#define Anum_pg_mv_statistic_deps_built 3
-#define Anum_pg_mv_statistic_stakeys 4
-#define Anum_pg_mv_statistic_stadeps 5
+#define Anum_pg_mv_statistic_mcv_enabled 3
+#define Anum_pg_mv_statistic_mcv_max_items 4
+#define Anum_pg_mv_statistic_deps_built 5
+#define Anum_pg_mv_statistic_mcv_built 6
+#define Anum_pg_mv_statistic_stakeys 7
+#define Anum_pg_mv_statistic_stadeps 8
+#define Anum_pg_mv_statistic_stamcv 9
#endif /* PG_MV_STATISTIC_H */
diff --git a/src/include/catalog/pg_proc.h b/src/include/catalog/pg_proc.h
index 9fb118a..b4e7b4f 100644
--- a/src/include/catalog/pg_proc.h
+++ b/src/include/catalog/pg_proc.h
@@ -2687,6 +2687,8 @@ DATA(insert OID = 3284 ( pg_mv_stats_dependencies_info PGNSP PGUID 12 1 0 0
DESCR("multivariate stats: functional dependencies info");
DATA(insert OID = 3285 ( pg_mv_stats_dependencies_show PGNSP PGUID 12 1 0 0 0 f f f f t f i 1 0 25 "17" _null_ _null_ _null_ _null_ pg_mv_stats_dependencies_show _null_ _null_ _null_ ));
DESCR("multivariate stats: functional dependencies show");
+DATA(insert OID = 3283 ( pg_mv_stats_mcvlist_info PGNSP PGUID 12 1 0 0 0 f f f f t f i 1 0 25 "17" _null_ _null_ _null_ _null_ pg_mv_stats_mcvlist_info _null_ _null_ _null_ ));
+DESCR("multi-variate statistics: MCV list info");
DATA(insert OID = 1928 ( pg_stat_get_numscans PGNSP PGUID 12 1 0 0 0 f f f f t f s 1 0 20 "26" _null_ _null_ _null_ _null_ pg_stat_get_numscans _null_ _null_ _null_ ));
DESCR("statistics: number of scans done for table/index");
diff --git a/src/include/utils/mvstats.h b/src/include/utils/mvstats.h
index a074253..e11aefc 100644
--- a/src/include/utils/mvstats.h
+++ b/src/include/utils/mvstats.h
@@ -25,9 +25,11 @@ typedef struct MVStatsData {
/* statistics requested in ALTER TABLE ... ADD STATISTICS */
bool deps_enabled; /* analyze functional dependencies */
+ bool mcv_enabled; /* analyze MCV lists */
/* available statistics (computed by ANALYZE) */
bool deps_built; /* functional dependencies available */
+ bool mcv_built; /* MCV list is already available */
} MVStatsData;
typedef struct MVStatsData *MVStats;
@@ -66,6 +68,47 @@ typedef MVDependenciesData* MVDependencies;
#define MVSTAT_DEPS_TYPE_BASIC 1 /* basic dependencies type */
/*
+ * Multivariate MCV (most-common value) lists
+ *
+ * A straight-forward extension of MCV items - i.e. a list (array) of
+ * combinations of attribute values, together with a frequency and
+ * null flags.
+ */
+typedef struct MCVItemData {
+ double frequency; /* frequency of this combination */
+ bool *isnull; /* lags of NULL values (up to 32 columns) */
+ Datum *values; /* variable-length (ndimensions) */
+} MCVItemData;
+
+typedef MCVItemData *MCVItem;
+
+/* multivariate MCV list - essentally an array of MCV items */
+typedef struct MCVListData {
+ uint32 magic; /* magic constant marker */
+ uint32 type; /* type of MCV list (BASIC) */
+ uint32 ndimensions; /* number of dimensions */
+ uint32 nitems; /* number of MCV items in the array */
+ MCVItem *items; /* array of MCV items */
+} MCVListData;
+
+typedef MCVListData *MCVList;
+
+/* used to flag stats serialized to bytea */
+#define MVSTAT_MCV_MAGIC 0xE1A651C2 /* marks serialized bytea */
+#define MVSTAT_MCV_TYPE_BASIC 1 /* basic MCV list type */
+
+/*
+ * Limits used for mcv_max_items option, i.e. we're always guaranteed
+ * to have space for at least MVSTAT_MCVLIST_MIN_ITEMS, and we cannot
+ * have more than MVSTAT_MCVLIST_MAX_ITEMS items.
+ *
+ * This is just a boundary for the 'max' threshold - the actual list
+ * may of course contain less items than MVSTAT_MCVLIST_MIN_ITEMS.
+ */
+#define MVSTAT_MCVLIST_MIN_ITEMS 128 /* min items in MCV list */
+#define MVSTAT_MCVLIST_MAX_ITEMS 8192 /* max items in MCV list */
+
+/*
* TODO Maybe fetching the histogram/MCV list separately is inefficient?
* Consider adding a single `fetch_stats` method, fetching all
* stats specified using flags (or something like that).
@@ -74,24 +117,39 @@ MVStats list_mv_stats(Oid relid, int *nstats, bool built_only);
bytea * fetch_mv_rules(Oid mvoid);
bytea * fetch_mv_dependencies(Oid mvoid);
+bytea * fetch_mv_mcvlist(Oid mvoid);
bytea * serialize_mv_dependencies(MVDependencies dependencies);
+bytea * serialize_mv_mcvlist(MCVList mcvlist, int2vector *attrs,
+ VacAttrStats **stats);
/* deserialization of stats (serialization is private to analyze) */
MVDependencies deserialize_mv_dependencies(bytea * data);
+MCVList deserialize_mv_mcvlist(bytea * data);
+
+/*
+ * Returns index of the attribute number within the vector (i.e. a
+ * dimension within the stats).
+ */
+int mv_get_index(AttrNumber varattno, int2vector * stakeys);
/* FIXME this probably belongs somewhere else (not to operations stats) */
extern Datum pg_mv_stats_dependencies_info(PG_FUNCTION_ARGS);
extern Datum pg_mv_stats_dependencies_show(PG_FUNCTION_ARGS);
+extern Datum pg_mv_stats_mcvlist_info(PG_FUNCTION_ARGS);
MVDependencies
-build_mv_dependencies(int numrows, HeapTuple *rows,
- int2vector *attrs,
- int natts, VacAttrStats **vacattrstats);
+build_mv_dependencies(int numrows, HeapTuple *rows, int2vector *attrs,
+ VacAttrStats **stats);
+
+MCVList
+build_mv_mcvlist(int numrows, HeapTuple *rows, int2vector *attrs,
+ VacAttrStats **stats, int *numrows_filtered);
void build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
- int natts, VacAttrStats **vacattrstats);
+ int natts, VacAttrStats **vacattrstats);
-void update_mv_stats(Oid relid, MVDependencies dependencies);
+void update_mv_stats(Oid relid, MVDependencies dependencies, MCVList mcvlist,
+ int2vector *attrs, VacAttrStats **stats);
#endif
diff --git a/src/test/regress/expected/mv_mcv.out b/src/test/regress/expected/mv_mcv.out
new file mode 100644
index 0000000..fa298ea
--- /dev/null
+++ b/src/test/regress/expected/mv_mcv.out
@@ -0,0 +1,210 @@
+-- data type passed by value
+CREATE TABLE mcv_list (
+ a INT,
+ b INT,
+ c INT
+);
+-- unknown column
+ALTER TABLE mcv_list ADD STATISTICS (mcv) ON (unknown_column);
+ERROR: column "unknown_column" referenced in statistics does not exist
+-- single column
+ALTER TABLE mcv_list ADD STATISTICS (mcv) ON (a);
+ERROR: multivariate stats require 2 or more columns
+-- single column, duplicated
+ALTER TABLE mcv_list ADD STATISTICS (mcv) ON (a, a);
+ERROR: duplicate column name in statistics definition
+-- two columns, one duplicated
+ALTER TABLE mcv_list ADD STATISTICS (mcv) ON (a, a, b);
+ERROR: duplicate column name in statistics definition
+-- unknown option
+ALTER TABLE mcv_list ADD STATISTICS (unknown_option) ON (a, b, c);
+ERROR: unrecognized STATISTICS option "unknown_option"
+-- missing MCV statistics
+ALTER TABLE mcv_list ADD STATISTICS (dependencies, max_mcv_items 200) ON (a, b, c);
+ERROR: option 'mcv' is required by other options(s)
+-- invalid mcv_max_items value / too low
+ALTER TABLE mcv_list ADD STATISTICS (mcv, max_mcv_items 10) ON (a, b, c);
+ERROR: max number of MCV items must be at least 128
+-- invalid mcv_max_items value / too high
+ALTER TABLE mcv_list ADD STATISTICS (mcv, max_mcv_items 10000) ON (a, b, c);
+ERROR: max number of MCV items is 8192
+-- correct command
+ALTER TABLE mcv_list ADD STATISTICS (mcv) ON (a, b, c);
+-- random data
+INSERT INTO mcv_list
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+ANALYZE mcv_list;
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+ mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info
+-------------+-----------+--------------------------
+ t | f |
+(1 row)
+
+TRUNCATE mcv_list;
+-- a => b, a => c, b => c
+INSERT INTO mcv_list
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE mcv_list;
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+ mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info
+-------------+-----------+--------------------------
+ t | t | nitems=1000
+(1 row)
+
+TRUNCATE mcv_list;
+-- a => b, a => c
+INSERT INTO mcv_list
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE mcv_list;
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+ mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info
+-------------+-----------+--------------------------
+ t | t | nitems=1000
+(1 row)
+
+TRUNCATE mcv_list;
+-- check explain (expect bitmap index scan, not plain index scan)
+INSERT INTO mcv_list
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX mcv_idx ON mcv_list (a, b);
+ANALYZE mcv_list;
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+ mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info
+-------------+-----------+--------------------------
+ t | t | nitems=100
+(1 row)
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mcv_list WHERE a = 10 AND b = 5;
+ QUERY PLAN
+--------------------------------------------
+ Bitmap Heap Scan on mcv_list
+ Recheck Cond: ((a = 10) AND (b = 5))
+ -> Bitmap Index Scan on mcv_idx
+ Index Cond: ((a = 10) AND (b = 5))
+(4 rows)
+
+DELETE FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+DROP TABLE mcv_list;
+-- varlena type (text)
+CREATE TABLE mcv_list (
+ a TEXT,
+ b TEXT,
+ c TEXT
+);
+ALTER TABLE mcv_list ADD STATISTICS (mcv) ON (a, b, c);
+-- random data
+INSERT INTO mcv_list
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+ANALYZE mcv_list;
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+ mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info
+-------------+-----------+--------------------------
+ t | f |
+(1 row)
+
+TRUNCATE mcv_list;
+-- a => b, a => c, b => c
+INSERT INTO mcv_list
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE mcv_list;
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+ mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info
+-------------+-----------+--------------------------
+ t | t | nitems=1000
+(1 row)
+
+TRUNCATE mcv_list;
+-- a => b, a => c
+INSERT INTO mcv_list
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE mcv_list;
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+ mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info
+-------------+-----------+--------------------------
+ t | t | nitems=1000
+(1 row)
+
+TRUNCATE mcv_list;
+-- check explain (expect bitmap index scan, not plain index scan)
+INSERT INTO mcv_list
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX mcv_idx ON mcv_list (a, b);
+ANALYZE mcv_list;
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+ mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info
+-------------+-----------+--------------------------
+ t | t | nitems=100
+(1 row)
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mcv_list WHERE a = '10' AND b = '5';
+ QUERY PLAN
+------------------------------------------------------------
+ Bitmap Heap Scan on mcv_list
+ Recheck Cond: ((a = '10'::text) AND (b = '5'::text))
+ -> Bitmap Index Scan on mcv_idx
+ Index Cond: ((a = '10'::text) AND (b = '5'::text))
+(4 rows)
+
+TRUNCATE mcv_list;
+-- check explain (expect bitmap index scan, not plain index scan) with NULLs
+INSERT INTO mcv_list
+ SELECT
+ (CASE WHEN i/10000 = 0 THEN NULL ELSE i/10000 END),
+ (CASE WHEN i/20000 = 0 THEN NULL ELSE i/20000 END),
+ (CASE WHEN i/40000 = 0 THEN NULL ELSE i/40000 END)
+ FROM generate_series(1,1000000) s(i);
+ANALYZE mcv_list;
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+ mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info
+-------------+-----------+--------------------------
+ t | t | nitems=100
+(1 row)
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mcv_list WHERE a IS NULL AND b IS NULL;
+ QUERY PLAN
+---------------------------------------------------
+ Bitmap Heap Scan on mcv_list
+ Recheck Cond: ((a IS NULL) AND (b IS NULL))
+ -> Bitmap Index Scan on mcv_idx
+ Index Cond: ((a IS NULL) AND (b IS NULL))
+(4 rows)
+
+DELETE FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+DROP TABLE mcv_list;
+-- NULL values (mix of int and text columns)
+CREATE TABLE mcv_list (
+ a INT,
+ b TEXT,
+ c INT,
+ d TEXT
+);
+ALTER TABLE mcv_list ADD STATISTICS (mcv) ON (a, b, c, d);
+INSERT INTO mcv_list
+ SELECT
+ mod(i, 100),
+ (CASE WHEN mod(i, 200) = 0 THEN NULL ELSE mod(i,200) END),
+ mod(i, 400),
+ (CASE WHEN mod(i, 300) = 0 THEN NULL ELSE mod(i,600) END)
+ FROM generate_series(1,10000) s(i);
+ANALYZE mcv_list;
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+ mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info
+-------------+-----------+--------------------------
+ t | t | nitems=1200
+(1 row)
+
+DELETE FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+DROP TABLE mcv_list;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 82c2659..80375b8 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1357,7 +1357,9 @@ pg_mv_stats| SELECT n.nspname AS schemaname,
c.relname AS tablename,
s.stakeys AS attnums,
length(s.stadeps) AS depsbytes,
- pg_mv_stats_dependencies_info(s.stadeps) AS depsinfo
+ pg_mv_stats_dependencies_info(s.stadeps) AS depsinfo,
+ length(s.stamcv) AS mcvbytes,
+ pg_mv_stats_mcvlist_info(s.stamcv) AS mcvinfo
FROM ((pg_mv_statistic s
JOIN pg_class c ON ((c.oid = s.starelid)))
LEFT JOIN pg_namespace n ON ((n.oid = c.relnamespace)));
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index c41762c..78c9b04 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -111,4 +111,4 @@ test: event_trigger
test: stats
# run tests of multivariate stats
-test: mv_dependencies
+test: mv_dependencies mv_mcv
diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule
index 3845b0f..3f9884f 100644
--- a/src/test/regress/serial_schedule
+++ b/src/test/regress/serial_schedule
@@ -153,3 +153,4 @@ test: xml
test: event_trigger
test: stats
test: mv_dependencies
+test: mv_mcv
diff --git a/src/test/regress/sql/mv_mcv.sql b/src/test/regress/sql/mv_mcv.sql
new file mode 100644
index 0000000..090731e
--- /dev/null
+++ b/src/test/regress/sql/mv_mcv.sql
@@ -0,0 +1,181 @@
+-- data type passed by value
+CREATE TABLE mcv_list (
+ a INT,
+ b INT,
+ c INT
+);
+
+-- unknown column
+ALTER TABLE mcv_list ADD STATISTICS (mcv) ON (unknown_column);
+
+-- single column
+ALTER TABLE mcv_list ADD STATISTICS (mcv) ON (a);
+
+-- single column, duplicated
+ALTER TABLE mcv_list ADD STATISTICS (mcv) ON (a, a);
+
+-- two columns, one duplicated
+ALTER TABLE mcv_list ADD STATISTICS (mcv) ON (a, a, b);
+
+-- unknown option
+ALTER TABLE mcv_list ADD STATISTICS (unknown_option) ON (a, b, c);
+
+-- missing MCV statistics
+ALTER TABLE mcv_list ADD STATISTICS (dependencies, max_mcv_items 200) ON (a, b, c);
+
+-- invalid mcv_max_items value / too low
+ALTER TABLE mcv_list ADD STATISTICS (mcv, max_mcv_items 10) ON (a, b, c);
+
+-- invalid mcv_max_items value / too high
+ALTER TABLE mcv_list ADD STATISTICS (mcv, max_mcv_items 10000) ON (a, b, c);
+
+-- correct command
+ALTER TABLE mcv_list ADD STATISTICS (mcv) ON (a, b, c);
+
+-- random data
+INSERT INTO mcv_list
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+
+ANALYZE mcv_list;
+
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+
+TRUNCATE mcv_list;
+
+-- a => b, a => c, b => c
+INSERT INTO mcv_list
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+
+ANALYZE mcv_list;
+
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+
+TRUNCATE mcv_list;
+
+-- a => b, a => c
+INSERT INTO mcv_list
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+
+ANALYZE mcv_list;
+
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+
+TRUNCATE mcv_list;
+
+-- check explain (expect bitmap index scan, not plain index scan)
+INSERT INTO mcv_list
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX mcv_idx ON mcv_list (a, b);
+
+ANALYZE mcv_list;
+
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mcv_list WHERE a = 10 AND b = 5;
+
+DELETE FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+DROP TABLE mcv_list;
+
+-- varlena type (text)
+CREATE TABLE mcv_list (
+ a TEXT,
+ b TEXT,
+ c TEXT
+);
+
+ALTER TABLE mcv_list ADD STATISTICS (mcv) ON (a, b, c);
+
+-- random data
+INSERT INTO mcv_list
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+
+ANALYZE mcv_list;
+
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+
+TRUNCATE mcv_list;
+
+-- a => b, a => c, b => c
+INSERT INTO mcv_list
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+
+ANALYZE mcv_list;
+
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+
+TRUNCATE mcv_list;
+
+-- a => b, a => c
+INSERT INTO mcv_list
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE mcv_list;
+
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+
+TRUNCATE mcv_list;
+
+-- check explain (expect bitmap index scan, not plain index scan)
+INSERT INTO mcv_list
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX mcv_idx ON mcv_list (a, b);
+ANALYZE mcv_list;
+
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mcv_list WHERE a = '10' AND b = '5';
+
+TRUNCATE mcv_list;
+
+-- check explain (expect bitmap index scan, not plain index scan) with NULLs
+INSERT INTO mcv_list
+ SELECT
+ (CASE WHEN i/10000 = 0 THEN NULL ELSE i/10000 END),
+ (CASE WHEN i/20000 = 0 THEN NULL ELSE i/20000 END),
+ (CASE WHEN i/40000 = 0 THEN NULL ELSE i/40000 END)
+ FROM generate_series(1,1000000) s(i);
+ANALYZE mcv_list;
+
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mcv_list WHERE a IS NULL AND b IS NULL;
+
+DELETE FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+DROP TABLE mcv_list;
+
+-- NULL values (mix of int and text columns)
+CREATE TABLE mcv_list (
+ a INT,
+ b TEXT,
+ c INT,
+ d TEXT
+);
+
+ALTER TABLE mcv_list ADD STATISTICS (mcv) ON (a, b, c, d);
+
+INSERT INTO mcv_list
+ SELECT
+ mod(i, 100),
+ (CASE WHEN mod(i, 200) = 0 THEN NULL ELSE mod(i,200) END),
+ mod(i, 400),
+ (CASE WHEN mod(i, 300) = 0 THEN NULL ELSE mod(i,600) END)
+ FROM generate_series(1,10000) s(i);
+
+ANALYZE mcv_list;
+
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+
+DELETE FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+DROP TABLE mcv_list;
--
2.0.5
[text/x-diff] 0004-multivariate-histograms.patch (114.6K, ../../54C3FED3.1060600@2ndquadrant.com/5-0004-multivariate-histograms.patch)
download | inline diff:
From 397a9b96670097df72f95af04687b1874fb6ae31 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Sun, 11 Jan 2015 20:18:24 +0100
Subject: [PATCH 4/4] multivariate histograms
- extends the pg_mv_statistic catalog (add 'hist' fields)
- building the histograms during ANALYZE
- simple estimation while planning the queries
FIX: don't build histogram by default
FIX: analyze histogram only when requested
FIX: improvements in clauselist_mv_selectivity_histogram()
FIX: minor cleanup in multivariate histograms
- move BUCKET_SIZE_SERIALIZED() macro to histogram.c
- rename MVHIST_ constants to MVSTAT_HIST
FIX: comment about building histograms on too many columns
FIX: added comment about check in ALTER TABLE ... ADD STATISTICS
FIX: added comment about handling DROP TABLE / DROP COLUMN
FIX: added comment about ALTER TABLE ... DROP STATISTICS
FIX: comment about building NULL-buckets for a histogram
FIX: initial support for all data types and NULL in MV histograms
This changes the serialize_histogram/update_mv_stats in a bit
strange way (passing VacAttrStats all over the place). This
needs to be improved, somehow, before rebasing into the
histogram part. Otherwise it'll cause needless conflicts.
FIX: refactoring lookup_var_attr_stats() / histograms
FIX: a set of regression tests for MV histograms
This is mostly equal to a combination of all the regression tests
for functional dependencies / MCV lists.
The last test fails due to Assert(!isNull) in partition_bucket()
which prevents NULL values in histograms.
FIX: remove the two memcmp-based comparators (used for histograms)
FIX: comment about memory corruption in deserializing histograms
FIX: remove CompareScalarsContext/ScalarMCVItem from common.h
FIX: fix the lookup_vac_attr() refactoring in histograms
FIX: get rid of the custom comparators in histogram.c
FIX: building NULL-buckets - buckets with just NULLs in some dimension(s)
FIX: fixed bugs in serialize/deserialize methods for histogram
When serializing, BUCKET_MIN_INDEXES were set twice (once instead
of BUCKET_MAX_INDEXES, which were not set at all).
When deserializing, the 'tmp' pointer was not advanced, so just
the first bucket was ever deserialized (and copied into all the
histogram buckets).
Added a few asserts into deserialize method, similarly to how
it's done in serialize.
FIX: formatting issues in the histogram regression test
FIX: remove sample-dependent results from histogram regression test
FIX: add USE_ASSERT_CHECKING to assert-only variable (histogram)
FIX: check ADD STATISTICS options (histograms)
FIX: improved comments/docs for the multivariate histograms
FIX: reuse DimensionInfo (after move to common.h)
FIX: remove obsolete TODO about NULL-buckets, improve comments
FIX: move multivariate histogram definitions after MCV lists
FIX: correct variable names in error message (dimension index 'j')
FIX: add support for 'IS [NOT] NULL' support to histograms
FIX: add regression test for ADD STATISTICS options (histograms)
FIX: added regression test to test IS [NOT] NULL with histograms
FIX: make regression tests parallel-happy (histograms)
---
src/backend/catalog/system_views.sql | 4 +-
src/backend/commands/tablecmds.c | 55 +-
src/backend/optimizer/path/clausesel.c | 391 +++++-
src/backend/utils/mvstats/Makefile | 2 +-
src/backend/utils/mvstats/common.c | 67 +-
src/backend/utils/mvstats/common.h | 14 -
src/backend/utils/mvstats/histogram.c | 1778 ++++++++++++++++++++++++++++
src/backend/utils/mvstats/mcv.c | 1 +
src/include/catalog/pg_mv_statistic.h | 24 +-
src/include/catalog/pg_proc.h | 2 +
src/include/utils/mvstats.h | 99 +-
src/test/regress/expected/mv_histogram.out | 210 ++++
src/test/regress/expected/rules.out | 4 +-
src/test/regress/parallel_schedule | 2 +-
src/test/regress/serial_schedule | 1 +
src/test/regress/sql/mv_histogram.sql | 179 +++
16 files changed, 2776 insertions(+), 57 deletions(-)
create mode 100644 src/backend/utils/mvstats/histogram.c
create mode 100644 src/test/regress/expected/mv_histogram.out
create mode 100644 src/test/regress/sql/mv_histogram.sql
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 8acf160..3aa7d2b 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -160,7 +160,9 @@ CREATE VIEW pg_mv_stats AS
length(S.stadeps) as depsbytes,
pg_mv_stats_dependencies_info(S.stadeps) as depsinfo,
length(S.stamcv) AS mcvbytes,
- pg_mv_stats_mcvlist_info(S.stamcv) AS mcvinfo
+ pg_mv_stats_mcvlist_info(S.stamcv) AS mcvinfo,
+ length(S.stahist) AS histbytes,
+ pg_mv_stats_histogram_info(S.stahist) AS histinfo
FROM (pg_mv_statistic S JOIN pg_class C ON (C.oid = S.starelid))
LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace);
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 1f08c1c..2bd3884 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -11635,6 +11635,16 @@ static int compare_int16(const void *a, const void *b)
* multiple stats on the same columns with different options
* (say, a detailed MCV-only stats for some queries, histogram
* for others, etc.)
+ *
+ * FIXME Check that at least one of the statistic types is enabled, and
+ * that only compatible options are used. For example if 'mcv' is
+ * not selected, then 'mcv_max_items' can't be used (alternative
+ * might be to enable it automatically).
+ *
+ * TODO It might be useful to have ALTER TABLE DROP STATISTICS too, but
+ * it's tricky because there may be multiple kinds of stats for the
+ * same list of columns, with different options (e.g. one just MCV
+ * list, another with histogram, etc.).
*/
static void ATExecAddStatistics(AlteredTableInfo *tab, Relation rel,
StatisticsDef *def, LOCKMODE lockmode)
@@ -11652,12 +11662,15 @@ static void ATExecAddStatistics(AlteredTableInfo *tab, Relation rel,
/* by default build everything */
bool build_dependencies = false,
- build_mcv = false;
+ build_mcv = false,
+ build_histogram = false;
- int32 max_mcv_items = -1;
+ int32 max_buckets = -1,
+ max_mcv_items = -1;
/* options required because of other options */
- bool require_mcv = false;
+ bool require_mcv = false,
+ require_histogram = false;
Assert(IsA(def, StatisticsDef));
@@ -11735,6 +11748,29 @@ static void ATExecAddStatistics(AlteredTableInfo *tab, Relation rel,
MVSTAT_MCVLIST_MAX_ITEMS)));
}
+ else if (strcmp(opt->defname, "histogram") == 0)
+ build_histogram = defGetBoolean(opt);
+ else if (strcmp(opt->defname, "max_buckets") == 0)
+ {
+ max_buckets = defGetInt32(opt);
+
+ /* this option requires 'histogram' to be enabled */
+ require_histogram = true;
+
+ /* sanity check */
+ if (max_buckets < MVSTAT_HIST_MIN_BUCKETS)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("minimum number of buckets is %d",
+ MVSTAT_HIST_MIN_BUCKETS)));
+
+ else if (max_buckets > MVSTAT_HIST_MAX_BUCKETS)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("minimum number of buckets is %d",
+ MVSTAT_HIST_MAX_BUCKETS)));
+
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -11743,10 +11779,10 @@ static void ATExecAddStatistics(AlteredTableInfo *tab, Relation rel,
}
/* check that at least some statistics were requested */
- if (! (build_dependencies || build_mcv))
+ if (! (build_dependencies || build_mcv || build_histogram))
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("no statistics type (dependencies, mcv) was requested")));
+ errmsg("no statistics type (dependencies, mcv, histogram) was requested")));
/* now do some checking of the options */
if (require_mcv && (! build_mcv))
@@ -11754,6 +11790,11 @@ static void ATExecAddStatistics(AlteredTableInfo *tab, Relation rel,
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("option 'mcv' is required by other options(s)")));
+ if (require_histogram && (! build_histogram))
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("option 'histogram' is required by other options(s)")));
+
/* sort the attnums and build int2vector */
qsort(attnums, numcols, sizeof(int16), compare_int16);
stakeys = buildint2vector(attnums, numcols);
@@ -11771,10 +11812,14 @@ static void ATExecAddStatistics(AlteredTableInfo *tab, Relation rel,
values[Anum_pg_mv_statistic_deps_enabled -1] = BoolGetDatum(build_dependencies);
values[Anum_pg_mv_statistic_mcv_enabled -1] = BoolGetDatum(build_mcv);
+ values[Anum_pg_mv_statistic_hist_enabled -1] = BoolGetDatum(build_histogram);
+
values[Anum_pg_mv_statistic_mcv_max_items -1] = Int32GetDatum(max_mcv_items);
+ values[Anum_pg_mv_statistic_hist_max_buckets -1] = Int32GetDatum(max_buckets);
nulls[Anum_pg_mv_statistic_stadeps -1] = true;
nulls[Anum_pg_mv_statistic_stamcv -1] = true;
+ nulls[Anum_pg_mv_statistic_stahist -1] = true;
/* insert the tuple into pg_mv_statistic */
mvstatrel = heap_open(MvStatisticRelationId, RowExclusiveLock);
diff --git a/src/backend/optimizer/path/clausesel.c b/src/backend/optimizer/path/clausesel.c
index 1446fa0..a4e6d16 100644
--- a/src/backend/optimizer/path/clausesel.c
+++ b/src/backend/optimizer/path/clausesel.c
@@ -70,6 +70,8 @@ static Selectivity clauselist_mv_selectivity(PlannerInfo *root,
static Selectivity clauselist_mv_selectivity_mcvlist(PlannerInfo *root,
List *clauses, MVStats mvstats,
bool *fullmatch, Selectivity *lowsel);
+static Selectivity clauselist_mv_selectivity_histogram(PlannerInfo *root,
+ List *clauses, MVStats mvstats);
/****************************************************************************
* ROUTINES TO COMPUTE SELECTIVITIES
@@ -1119,6 +1121,7 @@ static Selectivity
clauselist_mv_selectivity(PlannerInfo *root, List *clauses, MVStats mvstats)
{
bool fullmatch = false;
+ Selectivity s1 = 0.0, s2 = 0.0;
/*
* Lowest frequency in the MCV list (may be used as an upper bound
@@ -1132,9 +1135,24 @@ clauselist_mv_selectivity(PlannerInfo *root, List *clauses, MVStats mvstats)
* MCV/histogram evaluation).
*/
- /* Evaluate the MCV selectivity */
- return clauselist_mv_selectivity_mcvlist(root, clauses, mvstats,
+ /* Evaluate the MCV first. */
+ s1 = clauselist_mv_selectivity_mcvlist(root, clauses, mvstats,
&fullmatch, &mcv_low);
+
+ /*
+ * If we got a full equality match on the MCV list, we're done (and
+ * the estimate is pretty good).
+ */
+ if (fullmatch && (s1 > 0.0))
+ return s1;
+
+ /* FIXME if (fullmatch) without matching MCV item, use the mcv_low
+ * selectivity as upper bound */
+
+ s2 = clauselist_mv_selectivity_histogram(root, clauses, mvstats);
+
+ /* TODO clamp to <= 1.0 (or more strictly, when possible) */
+ return s1 + s2;
}
/*
@@ -2024,3 +2042,372 @@ clauselist_mv_selectivity_mcvlist(PlannerInfo *root, List *clauses,
return s;
}
+
+/*
+ * Estimate selectivity of clauses using a histogram.
+ *
+ * If there's no histogram for the stats, the function returns 0.0.
+ *
+ * The general idea of this method is similar to how MCV lists are
+ * processed, except that this introduces the concept of a partial
+ * match (MCV only works with full match / mismatch).
+ *
+ * The algorithm works like this:
+ *
+ * 1) mark all buckets as 'full match'
+ * 2) walk through all the clauses
+ * 3) for a particular clause, walk through all the buckets
+ * 4) skip buckets that are already 'no match'
+ * 5) check clause for buckets that still match (at least partially)
+ * 6) sum frequencies for buckets to get selectivity
+ *
+ * Unlike MCV lists, histograms have a concept of a partial match.
+ *
+ * TODO This only handles AND-ed clauses, but it might work for OR-ed
+ * lists too - it just needs to reverse the logic a bit. I.e. start
+ * with 'no match' for all buckets, and increase the match level
+ * for the clauses (and skip buckets that are 'full match').
+ *
+ * TODO This might use a similar shortcut to MCV lists - count buckets
+ * marked as partial/full match, and terminate once this drop to 0.
+ * Not sure if it's really worth it - for MCV lists a situation like
+ * this is not uncommon, but for histograms it's not that clear.
+ */
+static Selectivity
+clauselist_mv_selectivity_histogram(PlannerInfo *root, List *clauses,
+ MVStats mvstats)
+{
+ int i;
+ Selectivity s = 0.0;
+ ListCell * l;
+ char *matches = NULL;
+ MVHistogram mvhist = NULL;
+
+ /* there's no histogram */
+ if (! mvstats->hist_built)
+ return 0.0;
+
+ /* There may be no histogram in the stats (check hist_built flag) */
+ mvhist = deserialize_mv_histogram(fetch_mv_histogram(mvstats->mvoid));
+
+ Assert (mvhist != NULL);
+ Assert (clauses != NIL);
+ Assert (list_length(clauses) >= 2);
+
+ /*
+ * Bitmap of bucket matches (mismatch, partial, full). by default
+ * all buckets fully match (and we'll eliminate them).
+ */
+ matches = palloc0(sizeof(char) * mvhist->nbuckets);
+ memset(matches, MVSTATS_MATCH_FULL, sizeof(char)*mvhist->nbuckets);
+
+ /* loop through the clauses and do the estimation */
+ foreach (l, clauses)
+ {
+ Node * clause = (Node*)lfirst(l);
+
+ /* it's either OpClause, or NullTest */
+ if (is_opclause(clause))
+ {
+ OpExpr * expr = (OpExpr*)clause;
+ bool varonleft = true;
+ bool ok;
+
+ FmgrInfo opproc; /* operator */
+ fmgr_info(get_opcode(expr->opno), &opproc);
+
+ ok = (NumRelids(clause) == 1) &&
+ (is_pseudo_constant_clause(lsecond(expr->args)) ||
+ (varonleft = false,
+ is_pseudo_constant_clause(linitial(expr->args))));
+
+ if (ok)
+ {
+ FmgrInfo ltproc;
+ Var * var = (varonleft) ? linitial(expr->args) : lsecond(expr->args);
+ Const * cst = (varonleft) ? lsecond(expr->args) : linitial(expr->args);
+ bool isgt = (! varonleft);
+
+ /*
+ * TODO Fetch only when really needed (probably for equality only)
+ *
+ * TODO Technically either lt/gt is sufficient.
+ *
+ * FIXME The code in analyze.c creates histograms only for types
+ * with enough ordering (by calling get_sort_group_operators).
+ * Is this the same assumption, i.e. are we certain that we
+ * get the ltproc/gtproc every time we ask? Or are there types
+ * where get_sort_group_operators returns ltopr and here we
+ * get nothing?
+ */
+ TypeCacheEntry *typecache
+ = lookup_type_cache(var->vartype, TYPECACHE_EQ_OPR | TYPECACHE_LT_OPR
+ | TYPECACHE_GT_OPR);
+
+ /* lookup dimension for the attribute */
+ int idx = mv_get_index(var->varattno, mvstats->stakeys);
+
+ fmgr_info(get_opcode(typecache->lt_opr), <proc);
+
+ /*
+ * Check this for all buckets that still have "true" in the bitmap
+ *
+ * We already know the clauses use suitable operators (because that's
+ * how we filtered them).
+ */
+ for (i = 0; i < mvhist->nbuckets; i++)
+ {
+ bool tmp;
+ MVBucket bucket = mvhist->buckets[i];
+
+ /*
+ * Skip buckets that were already eliminated - this is impotant
+ * considering how we update the info (we only lower the match)
+ */
+ if (matches[i] == MVSTATS_MATCH_NONE)
+ continue;
+
+ /*
+ * If it's not a "<" or ">" or "=" operator, just ignore the
+ * clause. Otherwise note the relid and attnum for the variable.
+ *
+ * TODO I'm really unsure the handling of 'isgt' flag (that is, clauses
+ * with reverse order of variable/constant) is correct. I wouldn't
+ * be surprised if there was some mixup. Using the lt/gt operators
+ * instead of messing with the opproc could make it simpler.
+ * It would however be using a different operator than the query,
+ * although it's not any shadier than using the selectivity function
+ * as is done currently.
+ *
+ * FIXME Once the min/max values are deduplicated, we can easily minimize
+ * the number of calls to the comparator (assuming we keep the
+ * deduplicated structure). See the note on compression at MVBucket
+ * serialize/deserialize methods.
+ */
+ switch (get_oprrest(expr->opno))
+ {
+ case F_SCALARLTSEL: /* column < constant */
+
+ if (! isgt) /* (var < const) */
+ {
+ /*
+ * First check whether the constant is below the lower boundary (in that
+ * case we can skip the bucket, because there's no overlap).
+ */
+ tmp = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ cst->constvalue,
+ bucket->min[idx]));
+ if (tmp)
+ {
+ matches[i] = MVSTATS_MATCH_NONE; /* no match */
+ continue;
+ }
+
+ /*
+ * Now check whether the upper boundary is below the constant (in that
+ * case it's a partial match).
+ */
+ tmp = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ cst->constvalue,
+ bucket->max[idx]));
+
+ if (tmp)
+ matches[i] = MVSTATS_MATCH_PARTIAL; /* partial match */
+ }
+ else /* (const < var) */
+ {
+ /*
+ * First check whether the constant is above the upper boundary (in that
+ * case we can skip the bucket, because there's no overlap).
+ */
+ tmp = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ bucket->max[idx],
+ cst->constvalue));
+ if (tmp)
+ {
+ matches[i] = MVSTATS_MATCH_NONE; /* no match */
+ continue;
+ }
+
+ /*
+ * Now check whether the lower boundary is below the constant (in that
+ * case it's a partial match).
+ */
+ tmp = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ bucket->min[idx],
+ cst->constvalue));
+
+ if (tmp)
+ matches[i] = MVSTATS_MATCH_PARTIAL; /* partial match */
+ }
+ break;
+
+ case F_SCALARGTSEL: /* column > constant */
+
+ if (! isgt) /* (var > const) */
+ {
+ /*
+ * First check whether the constant is above the upper boundary (in that
+ * case we can skip the bucket, because there's no overlap).
+ */
+ tmp = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ cst->constvalue,
+ bucket->max[idx]));
+ if (tmp)
+ {
+ matches[i] = MVSTATS_MATCH_NONE; /* no match */
+ continue;
+ }
+
+ /*
+ * Now check whether the lower boundary is below the constant (in that
+ * case it's a partial match).
+ */
+ tmp = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ cst->constvalue,
+ bucket->min[idx]));
+
+ if (tmp)
+ matches[i] = MVSTATS_MATCH_PARTIAL; /* partial match */
+ }
+ else /* (const > var) */
+ {
+ /*
+ * First check whether the constant is below the lower boundary (in
+ * that case we can skip the bucket, because there's no overlap).
+ */
+ tmp = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ bucket->min[idx],
+ cst->constvalue));
+ if (tmp)
+ {
+ matches[i] = MVSTATS_MATCH_NONE; /* no match */
+ continue;
+ }
+
+ /*
+ * Now check whether the upper boundary is below the constant (in that
+ * case it's a partial match).
+ */
+ tmp = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ bucket->max[idx],
+ cst->constvalue));
+
+ if (tmp)
+ matches[i] = MVSTATS_MATCH_PARTIAL; /* partial match */
+ }
+
+ break;
+
+ case F_EQSEL:
+
+ /*
+ * We only check whether the value is within the bucket, using the lt/gt
+ * operators fetched from type cache.
+ *
+ * TODO We'll use the default 50% estimate, but that's probably way off
+ * if there are multiple distinct values. Consider tweaking this a
+ * somehow, e.g. using only a part inversely proportional to the
+ * estimated number of distinct values in the bucket.
+ *
+ * TODO This does not handle inclusion flags at the moment, thus counting
+ * some buckets twice (when hitting the boundary).
+ *
+ * TODO Optimization is that if max[i] == min[i], it's effectively a MCV
+ * item and we can count the whole bucket as a complete match (thus
+ * using 100% bucket selectivity and not just 50%).
+ *
+ * TODO Technically some buckets may "degenerate" into single-value
+ * buckets (not necessarily for all the dimensions) - maybe this
+ * is better than keeping a separate MCV list (multi-dimensional).
+ * Update: Actually, that's unlikely to be better than a separate
+ * MCV list for two reasons - first, it requires ~2x the space
+ * (because of storing lower/upper boundaries) and second because
+ * the buckets are ranges - depending on the partitioning algorithm
+ * it may not even degenerate into (min=max) bucket. For example the
+ * the current partitioning algorithm never does that.
+ */
+ tmp = DatumGetBool(FunctionCall2Coll(<proc,
+ DEFAULT_COLLATION_OID,
+ cst->constvalue,
+ bucket->min[idx]));
+
+ if (tmp)
+ {
+ matches[i] = MVSTATS_MATCH_NONE; /* constvalue < min */
+ continue;
+ }
+
+ tmp = DatumGetBool(FunctionCall2Coll(<proc,
+ DEFAULT_COLLATION_OID,
+ bucket->max[idx],
+ cst->constvalue));
+
+ if (tmp)
+ {
+ matches[i] = MVSTATS_MATCH_NONE; /* constvalue > max */
+ continue;
+ }
+
+ /* partial match */
+ matches[i] = MVSTATS_MATCH_PARTIAL;
+
+ break;
+ }
+ }
+ }
+ }
+ else if (IsA(clause, NullTest))
+ {
+ NullTest * expr = (NullTest*)clause;
+ Var * var = (Var*)(expr->arg);
+
+ /* FIXME proper matching attribute to dimension */
+ int idx = mv_get_index(var->varattno, mvstats->stakeys);
+
+ /*
+ * Walk through the buckets and evaluate the current clause. We can
+ * skip items that were already ruled out, and terminate if there are
+ * no remaining buckets that might possibly match.
+ */
+ for (i = 0; i < mvhist->nbuckets; i++)
+ {
+ MVBucket bucket = mvhist->buckets[i];
+
+ /*
+ * Skip buckets that were already eliminated - this is impotant
+ * considering how we update the info (we only lower the match)
+ */
+ if (matches[i] == MVSTATS_MATCH_NONE)
+ continue;
+
+ /* if the clause mismatches the MCV item, set it as MATCH_NONE */
+ if ((expr->nulltesttype == IS_NULL)
+ && (! bucket->nullsonly[idx]))
+ matches[i] = MVSTATS_MATCH_NONE;
+ else if ((expr->nulltesttype == IS_NOT_NULL) &&
+ (bucket->nullsonly[idx]))
+ matches[i] = MVSTATS_MATCH_NONE;
+ }
+ }
+ }
+
+ /* now, walk through the buckets and sum the selectivities */
+ for (i = 0; i < mvhist->nbuckets; i++)
+ {
+ if (matches[i] == MVSTATS_MATCH_FULL)
+ s += mvhist->buckets[i]->ntuples;
+ else if (matches[i] == MVSTATS_MATCH_PARTIAL)
+ s += 0.5 * mvhist->buckets[i]->ntuples;
+ }
+
+ return s;
+}
diff --git a/src/backend/utils/mvstats/Makefile b/src/backend/utils/mvstats/Makefile
index 3c0aff4..9dbb3b6 100644
--- a/src/backend/utils/mvstats/Makefile
+++ b/src/backend/utils/mvstats/Makefile
@@ -12,6 +12,6 @@ subdir = src/backend/utils/mvstats
top_builddir = ../../../..
include $(top_builddir)/src/Makefile.global
-OBJS = common.o mcv.o dependencies.o
+OBJS = common.o dependencies.o histogram.o mcv.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/utils/mvstats/common.c b/src/backend/utils/mvstats/common.c
index 69ab805..f6edb2f 100644
--- a/src/backend/utils/mvstats/common.c
+++ b/src/backend/utils/mvstats/common.c
@@ -45,7 +45,8 @@ build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
{
MVDependencies deps = NULL;
MCVList mcvlist = NULL;
- int numrows_filtered = 0;
+ MVHistogram histogram = NULL;
+ int numrows_filtered = numrows;
/* int2 vector of attnums the stats should be computed on */
int2vector * attrs = mvstats[i].stakeys;
@@ -66,8 +67,23 @@ build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
if (mvstats->mcv_enabled)
mcvlist = build_mv_mcvlist(numrows, rows, attrs, stats, &numrows_filtered);
+ /*
+ * Build a multivariate histogram on the columns.
+ *
+ * FIXME remove the rows used to build the MCV from the histogram.
+ * Another option might be subtracting the MCV selectivities
+ * from the histogram, but I'm not sure whether that works
+ * accurately (maybe it introduces additional errors).
+ */
+ if ((numrows_filtered > 0) && (mvstats->hist_enabled))
+ histogram = build_mv_histogram(numrows_filtered, rows, attrs, stats, numrows);
+
/* store the histogram / MCV list in the catalog */
- update_mv_stats(mvstats[i].mvoid, deps, mcvlist, attrs, stats);
+ update_mv_stats(mvstats[i].mvoid, deps, mcvlist, histogram, attrs, stats);
+
+#ifdef MVSTATS_DEBUG
+ print_mv_histogram_info(histogram);
+#endif
}
}
@@ -149,7 +165,7 @@ list_mv_stats(Oid relid, int *nstats, bool built_only)
* Skip statistics that were not computed yet (if only stats
* that were already built were requested)
*/
- if (built_only && (! (stats->mcv_built || stats->deps_built)))
+ if (built_only && (! (stats->mcv_built || stats->deps_built || stats->hist_built)))
continue;
/* double the array size if needed */
@@ -161,10 +177,15 @@ list_mv_stats(Oid relid, int *nstats, bool built_only)
result[*nstats].mvoid = HeapTupleGetOid(htup);
result[*nstats].stakeys = buildint2vector(stats->stakeys.values, stats->stakeys.dim1);
+
result[*nstats].deps_enabled = stats->deps_enabled;
result[*nstats].mcv_enabled = stats->mcv_enabled;
+ result[*nstats].hist_enabled = stats->hist_enabled;
+
result[*nstats].deps_built = stats->deps_built;
result[*nstats].mcv_built = stats->mcv_built;
+ result[*nstats].hist_built = stats->hist_built;
+
*nstats += 1;
}
@@ -178,9 +199,16 @@ list_mv_stats(Oid relid, int *nstats, bool built_only)
return result;
}
+/*
+ * FIXME This adds statistics, but we need to drop statistics when the
+ * table is dropped. Not sure what to do when a column is dropped.
+ * Either we can (a) remove all stats on that column, (b) remove
+ * the column from defined stats and force rebuild, (c) remove the
+ * column on next ANALYZE. Or maybe something else?
+ */
void
update_mv_stats(Oid mvoid,
- MVDependencies dependencies, MCVList mcvlist,
+ MVDependencies dependencies, MCVList mcvlist, MVHistogram histogram,
int2vector *attrs, VacAttrStats **stats)
{
HeapTuple stup,
@@ -213,19 +241,31 @@ update_mv_stats(Oid mvoid,
values[Anum_pg_mv_statistic_stamcv - 1] = PointerGetDatum(data);
}
+ if (histogram != NULL)
+ {
+ bytea * data = serialize_mv_histogram(histogram, attrs, stats);
+ nulls[Anum_pg_mv_statistic_stahist-1] = (data == NULL);
+ values[Anum_pg_mv_statistic_stahist - 1]
+ = PointerGetDatum(data);
+ }
+
/* always replace the value (either by bytea or NULL) */
replaces[Anum_pg_mv_statistic_stadeps -1] = true;
replaces[Anum_pg_mv_statistic_stamcv -1] = true;
+ replaces[Anum_pg_mv_statistic_stahist-1] = true;
/* always change the availability flags */
nulls[Anum_pg_mv_statistic_deps_built -1] = false;
nulls[Anum_pg_mv_statistic_mcv_built -1] = false;
+ nulls[Anum_pg_mv_statistic_hist_built-1] = false;
replaces[Anum_pg_mv_statistic_deps_built-1] = true;
replaces[Anum_pg_mv_statistic_mcv_built -1] = true;
+ replaces[Anum_pg_mv_statistic_hist_built -1] = true;
values[Anum_pg_mv_statistic_deps_built-1] = BoolGetDatum(dependencies != NULL);
values[Anum_pg_mv_statistic_mcv_built -1] = BoolGetDatum(mcvlist != NULL);
+ values[Anum_pg_mv_statistic_hist_built -1] = BoolGetDatum(histogram != NULL);
/* Is there already a pg_mv_statistic tuple for this attribute? */
oldtup = SearchSysCache1(MVSTATOID,
@@ -302,25 +342,6 @@ compare_scalars_partition(const void *a, const void *b, void *arg)
return ApplySortComparator(da, false, db, false, ssup);
}
-/*
- * qsort_arg comparator for sorting Datum[] (row of Datums) when
- * counting distinct values.
- */
-int
-compare_scalars_memcmp(const void *a, const void *b, void *arg)
-{
- Size len = *(Size*)arg;
-
- return memcmp(a, b, len);
-}
-
-int
-compare_scalars_memcmp_2(const void *a, const void *b)
-{
- return memcmp(a, b, sizeof(Datum));
-}
-
-
/* initialize multi-dimensional sort */
MultiSortSupport
multi_sort_init(int ndims)
diff --git a/src/backend/utils/mvstats/common.h b/src/backend/utils/mvstats/common.h
index fca2782..f4309f7 100644
--- a/src/backend/utils/mvstats/common.h
+++ b/src/backend/utils/mvstats/common.h
@@ -47,18 +47,6 @@ typedef struct
int tupno; /* position index for tuple it came from */
} ScalarItem;
-typedef struct
-{
- int count; /* # of duplicates */
- int first; /* values[] index of first occurrence */
-} ScalarMCVItem;
-
-typedef struct
-{
- SortSupport ssup;
- int *tupnoLink;
-} CompareScalarsContext;
-
/* (de)serialization info */
typedef struct DimensionInfo {
int nvalues; /* number of deduplicated values */
@@ -94,5 +82,3 @@ int multi_sort_compare_dim(int dim, const SortItem *a,
int compare_datums_simple(Datum a, Datum b, SortSupport ssup);
int compare_scalars_simple(const void *a, const void *b, void *arg);
int compare_scalars_partition(const void *a, const void *b, void *arg);
-int compare_scalars_memcmp(const void *a, const void *b, void *arg);
-int compare_scalars_memcmp_2(const void *a, const void *b);
diff --git a/src/backend/utils/mvstats/histogram.c b/src/backend/utils/mvstats/histogram.c
new file mode 100644
index 0000000..3acbea2
--- /dev/null
+++ b/src/backend/utils/mvstats/histogram.c
@@ -0,0 +1,1778 @@
+/*-------------------------------------------------------------------------
+ *
+ * histogram.c
+ * POSTGRES multivariate histograms
+ *
+ *
+ * Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/utils/mvstats/histogram.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "common.h"
+
+/*
+ * Multivariate histograms
+ *
+ * Histograms are a collection of buckets, represented by n-dimensional
+ * rectangles. Each rectangle is delimited by a min/max value in each
+ * dimension, stored in an array, so that the bucket includes values
+ * fulfilling condition
+ *
+ * min[i] <= value[i] <= max[i]
+ *
+ * where 'i' is the dimension. In 1D this corresponds to a simple
+ * interval, in 2D to a rectangle, and in 3D to a block. If you can
+ * imagine this in 4D, congrats!
+ *
+ * In addition to the bounaries, each bucket tracks additional details:
+ *
+ * * frequency (fraction of tuples it matches)
+ * * whether the boundaries are inclusive or exclusive
+ * * whether the dimension contains only NULL values
+ * * number of distinct values in each dimension (for building)
+ *
+ * and possibly some additional information.
+ *
+ * We do expect to support multiple histogram types, with different
+ * features etc. The 'type' field is used to identify those types.
+ * Technically some histogram types might use completely different
+ * bucket representation, but that's not expected at the moment.
+ *
+ * Although the current implementation builds non-overlapping buckets,
+ * the code does not rely on the non-overlapping nature - there are
+ * interesting types of histograms / histogram building algorithms
+ * producing overlapping buckets.
+ *
+ * TODO Currently the histogram does not include information about what
+ * part of the table it covers (because the frequencies are
+ * computed from the rows that may be filtered by MCV list). Seems
+ * wrong, possibly causing misestimates (when not matching the MCV
+ * list, we'll probably get much higher selectivity).
+ *
+ *
+ * Estimating selectivity
+ * ----------------------
+ * With histograms, we always "match" a whole bucket, not indivitual
+ * rows (or values), irrespectedly of the type of clause. Therefore we
+ * can't use the optimizations for equality clauses, as in MCV lists.
+ *
+ * The current implementation uses histograms to estimates those types
+ * of clauses (think of WHERE conditions):
+ *
+ * (a) equality clauses WHERE (a = 1) AND (b = 2)
+ * (b) inequality clauses WHERE (a < 1) AND (b >= 2)
+ *
+ * It's possible to add more clauses, for example:
+ *
+ * (a) NULL clauses WHERE (a IS NULL) AND (b IS NOT NULL)
+ * (b) multi-var clauses WHERE (a > b)
+ *
+ * and so on. These are tasks for the future, not yet implemented.
+ *
+ * When used on low-cardinality data, histograms usually perform
+ * considerably worse than MCV lists (which are a good fit for this
+ * kind of data). This is especially true on categorical data, where
+ * ordering of the values is only loosely related to meaning of the
+ * data, as proper ordering is crucial for histograms.
+ *
+ * On high-cardinality data the histograms are usually a better choice,
+ * because MCV lists can't accurately represent the distribution.
+ *
+ * By evaluating a clause on a bucket, we may get one of three results:
+ *
+ * (a) FULL_MATCH - The bucket definitely matches the clause.
+ *
+ * (b) PARTIAL_MATCH - The bucket matches the clause, but not
+ * necessarily all the tuples it represents.
+ *
+ * (c) NO_MATCH - The bucket definitely does not match the clause.
+ *
+ * This may be illustrated using a range [1, 5], which is essentially
+ * a 1D bucket. With clause
+ *
+ * WHERE (a < 10) => FULL_MATCH (all range values are below
+ * 10, so the whole bucket matches)
+ *
+ * WHERE (a < 3) => PARTIAL_MATCH (there may be values matching
+ * the clause, but we don't know how many)
+ *
+ * WHERE (a < 0) => NO_MATCH (all range values are above 1, so
+ * no values from the bucket match)
+ *
+ * Some clauses may produce only some of those results - for example
+ * equality clauses may never produce FULL_MATCH as we always hit only
+ * part of the bucket, not all the values. This results in less accurate
+ * estimates compared to MCV lists, where we can hit a MCV items exactly
+ * (an extreme case of that is 'full match').
+ *
+ * There are clauses that may not produce any PARTIAL_MATCH results.
+ * A nice example of that is 'IS [NOT] NULL' clause, which either
+ * matches the bucket completely (FULL_MATCH) or not at all (NO_MATCH),
+ * thanks to how the NULL-buckets are constructed.
+ *
+ * TODO The IS [NOT] NULL clause is not yet implemented, but should be
+ * rather trivial to.
+ *
+ * Computing the total selectivity estimate is trivial - simply sum
+ * selectivities from all the FULL_MATCH and PARTIAL_MATCH buckets, but
+ * multiply the PARTIAL_MATCH buckets by 0.5 to minimize average error.
+ *
+ *
+ * NULL handling
+ * -------------
+ * Buckets may not contain tuples with NULL and non-NULL values in
+ * a single dimension (attribute). To handle this, the histogram may
+ * contain NULL-buckets, i.e. buckets with one or more NULL-only
+ * dimensions.
+ *
+ * The maximum number of NULL-buckets is determined by the number of
+ * attributes the histogram is built on. For N-dimensional histogram,
+ * the maximum number of NULL-buckets is 2^N. So for 8 attributes
+ * (which is the current value of MVSTATS_MAX_DIMENSIONS), there may be
+ * up to 256 NULL-buckets.
+ *
+ * Those buckets are only built if needed - if there are no NULL values
+ * in the data, no such buckets are built.
+ *
+ *
+ * Serialization
+ * -------------
+ * After serialization, the histograms are marked with 'magic' constant.
+ * to make sure the bytea really is a histogram in serialized form.
+ *
+ * FIXME info about deduplication
+ *
+ *
+ * TODO This structure is used both when building the histogram, and
+ * then when using it to compute estimates. That's why the last
+ * few elements are not used once the histogram is built.
+ *
+ * Add pointer to 'private' data, meant for private data for
+ * other algorithms for building the histogram. It also removes
+ * the bogus / unnecessary fields.
+ *
+ * TODO The limit on number of buckets is quite arbitrary, aiming for
+ * sufficient accuracy while still being fast. Probably should be
+ * replaced with a dynamic limit dependent on statistics target,
+ * number of attributes (dimensions) and statistics target
+ * associated with the attributes. Also, this needs to be related
+ * to the number of sampled rows, by either clamping it to a
+ * reasonable number (after seeing the number of rows) or using
+ * it when computing the number of rows to sample. Something like
+ * 10 rows per bucket seems reasonable.
+ *
+ * TODO Add MVSTAT_HIST_ROWS_PER_BUCKET tracking minimal number of
+ * tuples per bucket (also, see the previous TODO).
+ *
+ * TODO We may replace the bool arrays with a suitably large data type
+ * (say, uint16 or uint32) and get rid of the allocations. It's
+ * unlikely we'll ever support more than 32 columns as that'd
+ * result in poor precision, huge histograms (splitting each
+ * dimension once would mean 2^32 buckets), and very expensive
+ * estimation. MCVItem already does it this way.
+ *
+ * Update: Actually, this is not 100% true, because we're splitting
+ * a single bucket, not all the buckets at the same time. So each
+ * split simply adds one new bucket, and we choose the bucket that
+ * is most in need of a slit. So even with 32 columns this might
+ * give reasonable accuracy, maybe? After 1000 splits we'll get
+ * about 1001 buckets, and some may be quite large (if that area
+ * frequency has low frequency of tuples).
+ *
+ * There are other challenges though - e.g. with this many columns
+ * it's more likely to reference both label/non-label columns,
+ * which is rather quirky (especially with histograms).
+ *
+ * However, while this would save some space for histograms built
+ * on many columns, it won't save anything for up to 4 columns
+ * (actually, on less than 3 columns it's probably wasteful).
+ *
+ * TODO Maybe the distinct stats (both for combination of all columns
+ * and for combinations of various subsets of columns) should be
+ * moved to a separate structure (next to histogram/MCV/...) to
+ * make it useful even without a histogram computed etc.
+ */
+
+static MVBucket create_initial_mv_bucket(int numrows, HeapTuple *rows,
+ int2vector *attrs,
+ VacAttrStats **stats);
+
+static MVBucket select_bucket_to_partition(int nbuckets, MVBucket * buckets);
+
+static MVBucket partition_bucket(MVBucket bucket, int2vector *attrs,
+ VacAttrStats **stats);
+
+static MVBucket copy_mv_bucket(MVBucket bucket, uint32 ndimensions);
+
+static void update_bucket_ndistinct(MVBucket bucket, int2vector *attrs,
+ VacAttrStats ** stats);
+
+static void update_dimension_ndistinct(MVBucket bucket, int dimension,
+ int2vector *attrs,
+ VacAttrStats ** stats,
+ bool update_boundaries);
+
+static void create_null_buckets(MVHistogram histogram, int bucket_idx,
+ int2vector *attrs, VacAttrStats ** stats);
+
+static int bsearch_comparator(const void * a, const void * b);
+
+/*
+ * Each serialized bucket needs to store (in this order):
+ *
+ * - number of tuples (float)
+ * - number of distinct (float)
+ * - min inclusive flags (ndim * sizeof(bool))
+ * - max inclusive flags (ndim * sizeof(bool))
+ * - null dimension flags (ndim * sizeof(bool))
+ * - min boundary indexes (2 * ndim * sizeof(int32))
+ * - max boundary indexes (2 * ndim * sizeof(int32))
+ *
+ * So in total:
+ *
+ * ndim * (4 * sizeof(int32) + 3 * sizeof(bool)) +
+ * 2 * sizeof(float)
+ */
+#define BUCKET_SIZE(ndims) \
+ (ndims * (4 * sizeof(int32) + 3 * sizeof(bool)) + 2 * sizeof(float))
+
+/* pointers into a flat serialized bucket of BUCKET_SIZE(n) bytes */
+#define BUCKET_NTUPLES(b) ((float*)b)
+#define BUCKET_NDISTINCT(b) ((float*)(b + sizeof(float)))
+#define BUCKET_MIN_INCL(b,n) ((bool*)(b + 2 * sizeof(float)))
+#define BUCKET_MAX_INCL(b,n) (BUCKET_MIN_INCL(b,n) + n)
+#define BUCKET_NULLS_ONLY(b,n) (BUCKET_MAX_INCL(b,n) + n)
+#define BUCKET_MIN_INDEXES(b,n) ((int32*)(BUCKET_NULLS_ONLY(b,n) + n))
+#define BUCKET_MAX_INDEXES(b,n) ((BUCKET_MIN_INDEXES(b,n) + n))
+
+/* some debugging methods */
+#ifdef MVSTATS_DEBUG
+static void print_mv_histogram_info(MVHistogram histogram);
+#endif
+
+/*
+ * Building a multivariate algorithm. In short it first creates a single
+ * bucket containing all the rows, and then repeatedly split is by first
+ * searching for the bucket / dimension most in need of a split.
+ *
+ * The current criteria is rather simple, by looking at the number of
+ * distinct values (combination of column values for bucket, column
+ * values for a dimension). This is somehow naive, but seems to work
+ * quite well. See the discussion at select_bucket_to_partition and
+ * partition_bucket for more details about alternative algorithms.
+ *
+ * So the current algorithm looks like this:
+ *
+ * build NULL-buckets (create_null_buckets)
+ *
+ * while [not reaching maximum number of buckets]
+ *
+ * choose bucket to partition (max distinct combinations)
+ * if no bucket to partition
+ * terminate the algorithm
+ *
+ * choose bucket dimension to partition (max distinct values)
+ * split the bucket into two buckets
+ */
+MVHistogram
+build_mv_histogram(int numrows, HeapTuple *rows, int2vector *attrs,
+ VacAttrStats **stats, int numrows_total)
+{
+ int i;
+ int ndistinct;
+ int numattrs = attrs->dim1;
+ int *ndistincts = (int*)palloc0(sizeof(int) * numattrs);
+
+ MVHistogram histogram = (MVHistogram)palloc0(sizeof(MVHistogramData));
+
+ HeapTuple * rows_copy = (HeapTuple*)palloc0(numrows * sizeof(HeapTuple));
+ memcpy(rows_copy, rows, sizeof(HeapTuple) * numrows);
+
+ Assert((numattrs >= 2) && (numattrs <= MVSTATS_MAX_DIMENSIONS));
+
+ histogram->ndimensions = numattrs;
+
+ histogram->magic = MVSTAT_HIST_MAGIC;
+ histogram->type = MVSTAT_HIST_TYPE_BASIC;
+ histogram->nbuckets = 1;
+
+ /* create max buckets (better than repalloc for short-lived objects) */
+ histogram->buckets = (MVBucket*)palloc0(MVSTAT_HIST_MAX_BUCKETS * sizeof(MVBucket));
+
+ /* create the initial bucket, covering the whole sample set */
+ histogram->buckets[0]
+ = create_initial_mv_bucket(numrows, rows_copy, attrs, stats);
+
+ /*
+ * We may use this to limit number of buckets too - there can never
+ * be more than ndistinct buckets (or ndistinct/k if we require at
+ * least k tuples per bucket.
+ *
+ * With NULL buckets it's a bit more complicated, because there may
+ * be 2^ndims NULL buckets, and if each contains a single tuple then
+ * there may be up to
+ *
+ * (ndistinct - 2^ndims)/k + 2^ndims
+ *
+ * buckets. But of course, it may happen that (ndistinct < 2^ndims)
+ * which needs to be checked.
+ *
+ * TODO Use this for alternative estimate of number of buckets.
+ */
+ ndistinct = histogram->buckets[0]->ndistinct;
+
+ /*
+ * The initial bucket may contain NULL values, so we have to create
+ * buckets with NULL-only dimensions.
+ *
+ * FIXME We may need up to 2^ndims buckets - check that there are
+ * enough buckets (MVSTAT_HIST_MAX_BUCKETS >= 2^ndims).
+ */
+ create_null_buckets(histogram, 0, attrs, stats);
+
+ /* keep the global ndistinct values */
+ for (i = 0; i < numattrs; i++)
+ ndistincts[i] = histogram->buckets[0]->ndistincts[i];
+
+ while (histogram->nbuckets < MVSTAT_HIST_MAX_BUCKETS)
+ {
+ MVBucket bucket = select_bucket_to_partition(histogram->nbuckets,
+ histogram->buckets);
+
+ /* no more buckets to partition */
+ if (bucket == NULL)
+ break;
+
+ histogram->buckets[histogram->nbuckets]
+ = partition_bucket(bucket, attrs, stats);
+
+ histogram->nbuckets += 1;
+ }
+
+ /* finalize the frequencies etc. */
+ for (i = 0; i < histogram->nbuckets; i++)
+ {
+ int d;
+ histogram->buckets[i]->ntuples
+ = (histogram->buckets[i]->numrows * 1.0) / numrows_total;
+ histogram->buckets[i]->ndistinct
+ = (histogram->buckets[i]->ndistinct * 1.0) / ndistinct;
+
+ for (d = 0; d < numattrs; d++)
+ histogram->buckets[i]->ndistincts[d]
+ = (histogram->buckets[i]->ndistincts[d] * 1.0) / ndistincts[d];
+ }
+
+ pfree(ndistincts);
+
+ return histogram;
+}
+
+/* fetch the histogram (as a bytea) from the pg_mv_statistic catalog */
+bytea *
+fetch_mv_histogram(Oid mvoid)
+{
+ Relation indrel;
+ SysScanDesc indscan;
+ ScanKeyData skey;
+ HeapTuple htup;
+ bytea *stahist = NULL;
+
+ /* Prepare to scan pg_mv_statistic for entries having indrelid = this rel. */
+ ScanKeyInit(&skey,
+ ObjectIdAttributeNumber,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(mvoid));
+
+ indrel = heap_open(MvStatisticRelationId, AccessShareLock);
+ indscan = systable_beginscan(indrel, MvStatisticOidIndexId, true,
+ NULL, 1, &skey);
+
+ while (HeapTupleIsValid(htup = systable_getnext(indscan)))
+ {
+ bool isnull = false;
+ Datum hist = SysCacheGetAttr(MVSTATOID, htup,
+ Anum_pg_mv_statistic_stahist, &isnull);
+
+ Assert(!isnull);
+
+ stahist = DatumGetByteaP(hist);
+
+ break;
+ }
+
+ systable_endscan(indscan);
+
+ heap_close(indrel, AccessShareLock);
+
+ /*
+ * TODO Maybe save the histogram into relcache, as in RelationGetIndexList
+ * (which was used as an inspiration of this one)?.
+ */
+
+ return stahist;
+}
+
+/* print some basic info about the histogram */
+Datum
+pg_mv_stats_histogram_info(PG_FUNCTION_ARGS)
+{
+ bytea *data = PG_GETARG_BYTEA_P(0);
+ char *result;
+
+ MVHistogram hist = deserialize_mv_histogram(data);
+
+ result = palloc0(128);
+ snprintf(result, 128, "nbuckets=%d", hist->nbuckets);
+
+ PG_RETURN_TEXT_P(cstring_to_text(result));
+}
+
+
+/* used to pass context into bsearch() */
+static SortSupport ssup_private = NULL;
+
+/*
+ * Serialize the MV histogram into a bytea value. The basic algorithm
+ * is simple, and mostly mimincs the MCV serialization:
+ *
+ * (1) perform deduplication for each attribute (separately)
+ * (a) collect all (non-NULL) attribute values from all buckets
+ * (b) sort the data (using 'lt' from VacAttrStats)
+ * (c) remove duplicate values from the array
+ *
+ * (2) serialize the arrays into a bytea value
+ *
+ * (3) process all buckets
+ * (a) replace min/max values with indexes into the arrays
+ *
+ * Each attribute has to be processed separately, because we're mixing
+ * different datatypes, and we don't know what equality means for them.
+ * We're also mixing pass-by-value and pass-by-ref types, and so on.
+ *
+ * We'll use 32-bit values for the indexes in step (3), although we
+ * could probably use just 16 bits as we don't allow more than 8k
+ * buckets in the histogram max_buckets (well, we might increase this
+ * to 16k and still fit into signed 16-bits). But let's be lazy and rely
+ * on the varlena compression to kick in. If most bytes will be 0x00
+ * so it should work nicely.
+ *
+ *
+ * Deduplication in serialization
+ * ------------------------------
+ * The deduplication is very effective and important here, because every
+ * time we split a bucket, we keep all the boundary values, except for
+ * the dimension that was used for the split. Another way to look at
+ * this is that each split introduces 1 new value (the value used to do
+ * the split). A histogram with M buckets was created by (M-1) splits
+ * of the initial bucket, and each bucket has 2*N boundary values. So
+ * assuming the initial bucket does not have any 'collapsed' dimensions,
+ * the number of distinct values is
+ *
+ * (2*N + (M-1))
+ *
+ * but the total number of boundary values is
+ *
+ * 2*N*M
+ *
+ * which is clearly much higher. For a histogram on two columns, with
+ * 1024 buckets, it's 1027 vs. 4096. Of course, we're not saving all
+ * the difference (because we'll use 32-bit indexes into the values).
+ * But with large values (e.g. stored as varlena), this saves a lot.
+ *
+ * An interesting feature is that the total number of distinct values
+ * does not really grow with the number of dimensions, except for the
+ * size of the initial bucket. After that it only depends on number of
+ * buckets (i.e. number of splits).
+ *
+ * XXX Of course this only holds for the current histogram building
+ * algorithm. Algorithms doing the splits differently (e.g.
+ * producing overlapping buckets) may behave differently.
+ *
+ * TODO This only confirms we can use the uint16 indexes. The worst
+ * that could happen is if all the splits happened by a single
+ * dimension. To exhaust the uint16 this would require ~64k
+ * splits (needs to be reflected in MVSTAT_HIST_MAX_BUCKETS).
+ *
+ * TODO We don't need to use a separate boolean for each flag, instead
+ * use a single char and set bits.
+ *
+ * TODO We might get a bit better compression by considering the actual
+ * data type length. The current implementation treats all data
+ * types passed by value as requiring 8B, but for INT it's actually
+ * just 4B etc.
+ *
+ * OTOH this is only related to the lookup table, and most of the
+ * space is occupied by the buckets (with int16 indexes).
+ *
+ *
+ * Varlena compression
+ * -------------------
+ * This encoding may prevent automatic varlena compression (similarly
+ * to JSONB), because first part of the serialized bytea will be an
+ * array of unique values (although sorted), and pglz decides whether
+ * to compress by trying to compress the first part (~1kB or so). Which
+ * is likely to be poor, due to the lack of repetition.
+ *
+ * One possible cure to that might be storing the buckets first, and
+ * then the deduplicated arrays. The buckets might be better suited
+ * for compression.
+ *
+ * On the other hand the encoding scheme is a context-aware compression,
+ * usually compressing to ~30% (or less, with large data types). So the
+ * lack of pglz compression may be OK.
+ *
+ * XXX But maybe we don't really want to compress this, to save on
+ * planning time?
+ *
+ * TODO Try storing the buckets / deduplicated arrays in reverse order,
+ * measure impact on compression.
+ *
+ *
+ * Deserialization
+ * ---------------
+ * The deserialization is currently implemented so that it reconstructs
+ * the histogram back into the same structures - this involves quite
+ * a few of memcpy() and palloc(), but maybe we could create a special
+ * structure for the serialized histogram, and access the data directly,
+ * without the unpacking.
+ *
+ * Not only it would save some memory and CPU time, but might actually
+ * work better with CPU caches (not polluting the caches).
+ *
+ * TODO Try to keep the compressed form, instead of deserializing it to
+ * MVHistogram/MVBucket.
+ *
+ *
+ * General TODOs
+ * -------------
+ * FIXME This probably leaks memory, or at least uses it inefficiently
+ * (many small palloc() calls instead of a large one).
+ *
+ * FIXME This probably leaks memory, or at least uses it inefficiently
+ * (many small palloc() calls instead of a large one).
+ *
+ * TODO Consider packing boolean flags (NULL) for each item into 'char'
+ * or a longer type (instead of using an array of bool items).
+ */
+bytea *
+serialize_mv_histogram(MVHistogram histogram, int2vector *attrs,
+ VacAttrStats **stats)
+{
+ int i = 0, j = 0;
+ Size total_length = 0;
+
+ bytea *output = NULL;
+ char *data = NULL;
+
+ int nbuckets = histogram->nbuckets;
+ int ndims = histogram->ndimensions;
+
+ /* allocated for serialized bucket data */
+ int bucketsize = BUCKET_SIZE(ndims);
+ char *bucket = palloc0(bucketsize);
+
+ /* values per dimension (and number of non-NULL values) */
+ Datum **values = (Datum**)palloc0(sizeof(Datum*) * ndims);
+ int *counts = (int*)palloc0(sizeof(int) * ndims);
+
+ /* info about dimensions (for deserialize) */
+ DimensionInfo * info
+ = (DimensionInfo *)palloc0(sizeof(DimensionInfo)*ndims);
+
+ /* sort support data */
+ SortSupport ssup = (SortSupport)palloc0(sizeof(SortSupportData)*ndims);
+
+ /* collect and deduplicate values for each dimension separately */
+ for (i = 0; i < ndims; i++)
+ {
+ int count;
+ StdAnalyzeData *tmp = (StdAnalyzeData *)stats[i]->extra_data;
+
+ /* keep important info about the data type */
+ info[i].typlen = stats[i]->attrtype->typlen;
+ info[i].typbyval = stats[i]->attrtype->typbyval;
+
+ /*
+ * Allocate space for all min/max values, including NULLs
+ * (we won't use them, but we don't know how many are there),
+ * and then collect all non-NULL values.
+ */
+ values[i] = (Datum*)palloc0(sizeof(Datum) * nbuckets * 2);
+
+ for (j = 0; j < histogram->nbuckets; j++)
+ {
+ /* skip buckets where this dimension is NULL-only */
+ if (! histogram->buckets[j]->nullsonly[i])
+ {
+ values[i][counts[i]] = histogram->buckets[j]->min[i];
+ counts[i] += 1;
+
+ values[i][counts[i]] = histogram->buckets[j]->max[i];
+ counts[i] += 1;
+ }
+ }
+
+ /* there are just NULL values in this dimension */
+ if (counts[i] == 0)
+ continue;
+
+ /* sort and deduplicate */
+ ssup[i].ssup_cxt = CurrentMemoryContext;
+ ssup[i].ssup_collation = DEFAULT_COLLATION_OID;
+ ssup[i].ssup_nulls_first = false;
+
+ PrepareSortSupportFromOrderingOp(tmp->ltopr, &ssup[i]);
+
+ qsort_arg(values[i], counts[i], sizeof(Datum),
+ compare_scalars_simple, &ssup[i]);
+
+ /*
+ * Walk through the array and eliminate duplicitate values, but
+ * keep the ordering (so that we can do bsearch later). We know
+ * there's at least 1 item, so we can skip the first element.
+ */
+ count = 1; /* number of deduplicated items */
+ for (j = 1; j < counts[i]; j++)
+ {
+ /* if it's different from the previous value, we need to keep it */
+ if (compare_datums_simple(values[i][j-1], values[i][j], &ssup[i]) != 0)
+ {
+ /* XXX: not needed if (count == j) */
+ values[i][count] = values[i][j];
+ count += 1;
+ }
+ }
+
+ /* keep info about the deduplicated count */
+ info[i].nvalues = count;
+
+ /* compute size of the serialized data */
+ if (info[i].typbyval)
+ /*
+ * passed by value, so just Datum array (int4, int8, ...)
+ *
+ * TODO Might save a few bytes here, by storing just typlen
+ * bytes instead of whole Datum (8B) on 64-bits.
+ */
+ info[i].nbytes = info[i].nvalues * sizeof(Datum);
+ else if (info[i].typlen > 0)
+ /* pased by reference, but fixed length (name, tid, ...) */
+ info[i].nbytes = info[i].nvalues * info[i].typlen;
+ else if (info[i].typlen == -1)
+ /* varlena, so just use VARSIZE_ANY */
+ for (j = 0; j < info[i].nvalues; j++)
+ info[i].nbytes += VARSIZE_ANY(values[i][j]);
+ else if (info[i].typlen == -2)
+ /* cstring, so simply strlen */
+ for (j = 0; j < info[i].nvalues; j++)
+ info[i].nbytes += strlen(DatumGetPointer(values[i][j]));
+ else
+ elog(ERROR, "unknown data type typbyval=%d typlen=%d",
+ info[i].typbyval, info[i].typlen);
+ }
+
+ /*
+ * Now we finally know how much space we'll need for the serialized
+ * histogram, as it contains these fields:
+ *
+ * - length (4B) for varlena
+ * - magic (4B)
+ * - type (4B)
+ * - ndimensions (4B)
+ * - nbuckets (4B)
+ * - info (ndim * sizeof(DimensionInfo)
+ * - arrays of values for each dimension
+ * - serialized buckets (nbuckets * bucketsize)
+ *
+ * So the 'header' size is 20B + ndim * sizeof(DimensionInfo) and
+ * then we'll place the data (and buckets).
+ */
+ total_length = (sizeof(int32) + offsetof(MVHistogramData, buckets)
+ + ndims * sizeof(DimensionInfo)
+ + nbuckets * bucketsize);
+
+ /* account for the deduplicated data */
+ for (i = 0; i < ndims; i++)
+ total_length += info[i].nbytes;
+
+ /* enforce arbitrary limit of 1MB */
+ if (total_length > 1024 * 1024)
+ elog(ERROR, "serialized histogram exceeds 1MB (%ld)", total_length);
+
+ /* allocate space for the serialized histogram list, set header */
+ output = (bytea*)palloc0(total_length);
+ SET_VARSIZE(output, total_length);
+
+ /* we'll use 'data' to keep track of the place to write data */
+ data = VARDATA(output);
+
+ memcpy(data, histogram, offsetof(MVHistogramData, buckets));
+ data += offsetof(MVHistogramData, buckets);
+
+ memcpy(data, info, sizeof(DimensionInfo) * ndims);
+ data += sizeof(DimensionInfo) * ndims;
+
+ /* value array for each dimension */
+ for (i = 0; i < ndims; i++)
+ {
+#ifdef USE_ASSERT_CHECKING
+ char *tmp = data;
+#endif
+ for (j = 0; j < info[i].nvalues; j++)
+ {
+ if (info[i].typbyval)
+ {
+ /* passed by value / Datum */
+ memcpy(data, &values[i][j], sizeof(Datum));
+ data += sizeof(Datum);
+ }
+ else if (info[i].typlen > 0)
+ {
+ /* pased by reference, but fixed length (name, tid, ...) */
+ memcpy(data, &values[i][j], info[i].typlen);
+ data += info[i].typlen;
+ }
+ else if (info[i].typlen == -1)
+ {
+ /* varlena */
+ memcpy(data, DatumGetPointer(values[i][j]),
+ VARSIZE_ANY(values[i][j]));
+ data += VARSIZE_ANY(values[i][j]);
+ }
+ else if (info[i].typlen == -2)
+ {
+ /* cstring (don't forget the \0 terminator!) */
+ memcpy(data, DatumGetPointer(values[i][j]),
+ strlen(DatumGetPointer(values[i][j])) + 1);
+ data += strlen(DatumGetPointer(values[i][j])) + 1;
+ }
+ }
+ Assert((data - tmp) == info[i].nbytes);
+ }
+
+ /* and finally, the histogram buckets */
+ for (i = 0; i < nbuckets; i++)
+ {
+ /* don't write beyond the allocated space */
+ Assert(data <= (char*)output + total_length - bucketsize);
+
+ /* reset the values for each item */
+ memset(bucket, 0, bucketsize);
+
+ *BUCKET_NTUPLES(bucket) = histogram->buckets[i]->ntuples;
+ *BUCKET_NDISTINCT(bucket) = histogram->buckets[i]->ndistinct;
+
+ for (j = 0; j < ndims; j++)
+ {
+ /* do the lookup only for non-NULL values */
+ if (! histogram->buckets[i]->nullsonly[j])
+ {
+ int idx;
+ Datum * v = NULL;
+ ssup_private = &ssup[j];
+
+ /* min boundary */
+ v = (Datum*)bsearch(&histogram->buckets[i]->min[j],
+ values[j], info[j].nvalues, sizeof(Datum),
+ bsearch_comparator);
+
+ if (v == NULL)
+ elog(ERROR, "value for dim %d not found in array", j);
+
+ /* compute index within the array */
+ idx = (v - values[j]);
+
+ Assert((idx >= 0) && (idx < info[j].nvalues));
+
+ BUCKET_MIN_INDEXES(bucket, ndims)[j] = idx;
+
+ /* max boundary */
+ v = (Datum*)bsearch(&histogram->buckets[i]->max[j],
+ values[j], info[j].nvalues, sizeof(Datum),
+ bsearch_comparator);
+
+ if (v == NULL)
+ elog(ERROR, "value for dim %d not found in array", j);
+
+ /* compute index within the array */
+ idx = (v - values[j]);
+
+ Assert((idx >= 0) && (idx < info[j].nvalues));
+
+ BUCKET_MAX_INDEXES(bucket, ndims)[j] = idx;
+ }
+ }
+
+ /* copy flags (nulls, min/max inclusive) */
+ memcpy(BUCKET_NULLS_ONLY(bucket, ndims),
+ histogram->buckets[i]->nullsonly, sizeof(bool) * ndims);
+
+ memcpy(BUCKET_MIN_INCL(bucket, ndims),
+ histogram->buckets[i]->min_inclusive, sizeof(bool) * ndims);
+
+ memcpy(BUCKET_MAX_INCL(bucket, ndims),
+ histogram->buckets[i]->max_inclusive, sizeof(bool) * ndims);
+
+ /* copy the item into the array */
+ memcpy(data, bucket, bucketsize);
+
+ data += bucketsize;
+ }
+
+ /* at this point we expect to match the total_length exactly */
+ Assert((data - (char*)output) == total_length);
+
+ /* FIXME free the values/counts arrays here */
+
+ return output;
+}
+
+/*
+ * Reverse to serialize histogram. This essentially expands the serialized
+ * form back to MVHistogram / MVBucket.
+ */
+MVHistogram
+deserialize_mv_histogram(bytea * data)
+{
+ int i = 0, j = 0;
+
+ Size expected_size;
+ char *tmp = NULL;
+ Datum **values = NULL;
+
+ MVHistogram histogram;
+ DimensionInfo *info;
+
+ int nbuckets;
+ int ndims;
+ int bucketsize;
+
+ if (data == NULL)
+ return NULL;
+
+ if (VARSIZE_ANY_EXHDR(data) < offsetof(MVHistogramData,buckets))
+ elog(ERROR, "invalid histogram size %ld (expected at least %ld)",
+ VARSIZE_ANY_EXHDR(data), offsetof(MVHistogramData,buckets));
+
+ /* read the histogram header */
+ histogram = (MVHistogram)palloc0(sizeof(MVHistogramData));
+
+ /* initialize pointer to the data part (skip the varlena header) */
+ tmp = VARDATA(data);
+
+ /* get the header and perform basic sanity checks */
+ memcpy(histogram, tmp, offsetof(MVHistogramData, buckets));
+ tmp += offsetof(MVHistogramData, buckets);
+
+ if (histogram->magic != MVSTAT_HIST_MAGIC)
+ elog(ERROR, "invalid histogram magic %d (expected %dd)",
+ histogram->magic, MVSTAT_HIST_MAGIC);
+
+ if (histogram->type != MVSTAT_HIST_TYPE_BASIC)
+ elog(ERROR, "invalid histogram type %d (expected %dd)",
+ histogram->type, MVSTAT_HIST_TYPE_BASIC);
+
+ nbuckets = histogram->nbuckets;
+ ndims = histogram->ndimensions;
+ bucketsize = BUCKET_SIZE(ndims);
+
+ Assert((nbuckets > 0) && (nbuckets <= MVSTAT_HIST_MAX_BUCKETS));
+ Assert((ndims >= 2) && (ndims <= MVSTATS_MAX_DIMENSIONS));
+
+ /*
+ * What size do we expect with those parameters (it's incomplete,
+ * as we yet have to count the array sizes (from DimensionInfo
+ * records).
+ */
+ expected_size = offsetof(MVHistogramData,buckets) +
+ ndims * sizeof(DimensionInfo) +
+ (nbuckets * bucketsize);
+
+ /* check that we have at least the DimensionInfo records */
+ if (VARSIZE_ANY_EXHDR(data) < expected_size)
+ elog(ERROR, "invalid histogram size %ld (expected %ld)",
+ VARSIZE_ANY_EXHDR(data), expected_size);
+
+ info = (DimensionInfo*)(tmp);
+ tmp += ndims * sizeof(DimensionInfo);
+
+ /* account for the value arrays */
+ for (i = 0; i < ndims; i++)
+ expected_size += info[i].nbytes;
+
+ if (VARSIZE_ANY_EXHDR(data) != expected_size)
+ elog(ERROR, "invalid histogram size %ld (expected %ld)",
+ VARSIZE_ANY_EXHDR(data), expected_size);
+
+ /* looks OK - not corrupted or something */
+
+ /* let's parse the value arrays */
+ values = (Datum**)palloc0(sizeof(Datum*) * ndims);
+
+ /*
+ * FIXME This uses pointers to the original data array (the types
+ * not passed by value), so when someone frees the memory,
+ * e.g. by doing something like this:
+ *
+ * bytea * data = ... fetch the data from catalog ...
+ * MCVList mcvlist = deserialize_mcv_list(data);
+ * pfree(data);
+ *
+ * then 'mcvlist' references the freed memory. This needs to
+ * copy the pieces.
+ *
+ * TODO same as in MCV deserialization / consider moving to common.c
+ */
+ for (i = 0; i < ndims; i++)
+ {
+ if (info[i].typbyval)
+ {
+ /* passed by value / Datum - simply reuse the array */
+ values[i] = (Datum*)tmp;
+ tmp += info[i].nbytes;
+ }
+ else if (info[i].typlen > 0)
+ {
+ /* pased by reference, but fixed length (name, tid, ...) */
+ values[i] = (Datum*)palloc0(sizeof(Datum) * info[i].nvalues);
+ for (j = 0; j < info[i].nvalues; j++)
+ {
+ /* just point into the array */
+ values[i][j] = PointerGetDatum(tmp);
+ tmp += info[i].typlen;
+ }
+ }
+ else if (info[i].typlen == -1)
+ {
+ /* varlena */
+ values[i] = (Datum*)palloc0(sizeof(Datum) * info[i].nvalues);
+ for (j = 0; j < info[i].nvalues; j++)
+ {
+ /* just point into the array */
+ values[i][j] = PointerGetDatum(tmp);
+ tmp += VARSIZE_ANY(tmp);
+ }
+ }
+ else if (info[i].typlen == -2)
+ {
+ /* cstring */
+ values[i] = (Datum*)palloc0(sizeof(Datum) * info[i].nvalues);
+ for (j = 0; j < info[i].nvalues; j++)
+ {
+ /* just point into the array */
+ values[i][j] = PointerGetDatum(tmp);
+ tmp += (strlen(tmp) + 1); /* don't forget the \0 */
+ }
+ }
+ }
+
+ /* allocate space for the buckets */
+ histogram->buckets = (MVBucket*)palloc0(sizeof(MVBucket) * nbuckets);
+
+ for (i = 0; i < nbuckets; i++)
+ {
+ MVBucket bucket = (MVBucket)palloc0(sizeof(MVBucketData));
+
+ bucket->nullsonly = (bool*) palloc0(sizeof(bool) * ndims);
+ bucket->min_inclusive = (bool*) palloc0(sizeof(bool) * ndims);
+ bucket->max_inclusive = (bool*) palloc0(sizeof(bool) * ndims);
+
+ bucket->min = (Datum*) palloc0(sizeof(Datum) * ndims);
+ bucket->max = (Datum*) palloc0(sizeof(Datum) * ndims);
+
+ bucket->ntuples = *BUCKET_NTUPLES(tmp);
+ bucket->ndistinct = *BUCKET_NDISTINCT(tmp);
+
+ memcpy(bucket->nullsonly, BUCKET_NULLS_ONLY(tmp, ndims),
+ sizeof(bool) * ndims);
+
+ memcpy(bucket->min_inclusive, BUCKET_MIN_INCL(tmp, ndims),
+ sizeof(bool) * ndims);
+
+ memcpy(bucket->max_inclusive, BUCKET_MAX_INCL(tmp, ndims),
+ sizeof(bool) * ndims);
+
+ /* translate the indexes to values */
+ for (j = 0; j < ndims; j++)
+ {
+ if (! bucket->nullsonly[j])
+ {
+ bucket->min[j] = values[j][BUCKET_MIN_INDEXES(tmp, ndims)[j]];
+ bucket->max[j] = values[j][BUCKET_MAX_INDEXES(tmp, ndims)[j]];
+ }
+ }
+
+ histogram->buckets[i] = bucket;
+
+ Assert(tmp <= (char*)data + VARSIZE_ANY(data));
+
+ tmp += bucketsize;
+ }
+
+ /* at this point we expect to match the total_length exactly */
+ Assert((tmp - VARDATA(data)) == expected_size);
+
+ return histogram;
+}
+
+/*
+ * Build the initial bucket, which will be then split into smaller
+ * buckets.
+ *
+ * TODO Add ndistinct estimation, probably the one described in "Towards
+ * Estimation Error Guarantees for Distinct Values, PODS 2000,
+ * p. 268-279" (the ones called GEE, or maybe AE).
+ *
+ * TODO The "combined" ndistinct is more likely to scale with the number
+ * of rows (in the table), because a single column behaving this
+ * way is sufficient for such behavior.
+ */
+static MVBucket
+create_initial_mv_bucket(int numrows, HeapTuple *rows, int2vector *attrs,
+ VacAttrStats **stats)
+{
+ int i;
+ int numattrs = attrs->dim1;
+
+ /* TODO allocate bucket as a single piece, including all the fields. */
+ MVBucket bucket = (MVBucket)palloc0(sizeof(MVBucketData));
+
+ Assert(numrows > 0);
+ Assert(rows != NULL);
+ Assert((numattrs >= 2) && (numattrs <= MVSTATS_MAX_DIMENSIONS));
+
+ /* allocate the per-dimension arrays */
+ bucket->ndistincts = (uint32*)palloc0(numattrs * sizeof(uint32));
+ bucket->nullsonly = (bool*)palloc0(numattrs * sizeof(bool));
+
+ /* inclusiveness boundaries - lower/upper bounds */
+ bucket->min_inclusive = (bool*)palloc0(numattrs * sizeof(bool));
+ bucket->max_inclusive = (bool*)palloc0(numattrs * sizeof(bool));
+
+ /* lower/upper boundaries */
+ bucket->min = (Datum*)palloc0(numattrs * sizeof(Datum));
+ bucket->max = (Datum*)palloc0(numattrs * sizeof(Datum));
+
+ /* all the sample rows fall into the initial bucket */
+ bucket->numrows = numrows;
+ bucket->ntuples = numrows;
+ bucket->rows = rows;
+
+ /*
+ * Update the number of ndistinct combinations in the bucket (which
+ * we use when selecting bucket to partition), and then number of
+ * distinct values for each partition (which we use when choosing
+ * which dimension to split).
+ */
+ update_bucket_ndistinct(bucket, attrs, stats);
+
+ /* Update ndistinct (and also set min/max) for all dimensions. */
+ for (i = 0; i < numattrs; i++)
+ update_dimension_ndistinct(bucket, i, attrs, stats, true);
+
+ /*
+ * The initial bucket was not split at all, so we'll start with the
+ * first dimension in the next round (index = 0).
+ */
+ bucket->last_split_dimension = -1;
+
+ return bucket;
+}
+
+/*
+ * TODO Fix to handle arbitrarily-sized histograms (not just 2D ones)
+ * and call the right output procedures (for the particular type).
+ *
+ * TODO This should somehow fetch info about the data types, and use
+ * the appropriate output functions to print the boundary values.
+ * Right now this prints the 8B value as an integer.
+ *
+ * TODO Also, provide a special function for 2D histogram, printing
+ * a gnuplot script (with rectangles).
+ *
+ * TODO For string types (once supported) we can sort the strings first,
+ * assign them a sequence of integers and use the original values
+ * as labels.
+ */
+#ifdef MVSTATS_DEBUG
+static void
+print_mv_histogram_info(MVHistogram histogram)
+{
+ int i = 0;
+
+ elog(WARNING, "histogram nbuckets=%d", histogram->nbuckets);
+
+ for (i = 0; i < histogram->nbuckets; i++)
+ {
+ MVBucket bucket = histogram->buckets[i];
+ elog(WARNING, " bucket %d : ndistinct=%f ntuples=%d min=[%ld, %ld], max=[%ld, %ld] distinct=[%d,%d]",
+ i, bucket->ndistinct, bucket->numrows,
+ bucket->min[0], bucket->min[1], bucket->max[0], bucket->max[1],
+ bucket->ndistincts[0], bucket->ndistincts[1]);
+ }
+}
+#endif
+
+/*
+ * A very simple partitioning selection criteria - choose the bucket
+ * with the highest number of distinct values.
+ *
+ * Returns either pointer to the bucket selected to be partitioned,
+ * or NULL if there are no buckets that may be split (i.e. all buckets
+ * contain a single distinct value).
+ *
+ * TODO Consider other partitioning criteria (v-optimal, maxdiff etc.).
+ *
+ * TODO Allowing the bucket to degenerate to a single combination of
+ * values makes it rather strange MCV list. Maybe we should use
+ * higher lower boundary, or maybe make the selection criteria
+ * more complex (e.g. consider number of rows in the bucket, etc.).
+ *
+ * That however is different from buckets 'degenerated' only for
+ * some dimensions (e.g. half of them), which is perfectly
+ * appropriate for statistics on a combination of low and high
+ * cardinality columns.
+ */
+static MVBucket
+select_bucket_to_partition(int nbuckets, MVBucket * buckets)
+{
+ int i;
+ int ndistinct = 1; /* if ndistinct=1, we can't split the bucket */
+ MVBucket bucket = NULL;
+
+ for (i = 0; i < nbuckets; i++)
+ {
+ /* if the ndistinct count is higher, use this bucket */
+ if (buckets[i]->ndistinct > ndistinct) {
+ bucket = buckets[i];
+ ndistinct = buckets[i]->ndistinct;
+ }
+ }
+
+ /* may be NULL if there are not buckets with (ndistinct>1) */
+ return bucket;
+}
+
+/*
+ * A simple bucket partitioning implementation - splits the dimensions in
+ * a round-robin manner (considering only those with ndistinct>1). That
+ * is first a dimension 0 is split, then 1, 2, ... until reaching the
+ * end of attribute list, and then wrapping back to 0. Of course,
+ * dimensions with a single distinct value are skipped.
+ *
+ * This is essentially what Muralikrishna/DeWitt described in their SIGMOD
+ * article (M. Muralikrishna, David J. DeWitt: Equi-Depth Histograms For
+ * Estimating Selectivity Factors For Multi-Dimensional Queries. SIGMOD
+ * Conference 1988: 28-36).
+ *
+ * There are multiple histogram options, centered around the partitioning
+ * criteria, specifying both how to choose a bucket and the dimension
+ * most in need of a split. For a nice summary and general overview, see
+ * "rK-Hist : an R-Tree based histogram for multi-dimensional selectivity
+ * estimation" thesis by J. A. Lopez, Concordia University, p.34-37 (and
+ * possibly p. 32-34 for explanation of the terms).
+ *
+ * This splits the bucket by tweaking the existing one, and returning the
+ * new bucket (essentially shrinking the existing one in-place and returning
+ * the other "half" as a new bucket). The caller is responsible for adding
+ * the new bucket into the list of buckets.
+ *
+ * TODO It requires care to prevent splitting only one dimension and not
+ * splitting another one at all (which might happen easily in case of
+ * strongly dependent columns - e.g. y=x).
+ *
+ * TODO Should probably consider statistics target for the columns (e.g. to
+ * split dimensions with higher statistics target more frequently).
+ */
+static MVBucket
+partition_bucket(MVBucket bucket, int2vector *attrs,
+ VacAttrStats **stats)
+{
+ int i;
+ int dimension;
+ int numattrs = attrs->dim1;
+
+ Datum split_value;
+ MVBucket new_bucket;
+
+ /* needed for sort, when looking for the split value */
+ bool isNull;
+ int nvalues = 0;
+ StdAnalyzeData * mystats = NULL;
+ ScalarItem * values = (ScalarItem*)palloc0(bucket->numrows * sizeof(ScalarItem));
+ SortSupportData ssup;
+
+ /* looking for the split value */
+ int ndistinct = 1; /* number of distinct values below current value */
+ int nrows = 1; /* number of rows below current value */
+
+ /* needed when splitting the values */
+ HeapTuple * oldrows = bucket->rows;
+ int oldnrows = bucket->numrows;
+
+ /*
+ * We can't split buckets with a single distinct value (this also
+ * disqualifies NULL-only dimensions). Also, there has to be multiple
+ * sample rows (otherwise, how could there be more distinct values).
+ */
+ Assert(bucket->ndistinct > 1);
+ Assert(bucket->numrows > 1);
+ Assert((numattrs >= 2) && (numattrs <= MVSTATS_MAX_DIMENSIONS));
+
+ /*
+ * Look for the next dimension to split, in a round robin manner.
+ * We'll use the first one with (ndistinct > 1).
+ *
+ * If we happen to wrap around, something clearly went wrong (we
+ * can't mess with the last_split_dimension directly, because we
+ * couldn't do this check).
+ */
+ dimension = bucket->last_split_dimension;
+ while (true)
+ {
+ dimension = (dimension + 1) % numattrs;
+
+ if (bucket->ndistincts[dimension] > 1)
+ break;
+
+ /* if we ran the previous split dimension, it's infinite loop */
+ Assert(dimension != bucket->last_split_dimension);
+ }
+
+ /* Remember the dimension for the next split of this bucket. */
+ bucket->last_split_dimension = dimension;
+
+ /*
+ * Walk through the selected dimension, collect and sort the values
+ * and then choose the value to use as the new boundary.
+ */
+ mystats = (StdAnalyzeData *) stats[dimension]->extra_data;
+
+ /* initialize sort support, etc. */
+ memset(&ssup, 0, sizeof(ssup));
+ ssup.ssup_cxt = CurrentMemoryContext;
+
+ /* We always use the default collation for statistics */
+ ssup.ssup_collation = DEFAULT_COLLATION_OID;
+ ssup.ssup_nulls_first = false;
+
+ PrepareSortSupportFromOrderingOp(mystats->ltopr, &ssup);
+
+ for (i = 0; i < bucket->numrows; i++)
+ {
+ /* remember the index of the sample row, to make the partitioning simpler */
+ values[nvalues].value = heap_getattr(bucket->rows[i], attrs->values[dimension],
+ stats[dimension]->tupDesc, &isNull);
+ values[nvalues].tupno = i;
+
+ /* no NULL values allowed here (we don't do splits by null-only dimensions) */
+ Assert(!isNull);
+
+ nvalues++;
+ }
+
+ /* sort the array of values */
+ qsort_arg((void *) values, nvalues, sizeof(ScalarItem),
+ compare_scalars_partition, (void *) &ssup);
+
+ /*
+ * We know there are bucket->ndistincts[dimension] distinct values
+ * in this dimension, and we want to split this into half, so walk
+ * through the array and stop once we see (ndistinct/2) values.
+ *
+ * We always choose the "next" value, i.e. (n/2+1)-th distinct value,
+ * and use it as an exclusive upper boundary (and inclusive lower
+ * boundary).
+ *
+ * TODO Maybe we should use "average" of the two middle distinct
+ * values (at least for even distinct counts), but that would
+ * require being able to do an average (which does not work
+ * for non-arithmetic types).
+ *
+ * TODO Another option is to look for a split that'd give about
+ * 50% tuples (not distinct values) in each partition. That
+ * might work better when there are a few very frequent
+ * values, and many rare ones.
+ */
+ split_value = values[0].value;
+ for (i = 1; i < bucket->numrows; i++)
+ {
+ /* count distinct values */
+ if (values[i].value != values[i-1].value)
+ ndistinct += 1;
+
+ /* once we've seen 1/2 distinct values (and use the value) */
+ if (ndistinct > bucket->ndistincts[dimension] / 2)
+ {
+ split_value = values[i].value;
+ break;
+ }
+
+ /* keep track how many rows belong to the first bucket */
+ nrows += 1;
+ }
+
+ Assert(nrows > 0);
+ Assert(nrows < bucket->numrows);
+
+ /* create the new bucket as a (incomplete) copy of the one being partitioned. */
+ new_bucket = copy_mv_bucket(bucket, numattrs);
+
+ /*
+ * Do the actual split of the chosen dimension, using the split value as the
+ * upper bound for the existing bucket, and lower bound for the new one.
+ */
+ bucket->max[dimension] = split_value;
+ new_bucket->min[dimension] = split_value;
+
+ bucket->max_inclusive[dimension] = false;
+ new_bucket->max_inclusive[dimension] = true;
+
+ /*
+ * Redistribute the sample tuples using the 'ScalarItem->tupno'
+ * index. We know 'nrows' rows should remain in the original
+ * bucket and the rest goes to the new one.
+ */
+
+ bucket->rows = (HeapTuple*)palloc0(nrows * sizeof(HeapTuple));
+ new_bucket->rows = (HeapTuple*)palloc0((oldnrows - nrows) * sizeof(HeapTuple));
+
+ bucket->numrows = nrows;
+ new_bucket->numrows = (oldnrows - nrows);
+
+ /*
+ * The first nrows should go to the first bucket, the rest should
+ * go to the new one. Use the tupno field to get the actual HeapTuple
+ * row from the original array of sample rows.
+ */
+ for (i = 0; i < nrows; i++)
+ memcpy(&bucket->rows[i], &oldrows[values[i].tupno], sizeof(HeapTuple));
+
+ for (i = nrows; i < oldnrows; i++)
+ memcpy(&new_bucket->rows[i-nrows], &oldrows[values[i].tupno], sizeof(HeapTuple));
+
+ /* update ndistinct values for the buckets (total and per dimension) */
+ update_bucket_ndistinct(bucket, attrs, stats);
+ update_bucket_ndistinct(new_bucket, attrs, stats);
+
+ /*
+ * TODO We don't need to do this for the dimension we used for split,
+ * because we know how many distinct values went to each partition.
+ */
+ for (i = 0; i < numattrs; i++)
+ {
+ update_dimension_ndistinct(bucket, i, attrs, stats, false);
+ update_dimension_ndistinct(new_bucket, i, attrs, stats, false);
+ }
+
+ pfree(oldrows);
+ pfree(values);
+
+ return new_bucket;
+}
+
+/*
+ * Copy a histogram bucket. The copy does not include the build-time
+ * data, i.e. sampled rows etc.
+ */
+static MVBucket
+copy_mv_bucket(MVBucket bucket, uint32 ndimensions)
+{
+ /* TODO allocate as a single piece (including all the fields) */
+ MVBucket new_bucket = (MVBucket)palloc0(sizeof(MVBucketData));
+
+ /* Copy only the attributes that will stay the same after the split, and
+ * we'll recompute the rest after the split. */
+
+ new_bucket->last_split_dimension = bucket->last_split_dimension;
+
+ /* allocate the per-dimension arrays */
+ new_bucket->ndistincts = (uint32*)palloc0(ndimensions * sizeof(uint32));
+ new_bucket->nullsonly = (bool*)palloc0(ndimensions * sizeof(bool));
+
+ /* inclusiveness boundaries - lower/upper bounds */
+ new_bucket->min_inclusive = (bool*)palloc0(ndimensions * sizeof(bool));
+ new_bucket->max_inclusive = (bool*)palloc0(ndimensions * sizeof(bool));
+
+ /* lower/upper boundaries */
+ new_bucket->min = (Datum*)palloc0(ndimensions * sizeof(Datum));
+ new_bucket->max = (Datum*)palloc0(ndimensions * sizeof(Datum));
+
+ /* copy data */
+ memcpy(new_bucket->nullsonly, bucket->nullsonly, ndimensions * sizeof(bool));
+
+ memcpy(new_bucket->min_inclusive, bucket->min_inclusive, ndimensions*sizeof(bool));
+ memcpy(new_bucket->min, bucket->min, ndimensions*sizeof(Datum));
+
+ memcpy(new_bucket->max_inclusive, bucket->max_inclusive, ndimensions*sizeof(bool));
+ memcpy(new_bucket->max, bucket->max, ndimensions*sizeof(Datum));
+
+ return new_bucket;
+}
+
+/*
+ * Counts the number of distinct values in the bucket. This just copies
+ * the Datum values into a simple array, and sorts them using memcmp-based
+ * comparator. That means it only works for pass-by-value data types
+ * (assuming they don't use collations etc.)
+ *
+ * TODO This might evaluate and store the distinct counts for all
+ * possible attribute combinations. The assumption is this might be
+ * useful for estimating things like GROUP BY cardinalities (e.g.
+ * in cases when some buckets contain a lot of low-frequency
+ * combinations, and other buckets contain few high-frequency ones).
+ *
+ * But it's unclear whether it's worth the price. Computing this
+ * is actually quite cheap, because it may be evaluated at the very
+ * end, when the buckets are rather small (so sorting it in 2^N ways
+ * is not a big deal). Assuming the partitioning algorithm does not
+ * use these values to do the decisions, of course (the current
+ * algorithm does not).
+ *
+ * The overhead with storing, fetching and parsing the data is more
+ * concerning - adding 2^N values per bucket (even if it's just
+ * a 1B or 2B value) would significantly bloat the histogram, and
+ * thus the impact on optimizer. Which is not really desirable.
+ *
+ * TODO This only updates the ndistinct for the sample (or bucket), but
+ * we eventually need an estimate of the total number of distinct
+ * values in the dataset. It's possible to either use the current
+ * 1D approach (i.e., if it's more than 10% of the sample, assume
+ * it's proportional to the number of rows). Or it's possible to
+ * implement the estimator suggested in the article, supposedly
+ * giving 'optimal' estimates (w.r.t. probability of error).
+ */
+static void
+update_bucket_ndistinct(MVBucket bucket, int2vector *attrs, VacAttrStats ** stats)
+{
+ int i, j;
+ int numrows = bucket->numrows;
+ int numattrs = attrs->dim1;
+
+ MultiSortSupport mss = multi_sort_init(numattrs);
+
+ /*
+ * We could collect this while walking through all the attributes
+ * above (this way we have to call heap_getattr twice).
+ */
+ SortItem *items = (SortItem*)palloc0(numrows * sizeof(SortItem));
+ Datum *values = (Datum*)palloc0(numrows * sizeof(Datum) * numattrs);
+ bool *isnull = (bool*)palloc0(numrows * sizeof(bool) * numattrs);
+
+ for (i = 0; i < numrows; i++)
+ {
+ items[i].values = &values[i * numattrs];
+ items[i].isnull = &isnull[i * numattrs];
+ }
+
+ /* prepare the sort function for the first dimension */
+ for (i = 0; i < numattrs; i++)
+ multi_sort_add_dimension(mss, i, i, stats);
+
+ /* collect the values */
+ for (i = 0; i < numrows; i++)
+ for (j = 0; j < numattrs; j++)
+ items[i].values[j]
+ = heap_getattr(bucket->rows[i], attrs->values[j],
+ stats[j]->tupDesc, &items[i].isnull[j]);
+
+ qsort_arg((void *) items, numrows, sizeof(SortItem),
+ multi_sort_compare, mss);
+
+ bucket->ndistinct = 1;
+
+ for (i = 1; i < numrows; i++)
+ if (multi_sort_compare(&items[i], &items[i-1], mss) != 0)
+ bucket->ndistinct += 1;
+
+ pfree(items);
+ pfree(values);
+ pfree(isnull);
+}
+
+/*
+ * Count distinct values per bucket dimension.
+ */
+static void
+update_dimension_ndistinct(MVBucket bucket, int dimension, int2vector *attrs,
+ VacAttrStats ** stats, bool update_boundaries)
+{
+ int j;
+ int nvalues = 0;
+ bool isNull;
+ Datum * values = (Datum*)palloc0(bucket->numrows * sizeof(Datum));
+ SortSupportData ssup;
+
+ StdAnalyzeData * mystats = (StdAnalyzeData *) stats[dimension]->extra_data;
+
+ /* we may already know this is a NULL-only dimension */
+ if (bucket->nullsonly[dimension])
+ bucket->ndistincts[dimension] = 1;
+
+ memset(&ssup, 0, sizeof(ssup));
+ ssup.ssup_cxt = CurrentMemoryContext;
+
+ /* We always use the default collation for statistics */
+ ssup.ssup_collation = DEFAULT_COLLATION_OID;
+ ssup.ssup_nulls_first = false;
+
+ PrepareSortSupportFromOrderingOp(mystats->ltopr, &ssup);
+
+ for (j = 0; j < bucket->numrows; j++)
+ {
+ values[nvalues] = heap_getattr(bucket->rows[j], attrs->values[dimension],
+ stats[dimension]->tupDesc, &isNull);
+
+ /* ignore NULL values */
+ if (! isNull)
+ nvalues++;
+ }
+
+ /* there's always at least 1 distinct value (may be NULL) */
+ bucket->ndistincts[dimension] = 1;
+
+ /* if there are only NULL values in the column, mark it so and continue
+ * with the next one */
+ if (nvalues == 0)
+ {
+ pfree(values);
+ bucket->nullsonly[dimension] = true;
+ return;
+ }
+
+ /* sort the array (pass-by-value datum */
+ qsort_arg((void *) values, nvalues, sizeof(Datum),
+ compare_scalars_simple, (void *) &ssup);
+
+ /*
+ * Update min/max boundaries to the smallest bounding box. Generally, this
+ * needs to be done only when constructing the initial bucket.
+ */
+ if (update_boundaries)
+ {
+ /* store the min/max values */
+ bucket->min[dimension] = values[0];
+ bucket->min_inclusive[dimension] = true;
+
+ bucket->max[dimension] = values[nvalues-1];
+ bucket->max_inclusive[dimension] = true;
+ }
+
+ /*
+ * Walk through the array and count distinct values by comparing
+ * succeeding values.
+ *
+ * FIXME This only works for pass-by-value types (i.e. not VARCHARs
+ * etc.). Although thanks to the deduplication it might work
+ * even for those types (equal values will get the same item
+ * in the deduplicated array).
+ */
+ for (j = 1; j < nvalues; j++) {
+ if (values[j] != values[j-1])
+ bucket->ndistincts[dimension] += 1;
+ }
+
+ pfree(values);
+}
+
+/*
+ * A properly built histogram must not contain buckets mixing NULL and
+ * non-NULL values in a single dimension. Each dimension may either be
+ * marked as 'nulls only', and thus containing only NULL values, or
+ * it must not contain any NULL values.
+ *
+ * Therefore, if the sample contains NULL values in any of the columns,
+ * it's necessary to build those NULL-buckets. This is done in an
+ * iterative way using this algorithm, operating on a single bucket:
+ *
+ * (1) Check that all dimensions are well-formed (not mixing NULL
+ * and non-NULL values).
+ *
+ * (2) If all dimensions are well-formed, terminate.
+ *
+ * (3) If the dimension contains only NULL values, but is not
+ * marked as NULL-only, mark it as NULL-only and run the
+ * algorithm again (on this bucket).
+ *
+ * (4) If the dimension mixes NULL and non-NULL values, split the
+ * bucket into two parts - one with NULL values, one with
+ * non-NULL values (replacing the current one). Then run
+ * the algorithm on both buckets.
+ *
+ * This is executed in a recursive manner, but the number of executions
+ * should be quite low - limited by the number of NULL-buckets. Also,
+ * in each branch the number of nested calls is limited by the number
+ * of dimensions (attributes) of the histogram.
+ *
+ * At the end, there should be buckets with no mixed dimensions. The
+ * number of buckets produced by this algorithm is rather limited - with
+ * N dimensions, there may be only 2^N such buckets (each dimension may
+ * be either NULL or non-NULL). So with 8 dimensions (current value of
+ * MVSTATS_MAX_DIMENSIONS) there may be only 256 such buckets.
+ *
+ * After this, a 'regular' bucket-split algorithm shall run, further
+ * optimizing the histogram.
+ */
+static void
+create_null_buckets(MVHistogram histogram, int bucket_idx,
+ int2vector *attrs, VacAttrStats ** stats)
+{
+ int i, j;
+ int null_dim = -1;
+ int null_count = 0;
+ bool null_found = false;
+ MVBucket bucket, null_bucket;
+ int null_idx, curr_idx;
+
+ /* remember original values from the bucket */
+ int numrows;
+ HeapTuple *oldrows = NULL;
+
+ Assert(bucket_idx < histogram->nbuckets);
+ Assert(histogram->ndimensions == attrs->dim1);
+
+ bucket = histogram->buckets[bucket_idx];
+
+ numrows = bucket->numrows;
+ oldrows = bucket->rows;
+
+ /*
+ * Walk through all rows / dimensions, and stop once we find NULL
+ * in a dimension not yet marked as NULL-only.
+ */
+ for (i = 0; i < bucket->numrows; i++)
+ {
+ for (j = 0; j < histogram->ndimensions; j++)
+ {
+ /* Is this a NULL-only dimension? If yes, skip. */
+ if (bucket->nullsonly[j])
+ continue;
+
+ /* found a NULL in that dimension? */
+ if (heap_attisnull(bucket->rows[i], attrs->values[j]))
+ {
+ null_found = true;
+ null_dim = j;
+ break;
+ }
+ }
+
+ /* terminate if we found attribute with NULL values */
+ if (null_found)
+ break;
+ }
+
+ /* no regular dimension contains NULL values => we're done */
+ if (! null_found)
+ return;
+
+ /* walk through the rows again, count NULL values in 'null_dim' */
+ for (i = 0; i < bucket->numrows; i++)
+ {
+ if (heap_attisnull(bucket->rows[i], attrs->values[null_dim]))
+ null_count += 1;
+ }
+
+ Assert(null_count <= bucket->numrows);
+
+ /*
+ * If (null_count == numrows) the dimension already is NULL-only,
+ * but is not yet marked like that. It's enough to mark it and
+ * repeat the process recursively (until we run out of dimensions).
+ */
+ if (null_count == bucket->numrows)
+ {
+ bucket->nullsonly[null_dim] = true;
+ create_null_buckets(histogram, bucket_idx, attrs, stats);
+ return;
+ }
+
+ /*
+ * We have to split the bucket into two - one with NULL values in
+ * the dimension, one with non-NULL values. We don't need to sort
+ * the data or anything, but otherwise it's similar to what's done
+ * in partition_bucket().
+ */
+
+ /* create bucket with NULL-only dimension 'dim' */
+ null_bucket = copy_mv_bucket(bucket, histogram->ndimensions);
+
+ /* remember the current array info */
+ oldrows = bucket->rows;
+ numrows = bucket->numrows;
+
+ /* we'll keep non-NULL values in the current bucket */
+ bucket->numrows = (numrows - null_count);
+ bucket->rows
+ = (HeapTuple*)palloc0(bucket->numrows * sizeof(HeapTuple));
+
+ /* and the NULL values will go to the new one */
+ null_bucket->numrows = null_count;
+ null_bucket->rows
+ = (HeapTuple*)palloc0(null_bucket->numrows * sizeof(HeapTuple));
+
+ /* mark the dimension as NULL-only (in the new bucket) */
+ null_bucket->nullsonly[null_dim] = true;
+
+ /* walk through the sample rows and distribute them accordingly */
+ null_idx = 0;
+ curr_idx = 0;
+ for (i = 0; i < numrows; i++)
+ {
+ if (heap_attisnull(oldrows[i], attrs->values[null_dim]))
+ /* NULL => copy to the new bucket */
+ memcpy(&null_bucket->rows[null_idx++], &oldrows[i],
+ sizeof(HeapTuple));
+ else
+ memcpy(&bucket->rows[curr_idx++], &oldrows[i],
+ sizeof(HeapTuple));
+ }
+
+ /* update ndistinct values for the buckets (total and per dimension) */
+ update_bucket_ndistinct(bucket, attrs, stats);
+ update_bucket_ndistinct(null_bucket, attrs, stats);
+
+ /*
+ * TODO We don't need to do this for the dimension we used for split,
+ * because we know how many distinct values went to each
+ * bucket (NULL is not a value, so 0, and the other bucket got
+ * all the ndistinct values).
+ */
+ for (i = 0; i < histogram->ndimensions; i++)
+ {
+ update_dimension_ndistinct(bucket, i, attrs, stats, false);
+ update_dimension_ndistinct(null_bucket, i, attrs, stats, false);
+ }
+
+ pfree(oldrows);
+
+ /* add the NULL bucket to the histogram */
+ histogram->buckets[histogram->nbuckets++] = null_bucket;
+
+ /*
+ * And now run the function recursively on both buckets (the new
+ * one first, because the call may change number of buckets, and
+ * it's used as an index).
+ */
+ create_null_buckets(histogram, (histogram->nbuckets-1), attrs, stats);
+ create_null_buckets(histogram, bucket_idx, attrs, stats);
+
+}
+
+/*
+ * We need to pass the SortSupport to the comparator, but bsearch()
+ * has no 'context' parameter, so we use a global variable (ugly).
+ */
+static int
+bsearch_comparator(const void * a, const void * b)
+{
+ Assert(ssup_private != NULL);
+ return compare_scalars_simple(a, b, (void*)ssup_private);
+}
diff --git a/src/backend/utils/mvstats/mcv.c b/src/backend/utils/mvstats/mcv.c
index 2b3d171..b0cea61 100644
--- a/src/backend/utils/mvstats/mcv.c
+++ b/src/backend/utils/mvstats/mcv.c
@@ -961,6 +961,7 @@ MCVList deserialize_mv_mcvlist(bytea * data)
for (i = 0; i < nitems; i++)
{
+ /* FIXME allocate as a single chunk (minimize palloc overhead) */
MCVItem item = (MCVItem)palloc0(sizeof(MCVItemData));
item->values = (Datum*)palloc0(sizeof(Datum)*ndims);
diff --git a/src/include/catalog/pg_mv_statistic.h b/src/include/catalog/pg_mv_statistic.h
index f88e200..08424bd 100644
--- a/src/include/catalog/pg_mv_statistic.h
+++ b/src/include/catalog/pg_mv_statistic.h
@@ -36,13 +36,16 @@ CATALOG(pg_mv_statistic,3281)
/* statistics requested to build */
bool deps_enabled; /* analyze dependencies? */
bool mcv_enabled; /* build MCV list? */
+ bool hist_enabled; /* build histogram? */
- /* MCV size */
+ /* histogram / MCV size */
int32 mcv_max_items; /* max MCV items */
+ int32 hist_max_buckets; /* max histogram buckets */
/* statistics that are available (if requested) */
bool deps_built; /* dependencies were built */
bool mcv_built; /* MCV list was built */
+ bool hist_built; /* histogram was built */
/* variable-length fields start here, but we allow direct access to stakeys */
int2vector stakeys; /* array of column keys */
@@ -50,6 +53,7 @@ CATALOG(pg_mv_statistic,3281)
#ifdef CATALOG_VARLEN
bytea stadeps; /* dependencies (serialized) */
bytea stamcv; /* MCV list (serialized) */
+ bytea stahist; /* MV histogram (serialized) */
#endif
} FormData_pg_mv_statistic;
@@ -65,15 +69,19 @@ typedef FormData_pg_mv_statistic *Form_pg_mv_statistic;
* compiler constants for pg_attrdef
* ----------------
*/
-#define Natts_pg_mv_statistic 9
+#define Natts_pg_mv_statistic 13
#define Anum_pg_mv_statistic_starelid 1
#define Anum_pg_mv_statistic_deps_enabled 2
#define Anum_pg_mv_statistic_mcv_enabled 3
-#define Anum_pg_mv_statistic_mcv_max_items 4
-#define Anum_pg_mv_statistic_deps_built 5
-#define Anum_pg_mv_statistic_mcv_built 6
-#define Anum_pg_mv_statistic_stakeys 7
-#define Anum_pg_mv_statistic_stadeps 8
-#define Anum_pg_mv_statistic_stamcv 9
+#define Anum_pg_mv_statistic_hist_enabled 4
+#define Anum_pg_mv_statistic_mcv_max_items 5
+#define Anum_pg_mv_statistic_hist_max_buckets 6
+#define Anum_pg_mv_statistic_deps_built 7
+#define Anum_pg_mv_statistic_mcv_built 8
+#define Anum_pg_mv_statistic_hist_built 9
+#define Anum_pg_mv_statistic_stakeys 10
+#define Anum_pg_mv_statistic_stadeps 11
+#define Anum_pg_mv_statistic_stamcv 12
+#define Anum_pg_mv_statistic_stahist 13
#endif /* PG_MV_STATISTIC_H */
diff --git a/src/include/catalog/pg_proc.h b/src/include/catalog/pg_proc.h
index b4e7b4f..448e76a 100644
--- a/src/include/catalog/pg_proc.h
+++ b/src/include/catalog/pg_proc.h
@@ -2689,6 +2689,8 @@ DATA(insert OID = 3285 ( pg_mv_stats_dependencies_show PGNSP PGUID 12 1 0 0
DESCR("multivariate stats: functional dependencies show");
DATA(insert OID = 3283 ( pg_mv_stats_mcvlist_info PGNSP PGUID 12 1 0 0 0 f f f f t f i 1 0 25 "17" _null_ _null_ _null_ _null_ pg_mv_stats_mcvlist_info _null_ _null_ _null_ ));
DESCR("multi-variate statistics: MCV list info");
+DATA(insert OID = 3282 ( pg_mv_stats_histogram_info PGNSP PGUID 12 1 0 0 0 f f f f t f i 1 0 25 "17" _null_ _null_ _null_ _null_ pg_mv_stats_histogram_info _null_ _null_ _null_ ));
+DESCR("multi-variate statistics: histogram info");
DATA(insert OID = 1928 ( pg_stat_get_numscans PGNSP PGUID 12 1 0 0 0 f f f f t f s 1 0 20 "26" _null_ _null_ _null_ _null_ pg_stat_get_numscans _null_ _null_ _null_ ));
DESCR("statistics: number of scans done for table/index");
diff --git a/src/include/utils/mvstats.h b/src/include/utils/mvstats.h
index e11aefc..028a634 100644
--- a/src/include/utils/mvstats.h
+++ b/src/include/utils/mvstats.h
@@ -26,10 +26,12 @@ typedef struct MVStatsData {
/* statistics requested in ALTER TABLE ... ADD STATISTICS */
bool deps_enabled; /* analyze functional dependencies */
bool mcv_enabled; /* analyze MCV lists */
+ bool hist_enabled; /* analyze histogram */
/* available statistics (computed by ANALYZE) */
bool deps_built; /* functional dependencies available */
bool mcv_built; /* MCV list is already available */
+ bool hist_built; /* histogram is already available */
} MVStatsData;
typedef struct MVStatsData *MVStats;
@@ -109,6 +111,91 @@ typedef MCVListData *MCVList;
#define MVSTAT_MCVLIST_MAX_ITEMS 8192 /* max items in MCV list */
/*
+ * Multivariate histograms
+ */
+typedef struct MVBucketData {
+
+ /* Frequencies of this bucket. */
+ float ntuples; /* frequency of tuples tuples */
+ float ndistinct; /* frequency of distinct values */
+
+ /*
+ * Number of distinct values in each dimension. This is used when
+ * building the histogram (and is not serialized/deserialized), but
+ * it could be useful for estimating ndistinct for combinations of
+ * columns.
+ *
+ * It would mean tracking 2^N values for each bucket, and even if
+ * those values might be stores in 1B it's still a lot of space
+ * (considering the expected number of buckets).
+ *
+ * TODO Consider tracking ndistincts for all attribute combinations.
+ */
+ uint32 *ndistincts;
+
+ /*
+ * Information about dimensions being NULL-only. Not yet used.
+ */
+ bool *nullsonly;
+
+ /* lower boundaries - values and information about the inequalities */
+ Datum *min;
+ bool *min_inclusive;
+
+ /* upper boundaries - values and information about the inequalities */
+ Datum *max;
+ bool *max_inclusive;
+
+ /*
+ * Sample tuples falling into this bucket, index of the dimension
+ * the bucket was split by in the last step.
+ *
+ * XXX These fields are needed only while building the histogram,
+ * and are not serialized at all.
+ */
+ HeapTuple *rows;
+ uint32 numrows;
+ int last_split_dimension;
+
+} MVBucketData;
+
+typedef MVBucketData *MVBucket;
+
+
+typedef struct MVHistogramData {
+
+ uint32 magic; /* magic constant marker */
+ uint32 type; /* type of histogram (BASIC) */
+ uint32 nbuckets; /* number of buckets (buckets array) */
+ uint32 ndimensions; /* number of dimensions */
+
+ MVBucket *buckets; /* array of buckets */
+
+} MVHistogramData;
+
+typedef MVHistogramData *MVHistogram;
+
+/* used to flag stats serialized to bytea */
+#define MVSTAT_HIST_MAGIC 0x7F8C5670 /* marks serialized bytea */
+#define MVSTAT_HIST_TYPE_BASIC 1 /* basic histogram type */
+
+/*
+ * Limits used for max_buckets option, i.e. we're always guaranteed
+ * to have space for at least MVSTAT_HIST_MIN_BUCKETS, and we cannot
+ * have more than MVSTAT_HIST_MAX_BUCKETS buckets.
+ *
+ * This is just a boundary for the 'max' threshold - the actual
+ * histogram may use less buckets than MVSTAT_HIST_MAX_BUCKETS.
+ *
+ * TODO The MVSTAT_HIST_MIN_BUCKETS should be related to the number of
+ * attributes (MVSTATS_MAX_DIMENSIONS) because of NULL-buckets.
+ * There should be at least 2^N buckets, otherwise we may be unable
+ * to build the NULL buckets.
+ */
+#define MVSTAT_HIST_MIN_BUCKETS 128 /* min number of buckets */
+#define MVSTAT_HIST_MAX_BUCKETS 16384 /* max number of buckets */
+
+/*
* TODO Maybe fetching the histogram/MCV list separately is inefficient?
* Consider adding a single `fetch_stats` method, fetching all
* stats specified using flags (or something like that).
@@ -118,14 +205,18 @@ bytea * fetch_mv_rules(Oid mvoid);
bytea * fetch_mv_dependencies(Oid mvoid);
bytea * fetch_mv_mcvlist(Oid mvoid);
+bytea * fetch_mv_histogram(Oid mvoid);
bytea * serialize_mv_dependencies(MVDependencies dependencies);
bytea * serialize_mv_mcvlist(MCVList mcvlist, int2vector *attrs,
VacAttrStats **stats);
+bytea * serialize_mv_histogram(MVHistogram histogram, int2vector *attrs,
+ VacAttrStats **stats);
/* deserialization of stats (serialization is private to analyze) */
MVDependencies deserialize_mv_dependencies(bytea * data);
MCVList deserialize_mv_mcvlist(bytea * data);
+MVHistogram deserialize_mv_histogram(bytea * data);
/*
* Returns index of the attribute number within the vector (i.e. a
@@ -137,6 +228,7 @@ int mv_get_index(AttrNumber varattno, int2vector * stakeys);
extern Datum pg_mv_stats_dependencies_info(PG_FUNCTION_ARGS);
extern Datum pg_mv_stats_dependencies_show(PG_FUNCTION_ARGS);
extern Datum pg_mv_stats_mcvlist_info(PG_FUNCTION_ARGS);
+extern Datum pg_mv_stats_histogram_info(PG_FUNCTION_ARGS);
MVDependencies
build_mv_dependencies(int numrows, HeapTuple *rows, int2vector *attrs,
@@ -146,10 +238,15 @@ MCVList
build_mv_mcvlist(int numrows, HeapTuple *rows, int2vector *attrs,
VacAttrStats **stats, int *numrows_filtered);
+MVHistogram
+build_mv_histogram(int numrows, HeapTuple *rows, int2vector *attrs,
+ VacAttrStats **stats, int numrows_total);
+
void build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
int natts, VacAttrStats **vacattrstats);
-void update_mv_stats(Oid relid, MVDependencies dependencies, MCVList mcvlist,
+void update_mv_stats(Oid relid, MVDependencies dependencies,
+ MCVList mcvlist, MVHistogram histogram,
int2vector *attrs, VacAttrStats **stats);
#endif
diff --git a/src/test/regress/expected/mv_histogram.out b/src/test/regress/expected/mv_histogram.out
new file mode 100644
index 0000000..a0cf37f
--- /dev/null
+++ b/src/test/regress/expected/mv_histogram.out
@@ -0,0 +1,210 @@
+-- data type passed by value
+CREATE TABLE mv_histogram (
+ a INT,
+ b INT,
+ c INT
+);
+-- unknown column
+ALTER TABLE mv_histogram ADD STATISTICS (histogram) ON (unknown_column);
+ERROR: column "unknown_column" referenced in statistics does not exist
+-- single column
+ALTER TABLE mv_histogram ADD STATISTICS (histogram) ON (a);
+ERROR: multivariate stats require 2 or more columns
+-- single column, duplicated
+ALTER TABLE mv_histogram ADD STATISTICS (histogram) ON (a, a);
+ERROR: duplicate column name in statistics definition
+-- two columns, one duplicated
+ALTER TABLE mv_histogram ADD STATISTICS (histogram) ON (a, a, b);
+ERROR: duplicate column name in statistics definition
+-- unknown option
+ALTER TABLE mv_histogram ADD STATISTICS (unknown_option) ON (a, b, c);
+ERROR: unrecognized STATISTICS option "unknown_option"
+-- missing histogram statistics
+ALTER TABLE mv_histogram ADD STATISTICS (dependencies, max_buckets 200) ON (a, b, c);
+ERROR: option 'histogram' is required by other options(s)
+-- invalid max_buckets value / too low
+ALTER TABLE mv_histogram ADD STATISTICS (mcv, max_buckets 10) ON (a, b, c);
+ERROR: minimum number of buckets is 128
+-- invalid max_buckets value / too high
+ALTER TABLE mv_histogram ADD STATISTICS (mcv, max_buckets 100000) ON (a, b, c);
+ERROR: minimum number of buckets is 16384
+-- correct command
+ALTER TABLE mv_histogram ADD STATISTICS (histogram) ON (a, b, c);
+-- random data (no functional dependencies)
+INSERT INTO mv_histogram
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+ANALYZE mv_histogram;
+SELECT hist_enabled, hist_built, pg_mv_stats_histogram_info(stahist)
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+ hist_enabled | hist_built | pg_mv_stats_histogram_info
+--------------+------------+----------------------------
+ t | t | nbuckets=10000
+(1 row)
+
+TRUNCATE mv_histogram;
+-- a => b, a => c, b => c
+INSERT INTO mv_histogram
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE mv_histogram;
+SELECT hist_enabled, hist_built, pg_mv_stats_histogram_info(stahist)
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+ hist_enabled | hist_built | pg_mv_stats_histogram_info
+--------------+------------+----------------------------
+ t | t | nbuckets=1001
+(1 row)
+
+TRUNCATE mv_histogram;
+-- a => b, a => c
+INSERT INTO mv_histogram
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE mv_histogram;
+SELECT hist_enabled, hist_built, pg_mv_stats_histogram_info(stahist)
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+ hist_enabled | hist_built | pg_mv_stats_histogram_info
+--------------+------------+----------------------------
+ t | t | nbuckets=1001
+(1 row)
+
+TRUNCATE mv_histogram;
+-- check explain (expect bitmap index scan, not plain index scan)
+INSERT INTO mv_histogram
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX hist_idx ON mv_histogram (a, b);
+ANALYZE mv_histogram;
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+ hist_enabled | hist_built
+--------------+------------
+ t | t
+(1 row)
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mv_histogram WHERE a = 10 AND b = 5;
+ QUERY PLAN
+--------------------------------------------
+ Bitmap Heap Scan on mv_histogram
+ Recheck Cond: ((a = 10) AND (b = 5))
+ -> Bitmap Index Scan on hist_idx
+ Index Cond: ((a = 10) AND (b = 5))
+(4 rows)
+
+DELETE FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+DROP TABLE mv_histogram;
+-- varlena type (text)
+CREATE TABLE mv_histogram (
+ a TEXT,
+ b TEXT,
+ c TEXT
+);
+ALTER TABLE mv_histogram ADD STATISTICS (histogram) ON (a, b, c);
+-- random data (no functional dependencies)
+INSERT INTO mv_histogram
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+ANALYZE mv_histogram;
+SELECT hist_enabled, hist_built, pg_mv_stats_histogram_info(stahist)
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+ hist_enabled | hist_built | pg_mv_stats_histogram_info
+--------------+------------+----------------------------
+ t | t | nbuckets=10000
+(1 row)
+
+TRUNCATE mv_histogram;
+-- a => b, a => c, b => c
+INSERT INTO mv_histogram
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE mv_histogram;
+SELECT hist_enabled, hist_built, pg_mv_stats_histogram_info(stahist)
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+ hist_enabled | hist_built | pg_mv_stats_histogram_info
+--------------+------------+----------------------------
+ t | t | nbuckets=3492
+(1 row)
+
+TRUNCATE mv_histogram;
+-- a => b, a => c
+INSERT INTO mv_histogram
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE mv_histogram;
+SELECT hist_enabled, hist_built, pg_mv_stats_histogram_info(stahist)
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+ hist_enabled | hist_built | pg_mv_stats_histogram_info
+--------------+------------+----------------------------
+ t | t | nbuckets=3433
+(1 row)
+
+TRUNCATE mv_histogram;
+-- check explain (expect bitmap index scan, not plain index scan)
+INSERT INTO mv_histogram
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX hist_idx ON mv_histogram (a, b);
+ANALYZE mv_histogram;
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+ hist_enabled | hist_built
+--------------+------------
+ t | t
+(1 row)
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mv_histogram WHERE a = '10' AND b = '5';
+ QUERY PLAN
+------------------------------------------------------------
+ Bitmap Heap Scan on mv_histogram
+ Recheck Cond: ((a = '10'::text) AND (b = '5'::text))
+ -> Bitmap Index Scan on hist_idx
+ Index Cond: ((a = '10'::text) AND (b = '5'::text))
+(4 rows)
+
+TRUNCATE mv_histogram;
+-- check explain (expect bitmap index scan, not plain index scan) with NULLs
+INSERT INTO mv_histogram
+ SELECT
+ (CASE WHEN i/10000 = 0 THEN NULL ELSE i/10000 END),
+ (CASE WHEN i/20000 = 0 THEN NULL ELSE i/20000 END),
+ (CASE WHEN i/40000 = 0 THEN NULL ELSE i/40000 END)
+ FROM generate_series(1,1000000) s(i);
+ANALYZE mv_histogram;
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+ hist_enabled | hist_built
+--------------+------------
+ t | t
+(1 row)
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mv_histogram WHERE a IS NULL AND b IS NULL;
+ QUERY PLAN
+---------------------------------------------------
+ Bitmap Heap Scan on mv_histogram
+ Recheck Cond: ((a IS NULL) AND (b IS NULL))
+ -> Bitmap Index Scan on hist_idx
+ Index Cond: ((a IS NULL) AND (b IS NULL))
+(4 rows)
+
+DELETE FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+DROP TABLE mv_histogram;
+-- NULL values (mix of int and text columns)
+CREATE TABLE mv_histogram (
+ a INT,
+ b TEXT,
+ c INT,
+ d TEXT
+);
+ALTER TABLE mv_histogram ADD STATISTICS (histogram) ON (a, b, c, d);
+INSERT INTO mv_histogram
+ SELECT
+ mod(i, 100),
+ (CASE WHEN mod(i, 200) = 0 THEN NULL ELSE mod(i,200) END),
+ mod(i, 400),
+ (CASE WHEN mod(i, 300) = 0 THEN NULL ELSE mod(i,600) END)
+ FROM generate_series(1,10000) s(i);
+ANALYZE mv_histogram;
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+ hist_enabled | hist_built
+--------------+------------
+ t | t
+(1 row)
+
+DELETE FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+DROP TABLE mv_histogram;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 80375b8..07896b4 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1359,7 +1359,9 @@ pg_mv_stats| SELECT n.nspname AS schemaname,
length(s.stadeps) AS depsbytes,
pg_mv_stats_dependencies_info(s.stadeps) AS depsinfo,
length(s.stamcv) AS mcvbytes,
- pg_mv_stats_mcvlist_info(s.stamcv) AS mcvinfo
+ pg_mv_stats_mcvlist_info(s.stamcv) AS mcvinfo,
+ length(s.stahist) AS histbytes,
+ pg_mv_stats_histogram_info(s.stahist) AS histinfo
FROM ((pg_mv_statistic s
JOIN pg_class c ON ((c.oid = s.starelid)))
LEFT JOIN pg_namespace n ON ((n.oid = c.relnamespace)));
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index 78c9b04..d9864b7 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -111,4 +111,4 @@ test: event_trigger
test: stats
# run tests of multivariate stats
-test: mv_dependencies mv_mcv
+test: mv_dependencies mv_mcv mv_histogram
diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule
index 3f9884f..d901a78 100644
--- a/src/test/regress/serial_schedule
+++ b/src/test/regress/serial_schedule
@@ -154,3 +154,4 @@ test: event_trigger
test: stats
test: mv_dependencies
test: mv_mcv
+test: mv_histogram
diff --git a/src/test/regress/sql/mv_histogram.sql b/src/test/regress/sql/mv_histogram.sql
new file mode 100644
index 0000000..a693e35
--- /dev/null
+++ b/src/test/regress/sql/mv_histogram.sql
@@ -0,0 +1,179 @@
+-- data type passed by value
+CREATE TABLE mv_histogram (
+ a INT,
+ b INT,
+ c INT
+);
+
+-- unknown column
+ALTER TABLE mv_histogram ADD STATISTICS (histogram) ON (unknown_column);
+
+-- single column
+ALTER TABLE mv_histogram ADD STATISTICS (histogram) ON (a);
+
+-- single column, duplicated
+ALTER TABLE mv_histogram ADD STATISTICS (histogram) ON (a, a);
+
+-- two columns, one duplicated
+ALTER TABLE mv_histogram ADD STATISTICS (histogram) ON (a, a, b);
+
+-- unknown option
+ALTER TABLE mv_histogram ADD STATISTICS (unknown_option) ON (a, b, c);
+
+-- missing histogram statistics
+ALTER TABLE mv_histogram ADD STATISTICS (dependencies, max_buckets 200) ON (a, b, c);
+
+-- invalid max_buckets value / too low
+ALTER TABLE mv_histogram ADD STATISTICS (mcv, max_buckets 10) ON (a, b, c);
+
+-- invalid max_buckets value / too high
+ALTER TABLE mv_histogram ADD STATISTICS (mcv, max_buckets 100000) ON (a, b, c);
+
+-- correct command
+ALTER TABLE mv_histogram ADD STATISTICS (histogram) ON (a, b, c);
+
+-- random data (no functional dependencies)
+INSERT INTO mv_histogram
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+
+ANALYZE mv_histogram;
+
+SELECT hist_enabled, hist_built, pg_mv_stats_histogram_info(stahist)
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+
+TRUNCATE mv_histogram;
+
+-- a => b, a => c, b => c
+INSERT INTO mv_histogram
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+
+ANALYZE mv_histogram;
+
+SELECT hist_enabled, hist_built, pg_mv_stats_histogram_info(stahist)
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+
+TRUNCATE mv_histogram;
+
+-- a => b, a => c
+INSERT INTO mv_histogram
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE mv_histogram;
+
+SELECT hist_enabled, hist_built, pg_mv_stats_histogram_info(stahist)
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+
+TRUNCATE mv_histogram;
+
+-- check explain (expect bitmap index scan, not plain index scan)
+INSERT INTO mv_histogram
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX hist_idx ON mv_histogram (a, b);
+ANALYZE mv_histogram;
+
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mv_histogram WHERE a = 10 AND b = 5;
+
+DELETE FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+DROP TABLE mv_histogram;
+
+-- varlena type (text)
+CREATE TABLE mv_histogram (
+ a TEXT,
+ b TEXT,
+ c TEXT
+);
+
+ALTER TABLE mv_histogram ADD STATISTICS (histogram) ON (a, b, c);
+
+-- random data (no functional dependencies)
+INSERT INTO mv_histogram
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+
+ANALYZE mv_histogram;
+
+SELECT hist_enabled, hist_built, pg_mv_stats_histogram_info(stahist)
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+
+TRUNCATE mv_histogram;
+
+-- a => b, a => c, b => c
+INSERT INTO mv_histogram
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+
+ANALYZE mv_histogram;
+
+SELECT hist_enabled, hist_built, pg_mv_stats_histogram_info(stahist)
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+
+TRUNCATE mv_histogram;
+
+-- a => b, a => c
+INSERT INTO mv_histogram
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE mv_histogram;
+
+SELECT hist_enabled, hist_built, pg_mv_stats_histogram_info(stahist)
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+
+TRUNCATE mv_histogram;
+
+-- check explain (expect bitmap index scan, not plain index scan)
+INSERT INTO mv_histogram
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX hist_idx ON mv_histogram (a, b);
+ANALYZE mv_histogram;
+
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mv_histogram WHERE a = '10' AND b = '5';
+
+TRUNCATE mv_histogram;
+
+-- check explain (expect bitmap index scan, not plain index scan) with NULLs
+INSERT INTO mv_histogram
+ SELECT
+ (CASE WHEN i/10000 = 0 THEN NULL ELSE i/10000 END),
+ (CASE WHEN i/20000 = 0 THEN NULL ELSE i/20000 END),
+ (CASE WHEN i/40000 = 0 THEN NULL ELSE i/40000 END)
+ FROM generate_series(1,1000000) s(i);
+ANALYZE mv_histogram;
+
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mv_histogram WHERE a IS NULL AND b IS NULL;
+
+DELETE FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+DROP TABLE mv_histogram;
+
+-- NULL values (mix of int and text columns)
+CREATE TABLE mv_histogram (
+ a INT,
+ b TEXT,
+ c INT,
+ d TEXT
+);
+
+ALTER TABLE mv_histogram ADD STATISTICS (histogram) ON (a, b, c, d);
+
+INSERT INTO mv_histogram
+ SELECT
+ mod(i, 100),
+ (CASE WHEN mod(i, 200) = 0 THEN NULL ELSE mod(i,200) END),
+ mod(i, 400),
+ (CASE WHEN mod(i, 300) = 0 THEN NULL ELSE mod(i,600) END)
+ FROM generate_series(1,10000) s(i);
+
+ANALYZE mv_histogram;
+
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+
+DELETE FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+DROP TABLE mv_histogram;
--
2.0.5
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: WIP: multivariate statistics / proof of concept
2014-10-12 22:00 WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-11-13 11:31 ` Re: WIP: multivariate statistics / proof of concept Simon Riggs <simon@2ndQuadrant.com>
2014-11-13 13:11 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-11-13 15:51 ` Re: WIP: multivariate statistics / proof of concept Katharina Büchse <katharina.buechse@uni-jena.de>
2014-11-13 16:42 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-11-15 17:49 ` Re: WIP: multivariate statistics / proof of concept Kevin Grittner <kgrittn@ymail.com>
2014-11-15 18:35 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-12-08 01:01 ` Re: WIP: multivariate statistics / proof of concept Michael Paquier <michael.paquier@gmail.com>
2014-12-09 20:15 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-12-15 02:55 ` Re: WIP: multivariate statistics / proof of concept Michael Paquier <michael.paquier@gmail.com>
2015-01-15 08:00 ` Re: WIP: multivariate statistics / proof of concept Michael Paquier <michael.paquier@gmail.com>
2015-01-24 20:21 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tomas.vondra@2ndquadrant.com>
@ 2015-03-20 08:33 ` Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
2015-03-20 21:17 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 1 reply; 37+ messages in thread
From: Kyotaro HORIGUCHI @ 2015-03-20 08:33 UTC (permalink / raw)
To: tomas.vondra@2ndquadrant.com; +Cc: pgsql-hackers
Hello,
Patch 0001 needs changes for OIDs since my patch was
committed. The attached is compatible with current master.
And I tried this like this, and got the following error on
analyze. But unfortunately I don't have enough time to
investigate it now.
postgres=# create table t1 (a int, b int, c int);
insert into t1 (select a/ 10000, a / 10000, a / 10000 from generate_series(0, 99999) a);
postgres=# analyze t1;
ERROR: invalid memory alloc request size 1485176862
regards,
At Sat, 24 Jan 2015 21:21:39 +0100, Tomas Vondra <tomas.vondra@2ndquadrant.com> wrote in <54C3FED3.1060600@2ndquadrant.com>
> Hi,
>
> attached is an updated version of the multivariate stats patch. This is
> going to be a bit longer mail, so I'll put here a small ToC ;-)
>
> 1) patch split into 4 parts
> 2) where to start / documentation
> 3) state of the code
> 4) main changes/improvements
> 5) remaining limitations
>
> The motivation and design ideas, explained in the first message of this
> thread are still valid. It might be a good idea to read it first:
>
> http://www.postgresql.org/message-id/flat/543AFA15.4080608@fuzzy.cz
>
> BTW if you happen to go to FOSDEM [PGDay], I'll gladly give you an intro
> into the patch in person, or discuss the patch in general.
>
>
> 1) Patch split into 4 parts
> ---------------------------
> Firstly, the patch got broken into the following four pieces, to make
> the reviews somewhat easier:
>
> 1) 0001-shared-infrastructure-and-functional-dependencies.patch
>
> - infrastructure, shared by all the kinds of stats added
> in the following patches (catalog, ALTER TABLE, ANALYZE ...)
>
> - implementation of a simple statistics, tracking functional
> dependencies between columns (previously called "associative
> rules", but that's incorrect for several reasons)
>
> - this does not modify the optimizer in any way
> 2) 0002-clause-reduction-using-functional-dependencies.patch
>
> - applies the functional dependencies to optimizer (i.e. considers
> the rules in clauselist_selectivity())
>
> 3) 0003-multivariate-MCV-lists.patch
>
> - multivariate MCV lists (both ANALYZE and optimizer parts)
>
> 4) 0004-multivariate-histograms.patch
>
> - multivariate histograms (both ANALYZE and optimizer parts)
>
>
> You may look at the patches at github here:
>
> https://github.com/tvondra/postgres/tree/multivariate-stats-squashed
>
> The branch is not stable, i.e. I'll rebase / squash / force-push changes
> in the future. (There's also multivariate-stats development branch with
> unsquashed changes, but you don't want to look at that, trust me.)
>
> The patches are not exactly small (being in the 50-100 kB range), but
> that's mostly because of the amount of comments explaining the goals and
> implementation details.
>
>
> 2) Where to start / documentation
> ---------------------------------
> I strived to document all the pieces properly, mostly in the form of
> comments. There's no sgml documentation at this point, which should
> obviously change in the future.
>
> Anyway, I'd suggest reading the first e-mail in this thread, explaining
> the ideas, and then these comments:
>
> 1) functional dependencies (patch 0001)
> - src/backend/utils/mvstats/dependencies.c
>
> 2) MCV lists (patch 0003)
> - src/backend/utils/mvstats/mcv.c
>
> 3) histograms (patch 0004)
> - src/backend/utils/mvstats/mcv.c
>
> - also see clauselist_mv_selectivity_mcvlist() in clausesel.c
> - also see clauselist_mv_selectivity_histogram() in clausesel.c
>
> 4) selectivity estimation (patches 0002-0004)
> - all in src/backend/optimizer/path/clausesel.c
> - clauselist_selectivity() - overview of how the stats are applied
> - clauselist_apply_dependencies() - functional dependencies reduction
> - clauselist_mv_selectivity_mcvlist() - MCV list estimation
> - clauselist_mv_selectivity_histogram() - histogram estimation
>
>
> 3) State of the code
> --------------------
> I've spent a fair amount of time testing the patches, and while I
> believe there are no segfaults or so, I know parts of the code need a
> bit more love.
>
> The part most in need of improvements / comments is probably the code in
> clausesel.c - that seems a bit quirky. Reviews / comments regarding this
> part of the code are very welcome - I'm sure there are many ways to
> improve this part.
>
> There are a few FIXMEs elsewhere (e.g. about memory allocation in the
> (de)serialization code), but those are mostly well-defined issues that I
> know how to address (at least I believe so).
>
>
> 4) Main changes/improvements
> ----------------------------
> There are many significant improvements. The previous patch version was
> in the 'proof of concept' category (missing pieces, knowingly broken in
> some areas), the current patch should 'mostly work'.
>
> The patch fixes two most annoying limitations of the first version:
>
> (a) support for all data types (not just those passed by value)
> (b) handles NULL values properly
> (c) adds support for IS [NOT] NULL clauses
>
> Aside from that the code was significantly improved, there are proper
> regression tests and plenty of comments explaining the details.
>
>
> 5) Remaining limitations
> ------------------------
>
> (a) limited to stats on 8 columns
>
> This is mostly just a 'safeguard' restriction.
>
> (b) only data types with '<' operator
>
> I don't think this will change anytime soon, because all the
> algorithms for building the stats rely on this. I don't see
> this as a serious limitation though.
>
> (c) not handling DROP COLUMN or DROP TABLE and so on
>
> Currently this is not handled at all (so the regression tests
> do an explicit DELETE from the pg_mv_statistic catalog).
>
> Handling the DROP TABLE won't be difficult, it's similar to the
> current stats. Handling ALTER TABLE ... DROP COLUMN will be much
> more tricky I guess - should we drop all the stats referencing
> that column, or should we just remove it from the stats? Or
> should we keep it and treat it as NULL? Not sure what's the best
> solution.
>
> (d) limited list of compatible WHERE clauses
>
> The initial patch handled only simple operator clauses
>
> (Var op Constant)
>
> where operator is one of ('<', '<=', '=', '>=', '>'). Now it also
> handles IS [NOT] NULL clauses. Adding more clause types should
> not be overly difficult - starting with more traditional
> 'BooleanTest' conditions, or even multi-column conditions
> (Var op Var)
>
> which are difficult to estimate using simple-column stats.
>
> (e) optimizer uses single stats per table
>
> This is still true and I don't think this will change soon. i do
> have some ideas on how to merge multiple stats etc. but it's
> certainly complex stuff, unlikely to happen within this CF. The
> patch makes a lot of sense even without this particular feature,
> because you can create multiple stats, each suitable for different
> queries.
>
> (f) no JOIN conditions
>
> Similarly to the previous point, it's on the TODO but it's not
> going to happen in this CF.
>
>
> kind regards
>
> --
> Tomas Vondra http://www.2ndQuadrant.com/
> PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[text/x-patch] 0001-shared-infrastructure-and-functional-dependencies.patch (59.6K, ../../20150320.173307.249113775.horiguchi.kyotaro@lab.ntt.co.jp/2-0001-shared-infrastructure-and-functional-dependencies.patch)
download | inline diff:
From 9ebfadb5d6cd9b55dd2707cfc8c789884dafa7fa Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Sun, 11 Jan 2015 19:51:48 +0100
Subject: [PATCH 1/4] shared infrastructure and functional dependencies
Basic infrastructure shared by all kinds of multivariate
stats, most importantly:
- adds a new system catalog (pg_mv_statistic)
- ALTER TABLE ... ADD STATISTICS syntax
- implementation of functional dependencies (the simplest
type of multivariate statistics)
- building functional dependencies in ANALYZE
- updates regression tests (new catalog etc.)
This does not include any changes to the optimizer, i.e.
it does not influence the query planning.
FIX: invalid assert in lookup_var_attr_stats()
The current implementation requires a valid 'ltopr'
so that we can sort the sample rows in various ways,
and the assert did verify this by checking that the
function is 'compute_scalar_stats'. This is however
private function in analyze.c, so the check failed
after moving the code into common.c.
Fixed by checking the 'ltopr' operator directly.
Eventually this will be removed, as ltopr is only
needed for histograms (functional dependencies and
MVC lists may be built without it).
FIX: improved comments about functional dependencies
FIX: add magic (MVSTAT_DEPS_MAGIC) into MVDependencies
FIX: improved analysis of functional dependencies
Changes:
- decreased minimum group size
- count contradicting rows ('not supporting' ones)
The algorithm is still rather simple and probably needs
other improvements.
FIX: add pg_mv_stats_dependencies_show() function
This function actually prints the rules, not just some basic
info (number of rules) as pg_mv_stats_dependencies_info().
FIX: (dependencies != NULL) in pg_mv_stats_dependencies_info()
STRICT is not a solution, because the deserialization may fail
for some reason (corrupted data, ...)
FIX: rename 'associative rules' to 'functional dependencies'
It's a more appropriate name as functional dependencies,
as defined in relational theory (esp. Normal Forms) are
tracking column-level dependencies.
Associative (or more correctly 'association') rules are
tracking dependencies between particular values, and not
necessarily in different columns (shopping bag analysis).
Also, did a bunch of comment improvements, minor fixes.
This does not include changes in clausesel.c!
FIX: remove obsolete Assert() enforcing typbyval types
---
src/backend/catalog/Makefile | 1 +
src/backend/catalog/system_views.sql | 10 +
src/backend/commands/analyze.c | 17 +-
src/backend/commands/tablecmds.c | 149 +++++++-
src/backend/nodes/copyfuncs.c | 15 +-
src/backend/parser/gram.y | 67 +++-
src/backend/utils/Makefile | 2 +-
src/backend/utils/cache/syscache.c | 12 +
src/backend/utils/mvstats/Makefile | 17 +
src/backend/utils/mvstats/common.c | 272 ++++++++++++++
src/backend/utils/mvstats/common.h | 70 ++++
src/backend/utils/mvstats/dependencies.c | 554 +++++++++++++++++++++++++++++
src/include/catalog/indexing.h | 5 +
src/include/catalog/pg_mv_statistic.h | 69 ++++
src/include/catalog/pg_proc.h | 5 +
src/include/catalog/toasting.h | 1 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 11 +-
src/include/utils/mvstats.h | 86 +++++
src/include/utils/syscache.h | 1 +
src/test/regress/expected/rules.out | 8 +
src/test/regress/expected/sanity_check.out | 1 +
22 files changed, 1365 insertions(+), 9 deletions(-)
create mode 100644 src/backend/utils/mvstats/Makefile
create mode 100644 src/backend/utils/mvstats/common.c
create mode 100644 src/backend/utils/mvstats/common.h
create mode 100644 src/backend/utils/mvstats/dependencies.c
create mode 100644 src/include/catalog/pg_mv_statistic.h
create mode 100644 src/include/utils/mvstats.h
diff --git a/src/backend/catalog/Makefile b/src/backend/catalog/Makefile
index a403c64..d6c16f8 100644
--- a/src/backend/catalog/Makefile
+++ b/src/backend/catalog/Makefile
@@ -32,6 +32,7 @@ POSTGRES_BKI_SRCS = $(addprefix $(top_srcdir)/src/include/catalog/,\
pg_attrdef.h pg_constraint.h pg_inherits.h pg_index.h pg_operator.h \
pg_opfamily.h pg_opclass.h pg_am.h pg_amop.h pg_amproc.h \
pg_language.h pg_largeobject_metadata.h pg_largeobject.h pg_aggregate.h \
+ pg_mv_statistic.h \
pg_statistic.h pg_rewrite.h pg_trigger.h pg_event_trigger.h pg_description.h \
pg_cast.h pg_enum.h pg_namespace.h pg_conversion.h pg_depend.h \
pg_database.h pg_db_role_setting.h pg_tablespace.h pg_pltemplate.h \
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 2800f73..d05a716 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -150,6 +150,16 @@ CREATE VIEW pg_indexes AS
LEFT JOIN pg_tablespace T ON (T.oid = I.reltablespace)
WHERE C.relkind IN ('r', 'm') AND I.relkind = 'i';
+CREATE VIEW pg_mv_stats AS
+ SELECT
+ N.nspname AS schemaname,
+ C.relname AS tablename,
+ S.stakeys AS attnums,
+ length(S.stadeps) as depsbytes,
+ pg_mv_stats_dependencies_info(S.stadeps) as depsinfo
+ FROM (pg_mv_statistic S JOIN pg_class C ON (C.oid = S.starelid))
+ LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace);
+
CREATE VIEW pg_stats AS
SELECT
nspname AS schemaname,
diff --git a/src/backend/commands/analyze.c b/src/backend/commands/analyze.c
index 75b45f7..da98d54 100644
--- a/src/backend/commands/analyze.c
+++ b/src/backend/commands/analyze.c
@@ -27,6 +27,7 @@
#include "catalog/indexing.h"
#include "catalog/pg_collation.h"
#include "catalog/pg_inherits_fn.h"
+#include "catalog/pg_mv_statistic.h"
#include "catalog/pg_namespace.h"
#include "commands/dbcommands.h"
#include "commands/tablecmds.h"
@@ -54,7 +55,11 @@
#include "utils/syscache.h"
#include "utils/timestamp.h"
#include "utils/tqual.h"
+#include "utils/fmgroids.h"
+#include "utils/builtins.h"
+#include "utils/mvstats.h"
+#include "access/sysattr.h"
/* Data structure for Algorithm S from Knuth 3.4.2 */
typedef struct
@@ -110,7 +115,6 @@ static void update_attstats(Oid relid, bool inh,
static Datum std_fetch_func(VacAttrStatsP stats, int rownum, bool *isNull);
static Datum ind_fetch_func(VacAttrStatsP stats, int rownum, bool *isNull);
-
/*
* analyze_rel() -- analyze one relation
*/
@@ -472,6 +476,13 @@ do_analyze_rel(Relation onerel, int options, List *va_cols,
* all analyzable columns. We use a lower bound of 100 rows to avoid
* possible overflow in Vitter's algorithm. (Note: that will also be the
* target in the corner case where there are no analyzable columns.)
+ *
+ * FIXME This sample sizing is mostly OK when computing stats for
+ * individual columns, but when computing multi-variate stats
+ * for multivariate stats (histograms, mcv, ...) it's rather
+ * insufficient. For small number of dimensions it works, but
+ * for complex stats it'd be nice use sample proportional to
+ * the table (say, 0.5% - 1%) instead of a fixed size.
*/
targrows = 100;
for (i = 0; i < attr_cnt; i++)
@@ -574,6 +585,9 @@ do_analyze_rel(Relation onerel, int options, List *va_cols,
update_attstats(RelationGetRelid(Irel[ind]), false,
thisdata->attr_cnt, thisdata->vacattrstats);
}
+
+ /* Build multivariate stats (if there are any). */
+ build_mv_stats(onerel, numrows, rows, attr_cnt, vacattrstats);
}
/*
@@ -2825,3 +2839,4 @@ compare_mcvs(const void *a, const void *b)
return da - db;
}
+
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 623e6bf..0df7f03 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -35,6 +35,7 @@
#include "catalog/pg_foreign_table.h"
#include "catalog/pg_inherits.h"
#include "catalog/pg_inherits_fn.h"
+#include "catalog/pg_mv_statistic.h"
#include "catalog/pg_namespace.h"
#include "catalog/pg_opclass.h"
#include "catalog/pg_tablespace.h"
@@ -92,7 +93,7 @@
#include "utils/syscache.h"
#include "utils/tqual.h"
#include "utils/typcache.h"
-
+#include "utils/mvstats.h"
/*
* ON COMMIT action list
@@ -140,8 +141,9 @@ static List *on_commits = NIL;
#define AT_PASS_ADD_COL 5 /* ADD COLUMN */
#define AT_PASS_ADD_INDEX 6 /* ADD indexes */
#define AT_PASS_ADD_CONSTR 7 /* ADD constraints, defaults */
-#define AT_PASS_MISC 8 /* other stuff */
-#define AT_NUM_PASSES 9
+#define AT_PASS_ADD_STATS 8 /* ADD statistics */
+#define AT_PASS_MISC 9 /* other stuff */
+#define AT_NUM_PASSES 10
typedef struct AlteredTableInfo
{
@@ -416,7 +418,8 @@ static void ATExecReplicaIdentity(Relation rel, ReplicaIdentityStmt *stmt, LOCKM
static void ATExecGenericOptions(Relation rel, List *options);
static void ATExecEnableRowSecurity(Relation rel);
static void ATExecDisableRowSecurity(Relation rel);
-
+static void ATExecAddStatistics(AlteredTableInfo *tab, Relation rel,
+ StatisticsDef *def, LOCKMODE lockmode);
static void copy_relation_data(SMgrRelation rel, SMgrRelation dst,
ForkNumber forkNum, char relpersistence);
static const char *storage_name(char c);
@@ -2989,6 +2992,7 @@ AlterTableGetLockLevel(List *cmds)
* updates.
*/
case AT_SetStatistics: /* Uses MVCC in getTableAttrs() */
+ case AT_AddStatistics: /* XXX not sure if the right level */
case AT_ClusterOn: /* Uses MVCC in getIndexes() */
case AT_DropCluster: /* Uses MVCC in getIndexes() */
case AT_SetOptions: /* Uses MVCC in getTableAttrs() */
@@ -3145,6 +3149,7 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd,
pass = AT_PASS_ADD_CONSTR;
break;
case AT_SetStatistics: /* ALTER COLUMN SET STATISTICS */
+ case AT_AddStatistics: /* XXX maybe not the right place */
ATSimpleRecursion(wqueue, rel, cmd, recurse, lockmode);
/* Performs own permission checks */
ATPrepSetStatistics(rel, cmd->name, cmd->def, lockmode);
@@ -3440,6 +3445,9 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,
case AT_SetStatistics: /* ALTER COLUMN SET STATISTICS */
ATExecSetStatistics(rel, cmd->name, cmd->def, lockmode);
break;
+ case AT_AddStatistics: /* ADD STATISTICS */
+ ATExecAddStatistics(tab, rel, (StatisticsDef *) cmd->def, lockmode);
+ break;
case AT_SetOptions: /* ALTER COLUMN SET ( options ) */
ATExecSetOptions(rel, cmd->name, cmd->def, false, lockmode);
break;
@@ -11638,3 +11646,136 @@ RangeVarCallbackForAlterRelation(const RangeVar *rv, Oid relid, Oid oldrelid,
ReleaseSysCache(tuple);
}
+
+/* used for sorting the attnums in ATExecAddStatistics */
+static int compare_int16(const void *a, const void *b)
+{
+ return memcmp(a, b, sizeof(int16));
+}
+
+/*
+ * Implements the ALTER TABLE ... ADD STATISTICS (options) ON (columns).
+ *
+ * The code is an unholy mix of pieces that really belong to other parts
+ * of the source tree.
+ *
+ * FIXME Check that the types are pass-by-value and support sort,
+ * although maybe we can live without the sort (and only build
+ * MCV list / association rules).
+ *
+ * FIXME This should probably check for duplicate stats (i.e. same
+ * keys, same options). Although maybe it's useful to have
+ * multiple stats on the same columns with different options
+ * (say, a detailed MCV-only stats for some queries, histogram
+ * for others, etc.)
+ */
+static void ATExecAddStatistics(AlteredTableInfo *tab, Relation rel,
+ StatisticsDef *def, LOCKMODE lockmode)
+{
+ int i, j;
+ ListCell *l;
+ int16 attnums[INDEX_MAX_KEYS];
+ int numcols = 0;
+
+ HeapTuple htup;
+ Datum values[Natts_pg_mv_statistic];
+ bool nulls[Natts_pg_mv_statistic];
+ int2vector *stakeys;
+ Relation mvstatrel;
+
+ /* by default build everything */
+ bool build_dependencies = true;
+
+ Assert(IsA(def, StatisticsDef));
+
+ /* transform the column names to attnum values */
+
+ foreach(l, def->keys)
+ {
+ char *attname = strVal(lfirst(l));
+ HeapTuple atttuple;
+
+ atttuple = SearchSysCacheAttName(RelationGetRelid(rel), attname);
+
+ if (!HeapTupleIsValid(atttuple))
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_COLUMN),
+ errmsg("column \"%s\" referenced in statistics does not exist",
+ attname)));
+
+ /* more than MVHIST_MAX_DIMENSIONS columns not allowed */
+ if (numcols >= MVSTATS_MAX_DIMENSIONS)
+ ereport(ERROR,
+ (errcode(ERRCODE_TOO_MANY_COLUMNS),
+ errmsg("cannot have more than %d keys in a statistics",
+ MVSTATS_MAX_DIMENSIONS)));
+
+ attnums[numcols] = ((Form_pg_attribute) GETSTRUCT(atttuple))->attnum;
+ ReleaseSysCache(atttuple);
+ numcols++;
+ }
+
+ /*
+ * Check the lower bound (at least 2 columns), the upper bound was
+ * already checked in the loop.
+ */
+ if (numcols < 2)
+ ereport(ERROR,
+ (errcode(ERRCODE_TOO_MANY_COLUMNS),
+ errmsg("multivariate stats require 2 or more columns")));
+
+ /* look for duplicities */
+ for (i = 0; i < numcols; i++)
+ for (j = 0; j < numcols; j++)
+ if ((i != j) && (attnums[i] == attnums[j]))
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_COLUMN),
+ errmsg("duplicate column name in statistics definition")));
+
+ /* parse the statistics options */
+ foreach (l, def->options)
+ {
+ DefElem *opt = (DefElem*)lfirst(l);
+
+ if (strcmp(opt->defname, "dependencies") == 0)
+ build_dependencies = defGetBoolean(opt);
+ else
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("unrecognized STATISTICS option \"%s\"",
+ opt->defname)));
+ }
+
+ /* sort the attnums and build int2vector */
+ qsort(attnums, numcols, sizeof(int16), compare_int16);
+ stakeys = buildint2vector(attnums, numcols);
+
+ /*
+ * Okay, let's create the pg_mv_statistic entry.
+ */
+ memset(values, 0, sizeof(values));
+ memset(nulls, false, sizeof(nulls));
+
+ /* no stats collected yet, so just the keys */
+ values[Anum_pg_mv_statistic_starelid-1] = ObjectIdGetDatum(RelationGetRelid(rel));
+
+ values[Anum_pg_mv_statistic_stakeys -1] = PointerGetDatum(stakeys);
+ values[Anum_pg_mv_statistic_deps_enabled -1] = BoolGetDatum(build_dependencies);
+
+ nulls[Anum_pg_mv_statistic_stadeps -1] = true;
+
+ /* insert the tuple into pg_mv_statistic */
+ mvstatrel = heap_open(MvStatisticRelationId, RowExclusiveLock);
+
+ htup = heap_form_tuple(mvstatrel->rd_att, values, nulls);
+
+ simple_heap_insert(mvstatrel, htup);
+
+ CatalogUpdateIndexes(mvstatrel, htup);
+
+ heap_freetuple(htup);
+
+ heap_close(mvstatrel, RowExclusiveLock);
+
+ return;
+}
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 029761e..df230d6 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -3918,6 +3918,17 @@ _copyAlterPolicyStmt(const AlterPolicyStmt *from)
return newnode;
}
+static StatisticsDef *
+_copyStatisticsDef(const StatisticsDef *from)
+{
+ StatisticsDef *newnode = makeNode(StatisticsDef);
+
+ COPY_NODE_FIELD(keys);
+ COPY_NODE_FIELD(options);
+
+ return newnode;
+}
+
/* ****************************************************************
* pg_list.h copy functions
* ****************************************************************
@@ -4744,7 +4755,9 @@ copyObject(const void *from)
case T_RoleSpec:
retval = _copyRoleSpec(from);
break;
-
+ case T_StatisticsDef:
+ retval = _copyStatisticsDef(from);
+ break;
default:
elog(ERROR, "unrecognized node type: %d", (int) nodeTag(from));
retval = 0; /* keep compiler quiet */
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 82405b9..0346a00 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -367,6 +367,13 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
create_generic_options alter_generic_options
relation_expr_list dostmt_opt_list
+%type <list> OptStatsOptions
+%type <str> stats_options_name
+%type <node> stats_options_arg
+%type <defelt> stats_options_elem
+%type <list> stats_options_list
+
+
%type <list> opt_fdw_options fdw_options
%type <defelt> fdw_option
@@ -486,7 +493,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <keyword> unreserved_keyword type_func_name_keyword
%type <keyword> col_name_keyword reserved_keyword
-%type <node> TableConstraint TableLikeClause
+%type <node> TableConstraint TableLikeClause TableStatistics
%type <ival> TableLikeOptionList TableLikeOption
%type <list> ColQualList
%type <node> ColConstraint ColConstraintElem ConstraintAttr
@@ -2311,6 +2318,14 @@ alter_table_cmd:
n->subtype = AT_DisableRowSecurity;
$$ = (Node *)n;
}
+ /* ALTER TABLE <name> ADD STATISTICS (options) ON (columns) ... */
+ | ADD_P TableStatistics
+ {
+ AlterTableCmd *n = makeNode(AlterTableCmd);
+ n->subtype = AT_AddStatistics;
+ n->def = $2;
+ $$ = (Node *)n;
+ }
| alter_generic_options
{
AlterTableCmd *n = makeNode(AlterTableCmd);
@@ -3381,6 +3396,56 @@ OptConsTableSpace: USING INDEX TABLESPACE name { $$ = $4; }
ExistingIndex: USING INDEX index_name { $$ = $3; }
;
+/*****************************************************************************
+ *
+ * QUERY :
+ * ALTER TABLE relname ADD STATISTICS (columns) WITH (options)
+ *
+ *****************************************************************************/
+
+TableStatistics:
+ STATISTICS OptStatsOptions ON '(' columnList ')'
+ {
+ StatisticsDef *n = makeNode(StatisticsDef);
+ n->keys = $5;
+ n->options = $2;
+ $$ = (Node *) n;
+ }
+ ;
+
+OptStatsOptions:
+ '(' stats_options_list ')' { $$ = $2; }
+ | /*EMPTY*/ { $$ = NIL; }
+ ;
+
+stats_options_list:
+ stats_options_elem
+ {
+ $$ = list_make1($1);
+ }
+ | stats_options_list ',' stats_options_elem
+ {
+ $$ = lappend($1, $3);
+ }
+ ;
+
+stats_options_elem:
+ stats_options_name stats_options_arg
+ {
+ $$ = makeDefElem($1, $2);
+ }
+ ;
+
+stats_options_name:
+ NonReservedWord { $$ = $1; }
+ ;
+
+stats_options_arg:
+ opt_boolean_or_string { $$ = (Node *) makeString($1); }
+ | NumericOnly { $$ = (Node *) $1; }
+ | /* EMPTY */ { $$ = NULL; }
+ ;
+
/*****************************************************************************
*
diff --git a/src/backend/utils/Makefile b/src/backend/utils/Makefile
index 8374533..eba0352 100644
--- a/src/backend/utils/Makefile
+++ b/src/backend/utils/Makefile
@@ -9,7 +9,7 @@ top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
OBJS = fmgrtab.o
-SUBDIRS = adt cache error fmgr hash init mb misc mmgr resowner sort time
+SUBDIRS = adt cache error fmgr hash init mb misc mmgr mvstats resowner sort time
# location of Catalog.pm
catalogdir = $(top_srcdir)/src/backend/catalog
diff --git a/src/backend/utils/cache/syscache.c b/src/backend/utils/cache/syscache.c
index bd27168..f61ef7e 100644
--- a/src/backend/utils/cache/syscache.c
+++ b/src/backend/utils/cache/syscache.c
@@ -43,6 +43,7 @@
#include "catalog/pg_foreign_server.h"
#include "catalog/pg_foreign_table.h"
#include "catalog/pg_language.h"
+#include "catalog/pg_mv_statistic.h"
#include "catalog/pg_namespace.h"
#include "catalog/pg_opclass.h"
#include "catalog/pg_operator.h"
@@ -499,6 +500,17 @@ static const struct cachedesc cacheinfo[] = {
},
4
},
+ {MvStatisticRelationId, /* MVSTATOID */
+ MvStatisticOidIndexId,
+ 1,
+ {
+ ObjectIdAttributeNumber,
+ 0,
+ 0,
+ 0
+ },
+ 128
+ },
{NamespaceRelationId, /* NAMESPACENAME */
NamespaceNameIndexId,
1,
diff --git a/src/backend/utils/mvstats/Makefile b/src/backend/utils/mvstats/Makefile
new file mode 100644
index 0000000..099f1ed
--- /dev/null
+++ b/src/backend/utils/mvstats/Makefile
@@ -0,0 +1,17 @@
+#-------------------------------------------------------------------------
+#
+# Makefile--
+# Makefile for utils/mvstats
+#
+# IDENTIFICATION
+# src/backend/utils/mvstats/Makefile
+#
+#-------------------------------------------------------------------------
+
+subdir = src/backend/utils/mvstats
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+
+OBJS = common.o dependencies.o
+
+include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/utils/mvstats/common.c b/src/backend/utils/mvstats/common.c
new file mode 100644
index 0000000..36757d5
--- /dev/null
+++ b/src/backend/utils/mvstats/common.c
@@ -0,0 +1,272 @@
+/*-------------------------------------------------------------------------
+ *
+ * common.c
+ * POSTGRES multivariate statistics
+ *
+ *
+ * Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/utils/mvstats/common.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "common.h"
+
+/*
+ * Compute requested multivariate stats, using the rows sampled for the
+ * plain (single-column) stats.
+ *
+ * This fetches a list of stats from pg_mv_statistic, computes the stats
+ * and serializes them back into the catalog (as bytea values).
+ */
+void
+build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
+ int natts, VacAttrStats **vacattrstats)
+{
+ int i;
+ MVStats mvstats;
+ int nmvstats;
+
+ /*
+ * Fetch defined MV groups from pg_mv_statistic, and then compute
+ * the MV statistics (histograms for now).
+ */
+ mvstats = list_mv_stats(RelationGetRelid(onerel), &nmvstats, false);
+
+ for (i = 0; i < nmvstats; i++)
+ {
+ MVDependencies deps = NULL;
+
+ /* int2 vector of attnums the stats should be computed on */
+ int2vector * attrs = mvstats[i].stakeys;
+
+ /* check allowed number of dimensions */
+ Assert((attrs->dim1 >= 2) && (attrs->dim1 <= MVSTATS_MAX_DIMENSIONS));
+
+ /*
+ * Analyze functional dependencies of columns.
+ */
+ deps = build_mv_dependencies(numrows, rows, attrs, natts, vacattrstats);
+
+ /* store the histogram / MCV list in the catalog */
+ update_mv_stats(mvstats[i].mvoid, deps);
+ }
+}
+
+/*
+ * Lookup the VacAttrStats info for the selected columns, with indexes
+ * matching the attrs vector (to make it easy to work with when
+ * computing multivariate stats).
+ */
+VacAttrStats **
+lookup_var_attr_stats(int2vector *attrs, int natts, VacAttrStats **vacattrstats)
+{
+ int i, j;
+ int numattrs = attrs->dim1;
+ VacAttrStats **stats = (VacAttrStats**)palloc0(numattrs * sizeof(VacAttrStats*));
+
+ /* lookup VacAttrStats info for the requested columns (same attnum) */
+ for (i = 0; i < numattrs; i++)
+ {
+ stats[i] = NULL;
+ for (j = 0; j < natts; j++)
+ {
+ if (attrs->values[i] == vacattrstats[j]->tupattnum)
+ {
+ stats[i] = vacattrstats[j];
+ break;
+ }
+ }
+
+ /*
+ * Check that we found the info, that the attnum matches and
+ * that there's the requested 'lt' operator and that the type
+ * is 'passed-by-value'.
+ */
+ Assert(stats[i] != NULL);
+ Assert(stats[i]->tupattnum == attrs->values[i]);
+
+ /* FIXME This is rather ugly way to check for 'ltopr' (which
+ * is defined for 'scalar' attributes).
+ */
+ Assert(((StdAnalyzeData *)stats[i]->extra_data)->ltopr != InvalidOid);
+ }
+
+ return stats;
+}
+
+/*
+ * Fetch list of MV stats defined on a table, without the actual data
+ * for histograms, MCV lists etc.
+ */
+MVStats
+list_mv_stats(Oid relid, int *nstats, bool built_only)
+{
+ Relation indrel;
+ SysScanDesc indscan;
+ ScanKeyData skey;
+ HeapTuple htup;
+ MVStats result;
+
+ /* start with 16 items, that should be enough for most cases */
+ int maxitems = 16;
+ result = (MVStats)palloc0(sizeof(MVStatsData) * maxitems);
+ *nstats = 0;
+
+ /* Prepare to scan pg_mv_statistic for entries having indrelid = this rel. */
+ ScanKeyInit(&skey,
+ Anum_pg_mv_statistic_starelid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(relid));
+
+ indrel = heap_open(MvStatisticRelationId, AccessShareLock);
+ indscan = systable_beginscan(indrel, MvStatisticRelidIndexId, true,
+ NULL, 1, &skey);
+
+ while (HeapTupleIsValid(htup = systable_getnext(indscan)))
+ {
+ Form_pg_mv_statistic stats = (Form_pg_mv_statistic) GETSTRUCT(htup);
+
+ /*
+ * Skip statistics that were not computed yet (if only stats
+ * that were already built were requested)
+ */
+ if (built_only && (! stats->deps_built))
+ continue;
+
+ /* double the array size if needed */
+ if (*nstats == maxitems)
+ {
+ maxitems *= 2;
+ result = (MVStats)repalloc(result, sizeof(MVStatsData) * maxitems);
+ }
+
+ result[*nstats].mvoid = HeapTupleGetOid(htup);
+ result[*nstats].stakeys = buildint2vector(stats->stakeys.values, stats->stakeys.dim1);
+ result[*nstats].deps_built = stats->deps_built;
+ *nstats += 1;
+ }
+
+ systable_endscan(indscan);
+
+ heap_close(indrel, AccessShareLock);
+
+ /* TODO maybe save the list into relcache, as in RelationGetIndexList
+ * (which was used as an inspiration of this one)?. */
+
+ return result;
+}
+
+void
+update_mv_stats(Oid mvoid, MVDependencies dependencies)
+{
+ HeapTuple stup,
+ oldtup;
+ Datum values[Natts_pg_mv_statistic];
+ bool nulls[Natts_pg_mv_statistic];
+ bool replaces[Natts_pg_mv_statistic];
+
+ Relation sd = heap_open(MvStatisticRelationId, RowExclusiveLock);
+
+ memset(nulls, 1, Natts_pg_mv_statistic * sizeof(bool));
+ memset(replaces, 0, Natts_pg_mv_statistic * sizeof(bool));
+ memset(values, 0, Natts_pg_mv_statistic * sizeof(Datum));
+
+ /*
+ * Construct a new pg_mv_statistic tuple - replace only the histogram
+ * and MCV list, depending whether it actually was computed.
+ */
+ if (dependencies != NULL)
+ {
+ nulls[Anum_pg_mv_statistic_stadeps -1] = false;
+ values[Anum_pg_mv_statistic_stadeps - 1]
+ = PointerGetDatum(serialize_mv_dependencies(dependencies));
+ }
+
+ /* always replace the value (either by bytea or NULL) */
+ replaces[Anum_pg_mv_statistic_stadeps -1] = true;
+
+ /* always change the availability flags */
+ nulls[Anum_pg_mv_statistic_deps_built -1] = false;
+
+ replaces[Anum_pg_mv_statistic_deps_built-1] = true;
+
+ values[Anum_pg_mv_statistic_deps_built-1] = BoolGetDatum(dependencies != NULL);
+
+ /* Is there already a pg_mv_statistic tuple for this attribute? */
+ oldtup = SearchSysCache1(MVSTATOID,
+ ObjectIdGetDatum(mvoid));
+
+ if (HeapTupleIsValid(oldtup))
+ {
+ /* Yes, replace it */
+ stup = heap_modify_tuple(oldtup,
+ RelationGetDescr(sd),
+ values,
+ nulls,
+ replaces);
+ ReleaseSysCache(oldtup);
+ simple_heap_update(sd, &stup->t_self, stup);
+ }
+ else
+ elog(ERROR, "invalid pg_mv_statistic record (oid=%d)", mvoid);
+
+ /* update indexes too */
+ CatalogUpdateIndexes(sd, stup);
+
+ heap_freetuple(stup);
+
+ heap_close(sd, RowExclusiveLock);
+}
+
+/* multi-variate stats comparator */
+
+/*
+ * qsort_arg comparator for sorting Datums (MV stats)
+ *
+ * This does not maintain the tupnoLink array.
+ */
+int
+compare_scalars_simple(const void *a, const void *b, void *arg)
+{
+ Datum da = *(Datum*)a;
+ Datum db = *(Datum*)b;
+ SortSupport ssup= (SortSupport) arg;
+
+ return ApplySortComparator(da, false, db, false, ssup);
+}
+
+/*
+ * qsort_arg comparator for sorting data when partitioning a MV bucket
+ */
+int
+compare_scalars_partition(const void *a, const void *b, void *arg)
+{
+ Datum da = ((ScalarItem*)a)->value;
+ Datum db = ((ScalarItem*)b)->value;
+ SortSupport ssup= (SortSupport) arg;
+
+ return ApplySortComparator(da, false, db, false, ssup);
+}
+
+/*
+ * qsort_arg comparator for sorting Datum[] (row of Datums) when
+ * counting distinct values.
+ */
+int
+compare_scalars_memcmp(const void *a, const void *b, void *arg)
+{
+ Size len = *(Size*)arg;
+
+ return memcmp(a, b, len);
+}
+
+int
+compare_scalars_memcmp_2(const void *a, const void *b)
+{
+ return memcmp(a, b, sizeof(Datum));
+}
diff --git a/src/backend/utils/mvstats/common.h b/src/backend/utils/mvstats/common.h
new file mode 100644
index 0000000..f511c4e
--- /dev/null
+++ b/src/backend/utils/mvstats/common.h
@@ -0,0 +1,70 @@
+/*-------------------------------------------------------------------------
+ *
+ * common.h
+ * POSTGRES multivariate statistics
+ *
+ *
+ * Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/utils/mvstats/common.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/tuptoaster.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_collation.h"
+#include "catalog/pg_mv_statistic.h"
+#include "foreign/fdwapi.h"
+#include "postmaster/autovacuum.h"
+#include "storage/lmgr.h"
+#include "utils/datum.h"
+#include "utils/sortsupport.h"
+#include "utils/syscache.h"
+#include "utils/fmgroids.h"
+#include "utils/builtins.h"
+#include "access/sysattr.h"
+
+#include "utils/mvstats.h"
+
+/* FIXME private structure copied from analyze.c */
+
+typedef struct
+{
+ Oid eqopr; /* '=' operator for datatype, if any */
+ Oid eqfunc; /* and associated function */
+ Oid ltopr; /* '<' operator for datatype, if any */
+} StdAnalyzeData;
+
+typedef struct
+{
+ Datum value; /* a data value */
+ int tupno; /* position index for tuple it came from */
+} ScalarItem;
+
+typedef struct
+{
+ int count; /* # of duplicates */
+ int first; /* values[] index of first occurrence */
+} ScalarMCVItem;
+
+typedef struct
+{
+ SortSupport ssup;
+ int *tupnoLink;
+} CompareScalarsContext;
+
+
+VacAttrStats ** lookup_var_attr_stats(int2vector *attrs,
+ int natts, VacAttrStats **vacattrstats);
+
+/* comparators, used when constructing multivariate stats */
+int compare_scalars_simple(const void *a, const void *b, void *arg);
+int compare_scalars_partition(const void *a, const void *b, void *arg);
+int compare_scalars_memcmp(const void *a, const void *b, void *arg);
+int compare_scalars_memcmp_2(const void *a, const void *b);
diff --git a/src/backend/utils/mvstats/dependencies.c b/src/backend/utils/mvstats/dependencies.c
new file mode 100644
index 0000000..b900efd
--- /dev/null
+++ b/src/backend/utils/mvstats/dependencies.c
@@ -0,0 +1,554 @@
+/*-------------------------------------------------------------------------
+ *
+ * dependencies.c
+ * POSTGRES multivariate functional dependencies
+ *
+ *
+ * Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/utils/mvstats/dependencies.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "common.h"
+
+/*
+ * Mine functional dependencies between columns, in the form (A => B),
+ * meaning that a value in column 'A' determines value in 'B'. A simple
+ * artificial example may be a table created like this
+ *
+ * CREATE TABLE deptest (a INT, b INT)
+ * AS SELECT i, i/10 FROM generate_series(1,100000) s(i);
+ *
+ * Clearly, once we know the value for 'A' we can easily determine the
+ * value of 'B' by dividing (A/10). A more practical example may be
+ * addresses, where (ZIP code => city name), i.e. once we know the ZIP,
+ * we probably know which city it belongs to. Larger cities usually have
+ * multiple ZIP codes, so the dependency can't be reversed.
+ *
+ * Functional dependencies are a concept well described in relational
+ * theory, especially in definition of normalization and "normal forms".
+ * Wikipedia has a nice definition of a functional dependency [1]:
+ *
+ * In a given table, an attribute Y is said to have a functional
+ * dependency on a set of attributes X (written X -> Y) if and only
+ * if each X value is associated with precisely one Y value. For
+ * example, in an "Employee" table that includes the attributes
+ * "Employee ID" and "Employee Date of Birth", the functional
+ * dependency {Employee ID} -> {Employee Date of Birth} would hold.
+ * It follows from the previous two sentences that each {Employee ID}
+ * is associated with precisely one {Employee Date of Birth}.
+ *
+ * [1] http://en.wikipedia.org/wiki/Database_normalization
+ *
+ * Most datasets might be normalized not to contain any such functional
+ * dependencies, but sometimes it's not practical. In some cases it's
+ * actually a conscious choice to model the dataset in denormalized way,
+ * either because of performance or to make querying easier.
+ *
+ * The current implementation supports only dependencies between two
+ * columns, but this is merely a simplification of the initial patch.
+ * It's certainly useful to mine for dependencies involving multiple
+ * columns on the 'left' side, i.e. a condition for the dependency.
+ * That is dependencies [A,B] => C and so on.
+ *
+ * Handling multiple columns on the right side is not necessary, as such
+ * dependencies may be decomposed into a set of dependencies with
+ * the same meaning, one for each column on the right side. For example
+ *
+ * A => [B,C]
+ *
+ * is exactly the same as
+ *
+ * (A => B) & (A => C).
+ *
+ * Of course, storing (A => [B, C]) may be more efficient thant storing
+ * the two dependencies (A => B) and (A => C) separately.
+ *
+ *
+ * Dependency mining (ANALYZE)
+ * ---------------------------
+ *
+ * FIXME Add more details about how build_mv_dependencies() works
+ * (minimum group size, supporting/contradicting etc.).
+ *
+ * Real-world datasets are imperfect - there may be errors (e.g. due to
+ * data-entry mistakes), or factually correct records, yet contradicting
+ * the dependency (e.g. when a city splits into two, but both keep the
+ * same ZIP code). A strict ANALYZE implementation (where the functional
+ * dependencies are identified) would ignore dependencies on such noisy
+ * data, making the approach unusable in practice.
+ *
+ * The proposed implementation attempts to handle such noisy cases
+ * gracefully, by tolerating small number of contradicting cases.
+ *
+ * In the future this might also perform some sort of test and decide
+ * whether it's worth building any other kind of multivariate stats,
+ * or whether the dependencies sufficiently describe the data. Or at
+ * least not build the MCV list / histogram on the implied columns.
+ * Such reduction would however make the 'verification' (see the next
+ * section) impossible.
+ *
+ *
+ * Clause reduction (planner/optimizer)
+ * ------------------------------------
+ *
+ * FIXME Explain how reduction works.
+ *
+ * The problem with the reduction is that the query may use conditions
+ * that are not redundant, but in fact contradictory - e.g. the user
+ * may search for a ZIP code and a city name not matching the ZIP code.
+ *
+ * In such cases, the condition on the city name is not actually
+ * redundant, but actually contradictory (making the result empty), and
+ * removing it while estimating the cardinality will make the estimate
+ * worse.
+ *
+ * The current estimation assuming independence (and multiplying the
+ * selectivities) works better in this case, but only by utter luck.
+ *
+ * In some cases this might be verified using the other multivariate
+ * statistics - MCV lists and histograms. For MCV lists the verification
+ * might be very simple - peek into the list if there are any items
+ * matching the clause on the 'A' column (e.g. ZIP code), and if such
+ * item is found, check that the 'B' column matches the other clause.
+ * If it does not, the clauses are contradictory. We can't really say
+ * if such item was not found, except maybe restricting the selectivity
+ * using the MCV data (e.g. using min/max selectivity, or something).
+ *
+ * With histograms, it might work similarly - we can't check the values
+ * directly (because histograms use buckets, unlike MCV lists, storing
+ * the actual values). So we can only observe the buckets matching the
+ * clauses - if those buckets have very low frequency, it probably means
+ * the two clauses are incompatible.
+ *
+ * It's unclear what 'low frequency' is, but if one of the clauses is
+ * implied (automatically true because of the other clause), then
+ *
+ * selectivity[clause(A)] = selectivity[clause(A) & clause(B)]
+ *
+ * So we might compute selectivity of the first clause (on the column
+ * A in dependency [A=>B]) - for example using regular statistics.
+ * And then check if the selectivity computed from the histogram is
+ * about the same (or significantly lower).
+ *
+ * The problem is that histograms work well only when the data ordering
+ * matches the natural meaning. For values that serve as labels - like
+ * city names or ZIP codes, or even generated IDs, histograms really
+ * don't work all that well. For example sorting cities by name won't
+ * match the sorting of ZIP codes, rendering the histogram unusable.
+ *
+ * The MCV are probably going to work much better, because they don't
+ * really assume any sort of ordering. And it's probably more appropriate
+ * for the label-like data.
+ *
+ * TODO Support dependencies with multiple columns on left/right.
+ *
+ * TODO Investigate using histogram and MCV list to confirm the
+ * functional dependencies.
+ *
+ * TODO Investigate statistical testing of the distribution (to decide
+ * whether it makes sense to build the histogram/MCV list).
+ *
+ * TODO Using a min/max of selectivities would probably make more sense
+ * for the associated columns.
+ *
+ * TODO Consider eliminating the implied columns from the histogram and
+ * MCV lists (but maybe that's not a good idea).
+ *
+ * FIXME Not sure if this handles NULL values properly (not sure how to
+ * do that). We assume that NULL means 0 for now, handling it just
+ * like any other value.
+ */
+MVDependencies
+build_mv_dependencies(int numrows, HeapTuple *rows, int2vector *attrs,
+ int natts, VacAttrStats **vacattrstats)
+{
+ int i;
+ bool isNull;
+ Size len = 2 * sizeof(Datum); /* only simple associations a => b */
+ int numattrs = attrs->dim1;
+
+ /* result */
+ int ndeps = 0;
+ MVDependencies dependencies = NULL;
+
+ /* TODO Maybe this should be somehow related to the number of
+ * distinct columns in the two columns we're currently analyzing.
+ * Assuming the distribution is uniform, we should expected to
+ * observe in the sample - we can then use the average group
+ * size as a threshold. That seems better than a static approach.
+ */
+ int min_group_size = 3;
+
+ /* dimension indexes we'll check for associations [a => b] */
+ int dima, dimb;
+
+ /* info for the interesting attributes only
+ *
+ * TODO Compute this only once and pass it to all the methods
+ * that need it.
+ */
+ VacAttrStats **stats = lookup_var_attr_stats(attrs, natts, vacattrstats);
+
+ /* We'll reuse the same array for all the combinations */
+ Datum * values = (Datum*)palloc0(numrows * 2 * sizeof(Datum));
+
+ Assert(numattrs >= 2);
+
+ /*
+ * Evaluate all possible combinations of [A => B], using a simple algorithm:
+ *
+ * (a) sort the data by [A,B]
+ * (b) split the data into groups by A (new group whenever a value changes)
+ * (c) count different values in the B column (again, value changes)
+ *
+ * TODO It should be rather simple to merge [A => B] and [A => C] into
+ * [A => B,C]. Just keep A constant, collect all the "implied" columns
+ * and you're done.
+ */
+ for (dima = 0; dima < numattrs; dima++)
+ {
+ for (dimb = 0; dimb < numattrs; dimb++)
+ {
+ Datum val_a, val_b;
+
+ /* number of groups supporting / contradicting the dependency */
+ int n_supporting = 0;
+ int n_contradicting = 0;
+
+ /* counters valid within a group */
+ int group_size = 0;
+ int n_violations = 0;
+
+ int n_supporting_rows = 0;
+ int n_contradicting_rows = 0;
+
+ /* make sure the columns are different (A => A) */
+ if (dima == dimb)
+ continue;
+
+ /* accumulate all the data for both columns into an array and sort it */
+ for (i = 0; i < numrows; i++)
+ {
+ values[i*2] = heap_getattr(rows[i], attrs->values[dima], stats[dima]->tupDesc, &isNull);
+ values[i*2+1] = heap_getattr(rows[i], attrs->values[dimb], stats[dimb]->tupDesc, &isNull);
+ }
+
+ qsort_arg((void *) values, numrows, sizeof(Datum) * 2, compare_scalars_memcmp, &len);
+
+ /*
+ * Walk through the array, split it into rows according to
+ * the A value, and count distinct values in the other one.
+ * If there's a single B value for the whole group, we count
+ * it as supporting the association, otherwise we count it
+ * as contradicting.
+ *
+ * Furthermore we require a group to have at least a certain
+ * number of rows to be considered useful for supporting the
+ * dependency. But when it's contradicting, use it always useful.
+ */
+
+ /* start with values from the first row */
+ val_a = values[0];
+ val_b = values[1];
+ group_size = 1;
+
+ for (i = 1; i < numrows; i++)
+ {
+ if (values[2*i] != val_a) /* end of the group */
+ {
+ /*
+ * If there are no contradicting rows, count it as
+ * supporting (otherwise contradicting), but only if
+ * the group is large enough.
+ *
+ * The requirement of a minimum group size makes it
+ * impossible to identify [unique,unique] cases, but
+ * that's probably a different case. This is more
+ * about [zip => city] associations etc.
+ */
+ n_supporting += ((n_violations == 0) && (group_size >= min_group_size)) ? 1 : 0;
+ n_contradicting += (n_violations != 0) ? 1 : 0;
+
+ n_supporting_rows += ((n_violations == 0) && (group_size >= min_group_size)) ? group_size : 0;
+ n_contradicting_rows += (n_violations > 0) ? group_size : 0;
+
+ /* current values start a new group */
+ val_a = values[2*i];
+ val_b = values[2*i+1];
+ n_violations = 0;
+ group_size = 1;
+ }
+ else
+ {
+ if (values[2*i+1] != val_b) /* mismatch of a B value is contradicting */
+ {
+ val_b = values[2*i+1];
+ n_violations += 1;
+ }
+
+ group_size += 1;
+ }
+ }
+
+ /* handle the last group */
+ n_supporting += ((n_violations == 0) && (group_size >= min_group_size)) ? 1 : 0;
+ n_contradicting += (n_violations != 0) ? 1 : 0;
+ n_supporting_rows += ((n_violations == 0) && (group_size >= min_group_size)) ? group_size : 0;
+ n_contradicting_rows += (n_violations > 0) ? group_size : 0;
+
+ /*
+ * See if the number of rows supporting the association is at least
+ * 10x the number of rows violating the hypothetical dependency.
+ *
+ * TODO This is rather arbitrary limit - I guess it's possible to do
+ * some math to come up with a better rule (e.g. testing a hypothesis
+ * 'this is due to randomness'). We can create a contingency table
+ * from the values and use it for testing. Possibly only when
+ * there are no contradicting rows?
+ *
+ * TODO Also, if (a => b) and (b => a) at the same time, it pretty much
+ * means the columns have the same values (or one is a 'label'),
+ * making the conditions rather redundant. Although it's possible
+ * that the query uses incompatible combination of values.
+ */
+ if (n_supporting_rows > (n_contradicting_rows * 10))
+ {
+ if (dependencies == NULL)
+ {
+ dependencies = (MVDependencies)palloc0(sizeof(MVDependenciesData));
+ dependencies->magic = MVSTAT_DEPS_MAGIC;
+ }
+ else
+ dependencies = repalloc(dependencies, offsetof(MVDependenciesData, deps)
+ + sizeof(MVDependency) * (dependencies->ndeps + 1));
+
+ /* update the */
+ dependencies->deps[ndeps] = (MVDependency)palloc0(sizeof(MVDependencyData));
+ dependencies->deps[ndeps]->a = attrs->values[dima];
+ dependencies->deps[ndeps]->b = attrs->values[dimb];
+
+ dependencies->ndeps = (++ndeps);
+ }
+ }
+ }
+
+ pfree(values);
+
+ return dependencies;
+}
+
+/*
+ * Store the dependencies into a bytea, so that it can be stored in the
+ * pg_mv_statistic catalog.
+ *
+ * Currently this only supports simple two-column rules, and stores them
+ * as a sequence of attnum pairs. In the future, this needs to be made
+ * more complex to support multiple columns on both sides of the
+ * implication (using AND on left, OR on right).
+ */
+bytea *
+serialize_mv_dependencies(MVDependencies dependencies)
+{
+ int i;
+
+ /* we need to store ndeps, and each needs 2 * int16 */
+ Size len = VARHDRSZ + offsetof(MVDependenciesData, deps)
+ + dependencies->ndeps * (sizeof(int16) * 2);
+
+ bytea * output = (bytea*)palloc0(len);
+
+ char * tmp = VARDATA(output);
+
+ SET_VARSIZE(output, len);
+
+ /* first, store the number of dimensions / items */
+ memcpy(tmp, dependencies, offsetof(MVDependenciesData, deps));
+ tmp += offsetof(MVDependenciesData, deps);
+
+ /* walk through the dependencies and copy both columns into the bytea */
+ for (i = 0; i < dependencies->ndeps; i++)
+ {
+ memcpy(tmp, &(dependencies->deps[i]->a), sizeof(int16));
+ tmp += sizeof(int16);
+
+ memcpy(tmp, &(dependencies->deps[i]->b), sizeof(int16));
+ tmp += sizeof(int16);
+ }
+
+ return output;
+}
+
+/*
+ * Reads serialized dependencies into MVDependencies structure.
+ */
+MVDependencies
+deserialize_mv_dependencies(bytea * data)
+{
+ int i;
+ Size expected_size;
+ MVDependencies dependencies;
+ char *tmp;
+
+ if (data == NULL)
+ return NULL;
+
+ if (VARSIZE_ANY_EXHDR(data) < offsetof(MVDependenciesData,deps))
+ elog(ERROR, "invalid MVDependencies size %ld (expected at least %ld)",
+ VARSIZE_ANY_EXHDR(data), offsetof(MVDependenciesData,deps));
+
+ /* read the MVDependencies header */
+ dependencies = (MVDependencies)palloc0(sizeof(MVDependenciesData));
+
+ /* initialize pointer to the data part (skip the varlena header) */
+ tmp = VARDATA(data);
+
+ /* get the header and perform basic sanity checks */
+ memcpy(dependencies, tmp, offsetof(MVDependenciesData, deps));
+ tmp += offsetof(MVDependenciesData, deps);
+
+ if (dependencies->magic != MVSTAT_DEPS_MAGIC)
+ {
+ pfree(dependencies);
+ elog(WARNING, "not a MV Dependencies (magic number mismatch)");
+ return NULL;
+ }
+
+ Assert(dependencies->ndeps > 0);
+
+ /* what bytea size do we expect for those parameters */
+ expected_size = offsetof(MVDependenciesData,deps) +
+ dependencies->ndeps * sizeof(int16) * 2;
+
+ if (VARSIZE_ANY_EXHDR(data) != expected_size)
+ elog(ERROR, "invalid dependencies size %ld (expected %ld)",
+ VARSIZE_ANY_EXHDR(data), expected_size);
+
+ /* allocate space for the MCV items */
+ dependencies = repalloc(dependencies, offsetof(MVDependenciesData,deps)
+ + (dependencies->ndeps * sizeof(MVDependency)));
+
+ for (i = 0; i < dependencies->ndeps; i++)
+ {
+ dependencies->deps[i] = (MVDependency)palloc0(sizeof(MVDependencyData));
+
+ memcpy(&(dependencies->deps[i]->a), tmp, sizeof(int16));
+ tmp += sizeof(int16);
+
+ memcpy(&(dependencies->deps[i]->b), tmp, sizeof(int16));
+ tmp += sizeof(int16);
+ }
+
+ return dependencies;
+}
+
+/* print some basic info about dependencies (number of dependencies) */
+Datum
+pg_mv_stats_dependencies_info(PG_FUNCTION_ARGS)
+{
+ bytea *data = PG_GETARG_BYTEA_P(0);
+ char *result;
+
+ MVDependencies dependencies = deserialize_mv_dependencies(data);
+
+ if (dependencies == NULL)
+ PG_RETURN_NULL();
+
+ result = palloc0(128);
+ snprintf(result, 128, "dependencies=%d", dependencies->ndeps);
+
+ /* FIXME free the deserialized data (pfree is not enough) */
+
+ PG_RETURN_TEXT_P(cstring_to_text(result));
+}
+
+/* print the dependencies
+ *
+ * TODO Would be nice if this knew the actual column names (instead of
+ * the attnums).
+ *
+ * FIXME This is really ugly and does not really check the lengths and
+ * strcpy/snprintf return values properly. Needs to be fixed.
+ */
+Datum
+pg_mv_stats_dependencies_show(PG_FUNCTION_ARGS)
+{
+ int i = 0;
+ bytea *data = PG_GETARG_BYTEA_P(0);
+ char *result = NULL;
+ int len = 0;
+
+ MVDependencies dependencies = deserialize_mv_dependencies(data);
+
+ if (dependencies == NULL)
+ PG_RETURN_NULL();
+
+ for (i = 0; i < dependencies->ndeps; i++)
+ {
+ MVDependency dependency = dependencies->deps[i];
+ char buffer[128];
+
+ int tmp = snprintf(buffer, 128, "%s%d => %d",
+ ((i == 0) ? "" : ", "), dependency->a, dependency->b);
+
+ if (tmp < 127)
+ {
+ if (result == NULL)
+ result = palloc0(len + tmp + 1);
+ else
+ result = repalloc(result, len + tmp + 1);
+
+ strcpy(result + len, buffer);
+ len += tmp;
+ }
+ }
+
+ PG_RETURN_TEXT_P(cstring_to_text(result));
+}
+
+bytea *
+fetch_mv_dependencies(Oid mvoid)
+{
+ Relation indrel;
+ SysScanDesc indscan;
+ ScanKeyData skey;
+ HeapTuple htup;
+ bytea *stadeps = NULL;
+
+ /* Prepare to scan pg_mv_statistic for entries having indrelid = this rel. */
+ ScanKeyInit(&skey,
+ ObjectIdAttributeNumber,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(mvoid));
+
+ indrel = heap_open(MvStatisticRelationId, AccessShareLock);
+ indscan = systable_beginscan(indrel, MvStatisticOidIndexId, true,
+ NULL, 1, &skey);
+
+ while (HeapTupleIsValid(htup = systable_getnext(indscan)))
+ {
+ bool isnull = false;
+ Datum deps = SysCacheGetAttr(MVSTATOID, htup,
+ Anum_pg_mv_statistic_stadeps, &isnull);
+
+ Assert(!isnull);
+
+ stadeps = DatumGetByteaP(deps);
+
+ break;
+ }
+
+ systable_endscan(indscan);
+
+ heap_close(indrel, AccessShareLock);
+
+ /* TODO maybe save the list into relcache, as in RelationGetIndexList
+ * (which was used as an inspiration of this one)?. */
+
+ return stadeps;
+}
diff --git a/src/include/catalog/indexing.h b/src/include/catalog/indexing.h
index a680229..f69eb7c 100644
--- a/src/include/catalog/indexing.h
+++ b/src/include/catalog/indexing.h
@@ -173,6 +173,11 @@ DECLARE_UNIQUE_INDEX(pg_largeobject_loid_pn_index, 2683, on pg_largeobject using
DECLARE_UNIQUE_INDEX(pg_largeobject_metadata_oid_index, 2996, on pg_largeobject_metadata using btree(oid oid_ops));
#define LargeObjectMetadataOidIndexId 2996
+DECLARE_UNIQUE_INDEX(pg_mv_statistic_oid_index, 3286, on pg_mv_statistic using btree(oid oid_ops));
+#define MvStatisticOidIndexId 3286
+DECLARE_INDEX(pg_mv_statistic_relid_index, 3287, on pg_mv_statistic using btree(starelid oid_ops));
+#define MvStatisticRelidIndexId 3287
+
DECLARE_UNIQUE_INDEX(pg_namespace_nspname_index, 2684, on pg_namespace using btree(nspname name_ops));
#define NamespaceNameIndexId 2684
DECLARE_UNIQUE_INDEX(pg_namespace_oid_index, 2685, on pg_namespace using btree(oid oid_ops));
diff --git a/src/include/catalog/pg_mv_statistic.h b/src/include/catalog/pg_mv_statistic.h
new file mode 100644
index 0000000..76b7db7
--- /dev/null
+++ b/src/include/catalog/pg_mv_statistic.h
@@ -0,0 +1,69 @@
+/*-------------------------------------------------------------------------
+ *
+ * pg_mv_statistic.h
+ * definition of the system "multivariate statistic" relation (pg_mv_statistic)
+ * along with the relation's initial contents.
+ *
+ *
+ * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/catalog/pg_mv_statistic.h
+ *
+ * NOTES
+ * the genbki.pl script reads this file and generates .bki
+ * information from the DATA() statements.
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef PG_MV_STATISTIC_H
+#define PG_MV_STATISTIC_H
+
+#include "catalog/genbki.h"
+
+/* ----------------
+ * pg_mv_statistic definition. cpp turns this into
+ * typedef struct FormData_pg_mv_statistic
+ * ----------------
+ */
+#define MvStatisticRelationId 3281
+
+CATALOG(pg_mv_statistic,3281)
+{
+ /* These fields form the unique key for the entry: */
+ Oid starelid; /* relation containing attributes */
+
+ /* statistics requested to build */
+ bool deps_enabled; /* analyze dependencies? */
+
+ /* statistics that are available (if requested) */
+ bool deps_built; /* dependencies were built */
+
+ /* variable-length fields start here, but we allow direct access to stakeys */
+ int2vector stakeys; /* array of column keys */
+
+#ifdef CATALOG_VARLEN
+ bytea stadeps; /* dependencies (serialized) */
+#endif
+
+} FormData_pg_mv_statistic;
+
+/* ----------------
+ * Form_pg_mv_statistic corresponds to a pointer to a tuple with
+ * the format of pg_mv_statistic relation.
+ * ----------------
+ */
+typedef FormData_pg_mv_statistic *Form_pg_mv_statistic;
+
+/* ----------------
+ * compiler constants for pg_attrdef
+ * ----------------
+ */
+#define Natts_pg_mv_statistic 5
+#define Anum_pg_mv_statistic_starelid 1
+#define Anum_pg_mv_statistic_deps_enabled 2
+#define Anum_pg_mv_statistic_deps_built 3
+#define Anum_pg_mv_statistic_stakeys 4
+#define Anum_pg_mv_statistic_stadeps 5
+
+#endif /* PG_MV_STATISTIC_H */
diff --git a/src/include/catalog/pg_proc.h b/src/include/catalog/pg_proc.h
index 6a757f3..4b7ae1f 100644
--- a/src/include/catalog/pg_proc.h
+++ b/src/include/catalog/pg_proc.h
@@ -2693,6 +2693,11 @@ DESCR("current user privilege on any column by rel name");
DATA(insert OID = 3029 ( has_any_column_privilege PGNSP PGUID 12 10 0 0 0 f f f f t f s 2 0 16 "26 25" _null_ _null_ _null_ _null_ has_any_column_privilege_id _null_ _null_ _null_ ));
DESCR("current user privilege on any column by rel oid");
+DATA(insert OID = 3284 ( pg_mv_stats_dependencies_info PGNSP PGUID 12 1 0 0 0 f f f f t f i 1 0 25 "17" _null_ _null_ _null_ _null_ pg_mv_stats_dependencies_info _null_ _null_ _null_ ));
+DESCR("multivariate stats: functional dependencies info");
+DATA(insert OID = 3285 ( pg_mv_stats_dependencies_show PGNSP PGUID 12 1 0 0 0 f f f f t f i 1 0 25 "17" _null_ _null_ _null_ _null_ pg_mv_stats_dependencies_show _null_ _null_ _null_ ));
+DESCR("multivariate stats: functional dependencies show");
+
DATA(insert OID = 1928 ( pg_stat_get_numscans PGNSP PGUID 12 1 0 0 0 f f f f t f s 1 0 20 "26" _null_ _null_ _null_ _null_ pg_stat_get_numscans _null_ _null_ _null_ ));
DESCR("statistics: number of scans done for table/index");
DATA(insert OID = 1929 ( pg_stat_get_tuples_returned PGNSP PGUID 12 1 0 0 0 f f f f t f s 1 0 20 "26" _null_ _null_ _null_ _null_ pg_stat_get_tuples_returned _null_ _null_ _null_ ));
diff --git a/src/include/catalog/toasting.h b/src/include/catalog/toasting.h
index cba4ae7..45d3b5a 100644
--- a/src/include/catalog/toasting.h
+++ b/src/include/catalog/toasting.h
@@ -49,6 +49,7 @@ extern void BootstrapToastTable(char *relName,
DECLARE_TOAST(pg_attrdef, 2830, 2831);
DECLARE_TOAST(pg_constraint, 2832, 2833);
DECLARE_TOAST(pg_description, 2834, 2835);
+DECLARE_TOAST(pg_mv_statistic, 3288, 3289);
DECLARE_TOAST(pg_proc, 2836, 2837);
DECLARE_TOAST(pg_rewrite, 2838, 2839);
DECLARE_TOAST(pg_seclabel, 3598, 3599);
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 38469ef..3a0e7c4 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -414,6 +414,7 @@ typedef enum NodeTag
T_WithClause,
T_CommonTableExpr,
T_RoleSpec,
+ T_StatisticsDef,
/*
* TAGS FOR REPLICATION GRAMMAR PARSE NODES (replnodes.h)
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index ec0d0ea..b256162 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -570,6 +570,14 @@ typedef struct ColumnDef
int location; /* parse location, or -1 if none/unknown */
} ColumnDef;
+typedef struct StatisticsDef
+{
+ NodeTag type;
+ List *keys; /* String nodes naming referenced column(s) */
+ List *options; /* list of DefElem nodes */
+} StatisticsDef;
+
+
/*
* TableLikeClause - CREATE TABLE ( ... LIKE ... ) clause
*/
@@ -1362,7 +1370,8 @@ typedef enum AlterTableType
AT_ReplicaIdentity, /* REPLICA IDENTITY */
AT_EnableRowSecurity, /* ENABLE ROW SECURITY */
AT_DisableRowSecurity, /* DISABLE ROW SECURITY */
- AT_GenericOptions /* OPTIONS (...) */
+ AT_GenericOptions, /* OPTIONS (...) */
+ AT_AddStatistics /* add statistics */
} AlterTableType;
typedef struct ReplicaIdentityStmt
diff --git a/src/include/utils/mvstats.h b/src/include/utils/mvstats.h
new file mode 100644
index 0000000..2b59c2d
--- /dev/null
+++ b/src/include/utils/mvstats.h
@@ -0,0 +1,86 @@
+/*-------------------------------------------------------------------------
+ *
+ * mvstats.h
+ * Multivariate statistics and selectivity estimation functions.
+ *
+ *
+ * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/utils/mvstats.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef MVSTATS_H
+#define MVSTATS_H
+
+#include "commands/vacuum.h"
+
+/*
+ * Basic info about the stats, used when choosing what to use
+ *
+ * TODO Add info about what statistics is available (histogram, MCV,
+ * hashed MCV, functional dependencies).
+ */
+typedef struct MVStatsData {
+ Oid mvoid; /* OID of the stats in pg_mv_statistic */
+ int2vector *stakeys; /* attnums for columns in the stats */
+ bool deps_built; /* functional dependencies available */
+} MVStatsData;
+
+typedef struct MVStatsData *MVStats;
+
+
+#define MVSTATS_MAX_DIMENSIONS 8 /* max number of attributes */
+
+/* An associative rule, tracking [a => b] dependency.
+ *
+ * TODO Make this work with multiple columns on both sides.
+ */
+typedef struct MVDependencyData {
+ int16 a;
+ int16 b;
+} MVDependencyData;
+
+typedef MVDependencyData* MVDependency;
+
+typedef struct MVDependenciesData {
+ uint32 magic; /* magic constant marker */
+ int32 ndeps; /* number of dependencies */
+ MVDependency deps[1]; /* XXX why not a pointer? */
+} MVDependenciesData;
+
+typedef MVDependenciesData* MVDependencies;
+
+#define MVSTAT_DEPS_MAGIC 0xB4549A2C /* marks serialized bytea */
+#define MVSTAT_DEPS_TYPE_BASIC 1 /* basic dependencies type */
+
+/*
+ * TODO Maybe fetching the histogram/MCV list separately is inefficient?
+ * Consider adding a single `fetch_stats` method, fetching all
+ * stats specified using flags (or something like that).
+ */
+MVStats list_mv_stats(Oid relid, int *nstats, bool built_only);
+
+bytea * fetch_mv_dependencies(Oid mvoid);
+
+bytea * serialize_mv_dependencies(MVDependencies dependencies);
+
+/* deserialization of stats (serialization is private to analyze) */
+MVDependencies deserialize_mv_dependencies(bytea * data);
+
+/* FIXME this probably belongs somewhere else (not to operations stats) */
+extern Datum pg_mv_stats_dependencies_info(PG_FUNCTION_ARGS);
+extern Datum pg_mv_stats_dependencies_show(PG_FUNCTION_ARGS);
+
+MVDependencies
+build_mv_dependencies(int numrows, HeapTuple *rows,
+ int2vector *attrs,
+ int natts, VacAttrStats **vacattrstats);
+
+void build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
+ int natts, VacAttrStats **vacattrstats);
+
+void update_mv_stats(Oid relid, MVDependencies dependencies);
+
+#endif
diff --git a/src/include/utils/syscache.h b/src/include/utils/syscache.h
index ba0b090..12147ab 100644
--- a/src/include/utils/syscache.h
+++ b/src/include/utils/syscache.h
@@ -66,6 +66,7 @@ enum SysCacheIdentifier
INDEXRELID,
LANGNAME,
LANGOID,
+ MVSTATOID,
NAMESPACENAME,
NAMESPACEOID,
OPERNAMENSP,
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 1788270..f0117ca 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1353,6 +1353,14 @@ pg_matviews| SELECT n.nspname AS schemaname,
LEFT JOIN pg_namespace n ON ((n.oid = c.relnamespace)))
LEFT JOIN pg_tablespace t ON ((t.oid = c.reltablespace)))
WHERE (c.relkind = 'm'::"char");
+pg_mv_stats| SELECT n.nspname AS schemaname,
+ c.relname AS tablename,
+ s.stakeys AS attnums,
+ length(s.stadeps) AS depsbytes,
+ pg_mv_stats_dependencies_info(s.stadeps) AS depsinfo
+ FROM ((pg_mv_statistic s
+ JOIN pg_class c ON ((c.oid = s.starelid)))
+ LEFT JOIN pg_namespace n ON ((n.oid = c.relnamespace)));
pg_policies| SELECT n.nspname AS schemaname,
c.relname AS tablename,
pol.polname AS policyname,
diff --git a/src/test/regress/expected/sanity_check.out b/src/test/regress/expected/sanity_check.out
index c7be273..00f5fe7 100644
--- a/src/test/regress/expected/sanity_check.out
+++ b/src/test/regress/expected/sanity_check.out
@@ -113,6 +113,7 @@ pg_inherits|t
pg_language|t
pg_largeobject|t
pg_largeobject_metadata|t
+pg_mv_statistic|t
pg_namespace|t
pg_opclass|t
pg_operator|t
--
2.1.0.GIT
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: WIP: multivariate statistics / proof of concept
2014-10-12 22:00 WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-11-13 11:31 ` Re: WIP: multivariate statistics / proof of concept Simon Riggs <simon@2ndQuadrant.com>
2014-11-13 13:11 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-11-13 15:51 ` Re: WIP: multivariate statistics / proof of concept Katharina Büchse <katharina.buechse@uni-jena.de>
2014-11-13 16:42 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-11-15 17:49 ` Re: WIP: multivariate statistics / proof of concept Kevin Grittner <kgrittn@ymail.com>
2014-11-15 18:35 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-12-08 01:01 ` Re: WIP: multivariate statistics / proof of concept Michael Paquier <michael.paquier@gmail.com>
2014-12-09 20:15 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-12-15 02:55 ` Re: WIP: multivariate statistics / proof of concept Michael Paquier <michael.paquier@gmail.com>
2015-01-15 08:00 ` Re: WIP: multivariate statistics / proof of concept Michael Paquier <michael.paquier@gmail.com>
2015-01-24 20:21 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-20 08:33 ` Re: WIP: multivariate statistics / proof of concept Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
@ 2015-03-20 21:17 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-24 05:34 ` Re: WIP: multivariate statistics / proof of concept Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
0 siblings, 1 reply; 37+ messages in thread
From: Tomas Vondra @ 2015-03-20 21:17 UTC (permalink / raw)
To: Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>; +Cc: pgsql-hackers
Hello,
On 20.3.2015 09:33, Kyotaro HORIGUCHI wrote:
> Hello,
>
>
> Patch 0001 needs changes for OIDs since my patch was
> committed. The attached is compatible with current master.
Thanks. I plan to submit a new version of the patch in a few days, with
significant progress in various directions. I'll have to rebase to
current master before submitting the new version anyway (which includes
fixing duplicate OIDs).
> And I tried this like this, and got the following error on
> analyze. But unfortunately I don't have enough time to
> investigate it now.
>
> postgres=# create table t1 (a int, b int, c int);
> insert into t1 (select a/ 10000, a / 10000, a / 10000 from
> generate_series(0, 99999) a);
> postgres=# analyze t1;
> ERROR: invalid memory alloc request size 1485176862
Interesting - particularly because this does not involve any
multivariate stats. I can't reproduce it with the current version of the
patch, so either it's unrelated, or I've fixed it since posting the last
version.
regards
--
Tomas Vondra http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: WIP: multivariate statistics / proof of concept
2014-10-12 22:00 WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-11-13 11:31 ` Re: WIP: multivariate statistics / proof of concept Simon Riggs <simon@2ndQuadrant.com>
2014-11-13 13:11 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-11-13 15:51 ` Re: WIP: multivariate statistics / proof of concept Katharina Büchse <katharina.buechse@uni-jena.de>
2014-11-13 16:42 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-11-15 17:49 ` Re: WIP: multivariate statistics / proof of concept Kevin Grittner <kgrittn@ymail.com>
2014-11-15 18:35 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-12-08 01:01 ` Re: WIP: multivariate statistics / proof of concept Michael Paquier <michael.paquier@gmail.com>
2014-12-09 20:15 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-12-15 02:55 ` Re: WIP: multivariate statistics / proof of concept Michael Paquier <michael.paquier@gmail.com>
2015-01-15 08:00 ` Re: WIP: multivariate statistics / proof of concept Michael Paquier <michael.paquier@gmail.com>
2015-01-24 20:21 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-20 08:33 ` Re: WIP: multivariate statistics / proof of concept Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
2015-03-20 21:17 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tomas.vondra@2ndquadrant.com>
@ 2015-03-24 05:34 ` Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
2015-03-24 10:24 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-31 00:26 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 2 replies; 37+ messages in thread
From: Kyotaro HORIGUCHI @ 2015-03-24 05:34 UTC (permalink / raw)
To: tomas.vondra@2ndquadrant.com; +Cc: pgsql-hackers
Hello,
> > Patch 0001 needs changes for OIDs since my patch was
> > committed. The attached is compatible with current master.
>
> Thanks. I plan to submit a new version of the patch in a few days, with
> significant progress in various directions. I'll have to rebase to
> current master before submitting the new version anyway (which includes
> fixing duplicate OIDs).
>
> > And I tried this like this, and got the following error on
> > analyze. But unfortunately I don't have enough time to
> > investigate it now.
> >
> > postgres=# create table t1 (a int, b int, c int);
> > insert into t1 (select a/ 10000, a / 10000, a / 10000 from
> > generate_series(0, 99999) a);
> > postgres=# analyze t1;
> > ERROR: invalid memory alloc request size 1485176862
>
> Interesting - particularly because this does not involve any
> multivariate stats. I can't reproduce it with the current version of the
> patch, so either it's unrelated, or I've fixed it since posting the last
> version.
Sorry, not shown above, the *previous* t1 had been done "alter
table t1 add statistics (a, b, c)". Removing t1 didn't remove the
setting. reiniting cluster let me do that without error.
The steps throughout was as following.
===
create table t1 (a int, b int, c int);
alter table t1 add statistics (histogram) on (a, b, c);
drop table t1; -- This does not remove the above setting.
create table t1 (a int, b int, c int);
insert into t1 (select a/ 10000, a / 10000, a / 10000 from generate_series(0, 99999) a);insert into t1 ...
regards,
--
Kyotaro Horiguchi
NTT Open Source Software Center
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: WIP: multivariate statistics / proof of concept
2014-10-12 22:00 WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-11-13 11:31 ` Re: WIP: multivariate statistics / proof of concept Simon Riggs <simon@2ndQuadrant.com>
2014-11-13 13:11 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-11-13 15:51 ` Re: WIP: multivariate statistics / proof of concept Katharina Büchse <katharina.buechse@uni-jena.de>
2014-11-13 16:42 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-11-15 17:49 ` Re: WIP: multivariate statistics / proof of concept Kevin Grittner <kgrittn@ymail.com>
2014-11-15 18:35 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-12-08 01:01 ` Re: WIP: multivariate statistics / proof of concept Michael Paquier <michael.paquier@gmail.com>
2014-12-09 20:15 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-12-15 02:55 ` Re: WIP: multivariate statistics / proof of concept Michael Paquier <michael.paquier@gmail.com>
2015-01-15 08:00 ` Re: WIP: multivariate statistics / proof of concept Michael Paquier <michael.paquier@gmail.com>
2015-01-24 20:21 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-20 08:33 ` Re: WIP: multivariate statistics / proof of concept Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
2015-03-20 21:17 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-24 05:34 ` Re: WIP: multivariate statistics / proof of concept Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
@ 2015-03-24 10:24 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
1 sibling, 0 replies; 37+ messages in thread
From: Tomas Vondra @ 2015-03-24 10:24 UTC (permalink / raw)
To: Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>; +Cc: pgsql-hackers
Hello,
On 03/24/15 06:34, Kyotaro HORIGUCHI wrote:
>
> Sorry, not shown above, the *previous* t1 had been done "alter table
> t1 add statistics (a, b, c)". Removing t1 didn't remove the setting.
> reiniting cluster let me do that without error.
OK, thanks. My guess is this issue got already fixed in my working copy,
but I will double-check that.
Admittedly, the management of the stats (e.g. removing stats when the
table is dropped) is one of the incomplete parts. You have to delete the
rows manually from pg_mv_statistic.
--
--
Tomas Vondra http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: WIP: multivariate statistics / proof of concept
2014-10-12 22:00 WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-11-13 11:31 ` Re: WIP: multivariate statistics / proof of concept Simon Riggs <simon@2ndQuadrant.com>
2014-11-13 13:11 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-11-13 15:51 ` Re: WIP: multivariate statistics / proof of concept Katharina Büchse <katharina.buechse@uni-jena.de>
2014-11-13 16:42 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-11-15 17:49 ` Re: WIP: multivariate statistics / proof of concept Kevin Grittner <kgrittn@ymail.com>
2014-11-15 18:35 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-12-08 01:01 ` Re: WIP: multivariate statistics / proof of concept Michael Paquier <michael.paquier@gmail.com>
2014-12-09 20:15 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-12-15 02:55 ` Re: WIP: multivariate statistics / proof of concept Michael Paquier <michael.paquier@gmail.com>
2015-01-15 08:00 ` Re: WIP: multivariate statistics / proof of concept Michael Paquier <michael.paquier@gmail.com>
2015-01-24 20:21 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-20 08:33 ` Re: WIP: multivariate statistics / proof of concept Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
2015-03-20 21:17 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-24 05:34 ` Re: WIP: multivariate statistics / proof of concept Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
@ 2015-03-31 00:26 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-04-28 16:09 ` Re: WIP: multivariate statistics / proof of concept Jeff Janes <jeff.janes@gmail.com>
1 sibling, 1 reply; 37+ messages in thread
From: Tomas Vondra @ 2015-03-31 00:26 UTC (permalink / raw)
To: Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>; +Cc: pgsql-hackers
Hello,
attached is a new version of the patch series. Aside from fixing various
issues (crashes, memory leaks). The patches are rebased to current
master, and I also attach a few SQL scripts I used for testing (nothing
fancy, just stress-testing all the parts the patch touches).
The main changes in the patches (requiring plenty of changes in the
other parts) are about these:
(1) combining multiple statistics on a table
--------------------------------------------
In the previous version of the patch, it was only possible to use a
single statistics on a table - when there was a statistics "covering"
all the conditions it worked fine, but that's not always the case.
The new patch is able to combine multiple statistics by decomposing the
probability (=selectivity) into conditional probabilities. Imagine
estimating selectivity of clauses
WHERE (a=1) AND (b=1) AND (c=1) AND (d=1)
with statistics on [a,b,c] and [b,c,d]. The selectivity may be split for
example like this:
P(a=1,b=1,c=1,d=1) = P(a=1,b=1,c=1) * P(d=1|a=1,b=1,c=1)
where P(a=1,b=1,c=1) may be estimated using statistics [a,b,c], and the
second may be simplified like this:
P(d=1|a=1,b=1,c=1) = P(d=1|b=1,c=1)
using the assumption "no multivariate stats => independent". Both these
probabilities match the existing statistics.
The idea is described a bit more in the part #5 of the patch.
(2) choosing the best combination of statistics
-----------------------------------------------
There may be more statistics on a table, and multiple possible ways to
use them to estimate the clauses (different ordering, overlapping
statistics, etc.).
The patch formulates this as an optimization task with two goals.
(a) cover as many clauses as possible
(b) reuse as many conditions (i.e. dependencies) as possible
and implements two algorithms to solve this: (a) exhaustive, walking
through all possible states (using dynamic programming), and (b) greedy,
choosing the best local solution in each step.
The time requirements for the exhaustive solution grows pretty quickly
with the number of clauses and statistics on a table (~ O(N!)). The
greedy is much faster, as it's ~O(N) and in fact much more time is spent
in actually processing the selected statistics (walking through the
histograms etc.).
I assume the exhaustive search may find a better solution in some cases
(that the greedy algorithm misses), but so far I've been unable to come
up with such example.
To make this easier to test, I've added GUC to switch between these
algorithms easily (set to 'greedy' by default)
mvstat_search = {'greedy', 'exhaustive'}
I assume this GUC will be removed eventually, after we figure out which
algorithm is the right one.
(3) estimation of more complex conditions (AND/OR clauses)
----------------------------------------------------------
I've added ability to estimate more complex clauses - combinations of
AND/OR clauses and such. It's somewhat incomplete at the moment, but
hopefully the ideas will be clear from the TODOs/FIXMEs along the way.
Let me know if you have any questions about this version of the patch,
or about the ideas it implements in general.
I also welcome real-world examples of poorly estimated queries, so that
I can test if these patches improve that particular case situation.
regards
--
Tomas Vondra http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[text/x-diff] 0001-shared-infrastructure-and-functional-dependencies.patch (65.9K, ../../5519E9B2.5080105@2ndquadrant.com/2-0001-shared-infrastructure-and-functional-dependencies.patch)
download | inline diff:
From 7c8f0ce0017beea314219c24146cbb64d0d37a3d Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Sun, 11 Jan 2015 19:51:48 +0100
Subject: [PATCH 1/5] shared infrastructure and functional dependencies
Basic infrastructure shared by all kinds of multivariate
stats, most importantly:
- adds a new system catalog (pg_mv_statistic)
- ALTER TABLE ... ADD STATISTICS syntax
- implementation of functional dependencies (the simplest
type of multivariate statistics)
- building functional dependencies in ANALYZE
- updates regression tests (new catalog etc.)
This does not include any changes to the optimizer, i.e.
it does not influence the query planning (subject to
follow-up patches).
The current implementation requires a valid 'ltopr' for
the columns, so that we can sort the sample rows in various
ways, both in this patch and other kinds of statistics.
Maybe this restriction could be relaxed in the future,
requiring just 'eqopr' in case of stats not sorting the
data (e.g. functional dependencies and MCV lists).
The algorithm detecting the dependencies is rather simple
and probably needs improvements.
The name 'functional dependencies' is more correct (than
'association rules') as it's exactly the name used in
relational theory (esp. Normal Forms) for tracking
column-level dependencies.
---
src/backend/catalog/Makefile | 1 +
src/backend/catalog/system_views.sql | 10 +
src/backend/commands/analyze.c | 20 +-
src/backend/commands/tablecmds.c | 149 ++++++-
src/backend/nodes/copyfuncs.c | 14 +
src/backend/parser/gram.y | 67 ++-
src/backend/utils/Makefile | 2 +-
src/backend/utils/cache/syscache.c | 12 +
src/backend/utils/mvstats/Makefile | 17 +
src/backend/utils/mvstats/common.c | 342 +++++++++++++++
src/backend/utils/mvstats/common.h | 75 ++++
src/backend/utils/mvstats/dependencies.c | 680 +++++++++++++++++++++++++++++
src/include/catalog/indexing.h | 5 +
src/include/catalog/pg_mv_statistic.h | 69 +++
src/include/catalog/pg_proc.h | 5 +
src/include/catalog/toasting.h | 1 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 11 +-
src/include/utils/mvstats.h | 86 ++++
src/include/utils/syscache.h | 1 +
src/test/regress/expected/rules.out | 8 +
src/test/regress/expected/sanity_check.out | 1 +
22 files changed, 1569 insertions(+), 8 deletions(-)
create mode 100644 src/backend/utils/mvstats/Makefile
create mode 100644 src/backend/utils/mvstats/common.c
create mode 100644 src/backend/utils/mvstats/common.h
create mode 100644 src/backend/utils/mvstats/dependencies.c
create mode 100644 src/include/catalog/pg_mv_statistic.h
create mode 100644 src/include/utils/mvstats.h
diff --git a/src/backend/catalog/Makefile b/src/backend/catalog/Makefile
index a403c64..d6c16f8 100644
--- a/src/backend/catalog/Makefile
+++ b/src/backend/catalog/Makefile
@@ -32,6 +32,7 @@ POSTGRES_BKI_SRCS = $(addprefix $(top_srcdir)/src/include/catalog/,\
pg_attrdef.h pg_constraint.h pg_inherits.h pg_index.h pg_operator.h \
pg_opfamily.h pg_opclass.h pg_am.h pg_amop.h pg_amproc.h \
pg_language.h pg_largeobject_metadata.h pg_largeobject.h pg_aggregate.h \
+ pg_mv_statistic.h \
pg_statistic.h pg_rewrite.h pg_trigger.h pg_event_trigger.h pg_description.h \
pg_cast.h pg_enum.h pg_namespace.h pg_conversion.h pg_depend.h \
pg_database.h pg_db_role_setting.h pg_tablespace.h pg_pltemplate.h \
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 2800f73..d05a716 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -150,6 +150,16 @@ CREATE VIEW pg_indexes AS
LEFT JOIN pg_tablespace T ON (T.oid = I.reltablespace)
WHERE C.relkind IN ('r', 'm') AND I.relkind = 'i';
+CREATE VIEW pg_mv_stats AS
+ SELECT
+ N.nspname AS schemaname,
+ C.relname AS tablename,
+ S.stakeys AS attnums,
+ length(S.stadeps) as depsbytes,
+ pg_mv_stats_dependencies_info(S.stadeps) as depsinfo
+ FROM (pg_mv_statistic S JOIN pg_class C ON (C.oid = S.starelid))
+ LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace);
+
CREATE VIEW pg_stats AS
SELECT
nspname AS schemaname,
diff --git a/src/backend/commands/analyze.c b/src/backend/commands/analyze.c
index d4d1914..f82fcf5 100644
--- a/src/backend/commands/analyze.c
+++ b/src/backend/commands/analyze.c
@@ -27,6 +27,7 @@
#include "catalog/indexing.h"
#include "catalog/pg_collation.h"
#include "catalog/pg_inherits_fn.h"
+#include "catalog/pg_mv_statistic.h"
#include "catalog/pg_namespace.h"
#include "commands/dbcommands.h"
#include "commands/tablecmds.h"
@@ -54,7 +55,11 @@
#include "utils/syscache.h"
#include "utils/timestamp.h"
#include "utils/tqual.h"
+#include "utils/fmgroids.h"
+#include "utils/builtins.h"
+#include "utils/mvstats.h"
+#include "access/sysattr.h"
/* Data structure for Algorithm S from Knuth 3.4.2 */
typedef struct
@@ -110,7 +115,6 @@ static void update_attstats(Oid relid, bool inh,
static Datum std_fetch_func(VacAttrStatsP stats, int rownum, bool *isNull);
static Datum ind_fetch_func(VacAttrStatsP stats, int rownum, bool *isNull);
-
/*
* analyze_rel() -- analyze one relation
*/
@@ -471,6 +475,17 @@ do_analyze_rel(Relation onerel, int options, List *va_cols,
* all analyzable columns. We use a lower bound of 100 rows to avoid
* possible overflow in Vitter's algorithm. (Note: that will also be the
* target in the corner case where there are no analyzable columns.)
+ *
+ * FIXME This sample sizing is mostly OK when computing stats for
+ * individual columns, but when computing multi-variate stats
+ * for multivariate stats (histograms, mcv, ...) it's rather
+ * insufficient. For stats on multiple columns / complex stats
+ * we need larger sample sizes, and in some cases samples
+ * proportional to the table (say, 0.5% - 1%) instead of a
+ * fixed size might be more appropriate. Also, this should be
+ * bound to the requested statistics size - e.g. number of MCV
+ * items or histogram buckets should require several sample
+ * rows per item/bucket (so the sample should be k*size).
*/
targrows = 100;
for (i = 0; i < attr_cnt; i++)
@@ -573,6 +588,9 @@ do_analyze_rel(Relation onerel, int options, List *va_cols,
update_attstats(RelationGetRelid(Irel[ind]), false,
thisdata->attr_cnt, thisdata->vacattrstats);
}
+
+ /* Build multivariate stats (if there are any). */
+ build_mv_stats(onerel, numrows, rows, attr_cnt, vacattrstats);
}
/*
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 002319e..a321755 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -35,6 +35,7 @@
#include "catalog/pg_foreign_table.h"
#include "catalog/pg_inherits.h"
#include "catalog/pg_inherits_fn.h"
+#include "catalog/pg_mv_statistic.h"
#include "catalog/pg_namespace.h"
#include "catalog/pg_opclass.h"
#include "catalog/pg_tablespace.h"
@@ -92,7 +93,7 @@
#include "utils/syscache.h"
#include "utils/tqual.h"
#include "utils/typcache.h"
-
+#include "utils/mvstats.h"
/*
* ON COMMIT action list
@@ -140,8 +141,9 @@ static List *on_commits = NIL;
#define AT_PASS_ADD_COL 5 /* ADD COLUMN */
#define AT_PASS_ADD_INDEX 6 /* ADD indexes */
#define AT_PASS_ADD_CONSTR 7 /* ADD constraints, defaults */
-#define AT_PASS_MISC 8 /* other stuff */
-#define AT_NUM_PASSES 9
+#define AT_PASS_ADD_STATS 8 /* ADD statistics */
+#define AT_PASS_MISC 9 /* other stuff */
+#define AT_NUM_PASSES 10
typedef struct AlteredTableInfo
{
@@ -416,7 +418,8 @@ static void ATExecReplicaIdentity(Relation rel, ReplicaIdentityStmt *stmt, LOCKM
static void ATExecGenericOptions(Relation rel, List *options);
static void ATExecEnableRowSecurity(Relation rel);
static void ATExecDisableRowSecurity(Relation rel);
-
+static void ATExecAddStatistics(AlteredTableInfo *tab, Relation rel,
+ StatisticsDef *def, LOCKMODE lockmode);
static void copy_relation_data(SMgrRelation rel, SMgrRelation dst,
ForkNumber forkNum, char relpersistence);
static const char *storage_name(char c);
@@ -2999,6 +3002,7 @@ AlterTableGetLockLevel(List *cmds)
* updates.
*/
case AT_SetStatistics: /* Uses MVCC in getTableAttrs() */
+ case AT_AddStatistics: /* XXX not sure if the right level */
case AT_ClusterOn: /* Uses MVCC in getIndexes() */
case AT_DropCluster: /* Uses MVCC in getIndexes() */
case AT_SetOptions: /* Uses MVCC in getTableAttrs() */
@@ -3155,6 +3159,7 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd,
pass = AT_PASS_ADD_CONSTR;
break;
case AT_SetStatistics: /* ALTER COLUMN SET STATISTICS */
+ case AT_AddStatistics: /* XXX maybe not the right place */
ATSimpleRecursion(wqueue, rel, cmd, recurse, lockmode);
/* Performs own permission checks */
ATPrepSetStatistics(rel, cmd->name, cmd->def, lockmode);
@@ -3457,6 +3462,9 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,
case AT_SetStatistics: /* ALTER COLUMN SET STATISTICS */
address = ATExecSetStatistics(rel, cmd->name, cmd->def, lockmode);
break;
+ case AT_AddStatistics: /* ADD STATISTICS */
+ ATExecAddStatistics(tab, rel, (StatisticsDef *) cmd->def, lockmode);
+ break;
case AT_SetOptions: /* ALTER COLUMN SET ( options ) */
address = ATExecSetOptions(rel, cmd->name, cmd->def, false, lockmode);
break;
@@ -11854,3 +11862,136 @@ RangeVarCallbackForAlterRelation(const RangeVar *rv, Oid relid, Oid oldrelid,
ReleaseSysCache(tuple);
}
+
+/* used for sorting the attnums in ATExecAddStatistics */
+static int compare_int16(const void *a, const void *b)
+{
+ return memcmp(a, b, sizeof(int16));
+}
+
+/*
+ * Implements the ALTER TABLE ... ADD STATISTICS (options) ON (columns).
+ *
+ * The code is an unholy mix of pieces that really belong to other parts
+ * of the source tree.
+ *
+ * FIXME Check that the types are pass-by-value and support sort,
+ * although maybe we can live without the sort (and only build
+ * MCV list / association rules).
+ *
+ * FIXME This should probably check for duplicate stats (i.e. same
+ * keys, same options). Although maybe it's useful to have
+ * multiple stats on the same columns with different options
+ * (say, a detailed MCV-only stats for some queries, histogram
+ * for others, etc.)
+ */
+static void ATExecAddStatistics(AlteredTableInfo *tab, Relation rel,
+ StatisticsDef *def, LOCKMODE lockmode)
+{
+ int i, j;
+ ListCell *l;
+ int16 attnums[INDEX_MAX_KEYS];
+ int numcols = 0;
+
+ HeapTuple htup;
+ Datum values[Natts_pg_mv_statistic];
+ bool nulls[Natts_pg_mv_statistic];
+ int2vector *stakeys;
+ Relation mvstatrel;
+
+ /* by default build everything */
+ bool build_dependencies = true;
+
+ Assert(IsA(def, StatisticsDef));
+
+ /* transform the column names to attnum values */
+
+ foreach(l, def->keys)
+ {
+ char *attname = strVal(lfirst(l));
+ HeapTuple atttuple;
+
+ atttuple = SearchSysCacheAttName(RelationGetRelid(rel), attname);
+
+ if (!HeapTupleIsValid(atttuple))
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_COLUMN),
+ errmsg("column \"%s\" referenced in statistics does not exist",
+ attname)));
+
+ /* more than MVHIST_MAX_DIMENSIONS columns not allowed */
+ if (numcols >= MVSTATS_MAX_DIMENSIONS)
+ ereport(ERROR,
+ (errcode(ERRCODE_TOO_MANY_COLUMNS),
+ errmsg("cannot have more than %d keys in a statistics",
+ MVSTATS_MAX_DIMENSIONS)));
+
+ attnums[numcols] = ((Form_pg_attribute) GETSTRUCT(atttuple))->attnum;
+ ReleaseSysCache(atttuple);
+ numcols++;
+ }
+
+ /*
+ * Check the lower bound (at least 2 columns), the upper bound was
+ * already checked in the loop.
+ */
+ if (numcols < 2)
+ ereport(ERROR,
+ (errcode(ERRCODE_TOO_MANY_COLUMNS),
+ errmsg("multivariate stats require 2 or more columns")));
+
+ /* look for duplicities */
+ for (i = 0; i < numcols; i++)
+ for (j = 0; j < numcols; j++)
+ if ((i != j) && (attnums[i] == attnums[j]))
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_COLUMN),
+ errmsg("duplicate column name in statistics definition")));
+
+ /* parse the statistics options */
+ foreach (l, def->options)
+ {
+ DefElem *opt = (DefElem*)lfirst(l);
+
+ if (strcmp(opt->defname, "dependencies") == 0)
+ build_dependencies = defGetBoolean(opt);
+ else
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("unrecognized STATISTICS option \"%s\"",
+ opt->defname)));
+ }
+
+ /* sort the attnums and build int2vector */
+ qsort(attnums, numcols, sizeof(int16), compare_int16);
+ stakeys = buildint2vector(attnums, numcols);
+
+ /*
+ * Okay, let's create the pg_mv_statistic entry.
+ */
+ memset(values, 0, sizeof(values));
+ memset(nulls, false, sizeof(nulls));
+
+ /* no stats collected yet, so just the keys */
+ values[Anum_pg_mv_statistic_starelid-1] = ObjectIdGetDatum(RelationGetRelid(rel));
+
+ values[Anum_pg_mv_statistic_stakeys -1] = PointerGetDatum(stakeys);
+ values[Anum_pg_mv_statistic_deps_enabled -1] = BoolGetDatum(build_dependencies);
+
+ nulls[Anum_pg_mv_statistic_stadeps -1] = true;
+
+ /* insert the tuple into pg_mv_statistic */
+ mvstatrel = heap_open(MvStatisticRelationId, RowExclusiveLock);
+
+ htup = heap_form_tuple(mvstatrel->rd_att, values, nulls);
+
+ simple_heap_insert(mvstatrel, htup);
+
+ CatalogUpdateIndexes(mvstatrel, htup);
+
+ heap_freetuple(htup);
+
+ heap_close(mvstatrel, RowExclusiveLock);
+
+ return;
+}
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 029761e..a4ce2c9 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -3918,6 +3918,17 @@ _copyAlterPolicyStmt(const AlterPolicyStmt *from)
return newnode;
}
+static StatisticsDef *
+_copyStatisticsDef(const StatisticsDef *from)
+{
+ StatisticsDef *newnode = makeNode(StatisticsDef);
+
+ COPY_NODE_FIELD(keys);
+ COPY_NODE_FIELD(options);
+
+ return newnode;
+}
+
/* ****************************************************************
* pg_list.h copy functions
* ****************************************************************
@@ -4732,6 +4743,9 @@ copyObject(const void *from)
case T_CommonTableExpr:
retval = _copyCommonTableExpr(from);
break;
+ case T_StatisticsDef:
+ retval = _copyStatisticsDef(from);
+ break;
case T_FuncWithArgs:
retval = _copyFuncWithArgs(from);
break;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 3aa9e42..17183ef 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -367,6 +367,13 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
create_generic_options alter_generic_options
relation_expr_list dostmt_opt_list
+%type <list> OptStatsOptions
+%type <str> stats_options_name
+%type <node> stats_options_arg
+%type <defelt> stats_options_elem
+%type <list> stats_options_list
+
+
%type <list> opt_fdw_options fdw_options
%type <defelt> fdw_option
@@ -486,7 +493,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <keyword> unreserved_keyword type_func_name_keyword
%type <keyword> col_name_keyword reserved_keyword
-%type <node> TableConstraint TableLikeClause
+%type <node> TableConstraint TableLikeClause TableStatistics
%type <ival> TableLikeOptionList TableLikeOption
%type <list> ColQualList
%type <node> ColConstraint ColConstraintElem ConstraintAttr
@@ -2311,6 +2318,14 @@ alter_table_cmd:
n->subtype = AT_DisableRowSecurity;
$$ = (Node *)n;
}
+ /* ALTER TABLE <name> ADD STATISTICS (options) ON (columns) ... */
+ | ADD_P TableStatistics
+ {
+ AlterTableCmd *n = makeNode(AlterTableCmd);
+ n->subtype = AT_AddStatistics;
+ n->def = $2;
+ $$ = (Node *)n;
+ }
| alter_generic_options
{
AlterTableCmd *n = makeNode(AlterTableCmd);
@@ -3385,6 +3400,56 @@ OptConsTableSpace: USING INDEX TABLESPACE name { $$ = $4; }
ExistingIndex: USING INDEX index_name { $$ = $3; }
;
+/*****************************************************************************
+ *
+ * QUERY :
+ * ALTER TABLE relname ADD STATISTICS (columns) WITH (options)
+ *
+ *****************************************************************************/
+
+TableStatistics:
+ STATISTICS OptStatsOptions ON '(' columnList ')'
+ {
+ StatisticsDef *n = makeNode(StatisticsDef);
+ n->keys = $5;
+ n->options = $2;
+ $$ = (Node *) n;
+ }
+ ;
+
+OptStatsOptions:
+ '(' stats_options_list ')' { $$ = $2; }
+ | /*EMPTY*/ { $$ = NIL; }
+ ;
+
+stats_options_list:
+ stats_options_elem
+ {
+ $$ = list_make1($1);
+ }
+ | stats_options_list ',' stats_options_elem
+ {
+ $$ = lappend($1, $3);
+ }
+ ;
+
+stats_options_elem:
+ stats_options_name stats_options_arg
+ {
+ $$ = makeDefElem($1, $2);
+ }
+ ;
+
+stats_options_name:
+ NonReservedWord { $$ = $1; }
+ ;
+
+stats_options_arg:
+ opt_boolean_or_string { $$ = (Node *) makeString($1); }
+ | NumericOnly { $$ = (Node *) $1; }
+ | /* EMPTY */ { $$ = NULL; }
+ ;
+
/*****************************************************************************
*
diff --git a/src/backend/utils/Makefile b/src/backend/utils/Makefile
index 8374533..eba0352 100644
--- a/src/backend/utils/Makefile
+++ b/src/backend/utils/Makefile
@@ -9,7 +9,7 @@ top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
OBJS = fmgrtab.o
-SUBDIRS = adt cache error fmgr hash init mb misc mmgr resowner sort time
+SUBDIRS = adt cache error fmgr hash init mb misc mmgr mvstats resowner sort time
# location of Catalog.pm
catalogdir = $(top_srcdir)/src/backend/catalog
diff --git a/src/backend/utils/cache/syscache.c b/src/backend/utils/cache/syscache.c
index bd27168..f61ef7e 100644
--- a/src/backend/utils/cache/syscache.c
+++ b/src/backend/utils/cache/syscache.c
@@ -43,6 +43,7 @@
#include "catalog/pg_foreign_server.h"
#include "catalog/pg_foreign_table.h"
#include "catalog/pg_language.h"
+#include "catalog/pg_mv_statistic.h"
#include "catalog/pg_namespace.h"
#include "catalog/pg_opclass.h"
#include "catalog/pg_operator.h"
@@ -499,6 +500,17 @@ static const struct cachedesc cacheinfo[] = {
},
4
},
+ {MvStatisticRelationId, /* MVSTATOID */
+ MvStatisticOidIndexId,
+ 1,
+ {
+ ObjectIdAttributeNumber,
+ 0,
+ 0,
+ 0
+ },
+ 128
+ },
{NamespaceRelationId, /* NAMESPACENAME */
NamespaceNameIndexId,
1,
diff --git a/src/backend/utils/mvstats/Makefile b/src/backend/utils/mvstats/Makefile
new file mode 100644
index 0000000..099f1ed
--- /dev/null
+++ b/src/backend/utils/mvstats/Makefile
@@ -0,0 +1,17 @@
+#-------------------------------------------------------------------------
+#
+# Makefile--
+# Makefile for utils/mvstats
+#
+# IDENTIFICATION
+# src/backend/utils/mvstats/Makefile
+#
+#-------------------------------------------------------------------------
+
+subdir = src/backend/utils/mvstats
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+
+OBJS = common.o dependencies.o
+
+include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/utils/mvstats/common.c b/src/backend/utils/mvstats/common.c
new file mode 100644
index 0000000..8efc5ba
--- /dev/null
+++ b/src/backend/utils/mvstats/common.c
@@ -0,0 +1,342 @@
+/*-------------------------------------------------------------------------
+ *
+ * common.c
+ * POSTGRES multivariate statistics
+ *
+ *
+ * Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/utils/mvstats/common.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "common.h"
+
+static VacAttrStats ** lookup_var_attr_stats(int2vector *attrs,
+ int natts, VacAttrStats **vacattrstats);
+
+
+/*
+ * Compute requested multivariate stats, using the rows sampled for the
+ * plain (single-column) stats.
+ *
+ * This fetches a list of stats from pg_mv_statistic, computes the stats
+ * and serializes them back into the catalog (as bytea values).
+ */
+void
+build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
+ int natts, VacAttrStats **vacattrstats)
+{
+ int i;
+ MVStats mvstats;
+ int nmvstats;
+
+ /*
+ * Fetch defined MV groups from pg_mv_statistic, and then compute
+ * the MV statistics (histograms for now).
+ */
+ mvstats = list_mv_stats(RelationGetRelid(onerel), &nmvstats, false);
+
+ for (i = 0; i < nmvstats; i++)
+ {
+ MVDependencies deps = NULL;
+
+ /* int2 vector of attnums the stats should be computed on */
+ int2vector * attrs = mvstats[i].stakeys;
+
+ /* filter only the interesting vacattrstats records */
+ VacAttrStats **stats = lookup_var_attr_stats(attrs, natts, vacattrstats);
+
+ /* check allowed number of dimensions */
+ Assert((attrs->dim1 >= 2) && (attrs->dim1 <= MVSTATS_MAX_DIMENSIONS));
+
+ /*
+ * Analyze functional dependencies of columns.
+ */
+ deps = build_mv_dependencies(numrows, rows, attrs, stats);
+
+ /* store the histogram / MCV list in the catalog */
+ update_mv_stats(mvstats[i].mvoid, deps);
+ }
+}
+
+/*
+ * Lookup the VacAttrStats info for the selected columns, with indexes
+ * matching the attrs vector (to make it easy to work with when
+ * computing multivariate stats).
+ */
+static VacAttrStats **
+lookup_var_attr_stats(int2vector *attrs, int natts, VacAttrStats **vacattrstats)
+{
+ int i, j;
+ int numattrs = attrs->dim1;
+ VacAttrStats **stats = (VacAttrStats**)palloc0(numattrs * sizeof(VacAttrStats*));
+
+ /* lookup VacAttrStats info for the requested columns (same attnum) */
+ for (i = 0; i < numattrs; i++)
+ {
+ stats[i] = NULL;
+ for (j = 0; j < natts; j++)
+ {
+ if (attrs->values[i] == vacattrstats[j]->tupattnum)
+ {
+ stats[i] = vacattrstats[j];
+ break;
+ }
+ }
+
+ /*
+ * Check that we found the info, that the attnum matches and
+ * that there's the requested 'lt' operator and that the type
+ * is 'passed-by-value'.
+ */
+ Assert(stats[i] != NULL);
+ Assert(stats[i]->tupattnum == attrs->values[i]);
+
+ /* FIXME This is rather ugly way to check for 'ltopr' (which
+ * is defined for 'scalar' attributes).
+ */
+ Assert(((StdAnalyzeData *)stats[i]->extra_data)->ltopr != InvalidOid);
+ }
+
+ return stats;
+}
+
+/*
+ * Fetch list of MV stats defined on a table, without the actual data
+ * for histograms, MCV lists etc.
+ */
+MVStats
+list_mv_stats(Oid relid, int *nstats, bool built_only)
+{
+ Relation indrel;
+ SysScanDesc indscan;
+ ScanKeyData skey;
+ HeapTuple htup;
+ MVStats result;
+
+ /* start with 16 items, that should be enough for most cases */
+ int maxitems = 16;
+ result = (MVStats)palloc0(sizeof(MVStatsData) * maxitems);
+ *nstats = 0;
+
+ /* Prepare to scan pg_mv_statistic for entries having indrelid = this rel. */
+ ScanKeyInit(&skey,
+ Anum_pg_mv_statistic_starelid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(relid));
+
+ indrel = heap_open(MvStatisticRelationId, AccessShareLock);
+ indscan = systable_beginscan(indrel, MvStatisticRelidIndexId, true,
+ NULL, 1, &skey);
+
+ while (HeapTupleIsValid(htup = systable_getnext(indscan)))
+ {
+ Form_pg_mv_statistic stats = (Form_pg_mv_statistic) GETSTRUCT(htup);
+
+ /*
+ * Skip statistics that were not computed yet (if only stats
+ * that were already built were requested)
+ */
+ if (built_only && (! stats->deps_built))
+ continue;
+
+ /* double the array size if needed */
+ if (*nstats == maxitems)
+ {
+ maxitems *= 2;
+ result = (MVStats)repalloc(result, sizeof(MVStatsData) * maxitems);
+ }
+
+ result[*nstats].mvoid = HeapTupleGetOid(htup);
+ result[*nstats].stakeys = buildint2vector(stats->stakeys.values, stats->stakeys.dim1);
+ result[*nstats].deps_built = stats->deps_built;
+ *nstats += 1;
+ }
+
+ systable_endscan(indscan);
+
+ heap_close(indrel, AccessShareLock);
+
+ /* TODO maybe save the list into relcache, as in RelationGetIndexList
+ * (which was used as an inspiration of this one)?. */
+
+ return result;
+}
+
+void
+update_mv_stats(Oid mvoid, MVDependencies dependencies)
+{
+ HeapTuple stup,
+ oldtup;
+ Datum values[Natts_pg_mv_statistic];
+ bool nulls[Natts_pg_mv_statistic];
+ bool replaces[Natts_pg_mv_statistic];
+
+ Relation sd = heap_open(MvStatisticRelationId, RowExclusiveLock);
+
+ memset(nulls, 1, Natts_pg_mv_statistic * sizeof(bool));
+ memset(replaces, 0, Natts_pg_mv_statistic * sizeof(bool));
+ memset(values, 0, Natts_pg_mv_statistic * sizeof(Datum));
+
+ /*
+ * Construct a new pg_mv_statistic tuple - replace only the histogram
+ * and MCV list, depending whether it actually was computed.
+ */
+ if (dependencies != NULL)
+ {
+ nulls[Anum_pg_mv_statistic_stadeps -1] = false;
+ values[Anum_pg_mv_statistic_stadeps - 1]
+ = PointerGetDatum(serialize_mv_dependencies(dependencies));
+ }
+
+ /* always replace the value (either by bytea or NULL) */
+ replaces[Anum_pg_mv_statistic_stadeps -1] = true;
+
+ /* always change the availability flags */
+ nulls[Anum_pg_mv_statistic_deps_built -1] = false;
+
+ replaces[Anum_pg_mv_statistic_deps_built-1] = true;
+
+ values[Anum_pg_mv_statistic_deps_built-1] = BoolGetDatum(dependencies != NULL);
+
+ /* Is there already a pg_mv_statistic tuple for this attribute? */
+ oldtup = SearchSysCache1(MVSTATOID,
+ ObjectIdGetDatum(mvoid));
+
+ if (HeapTupleIsValid(oldtup))
+ {
+ /* Yes, replace it */
+ stup = heap_modify_tuple(oldtup,
+ RelationGetDescr(sd),
+ values,
+ nulls,
+ replaces);
+ ReleaseSysCache(oldtup);
+ simple_heap_update(sd, &stup->t_self, stup);
+ }
+ else
+ elog(ERROR, "invalid pg_mv_statistic record (oid=%d)", mvoid);
+
+ /* update indexes too */
+ CatalogUpdateIndexes(sd, stup);
+
+ heap_freetuple(stup);
+
+ heap_close(sd, RowExclusiveLock);
+}
+
+/* multi-variate stats comparator */
+
+/*
+ * qsort_arg comparator for sorting Datums (MV stats)
+ *
+ * This does not maintain the tupnoLink array.
+ */
+int
+compare_scalars_simple(const void *a, const void *b, void *arg)
+{
+ Datum da = *(Datum*)a;
+ Datum db = *(Datum*)b;
+ SortSupport ssup= (SortSupport) arg;
+
+ return ApplySortComparator(da, false, db, false, ssup);
+}
+
+/*
+ * qsort_arg comparator for sorting data when partitioning a MV bucket
+ */
+int
+compare_scalars_partition(const void *a, const void *b, void *arg)
+{
+ Datum da = ((ScalarItem*)a)->value;
+ Datum db = ((ScalarItem*)b)->value;
+ SortSupport ssup= (SortSupport) arg;
+
+ return ApplySortComparator(da, false, db, false, ssup);
+}
+
+/* initialize multi-dimensional sort */
+MultiSortSupport
+multi_sort_init(int ndims)
+{
+ MultiSortSupport mss;
+
+ Assert(ndims >= 2);
+
+ mss = (MultiSortSupport)palloc0(offsetof(MultiSortSupportData, ssup)
+ + sizeof(SortSupportData)*ndims);
+
+ mss->ndims = ndims;
+
+ return mss;
+}
+
+/*
+ * add sort into for dimension 'dim' (index into vacattrstats) to mss,
+ * at the position 'sortattr'
+ */
+void
+multi_sort_add_dimension(MultiSortSupport mss, int sortdim,
+ int dim, VacAttrStats **vacattrstats)
+{
+ /* first, lookup StdAnalyzeData for the dimension (attribute) */
+ SortSupportData ssup;
+ StdAnalyzeData *tmp = (StdAnalyzeData *)vacattrstats[dim]->extra_data;
+
+ Assert(mss != NULL);
+ Assert(sortdim < mss->ndims);
+
+ /* initialize sort support, etc. */
+ memset(&ssup, 0, sizeof(ssup));
+ ssup.ssup_cxt = CurrentMemoryContext;
+
+ /* We always use the default collation for statistics */
+ ssup.ssup_collation = DEFAULT_COLLATION_OID;
+ ssup.ssup_nulls_first = false;
+
+ PrepareSortSupportFromOrderingOp(tmp->ltopr, &ssup);
+
+ mss->ssup[sortdim] = ssup;
+}
+
+/* compare all the dimensions in the selected order */
+int
+multi_sort_compare(const void *a, const void *b, void *arg)
+{
+ int i;
+ SortItem *ia = (SortItem*)a;
+ SortItem *ib = (SortItem*)b;
+
+ MultiSortSupport mss = (MultiSortSupport)arg;
+
+ for (i = 0; i < mss->ndims; i++)
+ {
+ int compare;
+
+ compare = ApplySortComparator(ia->values[i], ia->isnull[i],
+ ib->values[i], ib->isnull[i],
+ &mss->ssup[i]);
+
+ if (compare != 0)
+ return compare;
+
+ }
+
+ /* equal by default */
+ return 0;
+}
+
+/* compare selected dimension */
+int
+multi_sort_compare_dim(int dim, const SortItem *a, const SortItem *b,
+ MultiSortSupport mss)
+{
+ return ApplySortComparator(a->values[dim], a->isnull[dim],
+ b->values[dim], b->isnull[dim],
+ &mss->ssup[dim]);
+}
diff --git a/src/backend/utils/mvstats/common.h b/src/backend/utils/mvstats/common.h
new file mode 100644
index 0000000..6d5465b
--- /dev/null
+++ b/src/backend/utils/mvstats/common.h
@@ -0,0 +1,75 @@
+/*-------------------------------------------------------------------------
+ *
+ * common.h
+ * POSTGRES multivariate statistics
+ *
+ *
+ * Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/utils/mvstats/common.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/tuptoaster.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_collation.h"
+#include "catalog/pg_mv_statistic.h"
+#include "foreign/fdwapi.h"
+#include "postmaster/autovacuum.h"
+#include "storage/lmgr.h"
+#include "utils/datum.h"
+#include "utils/sortsupport.h"
+#include "utils/syscache.h"
+#include "utils/fmgroids.h"
+#include "utils/builtins.h"
+#include "access/sysattr.h"
+
+#include "utils/mvstats.h"
+
+/* FIXME private structure copied from analyze.c */
+
+typedef struct
+{
+ Oid eqopr; /* '=' operator for datatype, if any */
+ Oid eqfunc; /* and associated function */
+ Oid ltopr; /* '<' operator for datatype, if any */
+} StdAnalyzeData;
+
+typedef struct
+{
+ Datum value; /* a data value */
+ int tupno; /* position index for tuple it came from */
+} ScalarItem;
+
+/* multi-sort */
+typedef struct MultiSortSupportData {
+ int ndims; /* number of dimensions supported by the */
+ SortSupportData ssup[1]; /* sort support data for each dimension */
+} MultiSortSupportData;
+
+typedef MultiSortSupportData* MultiSortSupport;
+
+typedef struct SortItem {
+ Datum *values;
+ bool *isnull;
+} SortItem;
+
+MultiSortSupport multi_sort_init(int ndims);
+
+void multi_sort_add_dimension(MultiSortSupport mss, int sortdim,
+ int dim, VacAttrStats **vacattrstats);
+
+int multi_sort_compare(const void *a, const void *b, void *arg);
+
+int multi_sort_compare_dim(int dim, const SortItem *a,
+ const SortItem *b, MultiSortSupport mss);
+
+/* comparators, used when constructing multivariate stats */
+int compare_scalars_simple(const void *a, const void *b, void *arg);
+int compare_scalars_partition(const void *a, const void *b, void *arg);
diff --git a/src/backend/utils/mvstats/dependencies.c b/src/backend/utils/mvstats/dependencies.c
new file mode 100644
index 0000000..9e7f294
--- /dev/null
+++ b/src/backend/utils/mvstats/dependencies.c
@@ -0,0 +1,680 @@
+/*-------------------------------------------------------------------------
+ *
+ * dependencies.c
+ * POSTGRES multivariate functional dependencies
+ *
+ *
+ * Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/utils/mvstats/dependencies.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "common.h"
+#include "utils/lsyscache.h"
+
+/*
+ * Mine functional dependencies between columns, in the form (A => B),
+ * meaning that a value in column 'A' determines value in 'B'. A simple
+ * artificial example may be a table created like this
+ *
+ * CREATE TABLE deptest (a INT, b INT)
+ * AS SELECT i, i/10 FROM generate_series(1,100000) s(i);
+ *
+ * Clearly, once we know the value for 'A' we can easily determine the
+ * value of 'B' by dividing (A/10). A more practical example may be
+ * addresses, where (ZIP code => city name), i.e. once we know the ZIP,
+ * we probably know which city it belongs to. Larger cities usually have
+ * multiple ZIP codes, so the dependency can't be reversed.
+ *
+ * Functional dependencies are a concept well described in relational
+ * theory, especially in definition of normalization and "normal forms".
+ * Wikipedia has a nice definition of a functional dependency [1]:
+ *
+ * In a given table, an attribute Y is said to have a functional
+ * dependency on a set of attributes X (written X -> Y) if and only
+ * if each X value is associated with precisely one Y value. For
+ * example, in an "Employee" table that includes the attributes
+ * "Employee ID" and "Employee Date of Birth", the functional
+ * dependency {Employee ID} -> {Employee Date of Birth} would hold.
+ * It follows from the previous two sentences that each {Employee ID}
+ * is associated with precisely one {Employee Date of Birth}.
+ *
+ * [1] http://en.wikipedia.org/wiki/Database_normalization
+ *
+ * Most datasets might be normalized not to contain any such functional
+ * dependencies, but sometimes it's not practical. In some cases it's
+ * actually a conscious choice to model the dataset in denormalized way,
+ * either because of performance or to make querying easier.
+ *
+ * The current implementation supports only dependencies between two
+ * columns, but this is merely a simplification of the initial patch.
+ * It's certainly useful to mine for dependencies involving multiple
+ * columns on the 'left' side, i.e. a condition for the dependency.
+ * That is dependencies [A,B] => C and so on.
+ *
+ * TODO The implementation may/should be smart enough not to mine both
+ * [A => B] and [A,C => B], because the second dependency is a
+ * consequence of the first one (if values of A determine values
+ * of B, adding another column won't change that). The ANALYZE
+ * should first analyze 1:1 dependencies, then 2:1 dependencies
+ * (and skip the already identified ones), etc.
+ *
+ * For example the dependency [city name => zip code] is much weaker
+ * than [city name, state name => zip code], because there may be
+ * multiple cities with the same name in various states. It's not
+ * perfect though - there are probably cities with the same name within
+ * the same state, but this is relatively rare occurence hopefully.
+ * More about this in the section about dependency mining.
+ *
+ * Handling multiple columns on the right side is not necessary, as such
+ * dependencies may be decomposed into a set of dependencies with
+ * the same meaning, one for each column on the right side. For example
+ *
+ * A => [B,C]
+ *
+ * is exactly the same as
+ *
+ * (A => B) & (A => C).
+ *
+ * Of course, storing (A => [B, C]) may be more efficient thant storing
+ * the two dependencies (A => B) and (A => C) separately.
+ *
+ *
+ * Dependency mining (ANALYZE)
+ * ---------------------------
+ *
+ * The current build algorithm is rather simple - for each pair [A,B] of
+ * columns, the data are sorted lexicographically (first by A, then B),
+ * and then a number of metrics is computed by walking the sorted data.
+ *
+ * In general the algorithm counts distict values of A (forming groups
+ * thanks to the sorting), supporting or contradicting the hypothesis
+ * that A => B (i.e. that values of B are predetermined by A). If there
+ * are multiple values of B for a single value of A, it's counted as
+ * contradicting.
+ *
+ * A group may be neither supporting nor contradicting. To be counted as
+ * supporting, the group has to have at least min_group_size(=3) rows.
+ * Smaller 'supporting' groups are counted as neutral.
+ *
+ * Finally, the number of rows in supporting and contradicting groups is
+ * compared, and if there is at least 10x more supporting rows, the
+ * dependency is considered valid.
+ *
+ *
+ * Real-world datasets are imperfect - there may be errors (e.g. due to
+ * data-entry mistakes), or factually correct records, yet contradicting
+ * the dependency (e.g. when a city splits into two, but both keep the
+ * same ZIP code). A strict ANALYZE implementation (where the functional
+ * dependencies are identified) would ignore dependencies on such noisy
+ * data, making the approach unusable in practice.
+ *
+ * The proposed implementation attempts to handle such noisy cases
+ * gracefully, by tolerating small number of contradicting cases.
+ *
+ * In the future this might also perform some sort of test and decide
+ * whether it's worth building any other kind of multivariate stats,
+ * or whether the dependencies sufficiently describe the data. Or at
+ * least not build the MCV list / histogram on the implied columns.
+ * Such reduction would however make the 'verification' (see the next
+ * section) impossible.
+ *
+ *
+ * Clause reduction (planner/optimizer)
+ * ------------------------------------
+ *
+ * Apllying the dependencies is quite simple - given a list of clauses,
+ * try to apply all the dependencies. For example given clause list
+ *
+ * (a = 1) AND (b = 1) AND (c = 1) AND (d < 100)
+ *
+ * and dependencies [a=>b] and [a=>d], this may be reduced to
+ *
+ * (a = 1) AND (c = 1) AND (d < 100)
+ *
+ * The (d<100) can't be reduced as it's not an equality clause, so the
+ * dependency [a=>d] can't be applied.
+ *
+ * See clauselist_apply_dependencies() for more details.
+ *
+ * The problem with the reduction is that the query may use conditions
+ * that are not redundant, but in fact contradictory - e.g. the user
+ * may search for a ZIP code and a city name not matching the ZIP code.
+ *
+ * In such cases, the condition on the city name is not actually
+ * redundant, but actually contradictory (making the result empty), and
+ * removing it while estimating the cardinality will make the estimate
+ * worse.
+ *
+ * The current estimation assuming independence (and multiplying the
+ * selectivities) works better in this case, but only by utter luck.
+ *
+ * In some cases this might be verified using the other multivariate
+ * statistics - MCV lists and histograms. For MCV lists the verification
+ * might be very simple - peek into the list if there are any items
+ * matching the clause on the 'A' column (e.g. ZIP code), and if such
+ * item is found, check that the 'B' column matches the other clause.
+ * If it does not, the clauses are contradictory. We can't really say
+ * if such item was not found, except maybe restricting the selectivity
+ * using the MCV data (e.g. using min/max selectivity, or something).
+ *
+ * With histograms, it might work similarly - we can't check the values
+ * directly (because histograms use buckets, unlike MCV lists, storing
+ * the actual values). So we can only observe the buckets matching the
+ * clauses - if those buckets have very low frequency, it probably means
+ * the two clauses are incompatible.
+ *
+ * It's unclear what 'low frequency' is, but if one of the clauses is
+ * implied (automatically true because of the other clause), then
+ *
+ * selectivity[clause(A)] = selectivity[clause(A) & clause(B)]
+ *
+ * So we might compute selectivity of the first clause (on the column
+ * A in dependency [A=>B]) - for example using regular statistics.
+ * And then check if the selectivity computed from the histogram is
+ * about the same (or significantly lower).
+ *
+ * The problem is that histograms work well only when the data ordering
+ * matches the natural meaning. For values that serve as labels - like
+ * city names or ZIP codes, or even generated IDs, histograms really
+ * don't work all that well. For example sorting cities by name won't
+ * match the sorting of ZIP codes, rendering the histogram unusable.
+ *
+ * The MCV are probably going to work much better, because they don't
+ * really assume any sort of ordering. And it's probably more appropriate
+ * for the label-like data.
+ *
+ * TODO Support dependencies with multiple columns on left/right.
+ *
+ * TODO Investigate using histogram and MCV list to confirm the
+ * functional dependencies.
+ *
+ * TODO Investigate statistical testing of the distribution (to decide
+ * whether it makes sense to build the histogram/MCV list).
+ *
+ * TODO Using a min/max of selectivities would probably make more sense
+ * for the associated columns.
+ *
+ * TODO Consider eliminating the implied columns from the histogram and
+ * MCV lists (but maybe that's not a good idea, because that'd make
+ * it impossible to use these stats for non-equality clauses and
+ * also it wouldn't be possible to use the stats for verification
+ * of the dependencies as proposed in another TODO).
+ *
+ * TODO This builds a complete set of dependencies, i.e. including
+ * transitive dependencies - if we identify [A => B] and [B => C],
+ * we're likely to identify [A => C] too. It might be better to
+ * keep only the minimal set of dependencies, i.e. prune all the
+ * dependencies that we can recreate by transivitity.
+ *
+ * There are two conceptual ways to do that:
+ *
+ * (a) generate all the rules, and then prune the rules that may
+ * be recteated by combining other dependencies, or
+ *
+ * (b) performing the 'is combination of other dependencies' check
+ * before actually doing the work
+ *
+ * The second option has the advantage that we don't really need
+ * to perform the sort/count. It's not sufficient alone, though,
+ * because we may discover the dependencies in the wrong order.
+ * For example [A => B], [A => C] and then [B => C]. None of those
+ * dependencies is a combination of the already known ones, yet
+ * [A => C] is a combination of [A => B] and [B => C].
+ *
+ * FIXME Not sure the current NULL handling makes much sense. We assume
+ * that NULL is 0, so it's handled like a regular value
+ * (NULL == NULL), so all NULLs in a single column form a single
+ * group. Maybe that's not the right thing to do, especially with
+ * equality conditions - in that case NULLs are irrelevant. So
+ * maybe the right solution would be to just ignore NULL values?
+ *
+ * However simply "ignoring" the NULL values does not seem like
+ * a good idea - imagine columns A and B, where for each value of
+ * A, values in B are constant (same for the whole group) or NULL.
+ * Let's say only 10% of B values in each group is not NULL. Then
+ * ignoring the NULL values will result in 10x misestimate (and
+ * it's trivial to construct arbitrary errors). So maybe handling
+ * NULL values just like a regular value is the right thing here.
+ *
+ * Or maybe NULL values should be treated differently on each side
+ * of the dependency? E.g. as ignored on the left (condition) and
+ * as regular values on the right - this seems consistent with how
+ * equality clauses work, as equality clause means 'NOT NULL'.
+ * So if we say [A => B] then it may also imply "NOT NULL" on the
+ * right side.
+ */
+MVDependencies
+build_mv_dependencies(int numrows, HeapTuple *rows, int2vector *attrs,
+ VacAttrStats **stats)
+{
+ int i;
+ int numattrs = attrs->dim1;
+
+ /* result */
+ int ndeps = 0;
+ MVDependencies dependencies = NULL;
+ MultiSortSupport mss = multi_sort_init(2); /* 2 dimensions for now */
+
+ /* TODO Maybe this should be somehow related to the number of
+ * distinct columns in the two columns we're currently analyzing.
+ * Assuming the distribution is uniform, we should expected to
+ * observe in the sample - we can then use the average group
+ * size as a threshold. That seems better than a static approach.
+ */
+ int min_group_size = 3;
+
+ /* dimension indexes we'll check for associations [a => b] */
+ int dima, dimb;
+
+ /*
+ * We'll reuse the same array for all the 2-column combinations.
+ *
+ * It's possible to sort the sample rows directly, but this seemed
+ * somehow simples / less error prone. Another option would be to
+ * allocate the arrays for each SortItem separately, but that'd be
+ * significant overhead (not just CPU, but especially memory bloat).
+ */
+ SortItem * items = (SortItem*)palloc0(numrows * sizeof(SortItem));
+
+ Datum *values = (Datum*)palloc0(sizeof(Datum) * numrows * 2);
+ bool *isnull = (bool*)palloc0(sizeof(bool) * numrows * 2);
+
+ for (i = 0; i < numrows; i++)
+ {
+ items[i].values = &values[i * 2];
+ items[i].isnull = &isnull[i * 2];
+ }
+
+ Assert(numattrs >= 2);
+
+ /*
+ * Evaluate all possible combinations of [A => B], using a simple algorithm:
+ *
+ * (a) sort the data by [A,B]
+ * (b) split the data into groups by A (new group whenever a value changes)
+ * (c) count different values in the B column (again, value changes)
+ *
+ * TODO It should be rather simple to merge [A => B] and [A => C] into
+ * [A => B,C]. Just keep A constant, collect all the "implied" columns
+ * and you're done.
+ */
+ for (dima = 0; dima < numattrs; dima++)
+ {
+ /* prepare the sort function for the first dimension */
+ multi_sort_add_dimension(mss, 0, dima, stats);
+
+ for (dimb = 0; dimb < numattrs; dimb++)
+ {
+ SortItem current;
+
+ /* number of groups supporting / contradicting the dependency */
+ int n_supporting = 0;
+ int n_contradicting = 0;
+
+ /* counters valid within a group */
+ int group_size = 0;
+ int n_violations = 0;
+
+ int n_supporting_rows = 0;
+ int n_contradicting_rows = 0;
+
+ /* make sure the columns are different (A => A) */
+ if (dima == dimb)
+ continue;
+
+ /* prepare the sort function for the second dimension */
+ multi_sort_add_dimension(mss, 1, dimb, stats);
+
+ /* reset the values and isnull flags */
+ memset(values, 0, sizeof(Datum) * numrows * 2);
+ memset(isnull, 0, sizeof(bool) * numrows * 2);
+
+ /* accumulate all the data for both columns into an array and sort it */
+ for (i = 0; i < numrows; i++)
+ {
+ items[i].values[0]
+ = heap_getattr(rows[i], attrs->values[dima],
+ stats[dima]->tupDesc, &items[i].isnull[0]);
+
+ items[i].values[1]
+ = heap_getattr(rows[i], attrs->values[dimb],
+ stats[dimb]->tupDesc, &items[i].isnull[1]);
+ }
+
+ qsort_arg((void *) items, numrows, sizeof(SortItem),
+ multi_sort_compare, mss);
+
+ /*
+ * Walk through the array, split it into rows according to
+ * the A value, and count distinct values in the other one.
+ * If there's a single B value for the whole group, we count
+ * it as supporting the association, otherwise we count it
+ * as contradicting.
+ *
+ * Furthermore we require a group to have at least a certain
+ * number of rows to be considered useful for supporting the
+ * dependency. But when it's contradicting, use it always useful.
+ */
+
+ /* start with values from the first row */
+ current = items[0];
+ group_size = 1;
+
+ for (i = 1; i < numrows; i++)
+ {
+ /* end of the group */
+ if (multi_sort_compare_dim(0, &items[i], ¤t, mss) != 0)
+ {
+ /*
+ * If there are no contradicting rows, count it as
+ * supporting (otherwise contradicting), but only if
+ * the group is large enough.
+ *
+ * The requirement of a minimum group size makes it
+ * impossible to identify [unique,unique] cases, but
+ * that's probably a different case. This is more
+ * about [zip => city] associations etc.
+ *
+ * If there are violations, count the group/rows as
+ * a violation.
+ *
+ * It may ne neither, if the group is too small (does
+ * not contain at least min_group_size rows).
+ */
+ if ((n_violations == 0) && (group_size >= min_group_size))
+ {
+ n_supporting += 1;
+ n_supporting_rows += group_size;
+ }
+ else if (n_violations > 0)
+ {
+ n_contradicting += 1;
+ n_contradicting_rows += group_size;
+ }
+
+ /* current values start a new group */
+ n_violations = 0;
+ group_size = 0;
+ }
+ /* mismatch of a B value is contradicting */
+ else if (multi_sort_compare_dim(1, &items[i], ¤t, mss) != 0)
+ {
+ n_violations += 1;
+ }
+
+ current = items[i];
+ group_size += 1;
+ }
+
+ /* handle the last group (just like above) */
+ if ((n_violations == 0) && (group_size >= min_group_size))
+ {
+ n_supporting += 1;
+ n_supporting_rows += group_size;
+ }
+ else if (n_violations)
+ {
+ n_contradicting += 1;
+ n_contradicting_rows += group_size;
+ }
+
+ /*
+ * See if the number of rows supporting the association is at least
+ * 10x the number of rows violating the hypothetical dependency.
+ *
+ * TODO This is rather arbitrary limit - I guess it's possible to do
+ * some math to come up with a better rule (e.g. testing a hypothesis
+ * 'this is due to randomness'). We can create a contingency table
+ * from the values and use it for testing. Possibly only when
+ * there are no contradicting rows?
+ *
+ * TODO Also, if (a => b) and (b => a) at the same time, it pretty much
+ * means there's a 1:1 relation (or one is a 'label'), making the
+ * conditions rather redundant. Although it's possible that the
+ * query uses incompatible combination of values.
+ */
+ if (n_supporting_rows > (n_contradicting_rows * 10))
+ {
+ if (dependencies == NULL)
+ {
+ dependencies = (MVDependencies)palloc0(sizeof(MVDependenciesData));
+ dependencies->magic = MVSTAT_DEPS_MAGIC;
+ }
+ else
+ dependencies = repalloc(dependencies, offsetof(MVDependenciesData, deps)
+ + sizeof(MVDependency) * (dependencies->ndeps + 1));
+
+ /* update the */
+ dependencies->deps[ndeps] = (MVDependency)palloc0(sizeof(MVDependencyData));
+ dependencies->deps[ndeps]->a = attrs->values[dima];
+ dependencies->deps[ndeps]->b = attrs->values[dimb];
+
+ dependencies->ndeps = (++ndeps);
+ }
+ }
+ }
+
+ pfree(items);
+ pfree(values);
+ pfree(isnull);
+ pfree(stats);
+ pfree(mss);
+
+ return dependencies;
+}
+
+/*
+ * Store the dependencies into a bytea, so that it can be stored in the
+ * pg_mv_statistic catalog.
+ *
+ * Currently this only supports simple two-column rules, and stores them
+ * as a sequence of attnum pairs. In the future, this needs to be made
+ * more complex to support multiple columns on both sides of the
+ * implication (using AND on left, OR on right).
+ */
+bytea *
+serialize_mv_dependencies(MVDependencies dependencies)
+{
+ int i;
+
+ /* we need to store ndeps, and each needs 2 * int16 */
+ Size len = VARHDRSZ + offsetof(MVDependenciesData, deps)
+ + dependencies->ndeps * (sizeof(int16) * 2);
+
+ bytea * output = (bytea*)palloc0(len);
+
+ char * tmp = VARDATA(output);
+
+ SET_VARSIZE(output, len);
+
+ /* first, store the number of dimensions / items */
+ memcpy(tmp, dependencies, offsetof(MVDependenciesData, deps));
+ tmp += offsetof(MVDependenciesData, deps);
+
+ /* walk through the dependencies and copy both columns into the bytea */
+ for (i = 0; i < dependencies->ndeps; i++)
+ {
+ memcpy(tmp, &(dependencies->deps[i]->a), sizeof(int16));
+ tmp += sizeof(int16);
+
+ memcpy(tmp, &(dependencies->deps[i]->b), sizeof(int16));
+ tmp += sizeof(int16);
+ }
+
+ return output;
+}
+
+/*
+ * Reads serialized dependencies into MVDependencies structure.
+ */
+MVDependencies
+deserialize_mv_dependencies(bytea * data)
+{
+ int i;
+ Size expected_size;
+ MVDependencies dependencies;
+ char *tmp;
+
+ if (data == NULL)
+ return NULL;
+
+ if (VARSIZE_ANY_EXHDR(data) < offsetof(MVDependenciesData,deps))
+ elog(ERROR, "invalid MVDependencies size %ld (expected at least %ld)",
+ VARSIZE_ANY_EXHDR(data), offsetof(MVDependenciesData,deps));
+
+ /* read the MVDependencies header */
+ dependencies = (MVDependencies)palloc0(sizeof(MVDependenciesData));
+
+ /* initialize pointer to the data part (skip the varlena header) */
+ tmp = VARDATA(data);
+
+ /* get the header and perform basic sanity checks */
+ memcpy(dependencies, tmp, offsetof(MVDependenciesData, deps));
+ tmp += offsetof(MVDependenciesData, deps);
+
+ if (dependencies->magic != MVSTAT_DEPS_MAGIC)
+ {
+ pfree(dependencies);
+ elog(WARNING, "not a MV Dependencies (magic number mismatch)");
+ return NULL;
+ }
+
+ Assert(dependencies->ndeps > 0);
+
+ /* what bytea size do we expect for those parameters */
+ expected_size = offsetof(MVDependenciesData,deps) +
+ dependencies->ndeps * sizeof(int16) * 2;
+
+ if (VARSIZE_ANY_EXHDR(data) != expected_size)
+ elog(ERROR, "invalid dependencies size %ld (expected %ld)",
+ VARSIZE_ANY_EXHDR(data), expected_size);
+
+ /* allocate space for the MCV items */
+ dependencies = repalloc(dependencies, offsetof(MVDependenciesData,deps)
+ + (dependencies->ndeps * sizeof(MVDependency)));
+
+ for (i = 0; i < dependencies->ndeps; i++)
+ {
+ dependencies->deps[i] = (MVDependency)palloc0(sizeof(MVDependencyData));
+
+ memcpy(&(dependencies->deps[i]->a), tmp, sizeof(int16));
+ tmp += sizeof(int16);
+
+ memcpy(&(dependencies->deps[i]->b), tmp, sizeof(int16));
+ tmp += sizeof(int16);
+ }
+
+ return dependencies;
+}
+
+/* print some basic info about dependencies (number of dependencies) */
+Datum
+pg_mv_stats_dependencies_info(PG_FUNCTION_ARGS)
+{
+ bytea *data = PG_GETARG_BYTEA_P(0);
+ char *result;
+
+ MVDependencies dependencies = deserialize_mv_dependencies(data);
+
+ if (dependencies == NULL)
+ PG_RETURN_NULL();
+
+ result = palloc0(128);
+ snprintf(result, 128, "dependencies=%d", dependencies->ndeps);
+
+ /* FIXME free the deserialized data (pfree is not enough) */
+
+ PG_RETURN_TEXT_P(cstring_to_text(result));
+}
+
+/* print the dependencies
+ *
+ * TODO Would be nice if this knew the actual column names (instead of
+ * the attnums).
+ *
+ * FIXME This is really ugly and does not really check the lengths and
+ * strcpy/snprintf return values properly. Needs to be fixed.
+ */
+Datum
+pg_mv_stats_dependencies_show(PG_FUNCTION_ARGS)
+{
+ int i = 0;
+ bytea *data = PG_GETARG_BYTEA_P(0);
+ char *result = NULL;
+ int len = 0;
+
+ MVDependencies dependencies = deserialize_mv_dependencies(data);
+
+ if (dependencies == NULL)
+ PG_RETURN_NULL();
+
+ for (i = 0; i < dependencies->ndeps; i++)
+ {
+ MVDependency dependency = dependencies->deps[i];
+ char buffer[128];
+
+ int tmp = snprintf(buffer, 128, "%s%d => %d",
+ ((i == 0) ? "" : ", "), dependency->a, dependency->b);
+
+ if (tmp < 127)
+ {
+ if (result == NULL)
+ result = palloc0(len + tmp + 1);
+ else
+ result = repalloc(result, len + tmp + 1);
+
+ strcpy(result + len, buffer);
+ len += tmp;
+ }
+ }
+
+ PG_RETURN_TEXT_P(cstring_to_text(result));
+}
+
+bytea *
+fetch_mv_dependencies(Oid mvoid)
+{
+ Relation indrel;
+ SysScanDesc indscan;
+ ScanKeyData skey;
+ HeapTuple htup;
+ bytea *stadeps = NULL;
+
+ /* Prepare to scan pg_mv_statistic for entries having indrelid = this rel. */
+ ScanKeyInit(&skey,
+ ObjectIdAttributeNumber,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(mvoid));
+
+ indrel = heap_open(MvStatisticRelationId, AccessShareLock);
+ indscan = systable_beginscan(indrel, MvStatisticOidIndexId, true,
+ NULL, 1, &skey);
+
+ while (HeapTupleIsValid(htup = systable_getnext(indscan)))
+ {
+ bool isnull = false;
+ Datum deps = SysCacheGetAttr(MVSTATOID, htup,
+ Anum_pg_mv_statistic_stadeps, &isnull);
+
+ Assert(!isnull);
+
+ stadeps = DatumGetByteaP(deps);
+
+ break;
+ }
+
+ systable_endscan(indscan);
+
+ heap_close(indrel, AccessShareLock);
+
+ /* TODO maybe save the list into relcache, as in RelationGetIndexList
+ * (which was used as an inspiration of this one)?. */
+
+ return stadeps;
+}
diff --git a/src/include/catalog/indexing.h b/src/include/catalog/indexing.h
index a680229..22bb781 100644
--- a/src/include/catalog/indexing.h
+++ b/src/include/catalog/indexing.h
@@ -173,6 +173,11 @@ DECLARE_UNIQUE_INDEX(pg_largeobject_loid_pn_index, 2683, on pg_largeobject using
DECLARE_UNIQUE_INDEX(pg_largeobject_metadata_oid_index, 2996, on pg_largeobject_metadata using btree(oid oid_ops));
#define LargeObjectMetadataOidIndexId 2996
+DECLARE_UNIQUE_INDEX(pg_mv_statistic_oid_index, 3380, on pg_mv_statistic using btree(oid oid_ops));
+#define MvStatisticOidIndexId 3380
+DECLARE_INDEX(pg_mv_statistic_relid_index, 3379, on pg_mv_statistic using btree(starelid oid_ops));
+#define MvStatisticRelidIndexId 3379
+
DECLARE_UNIQUE_INDEX(pg_namespace_nspname_index, 2684, on pg_namespace using btree(nspname name_ops));
#define NamespaceNameIndexId 2684
DECLARE_UNIQUE_INDEX(pg_namespace_oid_index, 2685, on pg_namespace using btree(oid oid_ops));
diff --git a/src/include/catalog/pg_mv_statistic.h b/src/include/catalog/pg_mv_statistic.h
new file mode 100644
index 0000000..81ec23b
--- /dev/null
+++ b/src/include/catalog/pg_mv_statistic.h
@@ -0,0 +1,69 @@
+/*-------------------------------------------------------------------------
+ *
+ * pg_mv_statistic.h
+ * definition of the system "multivariate statistic" relation (pg_mv_statistic)
+ * along with the relation's initial contents.
+ *
+ *
+ * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/catalog/pg_mv_statistic.h
+ *
+ * NOTES
+ * the genbki.pl script reads this file and generates .bki
+ * information from the DATA() statements.
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef PG_MV_STATISTIC_H
+#define PG_MV_STATISTIC_H
+
+#include "catalog/genbki.h"
+
+/* ----------------
+ * pg_mv_statistic definition. cpp turns this into
+ * typedef struct FormData_pg_mv_statistic
+ * ----------------
+ */
+#define MvStatisticRelationId 3381
+
+CATALOG(pg_mv_statistic,3381)
+{
+ /* These fields form the unique key for the entry: */
+ Oid starelid; /* relation containing attributes */
+
+ /* statistics requested to build */
+ bool deps_enabled; /* analyze dependencies? */
+
+ /* statistics that are available (if requested) */
+ bool deps_built; /* dependencies were built */
+
+ /* variable-length fields start here, but we allow direct access to stakeys */
+ int2vector stakeys; /* array of column keys */
+
+#ifdef CATALOG_VARLEN
+ bytea stadeps; /* dependencies (serialized) */
+#endif
+
+} FormData_pg_mv_statistic;
+
+/* ----------------
+ * Form_pg_mv_statistic corresponds to a pointer to a tuple with
+ * the format of pg_mv_statistic relation.
+ * ----------------
+ */
+typedef FormData_pg_mv_statistic *Form_pg_mv_statistic;
+
+/* ----------------
+ * compiler constants for pg_attrdef
+ * ----------------
+ */
+#define Natts_pg_mv_statistic 5
+#define Anum_pg_mv_statistic_starelid 1
+#define Anum_pg_mv_statistic_deps_enabled 2
+#define Anum_pg_mv_statistic_deps_built 3
+#define Anum_pg_mv_statistic_stakeys 4
+#define Anum_pg_mv_statistic_stadeps 5
+
+#endif /* PG_MV_STATISTIC_H */
diff --git a/src/include/catalog/pg_proc.h b/src/include/catalog/pg_proc.h
index 8890ade..f728d88 100644
--- a/src/include/catalog/pg_proc.h
+++ b/src/include/catalog/pg_proc.h
@@ -2712,6 +2712,11 @@ DESCR("current user privilege on any column by rel name");
DATA(insert OID = 3029 ( has_any_column_privilege PGNSP PGUID 12 10 0 0 0 f f f f t f s 2 0 16 "26 25" _null_ _null_ _null_ _null_ has_any_column_privilege_id _null_ _null_ _null_ ));
DESCR("current user privilege on any column by rel oid");
+DATA(insert OID = 3284 ( pg_mv_stats_dependencies_info PGNSP PGUID 12 1 0 0 0 f f f f t f i 1 0 25 "17" _null_ _null_ _null_ _null_ pg_mv_stats_dependencies_info _null_ _null_ _null_ ));
+DESCR("multivariate stats: functional dependencies info");
+DATA(insert OID = 3285 ( pg_mv_stats_dependencies_show PGNSP PGUID 12 1 0 0 0 f f f f t f i 1 0 25 "17" _null_ _null_ _null_ _null_ pg_mv_stats_dependencies_show _null_ _null_ _null_ ));
+DESCR("multivariate stats: functional dependencies show");
+
DATA(insert OID = 1928 ( pg_stat_get_numscans PGNSP PGUID 12 1 0 0 0 f f f f t f s 1 0 20 "26" _null_ _null_ _null_ _null_ pg_stat_get_numscans _null_ _null_ _null_ ));
DESCR("statistics: number of scans done for table/index");
DATA(insert OID = 1929 ( pg_stat_get_tuples_returned PGNSP PGUID 12 1 0 0 0 f f f f t f s 1 0 20 "26" _null_ _null_ _null_ _null_ pg_stat_get_tuples_returned _null_ _null_ _null_ ));
diff --git a/src/include/catalog/toasting.h b/src/include/catalog/toasting.h
index fb2f035..724a169 100644
--- a/src/include/catalog/toasting.h
+++ b/src/include/catalog/toasting.h
@@ -49,6 +49,7 @@ extern void BootstrapToastTable(char *relName,
DECLARE_TOAST(pg_attrdef, 2830, 2831);
DECLARE_TOAST(pg_constraint, 2832, 2833);
DECLARE_TOAST(pg_description, 2834, 2835);
+DECLARE_TOAST(pg_mv_statistic, 3288, 3289);
DECLARE_TOAST(pg_proc, 2836, 2837);
DECLARE_TOAST(pg_rewrite, 2838, 2839);
DECLARE_TOAST(pg_seclabel, 3598, 3599);
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 38469ef..3a0e7c4 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -414,6 +414,7 @@ typedef enum NodeTag
T_WithClause,
T_CommonTableExpr,
T_RoleSpec,
+ T_StatisticsDef,
/*
* TAGS FOR REPLICATION GRAMMAR PARSE NODES (replnodes.h)
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 2893cef..81ca159 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -570,6 +570,14 @@ typedef struct ColumnDef
int location; /* parse location, or -1 if none/unknown */
} ColumnDef;
+typedef struct StatisticsDef
+{
+ NodeTag type;
+ List *keys; /* String nodes naming referenced column(s) */
+ List *options; /* list of DefElem nodes */
+} StatisticsDef;
+
+
/*
* TableLikeClause - CREATE TABLE ( ... LIKE ... ) clause
*/
@@ -1362,7 +1370,8 @@ typedef enum AlterTableType
AT_ReplicaIdentity, /* REPLICA IDENTITY */
AT_EnableRowSecurity, /* ENABLE ROW SECURITY */
AT_DisableRowSecurity, /* DISABLE ROW SECURITY */
- AT_GenericOptions /* OPTIONS (...) */
+ AT_GenericOptions, /* OPTIONS (...) */
+ AT_AddStatistics /* add statistics */
} AlterTableType;
typedef struct ReplicaIdentityStmt
diff --git a/src/include/utils/mvstats.h b/src/include/utils/mvstats.h
new file mode 100644
index 0000000..5c8643d
--- /dev/null
+++ b/src/include/utils/mvstats.h
@@ -0,0 +1,86 @@
+/*-------------------------------------------------------------------------
+ *
+ * mvstats.h
+ * Multivariate statistics and selectivity estimation functions.
+ *
+ *
+ * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/utils/mvstats.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef MVSTATS_H
+#define MVSTATS_H
+
+#include "commands/vacuum.h"
+
+/*
+ * Basic info about the stats, used when choosing what to use
+ *
+ * TODO Add info about what statistics is available (histogram, MCV,
+ * hashed MCV, functional dependencies).
+ */
+typedef struct MVStatsData {
+ Oid mvoid; /* OID of the stats in pg_mv_statistic */
+ int2vector *stakeys; /* attnums for columns in the stats */
+ bool deps_built; /* functional dependencies available */
+} MVStatsData;
+
+typedef struct MVStatsData *MVStats;
+
+
+#define MVSTATS_MAX_DIMENSIONS 8 /* max number of attributes */
+
+/* An associative rule, tracking [a => b] dependency.
+ *
+ * TODO Make this work with multiple columns on both sides.
+ */
+typedef struct MVDependencyData {
+ int16 a;
+ int16 b;
+} MVDependencyData;
+
+typedef MVDependencyData* MVDependency;
+
+typedef struct MVDependenciesData {
+ uint32 magic; /* magic constant marker */
+ int32 ndeps; /* number of dependencies */
+ MVDependency deps[1]; /* XXX why not a pointer? */
+} MVDependenciesData;
+
+typedef MVDependenciesData* MVDependencies;
+
+#define MVSTAT_DEPS_MAGIC 0xB4549A2C /* marks serialized bytea */
+#define MVSTAT_DEPS_TYPE_BASIC 1 /* basic dependencies type */
+
+/*
+ * TODO Maybe fetching the histogram/MCV list separately is inefficient?
+ * Consider adding a single `fetch_stats` method, fetching all
+ * stats specified using flags (or something like that).
+ */
+MVStats list_mv_stats(Oid relid, int *nstats, bool built_only);
+
+bytea * fetch_mv_dependencies(Oid mvoid);
+
+bytea * serialize_mv_dependencies(MVDependencies dependencies);
+
+/* deserialization of stats (serialization is private to analyze) */
+MVDependencies deserialize_mv_dependencies(bytea * data);
+
+/* FIXME this probably belongs somewhere else (not to operations stats) */
+extern Datum pg_mv_stats_dependencies_info(PG_FUNCTION_ARGS);
+extern Datum pg_mv_stats_dependencies_show(PG_FUNCTION_ARGS);
+
+MVDependencies
+build_mv_dependencies(int numrows, HeapTuple *rows,
+ int2vector *attrs,
+ VacAttrStats **stats);
+
+void build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
+ int natts, VacAttrStats **vacattrstats);
+
+void update_mv_stats(Oid relid, MVDependencies dependencies);
+
+#endif
diff --git a/src/include/utils/syscache.h b/src/include/utils/syscache.h
index ba0b090..12147ab 100644
--- a/src/include/utils/syscache.h
+++ b/src/include/utils/syscache.h
@@ -66,6 +66,7 @@ enum SysCacheIdentifier
INDEXRELID,
LANGNAME,
LANGOID,
+ MVSTATOID,
NAMESPACENAME,
NAMESPACEOID,
OPERNAMENSP,
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 1788270..f0117ca 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1353,6 +1353,14 @@ pg_matviews| SELECT n.nspname AS schemaname,
LEFT JOIN pg_namespace n ON ((n.oid = c.relnamespace)))
LEFT JOIN pg_tablespace t ON ((t.oid = c.reltablespace)))
WHERE (c.relkind = 'm'::"char");
+pg_mv_stats| SELECT n.nspname AS schemaname,
+ c.relname AS tablename,
+ s.stakeys AS attnums,
+ length(s.stadeps) AS depsbytes,
+ pg_mv_stats_dependencies_info(s.stadeps) AS depsinfo
+ FROM ((pg_mv_statistic s
+ JOIN pg_class c ON ((c.oid = s.starelid)))
+ LEFT JOIN pg_namespace n ON ((n.oid = c.relnamespace)));
pg_policies| SELECT n.nspname AS schemaname,
c.relname AS tablename,
pol.polname AS policyname,
diff --git a/src/test/regress/expected/sanity_check.out b/src/test/regress/expected/sanity_check.out
index c7be273..00f5fe7 100644
--- a/src/test/regress/expected/sanity_check.out
+++ b/src/test/regress/expected/sanity_check.out
@@ -113,6 +113,7 @@ pg_inherits|t
pg_language|t
pg_largeobject|t
pg_largeobject_metadata|t
+pg_mv_statistic|t
pg_namespace|t
pg_opclass|t
pg_operator|t
--
2.0.5
[text/x-diff] 0002-clause-reduction-using-functional-dependencies.patch (44.5K, ../../5519E9B2.5080105@2ndquadrant.com/3-0002-clause-reduction-using-functional-dependencies.patch)
download | inline diff:
From 47a48180be115db2fa29ac659f4e4f259e01600d Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@pgaddict.com>
Date: Fri, 16 Jan 2015 22:33:41 +0100
Subject: [PATCH 2/5] clause reduction using functional dependencies
During planning, use functional dependencies to decide
which clauses to skip during cardinality estimation.
Initial and rather simplistic implementation.
This only works with regular WHERE clauses, not clauses
used for joining.
Note: The clause_is_mv_compatible() needs to identify the
relation (so that we can fetch the list of multivariate stats
by OID). planner_rt_fetch() seems like the appropriate way to
get the relation OID, but apparently it only works with simple
vars. Maybe examine_variable() would make this work with more
complex vars too?
Includes regression tests analyzing functional dependencies
(part of ANALYZE) on several datasets (no dependencies, no
transitive dependencies, ...).
Checks that a query with conditions on two columns, where one (B)
is functionally dependent on the other one (A), correctly ignores
the clause on (B) and chooses bitmap index scan instead of plain
index scan (which is what happens otherwise, thanks to assumption
of independence).
Note: Functional dependencies only work with equality clauses,
no inequalities etc.
---
src/backend/commands/analyze.c | 1 +
src/backend/commands/tablecmds.c | 9 +-
src/backend/optimizer/path/clausesel.c | 650 +++++++++++++++++++++++++-
src/backend/utils/mvstats/common.c | 5 +-
src/include/catalog/pg_proc.h | 4 +-
src/include/utils/mvstats.h | 23 +-
src/test/regress/expected/mv_dependencies.out | 175 +++++++
src/test/regress/parallel_schedule | 3 +
src/test/regress/serial_schedule | 1 +
src/test/regress/sql/mv_dependencies.sql | 153 ++++++
10 files changed, 1013 insertions(+), 11 deletions(-)
create mode 100644 src/test/regress/expected/mv_dependencies.out
create mode 100644 src/test/regress/sql/mv_dependencies.sql
diff --git a/src/backend/commands/analyze.c b/src/backend/commands/analyze.c
index f82fcf5..e247f84 100644
--- a/src/backend/commands/analyze.c
+++ b/src/backend/commands/analyze.c
@@ -115,6 +115,7 @@ static void update_attstats(Oid relid, bool inh,
static Datum std_fetch_func(VacAttrStatsP stats, int rownum, bool *isNull);
static Datum ind_fetch_func(VacAttrStatsP stats, int rownum, bool *isNull);
+
/*
* analyze_rel() -- analyze one relation
*/
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index a321755..965d342 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -420,6 +420,7 @@ static void ATExecEnableRowSecurity(Relation rel);
static void ATExecDisableRowSecurity(Relation rel);
static void ATExecAddStatistics(AlteredTableInfo *tab, Relation rel,
StatisticsDef *def, LOCKMODE lockmode);
+
static void copy_relation_data(SMgrRelation rel, SMgrRelation dst,
ForkNumber forkNum, char relpersistence);
static const char *storage_name(char c);
@@ -11900,7 +11901,7 @@ static void ATExecAddStatistics(AlteredTableInfo *tab, Relation rel,
Relation mvstatrel;
/* by default build everything */
- bool build_dependencies = true;
+ bool build_dependencies = false;
Assert(IsA(def, StatisticsDef));
@@ -11962,6 +11963,12 @@ static void ATExecAddStatistics(AlteredTableInfo *tab, Relation rel,
opt->defname)));
}
+ /* check that at least some statistics were requested */
+ if (! build_dependencies)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("no statistics type (dependencies) was requested")));
+
/* sort the attnums and build int2vector */
qsort(attnums, numcols, sizeof(int16), compare_int16);
stakeys = buildint2vector(attnums, numcols);
diff --git a/src/backend/optimizer/path/clausesel.c b/src/backend/optimizer/path/clausesel.c
index dcac1c1..e742827 100644
--- a/src/backend/optimizer/path/clausesel.c
+++ b/src/backend/optimizer/path/clausesel.c
@@ -24,6 +24,14 @@
#include "utils/lsyscache.h"
#include "utils/selfuncs.h"
+#include "utils/mvstats.h"
+#include "catalog/pg_collation.h"
+#include "utils/typcache.h"
+
+#include "parser/parsetree.h"
+
+
+#include <stdio.h>
/*
* Data structure for accumulating info about possible range-query
@@ -43,6 +51,16 @@ static void addRangeClause(RangeQueryClause **rqlist, Node *clause,
bool varonleft, bool isLTsel, Selectivity s2);
+static bool clause_is_mv_compatible(PlannerInfo *root, Node *clause, Oid varRelid,
+ Oid *relid, AttrNumber *attnum, SpecialJoinInfo *sjinfo);
+
+static Bitmapset *collect_mv_attnums(PlannerInfo *root, List *clauses,
+ Oid varRelid, Oid *relid, SpecialJoinInfo *sjinfo);
+
+static List *clauselist_apply_dependencies(PlannerInfo *root, List *clauses,
+ Oid varRelid, int nmvstats, MVStats mvstats,
+ SpecialJoinInfo *sjinfo);
+
/****************************************************************************
* ROUTINES TO COMPUTE SELECTIVITIES
****************************************************************************/
@@ -61,7 +79,7 @@ static void addRangeClause(RangeQueryClause **rqlist, Node *clause,
* subclauses. However, that's only right if the subclauses have independent
* probabilities, and in reality they are often NOT independent. So,
* we want to be smarter where we can.
-
+ *
* Currently, the only extra smarts we have is to recognize "range queries",
* such as "x > 34 AND x < 42". Clauses are recognized as possible range
* query components if they are restriction opclauses whose operators have
@@ -88,6 +106,76 @@ static void addRangeClause(RangeQueryClause **rqlist, Node *clause,
*
* Of course this is all very dependent on the behavior of
* scalarltsel/scalargtsel; perhaps some day we can generalize the approach.
+ *
+ *
+ * Multivariate statististics
+ * --------------------------
+ * This also uses multivariate stats to estimate combinations of conditions,
+ * in a way attempting to minimize the overhead when there are no suitable
+ * multivariate stats.
+ *
+ * The following checks are performed (in this order), and the optimizer
+ * falls back to regular stats on the first 'false'.
+ *
+ * NOTE: This explains how this works with all the patches applied, not
+ * just the functional dependencies.
+ *
+ * (1) check that at least two columns are referenced from conditions
+ * compatible with multivariate stats
+ *
+ * If there are no conditions that might be handled by multivariate
+ * stats, or if the conditions reference just a single column, it
+ * makes no sense to use multivariate stats.
+ *
+ * What conditions are compatible with multivariate stats is decided
+ * by clause_is_mv_compatible(). At this moment, only simple conditions
+ * of the form "column operator constant" (for simple comparison
+ * operators), and IS NULL / IS NOT NULL are considered compatible
+ * with multivariate statistics.
+ *
+ * (2) reduce the clauses using functional dependencies
+ *
+ * This simply attempts to 'reduce' the clauses by applying functional
+ * dependencies. For example if there are two clauses:
+ *
+ * WHERE (a = 1) AND (b = 2)
+ *
+ * and we know that 'a' determines the value of 'b', we may remove
+ * the second condition (b = 2) when computing the selectivity.
+ * This is of course tricky - see mvstats/dependencies.c for details.
+ *
+ * After the reduction, step (1) is to be repeated.
+ *
+ * (3) check if there are multivariate stats built on the columns
+ *
+ * If there are no multivariate statistics, we have to fall back to
+ * the regular stats. We might perform checks (1) and (2) in reverse
+ * order, i.e. first check if there are multivariate statistics and
+ * then collect the attributes only if needed. The assumption is
+ * that checking the clauses is cheaper than querying the catalog,
+ * so this check is performed first.
+ *
+ * (4) choose the stats matching the most columns (at least two)
+ *
+ * If there are multiple instances of multivariate statistics (e.g.
+ * built on different sets of columns), we choose the stats covering
+ * the most columns from step (1). It may happen that all available
+ * stats match just a single column - for example with conditions
+ *
+ * WHERE a = 1 AND b = 2
+ *
+ * and statistics built on (a,c) and (b,c). In such case just fall
+ * back to the regular stats because it makes no sense to use the
+ * multivariate statistics.
+ *
+ * This selection criteria (the most columns) is certainly very
+ * simple and definitely not optimal - it's simple to come up with
+ * examples where other approaches work better. More about this
+ * at choose_mv_statistics().
+ *
+ * (5) use the multivariate stats to estimate matching clauses
+ *
+ * (6) estimate the remaining clauses using the regular statistics
*/
Selectivity
clauselist_selectivity(PlannerInfo *root,
@@ -100,6 +188,14 @@ clauselist_selectivity(PlannerInfo *root,
RangeQueryClause *rqlist = NULL;
ListCell *l;
+ /* processing mv stats */
+ Oid relid = InvalidOid;
+ int nmvstats = 0;
+ MVStats mvstats = NULL;
+
+ /* attributes in mv-compatible clauses */
+ Bitmapset *mvattnums = NULL;
+
/*
* If there's exactly one clause, then no use in trying to match up pairs,
* so just go directly to clause_selectivity().
@@ -108,6 +204,28 @@ clauselist_selectivity(PlannerInfo *root,
return clause_selectivity(root, (Node *) linitial(clauses),
varRelid, jointype, sjinfo);
+ /* collect attributes referenced by mv-compatible clauses */
+ mvattnums = collect_mv_attnums(root, clauses, varRelid, &relid, sjinfo);
+
+ /*
+ * If there are mv-compatible clauses, referencing at least two
+ * different columns (otherwise it makes no sense to use mv stats),
+ * try to reduce the clauses using functional dependencies, and
+ * recollect the attributes from the reduced list.
+ *
+ * We don't need to select a single statistics for this - we can
+ * apply all the functional dependencies we have.
+ */
+ if (bms_num_members(mvattnums) >= 2)
+ {
+ /* fetch info from the catalog (not the serialized stats yet) */
+ mvstats = list_mv_stats(relid, &nmvstats, true);
+
+ /* reduce clauses by applying functional dependencies rules */
+ clauses = clauselist_apply_dependencies(root, clauses, varRelid,
+ nmvstats, mvstats, sjinfo);
+ }
+
/*
* Initial scan over clauses. Anything that doesn't look like a potential
* rangequery clause gets multiplied into s1 and forgotten. Anything that
@@ -782,3 +900,533 @@ clause_selectivity(PlannerInfo *root,
return s1;
}
+
+/*
+ * Collect attributes from mv-compatible clauses.
+ */
+static Bitmapset *
+collect_mv_attnums(PlannerInfo *root, List *clauses, Oid varRelid,
+ Oid *relid, SpecialJoinInfo *sjinfo)
+{
+ Bitmapset *attnums = NULL;
+ ListCell *l;
+
+ /*
+ * Walk through the clauses and identify the ones we can estimate
+ * using multivariate stats, and remember the relid/columns. We'll
+ * then cross-check if we have suitable stats, and only if needed
+ * we'll split the clauses into multivariate and regular lists.
+ *
+ * For now we're only interested in RestrictInfo nodes with nested
+ * OpExpr, using either a range or equality.
+ */
+ foreach (l, clauses)
+ {
+ AttrNumber attnum;
+ Node *clause = (Node *) lfirst(l);
+
+ /* ignore the result for now - we only need the info */
+ if (clause_is_mv_compatible(root, clause, varRelid, relid, &attnum, sjinfo))
+ attnums = bms_add_member(attnums, attnum);
+ }
+
+ /*
+ * If there are not at least two attributes referenced by the clause(s),
+ * we can throw everything out (as we'll revert to simple stats).
+ */
+ if (bms_num_members(attnums) <= 1)
+ {
+ if (attnums != NULL)
+ pfree(attnums);
+ attnums = NULL;
+ *relid = InvalidOid;
+ }
+
+ return attnums;
+}
+
+/*
+ * Determines whether the clause is compatible with multivariate stats,
+ * and if it is, returns some additional information - varno (index
+ * into simple_rte_array) and a bitmap of attributes. This is then
+ * used to fetch related multivariate statistics.
+ *
+ * At this moment we only support basic conditions of the form
+ *
+ * variable OP constant
+ *
+ * where OP is one of [=,<,<=,>=,>] (which is however determined by
+ * looking at the associated function for estimating selectivity, just
+ * like with the single-dimensional case).
+ *
+ * TODO Support 'OR clauses' - shouldn't be all that difficult to
+ * evaluate them using multivariate stats.
+ */
+static bool
+clause_is_mv_compatible(PlannerInfo *root, Node *clause, Oid varRelid,
+ Oid *relid, AttrNumber *attnum, SpecialJoinInfo *sjinfo)
+{
+
+ if (IsA(clause, RestrictInfo))
+ {
+ RestrictInfo *rinfo = (RestrictInfo *) clause;
+
+ /* Pseudoconstants are not really interesting here. */
+ if (rinfo->pseudoconstant)
+ return false;
+
+ /* no support for OR clauses at this point */
+ if (rinfo->orclause)
+ return false;
+
+ /* get the actual clause from the RestrictInfo (it's not an OR clause) */
+ clause = (Node*)rinfo->clause;
+
+ /* only simple opclauses are compatible with multivariate stats */
+ if (! is_opclause(clause))
+ return false;
+
+ /* we don't support join conditions at this moment */
+ if (treat_as_join_clause(clause, rinfo, varRelid, sjinfo))
+ return false;
+
+ /* is it 'variable op constant' ? */
+ if (list_length(((OpExpr *) clause)->args) == 2)
+ {
+ OpExpr *expr = (OpExpr *) clause;
+ bool varonleft = true;
+ bool ok;
+
+ ok = (bms_membership(rinfo->clause_relids) == BMS_SINGLETON) &&
+ (is_pseudo_constant_clause_relids(lsecond(expr->args),
+ rinfo->right_relids) ||
+ (varonleft = false,
+ is_pseudo_constant_clause_relids(linitial(expr->args),
+ rinfo->left_relids)));
+
+ if (ok)
+ {
+ RangeTblEntry * rte;
+ Var * var = (varonleft) ? linitial(expr->args) : lsecond(expr->args);
+
+ /*
+ * Simple variables only - otherwise the planner_rt_fetch seems to fail
+ * (return NULL).
+ *
+ * TODO Maybe use examine_variable() would fix that?
+ */
+ if (! (IsA(var, Var) && (varRelid == 0 || varRelid == var->varno)))
+ return false;
+
+ /*
+ * Only consider this variable if (varRelid == 0) or when the varno
+ * matches varRelid (see explanation at clause_selectivity).
+ *
+ * FIXME I suspect this may not be really necessary. The (varRelid == 0)
+ * part seems to be enforced by treat_as_join_clause().
+ */
+ if (! ((varRelid == 0) || (varRelid == var->varno)))
+ return false;
+
+ /* Also skip special varno values, and system attributes ... */
+ if ((IS_SPECIAL_VARNO(var->varno)) || (! AttrNumberIsForUserDefinedAttr(var->varattno)))
+ return false;
+
+ /* Lookup info about the base relation (we need to pass the OID out) */
+ rte = planner_rt_fetch(var->varno, root);
+ *relid = rte->relid;
+
+ /*
+ * If it's not a "<" or ">" or "=" operator, just ignore the
+ * clause. Otherwise note the relid and attnum for the variable.
+ * This uses the function for estimating selectivity, ont the
+ * operator directly (a bit awkward, but well ...).
+ */
+ switch (get_oprrest(expr->opno))
+ {
+ case F_EQSEL:
+ *attnum = var->varattno;
+ return true;
+ }
+ }
+ }
+ }
+
+ return false;
+
+}
+
+/*
+ * Performs reduction of clauses using functional dependencies, i.e.
+ * removes clauses that are considered redundant. It simply walks
+ * through dependencies, and checks whether the dependency 'matches'
+ * the clauses, i.e. if there's a clause matching the condition. If yes,
+ * all clauses matching the implied part of the dependency are removed
+ * from the list.
+ *
+ * This simply looks at attnums references by the clauses, not at the
+ * type of the operator (equality, inequality, ...). This may not be the
+ * right way to do - it certainly works best for equalities, which is
+ * naturally consistent with functional dependencies (implications).
+ * It's not clear that other operators are handled sensibly - for
+ * example for inequalities, like
+ *
+ * WHERE (A >= 10) AND (B <= 20)
+ *
+ * and a trivial case where [A == B], resulting in symmetric pair of
+ * rules [A => B], [B => A], it's rather clear we can't remove either of
+ * those clauses.
+ *
+ * That only highlights that functional dependencies are most suitable
+ * for label-like data, where using non-equality operators is very rare.
+ * Using the common city/zipcode example, clauses like
+ *
+ * (zipcode <= 12345)
+ *
+ * or
+ *
+ * (cityname >= 'Washington')
+ *
+ * are rare. So restricting the reduction to equality should not harm
+ * the usefulness / applicability.
+ *
+ * The other assumption is that this assumes 'compatible' clauses. For
+ * example by using mismatching zip code and city name, this is unable
+ * to identify the discrepancy and eliminates one of the clauses. The
+ * usual approach (multiplying both selectivities) thus produces a more
+ * accurate estimate, although mostly by luck - the multiplication
+ * comes from assumption of statistical independence of the two
+ * conditions (which is not not valid in this case), but moves the
+ * estimate in the right direction (towards 0%).
+ *
+ * This might be somewhat improved by cross-checking the selectivities
+ * against MCV and/or histogram.
+ *
+ * The implementation needs to be careful about cyclic rules, i.e. rules
+ * like [A => B] and [B => A] at the same time. This must not reduce
+ * clauses on both attributes at the same time.
+ *
+ * Technically we might consider selectivities here too, somehow. E.g.
+ * when (A => B) and (B => A), we might use the clauses with minimum
+ * selectivity.
+ *
+ * TODO Consider restricting the reduction to equality clauses. Or maybe
+ * use equality classes somehow?
+ *
+ * TODO Merge this docs to dependencies.c, as it's saying mostly the
+ * same things as the comments there.
+ */
+static List *
+clauselist_apply_dependencies(PlannerInfo *root, List *clauses, Oid varRelid,
+ int nmvstats, MVStats mvstats, SpecialJoinInfo *sjinfo)
+{
+ int i;
+ ListCell *lc;
+ List * reduced_clauses = NIL;
+ Oid relid;
+
+ /*
+ * preallocate space for all clauses, including non-mv-compatible,
+ * so that we don't need to reallocate the arrays repeatedly
+ *
+ * XXX This assumes each clause references exactly one Var, so the
+ * arrays are sized accordingly - for functional dependencies
+ * this is safe, because it only works with Var=Const.
+ */
+ bool *reduced;
+ AttrNumber *mvattnums;
+ Node **mvclauses;
+ int nmvclauses = 0; /* number clauses in the arrays */
+
+ /*
+ * matrix of (natts x natts), 1 means x=>y
+ *
+ * This serves two purposes - first, it merges dependencies from all
+ * the statistics, second it makes generating all the transitive
+ * dependencies easier.
+ *
+ * We need to build this only for attributes from the dependencies,
+ * not for all attributes in the table.
+ *
+ * We can't do that only for attributes from the clauses, because we
+ * want to build transitive dependencies (including those going
+ * through attributes not listed in the stats).
+ *
+ * This only works for A=>B dependencies, not sure how to do that
+ * for complex dependencies.
+ */
+ bool *deps_matrix;
+ int deps_natts; /* size of the matric */
+
+ /* mapping attnum <=> matrix index */
+ int *deps_idx_to_attnum;
+ int *deps_attnum_to_idx;
+
+ /* attnums in dependencies and clauses (and intersection) */
+ Bitmapset *deps_attnums = NULL;
+ Bitmapset *clause_attnums = NULL;
+ Bitmapset *intersect_attnums = NULL;
+
+ int attnum, attidx, attnum_max;
+
+ bool has_deps_built = false;
+
+ /* see if there's at least one statistics with dependencies */
+ for (i = 0; i < nmvstats; i++)
+ {
+ if (mvstats[i].deps_built)
+ {
+ has_deps_built = true;
+ break;
+ }
+ }
+
+ /* no dependencies available - return the original clauses */
+ if (! has_deps_built)
+ return clauses;
+
+ mvclauses = (Node**)palloc0(list_length(clauses) * sizeof(Node*));
+ mvattnums = (AttrNumber*)palloc0(list_length(clauses) * sizeof(AttrNumber));
+
+ /*
+ * Walk through the clauses - clauses that are not mv-compatible copy
+ * directly into the result list, and mv-compatible ones store into
+ * an array of clauses (and remember the attnumb in another array).
+ */
+ foreach (lc, clauses)
+ {
+ AttrNumber attnum;
+ Node *clause = (Node *) lfirst(lc);
+ if (! clause_is_mv_compatible(root, clause, varRelid, &relid, &attnum, sjinfo))
+ reduced_clauses = lappend(reduced_clauses, clause);
+ else
+ {
+ mvclauses[nmvclauses] = clause;
+ mvattnums[nmvclauses] = attnum;
+ nmvclauses++;
+
+ clause_attnums = bms_add_member(clause_attnums, attnum);
+ }
+ }
+
+ /*
+ * we need at least two clauses referencing two different attributes
+ * referencing to do the reduction
+ */
+ if ((nmvclauses < 2) || (bms_num_members(clause_attnums) < 2))
+ {
+ pfree(mvattnums);
+ pfree(mvclauses);
+
+ bms_free(clause_attnums);
+ list_free(reduced_clauses);
+
+ return clauses;
+ }
+
+ reduced = (bool*)palloc0(list_length(clauses) * sizeof(bool));
+
+ /* build the dependency matrix */
+ attnum_max = -1;
+ for (i = 0; i < nmvstats; i++)
+ {
+ int j;
+ int2vector *stakeys = mvstats[i].stakeys;
+
+ /* skip stats without functional dependencies built */
+ if (! mvstats[i].deps_built)
+ continue;
+
+ for (j = 0; j < stakeys->dim1; j++)
+ {
+ int attnum = stakeys->values[j];
+ deps_attnums = bms_add_member(deps_attnums, attnum);
+
+ /* keep the max attnum in the dependencies */
+ attnum_max = (attnum > attnum_max) ? attnum : attnum_max;
+ }
+ }
+
+ /*
+ * We need at least two matching attributes in the clauses and
+ * dependencies, otherwise we can't reduce anything.
+ */
+ intersect_attnums = bms_intersect(clause_attnums, deps_attnums);
+ if (bms_num_members(intersect_attnums) < 2)
+ {
+ pfree(mvattnums);
+ pfree(mvclauses);
+
+ bms_free(clause_attnums);
+ bms_free(deps_attnums);
+ bms_free(intersect_attnums);
+
+ list_free(reduced_clauses);
+
+ return clauses;
+ }
+
+ /* allocate the matrix and mappings */
+ deps_natts = bms_num_members(deps_attnums);
+ deps_matrix = (bool*)palloc0(deps_natts * deps_natts * sizeof(int));
+ deps_idx_to_attnum = (int*)palloc0(deps_natts * sizeof(int));
+ deps_attnum_to_idx = (int*)palloc0((attnum_max+1) * sizeof(int));
+
+ /* build the (attnum => attidx) and (attidx => attnum) mappings */
+ attidx = 0;
+ attnum = -1;
+
+ while (true)
+ {
+ attnum = bms_next_member(deps_attnums, attnum);
+ if (attnum == -2)
+ break;
+
+ deps_idx_to_attnum[attidx] = attnum;
+ deps_attnum_to_idx[attnum] = attidx;
+
+ attidx += 1;
+ }
+
+ /* do we have all the attributes mapped? */
+ Assert(attidx == deps_natts);
+
+ /* walk through all the mvstats, build the adjacency matrix */
+ for (i = 0; i < nmvstats; i++)
+ {
+ int j;
+ MVDependencies dependencies = NULL;
+
+ /* skip stats without functional dependencies built */
+ if (! mvstats[i].deps_built)
+ continue;
+
+ /* fetch dependencies */
+ dependencies = deserialize_mv_dependencies(fetch_mv_dependencies(mvstats[i].mvoid));
+ if (dependencies == NULL)
+ continue;
+
+ /* set deps_matrix[a,b] to 'true' if 'a=>b' */
+ for (j = 0; j < dependencies->ndeps; j++)
+ {
+ int aidx = deps_attnum_to_idx[dependencies->deps[j]->a];
+ int bidx = deps_attnum_to_idx[dependencies->deps[j]->b];
+
+ /* a=> b */
+ deps_matrix[aidx * deps_natts + bidx] = true;
+ }
+ }
+
+ /*
+ * Multiply the matrix N-times (N = size of the matrix), so that we
+ * get all the transitive dependencies. That makes the next step
+ * much easier and faster.
+ *
+ * This is essentially an adjacency matrix from graph theory, and
+ * by multiplying it we get transitive edges. We don't really care
+ * about the exact number (number of paths between vertices) though,
+ * so we can do the multiplication in-place (we don't care whether
+ * we found the dependency in this round or in the previous one).
+ *
+ * Track how many new dependencies were added, and stop when 0, but
+ * we can't multiply more than N-times (longest path in the graph).
+ */
+ for (i = 0; i < deps_natts; i++)
+ {
+ int k, l, m;
+ int nchanges = 0;
+
+ /* k => l */
+ for (k = 0; k < deps_natts; k++)
+ {
+ for (l = 0; l < deps_natts; l++)
+ {
+ /* we already have this dependency */
+ if (deps_matrix[k * deps_natts + l])
+ continue;
+
+ /* we don't really care about the exact value, just 0/1 */
+ for (m = 0; m < deps_natts; m++)
+ {
+ if (deps_matrix[k * deps_natts + m] * deps_matrix[m * deps_natts + l])
+ {
+ deps_matrix[k * deps_natts + l] = true;
+ nchanges += 1;
+ break;
+ }
+ }
+ }
+ }
+
+ /* no transitive dependency added here, so terminate */
+ if (nchanges == 0)
+ break;
+ }
+
+ /*
+ * Walk through the clauses, and see which other clauses we may
+ * reduce. The matrix contains all transitive dependencies, which
+ * makes this very fast.
+ *
+ * We have to be careful not to reduce the clause using itself, or
+ * reducing all clauses forming a cycle (so we have to skip already
+ * eliminated clauses).
+ *
+ * I'm not sure whether this guarantees finding the best solution,
+ * i.e. reducing the most clauses, but it probably does (thanks to
+ * having all the transitive dependencies).
+ */
+ for (i = 0; i < nmvclauses; i++)
+ {
+ int j;
+
+ /* not covered by dependencies */
+ if (! bms_is_member(mvattnums[i], deps_attnums))
+ continue;
+
+ /* this clause was already reduced, so let's skip it */
+ if (reduced[i])
+ continue;
+
+ /* walk the potentially 'implied' clauses */
+ for (j = 0; j < nmvclauses; j++)
+ {
+ int aidx, bidx;
+
+ /* not covered by dependencies */
+ if (! bms_is_member(mvattnums[j], deps_attnums))
+ continue;
+
+ aidx = deps_attnum_to_idx[mvattnums[i]];
+ bidx = deps_attnum_to_idx[mvattnums[j]];
+
+ /* can't reduce the clause by itself, or if already reduced */
+ if ((i == j) || reduced[j])
+ continue;
+
+ /* mark the clause as reduced (if aidx => bidx) */
+ reduced[j] = deps_matrix[aidx * deps_natts + bidx];
+ }
+ }
+
+ /* now walk through the clauses, and keep only those not reduced */
+ for (i = 0; i < nmvclauses; i++)
+ {
+ if (! reduced[i])
+ reduced_clauses = lappend(reduced_clauses, mvclauses[i]);
+ }
+
+ pfree(reduced);
+ pfree(mvclauses);
+ pfree(mvattnums);
+
+ pfree(deps_matrix);
+ pfree(deps_idx_to_attnum);
+ pfree(deps_attnum_to_idx);
+
+ bms_free(deps_attnums);
+ bms_free(clause_attnums);
+ bms_free(intersect_attnums);
+
+ return reduced_clauses;
+}
diff --git a/src/backend/utils/mvstats/common.c b/src/backend/utils/mvstats/common.c
index 8efc5ba..d44b95a 100644
--- a/src/backend/utils/mvstats/common.c
+++ b/src/backend/utils/mvstats/common.c
@@ -57,7 +57,8 @@ build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
/*
* Analyze functional dependencies of columns.
*/
- deps = build_mv_dependencies(numrows, rows, attrs, stats);
+ if (mvstats->deps_enabled)
+ deps = build_mv_dependencies(numrows, rows, attrs, stats);
/* store the histogram / MCV list in the catalog */
update_mv_stats(mvstats[i].mvoid, deps);
@@ -154,6 +155,7 @@ list_mv_stats(Oid relid, int *nstats, bool built_only)
result[*nstats].mvoid = HeapTupleGetOid(htup);
result[*nstats].stakeys = buildint2vector(stats->stakeys.values, stats->stakeys.dim1);
+ result[*nstats].deps_enabled = stats->deps_enabled;
result[*nstats].deps_built = stats->deps_built;
*nstats += 1;
}
@@ -260,6 +262,7 @@ compare_scalars_partition(const void *a, const void *b, void *arg)
return ApplySortComparator(da, false, db, false, ssup);
}
+
/* initialize multi-dimensional sort */
MultiSortSupport
multi_sort_init(int ndims)
diff --git a/src/include/catalog/pg_proc.h b/src/include/catalog/pg_proc.h
index f728d88..2916f11 100644
--- a/src/include/catalog/pg_proc.h
+++ b/src/include/catalog/pg_proc.h
@@ -2712,9 +2712,9 @@ DESCR("current user privilege on any column by rel name");
DATA(insert OID = 3029 ( has_any_column_privilege PGNSP PGUID 12 10 0 0 0 f f f f t f s 2 0 16 "26 25" _null_ _null_ _null_ _null_ has_any_column_privilege_id _null_ _null_ _null_ ));
DESCR("current user privilege on any column by rel oid");
-DATA(insert OID = 3284 ( pg_mv_stats_dependencies_info PGNSP PGUID 12 1 0 0 0 f f f f t f i 1 0 25 "17" _null_ _null_ _null_ _null_ pg_mv_stats_dependencies_info _null_ _null_ _null_ ));
+DATA(insert OID = 3377 ( pg_mv_stats_dependencies_info PGNSP PGUID 12 1 0 0 0 f f f f t f i 1 0 25 "17" _null_ _null_ _null_ _null_ pg_mv_stats_dependencies_info _null_ _null_ _null_ ));
DESCR("multivariate stats: functional dependencies info");
-DATA(insert OID = 3285 ( pg_mv_stats_dependencies_show PGNSP PGUID 12 1 0 0 0 f f f f t f i 1 0 25 "17" _null_ _null_ _null_ _null_ pg_mv_stats_dependencies_show _null_ _null_ _null_ ));
+DATA(insert OID = 3378 ( pg_mv_stats_dependencies_show PGNSP PGUID 12 1 0 0 0 f f f f t f i 1 0 25 "17" _null_ _null_ _null_ _null_ pg_mv_stats_dependencies_show _null_ _null_ _null_ ));
DESCR("multivariate stats: functional dependencies show");
DATA(insert OID = 1928 ( pg_stat_get_numscans PGNSP PGUID 12 1 0 0 0 f f f f t f s 1 0 20 "26" _null_ _null_ _null_ _null_ pg_stat_get_numscans _null_ _null_ _null_ ));
diff --git a/src/include/utils/mvstats.h b/src/include/utils/mvstats.h
index 5c8643d..ec6764b 100644
--- a/src/include/utils/mvstats.h
+++ b/src/include/utils/mvstats.h
@@ -18,24 +18,34 @@
/*
* Basic info about the stats, used when choosing what to use
- *
- * TODO Add info about what statistics is available (histogram, MCV,
- * hashed MCV, functional dependencies).
*/
typedef struct MVStatsData {
Oid mvoid; /* OID of the stats in pg_mv_statistic */
int2vector *stakeys; /* attnums for columns in the stats */
+
+ /* statistics requested in ALTER TABLE ... ADD STATISTICS */
+ bool deps_enabled; /* analyze functional dependencies */
+
+ /* available statistics (computed by ANALYZE) */
bool deps_built; /* functional dependencies available */
} MVStatsData;
typedef struct MVStatsData *MVStats;
+/*
+ * Degree of how much MCV item / histogram bucket matches a clause.
+ * This is then considered when computing the selectivity.
+ */
+#define MVSTATS_MATCH_NONE 0 /* no match at all */
+#define MVSTATS_MATCH_PARTIAL 1 /* partial match */
+#define MVSTATS_MATCH_FULL 2 /* full match */
#define MVSTATS_MAX_DIMENSIONS 8 /* max number of attributes */
-/* An associative rule, tracking [a => b] dependency.
- *
- * TODO Make this work with multiple columns on both sides.
+
+/*
+ * Functional dependencies, tracking column-level relationships (values
+ * in one column determine values in another one).
*/
typedef struct MVDependencyData {
int16 a;
@@ -61,6 +71,7 @@ typedef MVDependenciesData* MVDependencies;
* stats specified using flags (or something like that).
*/
MVStats list_mv_stats(Oid relid, int *nstats, bool built_only);
+bytea * fetch_mv_rules(Oid mvoid);
bytea * fetch_mv_dependencies(Oid mvoid);
diff --git a/src/test/regress/expected/mv_dependencies.out b/src/test/regress/expected/mv_dependencies.out
new file mode 100644
index 0000000..159d317
--- /dev/null
+++ b/src/test/regress/expected/mv_dependencies.out
@@ -0,0 +1,175 @@
+-- data type passed by value
+CREATE TABLE functional_dependencies (
+ a INT,
+ b INT,
+ c INT
+);
+-- unknown column
+ALTER TABLE functional_dependencies ADD STATISTICS (dependencies) ON (unknown_column);
+ERROR: column "unknown_column" referenced in statistics does not exist
+-- single column
+ALTER TABLE functional_dependencies ADD STATISTICS (dependencies) ON (a);
+ERROR: multivariate stats require 2 or more columns
+-- single column, duplicated
+ALTER TABLE functional_dependencies ADD STATISTICS (dependencies) ON (a, a);
+ERROR: duplicate column name in statistics definition
+-- two columns, one duplicated
+ALTER TABLE functional_dependencies ADD STATISTICS (dependencies) ON (a, a, b);
+ERROR: duplicate column name in statistics definition
+-- unknown option
+ALTER TABLE functional_dependencies ADD STATISTICS (unknown_option) ON (a, b, c);
+ERROR: unrecognized STATISTICS option "unknown_option"
+-- correct command
+ALTER TABLE functional_dependencies ADD STATISTICS (dependencies) ON (a, b, c);
+-- random data (no functional dependencies)
+INSERT INTO functional_dependencies
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+ANALYZE functional_dependencies;
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+ deps_enabled | deps_built | pg_mv_stats_dependencies_show
+--------------+------------+-------------------------------
+ t | f |
+(1 row)
+
+TRUNCATE functional_dependencies;
+-- a => b, a => c, b => c
+INSERT INTO functional_dependencies
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE functional_dependencies;
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+ deps_enabled | deps_built | pg_mv_stats_dependencies_show
+--------------+------------+-------------------------------
+ t | t | 1 => 2, 1 => 3, 2 => 3
+(1 row)
+
+TRUNCATE functional_dependencies;
+-- a => b, a => c
+INSERT INTO functional_dependencies
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE functional_dependencies;
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+ deps_enabled | deps_built | pg_mv_stats_dependencies_show
+--------------+------------+-------------------------------
+ t | t | 1 => 2, 1 => 3
+(1 row)
+
+TRUNCATE functional_dependencies;
+-- check explain (expect bitmap index scan, not plain index scan)
+INSERT INTO functional_dependencies
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX fdeps_idx ON functional_dependencies (a, b);
+ANALYZE functional_dependencies;
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+ deps_enabled | deps_built | pg_mv_stats_dependencies_show
+--------------+------------+-------------------------------
+ t | t | 1 => 2, 1 => 3, 2 => 3
+(1 row)
+
+EXPLAIN (COSTS off)
+ SELECT * FROM functional_dependencies WHERE a = 10 AND b = 5;
+ QUERY PLAN
+---------------------------------------------
+ Bitmap Heap Scan on functional_dependencies
+ Recheck Cond: ((a = 10) AND (b = 5))
+ -> Bitmap Index Scan on fdeps_idx
+ Index Cond: ((a = 10) AND (b = 5))
+(4 rows)
+
+DELETE FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+DROP TABLE functional_dependencies;
+-- varlena type (text)
+CREATE TABLE functional_dependencies (
+ a TEXT,
+ b TEXT,
+ c TEXT
+);
+ALTER TABLE functional_dependencies ADD STATISTICS (dependencies) ON (a, b, c);
+-- random data (no functional dependencies)
+INSERT INTO functional_dependencies
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+ANALYZE functional_dependencies;
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+ deps_enabled | deps_built | pg_mv_stats_dependencies_show
+--------------+------------+-------------------------------
+ t | f |
+(1 row)
+
+TRUNCATE functional_dependencies;
+-- a => b, a => c, b => c
+INSERT INTO functional_dependencies
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE functional_dependencies;
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+ deps_enabled | deps_built | pg_mv_stats_dependencies_show
+--------------+------------+-------------------------------
+ t | t | 1 => 2, 1 => 3, 2 => 3
+(1 row)
+
+TRUNCATE functional_dependencies;
+-- a => b, a => c
+INSERT INTO functional_dependencies
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE functional_dependencies;
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+ deps_enabled | deps_built | pg_mv_stats_dependencies_show
+--------------+------------+-------------------------------
+ t | t | 1 => 2, 1 => 3
+(1 row)
+
+TRUNCATE functional_dependencies;
+-- check explain (expect bitmap index scan, not plain index scan)
+INSERT INTO functional_dependencies
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX fdeps_idx ON functional_dependencies (a, b);
+ANALYZE functional_dependencies;
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+ deps_enabled | deps_built | pg_mv_stats_dependencies_show
+--------------+------------+-------------------------------
+ t | t | 1 => 2, 1 => 3, 2 => 3
+(1 row)
+
+EXPLAIN (COSTS off)
+ SELECT * FROM functional_dependencies WHERE a = '10' AND b = '5';
+ QUERY PLAN
+------------------------------------------------------------
+ Bitmap Heap Scan on functional_dependencies
+ Recheck Cond: ((a = '10'::text) AND (b = '5'::text))
+ -> Bitmap Index Scan on fdeps_idx
+ Index Cond: ((a = '10'::text) AND (b = '5'::text))
+(4 rows)
+
+DELETE FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+DROP TABLE functional_dependencies;
+-- NULL values (mix of int and text columns)
+CREATE TABLE functional_dependencies (
+ a INT,
+ b TEXT,
+ c INT,
+ d TEXT
+);
+ALTER TABLE functional_dependencies ADD STATISTICS (dependencies) ON (a, b, c, d);
+INSERT INTO functional_dependencies
+ SELECT
+ mod(i, 100),
+ (CASE WHEN mod(i, 200) = 0 THEN NULL ELSE mod(i,200) END),
+ mod(i, 400),
+ (CASE WHEN mod(i, 300) = 0 THEN NULL ELSE mod(i,600) END)
+ FROM generate_series(1,10000) s(i);
+ANALYZE functional_dependencies;
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+ deps_enabled | deps_built | pg_mv_stats_dependencies_show
+--------------+------------+----------------------------------------
+ t | t | 2 => 1, 3 => 1, 3 => 2, 4 => 1, 4 => 2
+(1 row)
+
+DELETE FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+DROP TABLE functional_dependencies;
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index 6d3b865..00c6ddf 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -109,3 +109,6 @@ test: event_trigger
# run stats by itself because its delay may be insufficient under heavy load
test: stats
+
+# run tests of multivariate stats
+test: mv_dependencies
diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule
index 8326894..b818be9 100644
--- a/src/test/regress/serial_schedule
+++ b/src/test/regress/serial_schedule
@@ -153,3 +153,4 @@ test: with
test: xml
test: event_trigger
test: stats
+test: mv_dependencies
diff --git a/src/test/regress/sql/mv_dependencies.sql b/src/test/regress/sql/mv_dependencies.sql
new file mode 100644
index 0000000..f95dbf5
--- /dev/null
+++ b/src/test/regress/sql/mv_dependencies.sql
@@ -0,0 +1,153 @@
+-- data type passed by value
+CREATE TABLE functional_dependencies (
+ a INT,
+ b INT,
+ c INT
+);
+
+-- unknown column
+ALTER TABLE functional_dependencies ADD STATISTICS (dependencies) ON (unknown_column);
+
+-- single column
+ALTER TABLE functional_dependencies ADD STATISTICS (dependencies) ON (a);
+
+-- single column, duplicated
+ALTER TABLE functional_dependencies ADD STATISTICS (dependencies) ON (a, a);
+
+-- two columns, one duplicated
+ALTER TABLE functional_dependencies ADD STATISTICS (dependencies) ON (a, a, b);
+
+-- unknown option
+ALTER TABLE functional_dependencies ADD STATISTICS (unknown_option) ON (a, b, c);
+
+-- correct command
+ALTER TABLE functional_dependencies ADD STATISTICS (dependencies) ON (a, b, c);
+
+-- random data (no functional dependencies)
+INSERT INTO functional_dependencies
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+
+ANALYZE functional_dependencies;
+
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+
+TRUNCATE functional_dependencies;
+
+-- a => b, a => c, b => c
+INSERT INTO functional_dependencies
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+
+ANALYZE functional_dependencies;
+
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+
+TRUNCATE functional_dependencies;
+
+-- a => b, a => c
+INSERT INTO functional_dependencies
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE functional_dependencies;
+
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+
+TRUNCATE functional_dependencies;
+
+-- check explain (expect bitmap index scan, not plain index scan)
+INSERT INTO functional_dependencies
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX fdeps_idx ON functional_dependencies (a, b);
+ANALYZE functional_dependencies;
+
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+
+EXPLAIN (COSTS off)
+ SELECT * FROM functional_dependencies WHERE a = 10 AND b = 5;
+
+DELETE FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+DROP TABLE functional_dependencies;
+
+-- varlena type (text)
+CREATE TABLE functional_dependencies (
+ a TEXT,
+ b TEXT,
+ c TEXT
+);
+
+ALTER TABLE functional_dependencies ADD STATISTICS (dependencies) ON (a, b, c);
+
+-- random data (no functional dependencies)
+INSERT INTO functional_dependencies
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+
+ANALYZE functional_dependencies;
+
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+
+TRUNCATE functional_dependencies;
+
+-- a => b, a => c, b => c
+INSERT INTO functional_dependencies
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+
+ANALYZE functional_dependencies;
+
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+
+TRUNCATE functional_dependencies;
+
+-- a => b, a => c
+INSERT INTO functional_dependencies
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE functional_dependencies;
+
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+
+TRUNCATE functional_dependencies;
+
+-- check explain (expect bitmap index scan, not plain index scan)
+INSERT INTO functional_dependencies
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX fdeps_idx ON functional_dependencies (a, b);
+ANALYZE functional_dependencies;
+
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+
+EXPLAIN (COSTS off)
+ SELECT * FROM functional_dependencies WHERE a = '10' AND b = '5';
+
+DELETE FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+DROP TABLE functional_dependencies;
+
+-- NULL values (mix of int and text columns)
+CREATE TABLE functional_dependencies (
+ a INT,
+ b TEXT,
+ c INT,
+ d TEXT
+);
+
+ALTER TABLE functional_dependencies ADD STATISTICS (dependencies) ON (a, b, c, d);
+
+INSERT INTO functional_dependencies
+ SELECT
+ mod(i, 100),
+ (CASE WHEN mod(i, 200) = 0 THEN NULL ELSE mod(i,200) END),
+ mod(i, 400),
+ (CASE WHEN mod(i, 300) = 0 THEN NULL ELSE mod(i,600) END)
+ FROM generate_series(1,10000) s(i);
+
+ANALYZE functional_dependencies;
+
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+
+DELETE FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+DROP TABLE functional_dependencies;
--
2.0.5
[text/x-diff] 0003-multivariate-MCV-lists.patch (109.4K, ../../5519E9B2.5080105@2ndquadrant.com/4-0003-multivariate-MCV-lists.patch)
download | inline diff:
From 13c3d4cbe85bbbe6b9509de15dd08384df1df97f Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Sun, 11 Jan 2015 20:15:37 +0100
Subject: [PATCH 3/5] multivariate MCV lists
- extends the pg_mv_statistic catalog (add 'mcv' fields)
- building the MCV lists during ANALYZE
- simple estimation while planning the queries
Includes regression tests, mostly equal to regression tests for
functional dependencies.
---
src/backend/catalog/system_views.sql | 4 +-
src/backend/commands/tablecmds.c | 47 +-
src/backend/optimizer/path/clausesel.c | 1153 ++++++++++++++++++++++++++++++--
src/backend/utils/mvstats/Makefile | 2 +-
src/backend/utils/mvstats/common.c | 58 +-
src/backend/utils/mvstats/common.h | 11 +-
src/backend/utils/mvstats/mcv.c | 1002 +++++++++++++++++++++++++++
src/include/catalog/pg_mv_statistic.h | 18 +-
src/include/catalog/pg_proc.h | 2 +
src/include/utils/mvstats.h | 68 +-
src/test/regress/expected/mv_mcv.out | 210 ++++++
src/test/regress/expected/rules.out | 4 +-
src/test/regress/parallel_schedule | 2 +-
src/test/regress/serial_schedule | 1 +
src/test/regress/sql/mv_mcv.sql | 181 +++++
15 files changed, 2662 insertions(+), 101 deletions(-)
create mode 100644 src/backend/utils/mvstats/mcv.c
create mode 100644 src/test/regress/expected/mv_mcv.out
create mode 100644 src/test/regress/sql/mv_mcv.sql
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index d05a716..4538e63 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -156,7 +156,9 @@ CREATE VIEW pg_mv_stats AS
C.relname AS tablename,
S.stakeys AS attnums,
length(S.stadeps) as depsbytes,
- pg_mv_stats_dependencies_info(S.stadeps) as depsinfo
+ pg_mv_stats_dependencies_info(S.stadeps) as depsinfo,
+ length(S.stamcv) AS mcvbytes,
+ pg_mv_stats_mcvlist_info(S.stamcv) AS mcvinfo
FROM (pg_mv_statistic S JOIN pg_class C ON (C.oid = S.starelid))
LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace);
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 965d342..fae0fc7 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -11901,7 +11901,13 @@ static void ATExecAddStatistics(AlteredTableInfo *tab, Relation rel,
Relation mvstatrel;
/* by default build everything */
- bool build_dependencies = false;
+ bool build_dependencies = false,
+ build_mcv = false;
+
+ int32 max_mcv_items = -1;
+
+ /* options required because of other options */
+ bool require_mcv = false;
Assert(IsA(def, StatisticsDef));
@@ -11956,6 +11962,29 @@ static void ATExecAddStatistics(AlteredTableInfo *tab, Relation rel,
if (strcmp(opt->defname, "dependencies") == 0)
build_dependencies = defGetBoolean(opt);
+ else if (strcmp(opt->defname, "mcv") == 0)
+ build_mcv = defGetBoolean(opt);
+ else if (strcmp(opt->defname, "max_mcv_items") == 0)
+ {
+ max_mcv_items = defGetInt32(opt);
+
+ /* this option requires 'mcv' to be enabled */
+ require_mcv = true;
+
+ /* sanity check */
+ if (max_mcv_items < MVSTAT_MCVLIST_MIN_ITEMS)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("max number of MCV items must be at least %d",
+ MVSTAT_MCVLIST_MIN_ITEMS)));
+
+ else if (max_mcv_items > MVSTAT_MCVLIST_MAX_ITEMS)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("max number of MCV items is %d",
+ MVSTAT_MCVLIST_MAX_ITEMS)));
+
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -11964,10 +11993,16 @@ static void ATExecAddStatistics(AlteredTableInfo *tab, Relation rel,
}
/* check that at least some statistics were requested */
- if (! build_dependencies)
+ if (! (build_dependencies || build_mcv))
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("no statistics type (dependencies) was requested")));
+ errmsg("no statistics type (dependencies, mcv) was requested")));
+
+ /* now do some checking of the options */
+ if (require_mcv && (! build_mcv))
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("option 'mcv' is required by other options(s)")));
/* sort the attnums and build int2vector */
qsort(attnums, numcols, sizeof(int16), compare_int16);
@@ -11983,9 +12018,13 @@ static void ATExecAddStatistics(AlteredTableInfo *tab, Relation rel,
values[Anum_pg_mv_statistic_starelid-1] = ObjectIdGetDatum(RelationGetRelid(rel));
values[Anum_pg_mv_statistic_stakeys -1] = PointerGetDatum(stakeys);
+
values[Anum_pg_mv_statistic_deps_enabled -1] = BoolGetDatum(build_dependencies);
+ values[Anum_pg_mv_statistic_mcv_enabled -1] = BoolGetDatum(build_mcv);
+ values[Anum_pg_mv_statistic_mcv_max_items -1] = Int32GetDatum(max_mcv_items);
- nulls[Anum_pg_mv_statistic_stadeps -1] = true;
+ nulls[Anum_pg_mv_statistic_stadeps -1] = true;
+ nulls[Anum_pg_mv_statistic_stamcv -1] = true;
/* insert the tuple into pg_mv_statistic */
mvstatrel = heap_open(MvStatisticRelationId, RowExclusiveLock);
diff --git a/src/backend/optimizer/path/clausesel.c b/src/backend/optimizer/path/clausesel.c
index e742827..d24aedf 100644
--- a/src/backend/optimizer/path/clausesel.c
+++ b/src/backend/optimizer/path/clausesel.c
@@ -20,6 +20,7 @@
#include "optimizer/cost.h"
#include "optimizer/pathnode.h"
#include "optimizer/plancat.h"
+#include "optimizer/var.h"
#include "utils/fmgroids.h"
#include "utils/lsyscache.h"
#include "utils/selfuncs.h"
@@ -50,17 +51,46 @@ typedef struct RangeQueryClause
static void addRangeClause(RangeQueryClause **rqlist, Node *clause,
bool varonleft, bool isLTsel, Selectivity s2);
+#define MV_CLAUSE_TYPE_FDEP 0x01
+#define MV_CLAUSE_TYPE_MCV 0x02
static bool clause_is_mv_compatible(PlannerInfo *root, Node *clause, Oid varRelid,
- Oid *relid, AttrNumber *attnum, SpecialJoinInfo *sjinfo);
+ Oid *relid, Bitmapset **attnums, SpecialJoinInfo *sjinfo,
+ int type);
static Bitmapset *collect_mv_attnums(PlannerInfo *root, List *clauses,
- Oid varRelid, Oid *relid, SpecialJoinInfo *sjinfo);
+ Oid varRelid, Oid *relid, SpecialJoinInfo *sjinfo,
+ int type);
static List *clauselist_apply_dependencies(PlannerInfo *root, List *clauses,
Oid varRelid, int nmvstats, MVStats mvstats,
SpecialJoinInfo *sjinfo);
+static int choose_mv_statistics(int nmvstats, MVStats mvstats,
+ Bitmapset *attnums);
+static List *clauselist_mv_split(PlannerInfo *root, SpecialJoinInfo *sjinfo,
+ List *clauses, Oid varRelid,
+ List **mvclauses, MVStats mvstats, int types);
+
+static Selectivity clauselist_mv_selectivity(PlannerInfo *root,
+ List *clauses, MVStats mvstats);
+static Selectivity clauselist_mv_selectivity_mcvlist(PlannerInfo *root,
+ List *clauses, MVStats mvstats,
+ bool *fullmatch, Selectivity *lowsel);
+
+static int update_match_bitmap_mcvlist(PlannerInfo *root, List *clauses,
+ int2vector *stakeys, MCVList mcvlist,
+ int nmatches, char * matches,
+ Selectivity *lowsel, bool *fullmatch,
+ bool is_or);
+
+/* used for merging bitmaps - AND (min), OR (max) */
+#define MAX(x, y) (((x) > (y)) ? (x) : (y))
+#define MIN(x, y) (((x) < (y)) ? (x) : (y))
+
+#define UPDATE_RESULT(m,r,or) \
+ (m) = (or) ? (MAX(m,r)) : (MIN(m,r))
+
/****************************************************************************
* ROUTINES TO COMPUTE SELECTIVITIES
****************************************************************************/
@@ -197,15 +227,19 @@ clauselist_selectivity(PlannerInfo *root,
Bitmapset *mvattnums = NULL;
/*
- * If there's exactly one clause, then no use in trying to match up pairs,
- * so just go directly to clause_selectivity().
+ * If there's exactly one clause, then no use in trying to match up
+ * pairs, so just go directly to clause_selectivity().
*/
if (list_length(clauses) == 1)
return clause_selectivity(root, (Node *) linitial(clauses),
varRelid, jointype, sjinfo);
- /* collect attributes referenced by mv-compatible clauses */
- mvattnums = collect_mv_attnums(root, clauses, varRelid, &relid, sjinfo);
+ /*
+ * Collect attributes referenced by mv-compatible clauses (looking
+ * for clauses compatible with functional dependencies for now).
+ */
+ mvattnums = collect_mv_attnums(root, clauses, varRelid, &relid, sjinfo,
+ MV_CLAUSE_TYPE_FDEP);
/*
* If there are mv-compatible clauses, referencing at least two
@@ -227,6 +261,49 @@ clauselist_selectivity(PlannerInfo *root,
}
/*
+ * Recollect attributes from mv-compatible clauses (maybe we've
+ * removed so many clauses we have a single mv-compatible attnum).
+ * From now on we're only interested in MCV-compatible clauses.
+ */
+ mvattnums = collect_mv_attnums(root, clauses, varRelid, &relid, sjinfo,
+ MV_CLAUSE_TYPE_MCV);
+
+ /*
+ * If there still are at least two columns, we'll try to select
+ * a suitable multivariate stats.
+ */
+ if (bms_num_members(mvattnums) >= 2)
+ {
+ /* fetch info from the catalog (not the serialized stats yet) */
+ mvstats = list_mv_stats(relid, &nmvstats, true);
+
+ /* see choose_mv_statistics() for details */
+ if (nmvstats > 0)
+ {
+ int idx = choose_mv_statistics(nmvstats, mvstats, mvattnums);
+
+ if (idx >= 0) /* we have a matching stats */
+ {
+ MVStats mvstat = &mvstats[idx];
+
+ /* clauses compatible with multi-variate stats */
+ List *mvclauses = NIL;
+
+ /* split the clauselist into regular and mv-clauses */
+ clauses = clauselist_mv_split(root, sjinfo, clauses,
+ varRelid, &mvclauses, mvstat,
+ MV_CLAUSE_TYPE_MCV);
+
+ /* we've chosen the histogram to match the clauses */
+ Assert(mvclauses != NIL);
+
+ /* compute the multivariate stats */
+ s1 *= clauselist_mv_selectivity(root, mvclauses, mvstat);
+ }
+ }
+ }
+
+ /*
* Initial scan over clauses. Anything that doesn't look like a potential
* rangequery clause gets multiplied into s1 and forgotten. Anything that
* does gets inserted into an rqlist entry.
@@ -901,12 +978,198 @@ clause_selectivity(PlannerInfo *root,
return s1;
}
+
+/*
+ * Estimate selectivity for the list of MV-compatible clauses, using that
+ * particular histogram.
+ *
+ * When we hit a single bucket, we don't know what portion of it actually
+ * matches the clauses (e.g. equality), and we use 1/2 the bucket by
+ * default. However, the MV histograms are usually less detailed than
+ * the per-column ones, meaning the sum of buckets is often quite high
+ * (thanks to combining a lot of "partially hit" buckets).
+ *
+ * There are several ways to improve this, usually with cases when it
+ * won't really help. Also, the more complex the process, the worse
+ * the failures (i.e. misestimates).
+ *
+ * (1) Use the MV histogram only as a way to combine multiple
+ * per-column histograms, essentially rewriting
+ *
+ * P(A & B) = P(A) * P(B|A)
+ *
+ * where P(B|A) may be computed using a proper "slice" of the
+ * histogram, by first selecting only buckets where A is true, and
+ * then using the boundaries to 'restrict' the per-colunm histogram.
+ *
+ * With more clauses, it gets more complicated, of course
+ *
+ * P(A & B & C) = P(A & C) * P(B|A & C)
+ * = P(A) * P(C|A) * P(B|A & C)
+ *
+ * and so on.
+ *
+ * Of course, the question is how well and efficiently we can
+ * compute the conditional probabilities - whether this approach
+ * can improve the estimates (instead of amplifying the errors).
+ *
+ * Also, this does not eliminate the need for histogram on [A,B,C].
+ *
+ * (2) Use multiple smaller (and more accurate) histograms, and combine
+ * them using a process similar to the above. E.g. by assuming that
+ * B and C are independent, we can rewrite
+ *
+ * P(B|A & C) = P(B|A)
+ *
+ * so we can rewrite the whole formula to
+ *
+ * P(A & B & C) = P(A) * P(C|A) * P(B|A)
+ *
+ * and we're OK with two 2D histograms [A,C] and [A,B].
+ *
+ * It'd be nice to perform some sort of statistical test (Fisher
+ * or another chi-squared test) to identify independent components
+ * and automatically separate them into smaller histograms.
+ *
+ * (3) Using the estimated number of distinct values in a bucket to
+ * decide the selectivity of equality in the bucket (instead of
+ * blindly using 1/2 of the bucket, we may use 1/ndistinct).
+ * Of course, if the ndistinct estimate is way off, or when the
+ * distribution is not uniform (one distict items get much more
+ * items), this will fail. Also, we currently don't have ndistinct
+ * estimate available at this moment (but it shouldn't be that
+ * difficult to compute as ndistinct and ntuples should be available).
+ *
+ * TODO Clamp the selectivity by min of the per-clause selectivities
+ * (i.e. the selectivity of the most restrictive clause), because
+ * that's the maximum we can ever get from ANDed list of clauses.
+ * This may probably prevent issues with hitting too many buckets
+ * and low precision histograms.
+ *
+ * TODO We may support some additional conditions, most importantly
+ * those matching multiple columns (e.g. "a = b" or "a < b").
+ * Ultimately we could track multi-table histograms for join
+ * cardinality estimation.
+ *
+ * TODO Currently this is only estimating all clauses, or clauses
+ * matching varRelid (when it's not 0). I'm not sure what's the
+ * purpose of varRelid, but my assumption is this is used for
+ * join conditions and such. In that case we can use those clauses
+ * to restrict the other (i.e. filter the histogram buckets first,
+ * before estimating the other clauses). This is essentially equal
+ * to computing P(A|B) where "B" are the clauses not matching the
+ * varRelid.
+ *
+ * TODO Further thoughts on processing equality clauses - maybe it'd be
+ * better to look for stats (with MCV) covered by the equality
+ * clauses, because then we have a chance to find an exact match
+ * in the MCV list, which is pretty much the best we can do. We may
+ * also look at the least frequent MCV item, and use it as a upper
+ * boundary for the selectivity (had there been a more frequent
+ * item, it'd be in the MCV list).
+ *
+ * These conditions may then be used as a condition for the other
+ * selectivities, i.e. we may estimate P(A,B) first, and then
+ * compute P(C|A,B) from another histogram. This may be useful when
+ * we can estimate P(A,B) accurately (e.g. because it's a complete
+ * equality match evaluated on MCV list), and then compute the
+ * conditional probability P(C|A,B), giving us the requested stats
+ *
+ * P(A,B,C) = P(A,B) * P(C|A,B)
+ *
+ * TODO There are several options for 'sanity clamping' the estimates.
+ *
+ * First, if we have selectivities for each condition, then
+ *
+ * P(A,B) <= MIN(P(A), P(B))
+ *
+ * Because additional conditions (connected by AND) can only lower
+ * the probability.
+ *
+ * So we can do some basic sanity checks using the single-variate
+ * stats (the ones we have right now).
+ *
+ * Second, when we have multivariate stats with a MCV list, then
+ *
+ * (a) if we have a full equality condition (one equality condition
+ * on each column) and we found a match in the MCV list, this is
+ * the selectivity (and it's supposed to be exact)
+ *
+ * (b) if we have a full equality condition and we haven't found a
+ * match in the MCV list, then the selectivity is below the
+ * lowest selectivity in the MCV list
+ *
+ * (c) if we have a equality condition (not full), we can still
+ * search the MCV for matches and use the sum of probabilities
+ * as a lower boundary for the histogram (if there are no
+ * matches in the MCV list, then we have no boundary)
+ *
+ * Third, if there are multiple multivariate stats for a set of
+ * clauses, we may compute all of them and then somehow aggregate
+ * them - e.g. by choosing the minimum, median or average. The
+ * multi-variate stats are susceptible to overestimation (because
+ * we take 50% of the bucket for partial matches). Some stats may
+ * give better estimates than others, but it's very difficult to
+ * say determine that in advance which one is the best (it depends
+ * on the number of buckets, number of additional columns not
+ * referenced in the clauses etc.) so we may compute all and then
+ * choose a sane aggregation (minimum seems like a good approach).
+ * Of course, this may result in longer / more expensive estimation
+ * (CPU-wise), but it may be worth it.
+ *
+ * There are ways to address this, though. First, it's possible to
+ * add a GUC choosing whether to do a 'simple' (using a single
+ * stats expected to give the best estimate) and 'complex' (combining
+ * the multiple estimates).
+ *
+ * multivariate_estimates = (simple|full)
+ *
+ * Also, this might be enabled at a table level, by something like
+ *
+ * ALTER TABLE ... SET STATISTICS (simple|full)
+ *
+ * Which would make it possible to use this only for the tables
+ * where the simple approach does not work.
+ *
+ * Also, there are ways to optimize this algorithmically. E.g. we
+ * may try to get an estimate from a matching MCV list first, and
+ * if we happen to get a "full equality match" we may stop computing
+ * the estimates from other stats (for this condition) because
+ * that's probably the best estimate we can really get.
+ *
+ * TODO When applying the clauses to the histogram/MCV list, we can do
+ * that from the most selective clauses first, because that'll
+ * eliminate the buckets/items sooner (so we'll be able to skip
+ * them without inspection, which is more expensive).
+ */
+static Selectivity
+clauselist_mv_selectivity(PlannerInfo *root, List *clauses, MVStats mvstats)
+{
+ bool fullmatch = false;
+
+ /*
+ * Lowest frequency in the MCV list (may be used as an upper bound
+ * for full equality conditions that did not match any MCV item).
+ */
+ Selectivity mcv_low = 0.0;
+
+ /* TODO Evaluate simple 1D selectivities, use the smallest one as
+ * an upper bound, product as lower bound, and sort the
+ * clauses in ascending order by selectivity (to optimize the
+ * MCV/histogram evaluation).
+ */
+
+ /* Evaluate the MCV selectivity */
+ return clauselist_mv_selectivity_mcvlist(root, clauses, mvstats,
+ &fullmatch, &mcv_low);
+}
+
/*
* Collect attributes from mv-compatible clauses.
*/
static Bitmapset *
collect_mv_attnums(PlannerInfo *root, List *clauses, Oid varRelid,
- Oid *relid, SpecialJoinInfo *sjinfo)
+ Oid *relid, SpecialJoinInfo *sjinfo, int types)
{
Bitmapset *attnums = NULL;
ListCell *l;
@@ -922,12 +1185,11 @@ collect_mv_attnums(PlannerInfo *root, List *clauses, Oid varRelid,
*/
foreach (l, clauses)
{
- AttrNumber attnum;
Node *clause = (Node *) lfirst(l);
- /* ignore the result for now - we only need the info */
- if (clause_is_mv_compatible(root, clause, varRelid, relid, &attnum, sjinfo))
- attnums = bms_add_member(attnums, attnum);
+ /* ignore the result here - we only need the attnums */
+ clause_is_mv_compatible(root, clause, varRelid, relid, &attnums,
+ sjinfo, types);
}
/*
@@ -946,6 +1208,180 @@ collect_mv_attnums(PlannerInfo *root, List *clauses, Oid varRelid,
}
/*
+ * We're looking for statistics matching at least 2 attributes,
+ * referenced in the clauses compatible with multivariate statistics.
+ * The current selection criteria is very simple - we choose the
+ * statistics referencing the most attributes.
+ *
+ * If there are multiple statistics referencing the same number of
+ * columns (from the clauses), the one with less source columns
+ * (as listed in the ADD STATISTICS when creating the statistics) wins.
+ * Other wise the first one wins.
+ *
+ * This is a very simple criteria, and has several weaknesses:
+ *
+ * (a) does not consider the accuracy of the statistics
+ *
+ * If there are two histograms built on the same set of columns,
+ * but one has 100 buckets and the other one has 1000 buckets (thus
+ * likely providing better estimates), this is not currently
+ * considered.
+ *
+ * (b) does not consider the type of statistics
+ *
+ * If there are three statistics - one containing just a MCV list,
+ * another one with just a histogram and a third one with both,
+ * this is not considered.
+ *
+ * (c) does not consider the number of clauses
+ *
+ * As explained, only the number of referenced attributes counts,
+ * so if there are multiple clauses on a single attribute, this
+ * still counts as a single attribute.
+ *
+ * (d) does not consider type of condition
+ *
+ * Some clauses may work better with some statistics - for example
+ * equality clauses probably work better with MCV lists than with
+ * histograms. But IS [NOT] NULL conditions may often work better
+ * with histograms (thanks to NULL-buckets).
+ *
+ * So for example with five WHERE conditions
+ *
+ * WHERE (a = 1) AND (b = 1) AND (c = 1) AND (d = 1) AND (e = 1)
+ *
+ * and statistics on (a,b), (a,b,e) and (a,b,c,d), the last one will be
+ * selected as it references the most columns.
+ *
+ * Once we have selected the multivariate statistics, we split the list
+ * of clauses into two parts - conditions that are compatible with the
+ * selected stats, and conditions are estimated using simple statistics.
+ *
+ * From the example above, conditions
+ *
+ * (a = 1) AND (b = 1) AND (c = 1) AND (d = 1)
+ *
+ * will be estimated using the multivariate statistics (a,b,c,d) while
+ * the last condition (e = 1) will get estimated using the regular ones.
+ *
+ * There are various alternative selection criteria (e.g. counting
+ * conditions instead of just referenced attributes), but eventually
+ * the best option should be to combine multiple statistics. But that's
+ * much harder to do correctly.
+ *
+ * TODO Select multiple statistics and combine them when computing
+ * the estimate.
+ *
+ * TODO This will probably have to consider compatibility of clauses,
+ * because 'dependencies' will probably work only with equality
+ * clauses.
+ */
+static int
+choose_mv_statistics(int nmvstats, MVStats mvstats, Bitmapset *attnums)
+{
+ int i, j;
+
+ int choice = -1;
+ int current_matches = 1; /* goal #1: maximize */
+ int current_dims = (MVSTATS_MAX_DIMENSIONS+1); /* goal #2: minimize */
+
+ /*
+ * Walk through the statistics (simple array with nmvstats elements)
+ * and for each one count the referenced attributes (encoded in
+ * the 'attnums' bitmap).
+ */
+ for (i = 0; i < nmvstats; i++)
+ {
+ /* columns matching this statistics */
+ int matches = 0;
+
+ int2vector * attrs = mvstats[i].stakeys;
+ int numattrs = mvstats[i].stakeys->dim1;
+
+ /* count columns covered by the histogram */
+ for (j = 0; j < numattrs; j++)
+ if (bms_is_member(attrs->values[j], attnums))
+ matches++;
+
+ /*
+ * Use this statistics when it improves the number of matches or
+ * when it matches the same number of attributes but is smaller.
+ */
+ if ((matches > current_matches) ||
+ ((matches == current_matches) && (current_dims > numattrs)))
+ {
+ choice = i;
+ current_matches = matches;
+ current_dims = numattrs;
+ }
+ }
+
+ return choice;
+}
+
+
+/*
+ * This splits the clauses list into two parts - one containing clauses
+ * that will be evaluated using the chosen statistics, and the remaining
+ * clauses (either non-mvcompatible, or not related to the histogram).
+ */
+static List *
+clauselist_mv_split(PlannerInfo *root, SpecialJoinInfo *sjinfo,
+ List *clauses, Oid varRelid, List **mvclauses,
+ MVStats mvstats, int types)
+{
+ int i;
+ ListCell *l;
+ List *non_mvclauses = NIL;
+
+ /* FIXME is there a better way to get info on int2vector? */
+ int2vector * attrs = mvstats->stakeys;
+ int numattrs = mvstats->stakeys->dim1;
+
+ Bitmapset *mvattnums = NULL;
+
+ /* build bitmap of attributes covered by the stats, so we can
+ * do bms_is_subset later */
+ for (i = 0; i < numattrs; i++)
+ mvattnums = bms_add_member(mvattnums, attrs->values[i]);
+
+ /* erase the list of mv-compatible clauses */
+ *mvclauses = NIL;
+
+ foreach (l, clauses)
+ {
+ bool match = false; /* by default not mv-compatible */
+ Bitmapset *attnums = NULL;
+ Node *clause = (Node *) lfirst(l);
+
+ if (clause_is_mv_compatible(root, clause, varRelid, NULL,
+ &attnums, sjinfo, types))
+ {
+ /* are all the attributes part of the selected stats? */
+ if (bms_is_subset(attnums, mvattnums))
+ match = true;
+ }
+
+ /*
+ * The clause matches the selected stats, so put it to the list
+ * of mv-compatible clauses. Otherwise, keep it in the list of
+ * 'regular' clauses (that may be selected later).
+ */
+ if (match)
+ *mvclauses = lappend(*mvclauses, clause);
+ else
+ non_mvclauses = lappend(non_mvclauses, clause);
+ }
+
+ /*
+ * Perform regular estimation using the clauses incompatible
+ * with the chosen histogram (or MV stats in general).
+ */
+ return non_mvclauses;
+
+}
+
+/*
* Determines whether the clause is compatible with multivariate stats,
* and if it is, returns some additional information - varno (index
* into simple_rte_array) and a bitmap of attributes. This is then
@@ -964,96 +1400,205 @@ collect_mv_attnums(PlannerInfo *root, List *clauses, Oid varRelid,
*/
static bool
clause_is_mv_compatible(PlannerInfo *root, Node *clause, Oid varRelid,
- Oid *relid, AttrNumber *attnum, SpecialJoinInfo *sjinfo)
+ Oid *relid, Bitmapset **attnums, SpecialJoinInfo *sjinfo,
+ int types)
{
+ Relids clause_relids;
+ Relids left_relids;
+ Relids right_relids;
if (IsA(clause, RestrictInfo))
{
RestrictInfo *rinfo = (RestrictInfo *) clause;
- /* Pseudoconstants are not really interesting here. */
- if (rinfo->pseudoconstant)
+ if (! IsA(clause, RestrictInfo))
+ {
+ elog(WARNING, "expected RestrictInfo, got type %d", clause->type);
return false;
+ }
- /* no support for OR clauses at this point */
- if (rinfo->orclause)
+ /* Pseudoconstants are not really interesting here. */
+ if (rinfo->pseudoconstant)
return false;
/* get the actual clause from the RestrictInfo (it's not an OR clause) */
clause = (Node*)rinfo->clause;
- /* only simple opclauses are compatible with multivariate stats */
- if (! is_opclause(clause))
- return false;
-
/* we don't support join conditions at this moment */
if (treat_as_join_clause(clause, rinfo, varRelid, sjinfo))
return false;
+ clause_relids = rinfo->clause_relids;
+ left_relids = rinfo->left_relids;
+ right_relids = rinfo->right_relids;
+ }
+ else if (is_opclause(clause) && list_length(((OpExpr *) clause)->args) == 2)
+ {
+ left_relids = pull_varnos(get_leftop((Expr*)clause));
+ right_relids = pull_varnos(get_rightop((Expr*)clause));
+
+ clause_relids = bms_union(left_relids,
+ right_relids);
+ }
+ else
+ {
+ /* Not a binary opclause, so mark left/right relid sets as empty */
+ left_relids = NULL;
+ right_relids = NULL;
+ /* and get the total relid set the hard way */
+ clause_relids = pull_varnos((Node *) clause);
+ }
+
+ /*
+ * Only simple opclauses and IS NULL tests are compatible with
+ * multivariate stats at this point.
+ */
+ if ((is_opclause(clause))
+ && (list_length(((OpExpr *) clause)->args) == 2))
+ {
+ OpExpr *expr = (OpExpr *) clause;
+ bool varonleft = true;
+ bool ok;
+
/* is it 'variable op constant' ? */
- if (list_length(((OpExpr *) clause)->args) == 2)
- {
- OpExpr *expr = (OpExpr *) clause;
- bool varonleft = true;
- bool ok;
- ok = (bms_membership(rinfo->clause_relids) == BMS_SINGLETON) &&
- (is_pseudo_constant_clause_relids(lsecond(expr->args),
- rinfo->right_relids) ||
- (varonleft = false,
- is_pseudo_constant_clause_relids(linitial(expr->args),
- rinfo->left_relids)));
+ ok = (bms_membership(clause_relids) == BMS_SINGLETON) &&
+ (is_pseudo_constant_clause_relids(lsecond(expr->args),
+ right_relids) ||
+ (varonleft = false,
+ is_pseudo_constant_clause_relids(linitial(expr->args),
+ left_relids)));
- if (ok)
- {
- RangeTblEntry * rte;
- Var * var = (varonleft) ? linitial(expr->args) : lsecond(expr->args);
+ if (ok)
+ {
+ RangeTblEntry * rte;
+ Var * var = (varonleft) ? linitial(expr->args) : lsecond(expr->args);
- /*
- * Simple variables only - otherwise the planner_rt_fetch seems to fail
- * (return NULL).
- *
- * TODO Maybe use examine_variable() would fix that?
- */
- if (! (IsA(var, Var) && (varRelid == 0 || varRelid == var->varno)))
- return false;
+ /*
+ * Simple variables only - otherwise the planner_rt_fetch seems to fail
+ * (return NULL).
+ *
+ * TODO Maybe use examine_variable() would fix that?
+ */
+ if (! (IsA(var, Var) && (varRelid == 0 || varRelid == var->varno)))
+ return false;
- /*
- * Only consider this variable if (varRelid == 0) or when the varno
- * matches varRelid (see explanation at clause_selectivity).
- *
- * FIXME I suspect this may not be really necessary. The (varRelid == 0)
- * part seems to be enforced by treat_as_join_clause().
- */
- if (! ((varRelid == 0) || (varRelid == var->varno)))
- return false;
+ /*
+ * Only consider this variable if (varRelid == 0) or when the varno
+ * matches varRelid (see explanation at clause_selectivity).
+ *
+ * FIXME I suspect this may not be really necessary. The (varRelid == 0)
+ * part seems to be enforced by treat_as_join_clause().
+ */
+ if (! ((varRelid == 0) || (varRelid == var->varno)))
+ return false;
- /* Also skip special varno values, and system attributes ... */
- if ((IS_SPECIAL_VARNO(var->varno)) || (! AttrNumberIsForUserDefinedAttr(var->varattno)))
- return false;
+ /* Also skip special varno values, and system attributes ... */
+ if ((IS_SPECIAL_VARNO(var->varno)) || (! AttrNumberIsForUserDefinedAttr(var->varattno)))
+ return false;
- /* Lookup info about the base relation (we need to pass the OID out) */
+ /* Lookup info about the base relation (we need to pass the OID out) */
+ if (relid != NULL)
+ {
rte = planner_rt_fetch(var->varno, root);
*relid = rte->relid;
-
- /*
- * If it's not a "<" or ">" or "=" operator, just ignore the
- * clause. Otherwise note the relid and attnum for the variable.
- * This uses the function for estimating selectivity, ont the
- * operator directly (a bit awkward, but well ...).
- */
- switch (get_oprrest(expr->opno))
- {
- case F_EQSEL:
- *attnum = var->varattno;
- return true;
- }
}
+
+ /*
+ * If it's not a "<" or ">" or "=" operator, just ignore the
+ * clause. Otherwise note the relid and attnum for the variable.
+ * This uses the function for estimating selectivity, ont the
+ * operator directly (a bit awkward, but well ...).
+ */
+ switch (get_oprrest(expr->opno))
+ {
+ case F_SCALARLTSEL:
+ case F_SCALARGTSEL:
+ /* not compatible with functional dependencies */
+ if (types & MV_CLAUSE_TYPE_MCV)
+ {
+ *attnums = bms_add_member(*attnums, var->varattno);
+ return (types & MV_CLAUSE_TYPE_MCV);
+ }
+ return false;
+
+ case F_EQSEL:
+ *attnums = bms_add_member(*attnums, var->varattno);
+ return true;
+ }
}
}
+ else if (IsA(clause, NullTest)
+ && IsA(((NullTest*)clause)->arg, Var))
+ {
+ RangeTblEntry * rte;
+ Var * var = (Var*)((NullTest*)clause)->arg;
- return false;
+ /*
+ * Simple variables only - otherwise the planner_rt_fetch seems to fail
+ * (return NULL).
+ *
+ * TODO Maybe use examine_variable() would fix that?
+ */
+ if (! (IsA(var, Var) && (varRelid == 0 || varRelid == var->varno)))
+ return false;
+ /*
+ * Only consider this variable if (varRelid == 0) or when the varno
+ * matches varRelid (see explanation at clause_selectivity).
+ *
+ * FIXME I suspect this may not be really necessary. The (varRelid == 0)
+ * part seems to be enforced by treat_as_join_clause().
+ */
+ if (! ((varRelid == 0) || (varRelid == var->varno)))
+ return false;
+
+ /* Also skip special varno values, and system attributes ... */
+ if ((IS_SPECIAL_VARNO(var->varno)) || (! AttrNumberIsForUserDefinedAttr(var->varattno)))
+ return false;
+
+ /* Lookup info about the base relation (we need to pass the OID out) */
+ if (relid != NULL)
+ {
+ rte = planner_rt_fetch(var->varno, root);
+ *relid = rte->relid;
+ }
+
+ *attnums = bms_add_member(*attnums, var->varattno);
+
+ return true;
+ }
+ else if (or_clause(clause) || and_clause(clause))
+ {
+ /*
+ * AND/OR-clauses are supported if all sub-clauses are supported
+ *
+ * TODO We might support mixed case, where some of the clauses
+ * are supported and some are not, and treat all supported
+ * subclauses as a single clause, compute it's selectivity
+ * using mv stats, and compute the total selectivity using
+ * the current algorithm.
+ *
+ * TODO For RestrictInfo above an OR-clause, we might use the
+ * orclause with nested RestrictInfo - we won't have to
+ * call pull_varnos() for each clause, saving time.
+ */
+ Bitmapset *tmp = NULL;
+ ListCell *l;
+ foreach (l, ((BoolExpr*)clause)->args)
+ {
+ if (! clause_is_mv_compatible(root, (Node*)lfirst(l),
+ varRelid, relid, &tmp, sjinfo, types))
+ return false;
+ }
+
+ /* add the attnums from the OR-clause to the set of attnums */
+ *attnums = bms_join(*attnums, tmp);
+
+ return true;
+ }
+
+ return false;
}
/*
@@ -1115,6 +1660,13 @@ clause_is_mv_compatible(PlannerInfo *root, Node *clause, Oid varRelid,
*
* TODO Merge this docs to dependencies.c, as it's saying mostly the
* same things as the comments there.
+ *
+ * TODO Currently this is applied only to the top-level clauses, but
+ * maybe we could apply it to lists at subtrees too, e.g. to the
+ * two AND-clauses in
+ *
+ * (x=1 AND y=2) OR (z=3 AND q=10)
+ *
*/
static List *
clauselist_apply_dependencies(PlannerInfo *root, List *clauses, Oid varRelid,
@@ -1195,17 +1747,27 @@ clauselist_apply_dependencies(PlannerInfo *root, List *clauses, Oid varRelid,
*/
foreach (lc, clauses)
{
- AttrNumber attnum;
+ Bitmapset *attnums = NULL;
Node *clause = (Node *) lfirst(lc);
- if (! clause_is_mv_compatible(root, clause, varRelid, &relid, &attnum, sjinfo))
+ if (! clause_is_mv_compatible(root, clause, varRelid, &relid, &attnums,
+ sjinfo, MV_CLAUSE_TYPE_FDEP))
+ reduced_clauses = lappend(reduced_clauses, clause);
+ else if (bms_num_members(attnums) > 1)
+ /* FIXME This may happen thanks to OR-clauses, which should
+ * really be handled differently for functional
+ * dependencies.
+ */
reduced_clauses = lappend(reduced_clauses, clause);
else
{
+ /* functional dependencies support only [Var = Const] */
+ Assert(bms_num_members(attnums) == 1);
mvclauses[nmvclauses] = clause;
- mvattnums[nmvclauses] = attnum;
+ mvattnums[nmvclauses] = bms_singleton_member(attnums);
nmvclauses++;
- clause_attnums = bms_add_member(clause_attnums, attnum);
+ clause_attnums = bms_add_member(clause_attnums,
+ bms_singleton_member(attnums));
}
}
@@ -1430,3 +1992,446 @@ clauselist_apply_dependencies(PlannerInfo *root, List *clauses, Oid varRelid,
return reduced_clauses;
}
+
+/*
+ * Estimate selectivity of clauses using a MCV list.
+ *
+ * If there's no MCV list for the stats, the function returns 0.0.
+ *
+ * While computing the estimate, the function checks whether all the
+ * columns were matched with an equality condition. If that's the case,
+ * we can skip processing the histogram, as there can be no rows in
+ * it with the same values - all the rows matching the condition are
+ * represented by the MCV item. This can only happen with equality
+ * on all the attributes.
+ *
+ * The algorithm works like this:
+ *
+ * 1) mark all items as 'match'
+ * 2) walk through all the clauses
+ * 3) for a particular clause, walk through all the items
+ * 4) skip items that are already 'no match'
+ * 5) check clause for items that still match
+ * 6) sum frequencies for items to get selectivity
+ *
+ * The function also returns the frequency of the least frequent item
+ * on the MCV list, which may be useful for clamping estimate from the
+ * histogram (all items not present in the MCV list are less frequent).
+ * This however seems useful only for cases with conditions on all
+ * attributes.
+ *
+ * TODO This only handles AND-ed clauses, but it might work for OR-ed
+ * lists too - it just needs to reverse the logic a bit. I.e. start
+ * with 'no match' for all items, and mark the items as a match
+ * as the clauses are processed (and skip items that are 'match').
+ */
+static Selectivity
+clauselist_mv_selectivity_mcvlist(PlannerInfo *root, List *clauses,
+ MVStats mvstats, bool *fullmatch,
+ Selectivity *lowsel)
+{
+ int i;
+ Selectivity s = 0.0;
+ MCVList mcvlist = NULL;
+ int nmatches = 0;
+
+ /* match/mismatch bitmap for each MCV item */
+ char * matches = NULL;
+
+ Assert(clauses != NIL);
+ Assert(list_length(clauses) >= 2);
+
+ /* there's no MCV list built yet */
+ if (! mvstats->mcv_built)
+ return 0.0;
+
+ mcvlist = deserialize_mv_mcvlist(fetch_mv_mcvlist(mvstats->mvoid));
+
+ Assert(mcvlist != NULL);
+ Assert(mcvlist->nitems > 0);
+
+ /* by default all the MCV items match the clauses fully */
+ matches = palloc0(sizeof(char) * mcvlist->nitems);
+ memset(matches, MVSTATS_MATCH_FULL, sizeof(char)*mcvlist->nitems);
+
+ /* number of matching MCV items */
+ nmatches = mcvlist->nitems;
+
+ nmatches = update_match_bitmap_mcvlist(root, clauses,
+ mvstats->stakeys, mcvlist,
+ nmatches, matches,
+ lowsel, fullmatch, false);
+
+ /* sum frequencies for all the matching MCV items */
+ for (i = 0; i < mcvlist->nitems; i++)
+ {
+ if (matches[i] != MVSTATS_MATCH_NONE)
+ s += mcvlist->items[i]->frequency;
+ }
+
+ pfree(matches);
+ pfree(mcvlist);
+
+ return s;
+}
+
+/*
+ * Evaluate clauses using the MCV list, and update the match bitmap.
+ *
+ * The bitmap may be already partially set, so this is really a way to
+ * combine results of several clause lists - either when computing
+ * conditional probability P(A|B) or a combination of AND/OR clauses.
+ *
+ * TODO This works with 'bitmap' where each bit is represented as a char,
+ * which is slightly wasteful. Instead, we could use a regular
+ * bitmap, reducing the size to ~1/8. Another thing is merging the
+ * bitmaps using & and |, which might be faster than min/max.
+ */
+static int
+update_match_bitmap_mcvlist(PlannerInfo *root, List *clauses,
+ int2vector *stakeys, MCVList mcvlist,
+ int nmatches, char * matches,
+ Selectivity *lowsel, bool *fullmatch,
+ bool is_or)
+{
+ int i;
+ ListCell * l;
+
+ Bitmapset *eqmatches = NULL; /* attributes with equality matches */
+
+ /* The bitmap may be partially built. */
+ Assert(nmatches >= 0);
+ Assert(nmatches <= mcvlist->nitems);
+ Assert(clauses != NIL);
+ Assert(list_length(clauses) >= 1);
+ Assert(mcvlist != NULL);
+ Assert(mcvlist->nitems > 0);
+
+ /* No possible matches (only works for AND-ded clauses) */
+ if (((nmatches == 0) && (! is_or)) ||
+ ((nmatches == mcvlist->nitems) && is_or))
+ return nmatches;
+
+ /* frequency of the lowest MCV item */
+ *lowsel = 1.0;
+
+ /*
+ * Loop through the list of clauses, and for each of them evaluate
+ * all the MCV items not yet eliminated by the preceding clauses.
+ *
+ * FIXME This would probably deserve a refactoring, I guess. Unify
+ * the two loops and put the checks inside, or something like
+ * that.
+ */
+ foreach (l, clauses)
+ {
+ Node * clause = (Node*)lfirst(l);
+
+ /* if it's a RestrictInfo, then extract the clause */
+ if (IsA(clause, RestrictInfo))
+ clause = (Node*)((RestrictInfo*)clause)->clause;
+
+ /* if there are no remaining matches possible, we can stop */
+ if (((nmatches == 0) && (! is_or)) ||
+ ((nmatches == mcvlist->nitems) && is_or))
+ break;
+
+ /* it's either OpClause, or NullTest */
+ if (is_opclause(clause))
+ {
+ OpExpr * expr = (OpExpr*)clause;
+ bool varonleft = true;
+ bool ok;
+
+ /* operator */
+ FmgrInfo opproc;
+
+ fmgr_info(get_opcode(expr->opno), &opproc);
+
+ ok = (NumRelids(clause) == 1) &&
+ (is_pseudo_constant_clause(lsecond(expr->args)) ||
+ (varonleft = false,
+ is_pseudo_constant_clause(linitial(expr->args))));
+
+ if (ok)
+ {
+
+ FmgrInfo ltproc, gtproc;
+ Var * var = (varonleft) ? linitial(expr->args) : lsecond(expr->args);
+ Const * cst = (varonleft) ? lsecond(expr->args) : linitial(expr->args);
+ bool isgt = (! varonleft);
+
+ /*
+ * TODO Fetch only when really needed (probably for equality only)
+ * TODO Technically either lt/gt is sufficient.
+ *
+ * FIXME The code in analyze.c creates histograms only for types
+ * with enough ordering (by calling get_sort_group_operators).
+ * Is this the same assumption, i.e. are we certain that we
+ * get the ltproc/gtproc every time we ask? Or are there types
+ * where get_sort_group_operators returns ltopr and here we
+ * get nothing?
+ */
+ TypeCacheEntry *typecache
+ = lookup_type_cache(var->vartype,
+ TYPECACHE_EQ_OPR | TYPECACHE_LT_OPR | TYPECACHE_GT_OPR);
+
+ /* FIXME proper matching attribute to dimension */
+ int idx = mv_get_index(var->varattno, stakeys);
+
+ fmgr_info(get_opcode(typecache->lt_opr), <proc);
+ fmgr_info(get_opcode(typecache->gt_opr), >proc);
+
+ /*
+ * Walk through the MCV items and evaluate the current clause. We can
+ * skip items that were already ruled out, and terminate if there are
+ * no remaining MCV items that might possibly match.
+ */
+ for (i = 0; i < mcvlist->nitems; i++)
+ {
+ bool mismatch = false;
+ MCVItem item = mcvlist->items[i];
+
+ /*
+ * find the lowest selectivity in the MCV
+ * FIXME Maybe not the best place do do this (in for all clauses).
+ */
+ if (item->frequency < *lowsel)
+ *lowsel = item->frequency;
+
+ /*
+ * If there are no more matches (AND) or no remaining unmatched
+ * items (OR), we can stop processing this clause.
+ */
+ if (((nmatches == 0) && (! is_or)) ||
+ ((nmatches == mcvlist->nitems) && is_or))
+ break;
+
+ /*
+ * For AND-lists, we can also mark NULL items as 'no match' (and
+ * then skip them). For OR-lists this is not possible.
+ */
+ if ((! is_or) && item->isnull[idx])
+ matches[i] = MVSTATS_MATCH_NONE;
+
+ /* skip MCV items that were already ruled out */
+ if ((! is_or) && (matches[i] == MVSTATS_MATCH_NONE))
+ continue;
+ else if (is_or && (matches[i] == MVSTATS_MATCH_FULL))
+ continue;
+
+ /* TODO consider bsearch here (list is sorted by values)
+ * TODO handle other operators too (LT, GT)
+ * TODO identify "full match" when the clauses fully
+ * match the whole MCV list (so that checking the
+ * histogram is not needed)
+ */
+ if (get_oprrest(expr->opno) == F_EQSEL)
+ {
+ /*
+ * We don't care about isgt in equality, because it does not
+ * matter whether it's (var = const) or (const = var).
+ */
+ bool match = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ cst->constvalue,
+ item->values[idx]));
+
+ if (match)
+ eqmatches = bms_add_member(eqmatches, idx);
+
+ mismatch = (! match);
+ }
+ else if (get_oprrest(expr->opno) == F_SCALARLTSEL) /* column < constant */
+ {
+
+ if (! isgt) /* (var < const) */
+ {
+ /*
+ * First check whether the constant is below the lower boundary (in that
+ * case we can skip the bucket, because there's no overlap).
+ */
+ mismatch = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ cst->constvalue,
+ item->values[idx]));
+
+ } /* (get_oprrest(expr->opno) == F_SCALARLTSEL) */
+ else /* (const < var) */
+ {
+ /*
+ * First check whether the constant is above the upper boundary (in that
+ * case we can skip the bucket, because there's no overlap).
+ */
+ mismatch = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ item->values[idx],
+ cst->constvalue));
+ }
+ }
+ else if (get_oprrest(expr->opno) == F_SCALARGTSEL) /* column > constant */
+ {
+
+ if (! isgt) /* (var > const) */
+ {
+ /*
+ * First check whether the constant is above the upper boundary (in that
+ * case we can skip the bucket, because there's no overlap).
+ */
+ mismatch = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ cst->constvalue,
+ item->values[idx]));
+ }
+ else /* (const > var) */
+ {
+ /*
+ * First check whether the constant is below the lower boundary (in
+ * that case we can skip the bucket, because there's no overlap).
+ */
+ mismatch = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ item->values[idx],
+ cst->constvalue));
+ }
+
+ } /* (get_oprrest(expr->opno) == F_SCALARGTSEL) */
+
+ /* XXX The conditions on matches[i] are not needed, as we
+ * skip MCV items that can't become true/false, depending
+ * on the current flag. See beginning of the loop over
+ * MCV items.
+ */
+
+ if ((is_or) && (matches[i] == MVSTATS_MATCH_NONE) && (! mismatch))
+ {
+ /* OR - was MATCH_NONE, but will be MATCH_FULL */
+ matches[i] = MVSTATS_MATCH_FULL;
+ ++nmatches;
+ continue;
+ }
+ else if ((! is_or) && (matches[i] == MVSTATS_MATCH_FULL) && mismatch)
+ {
+ /* AND - was MATC_FULL, but will be MATCH_NONE */
+ matches[i] = MVSTATS_MATCH_NONE;
+ --nmatches;
+ continue;
+ }
+
+ }
+ }
+ }
+ else if (IsA(clause, NullTest))
+ {
+ NullTest * expr = (NullTest*)clause;
+ Var * var = (Var*)(expr->arg);
+
+ /* FIXME proper matching attribute to dimension */
+ int idx = mv_get_index(var->varattno, stakeys);
+
+ /*
+ * Walk through the MCV items and evaluate the current clause. We can
+ * skip items that were already ruled out, and terminate if there are
+ * no remaining MCV items that might possibly match.
+ */
+ for (i = 0; i < mcvlist->nitems; i++)
+ {
+ MCVItem item = mcvlist->items[i];
+
+ /*
+ * find the lowest selectivity in the MCV
+ * FIXME Maybe not the best place do do this (in for all clauses).
+ */
+ if (item->frequency < *lowsel)
+ *lowsel = item->frequency;
+
+ /* if there are no more matches, we can stop processing this clause */
+ if (nmatches == 0)
+ break;
+
+ /* skip MCV items that were already ruled out */
+ if (matches[i] == MVSTATS_MATCH_NONE)
+ continue;
+
+ /* if the clause mismatches the MCV item, set it as MATCH_NONE */
+ if (((expr->nulltesttype == IS_NULL) && (! mcvlist->items[i]->isnull[idx])) ||
+ ((expr->nulltesttype == IS_NOT_NULL) && (mcvlist->items[i]->isnull[idx])))
+ {
+ matches[i] = MVSTATS_MATCH_NONE;
+ --nmatches;
+ }
+ }
+ }
+ else if (or_clause(clause) || and_clause(clause))
+ {
+ /* AND/OR clause, with all clauses compatible with the selected MV stat */
+
+ int i;
+ BoolExpr *orclause = ((BoolExpr*)clause);
+ List *orclauses = orclause->args;
+
+ /* match/mismatch bitmap for each MCV item */
+ int or_nmatches = 0;
+ char * or_matches = NULL;
+
+ Assert(orclauses != NIL);
+ Assert(list_length(orclauses) >= 2);
+
+ /* number of matching MCV items */
+ or_nmatches = mcvlist->nitems;
+
+ /* by default none of the MCV items matches the clauses */
+ or_matches = palloc0(sizeof(char) * or_nmatches);
+
+ if (or_clause(clause))
+ {
+ /* OR clauses assume nothing matches, initially */
+ memset(or_matches, MVSTATS_MATCH_NONE, sizeof(char)*or_nmatches);
+ or_nmatches = 0;
+ }
+ else
+ {
+ /* AND clauses assume nothing matches, initially */
+ memset(or_matches, MVSTATS_MATCH_FULL, sizeof(char)*or_nmatches);
+ }
+
+ /* build the match bitmap for the OR-clauses */
+ or_nmatches = update_match_bitmap_mcvlist(root, orclauses,
+ stakeys, mcvlist,
+ or_nmatches, or_matches,
+ lowsel, fullmatch, or_clause(clause));
+
+ /* merge the bitmap into the existing one*/
+ for (i = 0; i < mcvlist->nitems; i++)
+ {
+ /*
+ * To AND-merge the bitmaps, a MIN() semantics is used.
+ * For OR-merge, use MAX().
+ *
+ * FIXME this does not decrease the number of matches
+ */
+ UPDATE_RESULT(matches[i], or_matches[i], is_or);
+ }
+
+ pfree(or_matches);
+
+ }
+ else
+ {
+ elog(ERROR, "unknown clause type: %d", clause->type);
+ }
+ }
+
+ /*
+ * If all the columns were matched by equality, it's a full match.
+ * In this case there can be just a single MCV item, matching the
+ * clause (if there were two, both would match the other one).
+ */
+ *fullmatch = (bms_num_members(eqmatches) == mcvlist->ndimensions);
+
+ /* free the allocated pieces */
+ if (eqmatches)
+ pfree(eqmatches);
+
+ return nmatches;
+}
diff --git a/src/backend/utils/mvstats/Makefile b/src/backend/utils/mvstats/Makefile
index 099f1ed..3c0aff4 100644
--- a/src/backend/utils/mvstats/Makefile
+++ b/src/backend/utils/mvstats/Makefile
@@ -12,6 +12,6 @@ subdir = src/backend/utils/mvstats
top_builddir = ../../../..
include $(top_builddir)/src/Makefile.global
-OBJS = common.o dependencies.o
+OBJS = common.o mcv.o dependencies.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/utils/mvstats/common.c b/src/backend/utils/mvstats/common.c
index d44b95a..bd952c6 100644
--- a/src/backend/utils/mvstats/common.c
+++ b/src/backend/utils/mvstats/common.c
@@ -17,8 +17,8 @@
#include "common.h"
static VacAttrStats ** lookup_var_attr_stats(int2vector *attrs,
- int natts, VacAttrStats **vacattrstats);
-
+ int natts,
+ VacAttrStats **vacattrstats);
/*
* Compute requested multivariate stats, using the rows sampled for the
@@ -44,6 +44,8 @@ build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
for (i = 0; i < nmvstats; i++)
{
MVDependencies deps = NULL;
+ MCVList mcvlist = NULL;
+ int numrows_filtered = 0;
/* int2 vector of attnums the stats should be computed on */
int2vector * attrs = mvstats[i].stakeys;
@@ -60,8 +62,12 @@ build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
if (mvstats->deps_enabled)
deps = build_mv_dependencies(numrows, rows, attrs, stats);
+ /* build the MCV list */
+ if (mvstats->mcv_enabled)
+ mcvlist = build_mv_mcvlist(numrows, rows, attrs, stats, &numrows_filtered);
+
/* store the histogram / MCV list in the catalog */
- update_mv_stats(mvstats[i].mvoid, deps);
+ update_mv_stats(mvstats[i].mvoid, deps, mcvlist, attrs, stats);
}
}
@@ -143,7 +149,7 @@ list_mv_stats(Oid relid, int *nstats, bool built_only)
* Skip statistics that were not computed yet (if only stats
* that were already built were requested)
*/
- if (built_only && (! stats->deps_built))
+ if (built_only && (! (stats->mcv_built || stats->deps_built)))
continue;
/* double the array size if needed */
@@ -156,7 +162,9 @@ list_mv_stats(Oid relid, int *nstats, bool built_only)
result[*nstats].mvoid = HeapTupleGetOid(htup);
result[*nstats].stakeys = buildint2vector(stats->stakeys.values, stats->stakeys.dim1);
result[*nstats].deps_enabled = stats->deps_enabled;
+ result[*nstats].mcv_enabled = stats->mcv_enabled;
result[*nstats].deps_built = stats->deps_built;
+ result[*nstats].mcv_built = stats->mcv_built;
*nstats += 1;
}
@@ -171,7 +179,9 @@ list_mv_stats(Oid relid, int *nstats, bool built_only)
}
void
-update_mv_stats(Oid mvoid, MVDependencies dependencies)
+update_mv_stats(Oid mvoid,
+ MVDependencies dependencies, MCVList mcvlist,
+ int2vector *attrs, VacAttrStats **stats)
{
HeapTuple stup,
oldtup;
@@ -196,15 +206,26 @@ update_mv_stats(Oid mvoid, MVDependencies dependencies)
= PointerGetDatum(serialize_mv_dependencies(dependencies));
}
+ if (mcvlist != NULL)
+ {
+ bytea * data = serialize_mv_mcvlist(mcvlist, attrs, stats);
+ nulls[Anum_pg_mv_statistic_stamcv -1] = (data == NULL);
+ values[Anum_pg_mv_statistic_stamcv - 1] = PointerGetDatum(data);
+ }
+
/* always replace the value (either by bytea or NULL) */
replaces[Anum_pg_mv_statistic_stadeps -1] = true;
+ replaces[Anum_pg_mv_statistic_stamcv -1] = true;
/* always change the availability flags */
nulls[Anum_pg_mv_statistic_deps_built -1] = false;
+ nulls[Anum_pg_mv_statistic_mcv_built -1] = false;
replaces[Anum_pg_mv_statistic_deps_built-1] = true;
+ replaces[Anum_pg_mv_statistic_mcv_built -1] = true;
values[Anum_pg_mv_statistic_deps_built-1] = BoolGetDatum(dependencies != NULL);
+ values[Anum_pg_mv_statistic_mcv_built -1] = BoolGetDatum(mcvlist != NULL);
/* Is there already a pg_mv_statistic tuple for this attribute? */
oldtup = SearchSysCache1(MVSTATOID,
@@ -232,6 +253,21 @@ update_mv_stats(Oid mvoid, MVDependencies dependencies)
heap_close(sd, RowExclusiveLock);
}
+
+int
+mv_get_index(AttrNumber varattno, int2vector * stakeys)
+{
+ int i, idx = 0;
+ for (i = 0; i < stakeys->dim1; i++)
+ {
+ if (stakeys->values[i] < varattno)
+ idx += 1;
+ else
+ break;
+ }
+ return idx;
+}
+
/* multi-variate stats comparator */
/*
@@ -242,11 +278,15 @@ update_mv_stats(Oid mvoid, MVDependencies dependencies)
int
compare_scalars_simple(const void *a, const void *b, void *arg)
{
- Datum da = *(Datum*)a;
- Datum db = *(Datum*)b;
- SortSupport ssup= (SortSupport) arg;
+ return compare_datums_simple(*(Datum*)a,
+ *(Datum*)b,
+ (SortSupport)arg);
+}
- return ApplySortComparator(da, false, db, false, ssup);
+int
+compare_datums_simple(Datum a, Datum b, SortSupport ssup)
+{
+ return ApplySortComparator(a, false, b, false, ssup);
}
/*
diff --git a/src/backend/utils/mvstats/common.h b/src/backend/utils/mvstats/common.h
index 6d5465b..f4309f7 100644
--- a/src/backend/utils/mvstats/common.h
+++ b/src/backend/utils/mvstats/common.h
@@ -46,7 +46,15 @@ typedef struct
Datum value; /* a data value */
int tupno; /* position index for tuple it came from */
} ScalarItem;
-
+
+/* (de)serialization info */
+typedef struct DimensionInfo {
+ int nvalues; /* number of deduplicated values */
+ int nbytes; /* number of bytes (serialized) */
+ int typlen; /* pg_type.typlen */
+ bool typbyval; /* pg_type.typbyval */
+} DimensionInfo;
+
/* multi-sort */
typedef struct MultiSortSupportData {
int ndims; /* number of dimensions supported by the */
@@ -71,5 +79,6 @@ int multi_sort_compare_dim(int dim, const SortItem *a,
const SortItem *b, MultiSortSupport mss);
/* comparators, used when constructing multivariate stats */
+int compare_datums_simple(Datum a, Datum b, SortSupport ssup);
int compare_scalars_simple(const void *a, const void *b, void *arg);
int compare_scalars_partition(const void *a, const void *b, void *arg);
diff --git a/src/backend/utils/mvstats/mcv.c b/src/backend/utils/mvstats/mcv.c
new file mode 100644
index 0000000..4466cee
--- /dev/null
+++ b/src/backend/utils/mvstats/mcv.c
@@ -0,0 +1,1002 @@
+/*-------------------------------------------------------------------------
+ *
+ * mcv.c
+ * POSTGRES multivariate MCV lists
+ *
+ *
+ * Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/utils/mvstats/mcv.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "common.h"
+
+/*
+ * Multivariate MCVs (most-common values lists) are a straightforward
+ * extension of regular MCV list by tracking combinations of values for
+ * several attributes (columns), including NULL flags, and frequency
+ * of the combination.
+ *
+ * For columns small number of distinct values, this works quite well
+ * and may represent the distribution pretty exactly. For columns with
+ * large number of distinct values (e.g. stored as FLOAT), this does
+ * not work that well.
+ *
+ * If we can represent the distribution as a MCV list, we can estimate
+ * some clauses (e.g. equality clauses) much accurately than using
+ * histograms for example.
+ *
+ * Discrete distributions are also easier to combine into a larger
+ * distribution (but this is not yet implemented).
+ *
+ *
+ * TODO For types that don't reasonably support ordering (either because
+ * the type does not support that or when the user adds some option
+ * to the ADD STATISTICS command - e.g. UNSORTED_STATS), building
+ * the histogram may be pointless and inefficient. This is esp.
+ * true for varlena types that may be quite large and a large MCV
+ * list may be a better choice, because it makes equality estimates
+ * more accurate. Due to the unsorted nature, range queries on those
+ * attributes are rather useless anyway.
+ *
+ * Another thing is that by restricting to MCV list and equality
+ * conditions, we can use hash values instead of long varlena values.
+ * The equality estimation will be very accurate.
+ *
+ * This however complicates matching the columns to available
+ * statistics, as it will require matching clauses (not columns) to
+ * stats. And it may get quite complex - e.g. what if there are
+ * multiple clauses, each compatible with different stats subset?
+ *
+ *
+ * Selectivity estimation
+ * ----------------------
+ * The estimation, implemented in clauselist_mv_selectivity_mcvlist(),
+ * is quite simple in principle - walk through the MCV items and sum
+ * frequencies of all the items that match all the clauses.
+ *
+ * The current implementation uses MCV lists to estimates those types
+ * of clauses (think of WHERE conditions):
+ *
+ * (a) equality clauses WHERE (a = 1) AND (b = 2)
+ *
+ * (b) inequality clauses WHERE (a < 1) AND (b >= 2)
+ *
+ * It's possible to add more clauses, for example:
+ *
+ * (a) NULL clauses WHERE (a IS NULL) AND (b IS NOT NULL)
+ *
+ * (b) multi-var clauses WHERE (a > b)
+ *
+ * and so on. These are tasks for the future, not yet implemented.
+ *
+ *
+ * Estimating equality clauses
+ * ---------------------------
+ * When computing selectivity estimate for equality clauses
+ *
+ * (a = 1) AND (b = 2)
+ *
+ * we can do this estimate pretty exactly assuming that two conditions
+ * are met:
+ *
+ * (1) there's an equality condition on each attribute
+ *
+ * (2) we find a matching item in the MCV list
+ *
+ * In that case we know the MCV item represents all the tuples matching
+ * the clauses, and the selectivity estimate is complete. This is what
+ * we call 'full match'.
+ *
+ * When only (1) holds, but there's no matching MCV item, we don't know
+ * whether there are no such rows or just are not very frequent. We can
+ * however use the frequency of the least frequent MCV item as an upper
+ * bound for the selectivity.
+ *
+ * If the equality conditions match only a subset of the attributes
+ * the MCV list is built on (i.e. we can't get a full match - we may get
+ * multiple MCV items matching the clauses, but even if we get a single
+ * match there may be items that did not get into the MCV list. But in
+ * this case we can still use the frequency of the last MCV item to clam
+ * the 'additional' selectivity not accounted for by the matching items.
+ *
+ * If there's no histogram, because the MCV list approximates the
+ * distribution accurately (not because the histogram was disabled),
+ * it does not really matter whether there are equality conditions on
+ * all the columns - we can do pretty accurate estimation using the MCV.
+ *
+ * TODO For a combination of equality conditions (not full-match case)
+ * we probably can clamp the selectivity by the minimum of
+ * selectivities for each condition. For example if we know the
+ * number of distinct values for each column, we can use 1/ndistinct
+ * as a per-column estimate. Or rather 1/ndistinct + selectivity
+ * derived from the MCV list.
+ *
+ * If we know the estimate of number of combinations of the columns
+ * (i.e. ndistinct(A,B)), we may estimate the average frequency of
+ * items in the remaining 10% as [10% / ndistinct(A,B)].
+ *
+ *
+ * Bounding estimates
+ * ------------------
+ * In general the MCV lists may not provide estimates as accurate as
+ * for the full-match equality case, but may provide some useful
+ * lower/upper boundaries for the estimation error.
+ *
+ * With equality clauses we can do a few more tricks to narrow this
+ * error range (see the previous section and TODO), but with inequality
+ * clauses (or generally non-equality clauses), it's rather dificult.
+ * There's nothing like a 'full match' - we have to consider both the
+ * MCV items and the remaining part every time. We can't use the minimum
+ * selectivity of MCV items, as the clauses may match multiple items.
+ *
+ * For example with a MCV list on columns (A, B), covering 90% of the
+ * table (computed while building the MCV list), about ~10% of the table
+ * is not represented by the MCV list. So even if the conditions match
+ * all the remaining rows (not represented by the MCV items), we can't
+ * get selectivity higher than those 10%. We may use 1/2 the remaining
+ * selectivity as an estimate (minimizing average error).
+ *
+ * TODO Most of these ideas (error limiting) are not yet implemented.
+ *
+ *
+ * General TODO
+ * ------------
+ *
+ * FIXME Use max_mcv_items from ALTER TABLE ADD STATISTICS command.
+ *
+ * TODO Add support for IS [NOT] NULL clauses, and clauses referencing
+ * multiple columns (a < b).
+ *
+ * TODO It's possible to build a special case of MCV list, storing not
+ * the actual values but only 32/64-bit hash. This is only useful
+ * for estimating equality clauses and for large varlena types,
+ * which are very impractical for plain MCV list because of size.
+ * But for those data types we really want just the equality
+ * clauses, so it's actually a good solution.
+ *
+ * TODO Currently there's no logic to consider building only a MCV list
+ * (and not building the histogram at all), except for doing this
+ * decision manually in ADD STATISTICS.
+ */
+
+/*
+ * Each serialized item needs to store (in this order):
+ *
+ * - indexes (ndim * sizeof(int32))
+ * - null flags (ndim * sizeof(bool))
+ * - frequency (sizeof(double))
+ *
+ * So in total:
+ *
+ * ndim * (sizeof(int32) + sizeof(bool)) + sizeof(double)
+ */
+#define ITEM_SIZE(ndims) \
+ (ndims * (sizeof(int32) + sizeof(bool)) + sizeof(double))
+
+/* pointers into a flat serialized item of ITEM_SIZE(n) bytes */
+#define ITEM_INDEXES(item) ((int32*)item)
+#define ITEM_NULLS(item,ndims) ((bool*)(ITEM_INDEXES(item) + ndims))
+#define ITEM_FREQUENCY(item,ndims) ((double*)(ITEM_NULLS(item,ndims) + ndims))
+
+/*
+ * Builds MCV list from sample rows, and removes rows represented by
+ * the MCV list from the sample (the number of remaining sample rows is
+ * returned by the numrows_filtered parameter).
+ *
+ * The method is quite simple - in short it does about these steps:
+ *
+ * (1) sort the data (default collation, '<' for the data type)
+ *
+ * (2) count distinct groups, decide how many to keep
+ *
+ * (3) build the MCV list using the threshold determined in (2)
+ *
+ * (4) remove rows represented by the MCV from the sample
+ *
+ * For more details, see the comments in the code.
+ *
+ * FIXME Single-dimensional MCV is sorted by frequency (descending). We
+ * should do that too, because when walking through the list we
+ * want to check the most frequent items first.
+ *
+ * TODO We're using Datum (8B), even for data types (e.g. int4 or
+ * float4). Maybe we could save some space here, but the bytea
+ * compression should handle it just fine.
+ *
+ * TODO This probably should not use the ndistinct directly (as computed
+ * from the table, but rather estimate the number of distinct
+ * values in the table), no?
+ */
+MCVList
+build_mv_mcvlist(int numrows, HeapTuple *rows, int2vector *attrs,
+ VacAttrStats **stats, int *numrows_filtered)
+{
+ int i, j;
+ int numattrs = attrs->dim1;
+ int ndistinct = 0;
+ int mcv_threshold = 0;
+ int count = 0;
+ int nitems = 0;
+
+ MCVList mcvlist = NULL;
+
+ /* Sort by multiple columns (using array of SortSupport) */
+ MultiSortSupport mss = multi_sort_init(numattrs);
+
+ /*
+ * Preallocate space for all the items as a single chunk, and point
+ * the items to the appropriate parts of the array.
+ */
+ SortItem *items = (SortItem*)palloc0(numrows * sizeof(SortItem));
+ Datum *values = (Datum*)palloc0(sizeof(Datum) * numrows * numattrs);
+ bool *isnull = (bool*)palloc0(sizeof(bool) * numrows * numattrs);
+
+ /* keep all the rows by default (as if there was no MCV list) */
+ *numrows_filtered = numrows;
+
+ for (i = 0; i < numrows; i++)
+ {
+ items[i].values = &values[i * numattrs];
+ items[i].isnull = &isnull[i * numattrs];
+ }
+
+ /* load the values/null flags from sample rows */
+ for (j = 0; j < numrows; j++)
+ for (i = 0; i < numattrs; i++)
+ items[j].values[i] = heap_getattr(rows[j], attrs->values[i],
+ stats[i]->tupDesc, &items[j].isnull[i]);
+
+ /* prepare the sort functions for all the attributes */
+ for (i = 0; i < numattrs; i++)
+ multi_sort_add_dimension(mss, i, i, stats);
+
+ /* do the sort, using the multi-sort */
+ qsort_arg((void *) items, numrows, sizeof(SortItem),
+ multi_sort_compare, mss);
+
+ /*
+ * Count the number of distinct groups - just walk through the
+ * sorted list and count the number of key changes. We use this to
+ * determine the threshold (125% of the average frequency).
+ */
+ ndistinct = 1;
+ for (i = 1; i < numrows; i++)
+ if (multi_sort_compare(&items[i], &items[i-1], mss) != 0)
+ ndistinct += 1;
+
+ /*
+ * Determine how many groups actually exceed the threshold, and then
+ * walk the array again and collect them into an array. We'll always
+ * require at least 4 rows per group.
+ *
+ * But if we can fit all the distinct values in the MCV list (i.e.
+ * if there are less distinct groups than MVSTAT_MCVLIST_MAX_ITEMS),
+ * we'll require only 2 rows per group.
+ *
+ * TODO For now the threshold is the same as in the single-column
+ * case (average + 25%), but maybe that's worth revisiting
+ * for the multivariate case.
+ *
+ * TODO We can do this only if we believe we got all the distinct
+ * values of the table.
+ *
+ * FIXME This should really reference mcv_max_items (from catalog)
+ * instead of the constant MVSTAT_MCVLIST_MAX_ITEMS.
+ */
+ mcv_threshold = 1.25 * numrows / ndistinct;
+ mcv_threshold = (mcv_threshold < 4) ? 4 : mcv_threshold;
+
+ if (ndistinct <= MVSTAT_MCVLIST_MAX_ITEMS)
+ mcv_threshold = 2;
+
+ /*
+ * Walk through the sorted data again, and see how many groups
+ * reach the mcv_threshold (and become an item in the MCV list).
+ */
+ count = 1;
+ for (i = 1; i <= numrows; i++)
+ {
+ /* last row or new group, so check if we exceed mcv_threshold */
+ if ((i == numrows) || (multi_sort_compare(&items[i], &items[i-1], mss) != 0))
+ {
+ /* group hits the threshold, count the group as MCV item */
+ if (count >= mcv_threshold)
+ nitems += 1;
+
+ count = 1;
+ }
+ else /* within group, so increase the number of items */
+ count += 1;
+ }
+
+ /* we know the number of MCV list items, so let's build the list */
+ if (nitems > 0)
+ {
+ /* allocate the MCV list structure, set parameters we know */
+ mcvlist = (MCVList)palloc0(sizeof(MCVListData));
+
+ mcvlist->magic = MVSTAT_MCV_MAGIC;
+ mcvlist->type = MVSTAT_MCV_TYPE_BASIC;
+ mcvlist->ndimensions = numattrs;
+ mcvlist->nitems = nitems;
+
+ /*
+ * Preallocate Datum/isnull arrays (not as a single chunk, as
+ * we'll pass this outside this method and thus it needs to be
+ * easy to pfree() the data (and we wouldn't know where the
+ * arrays start).
+ *
+ * TODO Maybe the reasoning that we can't allocate a single
+ * piece because we're passing it out is bogus? Who'd
+ * free a single item of the MCV list, anyway?
+ *
+ * TODO Maybe with a proper encoding (stuffing all the values
+ * into a list-level array, this will be untrue)?
+ */
+ mcvlist->items = (MCVItem*)palloc0(sizeof(MCVItem)*nitems);
+
+ for (i = 0; i < nitems; i++)
+ {
+ mcvlist->items[i] = (MCVItem)palloc0(sizeof(MCVItemData));
+ mcvlist->items[i]->values = (Datum*)palloc0(sizeof(Datum)*numattrs);
+ mcvlist->items[i]->isnull = (bool*)palloc0(sizeof(bool)*numattrs);
+ }
+
+ /*
+ * Repeat the same loop as above, but this time copy the data
+ * into the MCV list (for items exceeding the threshold).
+ *
+ * TODO Maybe we could simply remember indexes of the last item
+ * in each group (from the previous loop)?
+ */
+ count = 1;
+ nitems = 0;
+ for (i = 1; i <= numrows; i++)
+ {
+ /* last row or a new group */
+ if ((i == numrows) || (multi_sort_compare(&items[i], &items[i-1], mss) != 0))
+ {
+ /* count the MCV item if exceeding the threshold (and copy into the array) */
+ if (count >= mcv_threshold)
+ {
+ /* just pointer to the proper place in the list */
+ MCVItem item = mcvlist->items[nitems];
+
+ /* copy values from the _previous_ group (last item of) */
+ memcpy(item->values, items[(i-1)].values, sizeof(Datum) * numattrs);
+ memcpy(item->isnull, items[(i-1)].isnull, sizeof(bool) * numattrs);
+
+
+ /* and finally the group frequency */
+ item->frequency = (double)count / numrows;
+
+ /* next item */
+ nitems += 1;
+ }
+
+ count = 1;
+ }
+ else /* same group, just increase the number of items */
+ count += 1;
+ }
+
+ /* make sure the loops are consistent */
+ Assert(nitems == mcvlist->nitems);
+
+ /*
+ * Remove the rows matching the MCV list (i.e. keep only rows
+ * that are not represented by the MCV list).
+ *
+ * FIXME This implementation is rather naive, effectively O(N^2).
+ * As the MCV list grows, the check will take longer and
+ * longer. And as the number of sampled rows increases (by
+ * increasing statistics target), it will take longer and
+ * longer. One option is to sort the MCV items first and
+ * then perform a binary search.
+ *
+ * A better option would be keeping the ID of the row in
+ * the sort item, and then just walk through the items and
+ * mark rows to remove (in a bitmap of the same size).
+ * There's not space for that in SortItem at this moment,
+ * but it's trivial to add 'private' pointer, or just
+ * using another structure with extra field (starting with
+ * SortItem, so that the comparators etc. still work).
+ *
+ * Another option is to use the sorted array of items
+ * (because that's how we sorted the source data), and
+ * simply do a bsearch() into it. If we find a matching
+ * item, the row belongs to the MCV list.
+ */
+ if (nitems == ndistinct) /* all rows are covered by MCV items */
+ *numrows_filtered = 0;
+ else /* (nitems < ndistinct) && (nitems > 0) */
+ {
+ int nfiltered = 0;
+ HeapTuple *rows_filtered = (HeapTuple*)palloc0(sizeof(HeapTuple) * numrows);
+
+ /* used for the searches */
+ SortItem item, mcvitem;;
+
+ item.values = (Datum*)palloc0(numattrs * sizeof(Datum));
+ item.isnull = (bool*)palloc0(numattrs * sizeof(bool));
+
+ /*
+ * FIXME we don't need to allocate this, we can reference
+ * the MCV item directly ...
+ */
+ mcvitem.values = (Datum*)palloc0(numattrs * sizeof(Datum));
+ mcvitem.isnull = (bool*)palloc0(numattrs * sizeof(bool));
+
+ /* walk through the tuples, compare the values to MCV items */
+ for (i = 0; i < numrows; i++)
+ {
+ bool match = false;
+
+ /* collect the key values from the row */
+ for (j = 0; j < numattrs; j++)
+ item.values[j] = heap_getattr(rows[i], attrs->values[j],
+ stats[j]->tupDesc, &item.isnull[j]);
+
+ /* scan through the MCV list for matches */
+ for (j = 0; j < mcvlist->nitems; j++)
+ {
+ /*
+ * TODO Create a SortItem/MCVItem comparator so that
+ * we don't need to do memcpy() like crazy.
+ */
+ memcpy(mcvitem.values, mcvlist->items[j]->values,
+ numattrs * sizeof(Datum));
+ memcpy(mcvitem.isnull, mcvlist->items[j]->isnull,
+ numattrs * sizeof(bool));
+
+ if (multi_sort_compare(&item, &mcvitem, mss) == 0)
+ {
+ match = true;
+ break;
+ }
+ }
+
+ /* if no match in the MCV list, copy the row into the filtered ones */
+ if (! match)
+ memcpy(&rows_filtered[nfiltered++], &rows[i], sizeof(HeapTuple));
+ }
+
+ /* replace the rows and remember how many rows we kept */
+ memcpy(rows, rows_filtered, sizeof(HeapTuple) * nfiltered);
+ *numrows_filtered = nfiltered;
+
+ /* free all the data used here */
+ pfree(rows_filtered);
+ pfree(item.values);
+ pfree(item.isnull);
+ pfree(mcvitem.values);
+ pfree(mcvitem.isnull);
+ }
+ }
+
+ pfree(values);
+ pfree(items);
+ pfree(isnull);
+
+ return mcvlist;
+}
+
+
+/* fetch the MCV list (as a bytea) from the pg_mv_statistic catalog */
+bytea *
+fetch_mv_mcvlist(Oid mvoid)
+{
+ Relation indrel;
+ SysScanDesc indscan;
+ ScanKeyData skey;
+ HeapTuple htup;
+ bytea *mcvlist = NULL;
+
+ /* Prepare to scan pg_mv_statistic for entries having indrelid = this rel. */
+ ScanKeyInit(&skey,
+ ObjectIdAttributeNumber,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(mvoid));
+
+ indrel = heap_open(MvStatisticRelationId, AccessShareLock);
+ indscan = systable_beginscan(indrel, MvStatisticOidIndexId, true,
+ NULL, 1, &skey);
+
+ while (HeapTupleIsValid(htup = systable_getnext(indscan)))
+ {
+ bool isnull = false;
+ Datum tmp = SysCacheGetAttr(MVSTATOID, htup,
+ Anum_pg_mv_statistic_stamcv, &isnull);
+
+ Assert(!isnull);
+
+ mcvlist = DatumGetByteaP(tmp);
+
+ break;
+ }
+
+ systable_endscan(indscan);
+
+ heap_close(indrel, AccessShareLock);
+
+ /* TODO maybe save the list into relcache, as in RelationGetIndexList
+ * (which was used as an inspiration of this one)?. */
+
+ return mcvlist;
+}
+
+/* print some basic info about the MCV list
+ *
+ * TODO Add info about what part of the table this covers.
+ */
+Datum
+pg_mv_stats_mcvlist_info(PG_FUNCTION_ARGS)
+{
+ bytea *data = PG_GETARG_BYTEA_P(0);
+ char *result;
+
+ MCVList mcvlist = deserialize_mv_mcvlist(data);
+
+ result = palloc0(128);
+ snprintf(result, 128, "nitems=%d", mcvlist->nitems);
+
+ pfree(mcvlist);
+
+ PG_RETURN_TEXT_P(cstring_to_text(result));
+}
+
+/* used to pass context into bsearch() */
+static SortSupport ssup_private = NULL;
+
+static int bsearch_comparator(const void * a, const void * b);
+
+/*
+ * Serialize MCV list into a bytea value. The basic algorithm is simple:
+ *
+ * (1) perform deduplication for each attribute (separately)
+ * (a) collect all (non-NULL) attribute values from all MCV items
+ * (b) sort the data (using 'lt' from VacAttrStats)
+ * (c) remove duplicate values from the array
+ *
+ * (2) serialize the arrays into a bytea value
+ *
+ * (3) process all MCV list items
+ * (a) replace values with indexes into the arrays
+ *
+ * Each attribute has to be processed separately, because we're mixing
+ * different datatypes, and we don't know what equality means for them.
+ * We're also mixing pass-by-value and pass-by-ref types, and so on.
+ *
+ * We'll use 32-bit values for the indexes in step (3), although we
+ * could probably use just 16 bits as we don't allow more than 8k
+ * items in the MCV list max_mcv_items (well, we might increase this to
+ * 32k and still fit into signed 16-bits). But let's be lazy and rely
+ * on the varlena compression to kick in. If most bytes will be 0x00
+ * so it should work nicely.
+ *
+ * FIXME This probably leaks memory, or at least uses it inefficiently
+ * (many small palloc() calls instead of a large one).
+ *
+ * TODO Consider using 16-bit values for the indexes in step (3).
+ *
+ * TODO Consider packing boolean flags (NULL) for each item into 'char'
+ * or a longer type (instead of using an array of bool items).
+ */
+bytea *
+serialize_mv_mcvlist(MCVList mcvlist, int2vector *attrs,
+ VacAttrStats **stats)
+{
+ int i, j;
+ int ndims = mcvlist->ndimensions;
+ int itemsize = ITEM_SIZE(ndims);
+
+ Size total_length = 0;
+
+ char *item = palloc0(itemsize);
+
+ /* serialized items (indexes into arrays, etc.) */
+ bytea *output;
+ char *data = NULL;
+
+ /* values per dimension (and number of non-NULL values) */
+ Datum **values = (Datum**)palloc0(sizeof(Datum*) * ndims);
+ int *counts = (int*)palloc0(sizeof(int) * ndims);
+
+ /* info about dimensions (for deserialize) */
+ DimensionInfo * info
+ = (DimensionInfo *)palloc0(sizeof(DimensionInfo)*ndims);
+
+ /* sort support data */
+ SortSupport ssup = (SortSupport)palloc0(sizeof(SortSupportData)*ndims);
+
+ /* collect and deduplicate values for each dimension */
+ for (i = 0; i < ndims; i++)
+ {
+ int count;
+ StdAnalyzeData *tmp = (StdAnalyzeData *)stats[i]->extra_data;
+
+ /* keep important info about the data type */
+ info[i].typlen = stats[i]->attrtype->typlen;
+ info[i].typbyval = stats[i]->attrtype->typbyval;
+
+ /* allocate space for all values, including NULLs (won't use them) */
+ values[i] = (Datum*)palloc0(sizeof(Datum) * mcvlist->nitems);
+
+ for (j = 0; j < mcvlist->nitems; j++)
+ {
+ if (! mcvlist->items[j]->isnull[i]) /* skip NULL values */
+ {
+ values[i][counts[i]] = mcvlist->items[j]->values[i];
+ counts[i] += 1;
+ }
+ }
+
+ /* there are just NULL values in this dimension */
+ if (counts[i] == 0)
+ continue;
+
+ /* sort and deduplicate */
+ ssup[i].ssup_cxt = CurrentMemoryContext;
+ ssup[i].ssup_collation = DEFAULT_COLLATION_OID;
+ ssup[i].ssup_nulls_first = false;
+
+ PrepareSortSupportFromOrderingOp(tmp->ltopr, &ssup[i]);
+
+ qsort_arg(values[i], counts[i], sizeof(Datum),
+ compare_scalars_simple, &ssup[i]);
+
+ /*
+ * Walk through the array and eliminate duplicitate values, but
+ * keep the ordering (so that we can do bsearch later). We know
+ * there's at least 1 item, so we can skip the first element.
+ */
+ count = 1; /* number of deduplicated items */
+ for (j = 1; j < counts[i]; j++)
+ {
+ /* if it's different from the previous value, we need to keep it */
+ if (compare_datums_simple(values[i][j-1], values[i][j], &ssup[i]) != 0)
+ {
+ /* XXX: not needed if (count == j) */
+ values[i][count] = values[i][j];
+ count += 1;
+ }
+ }
+
+ /* keep info about the deduplicated count */
+ info[i].nvalues = count;
+
+ /* compute size of the serialized data */
+ if (info[i].typbyval)
+ /*
+ * passed by value, so just Datum array (int4, int8, ...)
+ *
+ * TODO Might save a few bytes here, by storing just typlen
+ * bytes instead of whole Datum (8B) on 64-bits.
+ */
+ info[i].nbytes = info[i].nvalues * sizeof(Datum);
+ else if (info[i].typlen > 0)
+ /* pased by reference, but fixed length (name, tid, ...) */
+ info[i].nbytes = info[i].nvalues * info[i].typlen;
+ else if (info[i].typlen == -1)
+ /* varlena, so just use VARSIZE_ANY */
+ for (j = 0; j < info[i].nvalues; j++)
+ info[i].nbytes += VARSIZE_ANY(values[i][j]);
+ else if (info[i].typlen == -2)
+ /* cstring, so simply strlen */
+ for (j = 0; j < info[i].nvalues; j++)
+ info[i].nbytes += strlen(DatumGetPointer(values[i][j]));
+ else
+ elog(ERROR, "unknown data type typbyval=%d typlen=%d",
+ info[i].typbyval, info[i].typlen);
+ }
+
+ /*
+ * Now we finally know how much space we'll need for the serialized
+ * MCV list, as it contains these fields:
+ *
+ * - length (4B) for varlena
+ * - magic (4B)
+ * - type (4B)
+ * - ndimensions (4B)
+ * - nitems (4B)
+ * - info (ndim * sizeof(DimensionInfo)
+ * - arrays of values for each dimension
+ * - serialized items (nitems * itemsize)
+ *
+ * So the 'header' size is 20B + ndim * sizeof(DimensionInfo) and
+ * then we'll place the data.
+ */
+ total_length = (sizeof(int32) + offsetof(MCVListData, items)
+ + ndims * sizeof(DimensionInfo)
+ + mcvlist->nitems * itemsize);
+
+ for (i = 0; i < ndims; i++)
+ total_length += info[i].nbytes;
+
+ /* enforce arbitrary limit of 1MB */
+ if (total_length > 1024 * 1024)
+ elog(ERROR, "serialized MCV exceeds 1MB (%ld)", total_length);
+
+ /* allocate space for the serialized MCV list, set header fields */
+ output = (bytea*)palloc0(total_length);
+ SET_VARSIZE(output, total_length);
+
+ /* we'll use 'ptr' to keep track of the place to write data */
+ data = VARDATA(output);
+
+ memcpy(data, mcvlist, offsetof(MCVListData, items));
+ data += offsetof(MCVListData, items);
+
+ memcpy(data, info, sizeof(DimensionInfo) * ndims);
+ data += sizeof(DimensionInfo) * ndims;
+
+ /* value array for each dimension */
+ for (i = 0; i < ndims; i++)
+ {
+#ifdef USE_ASSERT_CHECKING
+ char *tmp = data;
+#endif
+ for (j = 0; j < info[i].nvalues; j++)
+ {
+ if (info[i].typbyval)
+ {
+ /* passed by value / Datum */
+ memcpy(data, &values[i][j], sizeof(Datum));
+ data += sizeof(Datum);
+ }
+ else if (info[i].typlen > 0)
+ {
+ /* pased by reference, but fixed length (name, tid, ...) */
+ memcpy(data, &values[i][j], info[i].typlen);
+ data += info[i].typlen;
+ }
+ else if (info[i].typlen == -1)
+ {
+ /* varlena */
+ memcpy(data, DatumGetPointer(values[i][j]),
+ VARSIZE_ANY(values[i][j]));
+ data += VARSIZE_ANY(values[i][j]);
+ }
+ else if (info[i].typlen == -2)
+ {
+ /* cstring (don't forget the \0 terminator!) */
+ memcpy(data, DatumGetPointer(values[i][j]),
+ strlen(DatumGetPointer(values[i][j])) + 1);
+ data += strlen(DatumGetPointer(values[i][j])) + 1;
+ }
+ }
+ Assert((data - tmp) == info[i].nbytes);
+ }
+
+ /* and finally, the MCV items */
+ for (i = 0; i < mcvlist->nitems; i++)
+ {
+ /* don't write beyond the allocated space */
+ Assert(data <= (char*)output + total_length - itemsize);
+
+ /* reset the values for each item */
+ memset(item, 0, itemsize);
+
+ for (j = 0; j < ndims; j++)
+ {
+ /* do the lookup only for non-NULL values */
+ if (! mcvlist->items[i]->isnull[j])
+ {
+ Datum * v = NULL;
+ ssup_private = &ssup[j];
+
+ v = (Datum*)bsearch(&mcvlist->items[i]->values[j],
+ values[j], info[j].nvalues, sizeof(Datum),
+ bsearch_comparator);
+
+ if (v == NULL)
+ elog(ERROR, "value for dim %d not found in array", j);
+
+ /* compute index within the array */
+ ITEM_INDEXES(item)[j] = (v - values[j]);
+
+ /* check the index is within expected bounds */
+ Assert(ITEM_INDEXES(item)[j] >= 0);
+ Assert(ITEM_INDEXES(item)[j] < info[j].nvalues);
+ }
+ }
+
+ /* copy NULL and frequency flags into the item */
+ memcpy(ITEM_NULLS(item, ndims),
+ mcvlist->items[i]->isnull, sizeof(bool) * ndims);
+ memcpy(ITEM_FREQUENCY(item, ndims),
+ &mcvlist->items[i]->frequency, sizeof(double));
+
+ /* copy the item into the array */
+ memcpy(data, item, itemsize);
+
+ data += itemsize;
+ }
+
+ /* at this point we expect to match the total_length exactly */
+ Assert((data - (char*)output) == total_length);
+
+ return output;
+}
+
+/* inverse to serialize_mv_mcvlist() - see the comment there */
+MCVList deserialize_mv_mcvlist(bytea * data)
+{
+ int i, j;
+ Size expected_size;
+ MCVList mcvlist;
+ char *tmp;
+
+ int ndims, nitems, itemsize;
+ DimensionInfo *info = NULL;
+
+ int32 *indexes = NULL;
+ Datum **values = NULL;
+
+ if (data == NULL)
+ return NULL;
+
+ if (VARSIZE_ANY_EXHDR(data) < offsetof(MCVListData,items))
+ elog(ERROR, "invalid MCV Size %ld (expected at least %ld)",
+ VARSIZE_ANY_EXHDR(data), offsetof(MCVListData,items));
+
+ /* read the MCV list header */
+ mcvlist = (MCVList)palloc0(sizeof(MCVListData));
+
+ /* initialize pointer to the data part (skip the varlena header) */
+ tmp = VARDATA(data);
+
+ /* get the header and perform basic sanity checks */
+ memcpy(mcvlist, tmp, offsetof(MCVListData,items));
+ tmp += offsetof(MCVListData,items);
+
+ if (mcvlist->magic != MVSTAT_MCV_MAGIC)
+ elog(ERROR, "invalid MCV magic %d (expected %dd)",
+ mcvlist->magic, MVSTAT_MCV_MAGIC);
+
+ if (mcvlist->type != MVSTAT_MCV_TYPE_BASIC)
+ elog(ERROR, "invalid MCV type %d (expected %dd)",
+ mcvlist->type, MVSTAT_MCV_TYPE_BASIC);
+
+ nitems = mcvlist->nitems;
+ ndims = mcvlist->ndimensions;
+ itemsize = ITEM_SIZE(ndims);
+
+ Assert(nitems > 0);
+ Assert((ndims >= 2) && (ndims <= MVSTATS_MAX_DIMENSIONS));
+
+ /*
+ * What size do we expect with those parameters (it's incomplete,
+ * as we yet have to count the array sizes (from DimensionInfo
+ * records).
+ */
+ expected_size = offsetof(MCVListData,items) +
+ ndims * sizeof(DimensionInfo) +
+ (nitems * itemsize);
+
+ /* check that we have at least the DimensionInfo records */
+ if (VARSIZE_ANY_EXHDR(data) < expected_size)
+ elog(ERROR, "invalid MCV Size %ld (expected %ld)",
+ VARSIZE_ANY_EXHDR(data), expected_size);
+
+ info = (DimensionInfo*)(tmp);
+ tmp += ndims * sizeof(DimensionInfo);
+
+ /* account for the value arrays */
+ for (i = 0; i < ndims; i++)
+ expected_size += info[i].nbytes;
+
+ if (VARSIZE_ANY_EXHDR(data) != expected_size)
+ elog(ERROR, "invalid MCV Size %ld (expected %ld)",
+ VARSIZE_ANY_EXHDR(data), expected_size);
+
+ /* looks OK - not corrupted or something */
+
+ /* let's parse the value arrays */
+ values = (Datum**)palloc0(sizeof(Datum*) * ndims);
+
+ /*
+ * FIXME This uses pointers to the original data array (the types
+ * not passed by value), so when someone frees the memory,
+ * e.g. by doing something like this:
+ *
+ * bytea * data = ... fetch the data from catalog ...
+ * MCVList mcvlist = deserialize_mcv_list(data);
+ * pfree(data);
+ *
+ * then 'mcvlist' references the freed memory. This needs to
+ * copy the pieces.
+ */
+ for (i = 0; i < ndims; i++)
+ {
+ if (info[i].typbyval)
+ {
+ /* passed by value / Datum - simply reuse the array */
+ values[i] = (Datum*)tmp;
+ tmp += info[i].nbytes;
+ }
+ else if (info[i].typlen > 0)
+ {
+ /* pased by reference, but fixed length (name, tid, ...) */
+ values[i] = (Datum*)palloc0(sizeof(Datum) * info[i].nvalues);
+ for (j = 0; j < info[i].nvalues; j++)
+ {
+ /* just point into the array */
+ values[i][j] = PointerGetDatum(tmp);
+ tmp += info[i].typlen;
+ }
+ }
+ else if (info[i].typlen == -1)
+ {
+ /* varlena */
+ values[i] = (Datum*)palloc0(sizeof(Datum) * info[i].nvalues);
+ for (j = 0; j < info[i].nvalues; j++)
+ {
+ /* just point into the array */
+ values[i][j] = PointerGetDatum(tmp);
+ tmp += VARSIZE_ANY(tmp);
+ }
+ }
+ else if (info[i].typlen == -2)
+ {
+ /* cstring */
+ values[i] = (Datum*)palloc0(sizeof(Datum) * info[i].nvalues);
+ for (j = 0; j < info[i].nvalues; j++)
+ {
+ /* just point into the array */
+ values[i][j] = PointerGetDatum(tmp);
+ tmp += (strlen(tmp) + 1); /* don't forget the \0 */
+ }
+ }
+ }
+
+ /* allocate space for the MCV items */
+ mcvlist->items = (MCVItem*)palloc0(sizeof(MCVItem) * nitems);
+
+ for (i = 0; i < nitems; i++)
+ {
+ MCVItem item = (MCVItem)palloc0(sizeof(MCVItemData));
+
+ item->values = (Datum*)palloc0(sizeof(Datum)*ndims);
+ item->isnull = (bool*) palloc0(sizeof(bool) *ndims);
+
+ /* just point to the right place */
+ indexes = ITEM_INDEXES(tmp);
+
+ memcpy(item->isnull, ITEM_NULLS(tmp, ndims), sizeof(bool) * ndims);
+ memcpy(&item->frequency, ITEM_FREQUENCY(tmp, ndims), sizeof(double));
+
+ /* translate the values */
+ for (j = 0; j < ndims; j++)
+ if (! item->isnull[j])
+ item->values[j] = values[j][indexes[j]];
+
+ mcvlist->items[i] = item;
+
+ tmp += ITEM_SIZE(ndims);
+
+ Assert(tmp <= (char*)data + VARSIZE_ANY(data));
+ }
+
+ /* check that we processed all the data */
+ Assert(tmp == (char*)data + VARSIZE_ANY(data));
+
+ return mcvlist;
+}
+
+/*
+ * We need to pass the SortSupport to the comparator, but bsearch()
+ * has no 'context' parameter, so we use a global variable (ugly).
+ */
+static int
+bsearch_comparator(const void * a, const void * b)
+{
+ Assert(ssup_private != NULL);
+ return compare_scalars_simple(a, b, (void*)ssup_private);
+}
diff --git a/src/include/catalog/pg_mv_statistic.h b/src/include/catalog/pg_mv_statistic.h
index 81ec23b..c6e7d74 100644
--- a/src/include/catalog/pg_mv_statistic.h
+++ b/src/include/catalog/pg_mv_statistic.h
@@ -35,15 +35,21 @@ CATALOG(pg_mv_statistic,3381)
/* statistics requested to build */
bool deps_enabled; /* analyze dependencies? */
+ bool mcv_enabled; /* build MCV list? */
+
+ /* MCV size */
+ int32 mcv_max_items; /* max MCV items */
/* statistics that are available (if requested) */
bool deps_built; /* dependencies were built */
+ bool mcv_built; /* MCV list was built */
/* variable-length fields start here, but we allow direct access to stakeys */
int2vector stakeys; /* array of column keys */
#ifdef CATALOG_VARLEN
bytea stadeps; /* dependencies (serialized) */
+ bytea stamcv; /* MCV list (serialized) */
#endif
} FormData_pg_mv_statistic;
@@ -59,11 +65,15 @@ typedef FormData_pg_mv_statistic *Form_pg_mv_statistic;
* compiler constants for pg_attrdef
* ----------------
*/
-#define Natts_pg_mv_statistic 5
+#define Natts_pg_mv_statistic 9
#define Anum_pg_mv_statistic_starelid 1
#define Anum_pg_mv_statistic_deps_enabled 2
-#define Anum_pg_mv_statistic_deps_built 3
-#define Anum_pg_mv_statistic_stakeys 4
-#define Anum_pg_mv_statistic_stadeps 5
+#define Anum_pg_mv_statistic_mcv_enabled 3
+#define Anum_pg_mv_statistic_mcv_max_items 4
+#define Anum_pg_mv_statistic_deps_built 5
+#define Anum_pg_mv_statistic_mcv_built 6
+#define Anum_pg_mv_statistic_stakeys 7
+#define Anum_pg_mv_statistic_stadeps 8
+#define Anum_pg_mv_statistic_stamcv 9
#endif /* PG_MV_STATISTIC_H */
diff --git a/src/include/catalog/pg_proc.h b/src/include/catalog/pg_proc.h
index 2916f11..b2aa815 100644
--- a/src/include/catalog/pg_proc.h
+++ b/src/include/catalog/pg_proc.h
@@ -2716,6 +2716,8 @@ DATA(insert OID = 3377 ( pg_mv_stats_dependencies_info PGNSP PGUID 12 1 0 0
DESCR("multivariate stats: functional dependencies info");
DATA(insert OID = 3378 ( pg_mv_stats_dependencies_show PGNSP PGUID 12 1 0 0 0 f f f f t f i 1 0 25 "17" _null_ _null_ _null_ _null_ pg_mv_stats_dependencies_show _null_ _null_ _null_ ));
DESCR("multivariate stats: functional dependencies show");
+DATA(insert OID = 3376 ( pg_mv_stats_mcvlist_info PGNSP PGUID 12 1 0 0 0 f f f f t f i 1 0 25 "17" _null_ _null_ _null_ _null_ pg_mv_stats_mcvlist_info _null_ _null_ _null_ ));
+DESCR("multi-variate statistics: MCV list info");
DATA(insert OID = 1928 ( pg_stat_get_numscans PGNSP PGUID 12 1 0 0 0 f f f f t f s 1 0 20 "26" _null_ _null_ _null_ _null_ pg_stat_get_numscans _null_ _null_ _null_ ));
DESCR("statistics: number of scans done for table/index");
diff --git a/src/include/utils/mvstats.h b/src/include/utils/mvstats.h
index ec6764b..6ff29d6 100644
--- a/src/include/utils/mvstats.h
+++ b/src/include/utils/mvstats.h
@@ -25,9 +25,11 @@ typedef struct MVStatsData {
/* statistics requested in ALTER TABLE ... ADD STATISTICS */
bool deps_enabled; /* analyze functional dependencies */
+ bool mcv_enabled; /* analyze MCV lists */
/* available statistics (computed by ANALYZE) */
bool deps_built; /* functional dependencies available */
+ bool mcv_built; /* MCV list is already available */
} MVStatsData;
typedef struct MVStatsData *MVStats;
@@ -66,6 +68,47 @@ typedef MVDependenciesData* MVDependencies;
#define MVSTAT_DEPS_TYPE_BASIC 1 /* basic dependencies type */
/*
+ * Multivariate MCV (most-common value) lists
+ *
+ * A straight-forward extension of MCV items - i.e. a list (array) of
+ * combinations of attribute values, together with a frequency and
+ * null flags.
+ */
+typedef struct MCVItemData {
+ double frequency; /* frequency of this combination */
+ bool *isnull; /* lags of NULL values (up to 32 columns) */
+ Datum *values; /* variable-length (ndimensions) */
+} MCVItemData;
+
+typedef MCVItemData *MCVItem;
+
+/* multivariate MCV list - essentally an array of MCV items */
+typedef struct MCVListData {
+ uint32 magic; /* magic constant marker */
+ uint32 type; /* type of MCV list (BASIC) */
+ uint32 ndimensions; /* number of dimensions */
+ uint32 nitems; /* number of MCV items in the array */
+ MCVItem *items; /* array of MCV items */
+} MCVListData;
+
+typedef MCVListData *MCVList;
+
+/* used to flag stats serialized to bytea */
+#define MVSTAT_MCV_MAGIC 0xE1A651C2 /* marks serialized bytea */
+#define MVSTAT_MCV_TYPE_BASIC 1 /* basic MCV list type */
+
+/*
+ * Limits used for mcv_max_items option, i.e. we're always guaranteed
+ * to have space for at least MVSTAT_MCVLIST_MIN_ITEMS, and we cannot
+ * have more than MVSTAT_MCVLIST_MAX_ITEMS items.
+ *
+ * This is just a boundary for the 'max' threshold - the actual list
+ * may of course contain less items than MVSTAT_MCVLIST_MIN_ITEMS.
+ */
+#define MVSTAT_MCVLIST_MIN_ITEMS 128 /* min items in MCV list */
+#define MVSTAT_MCVLIST_MAX_ITEMS 8192 /* max items in MCV list */
+
+/*
* TODO Maybe fetching the histogram/MCV list separately is inefficient?
* Consider adding a single `fetch_stats` method, fetching all
* stats specified using flags (or something like that).
@@ -74,24 +117,39 @@ MVStats list_mv_stats(Oid relid, int *nstats, bool built_only);
bytea * fetch_mv_rules(Oid mvoid);
bytea * fetch_mv_dependencies(Oid mvoid);
+bytea * fetch_mv_mcvlist(Oid mvoid);
bytea * serialize_mv_dependencies(MVDependencies dependencies);
+bytea * serialize_mv_mcvlist(MCVList mcvlist, int2vector *attrs,
+ VacAttrStats **stats);
/* deserialization of stats (serialization is private to analyze) */
MVDependencies deserialize_mv_dependencies(bytea * data);
+MCVList deserialize_mv_mcvlist(bytea * data);
+
+/*
+ * Returns index of the attribute number within the vector (i.e. a
+ * dimension within the stats).
+ */
+int mv_get_index(AttrNumber varattno, int2vector * stakeys);
/* FIXME this probably belongs somewhere else (not to operations stats) */
extern Datum pg_mv_stats_dependencies_info(PG_FUNCTION_ARGS);
extern Datum pg_mv_stats_dependencies_show(PG_FUNCTION_ARGS);
+extern Datum pg_mv_stats_mcvlist_info(PG_FUNCTION_ARGS);
MVDependencies
-build_mv_dependencies(int numrows, HeapTuple *rows,
- int2vector *attrs,
- VacAttrStats **stats);
+build_mv_dependencies(int numrows, HeapTuple *rows, int2vector *attrs,
+ VacAttrStats **stats);
+
+MCVList
+build_mv_mcvlist(int numrows, HeapTuple *rows, int2vector *attrs,
+ VacAttrStats **stats, int *numrows_filtered);
void build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
- int natts, VacAttrStats **vacattrstats);
+ int natts, VacAttrStats **vacattrstats);
-void update_mv_stats(Oid relid, MVDependencies dependencies);
+void update_mv_stats(Oid relid, MVDependencies dependencies, MCVList mcvlist,
+ int2vector *attrs, VacAttrStats **stats);
#endif
diff --git a/src/test/regress/expected/mv_mcv.out b/src/test/regress/expected/mv_mcv.out
new file mode 100644
index 0000000..595cfbf
--- /dev/null
+++ b/src/test/regress/expected/mv_mcv.out
@@ -0,0 +1,210 @@
+-- data type passed by value
+CREATE TABLE mcv_list (
+ a INT,
+ b INT,
+ c INT
+);
+-- unknown column
+ALTER TABLE mcv_list ADD STATISTICS (mcv) ON (unknown_column);
+ERROR: column "unknown_column" referenced in statistics does not exist
+-- single column
+ALTER TABLE mcv_list ADD STATISTICS (mcv) ON (a);
+ERROR: multivariate stats require 2 or more columns
+-- single column, duplicated
+ALTER TABLE mcv_list ADD STATISTICS (mcv) ON (a, a);
+ERROR: duplicate column name in statistics definition
+-- two columns, one duplicated
+ALTER TABLE mcv_list ADD STATISTICS (mcv) ON (a, a, b);
+ERROR: duplicate column name in statistics definition
+-- unknown option
+ALTER TABLE mcv_list ADD STATISTICS (unknown_option) ON (a, b, c);
+ERROR: unrecognized STATISTICS option "unknown_option"
+-- missing MCV statistics
+ALTER TABLE mcv_list ADD STATISTICS (dependencies, max_mcv_items 200) ON (a, b, c);
+ERROR: option 'mcv' is required by other options(s)
+-- invalid mcv_max_items value / too low
+ALTER TABLE mcv_list ADD STATISTICS (mcv, max_mcv_items 10) ON (a, b, c);
+ERROR: max number of MCV items must be at least 128
+-- invalid mcv_max_items value / too high
+ALTER TABLE mcv_list ADD STATISTICS (mcv, max_mcv_items 10000) ON (a, b, c);
+ERROR: max number of MCV items is 8192
+-- correct command
+ALTER TABLE mcv_list ADD STATISTICS (mcv) ON (a, b, c);
+-- random data
+INSERT INTO mcv_list
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+ANALYZE mcv_list;
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+ mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info
+-------------+-----------+--------------------------
+ t | f |
+(1 row)
+
+TRUNCATE mcv_list;
+-- a => b, a => c, b => c
+INSERT INTO mcv_list
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE mcv_list;
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+ mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info
+-------------+-----------+--------------------------
+ t | t | nitems=1000
+(1 row)
+
+TRUNCATE mcv_list;
+-- a => b, a => c
+INSERT INTO mcv_list
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE mcv_list;
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+ mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info
+-------------+-----------+--------------------------
+ t | t | nitems=1000
+(1 row)
+
+TRUNCATE mcv_list;
+-- check explain (expect bitmap index scan, not plain index scan)
+INSERT INTO mcv_list
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX mcv_idx ON mcv_list (a, b);
+ANALYZE mcv_list;
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+ mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info
+-------------+-----------+--------------------------
+ t | t | nitems=100
+(1 row)
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mcv_list WHERE a = 10 AND b = 5;
+ QUERY PLAN
+--------------------------------------------
+ Bitmap Heap Scan on mcv_list
+ Recheck Cond: ((a = 10) AND (b = 5))
+ -> Bitmap Index Scan on mcv_idx
+ Index Cond: ((a = 10) AND (b = 5))
+(4 rows)
+
+DELETE FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+DROP TABLE mcv_list;
+-- varlena type (text)
+CREATE TABLE mcv_list (
+ a TEXT,
+ b TEXT,
+ c TEXT
+);
+ALTER TABLE mcv_list ADD STATISTICS (mcv) ON (a, b, c);
+-- random data
+INSERT INTO mcv_list
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+ANALYZE mcv_list;
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+ mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info
+-------------+-----------+--------------------------
+ t | f |
+(1 row)
+
+TRUNCATE mcv_list;
+-- a => b, a => c, b => c
+INSERT INTO mcv_list
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE mcv_list;
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+ mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info
+-------------+-----------+--------------------------
+ t | t | nitems=1000
+(1 row)
+
+TRUNCATE mcv_list;
+-- a => b, a => c
+INSERT INTO mcv_list
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE mcv_list;
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+ mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info
+-------------+-----------+--------------------------
+ t | t | nitems=1000
+(1 row)
+
+TRUNCATE mcv_list;
+-- check explain (expect bitmap index scan, not plain index scan)
+INSERT INTO mcv_list
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX mcv_idx ON mcv_list (a, b);
+ANALYZE mcv_list;
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+ mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info
+-------------+-----------+--------------------------
+ t | t | nitems=100
+(1 row)
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mcv_list WHERE a = '10' AND b = '5';
+ QUERY PLAN
+------------------------------------------------------------
+ Bitmap Heap Scan on mcv_list
+ Recheck Cond: ((a = '10'::text) AND (b = '5'::text))
+ -> Bitmap Index Scan on mcv_idx
+ Index Cond: ((a = '10'::text) AND (b = '5'::text))
+(4 rows)
+
+TRUNCATE mcv_list;
+-- check explain (expect bitmap index scan, not plain index scan) with NULLs
+INSERT INTO mcv_list
+ SELECT
+ (CASE WHEN i/10000 = 0 THEN NULL ELSE i/10000 END),
+ (CASE WHEN i/20000 = 0 THEN NULL ELSE i/20000 END),
+ (CASE WHEN i/40000 = 0 THEN NULL ELSE i/40000 END)
+ FROM generate_series(1,1000000) s(i);
+ANALYZE mcv_list;
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+ mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info
+-------------+-----------+--------------------------
+ t | t | nitems=100
+(1 row)
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mcv_list WHERE a IS NULL AND b IS NULL;
+ QUERY PLAN
+---------------------------------------------------
+ Bitmap Heap Scan on mcv_list
+ Recheck Cond: ((a IS NULL) AND (b IS NULL))
+ -> Bitmap Index Scan on mcv_idx
+ Index Cond: ((a IS NULL) AND (b IS NULL))
+(4 rows)
+
+DELETE FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+DROP TABLE mcv_list;
+-- NULL values (mix of int and text columns)
+CREATE TABLE mcv_list (
+ a INT,
+ b TEXT,
+ c INT,
+ d TEXT
+);
+ALTER TABLE mcv_list ADD STATISTICS (mcv) ON (a, b, c, d);
+INSERT INTO mcv_list
+ SELECT
+ mod(i, 100),
+ (CASE WHEN mod(i, 200) = 0 THEN NULL ELSE mod(i,200) END),
+ mod(i, 400),
+ (CASE WHEN mod(i, 300) = 0 THEN NULL ELSE mod(i,600) END)
+ FROM generate_series(1,10000) s(i);
+ANALYZE mcv_list;
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+ mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info
+-------------+-----------+--------------------------
+ t | t | nitems=1200
+(1 row)
+
+DELETE FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+DROP TABLE mcv_list;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index f0117ca..6d9ab2f 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1357,7 +1357,9 @@ pg_mv_stats| SELECT n.nspname AS schemaname,
c.relname AS tablename,
s.stakeys AS attnums,
length(s.stadeps) AS depsbytes,
- pg_mv_stats_dependencies_info(s.stadeps) AS depsinfo
+ pg_mv_stats_dependencies_info(s.stadeps) AS depsinfo,
+ length(s.stamcv) AS mcvbytes,
+ pg_mv_stats_mcvlist_info(s.stamcv) AS mcvinfo
FROM ((pg_mv_statistic s
JOIN pg_class c ON ((c.oid = s.starelid)))
LEFT JOIN pg_namespace n ON ((n.oid = c.relnamespace)));
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index 00c6ddf..63727a4 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -111,4 +111,4 @@ test: event_trigger
test: stats
# run tests of multivariate stats
-test: mv_dependencies
+test: mv_dependencies mv_mcv
diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule
index b818be9..5b07b3b 100644
--- a/src/test/regress/serial_schedule
+++ b/src/test/regress/serial_schedule
@@ -154,3 +154,4 @@ test: xml
test: event_trigger
test: stats
test: mv_dependencies
+test: mv_mcv
diff --git a/src/test/regress/sql/mv_mcv.sql b/src/test/regress/sql/mv_mcv.sql
new file mode 100644
index 0000000..410b52d
--- /dev/null
+++ b/src/test/regress/sql/mv_mcv.sql
@@ -0,0 +1,181 @@
+-- data type passed by value
+CREATE TABLE mcv_list (
+ a INT,
+ b INT,
+ c INT
+);
+
+-- unknown column
+ALTER TABLE mcv_list ADD STATISTICS (mcv) ON (unknown_column);
+
+-- single column
+ALTER TABLE mcv_list ADD STATISTICS (mcv) ON (a);
+
+-- single column, duplicated
+ALTER TABLE mcv_list ADD STATISTICS (mcv) ON (a, a);
+
+-- two columns, one duplicated
+ALTER TABLE mcv_list ADD STATISTICS (mcv) ON (a, a, b);
+
+-- unknown option
+ALTER TABLE mcv_list ADD STATISTICS (unknown_option) ON (a, b, c);
+
+-- missing MCV statistics
+ALTER TABLE mcv_list ADD STATISTICS (dependencies, max_mcv_items 200) ON (a, b, c);
+
+-- invalid mcv_max_items value / too low
+ALTER TABLE mcv_list ADD STATISTICS (mcv, max_mcv_items 10) ON (a, b, c);
+
+-- invalid mcv_max_items value / too high
+ALTER TABLE mcv_list ADD STATISTICS (mcv, max_mcv_items 10000) ON (a, b, c);
+
+-- correct command
+ALTER TABLE mcv_list ADD STATISTICS (mcv) ON (a, b, c);
+
+-- random data
+INSERT INTO mcv_list
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+
+ANALYZE mcv_list;
+
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+
+TRUNCATE mcv_list;
+
+-- a => b, a => c, b => c
+INSERT INTO mcv_list
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+
+ANALYZE mcv_list;
+
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+
+TRUNCATE mcv_list;
+
+-- a => b, a => c
+INSERT INTO mcv_list
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+
+ANALYZE mcv_list;
+
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+
+TRUNCATE mcv_list;
+
+-- check explain (expect bitmap index scan, not plain index scan)
+INSERT INTO mcv_list
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX mcv_idx ON mcv_list (a, b);
+
+ANALYZE mcv_list;
+
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mcv_list WHERE a = 10 AND b = 5;
+
+DELETE FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+DROP TABLE mcv_list;
+
+-- varlena type (text)
+CREATE TABLE mcv_list (
+ a TEXT,
+ b TEXT,
+ c TEXT
+);
+
+ALTER TABLE mcv_list ADD STATISTICS (mcv) ON (a, b, c);
+
+-- random data
+INSERT INTO mcv_list
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+
+ANALYZE mcv_list;
+
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+
+TRUNCATE mcv_list;
+
+-- a => b, a => c, b => c
+INSERT INTO mcv_list
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+
+ANALYZE mcv_list;
+
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+
+TRUNCATE mcv_list;
+
+-- a => b, a => c
+INSERT INTO mcv_list
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE mcv_list;
+
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+
+TRUNCATE mcv_list;
+
+-- check explain (expect bitmap index scan, not plain index scan)
+INSERT INTO mcv_list
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX mcv_idx ON mcv_list (a, b);
+ANALYZE mcv_list;
+
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mcv_list WHERE a = '10' AND b = '5';
+
+TRUNCATE mcv_list;
+
+-- check explain (expect bitmap index scan, not plain index scan) with NULLs
+INSERT INTO mcv_list
+ SELECT
+ (CASE WHEN i/10000 = 0 THEN NULL ELSE i/10000 END),
+ (CASE WHEN i/20000 = 0 THEN NULL ELSE i/20000 END),
+ (CASE WHEN i/40000 = 0 THEN NULL ELSE i/40000 END)
+ FROM generate_series(1,1000000) s(i);
+ANALYZE mcv_list;
+
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mcv_list WHERE a IS NULL AND b IS NULL;
+
+DELETE FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+DROP TABLE mcv_list;
+
+-- NULL values (mix of int and text columns)
+CREATE TABLE mcv_list (
+ a INT,
+ b TEXT,
+ c INT,
+ d TEXT
+);
+
+ALTER TABLE mcv_list ADD STATISTICS (mcv) ON (a, b, c, d);
+
+INSERT INTO mcv_list
+ SELECT
+ mod(i, 100),
+ (CASE WHEN mod(i, 200) = 0 THEN NULL ELSE mod(i,200) END),
+ mod(i, 400),
+ (CASE WHEN mod(i, 300) = 0 THEN NULL ELSE mod(i,600) END)
+ FROM generate_series(1,10000) s(i);
+
+ANALYZE mcv_list;
+
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+
+DELETE FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+DROP TABLE mcv_list;
--
2.0.5
[text/x-diff] 0004-multivariate-histograms.patch (117.4K, ../../5519E9B2.5080105@2ndquadrant.com/5-0004-multivariate-histograms.patch)
download | inline diff:
From 166a13ed6152ebc0e384c53f765946ae8be5193f Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Sun, 11 Jan 2015 20:18:24 +0100
Subject: [PATCH 4/5] multivariate histograms
- extends the pg_mv_statistic catalog (add 'hist' fields)
- building the histograms during ANALYZE
- simple estimation while planning the queries
Includes regression tests mostly equal to those for functional
dependencies / MCV lists.
---
src/backend/catalog/system_views.sql | 4 +-
src/backend/commands/tablecmds.c | 67 +-
src/backend/optimizer/path/clausesel.c | 549 ++++++++-
src/backend/utils/mvstats/Makefile | 2 +-
src/backend/utils/mvstats/common.c | 41 +-
src/backend/utils/mvstats/histogram.c | 1800 ++++++++++++++++++++++++++++
src/backend/utils/mvstats/mcv.c | 1 +
src/include/catalog/pg_mv_statistic.h | 24 +-
src/include/catalog/pg_proc.h | 2 +
src/include/utils/mvstats.h | 76 +-
src/test/regress/expected/mv_histogram.out | 210 ++++
src/test/regress/expected/rules.out | 4 +-
src/test/regress/parallel_schedule | 2 +-
src/test/regress/serial_schedule | 1 +
src/test/regress/sql/mv_histogram.sql | 179 +++
15 files changed, 2924 insertions(+), 38 deletions(-)
create mode 100644 src/backend/utils/mvstats/histogram.c
create mode 100644 src/test/regress/expected/mv_histogram.out
create mode 100644 src/test/regress/sql/mv_histogram.sql
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 4538e63..87086f9 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -158,7 +158,9 @@ CREATE VIEW pg_mv_stats AS
length(S.stadeps) as depsbytes,
pg_mv_stats_dependencies_info(S.stadeps) as depsinfo,
length(S.stamcv) AS mcvbytes,
- pg_mv_stats_mcvlist_info(S.stamcv) AS mcvinfo
+ pg_mv_stats_mcvlist_info(S.stamcv) AS mcvinfo,
+ length(S.stahist) AS histbytes,
+ pg_mv_stats_histogram_info(S.stahist) AS histinfo
FROM (pg_mv_statistic S JOIN pg_class C ON (C.oid = S.starelid))
LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace);
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index fae0fc7..6b01660 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -11876,15 +11876,19 @@ static int compare_int16(const void *a, const void *b)
* The code is an unholy mix of pieces that really belong to other parts
* of the source tree.
*
- * FIXME Check that the types are pass-by-value and support sort,
- * although maybe we can live without the sort (and only build
- * MCV list / association rules).
- *
- * FIXME This should probably check for duplicate stats (i.e. same
- * keys, same options). Although maybe it's useful to have
- * multiple stats on the same columns with different options
- * (say, a detailed MCV-only stats for some queries, histogram
- * for others, etc.)
+ * TODO Check that the types support sort, although maybe we can live
+ * without it (and only build MCV list / association rules).
+ *
+ * TODO This should probably check for duplicate stats (i.e. same
+ * keys, same options). Although maybe it's useful to have
+ * multiple stats on the same columns with different options
+ * (say, a detailed MCV-only stats for some queries, histogram
+ * for others, etc.)
+ *
+ * TODO It might be useful to have ALTER TABLE DROP STATISTICS too, but
+ * it's tricky because there may be multiple kinds of stats for the
+ * same list of columns, with different options (e.g. one just MCV
+ * list, another with histogram, etc.).
*/
static void ATExecAddStatistics(AlteredTableInfo *tab, Relation rel,
StatisticsDef *def, LOCKMODE lockmode)
@@ -11902,12 +11906,15 @@ static void ATExecAddStatistics(AlteredTableInfo *tab, Relation rel,
/* by default build everything */
bool build_dependencies = false,
- build_mcv = false;
+ build_mcv = false,
+ build_histogram = false;
- int32 max_mcv_items = -1;
+ int32 max_buckets = -1,
+ max_mcv_items = -1;
/* options required because of other options */
- bool require_mcv = false;
+ bool require_mcv = false,
+ require_histogram = false;
Assert(IsA(def, StatisticsDef));
@@ -11985,6 +11992,29 @@ static void ATExecAddStatistics(AlteredTableInfo *tab, Relation rel,
MVSTAT_MCVLIST_MAX_ITEMS)));
}
+ else if (strcmp(opt->defname, "histogram") == 0)
+ build_histogram = defGetBoolean(opt);
+ else if (strcmp(opt->defname, "max_buckets") == 0)
+ {
+ max_buckets = defGetInt32(opt);
+
+ /* this option requires 'histogram' to be enabled */
+ require_histogram = true;
+
+ /* sanity check */
+ if (max_buckets < MVSTAT_HIST_MIN_BUCKETS)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("minimum number of buckets is %d",
+ MVSTAT_HIST_MIN_BUCKETS)));
+
+ else if (max_buckets > MVSTAT_HIST_MAX_BUCKETS)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("minimum number of buckets is %d",
+ MVSTAT_HIST_MAX_BUCKETS)));
+
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -11993,10 +12023,10 @@ static void ATExecAddStatistics(AlteredTableInfo *tab, Relation rel,
}
/* check that at least some statistics were requested */
- if (! (build_dependencies || build_mcv))
+ if (! (build_dependencies || build_mcv || build_histogram))
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("no statistics type (dependencies, mcv) was requested")));
+ errmsg("no statistics type (dependencies, mcv, histogram) was requested")));
/* now do some checking of the options */
if (require_mcv && (! build_mcv))
@@ -12004,6 +12034,11 @@ static void ATExecAddStatistics(AlteredTableInfo *tab, Relation rel,
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("option 'mcv' is required by other options(s)")));
+ if (require_histogram && (! build_histogram))
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("option 'histogram' is required by other options(s)")));
+
/* sort the attnums and build int2vector */
qsort(attnums, numcols, sizeof(int16), compare_int16);
stakeys = buildint2vector(attnums, numcols);
@@ -12021,10 +12056,14 @@ static void ATExecAddStatistics(AlteredTableInfo *tab, Relation rel,
values[Anum_pg_mv_statistic_deps_enabled -1] = BoolGetDatum(build_dependencies);
values[Anum_pg_mv_statistic_mcv_enabled -1] = BoolGetDatum(build_mcv);
+ values[Anum_pg_mv_statistic_hist_enabled -1] = BoolGetDatum(build_histogram);
+
values[Anum_pg_mv_statistic_mcv_max_items -1] = Int32GetDatum(max_mcv_items);
+ values[Anum_pg_mv_statistic_hist_max_buckets -1] = Int32GetDatum(max_buckets);
nulls[Anum_pg_mv_statistic_stadeps -1] = true;
nulls[Anum_pg_mv_statistic_stamcv -1] = true;
+ nulls[Anum_pg_mv_statistic_stahist -1] = true;
/* insert the tuple into pg_mv_statistic */
mvstatrel = heap_open(MvStatisticRelationId, RowExclusiveLock);
diff --git a/src/backend/optimizer/path/clausesel.c b/src/backend/optimizer/path/clausesel.c
index d24aedf..ea4d588 100644
--- a/src/backend/optimizer/path/clausesel.c
+++ b/src/backend/optimizer/path/clausesel.c
@@ -53,6 +53,7 @@ static void addRangeClause(RangeQueryClause **rqlist, Node *clause,
#define MV_CLAUSE_TYPE_FDEP 0x01
#define MV_CLAUSE_TYPE_MCV 0x02
+#define MV_CLAUSE_TYPE_HIST 0x04
static bool clause_is_mv_compatible(PlannerInfo *root, Node *clause, Oid varRelid,
Oid *relid, Bitmapset **attnums, SpecialJoinInfo *sjinfo,
@@ -77,6 +78,8 @@ static Selectivity clauselist_mv_selectivity(PlannerInfo *root,
static Selectivity clauselist_mv_selectivity_mcvlist(PlannerInfo *root,
List *clauses, MVStats mvstats,
bool *fullmatch, Selectivity *lowsel);
+static Selectivity clauselist_mv_selectivity_histogram(PlannerInfo *root,
+ List *clauses, MVStats mvstats);
static int update_match_bitmap_mcvlist(PlannerInfo *root, List *clauses,
int2vector *stakeys, MCVList mcvlist,
@@ -84,6 +87,11 @@ static int update_match_bitmap_mcvlist(PlannerInfo *root, List *clauses,
Selectivity *lowsel, bool *fullmatch,
bool is_or);
+static int update_match_bitmap_histogram(PlannerInfo *root, List *clauses,
+ int2vector *stakeys, MVHistogram mvhist,
+ int nmatches, char * matches,
+ bool is_or);
+
/* used for merging bitmaps - AND (min), OR (max) */
#define MAX(x, y) (((x) > (y)) ? (x) : (y))
#define MIN(x, y) (((x) < (y)) ? (x) : (y))
@@ -266,7 +274,7 @@ clauselist_selectivity(PlannerInfo *root,
* From now on we're only interested in MCV-compatible clauses.
*/
mvattnums = collect_mv_attnums(root, clauses, varRelid, &relid, sjinfo,
- MV_CLAUSE_TYPE_MCV);
+ (MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST));
/*
* If there still are at least two columns, we'll try to select
@@ -292,7 +300,7 @@ clauselist_selectivity(PlannerInfo *root,
/* split the clauselist into regular and mv-clauses */
clauses = clauselist_mv_split(root, sjinfo, clauses,
varRelid, &mvclauses, mvstat,
- MV_CLAUSE_TYPE_MCV);
+ (MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST));
/* we've chosen the histogram to match the clauses */
Assert(mvclauses != NIL);
@@ -1146,6 +1154,7 @@ static Selectivity
clauselist_mv_selectivity(PlannerInfo *root, List *clauses, MVStats mvstats)
{
bool fullmatch = false;
+ Selectivity s1 = 0.0, s2 = 0.0;
/*
* Lowest frequency in the MCV list (may be used as an upper bound
@@ -1159,9 +1168,24 @@ clauselist_mv_selectivity(PlannerInfo *root, List *clauses, MVStats mvstats)
* MCV/histogram evaluation).
*/
- /* Evaluate the MCV selectivity */
- return clauselist_mv_selectivity_mcvlist(root, clauses, mvstats,
+ /* Evaluate the MCV first. */
+ s1 = clauselist_mv_selectivity_mcvlist(root, clauses, mvstats,
&fullmatch, &mcv_low);
+
+ /*
+ * If we got a full equality match on the MCV list, we're done (and
+ * the estimate is pretty good).
+ */
+ if (fullmatch && (s1 > 0.0))
+ return s1;
+
+ /* FIXME if (fullmatch) without matching MCV item, use the mcv_low
+ * selectivity as upper bound */
+
+ s2 = clauselist_mv_selectivity_histogram(root, clauses, mvstats);
+
+ /* TODO clamp to <= 1.0 (or more strictly, when possible) */
+ return s1 + s2;
}
/*
@@ -1461,7 +1485,6 @@ clause_is_mv_compatible(PlannerInfo *root, Node *clause, Oid varRelid,
bool ok;
/* is it 'variable op constant' ? */
-
ok = (bms_membership(clause_relids) == BMS_SINGLETON) &&
(is_pseudo_constant_clause_relids(lsecond(expr->args),
right_relids) ||
@@ -1515,10 +1538,10 @@ clause_is_mv_compatible(PlannerInfo *root, Node *clause, Oid varRelid,
case F_SCALARLTSEL:
case F_SCALARGTSEL:
/* not compatible with functional dependencies */
- if (types & MV_CLAUSE_TYPE_MCV)
+ if (types & (MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST))
{
*attnums = bms_add_member(*attnums, var->varattno);
- return (types & MV_CLAUSE_TYPE_MCV);
+ return (types & (MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST));
}
return false;
@@ -2435,3 +2458,515 @@ update_match_bitmap_mcvlist(PlannerInfo *root, List *clauses,
return nmatches;
}
+
+/*
+ * Estimate selectivity of clauses using a histogram.
+ *
+ * If there's no histogram for the stats, the function returns 0.0.
+ *
+ * The general idea of this method is similar to how MCV lists are
+ * processed, except that this introduces the concept of a partial
+ * match (MCV only works with full match / mismatch).
+ *
+ * The algorithm works like this:
+ *
+ * 1) mark all buckets as 'full match'
+ * 2) walk through all the clauses
+ * 3) for a particular clause, walk through all the buckets
+ * 4) skip buckets that are already 'no match'
+ * 5) check clause for buckets that still match (at least partially)
+ * 6) sum frequencies for buckets to get selectivity
+ *
+ * Unlike MCV lists, histograms have a concept of a partial match.
+ *
+ * TODO This only handles AND-ed clauses, but it might work for OR-ed
+ * lists too - it just needs to reverse the logic a bit. I.e. start
+ * with 'no match' for all buckets, and increase the match level
+ * for the clauses (and skip buckets that are 'full match').
+ *
+ * TODO This might use a similar shortcut to MCV lists - count buckets
+ * marked as partial/full match, and terminate once this drop to 0.
+ * Not sure if it's really worth it - for MCV lists a situation like
+ * this is not uncommon, but for histograms it's not that clear.
+ */
+static Selectivity
+clauselist_mv_selectivity_histogram(PlannerInfo *root, List *clauses,
+ MVStats mvstats)
+{
+ int i;
+ Selectivity s = 0.0;
+ int nmatches = 0;
+ char *matches = NULL;
+ MVHistogram mvhist = NULL;
+
+ /* there's no histogram */
+ if (! mvstats->hist_built)
+ return 0.0;
+
+ /* There may be no histogram in the stats (check hist_built flag) */
+ mvhist = deserialize_mv_histogram(fetch_mv_histogram(mvstats->mvoid));
+
+ Assert (mvhist != NULL);
+ Assert (clauses != NIL);
+ Assert (list_length(clauses) >= 2);
+
+ /*
+ * Bitmap of bucket matches (mismatch, partial, full). by default
+ * all buckets fully match (and we'll eliminate them).
+ */
+ matches = palloc0(sizeof(char) * mvhist->nbuckets);
+ memset(matches, MVSTATS_MATCH_FULL, sizeof(char)*mvhist->nbuckets);
+
+ nmatches = mvhist->nbuckets;
+
+ /* build the match bitmap */
+ update_match_bitmap_histogram(root, clauses,
+ mvstats->stakeys, mvhist,
+ nmatches, matches, false);
+
+ /* now, walk through the buckets and sum the selectivities */
+ for (i = 0; i < mvhist->nbuckets; i++)
+ {
+ if (matches[i] == MVSTATS_MATCH_FULL)
+ s += mvhist->buckets[i]->ntuples;
+ else if (matches[i] == MVSTATS_MATCH_PARTIAL)
+ s += 0.5 * mvhist->buckets[i]->ntuples;
+ }
+
+ /* release the allocated bitmap and deserialized histogram */
+ pfree(matches);
+ pfree(mvhist);
+
+ return s;
+}
+
+/*
+ * Evaluate clauses using the histogram, and update the match bitmap.
+ *
+ * The bitmap may be already partially set, so this is really a way to
+ * combine results of several clause lists - either when computing
+ * conditional probability P(A|B) or a combination of AND/OR clauses.
+ *
+ * Note: This is not a simple bitmap in the sense that there are more
+ * than two possible values for each item - no match, partial
+ * match and full match. So we need 2 bits per item.
+ *
+ * TODO This works with 'bitmap' where each item is represented as a
+ * char, which is slightly wasteful. Instead, we could use a bitmap
+ * with 2 bits per item, reducing the size to ~1/4. By using values
+ * 0, 1 and 3 (instead of 0, 1 and 2), the operations (merging etc.)
+ * might be performed just like for simple bitmap by using & and |,
+ * which might be faster than min/max.
+ */
+static int
+update_match_bitmap_histogram(PlannerInfo *root, List *clauses,
+ int2vector *stakeys, MVHistogram mvhist,
+ int nmatches, char * matches,
+ bool is_or)
+{
+ int i;
+ ListCell * l;
+
+ Assert (mvhist != NULL);
+ Assert(mvhist->nbuckets > 0);
+ Assert(nmatches >= 0);
+ Assert(nmatches <= mvhist->nbuckets);
+
+ Assert (clauses != NIL);
+ Assert(list_length(clauses) >= 1);
+
+ /* loop through the clauses and do the estimation */
+ foreach (l, clauses)
+ {
+ Node * clause = (Node*)lfirst(l);
+
+ /* if it's a RestrictInfo, then extract the clause */
+ if (IsA(clause, RestrictInfo))
+ clause = (Node*)((RestrictInfo*)clause)->clause;
+
+ /* it's either OpClause, or NullTest */
+ if (is_opclause(clause))
+ {
+ OpExpr * expr = (OpExpr*)clause;
+ bool varonleft = true;
+ bool ok;
+
+ FmgrInfo opproc; /* operator */
+ fmgr_info(get_opcode(expr->opno), &opproc);
+
+ ok = (NumRelids(clause) == 1) &&
+ (is_pseudo_constant_clause(lsecond(expr->args)) ||
+ (varonleft = false,
+ is_pseudo_constant_clause(linitial(expr->args))));
+
+ if (ok)
+ {
+ FmgrInfo ltproc;
+ Var * var = (varonleft) ? linitial(expr->args) : lsecond(expr->args);
+ Const * cst = (varonleft) ? lsecond(expr->args) : linitial(expr->args);
+ bool isgt = (! varonleft);
+
+ /*
+ * TODO Fetch only when really needed (probably for equality only)
+ *
+ * TODO Technically either lt/gt is sufficient.
+ *
+ * FIXME The code in analyze.c creates histograms only for types
+ * with enough ordering (by calling get_sort_group_operators).
+ * Is this the same assumption, i.e. are we certain that we
+ * get the ltproc/gtproc every time we ask? Or are there types
+ * where get_sort_group_operators returns ltopr and here we
+ * get nothing?
+ */
+ TypeCacheEntry *typecache
+ = lookup_type_cache(var->vartype, TYPECACHE_EQ_OPR | TYPECACHE_LT_OPR
+ | TYPECACHE_GT_OPR);
+
+ /* lookup dimension for the attribute */
+ int idx = mv_get_index(var->varattno, stakeys);
+
+ fmgr_info(get_opcode(typecache->lt_opr), <proc);
+
+ /*
+ * Check this for all buckets that still have "true" in the bitmap
+ *
+ * We already know the clauses use suitable operators (because that's
+ * how we filtered them).
+ */
+ for (i = 0; i < mvhist->nbuckets; i++)
+ {
+ bool tmp;
+ MVBucket bucket = mvhist->buckets[i];
+
+ /*
+ * For AND-lists, we can also mark NULL buckets as 'no match'
+ * (and then skip them). For OR-lists this is not possible.
+ */
+ if ((! is_or) && bucket->nullsonly[idx])
+ matches[i] = MVSTATS_MATCH_NONE;
+
+ /*
+ * Skip buckets that were already eliminated - this is impotant
+ * considering how we update the info (we only lower the match).
+ * We can't really do anything about the MATCH_PARTIAL buckets.
+ */
+ if ((! is_or) && (matches[i] == MVSTATS_MATCH_NONE))
+ continue;
+ else if (is_or && (matches[i] == MVSTATS_MATCH_FULL))
+ continue;
+
+ /*
+ * TODO Maybe it's possible to add here a similar optimization
+ * as for the MCV lists:
+ *
+ * (nmatches == 0) && AND-list => all eliminated (FALSE)
+ * (nmatches == N) && OR-list => all eliminated (TRUE)
+ *
+ * But it's more complex because of the partial matches.
+ */
+
+ /*
+ * If it's not a "<" or ">" or "=" operator, just ignore the
+ * clause. Otherwise note the relid and attnum for the variable.
+ *
+ * TODO I'm really unsure the handling of 'isgt' flag (that is, clauses
+ * with reverse order of variable/constant) is correct. I wouldn't
+ * be surprised if there was some mixup. Using the lt/gt operators
+ * instead of messing with the opproc could make it simpler.
+ * It would however be using a different operator than the query,
+ * although it's not any shadier than using the selectivity function
+ * as is done currently.
+ *
+ * FIXME Once the min/max values are deduplicated, we can easily minimize
+ * the number of calls to the comparator (assuming we keep the
+ * deduplicated structure). See the note on compression at MVBucket
+ * serialize/deserialize methods.
+ */
+ switch (get_oprrest(expr->opno))
+ {
+ case F_SCALARLTSEL: /* column < constant */
+
+ if (! isgt) /* (var < const) */
+ {
+ /*
+ * First check whether the constant is below the lower boundary (in that
+ * case we can skip the bucket, because there's no overlap).
+ */
+ tmp = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ cst->constvalue,
+ bucket->min[idx]));
+ if (tmp)
+ {
+ /* no match */
+ UPDATE_RESULT(matches[i], MVSTATS_MATCH_NONE, is_or);
+ continue;
+ }
+
+ /*
+ * Now check whether the upper boundary is below the constant (in that
+ * case it's a partial match).
+ */
+ tmp = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ cst->constvalue,
+ bucket->max[idx]));
+
+ if (tmp)
+ /* partial match */
+ UPDATE_RESULT(matches[i], MVSTATS_MATCH_PARTIAL, is_or);
+
+ }
+ else /* (const < var) */
+ {
+ /*
+ * First check whether the constant is above the upper boundary (in that
+ * case we can skip the bucket, because there's no overlap).
+ */
+ tmp = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ bucket->max[idx],
+ cst->constvalue));
+ if (tmp)
+ {
+ /* no match */
+ UPDATE_RESULT(matches[i], MVSTATS_MATCH_NONE, is_or);
+ continue;
+ }
+
+ /*
+ * Now check whether the lower boundary is below the constant (in that
+ * case it's a partial match).
+ */
+ tmp = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ bucket->min[idx],
+ cst->constvalue));
+
+ if (tmp)
+ /* partial match */
+ UPDATE_RESULT(matches[i], MVSTATS_MATCH_PARTIAL, is_or);
+ }
+ break;
+
+ case F_SCALARGTSEL: /* column > constant */
+
+ if (! isgt) /* (var > const) */
+ {
+ /*
+ * First check whether the constant is above the upper boundary (in that
+ * case we can skip the bucket, because there's no overlap).
+ */
+ tmp = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ cst->constvalue,
+ bucket->max[idx]));
+ if (tmp)
+ {
+ /* no match */
+ UPDATE_RESULT(matches[i], MVSTATS_MATCH_NONE, is_or);
+ continue;
+ }
+
+ /*
+ * Now check whether the lower boundary is below the constant (in that
+ * case it's a partial match).
+ */
+ tmp = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ cst->constvalue,
+ bucket->min[idx]));
+
+ if (tmp)
+ /* partial match */
+ UPDATE_RESULT(matches[i], MVSTATS_MATCH_PARTIAL, is_or);
+ }
+ else /* (const > var) */
+ {
+ /*
+ * First check whether the constant is below the lower boundary (in
+ * that case we can skip the bucket, because there's no overlap).
+ */
+ tmp = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ bucket->min[idx],
+ cst->constvalue));
+ if (tmp)
+ {
+ /* no match */
+ UPDATE_RESULT(matches[i], MVSTATS_MATCH_NONE, is_or);
+ continue;
+ }
+
+ /*
+ * Now check whether the upper boundary is below the constant (in that
+ * case it's a partial match).
+ */
+ tmp = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ bucket->max[idx],
+ cst->constvalue));
+
+ if (tmp)
+ /* partial match */
+ UPDATE_RESULT(matches[i], MVSTATS_MATCH_PARTIAL, is_or);
+ }
+ break;
+
+ case F_EQSEL:
+
+ /*
+ * We only check whether the value is within the bucket, using the lt/gt
+ * operators fetched from type cache.
+ *
+ * TODO We'll use the default 50% estimate, but that's probably way off
+ * if there are multiple distinct values. Consider tweaking this a
+ * somehow, e.g. using only a part inversely proportional to the
+ * estimated number of distinct values in the bucket.
+ *
+ * TODO This does not handle inclusion flags at the moment, thus counting
+ * some buckets twice (when hitting the boundary).
+ *
+ * TODO Optimization is that if max[i] == min[i], it's effectively a MCV
+ * item and we can count the whole bucket as a complete match (thus
+ * using 100% bucket selectivity and not just 50%).
+ *
+ * TODO Technically some buckets may "degenerate" into single-value
+ * buckets (not necessarily for all the dimensions) - maybe this
+ * is better than keeping a separate MCV list (multi-dimensional).
+ * Update: Actually, that's unlikely to be better than a separate
+ * MCV list for two reasons - first, it requires ~2x the space
+ * (because of storing lower/upper boundaries) and second because
+ * the buckets are ranges - depending on the partitioning algorithm
+ * it may not even degenerate into (min=max) bucket. For example the
+ * the current partitioning algorithm never does that.
+ */
+ tmp = DatumGetBool(FunctionCall2Coll(<proc,
+ DEFAULT_COLLATION_OID,
+ cst->constvalue,
+ bucket->min[idx]));
+
+ if (tmp)
+ {
+ /* no match */
+ UPDATE_RESULT(matches[i], MVSTATS_MATCH_NONE, is_or);
+ continue;
+ }
+
+ tmp = DatumGetBool(FunctionCall2Coll(<proc,
+ DEFAULT_COLLATION_OID,
+ bucket->max[idx],
+ cst->constvalue));
+
+ if (tmp)
+ {
+ /* no match */
+ UPDATE_RESULT(matches[i], MVSTATS_MATCH_NONE, is_or);
+ continue;
+ }
+
+ /* partial match */
+ UPDATE_RESULT(matches[i], MVSTATS_MATCH_PARTIAL, is_or);
+
+ break;
+ }
+ }
+ }
+ }
+ else if (IsA(clause, NullTest))
+ {
+ NullTest * expr = (NullTest*)clause;
+ Var * var = (Var*)(expr->arg);
+
+ /* FIXME proper matching attribute to dimension */
+ int idx = mv_get_index(var->varattno, stakeys);
+
+ /*
+ * Walk through the buckets and evaluate the current clause. We can
+ * skip items that were already ruled out, and terminate if there are
+ * no remaining buckets that might possibly match.
+ */
+ for (i = 0; i < mvhist->nbuckets; i++)
+ {
+ MVBucket bucket = mvhist->buckets[i];
+
+ /*
+ * Skip buckets that were already eliminated - this is impotant
+ * considering how we update the info (we only lower the match)
+ */
+ if ((! is_or) && (matches[i] == MVSTATS_MATCH_NONE))
+ continue;
+ else if (is_or && (matches[i] == MVSTATS_MATCH_FULL))
+ continue;
+
+ /* if the clause mismatches the MCV item, set it as MATCH_NONE */
+ if ((expr->nulltesttype == IS_NULL)
+ && (! bucket->nullsonly[idx]))
+ UPDATE_RESULT(matches[i], MVSTATS_MATCH_NONE, is_or);
+
+ else if ((expr->nulltesttype == IS_NOT_NULL) &&
+ (bucket->nullsonly[idx]))
+ UPDATE_RESULT(matches[i], MVSTATS_MATCH_NONE, is_or);
+ }
+ }
+ else if (or_clause(clause) || and_clause(clause))
+ {
+ /* AND/OR clause, with all clauses compatible with the selected MV stat */
+
+ int i;
+ BoolExpr *orclause = ((BoolExpr*)clause);
+ List *orclauses = orclause->args;
+
+ /* match/mismatch bitmap for each bucket */
+ int or_nmatches = 0;
+ char * or_matches = NULL;
+
+ Assert(orclauses != NIL);
+ Assert(list_length(orclauses) >= 2);
+
+ /* number of matching buckets */
+ or_nmatches = mvhist->nbuckets;
+
+ /* by default none of the buckets matches the clauses */
+ or_matches = palloc0(sizeof(char) * or_nmatches);
+
+ if (or_clause(clause))
+ {
+ /* OR clauses assume nothing matches, initially */
+ memset(or_matches, MVSTATS_MATCH_NONE, sizeof(char)*or_nmatches);
+ or_nmatches = 0;
+ }
+ else
+ {
+ /* AND clauses assume nothing matches, initially */
+ memset(or_matches, MVSTATS_MATCH_FULL, sizeof(char)*or_nmatches);
+ }
+
+ /* build the match bitmap for the OR-clauses */
+ or_nmatches = update_match_bitmap_histogram(root, orclauses,
+ stakeys, mvhist,
+ or_nmatches, or_matches, or_clause(clause));
+
+ /* merge the bitmap into the existing one*/
+ for (i = 0; i < mvhist->nbuckets; i++)
+ {
+ /*
+ * To AND-merge the bitmaps, a MIN() semantics is used.
+ * For OR-merge, use MAX().
+ *
+ * FIXME this does not decrease the number of matches
+ */
+ UPDATE_RESULT(matches[i], or_matches[i], is_or);
+ }
+
+ pfree(or_matches);
+
+ }
+ else
+ {
+ elog(ERROR, "unknown clause type: %d", clause->type);
+ }
+ }
+
+ return nmatches;
+}
diff --git a/src/backend/utils/mvstats/Makefile b/src/backend/utils/mvstats/Makefile
index 3c0aff4..9dbb3b6 100644
--- a/src/backend/utils/mvstats/Makefile
+++ b/src/backend/utils/mvstats/Makefile
@@ -12,6 +12,6 @@ subdir = src/backend/utils/mvstats
top_builddir = ../../../..
include $(top_builddir)/src/Makefile.global
-OBJS = common.o mcv.o dependencies.o
+OBJS = common.o dependencies.o histogram.o mcv.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/utils/mvstats/common.c b/src/backend/utils/mvstats/common.c
index bd952c6..6e824bd 100644
--- a/src/backend/utils/mvstats/common.c
+++ b/src/backend/utils/mvstats/common.c
@@ -45,7 +45,8 @@ build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
{
MVDependencies deps = NULL;
MCVList mcvlist = NULL;
- int numrows_filtered = 0;
+ MVHistogram histogram = NULL;
+ int numrows_filtered = numrows;
/* int2 vector of attnums the stats should be computed on */
int2vector * attrs = mvstats[i].stakeys;
@@ -66,8 +67,16 @@ build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
if (mvstats->mcv_enabled)
mcvlist = build_mv_mcvlist(numrows, rows, attrs, stats, &numrows_filtered);
+ /* build a multivariate histogram on the columns */
+ if ((numrows_filtered > 0) && (mvstats->hist_enabled))
+ histogram = build_mv_histogram(numrows_filtered, rows, attrs, stats, numrows);
+
/* store the histogram / MCV list in the catalog */
- update_mv_stats(mvstats[i].mvoid, deps, mcvlist, attrs, stats);
+ update_mv_stats(mvstats[i].mvoid, deps, mcvlist, histogram, attrs, stats);
+
+#ifdef MVSTATS_DEBUG
+ print_mv_histogram_info(histogram);
+#endif
}
}
@@ -149,7 +158,7 @@ list_mv_stats(Oid relid, int *nstats, bool built_only)
* Skip statistics that were not computed yet (if only stats
* that were already built were requested)
*/
- if (built_only && (! (stats->mcv_built || stats->deps_built)))
+ if (built_only && (! (stats->mcv_built || stats->deps_built || stats->hist_built)))
continue;
/* double the array size if needed */
@@ -161,10 +170,15 @@ list_mv_stats(Oid relid, int *nstats, bool built_only)
result[*nstats].mvoid = HeapTupleGetOid(htup);
result[*nstats].stakeys = buildint2vector(stats->stakeys.values, stats->stakeys.dim1);
+
result[*nstats].deps_enabled = stats->deps_enabled;
result[*nstats].mcv_enabled = stats->mcv_enabled;
+ result[*nstats].hist_enabled = stats->hist_enabled;
+
result[*nstats].deps_built = stats->deps_built;
result[*nstats].mcv_built = stats->mcv_built;
+ result[*nstats].hist_built = stats->hist_built;
+
*nstats += 1;
}
@@ -178,9 +192,16 @@ list_mv_stats(Oid relid, int *nstats, bool built_only)
return result;
}
+/*
+ * FIXME This adds statistics, but we need to drop statistics when the
+ * table is dropped. Not sure what to do when a column is dropped.
+ * Either we can (a) remove all stats on that column, (b) remove
+ * the column from defined stats and force rebuild, (c) remove the
+ * column on next ANALYZE. Or maybe something else?
+ */
void
update_mv_stats(Oid mvoid,
- MVDependencies dependencies, MCVList mcvlist,
+ MVDependencies dependencies, MCVList mcvlist, MVHistogram histogram,
int2vector *attrs, VacAttrStats **stats)
{
HeapTuple stup,
@@ -213,19 +234,31 @@ update_mv_stats(Oid mvoid,
values[Anum_pg_mv_statistic_stamcv - 1] = PointerGetDatum(data);
}
+ if (histogram != NULL)
+ {
+ bytea * data = serialize_mv_histogram(histogram, attrs, stats);
+ nulls[Anum_pg_mv_statistic_stahist-1] = (data == NULL);
+ values[Anum_pg_mv_statistic_stahist - 1]
+ = PointerGetDatum(data);
+ }
+
/* always replace the value (either by bytea or NULL) */
replaces[Anum_pg_mv_statistic_stadeps -1] = true;
replaces[Anum_pg_mv_statistic_stamcv -1] = true;
+ replaces[Anum_pg_mv_statistic_stahist-1] = true;
/* always change the availability flags */
nulls[Anum_pg_mv_statistic_deps_built -1] = false;
nulls[Anum_pg_mv_statistic_mcv_built -1] = false;
+ nulls[Anum_pg_mv_statistic_hist_built-1] = false;
replaces[Anum_pg_mv_statistic_deps_built-1] = true;
replaces[Anum_pg_mv_statistic_mcv_built -1] = true;
+ replaces[Anum_pg_mv_statistic_hist_built -1] = true;
values[Anum_pg_mv_statistic_deps_built-1] = BoolGetDatum(dependencies != NULL);
values[Anum_pg_mv_statistic_mcv_built -1] = BoolGetDatum(mcvlist != NULL);
+ values[Anum_pg_mv_statistic_hist_built -1] = BoolGetDatum(histogram != NULL);
/* Is there already a pg_mv_statistic tuple for this attribute? */
oldtup = SearchSysCache1(MVSTATOID,
diff --git a/src/backend/utils/mvstats/histogram.c b/src/backend/utils/mvstats/histogram.c
new file mode 100644
index 0000000..2a7f660
--- /dev/null
+++ b/src/backend/utils/mvstats/histogram.c
@@ -0,0 +1,1800 @@
+/*-------------------------------------------------------------------------
+ *
+ * histogram.c
+ * POSTGRES multivariate histograms
+ *
+ *
+ * Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/utils/mvstats/histogram.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "common.h"
+
+/*
+ * Multivariate histograms
+ *
+ * Histograms are a collection of buckets, represented by n-dimensional
+ * rectangles. Each rectangle is delimited by a min/max value in each
+ * dimension, stored in an array, so that the bucket includes values
+ * fulfilling condition
+ *
+ * min[i] <= value[i] <= max[i]
+ *
+ * where 'i' is the dimension. In 1D this corresponds to a simple
+ * interval, in 2D to a rectangle, and in 3D to a block. If you can
+ * imagine this in 4D, congrats!
+ *
+ * In addition to the bounaries, each bucket tracks additional details:
+ *
+ * * frequency (fraction of tuples it matches)
+ * * whether the boundaries are inclusive or exclusive
+ * * whether the dimension contains only NULL values
+ * * number of distinct values in each dimension (for building)
+ *
+ * and possibly some additional information.
+ *
+ * We do expect to support multiple histogram types, with different
+ * features etc. The 'type' field is used to identify those types.
+ * Technically some histogram types might use completely different
+ * bucket representation, but that's not expected at the moment.
+ *
+ * Although the current implementation builds non-overlapping buckets,
+ * the code does not rely on the non-overlapping nature - there are
+ * interesting types of histograms / histogram building algorithms
+ * producing overlapping buckets.
+ *
+ * TODO Currently the histogram does not include information about what
+ * part of the table it covers (because the frequencies are
+ * computed from the rows that may be filtered by MCV list). Seems
+ * wrong, possibly causing misestimates (when not matching the MCV
+ * list, we'll probably get much higher selectivity).
+ *
+ *
+ * Estimating selectivity
+ * ----------------------
+ * With histograms, we always "match" a whole bucket, not indivitual
+ * rows (or values), irrespectedly of the type of clause. Therefore we
+ * can't use the optimizations for equality clauses, as in MCV lists.
+ *
+ * The current implementation uses histograms to estimates those types
+ * of clauses (think of WHERE conditions):
+ *
+ * (a) equality clauses WHERE (a = 1) AND (b = 2)
+ * (b) inequality clauses WHERE (a < 1) AND (b >= 2)
+ *
+ * It's possible to add more clauses, for example:
+ *
+ * (a) NULL clauses WHERE (a IS NULL) AND (b IS NOT NULL)
+ * (b) multi-var clauses WHERE (a > b)
+ *
+ * and so on. These are tasks for the future, not yet implemented.
+ *
+ * When used on low-cardinality data, histograms usually perform
+ * considerably worse than MCV lists (which are a good fit for this
+ * kind of data). This is especially true on categorical data, where
+ * ordering of the values is only loosely related to meaning of the
+ * data, as proper ordering is crucial for histograms.
+ *
+ * On high-cardinality data the histograms are usually a better choice,
+ * because MCV lists can't accurately represent the distribution.
+ *
+ * By evaluating a clause on a bucket, we may get one of three results:
+ *
+ * (a) FULL_MATCH - The bucket definitely matches the clause.
+ *
+ * (b) PARTIAL_MATCH - The bucket matches the clause, but not
+ * necessarily all the tuples it represents.
+ *
+ * (c) NO_MATCH - The bucket definitely does not match the clause.
+ *
+ * This may be illustrated using a range [1, 5], which is essentially
+ * a 1D bucket. With clause
+ *
+ * WHERE (a < 10) => FULL_MATCH (all range values are below
+ * 10, so the whole bucket matches)
+ *
+ * WHERE (a < 3) => PARTIAL_MATCH (there may be values matching
+ * the clause, but we don't know how many)
+ *
+ * WHERE (a < 0) => NO_MATCH (all range values are above 1, so
+ * no values from the bucket match)
+ *
+ * Some clauses may produce only some of those results - for example
+ * equality clauses may never produce FULL_MATCH as we always hit only
+ * part of the bucket, not all the values. This results in less accurate
+ * estimates compared to MCV lists, where we can hit a MCV items exactly
+ * (an extreme case of that is 'full match').
+ *
+ * There are clauses that may not produce any PARTIAL_MATCH results.
+ * A nice example of that is 'IS [NOT] NULL' clause, which either
+ * matches the bucket completely (FULL_MATCH) or not at all (NO_MATCH),
+ * thanks to how the NULL-buckets are constructed.
+ *
+ * TODO The IS [NOT] NULL clause is not yet implemented, but should be
+ * rather trivial to.
+ *
+ * Computing the total selectivity estimate is trivial - simply sum
+ * selectivities from all the FULL_MATCH and PARTIAL_MATCH buckets, but
+ * multiply the PARTIAL_MATCH buckets by 0.5 to minimize average error.
+ *
+ *
+ * NULL handling
+ * -------------
+ * Buckets may not contain tuples with NULL and non-NULL values in
+ * a single dimension (attribute). To handle this, the histogram may
+ * contain NULL-buckets, i.e. buckets with one or more NULL-only
+ * dimensions.
+ *
+ * The maximum number of NULL-buckets is determined by the number of
+ * attributes the histogram is built on. For N-dimensional histogram,
+ * the maximum number of NULL-buckets is 2^N. So for 8 attributes
+ * (which is the current value of MVSTATS_MAX_DIMENSIONS), there may be
+ * up to 256 NULL-buckets.
+ *
+ * Those buckets are only built if needed - if there are no NULL values
+ * in the data, no such buckets are built.
+ *
+ *
+ * Serialization
+ * -------------
+ * After building, the histogram is serialized into a more efficient
+ * form (dedup boundary values etc.). See serialize_mv_histogram() for
+ * more details about how it's done.
+ *
+ * Serialized histograms are marked with 'magic' constant, to make it
+ * easier to check the bytea really is a histogram in serialized form.
+ *
+ *
+ * TODO This structure is used both when building the histogram, and
+ * then when using it to compute estimates. That's why the last
+ * few elements are not used once the histogram is built.
+ *
+ * Add pointer to 'private' data, meant for private data for
+ * other algorithms for building the histogram. It also removes
+ * the bogus / unnecessary fields.
+ *
+ * TODO The limit on number of buckets is quite arbitrary, aiming for
+ * sufficient accuracy while still being fast. Probably should be
+ * replaced with a dynamic limit dependent on statistics target,
+ * number of attributes (dimensions) and statistics target
+ * associated with the attributes. Also, this needs to be related
+ * to the number of sampled rows, by either clamping it to a
+ * reasonable number (after seeing the number of rows) or using
+ * it when computing the number of rows to sample. Something like
+ * 10 rows per bucket seems reasonable.
+ *
+ * TODO Add MVSTAT_HIST_ROWS_PER_BUCKET tracking minimal number of
+ * tuples per bucket (also, see the previous TODO).
+ *
+ * TODO We may replace the bool arrays with a suitably large data type
+ * (say, uint16 or uint32) and get rid of the allocations. It's
+ * unlikely we'll ever support more than 32 columns as that'd
+ * result in poor precision, huge histograms (splitting each
+ * dimension once would mean 2^32 buckets), and very expensive
+ * estimation. MCVItem already does it this way.
+ *
+ * Update: Actually, this is not 100% true, because we're splitting
+ * a single bucket, not all the buckets at the same time. So each
+ * split simply adds one new bucket, and we choose the bucket that
+ * is most in need of a slit. So even with 32 columns this might
+ * give reasonable accuracy, maybe? After 1000 splits we'll get
+ * about 1001 buckets, and some may be quite large (if that area
+ * frequency has low frequency of tuples).
+ *
+ * There are other challenges though - e.g. with this many columns
+ * it's more likely to reference both label/non-label columns,
+ * which is rather quirky (especially with histograms).
+ *
+ * However, while this would save some space for histograms built
+ * on many columns, it won't save anything for up to 4 columns
+ * (actually, on less than 3 columns it's probably wasteful).
+ *
+ * TODO Maybe the distinct stats (both for combination of all columns
+ * and for combinations of various subsets of columns) should be
+ * moved to a separate structure (next to histogram/MCV/...) to
+ * make it useful even without a histogram computed etc.
+ */
+
+static MVBucket create_initial_mv_bucket(int numrows, HeapTuple *rows,
+ int2vector *attrs,
+ VacAttrStats **stats);
+
+static MVBucket select_bucket_to_partition(int nbuckets, MVBucket * buckets);
+
+static MVBucket partition_bucket(MVBucket bucket, int2vector *attrs,
+ VacAttrStats **stats);
+
+static MVBucket copy_mv_bucket(MVBucket bucket, uint32 ndimensions);
+
+static void update_bucket_ndistinct(MVBucket bucket, int2vector *attrs,
+ VacAttrStats ** stats);
+
+static void update_dimension_ndistinct(MVBucket bucket, int dimension,
+ int2vector *attrs,
+ VacAttrStats ** stats,
+ bool update_boundaries);
+
+static void create_null_buckets(MVHistogram histogram, int bucket_idx,
+ int2vector *attrs, VacAttrStats ** stats);
+
+static int bsearch_comparator(const void * a, const void * b);
+
+/*
+ * Each serialized bucket needs to store (in this order):
+ *
+ * - number of tuples (float)
+ * - number of distinct (float)
+ * - min inclusive flags (ndim * sizeof(bool))
+ * - max inclusive flags (ndim * sizeof(bool))
+ * - null dimension flags (ndim * sizeof(bool))
+ * - min boundary indexes (2 * ndim * sizeof(int32))
+ * - max boundary indexes (2 * ndim * sizeof(int32))
+ *
+ * So in total:
+ *
+ * ndim * (4 * sizeof(int32) + 3 * sizeof(bool)) +
+ * 2 * sizeof(float)
+ */
+#define BUCKET_SIZE(ndims) \
+ (ndims * (4 * sizeof(int32) + 3 * sizeof(bool)) + sizeof(float))
+
+/* pointers into a flat serialized bucket of BUCKET_SIZE(n) bytes */
+#define BUCKET_NTUPLES(b) ((float*)b)
+#define BUCKET_MIN_INCL(b,n) ((bool*)(b + sizeof(float)))
+#define BUCKET_MAX_INCL(b,n) (BUCKET_MIN_INCL(b,n) + n)
+#define BUCKET_NULLS_ONLY(b,n) (BUCKET_MAX_INCL(b,n) + n)
+#define BUCKET_MIN_INDEXES(b,n) ((int32*)(BUCKET_NULLS_ONLY(b,n) + n))
+#define BUCKET_MAX_INDEXES(b,n) ((BUCKET_MIN_INDEXES(b,n) + n))
+
+/* some debugging methods */
+#ifdef MVSTATS_DEBUG
+static void print_mv_histogram_info(MVHistogram histogram);
+#endif
+
+/*
+ * Data used while building the histogram.
+ */
+typedef struct HistogramBuildData {
+
+ float ndistinct; /* frequency of distinct values */
+
+ HeapTuple *rows; /* aray of sample rows */
+ uint32 numrows; /* number of sample rows (array size) */
+
+ /* index of the dimension the bucket was split previously */
+ int last_split_dimension;
+
+ /*
+ * Number of distinct values in each dimension. This is used when
+ * building the histogram (and is not serialized/deserialized).
+ *
+ * XXX Maybe it could be useful for improving ndistinct estimates for
+ * combinations of columns (e.g. in GROUP BY queries). It would
+ * probably mean tracking 2^N values for each bucket, and even if
+ * those values might be stores in 1B (which is unlikely) it's
+ * still a lot of space (considering the expected number of
+ * buckets). So maybe that might be tracked just at the top level.
+ *
+ * TODO Consider tracking ndistincts for all attribute combinations.
+ */
+ uint32 *ndistincts;
+
+} HistogramBuildData;
+
+typedef HistogramBuildData *HistogramBuild;
+
+/*
+ * Building a multivariate algorithm. In short it first creates a single
+ * bucket containing all the rows, and then repeatedly split is by first
+ * searching for the bucket / dimension most in need of a split.
+ *
+ * The current criteria is rather simple, by looking at the number of
+ * distinct values (combination of column values for bucket, column
+ * values for a dimension). This is somehow naive, but seems to work
+ * quite well. See the discussion at select_bucket_to_partition and
+ * partition_bucket for more details about alternative algorithms.
+ *
+ * So the current algorithm looks like this:
+ *
+ * build NULL-buckets (create_null_buckets)
+ *
+ * while [not reaching maximum number of buckets]
+ *
+ * choose bucket to partition (max distinct combinations)
+ * if no bucket to partition
+ * terminate the algorithm
+ *
+ * choose bucket dimension to partition (max distinct values)
+ * split the bucket into two buckets
+ */
+MVHistogram
+build_mv_histogram(int numrows, HeapTuple *rows, int2vector *attrs,
+ VacAttrStats **stats, int numrows_total)
+{
+ int i;
+ int numattrs = attrs->dim1;
+
+ MVHistogram histogram = (MVHistogram)palloc0(sizeof(MVHistogramData));
+
+ HeapTuple * rows_copy = (HeapTuple*)palloc0(numrows * sizeof(HeapTuple));
+ memcpy(rows_copy, rows, sizeof(HeapTuple) * numrows);
+
+ Assert((numattrs >= 2) && (numattrs <= MVSTATS_MAX_DIMENSIONS));
+
+ histogram->ndimensions = numattrs;
+
+ histogram->magic = MVSTAT_HIST_MAGIC;
+ histogram->type = MVSTAT_HIST_TYPE_BASIC;
+ histogram->nbuckets = 1;
+
+ /* create max buckets (better than repalloc for short-lived objects) */
+ histogram->buckets = (MVBucket*)palloc0(MVSTAT_HIST_MAX_BUCKETS * sizeof(MVBucket));
+
+ /* create the initial bucket, covering the whole sample set */
+ histogram->buckets[0]
+ = create_initial_mv_bucket(numrows, rows_copy, attrs, stats);
+
+ /*
+ * The initial bucket may contain NULL values, so we have to create
+ * buckets with NULL-only dimensions.
+ *
+ * FIXME We may need up to 2^ndims buckets - check that there are
+ * enough buckets (MVSTAT_HIST_MAX_BUCKETS >= 2^ndims).
+ */
+ create_null_buckets(histogram, 0, attrs, stats);
+
+ while (histogram->nbuckets < MVSTAT_HIST_MAX_BUCKETS)
+ {
+ MVBucket bucket = select_bucket_to_partition(histogram->nbuckets,
+ histogram->buckets);
+
+ /* no more buckets to partition */
+ if (bucket == NULL)
+ break;
+
+ histogram->buckets[histogram->nbuckets]
+ = partition_bucket(bucket, attrs, stats);
+
+ histogram->nbuckets += 1;
+ }
+
+ /* finalize the frequencies etc. */
+ for (i = 0; i < histogram->nbuckets; i++)
+ {
+ HistogramBuild build_data = ((HistogramBuild)histogram->buckets[i]->build_data);
+ histogram->buckets[i]->ntuples
+ = (build_data->numrows * 1.0) / numrows_total;
+ }
+
+ return histogram;
+}
+
+/* fetch the histogram (as a bytea) from the pg_mv_statistic catalog */
+bytea *
+fetch_mv_histogram(Oid mvoid)
+{
+ Relation indrel;
+ SysScanDesc indscan;
+ ScanKeyData skey;
+ HeapTuple htup;
+ bytea *stahist = NULL;
+
+ /* Prepare to scan pg_mv_statistic for entries having indrelid = this rel. */
+ ScanKeyInit(&skey,
+ ObjectIdAttributeNumber,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(mvoid));
+
+ indrel = heap_open(MvStatisticRelationId, AccessShareLock);
+ indscan = systable_beginscan(indrel, MvStatisticOidIndexId, true,
+ NULL, 1, &skey);
+
+ while (HeapTupleIsValid(htup = systable_getnext(indscan)))
+ {
+ bool isnull = false;
+ Datum hist = SysCacheGetAttr(MVSTATOID, htup,
+ Anum_pg_mv_statistic_stahist, &isnull);
+
+ Assert(!isnull);
+
+ stahist = DatumGetByteaP(hist);
+
+ break;
+ }
+
+ systable_endscan(indscan);
+
+ heap_close(indrel, AccessShareLock);
+
+ /*
+ * TODO Maybe save the histogram into relcache, as in RelationGetIndexList
+ * (which was used as an inspiration of this one)?.
+ */
+
+ return stahist;
+}
+
+/* print some basic info about the histogram */
+Datum
+pg_mv_stats_histogram_info(PG_FUNCTION_ARGS)
+{
+ bytea *data = PG_GETARG_BYTEA_P(0);
+ char *result;
+
+ MVHistogram hist = deserialize_mv_histogram(data);
+
+ result = palloc0(128);
+ snprintf(result, 128, "nbuckets=%d", hist->nbuckets);
+
+ PG_RETURN_TEXT_P(cstring_to_text(result));
+}
+
+
+/* used to pass context into bsearch() */
+static SortSupport ssup_private = NULL;
+
+/*
+ * Serialize the MV histogram into a bytea value. The basic algorithm
+ * is simple, and mostly mimincs the MCV serialization:
+ *
+ * (1) perform deduplication for each attribute (separately)
+ * (a) collect all (non-NULL) attribute values from all buckets
+ * (b) sort the data (using 'lt' from VacAttrStats)
+ * (c) remove duplicate values from the array
+ *
+ * (2) serialize the arrays into a bytea value
+ *
+ * (3) process all buckets
+ * (a) replace min/max values with indexes into the arrays
+ *
+ * Each attribute has to be processed separately, because we're mixing
+ * different datatypes, and we don't know what equality means for them.
+ * We're also mixing pass-by-value and pass-by-ref types, and so on.
+ *
+ * We'll use 32-bit values for the indexes in step (3), although we
+ * could probably use just 16 bits as we don't allow more than 8k
+ * buckets in the histogram max_buckets (well, we might increase this
+ * to 16k and still fit into signed 16-bits). But let's be lazy and rely
+ * on the varlena compression to kick in. If most bytes will be 0x00
+ * so it should work nicely.
+ *
+ *
+ * Deduplication in serialization
+ * ------------------------------
+ * The deduplication is very effective and important here, because every
+ * time we split a bucket, we keep all the boundary values, except for
+ * the dimension that was used for the split. Another way to look at
+ * this is that each split introduces 1 new value (the value used to do
+ * the split). A histogram with M buckets was created by (M-1) splits
+ * of the initial bucket, and each bucket has 2*N boundary values. So
+ * assuming the initial bucket does not have any 'collapsed' dimensions,
+ * the number of distinct values is
+ *
+ * (2*N + (M-1))
+ *
+ * but the total number of boundary values is
+ *
+ * 2*N*M
+ *
+ * which is clearly much higher. For a histogram on two columns, with
+ * 1024 buckets, it's 1027 vs. 4096. Of course, we're not saving all
+ * the difference (because we'll use 32-bit indexes into the values).
+ * But with large values (e.g. stored as varlena), this saves a lot.
+ *
+ * An interesting feature is that the total number of distinct values
+ * does not really grow with the number of dimensions, except for the
+ * size of the initial bucket. After that it only depends on number of
+ * buckets (i.e. number of splits).
+ *
+ * XXX Of course this only holds for the current histogram building
+ * algorithm. Algorithms doing the splits differently (e.g.
+ * producing overlapping buckets) may behave differently.
+ *
+ * TODO This only confirms we can use the uint16 indexes. The worst
+ * that could happen is if all the splits happened by a single
+ * dimension. To exhaust the uint16 this would require ~64k
+ * splits (needs to be reflected in MVSTAT_HIST_MAX_BUCKETS).
+ *
+ * TODO We don't need to use a separate boolean for each flag, instead
+ * use a single char and set bits.
+ *
+ * TODO We might get a bit better compression by considering the actual
+ * data type length. The current implementation treats all data
+ * types passed by value as requiring 8B, but for INT it's actually
+ * just 4B etc.
+ *
+ * OTOH this is only related to the lookup table, and most of the
+ * space is occupied by the buckets (with int16 indexes).
+ *
+ *
+ * Varlena compression
+ * -------------------
+ * This encoding may prevent automatic varlena compression (similarly
+ * to JSONB), because first part of the serialized bytea will be an
+ * array of unique values (although sorted), and pglz decides whether
+ * to compress by trying to compress the first part (~1kB or so). Which
+ * is likely to be poor, due to the lack of repetition.
+ *
+ * One possible cure to that might be storing the buckets first, and
+ * then the deduplicated arrays. The buckets might be better suited
+ * for compression.
+ *
+ * On the other hand the encoding scheme is a context-aware compression,
+ * usually compressing to ~30% (or less, with large data types). So the
+ * lack of pglz compression may be OK.
+ *
+ * XXX But maybe we don't really want to compress this, to save on
+ * planning time?
+ *
+ * TODO Try storing the buckets / deduplicated arrays in reverse order,
+ * measure impact on compression.
+ *
+ *
+ * Deserialization
+ * ---------------
+ * The deserialization is currently implemented so that it reconstructs
+ * the histogram back into the same structures - this involves quite
+ * a few of memcpy() and palloc(), but maybe we could create a special
+ * structure for the serialized histogram, and access the data directly,
+ * without the unpacking.
+ *
+ * Not only it would save some memory and CPU time, but might actually
+ * work better with CPU caches (not polluting the caches).
+ *
+ * TODO Try to keep the compressed form, instead of deserializing it to
+ * MVHistogram/MVBucket.
+ *
+ *
+ * General TODOs
+ * -------------
+ * FIXME This probably leaks memory, or at least uses it inefficiently
+ * (many small palloc() calls instead of a large one).
+ *
+ * FIXME This probably leaks memory, or at least uses it inefficiently
+ * (many small palloc() calls instead of a large one).
+ *
+ * TODO Consider packing boolean flags (NULL) for each item into 'char'
+ * or a longer type (instead of using an array of bool items).
+ */
+bytea *
+serialize_mv_histogram(MVHistogram histogram, int2vector *attrs,
+ VacAttrStats **stats)
+{
+ int i = 0, j = 0;
+ Size total_length = 0;
+
+ bytea *output = NULL;
+ char *data = NULL;
+
+ int nbuckets = histogram->nbuckets;
+ int ndims = histogram->ndimensions;
+
+ /* allocated for serialized bucket data */
+ int bucketsize = BUCKET_SIZE(ndims);
+ char *bucket = palloc0(bucketsize);
+
+ /* values per dimension (and number of non-NULL values) */
+ Datum **values = (Datum**)palloc0(sizeof(Datum*) * ndims);
+ int *counts = (int*)palloc0(sizeof(int) * ndims);
+
+ /* info about dimensions (for deserialize) */
+ DimensionInfo * info
+ = (DimensionInfo *)palloc0(sizeof(DimensionInfo)*ndims);
+
+ /* sort support data */
+ SortSupport ssup = (SortSupport)palloc0(sizeof(SortSupportData)*ndims);
+
+ /* collect and deduplicate values for each dimension separately */
+ for (i = 0; i < ndims; i++)
+ {
+ int count;
+ StdAnalyzeData *tmp = (StdAnalyzeData *)stats[i]->extra_data;
+
+ /* keep important info about the data type */
+ info[i].typlen = stats[i]->attrtype->typlen;
+ info[i].typbyval = stats[i]->attrtype->typbyval;
+
+ /*
+ * Allocate space for all min/max values, including NULLs
+ * (we won't use them, but we don't know how many are there),
+ * and then collect all non-NULL values.
+ */
+ values[i] = (Datum*)palloc0(sizeof(Datum) * nbuckets * 2);
+
+ for (j = 0; j < histogram->nbuckets; j++)
+ {
+ /* skip buckets where this dimension is NULL-only */
+ if (! histogram->buckets[j]->nullsonly[i])
+ {
+ values[i][counts[i]] = histogram->buckets[j]->min[i];
+ counts[i] += 1;
+
+ values[i][counts[i]] = histogram->buckets[j]->max[i];
+ counts[i] += 1;
+ }
+ }
+
+ /* there are just NULL values in this dimension */
+ if (counts[i] == 0)
+ continue;
+
+ /* sort and deduplicate */
+ ssup[i].ssup_cxt = CurrentMemoryContext;
+ ssup[i].ssup_collation = DEFAULT_COLLATION_OID;
+ ssup[i].ssup_nulls_first = false;
+
+ PrepareSortSupportFromOrderingOp(tmp->ltopr, &ssup[i]);
+
+ qsort_arg(values[i], counts[i], sizeof(Datum),
+ compare_scalars_simple, &ssup[i]);
+
+ /*
+ * Walk through the array and eliminate duplicitate values, but
+ * keep the ordering (so that we can do bsearch later). We know
+ * there's at least 1 item, so we can skip the first element.
+ */
+ count = 1; /* number of deduplicated items */
+ for (j = 1; j < counts[i]; j++)
+ {
+ /* if it's different from the previous value, we need to keep it */
+ if (compare_datums_simple(values[i][j-1], values[i][j], &ssup[i]) != 0)
+ {
+ /* XXX: not needed if (count == j) */
+ values[i][count] = values[i][j];
+ count += 1;
+ }
+ }
+
+ /* keep info about the deduplicated count */
+ info[i].nvalues = count;
+
+ /* compute size of the serialized data */
+ if (info[i].typbyval)
+ /*
+ * passed by value, so just Datum array (int4, int8, ...)
+ *
+ * TODO Might save a few bytes here, by storing just typlen
+ * bytes instead of whole Datum (8B) on 64-bits.
+ */
+ info[i].nbytes = info[i].nvalues * sizeof(Datum);
+ else if (info[i].typlen > 0)
+ /* pased by reference, but fixed length (name, tid, ...) */
+ info[i].nbytes = info[i].nvalues * info[i].typlen;
+ else if (info[i].typlen == -1)
+ /* varlena, so just use VARSIZE_ANY */
+ for (j = 0; j < info[i].nvalues; j++)
+ info[i].nbytes += VARSIZE_ANY(values[i][j]);
+ else if (info[i].typlen == -2)
+ /* cstring, so simply strlen */
+ for (j = 0; j < info[i].nvalues; j++)
+ info[i].nbytes += strlen(DatumGetPointer(values[i][j]));
+ else
+ elog(ERROR, "unknown data type typbyval=%d typlen=%d",
+ info[i].typbyval, info[i].typlen);
+ }
+
+ /*
+ * Now we finally know how much space we'll need for the serialized
+ * histogram, as it contains these fields:
+ *
+ * - length (4B) for varlena
+ * - magic (4B)
+ * - type (4B)
+ * - ndimensions (4B)
+ * - nbuckets (4B)
+ * - info (ndim * sizeof(DimensionInfo)
+ * - arrays of values for each dimension
+ * - serialized buckets (nbuckets * bucketsize)
+ *
+ * So the 'header' size is 20B + ndim * sizeof(DimensionInfo) and
+ * then we'll place the data (and buckets).
+ */
+ total_length = (sizeof(int32) + offsetof(MVHistogramData, buckets)
+ + ndims * sizeof(DimensionInfo)
+ + nbuckets * bucketsize);
+
+ /* account for the deduplicated data */
+ for (i = 0; i < ndims; i++)
+ total_length += info[i].nbytes;
+
+ /* enforce arbitrary limit of 1MB */
+ if (total_length > 1024 * 1024)
+ elog(ERROR, "serialized histogram exceeds 1MB (%ld)", total_length);
+
+ /* allocate space for the serialized histogram list, set header */
+ output = (bytea*)palloc0(total_length);
+ SET_VARSIZE(output, total_length);
+
+ /* we'll use 'data' to keep track of the place to write data */
+ data = VARDATA(output);
+
+ memcpy(data, histogram, offsetof(MVHistogramData, buckets));
+ data += offsetof(MVHistogramData, buckets);
+
+ memcpy(data, info, sizeof(DimensionInfo) * ndims);
+ data += sizeof(DimensionInfo) * ndims;
+
+ /* value array for each dimension */
+ for (i = 0; i < ndims; i++)
+ {
+#ifdef USE_ASSERT_CHECKING
+ char *tmp = data;
+#endif
+ for (j = 0; j < info[i].nvalues; j++)
+ {
+ if (info[i].typbyval)
+ {
+ /* passed by value / Datum */
+ memcpy(data, &values[i][j], sizeof(Datum));
+ data += sizeof(Datum);
+ }
+ else if (info[i].typlen > 0)
+ {
+ /* pased by reference, but fixed length (name, tid, ...) */
+ memcpy(data, &values[i][j], info[i].typlen);
+ data += info[i].typlen;
+ }
+ else if (info[i].typlen == -1)
+ {
+ /* varlena */
+ memcpy(data, DatumGetPointer(values[i][j]),
+ VARSIZE_ANY(values[i][j]));
+ data += VARSIZE_ANY(values[i][j]);
+ }
+ else if (info[i].typlen == -2)
+ {
+ /* cstring (don't forget the \0 terminator!) */
+ memcpy(data, DatumGetPointer(values[i][j]),
+ strlen(DatumGetPointer(values[i][j])) + 1);
+ data += strlen(DatumGetPointer(values[i][j])) + 1;
+ }
+ }
+ Assert((data - tmp) == info[i].nbytes);
+ }
+
+ /* and finally, the histogram buckets */
+ for (i = 0; i < nbuckets; i++)
+ {
+ /* don't write beyond the allocated space */
+ Assert(data <= (char*)output + total_length - bucketsize);
+
+ /* reset the values for each item */
+ memset(bucket, 0, bucketsize);
+
+ *BUCKET_NTUPLES(bucket) = histogram->buckets[i]->ntuples;
+
+ for (j = 0; j < ndims; j++)
+ {
+ /* do the lookup only for non-NULL values */
+ if (! histogram->buckets[i]->nullsonly[j])
+ {
+ int idx;
+ Datum * v = NULL;
+ ssup_private = &ssup[j];
+
+ /* min boundary */
+ v = (Datum*)bsearch(&histogram->buckets[i]->min[j],
+ values[j], info[j].nvalues, sizeof(Datum),
+ bsearch_comparator);
+
+ if (v == NULL)
+ elog(ERROR, "value for dim %d not found in array", j);
+
+ /* compute index within the array */
+ idx = (v - values[j]);
+
+ Assert((idx >= 0) && (idx < info[j].nvalues));
+
+ BUCKET_MIN_INDEXES(bucket, ndims)[j] = idx;
+
+ /* max boundary */
+ v = (Datum*)bsearch(&histogram->buckets[i]->max[j],
+ values[j], info[j].nvalues, sizeof(Datum),
+ bsearch_comparator);
+
+ if (v == NULL)
+ elog(ERROR, "value for dim %d not found in array", j);
+
+ /* compute index within the array */
+ idx = (v - values[j]);
+
+ Assert((idx >= 0) && (idx < info[j].nvalues));
+
+ BUCKET_MAX_INDEXES(bucket, ndims)[j] = idx;
+ }
+ }
+
+ /* copy flags (nulls, min/max inclusive) */
+ memcpy(BUCKET_NULLS_ONLY(bucket, ndims),
+ histogram->buckets[i]->nullsonly, sizeof(bool) * ndims);
+
+ memcpy(BUCKET_MIN_INCL(bucket, ndims),
+ histogram->buckets[i]->min_inclusive, sizeof(bool) * ndims);
+
+ memcpy(BUCKET_MAX_INCL(bucket, ndims),
+ histogram->buckets[i]->max_inclusive, sizeof(bool) * ndims);
+
+ /* copy the item into the array */
+ memcpy(data, bucket, bucketsize);
+
+ data += bucketsize;
+ }
+
+ /* at this point we expect to match the total_length exactly */
+ Assert((data - (char*)output) == total_length);
+
+ /* FIXME free the values/counts arrays here */
+
+ return output;
+}
+
+/*
+ * Reverse to serialize histogram. This essentially expands the serialized
+ * form back to MVHistogram / MVBucket.
+ */
+MVHistogram
+deserialize_mv_histogram(bytea * data)
+{
+ int i = 0, j = 0;
+
+ Size expected_size;
+ char *tmp = NULL;
+ Datum **values = NULL;
+
+ MVHistogram histogram;
+ DimensionInfo *info;
+
+ int nbuckets;
+ int ndims;
+ int bucketsize;
+
+ if (data == NULL)
+ return NULL;
+
+ if (VARSIZE_ANY_EXHDR(data) < offsetof(MVHistogramData,buckets))
+ elog(ERROR, "invalid histogram size %ld (expected at least %ld)",
+ VARSIZE_ANY_EXHDR(data), offsetof(MVHistogramData,buckets));
+
+ /* read the histogram header */
+ histogram = (MVHistogram)palloc0(sizeof(MVHistogramData));
+
+ /* initialize pointer to the data part (skip the varlena header) */
+ tmp = VARDATA(data);
+
+ /* get the header and perform basic sanity checks */
+ memcpy(histogram, tmp, offsetof(MVHistogramData, buckets));
+ tmp += offsetof(MVHistogramData, buckets);
+
+ if (histogram->magic != MVSTAT_HIST_MAGIC)
+ elog(ERROR, "invalid histogram magic %d (expected %dd)",
+ histogram->magic, MVSTAT_HIST_MAGIC);
+
+ if (histogram->type != MVSTAT_HIST_TYPE_BASIC)
+ elog(ERROR, "invalid histogram type %d (expected %dd)",
+ histogram->type, MVSTAT_HIST_TYPE_BASIC);
+
+ nbuckets = histogram->nbuckets;
+ ndims = histogram->ndimensions;
+ bucketsize = BUCKET_SIZE(ndims);
+
+ Assert((nbuckets > 0) && (nbuckets <= MVSTAT_HIST_MAX_BUCKETS));
+ Assert((ndims >= 2) && (ndims <= MVSTATS_MAX_DIMENSIONS));
+
+ /*
+ * What size do we expect with those parameters (it's incomplete,
+ * as we yet have to count the array sizes (from DimensionInfo
+ * records).
+ */
+ expected_size = offsetof(MVHistogramData,buckets) +
+ ndims * sizeof(DimensionInfo) +
+ (nbuckets * bucketsize);
+
+ /* check that we have at least the DimensionInfo records */
+ if (VARSIZE_ANY_EXHDR(data) < expected_size)
+ elog(ERROR, "invalid histogram size %ld (expected %ld)",
+ VARSIZE_ANY_EXHDR(data), expected_size);
+
+ info = (DimensionInfo*)(tmp);
+ tmp += ndims * sizeof(DimensionInfo);
+
+ /* account for the value arrays */
+ for (i = 0; i < ndims; i++)
+ expected_size += info[i].nbytes;
+
+ if (VARSIZE_ANY_EXHDR(data) != expected_size)
+ elog(ERROR, "invalid histogram size %ld (expected %ld)",
+ VARSIZE_ANY_EXHDR(data), expected_size);
+
+ /* looks OK - not corrupted or something */
+
+ /* let's parse the value arrays */
+ values = (Datum**)palloc0(sizeof(Datum*) * ndims);
+
+ /*
+ * FIXME This uses pointers to the original data array (the types
+ * not passed by value), so when someone frees the memory,
+ * e.g. by doing something like this:
+ *
+ * bytea * data = ... fetch the data from catalog ...
+ * MCVList mcvlist = deserialize_mcv_list(data);
+ * pfree(data);
+ *
+ * then 'mcvlist' references the freed memory. This needs to
+ * copy the pieces.
+ *
+ * TODO same as in MCV deserialization / consider moving to common.c
+ */
+ for (i = 0; i < ndims; i++)
+ {
+ if (info[i].typbyval)
+ {
+ /* passed by value / Datum - simply reuse the array */
+ values[i] = (Datum*)tmp;
+ tmp += info[i].nbytes;
+ }
+ else if (info[i].typlen > 0)
+ {
+ /* pased by reference, but fixed length (name, tid, ...) */
+ values[i] = (Datum*)palloc0(sizeof(Datum) * info[i].nvalues);
+ for (j = 0; j < info[i].nvalues; j++)
+ {
+ /* just point into the array */
+ values[i][j] = PointerGetDatum(tmp);
+ tmp += info[i].typlen;
+ }
+ }
+ else if (info[i].typlen == -1)
+ {
+ /* varlena */
+ values[i] = (Datum*)palloc0(sizeof(Datum) * info[i].nvalues);
+ for (j = 0; j < info[i].nvalues; j++)
+ {
+ /* just point into the array */
+ values[i][j] = PointerGetDatum(tmp);
+ tmp += VARSIZE_ANY(tmp);
+ }
+ }
+ else if (info[i].typlen == -2)
+ {
+ /* cstring */
+ values[i] = (Datum*)palloc0(sizeof(Datum) * info[i].nvalues);
+ for (j = 0; j < info[i].nvalues; j++)
+ {
+ /* just point into the array */
+ values[i][j] = PointerGetDatum(tmp);
+ tmp += (strlen(tmp) + 1); /* don't forget the \0 */
+ }
+ }
+ }
+
+ /* allocate space for the buckets */
+ histogram->buckets = (MVBucket*)palloc0(sizeof(MVBucket) * nbuckets);
+
+ for (i = 0; i < nbuckets; i++)
+ {
+ MVBucket bucket = (MVBucket)palloc0(sizeof(MVBucketData));
+
+ bucket->nullsonly = (bool*) palloc0(sizeof(bool) * ndims);
+ bucket->min_inclusive = (bool*) palloc0(sizeof(bool) * ndims);
+ bucket->max_inclusive = (bool*) palloc0(sizeof(bool) * ndims);
+
+ bucket->min = (Datum*) palloc0(sizeof(Datum) * ndims);
+ bucket->max = (Datum*) palloc0(sizeof(Datum) * ndims);
+
+ bucket->ntuples = *BUCKET_NTUPLES(tmp);
+
+ memcpy(bucket->nullsonly, BUCKET_NULLS_ONLY(tmp, ndims),
+ sizeof(bool) * ndims);
+
+ memcpy(bucket->min_inclusive, BUCKET_MIN_INCL(tmp, ndims),
+ sizeof(bool) * ndims);
+
+ memcpy(bucket->max_inclusive, BUCKET_MAX_INCL(tmp, ndims),
+ sizeof(bool) * ndims);
+
+ /* translate the indexes to values */
+ for (j = 0; j < ndims; j++)
+ {
+ if (! bucket->nullsonly[j])
+ {
+ bucket->min[j] = values[j][BUCKET_MIN_INDEXES(tmp, ndims)[j]];
+ bucket->max[j] = values[j][BUCKET_MAX_INDEXES(tmp, ndims)[j]];
+ }
+ }
+
+ histogram->buckets[i] = bucket;
+
+ Assert(tmp <= (char*)data + VARSIZE_ANY(data));
+
+ tmp += bucketsize;
+ }
+
+ /* at this point we expect to match the total_length exactly */
+ Assert((tmp - VARDATA(data)) == expected_size);
+
+ return histogram;
+}
+
+/*
+ * Build the initial bucket, which will be then split into smaller
+ * buckets.
+ *
+ * TODO Add ndistinct estimation, probably the one described in "Towards
+ * Estimation Error Guarantees for Distinct Values, PODS 2000,
+ * p. 268-279" (the ones called GEE, or maybe AE).
+ *
+ * TODO The "combined" ndistinct is more likely to scale with the number
+ * of rows (in the table), because a single column behaving this
+ * way is sufficient for such behavior.
+ */
+static MVBucket
+create_initial_mv_bucket(int numrows, HeapTuple *rows, int2vector *attrs,
+ VacAttrStats **stats)
+{
+ int i;
+ int numattrs = attrs->dim1;
+ HistogramBuild data = NULL;
+
+ /* TODO allocate bucket as a single piece, including all the fields. */
+ MVBucket bucket = (MVBucket)palloc0(sizeof(MVBucketData));
+
+ Assert(numrows > 0);
+ Assert(rows != NULL);
+ Assert((numattrs >= 2) && (numattrs <= MVSTATS_MAX_DIMENSIONS));
+
+ /* allocate the per-dimension arrays */
+
+ /* flags for null-only dimensions */
+ bucket->nullsonly = (bool*)palloc0(numattrs * sizeof(bool));
+
+ /* inclusiveness boundaries - lower/upper bounds */
+ bucket->min_inclusive = (bool*)palloc0(numattrs * sizeof(bool));
+ bucket->max_inclusive = (bool*)palloc0(numattrs * sizeof(bool));
+
+ /* lower/upper boundaries */
+ bucket->min = (Datum*)palloc0(numattrs * sizeof(Datum));
+ bucket->max = (Datum*)palloc0(numattrs * sizeof(Datum));
+
+ /* build-data */
+ data = (HistogramBuild)palloc0(sizeof(HistogramBuildData));
+
+ /* number of distinct values (per dimension) */
+ data->ndistincts = (uint32*)palloc0(numattrs * sizeof(uint32));
+
+ /* all the sample rows fall into the initial bucket */
+ data->numrows = numrows;
+ data->rows = rows;
+
+ /*
+ * The initial bucket was not split at all, so we'll start with the
+ * first dimension in the next round (index = 0).
+ */
+ data->last_split_dimension = -1;
+
+ bucket->build_data = data;
+
+ /*
+ * Update the number of ndistinct combinations in the bucket (which
+ * we use when selecting bucket to partition), and then number of
+ * distinct values for each partition (which we use when choosing
+ * which dimension to split).
+ */
+ update_bucket_ndistinct(bucket, attrs, stats);
+
+ /* Update ndistinct (and also set min/max) for all dimensions. */
+ for (i = 0; i < numattrs; i++)
+ update_dimension_ndistinct(bucket, i, attrs, stats, true);
+
+ return bucket;
+}
+
+/*
+ * TODO Fix to handle arbitrarily-sized histograms (not just 2D ones)
+ * and call the right output procedures (for the particular type).
+ *
+ * TODO This should somehow fetch info about the data types, and use
+ * the appropriate output functions to print the boundary values.
+ * Right now this prints the 8B value as an integer.
+ *
+ * TODO Also, provide a special function for 2D histogram, printing
+ * a gnuplot script (with rectangles).
+ *
+ * TODO For string types (once supported) we can sort the strings first,
+ * assign them a sequence of integers and use the original values
+ * as labels.
+ */
+#ifdef MVSTATS_DEBUG
+static void
+print_mv_histogram_info(MVHistogram histogram)
+{
+ int i = 0;
+
+ elog(WARNING, "histogram nbuckets=%d", histogram->nbuckets);
+
+ for (i = 0; i < histogram->nbuckets; i++)
+ {
+ MVBucket bucket = histogram->buckets[i];
+ elog(WARNING, " bucket %d : ndistinct=%f ntuples=%d min=[%ld, %ld], max=[%ld, %ld] distinct=[%d,%d]",
+ i, bucket->ndistinct, bucket->numrows,
+ bucket->min[0], bucket->min[1], bucket->max[0], bucket->max[1],
+ bucket->ndistincts[0], bucket->ndistincts[1]);
+ }
+}
+#endif
+
+/*
+ * A very simple partitioning selection criteria - choose the bucket
+ * with the highest number of distinct values.
+ *
+ * Returns either pointer to the bucket selected to be partitioned,
+ * or NULL if there are no buckets that may be split (i.e. all buckets
+ * contain a single distinct value).
+ *
+ * TODO Consider other partitioning criteria (v-optimal, maxdiff etc.).
+ *
+ * TODO Allowing the bucket to degenerate to a single combination of
+ * values makes it rather strange MCV list. Maybe we should use
+ * higher lower boundary, or maybe make the selection criteria
+ * more complex (e.g. consider number of rows in the bucket, etc.).
+ *
+ * That however is different from buckets 'degenerated' only for
+ * some dimensions (e.g. half of them), which is perfectly
+ * appropriate for statistics on a combination of low and high
+ * cardinality columns.
+ */
+static MVBucket
+select_bucket_to_partition(int nbuckets, MVBucket * buckets)
+{
+ int i;
+ int ndistinct = 1; /* if ndistinct=1, we can't split the bucket */
+ MVBucket bucket = NULL;
+
+ for (i = 0; i < nbuckets; i++)
+ {
+ HistogramBuild data = (HistogramBuild)buckets[i]->build_data;
+ /* if the ndistinct count is higher, use this bucket */
+ if (data->ndistinct > ndistinct) {
+ bucket = buckets[i];
+ ndistinct = data->ndistinct;
+ }
+ }
+
+ /* may be NULL if there are not buckets with (ndistinct>1) */
+ return bucket;
+}
+
+/*
+ * A simple bucket partitioning implementation - splits the dimensions in
+ * a round-robin manner (considering only those with ndistinct>1). That
+ * is first a dimension 0 is split, then 1, 2, ... until reaching the
+ * end of attribute list, and then wrapping back to 0. Of course,
+ * dimensions with a single distinct value are skipped.
+ *
+ * This is essentially what Muralikrishna/DeWitt described in their SIGMOD
+ * article (M. Muralikrishna, David J. DeWitt: Equi-Depth Histograms For
+ * Estimating Selectivity Factors For Multi-Dimensional Queries. SIGMOD
+ * Conference 1988: 28-36).
+ *
+ * There are multiple histogram options, centered around the partitioning
+ * criteria, specifying both how to choose a bucket and the dimension
+ * most in need of a split. For a nice summary and general overview, see
+ * "rK-Hist : an R-Tree based histogram for multi-dimensional selectivity
+ * estimation" thesis by J. A. Lopez, Concordia University, p.34-37 (and
+ * possibly p. 32-34 for explanation of the terms).
+ *
+ * This splits the bucket by tweaking the existing one, and returning the
+ * new bucket (essentially shrinking the existing one in-place and returning
+ * the other "half" as a new bucket). The caller is responsible for adding
+ * the new bucket into the list of buckets.
+ *
+ * TODO It requires care to prevent splitting only one dimension and not
+ * splitting another one at all (which might happen easily in case of
+ * strongly dependent columns - e.g. y=x).
+ *
+ * TODO Should probably consider statistics target for the columns (e.g. to
+ * split dimensions with higher statistics target more frequently).
+ */
+static MVBucket
+partition_bucket(MVBucket bucket, int2vector *attrs,
+ VacAttrStats **stats)
+{
+ int i;
+ int dimension;
+ int numattrs = attrs->dim1;
+
+ Datum split_value;
+ MVBucket new_bucket;
+ HistogramBuild new_data;
+
+ /* needed for sort, when looking for the split value */
+ bool isNull;
+ int nvalues = 0;
+ HistogramBuild data = (HistogramBuild)bucket->build_data;
+ StdAnalyzeData * mystats = NULL;
+ ScalarItem * values = (ScalarItem*)palloc0(data->numrows * sizeof(ScalarItem));
+ SortSupportData ssup;
+
+ /* looking for the split value */
+ int ndistinct = 1; /* number of distinct values below current value */
+ int nrows = 1; /* number of rows below current value */
+
+ /* needed when splitting the values */
+ HeapTuple * oldrows = data->rows;
+ int oldnrows = data->numrows;
+
+ /*
+ * We can't split buckets with a single distinct value (this also
+ * disqualifies NULL-only dimensions). Also, there has to be multiple
+ * sample rows (otherwise, how could there be more distinct values).
+ */
+ Assert(data->ndistinct > 1);
+ Assert(data->numrows > 1);
+ Assert((numattrs >= 2) && (numattrs <= MVSTATS_MAX_DIMENSIONS));
+
+ /*
+ * Look for the next dimension to split, in a round robin manner.
+ * We'll use the first one with (ndistinct > 1).
+ *
+ * If we happen to wrap around, something clearly went wrong (we
+ * can't mess with the last_split_dimension directly, because we
+ * couldn't do this check).
+ */
+ dimension = data->last_split_dimension;
+ while (true)
+ {
+ dimension = (dimension + 1) % numattrs;
+
+ if (data->ndistincts[dimension] > 1)
+ break;
+
+ /* if we ran the previous split dimension, it's infinite loop */
+ Assert(dimension != data->last_split_dimension);
+ }
+
+ /* Remember the dimension for the next split of this bucket. */
+ data->last_split_dimension = dimension;
+
+ /*
+ * Walk through the selected dimension, collect and sort the values
+ * and then choose the value to use as the new boundary.
+ */
+ mystats = (StdAnalyzeData *) stats[dimension]->extra_data;
+
+ /* initialize sort support, etc. */
+ memset(&ssup, 0, sizeof(ssup));
+ ssup.ssup_cxt = CurrentMemoryContext;
+
+ /* We always use the default collation for statistics */
+ ssup.ssup_collation = DEFAULT_COLLATION_OID;
+ ssup.ssup_nulls_first = false;
+
+ PrepareSortSupportFromOrderingOp(mystats->ltopr, &ssup);
+
+ for (i = 0; i < data->numrows; i++)
+ {
+ /* remember the index of the sample row, to make the partitioning simpler */
+ values[nvalues].value = heap_getattr(data->rows[i], attrs->values[dimension],
+ stats[dimension]->tupDesc, &isNull);
+ values[nvalues].tupno = i;
+
+ /* no NULL values allowed here (we don't do splits by null-only dimensions) */
+ Assert(!isNull);
+
+ nvalues++;
+ }
+
+ /* sort the array of values */
+ qsort_arg((void *) values, nvalues, sizeof(ScalarItem),
+ compare_scalars_partition, (void *) &ssup);
+
+ /*
+ * We know there are bucket->ndistincts[dimension] distinct values
+ * in this dimension, and we want to split this into half, so walk
+ * through the array and stop once we see (ndistinct/2) values.
+ *
+ * We always choose the "next" value, i.e. (n/2+1)-th distinct value,
+ * and use it as an exclusive upper boundary (and inclusive lower
+ * boundary).
+ *
+ * TODO Maybe we should use "average" of the two middle distinct
+ * values (at least for even distinct counts), but that would
+ * require being able to do an average (which does not work
+ * for non-arithmetic types).
+ *
+ * TODO Another option is to look for a split that'd give about
+ * 50% tuples (not distinct values) in each partition. That
+ * might work better when there are a few very frequent
+ * values, and many rare ones.
+ */
+ split_value = values[0].value;
+ for (i = 1; i < data->numrows; i++)
+ {
+ /* count distinct values */
+ if (values[i].value != values[i-1].value)
+ ndistinct += 1;
+
+ /* once we've seen 1/2 distinct values (and use the value) */
+ if (ndistinct > data->ndistincts[dimension] / 2)
+ {
+ split_value = values[i].value;
+ break;
+ }
+
+ /* keep track how many rows belong to the first bucket */
+ nrows += 1;
+ }
+
+ Assert(nrows > 0);
+ Assert(nrows < data->numrows);
+
+ /* create the new bucket as a (incomplete) copy of the one being partitioned. */
+ new_bucket = copy_mv_bucket(bucket, numattrs);
+ new_data = (HistogramBuild)new_bucket->build_data;
+
+ /*
+ * Do the actual split of the chosen dimension, using the split value as the
+ * upper bound for the existing bucket, and lower bound for the new one.
+ */
+ bucket->max[dimension] = split_value;
+ new_bucket->min[dimension] = split_value;
+
+ bucket->max_inclusive[dimension] = false;
+ new_bucket->max_inclusive[dimension] = true;
+
+ /*
+ * Redistribute the sample tuples using the 'ScalarItem->tupno'
+ * index. We know 'nrows' rows should remain in the original
+ * bucket and the rest goes to the new one.
+ */
+
+ data->rows = (HeapTuple*)palloc0(nrows * sizeof(HeapTuple));
+ new_data->rows = (HeapTuple*)palloc0((oldnrows - nrows) * sizeof(HeapTuple));
+
+ data->numrows = nrows;
+ new_data->numrows = (oldnrows - nrows);
+
+ /*
+ * The first nrows should go to the first bucket, the rest should
+ * go to the new one. Use the tupno field to get the actual HeapTuple
+ * row from the original array of sample rows.
+ */
+ for (i = 0; i < nrows; i++)
+ memcpy(&data->rows[i], &oldrows[values[i].tupno], sizeof(HeapTuple));
+
+ for (i = nrows; i < oldnrows; i++)
+ memcpy(&new_data->rows[i-nrows], &oldrows[values[i].tupno], sizeof(HeapTuple));
+
+ /* update ndistinct values for the buckets (total and per dimension) */
+ update_bucket_ndistinct(bucket, attrs, stats);
+ update_bucket_ndistinct(new_bucket, attrs, stats);
+
+ /*
+ * TODO We don't need to do this for the dimension we used for split,
+ * because we know how many distinct values went to each partition.
+ */
+ for (i = 0; i < numattrs; i++)
+ {
+ update_dimension_ndistinct(bucket, i, attrs, stats, false);
+ update_dimension_ndistinct(new_bucket, i, attrs, stats, false);
+ }
+
+ pfree(oldrows);
+ pfree(values);
+
+ return new_bucket;
+}
+
+/*
+ * Copy a histogram bucket. The copy does not include the build-time
+ * data, i.e. sampled rows etc.
+ */
+static MVBucket
+copy_mv_bucket(MVBucket bucket, uint32 ndimensions)
+{
+ /* TODO allocate as a single piece (including all the fields) */
+ MVBucket new_bucket = (MVBucket)palloc0(sizeof(MVBucketData));
+ HistogramBuild data = (HistogramBuild)palloc0(sizeof(HistogramBuildData));
+
+ /* Copy only the attributes that will stay the same after the split, and
+ * we'll recompute the rest after the split. */
+
+ /* allocate the per-dimension arrays */
+ new_bucket->nullsonly = (bool*)palloc0(ndimensions * sizeof(bool));
+
+ /* inclusiveness boundaries - lower/upper bounds */
+ new_bucket->min_inclusive = (bool*)palloc0(ndimensions * sizeof(bool));
+ new_bucket->max_inclusive = (bool*)palloc0(ndimensions * sizeof(bool));
+
+ /* lower/upper boundaries */
+ new_bucket->min = (Datum*)palloc0(ndimensions * sizeof(Datum));
+ new_bucket->max = (Datum*)palloc0(ndimensions * sizeof(Datum));
+
+ /* copy data */
+ memcpy(new_bucket->nullsonly, bucket->nullsonly, ndimensions * sizeof(bool));
+
+ memcpy(new_bucket->min_inclusive, bucket->min_inclusive, ndimensions*sizeof(bool));
+ memcpy(new_bucket->min, bucket->min, ndimensions*sizeof(Datum));
+
+ memcpy(new_bucket->max_inclusive, bucket->max_inclusive, ndimensions*sizeof(bool));
+ memcpy(new_bucket->max, bucket->max, ndimensions*sizeof(Datum));
+
+ /* allocate and copy the interesting part of the build data */
+ data->last_split_dimension = ((HistogramBuild)bucket->build_data)->last_split_dimension;
+ data->ndistincts = (uint32*)palloc0(ndimensions * sizeof(uint32));
+
+ new_bucket->build_data = data;
+
+ return new_bucket;
+}
+
+/*
+ * Counts the number of distinct values in the bucket. This just copies
+ * the Datum values into a simple array, and sorts them using memcmp-based
+ * comparator. That means it only works for pass-by-value data types
+ * (assuming they don't use collations etc.)
+ *
+ * TODO This might evaluate and store the distinct counts for all
+ * possible attribute combinations. The assumption is this might be
+ * useful for estimating things like GROUP BY cardinalities (e.g.
+ * in cases when some buckets contain a lot of low-frequency
+ * combinations, and other buckets contain few high-frequency ones).
+ *
+ * But it's unclear whether it's worth the price. Computing this
+ * is actually quite cheap, because it may be evaluated at the very
+ * end, when the buckets are rather small (so sorting it in 2^N ways
+ * is not a big deal). Assuming the partitioning algorithm does not
+ * use these values to do the decisions, of course (the current
+ * algorithm does not).
+ *
+ * The overhead with storing, fetching and parsing the data is more
+ * concerning - adding 2^N values per bucket (even if it's just
+ * a 1B or 2B value) would significantly bloat the histogram, and
+ * thus the impact on optimizer. Which is not really desirable.
+ *
+ * TODO This only updates the ndistinct for the sample (or bucket), but
+ * we eventually need an estimate of the total number of distinct
+ * values in the dataset. It's possible to either use the current
+ * 1D approach (i.e., if it's more than 10% of the sample, assume
+ * it's proportional to the number of rows). Or it's possible to
+ * implement the estimator suggested in the article, supposedly
+ * giving 'optimal' estimates (w.r.t. probability of error).
+ */
+static void
+update_bucket_ndistinct(MVBucket bucket, int2vector *attrs, VacAttrStats ** stats)
+{
+ int i, j;
+ int numattrs = attrs->dim1;
+
+ HistogramBuild data = (HistogramBuild)bucket->build_data;
+ int numrows = data->numrows;
+
+ MultiSortSupport mss = multi_sort_init(numattrs);
+
+ /*
+ * We could collect this while walking through all the attributes
+ * above (this way we have to call heap_getattr twice).
+ */
+ SortItem *items = (SortItem*)palloc0(numrows * sizeof(SortItem));
+ Datum *values = (Datum*)palloc0(numrows * sizeof(Datum) * numattrs);
+ bool *isnull = (bool*)palloc0(numrows * sizeof(bool) * numattrs);
+
+ for (i = 0; i < numrows; i++)
+ {
+ items[i].values = &values[i * numattrs];
+ items[i].isnull = &isnull[i * numattrs];
+ }
+
+ /* prepare the sort function for the first dimension */
+ for (i = 0; i < numattrs; i++)
+ multi_sort_add_dimension(mss, i, i, stats);
+
+ /* collect the values */
+ for (i = 0; i < numrows; i++)
+ for (j = 0; j < numattrs; j++)
+ items[i].values[j]
+ = heap_getattr(data->rows[i], attrs->values[j],
+ stats[j]->tupDesc, &items[i].isnull[j]);
+
+ qsort_arg((void *) items, numrows, sizeof(SortItem),
+ multi_sort_compare, mss);
+
+ data->ndistinct = 1;
+
+ for (i = 1; i < numrows; i++)
+ if (multi_sort_compare(&items[i], &items[i-1], mss) != 0)
+ data->ndistinct += 1;
+
+ pfree(items);
+ pfree(values);
+ pfree(isnull);
+}
+
+/*
+ * Count distinct values per bucket dimension.
+ */
+static void
+update_dimension_ndistinct(MVBucket bucket, int dimension, int2vector *attrs,
+ VacAttrStats ** stats, bool update_boundaries)
+{
+ int j;
+ int nvalues = 0;
+ bool isNull;
+ HistogramBuild data = (HistogramBuild)bucket->build_data;
+ Datum * values = (Datum*)palloc0(data->numrows * sizeof(Datum));
+ SortSupportData ssup;
+
+ StdAnalyzeData * mystats = (StdAnalyzeData *) stats[dimension]->extra_data;
+
+ /* we may already know this is a NULL-only dimension */
+ if (bucket->nullsonly[dimension])
+ data->ndistincts[dimension] = 1;
+
+ memset(&ssup, 0, sizeof(ssup));
+ ssup.ssup_cxt = CurrentMemoryContext;
+
+ /* We always use the default collation for statistics */
+ ssup.ssup_collation = DEFAULT_COLLATION_OID;
+ ssup.ssup_nulls_first = false;
+
+ PrepareSortSupportFromOrderingOp(mystats->ltopr, &ssup);
+
+ for (j = 0; j < data->numrows; j++)
+ {
+ values[nvalues] = heap_getattr(data->rows[j], attrs->values[dimension],
+ stats[dimension]->tupDesc, &isNull);
+
+ /* ignore NULL values */
+ if (! isNull)
+ nvalues++;
+ }
+
+ /* there's always at least 1 distinct value (may be NULL) */
+ data->ndistincts[dimension] = 1;
+
+ /* if there are only NULL values in the column, mark it so and continue
+ * with the next one */
+ if (nvalues == 0)
+ {
+ pfree(values);
+ bucket->nullsonly[dimension] = true;
+ return;
+ }
+
+ /* sort the array (pass-by-value datum */
+ qsort_arg((void *) values, nvalues, sizeof(Datum),
+ compare_scalars_simple, (void *) &ssup);
+
+ /*
+ * Update min/max boundaries to the smallest bounding box. Generally, this
+ * needs to be done only when constructing the initial bucket.
+ */
+ if (update_boundaries)
+ {
+ /* store the min/max values */
+ bucket->min[dimension] = values[0];
+ bucket->min_inclusive[dimension] = true;
+
+ bucket->max[dimension] = values[nvalues-1];
+ bucket->max_inclusive[dimension] = true;
+ }
+
+ /*
+ * Walk through the array and count distinct values by comparing
+ * succeeding values.
+ *
+ * FIXME This only works for pass-by-value types (i.e. not VARCHARs
+ * etc.). Although thanks to the deduplication it might work
+ * even for those types (equal values will get the same item
+ * in the deduplicated array).
+ */
+ for (j = 1; j < nvalues; j++) {
+ if (values[j] != values[j-1])
+ data->ndistincts[dimension] += 1;
+ }
+
+ pfree(values);
+}
+
+/*
+ * A properly built histogram must not contain buckets mixing NULL and
+ * non-NULL values in a single dimension. Each dimension may either be
+ * marked as 'nulls only', and thus containing only NULL values, or
+ * it must not contain any NULL values.
+ *
+ * Therefore, if the sample contains NULL values in any of the columns,
+ * it's necessary to build those NULL-buckets. This is done in an
+ * iterative way using this algorithm, operating on a single bucket:
+ *
+ * (1) Check that all dimensions are well-formed (not mixing NULL
+ * and non-NULL values).
+ *
+ * (2) If all dimensions are well-formed, terminate.
+ *
+ * (3) If the dimension contains only NULL values, but is not
+ * marked as NULL-only, mark it as NULL-only and run the
+ * algorithm again (on this bucket).
+ *
+ * (4) If the dimension mixes NULL and non-NULL values, split the
+ * bucket into two parts - one with NULL values, one with
+ * non-NULL values (replacing the current one). Then run
+ * the algorithm on both buckets.
+ *
+ * This is executed in a recursive manner, but the number of executions
+ * should be quite low - limited by the number of NULL-buckets. Also,
+ * in each branch the number of nested calls is limited by the number
+ * of dimensions (attributes) of the histogram.
+ *
+ * At the end, there should be buckets with no mixed dimensions. The
+ * number of buckets produced by this algorithm is rather limited - with
+ * N dimensions, there may be only 2^N such buckets (each dimension may
+ * be either NULL or non-NULL). So with 8 dimensions (current value of
+ * MVSTATS_MAX_DIMENSIONS) there may be only 256 such buckets.
+ *
+ * After this, a 'regular' bucket-split algorithm shall run, further
+ * optimizing the histogram.
+ */
+static void
+create_null_buckets(MVHistogram histogram, int bucket_idx,
+ int2vector *attrs, VacAttrStats ** stats)
+{
+ int i, j;
+ int null_dim = -1;
+ int null_count = 0;
+ bool null_found = false;
+ MVBucket bucket, null_bucket;
+ int null_idx, curr_idx;
+ HistogramBuild data, null_data;
+
+ /* remember original values from the bucket */
+ int numrows;
+ HeapTuple *oldrows = NULL;
+
+ Assert(bucket_idx < histogram->nbuckets);
+ Assert(histogram->ndimensions == attrs->dim1);
+
+ bucket = histogram->buckets[bucket_idx];
+ data = (HistogramBuild)bucket->build_data;
+
+ numrows = data->numrows;
+ oldrows = data->rows;
+
+ /*
+ * Walk through all rows / dimensions, and stop once we find NULL
+ * in a dimension not yet marked as NULL-only.
+ */
+ for (i = 0; i < data->numrows; i++)
+ {
+ for (j = 0; j < histogram->ndimensions; j++)
+ {
+ /* Is this a NULL-only dimension? If yes, skip. */
+ if (bucket->nullsonly[j])
+ continue;
+
+ /* found a NULL in that dimension? */
+ if (heap_attisnull(data->rows[i], attrs->values[j]))
+ {
+ null_found = true;
+ null_dim = j;
+ break;
+ }
+ }
+
+ /* terminate if we found attribute with NULL values */
+ if (null_found)
+ break;
+ }
+
+ /* no regular dimension contains NULL values => we're done */
+ if (! null_found)
+ return;
+
+ /* walk through the rows again, count NULL values in 'null_dim' */
+ for (i = 0; i < data->numrows; i++)
+ {
+ if (heap_attisnull(data->rows[i], attrs->values[null_dim]))
+ null_count += 1;
+ }
+
+ Assert(null_count <= data->numrows);
+
+ /*
+ * If (null_count == numrows) the dimension already is NULL-only,
+ * but is not yet marked like that. It's enough to mark it and
+ * repeat the process recursively (until we run out of dimensions).
+ */
+ if (null_count == data->numrows)
+ {
+ bucket->nullsonly[null_dim] = true;
+ create_null_buckets(histogram, bucket_idx, attrs, stats);
+ return;
+ }
+
+ /*
+ * We have to split the bucket into two - one with NULL values in
+ * the dimension, one with non-NULL values. We don't need to sort
+ * the data or anything, but otherwise it's similar to what's done
+ * in partition_bucket().
+ */
+
+ /* create bucket with NULL-only dimension 'dim' */
+ null_bucket = copy_mv_bucket(bucket, histogram->ndimensions);
+ null_data = (HistogramBuild)null_bucket->build_data;
+
+ /* remember the current array info */
+ oldrows = data->rows;
+ numrows = data->numrows;
+
+ /* we'll keep non-NULL values in the current bucket */
+ data->numrows = (numrows - null_count);
+ data->rows
+ = (HeapTuple*)palloc0(data->numrows * sizeof(HeapTuple));
+
+ /* and the NULL values will go to the new one */
+ null_data->numrows = null_count;
+ null_data->rows
+ = (HeapTuple*)palloc0(null_data->numrows * sizeof(HeapTuple));
+
+ /* mark the dimension as NULL-only (in the new bucket) */
+ null_bucket->nullsonly[null_dim] = true;
+
+ /* walk through the sample rows and distribute them accordingly */
+ null_idx = 0;
+ curr_idx = 0;
+ for (i = 0; i < numrows; i++)
+ {
+ if (heap_attisnull(oldrows[i], attrs->values[null_dim]))
+ /* NULL => copy to the new bucket */
+ memcpy(&null_data->rows[null_idx++], &oldrows[i],
+ sizeof(HeapTuple));
+ else
+ memcpy(&data->rows[curr_idx++], &oldrows[i],
+ sizeof(HeapTuple));
+ }
+
+ /* update ndistinct values for the buckets (total and per dimension) */
+ update_bucket_ndistinct(bucket, attrs, stats);
+ update_bucket_ndistinct(null_bucket, attrs, stats);
+
+ /*
+ * TODO We don't need to do this for the dimension we used for split,
+ * because we know how many distinct values went to each
+ * bucket (NULL is not a value, so 0, and the other bucket got
+ * all the ndistinct values).
+ */
+ for (i = 0; i < histogram->ndimensions; i++)
+ {
+ update_dimension_ndistinct(bucket, i, attrs, stats, false);
+ update_dimension_ndistinct(null_bucket, i, attrs, stats, false);
+ }
+
+ pfree(oldrows);
+
+ /* add the NULL bucket to the histogram */
+ histogram->buckets[histogram->nbuckets++] = null_bucket;
+
+ /*
+ * And now run the function recursively on both buckets (the new
+ * one first, because the call may change number of buckets, and
+ * it's used as an index).
+ */
+ create_null_buckets(histogram, (histogram->nbuckets-1), attrs, stats);
+ create_null_buckets(histogram, bucket_idx, attrs, stats);
+
+}
+
+/*
+ * We need to pass the SortSupport to the comparator, but bsearch()
+ * has no 'context' parameter, so we use a global variable (ugly).
+ */
+static int
+bsearch_comparator(const void * a, const void * b)
+{
+ Assert(ssup_private != NULL);
+ return compare_scalars_simple(a, b, (void*)ssup_private);
+}
diff --git a/src/backend/utils/mvstats/mcv.c b/src/backend/utils/mvstats/mcv.c
index 4466cee..4f60bd1 100644
--- a/src/backend/utils/mvstats/mcv.c
+++ b/src/backend/utils/mvstats/mcv.c
@@ -961,6 +961,7 @@ MCVList deserialize_mv_mcvlist(bytea * data)
for (i = 0; i < nitems; i++)
{
+ /* FIXME allocate as a single chunk (minimize palloc overhead) */
MCVItem item = (MCVItem)palloc0(sizeof(MCVItemData));
item->values = (Datum*)palloc0(sizeof(Datum)*ndims);
diff --git a/src/include/catalog/pg_mv_statistic.h b/src/include/catalog/pg_mv_statistic.h
index c6e7d74..84579da 100644
--- a/src/include/catalog/pg_mv_statistic.h
+++ b/src/include/catalog/pg_mv_statistic.h
@@ -36,13 +36,16 @@ CATALOG(pg_mv_statistic,3381)
/* statistics requested to build */
bool deps_enabled; /* analyze dependencies? */
bool mcv_enabled; /* build MCV list? */
+ bool hist_enabled; /* build histogram? */
- /* MCV size */
+ /* histogram / MCV size */
int32 mcv_max_items; /* max MCV items */
+ int32 hist_max_buckets; /* max histogram buckets */
/* statistics that are available (if requested) */
bool deps_built; /* dependencies were built */
bool mcv_built; /* MCV list was built */
+ bool hist_built; /* histogram was built */
/* variable-length fields start here, but we allow direct access to stakeys */
int2vector stakeys; /* array of column keys */
@@ -50,6 +53,7 @@ CATALOG(pg_mv_statistic,3381)
#ifdef CATALOG_VARLEN
bytea stadeps; /* dependencies (serialized) */
bytea stamcv; /* MCV list (serialized) */
+ bytea stahist; /* MV histogram (serialized) */
#endif
} FormData_pg_mv_statistic;
@@ -65,15 +69,19 @@ typedef FormData_pg_mv_statistic *Form_pg_mv_statistic;
* compiler constants for pg_attrdef
* ----------------
*/
-#define Natts_pg_mv_statistic 9
+#define Natts_pg_mv_statistic 13
#define Anum_pg_mv_statistic_starelid 1
#define Anum_pg_mv_statistic_deps_enabled 2
#define Anum_pg_mv_statistic_mcv_enabled 3
-#define Anum_pg_mv_statistic_mcv_max_items 4
-#define Anum_pg_mv_statistic_deps_built 5
-#define Anum_pg_mv_statistic_mcv_built 6
-#define Anum_pg_mv_statistic_stakeys 7
-#define Anum_pg_mv_statistic_stadeps 8
-#define Anum_pg_mv_statistic_stamcv 9
+#define Anum_pg_mv_statistic_hist_enabled 4
+#define Anum_pg_mv_statistic_mcv_max_items 5
+#define Anum_pg_mv_statistic_hist_max_buckets 6
+#define Anum_pg_mv_statistic_deps_built 7
+#define Anum_pg_mv_statistic_mcv_built 8
+#define Anum_pg_mv_statistic_hist_built 9
+#define Anum_pg_mv_statistic_stakeys 10
+#define Anum_pg_mv_statistic_stadeps 11
+#define Anum_pg_mv_statistic_stamcv 12
+#define Anum_pg_mv_statistic_stahist 13
#endif /* PG_MV_STATISTIC_H */
diff --git a/src/include/catalog/pg_proc.h b/src/include/catalog/pg_proc.h
index b2aa815..a1b5e2b 100644
--- a/src/include/catalog/pg_proc.h
+++ b/src/include/catalog/pg_proc.h
@@ -2718,6 +2718,8 @@ DATA(insert OID = 3378 ( pg_mv_stats_dependencies_show PGNSP PGUID 12 1 0 0
DESCR("multivariate stats: functional dependencies show");
DATA(insert OID = 3376 ( pg_mv_stats_mcvlist_info PGNSP PGUID 12 1 0 0 0 f f f f t f i 1 0 25 "17" _null_ _null_ _null_ _null_ pg_mv_stats_mcvlist_info _null_ _null_ _null_ ));
DESCR("multi-variate statistics: MCV list info");
+DATA(insert OID = 3375 ( pg_mv_stats_histogram_info PGNSP PGUID 12 1 0 0 0 f f f f t f i 1 0 25 "17" _null_ _null_ _null_ _null_ pg_mv_stats_histogram_info _null_ _null_ _null_ ));
+DESCR("multi-variate statistics: histogram info");
DATA(insert OID = 1928 ( pg_stat_get_numscans PGNSP PGUID 12 1 0 0 0 f f f f t f s 1 0 20 "26" _null_ _null_ _null_ _null_ pg_stat_get_numscans _null_ _null_ _null_ ));
DESCR("statistics: number of scans done for table/index");
diff --git a/src/include/utils/mvstats.h b/src/include/utils/mvstats.h
index 6ff29d6..673e546 100644
--- a/src/include/utils/mvstats.h
+++ b/src/include/utils/mvstats.h
@@ -26,10 +26,12 @@ typedef struct MVStatsData {
/* statistics requested in ALTER TABLE ... ADD STATISTICS */
bool deps_enabled; /* analyze functional dependencies */
bool mcv_enabled; /* analyze MCV lists */
+ bool hist_enabled; /* analyze histogram */
/* available statistics (computed by ANALYZE) */
bool deps_built; /* functional dependencies available */
bool mcv_built; /* MCV list is already available */
+ bool hist_built; /* histogram is already available */
} MVStatsData;
typedef struct MVStatsData *MVStats;
@@ -109,6 +111,68 @@ typedef MCVListData *MCVList;
#define MVSTAT_MCVLIST_MAX_ITEMS 8192 /* max items in MCV list */
/*
+ * Multivariate histograms
+ */
+typedef struct MVBucketData {
+
+ /* Frequencies of this bucket. */
+ float ntuples; /* frequency of tuples tuples */
+
+ /*
+ * Information about dimensions being NULL-only. Not yet used.
+ */
+ bool *nullsonly;
+
+ /* lower boundaries - values and information about the inequalities */
+ Datum *min;
+ bool *min_inclusive;
+
+ /* upper boundaries - values and information about the inequalities */
+ Datum *max;
+ bool *max_inclusive;
+
+ /* used when building the histogram (not serialized/deserialized) */
+ void *build_data;
+
+} MVBucketData;
+
+typedef MVBucketData *MVBucket;
+
+
+typedef struct MVHistogramData {
+
+ uint32 magic; /* magic constant marker */
+ uint32 type; /* type of histogram (BASIC) */
+ uint32 nbuckets; /* number of buckets (buckets array) */
+ uint32 ndimensions; /* number of dimensions */
+
+ MVBucket *buckets; /* array of buckets */
+
+} MVHistogramData;
+
+typedef MVHistogramData *MVHistogram;
+
+/* used to flag stats serialized to bytea */
+#define MVSTAT_HIST_MAGIC 0x7F8C5670 /* marks serialized bytea */
+#define MVSTAT_HIST_TYPE_BASIC 1 /* basic histogram type */
+
+/*
+ * Limits used for max_buckets option, i.e. we're always guaranteed
+ * to have space for at least MVSTAT_HIST_MIN_BUCKETS, and we cannot
+ * have more than MVSTAT_HIST_MAX_BUCKETS buckets.
+ *
+ * This is just a boundary for the 'max' threshold - the actual
+ * histogram may use less buckets than MVSTAT_HIST_MAX_BUCKETS.
+ *
+ * TODO The MVSTAT_HIST_MIN_BUCKETS should be related to the number of
+ * attributes (MVSTATS_MAX_DIMENSIONS) because of NULL-buckets.
+ * There should be at least 2^N buckets, otherwise we may be unable
+ * to build the NULL buckets.
+ */
+#define MVSTAT_HIST_MIN_BUCKETS 128 /* min number of buckets */
+#define MVSTAT_HIST_MAX_BUCKETS 16384 /* max number of buckets */
+
+/*
* TODO Maybe fetching the histogram/MCV list separately is inefficient?
* Consider adding a single `fetch_stats` method, fetching all
* stats specified using flags (or something like that).
@@ -118,14 +182,18 @@ bytea * fetch_mv_rules(Oid mvoid);
bytea * fetch_mv_dependencies(Oid mvoid);
bytea * fetch_mv_mcvlist(Oid mvoid);
+bytea * fetch_mv_histogram(Oid mvoid);
bytea * serialize_mv_dependencies(MVDependencies dependencies);
bytea * serialize_mv_mcvlist(MCVList mcvlist, int2vector *attrs,
VacAttrStats **stats);
+bytea * serialize_mv_histogram(MVHistogram histogram, int2vector *attrs,
+ VacAttrStats **stats);
/* deserialization of stats (serialization is private to analyze) */
MVDependencies deserialize_mv_dependencies(bytea * data);
MCVList deserialize_mv_mcvlist(bytea * data);
+MVHistogram deserialize_mv_histogram(bytea * data);
/*
* Returns index of the attribute number within the vector (i.e. a
@@ -137,6 +205,7 @@ int mv_get_index(AttrNumber varattno, int2vector * stakeys);
extern Datum pg_mv_stats_dependencies_info(PG_FUNCTION_ARGS);
extern Datum pg_mv_stats_dependencies_show(PG_FUNCTION_ARGS);
extern Datum pg_mv_stats_mcvlist_info(PG_FUNCTION_ARGS);
+extern Datum pg_mv_stats_histogram_info(PG_FUNCTION_ARGS);
MVDependencies
build_mv_dependencies(int numrows, HeapTuple *rows, int2vector *attrs,
@@ -146,10 +215,15 @@ MCVList
build_mv_mcvlist(int numrows, HeapTuple *rows, int2vector *attrs,
VacAttrStats **stats, int *numrows_filtered);
+MVHistogram
+build_mv_histogram(int numrows, HeapTuple *rows, int2vector *attrs,
+ VacAttrStats **stats, int numrows_total);
+
void build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
int natts, VacAttrStats **vacattrstats);
-void update_mv_stats(Oid relid, MVDependencies dependencies, MCVList mcvlist,
+void update_mv_stats(Oid relid, MVDependencies dependencies,
+ MCVList mcvlist, MVHistogram histogram,
int2vector *attrs, VacAttrStats **stats);
#endif
diff --git a/src/test/regress/expected/mv_histogram.out b/src/test/regress/expected/mv_histogram.out
new file mode 100644
index 0000000..ff2f0cc
--- /dev/null
+++ b/src/test/regress/expected/mv_histogram.out
@@ -0,0 +1,210 @@
+-- data type passed by value
+CREATE TABLE mv_histogram (
+ a INT,
+ b INT,
+ c INT
+);
+-- unknown column
+ALTER TABLE mv_histogram ADD STATISTICS (histogram) ON (unknown_column);
+ERROR: column "unknown_column" referenced in statistics does not exist
+-- single column
+ALTER TABLE mv_histogram ADD STATISTICS (histogram) ON (a);
+ERROR: multivariate stats require 2 or more columns
+-- single column, duplicated
+ALTER TABLE mv_histogram ADD STATISTICS (histogram) ON (a, a);
+ERROR: duplicate column name in statistics definition
+-- two columns, one duplicated
+ALTER TABLE mv_histogram ADD STATISTICS (histogram) ON (a, a, b);
+ERROR: duplicate column name in statistics definition
+-- unknown option
+ALTER TABLE mv_histogram ADD STATISTICS (unknown_option) ON (a, b, c);
+ERROR: unrecognized STATISTICS option "unknown_option"
+-- missing histogram statistics
+ALTER TABLE mv_histogram ADD STATISTICS (dependencies, max_buckets 200) ON (a, b, c);
+ERROR: option 'histogram' is required by other options(s)
+-- invalid max_buckets value / too low
+ALTER TABLE mv_histogram ADD STATISTICS (mcv, max_buckets 10) ON (a, b, c);
+ERROR: minimum number of buckets is 128
+-- invalid max_buckets value / too high
+ALTER TABLE mv_histogram ADD STATISTICS (mcv, max_buckets 100000) ON (a, b, c);
+ERROR: minimum number of buckets is 16384
+-- correct command
+ALTER TABLE mv_histogram ADD STATISTICS (histogram) ON (a, b, c);
+-- random data (no functional dependencies)
+INSERT INTO mv_histogram
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+ANALYZE mv_histogram;
+SELECT hist_enabled, hist_built, pg_mv_stats_histogram_info(stahist)
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+ hist_enabled | hist_built | pg_mv_stats_histogram_info
+--------------+------------+----------------------------
+ t | t | nbuckets=10000
+(1 row)
+
+TRUNCATE mv_histogram;
+-- a => b, a => c, b => c
+INSERT INTO mv_histogram
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE mv_histogram;
+SELECT hist_enabled, hist_built, pg_mv_stats_histogram_info(stahist)
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+ hist_enabled | hist_built | pg_mv_stats_histogram_info
+--------------+------------+----------------------------
+ t | t | nbuckets=1001
+(1 row)
+
+TRUNCATE mv_histogram;
+-- a => b, a => c
+INSERT INTO mv_histogram
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE mv_histogram;
+SELECT hist_enabled, hist_built, pg_mv_stats_histogram_info(stahist)
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+ hist_enabled | hist_built | pg_mv_stats_histogram_info
+--------------+------------+----------------------------
+ t | t | nbuckets=1001
+(1 row)
+
+TRUNCATE mv_histogram;
+-- check explain (expect bitmap index scan, not plain index scan)
+INSERT INTO mv_histogram
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX hist_idx ON mv_histogram (a, b);
+ANALYZE mv_histogram;
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+ hist_enabled | hist_built
+--------------+------------
+ t | t
+(1 row)
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mv_histogram WHERE a = 10 AND b = 5;
+ QUERY PLAN
+--------------------------------------------
+ Bitmap Heap Scan on mv_histogram
+ Recheck Cond: ((a = 10) AND (b = 5))
+ -> Bitmap Index Scan on hist_idx
+ Index Cond: ((a = 10) AND (b = 5))
+(4 rows)
+
+DELETE FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+DROP TABLE mv_histogram;
+-- varlena type (text)
+CREATE TABLE mv_histogram (
+ a TEXT,
+ b TEXT,
+ c TEXT
+);
+ALTER TABLE mv_histogram ADD STATISTICS (histogram) ON (a, b, c);
+-- random data (no functional dependencies)
+INSERT INTO mv_histogram
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+ANALYZE mv_histogram;
+SELECT hist_enabled, hist_built, pg_mv_stats_histogram_info(stahist)
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+ hist_enabled | hist_built | pg_mv_stats_histogram_info
+--------------+------------+----------------------------
+ t | t | nbuckets=10000
+(1 row)
+
+TRUNCATE mv_histogram;
+-- a => b, a => c, b => c
+INSERT INTO mv_histogram
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE mv_histogram;
+SELECT hist_enabled, hist_built, pg_mv_stats_histogram_info(stahist)
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+ hist_enabled | hist_built | pg_mv_stats_histogram_info
+--------------+------------+----------------------------
+ t | t | nbuckets=3492
+(1 row)
+
+TRUNCATE mv_histogram;
+-- a => b, a => c
+INSERT INTO mv_histogram
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE mv_histogram;
+SELECT hist_enabled, hist_built, pg_mv_stats_histogram_info(stahist)
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+ hist_enabled | hist_built | pg_mv_stats_histogram_info
+--------------+------------+----------------------------
+ t | t | nbuckets=3433
+(1 row)
+
+TRUNCATE mv_histogram;
+-- check explain (expect bitmap index scan, not plain index scan)
+INSERT INTO mv_histogram
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX hist_idx ON mv_histogram (a, b);
+ANALYZE mv_histogram;
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+ hist_enabled | hist_built
+--------------+------------
+ t | t
+(1 row)
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mv_histogram WHERE a = '10' AND b = '5';
+ QUERY PLAN
+------------------------------------------------------------
+ Bitmap Heap Scan on mv_histogram
+ Recheck Cond: ((a = '10'::text) AND (b = '5'::text))
+ -> Bitmap Index Scan on hist_idx
+ Index Cond: ((a = '10'::text) AND (b = '5'::text))
+(4 rows)
+
+TRUNCATE mv_histogram;
+-- check explain (expect bitmap index scan, not plain index scan) with NULLs
+INSERT INTO mv_histogram
+ SELECT
+ (CASE WHEN i/10000 = 0 THEN NULL ELSE i/10000 END),
+ (CASE WHEN i/20000 = 0 THEN NULL ELSE i/20000 END),
+ (CASE WHEN i/40000 = 0 THEN NULL ELSE i/40000 END)
+ FROM generate_series(1,1000000) s(i);
+ANALYZE mv_histogram;
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+ hist_enabled | hist_built
+--------------+------------
+ t | t
+(1 row)
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mv_histogram WHERE a IS NULL AND b IS NULL;
+ QUERY PLAN
+---------------------------------------------------
+ Bitmap Heap Scan on mv_histogram
+ Recheck Cond: ((a IS NULL) AND (b IS NULL))
+ -> Bitmap Index Scan on hist_idx
+ Index Cond: ((a IS NULL) AND (b IS NULL))
+(4 rows)
+
+DELETE FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+DROP TABLE mv_histogram;
+-- NULL values (mix of int and text columns)
+CREATE TABLE mv_histogram (
+ a INT,
+ b TEXT,
+ c INT,
+ d TEXT
+);
+ALTER TABLE mv_histogram ADD STATISTICS (histogram) ON (a, b, c, d);
+INSERT INTO mv_histogram
+ SELECT
+ mod(i, 100),
+ (CASE WHEN mod(i, 200) = 0 THEN NULL ELSE mod(i,200) END),
+ mod(i, 400),
+ (CASE WHEN mod(i, 300) = 0 THEN NULL ELSE mod(i,600) END)
+ FROM generate_series(1,10000) s(i);
+ANALYZE mv_histogram;
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+ hist_enabled | hist_built
+--------------+------------
+ t | t
+(1 row)
+
+DELETE FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+DROP TABLE mv_histogram;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 6d9ab2f..ccc778a 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1359,7 +1359,9 @@ pg_mv_stats| SELECT n.nspname AS schemaname,
length(s.stadeps) AS depsbytes,
pg_mv_stats_dependencies_info(s.stadeps) AS depsinfo,
length(s.stamcv) AS mcvbytes,
- pg_mv_stats_mcvlist_info(s.stamcv) AS mcvinfo
+ pg_mv_stats_mcvlist_info(s.stamcv) AS mcvinfo,
+ length(s.stahist) AS histbytes,
+ pg_mv_stats_histogram_info(s.stahist) AS histinfo
FROM ((pg_mv_statistic s
JOIN pg_class c ON ((c.oid = s.starelid)))
LEFT JOIN pg_namespace n ON ((n.oid = c.relnamespace)));
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index 63727a4..aeb89f8 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -111,4 +111,4 @@ test: event_trigger
test: stats
# run tests of multivariate stats
-test: mv_dependencies mv_mcv
+test: mv_dependencies mv_mcv mv_histogram
diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule
index 5b07b3b..ee1468d 100644
--- a/src/test/regress/serial_schedule
+++ b/src/test/regress/serial_schedule
@@ -155,3 +155,4 @@ test: event_trigger
test: stats
test: mv_dependencies
test: mv_mcv
+test: mv_histogram
diff --git a/src/test/regress/sql/mv_histogram.sql b/src/test/regress/sql/mv_histogram.sql
new file mode 100644
index 0000000..78890c8
--- /dev/null
+++ b/src/test/regress/sql/mv_histogram.sql
@@ -0,0 +1,179 @@
+-- data type passed by value
+CREATE TABLE mv_histogram (
+ a INT,
+ b INT,
+ c INT
+);
+
+-- unknown column
+ALTER TABLE mv_histogram ADD STATISTICS (histogram) ON (unknown_column);
+
+-- single column
+ALTER TABLE mv_histogram ADD STATISTICS (histogram) ON (a);
+
+-- single column, duplicated
+ALTER TABLE mv_histogram ADD STATISTICS (histogram) ON (a, a);
+
+-- two columns, one duplicated
+ALTER TABLE mv_histogram ADD STATISTICS (histogram) ON (a, a, b);
+
+-- unknown option
+ALTER TABLE mv_histogram ADD STATISTICS (unknown_option) ON (a, b, c);
+
+-- missing histogram statistics
+ALTER TABLE mv_histogram ADD STATISTICS (dependencies, max_buckets 200) ON (a, b, c);
+
+-- invalid max_buckets value / too low
+ALTER TABLE mv_histogram ADD STATISTICS (mcv, max_buckets 10) ON (a, b, c);
+
+-- invalid max_buckets value / too high
+ALTER TABLE mv_histogram ADD STATISTICS (mcv, max_buckets 100000) ON (a, b, c);
+
+-- correct command
+ALTER TABLE mv_histogram ADD STATISTICS (histogram) ON (a, b, c);
+
+-- random data (no functional dependencies)
+INSERT INTO mv_histogram
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+
+ANALYZE mv_histogram;
+
+SELECT hist_enabled, hist_built, pg_mv_stats_histogram_info(stahist)
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+
+TRUNCATE mv_histogram;
+
+-- a => b, a => c, b => c
+INSERT INTO mv_histogram
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+
+ANALYZE mv_histogram;
+
+SELECT hist_enabled, hist_built, pg_mv_stats_histogram_info(stahist)
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+
+TRUNCATE mv_histogram;
+
+-- a => b, a => c
+INSERT INTO mv_histogram
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE mv_histogram;
+
+SELECT hist_enabled, hist_built, pg_mv_stats_histogram_info(stahist)
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+
+TRUNCATE mv_histogram;
+
+-- check explain (expect bitmap index scan, not plain index scan)
+INSERT INTO mv_histogram
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX hist_idx ON mv_histogram (a, b);
+ANALYZE mv_histogram;
+
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mv_histogram WHERE a = 10 AND b = 5;
+
+DELETE FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+DROP TABLE mv_histogram;
+
+-- varlena type (text)
+CREATE TABLE mv_histogram (
+ a TEXT,
+ b TEXT,
+ c TEXT
+);
+
+ALTER TABLE mv_histogram ADD STATISTICS (histogram) ON (a, b, c);
+
+-- random data (no functional dependencies)
+INSERT INTO mv_histogram
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+
+ANALYZE mv_histogram;
+
+SELECT hist_enabled, hist_built, pg_mv_stats_histogram_info(stahist)
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+
+TRUNCATE mv_histogram;
+
+-- a => b, a => c, b => c
+INSERT INTO mv_histogram
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+
+ANALYZE mv_histogram;
+
+SELECT hist_enabled, hist_built, pg_mv_stats_histogram_info(stahist)
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+
+TRUNCATE mv_histogram;
+
+-- a => b, a => c
+INSERT INTO mv_histogram
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE mv_histogram;
+
+SELECT hist_enabled, hist_built, pg_mv_stats_histogram_info(stahist)
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+
+TRUNCATE mv_histogram;
+
+-- check explain (expect bitmap index scan, not plain index scan)
+INSERT INTO mv_histogram
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX hist_idx ON mv_histogram (a, b);
+ANALYZE mv_histogram;
+
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mv_histogram WHERE a = '10' AND b = '5';
+
+TRUNCATE mv_histogram;
+
+-- check explain (expect bitmap index scan, not plain index scan) with NULLs
+INSERT INTO mv_histogram
+ SELECT
+ (CASE WHEN i/10000 = 0 THEN NULL ELSE i/10000 END),
+ (CASE WHEN i/20000 = 0 THEN NULL ELSE i/20000 END),
+ (CASE WHEN i/40000 = 0 THEN NULL ELSE i/40000 END)
+ FROM generate_series(1,1000000) s(i);
+ANALYZE mv_histogram;
+
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mv_histogram WHERE a IS NULL AND b IS NULL;
+
+DELETE FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+DROP TABLE mv_histogram;
+
+-- NULL values (mix of int and text columns)
+CREATE TABLE mv_histogram (
+ a INT,
+ b TEXT,
+ c INT,
+ d TEXT
+);
+
+ALTER TABLE mv_histogram ADD STATISTICS (histogram) ON (a, b, c, d);
+
+INSERT INTO mv_histogram
+ SELECT
+ mod(i, 100),
+ (CASE WHEN mod(i, 200) = 0 THEN NULL ELSE mod(i,200) END),
+ mod(i, 400),
+ (CASE WHEN mod(i, 300) = 0 THEN NULL ELSE mod(i,600) END)
+ FROM generate_series(1,10000) s(i);
+
+ANALYZE mv_histogram;
+
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+
+DELETE FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+DROP TABLE mv_histogram;
--
2.0.5
[text/x-diff] 0005-multi-statistics-estimation.patch (71.2K, ../../5519E9B2.5080105@2ndquadrant.com/6-0005-multi-statistics-estimation.patch)
download | inline diff:
From db24cc534985ce97b238ea539b4216d8e33397a5 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@pgaddict.com>
Date: Fri, 6 Feb 2015 01:42:38 +0100
Subject: [PATCH 5/5] multi-statistics estimation
The general idea is that a probability (which
is what selectivity is) can be split into a product of
conditional probabilities like this:
P(A & B & C) = P(A & B) * P(C|A & B)
If we assume that C and B are independent, the last part
may be simplified like this
P(A & B & C) = P(A & B) * P(C|A)
we only need probabilities on [A,B] and [C,A] to compute
the original probability.
The implementation works in the other direction, though.
We know what probability P(A & B & C) we need to compute,
and also what statistics are available.
So we search for a combinations of statistics, covering
the clauses in an optimal way (most clauses covered, most
dependencies exploited).
There are two possible approaches - exhaustive and greedy.
The exhaustive one walks through all permutations of
stats using dynamic programming, so it's guaranteed to
find the optimal solution, but it soon gets very slow as
it's roughly O(N!). The dynamic programming may improve
that a bit, but it's still far too expensive for large
numbers of statistics (on a single table).
The greedy algorithm is very simple - in every step choose
the best solution. That may not guarantee the best solution
globally (but maybe it does?), but it only needs N steps
to find the solution, so it's very fast (processing the
selected stats is usually way more expensive).
There's a GUC for selecting the search algorithm
mvstat_search = {'greedy', 'exhaustive'}
The default value is 'greedy' as that's much safer (with
respect to runtime). See choose_mv_statistics().
Once we have found a sequence of statistics, we apply
them to the clauses using the conditional probabilities.
We process the selected stats one by one, and for each
we select the estimated clauses and conditions. See
clauselist_selectivity() for more details.
Limitations
-----------
It's still true that each clause at a given level has to
be covered by a single MV statistics. So with this query
WHERE (clause1) AND (clause2) AND (clause3 OR clause4)
each parenthesized clause has to be covered by a single
multivariate statistics.
Clauses not covered by a single statistics at this level
will be passed to clause_selectivity() but this will treat
them as a collection of simpler clauses (connected by AND
or OR), and the clauses from the previous level will be
used as conditions.
So using the same example, the last clause will be passed
to clause_selectivity() with 'clause1' and 'clause2' as
conditions, and it will be processed using multivariate
stats if possible.
The other limitation is that all the expressions have to
be mv-compatible, i.e. there can't be a mix of expressions.
Fixing this should be relatively simple - just split the
list into two parts (mv-compatible/incompatible), as at
the top level.
---
src/backend/optimizer/path/clausesel.c | 1533 ++++++++++++++++++++++++++++++--
src/backend/optimizer/path/costsize.c | 23 +-
src/backend/optimizer/util/orclauses.c | 4 +-
src/backend/utils/adt/selfuncs.c | 17 +-
src/backend/utils/misc/guc.c | 20 +
src/include/optimizer/cost.h | 6 +-
src/include/utils/mvstats.h | 8 +
7 files changed, 1513 insertions(+), 98 deletions(-)
diff --git a/src/backend/optimizer/path/clausesel.c b/src/backend/optimizer/path/clausesel.c
index ea4d588..98ad802 100644
--- a/src/backend/optimizer/path/clausesel.c
+++ b/src/backend/optimizer/path/clausesel.c
@@ -30,7 +30,7 @@
#include "utils/typcache.h"
#include "parser/parsetree.h"
-
+#include "miscadmin.h"
#include <stdio.h>
@@ -63,23 +63,25 @@ static Bitmapset *collect_mv_attnums(PlannerInfo *root, List *clauses,
Oid varRelid, Oid *relid, SpecialJoinInfo *sjinfo,
int type);
+static Bitmapset *clause_mv_get_attnums(PlannerInfo *root, Node *clause);
+
static List *clauselist_apply_dependencies(PlannerInfo *root, List *clauses,
Oid varRelid, int nmvstats, MVStats mvstats,
SpecialJoinInfo *sjinfo);
-static int choose_mv_statistics(int nmvstats, MVStats mvstats,
- Bitmapset *attnums);
static List *clauselist_mv_split(PlannerInfo *root, SpecialJoinInfo *sjinfo,
List *clauses, Oid varRelid,
List **mvclauses, MVStats mvstats, int types);
static Selectivity clauselist_mv_selectivity(PlannerInfo *root,
- List *clauses, MVStats mvstats);
+ MVStats mvstats, List *clauses, List *conditions);
static Selectivity clauselist_mv_selectivity_mcvlist(PlannerInfo *root,
- List *clauses, MVStats mvstats,
+ MVStats mvstats,
+ List *clauses, List *conditions,
bool *fullmatch, Selectivity *lowsel);
static Selectivity clauselist_mv_selectivity_histogram(PlannerInfo *root,
- List *clauses, MVStats mvstats);
+ MVStats mvstats,
+ List *clauses, List *conditions);
static int update_match_bitmap_mcvlist(PlannerInfo *root, List *clauses,
int2vector *stakeys, MCVList mcvlist,
@@ -92,6 +94,31 @@ static int update_match_bitmap_histogram(PlannerInfo *root, List *clauses,
int nmatches, char * matches,
bool is_or);
+/*
+ * Describes a combination of multiple statistics to cover attributes
+ * referenced by the clauses. The array 'stats' (with nstats elements)
+ * lists attributes (in the order as they are applied), and number of
+ * clause attributes covered by this solution.
+ *
+ * choose_mv_statistics_exhaustive() uses this to track both the current
+ * and the best solutions, while walking through the state of possible
+ * combination.
+ */
+typedef struct mv_solution_t {
+ int nclauses; /* number of clauses covered */
+ int nconditions; /* number of conditions covered */
+ int nstats; /* number of stats applied */
+ int *stats; /* stats (in the apply order) */
+} mv_solution_t;
+
+static mv_solution_t *choose_mv_statistics(PlannerInfo *root,
+ int nmvstats, MVStats mvstats,
+ List *clauses, List *conditions,
+ Oid varRelid,
+ SpecialJoinInfo *sjinfo, int type);
+
+int mvstat_search_type = MVSTAT_SEARCH_GREEDY;
+
/* used for merging bitmaps - AND (min), OR (max) */
#define MAX(x, y) (((x) > (y)) ? (x) : (y))
#define MIN(x, y) (((x) < (y)) ? (x) : (y))
@@ -220,7 +247,8 @@ clauselist_selectivity(PlannerInfo *root,
List *clauses,
int varRelid,
JoinType jointype,
- SpecialJoinInfo *sjinfo)
+ SpecialJoinInfo *sjinfo,
+ List *conditions)
{
Selectivity s1 = 1.0;
RangeQueryClause *rqlist = NULL;
@@ -234,13 +262,8 @@ clauselist_selectivity(PlannerInfo *root,
/* attributes in mv-compatible clauses */
Bitmapset *mvattnums = NULL;
- /*
- * If there's exactly one clause, then no use in trying to match up
- * pairs, so just go directly to clause_selectivity().
- */
- if (list_length(clauses) == 1)
- return clause_selectivity(root, (Node *) linitial(clauses),
- varRelid, jointype, sjinfo);
+ /* local conditions, accumulated and passed to clauses in this list */
+ List *conditions_local = list_copy(conditions);
/*
* Collect attributes referenced by mv-compatible clauses (looking
@@ -288,30 +311,185 @@ clauselist_selectivity(PlannerInfo *root,
/* see choose_mv_statistics() for details */
if (nmvstats > 0)
{
- int idx = choose_mv_statistics(nmvstats, mvstats, mvattnums);
+ mv_solution_t * solution
+ = choose_mv_statistics(root, nmvstats, mvstats,
+ clauses, conditions,
+ varRelid, sjinfo,
+ (MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST));
- if (idx >= 0) /* we have a matching stats */
+ /*
+ * FIXME This probaly leaks memory a bit - the lists of clauses
+ * should be freed properly.
+ */
+
+ /* we have a good solution stats */
+ if (solution != NULL)
{
- MVStats mvstat = &mvstats[idx];
+ int i, j, k;
+
+ for (i = 0; i < solution->nstats; i++)
+ {
+ /* clauses compatible with multi-variate stats */
+ List *mvclauses = NIL;
+ List *mvclauses_new = NIL;
+ List *mvclauses_conditions = NIL;
+ Bitmapset *stat_attnums = NULL;
+
+ MVStats mvstat = &mvstats[solution->stats[i]];
+
+ /* build attnum bitmapset for this statistics */
+ for (k = 0; k < mvstat->stakeys->dim1; k++)
+ stat_attnums = bms_add_member(stat_attnums,
+ mvstat->stakeys->values[k]);
+
+ /*
+ * Append the compatible conditions (passed from above)
+ * to mvclauses_conditions.
+ */
+ foreach (l, conditions)
+ {
+ Node *c = (Node*)lfirst(l);
+ Bitmapset *tmp = clause_mv_get_attnums(root, c);
- /* clauses compatible with multi-variate stats */
- List *mvclauses = NIL;
+ if (bms_is_subset(tmp, stat_attnums))
+ mvclauses_conditions
+ = lappend(mvclauses_conditions, c);
+
+ bms_free(tmp);
+ }
+
+ /* split the clauselist into regular and mv-clauses
+ *
+ * We keep the list of clauses (we don't remove the
+ * clauses yet, because we want to use the clauses
+ * as conditions of other clauses).
+ *
+ * FIXME Do this only once, i.e. filter the clauses
+ * once (selecting clauses covered by at least
+ * one statistics) and then convert them into
+ * smaller per-statistics lists of conditions
+ * and estimated clauses.
+ */
+ clauselist_mv_split(root, sjinfo, clauses,
+ varRelid, &mvclauses, mvstat,
+ (MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST));
+
+ /*
+ * We've chosen the statistics to match the clauses, so
+ * each statistics from the solution should have at least
+ * one new clause (not covered by the previous stats).
+ */
+ Assert(mvclauses != NIL);
+
+ /*
+ * Mvclauses now contains only clauses compatible
+ * with the currently selected stats, but we have to
+ * split that into conditions (already matched by
+ * the previous stats), and the new clauses we need
+ * to estimate using this stats.
+ */
+ foreach (l, mvclauses)
+ {
+ bool covered = false;
+ Node *clause = (Node *) lfirst(l);
+ Bitmapset *clause_attnums = clause_mv_get_attnums(root, clause);
+
+ /*
+ * If already covered by previous stats, add it to
+ * conditions.
+ *
+ * TODO Maybe this could be relaxed a bit? Because
+ * with complex and/or clauses, this might
+ * mean no statistics actually covers such
+ * complex clause.
+ */
+ for (j = 0; j < i; j++)
+ {
+ int k;
+ Bitmapset *stat_attnums = NULL;
+ MVStats prev_stat = &mvstats[solution->stats[j]];
+
+ for (k = 0; k < prev_stat->stakeys->dim1; k++)
+ stat_attnums = bms_add_member(stat_attnums,
+ prev_stat->stakeys->values[k]);
+
+ covered = bms_is_subset(clause_attnums, stat_attnums);
+
+ bms_free(stat_attnums);
+
+ if (covered)
+ break;
+ }
+
+ if (covered)
+ mvclauses_conditions
+ = lappend(mvclauses_conditions, clause);
+ else
+ mvclauses_new
+ = lappend(mvclauses_new, clause);
+ }
+
+ /*
+ * We need at least one new clause (not just conditions).
+ */
+ Assert(mvclauses_new != NIL);
+
+ /* compute the multivariate stats */
+ s1 *= clauselist_mv_selectivity(root, mvstat,
+ mvclauses_new,
+ mvclauses_conditions);
+ }
+
+ /*
+ * And now finally remove all the mv-compatible clauses.
+ *
+ * This only repeats the same split as above, but this
+ * time we actually use the result list (and feed it to
+ * the next call).
+ */
+ for (i = 0; i < solution->nstats; i++)
+ {
+ /* clauses compatible with multi-variate stats */
+ List *mvclauses = NIL;
- /* split the clauselist into regular and mv-clauses */
- clauses = clauselist_mv_split(root, sjinfo, clauses,
- varRelid, &mvclauses, mvstat,
- (MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST));
+ MVStats mvstat = &mvstats[solution->stats[i]];
- /* we've chosen the histogram to match the clauses */
- Assert(mvclauses != NIL);
+ /* split the list into regular and mv-clauses */
+ clauses = clauselist_mv_split(root, sjinfo, clauses,
+ varRelid, &mvclauses, mvstat,
+ (MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST));
- /* compute the multivariate stats */
- s1 *= clauselist_mv_selectivity(root, mvclauses, mvstat);
+ /*
+ * Add the clauses to the conditions (to be passed
+ * to regular clauses), irrespectedly whether it
+ * will be used as a condition or a clause here.
+ *
+ * We only keep the remaining conditions in the
+ * clauses (we keep what clauselist_mv_split returns)
+ * so we add each MV condition exactly once.
+ */
+ foreach (l, mvclauses)
+ conditions_local = lappend(conditions_local,
+ (Node*)lfirst(l));
+ }
}
}
}
/*
+ * If there's exactly one clause, then no use in trying to match up
+ * pairs, so just go directly to clause_selectivity().
+ */
+ if (list_length(clauses) == 1)
+ {
+ Selectivity s = clause_selectivity(root, (Node *) linitial(clauses),
+ varRelid, jointype, sjinfo,
+ conditions_local);
+ list_free(conditions_local);
+ return s;
+ }
+
+ /*
* Initial scan over clauses. Anything that doesn't look like a potential
* rangequery clause gets multiplied into s1 and forgotten. Anything that
* does gets inserted into an rqlist entry.
@@ -323,7 +501,8 @@ clauselist_selectivity(PlannerInfo *root,
Selectivity s2;
/* Always compute the selectivity using clause_selectivity */
- s2 = clause_selectivity(root, clause, varRelid, jointype, sjinfo);
+ s2 = clause_selectivity(root, clause, varRelid, jointype, sjinfo,
+ conditions_local);
/*
* Check for being passed a RestrictInfo.
@@ -478,6 +657,9 @@ clauselist_selectivity(PlannerInfo *root,
rqlist = rqnext;
}
+ /* free the local conditions */
+ list_free(conditions_local);
+
return s1;
}
@@ -688,7 +870,8 @@ clause_selectivity(PlannerInfo *root,
Node *clause,
int varRelid,
JoinType jointype,
- SpecialJoinInfo *sjinfo)
+ SpecialJoinInfo *sjinfo,
+ List *conditions)
{
Selectivity s1 = 0.5; /* default for any unhandled clause type */
RestrictInfo *rinfo = NULL;
@@ -818,7 +1001,8 @@ clause_selectivity(PlannerInfo *root,
(Node *) get_notclausearg((Expr *) clause),
varRelid,
jointype,
- sjinfo);
+ sjinfo,
+ conditions);
}
else if (and_clause(clause))
{
@@ -827,7 +1011,8 @@ clause_selectivity(PlannerInfo *root,
((BoolExpr *) clause)->args,
varRelid,
jointype,
- sjinfo);
+ sjinfo,
+ conditions);
}
else if (or_clause(clause))
{
@@ -839,6 +1024,13 @@ clause_selectivity(PlannerInfo *root,
*/
ListCell *arg;
+ /* TODO Split the clause list into mv-compatible part, pretty
+ * much just like in clauselist_selectivity(), and call
+ * clauselist_mv_selectivity(). It has to be taught about
+ * OR-semantics (right now it assumes AND) or maybe just
+ * create a fake OR clause here, and pass it in.
+ */
+
s1 = 0.0;
foreach(arg, ((BoolExpr *) clause)->args)
{
@@ -846,7 +1038,8 @@ clause_selectivity(PlannerInfo *root,
(Node *) lfirst(arg),
varRelid,
jointype,
- sjinfo);
+ sjinfo,
+ conditions);
s1 = s1 + s2 - s1 * s2;
}
@@ -958,7 +1151,8 @@ clause_selectivity(PlannerInfo *root,
(Node *) ((RelabelType *) clause)->arg,
varRelid,
jointype,
- sjinfo);
+ sjinfo,
+ conditions);
}
else if (IsA(clause, CoerceToDomain))
{
@@ -967,7 +1161,8 @@ clause_selectivity(PlannerInfo *root,
(Node *) ((CoerceToDomain *) clause)->arg,
varRelid,
jointype,
- sjinfo);
+ sjinfo,
+ conditions);
}
/* Cache the result if possible */
@@ -1149,9 +1344,67 @@ clause_selectivity(PlannerInfo *root,
* that from the most selective clauses first, because that'll
* eliminate the buckets/items sooner (so we'll be able to skip
* them without inspection, which is more expensive).
+ *
+ * TODO All this is based on the assumption that the statistics represent
+ * the necessary dependencies, i.e. that if two colunms are not in
+ * the same statistics, there's no dependency. If that's not the
+ * case, we may get misestimates, just like before. For example
+ * assume we have a table with three columns [a,b,c] with exactly
+ * the same values, and statistics on [a,b] and [b,c]. So somthing
+ * like this:
+ *
+ * CREATE TABLE test AS SELECT i, i, i
+ FROM generate_series(1,1000);
+ *
+ * ALTER TABLE test ADD STATISTICS (mcv) ON (a,b);
+ * ALTER TABLE test ADD STATISTICS (mcv) ON (b,c);
+ *
+ * ANALYZE test;
+ *
+ * EXPLAIN ANALYZE SELECT * FROM test
+ * WHERE (a < 10) AND (b < 20) AND (c < 10);
+ *
+ * The problem here is that the only shared column between the two
+ * statistics is 'b' so the probability will be computed like this
+ *
+ * P[(a < 10) & (b < 20) & (c < 10)]
+ * = P[(a < 10) & (b < 20)] * P[(c < 10) | (a < 10) & (b < 20)]
+ * = P[(a < 10) & (b < 20)] * P[(c < 10) | (b < 20)]
+ *
+ * or like this
+ *
+ * P[(a < 10) & (b < 20) & (c < 10)]
+ * = P[(b < 20) & (c < 10)] * P[(a < 10) | (b < 20) & (c < 10)]
+ * = P[(b < 20) & (c < 10)] * P[(a < 10) | (b < 20)]
+ *
+ * In both cases the conditional probabilities will be evaluated as
+ * 0.5, because they lack the other column (which would make it 1.0).
+ *
+ * Theoretically it might be possible to transfer the dependency,
+ * e.g. by building bitmap for [a,b] and then combine it with [b,c]
+ * by doing something like this:
+ *
+ * 1) build bitmap on [a,b] using [(a<10) & (b < 20)]
+ * 2) for each element in [b,c] check the bitmap
+ *
+ * But that's certainly nontrivial - for example the statistics may
+ * be different (MCV list vs. histogram) and/or the items may not
+ * match (e.g. MCV items or histogram buckets will be built
+ * differently). Also, for one value of 'b' there might be multiple
+ * MCV items (because of the other column values) with different
+ * bitmap values (some will match, some won't) - so it's not exactly
+ * bitmap but a partial match.
+ *
+ * Maybe a hash table with number of matches and mismatches (or
+ * maybe sums of frequencies) would work? The step (2) would then
+ * lookup the values and use that to weight the item somehow.
+ *
+ * Currently the only solution is to build statistics on all three
+ * columns.
*/
static Selectivity
-clauselist_mv_selectivity(PlannerInfo *root, List *clauses, MVStats mvstats)
+clauselist_mv_selectivity(PlannerInfo *root, MVStats mvstats,
+ List *clauses, List *conditions)
{
bool fullmatch = false;
Selectivity s1 = 0.0, s2 = 0.0;
@@ -1169,7 +1422,8 @@ clauselist_mv_selectivity(PlannerInfo *root, List *clauses, MVStats mvstats)
*/
/* Evaluate the MCV first. */
- s1 = clauselist_mv_selectivity_mcvlist(root, clauses, mvstats,
+ s1 = clauselist_mv_selectivity_mcvlist(root, mvstats,
+ clauses, conditions,
&fullmatch, &mcv_low);
/*
@@ -1182,7 +1436,8 @@ clauselist_mv_selectivity(PlannerInfo *root, List *clauses, MVStats mvstats)
/* FIXME if (fullmatch) without matching MCV item, use the mcv_low
* selectivity as upper bound */
- s2 = clauselist_mv_selectivity_histogram(root, clauses, mvstats);
+ s2 = clauselist_mv_selectivity_histogram(root, mvstats,
+ clauses, conditions);
/* TODO clamp to <= 1.0 (or more strictly, when possible) */
return s1 + s2;
@@ -1232,6 +1487,665 @@ collect_mv_attnums(PlannerInfo *root, List *clauses, Oid varRelid,
}
/*
+ * Selects the best combination of multivariate statistics, where
+ * 'best' means:
+ *
+ * (a) covering the most attributes (referenced by clauses)
+ * (b) using the least number of multivariate stats
+ *
+ * There may be other optimality criteria, not considered in the initial
+ * implementation (more on that 'weaknesses' section).
+ *
+ * This is pretty much equal to splitting the probability of clauses
+ * (aka selectivity) into a sequence of conditional probabilities, like
+ * this
+ *
+ * P(A,B,C,D) = P(A,B) * P(C|A,B) * P(D|A,B,C)
+ *
+ * and removing the attributes not referenced by the existing stats,
+ * under the assumption that there's no dependency (otherwise the DBA
+ * would create the stats).
+ *
+ *
+ * Algorithm
+ * ---------
+ * The algorithm is a recursive implementation of backtracking, with
+ * maximum 'depth' equal to the number of multi-variate statistics
+ * available on the table.
+ *
+ * It explores all the possible permutations of the stats.
+ *
+ * Whenever it considers adding the next statistics, the clauses it
+ * matches are divided into 'conditions' (clauses already matched by at
+ * least one previous statistics) and clauses that are estimated.
+ *
+ * Then several checks are performed:
+ *
+ * (a) The statistics covers at least 2 columns, referenced in the
+ * estimated clauses (otherwise multi-variate stats are useless).
+ *
+ * (b) The statistics covers at least 1 new column, i.e. column not
+ * refefenced by the already used stats (and the new column has
+ * to be referenced by the clauses, of couse). Otherwise the
+ * statistics would not add any new information.
+ *
+ * There are some other sanity checks (e.g. that the stats must not be
+ * used twice etc.).
+ *
+ * Finally the new solution is compared to the currently best one, and
+ * if it's considered better, it's used instead.
+ *
+ *
+ * Weaknesses
+ * ----------
+ * The current implemetation uses a somewhat simple optimality criteria,
+ * suffering by the following weaknesses.
+ *
+ * (a) There may be multiple solutions with the same number of covered
+ * attributes and number of statistics (e.g. the same solution but
+ * with statistics in a different order). It's unclear which solution
+ * is the best one - in a sense all of them are equal.
+ *
+ * TODO It might be possible to compute estimate for each of those
+ * solutions, and then combine them to get the final estimate
+ * (e.g. by using average or median).
+ *
+ * (b) Does not consider that some types of stats are a better match for
+ * some types of clauses (e.g. MCV list is a good match for equality
+ * than a histogram).
+ *
+ * XXX Maybe MCV is almost always better / more accurate?
+ *
+ * But maybe this is pointless - generally, each column is either
+ * a label (it's not important whether because of the data type or
+ * how it's used), or a value with ordering that makes sense. So
+ * either a MCV list is more appropriate (labels) or a histogram
+ * (values with orderings).
+ *
+ * Now sure what to do with statistics on columns mixing columns of
+ * both types - maybe it'd be beeter to invent a new type of stats
+ * combining MCV list and histogram (keeping a small histogram for
+ * each MCV item, and a separate histogram for values not on the
+ * MCV list). But that's not implemented at this moment.
+ *
+ * (c) Does not consider that some solutions may better exploit the
+ * dependencies. For example with clauses on columns [A,B,C,D] and
+ * statistics on [A,B,C] and [C,D] cover all the columns just like
+ * [A,B,C] and [B,C,D], but the latter probably exploits additional
+ * dependencies thanks to having 'B' in both stats (thus allowing
+ * using it as a condition for the second stats). Of course, if
+ * B and [C,D] are independent, this is untrue - but if we have that
+ * statistics created, it's a sign that the DBA/developer believes
+ * there's a dependency.
+ *
+ * (d) Does not consider the order of clauses, which may be significant.
+ * For example, when there's a mix of simple and complex clauses,
+ * i.e. something like
+ *
+ * (a=2) AND (b=3 OR (c=3 AND d=4)) AND (c=3)
+ *
+ * It may be better to evaluate the simple clauses first, and then
+ * use them as conditions for the complex clause.
+ *
+ * We can for example count number of different attributes
+ * referenced in the clause, and use that as a metric of complexity
+ * (lower number -> simpler). Maybe use ratio (#vars/#atts) or
+ * (#clauses/#atts) as secondary metrics? Also the general complexity
+ * of the clause (levels of nesting etc.) might be useful.
+ *
+ * Hopefully most clauses will be reasonably simple, though.
+ *
+ * Update: On second thought, I believe the order of clauses is
+ * determined by choosing the order of statistics, and therefore
+ * optimized by the current algorithm.
+ *
+ * TODO Consider adding a counter of attributes covered by previous
+ * stats (possibly tracking the number of how many stats reference
+ * it too), and use this 'dependency_count' when selecting the best
+ * solution (not sure how). Similarly to (a) it might be possible
+ * to build estimate for each solution (different criteria) and then
+ * combine them somehow.
+ *
+ * TODO The current implementation repeatedly walks through the previous
+ * stats, just to compute the number of covered attributes over and
+ * over. With non-trivial number of statistics this might be an
+ * issue, so maybe we should keep track of 'covered' attributes by
+ * each step, so that we can get rid of this. We'll need this
+ * information anyway (when splitting clauses into condition and
+ * the estimated part).
+ *
+ * TODO This needs to consider the conditions passed from the preceding
+ * and upper clauses (in complex cases), but only as conditions
+ * and not as estimated clauses. So it needs to somehow affect the
+ * score (the more conditions we use the better).
+ *
+ * TODO The algorithm should probably count number of Vars (not just
+ * attnums) when computing the 'score' of each solution. Computing
+ * the ratio of (num of all vars) / (num of condition vars) as a
+ * measure of how well the solution uses conditions might be
+ * useful.
+ *
+ * TODO This might be much easier if we kept Bitmapset of attributes
+ * covered by the stats up to that step.
+ *
+ * FIXME When comparing the solutions, we currently use this condition:
+ *
+ * ((current->nstats > (*best)->nstats))
+ *
+ * i.e. we're choosing solution with more stats, because with
+ * clauses
+ *
+ * (a = 1) AND (b = 1) AND (c = 1) AND (d = 1)
+ *
+ * and stats on [a,b], [b,c], [c,d] we want to choose the solution
+ * with all three stats, and not just [a,b], [c,d]. Otherwise we'd
+ * fail to exploit one of the dependencies.
+ *
+ * This is however a workaround for another issue - we're not
+ * tracking number of 'dependencies' covered by the solution, only
+ * number of clauses, and that's the same for both solutions.
+ * ([a,b], [c,d]) and ([a,b], [b,c], [c,d]) both cover all 4 clauses.
+ *
+ * Once a suitable metric is added, we want to choose the solution
+ * with less stats, assuming it covers the same number of clauses
+ * and exploits the same number of dependencies.
+ */
+static void
+choose_mv_statistics_exhaustive(PlannerInfo *root, int step,
+ int nmvstats, MVStats mvstats, Bitmapset ** stats_attnums,
+ int nclauses, Node ** clauses, Bitmapset ** clauses_attnums,
+ int nconditions, Node ** conditions, Bitmapset ** conditions_attnums,
+ bool *cover_map, bool *condition_map, int *ruled_out,
+ mv_solution_t *current, mv_solution_t **best)
+{
+ int i, j;
+
+ Assert(best != NULL);
+ Assert((step == 0 && current == NULL) || (step > 0 && current != NULL));
+
+ CHECK_FOR_INTERRUPTS();
+
+ if (current == NULL)
+ {
+ current = (mv_solution_t*)palloc0(sizeof(mv_solution_t));
+ current->stats = (int*)palloc0(sizeof(int)*nmvstats);
+ current->nstats = 0;
+ current->nclauses = 0;
+ current->nconditions = 0;
+ }
+
+ /*
+ * Now try to apply each statistics, matching at least two attributes,
+ * unless it's already used in one of the previous steps.
+ */
+ for (i = 0; i < nmvstats; i++)
+ {
+ int c;
+
+ int ncovered_clauses = 0; /* number of covered clauses */
+ int ncovered_conditions = 0; /* number of covered conditions */
+ int nattnums = 0; /* number of covered attributes */
+
+ Bitmapset *all_attnums = NULL;
+ Bitmapset *new_attnums = NULL;
+
+ /* skip statistics that were already used or eliminated */
+ if (ruled_out[i] != -1)
+ continue;
+
+ /*
+ * See if we have clauses covered by this statistics, but not
+ * yet covered by any of the preceding onces.
+ */
+ for (c = 0; c < nclauses; c++)
+ {
+ bool covered = false;
+ Bitmapset *clause_attnums = clauses_attnums[c];
+ Bitmapset *tmp = NULL;
+
+ /*
+ * If this clause is not covered by this stats, we can't
+ * use the stats to estimate that at all.
+ */
+ if (! cover_map[i * nclauses + c])
+ continue;
+
+ /*
+ * Now we know we'll use this clause - either as a condition
+ * or as a new clause (the estimated one). So let's add the
+ * attributes to the attnums from all the clauses usable with
+ * this statistics.
+ */
+ tmp = bms_union(all_attnums, clause_attnums);
+
+ /* free the old bitmap */
+ bms_free(all_attnums);
+ all_attnums = tmp;
+
+ /* let's see if it's covered by any of the previous stats */
+ for (j = 0; j < step; j++)
+ {
+ /* already covered by the previous stats */
+ if (cover_map[current->stats[j] * nclauses + c])
+ covered = true;
+
+ if (covered)
+ break;
+ }
+
+ /* if already covered, continue with the next clause */
+ if (covered)
+ {
+ ncovered_conditions += 1;
+ continue;
+ }
+
+ /*
+ * OK, this clause is covered by this statistics (and not by
+ * any of the previous ones)
+ */
+ ncovered_clauses += 1;
+
+ /* add the attnums into attnums from 'new clauses' */
+ // new_attnums = bms_union(new_attnums, clause_attnums);
+ }
+
+ /* can't have more new clauses than original clauses */
+ Assert(nclauses >= ncovered_clauses);
+ Assert(ncovered_clauses >= 0); /* mostly paranoia */
+
+ nattnums = bms_num_members(all_attnums);
+
+ /* free all the bitmapsets - we don't need them anymore */
+ bms_free(all_attnums);
+ bms_free(new_attnums);
+
+ all_attnums = NULL;
+ new_attnums = NULL;
+
+ /*
+ * See if we have clauses covered by this statistics, but not
+ * yet covered by any of the preceding onces.
+ */
+ for (c = 0; c < nconditions; c++)
+ {
+ Bitmapset *clause_attnums = conditions_attnums[c];
+ Bitmapset *tmp = NULL;
+
+ /*
+ * If this clause is not covered by this stats, we can't
+ * use the stats to estimate that at all.
+ */
+ if (! condition_map[i * nconditions + c])
+ continue;
+
+ /* count this as a condition */
+ ncovered_conditions += 1;
+
+ /*
+ * Now we know we'll use this clause - either as a condition
+ * or as a new clause (the estimated one). So let's add the
+ * attributes to the attnums from all the clauses usable with
+ * this statistics.
+ */
+ tmp = bms_union(all_attnums, clause_attnums);
+
+ /* free the old bitmap */
+ bms_free(all_attnums);
+ all_attnums = tmp;
+ }
+
+ /*
+ * Let's mark the statistics as 'ruled out' - either we'll use
+ * it (and proceed to the next step), or it's incompatible.
+ */
+ ruled_out[i] = step;
+
+ /*
+ * There are no clauses usable with this statistics (not already
+ * covered by aome of the previous stats).
+ *
+ * Similarly, if the clauses only use a single attribute, we
+ * can't really use that.
+ */
+ if ((ncovered_clauses == 0) || (nattnums < 2))
+ continue;
+
+ /*
+ * TODO Not sure if it's possible to add a clause referencing
+ * only attributes already covered by previous stats?
+ * Introducing only some new dependency, not a new
+ * attribute. Couldn't come up with an example, though.
+ * Might be worth adding some assert.
+ */
+
+ /*
+ * got a suitable statistics - let's update the current solution,
+ * maybe use it as the best solution
+ */
+ current->nclauses += ncovered_clauses;
+ current->nconditions += ncovered_conditions;
+ current->nstats += 1;
+ current->stats[step] = i;
+
+ /*
+ * We can never cover more clauses, or use more stats that we
+ * actually have at the beginning.
+ */
+ Assert(nclauses >= current->nclauses);
+ Assert(nmvstats >= current->nstats);
+ Assert(step < nmvstats);
+
+ /* we can't get more conditions that clauses and conditions combined
+ *
+ * FIXME This assert does not work because we count the conditions
+ * repeatedly (once for each statistics covering it).
+ */
+ /* Assert((nconditions + nclauses) >= current->nconditions); */
+
+ if (*best == NULL)
+ {
+ *best = (mv_solution_t*)palloc0(sizeof(mv_solution_t));
+ (*best)->stats = (int*)palloc0(sizeof(int)*nmvstats);
+ (*best)->nstats = 0;
+ (*best)->nclauses = 0;
+ (*best)->nconditions = 0;
+ }
+
+ /* see if it's better than the current 'best' solution */
+ if ((current->nclauses > (*best)->nclauses) ||
+ ((current->nclauses == (*best)->nclauses) &&
+ ((current->nstats > (*best)->nstats))))
+ {
+ (*best)->nstats = current->nstats;
+ (*best)->nclauses = current->nclauses;
+ (*best)->nconditions = current->nconditions;
+ memcpy((*best)->stats, current->stats, nmvstats * sizeof(int));
+ }
+
+ /*
+ * The recursion only makes sense if we haven't covered all the
+ * attributes (then adding stats is not really possible).
+ */
+ if ((step + 1) < nmvstats)
+ choose_mv_statistics_exhaustive(root, step+1,
+ nmvstats, mvstats, stats_attnums,
+ nclauses, clauses, clauses_attnums,
+ nconditions, conditions, conditions_attnums,
+ cover_map, condition_map, ruled_out,
+ current, best);
+
+ /* reset the last step */
+ current->nclauses -= ncovered_clauses;
+ current->nconditions -= ncovered_conditions;
+ current->nstats -= 1;
+ current->stats[step] = 0;
+
+ /* mark the statistics as usable again */
+ ruled_out[i] = -1;
+
+ Assert(current->nclauses >= 0);
+ Assert(current->nstats >= 0);
+ }
+
+ /* reset all statistics as 'incompatible' in this step */
+ for (i = 0; i < nmvstats; i++)
+ if (ruled_out[i] == step)
+ ruled_out[i] = -1;
+
+}
+
+/*
+ * Greedy search for a multivariate solution - a sequence of statistics
+ * covering the clauses. This chooses the "best" statistics at each step,
+ * so the resulting solution may not be the best solution globally, but
+ * this produces the solution in only N steps (where N is the number of
+ * statistics), while the exhaustive approach may have to walk through
+ * ~N! combinations (although some of those are terminated early).
+ *
+ * TODO There are probably other metrics we might use - e.g. using
+ * number of columns (num_cond_columns / num_cov_columns), which
+ * might work better with a mix of simple and complex clauses.
+ *
+ * TODO Also the choice at the very first step should be handled
+ * in a special way, because there will be 0 conditions at that
+ * moment, so there needs to be some other criteria - e.g. using
+ * the simplest (or most complex?) clause might be a good idea.
+ *
+ * TODO We might also select multiple stats using different criteria,
+ * and branch the search. This is however tricky, because if we
+ * choose k statistics at each step, we get k^N branches to
+ * walk through (with N steps). That's not really good with
+ * large number of stats (yet better than exhaustive search).
+ */
+static void
+choose_mv_statistics_greedy(PlannerInfo *root, int step,
+ int nmvstats, MVStats mvstats, Bitmapset ** stats_attnums,
+ int nclauses, Node ** clauses, Bitmapset ** clauses_attnums,
+ int nconditions, Node ** conditions, Bitmapset ** conditions_attnums,
+ bool *cover_map, bool *condition_map, int *ruled_out,
+ mv_solution_t *current, mv_solution_t **best)
+{
+ int i, j;
+ int best_stat = -1;
+ double gain, max_gain = -1.0;
+
+ /*
+ * Bitmap tracking which clauses are already covered (by the previous
+ * statistics) and may thus serve only as a condition in this step.
+ */
+ bool *covered_clauses = (bool*)palloc0(nclauses);
+
+ /*
+ * Number of clauses and columns covered by each statistics - this
+ * includes both conditions and clauses covered by the statistics for
+ * the first time. The number of columns may count some columns
+ * repeatedly - if a column is shared by multiple clauses, it will
+ * be counted once for each clause (covered by the statistics).
+ * So with two clauses [(a=1 OR b=2),(a<2 OR c>1)] the column "a"
+ * will be counted twice (if both clauses are covered).
+ *
+ * The values for reduded statistics (that can't be applied) are
+ * not computed, because that'd be pointless.
+ */
+ int *num_cov_clauses = (int*)palloc0(sizeof(int) * nmvstats);
+ int *num_cov_columns = (int*)palloc0(sizeof(int) * nmvstats);
+
+ /*
+ * Same as above, but this only includes clauses that are already
+ * covered by the previous stats (and the current one).
+ */
+ int *num_cond_clauses = (int*)palloc0(sizeof(int) * nmvstats);
+ int *num_cond_columns = (int*)palloc0(sizeof(int) * nmvstats);
+
+ /*
+ * Number of attributes for each clause.
+ *
+ * TODO Might be computed in choose_mv_statistics() and then passed
+ * here, but then the function would not have the same signature
+ * as _exhaustive().
+ */
+ int *attnum_counts = (int*)palloc0(sizeof(int) * nclauses);
+ int *attnum_cond_counts = (int*)palloc0(sizeof(int) * nconditions);
+
+ CHECK_FOR_INTERRUPTS();
+
+ Assert(best != NULL);
+ Assert((step == 0 && current == NULL) || (step > 0 && current != NULL));
+
+ /* compute attributes (columns) for each clause */
+ for (i = 0; i < nclauses; i++)
+ attnum_counts[i] = bms_num_members(clauses_attnums[i]);
+
+ /* compute attributes (columns) for each condition */
+ for (i = 0; i < nconditions; i++)
+ attnum_cond_counts[i] = bms_num_members(conditions_attnums[i]);
+
+ /* see which clauses are already covered at this point (by previous stats) */
+ for (i = 0; i < step; i++)
+ for (j = 0; j < nclauses; j++)
+ covered_clauses[j] |= (cover_map[current->stats[i] * nclauses + j]);
+
+ /* which remaining statistics covers most clauses / uses most conditions? */
+ for (i = 0; i < nmvstats; i++)
+ {
+ Bitmapset *attnums_covered = NULL;
+ Bitmapset *attnums_conditions = NULL;
+
+ /* skip stats that are already ruled out (either used or inapplicable) */
+ if (ruled_out[i] != -1)
+ continue;
+
+ /* count covered clauses and conditions (for the statistics) */
+ for (j = 0; j < nclauses; j++)
+ {
+ if (cover_map[i * nclauses + j])
+ {
+ Bitmapset *new = bms_union(attnums_covered, clauses_attnums[j]);
+
+ /* get rid of the old bitmap and keep the unified result */
+ bms_free(attnums_covered);
+ attnums_covered = new;
+
+ num_cov_clauses[i] += 1;
+ num_cov_columns[i] += attnum_counts[j];
+
+ /* is the clause already covered (i.e. a condition)? */
+ if (covered_clauses[j])
+ {
+ num_cond_clauses[i] += 1;
+ num_cond_columns[i] += attnum_counts[j];
+ new = bms_union(attnums_conditions,
+ clauses_attnums[j]);
+
+ bms_free(attnums_conditions);
+ attnums_conditions = new;
+ }
+ }
+ }
+
+ /* if all covered clauses are covered by prev stats (thus conditions) */
+ if (num_cov_clauses[i] == num_cond_clauses[i])
+ ruled_out[i] = step;
+
+ /* same if there are no new attributes */
+ else if (bms_num_members(attnums_conditions) == bms_num_members(attnums_covered))
+ ruled_out[i] = step;
+
+ bms_free(attnums_covered);
+ bms_free(attnums_conditions);
+
+ /* if the statistics is inapplicable, try the next one */
+ if (ruled_out[i] != -1)
+ continue;
+
+ /* now let's walk through conditions and count the covered */
+ for (j = 0; j < nconditions; j++)
+ {
+ if (condition_map[i * nconditions + j])
+ {
+ num_cond_clauses[i] += 1;
+ num_cond_columns[i] += attnum_cond_counts[j];
+ }
+ }
+
+ /* otherwise see if this improves the interesting metrics */
+ gain = num_cond_columns[i] / (double)num_cov_columns[i];
+
+ if (gain > max_gain)
+ {
+ max_gain = gain;
+ best_stat = i;
+ }
+ }
+
+ /*
+ * Have we found a suitable statistics? Add it to the solution and
+ * try next step.
+ */
+ if (best_stat != -1)
+ {
+ /* mark the statistics, so that we skip it in next steps */
+ ruled_out[best_stat] = step;
+
+ /* allocate current solution if necessary */
+ if (current == NULL)
+ {
+ current = (mv_solution_t*)palloc0(sizeof(mv_solution_t));
+ current->stats = (int*)palloc0(sizeof(int)*nmvstats);
+ current->nstats = 0;
+ current->nclauses = 0;
+ current->nconditions = 0;
+ }
+
+ current->nclauses += num_cov_clauses[best_stat];
+ current->nconditions += num_cond_clauses[best_stat];
+ current->stats[step] = best_stat;
+ current->nstats++;
+
+ if (*best == NULL)
+ {
+ (*best) = (mv_solution_t*)palloc0(sizeof(mv_solution_t));
+ (*best)->nstats = current->nstats;
+ (*best)->nclauses = current->nclauses;
+ (*best)->nconditions = current->nconditions;
+
+ (*best)->stats = (int*)palloc0(sizeof(int)*nmvstats);
+ memcpy((*best)->stats, current->stats, nmvstats * sizeof(int));
+ }
+ else
+ {
+ /* see if this is a better solution */
+ double current_gain = (double)current->nconditions / current->nclauses;
+ double best_gain = (double)(*best)->nconditions / (*best)->nclauses;
+
+ if ((current_gain > best_gain) ||
+ ((current_gain == best_gain) && (current->nstats < (*best)->nstats)))
+ {
+ (*best)->nstats = current->nstats;
+ (*best)->nclauses = current->nclauses;
+ (*best)->nconditions = current->nconditions;
+ memcpy((*best)->stats, current->stats, nmvstats * sizeof(int));
+ }
+ }
+
+ /*
+ * The recursion only makes sense if we haven't covered all the
+ * attributes (then adding stats is not really possible).
+ */
+ if ((step + 1) < nmvstats)
+ choose_mv_statistics_greedy(root, step+1,
+ nmvstats, mvstats, stats_attnums,
+ nclauses, clauses, clauses_attnums,
+ nconditions, conditions, conditions_attnums,
+ cover_map, condition_map, ruled_out,
+ current, best);
+
+ /* reset the last step */
+ current->nclauses -= num_cov_clauses[best_stat];
+ current->nconditions -= num_cond_clauses[best_stat];
+ current->nstats -= 1;
+ current->stats[step] = 0;
+
+ /* mark the statistics as usable again */
+ ruled_out[best_stat] = -1;
+ }
+
+ /* reset all statistics eliminated in this step */
+ for (i = 0; i < nmvstats; i++)
+ if (ruled_out[i] == step)
+ ruled_out[i] = -1;
+
+ /* free everything allocated in this step */
+ pfree(covered_clauses);
+ pfree(attnum_counts);
+ pfree(num_cov_clauses);
+ pfree(num_cov_columns);
+ pfree(num_cond_clauses);
+ pfree(num_cond_columns);
+}
+
+/*
* We're looking for statistics matching at least 2 attributes,
* referenced in the clauses compatible with multivariate statistics.
* The current selection criteria is very simple - we choose the
@@ -1299,48 +2213,386 @@ collect_mv_attnums(PlannerInfo *root, List *clauses, Oid varRelid,
* TODO This will probably have to consider compatibility of clauses,
* because 'dependencies' will probably work only with equality
* clauses.
+ *
+ * TODO Another way to make the optimization problems smaller might
+ * be splitting the statistics into several disjoint subsets, i.e.
+ * if we can split the graph of statistics (after the elimination)
+ * into multiple components (so that stats in different components
+ * share no attributes), we can do the optimization for each
+ * component separately.
+ *
+ * TODO Another possible optimization might be removing redundant
+ * statistics - if statistics S1 covers S2 (covers S2 attributes
+ * and possibly some more), we can probably remove S2. What
+ * actually matters are attributes from covered clauses (not all
+ * the original attributes). This might however prefer larger,
+ * and thus less accurate, statistics.
+ *
+ * TODO If we could compute what is a "perfect solution" maybe we could
+ * terminate the search after reaching ~90% of it? Say, if we knew
+ * that we can cover 10 clauses and reuse 8 dependencies, maybe
+ * covering 9 clauses and 7 dependencies would be OK?
*/
-static int
-choose_mv_statistics(int nmvstats, MVStats mvstats, Bitmapset *attnums)
+static mv_solution_t *
+choose_mv_statistics(PlannerInfo *root, int nmvstats, MVStats mvstats,
+ List *clauses, List *conditions,
+ Oid varRelid, SpecialJoinInfo *sjinfo, int type)
{
int i, j;
+ mv_solution_t *best = NULL;
+ ListCell *l;
+
+ /* pass only stats matching at least two attributes (from clauses) */
+ MVStats mvstats_filtered = (MVStats)palloc0(nmvstats * sizeof(MVStatsData));
+ int nmvstats_filtered;
+ bool repeat = true;
+ bool *clause_cover_map = NULL,
+ *condition_cover_map = NULL;
+ int *ruled_out = NULL;
+
+ /* build bitmapsets for all stats and clauses */
+ Bitmapset **stats_attnums;
+ Bitmapset **clauses_attnums;
+ Bitmapset **conditions_attnums;
+
+ int nclauses, nconditions;
+ Node ** clauses_array;
+ Node ** conditions_array;
- int choice = -1;
- int current_matches = 1; /* goal #1: maximize */
- int current_dims = (MVSTATS_MAX_DIMENSIONS+1); /* goal #2: minimize */
+ /* copy lists, so that we can free them during elimination easily */
+ clauses = list_copy(clauses);
+ conditions = list_copy(conditions);
/*
- * Walk through the statistics (simple array with nmvstats elements)
- * and for each one count the referenced attributes (encoded in
- * the 'attnums' bitmap).
+ * Reduce the optimization problem size as much as possible.
+ *
+ * Eliminate clauses and conditions not covered by any statistics,
+ * or statistics not matching at least two attributes (one of them
+ * has to be in a regular clause).
+ *
+ * It's possible that removing a statistics in one iteration
+ * eliminates clause in the next one, so we'll repeat this until we
+ * eliminate no clauses/stats in that iteration.
+ *
+ * This can only happen after eliminating a statistics - clauses are
+ * eliminated first, so statistics always reflect that.
*/
+ while (repeat)
+ {
+ /* pass only mv-compatible clauses covered by at least one statistics */
+ List *compatible_clauses = NIL;
+ List *compatible_conditions = NIL;
+
+ Bitmapset *compatible_attnums = NULL;
+ Bitmapset *condition_attnums = NULL;
+ Bitmapset *all_attnums = NULL;
+
+ /*
+ * Clauses
+ *
+ * Walk through clauses and keep only those covered by at least
+ * one of the statistics we still have. Also, collect bitmap of
+ * attributes so that we can make sure we add at least one new.
+ */
+ foreach (l, clauses)
+ {
+ Node *clause = (Node*)lfirst(l);
+ Bitmapset *clause_attnums = NULL;
+ Oid relid;
+
+ /*
+ * The clause has to be mv-compatible (suitable operators etc.).
+ */
+ if (! clause_is_mv_compatible(root, clause, varRelid,
+ &relid, &clause_attnums, sjinfo, type))
+ continue;
+
+ /* is there a statistics covering this clause? */
+ for (i = 0; i < nmvstats; i++)
+ {
+ int k, matches = 0;
+ for (k = 0; k < mvstats[i].stakeys->dim1; k++)
+ {
+ if (bms_is_member(mvstats[i].stakeys->values[k],
+ clause_attnums))
+ matches += 1;
+ }
+
+ /*
+ * The clause is compatible if all attributes it references
+ * are covered by the statistics.
+ */
+ if (bms_num_members(clause_attnums) == matches)
+ {
+ compatible_attnums = bms_union(compatible_attnums,
+ clause_attnums);
+ compatible_clauses = lappend(compatible_clauses,
+ clause);
+ break;
+ }
+ }
+
+ bms_free(clause_attnums);
+ }
+
+ /* we can't have more compatible clauses that source clauses */
+ Assert(list_length(clauses) >= list_length(compatible_clauses));
+
+ /* work with only compatible clauses from now */
+ list_free(clauses);
+ clauses = compatible_clauses;
+
+ /*
+ * Conditions
+ *
+ * Walk through clauses and keep only those covered by at least
+ * one of the statistics we still have. Also, collect bitmap of
+ * attributes so that we can make sure we add at least one new.
+ */
+
+ /* next, generate bitmap of attnums from all mv_compatible conditions */
+ foreach (l, conditions)
+ {
+ Node *clause = (Node*)lfirst(l);
+ Bitmapset *clause_attnums = NULL;
+ Oid relid;
+
+ /*
+ * The clause has to be mv-compatible (suitable operators etc.).
+ */
+ if (! clause_is_mv_compatible(root, clause, varRelid,
+ &relid, &clause_attnums, sjinfo, type))
+ continue;
+
+ /* is there a statistics covering this clause? */
+ for (i = 0; i < nmvstats; i++)
+ {
+ int k, matches = 0;
+ for (k = 0; k < mvstats[i].stakeys->dim1; k++)
+ {
+ if (bms_is_member(mvstats[i].stakeys->values[k],
+ clause_attnums))
+ matches += 1;
+ }
+
+ if (bms_num_members(clause_attnums) == matches)
+ {
+ condition_attnums = bms_union(condition_attnums,
+ clause_attnums);
+ compatible_conditions = lappend(compatible_conditions,
+ clause);
+ break;
+ }
+ }
+
+ bms_free(clause_attnums);
+ }
+
+ /* we can't have more compatible conditions than source conditions */
+ Assert(list_length(conditions) >= list_length(compatible_conditions));
+
+ /* keep only compatible clauses */
+ list_free(conditions);
+ conditions = compatible_conditions;
+
+ /* get a union of attnums (from conditions and clauses) */
+ all_attnums = bms_union(compatible_attnums, condition_attnums);
+
+ /*
+ * Statisitics
+ *
+ * Walk through statistics and only keep those covering at least
+ * one new attribute (excluding conditions) and at two attributes
+ * in both clauses and conditions.
+ */
+ nmvstats_filtered = 0;
+
+ for (i = 0; i < nmvstats; i++)
+ {
+ int k;
+ int matches_new = 0,
+ matches_all = 0;
+
+ for (k = 0; k < mvstats[i].stakeys->dim1; k++)
+ {
+ /* attribute covered by new clause(s) */
+ if (bms_is_member(mvstats[i].stakeys->values[k],
+ compatible_attnums))
+ matches_new += 1;
+
+ /* attribute covered by clause(s) or ondition(s) */
+ if (bms_is_member(mvstats[i].stakeys->values[k],
+ all_attnums))
+ matches_all += 1;
+ }
+
+ /* check we have enough attributes for this statistics */
+ if ((matches_new >= 1) && (matches_all >= 2))
+ {
+ mvstats_filtered[nmvstats_filtered] = mvstats[i];
+ nmvstats_filtered += 1;
+ }
+ }
+
+ /* we can't have more useful stats than we had originally */
+ Assert(nmvstats >= nmvstats_filtered);
+
+ /* if we've eliminated a statistics, trigger another round */
+ repeat = (nmvstats > nmvstats_filtered);
+
+ /*
+ * work only with filtered statistics from now
+ *
+ * FIXME This rewrites the input 'mvstats' array, which is not
+ * exactly pretty as it's an unexpected side-effect (the
+ * caller may use the stats for something else). But the
+ * solution contains indexes into this 'reduced' array so
+ * we can't stop doing that easily.
+ *
+ * Another issue is that we only modify the local 'mvstats'
+ * value, so the caller will still see the original number
+ * of stats (and thus maybe duplicate entries).
+ *
+ * We should make a copy of the array, and only mess with
+ * that copy (and map the indexes to the original ones at
+ * the end, when returning the solution to the user). Or
+ * simply work with OIDs.
+ */
+ if (nmvstats_filtered < nmvstats)
+ {
+ nmvstats = nmvstats_filtered;
+ memcpy(mvstats, mvstats_filtered, sizeof(MVStatsData)*nmvstats);
+ nmvstats_filtered = 0;
+ }
+ }
+
+ /* only do the optimization if we have clauses/statistics */
+ if ((nmvstats == 0) || (list_length(clauses) == 0))
+ return NULL;
+
+ stats_attnums
+ = (Bitmapset **)palloc0(nmvstats * sizeof(Bitmapset *));
+
+ /*
+ * TODO We should sort the stats to make the order deterministic,
+ * otherwise we may get different estimates on different
+ * executions - if there are multiple "equally good" solutions,
+ * we'll keep the first solution we see.
+ *
+ * Sorting by OID probably is not the right solution though,
+ * because we'd like it to be somehow reproducible,
+ * irrespectedly of the order of ADD STATISTICS commands.
+ * So maybe statkeys?
+ */
+
for (i = 0; i < nmvstats; i++)
{
- /* columns matching this statistics */
- int matches = 0;
+ for (j = 0; j < mvstats[i].stakeys->dim1; j++)
+ stats_attnums[i] = bms_add_member(stats_attnums[i],
+ mvstats[i].stakeys->values[j]);
+ }
- int2vector * attrs = mvstats[i].stakeys;
- int numattrs = mvstats[i].stakeys->dim1;
+ /* collect clauses an bitmap of attnums */
+ nclauses = 0;
+ clauses_attnums = (Bitmapset **)palloc0(list_length(clauses)
+ * sizeof(Bitmapset *));
+ clauses_array = (Node **)palloc0(list_length(clauses)
+ * sizeof(Node *));
- /* count columns covered by the histogram */
- for (j = 0; j < numattrs; j++)
- if (bms_is_member(attrs->values[j], attnums))
- matches++;
+ foreach (l, clauses)
+ {
+ Oid relid;
+ Bitmapset * attnums = NULL;
/*
- * Use this statistics when it improves the number of matches or
- * when it matches the same number of attributes but is smaller.
+ * The clause has to be mv-compatible (suitable operators etc.).
*/
- if ((matches > current_matches) ||
- ((matches == current_matches) && (current_dims > numattrs)))
+ if (! clause_is_mv_compatible(root, (Node *)lfirst(l), varRelid,
+ &relid, &attnums, sjinfo, type))
+ elog(ERROR, "should not get non-mv-compatible cluase");
+
+ clauses_attnums[nclauses] = attnums;
+ clauses_array[nclauses] = (Node *)lfirst(l);
+ nclauses += 1;
+ }
+
+ /* collect conditions and bitmap of attnums */
+ nconditions = 0;
+ conditions_attnums = (Bitmapset **)palloc0(list_length(conditions)
+ * sizeof(Bitmapset *));
+ conditions_array = (Node **)palloc0(list_length(conditions)
+ * sizeof(Node *));
+
+ foreach (l, conditions)
+ {
+ Oid relid;
+ Bitmapset * attnums = NULL;
+
+ /* conditions are mv-compatible (thanks to the reduction) */
+ if (! clause_is_mv_compatible(root, (Node *)lfirst(l), varRelid,
+ &relid, &attnums, sjinfo, type))
+ elog(ERROR, "should not get non-mv-compatible cluase");
+
+ conditions_attnums[nconditions] = attnums;
+ conditions_array[nconditions] = (Node *)lfirst(l);
+ nconditions += 1;
+ }
+
+ /*
+ * Build bitmaps with info about which clauses/conditions are
+ * covered by each statistics (so that we don't need to call the
+ * bms_is_subset over and over again).
+ */
+ clause_cover_map = (bool*)palloc0(nclauses * nmvstats);
+ condition_cover_map = (bool*)palloc0(nconditions * nmvstats);
+ ruled_out = (int*)palloc0(nmvstats * sizeof(int));
+
+ for (i = 0; i < nmvstats; i++)
+ {
+ ruled_out[i] = -1; /* not ruled out by default */
+ for (j = 0; j < nclauses; j++)
+ {
+ clause_cover_map[i * nclauses + j]
+ = bms_is_subset(clauses_attnums[j],
+ stats_attnums[i]);
+ }
+
+ for (j = 0; j < nconditions; j++)
{
- choice = i;
- current_matches = matches;
- current_dims = numattrs;
+ condition_cover_map[i * nconditions + j]
+ = bms_is_subset(conditions_attnums[j],
+ stats_attnums[i]);
}
}
- return choice;
+ /* do the optimization itself */
+ if (mvstat_search_type == MVSTAT_SEARCH_EXHAUSTIVE)
+ choose_mv_statistics_exhaustive(root, 0,
+ nmvstats, mvstats, stats_attnums,
+ nclauses, clauses_array, clauses_attnums,
+ nconditions, conditions_array, conditions_attnums,
+ clause_cover_map, condition_cover_map,
+ ruled_out, NULL, &best);
+ else
+ choose_mv_statistics_greedy(root, 0,
+ nmvstats, mvstats, stats_attnums,
+ nclauses, clauses_array, clauses_attnums,
+ nconditions, conditions_array, conditions_attnums,
+ clause_cover_map, condition_cover_map,
+ ruled_out, NULL, &best);
+
+ /* maybe we should leave the cleanup up to the memory context */
+ pfree(mvstats_filtered);
+ pfree(stats_attnums);
+ pfree(clauses_attnums);
+ pfree(clauses_array);
+ pfree(conditions_attnums);
+ pfree(conditions_array);
+ pfree(clause_cover_map);
+ pfree(condition_cover_map);
+ pfree(ruled_out);
+
+ return best;
}
@@ -1624,6 +2876,51 @@ clause_is_mv_compatible(PlannerInfo *root, Node *clause, Oid varRelid,
return false;
}
+
+static Bitmapset *
+clause_mv_get_attnums(PlannerInfo *root, Node *clause)
+{
+ Bitmapset * attnums = NULL;
+
+ /* Extract clause from restrict info, if needed. */
+ if (IsA(clause, RestrictInfo))
+ clause = (Node*)((RestrictInfo*)clause)->clause;
+
+ /*
+ * Only simple opclauses and IS NULL tests are compatible with
+ * multivariate stats at this point.
+ */
+ if ((is_opclause(clause))
+ && (list_length(((OpExpr *) clause)->args) == 2))
+ {
+ OpExpr *expr = (OpExpr *) clause;
+
+ if (IsA(linitial(expr->args), Var))
+ attnums = bms_add_member(attnums,
+ ((Var*)linitial(expr->args))->varattno);
+ else
+ attnums = bms_add_member(attnums,
+ ((Var*)lsecond(expr->args))->varattno);
+ }
+ else if (IsA(clause, NullTest)
+ && IsA(((NullTest*)clause)->arg, Var))
+ {
+ attnums = bms_add_member(attnums,
+ ((Var*)((NullTest*)clause)->arg)->varattno);
+ }
+ else if (or_clause(clause) || and_clause(clause))
+ {
+ ListCell *l;
+ foreach (l, ((BoolExpr*)clause)->args)
+ {
+ attnums = bms_join(attnums,
+ clause_mv_get_attnums(root, (Node*)lfirst(l)));
+ }
+ }
+
+ return attnums;
+}
+
/*
* Performs reduction of clauses using functional dependencies, i.e.
* removes clauses that are considered redundant. It simply walks
@@ -2049,20 +3346,24 @@ clauselist_apply_dependencies(PlannerInfo *root, List *clauses, Oid varRelid,
* as the clauses are processed (and skip items that are 'match').
*/
static Selectivity
-clauselist_mv_selectivity_mcvlist(PlannerInfo *root, List *clauses,
- MVStats mvstats, bool *fullmatch,
- Selectivity *lowsel)
+clauselist_mv_selectivity_mcvlist(PlannerInfo *root, MVStats mvstats,
+ List *clauses, List *conditions,
+ bool *fullmatch, Selectivity *lowsel)
{
int i;
Selectivity s = 0.0;
+ Selectivity t = 0.0;
MCVList mcvlist = NULL;
+
int nmatches = 0;
+ int nconditions = 0;
/* match/mismatch bitmap for each MCV item */
char * matches = NULL;
+ char * condition_matches = NULL;
Assert(clauses != NIL);
- Assert(list_length(clauses) >= 2);
+ Assert(list_length(clauses) >= 1);
/* there's no MCV list built yet */
if (! mvstats->mcv_built)
@@ -2073,13 +3374,34 @@ clauselist_mv_selectivity_mcvlist(PlannerInfo *root, List *clauses,
Assert(mcvlist != NULL);
Assert(mcvlist->nitems > 0);
- /* by default all the MCV items match the clauses fully */
- matches = palloc0(sizeof(char) * mcvlist->nitems);
- memset(matches, MVSTATS_MATCH_FULL, sizeof(char)*mcvlist->nitems);
-
/* number of matching MCV items */
nmatches = mcvlist->nitems;
+ nconditions = mcvlist->nitems;
+
+ /* conditions */
+ condition_matches = palloc0(sizeof(char) * nconditions);
+ memset(condition_matches, MVSTATS_MATCH_FULL, sizeof(char)*nconditions);
+ /* by default all the MCV items match the clauses fully */
+ matches = palloc0(sizeof(char) * nmatches);
+ memset(matches, MVSTATS_MATCH_FULL, sizeof(char)*nmatches);
+
+ /*
+ * build the match bitmap for the conditions
+ */
+ if (conditions != NIL)
+ nconditions = update_match_bitmap_mcvlist(root, conditions,
+ mvstats->stakeys, mcvlist,
+ nconditions, condition_matches,
+ lowsel, fullmatch, false);
+
+ /*
+ * build the match bitmap for the estimated clauses
+ *
+ * TODO This evaluates the clauses for all MCV items, even those
+ * ruled out by the conditions. The final result should be the
+ * same, but it might be faster.
+ */
nmatches = update_match_bitmap_mcvlist(root, clauses,
mvstats->stakeys, mcvlist,
nmatches, matches,
@@ -2088,14 +3410,25 @@ clauselist_mv_selectivity_mcvlist(PlannerInfo *root, List *clauses,
/* sum frequencies for all the matching MCV items */
for (i = 0; i < mcvlist->nitems; i++)
{
+ /* skit MCV items not matching the conditions */
+ if (condition_matches[i] == MVSTATS_MATCH_NONE)
+ continue;
+
if (matches[i] != MVSTATS_MATCH_NONE)
s += mcvlist->items[i]->frequency;
+
+ t += mcvlist->items[i]->frequency;
}
pfree(matches);
+ pfree(condition_matches);
pfree(mcvlist);
- return s;
+ /* no condition matches */
+ if (t == 0.0)
+ return (Selectivity)0.0;
+
+ return (s / t);
}
/*
@@ -2490,13 +3823,17 @@ update_match_bitmap_mcvlist(PlannerInfo *root, List *clauses,
* this is not uncommon, but for histograms it's not that clear.
*/
static Selectivity
-clauselist_mv_selectivity_histogram(PlannerInfo *root, List *clauses,
- MVStats mvstats)
+clauselist_mv_selectivity_histogram(PlannerInfo *root, MVStats mvstats,
+ List *clauses, List *conditions)
{
int i;
Selectivity s = 0.0;
+ Selectivity t = 0.0;
int nmatches = 0;
+ int nconditions = 0;
char *matches = NULL;
+ char *condition_matches = NULL;
+
MVHistogram mvhist = NULL;
/* there's no histogram */
@@ -2508,18 +3845,34 @@ clauselist_mv_selectivity_histogram(PlannerInfo *root, List *clauses,
Assert (mvhist != NULL);
Assert (clauses != NIL);
- Assert (list_length(clauses) >= 2);
+ Assert (list_length(clauses) >= 1);
+
+ nmatches = mvhist->nbuckets;
+ nconditions = mvhist->nbuckets;
/*
* Bitmap of bucket matches (mismatch, partial, full). by default
* all buckets fully match (and we'll eliminate them).
*/
- matches = palloc0(sizeof(char) * mvhist->nbuckets);
- memset(matches, MVSTATS_MATCH_FULL, sizeof(char)*mvhist->nbuckets);
+ matches = palloc0(sizeof(char) * nmatches);
+ memset(matches, MVSTATS_MATCH_FULL, sizeof(char)*nmatches);
- nmatches = mvhist->nbuckets;
+ condition_matches = palloc0(sizeof(char)*nconditions);
+ memset(condition_matches, MVSTATS_MATCH_FULL, sizeof(char)*nconditions);
+
+ /* build the match bitmap for the conditions */
+ if (conditions != NIL)
+ update_match_bitmap_histogram(root, conditions,
+ mvstats->stakeys, mvhist,
+ nconditions, condition_matches, false);
- /* build the match bitmap */
+ /*
+ * build the match bitmap for the estimated clauses
+ *
+ * TODO This evaluates the clauses for all buckets, even those
+ * ruled out by the conditions. The final result should be
+ * the same, but it might be faster.
+ */
update_match_bitmap_histogram(root, clauses,
mvstats->stakeys, mvhist,
nmatches, matches, false);
@@ -2527,17 +3880,37 @@ clauselist_mv_selectivity_histogram(PlannerInfo *root, List *clauses,
/* now, walk through the buckets and sum the selectivities */
for (i = 0; i < mvhist->nbuckets; i++)
{
+ float coeff = 1.0;
+
+ /* skip buckets not matching the conditions */
+ if (condition_matches[i] == MVSTATS_MATCH_NONE)
+ continue;
+ else if (condition_matches[i] == MVSTATS_MATCH_PARTIAL)
+ coeff = 0.5;
+
+ t += coeff * mvhist->buckets[i]->ntuples;
+
if (matches[i] == MVSTATS_MATCH_FULL)
- s += mvhist->buckets[i]->ntuples;
+ s += coeff * mvhist->buckets[i]->ntuples;
else if (matches[i] == MVSTATS_MATCH_PARTIAL)
- s += 0.5 * mvhist->buckets[i]->ntuples;
+ /*
+ * TODO If both conditions and clauses match partially, this
+ * will use 0.25 match - not sure if that's the right
+ * thing solution, but seems about right.
+ */
+ s += coeff * 0.5 * mvhist->buckets[i]->ntuples;
}
/* release the allocated bitmap and deserialized histogram */
pfree(matches);
+ pfree(condition_matches);
pfree(mvhist);
- return s;
+ /* no condition matches */
+ if (t == 0.0)
+ return (Selectivity)0.0;
+
+ return (s / t);
}
/*
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index 1a0d358..71beb2e 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -3280,7 +3280,8 @@ compute_semi_anti_join_factors(PlannerInfo *root,
joinquals,
0,
jointype,
- sjinfo);
+ sjinfo,
+ NIL);
/*
* Also get the normal inner-join selectivity of the join clauses.
@@ -3303,7 +3304,8 @@ compute_semi_anti_join_factors(PlannerInfo *root,
joinquals,
0,
JOIN_INNER,
- &norm_sjinfo);
+ &norm_sjinfo,
+ NIL);
/* Avoid leaking a lot of ListCells */
if (jointype == JOIN_ANTI)
@@ -3470,7 +3472,7 @@ approx_tuple_count(PlannerInfo *root, JoinPath *path, List *quals)
Node *qual = (Node *) lfirst(l);
/* Note that clause_selectivity will be able to cache its result */
- selec *= clause_selectivity(root, qual, 0, JOIN_INNER, &sjinfo);
+ selec *= clause_selectivity(root, qual, 0, JOIN_INNER, &sjinfo, NIL);
}
/* Apply it to the input relation sizes */
@@ -3506,7 +3508,8 @@ set_baserel_size_estimates(PlannerInfo *root, RelOptInfo *rel)
rel->baserestrictinfo,
0,
JOIN_INNER,
- NULL);
+ NULL,
+ NIL);
rel->rows = clamp_row_est(nrows);
@@ -3543,7 +3546,8 @@ get_parameterized_baserel_size(PlannerInfo *root, RelOptInfo *rel,
allclauses,
rel->relid, /* do not use 0! */
JOIN_INNER,
- NULL);
+ NULL,
+ NIL);
nrows = clamp_row_est(nrows);
/* For safety, make sure result is not more than the base estimate */
if (nrows > rel->rows)
@@ -3681,12 +3685,14 @@ calc_joinrel_size_estimate(PlannerInfo *root,
joinquals,
0,
jointype,
- sjinfo);
+ sjinfo,
+ NIL);
pselec = clauselist_selectivity(root,
pushedquals,
0,
jointype,
- sjinfo);
+ sjinfo,
+ NIL);
/* Avoid leaking a lot of ListCells */
list_free(joinquals);
@@ -3698,7 +3704,8 @@ calc_joinrel_size_estimate(PlannerInfo *root,
restrictlist,
0,
jointype,
- sjinfo);
+ sjinfo,
+ NIL);
pselec = 0.0; /* not used, keep compiler quiet */
}
diff --git a/src/backend/optimizer/util/orclauses.c b/src/backend/optimizer/util/orclauses.c
index f0acc14..e41508b 100644
--- a/src/backend/optimizer/util/orclauses.c
+++ b/src/backend/optimizer/util/orclauses.c
@@ -280,7 +280,7 @@ consider_new_or_clause(PlannerInfo *root, RelOptInfo *rel,
* saving work later.)
*/
or_selec = clause_selectivity(root, (Node *) or_rinfo,
- 0, JOIN_INNER, NULL);
+ 0, JOIN_INNER, NULL, NIL);
/*
* The clause is only worth adding to the query if it rejects a useful
@@ -342,7 +342,7 @@ consider_new_or_clause(PlannerInfo *root, RelOptInfo *rel,
/* Compute inner-join size */
orig_selec = clause_selectivity(root, (Node *) join_or_rinfo,
- 0, JOIN_INNER, &sjinfo);
+ 0, JOIN_INNER, &sjinfo, NIL);
/* And hack cached selectivity so join size remains the same */
join_or_rinfo->norm_selec = orig_selec / or_selec;
diff --git a/src/backend/utils/adt/selfuncs.c b/src/backend/utils/adt/selfuncs.c
index 4dd3f9f..326dd36 100644
--- a/src/backend/utils/adt/selfuncs.c
+++ b/src/backend/utils/adt/selfuncs.c
@@ -1580,13 +1580,15 @@ booltestsel(PlannerInfo *root, BoolTestType booltesttype, Node *arg,
case IS_NOT_FALSE:
selec = (double) clause_selectivity(root, arg,
varRelid,
- jointype, sjinfo);
+ jointype, sjinfo,
+ NIL);
break;
case IS_FALSE:
case IS_NOT_TRUE:
selec = 1.0 - (double) clause_selectivity(root, arg,
varRelid,
- jointype, sjinfo);
+ jointype, sjinfo,
+ NIL);
break;
default:
elog(ERROR, "unrecognized booltesttype: %d",
@@ -6196,7 +6198,8 @@ genericcostestimate(PlannerInfo *root,
indexSelectivity = clauselist_selectivity(root, selectivityQuals,
index->rel->relid,
JOIN_INNER,
- NULL);
+ NULL,
+ NIL);
/*
* If caller didn't give us an estimate, estimate the number of index
@@ -6521,7 +6524,8 @@ btcostestimate(PG_FUNCTION_ARGS)
btreeSelectivity = clauselist_selectivity(root, selectivityQuals,
index->rel->relid,
JOIN_INNER,
- NULL);
+ NULL,
+ NIL);
numIndexTuples = btreeSelectivity * index->rel->tuples;
/*
@@ -7264,7 +7268,8 @@ gincostestimate(PG_FUNCTION_ARGS)
*indexSelectivity = clauselist_selectivity(root, selectivityQuals,
index->rel->relid,
JOIN_INNER,
- NULL);
+ NULL,
+ NIL);
/* fetch estimated page cost for tablespace containing index */
get_tablespace_page_costs(index->reltablespace,
@@ -7496,7 +7501,7 @@ brincostestimate(PG_FUNCTION_ARGS)
*indexSelectivity =
clauselist_selectivity(root, indexQuals,
path->indexinfo->rel->relid,
- JOIN_INNER, NULL);
+ JOIN_INNER, NULL, NIL);
*indexCorrelation = 1;
/*
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index b8a0f9f..5cd2583 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -75,6 +75,7 @@
#include "utils/bytea.h"
#include "utils/guc_tables.h"
#include "utils/memutils.h"
+#include "utils/mvstats.h"
#include "utils/pg_locale.h"
#include "utils/plancache.h"
#include "utils/portal.h"
@@ -396,6 +397,15 @@ static const struct config_enum_entry row_security_options[] = {
};
/*
+ * Search algorithm for multivariate stats.
+ */
+static const struct config_enum_entry mvstat_search_options[] = {
+ {"greedy", MVSTAT_SEARCH_GREEDY, false},
+ {"exhaustive", MVSTAT_SEARCH_EXHAUSTIVE, false},
+ {NULL, 0, false}
+};
+
+/*
* Options for enum values stored in other modules
*/
extern const struct config_enum_entry wal_level_options[];
@@ -3651,6 +3661,16 @@ static struct config_enum ConfigureNamesEnum[] =
NULL, NULL, NULL
},
+ {
+ {"mvstat_search", PGC_USERSET, QUERY_TUNING_OTHER,
+ gettext_noop("Sets the algorithm used for combining multivariate stats."),
+ NULL
+ },
+ &mvstat_search_type,
+ MVSTAT_SEARCH_GREEDY, mvstat_search_options,
+ NULL, NULL, NULL
+ },
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, 0, NULL, NULL, NULL, NULL
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 9c2000b..7a3835b 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -182,11 +182,13 @@ extern Selectivity clauselist_selectivity(PlannerInfo *root,
List *clauses,
int varRelid,
JoinType jointype,
- SpecialJoinInfo *sjinfo);
+ SpecialJoinInfo *sjinfo,
+ List *conditions);
extern Selectivity clause_selectivity(PlannerInfo *root,
Node *clause,
int varRelid,
JoinType jointype,
- SpecialJoinInfo *sjinfo);
+ SpecialJoinInfo *sjinfo,
+ List *conditions);
#endif /* COST_H */
diff --git a/src/include/utils/mvstats.h b/src/include/utils/mvstats.h
index 673e546..194bbf8 100644
--- a/src/include/utils/mvstats.h
+++ b/src/include/utils/mvstats.h
@@ -16,6 +16,14 @@
#include "commands/vacuum.h"
+typedef enum MVStatSearchType
+{
+ MVSTAT_SEARCH_EXHAUSTIVE, /* exhaustive search */
+ MVSTAT_SEARCH_GREEDY /* greedy search */
+} MVStatSearchType;
+
+extern int mvstat_search_type;
+
/*
* Basic info about the stats, used when choosing what to use
*/
--
2.0.5
[application/sql] mcv.sql (25.8K, ../../5519E9B2.5080105@2ndquadrant.com/7-mcv.sql)
download
[application/sql] histogram.sql (34.8K, ../../5519E9B2.5080105@2ndquadrant.com/8-histogram.sql)
download
[application/sql] dependencies.sql (19.0K, ../../5519E9B2.5080105@2ndquadrant.com/9-dependencies.sql)
download
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: WIP: multivariate statistics / proof of concept
2014-10-12 22:00 WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-11-13 11:31 ` Re: WIP: multivariate statistics / proof of concept Simon Riggs <simon@2ndQuadrant.com>
2014-11-13 13:11 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-11-13 15:51 ` Re: WIP: multivariate statistics / proof of concept Katharina Büchse <katharina.buechse@uni-jena.de>
2014-11-13 16:42 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-11-15 17:49 ` Re: WIP: multivariate statistics / proof of concept Kevin Grittner <kgrittn@ymail.com>
2014-11-15 18:35 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-12-08 01:01 ` Re: WIP: multivariate statistics / proof of concept Michael Paquier <michael.paquier@gmail.com>
2014-12-09 20:15 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-12-15 02:55 ` Re: WIP: multivariate statistics / proof of concept Michael Paquier <michael.paquier@gmail.com>
2015-01-15 08:00 ` Re: WIP: multivariate statistics / proof of concept Michael Paquier <michael.paquier@gmail.com>
2015-01-24 20:21 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-20 08:33 ` Re: WIP: multivariate statistics / proof of concept Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
2015-03-20 21:17 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-24 05:34 ` Re: WIP: multivariate statistics / proof of concept Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
2015-03-31 00:26 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tomas.vondra@2ndquadrant.com>
@ 2015-04-28 16:09 ` Jeff Janes <jeff.janes@gmail.com>
2015-04-28 16:13 ` Re: WIP: multivariate statistics / proof of concept Stephen Frost <sfrost@snowman.net>
0 siblings, 1 reply; 37+ messages in thread
From: Jeff Janes @ 2015-04-28 16:09 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>; pgsql-hackers
On Mon, Mar 30, 2015 at 5:26 PM, Tomas Vondra <tomas.vondra@2ndquadrant.com>
wrote:
> Hello,
>
> attached is a new version of the patch series. Aside from fixing various
> issues (crashes, memory leaks). The patches are rebased to current
> master, and I also attach a few SQL scripts I used for testing (nothing
> fancy, just stress-testing all the parts the patch touches).
>
Hi Tomas,
I get cascading conflicts in pg_proc.h. It looked easy enough to fix,
except then I get compiler errors:
funcapi.c: In function 'get_func_trftypes':
funcapi.c:890: warning: unused variable 'procStruct'
utils/fmgrtab.o:(.rodata+0x10cf8): undefined reference to `_null_'
utils/fmgrtab.o:(.rodata+0x10d18): undefined reference to `_null_'
utils/fmgrtab.o:(.rodata+0x10d38): undefined reference to `_null_'
utils/fmgrtab.o:(.rodata+0x10d58): undefined reference to `_null_'
collect2: ld returned 1 exit status
make[2]: *** [postgres] Error 1
make[1]: *** [all-backend-recurse] Error 2
make: *** [all-src-recurse] Error 2
make: *** Waiting for unfinished jobs....
make: *** [temp-install] Error 2
Cheers,
Jeff
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: WIP: multivariate statistics / proof of concept
2014-10-12 22:00 WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-11-13 11:31 ` Re: WIP: multivariate statistics / proof of concept Simon Riggs <simon@2ndQuadrant.com>
2014-11-13 13:11 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-11-13 15:51 ` Re: WIP: multivariate statistics / proof of concept Katharina Büchse <katharina.buechse@uni-jena.de>
2014-11-13 16:42 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-11-15 17:49 ` Re: WIP: multivariate statistics / proof of concept Kevin Grittner <kgrittn@ymail.com>
2014-11-15 18:35 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-12-08 01:01 ` Re: WIP: multivariate statistics / proof of concept Michael Paquier <michael.paquier@gmail.com>
2014-12-09 20:15 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-12-15 02:55 ` Re: WIP: multivariate statistics / proof of concept Michael Paquier <michael.paquier@gmail.com>
2015-01-15 08:00 ` Re: WIP: multivariate statistics / proof of concept Michael Paquier <michael.paquier@gmail.com>
2015-01-24 20:21 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-20 08:33 ` Re: WIP: multivariate statistics / proof of concept Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
2015-03-20 21:17 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-24 05:34 ` Re: WIP: multivariate statistics / proof of concept Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
2015-03-31 00:26 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-04-28 16:09 ` Re: WIP: multivariate statistics / proof of concept Jeff Janes <jeff.janes@gmail.com>
@ 2015-04-28 16:13 ` Stephen Frost <sfrost@snowman.net>
2015-04-28 17:36 ` Re: WIP: multivariate statistics / proof of concept Jeff Janes <jeff.janes@gmail.com>
0 siblings, 1 reply; 37+ messages in thread
From: Stephen Frost @ 2015-04-28 16:13 UTC (permalink / raw)
To: Jeff Janes <jeff.janes@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>; pgsql-hackers
* Jeff Janes (jeff.janes@gmail.com) wrote:
> On Mon, Mar 30, 2015 at 5:26 PM, Tomas Vondra <tomas.vondra@2ndquadrant.com>
> wrote:
> > attached is a new version of the patch series. Aside from fixing various
> > issues (crashes, memory leaks). The patches are rebased to current
> > master, and I also attach a few SQL scripts I used for testing (nothing
> > fancy, just stress-testing all the parts the patch touches).
>
> I get cascading conflicts in pg_proc.h. It looked easy enough to fix,
> except then I get compiler errors:
Yeah, those are because you didn't address the new column which was
added to pg_proc. You need to add another _null_ in the pg_proc.h lines
in the correct place, apparently on four lines.
Thanks!
Stephen
Attachments:
[application/pgp-signature] signature.asc (818B, ../../20150428161310.GR30322@tamriel.snowman.net/2-signature.asc)
download
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: WIP: multivariate statistics / proof of concept
2014-10-12 22:00 WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-11-13 11:31 ` Re: WIP: multivariate statistics / proof of concept Simon Riggs <simon@2ndQuadrant.com>
2014-11-13 13:11 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-11-13 15:51 ` Re: WIP: multivariate statistics / proof of concept Katharina Büchse <katharina.buechse@uni-jena.de>
2014-11-13 16:42 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-11-15 17:49 ` Re: WIP: multivariate statistics / proof of concept Kevin Grittner <kgrittn@ymail.com>
2014-11-15 18:35 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-12-08 01:01 ` Re: WIP: multivariate statistics / proof of concept Michael Paquier <michael.paquier@gmail.com>
2014-12-09 20:15 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-12-15 02:55 ` Re: WIP: multivariate statistics / proof of concept Michael Paquier <michael.paquier@gmail.com>
2015-01-15 08:00 ` Re: WIP: multivariate statistics / proof of concept Michael Paquier <michael.paquier@gmail.com>
2015-01-24 20:21 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-20 08:33 ` Re: WIP: multivariate statistics / proof of concept Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
2015-03-20 21:17 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-24 05:34 ` Re: WIP: multivariate statistics / proof of concept Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
2015-03-31 00:26 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-04-28 16:09 ` Re: WIP: multivariate statistics / proof of concept Jeff Janes <jeff.janes@gmail.com>
2015-04-28 16:13 ` Re: WIP: multivariate statistics / proof of concept Stephen Frost <sfrost@snowman.net>
@ 2015-04-28 17:36 ` Jeff Janes <jeff.janes@gmail.com>
2015-04-28 18:15 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 1 reply; 37+ messages in thread
From: Jeff Janes @ 2015-04-28 17:36 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>; pgsql-hackers
On Tue, Apr 28, 2015 at 9:13 AM, Stephen Frost <sfrost@snowman.net> wrote:
> * Jeff Janes (jeff.janes@gmail.com) wrote:
> > On Mon, Mar 30, 2015 at 5:26 PM, Tomas Vondra <
> tomas.vondra@2ndquadrant.com>
> > wrote:
> > > attached is a new version of the patch series. Aside from fixing
> various
> > > issues (crashes, memory leaks). The patches are rebased to current
> > > master, and I also attach a few SQL scripts I used for testing (nothing
> > > fancy, just stress-testing all the parts the patch touches).
> >
> > I get cascading conflicts in pg_proc.h. It looked easy enough to fix,
> > except then I get compiler errors:
>
> Yeah, those are because you didn't address the new column which was
> added to pg_proc. You need to add another _null_ in the pg_proc.h lines
> in the correct place, apparently on four lines.
>
Thanks. I think I tried that, but was still having trouble. But it turns
out that the trouble was for an unrelated reason, and I got it to compile
now.
Some of the fdw's need a patch as well in order to compile, see attached.
Cheers,
Jeff
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[application/octet-stream] multivariate_contrib.patch (1.8K, ../../CAMkU=1xrzZt3YGiux0tNkvXKGu4Jt3proMS+_X+UQhpB+qQ2Hw@mail.gmail.com/3-multivariate_contrib.patch)
download | inline diff:
diff --git a/contrib/file_fdw/file_fdw.c b/contrib/file_fdw/file_fdw.c
new file mode 100644
index 4368897..7b4839b
*** a/contrib/file_fdw/file_fdw.c
--- b/contrib/file_fdw/file_fdw.c
*************** estimate_size(PlannerInfo *root, RelOptI
*** 947,953 ****
baserel->baserestrictinfo,
0,
JOIN_INNER,
! NULL);
nrows = clamp_row_est(nrows);
--- 947,954 ----
baserel->baserestrictinfo,
0,
JOIN_INNER,
! NULL,
! NIL);
nrows = clamp_row_est(nrows);
diff --git a/contrib/postgres_fdw/postgres_fdw.c b/contrib/postgres_fdw/postgres_fdw.c
new file mode 100644
index 478e124..ff6b438
*** a/contrib/postgres_fdw/postgres_fdw.c
--- b/contrib/postgres_fdw/postgres_fdw.c
*************** postgresGetForeignRelSize(PlannerInfo *r
*** 478,484 ****
fpinfo->local_conds,
baserel->relid,
JOIN_INNER,
! NULL);
cost_qual_eval(&fpinfo->local_conds_cost, fpinfo->local_conds, root);
--- 478,485 ----
fpinfo->local_conds,
baserel->relid,
JOIN_INNER,
! NULL,
! NIL);
cost_qual_eval(&fpinfo->local_conds_cost, fpinfo->local_conds, root);
*************** estimate_path_cost_size(PlannerInfo *roo
*** 1770,1776 ****
local_join_conds,
baserel->relid,
JOIN_INNER,
! NULL);
local_sel *= fpinfo->local_conds_sel;
rows = clamp_row_est(rows * local_sel);
--- 1771,1778 ----
local_join_conds,
baserel->relid,
JOIN_INNER,
! NULL,
! NIL);
local_sel *= fpinfo->local_conds_sel;
rows = clamp_row_est(rows * local_sel);
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: WIP: multivariate statistics / proof of concept
2014-10-12 22:00 WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-11-13 11:31 ` Re: WIP: multivariate statistics / proof of concept Simon Riggs <simon@2ndQuadrant.com>
2014-11-13 13:11 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-11-13 15:51 ` Re: WIP: multivariate statistics / proof of concept Katharina Büchse <katharina.buechse@uni-jena.de>
2014-11-13 16:42 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-11-15 17:49 ` Re: WIP: multivariate statistics / proof of concept Kevin Grittner <kgrittn@ymail.com>
2014-11-15 18:35 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-12-08 01:01 ` Re: WIP: multivariate statistics / proof of concept Michael Paquier <michael.paquier@gmail.com>
2014-12-09 20:15 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-12-15 02:55 ` Re: WIP: multivariate statistics / proof of concept Michael Paquier <michael.paquier@gmail.com>
2015-01-15 08:00 ` Re: WIP: multivariate statistics / proof of concept Michael Paquier <michael.paquier@gmail.com>
2015-01-24 20:21 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-20 08:33 ` Re: WIP: multivariate statistics / proof of concept Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
2015-03-20 21:17 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-24 05:34 ` Re: WIP: multivariate statistics / proof of concept Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
2015-03-31 00:26 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-04-28 16:09 ` Re: WIP: multivariate statistics / proof of concept Jeff Janes <jeff.janes@gmail.com>
2015-04-28 16:13 ` Re: WIP: multivariate statistics / proof of concept Stephen Frost <sfrost@snowman.net>
2015-04-28 17:36 ` Re: WIP: multivariate statistics / proof of concept Jeff Janes <jeff.janes@gmail.com>
@ 2015-04-28 18:15 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 0 replies; 37+ messages in thread
From: Tomas Vondra @ 2015-04-28 18:15 UTC (permalink / raw)
To: Jeff Janes <jeff.janes@gmail.com>; Stephen Frost <sfrost@snowman.net>; +Cc: Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>; pgsql-hackers
Hi,
On 04/28/15 19:36, Jeff Janes wrote:
>
...
>
> Thanks. I think I tried that, but was still having trouble. But it
> turns out that the trouble was for an unrelated reason, and I got it
> to compile now.
Yeah, a new column was added to pg_proc the day after I submitted the
pacth. Will address that in a new version, hopefully in a few days.
>
> Some of the fdw's need a patch as well in order to compile, see
> attached.
Thanks, I forgot to tweak the clauselist_selectivity() calls contrib :-(
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: WIP: multivariate statistics / proof of concept
2014-10-12 22:00 WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-11-13 11:31 ` Re: WIP: multivariate statistics / proof of concept Simon Riggs <simon@2ndQuadrant.com>
@ 2016-11-21 22:10 ` Robert Haas <robertmhaas@gmail.com>
2016-11-22 03:42 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tomas.vondra@2ndquadrant.com>
1 sibling, 1 reply; 37+ messages in thread
From: Robert Haas @ 2016-11-21 22:10 UTC (permalink / raw)
To: Simon Riggs <simon@2ndquadrant.com>; +Cc: Tomas Vondra <tv@fuzzy.cz>; pgsql-hackers
[ reviving an old multivariate statistics thread ]
On Thu, Nov 13, 2014 at 6:31 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
> On 12 October 2014 23:00, Tomas Vondra <tv@fuzzy.cz> wrote:
>
>> It however seems to be working sufficiently well at this point, enough
>> to get some useful feedback. So here we go.
>
> This looks interesting and useful.
>
> What I'd like to check before a detailed review is that this has
> sufficient applicability to be useful.
>
> My understanding is that Q9 and Q18 of TPC-H have poor plans as a
> result of multi-column stats errors.
>
> Could you look at those queries and confirm that this patch can
> produce better plans for them?
Tomas, did you ever do any testing in this area? One of my
colleagues, Rafia Sabih, recently did some testing of TPC-H queries @
20 GB. Q18 actually doesn't complete at all right now because of an
issue with the new simplehash implementation. I reported it to Andres
and he tracked it down, but hasn't posted the patch yet - see
http://archives.postgresql.org/message-id/20161115192802.jfbec5s6ougxwicp@alap3.anarazel.de
Of the remaining queries, the slowest are Q9 and Q20, and both of them
have serious estimation errors. On Q9, things go wrong here:
-> Merge Join
(cost=5225092.04..6595105.57 rows=154 width=47) (actual
time=103592.821..149335.010 rows=6503988 loops=1)
Merge Cond:
(partsupp.ps_partkey = lineitem.l_partkey)
Join Filter:
(lineitem.l_suppkey = partsupp.ps_suppkey)
Rows Removed by Join Filter: 19511964
-> Index Scan using
idx_partsupp_partkey on partsupp (cost=0.43..781956.32 rows=15999792
width=22) (actual time=0.044..11825.481 rows=15999881 loops=1)
-> Sort
(cost=5224967.03..5245348.02 rows=8152396 width=45) (actual
time=103592.505..112205.444 rows=26015949 loops=1)
Sort Key: part.p_partkey
Sort Method: quicksort
Memory: 704733kB
-> Hash Join
(cost=127278.36..4289121.18 rows=8152396 width=45) (actual
time=1084.370..94732.951 rows=6503988 loops=1)
Hash Cond:
(lineitem.l_partkey = part.p_partkey)
-> Seq Scan on
lineitem (cost=0.00..3630339.08 rows=119994608 width=41) (actual
time=0.015..33355.637 rows=119994608 loops=1)
-> Hash
(cost=123743.07..123743.07 rows=282823 width=4) (actual
time=1083.686..1083.686 rows=216867 loops=1)
Buckets:
524288 Batches: 1 Memory Usage: 11721kB
-> Gather
(cost=1000.00..123743.07 rows=282823 width=4) (actual
time=0.418..926.283 rows=216867 loops=1)
Workers
Planned: 4
Workers
Launched: 4
->
Parallel Seq Scan on part (cost=0.00..94460.77 rows=70706 width=4)
(actual time=0.063..962.909 rows=43373 loops=5)
Filter: ((p_name)::text ~~ '%grey%'::text)
Rows Removed by Filter: 756627
The estimate for the index scan on partsupp is essentially perfect,
and the lineitem-part join is off by about 3x. However, the merge
join is off by about 4000x, which is real bad.
On Q20, things go wrong here:
-> Merge Join (cost=5928271.92..6411281.44
rows=278 width=16) (actual time=77887.963..136614.284 rows=118124
loops=1)
Merge Cond: ((lineitem.l_partkey =
partsupp.ps_partkey) AND (lineitem.l_suppkey = partsupp.ps_suppkey))
Join Filter:
((partsupp.ps_availqty)::numeric > ((0.5 * sum(lineitem.l_quantity))))
Rows Removed by Join Filter: 242
-> GroupAggregate
(cost=5363980.40..5691151.45 rows=9681876 width=48) (actual
time=76672.726..131482.677 rows=10890067 loops=1)
Group Key: lineitem.l_partkey,
lineitem.l_suppkey
-> Sort
(cost=5363980.40..5409466.13 rows=18194291 width=21) (actual
time=76672.661..86405.882 rows=18194084 loops=1)
Sort Key: lineitem.l_partkey,
lineitem.l_suppkey
Sort Method: external merge
Disk: 551376kB
-> Bitmap Heap Scan on
lineitem (cost=466716.05..3170023.42 rows=18194291 width=21) (actual
time=13735.552..39289.995 rows=18195269 loops=1)
Recheck Cond:
((l_shipdate >= '1994-01-01'::date) AND (l_shipdate < '1995-01-01
00:00:00'::timestamp without time zone))
Heap Blocks: exact=2230011
-> Bitmap Index Scan on
idx_lineitem_shipdate (cost=0.00..462167.48 rows=18194291 width=0)
(actual time=11771.173..11771.173 rows=18195269 loops=1)
Index Cond:
((l_shipdate >= '1994-01-01'::date) AND (l_shipdate < '1995-01-01
00:00:00'::timestamp without time zone))
-> Sort (cost=564291.52..567827.56
rows=1414417 width=24) (actual time=1214.812..1264.356 rows=173936
loops=1)
Sort Key: partsupp.ps_partkey,
partsupp.ps_suppkey
Sort Method: quicksort Memory: 19733kB
-> Nested Loop
(cost=1000.43..419796.26 rows=1414417 width=24) (actual
time=0.447..985.562 rows=173936 loops=1)
-> Gather
(cost=1000.00..99501.07 rows=40403 width=4) (actual time=0.390..34.476
rows=43484 loops=1)
Workers Planned: 4
Workers Launched: 4
-> Parallel Seq Scan on
part (cost=0.00..94460.77 rows=10101 width=4) (actual
time=0.143..527.665 rows=8697 loops=5)
Filter:
((p_name)::text ~~ 'beige%'::text)
Rows Removed by
Filter: 791303
-> Index Scan using
idx_partsupp_partkey on partsupp (cost=0.43..7.58 rows=35 width=20)
(actual time=0.017..0.019 rows=4 loops=43484)
Index Cond: (ps_partkey =
part.p_partkey)
The estimate for the GroupAggregate feeding one side of the merge join
is quite accurate. The estimate for the part-partsupp join on the
other side is off by 8x. Then things get much worse: the estimate for
the merge join is off by 400x.
I'm not really sure whether the multivariate statistics stuff will fix
this kind of case or not, but if it did it would be awesome.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: WIP: multivariate statistics / proof of concept
2014-10-12 22:00 WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-11-13 11:31 ` Re: WIP: multivariate statistics / proof of concept Simon Riggs <simon@2ndQuadrant.com>
2016-11-21 22:10 ` Re: WIP: multivariate statistics / proof of concept Robert Haas <robertmhaas@gmail.com>
@ 2016-11-22 03:42 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2016-12-02 12:02 ` Re: WIP: multivariate statistics / proof of concept Haribabu Kommi <kommi.haribabu@gmail.com>
0 siblings, 1 reply; 37+ messages in thread
From: Tomas Vondra @ 2016-11-22 03:42 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; Simon Riggs <simon@2ndquadrant.com>; +Cc: Tomas Vondra <tv@fuzzy.cz>; pgsql-hackers
On 11/21/2016 11:10 PM, Robert Haas wrote:
> [ reviving an old multivariate statistics thread ]
>
> On Thu, Nov 13, 2014 at 6:31 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
>> On 12 October 2014 23:00, Tomas Vondra <tv@fuzzy.cz> wrote:
>>
>>> It however seems to be working sufficiently well at this point, enough
>>> to get some useful feedback. So here we go.
>>
>> This looks interesting and useful.
>>
>> What I'd like to check before a detailed review is that this has
>> sufficient applicability to be useful.
>>
>> My understanding is that Q9 and Q18 of TPC-H have poor plans as a
>> result of multi-column stats errors.
>>
>> Could you look at those queries and confirm that this patch can
>> produce better plans for them?
>
> Tomas, did you ever do any testing in this area? One of my
> colleagues, Rafia Sabih, recently did some testing of TPC-H queries @
> 20 GB. Q18 actually doesn't complete at all right now because of an
> issue with the new simplehash implementation. I reported it to Andres
> and he tracked it down, but hasn't posted the patch yet - see
> http://archives.postgresql.org/message-id/20161115192802.jfbec5s6ougxwicp@alap3.anarazel.de
>
> Of the remaining queries, the slowest are Q9 and Q20, and both of them
> have serious estimation errors. On Q9, things go wrong here:
>
> -> Merge Join
> (cost=5225092.04..6595105.57 rows=154 width=47) (actual
> time=103592.821..149335.010 rows=6503988 loops=1)
> Merge Cond:
> (partsupp.ps_partkey = lineitem.l_partkey)
> Join Filter:
> (lineitem.l_suppkey = partsupp.ps_suppkey)
> Rows Removed by Join Filter: 19511964
> -> Index Scan using
> [snip]
>
> Rows Removed by Filter: 756627
>
> The estimate for the index scan on partsupp is essentially perfect,
> and the lineitem-part join is off by about 3x. However, the merge
> join is off by about 4000x, which is real bad.
>
The patch only deals with statistics on base relations, no joins, at
this point. It's meant to be extended in that direction, so the syntax
supports it, but at this point that's all. No joins.
That being said, this estimate should be improved in 9.6, when you
create a foreign key between the tables. In fact, that patch was exactly
about Q9.
This is how the join estimate looks on scale 1 without the FK between
the two tables:
QUERY PLAN
-----------------------------------------------------------------------
Merge Join (cost=19.19..700980.12 rows=2404 width=261)
Merge Cond: ((lineitem.l_partkey = partsupp.ps_partkey) AND
(lineitem.l_suppkey = partsupp.ps_suppkey))
-> Index Scan using idx_lineitem_part_supp on lineitem
(cost=0.43..605856.84 rows=6001117 width=117)
-> Index Scan using partsupp_pkey on partsupp
(cost=0.42..61141.76 rows=800000 width=144)
(4 rows)
and with the foreign key:
QUERY PLAN
-----------------------------------------------------------------------
Merge Join (cost=19.19..700980.12 rows=6001117 width=261)
(actual rows=6001215 loops=1)
Merge Cond: ((lineitem.l_partkey = partsupp.ps_partkey) AND
(lineitem.l_suppkey = partsupp.ps_suppkey))
-> Index Scan using idx_lineitem_part_supp on lineitem
(cost=0.43..605856.84 rows=6001117 width=117)
(actual rows=6001215 loops=1)
-> Index Scan using partsupp_pkey on partsupp
(cost=0.42..61141.76 rows=800000 width=144)
(actual rows=6001672 loops=1)
Planning time: 3.840 ms
Execution time: 21987.913 ms
(6 rows)
> On Q20, things go wrong here:
>
> [snip]
>
> The estimate for the GroupAggregate feeding one side of the merge join
> is quite accurate. The estimate for the part-partsupp join on the
> other side is off by 8x. Then things get much worse: the estimate for
> the merge join is off by 400x.
>
Well, most of the estimation error comes from the join, but sadly the
aggregate makes using the foreign keys impossible - at least in the
current version. I don't know if it can be improved, somehow.
> I'm not really sure whether the multivariate statistics stuff will fix
> this kind of case or not, but if it did it would be awesome.
>
Join statistics are something I'd like to add eventually, but I don't
see how it could happen in the first version. Also, the patch received
no reviews this CF, and making it even larger is unlikely to make it
more attractive.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: WIP: multivariate statistics / proof of concept
2014-10-12 22:00 WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-11-13 11:31 ` Re: WIP: multivariate statistics / proof of concept Simon Riggs <simon@2ndQuadrant.com>
2016-11-21 22:10 ` Re: WIP: multivariate statistics / proof of concept Robert Haas <robertmhaas@gmail.com>
2016-11-22 03:42 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tomas.vondra@2ndquadrant.com>
@ 2016-12-02 12:02 ` Haribabu Kommi <kommi.haribabu@gmail.com>
0 siblings, 0 replies; 37+ messages in thread
From: Haribabu Kommi @ 2016-12-02 12:02 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Robert Haas <robertmhaas@gmail.com>; Simon Riggs <simon@2ndquadrant.com>; Tomas Vondra <tv@fuzzy.cz>; pgsql-hackers
On Tue, Nov 22, 2016 at 2:42 PM, Tomas Vondra <tomas.vondra@2ndquadrant.com>
wrote:
> On 11/21/2016 11:10 PM, Robert Haas wrote:
>
>> [ reviving an old multivariate statistics thread ]
>>
>> On Thu, Nov 13, 2014 at 6:31 AM, Simon Riggs <simon@2ndquadrant.com>
>> wrote:
>>
>>> On 12 October 2014 23:00, Tomas Vondra <tv@fuzzy.cz> wrote:
>>>
>>> It however seems to be working sufficiently well at this point, enough
>>>> to get some useful feedback. So here we go.
>>>>
>>>
>>> This looks interesting and useful.
>>>
>>> What I'd like to check before a detailed review is that this has
>>> sufficient applicability to be useful.
>>>
>>> My understanding is that Q9 and Q18 of TPC-H have poor plans as a
>>> result of multi-column stats errors.
>>>
>>> Could you look at those queries and confirm that this patch can
>>> produce better plans for them?
>>>
>>
>> Tomas, did you ever do any testing in this area? One of my
>> colleagues, Rafia Sabih, recently did some testing of TPC-H queries @
>> 20 GB. Q18 actually doesn't complete at all right now because of an
>> issue with the new simplehash implementation. I reported it to Andres
>> and he tracked it down, but hasn't posted the patch yet - see
>> http://archives.postgresql.org/message-id/20161115192802.jfb
>> ec5s6ougxwicp@alap3.anarazel.de
>>
>> Of the remaining queries, the slowest are Q9 and Q20, and both of them
>> have serious estimation errors. On Q9, things go wrong here:
>>
>> -> Merge Join
>> (cost=5225092.04..6595105.57 rows=154 width=47) (actual
>> time=103592.821..149335.010 rows=6503988 loops=1)
>> Merge Cond:
>> (partsupp.ps_partkey = lineitem.l_partkey)
>> Join Filter:
>> (lineitem.l_suppkey = partsupp.ps_suppkey)
>> Rows Removed by Join Filter:
>> 19511964
>> -> Index Scan using
>>
> > [snip]
>
>>
>> Rows Removed by Filter: 756627
>>
>> The estimate for the index scan on partsupp is essentially perfect,
>> and the lineitem-part join is off by about 3x. However, the merge
>> join is off by about 4000x, which is real bad.
>>
>>
> The patch only deals with statistics on base relations, no joins, at this
> point. It's meant to be extended in that direction, so the syntax supports
> it, but at this point that's all. No joins.
>
> That being said, this estimate should be improved in 9.6, when you create
> a foreign key between the tables. In fact, that patch was exactly about Q9.
>
> This is how the join estimate looks on scale 1 without the FK between the
> two tables:
>
> QUERY PLAN
> -----------------------------------------------------------------------
> Merge Join (cost=19.19..700980.12 rows=2404 width=261)
> Merge Cond: ((lineitem.l_partkey = partsupp.ps_partkey) AND
> (lineitem.l_suppkey = partsupp.ps_suppkey))
> -> Index Scan using idx_lineitem_part_supp on lineitem
> (cost=0.43..605856.84 rows=6001117 width=117)
> -> Index Scan using partsupp_pkey on partsupp
> (cost=0.42..61141.76 rows=800000 width=144)
> (4 rows)
>
>
> and with the foreign key:
>
> QUERY PLAN
> -----------------------------------------------------------------------
> Merge Join (cost=19.19..700980.12 rows=6001117 width=261)
> (actual rows=6001215 loops=1)
> Merge Cond: ((lineitem.l_partkey = partsupp.ps_partkey) AND
> (lineitem.l_suppkey = partsupp.ps_suppkey))
> -> Index Scan using idx_lineitem_part_supp on lineitem
> (cost=0.43..605856.84 rows=6001117 width=117)
> (actual rows=6001215 loops=1)
> -> Index Scan using partsupp_pkey on partsupp
> (cost=0.42..61141.76 rows=800000 width=144)
> (actual rows=6001672 loops=1)
> Planning time: 3.840 ms
> Execution time: 21987.913 ms
> (6 rows)
>
>
> On Q20, things go wrong here:
>>
> >
>
>> [snip]
>>
>> The estimate for the GroupAggregate feeding one side of the merge join
>> is quite accurate. The estimate for the part-partsupp join on the
>> other side is off by 8x. Then things get much worse: the estimate for
>> the merge join is off by 400x.
>>
>>
> Well, most of the estimation error comes from the join, but sadly the
> aggregate makes using the foreign keys impossible - at least in the current
> version. I don't know if it can be improved, somehow.
>
> I'm not really sure whether the multivariate statistics stuff will fix
>> this kind of case or not, but if it did it would be awesome.
>>
>>
> Join statistics are something I'd like to add eventually, but I don't see
> how it could happen in the first version. Also, the patch received no
> reviews this CF, and making it even larger is unlikely to make it more
> attractive.
>
Moved to next CF with "needs review" status.
Regards,
Hari Babu
Fujitsu Australia
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: WIP: multivariate statistics / proof of concept
2014-10-12 22:00 WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
@ 2014-12-11 16:53 ` Heikki Linnakangas <hlinnakangas@vmware.com>
2014-12-11 20:07 ` Re: WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2016-01-19 08:34 ` Re: WIP: multivariate statistics / proof of concept Gavin Flower <GavinFlower@archidevsys.co.nz>
3 siblings, 2 replies; 37+ messages in thread
From: Heikki Linnakangas @ 2014-12-11 16:53 UTC (permalink / raw)
To: Tomas Vondra <tv@fuzzy.cz>; pgsql-hackers
On 10/13/2014 01:00 AM, Tomas Vondra wrote:
> Hi,
>
> attached is a WIP patch implementing multivariate statistics.
Great! Really glad to see you working on this.
> + * FIXME This sample sizing is mostly OK when computing stats for
> + * individual columns, but when computing multi-variate stats
> + * for multivariate stats (histograms, mcv, ...) it's rather
> + * insufficient. For small number of dimensions it works, but
> + * for complex stats it'd be nice use sample proportional to
> + * the table (say, 0.5% - 1%) instead of a fixed size.
I don't think a fraction of the table is appropriate. As long as the
sample is random, the accuracy of a sample doesn't depend much on the
size of the population. For example, if you sample 1,000 rows from a
table with 100,000 rows, or 1000 rows from a table with 100,000,000
rows, the accuracy is pretty much the same. That doesn't change when you
go from a single variable to multiple variables.
You do need a bigger sample with multiple variables, however. My gut
feeling is that if you sample N rows for a single variable, with two
variables you need to sample N^2 rows to get the same accuracy. But it's
not proportional to the table size. (I have no proof for that, but I'm
sure there is literature on this.)
> + * Multivariate histograms
> + *
> + * Histograms are a collection of buckets, represented by n-dimensional
> + * rectangles. Each rectangle is delimited by an array of lower and
> + * upper boundaries, so that for for the i-th attribute
> + *
> + * min[i] <= value[i] <= max[i]
> + *
> + * Each bucket tracks frequency (fraction of tuples it contains),
> + * information about the inequalities, number of distinct values in
> + * each dimension (which is used when building the histogram) etc.
> + *
> + * The boundaries may be either inclusive or exclusive, or the whole
> + * dimension may be NULL.
> + *
> + * The buckets may overlap (assuming the build algorithm keeps the
> + * frequencies additive) or may not cover the whole space (i.e. allow
> + * gaps). This entirely depends on the algorithm used to build the
> + * histogram.
That sounds pretty exotic. These buckets are quite different from the
single-dimension buckets we currently have.
The paper you reference in partition_bucket() function, M.
Muralikrishna, David J. DeWitt: Equi-Depth Histograms For Estimating
Selectivity Factors For Multi-Dimensional Queries. SIGMOD Conference
1988: 28-36, actually doesn't mention overlapping buckets at all. I
haven't read the code in detail, but if it implements the algorithm from
that paper, there will be no overlap.
- Heikki
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: WIP: multivariate statistics / proof of concept
2014-10-12 22:00 WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-12-11 16:53 ` Re: WIP: multivariate statistics / proof of concept Heikki Linnakangas <hlinnakangas@vmware.com>
@ 2014-12-11 20:07 ` Tomas Vondra <tv@fuzzy.cz>
1 sibling, 0 replies; 37+ messages in thread
From: Tomas Vondra @ 2014-12-11 20:07 UTC (permalink / raw)
To: pgsql-hackers
On 11.12.2014 17:53, Heikki Linnakangas wrote:
> On 10/13/2014 01:00 AM, Tomas Vondra wrote:
>> Hi,
>>
>> attached is a WIP patch implementing multivariate statistics.
>
> Great! Really glad to see you working on this.
>
>> + * FIXME This sample sizing is mostly OK when computing stats for
>> + * individual columns, but when computing multi-variate stats
>> + * for multivariate stats (histograms, mcv, ...) it's rather
>> + * insufficient. For small number of dimensions it works, but
>> + * for complex stats it'd be nice use sample proportional to
>> + * the table (say, 0.5% - 1%) instead of a fixed size.
>
> I don't think a fraction of the table is appropriate. As long as the
> sample is random, the accuracy of a sample doesn't depend much on
> the size of the population. For example, if you sample 1,000 rows
> from a table with 100,000 rows, or 1000 rows from a table with
> 100,000,000 rows, the accuracy is pretty much the same. That doesn't
> change when you go from a single variable to multiple variables.
I might be wrong, but I doubt that. First, I read a number of papers
while working on this patch, and all of them used samples proportional
to the data set. That's an indirect evidence, though.
> You do need a bigger sample with multiple variables, however. My gut
> feeling is that if you sample N rows for a single variable, with two
> variables you need to sample N^2 rows to get the same accuracy. But
> it's not proportional to the table size. (I have no proof for that,
> but I'm sure there is literature on this.)
Maybe. I think it's somehow related to the number of buckets (which
somehow determines the precision of the histogram). If you want 1000
buckets, the number of rows scanned needs to be e.g. 10x that. With
multi-variate histograms, we may shoot for more buckets (say, 100 in
each dimension).
>
>> + * Multivariate histograms
>> + *
>> + * Histograms are a collection of buckets, represented by n-dimensional
>> + * rectangles. Each rectangle is delimited by an array of lower and
>> + * upper boundaries, so that for for the i-th attribute
>> + *
>> + * min[i] <= value[i] <= max[i]
>> + *
>> + * Each bucket tracks frequency (fraction of tuples it contains),
>> + * information about the inequalities, number of distinct values in
>> + * each dimension (which is used when building the histogram) etc.
>> + *
>> + * The boundaries may be either inclusive or exclusive, or the whole
>> + * dimension may be NULL.
>> + *
>> + * The buckets may overlap (assuming the build algorithm keeps the
>> + * frequencies additive) or may not cover the whole space (i.e. allow
>> + * gaps). This entirely depends on the algorithm used to build the
>> + * histogram.
>
> That sounds pretty exotic. These buckets are quite different from
> the single-dimension buckets we currently have.
>
> The paper you reference in partition_bucket() function, M.
> Muralikrishna, David J. DeWitt: Equi-Depth Histograms For Estimating
> Selectivity Factors For Multi-Dimensional Queries. SIGMOD Conference
> 1988: 28-36, actually doesn't mention overlapping buckets at all. I
> haven't read the code in detail, but if it implements the algorithm
> from that paper, there will be no overlap.
The algorithm implemented in partition_bucket() is very simple and
naive, and it mostly resembles the algorithm described in the paper. I'm
sure there are differences, it's not a 1:1 implementation, but you're
right it produces non-overlapping buckets.
The point is that I envision more complex algorithms or different
histogram types, and some of them may produce overlapping buckets. Maybe
that's premature comment, and it will turn out it's not really necessary.
regards
Tomas
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 37+ messages in thread
* Re: WIP: multivariate statistics / proof of concept
2014-10-12 22:00 WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-12-11 16:53 ` Re: WIP: multivariate statistics / proof of concept Heikki Linnakangas <hlinnakangas@vmware.com>
@ 2016-01-19 08:34 ` Gavin Flower <GavinFlower@archidevsys.co.nz>
1 sibling, 0 replies; 37+ messages in thread
From: Gavin Flower @ 2016-01-19 08:34 UTC (permalink / raw)
To: Heikki Linnakangas <hlinnakangas@vmware.com>; Tomas Vondra <tv@fuzzy.cz>; pgsql-hackers
On 12/12/14 05:53, Heikki Linnakangas wrote:
> On 10/13/2014 01:00 AM, Tomas Vondra wrote:
>> Hi,
>>
>> attached is a WIP patch implementing multivariate statistics.
>
> Great! Really glad to see you working on this.
>
>> + * FIXME This sample sizing is mostly OK when computing stats for
>> + * individual columns, but when computing multi-variate stats
>> + * for multivariate stats (histograms, mcv, ...) it's rather
>> + * insufficient. For small number of dimensions it works, but
>> + * for complex stats it'd be nice use sample proportional to
>> + * the table (say, 0.5% - 1%) instead of a fixed size.
>
> I don't think a fraction of the table is appropriate. As long as the
> sample is random, the accuracy of a sample doesn't depend much on the
> size of the population. For example, if you sample 1,000 rows from a
> table with 100,000 rows, or 1000 rows from a table with 100,000,000
> rows, the accuracy is pretty much the same. That doesn't change when
> you go from a single variable to multiple variables.
>
> You do need a bigger sample with multiple variables, however. My gut
> feeling is that if you sample N rows for a single variable, with two
> variables you need to sample N^2 rows to get the same accuracy. But
> it's not proportional to the table size. (I have no proof for that,
> but I'm sure there is literature on this.)
[...]
I did stage III statistics at University many moons ago...
The accuracy of the sample only depends on the value of N, not the total
size of the population, with the obvious constraint that N <= population
size.
The standard deviation in a random sample is proportional to the square
root of N. So using N = 100 would have a standard deviation of about
10%, so to reduce it to 5% you would need N = 400.
For multiple variables, it will also be a function of N - I don't recall
precisely how, I suspect it might M * N were M is the number of
parameters (but I'm not as certain). I think M^N might be needed if you
want all the possible correlations between sets of variable to be
reasonably significant - but I'm mostly just guessing here.
So using a % of table size is somewhat silly, looking at the above.
However, if you want to detect frequencies that occur at the 1% level,
then you will need to sample 1% of the table or greater. So which
approach is 'best', depends on what you are trying to determine. The
sample size is more useful when you need to decide between 2 different
hypothesises.
The sampling methodology, is far more important than the ratio of N to
population size - consider the bias imposed by using random telephone
numbers, even before the event of mobile phones!
Cheers,
Gavin
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 37+ messages in thread
end of thread, other threads:[~2016-12-02 12:02 UTC | newest]
Thread overview: 37+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2014-10-12 22:00 WIP: multivariate statistics / proof of concept Tomas Vondra <tv@fuzzy.cz>
2014-10-13 07:36 ` Albe Laurenz <laurenz.albe@wien.gv.at>
2014-10-13 19:47 ` Tomas Vondra <tv@fuzzy.cz>
2014-10-29 09:41 ` David Rowley <dgrowleyml@gmail.com>
2014-10-29 11:21 ` Tomas Vondra <tv@fuzzy.cz>
2014-10-30 09:23 ` David Rowley <dgrowleyml@gmail.com>
2014-11-10 02:34 ` Tomas Vondra <tv@fuzzy.cz>
2014-10-29 11:31 ` Petr Jelinek <petr@2ndquadrant.com>
2014-10-29 11:48 ` Tomas Vondra <tv@fuzzy.cz>
2014-10-30 09:17 ` David Rowley <dgrowleyml@gmail.com>
2014-10-30 10:29 ` Tomas Vondra <tv@fuzzy.cz>
2014-11-13 11:31 ` Simon Riggs <simon@2ndQuadrant.com>
2014-11-13 13:11 ` Tomas Vondra <tv@fuzzy.cz>
2014-11-13 15:51 ` Katharina Büchse <katharina.buechse@uni-jena.de>
2014-11-13 16:42 ` Tomas Vondra <tv@fuzzy.cz>
2014-11-15 17:49 ` Kevin Grittner <kgrittn@ymail.com>
2014-11-15 18:35 ` Tomas Vondra <tv@fuzzy.cz>
2014-12-08 01:01 ` Michael Paquier <michael.paquier@gmail.com>
2014-12-09 20:15 ` Tomas Vondra <tv@fuzzy.cz>
2014-12-15 02:55 ` Michael Paquier <michael.paquier@gmail.com>
2015-01-15 08:00 ` Michael Paquier <michael.paquier@gmail.com>
2015-01-24 20:21 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-20 08:33 ` Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
2015-03-20 21:17 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-24 05:34 ` Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
2015-03-24 10:24 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-03-31 00:26 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2015-04-28 16:09 ` Jeff Janes <jeff.janes@gmail.com>
2015-04-28 16:13 ` Stephen Frost <sfrost@snowman.net>
2015-04-28 17:36 ` Jeff Janes <jeff.janes@gmail.com>
2015-04-28 18:15 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2016-11-21 22:10 ` Robert Haas <robertmhaas@gmail.com>
2016-11-22 03:42 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2016-12-02 12:02 ` Haribabu Kommi <kommi.haribabu@gmail.com>
2014-12-11 16:53 ` Heikki Linnakangas <hlinnakangas@vmware.com>
2014-12-11 20:07 ` Tomas Vondra <tv@fuzzy.cz>
2016-01-19 08:34 ` Gavin Flower <GavinFlower@archidevsys.co.nz>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox