agora inbox for [email protected]
help / color / mirror / Atom feed[PATCH 1/2] multivariate MCV lists
8+ messages / 4 participants
[nested] [flat]
* [PATCH 1/2] multivariate MCV lists
@ 2018-09-01 15:48 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 8+ messages in thread
From: Tomas Vondra @ 2018-09-01 15:48 UTC (permalink / raw)
---
doc/src/sgml/catalogs.sgml | 13 +-
doc/src/sgml/func.sgml | 77 +
doc/src/sgml/planstats.sgml | 114 +-
doc/src/sgml/ref/create_statistics.sgml | 28 +-
src/backend/commands/analyze.c | 8 +-
src/backend/commands/statscmds.c | 107 +-
src/backend/optimizer/path/clausesel.c | 89 +-
src/backend/optimizer/util/plancat.c | 12 +
src/backend/parser/parse_utilcmd.c | 2 +
src/backend/statistics/Makefile | 2 +-
src/backend/statistics/README | 4 +
src/backend/statistics/README.mcv | 140 ++
src/backend/statistics/dependencies.c | 83 +-
src/backend/statistics/extended_stats.c | 543 ++++++-
src/backend/statistics/mcv.c | 1656 ++++++++++++++++++++++
src/backend/statistics/mvdistinct.c | 28 -
src/backend/utils/adt/ruleutils.c | 24 +-
src/backend/utils/adt/selfuncs.c | 165 +++
src/bin/psql/describe.c | 9 +-
src/include/catalog/pg_cast.dat | 6 +
src/include/catalog/pg_proc.dat | 24 +
src/include/catalog/pg_statistic_ext.h | 2 +
src/include/catalog/pg_type.dat | 7 +
src/include/commands/vacuum.h | 6 +
src/include/optimizer/cost.h | 6 +
src/include/statistics/extended_stats_internal.h | 43 +
src/include/statistics/statistics.h | 49 +
src/include/utils/selfuncs.h | 2 +
src/test/regress/expected/create_table_like.out | 2 +-
src/test/regress/expected/opr_sanity.out | 3 +-
src/test/regress/expected/stats_ext.out | 319 ++++-
src/test/regress/expected/type_sanity.out | 3 +-
src/test/regress/sql/stats_ext.sql | 181 +++
33 files changed, 3630 insertions(+), 127 deletions(-)
create mode 100644 src/backend/statistics/README.mcv
create mode 100644 src/backend/statistics/mcv.c
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 07e8b3325f..dc7bbe5173 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -6571,7 +6571,8 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
An array containing codes for the enabled statistic kinds;
valid values are:
<literal>d</literal> for n-distinct statistics,
- <literal>f</literal> for functional dependency statistics
+ <literal>f</literal> for functional dependency statistics, and
+ <literal>m</literal> for most common values (MCV) list statistics
</entry>
</row>
@@ -6594,6 +6595,16 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</entry>
</row>
+ <row>
+ <entry><structfield>stxmcv</structfield></entry>
+ <entry><type>pg_mcv_list</type></entry>
+ <entry></entry>
+ <entry>
+ MCV (most-common values) list statistics, serialized as
+ <structname>pg_mcv_list</structname> type.
+ </entry>
+ </row>
+
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index bb794e044f..69cfe7bbe9 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -20917,4 +20917,81 @@ CREATE EVENT TRIGGER test_table_rewrite_oid
</sect2>
</sect1>
+ <sect1 id="functions-statistics">
+ <title>Statistics Information Functions</title>
+
+ <indexterm zone="functions-statistics">
+ <primary>function</primary>
+ <secondary>statistics</secondary>
+ </indexterm>
+
+ <para>
+ To inspect statistics defined using <command>CREATE STATISTICS</command>
+ command, <productname>PostgreSQL</productname> provides multiple functions.
+ </para>
+
+ <sect2 id="functions-statistics-mcv">
+ <title>Inspecting MCV lists</title>
+
+ <indexterm>
+ <primary>pg_mcv_list_items</primary>
+ <secondary>pg_mcv_list</secondary>
+ </indexterm>
+
+ <para>
+ <function>pg_mcv_list_items</function> returns a list of all items
+ stored in a multi-column <literal>MCV</literal> list, and returns the
+ following columns:
+
+ <informaltable>
+ <tgroup cols="3">
+ <thead>
+ <row>
+ <entry>Name</entry>
+ <entry>Type</entry>
+ <entry>Description</entry>
+ </row>
+ </thead>
+
+ <tbody>
+ <row>
+ <entry><literal>index</literal></entry>
+ <entry><type>int</type></entry>
+ <entry>index of the item in the <literal>MCV</literal> list</entry>
+ </row>
+ <row>
+ <entry><literal>values</literal></entry>
+ <entry><type>text[]</type></entry>
+ <entry>values stored in the MCV item</entry>
+ </row>
+ <row>
+ <entry><literal>nulls</literal></entry>
+ <entry><type>boolean[]</type></entry>
+ <entry>flags identifying <literal>NULL</literal> values</entry>
+ </row>
+ <row>
+ <entry><literal>frequency</literal></entry>
+ <entry><type>double precision</type></entry>
+ <entry>frequency of this <literal>MCV</literal> item</entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ </para>
+
+ <para>
+ The <function>pg_mcv_list_items</function> function can be used like this:
+
+<programlisting>
+SELECT m.* FROM pg_statistic_ext,
+ pg_mcv_list_items(stxmcv) m WHERE stxname = 'stts';
+</programlisting>
+
+ Values of the <type>pg_mcv_list</type> can be obtained only from the
+ <literal>pg_statistic.stxmcv</literal> column.
+ </para>
+ </sect2>
+
+ </sect1>
+
</chapter>
diff --git a/doc/src/sgml/planstats.sgml b/doc/src/sgml/planstats.sgml
index ef643ad064..de8ef165c9 100644
--- a/doc/src/sgml/planstats.sgml
+++ b/doc/src/sgml/planstats.sgml
@@ -455,7 +455,7 @@ rows = (outer_cardinality * inner_cardinality) * selectivity
<secondary>multivariate</secondary>
</indexterm>
- <sect2>
+ <sect2 id="functional-dependencies">
<title>Functional Dependencies</title>
<para>
@@ -540,7 +540,7 @@ EXPLAIN (ANALYZE, TIMING OFF) SELECT * FROM t WHERE a = 1 AND b = 1;
</para>
</sect2>
- <sect2>
+ <sect2 id="multivariate-ndistinct-counts">
<title>Multivariate N-Distinct Counts</title>
<para>
@@ -585,6 +585,116 @@ EXPLAIN (ANALYZE, TIMING OFF) SELECT COUNT(*) FROM t GROUP BY a, b;
</para>
</sect2>
+
+ <sect2 id="mcv-lists">
+ <title>MCV lists</title>
+
+ <para>
+ As explained in <xref linkend="functional-dependencies"/>, functional
+ dependencies are very cheap and efficient type of statistics, but their
+ main limitation is their global nature (only tracking dependencies at
+ the column level, not between individual column values).
+ </para>
+
+ <para>
+ This section introduces multivariate variant of <acronym>MCV</acronym>
+ (most-common values) lists, a straight-forward extension of the per-column
+ statistics described in <xref linkend="row-estimation-examples"/>. This
+ statistics adresses the limitation by storing individual values, but it
+ is naturally more expensive, both in terms of storage and planning time.
+ </para>
+
+ <para>
+ Let's look at the query from <xref linkend="functional-dependencies"/>
+ again, but this time with a <acronym>MCV</acronym> list created on the
+ same set of columns (be sure to drop the functional dependencies, to
+ make sure the planner uses the newly created statistics).
+
+<programlisting>
+DROP STATISTICS stts;
+CREATE STATISTICS stts2 (mcv) ON a, b FROM t;
+ANALYZE t;
+EXPLAIN (ANALYZE, TIMING OFF) SELECT * FROM t WHERE a = 1 AND b = 1;
+ QUERY PLAN
+-------------------------------------------------------------------------------
+ Seq Scan on t (cost=0.00..195.00 rows=100 width=8) (actual rows=100 loops=1)
+ Filter: ((a = 1) AND (b = 1))
+ Rows Removed by Filter: 9900
+</programlisting>
+
+ The estimate is as accurate as with the functional dependencies, mostly
+ thanks to the table being fairly small and having a simple distribution
+ with low number of distinct values. Before looking at the second query,
+ which was not handled by functional dependencies particularly well,
+ let's inspect the <acronym>MCV</acronym> list a bit.
+ </para>
+
+ <para>
+ Inspecting the <acronym>MCV</acronym> list is possible using
+ <function>pg_mcv_list_items</function> set-returning function.
+
+<programlisting>
+SELECT m.* FROM pg_statistic_ext,
+ pg_mcv_list_items(stxmcv) m WHERE stxname = 'stts2';
+ index | values | nulls | frequency
+-------+---------+-------+-----------
+ 0 | {0,0} | {f,f} | 0.01
+ 1 | {1,1} | {f,f} | 0.01
+ 2 | {2,2} | {f,f} | 0.01
+...
+ 49 | {49,49} | {f,f} | 0.01
+ 50 | {50,0} | {f,f} | 0.01
+...
+ 97 | {97,47} | {f,f} | 0.01
+ 98 | {98,48} | {f,f} | 0.01
+ 99 | {99,49} | {f,f} | 0.01
+(100 rows)
+</programlisting>
+
+ Which confirms there are 100 distinct combinations in the two columns,
+ and all of them are about equally likely (1% frequency for each one).
+ Had there been any null values in either of the columns, this would be
+ identified in the <structfield>nulls</structfield> column.
+ </para>
+
+ <para>
+ When estimating the selectivity, the planner applies all the conditions
+ on items in the <acronym>MCV</acronym> list, and them sums the frequencies
+ of the matching ones. See <function>mcv_clauselist_selectivity</function>
+ in <filename>src/backend/statistics/mcv.c</filename> for details.
+ </para>
+
+ <para>
+ Compared to functional dependencies, <acronym>MCV</acronym> lists have two
+ major advantages. Firstly, the list stores actual values, making it possible
+ to decide which combinations are compatible.
+
+<programlisting>
+EXPLAIN (ANALYZE, TIMING OFF) SELECT * FROM t WHERE a = 1 AND b = 10;
+ QUERY PLAN
+---------------------------------------------------------------------------
+ Seq Scan on t (cost=0.00..195.00 rows=1 width=8) (actual rows=0 loops=1)
+ Filter: ((a = 1) AND (b = 10))
+ Rows Removed by Filter: 10000
+</programlisting>
+
+ Secondly, <acronym>MCV</acronym> lists handle a wider range of clause types,
+ not just equality clauses like functional dependencies. See for example the
+ example range query, presented earlier:
+
+<programlisting>
+EXPLAIN (ANALYZE, TIMING OFF) SELECT * FROM t WHERE a <= 49 AND b > 49;
+ QUERY PLAN
+---------------------------------------------------------------------------
+ Seq Scan on t (cost=0.00..195.00 rows=1 width=8) (actual rows=0 loops=1)
+ Filter: ((a <= 49) AND (b > 49))
+ Rows Removed by Filter: 10000
+</programlisting>
+
+ </para>
+
+ </sect2>
+
</sect1>
<sect1 id="planner-stats-security">
diff --git a/doc/src/sgml/ref/create_statistics.sgml b/doc/src/sgml/ref/create_statistics.sgml
index 539f5bded5..fcbfa569d0 100644
--- a/doc/src/sgml/ref/create_statistics.sgml
+++ b/doc/src/sgml/ref/create_statistics.sgml
@@ -83,7 +83,8 @@ CREATE STATISTICS [ IF NOT EXISTS ] <replaceable class="parameter">statistics_na
Currently supported kinds are
<literal>ndistinct</literal>, which enables n-distinct statistics, and
<literal>dependencies</literal>, which enables functional
- dependency statistics.
+ dependency statistics, and <literal>mcv</literal> which enables
+ most-common values lists.
If this clause is omitted, all supported statistics kinds are
included in the statistics object.
For more information, see <xref linkend="planner-stats-extended"/>
@@ -164,6 +165,31 @@ EXPLAIN ANALYZE SELECT * FROM t1 WHERE (a = 1) AND (b = 0);
conditions are redundant and does not underestimate the row count.
</para>
+ <para>
+ Create table <structname>t2</structname> with two perfectly correlated columns
+ (containing identical data), and a MCV list on those columns:
+
+<programlisting>
+CREATE TABLE t2 (
+ a int,
+ b int
+);
+
+INSERT INTO t2 SELECT mod(i,100), mod(i,100)
+ FROM generate_series(1,1000000) s(i);
+
+CREATE STATISTICS s2 WITH (mcv) ON (a, b) FROM t2;
+
+ANALYZE t2;
+
+-- valid combination (found in MCV)
+EXPLAIN ANALYZE SELECT * FROM t2 WHERE (a = 1) AND (b = 1);
+
+-- invalid combination (not found in MCV)
+EXPLAIN ANALYZE SELECT * FROM t2 WHERE (a = 1) AND (b = 2);
+</programlisting>
+ </para>
+
</refsect1>
<refsect1>
diff --git a/src/backend/commands/analyze.c b/src/backend/commands/analyze.c
index edbdce81f2..bc369df4cd 100644
--- a/src/backend/commands/analyze.c
+++ b/src/backend/commands/analyze.c
@@ -1759,12 +1759,6 @@ static void compute_scalar_stats(VacAttrStatsP stats,
double totalrows);
static int compare_scalars(const void *a, const void *b, void *arg);
static int compare_mcvs(const void *a, const void *b);
-static int analyze_mcv_list(int *mcv_counts,
- int num_mcv,
- double stadistinct,
- double stanullfrac,
- int samplerows,
- double totalrows);
/*
@@ -2859,7 +2853,7 @@ compare_mcvs(const void *a, const void *b)
* number that are significantly more common than the values not in the list,
* and which are therefore deemed worth storing in the table's MCV list.
*/
-static int
+int
analyze_mcv_list(int *mcv_counts,
int num_mcv,
double stadistinct,
diff --git a/src/backend/commands/statscmds.c b/src/backend/commands/statscmds.c
index 3bb0d24cd2..903d8155e0 100644
--- a/src/backend/commands/statscmds.c
+++ b/src/backend/commands/statscmds.c
@@ -70,11 +70,12 @@ CreateStatistics(CreateStatsStmt *stmt)
Oid relid;
ObjectAddress parentobject,
myself;
- Datum types[2]; /* one for each possible type of statistic */
+ Datum types[3]; /* one for each possible type of statistic */
int ntypes;
ArrayType *stxkind;
bool build_ndistinct;
bool build_dependencies;
+ bool build_mcv;
bool requested_type = false;
int i;
ListCell *cell;
@@ -269,6 +270,7 @@ CreateStatistics(CreateStatsStmt *stmt)
*/
build_ndistinct = false;
build_dependencies = false;
+ build_mcv = false;
foreach(cell, stmt->stat_types)
{
char *type = strVal((Value *) lfirst(cell));
@@ -283,6 +285,11 @@ CreateStatistics(CreateStatsStmt *stmt)
build_dependencies = true;
requested_type = true;
}
+ else if (strcmp(type, "mcv") == 0)
+ {
+ build_mcv = true;
+ requested_type = true;
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -294,6 +301,7 @@ CreateStatistics(CreateStatsStmt *stmt)
{
build_ndistinct = true;
build_dependencies = true;
+ build_mcv = true;
}
/* construct the char array of enabled statistic types */
@@ -302,6 +310,8 @@ CreateStatistics(CreateStatsStmt *stmt)
types[ntypes++] = CharGetDatum(STATS_EXT_NDISTINCT);
if (build_dependencies)
types[ntypes++] = CharGetDatum(STATS_EXT_DEPENDENCIES);
+ if (build_mcv)
+ types[ntypes++] = CharGetDatum(STATS_EXT_MCV);
Assert(ntypes > 0 && ntypes <= lengthof(types));
stxkind = construct_array(types, ntypes, CHAROID, 1, true, 'c');
@@ -320,6 +330,7 @@ CreateStatistics(CreateStatsStmt *stmt)
/* no statistics built yet */
nulls[Anum_pg_statistic_ext_stxndistinct - 1] = true;
nulls[Anum_pg_statistic_ext_stxdependencies - 1] = true;
+ nulls[Anum_pg_statistic_ext_stxmcv - 1] = true;
/* insert it into pg_statistic_ext */
statrel = heap_open(StatisticExtRelationId, RowExclusiveLock);
@@ -415,23 +426,97 @@ RemoveStatisticsById(Oid statsOid)
* null until the next ANALYZE. (Note that the type change hasn't actually
* happened yet, so one option that's *not* on the table is to recompute
* immediately.)
+ *
+ * For both ndistinct and functional-dependencies stats, the on-disk
+ * representation is independent of the source column data types, and it is
+ * plausible to assume that the old statistic values will still be good for
+ * the new column contents. (Obviously, if the ALTER COLUMN TYPE has a USING
+ * expression that substantially alters the semantic meaning of the column
+ * values, this assumption could fail. But that seems like a corner case
+ * that doesn't justify zapping the stats in common cases.)
+ *
+ * For MCV lists that's not the case, as those statistics store the datums
+ * internally. In this case we simply reset the statistics value to NULL.
*/
void
UpdateStatisticsForTypeChange(Oid statsOid, Oid relationOid, int attnum,
Oid oldColumnType, Oid newColumnType)
{
+ Form_pg_statistic_ext staForm;
+ HeapTuple stup,
+ oldtup;
+ int i;
+
+ /* Do we need to reset anything? */
+ bool attribute_referenced;
+ bool reset_stats = false;
+
+ Relation rel;
+
+ Datum values[Natts_pg_statistic_ext];
+ bool nulls[Natts_pg_statistic_ext];
+ bool replaces[Natts_pg_statistic_ext];
+
+ oldtup = SearchSysCache1(STATEXTOID, ObjectIdGetDatum(statsOid));
+ if (!oldtup)
+ elog(ERROR, "cache lookup failed for statistics object %u", statsOid);
+ staForm = (Form_pg_statistic_ext) GETSTRUCT(oldtup);
+
+ /*
+ * If the modified attribute is not referenced by this statistic, we
+ * can simply leave the statistics alone.
+ */
+ attribute_referenced = false;
+ for (i = 0; i < staForm->stxkeys.dim1; i++)
+ if (attnum == staForm->stxkeys.values[i])
+ attribute_referenced = true;
+
/*
- * Currently, we don't actually need to do anything here. For both
- * ndistinct and functional-dependencies stats, the on-disk representation
- * is independent of the source column data types, and it is plausible to
- * assume that the old statistic values will still be good for the new
- * column contents. (Obviously, if the ALTER COLUMN TYPE has a USING
- * expression that substantially alters the semantic meaning of the column
- * values, this assumption could fail. But that seems like a corner case
- * that doesn't justify zapping the stats in common cases.)
- *
- * Future types of extended stats will likely require us to work harder.
+ * We can also leave the record as it is if there are no statistics
+ * including the datum values, like for example MCV lists.
*/
+ if (statext_is_kind_built(oldtup, STATS_EXT_MCV))
+ reset_stats = true;
+
+ /*
+ * If we can leave the statistics as it is, just do minimal cleanup
+ * and we're done.
+ */
+ if (!attribute_referenced && reset_stats)
+ {
+ ReleaseSysCache(oldtup);
+ return;
+ }
+
+ /*
+ * OK, we need to reset some statistics. So let's build the new tuple,
+ * replacing the affected statistics types with NULL.
+ */
+ memset(nulls, 0, Natts_pg_statistic_ext * sizeof(bool));
+ memset(replaces, 0, Natts_pg_statistic_ext * sizeof(bool));
+ memset(values, 0, Natts_pg_statistic_ext * sizeof(Datum));
+
+ if (statext_is_kind_built(oldtup, STATS_EXT_MCV))
+ {
+ replaces[Anum_pg_statistic_ext_stxmcv - 1] = true;
+ nulls[Anum_pg_statistic_ext_stxmcv - 1] = true;
+ }
+
+ rel = heap_open(StatisticExtRelationId, RowExclusiveLock);
+
+ /* replace the old tuple */
+ stup = heap_modify_tuple(oldtup,
+ RelationGetDescr(rel),
+ values,
+ nulls,
+ replaces);
+
+ ReleaseSysCache(oldtup);
+ CatalogTupleUpdate(rel, &stup->t_self, stup);
+
+ heap_freetuple(stup);
+
+ heap_close(rel, RowExclusiveLock);
}
/*
diff --git a/src/backend/optimizer/path/clausesel.c b/src/backend/optimizer/path/clausesel.c
index f4717942c3..fd09ad3c49 100644
--- a/src/backend/optimizer/path/clausesel.c
+++ b/src/backend/optimizer/path/clausesel.c
@@ -105,9 +105,6 @@ clauselist_selectivity(PlannerInfo *root,
Selectivity s1 = 1.0;
RelOptInfo *rel;
Bitmapset *estimatedclauses = NULL;
- RangeQueryClause *rqlist = NULL;
- ListCell *l;
- int listidx;
/*
* If there's exactly one clause, just go directly to
@@ -125,6 +122,25 @@ clauselist_selectivity(PlannerInfo *root,
if (rel && rel->rtekind == RTE_RELATION && rel->statlist != NIL)
{
/*
+ * Estimate selectivity on any clauses applicable by stats tracking
+ * actual values first, then apply functional dependencies on the
+ * remaining clauses. The reasoning for this particular order is that
+ * the more complex stats can track more complex correlations between
+ * the attributes, and may be considered more reliable.
+ *
+ * For example MCV list can give us an exact selectivity for values in
+ * two columns, while functional dependencies can only provide
+ * information about overall strength of the dependency.
+ *
+ * 'estimatedclauses' is a bitmap of 0-based list positions of clauses
+ * used that way, so that we can ignore them later (not to estimate
+ * them twice).
+ */
+ s1 *= statext_clauselist_selectivity(root, clauses, varRelid,
+ jointype, sjinfo, rel,
+ &estimatedclauses);
+
+ /*
* Perform selectivity estimations on any clauses found applicable by
* dependencies_clauselist_selectivity. 'estimatedclauses' will be
* filled with the 0-based list positions of clauses used that way, so
@@ -133,17 +149,72 @@ clauselist_selectivity(PlannerInfo *root,
s1 *= dependencies_clauselist_selectivity(root, clauses, varRelid,
jointype, sjinfo, rel,
&estimatedclauses);
-
- /*
- * This would be the place to apply any other types of extended
- * statistics selectivity estimations for remaining clauses.
- */
}
/*
* Apply normal selectivity estimates for remaining clauses. We'll be
* careful to skip any clauses which were already estimated above.
- *
+ */
+ return s1 * clauselist_selectivity_simple(root, clauses, varRelid,
+ jointype, sjinfo,
+ estimatedclauses);
+}
+
+/*
+ * clauselist_selectivity_simple -
+ * Compute the selectivity of an implicitly-ANDed list of boolean
+ * expression clauses. The list can be empty, in which case 1.0
+ * must be returned. List elements may be either RestrictInfos
+ * or bare expression clauses --- the former is preferred since
+ * it allows caching of results.
+ *
+ * See clause_selectivity() for the meaning of the additional parameters.
+ *
+ * Our basic approach is to take the product of the selectivities of the
+ * subclauses. However, that's only right if the subclauses have independent
+ * probabilities, and in reality they are often NOT independent. So,
+ * we want to be smarter where we can.
+ *
+ * We also recognize "range queries", such as "x > 34 AND x < 42". Clauses
+ * are recognized as possible range query components if they are restriction
+ * opclauses whose operators have scalarltsel or a related function as their
+ * restriction selectivity estimator. We pair up clauses of this form that
+ * refer to the same variable. An unpairable clause of this kind is simply
+ * multiplied into the selectivity product in the normal way. But when we
+ * find a pair, we know that the selectivities represent the relative
+ * positions of the low and high bounds within the column's range, so instead
+ * of figuring the selectivity as hisel * losel, we can figure it as hisel +
+ * losel - 1. (To visualize this, see that hisel is the fraction of the range
+ * below the high bound, while losel is the fraction above the low bound; so
+ * hisel can be interpreted directly as a 0..1 value but we need to convert
+ * losel to 1-losel before interpreting it as a value. Then the available
+ * range is 1-losel to hisel. However, this calculation double-excludes
+ * nulls, so really we need hisel + losel + null_frac - 1.)
+ *
+ * If either selectivity is exactly DEFAULT_INEQ_SEL, we forget this equation
+ * and instead use DEFAULT_RANGE_INEQ_SEL. The same applies if the equation
+ * yields an impossible (negative) result.
+ *
+ * A free side-effect is that we can recognize redundant inequalities such
+ * as "x < 4 AND x < 5"; only the tighter constraint will be counted.
+ *
+ * Of course this is all very dependent on the behavior of the inequality
+ * selectivity functions; perhaps some day we can generalize the approach.
+ */
+Selectivity
+clauselist_selectivity_simple(PlannerInfo *root,
+ List *clauses,
+ int varRelid,
+ JoinType jointype,
+ SpecialJoinInfo *sjinfo,
+ Bitmapset *estimatedclauses)
+{
+ Selectivity s1 = 1.0;
+ RangeQueryClause *rqlist = NULL;
+ ListCell *l;
+ int listidx;
+
+ /*
* Anything that doesn't look like a potential rangequery clause gets
* multiplied into s1 and forgotten. Anything that does gets inserted into
* an rqlist entry.
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index 8369e3ad62..0112450419 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -1363,6 +1363,18 @@ get_relation_statistics(RelOptInfo *rel, Relation relation)
stainfos = lcons(info, stainfos);
}
+ if (statext_is_kind_built(htup, STATS_EXT_MCV))
+ {
+ StatisticExtInfo *info = makeNode(StatisticExtInfo);
+
+ info->statOid = statOid;
+ info->rel = rel;
+ info->kind = STATS_EXT_MCV;
+ info->keys = bms_copy(keys);
+
+ stainfos = lcons(info, stainfos);
+ }
+
ReleaseSysCache(htup);
bms_free(keys);
}
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index 656b1b5f1b..29877126d7 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -1681,6 +1681,8 @@ generateClonedExtStatsStmt(RangeVar *heapRel, Oid heapRelid,
stat_types = lappend(stat_types, makeString("ndistinct"));
else if (enabled[i] == STATS_EXT_DEPENDENCIES)
stat_types = lappend(stat_types, makeString("dependencies"));
+ else if (enabled[i] == STATS_EXT_MCV)
+ stat_types = lappend(stat_types, makeString("mcv"));
else
elog(ERROR, "unrecognized statistics kind %c", enabled[i]);
}
diff --git a/src/backend/statistics/Makefile b/src/backend/statistics/Makefile
index 3404e4554a..d2815265fb 100644
--- a/src/backend/statistics/Makefile
+++ b/src/backend/statistics/Makefile
@@ -12,6 +12,6 @@ subdir = src/backend/statistics
top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
-OBJS = extended_stats.o dependencies.o mvdistinct.o
+OBJS = extended_stats.o dependencies.o mcv.o mvdistinct.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/statistics/README b/src/backend/statistics/README
index a8f00a590e..8f153a9e85 100644
--- a/src/backend/statistics/README
+++ b/src/backend/statistics/README
@@ -18,6 +18,8 @@ There are currently two kinds of extended statistics:
(b) soft functional dependencies (README.dependencies)
+ (c) MCV lists (README.mcv)
+
Compatible clause types
-----------------------
@@ -26,6 +28,8 @@ Each type of statistics may be used to estimate some subset of clause types.
(a) functional dependencies - equality clauses (AND), possibly IS NULL
+ (b) MCV lists - equality and inequality clauses (AND, OR, NOT), IS NULL
+
Currently, only OpExprs in the form Var op Const, or Const op Var are
supported, however it's feasible to expand the code later to also estimate the
selectivities on clauses such as Var op Var.
diff --git a/src/backend/statistics/README.mcv b/src/backend/statistics/README.mcv
new file mode 100644
index 0000000000..2910eca962
--- /dev/null
+++ b/src/backend/statistics/README.mcv
@@ -0,0 +1,140 @@
+MCV lists
+=========
+
+Multivariate MCV (most-common values) lists are a straightforward extension of
+regular MCV list, tracking most frequent combinations of values for a group of
+attributes.
+
+This works particularly well for columns with a small number of distinct values,
+as the list may include all the combinations and approximate the distribution
+very accurately.
+
+For columns with large number of distinct values (e.g. those with continuous
+domains), the list will only track the most frequent combinations. If the
+distribution is mostly uniform (all combinations about equally frequent), the
+MCV list will be empty.
+
+Estimates of some clauses (e.g. equality) based on MCV lists are more accurate
+than when using histograms.
+
+Also, MCV lists don't necessarily require sorting of the values (the fact that
+we use sorting when building them is implementation detail), but even more
+importantly the ordering is not built into the approximation (while histograms
+are built on ordering). So MCV lists work well even for attributes where the
+ordering of the data type is disconnected from the meaning of the data. For
+example we know how to sort strings, but it's unlikely to make much sense for
+city names (or other label-like attributes).
+
+
+Selectivity estimation
+----------------------
+
+The estimation, implemented in clauselist_mv_selectivity_mcvlist(), is quite
+simple in principle - we need to identify MCV items matching all the clauses
+and sum frequencies of all those items.
+
+Currently MCV lists support estimation of the following clause types:
+
+ (a) equality clauses WHERE (a = 1) AND (b = 2)
+ (b) inequality clauses WHERE (a < 1) AND (b >= 2)
+ (c) NULL clauses WHERE (a IS NULL) AND (b IS NOT NULL)
+ (d) OR clauses WHERE (a < 1) OR (b >= 2)
+
+It's possible to add support for additional clauses, for example:
+
+ (e) multi-var clauses WHERE (a > b)
+
+and possibly others. These are tasks for the future, not yet implemented.
+
+
+Estimating equality clauses
+---------------------------
+
+When computing selectivity estimate for equality clauses
+
+ (a = 1) AND (b = 2)
+
+we can do this estimate pretty exactly assuming that two conditions are met:
+
+ (1) there's an equality condition on all attributes of the statistic
+
+ (2) we find a matching item in the MCV list
+
+In this case we know the MCV item represents all tuples matching the clauses,
+and the selectivity estimate is complete (i.e. we don't need to perform
+estimation using the histogram). This is what we call 'full match'.
+
+When only (1) holds, but there's no matching MCV item, we don't know whether
+there are no such rows or just are not very frequent. We can however use the
+frequency of the least frequent MCV item as an upper bound for the selectivity.
+
+For a combination of equality conditions (not full-match case) we can clamp the
+selectivity by the minimum of selectivities for each condition. For example if
+we know the number of distinct values for each column, we can use 1/ndistinct
+as a per-column estimate. Or rather 1/ndistinct + selectivity derived from the
+MCV list.
+
+We should also probably only use the 'residual ndistinct' by exluding the items
+included in the MCV list (and also residual frequency):
+
+ f = (1.0 - sum(MCV frequencies)) / (ndistinct - ndistinct(MCV list))
+
+but it's worth pointing out the ndistinct values are multi-variate for the
+columns referenced by the equality conditions.
+
+Note: Only the "full match" limit is currently implemented.
+
+
+Hashed MCV (not yet implemented)
+--------------------------------
+
+Regular MCV lists have to include actual values for each item, so if those items
+are large the list may be quite large. This is especially true for multi-variate
+MCV lists, although the current implementation partially mitigates this by
+performing de-duplicating the values before storing them on disk.
+
+It's possible to only store hashes (32-bit values) instead of the actual values,
+significantly reducing the space requirements. Obviously, this would only make
+the MCV lists useful for estimating equality conditions (assuming the 32-bit
+hashes make the collisions rare enough).
+
+This might also complicate matching the columns to available stats.
+
+
+TODO Consider implementing hashed MCV list, storing just 32-bit hashes instead
+ of the actual values. This type of MCV list will be useful only for
+ estimating equality clauses, and will reduce space requirements for large
+ varlena types (in such cases we usually only want equality anyway).
+
+TODO Currently there's no logic to consider building only a MCV list (and not
+ building the histogram at all), except for doing this decision manually in
+ ADD STATISTICS.
+
+
+Inspecting the MCV list
+-----------------------
+
+Inspecting the regular (per-attribute) MCV lists is trivial, as it's enough
+to select the columns from pg_stats. The data is encoded as anyarrays, and
+all the items have the same data type, so anyarray provides a simple way to
+get a text representation.
+
+With multivariate MCV lists the columns may use different data types, making
+it impossible to use anyarrays. It might be possible to produce similar
+array-like representation, but that would complicate further processing and
+analysis of the MCV list.
+
+So instead the MCV lists are stored in a custom data type (pg_mcv_list),
+which however makes it more difficult to inspect the contents. To make that
+easier, there's a SRF returning detailed information about the MCV lists.
+
+ SELECT * FROM pg_mcv_list_items(stxmcv);
+
+It accepts one parameter - a pg_mcv_list value (which can only be obtained
+from pg_statistic_ext catalog, to defend against malicious input), and
+returns these columns:
+
+ - item index (0, ..., (nitems-1))
+ - values (string array)
+ - nulls only (boolean array)
+ - frequency (double precision)
diff --git a/src/backend/statistics/dependencies.c b/src/backend/statistics/dependencies.c
index 140783cfb3..29e816c4f7 100644
--- a/src/backend/statistics/dependencies.c
+++ b/src/backend/statistics/dependencies.c
@@ -201,14 +201,11 @@ static double
dependency_degree(int numrows, HeapTuple *rows, int k, AttrNumber *dependency,
VacAttrStats **stats, Bitmapset *attrs)
{
- int i,
- j;
- int nvalues = numrows * k;
+ int i;
MultiSortSupport mss;
SortItem *items;
- Datum *values;
- bool *isnull;
int *attnums;
+ int *attnums_dep;
/* counters valid within a group */
int group_size = 0;
@@ -223,26 +220,16 @@ dependency_degree(int numrows, HeapTuple *rows, int k, AttrNumber *dependency,
/* sort info for all attributes columns */
mss = multi_sort_init(k);
- /* data for the sort */
- items = (SortItem *) palloc(numrows * sizeof(SortItem));
- values = (Datum *) palloc(sizeof(Datum) * nvalues);
- isnull = (bool *) palloc(sizeof(bool) * nvalues);
-
- /* fix the pointers to values/isnull */
- for (i = 0; i < numrows; i++)
- {
- items[i].values = &values[i * k];
- items[i].isnull = &isnull[i * k];
- }
-
/*
- * Transform the bms into an array, to make accessing i-th member easier.
+ * Transform the bms into an array, to make accessing i-th member easier,
+ * and then construct a filtered version with only attnums referenced
+ * by the dependency we validate.
*/
- attnums = (int *) palloc(sizeof(int) * bms_num_members(attrs));
- i = 0;
- j = -1;
- while ((j = bms_next_member(attrs, j)) >= 0)
- attnums[i++] = j;
+ attnums = build_attnums(attrs);
+
+ attnums_dep = (int *)palloc(k * sizeof(int));
+ for (i = 0; i < k; i++)
+ attnums_dep[i] = attnums[dependency[i]];
/*
* Verify the dependency (a,b,...)->z, using a rather simple algorithm:
@@ -254,7 +241,7 @@ dependency_degree(int numrows, HeapTuple *rows, int k, AttrNumber *dependency,
* (c) for each group count different values in the last column
*/
- /* prepare the sort function for the first dimension, and SortItem array */
+ /* prepare the sort function for the dimensions */
for (i = 0; i < k; i++)
{
VacAttrStats *colstat = stats[dependency[i]];
@@ -267,19 +254,16 @@ dependency_degree(int numrows, HeapTuple *rows, int k, AttrNumber *dependency,
/* prepare the sort function for this dimension */
multi_sort_add_dimension(mss, i, type->lt_opr);
-
- /* accumulate all the data for both columns into an array and sort it */
- for (j = 0; j < numrows; j++)
- {
- items[j].values[i] =
- heap_getattr(rows[j], attnums[dependency[i]],
- stats[i]->tupDesc, &items[j].isnull[i]);
- }
}
- /* sort the items so that we can detect the groups */
- qsort_arg((void *) items, numrows, sizeof(SortItem),
- multi_sort_compare, mss);
+ /*
+ * build an array of SortItem(s) sorted using the multi-sort support
+ *
+ * XXX This relies on all stats entries pointing to the same tuple
+ * descriptor. Not sure if that might not be the case.
+ */
+ items = build_sorted_items(numrows, rows, stats[0]->tupDesc,
+ mss, k, attnums_dep);
/*
* Walk through the sorted array, split it into rows according to the
@@ -322,9 +306,9 @@ dependency_degree(int numrows, HeapTuple *rows, int k, AttrNumber *dependency,
}
pfree(items);
- pfree(values);
- pfree(isnull);
pfree(mss);
+ pfree(attnums);
+ pfree(attnums_dep);
/* Compute the 'degree of validity' as (supporting/total). */
return (n_supporting_rows * 1.0 / numrows);
@@ -351,7 +335,6 @@ statext_dependencies_build(int numrows, HeapTuple *rows, Bitmapset *attrs,
VacAttrStats **stats)
{
int i,
- j,
k;
int numattrs;
int *attnums;
@@ -364,11 +347,7 @@ statext_dependencies_build(int numrows, HeapTuple *rows, Bitmapset *attrs,
/*
* Transform the bms into an array, to make accessing i-th member easier.
*/
- attnums = palloc(sizeof(int) * bms_num_members(attrs));
- i = 0;
- j = -1;
- while ((j = bms_next_member(attrs, j)) >= 0)
- attnums[i++] = j;
+ attnums = build_attnums(attrs);
Assert(numattrs >= 2);
@@ -915,9 +894,9 @@ find_strongest_dependency(StatisticExtInfo *stats, MVDependencies *dependencies,
* using functional dependency statistics, or 1.0 if no useful functional
* dependency statistic exists.
*
- * 'estimatedclauses' is an output argument that gets a bit set corresponding
- * to the (zero-based) list index of each clause that is included in the
- * estimated selectivity.
+ * 'estimatedclauses' is an input/output argument that gets a bit set
+ * corresponding to the (zero-based) list index of each clause that is included
+ * in the estimated selectivity.
*
* Given equality clauses on attributes (a,b) we find the strongest dependency
* between them, i.e. either (a=>b) or (b=>a). Assuming (a=>b) is the selected
@@ -952,9 +931,6 @@ dependencies_clauselist_selectivity(PlannerInfo *root,
AttrNumber *list_attnums;
int listidx;
- /* initialize output argument */
- *estimatedclauses = NULL;
-
/* check if there's any stats that might be useful for us. */
if (!has_stats_of_kind(rel->statlist, STATS_EXT_DEPENDENCIES))
return 1.0;
@@ -969,6 +945,9 @@ dependencies_clauselist_selectivity(PlannerInfo *root,
* the attnums for each clause in a list which we'll reference later so we
* don't need to repeat the same work again. We'll also keep track of all
* attnums seen.
+ *
+ * We also skip clauses that we already estimated using different types of
+ * statistics (we treat them as incompatible).
*/
listidx = 0;
foreach(l, clauses)
@@ -976,7 +955,8 @@ dependencies_clauselist_selectivity(PlannerInfo *root,
Node *clause = (Node *) lfirst(l);
AttrNumber attnum;
- if (dependency_is_compatible_clause(clause, rel->relid, &attnum))
+ if ((dependency_is_compatible_clause(clause, rel->relid, &attnum)) &&
+ (!bms_is_member(listidx, *estimatedclauses)))
{
list_attnums[listidx] = attnum;
clauses_attnums = bms_add_member(clauses_attnums, attnum);
@@ -1046,8 +1026,7 @@ dependencies_clauselist_selectivity(PlannerInfo *root,
/*
* Skip incompatible clauses, and ones we've already estimated on.
*/
- if (list_attnums[listidx] == InvalidAttrNumber ||
- bms_is_member(listidx, *estimatedclauses))
+ if (list_attnums[listidx] == InvalidAttrNumber)
continue;
/*
diff --git a/src/backend/statistics/extended_stats.c b/src/backend/statistics/extended_stats.c
index 2df5f7dc3a..0b66000705 100644
--- a/src/backend/statistics/extended_stats.c
+++ b/src/backend/statistics/extended_stats.c
@@ -16,6 +16,8 @@
*/
#include "postgres.h"
+#include <math.h>
+
#include "access/genam.h"
#include "access/heapam.h"
#include "access/htup_details.h"
@@ -23,6 +25,8 @@
#include "catalog/pg_collation.h"
#include "catalog/pg_statistic_ext.h"
#include "nodes/relation.h"
+#include "optimizer/clauses.h"
+#include "optimizer/cost.h"
#include "postmaster/autovacuum.h"
#include "statistics/extended_stats_internal.h"
#include "statistics/statistics.h"
@@ -31,6 +35,7 @@
#include "utils/lsyscache.h"
#include "utils/memutils.h"
#include "utils/rel.h"
+#include "utils/selfuncs.h"
#include "utils/syscache.h"
@@ -53,7 +58,7 @@ static VacAttrStats **lookup_var_attr_stats(Relation rel, Bitmapset *attrs,
int nvacatts, VacAttrStats **vacatts);
static void statext_store(Relation pg_stext, Oid relid,
MVNDistinct *ndistinct, MVDependencies *dependencies,
- VacAttrStats **stats);
+ MCVList * mcvlist, VacAttrStats **stats);
/*
@@ -87,6 +92,7 @@ BuildRelationExtStatistics(Relation onerel, double totalrows,
StatExtEntry *stat = (StatExtEntry *) lfirst(lc);
MVNDistinct *ndistinct = NULL;
MVDependencies *dependencies = NULL;
+ MCVList *mcv = NULL;
VacAttrStats **stats;
ListCell *lc2;
@@ -124,10 +130,13 @@ BuildRelationExtStatistics(Relation onerel, double totalrows,
else if (t == STATS_EXT_DEPENDENCIES)
dependencies = statext_dependencies_build(numrows, rows,
stat->columns, stats);
+ else if (t == STATS_EXT_MCV)
+ mcv = statext_mcv_build(numrows, rows, stat->columns, stats,
+ totalrows);
}
/* store the statistics in the catalog */
- statext_store(pg_stext, stat->statOid, ndistinct, dependencies, stats);
+ statext_store(pg_stext, stat->statOid, ndistinct, dependencies, mcv, stats);
}
heap_close(pg_stext, RowExclusiveLock);
@@ -155,6 +164,10 @@ statext_is_kind_built(HeapTuple htup, char type)
attnum = Anum_pg_statistic_ext_stxdependencies;
break;
+ case STATS_EXT_MCV:
+ attnum = Anum_pg_statistic_ext_stxmcv;
+ break;
+
default:
elog(ERROR, "unexpected statistics type requested: %d", type);
}
@@ -219,7 +232,8 @@ fetch_statentries_for_relation(Relation pg_statext, Oid relid)
for (i = 0; i < ARR_DIMS(arr)[0]; i++)
{
Assert((enabled[i] == STATS_EXT_NDISTINCT) ||
- (enabled[i] == STATS_EXT_DEPENDENCIES));
+ (enabled[i] == STATS_EXT_DEPENDENCIES) ||
+ (enabled[i] == STATS_EXT_MCV));
entry->types = lappend_int(entry->types, (int) enabled[i]);
}
@@ -294,7 +308,7 @@ lookup_var_attr_stats(Relation rel, Bitmapset *attrs,
static void
statext_store(Relation pg_stext, Oid statOid,
MVNDistinct *ndistinct, MVDependencies *dependencies,
- VacAttrStats **stats)
+ MCVList * mcv, VacAttrStats **stats)
{
HeapTuple stup,
oldtup;
@@ -325,9 +339,18 @@ statext_store(Relation pg_stext, Oid statOid,
values[Anum_pg_statistic_ext_stxdependencies - 1] = PointerGetDatum(data);
}
+ if (mcv != NULL)
+ {
+ bytea *data = statext_mcv_serialize(mcv, stats);
+
+ nulls[Anum_pg_statistic_ext_stxmcv - 1] = (data == NULL);
+ values[Anum_pg_statistic_ext_stxmcv - 1] = PointerGetDatum(data);
+ }
+
/* always replace the value (either by bytea or NULL) */
replaces[Anum_pg_statistic_ext_stxndistinct - 1] = true;
replaces[Anum_pg_statistic_ext_stxdependencies - 1] = true;
+ replaces[Anum_pg_statistic_ext_stxmcv - 1] = true;
/* there should already be a pg_statistic_ext tuple */
oldtup = SearchSysCache1(STATEXTOID, ObjectIdGetDatum(statOid));
@@ -434,6 +457,137 @@ multi_sort_compare_dims(int start, int end,
return 0;
}
+int
+compare_scalars_simple(const void *a, const void *b, void *arg)
+{
+ return compare_datums_simple(*(Datum *) a,
+ *(Datum *) b,
+ (SortSupport) arg);
+}
+
+int
+compare_datums_simple(Datum a, Datum b, SortSupport ssup)
+{
+ return ApplySortComparator(a, false, b, false, ssup);
+}
+
+/* simple counterpart to qsort_arg */
+void *
+bsearch_arg(const void *key, const void *base, size_t nmemb, size_t size,
+ int (*compar) (const void *, const void *, void *),
+ void *arg)
+{
+ size_t l,
+ u,
+ idx;
+ const void *p;
+ int comparison;
+
+ l = 0;
+ u = nmemb;
+ while (l < u)
+ {
+ idx = (l + u) / 2;
+ p = (void *) (((const char *) base) + (idx * size));
+ comparison = (*compar) (key, p, arg);
+
+ if (comparison < 0)
+ u = idx;
+ else if (comparison > 0)
+ l = idx + 1;
+ else
+ return (void *) p;
+ }
+
+ return NULL;
+}
+
+int *
+build_attnums(Bitmapset *attrs)
+{
+ int i,
+ j;
+ int numattrs = bms_num_members(attrs);
+ int *attnums;
+
+ /* build attnums from the bitmapset */
+ attnums = (int *) palloc(sizeof(int) * numattrs);
+ i = 0;
+ j = -1;
+ while ((j = bms_next_member(attrs, j)) >= 0)
+ attnums[i++] = j;
+
+ return attnums;
+}
+
+/* build_sorted_items
+ * build sorted array of SortItem with values from rows
+ *
+ * XXX All the memory is allocated in a single chunk, so that the caller
+ * can simply pfree the return value to release all of it.
+ */
+SortItem *
+build_sorted_items(int numrows, HeapTuple *rows, TupleDesc tdesc,
+ MultiSortSupport mss, int numattrs, int *attnums)
+{
+ int i,
+ j,
+ len;
+ int nvalues = numrows * numattrs;
+
+ /*
+ * We won't allocate the arrays for each item independenly, but in one
+ * large chunk and then just set the pointers. This allows the caller to
+ * simply pfree the return value to release all the memory.
+ */
+ SortItem *items;
+ Datum *values;
+ bool *isnull;
+ char *ptr;
+
+ /* Compute the total amount of memory we need (both items and values). */
+ len = numrows * sizeof(SortItem) + nvalues * (sizeof(Datum) + sizeof(bool));
+
+ /* Allocate the memory and split it into the pieces. */
+ ptr = palloc0(len);
+
+ /* items to sort */
+ items = (SortItem *) ptr;
+ ptr += numrows * sizeof(SortItem);
+
+ /* values and null flags */
+ values = (Datum *) ptr;
+ ptr += nvalues * sizeof(Datum);
+
+ isnull = (bool *) ptr;
+ ptr += nvalues * sizeof(bool);
+
+ /* make sure we consumed the whole buffer exactly */
+ Assert((ptr - (char *) items) == len);
+
+ /* fix the pointers to Datum and bool arrays */
+ for (i = 0; i < numrows; i++)
+ {
+ items[i].values = &values[i * numattrs];
+ items[i].isnull = &isnull[i * numattrs];
+
+ /* load the values/null flags from sample rows */
+ for (j = 0; j < numattrs; j++)
+ {
+ items[i].values[j] = heap_getattr(rows[i],
+ attnums[j], /* attnum */
+ tdesc,
+ &items[i].isnull[j]); /* isnull */
+ }
+ }
+
+ /* do the sort, using the multi-sort */
+ qsort_arg((void *) items, numrows, sizeof(SortItem),
+ multi_sort_compare, mss);
+
+ return items;
+}
+
/*
* has_stats_of_kind
* Check whether the list contains statistic of a given kind
@@ -464,7 +618,7 @@ has_stats_of_kind(List *stats, char requiredkind)
* object referencing the most of the requested attributes, breaking ties
* in favor of objects with fewer keys overall.
*
- * XXX if multiple statistics objects tie on both criteria, then which object
+ * XXX If multiple statistics objects tie on both criteria, then which object
* is chosen depends on the order that they appear in the stats list. Perhaps
* further tiebreakers are needed.
*/
@@ -514,3 +668,382 @@ choose_best_statistics(List *stats, Bitmapset *attnums, char requiredkind)
return best_match;
}
+
+int
+bms_member_index(Bitmapset *keys, AttrNumber varattno)
+{
+ int i,
+ j;
+
+ i = -1;
+ j = 0;
+ while (((i = bms_next_member(keys, i)) >= 0) && (i < varattno))
+ j += 1;
+
+ return j;
+}
+
+/* The Duj1 estimator (already used in analyze.c). */
+double
+estimate_ndistinct(double totalrows, int numrows, int d, int f1)
+{
+ double numer,
+ denom,
+ ndistinct;
+
+ numer = (double) numrows * (double) d;
+
+ denom = (double) (numrows - f1) +
+ (double) f1 * (double) numrows / totalrows;
+
+ ndistinct = numer / denom;
+
+ /* Clamp to sane range in case of roundoff error */
+ if (ndistinct < (double) d)
+ ndistinct = (double) d;
+
+ if (ndistinct > totalrows)
+ ndistinct = totalrows;
+
+ return floor(ndistinct + 0.5);
+}
+
+/*
+ * statext_is_compatible_clause_internal
+ * Does the heavy lifting of actually inspecting the clauses for
+ * statext_is_compatible_clause.
+ */
+static bool
+statext_is_compatible_clause_internal(Node *clause, Index relid, Bitmapset **attnums)
+{
+ /* We only support plain Vars for now */
+ if (IsA(clause, Var))
+ {
+ Var *var = (Var *) clause;
+
+ /* Ensure var is from the correct relation */
+ if (var->varno != relid)
+ return false;
+
+ /* we also better ensure the Var is from the current level */
+ if (var->varlevelsup > 0)
+ return false;
+
+ /* Also skip system attributes (we don't allow stats on those). */
+ if (!AttrNumberIsForUserDefinedAttr(var->varattno))
+ return false;
+
+ *attnums = bms_add_member(*attnums, var->varattno);
+
+ return true;
+ }
+
+ /* Var = Const */
+ if (is_opclause(clause))
+ {
+ OpExpr *expr = (OpExpr *) clause;
+ Var *var;
+ bool varonleft = true;
+ bool ok;
+
+ /* Only expressions with two arguments are considered compatible. */
+ if (list_length(expr->args) != 2)
+ return false;
+
+ /* see if it actually has the right */
+ ok = (NumRelids((Node *) expr) == 1) &&
+ (is_pseudo_constant_clause(lsecond(expr->args)) ||
+ (varonleft = false,
+ is_pseudo_constant_clause(linitial(expr->args))));
+
+ /* unsupported structure (two variables or so) */
+ if (!ok)
+ return false;
+
+ /*
+ * If it's not one of the supported operators ("=", "<", ">", etc.),
+ * just ignore the clause, as it's not compatible with MCV lists.
+ *
+ * This uses the function for estimating selectivity, not the operator
+ * directly (a bit awkward, but well ...).
+ */
+ if ((get_oprrest(expr->opno) != F_EQSEL) &&
+ (get_oprrest(expr->opno) != F_NEQSEL) &&
+ (get_oprrest(expr->opno) != F_SCALARLTSEL) &&
+ (get_oprrest(expr->opno) != F_SCALARLESEL) &&
+ (get_oprrest(expr->opno) != F_SCALARGTSEL) &&
+ (get_oprrest(expr->opno) != F_SCALARGESEL))
+ return false;
+
+ var = (varonleft) ? linitial(expr->args) : lsecond(expr->args);
+
+ return statext_is_compatible_clause_internal((Node *) var, relid, attnums);
+ }
+
+ /* NOT/AND/OR clause */
+ if (or_clause(clause) ||
+ and_clause(clause) ||
+ not_clause(clause))
+ {
+ /*
+ * AND/OR/NOT-clauses are supported if all sub-clauses are supported
+ *
+ * Perhaps we could improve this by handling mixed cases, when some of
+ * the clauses are supported and some are not. Selectivity for the
+ * supported subclauses would be computed using extended statistics,
+ * and the remaining clauses would be estimated using the traditional
+ * algorithm (product of selectivities).
+ *
+ * It however seems overly complex, and in a way we already do that
+ * because if we reject the whole clause as unsupported here, it will
+ * be eventually passed to clauselist_selectivity() which does exactly
+ * this (split into supported/unsupported clauses etc).
+ */
+ BoolExpr *expr = (BoolExpr *) clause;
+ ListCell *lc;
+ Bitmapset *clause_attnums = NULL;
+
+ foreach(lc, expr->args)
+ {
+ /*
+ * Had we found incompatible clause in the arguments, treat the
+ * whole clause as incompatible.
+ */
+ if (!statext_is_compatible_clause_internal((Node *) lfirst(lc),
+ relid, &clause_attnums))
+ return false;
+ }
+
+ /*
+ * Otherwise the clause is compatible, and we need to merge the
+ * attnums into the main bitmapset.
+ */
+ *attnums = bms_join(*attnums, clause_attnums);
+
+ return true;
+ }
+
+ /* Var IS NULL */
+ if (IsA(clause, NullTest))
+ {
+ NullTest *nt = (NullTest *) clause;
+
+ /*
+ * Only simple (Var IS NULL) expressions supported for now. Maybe we
+ * could use examine_variable to fix this?
+ */
+ if (!IsA(nt->arg, Var))
+ return false;
+
+ return statext_is_compatible_clause_internal((Node *) (nt->arg), relid, attnums);
+ }
+
+ return false;
+}
+
+/*
+ * statext_is_compatible_clause
+ * Determines if the clause is compatible with MCV lists.
+ *
+ * Only OpExprs with two arguments using an equality operator are supported.
+ * When returning True attnum is set to the attribute number of the Var within
+ * the supported clause.
+ *
+ * Currently we only support Var = Const, or Const = Var. It may be possible
+ * to expand on this later.
+ */
+static bool
+statext_is_compatible_clause(Node *clause, Index relid, Bitmapset **attnums)
+{
+ RestrictInfo *rinfo = (RestrictInfo *) clause;
+
+ if (!IsA(rinfo, RestrictInfo))
+ return false;
+
+ /* Pseudoconstants are not really interesting here. */
+ if (rinfo->pseudoconstant)
+ return false;
+
+ /* clauses referencing multiple varnos are incompatible */
+ if (bms_membership(rinfo->clause_relids) != BMS_SINGLETON)
+ return false;
+
+ return statext_is_compatible_clause_internal((Node *) rinfo->clause,
+ relid, attnums);
+}
+
+/*
+ * statext_clauselist_selectivity
+ * Estimate clauses using the best multi-column statistics.
+ *
+ * Selects the best extended (multi-column) statistic on a table (measured by
+ * a number of attributes extracted from the clauses and covered by it), and
+ * computes the selectivity for supplied clauses.
+ *
+ * One of the main challenges with using MCV lists is how to extrapolate the
+ * estimate to the data not covered by the MCV list. To do that, we compute
+ * not only the "MCV selectivity" (selectivities for MCV items matching the
+ * supplied clauses), but also a couple of derived selectivities:
+ *
+ * - simple selectivity: Computed without extended statistic, i.e. as if the
+ * columns/clauses were independent
+ *
+ * - base selectivity: Similar to simple selectivity, but is computed using
+ * the extended statistic by adding up the base frequencies (that we compute
+ * and store for each MCV item) of matching MCV items.
+ *
+ * - total selectivity: Selectivity covered by the whole MCV list.
+ *
+ * - other selectivity: A selectivity estimate for data not covered by the MCV
+ * list (i.e. satisfying the clauses, but not common enough to make it into
+ * the MCV list)
+ *
+ * Note: While simple and base selectivities are defined in a quite similar
+ * way, the values are computed differently and are not therefore equal. The
+ * simple selectivity is computed as a product of per-clause estimates, while
+ * the base selectivity is computed by adding up base frequencies of matching
+ * items of the multi-column MCV list. So the values may differ for two main
+ * reasons - (a) the MCV list may not cover 100% of the data and (b) some of
+ * the MCV items did not match the estimated clauses.
+ *
+ * As both (a) and (b) reduce the base selectivity value, it generally holds
+ * that (simple_selectivity >= base_selectivity). If the MCV list covers all
+ * the data, the values may be equal.
+ *
+ * So (simple_selectivity - base_selectivity) may be seen as a correction for
+ * the part not covered by the MCV list.
+ *
+ * Note: Due to rounding errors and minor differences in how the estimates
+ * are computed, the inequality may not always hold. Which is why we clamp
+ * the selectivities to prevent strange estimate (negative etc.).
+ *
+ * XXX If we were to use multiple statistics, this is where it would happen.
+ * We would simply repeat this on a loop on the "remaining" clauses, possibly
+ * using the already estimated clauses as conditions (and combining the values
+ * using conditional probability formula).
+ */
+Selectivity
+statext_clauselist_selectivity(PlannerInfo *root, List *clauses, int varRelid,
+ JoinType jointype, SpecialJoinInfo *sjinfo,
+ RelOptInfo *rel, Bitmapset **estimatedclauses)
+{
+ ListCell *l;
+ Bitmapset *clauses_attnums = NULL;
+ Bitmapset **list_attnums;
+ int listidx;
+ StatisticExtInfo *stat;
+ List *stat_clauses;
+ Selectivity simple_sel,
+ mcv_sel,
+ mcv_basesel,
+ mcv_totalsel,
+ other_sel,
+ sel;
+
+ /* we're interested in MCV lists */
+ int types = STATS_EXT_MCV;
+
+ /* check if there's any stats that might be useful for us. */
+ if (!has_stats_of_kind(rel->statlist, types))
+ return (Selectivity) 1.0;
+
+ list_attnums = (Bitmapset **) palloc(sizeof(Bitmapset *) *
+ list_length(clauses));
+
+ /*
+ * Pre-process the clauses list to extract the attnums seen in each item.
+ * We need to determine if there's any clauses which will be useful for
+ * dependency selectivity estimations. Along the way we'll record all of
+ * the attnums for each clause in a list which we'll reference later so we
+ * don't need to repeat the same work again. We'll also keep track of all
+ * attnums seen.
+ *
+ * We also skip clauses that we already estimated using different types of
+ * statistics (we treat them as incompatible).
+ *
+ * XXX Currently, the estimated clauses are always empty because the extra
+ * statistics are applied before functional dependencies. Once we decide
+ * to apply multiple statistics, this may change.
+ */
+ listidx = 0;
+ foreach(l, clauses)
+ {
+ Node *clause = (Node *) lfirst(l);
+ Bitmapset *attnums = NULL;
+
+ if ((statext_is_compatible_clause(clause, rel->relid, &attnums)) &&
+ (!bms_is_member(listidx, *estimatedclauses)))
+ {
+ list_attnums[listidx] = attnums;
+ clauses_attnums = bms_add_members(clauses_attnums, attnums);
+ }
+ else
+ list_attnums[listidx] = NULL;
+
+ listidx++;
+ }
+
+ /* We need at least two attributes for MCV lists. */
+ if (bms_num_members(clauses_attnums) < 2)
+ return 1.0;
+
+ /* find the best suited statistics object for these attnums */
+ stat = choose_best_statistics(rel->statlist, clauses_attnums, types);
+
+ /* if no matching stats could be found then we've nothing to do */
+ if (!stat)
+ return (Selectivity) 1.0;
+
+ /* We only understand MCV lists for now. */
+ Assert(stat->kind == STATS_EXT_MCV);
+
+ /* now filter the clauses to be estimated using the selected MCV */
+ stat_clauses = NIL;
+
+ listidx = 0;
+ foreach(l, clauses)
+ {
+ /*
+ * If the clause is compatible with the selected statistics, mark it
+ * as estimated and add it to the list to estimate.
+ */
+ if ((list_attnums[listidx] != NULL) &&
+ (bms_is_subset(list_attnums[listidx], stat->keys)))
+ {
+ stat_clauses = lappend(stat_clauses, (Node *) lfirst(l));
+ *estimatedclauses = bms_add_member(*estimatedclauses, listidx);
+ }
+
+ listidx++;
+ }
+
+ /*
+ * First compute "simple" selectivity, i.e. without the extended statistics,
+ * and essentially assuming independence of the columns/clauses. We'll then
+ * use the various selectivities computed from MCV list to improve it.
+ */
+ simple_sel = clauselist_selectivity_simple(root, stat_clauses, varRelid,
+ jointype, sjinfo, NULL);
+
+ /*
+ * Now compute the multi-column estimate from the MCV list, along with the
+ * other selectivities (base & total selectivity).
+ */
+ mcv_sel = mcv_clauselist_selectivity(root, stat, stat_clauses, varRelid,
+ jointype, sjinfo, rel,
+ &mcv_basesel, &mcv_totalsel);
+
+ /* Estimated selectivity of values not covered by MCV matches */
+ other_sel = simple_sel - mcv_basesel;
+ CLAMP_PROBABILITY(other_sel);
+
+ /* The non-MCV selectivity can't exceed the 1 - mcv_totalsel. */
+ if (other_sel > 1.0 - mcv_totalsel)
+ other_sel = 1.0 - mcv_totalsel;
+
+ /* Overall selectivity is the combination of MCV and non-MCV estimates. */
+ sel = mcv_sel + other_sel;
+ CLAMP_PROBABILITY(sel);
+
+ return sel;
+}
diff --git a/src/backend/statistics/mcv.c b/src/backend/statistics/mcv.c
new file mode 100644
index 0000000000..533fbdc037
--- /dev/null
+++ b/src/backend/statistics/mcv.c
@@ -0,0 +1,1656 @@
+/*-------------------------------------------------------------------------
+ *
+ * mcv.c
+ * POSTGRES multivariate MCV lists
+ *
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/statistics/mcv.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/htup_details.h"
+#include "catalog/pg_collation.h"
+#include "catalog/pg_statistic_ext.h"
+#include "fmgr.h"
+#include "funcapi.h"
+#include "optimizer/clauses.h"
+#include "statistics/extended_stats_internal.h"
+#include "statistics/statistics.h"
+#include "utils/builtins.h"
+#include "utils/bytea.h"
+#include "utils/fmgroids.h"
+#include "utils/fmgrprotos.h"
+#include "utils/lsyscache.h"
+#include "utils/syscache.h"
+#include "utils/typcache.h"
+
+#include <math.h>
+
+/*
+ * Computes size of a serialized MCV item, depending on the number of
+ * dimensions (columns) the statistic is defined on. The datum values are
+ * stored in a separate array (deduplicated, to minimize the size), and
+ * so the serialized items only store uint16 indexes into that array.
+ *
+ * Each serialized item store (in this order):
+ *
+ * - indexes to values (ndim * sizeof(uint16))
+ * - null flags (ndim * sizeof(bool))
+ * - frequency (sizeof(double))
+ * - base_frequency (sizeof(double))
+ *
+ * So in total each MCV item requires this many bytes:
+ *
+ * ndim * (sizeof(uint16) + sizeof(bool)) + 2 * sizeof(double)
+ */
+#define ITEM_SIZE(ndims) \
+ (ndims * (sizeof(uint16) + sizeof(bool)) + 2 * sizeof(double))
+
+/*
+ * Macros for convenient access to parts of a serialized MCV item.
+ */
+#define ITEM_INDEXES(item) ((uint16*)item)
+#define ITEM_NULLS(item,ndims) ((bool*)(ITEM_INDEXES(item) + ndims))
+#define ITEM_FREQUENCY(item,ndims) ((double*)(ITEM_NULLS(item,ndims) + ndims))
+#define ITEM_BASE_FREQUENCY(item,ndims) ((double*)(ITEM_FREQUENCY(item,ndims) + 1))
+
+
+static MultiSortSupport build_mss(VacAttrStats **stats, Bitmapset *attrs);
+
+static SortItem *build_distinct_groups(int numrows, SortItem *items,
+ MultiSortSupport mss, int *ndistinct);
+
+static int count_distinct_groups(int numrows, SortItem *items,
+ MultiSortSupport mss);
+
+/*
+ * Builds MCV list from the set of sampled rows.
+ *
+ * The algorithm is quite simple:
+ *
+ * (1) sort the data (default collation, '<' for the data type)
+ *
+ * (2) count distinct groups, decide how many to keep
+ *
+ * (3) build the MCV list using the threshold determined in (2)
+ *
+ * (4) remove rows represented by the MCV from the sample
+ *
+ */
+MCVList *
+statext_mcv_build(int numrows, HeapTuple *rows, Bitmapset *attrs,
+ VacAttrStats **stats, double totalrows)
+{
+ int i,
+ j,
+ k;
+ int numattrs = bms_num_members(attrs);
+ int ngroups;
+ int nitems;
+ double stadistinct;
+ int *mcv_counts;
+ int f1;
+
+ int *attnums = build_attnums(attrs);
+
+ MCVList *mcvlist = NULL;
+
+ /* comparator for all the columns */
+ MultiSortSupport mss = build_mss(stats, attrs);
+
+ /* sort the rows */
+ SortItem *items = build_sorted_items(numrows, rows, stats[0]->tupDesc,
+ mss, numattrs, attnums);
+
+ /* transform the sorted rows into groups (sorted by frequency) */
+ SortItem *groups = build_distinct_groups(numrows, items, mss, &ngroups);
+
+ /*
+ * Maximum number of MCV items to store, based on the attribute with the
+ * largest stats target (and the number of groups we have available).
+ */
+ nitems = stats[0]->attr->attstattarget;
+ for (i = 1; i < numattrs; i++)
+ {
+ if (stats[i]->attr->attstattarget > nitems)
+ nitems = stats[i]->attr->attstattarget;
+ }
+ if (nitems > ngroups)
+ nitems = ngroups;
+
+ /*
+ * Decide how many items to keep in the MCV list. We simply use the same
+ * algorithm as for per-column MCV lists, to keep it consistent.
+ *
+ * One difference is that we do not have a multi-column stanullfrac, and
+ * we simply treat it as a special item in the MCV list (it it makes it).
+ * We could compute and store it, of course, but we may have statistics
+ * on more than two columns, so we'd probably want to store this for
+ * various combinations of columns - for K columns that'd be 2^K values.
+ * So we instead store those as items of the multi-column MCV list (if
+ * common enough).
+ *
+ * XXX Conceptually this is similar to the NULL-buckets of histograms.
+ */
+ mcv_counts = (int *) palloc(sizeof(int) * nitems);
+ f1 = 0;
+
+ for (i = 0; i < nitems; i++)
+ {
+ mcv_counts[i] = groups[i].count;
+
+ /* count values that occur exactly once for the ndistinct estimate */
+ if (groups[i].count == 1)
+ f1 += 1;
+ }
+
+ stadistinct = estimate_ndistinct(totalrows, numrows, ngroups, f1);
+
+ /*
+ * If we can fit all the items onto the MCV list, do that. Otherwise use
+ * analyze_mcv_list to decide how many items to keep in the MCV list, just
+ * like for the single-dimensional MCV list.
+ *
+ * XXX Should we also consider stadistinct here, to see if the groups do
+ * represent all the distinct combinations.
+ */
+ if (ngroups > nitems)
+ {
+ nitems = analyze_mcv_list(mcv_counts, nitems, stadistinct,
+ 0.0, /* stanullfrac */
+ numrows, totalrows);
+ }
+
+ /*
+ * At this point we know the number of items for the MCV list. There might
+ * be none (for uniform distribution with many groups), and in that case
+ * there will be no MCV list. Otherwise construct the MCV list.
+ */
+ if (nitems > 0)
+ {
+ /*
+ * Allocate the MCV list structure, set the global parameters.
+ */
+ mcvlist = (MCVList *) palloc0(sizeof(MCVList));
+
+ mcvlist->magic = STATS_MCV_MAGIC;
+ mcvlist->type = STATS_MCV_TYPE_BASIC;
+ mcvlist->ndimensions = numattrs;
+ mcvlist->nitems = nitems;
+
+ /* store info about data type OIDs */
+ i = 0;
+ j = -1;
+ while ((j = bms_next_member(attrs, j)) >= 0)
+ {
+ VacAttrStats *colstat = stats[i];
+
+ mcvlist->types[i] = colstat->attrtypid;
+ i++;
+ }
+
+ /*
+ * Preallocate Datum/isnull arrays (not as a single chunk, as we will
+ * pass the result outside and thus it needs to be easy to pfree().
+ *
+ * XXX On second thought, we're the only ones dealing with MCV lists,
+ * so we might allocate everything as a single chunk to reduce palloc
+ * overhead (chunk headers, etc.) without significant risk. Not sure
+ * it's worth it, though, as we're not re-building stats very often.
+ */
+ mcvlist->items = (MCVItem * *) palloc0(sizeof(MCVItem *) * nitems);
+
+ for (i = 0; i < nitems; i++)
+ {
+ mcvlist->items[i] = (MCVItem *) palloc(sizeof(MCVItem));
+ mcvlist->items[i]->values = (Datum *) palloc(sizeof(Datum) * numattrs);
+ mcvlist->items[i]->isnull = (bool *) palloc(sizeof(bool) * numattrs);
+ }
+
+ /* Copy the first chunk of groups into the result. */
+ for (i = 0; i < nitems; i++)
+ {
+ /* just pointer to the proper place in the list */
+ MCVItem *item = mcvlist->items[i];
+
+ /* copy values from the _previous_ group (last item of) */
+ memcpy(item->values, groups[i].values, sizeof(Datum) * numattrs);
+ memcpy(item->isnull, groups[i].isnull, sizeof(bool) * numattrs);
+
+ /* groups should be sorted by frequency in descending order */
+ Assert((i == 0) || (groups[i - 1].count >= groups[i].count));
+
+ /* group frequency */
+ item->frequency = (double) groups[i].count / numrows;
+
+ /* base frequency, if the attributes were independent */
+ item->base_frequency = 1.0;
+ for (j = 0; j < numattrs; j++)
+ {
+ int count = 0;
+
+ for (k = 0; k < ngroups; k++)
+ {
+ if (multi_sort_compare_dim(j, &groups[i], &groups[k], mss) == 0)
+ count += groups[k].count;
+ }
+
+ item->base_frequency *= (double) count / numrows;
+ }
+ }
+ }
+
+ pfree(items);
+ pfree(groups);
+ pfree(mcv_counts);
+
+ return mcvlist;
+}
+
+/*
+ * build_mss
+ * build MultiSortSupport for the attributes passed in attrs
+ */
+static MultiSortSupport
+build_mss(VacAttrStats **stats, Bitmapset *attrs)
+{
+ int i,
+ j;
+ int numattrs = bms_num_members(attrs);
+
+ /* Sort by multiple columns (using array of SortSupport) */
+ MultiSortSupport mss = multi_sort_init(numattrs);
+
+ /* prepare the sort functions for all the attributes */
+ i = 0;
+ j = -1;
+ while ((j = bms_next_member(attrs, j)) >= 0)
+ {
+ VacAttrStats *colstat = stats[i];
+ TypeCacheEntry *type;
+
+ type = lookup_type_cache(colstat->attrtypid, TYPECACHE_LT_OPR);
+ if (type->lt_opr == InvalidOid) /* shouldn't happen */
+ elog(ERROR, "cache lookup failed for ordering operator for type %u",
+ colstat->attrtypid);
+
+ multi_sort_add_dimension(mss, i, type->lt_opr);
+ i++;
+ }
+
+ return mss;
+}
+
+/*
+ * count_distinct_groups
+ * count distinct combinations of SortItems in the array
+ *
+ * The array is assumed to be sorted according to the MultiSortSupport.
+ */
+static int
+count_distinct_groups(int numrows, SortItem *items, MultiSortSupport mss)
+{
+ int i;
+ int ndistinct;
+
+ ndistinct = 1;
+ for (i = 1; i < numrows; i++)
+ {
+ /* make sure the array really is sorted */
+ Assert(multi_sort_compare(&items[i], &items[i - 1], mss) >= 0);
+
+ if (multi_sort_compare(&items[i], &items[i - 1], mss) != 0)
+ ndistinct += 1;
+ }
+
+ return ndistinct;
+}
+
+/*
+ * compare_sort_item_count
+ * comparator for sorting items by count (frequencies) in descending order
+ */
+static int
+compare_sort_item_count(const void *a, const void *b)
+{
+ SortItem *ia = (SortItem *) a;
+ SortItem *ib = (SortItem *) b;
+
+ if (ia->count == ib->count)
+ return 0;
+ else if (ia->count > ib->count)
+ return -1;
+
+ return 1;
+}
+
+/*
+ * build_distinct_groups
+ * build array of SortItems for distinct groups and counts matching items
+ *
+ * The input array is assumed to be sorted
+ */
+static SortItem *
+build_distinct_groups(int numrows, SortItem *items, MultiSortSupport mss,
+ int *ndistinct)
+{
+ int i,
+ j;
+ int ngroups = count_distinct_groups(numrows, items, mss);
+
+ SortItem *groups = (SortItem *) palloc0(ngroups * sizeof(SortItem));
+
+ j = 0;
+ groups[0] = items[0];
+ groups[0].count = 1;
+
+ for (i = 1; i < numrows; i++)
+ {
+ /* Assume sorted in ascending order. */
+ Assert(multi_sort_compare(&items[i], &items[i - 1], mss) >= 0);
+
+ /* New distinct group detected. */
+ if (multi_sort_compare(&items[i], &items[i - 1], mss) != 0)
+ groups[++j] = items[i];
+
+ groups[j].count++;
+ }
+
+ /* Sort the distinct groups by frequency (in descending order). */
+ pg_qsort((void *) groups, ngroups, sizeof(SortItem),
+ compare_sort_item_count);
+
+ *ndistinct = ngroups;
+ return groups;
+}
+
+
+/*
+ * statext_mcv_load
+ * Load the MCV list for the indicated pg_statistic_ext tuple
+ */
+MCVList *
+statext_mcv_load(Oid mvoid)
+{
+ bool isnull = false;
+ Datum mcvlist;
+ HeapTuple htup = SearchSysCache1(STATEXTOID, ObjectIdGetDatum(mvoid));
+
+ if (!HeapTupleIsValid(htup))
+ elog(ERROR, "cache lookup failed for statistics object %u", mvoid);
+
+ mcvlist = SysCacheGetAttr(STATEXTOID, htup,
+ Anum_pg_statistic_ext_stxmcv, &isnull);
+
+ ReleaseSysCache(htup);
+
+ if (isnull)
+ return NULL;
+
+ return statext_mcv_deserialize(DatumGetByteaP(mcvlist));
+}
+
+
+/*
+ * Serialize MCV list into a bytea value.
+ *
+ * The basic algorithm is simple:
+ *
+ * (1) perform deduplication (for each attribute separately)
+ * (a) collect all (non-NULL) attribute values from all MCV items
+ * (b) sort the data (using 'lt' from VacAttrStats)
+ * (c) remove duplicate values from the array
+ *
+ * (2) serialize the arrays into a bytea value
+ *
+ * (3) process all MCV list items
+ * (a) replace values with indexes into the arrays
+ *
+ * Each attribute has to be processed separately, as we may be mixing different
+ * datatypes, with different sort operators, etc.
+ *
+ * We use uint16 values for the indexes in step (3), as we currently don't allow
+ * more than 8k MCV items anyway, although that's mostly arbitrary limit. We might
+ * increase this to 65k and still fit into uint16. Furthermore, this limit is on
+ * the number of distinct values per column, and we usually have few of those
+ * (and various combinations of them for the those MCV list). So uint16 seems fine.
+ *
+ * We don't really expect the serialization to save as much space as for
+ * histograms, as we are not doing any bucket splits (which is the source
+ * of high redundancy in histograms).
+ *
+ * TODO: Consider packing boolean flags (NULL) for each item into a single char
+ * (or a longer type) instead of using an array of bool items.
+ */
+bytea *
+statext_mcv_serialize(MCVList * mcvlist, VacAttrStats **stats)
+{
+ int i;
+ int dim;
+ int ndims = mcvlist->ndimensions;
+ int itemsize = ITEM_SIZE(ndims);
+
+ SortSupport ssup;
+ DimensionInfo *info;
+
+ Size total_length;
+
+ /* allocate the item just once */
+ char *item = palloc0(itemsize);
+
+ /* serialized items (indexes into arrays, etc.) */
+ bytea *output;
+ char *data = NULL;
+
+ /* values per dimension (and number of non-NULL values) */
+ Datum **values = (Datum **) palloc0(sizeof(Datum *) * ndims);
+ int *counts = (int *) palloc0(sizeof(int) * ndims);
+
+ /*
+ * We'll include some rudimentary information about the attributes (type
+ * length, etc.), so that we don't have to look them up while
+ * deserializing the MCV list.
+ *
+ * XXX Maybe this is not a great idea? Or maybe we should actually copy
+ * more fields, e.g. typeid, which would allow us to display the MCV list
+ * using only the serialized representation (currently we have to fetch
+ * this info from the relation).
+ */
+ info = (DimensionInfo *) palloc0(sizeof(DimensionInfo) * ndims);
+
+ /* sort support data for all attributes included in the MCV list */
+ ssup = (SortSupport) palloc0(sizeof(SortSupportData) * ndims);
+
+ /* collect and deduplicate values for each dimension (attribute) */
+ for (dim = 0; dim < ndims; dim++)
+ {
+ int ndistinct;
+ TypeCacheEntry *typentry;
+
+ /*
+ * Lookup the LT operator (can't get it from stats extra_data, as we
+ * don't know how to interpret that - scalar vs. array etc.).
+ */
+ typentry = lookup_type_cache(stats[dim]->attrtypid, TYPECACHE_LT_OPR);
+
+ /* copy important info about the data type (length, by-value) */
+ info[dim].typlen = stats[dim]->attrtype->typlen;
+ info[dim].typbyval = stats[dim]->attrtype->typbyval;
+
+ /* allocate space for values in the attribute and collect them */
+ values[dim] = (Datum *) palloc0(sizeof(Datum) * mcvlist->nitems);
+
+ for (i = 0; i < mcvlist->nitems; i++)
+ {
+ /* skip NULL values - we don't need to deduplicate those */
+ if (mcvlist->items[i]->isnull[dim])
+ continue;
+
+ values[dim][counts[dim]] = mcvlist->items[i]->values[dim];
+ counts[dim] += 1;
+ }
+
+ /* if there are just NULL values in this dimension, we're done */
+ if (counts[dim] == 0)
+ continue;
+
+ /* sort and deduplicate the data */
+ ssup[dim].ssup_cxt = CurrentMemoryContext;
+ ssup[dim].ssup_collation = DEFAULT_COLLATION_OID;
+ ssup[dim].ssup_nulls_first = false;
+
+ PrepareSortSupportFromOrderingOp(typentry->lt_opr, &ssup[dim]);
+
+ qsort_arg(values[dim], counts[dim], sizeof(Datum),
+ compare_scalars_simple, &ssup[dim]);
+
+ /*
+ * Walk through the array and eliminate duplicate values, but keep the
+ * ordering (so that we can do bsearch later). We know there's at
+ * least one item as (counts[dim] != 0), so we can skip the first
+ * element.
+ */
+ ndistinct = 1; /* number of distinct values */
+ for (i = 1; i < counts[dim]; i++)
+ {
+ /* expect sorted array */
+ Assert(compare_datums_simple(values[dim][i - 1], values[dim][i], &ssup[dim]) <= 0);
+
+ /* if the value is the same as the previous one, we can skip it */
+ if (!compare_datums_simple(values[dim][i - 1], values[dim][i], &ssup[dim]))
+ continue;
+
+ values[dim][ndistinct] = values[dim][i];
+ ndistinct += 1;
+ }
+
+ /* we must not exceed UINT16_MAX, as we use uint16 indexes */
+ Assert(ndistinct <= UINT16_MAX);
+
+ /*
+ * Store additional info about the attribute - number of deduplicated
+ * values, and also size of the serialized data. For fixed-length data
+ * types this is trivial to compute, for varwidth types we need to
+ * actually walk the array and sum the sizes.
+ */
+ info[dim].nvalues = ndistinct;
+
+ if (info[dim].typlen > 0) /* fixed-length data types */
+ info[dim].nbytes = info[dim].nvalues * info[dim].typlen;
+ else if (info[dim].typlen == -1) /* varlena */
+ {
+ info[dim].nbytes = 0;
+ for (i = 0; i < info[dim].nvalues; i++)
+ info[dim].nbytes += VARSIZE_ANY(values[dim][i]);
+ }
+ else if (info[dim].typlen == -2) /* cstring */
+ {
+ info[dim].nbytes = 0;
+ for (i = 0; i < info[dim].nvalues; i++)
+ info[dim].nbytes += strlen(DatumGetPointer(values[dim][i]));
+ }
+
+ /* we know (count>0) so there must be some data */
+ Assert(info[dim].nbytes > 0);
+ }
+
+ /*
+ * Now we can finally compute how much space we'll actually need for the
+ * whole serialized MCV list, as it contains these fields:
+ *
+ * - length (4B) for varlena - magic (4B) - type (4B) - ndimensions (4B) -
+ * nitems (4B) - info (ndim * sizeof(DimensionInfo) - arrays of values for
+ * each dimension - serialized items (nitems * itemsize)
+ *
+ * So the 'header' size is 20B + ndim * sizeof(DimensionInfo) and then we
+ * will place all the data (values + indexes). We'll however use offsetof
+ * and sizeof to compute sizes of the structs.
+ */
+ total_length = (sizeof(int32) + offsetof(MCVList, items)
+ + (ndims * sizeof(DimensionInfo))
+ + mcvlist->nitems * itemsize);
+
+ /* add space for the arrays of deduplicated values */
+ for (i = 0; i < ndims; i++)
+ total_length += info[i].nbytes;
+
+ /*
+ * Enforce arbitrary limit of 1MB on the size of the serialized MCV list.
+ * This is meant as a protection against someone building MCV list on long
+ * values (e.g. text documents).
+ *
+ * XXX Should we enforce arbitrary limits like this one? Maybe it's not
+ * even necessary, as long values are usually unique and so won't make it
+ * into the MCV list in the first place. In the end, we have a 1GB limit
+ * on bytea values.
+ */
+ if (total_length > (1024 * 1024))
+ elog(ERROR, "serialized MCV list exceeds 1MB (%ld)", total_length);
+
+ /* allocate space for the serialized MCV list, set header fields */
+ output = (bytea *) palloc0(total_length);
+ SET_VARSIZE(output, total_length);
+
+ /* 'data' points to the current position in the output buffer */
+ data = VARDATA(output);
+
+ /* MCV list header (number of items, ...) */
+ memcpy(data, mcvlist, offsetof(MCVList, items));
+ data += offsetof(MCVList, items);
+
+ /* information about the attributes */
+ memcpy(data, info, sizeof(DimensionInfo) * ndims);
+ data += sizeof(DimensionInfo) * ndims;
+
+ /* Copy the deduplicated values for all attributes to the output. */
+ for (dim = 0; dim < ndims; dim++)
+ {
+#ifdef USE_ASSERT_CHECKING
+ /* remember the starting point for Asserts later */
+ char *tmp = data;
+#endif
+ for (i = 0; i < info[dim].nvalues; i++)
+ {
+ Datum v = values[dim][i];
+
+ if (info[dim].typbyval) /* passed by value */
+ {
+ memcpy(data, &v, info[dim].typlen);
+ data += info[dim].typlen;
+ }
+ else if (info[dim].typlen > 0) /* pased by reference */
+ {
+ memcpy(data, DatumGetPointer(v), info[dim].typlen);
+ data += info[dim].typlen;
+ }
+ else if (info[dim].typlen == -1) /* varlena */
+ {
+ memcpy(data, DatumGetPointer(v), VARSIZE_ANY(v));
+ data += VARSIZE_ANY(v);
+ }
+ else if (info[dim].typlen == -2) /* cstring */
+ {
+ memcpy(data, DatumGetPointer(v), strlen(DatumGetPointer(v)) + 1);
+ data += strlen(DatumGetPointer(v)) + 1; /* terminator */
+ }
+
+ /* no underflows or overflows */
+ Assert((data > tmp) && ((data - tmp) <= info[dim].nbytes));
+ }
+
+ /*
+ * check we got exactly the amount of data we expected for this
+ * dimension
+ */
+ Assert((data - tmp) == info[dim].nbytes);
+ }
+
+ /* Serialize the items, with uint16 indexes instead of the values. */
+ for (i = 0; i < mcvlist->nitems; i++)
+ {
+ MCVItem *mcvitem = mcvlist->items[i];
+
+ /* don't write beyond the allocated space */
+ Assert(data <= (char *) output + total_length - itemsize);
+
+ /* reset the item (we only allocate it once and reuse it) */
+ memset(item, 0, itemsize);
+
+ for (dim = 0; dim < ndims; dim++)
+ {
+ Datum *v = NULL;
+
+ /* do the lookup only for non-NULL values */
+ if (mcvlist->items[i]->isnull[dim])
+ continue;
+
+ v = (Datum *) bsearch_arg(&mcvitem->values[dim], values[dim],
+ info[dim].nvalues, sizeof(Datum),
+ compare_scalars_simple, &ssup[dim]);
+
+ Assert(v != NULL); /* serialization or deduplication error */
+
+ /* compute index within the array */
+ ITEM_INDEXES(item)[dim] = (v - values[dim]);
+
+ /* check the index is within expected bounds */
+ Assert(ITEM_INDEXES(item)[dim] >= 0);
+ Assert(ITEM_INDEXES(item)[dim] < info[dim].nvalues);
+ }
+
+ /* copy NULL and frequency flags into the item */
+ memcpy(ITEM_NULLS(item, ndims), mcvitem->isnull, sizeof(bool) * ndims);
+ memcpy(ITEM_FREQUENCY(item, ndims), &mcvitem->frequency, sizeof(double));
+ memcpy(ITEM_BASE_FREQUENCY(item, ndims), &mcvitem->base_frequency, sizeof(double));
+
+ /* copy the serialized item into the array */
+ memcpy(data, item, itemsize);
+
+ data += itemsize;
+ }
+
+ /* at this point we expect to match the total_length exactly */
+ Assert((data - (char *) output) == total_length);
+
+ pfree(item);
+ pfree(values);
+ pfree(counts);
+
+ return output;
+}
+
+/*
+ * Reads serialized MCV list into MCVList structure.
+ *
+ * Unlike with histograms, we deserialize the MCV list fully (i.e. we don't
+ * keep the deduplicated arrays and pointers into them), as we don't expect
+ * there to be a lot of duplicate values. But perhaps that's not true and we
+ * should keep the MCV in serialized form too.
+ *
+ * XXX See how much memory we could save by keeping the deduplicated version
+ * (both for typical and corner cases with few distinct values but many items).
+ */
+MCVList *
+statext_mcv_deserialize(bytea *data)
+{
+ int dim,
+ i;
+ Size expected_size;
+ MCVList *mcvlist;
+ char *tmp;
+
+ int ndims,
+ nitems,
+ itemsize;
+ DimensionInfo *info = NULL;
+ Datum **values = NULL;
+
+ /* local allocation buffer (used only for deserialization) */
+ int bufflen;
+ char *buff;
+ char *ptr;
+
+ /* buffer used for the result */
+ int rbufflen;
+ char *rbuff;
+ char *rptr;
+
+ if (data == NULL)
+ return NULL;
+
+ /*
+ * We can't possibly deserialize a MCV list if there's not even a complete
+ * header.
+ */
+ if (VARSIZE_ANY_EXHDR(data) < offsetof(MCVList, items))
+ elog(ERROR, "invalid MCV Size %ld (expected at least %ld)",
+ VARSIZE_ANY_EXHDR(data), offsetof(MCVList, items));
+
+ /* read the MCV list header */
+ mcvlist = (MCVList *) palloc0(sizeof(MCVList));
+
+ /* initialize pointer to the data part (skip the varlena header) */
+ tmp = VARDATA_ANY(data);
+
+ /* get the header and perform further sanity checks */
+ memcpy(mcvlist, tmp, offsetof(MCVList, items));
+ tmp += offsetof(MCVList, items);
+
+ if (mcvlist->magic != STATS_MCV_MAGIC)
+ elog(ERROR, "invalid MCV magic %d (expected %dd)",
+ mcvlist->magic, STATS_MCV_MAGIC);
+
+ if (mcvlist->type != STATS_MCV_TYPE_BASIC)
+ elog(ERROR, "invalid MCV type %d (expected %dd)",
+ mcvlist->type, STATS_MCV_TYPE_BASIC);
+
+ if (mcvlist->ndimensions == 0)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("invalid zero-length dimension array in MCVList")));
+ else if (mcvlist->ndimensions > STATS_MAX_DIMENSIONS)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("invalid length (%d) dimension array in MCVList",
+ mcvlist->ndimensions)));
+
+ if (mcvlist->nitems == 0)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("invalid zero-length item array in MCVList")));
+ else if (mcvlist->nitems > STATS_MCVLIST_MAX_ITEMS)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("invalid length (%d) item array in MCVList",
+ mcvlist->nitems)));
+
+ nitems = mcvlist->nitems;
+ ndims = mcvlist->ndimensions;
+ itemsize = ITEM_SIZE(ndims);
+
+ /*
+ * Check amount of data including DimensionInfo for all dimensions and
+ * also the serialized items (including uint16 indexes). Also, walk
+ * through the dimension information and add it to the sum.
+ */
+ expected_size = offsetof(MCVList, items) +
+ ndims * sizeof(DimensionInfo) +
+ (nitems * itemsize);
+
+ /*
+ * Check that we have at least the dimension and info records, along with
+ * the items. We don't know the size of the serialized values yet. We need
+ * to do this check first, before accessing the dimension info.
+ */
+ if (VARSIZE_ANY_EXHDR(data) < expected_size)
+ elog(ERROR, "invalid MCV size %ld (expected %ld)",
+ VARSIZE_ANY_EXHDR(data), expected_size);
+
+ /* Now it's safe to access the dimension info. */
+ info = (DimensionInfo *) (tmp);
+ tmp += ndims * sizeof(DimensionInfo);
+
+ /* account for the value arrays */
+ for (dim = 0; dim < ndims; dim++)
+ {
+ /*
+ * XXX I wonder if we can/should rely on asserts here. Maybe those
+ * checks should be done every time?
+ */
+ Assert(info[dim].nvalues >= 0);
+ Assert(info[dim].nbytes >= 0);
+
+ expected_size += info[dim].nbytes;
+ }
+
+ /*
+ * Now we know the total expected MCV size, including all the pieces
+ * (header, dimension info. items and deduplicated data). So do the final
+ * check on size.
+ */
+ if (VARSIZE_ANY_EXHDR(data) != expected_size)
+ elog(ERROR, "invalid MCV size %ld (expected %ld)",
+ VARSIZE_ANY_EXHDR(data), expected_size);
+
+ /*
+ * Allocate one large chunk of memory for the intermediate data, needed
+ * only for deserializing the MCV list (and allocate densely to minimize
+ * the palloc overhead).
+ *
+ * Let's see how much space we'll actually need, and also include space
+ * for the array with pointers.
+ *
+ * We need an array of Datum pointers values for each dimension, so that
+ * we can easily translate the uint16 indexes. We also need a top-level
+ * array of pointers to those per-dimension arrays.
+ *
+ * For byval types with size matching sizeof(Datum) we can reuse the
+ * serialized array directly.
+ */
+ bufflen = sizeof(Datum **) * ndims; /* space for top-level pointers */
+
+ for (dim = 0; dim < ndims; dim++)
+ {
+ /* for full-size byval types, we reuse the serialized value */
+ if (!(info[dim].typbyval && info[dim].typlen == sizeof(Datum)))
+ bufflen += (sizeof(Datum) * info[dim].nvalues);
+ }
+
+ buff = palloc0(bufflen);
+ ptr = buff;
+
+ values = (Datum **) buff;
+ ptr += (sizeof(Datum *) * ndims);
+
+ /*
+ * XXX This uses pointers to the original data array (the types not passed
+ * by value), so when someone frees the memory, e.g. by doing something
+ * like this:
+ *
+ * bytea * data = ... fetch the data from catalog ...
+ *
+ * MCVList mcvlist = deserialize_mcv_list(data);
+ *
+ * pfree(data);
+ *
+ * then 'mcvlist' references the freed memory. Should copy the pieces.
+ */
+ for (dim = 0; dim < ndims; dim++)
+ {
+#ifdef USE_ASSERT_CHECKING
+ /* remember where data for this dimension starts */
+ char *start = tmp;
+#endif
+ if (info[dim].typbyval)
+ {
+ /* passed by value / size matches Datum - just reuse the array */
+ if (info[dim].typlen == sizeof(Datum))
+ {
+ values[dim] = (Datum *) tmp;
+ tmp += info[dim].nbytes;
+
+ /* no overflow of input array */
+ Assert(tmp <= start + info[dim].nbytes);
+ }
+ else
+ {
+ values[dim] = (Datum *) ptr;
+ ptr += (sizeof(Datum) * info[dim].nvalues);
+
+ for (i = 0; i < info[dim].nvalues; i++)
+ {
+ /* just point into the array */
+ memcpy(&values[dim][i], tmp, info[dim].typlen);
+ tmp += info[dim].typlen;
+
+ /* no overflow of input array */
+ Assert(tmp <= start + info[dim].nbytes);
+ }
+ }
+ }
+ else
+ {
+ /* all the other types need a chunk of the buffer */
+ values[dim] = (Datum *) ptr;
+ ptr += (sizeof(Datum) * info[dim].nvalues);
+
+ /* passed by reference, but fixed length (name, tid, ...) */
+ if (info[dim].typlen > 0)
+ {
+ for (i = 0; i < info[dim].nvalues; i++)
+ {
+ /* just point into the array */
+ values[dim][i] = PointerGetDatum(tmp);
+ tmp += info[dim].typlen;
+
+ /* no overflow of input array */
+ Assert(tmp <= start + info[dim].nbytes);
+ }
+ }
+ else if (info[dim].typlen == -1)
+ {
+ /* varlena */
+ for (i = 0; i < info[dim].nvalues; i++)
+ {
+ /* just point into the array */
+ values[dim][i] = PointerGetDatum(tmp);
+ tmp += VARSIZE_ANY(tmp);
+
+ /* no overflow of input array */
+ Assert(tmp <= start + info[dim].nbytes);
+ }
+ }
+ else if (info[dim].typlen == -2)
+ {
+ /* cstring */
+ for (i = 0; i < info[dim].nvalues; i++)
+ {
+ /* just point into the array */
+ values[dim][i] = PointerGetDatum(tmp);
+ tmp += (strlen(tmp) + 1); /* don't forget the \0 */
+
+ /* no overflow of input array */
+ Assert(tmp <= start + info[dim].nbytes);
+ }
+ }
+ }
+
+ /* check we consumed the serialized data for this dimension exactly */
+ Assert((tmp - start) == info[dim].nbytes);
+ }
+
+ /* we should have exhausted the buffer exactly */
+ Assert((ptr - buff) == bufflen);
+
+ /* allocate space for all the MCV items in a single piece */
+ rbufflen = (sizeof(MCVItem *) + sizeof(MCVItem) +
+ sizeof(Datum) * ndims + sizeof(bool) * ndims) * nitems;
+
+ rbuff = palloc0(rbufflen);
+ rptr = rbuff;
+
+ mcvlist->items = (MCVItem * *) rbuff;
+ rptr += (sizeof(MCVItem *) * nitems);
+
+ /* deserialize the MCV items and translate the indexes to Datums */
+ for (i = 0; i < nitems; i++)
+ {
+ uint16 *indexes = NULL;
+ MCVItem *item = (MCVItem *) rptr;
+
+ rptr += (sizeof(MCVItem));
+
+ item->values = (Datum *) rptr;
+ rptr += (sizeof(Datum) * ndims);
+
+ item->isnull = (bool *) rptr;
+ rptr += (sizeof(bool) * ndims);
+
+ /* just point to the right place */
+ indexes = ITEM_INDEXES(tmp);
+
+ memcpy(item->isnull, ITEM_NULLS(tmp, ndims), sizeof(bool) * ndims);
+ memcpy(&item->frequency, ITEM_FREQUENCY(tmp, ndims), sizeof(double));
+ memcpy(&item->base_frequency, ITEM_BASE_FREQUENCY(tmp, ndims), sizeof(double));
+
+ /* translate the values */
+ for (dim = 0; dim < ndims; dim++)
+ if (!item->isnull[dim])
+ item->values[dim] = values[dim][indexes[dim]];
+
+ mcvlist->items[i] = item;
+
+ tmp += ITEM_SIZE(ndims);
+
+ /* check we're not overflowing the input */
+ Assert(tmp <= (char *) data + VARSIZE_ANY(data));
+ }
+
+ /* check that we processed all the data */
+ Assert(tmp == (char *) data + VARSIZE_ANY(data));
+
+ /* release the temporary buffer */
+ pfree(buff);
+
+ return mcvlist;
+}
+
+/*
+ * SRF with details about buckets of a histogram:
+ *
+ * - item ID (0...nitems)
+ * - values (string array)
+ * - nulls only (boolean array)
+ * - frequency (double precision)
+ * - base_frequency (double precision)
+ *
+ * The input is the OID of the statistics, and there are no rows returned if
+ * the statistics contains no histogram.
+ */
+PG_FUNCTION_INFO_V1(pg_stats_ext_mcvlist_items);
+
+Datum
+pg_stats_ext_mcvlist_items(PG_FUNCTION_ARGS)
+{
+ FuncCallContext *funcctx;
+ int call_cntr;
+ int max_calls;
+ TupleDesc tupdesc;
+ AttInMetadata *attinmeta;
+
+ /* stuff done only on the first call of the function */
+ if (SRF_IS_FIRSTCALL())
+ {
+ MemoryContext oldcontext;
+ MCVList *mcvlist;
+
+ /* create a function context for cross-call persistence */
+ funcctx = SRF_FIRSTCALL_INIT();
+
+ /* switch to memory context appropriate for multiple function calls */
+ oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
+
+ mcvlist = statext_mcv_deserialize(PG_GETARG_BYTEA_P(0));
+
+ funcctx->user_fctx = mcvlist;
+
+ /* total number of tuples to be returned */
+ funcctx->max_calls = 0;
+ if (funcctx->user_fctx != NULL)
+ funcctx->max_calls = mcvlist->nitems;
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("function returning record called in context "
+ "that cannot accept type record")));
+
+ /* build metadata needed later to produce tuples from raw C-strings */
+ attinmeta = TupleDescGetAttInMetadata(tupdesc);
+ funcctx->attinmeta = attinmeta;
+
+ MemoryContextSwitchTo(oldcontext);
+ }
+
+ /* stuff done on every call of the function */
+ funcctx = SRF_PERCALL_SETUP();
+
+ call_cntr = funcctx->call_cntr;
+ max_calls = funcctx->max_calls;
+ attinmeta = funcctx->attinmeta;
+
+ if (call_cntr < max_calls) /* do when there is more left to send */
+ {
+ char **values;
+ HeapTuple tuple;
+ Datum result;
+
+ char *buff = palloc0(1024);
+ char *format;
+
+ int i;
+
+ Oid *outfuncs;
+ FmgrInfo *fmgrinfo;
+
+ MCVList *mcvlist;
+ MCVItem *item;
+
+ mcvlist = (MCVList *) funcctx->user_fctx;
+
+ Assert(call_cntr < mcvlist->nitems);
+
+ item = mcvlist->items[call_cntr];
+
+ /*
+ * Prepare a values array for building the returned tuple. This should
+ * be an array of C strings which will be processed later by the type
+ * input functions.
+ */
+ values = (char **) palloc(5 * sizeof(char *));
+
+ values[0] = (char *) palloc(64 * sizeof(char));
+
+ /* arrays */
+ values[1] = (char *) palloc0(1024 * sizeof(char));
+ values[2] = (char *) palloc0(1024 * sizeof(char));
+
+ /* frequency */
+ values[3] = (char *) palloc(64 * sizeof(char));
+
+ /* base frequency */
+ values[4] = (char *) palloc(64 * sizeof(char));
+
+ outfuncs = (Oid *) palloc0(sizeof(Oid) * mcvlist->ndimensions);
+ fmgrinfo = (FmgrInfo *) palloc0(sizeof(FmgrInfo) * mcvlist->ndimensions);
+
+ for (i = 0; i < mcvlist->ndimensions; i++)
+ {
+ bool isvarlena;
+
+ getTypeOutputInfo(mcvlist->types[i], &outfuncs[i], &isvarlena);
+
+ fmgr_info(outfuncs[i], &fmgrinfo[i]);
+ }
+
+ snprintf(values[0], 64, "%d", call_cntr); /* item ID */
+
+ for (i = 0; i < mcvlist->ndimensions; i++)
+ {
+ Datum val,
+ valout;
+
+ format = "%s, %s";
+ if (i == 0)
+ format = "{%s%s";
+ else if (i == mcvlist->ndimensions - 1)
+ format = "%s, %s}";
+
+ if (item->isnull[i])
+ valout = CStringGetDatum("NULL");
+ else
+ {
+ val = item->values[i];
+ valout = FunctionCall1(&fmgrinfo[i], val);
+ }
+
+ snprintf(buff, 1024, format, values[1], DatumGetPointer(valout));
+ strncpy(values[1], buff, 1023);
+ buff[0] = '\0';
+
+ snprintf(buff, 1024, format, values[2], item->isnull[i] ? "t" : "f");
+ strncpy(values[2], buff, 1023);
+ buff[0] = '\0';
+ }
+
+ snprintf(values[3], 64, "%f", item->frequency); /* frequency */
+ snprintf(values[4], 64, "%f", item->base_frequency); /* base frequency */
+
+ /* build a tuple */
+ tuple = BuildTupleFromCStrings(attinmeta, values);
+
+ /* make the tuple into a datum */
+ result = HeapTupleGetDatum(tuple);
+
+ /* clean up (this is not really necessary) */
+ pfree(values[0]);
+ pfree(values[1]);
+ pfree(values[2]);
+ pfree(values[3]);
+ pfree(values[4]);
+
+ pfree(values);
+
+ SRF_RETURN_NEXT(funcctx, result);
+ }
+ else /* do when there is no more left */
+ {
+ SRF_RETURN_DONE(funcctx);
+ }
+}
+
+/*
+ * pg_mcv_list_in - input routine for type pg_mcv_list.
+ *
+ * pg_mcv_list is real enough to be a table column, but it has no operations
+ * of its own, and disallows input too
+ */
+Datum
+pg_mcv_list_in(PG_FUNCTION_ARGS)
+{
+ /*
+ * pg_mcv_list stores the data in binary form and parsing text input is
+ * not needed, so disallow this.
+ */
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("cannot accept a value of type %s", "pg_mcv_list")));
+
+ PG_RETURN_VOID(); /* keep compiler quiet */
+}
+
+
+/*
+ * pg_mcv_list_out - output routine for type PG_MCV_LIST.
+ *
+ * MCV lists are serialized into a bytea value, so we simply call byteaout()
+ * to serialize the value into text. But it'd be nice to serialize that into
+ * a meaningful representation (e.g. for inspection by people).
+ *
+ * XXX This should probably return something meaningful, similar to what
+ * pg_dependencies_out does. Not sure how to deal with the deduplicated
+ * values, though - do we want to expand that or not?
+ */
+Datum
+pg_mcv_list_out(PG_FUNCTION_ARGS)
+{
+ return byteaout(fcinfo);
+}
+
+/*
+ * pg_mcv_list_recv - binary input routine for type pg_mcv_list.
+ */
+Datum
+pg_mcv_list_recv(PG_FUNCTION_ARGS)
+{
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("cannot accept a value of type %s", "pg_mcv_list")));
+
+ PG_RETURN_VOID(); /* keep compiler quiet */
+}
+
+/*
+ * pg_mcv_list_send - binary output routine for type pg_mcv_list.
+ *
+ * MCV lists are serialized in a bytea value (although the type is named
+ * differently), so let's just send that.
+ */
+Datum
+pg_mcv_list_send(PG_FUNCTION_ARGS)
+{
+ return byteasend(fcinfo);
+}
+
+/*
+ * mcv_update_match_bitmap
+ * Evaluate clauses using the MCV list, and update the match bitmap.
+ *
+ * A match bitmap keeps match/mismatch status for each MCV item, and we
+ * update it based on additional clauses. We also use it to skip items
+ * that can't possibly match (e.g. item marked as "mismatch" can't change
+ * to "match" when evaluating AND clause list).
+ *
+ * The function also returns a flag indicating whether there was an
+ * equality condition for all attributes, the minimum frequency in the MCV
+ * list, and a total MCV frequency (sum of frequencies for all items).
+ *
+ * XXX Currently the match bitmap uses a char for each MCV item, which is
+ * somewhat wasteful as we could do with just a single bit, thus reducing
+ * the size to ~1/8. It would also allow us to combine bitmaps simply using
+ * & and |, which should be faster than min/max. The bitmaps are fairly
+ * small, though (as we cap the MCV list size to 8k items).
+ */
+static void
+mcv_update_match_bitmap(PlannerInfo *root, List *clauses,
+ Bitmapset *keys, MCVList * mcvlist, char *matches,
+ bool is_or)
+{
+ int i;
+ ListCell *l;
+
+ /* The bitmap may be partially built. */
+ Assert(clauses != NIL);
+ Assert(list_length(clauses) >= 1);
+ Assert(mcvlist != NULL);
+ Assert(mcvlist->nitems > 0);
+ Assert(mcvlist->nitems <= STATS_MCVLIST_MAX_ITEMS);
+
+ /*
+ * Loop through the list of clauses, and for each of them evaluate all the
+ * MCV items not yet eliminated by the preceding clauses.
+ */
+ foreach(l, clauses)
+ {
+ Node *clause = (Node *) lfirst(l);
+
+ /* if it's a RestrictInfo, then extract the clause */
+ if (IsA(clause, RestrictInfo))
+ clause = (Node *) ((RestrictInfo *) clause)->clause;
+
+ /*
+ * Handle the various types of clauses - OpClause, NullTest and
+ * AND/OR/NOT
+ */
+ if (is_opclause(clause))
+ {
+ OpExpr *expr = (OpExpr *) clause;
+ bool varonleft = true;
+ bool ok;
+ FmgrInfo opproc;
+
+ /* get procedure computing operator selectivity */
+ RegProcedure oprrest = get_oprrest(expr->opno);
+
+ fmgr_info(get_opcode(expr->opno), &opproc);
+
+ ok = (NumRelids(clause) == 1) &&
+ (is_pseudo_constant_clause(lsecond(expr->args)) ||
+ (varonleft = false,
+ is_pseudo_constant_clause(linitial(expr->args))));
+
+ if (ok)
+ {
+
+ FmgrInfo gtproc;
+ Var *var = (varonleft) ? linitial(expr->args) : lsecond(expr->args);
+ Const *cst = (varonleft) ? lsecond(expr->args) : linitial(expr->args);
+ bool isgt = (!varonleft);
+
+ TypeCacheEntry *typecache
+ = lookup_type_cache(var->vartype, TYPECACHE_GT_OPR);
+
+ /* match the attribute to a dimension of the statistic */
+ int idx = bms_member_index(keys, var->varattno);
+
+ fmgr_info(get_opcode(typecache->gt_opr), >proc);
+
+ /*
+ * Walk through the MCV items and evaluate the current clause.
+ * We can skip items that were already ruled out, and
+ * terminate if there are no remaining MCV items that might
+ * possibly match.
+ */
+ for (i = 0; i < mcvlist->nitems; i++)
+ {
+ bool mismatch = false;
+ MCVItem *item = mcvlist->items[i];
+
+ /*
+ * For AND-lists, we can also mark NULL items as 'no
+ * match' (and then skip them). For OR-lists this is not
+ * possible.
+ */
+ if ((!is_or) && item->isnull[idx])
+ matches[i] = STATS_MATCH_NONE;
+
+ /* skip MCV items that were already ruled out */
+ if ((!is_or) && (matches[i] == STATS_MATCH_NONE))
+ continue;
+ else if (is_or && (matches[i] == STATS_MATCH_FULL))
+ continue;
+
+ switch (oprrest)
+ {
+ case F_EQSEL:
+ case F_NEQSEL:
+
+ /*
+ * We don't care about isgt in equality, because
+ * it does not matter whether it's (var op const)
+ * or (const op var).
+ */
+ mismatch = !DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ cst->constvalue,
+ item->values[idx]));
+
+ break;
+
+ case F_SCALARLTSEL: /* column < constant */
+ case F_SCALARLESEL: /* column <= constant */
+ case F_SCALARGTSEL: /* column > constant */
+ case F_SCALARGESEL: /* column >= constant */
+
+ /*
+ * First check whether the constant is below the
+ * lower boundary (in that case we can skip the
+ * bucket, because there's no overlap).
+ */
+ if (isgt)
+ mismatch = !DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ cst->constvalue,
+ item->values[idx]));
+ else
+ mismatch = !DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ item->values[idx],
+ cst->constvalue));
+
+ break;
+ }
+
+ /*
+ * XXX The conditions on matches[i] are not needed, as we
+ * skip MCV items that can't become true/false, depending
+ * on the current flag. See beginning of the loop over MCV
+ * items.
+ */
+
+ if ((is_or) && (!mismatch))
+ {
+ /* OR - was MATCH_NONE, but will be MATCH_FULL */
+ matches[i] = STATS_MATCH_FULL;
+ continue;
+ }
+ else if ((!is_or) && mismatch)
+ {
+ /* AND - was MATC_FULL, but will be MATCH_NONE */
+ matches[i] = STATS_MATCH_NONE;
+ continue;
+ }
+
+ }
+ }
+ }
+ else if (IsA(clause, NullTest))
+ {
+ NullTest *expr = (NullTest *) clause;
+ Var *var = (Var *) (expr->arg);
+
+ /* match the attribute to a dimension of the statistic */
+ int idx = bms_member_index(keys, var->varattno);
+
+ /*
+ * Walk through the MCV items and evaluate the current clause. We
+ * can skip items that were already ruled out, and terminate if
+ * there are no remaining MCV items that might possibly match.
+ */
+ for (i = 0; i < mcvlist->nitems; i++)
+ {
+ char match = STATS_MATCH_NONE; /* assume mismatch */
+ MCVItem *item = mcvlist->items[i];
+
+ /* if the clause mismatches the MCV item, set it as MATCH_NONE */
+ switch (expr->nulltesttype)
+ {
+ case IS_NULL:
+ match = (item->isnull[idx]) ? STATS_MATCH_FULL : match;
+ break;
+
+ case IS_NOT_NULL:
+ match = (!item->isnull[idx]) ? STATS_MATCH_FULL : match;
+ break;
+ }
+
+ /* now, update the match bitmap, depending on OR/AND type */
+ if (is_or)
+ matches[i] = Max(matches[i], match);
+ else
+ matches[i] = Min(matches[i], match);
+ }
+ }
+ else if (or_clause(clause) || and_clause(clause))
+ {
+ /* AND/OR clause, with all subclauses being compatible */
+
+ int i;
+ BoolExpr *bool_clause = ((BoolExpr *) clause);
+ List *bool_clauses = bool_clause->args;
+
+ /* match/mismatch bitmap for each MCV item */
+ char *bool_matches = NULL;
+
+ Assert(bool_clauses != NIL);
+ Assert(list_length(bool_clauses) >= 2);
+
+ /* by default none of the MCV items matches the clauses */
+ bool_matches = palloc0(sizeof(char) * mcvlist->nitems);
+
+ if (or_clause(clause))
+ {
+ /* OR clauses assume nothing matches, initially */
+ memset(bool_matches, STATS_MATCH_NONE, sizeof(char) * mcvlist->nitems);
+ }
+ else
+ {
+ /* AND clauses assume everything matches, initially */
+ memset(bool_matches, STATS_MATCH_FULL, sizeof(char) * mcvlist->nitems);
+ }
+
+ /* build the match bitmap for the OR-clauses */
+ mcv_update_match_bitmap(root, bool_clauses, keys,
+ mcvlist, bool_matches,
+ or_clause(clause));
+
+ /*
+ * Merge the bitmap produced by mcv_update_match_bitmap into the
+ * current one. We need to consider if we're evaluating AND or OR
+ * condition when merging the results.
+ */
+ for (i = 0; i < mcvlist->nitems; i++)
+ {
+ /* Is this OR or AND clause? */
+ if (is_or)
+ matches[i] = Max(matches[i], bool_matches[i]);
+ else
+ matches[i] = Min(matches[i], bool_matches[i]);
+ }
+
+ pfree(bool_matches);
+ }
+ else if (not_clause(clause))
+ {
+ /* NOT clause, with all subclauses compatible */
+
+ int i;
+ BoolExpr *not_clause = ((BoolExpr *) clause);
+ List *not_args = not_clause->args;
+
+ /* match/mismatch bitmap for each MCV item */
+ char *not_matches = NULL;
+
+ Assert(not_args != NIL);
+ Assert(list_length(not_args) == 1);
+
+ /* by default none of the MCV items matches the clauses */
+ not_matches = palloc0(sizeof(char) * mcvlist->nitems);
+
+ /* NOT clauses assume nothing matches, initially */
+ memset(not_matches, STATS_MATCH_FULL, sizeof(char) * mcvlist->nitems);
+
+ /* build the match bitmap for the NOT-clause */
+ mcv_update_match_bitmap(root, not_args, keys,
+ mcvlist, not_matches, false);
+
+ /*
+ * Merge the bitmap produced by mcv_update_match_bitmap into the
+ * current one.
+ */
+ for (i = 0; i < mcvlist->nitems; i++)
+ {
+ /*
+ * When handling a NOT clause, we need to invert the result
+ * before merging it into the global result.
+ */
+ if (not_matches[i] == STATS_MATCH_NONE)
+ not_matches[i] = STATS_MATCH_FULL;
+ else
+ not_matches[i] = STATS_MATCH_NONE;
+
+ /* Is this OR or AND clause? */
+ if (is_or)
+ matches[i] = Max(matches[i], not_matches[i]);
+ else
+ matches[i] = Min(matches[i], not_matches[i]);
+ }
+
+ pfree(not_matches);
+ }
+ else if (IsA(clause, Var))
+ {
+ /* Var (has to be a boolean Var, possibly from below NOT) */
+
+ Var *var = (Var *) (clause);
+
+ /* match the attribute to a dimension of the statistic */
+ int idx = bms_member_index(keys, var->varattno);
+
+ Assert(var->vartype == BOOLOID);
+
+ /*
+ * Walk through the MCV items and evaluate the current clause. We
+ * can skip items that were already ruled out, and terminate if
+ * there are no remaining MCV items that might possibly match.
+ */
+ for (i = 0; i < mcvlist->nitems; i++)
+ {
+ MCVItem *item = mcvlist->items[i];
+ bool match = STATS_MATCH_NONE;
+
+ /* if the item is NULL, it's a mismatch */
+ if (!item->isnull[idx] && DatumGetBool(item->values[idx]))
+ match = STATS_MATCH_FULL;
+
+ /* now, update the match bitmap, depending on OR/AND type */
+ if (is_or)
+ matches[i] = Max(matches[i], match);
+ else
+ matches[i] = Min(matches[i], match);
+ }
+ }
+ else
+ {
+ elog(ERROR, "unknown clause type: %d", clause->type);
+ }
+ }
+}
+
+
+/*
+ * mcv_clauselist_selectivity
+ * Return the selectivity estimate of clauses using MCV list.
+ *
+ * It also produces two interesting selectivities - total selectivity of
+ * all the MCV items combined, and selectivity of the least frequent item
+ * in the list.
+ */
+Selectivity
+mcv_clauselist_selectivity(PlannerInfo *root, StatisticExtInfo *stat,
+ List *clauses, int varRelid,
+ JoinType jointype, SpecialJoinInfo *sjinfo,
+ RelOptInfo *rel,
+ Selectivity *basesel, Selectivity *totalsel)
+{
+ int i;
+ MCVList *mcv;
+ Selectivity s = 0.0;
+
+ /* match/mismatch bitmap for each MCV item */
+ char *matches = NULL;
+
+ /* load the MCV list stored in the statistics object */
+ mcv = statext_mcv_load(stat->statOid);
+
+ /* by default all the MCV items match the clauses fully */
+ matches = palloc0(sizeof(char) * mcv->nitems);
+ memset(matches, STATS_MATCH_FULL, sizeof(char) * mcv->nitems);
+
+ mcv_update_match_bitmap(root, clauses, stat->keys, mcv,
+ matches, false);
+
+ /* sum frequencies for all the matching MCV items */
+ *basesel = 0.0;
+ *totalsel = 0.0;
+ for (i = 0; i < mcv->nitems; i++)
+ {
+ *totalsel += mcv->items[i]->frequency;
+
+ if (matches[i] != STATS_MATCH_NONE)
+ {
+ /* XXX Shouldn't the basesel be outside the if condition? */
+ *basesel += mcv->items[i]->base_frequency;
+ s += mcv->items[i]->frequency;
+ }
+ }
+
+ return s;
+}
diff --git a/src/backend/statistics/mvdistinct.c b/src/backend/statistics/mvdistinct.c
index 593c219839..fb74ed3618 100644
--- a/src/backend/statistics/mvdistinct.c
+++ b/src/backend/statistics/mvdistinct.c
@@ -23,8 +23,6 @@
*/
#include "postgres.h"
-#include <math.h>
-
#include "access/htup_details.h"
#include "catalog/pg_statistic_ext.h"
#include "utils/fmgrprotos.h"
@@ -39,7 +37,6 @@
static double ndistinct_for_combination(double totalrows, int numrows,
HeapTuple *rows, VacAttrStats **stats,
int k, int *combination);
-static double estimate_ndistinct(double totalrows, int numrows, int d, int f1);
static int n_choose_k(int n, int k);
static int num_combinations(int n);
@@ -508,31 +505,6 @@ ndistinct_for_combination(double totalrows, int numrows, HeapTuple *rows,
return estimate_ndistinct(totalrows, numrows, d, f1);
}
-/* The Duj1 estimator (already used in analyze.c). */
-static double
-estimate_ndistinct(double totalrows, int numrows, int d, int f1)
-{
- double numer,
- denom,
- ndistinct;
-
- numer = (double) numrows * (double) d;
-
- denom = (double) (numrows - f1) +
- (double) f1 * (double) numrows / totalrows;
-
- ndistinct = numer / denom;
-
- /* Clamp to sane range in case of roundoff error */
- if (ndistinct < (double) d)
- ndistinct = (double) d;
-
- if (ndistinct > totalrows)
- ndistinct = totalrows;
-
- return floor(ndistinct + 0.5);
-}
-
/*
* n_choose_k
* computes binomial coefficients using an algorithm that is both
diff --git a/src/backend/utils/adt/ruleutils.c b/src/backend/utils/adt/ruleutils.c
index 03e9a28a63..c941c3310b 100644
--- a/src/backend/utils/adt/ruleutils.c
+++ b/src/backend/utils/adt/ruleutils.c
@@ -1504,6 +1504,7 @@ pg_get_statisticsobj_worker(Oid statextid, bool missing_ok)
bool isnull;
bool ndistinct_enabled;
bool dependencies_enabled;
+ bool mcv_enabled;
int i;
statexttup = SearchSysCache1(STATEXTOID, ObjectIdGetDatum(statextid));
@@ -1539,6 +1540,7 @@ pg_get_statisticsobj_worker(Oid statextid, bool missing_ok)
ndistinct_enabled = false;
dependencies_enabled = false;
+ mcv_enabled = false;
for (i = 0; i < ARR_DIMS(arr)[0]; i++)
{
@@ -1546,6 +1548,8 @@ pg_get_statisticsobj_worker(Oid statextid, bool missing_ok)
ndistinct_enabled = true;
if (enabled[i] == STATS_EXT_DEPENDENCIES)
dependencies_enabled = true;
+ if (enabled[i] == STATS_EXT_MCV)
+ mcv_enabled = true;
}
/*
@@ -1555,13 +1559,27 @@ pg_get_statisticsobj_worker(Oid statextid, bool missing_ok)
* statistics types on a newer postgres version, if the statistics had all
* options enabled on the original version.
*/
- if (!ndistinct_enabled || !dependencies_enabled)
+ if (!ndistinct_enabled || !dependencies_enabled || !mcv_enabled)
{
+ bool gotone = false;
+
appendStringInfoString(&buf, " (");
+
if (ndistinct_enabled)
+ {
appendStringInfoString(&buf, "ndistinct");
- else if (dependencies_enabled)
- appendStringInfoString(&buf, "dependencies");
+ gotone = true;
+ }
+
+ if (dependencies_enabled)
+ {
+ appendStringInfo(&buf, "%sdependencies", gotone ? ", " : "");
+ gotone = true;
+ }
+
+ if (mcv_enabled)
+ appendStringInfo(&buf, "%smcv", gotone ? ", " : "");
+
appendStringInfoChar(&buf, ')');
}
diff --git a/src/backend/utils/adt/selfuncs.c b/src/backend/utils/adt/selfuncs.c
index f1c78ffb65..fdfc0d6a1b 100644
--- a/src/backend/utils/adt/selfuncs.c
+++ b/src/backend/utils/adt/selfuncs.c
@@ -3735,6 +3735,171 @@ estimate_num_groups(PlannerInfo *root, List *groupExprs, double input_rows,
}
/*
+ * estimate_num_groups_simple
+ * Estimate number of groups in a relation.
+ *
+ * A simplified version of estimate_num_groups, assuming all expressions
+ * are only plain Vars from a single relation, and that no filtering is
+ * happenning.
+ */
+double
+estimate_num_groups_simple(PlannerInfo *root, List *vars)
+{
+ List *varinfos = NIL;
+ double numdistinct;
+ ListCell *l;
+
+ RelOptInfo *rel;
+ double reldistinct = 1;
+ double relmaxndistinct = reldistinct;
+ int relvarcount = 0;
+
+
+ /*
+ * If no grouping columns, there's exactly one group. (This can't happen
+ * for normal cases with GROUP BY or DISTINCT, but it is possible for
+ * corner cases with set operations.)
+ */
+ if (vars == NIL)
+ return 1.0;
+
+ /*
+ * We expect only variables from a single relation.
+ */
+ Assert(NumRelids((Node *) vars) == 1);
+
+ /*
+ * Find the unique Vars used, treating an expression as a Var if we can
+ * find stats for it. For each one, record the statistical estimate of
+ * number of distinct values (total in its table).
+ */
+ numdistinct = 1.0;
+
+ foreach(l, vars)
+ {
+ Var *var = (Var *) lfirst(l);
+ VariableStatData vardata;
+
+ Assert(IsA(var, Var));
+
+ /*
+ * If examine_variable is able to deduce anything about the GROUP BY
+ * expression, treat it as a single variable even if it's really more
+ * complicated.
+ */
+ examine_variable(root, (Node *) var, 0, &vardata);
+ if (HeapTupleIsValid(vardata.statsTuple) || vardata.isunique)
+ {
+ varinfos = add_unique_group_var(root, varinfos,
+ (Node *) var, &vardata);
+ ReleaseVariableStats(vardata);
+ continue;
+ }
+ ReleaseVariableStats(vardata);
+ }
+
+ Assert(varinfos);
+
+ /*
+ * Get the numdistinct estimate for the Vars of this rel.
+ *
+ * We
+ * iteratively search for multivariate n-distinct with maximum number
+ * of vars; assuming that each var group is independent of the others,
+ * we multiply them together. Any remaining relvarinfos after no more
+ * multivariate matches are found are assumed independent too, so
+ * their individual ndistinct estimates are multiplied also.
+ *
+ * While iterating, count how many separate numdistinct values we
+ * apply. We apply a fudge factor below, but only if we multiplied
+ * more than one such values.
+ */
+ while (varinfos)
+ {
+ double mvndistinct;
+
+ rel = ((GroupVarInfo *) linitial(varinfos))->rel;
+
+ if (estimate_multivariate_ndistinct(root, rel, &varinfos,
+ &mvndistinct))
+ {
+ reldistinct *= mvndistinct;
+ if (relmaxndistinct < mvndistinct)
+ relmaxndistinct = mvndistinct;
+ relvarcount++;
+ }
+ else
+ {
+ foreach(l, varinfos)
+ {
+ GroupVarInfo *varinfo = (GroupVarInfo *) lfirst(l);
+
+ reldistinct *= varinfo->ndistinct;
+ if (relmaxndistinct < varinfo->ndistinct)
+ relmaxndistinct = varinfo->ndistinct;
+ relvarcount++;
+ }
+
+ /* we're done with this relation */
+ varinfos = NIL;
+ }
+ }
+
+ /*
+ * Sanity check --- don't divide by zero if empty relation.
+ */
+ Assert(IS_SIMPLE_REL(rel));
+ if (rel->tuples > 0)
+ {
+ /*
+ * Clamp to size of rel, or size of rel / 10 if multiple Vars. The
+ * fudge factor is because the Vars are probably correlated but we
+ * don't know by how much. We should never clamp to less than the
+ * largest ndistinct value for any of the Vars, though, since
+ * there will surely be at least that many groups.
+ */
+ double clamp = rel->tuples;
+
+ if (relvarcount > 1)
+ {
+ clamp *= 0.1;
+ if (clamp < relmaxndistinct)
+ {
+ clamp = relmaxndistinct;
+ /* for sanity in case some ndistinct is too large: */
+ if (clamp > rel->tuples)
+ clamp = rel->tuples;
+ }
+ }
+ if (reldistinct > clamp)
+ reldistinct = clamp;
+
+ /*
+ * We're assuming we are returning all rows.
+ */
+ reldistinct = clamp_row_est(reldistinct);
+
+ /*
+ * Update estimate of total distinct groups.
+ */
+ numdistinct *= reldistinct;
+
+ /* Guard against out-of-range answers */
+ if (numdistinct > rel->tuples)
+ numdistinct = rel->tuples;
+ }
+
+ if (numdistinct < 1.0)
+ numdistinct = 1.0;
+
+ /* Round off */
+ numdistinct = ceil(numdistinct);
+
+ return numdistinct;
+
+}
+
+/*
* Estimate hash bucket statistics when the specified expression is used
* as a hash key for the given number of buckets.
*
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 4ca0db1d0c..394301caa8 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -2518,7 +2518,8 @@ describeOneTableDetails(const char *schemaname,
" JOIN pg_catalog.pg_attribute a ON (stxrelid = a.attrelid AND\n"
" a.attnum = s.attnum AND NOT attisdropped)) AS columns,\n"
" 'd' = any(stxkind) AS ndist_enabled,\n"
- " 'f' = any(stxkind) AS deps_enabled\n"
+ " 'f' = any(stxkind) AS deps_enabled,\n"
+ " 'm' = any(stxkind) AS mcv_enabled\n"
"FROM pg_catalog.pg_statistic_ext stat "
"WHERE stxrelid = '%s'\n"
"ORDER BY 1;",
@@ -2555,6 +2556,12 @@ describeOneTableDetails(const char *schemaname,
if (strcmp(PQgetvalue(result, i, 6), "t") == 0)
{
appendPQExpBuffer(&buf, "%sdependencies", gotone ? ", " : "");
+ gotone = true;
+ }
+
+ if (strcmp(PQgetvalue(result, i, 7), "t") == 0)
+ {
+ appendPQExpBuffer(&buf, "%smcv", gotone ? ", " : "");
}
appendPQExpBuffer(&buf, ") ON %s FROM %s",
diff --git a/src/include/catalog/pg_cast.dat b/src/include/catalog/pg_cast.dat
index cf007528fd..dff3a9a08a 100644
--- a/src/include/catalog/pg_cast.dat
+++ b/src/include/catalog/pg_cast.dat
@@ -324,6 +324,12 @@
{ castsource => 'pg_dependencies', casttarget => 'text', castfunc => '0',
castcontext => 'i', castmethod => 'i' },
+# pg_mcv_list can be coerced to, but not from, bytea and text
+{ castsource => 'pg_mcv_list', casttarget => 'bytea', castfunc => '0',
+ castcontext => 'i', castmethod => 'b' },
+{ castsource => 'pg_mcv_list', casttarget => 'text', castfunc => '0',
+ castcontext => 'i', castmethod => 'i' },
+
# Datetime category
{ castsource => 'abstime', casttarget => 'date', castfunc => 'date(abstime)',
castcontext => 'a', castmethod => 'f' },
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index a14651010f..3cfcafcb11 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -5073,6 +5073,30 @@
proname => 'pg_dependencies_send', provolatile => 's', prorettype => 'bytea',
proargtypes => 'pg_dependencies', prosrc => 'pg_dependencies_send' },
+{ oid => '4002', descr => 'I/O',
+ proname => 'pg_mcv_list_in', prorettype => 'pg_mcv_list',
+ proargtypes => 'cstring', prosrc => 'pg_mcv_list_in' },
+{ oid => '4003', descr => 'I/O',
+ proname => 'pg_mcv_list_out', prorettype => 'cstring',
+ proargtypes => 'pg_mcv_list', prosrc => 'pg_mcv_list_out' },
+{ oid => '4004', descr => 'I/O',
+ proname => 'pg_mcv_list_recv', provolatile => 's',
+ prorettype => 'pg_mcv_list', proargtypes => 'internal',
+ prosrc => 'pg_mcv_list_recv' },
+{ oid => '4005', descr => 'I/O',
+ proname => 'pg_mcv_list_send', provolatile => 's', prorettype => 'bytea',
+ proargtypes => 'pg_mcv_list', prosrc => 'pg_mcv_list_send' },
+
+{ oid => '3424',
+ descr => 'details about MCV list items',
+ proname => 'pg_mcv_list_items', prorows => '1000', proisstrict => 'f',
+ proretset => 't', provolatile => 's', prorettype => 'record',
+ proargtypes => 'pg_mcv_list',
+ proallargtypes => '{pg_mcv_list,int4,text,_bool,float8,float8}',
+ proargmodes => '{i,o,o,o,o,o}',
+ proargnames => '{mcv_list,index,values,nulls,frequency,base_frequency}',
+ prosrc => 'pg_stats_ext_mcvlist_items' },
+
{ oid => '1928', descr => 'statistics: number of scans done for table/index',
proname => 'pg_stat_get_numscans', provolatile => 's', proparallel => 'r',
prorettype => 'int8', proargtypes => 'oid',
diff --git a/src/include/catalog/pg_statistic_ext.h b/src/include/catalog/pg_statistic_ext.h
index 443798ae52..7ddbee63c9 100644
--- a/src/include/catalog/pg_statistic_ext.h
+++ b/src/include/catalog/pg_statistic_ext.h
@@ -47,6 +47,7 @@ CATALOG(pg_statistic_ext,3381,StatisticExtRelationId)
* to build */
pg_ndistinct stxndistinct; /* ndistinct coefficients (serialized) */
pg_dependencies stxdependencies; /* dependencies (serialized) */
+ pg_mcv_list stxmcv; /* MCV (serialized) */
#endif
} FormData_pg_statistic_ext;
@@ -62,6 +63,7 @@ typedef FormData_pg_statistic_ext *Form_pg_statistic_ext;
#define STATS_EXT_NDISTINCT 'd'
#define STATS_EXT_DEPENDENCIES 'f'
+#define STATS_EXT_MCV 'm'
#endif /* EXPOSE_TO_CLIENT_CODE */
diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat
index 48e01cd694..0ff25e87a7 100644
--- a/src/include/catalog/pg_type.dat
+++ b/src/include/catalog/pg_type.dat
@@ -165,6 +165,13 @@
typoutput => 'pg_dependencies_out', typreceive => 'pg_dependencies_recv',
typsend => 'pg_dependencies_send', typalign => 'i', typstorage => 'x',
typcollation => '100' },
+{ oid => '4001', oid_symbol => 'PGMCVLISTOID',
+ descr => 'multivariate MCV list',
+ typname => 'pg_mcv_list', typlen => '-1', typbyval => 'f',
+ typcategory => 'S', typinput => 'pg_mcv_list_in',
+ typoutput => 'pg_mcv_list_out', typreceive => 'pg_mcv_list_recv',
+ typsend => 'pg_mcv_list_send', typalign => 'i', typstorage => 'x',
+ typcollation => '100' },
{ oid => '32', oid_symbol => 'PGDDLCOMMANDOID',
descr => 'internal type for passing CollectedCommand',
typname => 'pg_ddl_command', typlen => 'SIZEOF_POINTER', typbyval => 't',
diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index 5af96fdc8a..775a2bd1c6 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -198,6 +198,12 @@ extern void analyze_rel(Oid relid, RangeVar *relation, int options,
VacuumParams *params, List *va_cols, bool in_outer_xact,
BufferAccessStrategy bstrategy);
extern bool std_typanalyze(VacAttrStats *stats);
+extern int analyze_mcv_list(int *mcv_counts,
+ int num_mcv,
+ double stadistinct,
+ double stanullfrac,
+ int samplerows,
+ double totalrows);
/* in utils/misc/sampling.c --- duplicate of declarations in utils/sampling.h */
extern double anl_random_fract(void);
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 77ca7ff837..e6cded0597 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -215,6 +215,12 @@ extern Selectivity clause_selectivity(PlannerInfo *root,
int varRelid,
JoinType jointype,
SpecialJoinInfo *sjinfo);
+extern Selectivity clauselist_selectivity_simple(PlannerInfo *root,
+ List *clauses,
+ int varRelid,
+ JoinType jointype,
+ SpecialJoinInfo *sjinfo,
+ Bitmapset *estimatedclauses);
extern void cost_gather_merge(GatherMergePath *path, PlannerInfo *root,
RelOptInfo *rel, ParamPathInfo *param_info,
Cost input_startup_cost, Cost input_total_cost,
diff --git a/src/include/statistics/extended_stats_internal.h b/src/include/statistics/extended_stats_internal.h
index b3ca0c1229..11159b58ee 100644
--- a/src/include/statistics/extended_stats_internal.h
+++ b/src/include/statistics/extended_stats_internal.h
@@ -31,6 +31,15 @@ typedef struct
int tupno; /* position index for tuple it came from */
} ScalarItem;
+/* (de)serialization info */
+typedef struct DimensionInfo
+{
+ int nvalues; /* number of deduplicated values */
+ int nbytes; /* number of bytes (serialized) */
+ int typlen; /* pg_type.typlen */
+ bool typbyval; /* pg_type.typbyval */
+} DimensionInfo;
+
/* multi-sort */
typedef struct MultiSortSupportData
{
@@ -44,6 +53,7 @@ typedef struct SortItem
{
Datum *values;
bool *isnull;
+ int count;
} SortItem;
extern MVNDistinct *statext_ndistinct_build(double totalrows,
@@ -57,6 +67,12 @@ extern MVDependencies *statext_dependencies_build(int numrows, HeapTuple *rows,
extern bytea *statext_dependencies_serialize(MVDependencies *dependencies);
extern MVDependencies *statext_dependencies_deserialize(bytea *data);
+extern MCVList * statext_mcv_build(int numrows, HeapTuple *rows,
+ Bitmapset *attrs, VacAttrStats **stats,
+ double totalrows);
+extern bytea *statext_mcv_serialize(MCVList * mcv, VacAttrStats **stats);
+extern MCVList * statext_mcv_deserialize(bytea *data);
+
extern MultiSortSupport multi_sort_init(int ndims);
extern void multi_sort_add_dimension(MultiSortSupport mss, int sortdim,
Oid oper);
@@ -65,5 +81,32 @@ extern int multi_sort_compare_dim(int dim, const SortItem *a,
const SortItem *b, MultiSortSupport mss);
extern int multi_sort_compare_dims(int start, int end, const SortItem *a,
const SortItem *b, MultiSortSupport mss);
+extern int compare_scalars_simple(const void *a, const void *b, void *arg);
+extern int compare_datums_simple(Datum a, Datum b, SortSupport ssup);
+
+extern void *bsearch_arg(const void *key, const void *base,
+ size_t nmemb, size_t size,
+ int (*compar) (const void *, const void *, void *),
+ void *arg);
+
+extern int *build_attnums(Bitmapset *attrs);
+
+extern SortItem *build_sorted_items(int numrows, HeapTuple *rows,
+ TupleDesc tdesc, MultiSortSupport mss,
+ int numattrs, int *attnums);
+
+extern int bms_member_index(Bitmapset *keys, AttrNumber varattno);
+
+extern double estimate_ndistinct(double totalrows, int numrows, int d, int f1);
+
+extern Selectivity mcv_clauselist_selectivity(PlannerInfo *root,
+ StatisticExtInfo *stat,
+ List *clauses,
+ int varRelid,
+ JoinType jointype,
+ SpecialJoinInfo *sjinfo,
+ RelOptInfo *rel,
+ Selectivity *basesel,
+ Selectivity *totalsel);
#endif /* EXTENDED_STATS_INTERNAL_H */
diff --git a/src/include/statistics/statistics.h b/src/include/statistics/statistics.h
index 8009fee322..e69d6a0232 100644
--- a/src/include/statistics/statistics.h
+++ b/src/include/statistics/statistics.h
@@ -16,6 +16,14 @@
#include "commands/vacuum.h"
#include "nodes/relation.h"
+/*
+ * Degree of how much MCV item matches a clause.
+ * This is then considered when computing the selectivity.
+ */
+#define STATS_MATCH_NONE 0 /* no match at all */
+#define STATS_MATCH_PARTIAL 1 /* partial match */
+#define STATS_MATCH_FULL 2 /* full match */
+
#define STATS_MAX_DIMENSIONS 8 /* max number of attributes */
/* Multivariate distinct coefficients */
@@ -78,8 +86,42 @@ typedef struct MVDependencies
/* size of the struct excluding the deps array */
#define SizeOfDependencies (offsetof(MVDependencies, ndeps) + sizeof(uint32))
+/* used to flag stats serialized to bytea */
+#define STATS_MCV_MAGIC 0xE1A651C2 /* marks serialized
+ * bytea */
+#define STATS_MCV_TYPE_BASIC 1 /* basic MCV list type */
+
+/* max items in MCV list (mostly arbitrary number) */
+#define STATS_MCVLIST_MAX_ITEMS 8192
+
+/*
+ * Multivariate MCV (most-common value) lists
+ *
+ * A straight-forward extension of MCV items - i.e. a list (array) of
+ * combinations of attribute values, together with a frequency and null flags.
+ */
+typedef struct MCVItem
+{
+ double frequency; /* frequency of this combination */
+ double base_frequency; /* frequency if independent */
+ bool *isnull; /* lags of NULL values (up to 32 columns) */
+ Datum *values; /* variable-length (ndimensions) */
+} MCVItem;
+
+/* multivariate MCV list - essentally an array of MCV items */
+typedef struct MCVList
+{
+ uint32 magic; /* magic constant marker */
+ uint32 type; /* type of MCV list (BASIC) */
+ uint32 nitems; /* number of MCV items in the array */
+ AttrNumber ndimensions; /* number of dimensions */
+ Oid types[STATS_MAX_DIMENSIONS]; /* OIDs of data types */
+ MCVItem **items; /* array of MCV items */
+} MCVList;
+
extern MVNDistinct *statext_ndistinct_load(Oid mvoid);
extern MVDependencies *statext_dependencies_load(Oid mvoid);
+extern MCVList * statext_mcv_load(Oid mvoid);
extern void BuildRelationExtStatistics(Relation onerel, double totalrows,
int numrows, HeapTuple *rows,
@@ -92,6 +134,13 @@ extern Selectivity dependencies_clauselist_selectivity(PlannerInfo *root,
SpecialJoinInfo *sjinfo,
RelOptInfo *rel,
Bitmapset **estimatedclauses);
+extern Selectivity statext_clauselist_selectivity(PlannerInfo *root,
+ List *clauses,
+ int varRelid,
+ JoinType jointype,
+ SpecialJoinInfo *sjinfo,
+ RelOptInfo *rel,
+ Bitmapset **estimatedclauses);
extern bool has_stats_of_kind(List *stats, char requiredkind);
extern StatisticExtInfo *choose_best_statistics(List *stats,
Bitmapset *attnums, char requiredkind);
diff --git a/src/include/utils/selfuncs.h b/src/include/utils/selfuncs.h
index 95e44280c4..4e9aaca6b5 100644
--- a/src/include/utils/selfuncs.h
+++ b/src/include/utils/selfuncs.h
@@ -209,6 +209,8 @@ extern void mergejoinscansel(PlannerInfo *root, Node *clause,
extern double estimate_num_groups(PlannerInfo *root, List *groupExprs,
double input_rows, List **pgset);
+extern double estimate_num_groups_simple(PlannerInfo *root, List *vars);
+
extern void estimate_hash_bucket_stats(PlannerInfo *root,
Node *hashkey, double nbuckets,
Selectivity *mcv_freq,
diff --git a/src/test/regress/expected/create_table_like.out b/src/test/regress/expected/create_table_like.out
index 8d4543bfe8..0f97355165 100644
--- a/src/test/regress/expected/create_table_like.out
+++ b/src/test/regress/expected/create_table_like.out
@@ -243,7 +243,7 @@ Indexes:
Check constraints:
"ctlt1_a_check" CHECK (length(a) > 2)
Statistics objects:
- "public"."ctlt_all_a_b_stat" (ndistinct, dependencies) ON a, b FROM ctlt_all
+ "public"."ctlt_all_a_b_stat" (ndistinct, dependencies, mcv) ON a, b FROM ctlt_all
SELECT c.relname, objsubid, description FROM pg_description, pg_index i, pg_class c WHERE classoid = 'pg_class'::regclass AND objoid = i.indexrelid AND c.oid = i.indexrelid AND i.indrelid = 'ctlt_all'::regclass ORDER BY c.relname, objsubid;
relname | objsubid | description
diff --git a/src/test/regress/expected/opr_sanity.out b/src/test/regress/expected/opr_sanity.out
index 3c6d853ffb..d13f0928d0 100644
--- a/src/test/regress/expected/opr_sanity.out
+++ b/src/test/regress/expected/opr_sanity.out
@@ -902,11 +902,12 @@ WHERE c.castmethod = 'b' AND
pg_node_tree | text | 0 | i
pg_ndistinct | bytea | 0 | i
pg_dependencies | bytea | 0 | i
+ pg_mcv_list | bytea | 0 | i
cidr | inet | 0 | i
xml | text | 0 | a
xml | character varying | 0 | a
xml | character | 0 | a
-(9 rows)
+(10 rows)
-- **************** pg_conversion ****************
-- Look for illegal values in pg_conversion fields.
diff --git a/src/test/regress/expected/stats_ext.out b/src/test/regress/expected/stats_ext.out
index 054a381dad..5d05962c04 100644
--- a/src/test/regress/expected/stats_ext.out
+++ b/src/test/regress/expected/stats_ext.out
@@ -58,7 +58,7 @@ ALTER TABLE ab1 DROP COLUMN a;
b | integer | | |
c | integer | | |
Statistics objects:
- "public"."ab1_b_c_stats" (ndistinct, dependencies) ON b, c FROM ab1
+ "public"."ab1_b_c_stats" (ndistinct, dependencies, mcv) ON b, c FROM ab1
-- Ensure statistics are dropped when table is
SELECT stxname FROM pg_statistic_ext WHERE stxname LIKE 'ab1%';
@@ -206,7 +206,7 @@ SELECT stxkind, stxndistinct
FROM pg_statistic_ext WHERE stxrelid = 'ndistinct'::regclass;
stxkind | stxndistinct
---------+---------------------------------------------------------
- {d,f} | {"3, 4": 301, "3, 6": 301, "4, 6": 301, "3, 4, 6": 301}
+ {d,f,m} | {"3, 4": 301, "3, 6": 301, "4, 6": 301, "3, 4, 6": 301}
(1 row)
-- Hash Aggregate, thanks to estimates improved by the statistic
@@ -272,7 +272,7 @@ SELECT stxkind, stxndistinct
FROM pg_statistic_ext WHERE stxrelid = 'ndistinct'::regclass;
stxkind | stxndistinct
---------+-------------------------------------------------------------
- {d,f} | {"3, 4": 2550, "3, 6": 800, "4, 6": 1632, "3, 4, 6": 10000}
+ {d,f,m} | {"3, 4": 2550, "3, 6": 800, "4, 6": 1632, "3, 4, 6": 10000}
(1 row)
-- plans using Group Aggregate, thanks to using correct esimates
@@ -509,3 +509,316 @@ EXPLAIN (COSTS OFF)
(5 rows)
RESET random_page_cost;
+-- MCV lists
+CREATE TABLE mcv_lists (
+ filler1 TEXT,
+ filler2 NUMERIC,
+ a INT,
+ b TEXT,
+ filler3 DATE,
+ c INT,
+ d TEXT
+);
+SET random_page_cost = 1.2;
+CREATE INDEX mcv_lists_ab_idx ON mcv_lists (a, b);
+CREATE INDEX mcv_lists_abc_idx ON mcv_lists (a, b, c);
+-- random data (no MCV list)
+INSERT INTO mcv_lists (a, b, c, filler1)
+ SELECT mod(i,37), mod(i,41), mod(i,43), mod(i,47) FROM generate_series(1,5000) s(i);
+ANALYZE mcv_lists;
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a = 1 AND b = '1';
+ QUERY PLAN
+---------------------------------------------------
+ Bitmap Heap Scan on mcv_lists
+ Recheck Cond: ((a = 1) AND (b = '1'::text))
+ -> Bitmap Index Scan on mcv_lists_abc_idx
+ Index Cond: ((a = 1) AND (b = '1'::text))
+(4 rows)
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a = 1 AND b = '1' AND c = 1;
+ QUERY PLAN
+---------------------------------------------------------
+ Index Scan using mcv_lists_abc_idx on mcv_lists
+ Index Cond: ((a = 1) AND (b = '1'::text) AND (c = 1))
+(2 rows)
+
+-- create statistics
+CREATE STATISTICS mcv_lists_stats (mcv) ON a, b, c FROM mcv_lists;
+ANALYZE mcv_lists;
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a = 1 AND b = '1';
+ QUERY PLAN
+---------------------------------------------------
+ Bitmap Heap Scan on mcv_lists
+ Recheck Cond: ((a = 1) AND (b = '1'::text))
+ -> Bitmap Index Scan on mcv_lists_abc_idx
+ Index Cond: ((a = 1) AND (b = '1'::text))
+(4 rows)
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a = 1 AND b = '1' AND c = 1;
+ QUERY PLAN
+---------------------------------------------------------
+ Index Scan using mcv_lists_abc_idx on mcv_lists
+ Index Cond: ((a = 1) AND (b = '1'::text) AND (c = 1))
+(2 rows)
+
+-- 100 distinct combinations, all in the MCV list
+TRUNCATE mcv_lists;
+DROP STATISTICS mcv_lists_stats;
+INSERT INTO mcv_lists (a, b, c, filler1)
+ SELECT mod(i,100), mod(i,50), mod(i,25), i FROM generate_series(1,5000) s(i);
+ANALYZE mcv_lists;
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a = 1 AND b = '1';
+ QUERY PLAN
+-------------------------------------------------
+ Index Scan using mcv_lists_abc_idx on mcv_lists
+ Index Cond: ((a = 1) AND (b = '1'::text))
+(2 rows)
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a < 1 AND b < '1';
+ QUERY PLAN
+-------------------------------------------------
+ Index Scan using mcv_lists_abc_idx on mcv_lists
+ Index Cond: ((a < 1) AND (b < '1'::text))
+(2 rows)
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a <= 0 AND b <= '0';
+ QUERY PLAN
+-------------------------------------------------
+ Index Scan using mcv_lists_abc_idx on mcv_lists
+ Index Cond: ((a <= 0) AND (b <= '0'::text))
+(2 rows)
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a = 1 AND b = '1' AND c = 1;
+ QUERY PLAN
+---------------------------------------------------------
+ Index Scan using mcv_lists_abc_idx on mcv_lists
+ Index Cond: ((a = 1) AND (b = '1'::text) AND (c = 1))
+(2 rows)
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a < 5 AND b < '1' AND c < 5;
+ QUERY PLAN
+---------------------------------------------------------
+ Index Scan using mcv_lists_abc_idx on mcv_lists
+ Index Cond: ((a < 5) AND (b < '1'::text) AND (c < 5))
+(2 rows)
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a <= 4 AND b <= '0' AND c <= 4;
+ QUERY PLAN
+------------------------------------------------------------
+ Index Scan using mcv_lists_abc_idx on mcv_lists
+ Index Cond: ((a <= 4) AND (b <= '0'::text) AND (c <= 4))
+(2 rows)
+
+-- create statistics
+CREATE STATISTICS mcv_lists_stats (mcv) ON a, b, c FROM mcv_lists;
+ANALYZE mcv_lists;
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a = 1 AND b = '1';
+ QUERY PLAN
+---------------------------------------------------
+ Bitmap Heap Scan on mcv_lists
+ Recheck Cond: ((a = 1) AND (b = '1'::text))
+ -> Bitmap Index Scan on mcv_lists_abc_idx
+ Index Cond: ((a = 1) AND (b = '1'::text))
+(4 rows)
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a < 1 AND b < '1';
+ QUERY PLAN
+---------------------------------------------------
+ Bitmap Heap Scan on mcv_lists
+ Recheck Cond: ((a < 1) AND (b < '1'::text))
+ -> Bitmap Index Scan on mcv_lists_abc_idx
+ Index Cond: ((a < 1) AND (b < '1'::text))
+(4 rows)
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a <= 0 AND b <= '0';
+ QUERY PLAN
+-----------------------------------------------------
+ Bitmap Heap Scan on mcv_lists
+ Recheck Cond: ((a <= 0) AND (b <= '0'::text))
+ -> Bitmap Index Scan on mcv_lists_abc_idx
+ Index Cond: ((a <= 0) AND (b <= '0'::text))
+(4 rows)
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a = 1 AND b = '1' AND c = 1;
+ QUERY PLAN
+---------------------------------------------------
+ Bitmap Heap Scan on mcv_lists
+ Recheck Cond: ((a = 1) AND (b = '1'::text))
+ Filter: (c = 1)
+ -> Bitmap Index Scan on mcv_lists_ab_idx
+ Index Cond: ((a = 1) AND (b = '1'::text))
+(5 rows)
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a < 5 AND b < '1' AND c < 5;
+ QUERY PLAN
+---------------------------------------------------
+ Bitmap Heap Scan on mcv_lists
+ Recheck Cond: ((a < 5) AND (b < '1'::text))
+ Filter: (c < 5)
+ -> Bitmap Index Scan on mcv_lists_ab_idx
+ Index Cond: ((a < 5) AND (b < '1'::text))
+(5 rows)
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a <= 4 AND b <= '0' AND c <= 4;
+ QUERY PLAN
+-----------------------------------------------------
+ Bitmap Heap Scan on mcv_lists
+ Recheck Cond: ((a <= 4) AND (b <= '0'::text))
+ Filter: (c <= 4)
+ -> Bitmap Index Scan on mcv_lists_ab_idx
+ Index Cond: ((a <= 4) AND (b <= '0'::text))
+(5 rows)
+
+-- check change of column type resets the MCV statistics
+ALTER TABLE mcv_lists ALTER COLUMN c TYPE numeric;
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a = 1 AND b = '1';
+ QUERY PLAN
+-------------------------------------------------
+ Index Scan using mcv_lists_abc_idx on mcv_lists
+ Index Cond: ((a = 1) AND (b = '1'::text))
+(2 rows)
+
+ANALYZE mcv_lists;
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a = 1 AND b = '1';
+ QUERY PLAN
+---------------------------------------------------
+ Bitmap Heap Scan on mcv_lists
+ Recheck Cond: ((a = 1) AND (b = '1'::text))
+ -> Bitmap Index Scan on mcv_lists_abc_idx
+ Index Cond: ((a = 1) AND (b = '1'::text))
+(4 rows)
+
+-- 100 distinct combinations with NULL values, all in the MCV list
+TRUNCATE mcv_lists;
+DROP STATISTICS mcv_lists_stats;
+INSERT INTO mcv_lists (a, b, c, filler1)
+ SELECT
+ (CASE WHEN mod(i,100) = 1 THEN NULL ELSE mod(i,100) END),
+ (CASE WHEN mod(i,50) = 1 THEN NULL ELSE mod(i,50) END),
+ (CASE WHEN mod(i,25) = 1 THEN NULL ELSE mod(i,25) END),
+ i
+ FROM generate_series(1,5000) s(i);
+ANALYZE mcv_lists;
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a IS NULL AND b IS NULL;
+ QUERY PLAN
+-------------------------------------------------
+ Index Scan using mcv_lists_abc_idx on mcv_lists
+ Index Cond: ((a IS NULL) AND (b IS NULL))
+(2 rows)
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a IS NULL AND b IS NULL AND c IS NULL;
+ QUERY PLAN
+------------------------------------------------
+ Index Scan using mcv_lists_ab_idx on mcv_lists
+ Index Cond: ((a IS NULL) AND (b IS NULL))
+ Filter: (c IS NULL)
+(3 rows)
+
+-- create statistics
+CREATE STATISTICS mcv_lists_stats (mcv) ON a, b, c FROM mcv_lists;
+ANALYZE mcv_lists;
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a IS NULL AND b IS NULL;
+ QUERY PLAN
+---------------------------------------------------
+ Bitmap Heap Scan on mcv_lists
+ Recheck Cond: ((a IS NULL) AND (b IS NULL))
+ -> Bitmap Index Scan on mcv_lists_abc_idx
+ Index Cond: ((a IS NULL) AND (b IS NULL))
+(4 rows)
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a IS NULL AND b IS NULL AND c IS NULL;
+ QUERY PLAN
+---------------------------------------------------
+ Bitmap Heap Scan on mcv_lists
+ Recheck Cond: ((a IS NULL) AND (b IS NULL))
+ Filter: (c IS NULL)
+ -> Bitmap Index Scan on mcv_lists_ab_idx
+ Index Cond: ((a IS NULL) AND (b IS NULL))
+(5 rows)
+
+RESET random_page_cost;
+-- mcv with arrays
+CREATE TABLE mcv_lists_arrays (
+ a TEXT[],
+ b NUMERIC[],
+ c INT[]
+);
+INSERT INTO mcv_lists_arrays (a, b, c)
+ SELECT
+ ARRAY[md5((i/100)::text), md5((i/100-1)::text), md5((i/100+1)::text)],
+ ARRAY[(i/100-1)::numeric/1000, (i/100)::numeric/1000, (i/100+1)::numeric/1000],
+ ARRAY[(i/100-1), i/100, (i/100+1)]
+ FROM generate_series(1,5000) s(i);
+CREATE STATISTICS mcv_lists_arrays_stats (mcv) ON a, b, c
+ FROM mcv_lists_arrays;
+ANALYZE mcv_lists_arrays;
+-- mcv with bool
+CREATE TABLE mcv_lists_bool (
+ a BOOL,
+ b BOOL,
+ c BOOL
+);
+INSERT INTO mcv_lists_bool (a, b, c)
+ SELECT
+ (mod(i,2) = 0), (mod(i,4) = 0), (mod(i,8) = 0)
+ FROM generate_series(1,10000) s(i);
+CREATE INDEX mcv_lists_bool_ab_idx ON mcv_lists_bool (a, b);
+CREATE INDEX mcv_lists_bool_abc_idx ON mcv_lists_bool (a, b, c);
+CREATE STATISTICS mcv_lists_bool_stats (mcv) ON a, b, c
+ FROM mcv_lists_bool;
+ANALYZE mcv_lists_bool;
+EXPLAIN (COSTS OFF) SELECT * FROM mcv_lists_bool WHERE a AND b AND c;
+ QUERY PLAN
+----------------------------------------------------------------
+ Bitmap Heap Scan on mcv_lists_bool
+ Filter: (a AND b AND c)
+ -> Bitmap Index Scan on mcv_lists_bool_abc_idx
+ Index Cond: ((a = true) AND (b = true) AND (c = true))
+(4 rows)
+
+EXPLAIN (COSTS OFF) SELECT * FROM mcv_lists_bool WHERE NOT a AND b AND c;
+ QUERY PLAN
+----------------------------------------------------------
+ Index Scan using mcv_lists_bool_ab_idx on mcv_lists_bool
+ Index Cond: ((a = false) AND (b = true))
+ Filter: ((NOT a) AND b AND c)
+(3 rows)
+
+EXPLAIN (COSTS OFF) SELECT * FROM mcv_lists_bool WHERE NOT a AND NOT b AND c;
+ QUERY PLAN
+----------------------------------------------------------------
+ Index Only Scan using mcv_lists_bool_abc_idx on mcv_lists_bool
+ Index Cond: ((a = false) AND (b = false) AND (c = true))
+ Filter: ((NOT a) AND (NOT b) AND c)
+(3 rows)
+
+EXPLAIN (COSTS OFF) SELECT * FROM mcv_lists_bool WHERE NOT a AND b AND NOT c;
+ QUERY PLAN
+----------------------------------------------------------
+ Index Scan using mcv_lists_bool_ab_idx on mcv_lists_bool
+ Index Cond: ((a = false) AND (b = true))
+ Filter: ((NOT a) AND b AND (NOT c))
+(3 rows)
+
diff --git a/src/test/regress/expected/type_sanity.out b/src/test/regress/expected/type_sanity.out
index b1419d4bc2..a56d6c5231 100644
--- a/src/test/regress/expected/type_sanity.out
+++ b/src/test/regress/expected/type_sanity.out
@@ -72,8 +72,9 @@ WHERE p1.typtype not in ('c','d','p') AND p1.typname NOT LIKE E'\\_%'
194 | pg_node_tree
3361 | pg_ndistinct
3402 | pg_dependencies
+ 4001 | pg_mcv_list
210 | smgr
-(4 rows)
+(5 rows)
-- Make sure typarray points to a varlena array type of our own base
SELECT p1.oid, p1.typname as basetype, p2.typname as arraytype,
diff --git a/src/test/regress/sql/stats_ext.sql b/src/test/regress/sql/stats_ext.sql
index 46acaadb39..ad1f103217 100644
--- a/src/test/regress/sql/stats_ext.sql
+++ b/src/test/regress/sql/stats_ext.sql
@@ -282,3 +282,184 @@ EXPLAIN (COSTS OFF)
SELECT * FROM functional_dependencies WHERE a = 1 AND b = '1' AND c = 1;
RESET random_page_cost;
+
+-- MCV lists
+CREATE TABLE mcv_lists (
+ filler1 TEXT,
+ filler2 NUMERIC,
+ a INT,
+ b TEXT,
+ filler3 DATE,
+ c INT,
+ d TEXT
+);
+
+SET random_page_cost = 1.2;
+
+CREATE INDEX mcv_lists_ab_idx ON mcv_lists (a, b);
+CREATE INDEX mcv_lists_abc_idx ON mcv_lists (a, b, c);
+
+-- random data (no MCV list)
+INSERT INTO mcv_lists (a, b, c, filler1)
+ SELECT mod(i,37), mod(i,41), mod(i,43), mod(i,47) FROM generate_series(1,5000) s(i);
+
+ANALYZE mcv_lists;
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a = 1 AND b = '1';
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a = 1 AND b = '1' AND c = 1;
+
+-- create statistics
+CREATE STATISTICS mcv_lists_stats (mcv) ON a, b, c FROM mcv_lists;
+
+ANALYZE mcv_lists;
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a = 1 AND b = '1';
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a = 1 AND b = '1' AND c = 1;
+
+-- 100 distinct combinations, all in the MCV list
+TRUNCATE mcv_lists;
+DROP STATISTICS mcv_lists_stats;
+
+INSERT INTO mcv_lists (a, b, c, filler1)
+ SELECT mod(i,100), mod(i,50), mod(i,25), i FROM generate_series(1,5000) s(i);
+
+ANALYZE mcv_lists;
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a = 1 AND b = '1';
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a < 1 AND b < '1';
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a <= 0 AND b <= '0';
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a = 1 AND b = '1' AND c = 1;
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a < 5 AND b < '1' AND c < 5;
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a <= 4 AND b <= '0' AND c <= 4;
+
+-- create statistics
+CREATE STATISTICS mcv_lists_stats (mcv) ON a, b, c FROM mcv_lists;
+
+ANALYZE mcv_lists;
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a = 1 AND b = '1';
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a < 1 AND b < '1';
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a <= 0 AND b <= '0';
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a = 1 AND b = '1' AND c = 1;
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a < 5 AND b < '1' AND c < 5;
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a <= 4 AND b <= '0' AND c <= 4;
+
+-- check change of column type resets the MCV statistics
+ALTER TABLE mcv_lists ALTER COLUMN c TYPE numeric;
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a = 1 AND b = '1';
+
+ANALYZE mcv_lists;
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a = 1 AND b = '1';
+
+-- 100 distinct combinations with NULL values, all in the MCV list
+TRUNCATE mcv_lists;
+DROP STATISTICS mcv_lists_stats;
+
+INSERT INTO mcv_lists (a, b, c, filler1)
+ SELECT
+ (CASE WHEN mod(i,100) = 1 THEN NULL ELSE mod(i,100) END),
+ (CASE WHEN mod(i,50) = 1 THEN NULL ELSE mod(i,50) END),
+ (CASE WHEN mod(i,25) = 1 THEN NULL ELSE mod(i,25) END),
+ i
+ FROM generate_series(1,5000) s(i);
+
+ANALYZE mcv_lists;
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a IS NULL AND b IS NULL;
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a IS NULL AND b IS NULL AND c IS NULL;
+
+-- create statistics
+CREATE STATISTICS mcv_lists_stats (mcv) ON a, b, c FROM mcv_lists;
+
+ANALYZE mcv_lists;
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a IS NULL AND b IS NULL;
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a IS NULL AND b IS NULL AND c IS NULL;
+
+RESET random_page_cost;
+
+-- mcv with arrays
+CREATE TABLE mcv_lists_arrays (
+ a TEXT[],
+ b NUMERIC[],
+ c INT[]
+);
+
+INSERT INTO mcv_lists_arrays (a, b, c)
+ SELECT
+ ARRAY[md5((i/100)::text), md5((i/100-1)::text), md5((i/100+1)::text)],
+ ARRAY[(i/100-1)::numeric/1000, (i/100)::numeric/1000, (i/100+1)::numeric/1000],
+ ARRAY[(i/100-1), i/100, (i/100+1)]
+ FROM generate_series(1,5000) s(i);
+
+CREATE STATISTICS mcv_lists_arrays_stats (mcv) ON a, b, c
+ FROM mcv_lists_arrays;
+
+ANALYZE mcv_lists_arrays;
+
+-- mcv with bool
+CREATE TABLE mcv_lists_bool (
+ a BOOL,
+ b BOOL,
+ c BOOL
+);
+
+INSERT INTO mcv_lists_bool (a, b, c)
+ SELECT
+ (mod(i,2) = 0), (mod(i,4) = 0), (mod(i,8) = 0)
+ FROM generate_series(1,10000) s(i);
+
+CREATE INDEX mcv_lists_bool_ab_idx ON mcv_lists_bool (a, b);
+
+CREATE INDEX mcv_lists_bool_abc_idx ON mcv_lists_bool (a, b, c);
+
+CREATE STATISTICS mcv_lists_bool_stats (mcv) ON a, b, c
+ FROM mcv_lists_bool;
+
+ANALYZE mcv_lists_bool;
+
+EXPLAIN (COSTS OFF) SELECT * FROM mcv_lists_bool WHERE a AND b AND c;
+
+EXPLAIN (COSTS OFF) SELECT * FROM mcv_lists_bool WHERE NOT a AND b AND c;
+
+EXPLAIN (COSTS OFF) SELECT * FROM mcv_lists_bool WHERE NOT a AND NOT b AND c;
+
+EXPLAIN (COSTS OFF) SELECT * FROM mcv_lists_bool WHERE NOT a AND b AND NOT c;
--
2.13.6
--------------0E1BE69E814B19BC8B911192--
^ permalink raw reply [nested|flat] 8+ messages in thread
* [PATCH 1/2] multivariate MCV lists
@ 2018-12-26 19:59 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 8+ messages in thread
From: Tomas Vondra @ 2018-12-26 19:59 UTC (permalink / raw)
---
doc/src/sgml/catalogs.sgml | 13 +-
doc/src/sgml/func.sgml | 77 +
doc/src/sgml/planstats.sgml | 114 +-
doc/src/sgml/ref/create_statistics.sgml | 28 +-
src/backend/commands/analyze.c | 8 +-
src/backend/commands/statscmds.c | 107 +-
src/backend/optimizer/path/clausesel.c | 89 +-
src/backend/optimizer/util/plancat.c | 12 +
src/backend/parser/parse_utilcmd.c | 2 +
src/backend/statistics/Makefile | 2 +-
src/backend/statistics/README | 4 +
src/backend/statistics/README.mcv | 140 ++
src/backend/statistics/dependencies.c | 83 +-
src/backend/statistics/extended_stats.c | 543 +++++-
src/backend/statistics/mcv.c | 1656 +++++++++++++++++
src/backend/statistics/mvdistinct.c | 28 -
src/backend/utils/adt/ruleutils.c | 24 +-
src/backend/utils/adt/selfuncs.c | 165 ++
src/bin/psql/describe.c | 9 +-
src/include/catalog/pg_cast.dat | 6 +
src/include/catalog/pg_proc.dat | 24 +
src/include/catalog/pg_statistic_ext.h | 2 +
src/include/catalog/pg_type.dat | 7 +
src/include/commands/vacuum.h | 6 +
src/include/optimizer/cost.h | 6 +
.../statistics/extended_stats_internal.h | 43 +
src/include/statistics/statistics.h | 49 +
src/include/utils/selfuncs.h | 2 +
.../regress/expected/create_table_like.out | 2 +-
src/test/regress/expected/opr_sanity.out | 3 +-
src/test/regress/expected/stats_ext.out | 319 +++-
src/test/regress/expected/type_sanity.out | 3 +-
src/test/regress/sql/stats_ext.sql | 181 ++
33 files changed, 3630 insertions(+), 127 deletions(-)
create mode 100644 src/backend/statistics/README.mcv
create mode 100644 src/backend/statistics/mcv.c
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index af4d0625ea..1f2a45c442 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -6515,7 +6515,8 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
An array containing codes for the enabled statistic kinds;
valid values are:
<literal>d</literal> for n-distinct statistics,
- <literal>f</literal> for functional dependency statistics
+ <literal>f</literal> for functional dependency statistics, and
+ <literal>m</literal> for most common values (MCV) list statistics
</entry>
</row>
@@ -6538,6 +6539,16 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</entry>
</row>
+ <row>
+ <entry><structfield>stxmcv</structfield></entry>
+ <entry><type>pg_mcv_list</type></entry>
+ <entry></entry>
+ <entry>
+ MCV (most-common values) list statistics, serialized as
+ <structname>pg_mcv_list</structname> type.
+ </entry>
+ </row>
+
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 37860996a6..dcfa5a0311 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21263,4 +21263,81 @@ CREATE EVENT TRIGGER test_table_rewrite_oid
</sect2>
</sect1>
+ <sect1 id="functions-statistics">
+ <title>Statistics Information Functions</title>
+
+ <indexterm zone="functions-statistics">
+ <primary>function</primary>
+ <secondary>statistics</secondary>
+ </indexterm>
+
+ <para>
+ To inspect statistics defined using <command>CREATE STATISTICS</command>
+ command, <productname>PostgreSQL</productname> provides multiple functions.
+ </para>
+
+ <sect2 id="functions-statistics-mcv">
+ <title>Inspecting MCV lists</title>
+
+ <indexterm>
+ <primary>pg_mcv_list_items</primary>
+ <secondary>pg_mcv_list</secondary>
+ </indexterm>
+
+ <para>
+ <function>pg_mcv_list_items</function> returns a list of all items
+ stored in a multi-column <literal>MCV</literal> list, and returns the
+ following columns:
+
+ <informaltable>
+ <tgroup cols="3">
+ <thead>
+ <row>
+ <entry>Name</entry>
+ <entry>Type</entry>
+ <entry>Description</entry>
+ </row>
+ </thead>
+
+ <tbody>
+ <row>
+ <entry><literal>index</literal></entry>
+ <entry><type>int</type></entry>
+ <entry>index of the item in the <literal>MCV</literal> list</entry>
+ </row>
+ <row>
+ <entry><literal>values</literal></entry>
+ <entry><type>text[]</type></entry>
+ <entry>values stored in the MCV item</entry>
+ </row>
+ <row>
+ <entry><literal>nulls</literal></entry>
+ <entry><type>boolean[]</type></entry>
+ <entry>flags identifying <literal>NULL</literal> values</entry>
+ </row>
+ <row>
+ <entry><literal>frequency</literal></entry>
+ <entry><type>double precision</type></entry>
+ <entry>frequency of this <literal>MCV</literal> item</entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ </para>
+
+ <para>
+ The <function>pg_mcv_list_items</function> function can be used like this:
+
+<programlisting>
+SELECT m.* FROM pg_statistic_ext,
+ pg_mcv_list_items(stxmcv) m WHERE stxname = 'stts';
+</programlisting>
+
+ Values of the <type>pg_mcv_list</type> can be obtained only from the
+ <literal>pg_statistic.stxmcv</literal> column.
+ </para>
+ </sect2>
+
+ </sect1>
+
</chapter>
diff --git a/doc/src/sgml/planstats.sgml b/doc/src/sgml/planstats.sgml
index ef643ad064..de8ef165c9 100644
--- a/doc/src/sgml/planstats.sgml
+++ b/doc/src/sgml/planstats.sgml
@@ -455,7 +455,7 @@ rows = (outer_cardinality * inner_cardinality) * selectivity
<secondary>multivariate</secondary>
</indexterm>
- <sect2>
+ <sect2 id="functional-dependencies">
<title>Functional Dependencies</title>
<para>
@@ -540,7 +540,7 @@ EXPLAIN (ANALYZE, TIMING OFF) SELECT * FROM t WHERE a = 1 AND b = 1;
</para>
</sect2>
- <sect2>
+ <sect2 id="multivariate-ndistinct-counts">
<title>Multivariate N-Distinct Counts</title>
<para>
@@ -585,6 +585,116 @@ EXPLAIN (ANALYZE, TIMING OFF) SELECT COUNT(*) FROM t GROUP BY a, b;
</para>
</sect2>
+
+ <sect2 id="mcv-lists">
+ <title>MCV lists</title>
+
+ <para>
+ As explained in <xref linkend="functional-dependencies"/>, functional
+ dependencies are very cheap and efficient type of statistics, but their
+ main limitation is their global nature (only tracking dependencies at
+ the column level, not between individual column values).
+ </para>
+
+ <para>
+ This section introduces multivariate variant of <acronym>MCV</acronym>
+ (most-common values) lists, a straight-forward extension of the per-column
+ statistics described in <xref linkend="row-estimation-examples"/>. This
+ statistics adresses the limitation by storing individual values, but it
+ is naturally more expensive, both in terms of storage and planning time.
+ </para>
+
+ <para>
+ Let's look at the query from <xref linkend="functional-dependencies"/>
+ again, but this time with a <acronym>MCV</acronym> list created on the
+ same set of columns (be sure to drop the functional dependencies, to
+ make sure the planner uses the newly created statistics).
+
+<programlisting>
+DROP STATISTICS stts;
+CREATE STATISTICS stts2 (mcv) ON a, b FROM t;
+ANALYZE t;
+EXPLAIN (ANALYZE, TIMING OFF) SELECT * FROM t WHERE a = 1 AND b = 1;
+ QUERY PLAN
+-------------------------------------------------------------------------------
+ Seq Scan on t (cost=0.00..195.00 rows=100 width=8) (actual rows=100 loops=1)
+ Filter: ((a = 1) AND (b = 1))
+ Rows Removed by Filter: 9900
+</programlisting>
+
+ The estimate is as accurate as with the functional dependencies, mostly
+ thanks to the table being fairly small and having a simple distribution
+ with low number of distinct values. Before looking at the second query,
+ which was not handled by functional dependencies particularly well,
+ let's inspect the <acronym>MCV</acronym> list a bit.
+ </para>
+
+ <para>
+ Inspecting the <acronym>MCV</acronym> list is possible using
+ <function>pg_mcv_list_items</function> set-returning function.
+
+<programlisting>
+SELECT m.* FROM pg_statistic_ext,
+ pg_mcv_list_items(stxmcv) m WHERE stxname = 'stts2';
+ index | values | nulls | frequency
+-------+---------+-------+-----------
+ 0 | {0,0} | {f,f} | 0.01
+ 1 | {1,1} | {f,f} | 0.01
+ 2 | {2,2} | {f,f} | 0.01
+...
+ 49 | {49,49} | {f,f} | 0.01
+ 50 | {50,0} | {f,f} | 0.01
+...
+ 97 | {97,47} | {f,f} | 0.01
+ 98 | {98,48} | {f,f} | 0.01
+ 99 | {99,49} | {f,f} | 0.01
+(100 rows)
+</programlisting>
+
+ Which confirms there are 100 distinct combinations in the two columns,
+ and all of them are about equally likely (1% frequency for each one).
+ Had there been any null values in either of the columns, this would be
+ identified in the <structfield>nulls</structfield> column.
+ </para>
+
+ <para>
+ When estimating the selectivity, the planner applies all the conditions
+ on items in the <acronym>MCV</acronym> list, and them sums the frequencies
+ of the matching ones. See <function>mcv_clauselist_selectivity</function>
+ in <filename>src/backend/statistics/mcv.c</filename> for details.
+ </para>
+
+ <para>
+ Compared to functional dependencies, <acronym>MCV</acronym> lists have two
+ major advantages. Firstly, the list stores actual values, making it possible
+ to decide which combinations are compatible.
+
+<programlisting>
+EXPLAIN (ANALYZE, TIMING OFF) SELECT * FROM t WHERE a = 1 AND b = 10;
+ QUERY PLAN
+---------------------------------------------------------------------------
+ Seq Scan on t (cost=0.00..195.00 rows=1 width=8) (actual rows=0 loops=1)
+ Filter: ((a = 1) AND (b = 10))
+ Rows Removed by Filter: 10000
+</programlisting>
+
+ Secondly, <acronym>MCV</acronym> lists handle a wider range of clause types,
+ not just equality clauses like functional dependencies. See for example the
+ example range query, presented earlier:
+
+<programlisting>
+EXPLAIN (ANALYZE, TIMING OFF) SELECT * FROM t WHERE a <= 49 AND b > 49;
+ QUERY PLAN
+---------------------------------------------------------------------------
+ Seq Scan on t (cost=0.00..195.00 rows=1 width=8) (actual rows=0 loops=1)
+ Filter: ((a <= 49) AND (b > 49))
+ Rows Removed by Filter: 10000
+</programlisting>
+
+ </para>
+
+ </sect2>
+
</sect1>
<sect1 id="planner-stats-security">
diff --git a/doc/src/sgml/ref/create_statistics.sgml b/doc/src/sgml/ref/create_statistics.sgml
index 539f5bded5..fcbfa569d0 100644
--- a/doc/src/sgml/ref/create_statistics.sgml
+++ b/doc/src/sgml/ref/create_statistics.sgml
@@ -83,7 +83,8 @@ CREATE STATISTICS [ IF NOT EXISTS ] <replaceable class="parameter">statistics_na
Currently supported kinds are
<literal>ndistinct</literal>, which enables n-distinct statistics, and
<literal>dependencies</literal>, which enables functional
- dependency statistics.
+ dependency statistics, and <literal>mcv</literal> which enables
+ most-common values lists.
If this clause is omitted, all supported statistics kinds are
included in the statistics object.
For more information, see <xref linkend="planner-stats-extended"/>
@@ -164,6 +165,31 @@ EXPLAIN ANALYZE SELECT * FROM t1 WHERE (a = 1) AND (b = 0);
conditions are redundant and does not underestimate the row count.
</para>
+ <para>
+ Create table <structname>t2</structname> with two perfectly correlated columns
+ (containing identical data), and a MCV list on those columns:
+
+<programlisting>
+CREATE TABLE t2 (
+ a int,
+ b int
+);
+
+INSERT INTO t2 SELECT mod(i,100), mod(i,100)
+ FROM generate_series(1,1000000) s(i);
+
+CREATE STATISTICS s2 WITH (mcv) ON (a, b) FROM t2;
+
+ANALYZE t2;
+
+-- valid combination (found in MCV)
+EXPLAIN ANALYZE SELECT * FROM t2 WHERE (a = 1) AND (b = 1);
+
+-- invalid combination (not found in MCV)
+EXPLAIN ANALYZE SELECT * FROM t2 WHERE (a = 1) AND (b = 2);
+</programlisting>
+ </para>
+
</refsect1>
<refsect1>
diff --git a/src/backend/commands/analyze.c b/src/backend/commands/analyze.c
index b5a7475db9..64ec958a52 100644
--- a/src/backend/commands/analyze.c
+++ b/src/backend/commands/analyze.c
@@ -1733,12 +1733,6 @@ static void compute_scalar_stats(VacAttrStatsP stats,
double totalrows);
static int compare_scalars(const void *a, const void *b, void *arg);
static int compare_mcvs(const void *a, const void *b);
-static int analyze_mcv_list(int *mcv_counts,
- int num_mcv,
- double stadistinct,
- double stanullfrac,
- int samplerows,
- double totalrows);
/*
@@ -2835,7 +2829,7 @@ compare_mcvs(const void *a, const void *b)
* number that are significantly more common than the values not in the list,
* and which are therefore deemed worth storing in the table's MCV list.
*/
-static int
+int
analyze_mcv_list(int *mcv_counts,
int num_mcv,
double stadistinct,
diff --git a/src/backend/commands/statscmds.c b/src/backend/commands/statscmds.c
index bfc0f1d1fa..0ea3ff2c34 100644
--- a/src/backend/commands/statscmds.c
+++ b/src/backend/commands/statscmds.c
@@ -71,11 +71,12 @@ CreateStatistics(CreateStatsStmt *stmt)
Oid relid;
ObjectAddress parentobject,
myself;
- Datum types[2]; /* one for each possible type of statistic */
+ Datum types[3]; /* one for each possible type of statistic */
int ntypes;
ArrayType *stxkind;
bool build_ndistinct;
bool build_dependencies;
+ bool build_mcv;
bool requested_type = false;
int i;
ListCell *cell;
@@ -270,6 +271,7 @@ CreateStatistics(CreateStatsStmt *stmt)
*/
build_ndistinct = false;
build_dependencies = false;
+ build_mcv = false;
foreach(cell, stmt->stat_types)
{
char *type = strVal((Value *) lfirst(cell));
@@ -284,6 +286,11 @@ CreateStatistics(CreateStatsStmt *stmt)
build_dependencies = true;
requested_type = true;
}
+ else if (strcmp(type, "mcv") == 0)
+ {
+ build_mcv = true;
+ requested_type = true;
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -295,6 +302,7 @@ CreateStatistics(CreateStatsStmt *stmt)
{
build_ndistinct = true;
build_dependencies = true;
+ build_mcv = true;
}
/* construct the char array of enabled statistic types */
@@ -303,6 +311,8 @@ CreateStatistics(CreateStatsStmt *stmt)
types[ntypes++] = CharGetDatum(STATS_EXT_NDISTINCT);
if (build_dependencies)
types[ntypes++] = CharGetDatum(STATS_EXT_DEPENDENCIES);
+ if (build_mcv)
+ types[ntypes++] = CharGetDatum(STATS_EXT_MCV);
Assert(ntypes > 0 && ntypes <= lengthof(types));
stxkind = construct_array(types, ntypes, CHAROID, 1, true, 'c');
@@ -327,6 +337,7 @@ CreateStatistics(CreateStatsStmt *stmt)
/* no statistics built yet */
nulls[Anum_pg_statistic_ext_stxndistinct - 1] = true;
nulls[Anum_pg_statistic_ext_stxdependencies - 1] = true;
+ nulls[Anum_pg_statistic_ext_stxmcv - 1] = true;
/* insert it into pg_statistic_ext */
htup = heap_form_tuple(statrel->rd_att, values, nulls);
@@ -422,23 +433,97 @@ RemoveStatisticsById(Oid statsOid)
* null until the next ANALYZE. (Note that the type change hasn't actually
* happened yet, so one option that's *not* on the table is to recompute
* immediately.)
+ *
+ * For both ndistinct and functional-dependencies stats, the on-disk
+ * representation is independent of the source column data types, and it is
+ * plausible to assume that the old statistic values will still be good for
+ * the new column contents. (Obviously, if the ALTER COLUMN TYPE has a USING
+ * expression that substantially alters the semantic meaning of the column
+ * values, this assumption could fail. But that seems like a corner case
+ * that doesn't justify zapping the stats in common cases.)
+ *
+ * For MCV lists that's not the case, as those statistics store the datums
+ * internally. In this case we simply reset the statistics value to NULL.
*/
void
UpdateStatisticsForTypeChange(Oid statsOid, Oid relationOid, int attnum,
Oid oldColumnType, Oid newColumnType)
{
+ Form_pg_statistic_ext staForm;
+ HeapTuple stup,
+ oldtup;
+ int i;
+
+ /* Do we need to reset anything? */
+ bool attribute_referenced;
+ bool reset_stats = false;
+
+ Relation rel;
+
+ Datum values[Natts_pg_statistic_ext];
+ bool nulls[Natts_pg_statistic_ext];
+ bool replaces[Natts_pg_statistic_ext];
+
+ oldtup = SearchSysCache1(STATEXTOID, ObjectIdGetDatum(statsOid));
+ if (!oldtup)
+ elog(ERROR, "cache lookup failed for statistics object %u", statsOid);
+ staForm = (Form_pg_statistic_ext) GETSTRUCT(oldtup);
+
+ /*
+ * If the modified attribute is not referenced by this statistic, we
+ * can simply leave the statistics alone.
+ */
+ attribute_referenced = false;
+ for (i = 0; i < staForm->stxkeys.dim1; i++)
+ if (attnum == staForm->stxkeys.values[i])
+ attribute_referenced = true;
+
/*
- * Currently, we don't actually need to do anything here. For both
- * ndistinct and functional-dependencies stats, the on-disk representation
- * is independent of the source column data types, and it is plausible to
- * assume that the old statistic values will still be good for the new
- * column contents. (Obviously, if the ALTER COLUMN TYPE has a USING
- * expression that substantially alters the semantic meaning of the column
- * values, this assumption could fail. But that seems like a corner case
- * that doesn't justify zapping the stats in common cases.)
- *
- * Future types of extended stats will likely require us to work harder.
+ * We can also leave the record as it is if there are no statistics
+ * including the datum values, like for example MCV lists.
*/
+ if (statext_is_kind_built(oldtup, STATS_EXT_MCV))
+ reset_stats = true;
+
+ /*
+ * If we can leave the statistics as it is, just do minimal cleanup
+ * and we're done.
+ */
+ if (!attribute_referenced && reset_stats)
+ {
+ ReleaseSysCache(oldtup);
+ return;
+ }
+
+ /*
+ * OK, we need to reset some statistics. So let's build the new tuple,
+ * replacing the affected statistics types with NULL.
+ */
+ memset(nulls, 0, Natts_pg_statistic_ext * sizeof(bool));
+ memset(replaces, 0, Natts_pg_statistic_ext * sizeof(bool));
+ memset(values, 0, Natts_pg_statistic_ext * sizeof(Datum));
+
+ if (statext_is_kind_built(oldtup, STATS_EXT_MCV))
+ {
+ replaces[Anum_pg_statistic_ext_stxmcv - 1] = true;
+ nulls[Anum_pg_statistic_ext_stxmcv - 1] = true;
+ }
+
+ rel = heap_open(StatisticExtRelationId, RowExclusiveLock);
+
+ /* replace the old tuple */
+ stup = heap_modify_tuple(oldtup,
+ RelationGetDescr(rel),
+ values,
+ nulls,
+ replaces);
+
+ ReleaseSysCache(oldtup);
+ CatalogTupleUpdate(rel, &stup->t_self, stup);
+
+ heap_freetuple(stup);
+
+ heap_close(rel, RowExclusiveLock);
}
/*
diff --git a/src/backend/optimizer/path/clausesel.c b/src/backend/optimizer/path/clausesel.c
index f4717942c3..fd09ad3c49 100644
--- a/src/backend/optimizer/path/clausesel.c
+++ b/src/backend/optimizer/path/clausesel.c
@@ -105,9 +105,6 @@ clauselist_selectivity(PlannerInfo *root,
Selectivity s1 = 1.0;
RelOptInfo *rel;
Bitmapset *estimatedclauses = NULL;
- RangeQueryClause *rqlist = NULL;
- ListCell *l;
- int listidx;
/*
* If there's exactly one clause, just go directly to
@@ -124,6 +121,25 @@ clauselist_selectivity(PlannerInfo *root,
rel = find_single_rel_for_clauses(root, clauses);
if (rel && rel->rtekind == RTE_RELATION && rel->statlist != NIL)
{
+ /*
+ * Estimate selectivity on any clauses applicable by stats tracking
+ * actual values first, then apply functional dependencies on the
+ * remaining clauses. The reasoning for this particular order is that
+ * the more complex stats can track more complex correlations between
+ * the attributes, and may be considered more reliable.
+ *
+ * For example MCV list can give us an exact selectivity for values in
+ * two columns, while functional dependencies can only provide
+ * information about overall strength of the dependency.
+ *
+ * 'estimatedclauses' is a bitmap of 0-based list positions of clauses
+ * used that way, so that we can ignore them later (not to estimate
+ * them twice).
+ */
+ s1 *= statext_clauselist_selectivity(root, clauses, varRelid,
+ jointype, sjinfo, rel,
+ &estimatedclauses);
+
/*
* Perform selectivity estimations on any clauses found applicable by
* dependencies_clauselist_selectivity. 'estimatedclauses' will be
@@ -133,17 +149,72 @@ clauselist_selectivity(PlannerInfo *root,
s1 *= dependencies_clauselist_selectivity(root, clauses, varRelid,
jointype, sjinfo, rel,
&estimatedclauses);
-
- /*
- * This would be the place to apply any other types of extended
- * statistics selectivity estimations for remaining clauses.
- */
}
/*
* Apply normal selectivity estimates for remaining clauses. We'll be
* careful to skip any clauses which were already estimated above.
- *
+ */
+ return s1 * clauselist_selectivity_simple(root, clauses, varRelid,
+ jointype, sjinfo,
+ estimatedclauses);
+}
+
+/*
+ * clauselist_selectivity_simple -
+ * Compute the selectivity of an implicitly-ANDed list of boolean
+ * expression clauses. The list can be empty, in which case 1.0
+ * must be returned. List elements may be either RestrictInfos
+ * or bare expression clauses --- the former is preferred since
+ * it allows caching of results.
+ *
+ * See clause_selectivity() for the meaning of the additional parameters.
+ *
+ * Our basic approach is to take the product of the selectivities of the
+ * subclauses. However, that's only right if the subclauses have independent
+ * probabilities, and in reality they are often NOT independent. So,
+ * we want to be smarter where we can.
+ *
+ * We also recognize "range queries", such as "x > 34 AND x < 42". Clauses
+ * are recognized as possible range query components if they are restriction
+ * opclauses whose operators have scalarltsel or a related function as their
+ * restriction selectivity estimator. We pair up clauses of this form that
+ * refer to the same variable. An unpairable clause of this kind is simply
+ * multiplied into the selectivity product in the normal way. But when we
+ * find a pair, we know that the selectivities represent the relative
+ * positions of the low and high bounds within the column's range, so instead
+ * of figuring the selectivity as hisel * losel, we can figure it as hisel +
+ * losel - 1. (To visualize this, see that hisel is the fraction of the range
+ * below the high bound, while losel is the fraction above the low bound; so
+ * hisel can be interpreted directly as a 0..1 value but we need to convert
+ * losel to 1-losel before interpreting it as a value. Then the available
+ * range is 1-losel to hisel. However, this calculation double-excludes
+ * nulls, so really we need hisel + losel + null_frac - 1.)
+ *
+ * If either selectivity is exactly DEFAULT_INEQ_SEL, we forget this equation
+ * and instead use DEFAULT_RANGE_INEQ_SEL. The same applies if the equation
+ * yields an impossible (negative) result.
+ *
+ * A free side-effect is that we can recognize redundant inequalities such
+ * as "x < 4 AND x < 5"; only the tighter constraint will be counted.
+ *
+ * Of course this is all very dependent on the behavior of the inequality
+ * selectivity functions; perhaps some day we can generalize the approach.
+ */
+Selectivity
+clauselist_selectivity_simple(PlannerInfo *root,
+ List *clauses,
+ int varRelid,
+ JoinType jointype,
+ SpecialJoinInfo *sjinfo,
+ Bitmapset *estimatedclauses)
+{
+ Selectivity s1 = 1.0;
+ RangeQueryClause *rqlist = NULL;
+ ListCell *l;
+ int listidx;
+
+ /*
* Anything that doesn't look like a potential rangequery clause gets
* multiplied into s1 and forgotten. Anything that does gets inserted into
* an rqlist entry.
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index a570ac0aab..2ae9945a28 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -1362,6 +1362,18 @@ get_relation_statistics(RelOptInfo *rel, Relation relation)
stainfos = lcons(info, stainfos);
}
+ if (statext_is_kind_built(htup, STATS_EXT_MCV))
+ {
+ StatisticExtInfo *info = makeNode(StatisticExtInfo);
+
+ info->statOid = statOid;
+ info->rel = rel;
+ info->kind = STATS_EXT_MCV;
+ info->keys = bms_copy(keys);
+
+ stainfos = lcons(info, stainfos);
+ }
+
ReleaseSysCache(htup);
bms_free(keys);
}
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index 52582d0a13..3ffd4fb210 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -1648,6 +1648,8 @@ generateClonedExtStatsStmt(RangeVar *heapRel, Oid heapRelid,
stat_types = lappend(stat_types, makeString("ndistinct"));
else if (enabled[i] == STATS_EXT_DEPENDENCIES)
stat_types = lappend(stat_types, makeString("dependencies"));
+ else if (enabled[i] == STATS_EXT_MCV)
+ stat_types = lappend(stat_types, makeString("mcv"));
else
elog(ERROR, "unrecognized statistics kind %c", enabled[i]);
}
diff --git a/src/backend/statistics/Makefile b/src/backend/statistics/Makefile
index 3404e4554a..d2815265fb 100644
--- a/src/backend/statistics/Makefile
+++ b/src/backend/statistics/Makefile
@@ -12,6 +12,6 @@ subdir = src/backend/statistics
top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
-OBJS = extended_stats.o dependencies.o mvdistinct.o
+OBJS = extended_stats.o dependencies.o mcv.o mvdistinct.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/statistics/README b/src/backend/statistics/README
index a8f00a590e..8f153a9e85 100644
--- a/src/backend/statistics/README
+++ b/src/backend/statistics/README
@@ -18,6 +18,8 @@ There are currently two kinds of extended statistics:
(b) soft functional dependencies (README.dependencies)
+ (c) MCV lists (README.mcv)
+
Compatible clause types
-----------------------
@@ -26,6 +28,8 @@ Each type of statistics may be used to estimate some subset of clause types.
(a) functional dependencies - equality clauses (AND), possibly IS NULL
+ (b) MCV lists - equality and inequality clauses (AND, OR, NOT), IS NULL
+
Currently, only OpExprs in the form Var op Const, or Const op Var are
supported, however it's feasible to expand the code later to also estimate the
selectivities on clauses such as Var op Var.
diff --git a/src/backend/statistics/README.mcv b/src/backend/statistics/README.mcv
new file mode 100644
index 0000000000..2910eca962
--- /dev/null
+++ b/src/backend/statistics/README.mcv
@@ -0,0 +1,140 @@
+MCV lists
+=========
+
+Multivariate MCV (most-common values) lists are a straightforward extension of
+regular MCV list, tracking most frequent combinations of values for a group of
+attributes.
+
+This works particularly well for columns with a small number of distinct values,
+as the list may include all the combinations and approximate the distribution
+very accurately.
+
+For columns with large number of distinct values (e.g. those with continuous
+domains), the list will only track the most frequent combinations. If the
+distribution is mostly uniform (all combinations about equally frequent), the
+MCV list will be empty.
+
+Estimates of some clauses (e.g. equality) based on MCV lists are more accurate
+than when using histograms.
+
+Also, MCV lists don't necessarily require sorting of the values (the fact that
+we use sorting when building them is implementation detail), but even more
+importantly the ordering is not built into the approximation (while histograms
+are built on ordering). So MCV lists work well even for attributes where the
+ordering of the data type is disconnected from the meaning of the data. For
+example we know how to sort strings, but it's unlikely to make much sense for
+city names (or other label-like attributes).
+
+
+Selectivity estimation
+----------------------
+
+The estimation, implemented in clauselist_mv_selectivity_mcvlist(), is quite
+simple in principle - we need to identify MCV items matching all the clauses
+and sum frequencies of all those items.
+
+Currently MCV lists support estimation of the following clause types:
+
+ (a) equality clauses WHERE (a = 1) AND (b = 2)
+ (b) inequality clauses WHERE (a < 1) AND (b >= 2)
+ (c) NULL clauses WHERE (a IS NULL) AND (b IS NOT NULL)
+ (d) OR clauses WHERE (a < 1) OR (b >= 2)
+
+It's possible to add support for additional clauses, for example:
+
+ (e) multi-var clauses WHERE (a > b)
+
+and possibly others. These are tasks for the future, not yet implemented.
+
+
+Estimating equality clauses
+---------------------------
+
+When computing selectivity estimate for equality clauses
+
+ (a = 1) AND (b = 2)
+
+we can do this estimate pretty exactly assuming that two conditions are met:
+
+ (1) there's an equality condition on all attributes of the statistic
+
+ (2) we find a matching item in the MCV list
+
+In this case we know the MCV item represents all tuples matching the clauses,
+and the selectivity estimate is complete (i.e. we don't need to perform
+estimation using the histogram). This is what we call 'full match'.
+
+When only (1) holds, but there's no matching MCV item, we don't know whether
+there are no such rows or just are not very frequent. We can however use the
+frequency of the least frequent MCV item as an upper bound for the selectivity.
+
+For a combination of equality conditions (not full-match case) we can clamp the
+selectivity by the minimum of selectivities for each condition. For example if
+we know the number of distinct values for each column, we can use 1/ndistinct
+as a per-column estimate. Or rather 1/ndistinct + selectivity derived from the
+MCV list.
+
+We should also probably only use the 'residual ndistinct' by exluding the items
+included in the MCV list (and also residual frequency):
+
+ f = (1.0 - sum(MCV frequencies)) / (ndistinct - ndistinct(MCV list))
+
+but it's worth pointing out the ndistinct values are multi-variate for the
+columns referenced by the equality conditions.
+
+Note: Only the "full match" limit is currently implemented.
+
+
+Hashed MCV (not yet implemented)
+--------------------------------
+
+Regular MCV lists have to include actual values for each item, so if those items
+are large the list may be quite large. This is especially true for multi-variate
+MCV lists, although the current implementation partially mitigates this by
+performing de-duplicating the values before storing them on disk.
+
+It's possible to only store hashes (32-bit values) instead of the actual values,
+significantly reducing the space requirements. Obviously, this would only make
+the MCV lists useful for estimating equality conditions (assuming the 32-bit
+hashes make the collisions rare enough).
+
+This might also complicate matching the columns to available stats.
+
+
+TODO Consider implementing hashed MCV list, storing just 32-bit hashes instead
+ of the actual values. This type of MCV list will be useful only for
+ estimating equality clauses, and will reduce space requirements for large
+ varlena types (in such cases we usually only want equality anyway).
+
+TODO Currently there's no logic to consider building only a MCV list (and not
+ building the histogram at all), except for doing this decision manually in
+ ADD STATISTICS.
+
+
+Inspecting the MCV list
+-----------------------
+
+Inspecting the regular (per-attribute) MCV lists is trivial, as it's enough
+to select the columns from pg_stats. The data is encoded as anyarrays, and
+all the items have the same data type, so anyarray provides a simple way to
+get a text representation.
+
+With multivariate MCV lists the columns may use different data types, making
+it impossible to use anyarrays. It might be possible to produce similar
+array-like representation, but that would complicate further processing and
+analysis of the MCV list.
+
+So instead the MCV lists are stored in a custom data type (pg_mcv_list),
+which however makes it more difficult to inspect the contents. To make that
+easier, there's a SRF returning detailed information about the MCV lists.
+
+ SELECT * FROM pg_mcv_list_items(stxmcv);
+
+It accepts one parameter - a pg_mcv_list value (which can only be obtained
+from pg_statistic_ext catalog, to defend against malicious input), and
+returns these columns:
+
+ - item index (0, ..., (nitems-1))
+ - values (string array)
+ - nulls only (boolean array)
+ - frequency (double precision)
diff --git a/src/backend/statistics/dependencies.c b/src/backend/statistics/dependencies.c
index 58d0df20f6..a4e8eef52f 100644
--- a/src/backend/statistics/dependencies.c
+++ b/src/backend/statistics/dependencies.c
@@ -201,14 +201,11 @@ static double
dependency_degree(int numrows, HeapTuple *rows, int k, AttrNumber *dependency,
VacAttrStats **stats, Bitmapset *attrs)
{
- int i,
- j;
- int nvalues = numrows * k;
+ int i;
MultiSortSupport mss;
SortItem *items;
- Datum *values;
- bool *isnull;
int *attnums;
+ int *attnums_dep;
/* counters valid within a group */
int group_size = 0;
@@ -223,26 +220,16 @@ dependency_degree(int numrows, HeapTuple *rows, int k, AttrNumber *dependency,
/* sort info for all attributes columns */
mss = multi_sort_init(k);
- /* data for the sort */
- items = (SortItem *) palloc(numrows * sizeof(SortItem));
- values = (Datum *) palloc(sizeof(Datum) * nvalues);
- isnull = (bool *) palloc(sizeof(bool) * nvalues);
-
- /* fix the pointers to values/isnull */
- for (i = 0; i < numrows; i++)
- {
- items[i].values = &values[i * k];
- items[i].isnull = &isnull[i * k];
- }
-
/*
- * Transform the bms into an array, to make accessing i-th member easier.
+ * Transform the bms into an array, to make accessing i-th member easier,
+ * and then construct a filtered version with only attnums referenced
+ * by the dependency we validate.
*/
- attnums = (int *) palloc(sizeof(int) * bms_num_members(attrs));
- i = 0;
- j = -1;
- while ((j = bms_next_member(attrs, j)) >= 0)
- attnums[i++] = j;
+ attnums = build_attnums(attrs);
+
+ attnums_dep = (int *)palloc(k * sizeof(int));
+ for (i = 0; i < k; i++)
+ attnums_dep[i] = attnums[dependency[i]];
/*
* Verify the dependency (a,b,...)->z, using a rather simple algorithm:
@@ -257,7 +244,7 @@ dependency_degree(int numrows, HeapTuple *rows, int k, AttrNumber *dependency,
* perhaps at some point it'd be worth using column-specific collations?
*/
- /* prepare the sort function for the first dimension, and SortItem array */
+ /* prepare the sort function for the dimensions */
for (i = 0; i < k; i++)
{
VacAttrStats *colstat = stats[dependency[i]];
@@ -270,19 +257,16 @@ dependency_degree(int numrows, HeapTuple *rows, int k, AttrNumber *dependency,
/* prepare the sort function for this dimension */
multi_sort_add_dimension(mss, i, type->lt_opr, type->typcollation);
-
- /* accumulate all the data for both columns into an array and sort it */
- for (j = 0; j < numrows; j++)
- {
- items[j].values[i] =
- heap_getattr(rows[j], attnums[dependency[i]],
- stats[i]->tupDesc, &items[j].isnull[i]);
- }
}
- /* sort the items so that we can detect the groups */
- qsort_arg((void *) items, numrows, sizeof(SortItem),
- multi_sort_compare, mss);
+ /*
+ * build an array of SortItem(s) sorted using the multi-sort support
+ *
+ * XXX This relies on all stats entries pointing to the same tuple
+ * descriptor. Not sure if that might not be the case.
+ */
+ items = build_sorted_items(numrows, rows, stats[0]->tupDesc,
+ mss, k, attnums_dep);
/*
* Walk through the sorted array, split it into rows according to the
@@ -325,9 +309,9 @@ dependency_degree(int numrows, HeapTuple *rows, int k, AttrNumber *dependency,
}
pfree(items);
- pfree(values);
- pfree(isnull);
pfree(mss);
+ pfree(attnums);
+ pfree(attnums_dep);
/* Compute the 'degree of validity' as (supporting/total). */
return (n_supporting_rows * 1.0 / numrows);
@@ -354,7 +338,6 @@ statext_dependencies_build(int numrows, HeapTuple *rows, Bitmapset *attrs,
VacAttrStats **stats)
{
int i,
- j,
k;
int numattrs;
int *attnums;
@@ -367,11 +350,7 @@ statext_dependencies_build(int numrows, HeapTuple *rows, Bitmapset *attrs,
/*
* Transform the bms into an array, to make accessing i-th member easier.
*/
- attnums = palloc(sizeof(int) * bms_num_members(attrs));
- i = 0;
- j = -1;
- while ((j = bms_next_member(attrs, j)) >= 0)
- attnums[i++] = j;
+ attnums = build_attnums(attrs);
Assert(numattrs >= 2);
@@ -918,9 +897,9 @@ find_strongest_dependency(StatisticExtInfo *stats, MVDependencies *dependencies,
* using functional dependency statistics, or 1.0 if no useful functional
* dependency statistic exists.
*
- * 'estimatedclauses' is an output argument that gets a bit set corresponding
- * to the (zero-based) list index of each clause that is included in the
- * estimated selectivity.
+ * 'estimatedclauses' is an input/output argument that gets a bit set
+ * corresponding to the (zero-based) list index of each clause that is included
+ * in the estimated selectivity.
*
* Given equality clauses on attributes (a,b) we find the strongest dependency
* between them, i.e. either (a=>b) or (b=>a). Assuming (a=>b) is the selected
@@ -955,9 +934,6 @@ dependencies_clauselist_selectivity(PlannerInfo *root,
AttrNumber *list_attnums;
int listidx;
- /* initialize output argument */
- *estimatedclauses = NULL;
-
/* check if there's any stats that might be useful for us. */
if (!has_stats_of_kind(rel->statlist, STATS_EXT_DEPENDENCIES))
return 1.0;
@@ -972,6 +948,9 @@ dependencies_clauselist_selectivity(PlannerInfo *root,
* the attnums for each clause in a list which we'll reference later so we
* don't need to repeat the same work again. We'll also keep track of all
* attnums seen.
+ *
+ * We also skip clauses that we already estimated using different types of
+ * statistics (we treat them as incompatible).
*/
listidx = 0;
foreach(l, clauses)
@@ -979,7 +958,8 @@ dependencies_clauselist_selectivity(PlannerInfo *root,
Node *clause = (Node *) lfirst(l);
AttrNumber attnum;
- if (dependency_is_compatible_clause(clause, rel->relid, &attnum))
+ if ((dependency_is_compatible_clause(clause, rel->relid, &attnum)) &&
+ (!bms_is_member(listidx, *estimatedclauses)))
{
list_attnums[listidx] = attnum;
clauses_attnums = bms_add_member(clauses_attnums, attnum);
@@ -1049,8 +1029,7 @@ dependencies_clauselist_selectivity(PlannerInfo *root,
/*
* Skip incompatible clauses, and ones we've already estimated on.
*/
- if (list_attnums[listidx] == InvalidAttrNumber ||
- bms_is_member(listidx, *estimatedclauses))
+ if (list_attnums[listidx] == InvalidAttrNumber)
continue;
/*
diff --git a/src/backend/statistics/extended_stats.c b/src/backend/statistics/extended_stats.c
index 082f0506da..1628daae83 100644
--- a/src/backend/statistics/extended_stats.c
+++ b/src/backend/statistics/extended_stats.c
@@ -16,6 +16,8 @@
*/
#include "postgres.h"
+#include <math.h>
+
#include "access/genam.h"
#include "access/heapam.h"
#include "access/htup_details.h"
@@ -23,6 +25,8 @@
#include "catalog/pg_collation.h"
#include "catalog/pg_statistic_ext.h"
#include "nodes/relation.h"
+#include "optimizer/clauses.h"
+#include "optimizer/cost.h"
#include "postmaster/autovacuum.h"
#include "statistics/extended_stats_internal.h"
#include "statistics/statistics.h"
@@ -31,6 +35,7 @@
#include "utils/lsyscache.h"
#include "utils/memutils.h"
#include "utils/rel.h"
+#include "utils/selfuncs.h"
#include "utils/syscache.h"
@@ -53,7 +58,7 @@ static VacAttrStats **lookup_var_attr_stats(Relation rel, Bitmapset *attrs,
int nvacatts, VacAttrStats **vacatts);
static void statext_store(Relation pg_stext, Oid relid,
MVNDistinct *ndistinct, MVDependencies *dependencies,
- VacAttrStats **stats);
+ MCVList * mcvlist, VacAttrStats **stats);
/*
@@ -87,6 +92,7 @@ BuildRelationExtStatistics(Relation onerel, double totalrows,
StatExtEntry *stat = (StatExtEntry *) lfirst(lc);
MVNDistinct *ndistinct = NULL;
MVDependencies *dependencies = NULL;
+ MCVList *mcv = NULL;
VacAttrStats **stats;
ListCell *lc2;
@@ -124,10 +130,13 @@ BuildRelationExtStatistics(Relation onerel, double totalrows,
else if (t == STATS_EXT_DEPENDENCIES)
dependencies = statext_dependencies_build(numrows, rows,
stat->columns, stats);
+ else if (t == STATS_EXT_MCV)
+ mcv = statext_mcv_build(numrows, rows, stat->columns, stats,
+ totalrows);
}
/* store the statistics in the catalog */
- statext_store(pg_stext, stat->statOid, ndistinct, dependencies, stats);
+ statext_store(pg_stext, stat->statOid, ndistinct, dependencies, mcv, stats);
}
heap_close(pg_stext, RowExclusiveLock);
@@ -155,6 +164,10 @@ statext_is_kind_built(HeapTuple htup, char type)
attnum = Anum_pg_statistic_ext_stxdependencies;
break;
+ case STATS_EXT_MCV:
+ attnum = Anum_pg_statistic_ext_stxmcv;
+ break;
+
default:
elog(ERROR, "unexpected statistics type requested: %d", type);
}
@@ -219,7 +232,8 @@ fetch_statentries_for_relation(Relation pg_statext, Oid relid)
for (i = 0; i < ARR_DIMS(arr)[0]; i++)
{
Assert((enabled[i] == STATS_EXT_NDISTINCT) ||
- (enabled[i] == STATS_EXT_DEPENDENCIES));
+ (enabled[i] == STATS_EXT_DEPENDENCIES) ||
+ (enabled[i] == STATS_EXT_MCV));
entry->types = lappend_int(entry->types, (int) enabled[i]);
}
@@ -294,7 +308,7 @@ lookup_var_attr_stats(Relation rel, Bitmapset *attrs,
static void
statext_store(Relation pg_stext, Oid statOid,
MVNDistinct *ndistinct, MVDependencies *dependencies,
- VacAttrStats **stats)
+ MCVList * mcv, VacAttrStats **stats)
{
HeapTuple stup,
oldtup;
@@ -325,9 +339,18 @@ statext_store(Relation pg_stext, Oid statOid,
values[Anum_pg_statistic_ext_stxdependencies - 1] = PointerGetDatum(data);
}
+ if (mcv != NULL)
+ {
+ bytea *data = statext_mcv_serialize(mcv, stats);
+
+ nulls[Anum_pg_statistic_ext_stxmcv - 1] = (data == NULL);
+ values[Anum_pg_statistic_ext_stxmcv - 1] = PointerGetDatum(data);
+ }
+
/* always replace the value (either by bytea or NULL) */
replaces[Anum_pg_statistic_ext_stxndistinct - 1] = true;
replaces[Anum_pg_statistic_ext_stxdependencies - 1] = true;
+ replaces[Anum_pg_statistic_ext_stxmcv - 1] = true;
/* there should already be a pg_statistic_ext tuple */
oldtup = SearchSysCache1(STATEXTOID, ObjectIdGetDatum(statOid));
@@ -434,6 +457,137 @@ multi_sort_compare_dims(int start, int end,
return 0;
}
+int
+compare_scalars_simple(const void *a, const void *b, void *arg)
+{
+ return compare_datums_simple(*(Datum *) a,
+ *(Datum *) b,
+ (SortSupport) arg);
+}
+
+int
+compare_datums_simple(Datum a, Datum b, SortSupport ssup)
+{
+ return ApplySortComparator(a, false, b, false, ssup);
+}
+
+/* simple counterpart to qsort_arg */
+void *
+bsearch_arg(const void *key, const void *base, size_t nmemb, size_t size,
+ int (*compar) (const void *, const void *, void *),
+ void *arg)
+{
+ size_t l,
+ u,
+ idx;
+ const void *p;
+ int comparison;
+
+ l = 0;
+ u = nmemb;
+ while (l < u)
+ {
+ idx = (l + u) / 2;
+ p = (void *) (((const char *) base) + (idx * size));
+ comparison = (*compar) (key, p, arg);
+
+ if (comparison < 0)
+ u = idx;
+ else if (comparison > 0)
+ l = idx + 1;
+ else
+ return (void *) p;
+ }
+
+ return NULL;
+}
+
+int *
+build_attnums(Bitmapset *attrs)
+{
+ int i,
+ j;
+ int numattrs = bms_num_members(attrs);
+ int *attnums;
+
+ /* build attnums from the bitmapset */
+ attnums = (int *) palloc(sizeof(int) * numattrs);
+ i = 0;
+ j = -1;
+ while ((j = bms_next_member(attrs, j)) >= 0)
+ attnums[i++] = j;
+
+ return attnums;
+}
+
+/* build_sorted_items
+ * build sorted array of SortItem with values from rows
+ *
+ * XXX All the memory is allocated in a single chunk, so that the caller
+ * can simply pfree the return value to release all of it.
+ */
+SortItem *
+build_sorted_items(int numrows, HeapTuple *rows, TupleDesc tdesc,
+ MultiSortSupport mss, int numattrs, int *attnums)
+{
+ int i,
+ j,
+ len;
+ int nvalues = numrows * numattrs;
+
+ /*
+ * We won't allocate the arrays for each item independenly, but in one
+ * large chunk and then just set the pointers. This allows the caller to
+ * simply pfree the return value to release all the memory.
+ */
+ SortItem *items;
+ Datum *values;
+ bool *isnull;
+ char *ptr;
+
+ /* Compute the total amount of memory we need (both items and values). */
+ len = numrows * sizeof(SortItem) + nvalues * (sizeof(Datum) + sizeof(bool));
+
+ /* Allocate the memory and split it into the pieces. */
+ ptr = palloc0(len);
+
+ /* items to sort */
+ items = (SortItem *) ptr;
+ ptr += numrows * sizeof(SortItem);
+
+ /* values and null flags */
+ values = (Datum *) ptr;
+ ptr += nvalues * sizeof(Datum);
+
+ isnull = (bool *) ptr;
+ ptr += nvalues * sizeof(bool);
+
+ /* make sure we consumed the whole buffer exactly */
+ Assert((ptr - (char *) items) == len);
+
+ /* fix the pointers to Datum and bool arrays */
+ for (i = 0; i < numrows; i++)
+ {
+ items[i].values = &values[i * numattrs];
+ items[i].isnull = &isnull[i * numattrs];
+
+ /* load the values/null flags from sample rows */
+ for (j = 0; j < numattrs; j++)
+ {
+ items[i].values[j] = heap_getattr(rows[i],
+ attnums[j], /* attnum */
+ tdesc,
+ &items[i].isnull[j]); /* isnull */
+ }
+ }
+
+ /* do the sort, using the multi-sort */
+ qsort_arg((void *) items, numrows, sizeof(SortItem),
+ multi_sort_compare, mss);
+
+ return items;
+}
+
/*
* has_stats_of_kind
* Check whether the list contains statistic of a given kind
@@ -464,7 +618,7 @@ has_stats_of_kind(List *stats, char requiredkind)
* object referencing the most of the requested attributes, breaking ties
* in favor of objects with fewer keys overall.
*
- * XXX if multiple statistics objects tie on both criteria, then which object
+ * XXX If multiple statistics objects tie on both criteria, then which object
* is chosen depends on the order that they appear in the stats list. Perhaps
* further tiebreakers are needed.
*/
@@ -514,3 +668,382 @@ choose_best_statistics(List *stats, Bitmapset *attnums, char requiredkind)
return best_match;
}
+
+int
+bms_member_index(Bitmapset *keys, AttrNumber varattno)
+{
+ int i,
+ j;
+
+ i = -1;
+ j = 0;
+ while (((i = bms_next_member(keys, i)) >= 0) && (i < varattno))
+ j += 1;
+
+ return j;
+}
+
+/* The Duj1 estimator (already used in analyze.c). */
+double
+estimate_ndistinct(double totalrows, int numrows, int d, int f1)
+{
+ double numer,
+ denom,
+ ndistinct;
+
+ numer = (double) numrows * (double) d;
+
+ denom = (double) (numrows - f1) +
+ (double) f1 * (double) numrows / totalrows;
+
+ ndistinct = numer / denom;
+
+ /* Clamp to sane range in case of roundoff error */
+ if (ndistinct < (double) d)
+ ndistinct = (double) d;
+
+ if (ndistinct > totalrows)
+ ndistinct = totalrows;
+
+ return floor(ndistinct + 0.5);
+}
+
+/*
+ * statext_is_compatible_clause_internal
+ * Does the heavy lifting of actually inspecting the clauses for
+ * statext_is_compatible_clause.
+ */
+static bool
+statext_is_compatible_clause_internal(Node *clause, Index relid, Bitmapset **attnums)
+{
+ /* We only support plain Vars for now */
+ if (IsA(clause, Var))
+ {
+ Var *var = (Var *) clause;
+
+ /* Ensure var is from the correct relation */
+ if (var->varno != relid)
+ return false;
+
+ /* we also better ensure the Var is from the current level */
+ if (var->varlevelsup > 0)
+ return false;
+
+ /* Also skip system attributes (we don't allow stats on those). */
+ if (!AttrNumberIsForUserDefinedAttr(var->varattno))
+ return false;
+
+ *attnums = bms_add_member(*attnums, var->varattno);
+
+ return true;
+ }
+
+ /* Var = Const */
+ if (is_opclause(clause))
+ {
+ OpExpr *expr = (OpExpr *) clause;
+ Var *var;
+ bool varonleft = true;
+ bool ok;
+
+ /* Only expressions with two arguments are considered compatible. */
+ if (list_length(expr->args) != 2)
+ return false;
+
+ /* see if it actually has the right */
+ ok = (NumRelids((Node *) expr) == 1) &&
+ (is_pseudo_constant_clause(lsecond(expr->args)) ||
+ (varonleft = false,
+ is_pseudo_constant_clause(linitial(expr->args))));
+
+ /* unsupported structure (two variables or so) */
+ if (!ok)
+ return false;
+
+ /*
+ * If it's not one of the supported operators ("=", "<", ">", etc.),
+ * just ignore the clause, as it's not compatible with MCV lists.
+ *
+ * This uses the function for estimating selectivity, not the operator
+ * directly (a bit awkward, but well ...).
+ */
+ if ((get_oprrest(expr->opno) != F_EQSEL) &&
+ (get_oprrest(expr->opno) != F_NEQSEL) &&
+ (get_oprrest(expr->opno) != F_SCALARLTSEL) &&
+ (get_oprrest(expr->opno) != F_SCALARLESEL) &&
+ (get_oprrest(expr->opno) != F_SCALARGTSEL) &&
+ (get_oprrest(expr->opno) != F_SCALARGESEL))
+ return false;
+
+ var = (varonleft) ? linitial(expr->args) : lsecond(expr->args);
+
+ return statext_is_compatible_clause_internal((Node *) var, relid, attnums);
+ }
+
+ /* NOT/AND/OR clause */
+ if (or_clause(clause) ||
+ and_clause(clause) ||
+ not_clause(clause))
+ {
+ /*
+ * AND/OR/NOT-clauses are supported if all sub-clauses are supported
+ *
+ * Perhaps we could improve this by handling mixed cases, when some of
+ * the clauses are supported and some are not. Selectivity for the
+ * supported subclauses would be computed using extended statistics,
+ * and the remaining clauses would be estimated using the traditional
+ * algorithm (product of selectivities).
+ *
+ * It however seems overly complex, and in a way we already do that
+ * because if we reject the whole clause as unsupported here, it will
+ * be eventually passed to clauselist_selectivity() which does exactly
+ * this (split into supported/unsupported clauses etc).
+ */
+ BoolExpr *expr = (BoolExpr *) clause;
+ ListCell *lc;
+ Bitmapset *clause_attnums = NULL;
+
+ foreach(lc, expr->args)
+ {
+ /*
+ * Had we found incompatible clause in the arguments, treat the
+ * whole clause as incompatible.
+ */
+ if (!statext_is_compatible_clause_internal((Node *) lfirst(lc),
+ relid, &clause_attnums))
+ return false;
+ }
+
+ /*
+ * Otherwise the clause is compatible, and we need to merge the
+ * attnums into the main bitmapset.
+ */
+ *attnums = bms_join(*attnums, clause_attnums);
+
+ return true;
+ }
+
+ /* Var IS NULL */
+ if (IsA(clause, NullTest))
+ {
+ NullTest *nt = (NullTest *) clause;
+
+ /*
+ * Only simple (Var IS NULL) expressions supported for now. Maybe we
+ * could use examine_variable to fix this?
+ */
+ if (!IsA(nt->arg, Var))
+ return false;
+
+ return statext_is_compatible_clause_internal((Node *) (nt->arg), relid, attnums);
+ }
+
+ return false;
+}
+
+/*
+ * statext_is_compatible_clause
+ * Determines if the clause is compatible with MCV lists.
+ *
+ * Only OpExprs with two arguments using an equality operator are supported.
+ * When returning True attnum is set to the attribute number of the Var within
+ * the supported clause.
+ *
+ * Currently we only support Var = Const, or Const = Var. It may be possible
+ * to expand on this later.
+ */
+static bool
+statext_is_compatible_clause(Node *clause, Index relid, Bitmapset **attnums)
+{
+ RestrictInfo *rinfo = (RestrictInfo *) clause;
+
+ if (!IsA(rinfo, RestrictInfo))
+ return false;
+
+ /* Pseudoconstants are not really interesting here. */
+ if (rinfo->pseudoconstant)
+ return false;
+
+ /* clauses referencing multiple varnos are incompatible */
+ if (bms_membership(rinfo->clause_relids) != BMS_SINGLETON)
+ return false;
+
+ return statext_is_compatible_clause_internal((Node *) rinfo->clause,
+ relid, attnums);
+}
+
+/*
+ * statext_clauselist_selectivity
+ * Estimate clauses using the best multi-column statistics.
+ *
+ * Selects the best extended (multi-column) statistic on a table (measured by
+ * a number of attributes extracted from the clauses and covered by it), and
+ * computes the selectivity for supplied clauses.
+ *
+ * One of the main challenges with using MCV lists is how to extrapolate the
+ * estimate to the data not covered by the MCV list. To do that, we compute
+ * not only the "MCV selectivity" (selectivities for MCV items matching the
+ * supplied clauses), but also a couple of derived selectivities:
+ *
+ * - simple selectivity: Computed without extended statistic, i.e. as if the
+ * columns/clauses were independent
+ *
+ * - base selectivity: Similar to simple selectivity, but is computed using
+ * the extended statistic by adding up the base frequencies (that we compute
+ * and store for each MCV item) of matching MCV items.
+ *
+ * - total selectivity: Selectivity covered by the whole MCV list.
+ *
+ * - other selectivity: A selectivity estimate for data not covered by the MCV
+ * list (i.e. satisfying the clauses, but not common enough to make it into
+ * the MCV list)
+ *
+ * Note: While simple and base selectivities are defined in a quite similar
+ * way, the values are computed differently and are not therefore equal. The
+ * simple selectivity is computed as a product of per-clause estimates, while
+ * the base selectivity is computed by adding up base frequencies of matching
+ * items of the multi-column MCV list. So the values may differ for two main
+ * reasons - (a) the MCV list may not cover 100% of the data and (b) some of
+ * the MCV items did not match the estimated clauses.
+ *
+ * As both (a) and (b) reduce the base selectivity value, it generally holds
+ * that (simple_selectivity >= base_selectivity). If the MCV list covers all
+ * the data, the values may be equal.
+ *
+ * So (simple_selectivity - base_selectivity) may be seen as a correction for
+ * the part not covered by the MCV list.
+ *
+ * Note: Due to rounding errors and minor differences in how the estimates
+ * are computed, the inequality may not always hold. Which is why we clamp
+ * the selectivities to prevent strange estimate (negative etc.).
+ *
+ * XXX If we were to use multiple statistics, this is where it would happen.
+ * We would simply repeat this on a loop on the "remaining" clauses, possibly
+ * using the already estimated clauses as conditions (and combining the values
+ * using conditional probability formula).
+ */
+Selectivity
+statext_clauselist_selectivity(PlannerInfo *root, List *clauses, int varRelid,
+ JoinType jointype, SpecialJoinInfo *sjinfo,
+ RelOptInfo *rel, Bitmapset **estimatedclauses)
+{
+ ListCell *l;
+ Bitmapset *clauses_attnums = NULL;
+ Bitmapset **list_attnums;
+ int listidx;
+ StatisticExtInfo *stat;
+ List *stat_clauses;
+ Selectivity simple_sel,
+ mcv_sel,
+ mcv_basesel,
+ mcv_totalsel,
+ other_sel,
+ sel;
+
+ /* we're interested in MCV lists */
+ int types = STATS_EXT_MCV;
+
+ /* check if there's any stats that might be useful for us. */
+ if (!has_stats_of_kind(rel->statlist, types))
+ return (Selectivity) 1.0;
+
+ list_attnums = (Bitmapset **) palloc(sizeof(Bitmapset *) *
+ list_length(clauses));
+
+ /*
+ * Pre-process the clauses list to extract the attnums seen in each item.
+ * We need to determine if there's any clauses which will be useful for
+ * dependency selectivity estimations. Along the way we'll record all of
+ * the attnums for each clause in a list which we'll reference later so we
+ * don't need to repeat the same work again. We'll also keep track of all
+ * attnums seen.
+ *
+ * We also skip clauses that we already estimated using different types of
+ * statistics (we treat them as incompatible).
+ *
+ * XXX Currently, the estimated clauses are always empty because the extra
+ * statistics are applied before functional dependencies. Once we decide
+ * to apply multiple statistics, this may change.
+ */
+ listidx = 0;
+ foreach(l, clauses)
+ {
+ Node *clause = (Node *) lfirst(l);
+ Bitmapset *attnums = NULL;
+
+ if ((statext_is_compatible_clause(clause, rel->relid, &attnums)) &&
+ (!bms_is_member(listidx, *estimatedclauses)))
+ {
+ list_attnums[listidx] = attnums;
+ clauses_attnums = bms_add_members(clauses_attnums, attnums);
+ }
+ else
+ list_attnums[listidx] = NULL;
+
+ listidx++;
+ }
+
+ /* We need at least two attributes for MCV lists. */
+ if (bms_num_members(clauses_attnums) < 2)
+ return 1.0;
+
+ /* find the best suited statistics object for these attnums */
+ stat = choose_best_statistics(rel->statlist, clauses_attnums, types);
+
+ /* if no matching stats could be found then we've nothing to do */
+ if (!stat)
+ return (Selectivity) 1.0;
+
+ /* We only understand MCV lists for now. */
+ Assert(stat->kind == STATS_EXT_MCV);
+
+ /* now filter the clauses to be estimated using the selected MCV */
+ stat_clauses = NIL;
+
+ listidx = 0;
+ foreach(l, clauses)
+ {
+ /*
+ * If the clause is compatible with the selected statistics, mark it
+ * as estimated and add it to the list to estimate.
+ */
+ if ((list_attnums[listidx] != NULL) &&
+ (bms_is_subset(list_attnums[listidx], stat->keys)))
+ {
+ stat_clauses = lappend(stat_clauses, (Node *) lfirst(l));
+ *estimatedclauses = bms_add_member(*estimatedclauses, listidx);
+ }
+
+ listidx++;
+ }
+
+ /*
+ * First compute "simple" selectivity, i.e. without the extended statistics,
+ * and essentially assuming independence of the columns/clauses. We'll then
+ * use the various selectivities computed from MCV list to improve it.
+ */
+ simple_sel = clauselist_selectivity_simple(root, stat_clauses, varRelid,
+ jointype, sjinfo, NULL);
+
+ /*
+ * Now compute the multi-column estimate from the MCV list, along with the
+ * other selectivities (base & total selectivity).
+ */
+ mcv_sel = mcv_clauselist_selectivity(root, stat, stat_clauses, varRelid,
+ jointype, sjinfo, rel,
+ &mcv_basesel, &mcv_totalsel);
+
+ /* Estimated selectivity of values not covered by MCV matches */
+ other_sel = simple_sel - mcv_basesel;
+ CLAMP_PROBABILITY(other_sel);
+
+ /* The non-MCV selectivity can't exceed the 1 - mcv_totalsel. */
+ if (other_sel > 1.0 - mcv_totalsel)
+ other_sel = 1.0 - mcv_totalsel;
+
+ /* Overall selectivity is the combination of MCV and non-MCV estimates. */
+ sel = mcv_sel + other_sel;
+ CLAMP_PROBABILITY(sel);
+
+ return sel;
+}
diff --git a/src/backend/statistics/mcv.c b/src/backend/statistics/mcv.c
new file mode 100644
index 0000000000..9bc2d07e90
--- /dev/null
+++ b/src/backend/statistics/mcv.c
@@ -0,0 +1,1656 @@
+/*-------------------------------------------------------------------------
+ *
+ * mcv.c
+ * POSTGRES multivariate MCV lists
+ *
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/statistics/mcv.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/htup_details.h"
+#include "catalog/pg_collation.h"
+#include "catalog/pg_statistic_ext.h"
+#include "fmgr.h"
+#include "funcapi.h"
+#include "optimizer/clauses.h"
+#include "statistics/extended_stats_internal.h"
+#include "statistics/statistics.h"
+#include "utils/builtins.h"
+#include "utils/bytea.h"
+#include "utils/fmgroids.h"
+#include "utils/fmgrprotos.h"
+#include "utils/lsyscache.h"
+#include "utils/syscache.h"
+#include "utils/typcache.h"
+
+#include <math.h>
+
+/*
+ * Computes size of a serialized MCV item, depending on the number of
+ * dimensions (columns) the statistic is defined on. The datum values are
+ * stored in a separate array (deduplicated, to minimize the size), and
+ * so the serialized items only store uint16 indexes into that array.
+ *
+ * Each serialized item store (in this order):
+ *
+ * - indexes to values (ndim * sizeof(uint16))
+ * - null flags (ndim * sizeof(bool))
+ * - frequency (sizeof(double))
+ * - base_frequency (sizeof(double))
+ *
+ * So in total each MCV item requires this many bytes:
+ *
+ * ndim * (sizeof(uint16) + sizeof(bool)) + 2 * sizeof(double)
+ */
+#define ITEM_SIZE(ndims) \
+ (ndims * (sizeof(uint16) + sizeof(bool)) + 2 * sizeof(double))
+
+/*
+ * Macros for convenient access to parts of a serialized MCV item.
+ */
+#define ITEM_INDEXES(item) ((uint16*)item)
+#define ITEM_NULLS(item,ndims) ((bool*)(ITEM_INDEXES(item) + ndims))
+#define ITEM_FREQUENCY(item,ndims) ((double*)(ITEM_NULLS(item,ndims) + ndims))
+#define ITEM_BASE_FREQUENCY(item,ndims) ((double*)(ITEM_FREQUENCY(item,ndims) + 1))
+
+
+static MultiSortSupport build_mss(VacAttrStats **stats, Bitmapset *attrs);
+
+static SortItem *build_distinct_groups(int numrows, SortItem *items,
+ MultiSortSupport mss, int *ndistinct);
+
+static int count_distinct_groups(int numrows, SortItem *items,
+ MultiSortSupport mss);
+
+/*
+ * Builds MCV list from the set of sampled rows.
+ *
+ * The algorithm is quite simple:
+ *
+ * (1) sort the data (default collation, '<' for the data type)
+ *
+ * (2) count distinct groups, decide how many to keep
+ *
+ * (3) build the MCV list using the threshold determined in (2)
+ *
+ * (4) remove rows represented by the MCV from the sample
+ *
+ */
+MCVList *
+statext_mcv_build(int numrows, HeapTuple *rows, Bitmapset *attrs,
+ VacAttrStats **stats, double totalrows)
+{
+ int i,
+ j,
+ k;
+ int numattrs = bms_num_members(attrs);
+ int ngroups;
+ int nitems;
+ double stadistinct;
+ int *mcv_counts;
+ int f1;
+
+ int *attnums = build_attnums(attrs);
+
+ MCVList *mcvlist = NULL;
+
+ /* comparator for all the columns */
+ MultiSortSupport mss = build_mss(stats, attrs);
+
+ /* sort the rows */
+ SortItem *items = build_sorted_items(numrows, rows, stats[0]->tupDesc,
+ mss, numattrs, attnums);
+
+ /* transform the sorted rows into groups (sorted by frequency) */
+ SortItem *groups = build_distinct_groups(numrows, items, mss, &ngroups);
+
+ /*
+ * Maximum number of MCV items to store, based on the attribute with the
+ * largest stats target (and the number of groups we have available).
+ */
+ nitems = stats[0]->attr->attstattarget;
+ for (i = 1; i < numattrs; i++)
+ {
+ if (stats[i]->attr->attstattarget > nitems)
+ nitems = stats[i]->attr->attstattarget;
+ }
+ if (nitems > ngroups)
+ nitems = ngroups;
+
+ /*
+ * Decide how many items to keep in the MCV list. We simply use the same
+ * algorithm as for per-column MCV lists, to keep it consistent.
+ *
+ * One difference is that we do not have a multi-column stanullfrac, and
+ * we simply treat it as a special item in the MCV list (it it makes it).
+ * We could compute and store it, of course, but we may have statistics
+ * on more than two columns, so we'd probably want to store this for
+ * various combinations of columns - for K columns that'd be 2^K values.
+ * So we instead store those as items of the multi-column MCV list (if
+ * common enough).
+ *
+ * XXX Conceptually this is similar to the NULL-buckets of histograms.
+ */
+ mcv_counts = (int *) palloc(sizeof(int) * nitems);
+ f1 = 0;
+
+ for (i = 0; i < nitems; i++)
+ {
+ mcv_counts[i] = groups[i].count;
+
+ /* count values that occur exactly once for the ndistinct estimate */
+ if (groups[i].count == 1)
+ f1 += 1;
+ }
+
+ stadistinct = estimate_ndistinct(totalrows, numrows, ngroups, f1);
+
+ /*
+ * If we can fit all the items onto the MCV list, do that. Otherwise use
+ * analyze_mcv_list to decide how many items to keep in the MCV list, just
+ * like for the single-dimensional MCV list.
+ *
+ * XXX Should we also consider stadistinct here, to see if the groups do
+ * represent all the distinct combinations.
+ */
+ if (ngroups > nitems)
+ {
+ nitems = analyze_mcv_list(mcv_counts, nitems, stadistinct,
+ 0.0, /* stanullfrac */
+ numrows, totalrows);
+ }
+
+ /*
+ * At this point we know the number of items for the MCV list. There might
+ * be none (for uniform distribution with many groups), and in that case
+ * there will be no MCV list. Otherwise construct the MCV list.
+ */
+ if (nitems > 0)
+ {
+ /*
+ * Allocate the MCV list structure, set the global parameters.
+ */
+ mcvlist = (MCVList *) palloc0(sizeof(MCVList));
+
+ mcvlist->magic = STATS_MCV_MAGIC;
+ mcvlist->type = STATS_MCV_TYPE_BASIC;
+ mcvlist->ndimensions = numattrs;
+ mcvlist->nitems = nitems;
+
+ /* store info about data type OIDs */
+ i = 0;
+ j = -1;
+ while ((j = bms_next_member(attrs, j)) >= 0)
+ {
+ VacAttrStats *colstat = stats[i];
+
+ mcvlist->types[i] = colstat->attrtypid;
+ i++;
+ }
+
+ /*
+ * Preallocate Datum/isnull arrays (not as a single chunk, as we will
+ * pass the result outside and thus it needs to be easy to pfree().
+ *
+ * XXX On second thought, we're the only ones dealing with MCV lists,
+ * so we might allocate everything as a single chunk to reduce palloc
+ * overhead (chunk headers, etc.) without significant risk. Not sure
+ * it's worth it, though, as we're not re-building stats very often.
+ */
+ mcvlist->items = (MCVItem * *) palloc0(sizeof(MCVItem *) * nitems);
+
+ for (i = 0; i < nitems; i++)
+ {
+ mcvlist->items[i] = (MCVItem *) palloc(sizeof(MCVItem));
+ mcvlist->items[i]->values = (Datum *) palloc(sizeof(Datum) * numattrs);
+ mcvlist->items[i]->isnull = (bool *) palloc(sizeof(bool) * numattrs);
+ }
+
+ /* Copy the first chunk of groups into the result. */
+ for (i = 0; i < nitems; i++)
+ {
+ /* just pointer to the proper place in the list */
+ MCVItem *item = mcvlist->items[i];
+
+ /* copy values from the _previous_ group (last item of) */
+ memcpy(item->values, groups[i].values, sizeof(Datum) * numattrs);
+ memcpy(item->isnull, groups[i].isnull, sizeof(bool) * numattrs);
+
+ /* groups should be sorted by frequency in descending order */
+ Assert((i == 0) || (groups[i - 1].count >= groups[i].count));
+
+ /* group frequency */
+ item->frequency = (double) groups[i].count / numrows;
+
+ /* base frequency, if the attributes were independent */
+ item->base_frequency = 1.0;
+ for (j = 0; j < numattrs; j++)
+ {
+ int count = 0;
+
+ for (k = 0; k < ngroups; k++)
+ {
+ if (multi_sort_compare_dim(j, &groups[i], &groups[k], mss) == 0)
+ count += groups[k].count;
+ }
+
+ item->base_frequency *= (double) count / numrows;
+ }
+ }
+ }
+
+ pfree(items);
+ pfree(groups);
+ pfree(mcv_counts);
+
+ return mcvlist;
+}
+
+/*
+ * build_mss
+ * build MultiSortSupport for the attributes passed in attrs
+ */
+static MultiSortSupport
+build_mss(VacAttrStats **stats, Bitmapset *attrs)
+{
+ int i,
+ j;
+ int numattrs = bms_num_members(attrs);
+
+ /* Sort by multiple columns (using array of SortSupport) */
+ MultiSortSupport mss = multi_sort_init(numattrs);
+
+ /* prepare the sort functions for all the attributes */
+ i = 0;
+ j = -1;
+ while ((j = bms_next_member(attrs, j)) >= 0)
+ {
+ VacAttrStats *colstat = stats[i];
+ TypeCacheEntry *type;
+
+ type = lookup_type_cache(colstat->attrtypid, TYPECACHE_LT_OPR);
+ if (type->lt_opr == InvalidOid) /* shouldn't happen */
+ elog(ERROR, "cache lookup failed for ordering operator for type %u",
+ colstat->attrtypid);
+
+ multi_sort_add_dimension(mss, i, type->lt_opr, type->typcollation);
+ i++;
+ }
+
+ return mss;
+}
+
+/*
+ * count_distinct_groups
+ * count distinct combinations of SortItems in the array
+ *
+ * The array is assumed to be sorted according to the MultiSortSupport.
+ */
+static int
+count_distinct_groups(int numrows, SortItem *items, MultiSortSupport mss)
+{
+ int i;
+ int ndistinct;
+
+ ndistinct = 1;
+ for (i = 1; i < numrows; i++)
+ {
+ /* make sure the array really is sorted */
+ Assert(multi_sort_compare(&items[i], &items[i - 1], mss) >= 0);
+
+ if (multi_sort_compare(&items[i], &items[i - 1], mss) != 0)
+ ndistinct += 1;
+ }
+
+ return ndistinct;
+}
+
+/*
+ * compare_sort_item_count
+ * comparator for sorting items by count (frequencies) in descending order
+ */
+static int
+compare_sort_item_count(const void *a, const void *b)
+{
+ SortItem *ia = (SortItem *) a;
+ SortItem *ib = (SortItem *) b;
+
+ if (ia->count == ib->count)
+ return 0;
+ else if (ia->count > ib->count)
+ return -1;
+
+ return 1;
+}
+
+/*
+ * build_distinct_groups
+ * build array of SortItems for distinct groups and counts matching items
+ *
+ * The input array is assumed to be sorted
+ */
+static SortItem *
+build_distinct_groups(int numrows, SortItem *items, MultiSortSupport mss,
+ int *ndistinct)
+{
+ int i,
+ j;
+ int ngroups = count_distinct_groups(numrows, items, mss);
+
+ SortItem *groups = (SortItem *) palloc0(ngroups * sizeof(SortItem));
+
+ j = 0;
+ groups[0] = items[0];
+ groups[0].count = 1;
+
+ for (i = 1; i < numrows; i++)
+ {
+ /* Assume sorted in ascending order. */
+ Assert(multi_sort_compare(&items[i], &items[i - 1], mss) >= 0);
+
+ /* New distinct group detected. */
+ if (multi_sort_compare(&items[i], &items[i - 1], mss) != 0)
+ groups[++j] = items[i];
+
+ groups[j].count++;
+ }
+
+ /* Sort the distinct groups by frequency (in descending order). */
+ pg_qsort((void *) groups, ngroups, sizeof(SortItem),
+ compare_sort_item_count);
+
+ *ndistinct = ngroups;
+ return groups;
+}
+
+
+/*
+ * statext_mcv_load
+ * Load the MCV list for the indicated pg_statistic_ext tuple
+ */
+MCVList *
+statext_mcv_load(Oid mvoid)
+{
+ bool isnull = false;
+ Datum mcvlist;
+ HeapTuple htup = SearchSysCache1(STATEXTOID, ObjectIdGetDatum(mvoid));
+
+ if (!HeapTupleIsValid(htup))
+ elog(ERROR, "cache lookup failed for statistics object %u", mvoid);
+
+ mcvlist = SysCacheGetAttr(STATEXTOID, htup,
+ Anum_pg_statistic_ext_stxmcv, &isnull);
+
+ ReleaseSysCache(htup);
+
+ if (isnull)
+ return NULL;
+
+ return statext_mcv_deserialize(DatumGetByteaP(mcvlist));
+}
+
+
+/*
+ * Serialize MCV list into a bytea value.
+ *
+ * The basic algorithm is simple:
+ *
+ * (1) perform deduplication (for each attribute separately)
+ * (a) collect all (non-NULL) attribute values from all MCV items
+ * (b) sort the data (using 'lt' from VacAttrStats)
+ * (c) remove duplicate values from the array
+ *
+ * (2) serialize the arrays into a bytea value
+ *
+ * (3) process all MCV list items
+ * (a) replace values with indexes into the arrays
+ *
+ * Each attribute has to be processed separately, as we may be mixing different
+ * datatypes, with different sort operators, etc.
+ *
+ * We use uint16 values for the indexes in step (3), as we currently don't allow
+ * more than 8k MCV items anyway, although that's mostly arbitrary limit. We might
+ * increase this to 65k and still fit into uint16. Furthermore, this limit is on
+ * the number of distinct values per column, and we usually have few of those
+ * (and various combinations of them for the those MCV list). So uint16 seems fine.
+ *
+ * We don't really expect the serialization to save as much space as for
+ * histograms, as we are not doing any bucket splits (which is the source
+ * of high redundancy in histograms).
+ *
+ * TODO: Consider packing boolean flags (NULL) for each item into a single char
+ * (or a longer type) instead of using an array of bool items.
+ */
+bytea *
+statext_mcv_serialize(MCVList * mcvlist, VacAttrStats **stats)
+{
+ int i;
+ int dim;
+ int ndims = mcvlist->ndimensions;
+ int itemsize = ITEM_SIZE(ndims);
+
+ SortSupport ssup;
+ DimensionInfo *info;
+
+ Size total_length;
+
+ /* allocate the item just once */
+ char *item = palloc0(itemsize);
+
+ /* serialized items (indexes into arrays, etc.) */
+ bytea *output;
+ char *data = NULL;
+
+ /* values per dimension (and number of non-NULL values) */
+ Datum **values = (Datum **) palloc0(sizeof(Datum *) * ndims);
+ int *counts = (int *) palloc0(sizeof(int) * ndims);
+
+ /*
+ * We'll include some rudimentary information about the attributes (type
+ * length, etc.), so that we don't have to look them up while
+ * deserializing the MCV list.
+ *
+ * XXX Maybe this is not a great idea? Or maybe we should actually copy
+ * more fields, e.g. typeid, which would allow us to display the MCV list
+ * using only the serialized representation (currently we have to fetch
+ * this info from the relation).
+ */
+ info = (DimensionInfo *) palloc0(sizeof(DimensionInfo) * ndims);
+
+ /* sort support data for all attributes included in the MCV list */
+ ssup = (SortSupport) palloc0(sizeof(SortSupportData) * ndims);
+
+ /* collect and deduplicate values for each dimension (attribute) */
+ for (dim = 0; dim < ndims; dim++)
+ {
+ int ndistinct;
+ TypeCacheEntry *typentry;
+
+ /*
+ * Lookup the LT operator (can't get it from stats extra_data, as we
+ * don't know how to interpret that - scalar vs. array etc.).
+ */
+ typentry = lookup_type_cache(stats[dim]->attrtypid, TYPECACHE_LT_OPR);
+
+ /* copy important info about the data type (length, by-value) */
+ info[dim].typlen = stats[dim]->attrtype->typlen;
+ info[dim].typbyval = stats[dim]->attrtype->typbyval;
+
+ /* allocate space for values in the attribute and collect them */
+ values[dim] = (Datum *) palloc0(sizeof(Datum) * mcvlist->nitems);
+
+ for (i = 0; i < mcvlist->nitems; i++)
+ {
+ /* skip NULL values - we don't need to deduplicate those */
+ if (mcvlist->items[i]->isnull[dim])
+ continue;
+
+ values[dim][counts[dim]] = mcvlist->items[i]->values[dim];
+ counts[dim] += 1;
+ }
+
+ /* if there are just NULL values in this dimension, we're done */
+ if (counts[dim] == 0)
+ continue;
+
+ /* sort and deduplicate the data */
+ ssup[dim].ssup_cxt = CurrentMemoryContext;
+ ssup[dim].ssup_collation = DEFAULT_COLLATION_OID;
+ ssup[dim].ssup_nulls_first = false;
+
+ PrepareSortSupportFromOrderingOp(typentry->lt_opr, &ssup[dim]);
+
+ qsort_arg(values[dim], counts[dim], sizeof(Datum),
+ compare_scalars_simple, &ssup[dim]);
+
+ /*
+ * Walk through the array and eliminate duplicate values, but keep the
+ * ordering (so that we can do bsearch later). We know there's at
+ * least one item as (counts[dim] != 0), so we can skip the first
+ * element.
+ */
+ ndistinct = 1; /* number of distinct values */
+ for (i = 1; i < counts[dim]; i++)
+ {
+ /* expect sorted array */
+ Assert(compare_datums_simple(values[dim][i - 1], values[dim][i], &ssup[dim]) <= 0);
+
+ /* if the value is the same as the previous one, we can skip it */
+ if (!compare_datums_simple(values[dim][i - 1], values[dim][i], &ssup[dim]))
+ continue;
+
+ values[dim][ndistinct] = values[dim][i];
+ ndistinct += 1;
+ }
+
+ /* we must not exceed UINT16_MAX, as we use uint16 indexes */
+ Assert(ndistinct <= UINT16_MAX);
+
+ /*
+ * Store additional info about the attribute - number of deduplicated
+ * values, and also size of the serialized data. For fixed-length data
+ * types this is trivial to compute, for varwidth types we need to
+ * actually walk the array and sum the sizes.
+ */
+ info[dim].nvalues = ndistinct;
+
+ if (info[dim].typlen > 0) /* fixed-length data types */
+ info[dim].nbytes = info[dim].nvalues * info[dim].typlen;
+ else if (info[dim].typlen == -1) /* varlena */
+ {
+ info[dim].nbytes = 0;
+ for (i = 0; i < info[dim].nvalues; i++)
+ info[dim].nbytes += VARSIZE_ANY(values[dim][i]);
+ }
+ else if (info[dim].typlen == -2) /* cstring */
+ {
+ info[dim].nbytes = 0;
+ for (i = 0; i < info[dim].nvalues; i++)
+ info[dim].nbytes += strlen(DatumGetPointer(values[dim][i]));
+ }
+
+ /* we know (count>0) so there must be some data */
+ Assert(info[dim].nbytes > 0);
+ }
+
+ /*
+ * Now we can finally compute how much space we'll actually need for the
+ * whole serialized MCV list, as it contains these fields:
+ *
+ * - length (4B) for varlena - magic (4B) - type (4B) - ndimensions (4B) -
+ * nitems (4B) - info (ndim * sizeof(DimensionInfo) - arrays of values for
+ * each dimension - serialized items (nitems * itemsize)
+ *
+ * So the 'header' size is 20B + ndim * sizeof(DimensionInfo) and then we
+ * will place all the data (values + indexes). We'll however use offsetof
+ * and sizeof to compute sizes of the structs.
+ */
+ total_length = (sizeof(int32) + offsetof(MCVList, items)
+ + (ndims * sizeof(DimensionInfo))
+ + mcvlist->nitems * itemsize);
+
+ /* add space for the arrays of deduplicated values */
+ for (i = 0; i < ndims; i++)
+ total_length += info[i].nbytes;
+
+ /*
+ * Enforce arbitrary limit of 1MB on the size of the serialized MCV list.
+ * This is meant as a protection against someone building MCV list on long
+ * values (e.g. text documents).
+ *
+ * XXX Should we enforce arbitrary limits like this one? Maybe it's not
+ * even necessary, as long values are usually unique and so won't make it
+ * into the MCV list in the first place. In the end, we have a 1GB limit
+ * on bytea values.
+ */
+ if (total_length > (1024 * 1024))
+ elog(ERROR, "serialized MCV list exceeds 1MB (%ld)", total_length);
+
+ /* allocate space for the serialized MCV list, set header fields */
+ output = (bytea *) palloc0(total_length);
+ SET_VARSIZE(output, total_length);
+
+ /* 'data' points to the current position in the output buffer */
+ data = VARDATA(output);
+
+ /* MCV list header (number of items, ...) */
+ memcpy(data, mcvlist, offsetof(MCVList, items));
+ data += offsetof(MCVList, items);
+
+ /* information about the attributes */
+ memcpy(data, info, sizeof(DimensionInfo) * ndims);
+ data += sizeof(DimensionInfo) * ndims;
+
+ /* Copy the deduplicated values for all attributes to the output. */
+ for (dim = 0; dim < ndims; dim++)
+ {
+#ifdef USE_ASSERT_CHECKING
+ /* remember the starting point for Asserts later */
+ char *tmp = data;
+#endif
+ for (i = 0; i < info[dim].nvalues; i++)
+ {
+ Datum v = values[dim][i];
+
+ if (info[dim].typbyval) /* passed by value */
+ {
+ memcpy(data, &v, info[dim].typlen);
+ data += info[dim].typlen;
+ }
+ else if (info[dim].typlen > 0) /* pased by reference */
+ {
+ memcpy(data, DatumGetPointer(v), info[dim].typlen);
+ data += info[dim].typlen;
+ }
+ else if (info[dim].typlen == -1) /* varlena */
+ {
+ memcpy(data, DatumGetPointer(v), VARSIZE_ANY(v));
+ data += VARSIZE_ANY(v);
+ }
+ else if (info[dim].typlen == -2) /* cstring */
+ {
+ memcpy(data, DatumGetPointer(v), strlen(DatumGetPointer(v)) + 1);
+ data += strlen(DatumGetPointer(v)) + 1; /* terminator */
+ }
+
+ /* no underflows or overflows */
+ Assert((data > tmp) && ((data - tmp) <= info[dim].nbytes));
+ }
+
+ /*
+ * check we got exactly the amount of data we expected for this
+ * dimension
+ */
+ Assert((data - tmp) == info[dim].nbytes);
+ }
+
+ /* Serialize the items, with uint16 indexes instead of the values. */
+ for (i = 0; i < mcvlist->nitems; i++)
+ {
+ MCVItem *mcvitem = mcvlist->items[i];
+
+ /* don't write beyond the allocated space */
+ Assert(data <= (char *) output + total_length - itemsize);
+
+ /* reset the item (we only allocate it once and reuse it) */
+ memset(item, 0, itemsize);
+
+ for (dim = 0; dim < ndims; dim++)
+ {
+ Datum *v = NULL;
+
+ /* do the lookup only for non-NULL values */
+ if (mcvlist->items[i]->isnull[dim])
+ continue;
+
+ v = (Datum *) bsearch_arg(&mcvitem->values[dim], values[dim],
+ info[dim].nvalues, sizeof(Datum),
+ compare_scalars_simple, &ssup[dim]);
+
+ Assert(v != NULL); /* serialization or deduplication error */
+
+ /* compute index within the array */
+ ITEM_INDEXES(item)[dim] = (v - values[dim]);
+
+ /* check the index is within expected bounds */
+ Assert(ITEM_INDEXES(item)[dim] >= 0);
+ Assert(ITEM_INDEXES(item)[dim] < info[dim].nvalues);
+ }
+
+ /* copy NULL and frequency flags into the item */
+ memcpy(ITEM_NULLS(item, ndims), mcvitem->isnull, sizeof(bool) * ndims);
+ memcpy(ITEM_FREQUENCY(item, ndims), &mcvitem->frequency, sizeof(double));
+ memcpy(ITEM_BASE_FREQUENCY(item, ndims), &mcvitem->base_frequency, sizeof(double));
+
+ /* copy the serialized item into the array */
+ memcpy(data, item, itemsize);
+
+ data += itemsize;
+ }
+
+ /* at this point we expect to match the total_length exactly */
+ Assert((data - (char *) output) == total_length);
+
+ pfree(item);
+ pfree(values);
+ pfree(counts);
+
+ return output;
+}
+
+/*
+ * Reads serialized MCV list into MCVList structure.
+ *
+ * Unlike with histograms, we deserialize the MCV list fully (i.e. we don't
+ * keep the deduplicated arrays and pointers into them), as we don't expect
+ * there to be a lot of duplicate values. But perhaps that's not true and we
+ * should keep the MCV in serialized form too.
+ *
+ * XXX See how much memory we could save by keeping the deduplicated version
+ * (both for typical and corner cases with few distinct values but many items).
+ */
+MCVList *
+statext_mcv_deserialize(bytea *data)
+{
+ int dim,
+ i;
+ Size expected_size;
+ MCVList *mcvlist;
+ char *tmp;
+
+ int ndims,
+ nitems,
+ itemsize;
+ DimensionInfo *info = NULL;
+ Datum **values = NULL;
+
+ /* local allocation buffer (used only for deserialization) */
+ int bufflen;
+ char *buff;
+ char *ptr;
+
+ /* buffer used for the result */
+ int rbufflen;
+ char *rbuff;
+ char *rptr;
+
+ if (data == NULL)
+ return NULL;
+
+ /*
+ * We can't possibly deserialize a MCV list if there's not even a complete
+ * header.
+ */
+ if (VARSIZE_ANY_EXHDR(data) < offsetof(MCVList, items))
+ elog(ERROR, "invalid MCV Size %ld (expected at least %ld)",
+ VARSIZE_ANY_EXHDR(data), offsetof(MCVList, items));
+
+ /* read the MCV list header */
+ mcvlist = (MCVList *) palloc0(sizeof(MCVList));
+
+ /* initialize pointer to the data part (skip the varlena header) */
+ tmp = VARDATA_ANY(data);
+
+ /* get the header and perform further sanity checks */
+ memcpy(mcvlist, tmp, offsetof(MCVList, items));
+ tmp += offsetof(MCVList, items);
+
+ if (mcvlist->magic != STATS_MCV_MAGIC)
+ elog(ERROR, "invalid MCV magic %d (expected %dd)",
+ mcvlist->magic, STATS_MCV_MAGIC);
+
+ if (mcvlist->type != STATS_MCV_TYPE_BASIC)
+ elog(ERROR, "invalid MCV type %d (expected %dd)",
+ mcvlist->type, STATS_MCV_TYPE_BASIC);
+
+ if (mcvlist->ndimensions == 0)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("invalid zero-length dimension array in MCVList")));
+ else if (mcvlist->ndimensions > STATS_MAX_DIMENSIONS)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("invalid length (%d) dimension array in MCVList",
+ mcvlist->ndimensions)));
+
+ if (mcvlist->nitems == 0)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("invalid zero-length item array in MCVList")));
+ else if (mcvlist->nitems > STATS_MCVLIST_MAX_ITEMS)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("invalid length (%d) item array in MCVList",
+ mcvlist->nitems)));
+
+ nitems = mcvlist->nitems;
+ ndims = mcvlist->ndimensions;
+ itemsize = ITEM_SIZE(ndims);
+
+ /*
+ * Check amount of data including DimensionInfo for all dimensions and
+ * also the serialized items (including uint16 indexes). Also, walk
+ * through the dimension information and add it to the sum.
+ */
+ expected_size = offsetof(MCVList, items) +
+ ndims * sizeof(DimensionInfo) +
+ (nitems * itemsize);
+
+ /*
+ * Check that we have at least the dimension and info records, along with
+ * the items. We don't know the size of the serialized values yet. We need
+ * to do this check first, before accessing the dimension info.
+ */
+ if (VARSIZE_ANY_EXHDR(data) < expected_size)
+ elog(ERROR, "invalid MCV size %ld (expected %ld)",
+ VARSIZE_ANY_EXHDR(data), expected_size);
+
+ /* Now it's safe to access the dimension info. */
+ info = (DimensionInfo *) (tmp);
+ tmp += ndims * sizeof(DimensionInfo);
+
+ /* account for the value arrays */
+ for (dim = 0; dim < ndims; dim++)
+ {
+ /*
+ * XXX I wonder if we can/should rely on asserts here. Maybe those
+ * checks should be done every time?
+ */
+ Assert(info[dim].nvalues >= 0);
+ Assert(info[dim].nbytes >= 0);
+
+ expected_size += info[dim].nbytes;
+ }
+
+ /*
+ * Now we know the total expected MCV size, including all the pieces
+ * (header, dimension info. items and deduplicated data). So do the final
+ * check on size.
+ */
+ if (VARSIZE_ANY_EXHDR(data) != expected_size)
+ elog(ERROR, "invalid MCV size %ld (expected %ld)",
+ VARSIZE_ANY_EXHDR(data), expected_size);
+
+ /*
+ * Allocate one large chunk of memory for the intermediate data, needed
+ * only for deserializing the MCV list (and allocate densely to minimize
+ * the palloc overhead).
+ *
+ * Let's see how much space we'll actually need, and also include space
+ * for the array with pointers.
+ *
+ * We need an array of Datum pointers values for each dimension, so that
+ * we can easily translate the uint16 indexes. We also need a top-level
+ * array of pointers to those per-dimension arrays.
+ *
+ * For byval types with size matching sizeof(Datum) we can reuse the
+ * serialized array directly.
+ */
+ bufflen = sizeof(Datum **) * ndims; /* space for top-level pointers */
+
+ for (dim = 0; dim < ndims; dim++)
+ {
+ /* for full-size byval types, we reuse the serialized value */
+ if (!(info[dim].typbyval && info[dim].typlen == sizeof(Datum)))
+ bufflen += (sizeof(Datum) * info[dim].nvalues);
+ }
+
+ buff = palloc0(bufflen);
+ ptr = buff;
+
+ values = (Datum **) buff;
+ ptr += (sizeof(Datum *) * ndims);
+
+ /*
+ * XXX This uses pointers to the original data array (the types not passed
+ * by value), so when someone frees the memory, e.g. by doing something
+ * like this:
+ *
+ * bytea * data = ... fetch the data from catalog ...
+ *
+ * MCVList mcvlist = deserialize_mcv_list(data);
+ *
+ * pfree(data);
+ *
+ * then 'mcvlist' references the freed memory. Should copy the pieces.
+ */
+ for (dim = 0; dim < ndims; dim++)
+ {
+#ifdef USE_ASSERT_CHECKING
+ /* remember where data for this dimension starts */
+ char *start = tmp;
+#endif
+ if (info[dim].typbyval)
+ {
+ /* passed by value / size matches Datum - just reuse the array */
+ if (info[dim].typlen == sizeof(Datum))
+ {
+ values[dim] = (Datum *) tmp;
+ tmp += info[dim].nbytes;
+
+ /* no overflow of input array */
+ Assert(tmp <= start + info[dim].nbytes);
+ }
+ else
+ {
+ values[dim] = (Datum *) ptr;
+ ptr += (sizeof(Datum) * info[dim].nvalues);
+
+ for (i = 0; i < info[dim].nvalues; i++)
+ {
+ /* just point into the array */
+ memcpy(&values[dim][i], tmp, info[dim].typlen);
+ tmp += info[dim].typlen;
+
+ /* no overflow of input array */
+ Assert(tmp <= start + info[dim].nbytes);
+ }
+ }
+ }
+ else
+ {
+ /* all the other types need a chunk of the buffer */
+ values[dim] = (Datum *) ptr;
+ ptr += (sizeof(Datum) * info[dim].nvalues);
+
+ /* passed by reference, but fixed length (name, tid, ...) */
+ if (info[dim].typlen > 0)
+ {
+ for (i = 0; i < info[dim].nvalues; i++)
+ {
+ /* just point into the array */
+ values[dim][i] = PointerGetDatum(tmp);
+ tmp += info[dim].typlen;
+
+ /* no overflow of input array */
+ Assert(tmp <= start + info[dim].nbytes);
+ }
+ }
+ else if (info[dim].typlen == -1)
+ {
+ /* varlena */
+ for (i = 0; i < info[dim].nvalues; i++)
+ {
+ /* just point into the array */
+ values[dim][i] = PointerGetDatum(tmp);
+ tmp += VARSIZE_ANY(tmp);
+
+ /* no overflow of input array */
+ Assert(tmp <= start + info[dim].nbytes);
+ }
+ }
+ else if (info[dim].typlen == -2)
+ {
+ /* cstring */
+ for (i = 0; i < info[dim].nvalues; i++)
+ {
+ /* just point into the array */
+ values[dim][i] = PointerGetDatum(tmp);
+ tmp += (strlen(tmp) + 1); /* don't forget the \0 */
+
+ /* no overflow of input array */
+ Assert(tmp <= start + info[dim].nbytes);
+ }
+ }
+ }
+
+ /* check we consumed the serialized data for this dimension exactly */
+ Assert((tmp - start) == info[dim].nbytes);
+ }
+
+ /* we should have exhausted the buffer exactly */
+ Assert((ptr - buff) == bufflen);
+
+ /* allocate space for all the MCV items in a single piece */
+ rbufflen = (sizeof(MCVItem *) + sizeof(MCVItem) +
+ sizeof(Datum) * ndims + sizeof(bool) * ndims) * nitems;
+
+ rbuff = palloc0(rbufflen);
+ rptr = rbuff;
+
+ mcvlist->items = (MCVItem * *) rbuff;
+ rptr += (sizeof(MCVItem *) * nitems);
+
+ /* deserialize the MCV items and translate the indexes to Datums */
+ for (i = 0; i < nitems; i++)
+ {
+ uint16 *indexes = NULL;
+ MCVItem *item = (MCVItem *) rptr;
+
+ rptr += (sizeof(MCVItem));
+
+ item->values = (Datum *) rptr;
+ rptr += (sizeof(Datum) * ndims);
+
+ item->isnull = (bool *) rptr;
+ rptr += (sizeof(bool) * ndims);
+
+ /* just point to the right place */
+ indexes = ITEM_INDEXES(tmp);
+
+ memcpy(item->isnull, ITEM_NULLS(tmp, ndims), sizeof(bool) * ndims);
+ memcpy(&item->frequency, ITEM_FREQUENCY(tmp, ndims), sizeof(double));
+ memcpy(&item->base_frequency, ITEM_BASE_FREQUENCY(tmp, ndims), sizeof(double));
+
+ /* translate the values */
+ for (dim = 0; dim < ndims; dim++)
+ if (!item->isnull[dim])
+ item->values[dim] = values[dim][indexes[dim]];
+
+ mcvlist->items[i] = item;
+
+ tmp += ITEM_SIZE(ndims);
+
+ /* check we're not overflowing the input */
+ Assert(tmp <= (char *) data + VARSIZE_ANY(data));
+ }
+
+ /* check that we processed all the data */
+ Assert(tmp == (char *) data + VARSIZE_ANY(data));
+
+ /* release the temporary buffer */
+ pfree(buff);
+
+ return mcvlist;
+}
+
+/*
+ * SRF with details about buckets of a histogram:
+ *
+ * - item ID (0...nitems)
+ * - values (string array)
+ * - nulls only (boolean array)
+ * - frequency (double precision)
+ * - base_frequency (double precision)
+ *
+ * The input is the OID of the statistics, and there are no rows returned if
+ * the statistics contains no histogram.
+ */
+PG_FUNCTION_INFO_V1(pg_stats_ext_mcvlist_items);
+
+Datum
+pg_stats_ext_mcvlist_items(PG_FUNCTION_ARGS)
+{
+ FuncCallContext *funcctx;
+ int call_cntr;
+ int max_calls;
+ TupleDesc tupdesc;
+ AttInMetadata *attinmeta;
+
+ /* stuff done only on the first call of the function */
+ if (SRF_IS_FIRSTCALL())
+ {
+ MemoryContext oldcontext;
+ MCVList *mcvlist;
+
+ /* create a function context for cross-call persistence */
+ funcctx = SRF_FIRSTCALL_INIT();
+
+ /* switch to memory context appropriate for multiple function calls */
+ oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
+
+ mcvlist = statext_mcv_deserialize(PG_GETARG_BYTEA_P(0));
+
+ funcctx->user_fctx = mcvlist;
+
+ /* total number of tuples to be returned */
+ funcctx->max_calls = 0;
+ if (funcctx->user_fctx != NULL)
+ funcctx->max_calls = mcvlist->nitems;
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("function returning record called in context "
+ "that cannot accept type record")));
+
+ /* build metadata needed later to produce tuples from raw C-strings */
+ attinmeta = TupleDescGetAttInMetadata(tupdesc);
+ funcctx->attinmeta = attinmeta;
+
+ MemoryContextSwitchTo(oldcontext);
+ }
+
+ /* stuff done on every call of the function */
+ funcctx = SRF_PERCALL_SETUP();
+
+ call_cntr = funcctx->call_cntr;
+ max_calls = funcctx->max_calls;
+ attinmeta = funcctx->attinmeta;
+
+ if (call_cntr < max_calls) /* do when there is more left to send */
+ {
+ char **values;
+ HeapTuple tuple;
+ Datum result;
+
+ char *buff = palloc0(1024);
+ char *format;
+
+ int i;
+
+ Oid *outfuncs;
+ FmgrInfo *fmgrinfo;
+
+ MCVList *mcvlist;
+ MCVItem *item;
+
+ mcvlist = (MCVList *) funcctx->user_fctx;
+
+ Assert(call_cntr < mcvlist->nitems);
+
+ item = mcvlist->items[call_cntr];
+
+ /*
+ * Prepare a values array for building the returned tuple. This should
+ * be an array of C strings which will be processed later by the type
+ * input functions.
+ */
+ values = (char **) palloc(5 * sizeof(char *));
+
+ values[0] = (char *) palloc(64 * sizeof(char));
+
+ /* arrays */
+ values[1] = (char *) palloc0(1024 * sizeof(char));
+ values[2] = (char *) palloc0(1024 * sizeof(char));
+
+ /* frequency */
+ values[3] = (char *) palloc(64 * sizeof(char));
+
+ /* base frequency */
+ values[4] = (char *) palloc(64 * sizeof(char));
+
+ outfuncs = (Oid *) palloc0(sizeof(Oid) * mcvlist->ndimensions);
+ fmgrinfo = (FmgrInfo *) palloc0(sizeof(FmgrInfo) * mcvlist->ndimensions);
+
+ for (i = 0; i < mcvlist->ndimensions; i++)
+ {
+ bool isvarlena;
+
+ getTypeOutputInfo(mcvlist->types[i], &outfuncs[i], &isvarlena);
+
+ fmgr_info(outfuncs[i], &fmgrinfo[i]);
+ }
+
+ snprintf(values[0], 64, "%d", call_cntr); /* item ID */
+
+ for (i = 0; i < mcvlist->ndimensions; i++)
+ {
+ Datum val,
+ valout;
+
+ format = "%s, %s";
+ if (i == 0)
+ format = "{%s%s";
+ else if (i == mcvlist->ndimensions - 1)
+ format = "%s, %s}";
+
+ if (item->isnull[i])
+ valout = CStringGetDatum("NULL");
+ else
+ {
+ val = item->values[i];
+ valout = FunctionCall1(&fmgrinfo[i], val);
+ }
+
+ snprintf(buff, 1024, format, values[1], DatumGetPointer(valout));
+ strncpy(values[1], buff, 1023);
+ buff[0] = '\0';
+
+ snprintf(buff, 1024, format, values[2], item->isnull[i] ? "t" : "f");
+ strncpy(values[2], buff, 1023);
+ buff[0] = '\0';
+ }
+
+ snprintf(values[3], 64, "%f", item->frequency); /* frequency */
+ snprintf(values[4], 64, "%f", item->base_frequency); /* base frequency */
+
+ /* build a tuple */
+ tuple = BuildTupleFromCStrings(attinmeta, values);
+
+ /* make the tuple into a datum */
+ result = HeapTupleGetDatum(tuple);
+
+ /* clean up (this is not really necessary) */
+ pfree(values[0]);
+ pfree(values[1]);
+ pfree(values[2]);
+ pfree(values[3]);
+ pfree(values[4]);
+
+ pfree(values);
+
+ SRF_RETURN_NEXT(funcctx, result);
+ }
+ else /* do when there is no more left */
+ {
+ SRF_RETURN_DONE(funcctx);
+ }
+}
+
+/*
+ * pg_mcv_list_in - input routine for type pg_mcv_list.
+ *
+ * pg_mcv_list is real enough to be a table column, but it has no operations
+ * of its own, and disallows input too
+ */
+Datum
+pg_mcv_list_in(PG_FUNCTION_ARGS)
+{
+ /*
+ * pg_mcv_list stores the data in binary form and parsing text input is
+ * not needed, so disallow this.
+ */
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("cannot accept a value of type %s", "pg_mcv_list")));
+
+ PG_RETURN_VOID(); /* keep compiler quiet */
+}
+
+
+/*
+ * pg_mcv_list_out - output routine for type PG_MCV_LIST.
+ *
+ * MCV lists are serialized into a bytea value, so we simply call byteaout()
+ * to serialize the value into text. But it'd be nice to serialize that into
+ * a meaningful representation (e.g. for inspection by people).
+ *
+ * XXX This should probably return something meaningful, similar to what
+ * pg_dependencies_out does. Not sure how to deal with the deduplicated
+ * values, though - do we want to expand that or not?
+ */
+Datum
+pg_mcv_list_out(PG_FUNCTION_ARGS)
+{
+ return byteaout(fcinfo);
+}
+
+/*
+ * pg_mcv_list_recv - binary input routine for type pg_mcv_list.
+ */
+Datum
+pg_mcv_list_recv(PG_FUNCTION_ARGS)
+{
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("cannot accept a value of type %s", "pg_mcv_list")));
+
+ PG_RETURN_VOID(); /* keep compiler quiet */
+}
+
+/*
+ * pg_mcv_list_send - binary output routine for type pg_mcv_list.
+ *
+ * MCV lists are serialized in a bytea value (although the type is named
+ * differently), so let's just send that.
+ */
+Datum
+pg_mcv_list_send(PG_FUNCTION_ARGS)
+{
+ return byteasend(fcinfo);
+}
+
+/*
+ * mcv_update_match_bitmap
+ * Evaluate clauses using the MCV list, and update the match bitmap.
+ *
+ * A match bitmap keeps match/mismatch status for each MCV item, and we
+ * update it based on additional clauses. We also use it to skip items
+ * that can't possibly match (e.g. item marked as "mismatch" can't change
+ * to "match" when evaluating AND clause list).
+ *
+ * The function also returns a flag indicating whether there was an
+ * equality condition for all attributes, the minimum frequency in the MCV
+ * list, and a total MCV frequency (sum of frequencies for all items).
+ *
+ * XXX Currently the match bitmap uses a char for each MCV item, which is
+ * somewhat wasteful as we could do with just a single bit, thus reducing
+ * the size to ~1/8. It would also allow us to combine bitmaps simply using
+ * & and |, which should be faster than min/max. The bitmaps are fairly
+ * small, though (as we cap the MCV list size to 8k items).
+ */
+static void
+mcv_update_match_bitmap(PlannerInfo *root, List *clauses,
+ Bitmapset *keys, MCVList * mcvlist, char *matches,
+ bool is_or)
+{
+ int i;
+ ListCell *l;
+
+ /* The bitmap may be partially built. */
+ Assert(clauses != NIL);
+ Assert(list_length(clauses) >= 1);
+ Assert(mcvlist != NULL);
+ Assert(mcvlist->nitems > 0);
+ Assert(mcvlist->nitems <= STATS_MCVLIST_MAX_ITEMS);
+
+ /*
+ * Loop through the list of clauses, and for each of them evaluate all the
+ * MCV items not yet eliminated by the preceding clauses.
+ */
+ foreach(l, clauses)
+ {
+ Node *clause = (Node *) lfirst(l);
+
+ /* if it's a RestrictInfo, then extract the clause */
+ if (IsA(clause, RestrictInfo))
+ clause = (Node *) ((RestrictInfo *) clause)->clause;
+
+ /*
+ * Handle the various types of clauses - OpClause, NullTest and
+ * AND/OR/NOT
+ */
+ if (is_opclause(clause))
+ {
+ OpExpr *expr = (OpExpr *) clause;
+ bool varonleft = true;
+ bool ok;
+ FmgrInfo opproc;
+
+ /* get procedure computing operator selectivity */
+ RegProcedure oprrest = get_oprrest(expr->opno);
+
+ fmgr_info(get_opcode(expr->opno), &opproc);
+
+ ok = (NumRelids(clause) == 1) &&
+ (is_pseudo_constant_clause(lsecond(expr->args)) ||
+ (varonleft = false,
+ is_pseudo_constant_clause(linitial(expr->args))));
+
+ if (ok)
+ {
+
+ FmgrInfo gtproc;
+ Var *var = (varonleft) ? linitial(expr->args) : lsecond(expr->args);
+ Const *cst = (varonleft) ? lsecond(expr->args) : linitial(expr->args);
+ bool isgt = (!varonleft);
+
+ TypeCacheEntry *typecache
+ = lookup_type_cache(var->vartype, TYPECACHE_GT_OPR);
+
+ /* match the attribute to a dimension of the statistic */
+ int idx = bms_member_index(keys, var->varattno);
+
+ fmgr_info(get_opcode(typecache->gt_opr), >proc);
+
+ /*
+ * Walk through the MCV items and evaluate the current clause.
+ * We can skip items that were already ruled out, and
+ * terminate if there are no remaining MCV items that might
+ * possibly match.
+ */
+ for (i = 0; i < mcvlist->nitems; i++)
+ {
+ bool mismatch = false;
+ MCVItem *item = mcvlist->items[i];
+
+ /*
+ * For AND-lists, we can also mark NULL items as 'no
+ * match' (and then skip them). For OR-lists this is not
+ * possible.
+ */
+ if ((!is_or) && item->isnull[idx])
+ matches[i] = STATS_MATCH_NONE;
+
+ /* skip MCV items that were already ruled out */
+ if ((!is_or) && (matches[i] == STATS_MATCH_NONE))
+ continue;
+ else if (is_or && (matches[i] == STATS_MATCH_FULL))
+ continue;
+
+ switch (oprrest)
+ {
+ case F_EQSEL:
+ case F_NEQSEL:
+
+ /*
+ * We don't care about isgt in equality, because
+ * it does not matter whether it's (var op const)
+ * or (const op var).
+ */
+ mismatch = !DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ cst->constvalue,
+ item->values[idx]));
+
+ break;
+
+ case F_SCALARLTSEL: /* column < constant */
+ case F_SCALARLESEL: /* column <= constant */
+ case F_SCALARGTSEL: /* column > constant */
+ case F_SCALARGESEL: /* column >= constant */
+
+ /*
+ * First check whether the constant is below the
+ * lower boundary (in that case we can skip the
+ * bucket, because there's no overlap).
+ */
+ if (isgt)
+ mismatch = !DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ cst->constvalue,
+ item->values[idx]));
+ else
+ mismatch = !DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ item->values[idx],
+ cst->constvalue));
+
+ break;
+ }
+
+ /*
+ * XXX The conditions on matches[i] are not needed, as we
+ * skip MCV items that can't become true/false, depending
+ * on the current flag. See beginning of the loop over MCV
+ * items.
+ */
+
+ if ((is_or) && (!mismatch))
+ {
+ /* OR - was MATCH_NONE, but will be MATCH_FULL */
+ matches[i] = STATS_MATCH_FULL;
+ continue;
+ }
+ else if ((!is_or) && mismatch)
+ {
+ /* AND - was MATC_FULL, but will be MATCH_NONE */
+ matches[i] = STATS_MATCH_NONE;
+ continue;
+ }
+
+ }
+ }
+ }
+ else if (IsA(clause, NullTest))
+ {
+ NullTest *expr = (NullTest *) clause;
+ Var *var = (Var *) (expr->arg);
+
+ /* match the attribute to a dimension of the statistic */
+ int idx = bms_member_index(keys, var->varattno);
+
+ /*
+ * Walk through the MCV items and evaluate the current clause. We
+ * can skip items that were already ruled out, and terminate if
+ * there are no remaining MCV items that might possibly match.
+ */
+ for (i = 0; i < mcvlist->nitems; i++)
+ {
+ char match = STATS_MATCH_NONE; /* assume mismatch */
+ MCVItem *item = mcvlist->items[i];
+
+ /* if the clause mismatches the MCV item, set it as MATCH_NONE */
+ switch (expr->nulltesttype)
+ {
+ case IS_NULL:
+ match = (item->isnull[idx]) ? STATS_MATCH_FULL : match;
+ break;
+
+ case IS_NOT_NULL:
+ match = (!item->isnull[idx]) ? STATS_MATCH_FULL : match;
+ break;
+ }
+
+ /* now, update the match bitmap, depending on OR/AND type */
+ if (is_or)
+ matches[i] = Max(matches[i], match);
+ else
+ matches[i] = Min(matches[i], match);
+ }
+ }
+ else if (or_clause(clause) || and_clause(clause))
+ {
+ /* AND/OR clause, with all subclauses being compatible */
+
+ int i;
+ BoolExpr *bool_clause = ((BoolExpr *) clause);
+ List *bool_clauses = bool_clause->args;
+
+ /* match/mismatch bitmap for each MCV item */
+ char *bool_matches = NULL;
+
+ Assert(bool_clauses != NIL);
+ Assert(list_length(bool_clauses) >= 2);
+
+ /* by default none of the MCV items matches the clauses */
+ bool_matches = palloc0(sizeof(char) * mcvlist->nitems);
+
+ if (or_clause(clause))
+ {
+ /* OR clauses assume nothing matches, initially */
+ memset(bool_matches, STATS_MATCH_NONE, sizeof(char) * mcvlist->nitems);
+ }
+ else
+ {
+ /* AND clauses assume everything matches, initially */
+ memset(bool_matches, STATS_MATCH_FULL, sizeof(char) * mcvlist->nitems);
+ }
+
+ /* build the match bitmap for the OR-clauses */
+ mcv_update_match_bitmap(root, bool_clauses, keys,
+ mcvlist, bool_matches,
+ or_clause(clause));
+
+ /*
+ * Merge the bitmap produced by mcv_update_match_bitmap into the
+ * current one. We need to consider if we're evaluating AND or OR
+ * condition when merging the results.
+ */
+ for (i = 0; i < mcvlist->nitems; i++)
+ {
+ /* Is this OR or AND clause? */
+ if (is_or)
+ matches[i] = Max(matches[i], bool_matches[i]);
+ else
+ matches[i] = Min(matches[i], bool_matches[i]);
+ }
+
+ pfree(bool_matches);
+ }
+ else if (not_clause(clause))
+ {
+ /* NOT clause, with all subclauses compatible */
+
+ int i;
+ BoolExpr *not_clause = ((BoolExpr *) clause);
+ List *not_args = not_clause->args;
+
+ /* match/mismatch bitmap for each MCV item */
+ char *not_matches = NULL;
+
+ Assert(not_args != NIL);
+ Assert(list_length(not_args) == 1);
+
+ /* by default none of the MCV items matches the clauses */
+ not_matches = palloc0(sizeof(char) * mcvlist->nitems);
+
+ /* NOT clauses assume nothing matches, initially */
+ memset(not_matches, STATS_MATCH_FULL, sizeof(char) * mcvlist->nitems);
+
+ /* build the match bitmap for the NOT-clause */
+ mcv_update_match_bitmap(root, not_args, keys,
+ mcvlist, not_matches, false);
+
+ /*
+ * Merge the bitmap produced by mcv_update_match_bitmap into the
+ * current one.
+ */
+ for (i = 0; i < mcvlist->nitems; i++)
+ {
+ /*
+ * When handling a NOT clause, we need to invert the result
+ * before merging it into the global result.
+ */
+ if (not_matches[i] == STATS_MATCH_NONE)
+ not_matches[i] = STATS_MATCH_FULL;
+ else
+ not_matches[i] = STATS_MATCH_NONE;
+
+ /* Is this OR or AND clause? */
+ if (is_or)
+ matches[i] = Max(matches[i], not_matches[i]);
+ else
+ matches[i] = Min(matches[i], not_matches[i]);
+ }
+
+ pfree(not_matches);
+ }
+ else if (IsA(clause, Var))
+ {
+ /* Var (has to be a boolean Var, possibly from below NOT) */
+
+ Var *var = (Var *) (clause);
+
+ /* match the attribute to a dimension of the statistic */
+ int idx = bms_member_index(keys, var->varattno);
+
+ Assert(var->vartype == BOOLOID);
+
+ /*
+ * Walk through the MCV items and evaluate the current clause. We
+ * can skip items that were already ruled out, and terminate if
+ * there are no remaining MCV items that might possibly match.
+ */
+ for (i = 0; i < mcvlist->nitems; i++)
+ {
+ MCVItem *item = mcvlist->items[i];
+ bool match = STATS_MATCH_NONE;
+
+ /* if the item is NULL, it's a mismatch */
+ if (!item->isnull[idx] && DatumGetBool(item->values[idx]))
+ match = STATS_MATCH_FULL;
+
+ /* now, update the match bitmap, depending on OR/AND type */
+ if (is_or)
+ matches[i] = Max(matches[i], match);
+ else
+ matches[i] = Min(matches[i], match);
+ }
+ }
+ else
+ {
+ elog(ERROR, "unknown clause type: %d", clause->type);
+ }
+ }
+}
+
+
+/*
+ * mcv_clauselist_selectivity
+ * Return the selectivity estimate of clauses using MCV list.
+ *
+ * It also produces two interesting selectivities - total selectivity of
+ * all the MCV items combined, and selectivity of the least frequent item
+ * in the list.
+ */
+Selectivity
+mcv_clauselist_selectivity(PlannerInfo *root, StatisticExtInfo *stat,
+ List *clauses, int varRelid,
+ JoinType jointype, SpecialJoinInfo *sjinfo,
+ RelOptInfo *rel,
+ Selectivity *basesel, Selectivity *totalsel)
+{
+ int i;
+ MCVList *mcv;
+ Selectivity s = 0.0;
+
+ /* match/mismatch bitmap for each MCV item */
+ char *matches = NULL;
+
+ /* load the MCV list stored in the statistics object */
+ mcv = statext_mcv_load(stat->statOid);
+
+ /* by default all the MCV items match the clauses fully */
+ matches = palloc0(sizeof(char) * mcv->nitems);
+ memset(matches, STATS_MATCH_FULL, sizeof(char) * mcv->nitems);
+
+ mcv_update_match_bitmap(root, clauses, stat->keys, mcv,
+ matches, false);
+
+ /* sum frequencies for all the matching MCV items */
+ *basesel = 0.0;
+ *totalsel = 0.0;
+ for (i = 0; i < mcv->nitems; i++)
+ {
+ *totalsel += mcv->items[i]->frequency;
+
+ if (matches[i] != STATS_MATCH_NONE)
+ {
+ /* XXX Shouldn't the basesel be outside the if condition? */
+ *basesel += mcv->items[i]->base_frequency;
+ s += mcv->items[i]->frequency;
+ }
+ }
+
+ return s;
+}
diff --git a/src/backend/statistics/mvdistinct.c b/src/backend/statistics/mvdistinct.c
index 3071e42d86..5ef243b8a3 100644
--- a/src/backend/statistics/mvdistinct.c
+++ b/src/backend/statistics/mvdistinct.c
@@ -23,8 +23,6 @@
*/
#include "postgres.h"
-#include <math.h>
-
#include "access/htup_details.h"
#include "catalog/pg_statistic_ext.h"
#include "utils/fmgrprotos.h"
@@ -39,7 +37,6 @@
static double ndistinct_for_combination(double totalrows, int numrows,
HeapTuple *rows, VacAttrStats **stats,
int k, int *combination);
-static double estimate_ndistinct(double totalrows, int numrows, int d, int f1);
static int n_choose_k(int n, int k);
static int num_combinations(int n);
@@ -511,31 +508,6 @@ ndistinct_for_combination(double totalrows, int numrows, HeapTuple *rows,
return estimate_ndistinct(totalrows, numrows, d, f1);
}
-/* The Duj1 estimator (already used in analyze.c). */
-static double
-estimate_ndistinct(double totalrows, int numrows, int d, int f1)
-{
- double numer,
- denom,
- ndistinct;
-
- numer = (double) numrows * (double) d;
-
- denom = (double) (numrows - f1) +
- (double) f1 * (double) numrows / totalrows;
-
- ndistinct = numer / denom;
-
- /* Clamp to sane range in case of roundoff error */
- if (ndistinct < (double) d)
- ndistinct = (double) d;
-
- if (ndistinct > totalrows)
- ndistinct = totalrows;
-
- return floor(ndistinct + 0.5);
-}
-
/*
* n_choose_k
* computes binomial coefficients using an algorithm that is both
diff --git a/src/backend/utils/adt/ruleutils.c b/src/backend/utils/adt/ruleutils.c
index 4857caecaa..64edd874c9 100644
--- a/src/backend/utils/adt/ruleutils.c
+++ b/src/backend/utils/adt/ruleutils.c
@@ -1508,6 +1508,7 @@ pg_get_statisticsobj_worker(Oid statextid, bool missing_ok)
bool isnull;
bool ndistinct_enabled;
bool dependencies_enabled;
+ bool mcv_enabled;
int i;
statexttup = SearchSysCache1(STATEXTOID, ObjectIdGetDatum(statextid));
@@ -1543,6 +1544,7 @@ pg_get_statisticsobj_worker(Oid statextid, bool missing_ok)
ndistinct_enabled = false;
dependencies_enabled = false;
+ mcv_enabled = false;
for (i = 0; i < ARR_DIMS(arr)[0]; i++)
{
@@ -1550,6 +1552,8 @@ pg_get_statisticsobj_worker(Oid statextid, bool missing_ok)
ndistinct_enabled = true;
if (enabled[i] == STATS_EXT_DEPENDENCIES)
dependencies_enabled = true;
+ if (enabled[i] == STATS_EXT_MCV)
+ mcv_enabled = true;
}
/*
@@ -1559,13 +1563,27 @@ pg_get_statisticsobj_worker(Oid statextid, bool missing_ok)
* statistics types on a newer postgres version, if the statistics had all
* options enabled on the original version.
*/
- if (!ndistinct_enabled || !dependencies_enabled)
+ if (!ndistinct_enabled || !dependencies_enabled || !mcv_enabled)
{
+ bool gotone = false;
+
appendStringInfoString(&buf, " (");
+
if (ndistinct_enabled)
+ {
appendStringInfoString(&buf, "ndistinct");
- else if (dependencies_enabled)
- appendStringInfoString(&buf, "dependencies");
+ gotone = true;
+ }
+
+ if (dependencies_enabled)
+ {
+ appendStringInfo(&buf, "%sdependencies", gotone ? ", " : "");
+ gotone = true;
+ }
+
+ if (mcv_enabled)
+ appendStringInfo(&buf, "%smcv", gotone ? ", " : "");
+
appendStringInfoChar(&buf, ')');
}
diff --git a/src/backend/utils/adt/selfuncs.c b/src/backend/utils/adt/selfuncs.c
index 7155dc9087..e727808eae 100644
--- a/src/backend/utils/adt/selfuncs.c
+++ b/src/backend/utils/adt/selfuncs.c
@@ -3753,6 +3753,171 @@ estimate_num_groups(PlannerInfo *root, List *groupExprs, double input_rows,
return numdistinct;
}
+/*
+ * estimate_num_groups_simple
+ * Estimate number of groups in a relation.
+ *
+ * A simplified version of estimate_num_groups, assuming all expressions
+ * are only plain Vars from a single relation, and that no filtering is
+ * happenning.
+ */
+double
+estimate_num_groups_simple(PlannerInfo *root, List *vars)
+{
+ List *varinfos = NIL;
+ double numdistinct;
+ ListCell *l;
+
+ RelOptInfo *rel;
+ double reldistinct = 1;
+ double relmaxndistinct = reldistinct;
+ int relvarcount = 0;
+
+
+ /*
+ * If no grouping columns, there's exactly one group. (This can't happen
+ * for normal cases with GROUP BY or DISTINCT, but it is possible for
+ * corner cases with set operations.)
+ */
+ if (vars == NIL)
+ return 1.0;
+
+ /*
+ * We expect only variables from a single relation.
+ */
+ Assert(NumRelids((Node *) vars) == 1);
+
+ /*
+ * Find the unique Vars used, treating an expression as a Var if we can
+ * find stats for it. For each one, record the statistical estimate of
+ * number of distinct values (total in its table).
+ */
+ numdistinct = 1.0;
+
+ foreach(l, vars)
+ {
+ Var *var = (Var *) lfirst(l);
+ VariableStatData vardata;
+
+ Assert(IsA(var, Var));
+
+ /*
+ * If examine_variable is able to deduce anything about the GROUP BY
+ * expression, treat it as a single variable even if it's really more
+ * complicated.
+ */
+ examine_variable(root, (Node *) var, 0, &vardata);
+ if (HeapTupleIsValid(vardata.statsTuple) || vardata.isunique)
+ {
+ varinfos = add_unique_group_var(root, varinfos,
+ (Node *) var, &vardata);
+ ReleaseVariableStats(vardata);
+ continue;
+ }
+ ReleaseVariableStats(vardata);
+ }
+
+ Assert(varinfos);
+
+ /*
+ * Get the numdistinct estimate for the Vars of this rel.
+ *
+ * We
+ * iteratively search for multivariate n-distinct with maximum number
+ * of vars; assuming that each var group is independent of the others,
+ * we multiply them together. Any remaining relvarinfos after no more
+ * multivariate matches are found are assumed independent too, so
+ * their individual ndistinct estimates are multiplied also.
+ *
+ * While iterating, count how many separate numdistinct values we
+ * apply. We apply a fudge factor below, but only if we multiplied
+ * more than one such values.
+ */
+ while (varinfos)
+ {
+ double mvndistinct;
+
+ rel = ((GroupVarInfo *) linitial(varinfos))->rel;
+
+ if (estimate_multivariate_ndistinct(root, rel, &varinfos,
+ &mvndistinct))
+ {
+ reldistinct *= mvndistinct;
+ if (relmaxndistinct < mvndistinct)
+ relmaxndistinct = mvndistinct;
+ relvarcount++;
+ }
+ else
+ {
+ foreach(l, varinfos)
+ {
+ GroupVarInfo *varinfo = (GroupVarInfo *) lfirst(l);
+
+ reldistinct *= varinfo->ndistinct;
+ if (relmaxndistinct < varinfo->ndistinct)
+ relmaxndistinct = varinfo->ndistinct;
+ relvarcount++;
+ }
+
+ /* we're done with this relation */
+ varinfos = NIL;
+ }
+ }
+
+ /*
+ * Sanity check --- don't divide by zero if empty relation.
+ */
+ Assert(IS_SIMPLE_REL(rel));
+ if (rel->tuples > 0)
+ {
+ /*
+ * Clamp to size of rel, or size of rel / 10 if multiple Vars. The
+ * fudge factor is because the Vars are probably correlated but we
+ * don't know by how much. We should never clamp to less than the
+ * largest ndistinct value for any of the Vars, though, since
+ * there will surely be at least that many groups.
+ */
+ double clamp = rel->tuples;
+
+ if (relvarcount > 1)
+ {
+ clamp *= 0.1;
+ if (clamp < relmaxndistinct)
+ {
+ clamp = relmaxndistinct;
+ /* for sanity in case some ndistinct is too large: */
+ if (clamp > rel->tuples)
+ clamp = rel->tuples;
+ }
+ }
+ if (reldistinct > clamp)
+ reldistinct = clamp;
+
+ /*
+ * We're assuming we are returning all rows.
+ */
+ reldistinct = clamp_row_est(reldistinct);
+
+ /*
+ * Update estimate of total distinct groups.
+ */
+ numdistinct *= reldistinct;
+
+ /* Guard against out-of-range answers */
+ if (numdistinct > rel->tuples)
+ numdistinct = rel->tuples;
+ }
+
+ if (numdistinct < 1.0)
+ numdistinct = 1.0;
+
+ /* Round off */
+ numdistinct = ceil(numdistinct);
+
+ return numdistinct;
+
+}
+
/*
* Estimate hash bucket statistics when the specified expression is used
* as a hash key for the given number of buckets.
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 0a181b01d9..3d68a7c0ea 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -2542,7 +2542,8 @@ describeOneTableDetails(const char *schemaname,
" JOIN pg_catalog.pg_attribute a ON (stxrelid = a.attrelid AND\n"
" a.attnum = s.attnum AND NOT attisdropped)) AS columns,\n"
" 'd' = any(stxkind) AS ndist_enabled,\n"
- " 'f' = any(stxkind) AS deps_enabled\n"
+ " 'f' = any(stxkind) AS deps_enabled,\n"
+ " 'm' = any(stxkind) AS mcv_enabled\n"
"FROM pg_catalog.pg_statistic_ext stat "
"WHERE stxrelid = '%s'\n"
"ORDER BY 1;",
@@ -2579,6 +2580,12 @@ describeOneTableDetails(const char *schemaname,
if (strcmp(PQgetvalue(result, i, 6), "t") == 0)
{
appendPQExpBuffer(&buf, "%sdependencies", gotone ? ", " : "");
+ gotone = true;
+ }
+
+ if (strcmp(PQgetvalue(result, i, 7), "t") == 0)
+ {
+ appendPQExpBuffer(&buf, "%smcv", gotone ? ", " : "");
}
appendPQExpBuffer(&buf, ") ON %s FROM %s",
diff --git a/src/include/catalog/pg_cast.dat b/src/include/catalog/pg_cast.dat
index 8cd65b3ab5..b382bdce5a 100644
--- a/src/include/catalog/pg_cast.dat
+++ b/src/include/catalog/pg_cast.dat
@@ -324,6 +324,12 @@
{ castsource => 'pg_dependencies', casttarget => 'text', castfunc => '0',
castcontext => 'i', castmethod => 'i' },
+# pg_mcv_list can be coerced to, but not from, bytea and text
+{ castsource => 'pg_mcv_list', casttarget => 'bytea', castfunc => '0',
+ castcontext => 'i', castmethod => 'b' },
+{ castsource => 'pg_mcv_list', casttarget => 'text', castfunc => '0',
+ castcontext => 'i', castmethod => 'i' },
+
# Datetime category
{ castsource => 'date', casttarget => 'timestamp',
castfunc => 'timestamp(date)', castcontext => 'i', castmethod => 'f' },
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index acb0154048..c08dcc55ec 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -4962,6 +4962,30 @@
proname => 'pg_dependencies_send', provolatile => 's', prorettype => 'bytea',
proargtypes => 'pg_dependencies', prosrc => 'pg_dependencies_send' },
+{ oid => '4002', descr => 'I/O',
+ proname => 'pg_mcv_list_in', prorettype => 'pg_mcv_list',
+ proargtypes => 'cstring', prosrc => 'pg_mcv_list_in' },
+{ oid => '4003', descr => 'I/O',
+ proname => 'pg_mcv_list_out', prorettype => 'cstring',
+ proargtypes => 'pg_mcv_list', prosrc => 'pg_mcv_list_out' },
+{ oid => '4004', descr => 'I/O',
+ proname => 'pg_mcv_list_recv', provolatile => 's',
+ prorettype => 'pg_mcv_list', proargtypes => 'internal',
+ prosrc => 'pg_mcv_list_recv' },
+{ oid => '4005', descr => 'I/O',
+ proname => 'pg_mcv_list_send', provolatile => 's', prorettype => 'bytea',
+ proargtypes => 'pg_mcv_list', prosrc => 'pg_mcv_list_send' },
+
+{ oid => '3424',
+ descr => 'details about MCV list items',
+ proname => 'pg_mcv_list_items', prorows => '1000', proisstrict => 'f',
+ proretset => 't', provolatile => 's', prorettype => 'record',
+ proargtypes => 'pg_mcv_list',
+ proallargtypes => '{pg_mcv_list,int4,text,_bool,float8,float8}',
+ proargmodes => '{i,o,o,o,o,o}',
+ proargnames => '{mcv_list,index,values,nulls,frequency,base_frequency}',
+ prosrc => 'pg_stats_ext_mcvlist_items' },
+
{ oid => '1928', descr => 'statistics: number of scans done for table/index',
proname => 'pg_stat_get_numscans', provolatile => 's', proparallel => 'r',
prorettype => 'int8', proargtypes => 'oid',
diff --git a/src/include/catalog/pg_statistic_ext.h b/src/include/catalog/pg_statistic_ext.h
index e2e8d4f6d4..c4d3270d3f 100644
--- a/src/include/catalog/pg_statistic_ext.h
+++ b/src/include/catalog/pg_statistic_ext.h
@@ -49,6 +49,7 @@ CATALOG(pg_statistic_ext,3381,StatisticExtRelationId)
* to build */
pg_ndistinct stxndistinct; /* ndistinct coefficients (serialized) */
pg_dependencies stxdependencies; /* dependencies (serialized) */
+ pg_mcv_list stxmcv; /* MCV (serialized) */
#endif
} FormData_pg_statistic_ext;
@@ -64,6 +65,7 @@ typedef FormData_pg_statistic_ext *Form_pg_statistic_ext;
#define STATS_EXT_NDISTINCT 'd'
#define STATS_EXT_DEPENDENCIES 'f'
+#define STATS_EXT_MCV 'm'
#endif /* EXPOSE_TO_CLIENT_CODE */
diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat
index 7bca400ec2..b87f6bc4d7 100644
--- a/src/include/catalog/pg_type.dat
+++ b/src/include/catalog/pg_type.dat
@@ -165,6 +165,13 @@
typoutput => 'pg_dependencies_out', typreceive => 'pg_dependencies_recv',
typsend => 'pg_dependencies_send', typalign => 'i', typstorage => 'x',
typcollation => '100' },
+{ oid => '4001', oid_symbol => 'PGMCVLISTOID',
+ descr => 'multivariate MCV list',
+ typname => 'pg_mcv_list', typlen => '-1', typbyval => 'f',
+ typcategory => 'S', typinput => 'pg_mcv_list_in',
+ typoutput => 'pg_mcv_list_out', typreceive => 'pg_mcv_list_recv',
+ typsend => 'pg_mcv_list_send', typalign => 'i', typstorage => 'x',
+ typcollation => '100' },
{ oid => '32', oid_symbol => 'PGDDLCOMMANDOID',
descr => 'internal type for passing CollectedCommand',
typname => 'pg_ddl_command', typlen => 'SIZEOF_POINTER', typbyval => 't',
diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index dfff23ac55..ac05684652 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -205,6 +205,12 @@ extern void analyze_rel(Oid relid, RangeVar *relation, int options,
VacuumParams *params, List *va_cols, bool in_outer_xact,
BufferAccessStrategy bstrategy);
extern bool std_typanalyze(VacAttrStats *stats);
+extern int analyze_mcv_list(int *mcv_counts,
+ int num_mcv,
+ double stadistinct,
+ double stanullfrac,
+ int samplerows,
+ double totalrows);
/* in utils/misc/sampling.c --- duplicate of declarations in utils/sampling.h */
extern double anl_random_fract(void);
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 77ca7ff837..e6cded0597 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -215,6 +215,12 @@ extern Selectivity clause_selectivity(PlannerInfo *root,
int varRelid,
JoinType jointype,
SpecialJoinInfo *sjinfo);
+extern Selectivity clauselist_selectivity_simple(PlannerInfo *root,
+ List *clauses,
+ int varRelid,
+ JoinType jointype,
+ SpecialJoinInfo *sjinfo,
+ Bitmapset *estimatedclauses);
extern void cost_gather_merge(GatherMergePath *path, PlannerInfo *root,
RelOptInfo *rel, ParamPathInfo *param_info,
Cost input_startup_cost, Cost input_total_cost,
diff --git a/src/include/statistics/extended_stats_internal.h b/src/include/statistics/extended_stats_internal.h
index fff6bc6799..f330f3c1d3 100644
--- a/src/include/statistics/extended_stats_internal.h
+++ b/src/include/statistics/extended_stats_internal.h
@@ -31,6 +31,15 @@ typedef struct
int tupno; /* position index for tuple it came from */
} ScalarItem;
+/* (de)serialization info */
+typedef struct DimensionInfo
+{
+ int nvalues; /* number of deduplicated values */
+ int nbytes; /* number of bytes (serialized) */
+ int typlen; /* pg_type.typlen */
+ bool typbyval; /* pg_type.typbyval */
+} DimensionInfo;
+
/* multi-sort */
typedef struct MultiSortSupportData
{
@@ -44,6 +53,7 @@ typedef struct SortItem
{
Datum *values;
bool *isnull;
+ int count;
} SortItem;
extern MVNDistinct *statext_ndistinct_build(double totalrows,
@@ -57,6 +67,12 @@ extern MVDependencies *statext_dependencies_build(int numrows, HeapTuple *rows,
extern bytea *statext_dependencies_serialize(MVDependencies *dependencies);
extern MVDependencies *statext_dependencies_deserialize(bytea *data);
+extern MCVList * statext_mcv_build(int numrows, HeapTuple *rows,
+ Bitmapset *attrs, VacAttrStats **stats,
+ double totalrows);
+extern bytea *statext_mcv_serialize(MCVList * mcv, VacAttrStats **stats);
+extern MCVList * statext_mcv_deserialize(bytea *data);
+
extern MultiSortSupport multi_sort_init(int ndims);
extern void multi_sort_add_dimension(MultiSortSupport mss, int sortdim,
Oid oper, Oid collation);
@@ -65,5 +81,32 @@ extern int multi_sort_compare_dim(int dim, const SortItem *a,
const SortItem *b, MultiSortSupport mss);
extern int multi_sort_compare_dims(int start, int end, const SortItem *a,
const SortItem *b, MultiSortSupport mss);
+extern int compare_scalars_simple(const void *a, const void *b, void *arg);
+extern int compare_datums_simple(Datum a, Datum b, SortSupport ssup);
+
+extern void *bsearch_arg(const void *key, const void *base,
+ size_t nmemb, size_t size,
+ int (*compar) (const void *, const void *, void *),
+ void *arg);
+
+extern int *build_attnums(Bitmapset *attrs);
+
+extern SortItem *build_sorted_items(int numrows, HeapTuple *rows,
+ TupleDesc tdesc, MultiSortSupport mss,
+ int numattrs, int *attnums);
+
+extern int bms_member_index(Bitmapset *keys, AttrNumber varattno);
+
+extern double estimate_ndistinct(double totalrows, int numrows, int d, int f1);
+
+extern Selectivity mcv_clauselist_selectivity(PlannerInfo *root,
+ StatisticExtInfo *stat,
+ List *clauses,
+ int varRelid,
+ JoinType jointype,
+ SpecialJoinInfo *sjinfo,
+ RelOptInfo *rel,
+ Selectivity *basesel,
+ Selectivity *totalsel);
#endif /* EXTENDED_STATS_INTERNAL_H */
diff --git a/src/include/statistics/statistics.h b/src/include/statistics/statistics.h
index 8009fee322..e69d6a0232 100644
--- a/src/include/statistics/statistics.h
+++ b/src/include/statistics/statistics.h
@@ -16,6 +16,14 @@
#include "commands/vacuum.h"
#include "nodes/relation.h"
+/*
+ * Degree of how much MCV item matches a clause.
+ * This is then considered when computing the selectivity.
+ */
+#define STATS_MATCH_NONE 0 /* no match at all */
+#define STATS_MATCH_PARTIAL 1 /* partial match */
+#define STATS_MATCH_FULL 2 /* full match */
+
#define STATS_MAX_DIMENSIONS 8 /* max number of attributes */
/* Multivariate distinct coefficients */
@@ -78,8 +86,42 @@ typedef struct MVDependencies
/* size of the struct excluding the deps array */
#define SizeOfDependencies (offsetof(MVDependencies, ndeps) + sizeof(uint32))
+/* used to flag stats serialized to bytea */
+#define STATS_MCV_MAGIC 0xE1A651C2 /* marks serialized
+ * bytea */
+#define STATS_MCV_TYPE_BASIC 1 /* basic MCV list type */
+
+/* max items in MCV list (mostly arbitrary number) */
+#define STATS_MCVLIST_MAX_ITEMS 8192
+
+/*
+ * Multivariate MCV (most-common value) lists
+ *
+ * A straight-forward extension of MCV items - i.e. a list (array) of
+ * combinations of attribute values, together with a frequency and null flags.
+ */
+typedef struct MCVItem
+{
+ double frequency; /* frequency of this combination */
+ double base_frequency; /* frequency if independent */
+ bool *isnull; /* lags of NULL values (up to 32 columns) */
+ Datum *values; /* variable-length (ndimensions) */
+} MCVItem;
+
+/* multivariate MCV list - essentally an array of MCV items */
+typedef struct MCVList
+{
+ uint32 magic; /* magic constant marker */
+ uint32 type; /* type of MCV list (BASIC) */
+ uint32 nitems; /* number of MCV items in the array */
+ AttrNumber ndimensions; /* number of dimensions */
+ Oid types[STATS_MAX_DIMENSIONS]; /* OIDs of data types */
+ MCVItem **items; /* array of MCV items */
+} MCVList;
+
extern MVNDistinct *statext_ndistinct_load(Oid mvoid);
extern MVDependencies *statext_dependencies_load(Oid mvoid);
+extern MCVList * statext_mcv_load(Oid mvoid);
extern void BuildRelationExtStatistics(Relation onerel, double totalrows,
int numrows, HeapTuple *rows,
@@ -92,6 +134,13 @@ extern Selectivity dependencies_clauselist_selectivity(PlannerInfo *root,
SpecialJoinInfo *sjinfo,
RelOptInfo *rel,
Bitmapset **estimatedclauses);
+extern Selectivity statext_clauselist_selectivity(PlannerInfo *root,
+ List *clauses,
+ int varRelid,
+ JoinType jointype,
+ SpecialJoinInfo *sjinfo,
+ RelOptInfo *rel,
+ Bitmapset **estimatedclauses);
extern bool has_stats_of_kind(List *stats, char requiredkind);
extern StatisticExtInfo *choose_best_statistics(List *stats,
Bitmapset *attnums, char requiredkind);
diff --git a/src/include/utils/selfuncs.h b/src/include/utils/selfuncs.h
index 95e44280c4..4e9aaca6b5 100644
--- a/src/include/utils/selfuncs.h
+++ b/src/include/utils/selfuncs.h
@@ -209,6 +209,8 @@ extern void mergejoinscansel(PlannerInfo *root, Node *clause,
extern double estimate_num_groups(PlannerInfo *root, List *groupExprs,
double input_rows, List **pgset);
+extern double estimate_num_groups_simple(PlannerInfo *root, List *vars);
+
extern void estimate_hash_bucket_stats(PlannerInfo *root,
Node *hashkey, double nbuckets,
Selectivity *mcv_freq,
diff --git a/src/test/regress/expected/create_table_like.out b/src/test/regress/expected/create_table_like.out
index b582211270..4c8a5bd7e2 100644
--- a/src/test/regress/expected/create_table_like.out
+++ b/src/test/regress/expected/create_table_like.out
@@ -243,7 +243,7 @@ Indexes:
Check constraints:
"ctlt1_a_check" CHECK (length(a) > 2)
Statistics objects:
- "public"."ctlt_all_a_b_stat" (ndistinct, dependencies) ON a, b FROM ctlt_all
+ "public"."ctlt_all_a_b_stat" (ndistinct, dependencies, mcv) ON a, b FROM ctlt_all
SELECT c.relname, objsubid, description FROM pg_description, pg_index i, pg_class c WHERE classoid = 'pg_class'::regclass AND objoid = i.indexrelid AND c.oid = i.indexrelid AND i.indrelid = 'ctlt_all'::regclass ORDER BY c.relname, objsubid;
relname | objsubid | description
diff --git a/src/test/regress/expected/opr_sanity.out b/src/test/regress/expected/opr_sanity.out
index 214ad2d619..fab4597908 100644
--- a/src/test/regress/expected/opr_sanity.out
+++ b/src/test/regress/expected/opr_sanity.out
@@ -893,11 +893,12 @@ WHERE c.castmethod = 'b' AND
pg_node_tree | text | 0 | i
pg_ndistinct | bytea | 0 | i
pg_dependencies | bytea | 0 | i
+ pg_mcv_list | bytea | 0 | i
cidr | inet | 0 | i
xml | text | 0 | a
xml | character varying | 0 | a
xml | character | 0 | a
-(9 rows)
+(10 rows)
-- **************** pg_conversion ****************
-- Look for illegal values in pg_conversion fields.
diff --git a/src/test/regress/expected/stats_ext.out b/src/test/regress/expected/stats_ext.out
index 054a381dad..5d05962c04 100644
--- a/src/test/regress/expected/stats_ext.out
+++ b/src/test/regress/expected/stats_ext.out
@@ -58,7 +58,7 @@ ALTER TABLE ab1 DROP COLUMN a;
b | integer | | |
c | integer | | |
Statistics objects:
- "public"."ab1_b_c_stats" (ndistinct, dependencies) ON b, c FROM ab1
+ "public"."ab1_b_c_stats" (ndistinct, dependencies, mcv) ON b, c FROM ab1
-- Ensure statistics are dropped when table is
SELECT stxname FROM pg_statistic_ext WHERE stxname LIKE 'ab1%';
@@ -206,7 +206,7 @@ SELECT stxkind, stxndistinct
FROM pg_statistic_ext WHERE stxrelid = 'ndistinct'::regclass;
stxkind | stxndistinct
---------+---------------------------------------------------------
- {d,f} | {"3, 4": 301, "3, 6": 301, "4, 6": 301, "3, 4, 6": 301}
+ {d,f,m} | {"3, 4": 301, "3, 6": 301, "4, 6": 301, "3, 4, 6": 301}
(1 row)
-- Hash Aggregate, thanks to estimates improved by the statistic
@@ -272,7 +272,7 @@ SELECT stxkind, stxndistinct
FROM pg_statistic_ext WHERE stxrelid = 'ndistinct'::regclass;
stxkind | stxndistinct
---------+-------------------------------------------------------------
- {d,f} | {"3, 4": 2550, "3, 6": 800, "4, 6": 1632, "3, 4, 6": 10000}
+ {d,f,m} | {"3, 4": 2550, "3, 6": 800, "4, 6": 1632, "3, 4, 6": 10000}
(1 row)
-- plans using Group Aggregate, thanks to using correct esimates
@@ -509,3 +509,316 @@ EXPLAIN (COSTS OFF)
(5 rows)
RESET random_page_cost;
+-- MCV lists
+CREATE TABLE mcv_lists (
+ filler1 TEXT,
+ filler2 NUMERIC,
+ a INT,
+ b TEXT,
+ filler3 DATE,
+ c INT,
+ d TEXT
+);
+SET random_page_cost = 1.2;
+CREATE INDEX mcv_lists_ab_idx ON mcv_lists (a, b);
+CREATE INDEX mcv_lists_abc_idx ON mcv_lists (a, b, c);
+-- random data (no MCV list)
+INSERT INTO mcv_lists (a, b, c, filler1)
+ SELECT mod(i,37), mod(i,41), mod(i,43), mod(i,47) FROM generate_series(1,5000) s(i);
+ANALYZE mcv_lists;
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a = 1 AND b = '1';
+ QUERY PLAN
+---------------------------------------------------
+ Bitmap Heap Scan on mcv_lists
+ Recheck Cond: ((a = 1) AND (b = '1'::text))
+ -> Bitmap Index Scan on mcv_lists_abc_idx
+ Index Cond: ((a = 1) AND (b = '1'::text))
+(4 rows)
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a = 1 AND b = '1' AND c = 1;
+ QUERY PLAN
+---------------------------------------------------------
+ Index Scan using mcv_lists_abc_idx on mcv_lists
+ Index Cond: ((a = 1) AND (b = '1'::text) AND (c = 1))
+(2 rows)
+
+-- create statistics
+CREATE STATISTICS mcv_lists_stats (mcv) ON a, b, c FROM mcv_lists;
+ANALYZE mcv_lists;
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a = 1 AND b = '1';
+ QUERY PLAN
+---------------------------------------------------
+ Bitmap Heap Scan on mcv_lists
+ Recheck Cond: ((a = 1) AND (b = '1'::text))
+ -> Bitmap Index Scan on mcv_lists_abc_idx
+ Index Cond: ((a = 1) AND (b = '1'::text))
+(4 rows)
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a = 1 AND b = '1' AND c = 1;
+ QUERY PLAN
+---------------------------------------------------------
+ Index Scan using mcv_lists_abc_idx on mcv_lists
+ Index Cond: ((a = 1) AND (b = '1'::text) AND (c = 1))
+(2 rows)
+
+-- 100 distinct combinations, all in the MCV list
+TRUNCATE mcv_lists;
+DROP STATISTICS mcv_lists_stats;
+INSERT INTO mcv_lists (a, b, c, filler1)
+ SELECT mod(i,100), mod(i,50), mod(i,25), i FROM generate_series(1,5000) s(i);
+ANALYZE mcv_lists;
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a = 1 AND b = '1';
+ QUERY PLAN
+-------------------------------------------------
+ Index Scan using mcv_lists_abc_idx on mcv_lists
+ Index Cond: ((a = 1) AND (b = '1'::text))
+(2 rows)
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a < 1 AND b < '1';
+ QUERY PLAN
+-------------------------------------------------
+ Index Scan using mcv_lists_abc_idx on mcv_lists
+ Index Cond: ((a < 1) AND (b < '1'::text))
+(2 rows)
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a <= 0 AND b <= '0';
+ QUERY PLAN
+-------------------------------------------------
+ Index Scan using mcv_lists_abc_idx on mcv_lists
+ Index Cond: ((a <= 0) AND (b <= '0'::text))
+(2 rows)
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a = 1 AND b = '1' AND c = 1;
+ QUERY PLAN
+---------------------------------------------------------
+ Index Scan using mcv_lists_abc_idx on mcv_lists
+ Index Cond: ((a = 1) AND (b = '1'::text) AND (c = 1))
+(2 rows)
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a < 5 AND b < '1' AND c < 5;
+ QUERY PLAN
+---------------------------------------------------------
+ Index Scan using mcv_lists_abc_idx on mcv_lists
+ Index Cond: ((a < 5) AND (b < '1'::text) AND (c < 5))
+(2 rows)
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a <= 4 AND b <= '0' AND c <= 4;
+ QUERY PLAN
+------------------------------------------------------------
+ Index Scan using mcv_lists_abc_idx on mcv_lists
+ Index Cond: ((a <= 4) AND (b <= '0'::text) AND (c <= 4))
+(2 rows)
+
+-- create statistics
+CREATE STATISTICS mcv_lists_stats (mcv) ON a, b, c FROM mcv_lists;
+ANALYZE mcv_lists;
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a = 1 AND b = '1';
+ QUERY PLAN
+---------------------------------------------------
+ Bitmap Heap Scan on mcv_lists
+ Recheck Cond: ((a = 1) AND (b = '1'::text))
+ -> Bitmap Index Scan on mcv_lists_abc_idx
+ Index Cond: ((a = 1) AND (b = '1'::text))
+(4 rows)
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a < 1 AND b < '1';
+ QUERY PLAN
+---------------------------------------------------
+ Bitmap Heap Scan on mcv_lists
+ Recheck Cond: ((a < 1) AND (b < '1'::text))
+ -> Bitmap Index Scan on mcv_lists_abc_idx
+ Index Cond: ((a < 1) AND (b < '1'::text))
+(4 rows)
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a <= 0 AND b <= '0';
+ QUERY PLAN
+-----------------------------------------------------
+ Bitmap Heap Scan on mcv_lists
+ Recheck Cond: ((a <= 0) AND (b <= '0'::text))
+ -> Bitmap Index Scan on mcv_lists_abc_idx
+ Index Cond: ((a <= 0) AND (b <= '0'::text))
+(4 rows)
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a = 1 AND b = '1' AND c = 1;
+ QUERY PLAN
+---------------------------------------------------
+ Bitmap Heap Scan on mcv_lists
+ Recheck Cond: ((a = 1) AND (b = '1'::text))
+ Filter: (c = 1)
+ -> Bitmap Index Scan on mcv_lists_ab_idx
+ Index Cond: ((a = 1) AND (b = '1'::text))
+(5 rows)
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a < 5 AND b < '1' AND c < 5;
+ QUERY PLAN
+---------------------------------------------------
+ Bitmap Heap Scan on mcv_lists
+ Recheck Cond: ((a < 5) AND (b < '1'::text))
+ Filter: (c < 5)
+ -> Bitmap Index Scan on mcv_lists_ab_idx
+ Index Cond: ((a < 5) AND (b < '1'::text))
+(5 rows)
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a <= 4 AND b <= '0' AND c <= 4;
+ QUERY PLAN
+-----------------------------------------------------
+ Bitmap Heap Scan on mcv_lists
+ Recheck Cond: ((a <= 4) AND (b <= '0'::text))
+ Filter: (c <= 4)
+ -> Bitmap Index Scan on mcv_lists_ab_idx
+ Index Cond: ((a <= 4) AND (b <= '0'::text))
+(5 rows)
+
+-- check change of column type resets the MCV statistics
+ALTER TABLE mcv_lists ALTER COLUMN c TYPE numeric;
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a = 1 AND b = '1';
+ QUERY PLAN
+-------------------------------------------------
+ Index Scan using mcv_lists_abc_idx on mcv_lists
+ Index Cond: ((a = 1) AND (b = '1'::text))
+(2 rows)
+
+ANALYZE mcv_lists;
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a = 1 AND b = '1';
+ QUERY PLAN
+---------------------------------------------------
+ Bitmap Heap Scan on mcv_lists
+ Recheck Cond: ((a = 1) AND (b = '1'::text))
+ -> Bitmap Index Scan on mcv_lists_abc_idx
+ Index Cond: ((a = 1) AND (b = '1'::text))
+(4 rows)
+
+-- 100 distinct combinations with NULL values, all in the MCV list
+TRUNCATE mcv_lists;
+DROP STATISTICS mcv_lists_stats;
+INSERT INTO mcv_lists (a, b, c, filler1)
+ SELECT
+ (CASE WHEN mod(i,100) = 1 THEN NULL ELSE mod(i,100) END),
+ (CASE WHEN mod(i,50) = 1 THEN NULL ELSE mod(i,50) END),
+ (CASE WHEN mod(i,25) = 1 THEN NULL ELSE mod(i,25) END),
+ i
+ FROM generate_series(1,5000) s(i);
+ANALYZE mcv_lists;
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a IS NULL AND b IS NULL;
+ QUERY PLAN
+-------------------------------------------------
+ Index Scan using mcv_lists_abc_idx on mcv_lists
+ Index Cond: ((a IS NULL) AND (b IS NULL))
+(2 rows)
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a IS NULL AND b IS NULL AND c IS NULL;
+ QUERY PLAN
+------------------------------------------------
+ Index Scan using mcv_lists_ab_idx on mcv_lists
+ Index Cond: ((a IS NULL) AND (b IS NULL))
+ Filter: (c IS NULL)
+(3 rows)
+
+-- create statistics
+CREATE STATISTICS mcv_lists_stats (mcv) ON a, b, c FROM mcv_lists;
+ANALYZE mcv_lists;
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a IS NULL AND b IS NULL;
+ QUERY PLAN
+---------------------------------------------------
+ Bitmap Heap Scan on mcv_lists
+ Recheck Cond: ((a IS NULL) AND (b IS NULL))
+ -> Bitmap Index Scan on mcv_lists_abc_idx
+ Index Cond: ((a IS NULL) AND (b IS NULL))
+(4 rows)
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a IS NULL AND b IS NULL AND c IS NULL;
+ QUERY PLAN
+---------------------------------------------------
+ Bitmap Heap Scan on mcv_lists
+ Recheck Cond: ((a IS NULL) AND (b IS NULL))
+ Filter: (c IS NULL)
+ -> Bitmap Index Scan on mcv_lists_ab_idx
+ Index Cond: ((a IS NULL) AND (b IS NULL))
+(5 rows)
+
+RESET random_page_cost;
+-- mcv with arrays
+CREATE TABLE mcv_lists_arrays (
+ a TEXT[],
+ b NUMERIC[],
+ c INT[]
+);
+INSERT INTO mcv_lists_arrays (a, b, c)
+ SELECT
+ ARRAY[md5((i/100)::text), md5((i/100-1)::text), md5((i/100+1)::text)],
+ ARRAY[(i/100-1)::numeric/1000, (i/100)::numeric/1000, (i/100+1)::numeric/1000],
+ ARRAY[(i/100-1), i/100, (i/100+1)]
+ FROM generate_series(1,5000) s(i);
+CREATE STATISTICS mcv_lists_arrays_stats (mcv) ON a, b, c
+ FROM mcv_lists_arrays;
+ANALYZE mcv_lists_arrays;
+-- mcv with bool
+CREATE TABLE mcv_lists_bool (
+ a BOOL,
+ b BOOL,
+ c BOOL
+);
+INSERT INTO mcv_lists_bool (a, b, c)
+ SELECT
+ (mod(i,2) = 0), (mod(i,4) = 0), (mod(i,8) = 0)
+ FROM generate_series(1,10000) s(i);
+CREATE INDEX mcv_lists_bool_ab_idx ON mcv_lists_bool (a, b);
+CREATE INDEX mcv_lists_bool_abc_idx ON mcv_lists_bool (a, b, c);
+CREATE STATISTICS mcv_lists_bool_stats (mcv) ON a, b, c
+ FROM mcv_lists_bool;
+ANALYZE mcv_lists_bool;
+EXPLAIN (COSTS OFF) SELECT * FROM mcv_lists_bool WHERE a AND b AND c;
+ QUERY PLAN
+----------------------------------------------------------------
+ Bitmap Heap Scan on mcv_lists_bool
+ Filter: (a AND b AND c)
+ -> Bitmap Index Scan on mcv_lists_bool_abc_idx
+ Index Cond: ((a = true) AND (b = true) AND (c = true))
+(4 rows)
+
+EXPLAIN (COSTS OFF) SELECT * FROM mcv_lists_bool WHERE NOT a AND b AND c;
+ QUERY PLAN
+----------------------------------------------------------
+ Index Scan using mcv_lists_bool_ab_idx on mcv_lists_bool
+ Index Cond: ((a = false) AND (b = true))
+ Filter: ((NOT a) AND b AND c)
+(3 rows)
+
+EXPLAIN (COSTS OFF) SELECT * FROM mcv_lists_bool WHERE NOT a AND NOT b AND c;
+ QUERY PLAN
+----------------------------------------------------------------
+ Index Only Scan using mcv_lists_bool_abc_idx on mcv_lists_bool
+ Index Cond: ((a = false) AND (b = false) AND (c = true))
+ Filter: ((NOT a) AND (NOT b) AND c)
+(3 rows)
+
+EXPLAIN (COSTS OFF) SELECT * FROM mcv_lists_bool WHERE NOT a AND b AND NOT c;
+ QUERY PLAN
+----------------------------------------------------------
+ Index Scan using mcv_lists_bool_ab_idx on mcv_lists_bool
+ Index Cond: ((a = false) AND (b = true))
+ Filter: ((NOT a) AND b AND (NOT c))
+(3 rows)
+
diff --git a/src/test/regress/expected/type_sanity.out b/src/test/regress/expected/type_sanity.out
index b1419d4bc2..a56d6c5231 100644
--- a/src/test/regress/expected/type_sanity.out
+++ b/src/test/regress/expected/type_sanity.out
@@ -72,8 +72,9 @@ WHERE p1.typtype not in ('c','d','p') AND p1.typname NOT LIKE E'\\_%'
194 | pg_node_tree
3361 | pg_ndistinct
3402 | pg_dependencies
+ 4001 | pg_mcv_list
210 | smgr
-(4 rows)
+(5 rows)
-- Make sure typarray points to a varlena array type of our own base
SELECT p1.oid, p1.typname as basetype, p2.typname as arraytype,
diff --git a/src/test/regress/sql/stats_ext.sql b/src/test/regress/sql/stats_ext.sql
index 46acaadb39..ad1f103217 100644
--- a/src/test/regress/sql/stats_ext.sql
+++ b/src/test/regress/sql/stats_ext.sql
@@ -282,3 +282,184 @@ EXPLAIN (COSTS OFF)
SELECT * FROM functional_dependencies WHERE a = 1 AND b = '1' AND c = 1;
RESET random_page_cost;
+
+-- MCV lists
+CREATE TABLE mcv_lists (
+ filler1 TEXT,
+ filler2 NUMERIC,
+ a INT,
+ b TEXT,
+ filler3 DATE,
+ c INT,
+ d TEXT
+);
+
+SET random_page_cost = 1.2;
+
+CREATE INDEX mcv_lists_ab_idx ON mcv_lists (a, b);
+CREATE INDEX mcv_lists_abc_idx ON mcv_lists (a, b, c);
+
+-- random data (no MCV list)
+INSERT INTO mcv_lists (a, b, c, filler1)
+ SELECT mod(i,37), mod(i,41), mod(i,43), mod(i,47) FROM generate_series(1,5000) s(i);
+
+ANALYZE mcv_lists;
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a = 1 AND b = '1';
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a = 1 AND b = '1' AND c = 1;
+
+-- create statistics
+CREATE STATISTICS mcv_lists_stats (mcv) ON a, b, c FROM mcv_lists;
+
+ANALYZE mcv_lists;
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a = 1 AND b = '1';
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a = 1 AND b = '1' AND c = 1;
+
+-- 100 distinct combinations, all in the MCV list
+TRUNCATE mcv_lists;
+DROP STATISTICS mcv_lists_stats;
+
+INSERT INTO mcv_lists (a, b, c, filler1)
+ SELECT mod(i,100), mod(i,50), mod(i,25), i FROM generate_series(1,5000) s(i);
+
+ANALYZE mcv_lists;
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a = 1 AND b = '1';
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a < 1 AND b < '1';
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a <= 0 AND b <= '0';
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a = 1 AND b = '1' AND c = 1;
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a < 5 AND b < '1' AND c < 5;
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a <= 4 AND b <= '0' AND c <= 4;
+
+-- create statistics
+CREATE STATISTICS mcv_lists_stats (mcv) ON a, b, c FROM mcv_lists;
+
+ANALYZE mcv_lists;
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a = 1 AND b = '1';
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a < 1 AND b < '1';
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a <= 0 AND b <= '0';
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a = 1 AND b = '1' AND c = 1;
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a < 5 AND b < '1' AND c < 5;
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a <= 4 AND b <= '0' AND c <= 4;
+
+-- check change of column type resets the MCV statistics
+ALTER TABLE mcv_lists ALTER COLUMN c TYPE numeric;
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a = 1 AND b = '1';
+
+ANALYZE mcv_lists;
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a = 1 AND b = '1';
+
+-- 100 distinct combinations with NULL values, all in the MCV list
+TRUNCATE mcv_lists;
+DROP STATISTICS mcv_lists_stats;
+
+INSERT INTO mcv_lists (a, b, c, filler1)
+ SELECT
+ (CASE WHEN mod(i,100) = 1 THEN NULL ELSE mod(i,100) END),
+ (CASE WHEN mod(i,50) = 1 THEN NULL ELSE mod(i,50) END),
+ (CASE WHEN mod(i,25) = 1 THEN NULL ELSE mod(i,25) END),
+ i
+ FROM generate_series(1,5000) s(i);
+
+ANALYZE mcv_lists;
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a IS NULL AND b IS NULL;
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a IS NULL AND b IS NULL AND c IS NULL;
+
+-- create statistics
+CREATE STATISTICS mcv_lists_stats (mcv) ON a, b, c FROM mcv_lists;
+
+ANALYZE mcv_lists;
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a IS NULL AND b IS NULL;
+
+EXPLAIN (COSTS OFF)
+ SELECT * FROM mcv_lists WHERE a IS NULL AND b IS NULL AND c IS NULL;
+
+RESET random_page_cost;
+
+-- mcv with arrays
+CREATE TABLE mcv_lists_arrays (
+ a TEXT[],
+ b NUMERIC[],
+ c INT[]
+);
+
+INSERT INTO mcv_lists_arrays (a, b, c)
+ SELECT
+ ARRAY[md5((i/100)::text), md5((i/100-1)::text), md5((i/100+1)::text)],
+ ARRAY[(i/100-1)::numeric/1000, (i/100)::numeric/1000, (i/100+1)::numeric/1000],
+ ARRAY[(i/100-1), i/100, (i/100+1)]
+ FROM generate_series(1,5000) s(i);
+
+CREATE STATISTICS mcv_lists_arrays_stats (mcv) ON a, b, c
+ FROM mcv_lists_arrays;
+
+ANALYZE mcv_lists_arrays;
+
+-- mcv with bool
+CREATE TABLE mcv_lists_bool (
+ a BOOL,
+ b BOOL,
+ c BOOL
+);
+
+INSERT INTO mcv_lists_bool (a, b, c)
+ SELECT
+ (mod(i,2) = 0), (mod(i,4) = 0), (mod(i,8) = 0)
+ FROM generate_series(1,10000) s(i);
+
+CREATE INDEX mcv_lists_bool_ab_idx ON mcv_lists_bool (a, b);
+
+CREATE INDEX mcv_lists_bool_abc_idx ON mcv_lists_bool (a, b, c);
+
+CREATE STATISTICS mcv_lists_bool_stats (mcv) ON a, b, c
+ FROM mcv_lists_bool;
+
+ANALYZE mcv_lists_bool;
+
+EXPLAIN (COSTS OFF) SELECT * FROM mcv_lists_bool WHERE a AND b AND c;
+
+EXPLAIN (COSTS OFF) SELECT * FROM mcv_lists_bool WHERE NOT a AND b AND c;
+
+EXPLAIN (COSTS OFF) SELECT * FROM mcv_lists_bool WHERE NOT a AND NOT b AND c;
+
+EXPLAIN (COSTS OFF) SELECT * FROM mcv_lists_bool WHERE NOT a AND b AND NOT c;
--
2.17.2
--------------30B3B270E7C397EF57716F4D--
^ permalink raw reply [nested|flat] 8+ messages in thread
* [PATCH v5 12/14] WIP: Don't initialize page in {vm,fsm}_extend(), not needed
@ 2023-03-01 04:56 Andres Freund <[email protected]>
0 siblings, 0 replies; 8+ messages in thread
From: Andres Freund @ 2023-03-01 04:56 UTC (permalink / raw)
---
src/backend/access/heap/visibilitymap.c | 6 +-----
src/backend/storage/freespace/freespace.c | 5 +----
2 files changed, 2 insertions(+), 9 deletions(-)
diff --git a/src/backend/access/heap/visibilitymap.c b/src/backend/access/heap/visibilitymap.c
index 74ff01bb172..709213d0d97 100644
--- a/src/backend/access/heap/visibilitymap.c
+++ b/src/backend/access/heap/visibilitymap.c
@@ -623,11 +623,9 @@ static void
vm_extend(Relation rel, BlockNumber vm_nblocks)
{
BlockNumber vm_nblocks_now;
- PGAlignedBlock pg;
+ PGAlignedBlock pg = {0};
SMgrRelation reln;
- PageInit((Page) pg.data, BLCKSZ, 0);
-
/*
* We use the relation extension lock to lock out other backends trying to
* extend the visibility map at the same time. It also locks out extension
@@ -663,8 +661,6 @@ vm_extend(Relation rel, BlockNumber vm_nblocks)
/* Now extend the file */
while (vm_nblocks_now < vm_nblocks)
{
- PageSetChecksumInplace((Page) pg.data, vm_nblocks_now);
-
smgrextend(reln, VISIBILITYMAP_FORKNUM, vm_nblocks_now, pg.data, false);
vm_nblocks_now++;
}
diff --git a/src/backend/storage/freespace/freespace.c b/src/backend/storage/freespace/freespace.c
index 3e9693b293b..90c529958e7 100644
--- a/src/backend/storage/freespace/freespace.c
+++ b/src/backend/storage/freespace/freespace.c
@@ -608,10 +608,9 @@ static void
fsm_extend(Relation rel, BlockNumber fsm_nblocks)
{
BlockNumber fsm_nblocks_now;
- PGAlignedBlock pg;
+ PGAlignedBlock pg = {0};
SMgrRelation reln;
- PageInit((Page) pg.data, BLCKSZ, 0);
/*
* We use the relation extension lock to lock out other backends trying to
@@ -649,8 +648,6 @@ fsm_extend(Relation rel, BlockNumber fsm_nblocks)
/* Extend as needed. */
while (fsm_nblocks_now < fsm_nblocks)
{
- PageSetChecksumInplace((Page) pg.data, fsm_nblocks_now);
-
smgrextend(reln, FSM_FORKNUM, fsm_nblocks_now,
pg.data, false);
fsm_nblocks_now++;
--
2.38.0
--wwosng5vofi3dmcs
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v5-0013-Convert-a-few-places-to-ExtendBufferedRelTo.patch"
^ permalink raw reply [nested|flat] 8+ messages in thread
* [PATCH v6 15/17] WIP: Don't initialize page in {vm,fsm}_extend(), not needed
@ 2023-03-01 04:56 Andres Freund <[email protected]>
0 siblings, 0 replies; 8+ messages in thread
From: Andres Freund @ 2023-03-01 04:56 UTC (permalink / raw)
---
src/backend/access/heap/visibilitymap.c | 6 +-----
src/backend/storage/freespace/freespace.c | 5 +----
2 files changed, 2 insertions(+), 9 deletions(-)
diff --git a/src/backend/access/heap/visibilitymap.c b/src/backend/access/heap/visibilitymap.c
index 74ff01bb172..709213d0d97 100644
--- a/src/backend/access/heap/visibilitymap.c
+++ b/src/backend/access/heap/visibilitymap.c
@@ -623,11 +623,9 @@ static void
vm_extend(Relation rel, BlockNumber vm_nblocks)
{
BlockNumber vm_nblocks_now;
- PGAlignedBlock pg;
+ PGAlignedBlock pg = {0};
SMgrRelation reln;
- PageInit((Page) pg.data, BLCKSZ, 0);
-
/*
* We use the relation extension lock to lock out other backends trying to
* extend the visibility map at the same time. It also locks out extension
@@ -663,8 +661,6 @@ vm_extend(Relation rel, BlockNumber vm_nblocks)
/* Now extend the file */
while (vm_nblocks_now < vm_nblocks)
{
- PageSetChecksumInplace((Page) pg.data, vm_nblocks_now);
-
smgrextend(reln, VISIBILITYMAP_FORKNUM, vm_nblocks_now, pg.data, false);
vm_nblocks_now++;
}
diff --git a/src/backend/storage/freespace/freespace.c b/src/backend/storage/freespace/freespace.c
index 3e9693b293b..90c529958e7 100644
--- a/src/backend/storage/freespace/freespace.c
+++ b/src/backend/storage/freespace/freespace.c
@@ -608,10 +608,9 @@ static void
fsm_extend(Relation rel, BlockNumber fsm_nblocks)
{
BlockNumber fsm_nblocks_now;
- PGAlignedBlock pg;
+ PGAlignedBlock pg = {0};
SMgrRelation reln;
- PageInit((Page) pg.data, BLCKSZ, 0);
/*
* We use the relation extension lock to lock out other backends trying to
@@ -649,8 +648,6 @@ fsm_extend(Relation rel, BlockNumber fsm_nblocks)
/* Extend as needed. */
while (fsm_nblocks_now < fsm_nblocks)
{
- PageSetChecksumInplace((Page) pg.data, fsm_nblocks_now);
-
smgrextend(reln, FSM_FORKNUM, fsm_nblocks_now,
pg.data, false);
fsm_nblocks_now++;
--
2.38.0
--cfn72vqnlbycypta
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v6-0016-Convert-a-few-places-to-ExtendBufferedRelTo.patch"
^ permalink raw reply [nested|flat] 8+ messages in thread
* [PATCH v4 14/15] WIP: Don't initialize page in {vm,fsm}_extend(), not needed
@ 2023-03-01 04:56 Andres Freund <[email protected]>
0 siblings, 0 replies; 8+ messages in thread
From: Andres Freund @ 2023-03-01 04:56 UTC (permalink / raw)
---
src/backend/access/heap/visibilitymap.c | 6 +-----
src/backend/storage/freespace/freespace.c | 5 +----
2 files changed, 2 insertions(+), 9 deletions(-)
diff --git a/src/backend/access/heap/visibilitymap.c b/src/backend/access/heap/visibilitymap.c
index 74ff01bb172..709213d0d97 100644
--- a/src/backend/access/heap/visibilitymap.c
+++ b/src/backend/access/heap/visibilitymap.c
@@ -623,11 +623,9 @@ static void
vm_extend(Relation rel, BlockNumber vm_nblocks)
{
BlockNumber vm_nblocks_now;
- PGAlignedBlock pg;
+ PGAlignedBlock pg = {0};
SMgrRelation reln;
- PageInit((Page) pg.data, BLCKSZ, 0);
-
/*
* We use the relation extension lock to lock out other backends trying to
* extend the visibility map at the same time. It also locks out extension
@@ -663,8 +661,6 @@ vm_extend(Relation rel, BlockNumber vm_nblocks)
/* Now extend the file */
while (vm_nblocks_now < vm_nblocks)
{
- PageSetChecksumInplace((Page) pg.data, vm_nblocks_now);
-
smgrextend(reln, VISIBILITYMAP_FORKNUM, vm_nblocks_now, pg.data, false);
vm_nblocks_now++;
}
diff --git a/src/backend/storage/freespace/freespace.c b/src/backend/storage/freespace/freespace.c
index 3e9693b293b..90c529958e7 100644
--- a/src/backend/storage/freespace/freespace.c
+++ b/src/backend/storage/freespace/freespace.c
@@ -608,10 +608,9 @@ static void
fsm_extend(Relation rel, BlockNumber fsm_nblocks)
{
BlockNumber fsm_nblocks_now;
- PGAlignedBlock pg;
+ PGAlignedBlock pg = {0};
SMgrRelation reln;
- PageInit((Page) pg.data, BLCKSZ, 0);
/*
* We use the relation extension lock to lock out other backends trying to
@@ -649,8 +648,6 @@ fsm_extend(Relation rel, BlockNumber fsm_nblocks)
/* Extend as needed. */
while (fsm_nblocks_now < fsm_nblocks)
{
- PageSetChecksumInplace((Page) pg.data, fsm_nblocks_now);
-
smgrextend(reln, FSM_FORKNUM, fsm_nblocks_now,
pg.data, false);
fsm_nblocks_now++;
--
2.38.0
--njltjzibte523gwd
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v4-0015-Convert-a-few-places-to-ExtendBufferedRelTo.patch"
^ permalink raw reply [nested|flat] 8+ messages in thread
* [PATCH v9 4/7] Row pattern recognition patch (executor).
@ 2023-10-04 05:51 Tatsuo Ishii <[email protected]>
0 siblings, 0 replies; 8+ messages in thread
From: Tatsuo Ishii @ 2023-10-04 05:51 UTC (permalink / raw)
---
src/backend/executor/nodeWindowAgg.c | 1021 +++++++++++++++++++++++++-
src/backend/utils/adt/windowfuncs.c | 37 +-
src/include/catalog/pg_proc.dat | 6 +
src/include/nodes/execnodes.h | 26 +
4 files changed, 1077 insertions(+), 13 deletions(-)
diff --git a/src/backend/executor/nodeWindowAgg.c b/src/backend/executor/nodeWindowAgg.c
index 77724a6daa..5da1bb5a6f 100644
--- a/src/backend/executor/nodeWindowAgg.c
+++ b/src/backend/executor/nodeWindowAgg.c
@@ -36,6 +36,7 @@
#include "access/htup_details.h"
#include "catalog/objectaccess.h"
#include "catalog/pg_aggregate.h"
+#include "catalog/pg_collation_d.h"
#include "catalog/pg_proc.h"
#include "executor/executor.h"
#include "executor/nodeWindowAgg.h"
@@ -48,6 +49,7 @@
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/datum.h"
+#include "utils/fmgroids.h"
#include "utils/expandeddatum.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
@@ -159,6 +161,12 @@ typedef struct WindowStatePerAggData
bool restart; /* need to restart this agg in this cycle? */
} WindowStatePerAggData;
+typedef struct StringSet {
+ StringInfo *str_set;
+ Size set_size; /* current array allocation size in number of items */
+ int set_index; /* current used size */
+} StringSet;
+
static void initialize_windowaggregate(WindowAggState *winstate,
WindowStatePerFunc perfuncstate,
WindowStatePerAgg peraggstate);
@@ -182,8 +190,9 @@ static void begin_partition(WindowAggState *winstate);
static void spool_tuples(WindowAggState *winstate, int64 pos);
static void release_partition(WindowAggState *winstate);
-static int row_is_in_frame(WindowAggState *winstate, int64 pos,
+static int row_is_in_frame(WindowAggState *winstate, int64 pos,
TupleTableSlot *slot);
+
static void update_frameheadpos(WindowAggState *winstate);
static void update_frametailpos(WindowAggState *winstate);
static void update_grouptailpos(WindowAggState *winstate);
@@ -195,9 +204,37 @@ static Datum GetAggInitVal(Datum textInitVal, Oid transtype);
static bool are_peers(WindowAggState *winstate, TupleTableSlot *slot1,
TupleTableSlot *slot2);
-static bool window_gettupleslot(WindowObject winobj, int64 pos,
- TupleTableSlot *slot);
+static int WinGetSlotInFrame(WindowObject winobj, TupleTableSlot *slot,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout);
+static bool window_gettupleslot(WindowObject winobj, int64 pos, TupleTableSlot *slot);
+
+static void attno_map(Node *node);
+static bool attno_map_walker(Node *node, void *context);
+static int row_is_in_reduced_frame(WindowObject winobj, int64 pos);
+static bool rpr_is_defined(WindowAggState *winstate);
+
+static void create_reduced_frame_map(WindowAggState *winstate);
+static int get_reduced_frame_map(WindowAggState *winstate, int64 pos);
+static void register_reduced_frame_map(WindowAggState *winstate, int64 pos, int val);
+static void clear_reduced_frame_map(WindowAggState *winstate);
+static void update_reduced_frame(WindowObject winobj, int64 pos);
+
+static int64 evaluate_pattern(WindowObject winobj, int64 current_pos,
+ char *vname, StringInfo encoded_str, bool *result);
+
+static bool get_slots(WindowObject winobj, int64 current_pos);
+
+static int search_str_set(char *pattern, StringSet *str_set);
+static char pattern_initial(WindowAggState *winstate, char *vname);
+static int do_pattern_match(char *pattern, char *encoded_str);
+
+static StringSet *string_set_init(void);
+static void string_set_add(StringSet *string_set, StringInfo str);
+static StringInfo string_set_get(StringSet *string_set, int index);
+static int string_set_get_size(StringSet *string_set);
+static void string_set_discard(StringSet *string_set);
/*
* initialize_windowaggregate
@@ -673,6 +710,7 @@ eval_windowaggregates(WindowAggState *winstate)
WindowObject agg_winobj;
TupleTableSlot *agg_row_slot;
TupleTableSlot *temp_slot;
+ bool agg_result_isnull;
numaggs = winstate->numaggs;
if (numaggs == 0)
@@ -778,6 +816,9 @@ eval_windowaggregates(WindowAggState *winstate)
* Note that we don't strictly need to restart in the last case, but if
* we're going to remove all rows from the aggregation anyway, a restart
* surely is faster.
+ *
+ * - if RPR is enabled and skip mode is SKIP TO NEXT ROW,
+ * we restart aggregation too.
*----------
*/
numaggs_restart = 0;
@@ -788,8 +829,11 @@ eval_windowaggregates(WindowAggState *winstate)
(winstate->aggregatedbase != winstate->frameheadpos &&
!OidIsValid(peraggstate->invtransfn_oid)) ||
(winstate->frameOptions & FRAMEOPTION_EXCLUSION) ||
- winstate->aggregatedupto <= winstate->frameheadpos)
+ winstate->aggregatedupto <= winstate->frameheadpos ||
+ (rpr_is_defined(winstate) &&
+ winstate->rpSkipTo == ST_NEXT_ROW))
{
+ elog(DEBUG1, "peraggstate->restart is set");
peraggstate->restart = true;
numaggs_restart++;
}
@@ -861,8 +905,10 @@ eval_windowaggregates(WindowAggState *winstate)
* If we created a mark pointer for aggregates, keep it pushed up to frame
* head, so that tuplestore can discard unnecessary rows.
*/
+#ifdef NOT_USED
if (agg_winobj->markptr >= 0)
WinSetMarkPosition(agg_winobj, winstate->frameheadpos);
+#endif
/*
* Now restart the aggregates that require it.
@@ -917,6 +963,29 @@ eval_windowaggregates(WindowAggState *winstate)
{
winstate->aggregatedupto = winstate->frameheadpos;
ExecClearTuple(agg_row_slot);
+
+ /*
+ * If RPR is defined, we do not use aggregatedupto_nonrestarted. To
+ * avoid assertion failure below, we reset aggregatedupto_nonrestarted
+ * to frameheadpos.
+ */
+ if (rpr_is_defined(winstate))
+ aggregatedupto_nonrestarted = winstate->frameheadpos;
+ }
+
+ agg_result_isnull = false;
+ /* RPR is defined? */
+ if (rpr_is_defined(winstate))
+ {
+ /*
+ * If the skip mode is SKIP TO PAST LAST ROW and we already know that
+ * current row is a skipped row, we don't need to accumulate rows,
+ * just return NULL. Note that for unamtched row, we need to do
+ * aggregation so that count(*) shows 0, rather than NULL.
+ */
+ if (winstate->rpSkipTo == ST_PAST_LAST_ROW &&
+ get_reduced_frame_map(winstate, winstate->currentpos) == RF_SKIPPED)
+ agg_result_isnull = true;
}
/*
@@ -930,6 +999,11 @@ eval_windowaggregates(WindowAggState *winstate)
{
int ret;
+ elog(DEBUG1, "===== loop in frame starts: " INT64_FORMAT, winstate->aggregatedupto);
+
+ if (agg_result_isnull)
+ break;
+
/* Fetch next row if we didn't already */
if (TupIsNull(agg_row_slot))
{
@@ -945,9 +1019,28 @@ eval_windowaggregates(WindowAggState *winstate)
ret = row_is_in_frame(winstate, winstate->aggregatedupto, agg_row_slot);
if (ret < 0)
break;
+
if (ret == 0)
goto next_tuple;
+ if (rpr_is_defined(winstate))
+ {
+ /*
+ * If the row status at currentpos is already decided and current
+ * row status is not decided yet, it means we passed the last
+ * reduced frame. Time to break the loop.
+ */
+ if (get_reduced_frame_map(winstate, winstate->currentpos) != RF_NOT_DETERMINED &&
+ get_reduced_frame_map(winstate, winstate->aggregatedupto) == RF_NOT_DETERMINED)
+ break;
+ /*
+ * Otherwise we need to calculate the reduced frame.
+ */
+ ret = row_is_in_reduced_frame(winstate->agg_winobj, winstate->aggregatedupto);
+ if (ret == -1) /* unmatched row */
+ break;
+ }
+
/* Set tuple context for evaluation of aggregate arguments */
winstate->tmpcontext->ecxt_outertuple = agg_row_slot;
@@ -976,6 +1069,7 @@ next_tuple:
ExecClearTuple(agg_row_slot);
}
+
/* The frame's end is not supposed to move backwards, ever */
Assert(aggregatedupto_nonrestarted <= winstate->aggregatedupto);
@@ -996,6 +1090,16 @@ next_tuple:
peraggstate,
result, isnull);
+ /*
+ * RPR is defined and we just return NULL because skip mode is SKIP
+ * TO PAST LAST ROW and current row is skipped row.
+ */
+ if (agg_result_isnull)
+ {
+ *isnull = true;
+ *result = (Datum) 0;
+ }
+
/*
* save the result in case next row shares the same frame.
*
@@ -1090,6 +1194,7 @@ begin_partition(WindowAggState *winstate)
winstate->framehead_valid = false;
winstate->frametail_valid = false;
winstate->grouptail_valid = false;
+ create_reduced_frame_map(winstate);
winstate->spooled_rows = 0;
winstate->currentpos = 0;
winstate->frameheadpos = 0;
@@ -2053,6 +2158,8 @@ ExecWindowAgg(PlanState *pstate)
CHECK_FOR_INTERRUPTS();
+ elog(DEBUG1, "ExecWindowAgg called. pos: " INT64_FORMAT , winstate->currentpos);
+
if (winstate->status == WINDOWAGG_DONE)
return NULL;
@@ -2221,6 +2328,17 @@ ExecWindowAgg(PlanState *pstate)
/* don't evaluate the window functions when we're in pass-through mode */
if (winstate->status == WINDOWAGG_RUN)
{
+ /*
+ * If RPR is defined and skip mode is next row, we need to clear existing
+ * reduced frame info so that we newly calculate the info starting from
+ * current row.
+ */
+ if (rpr_is_defined(winstate))
+ {
+ if (winstate->rpSkipTo == ST_NEXT_ROW)
+ clear_reduced_frame_map(winstate);
+ }
+
/*
* Evaluate true window functions
*/
@@ -2388,6 +2506,9 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
TupleDesc scanDesc;
ListCell *l;
+ TargetEntry *te;
+ Expr *expr;
+
/* check for unsupported flags */
Assert(!(eflags & (EXEC_FLAG_BACKWARD | EXEC_FLAG_MARK)));
@@ -2483,6 +2604,16 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winstate->temp_slot_2 = ExecInitExtraTupleSlot(estate, scanDesc,
&TTSOpsMinimalTuple);
+ winstate->prev_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+
+ winstate->next_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+
+ winstate->null_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+ winstate->null_slot = ExecStoreAllNullTuple(winstate->null_slot);
+
/*
* create frame head and tail slots only if needed (must create slots in
* exactly the same cases that update_frameheadpos and update_frametailpos
@@ -2667,6 +2798,39 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winstate->inRangeAsc = node->inRangeAsc;
winstate->inRangeNullsFirst = node->inRangeNullsFirst;
+ /* Set up SKIP TO type */
+ winstate->rpSkipTo = node->rpSkipTo;
+ /* Set up row pattern recognition PATTERN clause */
+ winstate->patternVariableList = node->patternVariable;
+ winstate->patternRegexpList = node->patternRegexp;
+
+ /* Set up row pattern recognition DEFINE clause */
+ winstate->defineInitial = node->defineInitial;
+ winstate->defineVariableList = NIL;
+ winstate->defineClauseList = NIL;
+ if (node->defineClause != NIL)
+ {
+ /*
+ * Tweak arg var of PREV/NEXT so that it refers to scan/inner slot.
+ */
+ foreach(l, node->defineClause)
+ {
+ char *name;
+ ExprState *exps;
+
+ te = lfirst(l);
+ name = te->resname;
+ expr = te->expr;
+
+ elog(DEBUG1, "defineVariable name: %s", name);
+ winstate->defineVariableList = lappend(winstate->defineVariableList,
+ makeString(pstrdup(name)));
+ attno_map((Node *)expr);
+ exps = ExecInitExpr(expr, (PlanState *) winstate);
+ winstate->defineClauseList = lappend(winstate->defineClauseList, exps);
+ }
+ }
+
winstate->all_first = true;
winstate->partition_spooled = false;
winstate->more_partitions = false;
@@ -2674,6 +2838,57 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
return winstate;
}
+/*
+ * Rewrite varno of Var node that is the argument of PREV/NET so that it sees
+ * scan tuple (PREV) or inner tuple (NEXT).
+ */
+static void
+attno_map(Node *node)
+{
+ (void) expression_tree_walker(node, attno_map_walker, NULL);
+}
+
+static bool
+attno_map_walker(Node *node, void *context)
+{
+ FuncExpr *func;
+ int nargs;
+ Expr *expr;
+ Var *var;
+
+ if (node == NULL)
+ return false;
+
+ if (IsA(node, FuncExpr))
+ {
+ func = (FuncExpr *)node;
+
+ if (func->funcid == F_PREV || func->funcid == F_NEXT)
+ {
+ /* sanity check */
+ nargs = list_length(func->args);
+ if (list_length(func->args) != 1)
+ elog(ERROR, "PREV/NEXT must have 1 argument but function %d has %d args", func->funcid, nargs);
+
+ expr = (Expr *) lfirst(list_head(func->args));
+ if (!IsA(expr, Var))
+ elog(ERROR, "PREV/NEXT's arg is not Var"); /* XXX: is it possible that arg type is Const? */
+ var = (Var *)expr;
+
+ if (func->funcid == F_PREV)
+ /*
+ * Rewrite varno from OUTER_VAR to regular var no so that the
+ * var references scan tuple.
+ */
+ var->varno = var->varnosyn;
+ else
+ var->varno = INNER_VAR;
+ elog(DEBUG1, "PREV/NEXT's varno is rewritten to: %d", var->varno);
+ }
+ }
+ return expression_tree_walker(node, attno_map_walker, NULL);
+}
+
/* -----------------
* ExecEndWindowAgg
* -----------------
@@ -2723,6 +2938,8 @@ ExecReScanWindowAgg(WindowAggState *node)
ExecClearTuple(node->agg_row_slot);
ExecClearTuple(node->temp_slot_1);
ExecClearTuple(node->temp_slot_2);
+ ExecClearTuple(node->prev_slot);
+ ExecClearTuple(node->next_slot);
if (node->framehead_slot)
ExecClearTuple(node->framehead_slot);
if (node->frametail_slot)
@@ -3083,7 +3300,7 @@ window_gettupleslot(WindowObject winobj, int64 pos, TupleTableSlot *slot)
return false;
if (pos < winobj->markpos)
- elog(ERROR, "cannot fetch row before WindowObject's mark position");
+ elog(ERROR, "cannot fetch row: " INT64_FORMAT " before WindowObject's mark position: " INT64_FORMAT, pos, winobj->markpos );
oldcontext = MemoryContextSwitchTo(winstate->ss.ps.ps_ExprContext->ecxt_per_query_memory);
@@ -3403,14 +3620,54 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
WindowAggState *winstate;
ExprContext *econtext;
TupleTableSlot *slot;
- int64 abs_pos;
- int64 mark_pos;
Assert(WindowObjectIsValid(winobj));
winstate = winobj->winstate;
econtext = winstate->ss.ps.ps_ExprContext;
slot = winstate->temp_slot_1;
+ if (WinGetSlotInFrame(winobj, slot,
+ relpos, seektype, set_mark,
+ isnull, isout) == 0)
+ {
+ econtext->ecxt_outertuple = slot;
+ return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
+ econtext, isnull);
+ }
+
+ if (isout)
+ *isout = true;
+ *isnull = true;
+ return (Datum) 0;
+}
+
+/*
+ * WinGetSlotInFrame
+ * slot: TupleTableSlot to store the result
+ * relpos: signed rowcount offset from the seek position
+ * seektype: WINDOW_SEEK_HEAD or WINDOW_SEEK_TAIL
+ * set_mark: If the row is found/in frame and set_mark is true, the mark is
+ * moved to the row as a side-effect.
+ * isnull: output argument, receives isnull status of result
+ * isout: output argument, set to indicate whether target row position
+ * is out of frame (can pass NULL if caller doesn't care about this)
+ *
+ * Returns 0 if we successfullt got the slot. false if out of frame.
+ * (also isout is set)
+ */
+static int
+WinGetSlotInFrame(WindowObject winobj, TupleTableSlot *slot,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout)
+{
+ WindowAggState *winstate;
+ int64 abs_pos;
+ int64 mark_pos;
+ int num_reduced_frame;
+
+ Assert(WindowObjectIsValid(winobj));
+ winstate = winobj->winstate;
+
switch (seektype)
{
case WINDOW_SEEK_CURRENT:
@@ -3477,11 +3734,21 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
winstate->frameOptions);
break;
}
+ num_reduced_frame = row_is_in_reduced_frame(winobj, winstate->frameheadpos);
+ if (num_reduced_frame < 0)
+ goto out_of_frame;
+ else if (num_reduced_frame > 0)
+ if (relpos >= num_reduced_frame)
+ goto out_of_frame;
break;
case WINDOW_SEEK_TAIL:
/* rejecting relpos > 0 is easy and simplifies code below */
if (relpos > 0)
goto out_of_frame;
+
+ /* RPR cares about frame head pos. Need to call update_frameheadpos */
+ update_frameheadpos(winstate);
+
update_frametailpos(winstate);
abs_pos = winstate->frametailpos - 1 + relpos;
@@ -3548,6 +3815,12 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
mark_pos = 0; /* keep compiler quiet */
break;
}
+
+ num_reduced_frame = row_is_in_reduced_frame(winobj, winstate->frameheadpos + relpos);
+ if (num_reduced_frame < 0)
+ goto out_of_frame;
+ else if (num_reduced_frame > 0)
+ abs_pos = winstate->frameheadpos + relpos + num_reduced_frame - 1;
break;
default:
elog(ERROR, "unrecognized window seek type: %d", seektype);
@@ -3566,15 +3839,13 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
*isout = false;
if (set_mark)
WinSetMarkPosition(winobj, mark_pos);
- econtext->ecxt_outertuple = slot;
- return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
- econtext, isnull);
+ return 0;
out_of_frame:
if (isout)
*isout = true;
*isnull = true;
- return (Datum) 0;
+ return -1;
}
/*
@@ -3605,3 +3876,731 @@ WinGetFuncArgCurrent(WindowObject winobj, int argno, bool *isnull)
return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
econtext, isnull);
}
+
+/*
+ * rpr_is_defined
+ * return true if Row pattern recognition is defined.
+ */
+static
+bool rpr_is_defined(WindowAggState *winstate)
+{
+ return winstate->patternVariableList != NIL;
+}
+
+/*
+ * row_is_in_reduced_frame
+ * Determine whether a row is in the current row's reduced window frame according
+ * to row pattern matching
+ *
+ * The row must has been already determined that it is in a full window frame
+ * and fetched it into slot.
+ *
+ * Returns:
+ * = 0, RPR is not defined.
+ * >0, if the row is the first in the reduced frame. Return the number of rows in the reduced frame.
+ * -1, if the row is unmatched row
+ * -2, if the row is in the reduced frame but needed to be skipped because of
+ * AFTER MATCH SKIP PAST LAST ROW
+ */
+static
+int row_is_in_reduced_frame(WindowObject winobj, int64 pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ int state;
+ int rtn;
+
+ if (!rpr_is_defined(winstate))
+ {
+ /*
+ * RPR is not defined. Assume that we are always in the the reduced
+ * window frame.
+ */
+ rtn = 0;
+ elog(DEBUG1, "row_is_in_reduced_frame returns %d: pos: " INT64_FORMAT, rtn, pos);
+ return rtn;
+ }
+
+ state = get_reduced_frame_map(winstate, pos);
+
+ if (state == RF_NOT_DETERMINED)
+ {
+ update_frameheadpos(winstate);
+ update_reduced_frame(winobj, pos);
+ }
+
+ state = get_reduced_frame_map(winstate, pos);
+
+ switch (state)
+ {
+ int64 i;
+ int num_reduced_rows;
+
+ case RF_FRAME_HEAD:
+ num_reduced_rows = 1;
+ for (i = pos + 1; get_reduced_frame_map(winstate,i) == RF_SKIPPED; i++)
+ num_reduced_rows++;
+ rtn = num_reduced_rows;
+ break;
+
+ case RF_SKIPPED:
+ rtn = -2;
+ break;
+
+ case RF_UNMATCHED:
+ rtn = -1;
+ break;
+
+ default:
+ elog(ERROR, "Unrecognized state: %d at: " INT64_FORMAT, state, pos);
+ break;
+ }
+
+ elog(DEBUG1, "row_is_in_reduced_frame returns %d: pos: " INT64_FORMAT, rtn, pos);
+ return rtn;
+}
+
+#define REDUCED_FRAME_MAP_INIT_SIZE 1024L
+
+/*
+ * Create reduced frame map
+ */
+static
+void create_reduced_frame_map(WindowAggState *winstate)
+{
+ winstate->reduced_frame_map =
+ MemoryContextAlloc(winstate->partcontext, REDUCED_FRAME_MAP_INIT_SIZE);
+ winstate->alloc_sz = REDUCED_FRAME_MAP_INIT_SIZE;
+ clear_reduced_frame_map(winstate);
+}
+
+/*
+ * Clear reduced frame map
+ */
+static
+void clear_reduced_frame_map(WindowAggState *winstate)
+{
+ Assert(winstate->reduced_frame_map != NULL);
+ MemSet(winstate->reduced_frame_map, RF_NOT_DETERMINED,
+ winstate->alloc_sz);
+}
+
+/*
+ * Get reduced frame map specified by pos
+ */
+static
+int get_reduced_frame_map(WindowAggState *winstate, int64 pos)
+{
+ Assert(winstate->reduced_frame_map != NULL);
+
+ if (pos < 0 || pos >= winstate->alloc_sz)
+ elog(ERROR, "wrong pos: " INT64_FORMAT, pos);
+
+ return winstate->reduced_frame_map[pos];
+}
+
+/*
+ * Add/replace reduced frame map member at pos.
+ * If there's no enough space, expand the map.
+ */
+static
+void register_reduced_frame_map(WindowAggState *winstate, int64 pos, int val)
+{
+ int64 realloc_sz;
+
+ Assert(winstate->reduced_frame_map != NULL);
+
+ if (pos < 0)
+ elog(ERROR, "wrong pos: " INT64_FORMAT, pos);
+
+ if (pos > winstate->alloc_sz - 1)
+ {
+ realloc_sz = winstate->alloc_sz * 2;
+
+ winstate->reduced_frame_map =
+ repalloc(winstate->reduced_frame_map, realloc_sz);
+
+ MemSet(winstate->reduced_frame_map + winstate->alloc_sz,
+ RF_NOT_DETERMINED, realloc_sz - winstate->alloc_sz);
+
+ winstate->alloc_sz = realloc_sz;
+ }
+
+ winstate->reduced_frame_map[pos] = val;
+}
+
+/*
+ * update_reduced_frame
+ * Update reduced frame info.
+ */
+static
+void update_reduced_frame(WindowObject winobj, int64 pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ ListCell *lc1, *lc2;
+ bool expression_result;
+ int num_matched_rows;
+ int64 original_pos;
+ bool anymatch;
+ StringInfo encoded_str;
+ StringInfo pattern_str = makeStringInfo();
+ StringSet *str_set;
+ int str_set_index = 0;
+
+ /*
+ * Set of pattern variables evaluted to true.
+ * Each character corresponds to pattern variable.
+ * Example:
+ * str_set[0] = "AB";
+ * str_set[1] = "AC";
+ * In this case at row 0 A and B are true, and A and C are true in row 1.
+ */
+
+ /* initialize pattern variables set */
+ str_set = string_set_init();
+
+ /* save original pos */
+ original_pos = pos;
+
+ /*
+ * Loop over until none of pattern matches or encounters end of frame.
+ */
+ for (;;)
+ {
+ int64 result_pos = -1;
+
+ /*
+ * Loop over each PATTERN variable.
+ */
+ anymatch = false;
+ encoded_str = makeStringInfo();
+
+ forboth(lc1, winstate->patternVariableList, lc2, winstate->patternRegexpList)
+ {
+ char *vname = strVal(lfirst(lc1));
+ char *quantifier = strVal(lfirst(lc2));
+
+ elog(DEBUG1, "pos: " INT64_FORMAT " pattern vname: %s quantifier: %s", pos, vname, quantifier);
+
+ expression_result = false;
+
+ /* evaluate row pattern against current row */
+ result_pos = evaluate_pattern(winobj, pos, vname, encoded_str, &expression_result);
+ if (expression_result)
+ {
+ elog(DEBUG1, "expression result is true");
+ anymatch = true;
+ }
+
+ /*
+ * If out of frame, we are done.
+ */
+ if (result_pos < 0)
+ break;
+ }
+
+ if (!anymatch)
+ {
+ /* none of patterns matched. */
+ break;
+ }
+
+ string_set_add(str_set, encoded_str);
+
+ elog(DEBUG1, "pos: " INT64_FORMAT " str_set_index: %d encoded_str: %s", pos, str_set_index, encoded_str->data);
+
+ /* move to next row */
+ pos++;
+
+ if (result_pos < 0)
+ {
+ /* out of frame */
+ break;
+ }
+ }
+
+ if (string_set_get_size(str_set) == 0)
+ {
+ /* no match found in the first row */
+ register_reduced_frame_map(winstate, original_pos, RF_UNMATCHED);
+ return;
+ }
+
+ elog(DEBUG2, "pos: " INT64_FORMAT " encoded_str: %s", pos, encoded_str->data);
+
+ /* build regular expression */
+ pattern_str = makeStringInfo();
+ appendStringInfoChar(pattern_str, '^');
+ forboth (lc1, winstate->patternVariableList, lc2, winstate->patternRegexpList)
+ {
+ char *vname = strVal(lfirst(lc1));
+ char *quantifier = strVal(lfirst(lc2));
+ char initial;
+
+ initial = pattern_initial(winstate, vname);
+ Assert(initial != 0);
+ appendStringInfoChar(pattern_str, initial);
+ if (quantifier[0])
+ appendStringInfoChar(pattern_str, quantifier[0]);
+ elog(DEBUG1, "vname: %s initial: %c quantifier: %s", vname, initial, quantifier);
+ }
+
+ elog(DEBUG2, "pos: " INT64_FORMAT " pattern: %s", pos, pattern_str->data);
+
+ /* look for matching pattern variable sequence */
+ num_matched_rows = search_str_set(pattern_str->data, str_set);
+ /*
+ * We are at the first row in the reduced frame. Save the number of
+ * matched rows as the number of rows in the reduced frame.
+ */
+ if (num_matched_rows <= 0)
+ {
+ /* no match */
+ register_reduced_frame_map(winstate, original_pos, RF_UNMATCHED);
+ }
+ else
+ {
+ int64 i;
+
+ register_reduced_frame_map(winstate, original_pos, RF_FRAME_HEAD);
+
+ for (i = original_pos + 1; i < original_pos + num_matched_rows; i++)
+ {
+ register_reduced_frame_map(winstate, i, RF_SKIPPED);
+ }
+ }
+
+ return;
+}
+
+/*
+ * Perform pattern matching using pattern against str_set. str_set is a set
+ * of StringInfo. Each StringInfo has a string comprising initial characters
+ * of pattern variables being true in a row. Returns the longest number of
+ * the matching rows.
+ */
+static
+int search_str_set(char *pattern, StringSet *str_set)
+{
+ int set_size; /* number of rows in the set */
+ int resultlen = 0;
+ int index;
+ StringSet *old_str_set, *new_str_set;
+ int new_str_size;
+
+ set_size = string_set_get_size(str_set);
+ new_str_set = string_set_init();
+
+ /*
+ * Generate all possible pattern variable name initials as a set of
+ * StringInfo named "new_str_set". For example, if we have two rows
+ * having "ab" (row 0) and "ac" (row 1) in the input str_set, new_str_set
+ * will have set of StringInfo "aa", "ac", "ba" and "bc" in the end.
+ */
+ for (index = 0; index < set_size; index++)
+ {
+ StringInfo str; /* search target row */
+ char *p;
+ int old_set_size;
+ int i;
+
+ if (index == 0)
+ {
+ /* copy variables in row 0 */
+ str = string_set_get(str_set, index);
+ p = str->data;
+
+ /*
+ * loop over each new pattern variable char in previous result
+ * char.
+ */
+ while (*p)
+ {
+ StringInfo new = makeStringInfo();
+
+ /* add pattern variable char */
+ appendStringInfoChar(new, *p);
+ /* add new one to string set */
+ string_set_add(new_str_set, new);
+ p++; /* next pattern variable */
+ }
+ }
+ else
+ {
+ old_str_set = new_str_set;
+ new_str_set = string_set_init();
+ str = string_set_get(str_set, index);
+ old_set_size = string_set_get_size(old_str_set);
+
+ /*
+ * loop over each rows in the previous result set.
+ */
+ for (i = 0; i < old_set_size ; i++)
+ {
+ StringInfo old = string_set_get(old_str_set, i);
+
+ p = str->data;
+
+ /*
+ * loop over each pattern variable char in the input set.
+ */
+ while (*p)
+ {
+ StringInfo new = makeStringInfo();
+
+ /* copy source string */
+ appendStringInfoString(new, old->data);
+ /* add pattern variable char */
+ appendStringInfoChar(new, *p);
+ /* add new one to string set */
+ string_set_add(new_str_set, new);
+ p++; /* next pattern variable */
+ }
+ }
+ /* we no long need old string set */
+ string_set_discard(old_str_set);
+ }
+ }
+
+ /*
+ * Perform pattern matching to find out the longest match.
+ */
+ new_str_size = string_set_get_size(new_str_set);
+
+ for (index = 0; index < new_str_size; index++)
+ {
+ int len;
+ StringInfo s;
+
+ s = string_set_get(new_str_set, index);
+ if (s == NULL)
+ continue; /* no data */
+
+ len = do_pattern_match(pattern, s->data);
+ if (len > resultlen)
+ {
+ /* remember the longest match */
+ resultlen = len;
+
+ /*
+ * If the size of result set is equal to the number of rows in the
+ * set, we are done because it's not possible that the number of
+ * matching rows exceeds the number of rows in the set.
+ */
+ if (resultlen >= set_size)
+ break;
+ }
+ }
+
+ /* we no long need new string set */
+ string_set_discard(new_str_set);
+
+ return resultlen;
+}
+
+/*
+ * do_pattern_match
+ * perform pattern match using pattern against encoded_str.
+ * returns matching number of rows if matching is succeeded.
+ * Otherwise returns 0.
+ */
+static
+int do_pattern_match(char *pattern, char *encoded_str)
+{
+ Datum d;
+ text *res;
+ char *substr;
+ int len = 0;
+
+
+ /*
+ * We first perform pattern matching using regexp_instr, then call
+ * textregexsubstr to get matched substring to know how long the
+ * matched string is. That is the number of rows in the reduced window
+ * frame. The reason why we can't call textregexsubstr in the first
+ * place is, it errors out if pattern does not match.
+ */
+ if (DatumGetInt32(DirectFunctionCall2Coll(regexp_instr, DEFAULT_COLLATION_OID,
+ PointerGetDatum(cstring_to_text(encoded_str)),
+ PointerGetDatum(cstring_to_text(pattern)))) > 0)
+ {
+ d = DirectFunctionCall2Coll(textregexsubstr,
+ DEFAULT_COLLATION_OID,
+ PointerGetDatum(cstring_to_text(encoded_str)),
+ PointerGetDatum(cstring_to_text(pattern)));
+ if (d != 0)
+ {
+ res = DatumGetTextPP(d);
+ substr = text_to_cstring(res);
+ len = strlen(substr);
+ }
+ }
+ return len;
+}
+
+
+/*
+ * Evaluate expression associated with PATTERN variable vname.
+ * relpos is relative row position in a frame (starting from 0).
+ * "quantifier" is the quatifier part of the PATTERN regular expression.
+ * Currently only '+' is allowed.
+ * result is out paramater representing the expression evaluation result
+ * is true of false.
+ * Return values are:
+ * >=0: the last match absolute row position
+ * other wise out of frame.
+ */
+static
+int64 evaluate_pattern(WindowObject winobj, int64 current_pos,
+ char *vname, StringInfo encoded_str, bool *result)
+{
+ WindowAggState *winstate = winobj->winstate;
+ ExprContext *econtext = winstate->ss.ps.ps_ExprContext;
+ ListCell *lc1, *lc2, *lc3;
+ ExprState *pat;
+ Datum eval_result;
+ bool out_of_frame = false;
+ bool isnull;
+
+ forthree (lc1, winstate->defineVariableList, lc2, winstate->defineClauseList, lc3, winstate->defineInitial)
+ {
+ char initial;
+ char *name = strVal(lfirst(lc1));
+
+ if (strcmp(vname, name))
+ continue;
+
+ initial = *(strVal(lfirst(lc3)));
+
+ /* set expression to evaluate */
+ pat = lfirst(lc2);
+
+ /* get current, previous and next tuples */
+ if (!get_slots(winobj, current_pos))
+ {
+ out_of_frame = true;
+ }
+ else
+ {
+ /* evaluate the expression */
+ eval_result = ExecEvalExpr(pat, econtext, &isnull);
+ if (isnull)
+ {
+ /* expression is NULL */
+ elog(DEBUG1, "expression for %s is NULL at row: " INT64_FORMAT, vname, current_pos);
+ *result = false;
+ }
+ else
+ {
+ if (!DatumGetBool(eval_result))
+ {
+ /* expression is false */
+ elog(DEBUG1, "expression for %s is false at row: " INT64_FORMAT, vname, current_pos);
+ *result = false;
+ }
+ else
+ {
+ /* expression is true */
+ elog(DEBUG1, "expression for %s is true at row: " INT64_FORMAT, vname, current_pos);
+ appendStringInfoChar(encoded_str, initial);
+ *result = true;
+ }
+ }
+ break;
+ }
+
+ if (out_of_frame)
+ {
+ *result = false;
+ return -1;
+ }
+ }
+ return current_pos;
+}
+
+/*
+ * Get current, previous and next tuples.
+ * Returns false if current row is out of partition/full frame.
+ */
+static
+bool get_slots(WindowObject winobj, int64 current_pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ TupleTableSlot *slot;
+ int ret;
+ ExprContext *econtext;
+
+ econtext = winstate->ss.ps.ps_ExprContext;
+
+ /* set up current row tuple slot */
+ slot = winstate->temp_slot_1;
+ if (!window_gettupleslot(winobj, current_pos, slot))
+ {
+ elog(DEBUG1, "current row is out of partition at:" INT64_FORMAT, current_pos);
+ return false;
+ }
+ ret = row_is_in_frame(winstate, current_pos, slot);
+ if (ret <= 0)
+ {
+ elog(DEBUG1, "current row is out of frame at: " INT64_FORMAT, current_pos);
+ return false;
+ }
+ econtext->ecxt_outertuple = slot;
+
+ /* for PREV */
+ if (current_pos > 0)
+ {
+ slot = winstate->prev_slot;
+ if (!window_gettupleslot(winobj, current_pos - 1, slot))
+ {
+ elog(DEBUG1, "previous row is out of partition at: " INT64_FORMAT, current_pos - 1);
+ econtext->ecxt_scantuple = winstate->null_slot;
+ }
+ else
+ {
+ ret = row_is_in_frame(winstate, current_pos - 1, slot);
+ if (ret <= 0)
+ {
+ elog(DEBUG1, "previous row is out of frame at: " INT64_FORMAT, current_pos - 1);
+ econtext->ecxt_scantuple = winstate->null_slot;
+ }
+ else
+ {
+ econtext->ecxt_scantuple = slot;
+ }
+ }
+ }
+ else
+ econtext->ecxt_scantuple = winstate->null_slot;
+
+ /* for NEXT */
+ slot = winstate->next_slot;
+ if (!window_gettupleslot(winobj, current_pos + 1, slot))
+ {
+ elog(DEBUG1, "next row is out of partiton at: " INT64_FORMAT, current_pos + 1);
+ econtext->ecxt_innertuple = winstate->null_slot;
+ }
+ else
+ {
+ ret = row_is_in_frame(winstate, current_pos + 1, slot);
+ if (ret <= 0)
+ {
+ elog(DEBUG1, "next row is out of frame at: " INT64_FORMAT, current_pos + 1);
+ econtext->ecxt_innertuple = winstate->null_slot;
+ }
+ else
+ econtext->ecxt_innertuple = slot;
+ }
+ return true;
+}
+
+/*
+ * Return pattern variable initial character
+ * matching with pattern variable name vname.
+ * If not found, return 0.
+ */
+static
+char pattern_initial(WindowAggState *winstate, char *vname)
+{
+ char initial;
+ char *name;
+ ListCell *lc1, *lc2;
+
+ forboth (lc1, winstate->defineVariableList, lc2, winstate->defineInitial)
+ {
+ name = strVal(lfirst(lc1)); /* DEFINE variable name */
+ initial = *(strVal(lfirst(lc2))); /* DEFINE variable initial */
+
+
+ if (!strcmp(name, vname))
+ return initial; /* found */
+ }
+ return 0;
+}
+
+/*
+ * string_set_init
+ * Create dynamic set of StringInfo.
+ */
+static
+StringSet *string_set_init(void)
+{
+/* Initial allocation size of str_set */
+#define STRING_SET_ALLOC_SIZE 1024
+
+ StringSet *string_set;
+ Size set_size;
+
+ string_set = palloc0(sizeof(StringSet));
+ string_set->set_index = 0;
+ set_size = STRING_SET_ALLOC_SIZE;
+ string_set->str_set = palloc(set_size * sizeof(StringInfo));
+ string_set->set_size = set_size;
+
+ return string_set;
+}
+
+/*
+ * Add StringInfo str to StringSet string_set.
+ */
+static
+void string_set_add(StringSet *string_set, StringInfo str)
+{
+ Size set_size;
+
+ set_size = string_set->set_size;
+ if (string_set->set_index >= set_size)
+ {
+ set_size *= 2;
+ string_set->str_set = repalloc(string_set->str_set,
+ set_size * sizeof(StringInfo));
+ string_set->set_size = set_size;
+ }
+
+ string_set->str_set[string_set->set_index++] = str;
+
+ return;
+}
+
+/*
+ * Returns StringInfo specified by index.
+ * If there's no data yet, returns NULL.
+ */
+static
+StringInfo string_set_get(StringSet *string_set, int index)
+{
+ /* no data? */
+ if (index == 0 && string_set->set_index == 0)
+ return NULL;
+
+ if (index < 0 ||index >= string_set->set_index)
+ elog(ERROR, "invalid index: %d", index);
+
+ return string_set->str_set[index];
+}
+
+/*
+ * Returns the size of StringSet.
+ */
+static
+int string_set_get_size(StringSet *string_set)
+{
+ return string_set->set_index;
+}
+
+/*
+ * Discard StringSet.
+ * All memory including StringSet itself is freed.
+ */
+static
+void string_set_discard(StringSet *string_set)
+{
+ int i;
+
+ for (i = 0; i < string_set->set_index; i++)
+ {
+ StringInfo str = string_set->str_set[i];
+ pfree(str->data);
+ pfree(str);
+ }
+ pfree(string_set);
+}
diff --git a/src/backend/utils/adt/windowfuncs.c b/src/backend/utils/adt/windowfuncs.c
index b87a624fb2..9ebcc7b5d2 100644
--- a/src/backend/utils/adt/windowfuncs.c
+++ b/src/backend/utils/adt/windowfuncs.c
@@ -13,6 +13,9 @@
*/
#include "postgres.h"
+#include "catalog/pg_collation_d.h"
+#include "executor/executor.h"
+#include "nodes/execnodes.h"
#include "nodes/supportnodes.h"
#include "utils/builtins.h"
#include "windowapi.h"
@@ -36,11 +39,19 @@ typedef struct
int64 remainder; /* (total rows) % (bucket num) */
} ntile_context;
+/*
+ * rpr process information.
+ * Used for AFTER MATCH SKIP PAST LAST ROW
+ */
+typedef struct SkipContext
+{
+ int64 pos; /* last row absolute position */
+} SkipContext;
+
static bool rank_up(WindowObject winobj);
static Datum leadlag_common(FunctionCallInfo fcinfo,
bool forward, bool withoffset, bool withdefault);
-
/*
* utility routine for *_rank functions.
*/
@@ -673,7 +684,7 @@ window_last_value(PG_FUNCTION_ARGS)
bool isnull;
result = WinGetFuncArgInFrame(winobj, 0,
- 0, WINDOW_SEEK_TAIL, true,
+ 0, WINDOW_SEEK_TAIL, false,
&isnull, NULL);
if (isnull)
PG_RETURN_NULL();
@@ -713,3 +724,25 @@ window_nth_value(PG_FUNCTION_ARGS)
PG_RETURN_DATUM(result);
}
+
+/*
+ * prev
+ * Dummy function to invoke RPR's navigation operator "PREV".
+ * This is *not* a window function.
+ */
+Datum
+window_prev(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_DATUM(PG_GETARG_DATUM(0));
+}
+
+/*
+ * next
+ * Dummy function to invoke RPR's navigation operation "NEXT".
+ * This is *not* a window function.
+ */
+Datum
+window_next(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_DATUM(PG_GETARG_DATUM(0));
+}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index f0b7b9cbd8..24d288336a 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10416,6 +10416,12 @@
{ oid => '3114', descr => 'fetch the Nth row value',
proname => 'nth_value', prokind => 'w', prorettype => 'anyelement',
proargtypes => 'anyelement int4', prosrc => 'window_nth_value' },
+{ oid => '6122', descr => 'previous value',
+ proname => 'prev', provolatile => 's', prorettype => 'anyelement',
+ proargtypes => 'anyelement', prosrc => 'window_prev' },
+{ oid => '6123', descr => 'next value',
+ proname => 'next', provolatile => 's', prorettype => 'anyelement',
+ proargtypes => 'anyelement', prosrc => 'window_next' },
# functions for range types
{ oid => '3832', descr => 'I/O',
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 108d69ba28..28d098b1cf 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -2470,6 +2470,11 @@ typedef enum WindowAggStatus
* tuples during spool */
} WindowAggStatus;
+#define RF_NOT_DETERMINED 0
+#define RF_FRAME_HEAD 1
+#define RF_SKIPPED 2
+#define RF_UNMATCHED 3
+
typedef struct WindowAggState
{
ScanState ss; /* its first field is NodeTag */
@@ -2518,6 +2523,15 @@ typedef struct WindowAggState
int64 groupheadpos; /* current row's peer group head position */
int64 grouptailpos; /* " " " " tail position (group end+1) */
+ /* these fields are used in Row pattern recognition: */
+ RPSkipTo rpSkipTo; /* Row Pattern Skip To type */
+ List *patternVariableList; /* list of row pattern variables names (list of String) */
+ List *patternRegexpList; /* list of row pattern regular expressions ('+' or ''. list of String) */
+ List *defineVariableList; /* list of row pattern definition variables (list of String) */
+ List *defineClauseList; /* expression for row pattern definition
+ * search conditions ExprState list */
+ List *defineInitial; /* list of row pattern definition variable initials (list of String) */
+
MemoryContext partcontext; /* context for partition-lifespan data */
MemoryContext aggcontext; /* shared context for aggregate working data */
MemoryContext curaggcontext; /* current aggregate's working data */
@@ -2554,6 +2568,18 @@ typedef struct WindowAggState
TupleTableSlot *agg_row_slot;
TupleTableSlot *temp_slot_1;
TupleTableSlot *temp_slot_2;
+
+ /* temporary slots for RPR */
+ TupleTableSlot *prev_slot; /* PREV row navigation operator */
+ TupleTableSlot *next_slot; /* NEXT row navigation operator */
+ TupleTableSlot *null_slot; /* all NULL slot */
+
+ /*
+ * Each byte corresponds to a row positioned at absolute its pos in
+ * partition. See above definition for RF_*
+ */
+ char *reduced_frame_map;
+ int64 alloc_sz; /* size of the map */
} WindowAggState;
/* ----------------
--
2.25.1
----Next_Part(Wed_Oct__4_15_03_28_2023_821)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v9-0005-Row-pattern-recognition-patch-docs.patch"
^ permalink raw reply [nested|flat] 8+ messages in thread
* New commitfest app release on March 4th
@ 2025-02-17 09:17 Jelte Fennema-Nio <[email protected]>
2025-03-04 00:42 ` Re: New commitfest app release on March 4th Jelte Fennema-Nio <[email protected]>
0 siblings, 1 reply; 8+ messages in thread
From: Jelte Fennema-Nio @ 2025-02-17 09:17 UTC (permalink / raw)
To: PostgreSQL Hackers <[email protected]>; Akshat Jaimini <[email protected]>
The next release of the commitfest app will take place March 4th.
The user facing changes are:
1. The form for creating a commitfest entry now contains all the
commitfest entry fields, instead of redirecting to the entries edit
page after creating it.
2. When creating a commitfest entry, the author field is defaulted to yourself.
3. When creating a commitfest entry, the description (aka title) is
autofilled based on the email you select from the "Find thread" popup.
4. Add reverse sorting on the commitfest page (done by clicking twice
on a column). Implemented by Akshat Jaimini.
5. Disallow assigning others as reviewers/committers on open
commitfest entries. This is to avoid having people mess with
"bookmarks" of others. (On closed entries it's still allowed)
Please test out the current staging website[1] to give some feedback.
HTTP auth user and password are both pgtest.
[1]: https://commitfest-test.postgresql.org/
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: New commitfest app release on March 4th
2025-02-17 09:17 New commitfest app release on March 4th Jelte Fennema-Nio <[email protected]>
@ 2025-03-04 00:42 ` Jelte Fennema-Nio <[email protected]>
0 siblings, 0 replies; 8+ messages in thread
From: Jelte Fennema-Nio @ 2025-03-04 00:42 UTC (permalink / raw)
To: PostgreSQL Hackers <[email protected]>; Akshat Jaimini <[email protected]>
On Mon, 17 Feb 2025 at 10:17, Jelte Fennema-Nio <[email protected]> wrote:
>
> The next release of the commitfest app will take place March 4th.
>
> The user facing changes are:
> 1. The form for creating a commitfest entry now contains all the
> commitfest entry fields, instead of redirecting to the entries edit
> page after creating it.
> 2. When creating a commitfest entry, the author field is defaulted to yourself.
> 3. When creating a commitfest entry, the description (aka title) is
> autofilled based on the email you select from the "Find thread" popup.
> 4. Add reverse sorting on the commitfest page (done by clicking twice
> on a column). Implemented by Akshat Jaimini.
> 5. Disallow assigning others as reviewers/committers on open
> commitfest entries. This is to avoid having people mess with
> "bookmarks" of others. (On closed entries it's still allowed)
This is deployed now. Please let me know if there are any issues.
^ permalink raw reply [nested|flat] 8+ messages in thread
end of thread, other threads:[~2025-03-04 00:42 UTC | newest]
Thread overview: 8+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2018-09-01 15:48 [PATCH 1/2] multivariate MCV lists Tomas Vondra <[email protected]>
2018-12-26 19:59 [PATCH 1/2] multivariate MCV lists Tomas Vondra <[email protected]>
2023-03-01 04:56 [PATCH v5 12/14] WIP: Don't initialize page in {vm,fsm}_extend(), not needed Andres Freund <[email protected]>
2023-03-01 04:56 [PATCH v6 15/17] WIP: Don't initialize page in {vm,fsm}_extend(), not needed Andres Freund <[email protected]>
2023-03-01 04:56 [PATCH v4 14/15] WIP: Don't initialize page in {vm,fsm}_extend(), not needed Andres Freund <[email protected]>
2023-10-04 05:51 [PATCH v9 4/7] Row pattern recognition patch (executor). Tatsuo Ishii <[email protected]>
2025-02-17 09:17 New commitfest app release on March 4th Jelte Fennema-Nio <[email protected]>
2025-03-04 00:42 ` Re: New commitfest app release on March 4th Jelte Fennema-Nio <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox