pg.ddx.io pgsql-hackers@postgresql.org mailing list archive
help / color / mirror / Atom feedRe: multivariate statistics v14
58+ messages / 10 participants
[nested] [flat]
* Re: multivariate statistics v14
@ 2016-03-09 15:02 Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 1 reply; 58+ messages in thread
From: Tomas Vondra @ 2016-03-09 15:02 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: Jeff Janes <jeff.janes@gmail.com>; pgsql-hackers
Hi,
thanks for the feedback. Attached is v14 of the patch series, fixing
most of the points you've raised.
On Wed, 2016-03-09 at 09:22 -0300, Alvaro Herrera wrote:
> Hi,
>
> I gave a very quick skim to patch 0002. Not a real review yet. But
> there are a few trivial points to fix:
>
> * You still have empty sections in the SGML docs (such as the EXAMPLES).
> I suppose the syntax is now firm enough that we can get some. (I looked
> at the other patches to see whether it was filled in, but couldn't find
> any additional text there.)
Yes, that's one of the items I plan to work on next. Until now the
regression tests were a sufficient source of examples, but it's time to
do the SGML piece.
>
> * check_object_ownership() needs to be filled in
Done.
I've added pg_statistics_ownercheck, which also required adding OID of
the owner to the catalog. Initially the plan was to use the same owner
as for the table, but now that we've switched to CREATE STATISTICS
partially because it will allow multi-table stats, that does not make
sense (multiple tables with different owners).
This probably means we also need an 'ALTER STATISTICS ... OWNER TO'
command, which does not exist at this point.
>
> * Since you're adding a new object type, please add a case to cover it
> in the object_address.sql pg_regress test.
Done.
Apparently there was a bunch of missing pieces in objectaddress.c, so
this adds them too.
>
> * in analyze.c (and elsewhere), please put new #include lines sorted.
Done.
I've also significantly reduced the excessive list of includes in
statscmds.c. I expect the headers to require a bit more love, especially
in the subsequent patches (MCV, histograms etc.).
>
> * I think the AT_PASS_ADD_STATS is a leftover which should be removed.
Yeah. Now that we've invented CREATE TABLE, all the changes to
tablecmds.c were just unnecessary leftovers. Removed.
>
> * The XXX comment in get_relation_info should probably be handled
> differently (namely, in a way that makes the syscache not contain OIDs
> of dropped stats)
I believe that was actually an obsolete comment. Removed.
>
> * The README.dependencies has a lot of TODOs. Do we need to get them
> done during the first cut? If not, I suggest creating a new section
> "Future work" in the file.
Right. Most of those TODOs are future work, or rather ideas (more or
less crazy). The one thing I definitely want to address now is support
for dependencies with multiple columns on the left side, because that
requires changes to serialized format. I might also look at handling IS
NULL clauses, but that may wait.
>
> * Please put the common.h header in src/include. Make sure not to
> include "postgres.h" in it -- our policy is that postgres.h goes at the
> top of every .c file and never in any .h file. Also please find a
> better name for it; even mvstats_common.h would be a lot more
> convincing. However:
>
> * ISTM that the code in common.c properly belongs in
> src/backend/catalog/pg_mvstats.c instead (or more properly
> catalog/pg_mv_statistics.c), which probably means the common.h file
> should be named something else; perhaps some of it could become
> pg_mv_statistic_fn.h, while the rest continues to be
> src/include/utils/mvstats_common.h? Not sure.
Hmmm, not sure either. The idea was that the "common.h" is pretty much
just a private header with stuff that's not very useful anywhere else.
No changes here, for now.
>
> * The version check in psql/describe.c uses 90500; should probably be
> updated to 90600.
Fixed.
>
> * _copyCreateStatsStmt is missing if_not_exists
Fixed.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[text/x-patch] 0001-teach-pull_-varno-varattno-_walker-about-RestrictInf.patch (1.4K, ../../1457535779.24545.71.camel@2ndquadrant.com/2-0001-teach-pull_-varno-varattno-_walker-about-RestrictInf.patch)
download | inline diff:
From 5c28e5ca8feb2c2010d98bc69de952355bd6f3a5 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@pgaddict.com>
Date: Tue, 28 Apr 2015 19:56:33 +0200
Subject: [PATCH 1/9] teach pull_(varno|varattno)_walker about RestrictInfo
otherwise pull_varnos fails when processing OR clauses
---
src/backend/optimizer/util/var.c | 16 ++++++++++++++++
1 file changed, 16 insertions(+)
diff --git a/src/backend/optimizer/util/var.c b/src/backend/optimizer/util/var.c
index dff52c4..80d01bd 100644
--- a/src/backend/optimizer/util/var.c
+++ b/src/backend/optimizer/util/var.c
@@ -197,6 +197,13 @@ pull_varnos_walker(Node *node, pull_varnos_context *context)
context->sublevels_up--;
return result;
}
+ if (IsA(node, RestrictInfo))
+ {
+ RestrictInfo *rinfo = (RestrictInfo*)node;
+ context->varnos = bms_add_members(context->varnos,
+ rinfo->clause_relids);
+ return false;
+ }
return expression_tree_walker(node, pull_varnos_walker,
(void *) context);
}
@@ -245,6 +252,15 @@ pull_varattnos_walker(Node *node, pull_varattnos_context *context)
return false;
}
+ if (IsA(node, RestrictInfo))
+ {
+ RestrictInfo *rinfo = (RestrictInfo *)node;
+
+ return expression_tree_walker((Node*)rinfo->clause,
+ pull_varattnos_walker,
+ (void*) context);
+ }
+
/* Should not find an unplanned subquery */
Assert(!IsA(node, Query));
--
2.1.0
[text/x-patch] 0002-shared-infrastructure-and-functional-dependencies.patch (110.7K, ../../1457535779.24545.71.camel@2ndquadrant.com/3-0002-shared-infrastructure-and-functional-dependencies.patch)
download | inline diff:
From 1c42a02189088ba194e30f5878bb67bc61953a11 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Sun, 11 Jan 2015 19:51:48 +0100
Subject: [PATCH 2/9] shared infrastructure and functional dependencies
Basic infrastructure shared by all kinds of multivariate stats, most
importantly:
- adds a new system catalog (pg_mv_statistic)
- CREATE STATISTICS name ON table (columns) WITH (options)
- DROP STATISTICS name
- implementation of functional dependencies (the simplest type of
multivariate statistics)
- building functional dependencies in ANALYZE
- updates regression tests (new catalog etc.)
This does not include any changes to the optimizer, i.e. it does not
influence the query planning (subject to follow-up patches).
The current implementation requires a valid 'ltopr' for the columns, so
that we can sort the sample rows in various ways, both in this patch
and other kinds of statistics. Maybe this restriction could be relaxed
in the future, requiring just 'eqopr' in case of stats not sorting the
data (e.g. functional dependencies and MCV lists).
Maybe some of the stats (functional dependencies and MCV list with
limited functionality) might be made to work with hashes of the values,
which is sufficient for equality comparisons. But the queries would
require the equality operator anyway, so it's not really a weaker
requirement. The hashes might reduce space requirements, though.
The algorithm detecting the dependencies is rather simple and probably
needs improvements, so that it detects more complicated dependencies,
and also validation of the math.
The name 'functional dependencies' is more correct (than 'association
rules') as it's exactly the name used in relational theory (esp. Normal
Forms) for tracking column-level dependencies.
The multivariate statistics are automatically removed in two situations
(a) after a DROP TABLE (obviously)
(b) after ALTER TABLE ... DROP COLUMN, if the statistics would be
defined on less than 2 columns (remaining)
If there are more at least two remaining columns, we keep the
statistics but perform cleanup on the next ANALYZE. The dropped columns
are removed from stakeys, and the new statistics is built on the
smaller set.
We can't do this at DROP COLUMN, because that'd leave us with invalid
statistics, or we'd have to throw it away although we can still use it.
This lazy approach lets us use the statistics although some of the
columns are dead.
This also adds a simple list of statistics to \d in psql.
This means the statistics are created within a schema by using a
qualified name (or using the default schema)
CREATE STATISTICS schema.statistics ON ...
and then dropped by specifying qualified name
DROP STATISTICS schema.statistics
or searching through search_path (just like with other objects).
This also gets rid of the "(opt_)stats_name" definitions in gram.y and
instead replaces them with just "opt_any_name", although the optional
case is not really handled currently - there's no generated name yet
(so either we should drop it or implement it).
I'm not entirely sure making statistics schema-specific is that a great
idea. Maybe it should be "global", but that does not seem right (e.g.
it makes multi-tenant systems based on schemas more difficult to
manage, because tenants would interact).
---
doc/src/sgml/ref/allfiles.sgml | 2 +
doc/src/sgml/ref/create_statistics.sgml | 174 ++++++++++
doc/src/sgml/ref/drop_statistics.sgml | 90 ++++++
doc/src/sgml/reference.sgml | 2 +
src/backend/catalog/Makefile | 1 +
src/backend/catalog/aclchk.c | 27 ++
src/backend/catalog/dependency.c | 11 +-
src/backend/catalog/heap.c | 102 ++++++
src/backend/catalog/namespace.c | 51 +++
src/backend/catalog/objectaddress.c | 54 ++++
src/backend/catalog/system_views.sql | 11 +
src/backend/commands/Makefile | 6 +-
src/backend/commands/analyze.c | 21 ++
src/backend/commands/dropcmds.c | 4 +
src/backend/commands/event_trigger.c | 3 +
src/backend/commands/statscmds.c | 266 ++++++++++++++++
src/backend/nodes/copyfuncs.c | 17 +
src/backend/nodes/outfuncs.c | 18 ++
src/backend/optimizer/util/plancat.c | 59 ++++
src/backend/parser/gram.y | 34 +-
src/backend/tcop/utility.c | 11 +
src/backend/utils/Makefile | 2 +-
src/backend/utils/cache/relcache.c | 59 ++++
src/backend/utils/cache/syscache.c | 23 ++
src/backend/utils/mvstats/Makefile | 17 +
src/backend/utils/mvstats/README.dependencies | 222 +++++++++++++
src/backend/utils/mvstats/common.c | 356 +++++++++++++++++++++
src/backend/utils/mvstats/common.h | 75 +++++
src/backend/utils/mvstats/dependencies.c | 437 ++++++++++++++++++++++++++
src/bin/psql/describe.c | 44 +++
src/include/catalog/dependency.h | 5 +-
src/include/catalog/heap.h | 1 +
src/include/catalog/indexing.h | 7 +
src/include/catalog/namespace.h | 2 +
src/include/catalog/pg_mv_statistic.h | 75 +++++
src/include/catalog/pg_proc.h | 5 +
src/include/catalog/toasting.h | 1 +
src/include/commands/defrem.h | 4 +
src/include/nodes/nodes.h | 2 +
src/include/nodes/parsenodes.h | 12 +
src/include/nodes/relation.h | 28 ++
src/include/utils/acl.h | 1 +
src/include/utils/mvstats.h | 70 +++++
src/include/utils/rel.h | 4 +
src/include/utils/relcache.h | 1 +
src/include/utils/syscache.h | 2 +
src/test/regress/expected/object_address.out | 7 +-
src/test/regress/expected/rules.out | 9 +
src/test/regress/expected/sanity_check.out | 1 +
src/test/regress/sql/object_address.sql | 4 +-
50 files changed, 2429 insertions(+), 11 deletions(-)
create mode 100644 doc/src/sgml/ref/create_statistics.sgml
create mode 100644 doc/src/sgml/ref/drop_statistics.sgml
create mode 100644 src/backend/commands/statscmds.c
create mode 100644 src/backend/utils/mvstats/Makefile
create mode 100644 src/backend/utils/mvstats/README.dependencies
create mode 100644 src/backend/utils/mvstats/common.c
create mode 100644 src/backend/utils/mvstats/common.h
create mode 100644 src/backend/utils/mvstats/dependencies.c
create mode 100644 src/include/catalog/pg_mv_statistic.h
create mode 100644 src/include/utils/mvstats.h
diff --git a/doc/src/sgml/ref/allfiles.sgml b/doc/src/sgml/ref/allfiles.sgml
index bf95453..c0f7653 100644
--- a/doc/src/sgml/ref/allfiles.sgml
+++ b/doc/src/sgml/ref/allfiles.sgml
@@ -76,6 +76,7 @@ Complete list of usable sgml source files in this directory.
<!ENTITY createSchema SYSTEM "create_schema.sgml">
<!ENTITY createSequence SYSTEM "create_sequence.sgml">
<!ENTITY createServer SYSTEM "create_server.sgml">
+<!ENTITY createStatistics SYSTEM "create_statistics.sgml">
<!ENTITY createTable SYSTEM "create_table.sgml">
<!ENTITY createTableAs SYSTEM "create_table_as.sgml">
<!ENTITY createTableSpace SYSTEM "create_tablespace.sgml">
@@ -119,6 +120,7 @@ Complete list of usable sgml source files in this directory.
<!ENTITY dropSchema SYSTEM "drop_schema.sgml">
<!ENTITY dropSequence SYSTEM "drop_sequence.sgml">
<!ENTITY dropServer SYSTEM "drop_server.sgml">
+<!ENTITY dropStatistics SYSTEM "drop_statistics.sgml">
<!ENTITY dropTable SYSTEM "drop_table.sgml">
<!ENTITY dropTableSpace SYSTEM "drop_tablespace.sgml">
<!ENTITY dropTransform SYSTEM "drop_transform.sgml">
diff --git a/doc/src/sgml/ref/create_statistics.sgml b/doc/src/sgml/ref/create_statistics.sgml
new file mode 100644
index 0000000..a86eae3
--- /dev/null
+++ b/doc/src/sgml/ref/create_statistics.sgml
@@ -0,0 +1,174 @@
+<!--
+doc/src/sgml/ref/create_statistics.sgml
+PostgreSQL documentation
+-->
+
+<refentry id="SQL-CREATESTATISTICS">
+ <indexterm zone="sql-createstatistics">
+ <primary>CREATE STATISTICS</primary>
+ </indexterm>
+
+ <refmeta>
+ <refentrytitle>CREATE STATISTICS</refentrytitle>
+ <manvolnum>7</manvolnum>
+ <refmiscinfo>SQL - Language Statements</refmiscinfo>
+ </refmeta>
+
+ <refnamediv>
+ <refname>CREATE STATISTICS</refname>
+ <refpurpose>define a new statistics</refpurpose>
+ </refnamediv>
+
+ <refsynopsisdiv>
+<synopsis>
+CREATE STATISTICS [ IF NOT EXISTS ] <replaceable class="PARAMETER">statistics_name</replaceable> ON <replaceable class="PARAMETER">table_name</replaceable> ( [
+ { <replaceable class="PARAMETER">column_name</replaceable> } ] [, ...])
+[ WITH ( <replaceable class="PARAMETER">statistics_parameter</replaceable> [= <replaceable class="PARAMETER">value</replaceable>] [, ... ] )
+</synopsis>
+
+ </refsynopsisdiv>
+
+ <refsect1 id="SQL-CREATESTATISTICS-description">
+ <title>Description</title>
+
+ <para>
+ <command>CREATE STATISTICS</command> will create a new multivariate
+ statistics on the table. The statistics will be created in the in the
+ current database. The statistics will be owned by the user issuing
+ the command.
+ </para>
+
+ <para>
+ If a schema name is given (for example, <literal>CREATE STATISTICS
+ myschema.mystat ...</>) then the statistics is created in the specified
+ schema. Otherwise it is created in the current schema. The name of
+ the table must be distinct from the name of any other statistics in the
+ same schema.
+ </para>
+
+ <para>
+ To be able to create a table, you must have <literal>USAGE</literal>
+ privilege on all column types or the type in the <literal>OF</literal>
+ clause, respectively.
+ </para>
+ </refsect1>
+
+ <refsect1>
+ <title>Parameters</title>
+
+ <variablelist>
+
+ <varlistentry>
+ <term><literal>IF NOT EXISTS</></term>
+ <listitem>
+ <para>
+ Do not throw an error if a statistics with the same name already exists.
+ A notice is issued in this case. Note that there is no guarantee that
+ the existing statistics is anything like the one that would have been
+ created.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><replaceable class="PARAMETER">statistics_name</replaceable></term>
+ <listitem>
+ <para>
+ The name (optionally schema-qualified) of the statistics to be created.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><replaceable class="PARAMETER">table_name</replaceable></term>
+ <listitem>
+ <para>
+ The name (optionally schema-qualified) of the table the statistics should
+ be created on.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><replaceable class="PARAMETER">column_name</replaceable></term>
+ <listitem>
+ <para>
+ The name of a column to be included in the statistics.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>WITH ( <replaceable class="PARAMETER">statistics_parameter</replaceable> [= <replaceable class="PARAMETER">value</replaceable>] [, ... ] )</literal></term>
+ <listitem>
+ <para>
+ ...
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ <refsect2 id="SQL-CREATESTATISTICS-parameters">
+ <title id="SQL-CREATESTATISTICS-parameters-title">Statistics Parameters</title>
+
+ <indexterm zone="sql-createstatistics-parameters">
+ <primary>statistics parameters</primary>
+ </indexterm>
+
+ <para>
+ The <literal>WITH</> clause can specify <firstterm>statistics parameters</>
+ for statistics. The currently available parameters are listed below.
+ </para>
+
+ <variablelist>
+
+ <varlistentry>
+ <term><literal>dependencies</> (<type>boolean</>)</term>
+ <listitem>
+ <para>
+ Enables functional dependencies for the statistics.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </refsect2>
+ </refsect1>
+
+ <refsect1 id="SQL-CREATESTATISTICS-notes">
+ <title>Notes</title>
+
+ <para>
+ ...
+ </para>
+
+ </refsect1>
+
+
+ <refsect1 id="SQL-CREATESTATISTICS-examples">
+ <title>Examples</title>
+
+ <para>
+ ...
+ </para>
+
+ </refsect1>
+
+ <refsect1>
+ <title>Compatibility</title>
+
+ <para>
+ There's no <command>CREATE STATISTICS</command> command in the SQL standard.
+ </para>
+ </refsect1>
+
+ <refsect1>
+ <title>See Also</title>
+
+ <simplelist type="inline">
+ <member><xref linkend="sql-dropstatistics"></member>
+ </simplelist>
+ </refsect1>
+</refentry>
diff --git a/doc/src/sgml/ref/drop_statistics.sgml b/doc/src/sgml/ref/drop_statistics.sgml
new file mode 100644
index 0000000..4cc0b70
--- /dev/null
+++ b/doc/src/sgml/ref/drop_statistics.sgml
@@ -0,0 +1,90 @@
+<!--
+doc/src/sgml/ref/drop_statistics.sgml
+PostgreSQL documentation
+-->
+
+<refentry id="SQL-DROPSTATISTICS">
+ <indexterm zone="sql-dropstatistics">
+ <primary>DROP STATISTICS</primary>
+ </indexterm>
+
+ <refmeta>
+ <refentrytitle>DROP STATISTICS</refentrytitle>
+ <manvolnum>7</manvolnum>
+ <refmiscinfo>SQL - Language Statements</refmiscinfo>
+ </refmeta>
+
+ <refnamediv>
+ <refname>DROP STATISTICS</refname>
+ <refpurpose>remove a statistics</refpurpose>
+ </refnamediv>
+
+ <refsynopsisdiv>
+<synopsis>
+DROP STATISTICS [ IF EXISTS ] <replaceable class="PARAMETER">name</replaceable> [, ...]
+</synopsis>
+ </refsynopsisdiv>
+
+ <refsect1>
+ <title>Description</title>
+
+ <para>
+ <command>DROP STATISTICS</command> removes statistics from the database.
+ Only the statistics owner, the schema owner, and superuser can drop a
+ statistics.
+ </para>
+
+ </refsect1>
+
+ <refsect1>
+ <title>Parameters</title>
+
+ <variablelist>
+ <varlistentry>
+ <term><literal>IF EXISTS</literal></term>
+ <listitem>
+ <para>
+ Do not throw an error if the statistics does not exist. A notice is
+ issued in this case.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><replaceable class="PARAMETER">name</replaceable></term>
+ <listitem>
+ <para>
+ The name (optionally schema-qualified) of the statistics to drop.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </refsect1>
+
+ <refsect1>
+ <title>Examples</title>
+
+ <para>
+ ...
+ </para>
+
+ </refsect1>
+
+ <refsect1>
+ <title>Compatibility</title>
+
+ <para>
+ There's no <command>DROP STATISTICS</command> command in the SQL standard.
+ </para>
+ </refsect1>
+
+ <refsect1>
+ <title>See Also</title>
+
+ <simplelist type="inline">
+ <member><xref linkend="sql-createstatistics"></member>
+ </simplelist>
+ </refsect1>
+
+</refentry>
diff --git a/doc/src/sgml/reference.sgml b/doc/src/sgml/reference.sgml
index 03020df..2b07b2d 100644
--- a/doc/src/sgml/reference.sgml
+++ b/doc/src/sgml/reference.sgml
@@ -104,6 +104,7 @@
&createSchema;
&createSequence;
&createServer;
+ &createStatistics;
&createTable;
&createTableAs;
&createTableSpace;
@@ -147,6 +148,7 @@
&dropSchema;
&dropSequence;
&dropServer;
+ &dropStatistics;
&dropTable;
&dropTableSpace;
&dropTSConfig;
diff --git a/src/backend/catalog/Makefile b/src/backend/catalog/Makefile
index 25130ec..058b8a9 100644
--- a/src/backend/catalog/Makefile
+++ b/src/backend/catalog/Makefile
@@ -32,6 +32,7 @@ POSTGRES_BKI_SRCS = $(addprefix $(top_srcdir)/src/include/catalog/,\
pg_attrdef.h pg_constraint.h pg_inherits.h pg_index.h pg_operator.h \
pg_opfamily.h pg_opclass.h pg_am.h pg_amop.h pg_amproc.h \
pg_language.h pg_largeobject_metadata.h pg_largeobject.h pg_aggregate.h \
+ pg_mv_statistic.h \
pg_statistic.h pg_rewrite.h pg_trigger.h pg_event_trigger.h pg_description.h \
pg_cast.h pg_enum.h pg_namespace.h pg_conversion.h pg_depend.h \
pg_database.h pg_db_role_setting.h pg_tablespace.h pg_pltemplate.h \
diff --git a/src/backend/catalog/aclchk.c b/src/backend/catalog/aclchk.c
index 0f3bc07..e21aacd 100644
--- a/src/backend/catalog/aclchk.c
+++ b/src/backend/catalog/aclchk.c
@@ -38,6 +38,7 @@
#include "catalog/pg_language.h"
#include "catalog/pg_largeobject.h"
#include "catalog/pg_largeobject_metadata.h"
+#include "catalog/pg_mv_statistic.h"
#include "catalog/pg_namespace.h"
#include "catalog/pg_opclass.h"
#include "catalog/pg_operator.h"
@@ -5021,6 +5022,32 @@ pg_extension_ownercheck(Oid ext_oid, Oid roleid)
}
/*
+ * Ownership check for a multivariate statistics (specified by OID).
+ */
+bool
+pg_statistics_ownercheck(Oid stat_oid, Oid roleid)
+{
+ HeapTuple tuple;
+ Oid ownerId;
+
+ /* Superusers bypass all permission checking. */
+ if (superuser_arg(roleid))
+ return true;
+
+ tuple = SearchSysCache1(MVSTATOID, ObjectIdGetDatum(stat_oid));
+ if (!HeapTupleIsValid(tuple))
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("statistics with OID %u does not exist", stat_oid)));
+
+ ownerId = ((Form_pg_mv_statistic) GETSTRUCT(tuple))->staowner;
+
+ ReleaseSysCache(tuple);
+
+ return has_privs_of_role(roleid, ownerId);
+}
+
+/*
* Check whether specified role has CREATEROLE privilege (or is a superuser)
*
* Note: roles do not have owners per se; instead we use this test in
diff --git a/src/backend/catalog/dependency.c b/src/backend/catalog/dependency.c
index c48e37b..8200454 100644
--- a/src/backend/catalog/dependency.c
+++ b/src/backend/catalog/dependency.c
@@ -40,6 +40,7 @@
#include "catalog/pg_foreign_server.h"
#include "catalog/pg_language.h"
#include "catalog/pg_largeobject.h"
+#include "catalog/pg_mv_statistic.h"
#include "catalog/pg_namespace.h"
#include "catalog/pg_opclass.h"
#include "catalog/pg_operator.h"
@@ -160,7 +161,8 @@ static const Oid object_classes[] = {
ExtensionRelationId, /* OCLASS_EXTENSION */
EventTriggerRelationId, /* OCLASS_EVENT_TRIGGER */
PolicyRelationId, /* OCLASS_POLICY */
- TransformRelationId /* OCLASS_TRANSFORM */
+ TransformRelationId, /* OCLASS_TRANSFORM */
+ MvStatisticRelationId /* OCLASS_STATISTICS */
};
@@ -1272,6 +1274,10 @@ doDeletion(const ObjectAddress *object, int flags)
DropTransformById(object->objectId);
break;
+ case OCLASS_STATISTICS:
+ RemoveStatisticsById(object->objectId);
+ break;
+
default:
elog(ERROR, "unrecognized object class: %u",
object->classId);
@@ -2415,6 +2421,9 @@ getObjectClass(const ObjectAddress *object)
case TransformRelationId:
return OCLASS_TRANSFORM;
+
+ case MvStatisticRelationId:
+ return OCLASS_STATISTICS;
}
/* shouldn't get here */
diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c
index 6a4a9d9..e7d9aaa 100644
--- a/src/backend/catalog/heap.c
+++ b/src/backend/catalog/heap.c
@@ -47,6 +47,7 @@
#include "catalog/pg_constraint_fn.h"
#include "catalog/pg_foreign_table.h"
#include "catalog/pg_inherits.h"
+#include "catalog/pg_mv_statistic.h"
#include "catalog/pg_namespace.h"
#include "catalog/pg_statistic.h"
#include "catalog/pg_tablespace.h"
@@ -1613,7 +1614,10 @@ RemoveAttributeById(Oid relid, AttrNumber attnum)
heap_close(attr_rel, RowExclusiveLock);
if (attnum > 0)
+ {
RemoveStatistics(relid, attnum);
+ RemoveMVStatistics(relid, attnum);
+ }
relation_close(rel, NoLock);
}
@@ -1841,6 +1845,11 @@ heap_drop_with_catalog(Oid relid)
RemoveStatistics(relid, 0);
/*
+ * delete multi-variate statistics
+ */
+ RemoveMVStatistics(relid, 0);
+
+ /*
* delete attribute tuples
*/
DeleteAttributeTuples(relid);
@@ -2696,6 +2705,99 @@ RemoveStatistics(Oid relid, AttrNumber attnum)
/*
+ * RemoveMVStatistics --- remove entries in pg_mv_statistic for a rel
+ *
+ * If attnum is zero, remove all entries for rel; else remove only the one(s)
+ * for that column.
+ */
+void
+RemoveMVStatistics(Oid relid, AttrNumber attnum)
+{
+ Relation pgmvstatistic;
+ TupleDesc tupdesc = NULL;
+ SysScanDesc scan;
+ ScanKeyData key;
+ HeapTuple tuple;
+
+ /*
+ * When dropping a column, we'll drop statistics with a single
+ * remaining (undropped column). To do that, we need the tuple
+ * descriptor.
+ *
+ * We already have the relation locked (as we're running ALTER
+ * TABLE ... DROP COLUMN), so we'll just get the descriptor here.
+ */
+ if (attnum != 0)
+ {
+ Relation rel = relation_open(relid, NoLock);
+
+ /* multivariate stats are supported on tables and matviews */
+ if (rel->rd_rel->relkind == RELKIND_RELATION ||
+ rel->rd_rel->relkind == RELKIND_MATVIEW)
+ tupdesc = RelationGetDescr(rel);
+
+ relation_close(rel, NoLock);
+ }
+
+ if (tupdesc == NULL)
+ return;
+
+ pgmvstatistic = heap_open(MvStatisticRelationId, RowExclusiveLock);
+
+ ScanKeyInit(&key,
+ Anum_pg_mv_statistic_starelid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(relid));
+
+ scan = systable_beginscan(pgmvstatistic,
+ MvStatisticRelidIndexId,
+ true, NULL, 1, &key);
+
+ /* we must loop even when attnum != 0, in case of inherited stats */
+ while (HeapTupleIsValid(tuple = systable_getnext(scan)))
+ {
+ bool delete = true;
+
+ if (attnum != 0)
+ {
+ Datum adatum;
+ bool isnull;
+ int i;
+ int ncolumns = 0;
+ ArrayType *arr;
+ int16 *attnums;
+
+ /* get the columns */
+ adatum = SysCacheGetAttr(MVSTATOID, tuple,
+ Anum_pg_mv_statistic_stakeys, &isnull);
+ Assert(!isnull);
+
+ arr = DatumGetArrayTypeP(adatum);
+ attnums = (int16*)ARR_DATA_PTR(arr);
+
+ for (i = 0; i < ARR_DIMS(arr)[0]; i++)
+ {
+ /* count the column unless it's has been / is being dropped */
+ if ((! tupdesc->attrs[attnums[i]-1]->attisdropped) &&
+ (attnums[i] != attnum))
+ ncolumns += 1;
+ }
+
+ /* delete if there are less than two attributes */
+ delete = (ncolumns < 2);
+ }
+
+ if (delete)
+ simple_heap_delete(pgmvstatistic, &tuple->t_self);
+ }
+
+ systable_endscan(scan);
+
+ heap_close(pgmvstatistic, RowExclusiveLock);
+}
+
+
+/*
* RelationTruncateIndexes - truncate all indexes associated
* with the heap relation to zero tuples.
*
diff --git a/src/backend/catalog/namespace.c b/src/backend/catalog/namespace.c
index 446b2ac..dfd5bef 100644
--- a/src/backend/catalog/namespace.c
+++ b/src/backend/catalog/namespace.c
@@ -4201,3 +4201,54 @@ pg_is_other_temp_schema(PG_FUNCTION_ARGS)
PG_RETURN_BOOL(isOtherTempNamespace(oid));
}
+
+Oid
+get_statistics_oid(List *names, bool missing_ok)
+{
+ char *schemaname;
+ char *stats_name;
+ Oid namespaceId;
+ Oid stats_oid = InvalidOid;
+ ListCell *l;
+
+ /* deconstruct the name list */
+ DeconstructQualifiedName(names, &schemaname, &stats_name);
+
+ if (schemaname)
+ {
+ /* use exact schema given */
+ namespaceId = LookupExplicitNamespace(schemaname, missing_ok);
+ if (missing_ok && !OidIsValid(namespaceId))
+ stats_oid = InvalidOid;
+ else
+ stats_oid = GetSysCacheOid2(MVSTATNAMENSP,
+ PointerGetDatum(stats_name),
+ ObjectIdGetDatum(namespaceId));
+ }
+ else
+ {
+ /* search for it in search path */
+ recomputeNamespacePath();
+
+ foreach(l, activeSearchPath)
+ {
+ namespaceId = lfirst_oid(l);
+
+ if (namespaceId == myTempNamespace)
+ continue; /* do not look in temp namespace */
+ stats_oid = GetSysCacheOid2(MVSTATNAMENSP,
+ PointerGetDatum(stats_name),
+ ObjectIdGetDatum(namespaceId));
+ if (OidIsValid(stats_oid))
+ break;
+ }
+ }
+
+ if (!OidIsValid(stats_oid) && !missing_ok)
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("statistics \"%s\" does not exist",
+ NameListToString(names))));
+
+ return stats_oid;
+}
diff --git a/src/backend/catalog/objectaddress.c b/src/backend/catalog/objectaddress.c
index d2aaa6d..85841e1 100644
--- a/src/backend/catalog/objectaddress.c
+++ b/src/backend/catalog/objectaddress.c
@@ -39,6 +39,7 @@
#include "catalog/pg_language.h"
#include "catalog/pg_largeobject.h"
#include "catalog/pg_largeobject_metadata.h"
+#include "catalog/pg_mv_statistic.h"
#include "catalog/pg_namespace.h"
#include "catalog/pg_opclass.h"
#include "catalog/pg_opfamily.h"
@@ -438,9 +439,22 @@ static const ObjectPropertyType ObjectProperty[] =
Anum_pg_type_typacl,
ACL_KIND_TYPE,
true
+ },
+ {
+ MvStatisticRelationId,
+ MvStatisticOidIndexId,
+ MVSTATOID,
+ MVSTATNAMENSP,
+ Anum_pg_mv_statistic_staname,
+ Anum_pg_mv_statistic_stanamespace,
+ InvalidAttrNumber, /* XXX same owner as relation */
+ InvalidAttrNumber, /* no ACL (same as relation) */
+ -1, /* no ACL */
+ true
}
};
+
/*
* This struct maps the string object types as returned by
* getObjectTypeDescription into ObjType enum values. Note that some enum
@@ -640,6 +654,10 @@ static const struct object_type_map
/* OCLASS_TRANSFORM */
{
"transform", OBJECT_TRANSFORM
+ },
+ /* OBJECT_STATISTICS */
+ {
+ "statistics", OBJECT_STATISTICS
}
};
@@ -913,6 +931,11 @@ get_object_address(ObjectType objtype, List *objname, List *objargs,
address = get_object_address_defacl(objname, objargs,
missing_ok);
break;
+ case OBJECT_STATISTICS:
+ address.classId = MvStatisticRelationId;
+ address.objectId = get_statistics_oid(objname, missing_ok);
+ address.objectSubId = 0;
+ break;
default:
elog(ERROR, "unrecognized objtype: %d", (int) objtype);
/* placate compiler, in case it thinks elog might return */
@@ -2185,6 +2208,10 @@ check_object_ownership(Oid roleid, ObjectType objtype, ObjectAddress address,
(errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
errmsg("must be superuser")));
break;
+ case OBJECT_STATISTICS:
+ if (!pg_statistics_ownercheck(address.objectId, roleid))
+ aclcheck_error_type(ACLCHECK_NOT_OWNER, address.objectId);
+ break;
default:
elog(ERROR, "unrecognized object type: %d",
(int) objtype);
@@ -3610,6 +3637,10 @@ getObjectTypeDescription(const ObjectAddress *object)
appendStringInfoString(&buffer, "transform");
break;
+ case OCLASS_STATISTICS:
+ appendStringInfoString(&buffer, "statistics");
+ break;
+
default:
appendStringInfo(&buffer, "unrecognized %u", object->classId);
break;
@@ -4566,6 +4597,29 @@ getObjectIdentityParts(const ObjectAddress *object,
}
break;
+ case OCLASS_STATISTICS:
+ {
+ HeapTuple tup;
+ Form_pg_mv_statistic formStatistic;
+ char *schema;
+
+ tup = SearchSysCache1(MVSTATOID,
+ ObjectIdGetDatum(object->objectId));
+ if (!HeapTupleIsValid(tup))
+ elog(ERROR, "cache lookup failed for statistics %u",
+ object->objectId);
+ formStatistic = (Form_pg_mv_statistic) GETSTRUCT(tup);
+ schema = get_namespace_name_or_temp(formStatistic->stanamespace);
+ appendStringInfoString(&buffer,
+ quote_qualified_identifier(schema,
+ NameStr(formStatistic->staname)));
+ if (objname)
+ *objname = list_make2(schema,
+ pstrdup(NameStr(formStatistic->staname)));
+ ReleaseSysCache(tup);
+ break;
+ }
+
default:
appendStringInfo(&buffer, "unrecognized object %u %u %d",
object->classId,
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index abf9a70..b8a264e 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -158,6 +158,17 @@ CREATE VIEW pg_indexes AS
LEFT JOIN pg_tablespace T ON (T.oid = I.reltablespace)
WHERE C.relkind IN ('r', 'm') AND I.relkind = 'i';
+CREATE VIEW pg_mv_stats AS
+ SELECT
+ N.nspname AS schemaname,
+ C.relname AS tablename,
+ S.staname AS staname,
+ S.stakeys AS attnums,
+ length(S.stadeps) as depsbytes,
+ pg_mv_stats_dependencies_info(S.stadeps) as depsinfo
+ FROM (pg_mv_statistic S JOIN pg_class C ON (C.oid = S.starelid))
+ LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace);
+
CREATE VIEW pg_stats WITH (security_barrier) AS
SELECT
nspname AS schemaname,
diff --git a/src/backend/commands/Makefile b/src/backend/commands/Makefile
index b1ac704..5151001 100644
--- a/src/backend/commands/Makefile
+++ b/src/backend/commands/Makefile
@@ -18,8 +18,8 @@ OBJS = aggregatecmds.o alter.o analyze.o async.o cluster.o comment.o \
event_trigger.o explain.o extension.o foreigncmds.o functioncmds.o \
indexcmds.o lockcmds.o matview.o operatorcmds.o opclasscmds.o \
policy.o portalcmds.o prepare.o proclang.o \
- schemacmds.o seclabel.o sequence.o tablecmds.o tablespace.o trigger.o \
- tsearchcmds.o typecmds.o user.o vacuum.o vacuumlazy.o \
- variable.o view.o
+ schemacmds.o seclabel.o sequence.o statscmds.o \
+ tablecmds.o tablespace.o trigger.o tsearchcmds.o typecmds.o \
+ user.o vacuum.o vacuumlazy.o variable.o view.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/commands/analyze.c b/src/backend/commands/analyze.c
index 8a5f07c..9087532 100644
--- a/src/backend/commands/analyze.c
+++ b/src/backend/commands/analyze.c
@@ -17,6 +17,7 @@
#include <math.h>
#include "access/multixact.h"
+#include "access/sysattr.h"
#include "access/transam.h"
#include "access/tupconvert.h"
#include "access/tuptoaster.h"
@@ -27,6 +28,7 @@
#include "catalog/indexing.h"
#include "catalog/pg_collation.h"
#include "catalog/pg_inherits_fn.h"
+#include "catalog/pg_mv_statistic.h"
#include "catalog/pg_namespace.h"
#include "commands/dbcommands.h"
#include "commands/tablecmds.h"
@@ -45,10 +47,13 @@
#include "storage/procarray.h"
#include "utils/acl.h"
#include "utils/attoptcache.h"
+#include "utils/builtins.h"
#include "utils/datum.h"
+#include "utils/fmgroids.h"
#include "utils/guc.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/mvstats.h"
#include "utils/pg_rusage.h"
#include "utils/sampling.h"
#include "utils/sortsupport.h"
@@ -460,6 +465,19 @@ do_analyze_rel(Relation onerel, int options, VacuumParams *params,
* all analyzable columns. We use a lower bound of 100 rows to avoid
* possible overflow in Vitter's algorithm. (Note: that will also be the
* target in the corner case where there are no analyzable columns.)
+ *
+ * FIXME This sample sizing is mostly OK when computing stats for
+ * individual columns, but when computing multi-variate stats
+ * for multivariate stats (histograms, mcv, ...) it's rather
+ * insufficient. For stats on multiple columns / complex stats
+ * we need larger sample sizes, because we need to build more
+ * detailed stats (more MCV items / histogram buckets) to get
+ * good accuracy. Maybe it'd be appropriate to use samples
+ * proportional to the table (say, 0.5% - 1%) instead of a
+ * fixed size might be more appropriate. Also, this should be
+ * bound to the requested statistics size - e.g. number of MCV
+ * items or histogram buckets should require several sample
+ * rows per item/bucket (so the sample should be k*size).
*/
targrows = 100;
for (i = 0; i < attr_cnt; i++)
@@ -562,6 +580,9 @@ do_analyze_rel(Relation onerel, int options, VacuumParams *params,
update_attstats(RelationGetRelid(Irel[ind]), false,
thisdata->attr_cnt, thisdata->vacattrstats);
}
+
+ /* Build multivariate stats (if there are any). */
+ build_mv_stats(onerel, numrows, rows, attr_cnt, vacattrstats);
}
/*
diff --git a/src/backend/commands/dropcmds.c b/src/backend/commands/dropcmds.c
index 522027a..cd65b58 100644
--- a/src/backend/commands/dropcmds.c
+++ b/src/backend/commands/dropcmds.c
@@ -292,6 +292,10 @@ does_not_exist_skipping(ObjectType objtype, List *objname, List *objargs)
msg = gettext_noop("schema \"%s\" does not exist, skipping");
name = NameListToString(objname);
break;
+ case OBJECT_STATISTICS:
+ msg = gettext_noop("statistics \"%s\" does not exist, skipping");
+ name = NameListToString(objname);
+ break;
case OBJECT_TSPARSER:
if (!schema_does_not_exist_skipping(objname, &msg, &name))
{
diff --git a/src/backend/commands/event_trigger.c b/src/backend/commands/event_trigger.c
index 9e32f8d..09061bb 100644
--- a/src/backend/commands/event_trigger.c
+++ b/src/backend/commands/event_trigger.c
@@ -110,6 +110,7 @@ static event_trigger_support_data event_trigger_support[] = {
{"SCHEMA", true},
{"SEQUENCE", true},
{"SERVER", true},
+ {"STATISTICS", true},
{"TABLE", true},
{"TABLESPACE", false},
{"TRANSFORM", true},
@@ -1106,6 +1107,7 @@ EventTriggerSupportsObjectType(ObjectType obtype)
case OBJECT_RULE:
case OBJECT_SCHEMA:
case OBJECT_SEQUENCE:
+ case OBJECT_STATISTICS:
case OBJECT_TABCONSTRAINT:
case OBJECT_TABLE:
case OBJECT_TRANSFORM:
@@ -1167,6 +1169,7 @@ EventTriggerSupportsObjectClass(ObjectClass objclass)
case OCLASS_DEFACL:
case OCLASS_EXTENSION:
case OCLASS_POLICY:
+ case OCLASS_STATISTICS:
return true;
}
diff --git a/src/backend/commands/statscmds.c b/src/backend/commands/statscmds.c
new file mode 100644
index 0000000..1b89bbe
--- /dev/null
+++ b/src/backend/commands/statscmds.c
@@ -0,0 +1,266 @@
+/*-------------------------------------------------------------------------
+ *
+ * statscmds.c
+ * Commands for creating and altering multivariate statistics
+ *
+ * Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/commands/statscmds.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/relscan.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/namespace.h"
+#include "catalog/pg_mv_statistic.h"
+#include "catalog/pg_namespace.h"
+#include "commands/defrem.h"
+#include "miscadmin.h"
+#include "utils/builtins.h"
+#include "utils/inval.h"
+#include "utils/memutils.h"
+#include "utils/mvstats.h"
+#include "utils/rel.h"
+#include "utils/syscache.h"
+
+
+/* used for sorting the attnums in ExecCreateStatistics */
+static int compare_int16(const void *a, const void *b)
+{
+ return memcmp(a, b, sizeof(int16));
+}
+
+/*
+ * Implements the CREATE STATISTICS name ON table (columns) WITH (options)
+ *
+ * TODO Check that the types support sort, although maybe we can live
+ * without it (and only build MCV list / association rules).
+ *
+ * TODO This should probably check for duplicate stats (i.e. same
+ * keys, same options). Although maybe it's useful to have
+ * multiple stats on the same columns with different options
+ * (say, a detailed MCV-only stats for some queries, histogram
+ * for others, etc.)
+ */
+ObjectAddress
+CreateStatistics(CreateStatsStmt *stmt)
+{
+ int i, j;
+ ListCell *l;
+ int16 attnums[INDEX_MAX_KEYS];
+ int numcols = 0;
+ ObjectAddress address = InvalidObjectAddress;
+ char *namestr;
+ NameData staname;
+ Oid statoid;
+ Oid namespaceId;
+
+ HeapTuple htup;
+ Datum values[Natts_pg_mv_statistic];
+ bool nulls[Natts_pg_mv_statistic];
+ int2vector *stakeys;
+ Relation mvstatrel;
+ Relation rel;
+ ObjectAddress parentobject, childobject;
+
+ /* by default build nothing */
+ bool build_dependencies = false;
+
+ Assert(IsA(stmt, CreateStatsStmt));
+
+ /* resolve the pieces of the name (namespace etc.) */
+ namespaceId = QualifiedNameGetCreationNamespace(stmt->defnames, &namestr);
+ namestrcpy(&staname, namestr);
+
+ /*
+ * If if_not_exists was given and the statistics already exists, bail out.
+ */
+ if (stmt->if_not_exists &&
+ SearchSysCacheExists2(MVSTATNAMENSP,
+ PointerGetDatum(&staname),
+ ObjectIdGetDatum(namespaceId)))
+ {
+ ereport(NOTICE,
+ (errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("statistics \"%s\" already exists, skipping",
+ namestr)));
+ return InvalidObjectAddress;
+ }
+
+ rel = heap_openrv(stmt->relation, AccessExclusiveLock);
+
+ /* transform the column names to attnum values */
+
+ foreach(l, stmt->keys)
+ {
+ char *attname = strVal(lfirst(l));
+ HeapTuple atttuple;
+
+ atttuple = SearchSysCacheAttName(RelationGetRelid(rel), attname);
+
+ if (!HeapTupleIsValid(atttuple))
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_COLUMN),
+ errmsg("column \"%s\" referenced in statistics does not exist",
+ attname)));
+
+ /* more than MVHIST_MAX_DIMENSIONS columns not allowed */
+ if (numcols >= MVSTATS_MAX_DIMENSIONS)
+ ereport(ERROR,
+ (errcode(ERRCODE_TOO_MANY_COLUMNS),
+ errmsg("cannot have more than %d keys in a statistics",
+ MVSTATS_MAX_DIMENSIONS)));
+
+ attnums[numcols] = ((Form_pg_attribute) GETSTRUCT(atttuple))->attnum;
+ ReleaseSysCache(atttuple);
+ numcols++;
+ }
+
+ /*
+ * Check the lower bound (at least 2 columns), the upper bound was
+ * already checked in the loop.
+ */
+ if (numcols < 2)
+ ereport(ERROR,
+ (errcode(ERRCODE_TOO_MANY_COLUMNS),
+ errmsg("multivariate stats require 2 or more columns")));
+
+ /* look for duplicities */
+ for (i = 0; i < numcols; i++)
+ for (j = 0; j < numcols; j++)
+ if ((i != j) && (attnums[i] == attnums[j]))
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_COLUMN),
+ errmsg("duplicate column name in statistics definition")));
+
+ /* parse the statistics options */
+ foreach (l, stmt->options)
+ {
+ DefElem *opt = (DefElem*)lfirst(l);
+
+ if (strcmp(opt->defname, "dependencies") == 0)
+ build_dependencies = defGetBoolean(opt);
+ else
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("unrecognized STATISTICS option \"%s\"",
+ opt->defname)));
+ }
+
+ /* check that at least some statistics were requested */
+ if (! build_dependencies)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("no statistics type (dependencies) was requested")));
+
+ /* sort the attnums and build int2vector */
+ qsort(attnums, numcols, sizeof(int16), compare_int16);
+ stakeys = buildint2vector(attnums, numcols);
+
+ /*
+ * Okay, let's create the pg_mv_statistic entry.
+ */
+ memset(values, 0, sizeof(values));
+ memset(nulls, false, sizeof(nulls));
+
+ /* no stats collected yet, so just the keys */
+ values[Anum_pg_mv_statistic_starelid-1] = ObjectIdGetDatum(RelationGetRelid(rel));
+ values[Anum_pg_mv_statistic_staname -1] = NameGetDatum(&staname);
+ values[Anum_pg_mv_statistic_stanamespace -1] = ObjectIdGetDatum(namespaceId);
+ values[Anum_pg_mv_statistic_staowner-1] = ObjectIdGetDatum(GetUserId());
+
+ values[Anum_pg_mv_statistic_stakeys -1] = PointerGetDatum(stakeys);
+
+ values[Anum_pg_mv_statistic_deps_enabled -1] = BoolGetDatum(build_dependencies);
+
+ nulls[Anum_pg_mv_statistic_stadeps -1] = true;
+
+ /* insert the tuple into pg_mv_statistic */
+ mvstatrel = heap_open(MvStatisticRelationId, RowExclusiveLock);
+
+ htup = heap_form_tuple(mvstatrel->rd_att, values, nulls);
+
+ simple_heap_insert(mvstatrel, htup);
+
+ CatalogUpdateIndexes(mvstatrel, htup);
+
+ statoid = HeapTupleGetOid(htup);
+
+ heap_freetuple(htup);
+
+
+ /*
+ * Store a dependency too, so that statistics are dropped on DROP TABLE
+ */
+ parentobject.classId = RelationRelationId;
+ parentobject.objectId = ObjectIdGetDatum(RelationGetRelid(rel));
+ parentobject.objectSubId = 0;
+ childobject.classId = MvStatisticRelationId;
+ childobject.objectId = statoid;
+ childobject.objectSubId = 0;
+
+ recordDependencyOn(&childobject, &parentobject, DEPENDENCY_AUTO);
+
+ /*
+ * Also record dependency on the schema (to drop statistics on DROP SCHEMA)
+ */
+ parentobject.classId = NamespaceRelationId;
+ parentobject.objectId = ObjectIdGetDatum(namespaceId);
+ parentobject.objectSubId = 0;
+ childobject.classId = MvStatisticRelationId;
+ childobject.objectId = statoid;
+ childobject.objectSubId = 0;
+
+ recordDependencyOn(&childobject, &parentobject, DEPENDENCY_AUTO);
+
+
+ heap_close(mvstatrel, RowExclusiveLock);
+
+ relation_close(rel, NoLock);
+
+ /*
+ * Invalidate relcache so that others see the new statistics.
+ */
+ CacheInvalidateRelcache(rel);
+
+ ObjectAddressSet(address, MvStatisticRelationId, statoid);
+
+ return address;
+}
+
+
+/*
+ * Implements the DROP STATISTICS
+ *
+ * DROP STATISTICS stats_name ON table_name
+ *
+ * The first one requires an exact match, the second one just drops
+ * all the statistics on a table.
+ */
+void
+RemoveStatisticsById(Oid statsOid)
+{
+ Relation relation;
+ HeapTuple tup;
+
+ /*
+ * Delete the pg_proc tuple.
+ */
+ relation = heap_open(MvStatisticRelationId, RowExclusiveLock);
+
+ tup = SearchSysCache1(MVSTATOID, ObjectIdGetDatum(statsOid));
+ if (!HeapTupleIsValid(tup)) /* should not happen */
+ elog(ERROR, "cache lookup failed for statistics %u", statsOid);
+
+ simple_heap_delete(relation, &tup->t_self);
+
+ ReleaseSysCache(tup);
+
+ heap_close(relation, RowExclusiveLock);
+}
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index df7c2fa..3b7c87f 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -4124,6 +4124,20 @@ _copyAlterPolicyStmt(const AlterPolicyStmt *from)
return newnode;
}
+static CreateStatsStmt *
+_copyCreateStatsStmt(const CreateStatsStmt *from)
+{
+ CreateStatsStmt *newnode = makeNode(CreateStatsStmt);
+
+ COPY_NODE_FIELD(defnames);
+ COPY_NODE_FIELD(relation);
+ COPY_NODE_FIELD(keys);
+ COPY_NODE_FIELD(options);
+ COPY_SCALAR_FIELD(if_not_exists);
+
+ return newnode;
+}
+
/* ****************************************************************
* pg_list.h copy functions
* ****************************************************************
@@ -4999,6 +5013,9 @@ copyObject(const void *from)
case T_CommonTableExpr:
retval = _copyCommonTableExpr(from);
break;
+ case T_CreateStatsStmt:
+ retval = _copyCreateStatsStmt(from);
+ break;
case T_FuncWithArgs:
retval = _copyFuncWithArgs(from);
break;
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index eb0fc1e..07206d7 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2153,6 +2153,21 @@ _outIndexOptInfo(StringInfo str, const IndexOptInfo *node)
}
static void
+_outMVStatisticInfo(StringInfo str, const MVStatisticInfo *node)
+{
+ WRITE_NODE_TYPE("MVSTATISTICINFO");
+
+ /* NB: this isn't a complete set of fields */
+ WRITE_OID_FIELD(mvoid);
+
+ /* enabled statistics */
+ WRITE_BOOL_FIELD(deps_enabled);
+
+ /* built/available statistics */
+ WRITE_BOOL_FIELD(deps_built);
+}
+
+static void
_outEquivalenceClass(StringInfo str, const EquivalenceClass *node)
{
/*
@@ -3636,6 +3651,9 @@ _outNode(StringInfo str, const void *obj)
case T_PlannerParamItem:
_outPlannerParamItem(str, obj);
break;
+ case T_MVStatisticInfo:
+ _outMVStatisticInfo(str, obj);
+ break;
case T_ExtensibleNode:
_outExtensibleNode(str, obj);
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index ad715bb..7fb2088 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -28,6 +28,7 @@
#include "catalog/dependency.h"
#include "catalog/heap.h"
#include "catalog/pg_am.h"
+#include "catalog/pg_mv_statistic.h"
#include "foreign/fdwapi.h"
#include "miscadmin.h"
#include "nodes/makefuncs.h"
@@ -40,7 +41,9 @@
#include "parser/parsetree.h"
#include "rewrite/rewriteManip.h"
#include "storage/bufmgr.h"
+#include "utils/builtins.h"
#include "utils/lsyscache.h"
+#include "utils/syscache.h"
#include "utils/rel.h"
#include "utils/snapmgr.h"
@@ -94,6 +97,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
Relation relation;
bool hasindex;
List *indexinfos = NIL;
+ List *stainfos = NIL;
/*
* We need not lock the relation since it was already locked, either by
@@ -387,6 +391,61 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
rel->indexlist = indexinfos;
+ if (true)
+ {
+ List *mvstatoidlist;
+ ListCell *l;
+
+ mvstatoidlist = RelationGetMVStatList(relation);
+
+ foreach(l, mvstatoidlist)
+ {
+ ArrayType *arr;
+ Datum adatum;
+ bool isnull;
+ Oid mvoid = lfirst_oid(l);
+ Form_pg_mv_statistic mvstat;
+ MVStatisticInfo *info;
+
+ HeapTuple htup = SearchSysCache1(MVSTATOID, ObjectIdGetDatum(mvoid));
+
+ mvstat = (Form_pg_mv_statistic) GETSTRUCT(htup);
+
+ /* unavailable stats are not interesting for the planner */
+ if (mvstat->deps_built)
+ {
+ info = makeNode(MVStatisticInfo);
+
+ info->mvoid = mvoid;
+ info->rel = rel;
+
+ /* enabled statistics */
+ info->deps_enabled = mvstat->deps_enabled;
+
+ /* built/available statistics */
+ info->deps_built = mvstat->deps_built;
+
+ /* stakeys */
+ adatum = SysCacheGetAttr(MVSTATOID, htup,
+ Anum_pg_mv_statistic_stakeys, &isnull);
+ Assert(!isnull);
+
+ arr = DatumGetArrayTypeP(adatum);
+
+ info->stakeys = buildint2vector((int16 *) ARR_DATA_PTR(arr),
+ ARR_DIMS(arr)[0]);
+
+ stainfos = lcons(info, stainfos);
+ }
+
+ ReleaseSysCache(htup);
+ }
+
+ list_free(mvstatoidlist);
+ }
+
+ rel->mvstatlist = stainfos;
+
/* Grab foreign-table info using the relcache, while we have it */
if (relation->rd_rel->relkind == RELKIND_FOREIGN_TABLE)
{
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index b307b48..3be3f02 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -241,7 +241,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
ConstraintsSetStmt CopyStmt CreateAsStmt CreateCastStmt
CreateDomainStmt CreateExtensionStmt CreateGroupStmt CreateOpClassStmt
CreateOpFamilyStmt AlterOpFamilyStmt CreatePLangStmt
- CreateSchemaStmt CreateSeqStmt CreateStmt CreateTableSpaceStmt
+ CreateSchemaStmt CreateSeqStmt CreateStmt CreateStatsStmt CreateTableSpaceStmt
CreateFdwStmt CreateForeignServerStmt CreateForeignTableStmt
CreateAssertStmt CreateTransformStmt CreateTrigStmt CreateEventTrigStmt
CreateUserStmt CreateUserMappingStmt CreateRoleStmt CreatePolicyStmt
@@ -809,6 +809,7 @@ stmt :
| CreateSchemaStmt
| CreateSeqStmt
| CreateStmt
+ | CreateStatsStmt
| CreateTableSpaceStmt
| CreateTransformStmt
| CreateTrigStmt
@@ -3436,6 +3437,36 @@ OptConsTableSpace: USING INDEX TABLESPACE name { $$ = $4; }
ExistingIndex: USING INDEX index_name { $$ = $3; }
;
+/*****************************************************************************
+ *
+ * QUERY :
+ * CREATE STATISTICS stats_name ON relname (columns) WITH (options)
+ *
+ *****************************************************************************/
+
+
+CreateStatsStmt: CREATE STATISTICS any_name ON qualified_name '(' columnList ')' opt_reloptions
+ {
+ CreateStatsStmt *n = makeNode(CreateStatsStmt);
+ n->defnames = $3;
+ n->relation = $5;
+ n->keys = $7;
+ n->options = $9;
+ n->if_not_exists = false;
+ $$ = (Node *)n;
+ }
+ | CREATE STATISTICS IF_P NOT EXISTS any_name ON qualified_name '(' columnList ')' opt_reloptions
+ {
+ CreateStatsStmt *n = makeNode(CreateStatsStmt);
+ n->defnames = $6;
+ n->relation = $8;
+ n->keys = $10;
+ n->options = $12;
+ n->if_not_exists = true;
+ $$ = (Node *)n;
+ }
+ ;
+
/*****************************************************************************
*
@@ -5621,6 +5652,7 @@ drop_type: TABLE { $$ = OBJECT_TABLE; }
| TEXT_P SEARCH DICTIONARY { $$ = OBJECT_TSDICTIONARY; }
| TEXT_P SEARCH TEMPLATE { $$ = OBJECT_TSTEMPLATE; }
| TEXT_P SEARCH CONFIGURATION { $$ = OBJECT_TSCONFIGURATION; }
+ | STATISTICS { $$ = OBJECT_STATISTICS; }
;
any_name_list:
diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c
index 045f7f0..2ba88e2 100644
--- a/src/backend/tcop/utility.c
+++ b/src/backend/tcop/utility.c
@@ -1520,6 +1520,10 @@ ProcessUtilitySlow(Node *parsetree,
address = ExecSecLabelStmt((SecLabelStmt *) parsetree);
break;
+ case T_CreateStatsStmt: /* CREATE STATISTICS */
+ address = CreateStatistics((CreateStatsStmt *) parsetree);
+ break;
+
default:
elog(ERROR, "unrecognized node type: %d",
(int) nodeTag(parsetree));
@@ -2160,6 +2164,9 @@ CreateCommandTag(Node *parsetree)
case OBJECT_TRANSFORM:
tag = "DROP TRANSFORM";
break;
+ case OBJECT_STATISTICS:
+ tag = "DROP STATISTICS";
+ break;
default:
tag = "???";
}
@@ -2527,6 +2534,10 @@ CreateCommandTag(Node *parsetree)
tag = "EXECUTE";
break;
+ case T_CreateStatsStmt:
+ tag = "CREATE STATISTICS";
+ break;
+
case T_DeallocateStmt:
{
DeallocateStmt *stmt = (DeallocateStmt *) parsetree;
diff --git a/src/backend/utils/Makefile b/src/backend/utils/Makefile
index 8374533..eba0352 100644
--- a/src/backend/utils/Makefile
+++ b/src/backend/utils/Makefile
@@ -9,7 +9,7 @@ top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
OBJS = fmgrtab.o
-SUBDIRS = adt cache error fmgr hash init mb misc mmgr resowner sort time
+SUBDIRS = adt cache error fmgr hash init mb misc mmgr mvstats resowner sort time
# location of Catalog.pm
catalogdir = $(top_srcdir)/src/backend/catalog
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 130c06d..3bc4c8a 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -47,6 +47,7 @@
#include "catalog/pg_auth_members.h"
#include "catalog/pg_constraint.h"
#include "catalog/pg_database.h"
+#include "catalog/pg_mv_statistic.h"
#include "catalog/pg_namespace.h"
#include "catalog/pg_opclass.h"
#include "catalog/pg_proc.h"
@@ -3956,6 +3957,62 @@ RelationGetIndexList(Relation relation)
return result;
}
+
+List *
+RelationGetMVStatList(Relation relation)
+{
+ Relation indrel;
+ SysScanDesc indscan;
+ ScanKeyData skey;
+ HeapTuple htup;
+ List *result;
+ List *oldlist;
+ MemoryContext oldcxt;
+
+ /* Quick exit if we already computed the list. */
+ if (relation->rd_mvstatvalid != 0)
+ return list_copy(relation->rd_mvstatlist);
+
+ /*
+ * We build the list we intend to return (in the caller's context) while
+ * doing the scan. After successfully completing the scan, we copy that
+ * list into the relcache entry. This avoids cache-context memory leakage
+ * if we get some sort of error partway through.
+ */
+ result = NIL;
+
+ /* Prepare to scan pg_index for entries having indrelid = this rel. */
+ ScanKeyInit(&skey,
+ Anum_pg_mv_statistic_starelid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationGetRelid(relation)));
+
+ indrel = heap_open(MvStatisticRelationId, AccessShareLock);
+ indscan = systable_beginscan(indrel, MvStatisticRelidIndexId, true,
+ NULL, 1, &skey);
+
+ while (HeapTupleIsValid(htup = systable_getnext(indscan)))
+ /* TODO maybe include only already built statistics? */
+ result = insert_ordered_oid(result, HeapTupleGetOid(htup));
+
+ systable_endscan(indscan);
+
+ heap_close(indrel, AccessShareLock);
+
+ /* Now save a copy of the completed list in the relcache entry. */
+ oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
+ oldlist = relation->rd_mvstatlist;
+ relation->rd_mvstatlist = list_copy(result);
+
+ relation->rd_mvstatvalid = true;
+ MemoryContextSwitchTo(oldcxt);
+
+ /* Don't leak the old list, if there is one */
+ list_free(oldlist);
+
+ return result;
+}
+
/*
* insert_ordered_oid
* Insert a new Oid into a sorted list of Oids, preserving ordering
@@ -4920,6 +4977,8 @@ load_relcache_init_file(bool shared)
rel->rd_indexattr = NULL;
rel->rd_keyattr = NULL;
rel->rd_idattr = NULL;
+ rel->rd_mvstatvalid = false;
+ rel->rd_mvstatlist = NIL;
rel->rd_createSubid = InvalidSubTransactionId;
rel->rd_newRelfilenodeSubid = InvalidSubTransactionId;
rel->rd_amcache = NULL;
diff --git a/src/backend/utils/cache/syscache.c b/src/backend/utils/cache/syscache.c
index 65ffe84..3c1bc4b 100644
--- a/src/backend/utils/cache/syscache.c
+++ b/src/backend/utils/cache/syscache.c
@@ -44,6 +44,7 @@
#include "catalog/pg_foreign_server.h"
#include "catalog/pg_foreign_table.h"
#include "catalog/pg_language.h"
+#include "catalog/pg_mv_statistic.h"
#include "catalog/pg_namespace.h"
#include "catalog/pg_opclass.h"
#include "catalog/pg_operator.h"
@@ -502,6 +503,28 @@ static const struct cachedesc cacheinfo[] = {
},
4
},
+ {MvStatisticRelationId, /* MVSTATNAMENSP */
+ MvStatisticNameIndexId,
+ 2,
+ {
+ Anum_pg_mv_statistic_staname,
+ Anum_pg_mv_statistic_stanamespace,
+ 0,
+ 0
+ },
+ 4
+ },
+ {MvStatisticRelationId, /* MVSTATOID */
+ MvStatisticOidIndexId,
+ 1,
+ {
+ ObjectIdAttributeNumber,
+ 0,
+ 0,
+ 0
+ },
+ 4
+ },
{NamespaceRelationId, /* NAMESPACENAME */
NamespaceNameIndexId,
1,
diff --git a/src/backend/utils/mvstats/Makefile b/src/backend/utils/mvstats/Makefile
new file mode 100644
index 0000000..099f1ed
--- /dev/null
+++ b/src/backend/utils/mvstats/Makefile
@@ -0,0 +1,17 @@
+#-------------------------------------------------------------------------
+#
+# Makefile--
+# Makefile for utils/mvstats
+#
+# IDENTIFICATION
+# src/backend/utils/mvstats/Makefile
+#
+#-------------------------------------------------------------------------
+
+subdir = src/backend/utils/mvstats
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+
+OBJS = common.o dependencies.o
+
+include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/utils/mvstats/README.dependencies b/src/backend/utils/mvstats/README.dependencies
new file mode 100644
index 0000000..1f96fbc
--- /dev/null
+++ b/src/backend/utils/mvstats/README.dependencies
@@ -0,0 +1,222 @@
+Soft functional dependencies
+============================
+
+A type of multivariate statistics used to capture cases when one column (or
+possibly a combination of columns) determines values in another column. We may
+also say that one column implies the other one.
+
+A simple artificial example may be a table with two columns, created like this
+
+ CREATE TABLE t (a INT, b INT)
+ AS SELECT i, i/10 FROM generate_series(1,100000) s(i);
+
+Clearly, once we know the value for column 'a' the value for 'b' is trivially
+determined, as it's simply (a/10). A more practical example may be addresses,
+where (ZIP code -> city name), i.e. once we know the ZIP, we probably know the
+city it belongs to, as ZIP codes are usually assigned to one city. Larger cities
+may have multiple ZIP codes, so the dependency can't be reversed.
+
+Functional dependencies are a concept well described in relational theory,
+particularly in definition of normalization and "normal forms". Wikipedia has a
+nice definition of a functional dependency [1]:
+
+ In a given table, an attribute Y is said to have a functional dependency on
+ a set of attributes X (written X -> Y) if and only if each X value is
+ associated with precisely one Y value. For example, in an "Employee" table
+ that includes the attributes "Employee ID" and "Employee Date of Birth", the
+ functional dependency {Employee ID} -> {Employee Date of Birth} would hold.
+ It follows from the previous two sentences that each {Employee ID} is
+ associated with precisely one {Employee Date of Birth}.
+
+ [1] http://en.wikipedia.org/wiki/Database_normalization
+
+Many datasets might be normalized not to contain such dependencies, but often
+it's not practical for various reasons. In some cases it's actually a conscious
+design choice to model the dataset in denormalized way, either because of
+performance or to make querying easier.
+
+The functional dependencies are called 'soft' because the implementation is
+meant to allow small number of rows contradicting the dependency. Many actual
+data sets contain some sort of errors, either because of data entry mistakes
+(user mistyping the ZIP code) or issues in generating the data (e.g. a ZIP code
+mistakenly assigned to two cities in different states). A strict implementation
+would ignore dependencies on such noisy data, rendering the approach unusable on
+such data sets.
+
+
+Mining dependencies (ANALYZE)
+-----------------------------
+
+The current build algorithm is rather simple - for each pair (a,b) of columns,
+the data are sorted lexicographically (first by 'a', then by 'b'). Then for each
+group (rows with the same 'a' value) we decide whether the group is neutral,
+supporting or contradicting the dependency (a->b).
+
+A group is considered neutral when it's too small - e.g. when there's a single
+row in the group, there can't possibly be multiple values in 'b'. For this
+reason we ignore groups smaller than a threshold (currently 3 rows).
+
+For sufficiently large groups (3 rows or more), we count the number of distinct
+values in 'b'. When there's a single 'b' value, the group is considered to
+support the dependency (a->b), otherwise it's condidered as contradicting it.
+
+At the end, we compare the number of rows in supporting and contradicting groups,
+and if there are at least 10x as many supporting rows, we consider the
+functional dependency to be valid.
+
+
+This approach has the negative property that the algorithm is that it's a bit
+fragile with respect to the sample - there may be data sets producing quite
+different results for each ANALYZE execution (as even a single row may change
+the outcome of the final 10x test).
+
+It was proposed to make the dependencies "fuzzy" - e.g. track some coefficient
+between [0,1] determining how much the dependency holds. That would however mean
+we have to keep all the dependencies, as eliminating them based on the value of
+the coefficient (e.g. throw away dependencies <= 0.5) would result in exactly
+the same fragility issues. This would also make it more complicated to combine
+dependencies. So this does not seem like a practical approach.
+
+A better approach might be to replace the constants (min_group_size=3 and 10x)
+with values somehow related to the particular data set.
+
+
+Clause reduction (planner/optimizer)
+------------------------------------
+
+Apllying the functional dependencies is quite simple - given a list of equality
+clauses, check which clauses are redundant (i.e. implied by some other clause).
+For example given clause list
+
+ (a = 2) AND (b = 2) AND (c = 3)
+
+and dependencies (a->b) and (a->d), the list of clauses may be simplified to
+
+ (a = 1) AND (c = 3)
+
+Functional dependencies may only be applied to equality clauses, all other types
+of clauses are ignored. See clauselist_apply_dependencies() for more details.
+
+
+Compatibility of clauses
+------------------------
+
+The reduction assumes the clauses really are redundant, and the value in the
+reduced clause (b=2) is the value determined by (a=1). If that's not the case
+and the values are "incompatible" the result will be over-estimation.
+
+This may happen for example when using conditions on ZIP and city name with
+mismatching values (ZIP for a different city), etc. In such case the result
+set will be empty, but we'll estimate the selectivity using the ZIP condition.
+
+In this case the default estimation based on AVIA principle happens to work
+better, but mostly by chance.
+
+
+Dependencies vs. MCV/histogram
+------------------------------
+
+In some cases the "compatibility" of the conditions might be verified using the
+other types of multivariate stats - MCV lists and histograms.
+
+For MCV lists the verification might be very simple - peek into the list if
+there are any items matching the clause on the 'a' column (e.g. ZIP code), and
+if such item is found, check that the 'b' column matches the other clause. If it
+does not, the clauses are contradictory. We can't really say if such item was
+not found, except maybe restricting the selectivity using the MCV data (e.g.
+using min/max selectivity, or something).
+
+With histograms, it might work similarly - we can't check the values directly
+(because histograms use buckets, unlike MCV lists, storing the actual values).
+So we can only observe the buckets matching the clauses - if those buckets have
+very low frequency, it probably means the two clauses are incompatible.
+
+It's unclear what 'low frequency' is, but if one of the clauses is implied
+(automatically true because of the other clause), then
+
+ selectivity[clause(A)] = selectivity[clause(A) & clause(B)]
+
+So we might compute selectivity of the first clause - for example using regular
+statistics. And then check if the selectivity computed from the histogram is
+about the same (or significantly lower).
+
+The problem is that histograms work well only when the data ordering matches the
+natural meaning. For values that serve as labels - like city names or ZIP codes,
+or even generated IDs, histograms really don't work all that well. For example
+sorting cities by name won't match the sorting of ZIP codes, rendering the
+histogram unusable.
+
+So MCVs are probably going to work much better, because they don't really assume
+any sort of ordering. And it's probably more appropriate for the label-like data.
+
+A good question however is why even use functional dependencies in such cases
+and not simply use the MCV/histogram instead. One reason is that the functional
+dependencies allow fallback to regular stats, and often produce more accurate
+estimates - especially compared to histograms, that are quite bad in estimating
+equality clauses.
+
+
+Limitations
+-----------
+
+Let's see the main liminations of functional dependencies, especially those
+related to the current implementation.
+
+The current implementation supports only dependencies between two columns, but
+this is merely a simplification of the initial implementation. It's certainly
+useful to mine for dependencies involving multiple columns on the 'left' side,
+i.e. a condition for the dependency. That is dependencies like (a,b -> c).
+
+The implementation may/should be smart enough not to mine redundant conditions,
+e.g. (a->b) and (a,c -> b), because the latter is a trivial consequence of the
+former one (if values of 'a' determine 'b', adding another column won't change
+that relationship). The ANALYZE should first analyze 1:1 dependencies, then 2:1
+dependencies (and skip the already identified ones), etc.
+
+For example the dependency
+
+ (city name -> zip code)
+
+is much stronger, i.e. whenever it hold, then
+
+ (city name, state name -> zip code)
+
+holds too. But in case there are cities with the same name in different states,
+then only the latter dependency will be valid.
+
+Of course, there probably are cities with the same name within a single state,
+but hopefully this is relatively rare occurence (and thus we'll still detect
+the 'soft' dependency).
+
+Handling multiple columns on the right side of the dependency, is not necessary,
+as those dependencies may be simply decomposed into a set of dependencies with
+the same meaning, one for each column on the right side. For example
+
+ (a -> b,c)
+
+is exactly the same as
+
+ (a -> b) & (a -> c)
+
+Of course, storing the first form may be more efficient thant storing multiple
+'simple' dependencies separately.
+
+
+TODO Support dependencies with multiple columns on left/right.
+
+TODO Investigate using histogram and MCV list to verify the dependencies.
+
+TODO Investigate statistical testing of the distribution (to decide whether it
+ makes sense to build the histogram/MCV list).
+
+TODO Using a min/max of selectivities would probably make more sense for the
+ associated columns.
+
+TODO Consider eliminating the implied columns from the histogram and MCV lists
+ (but maybe that's not a good idea, because that'd make it impossible to use
+ these stats for non-equality clauses and also it wouldn't be possible to
+ use the stats for verification of the dependencies).
+
+TODO The reduction probably might be extended to also handle IS NULL clauses,
+ assuming we fix the ANALYZE to properly handle NULL values. We however
+ won't be able to reduce IS NOT NULL (unless I'm missing something).
diff --git a/src/backend/utils/mvstats/common.c b/src/backend/utils/mvstats/common.c
new file mode 100644
index 0000000..a755c49
--- /dev/null
+++ b/src/backend/utils/mvstats/common.c
@@ -0,0 +1,356 @@
+/*-------------------------------------------------------------------------
+ *
+ * common.c
+ * POSTGRES multivariate statistics
+ *
+ *
+ * Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/utils/mvstats/common.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "common.h"
+
+static VacAttrStats ** lookup_var_attr_stats(int2vector *attrs,
+ int natts, VacAttrStats **vacattrstats);
+
+static List* list_mv_stats(Oid relid);
+
+
+/*
+ * Compute requested multivariate stats, using the rows sampled for the
+ * plain (single-column) stats.
+ *
+ * This fetches a list of stats from pg_mv_statistic, computes the stats
+ * and serializes them back into the catalog (as bytea values).
+ */
+void
+build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
+ int natts, VacAttrStats **vacattrstats)
+{
+ ListCell *lc;
+ List *mvstats;
+
+ TupleDesc tupdesc = RelationGetDescr(onerel);
+
+ /*
+ * Fetch defined MV groups from pg_mv_statistic, and then compute
+ * the MV statistics (histograms for now).
+ */
+ mvstats = list_mv_stats(RelationGetRelid(onerel));
+
+ foreach (lc, mvstats)
+ {
+ int j;
+ MVStatisticInfo *stat = (MVStatisticInfo *)lfirst(lc);
+ MVDependencies deps = NULL;
+
+ VacAttrStats **stats = NULL;
+ int numatts = 0;
+
+ /* int2 vector of attnums the stats should be computed on */
+ int2vector * attrs = stat->stakeys;
+
+ /* see how many of the columns are not dropped */
+ for (j = 0; j < attrs->dim1; j++)
+ if (! tupdesc->attrs[attrs->values[j]-1]->attisdropped)
+ numatts += 1;
+
+ /* if there are dropped attributes, build a filtered int2vector */
+ if (numatts != attrs->dim1)
+ {
+ int16 *tmp = palloc0(numatts * sizeof(int16));
+ int attnum = 0;
+
+ for (j = 0; j < attrs->dim1; j++)
+ if (! tupdesc->attrs[attrs->values[j]-1]->attisdropped)
+ tmp[attnum++] = attrs->values[j];
+
+ pfree(attrs);
+ attrs = buildint2vector(tmp, numatts);
+ }
+
+ /* filter only the interesting vacattrstats records */
+ stats = lookup_var_attr_stats(attrs, natts, vacattrstats);
+
+ /* check allowed number of dimensions */
+ Assert((attrs->dim1 >= 2) && (attrs->dim1 <= MVSTATS_MAX_DIMENSIONS));
+
+ /*
+ * Analyze functional dependencies of columns.
+ */
+ deps = build_mv_dependencies(numrows, rows, attrs, stats);
+
+ /* store the histogram / MCV list in the catalog */
+ update_mv_stats(stat->mvoid, deps, attrs);
+ }
+}
+
+/*
+ * Lookup the VacAttrStats info for the selected columns, with indexes
+ * matching the attrs vector (to make it easy to work with when
+ * computing multivariate stats).
+ */
+static VacAttrStats **
+lookup_var_attr_stats(int2vector *attrs, int natts, VacAttrStats **vacattrstats)
+{
+ int i, j;
+ int numattrs = attrs->dim1;
+ VacAttrStats **stats = (VacAttrStats**)palloc0(numattrs * sizeof(VacAttrStats*));
+
+ /* lookup VacAttrStats info for the requested columns (same attnum) */
+ for (i = 0; i < numattrs; i++)
+ {
+ stats[i] = NULL;
+ for (j = 0; j < natts; j++)
+ {
+ if (attrs->values[i] == vacattrstats[j]->tupattnum)
+ {
+ stats[i] = vacattrstats[j];
+ break;
+ }
+ }
+
+ /*
+ * Check that we found the info, that the attnum matches and
+ * that there's the requested 'lt' operator and that the type
+ * is 'passed-by-value'.
+ */
+ Assert(stats[i] != NULL);
+ Assert(stats[i]->tupattnum == attrs->values[i]);
+
+ /* FIXME This is rather ugly way to check for 'ltopr' (which
+ * is defined for 'scalar' attributes).
+ */
+ Assert(((StdAnalyzeData *)stats[i]->extra_data)->ltopr != InvalidOid);
+ }
+
+ return stats;
+}
+
+/*
+ * Fetch list of MV stats defined on a table, without the actual data
+ * for histograms, MCV lists etc.
+ */
+static List*
+list_mv_stats(Oid relid)
+{
+ Relation indrel;
+ SysScanDesc indscan;
+ ScanKeyData skey;
+ HeapTuple htup;
+ List *result = NIL;
+
+ /* Prepare to scan pg_mv_statistic for entries having indrelid = this rel. */
+ ScanKeyInit(&skey,
+ Anum_pg_mv_statistic_starelid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(relid));
+
+ indrel = heap_open(MvStatisticRelationId, AccessShareLock);
+ indscan = systable_beginscan(indrel, MvStatisticRelidIndexId, true,
+ NULL, 1, &skey);
+
+ while (HeapTupleIsValid(htup = systable_getnext(indscan)))
+ {
+ MVStatisticInfo *info = makeNode(MVStatisticInfo);
+ Form_pg_mv_statistic stats = (Form_pg_mv_statistic) GETSTRUCT(htup);
+
+ info->mvoid = HeapTupleGetOid(htup);
+ info->stakeys = buildint2vector(stats->stakeys.values, stats->stakeys.dim1);
+ info->deps_built = stats->deps_built;
+
+ result = lappend(result, info);
+ }
+
+ systable_endscan(indscan);
+
+ heap_close(indrel, AccessShareLock);
+
+ /* TODO maybe save the list into relcache, as in RelationGetIndexList
+ * (which was used as an inspiration of this one)?. */
+
+ return result;
+}
+
+void
+update_mv_stats(Oid mvoid, MVDependencies dependencies, int2vector *attrs)
+{
+ HeapTuple stup,
+ oldtup;
+ Datum values[Natts_pg_mv_statistic];
+ bool nulls[Natts_pg_mv_statistic];
+ bool replaces[Natts_pg_mv_statistic];
+
+ Relation sd = heap_open(MvStatisticRelationId, RowExclusiveLock);
+
+ memset(nulls, 1, Natts_pg_mv_statistic * sizeof(bool));
+ memset(replaces, 0, Natts_pg_mv_statistic * sizeof(bool));
+ memset(values, 0, Natts_pg_mv_statistic * sizeof(Datum));
+
+ /*
+ * Construct a new pg_mv_statistic tuple - replace only the histogram
+ * and MCV list, depending whether it actually was computed.
+ */
+ if (dependencies != NULL)
+ {
+ nulls[Anum_pg_mv_statistic_stadeps -1] = false;
+ values[Anum_pg_mv_statistic_stadeps - 1]
+ = PointerGetDatum(serialize_mv_dependencies(dependencies));
+ }
+
+ /* always replace the value (either by bytea or NULL) */
+ replaces[Anum_pg_mv_statistic_stadeps -1] = true;
+
+ /* always change the availability flags */
+ nulls[Anum_pg_mv_statistic_deps_built -1] = false;
+ nulls[Anum_pg_mv_statistic_stakeys-1] = false;
+
+ /* use the new attnums, in case we removed some dropped ones */
+ replaces[Anum_pg_mv_statistic_deps_built-1] = true;
+ replaces[Anum_pg_mv_statistic_stakeys -1] = true;
+
+ values[Anum_pg_mv_statistic_deps_built-1] = BoolGetDatum(dependencies != NULL);
+ values[Anum_pg_mv_statistic_stakeys -1] = PointerGetDatum(attrs);
+
+ /* Is there already a pg_mv_statistic tuple for this attribute? */
+ oldtup = SearchSysCache1(MVSTATOID,
+ ObjectIdGetDatum(mvoid));
+
+ if (HeapTupleIsValid(oldtup))
+ {
+ /* Yes, replace it */
+ stup = heap_modify_tuple(oldtup,
+ RelationGetDescr(sd),
+ values,
+ nulls,
+ replaces);
+ ReleaseSysCache(oldtup);
+ simple_heap_update(sd, &stup->t_self, stup);
+ }
+ else
+ elog(ERROR, "invalid pg_mv_statistic record (oid=%d)", mvoid);
+
+ /* update indexes too */
+ CatalogUpdateIndexes(sd, stup);
+
+ heap_freetuple(stup);
+
+ heap_close(sd, RowExclusiveLock);
+}
+
+/* multi-variate stats comparator */
+
+/*
+ * qsort_arg comparator for sorting Datums (MV stats)
+ *
+ * This does not maintain the tupnoLink array.
+ */
+int
+compare_scalars_simple(const void *a, const void *b, void *arg)
+{
+ Datum da = *(Datum*)a;
+ Datum db = *(Datum*)b;
+ SortSupport ssup= (SortSupport) arg;
+
+ return ApplySortComparator(da, false, db, false, ssup);
+}
+
+/*
+ * qsort_arg comparator for sorting data when partitioning a MV bucket
+ */
+int
+compare_scalars_partition(const void *a, const void *b, void *arg)
+{
+ Datum da = ((ScalarItem*)a)->value;
+ Datum db = ((ScalarItem*)b)->value;
+ SortSupport ssup= (SortSupport) arg;
+
+ return ApplySortComparator(da, false, db, false, ssup);
+}
+
+/* initialize multi-dimensional sort */
+MultiSortSupport
+multi_sort_init(int ndims)
+{
+ MultiSortSupport mss;
+
+ Assert(ndims >= 2);
+
+ mss = (MultiSortSupport)palloc0(offsetof(MultiSortSupportData, ssup)
+ + sizeof(SortSupportData)*ndims);
+
+ mss->ndims = ndims;
+
+ return mss;
+}
+
+/*
+ * add sort into for dimension 'dim' (index into vacattrstats) to mss,
+ * at the position 'sortattr'
+ */
+void
+multi_sort_add_dimension(MultiSortSupport mss, int sortdim,
+ int dim, VacAttrStats **vacattrstats)
+{
+ /* first, lookup StdAnalyzeData for the dimension (attribute) */
+ SortSupportData ssup;
+ StdAnalyzeData *tmp = (StdAnalyzeData *)vacattrstats[dim]->extra_data;
+
+ Assert(mss != NULL);
+ Assert(sortdim < mss->ndims);
+
+ /* initialize sort support, etc. */
+ memset(&ssup, 0, sizeof(ssup));
+ ssup.ssup_cxt = CurrentMemoryContext;
+
+ /* We always use the default collation for statistics */
+ ssup.ssup_collation = DEFAULT_COLLATION_OID;
+ ssup.ssup_nulls_first = false;
+
+ PrepareSortSupportFromOrderingOp(tmp->ltopr, &ssup);
+
+ mss->ssup[sortdim] = ssup;
+}
+
+/* compare all the dimensions in the selected order */
+int
+multi_sort_compare(const void *a, const void *b, void *arg)
+{
+ int i;
+ SortItem *ia = (SortItem*)a;
+ SortItem *ib = (SortItem*)b;
+
+ MultiSortSupport mss = (MultiSortSupport)arg;
+
+ for (i = 0; i < mss->ndims; i++)
+ {
+ int compare;
+
+ compare = ApplySortComparator(ia->values[i], ia->isnull[i],
+ ib->values[i], ib->isnull[i],
+ &mss->ssup[i]);
+
+ if (compare != 0)
+ return compare;
+
+ }
+
+ /* equal by default */
+ return 0;
+}
+
+/* compare selected dimension */
+int
+multi_sort_compare_dim(int dim, const SortItem *a, const SortItem *b,
+ MultiSortSupport mss)
+{
+ return ApplySortComparator(a->values[dim], a->isnull[dim],
+ b->values[dim], b->isnull[dim],
+ &mss->ssup[dim]);
+}
diff --git a/src/backend/utils/mvstats/common.h b/src/backend/utils/mvstats/common.h
new file mode 100644
index 0000000..d96422d
--- /dev/null
+++ b/src/backend/utils/mvstats/common.h
@@ -0,0 +1,75 @@
+/*-------------------------------------------------------------------------
+ *
+ * common.h
+ * POSTGRES multivariate statistics
+ *
+ *
+ * Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/utils/mvstats/common.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/sysattr.h"
+#include "access/tuptoaster.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_collation.h"
+#include "catalog/pg_mv_statistic.h"
+#include "foreign/fdwapi.h"
+#include "postmaster/autovacuum.h"
+#include "storage/lmgr.h"
+#include "utils/builtins.h"
+#include "utils/datum.h"
+#include "utils/fmgroids.h"
+#include "utils/mvstats.h"
+#include "utils/sortsupport.h"
+#include "utils/syscache.h"
+
+
+/* FIXME private structure copied from analyze.c */
+
+typedef struct
+{
+ Oid eqopr; /* '=' operator for datatype, if any */
+ Oid eqfunc; /* and associated function */
+ Oid ltopr; /* '<' operator for datatype, if any */
+} StdAnalyzeData;
+
+typedef struct
+{
+ Datum value; /* a data value */
+ int tupno; /* position index for tuple it came from */
+} ScalarItem;
+
+/* multi-sort */
+typedef struct MultiSortSupportData {
+ int ndims; /* number of dimensions supported by the */
+ SortSupportData ssup[1]; /* sort support data for each dimension */
+} MultiSortSupportData;
+
+typedef MultiSortSupportData* MultiSortSupport;
+
+typedef struct SortItem {
+ Datum *values;
+ bool *isnull;
+} SortItem;
+
+MultiSortSupport multi_sort_init(int ndims);
+
+void multi_sort_add_dimension(MultiSortSupport mss, int sortdim,
+ int dim, VacAttrStats **vacattrstats);
+
+int multi_sort_compare(const void *a, const void *b, void *arg);
+
+int multi_sort_compare_dim(int dim, const SortItem *a,
+ const SortItem *b, MultiSortSupport mss);
+
+/* comparators, used when constructing multivariate stats */
+int compare_scalars_simple(const void *a, const void *b, void *arg);
+int compare_scalars_partition(const void *a, const void *b, void *arg);
diff --git a/src/backend/utils/mvstats/dependencies.c b/src/backend/utils/mvstats/dependencies.c
new file mode 100644
index 0000000..2a064a0
--- /dev/null
+++ b/src/backend/utils/mvstats/dependencies.c
@@ -0,0 +1,437 @@
+/*-------------------------------------------------------------------------
+ *
+ * dependencies.c
+ * POSTGRES multivariate functional dependencies
+ *
+ *
+ * Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/utils/mvstats/dependencies.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "common.h"
+#include "utils/lsyscache.h"
+
+/*
+ * Detect functional dependencies between columns.
+ *
+ * TODO This builds a complete set of dependencies, i.e. including transitive
+ * dependencies - if we identify [A => B] and [B => C], we're likely to
+ * identify [A => C] too. It might be better to keep only the minimal set
+ * of dependencies, i.e. prune all the dependencies that we can recreate
+ * by transivitity.
+ *
+ * There are two conceptual ways to do that:
+ *
+ * (a) generate all the rules, and then prune the rules that may be
+ * recteated by combining other dependencies, or
+ *
+ * (b) performing the 'is combination of other dependencies' check before
+ * actually doing the work
+ *
+ * The second option has the advantage that we don't really need to perform
+ * the sort/count. It's not sufficient alone, though, because we may
+ * discover the dependencies in the wrong order. For example we may find
+ *
+ * (a -> b), (a -> c) and then (b -> c)
+ *
+ * None of those dependencies is a combination of the already known ones,
+ * yet (a -> C) is a combination of (a -> b) and (b -> c).
+ *
+ *
+ * FIXME Currently we simply replace NULL values with 0 and then handle is as
+ * a regular value, but that groups NULL and actual 0 values. That's
+ * clearly incorrect - we need to handle NULL values as a separate value.
+ */
+MVDependencies
+build_mv_dependencies(int numrows, HeapTuple *rows, int2vector *attrs,
+ VacAttrStats **stats)
+{
+ int i;
+ int numattrs = attrs->dim1;
+
+ /* result */
+ int ndeps = 0;
+ MVDependencies dependencies = NULL;
+ MultiSortSupport mss = multi_sort_init(2); /* 2 dimensions for now */
+
+ /* TODO Maybe this should be somehow related to the number of
+ * distinct values in the two columns we're currently analyzing.
+ * Assuming the distribution is uniform, we can estimate the
+ * average group size and use it as a threshold. Or something
+ * like that. Seems better than a static approach.
+ */
+ int min_group_size = 3;
+
+ /* dimension indexes we'll check for associations [a => b] */
+ int dima, dimb;
+
+ /*
+ * We'll reuse the same array for all the 2-column combinations.
+ *
+ * It's possible to sort the sample rows directly, but this seemed
+ * somehow simples / less error prone. Another option would be to
+ * allocate the arrays for each SortItem separately, but that'd be
+ * significant overhead (not just CPU, but especially memory bloat).
+ */
+ SortItem * items = (SortItem*)palloc0(numrows * sizeof(SortItem));
+
+ Datum *values = (Datum*)palloc0(sizeof(Datum) * numrows * 2);
+ bool *isnull = (bool*)palloc0(sizeof(bool) * numrows * 2);
+
+ for (i = 0; i < numrows; i++)
+ {
+ items[i].values = &values[i * 2];
+ items[i].isnull = &isnull[i * 2];
+ }
+
+ Assert(numattrs >= 2);
+
+ /*
+ * Evaluate all possible combinations of [A => B], using a simple algorithm:
+ *
+ * (a) sort the data by [A,B]
+ * (b) split the data into groups by A (new group whenever a value changes)
+ * (c) count different values in the B column (again, value changes)
+ *
+ * TODO It should be rather simple to merge [A => B] and [A => C] into
+ * [A => B,C]. Just keep A constant, collect all the "implied" columns
+ * and you're done.
+ */
+ for (dima = 0; dima < numattrs; dima++)
+ {
+ /* prepare the sort function for the first dimension */
+ multi_sort_add_dimension(mss, 0, dima, stats);
+
+ for (dimb = 0; dimb < numattrs; dimb++)
+ {
+ SortItem current;
+
+ /* number of groups supporting / contradicting the dependency */
+ int n_supporting = 0;
+ int n_contradicting = 0;
+
+ /* counters valid within a group */
+ int group_size = 0;
+ int n_violations = 0;
+
+ int n_supporting_rows = 0;
+ int n_contradicting_rows = 0;
+
+ /* make sure the columns are different (A => A) */
+ if (dima == dimb)
+ continue;
+
+ /* prepare the sort function for the second dimension */
+ multi_sort_add_dimension(mss, 1, dimb, stats);
+
+ /* reset the values and isnull flags */
+ memset(values, 0, sizeof(Datum) * numrows * 2);
+ memset(isnull, 0, sizeof(bool) * numrows * 2);
+
+ /* accumulate all the data for both columns into an array and sort it */
+ for (i = 0; i < numrows; i++)
+ {
+ items[i].values[0]
+ = heap_getattr(rows[i], attrs->values[dima],
+ stats[dima]->tupDesc, &items[i].isnull[0]);
+
+ items[i].values[1]
+ = heap_getattr(rows[i], attrs->values[dimb],
+ stats[dimb]->tupDesc, &items[i].isnull[1]);
+ }
+
+ qsort_arg((void *) items, numrows, sizeof(SortItem),
+ multi_sort_compare, mss);
+
+ /*
+ * Walk through the array, split it into rows according to
+ * the A value, and count distinct values in the other one.
+ * If there's a single B value for the whole group, we count
+ * it as supporting the association, otherwise we count it
+ * as contradicting.
+ *
+ * Furthermore we require a group to have at least a certain
+ * number of rows to be considered useful for supporting the
+ * dependency. But when it's contradicting, use it always useful.
+ */
+
+ /* start with values from the first row */
+ current = items[0];
+ group_size = 1;
+
+ for (i = 1; i < numrows; i++)
+ {
+ /* end of the group */
+ if (multi_sort_compare_dim(0, &items[i], ¤t, mss) != 0)
+ {
+ /*
+ * If there are no contradicting rows, count it as
+ * supporting (otherwise contradicting), but only if
+ * the group is large enough.
+ *
+ * The requirement of a minimum group size makes it
+ * impossible to identify [unique,unique] cases, but
+ * that's probably a different case. This is more
+ * about [zip => city] associations etc.
+ *
+ * If there are violations, count the group/rows as
+ * a violation.
+ *
+ * It may ne neither, if the group is too small (does
+ * not contain at least min_group_size rows).
+ */
+ if ((n_violations == 0) && (group_size >= min_group_size))
+ {
+ n_supporting += 1;
+ n_supporting_rows += group_size;
+ }
+ else if (n_violations > 0)
+ {
+ n_contradicting += 1;
+ n_contradicting_rows += group_size;
+ }
+
+ /* current values start a new group */
+ n_violations = 0;
+ group_size = 0;
+ }
+ /* mismatch of a B value is contradicting */
+ else if (multi_sort_compare_dim(1, &items[i], ¤t, mss) != 0)
+ {
+ n_violations += 1;
+ }
+
+ current = items[i];
+ group_size += 1;
+ }
+
+ /* handle the last group (just like above) */
+ if ((n_violations == 0) && (group_size >= min_group_size))
+ {
+ n_supporting += 1;
+ n_supporting_rows += group_size;
+ }
+ else if (n_violations)
+ {
+ n_contradicting += 1;
+ n_contradicting_rows += group_size;
+ }
+
+ /*
+ * See if the number of rows supporting the association is at least
+ * 10x the number of rows violating the hypothetical dependency.
+ *
+ * TODO This is rather arbitrary limit - I guess it's possible to do
+ * some math to come up with a better rule (e.g. testing a hypothesis
+ * 'this is due to randomness'). We can create a contingency table
+ * from the values and use it for testing. Possibly only when
+ * there are no contradicting rows?
+ *
+ * TODO Also, if (a => b) and (b => a) at the same time, it pretty much
+ * means there's a 1:1 relation (or one is a 'label'), making the
+ * conditions rather redundant. Although it's possible that the
+ * query uses incompatible combination of values.
+ */
+ if (n_supporting_rows > (n_contradicting_rows * 10))
+ {
+ if (dependencies == NULL)
+ {
+ dependencies = (MVDependencies)palloc0(sizeof(MVDependenciesData));
+ dependencies->magic = MVSTAT_DEPS_MAGIC;
+ }
+ else
+ dependencies = repalloc(dependencies, offsetof(MVDependenciesData, deps)
+ + sizeof(MVDependency) * (dependencies->ndeps + 1));
+
+ /* update the */
+ dependencies->deps[ndeps] = (MVDependency)palloc0(sizeof(MVDependencyData));
+ dependencies->deps[ndeps]->a = attrs->values[dima];
+ dependencies->deps[ndeps]->b = attrs->values[dimb];
+
+ dependencies->ndeps = (++ndeps);
+ }
+ }
+ }
+
+ pfree(items);
+ pfree(values);
+ pfree(isnull);
+ pfree(stats);
+ pfree(mss);
+
+ return dependencies;
+}
+
+/*
+ * Store the dependencies into a bytea, so that it can be stored in the
+ * pg_mv_statistic catalog.
+ *
+ * Currently this only supports simple two-column rules, and stores them
+ * as a sequence of attnum pairs. In the future, this needs to be made
+ * more complex to support multiple columns on both sides of the
+ * implication (using AND on left, OR on right).
+ */
+bytea *
+serialize_mv_dependencies(MVDependencies dependencies)
+{
+ int i;
+
+ /* we need to store ndeps, and each needs 2 * int16 */
+ Size len = VARHDRSZ + offsetof(MVDependenciesData, deps)
+ + dependencies->ndeps * (sizeof(int16) * 2);
+
+ bytea * output = (bytea*)palloc0(len);
+
+ char * tmp = VARDATA(output);
+
+ SET_VARSIZE(output, len);
+
+ /* first, store the number of dimensions / items */
+ memcpy(tmp, dependencies, offsetof(MVDependenciesData, deps));
+ tmp += offsetof(MVDependenciesData, deps);
+
+ /* walk through the dependencies and copy both columns into the bytea */
+ for (i = 0; i < dependencies->ndeps; i++)
+ {
+ memcpy(tmp, &(dependencies->deps[i]->a), sizeof(int16));
+ tmp += sizeof(int16);
+
+ memcpy(tmp, &(dependencies->deps[i]->b), sizeof(int16));
+ tmp += sizeof(int16);
+ }
+
+ return output;
+}
+
+/*
+ * Reads serialized dependencies into MVDependencies structure.
+ */
+MVDependencies
+deserialize_mv_dependencies(bytea * data)
+{
+ int i;
+ Size expected_size;
+ MVDependencies dependencies;
+ char *tmp;
+
+ if (data == NULL)
+ return NULL;
+
+ if (VARSIZE_ANY_EXHDR(data) < offsetof(MVDependenciesData,deps))
+ elog(ERROR, "invalid MVDependencies size %ld (expected at least %ld)",
+ VARSIZE_ANY_EXHDR(data), offsetof(MVDependenciesData,deps));
+
+ /* read the MVDependencies header */
+ dependencies = (MVDependencies)palloc0(sizeof(MVDependenciesData));
+
+ /* initialize pointer to the data part (skip the varlena header) */
+ tmp = VARDATA(data);
+
+ /* get the header and perform basic sanity checks */
+ memcpy(dependencies, tmp, offsetof(MVDependenciesData, deps));
+ tmp += offsetof(MVDependenciesData, deps);
+
+ if (dependencies->magic != MVSTAT_DEPS_MAGIC)
+ {
+ pfree(dependencies);
+ elog(WARNING, "not a MV Dependencies (magic number mismatch)");
+ return NULL;
+ }
+
+ Assert(dependencies->ndeps > 0);
+
+ /* what bytea size do we expect for those parameters */
+ expected_size = offsetof(MVDependenciesData,deps) +
+ dependencies->ndeps * sizeof(int16) * 2;
+
+ if (VARSIZE_ANY_EXHDR(data) != expected_size)
+ elog(ERROR, "invalid dependencies size %ld (expected %ld)",
+ VARSIZE_ANY_EXHDR(data), expected_size);
+
+ /* allocate space for the MCV items */
+ dependencies = repalloc(dependencies, offsetof(MVDependenciesData,deps)
+ + (dependencies->ndeps * sizeof(MVDependency)));
+
+ for (i = 0; i < dependencies->ndeps; i++)
+ {
+ dependencies->deps[i] = (MVDependency)palloc0(sizeof(MVDependencyData));
+
+ memcpy(&(dependencies->deps[i]->a), tmp, sizeof(int16));
+ tmp += sizeof(int16);
+
+ memcpy(&(dependencies->deps[i]->b), tmp, sizeof(int16));
+ tmp += sizeof(int16);
+ }
+
+ return dependencies;
+}
+
+/* print some basic info about dependencies (number of dependencies) */
+Datum
+pg_mv_stats_dependencies_info(PG_FUNCTION_ARGS)
+{
+ bytea *data = PG_GETARG_BYTEA_P(0);
+ char *result;
+
+ MVDependencies dependencies = deserialize_mv_dependencies(data);
+
+ if (dependencies == NULL)
+ PG_RETURN_NULL();
+
+ result = palloc0(128);
+ snprintf(result, 128, "dependencies=%d", dependencies->ndeps);
+
+ /* FIXME free the deserialized data (pfree is not enough) */
+
+ PG_RETURN_TEXT_P(cstring_to_text(result));
+}
+
+/* print the dependencies
+ *
+ * TODO Would be nice if this knew the actual column names (instead of
+ * the attnums).
+ *
+ * FIXME This is really ugly and does not really check the lengths and
+ * strcpy/snprintf return values properly. Needs to be fixed.
+ */
+Datum
+pg_mv_stats_dependencies_show(PG_FUNCTION_ARGS)
+{
+ int i = 0;
+ bytea *data = PG_GETARG_BYTEA_P(0);
+ char *result = NULL;
+ int len = 0;
+
+ MVDependencies dependencies = deserialize_mv_dependencies(data);
+
+ if (dependencies == NULL)
+ PG_RETURN_NULL();
+
+ for (i = 0; i < dependencies->ndeps; i++)
+ {
+ MVDependency dependency = dependencies->deps[i];
+ char buffer[128];
+
+ int tmp = snprintf(buffer, 128, "%s%d => %d",
+ ((i == 0) ? "" : ", "), dependency->a, dependency->b);
+
+ if (tmp < 127)
+ {
+ if (result == NULL)
+ result = palloc0(len + tmp + 1);
+ else
+ result = repalloc(result, len + tmp + 1);
+
+ strcpy(result + len, buffer);
+ len += tmp;
+ }
+ }
+
+ PG_RETURN_TEXT_P(cstring_to_text(result));
+}
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index fd8dc91..8ce9c0e 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -2104,6 +2104,50 @@ describeOneTableDetails(const char *schemaname,
PQclear(result);
}
+ /* print any multivariate statistics */
+ if (pset.sversion >= 90600)
+ {
+ printfPQExpBuffer(&buf,
+ "SELECT oid, stanamespace::regnamespace AS nsp, staname, stakeys,\n"
+ " deps_enabled,\n"
+ " deps_built,\n"
+ " (SELECT string_agg(attname::text,', ')\n"
+ " FROM ((SELECT unnest(stakeys) AS attnum) s\n"
+ " JOIN pg_attribute a ON (starelid = a.attrelid and a.attnum = s.attnum))) AS attnums\n"
+ "FROM pg_mv_statistic stat WHERE starelid = '%s' ORDER BY 1;",
+ oid);
+
+ result = PSQLexec(buf.data);
+ if (!result)
+ goto error_return;
+ else
+ tuples = PQntuples(result);
+
+ if (tuples > 0)
+ {
+ printTableAddFooter(&cont, _("Statistics:"));
+ for (i = 0; i < tuples; i++)
+ {
+ printfPQExpBuffer(&buf, " ");
+
+ /* statistics name (qualified with namespace) */
+ appendPQExpBuffer(&buf, "\"%s.%s\" ",
+ PQgetvalue(result, i, 1),
+ PQgetvalue(result, i, 2));
+
+ /* options */
+ if (!strcmp(PQgetvalue(result, i, 4), "t"))
+ appendPQExpBuffer(&buf, "(dependencies)");
+
+ appendPQExpBuffer(&buf, " ON (%s)",
+ PQgetvalue(result, i, 6));
+
+ printTableAddFooter(&cont, buf.data);
+ }
+ }
+ PQclear(result);
+ }
+
/* print rules */
if (tableinfo.hasrules && tableinfo.relkind != 'm')
{
diff --git a/src/include/catalog/dependency.h b/src/include/catalog/dependency.h
index 049bf9f..12211fe 100644
--- a/src/include/catalog/dependency.h
+++ b/src/include/catalog/dependency.h
@@ -153,10 +153,11 @@ typedef enum ObjectClass
OCLASS_EXTENSION, /* pg_extension */
OCLASS_EVENT_TRIGGER, /* pg_event_trigger */
OCLASS_POLICY, /* pg_policy */
- OCLASS_TRANSFORM /* pg_transform */
+ OCLASS_TRANSFORM, /* pg_transform */
+ OCLASS_STATISTICS /* pg_mv_statistics */
} ObjectClass;
-#define LAST_OCLASS OCLASS_TRANSFORM
+#define LAST_OCLASS OCLASS_STATISTICS
/* in dependency.c */
diff --git a/src/include/catalog/heap.h b/src/include/catalog/heap.h
index b80d8d8..5ae42f7 100644
--- a/src/include/catalog/heap.h
+++ b/src/include/catalog/heap.h
@@ -119,6 +119,7 @@ extern void RemoveAttrDefault(Oid relid, AttrNumber attnum,
DropBehavior behavior, bool complain, bool internal);
extern void RemoveAttrDefaultById(Oid attrdefId);
extern void RemoveStatistics(Oid relid, AttrNumber attnum);
+extern void RemoveMVStatistics(Oid relid, AttrNumber attnum);
extern Form_pg_attribute SystemAttributeDefinition(AttrNumber attno,
bool relhasoids);
diff --git a/src/include/catalog/indexing.h b/src/include/catalog/indexing.h
index ab2c1a8..a768bb5 100644
--- a/src/include/catalog/indexing.h
+++ b/src/include/catalog/indexing.h
@@ -173,6 +173,13 @@ DECLARE_UNIQUE_INDEX(pg_largeobject_loid_pn_index, 2683, on pg_largeobject using
DECLARE_UNIQUE_INDEX(pg_largeobject_metadata_oid_index, 2996, on pg_largeobject_metadata using btree(oid oid_ops));
#define LargeObjectMetadataOidIndexId 2996
+DECLARE_UNIQUE_INDEX(pg_mv_statistic_oid_index, 3380, on pg_mv_statistic using btree(oid oid_ops));
+#define MvStatisticOidIndexId 3380
+DECLARE_UNIQUE_INDEX(pg_mv_statistic_name_index, 3997, on pg_mv_statistic using btree(staname name_ops, stanamespace oid_ops));
+#define MvStatisticNameIndexId 3997
+DECLARE_INDEX(pg_mv_statistic_relid_index, 3379, on pg_mv_statistic using btree(starelid oid_ops));
+#define MvStatisticRelidIndexId 3379
+
DECLARE_UNIQUE_INDEX(pg_namespace_nspname_index, 2684, on pg_namespace using btree(nspname name_ops));
#define NamespaceNameIndexId 2684
DECLARE_UNIQUE_INDEX(pg_namespace_oid_index, 2685, on pg_namespace using btree(oid oid_ops));
diff --git a/src/include/catalog/namespace.h b/src/include/catalog/namespace.h
index 2ccb3a7..44cf9c6 100644
--- a/src/include/catalog/namespace.h
+++ b/src/include/catalog/namespace.h
@@ -137,6 +137,8 @@ extern Oid get_collation_oid(List *collname, bool missing_ok);
extern Oid get_conversion_oid(List *conname, bool missing_ok);
extern Oid FindDefaultConversionProc(int32 for_encoding, int32 to_encoding);
+extern Oid get_statistics_oid(List *names, bool missing_ok);
+
/* initialization & transaction cleanup code */
extern void InitializeSearchPath(void);
extern void AtEOXact_Namespace(bool isCommit, bool parallel);
diff --git a/src/include/catalog/pg_mv_statistic.h b/src/include/catalog/pg_mv_statistic.h
new file mode 100644
index 0000000..c74af47
--- /dev/null
+++ b/src/include/catalog/pg_mv_statistic.h
@@ -0,0 +1,75 @@
+/*-------------------------------------------------------------------------
+ *
+ * pg_mv_statistic.h
+ * definition of the system "multivariate statistic" relation (pg_mv_statistic)
+ * along with the relation's initial contents.
+ *
+ *
+ * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/catalog/pg_mv_statistic.h
+ *
+ * NOTES
+ * the genbki.pl script reads this file and generates .bki
+ * information from the DATA() statements.
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef PG_MV_STATISTIC_H
+#define PG_MV_STATISTIC_H
+
+#include "catalog/genbki.h"
+
+/* ----------------
+ * pg_mv_statistic definition. cpp turns this into
+ * typedef struct FormData_pg_mv_statistic
+ * ----------------
+ */
+#define MvStatisticRelationId 3381
+
+CATALOG(pg_mv_statistic,3381)
+{
+ /* These fields form the unique key for the entry: */
+ Oid starelid; /* relation containing attributes */
+ NameData staname; /* statistics name */
+ Oid stanamespace; /* OID of namespace containing this statistics */
+ Oid staowner; /* statistics owner */
+
+ /* statistics requested to build */
+ bool deps_enabled; /* analyze dependencies? */
+
+ /* statistics that are available (if requested) */
+ bool deps_built; /* dependencies were built */
+
+ /* variable-length fields start here, but we allow direct access to stakeys */
+ int2vector stakeys; /* array of column keys */
+
+#ifdef CATALOG_VARLEN
+ bytea stadeps; /* dependencies (serialized) */
+#endif
+
+} FormData_pg_mv_statistic;
+
+/* ----------------
+ * Form_pg_mv_statistic corresponds to a pointer to a tuple with
+ * the format of pg_mv_statistic relation.
+ * ----------------
+ */
+typedef FormData_pg_mv_statistic *Form_pg_mv_statistic;
+
+/* ----------------
+ * compiler constants for pg_mv_statistic
+ * ----------------
+ */
+#define Natts_pg_mv_statistic 8
+#define Anum_pg_mv_statistic_starelid 1
+#define Anum_pg_mv_statistic_staname 2
+#define Anum_pg_mv_statistic_stanamespace 3
+#define Anum_pg_mv_statistic_staowner 4
+#define Anum_pg_mv_statistic_deps_enabled 5
+#define Anum_pg_mv_statistic_deps_built 6
+#define Anum_pg_mv_statistic_stakeys 7
+#define Anum_pg_mv_statistic_stadeps 8
+
+#endif /* PG_MV_STATISTIC_H */
diff --git a/src/include/catalog/pg_proc.h b/src/include/catalog/pg_proc.h
index cbbb883..eecce40 100644
--- a/src/include/catalog/pg_proc.h
+++ b/src/include/catalog/pg_proc.h
@@ -2666,6 +2666,11 @@ DESCR("current user privilege on any column by rel name");
DATA(insert OID = 3029 ( has_any_column_privilege PGNSP PGUID 12 10 0 0 0 f f f f t f s s 2 0 16 "26 25" _null_ _null_ _null_ _null_ _null_ has_any_column_privilege_id _null_ _null_ _null_ ));
DESCR("current user privilege on any column by rel oid");
+DATA(insert OID = 3998 ( pg_mv_stats_dependencies_info PGNSP PGUID 12 1 0 0 0 f f f f t f i s 1 0 25 "17" _null_ _null_ _null_ _null_ _null_ pg_mv_stats_dependencies_info _null_ _null_ _null_ ));
+DESCR("multivariate stats: functional dependencies info");
+DATA(insert OID = 3999 ( pg_mv_stats_dependencies_show PGNSP PGUID 12 1 0 0 0 f f f f t f i s 1 0 25 "17" _null_ _null_ _null_ _null_ _null_ pg_mv_stats_dependencies_show _null_ _null_ _null_ ));
+DESCR("multivariate stats: functional dependencies show");
+
DATA(insert OID = 1928 ( pg_stat_get_numscans PGNSP PGUID 12 1 0 0 0 f f f f t f s r 1 0 20 "26" _null_ _null_ _null_ _null_ _null_ pg_stat_get_numscans _null_ _null_ _null_ ));
DESCR("statistics: number of scans done for table/index");
DATA(insert OID = 1929 ( pg_stat_get_tuples_returned PGNSP PGUID 12 1 0 0 0 f f f f t f s r 1 0 20 "26" _null_ _null_ _null_ _null_ _null_ pg_stat_get_tuples_returned _null_ _null_ _null_ ));
diff --git a/src/include/catalog/toasting.h b/src/include/catalog/toasting.h
index b7a38ce..a52096b 100644
--- a/src/include/catalog/toasting.h
+++ b/src/include/catalog/toasting.h
@@ -49,6 +49,7 @@ extern void BootstrapToastTable(char *relName,
DECLARE_TOAST(pg_attrdef, 2830, 2831);
DECLARE_TOAST(pg_constraint, 2832, 2833);
DECLARE_TOAST(pg_description, 2834, 2835);
+DECLARE_TOAST(pg_mv_statistic, 3577, 3578);
DECLARE_TOAST(pg_proc, 2836, 2837);
DECLARE_TOAST(pg_rewrite, 2838, 2839);
DECLARE_TOAST(pg_seclabel, 3598, 3599);
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index 54f67e9..99a6a62 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -75,6 +75,10 @@ extern ObjectAddress DefineOperator(List *names, List *parameters);
extern void RemoveOperatorById(Oid operOid);
extern ObjectAddress AlterOperator(AlterOperatorStmt *stmt);
+/* commands/statscmds.c */
+extern ObjectAddress CreateStatistics(CreateStatsStmt *stmt);
+extern void RemoveStatisticsById(Oid statsOid);
+
/* commands/aggregatecmds.c */
extern ObjectAddress DefineAggregate(List *name, List *args, bool oldstyle,
List *parameters, const char *queryString);
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index fad9988..545b62a 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -266,6 +266,7 @@ typedef enum NodeTag
T_PlaceHolderInfo,
T_MinMaxAggInfo,
T_PlannerParamItem,
+ T_MVStatisticInfo,
/*
* TAGS FOR MEMORY NODES (memnodes.h)
@@ -401,6 +402,7 @@ typedef enum NodeTag
T_CreatePolicyStmt,
T_AlterPolicyStmt,
T_CreateTransformStmt,
+ T_CreateStatsStmt,
/*
* TAGS FOR PARSE TREE NODES (parsenodes.h)
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 2fd0629..e1807fb 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -601,6 +601,17 @@ typedef struct ColumnDef
int location; /* parse location, or -1 if none/unknown */
} ColumnDef;
+typedef struct CreateStatsStmt
+{
+ NodeTag type;
+ List *defnames; /* qualified name (list of Value strings) */
+ RangeVar *relation; /* relation to build statistics on */
+ List *keys; /* String nodes naming referenced column(s) */
+ List *options; /* list of DefElem nodes */
+ bool if_not_exists; /* just do nothing if statistics already exists? */
+} CreateStatsStmt;
+
+
/*
* TableLikeClause - CREATE TABLE ( ... LIKE ... ) clause
*/
@@ -1410,6 +1421,7 @@ typedef enum ObjectType
OBJECT_RULE,
OBJECT_SCHEMA,
OBJECT_SEQUENCE,
+ OBJECT_STATISTICS,
OBJECT_TABCONSTRAINT,
OBJECT_TABLE,
OBJECT_TABLESPACE,
diff --git a/src/include/nodes/relation.h b/src/include/nodes/relation.h
index 641728b..e10dcf1 100644
--- a/src/include/nodes/relation.h
+++ b/src/include/nodes/relation.h
@@ -539,6 +539,7 @@ typedef struct RelOptInfo
List *lateral_vars; /* LATERAL Vars and PHVs referenced by rel */
Relids lateral_referencers; /* rels that reference me laterally */
List *indexlist; /* list of IndexOptInfo */
+ List *mvstatlist; /* list of MVStatisticInfo */
BlockNumber pages; /* size estimates derived from pg_class */
double tuples;
double allvisfrac;
@@ -634,6 +635,33 @@ typedef struct IndexOptInfo
void (*amcostestimate) (); /* AM's cost estimator */
} IndexOptInfo;
+/*
+ * MVStatisticInfo
+ * Information about multivariate stats for planning/optimization
+ *
+ * This contains information about which columns are covered by the
+ * statistics (stakeys), which options were requested while adding the
+ * statistics (*_enabled), and which kinds of statistics were actually
+ * built and are available for the optimizer (*_built).
+ */
+typedef struct MVStatisticInfo
+{
+ NodeTag type;
+
+ Oid mvoid; /* OID of the statistics row */
+ RelOptInfo *rel; /* back-link to index's table */
+
+ /* enabled statistics */
+ bool deps_enabled; /* functional dependencies enabled */
+
+ /* built/available statistics */
+ bool deps_built; /* functional dependencies built */
+
+ /* columns in the statistics (attnums) */
+ int2vector *stakeys; /* attnums of the columns covered */
+
+} MVStatisticInfo;
+
/*
* EquivalenceClasses
diff --git a/src/include/utils/acl.h b/src/include/utils/acl.h
index 4e15a14..3e11253 100644
--- a/src/include/utils/acl.h
+++ b/src/include/utils/acl.h
@@ -330,6 +330,7 @@ extern bool pg_foreign_data_wrapper_ownercheck(Oid srv_oid, Oid roleid);
extern bool pg_foreign_server_ownercheck(Oid srv_oid, Oid roleid);
extern bool pg_event_trigger_ownercheck(Oid et_oid, Oid roleid);
extern bool pg_extension_ownercheck(Oid ext_oid, Oid roleid);
+extern bool pg_statistics_ownercheck(Oid stat_oid, Oid roleid);
extern bool has_createrole_privilege(Oid roleid);
extern bool has_bypassrls_privilege(Oid roleid);
diff --git a/src/include/utils/mvstats.h b/src/include/utils/mvstats.h
new file mode 100644
index 0000000..7ebd961
--- /dev/null
+++ b/src/include/utils/mvstats.h
@@ -0,0 +1,70 @@
+/*-------------------------------------------------------------------------
+ *
+ * mvstats.h
+ * Multivariate statistics and selectivity estimation functions.
+ *
+ *
+ * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/utils/mvstats.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef MVSTATS_H
+#define MVSTATS_H
+
+#include "fmgr.h"
+#include "commands/vacuum.h"
+
+
+#define MVSTATS_MAX_DIMENSIONS 8 /* max number of attributes */
+
+/* An associative rule, tracking [a => b] dependency.
+ *
+ * TODO Make this work with multiple columns on both sides.
+ */
+typedef struct MVDependencyData {
+ int16 a;
+ int16 b;
+} MVDependencyData;
+
+typedef MVDependencyData* MVDependency;
+
+typedef struct MVDependenciesData {
+ uint32 magic; /* magic constant marker */
+ int32 ndeps; /* number of dependencies */
+ MVDependency deps[1]; /* XXX why not a pointer? */
+} MVDependenciesData;
+
+typedef MVDependenciesData* MVDependencies;
+
+#define MVSTAT_DEPS_MAGIC 0xB4549A2C /* marks serialized bytea */
+#define MVSTAT_DEPS_TYPE_BASIC 1 /* basic dependencies type */
+
+/*
+ * TODO Maybe fetching the histogram/MCV list separately is inefficient?
+ * Consider adding a single `fetch_stats` method, fetching all
+ * stats specified using flags (or something like that).
+ */
+
+bytea * serialize_mv_dependencies(MVDependencies dependencies);
+
+/* deserialization of stats (serialization is private to analyze) */
+MVDependencies deserialize_mv_dependencies(bytea * data);
+
+/* FIXME this probably belongs somewhere else (not to operations stats) */
+extern Datum pg_mv_stats_dependencies_info(PG_FUNCTION_ARGS);
+extern Datum pg_mv_stats_dependencies_show(PG_FUNCTION_ARGS);
+
+MVDependencies
+build_mv_dependencies(int numrows, HeapTuple *rows,
+ int2vector *attrs,
+ VacAttrStats **stats);
+
+void build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
+ int natts, VacAttrStats **vacattrstats);
+
+void update_mv_stats(Oid relid, MVDependencies dependencies, int2vector *attrs);
+
+#endif
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index f2bebf2..8771f9c 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -61,6 +61,7 @@ typedef struct RelationData
bool rd_isvalid; /* relcache entry is valid */
char rd_indexvalid; /* state of rd_indexlist: 0 = not valid, 1 =
* valid, 2 = temporarily forced */
+ bool rd_mvstatvalid; /* state of rd_mvstatlist: true/false */
/*
* rd_createSubid is the ID of the highest subtransaction the rel has
@@ -93,6 +94,9 @@ typedef struct RelationData
List *rd_indexlist; /* list of OIDs of indexes on relation */
Oid rd_oidindex; /* OID of unique index on OID, if any */
Oid rd_replidindex; /* OID of replica identity index, if any */
+
+ /* data managed by RelationGetMVStatList: */
+ List *rd_mvstatlist; /* list of OIDs of multivariate stats */
/* data managed by RelationGetIndexAttrBitmap: */
Bitmapset *rd_indexattr; /* identifies columns used in indexes */
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 1b48304..9f03c8d 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -38,6 +38,7 @@ extern void RelationClose(Relation relation);
* Routines to compute/retrieve additional cached information
*/
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetMVStatList(Relation relation);
extern Oid RelationGetOidIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
extern List *RelationGetIndexExpressions(Relation relation);
diff --git a/src/include/utils/syscache.h b/src/include/utils/syscache.h
index 256615b..0e0658d 100644
--- a/src/include/utils/syscache.h
+++ b/src/include/utils/syscache.h
@@ -66,6 +66,8 @@ enum SysCacheIdentifier
INDEXRELID,
LANGNAME,
LANGOID,
+ MVSTATNAMENSP,
+ MVSTATOID,
NAMESPACENAME,
NAMESPACEOID,
OPERNAMENSP,
diff --git a/src/test/regress/expected/object_address.out b/src/test/regress/expected/object_address.out
index 75751be..eb60960 100644
--- a/src/test/regress/expected/object_address.out
+++ b/src/test/regress/expected/object_address.out
@@ -35,6 +35,7 @@ ALTER DEFAULT PRIVILEGES FOR ROLE regtest_addr_user REVOKE DELETE ON TABLES FROM
CREATE TRANSFORM FOR int LANGUAGE SQL (
FROM SQL WITH FUNCTION varchar_transform(internal),
TO SQL WITH FUNCTION int4recv(internal));
+CREATE STATISTICS addr_nsp.gentable_stat ON addr_nsp.gentable(a,b) WITH (dependencies);
-- test some error cases
SELECT pg_get_object_address('stone', '{}', '{}');
ERROR: unrecognized object type "stone"
@@ -373,7 +374,8 @@ WITH objects (type, name, args) AS (VALUES
-- extension
-- event trigger
('policy', '{addr_nsp, gentable, genpol}', '{}'),
- ('transform', '{int}', '{sql}')
+ ('transform', '{int}', '{sql}'),
+ ('statistics', '{addr_nsp, gentable_stat}', '{}')
)
SELECT (pg_identify_object(addr1.classid, addr1.objid, addr1.subobjid)).*,
-- test roundtrip through pg_identify_object_as_address
@@ -420,13 +422,14 @@ SELECT (pg_identify_object(addr1.classid, addr1.objid, addr1.subobjid)).*,
trigger | | | t on addr_nsp.gentable | t
operator family | pg_catalog | integer_ops | pg_catalog.integer_ops USING btree | t
policy | | | genpol on addr_nsp.gentable | t
+ statistics | addr_nsp | gentable_stat | addr_nsp.gentable_stat | t
collation | pg_catalog | "default" | pg_catalog."default" | t
transform | | | for integer on language sql | t
text search dictionary | addr_nsp | addr_ts_dict | addr_nsp.addr_ts_dict | t
text search parser | addr_nsp | addr_ts_prs | addr_nsp.addr_ts_prs | t
text search configuration | addr_nsp | addr_ts_conf | addr_nsp.addr_ts_conf | t
text search template | addr_nsp | addr_ts_temp | addr_nsp.addr_ts_temp | t
-(41 rows)
+(42 rows)
---
--- Cleanup resources
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 81bc5c9..84b4425 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1368,6 +1368,15 @@ pg_matviews| SELECT n.nspname AS schemaname,
LEFT JOIN pg_namespace n ON ((n.oid = c.relnamespace)))
LEFT JOIN pg_tablespace t ON ((t.oid = c.reltablespace)))
WHERE (c.relkind = 'm'::"char");
+pg_mv_stats| SELECT n.nspname AS schemaname,
+ c.relname AS tablename,
+ s.staname,
+ s.stakeys AS attnums,
+ length(s.stadeps) AS depsbytes,
+ pg_mv_stats_dependencies_info(s.stadeps) AS depsinfo
+ FROM ((pg_mv_statistic s
+ JOIN pg_class c ON ((c.oid = s.starelid)))
+ LEFT JOIN pg_namespace n ON ((n.oid = c.relnamespace)));
pg_policies| SELECT n.nspname AS schemaname,
c.relname AS tablename,
pol.polname AS policyname,
diff --git a/src/test/regress/expected/sanity_check.out b/src/test/regress/expected/sanity_check.out
index eb0bc88..92a0d8a 100644
--- a/src/test/regress/expected/sanity_check.out
+++ b/src/test/regress/expected/sanity_check.out
@@ -113,6 +113,7 @@ pg_inherits|t
pg_language|t
pg_largeobject|t
pg_largeobject_metadata|t
+pg_mv_statistic|t
pg_namespace|t
pg_opclass|t
pg_operator|t
diff --git a/src/test/regress/sql/object_address.sql b/src/test/regress/sql/object_address.sql
index 68e7cb0..3775b28 100644
--- a/src/test/regress/sql/object_address.sql
+++ b/src/test/regress/sql/object_address.sql
@@ -39,6 +39,7 @@ ALTER DEFAULT PRIVILEGES FOR ROLE regtest_addr_user REVOKE DELETE ON TABLES FROM
CREATE TRANSFORM FOR int LANGUAGE SQL (
FROM SQL WITH FUNCTION varchar_transform(internal),
TO SQL WITH FUNCTION int4recv(internal));
+CREATE STATISTICS addr_nsp.gentable_stat ON addr_nsp.gentable(a,b) WITH (dependencies);
-- test some error cases
SELECT pg_get_object_address('stone', '{}', '{}');
@@ -166,7 +167,8 @@ WITH objects (type, name, args) AS (VALUES
-- extension
-- event trigger
('policy', '{addr_nsp, gentable, genpol}', '{}'),
- ('transform', '{int}', '{sql}')
+ ('transform', '{int}', '{sql}'),
+ ('statistics', '{addr_nsp, gentable_stat}', '{}')
)
SELECT (pg_identify_object(addr1.classid, addr1.objid, addr1.subobjid)).*,
-- test roundtrip through pg_identify_object_as_address
--
2.1.0
[text/x-patch] 0003-clause-reduction-using-functional-dependencies.patch (47.7K, ../../1457535779.24545.71.camel@2ndquadrant.com/4-0003-clause-reduction-using-functional-dependencies.patch)
download | inline diff:
From 2433b5b3cb25a093f78857adb7f9c0b12ac88967 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@pgaddict.com>
Date: Mon, 6 Apr 2015 19:42:18 +0200
Subject: [PATCH 3/9] clause reduction using functional dependencies
During planning, use functional dependencies to decide which clauses to
skip during cardinality estimation. Initial and rather simplistic
implementation.
This only works with regular WHERE clauses, not clauses used for join
clauses.
Note: The clause_is_mv_compatible() needs to identify the relation (so
that we can fetch the list of multivariate stats by OID).
planner_rt_fetch() seems like the appropriate way to get the relation
OID, but apparently it only works with simple vars. Maybe
examine_variable() would make this work with more complex vars too?
Includes regression tests analyzing functional dependencies (part of
ANALYZE) on several datasets (no dependencies, no transitive
dependencies, ...).
Checks that a query with conditions on two columns, where one (B) is
functionally dependent on the other one (A), correctly ignores the
clause on (B) and chooses bitmap index scan instead of plain index scan
(which is what happens otherwise, thanks to assumption of
independence).
Note: Functional dependencies only work with equality clauses, no
inequalities etc.
---
src/backend/optimizer/path/clausesel.c | 891 +++++++++++++++++++++++++-
src/backend/utils/mvstats/README.stats | 36 ++
src/backend/utils/mvstats/common.c | 5 +-
src/backend/utils/mvstats/dependencies.c | 24 +
src/include/utils/mvstats.h | 16 +-
src/test/regress/expected/mv_dependencies.out | 172 +++++
src/test/regress/parallel_schedule | 3 +
src/test/regress/serial_schedule | 1 +
src/test/regress/sql/mv_dependencies.sql | 150 +++++
9 files changed, 1293 insertions(+), 5 deletions(-)
create mode 100644 src/backend/utils/mvstats/README.stats
create mode 100644 src/test/regress/expected/mv_dependencies.out
create mode 100644 src/test/regress/sql/mv_dependencies.sql
diff --git a/src/backend/optimizer/path/clausesel.c b/src/backend/optimizer/path/clausesel.c
index 02660c2..80708fe 100644
--- a/src/backend/optimizer/path/clausesel.c
+++ b/src/backend/optimizer/path/clausesel.c
@@ -14,14 +14,19 @@
*/
#include "postgres.h"
+#include "access/sysattr.h"
+#include "catalog/pg_operator.h"
#include "nodes/makefuncs.h"
#include "optimizer/clauses.h"
#include "optimizer/cost.h"
#include "optimizer/pathnode.h"
#include "optimizer/plancat.h"
+#include "optimizer/var.h"
#include "utils/fmgroids.h"
#include "utils/lsyscache.h"
+#include "utils/mvstats.h"
#include "utils/selfuncs.h"
+#include "utils/typcache.h"
/*
@@ -41,6 +46,23 @@ typedef struct RangeQueryClause
static void addRangeClause(RangeQueryClause **rqlist, Node *clause,
bool varonleft, bool isLTsel, Selectivity s2);
+#define MV_CLAUSE_TYPE_FDEP 0x01
+
+static bool clause_is_mv_compatible(Node *clause, Index relid, AttrNumber *attnum);
+
+static Bitmapset *collect_mv_attnums(List *clauses, Index relid);
+
+static int count_mv_attnums(List *clauses, Index relid);
+
+static int count_varnos(List *clauses, Index *relid);
+
+static List *clauselist_apply_dependencies(PlannerInfo *root, List *clauses,
+ Index relid, List *stats);
+
+static bool has_stats(List *stats, int type);
+
+static List * find_stats(PlannerInfo *root, Index relid);
+
/****************************************************************************
* ROUTINES TO COMPUTE SELECTIVITIES
@@ -60,7 +82,19 @@ static void addRangeClause(RangeQueryClause **rqlist, Node *clause,
* subclauses. However, that's only right if the subclauses have independent
* probabilities, and in reality they are often NOT independent. So,
* we want to be smarter where we can.
-
+ *
+ * The first thing we try to do is applying multivariate statistics, in a way
+ * that intends to minimize the overhead when there are no multivariate stats
+ * on the relation. Thus we do several simple (and inexpensive) checks first,
+ * to verify that suitable multivariate statistics exist.
+ *
+ * If we identify such multivariate statistics apply, we try to apply them.
+ * Currently we only have (soft) functional dependencies, so we try to reduce
+ * the list of clauses.
+ *
+ * Then we remove the clauses estimated using multivariate stats, and process
+ * the rest of the clauses using the regular per-column stats.
+ *
* Currently, the only extra smarts we have is to recognize "range queries",
* such as "x > 34 AND x < 42". Clauses are recognized as possible range
* query components if they are restriction opclauses whose operators have
@@ -99,6 +133,22 @@ clauselist_selectivity(PlannerInfo *root,
RangeQueryClause *rqlist = NULL;
ListCell *l;
+ /* processing mv stats */
+ Oid relid = InvalidOid;
+
+ /* list of multivariate stats on the relation */
+ List *stats = NIL;
+
+ /*
+ * To fetch the statistics, we first need to determine the rel. Currently
+ * point we only support estimates of simple restrictions with all Vars
+ * referencing a single baserel. However set_baserel_size_estimates() sets
+ * varRelid=0 so we have to actually inspect the clauses by pull_varnos
+ * and see if there's just a single varno referenced.
+ */
+ if ((count_varnos(clauses, &relid) == 1) && ((varRelid == 0) || (varRelid == relid)))
+ stats = find_stats(root, relid);
+
/*
* If there's exactly one clause, then no use in trying to match up pairs,
* so just go directly to clause_selectivity().
@@ -108,6 +158,24 @@ clauselist_selectivity(PlannerInfo *root,
varRelid, jointype, sjinfo);
/*
+ * Apply functional dependencies, but first check that there are some stats
+ * with functional dependencies built (by simply walking the stats list),
+ * and that there are at two or more attributes referenced by clauses that
+ * may be reduced using functional dependencies.
+ *
+ * We would find that anyway when trying to actually apply the functional
+ * dependencies, but let's do the cheap checks first.
+ *
+ * After applying the functional dependencies we get the remainig clauses
+ * that need to be estimated by other types of stats (MCV, histograms etc).
+ */
+ if (has_stats(stats, MV_CLAUSE_TYPE_FDEP) &&
+ (count_mv_attnums(clauses, relid) >= 2))
+ {
+ clauses = clauselist_apply_dependencies(root, clauses, relid, stats);
+ }
+
+ /*
* Initial scan over clauses. Anything that doesn't look like a potential
* rangequery clause gets multiplied into s1 and forgotten. Anything that
* does gets inserted into an rqlist entry.
@@ -763,3 +831,824 @@ clause_selectivity(PlannerInfo *root,
return s1;
}
+
+/*
+ * Pull varattnos from the clauses, similarly to pull_varattnos() but:
+ *
+ * (a) only get attributes for a particular relation (relid)
+ * (b) ignore system attributes (we can't build stats on them anyway)
+ *
+ * This makes it possible to directly compare the result with attnum
+ * values from pg_attribute etc.
+ */
+static Bitmapset *
+get_varattnos(Node * node, Index relid)
+{
+ int k;
+ Bitmapset *varattnos = NULL;
+ Bitmapset *result = NULL;
+
+ /* get the varattnos */
+ pull_varattnos(node, relid, &varattnos);
+
+ k = -1;
+ while ((k = bms_next_member(varattnos, k)) >= 0)
+ {
+ if (k + FirstLowInvalidHeapAttributeNumber > 0)
+ result
+ = bms_add_member(result,
+ k + FirstLowInvalidHeapAttributeNumber);
+ }
+
+ bms_free(varattnos);
+
+ return result;
+}
+
+/*
+ * Collect attributes from mv-compatible clauses.
+ */
+static Bitmapset *
+collect_mv_attnums(List *clauses, Index relid)
+{
+ Bitmapset *attnums = NULL;
+ ListCell *l;
+
+ /*
+ * Walk through the clauses and identify the ones we can estimate
+ * using multivariate stats, and remember the relid/columns. We'll
+ * then cross-check if we have suitable stats, and only if needed
+ * we'll split the clauses into multivariate and regular lists.
+ *
+ * For now we're only interested in RestrictInfo nodes with nested
+ * OpExpr, using either a range or equality.
+ */
+ foreach (l, clauses)
+ {
+ AttrNumber attnum;
+ Node *clause = (Node *) lfirst(l);
+
+ /* ignore the result for now - we only need the info */
+ if (clause_is_mv_compatible(clause, relid, &attnum))
+ attnums = bms_add_member(attnums, attnum);
+ }
+
+ /*
+ * If there are not at least two attributes referenced by the clause(s),
+ * we can throw everything out (as we'll revert to simple stats).
+ */
+ if (bms_num_members(attnums) <= 1)
+ {
+ if (attnums != NULL)
+ pfree(attnums);
+ attnums = NULL;
+ }
+
+ return attnums;
+}
+
+/*
+ * Count the number of attributes in clauses compatible with multivariate stats.
+ */
+static int
+count_mv_attnums(List *clauses, Index relid)
+{
+ int c;
+ Bitmapset *attnums = collect_mv_attnums(clauses, relid);
+
+ c = bms_num_members(attnums);
+
+ bms_free(attnums);
+
+ return c;
+}
+
+/*
+ * Count varnos referenced in the clauses, and if there's a single varno then
+ * return the index in 'relid'.
+ */
+static int
+count_varnos(List *clauses, Index *relid)
+{
+ int cnt;
+ Bitmapset *varnos = NULL;
+
+ varnos = pull_varnos((Node *) clauses);
+ cnt = bms_num_members(varnos);
+
+ /* if there's a single varno in the clauses, remember it */
+ if (bms_num_members(varnos) == 1)
+ *relid = bms_singleton_member(varnos);
+
+ bms_free(varnos);
+
+ return cnt;
+}
+
+typedef struct
+{
+ Index varno; /* relid we're interested in */
+ Bitmapset *varattnos; /* attnums referenced by the clauses */
+} mv_compatible_context;
+
+/*
+ * Recursive walker that checks compatibility of the clause with multivariate
+ * statistics, and collects attnums from the Vars.
+ *
+ * XXX The original idea was to combine this with expression_tree_walker, but
+ * I've been unable to make that work - seems that does not quite allow
+ * checking the structure. Hence the explicit calls to the walker.
+ */
+static bool
+mv_compatible_walker(Node *node, mv_compatible_context *context)
+{
+ if (node == NULL)
+ return false;
+
+ if (IsA(node, RestrictInfo))
+ {
+ RestrictInfo *rinfo = (RestrictInfo *) node;
+
+ /* Pseudoconstants are not really interesting here. */
+ if (rinfo->pseudoconstant)
+ return true;
+
+ /* clauses referencing multiple varnos are incompatible */
+ if (bms_membership(rinfo->clause_relids) != BMS_SINGLETON)
+ return true;
+
+ /* check the clause inside the RestrictInfo */
+ return mv_compatible_walker((Node*)rinfo->clause, (void *) context);
+ }
+
+ if (IsA(node, Var))
+ {
+ Var * var = (Var*)node;
+
+ /*
+ * Also, the variable needs to reference the right relid (this might be
+ * unnecessary given the other checks, but let's be sure).
+ */
+ if (var->varno != context->varno)
+ return true;
+
+ /* Also skip system attributes (we don't allow stats on those). */
+ if (! AttrNumberIsForUserDefinedAttr(var->varattno))
+ return true;
+
+ /* Seems fine, so let's remember the attnum. */
+ context->varattnos = bms_add_member(context->varattnos, var->varattno);
+
+ return false;
+ }
+
+ /*
+ * And finally the operator expressions - we only allow simple expressions
+ * with two arguments, where one is a Var and the other is a constant, and
+ * it's a simple comparison (which we detect using estimator function).
+ */
+ if (is_opclause(node))
+ {
+ OpExpr *expr = (OpExpr *) node;
+ Var *var;
+ bool varonleft = true;
+ bool ok;
+
+ /*
+ * Only expressions with two arguments are considered compatible.
+ *
+ * XXX Possibly unnecessary (can OpExpr have different arg count?).
+ */
+ if (list_length(expr->args) != 2)
+ return true;
+
+ /* see if it actually has the right */
+ ok = (NumRelids((Node*)expr) == 1) &&
+ (is_pseudo_constant_clause(lsecond(expr->args)) ||
+ (varonleft = false,
+ is_pseudo_constant_clause(linitial(expr->args))));
+
+ /* unsupported structure (two variables or so) */
+ if (! ok)
+ return true;
+
+ /*
+ * If it's not a "<" or ">" or "=" operator, just ignore the clause.
+ * Otherwise note the relid and attnum for the variable. This uses the
+ * function for estimating selectivity, ont the operator directly (a bit
+ * awkward, but well ...).
+ */
+ switch (get_oprrest(expr->opno))
+ {
+ case F_EQSEL:
+
+ /* equality conditions are compatible with all statistics */
+ break;
+
+ default:
+
+ /* unknown estimator */
+ return true;
+ }
+
+ var = (varonleft) ? linitial(expr->args) : lsecond(expr->args);
+
+ return mv_compatible_walker((Node *) var, context);
+ }
+
+ /* Node not explicitly supported, so terminate */
+ return true;
+}
+
+/*
+ * Determines whether the clause is compatible with multivariate stats,
+ * and if it is, returns some additional information - varno (index
+ * into simple_rte_array) and a bitmap of attributes. This is then
+ * used to fetch related multivariate statistics.
+ *
+ * At this moment we only support basic conditions of the form
+ *
+ * variable OP constant
+ *
+ * where OP is one of [=,<,<=,>=,>] (which is however determined by
+ * looking at the associated function for estimating selectivity, just
+ * like with the single-dimensional case).
+ *
+ * TODO Support 'OR clauses' - shouldn't be all that difficult to
+ * evaluate them using multivariate stats.
+ */
+static bool
+clause_is_mv_compatible(Node *clause, Index relid, AttrNumber *attnum)
+{
+ mv_compatible_context context;
+
+ context.varno = relid;
+ context.varattnos = NULL; /* no attnums */
+
+ if (mv_compatible_walker(clause, (void *) &context))
+ return false;
+
+ /* remember the newly collected attnums */
+ *attnum = bms_singleton_member(context.varattnos);
+
+ return true;
+}
+
+/*
+ * collect attnums from functional dependencies
+ *
+ * Walk through all statistics on the relation, and collect attnums covered
+ * by those with functional dependencies. We only look at columns specified
+ * when creating the statistics, not at columns actually referenced by the
+ * dependencies (which may only be a subset of the attributes).
+ */
+static Bitmapset*
+fdeps_collect_attnums(List *stats)
+{
+ ListCell *lc;
+ Bitmapset *attnums = NULL;
+
+ foreach (lc, stats)
+ {
+ int j;
+ MVStatisticInfo *info = (MVStatisticInfo *)lfirst(lc);
+
+ int2vector *stakeys = info->stakeys;
+
+ /* skip stats without functional dependencies built */
+ if (! info->deps_built)
+ continue;
+
+ for (j = 0; j < stakeys->dim1; j++)
+ attnums = bms_add_member(attnums, stakeys->values[j]);
+ }
+
+ return attnums;
+}
+
+/* transforms bitmapset into an array (index => value) */
+static int*
+make_idx_to_attnum_mapping(Bitmapset *attnums)
+{
+ int attidx = 0;
+ int attnum;
+
+ int *mapping = (int*)palloc0(bms_num_members(attnums) * sizeof(int));
+
+ attnum = -1;
+ while ((attnum = bms_next_member(attnums, attnum)) >= 0)
+ mapping[attidx++] = attnum;
+
+ Assert(attidx == bms_num_members(attnums));
+
+ return mapping;
+}
+
+/* transforms bitmapset into an array (value => index) */
+static int*
+make_attnum_to_idx_mapping(Bitmapset *attnums)
+{
+ int attidx = 0;
+ int attnum;
+ int maxattnum = -1;
+ int *mapping;
+
+ attnum = -1;
+ while ((attnum = bms_next_member(attnums, attnum)) >= 0)
+ maxattnum = attnum;
+
+ mapping = (int*)palloc0((maxattnum+1) * sizeof(int));
+
+ attnum = -1;
+ while ((attnum = bms_next_member(attnums, attnum)) >= 0)
+ mapping[attnum] = attidx++;
+
+ Assert(attidx == bms_num_members(attnums));
+
+ return mapping;
+}
+
+/* build adjacency matrix for the dependencies */
+static bool*
+build_adjacency_matrix(List *stats, Bitmapset *attnums,
+ int *idx_to_attnum, int *attnum_to_idx)
+{
+ ListCell *lc;
+ int natts = bms_num_members(attnums);
+ bool *matrix = (bool*)palloc0(natts * natts * sizeof(bool));
+
+ foreach (lc, stats)
+ {
+ int j;
+ MVStatisticInfo *stat = (MVStatisticInfo *)lfirst(lc);
+ MVDependencies dependencies = NULL;
+
+ /* skip stats without functional dependencies built */
+ if (! stat->deps_built)
+ continue;
+
+ /* fetch and deserialize dependencies */
+ dependencies = load_mv_dependencies(stat->mvoid);
+ if (dependencies == NULL)
+ {
+ elog(WARNING, "failed to deserialize func deps %d", stat->mvoid);
+ continue;
+ }
+
+ /* set matrix[a,b] to 'true' if 'a=>b' */
+ for (j = 0; j < dependencies->ndeps; j++)
+ {
+ int aidx = attnum_to_idx[dependencies->deps[j]->a];
+ int bidx = attnum_to_idx[dependencies->deps[j]->b];
+
+ /* a=> b */
+ matrix[aidx * natts + bidx] = true;
+ }
+ }
+
+ return matrix;
+}
+
+/*
+ * multiply the adjacency matrix
+ *
+ * By multiplying the adjacency matrix, we derive dependencies implied by those
+ * stored in the catalog (but possibly in several separate rows). We need to
+ * repeat the multiplication until no new dependencies are discovered. The
+ * maximum number of multiplications is equal to the number of attributes.
+ *
+ * This is based on modeling the functional dependencies as edges in a directed
+ * graph with attributes as vertices.
+ */
+static void
+multiply_adjacency_matrix(bool *matrix, int natts)
+{
+ int i;
+
+ /* repeat the multiplication up to natts-times */
+ for (i = 0; i < natts; i++)
+ {
+ bool changed = false; /* no changes in this round */
+ int k, l, m;
+
+ /* k => l */
+ for (k = 0; k < natts; k++)
+ {
+ for (l = 0; l < natts; l++)
+ {
+ /* skip already known dependencies */
+ if (matrix[k * natts + l])
+ continue;
+
+ /*
+ * compute (k,l) in the multiplied matrix
+ *
+ * We don't really care about the exact value, just true/false,
+ * so terminate the loop once we get a hit. Also, this makes it
+ * safe to modify the matrix in-place.
+ */
+ for (m = 0; m < natts; m++)
+ {
+ if (matrix[k * natts + m] * matrix[m * natts + l])
+ {
+ matrix[k * natts + l] = true;
+ changed = true;
+ break;
+ }
+ }
+ }
+ }
+
+ /* no transitive dependency added in this round, so terminate */
+ if (! changed)
+ break;
+ }
+}
+
+/*
+ * Reduce clauses using functional dependencies
+ *
+ * Walk through clauses and eliminate the redundant ones (implied by other
+ * clauses). This is done by first deriving a transitive closure of all the
+ * functional dependencies (by multiplying the adjacency matrix).
+ */
+static List*
+fdeps_reduce_clauses(List *clauses, Bitmapset *attnums, bool *matrix,
+ int *idx_to_attnum, int *attnum_to_idx, Index relid)
+{
+ int i;
+ ListCell *lc;
+ List *reduced_clauses = NIL;
+
+ int nmvclauses; /* size of the arrays */
+ bool *reduced;
+ AttrNumber *mvattnums;
+ Node **mvclauses;
+
+ int natts = bms_num_members(attnums);
+
+ /*
+ * Preallocate space for all clauses (the list only containst
+ * compatible clauses at this point). This makes it somewhat easier
+ * to access the stats / attnums randomly.
+ *
+ * XXX This assumes each clause references exactly one Var, so the
+ * arrays are sized accordingly - for functional dependencies
+ * this is safe, because it only works with Var=Const.
+ */
+ mvclauses = (Node**)palloc0(list_length(clauses) * sizeof(Node*));
+ mvattnums = (AttrNumber*)palloc0(list_length(clauses) * sizeof(AttrNumber));
+ reduced = (bool*)palloc0(list_length(clauses) * sizeof(bool));
+
+ /* fill the arrays */
+ nmvclauses = 0;
+ foreach (lc, clauses)
+ {
+ Node * clause = (Node*)lfirst(lc);
+ Bitmapset * attnums = get_varattnos(clause, relid);
+
+ mvclauses[nmvclauses] = clause;
+ mvattnums[nmvclauses] = bms_singleton_member(attnums);
+ nmvclauses++;
+ }
+
+ Assert(nmvclauses == list_length(clauses));
+
+ /* now try to reduce the clauses (using the dependencies) */
+ for (i = 0; i < nmvclauses; i++)
+ {
+ int j;
+
+ /* not covered by dependencies */
+ if (! bms_is_member(mvattnums[i], attnums))
+ continue;
+
+ /* this clause was already reduced, so let's skip it */
+ if (reduced[i])
+ continue;
+
+ /* walk the potentially 'implied' clauses */
+ for (j = 0; j < nmvclauses; j++)
+ {
+ int aidx, bidx;
+
+ /* not covered by dependencies */
+ if (! bms_is_member(mvattnums[j], attnums))
+ continue;
+
+ aidx = attnum_to_idx[mvattnums[i]];
+ bidx = attnum_to_idx[mvattnums[j]];
+
+ /* can't reduce the clause by itself, or if already reduced */
+ if ((i == j) || reduced[j])
+ continue;
+
+ /* mark the clause as reduced (if aidx => bidx) */
+ reduced[j] = matrix[aidx * natts + bidx];
+ }
+ }
+
+ /* now walk through the clauses, and keep only those not reduced */
+ for (i = 0; i < nmvclauses; i++)
+ if (! reduced[i])
+ reduced_clauses = lappend(reduced_clauses, mvclauses[i]);
+
+ pfree(reduced);
+ pfree(mvclauses);
+ pfree(mvattnums);
+
+ return reduced_clauses;
+}
+
+/*
+ * filter clauses that are interesting for the reduction step
+ *
+ * Functional dependencies can only work with equality clauses with attributes
+ * covered by at least one of the statistics, so we walk through the clauses
+ * and copy the uninteresting ones directly to the result (reduced) clauses.
+ *
+ * That includes clauses that:
+ * (a) are not mv-compatible
+ * (b) reference more than a single attnum
+ * (c) use attnum not covered by functional depencencies
+ *
+ * The clauses interesting for the reduction step are copied to deps_clauses.
+ *
+ * root - planner root
+ * clauses - list of clauses (input)
+ * deps_attnums - attributes covered by dependencies
+ * reduced_clauses - resulting clauses (not subject to reduction step)
+ * deps_clauses - clauses to be processed by reduction
+ * relid - relid of the baserel
+ *
+ * The return value is a bitmap of attnums referenced by deps_clauses.
+ */
+static Bitmapset *
+fdeps_filter_clauses(PlannerInfo *root,
+ List *clauses, Bitmapset *deps_attnums,
+ List **reduced_clauses, List **deps_clauses,
+ Index relid)
+{
+ ListCell *lc;
+ Bitmapset *clause_attnums = NULL;
+
+ foreach (lc, clauses)
+ {
+ AttrNumber attnum;
+ Node *clause = (Node *) lfirst(lc);
+
+ if (! clause_is_mv_compatible(clause, relid, &attnum))
+
+ /* clause incompatible with functional dependencies */
+ *reduced_clauses = lappend(*reduced_clauses, clause);
+
+ else if (! bms_is_member(attnum, deps_attnums))
+
+ /* clause not covered by the dependencies */
+ *reduced_clauses = lappend(*reduced_clauses, clause);
+
+ else
+ {
+ *deps_clauses = lappend(*deps_clauses, clause);
+ clause_attnums = bms_add_member(clause_attnums, attnum);
+ }
+ }
+
+ return clause_attnums;
+}
+
+/*
+ * reduce list of equality clauses using soft functional dependencies
+ *
+ * We simply walk through list of functional dependencies, and for each one we
+ * check whether the dependency 'matches' the clauses, i.e. if there's a clause
+ * matching the condition. If yes, we attempt to remove all clauses matching
+ * the implied part of the dependency from the list.
+ *
+ * This only reduces equality clauses, and ignores all the other types. We might
+ * extend it to handle IS NULL clause, in the future.
+ *
+ * We also assume the equality clauses are 'compatible'. For example we can't
+ * identify when the clauses use a mismatching zip code and city name. In such
+ * case the usual approach (product of selectivities) would produce a better
+ * estimate, although mostly by chance.
+ *
+ * The implementation needs to be careful about cyclic dependencies, e.g. when
+ *
+ * (a -> b) and (b -> a)
+ *
+ * at the same time, which means there's 1:1 relationship between te columns.
+ * In this case we must not reduce clauses on both attributes at the same time.
+ *
+ * TODO Currently we only apply functional dependencies at the same level, but
+ * maybe we could transfer the clauses from upper levels to the subtrees?
+ * For example let's say we have (a->b) dependency, and condition
+ *
+ * (a=1) AND (b=2 OR c=3)
+ *
+ * Currently, we won't be able to perform any reduction, because we'll
+ * consider (a=1) and (b=2 OR c=3) independently. But maybe we could pass
+ * (a=1) into the other expression, and only check it against conditions
+ * of the functional dependencies?
+ *
+ * In this case we'd end up with
+ *
+ * (a=1)
+ *
+ * as we'd consider (b=2) implied thanks to the rule, rendering the whole
+ * OR clause valid.
+ */
+static List *
+clauselist_apply_dependencies(PlannerInfo *root, List *clauses,
+ Index relid, List *stats)
+{
+ List *reduced_clauses = NIL;
+
+ /*
+ * matrix of (natts x natts), 1 means x=>y
+ *
+ * This serves two purposes - first, it merges dependencies from all
+ * the statistics, second it makes generating all the transitive
+ * dependencies easier.
+ *
+ * We need to build this only for attributes from the dependencies,
+ * not for all attributes in the table.
+ *
+ * We can't do that only for attributes from the clauses, because we
+ * want to build transitive dependencies (including those going
+ * through attributes not listed in the stats).
+ *
+ * This only works for A=>B dependencies, not sure how to do that
+ * for complex dependencies.
+ */
+ bool *deps_matrix;
+ int deps_natts; /* size of the matric */
+
+ /* mapping attnum <=> matrix index */
+ int *deps_idx_to_attnum;
+ int *deps_attnum_to_idx;
+
+ /* attnums in dependencies and clauses (and intersection) */
+ List *deps_clauses = NIL;
+ Bitmapset *deps_attnums = NULL;
+ Bitmapset *clause_attnums = NULL;
+ Bitmapset *intersect_attnums = NULL;
+
+ /*
+ * Is there at least one statistics with functional dependencies?
+ * If not, return the original clauses right away.
+ *
+ * XXX Isn't this pointless, thanks to exactly the same check in
+ * clauselist_selectivity()? Can we trigger the condition here?
+ */
+ if (! has_stats(stats, MV_CLAUSE_TYPE_FDEP))
+ return clauses;
+
+ /*
+ * Build the dependency matrix, i.e. attribute adjacency matrix,
+ * where 1 means (a=>b). Once we have the adjacency matrix, we'll
+ * multiply it by itself, to get transitive dependencies.
+ *
+ * Note: This is pretty much transitive closure from graph theory.
+ *
+ * First, let's see what attributes are covered by functional
+ * dependencies (sides of the adjacency matrix), and also a maximum
+ * attribute (size of mapping to simple integer indexes);
+ */
+ deps_attnums = fdeps_collect_attnums(stats);
+
+ /*
+ * Walk through the clauses - clauses that are (one of)
+ *
+ * (a) not mv-compatible
+ * (b) are using more than a single attnum
+ * (c) using attnum not covered by functional depencencies
+ *
+ * may be copied directly to the result. The interesting clauses are
+ * kept in 'deps_clauses' and will be processed later.
+ */
+ clause_attnums = fdeps_filter_clauses(root, clauses, deps_attnums,
+ &reduced_clauses, &deps_clauses, relid);
+
+ /*
+ * we need at least two clauses referencing two different attributes
+ * referencing to do the reduction
+ */
+ if ((list_length(deps_clauses) < 2) || (bms_num_members(clause_attnums) < 2))
+ {
+ bms_free(clause_attnums);
+ list_free(reduced_clauses);
+ list_free(deps_clauses);
+
+ return clauses;
+ }
+
+
+ /*
+ * We need at least two matching attributes in the clauses and
+ * dependencies, otherwise we can't really reduce anything.
+ */
+ intersect_attnums = bms_intersect(clause_attnums, deps_attnums);
+ if (bms_num_members(intersect_attnums) < 2)
+ {
+ bms_free(clause_attnums);
+ bms_free(deps_attnums);
+ bms_free(intersect_attnums);
+
+ list_free(deps_clauses);
+ list_free(reduced_clauses);
+
+ return clauses;
+ }
+
+ /*
+ * Build mapping between matrix indexes and attnums, and then the
+ * adjacency matrix itself.
+ */
+ deps_idx_to_attnum = make_idx_to_attnum_mapping(deps_attnums);
+ deps_attnum_to_idx = make_attnum_to_idx_mapping(deps_attnums);
+
+ /* build the adjacency matrix */
+ deps_matrix = build_adjacency_matrix(stats, deps_attnums,
+ deps_idx_to_attnum,
+ deps_attnum_to_idx);
+
+ deps_natts = bms_num_members(deps_attnums);
+
+ /*
+ * Multiply the matrix N-times (N = size of the matrix), so that we
+ * get all the transitive dependencies. That makes the next step
+ * much easier and faster.
+ *
+ * This is essentially an adjacency matrix from graph theory, and
+ * by multiplying it we get transitive edges. We don't really care
+ * about the exact number (number of paths between vertices) though,
+ * so we can do the multiplication in-place (we don't care whether
+ * we found the dependency in this round or in the previous one).
+ *
+ * Track how many new dependencies were added, and stop when 0, but
+ * we can't multiply more than N-times (longest path in the graph).
+ */
+ multiply_adjacency_matrix(deps_matrix, deps_natts);
+
+ /*
+ * Walk through the clauses, and see which other clauses we may
+ * reduce. The matrix contains all transitive dependencies, which
+ * makes this very fast.
+ *
+ * We have to be careful not to reduce the clause using itself, or
+ * reducing all clauses forming a cycle (so we have to skip already
+ * eliminated clauses).
+ *
+ * I'm not sure whether this guarantees finding the best solution,
+ * i.e. reducing the most clauses, but it probably does (thanks to
+ * having all the transitive dependencies).
+ */
+ deps_clauses = fdeps_reduce_clauses(deps_clauses,
+ deps_attnums, deps_matrix,
+ deps_idx_to_attnum,
+ deps_attnum_to_idx, relid);
+
+ /* join the two lists of clauses */
+ reduced_clauses = list_union(reduced_clauses, deps_clauses);
+
+ pfree(deps_matrix);
+ pfree(deps_idx_to_attnum);
+ pfree(deps_attnum_to_idx);
+
+ bms_free(deps_attnums);
+ bms_free(clause_attnums);
+ bms_free(intersect_attnums);
+
+ return reduced_clauses;
+}
+
+/*
+ * Check that there are stats with at least one of the requested types.
+ */
+static bool
+has_stats(List *stats, int type)
+{
+ ListCell *s;
+
+ foreach (s, stats)
+ {
+ MVStatisticInfo *stat = (MVStatisticInfo *)lfirst(s);
+
+ if ((type & MV_CLAUSE_TYPE_FDEP) && stat->deps_built)
+ return true;
+ }
+
+ return false;
+}
+
+/*
+ * Lookups stats for a given baserel.
+ */
+static List *
+find_stats(PlannerInfo *root, Index relid)
+{
+ Assert(root->simple_rel_array[relid] != NULL);
+
+ return root->simple_rel_array[relid]->mvstatlist;
+}
diff --git a/src/backend/utils/mvstats/README.stats b/src/backend/utils/mvstats/README.stats
new file mode 100644
index 0000000..a38ea7b
--- /dev/null
+++ b/src/backend/utils/mvstats/README.stats
@@ -0,0 +1,36 @@
+Multivariate statististics
+==========================
+
+When estimating various quantities (e.g. condition selectivities) the default
+approach relies on the assumption of independence. In practice that's often
+not true, resulting in estimation errors.
+
+Multivariate stats track different types of dependencies between the columns,
+hopefully improving the estimates.
+
+Currently we only have one kind of multivariate statistics - soft functional
+dependencies, and we use it to improve estimates of equality clauses. See
+README.dependencies for details.
+
+
+Selectivity estimation
+----------------------
+
+When estimating selectivity, we aim to achieve several things:
+
+ (a) maximize the estimate accuracy
+
+ (b) minimize the overhead, especially when no suitable multivariate stats
+ exist (so if you are not using multivariate stats, there's no overhead)
+
+This clauselist_selectivity() performs several inexpensive checks first, before
+even attempting to do the more expensive estimation.
+
+ (1) check if there are multivariate stats on the relation
+
+ (2) check there are at least two attributes referenced by clauses compatible
+ with multivariate statistics (equality clauses for func. dependencies)
+
+ (3) perform reduction of equality clauses using func. dependencies
+
+ (4) estimate the reduced list of clauses using regular statistics
diff --git a/src/backend/utils/mvstats/common.c b/src/backend/utils/mvstats/common.c
index a755c49..bd200bc 100644
--- a/src/backend/utils/mvstats/common.c
+++ b/src/backend/utils/mvstats/common.c
@@ -84,7 +84,8 @@ build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
/*
* Analyze functional dependencies of columns.
*/
- deps = build_mv_dependencies(numrows, rows, attrs, stats);
+ if (stat->deps_enabled)
+ deps = build_mv_dependencies(numrows, rows, attrs, stats);
/* store the histogram / MCV list in the catalog */
update_mv_stats(stat->mvoid, deps, attrs);
@@ -163,6 +164,7 @@ list_mv_stats(Oid relid)
info->mvoid = HeapTupleGetOid(htup);
info->stakeys = buildint2vector(stats->stakeys.values, stats->stakeys.dim1);
+ info->deps_enabled = stats->deps_enabled;
info->deps_built = stats->deps_built;
result = lappend(result, info);
@@ -274,6 +276,7 @@ compare_scalars_partition(const void *a, const void *b, void *arg)
return ApplySortComparator(da, false, db, false, ssup);
}
+
/* initialize multi-dimensional sort */
MultiSortSupport
multi_sort_init(int ndims)
diff --git a/src/backend/utils/mvstats/dependencies.c b/src/backend/utils/mvstats/dependencies.c
index 2a064a0..c80ba33 100644
--- a/src/backend/utils/mvstats/dependencies.c
+++ b/src/backend/utils/mvstats/dependencies.c
@@ -435,3 +435,27 @@ pg_mv_stats_dependencies_show(PG_FUNCTION_ARGS)
PG_RETURN_TEXT_P(cstring_to_text(result));
}
+
+MVDependencies
+load_mv_dependencies(Oid mvoid)
+{
+ bool isnull = false;
+ Datum deps;
+
+ /* Prepare to scan pg_mv_statistic for entries having indrelid = this rel. */
+ HeapTuple htup = SearchSysCache1(MVSTATOID, ObjectIdGetDatum(mvoid));
+
+#ifdef USE_ASSERT_CHECKING
+ Form_pg_mv_statistic mvstat = (Form_pg_mv_statistic) GETSTRUCT(htup);
+ Assert(mvstat->deps_enabled && mvstat->deps_built);
+#endif
+
+ deps = SysCacheGetAttr(MVSTATOID, htup,
+ Anum_pg_mv_statistic_stadeps, &isnull);
+
+ Assert(!isnull);
+
+ ReleaseSysCache(htup);
+
+ return deserialize_mv_dependencies(DatumGetByteaP(deps));
+}
diff --git a/src/include/utils/mvstats.h b/src/include/utils/mvstats.h
index 7ebd961..cc43a79 100644
--- a/src/include/utils/mvstats.h
+++ b/src/include/utils/mvstats.h
@@ -17,12 +17,20 @@
#include "fmgr.h"
#include "commands/vacuum.h"
+/*
+ * Degree of how much MCV item / histogram bucket matches a clause.
+ * This is then considered when computing the selectivity.
+ */
+#define MVSTATS_MATCH_NONE 0 /* no match at all */
+#define MVSTATS_MATCH_PARTIAL 1 /* partial match */
+#define MVSTATS_MATCH_FULL 2 /* full match */
#define MVSTATS_MAX_DIMENSIONS 8 /* max number of attributes */
-/* An associative rule, tracking [a => b] dependency.
- *
- * TODO Make this work with multiple columns on both sides.
+
+/*
+ * Functional dependencies, tracking column-level relationships (values
+ * in one column determine values in another one).
*/
typedef struct MVDependencyData {
int16 a;
@@ -48,6 +56,8 @@ typedef MVDependenciesData* MVDependencies;
* stats specified using flags (or something like that).
*/
+MVDependencies load_mv_dependencies(Oid mvoid);
+
bytea * serialize_mv_dependencies(MVDependencies dependencies);
/* deserialization of stats (serialization is private to analyze) */
diff --git a/src/test/regress/expected/mv_dependencies.out b/src/test/regress/expected/mv_dependencies.out
new file mode 100644
index 0000000..e759997
--- /dev/null
+++ b/src/test/regress/expected/mv_dependencies.out
@@ -0,0 +1,172 @@
+-- data type passed by value
+CREATE TABLE functional_dependencies (
+ a INT,
+ b INT,
+ c INT
+);
+-- unknown column
+CREATE STATISTICS s1 ON functional_dependencies (unknown_column) WITH (dependencies);
+ERROR: column "unknown_column" referenced in statistics does not exist
+-- single column
+CREATE STATISTICS s1 ON functional_dependencies (a) WITH (dependencies);
+ERROR: multivariate stats require 2 or more columns
+-- single column, duplicated
+CREATE STATISTICS s1 ON functional_dependencies (a,a) WITH (dependencies);
+ERROR: duplicate column name in statistics definition
+-- two columns, one duplicated
+CREATE STATISTICS s1 ON functional_dependencies (a, a, b) WITH (dependencies);
+ERROR: duplicate column name in statistics definition
+-- unknown option
+CREATE STATISTICS s1 ON functional_dependencies (a, b, c) WITH (unknown_option);
+ERROR: unrecognized STATISTICS option "unknown_option"
+-- correct command
+CREATE STATISTICS s1 ON functional_dependencies (a, b, c) WITH (dependencies);
+-- random data (no functional dependencies)
+INSERT INTO functional_dependencies
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+ANALYZE functional_dependencies;
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+ deps_enabled | deps_built | pg_mv_stats_dependencies_show
+--------------+------------+-------------------------------
+ t | f |
+(1 row)
+
+TRUNCATE functional_dependencies;
+-- a => b, a => c, b => c
+INSERT INTO functional_dependencies
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE functional_dependencies;
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+ deps_enabled | deps_built | pg_mv_stats_dependencies_show
+--------------+------------+-------------------------------
+ t | t | 1 => 2, 1 => 3, 2 => 3
+(1 row)
+
+TRUNCATE functional_dependencies;
+-- a => b, a => c
+INSERT INTO functional_dependencies
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE functional_dependencies;
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+ deps_enabled | deps_built | pg_mv_stats_dependencies_show
+--------------+------------+-------------------------------
+ t | t | 1 => 2, 1 => 3
+(1 row)
+
+TRUNCATE functional_dependencies;
+-- check explain (expect bitmap index scan, not plain index scan)
+INSERT INTO functional_dependencies
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX fdeps_idx ON functional_dependencies (a, b);
+ANALYZE functional_dependencies;
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+ deps_enabled | deps_built | pg_mv_stats_dependencies_show
+--------------+------------+-------------------------------
+ t | t | 1 => 2, 1 => 3, 2 => 3
+(1 row)
+
+EXPLAIN (COSTS off)
+ SELECT * FROM functional_dependencies WHERE a = 10 AND b = 5;
+ QUERY PLAN
+---------------------------------------------
+ Bitmap Heap Scan on functional_dependencies
+ Recheck Cond: ((a = 10) AND (b = 5))
+ -> Bitmap Index Scan on fdeps_idx
+ Index Cond: ((a = 10) AND (b = 5))
+(4 rows)
+
+DROP TABLE functional_dependencies;
+-- varlena type (text)
+CREATE TABLE functional_dependencies (
+ a TEXT,
+ b TEXT,
+ c TEXT
+);
+CREATE STATISTICS s2 ON functional_dependencies (a, b, c) WITH (dependencies);
+-- random data (no functional dependencies)
+INSERT INTO functional_dependencies
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+ANALYZE functional_dependencies;
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+ deps_enabled | deps_built | pg_mv_stats_dependencies_show
+--------------+------------+-------------------------------
+ t | f |
+(1 row)
+
+TRUNCATE functional_dependencies;
+-- a => b, a => c, b => c
+INSERT INTO functional_dependencies
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE functional_dependencies;
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+ deps_enabled | deps_built | pg_mv_stats_dependencies_show
+--------------+------------+-------------------------------
+ t | t | 1 => 2, 1 => 3, 2 => 3
+(1 row)
+
+TRUNCATE functional_dependencies;
+-- a => b, a => c
+INSERT INTO functional_dependencies
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE functional_dependencies;
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+ deps_enabled | deps_built | pg_mv_stats_dependencies_show
+--------------+------------+-------------------------------
+ t | t | 1 => 2, 1 => 3
+(1 row)
+
+TRUNCATE functional_dependencies;
+-- check explain (expect bitmap index scan, not plain index scan)
+INSERT INTO functional_dependencies
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX fdeps_idx ON functional_dependencies (a, b);
+ANALYZE functional_dependencies;
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+ deps_enabled | deps_built | pg_mv_stats_dependencies_show
+--------------+------------+-------------------------------
+ t | t | 1 => 2, 1 => 3, 2 => 3
+(1 row)
+
+EXPLAIN (COSTS off)
+ SELECT * FROM functional_dependencies WHERE a = '10' AND b = '5';
+ QUERY PLAN
+------------------------------------------------------------
+ Bitmap Heap Scan on functional_dependencies
+ Recheck Cond: ((a = '10'::text) AND (b = '5'::text))
+ -> Bitmap Index Scan on fdeps_idx
+ Index Cond: ((a = '10'::text) AND (b = '5'::text))
+(4 rows)
+
+DROP TABLE functional_dependencies;
+-- NULL values (mix of int and text columns)
+CREATE TABLE functional_dependencies (
+ a INT,
+ b TEXT,
+ c INT,
+ d TEXT
+);
+CREATE STATISTICS s3 ON functional_dependencies (a, b, c, d) WITH (dependencies);
+INSERT INTO functional_dependencies
+ SELECT
+ mod(i, 100),
+ (CASE WHEN mod(i, 200) = 0 THEN NULL ELSE mod(i,200) END),
+ mod(i, 400),
+ (CASE WHEN mod(i, 300) = 0 THEN NULL ELSE mod(i,600) END)
+ FROM generate_series(1,10000) s(i);
+ANALYZE functional_dependencies;
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+ deps_enabled | deps_built | pg_mv_stats_dependencies_show
+--------------+------------+----------------------------------------
+ t | t | 2 => 1, 3 => 1, 3 => 2, 4 => 1, 4 => 2
+(1 row)
+
+DROP TABLE functional_dependencies;
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index bec0316..4f2ffb8 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -110,3 +110,6 @@ test: event_trigger
# run stats by itself because its delay may be insufficient under heavy load
test: stats
+
+# run tests of multivariate stats
+test: mv_dependencies
diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule
index 7e9b319..097a04f 100644
--- a/src/test/regress/serial_schedule
+++ b/src/test/regress/serial_schedule
@@ -162,3 +162,4 @@ test: with
test: xml
test: event_trigger
test: stats
+test: mv_dependencies
diff --git a/src/test/regress/sql/mv_dependencies.sql b/src/test/regress/sql/mv_dependencies.sql
new file mode 100644
index 0000000..48dea4d
--- /dev/null
+++ b/src/test/regress/sql/mv_dependencies.sql
@@ -0,0 +1,150 @@
+-- data type passed by value
+CREATE TABLE functional_dependencies (
+ a INT,
+ b INT,
+ c INT
+);
+
+-- unknown column
+CREATE STATISTICS s1 ON functional_dependencies (unknown_column) WITH (dependencies);
+
+-- single column
+CREATE STATISTICS s1 ON functional_dependencies (a) WITH (dependencies);
+
+-- single column, duplicated
+CREATE STATISTICS s1 ON functional_dependencies (a,a) WITH (dependencies);
+
+-- two columns, one duplicated
+CREATE STATISTICS s1 ON functional_dependencies (a, a, b) WITH (dependencies);
+
+-- unknown option
+CREATE STATISTICS s1 ON functional_dependencies (a, b, c) WITH (unknown_option);
+
+-- correct command
+CREATE STATISTICS s1 ON functional_dependencies (a, b, c) WITH (dependencies);
+
+-- random data (no functional dependencies)
+INSERT INTO functional_dependencies
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+
+ANALYZE functional_dependencies;
+
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+
+TRUNCATE functional_dependencies;
+
+-- a => b, a => c, b => c
+INSERT INTO functional_dependencies
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+
+ANALYZE functional_dependencies;
+
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+
+TRUNCATE functional_dependencies;
+
+-- a => b, a => c
+INSERT INTO functional_dependencies
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE functional_dependencies;
+
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+
+TRUNCATE functional_dependencies;
+
+-- check explain (expect bitmap index scan, not plain index scan)
+INSERT INTO functional_dependencies
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX fdeps_idx ON functional_dependencies (a, b);
+ANALYZE functional_dependencies;
+
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+
+EXPLAIN (COSTS off)
+ SELECT * FROM functional_dependencies WHERE a = 10 AND b = 5;
+
+DROP TABLE functional_dependencies;
+
+-- varlena type (text)
+CREATE TABLE functional_dependencies (
+ a TEXT,
+ b TEXT,
+ c TEXT
+);
+
+CREATE STATISTICS s2 ON functional_dependencies (a, b, c) WITH (dependencies);
+
+-- random data (no functional dependencies)
+INSERT INTO functional_dependencies
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+
+ANALYZE functional_dependencies;
+
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+
+TRUNCATE functional_dependencies;
+
+-- a => b, a => c, b => c
+INSERT INTO functional_dependencies
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+
+ANALYZE functional_dependencies;
+
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+
+TRUNCATE functional_dependencies;
+
+-- a => b, a => c
+INSERT INTO functional_dependencies
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE functional_dependencies;
+
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+
+TRUNCATE functional_dependencies;
+
+-- check explain (expect bitmap index scan, not plain index scan)
+INSERT INTO functional_dependencies
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX fdeps_idx ON functional_dependencies (a, b);
+ANALYZE functional_dependencies;
+
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+
+EXPLAIN (COSTS off)
+ SELECT * FROM functional_dependencies WHERE a = '10' AND b = '5';
+
+DROP TABLE functional_dependencies;
+
+-- NULL values (mix of int and text columns)
+CREATE TABLE functional_dependencies (
+ a INT,
+ b TEXT,
+ c INT,
+ d TEXT
+);
+
+CREATE STATISTICS s3 ON functional_dependencies (a, b, c, d) WITH (dependencies);
+
+INSERT INTO functional_dependencies
+ SELECT
+ mod(i, 100),
+ (CASE WHEN mod(i, 200) = 0 THEN NULL ELSE mod(i,200) END),
+ mod(i, 400),
+ (CASE WHEN mod(i, 300) = 0 THEN NULL ELSE mod(i,600) END)
+ FROM generate_series(1,10000) s(i);
+
+ANALYZE functional_dependencies;
+
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+
+DROP TABLE functional_dependencies;
--
2.1.0
[text/x-patch] 0004-multivariate-MCV-lists.patch (112.8K, ../../1457535779.24545.71.camel@2ndquadrant.com/5-0004-multivariate-MCV-lists.patch)
download | inline diff:
From 11e08f7a0ffc186dbc23605d522c278e9b393ea5 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@pgaddict.com>
Date: Mon, 6 Apr 2015 16:52:15 +0200
Subject: [PATCH 4/9] multivariate MCV lists
- extends the pg_mv_statistic catalog (add 'mcv' fields)
- building the MCV lists during ANALYZE
- simple estimation while planning the queries
Includes regression tests, mostly equal to regression tests for
functional dependencies.
---
doc/src/sgml/ref/create_statistics.sgml | 18 +
src/backend/catalog/system_views.sql | 4 +-
src/backend/commands/statscmds.c | 45 +-
src/backend/nodes/outfuncs.c | 2 +
src/backend/optimizer/path/clausesel.c | 829 ++++++++++++++++++++++-
src/backend/optimizer/util/plancat.c | 4 +-
src/backend/utils/mvstats/Makefile | 2 +-
src/backend/utils/mvstats/README.mcv | 137 ++++
src/backend/utils/mvstats/README.stats | 89 ++-
src/backend/utils/mvstats/common.c | 104 ++-
src/backend/utils/mvstats/common.h | 11 +-
src/backend/utils/mvstats/mcv.c | 1094 +++++++++++++++++++++++++++++++
src/bin/psql/describe.c | 25 +-
src/include/catalog/pg_mv_statistic.h | 18 +-
src/include/catalog/pg_proc.h | 4 +
src/include/nodes/relation.h | 2 +
src/include/utils/mvstats.h | 69 +-
src/test/regress/expected/mv_mcv.out | 207 ++++++
src/test/regress/expected/rules.out | 4 +-
src/test/regress/parallel_schedule | 2 +-
src/test/regress/serial_schedule | 1 +
src/test/regress/sql/mv_mcv.sql | 178 +++++
22 files changed, 2776 insertions(+), 73 deletions(-)
create mode 100644 src/backend/utils/mvstats/README.mcv
create mode 100644 src/backend/utils/mvstats/mcv.c
create mode 100644 src/test/regress/expected/mv_mcv.out
create mode 100644 src/test/regress/sql/mv_mcv.sql
diff --git a/doc/src/sgml/ref/create_statistics.sgml b/doc/src/sgml/ref/create_statistics.sgml
index a86eae3..193e4b0 100644
--- a/doc/src/sgml/ref/create_statistics.sgml
+++ b/doc/src/sgml/ref/create_statistics.sgml
@@ -132,6 +132,24 @@ CREATE STATISTICS [ IF NOT EXISTS ] <replaceable class="PARAMETER">statistics_na
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>max_mcv_items</> (<type>integer</>)</term>
+ <listitem>
+ <para>
+ Maximum number of MCV list items.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>mcv</> (<type>boolean</>)</term>
+ <listitem>
+ <para>
+ Enables MCV list for the statistics.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect2>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index b8a264e..2d570ee 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -165,7 +165,9 @@ CREATE VIEW pg_mv_stats AS
S.staname AS staname,
S.stakeys AS attnums,
length(S.stadeps) as depsbytes,
- pg_mv_stats_dependencies_info(S.stadeps) as depsinfo
+ pg_mv_stats_dependencies_info(S.stadeps) as depsinfo,
+ length(S.stamcv) AS mcvbytes,
+ pg_mv_stats_mcvlist_info(S.stamcv) AS mcvinfo
FROM (pg_mv_statistic S JOIN pg_class C ON (C.oid = S.starelid))
LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace);
diff --git a/src/backend/commands/statscmds.c b/src/backend/commands/statscmds.c
index 1b89bbe..b04c583 100644
--- a/src/backend/commands/statscmds.c
+++ b/src/backend/commands/statscmds.c
@@ -70,7 +70,13 @@ CreateStatistics(CreateStatsStmt *stmt)
ObjectAddress parentobject, childobject;
/* by default build nothing */
- bool build_dependencies = false;
+ bool build_dependencies = false,
+ build_mcv = false;
+
+ int32 max_mcv_items = -1;
+
+ /* options required because of other options */
+ bool require_mcv = false;
Assert(IsA(stmt, CreateStatsStmt));
@@ -146,6 +152,29 @@ CreateStatistics(CreateStatsStmt *stmt)
if (strcmp(opt->defname, "dependencies") == 0)
build_dependencies = defGetBoolean(opt);
+ else if (strcmp(opt->defname, "mcv") == 0)
+ build_mcv = defGetBoolean(opt);
+ else if (strcmp(opt->defname, "max_mcv_items") == 0)
+ {
+ max_mcv_items = defGetInt32(opt);
+
+ /* this option requires 'mcv' to be enabled */
+ require_mcv = true;
+
+ /* sanity check */
+ if (max_mcv_items < MVSTAT_MCVLIST_MIN_ITEMS)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("max number of MCV items must be at least %d",
+ MVSTAT_MCVLIST_MIN_ITEMS)));
+
+ else if (max_mcv_items > MVSTAT_MCVLIST_MAX_ITEMS)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("max number of MCV items is %d",
+ MVSTAT_MCVLIST_MAX_ITEMS)));
+
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -154,10 +183,16 @@ CreateStatistics(CreateStatsStmt *stmt)
}
/* check that at least some statistics were requested */
- if (! build_dependencies)
+ if (! (build_dependencies || build_mcv))
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("no statistics type (dependencies, mcv) was requested")));
+
+ /* now do some checking of the options */
+ if (require_mcv && (! build_mcv))
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("no statistics type (dependencies) was requested")));
+ errmsg("option 'mcv' is required by other options(s)")));
/* sort the attnums and build int2vector */
qsort(attnums, numcols, sizeof(int16), compare_int16);
@@ -178,8 +213,12 @@ CreateStatistics(CreateStatsStmt *stmt)
values[Anum_pg_mv_statistic_stakeys -1] = PointerGetDatum(stakeys);
values[Anum_pg_mv_statistic_deps_enabled -1] = BoolGetDatum(build_dependencies);
+ values[Anum_pg_mv_statistic_mcv_enabled -1] = BoolGetDatum(build_mcv);
+
+ values[Anum_pg_mv_statistic_mcv_max_items -1] = Int32GetDatum(max_mcv_items);
nulls[Anum_pg_mv_statistic_stadeps -1] = true;
+ nulls[Anum_pg_mv_statistic_stamcv -1] = true;
/* insert the tuple into pg_mv_statistic */
mvstatrel = heap_open(MvStatisticRelationId, RowExclusiveLock);
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 07206d7..333e24b 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2162,9 +2162,11 @@ _outMVStatisticInfo(StringInfo str, const MVStatisticInfo *node)
/* enabled statistics */
WRITE_BOOL_FIELD(deps_enabled);
+ WRITE_BOOL_FIELD(mcv_enabled);
/* built/available statistics */
WRITE_BOOL_FIELD(deps_built);
+ WRITE_BOOL_FIELD(mcv_built);
}
static void
diff --git a/src/backend/optimizer/path/clausesel.c b/src/backend/optimizer/path/clausesel.c
index 80708fe..977f88e 100644
--- a/src/backend/optimizer/path/clausesel.c
+++ b/src/backend/optimizer/path/clausesel.c
@@ -15,6 +15,7 @@
#include "postgres.h"
#include "access/sysattr.h"
+#include "catalog/pg_collation.h"
#include "catalog/pg_operator.h"
#include "nodes/makefuncs.h"
#include "optimizer/clauses.h"
@@ -47,23 +48,51 @@ static void addRangeClause(RangeQueryClause **rqlist, Node *clause,
bool varonleft, bool isLTsel, Selectivity s2);
#define MV_CLAUSE_TYPE_FDEP 0x01
+#define MV_CLAUSE_TYPE_MCV 0x02
-static bool clause_is_mv_compatible(Node *clause, Index relid, AttrNumber *attnum);
+static bool clause_is_mv_compatible(Node *clause, Index relid, Bitmapset **attnums,
+ int type);
-static Bitmapset *collect_mv_attnums(List *clauses, Index relid);
+static Bitmapset *collect_mv_attnums(List *clauses, Index relid, int type);
-static int count_mv_attnums(List *clauses, Index relid);
+static int count_mv_attnums(List *clauses, Index relid, int type);
static int count_varnos(List *clauses, Index *relid);
static List *clauselist_apply_dependencies(PlannerInfo *root, List *clauses,
Index relid, List *stats);
+static MVStatisticInfo *choose_mv_statistics(List *mvstats, Bitmapset *attnums);
+
+static List *clauselist_mv_split(PlannerInfo *root, Index relid,
+ List *clauses, List **mvclauses,
+ MVStatisticInfo *mvstats, int types);
+
+static Selectivity clauselist_mv_selectivity(PlannerInfo *root,
+ List *clauses, MVStatisticInfo *mvstats);
+
+static Selectivity clauselist_mv_selectivity_mcvlist(PlannerInfo *root,
+ List *clauses, MVStatisticInfo *mvstats,
+ bool *fullmatch, Selectivity *lowsel);
+
+static int update_match_bitmap_mcvlist(PlannerInfo *root, List *clauses,
+ int2vector *stakeys, MCVList mcvlist,
+ int nmatches, char * matches,
+ Selectivity *lowsel, bool *fullmatch,
+ bool is_or);
+
static bool has_stats(List *stats, int type);
static List * find_stats(PlannerInfo *root, Index relid);
+/* used for merging bitmaps - AND (min), OR (max) */
+#define MAX(x, y) (((x) > (y)) ? (x) : (y))
+#define MIN(x, y) (((x) < (y)) ? (x) : (y))
+
+#define UPDATE_RESULT(m,r,isor) \
+ (m) = (isor) ? (MAX(m,r)) : (MIN(m,r))
+
/****************************************************************************
* ROUTINES TO COMPUTE SELECTIVITIES
****************************************************************************/
@@ -89,11 +118,13 @@ static List * find_stats(PlannerInfo *root, Index relid);
* to verify that suitable multivariate statistics exist.
*
* If we identify such multivariate statistics apply, we try to apply them.
- * Currently we only have (soft) functional dependencies, so we try to reduce
- * the list of clauses.
*
- * Then we remove the clauses estimated using multivariate stats, and process
- * the rest of the clauses using the regular per-column stats.
+ * First we try to reduce the list of clauses by applying (soft) functional
+ * dependencies, and then we try to estimate the selectivity of the reduced
+ * list of clauses using the multivariate MCV list.
+ *
+ * Finally we remove the portion of clauses estimated using multivariate stats,
+ * and process the rest of the clauses using the regular per-column stats.
*
* Currently, the only extra smarts we have is to recognize "range queries",
* such as "x > 34 AND x < 42". Clauses are recognized as possible range
@@ -170,12 +201,46 @@ clauselist_selectivity(PlannerInfo *root,
* that need to be estimated by other types of stats (MCV, histograms etc).
*/
if (has_stats(stats, MV_CLAUSE_TYPE_FDEP) &&
- (count_mv_attnums(clauses, relid) >= 2))
+ (count_mv_attnums(clauses, relid, MV_CLAUSE_TYPE_FDEP) >= 2))
{
clauses = clauselist_apply_dependencies(root, clauses, relid, stats);
}
/*
+ * Check that there are statistics with MCV list or histogram, and also the
+ * number of attributes covered by these types of statistics.
+ *
+ * If there are no such stats or not enough attributes, don't waste time
+ * with the multivariate code and simply skip to estimation using the
+ * regular per-column stats.
+ */
+ if (has_stats(stats, MV_CLAUSE_TYPE_MCV) &&
+ (count_mv_attnums(clauses, relid, MV_CLAUSE_TYPE_MCV) >= 2))
+ {
+ /* collect attributes from the compatible conditions */
+ Bitmapset *mvattnums = collect_mv_attnums(clauses, relid, MV_CLAUSE_TYPE_MCV);
+
+ /* and search for the statistic covering the most attributes */
+ MVStatisticInfo *mvstat = choose_mv_statistics(stats, mvattnums);
+
+ if (mvstat != NULL) /* we have a matching stats */
+ {
+ /* clauses compatible with multi-variate stats */
+ List *mvclauses = NIL;
+
+ /* split the clauselist into regular and mv-clauses */
+ clauses = clauselist_mv_split(root, relid, clauses, &mvclauses,
+ mvstat, MV_CLAUSE_TYPE_MCV);
+
+ /* we've chosen the histogram to match the clauses */
+ Assert(mvclauses != NIL);
+
+ /* compute the multivariate stats */
+ s1 *= clauselist_mv_selectivity(root, mvclauses, mvstat);
+ }
+ }
+
+ /*
* Initial scan over clauses. Anything that doesn't look like a potential
* rangequery clause gets multiplied into s1 and forgotten. Anything that
* does gets inserted into an rqlist entry.
@@ -832,6 +897,69 @@ clause_selectivity(PlannerInfo *root,
return s1;
}
+
+/*
+ * estimate selectivity of clauses using multivariate statistic
+ *
+ * Perform estimation of the clauses using a MCV list.
+ *
+ * This assumes all the clauses are compatible with the selected statistics
+ * (e.g. only reference columns covered by the statistics, use supported
+ * operator, etc.).
+ *
+ * TODO We may support some additional conditions, most importantly those
+ * matching multiple columns (e.g. "a = b" or "a < b").
+ *
+ * TODO Clamp the selectivity by min of the per-clause selectivities (i.e. the
+ * selectivity of the most restrictive clause), because that's the maximum
+ * we can ever get from ANDed list of clauses. This may probably prevent
+ * issues with hitting too many buckets and low precision histograms.
+ *
+ * TODO We may remember the lowest frequency in the MCV list, and then later use
+ * it as a upper boundary for the selectivity (had there been a more
+ * frequent item, it'd be in the MCV list). This might improve cases with
+ * low-detail histograms.
+ *
+ * TODO We may also derive some additional boundaries for the selectivity from
+ * the MCV list, because
+ *
+ * (a) if we have a "full equality condition" (one equality condition on
+ * each column of the statistic) and we found a match in the MCV list,
+ * then this is the final selectivity (and pretty accurate),
+ *
+ * (b) if we have a "full equality condition" and we haven't found a match
+ * in the MCV list, then the selectivity is below the lowest frequency
+ * found in the MCV list,
+ *
+ * TODO When applying the clauses to the histogram/MCV list, we can do
+ * that from the most selective clauses first, because that'll
+ * eliminate the buckets/items sooner (so we'll be able to skip
+ * them without inspection, which is more expensive). But this
+ * requires really knowing the per-clause selectivities in advance,
+ * and that's not what we do now.
+ */
+static Selectivity
+clauselist_mv_selectivity(PlannerInfo *root, List *clauses, MVStatisticInfo *mvstats)
+{
+ bool fullmatch = false;
+
+ /*
+ * Lowest frequency in the MCV list (may be used as an upper bound
+ * for full equality conditions that did not match any MCV item).
+ */
+ Selectivity mcv_low = 0.0;
+
+ /* TODO Evaluate simple 1D selectivities, use the smallest one as
+ * an upper bound, product as lower bound, and sort the
+ * clauses in ascending order by selectivity (to optimize the
+ * MCV/histogram evaluation).
+ */
+
+ /* Evaluate the MCV selectivity */
+ return clauselist_mv_selectivity_mcvlist(root, clauses, mvstats,
+ &fullmatch, &mcv_low);
+}
+
/*
* Pull varattnos from the clauses, similarly to pull_varattnos() but:
*
@@ -869,28 +997,26 @@ get_varattnos(Node * node, Index relid)
* Collect attributes from mv-compatible clauses.
*/
static Bitmapset *
-collect_mv_attnums(List *clauses, Index relid)
+collect_mv_attnums(List *clauses, Index relid, int types)
{
Bitmapset *attnums = NULL;
ListCell *l;
/*
- * Walk through the clauses and identify the ones we can estimate
- * using multivariate stats, and remember the relid/columns. We'll
- * then cross-check if we have suitable stats, and only if needed
- * we'll split the clauses into multivariate and regular lists.
+ * Walk through the clauses and identify the ones we can estimate using
+ * multivariate stats, and remember the relid/columns. We'll then
+ * cross-check if we have suitable stats, and only if needed we'll split
+ * the clauses into multivariate and regular lists.
*
- * For now we're only interested in RestrictInfo nodes with nested
- * OpExpr, using either a range or equality.
+ * For now we're only interested in RestrictInfo nodes with nested OpExpr,
+ * using either a range or equality.
*/
foreach (l, clauses)
{
- AttrNumber attnum;
Node *clause = (Node *) lfirst(l);
- /* ignore the result for now - we only need the info */
- if (clause_is_mv_compatible(clause, relid, &attnum))
- attnums = bms_add_member(attnums, attnum);
+ /* ignore the result here - we only need the attnums */
+ clause_is_mv_compatible(clause, relid, &attnums, types);
}
/*
@@ -911,10 +1037,10 @@ collect_mv_attnums(List *clauses, Index relid)
* Count the number of attributes in clauses compatible with multivariate stats.
*/
static int
-count_mv_attnums(List *clauses, Index relid)
+count_mv_attnums(List *clauses, Index relid, int type)
{
int c;
- Bitmapset *attnums = collect_mv_attnums(clauses, relid);
+ Bitmapset *attnums = collect_mv_attnums(clauses, relid, type);
c = bms_num_members(attnums);
@@ -944,9 +1070,183 @@ count_varnos(List *clauses, Index *relid)
return cnt;
}
+
+/*
+ * We're looking for statistics matching at least 2 attributes, referenced in
+ * clauses compatible with multivariate statistics. The current selection
+ * criteria is very simple - we choose the statistics referencing the most
+ * attributes.
+ *
+ * If there are multiple statistics referencing the same number of columns
+ * (from the clauses), the one with less source columns (as listed in the
+ * ADD STATISTICS when creating the statistics) wins. Else the first one wins.
+ *
+ * This is a very simple criteria, and has several weaknesses:
+ *
+ * (a) does not consider the accuracy of the statistics
+ *
+ * If there are two histograms built on the same set of columns, but one
+ * has 100 buckets and the other one has 1000 buckets (thus likely
+ * providing better estimates), this is not currently considered.
+ *
+ * (b) does not consider the type of statistics
+ *
+ * If there are three statistics - one containing just a MCV list, another
+ * one with just a histogram and a third one with both, we treat them equally.
+ *
+ * (c) does not consider the number of clauses
+ *
+ * As explained, only the number of referenced attributes counts, so if
+ * there are multiple clauses on a single attribute, this still counts as
+ * a single attribute.
+ *
+ * (d) does not consider type of condition
+ *
+ * Some clauses may work better with some statistics - for example equality
+ * clauses probably work better with MCV lists than with histograms. But
+ * IS [NOT] NULL conditions may often work better with histograms (thanks
+ * to NULL-buckets).
+ *
+ * So for example with five WHERE conditions
+ *
+ * WHERE (a = 1) AND (b = 1) AND (c = 1) AND (d = 1) AND (e = 1)
+ *
+ * and statistics on (a,b), (a,b,e) and (a,b,c,d), the last one will be selected
+ * as it references the most columns.
+ *
+ * Once we have selected the multivariate statistics, we split the list of
+ * clauses into two parts - conditions that are compatible with the selected
+ * stats, and conditions are estimated using simple statistics.
+ *
+ * From the example above, conditions
+ *
+ * (a = 1) AND (b = 1) AND (c = 1) AND (d = 1)
+ *
+ * will be estimated using the multivariate statistics (a,b,c,d) while the last
+ * condition (e = 1) will get estimated using the regular ones.
+ *
+ * There are various alternative selection criteria (e.g. counting conditions
+ * instead of just referenced attributes), but eventually the best option should
+ * be to combine multiple statistics. But that's much harder to do correctly.
+ *
+ * TODO Select multiple statistics and combine them when computing the estimate.
+ *
+ * TODO This will probably have to consider compatibility of clauses, because
+ * 'dependencies' will probably work only with equality clauses.
+ */
+static MVStatisticInfo *
+choose_mv_statistics(List *stats, Bitmapset *attnums)
+{
+ int i;
+ ListCell *lc;
+
+ MVStatisticInfo *choice = NULL;
+
+ int current_matches = 1; /* goal #1: maximize */
+ int current_dims = (MVSTATS_MAX_DIMENSIONS+1); /* goal #2: minimize */
+
+ /*
+ * Walk through the statistics (simple array with nmvstats elements) and for
+ * each one count the referenced attributes (encoded in the 'attnums' bitmap).
+ */
+ foreach (lc, stats)
+ {
+ MVStatisticInfo *info = (MVStatisticInfo *)lfirst(lc);
+
+ /* columns matching this statistics */
+ int matches = 0;
+
+ int2vector * attrs = info->stakeys;
+ int numattrs = attrs->dim1;
+
+ /* skip dependencies-only stats */
+ if (! info->mcv_built)
+ continue;
+
+ /* count columns covered by the histogram */
+ for (i = 0; i < numattrs; i++)
+ if (bms_is_member(attrs->values[i], attnums))
+ matches++;
+
+ /*
+ * Use this statistics when it improves the number of matches or
+ * when it matches the same number of attributes but is smaller.
+ */
+ if ((matches > current_matches) ||
+ ((matches == current_matches) && (current_dims > numattrs)))
+ {
+ choice = info;
+ current_matches = matches;
+ current_dims = numattrs;
+ }
+ }
+
+ return choice;
+}
+
+
+/*
+ * This splits the clauses list into two parts - one containing clauses that
+ * will be evaluated using the chosen statistics, and the remaining clauses
+ * (either non-mvcompatible, or not related to the histogram).
+ */
+static List *
+clauselist_mv_split(PlannerInfo *root, Index relid,
+ List *clauses, List **mvclauses,
+ MVStatisticInfo *mvstats, int types)
+{
+ int i;
+ ListCell *l;
+ List *non_mvclauses = NIL;
+
+ /* FIXME is there a better way to get info on int2vector? */
+ int2vector * attrs = mvstats->stakeys;
+ int numattrs = mvstats->stakeys->dim1;
+
+ Bitmapset *mvattnums = NULL;
+
+ /* build bitmap of attributes, so we can do bms_is_subset later */
+ for (i = 0; i < numattrs; i++)
+ mvattnums = bms_add_member(mvattnums, attrs->values[i]);
+
+ /* erase the list of mv-compatible clauses */
+ *mvclauses = NIL;
+
+ foreach (l, clauses)
+ {
+ bool match = false; /* by default not mv-compatible */
+ Bitmapset *attnums = NULL;
+ Node *clause = (Node *) lfirst(l);
+
+ if (clause_is_mv_compatible(clause, relid, &attnums, types))
+ {
+ /* are all the attributes part of the selected stats? */
+ if (bms_is_subset(attnums, mvattnums))
+ match = true;
+ }
+
+ /*
+ * The clause matches the selected stats, so put it to the list of
+ * mv-compatible clauses. Otherwise, keep it in the list of 'regular'
+ * clauses (that may be selected later).
+ */
+ if (match)
+ *mvclauses = lappend(*mvclauses, clause);
+ else
+ non_mvclauses = lappend(non_mvclauses, clause);
+ }
+
+ /*
+ * Perform regular estimation using the clauses incompatible with the chosen
+ * histogram (or MV stats in general).
+ */
+ return non_mvclauses;
+
+}
typedef struct
{
+ int types; /* types of statistics ? */
Index varno; /* relid we're interested in */
Bitmapset *varattnos; /* attnums referenced by the clauses */
} mv_compatible_context;
@@ -964,23 +1264,66 @@ mv_compatible_walker(Node *node, mv_compatible_context *context)
{
if (node == NULL)
return false;
-
+
if (IsA(node, RestrictInfo))
{
RestrictInfo *rinfo = (RestrictInfo *) node;
-
+
/* Pseudoconstants are not really interesting here. */
if (rinfo->pseudoconstant)
return true;
-
+
/* clauses referencing multiple varnos are incompatible */
if (bms_membership(rinfo->clause_relids) != BMS_SINGLETON)
return true;
-
+
/* check the clause inside the RestrictInfo */
return mv_compatible_walker((Node*)rinfo->clause, (void *) context);
}
+ if (or_clause(node) || and_clause(node) || not_clause(node))
+ {
+ /*
+ * AND/OR/NOT-clauses are supported if all sub-clauses are supported
+ *
+ * TODO We might support mixed case, where some of the clauses are
+ * supported and some are not, and treat all supported subclauses
+ * as a single clause, compute it's selectivity using mv stats,
+ * and compute the total selectivity using the current algorithm.
+ *
+ * TODO For RestrictInfo above an OR-clause, we might use the orclause
+ * with nested RestrictInfo - we won't have to call pull_varnos()
+ * for each clause, saving time.
+ *
+ * TODO Perhaps this needs a bit more thought for functional
+ * dependencies? Those don't quite work for NOT cases.
+ */
+ BoolExpr *expr = (BoolExpr *) node;
+ ListCell *lc;
+
+ foreach (lc, expr->args)
+ {
+ if (mv_compatible_walker((Node *) lfirst(lc), context))
+ return true;
+ }
+
+ return false;
+ }
+
+ if (IsA(node, NullTest))
+ {
+ NullTest* nt = (NullTest*)node;
+
+ /*
+ * Only simple (Var IS NULL) expressions supported for now. Maybe we could
+ * use examine_variable to fix this?
+ */
+ if (! IsA(nt->arg, Var))
+ return true;
+
+ return mv_compatible_walker((Node*)(nt->arg), context);
+ }
+
if (IsA(node, Var))
{
Var * var = (Var*)node;
@@ -1031,7 +1374,7 @@ mv_compatible_walker(Node *node, mv_compatible_context *context)
/* unsupported structure (two variables or so) */
if (! ok)
return true;
-
+
/*
* If it's not a "<" or ">" or "=" operator, just ignore the clause.
* Otherwise note the relid and attnum for the variable. This uses the
@@ -1041,10 +1384,18 @@ mv_compatible_walker(Node *node, mv_compatible_context *context)
switch (get_oprrest(expr->opno))
{
case F_EQSEL:
-
/* equality conditions are compatible with all statistics */
break;
+ case F_SCALARLTSEL:
+ case F_SCALARGTSEL:
+
+ /* not compatible with functional dependencies */
+ if (! (context->types & MV_CLAUSE_TYPE_MCV))
+ return true; /* terminate */
+
+ break;
+
default:
/* unknown estimator */
@@ -1055,11 +1406,11 @@ mv_compatible_walker(Node *node, mv_compatible_context *context)
return mv_compatible_walker((Node *) var, context);
}
-
+
/* Node not explicitly supported, so terminate */
return true;
}
-
+
/*
* Determines whether the clause is compatible with multivariate stats,
* and if it is, returns some additional information - varno (index
@@ -1078,10 +1429,11 @@ mv_compatible_walker(Node *node, mv_compatible_context *context)
* evaluate them using multivariate stats.
*/
static bool
-clause_is_mv_compatible(Node *clause, Index relid, AttrNumber *attnum)
+clause_is_mv_compatible(Node *clause, Index relid, Bitmapset **attnums, int types)
{
mv_compatible_context context;
+ context.types = types;
context.varno = relid;
context.varattnos = NULL; /* no attnums */
@@ -1089,7 +1441,7 @@ clause_is_mv_compatible(Node *clause, Index relid, AttrNumber *attnum)
return false;
/* remember the newly collected attnums */
- *attnum = bms_singleton_member(context.varattnos);
+ *attnums = bms_add_members(*attnums, context.varattnos);
return true;
}
@@ -1394,24 +1746,39 @@ fdeps_filter_clauses(PlannerInfo *root,
foreach (lc, clauses)
{
- AttrNumber attnum;
+ Bitmapset *attnums = NULL;
Node *clause = (Node *) lfirst(lc);
- if (! clause_is_mv_compatible(clause, relid, &attnum))
+ if (! clause_is_mv_compatible(clause, relid, &attnums,
+ MV_CLAUSE_TYPE_FDEP))
/* clause incompatible with functional dependencies */
*reduced_clauses = lappend(*reduced_clauses, clause);
- else if (! bms_is_member(attnum, deps_attnums))
+ else if (bms_num_members(attnums) > 1)
+
+ /*
+ * clause referencing multiple attributes (strange, should
+ * this be handled by clause_is_mv_compatible directly)
+ */
+ *reduced_clauses = lappend(*reduced_clauses, clause);
+
+ else if (! bms_is_member(bms_singleton_member(attnums), deps_attnums))
/* clause not covered by the dependencies */
*reduced_clauses = lappend(*reduced_clauses, clause);
else
{
+ /* ok, clause compatible with existing dependencies */
+ Assert(bms_num_members(attnums) == 1);
+
*deps_clauses = lappend(*deps_clauses, clause);
- clause_attnums = bms_add_member(clause_attnums, attnum);
+ clause_attnums = bms_add_member(clause_attnums,
+ bms_singleton_member(attnums));
}
+
+ bms_free(attnums);
}
return clause_attnums;
@@ -1637,6 +2004,9 @@ has_stats(List *stats, int type)
if ((type & MV_CLAUSE_TYPE_FDEP) && stat->deps_built)
return true;
+
+ if ((type & MV_CLAUSE_TYPE_MCV) && stat->mcv_built)
+ return true;
}
return false;
@@ -1652,3 +2022,392 @@ find_stats(PlannerInfo *root, Index relid)
return root->simple_rel_array[relid]->mvstatlist;
}
+
+/*
+ * Estimate selectivity of clauses using a MCV list.
+ *
+ * If there's no MCV list for the stats, the function returns 0.0.
+ *
+ * While computing the estimate, the function checks whether all the
+ * columns were matched with an equality condition. If that's the case,
+ * we can skip processing the histogram, as there can be no rows in
+ * it with the same values - all the rows matching the condition are
+ * represented by the MCV item. This can only happen with equality
+ * on all the attributes.
+ *
+ * The algorithm works like this:
+ *
+ * 1) mark all items as 'match'
+ * 2) walk through all the clauses
+ * 3) for a particular clause, walk through all the items
+ * 4) skip items that are already 'no match'
+ * 5) check clause for items that still match
+ * 6) sum frequencies for items to get selectivity
+ *
+ * The function also returns the frequency of the least frequent item
+ * on the MCV list, which may be useful for clamping estimate from the
+ * histogram (all items not present in the MCV list are less frequent).
+ * This however seems useful only for cases with conditions on all
+ * attributes.
+ *
+ * TODO This only handles AND-ed clauses, but it might work for OR-ed
+ * lists too - it just needs to reverse the logic a bit. I.e. start
+ * with 'no match' for all items, and mark the items as a match
+ * as the clauses are processed (and skip items that are 'match').
+ */
+static Selectivity
+clauselist_mv_selectivity_mcvlist(PlannerInfo *root, List *clauses,
+ MVStatisticInfo *mvstats, bool *fullmatch,
+ Selectivity *lowsel)
+{
+ int i;
+ Selectivity s = 0.0;
+ Selectivity u = 0.0;
+
+ MCVList mcvlist = NULL;
+ int nmatches = 0;
+
+ /* match/mismatch bitmap for each MCV item */
+ char * matches = NULL;
+
+ Assert(clauses != NIL);
+ Assert(list_length(clauses) >= 2);
+
+ /* there's no MCV list built yet */
+ if (! mvstats->mcv_built)
+ return 0.0;
+
+ mcvlist = load_mv_mcvlist(mvstats->mvoid);
+
+ Assert(mcvlist != NULL);
+ Assert(mcvlist->nitems > 0);
+
+ /* by default all the MCV items match the clauses fully */
+ matches = palloc0(sizeof(char) * mcvlist->nitems);
+ memset(matches, MVSTATS_MATCH_FULL, sizeof(char)*mcvlist->nitems);
+
+ /* number of matching MCV items */
+ nmatches = mcvlist->nitems;
+
+ nmatches = update_match_bitmap_mcvlist(root, clauses,
+ mvstats->stakeys, mcvlist,
+ nmatches, matches,
+ lowsel, fullmatch, false);
+
+ /* sum frequencies for all the matching MCV items */
+ for (i = 0; i < mcvlist->nitems; i++)
+ {
+ /* used to 'scale' for MCV lists not covering all tuples */
+ u += mcvlist->items[i]->frequency;
+
+ if (matches[i] != MVSTATS_MATCH_NONE)
+ s += mcvlist->items[i]->frequency;
+ }
+
+ pfree(matches);
+ pfree(mcvlist);
+
+ return s*u;
+}
+
+/*
+ * Evaluate clauses using the MCV list, and update the match bitmap.
+ *
+ * The bitmap may be already partially set, so this is really a way to
+ * combine results of several clause lists - either when computing
+ * conditional probability P(A|B) or a combination of AND/OR clauses.
+ *
+ * TODO This works with 'bitmap' where each bit is represented as a char,
+ * which is slightly wasteful. Instead, we could use a regular
+ * bitmap, reducing the size to ~1/8. Another thing is merging the
+ * bitmaps using & and |, which might be faster than min/max.
+ */
+static int
+update_match_bitmap_mcvlist(PlannerInfo *root, List *clauses,
+ int2vector *stakeys, MCVList mcvlist,
+ int nmatches, char * matches,
+ Selectivity *lowsel, bool *fullmatch,
+ bool is_or)
+{
+ int i;
+ ListCell * l;
+
+ Bitmapset *eqmatches = NULL; /* attributes with equality matches */
+
+ /* The bitmap may be partially built. */
+ Assert(nmatches >= 0);
+ Assert(nmatches <= mcvlist->nitems);
+ Assert(clauses != NIL);
+ Assert(list_length(clauses) >= 1);
+ Assert(mcvlist != NULL);
+ Assert(mcvlist->nitems > 0);
+
+ /* No possible matches (only works for AND-ded clauses) */
+ if (((nmatches == 0) && (! is_or)) ||
+ ((nmatches == mcvlist->nitems) && is_or))
+ return nmatches;
+
+ /*
+ * find the lowest frequency in the MCV list
+ *
+ * We need to do that here, because we do various tricks in the following
+ * code - skipping items already ruled out, etc.
+ *
+ * XXX A loop is necessary because the MCV list is not sorted by frequency.
+ */
+ *lowsel = 1.0;
+ for (i = 0; i < mcvlist->nitems; i++)
+ {
+ MCVItem item = mcvlist->items[i];
+
+ if (item->frequency < *lowsel)
+ *lowsel = item->frequency;
+ }
+
+ /*
+ * Loop through the list of clauses, and for each of them evaluate
+ * all the MCV items not yet eliminated by the preceding clauses.
+ */
+ foreach (l, clauses)
+ {
+ Node * clause = (Node*)lfirst(l);
+
+ /* if it's a RestrictInfo, then extract the clause */
+ if (IsA(clause, RestrictInfo))
+ clause = (Node*)((RestrictInfo*)clause)->clause;
+
+ /* if there are no remaining matches possible, we can stop */
+ if (((nmatches == 0) && (! is_or)) ||
+ ((nmatches == mcvlist->nitems) && is_or))
+ break;
+
+ /* it's either OpClause, or NullTest */
+ if (is_opclause(clause))
+ {
+ OpExpr *expr = (OpExpr*)clause;
+ bool varonleft = true;
+ bool ok;
+ FmgrInfo opproc;
+
+ /* get procedure computing operator selectivity */
+ RegProcedure oprrest = get_oprrest(expr->opno);
+
+ fmgr_info(get_opcode(expr->opno), &opproc);
+
+ ok = (NumRelids(clause) == 1) &&
+ (is_pseudo_constant_clause(lsecond(expr->args)) ||
+ (varonleft = false,
+ is_pseudo_constant_clause(linitial(expr->args))));
+
+ if (ok)
+ {
+
+ FmgrInfo gtproc;
+ Var * var = (varonleft) ? linitial(expr->args) : lsecond(expr->args);
+ Const * cst = (varonleft) ? lsecond(expr->args) : linitial(expr->args);
+ bool isgt = (! varonleft);
+
+ TypeCacheEntry *typecache
+ = lookup_type_cache(var->vartype, TYPECACHE_GT_OPR);
+
+ /* FIXME proper matching attribute to dimension */
+ int idx = mv_get_index(var->varattno, stakeys);
+
+ fmgr_info(get_opcode(typecache->gt_opr), >proc);
+
+ /*
+ * Walk through the MCV items and evaluate the current clause. We can
+ * skip items that were already ruled out, and terminate if there are
+ * no remaining MCV items that might possibly match.
+ */
+ for (i = 0; i < mcvlist->nitems; i++)
+ {
+ bool mismatch = false;
+ MCVItem item = mcvlist->items[i];
+
+ /*
+ * If there are no more matches (AND) or no remaining unmatched
+ * items (OR), we can stop processing this clause.
+ */
+ if (((nmatches == 0) && (! is_or)) ||
+ ((nmatches == mcvlist->nitems) && is_or))
+ break;
+
+ /*
+ * For AND-lists, we can also mark NULL items as 'no match' (and
+ * then skip them). For OR-lists this is not possible.
+ */
+ if ((! is_or) && item->isnull[idx])
+ matches[i] = MVSTATS_MATCH_NONE;
+
+ /* skip MCV items that were already ruled out */
+ if ((! is_or) && (matches[i] == MVSTATS_MATCH_NONE))
+ continue;
+ else if (is_or && (matches[i] == MVSTATS_MATCH_FULL))
+ continue;
+
+ switch (oprrest)
+ {
+ case F_EQSEL:
+ /*
+ * We don't care about isgt in equality, because it does not
+ * matter whether it's (var = const) or (const = var).
+ */
+ mismatch = ! DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ cst->constvalue,
+ item->values[idx]));
+
+ if (! mismatch)
+ eqmatches = bms_add_member(eqmatches, idx);
+
+ break;
+
+ case F_SCALARLTSEL: /* column < constant */
+ case F_SCALARGTSEL: /* column > constant */
+
+ /*
+ * First check whether the constant is below the lower boundary (in that
+ * case we can skip the bucket, because there's no overlap).
+ */
+ mismatch = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ cst->constvalue,
+ item->values[idx]));
+
+ /* invert the result if isgt=true */
+ mismatch = (isgt) ? (! mismatch) : mismatch;
+ break;
+ }
+
+ /* XXX The conditions on matches[i] are not needed, as we
+ * skip MCV items that can't become true/false, depending
+ * on the current flag. See beginning of the loop over
+ * MCV items.
+ */
+
+ if ((is_or) && (matches[i] == MVSTATS_MATCH_NONE) && (! mismatch))
+ {
+ /* OR - was MATCH_NONE, but will be MATCH_FULL */
+ matches[i] = MVSTATS_MATCH_FULL;
+ ++nmatches;
+ continue;
+ }
+ else if ((! is_or) && (matches[i] == MVSTATS_MATCH_FULL) && mismatch)
+ {
+ /* AND - was MATC_FULL, but will be MATCH_NONE */
+ matches[i] = MVSTATS_MATCH_NONE;
+ --nmatches;
+ continue;
+ }
+
+ }
+ }
+ }
+ else if (IsA(clause, NullTest))
+ {
+ NullTest * expr = (NullTest*)clause;
+ Var * var = (Var*)(expr->arg);
+
+ /* FIXME proper matching attribute to dimension */
+ int idx = mv_get_index(var->varattno, stakeys);
+
+ /*
+ * Walk through the MCV items and evaluate the current clause. We can
+ * skip items that were already ruled out, and terminate if there are
+ * no remaining MCV items that might possibly match.
+ */
+ for (i = 0; i < mcvlist->nitems; i++)
+ {
+ MCVItem item = mcvlist->items[i];
+
+ /* if there are no more matches, we can stop processing this clause */
+ if (nmatches == 0)
+ break;
+
+ /* skip MCV items that were already ruled out */
+ if (matches[i] == MVSTATS_MATCH_NONE)
+ continue;
+
+ /* if the clause mismatches the MCV item, set it as MATCH_NONE */
+ if (((expr->nulltesttype == IS_NULL) && (! item->isnull[idx])) ||
+ ((expr->nulltesttype == IS_NOT_NULL) && (item->isnull[idx])))
+ {
+ matches[i] = MVSTATS_MATCH_NONE;
+ --nmatches;
+ }
+ }
+ }
+ else if (or_clause(clause) || and_clause(clause))
+ {
+ /* AND/OR clause, with all clauses compatible with the selected MV stat */
+
+ int i;
+ BoolExpr *orclause = ((BoolExpr*)clause);
+ List *orclauses = orclause->args;
+
+ /* match/mismatch bitmap for each MCV item */
+ int or_nmatches = 0;
+ char * or_matches = NULL;
+
+ Assert(orclauses != NIL);
+ Assert(list_length(orclauses) >= 2);
+
+ /* number of matching MCV items */
+ or_nmatches = mcvlist->nitems;
+
+ /* by default none of the MCV items matches the clauses */
+ or_matches = palloc0(sizeof(char) * or_nmatches);
+
+ if (or_clause(clause))
+ {
+ /* OR clauses assume nothing matches, initially */
+ memset(or_matches, MVSTATS_MATCH_NONE, sizeof(char)*or_nmatches);
+ or_nmatches = 0;
+ }
+ else
+ {
+ /* AND clauses assume nothing matches, initially */
+ memset(or_matches, MVSTATS_MATCH_FULL, sizeof(char)*or_nmatches);
+ }
+
+ /* build the match bitmap for the OR-clauses */
+ or_nmatches = update_match_bitmap_mcvlist(root, orclauses,
+ stakeys, mcvlist,
+ or_nmatches, or_matches,
+ lowsel, fullmatch, or_clause(clause));
+
+ /* merge the bitmap into the existing one*/
+ for (i = 0; i < mcvlist->nitems; i++)
+ {
+ /*
+ * To AND-merge the bitmaps, a MIN() semantics is used.
+ * For OR-merge, use MAX().
+ *
+ * FIXME this does not decrease the number of matches
+ */
+ UPDATE_RESULT(matches[i], or_matches[i], is_or);
+ }
+
+ pfree(or_matches);
+
+ }
+ else
+ {
+ elog(ERROR, "unknown clause type: %d", clause->type);
+ }
+ }
+
+ /*
+ * If all the columns were matched by equality, it's a full match.
+ * In this case there can be just a single MCV item, matching the
+ * clause (if there were two, both would match the other one).
+ */
+ *fullmatch = (bms_num_members(eqmatches) == mcvlist->ndimensions);
+
+ /* free the allocated pieces */
+ if (eqmatches)
+ pfree(eqmatches);
+
+ return nmatches;
+}
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index 7fb2088..8394111 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -412,7 +412,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
mvstat = (Form_pg_mv_statistic) GETSTRUCT(htup);
/* unavailable stats are not interesting for the planner */
- if (mvstat->deps_built)
+ if (mvstat->deps_built || mvstat->mcv_built)
{
info = makeNode(MVStatisticInfo);
@@ -421,9 +421,11 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
/* enabled statistics */
info->deps_enabled = mvstat->deps_enabled;
+ info->mcv_enabled = mvstat->mcv_enabled;
/* built/available statistics */
info->deps_built = mvstat->deps_built;
+ info->mcv_built = mvstat->mcv_built;
/* stakeys */
adatum = SysCacheGetAttr(MVSTATOID, htup,
diff --git a/src/backend/utils/mvstats/Makefile b/src/backend/utils/mvstats/Makefile
index 099f1ed..f9bf10c 100644
--- a/src/backend/utils/mvstats/Makefile
+++ b/src/backend/utils/mvstats/Makefile
@@ -12,6 +12,6 @@ subdir = src/backend/utils/mvstats
top_builddir = ../../../..
include $(top_builddir)/src/Makefile.global
-OBJS = common.o dependencies.o
+OBJS = common.o dependencies.o mcv.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/utils/mvstats/README.mcv b/src/backend/utils/mvstats/README.mcv
new file mode 100644
index 0000000..e93cfe4
--- /dev/null
+++ b/src/backend/utils/mvstats/README.mcv
@@ -0,0 +1,137 @@
+MCV lists
+=========
+
+Multivariate MCV (most-common values) lists are a straightforward extension of
+regular MCV list, tracking most frequent combinations of values for a group of
+attributes.
+
+This works particularly well for columns with a small number of distinct values,
+as the list may include all the combinations and approximate the distribution
+very accurately.
+
+For columns with large number of distinct values (e.g. those with continuous
+domains), the list will only track the most frequent combinations. If the
+distribution is mostly uniform (all combinations about equally frequent), the
+MCV list will be empty.
+
+Estimates of some clauses (e.g. equality) based on MCV lists are more accurate
+than when using histograms.
+
+Also, MCV lists don't necessarily require sorting of the values (the fact that
+we use sorting when building them is implementation detail), but even more
+importantly the ordering is not built into the approximation (while histograms
+are built on ordering). So MCV lists work well even for attributes where the
+ordering of the data type is disconnected from the meaning of the data. For
+example we know how to sort strings, but it's unlikely to make much sense for
+city names (or other label-like attributes).
+
+
+Selectivity estimation
+----------------------
+
+The estimation, implemented in clauselist_mv_selectivity_mcvlist(), is quite
+simple in principle - we need to identify MCV items matching all the clauses
+and sum frequencies of all those items.
+
+Currently MCV lists support estimation of the following clause types:
+
+ (a) equality clauses WHERE (a = 1) AND (b = 2)
+ (b) inequality clauses WHERE (a < 1) AND (b >= 2)
+ (c) NULL clauses WHERE (a IS NULL) AND (b IS NOT NULL)
+ (d) OR clauses WHERE (a < 1) OR (b >= 2)
+
+It's possible to add support for additional clauses, for example:
+
+ (e) multi-var clauses WHERE (a > b)
+
+and possibly others. These are tasks for the future, not yet implemented.
+
+
+Estimating equality clauses
+---------------------------
+
+When computing selectivity estimate for equality clauses
+
+ (a = 1) AND (b = 2)
+
+we can do this estimate pretty exactly assuming that two conditions are met:
+
+ (1) there's an equality condition on all attributes of the statistic
+
+ (2) we find a matching item in the MCV list
+
+In this case we know the MCV item represents all tuples matching the clauses,
+and the selectivity estimate is complete (i.e. we don't need to perform
+estimation using the histogram). This is what we call 'full match'.
+
+When only (1) holds, but there's no matching MCV item, we don't know whether
+there are no such rows or just are not very frequent. We can however use the
+frequency of the least frequent MCV item as an upper bound for the selectivity.
+
+For a combination of equality conditions (not full-match case) we can clamp the
+selectivity by the minimum of selectivities for each condition. For example if
+we know the number of distinct values for each column, we can use 1/ndistinct
+as a per-column estimate. Or rather 1/ndistinct + selectivity derived from the
+MCV list.
+
+We should also probably only use the 'residual ndistinct' by exluding the items
+included in the MCV list (and also residual frequency):
+
+ f = (1.0 - sum(MCV frequencies)) / (ndistinct - ndistinct(MCV list))
+
+but it's worth pointing out the ndistinct values are multi-variate for the
+columns referenced by the equality conditions.
+
+Note: Only the "full match" limit is currently implemented.
+
+
+Hashed MCV (not yet implemented)
+--------------------------------
+
+Regular MCV lists have to include actual values for each item, so if those items
+are large the list may be quite large. This is especially true for multi-variate
+MCV lists, although the current implementation partially mitigates this by
+performing de-duplicating the values before storing them on disk.
+
+It's possible to only store hashes (32-bit values) instead of the actual values,
+significantly reducing the space requirements. Obviously, this would only make
+the MCV lists useful for estimating equality conditions (assuming the 32-bit
+hashes make the collisions rare enough).
+
+This might also complicate matching the columns to available stats.
+
+
+TODO Consider implementing hashed MCV list, storing just 32-bit hashes instead
+ of the actual values. This type of MCV list will be useful only for
+ estimating equality clauses, and will reduce space requirements for large
+ varlena types (in such cases we usually only want equality anyway).
+
+TODO Currently there's no logic to consider building only a MCV list (and not
+ building the histogram at all), except for doing this decision manually in
+ ADD STATISTICS.
+
+
+Inspecting the MCV list
+-----------------------
+
+Inspecting the regular (per-attribute) MCV lists is trivial, as it's enough
+to select the columns from pg_stats - the data is encoded as anyarrays, so we
+simply get the text representation of the arrays.
+
+With multivariate MCV lits it's not that simple due to the possible mix of
+data types. It might be possible to produce similar array-like representation,
+but that'd unnecessarily complicate further processing and analysis of the MCV
+list. Instead, there's a SRF function providing values, frequencies etc.
+
+ SELECT * FROM pg_mv_mcv_items();
+
+It has two input parameters:
+
+ oid - OID of the MCV list (pg_mv_statistic.staoid)
+
+and produces a table with these columns:
+
+ - item ID (0...nitems-1)
+ - values (string array)
+ - nulls only (boolean array)
+ - frequency (double precision)
diff --git a/src/backend/utils/mvstats/README.stats b/src/backend/utils/mvstats/README.stats
index a38ea7b..5c5c59a 100644
--- a/src/backend/utils/mvstats/README.stats
+++ b/src/backend/utils/mvstats/README.stats
@@ -8,9 +8,50 @@ not true, resulting in estimation errors.
Multivariate stats track different types of dependencies between the columns,
hopefully improving the estimates.
-Currently we only have one kind of multivariate statistics - soft functional
-dependencies, and we use it to improve estimates of equality clauses. See
-README.dependencies for details.
+
+Types of statistics
+-------------------
+
+Currently we only have two kinds of multivariate statistics
+
+ (a) soft functional dependencies (README.dependencies)
+
+ (b) MCV lists (README.mcv)
+
+
+Compatible clause types
+-----------------------
+
+Each type of statistics may be used to estimate some subset of clause types.
+
+ (a) functional dependencies - equality clauses (AND), possibly IS NULL
+
+ (b) MCV list - equality and inequality clauses, IS [NOT] NULL, AND/OR
+
+Currently only simple operator clauses (Var op Const) are supported, but it's
+possible to support more complex clause types, e.g. (Var op Var).
+
+
+Complex clauses
+---------------
+
+We also support estimating more complex clauses - essentially AND/OR clauses
+with (Var op Const) as leaves, as long as all the referenced attributes are
+covered by a single statistics.
+
+For example this condition
+
+ (a=1) AND ((b=2) OR ((c=3) AND (d=4)))
+
+may be estimated using statistics on (a,b,c,d). If we only have statistics on
+(b,c,d) we may estimate the second part, and estimate (a=1) using simple stats.
+
+If we only have statistics on (a,b,c) we can't apply it at all at this point,
+but it's worth pointing out clauselist_selectivity() works recursively and when
+handling the second part (the OR-clause), we'll be able to apply the statistics.
+
+Note: The multi-statistics estimation patch also makes it possible to pass some
+clauses as 'conditions' into the deeper parts of the expression tree.
Selectivity estimation
@@ -23,14 +64,48 @@ When estimating selectivity, we aim to achieve several things:
(b) minimize the overhead, especially when no suitable multivariate stats
exist (so if you are not using multivariate stats, there's no overhead)
-This clauselist_selectivity() performs several inexpensive checks first, before
+Thus clauselist_selectivity() performs several inexpensive checks first, before
even attempting to do the more expensive estimation.
(1) check if there are multivariate stats on the relation
- (2) check there are at least two attributes referenced by clauses compatible
- with multivariate statistics (equality clauses for func. dependencies)
+ (2) check that there are functional dependencies on the table, and that
+ there are at least two attributes referenced by compatible clauses
+ (equality clauses for func. dependencies)
(3) perform reduction of equality clauses using func. dependencies
- (4) estimate the reduced list of clauses using regular statistics
+ (4) check that there are multivariate MCV lists on the table, and that
+ there are at least two attributes referenced by compatible clauses
+ (equalities, inequalities, etc.)
+
+ (5) find the best multivariate statistics (matching the most conditions)
+ and use it to compute the estimate
+
+ (6) estimate the remaining clauses (not estimated using multivariate stats)
+ using the regular per-column statistics
+
+Whenever we find there are no suitable stats, we skip the expensive steps.
+
+
+Further (possibly crazy) ideas
+------------------------------
+
+Currently the clauses are only estimated using a single statistics, even if
+there are multiple candidate statistics - for example assume we have statistics
+on (a,b,c) and (b,c,d), and estimate conditions
+
+ (b = 1) AND (c = 2)
+
+Then both statistics may be used, but we only use one of them. Maybe we could
+use compute estimates using all candidate stats, and somehow aggregate them
+into the final estimate by using average or median.
+
+Some stats may give better estimates than others, but it's very difficult to say
+in advance which stats are the best (it depends on the number of buckets, number
+of additional columns not referenced in the clauses, type of condition etc.).
+
+But of course, this may result in expensive estimation (CPU-wise).
+
+So we might add a GUC to choose between a simple (single statistics) and thus
+multi-statistic estimation, possibly table-level parameter (ALTER TABLE ...).
diff --git a/src/backend/utils/mvstats/common.c b/src/backend/utils/mvstats/common.c
index bd200bc..d1da714 100644
--- a/src/backend/utils/mvstats/common.c
+++ b/src/backend/utils/mvstats/common.c
@@ -16,12 +16,14 @@
#include "common.h"
+#include "utils/array.h"
+
static VacAttrStats ** lookup_var_attr_stats(int2vector *attrs,
- int natts, VacAttrStats **vacattrstats);
+ int natts,
+ VacAttrStats **vacattrstats);
static List* list_mv_stats(Oid relid);
-
/*
* Compute requested multivariate stats, using the rows sampled for the
* plain (single-column) stats.
@@ -49,6 +51,8 @@ build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
int j;
MVStatisticInfo *stat = (MVStatisticInfo *)lfirst(lc);
MVDependencies deps = NULL;
+ MCVList mcvlist = NULL;
+ int numrows_filtered = 0;
VacAttrStats **stats = NULL;
int numatts = 0;
@@ -87,8 +91,12 @@ build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
if (stat->deps_enabled)
deps = build_mv_dependencies(numrows, rows, attrs, stats);
+ /* build the MCV list */
+ if (stat->mcv_enabled)
+ mcvlist = build_mv_mcvlist(numrows, rows, attrs, stats, &numrows_filtered);
+
/* store the histogram / MCV list in the catalog */
- update_mv_stats(stat->mvoid, deps, attrs);
+ update_mv_stats(stat->mvoid, deps, mcvlist, attrs, stats);
}
}
@@ -166,6 +174,8 @@ list_mv_stats(Oid relid)
info->stakeys = buildint2vector(stats->stakeys.values, stats->stakeys.dim1);
info->deps_enabled = stats->deps_enabled;
info->deps_built = stats->deps_built;
+ info->mcv_enabled = stats->mcv_enabled;
+ info->mcv_built = stats->mcv_built;
result = lappend(result, info);
}
@@ -180,8 +190,56 @@ list_mv_stats(Oid relid)
return result;
}
+
+/*
+ * Find attnims of MV stats using the mvoid.
+ */
+int2vector*
+find_mv_attnums(Oid mvoid, Oid *relid)
+{
+ ArrayType *arr;
+ Datum adatum;
+ bool isnull;
+ HeapTuple htup;
+ int2vector *keys;
+
+ /* Prepare to scan pg_mv_statistic for entries having indrelid = this rel. */
+ htup = SearchSysCache1(MVSTATOID,
+ ObjectIdGetDatum(mvoid));
+
+ /* XXX syscache contains OIDs of deleted stats (not invalidated) */
+ if (! HeapTupleIsValid(htup))
+ return NULL;
+
+ /* starelid */
+ adatum = SysCacheGetAttr(MVSTATOID, htup,
+ Anum_pg_mv_statistic_starelid, &isnull);
+ Assert(!isnull);
+
+ *relid = DatumGetObjectId(adatum);
+
+ /* stakeys */
+ adatum = SysCacheGetAttr(MVSTATOID, htup,
+ Anum_pg_mv_statistic_stakeys, &isnull);
+ Assert(!isnull);
+
+ arr = DatumGetArrayTypeP(adatum);
+
+ keys = buildint2vector((int16 *) ARR_DATA_PTR(arr),
+ ARR_DIMS(arr)[0]);
+ ReleaseSysCache(htup);
+
+ /* TODO maybe save the list into relcache, as in RelationGetIndexList
+ * (which was used as an inspiration of this one)?. */
+
+ return keys;
+}
+
+
void
-update_mv_stats(Oid mvoid, MVDependencies dependencies, int2vector *attrs)
+update_mv_stats(Oid mvoid,
+ MVDependencies dependencies, MCVList mcvlist,
+ int2vector *attrs, VacAttrStats **stats)
{
HeapTuple stup,
oldtup;
@@ -206,18 +264,29 @@ update_mv_stats(Oid mvoid, MVDependencies dependencies, int2vector *attrs)
= PointerGetDatum(serialize_mv_dependencies(dependencies));
}
+ if (mcvlist != NULL)
+ {
+ bytea * data = serialize_mv_mcvlist(mcvlist, attrs, stats);
+ nulls[Anum_pg_mv_statistic_stamcv -1] = (data == NULL);
+ values[Anum_pg_mv_statistic_stamcv - 1] = PointerGetDatum(data);
+ }
+
/* always replace the value (either by bytea or NULL) */
replaces[Anum_pg_mv_statistic_stadeps -1] = true;
+ replaces[Anum_pg_mv_statistic_stamcv -1] = true;
/* always change the availability flags */
nulls[Anum_pg_mv_statistic_deps_built -1] = false;
+ nulls[Anum_pg_mv_statistic_mcv_built -1] = false;
nulls[Anum_pg_mv_statistic_stakeys-1] = false;
/* use the new attnums, in case we removed some dropped ones */
replaces[Anum_pg_mv_statistic_deps_built-1] = true;
+ replaces[Anum_pg_mv_statistic_mcv_built -1] = true;
replaces[Anum_pg_mv_statistic_stakeys -1] = true;
values[Anum_pg_mv_statistic_deps_built-1] = BoolGetDatum(dependencies != NULL);
+ values[Anum_pg_mv_statistic_mcv_built -1] = BoolGetDatum(mcvlist != NULL);
values[Anum_pg_mv_statistic_stakeys -1] = PointerGetDatum(attrs);
/* Is there already a pg_mv_statistic tuple for this attribute? */
@@ -246,6 +315,21 @@ update_mv_stats(Oid mvoid, MVDependencies dependencies, int2vector *attrs)
heap_close(sd, RowExclusiveLock);
}
+
+int
+mv_get_index(AttrNumber varattno, int2vector * stakeys)
+{
+ int i, idx = 0;
+ for (i = 0; i < stakeys->dim1; i++)
+ {
+ if (stakeys->values[i] < varattno)
+ idx += 1;
+ else
+ break;
+ }
+ return idx;
+}
+
/* multi-variate stats comparator */
/*
@@ -256,11 +340,15 @@ update_mv_stats(Oid mvoid, MVDependencies dependencies, int2vector *attrs)
int
compare_scalars_simple(const void *a, const void *b, void *arg)
{
- Datum da = *(Datum*)a;
- Datum db = *(Datum*)b;
- SortSupport ssup= (SortSupport) arg;
+ return compare_datums_simple(*(Datum*)a,
+ *(Datum*)b,
+ (SortSupport)arg);
+}
- return ApplySortComparator(da, false, db, false, ssup);
+int
+compare_datums_simple(Datum a, Datum b, SortSupport ssup)
+{
+ return ApplySortComparator(a, false, b, false, ssup);
}
/*
diff --git a/src/backend/utils/mvstats/common.h b/src/backend/utils/mvstats/common.h
index d96422d..9f1bd59 100644
--- a/src/backend/utils/mvstats/common.h
+++ b/src/backend/utils/mvstats/common.h
@@ -46,7 +46,15 @@ typedef struct
Datum value; /* a data value */
int tupno; /* position index for tuple it came from */
} ScalarItem;
-
+
+/* (de)serialization info */
+typedef struct DimensionInfo {
+ int nvalues; /* number of deduplicated values */
+ int nbytes; /* number of bytes (serialized) */
+ int typlen; /* pg_type.typlen */
+ bool typbyval; /* pg_type.typbyval */
+} DimensionInfo;
+
/* multi-sort */
typedef struct MultiSortSupportData {
int ndims; /* number of dimensions supported by the */
@@ -71,5 +79,6 @@ int multi_sort_compare_dim(int dim, const SortItem *a,
const SortItem *b, MultiSortSupport mss);
/* comparators, used when constructing multivariate stats */
+int compare_datums_simple(Datum a, Datum b, SortSupport ssup);
int compare_scalars_simple(const void *a, const void *b, void *arg);
int compare_scalars_partition(const void *a, const void *b, void *arg);
diff --git a/src/backend/utils/mvstats/mcv.c b/src/backend/utils/mvstats/mcv.c
new file mode 100644
index 0000000..551c934
--- /dev/null
+++ b/src/backend/utils/mvstats/mcv.c
@@ -0,0 +1,1094 @@
+/*-------------------------------------------------------------------------
+ *
+ * mcv.c
+ * POSTGRES multivariate MCV lists
+ *
+ *
+ * Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/utils/mvstats/mcv.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "fmgr.h"
+#include "funcapi.h"
+
+#include "utils/lsyscache.h"
+
+#include "common.h"
+
+/*
+ * Each serialized item needs to store (in this order):
+ *
+ * - indexes (ndim * sizeof(int32))
+ * - null flags (ndim * sizeof(bool))
+ * - frequency (sizeof(double))
+ *
+ * So in total:
+ *
+ * ndim * (sizeof(int32) + sizeof(bool)) + sizeof(double)
+ */
+#define ITEM_SIZE(ndims) \
+ (ndims * (sizeof(uint16) + sizeof(bool)) + sizeof(double))
+
+/* pointers into a flat serialized item of ITEM_SIZE(n) bytes */
+#define ITEM_INDEXES(item) ((uint16*)item)
+#define ITEM_NULLS(item,ndims) ((bool*)(ITEM_INDEXES(item) + ndims))
+#define ITEM_FREQUENCY(item,ndims) ((double*)(ITEM_NULLS(item,ndims) + ndims))
+
+/*
+ * Builds MCV list from sample rows, and removes rows represented by
+ * the MCV list from the sample (the number of remaining sample rows is
+ * returned by the numrows_filtered parameter).
+ *
+ * The method is quite simple - in short it does about these steps:
+ *
+ * (1) sort the data (default collation, '<' for the data type)
+ *
+ * (2) count distinct groups, decide how many to keep
+ *
+ * (3) build the MCV list using the threshold determined in (2)
+ *
+ * (4) remove rows represented by the MCV from the sample
+ *
+ * For more details, see the comments in the code.
+ *
+ * FIXME Use max_mcv_items from ALTER TABLE ADD STATISTICS command.
+ *
+ * FIXME Single-dimensional MCV is sorted by frequency (descending). We
+ * should do that too, because when walking through the list we
+ * want to check the most frequent items first.
+ *
+ * TODO We're using Datum (8B), even for data types (e.g. int4 or
+ * float4). Maybe we could save some space here, but the bytea
+ * compression should handle it just fine.
+ *
+ * TODO This probably should not use the ndistinct directly (as computed
+ * from the table, but rather estimate the number of distinct
+ * values in the table), no?
+ */
+MCVList
+build_mv_mcvlist(int numrows, HeapTuple *rows, int2vector *attrs,
+ VacAttrStats **stats, int *numrows_filtered)
+{
+ int i, j;
+ int numattrs = attrs->dim1;
+ int ndistinct = 0;
+ int mcv_threshold = 0;
+ int count = 0;
+ int nitems = 0;
+
+ MCVList mcvlist = NULL;
+
+ /* Sort by multiple columns (using array of SortSupport) */
+ MultiSortSupport mss = multi_sort_init(numattrs);
+
+ /*
+ * Preallocate space for all the items as a single chunk, and point
+ * the items to the appropriate parts of the array.
+ */
+ SortItem *items = (SortItem*)palloc0(numrows * sizeof(SortItem));
+ Datum *values = (Datum*)palloc0(sizeof(Datum) * numrows * numattrs);
+ bool *isnull = (bool*)palloc0(sizeof(bool) * numrows * numattrs);
+
+ /* keep all the rows by default (as if there was no MCV list) */
+ *numrows_filtered = numrows;
+
+ for (i = 0; i < numrows; i++)
+ {
+ items[i].values = &values[i * numattrs];
+ items[i].isnull = &isnull[i * numattrs];
+ }
+
+ /* load the values/null flags from sample rows */
+ for (j = 0; j < numrows; j++)
+ for (i = 0; i < numattrs; i++)
+ items[j].values[i] = heap_getattr(rows[j], attrs->values[i],
+ stats[i]->tupDesc, &items[j].isnull[i]);
+
+ /* prepare the sort functions for all the attributes */
+ for (i = 0; i < numattrs; i++)
+ multi_sort_add_dimension(mss, i, i, stats);
+
+ /* do the sort, using the multi-sort */
+ qsort_arg((void *) items, numrows, sizeof(SortItem),
+ multi_sort_compare, mss);
+
+ /*
+ * Count the number of distinct groups - just walk through the
+ * sorted list and count the number of key changes. We use this to
+ * determine the threshold (125% of the average frequency).
+ */
+ ndistinct = 1;
+ for (i = 1; i < numrows; i++)
+ if (multi_sort_compare(&items[i], &items[i-1], mss) != 0)
+ ndistinct += 1;
+
+ /*
+ * Determine how many groups actually exceed the threshold, and then
+ * walk the array again and collect them into an array. We'll always
+ * require at least 4 rows per group.
+ *
+ * But if we can fit all the distinct values in the MCV list (i.e.
+ * if there are less distinct groups than MVSTAT_MCVLIST_MAX_ITEMS),
+ * we'll require only 2 rows per group.
+ *
+ * TODO For now the threshold is the same as in the single-column
+ * case (average + 25%), but maybe that's worth revisiting
+ * for the multivariate case.
+ *
+ * TODO We can do this only if we believe we got all the distinct
+ * values of the table.
+ *
+ * FIXME This should really reference mcv_max_items (from catalog)
+ * instead of the constant MVSTAT_MCVLIST_MAX_ITEMS.
+ */
+ mcv_threshold = 1.25 * numrows / ndistinct;
+ mcv_threshold = (mcv_threshold < 4) ? 4 : mcv_threshold;
+
+ if (ndistinct <= MVSTAT_MCVLIST_MAX_ITEMS)
+ mcv_threshold = 2;
+
+ /*
+ * Walk through the sorted data again, and see how many groups
+ * reach the mcv_threshold (and become an item in the MCV list).
+ */
+ count = 1;
+ for (i = 1; i <= numrows; i++)
+ {
+ /* last row or new group, so check if we exceed mcv_threshold */
+ if ((i == numrows) || (multi_sort_compare(&items[i], &items[i-1], mss) != 0))
+ {
+ /* group hits the threshold, count the group as MCV item */
+ if (count >= mcv_threshold)
+ nitems += 1;
+
+ count = 1;
+ }
+ else /* within group, so increase the number of items */
+ count += 1;
+ }
+
+ /* we know the number of MCV list items, so let's build the list */
+ if (nitems > 0)
+ {
+ /* allocate the MCV list structure, set parameters we know */
+ mcvlist = (MCVList)palloc0(sizeof(MCVListData));
+
+ mcvlist->magic = MVSTAT_MCV_MAGIC;
+ mcvlist->type = MVSTAT_MCV_TYPE_BASIC;
+ mcvlist->ndimensions = numattrs;
+ mcvlist->nitems = nitems;
+
+ /*
+ * Preallocate Datum/isnull arrays (not as a single chunk, as
+ * we'll pass this outside this method and thus it needs to be
+ * easy to pfree() the data (and we wouldn't know where the
+ * arrays start).
+ *
+ * TODO Maybe the reasoning that we can't allocate a single
+ * piece because we're passing it out is bogus? Who'd
+ * free a single item of the MCV list, anyway?
+ *
+ * TODO Maybe with a proper encoding (stuffing all the values
+ * into a list-level array, this will be untrue)?
+ */
+ mcvlist->items = (MCVItem*)palloc0(sizeof(MCVItem)*nitems);
+
+ for (i = 0; i < nitems; i++)
+ {
+ mcvlist->items[i] = (MCVItem)palloc0(sizeof(MCVItemData));
+ mcvlist->items[i]->values = (Datum*)palloc0(sizeof(Datum)*numattrs);
+ mcvlist->items[i]->isnull = (bool*)palloc0(sizeof(bool)*numattrs);
+ }
+
+ /*
+ * Repeat the same loop as above, but this time copy the data
+ * into the MCV list (for items exceeding the threshold).
+ *
+ * TODO Maybe we could simply remember indexes of the last item
+ * in each group (from the previous loop)?
+ */
+ count = 1;
+ nitems = 0;
+ for (i = 1; i <= numrows; i++)
+ {
+ /* last row or a new group */
+ if ((i == numrows) || (multi_sort_compare(&items[i], &items[i-1], mss) != 0))
+ {
+ /* count the MCV item if exceeding the threshold (and copy into the array) */
+ if (count >= mcv_threshold)
+ {
+ /* just pointer to the proper place in the list */
+ MCVItem item = mcvlist->items[nitems];
+
+ /* copy values from the _previous_ group (last item of) */
+ memcpy(item->values, items[(i-1)].values, sizeof(Datum) * numattrs);
+ memcpy(item->isnull, items[(i-1)].isnull, sizeof(bool) * numattrs);
+
+
+ /* and finally the group frequency */
+ item->frequency = (double)count / numrows;
+
+ /* next item */
+ nitems += 1;
+ }
+
+ count = 1;
+ }
+ else /* same group, just increase the number of items */
+ count += 1;
+ }
+
+ /* make sure the loops are consistent */
+ Assert(nitems == mcvlist->nitems);
+
+ /*
+ * Remove the rows matching the MCV list (i.e. keep only rows
+ * that are not represented by the MCV list).
+ *
+ * FIXME This implementation is rather naive, effectively O(N^2).
+ * As the MCV list grows, the check will take longer and
+ * longer. And as the number of sampled rows increases (by
+ * increasing statistics target), it will take longer and
+ * longer. One option is to sort the MCV items first and
+ * then perform a binary search.
+ *
+ * A better option would be keeping the ID of the row in
+ * the sort item, and then just walk through the items and
+ * mark rows to remove (in a bitmap of the same size).
+ * There's not space for that in SortItem at this moment,
+ * but it's trivial to add 'private' pointer, or just
+ * using another structure with extra field (starting with
+ * SortItem, so that the comparators etc. still work).
+ *
+ * Another option is to use the sorted array of items
+ * (because that's how we sorted the source data), and
+ * simply do a bsearch() into it. If we find a matching
+ * item, the row belongs to the MCV list.
+ */
+ if (nitems == ndistinct) /* all rows are covered by MCV items */
+ *numrows_filtered = 0;
+ else /* (nitems < ndistinct) && (nitems > 0) */
+ {
+ int nfiltered = 0;
+ HeapTuple *rows_filtered = (HeapTuple*)palloc0(sizeof(HeapTuple) * numrows);
+
+ /* used for the searches */
+ SortItem item, mcvitem;;
+
+ item.values = (Datum*)palloc0(numattrs * sizeof(Datum));
+ item.isnull = (bool*)palloc0(numattrs * sizeof(bool));
+
+ /*
+ * FIXME we don't need to allocate this, we can reference
+ * the MCV item directly ...
+ */
+ mcvitem.values = (Datum*)palloc0(numattrs * sizeof(Datum));
+ mcvitem.isnull = (bool*)palloc0(numattrs * sizeof(bool));
+
+ /* walk through the tuples, compare the values to MCV items */
+ for (i = 0; i < numrows; i++)
+ {
+ bool match = false;
+
+ /* collect the key values from the row */
+ for (j = 0; j < numattrs; j++)
+ item.values[j] = heap_getattr(rows[i], attrs->values[j],
+ stats[j]->tupDesc, &item.isnull[j]);
+
+ /* scan through the MCV list for matches */
+ for (j = 0; j < mcvlist->nitems; j++)
+ {
+ /*
+ * TODO Create a SortItem/MCVItem comparator so that
+ * we don't need to do memcpy() like crazy.
+ */
+ memcpy(mcvitem.values, mcvlist->items[j]->values,
+ numattrs * sizeof(Datum));
+ memcpy(mcvitem.isnull, mcvlist->items[j]->isnull,
+ numattrs * sizeof(bool));
+
+ if (multi_sort_compare(&item, &mcvitem, mss) == 0)
+ {
+ match = true;
+ break;
+ }
+ }
+
+ /* if no match in the MCV list, copy the row into the filtered ones */
+ if (! match)
+ memcpy(&rows_filtered[nfiltered++], &rows[i], sizeof(HeapTuple));
+ }
+
+ /* replace the rows and remember how many rows we kept */
+ memcpy(rows, rows_filtered, sizeof(HeapTuple) * nfiltered);
+ *numrows_filtered = nfiltered;
+
+ /* free all the data used here */
+ pfree(rows_filtered);
+ pfree(item.values);
+ pfree(item.isnull);
+ pfree(mcvitem.values);
+ pfree(mcvitem.isnull);
+ }
+ }
+
+ pfree(values);
+ pfree(items);
+ pfree(isnull);
+
+ return mcvlist;
+}
+
+
+/* fetch the MCV list (as a bytea) from the pg_mv_statistic catalog */
+MCVList
+load_mv_mcvlist(Oid mvoid)
+{
+ bool isnull = false;
+ Datum mcvlist;
+
+#ifdef USE_ASSERT_CHECKING
+ Form_pg_mv_statistic mvstat;
+#endif
+
+ /* Prepare to scan pg_mv_statistic for entries having indrelid = this rel. */
+ HeapTuple htup = SearchSysCache1(MVSTATOID, ObjectIdGetDatum(mvoid));
+
+ if (! HeapTupleIsValid(htup))
+ return NULL;
+
+#ifdef USE_ASSERT_CHECKING
+ mvstat = (Form_pg_mv_statistic) GETSTRUCT(htup);
+ Assert(mvstat->mcv_enabled && mvstat->mcv_built);
+#endif
+
+ mcvlist = SysCacheGetAttr(MVSTATOID, htup,
+ Anum_pg_mv_statistic_stamcv, &isnull);
+
+ Assert(!isnull);
+
+ ReleaseSysCache(htup);
+
+ return deserialize_mv_mcvlist(DatumGetByteaP(mcvlist));
+}
+
+/* print some basic info about the MCV list
+ *
+ * TODO Add info about what part of the table this covers.
+ */
+Datum
+pg_mv_stats_mcvlist_info(PG_FUNCTION_ARGS)
+{
+ bytea *data = PG_GETARG_BYTEA_P(0);
+ char *result;
+
+ MCVList mcvlist = deserialize_mv_mcvlist(data);
+
+ result = palloc0(128);
+ snprintf(result, 128, "nitems=%d", mcvlist->nitems);
+
+ pfree(mcvlist);
+
+ PG_RETURN_TEXT_P(cstring_to_text(result));
+}
+
+/* used to pass context into bsearch() */
+static SortSupport ssup_private = NULL;
+
+static int bsearch_comparator(const void * a, const void * b);
+
+/*
+ * Serialize MCV list into a bytea value. The basic algorithm is simple:
+ *
+ * (1) perform deduplication for each attribute (separately)
+ * (a) collect all (non-NULL) attribute values from all MCV items
+ * (b) sort the data (using 'lt' from VacAttrStats)
+ * (c) remove duplicate values from the array
+ *
+ * (2) serialize the arrays into a bytea value
+ *
+ * (3) process all MCV list items
+ * (a) replace values with indexes into the arrays
+ *
+ * Each attribute has to be processed separately, because we're mixing
+ * different datatypes, and we don't know what equality means for them.
+ * We're also mixing pass-by-value and pass-by-ref types, and so on.
+ *
+ * We'll use uint16 values for the indexes in step (3), as we don't
+ * allow more than 8k MCV items (see list max_mcv_items). We might
+ * increase this to 65k and still fit into uint16.
+ *
+ * We don't really expect the high compression as with histograms,
+ * because we're not doing any bucket splits etc. (which is the source
+ * of high redundancy there), but we need to do it anyway as we need
+ * to serialize varlena values etc. We might invent another way to
+ * serialize MCV lists, but let's keep it consistent.
+ *
+ * FIXME This probably leaks memory, or at least uses it inefficiently
+ * (many small palloc() calls instead of a large one).
+ *
+ * TODO Consider using 16-bit values for the indexes in step (3).
+ *
+ * TODO Consider packing boolean flags (NULL) for each item into 'char'
+ * or a longer type (instead of using an array of bool items).
+ */
+bytea *
+serialize_mv_mcvlist(MCVList mcvlist, int2vector *attrs,
+ VacAttrStats **stats)
+{
+ int i, j;
+ int ndims = mcvlist->ndimensions;
+ int itemsize = ITEM_SIZE(ndims);
+
+ Size total_length = 0;
+
+ char *item = palloc0(itemsize);
+
+ /* serialized items (indexes into arrays, etc.) */
+ bytea *output;
+ char *data = NULL;
+
+ /* values per dimension (and number of non-NULL values) */
+ Datum **values = (Datum**)palloc0(sizeof(Datum*) * ndims);
+ int *counts = (int*)palloc0(sizeof(int) * ndims);
+
+ /* info about dimensions (for deserialize) */
+ DimensionInfo * info
+ = (DimensionInfo *)palloc0(sizeof(DimensionInfo)*ndims);
+
+ /* sort support data */
+ SortSupport ssup = (SortSupport)palloc0(sizeof(SortSupportData)*ndims);
+
+ /* collect and deduplicate values for each dimension */
+ for (i = 0; i < ndims; i++)
+ {
+ int count;
+ StdAnalyzeData *tmp = (StdAnalyzeData *)stats[i]->extra_data;
+
+ /* keep important info about the data type */
+ info[i].typlen = stats[i]->attrtype->typlen;
+ info[i].typbyval = stats[i]->attrtype->typbyval;
+
+ /* allocate space for all values, including NULLs (won't use them) */
+ values[i] = (Datum*)palloc0(sizeof(Datum) * mcvlist->nitems);
+
+ for (j = 0; j < mcvlist->nitems; j++)
+ {
+ if (! mcvlist->items[j]->isnull[i]) /* skip NULL values */
+ {
+ values[i][counts[i]] = mcvlist->items[j]->values[i];
+ counts[i] += 1;
+ }
+ }
+
+ /* there are just NULL values in this dimension */
+ if (counts[i] == 0)
+ continue;
+
+ /* sort and deduplicate */
+ ssup[i].ssup_cxt = CurrentMemoryContext;
+ ssup[i].ssup_collation = DEFAULT_COLLATION_OID;
+ ssup[i].ssup_nulls_first = false;
+
+ PrepareSortSupportFromOrderingOp(tmp->ltopr, &ssup[i]);
+
+ qsort_arg(values[i], counts[i], sizeof(Datum),
+ compare_scalars_simple, &ssup[i]);
+
+ /*
+ * Walk through the array and eliminate duplicitate values, but
+ * keep the ordering (so that we can do bsearch later). We know
+ * there's at least 1 item, so we can skip the first element.
+ */
+ count = 1; /* number of deduplicated items */
+ for (j = 1; j < counts[i]; j++)
+ {
+ /* if it's different from the previous value, we need to keep it */
+ if (compare_datums_simple(values[i][j-1], values[i][j], &ssup[i]) != 0)
+ {
+ /* XXX: not needed if (count == j) */
+ values[i][count] = values[i][j];
+ count += 1;
+ }
+ }
+
+ /* do not exceed UINT16_MAX */
+ Assert(count <= UINT16_MAX);
+
+ /* keep info about the deduplicated count */
+ info[i].nvalues = count;
+
+ /* compute size of the serialized data */
+ if (info[i].typbyval || (info[i].typlen > 0))
+ /* by value pased by reference, but fixed length */
+ info[i].nbytes = info[i].nvalues * info[i].typlen;
+ else if (info[i].typlen == -1)
+ /* varlena, so just use VARSIZE_ANY */
+ for (j = 0; j < info[i].nvalues; j++)
+ info[i].nbytes += VARSIZE_ANY(values[i][j]);
+ else if (info[i].typlen == -2)
+ /* cstring, so simply strlen */
+ for (j = 0; j < info[i].nvalues; j++)
+ info[i].nbytes += strlen(DatumGetPointer(values[i][j]));
+ else
+ elog(ERROR, "unknown data type typbyval=%d typlen=%d",
+ info[i].typbyval, info[i].typlen);
+ }
+
+ /*
+ * Now we finally know how much space we'll need for the serialized
+ * MCV list, as it contains these fields:
+ *
+ * - length (4B) for varlena
+ * - magic (4B)
+ * - type (4B)
+ * - ndimensions (4B)
+ * - nitems (4B)
+ * - info (ndim * sizeof(DimensionInfo)
+ * - arrays of values for each dimension
+ * - serialized items (nitems * itemsize)
+ *
+ * So the 'header' size is 20B + ndim * sizeof(DimensionInfo) and
+ * then we'll place the data.
+ */
+ total_length = (sizeof(int32) + offsetof(MCVListData, items)
+ + ndims * sizeof(DimensionInfo)
+ + mcvlist->nitems * itemsize);
+
+ for (i = 0; i < ndims; i++)
+ total_length += info[i].nbytes;
+
+ /* enforce arbitrary limit of 1MB */
+ if (total_length > 1024 * 1024)
+ elog(ERROR, "serialized MCV exceeds 1MB (%ld)", total_length);
+
+ /* allocate space for the serialized MCV list, set header fields */
+ output = (bytea*)palloc0(total_length);
+ SET_VARSIZE(output, total_length);
+
+ /* we'll use 'ptr' to keep track of the place to write data */
+ data = VARDATA(output);
+
+ memcpy(data, mcvlist, offsetof(MCVListData, items));
+ data += offsetof(MCVListData, items);
+
+ memcpy(data, info, sizeof(DimensionInfo) * ndims);
+ data += sizeof(DimensionInfo) * ndims;
+
+ /* value array for each dimension */
+ for (i = 0; i < ndims; i++)
+ {
+#ifdef USE_ASSERT_CHECKING
+ char *tmp = data;
+#endif
+ for (j = 0; j < info[i].nvalues; j++)
+ {
+ if (info[i].typbyval)
+ {
+ /* passed by value / Datum */
+ memcpy(data, &values[i][j], info[i].typlen);
+ data += info[i].typlen;
+ }
+ else if (info[i].typlen > 0)
+ {
+ /* pased by reference, but fixed length (name, tid, ...) */
+ memcpy(data, &values[i][j], info[i].typlen);
+ data += info[i].typlen;
+ }
+ else if (info[i].typlen == -1)
+ {
+ /* varlena */
+ memcpy(data, DatumGetPointer(values[i][j]),
+ VARSIZE_ANY(values[i][j]));
+ data += VARSIZE_ANY(values[i][j]);
+ }
+ else if (info[i].typlen == -2)
+ {
+ /* cstring (don't forget the \0 terminator!) */
+ memcpy(data, DatumGetPointer(values[i][j]),
+ strlen(DatumGetPointer(values[i][j])) + 1);
+ data += strlen(DatumGetPointer(values[i][j])) + 1;
+ }
+ }
+ Assert((data - tmp) == info[i].nbytes);
+ }
+
+ /* and finally, the MCV items */
+ for (i = 0; i < mcvlist->nitems; i++)
+ {
+ /* don't write beyond the allocated space */
+ Assert(data <= (char*)output + total_length - itemsize);
+
+ /* reset the values for each item */
+ memset(item, 0, itemsize);
+
+ for (j = 0; j < ndims; j++)
+ {
+ /* do the lookup only for non-NULL values */
+ if (! mcvlist->items[i]->isnull[j])
+ {
+ Datum * v = NULL;
+ ssup_private = &ssup[j];
+
+ v = (Datum*)bsearch(&mcvlist->items[i]->values[j],
+ values[j], info[j].nvalues, sizeof(Datum),
+ bsearch_comparator);
+
+ if (v == NULL)
+ elog(ERROR, "value for dim %d not found in array", j);
+
+ /* compute index within the array */
+ ITEM_INDEXES(item)[j] = (v - values[j]);
+
+ /* check the index is within expected bounds */
+ Assert(ITEM_INDEXES(item)[j] >= 0);
+ Assert(ITEM_INDEXES(item)[j] < info[j].nvalues);
+ }
+ }
+
+ /* copy NULL and frequency flags into the item */
+ memcpy(ITEM_NULLS(item, ndims),
+ mcvlist->items[i]->isnull, sizeof(bool) * ndims);
+ memcpy(ITEM_FREQUENCY(item, ndims),
+ &mcvlist->items[i]->frequency, sizeof(double));
+
+ /* copy the item into the array */
+ memcpy(data, item, itemsize);
+
+ data += itemsize;
+ }
+
+ /* at this point we expect to match the total_length exactly */
+ Assert((data - (char*)output) == total_length);
+
+ return output;
+}
+
+/*
+ * Inverse to serialize_mv_mcvlist() - see the comment there.
+ *
+ * We'll do full deserialization, because we don't really expect high
+ * duplication of values so the caching may not be as efficient as with
+ * histograms.
+ */
+MCVList deserialize_mv_mcvlist(bytea * data)
+{
+ int i, j;
+ Size expected_size;
+ MCVList mcvlist;
+ char *tmp;
+
+ int ndims, nitems, itemsize;
+ DimensionInfo *info = NULL;
+
+ uint16 *indexes = NULL;
+ Datum **values = NULL;
+
+ /* local allocation buffer (used only for deserialization) */
+ int bufflen;
+ char *buff;
+ char *ptr;
+
+ /* buffer used for the result */
+ int rbufflen;
+ char *rbuff;
+ char *rptr;
+
+ if (data == NULL)
+ return NULL;
+
+ if (VARSIZE_ANY_EXHDR(data) < offsetof(MCVListData,items))
+ elog(ERROR, "invalid MCV Size %ld (expected at least %ld)",
+ VARSIZE_ANY_EXHDR(data), offsetof(MCVListData,items));
+
+ /* read the MCV list header */
+ mcvlist = (MCVList)palloc0(sizeof(MCVListData));
+
+ /* initialize pointer to the data part (skip the varlena header) */
+ tmp = VARDATA(data);
+
+ /* get the header and perform basic sanity checks */
+ memcpy(mcvlist, tmp, offsetof(MCVListData,items));
+ tmp += offsetof(MCVListData,items);
+
+ if (mcvlist->magic != MVSTAT_MCV_MAGIC)
+ elog(ERROR, "invalid MCV magic %d (expected %dd)",
+ mcvlist->magic, MVSTAT_MCV_MAGIC);
+
+ if (mcvlist->type != MVSTAT_MCV_TYPE_BASIC)
+ elog(ERROR, "invalid MCV type %d (expected %dd)",
+ mcvlist->type, MVSTAT_MCV_TYPE_BASIC);
+
+ nitems = mcvlist->nitems;
+ ndims = mcvlist->ndimensions;
+ itemsize = ITEM_SIZE(ndims);
+
+ Assert(nitems > 0);
+ Assert((ndims >= 2) && (ndims <= MVSTATS_MAX_DIMENSIONS));
+
+ /*
+ * What size do we expect with those parameters (it's incomplete,
+ * as we yet have to count the array sizes (from DimensionInfo
+ * records).
+ */
+ expected_size = offsetof(MCVListData,items) +
+ ndims * sizeof(DimensionInfo) +
+ (nitems * itemsize);
+
+ /* check that we have at least the DimensionInfo records */
+ if (VARSIZE_ANY_EXHDR(data) < expected_size)
+ elog(ERROR, "invalid MCV Size %ld (expected %ld)",
+ VARSIZE_ANY_EXHDR(data), expected_size);
+
+ info = (DimensionInfo*)(tmp);
+ tmp += ndims * sizeof(DimensionInfo);
+
+ /* account for the value arrays */
+ for (i = 0; i < ndims; i++)
+ expected_size += info[i].nbytes;
+
+ if (VARSIZE_ANY_EXHDR(data) != expected_size)
+ elog(ERROR, "invalid MCV Size %ld (expected %ld)",
+ VARSIZE_ANY_EXHDR(data), expected_size);
+
+ /* looks OK - not corrupted or something */
+
+ /*
+ * We'll allocate one large chunk of memory for the intermediate
+ * data, needed only for deserializing the MCV list, and we'll pack
+ * use a local dense allocation to minimize the palloc overhead.
+ *
+ * Let's see how much space we'll actually need, and also include
+ * space for the array with pointers.
+ */
+ bufflen = sizeof(Datum*) * ndims; /* space for pointers */
+
+ for (i = 0; i < ndims; i++)
+ /* for full-size byval types, we reuse the serialized value */
+ if (! (info[i].typbyval && info[i].typlen == sizeof(Datum)))
+ bufflen += (sizeof(Datum) * info[i].nvalues);
+
+ buff = palloc0(bufflen);
+ ptr = buff;
+
+ values = (Datum**)buff;
+ ptr += (sizeof(Datum*) * ndims);
+
+ /*
+ * FIXME This uses pointers to the original data array (the types
+ * not passed by value), so when someone frees the memory,
+ * e.g. by doing something like this:
+ *
+ * bytea * data = ... fetch the data from catalog ...
+ * MCVList mcvlist = deserialize_mcv_list(data);
+ * pfree(data);
+ *
+ * then 'mcvlist' references the freed memory. This needs to
+ * copy the pieces.
+ */
+ for (i = 0; i < ndims; i++)
+ {
+ if (info[i].typbyval)
+ {
+ /* passed by value / Datum - simply reuse the array */
+ if (info[i].typlen == sizeof(Datum))
+ {
+ values[i] = (Datum*)tmp;
+ tmp += info[i].nbytes;
+ }
+ else
+ {
+ values[i] = (Datum*)ptr;
+ ptr += (sizeof(Datum) * info[i].nvalues);
+
+ for (j = 0; j < info[i].nvalues; j++)
+ {
+ /* just point into the array */
+ memcpy(&values[i][j], tmp, info[i].typlen);
+ tmp += info[i].typlen;
+ }
+ }
+ }
+ else
+ {
+ /* all the varlena data need a chunk from the buffer */
+ values[i] = (Datum*)ptr;
+ ptr += (sizeof(Datum) * info[i].nvalues);
+
+ /* pased by reference, but fixed length (name, tid, ...) */
+ if (info[i].typlen > 0)
+ {
+ for (j = 0; j < info[i].nvalues; j++)
+ {
+ /* just point into the array */
+ values[i][j] = PointerGetDatum(tmp);
+ tmp += info[i].typlen;
+ }
+ }
+ else if (info[i].typlen == -1)
+ {
+ /* varlena */
+ for (j = 0; j < info[i].nvalues; j++)
+ {
+ /* just point into the array */
+ values[i][j] = PointerGetDatum(tmp);
+ tmp += VARSIZE_ANY(tmp);
+ }
+ }
+ else if (info[i].typlen == -2)
+ {
+ /* cstring */
+ for (j = 0; j < info[i].nvalues; j++)
+ {
+ /* just point into the array */
+ values[i][j] = PointerGetDatum(tmp);
+ tmp += (strlen(tmp) + 1); /* don't forget the \0 */
+ }
+ }
+ }
+ }
+
+ /* we should exhaust the buffer exactly */
+ Assert((ptr - buff) == bufflen);
+
+ /* allocate space for the MCV items in a single piece */
+ rbufflen = (sizeof(MCVItem) + sizeof(MCVItemData) +
+ sizeof(Datum)*ndims + sizeof(bool)*ndims) * nitems;
+
+ rbuff = palloc(rbufflen);
+ rptr = rbuff;
+
+ mcvlist->items = (MCVItem*)rbuff;
+ rptr += (sizeof(MCVItem) * nitems);
+
+ for (i = 0; i < nitems; i++)
+ {
+ MCVItem item = (MCVItem)rptr;
+ rptr += (sizeof(MCVItemData));
+
+ item->values = (Datum*)rptr;
+ rptr += (sizeof(Datum)*ndims);
+
+ item->isnull = (bool*)rptr;
+ rptr += (sizeof(bool) *ndims);
+
+ /* just point to the right place */
+ indexes = ITEM_INDEXES(tmp);
+
+ memcpy(item->isnull, ITEM_NULLS(tmp, ndims), sizeof(bool) * ndims);
+ memcpy(&item->frequency, ITEM_FREQUENCY(tmp, ndims), sizeof(double));
+
+#ifdef ASSERT_CHECKING
+ for (j = 0; j < ndims; j++)
+ Assert(indexes[j] <= UINT16_MAX);
+#endif
+
+ /* translate the values */
+ for (j = 0; j < ndims; j++)
+ if (! item->isnull[j])
+ item->values[j] = values[j][indexes[j]];
+
+ mcvlist->items[i] = item;
+
+ tmp += ITEM_SIZE(ndims);
+
+ Assert(tmp <= (char*)data + VARSIZE_ANY(data));
+ }
+
+ /* check that we processed all the data */
+ Assert(tmp == (char*)data + VARSIZE_ANY(data));
+
+ /* release the temporary buffer */
+ pfree(buff);
+
+ return mcvlist;
+}
+
+/*
+ * We need to pass the SortSupport to the comparator, but bsearch()
+ * has no 'context' parameter, so we use a global variable (ugly).
+ */
+static int
+bsearch_comparator(const void * a, const void * b)
+{
+ Assert(ssup_private != NULL);
+ return compare_scalars_simple(a, b, (void*)ssup_private);
+}
+/*
+ * SRF with details about buckets of a histogram:
+ *
+ * - item ID (0...nitems)
+ * - values (string array)
+ * - nulls only (boolean array)
+ * - frequency (double precision)
+ *
+ * The input is the OID of the statistics, and there are no rows
+ * returned if the statistics contains no histogram.
+ */
+PG_FUNCTION_INFO_V1(pg_mv_mcv_items);
+
+Datum
+pg_mv_mcv_items(PG_FUNCTION_ARGS)
+{
+ FuncCallContext *funcctx;
+ int call_cntr;
+ int max_calls;
+ TupleDesc tupdesc;
+ AttInMetadata *attinmeta;
+
+ /* stuff done only on the first call of the function */
+ if (SRF_IS_FIRSTCALL())
+ {
+ MemoryContext oldcontext;
+ MCVList mcvlist;
+
+ /* create a function context for cross-call persistence */
+ funcctx = SRF_FIRSTCALL_INIT();
+
+ /* switch to memory context appropriate for multiple function calls */
+ oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
+
+ mcvlist = load_mv_mcvlist(PG_GETARG_OID(0));
+
+ funcctx->user_fctx = mcvlist;
+
+ /* total number of tuples to be returned */
+ funcctx->max_calls = 0;
+ if (funcctx->user_fctx != NULL)
+ funcctx->max_calls = mcvlist->nitems;
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("function returning record called in context "
+ "that cannot accept type record")));
+
+ /*
+ * generate attribute metadata needed later to produce tuples
+ * from raw C strings
+ */
+ attinmeta = TupleDescGetAttInMetadata(tupdesc);
+ funcctx->attinmeta = attinmeta;
+
+ MemoryContextSwitchTo(oldcontext);
+ }
+
+ /* stuff done on every call of the function */
+ funcctx = SRF_PERCALL_SETUP();
+
+ call_cntr = funcctx->call_cntr;
+ max_calls = funcctx->max_calls;
+ attinmeta = funcctx->attinmeta;
+
+ if (call_cntr < max_calls) /* do when there is more left to send */
+ {
+ char **values;
+ HeapTuple tuple;
+ Datum result;
+ int2vector *stakeys;
+ Oid relid;
+
+ char *buff = palloc0(1024);
+ char *format;
+
+ int i;
+
+ Oid *outfuncs;
+ FmgrInfo *fmgrinfo;
+
+ MCVList mcvlist;
+ MCVItem item;
+
+ mcvlist = (MCVList)funcctx->user_fctx;
+
+ Assert(call_cntr < mcvlist->nitems);
+
+ item = mcvlist->items[call_cntr];
+
+ stakeys = find_mv_attnums(PG_GETARG_OID(0), &relid);
+
+ /*
+ * Prepare a values array for building the returned tuple.
+ * This should be an array of C strings which will
+ * be processed later by the type input functions.
+ */
+ values = (char **) palloc(4 * sizeof(char *));
+
+ values[0] = (char *) palloc(64 * sizeof(char));
+
+ /* arrays */
+ values[1] = (char *) palloc0(1024 * sizeof(char));
+ values[2] = (char *) palloc0(1024 * sizeof(char));
+
+ /* frequency */
+ values[3] = (char *) palloc(64 * sizeof(char));
+
+ outfuncs = (Oid*)palloc0(sizeof(Oid) * mcvlist->ndimensions);
+ fmgrinfo = (FmgrInfo*)palloc0(sizeof(FmgrInfo) * mcvlist->ndimensions);
+
+ for (i = 0; i < mcvlist->ndimensions; i++)
+ {
+ bool isvarlena;
+
+ getTypeOutputInfo(get_atttype(relid, stakeys->values[i]),
+ &outfuncs[i], &isvarlena);
+
+ fmgr_info(outfuncs[i], &fmgrinfo[i]);
+ }
+
+ snprintf(values[0], 64, "%d", call_cntr); /* item ID */
+
+ for (i = 0; i < mcvlist->ndimensions; i++)
+ {
+ Datum val, valout;
+
+ format = "%s, %s";
+ if (i == 0)
+ format = "{%s%s";
+ else if (i == mcvlist->ndimensions-1)
+ format = "%s, %s}";
+
+ val = item->values[i];
+ valout = FunctionCall1(&fmgrinfo[i], val);
+
+ snprintf(buff, 1024, format, values[1], DatumGetPointer(valout));
+ strncpy(values[1], buff, 1023);
+ buff[0] = '\0';
+
+ snprintf(buff, 1024, format, values[2], item->isnull[i] ? "t" : "f");
+ strncpy(values[2], buff, 1023);
+ buff[0] = '\0';
+ }
+
+ snprintf(values[3], 64, "%f", item->frequency); /* frequency */
+
+ /* build a tuple */
+ tuple = BuildTupleFromCStrings(attinmeta, values);
+
+ /* make the tuple into a datum */
+ result = HeapTupleGetDatum(tuple);
+
+ /* clean up (this is not really necessary) */
+ pfree(values[0]);
+ pfree(values[1]);
+ pfree(values[2]);
+ pfree(values[3]);
+
+ pfree(values);
+
+ SRF_RETURN_NEXT(funcctx, result);
+ }
+ else /* do when there is no more left */
+ {
+ SRF_RETURN_DONE(funcctx);
+ }
+}
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 8ce9c0e..2c22d31 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -2109,8 +2109,9 @@ describeOneTableDetails(const char *schemaname,
{
printfPQExpBuffer(&buf,
"SELECT oid, stanamespace::regnamespace AS nsp, staname, stakeys,\n"
- " deps_enabled,\n"
- " deps_built,\n"
+ " deps_enabled, mcv_enabled,\n"
+ " deps_built, mcv_built,\n"
+ " mcv_max_items,\n"
" (SELECT string_agg(attname::text,', ')\n"
" FROM ((SELECT unnest(stakeys) AS attnum) s\n"
" JOIN pg_attribute a ON (starelid = a.attrelid and a.attnum = s.attnum))) AS attnums\n"
@@ -2128,6 +2129,8 @@ describeOneTableDetails(const char *schemaname,
printTableAddFooter(&cont, _("Statistics:"));
for (i = 0; i < tuples; i++)
{
+ bool first = true;
+
printfPQExpBuffer(&buf, " ");
/* statistics name (qualified with namespace) */
@@ -2137,10 +2140,22 @@ describeOneTableDetails(const char *schemaname,
/* options */
if (!strcmp(PQgetvalue(result, i, 4), "t"))
- appendPQExpBuffer(&buf, "(dependencies)");
+ {
+ appendPQExpBuffer(&buf, "(dependencies");
+ first = false;
+ }
+
+ if (!strcmp(PQgetvalue(result, i, 5), "t"))
+ {
+ if (! first)
+ appendPQExpBuffer(&buf, ", mcv");
+ else
+ appendPQExpBuffer(&buf, "(mcv");
+ first = false;
+ }
- appendPQExpBuffer(&buf, " ON (%s)",
- PQgetvalue(result, i, 6));
+ appendPQExpBuffer(&buf, ") ON (%s)",
+ PQgetvalue(result, i, 9));
printTableAddFooter(&cont, buf.data);
}
diff --git a/src/include/catalog/pg_mv_statistic.h b/src/include/catalog/pg_mv_statistic.h
index c74af47..3529b03 100644
--- a/src/include/catalog/pg_mv_statistic.h
+++ b/src/include/catalog/pg_mv_statistic.h
@@ -38,15 +38,21 @@ CATALOG(pg_mv_statistic,3381)
/* statistics requested to build */
bool deps_enabled; /* analyze dependencies? */
+ bool mcv_enabled; /* build MCV list? */
+
+ /* MCV size */
+ int32 mcv_max_items; /* max MCV items */
/* statistics that are available (if requested) */
bool deps_built; /* dependencies were built */
+ bool mcv_built; /* MCV list was built */
/* variable-length fields start here, but we allow direct access to stakeys */
int2vector stakeys; /* array of column keys */
#ifdef CATALOG_VARLEN
bytea stadeps; /* dependencies (serialized) */
+ bytea stamcv; /* MCV list (serialized) */
#endif
} FormData_pg_mv_statistic;
@@ -62,14 +68,18 @@ typedef FormData_pg_mv_statistic *Form_pg_mv_statistic;
* compiler constants for pg_mv_statistic
* ----------------
*/
-#define Natts_pg_mv_statistic 8
+#define Natts_pg_mv_statistic 12
#define Anum_pg_mv_statistic_starelid 1
#define Anum_pg_mv_statistic_staname 2
#define Anum_pg_mv_statistic_stanamespace 3
#define Anum_pg_mv_statistic_staowner 4
#define Anum_pg_mv_statistic_deps_enabled 5
-#define Anum_pg_mv_statistic_deps_built 6
-#define Anum_pg_mv_statistic_stakeys 7
-#define Anum_pg_mv_statistic_stadeps 8
+#define Anum_pg_mv_statistic_mcv_enabled 6
+#define Anum_pg_mv_statistic_mcv_max_items 7
+#define Anum_pg_mv_statistic_deps_built 8
+#define Anum_pg_mv_statistic_mcv_built 9
+#define Anum_pg_mv_statistic_stakeys 10
+#define Anum_pg_mv_statistic_stadeps 11
+#define Anum_pg_mv_statistic_stamcv 12
#endif /* PG_MV_STATISTIC_H */
diff --git a/src/include/catalog/pg_proc.h b/src/include/catalog/pg_proc.h
index eecce40..b16eebc 100644
--- a/src/include/catalog/pg_proc.h
+++ b/src/include/catalog/pg_proc.h
@@ -2670,6 +2670,10 @@ DATA(insert OID = 3998 ( pg_mv_stats_dependencies_info PGNSP PGUID 12 1 0 0
DESCR("multivariate stats: functional dependencies info");
DATA(insert OID = 3999 ( pg_mv_stats_dependencies_show PGNSP PGUID 12 1 0 0 0 f f f f t f i s 1 0 25 "17" _null_ _null_ _null_ _null_ _null_ pg_mv_stats_dependencies_show _null_ _null_ _null_ ));
DESCR("multivariate stats: functional dependencies show");
+DATA(insert OID = 3376 ( pg_mv_stats_mcvlist_info PGNSP PGUID 12 1 0 0 0 f f f f t f i s 1 0 25 "17" _null_ _null_ _null_ _null_ _null_ pg_mv_stats_mcvlist_info _null_ _null_ _null_ ));
+DESCR("multi-variate statistics: MCV list info");
+DATA(insert OID = 3373 ( pg_mv_mcv_items PGNSP PGUID 12 1 1000 0 0 f f f f t t i s 1 0 2249 "26" "{26,23,1009,1000,701}" "{i,o,o,o,o}" "{oid,index,values,nulls,frequency}" _null_ _null_ pg_mv_mcv_items _null_ _null_ _null_ ));
+DESCR("details about MCV list items");
DATA(insert OID = 1928 ( pg_stat_get_numscans PGNSP PGUID 12 1 0 0 0 f f f f t f s r 1 0 20 "26" _null_ _null_ _null_ _null_ _null_ pg_stat_get_numscans _null_ _null_ _null_ ));
DESCR("statistics: number of scans done for table/index");
diff --git a/src/include/nodes/relation.h b/src/include/nodes/relation.h
index e10dcf1..2bcd582 100644
--- a/src/include/nodes/relation.h
+++ b/src/include/nodes/relation.h
@@ -653,9 +653,11 @@ typedef struct MVStatisticInfo
/* enabled statistics */
bool deps_enabled; /* functional dependencies enabled */
+ bool mcv_enabled; /* MCV list enabled */
/* built/available statistics */
bool deps_built; /* functional dependencies built */
+ bool mcv_built; /* MCV list built */
/* columns in the statistics (attnums) */
int2vector *stakeys; /* attnums of the columns covered */
diff --git a/src/include/utils/mvstats.h b/src/include/utils/mvstats.h
index cc43a79..4535db7 100644
--- a/src/include/utils/mvstats.h
+++ b/src/include/utils/mvstats.h
@@ -51,30 +51,89 @@ typedef MVDependenciesData* MVDependencies;
#define MVSTAT_DEPS_TYPE_BASIC 1 /* basic dependencies type */
/*
+ * Multivariate MCV (most-common value) lists
+ *
+ * A straight-forward extension of MCV items - i.e. a list (array) of
+ * combinations of attribute values, together with a frequency and
+ * null flags.
+ */
+typedef struct MCVItemData {
+ double frequency; /* frequency of this combination */
+ bool *isnull; /* lags of NULL values (up to 32 columns) */
+ Datum *values; /* variable-length (ndimensions) */
+} MCVItemData;
+
+typedef MCVItemData *MCVItem;
+
+/* multivariate MCV list - essentally an array of MCV items */
+typedef struct MCVListData {
+ uint32 magic; /* magic constant marker */
+ uint32 type; /* type of MCV list (BASIC) */
+ uint32 ndimensions; /* number of dimensions */
+ uint32 nitems; /* number of MCV items in the array */
+ MCVItem *items; /* array of MCV items */
+} MCVListData;
+
+typedef MCVListData *MCVList;
+
+/* used to flag stats serialized to bytea */
+#define MVSTAT_MCV_MAGIC 0xE1A651C2 /* marks serialized bytea */
+#define MVSTAT_MCV_TYPE_BASIC 1 /* basic MCV list type */
+
+/*
+ * Limits used for mcv_max_items option, i.e. we're always guaranteed
+ * to have space for at least MVSTAT_MCVLIST_MIN_ITEMS, and we cannot
+ * have more than MVSTAT_MCVLIST_MAX_ITEMS items.
+ *
+ * This is just a boundary for the 'max' threshold - the actual list
+ * may of course contain less items than MVSTAT_MCVLIST_MIN_ITEMS.
+ */
+#define MVSTAT_MCVLIST_MIN_ITEMS 128 /* min items in MCV list */
+#define MVSTAT_MCVLIST_MAX_ITEMS 8192 /* max items in MCV list */
+
+/*
* TODO Maybe fetching the histogram/MCV list separately is inefficient?
* Consider adding a single `fetch_stats` method, fetching all
* stats specified using flags (or something like that).
*/
MVDependencies load_mv_dependencies(Oid mvoid);
+MCVList load_mv_mcvlist(Oid mvoid);
bytea * serialize_mv_dependencies(MVDependencies dependencies);
+bytea * serialize_mv_mcvlist(MCVList mcvlist, int2vector *attrs,
+ VacAttrStats **stats);
/* deserialization of stats (serialization is private to analyze) */
MVDependencies deserialize_mv_dependencies(bytea * data);
+MCVList deserialize_mv_mcvlist(bytea * data);
+
+/*
+ * Returns index of the attribute number within the vector (i.e. a
+ * dimension within the stats).
+ */
+int mv_get_index(AttrNumber varattno, int2vector * stakeys);
+
+int2vector* find_mv_attnums(Oid mvoid, Oid *relid);
/* FIXME this probably belongs somewhere else (not to operations stats) */
extern Datum pg_mv_stats_dependencies_info(PG_FUNCTION_ARGS);
extern Datum pg_mv_stats_dependencies_show(PG_FUNCTION_ARGS);
+extern Datum pg_mv_stats_mcvlist_info(PG_FUNCTION_ARGS);
+extern Datum pg_mv_mcvlist_items(PG_FUNCTION_ARGS);
MVDependencies
-build_mv_dependencies(int numrows, HeapTuple *rows,
- int2vector *attrs,
- VacAttrStats **stats);
+build_mv_dependencies(int numrows, HeapTuple *rows, int2vector *attrs,
+ VacAttrStats **stats);
+
+MCVList
+build_mv_mcvlist(int numrows, HeapTuple *rows, int2vector *attrs,
+ VacAttrStats **stats, int *numrows_filtered);
void build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
- int natts, VacAttrStats **vacattrstats);
+ int natts, VacAttrStats **vacattrstats);
-void update_mv_stats(Oid relid, MVDependencies dependencies, int2vector *attrs);
+void update_mv_stats(Oid relid, MVDependencies dependencies, MCVList mcvlist,
+ int2vector *attrs, VacAttrStats **stats);
#endif
diff --git a/src/test/regress/expected/mv_mcv.out b/src/test/regress/expected/mv_mcv.out
new file mode 100644
index 0000000..075320b
--- /dev/null
+++ b/src/test/regress/expected/mv_mcv.out
@@ -0,0 +1,207 @@
+-- data type passed by value
+CREATE TABLE mcv_list (
+ a INT,
+ b INT,
+ c INT
+);
+-- unknown column
+CREATE STATISTICS s4 ON mcv_list (unknown_column) WITH (mcv);
+ERROR: column "unknown_column" referenced in statistics does not exist
+-- single column
+CREATE STATISTICS s4 ON mcv_list (a) WITH (mcv);
+ERROR: multivariate stats require 2 or more columns
+-- single column, duplicated
+CREATE STATISTICS s4 ON mcv_list (a, a) WITH (mcv);
+ERROR: duplicate column name in statistics definition
+-- two columns, one duplicated
+CREATE STATISTICS s4 ON mcv_list (a, a, b) WITH (mcv);
+ERROR: duplicate column name in statistics definition
+-- unknown option
+CREATE STATISTICS s4 ON mcv_list (a, b, c) WITH (unknown_option);
+ERROR: unrecognized STATISTICS option "unknown_option"
+-- missing MCV statistics
+CREATE STATISTICS s4 ON mcv_list (a, b, c) WITH (dependencies, max_mcv_items=200);
+ERROR: option 'mcv' is required by other options(s)
+-- invalid mcv_max_items value / too low
+CREATE STATISTICS s4 ON mcv_list (a, b, c) WITH (mcv, max_mcv_items=10);
+ERROR: max number of MCV items must be at least 128
+-- invalid mcv_max_items value / too high
+CREATE STATISTICS s4 ON mcv_list (a, b, c) WITH (mcv, max_mcv_items=10000);
+ERROR: max number of MCV items is 8192
+-- correct command
+CREATE STATISTICS s4 ON mcv_list (a, b, c) WITH (mcv);
+-- random data
+INSERT INTO mcv_list
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+ANALYZE mcv_list;
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+ mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info
+-------------+-----------+--------------------------
+ t | f |
+(1 row)
+
+TRUNCATE mcv_list;
+-- a => b, a => c, b => c
+INSERT INTO mcv_list
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE mcv_list;
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+ mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info
+-------------+-----------+--------------------------
+ t | t | nitems=1000
+(1 row)
+
+TRUNCATE mcv_list;
+-- a => b, a => c
+INSERT INTO mcv_list
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE mcv_list;
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+ mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info
+-------------+-----------+--------------------------
+ t | t | nitems=1000
+(1 row)
+
+TRUNCATE mcv_list;
+-- check explain (expect bitmap index scan, not plain index scan)
+INSERT INTO mcv_list
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX mcv_idx ON mcv_list (a, b);
+ANALYZE mcv_list;
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+ mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info
+-------------+-----------+--------------------------
+ t | t | nitems=100
+(1 row)
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mcv_list WHERE a = 10 AND b = 5;
+ QUERY PLAN
+--------------------------------------------
+ Bitmap Heap Scan on mcv_list
+ Recheck Cond: ((a = 10) AND (b = 5))
+ -> Bitmap Index Scan on mcv_idx
+ Index Cond: ((a = 10) AND (b = 5))
+(4 rows)
+
+DROP TABLE mcv_list;
+-- varlena type (text)
+CREATE TABLE mcv_list (
+ a TEXT,
+ b TEXT,
+ c TEXT
+);
+CREATE STATISTICS s5 ON mcv_list (a, b, c) WITH (mcv);
+-- random data
+INSERT INTO mcv_list
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+ANALYZE mcv_list;
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+ mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info
+-------------+-----------+--------------------------
+ t | f |
+(1 row)
+
+TRUNCATE mcv_list;
+-- a => b, a => c, b => c
+INSERT INTO mcv_list
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE mcv_list;
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+ mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info
+-------------+-----------+--------------------------
+ t | t | nitems=1000
+(1 row)
+
+TRUNCATE mcv_list;
+-- a => b, a => c
+INSERT INTO mcv_list
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE mcv_list;
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+ mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info
+-------------+-----------+--------------------------
+ t | t | nitems=1000
+(1 row)
+
+TRUNCATE mcv_list;
+-- check explain (expect bitmap index scan, not plain index scan)
+INSERT INTO mcv_list
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX mcv_idx ON mcv_list (a, b);
+ANALYZE mcv_list;
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+ mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info
+-------------+-----------+--------------------------
+ t | t | nitems=100
+(1 row)
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mcv_list WHERE a = '10' AND b = '5';
+ QUERY PLAN
+------------------------------------------------------------
+ Bitmap Heap Scan on mcv_list
+ Recheck Cond: ((a = '10'::text) AND (b = '5'::text))
+ -> Bitmap Index Scan on mcv_idx
+ Index Cond: ((a = '10'::text) AND (b = '5'::text))
+(4 rows)
+
+TRUNCATE mcv_list;
+-- check explain (expect bitmap index scan, not plain index scan) with NULLs
+INSERT INTO mcv_list
+ SELECT
+ (CASE WHEN i/10000 = 0 THEN NULL ELSE i/10000 END),
+ (CASE WHEN i/20000 = 0 THEN NULL ELSE i/20000 END),
+ (CASE WHEN i/40000 = 0 THEN NULL ELSE i/40000 END)
+ FROM generate_series(1,1000000) s(i);
+ANALYZE mcv_list;
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+ mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info
+-------------+-----------+--------------------------
+ t | t | nitems=100
+(1 row)
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mcv_list WHERE a IS NULL AND b IS NULL;
+ QUERY PLAN
+---------------------------------------------------
+ Bitmap Heap Scan on mcv_list
+ Recheck Cond: ((a IS NULL) AND (b IS NULL))
+ -> Bitmap Index Scan on mcv_idx
+ Index Cond: ((a IS NULL) AND (b IS NULL))
+(4 rows)
+
+DROP TABLE mcv_list;
+-- NULL values (mix of int and text columns)
+CREATE TABLE mcv_list (
+ a INT,
+ b TEXT,
+ c INT,
+ d TEXT
+);
+CREATE STATISTICS s6 ON mcv_list (a, b, c, d) WITH (mcv);
+INSERT INTO mcv_list
+ SELECT
+ mod(i, 100),
+ (CASE WHEN mod(i, 200) = 0 THEN NULL ELSE mod(i,200) END),
+ mod(i, 400),
+ (CASE WHEN mod(i, 300) = 0 THEN NULL ELSE mod(i,600) END)
+ FROM generate_series(1,10000) s(i);
+ANALYZE mcv_list;
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+ mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info
+-------------+-----------+--------------------------
+ t | t | nitems=1200
+(1 row)
+
+DROP TABLE mcv_list;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 84b4425..66071d8 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1373,7 +1373,9 @@ pg_mv_stats| SELECT n.nspname AS schemaname,
s.staname,
s.stakeys AS attnums,
length(s.stadeps) AS depsbytes,
- pg_mv_stats_dependencies_info(s.stadeps) AS depsinfo
+ pg_mv_stats_dependencies_info(s.stadeps) AS depsinfo,
+ length(s.stamcv) AS mcvbytes,
+ pg_mv_stats_mcvlist_info(s.stamcv) AS mcvinfo
FROM ((pg_mv_statistic s
JOIN pg_class c ON ((c.oid = s.starelid)))
LEFT JOIN pg_namespace n ON ((n.oid = c.relnamespace)));
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index 4f2ffb8..85d94f1 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -112,4 +112,4 @@ test: event_trigger
test: stats
# run tests of multivariate stats
-test: mv_dependencies
+test: mv_dependencies mv_mcv
diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule
index 097a04f..6584d73 100644
--- a/src/test/regress/serial_schedule
+++ b/src/test/regress/serial_schedule
@@ -163,3 +163,4 @@ test: xml
test: event_trigger
test: stats
test: mv_dependencies
+test: mv_mcv
diff --git a/src/test/regress/sql/mv_mcv.sql b/src/test/regress/sql/mv_mcv.sql
new file mode 100644
index 0000000..b31d32d
--- /dev/null
+++ b/src/test/regress/sql/mv_mcv.sql
@@ -0,0 +1,178 @@
+-- data type passed by value
+CREATE TABLE mcv_list (
+ a INT,
+ b INT,
+ c INT
+);
+
+-- unknown column
+CREATE STATISTICS s4 ON mcv_list (unknown_column) WITH (mcv);
+
+-- single column
+CREATE STATISTICS s4 ON mcv_list (a) WITH (mcv);
+
+-- single column, duplicated
+CREATE STATISTICS s4 ON mcv_list (a, a) WITH (mcv);
+
+-- two columns, one duplicated
+CREATE STATISTICS s4 ON mcv_list (a, a, b) WITH (mcv);
+
+-- unknown option
+CREATE STATISTICS s4 ON mcv_list (a, b, c) WITH (unknown_option);
+
+-- missing MCV statistics
+CREATE STATISTICS s4 ON mcv_list (a, b, c) WITH (dependencies, max_mcv_items=200);
+
+-- invalid mcv_max_items value / too low
+CREATE STATISTICS s4 ON mcv_list (a, b, c) WITH (mcv, max_mcv_items=10);
+
+-- invalid mcv_max_items value / too high
+CREATE STATISTICS s4 ON mcv_list (a, b, c) WITH (mcv, max_mcv_items=10000);
+
+-- correct command
+CREATE STATISTICS s4 ON mcv_list (a, b, c) WITH (mcv);
+
+-- random data
+INSERT INTO mcv_list
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+
+ANALYZE mcv_list;
+
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+
+TRUNCATE mcv_list;
+
+-- a => b, a => c, b => c
+INSERT INTO mcv_list
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+
+ANALYZE mcv_list;
+
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+
+TRUNCATE mcv_list;
+
+-- a => b, a => c
+INSERT INTO mcv_list
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+
+ANALYZE mcv_list;
+
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+
+TRUNCATE mcv_list;
+
+-- check explain (expect bitmap index scan, not plain index scan)
+INSERT INTO mcv_list
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX mcv_idx ON mcv_list (a, b);
+
+ANALYZE mcv_list;
+
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mcv_list WHERE a = 10 AND b = 5;
+
+DROP TABLE mcv_list;
+
+-- varlena type (text)
+CREATE TABLE mcv_list (
+ a TEXT,
+ b TEXT,
+ c TEXT
+);
+
+CREATE STATISTICS s5 ON mcv_list (a, b, c) WITH (mcv);
+
+-- random data
+INSERT INTO mcv_list
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+
+ANALYZE mcv_list;
+
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+
+TRUNCATE mcv_list;
+
+-- a => b, a => c, b => c
+INSERT INTO mcv_list
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+
+ANALYZE mcv_list;
+
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+
+TRUNCATE mcv_list;
+
+-- a => b, a => c
+INSERT INTO mcv_list
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE mcv_list;
+
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+
+TRUNCATE mcv_list;
+
+-- check explain (expect bitmap index scan, not plain index scan)
+INSERT INTO mcv_list
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX mcv_idx ON mcv_list (a, b);
+ANALYZE mcv_list;
+
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mcv_list WHERE a = '10' AND b = '5';
+
+TRUNCATE mcv_list;
+
+-- check explain (expect bitmap index scan, not plain index scan) with NULLs
+INSERT INTO mcv_list
+ SELECT
+ (CASE WHEN i/10000 = 0 THEN NULL ELSE i/10000 END),
+ (CASE WHEN i/20000 = 0 THEN NULL ELSE i/20000 END),
+ (CASE WHEN i/40000 = 0 THEN NULL ELSE i/40000 END)
+ FROM generate_series(1,1000000) s(i);
+ANALYZE mcv_list;
+
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mcv_list WHERE a IS NULL AND b IS NULL;
+
+DROP TABLE mcv_list;
+
+-- NULL values (mix of int and text columns)
+CREATE TABLE mcv_list (
+ a INT,
+ b TEXT,
+ c INT,
+ d TEXT
+);
+
+CREATE STATISTICS s6 ON mcv_list (a, b, c, d) WITH (mcv);
+
+INSERT INTO mcv_list
+ SELECT
+ mod(i, 100),
+ (CASE WHEN mod(i, 200) = 0 THEN NULL ELSE mod(i,200) END),
+ mod(i, 400),
+ (CASE WHEN mod(i, 300) = 0 THEN NULL ELSE mod(i,600) END)
+ FROM generate_series(1,10000) s(i);
+
+ANALYZE mcv_list;
+
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+
+DROP TABLE mcv_list;
--
2.1.0
[text/x-patch] 0005-multivariate-histograms.patch (136.6K, ../../1457535779.24545.71.camel@2ndquadrant.com/6-0005-multivariate-histograms.patch)
download | inline diff:
From fea437ee38376fda67d177276ce9812f2b0e9d81 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Sun, 11 Jan 2015 20:18:24 +0100
Subject: [PATCH 5/9] multivariate histograms
- extends the pg_mv_statistic catalog (add 'hist' fields)
- building the histograms during ANALYZE
- simple estimation while planning the queries
Includes regression tests mostly equal to those for functional
dependencies / MCV lists.
---
doc/src/sgml/ref/create_statistics.sgml | 18 +
src/backend/catalog/system_views.sql | 4 +-
src/backend/commands/statscmds.c | 44 +-
src/backend/nodes/outfuncs.c | 2 +
src/backend/optimizer/path/clausesel.c | 571 +++++++-
src/backend/optimizer/util/plancat.c | 4 +-
src/backend/utils/mvstats/Makefile | 2 +-
src/backend/utils/mvstats/README.histogram | 287 ++++
src/backend/utils/mvstats/README.stats | 2 +
src/backend/utils/mvstats/common.c | 37 +-
src/backend/utils/mvstats/histogram.c | 2032 ++++++++++++++++++++++++++++
src/bin/psql/describe.c | 17 +-
src/include/catalog/pg_mv_statistic.h | 24 +-
src/include/catalog/pg_proc.h | 4 +
src/include/nodes/relation.h | 2 +
src/include/utils/mvstats.h | 136 +-
src/test/regress/expected/mv_histogram.out | 207 +++
src/test/regress/expected/rules.out | 4 +-
src/test/regress/parallel_schedule | 2 +-
src/test/regress/serial_schedule | 1 +
src/test/regress/sql/mv_histogram.sql | 176 +++
21 files changed, 3538 insertions(+), 38 deletions(-)
create mode 100644 src/backend/utils/mvstats/README.histogram
create mode 100644 src/backend/utils/mvstats/histogram.c
create mode 100644 src/test/regress/expected/mv_histogram.out
create mode 100644 src/test/regress/sql/mv_histogram.sql
diff --git a/doc/src/sgml/ref/create_statistics.sgml b/doc/src/sgml/ref/create_statistics.sgml
index 193e4b0..fd3382e 100644
--- a/doc/src/sgml/ref/create_statistics.sgml
+++ b/doc/src/sgml/ref/create_statistics.sgml
@@ -133,6 +133,24 @@ CREATE STATISTICS [ IF NOT EXISTS ] <replaceable class="PARAMETER">statistics_na
</varlistentry>
<varlistentry>
+ <term><literal>histogram</> (<type>boolean</>)</term>
+ <listitem>
+ <para>
+ Enables histogram for the statistics.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>max_buckets</> (<type>integer</>)</term>
+ <listitem>
+ <para>
+ Maximum number of histogram buckets.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
<term><literal>max_mcv_items</> (<type>integer</>)</term>
<listitem>
<para>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 2d570ee..6afdee0 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -167,7 +167,9 @@ CREATE VIEW pg_mv_stats AS
length(S.stadeps) as depsbytes,
pg_mv_stats_dependencies_info(S.stadeps) as depsinfo,
length(S.stamcv) AS mcvbytes,
- pg_mv_stats_mcvlist_info(S.stamcv) AS mcvinfo
+ pg_mv_stats_mcvlist_info(S.stamcv) AS mcvinfo,
+ length(S.stahist) AS histbytes,
+ pg_mv_stats_histogram_info(S.stahist) AS histinfo
FROM (pg_mv_statistic S JOIN pg_class C ON (C.oid = S.starelid))
LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace);
diff --git a/src/backend/commands/statscmds.c b/src/backend/commands/statscmds.c
index b04c583..e2f3ff1 100644
--- a/src/backend/commands/statscmds.c
+++ b/src/backend/commands/statscmds.c
@@ -71,12 +71,15 @@ CreateStatistics(CreateStatsStmt *stmt)
/* by default build nothing */
bool build_dependencies = false,
- build_mcv = false;
+ build_mcv = false,
+ build_histogram = false;
- int32 max_mcv_items = -1;
+ int32 max_buckets = -1,
+ max_mcv_items = -1;
/* options required because of other options */
- bool require_mcv = false;
+ bool require_mcv = false,
+ require_histogram = false;
Assert(IsA(stmt, CreateStatsStmt));
@@ -175,6 +178,29 @@ CreateStatistics(CreateStatsStmt *stmt)
MVSTAT_MCVLIST_MAX_ITEMS)));
}
+ else if (strcmp(opt->defname, "histogram") == 0)
+ build_histogram = defGetBoolean(opt);
+ else if (strcmp(opt->defname, "max_buckets") == 0)
+ {
+ max_buckets = defGetInt32(opt);
+
+ /* this option requires 'histogram' to be enabled */
+ require_histogram = true;
+
+ /* sanity check */
+ if (max_buckets < MVSTAT_HIST_MIN_BUCKETS)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("minimum number of buckets is %d",
+ MVSTAT_HIST_MIN_BUCKETS)));
+
+ else if (max_buckets > MVSTAT_HIST_MAX_BUCKETS)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("maximum number of buckets is %d",
+ MVSTAT_HIST_MAX_BUCKETS)));
+
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -183,10 +209,10 @@ CreateStatistics(CreateStatsStmt *stmt)
}
/* check that at least some statistics were requested */
- if (! (build_dependencies || build_mcv))
+ if (! (build_dependencies || build_mcv || build_histogram))
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("no statistics type (dependencies, mcv) was requested")));
+ errmsg("no statistics type (dependencies, mcv, histogram) was requested")));
/* now do some checking of the options */
if (require_mcv && (! build_mcv))
@@ -194,6 +220,11 @@ CreateStatistics(CreateStatsStmt *stmt)
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("option 'mcv' is required by other options(s)")));
+ if (require_histogram && (! build_histogram))
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("option 'histogram' is required by other options(s)")));
+
/* sort the attnums and build int2vector */
qsort(attnums, numcols, sizeof(int16), compare_int16);
stakeys = buildint2vector(attnums, numcols);
@@ -214,11 +245,14 @@ CreateStatistics(CreateStatsStmt *stmt)
values[Anum_pg_mv_statistic_deps_enabled -1] = BoolGetDatum(build_dependencies);
values[Anum_pg_mv_statistic_mcv_enabled -1] = BoolGetDatum(build_mcv);
+ values[Anum_pg_mv_statistic_hist_enabled -1] = BoolGetDatum(build_histogram);
values[Anum_pg_mv_statistic_mcv_max_items -1] = Int32GetDatum(max_mcv_items);
+ values[Anum_pg_mv_statistic_hist_max_buckets -1] = Int32GetDatum(max_buckets);
nulls[Anum_pg_mv_statistic_stadeps -1] = true;
nulls[Anum_pg_mv_statistic_stamcv -1] = true;
+ nulls[Anum_pg_mv_statistic_stahist -1] = true;
/* insert the tuple into pg_mv_statistic */
mvstatrel = heap_open(MvStatisticRelationId, RowExclusiveLock);
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 333e24b..9172f21 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2163,10 +2163,12 @@ _outMVStatisticInfo(StringInfo str, const MVStatisticInfo *node)
/* enabled statistics */
WRITE_BOOL_FIELD(deps_enabled);
WRITE_BOOL_FIELD(mcv_enabled);
+ WRITE_BOOL_FIELD(hist_enabled);
/* built/available statistics */
WRITE_BOOL_FIELD(deps_built);
WRITE_BOOL_FIELD(mcv_built);
+ WRITE_BOOL_FIELD(hist_built);
}
static void
diff --git a/src/backend/optimizer/path/clausesel.c b/src/backend/optimizer/path/clausesel.c
index 977f88e..0de2418 100644
--- a/src/backend/optimizer/path/clausesel.c
+++ b/src/backend/optimizer/path/clausesel.c
@@ -49,6 +49,7 @@ static void addRangeClause(RangeQueryClause **rqlist, Node *clause,
#define MV_CLAUSE_TYPE_FDEP 0x01
#define MV_CLAUSE_TYPE_MCV 0x02
+#define MV_CLAUSE_TYPE_HIST 0x04
static bool clause_is_mv_compatible(Node *clause, Index relid, Bitmapset **attnums,
int type);
@@ -74,6 +75,8 @@ static Selectivity clauselist_mv_selectivity(PlannerInfo *root,
static Selectivity clauselist_mv_selectivity_mcvlist(PlannerInfo *root,
List *clauses, MVStatisticInfo *mvstats,
bool *fullmatch, Selectivity *lowsel);
+static Selectivity clauselist_mv_selectivity_histogram(PlannerInfo *root,
+ List *clauses, MVStatisticInfo *mvstats);
static int update_match_bitmap_mcvlist(PlannerInfo *root, List *clauses,
int2vector *stakeys, MCVList mcvlist,
@@ -81,6 +84,12 @@ static int update_match_bitmap_mcvlist(PlannerInfo *root, List *clauses,
Selectivity *lowsel, bool *fullmatch,
bool is_or);
+static int update_match_bitmap_histogram(PlannerInfo *root, List *clauses,
+ int2vector *stakeys,
+ MVSerializedHistogram mvhist,
+ int nmatches, char * matches,
+ bool is_or);
+
static bool has_stats(List *stats, int type);
static List * find_stats(PlannerInfo *root, Index relid);
@@ -93,6 +102,7 @@ static List * find_stats(PlannerInfo *root, Index relid);
#define UPDATE_RESULT(m,r,isor) \
(m) = (isor) ? (MAX(m,r)) : (MIN(m,r))
+
/****************************************************************************
* ROUTINES TO COMPUTE SELECTIVITIES
****************************************************************************/
@@ -121,7 +131,7 @@ static List * find_stats(PlannerInfo *root, Index relid);
*
* First we try to reduce the list of clauses by applying (soft) functional
* dependencies, and then we try to estimate the selectivity of the reduced
- * list of clauses using the multivariate MCV list.
+ * list of clauses using the multivariate MCV list and histograms.
*
* Finally we remove the portion of clauses estimated using multivariate stats,
* and process the rest of the clauses using the regular per-column stats.
@@ -214,11 +224,13 @@ clauselist_selectivity(PlannerInfo *root,
* with the multivariate code and simply skip to estimation using the
* regular per-column stats.
*/
- if (has_stats(stats, MV_CLAUSE_TYPE_MCV) &&
- (count_mv_attnums(clauses, relid, MV_CLAUSE_TYPE_MCV) >= 2))
+ if (has_stats(stats, MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST) &&
+ (count_mv_attnums(clauses, relid,
+ MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST) >= 2))
{
/* collect attributes from the compatible conditions */
- Bitmapset *mvattnums = collect_mv_attnums(clauses, relid, MV_CLAUSE_TYPE_MCV);
+ Bitmapset *mvattnums = collect_mv_attnums(clauses, relid,
+ MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST);
/* and search for the statistic covering the most attributes */
MVStatisticInfo *mvstat = choose_mv_statistics(stats, mvattnums);
@@ -230,7 +242,7 @@ clauselist_selectivity(PlannerInfo *root,
/* split the clauselist into regular and mv-clauses */
clauses = clauselist_mv_split(root, relid, clauses, &mvclauses,
- mvstat, MV_CLAUSE_TYPE_MCV);
+ mvstat, MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST);
/* we've chosen the histogram to match the clauses */
Assert(mvclauses != NIL);
@@ -942,6 +954,7 @@ static Selectivity
clauselist_mv_selectivity(PlannerInfo *root, List *clauses, MVStatisticInfo *mvstats)
{
bool fullmatch = false;
+ Selectivity s1 = 0.0, s2 = 0.0;
/*
* Lowest frequency in the MCV list (may be used as an upper bound
@@ -955,9 +968,24 @@ clauselist_mv_selectivity(PlannerInfo *root, List *clauses, MVStatisticInfo *mvs
* MCV/histogram evaluation).
*/
- /* Evaluate the MCV selectivity */
- return clauselist_mv_selectivity_mcvlist(root, clauses, mvstats,
+ /* Evaluate the MCV first. */
+ s1 = clauselist_mv_selectivity_mcvlist(root, clauses, mvstats,
&fullmatch, &mcv_low);
+
+ /*
+ * If we got a full equality match on the MCV list, we're done (and
+ * the estimate is pretty good).
+ */
+ if (fullmatch && (s1 > 0.0))
+ return s1;
+
+ /* TODO if (fullmatch) without matching MCV item, use the mcv_low
+ * selectivity as upper bound */
+
+ s2 = clauselist_mv_selectivity_histogram(root, clauses, mvstats);
+
+ /* TODO clamp to <= 1.0 (or more strictly, when possible) */
+ return s1 + s2;
}
/*
@@ -1160,7 +1188,7 @@ choose_mv_statistics(List *stats, Bitmapset *attnums)
int numattrs = attrs->dim1;
/* skip dependencies-only stats */
- if (! info->mcv_built)
+ if (! (info->mcv_built || info->hist_built))
continue;
/* count columns covered by the histogram */
@@ -1391,7 +1419,7 @@ mv_compatible_walker(Node *node, mv_compatible_context *context)
case F_SCALARGTSEL:
/* not compatible with functional dependencies */
- if (! (context->types & MV_CLAUSE_TYPE_MCV))
+ if (! (context->types & (MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST)))
return true; /* terminate */
break;
@@ -2007,6 +2035,9 @@ has_stats(List *stats, int type)
if ((type & MV_CLAUSE_TYPE_MCV) && stat->mcv_built)
return true;
+
+ if ((type & MV_CLAUSE_TYPE_HIST) && stat->hist_built)
+ return true;
}
return false;
@@ -2411,3 +2442,525 @@ update_match_bitmap_mcvlist(PlannerInfo *root, List *clauses,
return nmatches;
}
+
+/*
+ * Estimate selectivity of clauses using a histogram.
+ *
+ * If there's no histogram for the stats, the function returns 0.0.
+ *
+ * The general idea of this method is similar to how MCV lists are
+ * processed, except that this introduces the concept of a partial
+ * match (MCV only works with full match / mismatch).
+ *
+ * The algorithm works like this:
+ *
+ * 1) mark all buckets as 'full match'
+ * 2) walk through all the clauses
+ * 3) for a particular clause, walk through all the buckets
+ * 4) skip buckets that are already 'no match'
+ * 5) check clause for buckets that still match (at least partially)
+ * 6) sum frequencies for buckets to get selectivity
+ *
+ * Unlike MCV lists, histograms have a concept of a partial match. In
+ * that case we use 1/2 the bucket, to minimize the average error. The
+ * MV histograms are usually less detailed than the per-column ones,
+ * meaning the sum is often quite high (thanks to combining a lot of
+ * "partially hit" buckets).
+ *
+ * Maybe we could use per-bucket information with number of distinct
+ * values it contains (for each dimension), and then use that to correct
+ * the estimate (so with 10 distinct values, we'd use 1/10 of the bucket
+ * frequency). We might also scale the value depending on the actual
+ * ndistinct estimate (not just the values observed in the sample).
+ *
+ * Another option would be to multiply the selectivities, i.e. if we get
+ * 'partial match' for a bucket for multiple conditions, we might use
+ * 0.5^k (where k is the number of conditions), instead of 0.5. This
+ * probably does not minimize the average error, though.
+ *
+ * TODO This might use a similar shortcut to MCV lists - count buckets
+ * marked as partial/full match, and terminate once this drop to 0.
+ * Not sure if it's really worth it - for MCV lists a situation like
+ * this is not uncommon, but for histograms it's not that clear.
+ */
+static Selectivity
+clauselist_mv_selectivity_histogram(PlannerInfo *root, List *clauses,
+ MVStatisticInfo *mvstats)
+{
+ int i;
+ Selectivity s = 0.0;
+ Selectivity u = 0.0;
+
+ int nmatches = 0;
+ char *matches = NULL;
+
+ MVSerializedHistogram mvhist = NULL;
+
+ /* there's no histogram */
+ if (! mvstats->hist_built)
+ return 0.0;
+
+ /* There may be no histogram in the stats (check hist_built flag) */
+ mvhist = load_mv_histogram(mvstats->mvoid);
+
+ Assert (mvhist != NULL);
+ Assert (clauses != NIL);
+ Assert (list_length(clauses) >= 2);
+
+ /*
+ * Bitmap of bucket matches (mismatch, partial, full). by default
+ * all buckets fully match (and we'll eliminate them).
+ */
+ matches = palloc0(sizeof(char) * mvhist->nbuckets);
+ memset(matches, MVSTATS_MATCH_FULL, sizeof(char)*mvhist->nbuckets);
+
+ nmatches = mvhist->nbuckets;
+
+ /* build the match bitmap */
+ update_match_bitmap_histogram(root, clauses,
+ mvstats->stakeys, mvhist,
+ nmatches, matches, false);
+
+ /* now, walk through the buckets and sum the selectivities */
+ for (i = 0; i < mvhist->nbuckets; i++)
+ {
+ /*
+ * Find out what part of the data is covered by the histogram,
+ * so that we can 'scale' the selectivity properly (e.g. when
+ * only 50% of the sample got into the histogram, and the rest
+ * is in a MCV list).
+ *
+ * TODO This might be handled by keeping a global "frequency"
+ * for the whole histogram, which might save us some time
+ * spent accessing the not-matching part of the histogram.
+ * Although it's likely in a cache, so it's very fast.
+ */
+ u += mvhist->buckets[i]->ntuples;
+
+ if (matches[i] == MVSTATS_MATCH_FULL)
+ s += mvhist->buckets[i]->ntuples;
+ else if (matches[i] == MVSTATS_MATCH_PARTIAL)
+ s += 0.5 * mvhist->buckets[i]->ntuples;
+ }
+
+#ifdef DEBUG_MVHIST
+ debug_histogram_matches(mvhist, matches);
+#endif
+
+ /* release the allocated bitmap and deserialized histogram */
+ pfree(matches);
+ pfree(mvhist);
+
+ return s * u;
+}
+
+/* cached result of bucket boundary comparison for a single dimension */
+
+#define HIST_CACHE_NOT_FOUND 0x00
+#define HIST_CACHE_FALSE 0x01
+#define HIST_CACHE_TRUE 0x03
+#define HIST_CACHE_MASK 0x02
+
+static char
+bucket_contains_value(FmgrInfo ltproc, Datum constvalue,
+ Datum min_value, Datum max_value,
+ int min_index, int max_index,
+ bool min_include, bool max_include,
+ char * callcache)
+{
+ bool a, b;
+
+ char min_cached = callcache[min_index];
+ char max_cached = callcache[max_index];
+
+ /*
+ * First some quick checks on equality - if any of the boundaries equals,
+ * we have a partial match (so no need to call the comparator).
+ */
+ if (((min_value == constvalue) && (min_include)) ||
+ ((max_value == constvalue) && (max_include)))
+ return MVSTATS_MATCH_PARTIAL;
+
+ /* Keep the values 0/1 because of the XOR at the end. */
+ a = ((min_cached & HIST_CACHE_MASK) >> 1);
+ b = ((max_cached & HIST_CACHE_MASK) >> 1);
+
+ /*
+ * If result for the bucket lower bound not in cache, evaluate the function
+ * and store the result in the cache.
+ */
+ if (! min_cached)
+ {
+ a = DatumGetBool(FunctionCall2Coll(<proc,
+ DEFAULT_COLLATION_OID,
+ constvalue, min_value));
+ /* remember the result */
+ callcache[min_index] = (a) ? HIST_CACHE_TRUE : HIST_CACHE_FALSE;
+ }
+
+ /* And do the same for the upper bound. */
+ if (! max_cached)
+ {
+ b = DatumGetBool(FunctionCall2Coll(<proc,
+ DEFAULT_COLLATION_OID,
+ constvalue, max_value));
+ /* remember the result */
+ callcache[max_index] = (b) ? HIST_CACHE_TRUE : HIST_CACHE_FALSE;
+ }
+
+ return (a ^ b) ? MVSTATS_MATCH_PARTIAL : MVSTATS_MATCH_NONE;
+}
+
+static char
+bucket_is_smaller_than_value(FmgrInfo opproc, Datum constvalue,
+ Datum min_value, Datum max_value,
+ int min_index, int max_index,
+ bool min_include, bool max_include,
+ char * callcache, bool isgt)
+{
+ char min_cached = callcache[min_index];
+ char max_cached = callcache[max_index];
+
+ /* Keep the values 0/1 because of the XOR at the end. */
+ bool a = ((min_cached & HIST_CACHE_MASK) >> 1);
+ bool b = ((max_cached & HIST_CACHE_MASK) >> 1);
+
+ if (! min_cached)
+ {
+ a = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ min_value,
+ constvalue));
+ /* remember the result */
+ callcache[min_index] = (a) ? HIST_CACHE_TRUE : HIST_CACHE_FALSE;
+ }
+
+ if (! max_cached)
+ {
+ b = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ max_value,
+ constvalue));
+ /* remember the result */
+ callcache[max_index] = (b) ? HIST_CACHE_TRUE : HIST_CACHE_FALSE;
+ }
+
+ /*
+ * Now, we need to combine both results into the final answer, and we need
+ * to be careful about the 'isgt' variable which kinda inverts the meaning.
+ *
+ * First, we handle the case when each boundary returns different results.
+ * In that case the outcome can only be 'partial' match.
+ */
+ if (a != b)
+ return MVSTATS_MATCH_PARTIAL;
+
+ /*
+ * When the results are the same, then it depends on the 'isgt' value. There
+ * are four options:
+ *
+ * isgt=false a=b=true => full match
+ * isgt=false a=b=false => empty
+ * isgt=true a=b=true => empty
+ * isgt=true a=b=false => full match
+ *
+ * We'll cheat a bit, because we know that (a=b) so we'll use just one of them.
+ */
+ if (isgt)
+ return (!a) ? MVSTATS_MATCH_FULL : MVSTATS_MATCH_NONE;
+ else
+ return ( a) ? MVSTATS_MATCH_FULL : MVSTATS_MATCH_NONE;
+}
+
+/*
+ * Evaluate clauses using the histogram, and update the match bitmap.
+ *
+ * The bitmap may be already partially set, so this is really a way to
+ * combine results of several clause lists - either when computing
+ * conditional probability P(A|B) or a combination of AND/OR clauses.
+ *
+ * Note: This is not a simple bitmap in the sense that there are more
+ * than two possible values for each item - no match, partial
+ * match and full match. So we need 2 bits per item.
+ *
+ * TODO This works with 'bitmap' where each item is represented as a
+ * char, which is slightly wasteful. Instead, we could use a bitmap
+ * with 2 bits per item, reducing the size to ~1/4. By using values
+ * 0, 1 and 3 (instead of 0, 1 and 2), the operations (merging etc.)
+ * might be performed just like for simple bitmap by using & and |,
+ * which might be faster than min/max.
+ */
+static int
+update_match_bitmap_histogram(PlannerInfo *root, List *clauses,
+ int2vector *stakeys,
+ MVSerializedHistogram mvhist,
+ int nmatches, char * matches,
+ bool is_or)
+{
+ int i;
+ ListCell * l;
+
+ /*
+ * Used for caching function calls, only once per deduplicated value.
+ *
+ * We know may have up to (2 * nbuckets) values per dimension. It's
+ * probably overkill, but let's allocate that once for all clauses,
+ * to minimize overhead.
+ *
+ * Also, we only need two bits per value, but this allocates byte
+ * per value. Might be worth optimizing.
+ *
+ * 0x00 - not yet called
+ * 0x01 - called, result is 'false'
+ * 0x03 - called, result is 'true'
+ */
+ char *callcache = palloc(mvhist->nbuckets);
+
+ Assert(mvhist != NULL);
+ Assert(mvhist->nbuckets > 0);
+ Assert(nmatches >= 0);
+ Assert(nmatches <= mvhist->nbuckets);
+
+ Assert(clauses != NIL);
+ Assert(list_length(clauses) >= 1);
+
+ /* loop through the clauses and do the estimation */
+ foreach (l, clauses)
+ {
+ Node * clause = (Node*)lfirst(l);
+
+ /* if it's a RestrictInfo, then extract the clause */
+ if (IsA(clause, RestrictInfo))
+ clause = (Node*)((RestrictInfo*)clause)->clause;
+
+ /* it's either OpClause, or NullTest */
+ if (is_opclause(clause))
+ {
+ OpExpr * expr = (OpExpr*)clause;
+ bool varonleft = true;
+ bool ok;
+
+ FmgrInfo opproc; /* operator */
+ fmgr_info(get_opcode(expr->opno), &opproc);
+
+ /* reset the cache (per clause) */
+ memset(callcache, 0, mvhist->nbuckets);
+
+ ok = (NumRelids(clause) == 1) &&
+ (is_pseudo_constant_clause(lsecond(expr->args)) ||
+ (varonleft = false,
+ is_pseudo_constant_clause(linitial(expr->args))));
+
+ if (ok)
+ {
+ FmgrInfo ltproc;
+ RegProcedure oprrest = get_oprrest(expr->opno);
+
+ Var * var = (varonleft) ? linitial(expr->args) : lsecond(expr->args);
+ Const * cst = (varonleft) ? lsecond(expr->args) : linitial(expr->args);
+ bool isgt = (! varonleft);
+
+ TypeCacheEntry *typecache
+ = lookup_type_cache(var->vartype, TYPECACHE_LT_OPR);
+
+ /* lookup dimension for the attribute */
+ int idx = mv_get_index(var->varattno, stakeys);
+
+ fmgr_info(get_opcode(typecache->lt_opr), <proc);
+
+ /*
+ * Check this for all buckets that still have "true" in the bitmap
+ *
+ * We already know the clauses use suitable operators (because that's
+ * how we filtered them).
+ */
+ for (i = 0; i < mvhist->nbuckets; i++)
+ {
+ char res = MVSTATS_MATCH_NONE;
+
+ MVSerializedBucket bucket = mvhist->buckets[i];
+
+ /* histogram boundaries */
+ Datum minval, maxval;
+ bool mininclude, maxinclude;
+ int minidx, maxidx;
+
+ /*
+ * For AND-lists, we can also mark NULL buckets as 'no match'
+ * (and then skip them). For OR-lists this is not possible.
+ */
+ if ((! is_or) && bucket->nullsonly[idx])
+ matches[i] = MVSTATS_MATCH_NONE;
+
+ /*
+ * Skip buckets that were already eliminated - this is impotant
+ * considering how we update the info (we only lower the match).
+ * We can't really do anything about the MATCH_PARTIAL buckets.
+ */
+ if ((! is_or) && (matches[i] == MVSTATS_MATCH_NONE))
+ continue;
+ else if (is_or && (matches[i] == MVSTATS_MATCH_FULL))
+ continue;
+
+ /* lookup the values and cache of function calls */
+ minidx = bucket->min[idx];
+ maxidx = bucket->max[idx];
+
+ minval = mvhist->values[idx][bucket->min[idx]];
+ maxval = mvhist->values[idx][bucket->max[idx]];
+
+ mininclude = bucket->min_inclusive[idx];
+ maxinclude = bucket->max_inclusive[idx];
+
+ /*
+ * TODO Maybe it's possible to add here a similar optimization
+ * as for the MCV lists:
+ *
+ * (nmatches == 0) && AND-list => all eliminated (FALSE)
+ * (nmatches == N) && OR-list => all eliminated (TRUE)
+ *
+ * But it's more complex because of the partial matches.
+ */
+
+ /*
+ * If it's not a "<" or ">" or "=" operator, just ignore the
+ * clause. Otherwise note the relid and attnum for the variable.
+ *
+ * TODO I'm really unsure the handling of 'isgt' flag (that is, clauses
+ * with reverse order of variable/constant) is correct. I wouldn't
+ * be surprised if there was some mixup. Using the lt/gt operators
+ * instead of messing with the opproc could make it simpler.
+ * It would however be using a different operator than the query,
+ * although it's not any shadier than using the selectivity function
+ * as is done currently.
+ */
+ switch (oprrest)
+ {
+ case F_SCALARLTSEL: /* Var < Const */
+ case F_SCALARGTSEL: /* Var > Const */
+
+ res = bucket_is_smaller_than_value(opproc, cst->constvalue,
+ minval, maxval,
+ minidx, maxidx,
+ mininclude, maxinclude,
+ callcache, isgt);
+ break;
+
+ case F_EQSEL:
+
+ /*
+ * We only check whether the value is within the bucket, using the
+ * lt operator, and we also check for equality with the boundaries.
+ */
+
+ res = bucket_contains_value(ltproc, cst->constvalue,
+ minval, maxval,
+ minidx, maxidx,
+ mininclude, maxinclude,
+ callcache);
+ break;
+ }
+
+ UPDATE_RESULT(matches[i], res, is_or);
+
+ }
+ }
+ }
+ else if (IsA(clause, NullTest))
+ {
+ NullTest * expr = (NullTest*)clause;
+ Var * var = (Var*)(expr->arg);
+
+ /* FIXME proper matching attribute to dimension */
+ int idx = mv_get_index(var->varattno, stakeys);
+
+ /*
+ * Walk through the buckets and evaluate the current clause. We can
+ * skip items that were already ruled out, and terminate if there are
+ * no remaining buckets that might possibly match.
+ */
+ for (i = 0; i < mvhist->nbuckets; i++)
+ {
+ MVSerializedBucket bucket = mvhist->buckets[i];
+
+ /*
+ * Skip buckets that were already eliminated - this is impotant
+ * considering how we update the info (we only lower the match)
+ */
+ if ((! is_or) && (matches[i] == MVSTATS_MATCH_NONE))
+ continue;
+ else if (is_or && (matches[i] == MVSTATS_MATCH_FULL))
+ continue;
+
+ /* if the clause mismatches the bucket, set it as MATCH_NONE */
+ if ((expr->nulltesttype == IS_NULL)
+ && (! bucket->nullsonly[idx]))
+ UPDATE_RESULT(matches[i], MVSTATS_MATCH_NONE, is_or);
+
+ else if ((expr->nulltesttype == IS_NOT_NULL) &&
+ (bucket->nullsonly[idx]))
+ UPDATE_RESULT(matches[i], MVSTATS_MATCH_NONE, is_or);
+ }
+ }
+ else if (or_clause(clause) || and_clause(clause))
+ {
+ /* AND/OR clause, with all clauses compatible with the selected MV stat */
+
+ int i;
+ BoolExpr *orclause = ((BoolExpr*)clause);
+ List *orclauses = orclause->args;
+
+ /* match/mismatch bitmap for each bucket */
+ int or_nmatches = 0;
+ char * or_matches = NULL;
+
+ Assert(orclauses != NIL);
+ Assert(list_length(orclauses) >= 2);
+
+ /* number of matching buckets */
+ or_nmatches = mvhist->nbuckets;
+
+ /* by default none of the buckets matches the clauses */
+ or_matches = palloc0(sizeof(char) * or_nmatches);
+
+ if (or_clause(clause))
+ {
+ /* OR clauses assume nothing matches, initially */
+ memset(or_matches, MVSTATS_MATCH_NONE, sizeof(char)*or_nmatches);
+ or_nmatches = 0;
+ }
+ else
+ {
+ /* AND clauses assume nothing matches, initially */
+ memset(or_matches, MVSTATS_MATCH_FULL, sizeof(char)*or_nmatches);
+ }
+
+ /* build the match bitmap for the OR-clauses */
+ or_nmatches = update_match_bitmap_histogram(root, orclauses,
+ stakeys, mvhist,
+ or_nmatches, or_matches, or_clause(clause));
+
+ /* merge the bitmap into the existing one*/
+ for (i = 0; i < mvhist->nbuckets; i++)
+ {
+ /*
+ * To AND-merge the bitmaps, a MIN() semantics is used.
+ * For OR-merge, use MAX().
+ *
+ * FIXME this does not decrease the number of matches
+ */
+ UPDATE_RESULT(matches[i], or_matches[i], is_or);
+ }
+
+ pfree(or_matches);
+
+ }
+ else
+ elog(ERROR, "unknown clause type: %d", clause->type);
+ }
+
+ /* free the call cache */
+ pfree(callcache);
+
+ return nmatches;
+}
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index 8394111..2519249 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -412,7 +412,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
mvstat = (Form_pg_mv_statistic) GETSTRUCT(htup);
/* unavailable stats are not interesting for the planner */
- if (mvstat->deps_built || mvstat->mcv_built)
+ if (mvstat->deps_built || mvstat->mcv_built || mvstat->hist_built)
{
info = makeNode(MVStatisticInfo);
@@ -422,10 +422,12 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
/* enabled statistics */
info->deps_enabled = mvstat->deps_enabled;
info->mcv_enabled = mvstat->mcv_enabled;
+ info->hist_enabled = mvstat->hist_enabled;
/* built/available statistics */
info->deps_built = mvstat->deps_built;
info->mcv_built = mvstat->mcv_built;
+ info->hist_built = mvstat->hist_built;
/* stakeys */
adatum = SysCacheGetAttr(MVSTATOID, htup,
diff --git a/src/backend/utils/mvstats/Makefile b/src/backend/utils/mvstats/Makefile
index f9bf10c..9dbb3b6 100644
--- a/src/backend/utils/mvstats/Makefile
+++ b/src/backend/utils/mvstats/Makefile
@@ -12,6 +12,6 @@ subdir = src/backend/utils/mvstats
top_builddir = ../../../..
include $(top_builddir)/src/Makefile.global
-OBJS = common.o dependencies.o mcv.o
+OBJS = common.o dependencies.o histogram.o mcv.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/utils/mvstats/README.histogram b/src/backend/utils/mvstats/README.histogram
new file mode 100644
index 0000000..8234d2c
--- /dev/null
+++ b/src/backend/utils/mvstats/README.histogram
@@ -0,0 +1,287 @@
+Multivariate histograms
+=======================
+
+Histograms on individual attributes consist of buckets represented by ranges,
+covering the domain of the attribute. That is, each bucket is a [min,max]
+interval, and contains all values in this range. The histogram is built in such
+a way that all buckets have about the same frequency.
+
+Multivariate histograms are an extension into n-dimensional space - the buckets
+are n-dimensional intervals (i.e. n-dimensional rectagles), covering the domain
+of the combination of attributes. That is, each bucket has a vector of lower
+and upper boundaries, denoted min[i] and max[i] (where i = 1..n).
+
+In addition to the boundaries, each bucket tracks additional info:
+
+ * frequency (fraction of tuples in the bucket)
+ * whether the boundaries are inclusive or exclusive
+ * whether the dimension contains only NULL values
+ * number of distinct values in each dimension (for building only)
+
+It's possible that in the future we'll multiple histogram types, with different
+features. We do however expect all the types to share the same representation
+(buckets as ranges) and only differ in how we build them.
+
+The current implementation builds non-overlapping buckets, that may not be true
+for some histogram types and the code should not rely on this assumption. There
+are interesting types of histograms (or algorithms) with overlapping buckets.
+
+When used on low-cardinality data, histograms usually perform considerably worse
+than MCV lists (which are a good fit for this kind of data). This is especially
+true on label-like values, where ordering of the values is mostly unrelated to
+meaning of the data, as proper ordering is crucial for histograms.
+
+On high-cardinality data the histograms are usually a better choice, because MCV
+lists can't represent the distribution accurately enough.
+
+
+Selectivity estimation
+----------------------
+
+The estimation is implemented in clauselist_mv_selectivity_histogram(), and
+works very similarly to clauselist_mv_selectivity_mcvlist.
+
+The main difference is that while MCV lists support exact matches, histograms
+often result in approximate matches - e.g. with equality we can only say if
+the constant would be part of the bucket, but not whether it really is there
+or what fraction of the bucket it corresponds to. In this case we rely on
+some defaults just like in the per-column histograms.
+
+The current implementation uses histograms to estimates those types of clauses
+(think of WHERE conditions):
+
+ (a) equality clauses WHERE (a = 1) AND (b = 2)
+ (b) inequality clauses WHERE (a < 1) AND (b >= 2)
+ (c) NULL clauses WHERE (a IS NULL) AND (b IS NOT NULL)
+ (d) OR-clauses WHERE (a = 1) OR (b = 2)
+
+Similarly to MCV lists, it's possible to add support for additional types of
+clauses, for example:
+
+ (e) multi-var clauses WHERE (a > b)
+
+and so on. These are tasks for the future, not yet implemented.
+
+
+When evaluating a clause on a bucket, we may get one of three results:
+
+ (a) FULL_MATCH - The bucket definitely matches the clause.
+
+ (b) PARTIAL_MATCH - The bucket matches the clause, but not necessarily all
+ the tuples it represents.
+
+ (c) NO_MATCH - The bucket definitely does not match the clause.
+
+This may be illustrated using a range [1, 5], which is essentially a 1-D bucket.
+With clause
+
+ WHERE (a < 10) => FULL_MATCH (all range values are below
+ 10, so the whole bucket matches)
+
+ WHERE (a < 3) => PARTIAL_MATCH (there may be values matching
+ the clause, but we don't know how many)
+
+ WHERE (a < 0) => NO_MATCH (the whole range is above 1, so
+ no values from the bucket can match)
+
+Some clauses may produce only some of those results - for example equality
+clauses may never produce FULL_MATCH as we always hit only part of the bucket
+(we can't match both boundaries at the same time). This results in less accurate
+estimates compared to MCV lists, where we can hit a MCV items exactly (there's
+no PARTIAL match in MCV).
+
+There are also clauses that may not produce any PARTIAL_MATCH results. A nice
+example of that is 'IS [NOT] NULL' clause, which either matches the bucket
+completely (FULL_MATCH) or not at all (NO_MATCH), thanks to how the NULL-buckets
+are constructed.
+
+Computing the total selectivity estimate is trivial - simply sum selectivities
+from all the FULL_MATCH and PARTIAL_MATCH buckets (but for buckets marked with
+PARTIAL_MATCH, multiply the frequency by 0.5 to minimize the average error).
+
+
+Building a histogram
+---------------------
+
+The algorithm of building a histogram in general is quite simple:
+
+ (a) create an initial bucket (containing all sample rows)
+
+ (b) create NULL buckets (by splitting the initial bucket)
+
+ (c) repeat
+
+ (1) choose bucket to split next
+
+ (2) terminate if no bucket that might be split found, or if we've
+ reached the maximum number of buckets (16384)
+
+ (3) choose dimension to partition the bucket by
+
+ (4) partition the bucket by the selected dimension
+
+The main complexity is hidden in steps (c.1) and (c.3), i.e. how we choose the
+bucket and dimension for the split.
+
+Similarly to one-dimensional histograms, we want to produce buckets with roughly
+the same frequency. We also need to produce "regular" buckets, because buckets
+with one "side" much longer than the others are very likely to match a lot of
+conditions (which increases error, even if the bucket frequency is very low).
+
+To achieve this, we choose the largest bucket (containing the most sample rows),
+but we only choose buckets that can actually be split (have at least 3 different
+combinations of values).
+
+Then we choose the "longest" dimension of the bucket, which is computed by using
+the distinct values in the sample as a measure.
+
+For details see functions select_bucket_to_partition() and partition_bucket().
+
+The current limit on number of buckets (16384) is mostly arbitrary, but chosen
+so that it guarantees we don't exceed the number of distinct values indexable by
+uint16 in any of the dimensions. In practice we could handle more buckets as we
+index each dimension separately and the splits should use the dimensions evenly.
+
+Also, histograms this large (with 16k values in multiple dimensions) would be
+quite expensive to build and process, so the 16k limit is rather reasonable.
+
+The actual number of buckets is also related to statistics target, because we
+require MIN_BUCKET_ROWS (10) tuples per bucket before a split, so we can't have
+more than (2 * 300 * target / 10) buckets. For the default target (100) this
+evaluates to ~6k.
+
+
+NULL handling (create_null_buckets)
+-----------------------------------
+
+When building histograms on a single attribute, we first filter out NULL values.
+In the multivariate case, we can't really do that because the rows may contain
+a mix of NULL and non-NULL values in different columns (so we can't simply
+filter all of them out).
+
+For this reason, the histograms are built in a way so that for each bucket, each
+dimension only contains only NULL or non-NULL values. Building the NULL-buckets
+happens as the first step in the build, by the create_null_buckets() function.
+The number of NULL buckets, as produced by this function, has a clear upper
+boundary (2^N) where N is the number of dimensions (attributes the histogram is
+built on). Or rather 2^K where K is the number of attributes that are not marked
+as not-NULL.
+
+The buckets with NULL dimensions are then subject to the same build algorithm
+(i.e. may be split into smaller buckets) just like any other bucket, but may
+only be split by non-NULL dimension.
+
+
+Serialization
+-------------
+
+To store the histogram in pg_mv_statistic table, it is serialized into a more
+efficient form. We also use the representation for estimation, i.e. we don't
+fully deserialize the histogram.
+
+For example the boundary values are deduplicated to minimize the required space.
+How much redundancy is there, actually? Let's assume there are no NULL values,
+so we start with a single bucket - in that case we have 2*N boundaries. Each
+time we split a bucket we introduce one new value (in the "middle" of one of
+the dimensions), and keep boundries for all the other dimensions. So after K
+splits, we have up to
+
+ 2*N + K
+
+unique boundary values (we may have fewe values, if the same value is used for
+several splits). But after K splits we do have (K+1) buckets, so
+
+ (K+1) * 2 * N
+
+boundary values. Using e.g. N=4 and K=999, we arrive to those numbers:
+
+ 2*N + K = 1007
+ (K+1) * 2 * N = 8000
+
+wich means a lot of redundancy. It's somewhat counter-intuitive that the number
+of distinct values does not really depend on the number of dimensions (except
+for the initial bucket, but that's negligible compared to the total).
+
+By deduplicating the values and replacing them with 16-bit indexes (uint16), we
+reduce the required space to
+
+ 1007 * 8 + 8000 * 2 ~= 24kB
+
+which is significantly less than 64kB required for the 'raw' histogram (assuming
+the values are 8B).
+
+While the bytea compression (pglz) might achieve the same reduction of space,
+the deduplicated representation is used to optimize the estimation by caching
+results of function calls for already visited values. This significantly
+reduces the number of calls to (often quite expensive) operators.
+
+Note: Of course, this reasoning only holds for histograms built by the algorithm
+that simply splits the buckets in half. Other histograms types (e.g. containing
+overlapping buckets) may behave differently and require different serialization.
+
+Serialized histograms are marked with 'magic' constant, to make it easier to
+check the bytea value really is a serialized histogram.
+
+
+varlena compression
+-------------------
+
+This serialization may however disable automatic varlena compression, the array
+of unique values is placed at the beginning of the serialized form. Which is
+exactly the chunk used by pglz to check if the data is compressible, and it
+will probably decide it's not very compressible. This is similar to the issue
+we had with JSONB initially.
+
+Maybe storing buckets first would make it work, as the buckets may be better
+compressible.
+
+On the other hand the serialization is actually a context-aware compression,
+usually compressing to ~30% (or even less, with large data types). So the lack
+of additional pglz compression may be acceptable.
+
+
+Deserialization
+---------------
+
+The deserialization is not a perfect inverse of the serialization, as we keep
+the deduplicated arrays. This reduces the amount of memory and also allows
+optimizations during estimation (e.g. we can cache results for the distinct
+values, saving expensive function calls).
+
+
+Inspecting the histogram
+------------------------
+
+Inspecting the regular (per-attribute) histograms is trivial, as it's enough
+to select the columns from pg_stats - the data is encoded as anyarray, so we
+simply get the text representation of the array.
+
+With multivariate histograms it's not that simple due to the possible mix of
+data types in the histogram. It might be possible to produce similar array-like
+text representation, but that'd unnecessarily complicate further processing
+and analysis of the histogram. Instead, there's a SRF function that allows
+access to lower/upper boundaries, frequencies etc.
+
+ SELECT * FROM pg_mv_histogram_buckets();
+
+It has two input parameters:
+
+ oid - OID of the histogram (pg_mv_statistic.staoid)
+ otype - type of output
+
+and produces a table with these columns:
+
+ - bucket ID (0...nbuckets-1)
+ - lower bucket boundaries (string array)
+ - upper bucket boundaries (string array)
+ - nulls only dimensions (boolean array)
+ - lower boundary inclusive (boolean array)
+ - upper boundary includive (boolean array)
+ - frequency (double precision)
+
+The 'otype' accepts three values, determining what will be returned in the
+lower/upper boundary arrays:
+
+ - 0 - values stored in the histogram, encoded as text
+ - 1 - indexes into the deduplicated arrays
+ - 2 - idnexes into the deduplicated arrays, scaled to [0,1]
diff --git a/src/backend/utils/mvstats/README.stats b/src/backend/utils/mvstats/README.stats
index 5c5c59a..3e4f4d1 100644
--- a/src/backend/utils/mvstats/README.stats
+++ b/src/backend/utils/mvstats/README.stats
@@ -18,6 +18,8 @@ Currently we only have two kinds of multivariate statistics
(b) MCV lists (README.mcv)
+ (c) multivariate histograms (README.histogram)
+
Compatible clause types
-----------------------
diff --git a/src/backend/utils/mvstats/common.c b/src/backend/utils/mvstats/common.c
index d1da714..ffb76f4 100644
--- a/src/backend/utils/mvstats/common.c
+++ b/src/backend/utils/mvstats/common.c
@@ -13,11 +13,11 @@
*
*-------------------------------------------------------------------------
*/
+#include "postgres.h"
+#include "utils/array.h"
#include "common.h"
-#include "utils/array.h"
-
static VacAttrStats ** lookup_var_attr_stats(int2vector *attrs,
int natts,
VacAttrStats **vacattrstats);
@@ -52,7 +52,8 @@ build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
MVStatisticInfo *stat = (MVStatisticInfo *)lfirst(lc);
MVDependencies deps = NULL;
MCVList mcvlist = NULL;
- int numrows_filtered = 0;
+ MVHistogram histogram = NULL;
+ int numrows_filtered = numrows;
VacAttrStats **stats = NULL;
int numatts = 0;
@@ -95,8 +96,12 @@ build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
if (stat->mcv_enabled)
mcvlist = build_mv_mcvlist(numrows, rows, attrs, stats, &numrows_filtered);
+ /* build a multivariate histogram on the columns */
+ if ((numrows_filtered > 0) && (stat->hist_enabled))
+ histogram = build_mv_histogram(numrows_filtered, rows, attrs, stats, numrows);
+
/* store the histogram / MCV list in the catalog */
- update_mv_stats(stat->mvoid, deps, mcvlist, attrs, stats);
+ update_mv_stats(stat->mvoid, deps, mcvlist, histogram, attrs, stats);
}
}
@@ -176,6 +181,8 @@ list_mv_stats(Oid relid)
info->deps_built = stats->deps_built;
info->mcv_enabled = stats->mcv_enabled;
info->mcv_built = stats->mcv_built;
+ info->hist_enabled = stats->hist_enabled;
+ info->hist_built = stats->hist_built;
result = lappend(result, info);
}
@@ -190,7 +197,6 @@ list_mv_stats(Oid relid)
return result;
}
-
/*
* Find attnims of MV stats using the mvoid.
*/
@@ -236,9 +242,16 @@ find_mv_attnums(Oid mvoid, Oid *relid)
}
+/*
+ * FIXME This adds statistics, but we need to drop statistics when the
+ * table is dropped. Not sure what to do when a column is dropped.
+ * Either we can (a) remove all stats on that column, (b) remove
+ * the column from defined stats and force rebuild, (c) remove the
+ * column on next ANALYZE. Or maybe something else?
+ */
void
update_mv_stats(Oid mvoid,
- MVDependencies dependencies, MCVList mcvlist,
+ MVDependencies dependencies, MCVList mcvlist, MVHistogram histogram,
int2vector *attrs, VacAttrStats **stats)
{
HeapTuple stup,
@@ -271,22 +284,34 @@ update_mv_stats(Oid mvoid,
values[Anum_pg_mv_statistic_stamcv - 1] = PointerGetDatum(data);
}
+ if (histogram != NULL)
+ {
+ bytea * data = serialize_mv_histogram(histogram, attrs, stats);
+ nulls[Anum_pg_mv_statistic_stahist-1] = (data == NULL);
+ values[Anum_pg_mv_statistic_stahist - 1]
+ = PointerGetDatum(data);
+ }
+
/* always replace the value (either by bytea or NULL) */
replaces[Anum_pg_mv_statistic_stadeps -1] = true;
replaces[Anum_pg_mv_statistic_stamcv -1] = true;
+ replaces[Anum_pg_mv_statistic_stahist-1] = true;
/* always change the availability flags */
nulls[Anum_pg_mv_statistic_deps_built -1] = false;
nulls[Anum_pg_mv_statistic_mcv_built -1] = false;
+ nulls[Anum_pg_mv_statistic_hist_built-1] = false;
nulls[Anum_pg_mv_statistic_stakeys-1] = false;
/* use the new attnums, in case we removed some dropped ones */
replaces[Anum_pg_mv_statistic_deps_built-1] = true;
replaces[Anum_pg_mv_statistic_mcv_built -1] = true;
+ replaces[Anum_pg_mv_statistic_hist_built -1] = true;
replaces[Anum_pg_mv_statistic_stakeys -1] = true;
values[Anum_pg_mv_statistic_deps_built-1] = BoolGetDatum(dependencies != NULL);
values[Anum_pg_mv_statistic_mcv_built -1] = BoolGetDatum(mcvlist != NULL);
+ values[Anum_pg_mv_statistic_hist_built -1] = BoolGetDatum(histogram != NULL);
values[Anum_pg_mv_statistic_stakeys -1] = PointerGetDatum(attrs);
/* Is there already a pg_mv_statistic tuple for this attribute? */
diff --git a/src/backend/utils/mvstats/histogram.c b/src/backend/utils/mvstats/histogram.c
new file mode 100644
index 0000000..9e5620a
--- /dev/null
+++ b/src/backend/utils/mvstats/histogram.c
@@ -0,0 +1,2032 @@
+/*-------------------------------------------------------------------------
+ *
+ * histogram.c
+ * POSTGRES multivariate histograms
+ *
+ *
+ * Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/utils/mvstats/histogram.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "fmgr.h"
+#include "funcapi.h"
+
+#include "utils/lsyscache.h"
+
+#include "common.h"
+#include <math.h>
+
+
+static MVBucket create_initial_mv_bucket(int numrows, HeapTuple *rows,
+ int2vector *attrs,
+ VacAttrStats **stats);
+
+static MVBucket select_bucket_to_partition(int nbuckets, MVBucket * buckets);
+
+static MVBucket partition_bucket(MVBucket bucket, int2vector *attrs,
+ VacAttrStats **stats,
+ int *ndistvalues, Datum **distvalues);
+
+static MVBucket copy_mv_bucket(MVBucket bucket, uint32 ndimensions);
+
+static void update_bucket_ndistinct(MVBucket bucket, int2vector *attrs,
+ VacAttrStats ** stats);
+
+static void update_dimension_ndistinct(MVBucket bucket, int dimension,
+ int2vector *attrs,
+ VacAttrStats ** stats,
+ bool update_boundaries);
+
+static void create_null_buckets(MVHistogram histogram, int bucket_idx,
+ int2vector *attrs, VacAttrStats ** stats);
+
+static int bsearch_comparator(const void * a, const void * b);
+
+/*
+ * Each serialized bucket needs to store (in this order):
+ *
+ * - number of tuples (float)
+ * - number of distinct (float)
+ * - min inclusive flags (ndim * sizeof(bool))
+ * - max inclusive flags (ndim * sizeof(bool))
+ * - null dimension flags (ndim * sizeof(bool))
+ * - min boundary indexes (2 * ndim * sizeof(int32))
+ * - max boundary indexes (2 * ndim * sizeof(int32))
+ *
+ * So in total:
+ *
+ * ndim * (4 * sizeof(int32) + 3 * sizeof(bool)) +
+ * 2 * sizeof(float)
+ */
+#define BUCKET_SIZE(ndims) \
+ (ndims * (4 * sizeof(uint16) + 3 * sizeof(bool)) + sizeof(float))
+
+/* pointers into a flat serialized bucket of BUCKET_SIZE(n) bytes */
+#define BUCKET_NTUPLES(b) ((float*)b)
+#define BUCKET_MIN_INCL(b,n) ((bool*)(b + sizeof(float)))
+#define BUCKET_MAX_INCL(b,n) (BUCKET_MIN_INCL(b,n) + n)
+#define BUCKET_NULLS_ONLY(b,n) (BUCKET_MAX_INCL(b,n) + n)
+#define BUCKET_MIN_INDEXES(b,n) ((uint16*)(BUCKET_NULLS_ONLY(b,n) + n))
+#define BUCKET_MAX_INDEXES(b,n) ((BUCKET_MIN_INDEXES(b,n) + n))
+
+/* can't split bucket with less than 10 rows */
+#define MIN_BUCKET_ROWS 10
+
+/*
+ * Data used while building the histogram.
+ */
+typedef struct HistogramBuildData {
+
+ float ndistinct; /* frequency of distinct values */
+
+ HeapTuple *rows; /* aray of sample rows */
+ uint32 numrows; /* number of sample rows (array size) */
+
+ /*
+ * Number of distinct values in each dimension. This is used when
+ * building the histogram (and is not serialized/deserialized).
+ */
+ uint32 *ndistincts;
+
+} HistogramBuildData;
+
+typedef HistogramBuildData *HistogramBuild;
+
+/*
+ * Building a multivariate algorithm. In short it first creates a single
+ * bucket containing all the rows, and then repeatedly split is by first
+ * searching for the bucket / dimension most in need of a split.
+ *
+ * The current criteria is rather simple, chosen so that the algorithm
+ * produces buckets with about equal frequency and regular size.
+ *
+ * See the discussion at select_bucket_to_partition and partition_bucket
+ * for more details about the algorithm.
+ *
+ * The current algorithm works like this:
+ *
+ * build NULL-buckets (create_null_buckets)
+ *
+ * while [not reaching maximum number of buckets]
+ *
+ * choose bucket to partition (largest bucket)
+ * if no bucket to partition
+ * terminate the algorithm
+ *
+ * choose bucket dimension to partition (largest dimension)
+ * split the bucket into two buckets
+ */
+MVHistogram
+build_mv_histogram(int numrows, HeapTuple *rows, int2vector *attrs,
+ VacAttrStats **stats, int numrows_total)
+{
+ int i;
+ int numattrs = attrs->dim1;
+
+ int *ndistvalues;
+ Datum **distvalues;
+
+ MVHistogram histogram = (MVHistogram)palloc0(sizeof(MVHistogramData));
+
+ HeapTuple * rows_copy = (HeapTuple*)palloc0(numrows * sizeof(HeapTuple));
+ memcpy(rows_copy, rows, sizeof(HeapTuple) * numrows);
+
+ Assert((numattrs >= 2) && (numattrs <= MVSTATS_MAX_DIMENSIONS));
+
+ histogram->ndimensions = numattrs;
+
+ histogram->magic = MVSTAT_HIST_MAGIC;
+ histogram->type = MVSTAT_HIST_TYPE_BASIC;
+ histogram->nbuckets = 1;
+
+ /* create max buckets (better than repalloc for short-lived objects) */
+ histogram->buckets
+ = (MVBucket*)palloc0(MVSTAT_HIST_MAX_BUCKETS * sizeof(MVBucket));
+
+ /* create the initial bucket, covering the whole sample set */
+ histogram->buckets[0]
+ = create_initial_mv_bucket(numrows, rows_copy, attrs, stats);
+
+ /*
+ * Collect info on distinct values in each dimension (used later
+ * to select dimension to partition).
+ */
+ ndistvalues = (int*)palloc0(sizeof(int) * numattrs);
+ distvalues = (Datum**)palloc0(sizeof(Datum*) * numattrs);
+
+ for (i = 0; i < numattrs; i++)
+ {
+ int j;
+ int nvals;
+ Datum *tmp;
+
+ SortSupportData ssup;
+ StdAnalyzeData *mystats = (StdAnalyzeData *) stats[i]->extra_data;
+
+ /* initialize sort support, etc. */
+ memset(&ssup, 0, sizeof(ssup));
+ ssup.ssup_cxt = CurrentMemoryContext;
+
+ /* We always use the default collation for statistics */
+ ssup.ssup_collation = DEFAULT_COLLATION_OID;
+ ssup.ssup_nulls_first = false;
+
+ PrepareSortSupportFromOrderingOp(mystats->ltopr, &ssup);
+
+ nvals = 0;
+ tmp = (Datum*)palloc0(sizeof(Datum) * numrows);
+
+ for (j = 0; j < numrows; j++)
+ {
+ bool isnull;
+
+ /* remember the index of the sample row, to make the partitioning simpler */
+ Datum value = heap_getattr(rows[j], attrs->values[i],
+ stats[i]->tupDesc, &isnull);
+
+ if (isnull)
+ continue;
+
+ tmp[nvals++] = value;
+ }
+
+ /* do the sort and stuff only if there are non-NULL values */
+ if (nvals > 0)
+ {
+ /* sort the array of values */
+ qsort_arg((void *) tmp, nvals, sizeof(Datum),
+ compare_scalars_simple, (void *) &ssup);
+
+ /* count distinct values */
+ ndistvalues[i] = 1;
+ for (j = 1; j < nvals; j++)
+ if (compare_scalars_simple(&tmp[j], &tmp[j-1], &ssup) != 0)
+ ndistvalues[i] += 1;
+
+ /* FIXME allocate only needed space (count ndistinct first) */
+ distvalues[i] = (Datum*)palloc0(sizeof(Datum) * ndistvalues[i]);
+
+ /* now collect distinct values into the array */
+ distvalues[i][0] = tmp[0];
+ ndistvalues[i] = 1;
+
+ for (j = 1; j < nvals; j++)
+ {
+ if (compare_scalars_simple(&tmp[j], &tmp[j-1], &ssup) != 0)
+ {
+ distvalues[i][ndistvalues[i]] = tmp[j];
+ ndistvalues[i] += 1;
+ }
+ }
+ }
+
+ pfree(tmp);
+ }
+
+ /*
+ * The initial bucket may contain NULL values, so we have to create
+ * buckets with NULL-only dimensions.
+ *
+ * FIXME We may need up to 2^ndims buckets - check that there are
+ * enough buckets (MVSTAT_HIST_MAX_BUCKETS >= 2^ndims).
+ */
+ create_null_buckets(histogram, 0, attrs, stats);
+
+ while (histogram->nbuckets < MVSTAT_HIST_MAX_BUCKETS)
+ {
+ MVBucket bucket = select_bucket_to_partition(histogram->nbuckets,
+ histogram->buckets);
+
+ /* no more buckets to partition */
+ if (bucket == NULL)
+ break;
+
+ histogram->buckets[histogram->nbuckets]
+ = partition_bucket(bucket, attrs, stats,
+ ndistvalues, distvalues);
+
+ histogram->nbuckets += 1;
+ }
+
+ /* finalize the frequencies etc. */
+ for (i = 0; i < histogram->nbuckets; i++)
+ {
+ HistogramBuild build_data
+ = ((HistogramBuild)histogram->buckets[i]->build_data);
+
+ /*
+ * The frequency has to be computed from the whole sample, in
+ * case some of the rows were used for MCV (and thus are missing
+ * from the histogram).
+ */
+ histogram->buckets[i]->ntuples
+ = (build_data->numrows * 1.0) / numrows_total;
+ }
+
+ return histogram;
+}
+
+/* fetch the histogram (as a bytea) from the pg_mv_statistic catalog */
+MVSerializedHistogram
+load_mv_histogram(Oid mvoid)
+{
+ bool isnull = false;
+ Datum histogram;
+
+#ifdef USE_ASSERT_CHECKING
+ Form_pg_mv_statistic mvstat;
+#endif
+
+ /* Prepare to scan pg_mv_statistic for entries having indrelid = this rel. */
+ HeapTuple htup = SearchSysCache1(MVSTATOID, ObjectIdGetDatum(mvoid));
+
+ if (! HeapTupleIsValid(htup))
+ return NULL;
+
+#ifdef USE_ASSERT_CHECKING
+ mvstat = (Form_pg_mv_statistic) GETSTRUCT(htup);
+ Assert(mvstat->hist_enabled && mvstat->hist_built);
+#endif
+
+ histogram = SysCacheGetAttr(MVSTATOID, htup,
+ Anum_pg_mv_statistic_stahist, &isnull);
+
+ Assert(!isnull);
+
+ ReleaseSysCache(htup);
+
+ return deserialize_mv_histogram(DatumGetByteaP(histogram));
+}
+
+/* print some basic info about the histogram */
+Datum
+pg_mv_stats_histogram_info(PG_FUNCTION_ARGS)
+{
+ bytea *data = PG_GETARG_BYTEA_P(0);
+ char *result;
+
+ MVSerializedHistogram hist = deserialize_mv_histogram(data);
+
+ result = palloc0(128);
+ snprintf(result, 128, "nbuckets=%d", hist->nbuckets);
+
+ PG_RETURN_TEXT_P(cstring_to_text(result));
+}
+
+
+/* used to pass context into bsearch() */
+static SortSupport ssup_private = NULL;
+
+/*
+ * Serialize the MV histogram into a bytea value. The basic algorithm is quite
+ * simple, and mostly mimincs the MCV serialization:
+ *
+ * (1) perform deduplication for each attribute (separately)
+ *
+ * (a) collect all (non-NULL) attribute values from all buckets
+ * (b) sort the data (using 'lt' from VacAttrStats)
+ * (c) remove duplicate values from the array
+ *
+ * (2) serialize the arrays into a bytea value
+ *
+ * (3) process all buckets
+ *
+ * (a) replace min/max values with indexes into the arrays
+ *
+ * Each attribute has to be processed separately, as we're mixing different
+ * datatypes, and we we need to use the right operators to compare/sort them.
+ * We're also mixing pass-by-value and pass-by-ref types, and so on.
+ *
+ *
+ * FIXME This probably leaks memory, or at least uses it inefficiently
+ * (many small palloc() calls instead of a large one).
+ *
+ * TODO Consider packing boolean flags (NULL) for each item into 'char'
+ * or a longer type (instead of using an array of bool items).
+ */
+bytea *
+serialize_mv_histogram(MVHistogram histogram, int2vector *attrs,
+ VacAttrStats **stats)
+{
+ int i = 0, j = 0;
+ Size total_length = 0;
+
+ bytea *output = NULL;
+ char *data = NULL;
+
+ int nbuckets = histogram->nbuckets;
+ int ndims = histogram->ndimensions;
+
+ /* allocated for serialized bucket data */
+ int bucketsize = BUCKET_SIZE(ndims);
+ char *bucket = palloc0(bucketsize);
+
+ /* values per dimension (and number of non-NULL values) */
+ Datum **values = (Datum**)palloc0(sizeof(Datum*) * ndims);
+ int *counts = (int*)palloc0(sizeof(int) * ndims);
+
+ /* info about dimensions (for deserialize) */
+ DimensionInfo * info
+ = (DimensionInfo *)palloc0(sizeof(DimensionInfo)*ndims);
+
+ /* sort support data */
+ SortSupport ssup = (SortSupport)palloc0(sizeof(SortSupportData)*ndims);
+
+ /* collect and deduplicate values for each dimension separately */
+ for (i = 0; i < ndims; i++)
+ {
+ int count;
+ StdAnalyzeData *tmp = (StdAnalyzeData *)stats[i]->extra_data;
+
+ /* keep important info about the data type */
+ info[i].typlen = stats[i]->attrtype->typlen;
+ info[i].typbyval = stats[i]->attrtype->typbyval;
+
+ /*
+ * Allocate space for all min/max values, including NULLs
+ * (we won't use them, but we don't know how many are there),
+ * and then collect all non-NULL values.
+ */
+ values[i] = (Datum*)palloc0(sizeof(Datum) * nbuckets * 2);
+
+ for (j = 0; j < histogram->nbuckets; j++)
+ {
+ /* skip buckets where this dimension is NULL-only */
+ if (! histogram->buckets[j]->nullsonly[i])
+ {
+ values[i][counts[i]] = histogram->buckets[j]->min[i];
+ counts[i] += 1;
+
+ values[i][counts[i]] = histogram->buckets[j]->max[i];
+ counts[i] += 1;
+ }
+ }
+
+ /* there are just NULL values in this dimension */
+ if (counts[i] == 0)
+ continue;
+
+ /* sort and deduplicate */
+ ssup[i].ssup_cxt = CurrentMemoryContext;
+ ssup[i].ssup_collation = DEFAULT_COLLATION_OID;
+ ssup[i].ssup_nulls_first = false;
+
+ PrepareSortSupportFromOrderingOp(tmp->ltopr, &ssup[i]);
+
+ qsort_arg(values[i], counts[i], sizeof(Datum),
+ compare_scalars_simple, &ssup[i]);
+
+ /*
+ * Walk through the array and eliminate duplicitate values, but
+ * keep the ordering (so that we can do bsearch later). We know
+ * there's at least 1 item, so we can skip the first element.
+ */
+ count = 1; /* number of deduplicated items */
+ for (j = 1; j < counts[i]; j++)
+ {
+ /* if it's different from the previous value, we need to keep it */
+ if (compare_datums_simple(values[i][j-1], values[i][j], &ssup[i]) != 0)
+ {
+ /* XXX: not needed if (count == j) */
+ values[i][count] = values[i][j];
+ count += 1;
+ }
+ }
+
+ /* make sure we fit into uint16 */
+ Assert(count <= UINT16_MAX);
+
+ /* keep info about the deduplicated count */
+ info[i].nvalues = count;
+
+ /* compute size of the serialized data */
+ if (info[i].typlen > 0)
+ /* byval or byref, but with fixed length (name, tid, ...) */
+ info[i].nbytes = info[i].nvalues * info[i].typlen;
+ else if (info[i].typlen == -1)
+ /* varlena, so just use VARSIZE_ANY */
+ for (j = 0; j < info[i].nvalues; j++)
+ info[i].nbytes += VARSIZE_ANY(values[i][j]);
+ else if (info[i].typlen == -2)
+ /* cstring, so simply strlen */
+ for (j = 0; j < info[i].nvalues; j++)
+ info[i].nbytes += strlen(DatumGetPointer(values[i][j]));
+ else
+ elog(ERROR, "unknown data type typbyval=%d typlen=%d",
+ info[i].typbyval, info[i].typlen);
+ }
+
+ /*
+ * Now we finally know how much space we'll need for the serialized
+ * histogram, as it contains these fields:
+ *
+ * - length (4B) for varlena
+ * - magic (4B)
+ * - type (4B)
+ * - ndimensions (4B)
+ * - nbuckets (4B)
+ * - info (ndim * sizeof(DimensionInfo)
+ * - arrays of values for each dimension
+ * - serialized buckets (nbuckets * bucketsize)
+ *
+ * So the 'header' size is 20B + ndim * sizeof(DimensionInfo) and
+ * then we'll place the data (and buckets).
+ */
+ total_length = (sizeof(int32) + offsetof(MVHistogramData, buckets)
+ + ndims * sizeof(DimensionInfo)
+ + nbuckets * bucketsize);
+
+ /* account for the deduplicated data */
+ for (i = 0; i < ndims; i++)
+ total_length += info[i].nbytes;
+
+ /* enforce arbitrary limit of 1MB */
+ if (total_length > (10 * 1024 * 1024))
+ elog(ERROR, "serialized histogram exceeds 10MB (%ld > %d)",
+ total_length, (10 * 1024 * 1024));
+
+ /* allocate space for the serialized histogram list, set header */
+ output = (bytea*)palloc0(total_length);
+ SET_VARSIZE(output, total_length);
+
+ /* we'll use 'data' to keep track of the place to write data */
+ data = VARDATA(output);
+
+ memcpy(data, histogram, offsetof(MVHistogramData, buckets));
+ data += offsetof(MVHistogramData, buckets);
+
+ memcpy(data, info, sizeof(DimensionInfo) * ndims);
+ data += sizeof(DimensionInfo) * ndims;
+
+ /* value array for each dimension */
+ for (i = 0; i < ndims; i++)
+ {
+#ifdef USE_ASSERT_CHECKING
+ char *tmp = data;
+#endif
+ for (j = 0; j < info[i].nvalues; j++)
+ {
+ if (info[i].typlen > 0)
+ {
+ /* pased by value or reference, but fixed length */
+ memcpy(data, &values[i][j], info[i].typlen);
+ data += info[i].typlen;
+ }
+ else if (info[i].typlen == -1)
+ {
+ /* varlena */
+ memcpy(data, DatumGetPointer(values[i][j]),
+ VARSIZE_ANY(values[i][j]));
+ data += VARSIZE_ANY(values[i][j]);
+ }
+ else if (info[i].typlen == -2)
+ {
+ /* cstring (don't forget the \0 terminator!) */
+ memcpy(data, DatumGetPointer(values[i][j]),
+ strlen(DatumGetPointer(values[i][j])) + 1);
+ data += strlen(DatumGetPointer(values[i][j])) + 1;
+ }
+ }
+ Assert((data - tmp) == info[i].nbytes);
+ }
+
+ /* and finally, the histogram buckets */
+ for (i = 0; i < nbuckets; i++)
+ {
+ /* don't write beyond the allocated space */
+ Assert(data <= (char*)output + total_length - bucketsize);
+
+ /* reset the values for each item */
+ memset(bucket, 0, bucketsize);
+
+ *BUCKET_NTUPLES(bucket) = histogram->buckets[i]->ntuples;
+
+ for (j = 0; j < ndims; j++)
+ {
+ /* do the lookup only for non-NULL values */
+ if (! histogram->buckets[i]->nullsonly[j])
+ {
+ uint16 idx;
+ Datum * v = NULL;
+ ssup_private = &ssup[j];
+
+ /* min boundary */
+ v = (Datum*)bsearch(&histogram->buckets[i]->min[j],
+ values[j], info[j].nvalues, sizeof(Datum),
+ bsearch_comparator);
+
+ if (v == NULL)
+ elog(ERROR, "value for dim %d not found in array", j);
+
+ /* compute index within the array */
+ idx = (v - values[j]);
+
+ Assert((idx >= 0) && (idx < info[j].nvalues));
+
+ BUCKET_MIN_INDEXES(bucket, ndims)[j] = idx;
+
+ /* max boundary */
+ v = (Datum*)bsearch(&histogram->buckets[i]->max[j],
+ values[j], info[j].nvalues, sizeof(Datum),
+ bsearch_comparator);
+
+ if (v == NULL)
+ elog(ERROR, "value for dim %d not found in array", j);
+
+ /* compute index within the array */
+ idx = (v - values[j]);
+
+ Assert((idx >= 0) && (idx < info[j].nvalues));
+
+ BUCKET_MAX_INDEXES(bucket, ndims)[j] = idx;
+ }
+ }
+
+ /* copy flags (nulls, min/max inclusive) */
+ memcpy(BUCKET_NULLS_ONLY(bucket, ndims),
+ histogram->buckets[i]->nullsonly, sizeof(bool) * ndims);
+
+ memcpy(BUCKET_MIN_INCL(bucket, ndims),
+ histogram->buckets[i]->min_inclusive, sizeof(bool) * ndims);
+
+ memcpy(BUCKET_MAX_INCL(bucket, ndims),
+ histogram->buckets[i]->max_inclusive, sizeof(bool) * ndims);
+
+ /* copy the item into the array */
+ memcpy(data, bucket, bucketsize);
+
+ data += bucketsize;
+ }
+
+ /* at this point we expect to match the total_length exactly */
+ Assert((data - (char*)output) == total_length);
+
+ /* FIXME free the values/counts arrays here */
+
+ return output;
+}
+
+/*
+ * Returns histogram in a partially-serialized form (keeps the boundary
+ * values deduplicated, so that it's possible to optimize the estimation
+ * part by caching function call results between buckets etc.).
+ */
+MVSerializedHistogram
+deserialize_mv_histogram(bytea * data)
+{
+ int i = 0, j = 0;
+
+ Size expected_size;
+ char *tmp = NULL;
+
+ MVSerializedHistogram histogram;
+ DimensionInfo *info;
+
+ int nbuckets;
+ int ndims;
+ int bucketsize;
+
+ /* temporary deserialization buffer */
+ int bufflen;
+ char *buff;
+ char *ptr;
+
+ if (data == NULL)
+ return NULL;
+
+ if (VARSIZE_ANY_EXHDR(data) < offsetof(MVSerializedHistogramData,buckets))
+ elog(ERROR, "invalid histogram size %ld (expected at least %ld)",
+ VARSIZE_ANY_EXHDR(data), offsetof(MVSerializedHistogramData,buckets));
+
+ /* read the histogram header */
+ histogram
+ = (MVSerializedHistogram)palloc(sizeof(MVSerializedHistogramData));
+
+ /* initialize pointer to the data part (skip the varlena header) */
+ tmp = VARDATA(data);
+
+ /* get the header and perform basic sanity checks */
+ memcpy(histogram, tmp, offsetof(MVSerializedHistogramData, buckets));
+ tmp += offsetof(MVSerializedHistogramData, buckets);
+
+ if (histogram->magic != MVSTAT_HIST_MAGIC)
+ elog(ERROR, "invalid histogram magic %d (expected %dd)",
+ histogram->magic, MVSTAT_HIST_MAGIC);
+
+ if (histogram->type != MVSTAT_HIST_TYPE_BASIC)
+ elog(ERROR, "invalid histogram type %d (expected %dd)",
+ histogram->type, MVSTAT_HIST_TYPE_BASIC);
+
+ nbuckets = histogram->nbuckets;
+ ndims = histogram->ndimensions;
+ bucketsize = BUCKET_SIZE(ndims);
+
+ Assert((nbuckets > 0) && (nbuckets <= MVSTAT_HIST_MAX_BUCKETS));
+ Assert((ndims >= 2) && (ndims <= MVSTATS_MAX_DIMENSIONS));
+
+ /*
+ * What size do we expect with those parameters (it's incomplete,
+ * as we yet have to count the array sizes (from DimensionInfo
+ * records).
+ */
+ expected_size = offsetof(MVSerializedHistogramData,buckets) +
+ ndims * sizeof(DimensionInfo) +
+ (nbuckets * bucketsize);
+
+ /* check that we have at least the DimensionInfo records */
+ if (VARSIZE_ANY_EXHDR(data) < expected_size)
+ elog(ERROR, "invalid histogram size %ld (expected %ld)",
+ VARSIZE_ANY_EXHDR(data), expected_size);
+
+ info = (DimensionInfo*)(tmp);
+ tmp += ndims * sizeof(DimensionInfo);
+
+ /* account for the value arrays */
+ for (i = 0; i < ndims; i++)
+ expected_size += info[i].nbytes;
+
+ if (VARSIZE_ANY_EXHDR(data) != expected_size)
+ elog(ERROR, "invalid histogram size %ld (expected %ld)",
+ VARSIZE_ANY_EXHDR(data), expected_size);
+
+ /* looks OK - not corrupted or something */
+
+ /* now let's allocate a single buffer for all the values and counts */
+
+ bufflen = (sizeof(int) + sizeof(Datum*)) * ndims;
+ for (i = 0; i < ndims; i++)
+ {
+ /* don't allocate space for byval types, matching Datum */
+ if (! (info[i].typbyval && (info[i].typlen == sizeof(Datum))))
+ bufflen += (sizeof(Datum) * info[i].nvalues);
+ }
+
+ /* also, include space for the result, tracking the buckets */
+ bufflen += nbuckets * (
+ sizeof(MVSerializedBucket) + /* bucket pointer */
+ sizeof(MVSerializedBucketData)); /* bucket data */
+
+ buff = palloc0(bufflen);
+ ptr = buff;
+
+ histogram->nvalues = (int*)ptr;
+ ptr += (sizeof(int) * ndims);
+
+ histogram->values = (Datum**)ptr;
+ ptr += (sizeof(Datum*) * ndims);
+
+ /*
+ * FIXME This uses pointers to the original data array (the types
+ * not passed by value), so when someone frees the memory,
+ * e.g. by doing something like this:
+ *
+ * bytea * data = ... fetch the data from catalog ...
+ * MCVList mcvlist = deserialize_mcv_list(data);
+ * pfree(data);
+ *
+ * then 'mcvlist' references the freed memory. This needs to
+ * copy the pieces.
+ *
+ * TODO same as in MCV deserialization / consider moving to common.c
+ */
+ for (i = 0; i < ndims; i++)
+ {
+ histogram->nvalues[i] = info[i].nvalues;
+
+ if (info[i].typbyval && info[i].typlen == sizeof(Datum))
+ {
+ /* passed by value / Datum - simply reuse the array */
+ histogram->values[i] = (Datum*)tmp;
+ tmp += info[i].nbytes;
+ }
+ else
+ {
+ /* all the varlena data need a chunk from the buffer */
+ histogram->values[i] = (Datum*)ptr;
+ ptr += (sizeof(Datum) * info[i].nvalues);
+
+ if (info[i].typbyval)
+ {
+ /* pased by value, but smaller than Datum */
+ for (j = 0; j < info[i].nvalues; j++)
+ {
+ /* just point into the array */
+ memcpy(&histogram->values[i][j], tmp, info[i].typlen);
+ tmp += info[i].typlen;
+ }
+ }
+ else if (info[i].typlen > 0)
+ {
+ /* pased by reference, but fixed length (name, tid, ...) */
+ for (j = 0; j < info[i].nvalues; j++)
+ {
+ /* just point into the array */
+ histogram->values[i][j] = PointerGetDatum(tmp);
+ tmp += info[i].typlen;
+ }
+ }
+ else if (info[i].typlen == -1)
+ {
+ /* varlena */
+ for (j = 0; j < info[i].nvalues; j++)
+ {
+ /* just point into the array */
+ histogram->values[i][j] = PointerGetDatum(tmp);
+ tmp += VARSIZE_ANY(tmp);
+ }
+ }
+ else if (info[i].typlen == -2)
+ {
+ /* cstring */
+ for (j = 0; j < info[i].nvalues; j++)
+ {
+ /* just point into the array */
+ histogram->values[i][j] = PointerGetDatum(tmp);
+ tmp += (strlen(tmp) + 1); /* don't forget the \0 */
+ }
+ }
+ }
+ }
+
+ histogram->buckets = (MVSerializedBucket*)ptr;
+ ptr += (sizeof(MVSerializedBucket) * nbuckets);
+
+ for (i = 0; i < nbuckets; i++)
+ {
+ MVSerializedBucket bucket = (MVSerializedBucket)ptr;
+ ptr += sizeof(MVSerializedBucketData);
+
+ bucket->ntuples = *BUCKET_NTUPLES(tmp);
+ bucket->nullsonly = BUCKET_NULLS_ONLY(tmp, ndims);
+ bucket->min_inclusive = BUCKET_MIN_INCL(tmp, ndims);
+ bucket->max_inclusive = BUCKET_MAX_INCL(tmp, ndims);
+
+ bucket->min = BUCKET_MIN_INDEXES(tmp, ndims);
+ bucket->max = BUCKET_MAX_INDEXES(tmp, ndims);
+
+ histogram->buckets[i] = bucket;
+
+ Assert(tmp <= (char*)data + VARSIZE_ANY(data));
+
+ tmp += bucketsize;
+ }
+
+ /* at this point we expect to match the total_length exactly */
+ Assert((tmp - VARDATA(data)) == expected_size);
+
+ /* we should exhaust the output buffer exactly */
+ Assert((ptr - buff) == bufflen);
+
+ return histogram;
+}
+
+/*
+ * Build the initial bucket, which will be then split into smaller ones.
+ */
+static MVBucket
+create_initial_mv_bucket(int numrows, HeapTuple *rows, int2vector *attrs,
+ VacAttrStats **stats)
+{
+ int i;
+ int numattrs = attrs->dim1;
+ HistogramBuild data = NULL;
+
+ /* TODO allocate bucket as a single piece, including all the fields. */
+ MVBucket bucket = (MVBucket)palloc0(sizeof(MVBucketData));
+
+ Assert(numrows > 0);
+ Assert(rows != NULL);
+ Assert((numattrs >= 2) && (numattrs <= MVSTATS_MAX_DIMENSIONS));
+
+ /* allocate the per-dimension arrays */
+
+ /* flags for null-only dimensions */
+ bucket->nullsonly = (bool*)palloc0(numattrs * sizeof(bool));
+
+ /* inclusiveness boundaries - lower/upper bounds */
+ bucket->min_inclusive = (bool*)palloc0(numattrs * sizeof(bool));
+ bucket->max_inclusive = (bool*)palloc0(numattrs * sizeof(bool));
+
+ /* lower/upper boundaries */
+ bucket->min = (Datum*)palloc0(numattrs * sizeof(Datum));
+ bucket->max = (Datum*)palloc0(numattrs * sizeof(Datum));
+
+ /* build-data */
+ data = (HistogramBuild)palloc0(sizeof(HistogramBuildData));
+
+ /* number of distinct values (per dimension) */
+ data->ndistincts = (uint32*)palloc0(numattrs * sizeof(uint32));
+
+ /* all the sample rows fall into the initial bucket */
+ data->numrows = numrows;
+ data->rows = rows;
+
+ bucket->build_data = data;
+
+ /*
+ * Update the number of ndistinct combinations in the bucket (which
+ * we use when selecting bucket to partition), and then number of
+ * distinct values for each partition (which we use when choosing
+ * which dimension to split).
+ */
+ update_bucket_ndistinct(bucket, attrs, stats);
+
+ /* Update ndistinct (and also set min/max) for all dimensions. */
+ for (i = 0; i < numattrs; i++)
+ update_dimension_ndistinct(bucket, i, attrs, stats, true);
+
+ return bucket;
+}
+
+/*
+ * Choose the bucket to partition next.
+ *
+ * The current criteria is rather simple, chosen so that the algorithm
+ * produces buckets with about equal frequency and regular size. We
+ * select the bucket with the highest number of distinct values, and
+ * then split it by the longest dimension.
+ *
+ * The distinct values are uniformly mapped to [0,1] interval, and this
+ * is used to compute length of the value range.
+ *
+ * NOTE: This is not the same array used for deduplication, as this
+ * contains values for all the tuples from the sample, not just
+ * the boundary values.
+ *
+ * Returns either pointer to the bucket selected to be partitioned,
+ * or NULL if there are no buckets that may be split (i.e. all buckets
+ * contain a single distinct value).
+ *
+ * TODO Consider other partitioning criteria (v-optimal, maxdiff etc.).
+ * For example use the "bucket volume" (product of dimension
+ * lengths) to select the bucket.
+ *
+ * We need buckets containing about the same number of tuples (so
+ * about the same frequency), as that limits the error when we
+ * match the bucket partially (in that case use 1/2 the bucket).
+ *
+ * We also need buckets with "regular" size, i.e. not "narrow" in
+ * some dimensions and "wide" in the others, because that makes
+ * partial matches more likely and increases the estimation error,
+ * especially when the clauses match many buckets partially. This
+ * is especially serious for OR-clauses, because in that case any
+ * of them may add the bucket as a (partial) match. With AND-clauses
+ * all the clauses have to match the bucket, which makes this issue
+ * somewhat less pressing.
+ *
+ * For example this table:
+ *
+ * CREATE TABLE t AS SELECT i AS a, i AS b
+ * FROM generate_series(1,1000000) s(i);
+ * ALTER TABLE t ADD STATISTICS (histogram) ON (a,b);
+ * ANALYZE t;
+ *
+ * It's a very specific (and perhaps artificial) example, because
+ * every bucket always has exactly the same number of distinct
+ * values in all dimensions, which makes the partitioning tricky.
+ *
+ * Then:
+ *
+ * SELECT * FROM t WHERE a < 10 AND b < 10;
+ *
+ * is estimated to return ~120 rows, while in reality it returns 9.
+ *
+ * QUERY PLAN
+ * ----------------------------------------------------------------
+ * Seq Scan on t (cost=0.00..19425.00 rows=117 width=8)
+ * (actual time=0.185..270.774 rows=9 loops=1)
+ * Filter: ((a < 10) AND (b < 10))
+ * Rows Removed by Filter: 999991
+ *
+ * while the query using OR clauses is estimated like this:
+ *
+ * QUERY PLAN
+ * ----------------------------------------------------------------
+ * Seq Scan on t (cost=0.00..19425.00 rows=8100 width=8)
+ * (actual time=0.118..189.919 rows=9 loops=1)
+ * Filter: ((a < 10) OR (b < 10))
+ * Rows Removed by Filter: 999991
+ *
+ * which is clearly much worse. This happens because the histogram
+ * contains buckets like this:
+ *
+ * bucket 592 [3 30310] [30134 30593] => [0.000233]
+ *
+ * i.e. the length of "a" dimension is (30310-3)=30307, while the
+ * length of "b" is (30593-30134)=459. So the "b" dimension is much
+ * narrower than "a". Of course, there are buckets where "b" is the
+ * wider dimension.
+ *
+ * This is partially mitigated by selecting the "longest" dimension
+ * in partition_bucket() but that only happens after we already
+ * selected the bucket. So if we never select the bucket, we can't
+ * really fix it there.
+ *
+ * The other reason why this particular example behaves so poorly
+ * is due to the way we split the partition in partition_bucket().
+ * Currently we attempt to divide the bucket into two parts with
+ * the same number of sampled tuples (frequency), but that does not
+ * work well when all the tuples are squashed on one end of the
+ * bucket (e.g. exactly at the diagonal, as a=b). In that case we
+ * split the bucket into a tiny bucket on the diagonal, and a huge
+ * remaining part of the bucket, which is still going to be narrow
+ * and we're unlikely to fix that.
+ *
+ * So perhaps we need two partitioning strategies - one aiming to
+ * split buckets with high frequency (number of sampled rows), the
+ * other aiming to split "large" buckets. And alternating between
+ * them, somehow.
+ *
+ * TODO Allowing the bucket to degenerate to a single combination of
+ * values makes it rather strange MCV list. Maybe we should use
+ * higher lower boundary, or maybe make the selection criteria
+ * more complex (e.g. consider number of rows in the bucket, etc.).
+ *
+ * That however is different from buckets 'degenerated' only for
+ * some dimensions (e.g. half of them), which is perfectly
+ * appropriate for statistics on a combination of low and high
+ * cardinality columns.
+ *
+ * TODO Consider using similar lower boundary for row count as for simple
+ * histograms, i.e. 300 tuples per bucket.
+ */
+static MVBucket
+select_bucket_to_partition(int nbuckets, MVBucket * buckets)
+{
+ int i;
+ int numrows = 0;
+ MVBucket bucket = NULL;
+
+ for (i = 0; i < nbuckets; i++)
+ {
+ HistogramBuild data = (HistogramBuild)buckets[i]->build_data;
+ /* if the number of rows is higher, use this bucket */
+ if ((data->ndistinct > 2) &&
+ (data->numrows > numrows) &&
+ (data->numrows >= MIN_BUCKET_ROWS)) {
+ bucket = buckets[i];
+ numrows = data->numrows;
+ }
+ }
+
+ /* may be NULL if there are not buckets with (ndistinct>1) */
+ return bucket;
+}
+
+/*
+ * A simple bucket partitioning implementation - we choose the longest
+ * bucket dimension, measured using the array of distinct values built
+ * at the very beginning of the build.
+ *
+ * We map all the distinct values to a [0,1] interval, uniformly
+ * distributed, and then use this to measure length. It's essentially
+ * a number of distinct values within the range, normalized to [0,1].
+ *
+ * Then we choose a 'middle' value splitting the bucket into two parts
+ * with roughly the same frequency.
+ *
+ * This splits the bucket by tweaking the existing one, and returning
+ * the new bucket (essentially shrinking the existing one in-place and
+ * returning the other "half" as a new bucket). The caller is responsible
+ * for adding the new bucket into the list of buckets.
+ *
+ * There are multiple histogram options, centered around the partitioning
+ * criteria, specifying both how to choose a bucket and the dimension
+ * most in need of a split. For a nice summary and general overview, see
+ * "rK-Hist : an R-Tree based histogram for multi-dimensional selectivity
+ * estimation" thesis by J. A. Lopez, Concordia University, p.34-37 (and
+ * possibly p. 32-34 for explanation of the terms).
+ *
+ * TODO It requires care to prevent splitting only one dimension and not
+ * splitting another one at all (which might happen easily in case
+ * of strongly dependent columns - e.g. y=x). The current algorithm
+ * minimizes this, but may still happen for perfectly dependent
+ * examples (when all the dimensions have equal length, the first
+ * one will be selected).
+ *
+ * TODO Should probably consider statistics target for the columns (e.g.
+ * to split dimensions with higher statistics target more frequently).
+ */
+static MVBucket
+partition_bucket(MVBucket bucket, int2vector *attrs,
+ VacAttrStats **stats,
+ int *ndistvalues, Datum **distvalues)
+{
+ int i;
+ int dimension;
+ int numattrs = attrs->dim1;
+
+ Datum split_value;
+ MVBucket new_bucket;
+ HistogramBuild new_data;
+
+ /* needed for sort, when looking for the split value */
+ bool isNull;
+ int nvalues = 0;
+ HistogramBuild data = (HistogramBuild)bucket->build_data;
+ StdAnalyzeData * mystats = NULL;
+ ScalarItem * values = (ScalarItem*)palloc0(data->numrows * sizeof(ScalarItem));
+ SortSupportData ssup;
+
+ /* looking for the split value */
+ // int ndistinct = 1; /* number of distinct values below current value */
+ int nrows = 1; /* number of rows below current value */
+ double delta;
+
+ /* needed when splitting the values */
+ HeapTuple * oldrows = data->rows;
+ int oldnrows = data->numrows;
+
+ /*
+ * We can't split buckets with a single distinct value (this also
+ * disqualifies NULL-only dimensions). Also, there has to be multiple
+ * sample rows (otherwise, how could there be more distinct values).
+ */
+ Assert(data->ndistinct > 1);
+ Assert(data->numrows > 1);
+ Assert((numattrs >= 2) && (numattrs <= MVSTATS_MAX_DIMENSIONS));
+
+ /*
+ * Look for the next dimension to split.
+ */
+ delta = 0.0;
+ dimension = -1;
+
+ for (i = 0; i < numattrs; i++)
+ {
+ Datum *a, *b;
+
+ mystats = (StdAnalyzeData *) stats[i]->extra_data;
+
+ /* initialize sort support, etc. */
+ memset(&ssup, 0, sizeof(ssup));
+ ssup.ssup_cxt = CurrentMemoryContext;
+
+ /* We always use the default collation for statistics */
+ ssup.ssup_collation = DEFAULT_COLLATION_OID;
+ ssup.ssup_nulls_first = false;
+
+ PrepareSortSupportFromOrderingOp(mystats->ltopr, &ssup);
+
+ /* can't split NULL-only dimension */
+ if (bucket->nullsonly[i])
+ continue;
+
+ /* can't split dimension with a single ndistinct value */
+ if (data->ndistincts[i] <= 1)
+ continue;
+
+ /* sort support for the bsearch_comparator */
+ ssup_private = &ssup;
+
+ /* search for min boundary in the distinct list */
+ a = (Datum*)bsearch(&bucket->min[i],
+ distvalues[i], ndistvalues[i],
+ sizeof(Datum), bsearch_comparator);
+
+ b = (Datum*)bsearch(&bucket->max[i],
+ distvalues[i], ndistvalues[i],
+ sizeof(Datum), bsearch_comparator);
+
+ /* if this dimension is 'larger' then partition by it */
+ if (((b-a)*1.0 / ndistvalues[i]) > delta)
+ {
+ delta = ((b-a)*1.0 / ndistvalues[i]);
+ dimension = i;
+ }
+ }
+
+ /*
+ * If we haven't found a dimension here, we've done something
+ * wrong in select_bucket_to_partition.
+ */
+ Assert(dimension != -1);
+
+ /*
+ * Walk through the selected dimension, collect and sort the values
+ * and then choose the value to use as the new boundary.
+ */
+ mystats = (StdAnalyzeData *) stats[dimension]->extra_data;
+
+ /* initialize sort support, etc. */
+ memset(&ssup, 0, sizeof(ssup));
+ ssup.ssup_cxt = CurrentMemoryContext;
+
+ /* We always use the default collation for statistics */
+ ssup.ssup_collation = DEFAULT_COLLATION_OID;
+ ssup.ssup_nulls_first = false;
+
+ PrepareSortSupportFromOrderingOp(mystats->ltopr, &ssup);
+
+ for (i = 0; i < data->numrows; i++)
+ {
+ /* remember the index of the sample row, to make the partitioning simpler */
+ values[nvalues].value = heap_getattr(data->rows[i], attrs->values[dimension],
+ stats[dimension]->tupDesc, &isNull);
+ values[nvalues].tupno = i;
+
+ /* no NULL values allowed here (we don't do splits by null-only dimensions) */
+ Assert(!isNull);
+
+ nvalues++;
+ }
+
+ /* sort the array of values */
+ qsort_arg((void *) values, nvalues, sizeof(ScalarItem),
+ compare_scalars_partition, (void *) &ssup);
+
+ /*
+ * We know there are bucket->ndistincts[dimension] distinct values
+ * in this dimension, and we want to split this into half, so walk
+ * through the array and stop once we see (ndistinct/2) values.
+ *
+ * We always choose the "next" value, i.e. (n/2+1)-th distinct value,
+ * and use it as an exclusive upper boundary (and inclusive lower
+ * boundary).
+ *
+ * TODO Maybe we should use "average" of the two middle distinct
+ * values (at least for even distinct counts), but that would
+ * require being able to do an average (which does not work
+ * for non-arithmetic types).
+ *
+ * TODO Another option is to look for a split that'd give about
+ * 50% tuples (not distinct values) in each partition. That
+ * might work better when there are a few very frequent
+ * values, and many rare ones.
+ */
+ delta = fabs(data->numrows);
+ split_value = values[0].value;
+
+ for (i = 1; i < data->numrows; i++)
+ {
+ if (values[i].value != values[i-1].value)
+ {
+ /* are we closer to splitting the bucket in half? */
+ if (fabs(i - data->numrows/2.0) < delta)
+ {
+ /* let's assume we'll use this value for the split */
+ split_value = values[i].value;
+ delta = fabs(i - data->numrows/2.0);
+ nrows = i;
+ }
+ }
+ }
+
+ Assert(nrows > 0);
+ Assert(nrows < data->numrows);
+
+ /* create the new bucket as a (incomplete) copy of the one being partitioned. */
+ new_bucket = copy_mv_bucket(bucket, numattrs);
+ new_data = (HistogramBuild)new_bucket->build_data;
+
+ /*
+ * Do the actual split of the chosen dimension, using the split value as the
+ * upper bound for the existing bucket, and lower bound for the new one.
+ */
+ bucket->max[dimension] = split_value;
+ new_bucket->min[dimension] = split_value;
+
+ bucket->max_inclusive[dimension] = false;
+ new_bucket->max_inclusive[dimension] = true;
+
+ /*
+ * Redistribute the sample tuples using the 'ScalarItem->tupno'
+ * index. We know 'nrows' rows should remain in the original
+ * bucket and the rest goes to the new one.
+ */
+
+ data->rows = (HeapTuple*)palloc0(nrows * sizeof(HeapTuple));
+ new_data->rows = (HeapTuple*)palloc0((oldnrows - nrows) * sizeof(HeapTuple));
+
+ data->numrows = nrows;
+ new_data->numrows = (oldnrows - nrows);
+
+ /*
+ * The first nrows should go to the first bucket, the rest should
+ * go to the new one. Use the tupno field to get the actual HeapTuple
+ * row from the original array of sample rows.
+ */
+ for (i = 0; i < nrows; i++)
+ memcpy(&data->rows[i], &oldrows[values[i].tupno], sizeof(HeapTuple));
+
+ for (i = nrows; i < oldnrows; i++)
+ memcpy(&new_data->rows[i-nrows], &oldrows[values[i].tupno], sizeof(HeapTuple));
+
+ /* update ndistinct values for the buckets (total and per dimension) */
+ update_bucket_ndistinct(bucket, attrs, stats);
+ update_bucket_ndistinct(new_bucket, attrs, stats);
+
+ /*
+ * TODO We don't need to do this for the dimension we used for split,
+ * because we know how many distinct values went to each partition.
+ */
+ for (i = 0; i < numattrs; i++)
+ {
+ update_dimension_ndistinct(bucket, i, attrs, stats, false);
+ update_dimension_ndistinct(new_bucket, i, attrs, stats, false);
+ }
+
+ pfree(oldrows);
+ pfree(values);
+
+ return new_bucket;
+}
+
+/*
+ * Copy a histogram bucket. The copy does not include the build-time
+ * data, i.e. sampled rows etc.
+ */
+static MVBucket
+copy_mv_bucket(MVBucket bucket, uint32 ndimensions)
+{
+ /* TODO allocate as a single piece (including all the fields) */
+ MVBucket new_bucket = (MVBucket)palloc0(sizeof(MVBucketData));
+ HistogramBuild data = (HistogramBuild)palloc0(sizeof(HistogramBuildData));
+
+ /* Copy only the attributes that will stay the same after the split, and
+ * we'll recompute the rest after the split. */
+
+ /* allocate the per-dimension arrays */
+ new_bucket->nullsonly = (bool*)palloc0(ndimensions * sizeof(bool));
+
+ /* inclusiveness boundaries - lower/upper bounds */
+ new_bucket->min_inclusive = (bool*)palloc0(ndimensions * sizeof(bool));
+ new_bucket->max_inclusive = (bool*)palloc0(ndimensions * sizeof(bool));
+
+ /* lower/upper boundaries */
+ new_bucket->min = (Datum*)palloc0(ndimensions * sizeof(Datum));
+ new_bucket->max = (Datum*)palloc0(ndimensions * sizeof(Datum));
+
+ /* copy data */
+ memcpy(new_bucket->nullsonly, bucket->nullsonly, ndimensions * sizeof(bool));
+
+ memcpy(new_bucket->min_inclusive, bucket->min_inclusive, ndimensions*sizeof(bool));
+ memcpy(new_bucket->min, bucket->min, ndimensions*sizeof(Datum));
+
+ memcpy(new_bucket->max_inclusive, bucket->max_inclusive, ndimensions*sizeof(bool));
+ memcpy(new_bucket->max, bucket->max, ndimensions*sizeof(Datum));
+
+ /* allocate and copy the interesting part of the build data */
+ data->ndistincts = (uint32*)palloc0(ndimensions * sizeof(uint32));
+
+ new_bucket->build_data = data;
+
+ return new_bucket;
+}
+
+/*
+ * Counts the number of distinct values in the bucket. This just copies
+ * the Datum values into a simple array, and sorts them using memcmp-based
+ * comparator. That means it only works for pass-by-value data types
+ * (assuming they don't use collations etc.)
+ *
+ * TODO This might evaluate and store the distinct counts for all
+ * possible attribute combinations. The assumption is this might be
+ * useful for estimating things like GROUP BY cardinalities (e.g.
+ * in cases when some buckets contain a lot of low-frequency
+ * combinations, and other buckets contain few high-frequency ones).
+ *
+ * But it's unclear whether it's worth the price. Computing this
+ * is actually quite cheap, because it may be evaluated at the very
+ * end, when the buckets are rather small (so sorting it in 2^N ways
+ * is not a big deal). Assuming the partitioning algorithm does not
+ * use these values to do the decisions, of course (the current
+ * algorithm does not).
+ *
+ * The overhead with storing, fetching and parsing the data is more
+ * concerning - adding 2^N values per bucket (even if it's just
+ * a 1B or 2B value) would significantly bloat the histogram, and
+ * thus the impact on optimizer. Which is not really desirable.
+ *
+ * TODO This only updates the ndistinct for the sample (or bucket), but
+ * we eventually need an estimate of the total number of distinct
+ * values in the dataset. It's possible to either use the current
+ * 1D approach (i.e., if it's more than 10% of the sample, assume
+ * it's proportional to the number of rows). Or it's possible to
+ * implement the estimator suggested in the article, supposedly
+ * giving 'optimal' estimates (w.r.t. probability of error).
+ */
+static void
+update_bucket_ndistinct(MVBucket bucket, int2vector *attrs, VacAttrStats ** stats)
+{
+ int i, j;
+ int numattrs = attrs->dim1;
+
+ HistogramBuild data = (HistogramBuild)bucket->build_data;
+ int numrows = data->numrows;
+
+ MultiSortSupport mss = multi_sort_init(numattrs);
+
+ /*
+ * We could collect this while walking through all the attributes
+ * above (this way we have to call heap_getattr twice).
+ */
+ SortItem *items = (SortItem*)palloc0(numrows * sizeof(SortItem));
+ Datum *values = (Datum*)palloc0(numrows * sizeof(Datum) * numattrs);
+ bool *isnull = (bool*)palloc0(numrows * sizeof(bool) * numattrs);
+
+ for (i = 0; i < numrows; i++)
+ {
+ items[i].values = &values[i * numattrs];
+ items[i].isnull = &isnull[i * numattrs];
+ }
+
+ /* prepare the sort function for the first dimension */
+ for (i = 0; i < numattrs; i++)
+ multi_sort_add_dimension(mss, i, i, stats);
+
+ /* collect the values */
+ for (i = 0; i < numrows; i++)
+ for (j = 0; j < numattrs; j++)
+ items[i].values[j]
+ = heap_getattr(data->rows[i], attrs->values[j],
+ stats[j]->tupDesc, &items[i].isnull[j]);
+
+ qsort_arg((void *) items, numrows, sizeof(SortItem),
+ multi_sort_compare, mss);
+
+ data->ndistinct = 1;
+
+ for (i = 1; i < numrows; i++)
+ if (multi_sort_compare(&items[i], &items[i-1], mss) != 0)
+ data->ndistinct += 1;
+
+ pfree(items);
+ pfree(values);
+ pfree(isnull);
+}
+
+/*
+ * Count distinct values per bucket dimension.
+ */
+static void
+update_dimension_ndistinct(MVBucket bucket, int dimension, int2vector *attrs,
+ VacAttrStats ** stats, bool update_boundaries)
+{
+ int j;
+ int nvalues = 0;
+ bool isNull;
+ HistogramBuild data = (HistogramBuild)bucket->build_data;
+ Datum * values = (Datum*)palloc0(data->numrows * sizeof(Datum));
+ SortSupportData ssup;
+
+ StdAnalyzeData * mystats = (StdAnalyzeData *) stats[dimension]->extra_data;
+
+ /* we may already know this is a NULL-only dimension */
+ if (bucket->nullsonly[dimension])
+ data->ndistincts[dimension] = 1;
+
+ memset(&ssup, 0, sizeof(ssup));
+ ssup.ssup_cxt = CurrentMemoryContext;
+
+ /* We always use the default collation for statistics */
+ ssup.ssup_collation = DEFAULT_COLLATION_OID;
+ ssup.ssup_nulls_first = false;
+
+ PrepareSortSupportFromOrderingOp(mystats->ltopr, &ssup);
+
+ for (j = 0; j < data->numrows; j++)
+ {
+ values[nvalues] = heap_getattr(data->rows[j], attrs->values[dimension],
+ stats[dimension]->tupDesc, &isNull);
+
+ /* ignore NULL values */
+ if (! isNull)
+ nvalues++;
+ }
+
+ /* there's always at least 1 distinct value (may be NULL) */
+ data->ndistincts[dimension] = 1;
+
+ /* if there are only NULL values in the column, mark it so and continue
+ * with the next one */
+ if (nvalues == 0)
+ {
+ pfree(values);
+ bucket->nullsonly[dimension] = true;
+ return;
+ }
+
+ /* sort the array (pass-by-value datum */
+ qsort_arg((void *) values, nvalues, sizeof(Datum),
+ compare_scalars_simple, (void *) &ssup);
+
+ /*
+ * Update min/max boundaries to the smallest bounding box. Generally, this
+ * needs to be done only when constructing the initial bucket.
+ */
+ if (update_boundaries)
+ {
+ /* store the min/max values */
+ bucket->min[dimension] = values[0];
+ bucket->min_inclusive[dimension] = true;
+
+ bucket->max[dimension] = values[nvalues-1];
+ bucket->max_inclusive[dimension] = true;
+ }
+
+ /*
+ * Walk through the array and count distinct values by comparing
+ * succeeding values.
+ *
+ * FIXME This only works for pass-by-value types (i.e. not VARCHARs
+ * etc.). Although thanks to the deduplication it might work
+ * even for those types (equal values will get the same item
+ * in the deduplicated array).
+ */
+ for (j = 1; j < nvalues; j++) {
+ if (values[j] != values[j-1])
+ data->ndistincts[dimension] += 1;
+ }
+
+ pfree(values);
+}
+
+/*
+ * A properly built histogram must not contain buckets mixing NULL and
+ * non-NULL values in a single dimension. Each dimension may either be
+ * marked as 'nulls only', and thus containing only NULL values, or
+ * it must not contain any NULL values.
+ *
+ * Therefore, if the sample contains NULL values in any of the columns,
+ * it's necessary to build those NULL-buckets. This is done in an
+ * iterative way using this algorithm, operating on a single bucket:
+ *
+ * (1) Check that all dimensions are well-formed (not mixing NULL
+ * and non-NULL values).
+ *
+ * (2) If all dimensions are well-formed, terminate.
+ *
+ * (3) If the dimension contains only NULL values, but is not
+ * marked as NULL-only, mark it as NULL-only and run the
+ * algorithm again (on this bucket).
+ *
+ * (4) If the dimension mixes NULL and non-NULL values, split the
+ * bucket into two parts - one with NULL values, one with
+ * non-NULL values (replacing the current one). Then run
+ * the algorithm on both buckets.
+ *
+ * This is executed in a recursive manner, but the number of executions
+ * should be quite low - limited by the number of NULL-buckets. Also,
+ * in each branch the number of nested calls is limited by the number
+ * of dimensions (attributes) of the histogram.
+ *
+ * At the end, there should be buckets with no mixed dimensions. The
+ * number of buckets produced by this algorithm is rather limited - with
+ * N dimensions, there may be only 2^N such buckets (each dimension may
+ * be either NULL or non-NULL). So with 8 dimensions (current value of
+ * MVSTATS_MAX_DIMENSIONS) there may be only 256 such buckets.
+ *
+ * After this, a 'regular' bucket-split algorithm shall run, further
+ * optimizing the histogram.
+ */
+static void
+create_null_buckets(MVHistogram histogram, int bucket_idx,
+ int2vector *attrs, VacAttrStats ** stats)
+{
+ int i, j;
+ int null_dim = -1;
+ int null_count = 0;
+ bool null_found = false;
+ MVBucket bucket, null_bucket;
+ int null_idx, curr_idx;
+ HistogramBuild data, null_data;
+
+ /* remember original values from the bucket */
+ int numrows;
+ HeapTuple *oldrows = NULL;
+
+ Assert(bucket_idx < histogram->nbuckets);
+ Assert(histogram->ndimensions == attrs->dim1);
+
+ bucket = histogram->buckets[bucket_idx];
+ data = (HistogramBuild)bucket->build_data;
+
+ numrows = data->numrows;
+ oldrows = data->rows;
+
+ /*
+ * Walk through all rows / dimensions, and stop once we find NULL
+ * in a dimension not yet marked as NULL-only.
+ */
+ for (i = 0; i < data->numrows; i++)
+ {
+ /*
+ * FIXME We don't need to start from the first attribute
+ * here - we can start from the last known dimension.
+ */
+ for (j = 0; j < histogram->ndimensions; j++)
+ {
+ /* Is this a NULL-only dimension? If yes, skip. */
+ if (bucket->nullsonly[j])
+ continue;
+
+ /* found a NULL in that dimension? */
+ if (heap_attisnull(data->rows[i], attrs->values[j]))
+ {
+ null_found = true;
+ null_dim = j;
+ break;
+ }
+ }
+
+ /* terminate if we found attribute with NULL values */
+ if (null_found)
+ break;
+ }
+
+ /* no regular dimension contains NULL values => we're done */
+ if (! null_found)
+ return;
+
+ /* walk through the rows again, count NULL values in 'null_dim' */
+ for (i = 0; i < data->numrows; i++)
+ {
+ if (heap_attisnull(data->rows[i], attrs->values[null_dim]))
+ null_count += 1;
+ }
+
+ Assert(null_count <= data->numrows);
+
+ /*
+ * If (null_count == numrows) the dimension already is NULL-only,
+ * but is not yet marked like that. It's enough to mark it and
+ * repeat the process recursively (until we run out of dimensions).
+ */
+ if (null_count == data->numrows)
+ {
+ bucket->nullsonly[null_dim] = true;
+ create_null_buckets(histogram, bucket_idx, attrs, stats);
+ return;
+ }
+
+ /*
+ * We have to split the bucket into two - one with NULL values in
+ * the dimension, one with non-NULL values. We don't need to sort
+ * the data or anything, but otherwise it's similar to what's done
+ * in partition_bucket().
+ */
+
+ /* create bucket with NULL-only dimension 'dim' */
+ null_bucket = copy_mv_bucket(bucket, histogram->ndimensions);
+ null_data = (HistogramBuild)null_bucket->build_data;
+
+ /* remember the current array info */
+ oldrows = data->rows;
+ numrows = data->numrows;
+
+ /* we'll keep non-NULL values in the current bucket */
+ data->numrows = (numrows - null_count);
+ data->rows
+ = (HeapTuple*)palloc0(data->numrows * sizeof(HeapTuple));
+
+ /* and the NULL values will go to the new one */
+ null_data->numrows = null_count;
+ null_data->rows
+ = (HeapTuple*)palloc0(null_data->numrows * sizeof(HeapTuple));
+
+ /* mark the dimension as NULL-only (in the new bucket) */
+ null_bucket->nullsonly[null_dim] = true;
+
+ /* walk through the sample rows and distribute them accordingly */
+ null_idx = 0;
+ curr_idx = 0;
+ for (i = 0; i < numrows; i++)
+ {
+ if (heap_attisnull(oldrows[i], attrs->values[null_dim]))
+ /* NULL => copy to the new bucket */
+ memcpy(&null_data->rows[null_idx++], &oldrows[i],
+ sizeof(HeapTuple));
+ else
+ memcpy(&data->rows[curr_idx++], &oldrows[i],
+ sizeof(HeapTuple));
+ }
+
+ /* update ndistinct values for the buckets (total and per dimension) */
+ update_bucket_ndistinct(bucket, attrs, stats);
+ update_bucket_ndistinct(null_bucket, attrs, stats);
+
+ /*
+ * TODO We don't need to do this for the dimension we used for split,
+ * because we know how many distinct values went to each
+ * bucket (NULL is not a value, so 0, and the other bucket got
+ * all the ndistinct values).
+ */
+ for (i = 0; i < histogram->ndimensions; i++)
+ {
+ update_dimension_ndistinct(bucket, i, attrs, stats, false);
+ update_dimension_ndistinct(null_bucket, i, attrs, stats, false);
+ }
+
+ pfree(oldrows);
+
+ /* add the NULL bucket to the histogram */
+ histogram->buckets[histogram->nbuckets++] = null_bucket;
+
+ /*
+ * And now run the function recursively on both buckets (the new
+ * one first, because the call may change number of buckets, and
+ * it's used as an index).
+ */
+ create_null_buckets(histogram, (histogram->nbuckets-1), attrs, stats);
+ create_null_buckets(histogram, bucket_idx, attrs, stats);
+
+}
+
+/*
+ * We need to pass the SortSupport to the comparator, but bsearch()
+ * has no 'context' parameter, so we use a global variable (ugly).
+ */
+static int
+bsearch_comparator(const void * a, const void * b)
+{
+ Assert(ssup_private != NULL);
+ return compare_scalars_simple(a, b, (void*)ssup_private);
+}
+
+/*
+ * SRF with details about buckets of a histogram:
+ *
+ * - bucket ID (0...nbuckets)
+ * - min values (string array)
+ * - max values (string array)
+ * - nulls only (boolean array)
+ * - min inclusive flags (boolean array)
+ * - max inclusive flags (boolean array)
+ * - frequency (double precision)
+ *
+ * The input is the OID of the statistics, and there are no rows
+ * returned if the statistics contains no histogram (or if there's no
+ * statistics for the OID).
+ *
+ * The second parameter (type) determines what values will be returned
+ * in the (minvals,maxvals). There are three possible values:
+ *
+ * 0 (actual values)
+ * -----------------
+ * - prints actual values
+ * - using the output function of the data type (as string)
+ * - handy for investigating the histogram
+ *
+ * 1 (distinct index)
+ * ------------------
+ * - prints index of the distinct value (into the serialized array)
+ * - makes it easier to spot neighbor buckets, etc.
+ * - handy for plotting the histogram
+ *
+ * 2 (normalized distinct index)
+ * -----------------------------
+ * - prints index of the distinct value, but normalized into [0,1]
+ * - similar to 1, but shows how 'long' the bucket range is
+ * - handy for plotting the histogram
+ *
+ * When plotting the histogram, be careful as the (1) and (2) options
+ * skew the lengths by distributing the distinct values uniformly. For
+ * data types without a clear meaning of 'distance' (e.g. strings) that
+ * is not a big deal, but for numbers it may be confusing.
+ */
+PG_FUNCTION_INFO_V1(pg_mv_histogram_buckets);
+
+Datum
+pg_mv_histogram_buckets(PG_FUNCTION_ARGS)
+{
+ FuncCallContext *funcctx;
+ int call_cntr;
+ int max_calls;
+ TupleDesc tupdesc;
+ AttInMetadata *attinmeta;
+
+ Oid mvoid = PG_GETARG_OID(0);
+ int otype = PG_GETARG_INT32(1);
+
+ if ((otype < 0) || (otype > 2))
+ elog(ERROR, "invalid output type specified");
+
+ /* stuff done only on the first call of the function */
+ if (SRF_IS_FIRSTCALL())
+ {
+ MemoryContext oldcontext;
+ MVSerializedHistogram histogram;
+
+ /* create a function context for cross-call persistence */
+ funcctx = SRF_FIRSTCALL_INIT();
+
+ /* switch to memory context appropriate for multiple function calls */
+ oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
+
+ histogram = load_mv_histogram(mvoid);
+
+ funcctx->user_fctx = histogram;
+
+ /* total number of tuples to be returned */
+ funcctx->max_calls = 0;
+ if (funcctx->user_fctx != NULL)
+ funcctx->max_calls = histogram->nbuckets;
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("function returning record called in context "
+ "that cannot accept type record")));
+
+ /*
+ * generate attribute metadata needed later to produce tuples
+ * from raw C strings
+ */
+ attinmeta = TupleDescGetAttInMetadata(tupdesc);
+ funcctx->attinmeta = attinmeta;
+
+ MemoryContextSwitchTo(oldcontext);
+ }
+
+ /* stuff done on every call of the function */
+ funcctx = SRF_PERCALL_SETUP();
+
+ call_cntr = funcctx->call_cntr;
+ max_calls = funcctx->max_calls;
+ attinmeta = funcctx->attinmeta;
+
+ if (call_cntr < max_calls) /* do when there is more left to send */
+ {
+ char **values;
+ HeapTuple tuple;
+ Datum result;
+ int2vector *stakeys;
+ Oid relid;
+ double bucket_size = 1.0;
+
+ char *buff = palloc0(1024);
+ char *format;
+
+ int i;
+
+ Oid *outfuncs;
+ FmgrInfo *fmgrinfo;
+
+ MVSerializedHistogram histogram;
+ MVSerializedBucket bucket;
+
+ histogram = (MVSerializedHistogram)funcctx->user_fctx;
+
+ Assert(call_cntr < histogram->nbuckets);
+
+ bucket = histogram->buckets[call_cntr];
+
+ stakeys = find_mv_attnums(mvoid, &relid);
+
+ /*
+ * Prepare a values array for building the returned tuple.
+ * This should be an array of C strings which will
+ * be processed later by the type input functions.
+ */
+ values = (char **) palloc(9 * sizeof(char *));
+
+ values[0] = (char *) palloc(64 * sizeof(char));
+
+ /* arrays */
+ values[1] = (char *) palloc0(1024 * sizeof(char));
+ values[2] = (char *) palloc0(1024 * sizeof(char));
+ values[3] = (char *) palloc0(1024 * sizeof(char));
+ values[4] = (char *) palloc0(1024 * sizeof(char));
+ values[5] = (char *) palloc0(1024 * sizeof(char));
+
+ values[6] = (char *) palloc(64 * sizeof(char));
+ values[7] = (char *) palloc(64 * sizeof(char));
+ values[8] = (char *) palloc(64 * sizeof(char));
+
+ /* we need to do this only when printing the actual values */
+ outfuncs = (Oid*)palloc0(sizeof(Oid) * histogram->ndimensions);
+ fmgrinfo = (FmgrInfo*)palloc0(sizeof(FmgrInfo) * histogram->ndimensions);
+
+ for (i = 0; i < histogram->ndimensions; i++)
+ {
+ bool isvarlena;
+
+ getTypeOutputInfo(get_atttype(relid, stakeys->values[i]),
+ &outfuncs[i], &isvarlena);
+
+ fmgr_info(outfuncs[i], &fmgrinfo[i]);
+ }
+
+ snprintf(values[0], 64, "%d", call_cntr); /* bucket ID */
+
+ /*
+ * currently we only print array of indexes, but the deduplicated
+ * values should be sorted, so this is actually quite useful
+ *
+ * TODO print the actual min/max values, using the output
+ * function of the attribute type
+ */
+
+ for (i = 0; i < histogram->ndimensions; i++)
+ {
+ bucket_size *= (bucket->max[i] - bucket->min[i]) * 1.0
+ / (histogram->nvalues[i]-1);
+
+ /* print the actual values, i.e. use output function etc. */
+ if (otype == 0)
+ {
+ Datum minval, maxval;
+ Datum minout, maxout;
+
+ format = "%s, %s";
+ if (i == 0)
+ format = "{%s%s";
+ else if (i == histogram->ndimensions-1)
+ format = "%s, %s}";
+
+ minval = histogram->values[i][bucket->min[i]];
+ minout = FunctionCall1(&fmgrinfo[i], minval);
+
+ maxval = histogram->values[i][bucket->max[i]];
+ maxout = FunctionCall1(&fmgrinfo[i], maxval);
+
+ // snprintf(buff, 1024, format, values[1], bucket->min[i]);
+ snprintf(buff, 1024, format, values[1], DatumGetPointer(minout));
+ strncpy(values[1], buff, 1023);
+ buff[0] = '\0';
+
+ // snprintf(buff, 1024, format, values[2], bucket->max[i]);
+ snprintf(buff, 1024, format, values[2], DatumGetPointer(maxout));
+ strncpy(values[2], buff, 1023);
+ buff[0] = '\0';
+ }
+ else if (otype == 1)
+ {
+ format = "%s, %d";
+ if (i == 0)
+ format = "{%s%d";
+ else if (i == histogram->ndimensions-1)
+ format = "%s, %d}";
+
+ snprintf(buff, 1024, format, values[1], bucket->min[i]);
+ strncpy(values[1], buff, 1023);
+ buff[0] = '\0';
+
+ snprintf(buff, 1024, format, values[2], bucket->max[i]);
+ strncpy(values[2], buff, 1023);
+ buff[0] = '\0';
+ }
+ else
+ {
+ format = "%s, %f";
+ if (i == 0)
+ format = "{%s%f";
+ else if (i == histogram->ndimensions-1)
+ format = "%s, %f}";
+
+ snprintf(buff, 1024, format, values[1],
+ bucket->min[i] * 1.0 / (histogram->nvalues[i]-1));
+ strncpy(values[1], buff, 1023);
+ buff[0] = '\0';
+
+ snprintf(buff, 1024, format, values[2],
+ bucket->max[i] * 1.0 / (histogram->nvalues[i]-1));
+ strncpy(values[2], buff, 1023);
+ buff[0] = '\0';
+ }
+
+ format = "%s, %s";
+ if (i == 0)
+ format = "{%s%s";
+ else if (i == histogram->ndimensions-1)
+ format = "%s, %s}";
+
+ snprintf(buff, 1024, format, values[3], bucket->nullsonly[i] ? "t" : "f");
+ strncpy(values[3], buff, 1023);
+ buff[0] = '\0';
+
+ snprintf(buff, 1024, format, values[4], bucket->min_inclusive[i] ? "t" : "f");
+ strncpy(values[4], buff, 1023);
+ buff[0] = '\0';
+
+ snprintf(buff, 1024, format, values[5], bucket->max_inclusive[i] ? "t" : "f");
+ strncpy(values[5], buff, 1023);
+ buff[0] = '\0';
+ }
+
+ snprintf(values[6], 64, "%f", bucket->ntuples); /* frequency */
+ snprintf(values[7], 64, "%f", bucket->ntuples / bucket_size); /* density */
+ snprintf(values[8], 64, "%f", bucket_size); /* bucket_size */
+
+ /* build a tuple */
+ tuple = BuildTupleFromCStrings(attinmeta, values);
+
+ /* make the tuple into a datum */
+ result = HeapTupleGetDatum(tuple);
+
+ /* clean up (this is not really necessary) */
+ pfree(values[0]);
+ pfree(values[1]);
+ pfree(values[2]);
+ pfree(values[3]);
+ pfree(values[4]);
+ pfree(values[5]);
+ pfree(values[6]);
+
+ pfree(values);
+
+ SRF_RETURN_NEXT(funcctx, result);
+ }
+ else /* do when there is no more left */
+ {
+ SRF_RETURN_DONE(funcctx);
+ }
+}
+
+#ifdef DEBUG_MVHIST
+/*
+ * prints debugging info about matched histogram buckets (full/partial)
+ *
+ * XXX Currently works only for INT data type.
+ */
+void
+debug_histogram_matches(MVSerializedHistogram mvhist, char *matches)
+{
+ int i, j;
+
+ float ffull = 0, fpartial = 0;
+ int nfull = 0, npartial = 0;
+
+ for (i = 0; i < mvhist->nbuckets; i++)
+ {
+ MVSerializedBucket bucket = mvhist->buckets[i];
+
+ char ranges[1024];
+
+ if (! matches[i])
+ continue;
+
+ /* increment the counters */
+ nfull += (matches[i] == MVSTATS_MATCH_FULL) ? 1 : 0;
+ npartial += (matches[i] == MVSTATS_MATCH_PARTIAL) ? 1 : 0;
+
+ /* and also update the frequencies */
+ ffull += (matches[i] == MVSTATS_MATCH_FULL) ? bucket->ntuples : 0;
+ fpartial += (matches[i] == MVSTATS_MATCH_PARTIAL) ? bucket->ntuples : 0;
+
+ memset(ranges, 0, sizeof(ranges));
+
+ /* build ranges for all the dimentions */
+ for (j = 0; j < mvhist->ndimensions; j++)
+ {
+ sprintf(ranges, "%s [%d %d]", ranges,
+ DatumGetInt32(mvhist->values[j][bucket->min[j]]),
+ DatumGetInt32(mvhist->values[j][bucket->max[j]]));
+ }
+
+ elog(WARNING, "bucket %d %s => %d [%f]", i, ranges, matches[i], bucket->ntuples);
+ }
+
+ elog(WARNING, "full=%f partial=%f (%f)", ffull, fpartial, (ffull + 0.5 * fpartial));
+}
+#endif
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 2c22d31..b693f36 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -2109,9 +2109,9 @@ describeOneTableDetails(const char *schemaname,
{
printfPQExpBuffer(&buf,
"SELECT oid, stanamespace::regnamespace AS nsp, staname, stakeys,\n"
- " deps_enabled, mcv_enabled,\n"
- " deps_built, mcv_built,\n"
- " mcv_max_items,\n"
+ " deps_enabled, mcv_enabled, hist_enabled,\n"
+ " deps_built, mcv_built, hist_built,\n"
+ " mcv_max_items, hist_max_buckets,\n"
" (SELECT string_agg(attname::text,', ')\n"
" FROM ((SELECT unnest(stakeys) AS attnum) s\n"
" JOIN pg_attribute a ON (starelid = a.attrelid and a.attnum = s.attnum))) AS attnums\n"
@@ -2154,8 +2154,17 @@ describeOneTableDetails(const char *schemaname,
first = false;
}
+ if (!strcmp(PQgetvalue(result, i, 6), "t"))
+ {
+ if (! first)
+ appendPQExpBuffer(&buf, ", histogram");
+ else
+ appendPQExpBuffer(&buf, "(histogram");
+ first = false;
+ }
+
appendPQExpBuffer(&buf, ") ON (%s)",
- PQgetvalue(result, i, 9));
+ PQgetvalue(result, i, 12));
printTableAddFooter(&cont, buf.data);
}
diff --git a/src/include/catalog/pg_mv_statistic.h b/src/include/catalog/pg_mv_statistic.h
index 3529b03..37f473f 100644
--- a/src/include/catalog/pg_mv_statistic.h
+++ b/src/include/catalog/pg_mv_statistic.h
@@ -39,13 +39,16 @@ CATALOG(pg_mv_statistic,3381)
/* statistics requested to build */
bool deps_enabled; /* analyze dependencies? */
bool mcv_enabled; /* build MCV list? */
+ bool hist_enabled; /* build histogram? */
- /* MCV size */
+ /* histogram / MCV size */
int32 mcv_max_items; /* max MCV items */
+ int32 hist_max_buckets; /* max histogram buckets */
/* statistics that are available (if requested) */
bool deps_built; /* dependencies were built */
bool mcv_built; /* MCV list was built */
+ bool hist_built; /* histogram was built */
/* variable-length fields start here, but we allow direct access to stakeys */
int2vector stakeys; /* array of column keys */
@@ -53,6 +56,7 @@ CATALOG(pg_mv_statistic,3381)
#ifdef CATALOG_VARLEN
bytea stadeps; /* dependencies (serialized) */
bytea stamcv; /* MCV list (serialized) */
+ bytea stahist; /* MV histogram (serialized) */
#endif
} FormData_pg_mv_statistic;
@@ -68,18 +72,22 @@ typedef FormData_pg_mv_statistic *Form_pg_mv_statistic;
* compiler constants for pg_mv_statistic
* ----------------
*/
-#define Natts_pg_mv_statistic 12
+#define Natts_pg_mv_statistic 15
#define Anum_pg_mv_statistic_starelid 1
#define Anum_pg_mv_statistic_staname 2
#define Anum_pg_mv_statistic_stanamespace 3
#define Anum_pg_mv_statistic_staowner 4
#define Anum_pg_mv_statistic_deps_enabled 5
#define Anum_pg_mv_statistic_mcv_enabled 6
-#define Anum_pg_mv_statistic_mcv_max_items 7
-#define Anum_pg_mv_statistic_deps_built 8
-#define Anum_pg_mv_statistic_mcv_built 9
-#define Anum_pg_mv_statistic_stakeys 10
-#define Anum_pg_mv_statistic_stadeps 11
-#define Anum_pg_mv_statistic_stamcv 12
+#define Anum_pg_mv_statistic_hist_enabled 7
+#define Anum_pg_mv_statistic_mcv_max_items 8
+#define Anum_pg_mv_statistic_hist_max_buckets 9
+#define Anum_pg_mv_statistic_deps_built 10
+#define Anum_pg_mv_statistic_mcv_built 11
+#define Anum_pg_mv_statistic_hist_built 12
+#define Anum_pg_mv_statistic_stakeys 13
+#define Anum_pg_mv_statistic_stadeps 14
+#define Anum_pg_mv_statistic_stamcv 15
+#define Anum_pg_mv_statistic_stahist 16
#endif /* PG_MV_STATISTIC_H */
diff --git a/src/include/catalog/pg_proc.h b/src/include/catalog/pg_proc.h
index b16eebc..19a490a 100644
--- a/src/include/catalog/pg_proc.h
+++ b/src/include/catalog/pg_proc.h
@@ -2674,6 +2674,10 @@ DATA(insert OID = 3376 ( pg_mv_stats_mcvlist_info PGNSP PGUID 12 1 0 0 0 f f f
DESCR("multi-variate statistics: MCV list info");
DATA(insert OID = 3373 ( pg_mv_mcv_items PGNSP PGUID 12 1 1000 0 0 f f f f t t i s 1 0 2249 "26" "{26,23,1009,1000,701}" "{i,o,o,o,o}" "{oid,index,values,nulls,frequency}" _null_ _null_ pg_mv_mcv_items _null_ _null_ _null_ ));
DESCR("details about MCV list items");
+DATA(insert OID = 3375 ( pg_mv_stats_histogram_info PGNSP PGUID 12 1 0 0 0 f f f f t f i s 1 0 25 "17" _null_ _null_ _null_ _null_ _null_ pg_mv_stats_histogram_info _null_ _null_ _null_ ));
+DESCR("multi-variate statistics: histogram info");
+DATA(insert OID = 3374 ( pg_mv_histogram_buckets PGNSP PGUID 12 1 1000 0 0 f f f f t t i s 2 0 2249 "26 23" "{26,23,23,1009,1009,1000,1000,1000,701,701,701}" "{i,i,o,o,o,o,o,o,o,o,o}" "{oid,otype,index,minvals,maxvals,nullsonly,mininclusive,maxinclusive,frequency,density,bucket_size}" _null_ _null_ pg_mv_histogram_buckets _null_ _null_ _null_ ));
+DESCR("details about histogram buckets");
DATA(insert OID = 1928 ( pg_stat_get_numscans PGNSP PGUID 12 1 0 0 0 f f f f t f s r 1 0 20 "26" _null_ _null_ _null_ _null_ _null_ pg_stat_get_numscans _null_ _null_ _null_ ));
DESCR("statistics: number of scans done for table/index");
diff --git a/src/include/nodes/relation.h b/src/include/nodes/relation.h
index 2bcd582..8c50bfb 100644
--- a/src/include/nodes/relation.h
+++ b/src/include/nodes/relation.h
@@ -654,10 +654,12 @@ typedef struct MVStatisticInfo
/* enabled statistics */
bool deps_enabled; /* functional dependencies enabled */
bool mcv_enabled; /* MCV list enabled */
+ bool hist_enabled; /* histogram enabled */
/* built/available statistics */
bool deps_built; /* functional dependencies built */
bool mcv_built; /* MCV list built */
+ bool hist_built; /* histogram built */
/* columns in the statistics (attnums) */
int2vector *stakeys; /* attnums of the columns covered */
diff --git a/src/include/utils/mvstats.h b/src/include/utils/mvstats.h
index 4535db7..f05a517 100644
--- a/src/include/utils/mvstats.h
+++ b/src/include/utils/mvstats.h
@@ -92,6 +92,123 @@ typedef MCVListData *MCVList;
#define MVSTAT_MCVLIST_MAX_ITEMS 8192 /* max items in MCV list */
/*
+ * Multivariate histograms
+ */
+typedef struct MVBucketData {
+
+ /* Frequencies of this bucket. */
+ float ntuples; /* frequency of tuples tuples */
+
+ /*
+ * Information about dimensions being NULL-only. Not yet used.
+ */
+ bool *nullsonly;
+
+ /* lower boundaries - values and information about the inequalities */
+ Datum *min;
+ bool *min_inclusive;
+
+ /* upper boundaries - values and information about the inequalities */
+ Datum *max;
+ bool *max_inclusive;
+
+ /* used when building the histogram (not serialized/deserialized) */
+ void *build_data;
+
+} MVBucketData;
+
+typedef MVBucketData *MVBucket;
+
+
+typedef struct MVHistogramData {
+
+ uint32 magic; /* magic constant marker */
+ uint32 type; /* type of histogram (BASIC) */
+ uint32 nbuckets; /* number of buckets (buckets array) */
+ uint32 ndimensions; /* number of dimensions */
+
+ MVBucket *buckets; /* array of buckets */
+
+} MVHistogramData;
+
+typedef MVHistogramData *MVHistogram;
+
+/*
+ * Histogram in a partially serialized form, with deduplicated boundary
+ * values etc.
+ *
+ * TODO add more detailed description here
+ */
+
+typedef struct MVSerializedBucketData {
+
+ /* Frequencies of this bucket. */
+ float ntuples; /* frequency of tuples tuples */
+
+ /*
+ * Information about dimensions being NULL-only. Not yet used.
+ */
+ bool *nullsonly;
+
+ /* lower boundaries - values and information about the inequalities */
+ uint16 *min;
+ bool *min_inclusive;
+
+ /* indexes of upper boundaries - values and information about the
+ * inequalities (exclusive vs. inclusive) */
+ uint16 *max;
+ bool *max_inclusive;
+
+} MVSerializedBucketData;
+
+typedef MVSerializedBucketData *MVSerializedBucket;
+
+typedef struct MVSerializedHistogramData {
+
+ uint32 magic; /* magic constant marker */
+ uint32 type; /* type of histogram (BASIC) */
+ uint32 nbuckets; /* number of buckets (buckets array) */
+ uint32 ndimensions; /* number of dimensions */
+
+ /*
+ * keep this the same with MVHistogramData, because of
+ * deserialization (same offset)
+ */
+ MVSerializedBucket *buckets; /* array of buckets */
+
+ /*
+ * serialized boundary values, one array per dimension, deduplicated
+ * (the min/max indexes point into these arrays)
+ */
+ int *nvalues;
+ Datum **values;
+
+} MVSerializedHistogramData;
+
+typedef MVSerializedHistogramData *MVSerializedHistogram;
+
+
+/* used to flag stats serialized to bytea */
+#define MVSTAT_HIST_MAGIC 0x7F8C5670 /* marks serialized bytea */
+#define MVSTAT_HIST_TYPE_BASIC 1 /* basic histogram type */
+
+/*
+ * Limits used for max_buckets option, i.e. we're always guaranteed
+ * to have space for at least MVSTAT_HIST_MIN_BUCKETS, and we cannot
+ * have more than MVSTAT_HIST_MAX_BUCKETS buckets.
+ *
+ * This is just a boundary for the 'max' threshold - the actual
+ * histogram may use less buckets than MVSTAT_HIST_MAX_BUCKETS.
+ *
+ * TODO The MVSTAT_HIST_MIN_BUCKETS should be related to the number of
+ * attributes (MVSTATS_MAX_DIMENSIONS) because of NULL-buckets.
+ * There should be at least 2^N buckets, otherwise we may be unable
+ * to build the NULL buckets.
+ */
+#define MVSTAT_HIST_MIN_BUCKETS 128 /* min number of buckets */
+#define MVSTAT_HIST_MAX_BUCKETS 16384 /* max number of buckets */
+
+/*
* TODO Maybe fetching the histogram/MCV list separately is inefficient?
* Consider adding a single `fetch_stats` method, fetching all
* stats specified using flags (or something like that).
@@ -99,20 +216,25 @@ typedef MCVListData *MCVList;
MVDependencies load_mv_dependencies(Oid mvoid);
MCVList load_mv_mcvlist(Oid mvoid);
+MVSerializedHistogram load_mv_histogram(Oid mvoid);
bytea * serialize_mv_dependencies(MVDependencies dependencies);
bytea * serialize_mv_mcvlist(MCVList mcvlist, int2vector *attrs,
VacAttrStats **stats);
+bytea * serialize_mv_histogram(MVHistogram histogram, int2vector *attrs,
+ VacAttrStats **stats);
/* deserialization of stats (serialization is private to analyze) */
MVDependencies deserialize_mv_dependencies(bytea * data);
MCVList deserialize_mv_mcvlist(bytea * data);
+MVSerializedHistogram deserialize_mv_histogram(bytea * data);
/*
* Returns index of the attribute number within the vector (i.e. a
* dimension within the stats).
*/
int mv_get_index(AttrNumber varattno, int2vector * stakeys);
+int2vector* find_mv_attnums(Oid mvoid, Oid *relid);
int2vector* find_mv_attnums(Oid mvoid, Oid *relid);
@@ -121,6 +243,8 @@ extern Datum pg_mv_stats_dependencies_info(PG_FUNCTION_ARGS);
extern Datum pg_mv_stats_dependencies_show(PG_FUNCTION_ARGS);
extern Datum pg_mv_stats_mcvlist_info(PG_FUNCTION_ARGS);
extern Datum pg_mv_mcvlist_items(PG_FUNCTION_ARGS);
+extern Datum pg_mv_stats_histogram_info(PG_FUNCTION_ARGS);
+extern Datum pg_mv_histogram_buckets(PG_FUNCTION_ARGS);
MVDependencies
build_mv_dependencies(int numrows, HeapTuple *rows, int2vector *attrs,
@@ -130,10 +254,20 @@ MCVList
build_mv_mcvlist(int numrows, HeapTuple *rows, int2vector *attrs,
VacAttrStats **stats, int *numrows_filtered);
+MVHistogram
+build_mv_histogram(int numrows, HeapTuple *rows, int2vector *attrs,
+ VacAttrStats **stats, int numrows_total);
+
void build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
int natts, VacAttrStats **vacattrstats);
-void update_mv_stats(Oid relid, MVDependencies dependencies, MCVList mcvlist,
+void update_mv_stats(Oid relid, MVDependencies dependencies,
+ MCVList mcvlist, MVHistogram histogram,
int2vector *attrs, VacAttrStats **stats);
+#ifdef DEBUG_MVHIST
+extern void debug_histogram_matches(MVSerializedHistogram mvhist, char *matches);
+#endif
+
+
#endif
diff --git a/src/test/regress/expected/mv_histogram.out b/src/test/regress/expected/mv_histogram.out
new file mode 100644
index 0000000..e830816
--- /dev/null
+++ b/src/test/regress/expected/mv_histogram.out
@@ -0,0 +1,207 @@
+-- data type passed by value
+CREATE TABLE mv_histogram (
+ a INT,
+ b INT,
+ c INT
+);
+-- unknown column
+CREATE STATISTICS s7 ON mv_histogram (unknown_column) WITH (histogram);
+ERROR: column "unknown_column" referenced in statistics does not exist
+-- single column
+CREATE STATISTICS s7 ON mv_histogram (a) WITH (histogram);
+ERROR: multivariate stats require 2 or more columns
+-- single column, duplicated
+CREATE STATISTICS s7 ON mv_histogram (a, a) WITH (histogram);
+ERROR: duplicate column name in statistics definition
+-- two columns, one duplicated
+CREATE STATISTICS s7 ON mv_histogram (a, a, b) WITH (histogram);
+ERROR: duplicate column name in statistics definition
+-- unknown option
+CREATE STATISTICS s7 ON mv_histogram (a, b, c) WITH (unknown_option);
+ERROR: unrecognized STATISTICS option "unknown_option"
+-- missing histogram statistics
+CREATE STATISTICS s7 ON mv_histogram (a, b, c) WITH (dependencies, max_buckets=200);
+ERROR: option 'histogram' is required by other options(s)
+-- invalid max_buckets value / too low
+CREATE STATISTICS s7 ON mv_histogram (a, b, c) WITH (mcv, max_buckets=10);
+ERROR: minimum number of buckets is 128
+-- invalid max_buckets value / too high
+CREATE STATISTICS s7 ON mv_histogram (a, b, c) WITH (mcv, max_buckets=100000);
+ERROR: maximum number of buckets is 16384
+-- correct command
+CREATE STATISTICS s7 ON mv_histogram (a, b, c) WITH (histogram);
+-- random data (no functional dependencies)
+INSERT INTO mv_histogram
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+ANALYZE mv_histogram;
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+ hist_enabled | hist_built
+--------------+------------
+ t | t
+(1 row)
+
+TRUNCATE mv_histogram;
+-- a => b, a => c, b => c
+INSERT INTO mv_histogram
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE mv_histogram;
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+ hist_enabled | hist_built
+--------------+------------
+ t | t
+(1 row)
+
+TRUNCATE mv_histogram;
+-- a => b, a => c
+INSERT INTO mv_histogram
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE mv_histogram;
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+ hist_enabled | hist_built
+--------------+------------
+ t | t
+(1 row)
+
+TRUNCATE mv_histogram;
+-- check explain (expect bitmap index scan, not plain index scan)
+INSERT INTO mv_histogram
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX hist_idx ON mv_histogram (a, b);
+ANALYZE mv_histogram;
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+ hist_enabled | hist_built
+--------------+------------
+ t | t
+(1 row)
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mv_histogram WHERE a = 10 AND b = 5;
+ QUERY PLAN
+--------------------------------------------
+ Bitmap Heap Scan on mv_histogram
+ Recheck Cond: ((a = 10) AND (b = 5))
+ -> Bitmap Index Scan on hist_idx
+ Index Cond: ((a = 10) AND (b = 5))
+(4 rows)
+
+DROP TABLE mv_histogram;
+-- varlena type (text)
+CREATE TABLE mv_histogram (
+ a TEXT,
+ b TEXT,
+ c TEXT
+);
+CREATE STATISTICS s8 ON mv_histogram (a, b, c) WITH (histogram);
+-- random data (no functional dependencies)
+INSERT INTO mv_histogram
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+ANALYZE mv_histogram;
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+ hist_enabled | hist_built
+--------------+------------
+ t | t
+(1 row)
+
+TRUNCATE mv_histogram;
+-- a => b, a => c, b => c
+INSERT INTO mv_histogram
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE mv_histogram;
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+ hist_enabled | hist_built
+--------------+------------
+ t | t
+(1 row)
+
+TRUNCATE mv_histogram;
+-- a => b, a => c
+INSERT INTO mv_histogram
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE mv_histogram;
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+ hist_enabled | hist_built
+--------------+------------
+ t | t
+(1 row)
+
+TRUNCATE mv_histogram;
+-- check explain (expect bitmap index scan, not plain index scan)
+INSERT INTO mv_histogram
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX hist_idx ON mv_histogram (a, b);
+ANALYZE mv_histogram;
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+ hist_enabled | hist_built
+--------------+------------
+ t | t
+(1 row)
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mv_histogram WHERE a = '10' AND b = '5';
+ QUERY PLAN
+------------------------------------------------------------
+ Bitmap Heap Scan on mv_histogram
+ Recheck Cond: ((a = '10'::text) AND (b = '5'::text))
+ -> Bitmap Index Scan on hist_idx
+ Index Cond: ((a = '10'::text) AND (b = '5'::text))
+(4 rows)
+
+TRUNCATE mv_histogram;
+-- check explain (expect bitmap index scan, not plain index scan) with NULLs
+INSERT INTO mv_histogram
+ SELECT
+ (CASE WHEN i/10000 = 0 THEN NULL ELSE i/10000 END),
+ (CASE WHEN i/20000 = 0 THEN NULL ELSE i/20000 END),
+ (CASE WHEN i/40000 = 0 THEN NULL ELSE i/40000 END)
+ FROM generate_series(1,1000000) s(i);
+ANALYZE mv_histogram;
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+ hist_enabled | hist_built
+--------------+------------
+ t | t
+(1 row)
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mv_histogram WHERE a IS NULL AND b IS NULL;
+ QUERY PLAN
+---------------------------------------------------
+ Bitmap Heap Scan on mv_histogram
+ Recheck Cond: ((a IS NULL) AND (b IS NULL))
+ -> Bitmap Index Scan on hist_idx
+ Index Cond: ((a IS NULL) AND (b IS NULL))
+(4 rows)
+
+DROP TABLE mv_histogram;
+-- NULL values (mix of int and text columns)
+CREATE TABLE mv_histogram (
+ a INT,
+ b TEXT,
+ c INT,
+ d TEXT
+);
+CREATE STATISTICS s9 ON mv_histogram (a, b, c, d) WITH (histogram);
+INSERT INTO mv_histogram
+ SELECT
+ mod(i, 100),
+ (CASE WHEN mod(i, 200) = 0 THEN NULL ELSE mod(i,200) END),
+ mod(i, 400),
+ (CASE WHEN mod(i, 300) = 0 THEN NULL ELSE mod(i,600) END)
+ FROM generate_series(1,10000) s(i);
+ANALYZE mv_histogram;
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+ hist_enabled | hist_built
+--------------+------------
+ t | t
+(1 row)
+
+DROP TABLE mv_histogram;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 66071d8..1a1a4ca 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1375,7 +1375,9 @@ pg_mv_stats| SELECT n.nspname AS schemaname,
length(s.stadeps) AS depsbytes,
pg_mv_stats_dependencies_info(s.stadeps) AS depsinfo,
length(s.stamcv) AS mcvbytes,
- pg_mv_stats_mcvlist_info(s.stamcv) AS mcvinfo
+ pg_mv_stats_mcvlist_info(s.stamcv) AS mcvinfo,
+ length(s.stahist) AS histbytes,
+ pg_mv_stats_histogram_info(s.stahist) AS histinfo
FROM ((pg_mv_statistic s
JOIN pg_class c ON ((c.oid = s.starelid)))
LEFT JOIN pg_namespace n ON ((n.oid = c.relnamespace)));
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index 85d94f1..a885235 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -112,4 +112,4 @@ test: event_trigger
test: stats
# run tests of multivariate stats
-test: mv_dependencies mv_mcv
+test: mv_dependencies mv_mcv mv_histogram
diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule
index 6584d73..2efdcd7 100644
--- a/src/test/regress/serial_schedule
+++ b/src/test/regress/serial_schedule
@@ -164,3 +164,4 @@ test: event_trigger
test: stats
test: mv_dependencies
test: mv_mcv
+test: mv_histogram
diff --git a/src/test/regress/sql/mv_histogram.sql b/src/test/regress/sql/mv_histogram.sql
new file mode 100644
index 0000000..27c2510
--- /dev/null
+++ b/src/test/regress/sql/mv_histogram.sql
@@ -0,0 +1,176 @@
+-- data type passed by value
+CREATE TABLE mv_histogram (
+ a INT,
+ b INT,
+ c INT
+);
+
+-- unknown column
+CREATE STATISTICS s7 ON mv_histogram (unknown_column) WITH (histogram);
+
+-- single column
+CREATE STATISTICS s7 ON mv_histogram (a) WITH (histogram);
+
+-- single column, duplicated
+CREATE STATISTICS s7 ON mv_histogram (a, a) WITH (histogram);
+
+-- two columns, one duplicated
+CREATE STATISTICS s7 ON mv_histogram (a, a, b) WITH (histogram);
+
+-- unknown option
+CREATE STATISTICS s7 ON mv_histogram (a, b, c) WITH (unknown_option);
+
+-- missing histogram statistics
+CREATE STATISTICS s7 ON mv_histogram (a, b, c) WITH (dependencies, max_buckets=200);
+
+-- invalid max_buckets value / too low
+CREATE STATISTICS s7 ON mv_histogram (a, b, c) WITH (mcv, max_buckets=10);
+
+-- invalid max_buckets value / too high
+CREATE STATISTICS s7 ON mv_histogram (a, b, c) WITH (mcv, max_buckets=100000);
+
+-- correct command
+CREATE STATISTICS s7 ON mv_histogram (a, b, c) WITH (histogram);
+
+-- random data (no functional dependencies)
+INSERT INTO mv_histogram
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+
+ANALYZE mv_histogram;
+
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+
+TRUNCATE mv_histogram;
+
+-- a => b, a => c, b => c
+INSERT INTO mv_histogram
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+
+ANALYZE mv_histogram;
+
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+
+TRUNCATE mv_histogram;
+
+-- a => b, a => c
+INSERT INTO mv_histogram
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE mv_histogram;
+
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+
+TRUNCATE mv_histogram;
+
+-- check explain (expect bitmap index scan, not plain index scan)
+INSERT INTO mv_histogram
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX hist_idx ON mv_histogram (a, b);
+ANALYZE mv_histogram;
+
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mv_histogram WHERE a = 10 AND b = 5;
+
+DROP TABLE mv_histogram;
+
+-- varlena type (text)
+CREATE TABLE mv_histogram (
+ a TEXT,
+ b TEXT,
+ c TEXT
+);
+
+CREATE STATISTICS s8 ON mv_histogram (a, b, c) WITH (histogram);
+
+-- random data (no functional dependencies)
+INSERT INTO mv_histogram
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+
+ANALYZE mv_histogram;
+
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+
+TRUNCATE mv_histogram;
+
+-- a => b, a => c, b => c
+INSERT INTO mv_histogram
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+
+ANALYZE mv_histogram;
+
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+
+TRUNCATE mv_histogram;
+
+-- a => b, a => c
+INSERT INTO mv_histogram
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE mv_histogram;
+
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+
+TRUNCATE mv_histogram;
+
+-- check explain (expect bitmap index scan, not plain index scan)
+INSERT INTO mv_histogram
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX hist_idx ON mv_histogram (a, b);
+ANALYZE mv_histogram;
+
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mv_histogram WHERE a = '10' AND b = '5';
+
+TRUNCATE mv_histogram;
+
+-- check explain (expect bitmap index scan, not plain index scan) with NULLs
+INSERT INTO mv_histogram
+ SELECT
+ (CASE WHEN i/10000 = 0 THEN NULL ELSE i/10000 END),
+ (CASE WHEN i/20000 = 0 THEN NULL ELSE i/20000 END),
+ (CASE WHEN i/40000 = 0 THEN NULL ELSE i/40000 END)
+ FROM generate_series(1,1000000) s(i);
+ANALYZE mv_histogram;
+
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mv_histogram WHERE a IS NULL AND b IS NULL;
+
+DROP TABLE mv_histogram;
+
+-- NULL values (mix of int and text columns)
+CREATE TABLE mv_histogram (
+ a INT,
+ b TEXT,
+ c INT,
+ d TEXT
+);
+
+CREATE STATISTICS s9 ON mv_histogram (a, b, c, d) WITH (histogram);
+
+INSERT INTO mv_histogram
+ SELECT
+ mod(i, 100),
+ (CASE WHEN mod(i, 200) = 0 THEN NULL ELSE mod(i,200) END),
+ mod(i, 400),
+ (CASE WHEN mod(i, 300) = 0 THEN NULL ELSE mod(i,600) END)
+ FROM generate_series(1,10000) s(i);
+
+ANALYZE mv_histogram;
+
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+
+DROP TABLE mv_histogram;
--
2.1.0
[text/x-patch] 0006-multi-statistics-estimation.patch (98.3K, ../../1457535779.24545.71.camel@2ndquadrant.com/7-0006-multi-statistics-estimation.patch)
download | inline diff:
From 3a564dbf9aa2c734d80c5e385f105cf8a48da1f5 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@pgaddict.com>
Date: Fri, 6 Feb 2015 01:42:38 +0100
Subject: [PATCH 6/9] multi-statistics estimation
The general idea is that a probability (which is what selectivity is)
can be split into a product of conditional probabilities like this:
P(A & B & C) = P(A & B) * P(C|A & B)
If we assume that C and B are independent, the last part may be
simplified like this
P(A & B & C) = P(A & B) * P(C|A)
we only need probabilities on [A,B] and [C,A] to compute the original
probability.
The implementation works in the other direction, though. We know what
probability P(A & B & C) we need to compute, and also what statistics
are available.
So we search for a combinations of statistics, covering the clauses in
an optimal way (most clauses covered, most dependencies exploited).
There are two possible approaches - exhaustive and greedy. The
exhaustive one walks through all permutations of stats using dynamic
programming, so it's guaranteed to find the optimal solution, but it
soon gets very slow as it's roughly O(N!). The dynamic programming may
improve that a bit, but it's still far too expensive for large numbers
of statistics (on a single table).
The greedy algorithm is very simple - in every step choose the best
solution. That may not guarantee the best solution globally (but maybe
it does?), but it only needs N steps to find the solution, so it's very
fast (processing the selected stats is usually way more expensive).
There's a GUC for selecting the search algorithm
mvstat_search = {'greedy', 'exhaustive'}
The default value is 'greedy' as that's much safer (with respect to
runtime). See choose_mv_statistics().
Once we have found a sequence of statistics, we apply them to the
clauses using the conditional probabilities. We process the selected
stats one by one, and for each we select the estimated clauses and
conditions. See clauselist_selectivity() for more details.
Limitations
-----------
It's still true that each clause at a given level has to be covered by
a single MV statistics. So with this query
WHERE (clause1) AND (clause2) AND (clause3 OR clause4)
each parenthesized clause has to be covered by a single multivariate
statistics.
Clauses not covered by a single statistics at this level will be passed
to clause_selectivity() but this will treat them as a collection of
simpler clauses (connected by AND or OR), and the clauses from the
previous level will be used as conditions.
So using the same example, the last clause will be passed to
clause_selectivity() with 'clause1' and 'clause2' as conditions, and it
will be processed using multivariate stats if possible.
The other limitation is that all the expressions have to be
mv-compatible, i.e. there can't be a mix of expressions. If this is
violated, the clause may be passed to the next level (just like with
list of clauses not covered by a single statistics), which splits that
into clauses handled by multivariate stats and clauses handler by
regular statistics.
rework clauselist_selectivity_or to handle OR-clauses correctly
We might invent a completely new set of functions here, resembling
clauselist_selectivity but adapting the ideas to OR-clauses.
But luckily we know that each OR-clause
(a OR b OR c)
may be rewritten as an equivalent AND-clause using negation:
NOT ((NOT a) AND (NOT b) AND (NOT c))
And that's something we can pass to clauselist_selectivity.
---
contrib/file_fdw/file_fdw.c | 3 +-
contrib/postgres_fdw/postgres_fdw.c | 11 +-
src/backend/optimizer/path/clausesel.c | 1990 ++++++++++++++++++++++++++------
src/backend/optimizer/path/costsize.c | 23 +-
src/backend/optimizer/util/orclauses.c | 4 +-
src/backend/utils/adt/selfuncs.c | 17 +-
src/backend/utils/misc/guc.c | 20 +
src/backend/utils/mvstats/README.stats | 166 +++
src/include/optimizer/cost.h | 6 +-
src/include/utils/mvstats.h | 8 +
10 files changed, 1890 insertions(+), 358 deletions(-)
diff --git a/contrib/file_fdw/file_fdw.c b/contrib/file_fdw/file_fdw.c
index dc035d7..8f11b7a 100644
--- a/contrib/file_fdw/file_fdw.c
+++ b/contrib/file_fdw/file_fdw.c
@@ -969,7 +969,8 @@ estimate_size(PlannerInfo *root, RelOptInfo *baserel,
baserel->baserestrictinfo,
0,
JOIN_INNER,
- NULL);
+ NULL,
+ NIL);
nrows = clamp_row_est(nrows);
diff --git a/contrib/postgres_fdw/postgres_fdw.c b/contrib/postgres_fdw/postgres_fdw.c
index 76d0e15..e78f140 100644
--- a/contrib/postgres_fdw/postgres_fdw.c
+++ b/contrib/postgres_fdw/postgres_fdw.c
@@ -498,7 +498,8 @@ postgresGetForeignRelSize(PlannerInfo *root,
fpinfo->local_conds,
baserel->relid,
JOIN_INNER,
- NULL);
+ NULL,
+ NIL);
cost_qual_eval(&fpinfo->local_conds_cost, fpinfo->local_conds, root);
@@ -2149,7 +2150,8 @@ estimate_path_cost_size(PlannerInfo *root,
local_param_join_conds,
foreignrel->relid,
JOIN_INNER,
- NULL);
+ NULL,
+ NIL);
local_sel *= fpinfo->local_conds_sel;
rows = clamp_row_est(rows * local_sel);
@@ -3618,7 +3620,8 @@ postgresGetForeignJoinPaths(PlannerInfo *root,
fpinfo->local_conds,
0,
JOIN_INNER,
- NULL);
+ NULL,
+ NIL);
cost_qual_eval(&fpinfo->local_conds_cost, fpinfo->local_conds, root);
/*
@@ -3637,7 +3640,7 @@ postgresGetForeignJoinPaths(PlannerInfo *root,
*/
fpinfo->joinclause_sel = clauselist_selectivity(root, fpinfo->joinclauses,
0, fpinfo->jointype,
- extra->sjinfo);
+ extra->sjinfo, NIL);
}
fpinfo->server = GetForeignServer(joinrel->serverid);
diff --git a/src/backend/optimizer/path/clausesel.c b/src/backend/optimizer/path/clausesel.c
index 0de2418..c1b8999 100644
--- a/src/backend/optimizer/path/clausesel.c
+++ b/src/backend/optimizer/path/clausesel.c
@@ -29,6 +29,8 @@
#include "utils/selfuncs.h"
#include "utils/typcache.h"
+#include "miscadmin.h"
+
/*
* Data structure for accumulating info about possible range-query
@@ -44,6 +46,13 @@ typedef struct RangeQueryClause
Selectivity hibound; /* Selectivity of a var < something clause */
} RangeQueryClause;
+static Selectivity clauselist_selectivity_or(PlannerInfo *root,
+ List *clauses,
+ int varRelid,
+ JoinType jointype,
+ SpecialJoinInfo *sjinfo,
+ List *conditions);
+
static void addRangeClause(RangeQueryClause **rqlist, Node *clause,
bool varonleft, bool isLTsel, Selectivity s2);
@@ -60,23 +69,25 @@ static int count_mv_attnums(List *clauses, Index relid, int type);
static int count_varnos(List *clauses, Index *relid);
+static List *clauses_matching_statistic(List **clauses, MVStatisticInfo *statistic,
+ Index relid, int types, bool remove);
+
static List *clauselist_apply_dependencies(PlannerInfo *root, List *clauses,
Index relid, List *stats);
-static MVStatisticInfo *choose_mv_statistics(List *mvstats, Bitmapset *attnums);
-
-static List *clauselist_mv_split(PlannerInfo *root, Index relid,
- List *clauses, List **mvclauses,
- MVStatisticInfo *mvstats, int types);
-
static Selectivity clauselist_mv_selectivity(PlannerInfo *root,
- List *clauses, MVStatisticInfo *mvstats);
+ MVStatisticInfo *mvstats, List *clauses,
+ List *conditions, bool is_or);
static Selectivity clauselist_mv_selectivity_mcvlist(PlannerInfo *root,
- List *clauses, MVStatisticInfo *mvstats,
- bool *fullmatch, Selectivity *lowsel);
+ MVStatisticInfo *mvstats,
+ List *clauses, List *conditions,
+ bool is_or, bool *fullmatch,
+ Selectivity *lowsel);
static Selectivity clauselist_mv_selectivity_histogram(PlannerInfo *root,
- List *clauses, MVStatisticInfo *mvstats);
+ MVStatisticInfo *mvstats,
+ List *clauses, List *conditions,
+ bool is_or);
static int update_match_bitmap_mcvlist(PlannerInfo *root, List *clauses,
int2vector *stakeys, MCVList mcvlist,
@@ -90,10 +101,33 @@ static int update_match_bitmap_histogram(PlannerInfo *root, List *clauses,
int nmatches, char * matches,
bool is_or);
+/*
+ * Describes a combination of multiple statistics to cover attributes
+ * referenced by the clauses. The array 'stats' (with nstats elements)
+ * lists attributes (in the order as they are applied), and number of
+ * clause attributes covered by this solution.
+ *
+ * choose_mv_statistics_exhaustive() uses this to track both the current
+ * and the best solutions, while walking through the state of possible
+ * combination.
+ */
+typedef struct mv_solution_t {
+ int nclauses; /* number of clauses covered */
+ int nconditions; /* number of conditions covered */
+ int nstats; /* number of stats applied */
+ int *stats; /* stats (in the apply order) */
+} mv_solution_t;
+
+static List *choose_mv_statistics(PlannerInfo *root, Index relid,
+ List *mvstats, List *clauses, List *conditions);
+
static bool has_stats(List *stats, int type);
static List * find_stats(PlannerInfo *root, Index relid);
+static bool stats_type_matches(MVStatisticInfo *stat, int type);
+
+int mvstat_search_type = MVSTAT_SEARCH_GREEDY;
/* used for merging bitmaps - AND (min), OR (max) */
#define MAX(x, y) (((x) > (y)) ? (x) : (y))
@@ -168,14 +202,15 @@ clauselist_selectivity(PlannerInfo *root,
List *clauses,
int varRelid,
JoinType jointype,
- SpecialJoinInfo *sjinfo)
+ SpecialJoinInfo *sjinfo,
+ List *conditions)
{
Selectivity s1 = 1.0;
RangeQueryClause *rqlist = NULL;
ListCell *l;
/* processing mv stats */
- Oid relid = InvalidOid;
+ Index relid = InvalidOid;
/* list of multivariate stats on the relation */
List *stats = NIL;
@@ -191,12 +226,13 @@ clauselist_selectivity(PlannerInfo *root,
stats = find_stats(root, relid);
/*
- * If there's exactly one clause, then no use in trying to match up pairs,
- * so just go directly to clause_selectivity().
+ * If there's exactly one clause, then no use in trying to match up
+ * pairs, or matching multivariate statistics, so just go directly
+ * to clause_selectivity().
*/
if (list_length(clauses) == 1)
return clause_selectivity(root, (Node *) linitial(clauses),
- varRelid, jointype, sjinfo);
+ varRelid, jointype, sjinfo, conditions);
/*
* Apply functional dependencies, but first check that there are some stats
@@ -228,31 +264,96 @@ clauselist_selectivity(PlannerInfo *root,
(count_mv_attnums(clauses, relid,
MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST) >= 2))
{
- /* collect attributes from the compatible conditions */
- Bitmapset *mvattnums = collect_mv_attnums(clauses, relid,
- MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST);
+ ListCell *s;
+
+ /*
+ * Copy the conditions we got from the upper part of the expression tree
+ * so that we can add local conditions to it (we need to keep the
+ * original list intact, for sibling expressions - other expressions
+ * at the same level).
+ */
+ List *conditions_local = list_copy(conditions);
- /* and search for the statistic covering the most attributes */
- MVStatisticInfo *mvstat = choose_mv_statistics(stats, mvattnums);
+ /* find the best combination of statistics */
+ List *solution = choose_mv_statistics(root, relid, stats,
+ clauses, conditions);
- if (mvstat != NULL) /* we have a matching stats */
+ /*
+ * We have a good solution, which is merely a list of statistics that
+ * we need to apply. We'll apply the statistics one by one (in the order
+ * as they appear in the list), and for each statistic we'll
+ *
+ * (1) find clauses compatible with the statistic (and remove them
+ * from the list)
+ *
+ * (2) find local conditions compatible with the statistic
+ *
+ * (3) do the estimation P(clauses | conditions)
+ *
+ * (4) append the estimated clauses to local conditions
+ *
+ * continuously modify
+ */
+ foreach (s, solution)
{
- /* clauses compatible with multi-variate stats */
- List *mvclauses = NIL;
+ MVStatisticInfo *mvstat = (MVStatisticInfo *)lfirst(s);
- /* split the clauselist into regular and mv-clauses */
- clauses = clauselist_mv_split(root, relid, clauses, &mvclauses,
- mvstat, MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST);
+ /* clauses compatible with the statistic we're applying right now */
+ List *stat_clauses = NIL;
+ List *stat_conditions = NIL;
- /* we've chosen the histogram to match the clauses */
- Assert(mvclauses != NIL);
+ /*
+ * Find clauses and conditions matching the statistic - the clauses
+ * need to be removed from the list, while conditions should remain
+ * there (so that we can apply them repeatedly).
+ */
+ stat_clauses
+ = clauses_matching_statistic(&clauses, mvstat, relid,
+ MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST,
+ true);
+
+ stat_conditions
+ = clauses_matching_statistic(&conditions_local, mvstat, relid,
+ MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST,
+ false);
+
+ /*
+ * If we got no clauses to estimate, we've done something wrong,
+ * either during the optimization, detecting compatible clause, or
+ * somewhere else.
+ *
+ * Also, we need at least two attributes in clauses and conditions.
+ */
+ Assert(stat_clauses != NIL);
+ Assert(count_mv_attnums(list_union(stat_clauses, stat_conditions),
+ relid, MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST) >= 2);
/* compute the multivariate stats */
- s1 *= clauselist_mv_selectivity(root, mvclauses, mvstat);
+ s1 *= clauselist_mv_selectivity(root, mvstat,
+ stat_clauses, stat_conditions,
+ false); /* AND */
+
+ /*
+ * Add the new clauses to the local conditions, so that we can use
+ * them for the subsequent statistics. We only add the clauses,
+ * because the conditions are already there (or should be).
+ */
+ conditions_local = list_concat(conditions_local, stat_clauses);
}
+
+ /* from now on, work only with the 'local' list of conditions */
+ conditions = conditions_local;
}
/*
+ * If there's exactly one clause, then no use in trying to match up
+ * pairs, so just go directly to clause_selectivity().
+ */
+ if (list_length(clauses) == 1)
+ return s1 * clause_selectivity(root, (Node *) linitial(clauses),
+ varRelid, jointype, sjinfo, conditions);
+
+ /*
* Initial scan over clauses. Anything that doesn't look like a potential
* rangequery clause gets multiplied into s1 and forgotten. Anything that
* does gets inserted into an rqlist entry.
@@ -264,7 +365,8 @@ clauselist_selectivity(PlannerInfo *root,
Selectivity s2;
/* Always compute the selectivity using clause_selectivity */
- s2 = clause_selectivity(root, clause, varRelid, jointype, sjinfo);
+ s2 = clause_selectivity(root, clause, varRelid, jointype, sjinfo,
+ conditions);
/*
* Check for being passed a RestrictInfo.
@@ -423,6 +525,55 @@ clauselist_selectivity(PlannerInfo *root,
}
/*
+ * Similar to clauselist_selectivity(), but for OR-clauses. We can't simply use
+ * the same multi-statistic estimation logic for AND-clauses, at least not
+ * directly, because there are a few key differences:
+ *
+ * - functional dependencies don't really apply to OR-clauses
+ *
+ * - clauselist_selectivity() is based on decomposing the selectivity into
+ * a sequence of conditional probabilities (selectivities), but that can
+ * be done only for AND-clauses
+ *
+ * We might invent a similar infrastructure for optimizing OR-clauses, doing
+ * something similar to what clause_selectivity does for AND-clauses, but
+ * luckily we know that each disjunctive normal form (aka OR-clause)
+ *
+ * (a OR b OR c)
+ *
+ * may be rewritten as an equivalent conjunctive normal form (aka AND-clause)
+ * by using negation:
+ *
+ * NOT ((NOT a) AND (NOT b) AND (NOT c))
+ *
+ * And that's something we can pass to clauselist_selectivity and let it do
+ * all the heavy lifting.
+ */
+static Selectivity
+clauselist_selectivity_or(PlannerInfo *root,
+ List *clauses,
+ int varRelid,
+ JoinType jointype,
+ SpecialJoinInfo *sjinfo,
+ List *conditions)
+{
+ List *args = NIL;
+ ListCell *l;
+ Expr *expr;
+
+ /* build arguments for the AND-clause by negating args of the OR-clause */
+ foreach (l, clauses)
+ args = lappend(args, makeBoolExpr(NOT_EXPR, list_make1(lfirst(l)), -1));
+
+ /* and then the actual OR-clause on the negated args */
+ expr = makeBoolExpr(AND_EXPR, args, -1);
+
+ /* instead of constructing NOT expression, just do (1.0 - s) */
+ return 1.0 - clauselist_selectivity(root, list_make1(expr), varRelid,
+ jointype, sjinfo, conditions);
+}
+
+/*
* addRangeClause --- add a new range clause for clauselist_selectivity
*
* Here is where we try to match up pairs of range-query clauses
@@ -629,7 +780,8 @@ clause_selectivity(PlannerInfo *root,
Node *clause,
int varRelid,
JoinType jointype,
- SpecialJoinInfo *sjinfo)
+ SpecialJoinInfo *sjinfo,
+ List *conditions)
{
Selectivity s1 = 0.5; /* default for any unhandled clause type */
RestrictInfo *rinfo = NULL;
@@ -749,7 +901,8 @@ clause_selectivity(PlannerInfo *root,
(Node *) get_notclausearg((Expr *) clause),
varRelid,
jointype,
- sjinfo);
+ sjinfo,
+ conditions);
}
else if (and_clause(clause))
{
@@ -758,29 +911,18 @@ clause_selectivity(PlannerInfo *root,
((BoolExpr *) clause)->args,
varRelid,
jointype,
- sjinfo);
+ sjinfo,
+ conditions);
}
else if (or_clause(clause))
{
- /*
- * Selectivities for an OR clause are computed as s1+s2 - s1*s2 to
- * account for the probable overlap of selected tuple sets.
- *
- * XXX is this too conservative?
- */
- ListCell *arg;
-
- s1 = 0.0;
- foreach(arg, ((BoolExpr *) clause)->args)
- {
- Selectivity s2 = clause_selectivity(root,
- (Node *) lfirst(arg),
- varRelid,
- jointype,
- sjinfo);
-
- s1 = s1 + s2 - s1 * s2;
- }
+ /* just call to clauselist_selectivity_or() */
+ s1 = clauselist_selectivity_or(root,
+ ((BoolExpr *) clause)->args,
+ varRelid,
+ jointype,
+ sjinfo,
+ conditions);
}
else if (is_opclause(clause) || IsA(clause, DistinctExpr))
{
@@ -870,7 +1012,8 @@ clause_selectivity(PlannerInfo *root,
(Node *) ((RelabelType *) clause)->arg,
varRelid,
jointype,
- sjinfo);
+ sjinfo,
+ conditions);
}
else if (IsA(clause, CoerceToDomain))
{
@@ -879,7 +1022,8 @@ clause_selectivity(PlannerInfo *root,
(Node *) ((CoerceToDomain *) clause)->arg,
varRelid,
jointype,
- sjinfo);
+ sjinfo,
+ conditions);
}
else
{
@@ -943,15 +1087,16 @@ clause_selectivity(PlannerInfo *root,
* in the MCV list, then the selectivity is below the lowest frequency
* found in the MCV list,
*
- * TODO When applying the clauses to the histogram/MCV list, we can do
- * that from the most selective clauses first, because that'll
- * eliminate the buckets/items sooner (so we'll be able to skip
- * them without inspection, which is more expensive). But this
- * requires really knowing the per-clause selectivities in advance,
- * and that's not what we do now.
+ * TODO When applying the clauses to the histogram/MCV list, we can do that from
+ * the most selective clauses first, because that'll eliminate the
+ * buckets/items sooner (so we'll be able to skip them without inspection,
+ * which is more expensive). But this requires really knowing the
+ * per-clause selectivities in advance, and that's not what we do now.
+ *
*/
static Selectivity
-clauselist_mv_selectivity(PlannerInfo *root, List *clauses, MVStatisticInfo *mvstats)
+clauselist_mv_selectivity(PlannerInfo *root, MVStatisticInfo *mvstats,
+ List *clauses, List *conditions, bool is_or)
{
bool fullmatch = false;
Selectivity s1 = 0.0, s2 = 0.0;
@@ -969,7 +1114,8 @@ clauselist_mv_selectivity(PlannerInfo *root, List *clauses, MVStatisticInfo *mvs
*/
/* Evaluate the MCV first. */
- s1 = clauselist_mv_selectivity_mcvlist(root, clauses, mvstats,
+ s1 = clauselist_mv_selectivity_mcvlist(root, mvstats,
+ clauses, conditions, is_or,
&fullmatch, &mcv_low);
/*
@@ -982,7 +1128,8 @@ clauselist_mv_selectivity(PlannerInfo *root, List *clauses, MVStatisticInfo *mvs
/* TODO if (fullmatch) without matching MCV item, use the mcv_low
* selectivity as upper bound */
- s2 = clauselist_mv_selectivity_histogram(root, clauses, mvstats);
+ s2 = clauselist_mv_selectivity_histogram(root, mvstats,
+ clauses, conditions, is_or);
/* TODO clamp to <= 1.0 (or more strictly, when possible) */
return s1 + s2;
@@ -1016,260 +1163,1325 @@ get_varattnos(Node * node, Index relid)
k + FirstLowInvalidHeapAttributeNumber);
}
- bms_free(varattnos);
+ bms_free(varattnos);
+
+ return result;
+}
+
+/*
+ * Collect attributes from mv-compatible clauses.
+ */
+static Bitmapset *
+collect_mv_attnums(List *clauses, Index relid, int types)
+{
+ Bitmapset *attnums = NULL;
+ ListCell *l;
+
+ /*
+ * Walk through the clauses and identify the ones we can estimate
+ * using multivariate stats, and remember the relid/columns. We'll
+ * then cross-check if we have suitable stats, and only if needed
+ * we'll split the clauses into multivariate and regular lists.
+ *
+ * For now we're only interested in RestrictInfo nodes with nested
+ * OpExpr, using either a range or equality.
+ */
+ foreach (l, clauses)
+ {
+ Node *clause = (Node *) lfirst(l);
+
+ /* ignore the result here - we only need the attnums */
+ clause_is_mv_compatible(clause, relid, &attnums, types);
+ }
+
+ /*
+ * If there are not at least two attributes referenced by the clause(s),
+ * we can throw everything out (as we'll revert to simple stats).
+ */
+ if (bms_num_members(attnums) <= 1)
+ {
+ bms_free(attnums);
+ attnums = NULL;
+ }
+
+ return attnums;
+}
+
+/*
+ * Count the number of attributes in clauses compatible with multivariate stats.
+ */
+static int
+count_mv_attnums(List *clauses, Index relid, int type)
+{
+ int c;
+ Bitmapset *attnums = collect_mv_attnums(clauses, relid, type);
+
+ c = bms_num_members(attnums);
+
+ bms_free(attnums);
+
+ return c;
+}
+
+/*
+ * Count varnos referenced in the clauses, and if there's a single varno then
+ * return the index in 'relid'.
+ */
+static int
+count_varnos(List *clauses, Index *relid)
+{
+ int cnt;
+ Bitmapset *varnos = NULL;
+
+ varnos = pull_varnos((Node *) clauses);
+ cnt = bms_num_members(varnos);
+
+ /* if there's a single varno in the clauses, remember it */
+ if (bms_num_members(varnos) == 1)
+ *relid = bms_singleton_member(varnos);
+
+ bms_free(varnos);
+
+ return cnt;
+}
+
+static List *
+clauses_matching_statistic(List **clauses, MVStatisticInfo *statistic,
+ Index relid, int types, bool remove)
+{
+ int i;
+ Bitmapset *stat_attnums = NULL;
+ List *matching_clauses = NIL;
+ ListCell *lc;
+
+ /* build attnum bitmapset for this statistics */
+ for (i = 0; i < statistic->stakeys->dim1; i++)
+ stat_attnums = bms_add_member(stat_attnums,
+ statistic->stakeys->values[i]);
+
+ /*
+ * We can't use foreach here, because we may need to remove some of the
+ * clauses if (remove=true).
+ */
+ lc = list_head(*clauses);
+ while (lc)
+ {
+ Node *clause = (Node*)lfirst(lc);
+ Bitmapset *attnums = NULL;
+
+ /* must advance lc before list_delete possibly pfree's it */
+ lc = lnext(lc);
+
+ /*
+ * skip clauses that are not compatible with stats (just leave them
+ * in the original list)
+ *
+ * XXX Perhaps this should check what stats are actually available in
+ * the statistics (not a big deal now, because MCV and histograms
+ * handle the same types of conditions).
+ */
+ if (! clause_is_mv_compatible(clause, relid, &attnums, types))
+ {
+ bms_free(attnums);
+ continue;
+ }
+
+ /* if the clause is covered by the statistic, add it to the list */
+ if (bms_is_subset(attnums, stat_attnums))
+ {
+ matching_clauses = lappend(matching_clauses, clause);
+
+ /* if remove=true, remove the matching item from the main list */
+ if (remove)
+ *clauses = list_delete_ptr(*clauses, clause);
+ }
+
+ bms_free(attnums);
+ }
+
+ bms_free(stat_attnums);
+
+ return matching_clauses;
+}
+
+/*
+ * Selects the best combination of multivariate statistics, in an exhaustive
+ * way, where 'best' means:
+ *
+ * (a) covering the most attributes (referenced by clauses)
+ * (b) using the least number of multivariate stats
+ * (c) using the most conditions to exploit dependency
+ *
+ * Don't call this directly but through choose_mv_statistics(), which does some
+ * additional tricks to minimize the runtime.
+ *
+ *
+ * Algorithm
+ * ---------
+ * The algorithm is a recursive implementation of backtracking, with maximum
+ * depth equal to the number of multi-variate statistics available on the table.
+ * It actually explores all valid combinations of stats.
+ *
+ * Whenever it considers adding the next statistics, the clauses it matches are
+ * divided into 'conditions' (clauses already matched by at least one previous
+ * statistics) and clauses that are estimated.
+ *
+ * Then several checks are performed:
+ *
+ * (a) The statistics covers at least 2 columns, referenced in the estimated
+ * clauses (otherwise multi-variate stats are useless).
+ *
+ * (b) The statistics covers at least 1 new column, i.e. column not refefenced
+ * by the already used stats (and the new column has to be referenced by
+ * the clauses, of couse). Otherwise the statistics would not add any new
+ * information.
+ *
+ * There are some other sanity checks (e.g. stats must not be used twice etc.).
+ *
+ *
+ * Weaknesses
+ * ----------
+ * The current implemetation uses a rather simple optimality criteria, so it may
+ * not do the best choice when
+ *
+ * (a) There may be multiple solutions with the same number of covered
+ * attributes and number of statistics (e.g. the same solution but with
+ * statistics in a different order). It's unclear which solution in the best
+ * one - in a sense all of them are equal.
+ *
+ * TODO It might be possible to compute estimate for each of those solutions,
+ * and then combine them to get the final estimate (e.g. by using average
+ * or median).
+ *
+ * (b) Does not consider that some types of stats are a better match for some
+ * types of clauses (e.g. MCV list is generally a better match for equality
+ * conditions than a histogram).
+ *
+ * But maybe this is pointless - generally, each column is either a label
+ * (it's not important whether because of the data type or how it's used),
+ * or a value with ordering that makes sense. So either a MCV list is more
+ * appropriate (labels) or a histogram (values with orderings).
+ *
+ * Now sure what to do with statistics on columns mixing both types of data
+ * (some columns would work best with MCVs, some with histograms). Maybe we
+ * could invent a new type of statistics combining MCV list and histogram
+ * (keeping a small histogram for each MCV item, and a separate histogram
+ * for values not on the MCV list).
+ *
+ * TODO The algorithm should probably count number of Vars (not just attnums)
+ * when computing the 'score' of each solution. Computing the ratio of
+ * (num of all vars) / (num of condition vars) as a measure of how well
+ * the solution uses conditions might be useful.
+ */
+static void
+choose_mv_statistics_exhaustive(PlannerInfo *root, int step,
+ int nmvstats, MVStatisticInfo *mvstats, Bitmapset ** stats_attnums,
+ int nclauses, Node ** clauses, Bitmapset ** clauses_attnums,
+ int nconditions, Node ** conditions, Bitmapset ** conditions_attnums,
+ bool *cover_map, bool *condition_map, int *ruled_out,
+ mv_solution_t *current, mv_solution_t **best)
+{
+ int i, j;
+
+ Assert(best != NULL);
+ Assert((step == 0 && current == NULL) || (step > 0 && current != NULL));
+
+ /* this may run for a long sime, so let's make it interruptible */
+ CHECK_FOR_INTERRUPTS();
+
+ if (current == NULL)
+ {
+ current = (mv_solution_t*)palloc0(sizeof(mv_solution_t));
+ current->stats = (int*)palloc0(sizeof(int)*nmvstats);
+ current->nstats = 0;
+ current->nclauses = 0;
+ current->nconditions = 0;
+ }
+
+ /*
+ * Now try to apply each statistics, matching at least two attributes,
+ * unless it's already used in one of the previous steps.
+ */
+ for (i = 0; i < nmvstats; i++)
+ {
+ int c;
+
+ int ncovered_clauses = 0; /* number of covered clauses */
+ int ncovered_conditions = 0; /* number of covered conditions */
+ int nattnums = 0; /* number of covered attributes */
+
+ Bitmapset *all_attnums = NULL;
+ Bitmapset *new_attnums = NULL;
+
+ /* skip statistics that were already used or eliminated */
+ if (ruled_out[i] != -1)
+ continue;
+
+ /*
+ * See if we have clauses covered by this statistics, but not
+ * yet covered by any of the preceding onces.
+ */
+ for (c = 0; c < nclauses; c++)
+ {
+ bool covered = false;
+ Bitmapset *clause_attnums = clauses_attnums[c];
+ Bitmapset *tmp = NULL;
+
+ /*
+ * If this clause is not covered by this stats, we can't
+ * use the stats to estimate that at all.
+ */
+ if (! cover_map[i * nclauses + c])
+ continue;
+
+ /*
+ * Now we know we'll use this clause - either as a condition
+ * or as a new clause (the estimated one). So let's add the
+ * attributes to the attnums from all the clauses usable with
+ * this statistics.
+ */
+ tmp = bms_union(all_attnums, clause_attnums);
+
+ /* free the old bitmap */
+ bms_free(all_attnums);
+ all_attnums = tmp;
+
+ /* let's see if it's covered by any of the previous stats */
+ for (j = 0; j < step; j++)
+ {
+ /* already covered by the previous stats */
+ if (cover_map[current->stats[j] * nclauses + c])
+ covered = true;
+
+ if (covered)
+ break;
+ }
+
+ /* if already covered, continue with the next clause */
+ if (covered)
+ {
+ ncovered_conditions += 1;
+ continue;
+ }
+
+ /*
+ * OK, this clause is covered by this statistics (and not by
+ * any of the previous ones)
+ */
+ ncovered_clauses += 1;
+
+ /* add the attnums into attnums from 'new clauses' */
+ // new_attnums = bms_union(new_attnums, clause_attnums);
+ }
+
+ /* can't have more new clauses than original clauses */
+ Assert(nclauses >= ncovered_clauses);
+ Assert(ncovered_clauses >= 0); /* mostly paranoia */
+
+ nattnums = bms_num_members(all_attnums);
+
+ /* free all the bitmapsets - we don't need them anymore */
+ bms_free(all_attnums);
+ bms_free(new_attnums);
+
+ all_attnums = NULL;
+ new_attnums = NULL;
+
+ /*
+ * See if we have clauses covered by this statistics, but not
+ * yet covered by any of the preceding onces.
+ */
+ for (c = 0; c < nconditions; c++)
+ {
+ Bitmapset *clause_attnums = conditions_attnums[c];
+ Bitmapset *tmp = NULL;
+
+ /*
+ * If this clause is not covered by this stats, we can't
+ * use the stats to estimate that at all.
+ */
+ if (! condition_map[i * nconditions + c])
+ continue;
+
+ /* count this as a condition */
+ ncovered_conditions += 1;
+
+ /*
+ * Now we know we'll use this clause - either as a condition
+ * or as a new clause (the estimated one). So let's add the
+ * attributes to the attnums from all the clauses usable with
+ * this statistics.
+ */
+ tmp = bms_union(all_attnums, clause_attnums);
+
+ /* free the old bitmap */
+ bms_free(all_attnums);
+ all_attnums = tmp;
+ }
+
+ /*
+ * Let's mark the statistics as 'ruled out' - either we'll use
+ * it (and proceed to the next step), or it's incompatible.
+ */
+ ruled_out[i] = step;
+
+ /*
+ * There are no clauses usable with this statistics (not already
+ * covered by aome of the previous stats).
+ *
+ * Similarly, if the clauses only use a single attribute, we
+ * can't really use that.
+ */
+ if ((ncovered_clauses == 0) || (nattnums < 2))
+ continue;
+
+ /*
+ * TODO Not sure if it's possible to add a clause referencing
+ * only attributes already covered by previous stats?
+ * Introducing only some new dependency, not a new
+ * attribute. Couldn't come up with an example, though.
+ * Might be worth adding some assert.
+ */
+
+ /*
+ * got a suitable statistics - let's update the current solution,
+ * maybe use it as the best solution
+ */
+ current->nclauses += ncovered_clauses;
+ current->nconditions += ncovered_conditions;
+ current->nstats += 1;
+ current->stats[step] = i;
+
+ /*
+ * We can never cover more clauses, or use more stats that we
+ * actually have at the beginning.
+ */
+ Assert(nclauses >= current->nclauses);
+ Assert(nmvstats >= current->nstats);
+ Assert(step < nmvstats);
+
+ if (*best == NULL)
+ {
+ *best = (mv_solution_t*)palloc0(sizeof(mv_solution_t));
+ (*best)->stats = (int*)palloc0(sizeof(int)*nmvstats);
+ (*best)->nstats = 0;
+ (*best)->nclauses = 0;
+ (*best)->nconditions = 0;
+ }
+
+ /* see if it's better than the current 'best' solution */
+ if ((current->nclauses > (*best)->nclauses) ||
+ ((current->nclauses == (*best)->nclauses) &&
+ ((current->nstats > (*best)->nstats))))
+ {
+ (*best)->nstats = current->nstats;
+ (*best)->nclauses = current->nclauses;
+ (*best)->nconditions = current->nconditions;
+ memcpy((*best)->stats, current->stats, nmvstats * sizeof(int));
+ }
+
+ /*
+ * The recursion only makes sense if we haven't covered all the
+ * attributes (then adding stats is not really possible).
+ */
+ if ((step + 1) < nmvstats)
+ choose_mv_statistics_exhaustive(root, step+1,
+ nmvstats, mvstats, stats_attnums,
+ nclauses, clauses, clauses_attnums,
+ nconditions, conditions, conditions_attnums,
+ cover_map, condition_map, ruled_out,
+ current, best);
+
+ /* reset the last step */
+ current->nclauses -= ncovered_clauses;
+ current->nconditions -= ncovered_conditions;
+ current->nstats -= 1;
+ current->stats[step] = 0;
+
+ /* mark the statistics as usable again */
+ ruled_out[i] = -1;
+
+ Assert(current->nclauses >= 0);
+ Assert(current->nstats >= 0);
+ }
+
+ /* reset all statistics as 'incompatible' in this step */
+ for (i = 0; i < nmvstats; i++)
+ if (ruled_out[i] == step)
+ ruled_out[i] = -1;
+
+}
+
+/*
+ * Greedy search for a multivariate solution - a sequence of statistics covering
+ * the clauses. This chooses the "best" statistics at each step, so the
+ * resulting solution may not be the best solution globally, but this produces
+ * the solution in only N steps (where N is the number of statistics), while
+ * the exhaustive approach may have to walk through ~N! combinations (although
+ * some of those are terminated early).
+ *
+ * See the comments at choose_mv_statistics_exhaustive() as this does the same
+ * thing (but in a different way).
+ *
+ * Don't call this directly, but through choose_mv_statistics().
+ *
+ * TODO There are probably other metrics we might use - e.g. using number of
+ * columns (num_cond_columns / num_cov_columns), which might work better
+ * with a mix of simple and complex clauses.
+ *
+ * TODO Also the choice at the very first step should be handled in a special
+ * way, because there will be 0 conditions at that moment, so there needs
+ * to be some other criteria - e.g. using the simplest (or most complex?)
+ * clause might be a good idea.
+ *
+ * TODO We might also select multiple stats using different criteria, and branch
+ * the search. This is however tricky, because if we choose k statistics at
+ * each step, we get k^N branches to walk through (with N steps). That's
+ * not really good with large number of stats (yet better than exhaustive
+ * search).
+ */
+static void
+choose_mv_statistics_greedy(PlannerInfo *root, int step,
+ int nmvstats, MVStatisticInfo *mvstats, Bitmapset ** stats_attnums,
+ int nclauses, Node ** clauses, Bitmapset ** clauses_attnums,
+ int nconditions, Node ** conditions, Bitmapset ** conditions_attnums,
+ bool *cover_map, bool *condition_map, int *ruled_out,
+ mv_solution_t *current, mv_solution_t **best)
+{
+ int i, j;
+ int best_stat = -1;
+ double gain, max_gain = -1.0;
+
+ /*
+ * Bitmap tracking which clauses are already covered (by the previous
+ * statistics) and may thus serve only as a condition in this step.
+ */
+ bool *covered_clauses = (bool*)palloc0(nclauses);
+
+ /*
+ * Number of clauses and columns covered by each statistics - this
+ * includes both conditions and clauses covered by the statistics for
+ * the first time. The number of columns may count some columns
+ * repeatedly - if a column is shared by multiple clauses, it will
+ * be counted once for each clause (covered by the statistics).
+ * So with two clauses [(a=1 OR b=2),(a<2 OR c>1)] the column "a"
+ * will be counted twice (if both clauses are covered).
+ *
+ * The values for reduded statistics (that can't be applied) are
+ * not computed, because that'd be pointless.
+ */
+ int *num_cov_clauses = (int*)palloc0(sizeof(int) * nmvstats);
+ int *num_cov_columns = (int*)palloc0(sizeof(int) * nmvstats);
+
+ /*
+ * Same as above, but this only includes clauses that are already
+ * covered by the previous stats (and the current one).
+ */
+ int *num_cond_clauses = (int*)palloc0(sizeof(int) * nmvstats);
+ int *num_cond_columns = (int*)palloc0(sizeof(int) * nmvstats);
+
+ /*
+ * Number of attributes for each clause.
+ *
+ * TODO Might be computed in choose_mv_statistics() and then passed
+ * here, but then the function would not have the same signature
+ * as _exhaustive().
+ */
+ int *attnum_counts = (int*)palloc0(sizeof(int) * nclauses);
+ int *attnum_cond_counts = (int*)palloc0(sizeof(int) * nconditions);
+
+ CHECK_FOR_INTERRUPTS();
+
+ Assert(best != NULL);
+ Assert((step == 0 && current == NULL) || (step > 0 && current != NULL));
+
+ /* compute attributes (columns) for each clause */
+ for (i = 0; i < nclauses; i++)
+ attnum_counts[i] = bms_num_members(clauses_attnums[i]);
+
+ /* compute attributes (columns) for each condition */
+ for (i = 0; i < nconditions; i++)
+ attnum_cond_counts[i] = bms_num_members(conditions_attnums[i]);
+
+ /* see which clauses are already covered at this point (by previous stats) */
+ for (i = 0; i < step; i++)
+ for (j = 0; j < nclauses; j++)
+ covered_clauses[j] |= (cover_map[current->stats[i] * nclauses + j]);
+
+ /* which remaining statistics covers most clauses / uses most conditions? */
+ for (i = 0; i < nmvstats; i++)
+ {
+ Bitmapset *attnums_covered = NULL;
+ Bitmapset *attnums_conditions = NULL;
+
+ /* skip stats that are already ruled out (either used or inapplicable) */
+ if (ruled_out[i] != -1)
+ continue;
+
+ /* count covered clauses and conditions (for the statistics) */
+ for (j = 0; j < nclauses; j++)
+ {
+ if (cover_map[i * nclauses + j])
+ {
+ Bitmapset *attnums_new
+ = bms_union(attnums_covered, clauses_attnums[j]);
+
+ /* get rid of the old bitmap and keep the unified result */
+ bms_free(attnums_covered);
+ attnums_covered = attnums_new;
+
+ num_cov_clauses[i] += 1;
+ num_cov_columns[i] += attnum_counts[j];
+
+ /* is the clause already covered (i.e. a condition)? */
+ if (covered_clauses[j])
+ {
+ num_cond_clauses[i] += 1;
+ num_cond_columns[i] += attnum_counts[j];
+ attnums_new = bms_union(attnums_conditions,
+ clauses_attnums[j]);
+
+ bms_free(attnums_conditions);
+ attnums_conditions = attnums_new;
+ }
+ }
+ }
+
+ /* if all covered clauses are covered by prev stats (thus conditions) */
+ if (num_cov_clauses[i] == num_cond_clauses[i])
+ ruled_out[i] = step;
+
+ /* same if there are no new attributes */
+ else if (bms_num_members(attnums_conditions) == bms_num_members(attnums_covered))
+ ruled_out[i] = step;
+
+ bms_free(attnums_covered);
+ bms_free(attnums_conditions);
+
+ /* if the statistics is inapplicable, try the next one */
+ if (ruled_out[i] != -1)
+ continue;
+
+ /* now let's walk through conditions and count the covered */
+ for (j = 0; j < nconditions; j++)
+ {
+ if (condition_map[i * nconditions + j])
+ {
+ num_cond_clauses[i] += 1;
+ num_cond_columns[i] += attnum_cond_counts[j];
+ }
+ }
+
+ /* otherwise see if this improves the interesting metrics */
+ gain = num_cond_columns[i] / (double)num_cov_columns[i];
+
+ if (gain > max_gain)
+ {
+ max_gain = gain;
+ best_stat = i;
+ }
+ }
+
+ /*
+ * Have we found a suitable statistics? Add it to the solution and
+ * try next step.
+ */
+ if (best_stat != -1)
+ {
+ /* mark the statistics, so that we skip it in next steps */
+ ruled_out[best_stat] = step;
+
+ /* allocate current solution if necessary */
+ if (current == NULL)
+ {
+ current = (mv_solution_t*)palloc0(sizeof(mv_solution_t));
+ current->stats = (int*)palloc0(sizeof(int)*nmvstats);
+ current->nstats = 0;
+ current->nclauses = 0;
+ current->nconditions = 0;
+ }
+
+ current->nclauses += num_cov_clauses[best_stat];
+ current->nconditions += num_cond_clauses[best_stat];
+ current->stats[step] = best_stat;
+ current->nstats++;
+
+ if (*best == NULL)
+ {
+ (*best) = (mv_solution_t*)palloc0(sizeof(mv_solution_t));
+ (*best)->nstats = current->nstats;
+ (*best)->nclauses = current->nclauses;
+ (*best)->nconditions = current->nconditions;
+
+ (*best)->stats = (int*)palloc0(sizeof(int)*nmvstats);
+ memcpy((*best)->stats, current->stats, nmvstats * sizeof(int));
+ }
+ else
+ {
+ /* see if this is a better solution */
+ double current_gain = (double)current->nconditions / current->nclauses;
+ double best_gain = (double)(*best)->nconditions / (*best)->nclauses;
+
+ if ((current_gain > best_gain) ||
+ ((current_gain == best_gain) && (current->nstats < (*best)->nstats)))
+ {
+ (*best)->nstats = current->nstats;
+ (*best)->nclauses = current->nclauses;
+ (*best)->nconditions = current->nconditions;
+ memcpy((*best)->stats, current->stats, nmvstats * sizeof(int));
+ }
+ }
+
+ /*
+ * The recursion only makes sense if we haven't covered all the
+ * attributes (then adding stats is not really possible).
+ */
+ if ((step + 1) < nmvstats)
+ choose_mv_statistics_greedy(root, step+1,
+ nmvstats, mvstats, stats_attnums,
+ nclauses, clauses, clauses_attnums,
+ nconditions, conditions, conditions_attnums,
+ cover_map, condition_map, ruled_out,
+ current, best);
+
+ /* reset the last step */
+ current->nclauses -= num_cov_clauses[best_stat];
+ current->nconditions -= num_cond_clauses[best_stat];
+ current->nstats -= 1;
+ current->stats[step] = 0;
+
+ /* mark the statistics as usable again */
+ ruled_out[best_stat] = -1;
+ }
+
+ /* reset all statistics eliminated in this step */
+ for (i = 0; i < nmvstats; i++)
+ if (ruled_out[i] == step)
+ ruled_out[i] = -1;
+
+ /* free everything allocated in this step */
+ pfree(covered_clauses);
+ pfree(attnum_counts);
+ pfree(num_cov_clauses);
+ pfree(num_cov_columns);
+ pfree(num_cond_clauses);
+ pfree(num_cond_columns);
+}
+
+/*
+ * Remove clauses not covered by any of the available statistics
+ *
+ * This helps us to reduce the amount of work done in choose_mv_statistics()
+ * by not having to deal with clauses that can't possibly be useful.
+ */
+static List *
+filter_clauses(PlannerInfo *root, Index relid, int type,
+ List *stats, List *clauses, Bitmapset **attnums)
+{
+ ListCell *c;
+ ListCell *s;
+
+ /* results (list of compatible clauses, attnums) */
+ List *rclauses = NIL;
+
+ foreach (c, clauses)
+ {
+ Node *clause = (Node*)lfirst(c);
+ Bitmapset *clause_attnums = NULL;
+
+ /*
+ * We do assume that thanks to previous checks, we should not run into
+ * clauses that are incompatible with multivariate stats here. We also
+ * need to collect the attnums for the clause.
+ *
+ * XXX Maybe turn this into an assert?
+ */
+ if (! clause_is_mv_compatible(clause, relid, &clause_attnums, type))
+ elog(ERROR, "should not get non-mv-compatible cluase");
+
+ /* Is there a multivariate statistics covering the clause? */
+ foreach (s, stats)
+ {
+ int k, matches = 0;
+ MVStatisticInfo *stat = (MVStatisticInfo *)lfirst(s);
+
+ /* skip statistics not matching the required type */
+ if (! stats_type_matches(stat, type))
+ continue;
+
+ /*
+ * see if all clause attributes are covered by the statistic
+ *
+ * We'll do that in the opposite direction, i.e. we'll see how many
+ * attributes of the statistic are referenced in the clause, and then
+ * compare the counts.
+ */
+ for (k = 0; k < stat->stakeys->dim1; k++)
+ if (bms_is_member(stat->stakeys->values[k], clause_attnums))
+ matches += 1;
+
+ /*
+ * If the number of matches is equal to attributes referenced by the
+ * clause, then the clause is covered by the statistic.
+ */
+ if (bms_num_members(clause_attnums) == matches)
+ {
+ *attnums = bms_union(*attnums, clause_attnums);
+ rclauses = lappend(rclauses, clause);
+ break;
+ }
+ }
+
+ bms_free(clause_attnums);
+ }
+
+ /* we can't have more compatible conditions than source conditions */
+ Assert(list_length(clauses) >= list_length(rclauses));
+
+ return rclauses;
+}
+
+/*
+ * Remove statistics not covering any new clauses
+ *
+ * Statistics not covering any new clauses (conditions don't count) are not
+ * really useful, so let's ignore them. Also, we need the statistics to
+ * reference at least two different attributes (both in conditions and clauses
+ * combined), and at least one of them in the clauses alone.
+ *
+ * This check might be made more strict by checking against individual clauses,
+ * because by using the bitmapsets of all attnums we may actually use attnums
+ * from clauses that are not covered by the statistics. For example, we may
+ * have a condition
+ *
+ * (a=1 AND b=2)
+ *
+ * and a new clause
+ *
+ * (c=1 AND d=1)
+ *
+ * With only bitmapsets, statistics on [b,c] will pass through this (assuming
+ * there are some statistics covering both clases).
+ *
+ * Parameters:
+ *
+ * stats - list of statistics to filter
+ * new_attnums - attnums referenced in new clauses
+ * all_attnums - attnums referenced by contidions and new clauses combined
+ *
+ * Returns filtered list of statistics.
+ *
+ * TODO Do the more strict check, i.e. walk through individual clauses and
+ * conditions and only use those covered by the statistics.
+ */
+static List *
+filter_stats(List *stats, Bitmapset *new_attnums, Bitmapset *all_attnums)
+{
+ ListCell *s;
+ List *stats_filtered = NIL;
+
+ foreach (s, stats)
+ {
+ int k;
+ int matches_new = 0,
+ matches_all = 0;
+
+ MVStatisticInfo *stat = (MVStatisticInfo *)lfirst(s);
+
+ /* see how many attributes the statistics covers */
+ for (k = 0; k < stat->stakeys->dim1; k++)
+ {
+ /* attributes from new clauses */
+ if (bms_is_member(stat->stakeys->values[k], new_attnums))
+ matches_new += 1;
+
+ /* attributes from onditions */
+ if (bms_is_member(stat->stakeys->values[k], all_attnums))
+ matches_all += 1;
+ }
+
+ /* check we have enough attributes for this statistics */
+ if ((matches_new >= 1) && (matches_all >= 2))
+ stats_filtered = lappend(stats_filtered, stat);
+ }
+
+ /* we can't have more useful stats than we had originally */
+ Assert(list_length(stats) >= list_length(stats_filtered));
+
+ return stats_filtered;
+}
+
+static MVStatisticInfo *
+make_stats_array(List *stats, int *nmvstats)
+{
+ int i;
+ ListCell *l;
+
+ MVStatisticInfo *mvstats = NULL;
+ *nmvstats = list_length(stats);
+
+ mvstats
+ = (MVStatisticInfo*)palloc0((*nmvstats) * sizeof(MVStatisticInfo));
+
+ i = 0;
+ foreach (l, stats)
+ {
+ MVStatisticInfo *stat = (MVStatisticInfo *)lfirst(l);
+ memcpy(&mvstats[i++], stat, sizeof(MVStatisticInfo));
+ }
+
+ return mvstats;
+}
+
+static Bitmapset **
+make_stats_attnums(MVStatisticInfo *mvstats, int nmvstats)
+{
+ int i, j;
+ Bitmapset **stats_attnums = NULL;
+
+ Assert(nmvstats > 0);
- return result;
+ /* build bitmaps of attnums for the stats (easier to compare) */
+ stats_attnums = (Bitmapset **)palloc0(nmvstats * sizeof(Bitmapset*));
+
+ for (i = 0; i < nmvstats; i++)
+ for (j = 0; j < mvstats[i].stakeys->dim1; j++)
+ stats_attnums[i]
+ = bms_add_member(stats_attnums[i],
+ mvstats[i].stakeys->values[j]);
+
+ return stats_attnums;
}
+
/*
- * Collect attributes from mv-compatible clauses.
+ * Remove redundant statistics
+ *
+ * If there are multiple statistics covering the same set of columns (counting
+ * only those referenced by clauses and conditions), we can apply one of those
+ * anyway and further reduce the size of the optimization problem.
+ *
+ * Thus when redundant stats are detected, we keep the smaller one (the one with
+ * fewer columns), based on the assumption that it's more accurate and also
+ * faster to process. That may be untrue for two reasons - first, the accuracy
+ * really depends on number of buckets/MCV items, not the number of columns.
+ * Second, some types of statistics may work better for certain types of clauses
+ * (e.g. MCV lists for equality conditions) etc.
*/
-static Bitmapset *
-collect_mv_attnums(List *clauses, Index relid, int types)
+static List*
+filter_redundant_stats(List *stats, List *clauses, List *conditions)
{
- Bitmapset *attnums = NULL;
- ListCell *l;
+ int i, j, nmvstats;
+
+ MVStatisticInfo *mvstats;
+ bool *redundant;
+ Bitmapset **stats_attnums;
+ Bitmapset *varattnos;
+ Index relid;
+
+ Assert(list_length(stats) > 0);
+ Assert(list_length(clauses) > 0);
/*
- * Walk through the clauses and identify the ones we can estimate using
- * multivariate stats, and remember the relid/columns. We'll then
- * cross-check if we have suitable stats, and only if needed we'll split
- * the clauses into multivariate and regular lists.
+ * We'll convert the list of statistics into an array now, because
+ * the reduction of redundant statistics is easier to do that way
+ * (we can mark previous stats as redundant, etc.).
+ */
+ mvstats = make_stats_array(stats, &nmvstats);
+ stats_attnums = make_stats_attnums(mvstats, nmvstats);
+
+ /* by default, none of the stats is redundant (so palloc0) */
+ redundant = palloc0(nmvstats * sizeof(bool));
+
+ /*
+ * We only expect a single relid here, and also we should get the
+ * same relid from clauses and conditions (but we get it from
+ * clauses, because those are certainly non-empty).
+ */
+ relid = bms_singleton_member(pull_varnos((Node*)clauses));
+
+ /*
+ * Get the varattnos from both conditions and clauses.
+ *
+ * This skips system attributes, although that should be impossible
+ * thanks to previous filtering out of incompatible clauses.
*
- * For now we're only interested in RestrictInfo nodes with nested OpExpr,
- * using either a range or equality.
+ * XXX Is that really true?
*/
- foreach (l, clauses)
+ varattnos = bms_union(get_varattnos((Node*)clauses, relid),
+ get_varattnos((Node*)conditions, relid));
+
+ for (i = 1; i < nmvstats; i++)
{
- Node *clause = (Node *) lfirst(l);
+ /* intersect with current statistics */
+ Bitmapset *curr = bms_intersect(stats_attnums[i], varattnos);
- /* ignore the result here - we only need the attnums */
- clause_is_mv_compatible(clause, relid, &attnums, types);
+ /* walk through 'previous' stats and check redundancy */
+ for (j = 0; j < i; j++)
+ {
+ /* intersect with current statistics */
+ Bitmapset *prev;
+
+ /* skip stats already identified as redundant */
+ if (redundant[j])
+ continue;
+
+ prev = bms_intersect(stats_attnums[j], varattnos);
+
+ switch (bms_subset_compare(curr, prev))
+ {
+ case BMS_EQUAL:
+ /*
+ * Use the smaller one (hopefully more accurate).
+ * If both have the same size, use the first one.
+ */
+ if (mvstats[i].stakeys->dim1 >= mvstats[j].stakeys->dim1)
+ redundant[i] = TRUE;
+ else
+ redundant[j] = TRUE;
+
+ break;
+
+ case BMS_SUBSET1: /* curr is subset of prev */
+ redundant[i] = TRUE;
+ break;
+
+ case BMS_SUBSET2: /* prev is subset of curr */
+ redundant[j] = TRUE;
+ break;
+
+ case BMS_DIFFERENT:
+ /* do nothing - keep both stats */
+ break;
+ }
+
+ bms_free(prev);
+ }
+
+ bms_free(curr);
}
- /*
- * If there are not at least two attributes referenced by the clause(s),
- * we can throw everything out (as we'll revert to simple stats).
- */
- if (bms_num_members(attnums) <= 1)
+ /* can't reduce all statistics (at least one has to remain) */
+ Assert(nmvstats > 0);
+
+ /* now, let's remove the reduced statistics from the arrays */
+ list_free(stats);
+ stats = NIL;
+
+ for (i = 0; i < nmvstats; i++)
{
- if (attnums != NULL)
- pfree(attnums);
- attnums = NULL;
+ MVStatisticInfo *info;
+
+ pfree(stats_attnums[i]);
+
+ if (redundant[i])
+ continue;
+
+ info = makeNode(MVStatisticInfo);
+ memcpy(info, &mvstats[i], sizeof(MVStatisticInfo));
+
+ stats = lappend(stats, info);
}
- return attnums;
+ pfree(mvstats);
+ pfree(stats_attnums);
+ pfree(redundant);
+
+ return stats;
}
-/*
- * Count the number of attributes in clauses compatible with multivariate stats.
- */
-static int
-count_mv_attnums(List *clauses, Index relid, int type)
+static Node**
+make_clauses_array(List *clauses, int *nclauses)
{
- int c;
- Bitmapset *attnums = collect_mv_attnums(clauses, relid, type);
+ int i;
+ ListCell *l;
- c = bms_num_members(attnums);
+ Node** clauses_array;
- bms_free(attnums);
+ *nclauses = list_length(clauses);
+ clauses_array = (Node **)palloc0((*nclauses) * sizeof(Node *));
- return c;
+ i = 0;
+ foreach (l, clauses)
+ clauses_array[i++] = (Node *)lfirst(l);
+
+ *nclauses = i;
+
+ return clauses_array;
}
-/*
- * Count varnos referenced in the clauses, and if there's a single varno then
- * return the index in 'relid'.
- */
-static int
-count_varnos(List *clauses, Index *relid)
+static Bitmapset **
+make_clauses_attnums(PlannerInfo *root, Index relid,
+ int type, Node **clauses, int nclauses)
{
- int cnt;
- Bitmapset *varnos = NULL;
+ int i;
+ Bitmapset **clauses_attnums
+ = (Bitmapset **)palloc0(nclauses * sizeof(Bitmapset *));
- varnos = pull_varnos((Node *) clauses);
- cnt = bms_num_members(varnos);
+ for (i = 0; i < nclauses; i++)
+ {
+ Bitmapset * attnums = NULL;
- /* if there's a single varno in the clauses, remember it */
- if (bms_num_members(varnos) == 1)
- *relid = bms_singleton_member(varnos);
+ if (! clause_is_mv_compatible(clauses[i], relid, &attnums, type))
+ elog(ERROR, "should not get non-mv-compatible clause");
- bms_free(varnos);
+ clauses_attnums[i] = attnums;
+ }
- return cnt;
+ return clauses_attnums;
}
-
+
+static bool*
+make_cover_map(Bitmapset **stats_attnums, int nmvstats,
+ Bitmapset **clauses_attnums, int nclauses)
+{
+ int i, j;
+ bool *cover_map = (bool*)palloc0(nclauses * nmvstats);
+
+ for (i = 0; i < nmvstats; i++)
+ for (j = 0; j < nclauses; j++)
+ cover_map[i * nclauses + j]
+ = bms_is_subset(clauses_attnums[j], stats_attnums[i]);
+
+ return cover_map;
+}
+
/*
- * We're looking for statistics matching at least 2 attributes, referenced in
- * clauses compatible with multivariate statistics. The current selection
- * criteria is very simple - we choose the statistics referencing the most
- * attributes.
- *
- * If there are multiple statistics referencing the same number of columns
- * (from the clauses), the one with less source columns (as listed in the
- * ADD STATISTICS when creating the statistics) wins. Else the first one wins.
- *
- * This is a very simple criteria, and has several weaknesses:
- *
- * (a) does not consider the accuracy of the statistics
- *
- * If there are two histograms built on the same set of columns, but one
- * has 100 buckets and the other one has 1000 buckets (thus likely
- * providing better estimates), this is not currently considered.
- *
- * (b) does not consider the type of statistics
- *
- * If there are three statistics - one containing just a MCV list, another
- * one with just a histogram and a third one with both, we treat them equally.
+ * Chooses the combination of statistics, optimal for estimation of a particular
+ * clause list.
*
- * (c) does not consider the number of clauses
+ * This only handles a 'preparation' shared by the exhaustive and greedy
+ * implementations (see the previous methods), mostly trying to reduce the size
+ * of the problem (eliminate clauses/statistics that can't be really used in
+ * the solution).
*
- * As explained, only the number of referenced attributes counts, so if
- * there are multiple clauses on a single attribute, this still counts as
- * a single attribute.
+ * It also precomputes bitmaps for attributes covered by clauses and statistics,
+ * so that we don't need to do that over and over in the actual optimizations
+ * (as it's both CPU and memory intensive).
*
- * (d) does not consider type of condition
*
- * Some clauses may work better with some statistics - for example equality
- * clauses probably work better with MCV lists than with histograms. But
- * IS [NOT] NULL conditions may often work better with histograms (thanks
- * to NULL-buckets).
+ * TODO Another way to make the optimization problems smaller might be splitting
+ * the statistics into several disjoint subsets, i.e. if we can split the
+ * graph of statistics (after the elimination) into multiple components
+ * (so that stats in different components share no attributes), we can do
+ * the optimization for each component separately.
*
- * So for example with five WHERE conditions
- *
- * WHERE (a = 1) AND (b = 1) AND (c = 1) AND (d = 1) AND (e = 1)
- *
- * and statistics on (a,b), (a,b,e) and (a,b,c,d), the last one will be selected
- * as it references the most columns.
- *
- * Once we have selected the multivariate statistics, we split the list of
- * clauses into two parts - conditions that are compatible with the selected
- * stats, and conditions are estimated using simple statistics.
- *
- * From the example above, conditions
- *
- * (a = 1) AND (b = 1) AND (c = 1) AND (d = 1)
- *
- * will be estimated using the multivariate statistics (a,b,c,d) while the last
- * condition (e = 1) will get estimated using the regular ones.
- *
- * There are various alternative selection criteria (e.g. counting conditions
- * instead of just referenced attributes), but eventually the best option should
- * be to combine multiple statistics. But that's much harder to do correctly.
- *
- * TODO Select multiple statistics and combine them when computing the estimate.
- *
- * TODO This will probably have to consider compatibility of clauses, because
- * 'dependencies' will probably work only with equality clauses.
+ * TODO If we could compute what is a "perfect solution" maybe we could
+ * terminate the search after reaching ~90% of it? Say, if we knew that we
+ * can cover 10 clauses and reuse 8 dependencies, maybe covering 9 clauses
+ * and 7 dependencies would be OK?
*/
-static MVStatisticInfo *
-choose_mv_statistics(List *stats, Bitmapset *attnums)
+static List*
+choose_mv_statistics(PlannerInfo *root, Index relid, List *stats,
+ List *clauses, List *conditions)
{
int i;
- ListCell *lc;
+ mv_solution_t *best = NULL;
+ List *result = NIL;
+
+ int nmvstats;
+ MVStatisticInfo *mvstats;
+
+ /* we only work with MCV lists and histograms here */
+ int type = (MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST);
+
+ bool *clause_cover_map = NULL,
+ *condition_cover_map = NULL;
+ int *ruled_out = NULL;
+
+ /* build bitmapsets for all stats and clauses */
+ Bitmapset **stats_attnums;
+ Bitmapset **clauses_attnums;
+ Bitmapset **conditions_attnums;
- MVStatisticInfo *choice = NULL;
+ int nclauses, nconditions;
+ Node ** clauses_array;
+ Node ** conditions_array;
- int current_matches = 1; /* goal #1: maximize */
- int current_dims = (MVSTATS_MAX_DIMENSIONS+1); /* goal #2: minimize */
+ /* copy lists, so that we can free them during elimination easily */
+ clauses = list_copy(clauses);
+ conditions = list_copy(conditions);
+ stats = list_copy(stats);
/*
- * Walk through the statistics (simple array with nmvstats elements) and for
- * each one count the referenced attributes (encoded in the 'attnums' bitmap).
+ * Reduce the optimization problem size as much as possible.
+ *
+ * Eliminate clauses and conditions not covered by any statistics,
+ * or statistics not matching at least two attributes (one of them
+ * has to be in a regular clause).
+ *
+ * It's possible that removing a statistics in one iteration
+ * eliminates clause in the next one, so we'll repeat this until we
+ * eliminate no clauses/stats in that iteration.
+ *
+ * This can only happen after eliminating a statistics - clauses are
+ * eliminated first, so statistics always reflect that.
*/
- foreach (lc, stats)
+ while (true)
{
- MVStatisticInfo *info = (MVStatisticInfo *)lfirst(lc);
-
- /* columns matching this statistics */
- int matches = 0;
+ List *tmp;
- int2vector * attrs = info->stakeys;
- int numattrs = attrs->dim1;
+ Bitmapset *compatible_attnums = NULL;
+ Bitmapset *condition_attnums = NULL;
+ Bitmapset *all_attnums = NULL;
- /* skip dependencies-only stats */
- if (! (info->mcv_built || info->hist_built))
- continue;
+ /*
+ * Clauses
+ *
+ * Walk through clauses and keep only those covered by at least
+ * one of the statistics we still have. We'll also keep info
+ * about attnums in clauses (without conditions) so that we can
+ * ignore stats covering just conditions (which is pointless).
+ */
+ tmp = filter_clauses(root, relid, type,
+ stats, clauses, &compatible_attnums);
- /* count columns covered by the histogram */
- for (i = 0; i < numattrs; i++)
- if (bms_is_member(attrs->values[i], attnums))
- matches++;
+ /* discard the original list */
+ list_free(clauses);
+ clauses = tmp;
/*
- * Use this statistics when it improves the number of matches or
- * when it matches the same number of attributes but is smaller.
+ * Conditions
+ *
+ * Walk through clauses and keep only those covered by at least
+ * one of the statistics we still have. Also, collect bitmap of
+ * attributes so that we can make sure we add at least one new
+ * attribute (by comparing with clauses).
*/
- if ((matches > current_matches) ||
- ((matches == current_matches) && (current_dims > numattrs)))
+ if (conditions != NIL)
{
- choice = info;
- current_matches = matches;
- current_dims = numattrs;
+ tmp = filter_clauses(root, relid, type,
+ stats, conditions, &condition_attnums);
+
+ /* discard the original list */
+ list_free(conditions);
+ conditions = tmp;
}
- }
- return choice;
-}
+ /* get a union of attnums (from conditions and new clauses) */
+ all_attnums = bms_union(compatible_attnums, condition_attnums);
+
+ /*
+ * Statisitics
+ *
+ * Walk through statistics and only keep those covering at least
+ * one new attribute (excluding conditions) and at two attributes
+ * in both clauses and conditions.
+ */
+ tmp = filter_stats(stats, compatible_attnums, all_attnums);
+ /* if we've not eliminated anything, terminate */
+ if (list_length(stats) == list_length(tmp))
+ break;
-/*
- * This splits the clauses list into two parts - one containing clauses that
- * will be evaluated using the chosen statistics, and the remaining clauses
- * (either non-mvcompatible, or not related to the histogram).
- */
-static List *
-clauselist_mv_split(PlannerInfo *root, Index relid,
- List *clauses, List **mvclauses,
- MVStatisticInfo *mvstats, int types)
-{
- int i;
- ListCell *l;
- List *non_mvclauses = NIL;
+ /* work only with filtered statistics from now */
+ list_free(stats);
+ stats = tmp;
+ }
- /* FIXME is there a better way to get info on int2vector? */
- int2vector * attrs = mvstats->stakeys;
- int numattrs = mvstats->stakeys->dim1;
+ /* only do the optimization if we have clauses/statistics */
+ if ((list_length(stats) == 0) || (list_length(clauses) == 0))
+ return NULL;
- Bitmapset *mvattnums = NULL;
+ /* remove redundant stats (stats covered by another stats) */
+ stats = filter_redundant_stats(stats, clauses, conditions);
- /* build bitmap of attributes, so we can do bms_is_subset later */
- for (i = 0; i < numattrs; i++)
- mvattnums = bms_add_member(mvattnums, attrs->values[i]);
+ /*
+ * TODO We should sort the stats to make the order deterministic,
+ * otherwise we may get different estimates on different
+ * executions - if there are multiple "equally good" solutions,
+ * we'll keep the first solution we see.
+ *
+ * Sorting by OID probably is not the right solution though,
+ * because we'd like it to be somehow reproducible,
+ * irrespectedly of the order of ADD STATISTICS commands.
+ * So maybe statkeys?
+ */
+ mvstats = make_stats_array(stats, &nmvstats);
+ stats_attnums = make_stats_attnums(mvstats, nmvstats);
- /* erase the list of mv-compatible clauses */
- *mvclauses = NIL;
+ /* collect clauses an bitmap of attnums */
+ clauses_array = make_clauses_array(clauses, &nclauses);
+ clauses_attnums = make_clauses_attnums(root, relid, type,
+ clauses_array, nclauses);
- foreach (l, clauses)
- {
- bool match = false; /* by default not mv-compatible */
- Bitmapset *attnums = NULL;
- Node *clause = (Node *) lfirst(l);
+ /* collect conditions and bitmap of attnums */
+ conditions_array = make_clauses_array(conditions, &nconditions);
+ conditions_attnums = make_clauses_attnums(root, relid, type,
+ conditions_array, nconditions);
- if (clause_is_mv_compatible(clause, relid, &attnums, types))
+ /*
+ * Build bitmaps with info about which clauses/conditions are
+ * covered by each statistics (so that we don't need to call the
+ * bms_is_subset over and over again).
+ */
+ clause_cover_map = make_cover_map(stats_attnums, nmvstats,
+ clauses_attnums, nclauses);
+
+ condition_cover_map = make_cover_map(stats_attnums, nmvstats,
+ conditions_attnums, nconditions);
+
+ ruled_out = (int*)palloc0(nmvstats * sizeof(int));
+
+ /* no stats are ruled out by default */
+ for (i = 0; i < nmvstats; i++)
+ ruled_out[i] = -1;
+
+ /* do the optimization itself */
+ if (mvstat_search_type == MVSTAT_SEARCH_EXHAUSTIVE)
+ choose_mv_statistics_exhaustive(root, 0,
+ nmvstats, mvstats, stats_attnums,
+ nclauses, clauses_array, clauses_attnums,
+ nconditions, conditions_array, conditions_attnums,
+ clause_cover_map, condition_cover_map,
+ ruled_out, NULL, &best);
+ else
+ choose_mv_statistics_greedy(root, 0,
+ nmvstats, mvstats, stats_attnums,
+ nclauses, clauses_array, clauses_attnums,
+ nconditions, conditions_array, conditions_attnums,
+ clause_cover_map, condition_cover_map,
+ ruled_out, NULL, &best);
+
+ /* create a list of statistics from the array */
+ if (best != NULL)
+ {
+ for (i = 0; i < best->nstats; i++)
{
- /* are all the attributes part of the selected stats? */
- if (bms_is_subset(attnums, mvattnums))
- match = true;
+ MVStatisticInfo *info = makeNode(MVStatisticInfo);
+ memcpy(info, &mvstats[best->stats[i]], sizeof(MVStatisticInfo));
+ result = lappend(result, info);
}
- /*
- * The clause matches the selected stats, so put it to the list of
- * mv-compatible clauses. Otherwise, keep it in the list of 'regular'
- * clauses (that may be selected later).
- */
- if (match)
- *mvclauses = lappend(*mvclauses, clause);
- else
- non_mvclauses = lappend(non_mvclauses, clause);
+ pfree(best);
}
- /*
- * Perform regular estimation using the clauses incompatible with the chosen
- * histogram (or MV stats in general).
- */
- return non_mvclauses;
+ /* cleanup (maybe leave it up to the memory context?) */
+ for (i = 0; i < nmvstats; i++)
+ bms_free(stats_attnums[i]);
+
+ for (i = 0; i < nclauses; i++)
+ bms_free(clauses_attnums[i]);
+
+ for (i = 0; i < nconditions; i++)
+ bms_free(conditions_attnums[i]);
+
+ pfree(stats_attnums);
+ pfree(clauses_attnums);
+ pfree(conditions_attnums);
+ pfree(clauses_array);
+ pfree(conditions_array);
+ pfree(clause_cover_map);
+ pfree(condition_cover_map);
+ pfree(ruled_out);
+ pfree(mvstats);
+
+ list_free(clauses);
+ list_free(conditions);
+ list_free(stats);
+
+ return result;
}
typedef struct
@@ -1474,6 +2686,7 @@ clause_is_mv_compatible(Node *clause, Index relid, Bitmapset **attnums, int type
return true;
}
+
/*
* collect attnums from functional dependencies
*
@@ -2022,6 +3235,24 @@ clauselist_apply_dependencies(PlannerInfo *root, List *clauses,
* Check that there are stats with at least one of the requested types.
*/
static bool
+stats_type_matches(MVStatisticInfo *stat, int type)
+{
+ if ((type & MV_CLAUSE_TYPE_FDEP) && stat->deps_built)
+ return true;
+
+ if ((type & MV_CLAUSE_TYPE_MCV) && stat->mcv_built)
+ return true;
+
+ if ((type & MV_CLAUSE_TYPE_HIST) && stat->hist_built)
+ return true;
+
+ return false;
+}
+
+/*
+ * Check that there are stats with at least one of the requested types.
+ */
+static bool
has_stats(List *stats, int type)
{
ListCell *s;
@@ -2030,13 +3261,8 @@ has_stats(List *stats, int type)
{
MVStatisticInfo *stat = (MVStatisticInfo *)lfirst(s);
- if ((type & MV_CLAUSE_TYPE_FDEP) && stat->deps_built)
- return true;
-
- if ((type & MV_CLAUSE_TYPE_MCV) && stat->mcv_built)
- return true;
-
- if ((type & MV_CLAUSE_TYPE_HIST) && stat->hist_built)
+ /* terminate if we've found at least one matching statistics */
+ if (stats_type_matches(stat, type))
return true;
}
@@ -2087,22 +3313,26 @@ find_stats(PlannerInfo *root, Index relid)
* as the clauses are processed (and skip items that are 'match').
*/
static Selectivity
-clauselist_mv_selectivity_mcvlist(PlannerInfo *root, List *clauses,
- MVStatisticInfo *mvstats, bool *fullmatch,
- Selectivity *lowsel)
+clauselist_mv_selectivity_mcvlist(PlannerInfo *root, MVStatisticInfo *mvstats,
+ List *clauses, List *conditions, bool is_or,
+ bool *fullmatch, Selectivity *lowsel)
{
int i;
Selectivity s = 0.0;
+ Selectivity t = 0.0;
Selectivity u = 0.0;
MCVList mcvlist = NULL;
+
int nmatches = 0;
+ int nconditions = 0;
/* match/mismatch bitmap for each MCV item */
char * matches = NULL;
+ char * condition_matches = NULL;
Assert(clauses != NIL);
- Assert(list_length(clauses) >= 2);
+ Assert(list_length(clauses) >= 1);
/* there's no MCV list built yet */
if (! mvstats->mcv_built)
@@ -2113,32 +3343,85 @@ clauselist_mv_selectivity_mcvlist(PlannerInfo *root, List *clauses,
Assert(mcvlist != NULL);
Assert(mcvlist->nitems > 0);
- /* by default all the MCV items match the clauses fully */
- matches = palloc0(sizeof(char) * mcvlist->nitems);
- memset(matches, MVSTATS_MATCH_FULL, sizeof(char)*mcvlist->nitems);
-
/* number of matching MCV items */
nmatches = mcvlist->nitems;
+ nconditions = mcvlist->nitems;
+
+ /*
+ * Bitmap of bucket matches (mismatch, partial, full).
+ *
+ * For AND clauses all buckets match (and we'll eliminate them).
+ * For OR clauses no buckets match (and we'll add them).
+ *
+ * We only need to do the memset for AND clauses (for OR clauses
+ * it's already set correctly by the palloc0).
+ */
+ matches = palloc0(sizeof(char) * nmatches);
+
+ if (! is_or) /* AND-clause */
+ memset(matches, MVSTATS_MATCH_FULL, sizeof(char)*nmatches);
+ /* Conditions are treated as AND clause, so match by default. */
+ condition_matches = palloc0(sizeof(char) * nconditions);
+ memset(condition_matches, MVSTATS_MATCH_FULL, sizeof(char)*nconditions);
+
+ /*
+ * build the match bitmap for the conditions (conditions are always
+ * connected by AND)
+ */
+ if (conditions != NIL)
+ nconditions = update_match_bitmap_mcvlist(root, conditions,
+ mvstats->stakeys, mcvlist,
+ nconditions, condition_matches,
+ lowsel, fullmatch, false);
+
+ /*
+ * build the match bitmap for the estimated clauses
+ *
+ * TODO This evaluates the clauses for all MCV items, even those
+ * ruled out by the conditions. The final result should be the
+ * same, but it might be faster.
+ */
nmatches = update_match_bitmap_mcvlist(root, clauses,
mvstats->stakeys, mcvlist,
- nmatches, matches,
- lowsel, fullmatch, false);
+ ((is_or) ? 0 : nmatches), matches,
+ lowsel, fullmatch, is_or);
/* sum frequencies for all the matching MCV items */
for (i = 0; i < mcvlist->nitems; i++)
{
- /* used to 'scale' for MCV lists not covering all tuples */
+ /*
+ * Find out what part of the data is covered by the MCV list,
+ * so that we can 'scale' the selectivity properly (e.g. when
+ * only 50% of the sample items got into the MCV, and the rest
+ * is either in a histogram, or not covered by stats).
+ *
+ * TODO This might be handled by keeping a global "frequency"
+ * for the whole list, which might save us a bit of time
+ * spent on accessing the not-matching part of the MCV list.
+ * Although it's likely in a cache, so it's very fast.
+ */
u += mcvlist->items[i]->frequency;
+ /* skit MCV items not matching the conditions */
+ if (condition_matches[i] == MVSTATS_MATCH_NONE)
+ continue;
+
if (matches[i] != MVSTATS_MATCH_NONE)
s += mcvlist->items[i]->frequency;
+
+ t += mcvlist->items[i]->frequency;
}
pfree(matches);
+ pfree(condition_matches);
pfree(mcvlist);
- return s*u;
+ /* no condition matches */
+ if (t == 0.0)
+ return (Selectivity)0.0;
+
+ return (s / t) * u;
}
/*
@@ -2369,64 +3652,57 @@ update_match_bitmap_mcvlist(PlannerInfo *root, List *clauses,
}
}
}
- else if (or_clause(clause) || and_clause(clause))
+ else if (or_clause(clause) || and_clause(clause) || not_clause(clause))
{
/* AND/OR clause, with all clauses compatible with the selected MV stat */
int i;
- BoolExpr *orclause = ((BoolExpr*)clause);
- List *orclauses = orclause->args;
+ List *tmp_clauses = ((BoolExpr*)clause)->args;
/* match/mismatch bitmap for each MCV item */
- int or_nmatches = 0;
- char * or_matches = NULL;
+ int tmp_nmatches = 0;
+ char * tmp_matches = NULL;
- Assert(orclauses != NIL);
- Assert(list_length(orclauses) >= 2);
+ Assert(tmp_clauses != NIL);
+ Assert((list_length(tmp_clauses) >= 2) || (not_clause(clause) && (list_length(tmp_clauses)==1)));
/* number of matching MCV items */
- or_nmatches = mcvlist->nitems;
+ tmp_nmatches = (or_clause(clause)) ? 0 : mcvlist->nitems;
/* by default none of the MCV items matches the clauses */
- or_matches = palloc0(sizeof(char) * or_nmatches);
+ tmp_matches = palloc0(sizeof(char) * mcvlist->nitems);
- if (or_clause(clause))
- {
- /* OR clauses assume nothing matches, initially */
- memset(or_matches, MVSTATS_MATCH_NONE, sizeof(char)*or_nmatches);
- or_nmatches = 0;
- }
- else
- {
- /* AND clauses assume nothing matches, initially */
- memset(or_matches, MVSTATS_MATCH_FULL, sizeof(char)*or_nmatches);
- }
+ /* AND (and NOT) clauses assume everything matches, initially */
+ if (! or_clause(clause))
+ memset(tmp_matches, MVSTATS_MATCH_FULL, sizeof(char)*mcvlist->nitems);
/* build the match bitmap for the OR-clauses */
- or_nmatches = update_match_bitmap_mcvlist(root, orclauses,
+ tmp_nmatches = update_match_bitmap_mcvlist(root, tmp_clauses,
stakeys, mcvlist,
- or_nmatches, or_matches,
+ tmp_nmatches, tmp_matches,
lowsel, fullmatch, or_clause(clause));
/* merge the bitmap into the existing one*/
for (i = 0; i < mcvlist->nitems; i++)
{
+ /* if this is a NOT clause, we need to invert the results first */
+ if (not_clause(clause))
+ tmp_matches[i] = (MVSTATS_MATCH_FULL - tmp_matches[i]);
+
/*
* To AND-merge the bitmaps, a MIN() semantics is used.
* For OR-merge, use MAX().
*
* FIXME this does not decrease the number of matches
*/
- UPDATE_RESULT(matches[i], or_matches[i], is_or);
+ UPDATE_RESULT(matches[i], tmp_matches[i], is_or);
}
- pfree(or_matches);
+ pfree(tmp_matches);
}
else
- {
elog(ERROR, "unknown clause type: %d", clause->type);
- }
}
/*
@@ -2484,15 +3760,18 @@ update_match_bitmap_mcvlist(PlannerInfo *root, List *clauses,
* this is not uncommon, but for histograms it's not that clear.
*/
static Selectivity
-clauselist_mv_selectivity_histogram(PlannerInfo *root, List *clauses,
- MVStatisticInfo *mvstats)
+clauselist_mv_selectivity_histogram(PlannerInfo *root, MVStatisticInfo *mvstats,
+ List *clauses, List *conditions, bool is_or)
{
int i;
Selectivity s = 0.0;
+ Selectivity t = 0.0;
Selectivity u = 0.0;
int nmatches = 0;
+ int nconditions = 0;
char *matches = NULL;
+ char *condition_matches = NULL;
MVSerializedHistogram mvhist = NULL;
@@ -2505,25 +3784,55 @@ clauselist_mv_selectivity_histogram(PlannerInfo *root, List *clauses,
Assert (mvhist != NULL);
Assert (clauses != NIL);
- Assert (list_length(clauses) >= 2);
+ Assert (list_length(clauses) >= 1);
+
+ nmatches = mvhist->nbuckets;
+ nconditions = mvhist->nbuckets;
/*
- * Bitmap of bucket matches (mismatch, partial, full). by default
- * all buckets fully match (and we'll eliminate them).
+ * Bitmap of bucket matches (mismatch, partial, full).
+ *
+ * For AND clauses all buckets match (and we'll eliminate them).
+ * For OR clauses no buckets match (and we'll add them).
+ *
+ * We only need to do the memset for AND clauses (for OR clauses
+ * it's already set correctly by the palloc0).
*/
- matches = palloc0(sizeof(char) * mvhist->nbuckets);
- memset(matches, MVSTATS_MATCH_FULL, sizeof(char)*mvhist->nbuckets);
+ matches = palloc0(sizeof(char) * nmatches);
- nmatches = mvhist->nbuckets;
+ if (! is_or) /* AND-clause */
+ memset(matches, MVSTATS_MATCH_FULL, sizeof(char)*nmatches);
+
+ /* Conditions are treated as AND clause, so match by default. */
+ condition_matches = palloc0(sizeof(char)*nconditions);
+ memset(condition_matches, MVSTATS_MATCH_FULL, sizeof(char)*nconditions);
+
+ /*
+ * build the match bitmap for the conditions (conditions are always
+ * connected by AND)
+ */
+ if (conditions != NIL)
+ update_match_bitmap_histogram(root, conditions,
+ mvstats->stakeys, mvhist,
+ nconditions, condition_matches, false);
- /* build the match bitmap */
+ /*
+ * build the match bitmap for the estimated clauses
+ *
+ * TODO This evaluates the clauses for all buckets, even those
+ * ruled out by the conditions. The final result should be
+ * the same, but it might be faster.
+ */
update_match_bitmap_histogram(root, clauses,
mvstats->stakeys, mvhist,
- nmatches, matches, false);
+ ((is_or) ? 0 : nmatches), matches,
+ is_or);
/* now, walk through the buckets and sum the selectivities */
for (i = 0; i < mvhist->nbuckets; i++)
{
+ float coeff = 1.0;
+
/*
* Find out what part of the data is covered by the histogram,
* so that we can 'scale' the selectivity properly (e.g. when
@@ -2537,10 +3846,23 @@ clauselist_mv_selectivity_histogram(PlannerInfo *root, List *clauses,
*/
u += mvhist->buckets[i]->ntuples;
+ /* skip buckets not matching the conditions */
+ if (condition_matches[i] == MVSTATS_MATCH_NONE)
+ continue;
+ else if (condition_matches[i] == MVSTATS_MATCH_PARTIAL)
+ coeff = 0.5;
+
+ t += coeff * mvhist->buckets[i]->ntuples;
+
if (matches[i] == MVSTATS_MATCH_FULL)
- s += mvhist->buckets[i]->ntuples;
+ s += coeff * mvhist->buckets[i]->ntuples;
else if (matches[i] == MVSTATS_MATCH_PARTIAL)
- s += 0.5 * mvhist->buckets[i]->ntuples;
+ /*
+ * TODO If both conditions and clauses match partially, this
+ * will use 0.25 match - not sure if that's the right
+ * thing solution, but seems about right.
+ */
+ s += coeff * 0.5 * mvhist->buckets[i]->ntuples;
}
#ifdef DEBUG_MVHIST
@@ -2549,9 +3871,14 @@ clauselist_mv_selectivity_histogram(PlannerInfo *root, List *clauses,
/* release the allocated bitmap and deserialized histogram */
pfree(matches);
+ pfree(condition_matches);
pfree(mvhist);
- return s * u;
+ /* no condition matches */
+ if (t == 0.0)
+ return (Selectivity)0.0;
+
+ return (s / t) * u;
}
/* cached result of bucket boundary comparison for a single dimension */
@@ -2699,7 +4026,7 @@ update_match_bitmap_histogram(PlannerInfo *root, List *clauses,
{
int i;
ListCell * l;
-
+
/*
* Used for caching function calls, only once per deduplicated value.
*
@@ -2742,7 +4069,7 @@ update_match_bitmap_histogram(PlannerInfo *root, List *clauses,
FmgrInfo opproc; /* operator */
fmgr_info(get_opcode(expr->opno), &opproc);
-
+
/* reset the cache (per clause) */
memset(callcache, 0, mvhist->nbuckets);
@@ -2902,64 +4229,57 @@ update_match_bitmap_histogram(PlannerInfo *root, List *clauses,
UPDATE_RESULT(matches[i], MVSTATS_MATCH_NONE, is_or);
}
}
- else if (or_clause(clause) || and_clause(clause))
+ else if (or_clause(clause) || and_clause(clause) || not_clause(clause))
{
/* AND/OR clause, with all clauses compatible with the selected MV stat */
int i;
- BoolExpr *orclause = ((BoolExpr*)clause);
- List *orclauses = orclause->args;
+ List *tmp_clauses = ((BoolExpr*)clause)->args;
/* match/mismatch bitmap for each bucket */
- int or_nmatches = 0;
- char * or_matches = NULL;
+ int tmp_nmatches = 0;
+ char * tmp_matches = NULL;
- Assert(orclauses != NIL);
- Assert(list_length(orclauses) >= 2);
+ Assert(tmp_clauses != NIL);
+ Assert((list_length(tmp_clauses) >= 2) || (not_clause(clause) && (list_length(tmp_clauses)==1)));
/* number of matching buckets */
- or_nmatches = mvhist->nbuckets;
+ tmp_nmatches = (or_clause(clause)) ? 0 : mvhist->nbuckets;
- /* by default none of the buckets matches the clauses */
- or_matches = palloc0(sizeof(char) * or_nmatches);
+ /* by default none of the buckets matches the clauses (OR clause) */
+ tmp_matches = palloc0(sizeof(char) * mvhist->nbuckets);
- if (or_clause(clause))
- {
- /* OR clauses assume nothing matches, initially */
- memset(or_matches, MVSTATS_MATCH_NONE, sizeof(char)*or_nmatches);
- or_nmatches = 0;
- }
- else
- {
- /* AND clauses assume nothing matches, initially */
- memset(or_matches, MVSTATS_MATCH_FULL, sizeof(char)*or_nmatches);
- }
+ /* but AND (and NOT) clauses assume everything matches, initially */
+ if (! or_clause(clause))
+ memset(tmp_matches, MVSTATS_MATCH_FULL, sizeof(char)*mvhist->nbuckets);
/* build the match bitmap for the OR-clauses */
- or_nmatches = update_match_bitmap_histogram(root, orclauses,
+ tmp_nmatches = update_match_bitmap_histogram(root, tmp_clauses,
stakeys, mvhist,
- or_nmatches, or_matches, or_clause(clause));
+ tmp_nmatches, tmp_matches, or_clause(clause));
/* merge the bitmap into the existing one*/
for (i = 0; i < mvhist->nbuckets; i++)
{
+ /* if this is a NOT clause, we need to invert the results first */
+ if (not_clause(clause))
+ tmp_matches[i] = (MVSTATS_MATCH_FULL - tmp_matches[i]);
+
/*
* To AND-merge the bitmaps, a MIN() semantics is used.
* For OR-merge, use MAX().
*
* FIXME this does not decrease the number of matches
*/
- UPDATE_RESULT(matches[i], or_matches[i], is_or);
+ UPDATE_RESULT(matches[i], tmp_matches[i], is_or);
}
- pfree(or_matches);
-
+ pfree(tmp_matches);
}
else
elog(ERROR, "unknown clause type: %d", clause->type);
}
- /* free the call cache */
pfree(callcache);
return nmatches;
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index 5350329..57214e0 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -3518,7 +3518,8 @@ compute_semi_anti_join_factors(PlannerInfo *root,
joinquals,
0,
jointype,
- sjinfo);
+ sjinfo,
+ NIL);
/*
* Also get the normal inner-join selectivity of the join clauses.
@@ -3541,7 +3542,8 @@ compute_semi_anti_join_factors(PlannerInfo *root,
joinquals,
0,
JOIN_INNER,
- &norm_sjinfo);
+ &norm_sjinfo,
+ NIL);
/* Avoid leaking a lot of ListCells */
if (jointype == JOIN_ANTI)
@@ -3708,7 +3710,7 @@ approx_tuple_count(PlannerInfo *root, JoinPath *path, List *quals)
Node *qual = (Node *) lfirst(l);
/* Note that clause_selectivity will be able to cache its result */
- selec *= clause_selectivity(root, qual, 0, JOIN_INNER, &sjinfo);
+ selec *= clause_selectivity(root, qual, 0, JOIN_INNER, &sjinfo, NIL);
}
/* Apply it to the input relation sizes */
@@ -3744,7 +3746,8 @@ set_baserel_size_estimates(PlannerInfo *root, RelOptInfo *rel)
rel->baserestrictinfo,
0,
JOIN_INNER,
- NULL);
+ NULL,
+ NIL);
rel->rows = clamp_row_est(nrows);
@@ -3781,7 +3784,8 @@ get_parameterized_baserel_size(PlannerInfo *root, RelOptInfo *rel,
allclauses,
rel->relid, /* do not use 0! */
JOIN_INNER,
- NULL);
+ NULL,
+ NIL);
nrows = clamp_row_est(nrows);
/* For safety, make sure result is not more than the base estimate */
if (nrows > rel->rows)
@@ -3919,12 +3923,14 @@ calc_joinrel_size_estimate(PlannerInfo *root,
joinquals,
0,
jointype,
- sjinfo);
+ sjinfo,
+ NIL);
pselec = clauselist_selectivity(root,
pushedquals,
0,
jointype,
- sjinfo);
+ sjinfo,
+ NIL);
/* Avoid leaking a lot of ListCells */
list_free(joinquals);
@@ -3936,7 +3942,8 @@ calc_joinrel_size_estimate(PlannerInfo *root,
restrictlist,
0,
jointype,
- sjinfo);
+ sjinfo,
+ NIL);
pselec = 0.0; /* not used, keep compiler quiet */
}
diff --git a/src/backend/optimizer/util/orclauses.c b/src/backend/optimizer/util/orclauses.c
index ea831f5..6299e75 100644
--- a/src/backend/optimizer/util/orclauses.c
+++ b/src/backend/optimizer/util/orclauses.c
@@ -280,7 +280,7 @@ consider_new_or_clause(PlannerInfo *root, RelOptInfo *rel,
* saving work later.)
*/
or_selec = clause_selectivity(root, (Node *) or_rinfo,
- 0, JOIN_INNER, NULL);
+ 0, JOIN_INNER, NULL, NIL);
/*
* The clause is only worth adding to the query if it rejects a useful
@@ -342,7 +342,7 @@ consider_new_or_clause(PlannerInfo *root, RelOptInfo *rel,
/* Compute inner-join size */
orig_selec = clause_selectivity(root, (Node *) join_or_rinfo,
- 0, JOIN_INNER, &sjinfo);
+ 0, JOIN_INNER, &sjinfo, NIL);
/* And hack cached selectivity so join size remains the same */
join_or_rinfo->norm_selec = orig_selec / or_selec;
diff --git a/src/backend/utils/adt/selfuncs.c b/src/backend/utils/adt/selfuncs.c
index 46c95b0..7d0a3a1 100644
--- a/src/backend/utils/adt/selfuncs.c
+++ b/src/backend/utils/adt/selfuncs.c
@@ -1627,13 +1627,15 @@ booltestsel(PlannerInfo *root, BoolTestType booltesttype, Node *arg,
case IS_NOT_FALSE:
selec = (double) clause_selectivity(root, arg,
varRelid,
- jointype, sjinfo);
+ jointype, sjinfo,
+ NIL);
break;
case IS_FALSE:
case IS_NOT_TRUE:
selec = 1.0 - (double) clause_selectivity(root, arg,
varRelid,
- jointype, sjinfo);
+ jointype, sjinfo,
+ NIL);
break;
default:
elog(ERROR, "unrecognized booltesttype: %d",
@@ -6259,7 +6261,8 @@ genericcostestimate(PlannerInfo *root,
indexSelectivity = clauselist_selectivity(root, selectivityQuals,
index->rel->relid,
JOIN_INNER,
- NULL);
+ NULL,
+ NIL);
/*
* If caller didn't give us an estimate, estimate the number of index
@@ -6579,7 +6582,8 @@ btcostestimate(PlannerInfo *root, IndexPath *path, double loop_count,
btreeSelectivity = clauselist_selectivity(root, selectivityQuals,
index->rel->relid,
JOIN_INNER,
- NULL);
+ NULL,
+ NIL);
numIndexTuples = btreeSelectivity * index->rel->tuples;
/*
@@ -7330,7 +7334,8 @@ gincostestimate(PlannerInfo *root, IndexPath *path, double loop_count,
*indexSelectivity = clauselist_selectivity(root, selectivityQuals,
index->rel->relid,
JOIN_INNER,
- NULL);
+ NULL,
+ NIL);
/* fetch estimated page cost for tablespace containing index */
get_tablespace_page_costs(index->reltablespace,
@@ -7560,7 +7565,7 @@ brincostestimate(PlannerInfo *root, IndexPath *path, double loop_count,
*indexSelectivity =
clauselist_selectivity(root, indexQuals,
path->indexinfo->rel->relid,
- JOIN_INNER, NULL);
+ JOIN_INNER, NULL, NIL);
*indexCorrelation = 1;
/*
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index ea5a09a..27a8de5 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -75,6 +75,7 @@
#include "utils/bytea.h"
#include "utils/guc_tables.h"
#include "utils/memutils.h"
+#include "utils/mvstats.h"
#include "utils/pg_locale.h"
#include "utils/plancache.h"
#include "utils/portal.h"
@@ -393,6 +394,15 @@ static const struct config_enum_entry force_parallel_mode_options[] = {
};
/*
+ * Search algorithm for multivariate stats.
+ */
+static const struct config_enum_entry mvstat_search_options[] = {
+ {"greedy", MVSTAT_SEARCH_GREEDY, false},
+ {"exhaustive", MVSTAT_SEARCH_EXHAUSTIVE, false},
+ {NULL, 0, false}
+};
+
+/*
* Options for enum values stored in other modules
*/
extern const struct config_enum_entry wal_level_options[];
@@ -3707,6 +3717,16 @@ static struct config_enum ConfigureNamesEnum[] =
NULL, NULL, NULL
},
+ {
+ {"mvstat_search", PGC_USERSET, QUERY_TUNING_OTHER,
+ gettext_noop("Sets the algorithm used for combining multivariate stats."),
+ NULL
+ },
+ &mvstat_search_type,
+ MVSTAT_SEARCH_GREEDY, mvstat_search_options,
+ NULL, NULL, NULL
+ },
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, 0, NULL, NULL, NULL, NULL
diff --git a/src/backend/utils/mvstats/README.stats b/src/backend/utils/mvstats/README.stats
index 3e4f4d1..d404914 100644
--- a/src/backend/utils/mvstats/README.stats
+++ b/src/backend/utils/mvstats/README.stats
@@ -90,6 +90,137 @@ even attempting to do the more expensive estimation.
Whenever we find there are no suitable stats, we skip the expensive steps.
+Combining multiple statistics
+-----------------------------
+
+When estimating selectivity of a list of clauses, there may exist no statistics
+covering all of them. If there are multiple statistics, each covering some
+subset of the attributes, the optimizer needs to figure out which of those
+statistics to apply.
+
+When the statistics do not overlap, the solution is trivial - we can simply
+split the groups of conditions by the matching statistics, and then multiply the
+selectivities. For example assume multivariate statistics on (b,c) and (d,e),
+and a condition like this:
+
+ (a=1) AND (b=2) AND (c=3) AND (d=4) AND (e=5)
+
+Then (a=1) is not covered by any of the statistics, so will be estimated using
+the regular per-column statistics. The two conditions ((b=2) AND (c=3)) will be
+estimated using the (b,c) statistics, and ((d=4) AND (e=5)) will be estimated
+using (d,e) statistics. And the resulting selectivities will be estimated.
+
+Now, what if the statistics overlap? For example assume the same condition as
+above, but let's say we have statistics on (a,b,c) and (a,c,d,e). What then?
+
+As selectivity is just a probability that the condition holds for a random row,
+we can write the selectivity like this:
+
+ P(a=1 & b=2 & c=3 & d=4 & e=5)
+
+and we can rewrite it using conditional probability like this
+
+ P(a=1 & b=2 & c=3) * P(d=4 & e=5 | a=1 & b=2 & c=3)
+
+Notice that the first part already matches to (a,b,c) statistics. If we assume
+that columns that are not referenced by the same statistics are independent, we
+may rewrite the second half like this
+
+ P(d=4 & e=5 | a=1 & b=2 & c=3) = P(d=4 & e=5 | a=1 & c=3)
+
+which corresponds to the statistics on (a,c,d,e).
+
+If there are multiple statistics defined on a table, it's not difficult to come
+up with examples when there are multiple ways to combine them to cover a list of
+clauses. We need a way to find the best combination of statistics.
+
+This is the purpose of choose_mv_statistics(). It searches through the possible
+combinations of statistics, and searches such combination that
+
+ (a) covers the most clauses of the list
+
+ (b) reuses the maximum number of clauses as conditions
+ (in conditional probabilities)
+
+While (a) criteria seems natural, the (b) may seem a bit awkward at first. The
+idea is that conditions in a way of transfering information about dependencies
+between statistics.
+
+There are two alternative implementations of choose_mv_statistics() - greedy
+and exhaustive. Exhaustive actually searches through all possible combinations
+of statistics, and for larger numbers of statistics may get quite expensive
+(as it, unsurprisingly, has exponential cost). Greedy terminates in less than
+K steps (when K is the number of clauses), and in each step chooses the best
+next statistics. I've been unable to come up with an example where those two
+approaches would produce different combinations.
+
+It's possible to choose the optimization using mvstat_search_type, with either
+'greedy' or 'exhaustive' values (default is 'greedy').
+
+ SET mvstat_search_type = 'exhaustive';
+
+Note: This is meant mostly for experimentation. I do expect we'll choose one of
+the algorithms and remove the GUC before commit.
+
+
+Limitations of combining statistics
+-----------------------------------
+
+As described in the section 'Combining multiple statistics', the current appoach
+is based on transfering information between statistics by means of conditional
+probabilities. This is a relatively cheap and efficient approach, but it is
+based on two assumptions:
+
+ (1) The overlap between the statistics needs to be sufficiently large, i.e.
+ there needs to be enough columns shared by the statistics to transfer
+ information about dependencies between the remaining columns.
+
+ (2) The query needs to include sufficient clauses on the shared columns.
+
+How a violation of those assumptions may be a problem can be illustrated by
+a simple example. Assume a table with three columns (a,b,c) containing exactly
+the same values, and statistics on (a,b) and (b,c):
+
+ CREATE TABLE test AS SELECT i, i, i
+ FROM generate_series(1,1000);
+
+ CREATE STATISTICS s1 ON test (a,b) WITH (mcv);
+ CREATE STATISTICS s2 ON test (b,c) WITH (mcv);
+
+ ANALYZE test;
+
+First, let's estimate this query:
+
+ SELECT * FROM test WHERE (a < 10) AND (c < 10);
+
+Clearly, there are no conditions on 'b' (which is the only column shared by the
+two statistics), so we'll end up with an estimate based on assumption of
+independence:
+
+ P(a < 10) * P(c < 10) = 0.01 * 0.01 = 0.0001
+
+Which is a significant under-estimate, as the proper selectivity is 0.01.
+
+But let's estimate another query:
+
+ SELECT * FROM test WHERE (a < 10) AND (b < 500) AND (c < 10);
+
+In this case, the estimate may be computed for example like this:
+
+ P[(a < 10) & (b < 500) & (c < 10)]
+ = P[(a < 10) & (b < 500)] * P[(c < 10) | (a < 10) & (b < 500)]
+ = P[(a < 10) & (b < 500)] * P[(c < 10) | (b < 500)]
+
+The trouble is the probability P(c < 10 | b < 500) evaluates to 0.02, because
+we have assumed (a) and (c) are independent because there is no statistic
+containing both these columns, and the condition on (b) does not transfer
+sufficient amount of information between the two statistics.
+
+Currently, the only solution is to build statistics on all three columns, but
+see the 'combining statistics using convolution' section for ideas on how to
+improve this.
+
+
Further (possibly crazy) ideas
------------------------------
@@ -111,3 +242,38 @@ But of course, this may result in expensive estimation (CPU-wise).
So we might add a GUC to choose between a simple (single statistics) and thus
multi-statistic estimation, possibly table-level parameter (ALTER TABLE ...).
+
+
+Combining stats using convolution
+---------------------------------
+
+While the current approach for combining statistics is based on conditional
+probabilities, and thus only works when the query includes conditions on the
+overlapping parts of the statistics. But there may be other ways to combine
+statistics, relaxing this requirement.
+
+Let's assume two histograms H1 and H2 - then combining them might work about
+like this:
+
+
+ for (buckets of H1, satisfying local conditions)
+ {
+ for (buckets of H2, overlapping with H1 bucket)
+ {
+ mark H2 bucket as 'valid'
+ }
+ }
+
+ s1 = s2 = 0.0
+ for (buckets of H2 marked as valid)
+ {
+ s1 += frequency
+
+ if (bucket satistifes local conditions)
+ s2 += frequency
+ }
+
+ s = (s2 / s1) /* final selectivity estimate */
+
+However this may quickly get non-trivial, e.g. when combining two statistics
+of different types (histogram vs. MCV).
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index fea2bb7..33f5a1b 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -192,11 +192,13 @@ extern Selectivity clauselist_selectivity(PlannerInfo *root,
List *clauses,
int varRelid,
JoinType jointype,
- SpecialJoinInfo *sjinfo);
+ SpecialJoinInfo *sjinfo,
+ List *conditions);
extern Selectivity clause_selectivity(PlannerInfo *root,
Node *clause,
int varRelid,
JoinType jointype,
- SpecialJoinInfo *sjinfo);
+ SpecialJoinInfo *sjinfo,
+ List *conditions);
#endif /* COST_H */
diff --git a/src/include/utils/mvstats.h b/src/include/utils/mvstats.h
index f05a517..35b2f8e 100644
--- a/src/include/utils/mvstats.h
+++ b/src/include/utils/mvstats.h
@@ -17,6 +17,14 @@
#include "fmgr.h"
#include "commands/vacuum.h"
+typedef enum MVStatSearchType
+{
+ MVSTAT_SEARCH_EXHAUSTIVE, /* exhaustive search */
+ MVSTAT_SEARCH_GREEDY /* greedy search */
+} MVStatSearchType;
+
+extern int mvstat_search_type;
+
/*
* Degree of how much MCV item / histogram bucket matches a clause.
* This is then considered when computing the selectivity.
--
2.1.0
[text/x-patch] 0007-multivariate-ndistinct-coefficients.patch (28.5K, ../../1457535779.24545.71.camel@2ndquadrant.com/8-0007-multivariate-ndistinct-coefficients.patch)
download | inline diff:
From d9b0afd75f2f678079d50f3d520bdd478c75bc89 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@pgaddict.com>
Date: Wed, 23 Dec 2015 02:07:58 +0100
Subject: [PATCH 7/9] multivariate ndistinct coefficients
---
doc/src/sgml/ref/create_statistics.sgml | 9 ++
src/backend/catalog/system_views.sql | 3 +-
src/backend/commands/analyze.c | 2 +-
src/backend/commands/statscmds.c | 11 +-
src/backend/optimizer/path/clausesel.c | 4 +
src/backend/optimizer/util/plancat.c | 4 +-
src/backend/utils/adt/selfuncs.c | 93 +++++++++++++++-
src/backend/utils/mvstats/Makefile | 2 +-
src/backend/utils/mvstats/README.ndistinct | 83 ++++++++++++++
src/backend/utils/mvstats/README.stats | 2 +
src/backend/utils/mvstats/common.c | 23 +++-
src/backend/utils/mvstats/mvdist.c | 171 +++++++++++++++++++++++++++++
src/include/catalog/pg_mv_statistic.h | 26 +++--
src/include/nodes/relation.h | 2 +
src/include/utils/mvstats.h | 9 +-
src/test/regress/expected/rules.out | 3 +-
16 files changed, 424 insertions(+), 23 deletions(-)
create mode 100644 src/backend/utils/mvstats/README.ndistinct
create mode 100644 src/backend/utils/mvstats/mvdist.c
diff --git a/doc/src/sgml/ref/create_statistics.sgml b/doc/src/sgml/ref/create_statistics.sgml
index fd3382e..80360a6 100644
--- a/doc/src/sgml/ref/create_statistics.sgml
+++ b/doc/src/sgml/ref/create_statistics.sgml
@@ -168,6 +168,15 @@ CREATE STATISTICS [ IF NOT EXISTS ] <replaceable class="PARAMETER">statistics_na
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>ndistinct</> (<type>boolean</>)</term>
+ <listitem>
+ <para>
+ Enables ndistinct coefficients for the statistics.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect2>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 6afdee0..a550141 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -169,7 +169,8 @@ CREATE VIEW pg_mv_stats AS
length(S.stamcv) AS mcvbytes,
pg_mv_stats_mcvlist_info(S.stamcv) AS mcvinfo,
length(S.stahist) AS histbytes,
- pg_mv_stats_histogram_info(S.stahist) AS histinfo
+ pg_mv_stats_histogram_info(S.stahist) AS histinfo,
+ standcoeff AS ndcoeff
FROM (pg_mv_statistic S JOIN pg_class C ON (C.oid = S.starelid))
LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace);
diff --git a/src/backend/commands/analyze.c b/src/backend/commands/analyze.c
index 9087532..c29f1be 100644
--- a/src/backend/commands/analyze.c
+++ b/src/backend/commands/analyze.c
@@ -582,7 +582,7 @@ do_analyze_rel(Relation onerel, int options, VacuumParams *params,
}
/* Build multivariate stats (if there are any). */
- build_mv_stats(onerel, numrows, rows, attr_cnt, vacattrstats);
+ build_mv_stats(onerel, totalrows, numrows, rows, attr_cnt, vacattrstats);
}
/*
diff --git a/src/backend/commands/statscmds.c b/src/backend/commands/statscmds.c
index e2f3ff1..11de1c5 100644
--- a/src/backend/commands/statscmds.c
+++ b/src/backend/commands/statscmds.c
@@ -72,7 +72,8 @@ CreateStatistics(CreateStatsStmt *stmt)
/* by default build nothing */
bool build_dependencies = false,
build_mcv = false,
- build_histogram = false;
+ build_histogram = false,
+ build_ndistinct = false;
int32 max_buckets = -1,
max_mcv_items = -1;
@@ -155,6 +156,8 @@ CreateStatistics(CreateStatsStmt *stmt)
if (strcmp(opt->defname, "dependencies") == 0)
build_dependencies = defGetBoolean(opt);
+ else if (strcmp(opt->defname, "ndistinct") == 0)
+ build_ndistinct = defGetBoolean(opt);
else if (strcmp(opt->defname, "mcv") == 0)
build_mcv = defGetBoolean(opt);
else if (strcmp(opt->defname, "max_mcv_items") == 0)
@@ -209,10 +212,10 @@ CreateStatistics(CreateStatsStmt *stmt)
}
/* check that at least some statistics were requested */
- if (! (build_dependencies || build_mcv || build_histogram))
+ if (! (build_dependencies || build_mcv || build_histogram || build_ndistinct))
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("no statistics type (dependencies, mcv, histogram) was requested")));
+ errmsg("no statistics type (dependencies, mcv, histogram, ndistinct) was requested")));
/* now do some checking of the options */
if (require_mcv && (! build_mcv))
@@ -246,6 +249,7 @@ CreateStatistics(CreateStatsStmt *stmt)
values[Anum_pg_mv_statistic_deps_enabled -1] = BoolGetDatum(build_dependencies);
values[Anum_pg_mv_statistic_mcv_enabled -1] = BoolGetDatum(build_mcv);
values[Anum_pg_mv_statistic_hist_enabled -1] = BoolGetDatum(build_histogram);
+ values[Anum_pg_mv_statistic_ndist_enabled-1] = BoolGetDatum(build_ndistinct);
values[Anum_pg_mv_statistic_mcv_max_items -1] = Int32GetDatum(max_mcv_items);
values[Anum_pg_mv_statistic_hist_max_buckets -1] = Int32GetDatum(max_buckets);
@@ -253,6 +257,7 @@ CreateStatistics(CreateStatsStmt *stmt)
nulls[Anum_pg_mv_statistic_stadeps -1] = true;
nulls[Anum_pg_mv_statistic_stamcv -1] = true;
nulls[Anum_pg_mv_statistic_stahist -1] = true;
+ nulls[Anum_pg_mv_statistic_standist -1] = true;
/* insert the tuple into pg_mv_statistic */
mvstatrel = heap_open(MvStatisticRelationId, RowExclusiveLock);
diff --git a/src/backend/optimizer/path/clausesel.c b/src/backend/optimizer/path/clausesel.c
index c1b8999..2540da9 100644
--- a/src/backend/optimizer/path/clausesel.c
+++ b/src/backend/optimizer/path/clausesel.c
@@ -59,6 +59,7 @@ static void addRangeClause(RangeQueryClause **rqlist, Node *clause,
#define MV_CLAUSE_TYPE_FDEP 0x01
#define MV_CLAUSE_TYPE_MCV 0x02
#define MV_CLAUSE_TYPE_HIST 0x04
+#define MV_CLAUSE_TYPE_NDIST 0x08
static bool clause_is_mv_compatible(Node *clause, Index relid, Bitmapset **attnums,
int type);
@@ -3246,6 +3247,9 @@ stats_type_matches(MVStatisticInfo *stat, int type)
if ((type & MV_CLAUSE_TYPE_HIST) && stat->hist_built)
return true;
+ if ((type & MV_CLAUSE_TYPE_NDIST) && stat->ndist_built)
+ return true;
+
return false;
}
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index 2519249..3741b7a 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -412,7 +412,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
mvstat = (Form_pg_mv_statistic) GETSTRUCT(htup);
/* unavailable stats are not interesting for the planner */
- if (mvstat->deps_built || mvstat->mcv_built || mvstat->hist_built)
+ if (mvstat->deps_built || mvstat->mcv_built || mvstat->hist_built || mvstat->ndist_built)
{
info = makeNode(MVStatisticInfo);
@@ -423,11 +423,13 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->deps_enabled = mvstat->deps_enabled;
info->mcv_enabled = mvstat->mcv_enabled;
info->hist_enabled = mvstat->hist_enabled;
+ info->ndist_enabled = mvstat->ndist_enabled;
/* built/available statistics */
info->deps_built = mvstat->deps_built;
info->mcv_built = mvstat->mcv_built;
info->hist_built = mvstat->hist_built;
+ info->ndist_built = mvstat->ndist_built;
/* stakeys */
adatum = SysCacheGetAttr(MVSTATOID, htup,
diff --git a/src/backend/utils/adt/selfuncs.c b/src/backend/utils/adt/selfuncs.c
index 7d0a3a1..a84dd2b 100644
--- a/src/backend/utils/adt/selfuncs.c
+++ b/src/backend/utils/adt/selfuncs.c
@@ -132,6 +132,7 @@
#include "utils/fmgroids.h"
#include "utils/index_selfuncs.h"
#include "utils/lsyscache.h"
+#include "utils/mvstats.h"
#include "utils/nabstime.h"
#include "utils/pg_locale.h"
#include "utils/rel.h"
@@ -206,6 +207,7 @@ static Const *string_to_const(const char *str, Oid datatype);
static Const *string_to_bytea_const(const char *str, size_t str_len);
static List *add_predicate_to_quals(IndexOptInfo *index, List *indexQuals);
+static Oid find_ndistinct_coeff(PlannerInfo *root, RelOptInfo *rel, List *varinfos);
/*
* eqsel - Selectivity of "=" for any data types.
@@ -3422,12 +3424,26 @@ estimate_num_groups(PlannerInfo *root, List *groupExprs, double input_rows,
* don't know by how much. We should never clamp to less than the
* largest ndistinct value for any of the Vars, though, since
* there will surely be at least that many groups.
+ *
+ * However we don't need to do this if we have ndistinct stats on
+ * the columns - in that case we can simply use the coefficient
+ * to get the (probably way more accurate) estimate.
+ *
+ * XXX Probably needs refactoring (don't like to mix with clamp
+ * and coeff at the same time).
*/
double clamp = rel->tuples;
+ double coeff = 1.0;
if (relvarcount > 1)
{
- clamp *= 0.1;
+ Oid oid = find_ndistinct_coeff(root, rel, varinfos);
+
+ if (oid != InvalidOid)
+ coeff = load_mv_ndistinct(oid);
+ else
+ clamp *= 0.1;
+
if (clamp < relmaxndistinct)
{
clamp = relmaxndistinct;
@@ -3436,6 +3452,13 @@ estimate_num_groups(PlannerInfo *root, List *groupExprs, double input_rows,
clamp = rel->tuples;
}
}
+
+ /*
+ * Apply ndistinct coefficient from multivar stats (we must do this
+ * before clamping the estimate in any way.
+ */
+ reldistinct /= coeff;
+
if (reldistinct > clamp)
reldistinct = clamp;
@@ -7582,3 +7605,71 @@ brincostestimate(PlannerInfo *root, IndexPath *path, double loop_count,
/* XXX what about pages_per_range? */
}
+
+/*
+ * Find applicable ndistinct statistics and compute the coefficient to
+ * correct the estimate (simply a product of per-column ndistincts).
+ *
+ * Currently we only look for a perfect match, i.e. a single ndistinct
+ * estimate exactly matching all the columns of the statistics.
+ */
+static Oid
+find_ndistinct_coeff(PlannerInfo *root, RelOptInfo *rel, List *varinfos)
+{
+ ListCell *lc;
+ Bitmapset *attnums = NULL;
+ VariableStatData vardata;
+
+ foreach(lc, varinfos)
+ {
+ GroupVarInfo *varinfo = (GroupVarInfo *) lfirst(lc);
+
+ if (varinfo->rel != rel)
+ continue;
+
+ /* FIXME handle expressions in general only */
+
+ /*
+ * examine the variable (or expression) so that we know which
+ * attribute we're dealing with - we need this for matching the
+ * ndistinct coefficient
+ *
+ * FIXME probably might remember this from estimate_num_groups
+ */
+ examine_variable(root, varinfo->var, 0, &vardata);
+
+ if (HeapTupleIsValid(vardata.statsTuple))
+ {
+ Form_pg_statistic stats
+ = (Form_pg_statistic) GETSTRUCT(vardata.statsTuple);
+
+ attnums = bms_add_member(attnums, stats->staattnum);
+
+ ReleaseVariableStats(vardata);
+ }
+ }
+
+ /* look for a matching ndistinct statistics */
+ foreach (lc, rel->mvstatlist)
+ {
+ int i;
+ MVStatisticInfo *info = (MVStatisticInfo *)lfirst(lc);
+
+ /* skip statistics without ndistinct coefficient built */
+ if (!info->ndist_built)
+ continue;
+
+ /* only exact matches for now (same set of columns) */
+ if (bms_num_members(attnums) != info->stakeys->dim1)
+ continue;
+
+ /* check that the columns match */
+ for (i = 0; i < info->stakeys->dim1; i++)
+ if (bms_is_member(info->stakeys->values[i], attnums))
+ continue;
+
+ return info->mvoid;
+ }
+
+ return InvalidOid;
+}
diff --git a/src/backend/utils/mvstats/Makefile b/src/backend/utils/mvstats/Makefile
index 9dbb3b6..d4b88e9 100644
--- a/src/backend/utils/mvstats/Makefile
+++ b/src/backend/utils/mvstats/Makefile
@@ -12,6 +12,6 @@ subdir = src/backend/utils/mvstats
top_builddir = ../../../..
include $(top_builddir)/src/Makefile.global
-OBJS = common.o dependencies.o histogram.o mcv.o
+OBJS = common.o dependencies.o histogram.o mcv.o mvdist.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/utils/mvstats/README.ndistinct b/src/backend/utils/mvstats/README.ndistinct
new file mode 100644
index 0000000..32d1624
--- /dev/null
+++ b/src/backend/utils/mvstats/README.ndistinct
@@ -0,0 +1,83 @@
+ndistinct coefficients
+======================
+
+Estimating number of distinct groups in a combination of columns is tricky,
+and the estimation error is often significant. By ndistinct coefficient we
+mean a ratio
+
+ q = ndistinct(a) * ndistinct(b) / ndistinct(a,b)
+
+where 'a' and 'b' are columns, ndistinct(a) is (an estimate of) a number of
+distinct values in column 'a'. And ndistinct(a,b) is the same thing for the
+pair of columns.
+
+The meaning of the coefficient may be illustrated by answering the following
+question: Given a combination of columns (a,b), how many distinct values of 'b'
+matches a chosen value of 'a' on average?
+
+Let's assume we know ndistinct(a) and ndistinct(a,b). Then the answer to the
+question clearly is
+
+ ndistinct(a,b) / ndistinct(a)
+
+and by using 'q' we may rewrite this as
+
+ ndistinct(b) / q
+
+so 'q' may be considered as a correction factor of the ndistinct estimate given
+a condition on one of the columns.
+
+This may be generalized to a combination of 'n' columns
+
+ [ndistinct(c1) * ... * ndistinct(cn)] / ndistinct(c1, ..., cn)
+
+and the meaning is very similar, except that we need to use conditions on (n-1)
+of the columns.
+
+
+Selectivity estimation
+----------------------
+
+As explained in the previous paragraph, ndistinct coefficients may be used to
+estimate cardinality of a column, given some apriori knowledge. Let's assume
+we need to estimate selectivity of a condition
+
+ (a=1) AND (b=2)
+
+which we can expand like this
+
+ P(a=1 & b=2) = P(a=1) * P(b=2 | a=1)
+
+Let's also assume that the distribution of 'b' is uniform, i.e. that
+
+ P(a=1) = 1/ndistinct(a)
+ P(b=2) = 1/ndistinct(b)
+ P(a=1 & b=2) = 1/ndistinct(a,b)
+
+ P(b=2 | a=1) = ndistinct(a) / ndistinct(a,b)
+
+which may be rewritten like
+
+ P(b=2 | a=1)
+ = ndistinct(a,b) / ndistinct(a)
+ = (1/ndistinct(b)) * [(ndistinct(a) * ndistinct(b)) / ndistinct(a,b)]
+ = (1/ndistinct(b)) * q
+
+and therefore
+
+ P(a=1 & b=2) = (1/ndistinct(a)) * (1/ndistinct(b)) * q
+
+This also illustrates 'q' as a correction coefficient.
+
+It also explains why we store the coefficient and not simply ndistinct(a,b).
+This way we can simply estimate individual clauses and then simply correct
+the estimate by multiplying the result with 'q' - we don't have to mess with
+ndistinct estimates at all.
+
+Naturally, as the coefficient is derives from ndistinct(a,b), it may be also
+used to estimate GROUP BY clauses on the combination of columns, replacing the
+existing heuristics in estimate_num_groups().
+
+Note: Currently only the GROUP BY estimation is implemented. It's a bit unclear
+how to implement the clause estimation when there are other statistics (esp.
+MCV lists and/or functional dependencies) available.
diff --git a/src/backend/utils/mvstats/README.stats b/src/backend/utils/mvstats/README.stats
index d404914..6d4b09b 100644
--- a/src/backend/utils/mvstats/README.stats
+++ b/src/backend/utils/mvstats/README.stats
@@ -20,6 +20,8 @@ Currently we only have two kinds of multivariate statistics
(c) multivariate histograms (README.histogram)
+ (d) ndistinct coefficients
+
Compatible clause types
-----------------------
diff --git a/src/backend/utils/mvstats/common.c b/src/backend/utils/mvstats/common.c
index ffb76f4..2be980d 100644
--- a/src/backend/utils/mvstats/common.c
+++ b/src/backend/utils/mvstats/common.c
@@ -32,7 +32,8 @@ static List* list_mv_stats(Oid relid);
* and serializes them back into the catalog (as bytea values).
*/
void
-build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
+build_mv_stats(Relation onerel, double totalrows,
+ int numrows, HeapTuple *rows,
int natts, VacAttrStats **vacattrstats)
{
ListCell *lc;
@@ -53,6 +54,7 @@ build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
MVDependencies deps = NULL;
MCVList mcvlist = NULL;
MVHistogram histogram = NULL;
+ double ndist = -1;
int numrows_filtered = numrows;
VacAttrStats **stats = NULL;
@@ -92,6 +94,9 @@ build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
if (stat->deps_enabled)
deps = build_mv_dependencies(numrows, rows, attrs, stats);
+ if (stat->ndist_enabled)
+ ndist = build_mv_ndistinct(totalrows, numrows, rows, attrs, stats);
+
/* build the MCV list */
if (stat->mcv_enabled)
mcvlist = build_mv_mcvlist(numrows, rows, attrs, stats, &numrows_filtered);
@@ -101,7 +106,7 @@ build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
histogram = build_mv_histogram(numrows_filtered, rows, attrs, stats, numrows);
/* store the histogram / MCV list in the catalog */
- update_mv_stats(stat->mvoid, deps, mcvlist, histogram, attrs, stats);
+ update_mv_stats(stat->mvoid, deps, mcvlist, histogram, ndist, attrs, stats);
}
}
@@ -183,6 +188,8 @@ list_mv_stats(Oid relid)
info->mcv_built = stats->mcv_built;
info->hist_enabled = stats->hist_enabled;
info->hist_built = stats->hist_built;
+ info->ndist_enabled = stats->ndist_enabled;
+ info->ndist_built = stats->ndist_built;
result = lappend(result, info);
}
@@ -252,7 +259,7 @@ find_mv_attnums(Oid mvoid, Oid *relid)
void
update_mv_stats(Oid mvoid,
MVDependencies dependencies, MCVList mcvlist, MVHistogram histogram,
- int2vector *attrs, VacAttrStats **stats)
+ double ndistcoeff, int2vector *attrs, VacAttrStats **stats)
{
HeapTuple stup,
oldtup;
@@ -292,26 +299,36 @@ update_mv_stats(Oid mvoid,
= PointerGetDatum(data);
}
+ if (ndistcoeff > 1.0)
+ {
+ nulls[Anum_pg_mv_statistic_standist -1] = false;
+ values[Anum_pg_mv_statistic_standist-1] = Float8GetDatum(ndistcoeff);
+ }
+
/* always replace the value (either by bytea or NULL) */
replaces[Anum_pg_mv_statistic_stadeps -1] = true;
replaces[Anum_pg_mv_statistic_stamcv -1] = true;
replaces[Anum_pg_mv_statistic_stahist-1] = true;
+ replaces[Anum_pg_mv_statistic_standist-1] = true;
/* always change the availability flags */
nulls[Anum_pg_mv_statistic_deps_built -1] = false;
nulls[Anum_pg_mv_statistic_mcv_built -1] = false;
nulls[Anum_pg_mv_statistic_hist_built-1] = false;
+ nulls[Anum_pg_mv_statistic_ndist_built-1] = false;
nulls[Anum_pg_mv_statistic_stakeys-1] = false;
/* use the new attnums, in case we removed some dropped ones */
replaces[Anum_pg_mv_statistic_deps_built-1] = true;
replaces[Anum_pg_mv_statistic_mcv_built -1] = true;
+ replaces[Anum_pg_mv_statistic_ndist_built-1] = true;
replaces[Anum_pg_mv_statistic_hist_built -1] = true;
replaces[Anum_pg_mv_statistic_stakeys -1] = true;
values[Anum_pg_mv_statistic_deps_built-1] = BoolGetDatum(dependencies != NULL);
values[Anum_pg_mv_statistic_mcv_built -1] = BoolGetDatum(mcvlist != NULL);
values[Anum_pg_mv_statistic_hist_built -1] = BoolGetDatum(histogram != NULL);
+ values[Anum_pg_mv_statistic_ndist_built-1] = BoolGetDatum(ndistcoeff > 1.0);
values[Anum_pg_mv_statistic_stakeys -1] = PointerGetDatum(attrs);
/* Is there already a pg_mv_statistic tuple for this attribute? */
diff --git a/src/backend/utils/mvstats/mvdist.c b/src/backend/utils/mvstats/mvdist.c
new file mode 100644
index 0000000..59b8358
--- /dev/null
+++ b/src/backend/utils/mvstats/mvdist.c
@@ -0,0 +1,171 @@
+/*-------------------------------------------------------------------------
+ *
+ * mvdist.c
+ * POSTGRES multivariate distinct coefficients
+ *
+ *
+ * Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/utils/mvstats/mvdist.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include <math.h>
+
+#include "common.h"
+#include "utils/lsyscache.h"
+
+static double estimate_ndistinct(double totalrows, int numrows, int d, int f1);
+
+/*
+ * Compute ndistinct coefficient for the combination of attributes. This
+ * computes the ndistinct estimate using the same estimator used in analyze.c
+ * and then computes the coefficient.
+ */
+double
+build_mv_ndistinct(double totalrows, int numrows, HeapTuple *rows,
+ int2vector *attrs, VacAttrStats **stats)
+{
+ int i, j;
+ int f1, cnt, d;
+ int nmultiple, summultiple;
+ int numattrs = attrs->dim1;
+ MultiSortSupport mss = multi_sort_init(numattrs);
+ double ndistcoeff;
+
+ /*
+ * It's possible to sort the sample rows directly, but this seemed
+ * somehow simpler / less error prone. Another option would be to
+ * allocate the arrays for each SortItem separately, but that'd be
+ * significant overhead (not just CPU, but especially memory bloat).
+ */
+ SortItem * items = (SortItem*)palloc0(numrows * sizeof(SortItem));
+
+ Datum *values = (Datum*)palloc0(sizeof(Datum) * numrows * numattrs);
+ bool *isnull = (bool*)palloc0(sizeof(bool) * numrows * numattrs);
+
+ for (i = 0; i < numrows; i++)
+ {
+ items[i].values = &values[i * numattrs];
+ items[i].isnull = &isnull[i * numattrs];
+ }
+
+ Assert(numattrs >= 2);
+
+ for (i = 0; i < numattrs; i++)
+ {
+ /* prepare the sort function for the first dimension */
+ multi_sort_add_dimension(mss, i, i, stats);
+
+ /* accumulate all the data into the array and sort it */
+ for (j = 0; j < numrows; j++)
+ {
+ items[j].values[i]
+ = heap_getattr(rows[j], attrs->values[i],
+ stats[i]->tupDesc, &items[j].isnull[i]);
+ }
+ }
+
+ qsort_arg((void *) items, numrows, sizeof(SortItem),
+ multi_sort_compare, mss);
+
+ /* count number of distinct combinations */
+
+ f1 = 0;
+ cnt = 1;
+ d = 1;
+ for (i = 1; i < numrows; i++)
+ {
+ if (multi_sort_compare(&items[i], &items[i-1], mss) != 0)
+ {
+ if (cnt == 1)
+ f1 += 1;
+ else
+ {
+ nmultiple += 1;
+ summultiple += cnt;
+ }
+
+ d++;
+ cnt = 0;
+ }
+
+ cnt += 1;
+ }
+
+ if (cnt == 1)
+ f1 += 1;
+ else
+ {
+ nmultiple += 1;
+ summultiple += cnt;
+ }
+
+ ndistcoeff = 1 / estimate_ndistinct(totalrows, numrows, d, f1);
+
+ /*
+ * now count distinct values for each attribute and incrementally
+ * compute ndistinct(a,b) / (ndistinct(a) * ndistinct(b))
+ *
+ * FIXME Probably need to handle cases when one of the ndistinct
+ * estimates is negative, and also check that the combined
+ * ndistinct is greater than any of those partial values.
+ */
+ for (i = 0; i < numattrs; i++)
+ ndistcoeff *= stats[i]->stadistinct;
+
+ return ndistcoeff;
+}
+
+double
+load_mv_ndistinct(Oid mvoid)
+{
+ bool isnull = false;
+ Datum deps;
+
+ /* Prepare to scan pg_mv_statistic for entries having indrelid = this rel. */
+ HeapTuple htup = SearchSysCache1(MVSTATOID, ObjectIdGetDatum(mvoid));
+
+#ifdef USE_ASSERT_CHECKING
+ Form_pg_mv_statistic mvstat = (Form_pg_mv_statistic) GETSTRUCT(htup);
+ Assert(mvstat->ndist_enabled && mvstat->ndist_built);
+#endif
+
+ deps = SysCacheGetAttr(MVSTATOID, htup,
+ Anum_pg_mv_statistic_standist, &isnull);
+
+ Assert(!isnull);
+
+ ReleaseSysCache(htup);
+
+ return DatumGetFloat8(deps);
+}
+
+/* The Duj1 estimator (already used in analyze.c). */
+static double
+estimate_ndistinct(double totalrows, int numrows, int d, int f1)
+{
+ double numer,
+ denom,
+ ndistinct;
+
+ numer = (double) numrows *(double) d;
+
+ denom = (double) (numrows - f1) +
+ (double) f1 * (double) numrows / totalrows;
+
+ ndistinct = numer / denom;
+
+ /* Clamp to sane range in case of roundoff error */
+ if (ndistinct < (double) d)
+ ndistinct = (double) d;
+
+ if (ndistinct > totalrows)
+ ndistinct = totalrows;
+
+ return floor(ndistinct + 0.5);
+}
diff --git a/src/include/catalog/pg_mv_statistic.h b/src/include/catalog/pg_mv_statistic.h
index 37f473f..e46cc6b 100644
--- a/src/include/catalog/pg_mv_statistic.h
+++ b/src/include/catalog/pg_mv_statistic.h
@@ -40,6 +40,7 @@ CATALOG(pg_mv_statistic,3381)
bool deps_enabled; /* analyze dependencies? */
bool mcv_enabled; /* build MCV list? */
bool hist_enabled; /* build histogram? */
+ bool ndist_enabled; /* build ndist coefficient? */
/* histogram / MCV size */
int32 mcv_max_items; /* max MCV items */
@@ -49,6 +50,7 @@ CATALOG(pg_mv_statistic,3381)
bool deps_built; /* dependencies were built */
bool mcv_built; /* MCV list was built */
bool hist_built; /* histogram was built */
+ bool ndist_built; /* ndistinct coeff built */
/* variable-length fields start here, but we allow direct access to stakeys */
int2vector stakeys; /* array of column keys */
@@ -57,6 +59,7 @@ CATALOG(pg_mv_statistic,3381)
bytea stadeps; /* dependencies (serialized) */
bytea stamcv; /* MCV list (serialized) */
bytea stahist; /* MV histogram (serialized) */
+ float8 standcoeff; /* ndistinct coeff (serialized) */
#endif
} FormData_pg_mv_statistic;
@@ -72,7 +75,7 @@ typedef FormData_pg_mv_statistic *Form_pg_mv_statistic;
* compiler constants for pg_mv_statistic
* ----------------
*/
-#define Natts_pg_mv_statistic 15
+#define Natts_pg_mv_statistic 19
#define Anum_pg_mv_statistic_starelid 1
#define Anum_pg_mv_statistic_staname 2
#define Anum_pg_mv_statistic_stanamespace 3
@@ -80,14 +83,17 @@ typedef FormData_pg_mv_statistic *Form_pg_mv_statistic;
#define Anum_pg_mv_statistic_deps_enabled 5
#define Anum_pg_mv_statistic_mcv_enabled 6
#define Anum_pg_mv_statistic_hist_enabled 7
-#define Anum_pg_mv_statistic_mcv_max_items 8
-#define Anum_pg_mv_statistic_hist_max_buckets 9
-#define Anum_pg_mv_statistic_deps_built 10
-#define Anum_pg_mv_statistic_mcv_built 11
-#define Anum_pg_mv_statistic_hist_built 12
-#define Anum_pg_mv_statistic_stakeys 13
-#define Anum_pg_mv_statistic_stadeps 14
-#define Anum_pg_mv_statistic_stamcv 15
-#define Anum_pg_mv_statistic_stahist 16
+#define Anum_pg_mv_statistic_ndist_enabled 8
+#define Anum_pg_mv_statistic_mcv_max_items 9
+#define Anum_pg_mv_statistic_hist_max_buckets 19
+#define Anum_pg_mv_statistic_deps_built 11
+#define Anum_pg_mv_statistic_mcv_built 12
+#define Anum_pg_mv_statistic_hist_built 13
+#define Anum_pg_mv_statistic_ndist_built 14
+#define Anum_pg_mv_statistic_stakeys 15
+#define Anum_pg_mv_statistic_stadeps 16
+#define Anum_pg_mv_statistic_stamcv 17
+#define Anum_pg_mv_statistic_stahist 18
+#define Anum_pg_mv_statistic_standist 19
#endif /* PG_MV_STATISTIC_H */
diff --git a/src/include/nodes/relation.h b/src/include/nodes/relation.h
index 8c50bfb..1923f2b 100644
--- a/src/include/nodes/relation.h
+++ b/src/include/nodes/relation.h
@@ -655,11 +655,13 @@ typedef struct MVStatisticInfo
bool deps_enabled; /* functional dependencies enabled */
bool mcv_enabled; /* MCV list enabled */
bool hist_enabled; /* histogram enabled */
+ bool ndist_enabled; /* ndistinct coefficient enabled */
/* built/available statistics */
bool deps_built; /* functional dependencies built */
bool mcv_built; /* MCV list built */
bool hist_built; /* histogram built */
+ bool ndist_built; /* ndistinct coefficient built */
/* columns in the statistics (attnums) */
int2vector *stakeys; /* attnums of the columns covered */
diff --git a/src/include/utils/mvstats.h b/src/include/utils/mvstats.h
index 35b2f8e..fb2c5d8 100644
--- a/src/include/utils/mvstats.h
+++ b/src/include/utils/mvstats.h
@@ -225,6 +225,7 @@ typedef MVSerializedHistogramData *MVSerializedHistogram;
MVDependencies load_mv_dependencies(Oid mvoid);
MCVList load_mv_mcvlist(Oid mvoid);
MVSerializedHistogram load_mv_histogram(Oid mvoid);
+double load_mv_ndistinct(Oid mvoid);
bytea * serialize_mv_dependencies(MVDependencies dependencies);
bytea * serialize_mv_mcvlist(MCVList mcvlist, int2vector *attrs,
@@ -266,11 +267,17 @@ MVHistogram
build_mv_histogram(int numrows, HeapTuple *rows, int2vector *attrs,
VacAttrStats **stats, int numrows_total);
-void build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
+double
+build_mv_ndistinct(double totalrows, int numrows, HeapTuple *rows,
+ int2vector *attrs, VacAttrStats **stats);
+
+void build_mv_stats(Relation onerel, double totalrows,
+ int numrows, HeapTuple *rows,
int natts, VacAttrStats **vacattrstats);
void update_mv_stats(Oid relid, MVDependencies dependencies,
MCVList mcvlist, MVHistogram histogram,
+ double ndistcoeff,
int2vector *attrs, VacAttrStats **stats);
#ifdef DEBUG_MVHIST
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 1a1a4ca..0ad935e 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1377,7 +1377,8 @@ pg_mv_stats| SELECT n.nspname AS schemaname,
length(s.stamcv) AS mcvbytes,
pg_mv_stats_mcvlist_info(s.stamcv) AS mcvinfo,
length(s.stahist) AS histbytes,
- pg_mv_stats_histogram_info(s.stahist) AS histinfo
+ pg_mv_stats_histogram_info(s.stahist) AS histinfo,
+ s.standcoeff AS ndcoeff
FROM ((pg_mv_statistic s
JOIN pg_class c ON ((c.oid = s.starelid)))
LEFT JOIN pg_namespace n ON ((n.oid = c.relnamespace)));
--
2.1.0
[text/x-patch] 0008-change-how-we-apply-selectivity-to-number-of-groups-.patch (1.4K, ../../1457535779.24545.71.camel@2ndquadrant.com/9-0008-change-how-we-apply-selectivity-to-number-of-groups-.patch)
download | inline diff:
From 3e6238b1651b37c0fc3f1dbad6be3c5bdbae5be8 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@pgaddict.com>
Date: Tue, 26 Jan 2016 18:14:33 +0100
Subject: [PATCH 8/9] change how we apply selectivity to number of groups
estimate
Instead of simply multiplying the ndistinct estimate with selecticity,
we instead use the formula for the expected number of distinct values
observed in 'k' rows when there are 'd' distinct values in the bin
d * (1 - ((d - 1) / d)^k)
This is 'with replacements' which seems appropriate for the use, and it
mostly assumes uniform distribution of the distinct values. So if the
distribution is not uniform (e.g. there are very frequent groups) this
may be less accurate than the current algorithm in some cases, giving
over-estimates. But that's probably better than OOM.
---
src/backend/utils/adt/selfuncs.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/backend/utils/adt/selfuncs.c b/src/backend/utils/adt/selfuncs.c
index a84dd2b..ce3ad19 100644
--- a/src/backend/utils/adt/selfuncs.c
+++ b/src/backend/utils/adt/selfuncs.c
@@ -3465,7 +3465,7 @@ estimate_num_groups(PlannerInfo *root, List *groupExprs, double input_rows,
/*
* Multiply by restriction selectivity.
*/
- reldistinct *= rel->rows / rel->tuples;
+ reldistinct = reldistinct * (1 - powl((reldistinct - 1) / reldistinct,rel->rows));
/*
* Update estimate of total distinct groups.
--
2.1.0
[text/x-patch] 0009-fixup-of-regression-tests-plans-changes-by-group-by-.patch (5.9K, ../../1457535779.24545.71.camel@2ndquadrant.com/10-0009-fixup-of-regression-tests-plans-changes-by-group-by-.patch)
download | inline diff:
From 82510eec9a98e24bc86deb313f3c031d54420996 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@pgaddict.com>
Date: Sun, 28 Feb 2016 21:16:40 +0100
Subject: [PATCH 9/9] fixup of regression tests (plans changes by group by
estimation)
---
src/test/regress/expected/join.out | 18 ++++++++++--------
src/test/regress/expected/subselect.out | 25 +++++++++++--------------
src/test/regress/expected/union.out | 16 ++++++++--------
3 files changed, 29 insertions(+), 30 deletions(-)
diff --git a/src/test/regress/expected/join.out b/src/test/regress/expected/join.out
index cafbc5e..151402d 100644
--- a/src/test/regress/expected/join.out
+++ b/src/test/regress/expected/join.out
@@ -3965,18 +3965,20 @@ select d.* from d left join (select * from b group by b.id, b.c_id) s
explain (costs off)
select d.* from d left join (select distinct * from b) s
on d.a = s.id;
- QUERY PLAN
---------------------------------------
+ QUERY PLAN
+---------------------------------------------
Merge Right Join
- Merge Cond: (b.id = d.a)
- -> Unique
- -> Sort
- Sort Key: b.id, b.c_id
- -> Seq Scan on b
+ Merge Cond: (s.id = d.a)
+ -> Sort
+ Sort Key: s.id
+ -> Subquery Scan on s
+ -> HashAggregate
+ Group Key: b.id, b.c_id
+ -> Seq Scan on b
-> Sort
Sort Key: d.a
-> Seq Scan on d
-(9 rows)
+(11 rows)
-- check join removal works when uniqueness of the join condition is enforced
-- by a UNION
diff --git a/src/test/regress/expected/subselect.out b/src/test/regress/expected/subselect.out
index de64ca7..0fc93d9 100644
--- a/src/test/regress/expected/subselect.out
+++ b/src/test/regress/expected/subselect.out
@@ -807,27 +807,24 @@ select * from int4_tbl where
explain (verbose, costs off)
select * from int4_tbl o where (f1, f1) in
(select f1, generate_series(1,2) / 10 g from int4_tbl i group by f1);
- QUERY PLAN
-----------------------------------------------------------------------
- Hash Join
+ QUERY PLAN
+----------------------------------------------------------------
+ Hash Semi Join
Output: o.f1
Hash Cond: (o.f1 = "ANY_subquery".f1)
-> Seq Scan on public.int4_tbl o
Output: o.f1
-> Hash
Output: "ANY_subquery".f1, "ANY_subquery".g
- -> HashAggregate
+ -> Subquery Scan on "ANY_subquery"
Output: "ANY_subquery".f1, "ANY_subquery".g
- Group Key: "ANY_subquery".f1, "ANY_subquery".g
- -> Subquery Scan on "ANY_subquery"
- Output: "ANY_subquery".f1, "ANY_subquery".g
- Filter: ("ANY_subquery".f1 = "ANY_subquery".g)
- -> HashAggregate
- Output: i.f1, (generate_series(1, 2) / 10)
- Group Key: i.f1
- -> Seq Scan on public.int4_tbl i
- Output: i.f1
-(18 rows)
+ Filter: ("ANY_subquery".f1 = "ANY_subquery".g)
+ -> HashAggregate
+ Output: i.f1, (generate_series(1, 2) / 10)
+ Group Key: i.f1
+ -> Seq Scan on public.int4_tbl i
+ Output: i.f1
+(15 rows)
select * from int4_tbl o where (f1, f1) in
(select f1, generate_series(1,2) / 10 g from int4_tbl i group by f1);
diff --git a/src/test/regress/expected/union.out b/src/test/regress/expected/union.out
index 016571b..f2e297e 100644
--- a/src/test/regress/expected/union.out
+++ b/src/test/regress/expected/union.out
@@ -263,16 +263,16 @@ ORDER BY 1;
SELECT q2 FROM int8_tbl INTERSECT SELECT q1 FROM int8_tbl;
q2
------------------
- 4567890123456789
123
+ 4567890123456789
(2 rows)
SELECT q2 FROM int8_tbl INTERSECT ALL SELECT q1 FROM int8_tbl;
q2
------------------
+ 123
4567890123456789
4567890123456789
- 123
(3 rows)
SELECT q2 FROM int8_tbl EXCEPT SELECT q1 FROM int8_tbl ORDER BY 1;
@@ -305,16 +305,16 @@ SELECT q1 FROM int8_tbl EXCEPT SELECT q2 FROM int8_tbl;
SELECT q1 FROM int8_tbl EXCEPT ALL SELECT q2 FROM int8_tbl;
q1
------------------
- 4567890123456789
123
+ 4567890123456789
(2 rows)
SELECT q1 FROM int8_tbl EXCEPT ALL SELECT DISTINCT q2 FROM int8_tbl;
q1
------------------
+ 123
4567890123456789
4567890123456789
- 123
(3 rows)
SELECT q1 FROM int8_tbl EXCEPT ALL SELECT q1 FROM int8_tbl FOR NO KEY UPDATE;
@@ -343,8 +343,8 @@ SELECT f1 FROM float8_tbl EXCEPT SELECT f1 FROM int4_tbl ORDER BY 1;
SELECT q1 FROM int8_tbl INTERSECT SELECT q2 FROM int8_tbl UNION ALL SELECT q2 FROM int8_tbl;
q1
-------------------
- 4567890123456789
123
+ 4567890123456789
456
4567890123456789
123
@@ -355,15 +355,15 @@ SELECT q1 FROM int8_tbl INTERSECT SELECT q2 FROM int8_tbl UNION ALL SELECT q2 FR
SELECT q1 FROM int8_tbl INTERSECT (((SELECT q2 FROM int8_tbl UNION ALL SELECT q2 FROM int8_tbl)));
q1
------------------
- 4567890123456789
123
+ 4567890123456789
(2 rows)
(((SELECT q1 FROM int8_tbl INTERSECT SELECT q2 FROM int8_tbl))) UNION ALL SELECT q2 FROM int8_tbl;
q1
-------------------
- 4567890123456789
123
+ 4567890123456789
456
4567890123456789
123
@@ -419,8 +419,8 @@ HINT: There is a column named "q2" in table "*SELECT* 2", but it cannot be refe
SELECT q1 FROM int8_tbl EXCEPT (((SELECT q2 FROM int8_tbl ORDER BY q2 LIMIT 1)));
q1
------------------
- 4567890123456789
123
+ 4567890123456789
(2 rows)
--
--
2.1.0
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: multivariate statistics v14
@ 2016-03-09 16:45 Jeff Janes <jeff.janes@gmail.com>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 1 reply; 58+ messages in thread
From: Jeff Janes @ 2016-03-09 16:45 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Alvaro Herrera <alvherre@2ndquadrant.com>; pgsql-hackers
On Wed, Mar 9, 2016 at 7:02 AM, Tomas Vondra
<tomas.vondra@2ndquadrant.com> wrote:
> Hi,
>
> thanks for the feedback. Attached is v14 of the patch series, fixing
> most of the points you've raised.
Hi Tomas,
Applied to aa09cd242fa7e3a694a31f, I still get the seg faults in make
check if I configure without --enable-cassert.
With --enable-cassert, it passes the regression test.
I got the core file, configured and compiled with:
CFLAGS="-fno-omit-frame-pointer" --enable-debug
The first core dump is on this statement:
-- check explain (expect bitmap index scan, not plain index scan)
INSERT INTO functional_dependencies
SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
bt
#0 0x00000000006e1160 in cost_qual_eval (cost=0x2494418,
quals=0x2495550, root=0x2541b88) at costsize.c:3181
#1 0x00000000006e1ee5 in set_baserel_size_estimates (root=0x2541b88,
rel=0x2494300) at costsize.c:3754
#2 0x00000000006d37e8 in set_plain_rel_size (root=0x2541b88,
rel=0x2494300, rte=0x247e660) at allpaths.c:480
#3 0x00000000006d353d in set_rel_size (root=0x2541b88, rel=0x2494300,
rti=1, rte=0x247e660) at allpaths.c:350
#4 0x00000000006d338f in set_base_rel_sizes (root=0x2541b88) at allpaths.c:270
#5 0x00000000006d3233 in make_one_rel (root=0x2541b88,
joinlist=0x2494628) at allpaths.c:169
#6 0x000000000070012e in query_planner (root=0x2541b88,
tlist=0x2541e58, qp_callback=0x7048d4 <standard_qp_callback>,
qp_extra=0x7ffefa6474e0)
at planmain.c:246
#7 0x0000000000702a33 in grouping_planner (root=0x2541b88,
inheritance_update=0 '\000', tuple_fraction=0) at planner.c:1647
#8 0x0000000000701310 in subquery_planner (glob=0x2541af8,
parse=0x246a838, parent_root=0x0, hasRecursion=0 '\000',
tuple_fraction=0) at planner.c:740
#9 0x000000000070055b in standard_planner (parse=0x246a838,
cursorOptions=256, boundParams=0x0) at planner.c:290
#10 0x000000000070023f in planner (parse=0x246a838, cursorOptions=256,
boundParams=0x0) at planner.c:160
#11 0x00000000007b8bf9 in pg_plan_query (querytree=0x246a838,
cursorOptions=256, boundParams=0x0) at postgres.c:798
#12 0x00000000005d1967 in ExplainOneQuery (query=0x246a838, into=0x0,
es=0x246a778,
queryString=0x2443d80 "EXPLAIN (COSTS off)\n SELECT * FROM
mcv_list WHERE a = 10 AND b = 5;", params=0x0) at explain.c:350
#13 0x00000000005d16a3 in ExplainQuery (stmt=0x2444f90,
queryString=0x2443d80 "EXPLAIN (COSTS off)\n SELECT * FROM mcv_list
WHERE a = 10 AND b = 5;",
params=0x0, dest=0x246a6e8) at explain.c:244
#14 0x00000000007c0afb in standard_ProcessUtility (parsetree=0x2444f90,
queryString=0x2443d80 "EXPLAIN (COSTS off)\n SELECT * FROM
mcv_list WHERE a = 10 AND b = 5;", context=PROCESS_UTILITY_TOPLEVEL,
params=0x0,
dest=0x246a6e8, completionTag=0x7ffefa647b60 "") at utility.c:659
#15 0x00000000007c0299 in ProcessUtility (parsetree=0x2444f90,
queryString=0x2443d80 "EXPLAIN (COSTS off)\n SELECT * FROM mcv_list
WHERE a = 10 AND b = 5;",
context=PROCESS_UTILITY_TOPLEVEL, params=0x0, dest=0x246a6e8,
completionTag=0x7ffefa647b60 "") at utility.c:335
#16 0x00000000007bf47b in PortalRunUtility (portal=0x23ed510,
utilityStmt=0x2444f90, isTopLevel=1 '\001', dest=0x246a6e8,
completionTag=0x7ffefa647b60 "")
at pquery.c:1183
#17 0x00000000007bf1ce in FillPortalStore (portal=0x23ed510,
isTopLevel=1 '\001') at pquery.c:1057
#18 0x00000000007beb19 in PortalRun (portal=0x23ed510,
count=9223372036854775807, isTopLevel=1 '\001', dest=0x253f6c0,
altdest=0x253f6c0,
completionTag=0x7ffefa647d40 "") at pquery.c:781
#19 0x00000000007b90ae in exec_simple_query (query_string=0x2443d80
"EXPLAIN (COSTS off)\n SELECT * FROM mcv_list WHERE a = 10 AND b =
5;")
at postgres.c:1094
#20 0x00000000007bcfac in PostgresMain (argc=1, argv=0x23d5070,
dbname=0x23d4e48 "regression", username=0x23d4e30 "jjanes") at
postgres.c:4021
#21 0x0000000000745a62 in BackendRun (port=0x23f4110) at postmaster.c:4258
#22 0x00000000007451d6 in BackendStartup (port=0x23f4110) at postmaster.c:3932
#23 0x0000000000741ab7 in ServerLoop () at postmaster.c:1690
#24 0x00000000007411c0 in PostmasterMain (argc=8, argv=0x23d3f20) at
postmaster.c:1298
#25 0x0000000000690026 in main (argc=8, argv=0x23d3f20) at main.c:223
Cheers,
Jeff
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: multivariate statistics v14
@ 2016-03-09 17:21 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Jeff Janes <jeff.janes@gmail.com>
0 siblings, 3 replies; 58+ messages in thread
From: Tomas Vondra @ 2016-03-09 17:21 UTC (permalink / raw)
To: Jeff Janes <jeff.janes@gmail.com>; +Cc: Alvaro Herrera <alvherre@2ndquadrant.com>; pgsql-hackers
Hi,
On Wed, 2016-03-09 at 08:45 -0800, Jeff Janes wrote:
> On Wed, Mar 9, 2016 at 7:02 AM, Tomas Vondra
> <tomas.vondra@2ndquadrant.com> wrote:
> > Hi,
> >
> > thanks for the feedback. Attached is v14 of the patch series, fixing
> > most of the points you've raised.
>
>
> Hi Tomas,
>
> Applied to aa09cd242fa7e3a694a31f, I still get the seg faults in make
> check if I configure without --enable-cassert.
Ah, after disabling asserts I can reproduce it too. And the reason why
it fails is quite simple - clauselist_selectivity modifies the original
list of clauses, which then confuses cost_qual_eval.
Can you try if the attached patch fixes the issue? I'll need to rework a
bit more of the code, but let's see if this fixes the issue on your
machine too.
> With --enable-cassert, it passes the regression test.
I wonder how can it work with casserts and fail without them. That's
kinda exactly the opposite to what I'd expect ...
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[text/x-patch] mvstats-segfault-fix.patch (642B, ../../1457544102.24545.79.camel@2ndquadrant.com/2-mvstats-segfault-fix.patch)
download | inline diff:
diff --git a/src/backend/optimizer/path/clausesel.c b/src/backend/optimizer/path/clausesel.c
index 2540da9..ddfdc3b 100644
--- a/src/backend/optimizer/path/clausesel.c
+++ b/src/backend/optimizer/path/clausesel.c
@@ -279,6 +279,10 @@ clauselist_selectivity(PlannerInfo *root,
List *solution = choose_mv_statistics(root, relid, stats,
clauses, conditions);
+ /* FIXME we must not scribble over the original list */
+ if (solution)
+ clauses = list_copy(clauses);
+
/*
* We have a good solution, which is merely a list of statistics that
* we need to apply. We'll apply the statistics one by one (in the order
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: multivariate statistics v14
@ 2016-03-09 18:09 Jeff Janes <jeff.janes@gmail.com>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
2 siblings, 0 replies; 58+ messages in thread
From: Jeff Janes @ 2016-03-09 18:09 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Alvaro Herrera <alvherre@2ndquadrant.com>; pgsql-hackers
On Wed, Mar 9, 2016 at 9:21 AM, Tomas Vondra
<tomas.vondra@2ndquadrant.com> wrote:
> Hi,
>
> On Wed, 2016-03-09 at 08:45 -0800, Jeff Janes wrote:
>> On Wed, Mar 9, 2016 at 7:02 AM, Tomas Vondra
>> <tomas.vondra@2ndquadrant.com> wrote:
>> > Hi,
>> >
>> > thanks for the feedback. Attached is v14 of the patch series, fixing
>> > most of the points you've raised.
>>
>>
>> Hi Tomas,
>>
>> Applied to aa09cd242fa7e3a694a31f, I still get the seg faults in make
>> check if I configure without --enable-cassert.
>
> Ah, after disabling asserts I can reproduce it too. And the reason why
> it fails is quite simple - clauselist_selectivity modifies the original
> list of clauses, which then confuses cost_qual_eval.
>
> Can you try if the attached patch fixes the issue? I'll need to rework a
> bit more of the code, but let's see if this fixes the issue on your
> machine too.
Yes, that fixes it.
>
>> With --enable-cassert, it passes the regression test.
>
> I wonder how can it work with casserts and fail without them. That's
> kinda exactly the opposite to what I'd expect ...
I too was surprised by that. Maybe cassert makes a copy of some data
structure which is used in-place without cassert?
Thanks,
Jeff
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: multivariate statistics v14
@ 2016-03-09 18:18 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
2 siblings, 0 replies; 58+ messages in thread
From: Tomas Vondra @ 2016-03-09 18:18 UTC (permalink / raw)
To: Jeff Janes <jeff.janes@gmail.com>; +Cc: Alvaro Herrera <alvherre@2ndquadrant.com>; pgsql-hackers
On Wed, 2016-03-09 at 18:21 +0100, Tomas Vondra wrote:
> Hi,
>
> On Wed, 2016-03-09 at 08:45 -0800, Jeff Janes wrote:
> > On Wed, Mar 9, 2016 at 7:02 AM, Tomas Vondra
> > <tomas.vondra@2ndquadrant.com> wrote:
> > > Hi,
> > >
> > > thanks for the feedback. Attached is v14 of the patch series, fixing
> > > most of the points you've raised.
> >
> >
> > Hi Tomas,
> >
> > Applied to aa09cd242fa7e3a694a31f, I still get the seg faults in make
> > check if I configure without --enable-cassert.
>
> Ah, after disabling asserts I can reproduce it too. And the reason why
> it fails is quite simple - clauselist_selectivity modifies the original
> list of clauses, which then confuses cost_qual_eval.
More precisely, it gets confused because the first clause in the list
gets deleted but cost_qual_eval never learns about that, and follows
stale pointer to the next cell, thus a segfault.
>
> Can you try if the attached patch fixes the issue? I'll need to rework a
> bit more of the code, but let's see if this fixes the issue on your
> machine too.
>
> > With --enable-cassert, it passes the regression test.
>
> I wonder how can it work with casserts and fail without them. That's
> kinda exactly the opposite to what I'd expect ...
FWIW it seems to be somehow related to this assert in clausesel.c:
Assert(count_mv_attnums(list_union(stat_clauses, stat_conditions),
relid, MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST) >= 2);
With the assert in place, the code passes without a failure. After
removing the assert (commenting it out), or even just changing it to
Assert(count_mv_attnums(stat_clauses, relid,
MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST)
+ count_mv_attnums(stat_conditions, relid,
MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST) >= 2);
i.e. removing the list_union, it fails as expected.
The only thing that I can think of is that list_union happens to place
the right stuff at the right position in memory - pure luck.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: multivariate statistics v14
@ 2016-03-13 07:30 Jeff Janes <jeff.janes@gmail.com>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
2 siblings, 1 reply; 58+ messages in thread
From: Jeff Janes @ 2016-03-13 07:30 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Alvaro Herrera <alvherre@2ndquadrant.com>; pgsql-hackers
On Wed, Mar 9, 2016 at 9:21 AM, Tomas Vondra
<tomas.vondra@2ndquadrant.com> wrote:
> Hi,
>
> On Wed, 2016-03-09 at 08:45 -0800, Jeff Janes wrote:
>> On Wed, Mar 9, 2016 at 7:02 AM, Tomas Vondra
>> <tomas.vondra@2ndquadrant.com> wrote:
>> > Hi,
>> >
>> > thanks for the feedback. Attached is v14 of the patch series, fixing
>> > most of the points you've raised.
>>
>>
>> Hi Tomas,
>>
>> Applied to aa09cd242fa7e3a694a31f, I still get the seg faults in make
>> check if I configure without --enable-cassert.
>
> Ah, after disabling asserts I can reproduce it too. And the reason why
> it fails is quite simple - clauselist_selectivity modifies the original
> list of clauses, which then confuses cost_qual_eval.
>
> Can you try if the attached patch fixes the issue? I'll need to rework a
> bit more of the code, but let's see if this fixes the issue on your
> machine too.
That patch on top of v14 did fix the original problem. But I got
another segfault:
jjanes=# create table foo as select x, floor(x/(10000000/500))::int as
y from generate_series(1,10000000) f(x);
jjanes=# create index on foo (x,y);
jjanes=# create index on foo (y,x);
jjanes=# create statistics jjj on foo (x,y) with (dependencies,histogram);
jjanes=# analyze ;
server closed the connection unexpectedly
#0 multi_sort_add_dimension (mss=mss@entry=0x7f45dafc7c88,
sortdim=sortdim@entry=0, dim=dim@entry=0,
vacattrstats=vacattrstats@entry=0x16f0dd0) at common.c:436
#1 0x00000000007d022a in update_bucket_ndistinct (attrs=0x166fdf8,
stats=0x16f0dd0, bucket=<optimized out>) at histogram.c:1384
#2 0x00000000007d09aa in create_initial_mv_bucket (stats=0x16f0dd0,
attrs=0x166fdf8, rows=0x17cda20, numrows=30000) at histogram.c:880
#3 build_mv_histogram (numrows=30000, rows=rows@entry=0x170ecf0,
attrs=attrs@entry=0x166fdf8, stats=stats@entry=0x16f0dd0,
numrows_total=numrows_total@entry=30000)
at histogram.c:156
#4 0x00000000007ced19 in build_mv_stats
(onerel=onerel@entry=0x7f45e797d040, totalrows=9999985,
numrows=numrows@entry=30000, rows=rows@entry=0x170ecf0,
natts=natts@entry=2,
vacattrstats=vacattrstats@entry=0x166efa0) at common.c:106
#5 0x000000000055ff6b in do_analyze_rel
(onerel=onerel@entry=0x7f45e797d040, options=options@entry=2,
va_cols=va_cols@entry=0x0, acquirefunc=<optimized out>,
relpages=44248,
inh=inh@entry=0 '\000', in_outer_xact=in_outer_xact@entry=0
'\000', elevel=elevel@entry=13, params=0x7ffcbe382a30) at
analyze.c:585
#6 0x0000000000560ced in analyze_rel (relid=relid@entry=16441,
relation=relation@entry=0x16bc9d0, options=options@entry=2,
params=params@entry=0x7ffcbe382a30,
va_cols=va_cols@entry=0x0, in_outer_xact=<optimized out>,
bstrategy=0x16640f0) at analyze.c:262
#7 0x00000000005b70fd in vacuum (options=2, relation=0x16bc9d0,
relid=relid@entry=0, params=params@entry=0x7ffcbe382a30, va_cols=0x0,
bstrategy=<optimized out>,
bstrategy@entry=0x0, isTopLevel=isTopLevel@entry=1 '\001') at vacuum.c:313
#8 0x00000000005b748e in ExecVacuum (vacstmt=vacstmt@entry=0x16bca20,
isTopLevel=isTopLevel@entry=1 '\001') at vacuum.c:121
#9 0x00000000006c90f3 in standard_ProcessUtility
(parsetree=0x16bca20, queryString=0x16bbfc0 "analyze foo ;",
context=<optimized out>, params=0x0, dest=0x16bcd60,
completionTag=0x7ffcbe382fa0 "") at utility.c:654
#10 0x00007f45e413b1d1 in pgss_ProcessUtility (parsetree=0x16bca20,
queryString=0x16bbfc0 "analyze foo ;",
context=PROCESS_UTILITY_TOPLEVEL, params=0x0, dest=0x16bcd60,
completionTag=0x7ffcbe382fa0 "") at pg_stat_statements.c:986
#11 0x00000000006c6841 in PortalRunUtility (portal=0x16f7700,
utilityStmt=0x16bca20, isTopLevel=<optimized out>, dest=0x16bcd60,
completionTag=0x7ffcbe382fa0 "") at pquery.c:1175
#12 0x00000000006c73c5 in PortalRunMulti
(portal=portal@entry=0x16f7700, isTopLevel=isTopLevel@entry=1 '\001',
dest=dest@entry=0x16bcd60, altdest=altdest@entry=0x16bcd60,
completionTag=completionTag@entry=0x7ffcbe382fa0 "") at pquery.c:1306
#13 0x00000000006c7dd9 in PortalRun (portal=portal@entry=0x16f7700,
count=count@entry=9223372036854775807, isTopLevel=isTopLevel@entry=1
'\001', dest=dest@entry=0x16bcd60,
altdest=altdest@entry=0x16bcd60,
completionTag=completionTag@entry=0x7ffcbe382fa0 "") at pquery.c:813
#14 0x00000000006c5c98 in exec_simple_query (query_string=0x16bbfc0
"analyze foo ;") at postgres.c:1094
#15 PostgresMain (argc=<optimized out>, argv=argv@entry=0x164baf8,
dbname=0x164b9a8 "jjanes", username=<optimized out>) at
postgres.c:4021
#16 0x000000000047cb1e in BackendRun (port=0x1669d40) at postmaster.c:4258
#17 BackendStartup (port=0x1669d40) at postmaster.c:3932
#18 ServerLoop () at postmaster.c:1690
#19 0x000000000066ff27 in PostmasterMain (argc=argc@entry=1,
argv=argv@entry=0x164aa10) at postmaster.c:1298
#20 0x000000000047d35e in main (argc=1, argv=0x164aa10) at main.c:228
Cheers,
Jeff
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: multivariate statistics v14
@ 2016-03-13 21:59 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Jeff Janes <jeff.janes@gmail.com>
0 siblings, 3 replies; 58+ messages in thread
From: Tomas Vondra @ 2016-03-13 21:59 UTC (permalink / raw)
To: Jeff Janes <jeff.janes@gmail.com>; +Cc: Alvaro Herrera <alvherre@2ndquadrant.com>; pgsql-hackers
On Sat, 2016-03-12 at 23:30 -0800, Jeff Janes wrote:
> On Wed, Mar 9, 2016 at 9:21 AM, Tomas Vondra
> <tomas.vondra@2ndquadrant.com> wrote:
> >
> > Hi,
> >
> > On Wed, 2016-03-09 at 08:45 -0800, Jeff Janes wrote:
> > >
> > > On Wed, Mar 9, 2016 at 7:02 AM, Tomas Vondra
> > > <tomas.vondra@2ndquadrant.com> wrote:
> > > >
> > > > Hi,
> > > >
> > > > thanks for the feedback. Attached is v14 of the patch series,
> > > > fixing
> > > > most of the points you've raised.
> > >
> > > Hi Tomas,
> > >
> > > Applied to aa09cd242fa7e3a694a31f, I still get the seg faults in
> > > make
> > > check if I configure without --enable-cassert.
> > Ah, after disabling asserts I can reproduce it too. And the reason
> > why
> > it fails is quite simple - clauselist_selectivity modifies the
> > original
> > list of clauses, which then confuses cost_qual_eval.
> >
> > Can you try if the attached patch fixes the issue? I'll need to
> > rework a
> > bit more of the code, but let's see if this fixes the issue on your
> > machine too.
> That patch on top of v14 did fix the original problem. But I got
> another segfault:
Oh, yeah. There was an extra pfree().
Attached is v15 of the patch series, fixing this and also doing quite a
few additional improvements:
* added some basic examples into the SGML documentation
* addressing the objectaddress omissions, as pointed out by Alvaro
* support for ALTER STATISTICS ... OWNER TO / RENAME / SET SCHEMA
* significant refactoring of MCV and histogram code, particularly
serialization, deserialization and building
* reworking the functional dependencies to support more complex
dependencies, with multiple columns as 'conditions'
* the reduction using functional dependencies is also significantly
simplified (I decided to get rid of computing the transitive closure
for now - it got too complex after the multi-condition dependencies,
so I'll leave that for the future
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[text/x-patch] 0008-change-how-we-apply-selectivity-to-number-of-groups-.patch (1.4K, ../../1457906378.27231.10.camel@2ndquadrant.com/2-0008-change-how-we-apply-selectivity-to-number-of-groups-.patch)
download | inline diff:
From 494a31e1ed7976e0f965a32e81c769e1c3dfad66 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@pgaddict.com>
Date: Tue, 26 Jan 2016 18:14:33 +0100
Subject: [PATCH 8/9] change how we apply selectivity to number of groups
estimate
Instead of simply multiplying the ndistinct estimate with selecticity,
we instead use the formula for the expected number of distinct values
observed in 'k' rows when there are 'd' distinct values in the bin
d * (1 - ((d - 1) / d)^k)
This is 'with replacements' which seems appropriate for the use, and it
mostly assumes uniform distribution of the distinct values. So if the
distribution is not uniform (e.g. there are very frequent groups) this
may be less accurate than the current algorithm in some cases, giving
over-estimates. But that's probably better than OOM.
---
src/backend/utils/adt/selfuncs.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/backend/utils/adt/selfuncs.c b/src/backend/utils/adt/selfuncs.c
index f8d39aa..6eceedf 100644
--- a/src/backend/utils/adt/selfuncs.c
+++ b/src/backend/utils/adt/selfuncs.c
@@ -3466,7 +3466,7 @@ estimate_num_groups(PlannerInfo *root, List *groupExprs, double input_rows,
/*
* Multiply by restriction selectivity.
*/
- reldistinct *= rel->rows / rel->tuples;
+ reldistinct = reldistinct * (1 - powl((reldistinct - 1) / reldistinct,rel->rows));
/*
* Update estimate of total distinct groups.
--
2.5.0
[text/x-patch] 0007-multivariate-ndistinct-coefficients.patch (28.5K, ../../1457906378.27231.10.camel@2ndquadrant.com/3-0007-multivariate-ndistinct-coefficients.patch)
download | inline diff:
From 1b905c77e851d34229da72c2a84107fa0925f54a Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@pgaddict.com>
Date: Wed, 23 Dec 2015 02:07:58 +0100
Subject: [PATCH 7/9] multivariate ndistinct coefficients
---
doc/src/sgml/ref/create_statistics.sgml | 9 ++
src/backend/catalog/system_views.sql | 3 +-
src/backend/commands/analyze.c | 2 +-
src/backend/commands/statscmds.c | 11 +-
src/backend/optimizer/path/clausesel.c | 4 +
src/backend/optimizer/util/plancat.c | 4 +-
src/backend/utils/adt/selfuncs.c | 93 +++++++++++++++-
src/backend/utils/mvstats/Makefile | 2 +-
src/backend/utils/mvstats/README.ndistinct | 83 ++++++++++++++
src/backend/utils/mvstats/README.stats | 2 +
src/backend/utils/mvstats/common.c | 23 +++-
src/backend/utils/mvstats/mvdist.c | 171 +++++++++++++++++++++++++++++
src/include/catalog/pg_mv_statistic.h | 26 +++--
src/include/nodes/relation.h | 2 +
src/include/utils/mvstats.h | 9 +-
src/test/regress/expected/rules.out | 3 +-
16 files changed, 424 insertions(+), 23 deletions(-)
create mode 100644 src/backend/utils/mvstats/README.ndistinct
create mode 100644 src/backend/utils/mvstats/mvdist.c
diff --git a/doc/src/sgml/ref/create_statistics.sgml b/doc/src/sgml/ref/create_statistics.sgml
index f7336fd..80e472f 100644
--- a/doc/src/sgml/ref/create_statistics.sgml
+++ b/doc/src/sgml/ref/create_statistics.sgml
@@ -168,6 +168,15 @@ CREATE STATISTICS [ IF NOT EXISTS ] <replaceable class="PARAMETER">statistics_na
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>ndistinct</> (<type>boolean</>)</term>
+ <listitem>
+ <para>
+ Enables ndistinct coefficients for the statistics.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect2>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index b151db1..8d2b435 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -169,7 +169,8 @@ CREATE VIEW pg_mv_stats AS
length(S.stamcv) AS mcvbytes,
pg_mv_stats_mcvlist_info(S.stamcv) AS mcvinfo,
length(S.stahist) AS histbytes,
- pg_mv_stats_histogram_info(S.stahist) AS histinfo
+ pg_mv_stats_histogram_info(S.stahist) AS histinfo,
+ standcoeff AS ndcoeff
FROM (pg_mv_statistic S JOIN pg_class C ON (C.oid = S.starelid))
LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace);
diff --git a/src/backend/commands/analyze.c b/src/backend/commands/analyze.c
index 9087532..c29f1be 100644
--- a/src/backend/commands/analyze.c
+++ b/src/backend/commands/analyze.c
@@ -582,7 +582,7 @@ do_analyze_rel(Relation onerel, int options, VacuumParams *params,
}
/* Build multivariate stats (if there are any). */
- build_mv_stats(onerel, numrows, rows, attr_cnt, vacattrstats);
+ build_mv_stats(onerel, totalrows, numrows, rows, attr_cnt, vacattrstats);
}
/*
diff --git a/src/backend/commands/statscmds.c b/src/backend/commands/statscmds.c
index e0b085f..a7c569d 100644
--- a/src/backend/commands/statscmds.c
+++ b/src/backend/commands/statscmds.c
@@ -72,7 +72,8 @@ CreateStatistics(CreateStatsStmt *stmt)
/* by default build nothing */
bool build_dependencies = false,
build_mcv = false,
- build_histogram = false;
+ build_histogram = false,
+ build_ndistinct = false;
int32 max_buckets = -1,
max_mcv_items = -1;
@@ -155,6 +156,8 @@ CreateStatistics(CreateStatsStmt *stmt)
if (strcmp(opt->defname, "dependencies") == 0)
build_dependencies = defGetBoolean(opt);
+ else if (strcmp(opt->defname, "ndistinct") == 0)
+ build_ndistinct = defGetBoolean(opt);
else if (strcmp(opt->defname, "mcv") == 0)
build_mcv = defGetBoolean(opt);
else if (strcmp(opt->defname, "max_mcv_items") == 0)
@@ -209,10 +212,10 @@ CreateStatistics(CreateStatsStmt *stmt)
}
/* check that at least some statistics were requested */
- if (! (build_dependencies || build_mcv || build_histogram))
+ if (! (build_dependencies || build_mcv || build_histogram || build_ndistinct))
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("no statistics type (dependencies, mcv, histogram) was requested")));
+ errmsg("no statistics type (dependencies, mcv, histogram, ndistinct) was requested")));
/* now do some checking of the options */
if (require_mcv && (! build_mcv))
@@ -246,6 +249,7 @@ CreateStatistics(CreateStatsStmt *stmt)
values[Anum_pg_mv_statistic_deps_enabled -1] = BoolGetDatum(build_dependencies);
values[Anum_pg_mv_statistic_mcv_enabled -1] = BoolGetDatum(build_mcv);
values[Anum_pg_mv_statistic_hist_enabled -1] = BoolGetDatum(build_histogram);
+ values[Anum_pg_mv_statistic_ndist_enabled-1] = BoolGetDatum(build_ndistinct);
values[Anum_pg_mv_statistic_mcv_max_items -1] = Int32GetDatum(max_mcv_items);
values[Anum_pg_mv_statistic_hist_max_buckets -1] = Int32GetDatum(max_buckets);
@@ -253,6 +257,7 @@ CreateStatistics(CreateStatsStmt *stmt)
nulls[Anum_pg_mv_statistic_stadeps -1] = true;
nulls[Anum_pg_mv_statistic_stamcv -1] = true;
nulls[Anum_pg_mv_statistic_stahist -1] = true;
+ nulls[Anum_pg_mv_statistic_standist -1] = true;
/* insert the tuple into pg_mv_statistic */
mvstatrel = heap_open(MvStatisticRelationId, RowExclusiveLock);
diff --git a/src/backend/optimizer/path/clausesel.c b/src/backend/optimizer/path/clausesel.c
index e06fd99..255d275 100644
--- a/src/backend/optimizer/path/clausesel.c
+++ b/src/backend/optimizer/path/clausesel.c
@@ -59,6 +59,7 @@ static void addRangeClause(RangeQueryClause **rqlist, Node *clause,
#define MV_CLAUSE_TYPE_FDEP 0x01
#define MV_CLAUSE_TYPE_MCV 0x02
#define MV_CLAUSE_TYPE_HIST 0x04
+#define MV_CLAUSE_TYPE_NDIST 0x08
static bool clause_is_mv_compatible(Node *clause, Index relid, Bitmapset **attnums,
int type);
@@ -2860,6 +2861,9 @@ stats_type_matches(MVStatisticInfo *stat, int type)
if ((type & MV_CLAUSE_TYPE_HIST) && stat->hist_built)
return true;
+ if ((type & MV_CLAUSE_TYPE_NDIST) && stat->ndist_built)
+ return true;
+
return false;
}
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index 2519249..3741b7a 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -412,7 +412,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
mvstat = (Form_pg_mv_statistic) GETSTRUCT(htup);
/* unavailable stats are not interesting for the planner */
- if (mvstat->deps_built || mvstat->mcv_built || mvstat->hist_built)
+ if (mvstat->deps_built || mvstat->mcv_built || mvstat->hist_built || mvstat->ndist_built)
{
info = makeNode(MVStatisticInfo);
@@ -423,11 +423,13 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->deps_enabled = mvstat->deps_enabled;
info->mcv_enabled = mvstat->mcv_enabled;
info->hist_enabled = mvstat->hist_enabled;
+ info->ndist_enabled = mvstat->ndist_enabled;
/* built/available statistics */
info->deps_built = mvstat->deps_built;
info->mcv_built = mvstat->mcv_built;
info->hist_built = mvstat->hist_built;
+ info->ndist_built = mvstat->ndist_built;
/* stakeys */
adatum = SysCacheGetAttr(MVSTATOID, htup,
diff --git a/src/backend/utils/adt/selfuncs.c b/src/backend/utils/adt/selfuncs.c
index 805d633..f8d39aa 100644
--- a/src/backend/utils/adt/selfuncs.c
+++ b/src/backend/utils/adt/selfuncs.c
@@ -132,6 +132,7 @@
#include "utils/fmgroids.h"
#include "utils/index_selfuncs.h"
#include "utils/lsyscache.h"
+#include "utils/mvstats.h"
#include "utils/nabstime.h"
#include "utils/pg_locale.h"
#include "utils/rel.h"
@@ -206,6 +207,7 @@ static Const *string_to_const(const char *str, Oid datatype);
static Const *string_to_bytea_const(const char *str, size_t str_len);
static List *add_predicate_to_quals(IndexOptInfo *index, List *indexQuals);
+static Oid find_ndistinct_coeff(PlannerInfo *root, RelOptInfo *rel, List *varinfos);
/*
* eqsel - Selectivity of "=" for any data types.
@@ -3423,12 +3425,26 @@ estimate_num_groups(PlannerInfo *root, List *groupExprs, double input_rows,
* don't know by how much. We should never clamp to less than the
* largest ndistinct value for any of the Vars, though, since
* there will surely be at least that many groups.
+ *
+ * However we don't need to do this if we have ndistinct stats on
+ * the columns - in that case we can simply use the coefficient
+ * to get the (probably way more accurate) estimate.
+ *
+ * XXX Probably needs refactoring (don't like to mix with clamp
+ * and coeff at the same time).
*/
double clamp = rel->tuples;
+ double coeff = 1.0;
if (relvarcount > 1)
{
- clamp *= 0.1;
+ Oid oid = find_ndistinct_coeff(root, rel, varinfos);
+
+ if (oid != InvalidOid)
+ coeff = load_mv_ndistinct(oid);
+ else
+ clamp *= 0.1;
+
if (clamp < relmaxndistinct)
{
clamp = relmaxndistinct;
@@ -3437,6 +3453,13 @@ estimate_num_groups(PlannerInfo *root, List *groupExprs, double input_rows,
clamp = rel->tuples;
}
}
+
+ /*
+ * Apply ndistinct coefficient from multivar stats (we must do this
+ * before clamping the estimate in any way.
+ */
+ reldistinct /= coeff;
+
if (reldistinct > clamp)
reldistinct = clamp;
@@ -7583,3 +7606,71 @@ brincostestimate(PlannerInfo *root, IndexPath *path, double loop_count,
/* XXX what about pages_per_range? */
}
+
+/*
+ * Find applicable ndistinct statistics and compute the coefficient to
+ * correct the estimate (simply a product of per-column ndistincts).
+ *
+ * Currently we only look for a perfect match, i.e. a single ndistinct
+ * estimate exactly matching all the columns of the statistics.
+ */
+static Oid
+find_ndistinct_coeff(PlannerInfo *root, RelOptInfo *rel, List *varinfos)
+{
+ ListCell *lc;
+ Bitmapset *attnums = NULL;
+ VariableStatData vardata;
+
+ foreach(lc, varinfos)
+ {
+ GroupVarInfo *varinfo = (GroupVarInfo *) lfirst(lc);
+
+ if (varinfo->rel != rel)
+ continue;
+
+ /* FIXME handle expressions in general only */
+
+ /*
+ * examine the variable (or expression) so that we know which
+ * attribute we're dealing with - we need this for matching the
+ * ndistinct coefficient
+ *
+ * FIXME probably might remember this from estimate_num_groups
+ */
+ examine_variable(root, varinfo->var, 0, &vardata);
+
+ if (HeapTupleIsValid(vardata.statsTuple))
+ {
+ Form_pg_statistic stats
+ = (Form_pg_statistic) GETSTRUCT(vardata.statsTuple);
+
+ attnums = bms_add_member(attnums, stats->staattnum);
+
+ ReleaseVariableStats(vardata);
+ }
+ }
+
+ /* look for a matching ndistinct statistics */
+ foreach (lc, rel->mvstatlist)
+ {
+ int i;
+ MVStatisticInfo *info = (MVStatisticInfo *)lfirst(lc);
+
+ /* skip statistics without ndistinct coefficient built */
+ if (!info->ndist_built)
+ continue;
+
+ /* only exact matches for now (same set of columns) */
+ if (bms_num_members(attnums) != info->stakeys->dim1)
+ continue;
+
+ /* check that the columns match */
+ for (i = 0; i < info->stakeys->dim1; i++)
+ if (bms_is_member(info->stakeys->values[i], attnums))
+ continue;
+
+ return info->mvoid;
+ }
+
+ return InvalidOid;
+}
diff --git a/src/backend/utils/mvstats/Makefile b/src/backend/utils/mvstats/Makefile
index 9dbb3b6..d4b88e9 100644
--- a/src/backend/utils/mvstats/Makefile
+++ b/src/backend/utils/mvstats/Makefile
@@ -12,6 +12,6 @@ subdir = src/backend/utils/mvstats
top_builddir = ../../../..
include $(top_builddir)/src/Makefile.global
-OBJS = common.o dependencies.o histogram.o mcv.o
+OBJS = common.o dependencies.o histogram.o mcv.o mvdist.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/utils/mvstats/README.ndistinct b/src/backend/utils/mvstats/README.ndistinct
new file mode 100644
index 0000000..32d1624
--- /dev/null
+++ b/src/backend/utils/mvstats/README.ndistinct
@@ -0,0 +1,83 @@
+ndistinct coefficients
+======================
+
+Estimating number of distinct groups in a combination of columns is tricky,
+and the estimation error is often significant. By ndistinct coefficient we
+mean a ratio
+
+ q = ndistinct(a) * ndistinct(b) / ndistinct(a,b)
+
+where 'a' and 'b' are columns, ndistinct(a) is (an estimate of) a number of
+distinct values in column 'a'. And ndistinct(a,b) is the same thing for the
+pair of columns.
+
+The meaning of the coefficient may be illustrated by answering the following
+question: Given a combination of columns (a,b), how many distinct values of 'b'
+matches a chosen value of 'a' on average?
+
+Let's assume we know ndistinct(a) and ndistinct(a,b). Then the answer to the
+question clearly is
+
+ ndistinct(a,b) / ndistinct(a)
+
+and by using 'q' we may rewrite this as
+
+ ndistinct(b) / q
+
+so 'q' may be considered as a correction factor of the ndistinct estimate given
+a condition on one of the columns.
+
+This may be generalized to a combination of 'n' columns
+
+ [ndistinct(c1) * ... * ndistinct(cn)] / ndistinct(c1, ..., cn)
+
+and the meaning is very similar, except that we need to use conditions on (n-1)
+of the columns.
+
+
+Selectivity estimation
+----------------------
+
+As explained in the previous paragraph, ndistinct coefficients may be used to
+estimate cardinality of a column, given some apriori knowledge. Let's assume
+we need to estimate selectivity of a condition
+
+ (a=1) AND (b=2)
+
+which we can expand like this
+
+ P(a=1 & b=2) = P(a=1) * P(b=2 | a=1)
+
+Let's also assume that the distribution of 'b' is uniform, i.e. that
+
+ P(a=1) = 1/ndistinct(a)
+ P(b=2) = 1/ndistinct(b)
+ P(a=1 & b=2) = 1/ndistinct(a,b)
+
+ P(b=2 | a=1) = ndistinct(a) / ndistinct(a,b)
+
+which may be rewritten like
+
+ P(b=2 | a=1)
+ = ndistinct(a,b) / ndistinct(a)
+ = (1/ndistinct(b)) * [(ndistinct(a) * ndistinct(b)) / ndistinct(a,b)]
+ = (1/ndistinct(b)) * q
+
+and therefore
+
+ P(a=1 & b=2) = (1/ndistinct(a)) * (1/ndistinct(b)) * q
+
+This also illustrates 'q' as a correction coefficient.
+
+It also explains why we store the coefficient and not simply ndistinct(a,b).
+This way we can simply estimate individual clauses and then simply correct
+the estimate by multiplying the result with 'q' - we don't have to mess with
+ndistinct estimates at all.
+
+Naturally, as the coefficient is derives from ndistinct(a,b), it may be also
+used to estimate GROUP BY clauses on the combination of columns, replacing the
+existing heuristics in estimate_num_groups().
+
+Note: Currently only the GROUP BY estimation is implemented. It's a bit unclear
+how to implement the clause estimation when there are other statistics (esp.
+MCV lists and/or functional dependencies) available.
diff --git a/src/backend/utils/mvstats/README.stats b/src/backend/utils/mvstats/README.stats
index d404914..6d4b09b 100644
--- a/src/backend/utils/mvstats/README.stats
+++ b/src/backend/utils/mvstats/README.stats
@@ -20,6 +20,8 @@ Currently we only have two kinds of multivariate statistics
(c) multivariate histograms (README.histogram)
+ (d) ndistinct coefficients
+
Compatible clause types
-----------------------
diff --git a/src/backend/utils/mvstats/common.c b/src/backend/utils/mvstats/common.c
index f6d1074..d34d072 100644
--- a/src/backend/utils/mvstats/common.c
+++ b/src/backend/utils/mvstats/common.c
@@ -32,7 +32,8 @@ static List* list_mv_stats(Oid relid);
* and serializes them back into the catalog (as bytea values).
*/
void
-build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
+build_mv_stats(Relation onerel, double totalrows,
+ int numrows, HeapTuple *rows,
int natts, VacAttrStats **vacattrstats)
{
ListCell *lc;
@@ -53,6 +54,7 @@ build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
MVDependencies deps = NULL;
MCVList mcvlist = NULL;
MVHistogram histogram = NULL;
+ double ndist = -1;
int numrows_filtered = numrows;
VacAttrStats **stats = NULL;
@@ -92,6 +94,9 @@ build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
if (stat->deps_enabled)
deps = build_mv_dependencies(numrows, rows, attrs, stats);
+ if (stat->ndist_enabled)
+ ndist = build_mv_ndistinct(totalrows, numrows, rows, attrs, stats);
+
/* build the MCV list */
if (stat->mcv_enabled)
mcvlist = build_mv_mcvlist(numrows, rows, attrs, stats, &numrows_filtered);
@@ -101,7 +106,7 @@ build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
histogram = build_mv_histogram(numrows_filtered, rows, attrs, stats, numrows);
/* store the histogram / MCV list in the catalog */
- update_mv_stats(stat->mvoid, deps, mcvlist, histogram, attrs, stats);
+ update_mv_stats(stat->mvoid, deps, mcvlist, histogram, ndist, attrs, stats);
}
}
@@ -183,6 +188,8 @@ list_mv_stats(Oid relid)
info->mcv_built = stats->mcv_built;
info->hist_enabled = stats->hist_enabled;
info->hist_built = stats->hist_built;
+ info->ndist_enabled = stats->ndist_enabled;
+ info->ndist_built = stats->ndist_built;
result = lappend(result, info);
}
@@ -252,7 +259,7 @@ find_mv_attnums(Oid mvoid, Oid *relid)
void
update_mv_stats(Oid mvoid,
MVDependencies dependencies, MCVList mcvlist, MVHistogram histogram,
- int2vector *attrs, VacAttrStats **stats)
+ double ndistcoeff, int2vector *attrs, VacAttrStats **stats)
{
HeapTuple stup,
oldtup;
@@ -292,26 +299,36 @@ update_mv_stats(Oid mvoid,
= PointerGetDatum(data);
}
+ if (ndistcoeff > 1.0)
+ {
+ nulls[Anum_pg_mv_statistic_standist -1] = false;
+ values[Anum_pg_mv_statistic_standist-1] = Float8GetDatum(ndistcoeff);
+ }
+
/* always replace the value (either by bytea or NULL) */
replaces[Anum_pg_mv_statistic_stadeps -1] = true;
replaces[Anum_pg_mv_statistic_stamcv -1] = true;
replaces[Anum_pg_mv_statistic_stahist-1] = true;
+ replaces[Anum_pg_mv_statistic_standist-1] = true;
/* always change the availability flags */
nulls[Anum_pg_mv_statistic_deps_built -1] = false;
nulls[Anum_pg_mv_statistic_mcv_built -1] = false;
nulls[Anum_pg_mv_statistic_hist_built-1] = false;
+ nulls[Anum_pg_mv_statistic_ndist_built-1] = false;
nulls[Anum_pg_mv_statistic_stakeys-1] = false;
/* use the new attnums, in case we removed some dropped ones */
replaces[Anum_pg_mv_statistic_deps_built-1] = true;
replaces[Anum_pg_mv_statistic_mcv_built -1] = true;
+ replaces[Anum_pg_mv_statistic_ndist_built-1] = true;
replaces[Anum_pg_mv_statistic_hist_built -1] = true;
replaces[Anum_pg_mv_statistic_stakeys -1] = true;
values[Anum_pg_mv_statistic_deps_built-1] = BoolGetDatum(dependencies != NULL);
values[Anum_pg_mv_statistic_mcv_built -1] = BoolGetDatum(mcvlist != NULL);
values[Anum_pg_mv_statistic_hist_built -1] = BoolGetDatum(histogram != NULL);
+ values[Anum_pg_mv_statistic_ndist_built-1] = BoolGetDatum(ndistcoeff > 1.0);
values[Anum_pg_mv_statistic_stakeys -1] = PointerGetDatum(attrs);
/* Is there already a pg_mv_statistic tuple for this attribute? */
diff --git a/src/backend/utils/mvstats/mvdist.c b/src/backend/utils/mvstats/mvdist.c
new file mode 100644
index 0000000..59b8358
--- /dev/null
+++ b/src/backend/utils/mvstats/mvdist.c
@@ -0,0 +1,171 @@
+/*-------------------------------------------------------------------------
+ *
+ * mvdist.c
+ * POSTGRES multivariate distinct coefficients
+ *
+ *
+ * Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/utils/mvstats/mvdist.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include <math.h>
+
+#include "common.h"
+#include "utils/lsyscache.h"
+
+static double estimate_ndistinct(double totalrows, int numrows, int d, int f1);
+
+/*
+ * Compute ndistinct coefficient for the combination of attributes. This
+ * computes the ndistinct estimate using the same estimator used in analyze.c
+ * and then computes the coefficient.
+ */
+double
+build_mv_ndistinct(double totalrows, int numrows, HeapTuple *rows,
+ int2vector *attrs, VacAttrStats **stats)
+{
+ int i, j;
+ int f1, cnt, d;
+ int nmultiple, summultiple;
+ int numattrs = attrs->dim1;
+ MultiSortSupport mss = multi_sort_init(numattrs);
+ double ndistcoeff;
+
+ /*
+ * It's possible to sort the sample rows directly, but this seemed
+ * somehow simpler / less error prone. Another option would be to
+ * allocate the arrays for each SortItem separately, but that'd be
+ * significant overhead (not just CPU, but especially memory bloat).
+ */
+ SortItem * items = (SortItem*)palloc0(numrows * sizeof(SortItem));
+
+ Datum *values = (Datum*)palloc0(sizeof(Datum) * numrows * numattrs);
+ bool *isnull = (bool*)palloc0(sizeof(bool) * numrows * numattrs);
+
+ for (i = 0; i < numrows; i++)
+ {
+ items[i].values = &values[i * numattrs];
+ items[i].isnull = &isnull[i * numattrs];
+ }
+
+ Assert(numattrs >= 2);
+
+ for (i = 0; i < numattrs; i++)
+ {
+ /* prepare the sort function for the first dimension */
+ multi_sort_add_dimension(mss, i, i, stats);
+
+ /* accumulate all the data into the array and sort it */
+ for (j = 0; j < numrows; j++)
+ {
+ items[j].values[i]
+ = heap_getattr(rows[j], attrs->values[i],
+ stats[i]->tupDesc, &items[j].isnull[i]);
+ }
+ }
+
+ qsort_arg((void *) items, numrows, sizeof(SortItem),
+ multi_sort_compare, mss);
+
+ /* count number of distinct combinations */
+
+ f1 = 0;
+ cnt = 1;
+ d = 1;
+ for (i = 1; i < numrows; i++)
+ {
+ if (multi_sort_compare(&items[i], &items[i-1], mss) != 0)
+ {
+ if (cnt == 1)
+ f1 += 1;
+ else
+ {
+ nmultiple += 1;
+ summultiple += cnt;
+ }
+
+ d++;
+ cnt = 0;
+ }
+
+ cnt += 1;
+ }
+
+ if (cnt == 1)
+ f1 += 1;
+ else
+ {
+ nmultiple += 1;
+ summultiple += cnt;
+ }
+
+ ndistcoeff = 1 / estimate_ndistinct(totalrows, numrows, d, f1);
+
+ /*
+ * now count distinct values for each attribute and incrementally
+ * compute ndistinct(a,b) / (ndistinct(a) * ndistinct(b))
+ *
+ * FIXME Probably need to handle cases when one of the ndistinct
+ * estimates is negative, and also check that the combined
+ * ndistinct is greater than any of those partial values.
+ */
+ for (i = 0; i < numattrs; i++)
+ ndistcoeff *= stats[i]->stadistinct;
+
+ return ndistcoeff;
+}
+
+double
+load_mv_ndistinct(Oid mvoid)
+{
+ bool isnull = false;
+ Datum deps;
+
+ /* Prepare to scan pg_mv_statistic for entries having indrelid = this rel. */
+ HeapTuple htup = SearchSysCache1(MVSTATOID, ObjectIdGetDatum(mvoid));
+
+#ifdef USE_ASSERT_CHECKING
+ Form_pg_mv_statistic mvstat = (Form_pg_mv_statistic) GETSTRUCT(htup);
+ Assert(mvstat->ndist_enabled && mvstat->ndist_built);
+#endif
+
+ deps = SysCacheGetAttr(MVSTATOID, htup,
+ Anum_pg_mv_statistic_standist, &isnull);
+
+ Assert(!isnull);
+
+ ReleaseSysCache(htup);
+
+ return DatumGetFloat8(deps);
+}
+
+/* The Duj1 estimator (already used in analyze.c). */
+static double
+estimate_ndistinct(double totalrows, int numrows, int d, int f1)
+{
+ double numer,
+ denom,
+ ndistinct;
+
+ numer = (double) numrows *(double) d;
+
+ denom = (double) (numrows - f1) +
+ (double) f1 * (double) numrows / totalrows;
+
+ ndistinct = numer / denom;
+
+ /* Clamp to sane range in case of roundoff error */
+ if (ndistinct < (double) d)
+ ndistinct = (double) d;
+
+ if (ndistinct > totalrows)
+ ndistinct = totalrows;
+
+ return floor(ndistinct + 0.5);
+}
diff --git a/src/include/catalog/pg_mv_statistic.h b/src/include/catalog/pg_mv_statistic.h
index 7020772..e46cc6b 100644
--- a/src/include/catalog/pg_mv_statistic.h
+++ b/src/include/catalog/pg_mv_statistic.h
@@ -40,6 +40,7 @@ CATALOG(pg_mv_statistic,3381)
bool deps_enabled; /* analyze dependencies? */
bool mcv_enabled; /* build MCV list? */
bool hist_enabled; /* build histogram? */
+ bool ndist_enabled; /* build ndist coefficient? */
/* histogram / MCV size */
int32 mcv_max_items; /* max MCV items */
@@ -49,6 +50,7 @@ CATALOG(pg_mv_statistic,3381)
bool deps_built; /* dependencies were built */
bool mcv_built; /* MCV list was built */
bool hist_built; /* histogram was built */
+ bool ndist_built; /* ndistinct coeff built */
/* variable-length fields start here, but we allow direct access to stakeys */
int2vector stakeys; /* array of column keys */
@@ -57,6 +59,7 @@ CATALOG(pg_mv_statistic,3381)
bytea stadeps; /* dependencies (serialized) */
bytea stamcv; /* MCV list (serialized) */
bytea stahist; /* MV histogram (serialized) */
+ float8 standcoeff; /* ndistinct coeff (serialized) */
#endif
} FormData_pg_mv_statistic;
@@ -72,7 +75,7 @@ typedef FormData_pg_mv_statistic *Form_pg_mv_statistic;
* compiler constants for pg_mv_statistic
* ----------------
*/
-#define Natts_pg_mv_statistic 16
+#define Natts_pg_mv_statistic 19
#define Anum_pg_mv_statistic_starelid 1
#define Anum_pg_mv_statistic_staname 2
#define Anum_pg_mv_statistic_stanamespace 3
@@ -80,14 +83,17 @@ typedef FormData_pg_mv_statistic *Form_pg_mv_statistic;
#define Anum_pg_mv_statistic_deps_enabled 5
#define Anum_pg_mv_statistic_mcv_enabled 6
#define Anum_pg_mv_statistic_hist_enabled 7
-#define Anum_pg_mv_statistic_mcv_max_items 8
-#define Anum_pg_mv_statistic_hist_max_buckets 9
-#define Anum_pg_mv_statistic_deps_built 10
-#define Anum_pg_mv_statistic_mcv_built 11
-#define Anum_pg_mv_statistic_hist_built 12
-#define Anum_pg_mv_statistic_stakeys 13
-#define Anum_pg_mv_statistic_stadeps 14
-#define Anum_pg_mv_statistic_stamcv 15
-#define Anum_pg_mv_statistic_stahist 16
+#define Anum_pg_mv_statistic_ndist_enabled 8
+#define Anum_pg_mv_statistic_mcv_max_items 9
+#define Anum_pg_mv_statistic_hist_max_buckets 19
+#define Anum_pg_mv_statistic_deps_built 11
+#define Anum_pg_mv_statistic_mcv_built 12
+#define Anum_pg_mv_statistic_hist_built 13
+#define Anum_pg_mv_statistic_ndist_built 14
+#define Anum_pg_mv_statistic_stakeys 15
+#define Anum_pg_mv_statistic_stadeps 16
+#define Anum_pg_mv_statistic_stamcv 17
+#define Anum_pg_mv_statistic_stahist 18
+#define Anum_pg_mv_statistic_standist 19
#endif /* PG_MV_STATISTIC_H */
diff --git a/src/include/nodes/relation.h b/src/include/nodes/relation.h
index 8c50bfb..1923f2b 100644
--- a/src/include/nodes/relation.h
+++ b/src/include/nodes/relation.h
@@ -655,11 +655,13 @@ typedef struct MVStatisticInfo
bool deps_enabled; /* functional dependencies enabled */
bool mcv_enabled; /* MCV list enabled */
bool hist_enabled; /* histogram enabled */
+ bool ndist_enabled; /* ndistinct coefficient enabled */
/* built/available statistics */
bool deps_built; /* functional dependencies built */
bool mcv_built; /* MCV list built */
bool hist_built; /* histogram built */
+ bool ndist_built; /* ndistinct coefficient built */
/* columns in the statistics (attnums) */
int2vector *stakeys; /* attnums of the columns covered */
diff --git a/src/include/utils/mvstats.h b/src/include/utils/mvstats.h
index 80bf96f..0ff24ce 100644
--- a/src/include/utils/mvstats.h
+++ b/src/include/utils/mvstats.h
@@ -226,6 +226,7 @@ typedef MVSerializedHistogramData *MVSerializedHistogram;
MVDependencies load_mv_dependencies(Oid mvoid);
MCVList load_mv_mcvlist(Oid mvoid);
MVSerializedHistogram load_mv_histogram(Oid mvoid);
+double load_mv_ndistinct(Oid mvoid);
bytea * serialize_mv_dependencies(MVDependencies dependencies);
bytea * serialize_mv_mcvlist(MCVList mcvlist, int2vector *attrs,
@@ -267,11 +268,17 @@ MVHistogram
build_mv_histogram(int numrows, HeapTuple *rows, int2vector *attrs,
VacAttrStats **stats, int numrows_total);
-void build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
+double
+build_mv_ndistinct(double totalrows, int numrows, HeapTuple *rows,
+ int2vector *attrs, VacAttrStats **stats);
+
+void build_mv_stats(Relation onerel, double totalrows,
+ int numrows, HeapTuple *rows,
int natts, VacAttrStats **vacattrstats);
void update_mv_stats(Oid relid, MVDependencies dependencies,
MCVList mcvlist, MVHistogram histogram,
+ double ndistcoeff,
int2vector *attrs, VacAttrStats **stats);
#ifdef DEBUG_MVHIST
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 528ac36..7a914da 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1377,7 +1377,8 @@ pg_mv_stats| SELECT n.nspname AS schemaname,
length(s.stamcv) AS mcvbytes,
pg_mv_stats_mcvlist_info(s.stamcv) AS mcvinfo,
length(s.stahist) AS histbytes,
- pg_mv_stats_histogram_info(s.stahist) AS histinfo
+ pg_mv_stats_histogram_info(s.stahist) AS histinfo,
+ s.standcoeff AS ndcoeff
FROM ((pg_mv_statistic s
JOIN pg_class c ON ((c.oid = s.starelid)))
LEFT JOIN pg_namespace n ON ((n.oid = c.relnamespace)));
--
2.5.0
[text/x-patch] 0006-multi-statistics-estimation.patch (98.8K, ../../1457906378.27231.10.camel@2ndquadrant.com/4-0006-multi-statistics-estimation.patch)
download | inline diff:
From 91b9b31cbeb22767b33c2f58b912b7a14c943b28 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@pgaddict.com>
Date: Fri, 6 Feb 2015 01:42:38 +0100
Subject: [PATCH 6/9] multi-statistics estimation
The general idea is that a probability (which is what selectivity is)
can be split into a product of conditional probabilities like this:
P(A & B & C) = P(A & B) * P(C|A & B)
If we assume that C and B are independent, the last part may be
simplified like this
P(A & B & C) = P(A & B) * P(C|A)
we only need probabilities on [A,B] and [C,A] to compute the original
probability.
The implementation works in the other direction, though. We know what
probability P(A & B & C) we need to compute, and also what statistics
are available.
So we search for a combinations of statistics, covering the clauses in
an optimal way (most clauses covered, most dependencies exploited).
There are two possible approaches - exhaustive and greedy. The
exhaustive one walks through all permutations of stats using dynamic
programming, so it's guaranteed to find the optimal solution, but it
soon gets very slow as it's roughly O(N!). The dynamic programming may
improve that a bit, but it's still far too expensive for large numbers
of statistics (on a single table).
The greedy algorithm is very simple - in every step choose the best
solution. That may not guarantee the best solution globally (but maybe
it does?), but it only needs N steps to find the solution, so it's very
fast (processing the selected stats is usually way more expensive).
There's a GUC for selecting the search algorithm
mvstat_search = {'greedy', 'exhaustive'}
The default value is 'greedy' as that's much safer (with respect to
runtime). See choose_mv_statistics().
Once we have found a sequence of statistics, we apply them to the
clauses using the conditional probabilities. We process the selected
stats one by one, and for each we select the estimated clauses and
conditions. See clauselist_selectivity() for more details.
Limitations
-----------
It's still true that each clause at a given level has to be covered by
a single MV statistics. So with this query
WHERE (clause1) AND (clause2) AND (clause3 OR clause4)
each parenthesized clause has to be covered by a single multivariate
statistics.
Clauses not covered by a single statistics at this level will be passed
to clause_selectivity() but this will treat them as a collection of
simpler clauses (connected by AND or OR), and the clauses from the
previous level will be used as conditions.
So using the same example, the last clause will be passed to
clause_selectivity() with 'clause1' and 'clause2' as conditions, and it
will be processed using multivariate stats if possible.
The other limitation is that all the expressions have to be
mv-compatible, i.e. there can't be a mix of expressions. If this is
violated, the clause may be passed to the next level (just like with
list of clauses not covered by a single statistics), which splits that
into clauses handled by multivariate stats and clauses handler by
regular statistics.
rework clauselist_selectivity_or to handle OR-clauses correctly
We might invent a completely new set of functions here, resembling
clauselist_selectivity but adapting the ideas to OR-clauses.
But luckily we know that each OR-clause
(a OR b OR c)
may be rewritten as an equivalent AND-clause using negation:
NOT ((NOT a) AND (NOT b) AND (NOT c))
And that's something we can pass to clauselist_selectivity.
---
contrib/file_fdw/file_fdw.c | 3 +-
contrib/postgres_fdw/postgres_fdw.c | 11 +-
src/backend/optimizer/path/clausesel.c | 2024 ++++++++++++++++++++++++++------
src/backend/optimizer/path/costsize.c | 23 +-
src/backend/optimizer/util/orclauses.c | 4 +-
src/backend/utils/adt/selfuncs.c | 17 +-
src/backend/utils/misc/guc.c | 20 +
src/backend/utils/mvstats/README.stats | 166 +++
src/include/optimizer/cost.h | 6 +-
src/include/utils/mvstats.h | 8 +
10 files changed, 1913 insertions(+), 369 deletions(-)
diff --git a/contrib/file_fdw/file_fdw.c b/contrib/file_fdw/file_fdw.c
index dc035d7..8f11b7a 100644
--- a/contrib/file_fdw/file_fdw.c
+++ b/contrib/file_fdw/file_fdw.c
@@ -969,7 +969,8 @@ estimate_size(PlannerInfo *root, RelOptInfo *baserel,
baserel->baserestrictinfo,
0,
JOIN_INNER,
- NULL);
+ NULL,
+ NIL);
nrows = clamp_row_est(nrows);
diff --git a/contrib/postgres_fdw/postgres_fdw.c b/contrib/postgres_fdw/postgres_fdw.c
index 40bffd6..d458a81 100644
--- a/contrib/postgres_fdw/postgres_fdw.c
+++ b/contrib/postgres_fdw/postgres_fdw.c
@@ -500,7 +500,8 @@ postgresGetForeignRelSize(PlannerInfo *root,
fpinfo->local_conds,
baserel->relid,
JOIN_INNER,
- NULL);
+ NULL,
+ NIL);
cost_qual_eval(&fpinfo->local_conds_cost, fpinfo->local_conds, root);
@@ -2136,7 +2137,8 @@ estimate_path_cost_size(PlannerInfo *root,
local_param_join_conds,
foreignrel->relid,
JOIN_INNER,
- NULL);
+ NULL,
+ NIL);
local_sel *= fpinfo->local_conds_sel;
rows = clamp_row_est(rows * local_sel);
@@ -3663,7 +3665,8 @@ postgresGetForeignJoinPaths(PlannerInfo *root,
fpinfo->local_conds,
0,
JOIN_INNER,
- NULL);
+ NULL,
+ NIL);
cost_qual_eval(&fpinfo->local_conds_cost, fpinfo->local_conds, root);
/*
@@ -3682,7 +3685,7 @@ postgresGetForeignJoinPaths(PlannerInfo *root,
*/
fpinfo->joinclause_sel = clauselist_selectivity(root, fpinfo->joinclauses,
0, fpinfo->jointype,
- extra->sjinfo);
+ extra->sjinfo, NIL);
}
fpinfo->server = GetForeignServer(joinrel->serverid);
diff --git a/src/backend/optimizer/path/clausesel.c b/src/backend/optimizer/path/clausesel.c
index 5e73a4e..e06fd99 100644
--- a/src/backend/optimizer/path/clausesel.c
+++ b/src/backend/optimizer/path/clausesel.c
@@ -29,6 +29,8 @@
#include "utils/selfuncs.h"
#include "utils/typcache.h"
+#include "miscadmin.h"
+
/*
* Data structure for accumulating info about possible range-query
@@ -44,6 +46,13 @@ typedef struct RangeQueryClause
Selectivity hibound; /* Selectivity of a var < something clause */
} RangeQueryClause;
+static Selectivity clauselist_selectivity_or(PlannerInfo *root,
+ List *clauses,
+ int varRelid,
+ JoinType jointype,
+ SpecialJoinInfo *sjinfo,
+ List *conditions);
+
static void addRangeClause(RangeQueryClause **rqlist, Node *clause,
bool varonleft, bool isLTsel, Selectivity s2);
@@ -60,23 +69,25 @@ static int count_mv_attnums(List *clauses, Index relid, int type);
static int count_varnos(List *clauses, Index *relid);
+static List *clauses_matching_statistic(List **clauses, MVStatisticInfo *statistic,
+ Index relid, int types, bool remove);
+
static List *clauselist_apply_dependencies(PlannerInfo *root, List *clauses,
Index relid, List *stats);
-static MVStatisticInfo *choose_mv_statistics(List *mvstats, Bitmapset *attnums);
-
-static List *clauselist_mv_split(PlannerInfo *root, Index relid,
- List *clauses, List **mvclauses,
- MVStatisticInfo *mvstats, int types);
-
static Selectivity clauselist_mv_selectivity(PlannerInfo *root,
- List *clauses, MVStatisticInfo *mvstats);
+ MVStatisticInfo *mvstats, List *clauses,
+ List *conditions, bool is_or);
static Selectivity clauselist_mv_selectivity_mcvlist(PlannerInfo *root,
- List *clauses, MVStatisticInfo *mvstats,
- bool *fullmatch, Selectivity *lowsel);
+ MVStatisticInfo *mvstats,
+ List *clauses, List *conditions,
+ bool is_or, bool *fullmatch,
+ Selectivity *lowsel);
static Selectivity clauselist_mv_selectivity_histogram(PlannerInfo *root,
- List *clauses, MVStatisticInfo *mvstats);
+ MVStatisticInfo *mvstats,
+ List *clauses, List *conditions,
+ bool is_or);
static int update_match_bitmap_mcvlist(PlannerInfo *root, List *clauses,
int2vector *stakeys, MCVList mcvlist,
@@ -90,12 +101,33 @@ static int update_match_bitmap_histogram(PlannerInfo *root, List *clauses,
int nmatches, char * matches,
bool is_or);
+/*
+ * Describes a combination of multiple statistics to cover attributes
+ * referenced by the clauses. The array 'stats' (with nstats elements)
+ * lists attributes (in the order as they are applied), and number of
+ * clause attributes covered by this solution.
+ *
+ * choose_mv_statistics_exhaustive() uses this to track both the current
+ * and the best solutions, while walking through the state of possible
+ * combination.
+ */
+typedef struct mv_solution_t {
+ int nclauses; /* number of clauses covered */
+ int nconditions; /* number of conditions covered */
+ int nstats; /* number of stats applied */
+ int *stats; /* stats (in the apply order) */
+} mv_solution_t;
+
+static List *choose_mv_statistics(PlannerInfo *root, Index relid,
+ List *mvstats, List *clauses, List *conditions);
+
static bool has_stats(List *stats, int type);
static List * find_stats(PlannerInfo *root, Index relid);
static bool stats_type_matches(MVStatisticInfo *stat, int type);
+int mvstat_search_type = MVSTAT_SEARCH_GREEDY;
/* used for merging bitmaps - AND (min), OR (max) */
#define MAX(x, y) (((x) > (y)) ? (x) : (y))
@@ -170,14 +202,15 @@ clauselist_selectivity(PlannerInfo *root,
List *clauses,
int varRelid,
JoinType jointype,
- SpecialJoinInfo *sjinfo)
+ SpecialJoinInfo *sjinfo,
+ List *conditions)
{
Selectivity s1 = 1.0;
RangeQueryClause *rqlist = NULL;
ListCell *l;
/* processing mv stats */
- Oid relid = InvalidOid;
+ Index relid = InvalidOid;
/* list of multivariate stats on the relation */
List *stats = NIL;
@@ -193,12 +226,13 @@ clauselist_selectivity(PlannerInfo *root,
stats = find_stats(root, relid);
/*
- * If there's exactly one clause, then no use in trying to match up pairs,
- * so just go directly to clause_selectivity().
+ * If there's exactly one clause, then no use in trying to match up
+ * pairs, or matching multivariate statistics, so just go directly
+ * to clause_selectivity().
*/
if (list_length(clauses) == 1)
return clause_selectivity(root, (Node *) linitial(clauses),
- varRelid, jointype, sjinfo);
+ varRelid, jointype, sjinfo, conditions);
/*
* Apply functional dependencies, but first check that there are some stats
@@ -230,31 +264,100 @@ clauselist_selectivity(PlannerInfo *root,
(count_mv_attnums(clauses, relid,
MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST) >= 2))
{
- /* collect attributes from the compatible conditions */
- Bitmapset *mvattnums = collect_mv_attnums(clauses, relid,
- MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST);
+ ListCell *s;
+
+ /*
+ * Copy the conditions we got from the upper part of the expression tree
+ * so that we can add local conditions to it (we need to keep the
+ * original list intact, for sibling expressions - other expressions
+ * at the same level).
+ */
+ List *conditions_local = list_copy(conditions);
+
+ /* find the best combination of statistics */
+ List *solution = choose_mv_statistics(root, relid, stats,
+ clauses, conditions);
- /* and search for the statistic covering the most attributes */
- MVStatisticInfo *mvstat = choose_mv_statistics(stats, mvattnums);
+ /* FIXME we must not scribble over the original list */
+ if (solution)
+ clauses = list_copy(clauses);
- if (mvstat != NULL) /* we have a matching stats */
+ /*
+ * We have a good solution, which is merely a list of statistics that
+ * we need to apply. We'll apply the statistics one by one (in the order
+ * as they appear in the list), and for each statistic we'll
+ *
+ * (1) find clauses compatible with the statistic (and remove them
+ * from the list)
+ *
+ * (2) find local conditions compatible with the statistic
+ *
+ * (3) do the estimation P(clauses | conditions)
+ *
+ * (4) append the estimated clauses to local conditions
+ *
+ * continuously modify
+ */
+ foreach (s, solution)
{
- /* clauses compatible with multi-variate stats */
- List *mvclauses = NIL;
+ MVStatisticInfo *mvstat = (MVStatisticInfo *)lfirst(s);
- /* split the clauselist into regular and mv-clauses */
- clauses = clauselist_mv_split(root, relid, clauses, &mvclauses,
- mvstat, MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST);
+ /* clauses compatible with the statistic we're applying right now */
+ List *stat_clauses = NIL;
+ List *stat_conditions = NIL;
- /* we've chosen the histogram to match the clauses */
- Assert(mvclauses != NIL);
+ /*
+ * Find clauses and conditions matching the statistic - the clauses
+ * need to be removed from the list, while conditions should remain
+ * there (so that we can apply them repeatedly).
+ */
+ stat_clauses
+ = clauses_matching_statistic(&clauses, mvstat, relid,
+ MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST,
+ true);
+
+ stat_conditions
+ = clauses_matching_statistic(&conditions_local, mvstat, relid,
+ MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST,
+ false);
+
+ /*
+ * If we got no clauses to estimate, we've done something wrong,
+ * either during the optimization, detecting compatible clause, or
+ * somewhere else.
+ *
+ * Also, we need at least two attributes in clauses and conditions.
+ */
+ Assert(stat_clauses != NIL);
+ Assert(count_mv_attnums(list_union(stat_clauses, stat_conditions),
+ relid, MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST) >= 2);
/* compute the multivariate stats */
- s1 *= clauselist_mv_selectivity(root, mvclauses, mvstat);
+ s1 *= clauselist_mv_selectivity(root, mvstat,
+ stat_clauses, stat_conditions,
+ false); /* AND */
+
+ /*
+ * Add the new clauses to the local conditions, so that we can use
+ * them for the subsequent statistics. We only add the clauses,
+ * because the conditions are already there (or should be).
+ */
+ conditions_local = list_concat(conditions_local, stat_clauses);
}
+
+ /* from now on, work only with the 'local' list of conditions */
+ conditions = conditions_local;
}
/*
+ * If there's exactly one clause, then no use in trying to match up
+ * pairs, so just go directly to clause_selectivity().
+ */
+ if (list_length(clauses) == 1)
+ return s1 * clause_selectivity(root, (Node *) linitial(clauses),
+ varRelid, jointype, sjinfo, conditions);
+
+ /*
* Initial scan over clauses. Anything that doesn't look like a potential
* rangequery clause gets multiplied into s1 and forgotten. Anything that
* does gets inserted into an rqlist entry.
@@ -266,7 +369,8 @@ clauselist_selectivity(PlannerInfo *root,
Selectivity s2;
/* Always compute the selectivity using clause_selectivity */
- s2 = clause_selectivity(root, clause, varRelid, jointype, sjinfo);
+ s2 = clause_selectivity(root, clause, varRelid, jointype, sjinfo,
+ conditions);
/*
* Check for being passed a RestrictInfo.
@@ -425,6 +529,55 @@ clauselist_selectivity(PlannerInfo *root,
}
/*
+ * Similar to clauselist_selectivity(), but for OR-clauses. We can't simply use
+ * the same multi-statistic estimation logic for AND-clauses, at least not
+ * directly, because there are a few key differences:
+ *
+ * - functional dependencies don't really apply to OR-clauses
+ *
+ * - clauselist_selectivity() is based on decomposing the selectivity into
+ * a sequence of conditional probabilities (selectivities), but that can
+ * be done only for AND-clauses
+ *
+ * We might invent a similar infrastructure for optimizing OR-clauses, doing
+ * something similar to what clause_selectivity does for AND-clauses, but
+ * luckily we know that each disjunctive normal form (aka OR-clause)
+ *
+ * (a OR b OR c)
+ *
+ * may be rewritten as an equivalent conjunctive normal form (aka AND-clause)
+ * by using negation:
+ *
+ * NOT ((NOT a) AND (NOT b) AND (NOT c))
+ *
+ * And that's something we can pass to clauselist_selectivity and let it do
+ * all the heavy lifting.
+ */
+static Selectivity
+clauselist_selectivity_or(PlannerInfo *root,
+ List *clauses,
+ int varRelid,
+ JoinType jointype,
+ SpecialJoinInfo *sjinfo,
+ List *conditions)
+{
+ List *args = NIL;
+ ListCell *l;
+ Expr *expr;
+
+ /* build arguments for the AND-clause by negating args of the OR-clause */
+ foreach (l, clauses)
+ args = lappend(args, makeBoolExpr(NOT_EXPR, list_make1(lfirst(l)), -1));
+
+ /* and then the actual OR-clause on the negated args */
+ expr = makeBoolExpr(AND_EXPR, args, -1);
+
+ /* instead of constructing NOT expression, just do (1.0 - s) */
+ return 1.0 - clauselist_selectivity(root, list_make1(expr), varRelid,
+ jointype, sjinfo, conditions);
+}
+
+/*
* addRangeClause --- add a new range clause for clauselist_selectivity
*
* Here is where we try to match up pairs of range-query clauses
@@ -631,7 +784,8 @@ clause_selectivity(PlannerInfo *root,
Node *clause,
int varRelid,
JoinType jointype,
- SpecialJoinInfo *sjinfo)
+ SpecialJoinInfo *sjinfo,
+ List *conditions)
{
Selectivity s1 = 0.5; /* default for any unhandled clause type */
RestrictInfo *rinfo = NULL;
@@ -751,7 +905,8 @@ clause_selectivity(PlannerInfo *root,
(Node *) get_notclausearg((Expr *) clause),
varRelid,
jointype,
- sjinfo);
+ sjinfo,
+ conditions);
}
else if (and_clause(clause))
{
@@ -760,29 +915,18 @@ clause_selectivity(PlannerInfo *root,
((BoolExpr *) clause)->args,
varRelid,
jointype,
- sjinfo);
+ sjinfo,
+ conditions);
}
else if (or_clause(clause))
{
- /*
- * Selectivities for an OR clause are computed as s1+s2 - s1*s2 to
- * account for the probable overlap of selected tuple sets.
- *
- * XXX is this too conservative?
- */
- ListCell *arg;
-
- s1 = 0.0;
- foreach(arg, ((BoolExpr *) clause)->args)
- {
- Selectivity s2 = clause_selectivity(root,
- (Node *) lfirst(arg),
- varRelid,
- jointype,
- sjinfo);
-
- s1 = s1 + s2 - s1 * s2;
- }
+ /* just call to clauselist_selectivity_or() */
+ s1 = clauselist_selectivity_or(root,
+ ((BoolExpr *) clause)->args,
+ varRelid,
+ jointype,
+ sjinfo,
+ conditions);
}
else if (is_opclause(clause) || IsA(clause, DistinctExpr))
{
@@ -872,7 +1016,8 @@ clause_selectivity(PlannerInfo *root,
(Node *) ((RelabelType *) clause)->arg,
varRelid,
jointype,
- sjinfo);
+ sjinfo,
+ conditions);
}
else if (IsA(clause, CoerceToDomain))
{
@@ -881,7 +1026,8 @@ clause_selectivity(PlannerInfo *root,
(Node *) ((CoerceToDomain *) clause)->arg,
varRelid,
jointype,
- sjinfo);
+ sjinfo,
+ conditions);
}
else
{
@@ -945,15 +1091,16 @@ clause_selectivity(PlannerInfo *root,
* in the MCV list, then the selectivity is below the lowest frequency
* found in the MCV list,
*
- * TODO When applying the clauses to the histogram/MCV list, we can do
- * that from the most selective clauses first, because that'll
- * eliminate the buckets/items sooner (so we'll be able to skip
- * them without inspection, which is more expensive). But this
- * requires really knowing the per-clause selectivities in advance,
- * and that's not what we do now.
+ * TODO When applying the clauses to the histogram/MCV list, we can do that from
+ * the most selective clauses first, because that'll eliminate the
+ * buckets/items sooner (so we'll be able to skip them without inspection,
+ * which is more expensive). But this requires really knowing the
+ * per-clause selectivities in advance, and that's not what we do now.
+ *
*/
static Selectivity
-clauselist_mv_selectivity(PlannerInfo *root, List *clauses, MVStatisticInfo *mvstats)
+clauselist_mv_selectivity(PlannerInfo *root, MVStatisticInfo *mvstats,
+ List *clauses, List *conditions, bool is_or)
{
bool fullmatch = false;
Selectivity s1 = 0.0, s2 = 0.0;
@@ -964,281 +1111,1375 @@ clauselist_mv_selectivity(PlannerInfo *root, List *clauses, MVStatisticInfo *mvs
*/
Selectivity mcv_low = 0.0;
- /* TODO Evaluate simple 1D selectivities, use the smallest one as
- * an upper bound, product as lower bound, and sort the
- * clauses in ascending order by selectivity (to optimize the
- * MCV/histogram evaluation).
- */
+ /* TODO Evaluate simple 1D selectivities, use the smallest one as
+ * an upper bound, product as lower bound, and sort the
+ * clauses in ascending order by selectivity (to optimize the
+ * MCV/histogram evaluation).
+ */
+
+ /* Evaluate the MCV first. */
+ s1 = clauselist_mv_selectivity_mcvlist(root, mvstats,
+ clauses, conditions, is_or,
+ &fullmatch, &mcv_low);
+
+ /*
+ * If we got a full equality match on the MCV list, we're done (and
+ * the estimate is pretty good).
+ */
+ if (fullmatch && (s1 > 0.0))
+ return s1;
+
+ /* TODO if (fullmatch) without matching MCV item, use the mcv_low
+ * selectivity as upper bound */
+
+ s2 = clauselist_mv_selectivity_histogram(root, mvstats,
+ clauses, conditions, is_or);
+
+ /* TODO clamp to <= 1.0 (or more strictly, when possible) */
+ return s1 + s2;
+}
+
+/*
+ * Pull varattnos from the clauses, similarly to pull_varattnos() but:
+ *
+ * (a) only get attributes for a particular relation (relid)
+ * (b) ignore system attributes (we can't build stats on them anyway)
+ *
+ * This makes it possible to directly compare the result with attnum
+ * values from pg_attribute etc.
+ */
+static Bitmapset *
+get_varattnos(Node * node, Index relid)
+{
+ int k;
+ Bitmapset *varattnos = NULL;
+ Bitmapset *result = NULL;
+
+ /* get the varattnos */
+ pull_varattnos(node, relid, &varattnos);
+
+ k = -1;
+ while ((k = bms_next_member(varattnos, k)) >= 0)
+ {
+ if (k + FirstLowInvalidHeapAttributeNumber > 0)
+ result
+ = bms_add_member(result,
+ k + FirstLowInvalidHeapAttributeNumber);
+ }
+
+ bms_free(varattnos);
+
+ return result;
+}
+
+/*
+ * Collect attributes from mv-compatible clauses.
+ */
+static Bitmapset *
+collect_mv_attnums(List *clauses, Index relid, int types)
+{
+ Bitmapset *attnums = NULL;
+ ListCell *l;
+
+ /*
+ * Walk through the clauses and identify the ones we can estimate
+ * using multivariate stats, and remember the relid/columns. We'll
+ * then cross-check if we have suitable stats, and only if needed
+ * we'll split the clauses into multivariate and regular lists.
+ *
+ * For now we're only interested in RestrictInfo nodes with nested
+ * OpExpr, using either a range or equality.
+ */
+ foreach (l, clauses)
+ {
+ Node *clause = (Node *) lfirst(l);
+
+ /* ignore the result here - we only need the attnums */
+ clause_is_mv_compatible(clause, relid, &attnums, types);
+ }
+
+ /*
+ * If there are not at least two attributes referenced by the clause(s),
+ * we can throw everything out (as we'll revert to simple stats).
+ */
+ if (bms_num_members(attnums) <= 1)
+ {
+ bms_free(attnums);
+ attnums = NULL;
+ }
+
+ return attnums;
+}
+
+/*
+ * Count the number of attributes in clauses compatible with multivariate stats.
+ */
+static int
+count_mv_attnums(List *clauses, Index relid, int type)
+{
+ int c;
+ Bitmapset *attnums = collect_mv_attnums(clauses, relid, type);
+
+ c = bms_num_members(attnums);
+
+ bms_free(attnums);
+
+ return c;
+}
+
+/*
+ * Count varnos referenced in the clauses, and if there's a single varno then
+ * return the index in 'relid'.
+ */
+static int
+count_varnos(List *clauses, Index *relid)
+{
+ int cnt;
+ Bitmapset *varnos = NULL;
+
+ varnos = pull_varnos((Node *) clauses);
+ cnt = bms_num_members(varnos);
+
+ /* if there's a single varno in the clauses, remember it */
+ if (bms_num_members(varnos) == 1)
+ *relid = bms_singleton_member(varnos);
+
+ bms_free(varnos);
+
+ return cnt;
+}
+
+static List *
+clauses_matching_statistic(List **clauses, MVStatisticInfo *statistic,
+ Index relid, int types, bool remove)
+{
+ int i;
+ Bitmapset *stat_attnums = NULL;
+ List *matching_clauses = NIL;
+ ListCell *lc;
+
+ /* build attnum bitmapset for this statistics */
+ for (i = 0; i < statistic->stakeys->dim1; i++)
+ stat_attnums = bms_add_member(stat_attnums,
+ statistic->stakeys->values[i]);
+
+ /*
+ * We can't use foreach here, because we may need to remove some of the
+ * clauses if (remove=true).
+ */
+ lc = list_head(*clauses);
+ while (lc)
+ {
+ Node *clause = (Node*)lfirst(lc);
+ Bitmapset *attnums = NULL;
+
+ /* must advance lc before list_delete possibly pfree's it */
+ lc = lnext(lc);
+
+ /*
+ * skip clauses that are not compatible with stats (just leave them
+ * in the original list)
+ *
+ * XXX Perhaps this should check what stats are actually available in
+ * the statistics (not a big deal now, because MCV and histograms
+ * handle the same types of conditions).
+ */
+ if (! clause_is_mv_compatible(clause, relid, &attnums, types))
+ {
+ bms_free(attnums);
+ continue;
+ }
+
+ /* if the clause is covered by the statistic, add it to the list */
+ if (bms_is_subset(attnums, stat_attnums))
+ {
+ matching_clauses = lappend(matching_clauses, clause);
+
+ /* if remove=true, remove the matching item from the main list */
+ if (remove)
+ *clauses = list_delete_ptr(*clauses, clause);
+ }
+
+ bms_free(attnums);
+ }
+
+ bms_free(stat_attnums);
+
+ return matching_clauses;
+}
+
+/*
+ * Selects the best combination of multivariate statistics, in an exhaustive
+ * way, where 'best' means:
+ *
+ * (a) covering the most attributes (referenced by clauses)
+ * (b) using the least number of multivariate stats
+ * (c) using the most conditions to exploit dependency
+ *
+ * Don't call this directly but through choose_mv_statistics(), which does some
+ * additional tricks to minimize the runtime.
+ *
+ *
+ * Algorithm
+ * ---------
+ * The algorithm is a recursive implementation of backtracking, with maximum
+ * depth equal to the number of multi-variate statistics available on the table.
+ * It actually explores all valid combinations of stats.
+ *
+ * Whenever it considers adding the next statistics, the clauses it matches are
+ * divided into 'conditions' (clauses already matched by at least one previous
+ * statistics) and clauses that are estimated.
+ *
+ * Then several checks are performed:
+ *
+ * (a) The statistics covers at least 2 columns, referenced in the estimated
+ * clauses (otherwise multi-variate stats are useless).
+ *
+ * (b) The statistics covers at least 1 new column, i.e. column not refefenced
+ * by the already used stats (and the new column has to be referenced by
+ * the clauses, of couse). Otherwise the statistics would not add any new
+ * information.
+ *
+ * There are some other sanity checks (e.g. stats must not be used twice etc.).
+ *
+ *
+ * Weaknesses
+ * ----------
+ * The current implemetation uses a rather simple optimality criteria, so it may
+ * not do the best choice when
+ *
+ * (a) There may be multiple solutions with the same number of covered
+ * attributes and number of statistics (e.g. the same solution but with
+ * statistics in a different order). It's unclear which solution in the best
+ * one - in a sense all of them are equal.
+ *
+ * TODO It might be possible to compute estimate for each of those solutions,
+ * and then combine them to get the final estimate (e.g. by using average
+ * or median).
+ *
+ * (b) Does not consider that some types of stats are a better match for some
+ * types of clauses (e.g. MCV list is generally a better match for equality
+ * conditions than a histogram).
+ *
+ * But maybe this is pointless - generally, each column is either a label
+ * (it's not important whether because of the data type or how it's used),
+ * or a value with ordering that makes sense. So either a MCV list is more
+ * appropriate (labels) or a histogram (values with orderings).
+ *
+ * Now sure what to do with statistics on columns mixing both types of data
+ * (some columns would work best with MCVs, some with histograms). Maybe we
+ * could invent a new type of statistics combining MCV list and histogram
+ * (keeping a small histogram for each MCV item, and a separate histogram
+ * for values not on the MCV list).
+ *
+ * TODO The algorithm should probably count number of Vars (not just attnums)
+ * when computing the 'score' of each solution. Computing the ratio of
+ * (num of all vars) / (num of condition vars) as a measure of how well
+ * the solution uses conditions might be useful.
+ */
+static void
+choose_mv_statistics_exhaustive(PlannerInfo *root, int step,
+ int nmvstats, MVStatisticInfo *mvstats, Bitmapset ** stats_attnums,
+ int nclauses, Node ** clauses, Bitmapset ** clauses_attnums,
+ int nconditions, Node ** conditions, Bitmapset ** conditions_attnums,
+ bool *cover_map, bool *condition_map, int *ruled_out,
+ mv_solution_t *current, mv_solution_t **best)
+{
+ int i, j;
+
+ Assert(best != NULL);
+ Assert((step == 0 && current == NULL) || (step > 0 && current != NULL));
+
+ /* this may run for a long sime, so let's make it interruptible */
+ CHECK_FOR_INTERRUPTS();
+
+ if (current == NULL)
+ {
+ current = (mv_solution_t*)palloc0(sizeof(mv_solution_t));
+ current->stats = (int*)palloc0(sizeof(int)*nmvstats);
+ current->nstats = 0;
+ current->nclauses = 0;
+ current->nconditions = 0;
+ }
+
+ /*
+ * Now try to apply each statistics, matching at least two attributes,
+ * unless it's already used in one of the previous steps.
+ */
+ for (i = 0; i < nmvstats; i++)
+ {
+ int c;
+
+ int ncovered_clauses = 0; /* number of covered clauses */
+ int ncovered_conditions = 0; /* number of covered conditions */
+ int nattnums = 0; /* number of covered attributes */
+
+ Bitmapset *all_attnums = NULL;
+
+ /* skip statistics that were already used or eliminated */
+ if (ruled_out[i] != -1)
+ continue;
+
+ /*
+ * See if we have clauses covered by this statistics, but not
+ * yet covered by any of the preceding onces.
+ */
+ for (c = 0; c < nclauses; c++)
+ {
+ bool covered = false;
+ Bitmapset *clause_attnums = clauses_attnums[c];
+ Bitmapset *tmp = NULL;
+
+ /*
+ * If this clause is not covered by this stats, we can't
+ * use the stats to estimate that at all.
+ */
+ if (! cover_map[i * nclauses + c])
+ continue;
+
+ /*
+ * Now we know we'll use this clause - either as a condition
+ * or as a new clause (the estimated one). So let's add the
+ * attributes to the attnums from all the clauses usable with
+ * this statistics.
+ */
+ tmp = bms_union(all_attnums, clause_attnums);
+
+ /* free the old bitmap */
+ bms_free(all_attnums);
+ all_attnums = tmp;
+
+ /* let's see if it's covered by any of the previous stats */
+ for (j = 0; j < step; j++)
+ {
+ /* already covered by the previous stats */
+ if (cover_map[current->stats[j] * nclauses + c])
+ covered = true;
+
+ if (covered)
+ break;
+ }
+
+ /* if already covered, continue with the next clause */
+ if (covered)
+ {
+ ncovered_conditions += 1;
+ continue;
+ }
+
+ /*
+ * OK, this clause is covered by this statistics (and not by
+ * any of the previous ones)
+ */
+ ncovered_clauses += 1;
+ }
+
+ /* can't have more new clauses than original clauses */
+ Assert(nclauses >= ncovered_clauses);
+ Assert(ncovered_clauses >= 0); /* mostly paranoia */
+
+ nattnums = bms_num_members(all_attnums);
+
+ /* free all the bitmapsets - we don't need them anymore */
+ bms_free(all_attnums);
+
+ all_attnums = NULL;
+
+ /*
+ * See if we have clauses covered by this statistics, but not
+ * yet covered by any of the preceding onces.
+ */
+ for (c = 0; c < nconditions; c++)
+ {
+ Bitmapset *clause_attnums = conditions_attnums[c];
+ Bitmapset *tmp = NULL;
+
+ /*
+ * If this clause is not covered by this stats, we can't
+ * use the stats to estimate that at all.
+ */
+ if (! condition_map[i * nconditions + c])
+ continue;
+
+ /* count this as a condition */
+ ncovered_conditions += 1;
+
+ /*
+ * Now we know we'll use this clause - either as a condition
+ * or as a new clause (the estimated one). So let's add the
+ * attributes to the attnums from all the clauses usable with
+ * this statistics.
+ */
+ tmp = bms_union(all_attnums, clause_attnums);
+
+ /* free the old bitmap */
+ bms_free(all_attnums);
+ all_attnums = tmp;
+ }
+
+ /*
+ * Let's mark the statistics as 'ruled out' - either we'll use
+ * it (and proceed to the next step), or it's incompatible.
+ */
+ ruled_out[i] = step;
+
+ /*
+ * There are no clauses usable with this statistics (not already
+ * covered by aome of the previous stats).
+ *
+ * Similarly, if the clauses only use a single attribute, we
+ * can't really use that.
+ */
+ if ((ncovered_clauses == 0) || (nattnums < 2))
+ continue;
+
+ /*
+ * TODO Not sure if it's possible to add a clause referencing
+ * only attributes already covered by previous stats?
+ * Introducing only some new dependency, not a new
+ * attribute. Couldn't come up with an example, though.
+ * Might be worth adding some assert.
+ */
+
+ /*
+ * got a suitable statistics - let's update the current solution,
+ * maybe use it as the best solution
+ */
+ current->nclauses += ncovered_clauses;
+ current->nconditions += ncovered_conditions;
+ current->nstats += 1;
+ current->stats[step] = i;
+
+ /*
+ * We can never cover more clauses, or use more stats that we
+ * actually have at the beginning.
+ */
+ Assert(nclauses >= current->nclauses);
+ Assert(nmvstats >= current->nstats);
+ Assert(step < nmvstats);
+
+ if (*best == NULL)
+ {
+ *best = (mv_solution_t*)palloc0(sizeof(mv_solution_t));
+ (*best)->stats = (int*)palloc0(sizeof(int)*nmvstats);
+ (*best)->nstats = 0;
+ (*best)->nclauses = 0;
+ (*best)->nconditions = 0;
+ }
+
+ /* see if it's better than the current 'best' solution */
+ if ((current->nclauses > (*best)->nclauses) ||
+ ((current->nclauses == (*best)->nclauses) &&
+ ((current->nstats > (*best)->nstats))))
+ {
+ (*best)->nstats = current->nstats;
+ (*best)->nclauses = current->nclauses;
+ (*best)->nconditions = current->nconditions;
+ memcpy((*best)->stats, current->stats, nmvstats * sizeof(int));
+ }
+
+ /*
+ * The recursion only makes sense if we haven't covered all the
+ * attributes (then adding stats is not really possible).
+ */
+ if ((step + 1) < nmvstats)
+ choose_mv_statistics_exhaustive(root, step+1,
+ nmvstats, mvstats, stats_attnums,
+ nclauses, clauses, clauses_attnums,
+ nconditions, conditions, conditions_attnums,
+ cover_map, condition_map, ruled_out,
+ current, best);
+
+ /* reset the last step */
+ current->nclauses -= ncovered_clauses;
+ current->nconditions -= ncovered_conditions;
+ current->nstats -= 1;
+ current->stats[step] = 0;
+
+ /* mark the statistics as usable again */
+ ruled_out[i] = -1;
+
+ Assert(current->nclauses >= 0);
+ Assert(current->nstats >= 0);
+ }
+
+ /* reset all statistics as 'incompatible' in this step */
+ for (i = 0; i < nmvstats; i++)
+ if (ruled_out[i] == step)
+ ruled_out[i] = -1;
+
+}
+
+/*
+ * Greedy search for a multivariate solution - a sequence of statistics covering
+ * the clauses. This chooses the "best" statistics at each step, so the
+ * resulting solution may not be the best solution globally, but this produces
+ * the solution in only N steps (where N is the number of statistics), while
+ * the exhaustive approach may have to walk through ~N! combinations (although
+ * some of those are terminated early).
+ *
+ * See the comments at choose_mv_statistics_exhaustive() as this does the same
+ * thing (but in a different way).
+ *
+ * Don't call this directly, but through choose_mv_statistics().
+ *
+ * TODO There are probably other metrics we might use - e.g. using number of
+ * columns (num_cond_columns / num_cov_columns), which might work better
+ * with a mix of simple and complex clauses.
+ *
+ * TODO Also the choice at the very first step should be handled in a special
+ * way, because there will be 0 conditions at that moment, so there needs
+ * to be some other criteria - e.g. using the simplest (or most complex?)
+ * clause might be a good idea.
+ *
+ * TODO We might also select multiple stats using different criteria, and branch
+ * the search. This is however tricky, because if we choose k statistics at
+ * each step, we get k^N branches to walk through (with N steps). That's
+ * not really good with large number of stats (yet better than exhaustive
+ * search).
+ */
+static void
+choose_mv_statistics_greedy(PlannerInfo *root, int step,
+ int nmvstats, MVStatisticInfo *mvstats, Bitmapset ** stats_attnums,
+ int nclauses, Node ** clauses, Bitmapset ** clauses_attnums,
+ int nconditions, Node ** conditions, Bitmapset ** conditions_attnums,
+ bool *cover_map, bool *condition_map, int *ruled_out,
+ mv_solution_t *current, mv_solution_t **best)
+{
+ int i, j;
+ int best_stat = -1;
+ double gain, max_gain = -1.0;
+
+ /*
+ * Bitmap tracking which clauses are already covered (by the previous
+ * statistics) and may thus serve only as a condition in this step.
+ */
+ bool *covered_clauses = (bool*)palloc0(nclauses);
+
+ /*
+ * Number of clauses and columns covered by each statistics - this
+ * includes both conditions and clauses covered by the statistics for
+ * the first time. The number of columns may count some columns
+ * repeatedly - if a column is shared by multiple clauses, it will
+ * be counted once for each clause (covered by the statistics).
+ * So with two clauses [(a=1 OR b=2),(a<2 OR c>1)] the column "a"
+ * will be counted twice (if both clauses are covered).
+ *
+ * The values for reduded statistics (that can't be applied) are
+ * not computed, because that'd be pointless.
+ */
+ int *num_cov_clauses = (int*)palloc0(sizeof(int) * nmvstats);
+ int *num_cov_columns = (int*)palloc0(sizeof(int) * nmvstats);
+
+ /*
+ * Same as above, but this only includes clauses that are already
+ * covered by the previous stats (and the current one).
+ */
+ int *num_cond_clauses = (int*)palloc0(sizeof(int) * nmvstats);
+ int *num_cond_columns = (int*)palloc0(sizeof(int) * nmvstats);
+
+ /*
+ * Number of attributes for each clause.
+ *
+ * TODO Might be computed in choose_mv_statistics() and then passed
+ * here, but then the function would not have the same signature
+ * as _exhaustive().
+ */
+ int *attnum_counts = (int*)palloc0(sizeof(int) * nclauses);
+ int *attnum_cond_counts = (int*)palloc0(sizeof(int) * nconditions);
+
+ CHECK_FOR_INTERRUPTS();
+
+ Assert(best != NULL);
+ Assert((step == 0 && current == NULL) || (step > 0 && current != NULL));
+
+ /* compute attributes (columns) for each clause */
+ for (i = 0; i < nclauses; i++)
+ attnum_counts[i] = bms_num_members(clauses_attnums[i]);
+
+ /* compute attributes (columns) for each condition */
+ for (i = 0; i < nconditions; i++)
+ attnum_cond_counts[i] = bms_num_members(conditions_attnums[i]);
+
+ /* see which clauses are already covered at this point (by previous stats) */
+ for (i = 0; i < step; i++)
+ for (j = 0; j < nclauses; j++)
+ covered_clauses[j] |= (cover_map[current->stats[i] * nclauses + j]);
+
+ /* which remaining statistics covers most clauses / uses most conditions? */
+ for (i = 0; i < nmvstats; i++)
+ {
+ Bitmapset *attnums_covered = NULL;
+ Bitmapset *attnums_conditions = NULL;
+
+ /* skip stats that are already ruled out (either used or inapplicable) */
+ if (ruled_out[i] != -1)
+ continue;
+
+ /* count covered clauses and conditions (for the statistics) */
+ for (j = 0; j < nclauses; j++)
+ {
+ if (cover_map[i * nclauses + j])
+ {
+ Bitmapset *attnums_new
+ = bms_union(attnums_covered, clauses_attnums[j]);
+
+ /* get rid of the old bitmap and keep the unified result */
+ bms_free(attnums_covered);
+ attnums_covered = attnums_new;
+
+ num_cov_clauses[i] += 1;
+ num_cov_columns[i] += attnum_counts[j];
+
+ /* is the clause already covered (i.e. a condition)? */
+ if (covered_clauses[j])
+ {
+ num_cond_clauses[i] += 1;
+ num_cond_columns[i] += attnum_counts[j];
+ attnums_new = bms_union(attnums_conditions,
+ clauses_attnums[j]);
+
+ bms_free(attnums_conditions);
+ attnums_conditions = attnums_new;
+ }
+ }
+ }
+
+ /* if all covered clauses are covered by prev stats (thus conditions) */
+ if (num_cov_clauses[i] == num_cond_clauses[i])
+ ruled_out[i] = step;
+
+ /* same if there are no new attributes */
+ else if (bms_num_members(attnums_conditions) == bms_num_members(attnums_covered))
+ ruled_out[i] = step;
+
+ bms_free(attnums_covered);
+ bms_free(attnums_conditions);
+
+ /* if the statistics is inapplicable, try the next one */
+ if (ruled_out[i] != -1)
+ continue;
+
+ /* now let's walk through conditions and count the covered */
+ for (j = 0; j < nconditions; j++)
+ {
+ if (condition_map[i * nconditions + j])
+ {
+ num_cond_clauses[i] += 1;
+ num_cond_columns[i] += attnum_cond_counts[j];
+ }
+ }
+
+ /* otherwise see if this improves the interesting metrics */
+ gain = num_cond_columns[i] / (double)num_cov_columns[i];
+
+ if (gain > max_gain)
+ {
+ max_gain = gain;
+ best_stat = i;
+ }
+ }
+
+ /*
+ * Have we found a suitable statistics? Add it to the solution and
+ * try next step.
+ */
+ if (best_stat != -1)
+ {
+ /* mark the statistics, so that we skip it in next steps */
+ ruled_out[best_stat] = step;
+
+ /* allocate current solution if necessary */
+ if (current == NULL)
+ {
+ current = (mv_solution_t*)palloc0(sizeof(mv_solution_t));
+ current->stats = (int*)palloc0(sizeof(int)*nmvstats);
+ current->nstats = 0;
+ current->nclauses = 0;
+ current->nconditions = 0;
+ }
+
+ current->nclauses += num_cov_clauses[best_stat];
+ current->nconditions += num_cond_clauses[best_stat];
+ current->stats[step] = best_stat;
+ current->nstats++;
+
+ if (*best == NULL)
+ {
+ (*best) = (mv_solution_t*)palloc0(sizeof(mv_solution_t));
+ (*best)->nstats = current->nstats;
+ (*best)->nclauses = current->nclauses;
+ (*best)->nconditions = current->nconditions;
+
+ (*best)->stats = (int*)palloc0(sizeof(int)*nmvstats);
+ memcpy((*best)->stats, current->stats, nmvstats * sizeof(int));
+ }
+ else
+ {
+ /* see if this is a better solution */
+ double current_gain = (double)current->nconditions / current->nclauses;
+ double best_gain = (double)(*best)->nconditions / (*best)->nclauses;
+
+ if ((current_gain > best_gain) ||
+ ((current_gain == best_gain) && (current->nstats < (*best)->nstats)))
+ {
+ (*best)->nstats = current->nstats;
+ (*best)->nclauses = current->nclauses;
+ (*best)->nconditions = current->nconditions;
+ memcpy((*best)->stats, current->stats, nmvstats * sizeof(int));
+ }
+ }
+
+ /*
+ * The recursion only makes sense if we haven't covered all the
+ * attributes (then adding stats is not really possible).
+ */
+ if ((step + 1) < nmvstats)
+ choose_mv_statistics_greedy(root, step+1,
+ nmvstats, mvstats, stats_attnums,
+ nclauses, clauses, clauses_attnums,
+ nconditions, conditions, conditions_attnums,
+ cover_map, condition_map, ruled_out,
+ current, best);
+
+ /* reset the last step */
+ current->nclauses -= num_cov_clauses[best_stat];
+ current->nconditions -= num_cond_clauses[best_stat];
+ current->nstats -= 1;
+ current->stats[step] = 0;
+
+ /* mark the statistics as usable again */
+ ruled_out[best_stat] = -1;
+ }
+
+ /* reset all statistics eliminated in this step */
+ for (i = 0; i < nmvstats; i++)
+ if (ruled_out[i] == step)
+ ruled_out[i] = -1;
+
+ /* free everything allocated in this step */
+ pfree(covered_clauses);
+ pfree(attnum_counts);
+ pfree(num_cov_clauses);
+ pfree(num_cov_columns);
+ pfree(num_cond_clauses);
+ pfree(num_cond_columns);
+}
+
+/*
+ * Remove clauses not covered by any of the available statistics
+ *
+ * This helps us to reduce the amount of work done in choose_mv_statistics()
+ * by not having to deal with clauses that can't possibly be useful.
+ */
+static List *
+filter_clauses(PlannerInfo *root, Index relid, int type,
+ List *stats, List *clauses, Bitmapset **attnums)
+{
+ ListCell *c;
+ ListCell *s;
+
+ /* results (list of compatible clauses, attnums) */
+ List *rclauses = NIL;
+
+ foreach (c, clauses)
+ {
+ Node *clause = (Node*)lfirst(c);
+ Bitmapset *clause_attnums = NULL;
+
+ /*
+ * We do assume that thanks to previous checks, we should not run into
+ * clauses that are incompatible with multivariate stats here. We also
+ * need to collect the attnums for the clause.
+ *
+ * XXX Maybe turn this into an assert?
+ */
+ if (! clause_is_mv_compatible(clause, relid, &clause_attnums, type))
+ elog(ERROR, "should not get non-mv-compatible cluase");
+
+ /* Is there a multivariate statistics covering the clause? */
+ foreach (s, stats)
+ {
+ int k, matches = 0;
+ MVStatisticInfo *stat = (MVStatisticInfo *)lfirst(s);
+
+ /* skip statistics not matching the required type */
+ if (! stats_type_matches(stat, type))
+ continue;
+
+ /*
+ * see if all clause attributes are covered by the statistic
+ *
+ * We'll do that in the opposite direction, i.e. we'll see how many
+ * attributes of the statistic are referenced in the clause, and then
+ * compare the counts.
+ */
+ for (k = 0; k < stat->stakeys->dim1; k++)
+ if (bms_is_member(stat->stakeys->values[k], clause_attnums))
+ matches += 1;
+
+ /*
+ * If the number of matches is equal to attributes referenced by the
+ * clause, then the clause is covered by the statistic.
+ */
+ if (bms_num_members(clause_attnums) == matches)
+ {
+ *attnums = bms_union(*attnums, clause_attnums);
+ rclauses = lappend(rclauses, clause);
+ break;
+ }
+ }
+
+ bms_free(clause_attnums);
+ }
+
+ /* we can't have more compatible conditions than source conditions */
+ Assert(list_length(clauses) >= list_length(rclauses));
+
+ return rclauses;
+}
+
+/*
+ * Remove statistics not covering any new clauses
+ *
+ * Statistics not covering any new clauses (conditions don't count) are not
+ * really useful, so let's ignore them. Also, we need the statistics to
+ * reference at least two different attributes (both in conditions and clauses
+ * combined), and at least one of them in the clauses alone.
+ *
+ * This check might be made more strict by checking against individual clauses,
+ * because by using the bitmapsets of all attnums we may actually use attnums
+ * from clauses that are not covered by the statistics. For example, we may
+ * have a condition
+ *
+ * (a=1 AND b=2)
+ *
+ * and a new clause
+ *
+ * (c=1 AND d=1)
+ *
+ * With only bitmapsets, statistics on [b,c] will pass through this (assuming
+ * there are some statistics covering both clases).
+ *
+ * Parameters:
+ *
+ * stats - list of statistics to filter
+ * new_attnums - attnums referenced in new clauses
+ * all_attnums - attnums referenced by contidions and new clauses combined
+ *
+ * Returns filtered list of statistics.
+ *
+ * TODO Do the more strict check, i.e. walk through individual clauses and
+ * conditions and only use those covered by the statistics.
+ */
+static List *
+filter_stats(List *stats, Bitmapset *new_attnums, Bitmapset *all_attnums)
+{
+ ListCell *s;
+ List *stats_filtered = NIL;
+
+ foreach (s, stats)
+ {
+ int k;
+ int matches_new = 0,
+ matches_all = 0;
+
+ MVStatisticInfo *stat = (MVStatisticInfo *)lfirst(s);
+
+ /* see how many attributes the statistics covers */
+ for (k = 0; k < stat->stakeys->dim1; k++)
+ {
+ /* attributes from new clauses */
+ if (bms_is_member(stat->stakeys->values[k], new_attnums))
+ matches_new += 1;
+
+ /* attributes from onditions */
+ if (bms_is_member(stat->stakeys->values[k], all_attnums))
+ matches_all += 1;
+ }
+
+ /* check we have enough attributes for this statistics */
+ if ((matches_new >= 1) && (matches_all >= 2))
+ stats_filtered = lappend(stats_filtered, stat);
+ }
+
+ /* we can't have more useful stats than we had originally */
+ Assert(list_length(stats) >= list_length(stats_filtered));
+
+ return stats_filtered;
+}
+
+static MVStatisticInfo *
+make_stats_array(List *stats, int *nmvstats)
+{
+ int i;
+ ListCell *l;
+
+ MVStatisticInfo *mvstats = NULL;
+ *nmvstats = list_length(stats);
- /* Evaluate the MCV first. */
- s1 = clauselist_mv_selectivity_mcvlist(root, clauses, mvstats,
- &fullmatch, &mcv_low);
+ mvstats
+ = (MVStatisticInfo*)palloc0((*nmvstats) * sizeof(MVStatisticInfo));
- /*
- * If we got a full equality match on the MCV list, we're done (and
- * the estimate is pretty good).
- */
- if (fullmatch && (s1 > 0.0))
- return s1;
+ i = 0;
+ foreach (l, stats)
+ {
+ MVStatisticInfo *stat = (MVStatisticInfo *)lfirst(l);
+ memcpy(&mvstats[i++], stat, sizeof(MVStatisticInfo));
+ }
- /* TODO if (fullmatch) without matching MCV item, use the mcv_low
- * selectivity as upper bound */
+ return mvstats;
+}
- s2 = clauselist_mv_selectivity_histogram(root, clauses, mvstats);
+static Bitmapset **
+make_stats_attnums(MVStatisticInfo *mvstats, int nmvstats)
+{
+ int i, j;
+ Bitmapset **stats_attnums = NULL;
- /* TODO clamp to <= 1.0 (or more strictly, when possible) */
- return s1 + s2;
+ Assert(nmvstats > 0);
+
+ /* build bitmaps of attnums for the stats (easier to compare) */
+ stats_attnums = (Bitmapset **)palloc0(nmvstats * sizeof(Bitmapset*));
+
+ for (i = 0; i < nmvstats; i++)
+ for (j = 0; j < mvstats[i].stakeys->dim1; j++)
+ stats_attnums[i]
+ = bms_add_member(stats_attnums[i],
+ mvstats[i].stakeys->values[j]);
+
+ return stats_attnums;
}
+
/*
- * Collect attributes from mv-compatible clauses.
+ * Remove redundant statistics
+ *
+ * If there are multiple statistics covering the same set of columns (counting
+ * only those referenced by clauses and conditions), we can apply one of those
+ * anyway and further reduce the size of the optimization problem.
+ *
+ * Thus when redundant stats are detected, we keep the smaller one (the one with
+ * fewer columns), based on the assumption that it's more accurate and also
+ * faster to process. That may be untrue for two reasons - first, the accuracy
+ * really depends on number of buckets/MCV items, not the number of columns.
+ * Second, some types of statistics may work better for certain types of clauses
+ * (e.g. MCV lists for equality conditions) etc.
*/
-static Bitmapset *
-collect_mv_attnums(List *clauses, Index relid, int types)
+static List*
+filter_redundant_stats(List *stats, List *clauses, List *conditions)
{
- Bitmapset *attnums = NULL;
- ListCell *l;
+ int i, j, nmvstats;
+
+ MVStatisticInfo *mvstats;
+ bool *redundant;
+ Bitmapset **stats_attnums;
+ Bitmapset *varattnos;
+ Index relid;
+
+ Assert(list_length(stats) > 0);
+ Assert(list_length(clauses) > 0);
+
+ /*
+ * We'll convert the list of statistics into an array now, because
+ * the reduction of redundant statistics is easier to do that way
+ * (we can mark previous stats as redundant, etc.).
+ */
+ mvstats = make_stats_array(stats, &nmvstats);
+ stats_attnums = make_stats_attnums(mvstats, nmvstats);
+
+ /* by default, none of the stats is redundant (so palloc0) */
+ redundant = palloc0(nmvstats * sizeof(bool));
+
+ /*
+ * We only expect a single relid here, and also we should get the
+ * same relid from clauses and conditions (but we get it from
+ * clauses, because those are certainly non-empty).
+ */
+ relid = bms_singleton_member(pull_varnos((Node*)clauses));
/*
- * Walk through the clauses and identify the ones we can estimate using
- * multivariate stats, and remember the relid/columns. We'll then
- * cross-check if we have suitable stats, and only if needed we'll split
- * the clauses into multivariate and regular lists.
+ * Get the varattnos from both conditions and clauses.
*
- * For now we're only interested in RestrictInfo nodes with nested OpExpr,
- * using either a range or equality.
+ * This skips system attributes, although that should be impossible
+ * thanks to previous filtering out of incompatible clauses.
+ *
+ * XXX Is that really true?
*/
- foreach (l, clauses)
+ varattnos = bms_union(get_varattnos((Node*)clauses, relid),
+ get_varattnos((Node*)conditions, relid));
+
+ for (i = 1; i < nmvstats; i++)
{
- Node *clause = (Node *) lfirst(l);
+ /* intersect with current statistics */
+ Bitmapset *curr = bms_intersect(stats_attnums[i], varattnos);
- /* ignore the result here - we only need the attnums */
- clause_is_mv_compatible(clause, relid, &attnums, types);
+ /* walk through 'previous' stats and check redundancy */
+ for (j = 0; j < i; j++)
+ {
+ /* intersect with current statistics */
+ Bitmapset *prev;
+
+ /* skip stats already identified as redundant */
+ if (redundant[j])
+ continue;
+
+ prev = bms_intersect(stats_attnums[j], varattnos);
+
+ switch (bms_subset_compare(curr, prev))
+ {
+ case BMS_EQUAL:
+ /*
+ * Use the smaller one (hopefully more accurate).
+ * If both have the same size, use the first one.
+ */
+ if (mvstats[i].stakeys->dim1 >= mvstats[j].stakeys->dim1)
+ redundant[i] = TRUE;
+ else
+ redundant[j] = TRUE;
+
+ break;
+
+ case BMS_SUBSET1: /* curr is subset of prev */
+ redundant[i] = TRUE;
+ break;
+
+ case BMS_SUBSET2: /* prev is subset of curr */
+ redundant[j] = TRUE;
+ break;
+
+ case BMS_DIFFERENT:
+ /* do nothing - keep both stats */
+ break;
+ }
+
+ bms_free(prev);
+ }
+
+ bms_free(curr);
}
- /*
- * If there are not at least two attributes referenced by the clause(s),
- * we can throw everything out (as we'll revert to simple stats).
- */
- if (bms_num_members(attnums) <= 1)
+ /* can't reduce all statistics (at least one has to remain) */
+ Assert(nmvstats > 0);
+
+ /* now, let's remove the reduced statistics from the arrays */
+ list_free(stats);
+ stats = NIL;
+
+ for (i = 0; i < nmvstats; i++)
{
- if (attnums != NULL)
- pfree(attnums);
- attnums = NULL;
+ MVStatisticInfo *info;
+
+ pfree(stats_attnums[i]);
+
+ if (redundant[i])
+ continue;
+
+ info = makeNode(MVStatisticInfo);
+ memcpy(info, &mvstats[i], sizeof(MVStatisticInfo));
+
+ stats = lappend(stats, info);
}
- return attnums;
+ pfree(mvstats);
+ pfree(stats_attnums);
+ pfree(redundant);
+
+ return stats;
}
-/*
- * Count the number of attributes in clauses compatible with multivariate stats.
- */
-static int
-count_mv_attnums(List *clauses, Index relid, int type)
+static Node**
+make_clauses_array(List *clauses, int *nclauses)
{
- int c;
- Bitmapset *attnums = collect_mv_attnums(clauses, relid, type);
+ int i;
+ ListCell *l;
- c = bms_num_members(attnums);
+ Node** clauses_array;
- bms_free(attnums);
+ *nclauses = list_length(clauses);
+ clauses_array = (Node **)palloc0((*nclauses) * sizeof(Node *));
- return c;
+ i = 0;
+ foreach (l, clauses)
+ clauses_array[i++] = (Node *)lfirst(l);
+
+ *nclauses = i;
+
+ return clauses_array;
}
-/*
- * Count varnos referenced in the clauses, and if there's a single varno then
- * return the index in 'relid'.
- */
-static int
-count_varnos(List *clauses, Index *relid)
+static Bitmapset **
+make_clauses_attnums(PlannerInfo *root, Index relid,
+ int type, Node **clauses, int nclauses)
{
- int cnt;
- Bitmapset *varnos = NULL;
+ int i;
+ Bitmapset **clauses_attnums
+ = (Bitmapset **)palloc0(nclauses * sizeof(Bitmapset *));
- varnos = pull_varnos((Node *) clauses);
- cnt = bms_num_members(varnos);
+ for (i = 0; i < nclauses; i++)
+ {
+ Bitmapset * attnums = NULL;
- /* if there's a single varno in the clauses, remember it */
- if (bms_num_members(varnos) == 1)
- *relid = bms_singleton_member(varnos);
+ if (! clause_is_mv_compatible(clauses[i], relid, &attnums, type))
+ elog(ERROR, "should not get non-mv-compatible clause");
- bms_free(varnos);
+ clauses_attnums[i] = attnums;
+ }
- return cnt;
+ return clauses_attnums;
}
-
+
+static bool*
+make_cover_map(Bitmapset **stats_attnums, int nmvstats,
+ Bitmapset **clauses_attnums, int nclauses)
+{
+ int i, j;
+ bool *cover_map = (bool*)palloc0(nclauses * nmvstats);
+
+ for (i = 0; i < nmvstats; i++)
+ for (j = 0; j < nclauses; j++)
+ cover_map[i * nclauses + j]
+ = bms_is_subset(clauses_attnums[j], stats_attnums[i]);
+
+ return cover_map;
+}
+
/*
- * We're looking for statistics matching at least 2 attributes, referenced in
- * clauses compatible with multivariate statistics. The current selection
- * criteria is very simple - we choose the statistics referencing the most
- * attributes.
- *
- * If there are multiple statistics referencing the same number of columns
- * (from the clauses), the one with less source columns (as listed in the
- * ADD STATISTICS when creating the statistics) wins. Else the first one wins.
- *
- * This is a very simple criteria, and has several weaknesses:
- *
- * (a) does not consider the accuracy of the statistics
- *
- * If there are two histograms built on the same set of columns, but one
- * has 100 buckets and the other one has 1000 buckets (thus likely
- * providing better estimates), this is not currently considered.
- *
- * (b) does not consider the type of statistics
- *
- * If there are three statistics - one containing just a MCV list, another
- * one with just a histogram and a third one with both, we treat them equally.
+ * Chooses the combination of statistics, optimal for estimation of a particular
+ * clause list.
*
- * (c) does not consider the number of clauses
+ * This only handles a 'preparation' shared by the exhaustive and greedy
+ * implementations (see the previous methods), mostly trying to reduce the size
+ * of the problem (eliminate clauses/statistics that can't be really used in
+ * the solution).
*
- * As explained, only the number of referenced attributes counts, so if
- * there are multiple clauses on a single attribute, this still counts as
- * a single attribute.
+ * It also precomputes bitmaps for attributes covered by clauses and statistics,
+ * so that we don't need to do that over and over in the actual optimizations
+ * (as it's both CPU and memory intensive).
*
- * (d) does not consider type of condition
*
- * Some clauses may work better with some statistics - for example equality
- * clauses probably work better with MCV lists than with histograms. But
- * IS [NOT] NULL conditions may often work better with histograms (thanks
- * to NULL-buckets).
+ * TODO Another way to make the optimization problems smaller might be splitting
+ * the statistics into several disjoint subsets, i.e. if we can split the
+ * graph of statistics (after the elimination) into multiple components
+ * (so that stats in different components share no attributes), we can do
+ * the optimization for each component separately.
*
- * So for example with five WHERE conditions
- *
- * WHERE (a = 1) AND (b = 1) AND (c = 1) AND (d = 1) AND (e = 1)
- *
- * and statistics on (a,b), (a,b,e) and (a,b,c,d), the last one will be selected
- * as it references the most columns.
- *
- * Once we have selected the multivariate statistics, we split the list of
- * clauses into two parts - conditions that are compatible with the selected
- * stats, and conditions are estimated using simple statistics.
- *
- * From the example above, conditions
- *
- * (a = 1) AND (b = 1) AND (c = 1) AND (d = 1)
- *
- * will be estimated using the multivariate statistics (a,b,c,d) while the last
- * condition (e = 1) will get estimated using the regular ones.
- *
- * There are various alternative selection criteria (e.g. counting conditions
- * instead of just referenced attributes), but eventually the best option should
- * be to combine multiple statistics. But that's much harder to do correctly.
- *
- * TODO Select multiple statistics and combine them when computing the estimate.
- *
- * TODO This will probably have to consider compatibility of clauses, because
- * 'dependencies' will probably work only with equality clauses.
+ * TODO If we could compute what is a "perfect solution" maybe we could
+ * terminate the search after reaching ~90% of it? Say, if we knew that we
+ * can cover 10 clauses and reuse 8 dependencies, maybe covering 9 clauses
+ * and 7 dependencies would be OK?
*/
-static MVStatisticInfo *
-choose_mv_statistics(List *stats, Bitmapset *attnums)
+static List*
+choose_mv_statistics(PlannerInfo *root, Index relid, List *stats,
+ List *clauses, List *conditions)
{
int i;
- ListCell *lc;
+ mv_solution_t *best = NULL;
+ List *result = NIL;
+
+ int nmvstats;
+ MVStatisticInfo *mvstats;
+
+ /* we only work with MCV lists and histograms here */
+ int type = (MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST);
+
+ bool *clause_cover_map = NULL,
+ *condition_cover_map = NULL;
+ int *ruled_out = NULL;
+
+ /* build bitmapsets for all stats and clauses */
+ Bitmapset **stats_attnums;
+ Bitmapset **clauses_attnums;
+ Bitmapset **conditions_attnums;
- MVStatisticInfo *choice = NULL;
+ int nclauses, nconditions;
+ Node ** clauses_array;
+ Node ** conditions_array;
- int current_matches = 1; /* goal #1: maximize */
- int current_dims = (MVSTATS_MAX_DIMENSIONS+1); /* goal #2: minimize */
+ /* copy lists, so that we can free them during elimination easily */
+ clauses = list_copy(clauses);
+ conditions = list_copy(conditions);
+ stats = list_copy(stats);
/*
- * Walk through the statistics (simple array with nmvstats elements) and for
- * each one count the referenced attributes (encoded in the 'attnums' bitmap).
+ * Reduce the optimization problem size as much as possible.
+ *
+ * Eliminate clauses and conditions not covered by any statistics,
+ * or statistics not matching at least two attributes (one of them
+ * has to be in a regular clause).
+ *
+ * It's possible that removing a statistics in one iteration
+ * eliminates clause in the next one, so we'll repeat this until we
+ * eliminate no clauses/stats in that iteration.
+ *
+ * This can only happen after eliminating a statistics - clauses are
+ * eliminated first, so statistics always reflect that.
*/
- foreach (lc, stats)
+ while (true)
{
- MVStatisticInfo *info = (MVStatisticInfo *)lfirst(lc);
-
- /* columns matching this statistics */
- int matches = 0;
+ List *tmp;
- int2vector * attrs = info->stakeys;
- int numattrs = attrs->dim1;
+ Bitmapset *compatible_attnums = NULL;
+ Bitmapset *condition_attnums = NULL;
+ Bitmapset *all_attnums = NULL;
- /* skip dependencies-only stats */
- if (! (info->mcv_built || info->hist_built))
- continue;
+ /*
+ * Clauses
+ *
+ * Walk through clauses and keep only those covered by at least
+ * one of the statistics we still have. We'll also keep info
+ * about attnums in clauses (without conditions) so that we can
+ * ignore stats covering just conditions (which is pointless).
+ */
+ tmp = filter_clauses(root, relid, type,
+ stats, clauses, &compatible_attnums);
- /* count columns covered by the histogram */
- for (i = 0; i < numattrs; i++)
- if (bms_is_member(attrs->values[i], attnums))
- matches++;
+ /* discard the original list */
+ list_free(clauses);
+ clauses = tmp;
/*
- * Use this statistics when it improves the number of matches or
- * when it matches the same number of attributes but is smaller.
+ * Conditions
+ *
+ * Walk through clauses and keep only those covered by at least
+ * one of the statistics we still have. Also, collect bitmap of
+ * attributes so that we can make sure we add at least one new
+ * attribute (by comparing with clauses).
*/
- if ((matches > current_matches) ||
- ((matches == current_matches) && (current_dims > numattrs)))
+ if (conditions != NIL)
{
- choice = info;
- current_matches = matches;
- current_dims = numattrs;
+ tmp = filter_clauses(root, relid, type,
+ stats, conditions, &condition_attnums);
+
+ /* discard the original list */
+ list_free(conditions);
+ conditions = tmp;
}
- }
- return choice;
-}
+ /* get a union of attnums (from conditions and new clauses) */
+ all_attnums = bms_union(compatible_attnums, condition_attnums);
+
+ /*
+ * Statisitics
+ *
+ * Walk through statistics and only keep those covering at least
+ * one new attribute (excluding conditions) and at two attributes
+ * in both clauses and conditions.
+ */
+ tmp = filter_stats(stats, compatible_attnums, all_attnums);
+ /* if we've not eliminated anything, terminate */
+ if (list_length(stats) == list_length(tmp))
+ break;
-/*
- * This splits the clauses list into two parts - one containing clauses that
- * will be evaluated using the chosen statistics, and the remaining clauses
- * (either non-mvcompatible, or not related to the histogram).
- */
-static List *
-clauselist_mv_split(PlannerInfo *root, Index relid,
- List *clauses, List **mvclauses,
- MVStatisticInfo *mvstats, int types)
-{
- int i;
- ListCell *l;
- List *non_mvclauses = NIL;
+ /* work only with filtered statistics from now */
+ list_free(stats);
+ stats = tmp;
+ }
- /* FIXME is there a better way to get info on int2vector? */
- int2vector * attrs = mvstats->stakeys;
- int numattrs = mvstats->stakeys->dim1;
+ /* only do the optimization if we have clauses/statistics */
+ if ((list_length(stats) == 0) || (list_length(clauses) == 0))
+ return NULL;
- Bitmapset *mvattnums = NULL;
+ /* remove redundant stats (stats covered by another stats) */
+ stats = filter_redundant_stats(stats, clauses, conditions);
- /* build bitmap of attributes, so we can do bms_is_subset later */
- for (i = 0; i < numattrs; i++)
- mvattnums = bms_add_member(mvattnums, attrs->values[i]);
+ /*
+ * TODO We should sort the stats to make the order deterministic,
+ * otherwise we may get different estimates on different
+ * executions - if there are multiple "equally good" solutions,
+ * we'll keep the first solution we see.
+ *
+ * Sorting by OID probably is not the right solution though,
+ * because we'd like it to be somehow reproducible,
+ * irrespectedly of the order of ADD STATISTICS commands.
+ * So maybe statkeys?
+ */
+ mvstats = make_stats_array(stats, &nmvstats);
+ stats_attnums = make_stats_attnums(mvstats, nmvstats);
- /* erase the list of mv-compatible clauses */
- *mvclauses = NIL;
+ /* collect clauses an bitmap of attnums */
+ clauses_array = make_clauses_array(clauses, &nclauses);
+ clauses_attnums = make_clauses_attnums(root, relid, type,
+ clauses_array, nclauses);
- foreach (l, clauses)
- {
- bool match = false; /* by default not mv-compatible */
- Bitmapset *attnums = NULL;
- Node *clause = (Node *) lfirst(l);
+ /* collect conditions and bitmap of attnums */
+ conditions_array = make_clauses_array(conditions, &nconditions);
+ conditions_attnums = make_clauses_attnums(root, relid, type,
+ conditions_array, nconditions);
- if (clause_is_mv_compatible(clause, relid, &attnums, types))
+ /*
+ * Build bitmaps with info about which clauses/conditions are
+ * covered by each statistics (so that we don't need to call the
+ * bms_is_subset over and over again).
+ */
+ clause_cover_map = make_cover_map(stats_attnums, nmvstats,
+ clauses_attnums, nclauses);
+
+ condition_cover_map = make_cover_map(stats_attnums, nmvstats,
+ conditions_attnums, nconditions);
+
+ ruled_out = (int*)palloc0(nmvstats * sizeof(int));
+
+ /* no stats are ruled out by default */
+ for (i = 0; i < nmvstats; i++)
+ ruled_out[i] = -1;
+
+ /* do the optimization itself */
+ if (mvstat_search_type == MVSTAT_SEARCH_EXHAUSTIVE)
+ choose_mv_statistics_exhaustive(root, 0,
+ nmvstats, mvstats, stats_attnums,
+ nclauses, clauses_array, clauses_attnums,
+ nconditions, conditions_array, conditions_attnums,
+ clause_cover_map, condition_cover_map,
+ ruled_out, NULL, &best);
+ else
+ choose_mv_statistics_greedy(root, 0,
+ nmvstats, mvstats, stats_attnums,
+ nclauses, clauses_array, clauses_attnums,
+ nconditions, conditions_array, conditions_attnums,
+ clause_cover_map, condition_cover_map,
+ ruled_out, NULL, &best);
+
+ /* create a list of statistics from the array */
+ if (best != NULL)
+ {
+ for (i = 0; i < best->nstats; i++)
{
- /* are all the attributes part of the selected stats? */
- if (bms_is_subset(attnums, mvattnums))
- match = true;
+ MVStatisticInfo *info = makeNode(MVStatisticInfo);
+ memcpy(info, &mvstats[best->stats[i]], sizeof(MVStatisticInfo));
+ result = lappend(result, info);
}
- /*
- * The clause matches the selected stats, so put it to the list of
- * mv-compatible clauses. Otherwise, keep it in the list of 'regular'
- * clauses (that may be selected later).
- */
- if (match)
- *mvclauses = lappend(*mvclauses, clause);
- else
- non_mvclauses = lappend(non_mvclauses, clause);
+ pfree(best);
}
- /*
- * Perform regular estimation using the clauses incompatible with the chosen
- * histogram (or MV stats in general).
- */
- return non_mvclauses;
+ /* cleanup (maybe leave it up to the memory context?) */
+ for (i = 0; i < nmvstats; i++)
+ bms_free(stats_attnums[i]);
+ for (i = 0; i < nclauses; i++)
+ bms_free(clauses_attnums[i]);
+
+ for (i = 0; i < nconditions; i++)
+ bms_free(conditions_attnums[i]);
+
+ pfree(stats_attnums);
+ pfree(clauses_attnums);
+ pfree(conditions_attnums);
+
+ pfree(clauses_array);
+ pfree(conditions_array);
+ pfree(clause_cover_map);
+ pfree(condition_cover_map);
+ pfree(ruled_out);
+ pfree(mvstats);
+
+ list_free(clauses);
+ list_free(conditions);
+ list_free(stats);
+
+ return result;
}
typedef struct
@@ -1637,9 +2878,6 @@ has_stats(List *stats, int type)
/* terminate if we've found at least one matching statistics */
if (stats_type_matches(stat, type))
return true;
-
- if ((type & MV_CLAUSE_TYPE_HIST) && stat->hist_built)
- return true;
}
return false;
@@ -1689,22 +2927,26 @@ find_stats(PlannerInfo *root, Index relid)
* as the clauses are processed (and skip items that are 'match').
*/
static Selectivity
-clauselist_mv_selectivity_mcvlist(PlannerInfo *root, List *clauses,
- MVStatisticInfo *mvstats, bool *fullmatch,
- Selectivity *lowsel)
+clauselist_mv_selectivity_mcvlist(PlannerInfo *root, MVStatisticInfo *mvstats,
+ List *clauses, List *conditions, bool is_or,
+ bool *fullmatch, Selectivity *lowsel)
{
int i;
Selectivity s = 0.0;
+ Selectivity t = 0.0;
Selectivity u = 0.0;
MCVList mcvlist = NULL;
+
int nmatches = 0;
+ int nconditions = 0;
/* match/mismatch bitmap for each MCV item */
char * matches = NULL;
+ char * condition_matches = NULL;
Assert(clauses != NIL);
- Assert(list_length(clauses) >= 2);
+ Assert(list_length(clauses) >= 1);
/* there's no MCV list built yet */
if (! mvstats->mcv_built)
@@ -1715,32 +2957,85 @@ clauselist_mv_selectivity_mcvlist(PlannerInfo *root, List *clauses,
Assert(mcvlist != NULL);
Assert(mcvlist->nitems > 0);
- /* by default all the MCV items match the clauses fully */
- matches = palloc0(sizeof(char) * mcvlist->nitems);
- memset(matches, MVSTATS_MATCH_FULL, sizeof(char)*mcvlist->nitems);
-
/* number of matching MCV items */
nmatches = mcvlist->nitems;
+ nconditions = mcvlist->nitems;
+
+ /*
+ * Bitmap of bucket matches (mismatch, partial, full).
+ *
+ * For AND clauses all buckets match (and we'll eliminate them).
+ * For OR clauses no buckets match (and we'll add them).
+ *
+ * We only need to do the memset for AND clauses (for OR clauses
+ * it's already set correctly by the palloc0).
+ */
+ matches = palloc0(sizeof(char) * nmatches);
+
+ if (! is_or) /* AND-clause */
+ memset(matches, MVSTATS_MATCH_FULL, sizeof(char)*nmatches);
+ /* Conditions are treated as AND clause, so match by default. */
+ condition_matches = palloc0(sizeof(char) * nconditions);
+ memset(condition_matches, MVSTATS_MATCH_FULL, sizeof(char)*nconditions);
+
+ /*
+ * build the match bitmap for the conditions (conditions are always
+ * connected by AND)
+ */
+ if (conditions != NIL)
+ nconditions = update_match_bitmap_mcvlist(root, conditions,
+ mvstats->stakeys, mcvlist,
+ nconditions, condition_matches,
+ lowsel, fullmatch, false);
+
+ /*
+ * build the match bitmap for the estimated clauses
+ *
+ * TODO This evaluates the clauses for all MCV items, even those
+ * ruled out by the conditions. The final result should be the
+ * same, but it might be faster.
+ */
nmatches = update_match_bitmap_mcvlist(root, clauses,
mvstats->stakeys, mcvlist,
- nmatches, matches,
- lowsel, fullmatch, false);
+ ((is_or) ? 0 : nmatches), matches,
+ lowsel, fullmatch, is_or);
/* sum frequencies for all the matching MCV items */
for (i = 0; i < mcvlist->nitems; i++)
{
- /* used to 'scale' for MCV lists not covering all tuples */
+ /*
+ * Find out what part of the data is covered by the MCV list,
+ * so that we can 'scale' the selectivity properly (e.g. when
+ * only 50% of the sample items got into the MCV, and the rest
+ * is either in a histogram, or not covered by stats).
+ *
+ * TODO This might be handled by keeping a global "frequency"
+ * for the whole list, which might save us a bit of time
+ * spent on accessing the not-matching part of the MCV list.
+ * Although it's likely in a cache, so it's very fast.
+ */
u += mcvlist->items[i]->frequency;
+ /* skit MCV items not matching the conditions */
+ if (condition_matches[i] == MVSTATS_MATCH_NONE)
+ continue;
+
if (matches[i] != MVSTATS_MATCH_NONE)
s += mcvlist->items[i]->frequency;
+
+ t += mcvlist->items[i]->frequency;
}
pfree(matches);
+ pfree(condition_matches);
pfree(mcvlist);
- return s*u;
+ /* no condition matches */
+ if (t == 0.0)
+ return (Selectivity)0.0;
+
+ return (s / t) * u;
}
/*
@@ -1971,64 +3266,57 @@ update_match_bitmap_mcvlist(PlannerInfo *root, List *clauses,
}
}
}
- else if (or_clause(clause) || and_clause(clause))
+ else if (or_clause(clause) || and_clause(clause) || not_clause(clause))
{
/* AND/OR clause, with all clauses compatible with the selected MV stat */
int i;
- BoolExpr *orclause = ((BoolExpr*)clause);
- List *orclauses = orclause->args;
+ List *tmp_clauses = ((BoolExpr*)clause)->args;
/* match/mismatch bitmap for each MCV item */
- int or_nmatches = 0;
- char * or_matches = NULL;
+ int tmp_nmatches = 0;
+ char * tmp_matches = NULL;
- Assert(orclauses != NIL);
- Assert(list_length(orclauses) >= 2);
+ Assert(tmp_clauses != NIL);
+ Assert((list_length(tmp_clauses) >= 2) || (not_clause(clause) && (list_length(tmp_clauses)==1)));
/* number of matching MCV items */
- or_nmatches = mcvlist->nitems;
+ tmp_nmatches = (or_clause(clause)) ? 0 : mcvlist->nitems;
/* by default none of the MCV items matches the clauses */
- or_matches = palloc0(sizeof(char) * or_nmatches);
+ tmp_matches = palloc0(sizeof(char) * mcvlist->nitems);
- if (or_clause(clause))
- {
- /* OR clauses assume nothing matches, initially */
- memset(or_matches, MVSTATS_MATCH_NONE, sizeof(char)*or_nmatches);
- or_nmatches = 0;
- }
- else
- {
- /* AND clauses assume nothing matches, initially */
- memset(or_matches, MVSTATS_MATCH_FULL, sizeof(char)*or_nmatches);
- }
+ /* AND (and NOT) clauses assume everything matches, initially */
+ if (! or_clause(clause))
+ memset(tmp_matches, MVSTATS_MATCH_FULL, sizeof(char)*mcvlist->nitems);
/* build the match bitmap for the OR-clauses */
- or_nmatches = update_match_bitmap_mcvlist(root, orclauses,
+ tmp_nmatches = update_match_bitmap_mcvlist(root, tmp_clauses,
stakeys, mcvlist,
- or_nmatches, or_matches,
+ tmp_nmatches, tmp_matches,
lowsel, fullmatch, or_clause(clause));
/* merge the bitmap into the existing one*/
for (i = 0; i < mcvlist->nitems; i++)
{
+ /* if this is a NOT clause, we need to invert the results first */
+ if (not_clause(clause))
+ tmp_matches[i] = (MVSTATS_MATCH_FULL - tmp_matches[i]);
+
/*
* To AND-merge the bitmaps, a MIN() semantics is used.
* For OR-merge, use MAX().
*
* FIXME this does not decrease the number of matches
*/
- UPDATE_RESULT(matches[i], or_matches[i], is_or);
+ UPDATE_RESULT(matches[i], tmp_matches[i], is_or);
}
- pfree(or_matches);
+ pfree(tmp_matches);
}
else
- {
elog(ERROR, "unknown clause type: %d", clause->type);
- }
}
/*
@@ -2086,15 +3374,18 @@ update_match_bitmap_mcvlist(PlannerInfo *root, List *clauses,
* this is not uncommon, but for histograms it's not that clear.
*/
static Selectivity
-clauselist_mv_selectivity_histogram(PlannerInfo *root, List *clauses,
- MVStatisticInfo *mvstats)
+clauselist_mv_selectivity_histogram(PlannerInfo *root, MVStatisticInfo *mvstats,
+ List *clauses, List *conditions, bool is_or)
{
int i;
Selectivity s = 0.0;
+ Selectivity t = 0.0;
Selectivity u = 0.0;
int nmatches = 0;
+ int nconditions = 0;
char *matches = NULL;
+ char *condition_matches = NULL;
MVSerializedHistogram mvhist = NULL;
@@ -2107,25 +3398,55 @@ clauselist_mv_selectivity_histogram(PlannerInfo *root, List *clauses,
Assert (mvhist != NULL);
Assert (clauses != NIL);
- Assert (list_length(clauses) >= 2);
+ Assert (list_length(clauses) >= 1);
+
+ nmatches = mvhist->nbuckets;
+ nconditions = mvhist->nbuckets;
/*
- * Bitmap of bucket matches (mismatch, partial, full). by default
- * all buckets fully match (and we'll eliminate them).
+ * Bitmap of bucket matches (mismatch, partial, full).
+ *
+ * For AND clauses all buckets match (and we'll eliminate them).
+ * For OR clauses no buckets match (and we'll add them).
+ *
+ * We only need to do the memset for AND clauses (for OR clauses
+ * it's already set correctly by the palloc0).
*/
- matches = palloc0(sizeof(char) * mvhist->nbuckets);
- memset(matches, MVSTATS_MATCH_FULL, sizeof(char)*mvhist->nbuckets);
+ matches = palloc0(sizeof(char) * nmatches);
- nmatches = mvhist->nbuckets;
+ if (! is_or) /* AND-clause */
+ memset(matches, MVSTATS_MATCH_FULL, sizeof(char)*nmatches);
+
+ /* Conditions are treated as AND clause, so match by default. */
+ condition_matches = palloc0(sizeof(char)*nconditions);
+ memset(condition_matches, MVSTATS_MATCH_FULL, sizeof(char)*nconditions);
+
+ /*
+ * build the match bitmap for the conditions (conditions are always
+ * connected by AND)
+ */
+ if (conditions != NIL)
+ update_match_bitmap_histogram(root, conditions,
+ mvstats->stakeys, mvhist,
+ nconditions, condition_matches, false);
- /* build the match bitmap */
+ /*
+ * build the match bitmap for the estimated clauses
+ *
+ * TODO This evaluates the clauses for all buckets, even those
+ * ruled out by the conditions. The final result should be
+ * the same, but it might be faster.
+ */
update_match_bitmap_histogram(root, clauses,
mvstats->stakeys, mvhist,
- nmatches, matches, false);
+ ((is_or) ? 0 : nmatches), matches,
+ is_or);
/* now, walk through the buckets and sum the selectivities */
for (i = 0; i < mvhist->nbuckets; i++)
{
+ float coeff = 1.0;
+
/*
* Find out what part of the data is covered by the histogram,
* so that we can 'scale' the selectivity properly (e.g. when
@@ -2139,10 +3460,23 @@ clauselist_mv_selectivity_histogram(PlannerInfo *root, List *clauses,
*/
u += mvhist->buckets[i]->ntuples;
+ /* skip buckets not matching the conditions */
+ if (condition_matches[i] == MVSTATS_MATCH_NONE)
+ continue;
+ else if (condition_matches[i] == MVSTATS_MATCH_PARTIAL)
+ coeff = 0.5;
+
+ t += coeff * mvhist->buckets[i]->ntuples;
+
if (matches[i] == MVSTATS_MATCH_FULL)
- s += mvhist->buckets[i]->ntuples;
+ s += coeff * mvhist->buckets[i]->ntuples;
else if (matches[i] == MVSTATS_MATCH_PARTIAL)
- s += 0.5 * mvhist->buckets[i]->ntuples;
+ /*
+ * TODO If both conditions and clauses match partially, this
+ * will use 0.25 match - not sure if that's the right
+ * thing solution, but seems about right.
+ */
+ s += coeff * 0.5 * mvhist->buckets[i]->ntuples;
}
#ifdef DEBUG_MVHIST
@@ -2151,9 +3485,14 @@ clauselist_mv_selectivity_histogram(PlannerInfo *root, List *clauses,
/* release the allocated bitmap and deserialized histogram */
pfree(matches);
+ pfree(condition_matches);
pfree(mvhist);
- return s * u;
+ /* no condition matches */
+ if (t == 0.0)
+ return (Selectivity)0.0;
+
+ return (s / t) * u;
}
/* cached result of bucket boundary comparison for a single dimension */
@@ -2301,7 +3640,7 @@ update_match_bitmap_histogram(PlannerInfo *root, List *clauses,
{
int i;
ListCell * l;
-
+
/*
* Used for caching function calls, only once per deduplicated value.
*
@@ -2344,7 +3683,7 @@ update_match_bitmap_histogram(PlannerInfo *root, List *clauses,
FmgrInfo opproc; /* operator */
fmgr_info(get_opcode(expr->opno), &opproc);
-
+
/* reset the cache (per clause) */
memset(callcache, 0, mvhist->nbuckets);
@@ -2504,64 +3843,57 @@ update_match_bitmap_histogram(PlannerInfo *root, List *clauses,
UPDATE_RESULT(matches[i], MVSTATS_MATCH_NONE, is_or);
}
}
- else if (or_clause(clause) || and_clause(clause))
+ else if (or_clause(clause) || and_clause(clause) || not_clause(clause))
{
/* AND/OR clause, with all clauses compatible with the selected MV stat */
int i;
- BoolExpr *orclause = ((BoolExpr*)clause);
- List *orclauses = orclause->args;
+ List *tmp_clauses = ((BoolExpr*)clause)->args;
/* match/mismatch bitmap for each bucket */
- int or_nmatches = 0;
- char * or_matches = NULL;
+ int tmp_nmatches = 0;
+ char * tmp_matches = NULL;
- Assert(orclauses != NIL);
- Assert(list_length(orclauses) >= 2);
+ Assert(tmp_clauses != NIL);
+ Assert((list_length(tmp_clauses) >= 2) || (not_clause(clause) && (list_length(tmp_clauses)==1)));
/* number of matching buckets */
- or_nmatches = mvhist->nbuckets;
+ tmp_nmatches = (or_clause(clause)) ? 0 : mvhist->nbuckets;
- /* by default none of the buckets matches the clauses */
- or_matches = palloc0(sizeof(char) * or_nmatches);
+ /* by default none of the buckets matches the clauses (OR clause) */
+ tmp_matches = palloc0(sizeof(char) * mvhist->nbuckets);
- if (or_clause(clause))
- {
- /* OR clauses assume nothing matches, initially */
- memset(or_matches, MVSTATS_MATCH_NONE, sizeof(char)*or_nmatches);
- or_nmatches = 0;
- }
- else
- {
- /* AND clauses assume nothing matches, initially */
- memset(or_matches, MVSTATS_MATCH_FULL, sizeof(char)*or_nmatches);
- }
+ /* but AND (and NOT) clauses assume everything matches, initially */
+ if (! or_clause(clause))
+ memset(tmp_matches, MVSTATS_MATCH_FULL, sizeof(char)*mvhist->nbuckets);
/* build the match bitmap for the OR-clauses */
- or_nmatches = update_match_bitmap_histogram(root, orclauses,
+ tmp_nmatches = update_match_bitmap_histogram(root, tmp_clauses,
stakeys, mvhist,
- or_nmatches, or_matches, or_clause(clause));
+ tmp_nmatches, tmp_matches, or_clause(clause));
/* merge the bitmap into the existing one*/
for (i = 0; i < mvhist->nbuckets; i++)
{
+ /* if this is a NOT clause, we need to invert the results first */
+ if (not_clause(clause))
+ tmp_matches[i] = (MVSTATS_MATCH_FULL - tmp_matches[i]);
+
/*
* To AND-merge the bitmaps, a MIN() semantics is used.
* For OR-merge, use MAX().
*
* FIXME this does not decrease the number of matches
*/
- UPDATE_RESULT(matches[i], or_matches[i], is_or);
+ UPDATE_RESULT(matches[i], tmp_matches[i], is_or);
}
- pfree(or_matches);
-
+ pfree(tmp_matches);
}
else
elog(ERROR, "unknown clause type: %d", clause->type);
}
- /* free the call cache */
pfree(callcache);
return nmatches;
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index 5350329..57214e0 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -3518,7 +3518,8 @@ compute_semi_anti_join_factors(PlannerInfo *root,
joinquals,
0,
jointype,
- sjinfo);
+ sjinfo,
+ NIL);
/*
* Also get the normal inner-join selectivity of the join clauses.
@@ -3541,7 +3542,8 @@ compute_semi_anti_join_factors(PlannerInfo *root,
joinquals,
0,
JOIN_INNER,
- &norm_sjinfo);
+ &norm_sjinfo,
+ NIL);
/* Avoid leaking a lot of ListCells */
if (jointype == JOIN_ANTI)
@@ -3708,7 +3710,7 @@ approx_tuple_count(PlannerInfo *root, JoinPath *path, List *quals)
Node *qual = (Node *) lfirst(l);
/* Note that clause_selectivity will be able to cache its result */
- selec *= clause_selectivity(root, qual, 0, JOIN_INNER, &sjinfo);
+ selec *= clause_selectivity(root, qual, 0, JOIN_INNER, &sjinfo, NIL);
}
/* Apply it to the input relation sizes */
@@ -3744,7 +3746,8 @@ set_baserel_size_estimates(PlannerInfo *root, RelOptInfo *rel)
rel->baserestrictinfo,
0,
JOIN_INNER,
- NULL);
+ NULL,
+ NIL);
rel->rows = clamp_row_est(nrows);
@@ -3781,7 +3784,8 @@ get_parameterized_baserel_size(PlannerInfo *root, RelOptInfo *rel,
allclauses,
rel->relid, /* do not use 0! */
JOIN_INNER,
- NULL);
+ NULL,
+ NIL);
nrows = clamp_row_est(nrows);
/* For safety, make sure result is not more than the base estimate */
if (nrows > rel->rows)
@@ -3919,12 +3923,14 @@ calc_joinrel_size_estimate(PlannerInfo *root,
joinquals,
0,
jointype,
- sjinfo);
+ sjinfo,
+ NIL);
pselec = clauselist_selectivity(root,
pushedquals,
0,
jointype,
- sjinfo);
+ sjinfo,
+ NIL);
/* Avoid leaking a lot of ListCells */
list_free(joinquals);
@@ -3936,7 +3942,8 @@ calc_joinrel_size_estimate(PlannerInfo *root,
restrictlist,
0,
jointype,
- sjinfo);
+ sjinfo,
+ NIL);
pselec = 0.0; /* not used, keep compiler quiet */
}
diff --git a/src/backend/optimizer/util/orclauses.c b/src/backend/optimizer/util/orclauses.c
index ea831f5..6299e75 100644
--- a/src/backend/optimizer/util/orclauses.c
+++ b/src/backend/optimizer/util/orclauses.c
@@ -280,7 +280,7 @@ consider_new_or_clause(PlannerInfo *root, RelOptInfo *rel,
* saving work later.)
*/
or_selec = clause_selectivity(root, (Node *) or_rinfo,
- 0, JOIN_INNER, NULL);
+ 0, JOIN_INNER, NULL, NIL);
/*
* The clause is only worth adding to the query if it rejects a useful
@@ -342,7 +342,7 @@ consider_new_or_clause(PlannerInfo *root, RelOptInfo *rel,
/* Compute inner-join size */
orig_selec = clause_selectivity(root, (Node *) join_or_rinfo,
- 0, JOIN_INNER, &sjinfo);
+ 0, JOIN_INNER, &sjinfo, NIL);
/* And hack cached selectivity so join size remains the same */
join_or_rinfo->norm_selec = orig_selec / or_selec;
diff --git a/src/backend/utils/adt/selfuncs.c b/src/backend/utils/adt/selfuncs.c
index d396ef1..805d633 100644
--- a/src/backend/utils/adt/selfuncs.c
+++ b/src/backend/utils/adt/selfuncs.c
@@ -1627,13 +1627,15 @@ booltestsel(PlannerInfo *root, BoolTestType booltesttype, Node *arg,
case IS_NOT_FALSE:
selec = (double) clause_selectivity(root, arg,
varRelid,
- jointype, sjinfo);
+ jointype, sjinfo,
+ NIL);
break;
case IS_FALSE:
case IS_NOT_TRUE:
selec = 1.0 - (double) clause_selectivity(root, arg,
varRelid,
- jointype, sjinfo);
+ jointype, sjinfo,
+ NIL);
break;
default:
elog(ERROR, "unrecognized booltesttype: %d",
@@ -6260,7 +6262,8 @@ genericcostestimate(PlannerInfo *root,
indexSelectivity = clauselist_selectivity(root, selectivityQuals,
index->rel->relid,
JOIN_INNER,
- NULL);
+ NULL,
+ NIL);
/*
* If caller didn't give us an estimate, estimate the number of index
@@ -6580,7 +6583,8 @@ btcostestimate(PlannerInfo *root, IndexPath *path, double loop_count,
btreeSelectivity = clauselist_selectivity(root, selectivityQuals,
index->rel->relid,
JOIN_INNER,
- NULL);
+ NULL,
+ NIL);
numIndexTuples = btreeSelectivity * index->rel->tuples;
/*
@@ -7331,7 +7335,8 @@ gincostestimate(PlannerInfo *root, IndexPath *path, double loop_count,
*indexSelectivity = clauselist_selectivity(root, selectivityQuals,
index->rel->relid,
JOIN_INNER,
- NULL);
+ NULL,
+ NIL);
/* fetch estimated page cost for tablespace containing index */
get_tablespace_page_costs(index->reltablespace,
@@ -7561,7 +7566,7 @@ brincostestimate(PlannerInfo *root, IndexPath *path, double loop_count,
*indexSelectivity =
clauselist_selectivity(root, indexQuals,
path->indexinfo->rel->relid,
- JOIN_INNER, NULL);
+ JOIN_INNER, NULL, NIL);
*indexCorrelation = 1;
/*
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index edcafce..b7aabed 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -75,6 +75,7 @@
#include "utils/bytea.h"
#include "utils/guc_tables.h"
#include "utils/memutils.h"
+#include "utils/mvstats.h"
#include "utils/pg_locale.h"
#include "utils/plancache.h"
#include "utils/portal.h"
@@ -393,6 +394,15 @@ static const struct config_enum_entry force_parallel_mode_options[] = {
};
/*
+ * Search algorithm for multivariate stats.
+ */
+static const struct config_enum_entry mvstat_search_options[] = {
+ {"greedy", MVSTAT_SEARCH_GREEDY, false},
+ {"exhaustive", MVSTAT_SEARCH_EXHAUSTIVE, false},
+ {NULL, 0, false}
+};
+
+/*
* Options for enum values stored in other modules
*/
extern const struct config_enum_entry wal_level_options[];
@@ -3743,6 +3753,16 @@ static struct config_enum ConfigureNamesEnum[] =
NULL, NULL, NULL
},
+ {
+ {"mvstat_search", PGC_USERSET, QUERY_TUNING_OTHER,
+ gettext_noop("Sets the algorithm used for combining multivariate stats."),
+ NULL
+ },
+ &mvstat_search_type,
+ MVSTAT_SEARCH_GREEDY, mvstat_search_options,
+ NULL, NULL, NULL
+ },
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, 0, NULL, NULL, NULL, NULL
diff --git a/src/backend/utils/mvstats/README.stats b/src/backend/utils/mvstats/README.stats
index 3e4f4d1..d404914 100644
--- a/src/backend/utils/mvstats/README.stats
+++ b/src/backend/utils/mvstats/README.stats
@@ -90,6 +90,137 @@ even attempting to do the more expensive estimation.
Whenever we find there are no suitable stats, we skip the expensive steps.
+Combining multiple statistics
+-----------------------------
+
+When estimating selectivity of a list of clauses, there may exist no statistics
+covering all of them. If there are multiple statistics, each covering some
+subset of the attributes, the optimizer needs to figure out which of those
+statistics to apply.
+
+When the statistics do not overlap, the solution is trivial - we can simply
+split the groups of conditions by the matching statistics, and then multiply the
+selectivities. For example assume multivariate statistics on (b,c) and (d,e),
+and a condition like this:
+
+ (a=1) AND (b=2) AND (c=3) AND (d=4) AND (e=5)
+
+Then (a=1) is not covered by any of the statistics, so will be estimated using
+the regular per-column statistics. The two conditions ((b=2) AND (c=3)) will be
+estimated using the (b,c) statistics, and ((d=4) AND (e=5)) will be estimated
+using (d,e) statistics. And the resulting selectivities will be estimated.
+
+Now, what if the statistics overlap? For example assume the same condition as
+above, but let's say we have statistics on (a,b,c) and (a,c,d,e). What then?
+
+As selectivity is just a probability that the condition holds for a random row,
+we can write the selectivity like this:
+
+ P(a=1 & b=2 & c=3 & d=4 & e=5)
+
+and we can rewrite it using conditional probability like this
+
+ P(a=1 & b=2 & c=3) * P(d=4 & e=5 | a=1 & b=2 & c=3)
+
+Notice that the first part already matches to (a,b,c) statistics. If we assume
+that columns that are not referenced by the same statistics are independent, we
+may rewrite the second half like this
+
+ P(d=4 & e=5 | a=1 & b=2 & c=3) = P(d=4 & e=5 | a=1 & c=3)
+
+which corresponds to the statistics on (a,c,d,e).
+
+If there are multiple statistics defined on a table, it's not difficult to come
+up with examples when there are multiple ways to combine them to cover a list of
+clauses. We need a way to find the best combination of statistics.
+
+This is the purpose of choose_mv_statistics(). It searches through the possible
+combinations of statistics, and searches such combination that
+
+ (a) covers the most clauses of the list
+
+ (b) reuses the maximum number of clauses as conditions
+ (in conditional probabilities)
+
+While (a) criteria seems natural, the (b) may seem a bit awkward at first. The
+idea is that conditions in a way of transfering information about dependencies
+between statistics.
+
+There are two alternative implementations of choose_mv_statistics() - greedy
+and exhaustive. Exhaustive actually searches through all possible combinations
+of statistics, and for larger numbers of statistics may get quite expensive
+(as it, unsurprisingly, has exponential cost). Greedy terminates in less than
+K steps (when K is the number of clauses), and in each step chooses the best
+next statistics. I've been unable to come up with an example where those two
+approaches would produce different combinations.
+
+It's possible to choose the optimization using mvstat_search_type, with either
+'greedy' or 'exhaustive' values (default is 'greedy').
+
+ SET mvstat_search_type = 'exhaustive';
+
+Note: This is meant mostly for experimentation. I do expect we'll choose one of
+the algorithms and remove the GUC before commit.
+
+
+Limitations of combining statistics
+-----------------------------------
+
+As described in the section 'Combining multiple statistics', the current appoach
+is based on transfering information between statistics by means of conditional
+probabilities. This is a relatively cheap and efficient approach, but it is
+based on two assumptions:
+
+ (1) The overlap between the statistics needs to be sufficiently large, i.e.
+ there needs to be enough columns shared by the statistics to transfer
+ information about dependencies between the remaining columns.
+
+ (2) The query needs to include sufficient clauses on the shared columns.
+
+How a violation of those assumptions may be a problem can be illustrated by
+a simple example. Assume a table with three columns (a,b,c) containing exactly
+the same values, and statistics on (a,b) and (b,c):
+
+ CREATE TABLE test AS SELECT i, i, i
+ FROM generate_series(1,1000);
+
+ CREATE STATISTICS s1 ON test (a,b) WITH (mcv);
+ CREATE STATISTICS s2 ON test (b,c) WITH (mcv);
+
+ ANALYZE test;
+
+First, let's estimate this query:
+
+ SELECT * FROM test WHERE (a < 10) AND (c < 10);
+
+Clearly, there are no conditions on 'b' (which is the only column shared by the
+two statistics), so we'll end up with an estimate based on assumption of
+independence:
+
+ P(a < 10) * P(c < 10) = 0.01 * 0.01 = 0.0001
+
+Which is a significant under-estimate, as the proper selectivity is 0.01.
+
+But let's estimate another query:
+
+ SELECT * FROM test WHERE (a < 10) AND (b < 500) AND (c < 10);
+
+In this case, the estimate may be computed for example like this:
+
+ P[(a < 10) & (b < 500) & (c < 10)]
+ = P[(a < 10) & (b < 500)] * P[(c < 10) | (a < 10) & (b < 500)]
+ = P[(a < 10) & (b < 500)] * P[(c < 10) | (b < 500)]
+
+The trouble is the probability P(c < 10 | b < 500) evaluates to 0.02, because
+we have assumed (a) and (c) are independent because there is no statistic
+containing both these columns, and the condition on (b) does not transfer
+sufficient amount of information between the two statistics.
+
+Currently, the only solution is to build statistics on all three columns, but
+see the 'combining statistics using convolution' section for ideas on how to
+improve this.
+
+
Further (possibly crazy) ideas
------------------------------
@@ -111,3 +242,38 @@ But of course, this may result in expensive estimation (CPU-wise).
So we might add a GUC to choose between a simple (single statistics) and thus
multi-statistic estimation, possibly table-level parameter (ALTER TABLE ...).
+
+
+Combining stats using convolution
+---------------------------------
+
+While the current approach for combining statistics is based on conditional
+probabilities, and thus only works when the query includes conditions on the
+overlapping parts of the statistics. But there may be other ways to combine
+statistics, relaxing this requirement.
+
+Let's assume two histograms H1 and H2 - then combining them might work about
+like this:
+
+
+ for (buckets of H1, satisfying local conditions)
+ {
+ for (buckets of H2, overlapping with H1 bucket)
+ {
+ mark H2 bucket as 'valid'
+ }
+ }
+
+ s1 = s2 = 0.0
+ for (buckets of H2 marked as valid)
+ {
+ s1 += frequency
+
+ if (bucket satistifes local conditions)
+ s2 += frequency
+ }
+
+ s = (s2 / s1) /* final selectivity estimate */
+
+However this may quickly get non-trivial, e.g. when combining two statistics
+of different types (histogram vs. MCV).
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index fea2bb7..33f5a1b 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -192,11 +192,13 @@ extern Selectivity clauselist_selectivity(PlannerInfo *root,
List *clauses,
int varRelid,
JoinType jointype,
- SpecialJoinInfo *sjinfo);
+ SpecialJoinInfo *sjinfo,
+ List *conditions);
extern Selectivity clause_selectivity(PlannerInfo *root,
Node *clause,
int varRelid,
JoinType jointype,
- SpecialJoinInfo *sjinfo);
+ SpecialJoinInfo *sjinfo,
+ List *conditions);
#endif /* COST_H */
diff --git a/src/include/utils/mvstats.h b/src/include/utils/mvstats.h
index 6708139..80bf96f 100644
--- a/src/include/utils/mvstats.h
+++ b/src/include/utils/mvstats.h
@@ -17,6 +17,14 @@
#include "fmgr.h"
#include "commands/vacuum.h"
+typedef enum MVStatSearchType
+{
+ MVSTAT_SEARCH_EXHAUSTIVE, /* exhaustive search */
+ MVSTAT_SEARCH_GREEDY /* greedy search */
+} MVStatSearchType;
+
+extern int mvstat_search_type;
+
/*
* Degree of how much MCV item / histogram bucket matches a clause.
* This is then considered when computing the selectivity.
--
2.5.0
[text/x-patch] 0005-multivariate-histograms.patch (136.1K, ../../1457906378.27231.10.camel@2ndquadrant.com/5-0005-multivariate-histograms.patch)
download | inline diff:
From 93e428970d3d814f6b61e4a5f4384237cf94ed41 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Sun, 11 Jan 2015 20:18:24 +0100
Subject: [PATCH 5/9] multivariate histograms
- extends the pg_mv_statistic catalog (add 'hist' fields)
- building the histograms during ANALYZE
- simple estimation while planning the queries
Includes regression tests mostly equal to those for functional
dependencies / MCV lists.
---
doc/src/sgml/ref/create_statistics.sgml | 44 +
src/backend/catalog/system_views.sql | 4 +-
src/backend/commands/statscmds.c | 44 +-
src/backend/nodes/outfuncs.c | 2 +
src/backend/optimizer/path/clausesel.c | 574 +++++++-
src/backend/optimizer/util/plancat.c | 4 +-
src/backend/utils/mvstats/Makefile | 2 +-
src/backend/utils/mvstats/README.histogram | 299 ++++
src/backend/utils/mvstats/README.stats | 2 +
src/backend/utils/mvstats/common.c | 37 +-
src/backend/utils/mvstats/histogram.c | 2023 ++++++++++++++++++++++++++++
src/bin/psql/describe.c | 17 +-
src/include/catalog/pg_mv_statistic.h | 24 +-
src/include/catalog/pg_proc.h | 4 +
src/include/nodes/relation.h | 2 +
src/include/utils/mvstats.h | 136 +-
src/test/regress/expected/mv_histogram.out | 207 +++
src/test/regress/expected/rules.out | 4 +-
src/test/regress/parallel_schedule | 2 +-
src/test/regress/serial_schedule | 1 +
src/test/regress/sql/mv_histogram.sql | 176 +++
21 files changed, 3570 insertions(+), 38 deletions(-)
create mode 100644 src/backend/utils/mvstats/README.histogram
create mode 100644 src/backend/utils/mvstats/histogram.c
create mode 100644 src/test/regress/expected/mv_histogram.out
create mode 100644 src/test/regress/sql/mv_histogram.sql
diff --git a/doc/src/sgml/ref/create_statistics.sgml b/doc/src/sgml/ref/create_statistics.sgml
index d6973e8..f7336fd 100644
--- a/doc/src/sgml/ref/create_statistics.sgml
+++ b/doc/src/sgml/ref/create_statistics.sgml
@@ -133,6 +133,24 @@ CREATE STATISTICS [ IF NOT EXISTS ] <replaceable class="PARAMETER">statistics_na
</varlistentry>
<varlistentry>
+ <term><literal>histogram</> (<type>boolean</>)</term>
+ <listitem>
+ <para>
+ Enables histogram for the statistics.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>max_buckets</> (<type>integer</>)</term>
+ <listitem>
+ <para>
+ Maximum number of histogram buckets.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
<term><literal>max_mcv_items</> (<type>integer</>)</term>
<listitem>
<para>
@@ -220,6 +238,32 @@ EXPLAIN ANALYZE SELECT * FROM t2 WHERE (a = 1) AND (b = 2);
</programlisting>
</para>
+ <para>
+ Create table <structname>t3</> with two strongly correlated columns, and
+ a histogram on those two columns:
+
+<programlisting>
+CREATE TABLE t3 (
+ a float,
+ b float
+);
+
+INSERT INTO t3 SELECT mod(i,1000), mod(i,1000) + 50 * (r - 0.5) FROM (
+ SELECT i, random() r FROM generate_series(1,1000000) s(i)
+ ) foo;
+
+CREATE STATISTICS s3 ON t3 (a, b) WITH (histogram);
+
+ANALYZE t2;
+
+-- small overlap
+EXPLAIN ANALYZE SELECT * FROM t3 WHERE (a < 500) AND (b > 500);
+
+-- no overlap
+EXPLAIN ANALYZE SELECT * FROM t3 WHERE (a < 400) AND (b > 600);
+</programlisting>
+ </para>
+
</refsect1>
<refsect1>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 5c40334..b151db1 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -167,7 +167,9 @@ CREATE VIEW pg_mv_stats AS
length(S.stadeps) as depsbytes,
pg_mv_stats_dependencies_info(S.stadeps) as depsinfo,
length(S.stamcv) AS mcvbytes,
- pg_mv_stats_mcvlist_info(S.stamcv) AS mcvinfo
+ pg_mv_stats_mcvlist_info(S.stamcv) AS mcvinfo,
+ length(S.stahist) AS histbytes,
+ pg_mv_stats_histogram_info(S.stahist) AS histinfo
FROM (pg_mv_statistic S JOIN pg_class C ON (C.oid = S.starelid))
LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace);
diff --git a/src/backend/commands/statscmds.c b/src/backend/commands/statscmds.c
index c480fbe..e0b085f 100644
--- a/src/backend/commands/statscmds.c
+++ b/src/backend/commands/statscmds.c
@@ -71,12 +71,15 @@ CreateStatistics(CreateStatsStmt *stmt)
/* by default build nothing */
bool build_dependencies = false,
- build_mcv = false;
+ build_mcv = false,
+ build_histogram = false;
- int32 max_mcv_items = -1;
+ int32 max_buckets = -1,
+ max_mcv_items = -1;
/* options required because of other options */
- bool require_mcv = false;
+ bool require_mcv = false,
+ require_histogram = false;
Assert(IsA(stmt, CreateStatsStmt));
@@ -175,6 +178,29 @@ CreateStatistics(CreateStatsStmt *stmt)
MVSTAT_MCVLIST_MAX_ITEMS)));
}
+ else if (strcmp(opt->defname, "histogram") == 0)
+ build_histogram = defGetBoolean(opt);
+ else if (strcmp(opt->defname, "max_buckets") == 0)
+ {
+ max_buckets = defGetInt32(opt);
+
+ /* this option requires 'histogram' to be enabled */
+ require_histogram = true;
+
+ /* sanity check */
+ if (max_buckets < MVSTAT_HIST_MIN_BUCKETS)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("minimum number of buckets is %d",
+ MVSTAT_HIST_MIN_BUCKETS)));
+
+ else if (max_buckets > MVSTAT_HIST_MAX_BUCKETS)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("maximum number of buckets is %d",
+ MVSTAT_HIST_MAX_BUCKETS)));
+
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -183,10 +209,10 @@ CreateStatistics(CreateStatsStmt *stmt)
}
/* check that at least some statistics were requested */
- if (! (build_dependencies || build_mcv))
+ if (! (build_dependencies || build_mcv || build_histogram))
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("no statistics type (dependencies, mcv) was requested")));
+ errmsg("no statistics type (dependencies, mcv, histogram) was requested")));
/* now do some checking of the options */
if (require_mcv && (! build_mcv))
@@ -194,6 +220,11 @@ CreateStatistics(CreateStatsStmt *stmt)
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("option 'mcv' is required by other options(s)")));
+ if (require_histogram && (! build_histogram))
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("option 'histogram' is required by other options(s)")));
+
/* sort the attnums and build int2vector */
qsort(attnums, numcols, sizeof(int16), compare_int16);
stakeys = buildint2vector(attnums, numcols);
@@ -214,11 +245,14 @@ CreateStatistics(CreateStatsStmt *stmt)
values[Anum_pg_mv_statistic_deps_enabled -1] = BoolGetDatum(build_dependencies);
values[Anum_pg_mv_statistic_mcv_enabled -1] = BoolGetDatum(build_mcv);
+ values[Anum_pg_mv_statistic_hist_enabled -1] = BoolGetDatum(build_histogram);
values[Anum_pg_mv_statistic_mcv_max_items -1] = Int32GetDatum(max_mcv_items);
+ values[Anum_pg_mv_statistic_hist_max_buckets -1] = Int32GetDatum(max_buckets);
nulls[Anum_pg_mv_statistic_stadeps -1] = true;
nulls[Anum_pg_mv_statistic_stamcv -1] = true;
+ nulls[Anum_pg_mv_statistic_stahist -1] = true;
/* insert the tuple into pg_mv_statistic */
mvstatrel = heap_open(MvStatisticRelationId, RowExclusiveLock);
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 333e24b..9172f21 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2163,10 +2163,12 @@ _outMVStatisticInfo(StringInfo str, const MVStatisticInfo *node)
/* enabled statistics */
WRITE_BOOL_FIELD(deps_enabled);
WRITE_BOOL_FIELD(mcv_enabled);
+ WRITE_BOOL_FIELD(hist_enabled);
/* built/available statistics */
WRITE_BOOL_FIELD(deps_built);
WRITE_BOOL_FIELD(mcv_built);
+ WRITE_BOOL_FIELD(hist_built);
}
static void
diff --git a/src/backend/optimizer/path/clausesel.c b/src/backend/optimizer/path/clausesel.c
index 7fc0c49..5e73a4e 100644
--- a/src/backend/optimizer/path/clausesel.c
+++ b/src/backend/optimizer/path/clausesel.c
@@ -49,6 +49,7 @@ static void addRangeClause(RangeQueryClause **rqlist, Node *clause,
#define MV_CLAUSE_TYPE_FDEP 0x01
#define MV_CLAUSE_TYPE_MCV 0x02
+#define MV_CLAUSE_TYPE_HIST 0x04
static bool clause_is_mv_compatible(Node *clause, Index relid, Bitmapset **attnums,
int type);
@@ -74,6 +75,8 @@ static Selectivity clauselist_mv_selectivity(PlannerInfo *root,
static Selectivity clauselist_mv_selectivity_mcvlist(PlannerInfo *root,
List *clauses, MVStatisticInfo *mvstats,
bool *fullmatch, Selectivity *lowsel);
+static Selectivity clauselist_mv_selectivity_histogram(PlannerInfo *root,
+ List *clauses, MVStatisticInfo *mvstats);
static int update_match_bitmap_mcvlist(PlannerInfo *root, List *clauses,
int2vector *stakeys, MCVList mcvlist,
@@ -81,6 +84,12 @@ static int update_match_bitmap_mcvlist(PlannerInfo *root, List *clauses,
Selectivity *lowsel, bool *fullmatch,
bool is_or);
+static int update_match_bitmap_histogram(PlannerInfo *root, List *clauses,
+ int2vector *stakeys,
+ MVSerializedHistogram mvhist,
+ int nmatches, char * matches,
+ bool is_or);
+
static bool has_stats(List *stats, int type);
static List * find_stats(PlannerInfo *root, Index relid);
@@ -95,6 +104,7 @@ static bool stats_type_matches(MVStatisticInfo *stat, int type);
#define UPDATE_RESULT(m,r,isor) \
(m) = (isor) ? (MAX(m,r)) : (MIN(m,r))
+
/****************************************************************************
* ROUTINES TO COMPUTE SELECTIVITIES
****************************************************************************/
@@ -123,7 +133,7 @@ static bool stats_type_matches(MVStatisticInfo *stat, int type);
*
* First we try to reduce the list of clauses by applying (soft) functional
* dependencies, and then we try to estimate the selectivity of the reduced
- * list of clauses using the multivariate MCV list.
+ * list of clauses using the multivariate MCV list and histograms.
*
* Finally we remove the portion of clauses estimated using multivariate stats,
* and process the rest of the clauses using the regular per-column stats.
@@ -216,11 +226,13 @@ clauselist_selectivity(PlannerInfo *root,
* with the multivariate code and simply skip to estimation using the
* regular per-column stats.
*/
- if (has_stats(stats, MV_CLAUSE_TYPE_MCV) &&
- (count_mv_attnums(clauses, relid, MV_CLAUSE_TYPE_MCV) >= 2))
+ if (has_stats(stats, MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST) &&
+ (count_mv_attnums(clauses, relid,
+ MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST) >= 2))
{
/* collect attributes from the compatible conditions */
- Bitmapset *mvattnums = collect_mv_attnums(clauses, relid, MV_CLAUSE_TYPE_MCV);
+ Bitmapset *mvattnums = collect_mv_attnums(clauses, relid,
+ MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST);
/* and search for the statistic covering the most attributes */
MVStatisticInfo *mvstat = choose_mv_statistics(stats, mvattnums);
@@ -232,7 +244,7 @@ clauselist_selectivity(PlannerInfo *root,
/* split the clauselist into regular and mv-clauses */
clauses = clauselist_mv_split(root, relid, clauses, &mvclauses,
- mvstat, MV_CLAUSE_TYPE_MCV);
+ mvstat, MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST);
/* we've chosen the histogram to match the clauses */
Assert(mvclauses != NIL);
@@ -944,6 +956,7 @@ static Selectivity
clauselist_mv_selectivity(PlannerInfo *root, List *clauses, MVStatisticInfo *mvstats)
{
bool fullmatch = false;
+ Selectivity s1 = 0.0, s2 = 0.0;
/*
* Lowest frequency in the MCV list (may be used as an upper bound
@@ -957,9 +970,24 @@ clauselist_mv_selectivity(PlannerInfo *root, List *clauses, MVStatisticInfo *mvs
* MCV/histogram evaluation).
*/
- /* Evaluate the MCV selectivity */
- return clauselist_mv_selectivity_mcvlist(root, clauses, mvstats,
+ /* Evaluate the MCV first. */
+ s1 = clauselist_mv_selectivity_mcvlist(root, clauses, mvstats,
&fullmatch, &mcv_low);
+
+ /*
+ * If we got a full equality match on the MCV list, we're done (and
+ * the estimate is pretty good).
+ */
+ if (fullmatch && (s1 > 0.0))
+ return s1;
+
+ /* TODO if (fullmatch) without matching MCV item, use the mcv_low
+ * selectivity as upper bound */
+
+ s2 = clauselist_mv_selectivity_histogram(root, clauses, mvstats);
+
+ /* TODO clamp to <= 1.0 (or more strictly, when possible) */
+ return s1 + s2;
}
/*
@@ -1129,7 +1157,7 @@ choose_mv_statistics(List *stats, Bitmapset *attnums)
int numattrs = attrs->dim1;
/* skip dependencies-only stats */
- if (! info->mcv_built)
+ if (! (info->mcv_built || info->hist_built))
continue;
/* count columns covered by the histogram */
@@ -1360,7 +1388,7 @@ mv_compatible_walker(Node *node, mv_compatible_context *context)
case F_SCALARGTSEL:
/* not compatible with functional dependencies */
- if (! (context->types & MV_CLAUSE_TYPE_MCV))
+ if (! (context->types & (MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST)))
return true; /* terminate */
break;
@@ -1588,6 +1616,9 @@ stats_type_matches(MVStatisticInfo *stat, int type)
if ((type & MV_CLAUSE_TYPE_MCV) && stat->mcv_built)
return true;
+ if ((type & MV_CLAUSE_TYPE_HIST) && stat->hist_built)
+ return true;
+
return false;
}
@@ -1606,6 +1637,9 @@ has_stats(List *stats, int type)
/* terminate if we've found at least one matching statistics */
if (stats_type_matches(stat, type))
return true;
+
+ if ((type & MV_CLAUSE_TYPE_HIST) && stat->hist_built)
+ return true;
}
return false;
@@ -2010,3 +2044,525 @@ update_match_bitmap_mcvlist(PlannerInfo *root, List *clauses,
return nmatches;
}
+
+/*
+ * Estimate selectivity of clauses using a histogram.
+ *
+ * If there's no histogram for the stats, the function returns 0.0.
+ *
+ * The general idea of this method is similar to how MCV lists are
+ * processed, except that this introduces the concept of a partial
+ * match (MCV only works with full match / mismatch).
+ *
+ * The algorithm works like this:
+ *
+ * 1) mark all buckets as 'full match'
+ * 2) walk through all the clauses
+ * 3) for a particular clause, walk through all the buckets
+ * 4) skip buckets that are already 'no match'
+ * 5) check clause for buckets that still match (at least partially)
+ * 6) sum frequencies for buckets to get selectivity
+ *
+ * Unlike MCV lists, histograms have a concept of a partial match. In
+ * that case we use 1/2 the bucket, to minimize the average error. The
+ * MV histograms are usually less detailed than the per-column ones,
+ * meaning the sum is often quite high (thanks to combining a lot of
+ * "partially hit" buckets).
+ *
+ * Maybe we could use per-bucket information with number of distinct
+ * values it contains (for each dimension), and then use that to correct
+ * the estimate (so with 10 distinct values, we'd use 1/10 of the bucket
+ * frequency). We might also scale the value depending on the actual
+ * ndistinct estimate (not just the values observed in the sample).
+ *
+ * Another option would be to multiply the selectivities, i.e. if we get
+ * 'partial match' for a bucket for multiple conditions, we might use
+ * 0.5^k (where k is the number of conditions), instead of 0.5. This
+ * probably does not minimize the average error, though.
+ *
+ * TODO This might use a similar shortcut to MCV lists - count buckets
+ * marked as partial/full match, and terminate once this drop to 0.
+ * Not sure if it's really worth it - for MCV lists a situation like
+ * this is not uncommon, but for histograms it's not that clear.
+ */
+static Selectivity
+clauselist_mv_selectivity_histogram(PlannerInfo *root, List *clauses,
+ MVStatisticInfo *mvstats)
+{
+ int i;
+ Selectivity s = 0.0;
+ Selectivity u = 0.0;
+
+ int nmatches = 0;
+ char *matches = NULL;
+
+ MVSerializedHistogram mvhist = NULL;
+
+ /* there's no histogram */
+ if (! mvstats->hist_built)
+ return 0.0;
+
+ /* There may be no histogram in the stats (check hist_built flag) */
+ mvhist = load_mv_histogram(mvstats->mvoid);
+
+ Assert (mvhist != NULL);
+ Assert (clauses != NIL);
+ Assert (list_length(clauses) >= 2);
+
+ /*
+ * Bitmap of bucket matches (mismatch, partial, full). by default
+ * all buckets fully match (and we'll eliminate them).
+ */
+ matches = palloc0(sizeof(char) * mvhist->nbuckets);
+ memset(matches, MVSTATS_MATCH_FULL, sizeof(char)*mvhist->nbuckets);
+
+ nmatches = mvhist->nbuckets;
+
+ /* build the match bitmap */
+ update_match_bitmap_histogram(root, clauses,
+ mvstats->stakeys, mvhist,
+ nmatches, matches, false);
+
+ /* now, walk through the buckets and sum the selectivities */
+ for (i = 0; i < mvhist->nbuckets; i++)
+ {
+ /*
+ * Find out what part of the data is covered by the histogram,
+ * so that we can 'scale' the selectivity properly (e.g. when
+ * only 50% of the sample got into the histogram, and the rest
+ * is in a MCV list).
+ *
+ * TODO This might be handled by keeping a global "frequency"
+ * for the whole histogram, which might save us some time
+ * spent accessing the not-matching part of the histogram.
+ * Although it's likely in a cache, so it's very fast.
+ */
+ u += mvhist->buckets[i]->ntuples;
+
+ if (matches[i] == MVSTATS_MATCH_FULL)
+ s += mvhist->buckets[i]->ntuples;
+ else if (matches[i] == MVSTATS_MATCH_PARTIAL)
+ s += 0.5 * mvhist->buckets[i]->ntuples;
+ }
+
+#ifdef DEBUG_MVHIST
+ debug_histogram_matches(mvhist, matches);
+#endif
+
+ /* release the allocated bitmap and deserialized histogram */
+ pfree(matches);
+ pfree(mvhist);
+
+ return s * u;
+}
+
+/* cached result of bucket boundary comparison for a single dimension */
+
+#define HIST_CACHE_NOT_FOUND 0x00
+#define HIST_CACHE_FALSE 0x01
+#define HIST_CACHE_TRUE 0x03
+#define HIST_CACHE_MASK 0x02
+
+static char
+bucket_contains_value(FmgrInfo ltproc, Datum constvalue,
+ Datum min_value, Datum max_value,
+ int min_index, int max_index,
+ bool min_include, bool max_include,
+ char * callcache)
+{
+ bool a, b;
+
+ char min_cached = callcache[min_index];
+ char max_cached = callcache[max_index];
+
+ /*
+ * First some quick checks on equality - if any of the boundaries equals,
+ * we have a partial match (so no need to call the comparator).
+ */
+ if (((min_value == constvalue) && (min_include)) ||
+ ((max_value == constvalue) && (max_include)))
+ return MVSTATS_MATCH_PARTIAL;
+
+ /* Keep the values 0/1 because of the XOR at the end. */
+ a = ((min_cached & HIST_CACHE_MASK) >> 1);
+ b = ((max_cached & HIST_CACHE_MASK) >> 1);
+
+ /*
+ * If result for the bucket lower bound not in cache, evaluate the function
+ * and store the result in the cache.
+ */
+ if (! min_cached)
+ {
+ a = DatumGetBool(FunctionCall2Coll(<proc,
+ DEFAULT_COLLATION_OID,
+ constvalue, min_value));
+ /* remember the result */
+ callcache[min_index] = (a) ? HIST_CACHE_TRUE : HIST_CACHE_FALSE;
+ }
+
+ /* And do the same for the upper bound. */
+ if (! max_cached)
+ {
+ b = DatumGetBool(FunctionCall2Coll(<proc,
+ DEFAULT_COLLATION_OID,
+ constvalue, max_value));
+ /* remember the result */
+ callcache[max_index] = (b) ? HIST_CACHE_TRUE : HIST_CACHE_FALSE;
+ }
+
+ return (a ^ b) ? MVSTATS_MATCH_PARTIAL : MVSTATS_MATCH_NONE;
+}
+
+static char
+bucket_is_smaller_than_value(FmgrInfo opproc, Datum constvalue,
+ Datum min_value, Datum max_value,
+ int min_index, int max_index,
+ bool min_include, bool max_include,
+ char * callcache, bool isgt)
+{
+ char min_cached = callcache[min_index];
+ char max_cached = callcache[max_index];
+
+ /* Keep the values 0/1 because of the XOR at the end. */
+ bool a = ((min_cached & HIST_CACHE_MASK) >> 1);
+ bool b = ((max_cached & HIST_CACHE_MASK) >> 1);
+
+ if (! min_cached)
+ {
+ a = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ min_value,
+ constvalue));
+ /* remember the result */
+ callcache[min_index] = (a) ? HIST_CACHE_TRUE : HIST_CACHE_FALSE;
+ }
+
+ if (! max_cached)
+ {
+ b = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ max_value,
+ constvalue));
+ /* remember the result */
+ callcache[max_index] = (b) ? HIST_CACHE_TRUE : HIST_CACHE_FALSE;
+ }
+
+ /*
+ * Now, we need to combine both results into the final answer, and we need
+ * to be careful about the 'isgt' variable which kinda inverts the meaning.
+ *
+ * First, we handle the case when each boundary returns different results.
+ * In that case the outcome can only be 'partial' match.
+ */
+ if (a != b)
+ return MVSTATS_MATCH_PARTIAL;
+
+ /*
+ * When the results are the same, then it depends on the 'isgt' value. There
+ * are four options:
+ *
+ * isgt=false a=b=true => full match
+ * isgt=false a=b=false => empty
+ * isgt=true a=b=true => empty
+ * isgt=true a=b=false => full match
+ *
+ * We'll cheat a bit, because we know that (a=b) so we'll use just one of them.
+ */
+ if (isgt)
+ return (!a) ? MVSTATS_MATCH_FULL : MVSTATS_MATCH_NONE;
+ else
+ return ( a) ? MVSTATS_MATCH_FULL : MVSTATS_MATCH_NONE;
+}
+
+/*
+ * Evaluate clauses using the histogram, and update the match bitmap.
+ *
+ * The bitmap may be already partially set, so this is really a way to
+ * combine results of several clause lists - either when computing
+ * conditional probability P(A|B) or a combination of AND/OR clauses.
+ *
+ * Note: This is not a simple bitmap in the sense that there are more
+ * than two possible values for each item - no match, partial
+ * match and full match. So we need 2 bits per item.
+ *
+ * TODO This works with 'bitmap' where each item is represented as a
+ * char, which is slightly wasteful. Instead, we could use a bitmap
+ * with 2 bits per item, reducing the size to ~1/4. By using values
+ * 0, 1 and 3 (instead of 0, 1 and 2), the operations (merging etc.)
+ * might be performed just like for simple bitmap by using & and |,
+ * which might be faster than min/max.
+ */
+static int
+update_match_bitmap_histogram(PlannerInfo *root, List *clauses,
+ int2vector *stakeys,
+ MVSerializedHistogram mvhist,
+ int nmatches, char * matches,
+ bool is_or)
+{
+ int i;
+ ListCell * l;
+
+ /*
+ * Used for caching function calls, only once per deduplicated value.
+ *
+ * We know may have up to (2 * nbuckets) values per dimension. It's
+ * probably overkill, but let's allocate that once for all clauses,
+ * to minimize overhead.
+ *
+ * Also, we only need two bits per value, but this allocates byte
+ * per value. Might be worth optimizing.
+ *
+ * 0x00 - not yet called
+ * 0x01 - called, result is 'false'
+ * 0x03 - called, result is 'true'
+ */
+ char *callcache = palloc(mvhist->nbuckets);
+
+ Assert(mvhist != NULL);
+ Assert(mvhist->nbuckets > 0);
+ Assert(nmatches >= 0);
+ Assert(nmatches <= mvhist->nbuckets);
+
+ Assert(clauses != NIL);
+ Assert(list_length(clauses) >= 1);
+
+ /* loop through the clauses and do the estimation */
+ foreach (l, clauses)
+ {
+ Node * clause = (Node*)lfirst(l);
+
+ /* if it's a RestrictInfo, then extract the clause */
+ if (IsA(clause, RestrictInfo))
+ clause = (Node*)((RestrictInfo*)clause)->clause;
+
+ /* it's either OpClause, or NullTest */
+ if (is_opclause(clause))
+ {
+ OpExpr * expr = (OpExpr*)clause;
+ bool varonleft = true;
+ bool ok;
+
+ FmgrInfo opproc; /* operator */
+ fmgr_info(get_opcode(expr->opno), &opproc);
+
+ /* reset the cache (per clause) */
+ memset(callcache, 0, mvhist->nbuckets);
+
+ ok = (NumRelids(clause) == 1) &&
+ (is_pseudo_constant_clause(lsecond(expr->args)) ||
+ (varonleft = false,
+ is_pseudo_constant_clause(linitial(expr->args))));
+
+ if (ok)
+ {
+ FmgrInfo ltproc;
+ RegProcedure oprrest = get_oprrest(expr->opno);
+
+ Var * var = (varonleft) ? linitial(expr->args) : lsecond(expr->args);
+ Const * cst = (varonleft) ? lsecond(expr->args) : linitial(expr->args);
+ bool isgt = (! varonleft);
+
+ TypeCacheEntry *typecache
+ = lookup_type_cache(var->vartype, TYPECACHE_LT_OPR);
+
+ /* lookup dimension for the attribute */
+ int idx = mv_get_index(var->varattno, stakeys);
+
+ fmgr_info(get_opcode(typecache->lt_opr), <proc);
+
+ /*
+ * Check this for all buckets that still have "true" in the bitmap
+ *
+ * We already know the clauses use suitable operators (because that's
+ * how we filtered them).
+ */
+ for (i = 0; i < mvhist->nbuckets; i++)
+ {
+ char res = MVSTATS_MATCH_NONE;
+
+ MVSerializedBucket bucket = mvhist->buckets[i];
+
+ /* histogram boundaries */
+ Datum minval, maxval;
+ bool mininclude, maxinclude;
+ int minidx, maxidx;
+
+ /*
+ * For AND-lists, we can also mark NULL buckets as 'no match'
+ * (and then skip them). For OR-lists this is not possible.
+ */
+ if ((! is_or) && bucket->nullsonly[idx])
+ matches[i] = MVSTATS_MATCH_NONE;
+
+ /*
+ * Skip buckets that were already eliminated - this is impotant
+ * considering how we update the info (we only lower the match).
+ * We can't really do anything about the MATCH_PARTIAL buckets.
+ */
+ if ((! is_or) && (matches[i] == MVSTATS_MATCH_NONE))
+ continue;
+ else if (is_or && (matches[i] == MVSTATS_MATCH_FULL))
+ continue;
+
+ /* lookup the values and cache of function calls */
+ minidx = bucket->min[idx];
+ maxidx = bucket->max[idx];
+
+ minval = mvhist->values[idx][bucket->min[idx]];
+ maxval = mvhist->values[idx][bucket->max[idx]];
+
+ mininclude = bucket->min_inclusive[idx];
+ maxinclude = bucket->max_inclusive[idx];
+
+ /*
+ * TODO Maybe it's possible to add here a similar optimization
+ * as for the MCV lists:
+ *
+ * (nmatches == 0) && AND-list => all eliminated (FALSE)
+ * (nmatches == N) && OR-list => all eliminated (TRUE)
+ *
+ * But it's more complex because of the partial matches.
+ */
+
+ /*
+ * If it's not a "<" or ">" or "=" operator, just ignore the
+ * clause. Otherwise note the relid and attnum for the variable.
+ *
+ * TODO I'm really unsure the handling of 'isgt' flag (that is, clauses
+ * with reverse order of variable/constant) is correct. I wouldn't
+ * be surprised if there was some mixup. Using the lt/gt operators
+ * instead of messing with the opproc could make it simpler.
+ * It would however be using a different operator than the query,
+ * although it's not any shadier than using the selectivity function
+ * as is done currently.
+ */
+ switch (oprrest)
+ {
+ case F_SCALARLTSEL: /* Var < Const */
+ case F_SCALARGTSEL: /* Var > Const */
+
+ res = bucket_is_smaller_than_value(opproc, cst->constvalue,
+ minval, maxval,
+ minidx, maxidx,
+ mininclude, maxinclude,
+ callcache, isgt);
+ break;
+
+ case F_EQSEL:
+
+ /*
+ * We only check whether the value is within the bucket, using the
+ * lt operator, and we also check for equality with the boundaries.
+ */
+
+ res = bucket_contains_value(ltproc, cst->constvalue,
+ minval, maxval,
+ minidx, maxidx,
+ mininclude, maxinclude,
+ callcache);
+ break;
+ }
+
+ UPDATE_RESULT(matches[i], res, is_or);
+
+ }
+ }
+ }
+ else if (IsA(clause, NullTest))
+ {
+ NullTest * expr = (NullTest*)clause;
+ Var * var = (Var*)(expr->arg);
+
+ /* FIXME proper matching attribute to dimension */
+ int idx = mv_get_index(var->varattno, stakeys);
+
+ /*
+ * Walk through the buckets and evaluate the current clause. We can
+ * skip items that were already ruled out, and terminate if there are
+ * no remaining buckets that might possibly match.
+ */
+ for (i = 0; i < mvhist->nbuckets; i++)
+ {
+ MVSerializedBucket bucket = mvhist->buckets[i];
+
+ /*
+ * Skip buckets that were already eliminated - this is impotant
+ * considering how we update the info (we only lower the match)
+ */
+ if ((! is_or) && (matches[i] == MVSTATS_MATCH_NONE))
+ continue;
+ else if (is_or && (matches[i] == MVSTATS_MATCH_FULL))
+ continue;
+
+ /* if the clause mismatches the bucket, set it as MATCH_NONE */
+ if ((expr->nulltesttype == IS_NULL)
+ && (! bucket->nullsonly[idx]))
+ UPDATE_RESULT(matches[i], MVSTATS_MATCH_NONE, is_or);
+
+ else if ((expr->nulltesttype == IS_NOT_NULL) &&
+ (bucket->nullsonly[idx]))
+ UPDATE_RESULT(matches[i], MVSTATS_MATCH_NONE, is_or);
+ }
+ }
+ else if (or_clause(clause) || and_clause(clause))
+ {
+ /* AND/OR clause, with all clauses compatible with the selected MV stat */
+
+ int i;
+ BoolExpr *orclause = ((BoolExpr*)clause);
+ List *orclauses = orclause->args;
+
+ /* match/mismatch bitmap for each bucket */
+ int or_nmatches = 0;
+ char * or_matches = NULL;
+
+ Assert(orclauses != NIL);
+ Assert(list_length(orclauses) >= 2);
+
+ /* number of matching buckets */
+ or_nmatches = mvhist->nbuckets;
+
+ /* by default none of the buckets matches the clauses */
+ or_matches = palloc0(sizeof(char) * or_nmatches);
+
+ if (or_clause(clause))
+ {
+ /* OR clauses assume nothing matches, initially */
+ memset(or_matches, MVSTATS_MATCH_NONE, sizeof(char)*or_nmatches);
+ or_nmatches = 0;
+ }
+ else
+ {
+ /* AND clauses assume nothing matches, initially */
+ memset(or_matches, MVSTATS_MATCH_FULL, sizeof(char)*or_nmatches);
+ }
+
+ /* build the match bitmap for the OR-clauses */
+ or_nmatches = update_match_bitmap_histogram(root, orclauses,
+ stakeys, mvhist,
+ or_nmatches, or_matches, or_clause(clause));
+
+ /* merge the bitmap into the existing one*/
+ for (i = 0; i < mvhist->nbuckets; i++)
+ {
+ /*
+ * To AND-merge the bitmaps, a MIN() semantics is used.
+ * For OR-merge, use MAX().
+ *
+ * FIXME this does not decrease the number of matches
+ */
+ UPDATE_RESULT(matches[i], or_matches[i], is_or);
+ }
+
+ pfree(or_matches);
+
+ }
+ else
+ elog(ERROR, "unknown clause type: %d", clause->type);
+ }
+
+ /* free the call cache */
+ pfree(callcache);
+
+ return nmatches;
+}
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index 8394111..2519249 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -412,7 +412,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
mvstat = (Form_pg_mv_statistic) GETSTRUCT(htup);
/* unavailable stats are not interesting for the planner */
- if (mvstat->deps_built || mvstat->mcv_built)
+ if (mvstat->deps_built || mvstat->mcv_built || mvstat->hist_built)
{
info = makeNode(MVStatisticInfo);
@@ -422,10 +422,12 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
/* enabled statistics */
info->deps_enabled = mvstat->deps_enabled;
info->mcv_enabled = mvstat->mcv_enabled;
+ info->hist_enabled = mvstat->hist_enabled;
/* built/available statistics */
info->deps_built = mvstat->deps_built;
info->mcv_built = mvstat->mcv_built;
+ info->hist_built = mvstat->hist_built;
/* stakeys */
adatum = SysCacheGetAttr(MVSTATOID, htup,
diff --git a/src/backend/utils/mvstats/Makefile b/src/backend/utils/mvstats/Makefile
index f9bf10c..9dbb3b6 100644
--- a/src/backend/utils/mvstats/Makefile
+++ b/src/backend/utils/mvstats/Makefile
@@ -12,6 +12,6 @@ subdir = src/backend/utils/mvstats
top_builddir = ../../../..
include $(top_builddir)/src/Makefile.global
-OBJS = common.o dependencies.o mcv.o
+OBJS = common.o dependencies.o histogram.o mcv.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/utils/mvstats/README.histogram b/src/backend/utils/mvstats/README.histogram
new file mode 100644
index 0000000..cd640e5
--- /dev/null
+++ b/src/backend/utils/mvstats/README.histogram
@@ -0,0 +1,299 @@
+Multivariate histograms
+=======================
+
+Histograms on individual attributes consist of buckets represented by ranges,
+covering the domain of the attribute. That is, each bucket is a [min,max]
+interval, and contains all values in this range. The histogram is built in such
+a way that all buckets have about the same frequency.
+
+Multivariate histograms are an extension into n-dimensional space - the buckets
+are n-dimensional intervals (i.e. n-dimensional rectagles), covering the domain
+of the combination of attributes. That is, each bucket has a vector of lower
+and upper boundaries, denoted min[i] and max[i] (where i = 1..n).
+
+In addition to the boundaries, each bucket tracks additional info:
+
+ * frequency (fraction of tuples in the bucket)
+ * whether the boundaries are inclusive or exclusive
+ * whether the dimension contains only NULL values
+ * number of distinct values in each dimension (for building only)
+
+It's possible that in the future we'll multiple histogram types, with different
+features. We do however expect all the types to share the same representation
+(buckets as ranges) and only differ in how we build them.
+
+The current implementation builds non-overlapping buckets, that may not be true
+for some histogram types and the code should not rely on this assumption. There
+are interesting types of histograms (or algorithms) with overlapping buckets.
+
+When used on low-cardinality data, histograms usually perform considerably worse
+than MCV lists (which are a good fit for this kind of data). This is especially
+true on label-like values, where ordering of the values is mostly unrelated to
+meaning of the data, as proper ordering is crucial for histograms.
+
+On high-cardinality data the histograms are usually a better choice, because MCV
+lists can't represent the distribution accurately enough.
+
+
+Selectivity estimation
+----------------------
+
+The estimation is implemented in clauselist_mv_selectivity_histogram(), and
+works very similarly to clauselist_mv_selectivity_mcvlist.
+
+The main difference is that while MCV lists support exact matches, histograms
+often result in approximate matches - e.g. with equality we can only say if
+the constant would be part of the bucket, but not whether it really is there
+or what fraction of the bucket it corresponds to. In this case we rely on
+some defaults just like in the per-column histograms.
+
+The current implementation uses histograms to estimates those types of clauses
+(think of WHERE conditions):
+
+ (a) equality clauses WHERE (a = 1) AND (b = 2)
+ (b) inequality clauses WHERE (a < 1) AND (b >= 2)
+ (c) NULL clauses WHERE (a IS NULL) AND (b IS NOT NULL)
+ (d) OR-clauses WHERE (a = 1) OR (b = 2)
+
+Similarly to MCV lists, it's possible to add support for additional types of
+clauses, for example:
+
+ (e) multi-var clauses WHERE (a > b)
+
+and so on. These are tasks for the future, not yet implemented.
+
+
+When evaluating a clause on a bucket, we may get one of three results:
+
+ (a) FULL_MATCH - The bucket definitely matches the clause.
+
+ (b) PARTIAL_MATCH - The bucket matches the clause, but not necessarily all
+ the tuples it represents.
+
+ (c) NO_MATCH - The bucket definitely does not match the clause.
+
+This may be illustrated using a range [1, 5], which is essentially a 1-D bucket.
+With clause
+
+ WHERE (a < 10) => FULL_MATCH (all range values are below
+ 10, so the whole bucket matches)
+
+ WHERE (a < 3) => PARTIAL_MATCH (there may be values matching
+ the clause, but we don't know how many)
+
+ WHERE (a < 0) => NO_MATCH (the whole range is above 1, so
+ no values from the bucket can match)
+
+Some clauses may produce only some of those results - for example equality
+clauses may never produce FULL_MATCH as we always hit only part of the bucket
+(we can't match both boundaries at the same time). This results in less accurate
+estimates compared to MCV lists, where we can hit a MCV items exactly (there's
+no PARTIAL match in MCV).
+
+There are also clauses that may not produce any PARTIAL_MATCH results. A nice
+example of that is 'IS [NOT] NULL' clause, which either matches the bucket
+completely (FULL_MATCH) or not at all (NO_MATCH), thanks to how the NULL-buckets
+are constructed.
+
+Computing the total selectivity estimate is trivial - simply sum selectivities
+from all the FULL_MATCH and PARTIAL_MATCH buckets (but for buckets marked with
+PARTIAL_MATCH, multiply the frequency by 0.5 to minimize the average error).
+
+
+Building a histogram
+---------------------
+
+The algorithm of building a histogram in general is quite simple:
+
+ (a) create an initial bucket (containing all sample rows)
+
+ (b) create NULL buckets (by splitting the initial bucket)
+
+ (c) repeat
+
+ (1) choose bucket to split next
+
+ (2) terminate if no bucket that might be split found, or if we've
+ reached the maximum number of buckets (16384)
+
+ (3) choose dimension to partition the bucket by
+
+ (4) partition the bucket by the selected dimension
+
+The main complexity is hidden in steps (c.1) and (c.3), i.e. how we choose the
+bucket and dimension for the split, as discussed in the next section.
+
+
+Partitioning criteria
+---------------------
+
+Similarly to one-dimensional histograms, we want to produce buckets with roughly
+the same frequency.
+
+We also need to produce "regular" buckets, because buckets with one dimension
+much longer than the others are very likely to match a lot of conditions (which
+increases error, even if the bucket frequency is very low).
+
+This is especially important when handling OR-clauses, because in that case each
+clause may add buckets independently. With AND-clauses all the clauses have to
+match each bucket, which makes this issue somewhat less concenrning.
+
+To achieve this, we choose the largest bucket (containing the most sample rows),
+but we only choose buckets that can actually be split (have at least 3 different
+combinations of values).
+
+Then we choose the "longest" dimension of the bucket, which is computed by using
+the distinct values in the sample as a measure.
+
+For details see functions select_bucket_to_partition() and partition_bucket(),
+which also includes further discussion.
+
+
+The current limit on number of buckets (16384) is mostly arbitrary, but chosen
+so that it guarantees we don't exceed the number of distinct values indexable by
+uint16 in any of the dimensions. In practice we could handle more buckets as we
+index each dimension separately and the splits should use the dimensions evenly.
+
+Also, histograms this large (with 16k values in multiple dimensions) would be
+quite expensive to build and process, so the 16k limit is rather reasonable.
+
+The actual number of buckets is also related to statistics target, because we
+require MIN_BUCKET_ROWS (10) tuples per bucket before a split, so we can't have
+more than (2 * 300 * target / 10) buckets. For the default target (100) this
+evaluates to ~6k.
+
+
+NULL handling (create_null_buckets)
+-----------------------------------
+
+When building histograms on a single attribute, we first filter out NULL values.
+In the multivariate case, we can't really do that because the rows may contain
+a mix of NULL and non-NULL values in different columns (so we can't simply
+filter all of them out).
+
+For this reason, the histograms are built in a way so that for each bucket, each
+dimension only contains only NULL or non-NULL values. Building the NULL-buckets
+happens as the first step in the build, by the create_null_buckets() function.
+The number of NULL buckets, as produced by this function, has a clear upper
+boundary (2^N) where N is the number of dimensions (attributes the histogram is
+built on). Or rather 2^K where K is the number of attributes that are not marked
+as not-NULL.
+
+The buckets with NULL dimensions are then subject to the same build algorithm
+(i.e. may be split into smaller buckets) just like any other bucket, but may
+only be split by non-NULL dimension.
+
+
+Serialization
+-------------
+
+To store the histogram in pg_mv_statistic table, it is serialized into a more
+efficient form. We also use the representation for estimation, i.e. we don't
+fully deserialize the histogram.
+
+For example the boundary values are deduplicated to minimize the required space.
+How much redundancy is there, actually? Let's assume there are no NULL values,
+so we start with a single bucket - in that case we have 2*N boundaries. Each
+time we split a bucket we introduce one new value (in the "middle" of one of
+the dimensions), and keep boundries for all the other dimensions. So after K
+splits, we have up to
+
+ 2*N + K
+
+unique boundary values (we may have fewe values, if the same value is used for
+several splits). But after K splits we do have (K+1) buckets, so
+
+ (K+1) * 2 * N
+
+boundary values. Using e.g. N=4 and K=999, we arrive to those numbers:
+
+ 2*N + K = 1007
+ (K+1) * 2 * N = 8000
+
+wich means a lot of redundancy. It's somewhat counter-intuitive that the number
+of distinct values does not really depend on the number of dimensions (except
+for the initial bucket, but that's negligible compared to the total).
+
+By deduplicating the values and replacing them with 16-bit indexes (uint16), we
+reduce the required space to
+
+ 1007 * 8 + 8000 * 2 ~= 24kB
+
+which is significantly less than 64kB required for the 'raw' histogram (assuming
+the values are 8B).
+
+While the bytea compression (pglz) might achieve the same reduction of space,
+the deduplicated representation is used to optimize the estimation by caching
+results of function calls for already visited values. This significantly
+reduces the number of calls to (often quite expensive) operators.
+
+Note: Of course, this reasoning only holds for histograms built by the algorithm
+that simply splits the buckets in half. Other histograms types (e.g. containing
+overlapping buckets) may behave differently and require different serialization.
+
+Serialized histograms are marked with 'magic' constant, to make it easier to
+check the bytea value really is a serialized histogram.
+
+
+varlena compression
+-------------------
+
+This serialization may however disable automatic varlena compression, the array
+of unique values is placed at the beginning of the serialized form. Which is
+exactly the chunk used by pglz to check if the data is compressible, and it
+will probably decide it's not very compressible. This is similar to the issue
+we had with JSONB initially.
+
+Maybe storing buckets first would make it work, as the buckets may be better
+compressible.
+
+On the other hand the serialization is actually a context-aware compression,
+usually compressing to ~30% (or even less, with large data types). So the lack
+of additional pglz compression may be acceptable.
+
+
+Deserialization
+---------------
+
+The deserialization is not a perfect inverse of the serialization, as we keep
+the deduplicated arrays. This reduces the amount of memory and also allows
+optimizations during estimation (e.g. we can cache results for the distinct
+values, saving expensive function calls).
+
+
+Inspecting the histogram
+------------------------
+
+Inspecting the regular (per-attribute) histograms is trivial, as it's enough
+to select the columns from pg_stats - the data is encoded as anyarray, so we
+simply get the text representation of the array.
+
+With multivariate histograms it's not that simple due to the possible mix of
+data types in the histogram. It might be possible to produce similar array-like
+text representation, but that'd unnecessarily complicate further processing
+and analysis of the histogram. Instead, there's a SRF function that allows
+access to lower/upper boundaries, frequencies etc.
+
+ SELECT * FROM pg_mv_histogram_buckets();
+
+It has two input parameters:
+
+ oid - OID of the histogram (pg_mv_statistic.staoid)
+ otype - type of output
+
+and produces a table with these columns:
+
+ - bucket ID (0...nbuckets-1)
+ - lower bucket boundaries (string array)
+ - upper bucket boundaries (string array)
+ - nulls only dimensions (boolean array)
+ - lower boundary inclusive (boolean array)
+ - upper boundary includive (boolean array)
+ - frequency (double precision)
+
+The 'otype' accepts three values, determining what will be returned in the
+lower/upper boundary arrays:
+
+ - 0 - values stored in the histogram, encoded as text
+ - 1 - indexes into the deduplicated arrays
+ - 2 - idnexes into the deduplicated arrays, scaled to [0,1]
diff --git a/src/backend/utils/mvstats/README.stats b/src/backend/utils/mvstats/README.stats
index 5c5c59a..3e4f4d1 100644
--- a/src/backend/utils/mvstats/README.stats
+++ b/src/backend/utils/mvstats/README.stats
@@ -18,6 +18,8 @@ Currently we only have two kinds of multivariate statistics
(b) MCV lists (README.mcv)
+ (c) multivariate histograms (README.histogram)
+
Compatible clause types
-----------------------
diff --git a/src/backend/utils/mvstats/common.c b/src/backend/utils/mvstats/common.c
index 4f5a842..f6d1074 100644
--- a/src/backend/utils/mvstats/common.c
+++ b/src/backend/utils/mvstats/common.c
@@ -13,11 +13,11 @@
*
*-------------------------------------------------------------------------
*/
+#include "postgres.h"
+#include "utils/array.h"
#include "common.h"
-#include "utils/array.h"
-
static VacAttrStats ** lookup_var_attr_stats(int2vector *attrs,
int natts,
VacAttrStats **vacattrstats);
@@ -52,7 +52,8 @@ build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
MVStatisticInfo *stat = (MVStatisticInfo *)lfirst(lc);
MVDependencies deps = NULL;
MCVList mcvlist = NULL;
- int numrows_filtered = 0;
+ MVHistogram histogram = NULL;
+ int numrows_filtered = numrows;
VacAttrStats **stats = NULL;
int numatts = 0;
@@ -95,8 +96,12 @@ build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
if (stat->mcv_enabled)
mcvlist = build_mv_mcvlist(numrows, rows, attrs, stats, &numrows_filtered);
+ /* build a multivariate histogram on the columns */
+ if ((numrows_filtered > 0) && (stat->hist_enabled))
+ histogram = build_mv_histogram(numrows_filtered, rows, attrs, stats, numrows);
+
/* store the histogram / MCV list in the catalog */
- update_mv_stats(stat->mvoid, deps, mcvlist, attrs, stats);
+ update_mv_stats(stat->mvoid, deps, mcvlist, histogram, attrs, stats);
}
}
@@ -176,6 +181,8 @@ list_mv_stats(Oid relid)
info->deps_built = stats->deps_built;
info->mcv_enabled = stats->mcv_enabled;
info->mcv_built = stats->mcv_built;
+ info->hist_enabled = stats->hist_enabled;
+ info->hist_built = stats->hist_built;
result = lappend(result, info);
}
@@ -190,7 +197,6 @@ list_mv_stats(Oid relid)
return result;
}
-
/*
* Find attnims of MV stats using the mvoid.
*/
@@ -236,9 +242,16 @@ find_mv_attnums(Oid mvoid, Oid *relid)
}
+/*
+ * FIXME This adds statistics, but we need to drop statistics when the
+ * table is dropped. Not sure what to do when a column is dropped.
+ * Either we can (a) remove all stats on that column, (b) remove
+ * the column from defined stats and force rebuild, (c) remove the
+ * column on next ANALYZE. Or maybe something else?
+ */
void
update_mv_stats(Oid mvoid,
- MVDependencies dependencies, MCVList mcvlist,
+ MVDependencies dependencies, MCVList mcvlist, MVHistogram histogram,
int2vector *attrs, VacAttrStats **stats)
{
HeapTuple stup,
@@ -271,22 +284,34 @@ update_mv_stats(Oid mvoid,
values[Anum_pg_mv_statistic_stamcv - 1] = PointerGetDatum(data);
}
+ if (histogram != NULL)
+ {
+ bytea * data = serialize_mv_histogram(histogram, attrs, stats);
+ nulls[Anum_pg_mv_statistic_stahist-1] = (data == NULL);
+ values[Anum_pg_mv_statistic_stahist - 1]
+ = PointerGetDatum(data);
+ }
+
/* always replace the value (either by bytea or NULL) */
replaces[Anum_pg_mv_statistic_stadeps -1] = true;
replaces[Anum_pg_mv_statistic_stamcv -1] = true;
+ replaces[Anum_pg_mv_statistic_stahist-1] = true;
/* always change the availability flags */
nulls[Anum_pg_mv_statistic_deps_built -1] = false;
nulls[Anum_pg_mv_statistic_mcv_built -1] = false;
+ nulls[Anum_pg_mv_statistic_hist_built-1] = false;
nulls[Anum_pg_mv_statistic_stakeys-1] = false;
/* use the new attnums, in case we removed some dropped ones */
replaces[Anum_pg_mv_statistic_deps_built-1] = true;
replaces[Anum_pg_mv_statistic_mcv_built -1] = true;
+ replaces[Anum_pg_mv_statistic_hist_built -1] = true;
replaces[Anum_pg_mv_statistic_stakeys -1] = true;
values[Anum_pg_mv_statistic_deps_built-1] = BoolGetDatum(dependencies != NULL);
values[Anum_pg_mv_statistic_mcv_built -1] = BoolGetDatum(mcvlist != NULL);
+ values[Anum_pg_mv_statistic_hist_built -1] = BoolGetDatum(histogram != NULL);
values[Anum_pg_mv_statistic_stakeys -1] = PointerGetDatum(attrs);
/* Is there already a pg_mv_statistic tuple for this attribute? */
diff --git a/src/backend/utils/mvstats/histogram.c b/src/backend/utils/mvstats/histogram.c
new file mode 100644
index 0000000..6b07b51
--- /dev/null
+++ b/src/backend/utils/mvstats/histogram.c
@@ -0,0 +1,2023 @@
+/*-------------------------------------------------------------------------
+ *
+ * histogram.c
+ * POSTGRES multivariate histograms
+ *
+ *
+ * Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/utils/mvstats/histogram.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "fmgr.h"
+#include "funcapi.h"
+
+#include "utils/lsyscache.h"
+
+#include "common.h"
+#include <math.h>
+
+
+static MVBucket create_initial_mv_bucket(int numrows, HeapTuple *rows,
+ int2vector *attrs,
+ VacAttrStats **stats);
+
+static MVBucket select_bucket_to_partition(int nbuckets, MVBucket * buckets);
+
+static MVBucket partition_bucket(MVBucket bucket, int2vector *attrs,
+ VacAttrStats **stats,
+ int *ndistvalues, Datum **distvalues);
+
+static MVBucket copy_mv_bucket(MVBucket bucket, uint32 ndimensions);
+
+static void update_bucket_ndistinct(MVBucket bucket, int2vector *attrs,
+ VacAttrStats ** stats);
+
+static void update_dimension_ndistinct(MVBucket bucket, int dimension,
+ int2vector *attrs,
+ VacAttrStats ** stats,
+ bool update_boundaries);
+
+static void create_null_buckets(MVHistogram histogram, int bucket_idx,
+ int2vector *attrs, VacAttrStats ** stats);
+
+static Datum * build_ndistinct(int numrows, HeapTuple *rows, int2vector *attrs,
+ VacAttrStats **stats, int i, int *nvals);
+
+/*
+ * Each serialized bucket needs to store (in this order):
+ *
+ * - number of tuples (float)
+ * - number of distinct (float)
+ * - min inclusive flags (ndim * sizeof(bool))
+ * - max inclusive flags (ndim * sizeof(bool))
+ * - null dimension flags (ndim * sizeof(bool))
+ * - min boundary indexes (2 * ndim * sizeof(uint16))
+ * - max boundary indexes (2 * ndim * sizeof(uint16))
+ *
+ * So in total:
+ *
+ * ndim * (4 * sizeof(uint16) + 3 * sizeof(bool)) + (2 * sizeof(float))
+ */
+#define BUCKET_SIZE(ndims) \
+ (ndims * (4 * sizeof(uint16) + 3 * sizeof(bool)) + sizeof(float))
+
+/* pointers into a flat serialized bucket of BUCKET_SIZE(n) bytes */
+#define BUCKET_NTUPLES(b) (*(float*)b)
+#define BUCKET_MIN_INCL(b,n) ((bool*)(b + sizeof(float)))
+#define BUCKET_MAX_INCL(b,n) (BUCKET_MIN_INCL(b,n) + n)
+#define BUCKET_NULLS_ONLY(b,n) (BUCKET_MAX_INCL(b,n) + n)
+#define BUCKET_MIN_INDEXES(b,n) ((uint16*)(BUCKET_NULLS_ONLY(b,n) + n))
+#define BUCKET_MAX_INDEXES(b,n) ((BUCKET_MIN_INDEXES(b,n) + n))
+
+/* can't split bucket with less than 10 rows */
+#define MIN_BUCKET_ROWS 10
+
+/*
+ * Data used while building the histogram.
+ */
+typedef struct HistogramBuildData {
+
+ float ndistinct; /* frequency of distinct values */
+
+ HeapTuple *rows; /* aray of sample rows */
+ uint32 numrows; /* number of sample rows (array size) */
+
+ /*
+ * Number of distinct values in each dimension. This is used when
+ * building the histogram (and is not serialized/deserialized).
+ */
+ uint32 *ndistincts;
+
+} HistogramBuildData;
+
+typedef HistogramBuildData *HistogramBuild;
+
+/*
+ * builds a multivariate algorithm
+ *
+ * The build algorithm is iterative - initially a single bucket containing all
+ * the sample rows is formed, and then repeatedly split into smaller buckets.
+ * In each step the largest bucket (in some sense) is chosen to be split next.
+ *
+ * The criteria for selecting the largest bucket (and the dimension for the
+ * split) needs to be elaborate enough to produce buckets of roughly the same
+ * size, and also regular shape (not very long in one dimension).
+ *
+ * The current algorithm works like this:
+ *
+ * build NULL-buckets (create_null_buckets)
+ *
+ * while [maximum number of buckets not reached]
+ *
+ * choose bucket to partition (largest bucket)
+ * if no bucket to partition
+ * terminate the algorithm
+ *
+ * choose bucket dimension to partition (largest dimension)
+ * split the bucket into two buckets
+ *
+ * See the discussion at select_bucket_to_partition and partition_bucket for
+ * more details about the algorithm.
+ */
+MVHistogram
+build_mv_histogram(int numrows, HeapTuple *rows, int2vector *attrs,
+ VacAttrStats **stats, int numrows_total)
+{
+ int i;
+ int numattrs = attrs->dim1;
+
+ int *ndistvalues;
+ Datum **distvalues;
+
+ MVHistogram histogram;
+
+ HeapTuple * rows_copy = (HeapTuple*)palloc0(numrows * sizeof(HeapTuple));
+ memcpy(rows_copy, rows, sizeof(HeapTuple) * numrows);
+
+ Assert((numattrs >= 2) && (numattrs <= MVSTATS_MAX_DIMENSIONS));
+
+ /* build histogram header */
+
+ histogram = (MVHistogram)palloc0(sizeof(MVHistogramData));
+
+ histogram->magic = MVSTAT_HIST_MAGIC;
+ histogram->type = MVSTAT_HIST_TYPE_BASIC;
+
+ histogram->nbuckets = 1;
+ histogram->ndimensions = numattrs;
+
+ /* create max buckets (better than repalloc for short-lived objects) */
+ histogram->buckets
+ = (MVBucket*)palloc0(MVSTAT_HIST_MAX_BUCKETS * sizeof(MVBucket));
+
+ /* create the initial bucket, covering the whole sample set */
+ histogram->buckets[0]
+ = create_initial_mv_bucket(numrows, rows_copy, attrs, stats);
+
+ /*
+ * Collect info on distinct values in each dimension (used later to select
+ * dimension to partition).
+ */
+ ndistvalues = (int*)palloc0(sizeof(int) * numattrs);
+ distvalues = (Datum**)palloc0(sizeof(Datum*) * numattrs);
+
+ for (i = 0; i < numattrs; i++)
+ distvalues[i] = build_ndistinct(numrows, rows, attrs, stats, i,
+ &ndistvalues[i]);
+
+ /*
+ * Split the initial bucket into buckets that don't mix NULL and non-NULL
+ * values in a single dimension.
+ */
+ create_null_buckets(histogram, 0, attrs, stats);
+
+ /*
+ * Do the actual histogram build - select a bucket and split it.
+ *
+ * FIXME This should use the max_buckets specified in CREATE STATISTICS.
+ */
+ while (histogram->nbuckets < MVSTAT_HIST_MAX_BUCKETS)
+ {
+ MVBucket bucket = select_bucket_to_partition(histogram->nbuckets,
+ histogram->buckets);
+
+ /* no buckets eligible for partitioning */
+ if (bucket == NULL)
+ break;
+
+ /* we modify the bucket in-place and add one new bucket */
+ histogram->buckets[histogram->nbuckets++]
+ = partition_bucket(bucket, attrs, stats, ndistvalues, distvalues);
+ }
+
+ /* finalize the histogram build - compute the frequencies etc. */
+ for (i = 0; i < histogram->nbuckets; i++)
+ {
+ HistogramBuild build_data
+ = ((HistogramBuild)histogram->buckets[i]->build_data);
+
+ /*
+ * The frequency has to be computed from the whole sample, in case some
+ * of the rows were used for MCV.
+ *
+ * XXX Perhaps this should simply compute frequency with respect to the
+ * local freuquency, and then factor-in the MCV later.
+ *
+ * FIXME The 'ntuples' sounds a bit inappropriate for frequency.
+ */
+ histogram->buckets[i]->ntuples
+ = (build_data->numrows * 1.0) / numrows_total;
+ }
+
+ return histogram;
+}
+
+/* build array of distinct values for a single attribute */
+static Datum *
+build_ndistinct(int numrows, HeapTuple *rows, int2vector *attrs,
+ VacAttrStats **stats, int i, int *nvals)
+{
+ int j;
+ int nvalues,
+ ndistinct;
+ Datum *values,
+ *distvalues;
+
+ SortSupportData ssup;
+ StdAnalyzeData *mystats = (StdAnalyzeData *) stats[i]->extra_data;
+
+ /* initialize sort support, etc. */
+ memset(&ssup, 0, sizeof(ssup));
+ ssup.ssup_cxt = CurrentMemoryContext;
+
+ /* We always use the default collation for statistics */
+ ssup.ssup_collation = DEFAULT_COLLATION_OID;
+ ssup.ssup_nulls_first = false;
+
+ PrepareSortSupportFromOrderingOp(mystats->ltopr, &ssup);
+
+ nvalues = 0;
+ values = (Datum*)palloc0(sizeof(Datum) * numrows);
+
+ /* collect values from the sample rows, ignore NULLs */
+ for (j = 0; j < numrows; j++)
+ {
+ Datum value;
+ bool isnull;
+
+ /* remember the index of the sample row, to make the partitioning simpler */
+ value = heap_getattr(rows[j], attrs->values[i],
+ stats[i]->tupDesc, &isnull);
+
+ if (isnull)
+ continue;
+
+ values[nvalues++] = value;
+ }
+
+ /* if no non-NULL values were found, free the memory and terminate */
+ if (nvalues == 0)
+ {
+ pfree(values);
+ return NULL;
+ }
+
+ /* sort the array of values using the SortSupport */
+ qsort_arg((void *) values, nvalues, sizeof(Datum),
+ compare_scalars_simple, (void *) &ssup);
+
+ /* count the distinct values first, and allocate just enough memory */
+ ndistinct = 1;
+ for (j = 1; j < nvalues; j++)
+ if (compare_scalars_simple(&values[j], &values[j-1], &ssup) != 0)
+ ndistinct += 1;
+
+ distvalues = (Datum*)palloc0(sizeof(Datum) * ndistinct);
+
+ /* now collect distinct values into the array */
+ distvalues[0] = values[0];
+ ndistinct = 1;
+
+ for (j = 1; j < nvalues; j++)
+ {
+ if (compare_scalars_simple(&values[j], &values[j-1], &ssup) != 0)
+ {
+ distvalues[ndistinct] = values[j];
+ ndistinct += 1;
+ }
+ }
+
+ pfree(values);
+
+ *nvals = ndistinct;
+ return distvalues;
+}
+
+/* fetch the histogram (as a bytea) from the pg_mv_statistic catalog */
+MVSerializedHistogram
+load_mv_histogram(Oid mvoid)
+{
+ bool isnull = false;
+ Datum histogram;
+
+#ifdef USE_ASSERT_CHECKING
+ Form_pg_mv_statistic mvstat;
+#endif
+
+ /* Prepare to scan pg_mv_statistic for entries having indrelid = this rel. */
+ HeapTuple htup = SearchSysCache1(MVSTATOID, ObjectIdGetDatum(mvoid));
+
+ if (! HeapTupleIsValid(htup))
+ return NULL;
+
+#ifdef USE_ASSERT_CHECKING
+ mvstat = (Form_pg_mv_statistic) GETSTRUCT(htup);
+ Assert(mvstat->hist_enabled && mvstat->hist_built);
+#endif
+
+ histogram = SysCacheGetAttr(MVSTATOID, htup,
+ Anum_pg_mv_statistic_stahist, &isnull);
+
+ Assert(!isnull);
+
+ ReleaseSysCache(htup);
+
+ return deserialize_mv_histogram(DatumGetByteaP(histogram));
+}
+
+/* print some basic info about the histogram */
+Datum
+pg_mv_stats_histogram_info(PG_FUNCTION_ARGS)
+{
+ bytea *data = PG_GETARG_BYTEA_P(0);
+ char *result;
+
+ MVSerializedHistogram hist = deserialize_mv_histogram(data);
+
+ result = palloc0(128);
+ snprintf(result, 128, "nbuckets=%d", hist->nbuckets);
+
+ PG_RETURN_TEXT_P(cstring_to_text(result));
+}
+
+/*
+ * Serialize the MV histogram into a bytea value. The basic algorithm is quite
+ * simple, and mostly mimincs the MCV serialization:
+ *
+ * (1) perform deduplication for each attribute (separately)
+ *
+ * (a) collect all (non-NULL) attribute values from all buckets
+ * (b) sort the data (using 'lt' from VacAttrStats)
+ * (c) remove duplicate values from the array
+ *
+ * (2) serialize the arrays into a bytea value
+ *
+ * (3) process all buckets
+ *
+ * (a) replace min/max values with indexes into the arrays
+ *
+ * Each attribute has to be processed separately, as we're mixing different
+ * datatypes, and we we need to use the right operators to compare/sort them.
+ * We're also mixing pass-by-value and pass-by-ref types, and so on.
+ *
+ *
+ * FIXME This probably leaks memory, or at least uses it inefficiently
+ * (many small palloc() calls instead of a large one).
+ *
+ * TODO Consider packing boolean flags (NULL) for each item into 'char'
+ * or a longer type (instead of using an array of bool items).
+ */
+bytea *
+serialize_mv_histogram(MVHistogram histogram, int2vector *attrs,
+ VacAttrStats **stats)
+{
+ int i = 0, j = 0;
+ Size total_length = 0;
+
+ bytea *output = NULL;
+ char *data = NULL;
+
+ DimensionInfo *info;
+ SortSupport ssup;
+
+ int nbuckets = histogram->nbuckets;
+ int ndims = histogram->ndimensions;
+
+ /* allocated for serialized bucket data */
+ int bucketsize = BUCKET_SIZE(ndims);
+ char *bucket = palloc0(bucketsize);
+
+ /* values per dimension (and number of non-NULL values) */
+ Datum **values = (Datum**)palloc0(sizeof(Datum*) * ndims);
+ int *counts = (int*)palloc0(sizeof(int) * ndims);
+
+ /* info about dimensions (for deserialize) */
+ info = (DimensionInfo *)palloc0(sizeof(DimensionInfo)*ndims);
+
+ /* sort support data */
+ ssup = (SortSupport)palloc0(sizeof(SortSupportData)*ndims);
+
+ /* collect and deduplicate values for each dimension separately */
+ for (i = 0; i < ndims; i++)
+ {
+ int count;
+ StdAnalyzeData *tmp = (StdAnalyzeData *)stats[i]->extra_data;
+
+ /* keep important info about the data type */
+ info[i].typlen = stats[i]->attrtype->typlen;
+ info[i].typbyval = stats[i]->attrtype->typbyval;
+
+ /*
+ * Allocate space for all min/max values, including NULLs (we won't use
+ * them, but we don't know how many are there), and then collect all
+ * non-NULL values.
+ */
+ values[i] = (Datum*)palloc0(sizeof(Datum) * nbuckets * 2);
+
+ for (j = 0; j < histogram->nbuckets; j++)
+ {
+ /* skip buckets where this dimension is NULL-only */
+ if (! histogram->buckets[j]->nullsonly[i])
+ {
+ values[i][counts[i]] = histogram->buckets[j]->min[i];
+ counts[i] += 1;
+
+ values[i][counts[i]] = histogram->buckets[j]->max[i];
+ counts[i] += 1;
+ }
+ }
+
+ /* there are just NULL values in this dimension */
+ if (counts[i] == 0)
+ continue;
+
+ /* sort and deduplicate */
+ ssup[i].ssup_cxt = CurrentMemoryContext;
+ ssup[i].ssup_collation = DEFAULT_COLLATION_OID;
+ ssup[i].ssup_nulls_first = false;
+
+ PrepareSortSupportFromOrderingOp(tmp->ltopr, &ssup[i]);
+
+ qsort_arg(values[i], counts[i], sizeof(Datum),
+ compare_scalars_simple, &ssup[i]);
+
+ /*
+ * Walk through the array and eliminate duplicitate values, but
+ * keep the ordering (so that we can do bsearch later). We know
+ * there's at least 1 item, so we can skip the first element.
+ */
+ count = 1; /* number of deduplicated items */
+ for (j = 1; j < counts[i]; j++)
+ {
+ /* if it's different from the previous value, we need to keep it */
+ if (compare_datums_simple(values[i][j-1], values[i][j], &ssup[i]) != 0)
+ {
+ /* XXX: not needed if (count == j) */
+ values[i][count] = values[i][j];
+ count += 1;
+ }
+ }
+
+ /* make sure we fit into uint16 */
+ Assert(count <= UINT16_MAX);
+
+ /* keep info about the deduplicated count */
+ info[i].nvalues = count;
+
+ /* compute size of the serialized data */
+ if (info[i].typlen > 0)
+ /* byval or byref, but with fixed length (name, tid, ...) */
+ info[i].nbytes = info[i].nvalues * info[i].typlen;
+ else if (info[i].typlen == -1)
+ /* varlena, so just use VARSIZE_ANY */
+ for (j = 0; j < info[i].nvalues; j++)
+ info[i].nbytes += VARSIZE_ANY(values[i][j]);
+ else if (info[i].typlen == -2)
+ /* cstring, so simply strlen */
+ for (j = 0; j < info[i].nvalues; j++)
+ info[i].nbytes += strlen(DatumGetPointer(values[i][j]));
+ else
+ elog(ERROR, "unknown data type typbyval=%d typlen=%d",
+ info[i].typbyval, info[i].typlen);
+ }
+
+ /*
+ * Now we finally know how much space we'll need for the serialized
+ * histogram, as it contains these fields:
+ *
+ * - length (4B) for varlena
+ * - magic (4B)
+ * - type (4B)
+ * - ndimensions (4B)
+ * - nbuckets (4B)
+ * - info (ndim * sizeof(DimensionInfo)
+ * - arrays of values for each dimension
+ * - serialized buckets (nbuckets * bucketsize)
+ *
+ * So the 'header' size is 20B + ndim * sizeof(DimensionInfo) and
+ * then we'll place the data (and buckets).
+ */
+ total_length = (sizeof(int32) + offsetof(MVHistogramData, buckets)
+ + ndims * sizeof(DimensionInfo)
+ + nbuckets * bucketsize);
+
+ /* account for the deduplicated data */
+ for (i = 0; i < ndims; i++)
+ total_length += info[i].nbytes;
+
+ /* enforce arbitrary limit of 1MB */
+ if (total_length > (1024 * 1024))
+ elog(ERROR, "serialized histogram exceeds 1MB (%ld > %d)",
+ total_length, (1024 * 1024));
+
+ /* allocate space for the serialized histogram list, set header */
+ output = (bytea*)palloc0(total_length);
+ SET_VARSIZE(output, total_length);
+
+ /* we'll use 'data' to keep track of the place to write data */
+ data = VARDATA(output);
+
+ memcpy(data, histogram, offsetof(MVHistogramData, buckets));
+ data += offsetof(MVHistogramData, buckets);
+
+ memcpy(data, info, sizeof(DimensionInfo) * ndims);
+ data += sizeof(DimensionInfo) * ndims;
+
+ /* serialize the deduplicated values for all attributes */
+ for (i = 0; i < ndims; i++)
+ {
+#ifdef USE_ASSERT_CHECKING
+ char *tmp = data;
+#endif
+ for (j = 0; j < info[i].nvalues; j++)
+ {
+ Datum v = values[i][j];
+
+ if (info[i].typbyval) /* passed by value */
+ {
+ memcpy(data, &v, info[i].typlen);
+ data += info[i].typlen;
+ }
+ else if (info[i].typlen > 0) /* pased by reference */
+ {
+ memcpy(data, DatumGetPointer(v), info[i].typlen);
+ data += info[i].typlen;
+ }
+ else if (info[i].typlen == -1) /* varlena */
+ {
+ memcpy(data, DatumGetPointer(v), VARSIZE_ANY(v));
+ data += VARSIZE_ANY(values[i][j]);
+ }
+ else if (info[i].typlen == -2) /* cstring */
+ {
+ memcpy(data, DatumGetPointer(v), strlen(DatumGetPointer(v))+1);
+ data += strlen(DatumGetPointer(v)) + 1;
+ }
+ }
+
+ /* make sure we got exactly the amount of data we expected */
+ Assert((data - tmp) == info[i].nbytes);
+ }
+
+ /* finally serialize the items, with uint16 indexes instead of the values */
+ for (i = 0; i < nbuckets; i++)
+ {
+ /* don't write beyond the allocated space */
+ Assert(data <= (char*)output + total_length - bucketsize);
+
+ /* reset the values for each item */
+ memset(bucket, 0, bucketsize);
+
+ BUCKET_NTUPLES(bucket) = histogram->buckets[i]->ntuples;
+
+ for (j = 0; j < ndims; j++)
+ {
+ /* do the lookup only for non-NULL values */
+ if (! histogram->buckets[i]->nullsonly[j])
+ {
+ uint16 idx;
+ Datum * v = NULL;
+
+ /* min boundary */
+ v = (Datum*)bsearch_arg(&histogram->buckets[i]->min[j],
+ values[j], info[j].nvalues, sizeof(Datum),
+ compare_scalars_simple, &ssup[j]);
+
+ Assert(v != NULL); /* serialization or deduplication error */
+
+ /* compute index within the array */
+ idx = (v - values[j]);
+
+ Assert((idx >= 0) && (idx < info[j].nvalues));
+
+ BUCKET_MIN_INDEXES(bucket, ndims)[j] = idx;
+
+ /* max boundary */
+ v = (Datum*)bsearch_arg(&histogram->buckets[i]->max[j],
+ values[j], info[j].nvalues, sizeof(Datum),
+ compare_scalars_simple, &ssup[j]);
+
+ Assert(v != NULL); /* serialization or deduplication error */
+
+ /* compute index within the array */
+ idx = (v - values[j]);
+
+ Assert((idx >= 0) && (idx < info[j].nvalues));
+
+ BUCKET_MAX_INDEXES(bucket, ndims)[j] = idx;
+ }
+ }
+
+ /* copy flags (nulls, min/max inclusive) */
+ memcpy(BUCKET_NULLS_ONLY(bucket, ndims),
+ histogram->buckets[i]->nullsonly, sizeof(bool) * ndims);
+
+ memcpy(BUCKET_MIN_INCL(bucket, ndims),
+ histogram->buckets[i]->min_inclusive, sizeof(bool) * ndims);
+
+ memcpy(BUCKET_MAX_INCL(bucket, ndims),
+ histogram->buckets[i]->max_inclusive, sizeof(bool) * ndims);
+
+ /* copy the item into the array */
+ memcpy(data, bucket, bucketsize);
+
+ data += bucketsize;
+ }
+
+ /* at this point we expect to match the total_length exactly */
+ Assert((data - (char*)output) == total_length);
+
+ /* free the values/counts arrays here */
+ pfree(counts);
+ pfree(info);
+ pfree(ssup);
+
+ for (i = 0; i < ndims; i++)
+ pfree(values[i]);
+
+ pfree(values);
+
+ return output;
+}
+
+/*
+ * Returns histogram in a partially-serialized form (keeps the boundary values
+ * deduplicated, so that it's possible to optimize the estimation part by
+ * caching function call results between buckets etc.).
+ */
+MVSerializedHistogram
+deserialize_mv_histogram(bytea * data)
+{
+ int i = 0, j = 0;
+
+ Size expected_size;
+ char *tmp = NULL;
+
+ MVSerializedHistogram histogram;
+ DimensionInfo *info;
+
+ int nbuckets;
+ int ndims;
+ int bucketsize;
+
+ /* temporary deserialization buffer */
+ int bufflen;
+ char *buff;
+ char *ptr;
+
+ if (data == NULL)
+ return NULL;
+
+ if (VARSIZE_ANY_EXHDR(data) < offsetof(MVSerializedHistogramData,buckets))
+ elog(ERROR, "invalid histogram size %ld (expected at least %ld)",
+ VARSIZE_ANY_EXHDR(data), offsetof(MVSerializedHistogramData,buckets));
+
+ /* read the histogram header */
+ histogram
+ = (MVSerializedHistogram)palloc(sizeof(MVSerializedHistogramData));
+
+ /* initialize pointer to the data part (skip the varlena header) */
+ tmp = VARDATA(data);
+
+ /* get the header and perform basic sanity checks */
+ memcpy(histogram, tmp, offsetof(MVSerializedHistogramData, buckets));
+ tmp += offsetof(MVSerializedHistogramData, buckets);
+
+ if (histogram->magic != MVSTAT_HIST_MAGIC)
+ elog(ERROR, "invalid histogram magic %d (expected %dd)",
+ histogram->magic, MVSTAT_HIST_MAGIC);
+
+ if (histogram->type != MVSTAT_HIST_TYPE_BASIC)
+ elog(ERROR, "invalid histogram type %d (expected %dd)",
+ histogram->type, MVSTAT_HIST_TYPE_BASIC);
+
+ nbuckets = histogram->nbuckets;
+ ndims = histogram->ndimensions;
+ bucketsize = BUCKET_SIZE(ndims);
+
+ Assert((nbuckets > 0) && (nbuckets <= MVSTAT_HIST_MAX_BUCKETS));
+ Assert((ndims >= 2) && (ndims <= MVSTATS_MAX_DIMENSIONS));
+
+ /*
+ * What size do we expect with those parameters (it's incomplete, as we yet
+ * have to count the array sizes (from DimensionInfo records).
+ */
+ expected_size = offsetof(MVSerializedHistogramData,buckets) +
+ ndims * sizeof(DimensionInfo) +
+ (nbuckets * bucketsize);
+
+ /* check that we have at least the DimensionInfo records */
+ if (VARSIZE_ANY_EXHDR(data) < expected_size)
+ elog(ERROR, "invalid histogram size %ld (expected %ld)",
+ VARSIZE_ANY_EXHDR(data), expected_size);
+
+ info = (DimensionInfo*)(tmp);
+ tmp += ndims * sizeof(DimensionInfo);
+
+ /* account for the value arrays */
+ for (i = 0; i < ndims; i++)
+ expected_size += info[i].nbytes;
+
+ if (VARSIZE_ANY_EXHDR(data) != expected_size)
+ elog(ERROR, "invalid histogram size %ld (expected %ld)",
+ VARSIZE_ANY_EXHDR(data), expected_size);
+
+ /* looks OK - not corrupted or something */
+
+ /* a single buffer for all the values and counts */
+ bufflen = (sizeof(int) + sizeof(Datum*)) * ndims;
+
+ for (i = 0; i < ndims; i++)
+ /* don't allocate space for byval types, matching Datum */
+ if (! (info[i].typbyval && (info[i].typlen == sizeof(Datum))))
+ bufflen += (sizeof(Datum) * info[i].nvalues);
+
+ /* also, include space for the result, tracking the buckets */
+ bufflen += nbuckets * (
+ sizeof(MVSerializedBucket) + /* bucket pointer */
+ sizeof(MVSerializedBucketData)); /* bucket data */
+
+ buff = palloc0(bufflen);
+ ptr = buff;
+
+ histogram->nvalues = (int*)ptr;
+ ptr += (sizeof(int) * ndims);
+
+ histogram->values = (Datum**)ptr;
+ ptr += (sizeof(Datum*) * ndims);
+
+ /*
+ * FIXME This uses pointers to the original data array (the types
+ * not passed by value), so when someone frees the memory,
+ * e.g. by doing something like this:
+ *
+ * bytea * data = ... fetch the data from catalog ...
+ * MCVList mcvlist = deserialize_mcv_list(data);
+ * pfree(data);
+ *
+ * then 'mcvlist' references the freed memory. This needs to
+ * copy the pieces.
+ *
+ * TODO same as in MCV deserialization / consider moving to common.c
+ */
+ for (i = 0; i < ndims; i++)
+ {
+ histogram->nvalues[i] = info[i].nvalues;
+
+ if (info[i].typbyval)
+ {
+ /* passed by value / Datum - simply reuse the array */
+ if (info[i].typlen == sizeof(Datum))
+ {
+ histogram->values[i] = (Datum*)tmp;
+ tmp += info[i].nbytes;
+ }
+ else
+ {
+ histogram->values[i] = (Datum*)ptr;
+ ptr += (sizeof(Datum) * info[i].nvalues);
+
+ for (j = 0; j < info[i].nvalues; j++)
+ {
+ /* just point into the array */
+ memcpy(&histogram->values[i][j], tmp, info[i].typlen);
+ tmp += info[i].typlen;
+ }
+ }
+ }
+ else
+ {
+ /* all the other types need a chunk of the buffer */
+ histogram->values[i] = (Datum*)ptr;
+ ptr += (sizeof(Datum) * info[i].nvalues);
+
+ if (info[i].typlen > 0)
+ {
+ /* pased by reference, but fixed length (name, tid, ...) */
+ for (j = 0; j < info[i].nvalues; j++)
+ {
+ /* just point into the array */
+ histogram->values[i][j] = PointerGetDatum(tmp);
+ tmp += info[i].typlen;
+ }
+ }
+ else if (info[i].typlen == -1)
+ {
+ /* varlena */
+ for (j = 0; j < info[i].nvalues; j++)
+ {
+ /* just point into the array */
+ histogram->values[i][j] = PointerGetDatum(tmp);
+ tmp += VARSIZE_ANY(tmp);
+ }
+ }
+ else if (info[i].typlen == -2)
+ {
+ /* cstring */
+ for (j = 0; j < info[i].nvalues; j++)
+ {
+ /* just point into the array */
+ histogram->values[i][j] = PointerGetDatum(tmp);
+ tmp += (strlen(tmp) + 1); /* don't forget the \0 */
+ }
+ }
+ }
+ }
+
+ histogram->buckets = (MVSerializedBucket*)ptr;
+ ptr += (sizeof(MVSerializedBucket) * nbuckets);
+
+ for (i = 0; i < nbuckets; i++)
+ {
+ MVSerializedBucket bucket = (MVSerializedBucket)ptr;
+ ptr += sizeof(MVSerializedBucketData);
+
+ bucket->ntuples = BUCKET_NTUPLES(tmp);
+ bucket->nullsonly = BUCKET_NULLS_ONLY(tmp, ndims);
+ bucket->min_inclusive = BUCKET_MIN_INCL(tmp, ndims);
+ bucket->max_inclusive = BUCKET_MAX_INCL(tmp, ndims);
+
+ bucket->min = BUCKET_MIN_INDEXES(tmp, ndims);
+ bucket->max = BUCKET_MAX_INDEXES(tmp, ndims);
+
+ histogram->buckets[i] = bucket;
+
+ Assert(tmp <= (char*)data + VARSIZE_ANY(data));
+
+ tmp += bucketsize;
+ }
+
+ /* at this point we expect to match the total_length exactly */
+ Assert((tmp - VARDATA(data)) == expected_size);
+
+ /* we should exhaust the output buffer exactly */
+ Assert((ptr - buff) == bufflen);
+
+ return histogram;
+}
+
+/*
+ * Build the initial bucket, which will be then split into smaller ones.
+ */
+static MVBucket
+create_initial_mv_bucket(int numrows, HeapTuple *rows, int2vector *attrs,
+ VacAttrStats **stats)
+{
+ int i;
+ int numattrs = attrs->dim1;
+ HistogramBuild data = NULL;
+
+ /* TODO allocate bucket as a single piece, including all the fields. */
+ MVBucket bucket = (MVBucket)palloc0(sizeof(MVBucketData));
+
+ Assert(numrows > 0);
+ Assert(rows != NULL);
+ Assert((numattrs >= 2) && (numattrs <= MVSTATS_MAX_DIMENSIONS));
+
+ /* allocate the per-dimension arrays */
+
+ /* flags for null-only dimensions */
+ bucket->nullsonly = (bool*)palloc0(numattrs * sizeof(bool));
+
+ /* inclusiveness boundaries - lower/upper bounds */
+ bucket->min_inclusive = (bool*)palloc0(numattrs * sizeof(bool));
+ bucket->max_inclusive = (bool*)palloc0(numattrs * sizeof(bool));
+
+ /* lower/upper boundaries */
+ bucket->min = (Datum*)palloc0(numattrs * sizeof(Datum));
+ bucket->max = (Datum*)palloc0(numattrs * sizeof(Datum));
+
+ /* build-data */
+ data = (HistogramBuild)palloc0(sizeof(HistogramBuildData));
+
+ /* number of distinct values (per dimension) */
+ data->ndistincts = (uint32*)palloc0(numattrs * sizeof(uint32));
+
+ /* all the sample rows fall into the initial bucket */
+ data->numrows = numrows;
+ data->rows = rows;
+
+ bucket->build_data = data;
+
+ /*
+ * Update the number of ndistinct combinations in the bucket (which we use
+ * when selecting bucket to partition), and then number of distinct values
+ * for each partition (which we use when choosing which dimension to split).
+ */
+ update_bucket_ndistinct(bucket, attrs, stats);
+
+ /* Update ndistinct (and also set min/max) for all dimensions. */
+ for (i = 0; i < numattrs; i++)
+ update_dimension_ndistinct(bucket, i, attrs, stats, true);
+
+ return bucket;
+}
+
+/*
+ * Choose the bucket to partition next.
+ *
+ * The current criteria is rather simple, chosen so that the algorithm produces
+ * buckets with about equal frequency and regular size. We select the bucket
+ * with the highest number of distinct values, and then split it by the longest
+ * dimension.
+ *
+ * The distinct values are uniformly mapped to [0,1] interval, and this is used
+ * to compute length of the value range.
+ *
+ * NOTE: This is not the same array used for deduplication, as this contains
+ * values for all the tuples from the sample, not just the boundary values.
+ *
+ * Returns either pointer to the bucket selected to be partitioned, or NULL if
+ * there are no buckets that may be split (e.g. if all buckets are too small
+ * or contain too few distinct values).
+ *
+ *
+ * Tricky example
+ * --------------
+ *
+ * Consider this table:
+ *
+ * CREATE TABLE t AS SELECT i AS a, i AS b
+ * FROM generate_series(1,1000000) s(i);
+ *
+ * CREATE STATISTICS s1 ON t (a,b) WITH (histogram);
+ *
+ * ANALYZE t;
+ *
+ * It's a very specific (and perhaps artificial) example, because every bucket
+ * always has exactly the same number of distinct values in all dimensions,
+ * which makes the partitioning tricky.
+ *
+ * Then:
+ *
+ * SELECT * FROM t WHERE (a < 100) AND (b < 100);
+ *
+ * is estimated to return ~120 rows, while in reality it returns only 99.
+ *
+ * QUERY PLAN
+ * -------------------------------------------------------------
+ * Seq Scan on t (cost=0.00..19425.00 rows=117 width=8)
+ * (actual time=0.129..82.776 rows=99 loops=1)
+ * Filter: ((a < 100) AND (b < 100))
+ * Rows Removed by Filter: 999901
+ * Planning time: 1.286 ms
+ * Execution time: 82.984 ms
+ * (5 rows)
+ *
+ * So this estimate is reasonably close. Let's change the query to OR clause:
+ *
+ * SELECT * FROM t WHERE (a < 100) OR (b < 100);
+ *
+ * QUERY PLAN
+ * -------------------------------------------------------------
+ * Seq Scan on t (cost=0.00..19425.00 rows=8100 width=8)
+ * (actual time=0.145..99.910 rows=99 loops=1)
+ * Filter: ((a < 100) OR (b < 100))
+ * Rows Removed by Filter: 999901
+ * Planning time: 1.578 ms
+ * Execution time: 100.132 ms
+ * (5 rows)
+ *
+ * That's clearly a much worse estimate. This happens because the histogram
+ * contains buckets like this:
+ *
+ * bucket 592 [3 30310] [30134 30593] => [0.000233]
+ *
+ * i.e. the length of "a" dimension is (30310-3)=30307, while the length of "b"
+ * is (30593-30134)=459. So the "b" dimension is much narrower than "a".
+ * Of course, there are also buckets where "b" is the wider dimension.
+ *
+ * This is partially mitigated by selecting the "longest" dimension but that
+ * only happens after we already selected the bucket. So if we never select the
+ * bucket, this optimization does not apply.
+ *
+ * The other reason why this particular example behaves so poorly is due to the
+ * way we actually split the selected bucket. We do attempt to divide the bucket
+ * into two parts containing about the same number of tuples, but that does not
+ * too well when most of the tuples is squashed on one side of the bucket.
+ *
+ * For example for columns with data on the diagonal (i.e. when a=b), we end up
+ * with a narrow bucket on the diagonal and a huge bucket overing the remaining
+ * part (with much lower density).
+ *
+ * So perhaps we need two partitioning strategies - one aiming to split buckets
+ * with high frequency (number of sampled rows), the other aiming to split
+ * "large" buckets. And alternating between them, somehow.
+ *
+ * TODO Consider using similar lower boundary for row count as for simple
+ * histograms, i.e. 300 tuples per bucket.
+ */
+static MVBucket
+select_bucket_to_partition(int nbuckets, MVBucket * buckets)
+{
+ int i;
+ int numrows = 0;
+ MVBucket bucket = NULL;
+
+ for (i = 0; i < nbuckets; i++)
+ {
+ HistogramBuild data = (HistogramBuild)buckets[i]->build_data;
+
+ /* if the number of rows is higher, use this bucket */
+ if ((data->ndistinct > 2) &&
+ (data->numrows > numrows) &&
+ (data->numrows >= MIN_BUCKET_ROWS)) {
+ bucket = buckets[i];
+ numrows = data->numrows;
+ }
+ }
+
+ /* may be NULL if there are not buckets with (ndistinct>1) */
+ return bucket;
+}
+
+/*
+ * A simple bucket partitioning implementation - we choose the longest bucket
+ * dimension, measured using the array of distinct values built at the very
+ * beginning of the build.
+ *
+ * We map all the distinct values to a [0,1] interval, uniformly distributed,
+ * and then use this to measure length. It's essentially a number of distinct
+ * values within the range, normalized to [0,1].
+ *
+ * Then we choose a 'middle' value splitting the bucket into two parts with
+ * roughly the same frequency.
+ *
+ * This splits the bucket by tweaking the existing one, and returning the new
+ * bucket (essentially shrinking the existing one in-place and returning the
+ * other "half" as a new bucket). The caller is responsible for adding the new
+ * bucket into the list of buckets.
+ *
+ * There are multiple histogram options, centered around the partitioning
+ * criteria, specifying both how to choose a bucket and the dimension most in
+ * need of a split. For a nice summary and general overview, see "rK-Hist : an
+ * R-Tree based histogram for multi-dimensional selectivity estimation" thesis
+ * by J. A. Lopez, Concordia University, p.34-37 (and possibly p. 32-34 for
+ * explanation of the terms).
+ *
+ * It requires care to prevent splitting only one dimension and not splitting
+ * another one at all (which might happen easily in case of strongly dependent
+ * columns - e.g. y=x). The current algorithm minimizes this, but may still
+ * happen for perfectly dependent examples (when all the dimensions have equal
+ * length, the first one will be selected).
+ *
+ * TODO Should probably consider statistics target for the columns (e.g.
+ * to split dimensions with higher statistics target more frequently).
+ */
+static MVBucket
+partition_bucket(MVBucket bucket, int2vector *attrs,
+ VacAttrStats **stats,
+ int *ndistvalues, Datum **distvalues)
+{
+ int i;
+ int dimension;
+ int numattrs = attrs->dim1;
+
+ Datum split_value;
+ MVBucket new_bucket;
+ HistogramBuild new_data;
+
+ /* needed for sort, when looking for the split value */
+ bool isNull;
+ int nvalues = 0;
+ HistogramBuild data = (HistogramBuild)bucket->build_data;
+ StdAnalyzeData * mystats = NULL;
+ ScalarItem * values = (ScalarItem*)palloc0(data->numrows * sizeof(ScalarItem));
+ SortSupportData ssup;
+
+ int nrows = 1; /* number of rows below current value */
+ double delta;
+
+ /* needed when splitting the values */
+ HeapTuple * oldrows = data->rows;
+ int oldnrows = data->numrows;
+
+ /*
+ * We can't split buckets with a single distinct value (this also
+ * disqualifies NULL-only dimensions). Also, there has to be multiple
+ * sample rows (otherwise, how could there be more distinct values).
+ */
+ Assert(data->ndistinct > 1);
+ Assert(data->numrows > 1);
+ Assert((numattrs >= 2) && (numattrs <= MVSTATS_MAX_DIMENSIONS));
+
+ /* Look for the next dimension to split. */
+ delta = 0.0;
+ dimension = -1;
+
+ for (i = 0; i < numattrs; i++)
+ {
+ Datum *a, *b;
+
+ mystats = (StdAnalyzeData *) stats[i]->extra_data;
+
+ /* initialize sort support, etc. */
+ memset(&ssup, 0, sizeof(ssup));
+ ssup.ssup_cxt = CurrentMemoryContext;
+
+ /* We always use the default collation for statistics */
+ ssup.ssup_collation = DEFAULT_COLLATION_OID;
+ ssup.ssup_nulls_first = false;
+
+ PrepareSortSupportFromOrderingOp(mystats->ltopr, &ssup);
+
+ /* can't split NULL-only dimension */
+ if (bucket->nullsonly[i])
+ continue;
+
+ /* can't split dimension with a single ndistinct value */
+ if (data->ndistincts[i] <= 1)
+ continue;
+
+ /* search for min boundary in the distinct list */
+ a = (Datum*)bsearch_arg(&bucket->min[i],
+ distvalues[i], ndistvalues[i],
+ sizeof(Datum), compare_scalars_simple, &ssup);
+
+ b = (Datum*)bsearch_arg(&bucket->max[i],
+ distvalues[i], ndistvalues[i],
+ sizeof(Datum), compare_scalars_simple, &ssup);
+
+ /* if this dimension is 'larger' then partition by it */
+ if (((b-a)*1.0 / ndistvalues[i]) > delta)
+ {
+ delta = ((b-a)*1.0 / ndistvalues[i]);
+ dimension = i;
+ }
+ }
+
+ /*
+ * If we haven't found a dimension here, we've done something
+ * wrong in select_bucket_to_partition.
+ */
+ Assert(dimension != -1);
+
+ /*
+ * Walk through the selected dimension, collect and sort the values and
+ * then choose the value to use as the new boundary.
+ */
+ mystats = (StdAnalyzeData *) stats[dimension]->extra_data;
+
+ /* initialize sort support, etc. */
+ memset(&ssup, 0, sizeof(ssup));
+ ssup.ssup_cxt = CurrentMemoryContext;
+
+ /* We always use the default collation for statistics */
+ ssup.ssup_collation = DEFAULT_COLLATION_OID;
+ ssup.ssup_nulls_first = false;
+
+ PrepareSortSupportFromOrderingOp(mystats->ltopr, &ssup);
+
+ for (i = 0; i < data->numrows; i++)
+ {
+ /* remember the index of the sample row, to make the partitioning simpler */
+ values[nvalues].value = heap_getattr(data->rows[i], attrs->values[dimension],
+ stats[dimension]->tupDesc, &isNull);
+ values[nvalues].tupno = i;
+
+ /* no NULL values allowed here (we never split null-only dimension) */
+ Assert(!isNull);
+
+ nvalues++;
+ }
+
+ /* sort the array of values */
+ qsort_arg((void *) values, nvalues, sizeof(ScalarItem),
+ compare_scalars_partition, (void *) &ssup);
+
+ /*
+ * We know there are bucket->ndistincts[dimension] distinct values in this
+ * dimension, and we want to split this into half, so walk through the
+ * array and stop once we see (ndistinct/2) values.
+ *
+ * We always choose the "next" value, i.e. (n/2+1)-th distinct value, and
+ * use it as an exclusive upper boundary (and inclusive lower boundary).
+ *
+ * TODO Maybe we should use "average" of the two middle distinct values
+ * (at least for even distinct counts), but that would require being
+ * able to do an average (which does not work for non-numeric types).
+ *
+ * TODO Another option is to look for a split that'd give about 50% tuples
+ * (not distinct values) in each partition. That might work better
+ * when there are a few very frequent values, and many rare ones.
+ */
+ delta = fabs(data->numrows);
+ split_value = values[0].value;
+
+ for (i = 1; i < data->numrows; i++)
+ {
+ if (values[i].value != values[i-1].value)
+ {
+ /* are we closer to splitting the bucket in half? */
+ if (fabs(i - data->numrows/2.0) < delta)
+ {
+ /* let's assume we'll use this value for the split */
+ split_value = values[i].value;
+ delta = fabs(i - data->numrows/2.0);
+ nrows = i;
+ }
+ }
+ }
+
+ Assert(nrows > 0);
+ Assert(nrows < data->numrows);
+
+ /* create the new bucket as a (incomplete) copy of the one being partitioned. */
+ new_bucket = copy_mv_bucket(bucket, numattrs);
+ new_data = (HistogramBuild)new_bucket->build_data;
+
+ /*
+ * Do the actual split of the chosen dimension, using the split value as the
+ * upper bound for the existing bucket, and lower bound for the new one.
+ */
+ bucket->max[dimension] = split_value;
+ new_bucket->min[dimension] = split_value;
+
+ bucket->max_inclusive[dimension] = false;
+ new_bucket->max_inclusive[dimension] = true;
+
+ /*
+ * Redistribute the sample tuples using the 'ScalarItem->tupno' index. We
+ * know 'nrows' rows should remain in the original bucket and the rest goes
+ * to the new one.
+ */
+
+ data->rows = (HeapTuple*)palloc0(nrows * sizeof(HeapTuple));
+ new_data->rows = (HeapTuple*)palloc0((oldnrows - nrows) * sizeof(HeapTuple));
+
+ data->numrows = nrows;
+ new_data->numrows = (oldnrows - nrows);
+
+ /*
+ * The first nrows should go to the first bucket, the rest should go to the
+ * new one. Use the tupno field to get the actual HeapTuple row from the
+ * original array of sample rows.
+ */
+ for (i = 0; i < nrows; i++)
+ memcpy(&data->rows[i], &oldrows[values[i].tupno], sizeof(HeapTuple));
+
+ for (i = nrows; i < oldnrows; i++)
+ memcpy(&new_data->rows[i-nrows], &oldrows[values[i].tupno], sizeof(HeapTuple));
+
+ /* update ndistinct values for the buckets (total and per dimension) */
+ update_bucket_ndistinct(bucket, attrs, stats);
+ update_bucket_ndistinct(new_bucket, attrs, stats);
+
+ /*
+ * TODO We don't need to do this for the dimension we used for split,
+ * because we know how many distinct values went to each partition.
+ */
+ for (i = 0; i < numattrs; i++)
+ {
+ update_dimension_ndistinct(bucket, i, attrs, stats, false);
+ update_dimension_ndistinct(new_bucket, i, attrs, stats, false);
+ }
+
+ pfree(oldrows);
+ pfree(values);
+
+ return new_bucket;
+}
+
+/*
+ * Copy a histogram bucket. The copy does not include the build-time data, i.e.
+ * sampled rows etc.
+ */
+static MVBucket
+copy_mv_bucket(MVBucket bucket, uint32 ndimensions)
+{
+ /* TODO allocate as a single piece (including all the fields) */
+ MVBucket new_bucket = (MVBucket)palloc0(sizeof(MVBucketData));
+ HistogramBuild data = (HistogramBuild)palloc0(sizeof(HistogramBuildData));
+
+ /* Copy only the attributes that will stay the same after the split, and
+ * we'll recompute the rest after the split. */
+
+ /* allocate the per-dimension arrays */
+ new_bucket->nullsonly = (bool*)palloc0(ndimensions * sizeof(bool));
+
+ /* inclusiveness boundaries - lower/upper bounds */
+ new_bucket->min_inclusive = (bool*)palloc0(ndimensions * sizeof(bool));
+ new_bucket->max_inclusive = (bool*)palloc0(ndimensions * sizeof(bool));
+
+ /* lower/upper boundaries */
+ new_bucket->min = (Datum*)palloc0(ndimensions * sizeof(Datum));
+ new_bucket->max = (Datum*)palloc0(ndimensions * sizeof(Datum));
+
+ /* copy data */
+ memcpy(new_bucket->nullsonly, bucket->nullsonly, ndimensions * sizeof(bool));
+
+ memcpy(new_bucket->min_inclusive, bucket->min_inclusive, ndimensions*sizeof(bool));
+ memcpy(new_bucket->min, bucket->min, ndimensions*sizeof(Datum));
+
+ memcpy(new_bucket->max_inclusive, bucket->max_inclusive, ndimensions*sizeof(bool));
+ memcpy(new_bucket->max, bucket->max, ndimensions*sizeof(Datum));
+
+ /* allocate and copy the interesting part of the build data */
+ data->ndistincts = (uint32*)palloc0(ndimensions * sizeof(uint32));
+
+ new_bucket->build_data = data;
+
+ return new_bucket;
+}
+
+/*
+ * Counts the number of distinct values in the bucket. This just copies the
+ * Datum values into a simple array, and sorts them using memcmp-based
+ * comparator. That means it only works for pass-by-value data types (assuming
+ * they don't use collations etc.)
+ */
+static void
+update_bucket_ndistinct(MVBucket bucket, int2vector *attrs, VacAttrStats ** stats)
+{
+ int i, j;
+ int numattrs = attrs->dim1;
+
+ HistogramBuild data = (HistogramBuild)bucket->build_data;
+ int numrows = data->numrows;
+
+ MultiSortSupport mss = multi_sort_init(numattrs);
+
+ /*
+ * We could collect this while walking through all the attributes above
+ * (this way we have to call heap_getattr twice).
+ */
+ SortItem *items = (SortItem*)palloc0(numrows * sizeof(SortItem));
+ Datum *values = (Datum*)palloc0(numrows * sizeof(Datum) * numattrs);
+ bool *isnull = (bool*)palloc0(numrows * sizeof(bool) * numattrs);
+
+ for (i = 0; i < numrows; i++)
+ {
+ items[i].values = &values[i * numattrs];
+ items[i].isnull = &isnull[i * numattrs];
+ }
+
+ /* prepare the sort function for the first dimension */
+ for (i = 0; i < numattrs; i++)
+ multi_sort_add_dimension(mss, i, i, stats);
+
+ /* collect the values */
+ for (i = 0; i < numrows; i++)
+ for (j = 0; j < numattrs; j++)
+ items[i].values[j]
+ = heap_getattr(data->rows[i], attrs->values[j],
+ stats[j]->tupDesc, &items[i].isnull[j]);
+
+ qsort_arg((void *) items, numrows, sizeof(SortItem),
+ multi_sort_compare, mss);
+
+ data->ndistinct = 1;
+
+ for (i = 1; i < numrows; i++)
+ if (multi_sort_compare(&items[i], &items[i-1], mss) != 0)
+ data->ndistinct += 1;
+
+ pfree(items);
+ pfree(values);
+ pfree(isnull);
+}
+
+/*
+ * Count distinct values per bucket dimension.
+ */
+static void
+update_dimension_ndistinct(MVBucket bucket, int dimension, int2vector *attrs,
+ VacAttrStats ** stats, bool update_boundaries)
+{
+ int j;
+ int nvalues = 0;
+ bool isNull;
+ HistogramBuild data = (HistogramBuild)bucket->build_data;
+ Datum * values = (Datum*)palloc0(data->numrows * sizeof(Datum));
+ SortSupportData ssup;
+
+ StdAnalyzeData * mystats = (StdAnalyzeData *) stats[dimension]->extra_data;
+
+ /* we may already know this is a NULL-only dimension */
+ if (bucket->nullsonly[dimension])
+ data->ndistincts[dimension] = 1;
+
+ memset(&ssup, 0, sizeof(ssup));
+ ssup.ssup_cxt = CurrentMemoryContext;
+
+ /* We always use the default collation for statistics */
+ ssup.ssup_collation = DEFAULT_COLLATION_OID;
+ ssup.ssup_nulls_first = false;
+
+ PrepareSortSupportFromOrderingOp(mystats->ltopr, &ssup);
+
+ for (j = 0; j < data->numrows; j++)
+ {
+ values[nvalues] = heap_getattr(data->rows[j], attrs->values[dimension],
+ stats[dimension]->tupDesc, &isNull);
+
+ /* ignore NULL values */
+ if (! isNull)
+ nvalues++;
+ }
+
+ /* there's always at least 1 distinct value (may be NULL) */
+ data->ndistincts[dimension] = 1;
+
+ /* if there are only NULL values in the column, mark it so and continue
+ * with the next one */
+ if (nvalues == 0)
+ {
+ pfree(values);
+ bucket->nullsonly[dimension] = true;
+ return;
+ }
+
+ /* sort the array (pass-by-value datum */
+ qsort_arg((void *) values, nvalues, sizeof(Datum),
+ compare_scalars_simple, (void *) &ssup);
+
+ /*
+ * Update min/max boundaries to the smallest bounding box. Generally, this
+ * needs to be done only when constructing the initial bucket.
+ */
+ if (update_boundaries)
+ {
+ /* store the min/max values */
+ bucket->min[dimension] = values[0];
+ bucket->min_inclusive[dimension] = true;
+
+ bucket->max[dimension] = values[nvalues-1];
+ bucket->max_inclusive[dimension] = true;
+ }
+
+ /*
+ * Walk through the array and count distinct values by comparing
+ * succeeding values.
+ *
+ * FIXME This only works for pass-by-value types (i.e. not VARCHARs
+ * etc.). Although thanks to the deduplication it might work
+ * even for those types (equal values will get the same item
+ * in the deduplicated array).
+ */
+ for (j = 1; j < nvalues; j++) {
+ if (values[j] != values[j-1])
+ data->ndistincts[dimension] += 1;
+ }
+
+ pfree(values);
+}
+
+/*
+ * A properly built histogram must not contain buckets mixing NULL and non-NULL
+ * values in a single dimension. Each dimension may either be marked as 'nulls
+ * only', and thus containing only NULL values, or it must not contain any NULL
+ * values.
+ *
+ * Therefore, if the sample contains NULL values in any of the columns, it's
+ * necessary to build those NULL-buckets. This is done in an iterative way
+ * using this algorithm, operating on a single bucket:
+ *
+ * (1) Check that all dimensions are well-formed (not mixing NULL and
+ * non-NULL values).
+ *
+ * (2) If all dimensions are well-formed, terminate.
+ *
+ * (3) If the dimension contains only NULL values, but is not marked as
+ * NULL-only, mark it as NULL-only and run the algorithm again (on
+ * this bucket).
+ *
+ * (4) If the dimension mixes NULL and non-NULL values, split the bucket
+ * into two parts - one with NULL values, one with non-NULL values
+ * (replacing the current one). Then run the algorithm on both buckets.
+ *
+ * This is executed in a recursive manner, but the number of executions should
+ * be quite low - limited by the number of NULL-buckets. Also, in each branch
+ * the number of nested calls is limited by the number of dimensions
+ * (attributes) of the histogram.
+ *
+ * At the end, there should be buckets with no mixed dimensions. The number of
+ * buckets produced by this algorithm is rather limited - with N dimensions,
+ * there may be only 2^N such buckets (each dimension may be either NULL or
+ * non-NULL). So with 8 dimensions (current value of MVSTATS_MAX_DIMENSIONS)
+ * there may be only 256 such buckets.
+ *
+ * After this, a 'regular' bucket-split algorithm shall run, further optimizing
+ * the histogram.
+ */
+static void
+create_null_buckets(MVHistogram histogram, int bucket_idx,
+ int2vector *attrs, VacAttrStats ** stats)
+{
+ int i, j;
+ int null_dim = -1;
+ int null_count = 0;
+ bool null_found = false;
+ MVBucket bucket, null_bucket;
+ int null_idx, curr_idx;
+ HistogramBuild data, null_data;
+
+ /* remember original values from the bucket */
+ int numrows;
+ HeapTuple *oldrows = NULL;
+
+ Assert(bucket_idx < histogram->nbuckets);
+ Assert(histogram->ndimensions == attrs->dim1);
+
+ bucket = histogram->buckets[bucket_idx];
+ data = (HistogramBuild)bucket->build_data;
+
+ numrows = data->numrows;
+ oldrows = data->rows;
+
+ /*
+ * Walk through all rows / dimensions, and stop once we find NULL in a
+ * dimension not yet marked as NULL-only.
+ */
+ for (i = 0; i < data->numrows; i++)
+ {
+ /*
+ * FIXME We don't need to start from the first attribute here - we can
+ * start from the last known dimension.
+ */
+ for (j = 0; j < histogram->ndimensions; j++)
+ {
+ /* Is this a NULL-only dimension? If yes, skip. */
+ if (bucket->nullsonly[j])
+ continue;
+
+ /* found a NULL in that dimension? */
+ if (heap_attisnull(data->rows[i], attrs->values[j]))
+ {
+ null_found = true;
+ null_dim = j;
+ break;
+ }
+ }
+
+ /* terminate if we found attribute with NULL values */
+ if (null_found)
+ break;
+ }
+
+ /* no regular dimension contains NULL values => we're done */
+ if (! null_found)
+ return;
+
+ /* walk through the rows again, count NULL values in 'null_dim' */
+ for (i = 0; i < data->numrows; i++)
+ {
+ if (heap_attisnull(data->rows[i], attrs->values[null_dim]))
+ null_count += 1;
+ }
+
+ Assert(null_count <= data->numrows);
+
+ /*
+ * If (null_count == numrows) the dimension already is NULL-only, but is
+ * not yet marked like that. It's enough to mark it and repeat the process
+ * recursively (until we run out of dimensions).
+ */
+ if (null_count == data->numrows)
+ {
+ bucket->nullsonly[null_dim] = true;
+ create_null_buckets(histogram, bucket_idx, attrs, stats);
+ return;
+ }
+
+ /*
+ * We have to split the bucket into two - one with NULL values in the
+ * dimension, one with non-NULL values. We don't need to sort the data or
+ * anything, but otherwise it's similar to what partition_bucket() does.
+ */
+
+ /* create bucket with NULL-only dimension 'dim' */
+ null_bucket = copy_mv_bucket(bucket, histogram->ndimensions);
+ null_data = (HistogramBuild)null_bucket->build_data;
+
+ /* remember the current array info */
+ oldrows = data->rows;
+ numrows = data->numrows;
+
+ /* we'll keep non-NULL values in the current bucket */
+ data->numrows = (numrows - null_count);
+ data->rows
+ = (HeapTuple*)palloc0(data->numrows * sizeof(HeapTuple));
+
+ /* and the NULL values will go to the new one */
+ null_data->numrows = null_count;
+ null_data->rows
+ = (HeapTuple*)palloc0(null_data->numrows * sizeof(HeapTuple));
+
+ /* mark the dimension as NULL-only (in the new bucket) */
+ null_bucket->nullsonly[null_dim] = true;
+
+ /* walk through the sample rows and distribute them accordingly */
+ null_idx = 0;
+ curr_idx = 0;
+ for (i = 0; i < numrows; i++)
+ {
+ if (heap_attisnull(oldrows[i], attrs->values[null_dim]))
+ /* NULL => copy to the new bucket */
+ memcpy(&null_data->rows[null_idx++], &oldrows[i],
+ sizeof(HeapTuple));
+ else
+ memcpy(&data->rows[curr_idx++], &oldrows[i],
+ sizeof(HeapTuple));
+ }
+
+ /* update ndistinct values for the buckets (total and per dimension) */
+ update_bucket_ndistinct(bucket, attrs, stats);
+ update_bucket_ndistinct(null_bucket, attrs, stats);
+
+ /*
+ * TODO We don't need to do this for the dimension we used for split,
+ * because we know how many distinct values went to each bucket (NULL
+ * is not a value, so NULL buckets get 0, and the other bucket got all
+ * the distinct values).
+ */
+ for (i = 0; i < histogram->ndimensions; i++)
+ {
+ update_dimension_ndistinct(bucket, i, attrs, stats, false);
+ update_dimension_ndistinct(null_bucket, i, attrs, stats, false);
+ }
+
+ pfree(oldrows);
+
+ /* add the NULL bucket to the histogram */
+ histogram->buckets[histogram->nbuckets++] = null_bucket;
+
+ /*
+ * And now run the function recursively on both buckets (the new
+ * one first, because the call may change number of buckets, and
+ * it's used as an index).
+ */
+ create_null_buckets(histogram, (histogram->nbuckets-1), attrs, stats);
+ create_null_buckets(histogram, bucket_idx, attrs, stats);
+}
+
+/*
+ * SRF with details about buckets of a histogram:
+ *
+ * - bucket ID (0...nbuckets)
+ * - min values (string array)
+ * - max values (string array)
+ * - nulls only (boolean array)
+ * - min inclusive flags (boolean array)
+ * - max inclusive flags (boolean array)
+ * - frequency (double precision)
+ *
+ * The input is the OID of the statistics, and there are no rows returned if the
+ * statistics contains no histogram (or if there's no statistics for the OID).
+ *
+ * The second parameter (type) determines what values will be returned
+ * in the (minvals,maxvals). There are three possible values:
+ *
+ * 0 (actual values)
+ * -----------------
+ * - prints actual values
+ * - using the output function of the data type (as string)
+ * - handy for investigating the histogram
+ *
+ * 1 (distinct index)
+ * ------------------
+ * - prints index of the distinct value (into the serialized array)
+ * - makes it easier to spot neighbor buckets, etc.
+ * - handy for plotting the histogram
+ *
+ * 2 (normalized distinct index)
+ * -----------------------------
+ * - prints index of the distinct value, but normalized into [0,1]
+ * - similar to 1, but shows how 'long' the bucket range is
+ * - handy for plotting the histogram
+ *
+ * When plotting the histogram, be careful as the (1) and (2) options skew the
+ * lengths by distributing the distinct values uniformly. For data types
+ * without a clear meaning of 'distance' (e.g. strings) that is not a big deal,
+ * but for numbers it may be confusing.
+ */
+PG_FUNCTION_INFO_V1(pg_mv_histogram_buckets);
+
+#define OUTPUT_FORMAT_RAW 0
+#define OUTPUT_FORMAT_INDEXES 1
+#define OUTPUT_FORMAT_DISTINCT 2
+
+Datum
+pg_mv_histogram_buckets(PG_FUNCTION_ARGS)
+{
+ FuncCallContext *funcctx;
+ int call_cntr;
+ int max_calls;
+ TupleDesc tupdesc;
+ AttInMetadata *attinmeta;
+
+ Oid mvoid = PG_GETARG_OID(0);
+ int otype = PG_GETARG_INT32(1);
+
+ if ((otype < 0) || (otype > 2))
+ elog(ERROR, "invalid output type specified");
+
+ /* stuff done only on the first call of the function */
+ if (SRF_IS_FIRSTCALL())
+ {
+ MemoryContext oldcontext;
+ MVSerializedHistogram histogram;
+
+ /* create a function context for cross-call persistence */
+ funcctx = SRF_FIRSTCALL_INIT();
+
+ /* switch to memory context appropriate for multiple function calls */
+ oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
+
+ histogram = load_mv_histogram(mvoid);
+
+ funcctx->user_fctx = histogram;
+
+ /* total number of tuples to be returned */
+ funcctx->max_calls = 0;
+ if (funcctx->user_fctx != NULL)
+ funcctx->max_calls = histogram->nbuckets;
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("function returning record called in context "
+ "that cannot accept type record")));
+
+ /*
+ * generate attribute metadata needed later to produce tuples
+ * from raw C strings
+ */
+ attinmeta = TupleDescGetAttInMetadata(tupdesc);
+ funcctx->attinmeta = attinmeta;
+
+ MemoryContextSwitchTo(oldcontext);
+ }
+
+ /* stuff done on every call of the function */
+ funcctx = SRF_PERCALL_SETUP();
+
+ call_cntr = funcctx->call_cntr;
+ max_calls = funcctx->max_calls;
+ attinmeta = funcctx->attinmeta;
+
+ if (call_cntr < max_calls) /* do when there is more left to send */
+ {
+ char **values;
+ HeapTuple tuple;
+ Datum result;
+ int2vector *stakeys;
+ Oid relid;
+ double bucket_volume = 1.0;
+ StringInfo bufs;
+
+ char *format;
+ int i;
+
+ Oid *outfuncs;
+ FmgrInfo *fmgrinfo;
+
+ MVSerializedHistogram histogram;
+ MVSerializedBucket bucket;
+
+ histogram = (MVSerializedHistogram)funcctx->user_fctx;
+
+ Assert(call_cntr < histogram->nbuckets);
+
+ bucket = histogram->buckets[call_cntr];
+
+ stakeys = find_mv_attnums(mvoid, &relid);
+
+ /*
+ * The scalar values will be formatted directly, using snprintf.
+ *
+ * The 'array' values will be formatted through StringInfo.
+ */
+ values = (char **) palloc0(9 * sizeof(char *));
+ bufs = (StringInfo) palloc0(9 * sizeof(StringInfoData));
+
+ values[0] = (char *) palloc(64 * sizeof(char));
+
+ initStringInfo(&bufs[1]); /* lower boundaries */
+ initStringInfo(&bufs[2]); /* upper boundaries */
+ initStringInfo(&bufs[3]); /* nulls-only */
+ initStringInfo(&bufs[4]); /* lower inclusive */
+ initStringInfo(&bufs[5]); /* upper inclusive */
+
+ values[6] = (char *) palloc(64 * sizeof(char));
+ values[7] = (char *) palloc(64 * sizeof(char));
+ values[8] = (char *) palloc(64 * sizeof(char));
+
+ /* we need to do this only when printing the actual values */
+ outfuncs = (Oid*)palloc0(sizeof(Oid) * histogram->ndimensions);
+ fmgrinfo = (FmgrInfo*)palloc0(sizeof(FmgrInfo) * histogram->ndimensions);
+
+ /*
+ * lookup output functions for all histogram dimensions
+ *
+ * XXX This might be one in the first call and stored in user_fctx.
+ */
+ for (i = 0; i < histogram->ndimensions; i++)
+ {
+ bool isvarlena;
+
+ getTypeOutputInfo(get_atttype(relid, stakeys->values[i]),
+ &outfuncs[i], &isvarlena);
+
+ fmgr_info(outfuncs[i], &fmgrinfo[i]);
+ }
+
+ snprintf(values[0], 64, "%d", call_cntr); /* bucket ID */
+
+ /* for the arrays of lower/upper boundaries, formated according to otype */
+ for (i = 0; i < histogram->ndimensions; i++)
+ {
+ Datum *vals = histogram->values[i];
+
+ uint16 minidx = bucket->min[i];
+ uint16 maxidx = bucket->max[i];
+
+ /* compute bucket volume, using distinct values as a measure
+ *
+ * XXX Not really sure what to do for NULL dimensions here, so let's
+ * simply count them as '1'.
+ */
+ bucket_volume
+ *= (double)(maxidx - minidx + 1) / (histogram->nvalues[i]-1);
+
+ if (i == 0)
+ format = "{%s"; /* fist dimension */
+ else if (i < (histogram->ndimensions - 1))
+ format = ", %s"; /* medium dimensions */
+ else
+ format = ", %s}"; /* last dimension */
+
+ appendStringInfo(&bufs[3], format, bucket->nullsonly[i] ? "t" : "f");
+ appendStringInfo(&bufs[4], format, bucket->min_inclusive[i] ? "t" : "f");
+ appendStringInfo(&bufs[5], format, bucket->max_inclusive[i] ? "t" : "f");
+
+ /* for NULL-only dimension, simply put there the NULL and continue */
+ if (bucket->nullsonly[i])
+ {
+ if (i == 0)
+ format = "{%s";
+ else if (i < (histogram->ndimensions - 1))
+ format = ", %s";
+ else
+ format = ", %s}";
+
+ appendStringInfo(&bufs[1], format, "NULL");
+ appendStringInfo(&bufs[2], format, "NULL");
+
+ continue;
+ }
+
+ /* otherwise we really need to format the value */
+ switch (otype)
+ {
+ case OUTPUT_FORMAT_RAW: /* actual boundary values */
+
+ if (i == 0)
+ format = "{%s";
+ else if (i < (histogram->ndimensions - 1))
+ format = ", %s";
+ else
+ format = ", %s}";
+
+ appendStringInfo(&bufs[1], format,
+ FunctionCall1(&fmgrinfo[i], vals[minidx]));
+
+ appendStringInfo(&bufs[2], format,
+ FunctionCall1(&fmgrinfo[i], vals[maxidx]));
+
+ break;
+
+ case OUTPUT_FORMAT_INDEXES: /* indexes into deduplicated arrays */
+
+ if (i == 0)
+ format = "{%d";
+ else if (i < (histogram->ndimensions - 1))
+ format = ", %d";
+ else
+ format = ", %d}";
+
+ appendStringInfo(&bufs[1], format, minidx);
+
+ appendStringInfo(&bufs[2], format, maxidx);
+
+ break;
+
+ case OUTPUT_FORMAT_DISTINCT: /* distinct arrays as measure */
+
+ if (i == 0)
+ format = "{%f";
+ else if (i < (histogram->ndimensions - 1))
+ format = ", %f";
+ else
+ format = ", %f}";
+
+ appendStringInfo(&bufs[1], format,
+ (minidx * 1.0 / (histogram->nvalues[i]-1)));
+
+ appendStringInfo(&bufs[2], format,
+ (maxidx * 1.0 / (histogram->nvalues[i]-1)));
+
+ break;
+
+ default:
+ elog(ERROR, "unknown output type: %d", otype);
+ }
+ }
+
+ values[1] = bufs[1].data;
+ values[2] = bufs[2].data;
+ values[3] = bufs[3].data;
+ values[4] = bufs[4].data;
+ values[5] = bufs[5].data;
+
+ snprintf(values[6], 64, "%f", bucket->ntuples); /* frequency */
+ snprintf(values[7], 64, "%f", bucket->ntuples / bucket_volume); /* density */
+ snprintf(values[8], 64, "%f", bucket_volume); /* volume (as a fraction) */
+
+ /* build a tuple */
+ tuple = BuildTupleFromCStrings(attinmeta, values);
+
+ /* make the tuple into a datum */
+ result = HeapTupleGetDatum(tuple);
+
+ /* clean up (this is not really necessary) */
+ pfree(values[0]);
+ pfree(values[6]);
+ pfree(values[7]);
+ pfree(values[8]);
+
+ resetStringInfo(&bufs[1]);
+ resetStringInfo(&bufs[2]);
+ resetStringInfo(&bufs[3]);
+ resetStringInfo(&bufs[4]);
+ resetStringInfo(&bufs[5]);
+
+ pfree(bufs);
+ pfree(values);
+
+ SRF_RETURN_NEXT(funcctx, result);
+ }
+ else /* do when there is no more left */
+ {
+ SRF_RETURN_DONE(funcctx);
+ }
+}
+
+#ifdef DEBUG_MVHIST
+/*
+ * prints debugging info about matched histogram buckets (full/partial)
+ *
+ * XXX Currently works only for INT data type.
+ */
+void
+debug_histogram_matches(MVSerializedHistogram mvhist, char *matches)
+{
+ int i, j;
+
+ float ffull = 0, fpartial = 0;
+ int nfull = 0, npartial = 0;
+
+ StringInfoData buf;
+
+ initStringInfo(&buf);
+
+ for (i = 0; i < mvhist->nbuckets; i++)
+ {
+ MVSerializedBucket bucket = mvhist->buckets[i];
+
+ if (! matches[i])
+ continue;
+
+ /* increment the counters */
+ nfull += (matches[i] == MVSTATS_MATCH_FULL) ? 1 : 0;
+ npartial += (matches[i] == MVSTATS_MATCH_PARTIAL) ? 1 : 0;
+
+ /* and also update the frequencies */
+ ffull += (matches[i] == MVSTATS_MATCH_FULL) ? bucket->ntuples : 0;
+ fpartial += (matches[i] == MVSTATS_MATCH_PARTIAL) ? bucket->ntuples : 0;
+
+ resetStringInfo(&buf);
+
+ /* build ranges for all the dimentions */
+ for (j = 0; j < mvhist->ndimensions; j++)
+ {
+ appendStringInfo(&buf, '[%d %d]',
+ DatumGetInt32(mvhist->values[j][bucket->min[j]]),
+ DatumGetInt32(mvhist->values[j][bucket->max[j]]));
+ }
+
+ elog(WARNING, "bucket %d %s => %d [%f]", i, buf.data, matches[i], bucket->ntuples);
+ }
+
+ elog(WARNING, "full=%f partial=%f (%f)", ffull, fpartial, (ffull + 0.5 * fpartial));
+}
+#endif
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 2c22d31..b693f36 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -2109,9 +2109,9 @@ describeOneTableDetails(const char *schemaname,
{
printfPQExpBuffer(&buf,
"SELECT oid, stanamespace::regnamespace AS nsp, staname, stakeys,\n"
- " deps_enabled, mcv_enabled,\n"
- " deps_built, mcv_built,\n"
- " mcv_max_items,\n"
+ " deps_enabled, mcv_enabled, hist_enabled,\n"
+ " deps_built, mcv_built, hist_built,\n"
+ " mcv_max_items, hist_max_buckets,\n"
" (SELECT string_agg(attname::text,', ')\n"
" FROM ((SELECT unnest(stakeys) AS attnum) s\n"
" JOIN pg_attribute a ON (starelid = a.attrelid and a.attnum = s.attnum))) AS attnums\n"
@@ -2154,8 +2154,17 @@ describeOneTableDetails(const char *schemaname,
first = false;
}
+ if (!strcmp(PQgetvalue(result, i, 6), "t"))
+ {
+ if (! first)
+ appendPQExpBuffer(&buf, ", histogram");
+ else
+ appendPQExpBuffer(&buf, "(histogram");
+ first = false;
+ }
+
appendPQExpBuffer(&buf, ") ON (%s)",
- PQgetvalue(result, i, 9));
+ PQgetvalue(result, i, 12));
printTableAddFooter(&cont, buf.data);
}
diff --git a/src/include/catalog/pg_mv_statistic.h b/src/include/catalog/pg_mv_statistic.h
index 3529b03..7020772 100644
--- a/src/include/catalog/pg_mv_statistic.h
+++ b/src/include/catalog/pg_mv_statistic.h
@@ -39,13 +39,16 @@ CATALOG(pg_mv_statistic,3381)
/* statistics requested to build */
bool deps_enabled; /* analyze dependencies? */
bool mcv_enabled; /* build MCV list? */
+ bool hist_enabled; /* build histogram? */
- /* MCV size */
+ /* histogram / MCV size */
int32 mcv_max_items; /* max MCV items */
+ int32 hist_max_buckets; /* max histogram buckets */
/* statistics that are available (if requested) */
bool deps_built; /* dependencies were built */
bool mcv_built; /* MCV list was built */
+ bool hist_built; /* histogram was built */
/* variable-length fields start here, but we allow direct access to stakeys */
int2vector stakeys; /* array of column keys */
@@ -53,6 +56,7 @@ CATALOG(pg_mv_statistic,3381)
#ifdef CATALOG_VARLEN
bytea stadeps; /* dependencies (serialized) */
bytea stamcv; /* MCV list (serialized) */
+ bytea stahist; /* MV histogram (serialized) */
#endif
} FormData_pg_mv_statistic;
@@ -68,18 +72,22 @@ typedef FormData_pg_mv_statistic *Form_pg_mv_statistic;
* compiler constants for pg_mv_statistic
* ----------------
*/
-#define Natts_pg_mv_statistic 12
+#define Natts_pg_mv_statistic 16
#define Anum_pg_mv_statistic_starelid 1
#define Anum_pg_mv_statistic_staname 2
#define Anum_pg_mv_statistic_stanamespace 3
#define Anum_pg_mv_statistic_staowner 4
#define Anum_pg_mv_statistic_deps_enabled 5
#define Anum_pg_mv_statistic_mcv_enabled 6
-#define Anum_pg_mv_statistic_mcv_max_items 7
-#define Anum_pg_mv_statistic_deps_built 8
-#define Anum_pg_mv_statistic_mcv_built 9
-#define Anum_pg_mv_statistic_stakeys 10
-#define Anum_pg_mv_statistic_stadeps 11
-#define Anum_pg_mv_statistic_stamcv 12
+#define Anum_pg_mv_statistic_hist_enabled 7
+#define Anum_pg_mv_statistic_mcv_max_items 8
+#define Anum_pg_mv_statistic_hist_max_buckets 9
+#define Anum_pg_mv_statistic_deps_built 10
+#define Anum_pg_mv_statistic_mcv_built 11
+#define Anum_pg_mv_statistic_hist_built 12
+#define Anum_pg_mv_statistic_stakeys 13
+#define Anum_pg_mv_statistic_stadeps 14
+#define Anum_pg_mv_statistic_stamcv 15
+#define Anum_pg_mv_statistic_stahist 16
#endif /* PG_MV_STATISTIC_H */
diff --git a/src/include/catalog/pg_proc.h b/src/include/catalog/pg_proc.h
index f8ceabf..0ca4957 100644
--- a/src/include/catalog/pg_proc.h
+++ b/src/include/catalog/pg_proc.h
@@ -2674,6 +2674,10 @@ DATA(insert OID = 3376 ( pg_mv_stats_mcvlist_info PGNSP PGUID 12 1 0 0 0 f f f
DESCR("multi-variate statistics: MCV list info");
DATA(insert OID = 3373 ( pg_mv_mcv_items PGNSP PGUID 12 1 1000 0 0 f f f f t t i s 1 0 2249 "26" "{26,23,1009,1000,701}" "{i,o,o,o,o}" "{oid,index,values,nulls,frequency}" _null_ _null_ pg_mv_mcv_items _null_ _null_ _null_ ));
DESCR("details about MCV list items");
+DATA(insert OID = 3375 ( pg_mv_stats_histogram_info PGNSP PGUID 12 1 0 0 0 f f f f t f i s 1 0 25 "17" _null_ _null_ _null_ _null_ _null_ pg_mv_stats_histogram_info _null_ _null_ _null_ ));
+DESCR("multi-variate statistics: histogram info");
+DATA(insert OID = 3374 ( pg_mv_histogram_buckets PGNSP PGUID 12 1 1000 0 0 f f f f t t i s 2 0 2249 "26 23" "{26,23,23,1009,1009,1000,1000,1000,701,701,701}" "{i,i,o,o,o,o,o,o,o,o,o}" "{oid,otype,index,minvals,maxvals,nullsonly,mininclusive,maxinclusive,frequency,density,bucket_volume}" _null_ _null_ pg_mv_histogram_buckets _null_ _null_ _null_ ));
+DESCR("details about histogram buckets");
DATA(insert OID = 1928 ( pg_stat_get_numscans PGNSP PGUID 12 1 0 0 0 f f f f t f s r 1 0 20 "26" _null_ _null_ _null_ _null_ _null_ pg_stat_get_numscans _null_ _null_ _null_ ));
DESCR("statistics: number of scans done for table/index");
diff --git a/src/include/nodes/relation.h b/src/include/nodes/relation.h
index 2bcd582..8c50bfb 100644
--- a/src/include/nodes/relation.h
+++ b/src/include/nodes/relation.h
@@ -654,10 +654,12 @@ typedef struct MVStatisticInfo
/* enabled statistics */
bool deps_enabled; /* functional dependencies enabled */
bool mcv_enabled; /* MCV list enabled */
+ bool hist_enabled; /* histogram enabled */
/* built/available statistics */
bool deps_built; /* functional dependencies built */
bool mcv_built; /* MCV list built */
+ bool hist_built; /* histogram built */
/* columns in the statistics (attnums) */
int2vector *stakeys; /* attnums of the columns covered */
diff --git a/src/include/utils/mvstats.h b/src/include/utils/mvstats.h
index ce7c3ad..6708139 100644
--- a/src/include/utils/mvstats.h
+++ b/src/include/utils/mvstats.h
@@ -93,6 +93,123 @@ typedef MCVListData *MCVList;
#define MVSTAT_MCVLIST_MAX_ITEMS 8192 /* max items in MCV list */
/*
+ * Multivariate histograms
+ */
+typedef struct MVBucketData {
+
+ /* Frequencies of this bucket. */
+ float ntuples; /* frequency of tuples tuples */
+
+ /*
+ * Information about dimensions being NULL-only. Not yet used.
+ */
+ bool *nullsonly;
+
+ /* lower boundaries - values and information about the inequalities */
+ Datum *min;
+ bool *min_inclusive;
+
+ /* upper boundaries - values and information about the inequalities */
+ Datum *max;
+ bool *max_inclusive;
+
+ /* used when building the histogram (not serialized/deserialized) */
+ void *build_data;
+
+} MVBucketData;
+
+typedef MVBucketData *MVBucket;
+
+
+typedef struct MVHistogramData {
+
+ uint32 magic; /* magic constant marker */
+ uint32 type; /* type of histogram (BASIC) */
+ uint32 nbuckets; /* number of buckets (buckets array) */
+ uint32 ndimensions; /* number of dimensions */
+
+ MVBucket *buckets; /* array of buckets */
+
+} MVHistogramData;
+
+typedef MVHistogramData *MVHistogram;
+
+/*
+ * Histogram in a partially serialized form, with deduplicated boundary
+ * values etc.
+ *
+ * TODO add more detailed description here
+ */
+
+typedef struct MVSerializedBucketData {
+
+ /* Frequencies of this bucket. */
+ float ntuples; /* frequency of tuples tuples */
+
+ /*
+ * Information about dimensions being NULL-only. Not yet used.
+ */
+ bool *nullsonly;
+
+ /* lower boundaries - values and information about the inequalities */
+ uint16 *min;
+ bool *min_inclusive;
+
+ /* indexes of upper boundaries - values and information about the
+ * inequalities (exclusive vs. inclusive) */
+ uint16 *max;
+ bool *max_inclusive;
+
+} MVSerializedBucketData;
+
+typedef MVSerializedBucketData *MVSerializedBucket;
+
+typedef struct MVSerializedHistogramData {
+
+ uint32 magic; /* magic constant marker */
+ uint32 type; /* type of histogram (BASIC) */
+ uint32 nbuckets; /* number of buckets (buckets array) */
+ uint32 ndimensions; /* number of dimensions */
+
+ /*
+ * keep this the same with MVHistogramData, because of
+ * deserialization (same offset)
+ */
+ MVSerializedBucket *buckets; /* array of buckets */
+
+ /*
+ * serialized boundary values, one array per dimension, deduplicated
+ * (the min/max indexes point into these arrays)
+ */
+ int *nvalues;
+ Datum **values;
+
+} MVSerializedHistogramData;
+
+typedef MVSerializedHistogramData *MVSerializedHistogram;
+
+
+/* used to flag stats serialized to bytea */
+#define MVSTAT_HIST_MAGIC 0x7F8C5670 /* marks serialized bytea */
+#define MVSTAT_HIST_TYPE_BASIC 1 /* basic histogram type */
+
+/*
+ * Limits used for max_buckets option, i.e. we're always guaranteed
+ * to have space for at least MVSTAT_HIST_MIN_BUCKETS, and we cannot
+ * have more than MVSTAT_HIST_MAX_BUCKETS buckets.
+ *
+ * This is just a boundary for the 'max' threshold - the actual
+ * histogram may use less buckets than MVSTAT_HIST_MAX_BUCKETS.
+ *
+ * TODO The MVSTAT_HIST_MIN_BUCKETS should be related to the number of
+ * attributes (MVSTATS_MAX_DIMENSIONS) because of NULL-buckets.
+ * There should be at least 2^N buckets, otherwise we may be unable
+ * to build the NULL buckets.
+ */
+#define MVSTAT_HIST_MIN_BUCKETS 128 /* min number of buckets */
+#define MVSTAT_HIST_MAX_BUCKETS 16384 /* max number of buckets */
+
+/*
* TODO Maybe fetching the histogram/MCV list separately is inefficient?
* Consider adding a single `fetch_stats` method, fetching all
* stats specified using flags (or something like that).
@@ -100,20 +217,25 @@ typedef MCVListData *MCVList;
MVDependencies load_mv_dependencies(Oid mvoid);
MCVList load_mv_mcvlist(Oid mvoid);
+MVSerializedHistogram load_mv_histogram(Oid mvoid);
bytea * serialize_mv_dependencies(MVDependencies dependencies);
bytea * serialize_mv_mcvlist(MCVList mcvlist, int2vector *attrs,
VacAttrStats **stats);
+bytea * serialize_mv_histogram(MVHistogram histogram, int2vector *attrs,
+ VacAttrStats **stats);
/* deserialization of stats (serialization is private to analyze) */
MVDependencies deserialize_mv_dependencies(bytea * data);
MCVList deserialize_mv_mcvlist(bytea * data);
+MVSerializedHistogram deserialize_mv_histogram(bytea * data);
/*
* Returns index of the attribute number within the vector (i.e. a
* dimension within the stats).
*/
int mv_get_index(AttrNumber varattno, int2vector * stakeys);
+int2vector* find_mv_attnums(Oid mvoid, Oid *relid);
int2vector* find_mv_attnums(Oid mvoid, Oid *relid);
@@ -122,6 +244,8 @@ extern Datum pg_mv_stats_dependencies_info(PG_FUNCTION_ARGS);
extern Datum pg_mv_stats_dependencies_show(PG_FUNCTION_ARGS);
extern Datum pg_mv_stats_mcvlist_info(PG_FUNCTION_ARGS);
extern Datum pg_mv_mcvlist_items(PG_FUNCTION_ARGS);
+extern Datum pg_mv_stats_histogram_info(PG_FUNCTION_ARGS);
+extern Datum pg_mv_histogram_buckets(PG_FUNCTION_ARGS);
MVDependencies
build_mv_dependencies(int numrows, HeapTuple *rows, int2vector *attrs,
@@ -131,10 +255,20 @@ MCVList
build_mv_mcvlist(int numrows, HeapTuple *rows, int2vector *attrs,
VacAttrStats **stats, int *numrows_filtered);
+MVHistogram
+build_mv_histogram(int numrows, HeapTuple *rows, int2vector *attrs,
+ VacAttrStats **stats, int numrows_total);
+
void build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
int natts, VacAttrStats **vacattrstats);
-void update_mv_stats(Oid relid, MVDependencies dependencies, MCVList mcvlist,
+void update_mv_stats(Oid relid, MVDependencies dependencies,
+ MCVList mcvlist, MVHistogram histogram,
int2vector *attrs, VacAttrStats **stats);
+#ifdef DEBUG_MVHIST
+extern void debug_histogram_matches(MVSerializedHistogram mvhist, char *matches);
+#endif
+
+
#endif
diff --git a/src/test/regress/expected/mv_histogram.out b/src/test/regress/expected/mv_histogram.out
new file mode 100644
index 0000000..e830816
--- /dev/null
+++ b/src/test/regress/expected/mv_histogram.out
@@ -0,0 +1,207 @@
+-- data type passed by value
+CREATE TABLE mv_histogram (
+ a INT,
+ b INT,
+ c INT
+);
+-- unknown column
+CREATE STATISTICS s7 ON mv_histogram (unknown_column) WITH (histogram);
+ERROR: column "unknown_column" referenced in statistics does not exist
+-- single column
+CREATE STATISTICS s7 ON mv_histogram (a) WITH (histogram);
+ERROR: multivariate stats require 2 or more columns
+-- single column, duplicated
+CREATE STATISTICS s7 ON mv_histogram (a, a) WITH (histogram);
+ERROR: duplicate column name in statistics definition
+-- two columns, one duplicated
+CREATE STATISTICS s7 ON mv_histogram (a, a, b) WITH (histogram);
+ERROR: duplicate column name in statistics definition
+-- unknown option
+CREATE STATISTICS s7 ON mv_histogram (a, b, c) WITH (unknown_option);
+ERROR: unrecognized STATISTICS option "unknown_option"
+-- missing histogram statistics
+CREATE STATISTICS s7 ON mv_histogram (a, b, c) WITH (dependencies, max_buckets=200);
+ERROR: option 'histogram' is required by other options(s)
+-- invalid max_buckets value / too low
+CREATE STATISTICS s7 ON mv_histogram (a, b, c) WITH (mcv, max_buckets=10);
+ERROR: minimum number of buckets is 128
+-- invalid max_buckets value / too high
+CREATE STATISTICS s7 ON mv_histogram (a, b, c) WITH (mcv, max_buckets=100000);
+ERROR: maximum number of buckets is 16384
+-- correct command
+CREATE STATISTICS s7 ON mv_histogram (a, b, c) WITH (histogram);
+-- random data (no functional dependencies)
+INSERT INTO mv_histogram
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+ANALYZE mv_histogram;
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+ hist_enabled | hist_built
+--------------+------------
+ t | t
+(1 row)
+
+TRUNCATE mv_histogram;
+-- a => b, a => c, b => c
+INSERT INTO mv_histogram
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE mv_histogram;
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+ hist_enabled | hist_built
+--------------+------------
+ t | t
+(1 row)
+
+TRUNCATE mv_histogram;
+-- a => b, a => c
+INSERT INTO mv_histogram
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE mv_histogram;
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+ hist_enabled | hist_built
+--------------+------------
+ t | t
+(1 row)
+
+TRUNCATE mv_histogram;
+-- check explain (expect bitmap index scan, not plain index scan)
+INSERT INTO mv_histogram
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX hist_idx ON mv_histogram (a, b);
+ANALYZE mv_histogram;
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+ hist_enabled | hist_built
+--------------+------------
+ t | t
+(1 row)
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mv_histogram WHERE a = 10 AND b = 5;
+ QUERY PLAN
+--------------------------------------------
+ Bitmap Heap Scan on mv_histogram
+ Recheck Cond: ((a = 10) AND (b = 5))
+ -> Bitmap Index Scan on hist_idx
+ Index Cond: ((a = 10) AND (b = 5))
+(4 rows)
+
+DROP TABLE mv_histogram;
+-- varlena type (text)
+CREATE TABLE mv_histogram (
+ a TEXT,
+ b TEXT,
+ c TEXT
+);
+CREATE STATISTICS s8 ON mv_histogram (a, b, c) WITH (histogram);
+-- random data (no functional dependencies)
+INSERT INTO mv_histogram
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+ANALYZE mv_histogram;
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+ hist_enabled | hist_built
+--------------+------------
+ t | t
+(1 row)
+
+TRUNCATE mv_histogram;
+-- a => b, a => c, b => c
+INSERT INTO mv_histogram
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE mv_histogram;
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+ hist_enabled | hist_built
+--------------+------------
+ t | t
+(1 row)
+
+TRUNCATE mv_histogram;
+-- a => b, a => c
+INSERT INTO mv_histogram
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE mv_histogram;
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+ hist_enabled | hist_built
+--------------+------------
+ t | t
+(1 row)
+
+TRUNCATE mv_histogram;
+-- check explain (expect bitmap index scan, not plain index scan)
+INSERT INTO mv_histogram
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX hist_idx ON mv_histogram (a, b);
+ANALYZE mv_histogram;
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+ hist_enabled | hist_built
+--------------+------------
+ t | t
+(1 row)
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mv_histogram WHERE a = '10' AND b = '5';
+ QUERY PLAN
+------------------------------------------------------------
+ Bitmap Heap Scan on mv_histogram
+ Recheck Cond: ((a = '10'::text) AND (b = '5'::text))
+ -> Bitmap Index Scan on hist_idx
+ Index Cond: ((a = '10'::text) AND (b = '5'::text))
+(4 rows)
+
+TRUNCATE mv_histogram;
+-- check explain (expect bitmap index scan, not plain index scan) with NULLs
+INSERT INTO mv_histogram
+ SELECT
+ (CASE WHEN i/10000 = 0 THEN NULL ELSE i/10000 END),
+ (CASE WHEN i/20000 = 0 THEN NULL ELSE i/20000 END),
+ (CASE WHEN i/40000 = 0 THEN NULL ELSE i/40000 END)
+ FROM generate_series(1,1000000) s(i);
+ANALYZE mv_histogram;
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+ hist_enabled | hist_built
+--------------+------------
+ t | t
+(1 row)
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mv_histogram WHERE a IS NULL AND b IS NULL;
+ QUERY PLAN
+---------------------------------------------------
+ Bitmap Heap Scan on mv_histogram
+ Recheck Cond: ((a IS NULL) AND (b IS NULL))
+ -> Bitmap Index Scan on hist_idx
+ Index Cond: ((a IS NULL) AND (b IS NULL))
+(4 rows)
+
+DROP TABLE mv_histogram;
+-- NULL values (mix of int and text columns)
+CREATE TABLE mv_histogram (
+ a INT,
+ b TEXT,
+ c INT,
+ d TEXT
+);
+CREATE STATISTICS s9 ON mv_histogram (a, b, c, d) WITH (histogram);
+INSERT INTO mv_histogram
+ SELECT
+ mod(i, 100),
+ (CASE WHEN mod(i, 200) = 0 THEN NULL ELSE mod(i,200) END),
+ mod(i, 400),
+ (CASE WHEN mod(i, 300) = 0 THEN NULL ELSE mod(i,600) END)
+ FROM generate_series(1,10000) s(i);
+ANALYZE mv_histogram;
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+ hist_enabled | hist_built
+--------------+------------
+ t | t
+(1 row)
+
+DROP TABLE mv_histogram;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 3d55ffe..528ac36 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1375,7 +1375,9 @@ pg_mv_stats| SELECT n.nspname AS schemaname,
length(s.stadeps) AS depsbytes,
pg_mv_stats_dependencies_info(s.stadeps) AS depsinfo,
length(s.stamcv) AS mcvbytes,
- pg_mv_stats_mcvlist_info(s.stamcv) AS mcvinfo
+ pg_mv_stats_mcvlist_info(s.stamcv) AS mcvinfo,
+ length(s.stahist) AS histbytes,
+ pg_mv_stats_histogram_info(s.stahist) AS histinfo
FROM ((pg_mv_statistic s
JOIN pg_class c ON ((c.oid = s.starelid)))
LEFT JOIN pg_namespace n ON ((n.oid = c.relnamespace)));
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index 85d94f1..a885235 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -112,4 +112,4 @@ test: event_trigger
test: stats
# run tests of multivariate stats
-test: mv_dependencies mv_mcv
+test: mv_dependencies mv_mcv mv_histogram
diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule
index 6584d73..2efdcd7 100644
--- a/src/test/regress/serial_schedule
+++ b/src/test/regress/serial_schedule
@@ -164,3 +164,4 @@ test: event_trigger
test: stats
test: mv_dependencies
test: mv_mcv
+test: mv_histogram
diff --git a/src/test/regress/sql/mv_histogram.sql b/src/test/regress/sql/mv_histogram.sql
new file mode 100644
index 0000000..27c2510
--- /dev/null
+++ b/src/test/regress/sql/mv_histogram.sql
@@ -0,0 +1,176 @@
+-- data type passed by value
+CREATE TABLE mv_histogram (
+ a INT,
+ b INT,
+ c INT
+);
+
+-- unknown column
+CREATE STATISTICS s7 ON mv_histogram (unknown_column) WITH (histogram);
+
+-- single column
+CREATE STATISTICS s7 ON mv_histogram (a) WITH (histogram);
+
+-- single column, duplicated
+CREATE STATISTICS s7 ON mv_histogram (a, a) WITH (histogram);
+
+-- two columns, one duplicated
+CREATE STATISTICS s7 ON mv_histogram (a, a, b) WITH (histogram);
+
+-- unknown option
+CREATE STATISTICS s7 ON mv_histogram (a, b, c) WITH (unknown_option);
+
+-- missing histogram statistics
+CREATE STATISTICS s7 ON mv_histogram (a, b, c) WITH (dependencies, max_buckets=200);
+
+-- invalid max_buckets value / too low
+CREATE STATISTICS s7 ON mv_histogram (a, b, c) WITH (mcv, max_buckets=10);
+
+-- invalid max_buckets value / too high
+CREATE STATISTICS s7 ON mv_histogram (a, b, c) WITH (mcv, max_buckets=100000);
+
+-- correct command
+CREATE STATISTICS s7 ON mv_histogram (a, b, c) WITH (histogram);
+
+-- random data (no functional dependencies)
+INSERT INTO mv_histogram
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+
+ANALYZE mv_histogram;
+
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+
+TRUNCATE mv_histogram;
+
+-- a => b, a => c, b => c
+INSERT INTO mv_histogram
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+
+ANALYZE mv_histogram;
+
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+
+TRUNCATE mv_histogram;
+
+-- a => b, a => c
+INSERT INTO mv_histogram
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE mv_histogram;
+
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+
+TRUNCATE mv_histogram;
+
+-- check explain (expect bitmap index scan, not plain index scan)
+INSERT INTO mv_histogram
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX hist_idx ON mv_histogram (a, b);
+ANALYZE mv_histogram;
+
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mv_histogram WHERE a = 10 AND b = 5;
+
+DROP TABLE mv_histogram;
+
+-- varlena type (text)
+CREATE TABLE mv_histogram (
+ a TEXT,
+ b TEXT,
+ c TEXT
+);
+
+CREATE STATISTICS s8 ON mv_histogram (a, b, c) WITH (histogram);
+
+-- random data (no functional dependencies)
+INSERT INTO mv_histogram
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+
+ANALYZE mv_histogram;
+
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+
+TRUNCATE mv_histogram;
+
+-- a => b, a => c, b => c
+INSERT INTO mv_histogram
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+
+ANALYZE mv_histogram;
+
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+
+TRUNCATE mv_histogram;
+
+-- a => b, a => c
+INSERT INTO mv_histogram
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE mv_histogram;
+
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+
+TRUNCATE mv_histogram;
+
+-- check explain (expect bitmap index scan, not plain index scan)
+INSERT INTO mv_histogram
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX hist_idx ON mv_histogram (a, b);
+ANALYZE mv_histogram;
+
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mv_histogram WHERE a = '10' AND b = '5';
+
+TRUNCATE mv_histogram;
+
+-- check explain (expect bitmap index scan, not plain index scan) with NULLs
+INSERT INTO mv_histogram
+ SELECT
+ (CASE WHEN i/10000 = 0 THEN NULL ELSE i/10000 END),
+ (CASE WHEN i/20000 = 0 THEN NULL ELSE i/20000 END),
+ (CASE WHEN i/40000 = 0 THEN NULL ELSE i/40000 END)
+ FROM generate_series(1,1000000) s(i);
+ANALYZE mv_histogram;
+
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mv_histogram WHERE a IS NULL AND b IS NULL;
+
+DROP TABLE mv_histogram;
+
+-- NULL values (mix of int and text columns)
+CREATE TABLE mv_histogram (
+ a INT,
+ b TEXT,
+ c INT,
+ d TEXT
+);
+
+CREATE STATISTICS s9 ON mv_histogram (a, b, c, d) WITH (histogram);
+
+INSERT INTO mv_histogram
+ SELECT
+ mod(i, 100),
+ (CASE WHEN mod(i, 200) = 0 THEN NULL ELSE mod(i,200) END),
+ mod(i, 400),
+ (CASE WHEN mod(i, 300) = 0 THEN NULL ELSE mod(i,600) END)
+ FROM generate_series(1,10000) s(i);
+
+ANALYZE mv_histogram;
+
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+
+DROP TABLE mv_histogram;
--
2.5.0
[text/x-patch] 0004-multivariate-MCV-lists.patch (113.9K, ../../1457906378.27231.10.camel@2ndquadrant.com/6-0004-multivariate-MCV-lists.patch)
download | inline diff:
From 9786256d4dec9b3d6ea90ebbbeebd41568453b1b Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@pgaddict.com>
Date: Mon, 6 Apr 2015 16:52:15 +0200
Subject: [PATCH 4/9] multivariate MCV lists
- extends the pg_mv_statistic catalog (add 'mcv' fields)
- building the MCV lists during ANALYZE
- simple estimation while planning the queries
Includes regression tests, mostly equal to regression tests for
functional dependencies.
---
doc/src/sgml/ref/create_statistics.sgml | 43 ++
src/backend/catalog/system_views.sql | 4 +-
src/backend/commands/statscmds.c | 45 +-
src/backend/nodes/outfuncs.c | 2 +
src/backend/optimizer/path/clausesel.c | 814 +++++++++++++++++++++-
src/backend/optimizer/util/plancat.c | 4 +-
src/backend/utils/mvstats/Makefile | 2 +-
src/backend/utils/mvstats/README.mcv | 137 ++++
src/backend/utils/mvstats/README.stats | 89 ++-
src/backend/utils/mvstats/common.c | 133 +++-
src/backend/utils/mvstats/common.h | 17 +-
src/backend/utils/mvstats/mcv.c | 1120 +++++++++++++++++++++++++++++++
src/bin/psql/describe.c | 25 +-
src/include/catalog/pg_mv_statistic.h | 18 +-
src/include/catalog/pg_proc.h | 4 +
src/include/nodes/relation.h | 2 +
src/include/utils/mvstats.h | 69 +-
src/test/regress/expected/mv_mcv.out | 207 ++++++
src/test/regress/expected/rules.out | 4 +-
src/test/regress/parallel_schedule | 2 +-
src/test/regress/serial_schedule | 1 +
src/test/regress/sql/mv_mcv.sql | 178 +++++
22 files changed, 2847 insertions(+), 73 deletions(-)
create mode 100644 src/backend/utils/mvstats/README.mcv
create mode 100644 src/backend/utils/mvstats/mcv.c
create mode 100644 src/test/regress/expected/mv_mcv.out
create mode 100644 src/test/regress/sql/mv_mcv.sql
diff --git a/doc/src/sgml/ref/create_statistics.sgml b/doc/src/sgml/ref/create_statistics.sgml
index ff09fa5..d6973e8 100644
--- a/doc/src/sgml/ref/create_statistics.sgml
+++ b/doc/src/sgml/ref/create_statistics.sgml
@@ -132,6 +132,24 @@ CREATE STATISTICS [ IF NOT EXISTS ] <replaceable class="PARAMETER">statistics_na
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>max_mcv_items</> (<type>integer</>)</term>
+ <listitem>
+ <para>
+ Maximum number of MCV list items.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>mcv</> (<type>boolean</>)</term>
+ <listitem>
+ <para>
+ Enables MCV list for the statistics.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect2>
@@ -177,6 +195,31 @@ EXPLAIN ANALYZE SELECT * FROM t1 WHERE (a = 1) AND (b = 2);
</programlisting>
</para>
+ <para>
+ Create table <structname>t2</> with two perfectly correlated columns
+ (containing identical data), and a MCV list on those columns:
+
+<programlisting>
+CREATE TABLE t2 (
+ a int,
+ b int
+);
+
+INSERT INTO t2 SELECT mod(i,100), mod(i,100)
+ FROM generate_series(1,1000000) s(i);
+
+CREATE STATISTICS s2 ON t2 (a, b) WITH (mcv);
+
+ANALYZE t2;
+
+-- valid combination (found in MCV)
+EXPLAIN ANALYZE SELECT * FROM t2 WHERE (a = 1) AND (b = 1);
+
+-- invalid combination (not found in MCV)
+EXPLAIN ANALYZE SELECT * FROM t2 WHERE (a = 1) AND (b = 2);
+</programlisting>
+ </para>
+
</refsect1>
<refsect1>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 31dbb2c..5c40334 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -165,7 +165,9 @@ CREATE VIEW pg_mv_stats AS
S.staname AS staname,
S.stakeys AS attnums,
length(S.stadeps) as depsbytes,
- pg_mv_stats_dependencies_info(S.stadeps) as depsinfo
+ pg_mv_stats_dependencies_info(S.stadeps) as depsinfo,
+ length(S.stamcv) AS mcvbytes,
+ pg_mv_stats_mcvlist_info(S.stamcv) AS mcvinfo
FROM (pg_mv_statistic S JOIN pg_class C ON (C.oid = S.starelid))
LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace);
diff --git a/src/backend/commands/statscmds.c b/src/backend/commands/statscmds.c
index f43b053..c480fbe 100644
--- a/src/backend/commands/statscmds.c
+++ b/src/backend/commands/statscmds.c
@@ -70,7 +70,13 @@ CreateStatistics(CreateStatsStmt *stmt)
ObjectAddress parentobject, childobject;
/* by default build nothing */
- bool build_dependencies = false;
+ bool build_dependencies = false,
+ build_mcv = false;
+
+ int32 max_mcv_items = -1;
+
+ /* options required because of other options */
+ bool require_mcv = false;
Assert(IsA(stmt, CreateStatsStmt));
@@ -146,6 +152,29 @@ CreateStatistics(CreateStatsStmt *stmt)
if (strcmp(opt->defname, "dependencies") == 0)
build_dependencies = defGetBoolean(opt);
+ else if (strcmp(opt->defname, "mcv") == 0)
+ build_mcv = defGetBoolean(opt);
+ else if (strcmp(opt->defname, "max_mcv_items") == 0)
+ {
+ max_mcv_items = defGetInt32(opt);
+
+ /* this option requires 'mcv' to be enabled */
+ require_mcv = true;
+
+ /* sanity check */
+ if (max_mcv_items < MVSTAT_MCVLIST_MIN_ITEMS)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("max number of MCV items must be at least %d",
+ MVSTAT_MCVLIST_MIN_ITEMS)));
+
+ else if (max_mcv_items > MVSTAT_MCVLIST_MAX_ITEMS)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("max number of MCV items is %d",
+ MVSTAT_MCVLIST_MAX_ITEMS)));
+
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -154,10 +183,16 @@ CreateStatistics(CreateStatsStmt *stmt)
}
/* check that at least some statistics were requested */
- if (! build_dependencies)
+ if (! (build_dependencies || build_mcv))
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("no statistics type (dependencies, mcv) was requested")));
+
+ /* now do some checking of the options */
+ if (require_mcv && (! build_mcv))
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("no statistics type (dependencies) was requested")));
+ errmsg("option 'mcv' is required by other options(s)")));
/* sort the attnums and build int2vector */
qsort(attnums, numcols, sizeof(int16), compare_int16);
@@ -178,8 +213,12 @@ CreateStatistics(CreateStatsStmt *stmt)
values[Anum_pg_mv_statistic_stakeys -1] = PointerGetDatum(stakeys);
values[Anum_pg_mv_statistic_deps_enabled -1] = BoolGetDatum(build_dependencies);
+ values[Anum_pg_mv_statistic_mcv_enabled -1] = BoolGetDatum(build_mcv);
+
+ values[Anum_pg_mv_statistic_mcv_max_items -1] = Int32GetDatum(max_mcv_items);
nulls[Anum_pg_mv_statistic_stadeps -1] = true;
+ nulls[Anum_pg_mv_statistic_stamcv -1] = true;
/* insert the tuple into pg_mv_statistic */
mvstatrel = heap_open(MvStatisticRelationId, RowExclusiveLock);
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 07206d7..333e24b 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2162,9 +2162,11 @@ _outMVStatisticInfo(StringInfo str, const MVStatisticInfo *node)
/* enabled statistics */
WRITE_BOOL_FIELD(deps_enabled);
+ WRITE_BOOL_FIELD(mcv_enabled);
/* built/available statistics */
WRITE_BOOL_FIELD(deps_built);
+ WRITE_BOOL_FIELD(mcv_built);
}
static void
diff --git a/src/backend/optimizer/path/clausesel.c b/src/backend/optimizer/path/clausesel.c
index 5ab7f15..7fc0c49 100644
--- a/src/backend/optimizer/path/clausesel.c
+++ b/src/backend/optimizer/path/clausesel.c
@@ -15,6 +15,7 @@
#include "postgres.h"
#include "access/sysattr.h"
+#include "catalog/pg_collation.h"
#include "catalog/pg_operator.h"
#include "nodes/makefuncs.h"
#include "optimizer/clauses.h"
@@ -47,18 +48,39 @@ static void addRangeClause(RangeQueryClause **rqlist, Node *clause,
bool varonleft, bool isLTsel, Selectivity s2);
#define MV_CLAUSE_TYPE_FDEP 0x01
+#define MV_CLAUSE_TYPE_MCV 0x02
-static bool clause_is_mv_compatible(Node *clause, Index relid, AttrNumber *attnum);
+static bool clause_is_mv_compatible(Node *clause, Index relid, Bitmapset **attnums,
+ int type);
-static Bitmapset *collect_mv_attnums(List *clauses, Index relid);
+static Bitmapset *collect_mv_attnums(List *clauses, Index relid, int type);
-static int count_mv_attnums(List *clauses, Index relid);
+static int count_mv_attnums(List *clauses, Index relid, int type);
static int count_varnos(List *clauses, Index *relid);
static List *clauselist_apply_dependencies(PlannerInfo *root, List *clauses,
Index relid, List *stats);
+static MVStatisticInfo *choose_mv_statistics(List *mvstats, Bitmapset *attnums);
+
+static List *clauselist_mv_split(PlannerInfo *root, Index relid,
+ List *clauses, List **mvclauses,
+ MVStatisticInfo *mvstats, int types);
+
+static Selectivity clauselist_mv_selectivity(PlannerInfo *root,
+ List *clauses, MVStatisticInfo *mvstats);
+
+static Selectivity clauselist_mv_selectivity_mcvlist(PlannerInfo *root,
+ List *clauses, MVStatisticInfo *mvstats,
+ bool *fullmatch, Selectivity *lowsel);
+
+static int update_match_bitmap_mcvlist(PlannerInfo *root, List *clauses,
+ int2vector *stakeys, MCVList mcvlist,
+ int nmatches, char * matches,
+ Selectivity *lowsel, bool *fullmatch,
+ bool is_or);
+
static bool has_stats(List *stats, int type);
static List * find_stats(PlannerInfo *root, Index relid);
@@ -66,6 +88,13 @@ static List * find_stats(PlannerInfo *root, Index relid);
static bool stats_type_matches(MVStatisticInfo *stat, int type);
+/* used for merging bitmaps - AND (min), OR (max) */
+#define MAX(x, y) (((x) > (y)) ? (x) : (y))
+#define MIN(x, y) (((x) < (y)) ? (x) : (y))
+
+#define UPDATE_RESULT(m,r,isor) \
+ (m) = (isor) ? (MAX(m,r)) : (MIN(m,r))
+
/****************************************************************************
* ROUTINES TO COMPUTE SELECTIVITIES
****************************************************************************/
@@ -91,11 +120,13 @@ static bool stats_type_matches(MVStatisticInfo *stat, int type);
* to verify that suitable multivariate statistics exist.
*
* If we identify such multivariate statistics apply, we try to apply them.
- * Currently we only have (soft) functional dependencies, so we try to reduce
- * the list of clauses.
*
- * Then we remove the clauses estimated using multivariate stats, and process
- * the rest of the clauses using the regular per-column stats.
+ * First we try to reduce the list of clauses by applying (soft) functional
+ * dependencies, and then we try to estimate the selectivity of the reduced
+ * list of clauses using the multivariate MCV list.
+ *
+ * Finally we remove the portion of clauses estimated using multivariate stats,
+ * and process the rest of the clauses using the regular per-column stats.
*
* Currently, the only extra smarts we have is to recognize "range queries",
* such as "x > 34 AND x < 42". Clauses are recognized as possible range
@@ -172,12 +203,46 @@ clauselist_selectivity(PlannerInfo *root,
* that need to be estimated by other types of stats (MCV, histograms etc).
*/
if (has_stats(stats, MV_CLAUSE_TYPE_FDEP) &&
- (count_mv_attnums(clauses, relid) >= 2))
+ (count_mv_attnums(clauses, relid, MV_CLAUSE_TYPE_FDEP) >= 2))
{
clauses = clauselist_apply_dependencies(root, clauses, relid, stats);
}
/*
+ * Check that there are statistics with MCV list or histogram, and also the
+ * number of attributes covered by these types of statistics.
+ *
+ * If there are no such stats or not enough attributes, don't waste time
+ * with the multivariate code and simply skip to estimation using the
+ * regular per-column stats.
+ */
+ if (has_stats(stats, MV_CLAUSE_TYPE_MCV) &&
+ (count_mv_attnums(clauses, relid, MV_CLAUSE_TYPE_MCV) >= 2))
+ {
+ /* collect attributes from the compatible conditions */
+ Bitmapset *mvattnums = collect_mv_attnums(clauses, relid, MV_CLAUSE_TYPE_MCV);
+
+ /* and search for the statistic covering the most attributes */
+ MVStatisticInfo *mvstat = choose_mv_statistics(stats, mvattnums);
+
+ if (mvstat != NULL) /* we have a matching stats */
+ {
+ /* clauses compatible with multi-variate stats */
+ List *mvclauses = NIL;
+
+ /* split the clauselist into regular and mv-clauses */
+ clauses = clauselist_mv_split(root, relid, clauses, &mvclauses,
+ mvstat, MV_CLAUSE_TYPE_MCV);
+
+ /* we've chosen the histogram to match the clauses */
+ Assert(mvclauses != NIL);
+
+ /* compute the multivariate stats */
+ s1 *= clauselist_mv_selectivity(root, mvclauses, mvstat);
+ }
+ }
+
+ /*
* Initial scan over clauses. Anything that doesn't look like a potential
* rangequery clause gets multiplied into s1 and forgotten. Anything that
* does gets inserted into an rqlist entry.
@@ -834,32 +899,93 @@ clause_selectivity(PlannerInfo *root,
return s1;
}
+
+/*
+ * estimate selectivity of clauses using multivariate statistic
+ *
+ * Perform estimation of the clauses using a MCV list.
+ *
+ * This assumes all the clauses are compatible with the selected statistics
+ * (e.g. only reference columns covered by the statistics, use supported
+ * operator, etc.).
+ *
+ * TODO We may support some additional conditions, most importantly those
+ * matching multiple columns (e.g. "a = b" or "a < b").
+ *
+ * TODO Clamp the selectivity by min of the per-clause selectivities (i.e. the
+ * selectivity of the most restrictive clause), because that's the maximum
+ * we can ever get from ANDed list of clauses. This may probably prevent
+ * issues with hitting too many buckets and low precision histograms.
+ *
+ * TODO We may remember the lowest frequency in the MCV list, and then later use
+ * it as a upper boundary for the selectivity (had there been a more
+ * frequent item, it'd be in the MCV list). This might improve cases with
+ * low-detail histograms.
+ *
+ * TODO We may also derive some additional boundaries for the selectivity from
+ * the MCV list, because
+ *
+ * (a) if we have a "full equality condition" (one equality condition on
+ * each column of the statistic) and we found a match in the MCV list,
+ * then this is the final selectivity (and pretty accurate),
+ *
+ * (b) if we have a "full equality condition" and we haven't found a match
+ * in the MCV list, then the selectivity is below the lowest frequency
+ * found in the MCV list,
+ *
+ * TODO When applying the clauses to the histogram/MCV list, we can do
+ * that from the most selective clauses first, because that'll
+ * eliminate the buckets/items sooner (so we'll be able to skip
+ * them without inspection, which is more expensive). But this
+ * requires really knowing the per-clause selectivities in advance,
+ * and that's not what we do now.
+ */
+static Selectivity
+clauselist_mv_selectivity(PlannerInfo *root, List *clauses, MVStatisticInfo *mvstats)
+{
+ bool fullmatch = false;
+
+ /*
+ * Lowest frequency in the MCV list (may be used as an upper bound
+ * for full equality conditions that did not match any MCV item).
+ */
+ Selectivity mcv_low = 0.0;
+
+ /* TODO Evaluate simple 1D selectivities, use the smallest one as
+ * an upper bound, product as lower bound, and sort the
+ * clauses in ascending order by selectivity (to optimize the
+ * MCV/histogram evaluation).
+ */
+
+ /* Evaluate the MCV selectivity */
+ return clauselist_mv_selectivity_mcvlist(root, clauses, mvstats,
+ &fullmatch, &mcv_low);
+}
+
/*
* Collect attributes from mv-compatible clauses.
*/
static Bitmapset *
-collect_mv_attnums(List *clauses, Index relid)
+collect_mv_attnums(List *clauses, Index relid, int types)
{
Bitmapset *attnums = NULL;
ListCell *l;
/*
- * Walk through the clauses and identify the ones we can estimate
- * using multivariate stats, and remember the relid/columns. We'll
- * then cross-check if we have suitable stats, and only if needed
- * we'll split the clauses into multivariate and regular lists.
+ * Walk through the clauses and identify the ones we can estimate using
+ * multivariate stats, and remember the relid/columns. We'll then
+ * cross-check if we have suitable stats, and only if needed we'll split
+ * the clauses into multivariate and regular lists.
*
- * For now we're only interested in RestrictInfo nodes with nested
- * OpExpr, using either a range or equality.
+ * For now we're only interested in RestrictInfo nodes with nested OpExpr,
+ * using either a range or equality.
*/
foreach (l, clauses)
{
- AttrNumber attnum;
Node *clause = (Node *) lfirst(l);
- /* ignore the result for now - we only need the info */
- if (clause_is_mv_compatible(clause, relid, &attnum))
- attnums = bms_add_member(attnums, attnum);
+ /* ignore the result here - we only need the attnums */
+ clause_is_mv_compatible(clause, relid, &attnums, types);
}
/*
@@ -880,10 +1006,10 @@ collect_mv_attnums(List *clauses, Index relid)
* Count the number of attributes in clauses compatible with multivariate stats.
*/
static int
-count_mv_attnums(List *clauses, Index relid)
+count_mv_attnums(List *clauses, Index relid, int type)
{
int c;
- Bitmapset *attnums = collect_mv_attnums(clauses, relid);
+ Bitmapset *attnums = collect_mv_attnums(clauses, relid, type);
c = bms_num_members(attnums);
@@ -913,9 +1039,183 @@ count_varnos(List *clauses, Index *relid)
return cnt;
}
+
+/*
+ * We're looking for statistics matching at least 2 attributes, referenced in
+ * clauses compatible with multivariate statistics. The current selection
+ * criteria is very simple - we choose the statistics referencing the most
+ * attributes.
+ *
+ * If there are multiple statistics referencing the same number of columns
+ * (from the clauses), the one with less source columns (as listed in the
+ * ADD STATISTICS when creating the statistics) wins. Else the first one wins.
+ *
+ * This is a very simple criteria, and has several weaknesses:
+ *
+ * (a) does not consider the accuracy of the statistics
+ *
+ * If there are two histograms built on the same set of columns, but one
+ * has 100 buckets and the other one has 1000 buckets (thus likely
+ * providing better estimates), this is not currently considered.
+ *
+ * (b) does not consider the type of statistics
+ *
+ * If there are three statistics - one containing just a MCV list, another
+ * one with just a histogram and a third one with both, we treat them equally.
+ *
+ * (c) does not consider the number of clauses
+ *
+ * As explained, only the number of referenced attributes counts, so if
+ * there are multiple clauses on a single attribute, this still counts as
+ * a single attribute.
+ *
+ * (d) does not consider type of condition
+ *
+ * Some clauses may work better with some statistics - for example equality
+ * clauses probably work better with MCV lists than with histograms. But
+ * IS [NOT] NULL conditions may often work better with histograms (thanks
+ * to NULL-buckets).
+ *
+ * So for example with five WHERE conditions
+ *
+ * WHERE (a = 1) AND (b = 1) AND (c = 1) AND (d = 1) AND (e = 1)
+ *
+ * and statistics on (a,b), (a,b,e) and (a,b,c,d), the last one will be selected
+ * as it references the most columns.
+ *
+ * Once we have selected the multivariate statistics, we split the list of
+ * clauses into two parts - conditions that are compatible with the selected
+ * stats, and conditions are estimated using simple statistics.
+ *
+ * From the example above, conditions
+ *
+ * (a = 1) AND (b = 1) AND (c = 1) AND (d = 1)
+ *
+ * will be estimated using the multivariate statistics (a,b,c,d) while the last
+ * condition (e = 1) will get estimated using the regular ones.
+ *
+ * There are various alternative selection criteria (e.g. counting conditions
+ * instead of just referenced attributes), but eventually the best option should
+ * be to combine multiple statistics. But that's much harder to do correctly.
+ *
+ * TODO Select multiple statistics and combine them when computing the estimate.
+ *
+ * TODO This will probably have to consider compatibility of clauses, because
+ * 'dependencies' will probably work only with equality clauses.
+ */
+static MVStatisticInfo *
+choose_mv_statistics(List *stats, Bitmapset *attnums)
+{
+ int i;
+ ListCell *lc;
+
+ MVStatisticInfo *choice = NULL;
+
+ int current_matches = 1; /* goal #1: maximize */
+ int current_dims = (MVSTATS_MAX_DIMENSIONS+1); /* goal #2: minimize */
+
+ /*
+ * Walk through the statistics (simple array with nmvstats elements) and for
+ * each one count the referenced attributes (encoded in the 'attnums' bitmap).
+ */
+ foreach (lc, stats)
+ {
+ MVStatisticInfo *info = (MVStatisticInfo *)lfirst(lc);
+
+ /* columns matching this statistics */
+ int matches = 0;
+
+ int2vector * attrs = info->stakeys;
+ int numattrs = attrs->dim1;
+
+ /* skip dependencies-only stats */
+ if (! info->mcv_built)
+ continue;
+
+ /* count columns covered by the histogram */
+ for (i = 0; i < numattrs; i++)
+ if (bms_is_member(attrs->values[i], attnums))
+ matches++;
+
+ /*
+ * Use this statistics when it improves the number of matches or
+ * when it matches the same number of attributes but is smaller.
+ */
+ if ((matches > current_matches) ||
+ ((matches == current_matches) && (current_dims > numattrs)))
+ {
+ choice = info;
+ current_matches = matches;
+ current_dims = numattrs;
+ }
+ }
+
+ return choice;
+}
+
+
+/*
+ * This splits the clauses list into two parts - one containing clauses that
+ * will be evaluated using the chosen statistics, and the remaining clauses
+ * (either non-mvcompatible, or not related to the histogram).
+ */
+static List *
+clauselist_mv_split(PlannerInfo *root, Index relid,
+ List *clauses, List **mvclauses,
+ MVStatisticInfo *mvstats, int types)
+{
+ int i;
+ ListCell *l;
+ List *non_mvclauses = NIL;
+
+ /* FIXME is there a better way to get info on int2vector? */
+ int2vector * attrs = mvstats->stakeys;
+ int numattrs = mvstats->stakeys->dim1;
+
+ Bitmapset *mvattnums = NULL;
+
+ /* build bitmap of attributes, so we can do bms_is_subset later */
+ for (i = 0; i < numattrs; i++)
+ mvattnums = bms_add_member(mvattnums, attrs->values[i]);
+
+ /* erase the list of mv-compatible clauses */
+ *mvclauses = NIL;
+
+ foreach (l, clauses)
+ {
+ bool match = false; /* by default not mv-compatible */
+ Bitmapset *attnums = NULL;
+ Node *clause = (Node *) lfirst(l);
+
+ if (clause_is_mv_compatible(clause, relid, &attnums, types))
+ {
+ /* are all the attributes part of the selected stats? */
+ if (bms_is_subset(attnums, mvattnums))
+ match = true;
+ }
+
+ /*
+ * The clause matches the selected stats, so put it to the list of
+ * mv-compatible clauses. Otherwise, keep it in the list of 'regular'
+ * clauses (that may be selected later).
+ */
+ if (match)
+ *mvclauses = lappend(*mvclauses, clause);
+ else
+ non_mvclauses = lappend(non_mvclauses, clause);
+ }
+
+ /*
+ * Perform regular estimation using the clauses incompatible with the chosen
+ * histogram (or MV stats in general).
+ */
+ return non_mvclauses;
+
+}
typedef struct
{
+ int types; /* types of statistics ? */
Index varno; /* relid we're interested in */
Bitmapset *varattnos; /* attnums referenced by the clauses */
} mv_compatible_context;
@@ -933,23 +1233,66 @@ mv_compatible_walker(Node *node, mv_compatible_context *context)
{
if (node == NULL)
return false;
-
+
if (IsA(node, RestrictInfo))
{
RestrictInfo *rinfo = (RestrictInfo *) node;
-
+
/* Pseudoconstants are not really interesting here. */
if (rinfo->pseudoconstant)
return true;
-
+
/* clauses referencing multiple varnos are incompatible */
if (bms_membership(rinfo->clause_relids) != BMS_SINGLETON)
return true;
-
+
/* check the clause inside the RestrictInfo */
return mv_compatible_walker((Node*)rinfo->clause, (void *) context);
}
+ if (or_clause(node) || and_clause(node) || not_clause(node))
+ {
+ /*
+ * AND/OR/NOT-clauses are supported if all sub-clauses are supported
+ *
+ * TODO We might support mixed case, where some of the clauses are
+ * supported and some are not, and treat all supported subclauses
+ * as a single clause, compute it's selectivity using mv stats,
+ * and compute the total selectivity using the current algorithm.
+ *
+ * TODO For RestrictInfo above an OR-clause, we might use the orclause
+ * with nested RestrictInfo - we won't have to call pull_varnos()
+ * for each clause, saving time.
+ *
+ * TODO Perhaps this needs a bit more thought for functional
+ * dependencies? Those don't quite work for NOT cases.
+ */
+ BoolExpr *expr = (BoolExpr *) node;
+ ListCell *lc;
+
+ foreach (lc, expr->args)
+ {
+ if (mv_compatible_walker((Node *) lfirst(lc), context))
+ return true;
+ }
+
+ return false;
+ }
+
+ if (IsA(node, NullTest))
+ {
+ NullTest* nt = (NullTest*)node;
+
+ /*
+ * Only simple (Var IS NULL) expressions supported for now. Maybe we could
+ * use examine_variable to fix this?
+ */
+ if (! IsA(nt->arg, Var))
+ return true;
+
+ return mv_compatible_walker((Node*)(nt->arg), context);
+ }
+
if (IsA(node, Var))
{
Var * var = (Var*)node;
@@ -1000,7 +1343,7 @@ mv_compatible_walker(Node *node, mv_compatible_context *context)
/* unsupported structure (two variables or so) */
if (! ok)
return true;
-
+
/*
* If it's not a "<" or ">" or "=" operator, just ignore the clause.
* Otherwise note the relid and attnum for the variable. This uses the
@@ -1010,10 +1353,18 @@ mv_compatible_walker(Node *node, mv_compatible_context *context)
switch (get_oprrest(expr->opno))
{
case F_EQSEL:
-
/* equality conditions are compatible with all statistics */
break;
+ case F_SCALARLTSEL:
+ case F_SCALARGTSEL:
+
+ /* not compatible with functional dependencies */
+ if (! (context->types & MV_CLAUSE_TYPE_MCV))
+ return true; /* terminate */
+
+ break;
+
default:
/* unknown estimator */
@@ -1024,11 +1375,11 @@ mv_compatible_walker(Node *node, mv_compatible_context *context)
return mv_compatible_walker((Node *) var, context);
}
-
+
/* Node not explicitly supported, so terminate */
return true;
}
-
+
/*
* Determines whether the clause is compatible with multivariate stats,
* and if it is, returns some additional information - varno (index
@@ -1047,10 +1398,11 @@ mv_compatible_walker(Node *node, mv_compatible_context *context)
* evaluate them using multivariate stats.
*/
static bool
-clause_is_mv_compatible(Node *clause, Index relid, AttrNumber *attnum)
+clause_is_mv_compatible(Node *clause, Index relid, Bitmapset **attnums, int types)
{
mv_compatible_context context;
+ context.types = types;
context.varno = relid;
context.varattnos = NULL; /* no attnums */
@@ -1058,7 +1410,7 @@ clause_is_mv_compatible(Node *clause, Index relid, AttrNumber *attnum)
return false;
/* remember the newly collected attnums */
- *attnum = bms_singleton_member(context.varattnos);
+ *attnums = bms_add_members(*attnums, context.varattnos);
return true;
}
@@ -1075,15 +1427,15 @@ fdeps_reduce_clauses(List *clauses, Index relid, Bitmapset *reduced_attnums)
foreach (lc, clauses)
{
- AttrNumber attnum = InvalidAttrNumber;
+ Bitmapset *attnums = NULL;
Node * clause = (Node*)lfirst(lc);
/* ignore clauses that are not compatible with functional dependencies */
- if (! clause_is_mv_compatible(clause, relid, &attnum))
+ if (! clause_is_mv_compatible(clause, relid, &attnums, MV_CLAUSE_TYPE_FDEP))
reduced_clauses = lappend(reduced_clauses, clause);
/* for equality clauses, only keep those not on reduced attributes */
- if (! bms_is_member(attnum, reduced_attnums))
+ if (! bms_is_subset(attnums, reduced_attnums))
reduced_clauses = lappend(reduced_clauses, clause);
}
@@ -1208,7 +1560,7 @@ clauselist_apply_dependencies(PlannerInfo *root, List *clauses,
return clauses;
/* collect attnums from clauses compatible with dependencies (equality) */
- clause_attnums = collect_mv_attnums(clauses, relid);
+ clause_attnums = collect_mv_attnums(clauses, relid, MV_CLAUSE_TYPE_FDEP);
/* decide which attnums may be eliminated */
reduced_attnums = fdeps_reduce_attnums(stats, clause_attnums);
@@ -1233,6 +1585,9 @@ stats_type_matches(MVStatisticInfo *stat, int type)
if ((type & MV_CLAUSE_TYPE_FDEP) && stat->deps_built)
return true;
+ if ((type & MV_CLAUSE_TYPE_MCV) && stat->mcv_built)
+ return true;
+
return false;
}
@@ -1266,3 +1621,392 @@ find_stats(PlannerInfo *root, Index relid)
return root->simple_rel_array[relid]->mvstatlist;
}
+
+/*
+ * Estimate selectivity of clauses using a MCV list.
+ *
+ * If there's no MCV list for the stats, the function returns 0.0.
+ *
+ * While computing the estimate, the function checks whether all the
+ * columns were matched with an equality condition. If that's the case,
+ * we can skip processing the histogram, as there can be no rows in
+ * it with the same values - all the rows matching the condition are
+ * represented by the MCV item. This can only happen with equality
+ * on all the attributes.
+ *
+ * The algorithm works like this:
+ *
+ * 1) mark all items as 'match'
+ * 2) walk through all the clauses
+ * 3) for a particular clause, walk through all the items
+ * 4) skip items that are already 'no match'
+ * 5) check clause for items that still match
+ * 6) sum frequencies for items to get selectivity
+ *
+ * The function also returns the frequency of the least frequent item
+ * on the MCV list, which may be useful for clamping estimate from the
+ * histogram (all items not present in the MCV list are less frequent).
+ * This however seems useful only for cases with conditions on all
+ * attributes.
+ *
+ * TODO This only handles AND-ed clauses, but it might work for OR-ed
+ * lists too - it just needs to reverse the logic a bit. I.e. start
+ * with 'no match' for all items, and mark the items as a match
+ * as the clauses are processed (and skip items that are 'match').
+ */
+static Selectivity
+clauselist_mv_selectivity_mcvlist(PlannerInfo *root, List *clauses,
+ MVStatisticInfo *mvstats, bool *fullmatch,
+ Selectivity *lowsel)
+{
+ int i;
+ Selectivity s = 0.0;
+ Selectivity u = 0.0;
+
+ MCVList mcvlist = NULL;
+ int nmatches = 0;
+
+ /* match/mismatch bitmap for each MCV item */
+ char * matches = NULL;
+
+ Assert(clauses != NIL);
+ Assert(list_length(clauses) >= 2);
+
+ /* there's no MCV list built yet */
+ if (! mvstats->mcv_built)
+ return 0.0;
+
+ mcvlist = load_mv_mcvlist(mvstats->mvoid);
+
+ Assert(mcvlist != NULL);
+ Assert(mcvlist->nitems > 0);
+
+ /* by default all the MCV items match the clauses fully */
+ matches = palloc0(sizeof(char) * mcvlist->nitems);
+ memset(matches, MVSTATS_MATCH_FULL, sizeof(char)*mcvlist->nitems);
+
+ /* number of matching MCV items */
+ nmatches = mcvlist->nitems;
+
+ nmatches = update_match_bitmap_mcvlist(root, clauses,
+ mvstats->stakeys, mcvlist,
+ nmatches, matches,
+ lowsel, fullmatch, false);
+
+ /* sum frequencies for all the matching MCV items */
+ for (i = 0; i < mcvlist->nitems; i++)
+ {
+ /* used to 'scale' for MCV lists not covering all tuples */
+ u += mcvlist->items[i]->frequency;
+
+ if (matches[i] != MVSTATS_MATCH_NONE)
+ s += mcvlist->items[i]->frequency;
+ }
+
+ pfree(matches);
+ pfree(mcvlist);
+
+ return s*u;
+}
+
+/*
+ * Evaluate clauses using the MCV list, and update the match bitmap.
+ *
+ * The bitmap may be already partially set, so this is really a way to
+ * combine results of several clause lists - either when computing
+ * conditional probability P(A|B) or a combination of AND/OR clauses.
+ *
+ * TODO This works with 'bitmap' where each bit is represented as a char,
+ * which is slightly wasteful. Instead, we could use a regular
+ * bitmap, reducing the size to ~1/8. Another thing is merging the
+ * bitmaps using & and |, which might be faster than min/max.
+ */
+static int
+update_match_bitmap_mcvlist(PlannerInfo *root, List *clauses,
+ int2vector *stakeys, MCVList mcvlist,
+ int nmatches, char * matches,
+ Selectivity *lowsel, bool *fullmatch,
+ bool is_or)
+{
+ int i;
+ ListCell * l;
+
+ Bitmapset *eqmatches = NULL; /* attributes with equality matches */
+
+ /* The bitmap may be partially built. */
+ Assert(nmatches >= 0);
+ Assert(nmatches <= mcvlist->nitems);
+ Assert(clauses != NIL);
+ Assert(list_length(clauses) >= 1);
+ Assert(mcvlist != NULL);
+ Assert(mcvlist->nitems > 0);
+
+ /* No possible matches (only works for AND-ded clauses) */
+ if (((nmatches == 0) && (! is_or)) ||
+ ((nmatches == mcvlist->nitems) && is_or))
+ return nmatches;
+
+ /*
+ * find the lowest frequency in the MCV list
+ *
+ * We need to do that here, because we do various tricks in the following
+ * code - skipping items already ruled out, etc.
+ *
+ * XXX A loop is necessary because the MCV list is not sorted by frequency.
+ */
+ *lowsel = 1.0;
+ for (i = 0; i < mcvlist->nitems; i++)
+ {
+ MCVItem item = mcvlist->items[i];
+
+ if (item->frequency < *lowsel)
+ *lowsel = item->frequency;
+ }
+
+ /*
+ * Loop through the list of clauses, and for each of them evaluate
+ * all the MCV items not yet eliminated by the preceding clauses.
+ */
+ foreach (l, clauses)
+ {
+ Node * clause = (Node*)lfirst(l);
+
+ /* if it's a RestrictInfo, then extract the clause */
+ if (IsA(clause, RestrictInfo))
+ clause = (Node*)((RestrictInfo*)clause)->clause;
+
+ /* if there are no remaining matches possible, we can stop */
+ if (((nmatches == 0) && (! is_or)) ||
+ ((nmatches == mcvlist->nitems) && is_or))
+ break;
+
+ /* it's either OpClause, or NullTest */
+ if (is_opclause(clause))
+ {
+ OpExpr *expr = (OpExpr*)clause;
+ bool varonleft = true;
+ bool ok;
+ FmgrInfo opproc;
+
+ /* get procedure computing operator selectivity */
+ RegProcedure oprrest = get_oprrest(expr->opno);
+
+ fmgr_info(get_opcode(expr->opno), &opproc);
+
+ ok = (NumRelids(clause) == 1) &&
+ (is_pseudo_constant_clause(lsecond(expr->args)) ||
+ (varonleft = false,
+ is_pseudo_constant_clause(linitial(expr->args))));
+
+ if (ok)
+ {
+
+ FmgrInfo gtproc;
+ Var * var = (varonleft) ? linitial(expr->args) : lsecond(expr->args);
+ Const * cst = (varonleft) ? lsecond(expr->args) : linitial(expr->args);
+ bool isgt = (! varonleft);
+
+ TypeCacheEntry *typecache
+ = lookup_type_cache(var->vartype, TYPECACHE_GT_OPR);
+
+ /* FIXME proper matching attribute to dimension */
+ int idx = mv_get_index(var->varattno, stakeys);
+
+ fmgr_info(get_opcode(typecache->gt_opr), >proc);
+
+ /*
+ * Walk through the MCV items and evaluate the current clause. We can
+ * skip items that were already ruled out, and terminate if there are
+ * no remaining MCV items that might possibly match.
+ */
+ for (i = 0; i < mcvlist->nitems; i++)
+ {
+ bool mismatch = false;
+ MCVItem item = mcvlist->items[i];
+
+ /*
+ * If there are no more matches (AND) or no remaining unmatched
+ * items (OR), we can stop processing this clause.
+ */
+ if (((nmatches == 0) && (! is_or)) ||
+ ((nmatches == mcvlist->nitems) && is_or))
+ break;
+
+ /*
+ * For AND-lists, we can also mark NULL items as 'no match' (and
+ * then skip them). For OR-lists this is not possible.
+ */
+ if ((! is_or) && item->isnull[idx])
+ matches[i] = MVSTATS_MATCH_NONE;
+
+ /* skip MCV items that were already ruled out */
+ if ((! is_or) && (matches[i] == MVSTATS_MATCH_NONE))
+ continue;
+ else if (is_or && (matches[i] == MVSTATS_MATCH_FULL))
+ continue;
+
+ switch (oprrest)
+ {
+ case F_EQSEL:
+ /*
+ * We don't care about isgt in equality, because it does not
+ * matter whether it's (var = const) or (const = var).
+ */
+ mismatch = ! DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ cst->constvalue,
+ item->values[idx]));
+
+ if (! mismatch)
+ eqmatches = bms_add_member(eqmatches, idx);
+
+ break;
+
+ case F_SCALARLTSEL: /* column < constant */
+ case F_SCALARGTSEL: /* column > constant */
+
+ /*
+ * First check whether the constant is below the lower boundary (in that
+ * case we can skip the bucket, because there's no overlap).
+ */
+ mismatch = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ cst->constvalue,
+ item->values[idx]));
+
+ /* invert the result if isgt=true */
+ mismatch = (isgt) ? (! mismatch) : mismatch;
+ break;
+ }
+
+ /* XXX The conditions on matches[i] are not needed, as we
+ * skip MCV items that can't become true/false, depending
+ * on the current flag. See beginning of the loop over
+ * MCV items.
+ */
+
+ if ((is_or) && (matches[i] == MVSTATS_MATCH_NONE) && (! mismatch))
+ {
+ /* OR - was MATCH_NONE, but will be MATCH_FULL */
+ matches[i] = MVSTATS_MATCH_FULL;
+ ++nmatches;
+ continue;
+ }
+ else if ((! is_or) && (matches[i] == MVSTATS_MATCH_FULL) && mismatch)
+ {
+ /* AND - was MATC_FULL, but will be MATCH_NONE */
+ matches[i] = MVSTATS_MATCH_NONE;
+ --nmatches;
+ continue;
+ }
+
+ }
+ }
+ }
+ else if (IsA(clause, NullTest))
+ {
+ NullTest * expr = (NullTest*)clause;
+ Var * var = (Var*)(expr->arg);
+
+ /* FIXME proper matching attribute to dimension */
+ int idx = mv_get_index(var->varattno, stakeys);
+
+ /*
+ * Walk through the MCV items and evaluate the current clause. We can
+ * skip items that were already ruled out, and terminate if there are
+ * no remaining MCV items that might possibly match.
+ */
+ for (i = 0; i < mcvlist->nitems; i++)
+ {
+ MCVItem item = mcvlist->items[i];
+
+ /* if there are no more matches, we can stop processing this clause */
+ if (nmatches == 0)
+ break;
+
+ /* skip MCV items that were already ruled out */
+ if (matches[i] == MVSTATS_MATCH_NONE)
+ continue;
+
+ /* if the clause mismatches the MCV item, set it as MATCH_NONE */
+ if (((expr->nulltesttype == IS_NULL) && (! item->isnull[idx])) ||
+ ((expr->nulltesttype == IS_NOT_NULL) && (item->isnull[idx])))
+ {
+ matches[i] = MVSTATS_MATCH_NONE;
+ --nmatches;
+ }
+ }
+ }
+ else if (or_clause(clause) || and_clause(clause))
+ {
+ /* AND/OR clause, with all clauses compatible with the selected MV stat */
+
+ int i;
+ BoolExpr *orclause = ((BoolExpr*)clause);
+ List *orclauses = orclause->args;
+
+ /* match/mismatch bitmap for each MCV item */
+ int or_nmatches = 0;
+ char * or_matches = NULL;
+
+ Assert(orclauses != NIL);
+ Assert(list_length(orclauses) >= 2);
+
+ /* number of matching MCV items */
+ or_nmatches = mcvlist->nitems;
+
+ /* by default none of the MCV items matches the clauses */
+ or_matches = palloc0(sizeof(char) * or_nmatches);
+
+ if (or_clause(clause))
+ {
+ /* OR clauses assume nothing matches, initially */
+ memset(or_matches, MVSTATS_MATCH_NONE, sizeof(char)*or_nmatches);
+ or_nmatches = 0;
+ }
+ else
+ {
+ /* AND clauses assume nothing matches, initially */
+ memset(or_matches, MVSTATS_MATCH_FULL, sizeof(char)*or_nmatches);
+ }
+
+ /* build the match bitmap for the OR-clauses */
+ or_nmatches = update_match_bitmap_mcvlist(root, orclauses,
+ stakeys, mcvlist,
+ or_nmatches, or_matches,
+ lowsel, fullmatch, or_clause(clause));
+
+ /* merge the bitmap into the existing one*/
+ for (i = 0; i < mcvlist->nitems; i++)
+ {
+ /*
+ * To AND-merge the bitmaps, a MIN() semantics is used.
+ * For OR-merge, use MAX().
+ *
+ * FIXME this does not decrease the number of matches
+ */
+ UPDATE_RESULT(matches[i], or_matches[i], is_or);
+ }
+
+ pfree(or_matches);
+
+ }
+ else
+ {
+ elog(ERROR, "unknown clause type: %d", clause->type);
+ }
+ }
+
+ /*
+ * If all the columns were matched by equality, it's a full match.
+ * In this case there can be just a single MCV item, matching the
+ * clause (if there were two, both would match the other one).
+ */
+ *fullmatch = (bms_num_members(eqmatches) == mcvlist->ndimensions);
+
+ /* free the allocated pieces */
+ if (eqmatches)
+ pfree(eqmatches);
+
+ return nmatches;
+}
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index 7fb2088..8394111 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -412,7 +412,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
mvstat = (Form_pg_mv_statistic) GETSTRUCT(htup);
/* unavailable stats are not interesting for the planner */
- if (mvstat->deps_built)
+ if (mvstat->deps_built || mvstat->mcv_built)
{
info = makeNode(MVStatisticInfo);
@@ -421,9 +421,11 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
/* enabled statistics */
info->deps_enabled = mvstat->deps_enabled;
+ info->mcv_enabled = mvstat->mcv_enabled;
/* built/available statistics */
info->deps_built = mvstat->deps_built;
+ info->mcv_built = mvstat->mcv_built;
/* stakeys */
adatum = SysCacheGetAttr(MVSTATOID, htup,
diff --git a/src/backend/utils/mvstats/Makefile b/src/backend/utils/mvstats/Makefile
index 099f1ed..f9bf10c 100644
--- a/src/backend/utils/mvstats/Makefile
+++ b/src/backend/utils/mvstats/Makefile
@@ -12,6 +12,6 @@ subdir = src/backend/utils/mvstats
top_builddir = ../../../..
include $(top_builddir)/src/Makefile.global
-OBJS = common.o dependencies.o
+OBJS = common.o dependencies.o mcv.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/utils/mvstats/README.mcv b/src/backend/utils/mvstats/README.mcv
new file mode 100644
index 0000000..e93cfe4
--- /dev/null
+++ b/src/backend/utils/mvstats/README.mcv
@@ -0,0 +1,137 @@
+MCV lists
+=========
+
+Multivariate MCV (most-common values) lists are a straightforward extension of
+regular MCV list, tracking most frequent combinations of values for a group of
+attributes.
+
+This works particularly well for columns with a small number of distinct values,
+as the list may include all the combinations and approximate the distribution
+very accurately.
+
+For columns with large number of distinct values (e.g. those with continuous
+domains), the list will only track the most frequent combinations. If the
+distribution is mostly uniform (all combinations about equally frequent), the
+MCV list will be empty.
+
+Estimates of some clauses (e.g. equality) based on MCV lists are more accurate
+than when using histograms.
+
+Also, MCV lists don't necessarily require sorting of the values (the fact that
+we use sorting when building them is implementation detail), but even more
+importantly the ordering is not built into the approximation (while histograms
+are built on ordering). So MCV lists work well even for attributes where the
+ordering of the data type is disconnected from the meaning of the data. For
+example we know how to sort strings, but it's unlikely to make much sense for
+city names (or other label-like attributes).
+
+
+Selectivity estimation
+----------------------
+
+The estimation, implemented in clauselist_mv_selectivity_mcvlist(), is quite
+simple in principle - we need to identify MCV items matching all the clauses
+and sum frequencies of all those items.
+
+Currently MCV lists support estimation of the following clause types:
+
+ (a) equality clauses WHERE (a = 1) AND (b = 2)
+ (b) inequality clauses WHERE (a < 1) AND (b >= 2)
+ (c) NULL clauses WHERE (a IS NULL) AND (b IS NOT NULL)
+ (d) OR clauses WHERE (a < 1) OR (b >= 2)
+
+It's possible to add support for additional clauses, for example:
+
+ (e) multi-var clauses WHERE (a > b)
+
+and possibly others. These are tasks for the future, not yet implemented.
+
+
+Estimating equality clauses
+---------------------------
+
+When computing selectivity estimate for equality clauses
+
+ (a = 1) AND (b = 2)
+
+we can do this estimate pretty exactly assuming that two conditions are met:
+
+ (1) there's an equality condition on all attributes of the statistic
+
+ (2) we find a matching item in the MCV list
+
+In this case we know the MCV item represents all tuples matching the clauses,
+and the selectivity estimate is complete (i.e. we don't need to perform
+estimation using the histogram). This is what we call 'full match'.
+
+When only (1) holds, but there's no matching MCV item, we don't know whether
+there are no such rows or just are not very frequent. We can however use the
+frequency of the least frequent MCV item as an upper bound for the selectivity.
+
+For a combination of equality conditions (not full-match case) we can clamp the
+selectivity by the minimum of selectivities for each condition. For example if
+we know the number of distinct values for each column, we can use 1/ndistinct
+as a per-column estimate. Or rather 1/ndistinct + selectivity derived from the
+MCV list.
+
+We should also probably only use the 'residual ndistinct' by exluding the items
+included in the MCV list (and also residual frequency):
+
+ f = (1.0 - sum(MCV frequencies)) / (ndistinct - ndistinct(MCV list))
+
+but it's worth pointing out the ndistinct values are multi-variate for the
+columns referenced by the equality conditions.
+
+Note: Only the "full match" limit is currently implemented.
+
+
+Hashed MCV (not yet implemented)
+--------------------------------
+
+Regular MCV lists have to include actual values for each item, so if those items
+are large the list may be quite large. This is especially true for multi-variate
+MCV lists, although the current implementation partially mitigates this by
+performing de-duplicating the values before storing them on disk.
+
+It's possible to only store hashes (32-bit values) instead of the actual values,
+significantly reducing the space requirements. Obviously, this would only make
+the MCV lists useful for estimating equality conditions (assuming the 32-bit
+hashes make the collisions rare enough).
+
+This might also complicate matching the columns to available stats.
+
+
+TODO Consider implementing hashed MCV list, storing just 32-bit hashes instead
+ of the actual values. This type of MCV list will be useful only for
+ estimating equality clauses, and will reduce space requirements for large
+ varlena types (in such cases we usually only want equality anyway).
+
+TODO Currently there's no logic to consider building only a MCV list (and not
+ building the histogram at all), except for doing this decision manually in
+ ADD STATISTICS.
+
+
+Inspecting the MCV list
+-----------------------
+
+Inspecting the regular (per-attribute) MCV lists is trivial, as it's enough
+to select the columns from pg_stats - the data is encoded as anyarrays, so we
+simply get the text representation of the arrays.
+
+With multivariate MCV lits it's not that simple due to the possible mix of
+data types. It might be possible to produce similar array-like representation,
+but that'd unnecessarily complicate further processing and analysis of the MCV
+list. Instead, there's a SRF function providing values, frequencies etc.
+
+ SELECT * FROM pg_mv_mcv_items();
+
+It has two input parameters:
+
+ oid - OID of the MCV list (pg_mv_statistic.staoid)
+
+and produces a table with these columns:
+
+ - item ID (0...nitems-1)
+ - values (string array)
+ - nulls only (boolean array)
+ - frequency (double precision)
diff --git a/src/backend/utils/mvstats/README.stats b/src/backend/utils/mvstats/README.stats
index a38ea7b..5c5c59a 100644
--- a/src/backend/utils/mvstats/README.stats
+++ b/src/backend/utils/mvstats/README.stats
@@ -8,9 +8,50 @@ not true, resulting in estimation errors.
Multivariate stats track different types of dependencies between the columns,
hopefully improving the estimates.
-Currently we only have one kind of multivariate statistics - soft functional
-dependencies, and we use it to improve estimates of equality clauses. See
-README.dependencies for details.
+
+Types of statistics
+-------------------
+
+Currently we only have two kinds of multivariate statistics
+
+ (a) soft functional dependencies (README.dependencies)
+
+ (b) MCV lists (README.mcv)
+
+
+Compatible clause types
+-----------------------
+
+Each type of statistics may be used to estimate some subset of clause types.
+
+ (a) functional dependencies - equality clauses (AND), possibly IS NULL
+
+ (b) MCV list - equality and inequality clauses, IS [NOT] NULL, AND/OR
+
+Currently only simple operator clauses (Var op Const) are supported, but it's
+possible to support more complex clause types, e.g. (Var op Var).
+
+
+Complex clauses
+---------------
+
+We also support estimating more complex clauses - essentially AND/OR clauses
+with (Var op Const) as leaves, as long as all the referenced attributes are
+covered by a single statistics.
+
+For example this condition
+
+ (a=1) AND ((b=2) OR ((c=3) AND (d=4)))
+
+may be estimated using statistics on (a,b,c,d). If we only have statistics on
+(b,c,d) we may estimate the second part, and estimate (a=1) using simple stats.
+
+If we only have statistics on (a,b,c) we can't apply it at all at this point,
+but it's worth pointing out clauselist_selectivity() works recursively and when
+handling the second part (the OR-clause), we'll be able to apply the statistics.
+
+Note: The multi-statistics estimation patch also makes it possible to pass some
+clauses as 'conditions' into the deeper parts of the expression tree.
Selectivity estimation
@@ -23,14 +64,48 @@ When estimating selectivity, we aim to achieve several things:
(b) minimize the overhead, especially when no suitable multivariate stats
exist (so if you are not using multivariate stats, there's no overhead)
-This clauselist_selectivity() performs several inexpensive checks first, before
+Thus clauselist_selectivity() performs several inexpensive checks first, before
even attempting to do the more expensive estimation.
(1) check if there are multivariate stats on the relation
- (2) check there are at least two attributes referenced by clauses compatible
- with multivariate statistics (equality clauses for func. dependencies)
+ (2) check that there are functional dependencies on the table, and that
+ there are at least two attributes referenced by compatible clauses
+ (equality clauses for func. dependencies)
(3) perform reduction of equality clauses using func. dependencies
- (4) estimate the reduced list of clauses using regular statistics
+ (4) check that there are multivariate MCV lists on the table, and that
+ there are at least two attributes referenced by compatible clauses
+ (equalities, inequalities, etc.)
+
+ (5) find the best multivariate statistics (matching the most conditions)
+ and use it to compute the estimate
+
+ (6) estimate the remaining clauses (not estimated using multivariate stats)
+ using the regular per-column statistics
+
+Whenever we find there are no suitable stats, we skip the expensive steps.
+
+
+Further (possibly crazy) ideas
+------------------------------
+
+Currently the clauses are only estimated using a single statistics, even if
+there are multiple candidate statistics - for example assume we have statistics
+on (a,b,c) and (b,c,d), and estimate conditions
+
+ (b = 1) AND (c = 2)
+
+Then both statistics may be used, but we only use one of them. Maybe we could
+use compute estimates using all candidate stats, and somehow aggregate them
+into the final estimate by using average or median.
+
+Some stats may give better estimates than others, but it's very difficult to say
+in advance which stats are the best (it depends on the number of buckets, number
+of additional columns not referenced in the clauses, type of condition etc.).
+
+But of course, this may result in expensive estimation (CPU-wise).
+
+So we might add a GUC to choose between a simple (single statistics) and thus
+multi-statistic estimation, possibly table-level parameter (ALTER TABLE ...).
diff --git a/src/backend/utils/mvstats/common.c b/src/backend/utils/mvstats/common.c
index dcb7c78..4f5a842 100644
--- a/src/backend/utils/mvstats/common.c
+++ b/src/backend/utils/mvstats/common.c
@@ -16,12 +16,14 @@
#include "common.h"
+#include "utils/array.h"
+
static VacAttrStats ** lookup_var_attr_stats(int2vector *attrs,
- int natts, VacAttrStats **vacattrstats);
+ int natts,
+ VacAttrStats **vacattrstats);
static List* list_mv_stats(Oid relid);
-
/*
* Compute requested multivariate stats, using the rows sampled for the
* plain (single-column) stats.
@@ -49,6 +51,8 @@ build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
int j;
MVStatisticInfo *stat = (MVStatisticInfo *)lfirst(lc);
MVDependencies deps = NULL;
+ MCVList mcvlist = NULL;
+ int numrows_filtered = 0;
VacAttrStats **stats = NULL;
int numatts = 0;
@@ -87,8 +91,12 @@ build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
if (stat->deps_enabled)
deps = build_mv_dependencies(numrows, rows, attrs, stats);
+ /* build the MCV list */
+ if (stat->mcv_enabled)
+ mcvlist = build_mv_mcvlist(numrows, rows, attrs, stats, &numrows_filtered);
+
/* store the histogram / MCV list in the catalog */
- update_mv_stats(stat->mvoid, deps, attrs);
+ update_mv_stats(stat->mvoid, deps, mcvlist, attrs, stats);
}
}
@@ -166,6 +174,8 @@ list_mv_stats(Oid relid)
info->stakeys = buildint2vector(stats->stakeys.values, stats->stakeys.dim1);
info->deps_enabled = stats->deps_enabled;
info->deps_built = stats->deps_built;
+ info->mcv_enabled = stats->mcv_enabled;
+ info->mcv_built = stats->mcv_built;
result = lappend(result, info);
}
@@ -180,8 +190,56 @@ list_mv_stats(Oid relid)
return result;
}
+
+/*
+ * Find attnims of MV stats using the mvoid.
+ */
+int2vector*
+find_mv_attnums(Oid mvoid, Oid *relid)
+{
+ ArrayType *arr;
+ Datum adatum;
+ bool isnull;
+ HeapTuple htup;
+ int2vector *keys;
+
+ /* Prepare to scan pg_mv_statistic for entries having indrelid = this rel. */
+ htup = SearchSysCache1(MVSTATOID,
+ ObjectIdGetDatum(mvoid));
+
+ /* XXX syscache contains OIDs of deleted stats (not invalidated) */
+ if (! HeapTupleIsValid(htup))
+ return NULL;
+
+ /* starelid */
+ adatum = SysCacheGetAttr(MVSTATOID, htup,
+ Anum_pg_mv_statistic_starelid, &isnull);
+ Assert(!isnull);
+
+ *relid = DatumGetObjectId(adatum);
+
+ /* stakeys */
+ adatum = SysCacheGetAttr(MVSTATOID, htup,
+ Anum_pg_mv_statistic_stakeys, &isnull);
+ Assert(!isnull);
+
+ arr = DatumGetArrayTypeP(adatum);
+
+ keys = buildint2vector((int16 *) ARR_DATA_PTR(arr),
+ ARR_DIMS(arr)[0]);
+ ReleaseSysCache(htup);
+
+ /* TODO maybe save the list into relcache, as in RelationGetIndexList
+ * (which was used as an inspiration of this one)?. */
+
+ return keys;
+}
+
+
void
-update_mv_stats(Oid mvoid, MVDependencies dependencies, int2vector *attrs)
+update_mv_stats(Oid mvoid,
+ MVDependencies dependencies, MCVList mcvlist,
+ int2vector *attrs, VacAttrStats **stats)
{
HeapTuple stup,
oldtup;
@@ -206,18 +264,29 @@ update_mv_stats(Oid mvoid, MVDependencies dependencies, int2vector *attrs)
= PointerGetDatum(serialize_mv_dependencies(dependencies));
}
+ if (mcvlist != NULL)
+ {
+ bytea * data = serialize_mv_mcvlist(mcvlist, attrs, stats);
+ nulls[Anum_pg_mv_statistic_stamcv -1] = (data == NULL);
+ values[Anum_pg_mv_statistic_stamcv - 1] = PointerGetDatum(data);
+ }
+
/* always replace the value (either by bytea or NULL) */
replaces[Anum_pg_mv_statistic_stadeps -1] = true;
+ replaces[Anum_pg_mv_statistic_stamcv -1] = true;
/* always change the availability flags */
nulls[Anum_pg_mv_statistic_deps_built -1] = false;
+ nulls[Anum_pg_mv_statistic_mcv_built -1] = false;
nulls[Anum_pg_mv_statistic_stakeys-1] = false;
/* use the new attnums, in case we removed some dropped ones */
replaces[Anum_pg_mv_statistic_deps_built-1] = true;
+ replaces[Anum_pg_mv_statistic_mcv_built -1] = true;
replaces[Anum_pg_mv_statistic_stakeys -1] = true;
values[Anum_pg_mv_statistic_deps_built-1] = BoolGetDatum(dependencies != NULL);
+ values[Anum_pg_mv_statistic_mcv_built -1] = BoolGetDatum(mcvlist != NULL);
values[Anum_pg_mv_statistic_stakeys -1] = PointerGetDatum(attrs);
/* Is there already a pg_mv_statistic tuple for this attribute? */
@@ -246,6 +315,21 @@ update_mv_stats(Oid mvoid, MVDependencies dependencies, int2vector *attrs)
heap_close(sd, RowExclusiveLock);
}
+
+int
+mv_get_index(AttrNumber varattno, int2vector * stakeys)
+{
+ int i, idx = 0;
+ for (i = 0; i < stakeys->dim1; i++)
+ {
+ if (stakeys->values[i] < varattno)
+ idx += 1;
+ else
+ break;
+ }
+ return idx;
+}
+
/* multi-variate stats comparator */
/*
@@ -256,11 +340,15 @@ update_mv_stats(Oid mvoid, MVDependencies dependencies, int2vector *attrs)
int
compare_scalars_simple(const void *a, const void *b, void *arg)
{
- Datum da = *(Datum*)a;
- Datum db = *(Datum*)b;
- SortSupport ssup= (SortSupport) arg;
+ return compare_datums_simple(*(Datum*)a,
+ *(Datum*)b,
+ (SortSupport)arg);
+}
- return ApplySortComparator(da, false, db, false, ssup);
+int
+compare_datums_simple(Datum a, Datum b, SortSupport ssup)
+{
+ return ApplySortComparator(a, false, b, false, ssup);
}
/*
@@ -377,3 +465,32 @@ multi_sort_compare_dims(int start, int end,
return 0;
}
+
+/* simple counterpart to qsort_arg */
+void *
+bsearch_arg(const void *key, const void *base, size_t nmemb, size_t size,
+ int (*compar) (const void *, const void *, void *),
+ void *arg)
+{
+ size_t l, u, idx;
+ const void *p;
+ int comparison;
+
+ l = 0;
+ u = nmemb;
+ while (l < u)
+ {
+ idx = (l + u) / 2;
+ p = (void *) (((const char *) base) + (idx * size));
+ comparison = (*compar) (key, p, arg);
+
+ if (comparison < 0)
+ u = idx;
+ else if (comparison > 0)
+ l = idx + 1;
+ else
+ return (void *) p;
+ }
+
+ return NULL;
+}
diff --git a/src/backend/utils/mvstats/common.h b/src/backend/utils/mvstats/common.h
index a019ea6..350760b 100644
--- a/src/backend/utils/mvstats/common.h
+++ b/src/backend/utils/mvstats/common.h
@@ -46,7 +46,15 @@ typedef struct
Datum value; /* a data value */
int tupno; /* position index for tuple it came from */
} ScalarItem;
-
+
+/* (de)serialization info */
+typedef struct DimensionInfo {
+ int nvalues; /* number of deduplicated values */
+ int nbytes; /* number of bytes (serialized) */
+ int typlen; /* pg_type.typlen */
+ bool typbyval; /* pg_type.typbyval */
+} DimensionInfo;
+
/* multi-sort */
typedef struct MultiSortSupportData {
int ndims; /* number of dimensions supported by the */
@@ -58,6 +66,7 @@ typedef MultiSortSupportData* MultiSortSupport;
typedef struct SortItem {
Datum *values;
bool *isnull;
+ int count;
} SortItem;
MultiSortSupport multi_sort_init(int ndims);
@@ -74,5 +83,11 @@ int multi_sort_compare_dims(int start, int end, const SortItem *a,
const SortItem *b, MultiSortSupport mss);
/* comparators, used when constructing multivariate stats */
+int compare_datums_simple(Datum a, Datum b, SortSupport ssup);
int compare_scalars_simple(const void *a, const void *b, void *arg);
int compare_scalars_partition(const void *a, const void *b, void *arg);
+
+void * bsearch_arg(const void *key, const void *base,
+ size_t nmemb, size_t size,
+ int (*compar) (const void *, const void *, void *),
+ void *arg);
diff --git a/src/backend/utils/mvstats/mcv.c b/src/backend/utils/mvstats/mcv.c
new file mode 100644
index 0000000..b300c1a
--- /dev/null
+++ b/src/backend/utils/mvstats/mcv.c
@@ -0,0 +1,1120 @@
+/*-------------------------------------------------------------------------
+ *
+ * mcv.c
+ * POSTGRES multivariate MCV lists
+ *
+ *
+ * Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/utils/mvstats/mcv.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "fmgr.h"
+#include "funcapi.h"
+
+#include "utils/lsyscache.h"
+
+#include "common.h"
+
+/*
+ * Each serialized item needs to store (in this order):
+ *
+ * - indexes (ndim * sizeof(uint16))
+ * - null flags (ndim * sizeof(bool))
+ * - frequency (sizeof(double))
+ *
+ * So in total:
+ *
+ * ndim * (sizeof(uint16) + sizeof(bool)) + sizeof(double)
+ */
+#define ITEM_SIZE(ndims) \
+ (ndims * (sizeof(uint16) + sizeof(bool)) + sizeof(double))
+
+/* Macros for convenient access to parts of the serialized MCV item */
+#define ITEM_INDEXES(item) ((uint16*)item)
+#define ITEM_NULLS(item,ndims) ((bool*)(ITEM_INDEXES(item) + ndims))
+#define ITEM_FREQUENCY(item,ndims) ((double*)(ITEM_NULLS(item,ndims) + ndims))
+
+static MultiSortSupport build_mss(VacAttrStats **stats, int2vector *attrs);
+
+static SortItem *build_sorted_items(int numrows, HeapTuple *rows,
+ TupleDesc tdesc, MultiSortSupport mss,
+ int2vector *attrs);
+
+static SortItem *build_distinct_groups(int numrows, SortItem *items,
+ MultiSortSupport mss, int *ndistinct);
+
+static int count_distinct_groups(int numrows, SortItem *items,
+ MultiSortSupport mss);
+
+/*
+ * Builds MCV list from the set of sampled rows.
+ *
+ * The algorithm is quite simple:
+ *
+ * (1) sort the data (default collation, '<' for the data type)
+ *
+ * (2) count distinct groups, decide how many to keep
+ *
+ * (3) build the MCV list using the threshold determined in (2)
+ *
+ * (4) remove rows represented by the MCV from the sample
+ *
+ * The method also removes rows matching the MCV items from the input array,
+ * and passes the number of remaining rows (useful for building histograms)
+ * using the numrows_filtered parameter.
+ *
+ * FIXME Use max_mcv_items from ALTER TABLE ADD STATISTICS command.
+ *
+ * FIXME Single-dimensional MCV is sorted by frequency (descending). We should
+ * do that too, because when walking through the list we want to check
+ * the most frequent items first.
+ *
+ * TODO We're using Datum (8B), even for data types (e.g. int4 or float4).
+ * Maybe we could save some space here, but the bytea compression should
+ * handle it just fine.
+ *
+ * TODO This probably should not use the ndistinct directly (as computed from
+ * the table, but rather estimate the number of distinct values in the
+ * table), no?
+ */
+MCVList
+build_mv_mcvlist(int numrows, HeapTuple *rows, int2vector *attrs,
+ VacAttrStats **stats, int *numrows_filtered)
+{
+ int i;
+ int numattrs = attrs->dim1;
+ int ndistinct = 0;
+ int mcv_threshold = 0;
+ int nitems = 0;
+
+ MCVList mcvlist = NULL;
+
+ /* comparator for all the columns */
+ MultiSortSupport mss = build_mss(stats, attrs);
+
+ /* sort the rows */
+ SortItem *items = build_sorted_items(numrows, rows, stats[0]->tupDesc,
+ mss, attrs);
+
+ /* transform the sorted rows into groups (sorted by frequency) */
+ SortItem *groups = build_distinct_groups(numrows, items, mss, &ndistinct);
+
+ /*
+ * Determine the minimum size of a group to be eligible for MCV list, and
+ * check how many groups actually pass that threshold. We use 1.25x the
+ * avarage group size, just like for regular statistics.
+ *
+ * But if we can fit all the distinct values in the MCV list (i.e. if there
+ * are less distinct groups than MVSTAT_MCVLIST_MAX_ITEMS), we'll require
+ * only 2 rows per group.
+ *
+ * FIXME This should really reference mcv_max_items (from catalog) instead
+ * of the constant MVSTAT_MCVLIST_MAX_ITEMS.
+ */
+ mcv_threshold = 1.25 * numrows / ndistinct;
+ mcv_threshold = (mcv_threshold < 4) ? 4 : mcv_threshold;
+
+ if (ndistinct <= MVSTAT_MCVLIST_MAX_ITEMS)
+ mcv_threshold = 2;
+
+ /* Walk through the groups and stop once we fall below the threshold. */
+ nitems = 0;
+ for (i = 0; i < ndistinct; i++)
+ {
+ if (groups[i].count < mcv_threshold)
+ break;
+
+ nitems++;
+ }
+
+ /* we know the number of MCV list items, so let's build the list */
+ if (nitems > 0)
+ {
+ /* allocate the MCV list structure, set parameters we know */
+ mcvlist = (MCVList)palloc0(sizeof(MCVListData));
+
+ mcvlist->magic = MVSTAT_MCV_MAGIC;
+ mcvlist->type = MVSTAT_MCV_TYPE_BASIC;
+ mcvlist->ndimensions = numattrs;
+ mcvlist->nitems = nitems;
+
+ /*
+ * Preallocate Datum/isnull arrays (not as a single chunk, as we will
+ * pass the result outside and thus it needs to be easy to pfree().
+ *
+ * XXX Although we're the only ones dealing with this.
+ */
+ mcvlist->items = (MCVItem*)palloc0(sizeof(MCVItem)*nitems);
+
+ for (i = 0; i < nitems; i++)
+ {
+ mcvlist->items[i] = (MCVItem)palloc0(sizeof(MCVItemData));
+ mcvlist->items[i]->values = (Datum*)palloc0(sizeof(Datum)*numattrs);
+ mcvlist->items[i]->isnull = (bool*)palloc0(sizeof(bool)*numattrs);
+ }
+
+ /* Copy the first chunk of groups into the result. */
+ for (i = 0; i < nitems; i++)
+ {
+ /* just pointer to the proper place in the list */
+ MCVItem item = mcvlist->items[i];
+
+ /* copy values from the _previous_ group (last item of) */
+ memcpy(item->values, groups[i].values, sizeof(Datum) * numattrs);
+ memcpy(item->isnull, groups[i].isnull, sizeof(bool) * numattrs);
+
+ /* and finally the group frequency */
+ item->frequency = (double)groups[i].count / numrows;
+ }
+
+ /* make sure the loops are consistent */
+ Assert(nitems == mcvlist->nitems);
+
+ /*
+ * Remove the rows matching the MCV list (i.e. keep only rows that are
+ * not represented by the MCV list). We will first sort the groups
+ * by the keys (not by count) and then use binary search.
+ */
+ if (nitems > ndistinct)
+ {
+ int i, j;
+ int nfiltered = 0;
+
+ /* used for the searches */
+ SortItem key;
+
+ /* wfill this with data from the rows */
+ key.values = (Datum*)palloc0(numattrs * sizeof(Datum));
+ key.isnull = (bool*)palloc0(numattrs * sizeof(bool));
+
+ /*
+ * Sort the groups for bsearch_r (but only the items that actually
+ * made it to the MCV list).
+ */
+ qsort_arg((void *) groups, nitems, sizeof(SortItem),
+ multi_sort_compare, mss);
+
+ /* walk through the tuples, compare the values to MCV items */
+ for (i = 0; i < numrows; i++)
+ {
+ /* collect the key values from the row */
+ for (j = 0; j < numattrs; j++)
+ key.values[j]
+ = heap_getattr(rows[i], attrs->values[j],
+ stats[j]->tupDesc, &key.isnull[j]);
+
+ /* if not included in the MCV list, keep it in the array */
+ if (bsearch_arg(&key, groups, nitems, sizeof(SortItem),
+ multi_sort_compare, mss) == NULL)
+ rows[nfiltered++] = rows[i];
+ }
+
+ /* remember how many rows we actually kept */
+ *numrows_filtered = nfiltered;
+
+ /* free all the data used here */
+ pfree(key.values);
+ pfree(key.isnull);
+ }
+ else
+ /* the MCV list convers all the rows */
+ *numrows_filtered = 0;
+ }
+
+ pfree(items);
+ pfree(groups);
+
+ return mcvlist;
+}
+
+/* build MultiSortSupport for the attributes passed in attrs */
+static MultiSortSupport
+build_mss(VacAttrStats **stats, int2vector *attrs)
+{
+ int i;
+ int numattrs = attrs->dim1;
+
+ /* Sort by multiple columns (using array of SortSupport) */
+ MultiSortSupport mss = multi_sort_init(numattrs);
+
+ /* prepare the sort functions for all the attributes */
+ for (i = 0; i < numattrs; i++)
+ multi_sort_add_dimension(mss, i, i, stats);
+
+ return mss;
+}
+
+/* build sorted array of SortItem with values from rows */
+static SortItem *
+build_sorted_items(int numrows, HeapTuple *rows, TupleDesc tdesc,
+ MultiSortSupport mss, int2vector *attrs)
+{
+ int i, j, len;
+ int numattrs = attrs->dim1;
+ int nvalues = numrows * numattrs;
+
+ /*
+ * We won't allocate the arrays for each item independenly, but in one large
+ * chunk and then just set the pointers.
+ */
+ SortItem *items;
+ Datum *values;
+ bool *isnull;
+ char *ptr;
+
+ /* Compute the total amount of memory we need (both items and values). */
+ len = numrows * sizeof(SortItem) + nvalues * (sizeof(Datum) + sizeof(bool));
+
+ /* Allocate the memory and split it into the pieces. */
+ ptr = palloc0(len);
+
+ /* items to sort */
+ items = (SortItem*)ptr;
+ ptr += numrows * sizeof(SortItem);
+
+ /* values and null flags */
+ values = (Datum*)ptr;
+ ptr += nvalues * sizeof(Datum);
+
+ isnull = (bool*)ptr;
+ ptr += nvalues * sizeof(bool);
+
+ /* make sure we consumed the whole buffer exactly */
+ Assert((ptr - (char*)items) == len);
+
+ /* fix the pointers to Datum and bool arrays */
+ for (i = 0; i < numrows; i++)
+ {
+ items[i].values = &values[i * numattrs];
+ items[i].isnull = &isnull[i * numattrs];
+
+ /* load the values/null flags from sample rows */
+ for (j = 0; j < numattrs; j++)
+ {
+ items[i].values[j] = heap_getattr(rows[i],
+ attrs->values[j], /* attnum */
+ tdesc,
+ &items[i].isnull[j]); /* isnull */
+ }
+ }
+
+ /* do the sort, using the multi-sort */
+ qsort_arg((void *) items, numrows, sizeof(SortItem),
+ multi_sort_compare, mss);
+
+ return items;
+}
+
+/* count distinct combinations of SortItems in the array */
+static int
+count_distinct_groups(int numrows, SortItem *items, MultiSortSupport mss)
+{
+ int i;
+ int ndistinct;
+
+ ndistinct = 1;
+ for (i = 1; i < numrows; i++)
+ if (multi_sort_compare(&items[i], &items[i-1], mss) != 0)
+ ndistinct += 1;
+
+ return ndistinct;
+}
+
+/* compares frequencies of the SortItem entries (in descending order) */
+static int
+compare_sort_item_count(const void *a, const void *b)
+{
+ SortItem *ia = (SortItem *)a;
+ SortItem *ib = (SortItem *)b;
+
+ if (ia->count == ib->count)
+ return 0;
+ else if (ia->count > ib->count)
+ return -1;
+
+ return 1;
+}
+
+/* builds SortItems for distinct groups and counts the matching items */
+static SortItem *
+build_distinct_groups(int numrows, SortItem *items, MultiSortSupport mss,
+ int *ndistinct)
+{
+ int i, j;
+ int ngroups = count_distinct_groups(numrows, items, mss);
+
+ SortItem *groups = (SortItem*)palloc0(ngroups * sizeof(SortItem));
+
+ j = 0;
+ groups[0] = items[0];
+ groups[0].count = 1;
+
+ for (i = 1; i < numrows; i++)
+ {
+ if (multi_sort_compare(&items[i], &items[i-1], mss) != 0)
+ groups[++j] = items[i];
+
+ groups[j].count++;
+ }
+
+ pg_qsort((void *) groups, ngroups, sizeof(SortItem),
+ compare_sort_item_count);
+
+ *ndistinct = ngroups;
+ return groups;
+}
+
+
+/* fetch the MCV list (as a bytea) from the pg_mv_statistic catalog */
+MCVList
+load_mv_mcvlist(Oid mvoid)
+{
+ bool isnull = false;
+ Datum mcvlist;
+
+#ifdef USE_ASSERT_CHECKING
+ Form_pg_mv_statistic mvstat;
+#endif
+
+ /* Prepare to scan pg_mv_statistic for entries having indrelid = this rel. */
+ HeapTuple htup = SearchSysCache1(MVSTATOID, ObjectIdGetDatum(mvoid));
+
+ if (! HeapTupleIsValid(htup))
+ return NULL;
+
+#ifdef USE_ASSERT_CHECKING
+ mvstat = (Form_pg_mv_statistic) GETSTRUCT(htup);
+ Assert(mvstat->mcv_enabled && mvstat->mcv_built);
+#endif
+
+ mcvlist = SysCacheGetAttr(MVSTATOID, htup,
+ Anum_pg_mv_statistic_stamcv, &isnull);
+
+ Assert(!isnull);
+
+ ReleaseSysCache(htup);
+
+ return deserialize_mv_mcvlist(DatumGetByteaP(mcvlist));
+}
+
+/* print some basic info about the MCV list
+ *
+ * TODO Add info about what part of the table this covers.
+ */
+Datum
+pg_mv_stats_mcvlist_info(PG_FUNCTION_ARGS)
+{
+ bytea *data = PG_GETARG_BYTEA_P(0);
+ char *result;
+
+ MCVList mcvlist = deserialize_mv_mcvlist(data);
+
+ result = palloc0(128);
+ snprintf(result, 128, "nitems=%d", mcvlist->nitems);
+
+ pfree(mcvlist);
+
+ PG_RETURN_TEXT_P(cstring_to_text(result));
+}
+
+/*
+ * serialize MCV list into a bytea value
+ *
+ *
+ * The basic algorithm is simple:
+ *
+ * (1) perform deduplication (for each attribute separately)
+ * (a) collect all (non-NULL) attribute values from all MCV items
+ * (b) sort the data (using 'lt' from VacAttrStats)
+ * (c) remove duplicate values from the array
+ *
+ * (2) serialize the arrays into a bytea value
+ *
+ * (3) process all MCV list items
+ * (a) replace values with indexes into the arrays
+ *
+ * Each attribute has to be processed separately, because we may be mixing
+ * different datatypes, with different sort operators, etc.
+ *
+ * We'll use uint16 values for the indexes in step (3), as we don't allow more
+ * than 8k MCV items (see list max_mcv_items), although that's mostly arbitrary
+ * limit. We might increase this to 65k and still fit into uint16.
+ *
+ * We don't really expect the serialization to save as much space as for
+ * histograms, because we are not doing any bucket splits (which is the source
+ * of high redundancy in histograms).
+ *
+ * TODO Consider packing boolean flags (NULL) for each item into a single char
+ * (or a longer type) instead of using an array of bool items.
+ */
+bytea *
+serialize_mv_mcvlist(MCVList mcvlist, int2vector *attrs,
+ VacAttrStats **stats)
+{
+ int i, j;
+ int ndims = mcvlist->ndimensions;
+ int itemsize = ITEM_SIZE(ndims);
+
+ SortSupport ssup;
+ DimensionInfo *info;
+
+ Size total_length;
+
+ /* allocate just once */
+ char *item = palloc0(itemsize);
+
+ /* serialized items (indexes into arrays, etc.) */
+ bytea *output;
+ char *data = NULL;
+
+ /* values per dimension (and number of non-NULL values) */
+ Datum **values = (Datum**)palloc0(sizeof(Datum*) * ndims);
+ int *counts = (int*)palloc0(sizeof(int) * ndims);
+
+ /*
+ * We'll include some rudimentary information about the attributes (type
+ * length, etc.), so that we don't have to look them up while deserializing
+ * the MCV list.
+ */
+ info = (DimensionInfo *)palloc0(sizeof(DimensionInfo)*ndims);
+
+ /* sort support data for all attributes included in the MCV list */
+ ssup = (SortSupport)palloc0(sizeof(SortSupportData)*ndims);
+
+ /* collect and deduplicate values for all attributes */
+ for (i = 0; i < ndims; i++)
+ {
+ int ndistinct;
+ StdAnalyzeData *tmp = (StdAnalyzeData *)stats[i]->extra_data;
+
+ /* copy important info about the data type (length, by-value) */
+ info[i].typlen = stats[i]->attrtype->typlen;
+ info[i].typbyval = stats[i]->attrtype->typbyval;
+
+ /* allocate space for values in the attribute and collect them */
+ values[i] = (Datum*)palloc0(sizeof(Datum) * mcvlist->nitems);
+
+ for (j = 0; j < mcvlist->nitems; j++)
+ {
+ /* skip NULL values - we don't need to serialize them */
+ if (mcvlist->items[j]->isnull[i])
+ continue;
+
+ values[i][counts[i]] = mcvlist->items[j]->values[i];
+ counts[i] += 1;
+ }
+
+ /* there are just NULL values in this dimension, we're done */
+ if (counts[i] == 0)
+ continue;
+
+ /* sort and deduplicate the data */
+ ssup[i].ssup_cxt = CurrentMemoryContext;
+ ssup[i].ssup_collation = DEFAULT_COLLATION_OID;
+ ssup[i].ssup_nulls_first = false;
+
+ PrepareSortSupportFromOrderingOp(tmp->ltopr, &ssup[i]);
+
+ qsort_arg(values[i], counts[i], sizeof(Datum),
+ compare_scalars_simple, &ssup[i]);
+
+ /*
+ * Walk through the array and eliminate duplicate values, but keep the
+ * ordering (so that we can do bsearch later). We know there's at least
+ * one item as (counts[i] != 0), so we can skip the first element.
+ */
+ ndistinct = 1; /* number of distinct values */
+ for (j = 1; j < counts[i]; j++)
+ {
+ /* if the value is the same as the previous one, we can skip it */
+ if (! compare_datums_simple(values[i][j-1], values[i][j], &ssup[i]))
+ continue;
+
+ values[i][ndistinct] = values[i][j];
+ ndistinct += 1;
+ }
+
+ /* we must not exceed UINT16_MAX, as we use uint16 indexes */
+ Assert(ndistinct <= UINT16_MAX);
+
+ /*
+ * Store additional info about the attribute - number of deduplicated
+ * values, and also size of the serialized data. For fixed-length data
+ * types this is trivial to compute, for varwidth types we need to
+ * actually walk the array and sum the sizes.
+ */
+ info[i].nvalues = ndistinct;
+
+ if (info[i].typlen > 0) /* fixed-length data types */
+ info[i].nbytes = info[i].nvalues * info[i].typlen;
+ else if (info[i].typlen == -1) /* varlena */
+ {
+ info[i].nbytes = 0;
+ for (j = 0; j < info[i].nvalues; j++)
+ info[i].nbytes += VARSIZE_ANY(values[i][j]);
+ }
+ else if (info[i].typlen == -2) /* cstring */
+ {
+ info[i].nbytes = 0;
+ for (j = 0; j < info[i].nvalues; j++)
+ info[i].nbytes += strlen(DatumGetPointer(values[i][j]));
+ }
+
+ /* we know (count>0) so there must be some data */
+ Assert(info[i].nbytes > 0);
+ }
+
+ /*
+ * Now we can finally compute how much space we'll actually need for the
+ * serialized MCV list, as it contains these fields:
+ *
+ * - length (4B) for varlena
+ * - magic (4B)
+ * - type (4B)
+ * - ndimensions (4B)
+ * - nitems (4B)
+ * - info (ndim * sizeof(DimensionInfo)
+ * - arrays of values for each dimension
+ * - serialized items (nitems * itemsize)
+ *
+ * So the 'header' size is 20B + ndim * sizeof(DimensionInfo) and then we
+ * will place all the data (values + indexes).
+ */
+ total_length = (sizeof(int32) + offsetof(MCVListData, items)
+ + ndims * sizeof(DimensionInfo)
+ + mcvlist->nitems * itemsize);
+
+ for (i = 0; i < ndims; i++)
+ total_length += info[i].nbytes;
+
+ /* enforce arbitrary limit of 1MB */
+ if (total_length > (1024 * 1024))
+ elog(ERROR, "serialized MCV list exceeds 1MB (%ld)", total_length);
+
+ /* allocate space for the serialized MCV list, set header fields */
+ output = (bytea*)palloc0(total_length);
+ SET_VARSIZE(output, total_length);
+
+ /* 'data' points to the current position in the output buffer */
+ data = VARDATA(output);
+
+ /* MCV list header (number of items, ...) */
+ memcpy(data, mcvlist, offsetof(MCVListData, items));
+ data += offsetof(MCVListData, items);
+
+ /* information about the attributes */
+ memcpy(data, info, sizeof(DimensionInfo) * ndims);
+ data += sizeof(DimensionInfo) * ndims;
+
+ /* now serialize the deduplicated values for all attributes */
+ for (i = 0; i < ndims; i++)
+ {
+#ifdef USE_ASSERT_CHECKING
+ char *tmp = data; /* remember the starting point */
+#endif
+ for (j = 0; j < info[i].nvalues; j++)
+ {
+ Datum v = values[i][j];
+
+ if (info[i].typbyval) /* passed by value */
+ {
+ memcpy(data, &v, info[i].typlen);
+ data += info[i].typlen;
+ }
+ else if (info[i].typlen > 0) /* pased by reference */
+ {
+ memcpy(data, DatumGetPointer(v), info[i].typlen);
+ data += info[i].typlen;
+ }
+ else if (info[i].typlen == -1) /* varlena */
+ {
+ memcpy(data, DatumGetPointer(v), VARSIZE_ANY(v));
+ data += VARSIZE_ANY(v);
+ }
+ else if (info[i].typlen == -2) /* cstring */
+ {
+ memcpy(data, DatumGetPointer(v), strlen(DatumGetPointer(v))+1);
+ data += strlen(DatumGetPointer(v)) + 1; /* terminator */
+ }
+ }
+
+ /* make sure we got exactly the amount of data we expected */
+ Assert((data - tmp) == info[i].nbytes);
+ }
+
+ /* finally serialize the items, with uint16 indexes instead of the values */
+ for (i = 0; i < mcvlist->nitems; i++)
+ {
+ MCVItem mcvitem = mcvlist->items[i];
+
+ /* don't write beyond the allocated space */
+ Assert(data <= (char*)output + total_length - itemsize);
+
+ /* reset the item (we only allocate it once and reuse it) */
+ memset(item, 0, itemsize);
+
+ for (j = 0; j < ndims; j++)
+ {
+ Datum *v = NULL;
+
+ /* do the lookup only for non-NULL values */
+ if (mcvlist->items[i]->isnull[j])
+ continue;
+
+ v = (Datum*)bsearch_arg(&mcvitem->values[j], values[j],
+ info[j].nvalues, sizeof(Datum),
+ compare_scalars_simple, &ssup[j]);
+
+ Assert(v != NULL); /* serialization or deduplication error */
+
+ /* compute index within the array */
+ ITEM_INDEXES(item)[j] = (v - values[j]);
+
+ /* check the index is within expected bounds */
+ Assert(ITEM_INDEXES(item)[j] >= 0);
+ Assert(ITEM_INDEXES(item)[j] < info[j].nvalues);
+ }
+
+ /* copy NULL and frequency flags into the item */
+ memcpy(ITEM_NULLS(item, ndims), mcvitem->isnull, sizeof(bool) * ndims);
+ memcpy(ITEM_FREQUENCY(item, ndims), &mcvitem->frequency, sizeof(double));
+
+ /* copy the serialized item into the array */
+ memcpy(data, item, itemsize);
+
+ data += itemsize;
+ }
+
+ /* at this point we expect to match the total_length exactly */
+ Assert((data - (char*)output) == total_length);
+
+ return output;
+}
+
+/*
+ * deserialize MCV list from the varlena value
+ *
+ *
+ * We deserialize the MCV list fully, because we don't expect there bo be a lot
+ * of duplicate values. But perhaps we should keep the MCV in serialized form
+ * just like histograms.
+ */
+MCVList deserialize_mv_mcvlist(bytea * data)
+{
+ int i, j;
+ Size expected_size;
+ MCVList mcvlist;
+ char *tmp;
+
+ int ndims, nitems, itemsize;
+ DimensionInfo *info = NULL;
+
+ uint16 *indexes = NULL;
+ Datum **values = NULL;
+
+ /* local allocation buffer (used only for deserialization) */
+ int bufflen;
+ char *buff;
+ char *ptr;
+
+ /* buffer used for the result */
+ int rbufflen;
+ char *rbuff;
+ char *rptr;
+
+ if (data == NULL)
+ return NULL;
+
+ /* we can't deserialize the MCV if there's not even a complete header */
+ expected_size = offsetof(MCVListData,items);
+
+ if (VARSIZE_ANY_EXHDR(data) < expected_size)
+ elog(ERROR, "invalid MCV Size %ld (expected at least %ld)",
+ VARSIZE_ANY_EXHDR(data), offsetof(MCVListData,items));
+
+ /* read the MCV list header */
+ mcvlist = (MCVList)palloc0(sizeof(MCVListData));
+
+ /* initialize pointer to the data part (skip the varlena header) */
+ tmp = VARDATA(data);
+
+ /* get the header and perform further sanity checks */
+ memcpy(mcvlist, tmp, offsetof(MCVListData,items));
+ tmp += offsetof(MCVListData,items);
+
+ if (mcvlist->magic != MVSTAT_MCV_MAGIC)
+ elog(ERROR, "invalid MCV magic %d (expected %dd)",
+ mcvlist->magic, MVSTAT_MCV_MAGIC);
+
+ if (mcvlist->type != MVSTAT_MCV_TYPE_BASIC)
+ elog(ERROR, "invalid MCV type %d (expected %dd)",
+ mcvlist->type, MVSTAT_MCV_TYPE_BASIC);
+
+ nitems = mcvlist->nitems;
+ ndims = mcvlist->ndimensions;
+ itemsize = ITEM_SIZE(ndims);
+
+ Assert((nitems > 0) && (nitems <= MVSTAT_MCVLIST_MAX_ITEMS));
+ Assert((ndims >= 2) && (ndims <= MVSTATS_MAX_DIMENSIONS));
+
+ /*
+ * Check amount of data including DimensionInfo for all dimensions and
+ * also the serialized items (including uint16 indexes). Also, walk
+ * through the dimension information and add it to the sum.
+ */
+ expected_size += ndims * sizeof(DimensionInfo) +
+ (nitems * itemsize);
+
+ /* check that we have at least the DimensionInfo records */
+ if (VARSIZE_ANY_EXHDR(data) < expected_size)
+ elog(ERROR, "invalid MCV size %ld (expected %ld)",
+ VARSIZE_ANY_EXHDR(data), expected_size);
+
+ info = (DimensionInfo*)(tmp);
+ tmp += ndims * sizeof(DimensionInfo);
+
+ /* account for the value arrays */
+ for (i = 0; i < ndims; i++)
+ {
+ Assert(info[i].nvalues >= 0);
+ Assert(info[i].nbytes >= 0);
+
+ expected_size += info[i].nbytes;
+ }
+
+ if (VARSIZE_ANY_EXHDR(data) != expected_size)
+ elog(ERROR, "invalid MCV size %ld (expected %ld)",
+ VARSIZE_ANY_EXHDR(data), expected_size);
+
+ /* looks OK - not corrupted or something */
+
+ /*
+ * Allocate one large chunk of memory for the intermediate data, needed
+ * only for deserializing the MCV list (and allocate densely to minimize
+ * the palloc overhead).
+ *
+ * Let's see how much space we'll actually need, and also include space
+ * for the array with pointers.
+ */
+ bufflen = sizeof(Datum*) * ndims; /* space for pointers */
+
+ for (i = 0; i < ndims; i++)
+ /* for full-size byval types, we reuse the serialized value */
+ if (! (info[i].typbyval && info[i].typlen == sizeof(Datum)))
+ bufflen += (sizeof(Datum) * info[i].nvalues);
+
+ buff = palloc0(bufflen);
+ ptr = buff;
+
+ values = (Datum**)buff;
+ ptr += (sizeof(Datum*) * ndims);
+
+ /*
+ * XXX This uses pointers to the original data array (the types not passed
+ * by value), so when someone frees the memory, e.g. by doing something
+ * like this:
+ *
+ * bytea * data = ... fetch the data from catalog ...
+ * MCVList mcvlist = deserialize_mcv_list(data);
+ * pfree(data);
+ *
+ * then 'mcvlist' references the freed memory. Should copy the pieces.
+ */
+ for (i = 0; i < ndims; i++)
+ {
+ if (info[i].typbyval)
+ {
+ /* passed by value / Datum - simply reuse the array */
+ if (info[i].typlen == sizeof(Datum))
+ {
+ values[i] = (Datum*)tmp;
+ tmp += info[i].nbytes;
+ }
+ else
+ {
+ values[i] = (Datum*)ptr;
+ ptr += (sizeof(Datum) * info[i].nvalues);
+
+ for (j = 0; j < info[i].nvalues; j++)
+ {
+ /* just point into the array */
+ memcpy(&values[i][j], tmp, info[i].typlen);
+ tmp += info[i].typlen;
+ }
+ }
+ }
+ else
+ {
+ /* all the other types need a chunk of the buffer */
+ values[i] = (Datum*)ptr;
+ ptr += (sizeof(Datum) * info[i].nvalues);
+
+ /* pased by reference, but fixed length (name, tid, ...) */
+ if (info[i].typlen > 0)
+ {
+ for (j = 0; j < info[i].nvalues; j++)
+ {
+ /* just point into the array */
+ values[i][j] = PointerGetDatum(tmp);
+ tmp += info[i].typlen;
+ }
+ }
+ else if (info[i].typlen == -1)
+ {
+ /* varlena */
+ for (j = 0; j < info[i].nvalues; j++)
+ {
+ /* just point into the array */
+ values[i][j] = PointerGetDatum(tmp);
+ tmp += VARSIZE_ANY(tmp);
+ }
+ }
+ else if (info[i].typlen == -2)
+ {
+ /* cstring */
+ for (j = 0; j < info[i].nvalues; j++)
+ {
+ /* just point into the array */
+ values[i][j] = PointerGetDatum(tmp);
+ tmp += (strlen(tmp) + 1); /* don't forget the \0 */
+ }
+ }
+ }
+ }
+
+ /* we should have exhausted the buffer exactly */
+ Assert((ptr - buff) == bufflen);
+
+ /* allocate space for all the MCV items in a single piece */
+ rbufflen = (sizeof(MCVItem) + sizeof(MCVItemData) +
+ sizeof(Datum)*ndims + sizeof(bool)*ndims) * nitems;
+
+ rbuff = palloc0(rbufflen);
+ rptr = rbuff;
+
+ mcvlist->items = (MCVItem*)rbuff;
+ rptr += (sizeof(MCVItem) * nitems);
+
+ for (i = 0; i < nitems; i++)
+ {
+ MCVItem item = (MCVItem)rptr;
+ rptr += (sizeof(MCVItemData));
+
+ item->values = (Datum*)rptr;
+ rptr += (sizeof(Datum)*ndims);
+
+ item->isnull = (bool*)rptr;
+ rptr += (sizeof(bool) *ndims);
+
+ /* just point to the right place */
+ indexes = ITEM_INDEXES(tmp);
+
+ memcpy(item->isnull, ITEM_NULLS(tmp, ndims), sizeof(bool) * ndims);
+ memcpy(&item->frequency, ITEM_FREQUENCY(tmp, ndims), sizeof(double));
+
+#ifdef ASSERT_CHECKING
+ for (j = 0; j < ndims; j++)
+ Assert(indexes[j] <= UINT16_MAX);
+#endif
+
+ /* translate the values */
+ for (j = 0; j < ndims; j++)
+ if (! item->isnull[j])
+ item->values[j] = values[j][indexes[j]];
+
+ mcvlist->items[i] = item;
+
+ tmp += ITEM_SIZE(ndims);
+
+ Assert(tmp <= (char*)data + VARSIZE_ANY(data));
+ }
+
+ /* check that we processed all the data */
+ Assert(tmp == (char*)data + VARSIZE_ANY(data));
+
+ /* release the temporary buffer */
+ pfree(buff);
+
+ return mcvlist;
+}
+
+/*
+ * SRF with details about buckets of a histogram:
+ *
+ * - item ID (0...nitems)
+ * - values (string array)
+ * - nulls only (boolean array)
+ * - frequency (double precision)
+ *
+ * The input is the OID of the statistics, and there are no rows returned if
+ * the statistics contains no histogram.
+ */
+PG_FUNCTION_INFO_V1(pg_mv_mcv_items);
+
+Datum
+pg_mv_mcv_items(PG_FUNCTION_ARGS)
+{
+ FuncCallContext *funcctx;
+ int call_cntr;
+ int max_calls;
+ TupleDesc tupdesc;
+ AttInMetadata *attinmeta;
+
+ /* stuff done only on the first call of the function */
+ if (SRF_IS_FIRSTCALL())
+ {
+ MemoryContext oldcontext;
+ MCVList mcvlist;
+
+ /* create a function context for cross-call persistence */
+ funcctx = SRF_FIRSTCALL_INIT();
+
+ /* switch to memory context appropriate for multiple function calls */
+ oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
+
+ mcvlist = load_mv_mcvlist(PG_GETARG_OID(0));
+
+ funcctx->user_fctx = mcvlist;
+
+ /* total number of tuples to be returned */
+ funcctx->max_calls = 0;
+ if (funcctx->user_fctx != NULL)
+ funcctx->max_calls = mcvlist->nitems;
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("function returning record called in context "
+ "that cannot accept type record")));
+
+ /* build metadata needed later to produce tuples from raw C-strings */
+ attinmeta = TupleDescGetAttInMetadata(tupdesc);
+ funcctx->attinmeta = attinmeta;
+
+ MemoryContextSwitchTo(oldcontext);
+ }
+
+ /* stuff done on every call of the function */
+ funcctx = SRF_PERCALL_SETUP();
+
+ call_cntr = funcctx->call_cntr;
+ max_calls = funcctx->max_calls;
+ attinmeta = funcctx->attinmeta;
+
+ if (call_cntr < max_calls) /* do when there is more left to send */
+ {
+ char **values;
+ HeapTuple tuple;
+ Datum result;
+ int2vector *stakeys;
+ Oid relid;
+
+ char *buff = palloc0(1024);
+ char *format;
+
+ int i;
+
+ Oid *outfuncs;
+ FmgrInfo *fmgrinfo;
+
+ MCVList mcvlist;
+ MCVItem item;
+
+ mcvlist = (MCVList)funcctx->user_fctx;
+
+ Assert(call_cntr < mcvlist->nitems);
+
+ item = mcvlist->items[call_cntr];
+
+ stakeys = find_mv_attnums(PG_GETARG_OID(0), &relid);
+
+ /*
+ * Prepare a values array for building the returned tuple. This should
+ * be an array of C strings which will be processed later by the type
+ * input functions.
+ */
+ values = (char **) palloc(4 * sizeof(char *));
+
+ values[0] = (char *) palloc(64 * sizeof(char));
+
+ /* arrays */
+ values[1] = (char *) palloc0(1024 * sizeof(char));
+ values[2] = (char *) palloc0(1024 * sizeof(char));
+
+ /* frequency */
+ values[3] = (char *) palloc(64 * sizeof(char));
+
+ outfuncs = (Oid*)palloc0(sizeof(Oid) * mcvlist->ndimensions);
+ fmgrinfo = (FmgrInfo*)palloc0(sizeof(FmgrInfo) * mcvlist->ndimensions);
+
+ for (i = 0; i < mcvlist->ndimensions; i++)
+ {
+ bool isvarlena;
+
+ getTypeOutputInfo(get_atttype(relid, stakeys->values[i]),
+ &outfuncs[i], &isvarlena);
+
+ fmgr_info(outfuncs[i], &fmgrinfo[i]);
+ }
+
+ snprintf(values[0], 64, "%d", call_cntr); /* item ID */
+
+ for (i = 0; i < mcvlist->ndimensions; i++)
+ {
+ Datum val, valout;
+
+ format = "%s, %s";
+ if (i == 0)
+ format = "{%s%s";
+ else if (i == mcvlist->ndimensions-1)
+ format = "%s, %s}";
+
+ if (item->isnull[i])
+ valout = CStringGetDatum("NULL");
+ else
+ {
+ val = item->values[i];
+ valout = FunctionCall1(&fmgrinfo[i], val);
+ }
+
+ snprintf(buff, 1024, format, values[1], DatumGetPointer(valout));
+ strncpy(values[1], buff, 1023);
+ buff[0] = '\0';
+
+ snprintf(buff, 1024, format, values[2], item->isnull[i] ? "t" : "f");
+ strncpy(values[2], buff, 1023);
+ buff[0] = '\0';
+ }
+
+ snprintf(values[3], 64, "%f", item->frequency); /* frequency */
+
+ /* build a tuple */
+ tuple = BuildTupleFromCStrings(attinmeta, values);
+
+ /* make the tuple into a datum */
+ result = HeapTupleGetDatum(tuple);
+
+ /* clean up (this is not really necessary) */
+ pfree(values[0]);
+ pfree(values[1]);
+ pfree(values[2]);
+ pfree(values[3]);
+
+ pfree(values);
+
+ SRF_RETURN_NEXT(funcctx, result);
+ }
+ else /* do when there is no more left */
+ {
+ SRF_RETURN_DONE(funcctx);
+ }
+}
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 8ce9c0e..2c22d31 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -2109,8 +2109,9 @@ describeOneTableDetails(const char *schemaname,
{
printfPQExpBuffer(&buf,
"SELECT oid, stanamespace::regnamespace AS nsp, staname, stakeys,\n"
- " deps_enabled,\n"
- " deps_built,\n"
+ " deps_enabled, mcv_enabled,\n"
+ " deps_built, mcv_built,\n"
+ " mcv_max_items,\n"
" (SELECT string_agg(attname::text,', ')\n"
" FROM ((SELECT unnest(stakeys) AS attnum) s\n"
" JOIN pg_attribute a ON (starelid = a.attrelid and a.attnum = s.attnum))) AS attnums\n"
@@ -2128,6 +2129,8 @@ describeOneTableDetails(const char *schemaname,
printTableAddFooter(&cont, _("Statistics:"));
for (i = 0; i < tuples; i++)
{
+ bool first = true;
+
printfPQExpBuffer(&buf, " ");
/* statistics name (qualified with namespace) */
@@ -2137,10 +2140,22 @@ describeOneTableDetails(const char *schemaname,
/* options */
if (!strcmp(PQgetvalue(result, i, 4), "t"))
- appendPQExpBuffer(&buf, "(dependencies)");
+ {
+ appendPQExpBuffer(&buf, "(dependencies");
+ first = false;
+ }
+
+ if (!strcmp(PQgetvalue(result, i, 5), "t"))
+ {
+ if (! first)
+ appendPQExpBuffer(&buf, ", mcv");
+ else
+ appendPQExpBuffer(&buf, "(mcv");
+ first = false;
+ }
- appendPQExpBuffer(&buf, " ON (%s)",
- PQgetvalue(result, i, 6));
+ appendPQExpBuffer(&buf, ") ON (%s)",
+ PQgetvalue(result, i, 9));
printTableAddFooter(&cont, buf.data);
}
diff --git a/src/include/catalog/pg_mv_statistic.h b/src/include/catalog/pg_mv_statistic.h
index c74af47..3529b03 100644
--- a/src/include/catalog/pg_mv_statistic.h
+++ b/src/include/catalog/pg_mv_statistic.h
@@ -38,15 +38,21 @@ CATALOG(pg_mv_statistic,3381)
/* statistics requested to build */
bool deps_enabled; /* analyze dependencies? */
+ bool mcv_enabled; /* build MCV list? */
+
+ /* MCV size */
+ int32 mcv_max_items; /* max MCV items */
/* statistics that are available (if requested) */
bool deps_built; /* dependencies were built */
+ bool mcv_built; /* MCV list was built */
/* variable-length fields start here, but we allow direct access to stakeys */
int2vector stakeys; /* array of column keys */
#ifdef CATALOG_VARLEN
bytea stadeps; /* dependencies (serialized) */
+ bytea stamcv; /* MCV list (serialized) */
#endif
} FormData_pg_mv_statistic;
@@ -62,14 +68,18 @@ typedef FormData_pg_mv_statistic *Form_pg_mv_statistic;
* compiler constants for pg_mv_statistic
* ----------------
*/
-#define Natts_pg_mv_statistic 8
+#define Natts_pg_mv_statistic 12
#define Anum_pg_mv_statistic_starelid 1
#define Anum_pg_mv_statistic_staname 2
#define Anum_pg_mv_statistic_stanamespace 3
#define Anum_pg_mv_statistic_staowner 4
#define Anum_pg_mv_statistic_deps_enabled 5
-#define Anum_pg_mv_statistic_deps_built 6
-#define Anum_pg_mv_statistic_stakeys 7
-#define Anum_pg_mv_statistic_stadeps 8
+#define Anum_pg_mv_statistic_mcv_enabled 6
+#define Anum_pg_mv_statistic_mcv_max_items 7
+#define Anum_pg_mv_statistic_deps_built 8
+#define Anum_pg_mv_statistic_mcv_built 9
+#define Anum_pg_mv_statistic_stakeys 10
+#define Anum_pg_mv_statistic_stadeps 11
+#define Anum_pg_mv_statistic_stamcv 12
#endif /* PG_MV_STATISTIC_H */
diff --git a/src/include/catalog/pg_proc.h b/src/include/catalog/pg_proc.h
index ff2d797..f8ceabf 100644
--- a/src/include/catalog/pg_proc.h
+++ b/src/include/catalog/pg_proc.h
@@ -2670,6 +2670,10 @@ DATA(insert OID = 3998 ( pg_mv_stats_dependencies_info PGNSP PGUID 12 1 0 0
DESCR("multivariate stats: functional dependencies info");
DATA(insert OID = 3999 ( pg_mv_stats_dependencies_show PGNSP PGUID 12 1 0 0 0 f f f f t f i s 1 0 25 "17" _null_ _null_ _null_ _null_ _null_ pg_mv_stats_dependencies_show _null_ _null_ _null_ ));
DESCR("multivariate stats: functional dependencies show");
+DATA(insert OID = 3376 ( pg_mv_stats_mcvlist_info PGNSP PGUID 12 1 0 0 0 f f f f t f i s 1 0 25 "17" _null_ _null_ _null_ _null_ _null_ pg_mv_stats_mcvlist_info _null_ _null_ _null_ ));
+DESCR("multi-variate statistics: MCV list info");
+DATA(insert OID = 3373 ( pg_mv_mcv_items PGNSP PGUID 12 1 1000 0 0 f f f f t t i s 1 0 2249 "26" "{26,23,1009,1000,701}" "{i,o,o,o,o}" "{oid,index,values,nulls,frequency}" _null_ _null_ pg_mv_mcv_items _null_ _null_ _null_ ));
+DESCR("details about MCV list items");
DATA(insert OID = 1928 ( pg_stat_get_numscans PGNSP PGUID 12 1 0 0 0 f f f f t f s r 1 0 20 "26" _null_ _null_ _null_ _null_ _null_ pg_stat_get_numscans _null_ _null_ _null_ ));
DESCR("statistics: number of scans done for table/index");
diff --git a/src/include/nodes/relation.h b/src/include/nodes/relation.h
index e10dcf1..2bcd582 100644
--- a/src/include/nodes/relation.h
+++ b/src/include/nodes/relation.h
@@ -653,9 +653,11 @@ typedef struct MVStatisticInfo
/* enabled statistics */
bool deps_enabled; /* functional dependencies enabled */
+ bool mcv_enabled; /* MCV list enabled */
/* built/available statistics */
bool deps_built; /* functional dependencies built */
+ bool mcv_built; /* MCV list built */
/* columns in the statistics (attnums) */
int2vector *stakeys; /* attnums of the columns covered */
diff --git a/src/include/utils/mvstats.h b/src/include/utils/mvstats.h
index c6f45ab..ce7c3ad 100644
--- a/src/include/utils/mvstats.h
+++ b/src/include/utils/mvstats.h
@@ -52,30 +52,89 @@ typedef MVDependenciesData* MVDependencies;
#define MVSTAT_DEPS_TYPE_BASIC 1 /* basic dependencies type */
/*
+ * Multivariate MCV (most-common value) lists
+ *
+ * A straight-forward extension of MCV items - i.e. a list (array) of
+ * combinations of attribute values, together with a frequency and
+ * null flags.
+ */
+typedef struct MCVItemData {
+ double frequency; /* frequency of this combination */
+ bool *isnull; /* lags of NULL values (up to 32 columns) */
+ Datum *values; /* variable-length (ndimensions) */
+} MCVItemData;
+
+typedef MCVItemData *MCVItem;
+
+/* multivariate MCV list - essentally an array of MCV items */
+typedef struct MCVListData {
+ uint32 magic; /* magic constant marker */
+ uint32 type; /* type of MCV list (BASIC) */
+ uint32 ndimensions; /* number of dimensions */
+ uint32 nitems; /* number of MCV items in the array */
+ MCVItem *items; /* array of MCV items */
+} MCVListData;
+
+typedef MCVListData *MCVList;
+
+/* used to flag stats serialized to bytea */
+#define MVSTAT_MCV_MAGIC 0xE1A651C2 /* marks serialized bytea */
+#define MVSTAT_MCV_TYPE_BASIC 1 /* basic MCV list type */
+
+/*
+ * Limits used for mcv_max_items option, i.e. we're always guaranteed
+ * to have space for at least MVSTAT_MCVLIST_MIN_ITEMS, and we cannot
+ * have more than MVSTAT_MCVLIST_MAX_ITEMS items.
+ *
+ * This is just a boundary for the 'max' threshold - the actual list
+ * may of course contain less items than MVSTAT_MCVLIST_MIN_ITEMS.
+ */
+#define MVSTAT_MCVLIST_MIN_ITEMS 128 /* min items in MCV list */
+#define MVSTAT_MCVLIST_MAX_ITEMS 8192 /* max items in MCV list */
+
+/*
* TODO Maybe fetching the histogram/MCV list separately is inefficient?
* Consider adding a single `fetch_stats` method, fetching all
* stats specified using flags (or something like that).
*/
MVDependencies load_mv_dependencies(Oid mvoid);
+MCVList load_mv_mcvlist(Oid mvoid);
bytea * serialize_mv_dependencies(MVDependencies dependencies);
+bytea * serialize_mv_mcvlist(MCVList mcvlist, int2vector *attrs,
+ VacAttrStats **stats);
/* deserialization of stats (serialization is private to analyze) */
MVDependencies deserialize_mv_dependencies(bytea * data);
+MCVList deserialize_mv_mcvlist(bytea * data);
+
+/*
+ * Returns index of the attribute number within the vector (i.e. a
+ * dimension within the stats).
+ */
+int mv_get_index(AttrNumber varattno, int2vector * stakeys);
+
+int2vector* find_mv_attnums(Oid mvoid, Oid *relid);
/* FIXME this probably belongs somewhere else (not to operations stats) */
extern Datum pg_mv_stats_dependencies_info(PG_FUNCTION_ARGS);
extern Datum pg_mv_stats_dependencies_show(PG_FUNCTION_ARGS);
+extern Datum pg_mv_stats_mcvlist_info(PG_FUNCTION_ARGS);
+extern Datum pg_mv_mcvlist_items(PG_FUNCTION_ARGS);
MVDependencies
-build_mv_dependencies(int numrows, HeapTuple *rows,
- int2vector *attrs,
- VacAttrStats **stats);
+build_mv_dependencies(int numrows, HeapTuple *rows, int2vector *attrs,
+ VacAttrStats **stats);
+
+MCVList
+build_mv_mcvlist(int numrows, HeapTuple *rows, int2vector *attrs,
+ VacAttrStats **stats, int *numrows_filtered);
void build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
- int natts, VacAttrStats **vacattrstats);
+ int natts, VacAttrStats **vacattrstats);
-void update_mv_stats(Oid relid, MVDependencies dependencies, int2vector *attrs);
+void update_mv_stats(Oid relid, MVDependencies dependencies, MCVList mcvlist,
+ int2vector *attrs, VacAttrStats **stats);
#endif
diff --git a/src/test/regress/expected/mv_mcv.out b/src/test/regress/expected/mv_mcv.out
new file mode 100644
index 0000000..075320b
--- /dev/null
+++ b/src/test/regress/expected/mv_mcv.out
@@ -0,0 +1,207 @@
+-- data type passed by value
+CREATE TABLE mcv_list (
+ a INT,
+ b INT,
+ c INT
+);
+-- unknown column
+CREATE STATISTICS s4 ON mcv_list (unknown_column) WITH (mcv);
+ERROR: column "unknown_column" referenced in statistics does not exist
+-- single column
+CREATE STATISTICS s4 ON mcv_list (a) WITH (mcv);
+ERROR: multivariate stats require 2 or more columns
+-- single column, duplicated
+CREATE STATISTICS s4 ON mcv_list (a, a) WITH (mcv);
+ERROR: duplicate column name in statistics definition
+-- two columns, one duplicated
+CREATE STATISTICS s4 ON mcv_list (a, a, b) WITH (mcv);
+ERROR: duplicate column name in statistics definition
+-- unknown option
+CREATE STATISTICS s4 ON mcv_list (a, b, c) WITH (unknown_option);
+ERROR: unrecognized STATISTICS option "unknown_option"
+-- missing MCV statistics
+CREATE STATISTICS s4 ON mcv_list (a, b, c) WITH (dependencies, max_mcv_items=200);
+ERROR: option 'mcv' is required by other options(s)
+-- invalid mcv_max_items value / too low
+CREATE STATISTICS s4 ON mcv_list (a, b, c) WITH (mcv, max_mcv_items=10);
+ERROR: max number of MCV items must be at least 128
+-- invalid mcv_max_items value / too high
+CREATE STATISTICS s4 ON mcv_list (a, b, c) WITH (mcv, max_mcv_items=10000);
+ERROR: max number of MCV items is 8192
+-- correct command
+CREATE STATISTICS s4 ON mcv_list (a, b, c) WITH (mcv);
+-- random data
+INSERT INTO mcv_list
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+ANALYZE mcv_list;
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+ mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info
+-------------+-----------+--------------------------
+ t | f |
+(1 row)
+
+TRUNCATE mcv_list;
+-- a => b, a => c, b => c
+INSERT INTO mcv_list
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE mcv_list;
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+ mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info
+-------------+-----------+--------------------------
+ t | t | nitems=1000
+(1 row)
+
+TRUNCATE mcv_list;
+-- a => b, a => c
+INSERT INTO mcv_list
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE mcv_list;
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+ mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info
+-------------+-----------+--------------------------
+ t | t | nitems=1000
+(1 row)
+
+TRUNCATE mcv_list;
+-- check explain (expect bitmap index scan, not plain index scan)
+INSERT INTO mcv_list
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX mcv_idx ON mcv_list (a, b);
+ANALYZE mcv_list;
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+ mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info
+-------------+-----------+--------------------------
+ t | t | nitems=100
+(1 row)
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mcv_list WHERE a = 10 AND b = 5;
+ QUERY PLAN
+--------------------------------------------
+ Bitmap Heap Scan on mcv_list
+ Recheck Cond: ((a = 10) AND (b = 5))
+ -> Bitmap Index Scan on mcv_idx
+ Index Cond: ((a = 10) AND (b = 5))
+(4 rows)
+
+DROP TABLE mcv_list;
+-- varlena type (text)
+CREATE TABLE mcv_list (
+ a TEXT,
+ b TEXT,
+ c TEXT
+);
+CREATE STATISTICS s5 ON mcv_list (a, b, c) WITH (mcv);
+-- random data
+INSERT INTO mcv_list
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+ANALYZE mcv_list;
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+ mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info
+-------------+-----------+--------------------------
+ t | f |
+(1 row)
+
+TRUNCATE mcv_list;
+-- a => b, a => c, b => c
+INSERT INTO mcv_list
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE mcv_list;
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+ mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info
+-------------+-----------+--------------------------
+ t | t | nitems=1000
+(1 row)
+
+TRUNCATE mcv_list;
+-- a => b, a => c
+INSERT INTO mcv_list
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE mcv_list;
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+ mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info
+-------------+-----------+--------------------------
+ t | t | nitems=1000
+(1 row)
+
+TRUNCATE mcv_list;
+-- check explain (expect bitmap index scan, not plain index scan)
+INSERT INTO mcv_list
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX mcv_idx ON mcv_list (a, b);
+ANALYZE mcv_list;
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+ mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info
+-------------+-----------+--------------------------
+ t | t | nitems=100
+(1 row)
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mcv_list WHERE a = '10' AND b = '5';
+ QUERY PLAN
+------------------------------------------------------------
+ Bitmap Heap Scan on mcv_list
+ Recheck Cond: ((a = '10'::text) AND (b = '5'::text))
+ -> Bitmap Index Scan on mcv_idx
+ Index Cond: ((a = '10'::text) AND (b = '5'::text))
+(4 rows)
+
+TRUNCATE mcv_list;
+-- check explain (expect bitmap index scan, not plain index scan) with NULLs
+INSERT INTO mcv_list
+ SELECT
+ (CASE WHEN i/10000 = 0 THEN NULL ELSE i/10000 END),
+ (CASE WHEN i/20000 = 0 THEN NULL ELSE i/20000 END),
+ (CASE WHEN i/40000 = 0 THEN NULL ELSE i/40000 END)
+ FROM generate_series(1,1000000) s(i);
+ANALYZE mcv_list;
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+ mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info
+-------------+-----------+--------------------------
+ t | t | nitems=100
+(1 row)
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mcv_list WHERE a IS NULL AND b IS NULL;
+ QUERY PLAN
+---------------------------------------------------
+ Bitmap Heap Scan on mcv_list
+ Recheck Cond: ((a IS NULL) AND (b IS NULL))
+ -> Bitmap Index Scan on mcv_idx
+ Index Cond: ((a IS NULL) AND (b IS NULL))
+(4 rows)
+
+DROP TABLE mcv_list;
+-- NULL values (mix of int and text columns)
+CREATE TABLE mcv_list (
+ a INT,
+ b TEXT,
+ c INT,
+ d TEXT
+);
+CREATE STATISTICS s6 ON mcv_list (a, b, c, d) WITH (mcv);
+INSERT INTO mcv_list
+ SELECT
+ mod(i, 100),
+ (CASE WHEN mod(i, 200) = 0 THEN NULL ELSE mod(i,200) END),
+ mod(i, 400),
+ (CASE WHEN mod(i, 300) = 0 THEN NULL ELSE mod(i,600) END)
+ FROM generate_series(1,10000) s(i);
+ANALYZE mcv_list;
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+ mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info
+-------------+-----------+--------------------------
+ t | t | nitems=1200
+(1 row)
+
+DROP TABLE mcv_list;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 06f2231..3d55ffe 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1373,7 +1373,9 @@ pg_mv_stats| SELECT n.nspname AS schemaname,
s.staname,
s.stakeys AS attnums,
length(s.stadeps) AS depsbytes,
- pg_mv_stats_dependencies_info(s.stadeps) AS depsinfo
+ pg_mv_stats_dependencies_info(s.stadeps) AS depsinfo,
+ length(s.stamcv) AS mcvbytes,
+ pg_mv_stats_mcvlist_info(s.stamcv) AS mcvinfo
FROM ((pg_mv_statistic s
JOIN pg_class c ON ((c.oid = s.starelid)))
LEFT JOIN pg_namespace n ON ((n.oid = c.relnamespace)));
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index 4f2ffb8..85d94f1 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -112,4 +112,4 @@ test: event_trigger
test: stats
# run tests of multivariate stats
-test: mv_dependencies
+test: mv_dependencies mv_mcv
diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule
index 097a04f..6584d73 100644
--- a/src/test/regress/serial_schedule
+++ b/src/test/regress/serial_schedule
@@ -163,3 +163,4 @@ test: xml
test: event_trigger
test: stats
test: mv_dependencies
+test: mv_mcv
diff --git a/src/test/regress/sql/mv_mcv.sql b/src/test/regress/sql/mv_mcv.sql
new file mode 100644
index 0000000..b31d32d
--- /dev/null
+++ b/src/test/regress/sql/mv_mcv.sql
@@ -0,0 +1,178 @@
+-- data type passed by value
+CREATE TABLE mcv_list (
+ a INT,
+ b INT,
+ c INT
+);
+
+-- unknown column
+CREATE STATISTICS s4 ON mcv_list (unknown_column) WITH (mcv);
+
+-- single column
+CREATE STATISTICS s4 ON mcv_list (a) WITH (mcv);
+
+-- single column, duplicated
+CREATE STATISTICS s4 ON mcv_list (a, a) WITH (mcv);
+
+-- two columns, one duplicated
+CREATE STATISTICS s4 ON mcv_list (a, a, b) WITH (mcv);
+
+-- unknown option
+CREATE STATISTICS s4 ON mcv_list (a, b, c) WITH (unknown_option);
+
+-- missing MCV statistics
+CREATE STATISTICS s4 ON mcv_list (a, b, c) WITH (dependencies, max_mcv_items=200);
+
+-- invalid mcv_max_items value / too low
+CREATE STATISTICS s4 ON mcv_list (a, b, c) WITH (mcv, max_mcv_items=10);
+
+-- invalid mcv_max_items value / too high
+CREATE STATISTICS s4 ON mcv_list (a, b, c) WITH (mcv, max_mcv_items=10000);
+
+-- correct command
+CREATE STATISTICS s4 ON mcv_list (a, b, c) WITH (mcv);
+
+-- random data
+INSERT INTO mcv_list
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+
+ANALYZE mcv_list;
+
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+
+TRUNCATE mcv_list;
+
+-- a => b, a => c, b => c
+INSERT INTO mcv_list
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+
+ANALYZE mcv_list;
+
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+
+TRUNCATE mcv_list;
+
+-- a => b, a => c
+INSERT INTO mcv_list
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+
+ANALYZE mcv_list;
+
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+
+TRUNCATE mcv_list;
+
+-- check explain (expect bitmap index scan, not plain index scan)
+INSERT INTO mcv_list
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX mcv_idx ON mcv_list (a, b);
+
+ANALYZE mcv_list;
+
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mcv_list WHERE a = 10 AND b = 5;
+
+DROP TABLE mcv_list;
+
+-- varlena type (text)
+CREATE TABLE mcv_list (
+ a TEXT,
+ b TEXT,
+ c TEXT
+);
+
+CREATE STATISTICS s5 ON mcv_list (a, b, c) WITH (mcv);
+
+-- random data
+INSERT INTO mcv_list
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+
+ANALYZE mcv_list;
+
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+
+TRUNCATE mcv_list;
+
+-- a => b, a => c, b => c
+INSERT INTO mcv_list
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+
+ANALYZE mcv_list;
+
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+
+TRUNCATE mcv_list;
+
+-- a => b, a => c
+INSERT INTO mcv_list
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE mcv_list;
+
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+
+TRUNCATE mcv_list;
+
+-- check explain (expect bitmap index scan, not plain index scan)
+INSERT INTO mcv_list
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX mcv_idx ON mcv_list (a, b);
+ANALYZE mcv_list;
+
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mcv_list WHERE a = '10' AND b = '5';
+
+TRUNCATE mcv_list;
+
+-- check explain (expect bitmap index scan, not plain index scan) with NULLs
+INSERT INTO mcv_list
+ SELECT
+ (CASE WHEN i/10000 = 0 THEN NULL ELSE i/10000 END),
+ (CASE WHEN i/20000 = 0 THEN NULL ELSE i/20000 END),
+ (CASE WHEN i/40000 = 0 THEN NULL ELSE i/40000 END)
+ FROM generate_series(1,1000000) s(i);
+ANALYZE mcv_list;
+
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mcv_list WHERE a IS NULL AND b IS NULL;
+
+DROP TABLE mcv_list;
+
+-- NULL values (mix of int and text columns)
+CREATE TABLE mcv_list (
+ a INT,
+ b TEXT,
+ c INT,
+ d TEXT
+);
+
+CREATE STATISTICS s6 ON mcv_list (a, b, c, d) WITH (mcv);
+
+INSERT INTO mcv_list
+ SELECT
+ mod(i, 100),
+ (CASE WHEN mod(i, 200) = 0 THEN NULL ELSE mod(i,200) END),
+ mod(i, 400),
+ (CASE WHEN mod(i, 300) = 0 THEN NULL ELSE mod(i,600) END)
+ FROM generate_series(1,10000) s(i);
+
+ANALYZE mcv_list;
+
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+
+DROP TABLE mcv_list;
--
2.5.0
[text/x-patch] 0003-clause-reduction-using-functional-dependencies.patch (69.4K, ../../1457906378.27231.10.camel@2ndquadrant.com/7-0003-clause-reduction-using-functional-dependencies.patch)
download | inline diff:
From 70cf8ed9f0d161a335be1e72a25f325261ea9e46 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@pgaddict.com>
Date: Mon, 6 Apr 2015 19:42:18 +0200
Subject: [PATCH 3/9] clause reduction using functional dependencies
During planning, use functional dependencies to decide which clauses to
skip during cardinality estimation. Initial and rather simplistic
implementation.
This only works with regular WHERE clauses, not clauses used for join
clauses.
Note: The clause_is_mv_compatible() needs to identify the relation (so
that we can fetch the list of multivariate stats by OID).
planner_rt_fetch() seems like the appropriate way to get the relation
OID, but apparently it only works with simple vars. Maybe
examine_variable() would make this work with more complex vars too?
Includes regression tests analyzing functional dependencies (part of
ANALYZE) on several datasets (no dependencies, no transitive
dependencies, ...).
Checks that a query with conditions on two columns, where one (B) is
functionally dependent on the other one (A), correctly ignores the
clause on (B) and chooses bitmap index scan instead of plain index scan
(which is what happens otherwise, thanks to assumption of
independence).
Note: Functional dependencies only work with equality clauses, no
inequalities etc.
---
src/backend/optimizer/path/clausesel.c | 505 ++++++++++++++++-
src/backend/utils/mvstats/README.dependencies | 63 +--
src/backend/utils/mvstats/README.stats | 36 ++
src/backend/utils/mvstats/common.c | 25 +-
src/backend/utils/mvstats/common.h | 3 +
src/backend/utils/mvstats/dependencies.c | 775 +++++++++++++++++---------
src/include/utils/mvstats.h | 21 +-
src/test/regress/expected/mv_dependencies.out | 172 ++++++
src/test/regress/parallel_schedule | 3 +
src/test/regress/serial_schedule | 1 +
src/test/regress/sql/mv_dependencies.sql | 150 +++++
11 files changed, 1457 insertions(+), 297 deletions(-)
create mode 100644 src/backend/utils/mvstats/README.stats
create mode 100644 src/test/regress/expected/mv_dependencies.out
create mode 100644 src/test/regress/sql/mv_dependencies.sql
diff --git a/src/backend/optimizer/path/clausesel.c b/src/backend/optimizer/path/clausesel.c
index 02660c2..5ab7f15 100644
--- a/src/backend/optimizer/path/clausesel.c
+++ b/src/backend/optimizer/path/clausesel.c
@@ -14,14 +14,19 @@
*/
#include "postgres.h"
+#include "access/sysattr.h"
+#include "catalog/pg_operator.h"
#include "nodes/makefuncs.h"
#include "optimizer/clauses.h"
#include "optimizer/cost.h"
#include "optimizer/pathnode.h"
#include "optimizer/plancat.h"
+#include "optimizer/var.h"
#include "utils/fmgroids.h"
#include "utils/lsyscache.h"
+#include "utils/mvstats.h"
#include "utils/selfuncs.h"
+#include "utils/typcache.h"
/*
@@ -41,6 +46,25 @@ typedef struct RangeQueryClause
static void addRangeClause(RangeQueryClause **rqlist, Node *clause,
bool varonleft, bool isLTsel, Selectivity s2);
+#define MV_CLAUSE_TYPE_FDEP 0x01
+
+static bool clause_is_mv_compatible(Node *clause, Index relid, AttrNumber *attnum);
+
+static Bitmapset *collect_mv_attnums(List *clauses, Index relid);
+
+static int count_mv_attnums(List *clauses, Index relid);
+
+static int count_varnos(List *clauses, Index *relid);
+
+static List *clauselist_apply_dependencies(PlannerInfo *root, List *clauses,
+ Index relid, List *stats);
+
+static bool has_stats(List *stats, int type);
+
+static List * find_stats(PlannerInfo *root, Index relid);
+
+static bool stats_type_matches(MVStatisticInfo *stat, int type);
+
/****************************************************************************
* ROUTINES TO COMPUTE SELECTIVITIES
@@ -60,7 +84,19 @@ static void addRangeClause(RangeQueryClause **rqlist, Node *clause,
* subclauses. However, that's only right if the subclauses have independent
* probabilities, and in reality they are often NOT independent. So,
* we want to be smarter where we can.
-
+ *
+ * The first thing we try to do is applying multivariate statistics, in a way
+ * that intends to minimize the overhead when there are no multivariate stats
+ * on the relation. Thus we do several simple (and inexpensive) checks first,
+ * to verify that suitable multivariate statistics exist.
+ *
+ * If we identify such multivariate statistics apply, we try to apply them.
+ * Currently we only have (soft) functional dependencies, so we try to reduce
+ * the list of clauses.
+ *
+ * Then we remove the clauses estimated using multivariate stats, and process
+ * the rest of the clauses using the regular per-column stats.
+ *
* Currently, the only extra smarts we have is to recognize "range queries",
* such as "x > 34 AND x < 42". Clauses are recognized as possible range
* query components if they are restriction opclauses whose operators have
@@ -99,6 +135,22 @@ clauselist_selectivity(PlannerInfo *root,
RangeQueryClause *rqlist = NULL;
ListCell *l;
+ /* processing mv stats */
+ Oid relid = InvalidOid;
+
+ /* list of multivariate stats on the relation */
+ List *stats = NIL;
+
+ /*
+ * To fetch the statistics, we first need to determine the rel. Currently
+ * point we only support estimates of simple restrictions with all Vars
+ * referencing a single baserel. However set_baserel_size_estimates() sets
+ * varRelid=0 so we have to actually inspect the clauses by pull_varnos
+ * and see if there's just a single varno referenced.
+ */
+ if ((count_varnos(clauses, &relid) == 1) && ((varRelid == 0) || (varRelid == relid)))
+ stats = find_stats(root, relid);
+
/*
* If there's exactly one clause, then no use in trying to match up pairs,
* so just go directly to clause_selectivity().
@@ -108,6 +160,24 @@ clauselist_selectivity(PlannerInfo *root,
varRelid, jointype, sjinfo);
/*
+ * Apply functional dependencies, but first check that there are some stats
+ * with functional dependencies built (by simply walking the stats list),
+ * and that there are at two or more attributes referenced by clauses that
+ * may be reduced using functional dependencies.
+ *
+ * We would find that anyway when trying to actually apply the functional
+ * dependencies, but let's do the cheap checks first.
+ *
+ * After applying the functional dependencies we get the remainig clauses
+ * that need to be estimated by other types of stats (MCV, histograms etc).
+ */
+ if (has_stats(stats, MV_CLAUSE_TYPE_FDEP) &&
+ (count_mv_attnums(clauses, relid) >= 2))
+ {
+ clauses = clauselist_apply_dependencies(root, clauses, relid, stats);
+ }
+
+ /*
* Initial scan over clauses. Anything that doesn't look like a potential
* rangequery clause gets multiplied into s1 and forgotten. Anything that
* does gets inserted into an rqlist entry.
@@ -763,3 +833,436 @@ clause_selectivity(PlannerInfo *root,
return s1;
}
+
+/*
+ * Collect attributes from mv-compatible clauses.
+ */
+static Bitmapset *
+collect_mv_attnums(List *clauses, Index relid)
+{
+ Bitmapset *attnums = NULL;
+ ListCell *l;
+
+ /*
+ * Walk through the clauses and identify the ones we can estimate
+ * using multivariate stats, and remember the relid/columns. We'll
+ * then cross-check if we have suitable stats, and only if needed
+ * we'll split the clauses into multivariate and regular lists.
+ *
+ * For now we're only interested in RestrictInfo nodes with nested
+ * OpExpr, using either a range or equality.
+ */
+ foreach (l, clauses)
+ {
+ AttrNumber attnum;
+ Node *clause = (Node *) lfirst(l);
+
+ /* ignore the result for now - we only need the info */
+ if (clause_is_mv_compatible(clause, relid, &attnum))
+ attnums = bms_add_member(attnums, attnum);
+ }
+
+ /*
+ * If there are not at least two attributes referenced by the clause(s),
+ * we can throw everything out (as we'll revert to simple stats).
+ */
+ if (bms_num_members(attnums) <= 1)
+ {
+ if (attnums != NULL)
+ pfree(attnums);
+ attnums = NULL;
+ }
+
+ return attnums;
+}
+
+/*
+ * Count the number of attributes in clauses compatible with multivariate stats.
+ */
+static int
+count_mv_attnums(List *clauses, Index relid)
+{
+ int c;
+ Bitmapset *attnums = collect_mv_attnums(clauses, relid);
+
+ c = bms_num_members(attnums);
+
+ bms_free(attnums);
+
+ return c;
+}
+
+/*
+ * Count varnos referenced in the clauses, and if there's a single varno then
+ * return the index in 'relid'.
+ */
+static int
+count_varnos(List *clauses, Index *relid)
+{
+ int cnt;
+ Bitmapset *varnos = NULL;
+
+ varnos = pull_varnos((Node *) clauses);
+ cnt = bms_num_members(varnos);
+
+ /* if there's a single varno in the clauses, remember it */
+ if (bms_num_members(varnos) == 1)
+ *relid = bms_singleton_member(varnos);
+
+ bms_free(varnos);
+
+ return cnt;
+}
+
+typedef struct
+{
+ Index varno; /* relid we're interested in */
+ Bitmapset *varattnos; /* attnums referenced by the clauses */
+} mv_compatible_context;
+
+/*
+ * Recursive walker that checks compatibility of the clause with multivariate
+ * statistics, and collects attnums from the Vars.
+ *
+ * XXX The original idea was to combine this with expression_tree_walker, but
+ * I've been unable to make that work - seems that does not quite allow
+ * checking the structure. Hence the explicit calls to the walker.
+ */
+static bool
+mv_compatible_walker(Node *node, mv_compatible_context *context)
+{
+ if (node == NULL)
+ return false;
+
+ if (IsA(node, RestrictInfo))
+ {
+ RestrictInfo *rinfo = (RestrictInfo *) node;
+
+ /* Pseudoconstants are not really interesting here. */
+ if (rinfo->pseudoconstant)
+ return true;
+
+ /* clauses referencing multiple varnos are incompatible */
+ if (bms_membership(rinfo->clause_relids) != BMS_SINGLETON)
+ return true;
+
+ /* check the clause inside the RestrictInfo */
+ return mv_compatible_walker((Node*)rinfo->clause, (void *) context);
+ }
+
+ if (IsA(node, Var))
+ {
+ Var * var = (Var*)node;
+
+ /*
+ * Also, the variable needs to reference the right relid (this might be
+ * unnecessary given the other checks, but let's be sure).
+ */
+ if (var->varno != context->varno)
+ return true;
+
+ /* Also skip system attributes (we don't allow stats on those). */
+ if (! AttrNumberIsForUserDefinedAttr(var->varattno))
+ return true;
+
+ /* Seems fine, so let's remember the attnum. */
+ context->varattnos = bms_add_member(context->varattnos, var->varattno);
+
+ return false;
+ }
+
+ /*
+ * And finally the operator expressions - we only allow simple expressions
+ * with two arguments, where one is a Var and the other is a constant, and
+ * it's a simple comparison (which we detect using estimator function).
+ */
+ if (is_opclause(node))
+ {
+ OpExpr *expr = (OpExpr *) node;
+ Var *var;
+ bool varonleft = true;
+ bool ok;
+
+ /*
+ * Only expressions with two arguments are considered compatible.
+ *
+ * XXX Possibly unnecessary (can OpExpr have different arg count?).
+ */
+ if (list_length(expr->args) != 2)
+ return true;
+
+ /* see if it actually has the right */
+ ok = (NumRelids((Node*)expr) == 1) &&
+ (is_pseudo_constant_clause(lsecond(expr->args)) ||
+ (varonleft = false,
+ is_pseudo_constant_clause(linitial(expr->args))));
+
+ /* unsupported structure (two variables or so) */
+ if (! ok)
+ return true;
+
+ /*
+ * If it's not a "<" or ">" or "=" operator, just ignore the clause.
+ * Otherwise note the relid and attnum for the variable. This uses the
+ * function for estimating selectivity, ont the operator directly (a bit
+ * awkward, but well ...).
+ */
+ switch (get_oprrest(expr->opno))
+ {
+ case F_EQSEL:
+
+ /* equality conditions are compatible with all statistics */
+ break;
+
+ default:
+
+ /* unknown estimator */
+ return true;
+ }
+
+ var = (varonleft) ? linitial(expr->args) : lsecond(expr->args);
+
+ return mv_compatible_walker((Node *) var, context);
+ }
+
+ /* Node not explicitly supported, so terminate */
+ return true;
+}
+
+/*
+ * Determines whether the clause is compatible with multivariate stats,
+ * and if it is, returns some additional information - varno (index
+ * into simple_rte_array) and a bitmap of attributes. This is then
+ * used to fetch related multivariate statistics.
+ *
+ * At this moment we only support basic conditions of the form
+ *
+ * variable OP constant
+ *
+ * where OP is one of [=,<,<=,>=,>] (which is however determined by
+ * looking at the associated function for estimating selectivity, just
+ * like with the single-dimensional case).
+ *
+ * TODO Support 'OR clauses' - shouldn't be all that difficult to
+ * evaluate them using multivariate stats.
+ */
+static bool
+clause_is_mv_compatible(Node *clause, Index relid, AttrNumber *attnum)
+{
+ mv_compatible_context context;
+
+ context.varno = relid;
+ context.varattnos = NULL; /* no attnums */
+
+ if (mv_compatible_walker(clause, (void *) &context))
+ return false;
+
+ /* remember the newly collected attnums */
+ *attnum = bms_singleton_member(context.varattnos);
+
+ return true;
+}
+
+
+/*
+ * Reduce clauses using functional dependencies
+ */
+static List*
+fdeps_reduce_clauses(List *clauses, Index relid, Bitmapset *reduced_attnums)
+{
+ ListCell *lc;
+ List *reduced_clauses = NIL;
+
+ foreach (lc, clauses)
+ {
+ AttrNumber attnum = InvalidAttrNumber;
+ Node * clause = (Node*)lfirst(lc);
+
+ /* ignore clauses that are not compatible with functional dependencies */
+ if (! clause_is_mv_compatible(clause, relid, &attnum))
+ reduced_clauses = lappend(reduced_clauses, clause);
+
+ /* for equality clauses, only keep those not on reduced attributes */
+ if (! bms_is_member(attnum, reduced_attnums))
+ reduced_clauses = lappend(reduced_clauses, clause);
+ }
+
+ return reduced_clauses;
+}
+
+/*
+ * decide which attributes are redundant (for equality clauses)
+ *
+ * We try to apply all functional dependencies available, and for each one we
+ * check if it matches attnums from equality clauses, but only those not yet
+ * reduced.
+ *
+ * XXX Not sure if the order in which we apply the dependencies matters.
+ *
+ * XXX We do not combine functional dependencies from separate stats. That is
+ * if we have dependencies on [a,b] and [b,c], then we don't deduce
+ * a->c from a->b and b->c. Computing such transitive closure is a possible
+ * future improvement.
+ */
+static Bitmapset *
+fdeps_reduce_attnums(List *stats, Bitmapset *attnums)
+{
+ ListCell *lc;
+ Bitmapset *reduced = NULL;
+
+ foreach (lc, stats)
+ {
+ int i;
+ MVDependencies dependencies = NULL;
+ MVStatisticInfo *info = (MVStatisticInfo *)lfirst(lc);
+
+ /* skip statistics without dependencies */
+ if (! stats_type_matches(info, MV_CLAUSE_TYPE_FDEP))
+ continue;
+
+ /* fetch and deserialize dependencies */
+ dependencies = load_mv_dependencies(info->mvoid);
+
+ for (i = 0; i < dependencies->ndeps; i++)
+ {
+ int j;
+ bool matched = true;
+ MVDependency dep = dependencies->deps[i];
+
+ /* we don't bother to break the loop early (only few attributes) */
+ for (j = 0; j < dep->nattributes; j++)
+ {
+ if (! bms_is_member(dep->attributes[j], attnums))
+ matched = false;
+
+ if (bms_is_member(dep->attributes[j], reduced))
+ matched = false;
+ }
+
+ /* if dependency applies, mark the last attribute as reduced */
+ if (matched)
+ reduced = bms_add_member(reduced,
+ dep->attributes[dep->nattributes-1]);
+ }
+ }
+
+ return reduced;
+}
+
+/*
+ * reduce list of equality clauses using soft functional dependencies
+ *
+ * We simply walk through list of functional dependencies, and for each one we
+ * check whether the dependency 'matches' the clauses, i.e. if there's a clause
+ * matching the condition. If yes, we attempt to remove all clauses matching
+ * the implied part of the dependency from the list.
+ *
+ * This only reduces equality clauses, and ignores all the other types. We might
+ * extend it to handle IS NULL clause, in the future.
+ *
+ * We also assume the equality clauses are 'compatible'. For example we can't
+ * identify when the clauses use a mismatching zip code and city name. In such
+ * case the usual approach (product of selectivities) would produce a better
+ * estimate, although mostly by chance.
+ *
+ * The implementation needs to be careful about cyclic dependencies, e.g. when
+ *
+ * (a -> b) and (b -> a)
+ *
+ * at the same time, which means there's 1:1 relationship between te columns.
+ * In this case we must not reduce clauses on both attributes at the same time.
+ *
+ * TODO Currently we only apply functional dependencies at the same level, but
+ * maybe we could transfer the clauses from upper levels to the subtrees?
+ * For example let's say we have (a->b) dependency, and condition
+ *
+ * (a=1) AND (b=2 OR c=3)
+ *
+ * Currently, we won't be able to perform any reduction, because we'll
+ * consider (a=1) and (b=2 OR c=3) independently. But maybe we could pass
+ * (a=1) into the other expression, and only check it against conditions
+ * of the functional dependencies?
+ *
+ * In this case we'd end up with
+ *
+ * (a=1)
+ *
+ * as we'd consider (b=2) implied thanks to the rule, rendering the whole
+ * OR clause valid.
+ */
+static List *
+clauselist_apply_dependencies(PlannerInfo *root, List *clauses,
+ Index relid, List *stats)
+{
+ Bitmapset *clause_attnums = NULL;
+ Bitmapset *reduced_attnums = NULL;
+
+ /*
+ * Is there at least one statistics with functional dependencies?
+ * If not, return the original clauses right away.
+ *
+ * XXX Isn't this a bit pointless, thanks to exactly the same check in
+ * clauselist_selectivity()? Can we trigger the condition here?
+ */
+ if (! has_stats(stats, MV_CLAUSE_TYPE_FDEP))
+ return clauses;
+
+ /* collect attnums from clauses compatible with dependencies (equality) */
+ clause_attnums = collect_mv_attnums(clauses, relid);
+
+ /* decide which attnums may be eliminated */
+ reduced_attnums = fdeps_reduce_attnums(stats, clause_attnums);
+
+ /*
+ * Walk through the clauses, and see which other clauses we may reduce.
+ */
+ clauses = fdeps_reduce_clauses(clauses, relid, reduced_attnums);
+
+ bms_free(clause_attnums);
+ bms_free(reduced_attnums);
+
+ return clauses;
+}
+
+/*
+ * Check that there are stats with at least one of the requested types.
+ */
+static bool
+stats_type_matches(MVStatisticInfo *stat, int type)
+{
+ if ((type & MV_CLAUSE_TYPE_FDEP) && stat->deps_built)
+ return true;
+
+ return false;
+}
+
+/*
+ * Check that there are stats with at least one of the requested types.
+ */
+static bool
+has_stats(List *stats, int type)
+{
+ ListCell *s;
+
+ foreach (s, stats)
+ {
+ MVStatisticInfo *stat = (MVStatisticInfo *)lfirst(s);
+
+ /* terminate if we've found at least one matching statistics */
+ if (stats_type_matches(stat, type))
+ return true;
+ }
+
+ return false;
+}
+
+/*
+ * Lookups stats for a given baserel.
+ */
+static List *
+find_stats(PlannerInfo *root, Index relid)
+{
+ Assert(root->simple_rel_array[relid] != NULL);
+
+ return root->simple_rel_array[relid]->mvstatlist;
+}
diff --git a/src/backend/utils/mvstats/README.dependencies b/src/backend/utils/mvstats/README.dependencies
index 1f96fbc..f248459 100644
--- a/src/backend/utils/mvstats/README.dependencies
+++ b/src/backend/utils/mvstats/README.dependencies
@@ -156,37 +156,24 @@ estimates - especially compared to histograms, that are quite bad in estimating
equality clauses.
-Limitations
------------
-
-Let's see the main liminations of functional dependencies, especially those
-related to the current implementation.
+Multi-column dependencies
+-------------------------
-The current implementation supports only dependencies between two columns, but
-this is merely a simplification of the initial implementation. It's certainly
-useful to mine for dependencies involving multiple columns on the 'left' side,
-i.e. a condition for the dependency. That is dependencies like (a,b -> c).
+The implementation supports dependencies with multiple columns on the left side
+(i.e. condition of the dependency). The detection starts from dependencies with
+a single condition, and then proceeds to higher condition counts.
-The implementation may/should be smart enough not to mine redundant conditions,
-e.g. (a->b) and (a,c -> b), because the latter is a trivial consequence of the
-former one (if values of 'a' determine 'b', adding another column won't change
-that relationship). The ANALYZE should first analyze 1:1 dependencies, then 2:1
-dependencies (and skip the already identified ones), etc.
+It also detects dependencies that are implied by already identified ones, and
+ignores them. For example if we know that (a->b) then we won't add (a,c->b) as
+this dependency is a trivial consequence of (a->b).
-For example the dependency
+For a more practical example, consider these two dependencies
(city name -> zip code)
-
-is much stronger, i.e. whenever it hold, then
-
(city name, state name -> zip code)
-holds too. But in case there are cities with the same name in different states,
-then only the latter dependency will be valid.
-
-Of course, there probably are cities with the same name within a single state,
-but hopefully this is relatively rare occurence (and thus we'll still detect
-the 'soft' dependency).
+We could say that the former dependency is stronger because if it's valid, then
+the second dependency is valid too.
Handling multiple columns on the right side of the dependency, is not necessary,
as those dependencies may be simply decomposed into a set of dependencies with
@@ -199,24 +186,22 @@ is exactly the same as
(a -> b) & (a -> c)
Of course, storing the first form may be more efficient thant storing multiple
-'simple' dependencies separately.
-
+'simple' dependencies separately. This is left as a future work.
-TODO Support dependencies with multiple columns on left/right.
-TODO Investigate using histogram and MCV list to verify the dependencies.
+Future work
+-----------
-TODO Investigate statistical testing of the distribution (to decide whether it
- makes sense to build the histogram/MCV list).
+* Investigate using histogram and MCV list to verify the dependencies.
-TODO Using a min/max of selectivities would probably make more sense for the
- associated columns.
+* Investigate statistical testing of the distribution (to decide whether it
+ makes sense to build the histogram/MCV list).
-TODO Consider eliminating the implied columns from the histogram and MCV lists
- (but maybe that's not a good idea, because that'd make it impossible to use
- these stats for non-equality clauses and also it wouldn't be possible to
- use the stats for verification of the dependencies).
+* Consider eliminating the implied columns from the histogram and MCV lists
+ (but maybe that's not a good idea, because that'd make it impossible to use
+ these stats for non-equality clauses and also it wouldn't be possible to
+ use the stats for verification of the dependencies).
-TODO The reduction probably might be extended to also handle IS NULL clauses,
- assuming we fix the ANALYZE to properly handle NULL values. We however
- won't be able to reduce IS NOT NULL (unless I'm missing something).
+* The reduction probably might be extended to also handle IS NULL clauses,
+ assuming we fix the ANALYZE to properly handle NULL values. We however
+ won't be able to reduce IS NOT NULL (unless I'm missing something).
diff --git a/src/backend/utils/mvstats/README.stats b/src/backend/utils/mvstats/README.stats
new file mode 100644
index 0000000..a38ea7b
--- /dev/null
+++ b/src/backend/utils/mvstats/README.stats
@@ -0,0 +1,36 @@
+Multivariate statististics
+==========================
+
+When estimating various quantities (e.g. condition selectivities) the default
+approach relies on the assumption of independence. In practice that's often
+not true, resulting in estimation errors.
+
+Multivariate stats track different types of dependencies between the columns,
+hopefully improving the estimates.
+
+Currently we only have one kind of multivariate statistics - soft functional
+dependencies, and we use it to improve estimates of equality clauses. See
+README.dependencies for details.
+
+
+Selectivity estimation
+----------------------
+
+When estimating selectivity, we aim to achieve several things:
+
+ (a) maximize the estimate accuracy
+
+ (b) minimize the overhead, especially when no suitable multivariate stats
+ exist (so if you are not using multivariate stats, there's no overhead)
+
+This clauselist_selectivity() performs several inexpensive checks first, before
+even attempting to do the more expensive estimation.
+
+ (1) check if there are multivariate stats on the relation
+
+ (2) check there are at least two attributes referenced by clauses compatible
+ with multivariate statistics (equality clauses for func. dependencies)
+
+ (3) perform reduction of equality clauses using func. dependencies
+
+ (4) estimate the reduced list of clauses using regular statistics
diff --git a/src/backend/utils/mvstats/common.c b/src/backend/utils/mvstats/common.c
index a755c49..dcb7c78 100644
--- a/src/backend/utils/mvstats/common.c
+++ b/src/backend/utils/mvstats/common.c
@@ -84,7 +84,8 @@ build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
/*
* Analyze functional dependencies of columns.
*/
- deps = build_mv_dependencies(numrows, rows, attrs, stats);
+ if (stat->deps_enabled)
+ deps = build_mv_dependencies(numrows, rows, attrs, stats);
/* store the histogram / MCV list in the catalog */
update_mv_stats(stat->mvoid, deps, attrs);
@@ -163,6 +164,7 @@ list_mv_stats(Oid relid)
info->mvoid = HeapTupleGetOid(htup);
info->stakeys = buildint2vector(stats->stakeys.values, stats->stakeys.dim1);
+ info->deps_enabled = stats->deps_enabled;
info->deps_built = stats->deps_built;
result = lappend(result, info);
@@ -274,6 +276,7 @@ compare_scalars_partition(const void *a, const void *b, void *arg)
return ApplySortComparator(da, false, db, false, ssup);
}
+
/* initialize multi-dimensional sort */
MultiSortSupport
multi_sort_init(int ndims)
@@ -354,3 +357,23 @@ multi_sort_compare_dim(int dim, const SortItem *a, const SortItem *b,
b->values[dim], b->isnull[dim],
&mss->ssup[dim]);
}
+
+int
+multi_sort_compare_dims(int start, int end,
+ const SortItem *a, const SortItem *b,
+ MultiSortSupport mss)
+{
+ int dim;
+
+ for (dim = start; dim <= end; dim++)
+ {
+ int r = ApplySortComparator(a->values[dim], a->isnull[dim],
+ b->values[dim], b->isnull[dim],
+ &mss->ssup[dim]);
+
+ if (r != 0)
+ return r;
+ }
+
+ return 0;
+}
diff --git a/src/backend/utils/mvstats/common.h b/src/backend/utils/mvstats/common.h
index d96422d..a019ea6 100644
--- a/src/backend/utils/mvstats/common.h
+++ b/src/backend/utils/mvstats/common.h
@@ -70,6 +70,9 @@ int multi_sort_compare(const void *a, const void *b, void *arg);
int multi_sort_compare_dim(int dim, const SortItem *a,
const SortItem *b, MultiSortSupport mss);
+int multi_sort_compare_dims(int start, int end, const SortItem *a,
+ const SortItem *b, MultiSortSupport mss);
+
/* comparators, used when constructing multivariate stats */
int compare_scalars_simple(const void *a, const void *b, void *arg);
int compare_scalars_partition(const void *a, const void *b, void *arg);
diff --git a/src/backend/utils/mvstats/dependencies.c b/src/backend/utils/mvstats/dependencies.c
index 2a064a0..412dc30 100644
--- a/src/backend/utils/mvstats/dependencies.c
+++ b/src/backend/utils/mvstats/dependencies.c
@@ -17,293 +17,521 @@
#include "common.h"
#include "utils/lsyscache.h"
+/* internal state for generator of variations (k-permutations of n elements) */
+typedef struct VariationGeneratorData {
+
+ int k; /* size of the k-permutation */
+ int current; /* index of the next variation to return */
+
+ int nvariations; /* number of variations generated (size of array) */
+ int variations[1]; /* array of pre-built variations */
+
+} VariationGeneratorData;
+
+typedef VariationGeneratorData* VariationGenerator;
+
+/*
+ * generate all variations (k-permutations of n elements)
+ */
+static void
+generate_variations(VariationGenerator state,
+ int n, int maxlevel, int level, int *current)
+{
+ int i, j;
+
+ /* initialize */
+ if (level == 0)
+ {
+ current = (int*)palloc0(sizeof(int) * (maxlevel+1));
+ state->current = 0;
+ }
+
+ for (i = 0; i < n; i++)
+ {
+ /* check if the value is already used current variation */
+ bool found = false;
+ for (j = 0; j < level; j++)
+ {
+ if (current[j] == i)
+ {
+ found = true;
+ break;
+ }
+ }
+
+ /* already used, so try the next element */
+ if (found)
+ continue;
+
+ /* ok, we can use this element, so store it */
+ current[level] = i;
+
+ /* and check if we do have a complete variation of k elements */
+ if (level == maxlevel)
+ {
+ /* yep, store the variation */
+ Assert(state->current < state->nvariations);
+ memcpy(&state->variations[(state->k * state->current)], current,
+ sizeof(int) * (maxlevel+1));
+ state->current++;
+ }
+ else
+ /* nope, look for additional elements */
+ generate_variations(state, n, maxlevel, level+1, current);
+ }
+
+ if (level == 0)
+ pfree(current);
+}
+
/*
- * Detect functional dependencies between columns.
+ * initialize the generator of variations, and prebuild the variations
*
- * TODO This builds a complete set of dependencies, i.e. including transitive
- * dependencies - if we identify [A => B] and [B => C], we're likely to
- * identify [A => C] too. It might be better to keep only the minimal set
- * of dependencies, i.e. prune all the dependencies that we can recreate
- * by transivitity.
- *
- * There are two conceptual ways to do that:
- *
- * (a) generate all the rules, and then prune the rules that may be
- * recteated by combining other dependencies, or
- *
- * (b) performing the 'is combination of other dependencies' check before
- * actually doing the work
- *
- * The second option has the advantage that we don't really need to perform
- * the sort/count. It's not sufficient alone, though, because we may
- * discover the dependencies in the wrong order. For example we may find
+ * This pre-builds all the variations. We could also generate them in
+ * generator_next(), but this seems simpler.
+ */
+static VariationGenerator
+generator_init(int2vector *attrs, int k)
+{
+ int i;
+ int n = attrs->dim1;
+ int nvariations;
+ VariationGenerator state;
+
+ Assert((n >= k) && (k > 0));
+
+ /* compute the total number of variations as n!/(n-k)! */
+ nvariations = n;
+ for (i = 1; i < k; i++)
+ nvariations *= (n - i);
+
+ /* allocate the generator state as a single chunk of memory */
+ state = (VariationGenerator)palloc0(
+ offsetof(VariationGeneratorData, variations)
+ + (nvariations * k * sizeof(int))); /* variations */
+
+ state->nvariations = nvariations;
+ state->k = k;
+
+ /* now actually pre-generate all the variations */
+ generate_variations(state, n, (k-1), 0, NULL);
+
+ /* we expect to generate exactly the right number of variations */
+ Assert(state->nvariations == state->current);
+
+ /* reset the index */
+ state->current = 0;
+
+ return state;
+}
+
+/* free the generator state */
+static void
+generator_free(VariationGenerator state)
+{
+ /* we've allocated a single chunk, so just free it */
+ pfree(state);
+}
+
+/* generate next combination */
+static int*
+generator_next(VariationGenerator state, int2vector *attrs)
+{
+ if (state->current == state->nvariations)
+ return NULL;
+
+ return &state->variations[state->k * state->current++];
+}
+
+/*
+ * check if the dependency is implied by existing dependencies
+ *
+ * A dependency is considered implied, if there exists a dependency with the
+ * same column on the left, and a subset of columns on the right side. So for
+ * example if we have a dependency
+ *
+ * (a,b,c) -> d
*
- * (a -> b), (a -> c) and then (b -> c)
+ * then we are looking for these six dependencies
*
- * None of those dependencies is a combination of the already known ones,
- * yet (a -> C) is a combination of (a -> b) and (b -> c).
+ * (a) -> d
+ * (b) -> d
+ * (c) -> d
+ * (a,b) -> d
+ * (a,c) -> d
+ * (b,c) -> d
*
- *
- * FIXME Currently we simply replace NULL values with 0 and then handle is as
- * a regular value, but that groups NULL and actual 0 values. That's
- * clearly incorrect - we need to handle NULL values as a separate value.
+ * This does not detect transitive dependencies. For example if we have
+ *
+ * (a) -> b
+ * (b) -> c
+ *
+ * then obviously
+ *
+ * (a) -> c
+ *
+ * but this is not detected. Extending the method to handle transitive cases
+ * is future work.
*/
-MVDependencies
-build_mv_dependencies(int numrows, HeapTuple *rows, int2vector *attrs,
- VacAttrStats **stats)
+static bool
+dependency_is_implied(MVDependencies dependencies, int k, int *dependency,
+ int2vector * attrs)
{
- int i;
- int numattrs = attrs->dim1;
+ bool implied = false;
+ int i, j, l;
+ int *tmp;
- /* result */
- int ndeps = 0;
- MVDependencies dependencies = NULL;
- MultiSortSupport mss = multi_sort_init(2); /* 2 dimensions for now */
+ if (dependencies == NULL)
+ return false;
- /* TODO Maybe this should be somehow related to the number of
- * distinct values in the two columns we're currently analyzing.
- * Assuming the distribution is uniform, we can estimate the
- * average group size and use it as a threshold. Or something
- * like that. Seems better than a static approach.
- */
- int min_group_size = 3;
+ tmp = (int*)palloc0(sizeof(int) * k);
+
+ /* translate the indexes to actual attribute numbers */
+ for (i = 0; i < k; i++)
+ tmp[i] = attrs->values[dependency[i]];
+
+ /* search for a smaller */
+ for (i = 0; i < dependencies->ndeps; i++)
+ {
+ bool contained = true;
+ MVDependency dep = dependencies->deps[i];
+
+ /* does the last attribute match? */
+ if (tmp[k-1] != dep->attributes[dep->nattributes-1])
+ continue; /* nope, no need to check this dependency further */
+
+ /* are the conditions superset of the existing dependency? */
+ for (j = 0; j < (dep->nattributes-1); j++)
+ {
+ bool found = false;
+
+ for (l = 0; l < (k-1); l++)
+ {
+ if (tmp[l] == dep->attributes[j])
+ {
+ found = true;
+ break;
+ }
+ }
+
+ /* we've found an attribute not included in the new dependency */
+ if (! found)
+ {
+ contained = false;
+ break;
+ }
+ }
+
+ /* we've found an existing dependency, trivially proving the new one */
+ if (contained)
+ {
+ implied = true;
+ break;
+ }
+ }
- /* dimension indexes we'll check for associations [a => b] */
- int dima, dimb;
+ pfree(tmp);
+
+ return implied;
+}
+
+/*
+ * validates functional dependency on the data
+ *
+ * An actual work horse of detecting functional dependencies. Given a variation
+ * of k attributes, it checks that the first (k-1) are sufficient to determine
+ * the last one.
+ */
+static bool
+dependency_is_valid(int numrows, HeapTuple *rows, int k, int * dependency,
+ VacAttrStats **stats, int2vector *attrs)
+{
+ int i, j;
+ int nvalues = numrows * k;
/*
- * We'll reuse the same array for all the 2-column combinations.
- *
- * It's possible to sort the sample rows directly, but this seemed
- * somehow simples / less error prone. Another option would be to
- * allocate the arrays for each SortItem separately, but that'd be
- * significant overhead (not just CPU, but especially memory bloat).
+ * XXX Maybe the threshold should be somehow related to the number of
+ * distinct values in the combination of columns we're analyzing.
+ * Assuming the distribution is uniform, we can estimate the average
+ * group size and use it as a threshold, similarly to what we do for
+ * MCV lists.
*/
- SortItem * items = (SortItem*)palloc0(numrows * sizeof(SortItem));
+ int min_group_size = 3;
+
+ /* number of groups supporting / contradicting the dependency */
+ int n_supporting = 0;
+ int n_contradicting = 0;
+
+ /* counters valid within a group */
+ int group_size = 0;
+ int n_violations = 0;
- Datum *values = (Datum*)palloc0(sizeof(Datum) * numrows * 2);
- bool *isnull = (bool*)palloc0(sizeof(bool) * numrows * 2);
+ int n_supporting_rows = 0;
+ int n_contradicting_rows = 0;
+ /* sort info for all attributes columns */
+ MultiSortSupport mss = multi_sort_init(k);
+
+ /* data for the sort */
+ SortItem *items = (SortItem*)palloc0(numrows * sizeof(SortItem));
+ Datum *values = (Datum*)palloc0(sizeof(Datum) * nvalues);
+ bool *isnull = (bool*)palloc0(sizeof(bool) * nvalues);
+
+ /* fix the pointers to values/isnull */
for (i = 0; i < numrows; i++)
{
- items[i].values = &values[i * 2];
- items[i].isnull = &isnull[i * 2];
+ items[i].values = &values[i * k];
+ items[i].isnull = &isnull[i * k];
}
- Assert(numattrs >= 2);
-
/*
- * Evaluate all possible combinations of [A => B], using a simple algorithm:
+ * Verify the dependency (a,b,...)->z, using a rather simple algorithm:
+ *
+ * (a) sort the data lexicographically
*
- * (a) sort the data by [A,B]
- * (b) split the data into groups by A (new group whenever a value changes)
- * (c) count different values in the B column (again, value changes)
+ * (b) split the data into groups by first (k-1) columns
*
- * TODO It should be rather simple to merge [A => B] and [A => C] into
- * [A => B,C]. Just keep A constant, collect all the "implied" columns
- * and you're done.
+ * (c) for each group count different values in the last column
*/
- for (dima = 0; dima < numattrs; dima++)
+
+ /* prepare the sort function for the first dimension, and SortItem array */
+ for (i = 0; i < k; i++)
{
- /* prepare the sort function for the first dimension */
- multi_sort_add_dimension(mss, 0, dima, stats);
+ multi_sort_add_dimension(mss, i, dependency[i], stats);
- for (dimb = 0; dimb < numattrs; dimb++)
+ /* accumulate all the data for both columns into an array and sort it */
+ for (j = 0; j < numrows; j++)
{
- SortItem current;
-
- /* number of groups supporting / contradicting the dependency */
- int n_supporting = 0;
- int n_contradicting = 0;
-
- /* counters valid within a group */
- int group_size = 0;
- int n_violations = 0;
-
- int n_supporting_rows = 0;
- int n_contradicting_rows = 0;
-
- /* make sure the columns are different (A => A) */
- if (dima == dimb)
- continue;
-
- /* prepare the sort function for the second dimension */
- multi_sort_add_dimension(mss, 1, dimb, stats);
-
- /* reset the values and isnull flags */
- memset(values, 0, sizeof(Datum) * numrows * 2);
- memset(isnull, 0, sizeof(bool) * numrows * 2);
+ items[j].values[i]
+ = heap_getattr(rows[j], attrs->values[dependency[i]],
+ stats[i]->tupDesc, &items[j].isnull[i]);
+ }
+ }
- /* accumulate all the data for both columns into an array and sort it */
- for (i = 0; i < numrows; i++)
- {
- items[i].values[0]
- = heap_getattr(rows[i], attrs->values[dima],
- stats[dima]->tupDesc, &items[i].isnull[0]);
+ /* sort the items so that we can detect the groups */
+ qsort_arg((void *) items, numrows, sizeof(SortItem),
+ multi_sort_compare, mss);
- items[i].values[1]
- = heap_getattr(rows[i], attrs->values[dimb],
- stats[dimb]->tupDesc, &items[i].isnull[1]);
- }
+ /*
+ * Walk through the sorted array, split it into rows according to the first
+ * (k-1) columns. If there's a single value in the last column, we count
+ * the group as 'supporting' the functional dependency. Otherwise we count
+ * it as contradicting.
+ *
+ * We also require a group to have a minimum number of rows to be considered
+ * useful for supporting the dependency. Contradicting groups may be of
+ * any size, though.
+ *
+ * XXX The minimum size requirement makes it impossible to identify case
+ * when both columns are unique (or nearly unique), and therefore
+ * trivially functionally dependent.
+ */
- qsort_arg((void *) items, numrows, sizeof(SortItem),
- multi_sort_compare, mss);
+ /* start with the first row forming a group */
+ group_size = 1;
+ for (i = 1; i < numrows; i++)
+ {
+ /* end of the preceding group */
+ if (multi_sort_compare_dims(0, (k-2), &items[i-1], &items[i], mss) != 0)
+ {
/*
- * Walk through the array, split it into rows according to
- * the A value, and count distinct values in the other one.
- * If there's a single B value for the whole group, we count
- * it as supporting the association, otherwise we count it
- * as contradicting.
- *
- * Furthermore we require a group to have at least a certain
- * number of rows to be considered useful for supporting the
- * dependency. But when it's contradicting, use it always useful.
+ * If there is a single are no contradicting rows, count the group
+ * as supporting, otherwise contradicting.
*/
-
- /* start with values from the first row */
- current = items[0];
- group_size = 1;
-
- for (i = 1; i < numrows; i++)
- {
- /* end of the group */
- if (multi_sort_compare_dim(0, &items[i], ¤t, mss) != 0)
- {
- /*
- * If there are no contradicting rows, count it as
- * supporting (otherwise contradicting), but only if
- * the group is large enough.
- *
- * The requirement of a minimum group size makes it
- * impossible to identify [unique,unique] cases, but
- * that's probably a different case. This is more
- * about [zip => city] associations etc.
- *
- * If there are violations, count the group/rows as
- * a violation.
- *
- * It may ne neither, if the group is too small (does
- * not contain at least min_group_size rows).
- */
- if ((n_violations == 0) && (group_size >= min_group_size))
- {
- n_supporting += 1;
- n_supporting_rows += group_size;
- }
- else if (n_violations > 0)
- {
- n_contradicting += 1;
- n_contradicting_rows += group_size;
- }
-
- /* current values start a new group */
- n_violations = 0;
- group_size = 0;
- }
- /* mismatch of a B value is contradicting */
- else if (multi_sort_compare_dim(1, &items[i], ¤t, mss) != 0)
- {
- n_violations += 1;
- }
-
- current = items[i];
- group_size += 1;
- }
-
- /* handle the last group (just like above) */
if ((n_violations == 0) && (group_size >= min_group_size))
{
- n_supporting += 1;
+ n_supporting += 1;
n_supporting_rows += group_size;
}
- else if (n_violations)
+ else if (n_violations > 0)
{
- n_contradicting += 1;
+ n_contradicting += 1;
n_contradicting_rows += group_size;
}
- /*
- * See if the number of rows supporting the association is at least
- * 10x the number of rows violating the hypothetical dependency.
- *
- * TODO This is rather arbitrary limit - I guess it's possible to do
- * some math to come up with a better rule (e.g. testing a hypothesis
- * 'this is due to randomness'). We can create a contingency table
- * from the values and use it for testing. Possibly only when
- * there are no contradicting rows?
- *
- * TODO Also, if (a => b) and (b => a) at the same time, it pretty much
- * means there's a 1:1 relation (or one is a 'label'), making the
- * conditions rather redundant. Although it's possible that the
- * query uses incompatible combination of values.
- */
- if (n_supporting_rows > (n_contradicting_rows * 10))
- {
- if (dependencies == NULL)
- {
- dependencies = (MVDependencies)palloc0(sizeof(MVDependenciesData));
- dependencies->magic = MVSTAT_DEPS_MAGIC;
- }
- else
- dependencies = repalloc(dependencies, offsetof(MVDependenciesData, deps)
- + sizeof(MVDependency) * (dependencies->ndeps + 1));
+ /* current values start a new group */
+ n_violations = 0;
+ group_size = 0;
+ }
+ /* first colums match, but the last one does not (so contradicting) */
+ else if (multi_sort_compare_dims((k-1), (k-1), &items[i-1], &items[i], mss) != 0)
+ n_violations += 1;
- /* update the */
- dependencies->deps[ndeps] = (MVDependency)palloc0(sizeof(MVDependencyData));
- dependencies->deps[ndeps]->a = attrs->values[dima];
- dependencies->deps[ndeps]->b = attrs->values[dimb];
+ group_size += 1;
+ }
- dependencies->ndeps = (++ndeps);
- }
- }
+ /* handle the last group (just like above) */
+ if ((n_violations == 0) && (group_size >= min_group_size))
+ {
+ n_supporting += 1;
+ n_supporting_rows += group_size;
+ }
+ else if (n_violations)
+ {
+ n_contradicting += 1;
+ n_contradicting_rows += group_size;
}
pfree(items);
pfree(values);
pfree(isnull);
- pfree(stats);
pfree(mss);
- return dependencies;
+ /*
+ * See if the number of rows supporting the association is at least 10x the
+ * number of rows violating the hypothetical dependency.
+ */
+ return (n_supporting_rows > (n_contradicting_rows * 10));
}
/*
- * Store the dependencies into a bytea, so that it can be stored in the
- * pg_mv_statistic catalog.
+ * detects functional dependencies between groups of columns
+ *
+ * Generates all possible subsets of columns (variations) and checks if the
+ * last one is determined by the preceding ones. For example given 3 columns,
+ * there are 12 variations (6 for variations on 2 columns, 6 for 3 columns):
+ *
+ * two columns three columns
+ * ----------- -------------
+ * (a) -> c (a,b) -> c
+ * (b) -> c (b,a) -> c
+ * (a) -> b (a,c) -> b
+ * (c) -> b (c,a) -> b
+ * (c) -> a (c,b) -> a
+ * (b) -> a (b,c) -> a
+ *
+ * Clearly some of the variations are redundant, as the order of columns on the
+ * left side does not matter. This is detected in dependency_is_implied, and
+ * those dependencies are ignored.
*
- * Currently this only supports simple two-column rules, and stores them
- * as a sequence of attnum pairs. In the future, this needs to be made
- * more complex to support multiple columns on both sides of the
- * implication (using AND on left, OR on right).
+ * We however do not detect that dependencies are transitively implied. For
+ * example given dependencies
+ *
+ * (a) -> b
+ * (b) -> c
+ *
+ * then
+ *
+ * (a) -> c
+ *
+ * is trivially implied. However we don't detect that and all three dependencies
+ * will get included in the resulting set. Eliminating such transitively implied
+ * dependencies is future work.
+ */
+MVDependencies
+build_mv_dependencies(int numrows, HeapTuple *rows, int2vector *attrs,
+ VacAttrStats **stats)
+{
+ int i;
+ int k;
+ int numattrs = attrs->dim1;
+
+ /* result */
+ MVDependencies dependencies = NULL;
+
+ Assert(numattrs >= 2);
+
+ /*
+ * We'll try build functional dependencies starting from the smallest ones
+ * covering jut 2 columns, to the largest ones, covering all columns
+ * included int the statistics. We start from the smallest ones because
+ * we want to be able to skip already implied ones.
+ */
+ for (k = 2; k <= numattrs; k++)
+ {
+ int *dependency; /* array with k elements */
+
+ /* prepare a generator of variation */
+ VariationGenerator generator = generator_init(attrs, k);
+
+ /* generate all possible variations of k values (out of n) */
+ while ((dependency = generator_next(generator, attrs)))
+ {
+ MVDependency d;
+
+ /* skip dependencies that are already trivially implied */
+ if (dependency_is_implied(dependencies, k, dependency, attrs))
+ continue;
+
+ /* also skip dependencies that don't seem to be valid */
+ if (! dependency_is_valid(numrows, rows, k, dependency, stats, attrs))
+ continue;
+
+ d = (MVDependency)palloc0(offsetof(MVDependencyData, attributes)
+ + k * sizeof(int));
+
+ /* copy the dependency, but translate it to actuall attnums */
+ d->nattributes = k;
+ for (i = 0; i < k; i++)
+ d->attributes[i] = attrs->values[dependency[i]];
+
+ /* initialize the list of dependencies */
+ if (dependencies == NULL)
+ {
+ dependencies
+ = (MVDependencies)palloc0(sizeof(MVDependenciesData));
+
+ dependencies->magic = MVSTAT_DEPS_MAGIC;
+ dependencies->type = MVSTAT_DEPS_TYPE_BASIC;
+ dependencies->ndeps = 0;
+ }
+
+ dependencies->ndeps++;
+ dependencies = (MVDependencies)repalloc(dependencies,
+ offsetof(MVDependenciesData, deps)
+ + dependencies->ndeps * sizeof(MVDependency));
+
+ dependencies->deps[dependencies->ndeps-1] = d;
+ }
+
+ /* we're done with variations of k elements, so free the generator */
+ generator_free(generator);
+ }
+
+ return dependencies;
+}
+
+
+/*
+ * serialize list of dependencies into a bytea
*/
bytea *
serialize_mv_dependencies(MVDependencies dependencies)
{
int i;
+ bytea * output;
+ char *tmp;
- /* we need to store ndeps, and each needs 2 * int16 */
+ /* we need to store ndeps, with a number of attributes for each one */
Size len = VARHDRSZ + offsetof(MVDependenciesData, deps)
- + dependencies->ndeps * (sizeof(int16) * 2);
-
- bytea * output = (bytea*)palloc0(len);
+ + sizeof(int) * dependencies->ndeps;
- char * tmp = VARDATA(output);
+ /* and also include space for the actual attribute numbers */
+ for (i = 0; i < dependencies->ndeps; i++)
+ len += (sizeof(int16) * dependencies->deps[i]->nattributes);
+ output = (bytea*)palloc0(len);
SET_VARSIZE(output, len);
+ tmp = VARDATA(output);
+
/* first, store the number of dimensions / items */
memcpy(tmp, dependencies, offsetof(MVDependenciesData, deps));
tmp += offsetof(MVDependenciesData, deps);
- /* walk through the dependencies and copy both columns into the bytea */
+ /* store number of attributes and attribute numbers for each dependency */
for (i = 0; i < dependencies->ndeps; i++)
{
- memcpy(tmp, &(dependencies->deps[i]->a), sizeof(int16));
- tmp += sizeof(int16);
+ MVDependency d = dependencies->deps[i];
+
+ memcpy(tmp, &(d->nattributes), sizeof(int));
+ tmp += sizeof(int);
- memcpy(tmp, &(dependencies->deps[i]->b), sizeof(int16));
- tmp += sizeof(int16);
+ memcpy(tmp, d->attributes, sizeof(int16) * d->nattributes);
+ tmp += sizeof(int16) * d->nattributes;
+
+ Assert(tmp <= ((char*)output + len));
}
return output;
@@ -338,20 +566,21 @@ deserialize_mv_dependencies(bytea * data)
tmp += offsetof(MVDependenciesData, deps);
if (dependencies->magic != MVSTAT_DEPS_MAGIC)
- {
- pfree(dependencies);
- elog(WARNING, "not a MV Dependencies (magic number mismatch)");
- return NULL;
- }
+ elog(ERROR, "invalid dependency type %d (expected %dd)",
+ dependencies->type, MVSTAT_DEPS_MAGIC);
+
+ if (dependencies->type != MVSTAT_DEPS_TYPE_BASIC)
+ elog(ERROR, "invalid dependency type %d (expected %dd)",
+ dependencies->type, MVSTAT_DEPS_TYPE_BASIC);
Assert(dependencies->ndeps > 0);
- /* what bytea size do we expect for those parameters */
+ /* what minimum bytea size do we expect for those parameters */
expected_size = offsetof(MVDependenciesData,deps) +
- dependencies->ndeps * sizeof(int16) * 2;
+ dependencies->ndeps * (sizeof(int) + sizeof(int16) * 2);
- if (VARSIZE_ANY_EXHDR(data) != expected_size)
- elog(ERROR, "invalid dependencies size %ld (expected %ld)",
+ if (VARSIZE_ANY_EXHDR(data) < expected_size)
+ elog(ERROR, "invalid dependencies size %ld (expected at least %ld)",
VARSIZE_ANY_EXHDR(data), expected_size);
/* allocate space for the MCV items */
@@ -360,15 +589,35 @@ deserialize_mv_dependencies(bytea * data)
for (i = 0; i < dependencies->ndeps; i++)
{
- dependencies->deps[i] = (MVDependency)palloc0(sizeof(MVDependencyData));
+ int k;
+ MVDependency d;
+
+ /* number of attributes */
+ memcpy(&k, tmp, sizeof(int));
+ tmp += sizeof(int);
+
+ /* is the number of attributes valid? */
+ Assert((k >= 2) && (k <= MVSTATS_MAX_DIMENSIONS));
- memcpy(&(dependencies->deps[i]->a), tmp, sizeof(int16));
- tmp += sizeof(int16);
+ /* now that we know the number of attributes, allocate the dependency */
+ d = (MVDependency)palloc0(offsetof(MVDependencyData, attributes)
+ + k * sizeof(int));
- memcpy(&(dependencies->deps[i]->b), tmp, sizeof(int16));
- tmp += sizeof(int16);
+ d->nattributes = k;
+
+ /* copy attribute numbers */
+ memcpy(d->attributes, tmp, sizeof(int16) * d->nattributes);
+ tmp += sizeof(int16) * d->nattributes;
+
+ dependencies->deps[i] = d;
+
+ /* still within the bytea */
+ Assert(tmp <= ((char*)data + VARSIZE_ANY(data)));
}
+ /* we should have consumed the whole bytea exactly */
+ Assert(tmp == ((char*)data + VARSIZE_ANY(data)));
+
return dependencies;
}
@@ -392,46 +641,70 @@ pg_mv_stats_dependencies_info(PG_FUNCTION_ARGS)
PG_RETURN_TEXT_P(cstring_to_text(result));
}
-/* print the dependencies
- *
- * TODO Would be nice if this knew the actual column names (instead of
- * the attnums).
+/*
+ * print the dependencies
*
- * FIXME This is really ugly and does not really check the lengths and
- * strcpy/snprintf return values properly. Needs to be fixed.
+ * TODO Would be nice if this printed column names (instead of just attnums).
*/
Datum
pg_mv_stats_dependencies_show(PG_FUNCTION_ARGS)
{
- int i = 0;
- bytea *data = PG_GETARG_BYTEA_P(0);
- char *result = NULL;
- int len = 0;
+ int i, j;
+ bytea *data = PG_GETARG_BYTEA_P(0);
+ StringInfoData buf;
MVDependencies dependencies = deserialize_mv_dependencies(data);
if (dependencies == NULL)
PG_RETURN_NULL();
+ initStringInfo(&buf);
+
for (i = 0; i < dependencies->ndeps; i++)
{
MVDependency dependency = dependencies->deps[i];
- char buffer[128];
- int tmp = snprintf(buffer, 128, "%s%d => %d",
- ((i == 0) ? "" : ", "), dependency->a, dependency->b);
+ if (i > 0)
+ appendStringInfo(&buf, ", ");
- if (tmp < 127)
+ /* conditions */
+ appendStringInfoChar(&buf, '(');
+ for (j = 0; j < dependency->nattributes-1; j++)
{
- if (result == NULL)
- result = palloc0(len + tmp + 1);
- else
- result = repalloc(result, len + tmp + 1);
+ if (j > 0)
+ appendStringInfoChar(&buf, ',');
- strcpy(result + len, buffer);
- len += tmp;
+ appendStringInfo(&buf, "%d", dependency->attributes[j]);
}
+
+ /* the implied attribute */
+ appendStringInfo(&buf, ") => %d",
+ dependency->attributes[dependency->nattributes-1]);
}
- PG_RETURN_TEXT_P(cstring_to_text(result));
+ PG_RETURN_TEXT_P(cstring_to_text(buf.data));
+}
+
+MVDependencies
+load_mv_dependencies(Oid mvoid)
+{
+ bool isnull = false;
+ Datum deps;
+
+ /* Prepare to scan pg_mv_statistic for entries having indrelid = this rel. */
+ HeapTuple htup = SearchSysCache1(MVSTATOID, ObjectIdGetDatum(mvoid));
+
+#ifdef USE_ASSERT_CHECKING
+ Form_pg_mv_statistic mvstat = (Form_pg_mv_statistic) GETSTRUCT(htup);
+ Assert(mvstat->deps_enabled && mvstat->deps_built);
+#endif
+
+ deps = SysCacheGetAttr(MVSTATOID, htup,
+ Anum_pg_mv_statistic_stadeps, &isnull);
+
+ Assert(!isnull);
+
+ ReleaseSysCache(htup);
+
+ return deserialize_mv_dependencies(DatumGetByteaP(deps));
}
diff --git a/src/include/utils/mvstats.h b/src/include/utils/mvstats.h
index 7ebd961..c6f45ab 100644
--- a/src/include/utils/mvstats.h
+++ b/src/include/utils/mvstats.h
@@ -17,22 +17,31 @@
#include "fmgr.h"
#include "commands/vacuum.h"
+/*
+ * Degree of how much MCV item / histogram bucket matches a clause.
+ * This is then considered when computing the selectivity.
+ */
+#define MVSTATS_MATCH_NONE 0 /* no match at all */
+#define MVSTATS_MATCH_PARTIAL 1 /* partial match */
+#define MVSTATS_MATCH_FULL 2 /* full match */
#define MVSTATS_MAX_DIMENSIONS 8 /* max number of attributes */
-/* An associative rule, tracking [a => b] dependency.
- *
- * TODO Make this work with multiple columns on both sides.
+
+/*
+ * Functional dependencies, tracking column-level relationships (values
+ * in one column determine values in another one).
*/
typedef struct MVDependencyData {
- int16 a;
- int16 b;
+ int nattributes; /* number of attributes */
+ int16 attributes[1]; /* attribute numbers */
} MVDependencyData;
typedef MVDependencyData* MVDependency;
typedef struct MVDependenciesData {
uint32 magic; /* magic constant marker */
+ uint32 type; /* type of MV Dependencies (BASIC) */
int32 ndeps; /* number of dependencies */
MVDependency deps[1]; /* XXX why not a pointer? */
} MVDependenciesData;
@@ -48,6 +57,8 @@ typedef MVDependenciesData* MVDependencies;
* stats specified using flags (or something like that).
*/
+MVDependencies load_mv_dependencies(Oid mvoid);
+
bytea * serialize_mv_dependencies(MVDependencies dependencies);
/* deserialization of stats (serialization is private to analyze) */
diff --git a/src/test/regress/expected/mv_dependencies.out b/src/test/regress/expected/mv_dependencies.out
new file mode 100644
index 0000000..e759997
--- /dev/null
+++ b/src/test/regress/expected/mv_dependencies.out
@@ -0,0 +1,172 @@
+-- data type passed by value
+CREATE TABLE functional_dependencies (
+ a INT,
+ b INT,
+ c INT
+);
+-- unknown column
+CREATE STATISTICS s1 ON functional_dependencies (unknown_column) WITH (dependencies);
+ERROR: column "unknown_column" referenced in statistics does not exist
+-- single column
+CREATE STATISTICS s1 ON functional_dependencies (a) WITH (dependencies);
+ERROR: multivariate stats require 2 or more columns
+-- single column, duplicated
+CREATE STATISTICS s1 ON functional_dependencies (a,a) WITH (dependencies);
+ERROR: duplicate column name in statistics definition
+-- two columns, one duplicated
+CREATE STATISTICS s1 ON functional_dependencies (a, a, b) WITH (dependencies);
+ERROR: duplicate column name in statistics definition
+-- unknown option
+CREATE STATISTICS s1 ON functional_dependencies (a, b, c) WITH (unknown_option);
+ERROR: unrecognized STATISTICS option "unknown_option"
+-- correct command
+CREATE STATISTICS s1 ON functional_dependencies (a, b, c) WITH (dependencies);
+-- random data (no functional dependencies)
+INSERT INTO functional_dependencies
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+ANALYZE functional_dependencies;
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+ deps_enabled | deps_built | pg_mv_stats_dependencies_show
+--------------+------------+-------------------------------
+ t | f |
+(1 row)
+
+TRUNCATE functional_dependencies;
+-- a => b, a => c, b => c
+INSERT INTO functional_dependencies
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE functional_dependencies;
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+ deps_enabled | deps_built | pg_mv_stats_dependencies_show
+--------------+------------+-------------------------------
+ t | t | 1 => 2, 1 => 3, 2 => 3
+(1 row)
+
+TRUNCATE functional_dependencies;
+-- a => b, a => c
+INSERT INTO functional_dependencies
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE functional_dependencies;
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+ deps_enabled | deps_built | pg_mv_stats_dependencies_show
+--------------+------------+-------------------------------
+ t | t | 1 => 2, 1 => 3
+(1 row)
+
+TRUNCATE functional_dependencies;
+-- check explain (expect bitmap index scan, not plain index scan)
+INSERT INTO functional_dependencies
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX fdeps_idx ON functional_dependencies (a, b);
+ANALYZE functional_dependencies;
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+ deps_enabled | deps_built | pg_mv_stats_dependencies_show
+--------------+------------+-------------------------------
+ t | t | 1 => 2, 1 => 3, 2 => 3
+(1 row)
+
+EXPLAIN (COSTS off)
+ SELECT * FROM functional_dependencies WHERE a = 10 AND b = 5;
+ QUERY PLAN
+---------------------------------------------
+ Bitmap Heap Scan on functional_dependencies
+ Recheck Cond: ((a = 10) AND (b = 5))
+ -> Bitmap Index Scan on fdeps_idx
+ Index Cond: ((a = 10) AND (b = 5))
+(4 rows)
+
+DROP TABLE functional_dependencies;
+-- varlena type (text)
+CREATE TABLE functional_dependencies (
+ a TEXT,
+ b TEXT,
+ c TEXT
+);
+CREATE STATISTICS s2 ON functional_dependencies (a, b, c) WITH (dependencies);
+-- random data (no functional dependencies)
+INSERT INTO functional_dependencies
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+ANALYZE functional_dependencies;
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+ deps_enabled | deps_built | pg_mv_stats_dependencies_show
+--------------+------------+-------------------------------
+ t | f |
+(1 row)
+
+TRUNCATE functional_dependencies;
+-- a => b, a => c, b => c
+INSERT INTO functional_dependencies
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE functional_dependencies;
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+ deps_enabled | deps_built | pg_mv_stats_dependencies_show
+--------------+------------+-------------------------------
+ t | t | 1 => 2, 1 => 3, 2 => 3
+(1 row)
+
+TRUNCATE functional_dependencies;
+-- a => b, a => c
+INSERT INTO functional_dependencies
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE functional_dependencies;
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+ deps_enabled | deps_built | pg_mv_stats_dependencies_show
+--------------+------------+-------------------------------
+ t | t | 1 => 2, 1 => 3
+(1 row)
+
+TRUNCATE functional_dependencies;
+-- check explain (expect bitmap index scan, not plain index scan)
+INSERT INTO functional_dependencies
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX fdeps_idx ON functional_dependencies (a, b);
+ANALYZE functional_dependencies;
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+ deps_enabled | deps_built | pg_mv_stats_dependencies_show
+--------------+------------+-------------------------------
+ t | t | 1 => 2, 1 => 3, 2 => 3
+(1 row)
+
+EXPLAIN (COSTS off)
+ SELECT * FROM functional_dependencies WHERE a = '10' AND b = '5';
+ QUERY PLAN
+------------------------------------------------------------
+ Bitmap Heap Scan on functional_dependencies
+ Recheck Cond: ((a = '10'::text) AND (b = '5'::text))
+ -> Bitmap Index Scan on fdeps_idx
+ Index Cond: ((a = '10'::text) AND (b = '5'::text))
+(4 rows)
+
+DROP TABLE functional_dependencies;
+-- NULL values (mix of int and text columns)
+CREATE TABLE functional_dependencies (
+ a INT,
+ b TEXT,
+ c INT,
+ d TEXT
+);
+CREATE STATISTICS s3 ON functional_dependencies (a, b, c, d) WITH (dependencies);
+INSERT INTO functional_dependencies
+ SELECT
+ mod(i, 100),
+ (CASE WHEN mod(i, 200) = 0 THEN NULL ELSE mod(i,200) END),
+ mod(i, 400),
+ (CASE WHEN mod(i, 300) = 0 THEN NULL ELSE mod(i,600) END)
+ FROM generate_series(1,10000) s(i);
+ANALYZE functional_dependencies;
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+ deps_enabled | deps_built | pg_mv_stats_dependencies_show
+--------------+------------+----------------------------------------
+ t | t | 2 => 1, 3 => 1, 3 => 2, 4 => 1, 4 => 2
+(1 row)
+
+DROP TABLE functional_dependencies;
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index bec0316..4f2ffb8 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -110,3 +110,6 @@ test: event_trigger
# run stats by itself because its delay may be insufficient under heavy load
test: stats
+
+# run tests of multivariate stats
+test: mv_dependencies
diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule
index 7e9b319..097a04f 100644
--- a/src/test/regress/serial_schedule
+++ b/src/test/regress/serial_schedule
@@ -162,3 +162,4 @@ test: with
test: xml
test: event_trigger
test: stats
+test: mv_dependencies
diff --git a/src/test/regress/sql/mv_dependencies.sql b/src/test/regress/sql/mv_dependencies.sql
new file mode 100644
index 0000000..48dea4d
--- /dev/null
+++ b/src/test/regress/sql/mv_dependencies.sql
@@ -0,0 +1,150 @@
+-- data type passed by value
+CREATE TABLE functional_dependencies (
+ a INT,
+ b INT,
+ c INT
+);
+
+-- unknown column
+CREATE STATISTICS s1 ON functional_dependencies (unknown_column) WITH (dependencies);
+
+-- single column
+CREATE STATISTICS s1 ON functional_dependencies (a) WITH (dependencies);
+
+-- single column, duplicated
+CREATE STATISTICS s1 ON functional_dependencies (a,a) WITH (dependencies);
+
+-- two columns, one duplicated
+CREATE STATISTICS s1 ON functional_dependencies (a, a, b) WITH (dependencies);
+
+-- unknown option
+CREATE STATISTICS s1 ON functional_dependencies (a, b, c) WITH (unknown_option);
+
+-- correct command
+CREATE STATISTICS s1 ON functional_dependencies (a, b, c) WITH (dependencies);
+
+-- random data (no functional dependencies)
+INSERT INTO functional_dependencies
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+
+ANALYZE functional_dependencies;
+
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+
+TRUNCATE functional_dependencies;
+
+-- a => b, a => c, b => c
+INSERT INTO functional_dependencies
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+
+ANALYZE functional_dependencies;
+
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+
+TRUNCATE functional_dependencies;
+
+-- a => b, a => c
+INSERT INTO functional_dependencies
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE functional_dependencies;
+
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+
+TRUNCATE functional_dependencies;
+
+-- check explain (expect bitmap index scan, not plain index scan)
+INSERT INTO functional_dependencies
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX fdeps_idx ON functional_dependencies (a, b);
+ANALYZE functional_dependencies;
+
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+
+EXPLAIN (COSTS off)
+ SELECT * FROM functional_dependencies WHERE a = 10 AND b = 5;
+
+DROP TABLE functional_dependencies;
+
+-- varlena type (text)
+CREATE TABLE functional_dependencies (
+ a TEXT,
+ b TEXT,
+ c TEXT
+);
+
+CREATE STATISTICS s2 ON functional_dependencies (a, b, c) WITH (dependencies);
+
+-- random data (no functional dependencies)
+INSERT INTO functional_dependencies
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+
+ANALYZE functional_dependencies;
+
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+
+TRUNCATE functional_dependencies;
+
+-- a => b, a => c, b => c
+INSERT INTO functional_dependencies
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+
+ANALYZE functional_dependencies;
+
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+
+TRUNCATE functional_dependencies;
+
+-- a => b, a => c
+INSERT INTO functional_dependencies
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE functional_dependencies;
+
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+
+TRUNCATE functional_dependencies;
+
+-- check explain (expect bitmap index scan, not plain index scan)
+INSERT INTO functional_dependencies
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX fdeps_idx ON functional_dependencies (a, b);
+ANALYZE functional_dependencies;
+
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+
+EXPLAIN (COSTS off)
+ SELECT * FROM functional_dependencies WHERE a = '10' AND b = '5';
+
+DROP TABLE functional_dependencies;
+
+-- NULL values (mix of int and text columns)
+CREATE TABLE functional_dependencies (
+ a INT,
+ b TEXT,
+ c INT,
+ d TEXT
+);
+
+CREATE STATISTICS s3 ON functional_dependencies (a, b, c, d) WITH (dependencies);
+
+INSERT INTO functional_dependencies
+ SELECT
+ mod(i, 100),
+ (CASE WHEN mod(i, 200) = 0 THEN NULL ELSE mod(i,200) END),
+ mod(i, 400),
+ (CASE WHEN mod(i, 300) = 0 THEN NULL ELSE mod(i,600) END)
+ FROM generate_series(1,10000) s(i);
+
+ANALYZE functional_dependencies;
+
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+
+DROP TABLE functional_dependencies;
--
2.5.0
[text/x-patch] 0002-shared-infrastructure-and-functional-dependencies.patch (118.4K, ../../1457906378.27231.10.camel@2ndquadrant.com/8-0002-shared-infrastructure-and-functional-dependencies.patch)
download | inline diff:
From 2c719f145ab1f74e87b0da2c4cff67af9f4a8e50 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Sun, 11 Jan 2015 19:51:48 +0100
Subject: [PATCH 2/9] shared infrastructure and functional dependencies
Basic infrastructure shared by all kinds of multivariate stats, most
importantly:
- adds a new system catalog (pg_mv_statistic)
- CREATE STATISTICS name ON table (columns) WITH (options)
- DROP STATISTICS name
- ALTER STATISTICS ... OWNER TO / SET SCHEMA / RENAME
- implementation of functional dependencies (the simplest type of
multivariate statistics)
- building functional dependencies in ANALYZE
- updates regression tests (new catalog etc.)
This does not include any changes to the optimizer, i.e. it does not
influence the query planning (subject to follow-up patches).
The current implementation requires a valid 'ltopr' for the columns, so
that we can sort the sample rows in various ways, both in this patch
and other kinds of statistics. Maybe this restriction could be relaxed
in the future, requiring just 'eqopr' in case of stats not sorting the
data (e.g. functional dependencies and MCV lists).
Maybe some of the stats (functional dependencies and MCV list with
limited functionality) might be made to work with hashes of the values,
which is sufficient for equality comparisons. But the queries would
require the equality operator anyway, so it's not really a weaker
requirement. The hashes might reduce space requirements, though.
The algorithm detecting the dependencies is rather simple and probably
needs improvements, so that it detects more complicated dependencies,
and also validation of the math.
The name 'functional dependencies' is more correct (than 'association
rules') as it's exactly the name used in relational theory (esp. Normal
Forms) for tracking column-level dependencies.
The multivariate statistics are automatically removed in two situations
(a) after a DROP TABLE (obviously)
(b) after ALTER TABLE ... DROP COLUMN, if the statistics would be
defined on less than 2 columns (remaining)
If there are more at least two remaining columns, we keep the
statistics but perform cleanup on the next ANALYZE. The dropped columns
are removed from stakeys, and the new statistics is built on the
smaller set.
We can't do this at DROP COLUMN, because that'd leave us with invalid
statistics, or we'd have to throw it away although we can still use it.
This lazy approach lets us use the statistics although some of the
columns are dead.
This also adds a simple list of statistics to \d in psql.
This means the statistics are created within a schema by using a
qualified name (or using the default schema)
CREATE STATISTICS schema.statistics ON ...
and then dropped by specifying qualified name
DROP STATISTICS schema.statistics
or searching through search_path (just like with other objects).
This also gets rid of the "(opt_)stats_name" definitions in gram.y and
instead replaces them with just "opt_any_name", although the optional
case is not really handled currently - there's no generated name yet
(so either we should drop it or implement it).
I'm not entirely sure making statistics schema-specific is that a great
idea. Maybe it should be "global", but that does not seem right (e.g.
it makes multi-tenant systems based on schemas more difficult to
manage, because tenants would interact).
---
doc/src/sgml/ref/allfiles.sgml | 3 +
doc/src/sgml/ref/alter_statistics.sgml | 115 +++++++
doc/src/sgml/ref/create_statistics.sgml | 198 ++++++++++++
doc/src/sgml/ref/drop_statistics.sgml | 91 ++++++
doc/src/sgml/reference.sgml | 2 +
src/backend/catalog/Makefile | 1 +
src/backend/catalog/aclchk.c | 27 ++
src/backend/catalog/dependency.c | 11 +-
src/backend/catalog/heap.c | 102 ++++++
src/backend/catalog/namespace.c | 51 +++
src/backend/catalog/objectaddress.c | 54 ++++
src/backend/catalog/system_views.sql | 11 +
src/backend/commands/Makefile | 6 +-
src/backend/commands/alter.c | 3 +
src/backend/commands/analyze.c | 21 ++
src/backend/commands/dropcmds.c | 4 +
src/backend/commands/event_trigger.c | 3 +
src/backend/commands/statscmds.c | 277 ++++++++++++++++
src/backend/nodes/copyfuncs.c | 17 +
src/backend/nodes/outfuncs.c | 18 ++
src/backend/optimizer/util/plancat.c | 59 ++++
src/backend/parser/gram.y | 60 +++-
src/backend/tcop/utility.c | 14 +
src/backend/utils/Makefile | 2 +-
src/backend/utils/cache/relcache.c | 59 ++++
src/backend/utils/cache/syscache.c | 23 ++
src/backend/utils/mvstats/Makefile | 17 +
src/backend/utils/mvstats/README.dependencies | 222 +++++++++++++
src/backend/utils/mvstats/common.c | 356 +++++++++++++++++++++
src/backend/utils/mvstats/common.h | 75 +++++
src/backend/utils/mvstats/dependencies.c | 437 ++++++++++++++++++++++++++
src/bin/psql/describe.c | 44 +++
src/include/catalog/dependency.h | 5 +-
src/include/catalog/heap.h | 1 +
src/include/catalog/indexing.h | 7 +
src/include/catalog/namespace.h | 2 +
src/include/catalog/pg_mv_statistic.h | 75 +++++
src/include/catalog/pg_proc.h | 5 +
src/include/catalog/toasting.h | 1 +
src/include/commands/defrem.h | 4 +
src/include/nodes/nodes.h | 2 +
src/include/nodes/parsenodes.h | 12 +
src/include/nodes/relation.h | 28 ++
src/include/utils/acl.h | 1 +
src/include/utils/mvstats.h | 70 +++++
src/include/utils/rel.h | 4 +
src/include/utils/relcache.h | 1 +
src/include/utils/syscache.h | 2 +
src/test/regress/expected/object_address.out | 7 +-
src/test/regress/expected/rules.out | 9 +
src/test/regress/expected/sanity_check.out | 1 +
src/test/regress/sql/object_address.sql | 4 +-
52 files changed, 2613 insertions(+), 11 deletions(-)
create mode 100644 doc/src/sgml/ref/alter_statistics.sgml
create mode 100644 doc/src/sgml/ref/create_statistics.sgml
create mode 100644 doc/src/sgml/ref/drop_statistics.sgml
create mode 100644 src/backend/commands/statscmds.c
create mode 100644 src/backend/utils/mvstats/Makefile
create mode 100644 src/backend/utils/mvstats/README.dependencies
create mode 100644 src/backend/utils/mvstats/common.c
create mode 100644 src/backend/utils/mvstats/common.h
create mode 100644 src/backend/utils/mvstats/dependencies.c
create mode 100644 src/include/catalog/pg_mv_statistic.h
create mode 100644 src/include/utils/mvstats.h
diff --git a/doc/src/sgml/ref/allfiles.sgml b/doc/src/sgml/ref/allfiles.sgml
index bf95453..524ed83 100644
--- a/doc/src/sgml/ref/allfiles.sgml
+++ b/doc/src/sgml/ref/allfiles.sgml
@@ -32,6 +32,7 @@ Complete list of usable sgml source files in this directory.
<!ENTITY alterServer SYSTEM "alter_server.sgml">
<!ENTITY alterSequence SYSTEM "alter_sequence.sgml">
<!ENTITY alterSystem SYSTEM "alter_system.sgml">
+<!ENTITY alterStatistics SYSTEM "alter_statistics.sgml">
<!ENTITY alterTable SYSTEM "alter_table.sgml">
<!ENTITY alterTableSpace SYSTEM "alter_tablespace.sgml">
<!ENTITY alterTSConfig SYSTEM "alter_tsconfig.sgml">
@@ -76,6 +77,7 @@ Complete list of usable sgml source files in this directory.
<!ENTITY createSchema SYSTEM "create_schema.sgml">
<!ENTITY createSequence SYSTEM "create_sequence.sgml">
<!ENTITY createServer SYSTEM "create_server.sgml">
+<!ENTITY createStatistics SYSTEM "create_statistics.sgml">
<!ENTITY createTable SYSTEM "create_table.sgml">
<!ENTITY createTableAs SYSTEM "create_table_as.sgml">
<!ENTITY createTableSpace SYSTEM "create_tablespace.sgml">
@@ -119,6 +121,7 @@ Complete list of usable sgml source files in this directory.
<!ENTITY dropSchema SYSTEM "drop_schema.sgml">
<!ENTITY dropSequence SYSTEM "drop_sequence.sgml">
<!ENTITY dropServer SYSTEM "drop_server.sgml">
+<!ENTITY dropStatistics SYSTEM "drop_statistics.sgml">
<!ENTITY dropTable SYSTEM "drop_table.sgml">
<!ENTITY dropTableSpace SYSTEM "drop_tablespace.sgml">
<!ENTITY dropTransform SYSTEM "drop_transform.sgml">
diff --git a/doc/src/sgml/ref/alter_statistics.sgml b/doc/src/sgml/ref/alter_statistics.sgml
new file mode 100644
index 0000000..aa421c0
--- /dev/null
+++ b/doc/src/sgml/ref/alter_statistics.sgml
@@ -0,0 +1,115 @@
+<!--
+doc/src/sgml/ref/alter_statistics.sgml
+PostgreSQL documentation
+-->
+
+<refentry id="SQL-ALTERSTATISTICS">
+ <indexterm zone="sql-alterstatistics">
+ <primary>ALTER STATISTICS</primary>
+ </indexterm>
+
+ <refmeta>
+ <refentrytitle>ALTER STATISTICS</refentrytitle>
+ <manvolnum>7</manvolnum>
+ <refmiscinfo>SQL - Language Statements</refmiscinfo>
+ </refmeta>
+
+ <refnamediv>
+ <refname>ALTER STATISTICS</refname>
+ <refpurpose>
+ change the definition of a multivariate statistics
+ </refpurpose>
+ </refnamediv>
+
+ <refsynopsisdiv>
+<synopsis>
+ALTER STATISTICS <replaceable class="parameter">name</replaceable> OWNER TO { <replaceable class="PARAMETER">new_owner</replaceable> | CURRENT_USER | SESSION_USER }
+ALTER STATISTICS <replaceable class="parameter">name</replaceable> RENAME TO <replaceable class="parameter">new_name</replaceable>
+ALTER STATISTICS <replaceable class="parameter">name</replaceable> SET SCHEMA <replaceable class="parameter">new_schema</replaceable>
+</synopsis>
+ </refsynopsisdiv>
+
+ <refsect1>
+ <title>Description</title>
+
+ <para>
+ <command>ALTER STATISTICS</command> changes the parameters of an existing
+ multivariate statistics. Any parameters not specifically set in the
+ <command>ALTER STATISTICS</command> command retain their prior settings.
+ </para>
+
+ <para>
+ You must own the statistics to use <command>ALTER STATISTICS</>.
+ To change a statistics' schema, you must also have <literal>CREATE</>
+ privilege on the new schema.
+ To alter the owner, you must also be a direct or indirect member of the new
+ owning role, and that role must have <literal>CREATE</literal> privilege on
+ the statistics' schema. (These restrictions enforce that altering the owner
+ doesn't do anything you couldn't do by dropping and recreating the statistics.
+ However, a superuser can alter ownership of any statistics anyway.)
+ </para>
+ </refsect1>
+
+ <refsect1>
+ <title>Parameters</title>
+
+ <para>
+ <variablelist>
+ <varlistentry>
+ <term><replaceable class="parameter">name</replaceable></term>
+ <listitem>
+ <para>
+ The name (optionally schema-qualified) of a statistics to be altered.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><replaceable class="PARAMETER">new_owner</replaceable></term>
+ <listitem>
+ <para>
+ The user name of the new owner of the statistics.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><replaceable class="parameter">new_name</replaceable></term>
+ <listitem>
+ <para>
+ The new name for the statistics.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><replaceable class="parameter">new_schema</replaceable></term>
+ <listitem>
+ <para>
+ The new schema for the statistics.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+ </refsect1>
+
+ <refsect1>
+ <title>Compatibility</title>
+
+ <para>
+ There's no <command>ALTER STATISTICS</command> command in the SQL standard.
+ </para>
+ </refsect1>
+
+ <refsect1>
+ <title>See Also</title>
+
+ <simplelist type="inline">
+ <member><xref linkend="sql-createstatistics"></member>
+ <member><xref linkend="sql-dropstatistics"></member>
+ </simplelist>
+ </refsect1>
+
+</refentry>
diff --git a/doc/src/sgml/ref/create_statistics.sgml b/doc/src/sgml/ref/create_statistics.sgml
new file mode 100644
index 0000000..ff09fa5
--- /dev/null
+++ b/doc/src/sgml/ref/create_statistics.sgml
@@ -0,0 +1,198 @@
+<!--
+doc/src/sgml/ref/create_statistics.sgml
+PostgreSQL documentation
+-->
+
+<refentry id="SQL-CREATESTATISTICS">
+ <indexterm zone="sql-createstatistics">
+ <primary>CREATE STATISTICS</primary>
+ </indexterm>
+
+ <refmeta>
+ <refentrytitle>CREATE STATISTICS</refentrytitle>
+ <manvolnum>7</manvolnum>
+ <refmiscinfo>SQL - Language Statements</refmiscinfo>
+ </refmeta>
+
+ <refnamediv>
+ <refname>CREATE STATISTICS</refname>
+ <refpurpose>define a new statistics</refpurpose>
+ </refnamediv>
+
+ <refsynopsisdiv>
+<synopsis>
+CREATE STATISTICS [ IF NOT EXISTS ] <replaceable class="PARAMETER">statistics_name</replaceable> ON <replaceable class="PARAMETER">table_name</replaceable> ( [
+ { <replaceable class="PARAMETER">column_name</replaceable> } ] [, ...])
+[ WITH ( <replaceable class="PARAMETER">statistics_parameter</replaceable> [= <replaceable class="PARAMETER">value</replaceable>] [, ... ] )
+</synopsis>
+
+ </refsynopsisdiv>
+
+ <refsect1 id="SQL-CREATESTATISTICS-description">
+ <title>Description</title>
+
+ <para>
+ <command>CREATE STATISTICS</command> will create a new multivariate
+ statistics on the table. The statistics will be created in the in the
+ current database. The statistics will be owned by the user issuing
+ the command.
+ </para>
+
+ <para>
+ If a schema name is given (for example, <literal>CREATE STATISTICS
+ myschema.mystat ...</>) then the statistics is created in the specified
+ schema. Otherwise it is created in the current schema. The name of
+ the table must be distinct from the name of any other statistics in the
+ same schema.
+ </para>
+
+ <para>
+ To be able to create a table, you must have <literal>USAGE</literal>
+ privilege on all column types or the type in the <literal>OF</literal>
+ clause, respectively.
+ </para>
+ </refsect1>
+
+ <refsect1>
+ <title>Parameters</title>
+
+ <variablelist>
+
+ <varlistentry>
+ <term><literal>IF NOT EXISTS</></term>
+ <listitem>
+ <para>
+ Do not throw an error if a statistics with the same name already exists.
+ A notice is issued in this case. Note that there is no guarantee that
+ the existing statistics is anything like the one that would have been
+ created.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><replaceable class="PARAMETER">statistics_name</replaceable></term>
+ <listitem>
+ <para>
+ The name (optionally schema-qualified) of the statistics to be created.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><replaceable class="PARAMETER">table_name</replaceable></term>
+ <listitem>
+ <para>
+ The name (optionally schema-qualified) of the table the statistics should
+ be created on.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><replaceable class="PARAMETER">column_name</replaceable></term>
+ <listitem>
+ <para>
+ The name of a column to be included in the statistics.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>WITH ( <replaceable class="PARAMETER">statistics_parameter</replaceable> [= <replaceable class="PARAMETER">value</replaceable>] [, ... ] )</literal></term>
+ <listitem>
+ <para>
+ ...
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ <refsect2 id="SQL-CREATESTATISTICS-parameters">
+ <title id="SQL-CREATESTATISTICS-parameters-title">Statistics Parameters</title>
+
+ <indexterm zone="sql-createstatistics-parameters">
+ <primary>statistics parameters</primary>
+ </indexterm>
+
+ <para>
+ The <literal>WITH</> clause can specify <firstterm>statistics parameters</>
+ for statistics. The currently available parameters are listed below.
+ </para>
+
+ <variablelist>
+
+ <varlistentry>
+ <term><literal>dependencies</> (<type>boolean</>)</term>
+ <listitem>
+ <para>
+ Enables functional dependencies for the statistics.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </refsect2>
+ </refsect1>
+
+ <refsect1 id="SQL-CREATESTATISTICS-notes">
+ <title>Notes</title>
+
+ <para>
+ ...
+ </para>
+
+ </refsect1>
+
+
+ <refsect1 id="SQL-CREATESTATISTICS-examples">
+ <title>Examples</title>
+
+ <para>
+ Create table <structname>t1</> with two functionally dependent columns, i.e.
+ knowledge of a value in the first column is sufficient for detemining the
+ value in the other column. Then functional dependencies are built on those
+ columns:
+
+<programlisting>
+CREATE TABLE t1 (
+ a int,
+ b int
+);
+
+INSERT INTO t1 SELECT i/100, i/500
+ FROM generate_series(1,1000000) s(i);
+
+CREATE STATISTICS s1 ON t1 (a, b) WITH (dependencies);
+
+ANALYZE t1;
+
+-- valid combination of values
+EXPLAIN ANALYZE SELECT * FROM t1 WHERE (a = 1) AND (b = 1);
+
+-- invalid combination of values
+EXPLAIN ANALYZE SELECT * FROM t1 WHERE (a = 1) AND (b = 2);
+</programlisting>
+ </para>
+
+ </refsect1>
+
+ <refsect1>
+ <title>Compatibility</title>
+
+ <para>
+ There's no <command>CREATE STATISTICS</command> command in the SQL standard.
+ </para>
+ </refsect1>
+
+ <refsect1>
+ <title>See Also</title>
+
+ <simplelist type="inline">
+ <member><xref linkend="sql-alterstatistics"></member>
+ <member><xref linkend="sql-dropstatistics"></member>
+ </simplelist>
+ </refsect1>
+</refentry>
diff --git a/doc/src/sgml/ref/drop_statistics.sgml b/doc/src/sgml/ref/drop_statistics.sgml
new file mode 100644
index 0000000..dd9047a
--- /dev/null
+++ b/doc/src/sgml/ref/drop_statistics.sgml
@@ -0,0 +1,91 @@
+<!--
+doc/src/sgml/ref/drop_statistics.sgml
+PostgreSQL documentation
+-->
+
+<refentry id="SQL-DROPSTATISTICS">
+ <indexterm zone="sql-dropstatistics">
+ <primary>DROP STATISTICS</primary>
+ </indexterm>
+
+ <refmeta>
+ <refentrytitle>DROP STATISTICS</refentrytitle>
+ <manvolnum>7</manvolnum>
+ <refmiscinfo>SQL - Language Statements</refmiscinfo>
+ </refmeta>
+
+ <refnamediv>
+ <refname>DROP STATISTICS</refname>
+ <refpurpose>remove a statistics</refpurpose>
+ </refnamediv>
+
+ <refsynopsisdiv>
+<synopsis>
+DROP STATISTICS [ IF EXISTS ] <replaceable class="PARAMETER">name</replaceable> [, ...]
+</synopsis>
+ </refsynopsisdiv>
+
+ <refsect1>
+ <title>Description</title>
+
+ <para>
+ <command>DROP STATISTICS</command> removes statistics from the database.
+ Only the statistics owner, the schema owner, and superuser can drop a
+ statistics.
+ </para>
+
+ </refsect1>
+
+ <refsect1>
+ <title>Parameters</title>
+
+ <variablelist>
+ <varlistentry>
+ <term><literal>IF EXISTS</literal></term>
+ <listitem>
+ <para>
+ Do not throw an error if the statistics does not exist. A notice is
+ issued in this case.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><replaceable class="PARAMETER">name</replaceable></term>
+ <listitem>
+ <para>
+ The name (optionally schema-qualified) of the statistics to drop.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </refsect1>
+
+ <refsect1>
+ <title>Examples</title>
+
+ <para>
+ ...
+ </para>
+
+ </refsect1>
+
+ <refsect1>
+ <title>Compatibility</title>
+
+ <para>
+ There's no <command>DROP STATISTICS</command> command in the SQL standard.
+ </para>
+ </refsect1>
+
+ <refsect1>
+ <title>See Also</title>
+
+ <simplelist type="inline">
+ <member><xref linkend="sql-alterstatistics"></member>
+ <member><xref linkend="sql-createstatistics"></member>
+ </simplelist>
+ </refsect1>
+
+</refentry>
diff --git a/doc/src/sgml/reference.sgml b/doc/src/sgml/reference.sgml
index 03020df..2b07b2d 100644
--- a/doc/src/sgml/reference.sgml
+++ b/doc/src/sgml/reference.sgml
@@ -104,6 +104,7 @@
&createSchema;
&createSequence;
&createServer;
+ &createStatistics;
&createTable;
&createTableAs;
&createTableSpace;
@@ -147,6 +148,7 @@
&dropSchema;
&dropSequence;
&dropServer;
+ &dropStatistics;
&dropTable;
&dropTableSpace;
&dropTSConfig;
diff --git a/src/backend/catalog/Makefile b/src/backend/catalog/Makefile
index 25130ec..058b8a9 100644
--- a/src/backend/catalog/Makefile
+++ b/src/backend/catalog/Makefile
@@ -32,6 +32,7 @@ POSTGRES_BKI_SRCS = $(addprefix $(top_srcdir)/src/include/catalog/,\
pg_attrdef.h pg_constraint.h pg_inherits.h pg_index.h pg_operator.h \
pg_opfamily.h pg_opclass.h pg_am.h pg_amop.h pg_amproc.h \
pg_language.h pg_largeobject_metadata.h pg_largeobject.h pg_aggregate.h \
+ pg_mv_statistic.h \
pg_statistic.h pg_rewrite.h pg_trigger.h pg_event_trigger.h pg_description.h \
pg_cast.h pg_enum.h pg_namespace.h pg_conversion.h pg_depend.h \
pg_database.h pg_db_role_setting.h pg_tablespace.h pg_pltemplate.h \
diff --git a/src/backend/catalog/aclchk.c b/src/backend/catalog/aclchk.c
index 0f3bc07..e21aacd 100644
--- a/src/backend/catalog/aclchk.c
+++ b/src/backend/catalog/aclchk.c
@@ -38,6 +38,7 @@
#include "catalog/pg_language.h"
#include "catalog/pg_largeobject.h"
#include "catalog/pg_largeobject_metadata.h"
+#include "catalog/pg_mv_statistic.h"
#include "catalog/pg_namespace.h"
#include "catalog/pg_opclass.h"
#include "catalog/pg_operator.h"
@@ -5021,6 +5022,32 @@ pg_extension_ownercheck(Oid ext_oid, Oid roleid)
}
/*
+ * Ownership check for a multivariate statistics (specified by OID).
+ */
+bool
+pg_statistics_ownercheck(Oid stat_oid, Oid roleid)
+{
+ HeapTuple tuple;
+ Oid ownerId;
+
+ /* Superusers bypass all permission checking. */
+ if (superuser_arg(roleid))
+ return true;
+
+ tuple = SearchSysCache1(MVSTATOID, ObjectIdGetDatum(stat_oid));
+ if (!HeapTupleIsValid(tuple))
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("statistics with OID %u does not exist", stat_oid)));
+
+ ownerId = ((Form_pg_mv_statistic) GETSTRUCT(tuple))->staowner;
+
+ ReleaseSysCache(tuple);
+
+ return has_privs_of_role(roleid, ownerId);
+}
+
+/*
* Check whether specified role has CREATEROLE privilege (or is a superuser)
*
* Note: roles do not have owners per se; instead we use this test in
diff --git a/src/backend/catalog/dependency.c b/src/backend/catalog/dependency.c
index c48e37b..8200454 100644
--- a/src/backend/catalog/dependency.c
+++ b/src/backend/catalog/dependency.c
@@ -40,6 +40,7 @@
#include "catalog/pg_foreign_server.h"
#include "catalog/pg_language.h"
#include "catalog/pg_largeobject.h"
+#include "catalog/pg_mv_statistic.h"
#include "catalog/pg_namespace.h"
#include "catalog/pg_opclass.h"
#include "catalog/pg_operator.h"
@@ -160,7 +161,8 @@ static const Oid object_classes[] = {
ExtensionRelationId, /* OCLASS_EXTENSION */
EventTriggerRelationId, /* OCLASS_EVENT_TRIGGER */
PolicyRelationId, /* OCLASS_POLICY */
- TransformRelationId /* OCLASS_TRANSFORM */
+ TransformRelationId, /* OCLASS_TRANSFORM */
+ MvStatisticRelationId /* OCLASS_STATISTICS */
};
@@ -1272,6 +1274,10 @@ doDeletion(const ObjectAddress *object, int flags)
DropTransformById(object->objectId);
break;
+ case OCLASS_STATISTICS:
+ RemoveStatisticsById(object->objectId);
+ break;
+
default:
elog(ERROR, "unrecognized object class: %u",
object->classId);
@@ -2415,6 +2421,9 @@ getObjectClass(const ObjectAddress *object)
case TransformRelationId:
return OCLASS_TRANSFORM;
+
+ case MvStatisticRelationId:
+ return OCLASS_STATISTICS;
}
/* shouldn't get here */
diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c
index e997b57..47ec8cc 100644
--- a/src/backend/catalog/heap.c
+++ b/src/backend/catalog/heap.c
@@ -47,6 +47,7 @@
#include "catalog/pg_constraint_fn.h"
#include "catalog/pg_foreign_table.h"
#include "catalog/pg_inherits.h"
+#include "catalog/pg_mv_statistic.h"
#include "catalog/pg_namespace.h"
#include "catalog/pg_statistic.h"
#include "catalog/pg_tablespace.h"
@@ -1613,7 +1614,10 @@ RemoveAttributeById(Oid relid, AttrNumber attnum)
heap_close(attr_rel, RowExclusiveLock);
if (attnum > 0)
+ {
RemoveStatistics(relid, attnum);
+ RemoveMVStatistics(relid, attnum);
+ }
relation_close(rel, NoLock);
}
@@ -1841,6 +1845,11 @@ heap_drop_with_catalog(Oid relid)
RemoveStatistics(relid, 0);
/*
+ * delete multi-variate statistics
+ */
+ RemoveMVStatistics(relid, 0);
+
+ /*
* delete attribute tuples
*/
DeleteAttributeTuples(relid);
@@ -2692,6 +2701,99 @@ RemoveStatistics(Oid relid, AttrNumber attnum)
/*
+ * RemoveMVStatistics --- remove entries in pg_mv_statistic for a rel
+ *
+ * If attnum is zero, remove all entries for rel; else remove only the one(s)
+ * for that column.
+ */
+void
+RemoveMVStatistics(Oid relid, AttrNumber attnum)
+{
+ Relation pgmvstatistic;
+ TupleDesc tupdesc = NULL;
+ SysScanDesc scan;
+ ScanKeyData key;
+ HeapTuple tuple;
+
+ /*
+ * When dropping a column, we'll drop statistics with a single
+ * remaining (undropped column). To do that, we need the tuple
+ * descriptor.
+ *
+ * We already have the relation locked (as we're running ALTER
+ * TABLE ... DROP COLUMN), so we'll just get the descriptor here.
+ */
+ if (attnum != 0)
+ {
+ Relation rel = relation_open(relid, NoLock);
+
+ /* multivariate stats are supported on tables and matviews */
+ if (rel->rd_rel->relkind == RELKIND_RELATION ||
+ rel->rd_rel->relkind == RELKIND_MATVIEW)
+ tupdesc = RelationGetDescr(rel);
+
+ relation_close(rel, NoLock);
+ }
+
+ if (tupdesc == NULL)
+ return;
+
+ pgmvstatistic = heap_open(MvStatisticRelationId, RowExclusiveLock);
+
+ ScanKeyInit(&key,
+ Anum_pg_mv_statistic_starelid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(relid));
+
+ scan = systable_beginscan(pgmvstatistic,
+ MvStatisticRelidIndexId,
+ true, NULL, 1, &key);
+
+ /* we must loop even when attnum != 0, in case of inherited stats */
+ while (HeapTupleIsValid(tuple = systable_getnext(scan)))
+ {
+ bool delete = true;
+
+ if (attnum != 0)
+ {
+ Datum adatum;
+ bool isnull;
+ int i;
+ int ncolumns = 0;
+ ArrayType *arr;
+ int16 *attnums;
+
+ /* get the columns */
+ adatum = SysCacheGetAttr(MVSTATOID, tuple,
+ Anum_pg_mv_statistic_stakeys, &isnull);
+ Assert(!isnull);
+
+ arr = DatumGetArrayTypeP(adatum);
+ attnums = (int16*)ARR_DATA_PTR(arr);
+
+ for (i = 0; i < ARR_DIMS(arr)[0]; i++)
+ {
+ /* count the column unless it's has been / is being dropped */
+ if ((! tupdesc->attrs[attnums[i]-1]->attisdropped) &&
+ (attnums[i] != attnum))
+ ncolumns += 1;
+ }
+
+ /* delete if there are less than two attributes */
+ delete = (ncolumns < 2);
+ }
+
+ if (delete)
+ simple_heap_delete(pgmvstatistic, &tuple->t_self);
+ }
+
+ systable_endscan(scan);
+
+ heap_close(pgmvstatistic, RowExclusiveLock);
+}
+
+
+/*
* RelationTruncateIndexes - truncate all indexes associated
* with the heap relation to zero tuples.
*
diff --git a/src/backend/catalog/namespace.c b/src/backend/catalog/namespace.c
index 446b2ac..dfd5bef 100644
--- a/src/backend/catalog/namespace.c
+++ b/src/backend/catalog/namespace.c
@@ -4201,3 +4201,54 @@ pg_is_other_temp_schema(PG_FUNCTION_ARGS)
PG_RETURN_BOOL(isOtherTempNamespace(oid));
}
+
+Oid
+get_statistics_oid(List *names, bool missing_ok)
+{
+ char *schemaname;
+ char *stats_name;
+ Oid namespaceId;
+ Oid stats_oid = InvalidOid;
+ ListCell *l;
+
+ /* deconstruct the name list */
+ DeconstructQualifiedName(names, &schemaname, &stats_name);
+
+ if (schemaname)
+ {
+ /* use exact schema given */
+ namespaceId = LookupExplicitNamespace(schemaname, missing_ok);
+ if (missing_ok && !OidIsValid(namespaceId))
+ stats_oid = InvalidOid;
+ else
+ stats_oid = GetSysCacheOid2(MVSTATNAMENSP,
+ PointerGetDatum(stats_name),
+ ObjectIdGetDatum(namespaceId));
+ }
+ else
+ {
+ /* search for it in search path */
+ recomputeNamespacePath();
+
+ foreach(l, activeSearchPath)
+ {
+ namespaceId = lfirst_oid(l);
+
+ if (namespaceId == myTempNamespace)
+ continue; /* do not look in temp namespace */
+ stats_oid = GetSysCacheOid2(MVSTATNAMENSP,
+ PointerGetDatum(stats_name),
+ ObjectIdGetDatum(namespaceId));
+ if (OidIsValid(stats_oid))
+ break;
+ }
+ }
+
+ if (!OidIsValid(stats_oid) && !missing_ok)
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("statistics \"%s\" does not exist",
+ NameListToString(names))));
+
+ return stats_oid;
+}
diff --git a/src/backend/catalog/objectaddress.c b/src/backend/catalog/objectaddress.c
index d2aaa6d..c13a569 100644
--- a/src/backend/catalog/objectaddress.c
+++ b/src/backend/catalog/objectaddress.c
@@ -39,6 +39,7 @@
#include "catalog/pg_language.h"
#include "catalog/pg_largeobject.h"
#include "catalog/pg_largeobject_metadata.h"
+#include "catalog/pg_mv_statistic.h"
#include "catalog/pg_namespace.h"
#include "catalog/pg_opclass.h"
#include "catalog/pg_opfamily.h"
@@ -438,9 +439,22 @@ static const ObjectPropertyType ObjectProperty[] =
Anum_pg_type_typacl,
ACL_KIND_TYPE,
true
+ },
+ {
+ MvStatisticRelationId,
+ MvStatisticOidIndexId,
+ MVSTATOID,
+ MVSTATNAMENSP,
+ Anum_pg_mv_statistic_staname,
+ Anum_pg_mv_statistic_stanamespace,
+ Anum_pg_mv_statistic_staowner,
+ InvalidAttrNumber, /* no ACL (same as relation) */
+ -1, /* no ACL */
+ true
}
};
+
/*
* This struct maps the string object types as returned by
* getObjectTypeDescription into ObjType enum values. Note that some enum
@@ -640,6 +654,10 @@ static const struct object_type_map
/* OCLASS_TRANSFORM */
{
"transform", OBJECT_TRANSFORM
+ },
+ /* OBJECT_STATISTICS */
+ {
+ "statistics", OBJECT_STATISTICS
}
};
@@ -913,6 +931,11 @@ get_object_address(ObjectType objtype, List *objname, List *objargs,
address = get_object_address_defacl(objname, objargs,
missing_ok);
break;
+ case OBJECT_STATISTICS:
+ address.classId = MvStatisticRelationId;
+ address.objectId = get_statistics_oid(objname, missing_ok);
+ address.objectSubId = 0;
+ break;
default:
elog(ERROR, "unrecognized objtype: %d", (int) objtype);
/* placate compiler, in case it thinks elog might return */
@@ -2185,6 +2208,10 @@ check_object_ownership(Oid roleid, ObjectType objtype, ObjectAddress address,
(errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
errmsg("must be superuser")));
break;
+ case OBJECT_STATISTICS:
+ if (!pg_statistics_ownercheck(address.objectId, roleid))
+ aclcheck_error_type(ACLCHECK_NOT_OWNER, address.objectId);
+ break;
default:
elog(ERROR, "unrecognized object type: %d",
(int) objtype);
@@ -3610,6 +3637,10 @@ getObjectTypeDescription(const ObjectAddress *object)
appendStringInfoString(&buffer, "transform");
break;
+ case OCLASS_STATISTICS:
+ appendStringInfoString(&buffer, "statistics");
+ break;
+
default:
appendStringInfo(&buffer, "unrecognized %u", object->classId);
break;
@@ -4566,6 +4597,29 @@ getObjectIdentityParts(const ObjectAddress *object,
}
break;
+ case OCLASS_STATISTICS:
+ {
+ HeapTuple tup;
+ Form_pg_mv_statistic formStatistic;
+ char *schema;
+
+ tup = SearchSysCache1(MVSTATOID,
+ ObjectIdGetDatum(object->objectId));
+ if (!HeapTupleIsValid(tup))
+ elog(ERROR, "cache lookup failed for statistics %u",
+ object->objectId);
+ formStatistic = (Form_pg_mv_statistic) GETSTRUCT(tup);
+ schema = get_namespace_name_or_temp(formStatistic->stanamespace);
+ appendStringInfoString(&buffer,
+ quote_qualified_identifier(schema,
+ NameStr(formStatistic->staname)));
+ if (objname)
+ *objname = list_make2(schema,
+ pstrdup(NameStr(formStatistic->staname)));
+ ReleaseSysCache(tup);
+ break;
+ }
+
default:
appendStringInfo(&buffer, "unrecognized object %u %u %d",
object->classId,
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 84aa061..31dbb2c 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -158,6 +158,17 @@ CREATE VIEW pg_indexes AS
LEFT JOIN pg_tablespace T ON (T.oid = I.reltablespace)
WHERE C.relkind IN ('r', 'm') AND I.relkind = 'i';
+CREATE VIEW pg_mv_stats AS
+ SELECT
+ N.nspname AS schemaname,
+ C.relname AS tablename,
+ S.staname AS staname,
+ S.stakeys AS attnums,
+ length(S.stadeps) as depsbytes,
+ pg_mv_stats_dependencies_info(S.stadeps) as depsinfo
+ FROM (pg_mv_statistic S JOIN pg_class C ON (C.oid = S.starelid))
+ LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace);
+
CREATE VIEW pg_stats WITH (security_barrier) AS
SELECT
nspname AS schemaname,
diff --git a/src/backend/commands/Makefile b/src/backend/commands/Makefile
index b1ac704..5151001 100644
--- a/src/backend/commands/Makefile
+++ b/src/backend/commands/Makefile
@@ -18,8 +18,8 @@ OBJS = aggregatecmds.o alter.o analyze.o async.o cluster.o comment.o \
event_trigger.o explain.o extension.o foreigncmds.o functioncmds.o \
indexcmds.o lockcmds.o matview.o operatorcmds.o opclasscmds.o \
policy.o portalcmds.o prepare.o proclang.o \
- schemacmds.o seclabel.o sequence.o tablecmds.o tablespace.o trigger.o \
- tsearchcmds.o typecmds.o user.o vacuum.o vacuumlazy.o \
- variable.o view.o
+ schemacmds.o seclabel.o sequence.o statscmds.o \
+ tablecmds.o tablespace.o trigger.o tsearchcmds.o typecmds.o \
+ user.o vacuum.o vacuumlazy.o variable.o view.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/commands/alter.c b/src/backend/commands/alter.c
index 5af0f2f..89985499 100644
--- a/src/backend/commands/alter.c
+++ b/src/backend/commands/alter.c
@@ -359,6 +359,7 @@ ExecRenameStmt(RenameStmt *stmt)
case OBJECT_OPCLASS:
case OBJECT_OPFAMILY:
case OBJECT_LANGUAGE:
+ case OBJECT_STATISTICS:
case OBJECT_TSCONFIGURATION:
case OBJECT_TSDICTIONARY:
case OBJECT_TSPARSER:
@@ -437,6 +438,7 @@ ExecAlterObjectSchemaStmt(AlterObjectSchemaStmt *stmt,
case OBJECT_OPERATOR:
case OBJECT_OPCLASS:
case OBJECT_OPFAMILY:
+ case OBJECT_STATISTICS:
case OBJECT_TSCONFIGURATION:
case OBJECT_TSDICTIONARY:
case OBJECT_TSPARSER:
@@ -745,6 +747,7 @@ ExecAlterOwnerStmt(AlterOwnerStmt *stmt)
case OBJECT_OPERATOR:
case OBJECT_OPCLASS:
case OBJECT_OPFAMILY:
+ case OBJECT_STATISTICS:
case OBJECT_TABLESPACE:
case OBJECT_TSDICTIONARY:
case OBJECT_TSCONFIGURATION:
diff --git a/src/backend/commands/analyze.c b/src/backend/commands/analyze.c
index 8a5f07c..9087532 100644
--- a/src/backend/commands/analyze.c
+++ b/src/backend/commands/analyze.c
@@ -17,6 +17,7 @@
#include <math.h>
#include "access/multixact.h"
+#include "access/sysattr.h"
#include "access/transam.h"
#include "access/tupconvert.h"
#include "access/tuptoaster.h"
@@ -27,6 +28,7 @@
#include "catalog/indexing.h"
#include "catalog/pg_collation.h"
#include "catalog/pg_inherits_fn.h"
+#include "catalog/pg_mv_statistic.h"
#include "catalog/pg_namespace.h"
#include "commands/dbcommands.h"
#include "commands/tablecmds.h"
@@ -45,10 +47,13 @@
#include "storage/procarray.h"
#include "utils/acl.h"
#include "utils/attoptcache.h"
+#include "utils/builtins.h"
#include "utils/datum.h"
+#include "utils/fmgroids.h"
#include "utils/guc.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/mvstats.h"
#include "utils/pg_rusage.h"
#include "utils/sampling.h"
#include "utils/sortsupport.h"
@@ -460,6 +465,19 @@ do_analyze_rel(Relation onerel, int options, VacuumParams *params,
* all analyzable columns. We use a lower bound of 100 rows to avoid
* possible overflow in Vitter's algorithm. (Note: that will also be the
* target in the corner case where there are no analyzable columns.)
+ *
+ * FIXME This sample sizing is mostly OK when computing stats for
+ * individual columns, but when computing multi-variate stats
+ * for multivariate stats (histograms, mcv, ...) it's rather
+ * insufficient. For stats on multiple columns / complex stats
+ * we need larger sample sizes, because we need to build more
+ * detailed stats (more MCV items / histogram buckets) to get
+ * good accuracy. Maybe it'd be appropriate to use samples
+ * proportional to the table (say, 0.5% - 1%) instead of a
+ * fixed size might be more appropriate. Also, this should be
+ * bound to the requested statistics size - e.g. number of MCV
+ * items or histogram buckets should require several sample
+ * rows per item/bucket (so the sample should be k*size).
*/
targrows = 100;
for (i = 0; i < attr_cnt; i++)
@@ -562,6 +580,9 @@ do_analyze_rel(Relation onerel, int options, VacuumParams *params,
update_attstats(RelationGetRelid(Irel[ind]), false,
thisdata->attr_cnt, thisdata->vacattrstats);
}
+
+ /* Build multivariate stats (if there are any). */
+ build_mv_stats(onerel, numrows, rows, attr_cnt, vacattrstats);
}
/*
diff --git a/src/backend/commands/dropcmds.c b/src/backend/commands/dropcmds.c
index 522027a..cd65b58 100644
--- a/src/backend/commands/dropcmds.c
+++ b/src/backend/commands/dropcmds.c
@@ -292,6 +292,10 @@ does_not_exist_skipping(ObjectType objtype, List *objname, List *objargs)
msg = gettext_noop("schema \"%s\" does not exist, skipping");
name = NameListToString(objname);
break;
+ case OBJECT_STATISTICS:
+ msg = gettext_noop("statistics \"%s\" does not exist, skipping");
+ name = NameListToString(objname);
+ break;
case OBJECT_TSPARSER:
if (!schema_does_not_exist_skipping(objname, &msg, &name))
{
diff --git a/src/backend/commands/event_trigger.c b/src/backend/commands/event_trigger.c
index 9e32f8d..09061bb 100644
--- a/src/backend/commands/event_trigger.c
+++ b/src/backend/commands/event_trigger.c
@@ -110,6 +110,7 @@ static event_trigger_support_data event_trigger_support[] = {
{"SCHEMA", true},
{"SEQUENCE", true},
{"SERVER", true},
+ {"STATISTICS", true},
{"TABLE", true},
{"TABLESPACE", false},
{"TRANSFORM", true},
@@ -1106,6 +1107,7 @@ EventTriggerSupportsObjectType(ObjectType obtype)
case OBJECT_RULE:
case OBJECT_SCHEMA:
case OBJECT_SEQUENCE:
+ case OBJECT_STATISTICS:
case OBJECT_TABCONSTRAINT:
case OBJECT_TABLE:
case OBJECT_TRANSFORM:
@@ -1167,6 +1169,7 @@ EventTriggerSupportsObjectClass(ObjectClass objclass)
case OCLASS_DEFACL:
case OCLASS_EXTENSION:
case OCLASS_POLICY:
+ case OCLASS_STATISTICS:
return true;
}
diff --git a/src/backend/commands/statscmds.c b/src/backend/commands/statscmds.c
new file mode 100644
index 0000000..f43b053
--- /dev/null
+++ b/src/backend/commands/statscmds.c
@@ -0,0 +1,277 @@
+/*-------------------------------------------------------------------------
+ *
+ * statscmds.c
+ * Commands for creating and altering multivariate statistics
+ *
+ * Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/commands/statscmds.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/relscan.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/namespace.h"
+#include "catalog/pg_mv_statistic.h"
+#include "catalog/pg_namespace.h"
+#include "commands/defrem.h"
+#include "miscadmin.h"
+#include "utils/builtins.h"
+#include "utils/inval.h"
+#include "utils/memutils.h"
+#include "utils/mvstats.h"
+#include "utils/rel.h"
+#include "utils/syscache.h"
+
+
+/* used for sorting the attnums in ExecCreateStatistics */
+static int compare_int16(const void *a, const void *b)
+{
+ return memcmp(a, b, sizeof(int16));
+}
+
+/*
+ * Implements the CREATE STATISTICS name ON table (columns) WITH (options)
+ *
+ * TODO Check that the types support sort, although maybe we can live
+ * without it (and only build MCV list / association rules).
+ *
+ * TODO This should probably check for duplicate stats (i.e. same
+ * keys, same options). Although maybe it's useful to have
+ * multiple stats on the same columns with different options
+ * (say, a detailed MCV-only stats for some queries, histogram
+ * for others, etc.)
+ */
+ObjectAddress
+CreateStatistics(CreateStatsStmt *stmt)
+{
+ int i, j;
+ ListCell *l;
+ int16 attnums[INDEX_MAX_KEYS];
+ int numcols = 0;
+ ObjectAddress address = InvalidObjectAddress;
+ char *namestr;
+ NameData staname;
+ Oid statoid;
+ Oid namespaceId;
+
+ HeapTuple htup;
+ Datum values[Natts_pg_mv_statistic];
+ bool nulls[Natts_pg_mv_statistic];
+ int2vector *stakeys;
+ Relation mvstatrel;
+ Relation rel;
+ ObjectAddress parentobject, childobject;
+
+ /* by default build nothing */
+ bool build_dependencies = false;
+
+ Assert(IsA(stmt, CreateStatsStmt));
+
+ /* resolve the pieces of the name (namespace etc.) */
+ namespaceId = QualifiedNameGetCreationNamespace(stmt->defnames, &namestr);
+ namestrcpy(&staname, namestr);
+
+ /*
+ * If if_not_exists was given and the statistics already exists, bail out.
+ */
+ if (stmt->if_not_exists &&
+ SearchSysCacheExists2(MVSTATNAMENSP,
+ PointerGetDatum(&staname),
+ ObjectIdGetDatum(namespaceId)))
+ {
+ ereport(NOTICE,
+ (errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("statistics \"%s\" already exists, skipping",
+ namestr)));
+ return InvalidObjectAddress;
+ }
+
+ rel = heap_openrv(stmt->relation, AccessExclusiveLock);
+
+ /* transform the column names to attnum values */
+
+ foreach(l, stmt->keys)
+ {
+ char *attname = strVal(lfirst(l));
+ HeapTuple atttuple;
+
+ atttuple = SearchSysCacheAttName(RelationGetRelid(rel), attname);
+
+ if (!HeapTupleIsValid(atttuple))
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_COLUMN),
+ errmsg("column \"%s\" referenced in statistics does not exist",
+ attname)));
+
+ /* more than MVHIST_MAX_DIMENSIONS columns not allowed */
+ if (numcols >= MVSTATS_MAX_DIMENSIONS)
+ ereport(ERROR,
+ (errcode(ERRCODE_TOO_MANY_COLUMNS),
+ errmsg("cannot have more than %d keys in a statistics",
+ MVSTATS_MAX_DIMENSIONS)));
+
+ attnums[numcols] = ((Form_pg_attribute) GETSTRUCT(atttuple))->attnum;
+ ReleaseSysCache(atttuple);
+ numcols++;
+ }
+
+ /*
+ * Check the lower bound (at least 2 columns), the upper bound was
+ * already checked in the loop.
+ */
+ if (numcols < 2)
+ ereport(ERROR,
+ (errcode(ERRCODE_TOO_MANY_COLUMNS),
+ errmsg("multivariate stats require 2 or more columns")));
+
+ /* look for duplicities */
+ for (i = 0; i < numcols; i++)
+ for (j = 0; j < numcols; j++)
+ if ((i != j) && (attnums[i] == attnums[j]))
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_COLUMN),
+ errmsg("duplicate column name in statistics definition")));
+
+ /* parse the statistics options */
+ foreach (l, stmt->options)
+ {
+ DefElem *opt = (DefElem*)lfirst(l);
+
+ if (strcmp(opt->defname, "dependencies") == 0)
+ build_dependencies = defGetBoolean(opt);
+ else
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("unrecognized STATISTICS option \"%s\"",
+ opt->defname)));
+ }
+
+ /* check that at least some statistics were requested */
+ if (! build_dependencies)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("no statistics type (dependencies) was requested")));
+
+ /* sort the attnums and build int2vector */
+ qsort(attnums, numcols, sizeof(int16), compare_int16);
+ stakeys = buildint2vector(attnums, numcols);
+
+ /*
+ * Okay, let's create the pg_mv_statistic entry.
+ */
+ memset(values, 0, sizeof(values));
+ memset(nulls, false, sizeof(nulls));
+
+ /* no stats collected yet, so just the keys */
+ values[Anum_pg_mv_statistic_starelid-1] = ObjectIdGetDatum(RelationGetRelid(rel));
+ values[Anum_pg_mv_statistic_staname -1] = NameGetDatum(&staname);
+ values[Anum_pg_mv_statistic_stanamespace -1] = ObjectIdGetDatum(namespaceId);
+ values[Anum_pg_mv_statistic_staowner-1] = ObjectIdGetDatum(GetUserId());
+
+ values[Anum_pg_mv_statistic_stakeys -1] = PointerGetDatum(stakeys);
+
+ values[Anum_pg_mv_statistic_deps_enabled -1] = BoolGetDatum(build_dependencies);
+
+ nulls[Anum_pg_mv_statistic_stadeps -1] = true;
+
+ /* insert the tuple into pg_mv_statistic */
+ mvstatrel = heap_open(MvStatisticRelationId, RowExclusiveLock);
+
+ htup = heap_form_tuple(mvstatrel->rd_att, values, nulls);
+
+ simple_heap_insert(mvstatrel, htup);
+
+ CatalogUpdateIndexes(mvstatrel, htup);
+
+ statoid = HeapTupleGetOid(htup);
+
+ heap_freetuple(htup);
+
+
+ /*
+ * Store a dependency too, so that statistics are dropped on DROP TABLE
+ */
+ parentobject.classId = RelationRelationId;
+ parentobject.objectId = ObjectIdGetDatum(RelationGetRelid(rel));
+ parentobject.objectSubId = 0;
+ childobject.classId = MvStatisticRelationId;
+ childobject.objectId = statoid;
+ childobject.objectSubId = 0;
+
+ recordDependencyOn(&childobject, &parentobject, DEPENDENCY_AUTO);
+
+ /*
+ * Also record dependency on the schema (to drop statistics on DROP SCHEMA)
+ */
+ parentobject.classId = NamespaceRelationId;
+ parentobject.objectId = ObjectIdGetDatum(namespaceId);
+ parentobject.objectSubId = 0;
+ childobject.classId = MvStatisticRelationId;
+ childobject.objectId = statoid;
+ childobject.objectSubId = 0;
+
+ recordDependencyOn(&childobject, &parentobject, DEPENDENCY_AUTO);
+
+
+ heap_close(mvstatrel, RowExclusiveLock);
+
+ relation_close(rel, NoLock);
+
+ /*
+ * Invalidate relcache so that others see the new statistics.
+ */
+ CacheInvalidateRelcache(rel);
+
+ ObjectAddressSet(address, MvStatisticRelationId, statoid);
+
+ return address;
+}
+
+
+/*
+ * Implements the DROP STATISTICS
+ *
+ * DROP STATISTICS stats_name ON table_name
+ *
+ * The first one requires an exact match, the second one just drops
+ * all the statistics on a table.
+ */
+void
+RemoveStatisticsById(Oid statsOid)
+{
+ Relation relation;
+ Oid relid;
+ Relation rel;
+ HeapTuple tup;
+ Form_pg_mv_statistic mvstat;
+
+ /*
+ * Delete the pg_proc tuple.
+ */
+ relation = heap_open(MvStatisticRelationId, RowExclusiveLock);
+
+ tup = SearchSysCache1(MVSTATOID, ObjectIdGetDatum(statsOid));
+ if (!HeapTupleIsValid(tup)) /* should not happen */
+ elog(ERROR, "cache lookup failed for statistics %u", statsOid);
+
+ mvstat = (Form_pg_mv_statistic) GETSTRUCT(tup);
+ relid = mvstat->starelid;
+
+ rel = heap_open(relid, AccessExclusiveLock);
+
+ simple_heap_delete(relation, &tup->t_self);
+
+ CacheInvalidateRelcache(rel);
+
+ ReleaseSysCache(tup);
+
+ heap_close(relation, RowExclusiveLock);
+ heap_close(rel, NoLock);
+}
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index df7c2fa..3b7c87f 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -4124,6 +4124,20 @@ _copyAlterPolicyStmt(const AlterPolicyStmt *from)
return newnode;
}
+static CreateStatsStmt *
+_copyCreateStatsStmt(const CreateStatsStmt *from)
+{
+ CreateStatsStmt *newnode = makeNode(CreateStatsStmt);
+
+ COPY_NODE_FIELD(defnames);
+ COPY_NODE_FIELD(relation);
+ COPY_NODE_FIELD(keys);
+ COPY_NODE_FIELD(options);
+ COPY_SCALAR_FIELD(if_not_exists);
+
+ return newnode;
+}
+
/* ****************************************************************
* pg_list.h copy functions
* ****************************************************************
@@ -4999,6 +5013,9 @@ copyObject(const void *from)
case T_CommonTableExpr:
retval = _copyCommonTableExpr(from);
break;
+ case T_CreateStatsStmt:
+ retval = _copyCreateStatsStmt(from);
+ break;
case T_FuncWithArgs:
retval = _copyFuncWithArgs(from);
break;
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index eb0fc1e..07206d7 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2153,6 +2153,21 @@ _outIndexOptInfo(StringInfo str, const IndexOptInfo *node)
}
static void
+_outMVStatisticInfo(StringInfo str, const MVStatisticInfo *node)
+{
+ WRITE_NODE_TYPE("MVSTATISTICINFO");
+
+ /* NB: this isn't a complete set of fields */
+ WRITE_OID_FIELD(mvoid);
+
+ /* enabled statistics */
+ WRITE_BOOL_FIELD(deps_enabled);
+
+ /* built/available statistics */
+ WRITE_BOOL_FIELD(deps_built);
+}
+
+static void
_outEquivalenceClass(StringInfo str, const EquivalenceClass *node)
{
/*
@@ -3636,6 +3651,9 @@ _outNode(StringInfo str, const void *obj)
case T_PlannerParamItem:
_outPlannerParamItem(str, obj);
break;
+ case T_MVStatisticInfo:
+ _outMVStatisticInfo(str, obj);
+ break;
case T_ExtensibleNode:
_outExtensibleNode(str, obj);
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index ad715bb..7fb2088 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -28,6 +28,7 @@
#include "catalog/dependency.h"
#include "catalog/heap.h"
#include "catalog/pg_am.h"
+#include "catalog/pg_mv_statistic.h"
#include "foreign/fdwapi.h"
#include "miscadmin.h"
#include "nodes/makefuncs.h"
@@ -40,7 +41,9 @@
#include "parser/parsetree.h"
#include "rewrite/rewriteManip.h"
#include "storage/bufmgr.h"
+#include "utils/builtins.h"
#include "utils/lsyscache.h"
+#include "utils/syscache.h"
#include "utils/rel.h"
#include "utils/snapmgr.h"
@@ -94,6 +97,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
Relation relation;
bool hasindex;
List *indexinfos = NIL;
+ List *stainfos = NIL;
/*
* We need not lock the relation since it was already locked, either by
@@ -387,6 +391,61 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
rel->indexlist = indexinfos;
+ if (true)
+ {
+ List *mvstatoidlist;
+ ListCell *l;
+
+ mvstatoidlist = RelationGetMVStatList(relation);
+
+ foreach(l, mvstatoidlist)
+ {
+ ArrayType *arr;
+ Datum adatum;
+ bool isnull;
+ Oid mvoid = lfirst_oid(l);
+ Form_pg_mv_statistic mvstat;
+ MVStatisticInfo *info;
+
+ HeapTuple htup = SearchSysCache1(MVSTATOID, ObjectIdGetDatum(mvoid));
+
+ mvstat = (Form_pg_mv_statistic) GETSTRUCT(htup);
+
+ /* unavailable stats are not interesting for the planner */
+ if (mvstat->deps_built)
+ {
+ info = makeNode(MVStatisticInfo);
+
+ info->mvoid = mvoid;
+ info->rel = rel;
+
+ /* enabled statistics */
+ info->deps_enabled = mvstat->deps_enabled;
+
+ /* built/available statistics */
+ info->deps_built = mvstat->deps_built;
+
+ /* stakeys */
+ adatum = SysCacheGetAttr(MVSTATOID, htup,
+ Anum_pg_mv_statistic_stakeys, &isnull);
+ Assert(!isnull);
+
+ arr = DatumGetArrayTypeP(adatum);
+
+ info->stakeys = buildint2vector((int16 *) ARR_DATA_PTR(arr),
+ ARR_DIMS(arr)[0]);
+
+ stainfos = lcons(info, stainfos);
+ }
+
+ ReleaseSysCache(htup);
+ }
+
+ list_free(mvstatoidlist);
+ }
+
+ rel->mvstatlist = stainfos;
+
/* Grab foreign-table info using the relcache, while we have it */
if (relation->rd_rel->relkind == RELKIND_FOREIGN_TABLE)
{
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index b9aeb31..eed9927 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -241,7 +241,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
ConstraintsSetStmt CopyStmt CreateAsStmt CreateCastStmt
CreateDomainStmt CreateExtensionStmt CreateGroupStmt CreateOpClassStmt
CreateOpFamilyStmt AlterOpFamilyStmt CreatePLangStmt
- CreateSchemaStmt CreateSeqStmt CreateStmt CreateTableSpaceStmt
+ CreateSchemaStmt CreateSeqStmt CreateStmt CreateStatsStmt CreateTableSpaceStmt
CreateFdwStmt CreateForeignServerStmt CreateForeignTableStmt
CreateAssertStmt CreateTransformStmt CreateTrigStmt CreateEventTrigStmt
CreateUserStmt CreateUserMappingStmt CreateRoleStmt CreatePolicyStmt
@@ -809,6 +809,7 @@ stmt :
| CreateSchemaStmt
| CreateSeqStmt
| CreateStmt
+ | CreateStatsStmt
| CreateTableSpaceStmt
| CreateTransformStmt
| CreateTrigStmt
@@ -3436,6 +3437,36 @@ OptConsTableSpace: USING INDEX TABLESPACE name { $$ = $4; }
ExistingIndex: USING INDEX index_name { $$ = $3; }
;
+/*****************************************************************************
+ *
+ * QUERY :
+ * CREATE STATISTICS stats_name ON relname (columns) WITH (options)
+ *
+ *****************************************************************************/
+
+
+CreateStatsStmt: CREATE STATISTICS any_name ON qualified_name '(' columnList ')' opt_reloptions
+ {
+ CreateStatsStmt *n = makeNode(CreateStatsStmt);
+ n->defnames = $3;
+ n->relation = $5;
+ n->keys = $7;
+ n->options = $9;
+ n->if_not_exists = false;
+ $$ = (Node *)n;
+ }
+ | CREATE STATISTICS IF_P NOT EXISTS any_name ON qualified_name '(' columnList ')' opt_reloptions
+ {
+ CreateStatsStmt *n = makeNode(CreateStatsStmt);
+ n->defnames = $6;
+ n->relation = $8;
+ n->keys = $10;
+ n->options = $12;
+ n->if_not_exists = true;
+ $$ = (Node *)n;
+ }
+ ;
+
/*****************************************************************************
*
@@ -5621,6 +5652,7 @@ drop_type: TABLE { $$ = OBJECT_TABLE; }
| TEXT_P SEARCH DICTIONARY { $$ = OBJECT_TSDICTIONARY; }
| TEXT_P SEARCH TEMPLATE { $$ = OBJECT_TSTEMPLATE; }
| TEXT_P SEARCH CONFIGURATION { $$ = OBJECT_TSCONFIGURATION; }
+ | STATISTICS { $$ = OBJECT_STATISTICS; }
;
any_name_list:
@@ -7995,6 +8027,15 @@ RenameStmt: ALTER AGGREGATE func_name aggr_args RENAME TO name
n->missing_ok = false;
$$ = (Node *)n;
}
+ | ALTER STATISTICS any_name RENAME TO name
+ {
+ RenameStmt *n = makeNode(RenameStmt);
+ n->renameType = OBJECT_STATISTICS;
+ n->object = $3;
+ n->newname = $6;
+ n->missing_ok = false;
+ $$ = (Node *)n;
+ }
;
opt_column: COLUMN { $$ = COLUMN; }
@@ -8231,6 +8272,15 @@ AlterObjectSchemaStmt:
n->missing_ok = false;
$$ = (Node *)n;
}
+ | ALTER STATISTICS any_name SET SCHEMA name
+ {
+ AlterObjectSchemaStmt *n = makeNode(AlterObjectSchemaStmt);
+ n->objectType = OBJECT_STATISTICS;
+ n->object = $3;
+ n->newschema = $6;
+ n->missing_ok = false;
+ $$ = (Node *)n;
+ }
;
/*****************************************************************************
@@ -8421,6 +8471,14 @@ AlterOwnerStmt: ALTER AGGREGATE func_name aggr_args OWNER TO RoleSpec
n->newowner = $7;
$$ = (Node *)n;
}
+ | ALTER STATISTICS name OWNER TO RoleSpec
+ {
+ AlterOwnerStmt *n = makeNode(AlterOwnerStmt);
+ n->objectType = OBJECT_STATISTICS;
+ n->object = list_make1(makeString($3));
+ n->newowner = $6;
+ $$ = (Node *)n;
+ }
;
diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c
index 045f7f0..96b58f8 100644
--- a/src/backend/tcop/utility.c
+++ b/src/backend/tcop/utility.c
@@ -1520,6 +1520,10 @@ ProcessUtilitySlow(Node *parsetree,
address = ExecSecLabelStmt((SecLabelStmt *) parsetree);
break;
+ case T_CreateStatsStmt: /* CREATE STATISTICS */
+ address = CreateStatistics((CreateStatsStmt *) parsetree);
+ break;
+
default:
elog(ERROR, "unrecognized node type: %d",
(int) nodeTag(parsetree));
@@ -1878,6 +1882,9 @@ AlterObjectTypeCommandTag(ObjectType objtype)
case OBJECT_MATVIEW:
tag = "ALTER MATERIALIZED VIEW";
break;
+ case OBJECT_STATISTICS:
+ tag = "ALTER STATISTICS";
+ break;
default:
tag = "???";
break;
@@ -2160,6 +2167,9 @@ CreateCommandTag(Node *parsetree)
case OBJECT_TRANSFORM:
tag = "DROP TRANSFORM";
break;
+ case OBJECT_STATISTICS:
+ tag = "DROP STATISTICS";
+ break;
default:
tag = "???";
}
@@ -2527,6 +2537,10 @@ CreateCommandTag(Node *parsetree)
tag = "EXECUTE";
break;
+ case T_CreateStatsStmt:
+ tag = "CREATE STATISTICS";
+ break;
+
case T_DeallocateStmt:
{
DeallocateStmt *stmt = (DeallocateStmt *) parsetree;
diff --git a/src/backend/utils/Makefile b/src/backend/utils/Makefile
index 8374533..eba0352 100644
--- a/src/backend/utils/Makefile
+++ b/src/backend/utils/Makefile
@@ -9,7 +9,7 @@ top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
OBJS = fmgrtab.o
-SUBDIRS = adt cache error fmgr hash init mb misc mmgr resowner sort time
+SUBDIRS = adt cache error fmgr hash init mb misc mmgr mvstats resowner sort time
# location of Catalog.pm
catalogdir = $(top_srcdir)/src/backend/catalog
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 130c06d..3bc4c8a 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -47,6 +47,7 @@
#include "catalog/pg_auth_members.h"
#include "catalog/pg_constraint.h"
#include "catalog/pg_database.h"
+#include "catalog/pg_mv_statistic.h"
#include "catalog/pg_namespace.h"
#include "catalog/pg_opclass.h"
#include "catalog/pg_proc.h"
@@ -3956,6 +3957,62 @@ RelationGetIndexList(Relation relation)
return result;
}
+
+List *
+RelationGetMVStatList(Relation relation)
+{
+ Relation indrel;
+ SysScanDesc indscan;
+ ScanKeyData skey;
+ HeapTuple htup;
+ List *result;
+ List *oldlist;
+ MemoryContext oldcxt;
+
+ /* Quick exit if we already computed the list. */
+ if (relation->rd_mvstatvalid != 0)
+ return list_copy(relation->rd_mvstatlist);
+
+ /*
+ * We build the list we intend to return (in the caller's context) while
+ * doing the scan. After successfully completing the scan, we copy that
+ * list into the relcache entry. This avoids cache-context memory leakage
+ * if we get some sort of error partway through.
+ */
+ result = NIL;
+
+ /* Prepare to scan pg_index for entries having indrelid = this rel. */
+ ScanKeyInit(&skey,
+ Anum_pg_mv_statistic_starelid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationGetRelid(relation)));
+
+ indrel = heap_open(MvStatisticRelationId, AccessShareLock);
+ indscan = systable_beginscan(indrel, MvStatisticRelidIndexId, true,
+ NULL, 1, &skey);
+
+ while (HeapTupleIsValid(htup = systable_getnext(indscan)))
+ /* TODO maybe include only already built statistics? */
+ result = insert_ordered_oid(result, HeapTupleGetOid(htup));
+
+ systable_endscan(indscan);
+
+ heap_close(indrel, AccessShareLock);
+
+ /* Now save a copy of the completed list in the relcache entry. */
+ oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
+ oldlist = relation->rd_mvstatlist;
+ relation->rd_mvstatlist = list_copy(result);
+
+ relation->rd_mvstatvalid = true;
+ MemoryContextSwitchTo(oldcxt);
+
+ /* Don't leak the old list, if there is one */
+ list_free(oldlist);
+
+ return result;
+}
+
/*
* insert_ordered_oid
* Insert a new Oid into a sorted list of Oids, preserving ordering
@@ -4920,6 +4977,8 @@ load_relcache_init_file(bool shared)
rel->rd_indexattr = NULL;
rel->rd_keyattr = NULL;
rel->rd_idattr = NULL;
+ rel->rd_mvstatvalid = false;
+ rel->rd_mvstatlist = NIL;
rel->rd_createSubid = InvalidSubTransactionId;
rel->rd_newRelfilenodeSubid = InvalidSubTransactionId;
rel->rd_amcache = NULL;
diff --git a/src/backend/utils/cache/syscache.c b/src/backend/utils/cache/syscache.c
index 65ffe84..3c1bc4b 100644
--- a/src/backend/utils/cache/syscache.c
+++ b/src/backend/utils/cache/syscache.c
@@ -44,6 +44,7 @@
#include "catalog/pg_foreign_server.h"
#include "catalog/pg_foreign_table.h"
#include "catalog/pg_language.h"
+#include "catalog/pg_mv_statistic.h"
#include "catalog/pg_namespace.h"
#include "catalog/pg_opclass.h"
#include "catalog/pg_operator.h"
@@ -502,6 +503,28 @@ static const struct cachedesc cacheinfo[] = {
},
4
},
+ {MvStatisticRelationId, /* MVSTATNAMENSP */
+ MvStatisticNameIndexId,
+ 2,
+ {
+ Anum_pg_mv_statistic_staname,
+ Anum_pg_mv_statistic_stanamespace,
+ 0,
+ 0
+ },
+ 4
+ },
+ {MvStatisticRelationId, /* MVSTATOID */
+ MvStatisticOidIndexId,
+ 1,
+ {
+ ObjectIdAttributeNumber,
+ 0,
+ 0,
+ 0
+ },
+ 4
+ },
{NamespaceRelationId, /* NAMESPACENAME */
NamespaceNameIndexId,
1,
diff --git a/src/backend/utils/mvstats/Makefile b/src/backend/utils/mvstats/Makefile
new file mode 100644
index 0000000..099f1ed
--- /dev/null
+++ b/src/backend/utils/mvstats/Makefile
@@ -0,0 +1,17 @@
+#-------------------------------------------------------------------------
+#
+# Makefile--
+# Makefile for utils/mvstats
+#
+# IDENTIFICATION
+# src/backend/utils/mvstats/Makefile
+#
+#-------------------------------------------------------------------------
+
+subdir = src/backend/utils/mvstats
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+
+OBJS = common.o dependencies.o
+
+include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/utils/mvstats/README.dependencies b/src/backend/utils/mvstats/README.dependencies
new file mode 100644
index 0000000..1f96fbc
--- /dev/null
+++ b/src/backend/utils/mvstats/README.dependencies
@@ -0,0 +1,222 @@
+Soft functional dependencies
+============================
+
+A type of multivariate statistics used to capture cases when one column (or
+possibly a combination of columns) determines values in another column. We may
+also say that one column implies the other one.
+
+A simple artificial example may be a table with two columns, created like this
+
+ CREATE TABLE t (a INT, b INT)
+ AS SELECT i, i/10 FROM generate_series(1,100000) s(i);
+
+Clearly, once we know the value for column 'a' the value for 'b' is trivially
+determined, as it's simply (a/10). A more practical example may be addresses,
+where (ZIP code -> city name), i.e. once we know the ZIP, we probably know the
+city it belongs to, as ZIP codes are usually assigned to one city. Larger cities
+may have multiple ZIP codes, so the dependency can't be reversed.
+
+Functional dependencies are a concept well described in relational theory,
+particularly in definition of normalization and "normal forms". Wikipedia has a
+nice definition of a functional dependency [1]:
+
+ In a given table, an attribute Y is said to have a functional dependency on
+ a set of attributes X (written X -> Y) if and only if each X value is
+ associated with precisely one Y value. For example, in an "Employee" table
+ that includes the attributes "Employee ID" and "Employee Date of Birth", the
+ functional dependency {Employee ID} -> {Employee Date of Birth} would hold.
+ It follows from the previous two sentences that each {Employee ID} is
+ associated with precisely one {Employee Date of Birth}.
+
+ [1] http://en.wikipedia.org/wiki/Database_normalization
+
+Many datasets might be normalized not to contain such dependencies, but often
+it's not practical for various reasons. In some cases it's actually a conscious
+design choice to model the dataset in denormalized way, either because of
+performance or to make querying easier.
+
+The functional dependencies are called 'soft' because the implementation is
+meant to allow small number of rows contradicting the dependency. Many actual
+data sets contain some sort of errors, either because of data entry mistakes
+(user mistyping the ZIP code) or issues in generating the data (e.g. a ZIP code
+mistakenly assigned to two cities in different states). A strict implementation
+would ignore dependencies on such noisy data, rendering the approach unusable on
+such data sets.
+
+
+Mining dependencies (ANALYZE)
+-----------------------------
+
+The current build algorithm is rather simple - for each pair (a,b) of columns,
+the data are sorted lexicographically (first by 'a', then by 'b'). Then for each
+group (rows with the same 'a' value) we decide whether the group is neutral,
+supporting or contradicting the dependency (a->b).
+
+A group is considered neutral when it's too small - e.g. when there's a single
+row in the group, there can't possibly be multiple values in 'b'. For this
+reason we ignore groups smaller than a threshold (currently 3 rows).
+
+For sufficiently large groups (3 rows or more), we count the number of distinct
+values in 'b'. When there's a single 'b' value, the group is considered to
+support the dependency (a->b), otherwise it's condidered as contradicting it.
+
+At the end, we compare the number of rows in supporting and contradicting groups,
+and if there are at least 10x as many supporting rows, we consider the
+functional dependency to be valid.
+
+
+This approach has the negative property that the algorithm is that it's a bit
+fragile with respect to the sample - there may be data sets producing quite
+different results for each ANALYZE execution (as even a single row may change
+the outcome of the final 10x test).
+
+It was proposed to make the dependencies "fuzzy" - e.g. track some coefficient
+between [0,1] determining how much the dependency holds. That would however mean
+we have to keep all the dependencies, as eliminating them based on the value of
+the coefficient (e.g. throw away dependencies <= 0.5) would result in exactly
+the same fragility issues. This would also make it more complicated to combine
+dependencies. So this does not seem like a practical approach.
+
+A better approach might be to replace the constants (min_group_size=3 and 10x)
+with values somehow related to the particular data set.
+
+
+Clause reduction (planner/optimizer)
+------------------------------------
+
+Apllying the functional dependencies is quite simple - given a list of equality
+clauses, check which clauses are redundant (i.e. implied by some other clause).
+For example given clause list
+
+ (a = 2) AND (b = 2) AND (c = 3)
+
+and dependencies (a->b) and (a->d), the list of clauses may be simplified to
+
+ (a = 1) AND (c = 3)
+
+Functional dependencies may only be applied to equality clauses, all other types
+of clauses are ignored. See clauselist_apply_dependencies() for more details.
+
+
+Compatibility of clauses
+------------------------
+
+The reduction assumes the clauses really are redundant, and the value in the
+reduced clause (b=2) is the value determined by (a=1). If that's not the case
+and the values are "incompatible" the result will be over-estimation.
+
+This may happen for example when using conditions on ZIP and city name with
+mismatching values (ZIP for a different city), etc. In such case the result
+set will be empty, but we'll estimate the selectivity using the ZIP condition.
+
+In this case the default estimation based on AVIA principle happens to work
+better, but mostly by chance.
+
+
+Dependencies vs. MCV/histogram
+------------------------------
+
+In some cases the "compatibility" of the conditions might be verified using the
+other types of multivariate stats - MCV lists and histograms.
+
+For MCV lists the verification might be very simple - peek into the list if
+there are any items matching the clause on the 'a' column (e.g. ZIP code), and
+if such item is found, check that the 'b' column matches the other clause. If it
+does not, the clauses are contradictory. We can't really say if such item was
+not found, except maybe restricting the selectivity using the MCV data (e.g.
+using min/max selectivity, or something).
+
+With histograms, it might work similarly - we can't check the values directly
+(because histograms use buckets, unlike MCV lists, storing the actual values).
+So we can only observe the buckets matching the clauses - if those buckets have
+very low frequency, it probably means the two clauses are incompatible.
+
+It's unclear what 'low frequency' is, but if one of the clauses is implied
+(automatically true because of the other clause), then
+
+ selectivity[clause(A)] = selectivity[clause(A) & clause(B)]
+
+So we might compute selectivity of the first clause - for example using regular
+statistics. And then check if the selectivity computed from the histogram is
+about the same (or significantly lower).
+
+The problem is that histograms work well only when the data ordering matches the
+natural meaning. For values that serve as labels - like city names or ZIP codes,
+or even generated IDs, histograms really don't work all that well. For example
+sorting cities by name won't match the sorting of ZIP codes, rendering the
+histogram unusable.
+
+So MCVs are probably going to work much better, because they don't really assume
+any sort of ordering. And it's probably more appropriate for the label-like data.
+
+A good question however is why even use functional dependencies in such cases
+and not simply use the MCV/histogram instead. One reason is that the functional
+dependencies allow fallback to regular stats, and often produce more accurate
+estimates - especially compared to histograms, that are quite bad in estimating
+equality clauses.
+
+
+Limitations
+-----------
+
+Let's see the main liminations of functional dependencies, especially those
+related to the current implementation.
+
+The current implementation supports only dependencies between two columns, but
+this is merely a simplification of the initial implementation. It's certainly
+useful to mine for dependencies involving multiple columns on the 'left' side,
+i.e. a condition for the dependency. That is dependencies like (a,b -> c).
+
+The implementation may/should be smart enough not to mine redundant conditions,
+e.g. (a->b) and (a,c -> b), because the latter is a trivial consequence of the
+former one (if values of 'a' determine 'b', adding another column won't change
+that relationship). The ANALYZE should first analyze 1:1 dependencies, then 2:1
+dependencies (and skip the already identified ones), etc.
+
+For example the dependency
+
+ (city name -> zip code)
+
+is much stronger, i.e. whenever it hold, then
+
+ (city name, state name -> zip code)
+
+holds too. But in case there are cities with the same name in different states,
+then only the latter dependency will be valid.
+
+Of course, there probably are cities with the same name within a single state,
+but hopefully this is relatively rare occurence (and thus we'll still detect
+the 'soft' dependency).
+
+Handling multiple columns on the right side of the dependency, is not necessary,
+as those dependencies may be simply decomposed into a set of dependencies with
+the same meaning, one for each column on the right side. For example
+
+ (a -> b,c)
+
+is exactly the same as
+
+ (a -> b) & (a -> c)
+
+Of course, storing the first form may be more efficient thant storing multiple
+'simple' dependencies separately.
+
+
+TODO Support dependencies with multiple columns on left/right.
+
+TODO Investigate using histogram and MCV list to verify the dependencies.
+
+TODO Investigate statistical testing of the distribution (to decide whether it
+ makes sense to build the histogram/MCV list).
+
+TODO Using a min/max of selectivities would probably make more sense for the
+ associated columns.
+
+TODO Consider eliminating the implied columns from the histogram and MCV lists
+ (but maybe that's not a good idea, because that'd make it impossible to use
+ these stats for non-equality clauses and also it wouldn't be possible to
+ use the stats for verification of the dependencies).
+
+TODO The reduction probably might be extended to also handle IS NULL clauses,
+ assuming we fix the ANALYZE to properly handle NULL values. We however
+ won't be able to reduce IS NOT NULL (unless I'm missing something).
diff --git a/src/backend/utils/mvstats/common.c b/src/backend/utils/mvstats/common.c
new file mode 100644
index 0000000..a755c49
--- /dev/null
+++ b/src/backend/utils/mvstats/common.c
@@ -0,0 +1,356 @@
+/*-------------------------------------------------------------------------
+ *
+ * common.c
+ * POSTGRES multivariate statistics
+ *
+ *
+ * Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/utils/mvstats/common.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "common.h"
+
+static VacAttrStats ** lookup_var_attr_stats(int2vector *attrs,
+ int natts, VacAttrStats **vacattrstats);
+
+static List* list_mv_stats(Oid relid);
+
+
+/*
+ * Compute requested multivariate stats, using the rows sampled for the
+ * plain (single-column) stats.
+ *
+ * This fetches a list of stats from pg_mv_statistic, computes the stats
+ * and serializes them back into the catalog (as bytea values).
+ */
+void
+build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
+ int natts, VacAttrStats **vacattrstats)
+{
+ ListCell *lc;
+ List *mvstats;
+
+ TupleDesc tupdesc = RelationGetDescr(onerel);
+
+ /*
+ * Fetch defined MV groups from pg_mv_statistic, and then compute
+ * the MV statistics (histograms for now).
+ */
+ mvstats = list_mv_stats(RelationGetRelid(onerel));
+
+ foreach (lc, mvstats)
+ {
+ int j;
+ MVStatisticInfo *stat = (MVStatisticInfo *)lfirst(lc);
+ MVDependencies deps = NULL;
+
+ VacAttrStats **stats = NULL;
+ int numatts = 0;
+
+ /* int2 vector of attnums the stats should be computed on */
+ int2vector * attrs = stat->stakeys;
+
+ /* see how many of the columns are not dropped */
+ for (j = 0; j < attrs->dim1; j++)
+ if (! tupdesc->attrs[attrs->values[j]-1]->attisdropped)
+ numatts += 1;
+
+ /* if there are dropped attributes, build a filtered int2vector */
+ if (numatts != attrs->dim1)
+ {
+ int16 *tmp = palloc0(numatts * sizeof(int16));
+ int attnum = 0;
+
+ for (j = 0; j < attrs->dim1; j++)
+ if (! tupdesc->attrs[attrs->values[j]-1]->attisdropped)
+ tmp[attnum++] = attrs->values[j];
+
+ pfree(attrs);
+ attrs = buildint2vector(tmp, numatts);
+ }
+
+ /* filter only the interesting vacattrstats records */
+ stats = lookup_var_attr_stats(attrs, natts, vacattrstats);
+
+ /* check allowed number of dimensions */
+ Assert((attrs->dim1 >= 2) && (attrs->dim1 <= MVSTATS_MAX_DIMENSIONS));
+
+ /*
+ * Analyze functional dependencies of columns.
+ */
+ deps = build_mv_dependencies(numrows, rows, attrs, stats);
+
+ /* store the histogram / MCV list in the catalog */
+ update_mv_stats(stat->mvoid, deps, attrs);
+ }
+}
+
+/*
+ * Lookup the VacAttrStats info for the selected columns, with indexes
+ * matching the attrs vector (to make it easy to work with when
+ * computing multivariate stats).
+ */
+static VacAttrStats **
+lookup_var_attr_stats(int2vector *attrs, int natts, VacAttrStats **vacattrstats)
+{
+ int i, j;
+ int numattrs = attrs->dim1;
+ VacAttrStats **stats = (VacAttrStats**)palloc0(numattrs * sizeof(VacAttrStats*));
+
+ /* lookup VacAttrStats info for the requested columns (same attnum) */
+ for (i = 0; i < numattrs; i++)
+ {
+ stats[i] = NULL;
+ for (j = 0; j < natts; j++)
+ {
+ if (attrs->values[i] == vacattrstats[j]->tupattnum)
+ {
+ stats[i] = vacattrstats[j];
+ break;
+ }
+ }
+
+ /*
+ * Check that we found the info, that the attnum matches and
+ * that there's the requested 'lt' operator and that the type
+ * is 'passed-by-value'.
+ */
+ Assert(stats[i] != NULL);
+ Assert(stats[i]->tupattnum == attrs->values[i]);
+
+ /* FIXME This is rather ugly way to check for 'ltopr' (which
+ * is defined for 'scalar' attributes).
+ */
+ Assert(((StdAnalyzeData *)stats[i]->extra_data)->ltopr != InvalidOid);
+ }
+
+ return stats;
+}
+
+/*
+ * Fetch list of MV stats defined on a table, without the actual data
+ * for histograms, MCV lists etc.
+ */
+static List*
+list_mv_stats(Oid relid)
+{
+ Relation indrel;
+ SysScanDesc indscan;
+ ScanKeyData skey;
+ HeapTuple htup;
+ List *result = NIL;
+
+ /* Prepare to scan pg_mv_statistic for entries having indrelid = this rel. */
+ ScanKeyInit(&skey,
+ Anum_pg_mv_statistic_starelid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(relid));
+
+ indrel = heap_open(MvStatisticRelationId, AccessShareLock);
+ indscan = systable_beginscan(indrel, MvStatisticRelidIndexId, true,
+ NULL, 1, &skey);
+
+ while (HeapTupleIsValid(htup = systable_getnext(indscan)))
+ {
+ MVStatisticInfo *info = makeNode(MVStatisticInfo);
+ Form_pg_mv_statistic stats = (Form_pg_mv_statistic) GETSTRUCT(htup);
+
+ info->mvoid = HeapTupleGetOid(htup);
+ info->stakeys = buildint2vector(stats->stakeys.values, stats->stakeys.dim1);
+ info->deps_built = stats->deps_built;
+
+ result = lappend(result, info);
+ }
+
+ systable_endscan(indscan);
+
+ heap_close(indrel, AccessShareLock);
+
+ /* TODO maybe save the list into relcache, as in RelationGetIndexList
+ * (which was used as an inspiration of this one)?. */
+
+ return result;
+}
+
+void
+update_mv_stats(Oid mvoid, MVDependencies dependencies, int2vector *attrs)
+{
+ HeapTuple stup,
+ oldtup;
+ Datum values[Natts_pg_mv_statistic];
+ bool nulls[Natts_pg_mv_statistic];
+ bool replaces[Natts_pg_mv_statistic];
+
+ Relation sd = heap_open(MvStatisticRelationId, RowExclusiveLock);
+
+ memset(nulls, 1, Natts_pg_mv_statistic * sizeof(bool));
+ memset(replaces, 0, Natts_pg_mv_statistic * sizeof(bool));
+ memset(values, 0, Natts_pg_mv_statistic * sizeof(Datum));
+
+ /*
+ * Construct a new pg_mv_statistic tuple - replace only the histogram
+ * and MCV list, depending whether it actually was computed.
+ */
+ if (dependencies != NULL)
+ {
+ nulls[Anum_pg_mv_statistic_stadeps -1] = false;
+ values[Anum_pg_mv_statistic_stadeps - 1]
+ = PointerGetDatum(serialize_mv_dependencies(dependencies));
+ }
+
+ /* always replace the value (either by bytea or NULL) */
+ replaces[Anum_pg_mv_statistic_stadeps -1] = true;
+
+ /* always change the availability flags */
+ nulls[Anum_pg_mv_statistic_deps_built -1] = false;
+ nulls[Anum_pg_mv_statistic_stakeys-1] = false;
+
+ /* use the new attnums, in case we removed some dropped ones */
+ replaces[Anum_pg_mv_statistic_deps_built-1] = true;
+ replaces[Anum_pg_mv_statistic_stakeys -1] = true;
+
+ values[Anum_pg_mv_statistic_deps_built-1] = BoolGetDatum(dependencies != NULL);
+ values[Anum_pg_mv_statistic_stakeys -1] = PointerGetDatum(attrs);
+
+ /* Is there already a pg_mv_statistic tuple for this attribute? */
+ oldtup = SearchSysCache1(MVSTATOID,
+ ObjectIdGetDatum(mvoid));
+
+ if (HeapTupleIsValid(oldtup))
+ {
+ /* Yes, replace it */
+ stup = heap_modify_tuple(oldtup,
+ RelationGetDescr(sd),
+ values,
+ nulls,
+ replaces);
+ ReleaseSysCache(oldtup);
+ simple_heap_update(sd, &stup->t_self, stup);
+ }
+ else
+ elog(ERROR, "invalid pg_mv_statistic record (oid=%d)", mvoid);
+
+ /* update indexes too */
+ CatalogUpdateIndexes(sd, stup);
+
+ heap_freetuple(stup);
+
+ heap_close(sd, RowExclusiveLock);
+}
+
+/* multi-variate stats comparator */
+
+/*
+ * qsort_arg comparator for sorting Datums (MV stats)
+ *
+ * This does not maintain the tupnoLink array.
+ */
+int
+compare_scalars_simple(const void *a, const void *b, void *arg)
+{
+ Datum da = *(Datum*)a;
+ Datum db = *(Datum*)b;
+ SortSupport ssup= (SortSupport) arg;
+
+ return ApplySortComparator(da, false, db, false, ssup);
+}
+
+/*
+ * qsort_arg comparator for sorting data when partitioning a MV bucket
+ */
+int
+compare_scalars_partition(const void *a, const void *b, void *arg)
+{
+ Datum da = ((ScalarItem*)a)->value;
+ Datum db = ((ScalarItem*)b)->value;
+ SortSupport ssup= (SortSupport) arg;
+
+ return ApplySortComparator(da, false, db, false, ssup);
+}
+
+/* initialize multi-dimensional sort */
+MultiSortSupport
+multi_sort_init(int ndims)
+{
+ MultiSortSupport mss;
+
+ Assert(ndims >= 2);
+
+ mss = (MultiSortSupport)palloc0(offsetof(MultiSortSupportData, ssup)
+ + sizeof(SortSupportData)*ndims);
+
+ mss->ndims = ndims;
+
+ return mss;
+}
+
+/*
+ * add sort into for dimension 'dim' (index into vacattrstats) to mss,
+ * at the position 'sortattr'
+ */
+void
+multi_sort_add_dimension(MultiSortSupport mss, int sortdim,
+ int dim, VacAttrStats **vacattrstats)
+{
+ /* first, lookup StdAnalyzeData for the dimension (attribute) */
+ SortSupportData ssup;
+ StdAnalyzeData *tmp = (StdAnalyzeData *)vacattrstats[dim]->extra_data;
+
+ Assert(mss != NULL);
+ Assert(sortdim < mss->ndims);
+
+ /* initialize sort support, etc. */
+ memset(&ssup, 0, sizeof(ssup));
+ ssup.ssup_cxt = CurrentMemoryContext;
+
+ /* We always use the default collation for statistics */
+ ssup.ssup_collation = DEFAULT_COLLATION_OID;
+ ssup.ssup_nulls_first = false;
+
+ PrepareSortSupportFromOrderingOp(tmp->ltopr, &ssup);
+
+ mss->ssup[sortdim] = ssup;
+}
+
+/* compare all the dimensions in the selected order */
+int
+multi_sort_compare(const void *a, const void *b, void *arg)
+{
+ int i;
+ SortItem *ia = (SortItem*)a;
+ SortItem *ib = (SortItem*)b;
+
+ MultiSortSupport mss = (MultiSortSupport)arg;
+
+ for (i = 0; i < mss->ndims; i++)
+ {
+ int compare;
+
+ compare = ApplySortComparator(ia->values[i], ia->isnull[i],
+ ib->values[i], ib->isnull[i],
+ &mss->ssup[i]);
+
+ if (compare != 0)
+ return compare;
+
+ }
+
+ /* equal by default */
+ return 0;
+}
+
+/* compare selected dimension */
+int
+multi_sort_compare_dim(int dim, const SortItem *a, const SortItem *b,
+ MultiSortSupport mss)
+{
+ return ApplySortComparator(a->values[dim], a->isnull[dim],
+ b->values[dim], b->isnull[dim],
+ &mss->ssup[dim]);
+}
diff --git a/src/backend/utils/mvstats/common.h b/src/backend/utils/mvstats/common.h
new file mode 100644
index 0000000..d96422d
--- /dev/null
+++ b/src/backend/utils/mvstats/common.h
@@ -0,0 +1,75 @@
+/*-------------------------------------------------------------------------
+ *
+ * common.h
+ * POSTGRES multivariate statistics
+ *
+ *
+ * Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/utils/mvstats/common.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/sysattr.h"
+#include "access/tuptoaster.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_collation.h"
+#include "catalog/pg_mv_statistic.h"
+#include "foreign/fdwapi.h"
+#include "postmaster/autovacuum.h"
+#include "storage/lmgr.h"
+#include "utils/builtins.h"
+#include "utils/datum.h"
+#include "utils/fmgroids.h"
+#include "utils/mvstats.h"
+#include "utils/sortsupport.h"
+#include "utils/syscache.h"
+
+
+/* FIXME private structure copied from analyze.c */
+
+typedef struct
+{
+ Oid eqopr; /* '=' operator for datatype, if any */
+ Oid eqfunc; /* and associated function */
+ Oid ltopr; /* '<' operator for datatype, if any */
+} StdAnalyzeData;
+
+typedef struct
+{
+ Datum value; /* a data value */
+ int tupno; /* position index for tuple it came from */
+} ScalarItem;
+
+/* multi-sort */
+typedef struct MultiSortSupportData {
+ int ndims; /* number of dimensions supported by the */
+ SortSupportData ssup[1]; /* sort support data for each dimension */
+} MultiSortSupportData;
+
+typedef MultiSortSupportData* MultiSortSupport;
+
+typedef struct SortItem {
+ Datum *values;
+ bool *isnull;
+} SortItem;
+
+MultiSortSupport multi_sort_init(int ndims);
+
+void multi_sort_add_dimension(MultiSortSupport mss, int sortdim,
+ int dim, VacAttrStats **vacattrstats);
+
+int multi_sort_compare(const void *a, const void *b, void *arg);
+
+int multi_sort_compare_dim(int dim, const SortItem *a,
+ const SortItem *b, MultiSortSupport mss);
+
+/* comparators, used when constructing multivariate stats */
+int compare_scalars_simple(const void *a, const void *b, void *arg);
+int compare_scalars_partition(const void *a, const void *b, void *arg);
diff --git a/src/backend/utils/mvstats/dependencies.c b/src/backend/utils/mvstats/dependencies.c
new file mode 100644
index 0000000..2a064a0
--- /dev/null
+++ b/src/backend/utils/mvstats/dependencies.c
@@ -0,0 +1,437 @@
+/*-------------------------------------------------------------------------
+ *
+ * dependencies.c
+ * POSTGRES multivariate functional dependencies
+ *
+ *
+ * Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/utils/mvstats/dependencies.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "common.h"
+#include "utils/lsyscache.h"
+
+/*
+ * Detect functional dependencies between columns.
+ *
+ * TODO This builds a complete set of dependencies, i.e. including transitive
+ * dependencies - if we identify [A => B] and [B => C], we're likely to
+ * identify [A => C] too. It might be better to keep only the minimal set
+ * of dependencies, i.e. prune all the dependencies that we can recreate
+ * by transivitity.
+ *
+ * There are two conceptual ways to do that:
+ *
+ * (a) generate all the rules, and then prune the rules that may be
+ * recteated by combining other dependencies, or
+ *
+ * (b) performing the 'is combination of other dependencies' check before
+ * actually doing the work
+ *
+ * The second option has the advantage that we don't really need to perform
+ * the sort/count. It's not sufficient alone, though, because we may
+ * discover the dependencies in the wrong order. For example we may find
+ *
+ * (a -> b), (a -> c) and then (b -> c)
+ *
+ * None of those dependencies is a combination of the already known ones,
+ * yet (a -> C) is a combination of (a -> b) and (b -> c).
+ *
+ *
+ * FIXME Currently we simply replace NULL values with 0 and then handle is as
+ * a regular value, but that groups NULL and actual 0 values. That's
+ * clearly incorrect - we need to handle NULL values as a separate value.
+ */
+MVDependencies
+build_mv_dependencies(int numrows, HeapTuple *rows, int2vector *attrs,
+ VacAttrStats **stats)
+{
+ int i;
+ int numattrs = attrs->dim1;
+
+ /* result */
+ int ndeps = 0;
+ MVDependencies dependencies = NULL;
+ MultiSortSupport mss = multi_sort_init(2); /* 2 dimensions for now */
+
+ /* TODO Maybe this should be somehow related to the number of
+ * distinct values in the two columns we're currently analyzing.
+ * Assuming the distribution is uniform, we can estimate the
+ * average group size and use it as a threshold. Or something
+ * like that. Seems better than a static approach.
+ */
+ int min_group_size = 3;
+
+ /* dimension indexes we'll check for associations [a => b] */
+ int dima, dimb;
+
+ /*
+ * We'll reuse the same array for all the 2-column combinations.
+ *
+ * It's possible to sort the sample rows directly, but this seemed
+ * somehow simples / less error prone. Another option would be to
+ * allocate the arrays for each SortItem separately, but that'd be
+ * significant overhead (not just CPU, but especially memory bloat).
+ */
+ SortItem * items = (SortItem*)palloc0(numrows * sizeof(SortItem));
+
+ Datum *values = (Datum*)palloc0(sizeof(Datum) * numrows * 2);
+ bool *isnull = (bool*)palloc0(sizeof(bool) * numrows * 2);
+
+ for (i = 0; i < numrows; i++)
+ {
+ items[i].values = &values[i * 2];
+ items[i].isnull = &isnull[i * 2];
+ }
+
+ Assert(numattrs >= 2);
+
+ /*
+ * Evaluate all possible combinations of [A => B], using a simple algorithm:
+ *
+ * (a) sort the data by [A,B]
+ * (b) split the data into groups by A (new group whenever a value changes)
+ * (c) count different values in the B column (again, value changes)
+ *
+ * TODO It should be rather simple to merge [A => B] and [A => C] into
+ * [A => B,C]. Just keep A constant, collect all the "implied" columns
+ * and you're done.
+ */
+ for (dima = 0; dima < numattrs; dima++)
+ {
+ /* prepare the sort function for the first dimension */
+ multi_sort_add_dimension(mss, 0, dima, stats);
+
+ for (dimb = 0; dimb < numattrs; dimb++)
+ {
+ SortItem current;
+
+ /* number of groups supporting / contradicting the dependency */
+ int n_supporting = 0;
+ int n_contradicting = 0;
+
+ /* counters valid within a group */
+ int group_size = 0;
+ int n_violations = 0;
+
+ int n_supporting_rows = 0;
+ int n_contradicting_rows = 0;
+
+ /* make sure the columns are different (A => A) */
+ if (dima == dimb)
+ continue;
+
+ /* prepare the sort function for the second dimension */
+ multi_sort_add_dimension(mss, 1, dimb, stats);
+
+ /* reset the values and isnull flags */
+ memset(values, 0, sizeof(Datum) * numrows * 2);
+ memset(isnull, 0, sizeof(bool) * numrows * 2);
+
+ /* accumulate all the data for both columns into an array and sort it */
+ for (i = 0; i < numrows; i++)
+ {
+ items[i].values[0]
+ = heap_getattr(rows[i], attrs->values[dima],
+ stats[dima]->tupDesc, &items[i].isnull[0]);
+
+ items[i].values[1]
+ = heap_getattr(rows[i], attrs->values[dimb],
+ stats[dimb]->tupDesc, &items[i].isnull[1]);
+ }
+
+ qsort_arg((void *) items, numrows, sizeof(SortItem),
+ multi_sort_compare, mss);
+
+ /*
+ * Walk through the array, split it into rows according to
+ * the A value, and count distinct values in the other one.
+ * If there's a single B value for the whole group, we count
+ * it as supporting the association, otherwise we count it
+ * as contradicting.
+ *
+ * Furthermore we require a group to have at least a certain
+ * number of rows to be considered useful for supporting the
+ * dependency. But when it's contradicting, use it always useful.
+ */
+
+ /* start with values from the first row */
+ current = items[0];
+ group_size = 1;
+
+ for (i = 1; i < numrows; i++)
+ {
+ /* end of the group */
+ if (multi_sort_compare_dim(0, &items[i], ¤t, mss) != 0)
+ {
+ /*
+ * If there are no contradicting rows, count it as
+ * supporting (otherwise contradicting), but only if
+ * the group is large enough.
+ *
+ * The requirement of a minimum group size makes it
+ * impossible to identify [unique,unique] cases, but
+ * that's probably a different case. This is more
+ * about [zip => city] associations etc.
+ *
+ * If there are violations, count the group/rows as
+ * a violation.
+ *
+ * It may ne neither, if the group is too small (does
+ * not contain at least min_group_size rows).
+ */
+ if ((n_violations == 0) && (group_size >= min_group_size))
+ {
+ n_supporting += 1;
+ n_supporting_rows += group_size;
+ }
+ else if (n_violations > 0)
+ {
+ n_contradicting += 1;
+ n_contradicting_rows += group_size;
+ }
+
+ /* current values start a new group */
+ n_violations = 0;
+ group_size = 0;
+ }
+ /* mismatch of a B value is contradicting */
+ else if (multi_sort_compare_dim(1, &items[i], ¤t, mss) != 0)
+ {
+ n_violations += 1;
+ }
+
+ current = items[i];
+ group_size += 1;
+ }
+
+ /* handle the last group (just like above) */
+ if ((n_violations == 0) && (group_size >= min_group_size))
+ {
+ n_supporting += 1;
+ n_supporting_rows += group_size;
+ }
+ else if (n_violations)
+ {
+ n_contradicting += 1;
+ n_contradicting_rows += group_size;
+ }
+
+ /*
+ * See if the number of rows supporting the association is at least
+ * 10x the number of rows violating the hypothetical dependency.
+ *
+ * TODO This is rather arbitrary limit - I guess it's possible to do
+ * some math to come up with a better rule (e.g. testing a hypothesis
+ * 'this is due to randomness'). We can create a contingency table
+ * from the values and use it for testing. Possibly only when
+ * there are no contradicting rows?
+ *
+ * TODO Also, if (a => b) and (b => a) at the same time, it pretty much
+ * means there's a 1:1 relation (or one is a 'label'), making the
+ * conditions rather redundant. Although it's possible that the
+ * query uses incompatible combination of values.
+ */
+ if (n_supporting_rows > (n_contradicting_rows * 10))
+ {
+ if (dependencies == NULL)
+ {
+ dependencies = (MVDependencies)palloc0(sizeof(MVDependenciesData));
+ dependencies->magic = MVSTAT_DEPS_MAGIC;
+ }
+ else
+ dependencies = repalloc(dependencies, offsetof(MVDependenciesData, deps)
+ + sizeof(MVDependency) * (dependencies->ndeps + 1));
+
+ /* update the */
+ dependencies->deps[ndeps] = (MVDependency)palloc0(sizeof(MVDependencyData));
+ dependencies->deps[ndeps]->a = attrs->values[dima];
+ dependencies->deps[ndeps]->b = attrs->values[dimb];
+
+ dependencies->ndeps = (++ndeps);
+ }
+ }
+ }
+
+ pfree(items);
+ pfree(values);
+ pfree(isnull);
+ pfree(stats);
+ pfree(mss);
+
+ return dependencies;
+}
+
+/*
+ * Store the dependencies into a bytea, so that it can be stored in the
+ * pg_mv_statistic catalog.
+ *
+ * Currently this only supports simple two-column rules, and stores them
+ * as a sequence of attnum pairs. In the future, this needs to be made
+ * more complex to support multiple columns on both sides of the
+ * implication (using AND on left, OR on right).
+ */
+bytea *
+serialize_mv_dependencies(MVDependencies dependencies)
+{
+ int i;
+
+ /* we need to store ndeps, and each needs 2 * int16 */
+ Size len = VARHDRSZ + offsetof(MVDependenciesData, deps)
+ + dependencies->ndeps * (sizeof(int16) * 2);
+
+ bytea * output = (bytea*)palloc0(len);
+
+ char * tmp = VARDATA(output);
+
+ SET_VARSIZE(output, len);
+
+ /* first, store the number of dimensions / items */
+ memcpy(tmp, dependencies, offsetof(MVDependenciesData, deps));
+ tmp += offsetof(MVDependenciesData, deps);
+
+ /* walk through the dependencies and copy both columns into the bytea */
+ for (i = 0; i < dependencies->ndeps; i++)
+ {
+ memcpy(tmp, &(dependencies->deps[i]->a), sizeof(int16));
+ tmp += sizeof(int16);
+
+ memcpy(tmp, &(dependencies->deps[i]->b), sizeof(int16));
+ tmp += sizeof(int16);
+ }
+
+ return output;
+}
+
+/*
+ * Reads serialized dependencies into MVDependencies structure.
+ */
+MVDependencies
+deserialize_mv_dependencies(bytea * data)
+{
+ int i;
+ Size expected_size;
+ MVDependencies dependencies;
+ char *tmp;
+
+ if (data == NULL)
+ return NULL;
+
+ if (VARSIZE_ANY_EXHDR(data) < offsetof(MVDependenciesData,deps))
+ elog(ERROR, "invalid MVDependencies size %ld (expected at least %ld)",
+ VARSIZE_ANY_EXHDR(data), offsetof(MVDependenciesData,deps));
+
+ /* read the MVDependencies header */
+ dependencies = (MVDependencies)palloc0(sizeof(MVDependenciesData));
+
+ /* initialize pointer to the data part (skip the varlena header) */
+ tmp = VARDATA(data);
+
+ /* get the header and perform basic sanity checks */
+ memcpy(dependencies, tmp, offsetof(MVDependenciesData, deps));
+ tmp += offsetof(MVDependenciesData, deps);
+
+ if (dependencies->magic != MVSTAT_DEPS_MAGIC)
+ {
+ pfree(dependencies);
+ elog(WARNING, "not a MV Dependencies (magic number mismatch)");
+ return NULL;
+ }
+
+ Assert(dependencies->ndeps > 0);
+
+ /* what bytea size do we expect for those parameters */
+ expected_size = offsetof(MVDependenciesData,deps) +
+ dependencies->ndeps * sizeof(int16) * 2;
+
+ if (VARSIZE_ANY_EXHDR(data) != expected_size)
+ elog(ERROR, "invalid dependencies size %ld (expected %ld)",
+ VARSIZE_ANY_EXHDR(data), expected_size);
+
+ /* allocate space for the MCV items */
+ dependencies = repalloc(dependencies, offsetof(MVDependenciesData,deps)
+ + (dependencies->ndeps * sizeof(MVDependency)));
+
+ for (i = 0; i < dependencies->ndeps; i++)
+ {
+ dependencies->deps[i] = (MVDependency)palloc0(sizeof(MVDependencyData));
+
+ memcpy(&(dependencies->deps[i]->a), tmp, sizeof(int16));
+ tmp += sizeof(int16);
+
+ memcpy(&(dependencies->deps[i]->b), tmp, sizeof(int16));
+ tmp += sizeof(int16);
+ }
+
+ return dependencies;
+}
+
+/* print some basic info about dependencies (number of dependencies) */
+Datum
+pg_mv_stats_dependencies_info(PG_FUNCTION_ARGS)
+{
+ bytea *data = PG_GETARG_BYTEA_P(0);
+ char *result;
+
+ MVDependencies dependencies = deserialize_mv_dependencies(data);
+
+ if (dependencies == NULL)
+ PG_RETURN_NULL();
+
+ result = palloc0(128);
+ snprintf(result, 128, "dependencies=%d", dependencies->ndeps);
+
+ /* FIXME free the deserialized data (pfree is not enough) */
+
+ PG_RETURN_TEXT_P(cstring_to_text(result));
+}
+
+/* print the dependencies
+ *
+ * TODO Would be nice if this knew the actual column names (instead of
+ * the attnums).
+ *
+ * FIXME This is really ugly and does not really check the lengths and
+ * strcpy/snprintf return values properly. Needs to be fixed.
+ */
+Datum
+pg_mv_stats_dependencies_show(PG_FUNCTION_ARGS)
+{
+ int i = 0;
+ bytea *data = PG_GETARG_BYTEA_P(0);
+ char *result = NULL;
+ int len = 0;
+
+ MVDependencies dependencies = deserialize_mv_dependencies(data);
+
+ if (dependencies == NULL)
+ PG_RETURN_NULL();
+
+ for (i = 0; i < dependencies->ndeps; i++)
+ {
+ MVDependency dependency = dependencies->deps[i];
+ char buffer[128];
+
+ int tmp = snprintf(buffer, 128, "%s%d => %d",
+ ((i == 0) ? "" : ", "), dependency->a, dependency->b);
+
+ if (tmp < 127)
+ {
+ if (result == NULL)
+ result = palloc0(len + tmp + 1);
+ else
+ result = repalloc(result, len + tmp + 1);
+
+ strcpy(result + len, buffer);
+ len += tmp;
+ }
+ }
+
+ PG_RETURN_TEXT_P(cstring_to_text(result));
+}
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index fd8dc91..8ce9c0e 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -2104,6 +2104,50 @@ describeOneTableDetails(const char *schemaname,
PQclear(result);
}
+ /* print any multivariate statistics */
+ if (pset.sversion >= 90600)
+ {
+ printfPQExpBuffer(&buf,
+ "SELECT oid, stanamespace::regnamespace AS nsp, staname, stakeys,\n"
+ " deps_enabled,\n"
+ " deps_built,\n"
+ " (SELECT string_agg(attname::text,', ')\n"
+ " FROM ((SELECT unnest(stakeys) AS attnum) s\n"
+ " JOIN pg_attribute a ON (starelid = a.attrelid and a.attnum = s.attnum))) AS attnums\n"
+ "FROM pg_mv_statistic stat WHERE starelid = '%s' ORDER BY 1;",
+ oid);
+
+ result = PSQLexec(buf.data);
+ if (!result)
+ goto error_return;
+ else
+ tuples = PQntuples(result);
+
+ if (tuples > 0)
+ {
+ printTableAddFooter(&cont, _("Statistics:"));
+ for (i = 0; i < tuples; i++)
+ {
+ printfPQExpBuffer(&buf, " ");
+
+ /* statistics name (qualified with namespace) */
+ appendPQExpBuffer(&buf, "\"%s.%s\" ",
+ PQgetvalue(result, i, 1),
+ PQgetvalue(result, i, 2));
+
+ /* options */
+ if (!strcmp(PQgetvalue(result, i, 4), "t"))
+ appendPQExpBuffer(&buf, "(dependencies)");
+
+ appendPQExpBuffer(&buf, " ON (%s)",
+ PQgetvalue(result, i, 6));
+
+ printTableAddFooter(&cont, buf.data);
+ }
+ }
+ PQclear(result);
+ }
+
/* print rules */
if (tableinfo.hasrules && tableinfo.relkind != 'm')
{
diff --git a/src/include/catalog/dependency.h b/src/include/catalog/dependency.h
index 049bf9f..12211fe 100644
--- a/src/include/catalog/dependency.h
+++ b/src/include/catalog/dependency.h
@@ -153,10 +153,11 @@ typedef enum ObjectClass
OCLASS_EXTENSION, /* pg_extension */
OCLASS_EVENT_TRIGGER, /* pg_event_trigger */
OCLASS_POLICY, /* pg_policy */
- OCLASS_TRANSFORM /* pg_transform */
+ OCLASS_TRANSFORM, /* pg_transform */
+ OCLASS_STATISTICS /* pg_mv_statistics */
} ObjectClass;
-#define LAST_OCLASS OCLASS_TRANSFORM
+#define LAST_OCLASS OCLASS_STATISTICS
/* in dependency.c */
diff --git a/src/include/catalog/heap.h b/src/include/catalog/heap.h
index b80d8d8..5ae42f7 100644
--- a/src/include/catalog/heap.h
+++ b/src/include/catalog/heap.h
@@ -119,6 +119,7 @@ extern void RemoveAttrDefault(Oid relid, AttrNumber attnum,
DropBehavior behavior, bool complain, bool internal);
extern void RemoveAttrDefaultById(Oid attrdefId);
extern void RemoveStatistics(Oid relid, AttrNumber attnum);
+extern void RemoveMVStatistics(Oid relid, AttrNumber attnum);
extern Form_pg_attribute SystemAttributeDefinition(AttrNumber attno,
bool relhasoids);
diff --git a/src/include/catalog/indexing.h b/src/include/catalog/indexing.h
index ab2c1a8..a768bb5 100644
--- a/src/include/catalog/indexing.h
+++ b/src/include/catalog/indexing.h
@@ -173,6 +173,13 @@ DECLARE_UNIQUE_INDEX(pg_largeobject_loid_pn_index, 2683, on pg_largeobject using
DECLARE_UNIQUE_INDEX(pg_largeobject_metadata_oid_index, 2996, on pg_largeobject_metadata using btree(oid oid_ops));
#define LargeObjectMetadataOidIndexId 2996
+DECLARE_UNIQUE_INDEX(pg_mv_statistic_oid_index, 3380, on pg_mv_statistic using btree(oid oid_ops));
+#define MvStatisticOidIndexId 3380
+DECLARE_UNIQUE_INDEX(pg_mv_statistic_name_index, 3997, on pg_mv_statistic using btree(staname name_ops, stanamespace oid_ops));
+#define MvStatisticNameIndexId 3997
+DECLARE_INDEX(pg_mv_statistic_relid_index, 3379, on pg_mv_statistic using btree(starelid oid_ops));
+#define MvStatisticRelidIndexId 3379
+
DECLARE_UNIQUE_INDEX(pg_namespace_nspname_index, 2684, on pg_namespace using btree(nspname name_ops));
#define NamespaceNameIndexId 2684
DECLARE_UNIQUE_INDEX(pg_namespace_oid_index, 2685, on pg_namespace using btree(oid oid_ops));
diff --git a/src/include/catalog/namespace.h b/src/include/catalog/namespace.h
index 2ccb3a7..44cf9c6 100644
--- a/src/include/catalog/namespace.h
+++ b/src/include/catalog/namespace.h
@@ -137,6 +137,8 @@ extern Oid get_collation_oid(List *collname, bool missing_ok);
extern Oid get_conversion_oid(List *conname, bool missing_ok);
extern Oid FindDefaultConversionProc(int32 for_encoding, int32 to_encoding);
+extern Oid get_statistics_oid(List *names, bool missing_ok);
+
/* initialization & transaction cleanup code */
extern void InitializeSearchPath(void);
extern void AtEOXact_Namespace(bool isCommit, bool parallel);
diff --git a/src/include/catalog/pg_mv_statistic.h b/src/include/catalog/pg_mv_statistic.h
new file mode 100644
index 0000000..c74af47
--- /dev/null
+++ b/src/include/catalog/pg_mv_statistic.h
@@ -0,0 +1,75 @@
+/*-------------------------------------------------------------------------
+ *
+ * pg_mv_statistic.h
+ * definition of the system "multivariate statistic" relation (pg_mv_statistic)
+ * along with the relation's initial contents.
+ *
+ *
+ * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/catalog/pg_mv_statistic.h
+ *
+ * NOTES
+ * the genbki.pl script reads this file and generates .bki
+ * information from the DATA() statements.
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef PG_MV_STATISTIC_H
+#define PG_MV_STATISTIC_H
+
+#include "catalog/genbki.h"
+
+/* ----------------
+ * pg_mv_statistic definition. cpp turns this into
+ * typedef struct FormData_pg_mv_statistic
+ * ----------------
+ */
+#define MvStatisticRelationId 3381
+
+CATALOG(pg_mv_statistic,3381)
+{
+ /* These fields form the unique key for the entry: */
+ Oid starelid; /* relation containing attributes */
+ NameData staname; /* statistics name */
+ Oid stanamespace; /* OID of namespace containing this statistics */
+ Oid staowner; /* statistics owner */
+
+ /* statistics requested to build */
+ bool deps_enabled; /* analyze dependencies? */
+
+ /* statistics that are available (if requested) */
+ bool deps_built; /* dependencies were built */
+
+ /* variable-length fields start here, but we allow direct access to stakeys */
+ int2vector stakeys; /* array of column keys */
+
+#ifdef CATALOG_VARLEN
+ bytea stadeps; /* dependencies (serialized) */
+#endif
+
+} FormData_pg_mv_statistic;
+
+/* ----------------
+ * Form_pg_mv_statistic corresponds to a pointer to a tuple with
+ * the format of pg_mv_statistic relation.
+ * ----------------
+ */
+typedef FormData_pg_mv_statistic *Form_pg_mv_statistic;
+
+/* ----------------
+ * compiler constants for pg_mv_statistic
+ * ----------------
+ */
+#define Natts_pg_mv_statistic 8
+#define Anum_pg_mv_statistic_starelid 1
+#define Anum_pg_mv_statistic_staname 2
+#define Anum_pg_mv_statistic_stanamespace 3
+#define Anum_pg_mv_statistic_staowner 4
+#define Anum_pg_mv_statistic_deps_enabled 5
+#define Anum_pg_mv_statistic_deps_built 6
+#define Anum_pg_mv_statistic_stakeys 7
+#define Anum_pg_mv_statistic_stadeps 8
+
+#endif /* PG_MV_STATISTIC_H */
diff --git a/src/include/catalog/pg_proc.h b/src/include/catalog/pg_proc.h
index 5c71bce..ff2d797 100644
--- a/src/include/catalog/pg_proc.h
+++ b/src/include/catalog/pg_proc.h
@@ -2666,6 +2666,11 @@ DESCR("current user privilege on any column by rel name");
DATA(insert OID = 3029 ( has_any_column_privilege PGNSP PGUID 12 10 0 0 0 f f f f t f s s 2 0 16 "26 25" _null_ _null_ _null_ _null_ _null_ has_any_column_privilege_id _null_ _null_ _null_ ));
DESCR("current user privilege on any column by rel oid");
+DATA(insert OID = 3998 ( pg_mv_stats_dependencies_info PGNSP PGUID 12 1 0 0 0 f f f f t f i s 1 0 25 "17" _null_ _null_ _null_ _null_ _null_ pg_mv_stats_dependencies_info _null_ _null_ _null_ ));
+DESCR("multivariate stats: functional dependencies info");
+DATA(insert OID = 3999 ( pg_mv_stats_dependencies_show PGNSP PGUID 12 1 0 0 0 f f f f t f i s 1 0 25 "17" _null_ _null_ _null_ _null_ _null_ pg_mv_stats_dependencies_show _null_ _null_ _null_ ));
+DESCR("multivariate stats: functional dependencies show");
+
DATA(insert OID = 1928 ( pg_stat_get_numscans PGNSP PGUID 12 1 0 0 0 f f f f t f s r 1 0 20 "26" _null_ _null_ _null_ _null_ _null_ pg_stat_get_numscans _null_ _null_ _null_ ));
DESCR("statistics: number of scans done for table/index");
DATA(insert OID = 1929 ( pg_stat_get_tuples_returned PGNSP PGUID 12 1 0 0 0 f f f f t f s r 1 0 20 "26" _null_ _null_ _null_ _null_ _null_ pg_stat_get_tuples_returned _null_ _null_ _null_ ));
diff --git a/src/include/catalog/toasting.h b/src/include/catalog/toasting.h
index b7a38ce..a52096b 100644
--- a/src/include/catalog/toasting.h
+++ b/src/include/catalog/toasting.h
@@ -49,6 +49,7 @@ extern void BootstrapToastTable(char *relName,
DECLARE_TOAST(pg_attrdef, 2830, 2831);
DECLARE_TOAST(pg_constraint, 2832, 2833);
DECLARE_TOAST(pg_description, 2834, 2835);
+DECLARE_TOAST(pg_mv_statistic, 3577, 3578);
DECLARE_TOAST(pg_proc, 2836, 2837);
DECLARE_TOAST(pg_rewrite, 2838, 2839);
DECLARE_TOAST(pg_seclabel, 3598, 3599);
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index 54f67e9..99a6a62 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -75,6 +75,10 @@ extern ObjectAddress DefineOperator(List *names, List *parameters);
extern void RemoveOperatorById(Oid operOid);
extern ObjectAddress AlterOperator(AlterOperatorStmt *stmt);
+/* commands/statscmds.c */
+extern ObjectAddress CreateStatistics(CreateStatsStmt *stmt);
+extern void RemoveStatisticsById(Oid statsOid);
+
/* commands/aggregatecmds.c */
extern ObjectAddress DefineAggregate(List *name, List *args, bool oldstyle,
List *parameters, const char *queryString);
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index fad9988..545b62a 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -266,6 +266,7 @@ typedef enum NodeTag
T_PlaceHolderInfo,
T_MinMaxAggInfo,
T_PlannerParamItem,
+ T_MVStatisticInfo,
/*
* TAGS FOR MEMORY NODES (memnodes.h)
@@ -401,6 +402,7 @@ typedef enum NodeTag
T_CreatePolicyStmt,
T_AlterPolicyStmt,
T_CreateTransformStmt,
+ T_CreateStatsStmt,
/*
* TAGS FOR PARSE TREE NODES (parsenodes.h)
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 2fd0629..e1807fb 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -601,6 +601,17 @@ typedef struct ColumnDef
int location; /* parse location, or -1 if none/unknown */
} ColumnDef;
+typedef struct CreateStatsStmt
+{
+ NodeTag type;
+ List *defnames; /* qualified name (list of Value strings) */
+ RangeVar *relation; /* relation to build statistics on */
+ List *keys; /* String nodes naming referenced column(s) */
+ List *options; /* list of DefElem nodes */
+ bool if_not_exists; /* just do nothing if statistics already exists? */
+} CreateStatsStmt;
+
+
/*
* TableLikeClause - CREATE TABLE ( ... LIKE ... ) clause
*/
@@ -1410,6 +1421,7 @@ typedef enum ObjectType
OBJECT_RULE,
OBJECT_SCHEMA,
OBJECT_SEQUENCE,
+ OBJECT_STATISTICS,
OBJECT_TABCONSTRAINT,
OBJECT_TABLE,
OBJECT_TABLESPACE,
diff --git a/src/include/nodes/relation.h b/src/include/nodes/relation.h
index 641728b..e10dcf1 100644
--- a/src/include/nodes/relation.h
+++ b/src/include/nodes/relation.h
@@ -539,6 +539,7 @@ typedef struct RelOptInfo
List *lateral_vars; /* LATERAL Vars and PHVs referenced by rel */
Relids lateral_referencers; /* rels that reference me laterally */
List *indexlist; /* list of IndexOptInfo */
+ List *mvstatlist; /* list of MVStatisticInfo */
BlockNumber pages; /* size estimates derived from pg_class */
double tuples;
double allvisfrac;
@@ -634,6 +635,33 @@ typedef struct IndexOptInfo
void (*amcostestimate) (); /* AM's cost estimator */
} IndexOptInfo;
+/*
+ * MVStatisticInfo
+ * Information about multivariate stats for planning/optimization
+ *
+ * This contains information about which columns are covered by the
+ * statistics (stakeys), which options were requested while adding the
+ * statistics (*_enabled), and which kinds of statistics were actually
+ * built and are available for the optimizer (*_built).
+ */
+typedef struct MVStatisticInfo
+{
+ NodeTag type;
+
+ Oid mvoid; /* OID of the statistics row */
+ RelOptInfo *rel; /* back-link to index's table */
+
+ /* enabled statistics */
+ bool deps_enabled; /* functional dependencies enabled */
+
+ /* built/available statistics */
+ bool deps_built; /* functional dependencies built */
+
+ /* columns in the statistics (attnums) */
+ int2vector *stakeys; /* attnums of the columns covered */
+
+} MVStatisticInfo;
+
/*
* EquivalenceClasses
diff --git a/src/include/utils/acl.h b/src/include/utils/acl.h
index 4e15a14..3e11253 100644
--- a/src/include/utils/acl.h
+++ b/src/include/utils/acl.h
@@ -330,6 +330,7 @@ extern bool pg_foreign_data_wrapper_ownercheck(Oid srv_oid, Oid roleid);
extern bool pg_foreign_server_ownercheck(Oid srv_oid, Oid roleid);
extern bool pg_event_trigger_ownercheck(Oid et_oid, Oid roleid);
extern bool pg_extension_ownercheck(Oid ext_oid, Oid roleid);
+extern bool pg_statistics_ownercheck(Oid stat_oid, Oid roleid);
extern bool has_createrole_privilege(Oid roleid);
extern bool has_bypassrls_privilege(Oid roleid);
diff --git a/src/include/utils/mvstats.h b/src/include/utils/mvstats.h
new file mode 100644
index 0000000..7ebd961
--- /dev/null
+++ b/src/include/utils/mvstats.h
@@ -0,0 +1,70 @@
+/*-------------------------------------------------------------------------
+ *
+ * mvstats.h
+ * Multivariate statistics and selectivity estimation functions.
+ *
+ *
+ * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/utils/mvstats.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef MVSTATS_H
+#define MVSTATS_H
+
+#include "fmgr.h"
+#include "commands/vacuum.h"
+
+
+#define MVSTATS_MAX_DIMENSIONS 8 /* max number of attributes */
+
+/* An associative rule, tracking [a => b] dependency.
+ *
+ * TODO Make this work with multiple columns on both sides.
+ */
+typedef struct MVDependencyData {
+ int16 a;
+ int16 b;
+} MVDependencyData;
+
+typedef MVDependencyData* MVDependency;
+
+typedef struct MVDependenciesData {
+ uint32 magic; /* magic constant marker */
+ int32 ndeps; /* number of dependencies */
+ MVDependency deps[1]; /* XXX why not a pointer? */
+} MVDependenciesData;
+
+typedef MVDependenciesData* MVDependencies;
+
+#define MVSTAT_DEPS_MAGIC 0xB4549A2C /* marks serialized bytea */
+#define MVSTAT_DEPS_TYPE_BASIC 1 /* basic dependencies type */
+
+/*
+ * TODO Maybe fetching the histogram/MCV list separately is inefficient?
+ * Consider adding a single `fetch_stats` method, fetching all
+ * stats specified using flags (or something like that).
+ */
+
+bytea * serialize_mv_dependencies(MVDependencies dependencies);
+
+/* deserialization of stats (serialization is private to analyze) */
+MVDependencies deserialize_mv_dependencies(bytea * data);
+
+/* FIXME this probably belongs somewhere else (not to operations stats) */
+extern Datum pg_mv_stats_dependencies_info(PG_FUNCTION_ARGS);
+extern Datum pg_mv_stats_dependencies_show(PG_FUNCTION_ARGS);
+
+MVDependencies
+build_mv_dependencies(int numrows, HeapTuple *rows,
+ int2vector *attrs,
+ VacAttrStats **stats);
+
+void build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
+ int natts, VacAttrStats **vacattrstats);
+
+void update_mv_stats(Oid relid, MVDependencies dependencies, int2vector *attrs);
+
+#endif
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index f2bebf2..8771f9c 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -61,6 +61,7 @@ typedef struct RelationData
bool rd_isvalid; /* relcache entry is valid */
char rd_indexvalid; /* state of rd_indexlist: 0 = not valid, 1 =
* valid, 2 = temporarily forced */
+ bool rd_mvstatvalid; /* state of rd_mvstatlist: true/false */
/*
* rd_createSubid is the ID of the highest subtransaction the rel has
@@ -93,6 +94,9 @@ typedef struct RelationData
List *rd_indexlist; /* list of OIDs of indexes on relation */
Oid rd_oidindex; /* OID of unique index on OID, if any */
Oid rd_replidindex; /* OID of replica identity index, if any */
+
+ /* data managed by RelationGetMVStatList: */
+ List *rd_mvstatlist; /* list of OIDs of multivariate stats */
/* data managed by RelationGetIndexAttrBitmap: */
Bitmapset *rd_indexattr; /* identifies columns used in indexes */
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 1b48304..9f03c8d 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -38,6 +38,7 @@ extern void RelationClose(Relation relation);
* Routines to compute/retrieve additional cached information
*/
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetMVStatList(Relation relation);
extern Oid RelationGetOidIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
extern List *RelationGetIndexExpressions(Relation relation);
diff --git a/src/include/utils/syscache.h b/src/include/utils/syscache.h
index 256615b..0e0658d 100644
--- a/src/include/utils/syscache.h
+++ b/src/include/utils/syscache.h
@@ -66,6 +66,8 @@ enum SysCacheIdentifier
INDEXRELID,
LANGNAME,
LANGOID,
+ MVSTATNAMENSP,
+ MVSTATOID,
NAMESPACENAME,
NAMESPACEOID,
OPERNAMENSP,
diff --git a/src/test/regress/expected/object_address.out b/src/test/regress/expected/object_address.out
index 75751be..eb60960 100644
--- a/src/test/regress/expected/object_address.out
+++ b/src/test/regress/expected/object_address.out
@@ -35,6 +35,7 @@ ALTER DEFAULT PRIVILEGES FOR ROLE regtest_addr_user REVOKE DELETE ON TABLES FROM
CREATE TRANSFORM FOR int LANGUAGE SQL (
FROM SQL WITH FUNCTION varchar_transform(internal),
TO SQL WITH FUNCTION int4recv(internal));
+CREATE STATISTICS addr_nsp.gentable_stat ON addr_nsp.gentable(a,b) WITH (dependencies);
-- test some error cases
SELECT pg_get_object_address('stone', '{}', '{}');
ERROR: unrecognized object type "stone"
@@ -373,7 +374,8 @@ WITH objects (type, name, args) AS (VALUES
-- extension
-- event trigger
('policy', '{addr_nsp, gentable, genpol}', '{}'),
- ('transform', '{int}', '{sql}')
+ ('transform', '{int}', '{sql}'),
+ ('statistics', '{addr_nsp, gentable_stat}', '{}')
)
SELECT (pg_identify_object(addr1.classid, addr1.objid, addr1.subobjid)).*,
-- test roundtrip through pg_identify_object_as_address
@@ -420,13 +422,14 @@ SELECT (pg_identify_object(addr1.classid, addr1.objid, addr1.subobjid)).*,
trigger | | | t on addr_nsp.gentable | t
operator family | pg_catalog | integer_ops | pg_catalog.integer_ops USING btree | t
policy | | | genpol on addr_nsp.gentable | t
+ statistics | addr_nsp | gentable_stat | addr_nsp.gentable_stat | t
collation | pg_catalog | "default" | pg_catalog."default" | t
transform | | | for integer on language sql | t
text search dictionary | addr_nsp | addr_ts_dict | addr_nsp.addr_ts_dict | t
text search parser | addr_nsp | addr_ts_prs | addr_nsp.addr_ts_prs | t
text search configuration | addr_nsp | addr_ts_conf | addr_nsp.addr_ts_conf | t
text search template | addr_nsp | addr_ts_temp | addr_nsp.addr_ts_temp | t
-(41 rows)
+(42 rows)
---
--- Cleanup resources
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 22ea06c..06f2231 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1368,6 +1368,15 @@ pg_matviews| SELECT n.nspname AS schemaname,
LEFT JOIN pg_namespace n ON ((n.oid = c.relnamespace)))
LEFT JOIN pg_tablespace t ON ((t.oid = c.reltablespace)))
WHERE (c.relkind = 'm'::"char");
+pg_mv_stats| SELECT n.nspname AS schemaname,
+ c.relname AS tablename,
+ s.staname,
+ s.stakeys AS attnums,
+ length(s.stadeps) AS depsbytes,
+ pg_mv_stats_dependencies_info(s.stadeps) AS depsinfo
+ FROM ((pg_mv_statistic s
+ JOIN pg_class c ON ((c.oid = s.starelid)))
+ LEFT JOIN pg_namespace n ON ((n.oid = c.relnamespace)));
pg_policies| SELECT n.nspname AS schemaname,
c.relname AS tablename,
pol.polname AS policyname,
diff --git a/src/test/regress/expected/sanity_check.out b/src/test/regress/expected/sanity_check.out
index eb0bc88..92a0d8a 100644
--- a/src/test/regress/expected/sanity_check.out
+++ b/src/test/regress/expected/sanity_check.out
@@ -113,6 +113,7 @@ pg_inherits|t
pg_language|t
pg_largeobject|t
pg_largeobject_metadata|t
+pg_mv_statistic|t
pg_namespace|t
pg_opclass|t
pg_operator|t
diff --git a/src/test/regress/sql/object_address.sql b/src/test/regress/sql/object_address.sql
index 68e7cb0..3775b28 100644
--- a/src/test/regress/sql/object_address.sql
+++ b/src/test/regress/sql/object_address.sql
@@ -39,6 +39,7 @@ ALTER DEFAULT PRIVILEGES FOR ROLE regtest_addr_user REVOKE DELETE ON TABLES FROM
CREATE TRANSFORM FOR int LANGUAGE SQL (
FROM SQL WITH FUNCTION varchar_transform(internal),
TO SQL WITH FUNCTION int4recv(internal));
+CREATE STATISTICS addr_nsp.gentable_stat ON addr_nsp.gentable(a,b) WITH (dependencies);
-- test some error cases
SELECT pg_get_object_address('stone', '{}', '{}');
@@ -166,7 +167,8 @@ WITH objects (type, name, args) AS (VALUES
-- extension
-- event trigger
('policy', '{addr_nsp, gentable, genpol}', '{}'),
- ('transform', '{int}', '{sql}')
+ ('transform', '{int}', '{sql}'),
+ ('statistics', '{addr_nsp, gentable_stat}', '{}')
)
SELECT (pg_identify_object(addr1.classid, addr1.objid, addr1.subobjid)).*,
-- test roundtrip through pg_identify_object_as_address
--
2.5.0
[text/x-patch] 0009-fixup-of-regression-tests-plans-changes-by-group-by-.patch (5.9K, ../../1457906378.27231.10.camel@2ndquadrant.com/9-0009-fixup-of-regression-tests-plans-changes-by-group-by-.patch)
download | inline diff:
From 771376751dfba0f469f6830c2a9eb545d1e25235 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@pgaddict.com>
Date: Sun, 28 Feb 2016 21:16:40 +0100
Subject: [PATCH 9/9] fixup of regression tests (plans changes by group by
estimation)
---
src/test/regress/expected/join.out | 18 ++++++++++--------
src/test/regress/expected/subselect.out | 25 +++++++++++--------------
src/test/regress/expected/union.out | 16 ++++++++--------
3 files changed, 29 insertions(+), 30 deletions(-)
diff --git a/src/test/regress/expected/join.out b/src/test/regress/expected/join.out
index cafbc5e..151402d 100644
--- a/src/test/regress/expected/join.out
+++ b/src/test/regress/expected/join.out
@@ -3965,18 +3965,20 @@ select d.* from d left join (select * from b group by b.id, b.c_id) s
explain (costs off)
select d.* from d left join (select distinct * from b) s
on d.a = s.id;
- QUERY PLAN
---------------------------------------
+ QUERY PLAN
+---------------------------------------------
Merge Right Join
- Merge Cond: (b.id = d.a)
- -> Unique
- -> Sort
- Sort Key: b.id, b.c_id
- -> Seq Scan on b
+ Merge Cond: (s.id = d.a)
+ -> Sort
+ Sort Key: s.id
+ -> Subquery Scan on s
+ -> HashAggregate
+ Group Key: b.id, b.c_id
+ -> Seq Scan on b
-> Sort
Sort Key: d.a
-> Seq Scan on d
-(9 rows)
+(11 rows)
-- check join removal works when uniqueness of the join condition is enforced
-- by a UNION
diff --git a/src/test/regress/expected/subselect.out b/src/test/regress/expected/subselect.out
index de64ca7..0fc93d9 100644
--- a/src/test/regress/expected/subselect.out
+++ b/src/test/regress/expected/subselect.out
@@ -807,27 +807,24 @@ select * from int4_tbl where
explain (verbose, costs off)
select * from int4_tbl o where (f1, f1) in
(select f1, generate_series(1,2) / 10 g from int4_tbl i group by f1);
- QUERY PLAN
-----------------------------------------------------------------------
- Hash Join
+ QUERY PLAN
+----------------------------------------------------------------
+ Hash Semi Join
Output: o.f1
Hash Cond: (o.f1 = "ANY_subquery".f1)
-> Seq Scan on public.int4_tbl o
Output: o.f1
-> Hash
Output: "ANY_subquery".f1, "ANY_subquery".g
- -> HashAggregate
+ -> Subquery Scan on "ANY_subquery"
Output: "ANY_subquery".f1, "ANY_subquery".g
- Group Key: "ANY_subquery".f1, "ANY_subquery".g
- -> Subquery Scan on "ANY_subquery"
- Output: "ANY_subquery".f1, "ANY_subquery".g
- Filter: ("ANY_subquery".f1 = "ANY_subquery".g)
- -> HashAggregate
- Output: i.f1, (generate_series(1, 2) / 10)
- Group Key: i.f1
- -> Seq Scan on public.int4_tbl i
- Output: i.f1
-(18 rows)
+ Filter: ("ANY_subquery".f1 = "ANY_subquery".g)
+ -> HashAggregate
+ Output: i.f1, (generate_series(1, 2) / 10)
+ Group Key: i.f1
+ -> Seq Scan on public.int4_tbl i
+ Output: i.f1
+(15 rows)
select * from int4_tbl o where (f1, f1) in
(select f1, generate_series(1,2) / 10 g from int4_tbl i group by f1);
diff --git a/src/test/regress/expected/union.out b/src/test/regress/expected/union.out
index 016571b..f2e297e 100644
--- a/src/test/regress/expected/union.out
+++ b/src/test/regress/expected/union.out
@@ -263,16 +263,16 @@ ORDER BY 1;
SELECT q2 FROM int8_tbl INTERSECT SELECT q1 FROM int8_tbl;
q2
------------------
- 4567890123456789
123
+ 4567890123456789
(2 rows)
SELECT q2 FROM int8_tbl INTERSECT ALL SELECT q1 FROM int8_tbl;
q2
------------------
+ 123
4567890123456789
4567890123456789
- 123
(3 rows)
SELECT q2 FROM int8_tbl EXCEPT SELECT q1 FROM int8_tbl ORDER BY 1;
@@ -305,16 +305,16 @@ SELECT q1 FROM int8_tbl EXCEPT SELECT q2 FROM int8_tbl;
SELECT q1 FROM int8_tbl EXCEPT ALL SELECT q2 FROM int8_tbl;
q1
------------------
- 4567890123456789
123
+ 4567890123456789
(2 rows)
SELECT q1 FROM int8_tbl EXCEPT ALL SELECT DISTINCT q2 FROM int8_tbl;
q1
------------------
+ 123
4567890123456789
4567890123456789
- 123
(3 rows)
SELECT q1 FROM int8_tbl EXCEPT ALL SELECT q1 FROM int8_tbl FOR NO KEY UPDATE;
@@ -343,8 +343,8 @@ SELECT f1 FROM float8_tbl EXCEPT SELECT f1 FROM int4_tbl ORDER BY 1;
SELECT q1 FROM int8_tbl INTERSECT SELECT q2 FROM int8_tbl UNION ALL SELECT q2 FROM int8_tbl;
q1
-------------------
- 4567890123456789
123
+ 4567890123456789
456
4567890123456789
123
@@ -355,15 +355,15 @@ SELECT q1 FROM int8_tbl INTERSECT SELECT q2 FROM int8_tbl UNION ALL SELECT q2 FR
SELECT q1 FROM int8_tbl INTERSECT (((SELECT q2 FROM int8_tbl UNION ALL SELECT q2 FROM int8_tbl)));
q1
------------------
- 4567890123456789
123
+ 4567890123456789
(2 rows)
(((SELECT q1 FROM int8_tbl INTERSECT SELECT q2 FROM int8_tbl))) UNION ALL SELECT q2 FROM int8_tbl;
q1
-------------------
- 4567890123456789
123
+ 4567890123456789
456
4567890123456789
123
@@ -419,8 +419,8 @@ HINT: There is a column named "q2" in table "*SELECT* 2", but it cannot be refe
SELECT q1 FROM int8_tbl EXCEPT (((SELECT q2 FROM int8_tbl ORDER BY q2 LIMIT 1)));
q1
------------------
- 4567890123456789
123
+ 4567890123456789
(2 rows)
--
--
2.5.0
[text/x-patch] 0001-teach-pull_-varno-varattno-_walker-about-RestrictInf.patch (1.4K, ../../1457906378.27231.10.camel@2ndquadrant.com/10-0001-teach-pull_-varno-varattno-_walker-about-RestrictInf.patch)
download | inline diff:
From e073a4d8368a0b0c66cd2933e6a7a210c1b5c53f Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@pgaddict.com>
Date: Tue, 28 Apr 2015 19:56:33 +0200
Subject: [PATCH 1/9] teach pull_(varno|varattno)_walker about RestrictInfo
otherwise pull_varnos fails when processing OR clauses
---
src/backend/optimizer/util/var.c | 16 ++++++++++++++++
1 file changed, 16 insertions(+)
diff --git a/src/backend/optimizer/util/var.c b/src/backend/optimizer/util/var.c
index 292e1f4..9228a46 100644
--- a/src/backend/optimizer/util/var.c
+++ b/src/backend/optimizer/util/var.c
@@ -196,6 +196,13 @@ pull_varnos_walker(Node *node, pull_varnos_context *context)
context->sublevels_up--;
return result;
}
+ if (IsA(node, RestrictInfo))
+ {
+ RestrictInfo *rinfo = (RestrictInfo*)node;
+ context->varnos = bms_add_members(context->varnos,
+ rinfo->clause_relids);
+ return false;
+ }
return expression_tree_walker(node, pull_varnos_walker,
(void *) context);
}
@@ -244,6 +251,15 @@ pull_varattnos_walker(Node *node, pull_varattnos_context *context)
return false;
}
+ if (IsA(node, RestrictInfo))
+ {
+ RestrictInfo *rinfo = (RestrictInfo *)node;
+
+ return expression_tree_walker((Node*)rinfo->clause,
+ pull_varattnos_walker,
+ (void*) context);
+ }
+
/* Should not find an unplanned subquery */
Assert(!IsA(node, Query));
--
2.5.0
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: multivariate statistics v14
@ 2016-03-16 02:29 Tatsuo Ishii <ishii@postgresql.org>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
2 siblings, 0 replies; 58+ messages in thread
From: Tatsuo Ishii @ 2016-03-16 02:29 UTC (permalink / raw)
To: tomas.vondra@2ndquadrant.com; +Cc: jeff.janes@gmail.com; alvherre@2ndquadrant.com; pgsql-hackers
> Instead of simply multiplying the ndistinct estimate with selecticity,
> we instead use the formula for the expected number of distinct values
> observed in 'k' rows when there are 'd' distinct values in the bin
>
> d * (1 - ((d - 1) / d)^k)
>
> This is 'with replacements' which seems appropriate for the use, and it
> mostly assumes uniform distribution of the distinct values. So if the
> distribution is not uniform (e.g. there are very frequent groups) this
> may be less accurate than the current algorithm in some cases, giving
> over-estimates. But that's probably better than OOM.
> ---
> src/backend/utils/adt/selfuncs.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/src/backend/utils/adt/selfuncs.c b/src/backend/utils/adt/selfuncs.c
> index f8d39aa..6eceedf 100644
> --- a/src/backend/utils/adt/selfuncs.c
> +++ b/src/backend/utils/adt/selfuncs.c
> @@ -3466,7 +3466,7 @@ estimate_num_groups(PlannerInfo *root, List *groupExprs, double input_rows,
> /*
> * Multiply by restriction selectivity.
> */
> - reldistinct *= rel->rows / rel->tuples;
> + reldistinct = reldistinct * (1 - powl((reldistinct - 1) / reldistinct,rel->rows));
Why do you change "*=" style? I see no reason to change this.
reldistinct *= 1 - powl((reldistinct - 1) / reldistinct, rel->rows);
Looks better to me because it's shorter and cleaner.
Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: multivariate statistics v14
@ 2016-03-16 02:58 Tatsuo Ishii <ishii@postgresql.org>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
2 siblings, 1 reply; 58+ messages in thread
From: Tatsuo Ishii @ 2016-03-16 02:58 UTC (permalink / raw)
To: tomas.vondra@2ndquadrant.com; +Cc: jeff.janes@gmail.com; alvherre@2ndquadrant.com; pgsql-hackers
I apology if it's already discussed. I am new to this patch.
> Attached is v15 of the patch series, fixing this and also doing quite a
> few additional improvements:
>
> * added some basic examples into the SGML documentation
>
> * addressing the objectaddress omissions, as pointed out by Alvaro
>
> * support for ALTER STATISTICS ... OWNER TO / RENAME / SET SCHEMA
>
> * significant refactoring of MCV and histogram code, particularly
> serialization, deserialization and building
>
> * reworking the functional dependencies to support more complex
> dependencies, with multiple columns as 'conditions'
>
> * the reduction using functional dependencies is also significantly
> simplified (I decided to get rid of computing the transitive closure
> for now - it got too complex after the multi-condition dependencies,
> so I'll leave that for the future
Do you have any other missing parts in this work? I am asking because
I wonder if you want to push this into 9.6 or rather 9.7.
Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: multivariate statistics v14
@ 2016-03-16 08:31 Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
2 siblings, 1 reply; 58+ messages in thread
From: Kyotaro HORIGUCHI @ 2016-03-16 08:31 UTC (permalink / raw)
To: tomas.vondra@2ndquadrant.com; +Cc: jeff.janes@gmail.com; alvherre@2ndquadrant.com; pgsql-hackers
Hello, I returned to this.
At Sun, 13 Mar 2016 22:59:38 +0100, Tomas Vondra <tomas.vondra@2ndquadrant.com> wrote in <camel@2ndquadrant.com/">1457906378.27231.10.camel@2ndquadrant.com>
> Oh, yeah. There was an extra pfree().
>
> Attached is v15 of the patch series, fixing this and also doing quite a
> few additional improvements:
>
> * added some basic examples into the SGML documentation
>
> * addressing the objectaddress omissions, as pointed out by Alvaro
>
> * support for ALTER STATISTICS ... OWNER TO / RENAME / SET SCHEMA
>
> * significant refactoring of MCV and histogram code, particularly
> serialization, deserialization and building
>
> * reworking the functional dependencies to support more complex
> dependencies, with multiple columns as 'conditions'
>
> * the reduction using functional dependencies is also significantly
> simplified (I decided to get rid of computing the transitive closure
> for now - it got too complex after the multi-condition dependencies,
> so I'll leave that for the future
Many trailing white spaces found.
0002
+ * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group
2014 should be 2016?
This patch defines many "magic"s for many structs, but
magic(number)s seems to be used to identify file or buffer page
in PostgreSQL. They wouldn't be needed if you don't intend to
dig out or identify the orphan memory blocks of mvstats.
+ MVDependency deps[1]; /* XXX why not a pointer? */
MVDependency seems to be a pointer type.
+ if (numcols >= MVSTATS_MAX_DIMENSIONS)
+ ereport(ERROR,
and
+ Assert((attrs->dim1 >= 2) && (attrs->dim1 <= MVSTATS_MAX_DIMENSIONS));
seem to be contradicting.
.. Sorry, time is up..
regards,
--
Kyotaro Horiguchi
NTT Open Source Software Center
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: multivariate statistics v14
@ 2016-03-16 13:32 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
0 siblings, 1 reply; 58+ messages in thread
From: Tomas Vondra @ 2016-03-16 13:32 UTC (permalink / raw)
To: Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>; +Cc: jeff.janes@gmail.com; alvherre@2ndquadrant.com; pgsql-hackers
On 03/16/2016 09:31 AM, Kyotaro HORIGUCHI wrote:
> Hello, I returned to this.
>
> At Sun, 13 Mar 2016 22:59:38 +0100, Tomas Vondra <tomas.vondra@2ndquadrant.com> wrote in <camel@2ndquadrant.com/">1457906378.27231.10.camel@2ndquadrant.com>
>> Oh, yeah. There was an extra pfree().
>>
>> Attached is v15 of the patch series, fixing this and also doing quite a
>> few additional improvements:
>>
>> * added some basic examples into the SGML documentation
>>
>> * addressing the objectaddress omissions, as pointed out by Alvaro
>>
>> * support for ALTER STATISTICS ... OWNER TO / RENAME / SET SCHEMA
>>
>> * significant refactoring of MCV and histogram code, particularly
>> serialization, deserialization and building
>>
>> * reworking the functional dependencies to support more complex
>> dependencies, with multiple columns as 'conditions'
>>
>> * the reduction using functional dependencies is also significantly
>> simplified (I decided to get rid of computing the transitive closure
>> for now - it got too complex after the multi-condition dependencies,
>> so I'll leave that for the future
>
> Many trailing white spaces found.
Sorry, haven't noticed that after one of the rebases. Fixed in the
attached v15 of the patch.
>
> 0002
>
> + * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group
>
> 2014 should be 2016?
Yes, the copyright info will need some tweaks. There's a few other files
with 2015, and I think the start should be the current year (and not 1996).
>
>
> This patch defines many "magic"s for many structs, but
> magic(number)s seems to be used to identify file or buffer page
> in PostgreSQL. They wouldn't be needed if you don't intend to
> dig out or identify the orphan memory blocks of mvstats.
>
> + MVDependency deps[1]; /* XXX why not a pointer? */
>
> MVDependency seems to be a pointer type.
Right, but we need an array of the structures here, so one way is to use
a pointer and the other one is using variable-length field. Will remove
the comment, I think the structure is fine as is.
>
> + if (numcols >= MVSTATS_MAX_DIMENSIONS)
> + ereport(ERROR,
> and
> + Assert((attrs->dim1 >= 2) && (attrs->dim1 <= MVSTATS_MAX_DIMENSIONS));
>
> seem to be contradicting.
Nope, because the first check is in a loop where 'numcols' is used as an
index into an array with MVSTATS_MAX_DIMENSIONS elements.
>
> .. Sorry, time is up..
Thanks for the comments!
Attached is v15 of the patch, that also fixes one mistake - after
reworking the functional dependencies to support multiple columns on the
left side (as conditions), I failed to move it to the proper place in
the patch series. So 0002 built the dependencies in the old way and 0003
changed it to the new one. That was pointless and added another 20kB to
the patch, so v15 moves the new code to 0002.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[text/x-patch] 0001-teach-pull_-varno-varattno-_walker-about-RestrictInf.patch (1.4K, ../../7aa6145f-302b-2077-b961-702cc61710bb@2ndquadrant.com/2-0001-teach-pull_-varno-varattno-_walker-about-RestrictInf.patch)
download | inline diff:
From 5de240e541a0893ed945b16ec1fe23522c00ae61 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@pgaddict.com>
Date: Tue, 28 Apr 2015 19:56:33 +0200
Subject: [PATCH 1/9] teach pull_(varno|varattno)_walker about RestrictInfo
otherwise pull_varnos fails when processing OR clauses
---
src/backend/optimizer/util/var.c | 16 ++++++++++++++++
1 file changed, 16 insertions(+)
diff --git a/src/backend/optimizer/util/var.c b/src/backend/optimizer/util/var.c
index 292e1f4..9228a46 100644
--- a/src/backend/optimizer/util/var.c
+++ b/src/backend/optimizer/util/var.c
@@ -196,6 +196,13 @@ pull_varnos_walker(Node *node, pull_varnos_context *context)
context->sublevels_up--;
return result;
}
+ if (IsA(node, RestrictInfo))
+ {
+ RestrictInfo *rinfo = (RestrictInfo*)node;
+ context->varnos = bms_add_members(context->varnos,
+ rinfo->clause_relids);
+ return false;
+ }
return expression_tree_walker(node, pull_varnos_walker,
(void *) context);
}
@@ -244,6 +251,15 @@ pull_varattnos_walker(Node *node, pull_varattnos_context *context)
return false;
}
+ if (IsA(node, RestrictInfo))
+ {
+ RestrictInfo *rinfo = (RestrictInfo *)node;
+
+ return expression_tree_walker((Node*)rinfo->clause,
+ pull_varattnos_walker,
+ (void*) context);
+ }
+
/* Should not find an unplanned subquery */
Assert(!IsA(node, Query));
--
2.5.0
[text/x-patch] 0002-shared-infrastructure-and-functional-dependencies.patch (136.0K, ../../7aa6145f-302b-2077-b961-702cc61710bb@2ndquadrant.com/3-0002-shared-infrastructure-and-functional-dependencies.patch)
download | inline diff:
From 6cf5d3b456bb294bb033b9e1e2eb545cfd4c1739 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Sun, 11 Jan 2015 19:51:48 +0100
Subject: [PATCH 2/9] shared infrastructure and functional dependencies
Basic infrastructure shared by all kinds of multivariate stats, most
importantly:
- adds a new system catalog (pg_mv_statistic)
- CREATE STATISTICS name ON table (columns) WITH (options)
- DROP STATISTICS name
- ALTER STATISTICS ... OWNER TO / SET SCHEMA / RENAME
- implementation of functional dependencies (the simplest type of
multivariate statistics)
- building functional dependencies in ANALYZE
- updates existing regression tests (new catalog etc.)
- adds a new regression test for functional dependencies
This does not include any changes to the optimizer, i.e. it does not
influence the query planning (subject to follow-up patches).
The current implementation requires a valid 'ltopr' for the columns, so
that we can sort the sample rows in various ways, both in this patch
and other kinds of statistics. Maybe this restriction could be relaxed
in the future, requiring just 'eqopr' in case of stats not sorting the
data (e.g. functional dependencies and MCV lists).
Maybe some of the stats (functional dependencies and MCV list with
limited functionality) might be made to work with hashes of the values,
which is sufficient for equality comparisons. But the queries would
require the equality operator anyway, so it's not really a weaker
requirement. The hashes might reduce space requirements, though.
The algorithm detecting the dependencies is rather simple and probably
needs improvements, so that it detects more complicated dependencies,
and also validation of the math.
The name 'functional dependencies' is more correct (than 'association
rules') as it's exactly the name used in relational theory (esp. Normal
Forms) for tracking column-level dependencies.
The multivariate statistics are automatically removed in two situations
(a) after a DROP TABLE (obviously)
(b) after ALTER TABLE ... DROP COLUMN, if the statistics would be
defined on less than 2 columns (remaining)
If there are more at least two remaining columns, we keep the
statistics but perform cleanup on the next ANALYZE. The dropped columns
are removed from stakeys, and the new statistics is built on the
smaller set.
We can't do this at DROP COLUMN, because that'd leave us with invalid
statistics, or we'd have to throw it away although we can still use it.
This lazy approach lets us use the statistics although some of the
columns are dead.
This also adds a simple list of statistics to \d in psql.
This means the statistics are created within a schema by using a
qualified name (or using the default schema)
CREATE STATISTICS schema.statistics ON ...
and then dropped by specifying qualified name
DROP STATISTICS schema.statistics
or searching through search_path (just like with other objects).
This also gets rid of the "(opt_)stats_name" definitions in gram.y and
instead replaces them with just "opt_any_name", although the optional
case is not really handled currently - there's no generated name yet
(so either we should drop it or implement it).
I'm not entirely sure making statistics schema-specific is that a great
idea. Maybe it should be "global", but that does not seem right (e.g.
it makes multi-tenant systems based on schemas more difficult to
manage, because tenants would interact).
---
doc/src/sgml/ref/allfiles.sgml | 3 +
doc/src/sgml/ref/alter_statistics.sgml | 115 +++++
doc/src/sgml/ref/create_statistics.sgml | 198 ++++++++
doc/src/sgml/ref/drop_statistics.sgml | 91 ++++
doc/src/sgml/reference.sgml | 2 +
src/backend/catalog/Makefile | 1 +
src/backend/catalog/aclchk.c | 27 +
src/backend/catalog/dependency.c | 11 +-
src/backend/catalog/heap.c | 102 ++++
src/backend/catalog/namespace.c | 51 ++
src/backend/catalog/objectaddress.c | 54 ++
src/backend/catalog/system_views.sql | 11 +
src/backend/commands/Makefile | 6 +-
src/backend/commands/alter.c | 3 +
src/backend/commands/analyze.c | 21 +
src/backend/commands/dropcmds.c | 4 +
src/backend/commands/event_trigger.c | 3 +
src/backend/commands/statscmds.c | 277 +++++++++++
src/backend/nodes/copyfuncs.c | 17 +
src/backend/nodes/outfuncs.c | 18 +
src/backend/optimizer/util/plancat.c | 59 +++
src/backend/parser/gram.y | 60 ++-
src/backend/tcop/utility.c | 14 +
src/backend/utils/Makefile | 2 +-
src/backend/utils/cache/relcache.c | 59 +++
src/backend/utils/cache/syscache.c | 23 +
src/backend/utils/mvstats/Makefile | 17 +
src/backend/utils/mvstats/README.dependencies | 222 +++++++++
src/backend/utils/mvstats/common.c | 376 ++++++++++++++
src/backend/utils/mvstats/common.h | 78 +++
src/backend/utils/mvstats/dependencies.c | 686 ++++++++++++++++++++++++++
src/bin/psql/describe.c | 44 ++
src/include/catalog/dependency.h | 5 +-
src/include/catalog/heap.h | 1 +
src/include/catalog/indexing.h | 7 +
src/include/catalog/namespace.h | 2 +
src/include/catalog/pg_mv_statistic.h | 75 +++
src/include/catalog/pg_proc.h | 5 +
src/include/catalog/toasting.h | 1 +
src/include/commands/defrem.h | 4 +
src/include/nodes/nodes.h | 2 +
src/include/nodes/parsenodes.h | 12 +
src/include/nodes/relation.h | 28 ++
src/include/utils/acl.h | 1 +
src/include/utils/mvstats.h | 71 +++
src/include/utils/rel.h | 4 +
src/include/utils/relcache.h | 1 +
src/include/utils/syscache.h | 2 +
src/test/regress/expected/mv_dependencies.out | 150 ++++++
src/test/regress/expected/object_address.out | 7 +-
src/test/regress/expected/rules.out | 9 +
src/test/regress/expected/sanity_check.out | 1 +
src/test/regress/serial_schedule | 1 +
src/test/regress/sql/mv_dependencies.sql | 142 ++++++
src/test/regress/sql/object_address.sql | 4 +-
55 files changed, 3179 insertions(+), 11 deletions(-)
create mode 100644 doc/src/sgml/ref/alter_statistics.sgml
create mode 100644 doc/src/sgml/ref/create_statistics.sgml
create mode 100644 doc/src/sgml/ref/drop_statistics.sgml
create mode 100644 src/backend/commands/statscmds.c
create mode 100644 src/backend/utils/mvstats/Makefile
create mode 100644 src/backend/utils/mvstats/README.dependencies
create mode 100644 src/backend/utils/mvstats/common.c
create mode 100644 src/backend/utils/mvstats/common.h
create mode 100644 src/backend/utils/mvstats/dependencies.c
create mode 100644 src/include/catalog/pg_mv_statistic.h
create mode 100644 src/include/utils/mvstats.h
create mode 100644 src/test/regress/expected/mv_dependencies.out
create mode 100644 src/test/regress/sql/mv_dependencies.sql
diff --git a/doc/src/sgml/ref/allfiles.sgml b/doc/src/sgml/ref/allfiles.sgml
index bf95453..524ed83 100644
--- a/doc/src/sgml/ref/allfiles.sgml
+++ b/doc/src/sgml/ref/allfiles.sgml
@@ -32,6 +32,7 @@ Complete list of usable sgml source files in this directory.
<!ENTITY alterServer SYSTEM "alter_server.sgml">
<!ENTITY alterSequence SYSTEM "alter_sequence.sgml">
<!ENTITY alterSystem SYSTEM "alter_system.sgml">
+<!ENTITY alterStatistics SYSTEM "alter_statistics.sgml">
<!ENTITY alterTable SYSTEM "alter_table.sgml">
<!ENTITY alterTableSpace SYSTEM "alter_tablespace.sgml">
<!ENTITY alterTSConfig SYSTEM "alter_tsconfig.sgml">
@@ -76,6 +77,7 @@ Complete list of usable sgml source files in this directory.
<!ENTITY createSchema SYSTEM "create_schema.sgml">
<!ENTITY createSequence SYSTEM "create_sequence.sgml">
<!ENTITY createServer SYSTEM "create_server.sgml">
+<!ENTITY createStatistics SYSTEM "create_statistics.sgml">
<!ENTITY createTable SYSTEM "create_table.sgml">
<!ENTITY createTableAs SYSTEM "create_table_as.sgml">
<!ENTITY createTableSpace SYSTEM "create_tablespace.sgml">
@@ -119,6 +121,7 @@ Complete list of usable sgml source files in this directory.
<!ENTITY dropSchema SYSTEM "drop_schema.sgml">
<!ENTITY dropSequence SYSTEM "drop_sequence.sgml">
<!ENTITY dropServer SYSTEM "drop_server.sgml">
+<!ENTITY dropStatistics SYSTEM "drop_statistics.sgml">
<!ENTITY dropTable SYSTEM "drop_table.sgml">
<!ENTITY dropTableSpace SYSTEM "drop_tablespace.sgml">
<!ENTITY dropTransform SYSTEM "drop_transform.sgml">
diff --git a/doc/src/sgml/ref/alter_statistics.sgml b/doc/src/sgml/ref/alter_statistics.sgml
new file mode 100644
index 0000000..aa421c0
--- /dev/null
+++ b/doc/src/sgml/ref/alter_statistics.sgml
@@ -0,0 +1,115 @@
+<!--
+doc/src/sgml/ref/alter_statistics.sgml
+PostgreSQL documentation
+-->
+
+<refentry id="SQL-ALTERSTATISTICS">
+ <indexterm zone="sql-alterstatistics">
+ <primary>ALTER STATISTICS</primary>
+ </indexterm>
+
+ <refmeta>
+ <refentrytitle>ALTER STATISTICS</refentrytitle>
+ <manvolnum>7</manvolnum>
+ <refmiscinfo>SQL - Language Statements</refmiscinfo>
+ </refmeta>
+
+ <refnamediv>
+ <refname>ALTER STATISTICS</refname>
+ <refpurpose>
+ change the definition of a multivariate statistics
+ </refpurpose>
+ </refnamediv>
+
+ <refsynopsisdiv>
+<synopsis>
+ALTER STATISTICS <replaceable class="parameter">name</replaceable> OWNER TO { <replaceable class="PARAMETER">new_owner</replaceable> | CURRENT_USER | SESSION_USER }
+ALTER STATISTICS <replaceable class="parameter">name</replaceable> RENAME TO <replaceable class="parameter">new_name</replaceable>
+ALTER STATISTICS <replaceable class="parameter">name</replaceable> SET SCHEMA <replaceable class="parameter">new_schema</replaceable>
+</synopsis>
+ </refsynopsisdiv>
+
+ <refsect1>
+ <title>Description</title>
+
+ <para>
+ <command>ALTER STATISTICS</command> changes the parameters of an existing
+ multivariate statistics. Any parameters not specifically set in the
+ <command>ALTER STATISTICS</command> command retain their prior settings.
+ </para>
+
+ <para>
+ You must own the statistics to use <command>ALTER STATISTICS</>.
+ To change a statistics' schema, you must also have <literal>CREATE</>
+ privilege on the new schema.
+ To alter the owner, you must also be a direct or indirect member of the new
+ owning role, and that role must have <literal>CREATE</literal> privilege on
+ the statistics' schema. (These restrictions enforce that altering the owner
+ doesn't do anything you couldn't do by dropping and recreating the statistics.
+ However, a superuser can alter ownership of any statistics anyway.)
+ </para>
+ </refsect1>
+
+ <refsect1>
+ <title>Parameters</title>
+
+ <para>
+ <variablelist>
+ <varlistentry>
+ <term><replaceable class="parameter">name</replaceable></term>
+ <listitem>
+ <para>
+ The name (optionally schema-qualified) of a statistics to be altered.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><replaceable class="PARAMETER">new_owner</replaceable></term>
+ <listitem>
+ <para>
+ The user name of the new owner of the statistics.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><replaceable class="parameter">new_name</replaceable></term>
+ <listitem>
+ <para>
+ The new name for the statistics.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><replaceable class="parameter">new_schema</replaceable></term>
+ <listitem>
+ <para>
+ The new schema for the statistics.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+ </refsect1>
+
+ <refsect1>
+ <title>Compatibility</title>
+
+ <para>
+ There's no <command>ALTER STATISTICS</command> command in the SQL standard.
+ </para>
+ </refsect1>
+
+ <refsect1>
+ <title>See Also</title>
+
+ <simplelist type="inline">
+ <member><xref linkend="sql-createstatistics"></member>
+ <member><xref linkend="sql-dropstatistics"></member>
+ </simplelist>
+ </refsect1>
+
+</refentry>
diff --git a/doc/src/sgml/ref/create_statistics.sgml b/doc/src/sgml/ref/create_statistics.sgml
new file mode 100644
index 0000000..ff09fa5
--- /dev/null
+++ b/doc/src/sgml/ref/create_statistics.sgml
@@ -0,0 +1,198 @@
+<!--
+doc/src/sgml/ref/create_statistics.sgml
+PostgreSQL documentation
+-->
+
+<refentry id="SQL-CREATESTATISTICS">
+ <indexterm zone="sql-createstatistics">
+ <primary>CREATE STATISTICS</primary>
+ </indexterm>
+
+ <refmeta>
+ <refentrytitle>CREATE STATISTICS</refentrytitle>
+ <manvolnum>7</manvolnum>
+ <refmiscinfo>SQL - Language Statements</refmiscinfo>
+ </refmeta>
+
+ <refnamediv>
+ <refname>CREATE STATISTICS</refname>
+ <refpurpose>define a new statistics</refpurpose>
+ </refnamediv>
+
+ <refsynopsisdiv>
+<synopsis>
+CREATE STATISTICS [ IF NOT EXISTS ] <replaceable class="PARAMETER">statistics_name</replaceable> ON <replaceable class="PARAMETER">table_name</replaceable> ( [
+ { <replaceable class="PARAMETER">column_name</replaceable> } ] [, ...])
+[ WITH ( <replaceable class="PARAMETER">statistics_parameter</replaceable> [= <replaceable class="PARAMETER">value</replaceable>] [, ... ] )
+</synopsis>
+
+ </refsynopsisdiv>
+
+ <refsect1 id="SQL-CREATESTATISTICS-description">
+ <title>Description</title>
+
+ <para>
+ <command>CREATE STATISTICS</command> will create a new multivariate
+ statistics on the table. The statistics will be created in the in the
+ current database. The statistics will be owned by the user issuing
+ the command.
+ </para>
+
+ <para>
+ If a schema name is given (for example, <literal>CREATE STATISTICS
+ myschema.mystat ...</>) then the statistics is created in the specified
+ schema. Otherwise it is created in the current schema. The name of
+ the table must be distinct from the name of any other statistics in the
+ same schema.
+ </para>
+
+ <para>
+ To be able to create a table, you must have <literal>USAGE</literal>
+ privilege on all column types or the type in the <literal>OF</literal>
+ clause, respectively.
+ </para>
+ </refsect1>
+
+ <refsect1>
+ <title>Parameters</title>
+
+ <variablelist>
+
+ <varlistentry>
+ <term><literal>IF NOT EXISTS</></term>
+ <listitem>
+ <para>
+ Do not throw an error if a statistics with the same name already exists.
+ A notice is issued in this case. Note that there is no guarantee that
+ the existing statistics is anything like the one that would have been
+ created.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><replaceable class="PARAMETER">statistics_name</replaceable></term>
+ <listitem>
+ <para>
+ The name (optionally schema-qualified) of the statistics to be created.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><replaceable class="PARAMETER">table_name</replaceable></term>
+ <listitem>
+ <para>
+ The name (optionally schema-qualified) of the table the statistics should
+ be created on.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><replaceable class="PARAMETER">column_name</replaceable></term>
+ <listitem>
+ <para>
+ The name of a column to be included in the statistics.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>WITH ( <replaceable class="PARAMETER">statistics_parameter</replaceable> [= <replaceable class="PARAMETER">value</replaceable>] [, ... ] )</literal></term>
+ <listitem>
+ <para>
+ ...
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ <refsect2 id="SQL-CREATESTATISTICS-parameters">
+ <title id="SQL-CREATESTATISTICS-parameters-title">Statistics Parameters</title>
+
+ <indexterm zone="sql-createstatistics-parameters">
+ <primary>statistics parameters</primary>
+ </indexterm>
+
+ <para>
+ The <literal>WITH</> clause can specify <firstterm>statistics parameters</>
+ for statistics. The currently available parameters are listed below.
+ </para>
+
+ <variablelist>
+
+ <varlistentry>
+ <term><literal>dependencies</> (<type>boolean</>)</term>
+ <listitem>
+ <para>
+ Enables functional dependencies for the statistics.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </refsect2>
+ </refsect1>
+
+ <refsect1 id="SQL-CREATESTATISTICS-notes">
+ <title>Notes</title>
+
+ <para>
+ ...
+ </para>
+
+ </refsect1>
+
+
+ <refsect1 id="SQL-CREATESTATISTICS-examples">
+ <title>Examples</title>
+
+ <para>
+ Create table <structname>t1</> with two functionally dependent columns, i.e.
+ knowledge of a value in the first column is sufficient for detemining the
+ value in the other column. Then functional dependencies are built on those
+ columns:
+
+<programlisting>
+CREATE TABLE t1 (
+ a int,
+ b int
+);
+
+INSERT INTO t1 SELECT i/100, i/500
+ FROM generate_series(1,1000000) s(i);
+
+CREATE STATISTICS s1 ON t1 (a, b) WITH (dependencies);
+
+ANALYZE t1;
+
+-- valid combination of values
+EXPLAIN ANALYZE SELECT * FROM t1 WHERE (a = 1) AND (b = 1);
+
+-- invalid combination of values
+EXPLAIN ANALYZE SELECT * FROM t1 WHERE (a = 1) AND (b = 2);
+</programlisting>
+ </para>
+
+ </refsect1>
+
+ <refsect1>
+ <title>Compatibility</title>
+
+ <para>
+ There's no <command>CREATE STATISTICS</command> command in the SQL standard.
+ </para>
+ </refsect1>
+
+ <refsect1>
+ <title>See Also</title>
+
+ <simplelist type="inline">
+ <member><xref linkend="sql-alterstatistics"></member>
+ <member><xref linkend="sql-dropstatistics"></member>
+ </simplelist>
+ </refsect1>
+</refentry>
diff --git a/doc/src/sgml/ref/drop_statistics.sgml b/doc/src/sgml/ref/drop_statistics.sgml
new file mode 100644
index 0000000..dd9047a
--- /dev/null
+++ b/doc/src/sgml/ref/drop_statistics.sgml
@@ -0,0 +1,91 @@
+<!--
+doc/src/sgml/ref/drop_statistics.sgml
+PostgreSQL documentation
+-->
+
+<refentry id="SQL-DROPSTATISTICS">
+ <indexterm zone="sql-dropstatistics">
+ <primary>DROP STATISTICS</primary>
+ </indexterm>
+
+ <refmeta>
+ <refentrytitle>DROP STATISTICS</refentrytitle>
+ <manvolnum>7</manvolnum>
+ <refmiscinfo>SQL - Language Statements</refmiscinfo>
+ </refmeta>
+
+ <refnamediv>
+ <refname>DROP STATISTICS</refname>
+ <refpurpose>remove a statistics</refpurpose>
+ </refnamediv>
+
+ <refsynopsisdiv>
+<synopsis>
+DROP STATISTICS [ IF EXISTS ] <replaceable class="PARAMETER">name</replaceable> [, ...]
+</synopsis>
+ </refsynopsisdiv>
+
+ <refsect1>
+ <title>Description</title>
+
+ <para>
+ <command>DROP STATISTICS</command> removes statistics from the database.
+ Only the statistics owner, the schema owner, and superuser can drop a
+ statistics.
+ </para>
+
+ </refsect1>
+
+ <refsect1>
+ <title>Parameters</title>
+
+ <variablelist>
+ <varlistentry>
+ <term><literal>IF EXISTS</literal></term>
+ <listitem>
+ <para>
+ Do not throw an error if the statistics does not exist. A notice is
+ issued in this case.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><replaceable class="PARAMETER">name</replaceable></term>
+ <listitem>
+ <para>
+ The name (optionally schema-qualified) of the statistics to drop.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </refsect1>
+
+ <refsect1>
+ <title>Examples</title>
+
+ <para>
+ ...
+ </para>
+
+ </refsect1>
+
+ <refsect1>
+ <title>Compatibility</title>
+
+ <para>
+ There's no <command>DROP STATISTICS</command> command in the SQL standard.
+ </para>
+ </refsect1>
+
+ <refsect1>
+ <title>See Also</title>
+
+ <simplelist type="inline">
+ <member><xref linkend="sql-alterstatistics"></member>
+ <member><xref linkend="sql-createstatistics"></member>
+ </simplelist>
+ </refsect1>
+
+</refentry>
diff --git a/doc/src/sgml/reference.sgml b/doc/src/sgml/reference.sgml
index 03020df..2b07b2d 100644
--- a/doc/src/sgml/reference.sgml
+++ b/doc/src/sgml/reference.sgml
@@ -104,6 +104,7 @@
&createSchema;
&createSequence;
&createServer;
+ &createStatistics;
&createTable;
&createTableAs;
&createTableSpace;
@@ -147,6 +148,7 @@
&dropSchema;
&dropSequence;
&dropServer;
+ &dropStatistics;
&dropTable;
&dropTableSpace;
&dropTSConfig;
diff --git a/src/backend/catalog/Makefile b/src/backend/catalog/Makefile
index 25130ec..058b8a9 100644
--- a/src/backend/catalog/Makefile
+++ b/src/backend/catalog/Makefile
@@ -32,6 +32,7 @@ POSTGRES_BKI_SRCS = $(addprefix $(top_srcdir)/src/include/catalog/,\
pg_attrdef.h pg_constraint.h pg_inherits.h pg_index.h pg_operator.h \
pg_opfamily.h pg_opclass.h pg_am.h pg_amop.h pg_amproc.h \
pg_language.h pg_largeobject_metadata.h pg_largeobject.h pg_aggregate.h \
+ pg_mv_statistic.h \
pg_statistic.h pg_rewrite.h pg_trigger.h pg_event_trigger.h pg_description.h \
pg_cast.h pg_enum.h pg_namespace.h pg_conversion.h pg_depend.h \
pg_database.h pg_db_role_setting.h pg_tablespace.h pg_pltemplate.h \
diff --git a/src/backend/catalog/aclchk.c b/src/backend/catalog/aclchk.c
index 0f3bc07..e21aacd 100644
--- a/src/backend/catalog/aclchk.c
+++ b/src/backend/catalog/aclchk.c
@@ -38,6 +38,7 @@
#include "catalog/pg_language.h"
#include "catalog/pg_largeobject.h"
#include "catalog/pg_largeobject_metadata.h"
+#include "catalog/pg_mv_statistic.h"
#include "catalog/pg_namespace.h"
#include "catalog/pg_opclass.h"
#include "catalog/pg_operator.h"
@@ -5021,6 +5022,32 @@ pg_extension_ownercheck(Oid ext_oid, Oid roleid)
}
/*
+ * Ownership check for a multivariate statistics (specified by OID).
+ */
+bool
+pg_statistics_ownercheck(Oid stat_oid, Oid roleid)
+{
+ HeapTuple tuple;
+ Oid ownerId;
+
+ /* Superusers bypass all permission checking. */
+ if (superuser_arg(roleid))
+ return true;
+
+ tuple = SearchSysCache1(MVSTATOID, ObjectIdGetDatum(stat_oid));
+ if (!HeapTupleIsValid(tuple))
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("statistics with OID %u does not exist", stat_oid)));
+
+ ownerId = ((Form_pg_mv_statistic) GETSTRUCT(tuple))->staowner;
+
+ ReleaseSysCache(tuple);
+
+ return has_privs_of_role(roleid, ownerId);
+}
+
+/*
* Check whether specified role has CREATEROLE privilege (or is a superuser)
*
* Note: roles do not have owners per se; instead we use this test in
diff --git a/src/backend/catalog/dependency.c b/src/backend/catalog/dependency.c
index c48e37b..8200454 100644
--- a/src/backend/catalog/dependency.c
+++ b/src/backend/catalog/dependency.c
@@ -40,6 +40,7 @@
#include "catalog/pg_foreign_server.h"
#include "catalog/pg_language.h"
#include "catalog/pg_largeobject.h"
+#include "catalog/pg_mv_statistic.h"
#include "catalog/pg_namespace.h"
#include "catalog/pg_opclass.h"
#include "catalog/pg_operator.h"
@@ -160,7 +161,8 @@ static const Oid object_classes[] = {
ExtensionRelationId, /* OCLASS_EXTENSION */
EventTriggerRelationId, /* OCLASS_EVENT_TRIGGER */
PolicyRelationId, /* OCLASS_POLICY */
- TransformRelationId /* OCLASS_TRANSFORM */
+ TransformRelationId, /* OCLASS_TRANSFORM */
+ MvStatisticRelationId /* OCLASS_STATISTICS */
};
@@ -1272,6 +1274,10 @@ doDeletion(const ObjectAddress *object, int flags)
DropTransformById(object->objectId);
break;
+ case OCLASS_STATISTICS:
+ RemoveStatisticsById(object->objectId);
+ break;
+
default:
elog(ERROR, "unrecognized object class: %u",
object->classId);
@@ -2415,6 +2421,9 @@ getObjectClass(const ObjectAddress *object)
case TransformRelationId:
return OCLASS_TRANSFORM;
+
+ case MvStatisticRelationId:
+ return OCLASS_STATISTICS;
}
/* shouldn't get here */
diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c
index e997b57..47ec8cc 100644
--- a/src/backend/catalog/heap.c
+++ b/src/backend/catalog/heap.c
@@ -47,6 +47,7 @@
#include "catalog/pg_constraint_fn.h"
#include "catalog/pg_foreign_table.h"
#include "catalog/pg_inherits.h"
+#include "catalog/pg_mv_statistic.h"
#include "catalog/pg_namespace.h"
#include "catalog/pg_statistic.h"
#include "catalog/pg_tablespace.h"
@@ -1613,7 +1614,10 @@ RemoveAttributeById(Oid relid, AttrNumber attnum)
heap_close(attr_rel, RowExclusiveLock);
if (attnum > 0)
+ {
RemoveStatistics(relid, attnum);
+ RemoveMVStatistics(relid, attnum);
+ }
relation_close(rel, NoLock);
}
@@ -1841,6 +1845,11 @@ heap_drop_with_catalog(Oid relid)
RemoveStatistics(relid, 0);
/*
+ * delete multi-variate statistics
+ */
+ RemoveMVStatistics(relid, 0);
+
+ /*
* delete attribute tuples
*/
DeleteAttributeTuples(relid);
@@ -2692,6 +2701,99 @@ RemoveStatistics(Oid relid, AttrNumber attnum)
/*
+ * RemoveMVStatistics --- remove entries in pg_mv_statistic for a rel
+ *
+ * If attnum is zero, remove all entries for rel; else remove only the one(s)
+ * for that column.
+ */
+void
+RemoveMVStatistics(Oid relid, AttrNumber attnum)
+{
+ Relation pgmvstatistic;
+ TupleDesc tupdesc = NULL;
+ SysScanDesc scan;
+ ScanKeyData key;
+ HeapTuple tuple;
+
+ /*
+ * When dropping a column, we'll drop statistics with a single
+ * remaining (undropped column). To do that, we need the tuple
+ * descriptor.
+ *
+ * We already have the relation locked (as we're running ALTER
+ * TABLE ... DROP COLUMN), so we'll just get the descriptor here.
+ */
+ if (attnum != 0)
+ {
+ Relation rel = relation_open(relid, NoLock);
+
+ /* multivariate stats are supported on tables and matviews */
+ if (rel->rd_rel->relkind == RELKIND_RELATION ||
+ rel->rd_rel->relkind == RELKIND_MATVIEW)
+ tupdesc = RelationGetDescr(rel);
+
+ relation_close(rel, NoLock);
+ }
+
+ if (tupdesc == NULL)
+ return;
+
+ pgmvstatistic = heap_open(MvStatisticRelationId, RowExclusiveLock);
+
+ ScanKeyInit(&key,
+ Anum_pg_mv_statistic_starelid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(relid));
+
+ scan = systable_beginscan(pgmvstatistic,
+ MvStatisticRelidIndexId,
+ true, NULL, 1, &key);
+
+ /* we must loop even when attnum != 0, in case of inherited stats */
+ while (HeapTupleIsValid(tuple = systable_getnext(scan)))
+ {
+ bool delete = true;
+
+ if (attnum != 0)
+ {
+ Datum adatum;
+ bool isnull;
+ int i;
+ int ncolumns = 0;
+ ArrayType *arr;
+ int16 *attnums;
+
+ /* get the columns */
+ adatum = SysCacheGetAttr(MVSTATOID, tuple,
+ Anum_pg_mv_statistic_stakeys, &isnull);
+ Assert(!isnull);
+
+ arr = DatumGetArrayTypeP(adatum);
+ attnums = (int16*)ARR_DATA_PTR(arr);
+
+ for (i = 0; i < ARR_DIMS(arr)[0]; i++)
+ {
+ /* count the column unless it's has been / is being dropped */
+ if ((! tupdesc->attrs[attnums[i]-1]->attisdropped) &&
+ (attnums[i] != attnum))
+ ncolumns += 1;
+ }
+
+ /* delete if there are less than two attributes */
+ delete = (ncolumns < 2);
+ }
+
+ if (delete)
+ simple_heap_delete(pgmvstatistic, &tuple->t_self);
+ }
+
+ systable_endscan(scan);
+
+ heap_close(pgmvstatistic, RowExclusiveLock);
+}
+
+
+/*
* RelationTruncateIndexes - truncate all indexes associated
* with the heap relation to zero tuples.
*
diff --git a/src/backend/catalog/namespace.c b/src/backend/catalog/namespace.c
index 446b2ac..dfd5bef 100644
--- a/src/backend/catalog/namespace.c
+++ b/src/backend/catalog/namespace.c
@@ -4201,3 +4201,54 @@ pg_is_other_temp_schema(PG_FUNCTION_ARGS)
PG_RETURN_BOOL(isOtherTempNamespace(oid));
}
+
+Oid
+get_statistics_oid(List *names, bool missing_ok)
+{
+ char *schemaname;
+ char *stats_name;
+ Oid namespaceId;
+ Oid stats_oid = InvalidOid;
+ ListCell *l;
+
+ /* deconstruct the name list */
+ DeconstructQualifiedName(names, &schemaname, &stats_name);
+
+ if (schemaname)
+ {
+ /* use exact schema given */
+ namespaceId = LookupExplicitNamespace(schemaname, missing_ok);
+ if (missing_ok && !OidIsValid(namespaceId))
+ stats_oid = InvalidOid;
+ else
+ stats_oid = GetSysCacheOid2(MVSTATNAMENSP,
+ PointerGetDatum(stats_name),
+ ObjectIdGetDatum(namespaceId));
+ }
+ else
+ {
+ /* search for it in search path */
+ recomputeNamespacePath();
+
+ foreach(l, activeSearchPath)
+ {
+ namespaceId = lfirst_oid(l);
+
+ if (namespaceId == myTempNamespace)
+ continue; /* do not look in temp namespace */
+ stats_oid = GetSysCacheOid2(MVSTATNAMENSP,
+ PointerGetDatum(stats_name),
+ ObjectIdGetDatum(namespaceId));
+ if (OidIsValid(stats_oid))
+ break;
+ }
+ }
+
+ if (!OidIsValid(stats_oid) && !missing_ok)
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("statistics \"%s\" does not exist",
+ NameListToString(names))));
+
+ return stats_oid;
+}
diff --git a/src/backend/catalog/objectaddress.c b/src/backend/catalog/objectaddress.c
index d2aaa6d..c13a569 100644
--- a/src/backend/catalog/objectaddress.c
+++ b/src/backend/catalog/objectaddress.c
@@ -39,6 +39,7 @@
#include "catalog/pg_language.h"
#include "catalog/pg_largeobject.h"
#include "catalog/pg_largeobject_metadata.h"
+#include "catalog/pg_mv_statistic.h"
#include "catalog/pg_namespace.h"
#include "catalog/pg_opclass.h"
#include "catalog/pg_opfamily.h"
@@ -438,9 +439,22 @@ static const ObjectPropertyType ObjectProperty[] =
Anum_pg_type_typacl,
ACL_KIND_TYPE,
true
+ },
+ {
+ MvStatisticRelationId,
+ MvStatisticOidIndexId,
+ MVSTATOID,
+ MVSTATNAMENSP,
+ Anum_pg_mv_statistic_staname,
+ Anum_pg_mv_statistic_stanamespace,
+ Anum_pg_mv_statistic_staowner,
+ InvalidAttrNumber, /* no ACL (same as relation) */
+ -1, /* no ACL */
+ true
}
};
+
/*
* This struct maps the string object types as returned by
* getObjectTypeDescription into ObjType enum values. Note that some enum
@@ -640,6 +654,10 @@ static const struct object_type_map
/* OCLASS_TRANSFORM */
{
"transform", OBJECT_TRANSFORM
+ },
+ /* OBJECT_STATISTICS */
+ {
+ "statistics", OBJECT_STATISTICS
}
};
@@ -913,6 +931,11 @@ get_object_address(ObjectType objtype, List *objname, List *objargs,
address = get_object_address_defacl(objname, objargs,
missing_ok);
break;
+ case OBJECT_STATISTICS:
+ address.classId = MvStatisticRelationId;
+ address.objectId = get_statistics_oid(objname, missing_ok);
+ address.objectSubId = 0;
+ break;
default:
elog(ERROR, "unrecognized objtype: %d", (int) objtype);
/* placate compiler, in case it thinks elog might return */
@@ -2185,6 +2208,10 @@ check_object_ownership(Oid roleid, ObjectType objtype, ObjectAddress address,
(errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
errmsg("must be superuser")));
break;
+ case OBJECT_STATISTICS:
+ if (!pg_statistics_ownercheck(address.objectId, roleid))
+ aclcheck_error_type(ACLCHECK_NOT_OWNER, address.objectId);
+ break;
default:
elog(ERROR, "unrecognized object type: %d",
(int) objtype);
@@ -3610,6 +3637,10 @@ getObjectTypeDescription(const ObjectAddress *object)
appendStringInfoString(&buffer, "transform");
break;
+ case OCLASS_STATISTICS:
+ appendStringInfoString(&buffer, "statistics");
+ break;
+
default:
appendStringInfo(&buffer, "unrecognized %u", object->classId);
break;
@@ -4566,6 +4597,29 @@ getObjectIdentityParts(const ObjectAddress *object,
}
break;
+ case OCLASS_STATISTICS:
+ {
+ HeapTuple tup;
+ Form_pg_mv_statistic formStatistic;
+ char *schema;
+
+ tup = SearchSysCache1(MVSTATOID,
+ ObjectIdGetDatum(object->objectId));
+ if (!HeapTupleIsValid(tup))
+ elog(ERROR, "cache lookup failed for statistics %u",
+ object->objectId);
+ formStatistic = (Form_pg_mv_statistic) GETSTRUCT(tup);
+ schema = get_namespace_name_or_temp(formStatistic->stanamespace);
+ appendStringInfoString(&buffer,
+ quote_qualified_identifier(schema,
+ NameStr(formStatistic->staname)));
+ if (objname)
+ *objname = list_make2(schema,
+ pstrdup(NameStr(formStatistic->staname)));
+ ReleaseSysCache(tup);
+ break;
+ }
+
default:
appendStringInfo(&buffer, "unrecognized object %u %u %d",
object->classId,
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 84aa061..31dbb2c 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -158,6 +158,17 @@ CREATE VIEW pg_indexes AS
LEFT JOIN pg_tablespace T ON (T.oid = I.reltablespace)
WHERE C.relkind IN ('r', 'm') AND I.relkind = 'i';
+CREATE VIEW pg_mv_stats AS
+ SELECT
+ N.nspname AS schemaname,
+ C.relname AS tablename,
+ S.staname AS staname,
+ S.stakeys AS attnums,
+ length(S.stadeps) as depsbytes,
+ pg_mv_stats_dependencies_info(S.stadeps) as depsinfo
+ FROM (pg_mv_statistic S JOIN pg_class C ON (C.oid = S.starelid))
+ LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace);
+
CREATE VIEW pg_stats WITH (security_barrier) AS
SELECT
nspname AS schemaname,
diff --git a/src/backend/commands/Makefile b/src/backend/commands/Makefile
index b1ac704..5151001 100644
--- a/src/backend/commands/Makefile
+++ b/src/backend/commands/Makefile
@@ -18,8 +18,8 @@ OBJS = aggregatecmds.o alter.o analyze.o async.o cluster.o comment.o \
event_trigger.o explain.o extension.o foreigncmds.o functioncmds.o \
indexcmds.o lockcmds.o matview.o operatorcmds.o opclasscmds.o \
policy.o portalcmds.o prepare.o proclang.o \
- schemacmds.o seclabel.o sequence.o tablecmds.o tablespace.o trigger.o \
- tsearchcmds.o typecmds.o user.o vacuum.o vacuumlazy.o \
- variable.o view.o
+ schemacmds.o seclabel.o sequence.o statscmds.o \
+ tablecmds.o tablespace.o trigger.o tsearchcmds.o typecmds.o \
+ user.o vacuum.o vacuumlazy.o variable.o view.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/commands/alter.c b/src/backend/commands/alter.c
index 5af0f2f..89985499 100644
--- a/src/backend/commands/alter.c
+++ b/src/backend/commands/alter.c
@@ -359,6 +359,7 @@ ExecRenameStmt(RenameStmt *stmt)
case OBJECT_OPCLASS:
case OBJECT_OPFAMILY:
case OBJECT_LANGUAGE:
+ case OBJECT_STATISTICS:
case OBJECT_TSCONFIGURATION:
case OBJECT_TSDICTIONARY:
case OBJECT_TSPARSER:
@@ -437,6 +438,7 @@ ExecAlterObjectSchemaStmt(AlterObjectSchemaStmt *stmt,
case OBJECT_OPERATOR:
case OBJECT_OPCLASS:
case OBJECT_OPFAMILY:
+ case OBJECT_STATISTICS:
case OBJECT_TSCONFIGURATION:
case OBJECT_TSDICTIONARY:
case OBJECT_TSPARSER:
@@ -745,6 +747,7 @@ ExecAlterOwnerStmt(AlterOwnerStmt *stmt)
case OBJECT_OPERATOR:
case OBJECT_OPCLASS:
case OBJECT_OPFAMILY:
+ case OBJECT_STATISTICS:
case OBJECT_TABLESPACE:
case OBJECT_TSDICTIONARY:
case OBJECT_TSCONFIGURATION:
diff --git a/src/backend/commands/analyze.c b/src/backend/commands/analyze.c
index 8a5f07c..9087532 100644
--- a/src/backend/commands/analyze.c
+++ b/src/backend/commands/analyze.c
@@ -17,6 +17,7 @@
#include <math.h>
#include "access/multixact.h"
+#include "access/sysattr.h"
#include "access/transam.h"
#include "access/tupconvert.h"
#include "access/tuptoaster.h"
@@ -27,6 +28,7 @@
#include "catalog/indexing.h"
#include "catalog/pg_collation.h"
#include "catalog/pg_inherits_fn.h"
+#include "catalog/pg_mv_statistic.h"
#include "catalog/pg_namespace.h"
#include "commands/dbcommands.h"
#include "commands/tablecmds.h"
@@ -45,10 +47,13 @@
#include "storage/procarray.h"
#include "utils/acl.h"
#include "utils/attoptcache.h"
+#include "utils/builtins.h"
#include "utils/datum.h"
+#include "utils/fmgroids.h"
#include "utils/guc.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/mvstats.h"
#include "utils/pg_rusage.h"
#include "utils/sampling.h"
#include "utils/sortsupport.h"
@@ -460,6 +465,19 @@ do_analyze_rel(Relation onerel, int options, VacuumParams *params,
* all analyzable columns. We use a lower bound of 100 rows to avoid
* possible overflow in Vitter's algorithm. (Note: that will also be the
* target in the corner case where there are no analyzable columns.)
+ *
+ * FIXME This sample sizing is mostly OK when computing stats for
+ * individual columns, but when computing multi-variate stats
+ * for multivariate stats (histograms, mcv, ...) it's rather
+ * insufficient. For stats on multiple columns / complex stats
+ * we need larger sample sizes, because we need to build more
+ * detailed stats (more MCV items / histogram buckets) to get
+ * good accuracy. Maybe it'd be appropriate to use samples
+ * proportional to the table (say, 0.5% - 1%) instead of a
+ * fixed size might be more appropriate. Also, this should be
+ * bound to the requested statistics size - e.g. number of MCV
+ * items or histogram buckets should require several sample
+ * rows per item/bucket (so the sample should be k*size).
*/
targrows = 100;
for (i = 0; i < attr_cnt; i++)
@@ -562,6 +580,9 @@ do_analyze_rel(Relation onerel, int options, VacuumParams *params,
update_attstats(RelationGetRelid(Irel[ind]), false,
thisdata->attr_cnt, thisdata->vacattrstats);
}
+
+ /* Build multivariate stats (if there are any). */
+ build_mv_stats(onerel, numrows, rows, attr_cnt, vacattrstats);
}
/*
diff --git a/src/backend/commands/dropcmds.c b/src/backend/commands/dropcmds.c
index 522027a..cd65b58 100644
--- a/src/backend/commands/dropcmds.c
+++ b/src/backend/commands/dropcmds.c
@@ -292,6 +292,10 @@ does_not_exist_skipping(ObjectType objtype, List *objname, List *objargs)
msg = gettext_noop("schema \"%s\" does not exist, skipping");
name = NameListToString(objname);
break;
+ case OBJECT_STATISTICS:
+ msg = gettext_noop("statistics \"%s\" does not exist, skipping");
+ name = NameListToString(objname);
+ break;
case OBJECT_TSPARSER:
if (!schema_does_not_exist_skipping(objname, &msg, &name))
{
diff --git a/src/backend/commands/event_trigger.c b/src/backend/commands/event_trigger.c
index 9e32f8d..09061bb 100644
--- a/src/backend/commands/event_trigger.c
+++ b/src/backend/commands/event_trigger.c
@@ -110,6 +110,7 @@ static event_trigger_support_data event_trigger_support[] = {
{"SCHEMA", true},
{"SEQUENCE", true},
{"SERVER", true},
+ {"STATISTICS", true},
{"TABLE", true},
{"TABLESPACE", false},
{"TRANSFORM", true},
@@ -1106,6 +1107,7 @@ EventTriggerSupportsObjectType(ObjectType obtype)
case OBJECT_RULE:
case OBJECT_SCHEMA:
case OBJECT_SEQUENCE:
+ case OBJECT_STATISTICS:
case OBJECT_TABCONSTRAINT:
case OBJECT_TABLE:
case OBJECT_TRANSFORM:
@@ -1167,6 +1169,7 @@ EventTriggerSupportsObjectClass(ObjectClass objclass)
case OCLASS_DEFACL:
case OCLASS_EXTENSION:
case OCLASS_POLICY:
+ case OCLASS_STATISTICS:
return true;
}
diff --git a/src/backend/commands/statscmds.c b/src/backend/commands/statscmds.c
new file mode 100644
index 0000000..f43b053
--- /dev/null
+++ b/src/backend/commands/statscmds.c
@@ -0,0 +1,277 @@
+/*-------------------------------------------------------------------------
+ *
+ * statscmds.c
+ * Commands for creating and altering multivariate statistics
+ *
+ * Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/commands/statscmds.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/relscan.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/namespace.h"
+#include "catalog/pg_mv_statistic.h"
+#include "catalog/pg_namespace.h"
+#include "commands/defrem.h"
+#include "miscadmin.h"
+#include "utils/builtins.h"
+#include "utils/inval.h"
+#include "utils/memutils.h"
+#include "utils/mvstats.h"
+#include "utils/rel.h"
+#include "utils/syscache.h"
+
+
+/* used for sorting the attnums in ExecCreateStatistics */
+static int compare_int16(const void *a, const void *b)
+{
+ return memcmp(a, b, sizeof(int16));
+}
+
+/*
+ * Implements the CREATE STATISTICS name ON table (columns) WITH (options)
+ *
+ * TODO Check that the types support sort, although maybe we can live
+ * without it (and only build MCV list / association rules).
+ *
+ * TODO This should probably check for duplicate stats (i.e. same
+ * keys, same options). Although maybe it's useful to have
+ * multiple stats on the same columns with different options
+ * (say, a detailed MCV-only stats for some queries, histogram
+ * for others, etc.)
+ */
+ObjectAddress
+CreateStatistics(CreateStatsStmt *stmt)
+{
+ int i, j;
+ ListCell *l;
+ int16 attnums[INDEX_MAX_KEYS];
+ int numcols = 0;
+ ObjectAddress address = InvalidObjectAddress;
+ char *namestr;
+ NameData staname;
+ Oid statoid;
+ Oid namespaceId;
+
+ HeapTuple htup;
+ Datum values[Natts_pg_mv_statistic];
+ bool nulls[Natts_pg_mv_statistic];
+ int2vector *stakeys;
+ Relation mvstatrel;
+ Relation rel;
+ ObjectAddress parentobject, childobject;
+
+ /* by default build nothing */
+ bool build_dependencies = false;
+
+ Assert(IsA(stmt, CreateStatsStmt));
+
+ /* resolve the pieces of the name (namespace etc.) */
+ namespaceId = QualifiedNameGetCreationNamespace(stmt->defnames, &namestr);
+ namestrcpy(&staname, namestr);
+
+ /*
+ * If if_not_exists was given and the statistics already exists, bail out.
+ */
+ if (stmt->if_not_exists &&
+ SearchSysCacheExists2(MVSTATNAMENSP,
+ PointerGetDatum(&staname),
+ ObjectIdGetDatum(namespaceId)))
+ {
+ ereport(NOTICE,
+ (errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("statistics \"%s\" already exists, skipping",
+ namestr)));
+ return InvalidObjectAddress;
+ }
+
+ rel = heap_openrv(stmt->relation, AccessExclusiveLock);
+
+ /* transform the column names to attnum values */
+
+ foreach(l, stmt->keys)
+ {
+ char *attname = strVal(lfirst(l));
+ HeapTuple atttuple;
+
+ atttuple = SearchSysCacheAttName(RelationGetRelid(rel), attname);
+
+ if (!HeapTupleIsValid(atttuple))
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_COLUMN),
+ errmsg("column \"%s\" referenced in statistics does not exist",
+ attname)));
+
+ /* more than MVHIST_MAX_DIMENSIONS columns not allowed */
+ if (numcols >= MVSTATS_MAX_DIMENSIONS)
+ ereport(ERROR,
+ (errcode(ERRCODE_TOO_MANY_COLUMNS),
+ errmsg("cannot have more than %d keys in a statistics",
+ MVSTATS_MAX_DIMENSIONS)));
+
+ attnums[numcols] = ((Form_pg_attribute) GETSTRUCT(atttuple))->attnum;
+ ReleaseSysCache(atttuple);
+ numcols++;
+ }
+
+ /*
+ * Check the lower bound (at least 2 columns), the upper bound was
+ * already checked in the loop.
+ */
+ if (numcols < 2)
+ ereport(ERROR,
+ (errcode(ERRCODE_TOO_MANY_COLUMNS),
+ errmsg("multivariate stats require 2 or more columns")));
+
+ /* look for duplicities */
+ for (i = 0; i < numcols; i++)
+ for (j = 0; j < numcols; j++)
+ if ((i != j) && (attnums[i] == attnums[j]))
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_COLUMN),
+ errmsg("duplicate column name in statistics definition")));
+
+ /* parse the statistics options */
+ foreach (l, stmt->options)
+ {
+ DefElem *opt = (DefElem*)lfirst(l);
+
+ if (strcmp(opt->defname, "dependencies") == 0)
+ build_dependencies = defGetBoolean(opt);
+ else
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("unrecognized STATISTICS option \"%s\"",
+ opt->defname)));
+ }
+
+ /* check that at least some statistics were requested */
+ if (! build_dependencies)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("no statistics type (dependencies) was requested")));
+
+ /* sort the attnums and build int2vector */
+ qsort(attnums, numcols, sizeof(int16), compare_int16);
+ stakeys = buildint2vector(attnums, numcols);
+
+ /*
+ * Okay, let's create the pg_mv_statistic entry.
+ */
+ memset(values, 0, sizeof(values));
+ memset(nulls, false, sizeof(nulls));
+
+ /* no stats collected yet, so just the keys */
+ values[Anum_pg_mv_statistic_starelid-1] = ObjectIdGetDatum(RelationGetRelid(rel));
+ values[Anum_pg_mv_statistic_staname -1] = NameGetDatum(&staname);
+ values[Anum_pg_mv_statistic_stanamespace -1] = ObjectIdGetDatum(namespaceId);
+ values[Anum_pg_mv_statistic_staowner-1] = ObjectIdGetDatum(GetUserId());
+
+ values[Anum_pg_mv_statistic_stakeys -1] = PointerGetDatum(stakeys);
+
+ values[Anum_pg_mv_statistic_deps_enabled -1] = BoolGetDatum(build_dependencies);
+
+ nulls[Anum_pg_mv_statistic_stadeps -1] = true;
+
+ /* insert the tuple into pg_mv_statistic */
+ mvstatrel = heap_open(MvStatisticRelationId, RowExclusiveLock);
+
+ htup = heap_form_tuple(mvstatrel->rd_att, values, nulls);
+
+ simple_heap_insert(mvstatrel, htup);
+
+ CatalogUpdateIndexes(mvstatrel, htup);
+
+ statoid = HeapTupleGetOid(htup);
+
+ heap_freetuple(htup);
+
+
+ /*
+ * Store a dependency too, so that statistics are dropped on DROP TABLE
+ */
+ parentobject.classId = RelationRelationId;
+ parentobject.objectId = ObjectIdGetDatum(RelationGetRelid(rel));
+ parentobject.objectSubId = 0;
+ childobject.classId = MvStatisticRelationId;
+ childobject.objectId = statoid;
+ childobject.objectSubId = 0;
+
+ recordDependencyOn(&childobject, &parentobject, DEPENDENCY_AUTO);
+
+ /*
+ * Also record dependency on the schema (to drop statistics on DROP SCHEMA)
+ */
+ parentobject.classId = NamespaceRelationId;
+ parentobject.objectId = ObjectIdGetDatum(namespaceId);
+ parentobject.objectSubId = 0;
+ childobject.classId = MvStatisticRelationId;
+ childobject.objectId = statoid;
+ childobject.objectSubId = 0;
+
+ recordDependencyOn(&childobject, &parentobject, DEPENDENCY_AUTO);
+
+
+ heap_close(mvstatrel, RowExclusiveLock);
+
+ relation_close(rel, NoLock);
+
+ /*
+ * Invalidate relcache so that others see the new statistics.
+ */
+ CacheInvalidateRelcache(rel);
+
+ ObjectAddressSet(address, MvStatisticRelationId, statoid);
+
+ return address;
+}
+
+
+/*
+ * Implements the DROP STATISTICS
+ *
+ * DROP STATISTICS stats_name ON table_name
+ *
+ * The first one requires an exact match, the second one just drops
+ * all the statistics on a table.
+ */
+void
+RemoveStatisticsById(Oid statsOid)
+{
+ Relation relation;
+ Oid relid;
+ Relation rel;
+ HeapTuple tup;
+ Form_pg_mv_statistic mvstat;
+
+ /*
+ * Delete the pg_proc tuple.
+ */
+ relation = heap_open(MvStatisticRelationId, RowExclusiveLock);
+
+ tup = SearchSysCache1(MVSTATOID, ObjectIdGetDatum(statsOid));
+ if (!HeapTupleIsValid(tup)) /* should not happen */
+ elog(ERROR, "cache lookup failed for statistics %u", statsOid);
+
+ mvstat = (Form_pg_mv_statistic) GETSTRUCT(tup);
+ relid = mvstat->starelid;
+
+ rel = heap_open(relid, AccessExclusiveLock);
+
+ simple_heap_delete(relation, &tup->t_self);
+
+ CacheInvalidateRelcache(rel);
+
+ ReleaseSysCache(tup);
+
+ heap_close(relation, RowExclusiveLock);
+ heap_close(rel, NoLock);
+}
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index df7c2fa..3b7c87f 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -4124,6 +4124,20 @@ _copyAlterPolicyStmt(const AlterPolicyStmt *from)
return newnode;
}
+static CreateStatsStmt *
+_copyCreateStatsStmt(const CreateStatsStmt *from)
+{
+ CreateStatsStmt *newnode = makeNode(CreateStatsStmt);
+
+ COPY_NODE_FIELD(defnames);
+ COPY_NODE_FIELD(relation);
+ COPY_NODE_FIELD(keys);
+ COPY_NODE_FIELD(options);
+ COPY_SCALAR_FIELD(if_not_exists);
+
+ return newnode;
+}
+
/* ****************************************************************
* pg_list.h copy functions
* ****************************************************************
@@ -4999,6 +5013,9 @@ copyObject(const void *from)
case T_CommonTableExpr:
retval = _copyCommonTableExpr(from);
break;
+ case T_CreateStatsStmt:
+ retval = _copyCreateStatsStmt(from);
+ break;
case T_FuncWithArgs:
retval = _copyFuncWithArgs(from);
break;
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index eb0fc1e..07206d7 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2153,6 +2153,21 @@ _outIndexOptInfo(StringInfo str, const IndexOptInfo *node)
}
static void
+_outMVStatisticInfo(StringInfo str, const MVStatisticInfo *node)
+{
+ WRITE_NODE_TYPE("MVSTATISTICINFO");
+
+ /* NB: this isn't a complete set of fields */
+ WRITE_OID_FIELD(mvoid);
+
+ /* enabled statistics */
+ WRITE_BOOL_FIELD(deps_enabled);
+
+ /* built/available statistics */
+ WRITE_BOOL_FIELD(deps_built);
+}
+
+static void
_outEquivalenceClass(StringInfo str, const EquivalenceClass *node)
{
/*
@@ -3636,6 +3651,9 @@ _outNode(StringInfo str, const void *obj)
case T_PlannerParamItem:
_outPlannerParamItem(str, obj);
break;
+ case T_MVStatisticInfo:
+ _outMVStatisticInfo(str, obj);
+ break;
case T_ExtensibleNode:
_outExtensibleNode(str, obj);
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index ad715bb..7fb2088 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -28,6 +28,7 @@
#include "catalog/dependency.h"
#include "catalog/heap.h"
#include "catalog/pg_am.h"
+#include "catalog/pg_mv_statistic.h"
#include "foreign/fdwapi.h"
#include "miscadmin.h"
#include "nodes/makefuncs.h"
@@ -40,7 +41,9 @@
#include "parser/parsetree.h"
#include "rewrite/rewriteManip.h"
#include "storage/bufmgr.h"
+#include "utils/builtins.h"
#include "utils/lsyscache.h"
+#include "utils/syscache.h"
#include "utils/rel.h"
#include "utils/snapmgr.h"
@@ -94,6 +97,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
Relation relation;
bool hasindex;
List *indexinfos = NIL;
+ List *stainfos = NIL;
/*
* We need not lock the relation since it was already locked, either by
@@ -387,6 +391,61 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
rel->indexlist = indexinfos;
+ if (true)
+ {
+ List *mvstatoidlist;
+ ListCell *l;
+
+ mvstatoidlist = RelationGetMVStatList(relation);
+
+ foreach(l, mvstatoidlist)
+ {
+ ArrayType *arr;
+ Datum adatum;
+ bool isnull;
+ Oid mvoid = lfirst_oid(l);
+ Form_pg_mv_statistic mvstat;
+ MVStatisticInfo *info;
+
+ HeapTuple htup = SearchSysCache1(MVSTATOID, ObjectIdGetDatum(mvoid));
+
+ mvstat = (Form_pg_mv_statistic) GETSTRUCT(htup);
+
+ /* unavailable stats are not interesting for the planner */
+ if (mvstat->deps_built)
+ {
+ info = makeNode(MVStatisticInfo);
+
+ info->mvoid = mvoid;
+ info->rel = rel;
+
+ /* enabled statistics */
+ info->deps_enabled = mvstat->deps_enabled;
+
+ /* built/available statistics */
+ info->deps_built = mvstat->deps_built;
+
+ /* stakeys */
+ adatum = SysCacheGetAttr(MVSTATOID, htup,
+ Anum_pg_mv_statistic_stakeys, &isnull);
+ Assert(!isnull);
+
+ arr = DatumGetArrayTypeP(adatum);
+
+ info->stakeys = buildint2vector((int16 *) ARR_DATA_PTR(arr),
+ ARR_DIMS(arr)[0]);
+
+ stainfos = lcons(info, stainfos);
+ }
+
+ ReleaseSysCache(htup);
+ }
+
+ list_free(mvstatoidlist);
+ }
+
+ rel->mvstatlist = stainfos;
+
/* Grab foreign-table info using the relcache, while we have it */
if (relation->rd_rel->relkind == RELKIND_FOREIGN_TABLE)
{
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index b9aeb31..eed9927 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -241,7 +241,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
ConstraintsSetStmt CopyStmt CreateAsStmt CreateCastStmt
CreateDomainStmt CreateExtensionStmt CreateGroupStmt CreateOpClassStmt
CreateOpFamilyStmt AlterOpFamilyStmt CreatePLangStmt
- CreateSchemaStmt CreateSeqStmt CreateStmt CreateTableSpaceStmt
+ CreateSchemaStmt CreateSeqStmt CreateStmt CreateStatsStmt CreateTableSpaceStmt
CreateFdwStmt CreateForeignServerStmt CreateForeignTableStmt
CreateAssertStmt CreateTransformStmt CreateTrigStmt CreateEventTrigStmt
CreateUserStmt CreateUserMappingStmt CreateRoleStmt CreatePolicyStmt
@@ -809,6 +809,7 @@ stmt :
| CreateSchemaStmt
| CreateSeqStmt
| CreateStmt
+ | CreateStatsStmt
| CreateTableSpaceStmt
| CreateTransformStmt
| CreateTrigStmt
@@ -3436,6 +3437,36 @@ OptConsTableSpace: USING INDEX TABLESPACE name { $$ = $4; }
ExistingIndex: USING INDEX index_name { $$ = $3; }
;
+/*****************************************************************************
+ *
+ * QUERY :
+ * CREATE STATISTICS stats_name ON relname (columns) WITH (options)
+ *
+ *****************************************************************************/
+
+
+CreateStatsStmt: CREATE STATISTICS any_name ON qualified_name '(' columnList ')' opt_reloptions
+ {
+ CreateStatsStmt *n = makeNode(CreateStatsStmt);
+ n->defnames = $3;
+ n->relation = $5;
+ n->keys = $7;
+ n->options = $9;
+ n->if_not_exists = false;
+ $$ = (Node *)n;
+ }
+ | CREATE STATISTICS IF_P NOT EXISTS any_name ON qualified_name '(' columnList ')' opt_reloptions
+ {
+ CreateStatsStmt *n = makeNode(CreateStatsStmt);
+ n->defnames = $6;
+ n->relation = $8;
+ n->keys = $10;
+ n->options = $12;
+ n->if_not_exists = true;
+ $$ = (Node *)n;
+ }
+ ;
+
/*****************************************************************************
*
@@ -5621,6 +5652,7 @@ drop_type: TABLE { $$ = OBJECT_TABLE; }
| TEXT_P SEARCH DICTIONARY { $$ = OBJECT_TSDICTIONARY; }
| TEXT_P SEARCH TEMPLATE { $$ = OBJECT_TSTEMPLATE; }
| TEXT_P SEARCH CONFIGURATION { $$ = OBJECT_TSCONFIGURATION; }
+ | STATISTICS { $$ = OBJECT_STATISTICS; }
;
any_name_list:
@@ -7995,6 +8027,15 @@ RenameStmt: ALTER AGGREGATE func_name aggr_args RENAME TO name
n->missing_ok = false;
$$ = (Node *)n;
}
+ | ALTER STATISTICS any_name RENAME TO name
+ {
+ RenameStmt *n = makeNode(RenameStmt);
+ n->renameType = OBJECT_STATISTICS;
+ n->object = $3;
+ n->newname = $6;
+ n->missing_ok = false;
+ $$ = (Node *)n;
+ }
;
opt_column: COLUMN { $$ = COLUMN; }
@@ -8231,6 +8272,15 @@ AlterObjectSchemaStmt:
n->missing_ok = false;
$$ = (Node *)n;
}
+ | ALTER STATISTICS any_name SET SCHEMA name
+ {
+ AlterObjectSchemaStmt *n = makeNode(AlterObjectSchemaStmt);
+ n->objectType = OBJECT_STATISTICS;
+ n->object = $3;
+ n->newschema = $6;
+ n->missing_ok = false;
+ $$ = (Node *)n;
+ }
;
/*****************************************************************************
@@ -8421,6 +8471,14 @@ AlterOwnerStmt: ALTER AGGREGATE func_name aggr_args OWNER TO RoleSpec
n->newowner = $7;
$$ = (Node *)n;
}
+ | ALTER STATISTICS name OWNER TO RoleSpec
+ {
+ AlterOwnerStmt *n = makeNode(AlterOwnerStmt);
+ n->objectType = OBJECT_STATISTICS;
+ n->object = list_make1(makeString($3));
+ n->newowner = $6;
+ $$ = (Node *)n;
+ }
;
diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c
index 045f7f0..96b58f8 100644
--- a/src/backend/tcop/utility.c
+++ b/src/backend/tcop/utility.c
@@ -1520,6 +1520,10 @@ ProcessUtilitySlow(Node *parsetree,
address = ExecSecLabelStmt((SecLabelStmt *) parsetree);
break;
+ case T_CreateStatsStmt: /* CREATE STATISTICS */
+ address = CreateStatistics((CreateStatsStmt *) parsetree);
+ break;
+
default:
elog(ERROR, "unrecognized node type: %d",
(int) nodeTag(parsetree));
@@ -1878,6 +1882,9 @@ AlterObjectTypeCommandTag(ObjectType objtype)
case OBJECT_MATVIEW:
tag = "ALTER MATERIALIZED VIEW";
break;
+ case OBJECT_STATISTICS:
+ tag = "ALTER STATISTICS";
+ break;
default:
tag = "???";
break;
@@ -2160,6 +2167,9 @@ CreateCommandTag(Node *parsetree)
case OBJECT_TRANSFORM:
tag = "DROP TRANSFORM";
break;
+ case OBJECT_STATISTICS:
+ tag = "DROP STATISTICS";
+ break;
default:
tag = "???";
}
@@ -2527,6 +2537,10 @@ CreateCommandTag(Node *parsetree)
tag = "EXECUTE";
break;
+ case T_CreateStatsStmt:
+ tag = "CREATE STATISTICS";
+ break;
+
case T_DeallocateStmt:
{
DeallocateStmt *stmt = (DeallocateStmt *) parsetree;
diff --git a/src/backend/utils/Makefile b/src/backend/utils/Makefile
index 8374533..eba0352 100644
--- a/src/backend/utils/Makefile
+++ b/src/backend/utils/Makefile
@@ -9,7 +9,7 @@ top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
OBJS = fmgrtab.o
-SUBDIRS = adt cache error fmgr hash init mb misc mmgr resowner sort time
+SUBDIRS = adt cache error fmgr hash init mb misc mmgr mvstats resowner sort time
# location of Catalog.pm
catalogdir = $(top_srcdir)/src/backend/catalog
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 130c06d..3bc4c8a 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -47,6 +47,7 @@
#include "catalog/pg_auth_members.h"
#include "catalog/pg_constraint.h"
#include "catalog/pg_database.h"
+#include "catalog/pg_mv_statistic.h"
#include "catalog/pg_namespace.h"
#include "catalog/pg_opclass.h"
#include "catalog/pg_proc.h"
@@ -3956,6 +3957,62 @@ RelationGetIndexList(Relation relation)
return result;
}
+
+List *
+RelationGetMVStatList(Relation relation)
+{
+ Relation indrel;
+ SysScanDesc indscan;
+ ScanKeyData skey;
+ HeapTuple htup;
+ List *result;
+ List *oldlist;
+ MemoryContext oldcxt;
+
+ /* Quick exit if we already computed the list. */
+ if (relation->rd_mvstatvalid != 0)
+ return list_copy(relation->rd_mvstatlist);
+
+ /*
+ * We build the list we intend to return (in the caller's context) while
+ * doing the scan. After successfully completing the scan, we copy that
+ * list into the relcache entry. This avoids cache-context memory leakage
+ * if we get some sort of error partway through.
+ */
+ result = NIL;
+
+ /* Prepare to scan pg_index for entries having indrelid = this rel. */
+ ScanKeyInit(&skey,
+ Anum_pg_mv_statistic_starelid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationGetRelid(relation)));
+
+ indrel = heap_open(MvStatisticRelationId, AccessShareLock);
+ indscan = systable_beginscan(indrel, MvStatisticRelidIndexId, true,
+ NULL, 1, &skey);
+
+ while (HeapTupleIsValid(htup = systable_getnext(indscan)))
+ /* TODO maybe include only already built statistics? */
+ result = insert_ordered_oid(result, HeapTupleGetOid(htup));
+
+ systable_endscan(indscan);
+
+ heap_close(indrel, AccessShareLock);
+
+ /* Now save a copy of the completed list in the relcache entry. */
+ oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
+ oldlist = relation->rd_mvstatlist;
+ relation->rd_mvstatlist = list_copy(result);
+
+ relation->rd_mvstatvalid = true;
+ MemoryContextSwitchTo(oldcxt);
+
+ /* Don't leak the old list, if there is one */
+ list_free(oldlist);
+
+ return result;
+}
+
/*
* insert_ordered_oid
* Insert a new Oid into a sorted list of Oids, preserving ordering
@@ -4920,6 +4977,8 @@ load_relcache_init_file(bool shared)
rel->rd_indexattr = NULL;
rel->rd_keyattr = NULL;
rel->rd_idattr = NULL;
+ rel->rd_mvstatvalid = false;
+ rel->rd_mvstatlist = NIL;
rel->rd_createSubid = InvalidSubTransactionId;
rel->rd_newRelfilenodeSubid = InvalidSubTransactionId;
rel->rd_amcache = NULL;
diff --git a/src/backend/utils/cache/syscache.c b/src/backend/utils/cache/syscache.c
index 65ffe84..3c1bc4b 100644
--- a/src/backend/utils/cache/syscache.c
+++ b/src/backend/utils/cache/syscache.c
@@ -44,6 +44,7 @@
#include "catalog/pg_foreign_server.h"
#include "catalog/pg_foreign_table.h"
#include "catalog/pg_language.h"
+#include "catalog/pg_mv_statistic.h"
#include "catalog/pg_namespace.h"
#include "catalog/pg_opclass.h"
#include "catalog/pg_operator.h"
@@ -502,6 +503,28 @@ static const struct cachedesc cacheinfo[] = {
},
4
},
+ {MvStatisticRelationId, /* MVSTATNAMENSP */
+ MvStatisticNameIndexId,
+ 2,
+ {
+ Anum_pg_mv_statistic_staname,
+ Anum_pg_mv_statistic_stanamespace,
+ 0,
+ 0
+ },
+ 4
+ },
+ {MvStatisticRelationId, /* MVSTATOID */
+ MvStatisticOidIndexId,
+ 1,
+ {
+ ObjectIdAttributeNumber,
+ 0,
+ 0,
+ 0
+ },
+ 4
+ },
{NamespaceRelationId, /* NAMESPACENAME */
NamespaceNameIndexId,
1,
diff --git a/src/backend/utils/mvstats/Makefile b/src/backend/utils/mvstats/Makefile
new file mode 100644
index 0000000..099f1ed
--- /dev/null
+++ b/src/backend/utils/mvstats/Makefile
@@ -0,0 +1,17 @@
+#-------------------------------------------------------------------------
+#
+# Makefile--
+# Makefile for utils/mvstats
+#
+# IDENTIFICATION
+# src/backend/utils/mvstats/Makefile
+#
+#-------------------------------------------------------------------------
+
+subdir = src/backend/utils/mvstats
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+
+OBJS = common.o dependencies.o
+
+include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/utils/mvstats/README.dependencies b/src/backend/utils/mvstats/README.dependencies
new file mode 100644
index 0000000..1f96fbc
--- /dev/null
+++ b/src/backend/utils/mvstats/README.dependencies
@@ -0,0 +1,222 @@
+Soft functional dependencies
+============================
+
+A type of multivariate statistics used to capture cases when one column (or
+possibly a combination of columns) determines values in another column. We may
+also say that one column implies the other one.
+
+A simple artificial example may be a table with two columns, created like this
+
+ CREATE TABLE t (a INT, b INT)
+ AS SELECT i, i/10 FROM generate_series(1,100000) s(i);
+
+Clearly, once we know the value for column 'a' the value for 'b' is trivially
+determined, as it's simply (a/10). A more practical example may be addresses,
+where (ZIP code -> city name), i.e. once we know the ZIP, we probably know the
+city it belongs to, as ZIP codes are usually assigned to one city. Larger cities
+may have multiple ZIP codes, so the dependency can't be reversed.
+
+Functional dependencies are a concept well described in relational theory,
+particularly in definition of normalization and "normal forms". Wikipedia has a
+nice definition of a functional dependency [1]:
+
+ In a given table, an attribute Y is said to have a functional dependency on
+ a set of attributes X (written X -> Y) if and only if each X value is
+ associated with precisely one Y value. For example, in an "Employee" table
+ that includes the attributes "Employee ID" and "Employee Date of Birth", the
+ functional dependency {Employee ID} -> {Employee Date of Birth} would hold.
+ It follows from the previous two sentences that each {Employee ID} is
+ associated with precisely one {Employee Date of Birth}.
+
+ [1] http://en.wikipedia.org/wiki/Database_normalization
+
+Many datasets might be normalized not to contain such dependencies, but often
+it's not practical for various reasons. In some cases it's actually a conscious
+design choice to model the dataset in denormalized way, either because of
+performance or to make querying easier.
+
+The functional dependencies are called 'soft' because the implementation is
+meant to allow small number of rows contradicting the dependency. Many actual
+data sets contain some sort of errors, either because of data entry mistakes
+(user mistyping the ZIP code) or issues in generating the data (e.g. a ZIP code
+mistakenly assigned to two cities in different states). A strict implementation
+would ignore dependencies on such noisy data, rendering the approach unusable on
+such data sets.
+
+
+Mining dependencies (ANALYZE)
+-----------------------------
+
+The current build algorithm is rather simple - for each pair (a,b) of columns,
+the data are sorted lexicographically (first by 'a', then by 'b'). Then for each
+group (rows with the same 'a' value) we decide whether the group is neutral,
+supporting or contradicting the dependency (a->b).
+
+A group is considered neutral when it's too small - e.g. when there's a single
+row in the group, there can't possibly be multiple values in 'b'. For this
+reason we ignore groups smaller than a threshold (currently 3 rows).
+
+For sufficiently large groups (3 rows or more), we count the number of distinct
+values in 'b'. When there's a single 'b' value, the group is considered to
+support the dependency (a->b), otherwise it's condidered as contradicting it.
+
+At the end, we compare the number of rows in supporting and contradicting groups,
+and if there are at least 10x as many supporting rows, we consider the
+functional dependency to be valid.
+
+
+This approach has the negative property that the algorithm is that it's a bit
+fragile with respect to the sample - there may be data sets producing quite
+different results for each ANALYZE execution (as even a single row may change
+the outcome of the final 10x test).
+
+It was proposed to make the dependencies "fuzzy" - e.g. track some coefficient
+between [0,1] determining how much the dependency holds. That would however mean
+we have to keep all the dependencies, as eliminating them based on the value of
+the coefficient (e.g. throw away dependencies <= 0.5) would result in exactly
+the same fragility issues. This would also make it more complicated to combine
+dependencies. So this does not seem like a practical approach.
+
+A better approach might be to replace the constants (min_group_size=3 and 10x)
+with values somehow related to the particular data set.
+
+
+Clause reduction (planner/optimizer)
+------------------------------------
+
+Apllying the functional dependencies is quite simple - given a list of equality
+clauses, check which clauses are redundant (i.e. implied by some other clause).
+For example given clause list
+
+ (a = 2) AND (b = 2) AND (c = 3)
+
+and dependencies (a->b) and (a->d), the list of clauses may be simplified to
+
+ (a = 1) AND (c = 3)
+
+Functional dependencies may only be applied to equality clauses, all other types
+of clauses are ignored. See clauselist_apply_dependencies() for more details.
+
+
+Compatibility of clauses
+------------------------
+
+The reduction assumes the clauses really are redundant, and the value in the
+reduced clause (b=2) is the value determined by (a=1). If that's not the case
+and the values are "incompatible" the result will be over-estimation.
+
+This may happen for example when using conditions on ZIP and city name with
+mismatching values (ZIP for a different city), etc. In such case the result
+set will be empty, but we'll estimate the selectivity using the ZIP condition.
+
+In this case the default estimation based on AVIA principle happens to work
+better, but mostly by chance.
+
+
+Dependencies vs. MCV/histogram
+------------------------------
+
+In some cases the "compatibility" of the conditions might be verified using the
+other types of multivariate stats - MCV lists and histograms.
+
+For MCV lists the verification might be very simple - peek into the list if
+there are any items matching the clause on the 'a' column (e.g. ZIP code), and
+if such item is found, check that the 'b' column matches the other clause. If it
+does not, the clauses are contradictory. We can't really say if such item was
+not found, except maybe restricting the selectivity using the MCV data (e.g.
+using min/max selectivity, or something).
+
+With histograms, it might work similarly - we can't check the values directly
+(because histograms use buckets, unlike MCV lists, storing the actual values).
+So we can only observe the buckets matching the clauses - if those buckets have
+very low frequency, it probably means the two clauses are incompatible.
+
+It's unclear what 'low frequency' is, but if one of the clauses is implied
+(automatically true because of the other clause), then
+
+ selectivity[clause(A)] = selectivity[clause(A) & clause(B)]
+
+So we might compute selectivity of the first clause - for example using regular
+statistics. And then check if the selectivity computed from the histogram is
+about the same (or significantly lower).
+
+The problem is that histograms work well only when the data ordering matches the
+natural meaning. For values that serve as labels - like city names or ZIP codes,
+or even generated IDs, histograms really don't work all that well. For example
+sorting cities by name won't match the sorting of ZIP codes, rendering the
+histogram unusable.
+
+So MCVs are probably going to work much better, because they don't really assume
+any sort of ordering. And it's probably more appropriate for the label-like data.
+
+A good question however is why even use functional dependencies in such cases
+and not simply use the MCV/histogram instead. One reason is that the functional
+dependencies allow fallback to regular stats, and often produce more accurate
+estimates - especially compared to histograms, that are quite bad in estimating
+equality clauses.
+
+
+Limitations
+-----------
+
+Let's see the main liminations of functional dependencies, especially those
+related to the current implementation.
+
+The current implementation supports only dependencies between two columns, but
+this is merely a simplification of the initial implementation. It's certainly
+useful to mine for dependencies involving multiple columns on the 'left' side,
+i.e. a condition for the dependency. That is dependencies like (a,b -> c).
+
+The implementation may/should be smart enough not to mine redundant conditions,
+e.g. (a->b) and (a,c -> b), because the latter is a trivial consequence of the
+former one (if values of 'a' determine 'b', adding another column won't change
+that relationship). The ANALYZE should first analyze 1:1 dependencies, then 2:1
+dependencies (and skip the already identified ones), etc.
+
+For example the dependency
+
+ (city name -> zip code)
+
+is much stronger, i.e. whenever it hold, then
+
+ (city name, state name -> zip code)
+
+holds too. But in case there are cities with the same name in different states,
+then only the latter dependency will be valid.
+
+Of course, there probably are cities with the same name within a single state,
+but hopefully this is relatively rare occurence (and thus we'll still detect
+the 'soft' dependency).
+
+Handling multiple columns on the right side of the dependency, is not necessary,
+as those dependencies may be simply decomposed into a set of dependencies with
+the same meaning, one for each column on the right side. For example
+
+ (a -> b,c)
+
+is exactly the same as
+
+ (a -> b) & (a -> c)
+
+Of course, storing the first form may be more efficient thant storing multiple
+'simple' dependencies separately.
+
+
+TODO Support dependencies with multiple columns on left/right.
+
+TODO Investigate using histogram and MCV list to verify the dependencies.
+
+TODO Investigate statistical testing of the distribution (to decide whether it
+ makes sense to build the histogram/MCV list).
+
+TODO Using a min/max of selectivities would probably make more sense for the
+ associated columns.
+
+TODO Consider eliminating the implied columns from the histogram and MCV lists
+ (but maybe that's not a good idea, because that'd make it impossible to use
+ these stats for non-equality clauses and also it wouldn't be possible to
+ use the stats for verification of the dependencies).
+
+TODO The reduction probably might be extended to also handle IS NULL clauses,
+ assuming we fix the ANALYZE to properly handle NULL values. We however
+ won't be able to reduce IS NOT NULL (unless I'm missing something).
diff --git a/src/backend/utils/mvstats/common.c b/src/backend/utils/mvstats/common.c
new file mode 100644
index 0000000..82f2177
--- /dev/null
+++ b/src/backend/utils/mvstats/common.c
@@ -0,0 +1,376 @@
+/*-------------------------------------------------------------------------
+ *
+ * common.c
+ * POSTGRES multivariate statistics
+ *
+ *
+ * Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/utils/mvstats/common.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "common.h"
+
+static VacAttrStats ** lookup_var_attr_stats(int2vector *attrs,
+ int natts, VacAttrStats **vacattrstats);
+
+static List* list_mv_stats(Oid relid);
+
+
+/*
+ * Compute requested multivariate stats, using the rows sampled for the
+ * plain (single-column) stats.
+ *
+ * This fetches a list of stats from pg_mv_statistic, computes the stats
+ * and serializes them back into the catalog (as bytea values).
+ */
+void
+build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
+ int natts, VacAttrStats **vacattrstats)
+{
+ ListCell *lc;
+ List *mvstats;
+
+ TupleDesc tupdesc = RelationGetDescr(onerel);
+
+ /*
+ * Fetch defined MV groups from pg_mv_statistic, and then compute
+ * the MV statistics (histograms for now).
+ */
+ mvstats = list_mv_stats(RelationGetRelid(onerel));
+
+ foreach (lc, mvstats)
+ {
+ int j;
+ MVStatisticInfo *stat = (MVStatisticInfo *)lfirst(lc);
+ MVDependencies deps = NULL;
+
+ VacAttrStats **stats = NULL;
+ int numatts = 0;
+
+ /* int2 vector of attnums the stats should be computed on */
+ int2vector * attrs = stat->stakeys;
+
+ /* see how many of the columns are not dropped */
+ for (j = 0; j < attrs->dim1; j++)
+ if (! tupdesc->attrs[attrs->values[j]-1]->attisdropped)
+ numatts += 1;
+
+ /* if there are dropped attributes, build a filtered int2vector */
+ if (numatts != attrs->dim1)
+ {
+ int16 *tmp = palloc0(numatts * sizeof(int16));
+ int attnum = 0;
+
+ for (j = 0; j < attrs->dim1; j++)
+ if (! tupdesc->attrs[attrs->values[j]-1]->attisdropped)
+ tmp[attnum++] = attrs->values[j];
+
+ pfree(attrs);
+ attrs = buildint2vector(tmp, numatts);
+ }
+
+ /* filter only the interesting vacattrstats records */
+ stats = lookup_var_attr_stats(attrs, natts, vacattrstats);
+
+ /* check allowed number of dimensions */
+ Assert((attrs->dim1 >= 2) && (attrs->dim1 <= MVSTATS_MAX_DIMENSIONS));
+
+ /*
+ * Analyze functional dependencies of columns.
+ */
+ deps = build_mv_dependencies(numrows, rows, attrs, stats);
+
+ /* store the histogram / MCV list in the catalog */
+ update_mv_stats(stat->mvoid, deps, attrs);
+ }
+}
+
+/*
+ * Lookup the VacAttrStats info for the selected columns, with indexes
+ * matching the attrs vector (to make it easy to work with when
+ * computing multivariate stats).
+ */
+static VacAttrStats **
+lookup_var_attr_stats(int2vector *attrs, int natts, VacAttrStats **vacattrstats)
+{
+ int i, j;
+ int numattrs = attrs->dim1;
+ VacAttrStats **stats = (VacAttrStats**)palloc0(numattrs * sizeof(VacAttrStats*));
+
+ /* lookup VacAttrStats info for the requested columns (same attnum) */
+ for (i = 0; i < numattrs; i++)
+ {
+ stats[i] = NULL;
+ for (j = 0; j < natts; j++)
+ {
+ if (attrs->values[i] == vacattrstats[j]->tupattnum)
+ {
+ stats[i] = vacattrstats[j];
+ break;
+ }
+ }
+
+ /*
+ * Check that we found the info, that the attnum matches and
+ * that there's the requested 'lt' operator and that the type
+ * is 'passed-by-value'.
+ */
+ Assert(stats[i] != NULL);
+ Assert(stats[i]->tupattnum == attrs->values[i]);
+
+ /* FIXME This is rather ugly way to check for 'ltopr' (which
+ * is defined for 'scalar' attributes).
+ */
+ Assert(((StdAnalyzeData *)stats[i]->extra_data)->ltopr != InvalidOid);
+ }
+
+ return stats;
+}
+
+/*
+ * Fetch list of MV stats defined on a table, without the actual data
+ * for histograms, MCV lists etc.
+ */
+static List*
+list_mv_stats(Oid relid)
+{
+ Relation indrel;
+ SysScanDesc indscan;
+ ScanKeyData skey;
+ HeapTuple htup;
+ List *result = NIL;
+
+ /* Prepare to scan pg_mv_statistic for entries having indrelid = this rel. */
+ ScanKeyInit(&skey,
+ Anum_pg_mv_statistic_starelid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(relid));
+
+ indrel = heap_open(MvStatisticRelationId, AccessShareLock);
+ indscan = systable_beginscan(indrel, MvStatisticRelidIndexId, true,
+ NULL, 1, &skey);
+
+ while (HeapTupleIsValid(htup = systable_getnext(indscan)))
+ {
+ MVStatisticInfo *info = makeNode(MVStatisticInfo);
+ Form_pg_mv_statistic stats = (Form_pg_mv_statistic) GETSTRUCT(htup);
+
+ info->mvoid = HeapTupleGetOid(htup);
+ info->stakeys = buildint2vector(stats->stakeys.values, stats->stakeys.dim1);
+ info->deps_built = stats->deps_built;
+
+ result = lappend(result, info);
+ }
+
+ systable_endscan(indscan);
+
+ heap_close(indrel, AccessShareLock);
+
+ /* TODO maybe save the list into relcache, as in RelationGetIndexList
+ * (which was used as an inspiration of this one)?. */
+
+ return result;
+}
+
+void
+update_mv_stats(Oid mvoid, MVDependencies dependencies, int2vector *attrs)
+{
+ HeapTuple stup,
+ oldtup;
+ Datum values[Natts_pg_mv_statistic];
+ bool nulls[Natts_pg_mv_statistic];
+ bool replaces[Natts_pg_mv_statistic];
+
+ Relation sd = heap_open(MvStatisticRelationId, RowExclusiveLock);
+
+ memset(nulls, 1, Natts_pg_mv_statistic * sizeof(bool));
+ memset(replaces, 0, Natts_pg_mv_statistic * sizeof(bool));
+ memset(values, 0, Natts_pg_mv_statistic * sizeof(Datum));
+
+ /*
+ * Construct a new pg_mv_statistic tuple - replace only the histogram
+ * and MCV list, depending whether it actually was computed.
+ */
+ if (dependencies != NULL)
+ {
+ nulls[Anum_pg_mv_statistic_stadeps -1] = false;
+ values[Anum_pg_mv_statistic_stadeps - 1]
+ = PointerGetDatum(serialize_mv_dependencies(dependencies));
+ }
+
+ /* always replace the value (either by bytea or NULL) */
+ replaces[Anum_pg_mv_statistic_stadeps -1] = true;
+
+ /* always change the availability flags */
+ nulls[Anum_pg_mv_statistic_deps_built -1] = false;
+ nulls[Anum_pg_mv_statistic_stakeys-1] = false;
+
+ /* use the new attnums, in case we removed some dropped ones */
+ replaces[Anum_pg_mv_statistic_deps_built-1] = true;
+ replaces[Anum_pg_mv_statistic_stakeys -1] = true;
+
+ values[Anum_pg_mv_statistic_deps_built-1] = BoolGetDatum(dependencies != NULL);
+ values[Anum_pg_mv_statistic_stakeys -1] = PointerGetDatum(attrs);
+
+ /* Is there already a pg_mv_statistic tuple for this attribute? */
+ oldtup = SearchSysCache1(MVSTATOID,
+ ObjectIdGetDatum(mvoid));
+
+ if (HeapTupleIsValid(oldtup))
+ {
+ /* Yes, replace it */
+ stup = heap_modify_tuple(oldtup,
+ RelationGetDescr(sd),
+ values,
+ nulls,
+ replaces);
+ ReleaseSysCache(oldtup);
+ simple_heap_update(sd, &stup->t_self, stup);
+ }
+ else
+ elog(ERROR, "invalid pg_mv_statistic record (oid=%d)", mvoid);
+
+ /* update indexes too */
+ CatalogUpdateIndexes(sd, stup);
+
+ heap_freetuple(stup);
+
+ heap_close(sd, RowExclusiveLock);
+}
+
+/* multi-variate stats comparator */
+
+/*
+ * qsort_arg comparator for sorting Datums (MV stats)
+ *
+ * This does not maintain the tupnoLink array.
+ */
+int
+compare_scalars_simple(const void *a, const void *b, void *arg)
+{
+ Datum da = *(Datum*)a;
+ Datum db = *(Datum*)b;
+ SortSupport ssup= (SortSupport) arg;
+
+ return ApplySortComparator(da, false, db, false, ssup);
+}
+
+/*
+ * qsort_arg comparator for sorting data when partitioning a MV bucket
+ */
+int
+compare_scalars_partition(const void *a, const void *b, void *arg)
+{
+ Datum da = ((ScalarItem*)a)->value;
+ Datum db = ((ScalarItem*)b)->value;
+ SortSupport ssup= (SortSupport) arg;
+
+ return ApplySortComparator(da, false, db, false, ssup);
+}
+
+/* initialize multi-dimensional sort */
+MultiSortSupport
+multi_sort_init(int ndims)
+{
+ MultiSortSupport mss;
+
+ Assert(ndims >= 2);
+
+ mss = (MultiSortSupport)palloc0(offsetof(MultiSortSupportData, ssup)
+ + sizeof(SortSupportData)*ndims);
+
+ mss->ndims = ndims;
+
+ return mss;
+}
+
+/*
+ * add sort into for dimension 'dim' (index into vacattrstats) to mss,
+ * at the position 'sortattr'
+ */
+void
+multi_sort_add_dimension(MultiSortSupport mss, int sortdim,
+ int dim, VacAttrStats **vacattrstats)
+{
+ /* first, lookup StdAnalyzeData for the dimension (attribute) */
+ SortSupportData ssup;
+ StdAnalyzeData *tmp = (StdAnalyzeData *)vacattrstats[dim]->extra_data;
+
+ Assert(mss != NULL);
+ Assert(sortdim < mss->ndims);
+
+ /* initialize sort support, etc. */
+ memset(&ssup, 0, sizeof(ssup));
+ ssup.ssup_cxt = CurrentMemoryContext;
+
+ /* We always use the default collation for statistics */
+ ssup.ssup_collation = DEFAULT_COLLATION_OID;
+ ssup.ssup_nulls_first = false;
+
+ PrepareSortSupportFromOrderingOp(tmp->ltopr, &ssup);
+
+ mss->ssup[sortdim] = ssup;
+}
+
+/* compare all the dimensions in the selected order */
+int
+multi_sort_compare(const void *a, const void *b, void *arg)
+{
+ int i;
+ SortItem *ia = (SortItem*)a;
+ SortItem *ib = (SortItem*)b;
+
+ MultiSortSupport mss = (MultiSortSupport)arg;
+
+ for (i = 0; i < mss->ndims; i++)
+ {
+ int compare;
+
+ compare = ApplySortComparator(ia->values[i], ia->isnull[i],
+ ib->values[i], ib->isnull[i],
+ &mss->ssup[i]);
+
+ if (compare != 0)
+ return compare;
+
+ }
+
+ /* equal by default */
+ return 0;
+}
+
+/* compare selected dimension */
+int
+multi_sort_compare_dim(int dim, const SortItem *a, const SortItem *b,
+ MultiSortSupport mss)
+{
+ return ApplySortComparator(a->values[dim], a->isnull[dim],
+ b->values[dim], b->isnull[dim],
+ &mss->ssup[dim]);
+}
+
+int
+multi_sort_compare_dims(int start, int end,
+ const SortItem *a, const SortItem *b,
+ MultiSortSupport mss)
+{
+ int dim;
+
+ for (dim = start; dim <= end; dim++)
+ {
+ int r = ApplySortComparator(a->values[dim], a->isnull[dim],
+ b->values[dim], b->isnull[dim],
+ &mss->ssup[dim]);
+
+ if (r != 0)
+ return r;
+ }
+
+ return 0;
+}
diff --git a/src/backend/utils/mvstats/common.h b/src/backend/utils/mvstats/common.h
new file mode 100644
index 0000000..75b9c54
--- /dev/null
+++ b/src/backend/utils/mvstats/common.h
@@ -0,0 +1,78 @@
+/*-------------------------------------------------------------------------
+ *
+ * common.h
+ * POSTGRES multivariate statistics
+ *
+ *
+ * Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/utils/mvstats/common.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/sysattr.h"
+#include "access/tuptoaster.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_collation.h"
+#include "catalog/pg_mv_statistic.h"
+#include "foreign/fdwapi.h"
+#include "postmaster/autovacuum.h"
+#include "storage/lmgr.h"
+#include "utils/builtins.h"
+#include "utils/datum.h"
+#include "utils/fmgroids.h"
+#include "utils/mvstats.h"
+#include "utils/sortsupport.h"
+#include "utils/syscache.h"
+
+
+/* FIXME private structure copied from analyze.c */
+
+typedef struct
+{
+ Oid eqopr; /* '=' operator for datatype, if any */
+ Oid eqfunc; /* and associated function */
+ Oid ltopr; /* '<' operator for datatype, if any */
+} StdAnalyzeData;
+
+typedef struct
+{
+ Datum value; /* a data value */
+ int tupno; /* position index for tuple it came from */
+} ScalarItem;
+
+/* multi-sort */
+typedef struct MultiSortSupportData {
+ int ndims; /* number of dimensions supported by the */
+ SortSupportData ssup[1]; /* sort support data for each dimension */
+} MultiSortSupportData;
+
+typedef MultiSortSupportData* MultiSortSupport;
+
+typedef struct SortItem {
+ Datum *values;
+ bool *isnull;
+} SortItem;
+
+MultiSortSupport multi_sort_init(int ndims);
+
+void multi_sort_add_dimension(MultiSortSupport mss, int sortdim,
+ int dim, VacAttrStats **vacattrstats);
+
+int multi_sort_compare(const void *a, const void *b, void *arg);
+
+int multi_sort_compare_dim(int dim, const SortItem *a,
+ const SortItem *b, MultiSortSupport mss);
+
+int multi_sort_compare_dims(int start, int end, const SortItem *a,
+ const SortItem *b, MultiSortSupport mss);
+
+/* comparators, used when constructing multivariate stats */
+int compare_scalars_simple(const void *a, const void *b, void *arg);
+int compare_scalars_partition(const void *a, const void *b, void *arg);
diff --git a/src/backend/utils/mvstats/dependencies.c b/src/backend/utils/mvstats/dependencies.c
new file mode 100644
index 0000000..5437bdf
--- /dev/null
+++ b/src/backend/utils/mvstats/dependencies.c
@@ -0,0 +1,686 @@
+/*-------------------------------------------------------------------------
+ *
+ * dependencies.c
+ * POSTGRES multivariate functional dependencies
+ *
+ *
+ * Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/utils/mvstats/dependencies.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "common.h"
+#include "utils/lsyscache.h"
+
+/* internal state for generator of variations (k-permutations of n elements) */
+typedef struct VariationGeneratorData {
+
+ int k; /* size of the k-permutation */
+ int current; /* index of the next variation to return */
+
+ int nvariations; /* number of variations generated (size of array) */
+ int variations[1]; /* array of pre-built variations */
+
+} VariationGeneratorData;
+
+typedef VariationGeneratorData* VariationGenerator;
+
+/*
+ * generate all variations (k-permutations of n elements)
+ */
+static void
+generate_variations(VariationGenerator state,
+ int n, int maxlevel, int level, int *current)
+{
+ int i, j;
+
+ /* initialize */
+ if (level == 0)
+ {
+ current = (int*)palloc0(sizeof(int) * (maxlevel+1));
+ state->current = 0;
+ }
+
+ for (i = 0; i < n; i++)
+ {
+ /* check if the value is already used current variation */
+ bool found = false;
+ for (j = 0; j < level; j++)
+ {
+ if (current[j] == i)
+ {
+ found = true;
+ break;
+ }
+ }
+
+ /* already used, so try the next element */
+ if (found)
+ continue;
+
+ /* ok, we can use this element, so store it */
+ current[level] = i;
+
+ /* and check if we do have a complete variation of k elements */
+ if (level == maxlevel)
+ {
+ /* yep, store the variation */
+ Assert(state->current < state->nvariations);
+ memcpy(&state->variations[(state->k * state->current)], current,
+ sizeof(int) * (maxlevel+1));
+ state->current++;
+ }
+ else
+ /* nope, look for additional elements */
+ generate_variations(state, n, maxlevel, level+1, current);
+ }
+
+ if (level == 0)
+ pfree(current);
+}
+
+/*
+ * initialize the generator of variations, and prebuild the variations
+ *
+ * This pre-builds all the variations. We could also generate them in
+ * generator_next(), but this seems simpler.
+ */
+static VariationGenerator
+generator_init(int2vector *attrs, int k)
+{
+ int i;
+ int n = attrs->dim1;
+ int nvariations;
+ VariationGenerator state;
+
+ Assert((n >= k) && (k > 0));
+
+ /* compute the total number of variations as n!/(n-k)! */
+ nvariations = n;
+ for (i = 1; i < k; i++)
+ nvariations *= (n - i);
+
+ /* allocate the generator state as a single chunk of memory */
+ state = (VariationGenerator)palloc0(
+ offsetof(VariationGeneratorData, variations)
+ + (nvariations * k * sizeof(int))); /* variations */
+
+ state->nvariations = nvariations;
+ state->k = k;
+
+ /* now actually pre-generate all the variations */
+ generate_variations(state, n, (k-1), 0, NULL);
+
+ /* we expect to generate exactly the right number of variations */
+ Assert(state->nvariations == state->current);
+
+ /* reset the index */
+ state->current = 0;
+
+ return state;
+}
+
+/* free the generator state */
+static void
+generator_free(VariationGenerator state)
+{
+ /* we've allocated a single chunk, so just free it */
+ pfree(state);
+}
+
+/* generate next combination */
+static int*
+generator_next(VariationGenerator state, int2vector *attrs)
+{
+ if (state->current == state->nvariations)
+ return NULL;
+
+ return &state->variations[state->k * state->current++];
+}
+
+/*
+ * check if the dependency is implied by existing dependencies
+ *
+ * A dependency is considered implied, if there exists a dependency with the
+ * same column on the left, and a subset of columns on the right side. So for
+ * example if we have a dependency
+ *
+ * (a,b,c) -> d
+ *
+ * then we are looking for these six dependencies
+ *
+ * (a) -> d
+ * (b) -> d
+ * (c) -> d
+ * (a,b) -> d
+ * (a,c) -> d
+ * (b,c) -> d
+ *
+ * This does not detect transitive dependencies. For example if we have
+ *
+ * (a) -> b
+ * (b) -> c
+ *
+ * then obviously
+ *
+ * (a) -> c
+ *
+ * but this is not detected. Extending the method to handle transitive cases
+ * is future work.
+ */
+static bool
+dependency_is_implied(MVDependencies dependencies, int k, int *dependency,
+ int2vector * attrs)
+{
+ bool implied = false;
+ int i, j, l;
+ int *tmp;
+
+ if (dependencies == NULL)
+ return false;
+
+ tmp = (int*)palloc0(sizeof(int) * k);
+
+ /* translate the indexes to actual attribute numbers */
+ for (i = 0; i < k; i++)
+ tmp[i] = attrs->values[dependency[i]];
+
+ /* search for a smaller */
+ for (i = 0; i < dependencies->ndeps; i++)
+ {
+ bool contained = true;
+ MVDependency dep = dependencies->deps[i];
+
+ /* does the last attribute match? */
+ if (tmp[k-1] != dep->attributes[dep->nattributes-1])
+ continue; /* nope, no need to check this dependency further */
+
+ /* are the conditions superset of the existing dependency? */
+ for (j = 0; j < (dep->nattributes-1); j++)
+ {
+ bool found = false;
+
+ for (l = 0; l < (k-1); l++)
+ {
+ if (tmp[l] == dep->attributes[j])
+ {
+ found = true;
+ break;
+ }
+ }
+
+ /* we've found an attribute not included in the new dependency */
+ if (! found)
+ {
+ contained = false;
+ break;
+ }
+ }
+
+ /* we've found an existing dependency, trivially proving the new one */
+ if (contained)
+ {
+ implied = true;
+ break;
+ }
+ }
+
+ pfree(tmp);
+
+ return implied;
+}
+
+/*
+ * validates functional dependency on the data
+ *
+ * An actual work horse of detecting functional dependencies. Given a variation
+ * of k attributes, it checks that the first (k-1) are sufficient to determine
+ * the last one.
+ */
+static bool
+dependency_is_valid(int numrows, HeapTuple *rows, int k, int * dependency,
+ VacAttrStats **stats, int2vector *attrs)
+{
+ int i, j;
+ int nvalues = numrows * k;
+
+ /*
+ * XXX Maybe the threshold should be somehow related to the number of
+ * distinct values in the combination of columns we're analyzing.
+ * Assuming the distribution is uniform, we can estimate the average
+ * group size and use it as a threshold, similarly to what we do for
+ * MCV lists.
+ */
+ int min_group_size = 3;
+
+ /* number of groups supporting / contradicting the dependency */
+ int n_supporting = 0;
+ int n_contradicting = 0;
+
+ /* counters valid within a group */
+ int group_size = 0;
+ int n_violations = 0;
+
+ int n_supporting_rows = 0;
+ int n_contradicting_rows = 0;
+
+ /* sort info for all attributes columns */
+ MultiSortSupport mss = multi_sort_init(k);
+
+ /* data for the sort */
+ SortItem *items = (SortItem*)palloc0(numrows * sizeof(SortItem));
+ Datum *values = (Datum*)palloc0(sizeof(Datum) * nvalues);
+ bool *isnull = (bool*)palloc0(sizeof(bool) * nvalues);
+
+ /* fix the pointers to values/isnull */
+ for (i = 0; i < numrows; i++)
+ {
+ items[i].values = &values[i * k];
+ items[i].isnull = &isnull[i * k];
+ }
+
+ /*
+ * Verify the dependency (a,b,...)->z, using a rather simple algorithm:
+ *
+ * (a) sort the data lexicographically
+ *
+ * (b) split the data into groups by first (k-1) columns
+ *
+ * (c) for each group count different values in the last column
+ */
+
+ /* prepare the sort function for the first dimension, and SortItem array */
+ for (i = 0; i < k; i++)
+ {
+ multi_sort_add_dimension(mss, i, dependency[i], stats);
+
+ /* accumulate all the data for both columns into an array and sort it */
+ for (j = 0; j < numrows; j++)
+ {
+ items[j].values[i]
+ = heap_getattr(rows[j], attrs->values[dependency[i]],
+ stats[i]->tupDesc, &items[j].isnull[i]);
+ }
+ }
+
+ /* sort the items so that we can detect the groups */
+ qsort_arg((void *) items, numrows, sizeof(SortItem),
+ multi_sort_compare, mss);
+
+ /*
+ * Walk through the sorted array, split it into rows according to the first
+ * (k-1) columns. If there's a single value in the last column, we count
+ * the group as 'supporting' the functional dependency. Otherwise we count
+ * it as contradicting.
+ *
+ * We also require a group to have a minimum number of rows to be considered
+ * useful for supporting the dependency. Contradicting groups may be of
+ * any size, though.
+ *
+ * XXX The minimum size requirement makes it impossible to identify case
+ * when both columns are unique (or nearly unique), and therefore
+ * trivially functionally dependent.
+ */
+
+ /* start with the first row forming a group */
+ group_size = 1;
+
+ for (i = 1; i < numrows; i++)
+ {
+ /* end of the preceding group */
+ if (multi_sort_compare_dims(0, (k-2), &items[i-1], &items[i], mss) != 0)
+ {
+ /*
+ * If there is a single are no contradicting rows, count the group
+ * as supporting, otherwise contradicting.
+ */
+ if ((n_violations == 0) && (group_size >= min_group_size))
+ {
+ n_supporting += 1;
+ n_supporting_rows += group_size;
+ }
+ else if (n_violations > 0)
+ {
+ n_contradicting += 1;
+ n_contradicting_rows += group_size;
+ }
+
+ /* current values start a new group */
+ n_violations = 0;
+ group_size = 0;
+ }
+ /* first colums match, but the last one does not (so contradicting) */
+ else if (multi_sort_compare_dims((k-1), (k-1), &items[i-1], &items[i], mss) != 0)
+ n_violations += 1;
+
+ group_size += 1;
+ }
+
+ /* handle the last group (just like above) */
+ if ((n_violations == 0) && (group_size >= min_group_size))
+ {
+ n_supporting += 1;
+ n_supporting_rows += group_size;
+ }
+ else if (n_violations)
+ {
+ n_contradicting += 1;
+ n_contradicting_rows += group_size;
+ }
+
+ pfree(items);
+ pfree(values);
+ pfree(isnull);
+ pfree(mss);
+
+ /*
+ * See if the number of rows supporting the association is at least 10x the
+ * number of rows violating the hypothetical dependency.
+ */
+ return (n_supporting_rows > (n_contradicting_rows * 10));
+}
+
+/*
+ * detects functional dependencies between groups of columns
+ *
+ * Generates all possible subsets of columns (variations) and checks if the
+ * last one is determined by the preceding ones. For example given 3 columns,
+ * there are 12 variations (6 for variations on 2 columns, 6 for 3 columns):
+ *
+ * two columns three columns
+ * ----------- -------------
+ * (a) -> c (a,b) -> c
+ * (b) -> c (b,a) -> c
+ * (a) -> b (a,c) -> b
+ * (c) -> b (c,a) -> b
+ * (c) -> a (c,b) -> a
+ * (b) -> a (b,c) -> a
+ *
+ * Clearly some of the variations are redundant, as the order of columns on the
+ * left side does not matter. This is detected in dependency_is_implied, and
+ * those dependencies are ignored.
+ *
+ * We however do not detect that dependencies are transitively implied. For
+ * example given dependencies
+ *
+ * (a) -> b
+ * (b) -> c
+ *
+ * then
+ *
+ * (a) -> c
+ *
+ * is trivially implied. However we don't detect that and all three dependencies
+ * will get included in the resulting set. Eliminating such transitively implied
+ * dependencies is future work.
+ */
+MVDependencies
+build_mv_dependencies(int numrows, HeapTuple *rows, int2vector *attrs,
+ VacAttrStats **stats)
+{
+ int i;
+ int k;
+ int numattrs = attrs->dim1;
+
+ /* result */
+ MVDependencies dependencies = NULL;
+
+ Assert(numattrs >= 2);
+
+ /*
+ * We'll try build functional dependencies starting from the smallest ones
+ * covering jut 2 columns, to the largest ones, covering all columns
+ * included int the statistics. We start from the smallest ones because
+ * we want to be able to skip already implied ones.
+ */
+ for (k = 2; k <= numattrs; k++)
+ {
+ int *dependency; /* array with k elements */
+
+ /* prepare a generator of variation */
+ VariationGenerator generator = generator_init(attrs, k);
+
+ /* generate all possible variations of k values (out of n) */
+ while ((dependency = generator_next(generator, attrs)))
+ {
+ MVDependency d;
+
+ /* skip dependencies that are already trivially implied */
+ if (dependency_is_implied(dependencies, k, dependency, attrs))
+ continue;
+
+ /* also skip dependencies that don't seem to be valid */
+ if (! dependency_is_valid(numrows, rows, k, dependency, stats, attrs))
+ continue;
+
+ d = (MVDependency)palloc0(offsetof(MVDependencyData, attributes)
+ + k * sizeof(int));
+
+ /* copy the dependency, but translate it to actuall attnums */
+ d->nattributes = k;
+ for (i = 0; i < k; i++)
+ d->attributes[i] = attrs->values[dependency[i]];
+
+ /* initialize the list of dependencies */
+ if (dependencies == NULL)
+ {
+ dependencies
+ = (MVDependencies)palloc0(sizeof(MVDependenciesData));
+
+ dependencies->magic = MVSTAT_DEPS_MAGIC;
+ dependencies->type = MVSTAT_DEPS_TYPE_BASIC;
+ dependencies->ndeps = 0;
+ }
+
+ dependencies->ndeps++;
+ dependencies = (MVDependencies)repalloc(dependencies,
+ offsetof(MVDependenciesData, deps)
+ + dependencies->ndeps * sizeof(MVDependency));
+
+ dependencies->deps[dependencies->ndeps-1] = d;
+ }
+
+ /* we're done with variations of k elements, so free the generator */
+ generator_free(generator);
+ }
+
+ return dependencies;
+}
+
+
+/*
+ * serialize list of dependencies into a bytea
+ */
+bytea *
+serialize_mv_dependencies(MVDependencies dependencies)
+{
+ int i;
+ bytea * output;
+ char *tmp;
+
+ /* we need to store ndeps, with a number of attributes for each one */
+ Size len = VARHDRSZ + offsetof(MVDependenciesData, deps)
+ + sizeof(int) * dependencies->ndeps;
+
+ /* and also include space for the actual attribute numbers */
+ for (i = 0; i < dependencies->ndeps; i++)
+ len += (sizeof(int16) * dependencies->deps[i]->nattributes);
+
+ output = (bytea*)palloc0(len);
+ SET_VARSIZE(output, len);
+
+ tmp = VARDATA(output);
+
+ /* first, store the number of dimensions / items */
+ memcpy(tmp, dependencies, offsetof(MVDependenciesData, deps));
+ tmp += offsetof(MVDependenciesData, deps);
+
+ /* store number of attributes and attribute numbers for each dependency */
+ for (i = 0; i < dependencies->ndeps; i++)
+ {
+ MVDependency d = dependencies->deps[i];
+
+ memcpy(tmp, &(d->nattributes), sizeof(int));
+ tmp += sizeof(int);
+
+ memcpy(tmp, d->attributes, sizeof(int16) * d->nattributes);
+ tmp += sizeof(int16) * d->nattributes;
+
+ Assert(tmp <= ((char*)output + len));
+ }
+
+ return output;
+}
+
+/*
+ * Reads serialized dependencies into MVDependencies structure.
+ */
+MVDependencies
+deserialize_mv_dependencies(bytea * data)
+{
+ int i;
+ Size expected_size;
+ MVDependencies dependencies;
+ char *tmp;
+
+ if (data == NULL)
+ return NULL;
+
+ if (VARSIZE_ANY_EXHDR(data) < offsetof(MVDependenciesData,deps))
+ elog(ERROR, "invalid MVDependencies size %ld (expected at least %ld)",
+ VARSIZE_ANY_EXHDR(data), offsetof(MVDependenciesData,deps));
+
+ /* read the MVDependencies header */
+ dependencies = (MVDependencies)palloc0(sizeof(MVDependenciesData));
+
+ /* initialize pointer to the data part (skip the varlena header) */
+ tmp = VARDATA(data);
+
+ /* get the header and perform basic sanity checks */
+ memcpy(dependencies, tmp, offsetof(MVDependenciesData, deps));
+ tmp += offsetof(MVDependenciesData, deps);
+
+ if (dependencies->magic != MVSTAT_DEPS_MAGIC)
+ elog(ERROR, "invalid dependency type %d (expected %dd)",
+ dependencies->type, MVSTAT_DEPS_MAGIC);
+
+ if (dependencies->type != MVSTAT_DEPS_TYPE_BASIC)
+ elog(ERROR, "invalid dependency type %d (expected %dd)",
+ dependencies->type, MVSTAT_DEPS_TYPE_BASIC);
+
+ Assert(dependencies->ndeps > 0);
+
+ /* what minimum bytea size do we expect for those parameters */
+ expected_size = offsetof(MVDependenciesData,deps) +
+ dependencies->ndeps * (sizeof(int) + sizeof(int16) * 2);
+
+ if (VARSIZE_ANY_EXHDR(data) < expected_size)
+ elog(ERROR, "invalid dependencies size %ld (expected at least %ld)",
+ VARSIZE_ANY_EXHDR(data), expected_size);
+
+ /* allocate space for the MCV items */
+ dependencies = repalloc(dependencies, offsetof(MVDependenciesData,deps)
+ + (dependencies->ndeps * sizeof(MVDependency)));
+
+ for (i = 0; i < dependencies->ndeps; i++)
+ {
+ int k;
+ MVDependency d;
+
+ /* number of attributes */
+ memcpy(&k, tmp, sizeof(int));
+ tmp += sizeof(int);
+
+ /* is the number of attributes valid? */
+ Assert((k >= 2) && (k <= MVSTATS_MAX_DIMENSIONS));
+
+ /* now that we know the number of attributes, allocate the dependency */
+ d = (MVDependency)palloc0(offsetof(MVDependencyData, attributes)
+ + k * sizeof(int));
+
+ d->nattributes = k;
+
+ /* copy attribute numbers */
+ memcpy(d->attributes, tmp, sizeof(int16) * d->nattributes);
+ tmp += sizeof(int16) * d->nattributes;
+
+ dependencies->deps[i] = d;
+
+ /* still within the bytea */
+ Assert(tmp <= ((char*)data + VARSIZE_ANY(data)));
+ }
+
+ /* we should have consumed the whole bytea exactly */
+ Assert(tmp == ((char*)data + VARSIZE_ANY(data)));
+
+ return dependencies;
+}
+
+/* print some basic info about dependencies (number of dependencies) */
+Datum
+pg_mv_stats_dependencies_info(PG_FUNCTION_ARGS)
+{
+ bytea *data = PG_GETARG_BYTEA_P(0);
+ char *result;
+
+ MVDependencies dependencies = deserialize_mv_dependencies(data);
+
+ if (dependencies == NULL)
+ PG_RETURN_NULL();
+
+ result = palloc0(128);
+ snprintf(result, 128, "dependencies=%d", dependencies->ndeps);
+
+ /* FIXME free the deserialized data (pfree is not enough) */
+
+ PG_RETURN_TEXT_P(cstring_to_text(result));
+}
+
+/*
+ * print the dependencies
+ *
+ * TODO Would be nice if this printed column names (instead of just attnums).
+ */
+Datum
+pg_mv_stats_dependencies_show(PG_FUNCTION_ARGS)
+{
+ int i, j;
+ bytea *data = PG_GETARG_BYTEA_P(0);
+ StringInfoData buf;
+
+ MVDependencies dependencies = deserialize_mv_dependencies(data);
+
+ if (dependencies == NULL)
+ PG_RETURN_NULL();
+
+ initStringInfo(&buf);
+
+ for (i = 0; i < dependencies->ndeps; i++)
+ {
+ MVDependency dependency = dependencies->deps[i];
+
+ if (i > 0)
+ appendStringInfo(&buf, ", ");
+
+ /* conditions */
+ appendStringInfoChar(&buf, '(');
+ for (j = 0; j < dependency->nattributes-1; j++)
+ {
+ if (j > 0)
+ appendStringInfoChar(&buf, ',');
+
+ appendStringInfo(&buf, "%d", dependency->attributes[j]);
+ }
+
+ /* the implied attribute */
+ appendStringInfo(&buf, ") => %d",
+ dependency->attributes[dependency->nattributes-1]);
+ }
+
+ PG_RETURN_TEXT_P(cstring_to_text(buf.data));
+}
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index fd8dc91..8ce9c0e 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -2104,6 +2104,50 @@ describeOneTableDetails(const char *schemaname,
PQclear(result);
}
+ /* print any multivariate statistics */
+ if (pset.sversion >= 90600)
+ {
+ printfPQExpBuffer(&buf,
+ "SELECT oid, stanamespace::regnamespace AS nsp, staname, stakeys,\n"
+ " deps_enabled,\n"
+ " deps_built,\n"
+ " (SELECT string_agg(attname::text,', ')\n"
+ " FROM ((SELECT unnest(stakeys) AS attnum) s\n"
+ " JOIN pg_attribute a ON (starelid = a.attrelid and a.attnum = s.attnum))) AS attnums\n"
+ "FROM pg_mv_statistic stat WHERE starelid = '%s' ORDER BY 1;",
+ oid);
+
+ result = PSQLexec(buf.data);
+ if (!result)
+ goto error_return;
+ else
+ tuples = PQntuples(result);
+
+ if (tuples > 0)
+ {
+ printTableAddFooter(&cont, _("Statistics:"));
+ for (i = 0; i < tuples; i++)
+ {
+ printfPQExpBuffer(&buf, " ");
+
+ /* statistics name (qualified with namespace) */
+ appendPQExpBuffer(&buf, "\"%s.%s\" ",
+ PQgetvalue(result, i, 1),
+ PQgetvalue(result, i, 2));
+
+ /* options */
+ if (!strcmp(PQgetvalue(result, i, 4), "t"))
+ appendPQExpBuffer(&buf, "(dependencies)");
+
+ appendPQExpBuffer(&buf, " ON (%s)",
+ PQgetvalue(result, i, 6));
+
+ printTableAddFooter(&cont, buf.data);
+ }
+ }
+ PQclear(result);
+ }
+
/* print rules */
if (tableinfo.hasrules && tableinfo.relkind != 'm')
{
diff --git a/src/include/catalog/dependency.h b/src/include/catalog/dependency.h
index 049bf9f..12211fe 100644
--- a/src/include/catalog/dependency.h
+++ b/src/include/catalog/dependency.h
@@ -153,10 +153,11 @@ typedef enum ObjectClass
OCLASS_EXTENSION, /* pg_extension */
OCLASS_EVENT_TRIGGER, /* pg_event_trigger */
OCLASS_POLICY, /* pg_policy */
- OCLASS_TRANSFORM /* pg_transform */
+ OCLASS_TRANSFORM, /* pg_transform */
+ OCLASS_STATISTICS /* pg_mv_statistics */
} ObjectClass;
-#define LAST_OCLASS OCLASS_TRANSFORM
+#define LAST_OCLASS OCLASS_STATISTICS
/* in dependency.c */
diff --git a/src/include/catalog/heap.h b/src/include/catalog/heap.h
index b80d8d8..5ae42f7 100644
--- a/src/include/catalog/heap.h
+++ b/src/include/catalog/heap.h
@@ -119,6 +119,7 @@ extern void RemoveAttrDefault(Oid relid, AttrNumber attnum,
DropBehavior behavior, bool complain, bool internal);
extern void RemoveAttrDefaultById(Oid attrdefId);
extern void RemoveStatistics(Oid relid, AttrNumber attnum);
+extern void RemoveMVStatistics(Oid relid, AttrNumber attnum);
extern Form_pg_attribute SystemAttributeDefinition(AttrNumber attno,
bool relhasoids);
diff --git a/src/include/catalog/indexing.h b/src/include/catalog/indexing.h
index ab2c1a8..a768bb5 100644
--- a/src/include/catalog/indexing.h
+++ b/src/include/catalog/indexing.h
@@ -173,6 +173,13 @@ DECLARE_UNIQUE_INDEX(pg_largeobject_loid_pn_index, 2683, on pg_largeobject using
DECLARE_UNIQUE_INDEX(pg_largeobject_metadata_oid_index, 2996, on pg_largeobject_metadata using btree(oid oid_ops));
#define LargeObjectMetadataOidIndexId 2996
+DECLARE_UNIQUE_INDEX(pg_mv_statistic_oid_index, 3380, on pg_mv_statistic using btree(oid oid_ops));
+#define MvStatisticOidIndexId 3380
+DECLARE_UNIQUE_INDEX(pg_mv_statistic_name_index, 3997, on pg_mv_statistic using btree(staname name_ops, stanamespace oid_ops));
+#define MvStatisticNameIndexId 3997
+DECLARE_INDEX(pg_mv_statistic_relid_index, 3379, on pg_mv_statistic using btree(starelid oid_ops));
+#define MvStatisticRelidIndexId 3379
+
DECLARE_UNIQUE_INDEX(pg_namespace_nspname_index, 2684, on pg_namespace using btree(nspname name_ops));
#define NamespaceNameIndexId 2684
DECLARE_UNIQUE_INDEX(pg_namespace_oid_index, 2685, on pg_namespace using btree(oid oid_ops));
diff --git a/src/include/catalog/namespace.h b/src/include/catalog/namespace.h
index 2ccb3a7..44cf9c6 100644
--- a/src/include/catalog/namespace.h
+++ b/src/include/catalog/namespace.h
@@ -137,6 +137,8 @@ extern Oid get_collation_oid(List *collname, bool missing_ok);
extern Oid get_conversion_oid(List *conname, bool missing_ok);
extern Oid FindDefaultConversionProc(int32 for_encoding, int32 to_encoding);
+extern Oid get_statistics_oid(List *names, bool missing_ok);
+
/* initialization & transaction cleanup code */
extern void InitializeSearchPath(void);
extern void AtEOXact_Namespace(bool isCommit, bool parallel);
diff --git a/src/include/catalog/pg_mv_statistic.h b/src/include/catalog/pg_mv_statistic.h
new file mode 100644
index 0000000..c74af47
--- /dev/null
+++ b/src/include/catalog/pg_mv_statistic.h
@@ -0,0 +1,75 @@
+/*-------------------------------------------------------------------------
+ *
+ * pg_mv_statistic.h
+ * definition of the system "multivariate statistic" relation (pg_mv_statistic)
+ * along with the relation's initial contents.
+ *
+ *
+ * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/catalog/pg_mv_statistic.h
+ *
+ * NOTES
+ * the genbki.pl script reads this file and generates .bki
+ * information from the DATA() statements.
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef PG_MV_STATISTIC_H
+#define PG_MV_STATISTIC_H
+
+#include "catalog/genbki.h"
+
+/* ----------------
+ * pg_mv_statistic definition. cpp turns this into
+ * typedef struct FormData_pg_mv_statistic
+ * ----------------
+ */
+#define MvStatisticRelationId 3381
+
+CATALOG(pg_mv_statistic,3381)
+{
+ /* These fields form the unique key for the entry: */
+ Oid starelid; /* relation containing attributes */
+ NameData staname; /* statistics name */
+ Oid stanamespace; /* OID of namespace containing this statistics */
+ Oid staowner; /* statistics owner */
+
+ /* statistics requested to build */
+ bool deps_enabled; /* analyze dependencies? */
+
+ /* statistics that are available (if requested) */
+ bool deps_built; /* dependencies were built */
+
+ /* variable-length fields start here, but we allow direct access to stakeys */
+ int2vector stakeys; /* array of column keys */
+
+#ifdef CATALOG_VARLEN
+ bytea stadeps; /* dependencies (serialized) */
+#endif
+
+} FormData_pg_mv_statistic;
+
+/* ----------------
+ * Form_pg_mv_statistic corresponds to a pointer to a tuple with
+ * the format of pg_mv_statistic relation.
+ * ----------------
+ */
+typedef FormData_pg_mv_statistic *Form_pg_mv_statistic;
+
+/* ----------------
+ * compiler constants for pg_mv_statistic
+ * ----------------
+ */
+#define Natts_pg_mv_statistic 8
+#define Anum_pg_mv_statistic_starelid 1
+#define Anum_pg_mv_statistic_staname 2
+#define Anum_pg_mv_statistic_stanamespace 3
+#define Anum_pg_mv_statistic_staowner 4
+#define Anum_pg_mv_statistic_deps_enabled 5
+#define Anum_pg_mv_statistic_deps_built 6
+#define Anum_pg_mv_statistic_stakeys 7
+#define Anum_pg_mv_statistic_stadeps 8
+
+#endif /* PG_MV_STATISTIC_H */
diff --git a/src/include/catalog/pg_proc.h b/src/include/catalog/pg_proc.h
index ceb8129..cdcbf95 100644
--- a/src/include/catalog/pg_proc.h
+++ b/src/include/catalog/pg_proc.h
@@ -2666,6 +2666,11 @@ DESCR("current user privilege on any column by rel name");
DATA(insert OID = 3029 ( has_any_column_privilege PGNSP PGUID 12 10 0 0 0 f f f f t f s s 2 0 16 "26 25" _null_ _null_ _null_ _null_ _null_ has_any_column_privilege_id _null_ _null_ _null_ ));
DESCR("current user privilege on any column by rel oid");
+DATA(insert OID = 3998 ( pg_mv_stats_dependencies_info PGNSP PGUID 12 1 0 0 0 f f f f t f i s 1 0 25 "17" _null_ _null_ _null_ _null_ _null_ pg_mv_stats_dependencies_info _null_ _null_ _null_ ));
+DESCR("multivariate stats: functional dependencies info");
+DATA(insert OID = 3999 ( pg_mv_stats_dependencies_show PGNSP PGUID 12 1 0 0 0 f f f f t f i s 1 0 25 "17" _null_ _null_ _null_ _null_ _null_ pg_mv_stats_dependencies_show _null_ _null_ _null_ ));
+DESCR("multivariate stats: functional dependencies show");
+
DATA(insert OID = 1928 ( pg_stat_get_numscans PGNSP PGUID 12 1 0 0 0 f f f f t f s r 1 0 20 "26" _null_ _null_ _null_ _null_ _null_ pg_stat_get_numscans _null_ _null_ _null_ ));
DESCR("statistics: number of scans done for table/index");
DATA(insert OID = 1929 ( pg_stat_get_tuples_returned PGNSP PGUID 12 1 0 0 0 f f f f t f s r 1 0 20 "26" _null_ _null_ _null_ _null_ _null_ pg_stat_get_tuples_returned _null_ _null_ _null_ ));
diff --git a/src/include/catalog/toasting.h b/src/include/catalog/toasting.h
index b7a38ce..a52096b 100644
--- a/src/include/catalog/toasting.h
+++ b/src/include/catalog/toasting.h
@@ -49,6 +49,7 @@ extern void BootstrapToastTable(char *relName,
DECLARE_TOAST(pg_attrdef, 2830, 2831);
DECLARE_TOAST(pg_constraint, 2832, 2833);
DECLARE_TOAST(pg_description, 2834, 2835);
+DECLARE_TOAST(pg_mv_statistic, 3577, 3578);
DECLARE_TOAST(pg_proc, 2836, 2837);
DECLARE_TOAST(pg_rewrite, 2838, 2839);
DECLARE_TOAST(pg_seclabel, 3598, 3599);
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index 54f67e9..99a6a62 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -75,6 +75,10 @@ extern ObjectAddress DefineOperator(List *names, List *parameters);
extern void RemoveOperatorById(Oid operOid);
extern ObjectAddress AlterOperator(AlterOperatorStmt *stmt);
+/* commands/statscmds.c */
+extern ObjectAddress CreateStatistics(CreateStatsStmt *stmt);
+extern void RemoveStatisticsById(Oid statsOid);
+
/* commands/aggregatecmds.c */
extern ObjectAddress DefineAggregate(List *name, List *args, bool oldstyle,
List *parameters, const char *queryString);
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index fad9988..545b62a 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -266,6 +266,7 @@ typedef enum NodeTag
T_PlaceHolderInfo,
T_MinMaxAggInfo,
T_PlannerParamItem,
+ T_MVStatisticInfo,
/*
* TAGS FOR MEMORY NODES (memnodes.h)
@@ -401,6 +402,7 @@ typedef enum NodeTag
T_CreatePolicyStmt,
T_AlterPolicyStmt,
T_CreateTransformStmt,
+ T_CreateStatsStmt,
/*
* TAGS FOR PARSE TREE NODES (parsenodes.h)
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 2fd0629..e1807fb 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -601,6 +601,17 @@ typedef struct ColumnDef
int location; /* parse location, or -1 if none/unknown */
} ColumnDef;
+typedef struct CreateStatsStmt
+{
+ NodeTag type;
+ List *defnames; /* qualified name (list of Value strings) */
+ RangeVar *relation; /* relation to build statistics on */
+ List *keys; /* String nodes naming referenced column(s) */
+ List *options; /* list of DefElem nodes */
+ bool if_not_exists; /* just do nothing if statistics already exists? */
+} CreateStatsStmt;
+
+
/*
* TableLikeClause - CREATE TABLE ( ... LIKE ... ) clause
*/
@@ -1410,6 +1421,7 @@ typedef enum ObjectType
OBJECT_RULE,
OBJECT_SCHEMA,
OBJECT_SEQUENCE,
+ OBJECT_STATISTICS,
OBJECT_TABCONSTRAINT,
OBJECT_TABLE,
OBJECT_TABLESPACE,
diff --git a/src/include/nodes/relation.h b/src/include/nodes/relation.h
index bdea72c..75c4752 100644
--- a/src/include/nodes/relation.h
+++ b/src/include/nodes/relation.h
@@ -541,6 +541,7 @@ typedef struct RelOptInfo
List *lateral_vars; /* LATERAL Vars and PHVs referenced by rel */
Relids lateral_referencers; /* rels that reference me laterally */
List *indexlist; /* list of IndexOptInfo */
+ List *mvstatlist; /* list of MVStatisticInfo */
BlockNumber pages; /* size estimates derived from pg_class */
double tuples;
double allvisfrac;
@@ -636,6 +637,33 @@ typedef struct IndexOptInfo
void (*amcostestimate) (); /* AM's cost estimator */
} IndexOptInfo;
+/*
+ * MVStatisticInfo
+ * Information about multivariate stats for planning/optimization
+ *
+ * This contains information about which columns are covered by the
+ * statistics (stakeys), which options were requested while adding the
+ * statistics (*_enabled), and which kinds of statistics were actually
+ * built and are available for the optimizer (*_built).
+ */
+typedef struct MVStatisticInfo
+{
+ NodeTag type;
+
+ Oid mvoid; /* OID of the statistics row */
+ RelOptInfo *rel; /* back-link to index's table */
+
+ /* enabled statistics */
+ bool deps_enabled; /* functional dependencies enabled */
+
+ /* built/available statistics */
+ bool deps_built; /* functional dependencies built */
+
+ /* columns in the statistics (attnums) */
+ int2vector *stakeys; /* attnums of the columns covered */
+
+} MVStatisticInfo;
+
/*
* EquivalenceClasses
diff --git a/src/include/utils/acl.h b/src/include/utils/acl.h
index 4e15a14..3e11253 100644
--- a/src/include/utils/acl.h
+++ b/src/include/utils/acl.h
@@ -330,6 +330,7 @@ extern bool pg_foreign_data_wrapper_ownercheck(Oid srv_oid, Oid roleid);
extern bool pg_foreign_server_ownercheck(Oid srv_oid, Oid roleid);
extern bool pg_event_trigger_ownercheck(Oid et_oid, Oid roleid);
extern bool pg_extension_ownercheck(Oid ext_oid, Oid roleid);
+extern bool pg_statistics_ownercheck(Oid stat_oid, Oid roleid);
extern bool has_createrole_privilege(Oid roleid);
extern bool has_bypassrls_privilege(Oid roleid);
diff --git a/src/include/utils/mvstats.h b/src/include/utils/mvstats.h
new file mode 100644
index 0000000..7837bc0
--- /dev/null
+++ b/src/include/utils/mvstats.h
@@ -0,0 +1,71 @@
+/*-------------------------------------------------------------------------
+ *
+ * mvstats.h
+ * Multivariate statistics and selectivity estimation functions.
+ *
+ *
+ * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/utils/mvstats.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef MVSTATS_H
+#define MVSTATS_H
+
+#include "fmgr.h"
+#include "commands/vacuum.h"
+
+
+#define MVSTATS_MAX_DIMENSIONS 8 /* max number of attributes */
+
+/*
+ * Functional dependencies, tracking column-level relationships (values
+ * in one column determine values in another one).
+ */
+typedef struct MVDependencyData {
+ int nattributes; /* number of attributes */
+ int16 attributes[1]; /* attribute numbers */
+} MVDependencyData;
+
+typedef MVDependencyData* MVDependency;
+
+typedef struct MVDependenciesData {
+ uint32 magic; /* magic constant marker */
+ uint32 type; /* type of MV Dependencies (BASIC) */
+ int32 ndeps; /* number of dependencies */
+ MVDependency deps[1]; /* XXX why not a pointer? */
+} MVDependenciesData;
+
+typedef MVDependenciesData* MVDependencies;
+
+#define MVSTAT_DEPS_MAGIC 0xB4549A2C /* marks serialized bytea */
+#define MVSTAT_DEPS_TYPE_BASIC 1 /* basic dependencies type */
+
+/*
+ * TODO Maybe fetching the histogram/MCV list separately is inefficient?
+ * Consider adding a single `fetch_stats` method, fetching all
+ * stats specified using flags (or something like that).
+ */
+
+bytea * serialize_mv_dependencies(MVDependencies dependencies);
+
+/* deserialization of stats (serialization is private to analyze) */
+MVDependencies deserialize_mv_dependencies(bytea * data);
+
+/* FIXME this probably belongs somewhere else (not to operations stats) */
+extern Datum pg_mv_stats_dependencies_info(PG_FUNCTION_ARGS);
+extern Datum pg_mv_stats_dependencies_show(PG_FUNCTION_ARGS);
+
+MVDependencies
+build_mv_dependencies(int numrows, HeapTuple *rows,
+ int2vector *attrs,
+ VacAttrStats **stats);
+
+void build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
+ int natts, VacAttrStats **vacattrstats);
+
+void update_mv_stats(Oid relid, MVDependencies dependencies, int2vector *attrs);
+
+#endif
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index f2bebf2..8771f9c 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -61,6 +61,7 @@ typedef struct RelationData
bool rd_isvalid; /* relcache entry is valid */
char rd_indexvalid; /* state of rd_indexlist: 0 = not valid, 1 =
* valid, 2 = temporarily forced */
+ bool rd_mvstatvalid; /* state of rd_mvstatlist: true/false */
/*
* rd_createSubid is the ID of the highest subtransaction the rel has
@@ -93,6 +94,9 @@ typedef struct RelationData
List *rd_indexlist; /* list of OIDs of indexes on relation */
Oid rd_oidindex; /* OID of unique index on OID, if any */
Oid rd_replidindex; /* OID of replica identity index, if any */
+
+ /* data managed by RelationGetMVStatList: */
+ List *rd_mvstatlist; /* list of OIDs of multivariate stats */
/* data managed by RelationGetIndexAttrBitmap: */
Bitmapset *rd_indexattr; /* identifies columns used in indexes */
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 1b48304..9f03c8d 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -38,6 +38,7 @@ extern void RelationClose(Relation relation);
* Routines to compute/retrieve additional cached information
*/
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetMVStatList(Relation relation);
extern Oid RelationGetOidIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
extern List *RelationGetIndexExpressions(Relation relation);
diff --git a/src/include/utils/syscache.h b/src/include/utils/syscache.h
index 256615b..0e0658d 100644
--- a/src/include/utils/syscache.h
+++ b/src/include/utils/syscache.h
@@ -66,6 +66,8 @@ enum SysCacheIdentifier
INDEXRELID,
LANGNAME,
LANGOID,
+ MVSTATNAMENSP,
+ MVSTATOID,
NAMESPACENAME,
NAMESPACEOID,
OPERNAMENSP,
diff --git a/src/test/regress/expected/mv_dependencies.out b/src/test/regress/expected/mv_dependencies.out
new file mode 100644
index 0000000..f54e1b7
--- /dev/null
+++ b/src/test/regress/expected/mv_dependencies.out
@@ -0,0 +1,150 @@
+-- data type passed by value
+CREATE TABLE functional_dependencies (
+ a INT,
+ b INT,
+ c INT
+);
+-- unknown column
+CREATE STATISTICS s1 ON functional_dependencies (unknown_column) WITH (dependencies);
+ERROR: column "unknown_column" referenced in statistics does not exist
+-- single column
+CREATE STATISTICS s1 ON functional_dependencies (a) WITH (dependencies);
+ERROR: multivariate stats require 2 or more columns
+-- single column, duplicated
+CREATE STATISTICS s1 ON functional_dependencies (a,a) WITH (dependencies);
+ERROR: duplicate column name in statistics definition
+-- two columns, one duplicated
+CREATE STATISTICS s1 ON functional_dependencies (a, a, b) WITH (dependencies);
+ERROR: duplicate column name in statistics definition
+-- unknown option
+CREATE STATISTICS s1 ON functional_dependencies (a, b, c) WITH (unknown_option);
+ERROR: unrecognized STATISTICS option "unknown_option"
+-- correct command
+CREATE STATISTICS s1 ON functional_dependencies (a, b, c) WITH (dependencies);
+-- random data (no functional dependencies)
+INSERT INTO functional_dependencies
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+ANALYZE functional_dependencies;
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+ deps_enabled | deps_built | pg_mv_stats_dependencies_show
+--------------+------------+-------------------------------
+ t | f |
+(1 row)
+
+TRUNCATE functional_dependencies;
+-- a => b, a => c, b => c
+INSERT INTO functional_dependencies
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE functional_dependencies;
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+ deps_enabled | deps_built | pg_mv_stats_dependencies_show
+--------------+------------+-------------------------------
+ t | t | (1) => 2, (1) => 3, (2) => 3
+(1 row)
+
+TRUNCATE functional_dependencies;
+-- a => b, a => c
+INSERT INTO functional_dependencies
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE functional_dependencies;
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+ deps_enabled | deps_built | pg_mv_stats_dependencies_show
+--------------+------------+-------------------------------
+ t | t | (1) => 2, (1) => 3
+(1 row)
+
+TRUNCATE functional_dependencies;
+-- a => b, a => c, b => c
+INSERT INTO functional_dependencies
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+ANALYZE functional_dependencies;
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+ deps_enabled | deps_built | pg_mv_stats_dependencies_show
+--------------+------------+-------------------------------
+ t | t | (1) => 2, (1) => 3, (2) => 3
+(1 row)
+
+DROP TABLE functional_dependencies;
+-- varlena type (text)
+CREATE TABLE functional_dependencies (
+ a TEXT,
+ b TEXT,
+ c TEXT
+);
+CREATE STATISTICS s2 ON functional_dependencies (a, b, c) WITH (dependencies);
+-- random data (no functional dependencies)
+INSERT INTO functional_dependencies
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+ANALYZE functional_dependencies;
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+ deps_enabled | deps_built | pg_mv_stats_dependencies_show
+--------------+------------+-------------------------------
+ t | f |
+(1 row)
+
+TRUNCATE functional_dependencies;
+-- a => b, a => c, b => c
+INSERT INTO functional_dependencies
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE functional_dependencies;
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+ deps_enabled | deps_built | pg_mv_stats_dependencies_show
+--------------+------------+-------------------------------
+ t | t | (1) => 2, (1) => 3, (2) => 3
+(1 row)
+
+TRUNCATE functional_dependencies;
+-- a => b, a => c
+INSERT INTO functional_dependencies
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE functional_dependencies;
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+ deps_enabled | deps_built | pg_mv_stats_dependencies_show
+--------------+------------+-------------------------------
+ t | t | (1) => 2, (1) => 3
+(1 row)
+
+TRUNCATE functional_dependencies;
+-- a => b, a => c, b => c
+INSERT INTO functional_dependencies
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+ANALYZE functional_dependencies;
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+ deps_enabled | deps_built | pg_mv_stats_dependencies_show
+--------------+------------+-------------------------------
+ t | t | (1) => 2, (1) => 3, (2) => 3
+(1 row)
+
+DROP TABLE functional_dependencies;
+-- NULL values (mix of int and text columns)
+CREATE TABLE functional_dependencies (
+ a INT,
+ b TEXT,
+ c INT,
+ d TEXT
+);
+CREATE STATISTICS s3 ON functional_dependencies (a, b, c, d) WITH (dependencies);
+INSERT INTO functional_dependencies
+ SELECT
+ mod(i, 100),
+ (CASE WHEN mod(i, 200) = 0 THEN NULL ELSE mod(i,200) END),
+ mod(i, 400),
+ (CASE WHEN mod(i, 300) = 0 THEN NULL ELSE mod(i,600) END)
+ FROM generate_series(1,10000) s(i);
+ANALYZE functional_dependencies;
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+ deps_enabled | deps_built | pg_mv_stats_dependencies_show
+--------------+------------+--------------------------------------------------
+ t | t | (2) => 1, (3) => 1, (3) => 2, (4) => 1, (4) => 2
+(1 row)
+
+DROP TABLE functional_dependencies;
diff --git a/src/test/regress/expected/object_address.out b/src/test/regress/expected/object_address.out
index 75751be..eb60960 100644
--- a/src/test/regress/expected/object_address.out
+++ b/src/test/regress/expected/object_address.out
@@ -35,6 +35,7 @@ ALTER DEFAULT PRIVILEGES FOR ROLE regtest_addr_user REVOKE DELETE ON TABLES FROM
CREATE TRANSFORM FOR int LANGUAGE SQL (
FROM SQL WITH FUNCTION varchar_transform(internal),
TO SQL WITH FUNCTION int4recv(internal));
+CREATE STATISTICS addr_nsp.gentable_stat ON addr_nsp.gentable(a,b) WITH (dependencies);
-- test some error cases
SELECT pg_get_object_address('stone', '{}', '{}');
ERROR: unrecognized object type "stone"
@@ -373,7 +374,8 @@ WITH objects (type, name, args) AS (VALUES
-- extension
-- event trigger
('policy', '{addr_nsp, gentable, genpol}', '{}'),
- ('transform', '{int}', '{sql}')
+ ('transform', '{int}', '{sql}'),
+ ('statistics', '{addr_nsp, gentable_stat}', '{}')
)
SELECT (pg_identify_object(addr1.classid, addr1.objid, addr1.subobjid)).*,
-- test roundtrip through pg_identify_object_as_address
@@ -420,13 +422,14 @@ SELECT (pg_identify_object(addr1.classid, addr1.objid, addr1.subobjid)).*,
trigger | | | t on addr_nsp.gentable | t
operator family | pg_catalog | integer_ops | pg_catalog.integer_ops USING btree | t
policy | | | genpol on addr_nsp.gentable | t
+ statistics | addr_nsp | gentable_stat | addr_nsp.gentable_stat | t
collation | pg_catalog | "default" | pg_catalog."default" | t
transform | | | for integer on language sql | t
text search dictionary | addr_nsp | addr_ts_dict | addr_nsp.addr_ts_dict | t
text search parser | addr_nsp | addr_ts_prs | addr_nsp.addr_ts_prs | t
text search configuration | addr_nsp | addr_ts_conf | addr_nsp.addr_ts_conf | t
text search template | addr_nsp | addr_ts_temp | addr_nsp.addr_ts_temp | t
-(41 rows)
+(42 rows)
---
--- Cleanup resources
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 22ea06c..06f2231 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1368,6 +1368,15 @@ pg_matviews| SELECT n.nspname AS schemaname,
LEFT JOIN pg_namespace n ON ((n.oid = c.relnamespace)))
LEFT JOIN pg_tablespace t ON ((t.oid = c.reltablespace)))
WHERE (c.relkind = 'm'::"char");
+pg_mv_stats| SELECT n.nspname AS schemaname,
+ c.relname AS tablename,
+ s.staname,
+ s.stakeys AS attnums,
+ length(s.stadeps) AS depsbytes,
+ pg_mv_stats_dependencies_info(s.stadeps) AS depsinfo
+ FROM ((pg_mv_statistic s
+ JOIN pg_class c ON ((c.oid = s.starelid)))
+ LEFT JOIN pg_namespace n ON ((n.oid = c.relnamespace)));
pg_policies| SELECT n.nspname AS schemaname,
c.relname AS tablename,
pol.polname AS policyname,
diff --git a/src/test/regress/expected/sanity_check.out b/src/test/regress/expected/sanity_check.out
index eb0bc88..92a0d8a 100644
--- a/src/test/regress/expected/sanity_check.out
+++ b/src/test/regress/expected/sanity_check.out
@@ -113,6 +113,7 @@ pg_inherits|t
pg_language|t
pg_largeobject|t
pg_largeobject_metadata|t
+pg_mv_statistic|t
pg_namespace|t
pg_opclass|t
pg_operator|t
diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule
index 7e9b319..097a04f 100644
--- a/src/test/regress/serial_schedule
+++ b/src/test/regress/serial_schedule
@@ -162,3 +162,4 @@ test: with
test: xml
test: event_trigger
test: stats
+test: mv_dependencies
diff --git a/src/test/regress/sql/mv_dependencies.sql b/src/test/regress/sql/mv_dependencies.sql
new file mode 100644
index 0000000..051633a
--- /dev/null
+++ b/src/test/regress/sql/mv_dependencies.sql
@@ -0,0 +1,142 @@
+-- data type passed by value
+CREATE TABLE functional_dependencies (
+ a INT,
+ b INT,
+ c INT
+);
+
+-- unknown column
+CREATE STATISTICS s1 ON functional_dependencies (unknown_column) WITH (dependencies);
+
+-- single column
+CREATE STATISTICS s1 ON functional_dependencies (a) WITH (dependencies);
+
+-- single column, duplicated
+CREATE STATISTICS s1 ON functional_dependencies (a,a) WITH (dependencies);
+
+-- two columns, one duplicated
+CREATE STATISTICS s1 ON functional_dependencies (a, a, b) WITH (dependencies);
+
+-- unknown option
+CREATE STATISTICS s1 ON functional_dependencies (a, b, c) WITH (unknown_option);
+
+-- correct command
+CREATE STATISTICS s1 ON functional_dependencies (a, b, c) WITH (dependencies);
+
+-- random data (no functional dependencies)
+INSERT INTO functional_dependencies
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+
+ANALYZE functional_dependencies;
+
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+
+TRUNCATE functional_dependencies;
+
+-- a => b, a => c, b => c
+INSERT INTO functional_dependencies
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+
+ANALYZE functional_dependencies;
+
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+
+TRUNCATE functional_dependencies;
+
+-- a => b, a => c
+INSERT INTO functional_dependencies
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE functional_dependencies;
+
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+
+TRUNCATE functional_dependencies;
+
+-- a => b, a => c, b => c
+INSERT INTO functional_dependencies
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+ANALYZE functional_dependencies;
+
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+
+DROP TABLE functional_dependencies;
+
+-- varlena type (text)
+CREATE TABLE functional_dependencies (
+ a TEXT,
+ b TEXT,
+ c TEXT
+);
+
+CREATE STATISTICS s2 ON functional_dependencies (a, b, c) WITH (dependencies);
+
+-- random data (no functional dependencies)
+INSERT INTO functional_dependencies
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+
+ANALYZE functional_dependencies;
+
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+
+TRUNCATE functional_dependencies;
+
+-- a => b, a => c, b => c
+INSERT INTO functional_dependencies
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+
+ANALYZE functional_dependencies;
+
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+
+TRUNCATE functional_dependencies;
+
+-- a => b, a => c
+INSERT INTO functional_dependencies
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE functional_dependencies;
+
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+
+TRUNCATE functional_dependencies;
+
+-- a => b, a => c, b => c
+INSERT INTO functional_dependencies
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+ANALYZE functional_dependencies;
+
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+
+DROP TABLE functional_dependencies;
+
+-- NULL values (mix of int and text columns)
+CREATE TABLE functional_dependencies (
+ a INT,
+ b TEXT,
+ c INT,
+ d TEXT
+);
+
+CREATE STATISTICS s3 ON functional_dependencies (a, b, c, d) WITH (dependencies);
+
+INSERT INTO functional_dependencies
+ SELECT
+ mod(i, 100),
+ (CASE WHEN mod(i, 200) = 0 THEN NULL ELSE mod(i,200) END),
+ mod(i, 400),
+ (CASE WHEN mod(i, 300) = 0 THEN NULL ELSE mod(i,600) END)
+ FROM generate_series(1,10000) s(i);
+
+ANALYZE functional_dependencies;
+
+SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
+ FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+
+DROP TABLE functional_dependencies;
diff --git a/src/test/regress/sql/object_address.sql b/src/test/regress/sql/object_address.sql
index 68e7cb0..3775b28 100644
--- a/src/test/regress/sql/object_address.sql
+++ b/src/test/regress/sql/object_address.sql
@@ -39,6 +39,7 @@ ALTER DEFAULT PRIVILEGES FOR ROLE regtest_addr_user REVOKE DELETE ON TABLES FROM
CREATE TRANSFORM FOR int LANGUAGE SQL (
FROM SQL WITH FUNCTION varchar_transform(internal),
TO SQL WITH FUNCTION int4recv(internal));
+CREATE STATISTICS addr_nsp.gentable_stat ON addr_nsp.gentable(a,b) WITH (dependencies);
-- test some error cases
SELECT pg_get_object_address('stone', '{}', '{}');
@@ -166,7 +167,8 @@ WITH objects (type, name, args) AS (VALUES
-- extension
-- event trigger
('policy', '{addr_nsp, gentable, genpol}', '{}'),
- ('transform', '{int}', '{sql}')
+ ('transform', '{int}', '{sql}'),
+ ('statistics', '{addr_nsp, gentable_stat}', '{}')
)
SELECT (pg_identify_object(addr1.classid, addr1.objid, addr1.subobjid)).*,
-- test roundtrip through pg_identify_object_as_address
--
2.5.0
[text/x-patch] 0003-clause-reduction-using-functional-dependencies.patch (32.7K, ../../7aa6145f-302b-2077-b961-702cc61710bb@2ndquadrant.com/4-0003-clause-reduction-using-functional-dependencies.patch)
download | inline diff:
From 6e3e16f46f93f045c137c070b48e387a470c3a08 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@pgaddict.com>
Date: Mon, 6 Apr 2015 19:42:18 +0200
Subject: [PATCH 3/9] clause reduction using functional dependencies
During planning, use functional dependencies to decide which clauses to
skip during cardinality estimation. Initial and rather simplistic
implementation.
This only works with regular WHERE clauses, not clauses used for join
clauses.
Note: The clause_is_mv_compatible() needs to identify the relation (so
that we can fetch the list of multivariate stats by OID).
planner_rt_fetch() seems like the appropriate way to get the relation
OID, but apparently it only works with simple vars. Maybe
examine_variable() would make this work with more complex vars too?
Includes regression tests analyzing functional dependencies (part of
ANALYZE) on several datasets (no dependencies, no transitive
dependencies, ...).
Checks that a query with conditions on two columns, where one (B) is
functionally dependent on the other one (A), correctly ignores the
clause on (B) and chooses bitmap index scan instead of plain index scan
(which is what happens otherwise, thanks to assumption of
independence).
Note: Functional dependencies only work with equality clauses, no
inequalities etc.
---
src/backend/optimizer/path/clausesel.c | 505 +++++++++++++++++++++++++-
src/backend/utils/mvstats/README.dependencies | 63 ++--
src/backend/utils/mvstats/README.stats | 36 ++
src/backend/utils/mvstats/common.c | 5 +-
src/backend/utils/mvstats/dependencies.c | 24 ++
src/include/utils/mvstats.h | 3 +-
src/include/utils/rel.h | 2 +-
src/test/regress/expected/mv_dependencies.out | 24 ++
src/test/regress/parallel_schedule | 3 +
src/test/regress/sql/mv_dependencies.sql | 15 +
10 files changed, 637 insertions(+), 43 deletions(-)
create mode 100644 src/backend/utils/mvstats/README.stats
diff --git a/src/backend/optimizer/path/clausesel.c b/src/backend/optimizer/path/clausesel.c
index 02660c2..a3afdf5 100644
--- a/src/backend/optimizer/path/clausesel.c
+++ b/src/backend/optimizer/path/clausesel.c
@@ -14,14 +14,19 @@
*/
#include "postgres.h"
+#include "access/sysattr.h"
+#include "catalog/pg_operator.h"
#include "nodes/makefuncs.h"
#include "optimizer/clauses.h"
#include "optimizer/cost.h"
#include "optimizer/pathnode.h"
#include "optimizer/plancat.h"
+#include "optimizer/var.h"
#include "utils/fmgroids.h"
#include "utils/lsyscache.h"
+#include "utils/mvstats.h"
#include "utils/selfuncs.h"
+#include "utils/typcache.h"
/*
@@ -41,6 +46,25 @@ typedef struct RangeQueryClause
static void addRangeClause(RangeQueryClause **rqlist, Node *clause,
bool varonleft, bool isLTsel, Selectivity s2);
+#define MV_CLAUSE_TYPE_FDEP 0x01
+
+static bool clause_is_mv_compatible(Node *clause, Index relid, AttrNumber *attnum);
+
+static Bitmapset *collect_mv_attnums(List *clauses, Index relid);
+
+static int count_mv_attnums(List *clauses, Index relid);
+
+static int count_varnos(List *clauses, Index *relid);
+
+static List *clauselist_apply_dependencies(PlannerInfo *root, List *clauses,
+ Index relid, List *stats);
+
+static bool has_stats(List *stats, int type);
+
+static List * find_stats(PlannerInfo *root, Index relid);
+
+static bool stats_type_matches(MVStatisticInfo *stat, int type);
+
/****************************************************************************
* ROUTINES TO COMPUTE SELECTIVITIES
@@ -60,7 +84,19 @@ static void addRangeClause(RangeQueryClause **rqlist, Node *clause,
* subclauses. However, that's only right if the subclauses have independent
* probabilities, and in reality they are often NOT independent. So,
* we want to be smarter where we can.
-
+ *
+ * The first thing we try to do is applying multivariate statistics, in a way
+ * that intends to minimize the overhead when there are no multivariate stats
+ * on the relation. Thus we do several simple (and inexpensive) checks first,
+ * to verify that suitable multivariate statistics exist.
+ *
+ * If we identify such multivariate statistics apply, we try to apply them.
+ * Currently we only have (soft) functional dependencies, so we try to reduce
+ * the list of clauses.
+ *
+ * Then we remove the clauses estimated using multivariate stats, and process
+ * the rest of the clauses using the regular per-column stats.
+ *
* Currently, the only extra smarts we have is to recognize "range queries",
* such as "x > 34 AND x < 42". Clauses are recognized as possible range
* query components if they are restriction opclauses whose operators have
@@ -99,6 +135,22 @@ clauselist_selectivity(PlannerInfo *root,
RangeQueryClause *rqlist = NULL;
ListCell *l;
+ /* processing mv stats */
+ Oid relid = InvalidOid;
+
+ /* list of multivariate stats on the relation */
+ List *stats = NIL;
+
+ /*
+ * To fetch the statistics, we first need to determine the rel. Currently
+ * point we only support estimates of simple restrictions with all Vars
+ * referencing a single baserel. However set_baserel_size_estimates() sets
+ * varRelid=0 so we have to actually inspect the clauses by pull_varnos
+ * and see if there's just a single varno referenced.
+ */
+ if ((count_varnos(clauses, &relid) == 1) && ((varRelid == 0) || (varRelid == relid)))
+ stats = find_stats(root, relid);
+
/*
* If there's exactly one clause, then no use in trying to match up pairs,
* so just go directly to clause_selectivity().
@@ -108,6 +160,24 @@ clauselist_selectivity(PlannerInfo *root,
varRelid, jointype, sjinfo);
/*
+ * Apply functional dependencies, but first check that there are some stats
+ * with functional dependencies built (by simply walking the stats list),
+ * and that there are at two or more attributes referenced by clauses that
+ * may be reduced using functional dependencies.
+ *
+ * We would find that anyway when trying to actually apply the functional
+ * dependencies, but let's do the cheap checks first.
+ *
+ * After applying the functional dependencies we get the remainig clauses
+ * that need to be estimated by other types of stats (MCV, histograms etc).
+ */
+ if (has_stats(stats, MV_CLAUSE_TYPE_FDEP) &&
+ (count_mv_attnums(clauses, relid) >= 2))
+ {
+ clauses = clauselist_apply_dependencies(root, clauses, relid, stats);
+ }
+
+ /*
* Initial scan over clauses. Anything that doesn't look like a potential
* rangequery clause gets multiplied into s1 and forgotten. Anything that
* does gets inserted into an rqlist entry.
@@ -763,3 +833,436 @@ clause_selectivity(PlannerInfo *root,
return s1;
}
+
+/*
+ * Collect attributes from mv-compatible clauses.
+ */
+static Bitmapset *
+collect_mv_attnums(List *clauses, Index relid)
+{
+ Bitmapset *attnums = NULL;
+ ListCell *l;
+
+ /*
+ * Walk through the clauses and identify the ones we can estimate
+ * using multivariate stats, and remember the relid/columns. We'll
+ * then cross-check if we have suitable stats, and only if needed
+ * we'll split the clauses into multivariate and regular lists.
+ *
+ * For now we're only interested in RestrictInfo nodes with nested
+ * OpExpr, using either a range or equality.
+ */
+ foreach (l, clauses)
+ {
+ AttrNumber attnum;
+ Node *clause = (Node *) lfirst(l);
+
+ /* ignore the result for now - we only need the info */
+ if (clause_is_mv_compatible(clause, relid, &attnum))
+ attnums = bms_add_member(attnums, attnum);
+ }
+
+ /*
+ * If there are not at least two attributes referenced by the clause(s),
+ * we can throw everything out (as we'll revert to simple stats).
+ */
+ if (bms_num_members(attnums) <= 1)
+ {
+ if (attnums != NULL)
+ pfree(attnums);
+ attnums = NULL;
+ }
+
+ return attnums;
+}
+
+/*
+ * Count the number of attributes in clauses compatible with multivariate stats.
+ */
+static int
+count_mv_attnums(List *clauses, Index relid)
+{
+ int c;
+ Bitmapset *attnums = collect_mv_attnums(clauses, relid);
+
+ c = bms_num_members(attnums);
+
+ bms_free(attnums);
+
+ return c;
+}
+
+/*
+ * Count varnos referenced in the clauses, and if there's a single varno then
+ * return the index in 'relid'.
+ */
+static int
+count_varnos(List *clauses, Index *relid)
+{
+ int cnt;
+ Bitmapset *varnos = NULL;
+
+ varnos = pull_varnos((Node *) clauses);
+ cnt = bms_num_members(varnos);
+
+ /* if there's a single varno in the clauses, remember it */
+ if (bms_num_members(varnos) == 1)
+ *relid = bms_singleton_member(varnos);
+
+ bms_free(varnos);
+
+ return cnt;
+}
+
+typedef struct
+{
+ Index varno; /* relid we're interested in */
+ Bitmapset *varattnos; /* attnums referenced by the clauses */
+} mv_compatible_context;
+
+/*
+ * Recursive walker that checks compatibility of the clause with multivariate
+ * statistics, and collects attnums from the Vars.
+ *
+ * XXX The original idea was to combine this with expression_tree_walker, but
+ * I've been unable to make that work - seems that does not quite allow
+ * checking the structure. Hence the explicit calls to the walker.
+ */
+static bool
+mv_compatible_walker(Node *node, mv_compatible_context *context)
+{
+ if (node == NULL)
+ return false;
+
+ if (IsA(node, RestrictInfo))
+ {
+ RestrictInfo *rinfo = (RestrictInfo *) node;
+
+ /* Pseudoconstants are not really interesting here. */
+ if (rinfo->pseudoconstant)
+ return true;
+
+ /* clauses referencing multiple varnos are incompatible */
+ if (bms_membership(rinfo->clause_relids) != BMS_SINGLETON)
+ return true;
+
+ /* check the clause inside the RestrictInfo */
+ return mv_compatible_walker((Node*)rinfo->clause, (void *) context);
+ }
+
+ if (IsA(node, Var))
+ {
+ Var * var = (Var*)node;
+
+ /*
+ * Also, the variable needs to reference the right relid (this might be
+ * unnecessary given the other checks, but let's be sure).
+ */
+ if (var->varno != context->varno)
+ return true;
+
+ /* Also skip system attributes (we don't allow stats on those). */
+ if (! AttrNumberIsForUserDefinedAttr(var->varattno))
+ return true;
+
+ /* Seems fine, so let's remember the attnum. */
+ context->varattnos = bms_add_member(context->varattnos, var->varattno);
+
+ return false;
+ }
+
+ /*
+ * And finally the operator expressions - we only allow simple expressions
+ * with two arguments, where one is a Var and the other is a constant, and
+ * it's a simple comparison (which we detect using estimator function).
+ */
+ if (is_opclause(node))
+ {
+ OpExpr *expr = (OpExpr *) node;
+ Var *var;
+ bool varonleft = true;
+ bool ok;
+
+ /*
+ * Only expressions with two arguments are considered compatible.
+ *
+ * XXX Possibly unnecessary (can OpExpr have different arg count?).
+ */
+ if (list_length(expr->args) != 2)
+ return true;
+
+ /* see if it actually has the right */
+ ok = (NumRelids((Node*)expr) == 1) &&
+ (is_pseudo_constant_clause(lsecond(expr->args)) ||
+ (varonleft = false,
+ is_pseudo_constant_clause(linitial(expr->args))));
+
+ /* unsupported structure (two variables or so) */
+ if (! ok)
+ return true;
+
+ /*
+ * If it's not a "<" or ">" or "=" operator, just ignore the clause.
+ * Otherwise note the relid and attnum for the variable. This uses the
+ * function for estimating selectivity, ont the operator directly (a bit
+ * awkward, but well ...).
+ */
+ switch (get_oprrest(expr->opno))
+ {
+ case F_EQSEL:
+
+ /* equality conditions are compatible with all statistics */
+ break;
+
+ default:
+
+ /* unknown estimator */
+ return true;
+ }
+
+ var = (varonleft) ? linitial(expr->args) : lsecond(expr->args);
+
+ return mv_compatible_walker((Node *) var, context);
+ }
+
+ /* Node not explicitly supported, so terminate */
+ return true;
+}
+
+/*
+ * Determines whether the clause is compatible with multivariate stats,
+ * and if it is, returns some additional information - varno (index
+ * into simple_rte_array) and a bitmap of attributes. This is then
+ * used to fetch related multivariate statistics.
+ *
+ * At this moment we only support basic conditions of the form
+ *
+ * variable OP constant
+ *
+ * where OP is one of [=,<,<=,>=,>] (which is however determined by
+ * looking at the associated function for estimating selectivity, just
+ * like with the single-dimensional case).
+ *
+ * TODO Support 'OR clauses' - shouldn't be all that difficult to
+ * evaluate them using multivariate stats.
+ */
+static bool
+clause_is_mv_compatible(Node *clause, Index relid, AttrNumber *attnum)
+{
+ mv_compatible_context context;
+
+ context.varno = relid;
+ context.varattnos = NULL; /* no attnums */
+
+ if (mv_compatible_walker(clause, (void *) &context))
+ return false;
+
+ /* remember the newly collected attnums */
+ *attnum = bms_singleton_member(context.varattnos);
+
+ return true;
+}
+
+
+/*
+ * Reduce clauses using functional dependencies
+ */
+static List*
+fdeps_reduce_clauses(List *clauses, Index relid, Bitmapset *reduced_attnums)
+{
+ ListCell *lc;
+ List *reduced_clauses = NIL;
+
+ foreach (lc, clauses)
+ {
+ AttrNumber attnum = InvalidAttrNumber;
+ Node * clause = (Node*)lfirst(lc);
+
+ /* ignore clauses that are not compatible with functional dependencies */
+ if (! clause_is_mv_compatible(clause, relid, &attnum))
+ reduced_clauses = lappend(reduced_clauses, clause);
+
+ /* for equality clauses, only keep those not on reduced attributes */
+ if (! bms_is_member(attnum, reduced_attnums))
+ reduced_clauses = lappend(reduced_clauses, clause);
+ }
+
+ return reduced_clauses;
+}
+
+/*
+ * decide which attributes are redundant (for equality clauses)
+ *
+ * We try to apply all functional dependencies available, and for each one we
+ * check if it matches attnums from equality clauses, but only those not yet
+ * reduced.
+ *
+ * XXX Not sure if the order in which we apply the dependencies matters.
+ *
+ * XXX We do not combine functional dependencies from separate stats. That is
+ * if we have dependencies on [a,b] and [b,c], then we don't deduce
+ * a->c from a->b and b->c. Computing such transitive closure is a possible
+ * future improvement.
+ */
+static Bitmapset *
+fdeps_reduce_attnums(List *stats, Bitmapset *attnums)
+{
+ ListCell *lc;
+ Bitmapset *reduced = NULL;
+
+ foreach (lc, stats)
+ {
+ int i;
+ MVDependencies dependencies = NULL;
+ MVStatisticInfo *info = (MVStatisticInfo *)lfirst(lc);
+
+ /* skip statistics without dependencies */
+ if (! stats_type_matches(info, MV_CLAUSE_TYPE_FDEP))
+ continue;
+
+ /* fetch and deserialize dependencies */
+ dependencies = load_mv_dependencies(info->mvoid);
+
+ for (i = 0; i < dependencies->ndeps; i++)
+ {
+ int j;
+ bool matched = true;
+ MVDependency dep = dependencies->deps[i];
+
+ /* we don't bother to break the loop early (only few attributes) */
+ for (j = 0; j < dep->nattributes; j++)
+ {
+ if (! bms_is_member(dep->attributes[j], attnums))
+ matched = false;
+
+ if (bms_is_member(dep->attributes[j], reduced))
+ matched = false;
+ }
+
+ /* if dependency applies, mark the last attribute as reduced */
+ if (matched)
+ reduced = bms_add_member(reduced,
+ dep->attributes[dep->nattributes-1]);
+ }
+ }
+
+ return reduced;
+}
+
+/*
+ * reduce list of equality clauses using soft functional dependencies
+ *
+ * We simply walk through list of functional dependencies, and for each one we
+ * check whether the dependency 'matches' the clauses, i.e. if there's a clause
+ * matching the condition. If yes, we attempt to remove all clauses matching
+ * the implied part of the dependency from the list.
+ *
+ * This only reduces equality clauses, and ignores all the other types. We might
+ * extend it to handle IS NULL clause, in the future.
+ *
+ * We also assume the equality clauses are 'compatible'. For example we can't
+ * identify when the clauses use a mismatching zip code and city name. In such
+ * case the usual approach (product of selectivities) would produce a better
+ * estimate, although mostly by chance.
+ *
+ * The implementation needs to be careful about cyclic dependencies, e.g. when
+ *
+ * (a -> b) and (b -> a)
+ *
+ * at the same time, which means there's 1:1 relationship between te columns.
+ * In this case we must not reduce clauses on both attributes at the same time.
+ *
+ * TODO Currently we only apply functional dependencies at the same level, but
+ * maybe we could transfer the clauses from upper levels to the subtrees?
+ * For example let's say we have (a->b) dependency, and condition
+ *
+ * (a=1) AND (b=2 OR c=3)
+ *
+ * Currently, we won't be able to perform any reduction, because we'll
+ * consider (a=1) and (b=2 OR c=3) independently. But maybe we could pass
+ * (a=1) into the other expression, and only check it against conditions
+ * of the functional dependencies?
+ *
+ * In this case we'd end up with
+ *
+ * (a=1)
+ *
+ * as we'd consider (b=2) implied thanks to the rule, rendering the whole
+ * OR clause valid.
+ */
+static List *
+clauselist_apply_dependencies(PlannerInfo *root, List *clauses,
+ Index relid, List *stats)
+{
+ Bitmapset *clause_attnums = NULL;
+ Bitmapset *reduced_attnums = NULL;
+
+ /*
+ * Is there at least one statistics with functional dependencies?
+ * If not, return the original clauses right away.
+ *
+ * XXX Isn't this a bit pointless, thanks to exactly the same check in
+ * clauselist_selectivity()? Can we trigger the condition here?
+ */
+ if (! has_stats(stats, MV_CLAUSE_TYPE_FDEP))
+ return clauses;
+
+ /* collect attnums from clauses compatible with dependencies (equality) */
+ clause_attnums = collect_mv_attnums(clauses, relid);
+
+ /* decide which attnums may be eliminated */
+ reduced_attnums = fdeps_reduce_attnums(stats, clause_attnums);
+
+ /*
+ * Walk through the clauses, and see which other clauses we may reduce.
+ */
+ clauses = fdeps_reduce_clauses(clauses, relid, reduced_attnums);
+
+ bms_free(clause_attnums);
+ bms_free(reduced_attnums);
+
+ return clauses;
+}
+
+/*
+ * Check that there are stats with at least one of the requested types.
+ */
+static bool
+stats_type_matches(MVStatisticInfo *stat, int type)
+{
+ if ((type & MV_CLAUSE_TYPE_FDEP) && stat->deps_built)
+ return true;
+
+ return false;
+}
+
+/*
+ * Check that there are stats with at least one of the requested types.
+ */
+static bool
+has_stats(List *stats, int type)
+{
+ ListCell *s;
+
+ foreach (s, stats)
+ {
+ MVStatisticInfo *stat = (MVStatisticInfo *)lfirst(s);
+
+ /* terminate if we've found at least one matching statistics */
+ if (stats_type_matches(stat, type))
+ return true;
+ }
+
+ return false;
+}
+
+/*
+ * Lookups stats for a given baserel.
+ */
+static List *
+find_stats(PlannerInfo *root, Index relid)
+{
+ Assert(root->simple_rel_array[relid] != NULL);
+
+ return root->simple_rel_array[relid]->mvstatlist;
+}
diff --git a/src/backend/utils/mvstats/README.dependencies b/src/backend/utils/mvstats/README.dependencies
index 1f96fbc..f248459 100644
--- a/src/backend/utils/mvstats/README.dependencies
+++ b/src/backend/utils/mvstats/README.dependencies
@@ -156,37 +156,24 @@ estimates - especially compared to histograms, that are quite bad in estimating
equality clauses.
-Limitations
------------
-
-Let's see the main liminations of functional dependencies, especially those
-related to the current implementation.
+Multi-column dependencies
+-------------------------
-The current implementation supports only dependencies between two columns, but
-this is merely a simplification of the initial implementation. It's certainly
-useful to mine for dependencies involving multiple columns on the 'left' side,
-i.e. a condition for the dependency. That is dependencies like (a,b -> c).
+The implementation supports dependencies with multiple columns on the left side
+(i.e. condition of the dependency). The detection starts from dependencies with
+a single condition, and then proceeds to higher condition counts.
-The implementation may/should be smart enough not to mine redundant conditions,
-e.g. (a->b) and (a,c -> b), because the latter is a trivial consequence of the
-former one (if values of 'a' determine 'b', adding another column won't change
-that relationship). The ANALYZE should first analyze 1:1 dependencies, then 2:1
-dependencies (and skip the already identified ones), etc.
+It also detects dependencies that are implied by already identified ones, and
+ignores them. For example if we know that (a->b) then we won't add (a,c->b) as
+this dependency is a trivial consequence of (a->b).
-For example the dependency
+For a more practical example, consider these two dependencies
(city name -> zip code)
-
-is much stronger, i.e. whenever it hold, then
-
(city name, state name -> zip code)
-holds too. But in case there are cities with the same name in different states,
-then only the latter dependency will be valid.
-
-Of course, there probably are cities with the same name within a single state,
-but hopefully this is relatively rare occurence (and thus we'll still detect
-the 'soft' dependency).
+We could say that the former dependency is stronger because if it's valid, then
+the second dependency is valid too.
Handling multiple columns on the right side of the dependency, is not necessary,
as those dependencies may be simply decomposed into a set of dependencies with
@@ -199,24 +186,22 @@ is exactly the same as
(a -> b) & (a -> c)
Of course, storing the first form may be more efficient thant storing multiple
-'simple' dependencies separately.
-
+'simple' dependencies separately. This is left as a future work.
-TODO Support dependencies with multiple columns on left/right.
-TODO Investigate using histogram and MCV list to verify the dependencies.
+Future work
+-----------
-TODO Investigate statistical testing of the distribution (to decide whether it
- makes sense to build the histogram/MCV list).
+* Investigate using histogram and MCV list to verify the dependencies.
-TODO Using a min/max of selectivities would probably make more sense for the
- associated columns.
+* Investigate statistical testing of the distribution (to decide whether it
+ makes sense to build the histogram/MCV list).
-TODO Consider eliminating the implied columns from the histogram and MCV lists
- (but maybe that's not a good idea, because that'd make it impossible to use
- these stats for non-equality clauses and also it wouldn't be possible to
- use the stats for verification of the dependencies).
+* Consider eliminating the implied columns from the histogram and MCV lists
+ (but maybe that's not a good idea, because that'd make it impossible to use
+ these stats for non-equality clauses and also it wouldn't be possible to
+ use the stats for verification of the dependencies).
-TODO The reduction probably might be extended to also handle IS NULL clauses,
- assuming we fix the ANALYZE to properly handle NULL values. We however
- won't be able to reduce IS NOT NULL (unless I'm missing something).
+* The reduction probably might be extended to also handle IS NULL clauses,
+ assuming we fix the ANALYZE to properly handle NULL values. We however
+ won't be able to reduce IS NOT NULL (unless I'm missing something).
diff --git a/src/backend/utils/mvstats/README.stats b/src/backend/utils/mvstats/README.stats
new file mode 100644
index 0000000..a38ea7b
--- /dev/null
+++ b/src/backend/utils/mvstats/README.stats
@@ -0,0 +1,36 @@
+Multivariate statististics
+==========================
+
+When estimating various quantities (e.g. condition selectivities) the default
+approach relies on the assumption of independence. In practice that's often
+not true, resulting in estimation errors.
+
+Multivariate stats track different types of dependencies between the columns,
+hopefully improving the estimates.
+
+Currently we only have one kind of multivariate statistics - soft functional
+dependencies, and we use it to improve estimates of equality clauses. See
+README.dependencies for details.
+
+
+Selectivity estimation
+----------------------
+
+When estimating selectivity, we aim to achieve several things:
+
+ (a) maximize the estimate accuracy
+
+ (b) minimize the overhead, especially when no suitable multivariate stats
+ exist (so if you are not using multivariate stats, there's no overhead)
+
+This clauselist_selectivity() performs several inexpensive checks first, before
+even attempting to do the more expensive estimation.
+
+ (1) check if there are multivariate stats on the relation
+
+ (2) check there are at least two attributes referenced by clauses compatible
+ with multivariate statistics (equality clauses for func. dependencies)
+
+ (3) perform reduction of equality clauses using func. dependencies
+
+ (4) estimate the reduced list of clauses using regular statistics
diff --git a/src/backend/utils/mvstats/common.c b/src/backend/utils/mvstats/common.c
index 82f2177..dcb7c78 100644
--- a/src/backend/utils/mvstats/common.c
+++ b/src/backend/utils/mvstats/common.c
@@ -84,7 +84,8 @@ build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
/*
* Analyze functional dependencies of columns.
*/
- deps = build_mv_dependencies(numrows, rows, attrs, stats);
+ if (stat->deps_enabled)
+ deps = build_mv_dependencies(numrows, rows, attrs, stats);
/* store the histogram / MCV list in the catalog */
update_mv_stats(stat->mvoid, deps, attrs);
@@ -163,6 +164,7 @@ list_mv_stats(Oid relid)
info->mvoid = HeapTupleGetOid(htup);
info->stakeys = buildint2vector(stats->stakeys.values, stats->stakeys.dim1);
+ info->deps_enabled = stats->deps_enabled;
info->deps_built = stats->deps_built;
result = lappend(result, info);
@@ -274,6 +276,7 @@ compare_scalars_partition(const void *a, const void *b, void *arg)
return ApplySortComparator(da, false, db, false, ssup);
}
+
/* initialize multi-dimensional sort */
MultiSortSupport
multi_sort_init(int ndims)
diff --git a/src/backend/utils/mvstats/dependencies.c b/src/backend/utils/mvstats/dependencies.c
index 5437bdf..412dc30 100644
--- a/src/backend/utils/mvstats/dependencies.c
+++ b/src/backend/utils/mvstats/dependencies.c
@@ -684,3 +684,27 @@ pg_mv_stats_dependencies_show(PG_FUNCTION_ARGS)
PG_RETURN_TEXT_P(cstring_to_text(buf.data));
}
+
+MVDependencies
+load_mv_dependencies(Oid mvoid)
+{
+ bool isnull = false;
+ Datum deps;
+
+ /* Prepare to scan pg_mv_statistic for entries having indrelid = this rel. */
+ HeapTuple htup = SearchSysCache1(MVSTATOID, ObjectIdGetDatum(mvoid));
+
+#ifdef USE_ASSERT_CHECKING
+ Form_pg_mv_statistic mvstat = (Form_pg_mv_statistic) GETSTRUCT(htup);
+ Assert(mvstat->deps_enabled && mvstat->deps_built);
+#endif
+
+ deps = SysCacheGetAttr(MVSTATOID, htup,
+ Anum_pg_mv_statistic_stadeps, &isnull);
+
+ Assert(!isnull);
+
+ ReleaseSysCache(htup);
+
+ return deserialize_mv_dependencies(DatumGetByteaP(deps));
+}
diff --git a/src/include/utils/mvstats.h b/src/include/utils/mvstats.h
index 7837bc0..ec55a09 100644
--- a/src/include/utils/mvstats.h
+++ b/src/include/utils/mvstats.h
@@ -17,7 +17,6 @@
#include "fmgr.h"
#include "commands/vacuum.h"
-
#define MVSTATS_MAX_DIMENSIONS 8 /* max number of attributes */
/*
@@ -49,6 +48,8 @@ typedef MVDependenciesData* MVDependencies;
* stats specified using flags (or something like that).
*/
+MVDependencies load_mv_dependencies(Oid mvoid);
+
bytea * serialize_mv_dependencies(MVDependencies dependencies);
/* deserialization of stats (serialization is private to analyze) */
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index 8771f9c..d09ba25 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -94,7 +94,7 @@ typedef struct RelationData
List *rd_indexlist; /* list of OIDs of indexes on relation */
Oid rd_oidindex; /* OID of unique index on OID, if any */
Oid rd_replidindex; /* OID of replica identity index, if any */
-
+
/* data managed by RelationGetMVStatList: */
List *rd_mvstatlist; /* list of OIDs of multivariate stats */
diff --git a/src/test/regress/expected/mv_dependencies.out b/src/test/regress/expected/mv_dependencies.out
index f54e1b7..ee8a9b2 100644
--- a/src/test/regress/expected/mv_dependencies.out
+++ b/src/test/regress/expected/mv_dependencies.out
@@ -58,8 +58,10 @@ SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
TRUNCATE functional_dependencies;
-- a => b, a => c, b => c
+-- check explain (expect bitmap index scan, not plain index scan)
INSERT INTO functional_dependencies
SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX fdeps_idx ON functional_dependencies (a, b);
ANALYZE functional_dependencies;
SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
@@ -68,6 +70,16 @@ SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
t | t | (1) => 2, (1) => 3, (2) => 3
(1 row)
+EXPLAIN (COSTS off)
+ SELECT * FROM functional_dependencies WHERE a = 10 AND b = 5;
+ QUERY PLAN
+---------------------------------------------
+ Bitmap Heap Scan on functional_dependencies
+ Recheck Cond: ((a = 10) AND (b = 5))
+ -> Bitmap Index Scan on fdeps_idx
+ Index Cond: ((a = 10) AND (b = 5))
+(4 rows)
+
DROP TABLE functional_dependencies;
-- varlena type (text)
CREATE TABLE functional_dependencies (
@@ -113,8 +125,10 @@ SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
TRUNCATE functional_dependencies;
-- a => b, a => c, b => c
+-- check explain (expect bitmap index scan, not plain index scan)
INSERT INTO functional_dependencies
SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX fdeps_idx ON functional_dependencies (a, b);
ANALYZE functional_dependencies;
SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
@@ -123,6 +137,16 @@ SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
t | t | (1) => 2, (1) => 3, (2) => 3
(1 row)
+EXPLAIN (COSTS off)
+ SELECT * FROM functional_dependencies WHERE a = '10' AND b = '5';
+ QUERY PLAN
+------------------------------------------------------------
+ Bitmap Heap Scan on functional_dependencies
+ Recheck Cond: ((a = '10'::text) AND (b = '5'::text))
+ -> Bitmap Index Scan on fdeps_idx
+ Index Cond: ((a = '10'::text) AND (b = '5'::text))
+(4 rows)
+
DROP TABLE functional_dependencies;
-- NULL values (mix of int and text columns)
CREATE TABLE functional_dependencies (
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index bec0316..4f2ffb8 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -110,3 +110,6 @@ test: event_trigger
# run stats by itself because its delay may be insufficient under heavy load
test: stats
+
+# run tests of multivariate stats
+test: mv_dependencies
diff --git a/src/test/regress/sql/mv_dependencies.sql b/src/test/regress/sql/mv_dependencies.sql
index 051633a..8ba72a4 100644
--- a/src/test/regress/sql/mv_dependencies.sql
+++ b/src/test/regress/sql/mv_dependencies.sql
@@ -56,13 +56,20 @@ SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
TRUNCATE functional_dependencies;
-- a => b, a => c, b => c
+-- check explain (expect bitmap index scan, not plain index scan)
INSERT INTO functional_dependencies
SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+
+CREATE INDEX fdeps_idx ON functional_dependencies (a, b);
+
ANALYZE functional_dependencies;
SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+EXPLAIN (COSTS off)
+ SELECT * FROM functional_dependencies WHERE a = 10 AND b = 5;
+
DROP TABLE functional_dependencies;
-- varlena type (text)
@@ -99,6 +106,7 @@ TRUNCATE functional_dependencies;
-- a => b, a => c
INSERT INTO functional_dependencies
SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+
ANALYZE functional_dependencies;
SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
@@ -107,13 +115,20 @@ SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
TRUNCATE functional_dependencies;
-- a => b, a => c, b => c
+-- check explain (expect bitmap index scan, not plain index scan)
INSERT INTO functional_dependencies
SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+
+CREATE INDEX fdeps_idx ON functional_dependencies (a, b);
+
ANALYZE functional_dependencies;
SELECT deps_enabled, deps_built, pg_mv_stats_dependencies_show(stadeps)
FROM pg_mv_statistic WHERE starelid = 'functional_dependencies'::regclass;
+EXPLAIN (COSTS off)
+ SELECT * FROM functional_dependencies WHERE a = '10' AND b = '5';
+
DROP TABLE functional_dependencies;
-- NULL values (mix of int and text columns)
--
2.5.0
[text/x-patch] 0004-multivariate-MCV-lists.patch (113.1K, ../../7aa6145f-302b-2077-b961-702cc61710bb@2ndquadrant.com/5-0004-multivariate-MCV-lists.patch)
download | inline diff:
From 6aba7480c5a4fd56896bf1a2d320e19ea231225d Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@pgaddict.com>
Date: Mon, 6 Apr 2015 16:52:15 +0200
Subject: [PATCH 4/9] multivariate MCV lists
- extends the pg_mv_statistic catalog (add 'mcv' fields)
- building the MCV lists during ANALYZE
- simple estimation while planning the queries
Includes regression tests, mostly equal to regression tests for
functional dependencies.
---
doc/src/sgml/ref/create_statistics.sgml | 43 ++
src/backend/catalog/system_views.sql | 4 +-
src/backend/commands/statscmds.c | 45 +-
src/backend/nodes/outfuncs.c | 2 +
src/backend/optimizer/path/clausesel.c | 800 +++++++++++++++++++++-
src/backend/optimizer/util/plancat.c | 4 +-
src/backend/utils/mvstats/Makefile | 2 +-
src/backend/utils/mvstats/README.mcv | 137 ++++
src/backend/utils/mvstats/README.stats | 89 ++-
src/backend/utils/mvstats/common.c | 133 +++-
src/backend/utils/mvstats/common.h | 15 +
src/backend/utils/mvstats/mcv.c | 1120 +++++++++++++++++++++++++++++++
src/bin/psql/describe.c | 25 +-
src/include/catalog/pg_mv_statistic.h | 18 +-
src/include/catalog/pg_proc.h | 4 +
src/include/nodes/relation.h | 2 +
src/include/utils/mvstats.h | 77 ++-
src/test/regress/expected/mv_mcv.out | 207 ++++++
src/test/regress/expected/rules.out | 4 +-
src/test/regress/parallel_schedule | 2 +-
src/test/regress/serial_schedule | 1 +
src/test/regress/sql/mv_mcv.sql | 178 +++++
22 files changed, 2847 insertions(+), 65 deletions(-)
create mode 100644 src/backend/utils/mvstats/README.mcv
create mode 100644 src/backend/utils/mvstats/mcv.c
create mode 100644 src/test/regress/expected/mv_mcv.out
create mode 100644 src/test/regress/sql/mv_mcv.sql
diff --git a/doc/src/sgml/ref/create_statistics.sgml b/doc/src/sgml/ref/create_statistics.sgml
index ff09fa5..d6973e8 100644
--- a/doc/src/sgml/ref/create_statistics.sgml
+++ b/doc/src/sgml/ref/create_statistics.sgml
@@ -132,6 +132,24 @@ CREATE STATISTICS [ IF NOT EXISTS ] <replaceable class="PARAMETER">statistics_na
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>max_mcv_items</> (<type>integer</>)</term>
+ <listitem>
+ <para>
+ Maximum number of MCV list items.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>mcv</> (<type>boolean</>)</term>
+ <listitem>
+ <para>
+ Enables MCV list for the statistics.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect2>
@@ -177,6 +195,31 @@ EXPLAIN ANALYZE SELECT * FROM t1 WHERE (a = 1) AND (b = 2);
</programlisting>
</para>
+ <para>
+ Create table <structname>t2</> with two perfectly correlated columns
+ (containing identical data), and a MCV list on those columns:
+
+<programlisting>
+CREATE TABLE t2 (
+ a int,
+ b int
+);
+
+INSERT INTO t2 SELECT mod(i,100), mod(i,100)
+ FROM generate_series(1,1000000) s(i);
+
+CREATE STATISTICS s2 ON t2 (a, b) WITH (mcv);
+
+ANALYZE t2;
+
+-- valid combination (found in MCV)
+EXPLAIN ANALYZE SELECT * FROM t2 WHERE (a = 1) AND (b = 1);
+
+-- invalid combination (not found in MCV)
+EXPLAIN ANALYZE SELECT * FROM t2 WHERE (a = 1) AND (b = 2);
+</programlisting>
+ </para>
+
</refsect1>
<refsect1>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 31dbb2c..5c40334 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -165,7 +165,9 @@ CREATE VIEW pg_mv_stats AS
S.staname AS staname,
S.stakeys AS attnums,
length(S.stadeps) as depsbytes,
- pg_mv_stats_dependencies_info(S.stadeps) as depsinfo
+ pg_mv_stats_dependencies_info(S.stadeps) as depsinfo,
+ length(S.stamcv) AS mcvbytes,
+ pg_mv_stats_mcvlist_info(S.stamcv) AS mcvinfo
FROM (pg_mv_statistic S JOIN pg_class C ON (C.oid = S.starelid))
LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace);
diff --git a/src/backend/commands/statscmds.c b/src/backend/commands/statscmds.c
index f43b053..c480fbe 100644
--- a/src/backend/commands/statscmds.c
+++ b/src/backend/commands/statscmds.c
@@ -70,7 +70,13 @@ CreateStatistics(CreateStatsStmt *stmt)
ObjectAddress parentobject, childobject;
/* by default build nothing */
- bool build_dependencies = false;
+ bool build_dependencies = false,
+ build_mcv = false;
+
+ int32 max_mcv_items = -1;
+
+ /* options required because of other options */
+ bool require_mcv = false;
Assert(IsA(stmt, CreateStatsStmt));
@@ -146,6 +152,29 @@ CreateStatistics(CreateStatsStmt *stmt)
if (strcmp(opt->defname, "dependencies") == 0)
build_dependencies = defGetBoolean(opt);
+ else if (strcmp(opt->defname, "mcv") == 0)
+ build_mcv = defGetBoolean(opt);
+ else if (strcmp(opt->defname, "max_mcv_items") == 0)
+ {
+ max_mcv_items = defGetInt32(opt);
+
+ /* this option requires 'mcv' to be enabled */
+ require_mcv = true;
+
+ /* sanity check */
+ if (max_mcv_items < MVSTAT_MCVLIST_MIN_ITEMS)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("max number of MCV items must be at least %d",
+ MVSTAT_MCVLIST_MIN_ITEMS)));
+
+ else if (max_mcv_items > MVSTAT_MCVLIST_MAX_ITEMS)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("max number of MCV items is %d",
+ MVSTAT_MCVLIST_MAX_ITEMS)));
+
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -154,10 +183,16 @@ CreateStatistics(CreateStatsStmt *stmt)
}
/* check that at least some statistics were requested */
- if (! build_dependencies)
+ if (! (build_dependencies || build_mcv))
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("no statistics type (dependencies, mcv) was requested")));
+
+ /* now do some checking of the options */
+ if (require_mcv && (! build_mcv))
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("no statistics type (dependencies) was requested")));
+ errmsg("option 'mcv' is required by other options(s)")));
/* sort the attnums and build int2vector */
qsort(attnums, numcols, sizeof(int16), compare_int16);
@@ -178,8 +213,12 @@ CreateStatistics(CreateStatsStmt *stmt)
values[Anum_pg_mv_statistic_stakeys -1] = PointerGetDatum(stakeys);
values[Anum_pg_mv_statistic_deps_enabled -1] = BoolGetDatum(build_dependencies);
+ values[Anum_pg_mv_statistic_mcv_enabled -1] = BoolGetDatum(build_mcv);
+
+ values[Anum_pg_mv_statistic_mcv_max_items -1] = Int32GetDatum(max_mcv_items);
nulls[Anum_pg_mv_statistic_stadeps -1] = true;
+ nulls[Anum_pg_mv_statistic_stamcv -1] = true;
/* insert the tuple into pg_mv_statistic */
mvstatrel = heap_open(MvStatisticRelationId, RowExclusiveLock);
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 07206d7..333e24b 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2162,9 +2162,11 @@ _outMVStatisticInfo(StringInfo str, const MVStatisticInfo *node)
/* enabled statistics */
WRITE_BOOL_FIELD(deps_enabled);
+ WRITE_BOOL_FIELD(mcv_enabled);
/* built/available statistics */
WRITE_BOOL_FIELD(deps_built);
+ WRITE_BOOL_FIELD(mcv_built);
}
static void
diff --git a/src/backend/optimizer/path/clausesel.c b/src/backend/optimizer/path/clausesel.c
index a3afdf5..c16d559 100644
--- a/src/backend/optimizer/path/clausesel.c
+++ b/src/backend/optimizer/path/clausesel.c
@@ -15,6 +15,7 @@
#include "postgres.h"
#include "access/sysattr.h"
+#include "catalog/pg_collation.h"
#include "catalog/pg_operator.h"
#include "nodes/makefuncs.h"
#include "optimizer/clauses.h"
@@ -47,18 +48,39 @@ static void addRangeClause(RangeQueryClause **rqlist, Node *clause,
bool varonleft, bool isLTsel, Selectivity s2);
#define MV_CLAUSE_TYPE_FDEP 0x01
+#define MV_CLAUSE_TYPE_MCV 0x02
-static bool clause_is_mv_compatible(Node *clause, Index relid, AttrNumber *attnum);
+static bool clause_is_mv_compatible(Node *clause, Index relid, Bitmapset **attnums,
+ int type);
-static Bitmapset *collect_mv_attnums(List *clauses, Index relid);
+static Bitmapset *collect_mv_attnums(List *clauses, Index relid, int type);
-static int count_mv_attnums(List *clauses, Index relid);
+static int count_mv_attnums(List *clauses, Index relid, int type);
static int count_varnos(List *clauses, Index *relid);
static List *clauselist_apply_dependencies(PlannerInfo *root, List *clauses,
Index relid, List *stats);
+static MVStatisticInfo *choose_mv_statistics(List *mvstats, Bitmapset *attnums);
+
+static List *clauselist_mv_split(PlannerInfo *root, Index relid,
+ List *clauses, List **mvclauses,
+ MVStatisticInfo *mvstats, int types);
+
+static Selectivity clauselist_mv_selectivity(PlannerInfo *root,
+ List *clauses, MVStatisticInfo *mvstats);
+
+static Selectivity clauselist_mv_selectivity_mcvlist(PlannerInfo *root,
+ List *clauses, MVStatisticInfo *mvstats,
+ bool *fullmatch, Selectivity *lowsel);
+
+static int update_match_bitmap_mcvlist(PlannerInfo *root, List *clauses,
+ int2vector *stakeys, MCVList mcvlist,
+ int nmatches, char * matches,
+ Selectivity *lowsel, bool *fullmatch,
+ bool is_or);
+
static bool has_stats(List *stats, int type);
static List * find_stats(PlannerInfo *root, Index relid);
@@ -66,6 +88,13 @@ static List * find_stats(PlannerInfo *root, Index relid);
static bool stats_type_matches(MVStatisticInfo *stat, int type);
+/* used for merging bitmaps - AND (min), OR (max) */
+#define MAX(x, y) (((x) > (y)) ? (x) : (y))
+#define MIN(x, y) (((x) < (y)) ? (x) : (y))
+
+#define UPDATE_RESULT(m,r,isor) \
+ (m) = (isor) ? (MAX(m,r)) : (MIN(m,r))
+
/****************************************************************************
* ROUTINES TO COMPUTE SELECTIVITIES
****************************************************************************/
@@ -91,11 +120,13 @@ static bool stats_type_matches(MVStatisticInfo *stat, int type);
* to verify that suitable multivariate statistics exist.
*
* If we identify such multivariate statistics apply, we try to apply them.
- * Currently we only have (soft) functional dependencies, so we try to reduce
- * the list of clauses.
*
- * Then we remove the clauses estimated using multivariate stats, and process
- * the rest of the clauses using the regular per-column stats.
+ * First we try to reduce the list of clauses by applying (soft) functional
+ * dependencies, and then we try to estimate the selectivity of the reduced
+ * list of clauses using the multivariate MCV list.
+ *
+ * Finally we remove the portion of clauses estimated using multivariate stats,
+ * and process the rest of the clauses using the regular per-column stats.
*
* Currently, the only extra smarts we have is to recognize "range queries",
* such as "x > 34 AND x < 42". Clauses are recognized as possible range
@@ -172,12 +203,46 @@ clauselist_selectivity(PlannerInfo *root,
* that need to be estimated by other types of stats (MCV, histograms etc).
*/
if (has_stats(stats, MV_CLAUSE_TYPE_FDEP) &&
- (count_mv_attnums(clauses, relid) >= 2))
+ (count_mv_attnums(clauses, relid, MV_CLAUSE_TYPE_FDEP) >= 2))
{
clauses = clauselist_apply_dependencies(root, clauses, relid, stats);
}
/*
+ * Check that there are statistics with MCV list or histogram, and also the
+ * number of attributes covered by these types of statistics.
+ *
+ * If there are no such stats or not enough attributes, don't waste time
+ * with the multivariate code and simply skip to estimation using the
+ * regular per-column stats.
+ */
+ if (has_stats(stats, MV_CLAUSE_TYPE_MCV) &&
+ (count_mv_attnums(clauses, relid, MV_CLAUSE_TYPE_MCV) >= 2))
+ {
+ /* collect attributes from the compatible conditions */
+ Bitmapset *mvattnums = collect_mv_attnums(clauses, relid, MV_CLAUSE_TYPE_MCV);
+
+ /* and search for the statistic covering the most attributes */
+ MVStatisticInfo *mvstat = choose_mv_statistics(stats, mvattnums);
+
+ if (mvstat != NULL) /* we have a matching stats */
+ {
+ /* clauses compatible with multi-variate stats */
+ List *mvclauses = NIL;
+
+ /* split the clauselist into regular and mv-clauses */
+ clauses = clauselist_mv_split(root, relid, clauses, &mvclauses,
+ mvstat, MV_CLAUSE_TYPE_MCV);
+
+ /* we've chosen the histogram to match the clauses */
+ Assert(mvclauses != NIL);
+
+ /* compute the multivariate stats */
+ s1 *= clauselist_mv_selectivity(root, mvclauses, mvstat);
+ }
+ }
+
+ /*
* Initial scan over clauses. Anything that doesn't look like a potential
* rangequery clause gets multiplied into s1 and forgotten. Anything that
* does gets inserted into an rqlist entry.
@@ -834,32 +899,93 @@ clause_selectivity(PlannerInfo *root,
return s1;
}
+
+/*
+ * estimate selectivity of clauses using multivariate statistic
+ *
+ * Perform estimation of the clauses using a MCV list.
+ *
+ * This assumes all the clauses are compatible with the selected statistics
+ * (e.g. only reference columns covered by the statistics, use supported
+ * operator, etc.).
+ *
+ * TODO We may support some additional conditions, most importantly those
+ * matching multiple columns (e.g. "a = b" or "a < b").
+ *
+ * TODO Clamp the selectivity by min of the per-clause selectivities (i.e. the
+ * selectivity of the most restrictive clause), because that's the maximum
+ * we can ever get from ANDed list of clauses. This may probably prevent
+ * issues with hitting too many buckets and low precision histograms.
+ *
+ * TODO We may remember the lowest frequency in the MCV list, and then later use
+ * it as a upper boundary for the selectivity (had there been a more
+ * frequent item, it'd be in the MCV list). This might improve cases with
+ * low-detail histograms.
+ *
+ * TODO We may also derive some additional boundaries for the selectivity from
+ * the MCV list, because
+ *
+ * (a) if we have a "full equality condition" (one equality condition on
+ * each column of the statistic) and we found a match in the MCV list,
+ * then this is the final selectivity (and pretty accurate),
+ *
+ * (b) if we have a "full equality condition" and we haven't found a match
+ * in the MCV list, then the selectivity is below the lowest frequency
+ * found in the MCV list,
+ *
+ * TODO When applying the clauses to the histogram/MCV list, we can do
+ * that from the most selective clauses first, because that'll
+ * eliminate the buckets/items sooner (so we'll be able to skip
+ * them without inspection, which is more expensive). But this
+ * requires really knowing the per-clause selectivities in advance,
+ * and that's not what we do now.
+ */
+static Selectivity
+clauselist_mv_selectivity(PlannerInfo *root, List *clauses, MVStatisticInfo *mvstats)
+{
+ bool fullmatch = false;
+
+ /*
+ * Lowest frequency in the MCV list (may be used as an upper bound
+ * for full equality conditions that did not match any MCV item).
+ */
+ Selectivity mcv_low = 0.0;
+
+ /* TODO Evaluate simple 1D selectivities, use the smallest one as
+ * an upper bound, product as lower bound, and sort the
+ * clauses in ascending order by selectivity (to optimize the
+ * MCV/histogram evaluation).
+ */
+
+ /* Evaluate the MCV selectivity */
+ return clauselist_mv_selectivity_mcvlist(root, clauses, mvstats,
+ &fullmatch, &mcv_low);
+}
+
/*
* Collect attributes from mv-compatible clauses.
*/
static Bitmapset *
-collect_mv_attnums(List *clauses, Index relid)
+collect_mv_attnums(List *clauses, Index relid, int types)
{
Bitmapset *attnums = NULL;
ListCell *l;
/*
- * Walk through the clauses and identify the ones we can estimate
- * using multivariate stats, and remember the relid/columns. We'll
- * then cross-check if we have suitable stats, and only if needed
- * we'll split the clauses into multivariate and regular lists.
+ * Walk through the clauses and identify the ones we can estimate using
+ * multivariate stats, and remember the relid/columns. We'll then
+ * cross-check if we have suitable stats, and only if needed we'll split
+ * the clauses into multivariate and regular lists.
*
- * For now we're only interested in RestrictInfo nodes with nested
- * OpExpr, using either a range or equality.
+ * For now we're only interested in RestrictInfo nodes with nested OpExpr,
+ * using either a range or equality.
*/
foreach (l, clauses)
{
- AttrNumber attnum;
Node *clause = (Node *) lfirst(l);
- /* ignore the result for now - we only need the info */
- if (clause_is_mv_compatible(clause, relid, &attnum))
- attnums = bms_add_member(attnums, attnum);
+ /* ignore the result here - we only need the attnums */
+ clause_is_mv_compatible(clause, relid, &attnums, types);
}
/*
@@ -880,10 +1006,10 @@ collect_mv_attnums(List *clauses, Index relid)
* Count the number of attributes in clauses compatible with multivariate stats.
*/
static int
-count_mv_attnums(List *clauses, Index relid)
+count_mv_attnums(List *clauses, Index relid, int type)
{
int c;
- Bitmapset *attnums = collect_mv_attnums(clauses, relid);
+ Bitmapset *attnums = collect_mv_attnums(clauses, relid, type);
c = bms_num_members(attnums);
@@ -913,9 +1039,183 @@ count_varnos(List *clauses, Index *relid)
return cnt;
}
+
+/*
+ * We're looking for statistics matching at least 2 attributes, referenced in
+ * clauses compatible with multivariate statistics. The current selection
+ * criteria is very simple - we choose the statistics referencing the most
+ * attributes.
+ *
+ * If there are multiple statistics referencing the same number of columns
+ * (from the clauses), the one with less source columns (as listed in the
+ * ADD STATISTICS when creating the statistics) wins. Else the first one wins.
+ *
+ * This is a very simple criteria, and has several weaknesses:
+ *
+ * (a) does not consider the accuracy of the statistics
+ *
+ * If there are two histograms built on the same set of columns, but one
+ * has 100 buckets and the other one has 1000 buckets (thus likely
+ * providing better estimates), this is not currently considered.
+ *
+ * (b) does not consider the type of statistics
+ *
+ * If there are three statistics - one containing just a MCV list, another
+ * one with just a histogram and a third one with both, we treat them equally.
+ *
+ * (c) does not consider the number of clauses
+ *
+ * As explained, only the number of referenced attributes counts, so if
+ * there are multiple clauses on a single attribute, this still counts as
+ * a single attribute.
+ *
+ * (d) does not consider type of condition
+ *
+ * Some clauses may work better with some statistics - for example equality
+ * clauses probably work better with MCV lists than with histograms. But
+ * IS [NOT] NULL conditions may often work better with histograms (thanks
+ * to NULL-buckets).
+ *
+ * So for example with five WHERE conditions
+ *
+ * WHERE (a = 1) AND (b = 1) AND (c = 1) AND (d = 1) AND (e = 1)
+ *
+ * and statistics on (a,b), (a,b,e) and (a,b,c,d), the last one will be selected
+ * as it references the most columns.
+ *
+ * Once we have selected the multivariate statistics, we split the list of
+ * clauses into two parts - conditions that are compatible with the selected
+ * stats, and conditions are estimated using simple statistics.
+ *
+ * From the example above, conditions
+ *
+ * (a = 1) AND (b = 1) AND (c = 1) AND (d = 1)
+ *
+ * will be estimated using the multivariate statistics (a,b,c,d) while the last
+ * condition (e = 1) will get estimated using the regular ones.
+ *
+ * There are various alternative selection criteria (e.g. counting conditions
+ * instead of just referenced attributes), but eventually the best option should
+ * be to combine multiple statistics. But that's much harder to do correctly.
+ *
+ * TODO Select multiple statistics and combine them when computing the estimate.
+ *
+ * TODO This will probably have to consider compatibility of clauses, because
+ * 'dependencies' will probably work only with equality clauses.
+ */
+static MVStatisticInfo *
+choose_mv_statistics(List *stats, Bitmapset *attnums)
+{
+ int i;
+ ListCell *lc;
+
+ MVStatisticInfo *choice = NULL;
+
+ int current_matches = 1; /* goal #1: maximize */
+ int current_dims = (MVSTATS_MAX_DIMENSIONS+1); /* goal #2: minimize */
+
+ /*
+ * Walk through the statistics (simple array with nmvstats elements) and for
+ * each one count the referenced attributes (encoded in the 'attnums' bitmap).
+ */
+ foreach (lc, stats)
+ {
+ MVStatisticInfo *info = (MVStatisticInfo *)lfirst(lc);
+
+ /* columns matching this statistics */
+ int matches = 0;
+
+ int2vector * attrs = info->stakeys;
+ int numattrs = attrs->dim1;
+
+ /* skip dependencies-only stats */
+ if (! info->mcv_built)
+ continue;
+
+ /* count columns covered by the histogram */
+ for (i = 0; i < numattrs; i++)
+ if (bms_is_member(attrs->values[i], attnums))
+ matches++;
+
+ /*
+ * Use this statistics when it improves the number of matches or
+ * when it matches the same number of attributes but is smaller.
+ */
+ if ((matches > current_matches) ||
+ ((matches == current_matches) && (current_dims > numattrs)))
+ {
+ choice = info;
+ current_matches = matches;
+ current_dims = numattrs;
+ }
+ }
+
+ return choice;
+}
+
+
+/*
+ * This splits the clauses list into two parts - one containing clauses that
+ * will be evaluated using the chosen statistics, and the remaining clauses
+ * (either non-mvcompatible, or not related to the histogram).
+ */
+static List *
+clauselist_mv_split(PlannerInfo *root, Index relid,
+ List *clauses, List **mvclauses,
+ MVStatisticInfo *mvstats, int types)
+{
+ int i;
+ ListCell *l;
+ List *non_mvclauses = NIL;
+
+ /* FIXME is there a better way to get info on int2vector? */
+ int2vector * attrs = mvstats->stakeys;
+ int numattrs = mvstats->stakeys->dim1;
+
+ Bitmapset *mvattnums = NULL;
+
+ /* build bitmap of attributes, so we can do bms_is_subset later */
+ for (i = 0; i < numattrs; i++)
+ mvattnums = bms_add_member(mvattnums, attrs->values[i]);
+
+ /* erase the list of mv-compatible clauses */
+ *mvclauses = NIL;
+
+ foreach (l, clauses)
+ {
+ bool match = false; /* by default not mv-compatible */
+ Bitmapset *attnums = NULL;
+ Node *clause = (Node *) lfirst(l);
+
+ if (clause_is_mv_compatible(clause, relid, &attnums, types))
+ {
+ /* are all the attributes part of the selected stats? */
+ if (bms_is_subset(attnums, mvattnums))
+ match = true;
+ }
+
+ /*
+ * The clause matches the selected stats, so put it to the list of
+ * mv-compatible clauses. Otherwise, keep it in the list of 'regular'
+ * clauses (that may be selected later).
+ */
+ if (match)
+ *mvclauses = lappend(*mvclauses, clause);
+ else
+ non_mvclauses = lappend(non_mvclauses, clause);
+ }
+
+ /*
+ * Perform regular estimation using the clauses incompatible with the chosen
+ * histogram (or MV stats in general).
+ */
+ return non_mvclauses;
+
+}
typedef struct
{
+ int types; /* types of statistics ? */
Index varno; /* relid we're interested in */
Bitmapset *varattnos; /* attnums referenced by the clauses */
} mv_compatible_context;
@@ -950,6 +1250,49 @@ mv_compatible_walker(Node *node, mv_compatible_context *context)
return mv_compatible_walker((Node*)rinfo->clause, (void *) context);
}
+ if (or_clause(node) || and_clause(node) || not_clause(node))
+ {
+ /*
+ * AND/OR/NOT-clauses are supported if all sub-clauses are supported
+ *
+ * TODO We might support mixed case, where some of the clauses are
+ * supported and some are not, and treat all supported subclauses
+ * as a single clause, compute it's selectivity using mv stats,
+ * and compute the total selectivity using the current algorithm.
+ *
+ * TODO For RestrictInfo above an OR-clause, we might use the orclause
+ * with nested RestrictInfo - we won't have to call pull_varnos()
+ * for each clause, saving time.
+ *
+ * TODO Perhaps this needs a bit more thought for functional
+ * dependencies? Those don't quite work for NOT cases.
+ */
+ BoolExpr *expr = (BoolExpr *) node;
+ ListCell *lc;
+
+ foreach (lc, expr->args)
+ {
+ if (mv_compatible_walker((Node *) lfirst(lc), context))
+ return true;
+ }
+
+ return false;
+ }
+
+ if (IsA(node, NullTest))
+ {
+ NullTest* nt = (NullTest*)node;
+
+ /*
+ * Only simple (Var IS NULL) expressions supported for now. Maybe we could
+ * use examine_variable to fix this?
+ */
+ if (! IsA(nt->arg, Var))
+ return true;
+
+ return mv_compatible_walker((Node*)(nt->arg), context);
+ }
+
if (IsA(node, Var))
{
Var * var = (Var*)node;
@@ -1010,10 +1353,18 @@ mv_compatible_walker(Node *node, mv_compatible_context *context)
switch (get_oprrest(expr->opno))
{
case F_EQSEL:
-
/* equality conditions are compatible with all statistics */
break;
+ case F_SCALARLTSEL:
+ case F_SCALARGTSEL:
+
+ /* not compatible with functional dependencies */
+ if (! (context->types & MV_CLAUSE_TYPE_MCV))
+ return true; /* terminate */
+
+ break;
+
default:
/* unknown estimator */
@@ -1047,10 +1398,11 @@ mv_compatible_walker(Node *node, mv_compatible_context *context)
* evaluate them using multivariate stats.
*/
static bool
-clause_is_mv_compatible(Node *clause, Index relid, AttrNumber *attnum)
+clause_is_mv_compatible(Node *clause, Index relid, Bitmapset **attnums, int types)
{
mv_compatible_context context;
+ context.types = types;
context.varno = relid;
context.varattnos = NULL; /* no attnums */
@@ -1058,7 +1410,7 @@ clause_is_mv_compatible(Node *clause, Index relid, AttrNumber *attnum)
return false;
/* remember the newly collected attnums */
- *attnum = bms_singleton_member(context.varattnos);
+ *attnums = bms_add_members(*attnums, context.varattnos);
return true;
}
@@ -1075,15 +1427,15 @@ fdeps_reduce_clauses(List *clauses, Index relid, Bitmapset *reduced_attnums)
foreach (lc, clauses)
{
- AttrNumber attnum = InvalidAttrNumber;
+ Bitmapset *attnums = NULL;
Node * clause = (Node*)lfirst(lc);
/* ignore clauses that are not compatible with functional dependencies */
- if (! clause_is_mv_compatible(clause, relid, &attnum))
+ if (! clause_is_mv_compatible(clause, relid, &attnums, MV_CLAUSE_TYPE_FDEP))
reduced_clauses = lappend(reduced_clauses, clause);
/* for equality clauses, only keep those not on reduced attributes */
- if (! bms_is_member(attnum, reduced_attnums))
+ if (! bms_is_subset(attnums, reduced_attnums))
reduced_clauses = lappend(reduced_clauses, clause);
}
@@ -1208,7 +1560,7 @@ clauselist_apply_dependencies(PlannerInfo *root, List *clauses,
return clauses;
/* collect attnums from clauses compatible with dependencies (equality) */
- clause_attnums = collect_mv_attnums(clauses, relid);
+ clause_attnums = collect_mv_attnums(clauses, relid, MV_CLAUSE_TYPE_FDEP);
/* decide which attnums may be eliminated */
reduced_attnums = fdeps_reduce_attnums(stats, clause_attnums);
@@ -1233,6 +1585,9 @@ stats_type_matches(MVStatisticInfo *stat, int type)
if ((type & MV_CLAUSE_TYPE_FDEP) && stat->deps_built)
return true;
+ if ((type & MV_CLAUSE_TYPE_MCV) && stat->mcv_built)
+ return true;
+
return false;
}
@@ -1266,3 +1621,392 @@ find_stats(PlannerInfo *root, Index relid)
return root->simple_rel_array[relid]->mvstatlist;
}
+
+/*
+ * Estimate selectivity of clauses using a MCV list.
+ *
+ * If there's no MCV list for the stats, the function returns 0.0.
+ *
+ * While computing the estimate, the function checks whether all the
+ * columns were matched with an equality condition. If that's the case,
+ * we can skip processing the histogram, as there can be no rows in
+ * it with the same values - all the rows matching the condition are
+ * represented by the MCV item. This can only happen with equality
+ * on all the attributes.
+ *
+ * The algorithm works like this:
+ *
+ * 1) mark all items as 'match'
+ * 2) walk through all the clauses
+ * 3) for a particular clause, walk through all the items
+ * 4) skip items that are already 'no match'
+ * 5) check clause for items that still match
+ * 6) sum frequencies for items to get selectivity
+ *
+ * The function also returns the frequency of the least frequent item
+ * on the MCV list, which may be useful for clamping estimate from the
+ * histogram (all items not present in the MCV list are less frequent).
+ * This however seems useful only for cases with conditions on all
+ * attributes.
+ *
+ * TODO This only handles AND-ed clauses, but it might work for OR-ed
+ * lists too - it just needs to reverse the logic a bit. I.e. start
+ * with 'no match' for all items, and mark the items as a match
+ * as the clauses are processed (and skip items that are 'match').
+ */
+static Selectivity
+clauselist_mv_selectivity_mcvlist(PlannerInfo *root, List *clauses,
+ MVStatisticInfo *mvstats, bool *fullmatch,
+ Selectivity *lowsel)
+{
+ int i;
+ Selectivity s = 0.0;
+ Selectivity u = 0.0;
+
+ MCVList mcvlist = NULL;
+ int nmatches = 0;
+
+ /* match/mismatch bitmap for each MCV item */
+ char * matches = NULL;
+
+ Assert(clauses != NIL);
+ Assert(list_length(clauses) >= 2);
+
+ /* there's no MCV list built yet */
+ if (! mvstats->mcv_built)
+ return 0.0;
+
+ mcvlist = load_mv_mcvlist(mvstats->mvoid);
+
+ Assert(mcvlist != NULL);
+ Assert(mcvlist->nitems > 0);
+
+ /* by default all the MCV items match the clauses fully */
+ matches = palloc0(sizeof(char) * mcvlist->nitems);
+ memset(matches, MVSTATS_MATCH_FULL, sizeof(char)*mcvlist->nitems);
+
+ /* number of matching MCV items */
+ nmatches = mcvlist->nitems;
+
+ nmatches = update_match_bitmap_mcvlist(root, clauses,
+ mvstats->stakeys, mcvlist,
+ nmatches, matches,
+ lowsel, fullmatch, false);
+
+ /* sum frequencies for all the matching MCV items */
+ for (i = 0; i < mcvlist->nitems; i++)
+ {
+ /* used to 'scale' for MCV lists not covering all tuples */
+ u += mcvlist->items[i]->frequency;
+
+ if (matches[i] != MVSTATS_MATCH_NONE)
+ s += mcvlist->items[i]->frequency;
+ }
+
+ pfree(matches);
+ pfree(mcvlist);
+
+ return s*u;
+}
+
+/*
+ * Evaluate clauses using the MCV list, and update the match bitmap.
+ *
+ * The bitmap may be already partially set, so this is really a way to
+ * combine results of several clause lists - either when computing
+ * conditional probability P(A|B) or a combination of AND/OR clauses.
+ *
+ * TODO This works with 'bitmap' where each bit is represented as a char,
+ * which is slightly wasteful. Instead, we could use a regular
+ * bitmap, reducing the size to ~1/8. Another thing is merging the
+ * bitmaps using & and |, which might be faster than min/max.
+ */
+static int
+update_match_bitmap_mcvlist(PlannerInfo *root, List *clauses,
+ int2vector *stakeys, MCVList mcvlist,
+ int nmatches, char * matches,
+ Selectivity *lowsel, bool *fullmatch,
+ bool is_or)
+{
+ int i;
+ ListCell * l;
+
+ Bitmapset *eqmatches = NULL; /* attributes with equality matches */
+
+ /* The bitmap may be partially built. */
+ Assert(nmatches >= 0);
+ Assert(nmatches <= mcvlist->nitems);
+ Assert(clauses != NIL);
+ Assert(list_length(clauses) >= 1);
+ Assert(mcvlist != NULL);
+ Assert(mcvlist->nitems > 0);
+
+ /* No possible matches (only works for AND-ded clauses) */
+ if (((nmatches == 0) && (! is_or)) ||
+ ((nmatches == mcvlist->nitems) && is_or))
+ return nmatches;
+
+ /*
+ * find the lowest frequency in the MCV list
+ *
+ * We need to do that here, because we do various tricks in the following
+ * code - skipping items already ruled out, etc.
+ *
+ * XXX A loop is necessary because the MCV list is not sorted by frequency.
+ */
+ *lowsel = 1.0;
+ for (i = 0; i < mcvlist->nitems; i++)
+ {
+ MCVItem item = mcvlist->items[i];
+
+ if (item->frequency < *lowsel)
+ *lowsel = item->frequency;
+ }
+
+ /*
+ * Loop through the list of clauses, and for each of them evaluate
+ * all the MCV items not yet eliminated by the preceding clauses.
+ */
+ foreach (l, clauses)
+ {
+ Node * clause = (Node*)lfirst(l);
+
+ /* if it's a RestrictInfo, then extract the clause */
+ if (IsA(clause, RestrictInfo))
+ clause = (Node*)((RestrictInfo*)clause)->clause;
+
+ /* if there are no remaining matches possible, we can stop */
+ if (((nmatches == 0) && (! is_or)) ||
+ ((nmatches == mcvlist->nitems) && is_or))
+ break;
+
+ /* it's either OpClause, or NullTest */
+ if (is_opclause(clause))
+ {
+ OpExpr *expr = (OpExpr*)clause;
+ bool varonleft = true;
+ bool ok;
+ FmgrInfo opproc;
+
+ /* get procedure computing operator selectivity */
+ RegProcedure oprrest = get_oprrest(expr->opno);
+
+ fmgr_info(get_opcode(expr->opno), &opproc);
+
+ ok = (NumRelids(clause) == 1) &&
+ (is_pseudo_constant_clause(lsecond(expr->args)) ||
+ (varonleft = false,
+ is_pseudo_constant_clause(linitial(expr->args))));
+
+ if (ok)
+ {
+
+ FmgrInfo gtproc;
+ Var * var = (varonleft) ? linitial(expr->args) : lsecond(expr->args);
+ Const * cst = (varonleft) ? lsecond(expr->args) : linitial(expr->args);
+ bool isgt = (! varonleft);
+
+ TypeCacheEntry *typecache
+ = lookup_type_cache(var->vartype, TYPECACHE_GT_OPR);
+
+ /* FIXME proper matching attribute to dimension */
+ int idx = mv_get_index(var->varattno, stakeys);
+
+ fmgr_info(get_opcode(typecache->gt_opr), >proc);
+
+ /*
+ * Walk through the MCV items and evaluate the current clause. We can
+ * skip items that were already ruled out, and terminate if there are
+ * no remaining MCV items that might possibly match.
+ */
+ for (i = 0; i < mcvlist->nitems; i++)
+ {
+ bool mismatch = false;
+ MCVItem item = mcvlist->items[i];
+
+ /*
+ * If there are no more matches (AND) or no remaining unmatched
+ * items (OR), we can stop processing this clause.
+ */
+ if (((nmatches == 0) && (! is_or)) ||
+ ((nmatches == mcvlist->nitems) && is_or))
+ break;
+
+ /*
+ * For AND-lists, we can also mark NULL items as 'no match' (and
+ * then skip them). For OR-lists this is not possible.
+ */
+ if ((! is_or) && item->isnull[idx])
+ matches[i] = MVSTATS_MATCH_NONE;
+
+ /* skip MCV items that were already ruled out */
+ if ((! is_or) && (matches[i] == MVSTATS_MATCH_NONE))
+ continue;
+ else if (is_or && (matches[i] == MVSTATS_MATCH_FULL))
+ continue;
+
+ switch (oprrest)
+ {
+ case F_EQSEL:
+ /*
+ * We don't care about isgt in equality, because it does not
+ * matter whether it's (var = const) or (const = var).
+ */
+ mismatch = ! DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ cst->constvalue,
+ item->values[idx]));
+
+ if (! mismatch)
+ eqmatches = bms_add_member(eqmatches, idx);
+
+ break;
+
+ case F_SCALARLTSEL: /* column < constant */
+ case F_SCALARGTSEL: /* column > constant */
+
+ /*
+ * First check whether the constant is below the lower boundary (in that
+ * case we can skip the bucket, because there's no overlap).
+ */
+ mismatch = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ cst->constvalue,
+ item->values[idx]));
+
+ /* invert the result if isgt=true */
+ mismatch = (isgt) ? (! mismatch) : mismatch;
+ break;
+ }
+
+ /* XXX The conditions on matches[i] are not needed, as we
+ * skip MCV items that can't become true/false, depending
+ * on the current flag. See beginning of the loop over
+ * MCV items.
+ */
+
+ if ((is_or) && (matches[i] == MVSTATS_MATCH_NONE) && (! mismatch))
+ {
+ /* OR - was MATCH_NONE, but will be MATCH_FULL */
+ matches[i] = MVSTATS_MATCH_FULL;
+ ++nmatches;
+ continue;
+ }
+ else if ((! is_or) && (matches[i] == MVSTATS_MATCH_FULL) && mismatch)
+ {
+ /* AND - was MATC_FULL, but will be MATCH_NONE */
+ matches[i] = MVSTATS_MATCH_NONE;
+ --nmatches;
+ continue;
+ }
+
+ }
+ }
+ }
+ else if (IsA(clause, NullTest))
+ {
+ NullTest * expr = (NullTest*)clause;
+ Var * var = (Var*)(expr->arg);
+
+ /* FIXME proper matching attribute to dimension */
+ int idx = mv_get_index(var->varattno, stakeys);
+
+ /*
+ * Walk through the MCV items and evaluate the current clause. We can
+ * skip items that were already ruled out, and terminate if there are
+ * no remaining MCV items that might possibly match.
+ */
+ for (i = 0; i < mcvlist->nitems; i++)
+ {
+ MCVItem item = mcvlist->items[i];
+
+ /* if there are no more matches, we can stop processing this clause */
+ if (nmatches == 0)
+ break;
+
+ /* skip MCV items that were already ruled out */
+ if (matches[i] == MVSTATS_MATCH_NONE)
+ continue;
+
+ /* if the clause mismatches the MCV item, set it as MATCH_NONE */
+ if (((expr->nulltesttype == IS_NULL) && (! item->isnull[idx])) ||
+ ((expr->nulltesttype == IS_NOT_NULL) && (item->isnull[idx])))
+ {
+ matches[i] = MVSTATS_MATCH_NONE;
+ --nmatches;
+ }
+ }
+ }
+ else if (or_clause(clause) || and_clause(clause))
+ {
+ /* AND/OR clause, with all clauses compatible with the selected MV stat */
+
+ int i;
+ BoolExpr *orclause = ((BoolExpr*)clause);
+ List *orclauses = orclause->args;
+
+ /* match/mismatch bitmap for each MCV item */
+ int or_nmatches = 0;
+ char * or_matches = NULL;
+
+ Assert(orclauses != NIL);
+ Assert(list_length(orclauses) >= 2);
+
+ /* number of matching MCV items */
+ or_nmatches = mcvlist->nitems;
+
+ /* by default none of the MCV items matches the clauses */
+ or_matches = palloc0(sizeof(char) * or_nmatches);
+
+ if (or_clause(clause))
+ {
+ /* OR clauses assume nothing matches, initially */
+ memset(or_matches, MVSTATS_MATCH_NONE, sizeof(char)*or_nmatches);
+ or_nmatches = 0;
+ }
+ else
+ {
+ /* AND clauses assume nothing matches, initially */
+ memset(or_matches, MVSTATS_MATCH_FULL, sizeof(char)*or_nmatches);
+ }
+
+ /* build the match bitmap for the OR-clauses */
+ or_nmatches = update_match_bitmap_mcvlist(root, orclauses,
+ stakeys, mcvlist,
+ or_nmatches, or_matches,
+ lowsel, fullmatch, or_clause(clause));
+
+ /* merge the bitmap into the existing one*/
+ for (i = 0; i < mcvlist->nitems; i++)
+ {
+ /*
+ * To AND-merge the bitmaps, a MIN() semantics is used.
+ * For OR-merge, use MAX().
+ *
+ * FIXME this does not decrease the number of matches
+ */
+ UPDATE_RESULT(matches[i], or_matches[i], is_or);
+ }
+
+ pfree(or_matches);
+
+ }
+ else
+ {
+ elog(ERROR, "unknown clause type: %d", clause->type);
+ }
+ }
+
+ /*
+ * If all the columns were matched by equality, it's a full match.
+ * In this case there can be just a single MCV item, matching the
+ * clause (if there were two, both would match the other one).
+ */
+ *fullmatch = (bms_num_members(eqmatches) == mcvlist->ndimensions);
+
+ /* free the allocated pieces */
+ if (eqmatches)
+ pfree(eqmatches);
+
+ return nmatches;
+}
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index 7fb2088..8394111 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -412,7 +412,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
mvstat = (Form_pg_mv_statistic) GETSTRUCT(htup);
/* unavailable stats are not interesting for the planner */
- if (mvstat->deps_built)
+ if (mvstat->deps_built || mvstat->mcv_built)
{
info = makeNode(MVStatisticInfo);
@@ -421,9 +421,11 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
/* enabled statistics */
info->deps_enabled = mvstat->deps_enabled;
+ info->mcv_enabled = mvstat->mcv_enabled;
/* built/available statistics */
info->deps_built = mvstat->deps_built;
+ info->mcv_built = mvstat->mcv_built;
/* stakeys */
adatum = SysCacheGetAttr(MVSTATOID, htup,
diff --git a/src/backend/utils/mvstats/Makefile b/src/backend/utils/mvstats/Makefile
index 099f1ed..f9bf10c 100644
--- a/src/backend/utils/mvstats/Makefile
+++ b/src/backend/utils/mvstats/Makefile
@@ -12,6 +12,6 @@ subdir = src/backend/utils/mvstats
top_builddir = ../../../..
include $(top_builddir)/src/Makefile.global
-OBJS = common.o dependencies.o
+OBJS = common.o dependencies.o mcv.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/utils/mvstats/README.mcv b/src/backend/utils/mvstats/README.mcv
new file mode 100644
index 0000000..e93cfe4
--- /dev/null
+++ b/src/backend/utils/mvstats/README.mcv
@@ -0,0 +1,137 @@
+MCV lists
+=========
+
+Multivariate MCV (most-common values) lists are a straightforward extension of
+regular MCV list, tracking most frequent combinations of values for a group of
+attributes.
+
+This works particularly well for columns with a small number of distinct values,
+as the list may include all the combinations and approximate the distribution
+very accurately.
+
+For columns with large number of distinct values (e.g. those with continuous
+domains), the list will only track the most frequent combinations. If the
+distribution is mostly uniform (all combinations about equally frequent), the
+MCV list will be empty.
+
+Estimates of some clauses (e.g. equality) based on MCV lists are more accurate
+than when using histograms.
+
+Also, MCV lists don't necessarily require sorting of the values (the fact that
+we use sorting when building them is implementation detail), but even more
+importantly the ordering is not built into the approximation (while histograms
+are built on ordering). So MCV lists work well even for attributes where the
+ordering of the data type is disconnected from the meaning of the data. For
+example we know how to sort strings, but it's unlikely to make much sense for
+city names (or other label-like attributes).
+
+
+Selectivity estimation
+----------------------
+
+The estimation, implemented in clauselist_mv_selectivity_mcvlist(), is quite
+simple in principle - we need to identify MCV items matching all the clauses
+and sum frequencies of all those items.
+
+Currently MCV lists support estimation of the following clause types:
+
+ (a) equality clauses WHERE (a = 1) AND (b = 2)
+ (b) inequality clauses WHERE (a < 1) AND (b >= 2)
+ (c) NULL clauses WHERE (a IS NULL) AND (b IS NOT NULL)
+ (d) OR clauses WHERE (a < 1) OR (b >= 2)
+
+It's possible to add support for additional clauses, for example:
+
+ (e) multi-var clauses WHERE (a > b)
+
+and possibly others. These are tasks for the future, not yet implemented.
+
+
+Estimating equality clauses
+---------------------------
+
+When computing selectivity estimate for equality clauses
+
+ (a = 1) AND (b = 2)
+
+we can do this estimate pretty exactly assuming that two conditions are met:
+
+ (1) there's an equality condition on all attributes of the statistic
+
+ (2) we find a matching item in the MCV list
+
+In this case we know the MCV item represents all tuples matching the clauses,
+and the selectivity estimate is complete (i.e. we don't need to perform
+estimation using the histogram). This is what we call 'full match'.
+
+When only (1) holds, but there's no matching MCV item, we don't know whether
+there are no such rows or just are not very frequent. We can however use the
+frequency of the least frequent MCV item as an upper bound for the selectivity.
+
+For a combination of equality conditions (not full-match case) we can clamp the
+selectivity by the minimum of selectivities for each condition. For example if
+we know the number of distinct values for each column, we can use 1/ndistinct
+as a per-column estimate. Or rather 1/ndistinct + selectivity derived from the
+MCV list.
+
+We should also probably only use the 'residual ndistinct' by exluding the items
+included in the MCV list (and also residual frequency):
+
+ f = (1.0 - sum(MCV frequencies)) / (ndistinct - ndistinct(MCV list))
+
+but it's worth pointing out the ndistinct values are multi-variate for the
+columns referenced by the equality conditions.
+
+Note: Only the "full match" limit is currently implemented.
+
+
+Hashed MCV (not yet implemented)
+--------------------------------
+
+Regular MCV lists have to include actual values for each item, so if those items
+are large the list may be quite large. This is especially true for multi-variate
+MCV lists, although the current implementation partially mitigates this by
+performing de-duplicating the values before storing them on disk.
+
+It's possible to only store hashes (32-bit values) instead of the actual values,
+significantly reducing the space requirements. Obviously, this would only make
+the MCV lists useful for estimating equality conditions (assuming the 32-bit
+hashes make the collisions rare enough).
+
+This might also complicate matching the columns to available stats.
+
+
+TODO Consider implementing hashed MCV list, storing just 32-bit hashes instead
+ of the actual values. This type of MCV list will be useful only for
+ estimating equality clauses, and will reduce space requirements for large
+ varlena types (in such cases we usually only want equality anyway).
+
+TODO Currently there's no logic to consider building only a MCV list (and not
+ building the histogram at all), except for doing this decision manually in
+ ADD STATISTICS.
+
+
+Inspecting the MCV list
+-----------------------
+
+Inspecting the regular (per-attribute) MCV lists is trivial, as it's enough
+to select the columns from pg_stats - the data is encoded as anyarrays, so we
+simply get the text representation of the arrays.
+
+With multivariate MCV lits it's not that simple due to the possible mix of
+data types. It might be possible to produce similar array-like representation,
+but that'd unnecessarily complicate further processing and analysis of the MCV
+list. Instead, there's a SRF function providing values, frequencies etc.
+
+ SELECT * FROM pg_mv_mcv_items();
+
+It has two input parameters:
+
+ oid - OID of the MCV list (pg_mv_statistic.staoid)
+
+and produces a table with these columns:
+
+ - item ID (0...nitems-1)
+ - values (string array)
+ - nulls only (boolean array)
+ - frequency (double precision)
diff --git a/src/backend/utils/mvstats/README.stats b/src/backend/utils/mvstats/README.stats
index a38ea7b..5c5c59a 100644
--- a/src/backend/utils/mvstats/README.stats
+++ b/src/backend/utils/mvstats/README.stats
@@ -8,9 +8,50 @@ not true, resulting in estimation errors.
Multivariate stats track different types of dependencies between the columns,
hopefully improving the estimates.
-Currently we only have one kind of multivariate statistics - soft functional
-dependencies, and we use it to improve estimates of equality clauses. See
-README.dependencies for details.
+
+Types of statistics
+-------------------
+
+Currently we only have two kinds of multivariate statistics
+
+ (a) soft functional dependencies (README.dependencies)
+
+ (b) MCV lists (README.mcv)
+
+
+Compatible clause types
+-----------------------
+
+Each type of statistics may be used to estimate some subset of clause types.
+
+ (a) functional dependencies - equality clauses (AND), possibly IS NULL
+
+ (b) MCV list - equality and inequality clauses, IS [NOT] NULL, AND/OR
+
+Currently only simple operator clauses (Var op Const) are supported, but it's
+possible to support more complex clause types, e.g. (Var op Var).
+
+
+Complex clauses
+---------------
+
+We also support estimating more complex clauses - essentially AND/OR clauses
+with (Var op Const) as leaves, as long as all the referenced attributes are
+covered by a single statistics.
+
+For example this condition
+
+ (a=1) AND ((b=2) OR ((c=3) AND (d=4)))
+
+may be estimated using statistics on (a,b,c,d). If we only have statistics on
+(b,c,d) we may estimate the second part, and estimate (a=1) using simple stats.
+
+If we only have statistics on (a,b,c) we can't apply it at all at this point,
+but it's worth pointing out clauselist_selectivity() works recursively and when
+handling the second part (the OR-clause), we'll be able to apply the statistics.
+
+Note: The multi-statistics estimation patch also makes it possible to pass some
+clauses as 'conditions' into the deeper parts of the expression tree.
Selectivity estimation
@@ -23,14 +64,48 @@ When estimating selectivity, we aim to achieve several things:
(b) minimize the overhead, especially when no suitable multivariate stats
exist (so if you are not using multivariate stats, there's no overhead)
-This clauselist_selectivity() performs several inexpensive checks first, before
+Thus clauselist_selectivity() performs several inexpensive checks first, before
even attempting to do the more expensive estimation.
(1) check if there are multivariate stats on the relation
- (2) check there are at least two attributes referenced by clauses compatible
- with multivariate statistics (equality clauses for func. dependencies)
+ (2) check that there are functional dependencies on the table, and that
+ there are at least two attributes referenced by compatible clauses
+ (equality clauses for func. dependencies)
(3) perform reduction of equality clauses using func. dependencies
- (4) estimate the reduced list of clauses using regular statistics
+ (4) check that there are multivariate MCV lists on the table, and that
+ there are at least two attributes referenced by compatible clauses
+ (equalities, inequalities, etc.)
+
+ (5) find the best multivariate statistics (matching the most conditions)
+ and use it to compute the estimate
+
+ (6) estimate the remaining clauses (not estimated using multivariate stats)
+ using the regular per-column statistics
+
+Whenever we find there are no suitable stats, we skip the expensive steps.
+
+
+Further (possibly crazy) ideas
+------------------------------
+
+Currently the clauses are only estimated using a single statistics, even if
+there are multiple candidate statistics - for example assume we have statistics
+on (a,b,c) and (b,c,d), and estimate conditions
+
+ (b = 1) AND (c = 2)
+
+Then both statistics may be used, but we only use one of them. Maybe we could
+use compute estimates using all candidate stats, and somehow aggregate them
+into the final estimate by using average or median.
+
+Some stats may give better estimates than others, but it's very difficult to say
+in advance which stats are the best (it depends on the number of buckets, number
+of additional columns not referenced in the clauses, type of condition etc.).
+
+But of course, this may result in expensive estimation (CPU-wise).
+
+So we might add a GUC to choose between a simple (single statistics) and thus
+multi-statistic estimation, possibly table-level parameter (ALTER TABLE ...).
diff --git a/src/backend/utils/mvstats/common.c b/src/backend/utils/mvstats/common.c
index dcb7c78..4f5a842 100644
--- a/src/backend/utils/mvstats/common.c
+++ b/src/backend/utils/mvstats/common.c
@@ -16,12 +16,14 @@
#include "common.h"
+#include "utils/array.h"
+
static VacAttrStats ** lookup_var_attr_stats(int2vector *attrs,
- int natts, VacAttrStats **vacattrstats);
+ int natts,
+ VacAttrStats **vacattrstats);
static List* list_mv_stats(Oid relid);
-
/*
* Compute requested multivariate stats, using the rows sampled for the
* plain (single-column) stats.
@@ -49,6 +51,8 @@ build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
int j;
MVStatisticInfo *stat = (MVStatisticInfo *)lfirst(lc);
MVDependencies deps = NULL;
+ MCVList mcvlist = NULL;
+ int numrows_filtered = 0;
VacAttrStats **stats = NULL;
int numatts = 0;
@@ -87,8 +91,12 @@ build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
if (stat->deps_enabled)
deps = build_mv_dependencies(numrows, rows, attrs, stats);
+ /* build the MCV list */
+ if (stat->mcv_enabled)
+ mcvlist = build_mv_mcvlist(numrows, rows, attrs, stats, &numrows_filtered);
+
/* store the histogram / MCV list in the catalog */
- update_mv_stats(stat->mvoid, deps, attrs);
+ update_mv_stats(stat->mvoid, deps, mcvlist, attrs, stats);
}
}
@@ -166,6 +174,8 @@ list_mv_stats(Oid relid)
info->stakeys = buildint2vector(stats->stakeys.values, stats->stakeys.dim1);
info->deps_enabled = stats->deps_enabled;
info->deps_built = stats->deps_built;
+ info->mcv_enabled = stats->mcv_enabled;
+ info->mcv_built = stats->mcv_built;
result = lappend(result, info);
}
@@ -180,8 +190,56 @@ list_mv_stats(Oid relid)
return result;
}
+
+/*
+ * Find attnims of MV stats using the mvoid.
+ */
+int2vector*
+find_mv_attnums(Oid mvoid, Oid *relid)
+{
+ ArrayType *arr;
+ Datum adatum;
+ bool isnull;
+ HeapTuple htup;
+ int2vector *keys;
+
+ /* Prepare to scan pg_mv_statistic for entries having indrelid = this rel. */
+ htup = SearchSysCache1(MVSTATOID,
+ ObjectIdGetDatum(mvoid));
+
+ /* XXX syscache contains OIDs of deleted stats (not invalidated) */
+ if (! HeapTupleIsValid(htup))
+ return NULL;
+
+ /* starelid */
+ adatum = SysCacheGetAttr(MVSTATOID, htup,
+ Anum_pg_mv_statistic_starelid, &isnull);
+ Assert(!isnull);
+
+ *relid = DatumGetObjectId(adatum);
+
+ /* stakeys */
+ adatum = SysCacheGetAttr(MVSTATOID, htup,
+ Anum_pg_mv_statistic_stakeys, &isnull);
+ Assert(!isnull);
+
+ arr = DatumGetArrayTypeP(adatum);
+
+ keys = buildint2vector((int16 *) ARR_DATA_PTR(arr),
+ ARR_DIMS(arr)[0]);
+ ReleaseSysCache(htup);
+
+ /* TODO maybe save the list into relcache, as in RelationGetIndexList
+ * (which was used as an inspiration of this one)?. */
+
+ return keys;
+}
+
+
void
-update_mv_stats(Oid mvoid, MVDependencies dependencies, int2vector *attrs)
+update_mv_stats(Oid mvoid,
+ MVDependencies dependencies, MCVList mcvlist,
+ int2vector *attrs, VacAttrStats **stats)
{
HeapTuple stup,
oldtup;
@@ -206,18 +264,29 @@ update_mv_stats(Oid mvoid, MVDependencies dependencies, int2vector *attrs)
= PointerGetDatum(serialize_mv_dependencies(dependencies));
}
+ if (mcvlist != NULL)
+ {
+ bytea * data = serialize_mv_mcvlist(mcvlist, attrs, stats);
+ nulls[Anum_pg_mv_statistic_stamcv -1] = (data == NULL);
+ values[Anum_pg_mv_statistic_stamcv - 1] = PointerGetDatum(data);
+ }
+
/* always replace the value (either by bytea or NULL) */
replaces[Anum_pg_mv_statistic_stadeps -1] = true;
+ replaces[Anum_pg_mv_statistic_stamcv -1] = true;
/* always change the availability flags */
nulls[Anum_pg_mv_statistic_deps_built -1] = false;
+ nulls[Anum_pg_mv_statistic_mcv_built -1] = false;
nulls[Anum_pg_mv_statistic_stakeys-1] = false;
/* use the new attnums, in case we removed some dropped ones */
replaces[Anum_pg_mv_statistic_deps_built-1] = true;
+ replaces[Anum_pg_mv_statistic_mcv_built -1] = true;
replaces[Anum_pg_mv_statistic_stakeys -1] = true;
values[Anum_pg_mv_statistic_deps_built-1] = BoolGetDatum(dependencies != NULL);
+ values[Anum_pg_mv_statistic_mcv_built -1] = BoolGetDatum(mcvlist != NULL);
values[Anum_pg_mv_statistic_stakeys -1] = PointerGetDatum(attrs);
/* Is there already a pg_mv_statistic tuple for this attribute? */
@@ -246,6 +315,21 @@ update_mv_stats(Oid mvoid, MVDependencies dependencies, int2vector *attrs)
heap_close(sd, RowExclusiveLock);
}
+
+int
+mv_get_index(AttrNumber varattno, int2vector * stakeys)
+{
+ int i, idx = 0;
+ for (i = 0; i < stakeys->dim1; i++)
+ {
+ if (stakeys->values[i] < varattno)
+ idx += 1;
+ else
+ break;
+ }
+ return idx;
+}
+
/* multi-variate stats comparator */
/*
@@ -256,11 +340,15 @@ update_mv_stats(Oid mvoid, MVDependencies dependencies, int2vector *attrs)
int
compare_scalars_simple(const void *a, const void *b, void *arg)
{
- Datum da = *(Datum*)a;
- Datum db = *(Datum*)b;
- SortSupport ssup= (SortSupport) arg;
+ return compare_datums_simple(*(Datum*)a,
+ *(Datum*)b,
+ (SortSupport)arg);
+}
- return ApplySortComparator(da, false, db, false, ssup);
+int
+compare_datums_simple(Datum a, Datum b, SortSupport ssup)
+{
+ return ApplySortComparator(a, false, b, false, ssup);
}
/*
@@ -377,3 +465,32 @@ multi_sort_compare_dims(int start, int end,
return 0;
}
+
+/* simple counterpart to qsort_arg */
+void *
+bsearch_arg(const void *key, const void *base, size_t nmemb, size_t size,
+ int (*compar) (const void *, const void *, void *),
+ void *arg)
+{
+ size_t l, u, idx;
+ const void *p;
+ int comparison;
+
+ l = 0;
+ u = nmemb;
+ while (l < u)
+ {
+ idx = (l + u) / 2;
+ p = (void *) (((const char *) base) + (idx * size));
+ comparison = (*compar) (key, p, arg);
+
+ if (comparison < 0)
+ u = idx;
+ else if (comparison > 0)
+ l = idx + 1;
+ else
+ return (void *) p;
+ }
+
+ return NULL;
+}
diff --git a/src/backend/utils/mvstats/common.h b/src/backend/utils/mvstats/common.h
index 75b9c54..350760b 100644
--- a/src/backend/utils/mvstats/common.h
+++ b/src/backend/utils/mvstats/common.h
@@ -47,6 +47,14 @@ typedef struct
int tupno; /* position index for tuple it came from */
} ScalarItem;
+/* (de)serialization info */
+typedef struct DimensionInfo {
+ int nvalues; /* number of deduplicated values */
+ int nbytes; /* number of bytes (serialized) */
+ int typlen; /* pg_type.typlen */
+ bool typbyval; /* pg_type.typbyval */
+} DimensionInfo;
+
/* multi-sort */
typedef struct MultiSortSupportData {
int ndims; /* number of dimensions supported by the */
@@ -58,6 +66,7 @@ typedef MultiSortSupportData* MultiSortSupport;
typedef struct SortItem {
Datum *values;
bool *isnull;
+ int count;
} SortItem;
MultiSortSupport multi_sort_init(int ndims);
@@ -74,5 +83,11 @@ int multi_sort_compare_dims(int start, int end, const SortItem *a,
const SortItem *b, MultiSortSupport mss);
/* comparators, used when constructing multivariate stats */
+int compare_datums_simple(Datum a, Datum b, SortSupport ssup);
int compare_scalars_simple(const void *a, const void *b, void *arg);
int compare_scalars_partition(const void *a, const void *b, void *arg);
+
+void * bsearch_arg(const void *key, const void *base,
+ size_t nmemb, size_t size,
+ int (*compar) (const void *, const void *, void *),
+ void *arg);
diff --git a/src/backend/utils/mvstats/mcv.c b/src/backend/utils/mvstats/mcv.c
new file mode 100644
index 0000000..b300c1a
--- /dev/null
+++ b/src/backend/utils/mvstats/mcv.c
@@ -0,0 +1,1120 @@
+/*-------------------------------------------------------------------------
+ *
+ * mcv.c
+ * POSTGRES multivariate MCV lists
+ *
+ *
+ * Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/utils/mvstats/mcv.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "fmgr.h"
+#include "funcapi.h"
+
+#include "utils/lsyscache.h"
+
+#include "common.h"
+
+/*
+ * Each serialized item needs to store (in this order):
+ *
+ * - indexes (ndim * sizeof(uint16))
+ * - null flags (ndim * sizeof(bool))
+ * - frequency (sizeof(double))
+ *
+ * So in total:
+ *
+ * ndim * (sizeof(uint16) + sizeof(bool)) + sizeof(double)
+ */
+#define ITEM_SIZE(ndims) \
+ (ndims * (sizeof(uint16) + sizeof(bool)) + sizeof(double))
+
+/* Macros for convenient access to parts of the serialized MCV item */
+#define ITEM_INDEXES(item) ((uint16*)item)
+#define ITEM_NULLS(item,ndims) ((bool*)(ITEM_INDEXES(item) + ndims))
+#define ITEM_FREQUENCY(item,ndims) ((double*)(ITEM_NULLS(item,ndims) + ndims))
+
+static MultiSortSupport build_mss(VacAttrStats **stats, int2vector *attrs);
+
+static SortItem *build_sorted_items(int numrows, HeapTuple *rows,
+ TupleDesc tdesc, MultiSortSupport mss,
+ int2vector *attrs);
+
+static SortItem *build_distinct_groups(int numrows, SortItem *items,
+ MultiSortSupport mss, int *ndistinct);
+
+static int count_distinct_groups(int numrows, SortItem *items,
+ MultiSortSupport mss);
+
+/*
+ * Builds MCV list from the set of sampled rows.
+ *
+ * The algorithm is quite simple:
+ *
+ * (1) sort the data (default collation, '<' for the data type)
+ *
+ * (2) count distinct groups, decide how many to keep
+ *
+ * (3) build the MCV list using the threshold determined in (2)
+ *
+ * (4) remove rows represented by the MCV from the sample
+ *
+ * The method also removes rows matching the MCV items from the input array,
+ * and passes the number of remaining rows (useful for building histograms)
+ * using the numrows_filtered parameter.
+ *
+ * FIXME Use max_mcv_items from ALTER TABLE ADD STATISTICS command.
+ *
+ * FIXME Single-dimensional MCV is sorted by frequency (descending). We should
+ * do that too, because when walking through the list we want to check
+ * the most frequent items first.
+ *
+ * TODO We're using Datum (8B), even for data types (e.g. int4 or float4).
+ * Maybe we could save some space here, but the bytea compression should
+ * handle it just fine.
+ *
+ * TODO This probably should not use the ndistinct directly (as computed from
+ * the table, but rather estimate the number of distinct values in the
+ * table), no?
+ */
+MCVList
+build_mv_mcvlist(int numrows, HeapTuple *rows, int2vector *attrs,
+ VacAttrStats **stats, int *numrows_filtered)
+{
+ int i;
+ int numattrs = attrs->dim1;
+ int ndistinct = 0;
+ int mcv_threshold = 0;
+ int nitems = 0;
+
+ MCVList mcvlist = NULL;
+
+ /* comparator for all the columns */
+ MultiSortSupport mss = build_mss(stats, attrs);
+
+ /* sort the rows */
+ SortItem *items = build_sorted_items(numrows, rows, stats[0]->tupDesc,
+ mss, attrs);
+
+ /* transform the sorted rows into groups (sorted by frequency) */
+ SortItem *groups = build_distinct_groups(numrows, items, mss, &ndistinct);
+
+ /*
+ * Determine the minimum size of a group to be eligible for MCV list, and
+ * check how many groups actually pass that threshold. We use 1.25x the
+ * avarage group size, just like for regular statistics.
+ *
+ * But if we can fit all the distinct values in the MCV list (i.e. if there
+ * are less distinct groups than MVSTAT_MCVLIST_MAX_ITEMS), we'll require
+ * only 2 rows per group.
+ *
+ * FIXME This should really reference mcv_max_items (from catalog) instead
+ * of the constant MVSTAT_MCVLIST_MAX_ITEMS.
+ */
+ mcv_threshold = 1.25 * numrows / ndistinct;
+ mcv_threshold = (mcv_threshold < 4) ? 4 : mcv_threshold;
+
+ if (ndistinct <= MVSTAT_MCVLIST_MAX_ITEMS)
+ mcv_threshold = 2;
+
+ /* Walk through the groups and stop once we fall below the threshold. */
+ nitems = 0;
+ for (i = 0; i < ndistinct; i++)
+ {
+ if (groups[i].count < mcv_threshold)
+ break;
+
+ nitems++;
+ }
+
+ /* we know the number of MCV list items, so let's build the list */
+ if (nitems > 0)
+ {
+ /* allocate the MCV list structure, set parameters we know */
+ mcvlist = (MCVList)palloc0(sizeof(MCVListData));
+
+ mcvlist->magic = MVSTAT_MCV_MAGIC;
+ mcvlist->type = MVSTAT_MCV_TYPE_BASIC;
+ mcvlist->ndimensions = numattrs;
+ mcvlist->nitems = nitems;
+
+ /*
+ * Preallocate Datum/isnull arrays (not as a single chunk, as we will
+ * pass the result outside and thus it needs to be easy to pfree().
+ *
+ * XXX Although we're the only ones dealing with this.
+ */
+ mcvlist->items = (MCVItem*)palloc0(sizeof(MCVItem)*nitems);
+
+ for (i = 0; i < nitems; i++)
+ {
+ mcvlist->items[i] = (MCVItem)palloc0(sizeof(MCVItemData));
+ mcvlist->items[i]->values = (Datum*)palloc0(sizeof(Datum)*numattrs);
+ mcvlist->items[i]->isnull = (bool*)palloc0(sizeof(bool)*numattrs);
+ }
+
+ /* Copy the first chunk of groups into the result. */
+ for (i = 0; i < nitems; i++)
+ {
+ /* just pointer to the proper place in the list */
+ MCVItem item = mcvlist->items[i];
+
+ /* copy values from the _previous_ group (last item of) */
+ memcpy(item->values, groups[i].values, sizeof(Datum) * numattrs);
+ memcpy(item->isnull, groups[i].isnull, sizeof(bool) * numattrs);
+
+ /* and finally the group frequency */
+ item->frequency = (double)groups[i].count / numrows;
+ }
+
+ /* make sure the loops are consistent */
+ Assert(nitems == mcvlist->nitems);
+
+ /*
+ * Remove the rows matching the MCV list (i.e. keep only rows that are
+ * not represented by the MCV list). We will first sort the groups
+ * by the keys (not by count) and then use binary search.
+ */
+ if (nitems > ndistinct)
+ {
+ int i, j;
+ int nfiltered = 0;
+
+ /* used for the searches */
+ SortItem key;
+
+ /* wfill this with data from the rows */
+ key.values = (Datum*)palloc0(numattrs * sizeof(Datum));
+ key.isnull = (bool*)palloc0(numattrs * sizeof(bool));
+
+ /*
+ * Sort the groups for bsearch_r (but only the items that actually
+ * made it to the MCV list).
+ */
+ qsort_arg((void *) groups, nitems, sizeof(SortItem),
+ multi_sort_compare, mss);
+
+ /* walk through the tuples, compare the values to MCV items */
+ for (i = 0; i < numrows; i++)
+ {
+ /* collect the key values from the row */
+ for (j = 0; j < numattrs; j++)
+ key.values[j]
+ = heap_getattr(rows[i], attrs->values[j],
+ stats[j]->tupDesc, &key.isnull[j]);
+
+ /* if not included in the MCV list, keep it in the array */
+ if (bsearch_arg(&key, groups, nitems, sizeof(SortItem),
+ multi_sort_compare, mss) == NULL)
+ rows[nfiltered++] = rows[i];
+ }
+
+ /* remember how many rows we actually kept */
+ *numrows_filtered = nfiltered;
+
+ /* free all the data used here */
+ pfree(key.values);
+ pfree(key.isnull);
+ }
+ else
+ /* the MCV list convers all the rows */
+ *numrows_filtered = 0;
+ }
+
+ pfree(items);
+ pfree(groups);
+
+ return mcvlist;
+}
+
+/* build MultiSortSupport for the attributes passed in attrs */
+static MultiSortSupport
+build_mss(VacAttrStats **stats, int2vector *attrs)
+{
+ int i;
+ int numattrs = attrs->dim1;
+
+ /* Sort by multiple columns (using array of SortSupport) */
+ MultiSortSupport mss = multi_sort_init(numattrs);
+
+ /* prepare the sort functions for all the attributes */
+ for (i = 0; i < numattrs; i++)
+ multi_sort_add_dimension(mss, i, i, stats);
+
+ return mss;
+}
+
+/* build sorted array of SortItem with values from rows */
+static SortItem *
+build_sorted_items(int numrows, HeapTuple *rows, TupleDesc tdesc,
+ MultiSortSupport mss, int2vector *attrs)
+{
+ int i, j, len;
+ int numattrs = attrs->dim1;
+ int nvalues = numrows * numattrs;
+
+ /*
+ * We won't allocate the arrays for each item independenly, but in one large
+ * chunk and then just set the pointers.
+ */
+ SortItem *items;
+ Datum *values;
+ bool *isnull;
+ char *ptr;
+
+ /* Compute the total amount of memory we need (both items and values). */
+ len = numrows * sizeof(SortItem) + nvalues * (sizeof(Datum) + sizeof(bool));
+
+ /* Allocate the memory and split it into the pieces. */
+ ptr = palloc0(len);
+
+ /* items to sort */
+ items = (SortItem*)ptr;
+ ptr += numrows * sizeof(SortItem);
+
+ /* values and null flags */
+ values = (Datum*)ptr;
+ ptr += nvalues * sizeof(Datum);
+
+ isnull = (bool*)ptr;
+ ptr += nvalues * sizeof(bool);
+
+ /* make sure we consumed the whole buffer exactly */
+ Assert((ptr - (char*)items) == len);
+
+ /* fix the pointers to Datum and bool arrays */
+ for (i = 0; i < numrows; i++)
+ {
+ items[i].values = &values[i * numattrs];
+ items[i].isnull = &isnull[i * numattrs];
+
+ /* load the values/null flags from sample rows */
+ for (j = 0; j < numattrs; j++)
+ {
+ items[i].values[j] = heap_getattr(rows[i],
+ attrs->values[j], /* attnum */
+ tdesc,
+ &items[i].isnull[j]); /* isnull */
+ }
+ }
+
+ /* do the sort, using the multi-sort */
+ qsort_arg((void *) items, numrows, sizeof(SortItem),
+ multi_sort_compare, mss);
+
+ return items;
+}
+
+/* count distinct combinations of SortItems in the array */
+static int
+count_distinct_groups(int numrows, SortItem *items, MultiSortSupport mss)
+{
+ int i;
+ int ndistinct;
+
+ ndistinct = 1;
+ for (i = 1; i < numrows; i++)
+ if (multi_sort_compare(&items[i], &items[i-1], mss) != 0)
+ ndistinct += 1;
+
+ return ndistinct;
+}
+
+/* compares frequencies of the SortItem entries (in descending order) */
+static int
+compare_sort_item_count(const void *a, const void *b)
+{
+ SortItem *ia = (SortItem *)a;
+ SortItem *ib = (SortItem *)b;
+
+ if (ia->count == ib->count)
+ return 0;
+ else if (ia->count > ib->count)
+ return -1;
+
+ return 1;
+}
+
+/* builds SortItems for distinct groups and counts the matching items */
+static SortItem *
+build_distinct_groups(int numrows, SortItem *items, MultiSortSupport mss,
+ int *ndistinct)
+{
+ int i, j;
+ int ngroups = count_distinct_groups(numrows, items, mss);
+
+ SortItem *groups = (SortItem*)palloc0(ngroups * sizeof(SortItem));
+
+ j = 0;
+ groups[0] = items[0];
+ groups[0].count = 1;
+
+ for (i = 1; i < numrows; i++)
+ {
+ if (multi_sort_compare(&items[i], &items[i-1], mss) != 0)
+ groups[++j] = items[i];
+
+ groups[j].count++;
+ }
+
+ pg_qsort((void *) groups, ngroups, sizeof(SortItem),
+ compare_sort_item_count);
+
+ *ndistinct = ngroups;
+ return groups;
+}
+
+
+/* fetch the MCV list (as a bytea) from the pg_mv_statistic catalog */
+MCVList
+load_mv_mcvlist(Oid mvoid)
+{
+ bool isnull = false;
+ Datum mcvlist;
+
+#ifdef USE_ASSERT_CHECKING
+ Form_pg_mv_statistic mvstat;
+#endif
+
+ /* Prepare to scan pg_mv_statistic for entries having indrelid = this rel. */
+ HeapTuple htup = SearchSysCache1(MVSTATOID, ObjectIdGetDatum(mvoid));
+
+ if (! HeapTupleIsValid(htup))
+ return NULL;
+
+#ifdef USE_ASSERT_CHECKING
+ mvstat = (Form_pg_mv_statistic) GETSTRUCT(htup);
+ Assert(mvstat->mcv_enabled && mvstat->mcv_built);
+#endif
+
+ mcvlist = SysCacheGetAttr(MVSTATOID, htup,
+ Anum_pg_mv_statistic_stamcv, &isnull);
+
+ Assert(!isnull);
+
+ ReleaseSysCache(htup);
+
+ return deserialize_mv_mcvlist(DatumGetByteaP(mcvlist));
+}
+
+/* print some basic info about the MCV list
+ *
+ * TODO Add info about what part of the table this covers.
+ */
+Datum
+pg_mv_stats_mcvlist_info(PG_FUNCTION_ARGS)
+{
+ bytea *data = PG_GETARG_BYTEA_P(0);
+ char *result;
+
+ MCVList mcvlist = deserialize_mv_mcvlist(data);
+
+ result = palloc0(128);
+ snprintf(result, 128, "nitems=%d", mcvlist->nitems);
+
+ pfree(mcvlist);
+
+ PG_RETURN_TEXT_P(cstring_to_text(result));
+}
+
+/*
+ * serialize MCV list into a bytea value
+ *
+ *
+ * The basic algorithm is simple:
+ *
+ * (1) perform deduplication (for each attribute separately)
+ * (a) collect all (non-NULL) attribute values from all MCV items
+ * (b) sort the data (using 'lt' from VacAttrStats)
+ * (c) remove duplicate values from the array
+ *
+ * (2) serialize the arrays into a bytea value
+ *
+ * (3) process all MCV list items
+ * (a) replace values with indexes into the arrays
+ *
+ * Each attribute has to be processed separately, because we may be mixing
+ * different datatypes, with different sort operators, etc.
+ *
+ * We'll use uint16 values for the indexes in step (3), as we don't allow more
+ * than 8k MCV items (see list max_mcv_items), although that's mostly arbitrary
+ * limit. We might increase this to 65k and still fit into uint16.
+ *
+ * We don't really expect the serialization to save as much space as for
+ * histograms, because we are not doing any bucket splits (which is the source
+ * of high redundancy in histograms).
+ *
+ * TODO Consider packing boolean flags (NULL) for each item into a single char
+ * (or a longer type) instead of using an array of bool items.
+ */
+bytea *
+serialize_mv_mcvlist(MCVList mcvlist, int2vector *attrs,
+ VacAttrStats **stats)
+{
+ int i, j;
+ int ndims = mcvlist->ndimensions;
+ int itemsize = ITEM_SIZE(ndims);
+
+ SortSupport ssup;
+ DimensionInfo *info;
+
+ Size total_length;
+
+ /* allocate just once */
+ char *item = palloc0(itemsize);
+
+ /* serialized items (indexes into arrays, etc.) */
+ bytea *output;
+ char *data = NULL;
+
+ /* values per dimension (and number of non-NULL values) */
+ Datum **values = (Datum**)palloc0(sizeof(Datum*) * ndims);
+ int *counts = (int*)palloc0(sizeof(int) * ndims);
+
+ /*
+ * We'll include some rudimentary information about the attributes (type
+ * length, etc.), so that we don't have to look them up while deserializing
+ * the MCV list.
+ */
+ info = (DimensionInfo *)palloc0(sizeof(DimensionInfo)*ndims);
+
+ /* sort support data for all attributes included in the MCV list */
+ ssup = (SortSupport)palloc0(sizeof(SortSupportData)*ndims);
+
+ /* collect and deduplicate values for all attributes */
+ for (i = 0; i < ndims; i++)
+ {
+ int ndistinct;
+ StdAnalyzeData *tmp = (StdAnalyzeData *)stats[i]->extra_data;
+
+ /* copy important info about the data type (length, by-value) */
+ info[i].typlen = stats[i]->attrtype->typlen;
+ info[i].typbyval = stats[i]->attrtype->typbyval;
+
+ /* allocate space for values in the attribute and collect them */
+ values[i] = (Datum*)palloc0(sizeof(Datum) * mcvlist->nitems);
+
+ for (j = 0; j < mcvlist->nitems; j++)
+ {
+ /* skip NULL values - we don't need to serialize them */
+ if (mcvlist->items[j]->isnull[i])
+ continue;
+
+ values[i][counts[i]] = mcvlist->items[j]->values[i];
+ counts[i] += 1;
+ }
+
+ /* there are just NULL values in this dimension, we're done */
+ if (counts[i] == 0)
+ continue;
+
+ /* sort and deduplicate the data */
+ ssup[i].ssup_cxt = CurrentMemoryContext;
+ ssup[i].ssup_collation = DEFAULT_COLLATION_OID;
+ ssup[i].ssup_nulls_first = false;
+
+ PrepareSortSupportFromOrderingOp(tmp->ltopr, &ssup[i]);
+
+ qsort_arg(values[i], counts[i], sizeof(Datum),
+ compare_scalars_simple, &ssup[i]);
+
+ /*
+ * Walk through the array and eliminate duplicate values, but keep the
+ * ordering (so that we can do bsearch later). We know there's at least
+ * one item as (counts[i] != 0), so we can skip the first element.
+ */
+ ndistinct = 1; /* number of distinct values */
+ for (j = 1; j < counts[i]; j++)
+ {
+ /* if the value is the same as the previous one, we can skip it */
+ if (! compare_datums_simple(values[i][j-1], values[i][j], &ssup[i]))
+ continue;
+
+ values[i][ndistinct] = values[i][j];
+ ndistinct += 1;
+ }
+
+ /* we must not exceed UINT16_MAX, as we use uint16 indexes */
+ Assert(ndistinct <= UINT16_MAX);
+
+ /*
+ * Store additional info about the attribute - number of deduplicated
+ * values, and also size of the serialized data. For fixed-length data
+ * types this is trivial to compute, for varwidth types we need to
+ * actually walk the array and sum the sizes.
+ */
+ info[i].nvalues = ndistinct;
+
+ if (info[i].typlen > 0) /* fixed-length data types */
+ info[i].nbytes = info[i].nvalues * info[i].typlen;
+ else if (info[i].typlen == -1) /* varlena */
+ {
+ info[i].nbytes = 0;
+ for (j = 0; j < info[i].nvalues; j++)
+ info[i].nbytes += VARSIZE_ANY(values[i][j]);
+ }
+ else if (info[i].typlen == -2) /* cstring */
+ {
+ info[i].nbytes = 0;
+ for (j = 0; j < info[i].nvalues; j++)
+ info[i].nbytes += strlen(DatumGetPointer(values[i][j]));
+ }
+
+ /* we know (count>0) so there must be some data */
+ Assert(info[i].nbytes > 0);
+ }
+
+ /*
+ * Now we can finally compute how much space we'll actually need for the
+ * serialized MCV list, as it contains these fields:
+ *
+ * - length (4B) for varlena
+ * - magic (4B)
+ * - type (4B)
+ * - ndimensions (4B)
+ * - nitems (4B)
+ * - info (ndim * sizeof(DimensionInfo)
+ * - arrays of values for each dimension
+ * - serialized items (nitems * itemsize)
+ *
+ * So the 'header' size is 20B + ndim * sizeof(DimensionInfo) and then we
+ * will place all the data (values + indexes).
+ */
+ total_length = (sizeof(int32) + offsetof(MCVListData, items)
+ + ndims * sizeof(DimensionInfo)
+ + mcvlist->nitems * itemsize);
+
+ for (i = 0; i < ndims; i++)
+ total_length += info[i].nbytes;
+
+ /* enforce arbitrary limit of 1MB */
+ if (total_length > (1024 * 1024))
+ elog(ERROR, "serialized MCV list exceeds 1MB (%ld)", total_length);
+
+ /* allocate space for the serialized MCV list, set header fields */
+ output = (bytea*)palloc0(total_length);
+ SET_VARSIZE(output, total_length);
+
+ /* 'data' points to the current position in the output buffer */
+ data = VARDATA(output);
+
+ /* MCV list header (number of items, ...) */
+ memcpy(data, mcvlist, offsetof(MCVListData, items));
+ data += offsetof(MCVListData, items);
+
+ /* information about the attributes */
+ memcpy(data, info, sizeof(DimensionInfo) * ndims);
+ data += sizeof(DimensionInfo) * ndims;
+
+ /* now serialize the deduplicated values for all attributes */
+ for (i = 0; i < ndims; i++)
+ {
+#ifdef USE_ASSERT_CHECKING
+ char *tmp = data; /* remember the starting point */
+#endif
+ for (j = 0; j < info[i].nvalues; j++)
+ {
+ Datum v = values[i][j];
+
+ if (info[i].typbyval) /* passed by value */
+ {
+ memcpy(data, &v, info[i].typlen);
+ data += info[i].typlen;
+ }
+ else if (info[i].typlen > 0) /* pased by reference */
+ {
+ memcpy(data, DatumGetPointer(v), info[i].typlen);
+ data += info[i].typlen;
+ }
+ else if (info[i].typlen == -1) /* varlena */
+ {
+ memcpy(data, DatumGetPointer(v), VARSIZE_ANY(v));
+ data += VARSIZE_ANY(v);
+ }
+ else if (info[i].typlen == -2) /* cstring */
+ {
+ memcpy(data, DatumGetPointer(v), strlen(DatumGetPointer(v))+1);
+ data += strlen(DatumGetPointer(v)) + 1; /* terminator */
+ }
+ }
+
+ /* make sure we got exactly the amount of data we expected */
+ Assert((data - tmp) == info[i].nbytes);
+ }
+
+ /* finally serialize the items, with uint16 indexes instead of the values */
+ for (i = 0; i < mcvlist->nitems; i++)
+ {
+ MCVItem mcvitem = mcvlist->items[i];
+
+ /* don't write beyond the allocated space */
+ Assert(data <= (char*)output + total_length - itemsize);
+
+ /* reset the item (we only allocate it once and reuse it) */
+ memset(item, 0, itemsize);
+
+ for (j = 0; j < ndims; j++)
+ {
+ Datum *v = NULL;
+
+ /* do the lookup only for non-NULL values */
+ if (mcvlist->items[i]->isnull[j])
+ continue;
+
+ v = (Datum*)bsearch_arg(&mcvitem->values[j], values[j],
+ info[j].nvalues, sizeof(Datum),
+ compare_scalars_simple, &ssup[j]);
+
+ Assert(v != NULL); /* serialization or deduplication error */
+
+ /* compute index within the array */
+ ITEM_INDEXES(item)[j] = (v - values[j]);
+
+ /* check the index is within expected bounds */
+ Assert(ITEM_INDEXES(item)[j] >= 0);
+ Assert(ITEM_INDEXES(item)[j] < info[j].nvalues);
+ }
+
+ /* copy NULL and frequency flags into the item */
+ memcpy(ITEM_NULLS(item, ndims), mcvitem->isnull, sizeof(bool) * ndims);
+ memcpy(ITEM_FREQUENCY(item, ndims), &mcvitem->frequency, sizeof(double));
+
+ /* copy the serialized item into the array */
+ memcpy(data, item, itemsize);
+
+ data += itemsize;
+ }
+
+ /* at this point we expect to match the total_length exactly */
+ Assert((data - (char*)output) == total_length);
+
+ return output;
+}
+
+/*
+ * deserialize MCV list from the varlena value
+ *
+ *
+ * We deserialize the MCV list fully, because we don't expect there bo be a lot
+ * of duplicate values. But perhaps we should keep the MCV in serialized form
+ * just like histograms.
+ */
+MCVList deserialize_mv_mcvlist(bytea * data)
+{
+ int i, j;
+ Size expected_size;
+ MCVList mcvlist;
+ char *tmp;
+
+ int ndims, nitems, itemsize;
+ DimensionInfo *info = NULL;
+
+ uint16 *indexes = NULL;
+ Datum **values = NULL;
+
+ /* local allocation buffer (used only for deserialization) */
+ int bufflen;
+ char *buff;
+ char *ptr;
+
+ /* buffer used for the result */
+ int rbufflen;
+ char *rbuff;
+ char *rptr;
+
+ if (data == NULL)
+ return NULL;
+
+ /* we can't deserialize the MCV if there's not even a complete header */
+ expected_size = offsetof(MCVListData,items);
+
+ if (VARSIZE_ANY_EXHDR(data) < expected_size)
+ elog(ERROR, "invalid MCV Size %ld (expected at least %ld)",
+ VARSIZE_ANY_EXHDR(data), offsetof(MCVListData,items));
+
+ /* read the MCV list header */
+ mcvlist = (MCVList)palloc0(sizeof(MCVListData));
+
+ /* initialize pointer to the data part (skip the varlena header) */
+ tmp = VARDATA(data);
+
+ /* get the header and perform further sanity checks */
+ memcpy(mcvlist, tmp, offsetof(MCVListData,items));
+ tmp += offsetof(MCVListData,items);
+
+ if (mcvlist->magic != MVSTAT_MCV_MAGIC)
+ elog(ERROR, "invalid MCV magic %d (expected %dd)",
+ mcvlist->magic, MVSTAT_MCV_MAGIC);
+
+ if (mcvlist->type != MVSTAT_MCV_TYPE_BASIC)
+ elog(ERROR, "invalid MCV type %d (expected %dd)",
+ mcvlist->type, MVSTAT_MCV_TYPE_BASIC);
+
+ nitems = mcvlist->nitems;
+ ndims = mcvlist->ndimensions;
+ itemsize = ITEM_SIZE(ndims);
+
+ Assert((nitems > 0) && (nitems <= MVSTAT_MCVLIST_MAX_ITEMS));
+ Assert((ndims >= 2) && (ndims <= MVSTATS_MAX_DIMENSIONS));
+
+ /*
+ * Check amount of data including DimensionInfo for all dimensions and
+ * also the serialized items (including uint16 indexes). Also, walk
+ * through the dimension information and add it to the sum.
+ */
+ expected_size += ndims * sizeof(DimensionInfo) +
+ (nitems * itemsize);
+
+ /* check that we have at least the DimensionInfo records */
+ if (VARSIZE_ANY_EXHDR(data) < expected_size)
+ elog(ERROR, "invalid MCV size %ld (expected %ld)",
+ VARSIZE_ANY_EXHDR(data), expected_size);
+
+ info = (DimensionInfo*)(tmp);
+ tmp += ndims * sizeof(DimensionInfo);
+
+ /* account for the value arrays */
+ for (i = 0; i < ndims; i++)
+ {
+ Assert(info[i].nvalues >= 0);
+ Assert(info[i].nbytes >= 0);
+
+ expected_size += info[i].nbytes;
+ }
+
+ if (VARSIZE_ANY_EXHDR(data) != expected_size)
+ elog(ERROR, "invalid MCV size %ld (expected %ld)",
+ VARSIZE_ANY_EXHDR(data), expected_size);
+
+ /* looks OK - not corrupted or something */
+
+ /*
+ * Allocate one large chunk of memory for the intermediate data, needed
+ * only for deserializing the MCV list (and allocate densely to minimize
+ * the palloc overhead).
+ *
+ * Let's see how much space we'll actually need, and also include space
+ * for the array with pointers.
+ */
+ bufflen = sizeof(Datum*) * ndims; /* space for pointers */
+
+ for (i = 0; i < ndims; i++)
+ /* for full-size byval types, we reuse the serialized value */
+ if (! (info[i].typbyval && info[i].typlen == sizeof(Datum)))
+ bufflen += (sizeof(Datum) * info[i].nvalues);
+
+ buff = palloc0(bufflen);
+ ptr = buff;
+
+ values = (Datum**)buff;
+ ptr += (sizeof(Datum*) * ndims);
+
+ /*
+ * XXX This uses pointers to the original data array (the types not passed
+ * by value), so when someone frees the memory, e.g. by doing something
+ * like this:
+ *
+ * bytea * data = ... fetch the data from catalog ...
+ * MCVList mcvlist = deserialize_mcv_list(data);
+ * pfree(data);
+ *
+ * then 'mcvlist' references the freed memory. Should copy the pieces.
+ */
+ for (i = 0; i < ndims; i++)
+ {
+ if (info[i].typbyval)
+ {
+ /* passed by value / Datum - simply reuse the array */
+ if (info[i].typlen == sizeof(Datum))
+ {
+ values[i] = (Datum*)tmp;
+ tmp += info[i].nbytes;
+ }
+ else
+ {
+ values[i] = (Datum*)ptr;
+ ptr += (sizeof(Datum) * info[i].nvalues);
+
+ for (j = 0; j < info[i].nvalues; j++)
+ {
+ /* just point into the array */
+ memcpy(&values[i][j], tmp, info[i].typlen);
+ tmp += info[i].typlen;
+ }
+ }
+ }
+ else
+ {
+ /* all the other types need a chunk of the buffer */
+ values[i] = (Datum*)ptr;
+ ptr += (sizeof(Datum) * info[i].nvalues);
+
+ /* pased by reference, but fixed length (name, tid, ...) */
+ if (info[i].typlen > 0)
+ {
+ for (j = 0; j < info[i].nvalues; j++)
+ {
+ /* just point into the array */
+ values[i][j] = PointerGetDatum(tmp);
+ tmp += info[i].typlen;
+ }
+ }
+ else if (info[i].typlen == -1)
+ {
+ /* varlena */
+ for (j = 0; j < info[i].nvalues; j++)
+ {
+ /* just point into the array */
+ values[i][j] = PointerGetDatum(tmp);
+ tmp += VARSIZE_ANY(tmp);
+ }
+ }
+ else if (info[i].typlen == -2)
+ {
+ /* cstring */
+ for (j = 0; j < info[i].nvalues; j++)
+ {
+ /* just point into the array */
+ values[i][j] = PointerGetDatum(tmp);
+ tmp += (strlen(tmp) + 1); /* don't forget the \0 */
+ }
+ }
+ }
+ }
+
+ /* we should have exhausted the buffer exactly */
+ Assert((ptr - buff) == bufflen);
+
+ /* allocate space for all the MCV items in a single piece */
+ rbufflen = (sizeof(MCVItem) + sizeof(MCVItemData) +
+ sizeof(Datum)*ndims + sizeof(bool)*ndims) * nitems;
+
+ rbuff = palloc0(rbufflen);
+ rptr = rbuff;
+
+ mcvlist->items = (MCVItem*)rbuff;
+ rptr += (sizeof(MCVItem) * nitems);
+
+ for (i = 0; i < nitems; i++)
+ {
+ MCVItem item = (MCVItem)rptr;
+ rptr += (sizeof(MCVItemData));
+
+ item->values = (Datum*)rptr;
+ rptr += (sizeof(Datum)*ndims);
+
+ item->isnull = (bool*)rptr;
+ rptr += (sizeof(bool) *ndims);
+
+ /* just point to the right place */
+ indexes = ITEM_INDEXES(tmp);
+
+ memcpy(item->isnull, ITEM_NULLS(tmp, ndims), sizeof(bool) * ndims);
+ memcpy(&item->frequency, ITEM_FREQUENCY(tmp, ndims), sizeof(double));
+
+#ifdef ASSERT_CHECKING
+ for (j = 0; j < ndims; j++)
+ Assert(indexes[j] <= UINT16_MAX);
+#endif
+
+ /* translate the values */
+ for (j = 0; j < ndims; j++)
+ if (! item->isnull[j])
+ item->values[j] = values[j][indexes[j]];
+
+ mcvlist->items[i] = item;
+
+ tmp += ITEM_SIZE(ndims);
+
+ Assert(tmp <= (char*)data + VARSIZE_ANY(data));
+ }
+
+ /* check that we processed all the data */
+ Assert(tmp == (char*)data + VARSIZE_ANY(data));
+
+ /* release the temporary buffer */
+ pfree(buff);
+
+ return mcvlist;
+}
+
+/*
+ * SRF with details about buckets of a histogram:
+ *
+ * - item ID (0...nitems)
+ * - values (string array)
+ * - nulls only (boolean array)
+ * - frequency (double precision)
+ *
+ * The input is the OID of the statistics, and there are no rows returned if
+ * the statistics contains no histogram.
+ */
+PG_FUNCTION_INFO_V1(pg_mv_mcv_items);
+
+Datum
+pg_mv_mcv_items(PG_FUNCTION_ARGS)
+{
+ FuncCallContext *funcctx;
+ int call_cntr;
+ int max_calls;
+ TupleDesc tupdesc;
+ AttInMetadata *attinmeta;
+
+ /* stuff done only on the first call of the function */
+ if (SRF_IS_FIRSTCALL())
+ {
+ MemoryContext oldcontext;
+ MCVList mcvlist;
+
+ /* create a function context for cross-call persistence */
+ funcctx = SRF_FIRSTCALL_INIT();
+
+ /* switch to memory context appropriate for multiple function calls */
+ oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
+
+ mcvlist = load_mv_mcvlist(PG_GETARG_OID(0));
+
+ funcctx->user_fctx = mcvlist;
+
+ /* total number of tuples to be returned */
+ funcctx->max_calls = 0;
+ if (funcctx->user_fctx != NULL)
+ funcctx->max_calls = mcvlist->nitems;
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("function returning record called in context "
+ "that cannot accept type record")));
+
+ /* build metadata needed later to produce tuples from raw C-strings */
+ attinmeta = TupleDescGetAttInMetadata(tupdesc);
+ funcctx->attinmeta = attinmeta;
+
+ MemoryContextSwitchTo(oldcontext);
+ }
+
+ /* stuff done on every call of the function */
+ funcctx = SRF_PERCALL_SETUP();
+
+ call_cntr = funcctx->call_cntr;
+ max_calls = funcctx->max_calls;
+ attinmeta = funcctx->attinmeta;
+
+ if (call_cntr < max_calls) /* do when there is more left to send */
+ {
+ char **values;
+ HeapTuple tuple;
+ Datum result;
+ int2vector *stakeys;
+ Oid relid;
+
+ char *buff = palloc0(1024);
+ char *format;
+
+ int i;
+
+ Oid *outfuncs;
+ FmgrInfo *fmgrinfo;
+
+ MCVList mcvlist;
+ MCVItem item;
+
+ mcvlist = (MCVList)funcctx->user_fctx;
+
+ Assert(call_cntr < mcvlist->nitems);
+
+ item = mcvlist->items[call_cntr];
+
+ stakeys = find_mv_attnums(PG_GETARG_OID(0), &relid);
+
+ /*
+ * Prepare a values array for building the returned tuple. This should
+ * be an array of C strings which will be processed later by the type
+ * input functions.
+ */
+ values = (char **) palloc(4 * sizeof(char *));
+
+ values[0] = (char *) palloc(64 * sizeof(char));
+
+ /* arrays */
+ values[1] = (char *) palloc0(1024 * sizeof(char));
+ values[2] = (char *) palloc0(1024 * sizeof(char));
+
+ /* frequency */
+ values[3] = (char *) palloc(64 * sizeof(char));
+
+ outfuncs = (Oid*)palloc0(sizeof(Oid) * mcvlist->ndimensions);
+ fmgrinfo = (FmgrInfo*)palloc0(sizeof(FmgrInfo) * mcvlist->ndimensions);
+
+ for (i = 0; i < mcvlist->ndimensions; i++)
+ {
+ bool isvarlena;
+
+ getTypeOutputInfo(get_atttype(relid, stakeys->values[i]),
+ &outfuncs[i], &isvarlena);
+
+ fmgr_info(outfuncs[i], &fmgrinfo[i]);
+ }
+
+ snprintf(values[0], 64, "%d", call_cntr); /* item ID */
+
+ for (i = 0; i < mcvlist->ndimensions; i++)
+ {
+ Datum val, valout;
+
+ format = "%s, %s";
+ if (i == 0)
+ format = "{%s%s";
+ else if (i == mcvlist->ndimensions-1)
+ format = "%s, %s}";
+
+ if (item->isnull[i])
+ valout = CStringGetDatum("NULL");
+ else
+ {
+ val = item->values[i];
+ valout = FunctionCall1(&fmgrinfo[i], val);
+ }
+
+ snprintf(buff, 1024, format, values[1], DatumGetPointer(valout));
+ strncpy(values[1], buff, 1023);
+ buff[0] = '\0';
+
+ snprintf(buff, 1024, format, values[2], item->isnull[i] ? "t" : "f");
+ strncpy(values[2], buff, 1023);
+ buff[0] = '\0';
+ }
+
+ snprintf(values[3], 64, "%f", item->frequency); /* frequency */
+
+ /* build a tuple */
+ tuple = BuildTupleFromCStrings(attinmeta, values);
+
+ /* make the tuple into a datum */
+ result = HeapTupleGetDatum(tuple);
+
+ /* clean up (this is not really necessary) */
+ pfree(values[0]);
+ pfree(values[1]);
+ pfree(values[2]);
+ pfree(values[3]);
+
+ pfree(values);
+
+ SRF_RETURN_NEXT(funcctx, result);
+ }
+ else /* do when there is no more left */
+ {
+ SRF_RETURN_DONE(funcctx);
+ }
+}
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 8ce9c0e..2c22d31 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -2109,8 +2109,9 @@ describeOneTableDetails(const char *schemaname,
{
printfPQExpBuffer(&buf,
"SELECT oid, stanamespace::regnamespace AS nsp, staname, stakeys,\n"
- " deps_enabled,\n"
- " deps_built,\n"
+ " deps_enabled, mcv_enabled,\n"
+ " deps_built, mcv_built,\n"
+ " mcv_max_items,\n"
" (SELECT string_agg(attname::text,', ')\n"
" FROM ((SELECT unnest(stakeys) AS attnum) s\n"
" JOIN pg_attribute a ON (starelid = a.attrelid and a.attnum = s.attnum))) AS attnums\n"
@@ -2128,6 +2129,8 @@ describeOneTableDetails(const char *schemaname,
printTableAddFooter(&cont, _("Statistics:"));
for (i = 0; i < tuples; i++)
{
+ bool first = true;
+
printfPQExpBuffer(&buf, " ");
/* statistics name (qualified with namespace) */
@@ -2137,10 +2140,22 @@ describeOneTableDetails(const char *schemaname,
/* options */
if (!strcmp(PQgetvalue(result, i, 4), "t"))
- appendPQExpBuffer(&buf, "(dependencies)");
+ {
+ appendPQExpBuffer(&buf, "(dependencies");
+ first = false;
+ }
+
+ if (!strcmp(PQgetvalue(result, i, 5), "t"))
+ {
+ if (! first)
+ appendPQExpBuffer(&buf, ", mcv");
+ else
+ appendPQExpBuffer(&buf, "(mcv");
+ first = false;
+ }
- appendPQExpBuffer(&buf, " ON (%s)",
- PQgetvalue(result, i, 6));
+ appendPQExpBuffer(&buf, ") ON (%s)",
+ PQgetvalue(result, i, 9));
printTableAddFooter(&cont, buf.data);
}
diff --git a/src/include/catalog/pg_mv_statistic.h b/src/include/catalog/pg_mv_statistic.h
index c74af47..3529b03 100644
--- a/src/include/catalog/pg_mv_statistic.h
+++ b/src/include/catalog/pg_mv_statistic.h
@@ -38,15 +38,21 @@ CATALOG(pg_mv_statistic,3381)
/* statistics requested to build */
bool deps_enabled; /* analyze dependencies? */
+ bool mcv_enabled; /* build MCV list? */
+
+ /* MCV size */
+ int32 mcv_max_items; /* max MCV items */
/* statistics that are available (if requested) */
bool deps_built; /* dependencies were built */
+ bool mcv_built; /* MCV list was built */
/* variable-length fields start here, but we allow direct access to stakeys */
int2vector stakeys; /* array of column keys */
#ifdef CATALOG_VARLEN
bytea stadeps; /* dependencies (serialized) */
+ bytea stamcv; /* MCV list (serialized) */
#endif
} FormData_pg_mv_statistic;
@@ -62,14 +68,18 @@ typedef FormData_pg_mv_statistic *Form_pg_mv_statistic;
* compiler constants for pg_mv_statistic
* ----------------
*/
-#define Natts_pg_mv_statistic 8
+#define Natts_pg_mv_statistic 12
#define Anum_pg_mv_statistic_starelid 1
#define Anum_pg_mv_statistic_staname 2
#define Anum_pg_mv_statistic_stanamespace 3
#define Anum_pg_mv_statistic_staowner 4
#define Anum_pg_mv_statistic_deps_enabled 5
-#define Anum_pg_mv_statistic_deps_built 6
-#define Anum_pg_mv_statistic_stakeys 7
-#define Anum_pg_mv_statistic_stadeps 8
+#define Anum_pg_mv_statistic_mcv_enabled 6
+#define Anum_pg_mv_statistic_mcv_max_items 7
+#define Anum_pg_mv_statistic_deps_built 8
+#define Anum_pg_mv_statistic_mcv_built 9
+#define Anum_pg_mv_statistic_stakeys 10
+#define Anum_pg_mv_statistic_stadeps 11
+#define Anum_pg_mv_statistic_stamcv 12
#endif /* PG_MV_STATISTIC_H */
diff --git a/src/include/catalog/pg_proc.h b/src/include/catalog/pg_proc.h
index cdcbf95..5640dc1 100644
--- a/src/include/catalog/pg_proc.h
+++ b/src/include/catalog/pg_proc.h
@@ -2670,6 +2670,10 @@ DATA(insert OID = 3998 ( pg_mv_stats_dependencies_info PGNSP PGUID 12 1 0 0
DESCR("multivariate stats: functional dependencies info");
DATA(insert OID = 3999 ( pg_mv_stats_dependencies_show PGNSP PGUID 12 1 0 0 0 f f f f t f i s 1 0 25 "17" _null_ _null_ _null_ _null_ _null_ pg_mv_stats_dependencies_show _null_ _null_ _null_ ));
DESCR("multivariate stats: functional dependencies show");
+DATA(insert OID = 3376 ( pg_mv_stats_mcvlist_info PGNSP PGUID 12 1 0 0 0 f f f f t f i s 1 0 25 "17" _null_ _null_ _null_ _null_ _null_ pg_mv_stats_mcvlist_info _null_ _null_ _null_ ));
+DESCR("multi-variate statistics: MCV list info");
+DATA(insert OID = 3373 ( pg_mv_mcv_items PGNSP PGUID 12 1 1000 0 0 f f f f t t i s 1 0 2249 "26" "{26,23,1009,1000,701}" "{i,o,o,o,o}" "{oid,index,values,nulls,frequency}" _null_ _null_ pg_mv_mcv_items _null_ _null_ _null_ ));
+DESCR("details about MCV list items");
DATA(insert OID = 1928 ( pg_stat_get_numscans PGNSP PGUID 12 1 0 0 0 f f f f t f s r 1 0 20 "26" _null_ _null_ _null_ _null_ _null_ pg_stat_get_numscans _null_ _null_ _null_ ));
DESCR("statistics: number of scans done for table/index");
diff --git a/src/include/nodes/relation.h b/src/include/nodes/relation.h
index 75c4752..f52884a 100644
--- a/src/include/nodes/relation.h
+++ b/src/include/nodes/relation.h
@@ -655,9 +655,11 @@ typedef struct MVStatisticInfo
/* enabled statistics */
bool deps_enabled; /* functional dependencies enabled */
+ bool mcv_enabled; /* MCV list enabled */
/* built/available statistics */
bool deps_built; /* functional dependencies built */
+ bool mcv_built; /* MCV list built */
/* columns in the statistics (attnums) */
int2vector *stakeys; /* attnums of the columns covered */
diff --git a/src/include/utils/mvstats.h b/src/include/utils/mvstats.h
index ec55a09..b2643ec 100644
--- a/src/include/utils/mvstats.h
+++ b/src/include/utils/mvstats.h
@@ -17,6 +17,14 @@
#include "fmgr.h"
#include "commands/vacuum.h"
+/*
+ * Degree of how much MCV item matches a clause.
+ * This is then considered when computing the selectivity.
+ */
+#define MVSTATS_MATCH_NONE 0 /* no match at all */
+#define MVSTATS_MATCH_PARTIAL 1 /* partial match */
+#define MVSTATS_MATCH_FULL 2 /* full match */
+
#define MVSTATS_MAX_DIMENSIONS 8 /* max number of attributes */
/*
@@ -43,30 +51,89 @@ typedef MVDependenciesData* MVDependencies;
#define MVSTAT_DEPS_TYPE_BASIC 1 /* basic dependencies type */
/*
+ * Multivariate MCV (most-common value) lists
+ *
+ * A straight-forward extension of MCV items - i.e. a list (array) of
+ * combinations of attribute values, together with a frequency and
+ * null flags.
+ */
+typedef struct MCVItemData {
+ double frequency; /* frequency of this combination */
+ bool *isnull; /* lags of NULL values (up to 32 columns) */
+ Datum *values; /* variable-length (ndimensions) */
+} MCVItemData;
+
+typedef MCVItemData *MCVItem;
+
+/* multivariate MCV list - essentally an array of MCV items */
+typedef struct MCVListData {
+ uint32 magic; /* magic constant marker */
+ uint32 type; /* type of MCV list (BASIC) */
+ uint32 ndimensions; /* number of dimensions */
+ uint32 nitems; /* number of MCV items in the array */
+ MCVItem *items; /* array of MCV items */
+} MCVListData;
+
+typedef MCVListData *MCVList;
+
+/* used to flag stats serialized to bytea */
+#define MVSTAT_MCV_MAGIC 0xE1A651C2 /* marks serialized bytea */
+#define MVSTAT_MCV_TYPE_BASIC 1 /* basic MCV list type */
+
+/*
+ * Limits used for mcv_max_items option, i.e. we're always guaranteed
+ * to have space for at least MVSTAT_MCVLIST_MIN_ITEMS, and we cannot
+ * have more than MVSTAT_MCVLIST_MAX_ITEMS items.
+ *
+ * This is just a boundary for the 'max' threshold - the actual list
+ * may of course contain less items than MVSTAT_MCVLIST_MIN_ITEMS.
+ */
+#define MVSTAT_MCVLIST_MIN_ITEMS 128 /* min items in MCV list */
+#define MVSTAT_MCVLIST_MAX_ITEMS 8192 /* max items in MCV list */
+
+/*
* TODO Maybe fetching the histogram/MCV list separately is inefficient?
* Consider adding a single `fetch_stats` method, fetching all
* stats specified using flags (or something like that).
*/
MVDependencies load_mv_dependencies(Oid mvoid);
+MCVList load_mv_mcvlist(Oid mvoid);
bytea * serialize_mv_dependencies(MVDependencies dependencies);
+bytea * serialize_mv_mcvlist(MCVList mcvlist, int2vector *attrs,
+ VacAttrStats **stats);
/* deserialization of stats (serialization is private to analyze) */
MVDependencies deserialize_mv_dependencies(bytea * data);
+MCVList deserialize_mv_mcvlist(bytea * data);
+
+/*
+ * Returns index of the attribute number within the vector (i.e. a
+ * dimension within the stats).
+ */
+int mv_get_index(AttrNumber varattno, int2vector * stakeys);
+
+int2vector* find_mv_attnums(Oid mvoid, Oid *relid);
/* FIXME this probably belongs somewhere else (not to operations stats) */
extern Datum pg_mv_stats_dependencies_info(PG_FUNCTION_ARGS);
extern Datum pg_mv_stats_dependencies_show(PG_FUNCTION_ARGS);
+extern Datum pg_mv_stats_mcvlist_info(PG_FUNCTION_ARGS);
+extern Datum pg_mv_mcvlist_items(PG_FUNCTION_ARGS);
MVDependencies
-build_mv_dependencies(int numrows, HeapTuple *rows,
- int2vector *attrs,
- VacAttrStats **stats);
+build_mv_dependencies(int numrows, HeapTuple *rows, int2vector *attrs,
+ VacAttrStats **stats);
+
+MCVList
+build_mv_mcvlist(int numrows, HeapTuple *rows, int2vector *attrs,
+ VacAttrStats **stats, int *numrows_filtered);
void build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
- int natts, VacAttrStats **vacattrstats);
+ int natts, VacAttrStats **vacattrstats);
-void update_mv_stats(Oid relid, MVDependencies dependencies, int2vector *attrs);
+void update_mv_stats(Oid relid, MVDependencies dependencies, MCVList mcvlist,
+ int2vector *attrs, VacAttrStats **stats);
#endif
diff --git a/src/test/regress/expected/mv_mcv.out b/src/test/regress/expected/mv_mcv.out
new file mode 100644
index 0000000..075320b
--- /dev/null
+++ b/src/test/regress/expected/mv_mcv.out
@@ -0,0 +1,207 @@
+-- data type passed by value
+CREATE TABLE mcv_list (
+ a INT,
+ b INT,
+ c INT
+);
+-- unknown column
+CREATE STATISTICS s4 ON mcv_list (unknown_column) WITH (mcv);
+ERROR: column "unknown_column" referenced in statistics does not exist
+-- single column
+CREATE STATISTICS s4 ON mcv_list (a) WITH (mcv);
+ERROR: multivariate stats require 2 or more columns
+-- single column, duplicated
+CREATE STATISTICS s4 ON mcv_list (a, a) WITH (mcv);
+ERROR: duplicate column name in statistics definition
+-- two columns, one duplicated
+CREATE STATISTICS s4 ON mcv_list (a, a, b) WITH (mcv);
+ERROR: duplicate column name in statistics definition
+-- unknown option
+CREATE STATISTICS s4 ON mcv_list (a, b, c) WITH (unknown_option);
+ERROR: unrecognized STATISTICS option "unknown_option"
+-- missing MCV statistics
+CREATE STATISTICS s4 ON mcv_list (a, b, c) WITH (dependencies, max_mcv_items=200);
+ERROR: option 'mcv' is required by other options(s)
+-- invalid mcv_max_items value / too low
+CREATE STATISTICS s4 ON mcv_list (a, b, c) WITH (mcv, max_mcv_items=10);
+ERROR: max number of MCV items must be at least 128
+-- invalid mcv_max_items value / too high
+CREATE STATISTICS s4 ON mcv_list (a, b, c) WITH (mcv, max_mcv_items=10000);
+ERROR: max number of MCV items is 8192
+-- correct command
+CREATE STATISTICS s4 ON mcv_list (a, b, c) WITH (mcv);
+-- random data
+INSERT INTO mcv_list
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+ANALYZE mcv_list;
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+ mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info
+-------------+-----------+--------------------------
+ t | f |
+(1 row)
+
+TRUNCATE mcv_list;
+-- a => b, a => c, b => c
+INSERT INTO mcv_list
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE mcv_list;
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+ mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info
+-------------+-----------+--------------------------
+ t | t | nitems=1000
+(1 row)
+
+TRUNCATE mcv_list;
+-- a => b, a => c
+INSERT INTO mcv_list
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE mcv_list;
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+ mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info
+-------------+-----------+--------------------------
+ t | t | nitems=1000
+(1 row)
+
+TRUNCATE mcv_list;
+-- check explain (expect bitmap index scan, not plain index scan)
+INSERT INTO mcv_list
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX mcv_idx ON mcv_list (a, b);
+ANALYZE mcv_list;
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+ mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info
+-------------+-----------+--------------------------
+ t | t | nitems=100
+(1 row)
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mcv_list WHERE a = 10 AND b = 5;
+ QUERY PLAN
+--------------------------------------------
+ Bitmap Heap Scan on mcv_list
+ Recheck Cond: ((a = 10) AND (b = 5))
+ -> Bitmap Index Scan on mcv_idx
+ Index Cond: ((a = 10) AND (b = 5))
+(4 rows)
+
+DROP TABLE mcv_list;
+-- varlena type (text)
+CREATE TABLE mcv_list (
+ a TEXT,
+ b TEXT,
+ c TEXT
+);
+CREATE STATISTICS s5 ON mcv_list (a, b, c) WITH (mcv);
+-- random data
+INSERT INTO mcv_list
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+ANALYZE mcv_list;
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+ mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info
+-------------+-----------+--------------------------
+ t | f |
+(1 row)
+
+TRUNCATE mcv_list;
+-- a => b, a => c, b => c
+INSERT INTO mcv_list
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE mcv_list;
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+ mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info
+-------------+-----------+--------------------------
+ t | t | nitems=1000
+(1 row)
+
+TRUNCATE mcv_list;
+-- a => b, a => c
+INSERT INTO mcv_list
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE mcv_list;
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+ mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info
+-------------+-----------+--------------------------
+ t | t | nitems=1000
+(1 row)
+
+TRUNCATE mcv_list;
+-- check explain (expect bitmap index scan, not plain index scan)
+INSERT INTO mcv_list
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX mcv_idx ON mcv_list (a, b);
+ANALYZE mcv_list;
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+ mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info
+-------------+-----------+--------------------------
+ t | t | nitems=100
+(1 row)
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mcv_list WHERE a = '10' AND b = '5';
+ QUERY PLAN
+------------------------------------------------------------
+ Bitmap Heap Scan on mcv_list
+ Recheck Cond: ((a = '10'::text) AND (b = '5'::text))
+ -> Bitmap Index Scan on mcv_idx
+ Index Cond: ((a = '10'::text) AND (b = '5'::text))
+(4 rows)
+
+TRUNCATE mcv_list;
+-- check explain (expect bitmap index scan, not plain index scan) with NULLs
+INSERT INTO mcv_list
+ SELECT
+ (CASE WHEN i/10000 = 0 THEN NULL ELSE i/10000 END),
+ (CASE WHEN i/20000 = 0 THEN NULL ELSE i/20000 END),
+ (CASE WHEN i/40000 = 0 THEN NULL ELSE i/40000 END)
+ FROM generate_series(1,1000000) s(i);
+ANALYZE mcv_list;
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+ mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info
+-------------+-----------+--------------------------
+ t | t | nitems=100
+(1 row)
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mcv_list WHERE a IS NULL AND b IS NULL;
+ QUERY PLAN
+---------------------------------------------------
+ Bitmap Heap Scan on mcv_list
+ Recheck Cond: ((a IS NULL) AND (b IS NULL))
+ -> Bitmap Index Scan on mcv_idx
+ Index Cond: ((a IS NULL) AND (b IS NULL))
+(4 rows)
+
+DROP TABLE mcv_list;
+-- NULL values (mix of int and text columns)
+CREATE TABLE mcv_list (
+ a INT,
+ b TEXT,
+ c INT,
+ d TEXT
+);
+CREATE STATISTICS s6 ON mcv_list (a, b, c, d) WITH (mcv);
+INSERT INTO mcv_list
+ SELECT
+ mod(i, 100),
+ (CASE WHEN mod(i, 200) = 0 THEN NULL ELSE mod(i,200) END),
+ mod(i, 400),
+ (CASE WHEN mod(i, 300) = 0 THEN NULL ELSE mod(i,600) END)
+ FROM generate_series(1,10000) s(i);
+ANALYZE mcv_list;
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+ mcv_enabled | mcv_built | pg_mv_stats_mcvlist_info
+-------------+-----------+--------------------------
+ t | t | nitems=1200
+(1 row)
+
+DROP TABLE mcv_list;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 06f2231..3d55ffe 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1373,7 +1373,9 @@ pg_mv_stats| SELECT n.nspname AS schemaname,
s.staname,
s.stakeys AS attnums,
length(s.stadeps) AS depsbytes,
- pg_mv_stats_dependencies_info(s.stadeps) AS depsinfo
+ pg_mv_stats_dependencies_info(s.stadeps) AS depsinfo,
+ length(s.stamcv) AS mcvbytes,
+ pg_mv_stats_mcvlist_info(s.stamcv) AS mcvinfo
FROM ((pg_mv_statistic s
JOIN pg_class c ON ((c.oid = s.starelid)))
LEFT JOIN pg_namespace n ON ((n.oid = c.relnamespace)));
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index 4f2ffb8..85d94f1 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -112,4 +112,4 @@ test: event_trigger
test: stats
# run tests of multivariate stats
-test: mv_dependencies
+test: mv_dependencies mv_mcv
diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule
index 097a04f..6584d73 100644
--- a/src/test/regress/serial_schedule
+++ b/src/test/regress/serial_schedule
@@ -163,3 +163,4 @@ test: xml
test: event_trigger
test: stats
test: mv_dependencies
+test: mv_mcv
diff --git a/src/test/regress/sql/mv_mcv.sql b/src/test/regress/sql/mv_mcv.sql
new file mode 100644
index 0000000..b31d32d
--- /dev/null
+++ b/src/test/regress/sql/mv_mcv.sql
@@ -0,0 +1,178 @@
+-- data type passed by value
+CREATE TABLE mcv_list (
+ a INT,
+ b INT,
+ c INT
+);
+
+-- unknown column
+CREATE STATISTICS s4 ON mcv_list (unknown_column) WITH (mcv);
+
+-- single column
+CREATE STATISTICS s4 ON mcv_list (a) WITH (mcv);
+
+-- single column, duplicated
+CREATE STATISTICS s4 ON mcv_list (a, a) WITH (mcv);
+
+-- two columns, one duplicated
+CREATE STATISTICS s4 ON mcv_list (a, a, b) WITH (mcv);
+
+-- unknown option
+CREATE STATISTICS s4 ON mcv_list (a, b, c) WITH (unknown_option);
+
+-- missing MCV statistics
+CREATE STATISTICS s4 ON mcv_list (a, b, c) WITH (dependencies, max_mcv_items=200);
+
+-- invalid mcv_max_items value / too low
+CREATE STATISTICS s4 ON mcv_list (a, b, c) WITH (mcv, max_mcv_items=10);
+
+-- invalid mcv_max_items value / too high
+CREATE STATISTICS s4 ON mcv_list (a, b, c) WITH (mcv, max_mcv_items=10000);
+
+-- correct command
+CREATE STATISTICS s4 ON mcv_list (a, b, c) WITH (mcv);
+
+-- random data
+INSERT INTO mcv_list
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+
+ANALYZE mcv_list;
+
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+
+TRUNCATE mcv_list;
+
+-- a => b, a => c, b => c
+INSERT INTO mcv_list
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+
+ANALYZE mcv_list;
+
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+
+TRUNCATE mcv_list;
+
+-- a => b, a => c
+INSERT INTO mcv_list
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+
+ANALYZE mcv_list;
+
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+
+TRUNCATE mcv_list;
+
+-- check explain (expect bitmap index scan, not plain index scan)
+INSERT INTO mcv_list
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX mcv_idx ON mcv_list (a, b);
+
+ANALYZE mcv_list;
+
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mcv_list WHERE a = 10 AND b = 5;
+
+DROP TABLE mcv_list;
+
+-- varlena type (text)
+CREATE TABLE mcv_list (
+ a TEXT,
+ b TEXT,
+ c TEXT
+);
+
+CREATE STATISTICS s5 ON mcv_list (a, b, c) WITH (mcv);
+
+-- random data
+INSERT INTO mcv_list
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+
+ANALYZE mcv_list;
+
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+
+TRUNCATE mcv_list;
+
+-- a => b, a => c, b => c
+INSERT INTO mcv_list
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+
+ANALYZE mcv_list;
+
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+
+TRUNCATE mcv_list;
+
+-- a => b, a => c
+INSERT INTO mcv_list
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE mcv_list;
+
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+
+TRUNCATE mcv_list;
+
+-- check explain (expect bitmap index scan, not plain index scan)
+INSERT INTO mcv_list
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX mcv_idx ON mcv_list (a, b);
+ANALYZE mcv_list;
+
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mcv_list WHERE a = '10' AND b = '5';
+
+TRUNCATE mcv_list;
+
+-- check explain (expect bitmap index scan, not plain index scan) with NULLs
+INSERT INTO mcv_list
+ SELECT
+ (CASE WHEN i/10000 = 0 THEN NULL ELSE i/10000 END),
+ (CASE WHEN i/20000 = 0 THEN NULL ELSE i/20000 END),
+ (CASE WHEN i/40000 = 0 THEN NULL ELSE i/40000 END)
+ FROM generate_series(1,1000000) s(i);
+ANALYZE mcv_list;
+
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mcv_list WHERE a IS NULL AND b IS NULL;
+
+DROP TABLE mcv_list;
+
+-- NULL values (mix of int and text columns)
+CREATE TABLE mcv_list (
+ a INT,
+ b TEXT,
+ c INT,
+ d TEXT
+);
+
+CREATE STATISTICS s6 ON mcv_list (a, b, c, d) WITH (mcv);
+
+INSERT INTO mcv_list
+ SELECT
+ mod(i, 100),
+ (CASE WHEN mod(i, 200) = 0 THEN NULL ELSE mod(i,200) END),
+ mod(i, 400),
+ (CASE WHEN mod(i, 300) = 0 THEN NULL ELSE mod(i,600) END)
+ FROM generate_series(1,10000) s(i);
+
+ANALYZE mcv_list;
+
+SELECT mcv_enabled, mcv_built, pg_mv_stats_mcvlist_info(stamcv)
+ FROM pg_mv_statistic WHERE starelid = 'mcv_list'::regclass;
+
+DROP TABLE mcv_list;
--
2.5.0
[text/x-patch] 0005-multivariate-histograms.patch (137.0K, ../../7aa6145f-302b-2077-b961-702cc61710bb@2ndquadrant.com/6-0005-multivariate-histograms.patch)
download | inline diff:
From eb184e590bc0d1e41b9bf69d7cdc6d09e28daac3 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Sun, 11 Jan 2015 20:18:24 +0100
Subject: [PATCH 5/9] multivariate histograms
- extends the pg_mv_statistic catalog (add 'hist' fields)
- building the histograms during ANALYZE
- simple estimation while planning the queries
Includes regression tests mostly equal to those for functional
dependencies / MCV lists.
---
doc/src/sgml/ref/create_statistics.sgml | 44 +
src/backend/catalog/system_views.sql | 4 +-
src/backend/commands/statscmds.c | 44 +-
src/backend/nodes/outfuncs.c | 2 +
src/backend/optimizer/path/clausesel.c | 584 +++++++-
src/backend/optimizer/util/plancat.c | 4 +-
src/backend/utils/mvstats/Makefile | 2 +-
src/backend/utils/mvstats/README.histogram | 299 ++++
src/backend/utils/mvstats/README.stats | 2 +
src/backend/utils/mvstats/common.c | 37 +-
src/backend/utils/mvstats/histogram.c | 2023 ++++++++++++++++++++++++++++
src/bin/psql/describe.c | 17 +-
src/include/catalog/pg_mv_statistic.h | 24 +-
src/include/catalog/pg_proc.h | 4 +
src/include/nodes/relation.h | 2 +
src/include/utils/mvstats.h | 138 +-
src/test/regress/expected/mv_histogram.out | 207 +++
src/test/regress/expected/rules.out | 4 +-
src/test/regress/parallel_schedule | 2 +-
src/test/regress/serial_schedule | 1 +
src/test/regress/sql/mv_histogram.sql | 176 +++
21 files changed, 3576 insertions(+), 44 deletions(-)
create mode 100644 src/backend/utils/mvstats/README.histogram
create mode 100644 src/backend/utils/mvstats/histogram.c
create mode 100644 src/test/regress/expected/mv_histogram.out
create mode 100644 src/test/regress/sql/mv_histogram.sql
diff --git a/doc/src/sgml/ref/create_statistics.sgml b/doc/src/sgml/ref/create_statistics.sgml
index d6973e8..f7336fd 100644
--- a/doc/src/sgml/ref/create_statistics.sgml
+++ b/doc/src/sgml/ref/create_statistics.sgml
@@ -133,6 +133,24 @@ CREATE STATISTICS [ IF NOT EXISTS ] <replaceable class="PARAMETER">statistics_na
</varlistentry>
<varlistentry>
+ <term><literal>histogram</> (<type>boolean</>)</term>
+ <listitem>
+ <para>
+ Enables histogram for the statistics.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>max_buckets</> (<type>integer</>)</term>
+ <listitem>
+ <para>
+ Maximum number of histogram buckets.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
<term><literal>max_mcv_items</> (<type>integer</>)</term>
<listitem>
<para>
@@ -220,6 +238,32 @@ EXPLAIN ANALYZE SELECT * FROM t2 WHERE (a = 1) AND (b = 2);
</programlisting>
</para>
+ <para>
+ Create table <structname>t3</> with two strongly correlated columns, and
+ a histogram on those two columns:
+
+<programlisting>
+CREATE TABLE t3 (
+ a float,
+ b float
+);
+
+INSERT INTO t3 SELECT mod(i,1000), mod(i,1000) + 50 * (r - 0.5) FROM (
+ SELECT i, random() r FROM generate_series(1,1000000) s(i)
+ ) foo;
+
+CREATE STATISTICS s3 ON t3 (a, b) WITH (histogram);
+
+ANALYZE t2;
+
+-- small overlap
+EXPLAIN ANALYZE SELECT * FROM t3 WHERE (a < 500) AND (b > 500);
+
+-- no overlap
+EXPLAIN ANALYZE SELECT * FROM t3 WHERE (a < 400) AND (b > 600);
+</programlisting>
+ </para>
+
</refsect1>
<refsect1>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 5c40334..b151db1 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -167,7 +167,9 @@ CREATE VIEW pg_mv_stats AS
length(S.stadeps) as depsbytes,
pg_mv_stats_dependencies_info(S.stadeps) as depsinfo,
length(S.stamcv) AS mcvbytes,
- pg_mv_stats_mcvlist_info(S.stamcv) AS mcvinfo
+ pg_mv_stats_mcvlist_info(S.stamcv) AS mcvinfo,
+ length(S.stahist) AS histbytes,
+ pg_mv_stats_histogram_info(S.stahist) AS histinfo
FROM (pg_mv_statistic S JOIN pg_class C ON (C.oid = S.starelid))
LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace);
diff --git a/src/backend/commands/statscmds.c b/src/backend/commands/statscmds.c
index c480fbe..e0b085f 100644
--- a/src/backend/commands/statscmds.c
+++ b/src/backend/commands/statscmds.c
@@ -71,12 +71,15 @@ CreateStatistics(CreateStatsStmt *stmt)
/* by default build nothing */
bool build_dependencies = false,
- build_mcv = false;
+ build_mcv = false,
+ build_histogram = false;
- int32 max_mcv_items = -1;
+ int32 max_buckets = -1,
+ max_mcv_items = -1;
/* options required because of other options */
- bool require_mcv = false;
+ bool require_mcv = false,
+ require_histogram = false;
Assert(IsA(stmt, CreateStatsStmt));
@@ -175,6 +178,29 @@ CreateStatistics(CreateStatsStmt *stmt)
MVSTAT_MCVLIST_MAX_ITEMS)));
}
+ else if (strcmp(opt->defname, "histogram") == 0)
+ build_histogram = defGetBoolean(opt);
+ else if (strcmp(opt->defname, "max_buckets") == 0)
+ {
+ max_buckets = defGetInt32(opt);
+
+ /* this option requires 'histogram' to be enabled */
+ require_histogram = true;
+
+ /* sanity check */
+ if (max_buckets < MVSTAT_HIST_MIN_BUCKETS)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("minimum number of buckets is %d",
+ MVSTAT_HIST_MIN_BUCKETS)));
+
+ else if (max_buckets > MVSTAT_HIST_MAX_BUCKETS)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("maximum number of buckets is %d",
+ MVSTAT_HIST_MAX_BUCKETS)));
+
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -183,10 +209,10 @@ CreateStatistics(CreateStatsStmt *stmt)
}
/* check that at least some statistics were requested */
- if (! (build_dependencies || build_mcv))
+ if (! (build_dependencies || build_mcv || build_histogram))
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("no statistics type (dependencies, mcv) was requested")));
+ errmsg("no statistics type (dependencies, mcv, histogram) was requested")));
/* now do some checking of the options */
if (require_mcv && (! build_mcv))
@@ -194,6 +220,11 @@ CreateStatistics(CreateStatsStmt *stmt)
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("option 'mcv' is required by other options(s)")));
+ if (require_histogram && (! build_histogram))
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("option 'histogram' is required by other options(s)")));
+
/* sort the attnums and build int2vector */
qsort(attnums, numcols, sizeof(int16), compare_int16);
stakeys = buildint2vector(attnums, numcols);
@@ -214,11 +245,14 @@ CreateStatistics(CreateStatsStmt *stmt)
values[Anum_pg_mv_statistic_deps_enabled -1] = BoolGetDatum(build_dependencies);
values[Anum_pg_mv_statistic_mcv_enabled -1] = BoolGetDatum(build_mcv);
+ values[Anum_pg_mv_statistic_hist_enabled -1] = BoolGetDatum(build_histogram);
values[Anum_pg_mv_statistic_mcv_max_items -1] = Int32GetDatum(max_mcv_items);
+ values[Anum_pg_mv_statistic_hist_max_buckets -1] = Int32GetDatum(max_buckets);
nulls[Anum_pg_mv_statistic_stadeps -1] = true;
nulls[Anum_pg_mv_statistic_stamcv -1] = true;
+ nulls[Anum_pg_mv_statistic_stahist -1] = true;
/* insert the tuple into pg_mv_statistic */
mvstatrel = heap_open(MvStatisticRelationId, RowExclusiveLock);
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 333e24b..9172f21 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2163,10 +2163,12 @@ _outMVStatisticInfo(StringInfo str, const MVStatisticInfo *node)
/* enabled statistics */
WRITE_BOOL_FIELD(deps_enabled);
WRITE_BOOL_FIELD(mcv_enabled);
+ WRITE_BOOL_FIELD(hist_enabled);
/* built/available statistics */
WRITE_BOOL_FIELD(deps_built);
WRITE_BOOL_FIELD(mcv_built);
+ WRITE_BOOL_FIELD(hist_built);
}
static void
diff --git a/src/backend/optimizer/path/clausesel.c b/src/backend/optimizer/path/clausesel.c
index c16d559..fe96a73 100644
--- a/src/backend/optimizer/path/clausesel.c
+++ b/src/backend/optimizer/path/clausesel.c
@@ -49,6 +49,7 @@ static void addRangeClause(RangeQueryClause **rqlist, Node *clause,
#define MV_CLAUSE_TYPE_FDEP 0x01
#define MV_CLAUSE_TYPE_MCV 0x02
+#define MV_CLAUSE_TYPE_HIST 0x04
static bool clause_is_mv_compatible(Node *clause, Index relid, Bitmapset **attnums,
int type);
@@ -74,6 +75,8 @@ static Selectivity clauselist_mv_selectivity(PlannerInfo *root,
static Selectivity clauselist_mv_selectivity_mcvlist(PlannerInfo *root,
List *clauses, MVStatisticInfo *mvstats,
bool *fullmatch, Selectivity *lowsel);
+static Selectivity clauselist_mv_selectivity_histogram(PlannerInfo *root,
+ List *clauses, MVStatisticInfo *mvstats);
static int update_match_bitmap_mcvlist(PlannerInfo *root, List *clauses,
int2vector *stakeys, MCVList mcvlist,
@@ -81,6 +84,12 @@ static int update_match_bitmap_mcvlist(PlannerInfo *root, List *clauses,
Selectivity *lowsel, bool *fullmatch,
bool is_or);
+static int update_match_bitmap_histogram(PlannerInfo *root, List *clauses,
+ int2vector *stakeys,
+ MVSerializedHistogram mvhist,
+ int nmatches, char * matches,
+ bool is_or);
+
static bool has_stats(List *stats, int type);
static List * find_stats(PlannerInfo *root, Index relid);
@@ -95,6 +104,7 @@ static bool stats_type_matches(MVStatisticInfo *stat, int type);
#define UPDATE_RESULT(m,r,isor) \
(m) = (isor) ? (MAX(m,r)) : (MIN(m,r))
+
/****************************************************************************
* ROUTINES TO COMPUTE SELECTIVITIES
****************************************************************************/
@@ -123,7 +133,7 @@ static bool stats_type_matches(MVStatisticInfo *stat, int type);
*
* First we try to reduce the list of clauses by applying (soft) functional
* dependencies, and then we try to estimate the selectivity of the reduced
- * list of clauses using the multivariate MCV list.
+ * list of clauses using the multivariate MCV list and histograms.
*
* Finally we remove the portion of clauses estimated using multivariate stats,
* and process the rest of the clauses using the regular per-column stats.
@@ -216,11 +226,13 @@ clauselist_selectivity(PlannerInfo *root,
* with the multivariate code and simply skip to estimation using the
* regular per-column stats.
*/
- if (has_stats(stats, MV_CLAUSE_TYPE_MCV) &&
- (count_mv_attnums(clauses, relid, MV_CLAUSE_TYPE_MCV) >= 2))
+ if (has_stats(stats, MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST) &&
+ (count_mv_attnums(clauses, relid,
+ MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST) >= 2))
{
/* collect attributes from the compatible conditions */
- Bitmapset *mvattnums = collect_mv_attnums(clauses, relid, MV_CLAUSE_TYPE_MCV);
+ Bitmapset *mvattnums = collect_mv_attnums(clauses, relid,
+ MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST);
/* and search for the statistic covering the most attributes */
MVStatisticInfo *mvstat = choose_mv_statistics(stats, mvattnums);
@@ -232,7 +244,7 @@ clauselist_selectivity(PlannerInfo *root,
/* split the clauselist into regular and mv-clauses */
clauses = clauselist_mv_split(root, relid, clauses, &mvclauses,
- mvstat, MV_CLAUSE_TYPE_MCV);
+ mvstat, MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST);
/* we've chosen the histogram to match the clauses */
Assert(mvclauses != NIL);
@@ -944,6 +956,7 @@ static Selectivity
clauselist_mv_selectivity(PlannerInfo *root, List *clauses, MVStatisticInfo *mvstats)
{
bool fullmatch = false;
+ Selectivity s1 = 0.0, s2 = 0.0;
/*
* Lowest frequency in the MCV list (may be used as an upper bound
@@ -957,9 +970,24 @@ clauselist_mv_selectivity(PlannerInfo *root, List *clauses, MVStatisticInfo *mvs
* MCV/histogram evaluation).
*/
- /* Evaluate the MCV selectivity */
- return clauselist_mv_selectivity_mcvlist(root, clauses, mvstats,
+ /* Evaluate the MCV first. */
+ s1 = clauselist_mv_selectivity_mcvlist(root, clauses, mvstats,
&fullmatch, &mcv_low);
+
+ /*
+ * If we got a full equality match on the MCV list, we're done (and
+ * the estimate is pretty good).
+ */
+ if (fullmatch && (s1 > 0.0))
+ return s1;
+
+ /* TODO if (fullmatch) without matching MCV item, use the mcv_low
+ * selectivity as upper bound */
+
+ s2 = clauselist_mv_selectivity_histogram(root, clauses, mvstats);
+
+ /* TODO clamp to <= 1.0 (or more strictly, when possible) */
+ return s1 + s2;
}
/*
@@ -1039,7 +1067,7 @@ count_varnos(List *clauses, Index *relid)
return cnt;
}
-
+
/*
* We're looking for statistics matching at least 2 attributes, referenced in
* clauses compatible with multivariate statistics. The current selection
@@ -1129,7 +1157,7 @@ choose_mv_statistics(List *stats, Bitmapset *attnums)
int numattrs = attrs->dim1;
/* skip dependencies-only stats */
- if (! info->mcv_built)
+ if (! (info->mcv_built || info->hist_built))
continue;
/* count columns covered by the histogram */
@@ -1251,7 +1279,7 @@ mv_compatible_walker(Node *node, mv_compatible_context *context)
}
if (or_clause(node) || and_clause(node) || not_clause(node))
- {
+ {
/*
* AND/OR/NOT-clauses are supported if all sub-clauses are supported
*
@@ -1277,10 +1305,10 @@ mv_compatible_walker(Node *node, mv_compatible_context *context)
}
return false;
- }
+ }
if (IsA(node, NullTest))
- {
+ {
NullTest* nt = (NullTest*)node;
/*
@@ -1360,9 +1388,9 @@ mv_compatible_walker(Node *node, mv_compatible_context *context)
case F_SCALARGTSEL:
/* not compatible with functional dependencies */
- if (! (context->types & MV_CLAUSE_TYPE_MCV))
+ if (! (context->types & (MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST)))
return true; /* terminate */
-
+
break;
default:
@@ -1588,6 +1616,9 @@ stats_type_matches(MVStatisticInfo *stat, int type)
if ((type & MV_CLAUSE_TYPE_MCV) && stat->mcv_built)
return true;
+ if ((type & MV_CLAUSE_TYPE_HIST) && stat->hist_built)
+ return true;
+
return false;
}
@@ -1606,6 +1637,9 @@ has_stats(List *stats, int type)
/* terminate if we've found at least one matching statistics */
if (stats_type_matches(stat, type))
return true;
+
+ if ((type & MV_CLAUSE_TYPE_HIST) && stat->hist_built)
+ return true;
}
return false;
@@ -2010,3 +2044,525 @@ update_match_bitmap_mcvlist(PlannerInfo *root, List *clauses,
return nmatches;
}
+
+/*
+ * Estimate selectivity of clauses using a histogram.
+ *
+ * If there's no histogram for the stats, the function returns 0.0.
+ *
+ * The general idea of this method is similar to how MCV lists are
+ * processed, except that this introduces the concept of a partial
+ * match (MCV only works with full match / mismatch).
+ *
+ * The algorithm works like this:
+ *
+ * 1) mark all buckets as 'full match'
+ * 2) walk through all the clauses
+ * 3) for a particular clause, walk through all the buckets
+ * 4) skip buckets that are already 'no match'
+ * 5) check clause for buckets that still match (at least partially)
+ * 6) sum frequencies for buckets to get selectivity
+ *
+ * Unlike MCV lists, histograms have a concept of a partial match. In
+ * that case we use 1/2 the bucket, to minimize the average error. The
+ * MV histograms are usually less detailed than the per-column ones,
+ * meaning the sum is often quite high (thanks to combining a lot of
+ * "partially hit" buckets).
+ *
+ * Maybe we could use per-bucket information with number of distinct
+ * values it contains (for each dimension), and then use that to correct
+ * the estimate (so with 10 distinct values, we'd use 1/10 of the bucket
+ * frequency). We might also scale the value depending on the actual
+ * ndistinct estimate (not just the values observed in the sample).
+ *
+ * Another option would be to multiply the selectivities, i.e. if we get
+ * 'partial match' for a bucket for multiple conditions, we might use
+ * 0.5^k (where k is the number of conditions), instead of 0.5. This
+ * probably does not minimize the average error, though.
+ *
+ * TODO This might use a similar shortcut to MCV lists - count buckets
+ * marked as partial/full match, and terminate once this drop to 0.
+ * Not sure if it's really worth it - for MCV lists a situation like
+ * this is not uncommon, but for histograms it's not that clear.
+ */
+static Selectivity
+clauselist_mv_selectivity_histogram(PlannerInfo *root, List *clauses,
+ MVStatisticInfo *mvstats)
+{
+ int i;
+ Selectivity s = 0.0;
+ Selectivity u = 0.0;
+
+ int nmatches = 0;
+ char *matches = NULL;
+
+ MVSerializedHistogram mvhist = NULL;
+
+ /* there's no histogram */
+ if (! mvstats->hist_built)
+ return 0.0;
+
+ /* There may be no histogram in the stats (check hist_built flag) */
+ mvhist = load_mv_histogram(mvstats->mvoid);
+
+ Assert (mvhist != NULL);
+ Assert (clauses != NIL);
+ Assert (list_length(clauses) >= 2);
+
+ /*
+ * Bitmap of bucket matches (mismatch, partial, full). by default
+ * all buckets fully match (and we'll eliminate them).
+ */
+ matches = palloc0(sizeof(char) * mvhist->nbuckets);
+ memset(matches, MVSTATS_MATCH_FULL, sizeof(char)*mvhist->nbuckets);
+
+ nmatches = mvhist->nbuckets;
+
+ /* build the match bitmap */
+ update_match_bitmap_histogram(root, clauses,
+ mvstats->stakeys, mvhist,
+ nmatches, matches, false);
+
+ /* now, walk through the buckets and sum the selectivities */
+ for (i = 0; i < mvhist->nbuckets; i++)
+ {
+ /*
+ * Find out what part of the data is covered by the histogram,
+ * so that we can 'scale' the selectivity properly (e.g. when
+ * only 50% of the sample got into the histogram, and the rest
+ * is in a MCV list).
+ *
+ * TODO This might be handled by keeping a global "frequency"
+ * for the whole histogram, which might save us some time
+ * spent accessing the not-matching part of the histogram.
+ * Although it's likely in a cache, so it's very fast.
+ */
+ u += mvhist->buckets[i]->ntuples;
+
+ if (matches[i] == MVSTATS_MATCH_FULL)
+ s += mvhist->buckets[i]->ntuples;
+ else if (matches[i] == MVSTATS_MATCH_PARTIAL)
+ s += 0.5 * mvhist->buckets[i]->ntuples;
+ }
+
+#ifdef DEBUG_MVHIST
+ debug_histogram_matches(mvhist, matches);
+#endif
+
+ /* release the allocated bitmap and deserialized histogram */
+ pfree(matches);
+ pfree(mvhist);
+
+ return s * u;
+}
+
+/* cached result of bucket boundary comparison for a single dimension */
+
+#define HIST_CACHE_NOT_FOUND 0x00
+#define HIST_CACHE_FALSE 0x01
+#define HIST_CACHE_TRUE 0x03
+#define HIST_CACHE_MASK 0x02
+
+static char
+bucket_contains_value(FmgrInfo ltproc, Datum constvalue,
+ Datum min_value, Datum max_value,
+ int min_index, int max_index,
+ bool min_include, bool max_include,
+ char * callcache)
+{
+ bool a, b;
+
+ char min_cached = callcache[min_index];
+ char max_cached = callcache[max_index];
+
+ /*
+ * First some quick checks on equality - if any of the boundaries equals,
+ * we have a partial match (so no need to call the comparator).
+ */
+ if (((min_value == constvalue) && (min_include)) ||
+ ((max_value == constvalue) && (max_include)))
+ return MVSTATS_MATCH_PARTIAL;
+
+ /* Keep the values 0/1 because of the XOR at the end. */
+ a = ((min_cached & HIST_CACHE_MASK) >> 1);
+ b = ((max_cached & HIST_CACHE_MASK) >> 1);
+
+ /*
+ * If result for the bucket lower bound not in cache, evaluate the function
+ * and store the result in the cache.
+ */
+ if (! min_cached)
+ {
+ a = DatumGetBool(FunctionCall2Coll(<proc,
+ DEFAULT_COLLATION_OID,
+ constvalue, min_value));
+ /* remember the result */
+ callcache[min_index] = (a) ? HIST_CACHE_TRUE : HIST_CACHE_FALSE;
+ }
+
+ /* And do the same for the upper bound. */
+ if (! max_cached)
+ {
+ b = DatumGetBool(FunctionCall2Coll(<proc,
+ DEFAULT_COLLATION_OID,
+ constvalue, max_value));
+ /* remember the result */
+ callcache[max_index] = (b) ? HIST_CACHE_TRUE : HIST_CACHE_FALSE;
+ }
+
+ return (a ^ b) ? MVSTATS_MATCH_PARTIAL : MVSTATS_MATCH_NONE;
+}
+
+static char
+bucket_is_smaller_than_value(FmgrInfo opproc, Datum constvalue,
+ Datum min_value, Datum max_value,
+ int min_index, int max_index,
+ bool min_include, bool max_include,
+ char * callcache, bool isgt)
+{
+ char min_cached = callcache[min_index];
+ char max_cached = callcache[max_index];
+
+ /* Keep the values 0/1 because of the XOR at the end. */
+ bool a = ((min_cached & HIST_CACHE_MASK) >> 1);
+ bool b = ((max_cached & HIST_CACHE_MASK) >> 1);
+
+ if (! min_cached)
+ {
+ a = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ min_value,
+ constvalue));
+ /* remember the result */
+ callcache[min_index] = (a) ? HIST_CACHE_TRUE : HIST_CACHE_FALSE;
+ }
+
+ if (! max_cached)
+ {
+ b = DatumGetBool(FunctionCall2Coll(&opproc,
+ DEFAULT_COLLATION_OID,
+ max_value,
+ constvalue));
+ /* remember the result */
+ callcache[max_index] = (b) ? HIST_CACHE_TRUE : HIST_CACHE_FALSE;
+ }
+
+ /*
+ * Now, we need to combine both results into the final answer, and we need
+ * to be careful about the 'isgt' variable which kinda inverts the meaning.
+ *
+ * First, we handle the case when each boundary returns different results.
+ * In that case the outcome can only be 'partial' match.
+ */
+ if (a != b)
+ return MVSTATS_MATCH_PARTIAL;
+
+ /*
+ * When the results are the same, then it depends on the 'isgt' value. There
+ * are four options:
+ *
+ * isgt=false a=b=true => full match
+ * isgt=false a=b=false => empty
+ * isgt=true a=b=true => empty
+ * isgt=true a=b=false => full match
+ *
+ * We'll cheat a bit, because we know that (a=b) so we'll use just one of them.
+ */
+ if (isgt)
+ return (!a) ? MVSTATS_MATCH_FULL : MVSTATS_MATCH_NONE;
+ else
+ return ( a) ? MVSTATS_MATCH_FULL : MVSTATS_MATCH_NONE;
+}
+
+/*
+ * Evaluate clauses using the histogram, and update the match bitmap.
+ *
+ * The bitmap may be already partially set, so this is really a way to
+ * combine results of several clause lists - either when computing
+ * conditional probability P(A|B) or a combination of AND/OR clauses.
+ *
+ * Note: This is not a simple bitmap in the sense that there are more
+ * than two possible values for each item - no match, partial
+ * match and full match. So we need 2 bits per item.
+ *
+ * TODO This works with 'bitmap' where each item is represented as a
+ * char, which is slightly wasteful. Instead, we could use a bitmap
+ * with 2 bits per item, reducing the size to ~1/4. By using values
+ * 0, 1 and 3 (instead of 0, 1 and 2), the operations (merging etc.)
+ * might be performed just like for simple bitmap by using & and |,
+ * which might be faster than min/max.
+ */
+static int
+update_match_bitmap_histogram(PlannerInfo *root, List *clauses,
+ int2vector *stakeys,
+ MVSerializedHistogram mvhist,
+ int nmatches, char * matches,
+ bool is_or)
+{
+ int i;
+ ListCell * l;
+
+ /*
+ * Used for caching function calls, only once per deduplicated value.
+ *
+ * We know may have up to (2 * nbuckets) values per dimension. It's
+ * probably overkill, but let's allocate that once for all clauses,
+ * to minimize overhead.
+ *
+ * Also, we only need two bits per value, but this allocates byte
+ * per value. Might be worth optimizing.
+ *
+ * 0x00 - not yet called
+ * 0x01 - called, result is 'false'
+ * 0x03 - called, result is 'true'
+ */
+ char *callcache = palloc(mvhist->nbuckets);
+
+ Assert(mvhist != NULL);
+ Assert(mvhist->nbuckets > 0);
+ Assert(nmatches >= 0);
+ Assert(nmatches <= mvhist->nbuckets);
+
+ Assert(clauses != NIL);
+ Assert(list_length(clauses) >= 1);
+
+ /* loop through the clauses and do the estimation */
+ foreach (l, clauses)
+ {
+ Node * clause = (Node*)lfirst(l);
+
+ /* if it's a RestrictInfo, then extract the clause */
+ if (IsA(clause, RestrictInfo))
+ clause = (Node*)((RestrictInfo*)clause)->clause;
+
+ /* it's either OpClause, or NullTest */
+ if (is_opclause(clause))
+ {
+ OpExpr * expr = (OpExpr*)clause;
+ bool varonleft = true;
+ bool ok;
+
+ FmgrInfo opproc; /* operator */
+ fmgr_info(get_opcode(expr->opno), &opproc);
+
+ /* reset the cache (per clause) */
+ memset(callcache, 0, mvhist->nbuckets);
+
+ ok = (NumRelids(clause) == 1) &&
+ (is_pseudo_constant_clause(lsecond(expr->args)) ||
+ (varonleft = false,
+ is_pseudo_constant_clause(linitial(expr->args))));
+
+ if (ok)
+ {
+ FmgrInfo ltproc;
+ RegProcedure oprrest = get_oprrest(expr->opno);
+
+ Var * var = (varonleft) ? linitial(expr->args) : lsecond(expr->args);
+ Const * cst = (varonleft) ? lsecond(expr->args) : linitial(expr->args);
+ bool isgt = (! varonleft);
+
+ TypeCacheEntry *typecache
+ = lookup_type_cache(var->vartype, TYPECACHE_LT_OPR);
+
+ /* lookup dimension for the attribute */
+ int idx = mv_get_index(var->varattno, stakeys);
+
+ fmgr_info(get_opcode(typecache->lt_opr), <proc);
+
+ /*
+ * Check this for all buckets that still have "true" in the bitmap
+ *
+ * We already know the clauses use suitable operators (because that's
+ * how we filtered them).
+ */
+ for (i = 0; i < mvhist->nbuckets; i++)
+ {
+ char res = MVSTATS_MATCH_NONE;
+
+ MVSerializedBucket bucket = mvhist->buckets[i];
+
+ /* histogram boundaries */
+ Datum minval, maxval;
+ bool mininclude, maxinclude;
+ int minidx, maxidx;
+
+ /*
+ * For AND-lists, we can also mark NULL buckets as 'no match'
+ * (and then skip them). For OR-lists this is not possible.
+ */
+ if ((! is_or) && bucket->nullsonly[idx])
+ matches[i] = MVSTATS_MATCH_NONE;
+
+ /*
+ * Skip buckets that were already eliminated - this is impotant
+ * considering how we update the info (we only lower the match).
+ * We can't really do anything about the MATCH_PARTIAL buckets.
+ */
+ if ((! is_or) && (matches[i] == MVSTATS_MATCH_NONE))
+ continue;
+ else if (is_or && (matches[i] == MVSTATS_MATCH_FULL))
+ continue;
+
+ /* lookup the values and cache of function calls */
+ minidx = bucket->min[idx];
+ maxidx = bucket->max[idx];
+
+ minval = mvhist->values[idx][bucket->min[idx]];
+ maxval = mvhist->values[idx][bucket->max[idx]];
+
+ mininclude = bucket->min_inclusive[idx];
+ maxinclude = bucket->max_inclusive[idx];
+
+ /*
+ * TODO Maybe it's possible to add here a similar optimization
+ * as for the MCV lists:
+ *
+ * (nmatches == 0) && AND-list => all eliminated (FALSE)
+ * (nmatches == N) && OR-list => all eliminated (TRUE)
+ *
+ * But it's more complex because of the partial matches.
+ */
+
+ /*
+ * If it's not a "<" or ">" or "=" operator, just ignore the
+ * clause. Otherwise note the relid and attnum for the variable.
+ *
+ * TODO I'm really unsure the handling of 'isgt' flag (that is, clauses
+ * with reverse order of variable/constant) is correct. I wouldn't
+ * be surprised if there was some mixup. Using the lt/gt operators
+ * instead of messing with the opproc could make it simpler.
+ * It would however be using a different operator than the query,
+ * although it's not any shadier than using the selectivity function
+ * as is done currently.
+ */
+ switch (oprrest)
+ {
+ case F_SCALARLTSEL: /* Var < Const */
+ case F_SCALARGTSEL: /* Var > Const */
+
+ res = bucket_is_smaller_than_value(opproc, cst->constvalue,
+ minval, maxval,
+ minidx, maxidx,
+ mininclude, maxinclude,
+ callcache, isgt);
+ break;
+
+ case F_EQSEL:
+
+ /*
+ * We only check whether the value is within the bucket, using the
+ * lt operator, and we also check for equality with the boundaries.
+ */
+
+ res = bucket_contains_value(ltproc, cst->constvalue,
+ minval, maxval,
+ minidx, maxidx,
+ mininclude, maxinclude,
+ callcache);
+ break;
+ }
+
+ UPDATE_RESULT(matches[i], res, is_or);
+
+ }
+ }
+ }
+ else if (IsA(clause, NullTest))
+ {
+ NullTest * expr = (NullTest*)clause;
+ Var * var = (Var*)(expr->arg);
+
+ /* FIXME proper matching attribute to dimension */
+ int idx = mv_get_index(var->varattno, stakeys);
+
+ /*
+ * Walk through the buckets and evaluate the current clause. We can
+ * skip items that were already ruled out, and terminate if there are
+ * no remaining buckets that might possibly match.
+ */
+ for (i = 0; i < mvhist->nbuckets; i++)
+ {
+ MVSerializedBucket bucket = mvhist->buckets[i];
+
+ /*
+ * Skip buckets that were already eliminated - this is impotant
+ * considering how we update the info (we only lower the match)
+ */
+ if ((! is_or) && (matches[i] == MVSTATS_MATCH_NONE))
+ continue;
+ else if (is_or && (matches[i] == MVSTATS_MATCH_FULL))
+ continue;
+
+ /* if the clause mismatches the bucket, set it as MATCH_NONE */
+ if ((expr->nulltesttype == IS_NULL)
+ && (! bucket->nullsonly[idx]))
+ UPDATE_RESULT(matches[i], MVSTATS_MATCH_NONE, is_or);
+
+ else if ((expr->nulltesttype == IS_NOT_NULL) &&
+ (bucket->nullsonly[idx]))
+ UPDATE_RESULT(matches[i], MVSTATS_MATCH_NONE, is_or);
+ }
+ }
+ else if (or_clause(clause) || and_clause(clause))
+ {
+ /* AND/OR clause, with all clauses compatible with the selected MV stat */
+
+ int i;
+ BoolExpr *orclause = ((BoolExpr*)clause);
+ List *orclauses = orclause->args;
+
+ /* match/mismatch bitmap for each bucket */
+ int or_nmatches = 0;
+ char * or_matches = NULL;
+
+ Assert(orclauses != NIL);
+ Assert(list_length(orclauses) >= 2);
+
+ /* number of matching buckets */
+ or_nmatches = mvhist->nbuckets;
+
+ /* by default none of the buckets matches the clauses */
+ or_matches = palloc0(sizeof(char) * or_nmatches);
+
+ if (or_clause(clause))
+ {
+ /* OR clauses assume nothing matches, initially */
+ memset(or_matches, MVSTATS_MATCH_NONE, sizeof(char)*or_nmatches);
+ or_nmatches = 0;
+ }
+ else
+ {
+ /* AND clauses assume nothing matches, initially */
+ memset(or_matches, MVSTATS_MATCH_FULL, sizeof(char)*or_nmatches);
+ }
+
+ /* build the match bitmap for the OR-clauses */
+ or_nmatches = update_match_bitmap_histogram(root, orclauses,
+ stakeys, mvhist,
+ or_nmatches, or_matches, or_clause(clause));
+
+ /* merge the bitmap into the existing one*/
+ for (i = 0; i < mvhist->nbuckets; i++)
+ {
+ /*
+ * To AND-merge the bitmaps, a MIN() semantics is used.
+ * For OR-merge, use MAX().
+ *
+ * FIXME this does not decrease the number of matches
+ */
+ UPDATE_RESULT(matches[i], or_matches[i], is_or);
+ }
+
+ pfree(or_matches);
+
+ }
+ else
+ elog(ERROR, "unknown clause type: %d", clause->type);
+ }
+
+ /* free the call cache */
+ pfree(callcache);
+
+ return nmatches;
+}
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index 8394111..2519249 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -412,7 +412,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
mvstat = (Form_pg_mv_statistic) GETSTRUCT(htup);
/* unavailable stats are not interesting for the planner */
- if (mvstat->deps_built || mvstat->mcv_built)
+ if (mvstat->deps_built || mvstat->mcv_built || mvstat->hist_built)
{
info = makeNode(MVStatisticInfo);
@@ -422,10 +422,12 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
/* enabled statistics */
info->deps_enabled = mvstat->deps_enabled;
info->mcv_enabled = mvstat->mcv_enabled;
+ info->hist_enabled = mvstat->hist_enabled;
/* built/available statistics */
info->deps_built = mvstat->deps_built;
info->mcv_built = mvstat->mcv_built;
+ info->hist_built = mvstat->hist_built;
/* stakeys */
adatum = SysCacheGetAttr(MVSTATOID, htup,
diff --git a/src/backend/utils/mvstats/Makefile b/src/backend/utils/mvstats/Makefile
index f9bf10c..9dbb3b6 100644
--- a/src/backend/utils/mvstats/Makefile
+++ b/src/backend/utils/mvstats/Makefile
@@ -12,6 +12,6 @@ subdir = src/backend/utils/mvstats
top_builddir = ../../../..
include $(top_builddir)/src/Makefile.global
-OBJS = common.o dependencies.o mcv.o
+OBJS = common.o dependencies.o histogram.o mcv.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/utils/mvstats/README.histogram b/src/backend/utils/mvstats/README.histogram
new file mode 100644
index 0000000..a182fa3
--- /dev/null
+++ b/src/backend/utils/mvstats/README.histogram
@@ -0,0 +1,299 @@
+Multivariate histograms
+=======================
+
+Histograms on individual attributes consist of buckets represented by ranges,
+covering the domain of the attribute. That is, each bucket is a [min,max]
+interval, and contains all values in this range. The histogram is built in such
+a way that all buckets have about the same frequency.
+
+Multivariate histograms are an extension into n-dimensional space - the buckets
+are n-dimensional intervals (i.e. n-dimensional rectagles), covering the domain
+of the combination of attributes. That is, each bucket has a vector of lower
+and upper boundaries, denoted min[i] and max[i] (where i = 1..n).
+
+In addition to the boundaries, each bucket tracks additional info:
+
+ * frequency (fraction of tuples in the bucket)
+ * whether the boundaries are inclusive or exclusive
+ * whether the dimension contains only NULL values
+ * number of distinct values in each dimension (for building only)
+
+It's possible that in the future we'll multiple histogram types, with different
+features. We do however expect all the types to share the same representation
+(buckets as ranges) and only differ in how we build them.
+
+The current implementation builds non-overlapping buckets, that may not be true
+for some histogram types and the code should not rely on this assumption. There
+are interesting types of histograms (or algorithms) with overlapping buckets.
+
+When used on low-cardinality data, histograms usually perform considerably worse
+than MCV lists (which are a good fit for this kind of data). This is especially
+true on label-like values, where ordering of the values is mostly unrelated to
+meaning of the data, as proper ordering is crucial for histograms.
+
+On high-cardinality data the histograms are usually a better choice, because MCV
+lists can't represent the distribution accurately enough.
+
+
+Selectivity estimation
+----------------------
+
+The estimation is implemented in clauselist_mv_selectivity_histogram(), and
+works very similarly to clauselist_mv_selectivity_mcvlist.
+
+The main difference is that while MCV lists support exact matches, histograms
+often result in approximate matches - e.g. with equality we can only say if
+the constant would be part of the bucket, but not whether it really is there
+or what fraction of the bucket it corresponds to. In this case we rely on
+some defaults just like in the per-column histograms.
+
+The current implementation uses histograms to estimates those types of clauses
+(think of WHERE conditions):
+
+ (a) equality clauses WHERE (a = 1) AND (b = 2)
+ (b) inequality clauses WHERE (a < 1) AND (b >= 2)
+ (c) NULL clauses WHERE (a IS NULL) AND (b IS NOT NULL)
+ (d) OR-clauses WHERE (a = 1) OR (b = 2)
+
+Similarly to MCV lists, it's possible to add support for additional types of
+clauses, for example:
+
+ (e) multi-var clauses WHERE (a > b)
+
+and so on. These are tasks for the future, not yet implemented.
+
+
+When evaluating a clause on a bucket, we may get one of three results:
+
+ (a) FULL_MATCH - The bucket definitely matches the clause.
+
+ (b) PARTIAL_MATCH - The bucket matches the clause, but not necessarily all
+ the tuples it represents.
+
+ (c) NO_MATCH - The bucket definitely does not match the clause.
+
+This may be illustrated using a range [1, 5], which is essentially a 1-D bucket.
+With clause
+
+ WHERE (a < 10) => FULL_MATCH (all range values are below
+ 10, so the whole bucket matches)
+
+ WHERE (a < 3) => PARTIAL_MATCH (there may be values matching
+ the clause, but we don't know how many)
+
+ WHERE (a < 0) => NO_MATCH (the whole range is above 1, so
+ no values from the bucket can match)
+
+Some clauses may produce only some of those results - for example equality
+clauses may never produce FULL_MATCH as we always hit only part of the bucket
+(we can't match both boundaries at the same time). This results in less accurate
+estimates compared to MCV lists, where we can hit a MCV items exactly (there's
+no PARTIAL match in MCV).
+
+There are also clauses that may not produce any PARTIAL_MATCH results. A nice
+example of that is 'IS [NOT] NULL' clause, which either matches the bucket
+completely (FULL_MATCH) or not at all (NO_MATCH), thanks to how the NULL-buckets
+are constructed.
+
+Computing the total selectivity estimate is trivial - simply sum selectivities
+from all the FULL_MATCH and PARTIAL_MATCH buckets (but for buckets marked with
+PARTIAL_MATCH, multiply the frequency by 0.5 to minimize the average error).
+
+
+Building a histogram
+---------------------
+
+The algorithm of building a histogram in general is quite simple:
+
+ (a) create an initial bucket (containing all sample rows)
+
+ (b) create NULL buckets (by splitting the initial bucket)
+
+ (c) repeat
+
+ (1) choose bucket to split next
+
+ (2) terminate if no bucket that might be split found, or if we've
+ reached the maximum number of buckets (16384)
+
+ (3) choose dimension to partition the bucket by
+
+ (4) partition the bucket by the selected dimension
+
+The main complexity is hidden in steps (c.1) and (c.3), i.e. how we choose the
+bucket and dimension for the split, as discussed in the next section.
+
+
+Partitioning criteria
+---------------------
+
+Similarly to one-dimensional histograms, we want to produce buckets with roughly
+the same frequency.
+
+We also need to produce "regular" buckets, because buckets with one dimension
+much longer than the others are very likely to match a lot of conditions (which
+increases error, even if the bucket frequency is very low).
+
+This is especially important when handling OR-clauses, because in that case each
+clause may add buckets independently. With AND-clauses all the clauses have to
+match each bucket, which makes this issue somewhat less concenrning.
+
+To achieve this, we choose the largest bucket (containing the most sample rows),
+but we only choose buckets that can actually be split (have at least 3 different
+combinations of values).
+
+Then we choose the "longest" dimension of the bucket, which is computed by using
+the distinct values in the sample as a measure.
+
+For details see functions select_bucket_to_partition() and partition_bucket(),
+which also includes further discussion.
+
+
+The current limit on number of buckets (16384) is mostly arbitrary, but chosen
+so that it guarantees we don't exceed the number of distinct values indexable by
+uint16 in any of the dimensions. In practice we could handle more buckets as we
+index each dimension separately and the splits should use the dimensions evenly.
+
+Also, histograms this large (with 16k values in multiple dimensions) would be
+quite expensive to build and process, so the 16k limit is rather reasonable.
+
+The actual number of buckets is also related to statistics target, because we
+require MIN_BUCKET_ROWS (10) tuples per bucket before a split, so we can't have
+more than (2 * 300 * target / 10) buckets. For the default target (100) this
+evaluates to ~6k.
+
+
+NULL handling (create_null_buckets)
+-----------------------------------
+
+When building histograms on a single attribute, we first filter out NULL values.
+In the multivariate case, we can't really do that because the rows may contain
+a mix of NULL and non-NULL values in different columns (so we can't simply
+filter all of them out).
+
+For this reason, the histograms are built in a way so that for each bucket, each
+dimension only contains only NULL or non-NULL values. Building the NULL-buckets
+happens as the first step in the build, by the create_null_buckets() function.
+The number of NULL buckets, as produced by this function, has a clear upper
+boundary (2^N) where N is the number of dimensions (attributes the histogram is
+built on). Or rather 2^K where K is the number of attributes that are not marked
+as not-NULL.
+
+The buckets with NULL dimensions are then subject to the same build algorithm
+(i.e. may be split into smaller buckets) just like any other bucket, but may
+only be split by non-NULL dimension.
+
+
+Serialization
+-------------
+
+To store the histogram in pg_mv_statistic table, it is serialized into a more
+efficient form. We also use the representation for estimation, i.e. we don't
+fully deserialize the histogram.
+
+For example the boundary values are deduplicated to minimize the required space.
+How much redundancy is there, actually? Let's assume there are no NULL values,
+so we start with a single bucket - in that case we have 2*N boundaries. Each
+time we split a bucket we introduce one new value (in the "middle" of one of
+the dimensions), and keep boundries for all the other dimensions. So after K
+splits, we have up to
+
+ 2*N + K
+
+unique boundary values (we may have fewe values, if the same value is used for
+several splits). But after K splits we do have (K+1) buckets, so
+
+ (K+1) * 2 * N
+
+boundary values. Using e.g. N=4 and K=999, we arrive to those numbers:
+
+ 2*N + K = 1007
+ (K+1) * 2 * N = 8000
+
+wich means a lot of redundancy. It's somewhat counter-intuitive that the number
+of distinct values does not really depend on the number of dimensions (except
+for the initial bucket, but that's negligible compared to the total).
+
+By deduplicating the values and replacing them with 16-bit indexes (uint16), we
+reduce the required space to
+
+ 1007 * 8 + 8000 * 2 ~= 24kB
+
+which is significantly less than 64kB required for the 'raw' histogram (assuming
+the values are 8B).
+
+While the bytea compression (pglz) might achieve the same reduction of space,
+the deduplicated representation is used to optimize the estimation by caching
+results of function calls for already visited values. This significantly
+reduces the number of calls to (often quite expensive) operators.
+
+Note: Of course, this reasoning only holds for histograms built by the algorithm
+that simply splits the buckets in half. Other histograms types (e.g. containing
+overlapping buckets) may behave differently and require different serialization.
+
+Serialized histograms are marked with 'magic' constant, to make it easier to
+check the bytea value really is a serialized histogram.
+
+
+varlena compression
+-------------------
+
+This serialization may however disable automatic varlena compression, the array
+of unique values is placed at the beginning of the serialized form. Which is
+exactly the chunk used by pglz to check if the data is compressible, and it
+will probably decide it's not very compressible. This is similar to the issue
+we had with JSONB initially.
+
+Maybe storing buckets first would make it work, as the buckets may be better
+compressible.
+
+On the other hand the serialization is actually a context-aware compression,
+usually compressing to ~30% (or even less, with large data types). So the lack
+of additional pglz compression may be acceptable.
+
+
+Deserialization
+---------------
+
+The deserialization is not a perfect inverse of the serialization, as we keep
+the deduplicated arrays. This reduces the amount of memory and also allows
+optimizations during estimation (e.g. we can cache results for the distinct
+values, saving expensive function calls).
+
+
+Inspecting the histogram
+------------------------
+
+Inspecting the regular (per-attribute) histograms is trivial, as it's enough
+to select the columns from pg_stats - the data is encoded as anyarray, so we
+simply get the text representation of the array.
+
+With multivariate histograms it's not that simple due to the possible mix of
+data types in the histogram. It might be possible to produce similar array-like
+text representation, but that'd unnecessarily complicate further processing
+and analysis of the histogram. Instead, there's a SRF function that allows
+access to lower/upper boundaries, frequencies etc.
+
+ SELECT * FROM pg_mv_histogram_buckets();
+
+It has two input parameters:
+
+ oid - OID of the histogram (pg_mv_statistic.staoid)
+ otype - type of output
+
+and produces a table with these columns:
+
+ - bucket ID (0...nbuckets-1)
+ - lower bucket boundaries (string array)
+ - upper bucket boundaries (string array)
+ - nulls only dimensions (boolean array)
+ - lower boundary inclusive (boolean array)
+ - upper boundary includive (boolean array)
+ - frequency (double precision)
+
+The 'otype' accepts three values, determining what will be returned in the
+lower/upper boundary arrays:
+
+ - 0 - values stored in the histogram, encoded as text
+ - 1 - indexes into the deduplicated arrays
+ - 2 - idnexes into the deduplicated arrays, scaled to [0,1]
diff --git a/src/backend/utils/mvstats/README.stats b/src/backend/utils/mvstats/README.stats
index 5c5c59a..3e4f4d1 100644
--- a/src/backend/utils/mvstats/README.stats
+++ b/src/backend/utils/mvstats/README.stats
@@ -18,6 +18,8 @@ Currently we only have two kinds of multivariate statistics
(b) MCV lists (README.mcv)
+ (c) multivariate histograms (README.histogram)
+
Compatible clause types
-----------------------
diff --git a/src/backend/utils/mvstats/common.c b/src/backend/utils/mvstats/common.c
index 4f5a842..f6d1074 100644
--- a/src/backend/utils/mvstats/common.c
+++ b/src/backend/utils/mvstats/common.c
@@ -13,11 +13,11 @@
*
*-------------------------------------------------------------------------
*/
+#include "postgres.h"
+#include "utils/array.h"
#include "common.h"
-#include "utils/array.h"
-
static VacAttrStats ** lookup_var_attr_stats(int2vector *attrs,
int natts,
VacAttrStats **vacattrstats);
@@ -52,7 +52,8 @@ build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
MVStatisticInfo *stat = (MVStatisticInfo *)lfirst(lc);
MVDependencies deps = NULL;
MCVList mcvlist = NULL;
- int numrows_filtered = 0;
+ MVHistogram histogram = NULL;
+ int numrows_filtered = numrows;
VacAttrStats **stats = NULL;
int numatts = 0;
@@ -95,8 +96,12 @@ build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
if (stat->mcv_enabled)
mcvlist = build_mv_mcvlist(numrows, rows, attrs, stats, &numrows_filtered);
+ /* build a multivariate histogram on the columns */
+ if ((numrows_filtered > 0) && (stat->hist_enabled))
+ histogram = build_mv_histogram(numrows_filtered, rows, attrs, stats, numrows);
+
/* store the histogram / MCV list in the catalog */
- update_mv_stats(stat->mvoid, deps, mcvlist, attrs, stats);
+ update_mv_stats(stat->mvoid, deps, mcvlist, histogram, attrs, stats);
}
}
@@ -176,6 +181,8 @@ list_mv_stats(Oid relid)
info->deps_built = stats->deps_built;
info->mcv_enabled = stats->mcv_enabled;
info->mcv_built = stats->mcv_built;
+ info->hist_enabled = stats->hist_enabled;
+ info->hist_built = stats->hist_built;
result = lappend(result, info);
}
@@ -190,7 +197,6 @@ list_mv_stats(Oid relid)
return result;
}
-
/*
* Find attnims of MV stats using the mvoid.
*/
@@ -236,9 +242,16 @@ find_mv_attnums(Oid mvoid, Oid *relid)
}
+/*
+ * FIXME This adds statistics, but we need to drop statistics when the
+ * table is dropped. Not sure what to do when a column is dropped.
+ * Either we can (a) remove all stats on that column, (b) remove
+ * the column from defined stats and force rebuild, (c) remove the
+ * column on next ANALYZE. Or maybe something else?
+ */
void
update_mv_stats(Oid mvoid,
- MVDependencies dependencies, MCVList mcvlist,
+ MVDependencies dependencies, MCVList mcvlist, MVHistogram histogram,
int2vector *attrs, VacAttrStats **stats)
{
HeapTuple stup,
@@ -271,22 +284,34 @@ update_mv_stats(Oid mvoid,
values[Anum_pg_mv_statistic_stamcv - 1] = PointerGetDatum(data);
}
+ if (histogram != NULL)
+ {
+ bytea * data = serialize_mv_histogram(histogram, attrs, stats);
+ nulls[Anum_pg_mv_statistic_stahist-1] = (data == NULL);
+ values[Anum_pg_mv_statistic_stahist - 1]
+ = PointerGetDatum(data);
+ }
+
/* always replace the value (either by bytea or NULL) */
replaces[Anum_pg_mv_statistic_stadeps -1] = true;
replaces[Anum_pg_mv_statistic_stamcv -1] = true;
+ replaces[Anum_pg_mv_statistic_stahist-1] = true;
/* always change the availability flags */
nulls[Anum_pg_mv_statistic_deps_built -1] = false;
nulls[Anum_pg_mv_statistic_mcv_built -1] = false;
+ nulls[Anum_pg_mv_statistic_hist_built-1] = false;
nulls[Anum_pg_mv_statistic_stakeys-1] = false;
/* use the new attnums, in case we removed some dropped ones */
replaces[Anum_pg_mv_statistic_deps_built-1] = true;
replaces[Anum_pg_mv_statistic_mcv_built -1] = true;
+ replaces[Anum_pg_mv_statistic_hist_built -1] = true;
replaces[Anum_pg_mv_statistic_stakeys -1] = true;
values[Anum_pg_mv_statistic_deps_built-1] = BoolGetDatum(dependencies != NULL);
values[Anum_pg_mv_statistic_mcv_built -1] = BoolGetDatum(mcvlist != NULL);
+ values[Anum_pg_mv_statistic_hist_built -1] = BoolGetDatum(histogram != NULL);
values[Anum_pg_mv_statistic_stakeys -1] = PointerGetDatum(attrs);
/* Is there already a pg_mv_statistic tuple for this attribute? */
diff --git a/src/backend/utils/mvstats/histogram.c b/src/backend/utils/mvstats/histogram.c
new file mode 100644
index 0000000..4bf7ec6
--- /dev/null
+++ b/src/backend/utils/mvstats/histogram.c
@@ -0,0 +1,2023 @@
+/*-------------------------------------------------------------------------
+ *
+ * histogram.c
+ * POSTGRES multivariate histograms
+ *
+ *
+ * Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/utils/mvstats/histogram.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "fmgr.h"
+#include "funcapi.h"
+
+#include "utils/lsyscache.h"
+
+#include "common.h"
+#include <math.h>
+
+
+static MVBucket create_initial_mv_bucket(int numrows, HeapTuple *rows,
+ int2vector *attrs,
+ VacAttrStats **stats);
+
+static MVBucket select_bucket_to_partition(int nbuckets, MVBucket * buckets);
+
+static MVBucket partition_bucket(MVBucket bucket, int2vector *attrs,
+ VacAttrStats **stats,
+ int *ndistvalues, Datum **distvalues);
+
+static MVBucket copy_mv_bucket(MVBucket bucket, uint32 ndimensions);
+
+static void update_bucket_ndistinct(MVBucket bucket, int2vector *attrs,
+ VacAttrStats ** stats);
+
+static void update_dimension_ndistinct(MVBucket bucket, int dimension,
+ int2vector *attrs,
+ VacAttrStats ** stats,
+ bool update_boundaries);
+
+static void create_null_buckets(MVHistogram histogram, int bucket_idx,
+ int2vector *attrs, VacAttrStats ** stats);
+
+static Datum * build_ndistinct(int numrows, HeapTuple *rows, int2vector *attrs,
+ VacAttrStats **stats, int i, int *nvals);
+
+/*
+ * Each serialized bucket needs to store (in this order):
+ *
+ * - number of tuples (float)
+ * - number of distinct (float)
+ * - min inclusive flags (ndim * sizeof(bool))
+ * - max inclusive flags (ndim * sizeof(bool))
+ * - null dimension flags (ndim * sizeof(bool))
+ * - min boundary indexes (2 * ndim * sizeof(uint16))
+ * - max boundary indexes (2 * ndim * sizeof(uint16))
+ *
+ * So in total:
+ *
+ * ndim * (4 * sizeof(uint16) + 3 * sizeof(bool)) + (2 * sizeof(float))
+ */
+#define BUCKET_SIZE(ndims) \
+ (ndims * (4 * sizeof(uint16) + 3 * sizeof(bool)) + sizeof(float))
+
+/* pointers into a flat serialized bucket of BUCKET_SIZE(n) bytes */
+#define BUCKET_NTUPLES(b) (*(float*)b)
+#define BUCKET_MIN_INCL(b,n) ((bool*)(b + sizeof(float)))
+#define BUCKET_MAX_INCL(b,n) (BUCKET_MIN_INCL(b,n) + n)
+#define BUCKET_NULLS_ONLY(b,n) (BUCKET_MAX_INCL(b,n) + n)
+#define BUCKET_MIN_INDEXES(b,n) ((uint16*)(BUCKET_NULLS_ONLY(b,n) + n))
+#define BUCKET_MAX_INDEXES(b,n) ((BUCKET_MIN_INDEXES(b,n) + n))
+
+/* can't split bucket with less than 10 rows */
+#define MIN_BUCKET_ROWS 10
+
+/*
+ * Data used while building the histogram.
+ */
+typedef struct HistogramBuildData {
+
+ float ndistinct; /* frequency of distinct values */
+
+ HeapTuple *rows; /* aray of sample rows */
+ uint32 numrows; /* number of sample rows (array size) */
+
+ /*
+ * Number of distinct values in each dimension. This is used when
+ * building the histogram (and is not serialized/deserialized).
+ */
+ uint32 *ndistincts;
+
+} HistogramBuildData;
+
+typedef HistogramBuildData *HistogramBuild;
+
+/*
+ * builds a multivariate algorithm
+ *
+ * The build algorithm is iterative - initially a single bucket containing all
+ * the sample rows is formed, and then repeatedly split into smaller buckets.
+ * In each step the largest bucket (in some sense) is chosen to be split next.
+ *
+ * The criteria for selecting the largest bucket (and the dimension for the
+ * split) needs to be elaborate enough to produce buckets of roughly the same
+ * size, and also regular shape (not very long in one dimension).
+ *
+ * The current algorithm works like this:
+ *
+ * build NULL-buckets (create_null_buckets)
+ *
+ * while [maximum number of buckets not reached]
+ *
+ * choose bucket to partition (largest bucket)
+ * if no bucket to partition
+ * terminate the algorithm
+ *
+ * choose bucket dimension to partition (largest dimension)
+ * split the bucket into two buckets
+ *
+ * See the discussion at select_bucket_to_partition and partition_bucket for
+ * more details about the algorithm.
+ */
+MVHistogram
+build_mv_histogram(int numrows, HeapTuple *rows, int2vector *attrs,
+ VacAttrStats **stats, int numrows_total)
+{
+ int i;
+ int numattrs = attrs->dim1;
+
+ int *ndistvalues;
+ Datum **distvalues;
+
+ MVHistogram histogram;
+
+ HeapTuple * rows_copy = (HeapTuple*)palloc0(numrows * sizeof(HeapTuple));
+ memcpy(rows_copy, rows, sizeof(HeapTuple) * numrows);
+
+ Assert((numattrs >= 2) && (numattrs <= MVSTATS_MAX_DIMENSIONS));
+
+ /* build histogram header */
+
+ histogram = (MVHistogram)palloc0(sizeof(MVHistogramData));
+
+ histogram->magic = MVSTAT_HIST_MAGIC;
+ histogram->type = MVSTAT_HIST_TYPE_BASIC;
+
+ histogram->nbuckets = 1;
+ histogram->ndimensions = numattrs;
+
+ /* create max buckets (better than repalloc for short-lived objects) */
+ histogram->buckets
+ = (MVBucket*)palloc0(MVSTAT_HIST_MAX_BUCKETS * sizeof(MVBucket));
+
+ /* create the initial bucket, covering the whole sample set */
+ histogram->buckets[0]
+ = create_initial_mv_bucket(numrows, rows_copy, attrs, stats);
+
+ /*
+ * Collect info on distinct values in each dimension (used later to select
+ * dimension to partition).
+ */
+ ndistvalues = (int*)palloc0(sizeof(int) * numattrs);
+ distvalues = (Datum**)palloc0(sizeof(Datum*) * numattrs);
+
+ for (i = 0; i < numattrs; i++)
+ distvalues[i] = build_ndistinct(numrows, rows, attrs, stats, i,
+ &ndistvalues[i]);
+
+ /*
+ * Split the initial bucket into buckets that don't mix NULL and non-NULL
+ * values in a single dimension.
+ */
+ create_null_buckets(histogram, 0, attrs, stats);
+
+ /*
+ * Do the actual histogram build - select a bucket and split it.
+ *
+ * FIXME This should use the max_buckets specified in CREATE STATISTICS.
+ */
+ while (histogram->nbuckets < MVSTAT_HIST_MAX_BUCKETS)
+ {
+ MVBucket bucket = select_bucket_to_partition(histogram->nbuckets,
+ histogram->buckets);
+
+ /* no buckets eligible for partitioning */
+ if (bucket == NULL)
+ break;
+
+ /* we modify the bucket in-place and add one new bucket */
+ histogram->buckets[histogram->nbuckets++]
+ = partition_bucket(bucket, attrs, stats, ndistvalues, distvalues);
+ }
+
+ /* finalize the histogram build - compute the frequencies etc. */
+ for (i = 0; i < histogram->nbuckets; i++)
+ {
+ HistogramBuild build_data
+ = ((HistogramBuild)histogram->buckets[i]->build_data);
+
+ /*
+ * The frequency has to be computed from the whole sample, in case some
+ * of the rows were used for MCV.
+ *
+ * XXX Perhaps this should simply compute frequency with respect to the
+ * local freuquency, and then factor-in the MCV later.
+ *
+ * FIXME The 'ntuples' sounds a bit inappropriate for frequency.
+ */
+ histogram->buckets[i]->ntuples
+ = (build_data->numrows * 1.0) / numrows_total;
+ }
+
+ return histogram;
+}
+
+/* build array of distinct values for a single attribute */
+static Datum *
+build_ndistinct(int numrows, HeapTuple *rows, int2vector *attrs,
+ VacAttrStats **stats, int i, int *nvals)
+{
+ int j;
+ int nvalues,
+ ndistinct;
+ Datum *values,
+ *distvalues;
+
+ SortSupportData ssup;
+ StdAnalyzeData *mystats = (StdAnalyzeData *) stats[i]->extra_data;
+
+ /* initialize sort support, etc. */
+ memset(&ssup, 0, sizeof(ssup));
+ ssup.ssup_cxt = CurrentMemoryContext;
+
+ /* We always use the default collation for statistics */
+ ssup.ssup_collation = DEFAULT_COLLATION_OID;
+ ssup.ssup_nulls_first = false;
+
+ PrepareSortSupportFromOrderingOp(mystats->ltopr, &ssup);
+
+ nvalues = 0;
+ values = (Datum*)palloc0(sizeof(Datum) * numrows);
+
+ /* collect values from the sample rows, ignore NULLs */
+ for (j = 0; j < numrows; j++)
+ {
+ Datum value;
+ bool isnull;
+
+ /* remember the index of the sample row, to make the partitioning simpler */
+ value = heap_getattr(rows[j], attrs->values[i],
+ stats[i]->tupDesc, &isnull);
+
+ if (isnull)
+ continue;
+
+ values[nvalues++] = value;
+ }
+
+ /* if no non-NULL values were found, free the memory and terminate */
+ if (nvalues == 0)
+ {
+ pfree(values);
+ return NULL;
+ }
+
+ /* sort the array of values using the SortSupport */
+ qsort_arg((void *) values, nvalues, sizeof(Datum),
+ compare_scalars_simple, (void *) &ssup);
+
+ /* count the distinct values first, and allocate just enough memory */
+ ndistinct = 1;
+ for (j = 1; j < nvalues; j++)
+ if (compare_scalars_simple(&values[j], &values[j-1], &ssup) != 0)
+ ndistinct += 1;
+
+ distvalues = (Datum*)palloc0(sizeof(Datum) * ndistinct);
+
+ /* now collect distinct values into the array */
+ distvalues[0] = values[0];
+ ndistinct = 1;
+
+ for (j = 1; j < nvalues; j++)
+ {
+ if (compare_scalars_simple(&values[j], &values[j-1], &ssup) != 0)
+ {
+ distvalues[ndistinct] = values[j];
+ ndistinct += 1;
+ }
+ }
+
+ pfree(values);
+
+ *nvals = ndistinct;
+ return distvalues;
+}
+
+/* fetch the histogram (as a bytea) from the pg_mv_statistic catalog */
+MVSerializedHistogram
+load_mv_histogram(Oid mvoid)
+{
+ bool isnull = false;
+ Datum histogram;
+
+#ifdef USE_ASSERT_CHECKING
+ Form_pg_mv_statistic mvstat;
+#endif
+
+ /* Prepare to scan pg_mv_statistic for entries having indrelid = this rel. */
+ HeapTuple htup = SearchSysCache1(MVSTATOID, ObjectIdGetDatum(mvoid));
+
+ if (! HeapTupleIsValid(htup))
+ return NULL;
+
+#ifdef USE_ASSERT_CHECKING
+ mvstat = (Form_pg_mv_statistic) GETSTRUCT(htup);
+ Assert(mvstat->hist_enabled && mvstat->hist_built);
+#endif
+
+ histogram = SysCacheGetAttr(MVSTATOID, htup,
+ Anum_pg_mv_statistic_stahist, &isnull);
+
+ Assert(!isnull);
+
+ ReleaseSysCache(htup);
+
+ return deserialize_mv_histogram(DatumGetByteaP(histogram));
+}
+
+/* print some basic info about the histogram */
+Datum
+pg_mv_stats_histogram_info(PG_FUNCTION_ARGS)
+{
+ bytea *data = PG_GETARG_BYTEA_P(0);
+ char *result;
+
+ MVSerializedHistogram hist = deserialize_mv_histogram(data);
+
+ result = palloc0(128);
+ snprintf(result, 128, "nbuckets=%d", hist->nbuckets);
+
+ PG_RETURN_TEXT_P(cstring_to_text(result));
+}
+
+/*
+ * Serialize the MV histogram into a bytea value. The basic algorithm is quite
+ * simple, and mostly mimincs the MCV serialization:
+ *
+ * (1) perform deduplication for each attribute (separately)
+ *
+ * (a) collect all (non-NULL) attribute values from all buckets
+ * (b) sort the data (using 'lt' from VacAttrStats)
+ * (c) remove duplicate values from the array
+ *
+ * (2) serialize the arrays into a bytea value
+ *
+ * (3) process all buckets
+ *
+ * (a) replace min/max values with indexes into the arrays
+ *
+ * Each attribute has to be processed separately, as we're mixing different
+ * datatypes, and we we need to use the right operators to compare/sort them.
+ * We're also mixing pass-by-value and pass-by-ref types, and so on.
+ *
+ *
+ * FIXME This probably leaks memory, or at least uses it inefficiently
+ * (many small palloc() calls instead of a large one).
+ *
+ * TODO Consider packing boolean flags (NULL) for each item into 'char'
+ * or a longer type (instead of using an array of bool items).
+ */
+bytea *
+serialize_mv_histogram(MVHistogram histogram, int2vector *attrs,
+ VacAttrStats **stats)
+{
+ int i = 0, j = 0;
+ Size total_length = 0;
+
+ bytea *output = NULL;
+ char *data = NULL;
+
+ DimensionInfo *info;
+ SortSupport ssup;
+
+ int nbuckets = histogram->nbuckets;
+ int ndims = histogram->ndimensions;
+
+ /* allocated for serialized bucket data */
+ int bucketsize = BUCKET_SIZE(ndims);
+ char *bucket = palloc0(bucketsize);
+
+ /* values per dimension (and number of non-NULL values) */
+ Datum **values = (Datum**)palloc0(sizeof(Datum*) * ndims);
+ int *counts = (int*)palloc0(sizeof(int) * ndims);
+
+ /* info about dimensions (for deserialize) */
+ info = (DimensionInfo *)palloc0(sizeof(DimensionInfo)*ndims);
+
+ /* sort support data */
+ ssup = (SortSupport)palloc0(sizeof(SortSupportData)*ndims);
+
+ /* collect and deduplicate values for each dimension separately */
+ for (i = 0; i < ndims; i++)
+ {
+ int count;
+ StdAnalyzeData *tmp = (StdAnalyzeData *)stats[i]->extra_data;
+
+ /* keep important info about the data type */
+ info[i].typlen = stats[i]->attrtype->typlen;
+ info[i].typbyval = stats[i]->attrtype->typbyval;
+
+ /*
+ * Allocate space for all min/max values, including NULLs (we won't use
+ * them, but we don't know how many are there), and then collect all
+ * non-NULL values.
+ */
+ values[i] = (Datum*)palloc0(sizeof(Datum) * nbuckets * 2);
+
+ for (j = 0; j < histogram->nbuckets; j++)
+ {
+ /* skip buckets where this dimension is NULL-only */
+ if (! histogram->buckets[j]->nullsonly[i])
+ {
+ values[i][counts[i]] = histogram->buckets[j]->min[i];
+ counts[i] += 1;
+
+ values[i][counts[i]] = histogram->buckets[j]->max[i];
+ counts[i] += 1;
+ }
+ }
+
+ /* there are just NULL values in this dimension */
+ if (counts[i] == 0)
+ continue;
+
+ /* sort and deduplicate */
+ ssup[i].ssup_cxt = CurrentMemoryContext;
+ ssup[i].ssup_collation = DEFAULT_COLLATION_OID;
+ ssup[i].ssup_nulls_first = false;
+
+ PrepareSortSupportFromOrderingOp(tmp->ltopr, &ssup[i]);
+
+ qsort_arg(values[i], counts[i], sizeof(Datum),
+ compare_scalars_simple, &ssup[i]);
+
+ /*
+ * Walk through the array and eliminate duplicitate values, but
+ * keep the ordering (so that we can do bsearch later). We know
+ * there's at least 1 item, so we can skip the first element.
+ */
+ count = 1; /* number of deduplicated items */
+ for (j = 1; j < counts[i]; j++)
+ {
+ /* if it's different from the previous value, we need to keep it */
+ if (compare_datums_simple(values[i][j-1], values[i][j], &ssup[i]) != 0)
+ {
+ /* XXX: not needed if (count == j) */
+ values[i][count] = values[i][j];
+ count += 1;
+ }
+ }
+
+ /* make sure we fit into uint16 */
+ Assert(count <= UINT16_MAX);
+
+ /* keep info about the deduplicated count */
+ info[i].nvalues = count;
+
+ /* compute size of the serialized data */
+ if (info[i].typlen > 0)
+ /* byval or byref, but with fixed length (name, tid, ...) */
+ info[i].nbytes = info[i].nvalues * info[i].typlen;
+ else if (info[i].typlen == -1)
+ /* varlena, so just use VARSIZE_ANY */
+ for (j = 0; j < info[i].nvalues; j++)
+ info[i].nbytes += VARSIZE_ANY(values[i][j]);
+ else if (info[i].typlen == -2)
+ /* cstring, so simply strlen */
+ for (j = 0; j < info[i].nvalues; j++)
+ info[i].nbytes += strlen(DatumGetPointer(values[i][j]));
+ else
+ elog(ERROR, "unknown data type typbyval=%d typlen=%d",
+ info[i].typbyval, info[i].typlen);
+ }
+
+ /*
+ * Now we finally know how much space we'll need for the serialized
+ * histogram, as it contains these fields:
+ *
+ * - length (4B) for varlena
+ * - magic (4B)
+ * - type (4B)
+ * - ndimensions (4B)
+ * - nbuckets (4B)
+ * - info (ndim * sizeof(DimensionInfo)
+ * - arrays of values for each dimension
+ * - serialized buckets (nbuckets * bucketsize)
+ *
+ * So the 'header' size is 20B + ndim * sizeof(DimensionInfo) and
+ * then we'll place the data (and buckets).
+ */
+ total_length = (sizeof(int32) + offsetof(MVHistogramData, buckets)
+ + ndims * sizeof(DimensionInfo)
+ + nbuckets * bucketsize);
+
+ /* account for the deduplicated data */
+ for (i = 0; i < ndims; i++)
+ total_length += info[i].nbytes;
+
+ /* enforce arbitrary limit of 1MB */
+ if (total_length > (1024 * 1024))
+ elog(ERROR, "serialized histogram exceeds 1MB (%ld > %d)",
+ total_length, (1024 * 1024));
+
+ /* allocate space for the serialized histogram list, set header */
+ output = (bytea*)palloc0(total_length);
+ SET_VARSIZE(output, total_length);
+
+ /* we'll use 'data' to keep track of the place to write data */
+ data = VARDATA(output);
+
+ memcpy(data, histogram, offsetof(MVHistogramData, buckets));
+ data += offsetof(MVHistogramData, buckets);
+
+ memcpy(data, info, sizeof(DimensionInfo) * ndims);
+ data += sizeof(DimensionInfo) * ndims;
+
+ /* serialize the deduplicated values for all attributes */
+ for (i = 0; i < ndims; i++)
+ {
+#ifdef USE_ASSERT_CHECKING
+ char *tmp = data;
+#endif
+ for (j = 0; j < info[i].nvalues; j++)
+ {
+ Datum v = values[i][j];
+
+ if (info[i].typbyval) /* passed by value */
+ {
+ memcpy(data, &v, info[i].typlen);
+ data += info[i].typlen;
+ }
+ else if (info[i].typlen > 0) /* pased by reference */
+ {
+ memcpy(data, DatumGetPointer(v), info[i].typlen);
+ data += info[i].typlen;
+ }
+ else if (info[i].typlen == -1) /* varlena */
+ {
+ memcpy(data, DatumGetPointer(v), VARSIZE_ANY(v));
+ data += VARSIZE_ANY(values[i][j]);
+ }
+ else if (info[i].typlen == -2) /* cstring */
+ {
+ memcpy(data, DatumGetPointer(v), strlen(DatumGetPointer(v))+1);
+ data += strlen(DatumGetPointer(v)) + 1;
+ }
+ }
+
+ /* make sure we got exactly the amount of data we expected */
+ Assert((data - tmp) == info[i].nbytes);
+ }
+
+ /* finally serialize the items, with uint16 indexes instead of the values */
+ for (i = 0; i < nbuckets; i++)
+ {
+ /* don't write beyond the allocated space */
+ Assert(data <= (char*)output + total_length - bucketsize);
+
+ /* reset the values for each item */
+ memset(bucket, 0, bucketsize);
+
+ BUCKET_NTUPLES(bucket) = histogram->buckets[i]->ntuples;
+
+ for (j = 0; j < ndims; j++)
+ {
+ /* do the lookup only for non-NULL values */
+ if (! histogram->buckets[i]->nullsonly[j])
+ {
+ uint16 idx;
+ Datum * v = NULL;
+
+ /* min boundary */
+ v = (Datum*)bsearch_arg(&histogram->buckets[i]->min[j],
+ values[j], info[j].nvalues, sizeof(Datum),
+ compare_scalars_simple, &ssup[j]);
+
+ Assert(v != NULL); /* serialization or deduplication error */
+
+ /* compute index within the array */
+ idx = (v - values[j]);
+
+ Assert((idx >= 0) && (idx < info[j].nvalues));
+
+ BUCKET_MIN_INDEXES(bucket, ndims)[j] = idx;
+
+ /* max boundary */
+ v = (Datum*)bsearch_arg(&histogram->buckets[i]->max[j],
+ values[j], info[j].nvalues, sizeof(Datum),
+ compare_scalars_simple, &ssup[j]);
+
+ Assert(v != NULL); /* serialization or deduplication error */
+
+ /* compute index within the array */
+ idx = (v - values[j]);
+
+ Assert((idx >= 0) && (idx < info[j].nvalues));
+
+ BUCKET_MAX_INDEXES(bucket, ndims)[j] = idx;
+ }
+ }
+
+ /* copy flags (nulls, min/max inclusive) */
+ memcpy(BUCKET_NULLS_ONLY(bucket, ndims),
+ histogram->buckets[i]->nullsonly, sizeof(bool) * ndims);
+
+ memcpy(BUCKET_MIN_INCL(bucket, ndims),
+ histogram->buckets[i]->min_inclusive, sizeof(bool) * ndims);
+
+ memcpy(BUCKET_MAX_INCL(bucket, ndims),
+ histogram->buckets[i]->max_inclusive, sizeof(bool) * ndims);
+
+ /* copy the item into the array */
+ memcpy(data, bucket, bucketsize);
+
+ data += bucketsize;
+ }
+
+ /* at this point we expect to match the total_length exactly */
+ Assert((data - (char*)output) == total_length);
+
+ /* free the values/counts arrays here */
+ pfree(counts);
+ pfree(info);
+ pfree(ssup);
+
+ for (i = 0; i < ndims; i++)
+ pfree(values[i]);
+
+ pfree(values);
+
+ return output;
+}
+
+/*
+ * Returns histogram in a partially-serialized form (keeps the boundary values
+ * deduplicated, so that it's possible to optimize the estimation part by
+ * caching function call results between buckets etc.).
+ */
+MVSerializedHistogram
+deserialize_mv_histogram(bytea * data)
+{
+ int i = 0, j = 0;
+
+ Size expected_size;
+ char *tmp = NULL;
+
+ MVSerializedHistogram histogram;
+ DimensionInfo *info;
+
+ int nbuckets;
+ int ndims;
+ int bucketsize;
+
+ /* temporary deserialization buffer */
+ int bufflen;
+ char *buff;
+ char *ptr;
+
+ if (data == NULL)
+ return NULL;
+
+ if (VARSIZE_ANY_EXHDR(data) < offsetof(MVSerializedHistogramData,buckets))
+ elog(ERROR, "invalid histogram size %ld (expected at least %ld)",
+ VARSIZE_ANY_EXHDR(data), offsetof(MVSerializedHistogramData,buckets));
+
+ /* read the histogram header */
+ histogram
+ = (MVSerializedHistogram)palloc(sizeof(MVSerializedHistogramData));
+
+ /* initialize pointer to the data part (skip the varlena header) */
+ tmp = VARDATA(data);
+
+ /* get the header and perform basic sanity checks */
+ memcpy(histogram, tmp, offsetof(MVSerializedHistogramData, buckets));
+ tmp += offsetof(MVSerializedHistogramData, buckets);
+
+ if (histogram->magic != MVSTAT_HIST_MAGIC)
+ elog(ERROR, "invalid histogram magic %d (expected %dd)",
+ histogram->magic, MVSTAT_HIST_MAGIC);
+
+ if (histogram->type != MVSTAT_HIST_TYPE_BASIC)
+ elog(ERROR, "invalid histogram type %d (expected %dd)",
+ histogram->type, MVSTAT_HIST_TYPE_BASIC);
+
+ nbuckets = histogram->nbuckets;
+ ndims = histogram->ndimensions;
+ bucketsize = BUCKET_SIZE(ndims);
+
+ Assert((nbuckets > 0) && (nbuckets <= MVSTAT_HIST_MAX_BUCKETS));
+ Assert((ndims >= 2) && (ndims <= MVSTATS_MAX_DIMENSIONS));
+
+ /*
+ * What size do we expect with those parameters (it's incomplete, as we yet
+ * have to count the array sizes (from DimensionInfo records).
+ */
+ expected_size = offsetof(MVSerializedHistogramData,buckets) +
+ ndims * sizeof(DimensionInfo) +
+ (nbuckets * bucketsize);
+
+ /* check that we have at least the DimensionInfo records */
+ if (VARSIZE_ANY_EXHDR(data) < expected_size)
+ elog(ERROR, "invalid histogram size %ld (expected %ld)",
+ VARSIZE_ANY_EXHDR(data), expected_size);
+
+ info = (DimensionInfo*)(tmp);
+ tmp += ndims * sizeof(DimensionInfo);
+
+ /* account for the value arrays */
+ for (i = 0; i < ndims; i++)
+ expected_size += info[i].nbytes;
+
+ if (VARSIZE_ANY_EXHDR(data) != expected_size)
+ elog(ERROR, "invalid histogram size %ld (expected %ld)",
+ VARSIZE_ANY_EXHDR(data), expected_size);
+
+ /* looks OK - not corrupted or something */
+
+ /* a single buffer for all the values and counts */
+ bufflen = (sizeof(int) + sizeof(Datum*)) * ndims;
+
+ for (i = 0; i < ndims; i++)
+ /* don't allocate space for byval types, matching Datum */
+ if (! (info[i].typbyval && (info[i].typlen == sizeof(Datum))))
+ bufflen += (sizeof(Datum) * info[i].nvalues);
+
+ /* also, include space for the result, tracking the buckets */
+ bufflen += nbuckets * (
+ sizeof(MVSerializedBucket) + /* bucket pointer */
+ sizeof(MVSerializedBucketData)); /* bucket data */
+
+ buff = palloc0(bufflen);
+ ptr = buff;
+
+ histogram->nvalues = (int*)ptr;
+ ptr += (sizeof(int) * ndims);
+
+ histogram->values = (Datum**)ptr;
+ ptr += (sizeof(Datum*) * ndims);
+
+ /*
+ * FIXME This uses pointers to the original data array (the types
+ * not passed by value), so when someone frees the memory,
+ * e.g. by doing something like this:
+ *
+ * bytea * data = ... fetch the data from catalog ...
+ * MCVList mcvlist = deserialize_mcv_list(data);
+ * pfree(data);
+ *
+ * then 'mcvlist' references the freed memory. This needs to
+ * copy the pieces.
+ *
+ * TODO same as in MCV deserialization / consider moving to common.c
+ */
+ for (i = 0; i < ndims; i++)
+ {
+ histogram->nvalues[i] = info[i].nvalues;
+
+ if (info[i].typbyval)
+ {
+ /* passed by value / Datum - simply reuse the array */
+ if (info[i].typlen == sizeof(Datum))
+ {
+ histogram->values[i] = (Datum*)tmp;
+ tmp += info[i].nbytes;
+ }
+ else
+ {
+ histogram->values[i] = (Datum*)ptr;
+ ptr += (sizeof(Datum) * info[i].nvalues);
+
+ for (j = 0; j < info[i].nvalues; j++)
+ {
+ /* just point into the array */
+ memcpy(&histogram->values[i][j], tmp, info[i].typlen);
+ tmp += info[i].typlen;
+ }
+ }
+ }
+ else
+ {
+ /* all the other types need a chunk of the buffer */
+ histogram->values[i] = (Datum*)ptr;
+ ptr += (sizeof(Datum) * info[i].nvalues);
+
+ if (info[i].typlen > 0)
+ {
+ /* pased by reference, but fixed length (name, tid, ...) */
+ for (j = 0; j < info[i].nvalues; j++)
+ {
+ /* just point into the array */
+ histogram->values[i][j] = PointerGetDatum(tmp);
+ tmp += info[i].typlen;
+ }
+ }
+ else if (info[i].typlen == -1)
+ {
+ /* varlena */
+ for (j = 0; j < info[i].nvalues; j++)
+ {
+ /* just point into the array */
+ histogram->values[i][j] = PointerGetDatum(tmp);
+ tmp += VARSIZE_ANY(tmp);
+ }
+ }
+ else if (info[i].typlen == -2)
+ {
+ /* cstring */
+ for (j = 0; j < info[i].nvalues; j++)
+ {
+ /* just point into the array */
+ histogram->values[i][j] = PointerGetDatum(tmp);
+ tmp += (strlen(tmp) + 1); /* don't forget the \0 */
+ }
+ }
+ }
+ }
+
+ histogram->buckets = (MVSerializedBucket*)ptr;
+ ptr += (sizeof(MVSerializedBucket) * nbuckets);
+
+ for (i = 0; i < nbuckets; i++)
+ {
+ MVSerializedBucket bucket = (MVSerializedBucket)ptr;
+ ptr += sizeof(MVSerializedBucketData);
+
+ bucket->ntuples = BUCKET_NTUPLES(tmp);
+ bucket->nullsonly = BUCKET_NULLS_ONLY(tmp, ndims);
+ bucket->min_inclusive = BUCKET_MIN_INCL(tmp, ndims);
+ bucket->max_inclusive = BUCKET_MAX_INCL(tmp, ndims);
+
+ bucket->min = BUCKET_MIN_INDEXES(tmp, ndims);
+ bucket->max = BUCKET_MAX_INDEXES(tmp, ndims);
+
+ histogram->buckets[i] = bucket;
+
+ Assert(tmp <= (char*)data + VARSIZE_ANY(data));
+
+ tmp += bucketsize;
+ }
+
+ /* at this point we expect to match the total_length exactly */
+ Assert((tmp - VARDATA(data)) == expected_size);
+
+ /* we should exhaust the output buffer exactly */
+ Assert((ptr - buff) == bufflen);
+
+ return histogram;
+}
+
+/*
+ * Build the initial bucket, which will be then split into smaller ones.
+ */
+static MVBucket
+create_initial_mv_bucket(int numrows, HeapTuple *rows, int2vector *attrs,
+ VacAttrStats **stats)
+{
+ int i;
+ int numattrs = attrs->dim1;
+ HistogramBuild data = NULL;
+
+ /* TODO allocate bucket as a single piece, including all the fields. */
+ MVBucket bucket = (MVBucket)palloc0(sizeof(MVBucketData));
+
+ Assert(numrows > 0);
+ Assert(rows != NULL);
+ Assert((numattrs >= 2) && (numattrs <= MVSTATS_MAX_DIMENSIONS));
+
+ /* allocate the per-dimension arrays */
+
+ /* flags for null-only dimensions */
+ bucket->nullsonly = (bool*)palloc0(numattrs * sizeof(bool));
+
+ /* inclusiveness boundaries - lower/upper bounds */
+ bucket->min_inclusive = (bool*)palloc0(numattrs * sizeof(bool));
+ bucket->max_inclusive = (bool*)palloc0(numattrs * sizeof(bool));
+
+ /* lower/upper boundaries */
+ bucket->min = (Datum*)palloc0(numattrs * sizeof(Datum));
+ bucket->max = (Datum*)palloc0(numattrs * sizeof(Datum));
+
+ /* build-data */
+ data = (HistogramBuild)palloc0(sizeof(HistogramBuildData));
+
+ /* number of distinct values (per dimension) */
+ data->ndistincts = (uint32*)palloc0(numattrs * sizeof(uint32));
+
+ /* all the sample rows fall into the initial bucket */
+ data->numrows = numrows;
+ data->rows = rows;
+
+ bucket->build_data = data;
+
+ /*
+ * Update the number of ndistinct combinations in the bucket (which we use
+ * when selecting bucket to partition), and then number of distinct values
+ * for each partition (which we use when choosing which dimension to split).
+ */
+ update_bucket_ndistinct(bucket, attrs, stats);
+
+ /* Update ndistinct (and also set min/max) for all dimensions. */
+ for (i = 0; i < numattrs; i++)
+ update_dimension_ndistinct(bucket, i, attrs, stats, true);
+
+ return bucket;
+}
+
+/*
+ * Choose the bucket to partition next.
+ *
+ * The current criteria is rather simple, chosen so that the algorithm produces
+ * buckets with about equal frequency and regular size. We select the bucket
+ * with the highest number of distinct values, and then split it by the longest
+ * dimension.
+ *
+ * The distinct values are uniformly mapped to [0,1] interval, and this is used
+ * to compute length of the value range.
+ *
+ * NOTE: This is not the same array used for deduplication, as this contains
+ * values for all the tuples from the sample, not just the boundary values.
+ *
+ * Returns either pointer to the bucket selected to be partitioned, or NULL if
+ * there are no buckets that may be split (e.g. if all buckets are too small
+ * or contain too few distinct values).
+ *
+ *
+ * Tricky example
+ * --------------
+ *
+ * Consider this table:
+ *
+ * CREATE TABLE t AS SELECT i AS a, i AS b
+ * FROM generate_series(1,1000000) s(i);
+ *
+ * CREATE STATISTICS s1 ON t (a,b) WITH (histogram);
+ *
+ * ANALYZE t;
+ *
+ * It's a very specific (and perhaps artificial) example, because every bucket
+ * always has exactly the same number of distinct values in all dimensions,
+ * which makes the partitioning tricky.
+ *
+ * Then:
+ *
+ * SELECT * FROM t WHERE (a < 100) AND (b < 100);
+ *
+ * is estimated to return ~120 rows, while in reality it returns only 99.
+ *
+ * QUERY PLAN
+ * -------------------------------------------------------------
+ * Seq Scan on t (cost=0.00..19425.00 rows=117 width=8)
+ * (actual time=0.129..82.776 rows=99 loops=1)
+ * Filter: ((a < 100) AND (b < 100))
+ * Rows Removed by Filter: 999901
+ * Planning time: 1.286 ms
+ * Execution time: 82.984 ms
+ * (5 rows)
+ *
+ * So this estimate is reasonably close. Let's change the query to OR clause:
+ *
+ * SELECT * FROM t WHERE (a < 100) OR (b < 100);
+ *
+ * QUERY PLAN
+ * -------------------------------------------------------------
+ * Seq Scan on t (cost=0.00..19425.00 rows=8100 width=8)
+ * (actual time=0.145..99.910 rows=99 loops=1)
+ * Filter: ((a < 100) OR (b < 100))
+ * Rows Removed by Filter: 999901
+ * Planning time: 1.578 ms
+ * Execution time: 100.132 ms
+ * (5 rows)
+ *
+ * That's clearly a much worse estimate. This happens because the histogram
+ * contains buckets like this:
+ *
+ * bucket 592 [3 30310] [30134 30593] => [0.000233]
+ *
+ * i.e. the length of "a" dimension is (30310-3)=30307, while the length of "b"
+ * is (30593-30134)=459. So the "b" dimension is much narrower than "a".
+ * Of course, there are also buckets where "b" is the wider dimension.
+ *
+ * This is partially mitigated by selecting the "longest" dimension but that
+ * only happens after we already selected the bucket. So if we never select the
+ * bucket, this optimization does not apply.
+ *
+ * The other reason why this particular example behaves so poorly is due to the
+ * way we actually split the selected bucket. We do attempt to divide the bucket
+ * into two parts containing about the same number of tuples, but that does not
+ * too well when most of the tuples is squashed on one side of the bucket.
+ *
+ * For example for columns with data on the diagonal (i.e. when a=b), we end up
+ * with a narrow bucket on the diagonal and a huge bucket overing the remaining
+ * part (with much lower density).
+ *
+ * So perhaps we need two partitioning strategies - one aiming to split buckets
+ * with high frequency (number of sampled rows), the other aiming to split
+ * "large" buckets. And alternating between them, somehow.
+ *
+ * TODO Consider using similar lower boundary for row count as for simple
+ * histograms, i.e. 300 tuples per bucket.
+ */
+static MVBucket
+select_bucket_to_partition(int nbuckets, MVBucket * buckets)
+{
+ int i;
+ int numrows = 0;
+ MVBucket bucket = NULL;
+
+ for (i = 0; i < nbuckets; i++)
+ {
+ HistogramBuild data = (HistogramBuild)buckets[i]->build_data;
+
+ /* if the number of rows is higher, use this bucket */
+ if ((data->ndistinct > 2) &&
+ (data->numrows > numrows) &&
+ (data->numrows >= MIN_BUCKET_ROWS)) {
+ bucket = buckets[i];
+ numrows = data->numrows;
+ }
+ }
+
+ /* may be NULL if there are not buckets with (ndistinct>1) */
+ return bucket;
+}
+
+/*
+ * A simple bucket partitioning implementation - we choose the longest bucket
+ * dimension, measured using the array of distinct values built at the very
+ * beginning of the build.
+ *
+ * We map all the distinct values to a [0,1] interval, uniformly distributed,
+ * and then use this to measure length. It's essentially a number of distinct
+ * values within the range, normalized to [0,1].
+ *
+ * Then we choose a 'middle' value splitting the bucket into two parts with
+ * roughly the same frequency.
+ *
+ * This splits the bucket by tweaking the existing one, and returning the new
+ * bucket (essentially shrinking the existing one in-place and returning the
+ * other "half" as a new bucket). The caller is responsible for adding the new
+ * bucket into the list of buckets.
+ *
+ * There are multiple histogram options, centered around the partitioning
+ * criteria, specifying both how to choose a bucket and the dimension most in
+ * need of a split. For a nice summary and general overview, see "rK-Hist : an
+ * R-Tree based histogram for multi-dimensional selectivity estimation" thesis
+ * by J. A. Lopez, Concordia University, p.34-37 (and possibly p. 32-34 for
+ * explanation of the terms).
+ *
+ * It requires care to prevent splitting only one dimension and not splitting
+ * another one at all (which might happen easily in case of strongly dependent
+ * columns - e.g. y=x). The current algorithm minimizes this, but may still
+ * happen for perfectly dependent examples (when all the dimensions have equal
+ * length, the first one will be selected).
+ *
+ * TODO Should probably consider statistics target for the columns (e.g.
+ * to split dimensions with higher statistics target more frequently).
+ */
+static MVBucket
+partition_bucket(MVBucket bucket, int2vector *attrs,
+ VacAttrStats **stats,
+ int *ndistvalues, Datum **distvalues)
+{
+ int i;
+ int dimension;
+ int numattrs = attrs->dim1;
+
+ Datum split_value;
+ MVBucket new_bucket;
+ HistogramBuild new_data;
+
+ /* needed for sort, when looking for the split value */
+ bool isNull;
+ int nvalues = 0;
+ HistogramBuild data = (HistogramBuild)bucket->build_data;
+ StdAnalyzeData * mystats = NULL;
+ ScalarItem * values = (ScalarItem*)palloc0(data->numrows * sizeof(ScalarItem));
+ SortSupportData ssup;
+
+ int nrows = 1; /* number of rows below current value */
+ double delta;
+
+ /* needed when splitting the values */
+ HeapTuple * oldrows = data->rows;
+ int oldnrows = data->numrows;
+
+ /*
+ * We can't split buckets with a single distinct value (this also
+ * disqualifies NULL-only dimensions). Also, there has to be multiple
+ * sample rows (otherwise, how could there be more distinct values).
+ */
+ Assert(data->ndistinct > 1);
+ Assert(data->numrows > 1);
+ Assert((numattrs >= 2) && (numattrs <= MVSTATS_MAX_DIMENSIONS));
+
+ /* Look for the next dimension to split. */
+ delta = 0.0;
+ dimension = -1;
+
+ for (i = 0; i < numattrs; i++)
+ {
+ Datum *a, *b;
+
+ mystats = (StdAnalyzeData *) stats[i]->extra_data;
+
+ /* initialize sort support, etc. */
+ memset(&ssup, 0, sizeof(ssup));
+ ssup.ssup_cxt = CurrentMemoryContext;
+
+ /* We always use the default collation for statistics */
+ ssup.ssup_collation = DEFAULT_COLLATION_OID;
+ ssup.ssup_nulls_first = false;
+
+ PrepareSortSupportFromOrderingOp(mystats->ltopr, &ssup);
+
+ /* can't split NULL-only dimension */
+ if (bucket->nullsonly[i])
+ continue;
+
+ /* can't split dimension with a single ndistinct value */
+ if (data->ndistincts[i] <= 1)
+ continue;
+
+ /* search for min boundary in the distinct list */
+ a = (Datum*)bsearch_arg(&bucket->min[i],
+ distvalues[i], ndistvalues[i],
+ sizeof(Datum), compare_scalars_simple, &ssup);
+
+ b = (Datum*)bsearch_arg(&bucket->max[i],
+ distvalues[i], ndistvalues[i],
+ sizeof(Datum), compare_scalars_simple, &ssup);
+
+ /* if this dimension is 'larger' then partition by it */
+ if (((b-a)*1.0 / ndistvalues[i]) > delta)
+ {
+ delta = ((b-a)*1.0 / ndistvalues[i]);
+ dimension = i;
+ }
+ }
+
+ /*
+ * If we haven't found a dimension here, we've done something
+ * wrong in select_bucket_to_partition.
+ */
+ Assert(dimension != -1);
+
+ /*
+ * Walk through the selected dimension, collect and sort the values and
+ * then choose the value to use as the new boundary.
+ */
+ mystats = (StdAnalyzeData *) stats[dimension]->extra_data;
+
+ /* initialize sort support, etc. */
+ memset(&ssup, 0, sizeof(ssup));
+ ssup.ssup_cxt = CurrentMemoryContext;
+
+ /* We always use the default collation for statistics */
+ ssup.ssup_collation = DEFAULT_COLLATION_OID;
+ ssup.ssup_nulls_first = false;
+
+ PrepareSortSupportFromOrderingOp(mystats->ltopr, &ssup);
+
+ for (i = 0; i < data->numrows; i++)
+ {
+ /* remember the index of the sample row, to make the partitioning simpler */
+ values[nvalues].value = heap_getattr(data->rows[i], attrs->values[dimension],
+ stats[dimension]->tupDesc, &isNull);
+ values[nvalues].tupno = i;
+
+ /* no NULL values allowed here (we never split null-only dimension) */
+ Assert(!isNull);
+
+ nvalues++;
+ }
+
+ /* sort the array of values */
+ qsort_arg((void *) values, nvalues, sizeof(ScalarItem),
+ compare_scalars_partition, (void *) &ssup);
+
+ /*
+ * We know there are bucket->ndistincts[dimension] distinct values in this
+ * dimension, and we want to split this into half, so walk through the
+ * array and stop once we see (ndistinct/2) values.
+ *
+ * We always choose the "next" value, i.e. (n/2+1)-th distinct value, and
+ * use it as an exclusive upper boundary (and inclusive lower boundary).
+ *
+ * TODO Maybe we should use "average" of the two middle distinct values
+ * (at least for even distinct counts), but that would require being
+ * able to do an average (which does not work for non-numeric types).
+ *
+ * TODO Another option is to look for a split that'd give about 50% tuples
+ * (not distinct values) in each partition. That might work better
+ * when there are a few very frequent values, and many rare ones.
+ */
+ delta = fabs(data->numrows);
+ split_value = values[0].value;
+
+ for (i = 1; i < data->numrows; i++)
+ {
+ if (values[i].value != values[i-1].value)
+ {
+ /* are we closer to splitting the bucket in half? */
+ if (fabs(i - data->numrows/2.0) < delta)
+ {
+ /* let's assume we'll use this value for the split */
+ split_value = values[i].value;
+ delta = fabs(i - data->numrows/2.0);
+ nrows = i;
+ }
+ }
+ }
+
+ Assert(nrows > 0);
+ Assert(nrows < data->numrows);
+
+ /* create the new bucket as a (incomplete) copy of the one being partitioned. */
+ new_bucket = copy_mv_bucket(bucket, numattrs);
+ new_data = (HistogramBuild)new_bucket->build_data;
+
+ /*
+ * Do the actual split of the chosen dimension, using the split value as the
+ * upper bound for the existing bucket, and lower bound for the new one.
+ */
+ bucket->max[dimension] = split_value;
+ new_bucket->min[dimension] = split_value;
+
+ bucket->max_inclusive[dimension] = false;
+ new_bucket->max_inclusive[dimension] = true;
+
+ /*
+ * Redistribute the sample tuples using the 'ScalarItem->tupno' index. We
+ * know 'nrows' rows should remain in the original bucket and the rest goes
+ * to the new one.
+ */
+
+ data->rows = (HeapTuple*)palloc0(nrows * sizeof(HeapTuple));
+ new_data->rows = (HeapTuple*)palloc0((oldnrows - nrows) * sizeof(HeapTuple));
+
+ data->numrows = nrows;
+ new_data->numrows = (oldnrows - nrows);
+
+ /*
+ * The first nrows should go to the first bucket, the rest should go to the
+ * new one. Use the tupno field to get the actual HeapTuple row from the
+ * original array of sample rows.
+ */
+ for (i = 0; i < nrows; i++)
+ memcpy(&data->rows[i], &oldrows[values[i].tupno], sizeof(HeapTuple));
+
+ for (i = nrows; i < oldnrows; i++)
+ memcpy(&new_data->rows[i-nrows], &oldrows[values[i].tupno], sizeof(HeapTuple));
+
+ /* update ndistinct values for the buckets (total and per dimension) */
+ update_bucket_ndistinct(bucket, attrs, stats);
+ update_bucket_ndistinct(new_bucket, attrs, stats);
+
+ /*
+ * TODO We don't need to do this for the dimension we used for split,
+ * because we know how many distinct values went to each partition.
+ */
+ for (i = 0; i < numattrs; i++)
+ {
+ update_dimension_ndistinct(bucket, i, attrs, stats, false);
+ update_dimension_ndistinct(new_bucket, i, attrs, stats, false);
+ }
+
+ pfree(oldrows);
+ pfree(values);
+
+ return new_bucket;
+}
+
+/*
+ * Copy a histogram bucket. The copy does not include the build-time data, i.e.
+ * sampled rows etc.
+ */
+static MVBucket
+copy_mv_bucket(MVBucket bucket, uint32 ndimensions)
+{
+ /* TODO allocate as a single piece (including all the fields) */
+ MVBucket new_bucket = (MVBucket)palloc0(sizeof(MVBucketData));
+ HistogramBuild data = (HistogramBuild)palloc0(sizeof(HistogramBuildData));
+
+ /* Copy only the attributes that will stay the same after the split, and
+ * we'll recompute the rest after the split. */
+
+ /* allocate the per-dimension arrays */
+ new_bucket->nullsonly = (bool*)palloc0(ndimensions * sizeof(bool));
+
+ /* inclusiveness boundaries - lower/upper bounds */
+ new_bucket->min_inclusive = (bool*)palloc0(ndimensions * sizeof(bool));
+ new_bucket->max_inclusive = (bool*)palloc0(ndimensions * sizeof(bool));
+
+ /* lower/upper boundaries */
+ new_bucket->min = (Datum*)palloc0(ndimensions * sizeof(Datum));
+ new_bucket->max = (Datum*)palloc0(ndimensions * sizeof(Datum));
+
+ /* copy data */
+ memcpy(new_bucket->nullsonly, bucket->nullsonly, ndimensions * sizeof(bool));
+
+ memcpy(new_bucket->min_inclusive, bucket->min_inclusive, ndimensions*sizeof(bool));
+ memcpy(new_bucket->min, bucket->min, ndimensions*sizeof(Datum));
+
+ memcpy(new_bucket->max_inclusive, bucket->max_inclusive, ndimensions*sizeof(bool));
+ memcpy(new_bucket->max, bucket->max, ndimensions*sizeof(Datum));
+
+ /* allocate and copy the interesting part of the build data */
+ data->ndistincts = (uint32*)palloc0(ndimensions * sizeof(uint32));
+
+ new_bucket->build_data = data;
+
+ return new_bucket;
+}
+
+/*
+ * Counts the number of distinct values in the bucket. This just copies the
+ * Datum values into a simple array, and sorts them using memcmp-based
+ * comparator. That means it only works for pass-by-value data types (assuming
+ * they don't use collations etc.)
+ */
+static void
+update_bucket_ndistinct(MVBucket bucket, int2vector *attrs, VacAttrStats ** stats)
+{
+ int i, j;
+ int numattrs = attrs->dim1;
+
+ HistogramBuild data = (HistogramBuild)bucket->build_data;
+ int numrows = data->numrows;
+
+ MultiSortSupport mss = multi_sort_init(numattrs);
+
+ /*
+ * We could collect this while walking through all the attributes above
+ * (this way we have to call heap_getattr twice).
+ */
+ SortItem *items = (SortItem*)palloc0(numrows * sizeof(SortItem));
+ Datum *values = (Datum*)palloc0(numrows * sizeof(Datum) * numattrs);
+ bool *isnull = (bool*)palloc0(numrows * sizeof(bool) * numattrs);
+
+ for (i = 0; i < numrows; i++)
+ {
+ items[i].values = &values[i * numattrs];
+ items[i].isnull = &isnull[i * numattrs];
+ }
+
+ /* prepare the sort function for the first dimension */
+ for (i = 0; i < numattrs; i++)
+ multi_sort_add_dimension(mss, i, i, stats);
+
+ /* collect the values */
+ for (i = 0; i < numrows; i++)
+ for (j = 0; j < numattrs; j++)
+ items[i].values[j]
+ = heap_getattr(data->rows[i], attrs->values[j],
+ stats[j]->tupDesc, &items[i].isnull[j]);
+
+ qsort_arg((void *) items, numrows, sizeof(SortItem),
+ multi_sort_compare, mss);
+
+ data->ndistinct = 1;
+
+ for (i = 1; i < numrows; i++)
+ if (multi_sort_compare(&items[i], &items[i-1], mss) != 0)
+ data->ndistinct += 1;
+
+ pfree(items);
+ pfree(values);
+ pfree(isnull);
+}
+
+/*
+ * Count distinct values per bucket dimension.
+ */
+static void
+update_dimension_ndistinct(MVBucket bucket, int dimension, int2vector *attrs,
+ VacAttrStats ** stats, bool update_boundaries)
+{
+ int j;
+ int nvalues = 0;
+ bool isNull;
+ HistogramBuild data = (HistogramBuild)bucket->build_data;
+ Datum * values = (Datum*)palloc0(data->numrows * sizeof(Datum));
+ SortSupportData ssup;
+
+ StdAnalyzeData * mystats = (StdAnalyzeData *) stats[dimension]->extra_data;
+
+ /* we may already know this is a NULL-only dimension */
+ if (bucket->nullsonly[dimension])
+ data->ndistincts[dimension] = 1;
+
+ memset(&ssup, 0, sizeof(ssup));
+ ssup.ssup_cxt = CurrentMemoryContext;
+
+ /* We always use the default collation for statistics */
+ ssup.ssup_collation = DEFAULT_COLLATION_OID;
+ ssup.ssup_nulls_first = false;
+
+ PrepareSortSupportFromOrderingOp(mystats->ltopr, &ssup);
+
+ for (j = 0; j < data->numrows; j++)
+ {
+ values[nvalues] = heap_getattr(data->rows[j], attrs->values[dimension],
+ stats[dimension]->tupDesc, &isNull);
+
+ /* ignore NULL values */
+ if (! isNull)
+ nvalues++;
+ }
+
+ /* there's always at least 1 distinct value (may be NULL) */
+ data->ndistincts[dimension] = 1;
+
+ /* if there are only NULL values in the column, mark it so and continue
+ * with the next one */
+ if (nvalues == 0)
+ {
+ pfree(values);
+ bucket->nullsonly[dimension] = true;
+ return;
+ }
+
+ /* sort the array (pass-by-value datum */
+ qsort_arg((void *) values, nvalues, sizeof(Datum),
+ compare_scalars_simple, (void *) &ssup);
+
+ /*
+ * Update min/max boundaries to the smallest bounding box. Generally, this
+ * needs to be done only when constructing the initial bucket.
+ */
+ if (update_boundaries)
+ {
+ /* store the min/max values */
+ bucket->min[dimension] = values[0];
+ bucket->min_inclusive[dimension] = true;
+
+ bucket->max[dimension] = values[nvalues-1];
+ bucket->max_inclusive[dimension] = true;
+ }
+
+ /*
+ * Walk through the array and count distinct values by comparing
+ * succeeding values.
+ *
+ * FIXME This only works for pass-by-value types (i.e. not VARCHARs
+ * etc.). Although thanks to the deduplication it might work
+ * even for those types (equal values will get the same item
+ * in the deduplicated array).
+ */
+ for (j = 1; j < nvalues; j++) {
+ if (values[j] != values[j-1])
+ data->ndistincts[dimension] += 1;
+ }
+
+ pfree(values);
+}
+
+/*
+ * A properly built histogram must not contain buckets mixing NULL and non-NULL
+ * values in a single dimension. Each dimension may either be marked as 'nulls
+ * only', and thus containing only NULL values, or it must not contain any NULL
+ * values.
+ *
+ * Therefore, if the sample contains NULL values in any of the columns, it's
+ * necessary to build those NULL-buckets. This is done in an iterative way
+ * using this algorithm, operating on a single bucket:
+ *
+ * (1) Check that all dimensions are well-formed (not mixing NULL and
+ * non-NULL values).
+ *
+ * (2) If all dimensions are well-formed, terminate.
+ *
+ * (3) If the dimension contains only NULL values, but is not marked as
+ * NULL-only, mark it as NULL-only and run the algorithm again (on
+ * this bucket).
+ *
+ * (4) If the dimension mixes NULL and non-NULL values, split the bucket
+ * into two parts - one with NULL values, one with non-NULL values
+ * (replacing the current one). Then run the algorithm on both buckets.
+ *
+ * This is executed in a recursive manner, but the number of executions should
+ * be quite low - limited by the number of NULL-buckets. Also, in each branch
+ * the number of nested calls is limited by the number of dimensions
+ * (attributes) of the histogram.
+ *
+ * At the end, there should be buckets with no mixed dimensions. The number of
+ * buckets produced by this algorithm is rather limited - with N dimensions,
+ * there may be only 2^N such buckets (each dimension may be either NULL or
+ * non-NULL). So with 8 dimensions (current value of MVSTATS_MAX_DIMENSIONS)
+ * there may be only 256 such buckets.
+ *
+ * After this, a 'regular' bucket-split algorithm shall run, further optimizing
+ * the histogram.
+ */
+static void
+create_null_buckets(MVHistogram histogram, int bucket_idx,
+ int2vector *attrs, VacAttrStats ** stats)
+{
+ int i, j;
+ int null_dim = -1;
+ int null_count = 0;
+ bool null_found = false;
+ MVBucket bucket, null_bucket;
+ int null_idx, curr_idx;
+ HistogramBuild data, null_data;
+
+ /* remember original values from the bucket */
+ int numrows;
+ HeapTuple *oldrows = NULL;
+
+ Assert(bucket_idx < histogram->nbuckets);
+ Assert(histogram->ndimensions == attrs->dim1);
+
+ bucket = histogram->buckets[bucket_idx];
+ data = (HistogramBuild)bucket->build_data;
+
+ numrows = data->numrows;
+ oldrows = data->rows;
+
+ /*
+ * Walk through all rows / dimensions, and stop once we find NULL in a
+ * dimension not yet marked as NULL-only.
+ */
+ for (i = 0; i < data->numrows; i++)
+ {
+ /*
+ * FIXME We don't need to start from the first attribute here - we can
+ * start from the last known dimension.
+ */
+ for (j = 0; j < histogram->ndimensions; j++)
+ {
+ /* Is this a NULL-only dimension? If yes, skip. */
+ if (bucket->nullsonly[j])
+ continue;
+
+ /* found a NULL in that dimension? */
+ if (heap_attisnull(data->rows[i], attrs->values[j]))
+ {
+ null_found = true;
+ null_dim = j;
+ break;
+ }
+ }
+
+ /* terminate if we found attribute with NULL values */
+ if (null_found)
+ break;
+ }
+
+ /* no regular dimension contains NULL values => we're done */
+ if (! null_found)
+ return;
+
+ /* walk through the rows again, count NULL values in 'null_dim' */
+ for (i = 0; i < data->numrows; i++)
+ {
+ if (heap_attisnull(data->rows[i], attrs->values[null_dim]))
+ null_count += 1;
+ }
+
+ Assert(null_count <= data->numrows);
+
+ /*
+ * If (null_count == numrows) the dimension already is NULL-only, but is
+ * not yet marked like that. It's enough to mark it and repeat the process
+ * recursively (until we run out of dimensions).
+ */
+ if (null_count == data->numrows)
+ {
+ bucket->nullsonly[null_dim] = true;
+ create_null_buckets(histogram, bucket_idx, attrs, stats);
+ return;
+ }
+
+ /*
+ * We have to split the bucket into two - one with NULL values in the
+ * dimension, one with non-NULL values. We don't need to sort the data or
+ * anything, but otherwise it's similar to what partition_bucket() does.
+ */
+
+ /* create bucket with NULL-only dimension 'dim' */
+ null_bucket = copy_mv_bucket(bucket, histogram->ndimensions);
+ null_data = (HistogramBuild)null_bucket->build_data;
+
+ /* remember the current array info */
+ oldrows = data->rows;
+ numrows = data->numrows;
+
+ /* we'll keep non-NULL values in the current bucket */
+ data->numrows = (numrows - null_count);
+ data->rows
+ = (HeapTuple*)palloc0(data->numrows * sizeof(HeapTuple));
+
+ /* and the NULL values will go to the new one */
+ null_data->numrows = null_count;
+ null_data->rows
+ = (HeapTuple*)palloc0(null_data->numrows * sizeof(HeapTuple));
+
+ /* mark the dimension as NULL-only (in the new bucket) */
+ null_bucket->nullsonly[null_dim] = true;
+
+ /* walk through the sample rows and distribute them accordingly */
+ null_idx = 0;
+ curr_idx = 0;
+ for (i = 0; i < numrows; i++)
+ {
+ if (heap_attisnull(oldrows[i], attrs->values[null_dim]))
+ /* NULL => copy to the new bucket */
+ memcpy(&null_data->rows[null_idx++], &oldrows[i],
+ sizeof(HeapTuple));
+ else
+ memcpy(&data->rows[curr_idx++], &oldrows[i],
+ sizeof(HeapTuple));
+ }
+
+ /* update ndistinct values for the buckets (total and per dimension) */
+ update_bucket_ndistinct(bucket, attrs, stats);
+ update_bucket_ndistinct(null_bucket, attrs, stats);
+
+ /*
+ * TODO We don't need to do this for the dimension we used for split,
+ * because we know how many distinct values went to each bucket (NULL
+ * is not a value, so NULL buckets get 0, and the other bucket got all
+ * the distinct values).
+ */
+ for (i = 0; i < histogram->ndimensions; i++)
+ {
+ update_dimension_ndistinct(bucket, i, attrs, stats, false);
+ update_dimension_ndistinct(null_bucket, i, attrs, stats, false);
+ }
+
+ pfree(oldrows);
+
+ /* add the NULL bucket to the histogram */
+ histogram->buckets[histogram->nbuckets++] = null_bucket;
+
+ /*
+ * And now run the function recursively on both buckets (the new
+ * one first, because the call may change number of buckets, and
+ * it's used as an index).
+ */
+ create_null_buckets(histogram, (histogram->nbuckets-1), attrs, stats);
+ create_null_buckets(histogram, bucket_idx, attrs, stats);
+}
+
+/*
+ * SRF with details about buckets of a histogram:
+ *
+ * - bucket ID (0...nbuckets)
+ * - min values (string array)
+ * - max values (string array)
+ * - nulls only (boolean array)
+ * - min inclusive flags (boolean array)
+ * - max inclusive flags (boolean array)
+ * - frequency (double precision)
+ *
+ * The input is the OID of the statistics, and there are no rows returned if the
+ * statistics contains no histogram (or if there's no statistics for the OID).
+ *
+ * The second parameter (type) determines what values will be returned
+ * in the (minvals,maxvals). There are three possible values:
+ *
+ * 0 (actual values)
+ * -----------------
+ * - prints actual values
+ * - using the output function of the data type (as string)
+ * - handy for investigating the histogram
+ *
+ * 1 (distinct index)
+ * ------------------
+ * - prints index of the distinct value (into the serialized array)
+ * - makes it easier to spot neighbor buckets, etc.
+ * - handy for plotting the histogram
+ *
+ * 2 (normalized distinct index)
+ * -----------------------------
+ * - prints index of the distinct value, but normalized into [0,1]
+ * - similar to 1, but shows how 'long' the bucket range is
+ * - handy for plotting the histogram
+ *
+ * When plotting the histogram, be careful as the (1) and (2) options skew the
+ * lengths by distributing the distinct values uniformly. For data types
+ * without a clear meaning of 'distance' (e.g. strings) that is not a big deal,
+ * but for numbers it may be confusing.
+ */
+PG_FUNCTION_INFO_V1(pg_mv_histogram_buckets);
+
+#define OUTPUT_FORMAT_RAW 0
+#define OUTPUT_FORMAT_INDEXES 1
+#define OUTPUT_FORMAT_DISTINCT 2
+
+Datum
+pg_mv_histogram_buckets(PG_FUNCTION_ARGS)
+{
+ FuncCallContext *funcctx;
+ int call_cntr;
+ int max_calls;
+ TupleDesc tupdesc;
+ AttInMetadata *attinmeta;
+
+ Oid mvoid = PG_GETARG_OID(0);
+ int otype = PG_GETARG_INT32(1);
+
+ if ((otype < 0) || (otype > 2))
+ elog(ERROR, "invalid output type specified");
+
+ /* stuff done only on the first call of the function */
+ if (SRF_IS_FIRSTCALL())
+ {
+ MemoryContext oldcontext;
+ MVSerializedHistogram histogram;
+
+ /* create a function context for cross-call persistence */
+ funcctx = SRF_FIRSTCALL_INIT();
+
+ /* switch to memory context appropriate for multiple function calls */
+ oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
+
+ histogram = load_mv_histogram(mvoid);
+
+ funcctx->user_fctx = histogram;
+
+ /* total number of tuples to be returned */
+ funcctx->max_calls = 0;
+ if (funcctx->user_fctx != NULL)
+ funcctx->max_calls = histogram->nbuckets;
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("function returning record called in context "
+ "that cannot accept type record")));
+
+ /*
+ * generate attribute metadata needed later to produce tuples
+ * from raw C strings
+ */
+ attinmeta = TupleDescGetAttInMetadata(tupdesc);
+ funcctx->attinmeta = attinmeta;
+
+ MemoryContextSwitchTo(oldcontext);
+ }
+
+ /* stuff done on every call of the function */
+ funcctx = SRF_PERCALL_SETUP();
+
+ call_cntr = funcctx->call_cntr;
+ max_calls = funcctx->max_calls;
+ attinmeta = funcctx->attinmeta;
+
+ if (call_cntr < max_calls) /* do when there is more left to send */
+ {
+ char **values;
+ HeapTuple tuple;
+ Datum result;
+ int2vector *stakeys;
+ Oid relid;
+ double bucket_volume = 1.0;
+ StringInfo bufs;
+
+ char *format;
+ int i;
+
+ Oid *outfuncs;
+ FmgrInfo *fmgrinfo;
+
+ MVSerializedHistogram histogram;
+ MVSerializedBucket bucket;
+
+ histogram = (MVSerializedHistogram)funcctx->user_fctx;
+
+ Assert(call_cntr < histogram->nbuckets);
+
+ bucket = histogram->buckets[call_cntr];
+
+ stakeys = find_mv_attnums(mvoid, &relid);
+
+ /*
+ * The scalar values will be formatted directly, using snprintf.
+ *
+ * The 'array' values will be formatted through StringInfo.
+ */
+ values = (char **) palloc0(9 * sizeof(char *));
+ bufs = (StringInfo) palloc0(9 * sizeof(StringInfoData));
+
+ values[0] = (char *) palloc(64 * sizeof(char));
+
+ initStringInfo(&bufs[1]); /* lower boundaries */
+ initStringInfo(&bufs[2]); /* upper boundaries */
+ initStringInfo(&bufs[3]); /* nulls-only */
+ initStringInfo(&bufs[4]); /* lower inclusive */
+ initStringInfo(&bufs[5]); /* upper inclusive */
+
+ values[6] = (char *) palloc(64 * sizeof(char));
+ values[7] = (char *) palloc(64 * sizeof(char));
+ values[8] = (char *) palloc(64 * sizeof(char));
+
+ /* we need to do this only when printing the actual values */
+ outfuncs = (Oid*)palloc0(sizeof(Oid) * histogram->ndimensions);
+ fmgrinfo = (FmgrInfo*)palloc0(sizeof(FmgrInfo) * histogram->ndimensions);
+
+ /*
+ * lookup output functions for all histogram dimensions
+ *
+ * XXX This might be one in the first call and stored in user_fctx.
+ */
+ for (i = 0; i < histogram->ndimensions; i++)
+ {
+ bool isvarlena;
+
+ getTypeOutputInfo(get_atttype(relid, stakeys->values[i]),
+ &outfuncs[i], &isvarlena);
+
+ fmgr_info(outfuncs[i], &fmgrinfo[i]);
+ }
+
+ snprintf(values[0], 64, "%d", call_cntr); /* bucket ID */
+
+ /* for the arrays of lower/upper boundaries, formated according to otype */
+ for (i = 0; i < histogram->ndimensions; i++)
+ {
+ Datum *vals = histogram->values[i];
+
+ uint16 minidx = bucket->min[i];
+ uint16 maxidx = bucket->max[i];
+
+ /* compute bucket volume, using distinct values as a measure
+ *
+ * XXX Not really sure what to do for NULL dimensions here, so let's
+ * simply count them as '1'.
+ */
+ bucket_volume
+ *= (double)(maxidx - minidx + 1) / (histogram->nvalues[i]-1);
+
+ if (i == 0)
+ format = "{%s"; /* fist dimension */
+ else if (i < (histogram->ndimensions - 1))
+ format = ", %s"; /* medium dimensions */
+ else
+ format = ", %s}"; /* last dimension */
+
+ appendStringInfo(&bufs[3], format, bucket->nullsonly[i] ? "t" : "f");
+ appendStringInfo(&bufs[4], format, bucket->min_inclusive[i] ? "t" : "f");
+ appendStringInfo(&bufs[5], format, bucket->max_inclusive[i] ? "t" : "f");
+
+ /* for NULL-only dimension, simply put there the NULL and continue */
+ if (bucket->nullsonly[i])
+ {
+ if (i == 0)
+ format = "{%s";
+ else if (i < (histogram->ndimensions - 1))
+ format = ", %s";
+ else
+ format = ", %s}";
+
+ appendStringInfo(&bufs[1], format, "NULL");
+ appendStringInfo(&bufs[2], format, "NULL");
+
+ continue;
+ }
+
+ /* otherwise we really need to format the value */
+ switch (otype)
+ {
+ case OUTPUT_FORMAT_RAW: /* actual boundary values */
+
+ if (i == 0)
+ format = "{%s";
+ else if (i < (histogram->ndimensions - 1))
+ format = ", %s";
+ else
+ format = ", %s}";
+
+ appendStringInfo(&bufs[1], format,
+ FunctionCall1(&fmgrinfo[i], vals[minidx]));
+
+ appendStringInfo(&bufs[2], format,
+ FunctionCall1(&fmgrinfo[i], vals[maxidx]));
+
+ break;
+
+ case OUTPUT_FORMAT_INDEXES: /* indexes into deduplicated arrays */
+
+ if (i == 0)
+ format = "{%d";
+ else if (i < (histogram->ndimensions - 1))
+ format = ", %d";
+ else
+ format = ", %d}";
+
+ appendStringInfo(&bufs[1], format, minidx);
+
+ appendStringInfo(&bufs[2], format, maxidx);
+
+ break;
+
+ case OUTPUT_FORMAT_DISTINCT: /* distinct arrays as measure */
+
+ if (i == 0)
+ format = "{%f";
+ else if (i < (histogram->ndimensions - 1))
+ format = ", %f";
+ else
+ format = ", %f}";
+
+ appendStringInfo(&bufs[1], format,
+ (minidx * 1.0 / (histogram->nvalues[i]-1)));
+
+ appendStringInfo(&bufs[2], format,
+ (maxidx * 1.0 / (histogram->nvalues[i]-1)));
+
+ break;
+
+ default:
+ elog(ERROR, "unknown output type: %d", otype);
+ }
+ }
+
+ values[1] = bufs[1].data;
+ values[2] = bufs[2].data;
+ values[3] = bufs[3].data;
+ values[4] = bufs[4].data;
+ values[5] = bufs[5].data;
+
+ snprintf(values[6], 64, "%f", bucket->ntuples); /* frequency */
+ snprintf(values[7], 64, "%f", bucket->ntuples / bucket_volume); /* density */
+ snprintf(values[8], 64, "%f", bucket_volume); /* volume (as a fraction) */
+
+ /* build a tuple */
+ tuple = BuildTupleFromCStrings(attinmeta, values);
+
+ /* make the tuple into a datum */
+ result = HeapTupleGetDatum(tuple);
+
+ /* clean up (this is not really necessary) */
+ pfree(values[0]);
+ pfree(values[6]);
+ pfree(values[7]);
+ pfree(values[8]);
+
+ resetStringInfo(&bufs[1]);
+ resetStringInfo(&bufs[2]);
+ resetStringInfo(&bufs[3]);
+ resetStringInfo(&bufs[4]);
+ resetStringInfo(&bufs[5]);
+
+ pfree(bufs);
+ pfree(values);
+
+ SRF_RETURN_NEXT(funcctx, result);
+ }
+ else /* do when there is no more left */
+ {
+ SRF_RETURN_DONE(funcctx);
+ }
+}
+
+#ifdef DEBUG_MVHIST
+/*
+ * prints debugging info about matched histogram buckets (full/partial)
+ *
+ * XXX Currently works only for INT data type.
+ */
+void
+debug_histogram_matches(MVSerializedHistogram mvhist, char *matches)
+{
+ int i, j;
+
+ float ffull = 0, fpartial = 0;
+ int nfull = 0, npartial = 0;
+
+ StringInfoData buf;
+
+ initStringInfo(&buf);
+
+ for (i = 0; i < mvhist->nbuckets; i++)
+ {
+ MVSerializedBucket bucket = mvhist->buckets[i];
+
+ if (! matches[i])
+ continue;
+
+ /* increment the counters */
+ nfull += (matches[i] == MVSTATS_MATCH_FULL) ? 1 : 0;
+ npartial += (matches[i] == MVSTATS_MATCH_PARTIAL) ? 1 : 0;
+
+ /* and also update the frequencies */
+ ffull += (matches[i] == MVSTATS_MATCH_FULL) ? bucket->ntuples : 0;
+ fpartial += (matches[i] == MVSTATS_MATCH_PARTIAL) ? bucket->ntuples : 0;
+
+ resetStringInfo(&buf);
+
+ /* build ranges for all the dimentions */
+ for (j = 0; j < mvhist->ndimensions; j++)
+ {
+ appendStringInfo(&buf, '[%d %d]',
+ DatumGetInt32(mvhist->values[j][bucket->min[j]]),
+ DatumGetInt32(mvhist->values[j][bucket->max[j]]));
+ }
+
+ elog(WARNING, "bucket %d %s => %d [%f]", i, buf.data, matches[i], bucket->ntuples);
+ }
+
+ elog(WARNING, "full=%f partial=%f (%f)", ffull, fpartial, (ffull + 0.5 * fpartial));
+}
+#endif
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 2c22d31..b693f36 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -2109,9 +2109,9 @@ describeOneTableDetails(const char *schemaname,
{
printfPQExpBuffer(&buf,
"SELECT oid, stanamespace::regnamespace AS nsp, staname, stakeys,\n"
- " deps_enabled, mcv_enabled,\n"
- " deps_built, mcv_built,\n"
- " mcv_max_items,\n"
+ " deps_enabled, mcv_enabled, hist_enabled,\n"
+ " deps_built, mcv_built, hist_built,\n"
+ " mcv_max_items, hist_max_buckets,\n"
" (SELECT string_agg(attname::text,', ')\n"
" FROM ((SELECT unnest(stakeys) AS attnum) s\n"
" JOIN pg_attribute a ON (starelid = a.attrelid and a.attnum = s.attnum))) AS attnums\n"
@@ -2154,8 +2154,17 @@ describeOneTableDetails(const char *schemaname,
first = false;
}
+ if (!strcmp(PQgetvalue(result, i, 6), "t"))
+ {
+ if (! first)
+ appendPQExpBuffer(&buf, ", histogram");
+ else
+ appendPQExpBuffer(&buf, "(histogram");
+ first = false;
+ }
+
appendPQExpBuffer(&buf, ") ON (%s)",
- PQgetvalue(result, i, 9));
+ PQgetvalue(result, i, 12));
printTableAddFooter(&cont, buf.data);
}
diff --git a/src/include/catalog/pg_mv_statistic.h b/src/include/catalog/pg_mv_statistic.h
index 3529b03..7020772 100644
--- a/src/include/catalog/pg_mv_statistic.h
+++ b/src/include/catalog/pg_mv_statistic.h
@@ -39,13 +39,16 @@ CATALOG(pg_mv_statistic,3381)
/* statistics requested to build */
bool deps_enabled; /* analyze dependencies? */
bool mcv_enabled; /* build MCV list? */
+ bool hist_enabled; /* build histogram? */
- /* MCV size */
+ /* histogram / MCV size */
int32 mcv_max_items; /* max MCV items */
+ int32 hist_max_buckets; /* max histogram buckets */
/* statistics that are available (if requested) */
bool deps_built; /* dependencies were built */
bool mcv_built; /* MCV list was built */
+ bool hist_built; /* histogram was built */
/* variable-length fields start here, but we allow direct access to stakeys */
int2vector stakeys; /* array of column keys */
@@ -53,6 +56,7 @@ CATALOG(pg_mv_statistic,3381)
#ifdef CATALOG_VARLEN
bytea stadeps; /* dependencies (serialized) */
bytea stamcv; /* MCV list (serialized) */
+ bytea stahist; /* MV histogram (serialized) */
#endif
} FormData_pg_mv_statistic;
@@ -68,18 +72,22 @@ typedef FormData_pg_mv_statistic *Form_pg_mv_statistic;
* compiler constants for pg_mv_statistic
* ----------------
*/
-#define Natts_pg_mv_statistic 12
+#define Natts_pg_mv_statistic 16
#define Anum_pg_mv_statistic_starelid 1
#define Anum_pg_mv_statistic_staname 2
#define Anum_pg_mv_statistic_stanamespace 3
#define Anum_pg_mv_statistic_staowner 4
#define Anum_pg_mv_statistic_deps_enabled 5
#define Anum_pg_mv_statistic_mcv_enabled 6
-#define Anum_pg_mv_statistic_mcv_max_items 7
-#define Anum_pg_mv_statistic_deps_built 8
-#define Anum_pg_mv_statistic_mcv_built 9
-#define Anum_pg_mv_statistic_stakeys 10
-#define Anum_pg_mv_statistic_stadeps 11
-#define Anum_pg_mv_statistic_stamcv 12
+#define Anum_pg_mv_statistic_hist_enabled 7
+#define Anum_pg_mv_statistic_mcv_max_items 8
+#define Anum_pg_mv_statistic_hist_max_buckets 9
+#define Anum_pg_mv_statistic_deps_built 10
+#define Anum_pg_mv_statistic_mcv_built 11
+#define Anum_pg_mv_statistic_hist_built 12
+#define Anum_pg_mv_statistic_stakeys 13
+#define Anum_pg_mv_statistic_stadeps 14
+#define Anum_pg_mv_statistic_stamcv 15
+#define Anum_pg_mv_statistic_stahist 16
#endif /* PG_MV_STATISTIC_H */
diff --git a/src/include/catalog/pg_proc.h b/src/include/catalog/pg_proc.h
index 5640dc1..f2f735d 100644
--- a/src/include/catalog/pg_proc.h
+++ b/src/include/catalog/pg_proc.h
@@ -2674,6 +2674,10 @@ DATA(insert OID = 3376 ( pg_mv_stats_mcvlist_info PGNSP PGUID 12 1 0 0 0 f f f
DESCR("multi-variate statistics: MCV list info");
DATA(insert OID = 3373 ( pg_mv_mcv_items PGNSP PGUID 12 1 1000 0 0 f f f f t t i s 1 0 2249 "26" "{26,23,1009,1000,701}" "{i,o,o,o,o}" "{oid,index,values,nulls,frequency}" _null_ _null_ pg_mv_mcv_items _null_ _null_ _null_ ));
DESCR("details about MCV list items");
+DATA(insert OID = 3375 ( pg_mv_stats_histogram_info PGNSP PGUID 12 1 0 0 0 f f f f t f i s 1 0 25 "17" _null_ _null_ _null_ _null_ _null_ pg_mv_stats_histogram_info _null_ _null_ _null_ ));
+DESCR("multi-variate statistics: histogram info");
+DATA(insert OID = 3374 ( pg_mv_histogram_buckets PGNSP PGUID 12 1 1000 0 0 f f f f t t i s 2 0 2249 "26 23" "{26,23,23,1009,1009,1000,1000,1000,701,701,701}" "{i,i,o,o,o,o,o,o,o,o,o}" "{oid,otype,index,minvals,maxvals,nullsonly,mininclusive,maxinclusive,frequency,density,bucket_volume}" _null_ _null_ pg_mv_histogram_buckets _null_ _null_ _null_ ));
+DESCR("details about histogram buckets");
DATA(insert OID = 1928 ( pg_stat_get_numscans PGNSP PGUID 12 1 0 0 0 f f f f t f s r 1 0 20 "26" _null_ _null_ _null_ _null_ _null_ pg_stat_get_numscans _null_ _null_ _null_ ));
DESCR("statistics: number of scans done for table/index");
diff --git a/src/include/nodes/relation.h b/src/include/nodes/relation.h
index f52884a..84be0ce 100644
--- a/src/include/nodes/relation.h
+++ b/src/include/nodes/relation.h
@@ -656,10 +656,12 @@ typedef struct MVStatisticInfo
/* enabled statistics */
bool deps_enabled; /* functional dependencies enabled */
bool mcv_enabled; /* MCV list enabled */
+ bool hist_enabled; /* histogram enabled */
/* built/available statistics */
bool deps_built; /* functional dependencies built */
bool mcv_built; /* MCV list built */
+ bool hist_built; /* histogram built */
/* columns in the statistics (attnums) */
int2vector *stakeys; /* attnums of the columns covered */
diff --git a/src/include/utils/mvstats.h b/src/include/utils/mvstats.h
index b2643ec..777c7da 100644
--- a/src/include/utils/mvstats.h
+++ b/src/include/utils/mvstats.h
@@ -18,7 +18,7 @@
#include "commands/vacuum.h"
/*
- * Degree of how much MCV item matches a clause.
+ * Degree of how much MCV item / histogram bucket matches a clause.
* This is then considered when computing the selectivity.
*/
#define MVSTATS_MATCH_NONE 0 /* no match at all */
@@ -92,6 +92,123 @@ typedef MCVListData *MCVList;
#define MVSTAT_MCVLIST_MAX_ITEMS 8192 /* max items in MCV list */
/*
+ * Multivariate histograms
+ */
+typedef struct MVBucketData {
+
+ /* Frequencies of this bucket. */
+ float ntuples; /* frequency of tuples tuples */
+
+ /*
+ * Information about dimensions being NULL-only. Not yet used.
+ */
+ bool *nullsonly;
+
+ /* lower boundaries - values and information about the inequalities */
+ Datum *min;
+ bool *min_inclusive;
+
+ /* upper boundaries - values and information about the inequalities */
+ Datum *max;
+ bool *max_inclusive;
+
+ /* used when building the histogram (not serialized/deserialized) */
+ void *build_data;
+
+} MVBucketData;
+
+typedef MVBucketData *MVBucket;
+
+
+typedef struct MVHistogramData {
+
+ uint32 magic; /* magic constant marker */
+ uint32 type; /* type of histogram (BASIC) */
+ uint32 nbuckets; /* number of buckets (buckets array) */
+ uint32 ndimensions; /* number of dimensions */
+
+ MVBucket *buckets; /* array of buckets */
+
+} MVHistogramData;
+
+typedef MVHistogramData *MVHistogram;
+
+/*
+ * Histogram in a partially serialized form, with deduplicated boundary
+ * values etc.
+ *
+ * TODO add more detailed description here
+ */
+
+typedef struct MVSerializedBucketData {
+
+ /* Frequencies of this bucket. */
+ float ntuples; /* frequency of tuples tuples */
+
+ /*
+ * Information about dimensions being NULL-only. Not yet used.
+ */
+ bool *nullsonly;
+
+ /* lower boundaries - values and information about the inequalities */
+ uint16 *min;
+ bool *min_inclusive;
+
+ /* indexes of upper boundaries - values and information about the
+ * inequalities (exclusive vs. inclusive) */
+ uint16 *max;
+ bool *max_inclusive;
+
+} MVSerializedBucketData;
+
+typedef MVSerializedBucketData *MVSerializedBucket;
+
+typedef struct MVSerializedHistogramData {
+
+ uint32 magic; /* magic constant marker */
+ uint32 type; /* type of histogram (BASIC) */
+ uint32 nbuckets; /* number of buckets (buckets array) */
+ uint32 ndimensions; /* number of dimensions */
+
+ /*
+ * keep this the same with MVHistogramData, because of
+ * deserialization (same offset)
+ */
+ MVSerializedBucket *buckets; /* array of buckets */
+
+ /*
+ * serialized boundary values, one array per dimension, deduplicated
+ * (the min/max indexes point into these arrays)
+ */
+ int *nvalues;
+ Datum **values;
+
+} MVSerializedHistogramData;
+
+typedef MVSerializedHistogramData *MVSerializedHistogram;
+
+
+/* used to flag stats serialized to bytea */
+#define MVSTAT_HIST_MAGIC 0x7F8C5670 /* marks serialized bytea */
+#define MVSTAT_HIST_TYPE_BASIC 1 /* basic histogram type */
+
+/*
+ * Limits used for max_buckets option, i.e. we're always guaranteed
+ * to have space for at least MVSTAT_HIST_MIN_BUCKETS, and we cannot
+ * have more than MVSTAT_HIST_MAX_BUCKETS buckets.
+ *
+ * This is just a boundary for the 'max' threshold - the actual
+ * histogram may use less buckets than MVSTAT_HIST_MAX_BUCKETS.
+ *
+ * TODO The MVSTAT_HIST_MIN_BUCKETS should be related to the number of
+ * attributes (MVSTATS_MAX_DIMENSIONS) because of NULL-buckets.
+ * There should be at least 2^N buckets, otherwise we may be unable
+ * to build the NULL buckets.
+ */
+#define MVSTAT_HIST_MIN_BUCKETS 128 /* min number of buckets */
+#define MVSTAT_HIST_MAX_BUCKETS 16384 /* max number of buckets */
+
+/*
* TODO Maybe fetching the histogram/MCV list separately is inefficient?
* Consider adding a single `fetch_stats` method, fetching all
* stats specified using flags (or something like that).
@@ -99,20 +216,25 @@ typedef MCVListData *MCVList;
MVDependencies load_mv_dependencies(Oid mvoid);
MCVList load_mv_mcvlist(Oid mvoid);
+MVSerializedHistogram load_mv_histogram(Oid mvoid);
bytea * serialize_mv_dependencies(MVDependencies dependencies);
bytea * serialize_mv_mcvlist(MCVList mcvlist, int2vector *attrs,
VacAttrStats **stats);
+bytea * serialize_mv_histogram(MVHistogram histogram, int2vector *attrs,
+ VacAttrStats **stats);
/* deserialization of stats (serialization is private to analyze) */
MVDependencies deserialize_mv_dependencies(bytea * data);
MCVList deserialize_mv_mcvlist(bytea * data);
+MVSerializedHistogram deserialize_mv_histogram(bytea * data);
/*
* Returns index of the attribute number within the vector (i.e. a
* dimension within the stats).
*/
int mv_get_index(AttrNumber varattno, int2vector * stakeys);
+int2vector* find_mv_attnums(Oid mvoid, Oid *relid);
int2vector* find_mv_attnums(Oid mvoid, Oid *relid);
@@ -121,6 +243,8 @@ extern Datum pg_mv_stats_dependencies_info(PG_FUNCTION_ARGS);
extern Datum pg_mv_stats_dependencies_show(PG_FUNCTION_ARGS);
extern Datum pg_mv_stats_mcvlist_info(PG_FUNCTION_ARGS);
extern Datum pg_mv_mcvlist_items(PG_FUNCTION_ARGS);
+extern Datum pg_mv_stats_histogram_info(PG_FUNCTION_ARGS);
+extern Datum pg_mv_histogram_buckets(PG_FUNCTION_ARGS);
MVDependencies
build_mv_dependencies(int numrows, HeapTuple *rows, int2vector *attrs,
@@ -130,10 +254,20 @@ MCVList
build_mv_mcvlist(int numrows, HeapTuple *rows, int2vector *attrs,
VacAttrStats **stats, int *numrows_filtered);
+MVHistogram
+build_mv_histogram(int numrows, HeapTuple *rows, int2vector *attrs,
+ VacAttrStats **stats, int numrows_total);
+
void build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
int natts, VacAttrStats **vacattrstats);
-void update_mv_stats(Oid relid, MVDependencies dependencies, MCVList mcvlist,
+void update_mv_stats(Oid relid, MVDependencies dependencies,
+ MCVList mcvlist, MVHistogram histogram,
int2vector *attrs, VacAttrStats **stats);
+#ifdef DEBUG_MVHIST
+extern void debug_histogram_matches(MVSerializedHistogram mvhist, char *matches);
+#endif
+
+
#endif
diff --git a/src/test/regress/expected/mv_histogram.out b/src/test/regress/expected/mv_histogram.out
new file mode 100644
index 0000000..e830816
--- /dev/null
+++ b/src/test/regress/expected/mv_histogram.out
@@ -0,0 +1,207 @@
+-- data type passed by value
+CREATE TABLE mv_histogram (
+ a INT,
+ b INT,
+ c INT
+);
+-- unknown column
+CREATE STATISTICS s7 ON mv_histogram (unknown_column) WITH (histogram);
+ERROR: column "unknown_column" referenced in statistics does not exist
+-- single column
+CREATE STATISTICS s7 ON mv_histogram (a) WITH (histogram);
+ERROR: multivariate stats require 2 or more columns
+-- single column, duplicated
+CREATE STATISTICS s7 ON mv_histogram (a, a) WITH (histogram);
+ERROR: duplicate column name in statistics definition
+-- two columns, one duplicated
+CREATE STATISTICS s7 ON mv_histogram (a, a, b) WITH (histogram);
+ERROR: duplicate column name in statistics definition
+-- unknown option
+CREATE STATISTICS s7 ON mv_histogram (a, b, c) WITH (unknown_option);
+ERROR: unrecognized STATISTICS option "unknown_option"
+-- missing histogram statistics
+CREATE STATISTICS s7 ON mv_histogram (a, b, c) WITH (dependencies, max_buckets=200);
+ERROR: option 'histogram' is required by other options(s)
+-- invalid max_buckets value / too low
+CREATE STATISTICS s7 ON mv_histogram (a, b, c) WITH (mcv, max_buckets=10);
+ERROR: minimum number of buckets is 128
+-- invalid max_buckets value / too high
+CREATE STATISTICS s7 ON mv_histogram (a, b, c) WITH (mcv, max_buckets=100000);
+ERROR: maximum number of buckets is 16384
+-- correct command
+CREATE STATISTICS s7 ON mv_histogram (a, b, c) WITH (histogram);
+-- random data (no functional dependencies)
+INSERT INTO mv_histogram
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+ANALYZE mv_histogram;
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+ hist_enabled | hist_built
+--------------+------------
+ t | t
+(1 row)
+
+TRUNCATE mv_histogram;
+-- a => b, a => c, b => c
+INSERT INTO mv_histogram
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE mv_histogram;
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+ hist_enabled | hist_built
+--------------+------------
+ t | t
+(1 row)
+
+TRUNCATE mv_histogram;
+-- a => b, a => c
+INSERT INTO mv_histogram
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE mv_histogram;
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+ hist_enabled | hist_built
+--------------+------------
+ t | t
+(1 row)
+
+TRUNCATE mv_histogram;
+-- check explain (expect bitmap index scan, not plain index scan)
+INSERT INTO mv_histogram
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX hist_idx ON mv_histogram (a, b);
+ANALYZE mv_histogram;
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+ hist_enabled | hist_built
+--------------+------------
+ t | t
+(1 row)
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mv_histogram WHERE a = 10 AND b = 5;
+ QUERY PLAN
+--------------------------------------------
+ Bitmap Heap Scan on mv_histogram
+ Recheck Cond: ((a = 10) AND (b = 5))
+ -> Bitmap Index Scan on hist_idx
+ Index Cond: ((a = 10) AND (b = 5))
+(4 rows)
+
+DROP TABLE mv_histogram;
+-- varlena type (text)
+CREATE TABLE mv_histogram (
+ a TEXT,
+ b TEXT,
+ c TEXT
+);
+CREATE STATISTICS s8 ON mv_histogram (a, b, c) WITH (histogram);
+-- random data (no functional dependencies)
+INSERT INTO mv_histogram
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+ANALYZE mv_histogram;
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+ hist_enabled | hist_built
+--------------+------------
+ t | t
+(1 row)
+
+TRUNCATE mv_histogram;
+-- a => b, a => c, b => c
+INSERT INTO mv_histogram
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE mv_histogram;
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+ hist_enabled | hist_built
+--------------+------------
+ t | t
+(1 row)
+
+TRUNCATE mv_histogram;
+-- a => b, a => c
+INSERT INTO mv_histogram
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE mv_histogram;
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+ hist_enabled | hist_built
+--------------+------------
+ t | t
+(1 row)
+
+TRUNCATE mv_histogram;
+-- check explain (expect bitmap index scan, not plain index scan)
+INSERT INTO mv_histogram
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX hist_idx ON mv_histogram (a, b);
+ANALYZE mv_histogram;
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+ hist_enabled | hist_built
+--------------+------------
+ t | t
+(1 row)
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mv_histogram WHERE a = '10' AND b = '5';
+ QUERY PLAN
+------------------------------------------------------------
+ Bitmap Heap Scan on mv_histogram
+ Recheck Cond: ((a = '10'::text) AND (b = '5'::text))
+ -> Bitmap Index Scan on hist_idx
+ Index Cond: ((a = '10'::text) AND (b = '5'::text))
+(4 rows)
+
+TRUNCATE mv_histogram;
+-- check explain (expect bitmap index scan, not plain index scan) with NULLs
+INSERT INTO mv_histogram
+ SELECT
+ (CASE WHEN i/10000 = 0 THEN NULL ELSE i/10000 END),
+ (CASE WHEN i/20000 = 0 THEN NULL ELSE i/20000 END),
+ (CASE WHEN i/40000 = 0 THEN NULL ELSE i/40000 END)
+ FROM generate_series(1,1000000) s(i);
+ANALYZE mv_histogram;
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+ hist_enabled | hist_built
+--------------+------------
+ t | t
+(1 row)
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mv_histogram WHERE a IS NULL AND b IS NULL;
+ QUERY PLAN
+---------------------------------------------------
+ Bitmap Heap Scan on mv_histogram
+ Recheck Cond: ((a IS NULL) AND (b IS NULL))
+ -> Bitmap Index Scan on hist_idx
+ Index Cond: ((a IS NULL) AND (b IS NULL))
+(4 rows)
+
+DROP TABLE mv_histogram;
+-- NULL values (mix of int and text columns)
+CREATE TABLE mv_histogram (
+ a INT,
+ b TEXT,
+ c INT,
+ d TEXT
+);
+CREATE STATISTICS s9 ON mv_histogram (a, b, c, d) WITH (histogram);
+INSERT INTO mv_histogram
+ SELECT
+ mod(i, 100),
+ (CASE WHEN mod(i, 200) = 0 THEN NULL ELSE mod(i,200) END),
+ mod(i, 400),
+ (CASE WHEN mod(i, 300) = 0 THEN NULL ELSE mod(i,600) END)
+ FROM generate_series(1,10000) s(i);
+ANALYZE mv_histogram;
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+ hist_enabled | hist_built
+--------------+------------
+ t | t
+(1 row)
+
+DROP TABLE mv_histogram;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 3d55ffe..528ac36 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1375,7 +1375,9 @@ pg_mv_stats| SELECT n.nspname AS schemaname,
length(s.stadeps) AS depsbytes,
pg_mv_stats_dependencies_info(s.stadeps) AS depsinfo,
length(s.stamcv) AS mcvbytes,
- pg_mv_stats_mcvlist_info(s.stamcv) AS mcvinfo
+ pg_mv_stats_mcvlist_info(s.stamcv) AS mcvinfo,
+ length(s.stahist) AS histbytes,
+ pg_mv_stats_histogram_info(s.stahist) AS histinfo
FROM ((pg_mv_statistic s
JOIN pg_class c ON ((c.oid = s.starelid)))
LEFT JOIN pg_namespace n ON ((n.oid = c.relnamespace)));
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index 85d94f1..a885235 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -112,4 +112,4 @@ test: event_trigger
test: stats
# run tests of multivariate stats
-test: mv_dependencies mv_mcv
+test: mv_dependencies mv_mcv mv_histogram
diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule
index 6584d73..2efdcd7 100644
--- a/src/test/regress/serial_schedule
+++ b/src/test/regress/serial_schedule
@@ -164,3 +164,4 @@ test: event_trigger
test: stats
test: mv_dependencies
test: mv_mcv
+test: mv_histogram
diff --git a/src/test/regress/sql/mv_histogram.sql b/src/test/regress/sql/mv_histogram.sql
new file mode 100644
index 0000000..27c2510
--- /dev/null
+++ b/src/test/regress/sql/mv_histogram.sql
@@ -0,0 +1,176 @@
+-- data type passed by value
+CREATE TABLE mv_histogram (
+ a INT,
+ b INT,
+ c INT
+);
+
+-- unknown column
+CREATE STATISTICS s7 ON mv_histogram (unknown_column) WITH (histogram);
+
+-- single column
+CREATE STATISTICS s7 ON mv_histogram (a) WITH (histogram);
+
+-- single column, duplicated
+CREATE STATISTICS s7 ON mv_histogram (a, a) WITH (histogram);
+
+-- two columns, one duplicated
+CREATE STATISTICS s7 ON mv_histogram (a, a, b) WITH (histogram);
+
+-- unknown option
+CREATE STATISTICS s7 ON mv_histogram (a, b, c) WITH (unknown_option);
+
+-- missing histogram statistics
+CREATE STATISTICS s7 ON mv_histogram (a, b, c) WITH (dependencies, max_buckets=200);
+
+-- invalid max_buckets value / too low
+CREATE STATISTICS s7 ON mv_histogram (a, b, c) WITH (mcv, max_buckets=10);
+
+-- invalid max_buckets value / too high
+CREATE STATISTICS s7 ON mv_histogram (a, b, c) WITH (mcv, max_buckets=100000);
+
+-- correct command
+CREATE STATISTICS s7 ON mv_histogram (a, b, c) WITH (histogram);
+
+-- random data (no functional dependencies)
+INSERT INTO mv_histogram
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+
+ANALYZE mv_histogram;
+
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+
+TRUNCATE mv_histogram;
+
+-- a => b, a => c, b => c
+INSERT INTO mv_histogram
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+
+ANALYZE mv_histogram;
+
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+
+TRUNCATE mv_histogram;
+
+-- a => b, a => c
+INSERT INTO mv_histogram
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE mv_histogram;
+
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+
+TRUNCATE mv_histogram;
+
+-- check explain (expect bitmap index scan, not plain index scan)
+INSERT INTO mv_histogram
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX hist_idx ON mv_histogram (a, b);
+ANALYZE mv_histogram;
+
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mv_histogram WHERE a = 10 AND b = 5;
+
+DROP TABLE mv_histogram;
+
+-- varlena type (text)
+CREATE TABLE mv_histogram (
+ a TEXT,
+ b TEXT,
+ c TEXT
+);
+
+CREATE STATISTICS s8 ON mv_histogram (a, b, c) WITH (histogram);
+
+-- random data (no functional dependencies)
+INSERT INTO mv_histogram
+ SELECT mod(i, 111), mod(i, 123), mod(i, 23) FROM generate_series(1,10000) s(i);
+
+ANALYZE mv_histogram;
+
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+
+TRUNCATE mv_histogram;
+
+-- a => b, a => c, b => c
+INSERT INTO mv_histogram
+ SELECT i/10, i/100, i/200 FROM generate_series(1,10000) s(i);
+
+ANALYZE mv_histogram;
+
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+
+TRUNCATE mv_histogram;
+
+-- a => b, a => c
+INSERT INTO mv_histogram
+ SELECT i/10, i/150, i/200 FROM generate_series(1,10000) s(i);
+ANALYZE mv_histogram;
+
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+
+TRUNCATE mv_histogram;
+
+-- check explain (expect bitmap index scan, not plain index scan)
+INSERT INTO mv_histogram
+ SELECT i/10000, i/20000, i/40000 FROM generate_series(1,1000000) s(i);
+CREATE INDEX hist_idx ON mv_histogram (a, b);
+ANALYZE mv_histogram;
+
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mv_histogram WHERE a = '10' AND b = '5';
+
+TRUNCATE mv_histogram;
+
+-- check explain (expect bitmap index scan, not plain index scan) with NULLs
+INSERT INTO mv_histogram
+ SELECT
+ (CASE WHEN i/10000 = 0 THEN NULL ELSE i/10000 END),
+ (CASE WHEN i/20000 = 0 THEN NULL ELSE i/20000 END),
+ (CASE WHEN i/40000 = 0 THEN NULL ELSE i/40000 END)
+ FROM generate_series(1,1000000) s(i);
+ANALYZE mv_histogram;
+
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+
+EXPLAIN (COSTS off)
+ SELECT * FROM mv_histogram WHERE a IS NULL AND b IS NULL;
+
+DROP TABLE mv_histogram;
+
+-- NULL values (mix of int and text columns)
+CREATE TABLE mv_histogram (
+ a INT,
+ b TEXT,
+ c INT,
+ d TEXT
+);
+
+CREATE STATISTICS s9 ON mv_histogram (a, b, c, d) WITH (histogram);
+
+INSERT INTO mv_histogram
+ SELECT
+ mod(i, 100),
+ (CASE WHEN mod(i, 200) = 0 THEN NULL ELSE mod(i,200) END),
+ mod(i, 400),
+ (CASE WHEN mod(i, 300) = 0 THEN NULL ELSE mod(i,600) END)
+ FROM generate_series(1,10000) s(i);
+
+ANALYZE mv_histogram;
+
+SELECT hist_enabled, hist_built
+ FROM pg_mv_statistic WHERE starelid = 'mv_histogram'::regclass;
+
+DROP TABLE mv_histogram;
--
2.5.0
[text/x-patch] 0006-multi-statistics-estimation.patch (98.8K, ../../7aa6145f-302b-2077-b961-702cc61710bb@2ndquadrant.com/7-0006-multi-statistics-estimation.patch)
download | inline diff:
From 6a965d339eca0b8573dbc709dc30eb9ac3c95e02 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@pgaddict.com>
Date: Fri, 6 Feb 2015 01:42:38 +0100
Subject: [PATCH 6/9] multi-statistics estimation
The general idea is that a probability (which is what selectivity is)
can be split into a product of conditional probabilities like this:
P(A & B & C) = P(A & B) * P(C|A & B)
If we assume that C and B are independent, the last part may be
simplified like this
P(A & B & C) = P(A & B) * P(C|A)
we only need probabilities on [A,B] and [C,A] to compute the original
probability.
The implementation works in the other direction, though. We know what
probability P(A & B & C) we need to compute, and also what statistics
are available.
So we search for a combinations of statistics, covering the clauses in
an optimal way (most clauses covered, most dependencies exploited).
There are two possible approaches - exhaustive and greedy. The
exhaustive one walks through all permutations of stats using dynamic
programming, so it's guaranteed to find the optimal solution, but it
soon gets very slow as it's roughly O(N!). The dynamic programming may
improve that a bit, but it's still far too expensive for large numbers
of statistics (on a single table).
The greedy algorithm is very simple - in every step choose the best
solution. That may not guarantee the best solution globally (but maybe
it does?), but it only needs N steps to find the solution, so it's very
fast (processing the selected stats is usually way more expensive).
There's a GUC for selecting the search algorithm
mvstat_search = {'greedy', 'exhaustive'}
The default value is 'greedy' as that's much safer (with respect to
runtime). See choose_mv_statistics().
Once we have found a sequence of statistics, we apply them to the
clauses using the conditional probabilities. We process the selected
stats one by one, and for each we select the estimated clauses and
conditions. See clauselist_selectivity() for more details.
Limitations
-----------
It's still true that each clause at a given level has to be covered by
a single MV statistics. So with this query
WHERE (clause1) AND (clause2) AND (clause3 OR clause4)
each parenthesized clause has to be covered by a single multivariate
statistics.
Clauses not covered by a single statistics at this level will be passed
to clause_selectivity() but this will treat them as a collection of
simpler clauses (connected by AND or OR), and the clauses from the
previous level will be used as conditions.
So using the same example, the last clause will be passed to
clause_selectivity() with 'clause1' and 'clause2' as conditions, and it
will be processed using multivariate stats if possible.
The other limitation is that all the expressions have to be
mv-compatible, i.e. there can't be a mix of expressions. If this is
violated, the clause may be passed to the next level (just like with
list of clauses not covered by a single statistics), which splits that
into clauses handled by multivariate stats and clauses handler by
regular statistics.
rework clauselist_selectivity_or to handle OR-clauses correctly
We might invent a completely new set of functions here, resembling
clauselist_selectivity but adapting the ideas to OR-clauses.
But luckily we know that each OR-clause
(a OR b OR c)
may be rewritten as an equivalent AND-clause using negation:
NOT ((NOT a) AND (NOT b) AND (NOT c))
And that's something we can pass to clauselist_selectivity.
---
contrib/file_fdw/file_fdw.c | 3 +-
contrib/postgres_fdw/postgres_fdw.c | 11 +-
src/backend/optimizer/path/clausesel.c | 2030 ++++++++++++++++++++++++++------
src/backend/optimizer/path/costsize.c | 23 +-
src/backend/optimizer/util/orclauses.c | 4 +-
src/backend/utils/adt/selfuncs.c | 17 +-
src/backend/utils/misc/guc.c | 20 +
src/backend/utils/mvstats/README.stats | 166 +++
src/include/optimizer/cost.h | 6 +-
src/include/utils/mvstats.h | 8 +
10 files changed, 1916 insertions(+), 372 deletions(-)
diff --git a/contrib/file_fdw/file_fdw.c b/contrib/file_fdw/file_fdw.c
index dc035d7..8f11b7a 100644
--- a/contrib/file_fdw/file_fdw.c
+++ b/contrib/file_fdw/file_fdw.c
@@ -969,7 +969,8 @@ estimate_size(PlannerInfo *root, RelOptInfo *baserel,
baserel->baserestrictinfo,
0,
JOIN_INNER,
- NULL);
+ NULL,
+ NIL);
nrows = clamp_row_est(nrows);
diff --git a/contrib/postgres_fdw/postgres_fdw.c b/contrib/postgres_fdw/postgres_fdw.c
index aa745f2..d89f9e3 100644
--- a/contrib/postgres_fdw/postgres_fdw.c
+++ b/contrib/postgres_fdw/postgres_fdw.c
@@ -500,7 +500,8 @@ postgresGetForeignRelSize(PlannerInfo *root,
fpinfo->local_conds,
baserel->relid,
JOIN_INNER,
- NULL);
+ NULL,
+ NIL);
cost_qual_eval(&fpinfo->local_conds_cost, fpinfo->local_conds, root);
@@ -2136,7 +2137,8 @@ estimate_path_cost_size(PlannerInfo *root,
local_param_join_conds,
foreignrel->relid,
JOIN_INNER,
- NULL);
+ NULL,
+ NIL);
local_sel *= fpinfo->local_conds_sel;
rows = clamp_row_est(rows * local_sel);
@@ -3661,7 +3663,8 @@ postgresGetForeignJoinPaths(PlannerInfo *root,
fpinfo->local_conds,
0,
JOIN_INNER,
- NULL);
+ NULL,
+ NIL);
cost_qual_eval(&fpinfo->local_conds_cost, fpinfo->local_conds, root);
/*
@@ -3680,7 +3683,7 @@ postgresGetForeignJoinPaths(PlannerInfo *root,
*/
fpinfo->joinclause_sel = clauselist_selectivity(root, fpinfo->joinclauses,
0, fpinfo->jointype,
- extra->sjinfo);
+ extra->sjinfo, NIL);
}
fpinfo->server = GetForeignServer(joinrel->serverid);
diff --git a/src/backend/optimizer/path/clausesel.c b/src/backend/optimizer/path/clausesel.c
index fe96a73..14e3444 100644
--- a/src/backend/optimizer/path/clausesel.c
+++ b/src/backend/optimizer/path/clausesel.c
@@ -29,6 +29,8 @@
#include "utils/selfuncs.h"
#include "utils/typcache.h"
+#include "miscadmin.h"
+
/*
* Data structure for accumulating info about possible range-query
@@ -44,6 +46,13 @@ typedef struct RangeQueryClause
Selectivity hibound; /* Selectivity of a var < something clause */
} RangeQueryClause;
+static Selectivity clauselist_selectivity_or(PlannerInfo *root,
+ List *clauses,
+ int varRelid,
+ JoinType jointype,
+ SpecialJoinInfo *sjinfo,
+ List *conditions);
+
static void addRangeClause(RangeQueryClause **rqlist, Node *clause,
bool varonleft, bool isLTsel, Selectivity s2);
@@ -60,23 +69,25 @@ static int count_mv_attnums(List *clauses, Index relid, int type);
static int count_varnos(List *clauses, Index *relid);
+static List *clauses_matching_statistic(List **clauses, MVStatisticInfo *statistic,
+ Index relid, int types, bool remove);
+
static List *clauselist_apply_dependencies(PlannerInfo *root, List *clauses,
Index relid, List *stats);
-static MVStatisticInfo *choose_mv_statistics(List *mvstats, Bitmapset *attnums);
-
-static List *clauselist_mv_split(PlannerInfo *root, Index relid,
- List *clauses, List **mvclauses,
- MVStatisticInfo *mvstats, int types);
-
static Selectivity clauselist_mv_selectivity(PlannerInfo *root,
- List *clauses, MVStatisticInfo *mvstats);
+ MVStatisticInfo *mvstats, List *clauses,
+ List *conditions, bool is_or);
static Selectivity clauselist_mv_selectivity_mcvlist(PlannerInfo *root,
- List *clauses, MVStatisticInfo *mvstats,
- bool *fullmatch, Selectivity *lowsel);
+ MVStatisticInfo *mvstats,
+ List *clauses, List *conditions,
+ bool is_or, bool *fullmatch,
+ Selectivity *lowsel);
static Selectivity clauselist_mv_selectivity_histogram(PlannerInfo *root,
- List *clauses, MVStatisticInfo *mvstats);
+ MVStatisticInfo *mvstats,
+ List *clauses, List *conditions,
+ bool is_or);
static int update_match_bitmap_mcvlist(PlannerInfo *root, List *clauses,
int2vector *stakeys, MCVList mcvlist,
@@ -90,12 +101,33 @@ static int update_match_bitmap_histogram(PlannerInfo *root, List *clauses,
int nmatches, char * matches,
bool is_or);
+/*
+ * Describes a combination of multiple statistics to cover attributes
+ * referenced by the clauses. The array 'stats' (with nstats elements)
+ * lists attributes (in the order as they are applied), and number of
+ * clause attributes covered by this solution.
+ *
+ * choose_mv_statistics_exhaustive() uses this to track both the current
+ * and the best solutions, while walking through the state of possible
+ * combination.
+ */
+typedef struct mv_solution_t {
+ int nclauses; /* number of clauses covered */
+ int nconditions; /* number of conditions covered */
+ int nstats; /* number of stats applied */
+ int *stats; /* stats (in the apply order) */
+} mv_solution_t;
+
+static List *choose_mv_statistics(PlannerInfo *root, Index relid,
+ List *mvstats, List *clauses, List *conditions);
+
static bool has_stats(List *stats, int type);
static List * find_stats(PlannerInfo *root, Index relid);
static bool stats_type_matches(MVStatisticInfo *stat, int type);
+int mvstat_search_type = MVSTAT_SEARCH_GREEDY;
/* used for merging bitmaps - AND (min), OR (max) */
#define MAX(x, y) (((x) > (y)) ? (x) : (y))
@@ -170,14 +202,15 @@ clauselist_selectivity(PlannerInfo *root,
List *clauses,
int varRelid,
JoinType jointype,
- SpecialJoinInfo *sjinfo)
+ SpecialJoinInfo *sjinfo,
+ List *conditions)
{
Selectivity s1 = 1.0;
RangeQueryClause *rqlist = NULL;
ListCell *l;
/* processing mv stats */
- Oid relid = InvalidOid;
+ Index relid = InvalidOid;
/* list of multivariate stats on the relation */
List *stats = NIL;
@@ -193,12 +226,13 @@ clauselist_selectivity(PlannerInfo *root,
stats = find_stats(root, relid);
/*
- * If there's exactly one clause, then no use in trying to match up pairs,
- * so just go directly to clause_selectivity().
+ * If there's exactly one clause, then no use in trying to match up
+ * pairs, or matching multivariate statistics, so just go directly
+ * to clause_selectivity().
*/
if (list_length(clauses) == 1)
return clause_selectivity(root, (Node *) linitial(clauses),
- varRelid, jointype, sjinfo);
+ varRelid, jointype, sjinfo, conditions);
/*
* Apply functional dependencies, but first check that there are some stats
@@ -230,31 +264,100 @@ clauselist_selectivity(PlannerInfo *root,
(count_mv_attnums(clauses, relid,
MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST) >= 2))
{
- /* collect attributes from the compatible conditions */
- Bitmapset *mvattnums = collect_mv_attnums(clauses, relid,
- MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST);
+ ListCell *s;
+
+ /*
+ * Copy the conditions we got from the upper part of the expression tree
+ * so that we can add local conditions to it (we need to keep the
+ * original list intact, for sibling expressions - other expressions
+ * at the same level).
+ */
+ List *conditions_local = list_copy(conditions);
+
+ /* find the best combination of statistics */
+ List *solution = choose_mv_statistics(root, relid, stats,
+ clauses, conditions);
- /* and search for the statistic covering the most attributes */
- MVStatisticInfo *mvstat = choose_mv_statistics(stats, mvattnums);
+ /* FIXME we must not scribble over the original list */
+ if (solution)
+ clauses = list_copy(clauses);
- if (mvstat != NULL) /* we have a matching stats */
+ /*
+ * We have a good solution, which is merely a list of statistics that
+ * we need to apply. We'll apply the statistics one by one (in the order
+ * as they appear in the list), and for each statistic we'll
+ *
+ * (1) find clauses compatible with the statistic (and remove them
+ * from the list)
+ *
+ * (2) find local conditions compatible with the statistic
+ *
+ * (3) do the estimation P(clauses | conditions)
+ *
+ * (4) append the estimated clauses to local conditions
+ *
+ * continuously modify
+ */
+ foreach (s, solution)
{
- /* clauses compatible with multi-variate stats */
- List *mvclauses = NIL;
+ MVStatisticInfo *mvstat = (MVStatisticInfo *)lfirst(s);
- /* split the clauselist into regular and mv-clauses */
- clauses = clauselist_mv_split(root, relid, clauses, &mvclauses,
- mvstat, MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST);
+ /* clauses compatible with the statistic we're applying right now */
+ List *stat_clauses = NIL;
+ List *stat_conditions = NIL;
- /* we've chosen the histogram to match the clauses */
- Assert(mvclauses != NIL);
+ /*
+ * Find clauses and conditions matching the statistic - the clauses
+ * need to be removed from the list, while conditions should remain
+ * there (so that we can apply them repeatedly).
+ */
+ stat_clauses
+ = clauses_matching_statistic(&clauses, mvstat, relid,
+ MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST,
+ true);
+
+ stat_conditions
+ = clauses_matching_statistic(&conditions_local, mvstat, relid,
+ MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST,
+ false);
+
+ /*
+ * If we got no clauses to estimate, we've done something wrong,
+ * either during the optimization, detecting compatible clause, or
+ * somewhere else.
+ *
+ * Also, we need at least two attributes in clauses and conditions.
+ */
+ Assert(stat_clauses != NIL);
+ Assert(count_mv_attnums(list_union(stat_clauses, stat_conditions),
+ relid, MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST) >= 2);
/* compute the multivariate stats */
- s1 *= clauselist_mv_selectivity(root, mvclauses, mvstat);
+ s1 *= clauselist_mv_selectivity(root, mvstat,
+ stat_clauses, stat_conditions,
+ false); /* AND */
+
+ /*
+ * Add the new clauses to the local conditions, so that we can use
+ * them for the subsequent statistics. We only add the clauses,
+ * because the conditions are already there (or should be).
+ */
+ conditions_local = list_concat(conditions_local, stat_clauses);
}
+
+ /* from now on, work only with the 'local' list of conditions */
+ conditions = conditions_local;
}
/*
+ * If there's exactly one clause, then no use in trying to match up
+ * pairs, so just go directly to clause_selectivity().
+ */
+ if (list_length(clauses) == 1)
+ return s1 * clause_selectivity(root, (Node *) linitial(clauses),
+ varRelid, jointype, sjinfo, conditions);
+
+ /*
* Initial scan over clauses. Anything that doesn't look like a potential
* rangequery clause gets multiplied into s1 and forgotten. Anything that
* does gets inserted into an rqlist entry.
@@ -266,7 +369,8 @@ clauselist_selectivity(PlannerInfo *root,
Selectivity s2;
/* Always compute the selectivity using clause_selectivity */
- s2 = clause_selectivity(root, clause, varRelid, jointype, sjinfo);
+ s2 = clause_selectivity(root, clause, varRelid, jointype, sjinfo,
+ conditions);
/*
* Check for being passed a RestrictInfo.
@@ -425,6 +529,55 @@ clauselist_selectivity(PlannerInfo *root,
}
/*
+ * Similar to clauselist_selectivity(), but for OR-clauses. We can't simply use
+ * the same multi-statistic estimation logic for AND-clauses, at least not
+ * directly, because there are a few key differences:
+ *
+ * - functional dependencies don't really apply to OR-clauses
+ *
+ * - clauselist_selectivity() is based on decomposing the selectivity into
+ * a sequence of conditional probabilities (selectivities), but that can
+ * be done only for AND-clauses
+ *
+ * We might invent a similar infrastructure for optimizing OR-clauses, doing
+ * something similar to what clause_selectivity does for AND-clauses, but
+ * luckily we know that each disjunctive normal form (aka OR-clause)
+ *
+ * (a OR b OR c)
+ *
+ * may be rewritten as an equivalent conjunctive normal form (aka AND-clause)
+ * by using negation:
+ *
+ * NOT ((NOT a) AND (NOT b) AND (NOT c))
+ *
+ * And that's something we can pass to clauselist_selectivity and let it do
+ * all the heavy lifting.
+ */
+static Selectivity
+clauselist_selectivity_or(PlannerInfo *root,
+ List *clauses,
+ int varRelid,
+ JoinType jointype,
+ SpecialJoinInfo *sjinfo,
+ List *conditions)
+{
+ List *args = NIL;
+ ListCell *l;
+ Expr *expr;
+
+ /* build arguments for the AND-clause by negating args of the OR-clause */
+ foreach (l, clauses)
+ args = lappend(args, makeBoolExpr(NOT_EXPR, list_make1(lfirst(l)), -1));
+
+ /* and then the actual OR-clause on the negated args */
+ expr = makeBoolExpr(AND_EXPR, args, -1);
+
+ /* instead of constructing NOT expression, just do (1.0 - s) */
+ return 1.0 - clauselist_selectivity(root, list_make1(expr), varRelid,
+ jointype, sjinfo, conditions);
+}
+
+/*
* addRangeClause --- add a new range clause for clauselist_selectivity
*
* Here is where we try to match up pairs of range-query clauses
@@ -631,7 +784,8 @@ clause_selectivity(PlannerInfo *root,
Node *clause,
int varRelid,
JoinType jointype,
- SpecialJoinInfo *sjinfo)
+ SpecialJoinInfo *sjinfo,
+ List *conditions)
{
Selectivity s1 = 0.5; /* default for any unhandled clause type */
RestrictInfo *rinfo = NULL;
@@ -751,7 +905,8 @@ clause_selectivity(PlannerInfo *root,
(Node *) get_notclausearg((Expr *) clause),
varRelid,
jointype,
- sjinfo);
+ sjinfo,
+ conditions);
}
else if (and_clause(clause))
{
@@ -760,29 +915,18 @@ clause_selectivity(PlannerInfo *root,
((BoolExpr *) clause)->args,
varRelid,
jointype,
- sjinfo);
+ sjinfo,
+ conditions);
}
else if (or_clause(clause))
{
- /*
- * Selectivities for an OR clause are computed as s1+s2 - s1*s2 to
- * account for the probable overlap of selected tuple sets.
- *
- * XXX is this too conservative?
- */
- ListCell *arg;
-
- s1 = 0.0;
- foreach(arg, ((BoolExpr *) clause)->args)
- {
- Selectivity s2 = clause_selectivity(root,
- (Node *) lfirst(arg),
- varRelid,
- jointype,
- sjinfo);
-
- s1 = s1 + s2 - s1 * s2;
- }
+ /* just call to clauselist_selectivity_or() */
+ s1 = clauselist_selectivity_or(root,
+ ((BoolExpr *) clause)->args,
+ varRelid,
+ jointype,
+ sjinfo,
+ conditions);
}
else if (is_opclause(clause) || IsA(clause, DistinctExpr))
{
@@ -872,7 +1016,8 @@ clause_selectivity(PlannerInfo *root,
(Node *) ((RelabelType *) clause)->arg,
varRelid,
jointype,
- sjinfo);
+ sjinfo,
+ conditions);
}
else if (IsA(clause, CoerceToDomain))
{
@@ -881,7 +1026,8 @@ clause_selectivity(PlannerInfo *root,
(Node *) ((CoerceToDomain *) clause)->arg,
varRelid,
jointype,
- sjinfo);
+ sjinfo,
+ conditions);
}
else
{
@@ -945,300 +1091,1395 @@ clause_selectivity(PlannerInfo *root,
* in the MCV list, then the selectivity is below the lowest frequency
* found in the MCV list,
*
- * TODO When applying the clauses to the histogram/MCV list, we can do
- * that from the most selective clauses first, because that'll
- * eliminate the buckets/items sooner (so we'll be able to skip
- * them without inspection, which is more expensive). But this
- * requires really knowing the per-clause selectivities in advance,
- * and that's not what we do now.
+ * TODO When applying the clauses to the histogram/MCV list, we can do that from
+ * the most selective clauses first, because that'll eliminate the
+ * buckets/items sooner (so we'll be able to skip them without inspection,
+ * which is more expensive). But this requires really knowing the
+ * per-clause selectivities in advance, and that's not what we do now.
+ *
*/
static Selectivity
-clauselist_mv_selectivity(PlannerInfo *root, List *clauses, MVStatisticInfo *mvstats)
+clauselist_mv_selectivity(PlannerInfo *root, MVStatisticInfo *mvstats,
+ List *clauses, List *conditions, bool is_or)
{
bool fullmatch = false;
Selectivity s1 = 0.0, s2 = 0.0;
- /*
- * Lowest frequency in the MCV list (may be used as an upper bound
- * for full equality conditions that did not match any MCV item).
- */
- Selectivity mcv_low = 0.0;
+ /*
+ * Lowest frequency in the MCV list (may be used as an upper bound
+ * for full equality conditions that did not match any MCV item).
+ */
+ Selectivity mcv_low = 0.0;
+
+ /* TODO Evaluate simple 1D selectivities, use the smallest one as
+ * an upper bound, product as lower bound, and sort the
+ * clauses in ascending order by selectivity (to optimize the
+ * MCV/histogram evaluation).
+ */
+
+ /* Evaluate the MCV first. */
+ s1 = clauselist_mv_selectivity_mcvlist(root, mvstats,
+ clauses, conditions, is_or,
+ &fullmatch, &mcv_low);
+
+ /*
+ * If we got a full equality match on the MCV list, we're done (and
+ * the estimate is pretty good).
+ */
+ if (fullmatch && (s1 > 0.0))
+ return s1;
+
+ /* TODO if (fullmatch) without matching MCV item, use the mcv_low
+ * selectivity as upper bound */
+
+ s2 = clauselist_mv_selectivity_histogram(root, mvstats,
+ clauses, conditions, is_or);
+
+ /* TODO clamp to <= 1.0 (or more strictly, when possible) */
+ return s1 + s2;
+}
+
+/*
+ * Pull varattnos from the clauses, similarly to pull_varattnos() but:
+ *
+ * (a) only get attributes for a particular relation (relid)
+ * (b) ignore system attributes (we can't build stats on them anyway)
+ *
+ * This makes it possible to directly compare the result with attnum
+ * values from pg_attribute etc.
+ */
+static Bitmapset *
+get_varattnos(Node * node, Index relid)
+{
+ int k;
+ Bitmapset *varattnos = NULL;
+ Bitmapset *result = NULL;
+
+ /* get the varattnos */
+ pull_varattnos(node, relid, &varattnos);
+
+ k = -1;
+ while ((k = bms_next_member(varattnos, k)) >= 0)
+ {
+ if (k + FirstLowInvalidHeapAttributeNumber > 0)
+ result
+ = bms_add_member(result,
+ k + FirstLowInvalidHeapAttributeNumber);
+ }
+
+ bms_free(varattnos);
+
+ return result;
+}
+
+/*
+ * Collect attributes from mv-compatible clauses.
+ */
+static Bitmapset *
+collect_mv_attnums(List *clauses, Index relid, int types)
+{
+ Bitmapset *attnums = NULL;
+ ListCell *l;
+
+ /*
+ * Walk through the clauses and identify the ones we can estimate
+ * using multivariate stats, and remember the relid/columns. We'll
+ * then cross-check if we have suitable stats, and only if needed
+ * we'll split the clauses into multivariate and regular lists.
+ *
+ * For now we're only interested in RestrictInfo nodes with nested
+ * OpExpr, using either a range or equality.
+ */
+ foreach (l, clauses)
+ {
+ Node *clause = (Node *) lfirst(l);
+
+ /* ignore the result here - we only need the attnums */
+ clause_is_mv_compatible(clause, relid, &attnums, types);
+ }
+
+ /*
+ * If there are not at least two attributes referenced by the clause(s),
+ * we can throw everything out (as we'll revert to simple stats).
+ */
+ if (bms_num_members(attnums) <= 1)
+ {
+ bms_free(attnums);
+ attnums = NULL;
+ }
+
+ return attnums;
+}
+
+/*
+ * Count the number of attributes in clauses compatible with multivariate stats.
+ */
+static int
+count_mv_attnums(List *clauses, Index relid, int type)
+{
+ int c;
+ Bitmapset *attnums = collect_mv_attnums(clauses, relid, type);
+
+ c = bms_num_members(attnums);
+
+ bms_free(attnums);
+
+ return c;
+}
+
+/*
+ * Count varnos referenced in the clauses, and if there's a single varno then
+ * return the index in 'relid'.
+ */
+static int
+count_varnos(List *clauses, Index *relid)
+{
+ int cnt;
+ Bitmapset *varnos = NULL;
+
+ varnos = pull_varnos((Node *) clauses);
+ cnt = bms_num_members(varnos);
+
+ /* if there's a single varno in the clauses, remember it */
+ if (bms_num_members(varnos) == 1)
+ *relid = bms_singleton_member(varnos);
+
+ bms_free(varnos);
+
+ return cnt;
+}
+
+static List *
+clauses_matching_statistic(List **clauses, MVStatisticInfo *statistic,
+ Index relid, int types, bool remove)
+{
+ int i;
+ Bitmapset *stat_attnums = NULL;
+ List *matching_clauses = NIL;
+ ListCell *lc;
+
+ /* build attnum bitmapset for this statistics */
+ for (i = 0; i < statistic->stakeys->dim1; i++)
+ stat_attnums = bms_add_member(stat_attnums,
+ statistic->stakeys->values[i]);
+
+ /*
+ * We can't use foreach here, because we may need to remove some of the
+ * clauses if (remove=true).
+ */
+ lc = list_head(*clauses);
+ while (lc)
+ {
+ Node *clause = (Node*)lfirst(lc);
+ Bitmapset *attnums = NULL;
+
+ /* must advance lc before list_delete possibly pfree's it */
+ lc = lnext(lc);
+
+ /*
+ * skip clauses that are not compatible with stats (just leave them
+ * in the original list)
+ *
+ * XXX Perhaps this should check what stats are actually available in
+ * the statistics (not a big deal now, because MCV and histograms
+ * handle the same types of conditions).
+ */
+ if (! clause_is_mv_compatible(clause, relid, &attnums, types))
+ {
+ bms_free(attnums);
+ continue;
+ }
+
+ /* if the clause is covered by the statistic, add it to the list */
+ if (bms_is_subset(attnums, stat_attnums))
+ {
+ matching_clauses = lappend(matching_clauses, clause);
+
+ /* if remove=true, remove the matching item from the main list */
+ if (remove)
+ *clauses = list_delete_ptr(*clauses, clause);
+ }
+
+ bms_free(attnums);
+ }
+
+ bms_free(stat_attnums);
+
+ return matching_clauses;
+}
+
+/*
+ * Selects the best combination of multivariate statistics, in an exhaustive
+ * way, where 'best' means:
+ *
+ * (a) covering the most attributes (referenced by clauses)
+ * (b) using the least number of multivariate stats
+ * (c) using the most conditions to exploit dependency
+ *
+ * Don't call this directly but through choose_mv_statistics(), which does some
+ * additional tricks to minimize the runtime.
+ *
+ *
+ * Algorithm
+ * ---------
+ * The algorithm is a recursive implementation of backtracking, with maximum
+ * depth equal to the number of multi-variate statistics available on the table.
+ * It actually explores all valid combinations of stats.
+ *
+ * Whenever it considers adding the next statistics, the clauses it matches are
+ * divided into 'conditions' (clauses already matched by at least one previous
+ * statistics) and clauses that are estimated.
+ *
+ * Then several checks are performed:
+ *
+ * (a) The statistics covers at least 2 columns, referenced in the estimated
+ * clauses (otherwise multi-variate stats are useless).
+ *
+ * (b) The statistics covers at least 1 new column, i.e. column not refefenced
+ * by the already used stats (and the new column has to be referenced by
+ * the clauses, of couse). Otherwise the statistics would not add any new
+ * information.
+ *
+ * There are some other sanity checks (e.g. stats must not be used twice etc.).
+ *
+ *
+ * Weaknesses
+ * ----------
+ * The current implemetation uses a rather simple optimality criteria, so it may
+ * not do the best choice when
+ *
+ * (a) There may be multiple solutions with the same number of covered
+ * attributes and number of statistics (e.g. the same solution but with
+ * statistics in a different order). It's unclear which solution in the best
+ * one - in a sense all of them are equal.
+ *
+ * TODO It might be possible to compute estimate for each of those solutions,
+ * and then combine them to get the final estimate (e.g. by using average
+ * or median).
+ *
+ * (b) Does not consider that some types of stats are a better match for some
+ * types of clauses (e.g. MCV list is generally a better match for equality
+ * conditions than a histogram).
+ *
+ * But maybe this is pointless - generally, each column is either a label
+ * (it's not important whether because of the data type or how it's used),
+ * or a value with ordering that makes sense. So either a MCV list is more
+ * appropriate (labels) or a histogram (values with orderings).
+ *
+ * Now sure what to do with statistics on columns mixing both types of data
+ * (some columns would work best with MCVs, some with histograms). Maybe we
+ * could invent a new type of statistics combining MCV list and histogram
+ * (keeping a small histogram for each MCV item, and a separate histogram
+ * for values not on the MCV list).
+ *
+ * TODO The algorithm should probably count number of Vars (not just attnums)
+ * when computing the 'score' of each solution. Computing the ratio of
+ * (num of all vars) / (num of condition vars) as a measure of how well
+ * the solution uses conditions might be useful.
+ */
+static void
+choose_mv_statistics_exhaustive(PlannerInfo *root, int step,
+ int nmvstats, MVStatisticInfo *mvstats, Bitmapset ** stats_attnums,
+ int nclauses, Node ** clauses, Bitmapset ** clauses_attnums,
+ int nconditions, Node ** conditions, Bitmapset ** conditions_attnums,
+ bool *cover_map, bool *condition_map, int *ruled_out,
+ mv_solution_t *current, mv_solution_t **best)
+{
+ int i, j;
+
+ Assert(best != NULL);
+ Assert((step == 0 && current == NULL) || (step > 0 && current != NULL));
+
+ /* this may run for a long sime, so let's make it interruptible */
+ CHECK_FOR_INTERRUPTS();
+
+ if (current == NULL)
+ {
+ current = (mv_solution_t*)palloc0(sizeof(mv_solution_t));
+ current->stats = (int*)palloc0(sizeof(int)*nmvstats);
+ current->nstats = 0;
+ current->nclauses = 0;
+ current->nconditions = 0;
+ }
+
+ /*
+ * Now try to apply each statistics, matching at least two attributes,
+ * unless it's already used in one of the previous steps.
+ */
+ for (i = 0; i < nmvstats; i++)
+ {
+ int c;
+
+ int ncovered_clauses = 0; /* number of covered clauses */
+ int ncovered_conditions = 0; /* number of covered conditions */
+ int nattnums = 0; /* number of covered attributes */
+
+ Bitmapset *all_attnums = NULL;
+
+ /* skip statistics that were already used or eliminated */
+ if (ruled_out[i] != -1)
+ continue;
+
+ /*
+ * See if we have clauses covered by this statistics, but not
+ * yet covered by any of the preceding onces.
+ */
+ for (c = 0; c < nclauses; c++)
+ {
+ bool covered = false;
+ Bitmapset *clause_attnums = clauses_attnums[c];
+ Bitmapset *tmp = NULL;
+
+ /*
+ * If this clause is not covered by this stats, we can't
+ * use the stats to estimate that at all.
+ */
+ if (! cover_map[i * nclauses + c])
+ continue;
+
+ /*
+ * Now we know we'll use this clause - either as a condition
+ * or as a new clause (the estimated one). So let's add the
+ * attributes to the attnums from all the clauses usable with
+ * this statistics.
+ */
+ tmp = bms_union(all_attnums, clause_attnums);
+
+ /* free the old bitmap */
+ bms_free(all_attnums);
+ all_attnums = tmp;
+
+ /* let's see if it's covered by any of the previous stats */
+ for (j = 0; j < step; j++)
+ {
+ /* already covered by the previous stats */
+ if (cover_map[current->stats[j] * nclauses + c])
+ covered = true;
+
+ if (covered)
+ break;
+ }
+
+ /* if already covered, continue with the next clause */
+ if (covered)
+ {
+ ncovered_conditions += 1;
+ continue;
+ }
+
+ /*
+ * OK, this clause is covered by this statistics (and not by
+ * any of the previous ones)
+ */
+ ncovered_clauses += 1;
+ }
+
+ /* can't have more new clauses than original clauses */
+ Assert(nclauses >= ncovered_clauses);
+ Assert(ncovered_clauses >= 0); /* mostly paranoia */
+
+ nattnums = bms_num_members(all_attnums);
+
+ /* free all the bitmapsets - we don't need them anymore */
+ bms_free(all_attnums);
+
+ all_attnums = NULL;
+
+ /*
+ * See if we have clauses covered by this statistics, but not
+ * yet covered by any of the preceding onces.
+ */
+ for (c = 0; c < nconditions; c++)
+ {
+ Bitmapset *clause_attnums = conditions_attnums[c];
+ Bitmapset *tmp = NULL;
+
+ /*
+ * If this clause is not covered by this stats, we can't
+ * use the stats to estimate that at all.
+ */
+ if (! condition_map[i * nconditions + c])
+ continue;
+
+ /* count this as a condition */
+ ncovered_conditions += 1;
+
+ /*
+ * Now we know we'll use this clause - either as a condition
+ * or as a new clause (the estimated one). So let's add the
+ * attributes to the attnums from all the clauses usable with
+ * this statistics.
+ */
+ tmp = bms_union(all_attnums, clause_attnums);
+
+ /* free the old bitmap */
+ bms_free(all_attnums);
+ all_attnums = tmp;
+ }
+
+ /*
+ * Let's mark the statistics as 'ruled out' - either we'll use
+ * it (and proceed to the next step), or it's incompatible.
+ */
+ ruled_out[i] = step;
+
+ /*
+ * There are no clauses usable with this statistics (not already
+ * covered by aome of the previous stats).
+ *
+ * Similarly, if the clauses only use a single attribute, we
+ * can't really use that.
+ */
+ if ((ncovered_clauses == 0) || (nattnums < 2))
+ continue;
+
+ /*
+ * TODO Not sure if it's possible to add a clause referencing
+ * only attributes already covered by previous stats?
+ * Introducing only some new dependency, not a new
+ * attribute. Couldn't come up with an example, though.
+ * Might be worth adding some assert.
+ */
+
+ /*
+ * got a suitable statistics - let's update the current solution,
+ * maybe use it as the best solution
+ */
+ current->nclauses += ncovered_clauses;
+ current->nconditions += ncovered_conditions;
+ current->nstats += 1;
+ current->stats[step] = i;
+
+ /*
+ * We can never cover more clauses, or use more stats that we
+ * actually have at the beginning.
+ */
+ Assert(nclauses >= current->nclauses);
+ Assert(nmvstats >= current->nstats);
+ Assert(step < nmvstats);
+
+ if (*best == NULL)
+ {
+ *best = (mv_solution_t*)palloc0(sizeof(mv_solution_t));
+ (*best)->stats = (int*)palloc0(sizeof(int)*nmvstats);
+ (*best)->nstats = 0;
+ (*best)->nclauses = 0;
+ (*best)->nconditions = 0;
+ }
+
+ /* see if it's better than the current 'best' solution */
+ if ((current->nclauses > (*best)->nclauses) ||
+ ((current->nclauses == (*best)->nclauses) &&
+ ((current->nstats > (*best)->nstats))))
+ {
+ (*best)->nstats = current->nstats;
+ (*best)->nclauses = current->nclauses;
+ (*best)->nconditions = current->nconditions;
+ memcpy((*best)->stats, current->stats, nmvstats * sizeof(int));
+ }
+
+ /*
+ * The recursion only makes sense if we haven't covered all the
+ * attributes (then adding stats is not really possible).
+ */
+ if ((step + 1) < nmvstats)
+ choose_mv_statistics_exhaustive(root, step+1,
+ nmvstats, mvstats, stats_attnums,
+ nclauses, clauses, clauses_attnums,
+ nconditions, conditions, conditions_attnums,
+ cover_map, condition_map, ruled_out,
+ current, best);
+
+ /* reset the last step */
+ current->nclauses -= ncovered_clauses;
+ current->nconditions -= ncovered_conditions;
+ current->nstats -= 1;
+ current->stats[step] = 0;
+
+ /* mark the statistics as usable again */
+ ruled_out[i] = -1;
+
+ Assert(current->nclauses >= 0);
+ Assert(current->nstats >= 0);
+ }
+
+ /* reset all statistics as 'incompatible' in this step */
+ for (i = 0; i < nmvstats; i++)
+ if (ruled_out[i] == step)
+ ruled_out[i] = -1;
+
+}
+
+/*
+ * Greedy search for a multivariate solution - a sequence of statistics covering
+ * the clauses. This chooses the "best" statistics at each step, so the
+ * resulting solution may not be the best solution globally, but this produces
+ * the solution in only N steps (where N is the number of statistics), while
+ * the exhaustive approach may have to walk through ~N! combinations (although
+ * some of those are terminated early).
+ *
+ * See the comments at choose_mv_statistics_exhaustive() as this does the same
+ * thing (but in a different way).
+ *
+ * Don't call this directly, but through choose_mv_statistics().
+ *
+ * TODO There are probably other metrics we might use - e.g. using number of
+ * columns (num_cond_columns / num_cov_columns), which might work better
+ * with a mix of simple and complex clauses.
+ *
+ * TODO Also the choice at the very first step should be handled in a special
+ * way, because there will be 0 conditions at that moment, so there needs
+ * to be some other criteria - e.g. using the simplest (or most complex?)
+ * clause might be a good idea.
+ *
+ * TODO We might also select multiple stats using different criteria, and branch
+ * the search. This is however tricky, because if we choose k statistics at
+ * each step, we get k^N branches to walk through (with N steps). That's
+ * not really good with large number of stats (yet better than exhaustive
+ * search).
+ */
+static void
+choose_mv_statistics_greedy(PlannerInfo *root, int step,
+ int nmvstats, MVStatisticInfo *mvstats, Bitmapset ** stats_attnums,
+ int nclauses, Node ** clauses, Bitmapset ** clauses_attnums,
+ int nconditions, Node ** conditions, Bitmapset ** conditions_attnums,
+ bool *cover_map, bool *condition_map, int *ruled_out,
+ mv_solution_t *current, mv_solution_t **best)
+{
+ int i, j;
+ int best_stat = -1;
+ double gain, max_gain = -1.0;
+
+ /*
+ * Bitmap tracking which clauses are already covered (by the previous
+ * statistics) and may thus serve only as a condition in this step.
+ */
+ bool *covered_clauses = (bool*)palloc0(nclauses);
+
+ /*
+ * Number of clauses and columns covered by each statistics - this
+ * includes both conditions and clauses covered by the statistics for
+ * the first time. The number of columns may count some columns
+ * repeatedly - if a column is shared by multiple clauses, it will
+ * be counted once for each clause (covered by the statistics).
+ * So with two clauses [(a=1 OR b=2),(a<2 OR c>1)] the column "a"
+ * will be counted twice (if both clauses are covered).
+ *
+ * The values for reduded statistics (that can't be applied) are
+ * not computed, because that'd be pointless.
+ */
+ int *num_cov_clauses = (int*)palloc0(sizeof(int) * nmvstats);
+ int *num_cov_columns = (int*)palloc0(sizeof(int) * nmvstats);
+
+ /*
+ * Same as above, but this only includes clauses that are already
+ * covered by the previous stats (and the current one).
+ */
+ int *num_cond_clauses = (int*)palloc0(sizeof(int) * nmvstats);
+ int *num_cond_columns = (int*)palloc0(sizeof(int) * nmvstats);
+
+ /*
+ * Number of attributes for each clause.
+ *
+ * TODO Might be computed in choose_mv_statistics() and then passed
+ * here, but then the function would not have the same signature
+ * as _exhaustive().
+ */
+ int *attnum_counts = (int*)palloc0(sizeof(int) * nclauses);
+ int *attnum_cond_counts = (int*)palloc0(sizeof(int) * nconditions);
+
+ CHECK_FOR_INTERRUPTS();
+
+ Assert(best != NULL);
+ Assert((step == 0 && current == NULL) || (step > 0 && current != NULL));
+
+ /* compute attributes (columns) for each clause */
+ for (i = 0; i < nclauses; i++)
+ attnum_counts[i] = bms_num_members(clauses_attnums[i]);
+
+ /* compute attributes (columns) for each condition */
+ for (i = 0; i < nconditions; i++)
+ attnum_cond_counts[i] = bms_num_members(conditions_attnums[i]);
+
+ /* see which clauses are already covered at this point (by previous stats) */
+ for (i = 0; i < step; i++)
+ for (j = 0; j < nclauses; j++)
+ covered_clauses[j] |= (cover_map[current->stats[i] * nclauses + j]);
+
+ /* which remaining statistics covers most clauses / uses most conditions? */
+ for (i = 0; i < nmvstats; i++)
+ {
+ Bitmapset *attnums_covered = NULL;
+ Bitmapset *attnums_conditions = NULL;
+
+ /* skip stats that are already ruled out (either used or inapplicable) */
+ if (ruled_out[i] != -1)
+ continue;
+
+ /* count covered clauses and conditions (for the statistics) */
+ for (j = 0; j < nclauses; j++)
+ {
+ if (cover_map[i * nclauses + j])
+ {
+ Bitmapset *attnums_new
+ = bms_union(attnums_covered, clauses_attnums[j]);
+
+ /* get rid of the old bitmap and keep the unified result */
+ bms_free(attnums_covered);
+ attnums_covered = attnums_new;
+
+ num_cov_clauses[i] += 1;
+ num_cov_columns[i] += attnum_counts[j];
+
+ /* is the clause already covered (i.e. a condition)? */
+ if (covered_clauses[j])
+ {
+ num_cond_clauses[i] += 1;
+ num_cond_columns[i] += attnum_counts[j];
+ attnums_new = bms_union(attnums_conditions,
+ clauses_attnums[j]);
+
+ bms_free(attnums_conditions);
+ attnums_conditions = attnums_new;
+ }
+ }
+ }
+
+ /* if all covered clauses are covered by prev stats (thus conditions) */
+ if (num_cov_clauses[i] == num_cond_clauses[i])
+ ruled_out[i] = step;
+
+ /* same if there are no new attributes */
+ else if (bms_num_members(attnums_conditions) == bms_num_members(attnums_covered))
+ ruled_out[i] = step;
+
+ bms_free(attnums_covered);
+ bms_free(attnums_conditions);
+
+ /* if the statistics is inapplicable, try the next one */
+ if (ruled_out[i] != -1)
+ continue;
+
+ /* now let's walk through conditions and count the covered */
+ for (j = 0; j < nconditions; j++)
+ {
+ if (condition_map[i * nconditions + j])
+ {
+ num_cond_clauses[i] += 1;
+ num_cond_columns[i] += attnum_cond_counts[j];
+ }
+ }
+
+ /* otherwise see if this improves the interesting metrics */
+ gain = num_cond_columns[i] / (double)num_cov_columns[i];
+
+ if (gain > max_gain)
+ {
+ max_gain = gain;
+ best_stat = i;
+ }
+ }
+
+ /*
+ * Have we found a suitable statistics? Add it to the solution and
+ * try next step.
+ */
+ if (best_stat != -1)
+ {
+ /* mark the statistics, so that we skip it in next steps */
+ ruled_out[best_stat] = step;
+
+ /* allocate current solution if necessary */
+ if (current == NULL)
+ {
+ current = (mv_solution_t*)palloc0(sizeof(mv_solution_t));
+ current->stats = (int*)palloc0(sizeof(int)*nmvstats);
+ current->nstats = 0;
+ current->nclauses = 0;
+ current->nconditions = 0;
+ }
+
+ current->nclauses += num_cov_clauses[best_stat];
+ current->nconditions += num_cond_clauses[best_stat];
+ current->stats[step] = best_stat;
+ current->nstats++;
+
+ if (*best == NULL)
+ {
+ (*best) = (mv_solution_t*)palloc0(sizeof(mv_solution_t));
+ (*best)->nstats = current->nstats;
+ (*best)->nclauses = current->nclauses;
+ (*best)->nconditions = current->nconditions;
+
+ (*best)->stats = (int*)palloc0(sizeof(int)*nmvstats);
+ memcpy((*best)->stats, current->stats, nmvstats * sizeof(int));
+ }
+ else
+ {
+ /* see if this is a better solution */
+ double current_gain = (double)current->nconditions / current->nclauses;
+ double best_gain = (double)(*best)->nconditions / (*best)->nclauses;
+
+ if ((current_gain > best_gain) ||
+ ((current_gain == best_gain) && (current->nstats < (*best)->nstats)))
+ {
+ (*best)->nstats = current->nstats;
+ (*best)->nclauses = current->nclauses;
+ (*best)->nconditions = current->nconditions;
+ memcpy((*best)->stats, current->stats, nmvstats * sizeof(int));
+ }
+ }
+
+ /*
+ * The recursion only makes sense if we haven't covered all the
+ * attributes (then adding stats is not really possible).
+ */
+ if ((step + 1) < nmvstats)
+ choose_mv_statistics_greedy(root, step+1,
+ nmvstats, mvstats, stats_attnums,
+ nclauses, clauses, clauses_attnums,
+ nconditions, conditions, conditions_attnums,
+ cover_map, condition_map, ruled_out,
+ current, best);
+
+ /* reset the last step */
+ current->nclauses -= num_cov_clauses[best_stat];
+ current->nconditions -= num_cond_clauses[best_stat];
+ current->nstats -= 1;
+ current->stats[step] = 0;
+
+ /* mark the statistics as usable again */
+ ruled_out[best_stat] = -1;
+ }
+
+ /* reset all statistics eliminated in this step */
+ for (i = 0; i < nmvstats; i++)
+ if (ruled_out[i] == step)
+ ruled_out[i] = -1;
+
+ /* free everything allocated in this step */
+ pfree(covered_clauses);
+ pfree(attnum_counts);
+ pfree(num_cov_clauses);
+ pfree(num_cov_columns);
+ pfree(num_cond_clauses);
+ pfree(num_cond_columns);
+}
+
+/*
+ * Remove clauses not covered by any of the available statistics
+ *
+ * This helps us to reduce the amount of work done in choose_mv_statistics()
+ * by not having to deal with clauses that can't possibly be useful.
+ */
+static List *
+filter_clauses(PlannerInfo *root, Index relid, int type,
+ List *stats, List *clauses, Bitmapset **attnums)
+{
+ ListCell *c;
+ ListCell *s;
+
+ /* results (list of compatible clauses, attnums) */
+ List *rclauses = NIL;
+
+ foreach (c, clauses)
+ {
+ Node *clause = (Node*)lfirst(c);
+ Bitmapset *clause_attnums = NULL;
+
+ /*
+ * We do assume that thanks to previous checks, we should not run into
+ * clauses that are incompatible with multivariate stats here. We also
+ * need to collect the attnums for the clause.
+ *
+ * XXX Maybe turn this into an assert?
+ */
+ if (! clause_is_mv_compatible(clause, relid, &clause_attnums, type))
+ elog(ERROR, "should not get non-mv-compatible cluase");
+
+ /* Is there a multivariate statistics covering the clause? */
+ foreach (s, stats)
+ {
+ int k, matches = 0;
+ MVStatisticInfo *stat = (MVStatisticInfo *)lfirst(s);
+
+ /* skip statistics not matching the required type */
+ if (! stats_type_matches(stat, type))
+ continue;
+
+ /*
+ * see if all clause attributes are covered by the statistic
+ *
+ * We'll do that in the opposite direction, i.e. we'll see how many
+ * attributes of the statistic are referenced in the clause, and then
+ * compare the counts.
+ */
+ for (k = 0; k < stat->stakeys->dim1; k++)
+ if (bms_is_member(stat->stakeys->values[k], clause_attnums))
+ matches += 1;
+
+ /*
+ * If the number of matches is equal to attributes referenced by the
+ * clause, then the clause is covered by the statistic.
+ */
+ if (bms_num_members(clause_attnums) == matches)
+ {
+ *attnums = bms_union(*attnums, clause_attnums);
+ rclauses = lappend(rclauses, clause);
+ break;
+ }
+ }
+
+ bms_free(clause_attnums);
+ }
+
+ /* we can't have more compatible conditions than source conditions */
+ Assert(list_length(clauses) >= list_length(rclauses));
+
+ return rclauses;
+}
+
+/*
+ * Remove statistics not covering any new clauses
+ *
+ * Statistics not covering any new clauses (conditions don't count) are not
+ * really useful, so let's ignore them. Also, we need the statistics to
+ * reference at least two different attributes (both in conditions and clauses
+ * combined), and at least one of them in the clauses alone.
+ *
+ * This check might be made more strict by checking against individual clauses,
+ * because by using the bitmapsets of all attnums we may actually use attnums
+ * from clauses that are not covered by the statistics. For example, we may
+ * have a condition
+ *
+ * (a=1 AND b=2)
+ *
+ * and a new clause
+ *
+ * (c=1 AND d=1)
+ *
+ * With only bitmapsets, statistics on [b,c] will pass through this (assuming
+ * there are some statistics covering both clases).
+ *
+ * Parameters:
+ *
+ * stats - list of statistics to filter
+ * new_attnums - attnums referenced in new clauses
+ * all_attnums - attnums referenced by contidions and new clauses combined
+ *
+ * Returns filtered list of statistics.
+ *
+ * TODO Do the more strict check, i.e. walk through individual clauses and
+ * conditions and only use those covered by the statistics.
+ */
+static List *
+filter_stats(List *stats, Bitmapset *new_attnums, Bitmapset *all_attnums)
+{
+ ListCell *s;
+ List *stats_filtered = NIL;
+
+ foreach (s, stats)
+ {
+ int k;
+ int matches_new = 0,
+ matches_all = 0;
+
+ MVStatisticInfo *stat = (MVStatisticInfo *)lfirst(s);
+
+ /* see how many attributes the statistics covers */
+ for (k = 0; k < stat->stakeys->dim1; k++)
+ {
+ /* attributes from new clauses */
+ if (bms_is_member(stat->stakeys->values[k], new_attnums))
+ matches_new += 1;
+
+ /* attributes from onditions */
+ if (bms_is_member(stat->stakeys->values[k], all_attnums))
+ matches_all += 1;
+ }
+
+ /* check we have enough attributes for this statistics */
+ if ((matches_new >= 1) && (matches_all >= 2))
+ stats_filtered = lappend(stats_filtered, stat);
+ }
+
+ /* we can't have more useful stats than we had originally */
+ Assert(list_length(stats) >= list_length(stats_filtered));
+
+ return stats_filtered;
+}
+
+static MVStatisticInfo *
+make_stats_array(List *stats, int *nmvstats)
+{
+ int i;
+ ListCell *l;
+
+ MVStatisticInfo *mvstats = NULL;
+ *nmvstats = list_length(stats);
- /* TODO Evaluate simple 1D selectivities, use the smallest one as
- * an upper bound, product as lower bound, and sort the
- * clauses in ascending order by selectivity (to optimize the
- * MCV/histogram evaluation).
- */
+ mvstats
+ = (MVStatisticInfo*)palloc0((*nmvstats) * sizeof(MVStatisticInfo));
- /* Evaluate the MCV first. */
- s1 = clauselist_mv_selectivity_mcvlist(root, clauses, mvstats,
- &fullmatch, &mcv_low);
+ i = 0;
+ foreach (l, stats)
+ {
+ MVStatisticInfo *stat = (MVStatisticInfo *)lfirst(l);
+ memcpy(&mvstats[i++], stat, sizeof(MVStatisticInfo));
+ }
- /*
- * If we got a full equality match on the MCV list, we're done (and
- * the estimate is pretty good).
- */
- if (fullmatch && (s1 > 0.0))
- return s1;
+ return mvstats;
+}
- /* TODO if (fullmatch) without matching MCV item, use the mcv_low
- * selectivity as upper bound */
+static Bitmapset **
+make_stats_attnums(MVStatisticInfo *mvstats, int nmvstats)
+{
+ int i, j;
+ Bitmapset **stats_attnums = NULL;
- s2 = clauselist_mv_selectivity_histogram(root, clauses, mvstats);
+ Assert(nmvstats > 0);
- /* TODO clamp to <= 1.0 (or more strictly, when possible) */
- return s1 + s2;
+ /* build bitmaps of attnums for the stats (easier to compare) */
+ stats_attnums = (Bitmapset **)palloc0(nmvstats * sizeof(Bitmapset*));
+
+ for (i = 0; i < nmvstats; i++)
+ for (j = 0; j < mvstats[i].stakeys->dim1; j++)
+ stats_attnums[i]
+ = bms_add_member(stats_attnums[i],
+ mvstats[i].stakeys->values[j]);
+
+ return stats_attnums;
}
+
/*
- * Collect attributes from mv-compatible clauses.
+ * Remove redundant statistics
+ *
+ * If there are multiple statistics covering the same set of columns (counting
+ * only those referenced by clauses and conditions), we can apply one of those
+ * anyway and further reduce the size of the optimization problem.
+ *
+ * Thus when redundant stats are detected, we keep the smaller one (the one with
+ * fewer columns), based on the assumption that it's more accurate and also
+ * faster to process. That may be untrue for two reasons - first, the accuracy
+ * really depends on number of buckets/MCV items, not the number of columns.
+ * Second, some types of statistics may work better for certain types of clauses
+ * (e.g. MCV lists for equality conditions) etc.
*/
-static Bitmapset *
-collect_mv_attnums(List *clauses, Index relid, int types)
+static List*
+filter_redundant_stats(List *stats, List *clauses, List *conditions)
{
- Bitmapset *attnums = NULL;
- ListCell *l;
+ int i, j, nmvstats;
+
+ MVStatisticInfo *mvstats;
+ bool *redundant;
+ Bitmapset **stats_attnums;
+ Bitmapset *varattnos;
+ Index relid;
+
+ Assert(list_length(stats) > 0);
+ Assert(list_length(clauses) > 0);
+
+ /*
+ * We'll convert the list of statistics into an array now, because
+ * the reduction of redundant statistics is easier to do that way
+ * (we can mark previous stats as redundant, etc.).
+ */
+ mvstats = make_stats_array(stats, &nmvstats);
+ stats_attnums = make_stats_attnums(mvstats, nmvstats);
+
+ /* by default, none of the stats is redundant (so palloc0) */
+ redundant = palloc0(nmvstats * sizeof(bool));
+
+ /*
+ * We only expect a single relid here, and also we should get the
+ * same relid from clauses and conditions (but we get it from
+ * clauses, because those are certainly non-empty).
+ */
+ relid = bms_singleton_member(pull_varnos((Node*)clauses));
/*
- * Walk through the clauses and identify the ones we can estimate using
- * multivariate stats, and remember the relid/columns. We'll then
- * cross-check if we have suitable stats, and only if needed we'll split
- * the clauses into multivariate and regular lists.
+ * Get the varattnos from both conditions and clauses.
+ *
+ * This skips system attributes, although that should be impossible
+ * thanks to previous filtering out of incompatible clauses.
*
- * For now we're only interested in RestrictInfo nodes with nested OpExpr,
- * using either a range or equality.
+ * XXX Is that really true?
*/
- foreach (l, clauses)
+ varattnos = bms_union(get_varattnos((Node*)clauses, relid),
+ get_varattnos((Node*)conditions, relid));
+
+ for (i = 1; i < nmvstats; i++)
{
- Node *clause = (Node *) lfirst(l);
+ /* intersect with current statistics */
+ Bitmapset *curr = bms_intersect(stats_attnums[i], varattnos);
- /* ignore the result here - we only need the attnums */
- clause_is_mv_compatible(clause, relid, &attnums, types);
+ /* walk through 'previous' stats and check redundancy */
+ for (j = 0; j < i; j++)
+ {
+ /* intersect with current statistics */
+ Bitmapset *prev;
+
+ /* skip stats already identified as redundant */
+ if (redundant[j])
+ continue;
+
+ prev = bms_intersect(stats_attnums[j], varattnos);
+
+ switch (bms_subset_compare(curr, prev))
+ {
+ case BMS_EQUAL:
+ /*
+ * Use the smaller one (hopefully more accurate).
+ * If both have the same size, use the first one.
+ */
+ if (mvstats[i].stakeys->dim1 >= mvstats[j].stakeys->dim1)
+ redundant[i] = TRUE;
+ else
+ redundant[j] = TRUE;
+
+ break;
+
+ case BMS_SUBSET1: /* curr is subset of prev */
+ redundant[i] = TRUE;
+ break;
+
+ case BMS_SUBSET2: /* prev is subset of curr */
+ redundant[j] = TRUE;
+ break;
+
+ case BMS_DIFFERENT:
+ /* do nothing - keep both stats */
+ break;
+ }
+
+ bms_free(prev);
+ }
+
+ bms_free(curr);
}
- /*
- * If there are not at least two attributes referenced by the clause(s),
- * we can throw everything out (as we'll revert to simple stats).
- */
- if (bms_num_members(attnums) <= 1)
+ /* can't reduce all statistics (at least one has to remain) */
+ Assert(nmvstats > 0);
+
+ /* now, let's remove the reduced statistics from the arrays */
+ list_free(stats);
+ stats = NIL;
+
+ for (i = 0; i < nmvstats; i++)
{
- if (attnums != NULL)
- pfree(attnums);
- attnums = NULL;
+ MVStatisticInfo *info;
+
+ pfree(stats_attnums[i]);
+
+ if (redundant[i])
+ continue;
+
+ info = makeNode(MVStatisticInfo);
+ memcpy(info, &mvstats[i], sizeof(MVStatisticInfo));
+
+ stats = lappend(stats, info);
}
- return attnums;
+ pfree(mvstats);
+ pfree(stats_attnums);
+ pfree(redundant);
+
+ return stats;
}
-/*
- * Count the number of attributes in clauses compatible with multivariate stats.
- */
-static int
-count_mv_attnums(List *clauses, Index relid, int type)
+static Node**
+make_clauses_array(List *clauses, int *nclauses)
{
- int c;
- Bitmapset *attnums = collect_mv_attnums(clauses, relid, type);
+ int i;
+ ListCell *l;
- c = bms_num_members(attnums);
+ Node** clauses_array;
- bms_free(attnums);
+ *nclauses = list_length(clauses);
+ clauses_array = (Node **)palloc0((*nclauses) * sizeof(Node *));
- return c;
+ i = 0;
+ foreach (l, clauses)
+ clauses_array[i++] = (Node *)lfirst(l);
+
+ *nclauses = i;
+
+ return clauses_array;
}
-/*
- * Count varnos referenced in the clauses, and if there's a single varno then
- * return the index in 'relid'.
- */
-static int
-count_varnos(List *clauses, Index *relid)
+static Bitmapset **
+make_clauses_attnums(PlannerInfo *root, Index relid,
+ int type, Node **clauses, int nclauses)
{
- int cnt;
- Bitmapset *varnos = NULL;
+ int i;
+ Bitmapset **clauses_attnums
+ = (Bitmapset **)palloc0(nclauses * sizeof(Bitmapset *));
- varnos = pull_varnos((Node *) clauses);
- cnt = bms_num_members(varnos);
+ for (i = 0; i < nclauses; i++)
+ {
+ Bitmapset * attnums = NULL;
- /* if there's a single varno in the clauses, remember it */
- if (bms_num_members(varnos) == 1)
- *relid = bms_singleton_member(varnos);
+ if (! clause_is_mv_compatible(clauses[i], relid, &attnums, type))
+ elog(ERROR, "should not get non-mv-compatible clause");
- bms_free(varnos);
+ clauses_attnums[i] = attnums;
+ }
- return cnt;
+ return clauses_attnums;
+}
+
+static bool*
+make_cover_map(Bitmapset **stats_attnums, int nmvstats,
+ Bitmapset **clauses_attnums, int nclauses)
+{
+ int i, j;
+ bool *cover_map = (bool*)palloc0(nclauses * nmvstats);
+
+ for (i = 0; i < nmvstats; i++)
+ for (j = 0; j < nclauses; j++)
+ cover_map[i * nclauses + j]
+ = bms_is_subset(clauses_attnums[j], stats_attnums[i]);
+
+ return cover_map;
}
/*
- * We're looking for statistics matching at least 2 attributes, referenced in
- * clauses compatible with multivariate statistics. The current selection
- * criteria is very simple - we choose the statistics referencing the most
- * attributes.
- *
- * If there are multiple statistics referencing the same number of columns
- * (from the clauses), the one with less source columns (as listed in the
- * ADD STATISTICS when creating the statistics) wins. Else the first one wins.
- *
- * This is a very simple criteria, and has several weaknesses:
- *
- * (a) does not consider the accuracy of the statistics
+ * Chooses the combination of statistics, optimal for estimation of a particular
+ * clause list.
*
- * If there are two histograms built on the same set of columns, but one
- * has 100 buckets and the other one has 1000 buckets (thus likely
- * providing better estimates), this is not currently considered.
+ * This only handles a 'preparation' shared by the exhaustive and greedy
+ * implementations (see the previous methods), mostly trying to reduce the size
+ * of the problem (eliminate clauses/statistics that can't be really used in
+ * the solution).
*
- * (b) does not consider the type of statistics
+ * It also precomputes bitmaps for attributes covered by clauses and statistics,
+ * so that we don't need to do that over and over in the actual optimizations
+ * (as it's both CPU and memory intensive).
*
- * If there are three statistics - one containing just a MCV list, another
- * one with just a histogram and a third one with both, we treat them equally.
*
- * (c) does not consider the number of clauses
+ * TODO Another way to make the optimization problems smaller might be splitting
+ * the statistics into several disjoint subsets, i.e. if we can split the
+ * graph of statistics (after the elimination) into multiple components
+ * (so that stats in different components share no attributes), we can do
+ * the optimization for each component separately.
*
- * As explained, only the number of referenced attributes counts, so if
- * there are multiple clauses on a single attribute, this still counts as
- * a single attribute.
- *
- * (d) does not consider type of condition
- *
- * Some clauses may work better with some statistics - for example equality
- * clauses probably work better with MCV lists than with histograms. But
- * IS [NOT] NULL conditions may often work better with histograms (thanks
- * to NULL-buckets).
- *
- * So for example with five WHERE conditions
- *
- * WHERE (a = 1) AND (b = 1) AND (c = 1) AND (d = 1) AND (e = 1)
- *
- * and statistics on (a,b), (a,b,e) and (a,b,c,d), the last one will be selected
- * as it references the most columns.
- *
- * Once we have selected the multivariate statistics, we split the list of
- * clauses into two parts - conditions that are compatible with the selected
- * stats, and conditions are estimated using simple statistics.
- *
- * From the example above, conditions
- *
- * (a = 1) AND (b = 1) AND (c = 1) AND (d = 1)
- *
- * will be estimated using the multivariate statistics (a,b,c,d) while the last
- * condition (e = 1) will get estimated using the regular ones.
- *
- * There are various alternative selection criteria (e.g. counting conditions
- * instead of just referenced attributes), but eventually the best option should
- * be to combine multiple statistics. But that's much harder to do correctly.
- *
- * TODO Select multiple statistics and combine them when computing the estimate.
- *
- * TODO This will probably have to consider compatibility of clauses, because
- * 'dependencies' will probably work only with equality clauses.
+ * TODO If we could compute what is a "perfect solution" maybe we could
+ * terminate the search after reaching ~90% of it? Say, if we knew that we
+ * can cover 10 clauses and reuse 8 dependencies, maybe covering 9 clauses
+ * and 7 dependencies would be OK?
*/
-static MVStatisticInfo *
-choose_mv_statistics(List *stats, Bitmapset *attnums)
+static List*
+choose_mv_statistics(PlannerInfo *root, Index relid, List *stats,
+ List *clauses, List *conditions)
{
int i;
- ListCell *lc;
+ mv_solution_t *best = NULL;
+ List *result = NIL;
+
+ int nmvstats;
+ MVStatisticInfo *mvstats;
- MVStatisticInfo *choice = NULL;
+ /* we only work with MCV lists and histograms here */
+ int type = (MV_CLAUSE_TYPE_MCV | MV_CLAUSE_TYPE_HIST);
- int current_matches = 1; /* goal #1: maximize */
- int current_dims = (MVSTATS_MAX_DIMENSIONS+1); /* goal #2: minimize */
+ bool *clause_cover_map = NULL,
+ *condition_cover_map = NULL;
+ int *ruled_out = NULL;
+
+ /* build bitmapsets for all stats and clauses */
+ Bitmapset **stats_attnums;
+ Bitmapset **clauses_attnums;
+ Bitmapset **conditions_attnums;
+
+ int nclauses, nconditions;
+ Node ** clauses_array;
+ Node ** conditions_array;
+
+ /* copy lists, so that we can free them during elimination easily */
+ clauses = list_copy(clauses);
+ conditions = list_copy(conditions);
+ stats = list_copy(stats);
/*
- * Walk through the statistics (simple array with nmvstats elements) and for
- * each one count the referenced attributes (encoded in the 'attnums' bitmap).
+ * Reduce the optimization problem size as much as possible.
+ *
+ * Eliminate clauses and conditions not covered by any statistics,
+ * or statistics not matching at least two attributes (one of them
+ * has to be in a regular clause).
+ *
+ * It's possible that removing a statistics in one iteration
+ * eliminates clause in the next one, so we'll repeat this until we
+ * eliminate no clauses/stats in that iteration.
+ *
+ * This can only happen after eliminating a statistics - clauses are
+ * eliminated first, so statistics always reflect that.
*/
- foreach (lc, stats)
+ while (true)
{
- MVStatisticInfo *info = (MVStatisticInfo *)lfirst(lc);
+ List *tmp;
- /* columns matching this statistics */
- int matches = 0;
+ Bitmapset *compatible_attnums = NULL;
+ Bitmapset *condition_attnums = NULL;
+ Bitmapset *all_attnums = NULL;
- int2vector * attrs = info->stakeys;
- int numattrs = attrs->dim1;
-
- /* skip dependencies-only stats */
- if (! (info->mcv_built || info->hist_built))
- continue;
+ /*
+ * Clauses
+ *
+ * Walk through clauses and keep only those covered by at least
+ * one of the statistics we still have. We'll also keep info
+ * about attnums in clauses (without conditions) so that we can
+ * ignore stats covering just conditions (which is pointless).
+ */
+ tmp = filter_clauses(root, relid, type,
+ stats, clauses, &compatible_attnums);
- /* count columns covered by the histogram */
- for (i = 0; i < numattrs; i++)
- if (bms_is_member(attrs->values[i], attnums))
- matches++;
+ /* discard the original list */
+ list_free(clauses);
+ clauses = tmp;
/*
- * Use this statistics when it improves the number of matches or
- * when it matches the same number of attributes but is smaller.
+ * Conditions
+ *
+ * Walk through clauses and keep only those covered by at least
+ * one of the statistics we still have. Also, collect bitmap of
+ * attributes so that we can make sure we add at least one new
+ * attribute (by comparing with clauses).
*/
- if ((matches > current_matches) ||
- ((matches == current_matches) && (current_dims > numattrs)))
+ if (conditions != NIL)
{
- choice = info;
- current_matches = matches;
- current_dims = numattrs;
+ tmp = filter_clauses(root, relid, type,
+ stats, conditions, &condition_attnums);
+
+ /* discard the original list */
+ list_free(conditions);
+ conditions = tmp;
}
- }
- return choice;
-}
+ /* get a union of attnums (from conditions and new clauses) */
+ all_attnums = bms_union(compatible_attnums, condition_attnums);
+ /*
+ * Statisitics
+ *
+ * Walk through statistics and only keep those covering at least
+ * one new attribute (excluding conditions) and at two attributes
+ * in both clauses and conditions.
+ */
+ tmp = filter_stats(stats, compatible_attnums, all_attnums);
-/*
- * This splits the clauses list into two parts - one containing clauses that
- * will be evaluated using the chosen statistics, and the remaining clauses
- * (either non-mvcompatible, or not related to the histogram).
- */
-static List *
-clauselist_mv_split(PlannerInfo *root, Index relid,
- List *clauses, List **mvclauses,
- MVStatisticInfo *mvstats, int types)
-{
- int i;
- ListCell *l;
- List *non_mvclauses = NIL;
+ /* if we've not eliminated anything, terminate */
+ if (list_length(stats) == list_length(tmp))
+ break;
- /* FIXME is there a better way to get info on int2vector? */
- int2vector * attrs = mvstats->stakeys;
- int numattrs = mvstats->stakeys->dim1;
+ /* work only with filtered statistics from now */
+ list_free(stats);
+ stats = tmp;
+ }
- Bitmapset *mvattnums = NULL;
+ /* only do the optimization if we have clauses/statistics */
+ if ((list_length(stats) == 0) || (list_length(clauses) == 0))
+ return NULL;
- /* build bitmap of attributes, so we can do bms_is_subset later */
- for (i = 0; i < numattrs; i++)
- mvattnums = bms_add_member(mvattnums, attrs->values[i]);
+ /* remove redundant stats (stats covered by another stats) */
+ stats = filter_redundant_stats(stats, clauses, conditions);
- /* erase the list of mv-compatible clauses */
- *mvclauses = NIL;
+ /*
+ * TODO We should sort the stats to make the order deterministic,
+ * otherwise we may get different estimates on different
+ * executions - if there are multiple "equally good" solutions,
+ * we'll keep the first solution we see.
+ *
+ * Sorting by OID probably is not the right solution though,
+ * because we'd like it to be somehow reproducible,
+ * irrespectedly of the order of ADD STATISTICS commands.
+ * So maybe statkeys?
+ */
+ mvstats = make_stats_array(stats, &nmvstats);
+ stats_attnums = make_stats_attnums(mvstats, nmvstats);
- foreach (l, clauses)
- {
- bool match = false; /* by default not mv-compatible */
- Bitmapset *attnums = NULL;
- Node *clause = (Node *) lfirst(l);
+ /* collect clauses an bitmap of attnums */
+ clauses_array = make_clauses_array(clauses, &nclauses);
+ clauses_attnums = make_clauses_attnums(root, relid, type,
+ clauses_array, nclauses);
+
+ /* collect conditions and bitmap of attnums */
+ conditions_array = make_clauses_array(conditions, &nconditions);
+ conditions_attnums = make_clauses_attnums(root, relid, type,
+ conditions_array, nconditions);
- if (clause_is_mv_compatible(clause, relid, &attnums, types))
+ /*
+ * Build bitmaps with info about which clauses/conditions are
+ * covered by each statistics (so that we don't need to call the
+ * bms_is_subset over and over again).
+ */
+ clause_cover_map = make_cover_map(stats_attnums, nmvstats,
+ clauses_attnums, nclauses);
+
+ condition_cover_map = make_cover_map(stats_attnums, nmvstats,
+ conditions_attnums, nconditions);
+
+ ruled_out = (int*)palloc0(nmvstats * sizeof(int));
+
+ /* no stats are ruled out by default */
+ for (i = 0; i < nmvstats; i++)
+ ruled_out[i] = -1;
+
+ /* do the optimization itself */
+ if (mvstat_search_type == MVSTAT_SEARCH_EXHAUSTIVE)
+ choose_mv_statistics_exhaustive(root, 0,
+ nmvstats, mvstats, stats_attnums,
+ nclauses, clauses_array, clauses_attnums,
+ nconditions, conditions_array, conditions_attnums,
+ clause_cover_map, condition_cover_map,
+ ruled_out, NULL, &best);
+ else
+ choose_mv_statistics_greedy(root, 0,
+ nmvstats, mvstats, stats_attnums,
+ nclauses, clauses_array, clauses_attnums,
+ nconditions, conditions_array, conditions_attnums,
+ clause_cover_map, condition_cover_map,
+ ruled_out, NULL, &best);
+
+ /* create a list of statistics from the array */
+ if (best != NULL)
+ {
+ for (i = 0; i < best->nstats; i++)
{
- /* are all the attributes part of the selected stats? */
- if (bms_is_subset(attnums, mvattnums))
- match = true;
+ MVStatisticInfo *info = makeNode(MVStatisticInfo);
+ memcpy(info, &mvstats[best->stats[i]], sizeof(MVStatisticInfo));
+ result = lappend(result, info);
}
- /*
- * The clause matches the selected stats, so put it to the list of
- * mv-compatible clauses. Otherwise, keep it in the list of 'regular'
- * clauses (that may be selected later).
- */
- if (match)
- *mvclauses = lappend(*mvclauses, clause);
- else
- non_mvclauses = lappend(non_mvclauses, clause);
+ pfree(best);
}
- /*
- * Perform regular estimation using the clauses incompatible with the chosen
- * histogram (or MV stats in general).
- */
- return non_mvclauses;
+ /* cleanup (maybe leave it up to the memory context?) */
+ for (i = 0; i < nmvstats; i++)
+ bms_free(stats_attnums[i]);
+
+ for (i = 0; i < nclauses; i++)
+ bms_free(clauses_attnums[i]);
+
+ for (i = 0; i < nconditions; i++)
+ bms_free(conditions_attnums[i]);
+
+ pfree(stats_attnums);
+ pfree(clauses_attnums);
+ pfree(conditions_attnums);
+
+ pfree(clauses_array);
+ pfree(conditions_array);
+ pfree(clause_cover_map);
+ pfree(condition_cover_map);
+ pfree(ruled_out);
+ pfree(mvstats);
+
+ list_free(clauses);
+ list_free(conditions);
+ list_free(stats);
+ return result;
}
typedef struct
@@ -1637,9 +2878,6 @@ has_stats(List *stats, int type)
/* terminate if we've found at least one matching statistics */
if (stats_type_matches(stat, type))
return true;
-
- if ((type & MV_CLAUSE_TYPE_HIST) && stat->hist_built)
- return true;
}
return false;
@@ -1689,22 +2927,26 @@ find_stats(PlannerInfo *root, Index relid)
* as the clauses are processed (and skip items that are 'match').
*/
static Selectivity
-clauselist_mv_selectivity_mcvlist(PlannerInfo *root, List *clauses,
- MVStatisticInfo *mvstats, bool *fullmatch,
- Selectivity *lowsel)
+clauselist_mv_selectivity_mcvlist(PlannerInfo *root, MVStatisticInfo *mvstats,
+ List *clauses, List *conditions, bool is_or,
+ bool *fullmatch, Selectivity *lowsel)
{
int i;
Selectivity s = 0.0;
+ Selectivity t = 0.0;
Selectivity u = 0.0;
MCVList mcvlist = NULL;
+
int nmatches = 0;
+ int nconditions = 0;
/* match/mismatch bitmap for each MCV item */
char * matches = NULL;
+ char * condition_matches = NULL;
Assert(clauses != NIL);
- Assert(list_length(clauses) >= 2);
+ Assert(list_length(clauses) >= 1);
/* there's no MCV list built yet */
if (! mvstats->mcv_built)
@@ -1715,32 +2957,85 @@ clauselist_mv_selectivity_mcvlist(PlannerInfo *root, List *clauses,
Assert(mcvlist != NULL);
Assert(mcvlist->nitems > 0);
- /* by default all the MCV items match the clauses fully */
- matches = palloc0(sizeof(char) * mcvlist->nitems);
- memset(matches, MVSTATS_MATCH_FULL, sizeof(char)*mcvlist->nitems);
-
/* number of matching MCV items */
nmatches = mcvlist->nitems;
+ nconditions = mcvlist->nitems;
+
+ /*
+ * Bitmap of bucket matches (mismatch, partial, full).
+ *
+ * For AND clauses all buckets match (and we'll eliminate them).
+ * For OR clauses no buckets match (and we'll add them).
+ *
+ * We only need to do the memset for AND clauses (for OR clauses
+ * it's already set correctly by the palloc0).
+ */
+ matches = palloc0(sizeof(char) * nmatches);
+
+ if (! is_or) /* AND-clause */
+ memset(matches, MVSTATS_MATCH_FULL, sizeof(char)*nmatches);
+
+ /* Conditions are treated as AND clause, so match by default. */
+ condition_matches = palloc0(sizeof(char) * nconditions);
+ memset(condition_matches, MVSTATS_MATCH_FULL, sizeof(char)*nconditions);
+ /*
+ * build the match bitmap for the conditions (conditions are always
+ * connected by AND)
+ */
+ if (conditions != NIL)
+ nconditions = update_match_bitmap_mcvlist(root, conditions,
+ mvstats->stakeys, mcvlist,
+ nconditions, condition_matches,
+ lowsel, fullmatch, false);
+
+ /*
+ * build the match bitmap for the estimated clauses
+ *
+ * TODO This evaluates the clauses for all MCV items, even those
+ * ruled out by the conditions. The final result should be the
+ * same, but it might be faster.
+ */
nmatches = update_match_bitmap_mcvlist(root, clauses,
mvstats->stakeys, mcvlist,
- nmatches, matches,
- lowsel, fullmatch, false);
+ ((is_or) ? 0 : nmatches), matches,
+ lowsel, fullmatch, is_or);
/* sum frequencies for all the matching MCV items */
for (i = 0; i < mcvlist->nitems; i++)
{
- /* used to 'scale' for MCV lists not covering all tuples */
+ /*
+ * Find out what part of the data is covered by the MCV list,
+ * so that we can 'scale' the selectivity properly (e.g. when
+ * only 50% of the sample items got into the MCV, and the rest
+ * is either in a histogram, or not covered by stats).
+ *
+ * TODO This might be handled by keeping a global "frequency"
+ * for the whole list, which might save us a bit of time
+ * spent on accessing the not-matching part of the MCV list.
+ * Although it's likely in a cache, so it's very fast.
+ */
u += mcvlist->items[i]->frequency;
+ /* skit MCV items not matching the conditions */
+ if (condition_matches[i] == MVSTATS_MATCH_NONE)
+ continue;
+
if (matches[i] != MVSTATS_MATCH_NONE)
s += mcvlist->items[i]->frequency;
+
+ t += mcvlist->items[i]->frequency;
}
pfree(matches);
+ pfree(condition_matches);
pfree(mcvlist);
- return s*u;
+ /* no condition matches */
+ if (t == 0.0)
+ return (Selectivity)0.0;
+
+ return (s / t) * u;
}
/*
@@ -1971,64 +3266,57 @@ update_match_bitmap_mcvlist(PlannerInfo *root, List *clauses,
}
}
}
- else if (or_clause(clause) || and_clause(clause))
+ else if (or_clause(clause) || and_clause(clause) || not_clause(clause))
{
/* AND/OR clause, with all clauses compatible with the selected MV stat */
int i;
- BoolExpr *orclause = ((BoolExpr*)clause);
- List *orclauses = orclause->args;
+ List *tmp_clauses = ((BoolExpr*)clause)->args;
/* match/mismatch bitmap for each MCV item */
- int or_nmatches = 0;
- char * or_matches = NULL;
+ int tmp_nmatches = 0;
+ char * tmp_matches = NULL;
- Assert(orclauses != NIL);
- Assert(list_length(orclauses) >= 2);
+ Assert(tmp_clauses != NIL);
+ Assert((list_length(tmp_clauses) >= 2) || (not_clause(clause) && (list_length(tmp_clauses)==1)));
/* number of matching MCV items */
- or_nmatches = mcvlist->nitems;
+ tmp_nmatches = (or_clause(clause)) ? 0 : mcvlist->nitems;
/* by default none of the MCV items matches the clauses */
- or_matches = palloc0(sizeof(char) * or_nmatches);
+ tmp_matches = palloc0(sizeof(char) * mcvlist->nitems);
- if (or_clause(clause))
- {
- /* OR clauses assume nothing matches, initially */
- memset(or_matches, MVSTATS_MATCH_NONE, sizeof(char)*or_nmatches);
- or_nmatches = 0;
- }
- else
- {
- /* AND clauses assume nothing matches, initially */
- memset(or_matches, MVSTATS_MATCH_FULL, sizeof(char)*or_nmatches);
- }
+ /* AND (and NOT) clauses assume everything matches, initially */
+ if (! or_clause(clause))
+ memset(tmp_matches, MVSTATS_MATCH_FULL, sizeof(char)*mcvlist->nitems);
/* build the match bitmap for the OR-clauses */
- or_nmatches = update_match_bitmap_mcvlist(root, orclauses,
+ tmp_nmatches = update_match_bitmap_mcvlist(root, tmp_clauses,
stakeys, mcvlist,
- or_nmatches, or_matches,
+ tmp_nmatches, tmp_matches,
lowsel, fullmatch, or_clause(clause));
/* merge the bitmap into the existing one*/
for (i = 0; i < mcvlist->nitems; i++)
{
+ /* if this is a NOT clause, we need to invert the results first */
+ if (not_clause(clause))
+ tmp_matches[i] = (MVSTATS_MATCH_FULL - tmp_matches[i]);
+
/*
* To AND-merge the bitmaps, a MIN() semantics is used.
* For OR-merge, use MAX().
*
* FIXME this does not decrease the number of matches
*/
- UPDATE_RESULT(matches[i], or_matches[i], is_or);
+ UPDATE_RESULT(matches[i], tmp_matches[i], is_or);
}
- pfree(or_matches);
+ pfree(tmp_matches);
}
else
- {
elog(ERROR, "unknown clause type: %d", clause->type);
- }
}
/*
@@ -2086,15 +3374,18 @@ update_match_bitmap_mcvlist(PlannerInfo *root, List *clauses,
* this is not uncommon, but for histograms it's not that clear.
*/
static Selectivity
-clauselist_mv_selectivity_histogram(PlannerInfo *root, List *clauses,
- MVStatisticInfo *mvstats)
+clauselist_mv_selectivity_histogram(PlannerInfo *root, MVStatisticInfo *mvstats,
+ List *clauses, List *conditions, bool is_or)
{
int i;
Selectivity s = 0.0;
+ Selectivity t = 0.0;
Selectivity u = 0.0;
int nmatches = 0;
+ int nconditions = 0;
char *matches = NULL;
+ char *condition_matches = NULL;
MVSerializedHistogram mvhist = NULL;
@@ -2107,25 +3398,55 @@ clauselist_mv_selectivity_histogram(PlannerInfo *root, List *clauses,
Assert (mvhist != NULL);
Assert (clauses != NIL);
- Assert (list_length(clauses) >= 2);
+ Assert (list_length(clauses) >= 1);
+
+ nmatches = mvhist->nbuckets;
+ nconditions = mvhist->nbuckets;
/*
- * Bitmap of bucket matches (mismatch, partial, full). by default
- * all buckets fully match (and we'll eliminate them).
+ * Bitmap of bucket matches (mismatch, partial, full).
+ *
+ * For AND clauses all buckets match (and we'll eliminate them).
+ * For OR clauses no buckets match (and we'll add them).
+ *
+ * We only need to do the memset for AND clauses (for OR clauses
+ * it's already set correctly by the palloc0).
*/
- matches = palloc0(sizeof(char) * mvhist->nbuckets);
- memset(matches, MVSTATS_MATCH_FULL, sizeof(char)*mvhist->nbuckets);
+ matches = palloc0(sizeof(char) * nmatches);
- nmatches = mvhist->nbuckets;
+ if (! is_or) /* AND-clause */
+ memset(matches, MVSTATS_MATCH_FULL, sizeof(char)*nmatches);
+
+ /* Conditions are treated as AND clause, so match by default. */
+ condition_matches = palloc0(sizeof(char)*nconditions);
+ memset(condition_matches, MVSTATS_MATCH_FULL, sizeof(char)*nconditions);
+
+ /*
+ * build the match bitmap for the conditions (conditions are always
+ * connected by AND)
+ */
+ if (conditions != NIL)
+ update_match_bitmap_histogram(root, conditions,
+ mvstats->stakeys, mvhist,
+ nconditions, condition_matches, false);
- /* build the match bitmap */
+ /*
+ * build the match bitmap for the estimated clauses
+ *
+ * TODO This evaluates the clauses for all buckets, even those
+ * ruled out by the conditions. The final result should be
+ * the same, but it might be faster.
+ */
update_match_bitmap_histogram(root, clauses,
mvstats->stakeys, mvhist,
- nmatches, matches, false);
+ ((is_or) ? 0 : nmatches), matches,
+ is_or);
/* now, walk through the buckets and sum the selectivities */
for (i = 0; i < mvhist->nbuckets; i++)
{
+ float coeff = 1.0;
+
/*
* Find out what part of the data is covered by the histogram,
* so that we can 'scale' the selectivity properly (e.g. when
@@ -2139,10 +3460,23 @@ clauselist_mv_selectivity_histogram(PlannerInfo *root, List *clauses,
*/
u += mvhist->buckets[i]->ntuples;
+ /* skip buckets not matching the conditions */
+ if (condition_matches[i] == MVSTATS_MATCH_NONE)
+ continue;
+ else if (condition_matches[i] == MVSTATS_MATCH_PARTIAL)
+ coeff = 0.5;
+
+ t += coeff * mvhist->buckets[i]->ntuples;
+
if (matches[i] == MVSTATS_MATCH_FULL)
- s += mvhist->buckets[i]->ntuples;
+ s += coeff * mvhist->buckets[i]->ntuples;
else if (matches[i] == MVSTATS_MATCH_PARTIAL)
- s += 0.5 * mvhist->buckets[i]->ntuples;
+ /*
+ * TODO If both conditions and clauses match partially, this
+ * will use 0.25 match - not sure if that's the right
+ * thing solution, but seems about right.
+ */
+ s += coeff * 0.5 * mvhist->buckets[i]->ntuples;
}
#ifdef DEBUG_MVHIST
@@ -2151,9 +3485,14 @@ clauselist_mv_selectivity_histogram(PlannerInfo *root, List *clauses,
/* release the allocated bitmap and deserialized histogram */
pfree(matches);
+ pfree(condition_matches);
pfree(mvhist);
- return s * u;
+ /* no condition matches */
+ if (t == 0.0)
+ return (Selectivity)0.0;
+
+ return (s / t) * u;
}
/* cached result of bucket boundary comparison for a single dimension */
@@ -2344,7 +3683,7 @@ update_match_bitmap_histogram(PlannerInfo *root, List *clauses,
FmgrInfo opproc; /* operator */
fmgr_info(get_opcode(expr->opno), &opproc);
-
+
/* reset the cache (per clause) */
memset(callcache, 0, mvhist->nbuckets);
@@ -2504,64 +3843,57 @@ update_match_bitmap_histogram(PlannerInfo *root, List *clauses,
UPDATE_RESULT(matches[i], MVSTATS_MATCH_NONE, is_or);
}
}
- else if (or_clause(clause) || and_clause(clause))
+ else if (or_clause(clause) || and_clause(clause) || not_clause(clause))
{
/* AND/OR clause, with all clauses compatible with the selected MV stat */
int i;
- BoolExpr *orclause = ((BoolExpr*)clause);
- List *orclauses = orclause->args;
+ List *tmp_clauses = ((BoolExpr*)clause)->args;
/* match/mismatch bitmap for each bucket */
- int or_nmatches = 0;
- char * or_matches = NULL;
+ int tmp_nmatches = 0;
+ char * tmp_matches = NULL;
- Assert(orclauses != NIL);
- Assert(list_length(orclauses) >= 2);
+ Assert(tmp_clauses != NIL);
+ Assert((list_length(tmp_clauses) >= 2) || (not_clause(clause) && (list_length(tmp_clauses)==1)));
/* number of matching buckets */
- or_nmatches = mvhist->nbuckets;
+ tmp_nmatches = (or_clause(clause)) ? 0 : mvhist->nbuckets;
- /* by default none of the buckets matches the clauses */
- or_matches = palloc0(sizeof(char) * or_nmatches);
+ /* by default none of the buckets matches the clauses (OR clause) */
+ tmp_matches = palloc0(sizeof(char) * mvhist->nbuckets);
- if (or_clause(clause))
- {
- /* OR clauses assume nothing matches, initially */
- memset(or_matches, MVSTATS_MATCH_NONE, sizeof(char)*or_nmatches);
- or_nmatches = 0;
- }
- else
- {
- /* AND clauses assume nothing matches, initially */
- memset(or_matches, MVSTATS_MATCH_FULL, sizeof(char)*or_nmatches);
- }
+ /* but AND (and NOT) clauses assume everything matches, initially */
+ if (! or_clause(clause))
+ memset(tmp_matches, MVSTATS_MATCH_FULL, sizeof(char)*mvhist->nbuckets);
/* build the match bitmap for the OR-clauses */
- or_nmatches = update_match_bitmap_histogram(root, orclauses,
+ tmp_nmatches = update_match_bitmap_histogram(root, tmp_clauses,
stakeys, mvhist,
- or_nmatches, or_matches, or_clause(clause));
+ tmp_nmatches, tmp_matches, or_clause(clause));
/* merge the bitmap into the existing one*/
for (i = 0; i < mvhist->nbuckets; i++)
{
+ /* if this is a NOT clause, we need to invert the results first */
+ if (not_clause(clause))
+ tmp_matches[i] = (MVSTATS_MATCH_FULL - tmp_matches[i]);
+
/*
* To AND-merge the bitmaps, a MIN() semantics is used.
* For OR-merge, use MAX().
*
* FIXME this does not decrease the number of matches
*/
- UPDATE_RESULT(matches[i], or_matches[i], is_or);
+ UPDATE_RESULT(matches[i], tmp_matches[i], is_or);
}
- pfree(or_matches);
-
+ pfree(tmp_matches);
}
else
elog(ERROR, "unknown clause type: %d", clause->type);
}
- /* free the call cache */
pfree(callcache);
return nmatches;
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index 5350329..57214e0 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -3518,7 +3518,8 @@ compute_semi_anti_join_factors(PlannerInfo *root,
joinquals,
0,
jointype,
- sjinfo);
+ sjinfo,
+ NIL);
/*
* Also get the normal inner-join selectivity of the join clauses.
@@ -3541,7 +3542,8 @@ compute_semi_anti_join_factors(PlannerInfo *root,
joinquals,
0,
JOIN_INNER,
- &norm_sjinfo);
+ &norm_sjinfo,
+ NIL);
/* Avoid leaking a lot of ListCells */
if (jointype == JOIN_ANTI)
@@ -3708,7 +3710,7 @@ approx_tuple_count(PlannerInfo *root, JoinPath *path, List *quals)
Node *qual = (Node *) lfirst(l);
/* Note that clause_selectivity will be able to cache its result */
- selec *= clause_selectivity(root, qual, 0, JOIN_INNER, &sjinfo);
+ selec *= clause_selectivity(root, qual, 0, JOIN_INNER, &sjinfo, NIL);
}
/* Apply it to the input relation sizes */
@@ -3744,7 +3746,8 @@ set_baserel_size_estimates(PlannerInfo *root, RelOptInfo *rel)
rel->baserestrictinfo,
0,
JOIN_INNER,
- NULL);
+ NULL,
+ NIL);
rel->rows = clamp_row_est(nrows);
@@ -3781,7 +3784,8 @@ get_parameterized_baserel_size(PlannerInfo *root, RelOptInfo *rel,
allclauses,
rel->relid, /* do not use 0! */
JOIN_INNER,
- NULL);
+ NULL,
+ NIL);
nrows = clamp_row_est(nrows);
/* For safety, make sure result is not more than the base estimate */
if (nrows > rel->rows)
@@ -3919,12 +3923,14 @@ calc_joinrel_size_estimate(PlannerInfo *root,
joinquals,
0,
jointype,
- sjinfo);
+ sjinfo,
+ NIL);
pselec = clauselist_selectivity(root,
pushedquals,
0,
jointype,
- sjinfo);
+ sjinfo,
+ NIL);
/* Avoid leaking a lot of ListCells */
list_free(joinquals);
@@ -3936,7 +3942,8 @@ calc_joinrel_size_estimate(PlannerInfo *root,
restrictlist,
0,
jointype,
- sjinfo);
+ sjinfo,
+ NIL);
pselec = 0.0; /* not used, keep compiler quiet */
}
diff --git a/src/backend/optimizer/util/orclauses.c b/src/backend/optimizer/util/orclauses.c
index ea831f5..6299e75 100644
--- a/src/backend/optimizer/util/orclauses.c
+++ b/src/backend/optimizer/util/orclauses.c
@@ -280,7 +280,7 @@ consider_new_or_clause(PlannerInfo *root, RelOptInfo *rel,
* saving work later.)
*/
or_selec = clause_selectivity(root, (Node *) or_rinfo,
- 0, JOIN_INNER, NULL);
+ 0, JOIN_INNER, NULL, NIL);
/*
* The clause is only worth adding to the query if it rejects a useful
@@ -342,7 +342,7 @@ consider_new_or_clause(PlannerInfo *root, RelOptInfo *rel,
/* Compute inner-join size */
orig_selec = clause_selectivity(root, (Node *) join_or_rinfo,
- 0, JOIN_INNER, &sjinfo);
+ 0, JOIN_INNER, &sjinfo, NIL);
/* And hack cached selectivity so join size remains the same */
join_or_rinfo->norm_selec = orig_selec / or_selec;
diff --git a/src/backend/utils/adt/selfuncs.c b/src/backend/utils/adt/selfuncs.c
index d396ef1..805d633 100644
--- a/src/backend/utils/adt/selfuncs.c
+++ b/src/backend/utils/adt/selfuncs.c
@@ -1627,13 +1627,15 @@ booltestsel(PlannerInfo *root, BoolTestType booltesttype, Node *arg,
case IS_NOT_FALSE:
selec = (double) clause_selectivity(root, arg,
varRelid,
- jointype, sjinfo);
+ jointype, sjinfo,
+ NIL);
break;
case IS_FALSE:
case IS_NOT_TRUE:
selec = 1.0 - (double) clause_selectivity(root, arg,
varRelid,
- jointype, sjinfo);
+ jointype, sjinfo,
+ NIL);
break;
default:
elog(ERROR, "unrecognized booltesttype: %d",
@@ -6260,7 +6262,8 @@ genericcostestimate(PlannerInfo *root,
indexSelectivity = clauselist_selectivity(root, selectivityQuals,
index->rel->relid,
JOIN_INNER,
- NULL);
+ NULL,
+ NIL);
/*
* If caller didn't give us an estimate, estimate the number of index
@@ -6580,7 +6583,8 @@ btcostestimate(PlannerInfo *root, IndexPath *path, double loop_count,
btreeSelectivity = clauselist_selectivity(root, selectivityQuals,
index->rel->relid,
JOIN_INNER,
- NULL);
+ NULL,
+ NIL);
numIndexTuples = btreeSelectivity * index->rel->tuples;
/*
@@ -7331,7 +7335,8 @@ gincostestimate(PlannerInfo *root, IndexPath *path, double loop_count,
*indexSelectivity = clauselist_selectivity(root, selectivityQuals,
index->rel->relid,
JOIN_INNER,
- NULL);
+ NULL,
+ NIL);
/* fetch estimated page cost for tablespace containing index */
get_tablespace_page_costs(index->reltablespace,
@@ -7561,7 +7566,7 @@ brincostestimate(PlannerInfo *root, IndexPath *path, double loop_count,
*indexSelectivity =
clauselist_selectivity(root, indexQuals,
path->indexinfo->rel->relid,
- JOIN_INNER, NULL);
+ JOIN_INNER, NULL, NIL);
*indexCorrelation = 1;
/*
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index edcafce..b7aabed 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -75,6 +75,7 @@
#include "utils/bytea.h"
#include "utils/guc_tables.h"
#include "utils/memutils.h"
+#include "utils/mvstats.h"
#include "utils/pg_locale.h"
#include "utils/plancache.h"
#include "utils/portal.h"
@@ -393,6 +394,15 @@ static const struct config_enum_entry force_parallel_mode_options[] = {
};
/*
+ * Search algorithm for multivariate stats.
+ */
+static const struct config_enum_entry mvstat_search_options[] = {
+ {"greedy", MVSTAT_SEARCH_GREEDY, false},
+ {"exhaustive", MVSTAT_SEARCH_EXHAUSTIVE, false},
+ {NULL, 0, false}
+};
+
+/*
* Options for enum values stored in other modules
*/
extern const struct config_enum_entry wal_level_options[];
@@ -3743,6 +3753,16 @@ static struct config_enum ConfigureNamesEnum[] =
NULL, NULL, NULL
},
+ {
+ {"mvstat_search", PGC_USERSET, QUERY_TUNING_OTHER,
+ gettext_noop("Sets the algorithm used for combining multivariate stats."),
+ NULL
+ },
+ &mvstat_search_type,
+ MVSTAT_SEARCH_GREEDY, mvstat_search_options,
+ NULL, NULL, NULL
+ },
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, 0, NULL, NULL, NULL, NULL
diff --git a/src/backend/utils/mvstats/README.stats b/src/backend/utils/mvstats/README.stats
index 3e4f4d1..d404914 100644
--- a/src/backend/utils/mvstats/README.stats
+++ b/src/backend/utils/mvstats/README.stats
@@ -90,6 +90,137 @@ even attempting to do the more expensive estimation.
Whenever we find there are no suitable stats, we skip the expensive steps.
+Combining multiple statistics
+-----------------------------
+
+When estimating selectivity of a list of clauses, there may exist no statistics
+covering all of them. If there are multiple statistics, each covering some
+subset of the attributes, the optimizer needs to figure out which of those
+statistics to apply.
+
+When the statistics do not overlap, the solution is trivial - we can simply
+split the groups of conditions by the matching statistics, and then multiply the
+selectivities. For example assume multivariate statistics on (b,c) and (d,e),
+and a condition like this:
+
+ (a=1) AND (b=2) AND (c=3) AND (d=4) AND (e=5)
+
+Then (a=1) is not covered by any of the statistics, so will be estimated using
+the regular per-column statistics. The two conditions ((b=2) AND (c=3)) will be
+estimated using the (b,c) statistics, and ((d=4) AND (e=5)) will be estimated
+using (d,e) statistics. And the resulting selectivities will be estimated.
+
+Now, what if the statistics overlap? For example assume the same condition as
+above, but let's say we have statistics on (a,b,c) and (a,c,d,e). What then?
+
+As selectivity is just a probability that the condition holds for a random row,
+we can write the selectivity like this:
+
+ P(a=1 & b=2 & c=3 & d=4 & e=5)
+
+and we can rewrite it using conditional probability like this
+
+ P(a=1 & b=2 & c=3) * P(d=4 & e=5 | a=1 & b=2 & c=3)
+
+Notice that the first part already matches to (a,b,c) statistics. If we assume
+that columns that are not referenced by the same statistics are independent, we
+may rewrite the second half like this
+
+ P(d=4 & e=5 | a=1 & b=2 & c=3) = P(d=4 & e=5 | a=1 & c=3)
+
+which corresponds to the statistics on (a,c,d,e).
+
+If there are multiple statistics defined on a table, it's not difficult to come
+up with examples when there are multiple ways to combine them to cover a list of
+clauses. We need a way to find the best combination of statistics.
+
+This is the purpose of choose_mv_statistics(). It searches through the possible
+combinations of statistics, and searches such combination that
+
+ (a) covers the most clauses of the list
+
+ (b) reuses the maximum number of clauses as conditions
+ (in conditional probabilities)
+
+While (a) criteria seems natural, the (b) may seem a bit awkward at first. The
+idea is that conditions in a way of transfering information about dependencies
+between statistics.
+
+There are two alternative implementations of choose_mv_statistics() - greedy
+and exhaustive. Exhaustive actually searches through all possible combinations
+of statistics, and for larger numbers of statistics may get quite expensive
+(as it, unsurprisingly, has exponential cost). Greedy terminates in less than
+K steps (when K is the number of clauses), and in each step chooses the best
+next statistics. I've been unable to come up with an example where those two
+approaches would produce different combinations.
+
+It's possible to choose the optimization using mvstat_search_type, with either
+'greedy' or 'exhaustive' values (default is 'greedy').
+
+ SET mvstat_search_type = 'exhaustive';
+
+Note: This is meant mostly for experimentation. I do expect we'll choose one of
+the algorithms and remove the GUC before commit.
+
+
+Limitations of combining statistics
+-----------------------------------
+
+As described in the section 'Combining multiple statistics', the current appoach
+is based on transfering information between statistics by means of conditional
+probabilities. This is a relatively cheap and efficient approach, but it is
+based on two assumptions:
+
+ (1) The overlap between the statistics needs to be sufficiently large, i.e.
+ there needs to be enough columns shared by the statistics to transfer
+ information about dependencies between the remaining columns.
+
+ (2) The query needs to include sufficient clauses on the shared columns.
+
+How a violation of those assumptions may be a problem can be illustrated by
+a simple example. Assume a table with three columns (a,b,c) containing exactly
+the same values, and statistics on (a,b) and (b,c):
+
+ CREATE TABLE test AS SELECT i, i, i
+ FROM generate_series(1,1000);
+
+ CREATE STATISTICS s1 ON test (a,b) WITH (mcv);
+ CREATE STATISTICS s2 ON test (b,c) WITH (mcv);
+
+ ANALYZE test;
+
+First, let's estimate this query:
+
+ SELECT * FROM test WHERE (a < 10) AND (c < 10);
+
+Clearly, there are no conditions on 'b' (which is the only column shared by the
+two statistics), so we'll end up with an estimate based on assumption of
+independence:
+
+ P(a < 10) * P(c < 10) = 0.01 * 0.01 = 0.0001
+
+Which is a significant under-estimate, as the proper selectivity is 0.01.
+
+But let's estimate another query:
+
+ SELECT * FROM test WHERE (a < 10) AND (b < 500) AND (c < 10);
+
+In this case, the estimate may be computed for example like this:
+
+ P[(a < 10) & (b < 500) & (c < 10)]
+ = P[(a < 10) & (b < 500)] * P[(c < 10) | (a < 10) & (b < 500)]
+ = P[(a < 10) & (b < 500)] * P[(c < 10) | (b < 500)]
+
+The trouble is the probability P(c < 10 | b < 500) evaluates to 0.02, because
+we have assumed (a) and (c) are independent because there is no statistic
+containing both these columns, and the condition on (b) does not transfer
+sufficient amount of information between the two statistics.
+
+Currently, the only solution is to build statistics on all three columns, but
+see the 'combining statistics using convolution' section for ideas on how to
+improve this.
+
+
Further (possibly crazy) ideas
------------------------------
@@ -111,3 +242,38 @@ But of course, this may result in expensive estimation (CPU-wise).
So we might add a GUC to choose between a simple (single statistics) and thus
multi-statistic estimation, possibly table-level parameter (ALTER TABLE ...).
+
+
+Combining stats using convolution
+---------------------------------
+
+While the current approach for combining statistics is based on conditional
+probabilities, and thus only works when the query includes conditions on the
+overlapping parts of the statistics. But there may be other ways to combine
+statistics, relaxing this requirement.
+
+Let's assume two histograms H1 and H2 - then combining them might work about
+like this:
+
+
+ for (buckets of H1, satisfying local conditions)
+ {
+ for (buckets of H2, overlapping with H1 bucket)
+ {
+ mark H2 bucket as 'valid'
+ }
+ }
+
+ s1 = s2 = 0.0
+ for (buckets of H2 marked as valid)
+ {
+ s1 += frequency
+
+ if (bucket satistifes local conditions)
+ s2 += frequency
+ }
+
+ s = (s2 / s1) /* final selectivity estimate */
+
+However this may quickly get non-trivial, e.g. when combining two statistics
+of different types (histogram vs. MCV).
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index fea2bb7..33f5a1b 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -192,11 +192,13 @@ extern Selectivity clauselist_selectivity(PlannerInfo *root,
List *clauses,
int varRelid,
JoinType jointype,
- SpecialJoinInfo *sjinfo);
+ SpecialJoinInfo *sjinfo,
+ List *conditions);
extern Selectivity clause_selectivity(PlannerInfo *root,
Node *clause,
int varRelid,
JoinType jointype,
- SpecialJoinInfo *sjinfo);
+ SpecialJoinInfo *sjinfo,
+ List *conditions);
#endif /* COST_H */
diff --git a/src/include/utils/mvstats.h b/src/include/utils/mvstats.h
index 777c7da..2b67772 100644
--- a/src/include/utils/mvstats.h
+++ b/src/include/utils/mvstats.h
@@ -17,6 +17,14 @@
#include "fmgr.h"
#include "commands/vacuum.h"
+typedef enum MVStatSearchType
+{
+ MVSTAT_SEARCH_EXHAUSTIVE, /* exhaustive search */
+ MVSTAT_SEARCH_GREEDY /* greedy search */
+} MVStatSearchType;
+
+extern int mvstat_search_type;
+
/*
* Degree of how much MCV item / histogram bucket matches a clause.
* This is then considered when computing the selectivity.
--
2.5.0
[text/x-patch] 0007-multivariate-ndistinct-coefficients.patch (28.5K, ../../7aa6145f-302b-2077-b961-702cc61710bb@2ndquadrant.com/8-0007-multivariate-ndistinct-coefficients.patch)
download | inline diff:
From 238d8994c85d8b64bd898604b6ad1219850a5a26 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@pgaddict.com>
Date: Wed, 23 Dec 2015 02:07:58 +0100
Subject: [PATCH 7/9] multivariate ndistinct coefficients
---
doc/src/sgml/ref/create_statistics.sgml | 9 ++
src/backend/catalog/system_views.sql | 3 +-
src/backend/commands/analyze.c | 2 +-
src/backend/commands/statscmds.c | 11 +-
src/backend/optimizer/path/clausesel.c | 4 +
src/backend/optimizer/util/plancat.c | 4 +-
src/backend/utils/adt/selfuncs.c | 93 +++++++++++++++-
src/backend/utils/mvstats/Makefile | 2 +-
src/backend/utils/mvstats/README.ndistinct | 83 ++++++++++++++
src/backend/utils/mvstats/README.stats | 2 +
src/backend/utils/mvstats/common.c | 23 +++-
src/backend/utils/mvstats/mvdist.c | 171 +++++++++++++++++++++++++++++
src/include/catalog/pg_mv_statistic.h | 26 +++--
src/include/nodes/relation.h | 2 +
src/include/utils/mvstats.h | 9 +-
src/test/regress/expected/rules.out | 3 +-
16 files changed, 424 insertions(+), 23 deletions(-)
create mode 100644 src/backend/utils/mvstats/README.ndistinct
create mode 100644 src/backend/utils/mvstats/mvdist.c
diff --git a/doc/src/sgml/ref/create_statistics.sgml b/doc/src/sgml/ref/create_statistics.sgml
index f7336fd..80e472f 100644
--- a/doc/src/sgml/ref/create_statistics.sgml
+++ b/doc/src/sgml/ref/create_statistics.sgml
@@ -168,6 +168,15 @@ CREATE STATISTICS [ IF NOT EXISTS ] <replaceable class="PARAMETER">statistics_na
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>ndistinct</> (<type>boolean</>)</term>
+ <listitem>
+ <para>
+ Enables ndistinct coefficients for the statistics.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect2>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index b151db1..8d2b435 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -169,7 +169,8 @@ CREATE VIEW pg_mv_stats AS
length(S.stamcv) AS mcvbytes,
pg_mv_stats_mcvlist_info(S.stamcv) AS mcvinfo,
length(S.stahist) AS histbytes,
- pg_mv_stats_histogram_info(S.stahist) AS histinfo
+ pg_mv_stats_histogram_info(S.stahist) AS histinfo,
+ standcoeff AS ndcoeff
FROM (pg_mv_statistic S JOIN pg_class C ON (C.oid = S.starelid))
LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace);
diff --git a/src/backend/commands/analyze.c b/src/backend/commands/analyze.c
index 9087532..c29f1be 100644
--- a/src/backend/commands/analyze.c
+++ b/src/backend/commands/analyze.c
@@ -582,7 +582,7 @@ do_analyze_rel(Relation onerel, int options, VacuumParams *params,
}
/* Build multivariate stats (if there are any). */
- build_mv_stats(onerel, numrows, rows, attr_cnt, vacattrstats);
+ build_mv_stats(onerel, totalrows, numrows, rows, attr_cnt, vacattrstats);
}
/*
diff --git a/src/backend/commands/statscmds.c b/src/backend/commands/statscmds.c
index e0b085f..a7c569d 100644
--- a/src/backend/commands/statscmds.c
+++ b/src/backend/commands/statscmds.c
@@ -72,7 +72,8 @@ CreateStatistics(CreateStatsStmt *stmt)
/* by default build nothing */
bool build_dependencies = false,
build_mcv = false,
- build_histogram = false;
+ build_histogram = false,
+ build_ndistinct = false;
int32 max_buckets = -1,
max_mcv_items = -1;
@@ -155,6 +156,8 @@ CreateStatistics(CreateStatsStmt *stmt)
if (strcmp(opt->defname, "dependencies") == 0)
build_dependencies = defGetBoolean(opt);
+ else if (strcmp(opt->defname, "ndistinct") == 0)
+ build_ndistinct = defGetBoolean(opt);
else if (strcmp(opt->defname, "mcv") == 0)
build_mcv = defGetBoolean(opt);
else if (strcmp(opt->defname, "max_mcv_items") == 0)
@@ -209,10 +212,10 @@ CreateStatistics(CreateStatsStmt *stmt)
}
/* check that at least some statistics were requested */
- if (! (build_dependencies || build_mcv || build_histogram))
+ if (! (build_dependencies || build_mcv || build_histogram || build_ndistinct))
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("no statistics type (dependencies, mcv, histogram) was requested")));
+ errmsg("no statistics type (dependencies, mcv, histogram, ndistinct) was requested")));
/* now do some checking of the options */
if (require_mcv && (! build_mcv))
@@ -246,6 +249,7 @@ CreateStatistics(CreateStatsStmt *stmt)
values[Anum_pg_mv_statistic_deps_enabled -1] = BoolGetDatum(build_dependencies);
values[Anum_pg_mv_statistic_mcv_enabled -1] = BoolGetDatum(build_mcv);
values[Anum_pg_mv_statistic_hist_enabled -1] = BoolGetDatum(build_histogram);
+ values[Anum_pg_mv_statistic_ndist_enabled-1] = BoolGetDatum(build_ndistinct);
values[Anum_pg_mv_statistic_mcv_max_items -1] = Int32GetDatum(max_mcv_items);
values[Anum_pg_mv_statistic_hist_max_buckets -1] = Int32GetDatum(max_buckets);
@@ -253,6 +257,7 @@ CreateStatistics(CreateStatsStmt *stmt)
nulls[Anum_pg_mv_statistic_stadeps -1] = true;
nulls[Anum_pg_mv_statistic_stamcv -1] = true;
nulls[Anum_pg_mv_statistic_stahist -1] = true;
+ nulls[Anum_pg_mv_statistic_standist -1] = true;
/* insert the tuple into pg_mv_statistic */
mvstatrel = heap_open(MvStatisticRelationId, RowExclusiveLock);
diff --git a/src/backend/optimizer/path/clausesel.c b/src/backend/optimizer/path/clausesel.c
index 14e3444..63baa73 100644
--- a/src/backend/optimizer/path/clausesel.c
+++ b/src/backend/optimizer/path/clausesel.c
@@ -59,6 +59,7 @@ static void addRangeClause(RangeQueryClause **rqlist, Node *clause,
#define MV_CLAUSE_TYPE_FDEP 0x01
#define MV_CLAUSE_TYPE_MCV 0x02
#define MV_CLAUSE_TYPE_HIST 0x04
+#define MV_CLAUSE_TYPE_NDIST 0x08
static bool clause_is_mv_compatible(Node *clause, Index relid, Bitmapset **attnums,
int type);
@@ -2860,6 +2861,9 @@ stats_type_matches(MVStatisticInfo *stat, int type)
if ((type & MV_CLAUSE_TYPE_HIST) && stat->hist_built)
return true;
+ if ((type & MV_CLAUSE_TYPE_NDIST) && stat->ndist_built)
+ return true;
+
return false;
}
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index 2519249..3741b7a 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -412,7 +412,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
mvstat = (Form_pg_mv_statistic) GETSTRUCT(htup);
/* unavailable stats are not interesting for the planner */
- if (mvstat->deps_built || mvstat->mcv_built || mvstat->hist_built)
+ if (mvstat->deps_built || mvstat->mcv_built || mvstat->hist_built || mvstat->ndist_built)
{
info = makeNode(MVStatisticInfo);
@@ -423,11 +423,13 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->deps_enabled = mvstat->deps_enabled;
info->mcv_enabled = mvstat->mcv_enabled;
info->hist_enabled = mvstat->hist_enabled;
+ info->ndist_enabled = mvstat->ndist_enabled;
/* built/available statistics */
info->deps_built = mvstat->deps_built;
info->mcv_built = mvstat->mcv_built;
info->hist_built = mvstat->hist_built;
+ info->ndist_built = mvstat->ndist_built;
/* stakeys */
adatum = SysCacheGetAttr(MVSTATOID, htup,
diff --git a/src/backend/utils/adt/selfuncs.c b/src/backend/utils/adt/selfuncs.c
index 805d633..f8d39aa 100644
--- a/src/backend/utils/adt/selfuncs.c
+++ b/src/backend/utils/adt/selfuncs.c
@@ -132,6 +132,7 @@
#include "utils/fmgroids.h"
#include "utils/index_selfuncs.h"
#include "utils/lsyscache.h"
+#include "utils/mvstats.h"
#include "utils/nabstime.h"
#include "utils/pg_locale.h"
#include "utils/rel.h"
@@ -206,6 +207,7 @@ static Const *string_to_const(const char *str, Oid datatype);
static Const *string_to_bytea_const(const char *str, size_t str_len);
static List *add_predicate_to_quals(IndexOptInfo *index, List *indexQuals);
+static Oid find_ndistinct_coeff(PlannerInfo *root, RelOptInfo *rel, List *varinfos);
/*
* eqsel - Selectivity of "=" for any data types.
@@ -3423,12 +3425,26 @@ estimate_num_groups(PlannerInfo *root, List *groupExprs, double input_rows,
* don't know by how much. We should never clamp to less than the
* largest ndistinct value for any of the Vars, though, since
* there will surely be at least that many groups.
+ *
+ * However we don't need to do this if we have ndistinct stats on
+ * the columns - in that case we can simply use the coefficient
+ * to get the (probably way more accurate) estimate.
+ *
+ * XXX Probably needs refactoring (don't like to mix with clamp
+ * and coeff at the same time).
*/
double clamp = rel->tuples;
+ double coeff = 1.0;
if (relvarcount > 1)
{
- clamp *= 0.1;
+ Oid oid = find_ndistinct_coeff(root, rel, varinfos);
+
+ if (oid != InvalidOid)
+ coeff = load_mv_ndistinct(oid);
+ else
+ clamp *= 0.1;
+
if (clamp < relmaxndistinct)
{
clamp = relmaxndistinct;
@@ -3437,6 +3453,13 @@ estimate_num_groups(PlannerInfo *root, List *groupExprs, double input_rows,
clamp = rel->tuples;
}
}
+
+ /*
+ * Apply ndistinct coefficient from multivar stats (we must do this
+ * before clamping the estimate in any way.
+ */
+ reldistinct /= coeff;
+
if (reldistinct > clamp)
reldistinct = clamp;
@@ -7583,3 +7606,71 @@ brincostestimate(PlannerInfo *root, IndexPath *path, double loop_count,
/* XXX what about pages_per_range? */
}
+
+/*
+ * Find applicable ndistinct statistics and compute the coefficient to
+ * correct the estimate (simply a product of per-column ndistincts).
+ *
+ * Currently we only look for a perfect match, i.e. a single ndistinct
+ * estimate exactly matching all the columns of the statistics.
+ */
+static Oid
+find_ndistinct_coeff(PlannerInfo *root, RelOptInfo *rel, List *varinfos)
+{
+ ListCell *lc;
+ Bitmapset *attnums = NULL;
+ VariableStatData vardata;
+
+ foreach(lc, varinfos)
+ {
+ GroupVarInfo *varinfo = (GroupVarInfo *) lfirst(lc);
+
+ if (varinfo->rel != rel)
+ continue;
+
+ /* FIXME handle expressions in general only */
+
+ /*
+ * examine the variable (or expression) so that we know which
+ * attribute we're dealing with - we need this for matching the
+ * ndistinct coefficient
+ *
+ * FIXME probably might remember this from estimate_num_groups
+ */
+ examine_variable(root, varinfo->var, 0, &vardata);
+
+ if (HeapTupleIsValid(vardata.statsTuple))
+ {
+ Form_pg_statistic stats
+ = (Form_pg_statistic) GETSTRUCT(vardata.statsTuple);
+
+ attnums = bms_add_member(attnums, stats->staattnum);
+
+ ReleaseVariableStats(vardata);
+ }
+ }
+
+ /* look for a matching ndistinct statistics */
+ foreach (lc, rel->mvstatlist)
+ {
+ int i;
+ MVStatisticInfo *info = (MVStatisticInfo *)lfirst(lc);
+
+ /* skip statistics without ndistinct coefficient built */
+ if (!info->ndist_built)
+ continue;
+
+ /* only exact matches for now (same set of columns) */
+ if (bms_num_members(attnums) != info->stakeys->dim1)
+ continue;
+
+ /* check that the columns match */
+ for (i = 0; i < info->stakeys->dim1; i++)
+ if (bms_is_member(info->stakeys->values[i], attnums))
+ continue;
+
+ return info->mvoid;
+ }
+
+ return InvalidOid;
+}
diff --git a/src/backend/utils/mvstats/Makefile b/src/backend/utils/mvstats/Makefile
index 9dbb3b6..d4b88e9 100644
--- a/src/backend/utils/mvstats/Makefile
+++ b/src/backend/utils/mvstats/Makefile
@@ -12,6 +12,6 @@ subdir = src/backend/utils/mvstats
top_builddir = ../../../..
include $(top_builddir)/src/Makefile.global
-OBJS = common.o dependencies.o histogram.o mcv.o
+OBJS = common.o dependencies.o histogram.o mcv.o mvdist.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/utils/mvstats/README.ndistinct b/src/backend/utils/mvstats/README.ndistinct
new file mode 100644
index 0000000..32d1624
--- /dev/null
+++ b/src/backend/utils/mvstats/README.ndistinct
@@ -0,0 +1,83 @@
+ndistinct coefficients
+======================
+
+Estimating number of distinct groups in a combination of columns is tricky,
+and the estimation error is often significant. By ndistinct coefficient we
+mean a ratio
+
+ q = ndistinct(a) * ndistinct(b) / ndistinct(a,b)
+
+where 'a' and 'b' are columns, ndistinct(a) is (an estimate of) a number of
+distinct values in column 'a'. And ndistinct(a,b) is the same thing for the
+pair of columns.
+
+The meaning of the coefficient may be illustrated by answering the following
+question: Given a combination of columns (a,b), how many distinct values of 'b'
+matches a chosen value of 'a' on average?
+
+Let's assume we know ndistinct(a) and ndistinct(a,b). Then the answer to the
+question clearly is
+
+ ndistinct(a,b) / ndistinct(a)
+
+and by using 'q' we may rewrite this as
+
+ ndistinct(b) / q
+
+so 'q' may be considered as a correction factor of the ndistinct estimate given
+a condition on one of the columns.
+
+This may be generalized to a combination of 'n' columns
+
+ [ndistinct(c1) * ... * ndistinct(cn)] / ndistinct(c1, ..., cn)
+
+and the meaning is very similar, except that we need to use conditions on (n-1)
+of the columns.
+
+
+Selectivity estimation
+----------------------
+
+As explained in the previous paragraph, ndistinct coefficients may be used to
+estimate cardinality of a column, given some apriori knowledge. Let's assume
+we need to estimate selectivity of a condition
+
+ (a=1) AND (b=2)
+
+which we can expand like this
+
+ P(a=1 & b=2) = P(a=1) * P(b=2 | a=1)
+
+Let's also assume that the distribution of 'b' is uniform, i.e. that
+
+ P(a=1) = 1/ndistinct(a)
+ P(b=2) = 1/ndistinct(b)
+ P(a=1 & b=2) = 1/ndistinct(a,b)
+
+ P(b=2 | a=1) = ndistinct(a) / ndistinct(a,b)
+
+which may be rewritten like
+
+ P(b=2 | a=1)
+ = ndistinct(a,b) / ndistinct(a)
+ = (1/ndistinct(b)) * [(ndistinct(a) * ndistinct(b)) / ndistinct(a,b)]
+ = (1/ndistinct(b)) * q
+
+and therefore
+
+ P(a=1 & b=2) = (1/ndistinct(a)) * (1/ndistinct(b)) * q
+
+This also illustrates 'q' as a correction coefficient.
+
+It also explains why we store the coefficient and not simply ndistinct(a,b).
+This way we can simply estimate individual clauses and then simply correct
+the estimate by multiplying the result with 'q' - we don't have to mess with
+ndistinct estimates at all.
+
+Naturally, as the coefficient is derives from ndistinct(a,b), it may be also
+used to estimate GROUP BY clauses on the combination of columns, replacing the
+existing heuristics in estimate_num_groups().
+
+Note: Currently only the GROUP BY estimation is implemented. It's a bit unclear
+how to implement the clause estimation when there are other statistics (esp.
+MCV lists and/or functional dependencies) available.
diff --git a/src/backend/utils/mvstats/README.stats b/src/backend/utils/mvstats/README.stats
index d404914..6d4b09b 100644
--- a/src/backend/utils/mvstats/README.stats
+++ b/src/backend/utils/mvstats/README.stats
@@ -20,6 +20,8 @@ Currently we only have two kinds of multivariate statistics
(c) multivariate histograms (README.histogram)
+ (d) ndistinct coefficients
+
Compatible clause types
-----------------------
diff --git a/src/backend/utils/mvstats/common.c b/src/backend/utils/mvstats/common.c
index f6d1074..d34d072 100644
--- a/src/backend/utils/mvstats/common.c
+++ b/src/backend/utils/mvstats/common.c
@@ -32,7 +32,8 @@ static List* list_mv_stats(Oid relid);
* and serializes them back into the catalog (as bytea values).
*/
void
-build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
+build_mv_stats(Relation onerel, double totalrows,
+ int numrows, HeapTuple *rows,
int natts, VacAttrStats **vacattrstats)
{
ListCell *lc;
@@ -53,6 +54,7 @@ build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
MVDependencies deps = NULL;
MCVList mcvlist = NULL;
MVHistogram histogram = NULL;
+ double ndist = -1;
int numrows_filtered = numrows;
VacAttrStats **stats = NULL;
@@ -92,6 +94,9 @@ build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
if (stat->deps_enabled)
deps = build_mv_dependencies(numrows, rows, attrs, stats);
+ if (stat->ndist_enabled)
+ ndist = build_mv_ndistinct(totalrows, numrows, rows, attrs, stats);
+
/* build the MCV list */
if (stat->mcv_enabled)
mcvlist = build_mv_mcvlist(numrows, rows, attrs, stats, &numrows_filtered);
@@ -101,7 +106,7 @@ build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
histogram = build_mv_histogram(numrows_filtered, rows, attrs, stats, numrows);
/* store the histogram / MCV list in the catalog */
- update_mv_stats(stat->mvoid, deps, mcvlist, histogram, attrs, stats);
+ update_mv_stats(stat->mvoid, deps, mcvlist, histogram, ndist, attrs, stats);
}
}
@@ -183,6 +188,8 @@ list_mv_stats(Oid relid)
info->mcv_built = stats->mcv_built;
info->hist_enabled = stats->hist_enabled;
info->hist_built = stats->hist_built;
+ info->ndist_enabled = stats->ndist_enabled;
+ info->ndist_built = stats->ndist_built;
result = lappend(result, info);
}
@@ -252,7 +259,7 @@ find_mv_attnums(Oid mvoid, Oid *relid)
void
update_mv_stats(Oid mvoid,
MVDependencies dependencies, MCVList mcvlist, MVHistogram histogram,
- int2vector *attrs, VacAttrStats **stats)
+ double ndistcoeff, int2vector *attrs, VacAttrStats **stats)
{
HeapTuple stup,
oldtup;
@@ -292,26 +299,36 @@ update_mv_stats(Oid mvoid,
= PointerGetDatum(data);
}
+ if (ndistcoeff > 1.0)
+ {
+ nulls[Anum_pg_mv_statistic_standist -1] = false;
+ values[Anum_pg_mv_statistic_standist-1] = Float8GetDatum(ndistcoeff);
+ }
+
/* always replace the value (either by bytea or NULL) */
replaces[Anum_pg_mv_statistic_stadeps -1] = true;
replaces[Anum_pg_mv_statistic_stamcv -1] = true;
replaces[Anum_pg_mv_statistic_stahist-1] = true;
+ replaces[Anum_pg_mv_statistic_standist-1] = true;
/* always change the availability flags */
nulls[Anum_pg_mv_statistic_deps_built -1] = false;
nulls[Anum_pg_mv_statistic_mcv_built -1] = false;
nulls[Anum_pg_mv_statistic_hist_built-1] = false;
+ nulls[Anum_pg_mv_statistic_ndist_built-1] = false;
nulls[Anum_pg_mv_statistic_stakeys-1] = false;
/* use the new attnums, in case we removed some dropped ones */
replaces[Anum_pg_mv_statistic_deps_built-1] = true;
replaces[Anum_pg_mv_statistic_mcv_built -1] = true;
+ replaces[Anum_pg_mv_statistic_ndist_built-1] = true;
replaces[Anum_pg_mv_statistic_hist_built -1] = true;
replaces[Anum_pg_mv_statistic_stakeys -1] = true;
values[Anum_pg_mv_statistic_deps_built-1] = BoolGetDatum(dependencies != NULL);
values[Anum_pg_mv_statistic_mcv_built -1] = BoolGetDatum(mcvlist != NULL);
values[Anum_pg_mv_statistic_hist_built -1] = BoolGetDatum(histogram != NULL);
+ values[Anum_pg_mv_statistic_ndist_built-1] = BoolGetDatum(ndistcoeff > 1.0);
values[Anum_pg_mv_statistic_stakeys -1] = PointerGetDatum(attrs);
/* Is there already a pg_mv_statistic tuple for this attribute? */
diff --git a/src/backend/utils/mvstats/mvdist.c b/src/backend/utils/mvstats/mvdist.c
new file mode 100644
index 0000000..59b8358
--- /dev/null
+++ b/src/backend/utils/mvstats/mvdist.c
@@ -0,0 +1,171 @@
+/*-------------------------------------------------------------------------
+ *
+ * mvdist.c
+ * POSTGRES multivariate distinct coefficients
+ *
+ *
+ * Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/utils/mvstats/mvdist.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include <math.h>
+
+#include "common.h"
+#include "utils/lsyscache.h"
+
+static double estimate_ndistinct(double totalrows, int numrows, int d, int f1);
+
+/*
+ * Compute ndistinct coefficient for the combination of attributes. This
+ * computes the ndistinct estimate using the same estimator used in analyze.c
+ * and then computes the coefficient.
+ */
+double
+build_mv_ndistinct(double totalrows, int numrows, HeapTuple *rows,
+ int2vector *attrs, VacAttrStats **stats)
+{
+ int i, j;
+ int f1, cnt, d;
+ int nmultiple, summultiple;
+ int numattrs = attrs->dim1;
+ MultiSortSupport mss = multi_sort_init(numattrs);
+ double ndistcoeff;
+
+ /*
+ * It's possible to sort the sample rows directly, but this seemed
+ * somehow simpler / less error prone. Another option would be to
+ * allocate the arrays for each SortItem separately, but that'd be
+ * significant overhead (not just CPU, but especially memory bloat).
+ */
+ SortItem * items = (SortItem*)palloc0(numrows * sizeof(SortItem));
+
+ Datum *values = (Datum*)palloc0(sizeof(Datum) * numrows * numattrs);
+ bool *isnull = (bool*)palloc0(sizeof(bool) * numrows * numattrs);
+
+ for (i = 0; i < numrows; i++)
+ {
+ items[i].values = &values[i * numattrs];
+ items[i].isnull = &isnull[i * numattrs];
+ }
+
+ Assert(numattrs >= 2);
+
+ for (i = 0; i < numattrs; i++)
+ {
+ /* prepare the sort function for the first dimension */
+ multi_sort_add_dimension(mss, i, i, stats);
+
+ /* accumulate all the data into the array and sort it */
+ for (j = 0; j < numrows; j++)
+ {
+ items[j].values[i]
+ = heap_getattr(rows[j], attrs->values[i],
+ stats[i]->tupDesc, &items[j].isnull[i]);
+ }
+ }
+
+ qsort_arg((void *) items, numrows, sizeof(SortItem),
+ multi_sort_compare, mss);
+
+ /* count number of distinct combinations */
+
+ f1 = 0;
+ cnt = 1;
+ d = 1;
+ for (i = 1; i < numrows; i++)
+ {
+ if (multi_sort_compare(&items[i], &items[i-1], mss) != 0)
+ {
+ if (cnt == 1)
+ f1 += 1;
+ else
+ {
+ nmultiple += 1;
+ summultiple += cnt;
+ }
+
+ d++;
+ cnt = 0;
+ }
+
+ cnt += 1;
+ }
+
+ if (cnt == 1)
+ f1 += 1;
+ else
+ {
+ nmultiple += 1;
+ summultiple += cnt;
+ }
+
+ ndistcoeff = 1 / estimate_ndistinct(totalrows, numrows, d, f1);
+
+ /*
+ * now count distinct values for each attribute and incrementally
+ * compute ndistinct(a,b) / (ndistinct(a) * ndistinct(b))
+ *
+ * FIXME Probably need to handle cases when one of the ndistinct
+ * estimates is negative, and also check that the combined
+ * ndistinct is greater than any of those partial values.
+ */
+ for (i = 0; i < numattrs; i++)
+ ndistcoeff *= stats[i]->stadistinct;
+
+ return ndistcoeff;
+}
+
+double
+load_mv_ndistinct(Oid mvoid)
+{
+ bool isnull = false;
+ Datum deps;
+
+ /* Prepare to scan pg_mv_statistic for entries having indrelid = this rel. */
+ HeapTuple htup = SearchSysCache1(MVSTATOID, ObjectIdGetDatum(mvoid));
+
+#ifdef USE_ASSERT_CHECKING
+ Form_pg_mv_statistic mvstat = (Form_pg_mv_statistic) GETSTRUCT(htup);
+ Assert(mvstat->ndist_enabled && mvstat->ndist_built);
+#endif
+
+ deps = SysCacheGetAttr(MVSTATOID, htup,
+ Anum_pg_mv_statistic_standist, &isnull);
+
+ Assert(!isnull);
+
+ ReleaseSysCache(htup);
+
+ return DatumGetFloat8(deps);
+}
+
+/* The Duj1 estimator (already used in analyze.c). */
+static double
+estimate_ndistinct(double totalrows, int numrows, int d, int f1)
+{
+ double numer,
+ denom,
+ ndistinct;
+
+ numer = (double) numrows *(double) d;
+
+ denom = (double) (numrows - f1) +
+ (double) f1 * (double) numrows / totalrows;
+
+ ndistinct = numer / denom;
+
+ /* Clamp to sane range in case of roundoff error */
+ if (ndistinct < (double) d)
+ ndistinct = (double) d;
+
+ if (ndistinct > totalrows)
+ ndistinct = totalrows;
+
+ return floor(ndistinct + 0.5);
+}
diff --git a/src/include/catalog/pg_mv_statistic.h b/src/include/catalog/pg_mv_statistic.h
index 7020772..e46cc6b 100644
--- a/src/include/catalog/pg_mv_statistic.h
+++ b/src/include/catalog/pg_mv_statistic.h
@@ -40,6 +40,7 @@ CATALOG(pg_mv_statistic,3381)
bool deps_enabled; /* analyze dependencies? */
bool mcv_enabled; /* build MCV list? */
bool hist_enabled; /* build histogram? */
+ bool ndist_enabled; /* build ndist coefficient? */
/* histogram / MCV size */
int32 mcv_max_items; /* max MCV items */
@@ -49,6 +50,7 @@ CATALOG(pg_mv_statistic,3381)
bool deps_built; /* dependencies were built */
bool mcv_built; /* MCV list was built */
bool hist_built; /* histogram was built */
+ bool ndist_built; /* ndistinct coeff built */
/* variable-length fields start here, but we allow direct access to stakeys */
int2vector stakeys; /* array of column keys */
@@ -57,6 +59,7 @@ CATALOG(pg_mv_statistic,3381)
bytea stadeps; /* dependencies (serialized) */
bytea stamcv; /* MCV list (serialized) */
bytea stahist; /* MV histogram (serialized) */
+ float8 standcoeff; /* ndistinct coeff (serialized) */
#endif
} FormData_pg_mv_statistic;
@@ -72,7 +75,7 @@ typedef FormData_pg_mv_statistic *Form_pg_mv_statistic;
* compiler constants for pg_mv_statistic
* ----------------
*/
-#define Natts_pg_mv_statistic 16
+#define Natts_pg_mv_statistic 19
#define Anum_pg_mv_statistic_starelid 1
#define Anum_pg_mv_statistic_staname 2
#define Anum_pg_mv_statistic_stanamespace 3
@@ -80,14 +83,17 @@ typedef FormData_pg_mv_statistic *Form_pg_mv_statistic;
#define Anum_pg_mv_statistic_deps_enabled 5
#define Anum_pg_mv_statistic_mcv_enabled 6
#define Anum_pg_mv_statistic_hist_enabled 7
-#define Anum_pg_mv_statistic_mcv_max_items 8
-#define Anum_pg_mv_statistic_hist_max_buckets 9
-#define Anum_pg_mv_statistic_deps_built 10
-#define Anum_pg_mv_statistic_mcv_built 11
-#define Anum_pg_mv_statistic_hist_built 12
-#define Anum_pg_mv_statistic_stakeys 13
-#define Anum_pg_mv_statistic_stadeps 14
-#define Anum_pg_mv_statistic_stamcv 15
-#define Anum_pg_mv_statistic_stahist 16
+#define Anum_pg_mv_statistic_ndist_enabled 8
+#define Anum_pg_mv_statistic_mcv_max_items 9
+#define Anum_pg_mv_statistic_hist_max_buckets 19
+#define Anum_pg_mv_statistic_deps_built 11
+#define Anum_pg_mv_statistic_mcv_built 12
+#define Anum_pg_mv_statistic_hist_built 13
+#define Anum_pg_mv_statistic_ndist_built 14
+#define Anum_pg_mv_statistic_stakeys 15
+#define Anum_pg_mv_statistic_stadeps 16
+#define Anum_pg_mv_statistic_stamcv 17
+#define Anum_pg_mv_statistic_stahist 18
+#define Anum_pg_mv_statistic_standist 19
#endif /* PG_MV_STATISTIC_H */
diff --git a/src/include/nodes/relation.h b/src/include/nodes/relation.h
index 84be0ce..ba587da 100644
--- a/src/include/nodes/relation.h
+++ b/src/include/nodes/relation.h
@@ -657,11 +657,13 @@ typedef struct MVStatisticInfo
bool deps_enabled; /* functional dependencies enabled */
bool mcv_enabled; /* MCV list enabled */
bool hist_enabled; /* histogram enabled */
+ bool ndist_enabled; /* ndistinct coefficient enabled */
/* built/available statistics */
bool deps_built; /* functional dependencies built */
bool mcv_built; /* MCV list built */
bool hist_built; /* histogram built */
+ bool ndist_built; /* ndistinct coefficient built */
/* columns in the statistics (attnums) */
int2vector *stakeys; /* attnums of the columns covered */
diff --git a/src/include/utils/mvstats.h b/src/include/utils/mvstats.h
index 2b67772..67ed2f8 100644
--- a/src/include/utils/mvstats.h
+++ b/src/include/utils/mvstats.h
@@ -225,6 +225,7 @@ typedef MVSerializedHistogramData *MVSerializedHistogram;
MVDependencies load_mv_dependencies(Oid mvoid);
MCVList load_mv_mcvlist(Oid mvoid);
MVSerializedHistogram load_mv_histogram(Oid mvoid);
+double load_mv_ndistinct(Oid mvoid);
bytea * serialize_mv_dependencies(MVDependencies dependencies);
bytea * serialize_mv_mcvlist(MCVList mcvlist, int2vector *attrs,
@@ -266,11 +267,17 @@ MVHistogram
build_mv_histogram(int numrows, HeapTuple *rows, int2vector *attrs,
VacAttrStats **stats, int numrows_total);
-void build_mv_stats(Relation onerel, int numrows, HeapTuple *rows,
+double
+build_mv_ndistinct(double totalrows, int numrows, HeapTuple *rows,
+ int2vector *attrs, VacAttrStats **stats);
+
+void build_mv_stats(Relation onerel, double totalrows,
+ int numrows, HeapTuple *rows,
int natts, VacAttrStats **vacattrstats);
void update_mv_stats(Oid relid, MVDependencies dependencies,
MCVList mcvlist, MVHistogram histogram,
+ double ndistcoeff,
int2vector *attrs, VacAttrStats **stats);
#ifdef DEBUG_MVHIST
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 528ac36..7a914da 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1377,7 +1377,8 @@ pg_mv_stats| SELECT n.nspname AS schemaname,
length(s.stamcv) AS mcvbytes,
pg_mv_stats_mcvlist_info(s.stamcv) AS mcvinfo,
length(s.stahist) AS histbytes,
- pg_mv_stats_histogram_info(s.stahist) AS histinfo
+ pg_mv_stats_histogram_info(s.stahist) AS histinfo,
+ s.standcoeff AS ndcoeff
FROM ((pg_mv_statistic s
JOIN pg_class c ON ((c.oid = s.starelid)))
LEFT JOIN pg_namespace n ON ((n.oid = c.relnamespace)));
--
2.5.0
[text/x-patch] 0008-change-how-we-apply-selectivity-to-number-of-groups-.patch (1.5K, ../../7aa6145f-302b-2077-b961-702cc61710bb@2ndquadrant.com/9-0008-change-how-we-apply-selectivity-to-number-of-groups-.patch)
download | inline diff:
From c968f05c26ecfa9344a8a9c9209bd755fa4ddf7b Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@pgaddict.com>
Date: Tue, 26 Jan 2016 18:14:33 +0100
Subject: [PATCH 8/9] change how we apply selectivity to number of groups
estimate
Instead of simply multiplying the ndistinct estimate with selecticity,
we instead use the formula for the expected number of distinct values
observed in 'k' rows when there are 'd' distinct values in the bin
d * (1 - ((d - 1) / d)^k)
This is 'with replacements' which seems appropriate for the use, and it
mostly assumes uniform distribution of the distinct values. So if the
distribution is not uniform (e.g. there are very frequent groups) this
may be less accurate than the current algorithm in some cases, giving
over-estimates. But that's probably better than OOM.
---
src/backend/utils/adt/selfuncs.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/src/backend/utils/adt/selfuncs.c b/src/backend/utils/adt/selfuncs.c
index f8d39aa..76be0e3 100644
--- a/src/backend/utils/adt/selfuncs.c
+++ b/src/backend/utils/adt/selfuncs.c
@@ -3464,9 +3464,9 @@ estimate_num_groups(PlannerInfo *root, List *groupExprs, double input_rows,
reldistinct = clamp;
/*
- * Multiply by restriction selectivity.
+ * Estimate the number of distinct values observed in rel->rows.
*/
- reldistinct *= rel->rows / rel->tuples;
+ reldistinct *= (1 - powl(1 - rel->rows/rel->tuples, rel->tuples/reldistinct));
/*
* Update estimate of total distinct groups.
--
2.5.0
[text/x-patch] 0009-fixup-of-regression-tests-plans-changes-by-group-by-.patch (2.3K, ../../7aa6145f-302b-2077-b961-702cc61710bb@2ndquadrant.com/10-0009-fixup-of-regression-tests-plans-changes-by-group-by-.patch)
download | inline diff:
From 29ea451f45fa5b8891ebde195551180f2841826d Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@pgaddict.com>
Date: Sun, 28 Feb 2016 21:16:40 +0100
Subject: [PATCH 9/9] fixup of regression tests (plans changes by group by
estimation)
---
src/test/regress/expected/subselect.out | 25 +++++++++++--------------
1 file changed, 11 insertions(+), 14 deletions(-)
diff --git a/src/test/regress/expected/subselect.out b/src/test/regress/expected/subselect.out
index de64ca7..0fc93d9 100644
--- a/src/test/regress/expected/subselect.out
+++ b/src/test/regress/expected/subselect.out
@@ -807,27 +807,24 @@ select * from int4_tbl where
explain (verbose, costs off)
select * from int4_tbl o where (f1, f1) in
(select f1, generate_series(1,2) / 10 g from int4_tbl i group by f1);
- QUERY PLAN
-----------------------------------------------------------------------
- Hash Join
+ QUERY PLAN
+----------------------------------------------------------------
+ Hash Semi Join
Output: o.f1
Hash Cond: (o.f1 = "ANY_subquery".f1)
-> Seq Scan on public.int4_tbl o
Output: o.f1
-> Hash
Output: "ANY_subquery".f1, "ANY_subquery".g
- -> HashAggregate
+ -> Subquery Scan on "ANY_subquery"
Output: "ANY_subquery".f1, "ANY_subquery".g
- Group Key: "ANY_subquery".f1, "ANY_subquery".g
- -> Subquery Scan on "ANY_subquery"
- Output: "ANY_subquery".f1, "ANY_subquery".g
- Filter: ("ANY_subquery".f1 = "ANY_subquery".g)
- -> HashAggregate
- Output: i.f1, (generate_series(1, 2) / 10)
- Group Key: i.f1
- -> Seq Scan on public.int4_tbl i
- Output: i.f1
-(18 rows)
+ Filter: ("ANY_subquery".f1 = "ANY_subquery".g)
+ -> HashAggregate
+ Output: i.f1, (generate_series(1, 2) / 10)
+ Group Key: i.f1
+ -> Seq Scan on public.int4_tbl i
+ Output: i.f1
+(15 rows)
select * from int4_tbl o where (f1, f1) in
(select f1, generate_series(1,2) / 10 g from int4_tbl i group by f1);
--
2.5.0
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: multivariate statistics v14
@ 2016-03-16 13:45 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Tatsuo Ishii <ishii@postgresql.org>
0 siblings, 1 reply; 58+ messages in thread
From: Tomas Vondra @ 2016-03-16 13:45 UTC (permalink / raw)
To: Tatsuo Ishii <ishii@postgresql.org>; +Cc: jeff.janes@gmail.com; alvherre@2ndquadrant.com; pgsql-hackers
Hi,
On 03/16/2016 03:58 AM, Tatsuo Ishii wrote:
> I apology if it's already discussed. I am new to this patch.
>
>> Attached is v15 of the patch series, fixing this and also doing quite a
>> few additional improvements:
>>
>> * added some basic examples into the SGML documentation
>>
>> * addressing the objectaddress omissions, as pointed out by Alvaro
>>
>> * support for ALTER STATISTICS ... OWNER TO / RENAME / SET SCHEMA
>>
>> * significant refactoring of MCV and histogram code, particularly
>> serialization, deserialization and building
>>
>> * reworking the functional dependencies to support more complex
>> dependencies, with multiple columns as 'conditions'
>>
>> * the reduction using functional dependencies is also significantly
>> simplified (I decided to get rid of computing the transitive closure
>> for now - it got too complex after the multi-condition dependencies,
>> so I'll leave that for the future
>
> Do you have any other missing parts in this work? I am asking
> because I wonder if you want to push this into 9.6 or rather 9.7.
I think the first few parts of the patch series, namely:
* shared infrastructure (0002)
* functional dependencies (0003)
* MCV lists (0004)
* histograms (0005)
might make it into 9.6. I believe the code for building and storing the
different kinds of stats is reasonably solid. What probably needs more
thorough review are the changes in clauselist_selectivity(), but the
code in these parts is reasonably simple as it only supports using a
single multi-variate statistics per relation.
The part (0006) that allows using multiple statistics (i.e. selects
which of the available stats to use and in what order) is probably the
most complex part of the whole patch, and I myself do have some
questions about some aspects of it. I don't think this part might get
into 9.6 at this point (although it'd be nice if we managed to do that).
I can also imagine moving the ndistinct pieces forward, in front of 0006
if that helps getting it into 9.6. There's a bit more work on making it
more flexible, though, to allow handling subsets columns (currently we
need a perfect match).
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: multivariate statistics v14
@ 2016-03-20 23:00 Tatsuo Ishii <ishii@postgresql.org>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 1 reply; 58+ messages in thread
From: Tatsuo Ishii @ 2016-03-20 23:00 UTC (permalink / raw)
To: tomas.vondra@2ndquadrant.com; +Cc: horiguchi.kyotaro@lab.ntt.co.jp; jeff.janes@gmail.com; alvherre@2ndquadrant.com; pgsql-hackers
>> Many trailing white spaces found.
>
> Sorry, haven't noticed that after one of the rebases. Fixed in the
> attached v15 of the patch.
There are still few of traling spaces.
/home/t-ishii/0002-shared-infrastructure-and-functional-dependencies.patch:3792: trailing whitespace.
/home/t-ishii/0004-multivariate-MCV-lists.patch:471: trailing whitespace.
/home/t-ishii/0004-multivariate-MCV-lists.patch:656: space before tab in indent.
{
/home/t-ishii/0004-multivariate-MCV-lists.patch:682: space before tab in indent.
}
/home/t-ishii/0004-multivariate-MCV-lists.patch:685: space before tab in indent.
{
/home/t-ishii/0004-multivariate-MCV-lists.patch:715: trailing whitespace.
/home/t-ishii/0006-multi-statistics-estimation.patch:2513: trailing whitespace.
Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: multivariate statistics v14
@ 2016-03-20 23:34 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Tatsuo Ishii <ishii@postgresql.org>
0 siblings, 2 replies; 58+ messages in thread
From: Tomas Vondra @ 2016-03-20 23:34 UTC (permalink / raw)
To: Tatsuo Ishii <ishii@postgresql.org>; +Cc: horiguchi.kyotaro@lab.ntt.co.jp; jeff.janes@gmail.com; alvherre@2ndquadrant.com; pgsql-hackers
On 03/21/2016 12:00 AM, Tatsuo Ishii wrote:
>>> Many trailing white spaces found.
>>
>> Sorry, haven't noticed that after one of the rebases. Fixed in the
>> attached v15 of the patch.
>
> There are still few of traling spaces.
>
> /home/t-ishii/0002-shared-infrastructure-and-functional-dependencies.patch:3792: trailing whitespace.
> /home/t-ishii/0004-multivariate-MCV-lists.patch:471: trailing whitespace.
> /home/t-ishii/0004-multivariate-MCV-lists.patch:656: space before tab in indent.
> {
> /home/t-ishii/0004-multivariate-MCV-lists.patch:682: space before tab in indent.
> }
> /home/t-ishii/0004-multivariate-MCV-lists.patch:685: space before tab in indent.
> {
> /home/t-ishii/0004-multivariate-MCV-lists.patch:715: trailing whitespace.
> /home/t-ishii/0006-multi-statistics-estimation.patch:2513: trailing whitespace.
>
> Best regards,
D'oh. Thanks for reporting. Attached is v16, hopefully fixing the few
remaining whitespace issues.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[application/x-compressed-tar] multivariate-stats-v16.tgz (144.9K, ../../1bc2bd16-8e33-9e22-cdb1-9ddc38fedc4c@2ndquadrant.com/2-multivariate-stats-v16.tgz)
download
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: multivariate statistics v14
@ 2016-03-21 03:34 Alvaro Herrera <alvherre@2ndquadrant.com>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
1 sibling, 2 replies; 58+ messages in thread
From: Alvaro Herrera @ 2016-03-21 03:34 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Tatsuo Ishii <ishii@postgresql.org>; horiguchi.kyotaro@lab.ntt.co.jp; jeff.janes@gmail.com; pgsql-hackers
Another skim on 0002:
reference.sgml is missing a call to &alterStatistic.
ObjectProperty[] contains a comment that the ACL is "same as relation",
but is that still correct, given that now stats may be related to more
than one relation? Do we even know what the rules for ACLs on
cross-relation stats are? One very simple way to get around this is to
dictate that all the rels must have the same owner. Perhaps we're not
considering the multi-relation case yet?
We have this FIXME comment in do_analyze_rel:
+ * FIXME This sample sizing is mostly OK when computing stats for
+ * individual columns, but when computing multi-variate stats
+ * for multivariate stats (histograms, mcv, ...) it's rather
+ * insufficient. For stats on multiple columns / complex stats
+ * we need larger sample sizes, because we need to build more
+ * detailed stats (more MCV items / histogram buckets) to get
+ * good accuracy. Maybe it'd be appropriate to use samples
+ * proportional to the table (say, 0.5% - 1%) instead of a
+ * fixed size might be more appropriate. Also, this should be
+ * bound to the requested statistics size - e.g. number of MCV
+ * items or histogram buckets should require several sample
+ * rows per item/bucket (so the sample should be k*size).
Maybe this merits more discussion. Right now we have an upper bound on
how much to scan for analyze; if we introduce the idea of scanning a
percentage of the relation, the time to analyze very large relations
could increase significantly. Do we have an idea of what to do for
this? For instance, a rule that would make me comfortable would say to
scan a sample 3x the current size when you have a mvstats on 3 columns;
then the size of fraction to scan is still bounded. But does that
actually work? From the wording of this comment, I assume you don't
actually know.
In this block (CreateStatistics)
+ /* look for duplicities */
+ for (i = 0; i < numcols; i++)
+ for (j = 0; j < numcols; j++)
+ if ((i != j) && (attnums[i] == attnums[j]))
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_COLUMN),
+ errmsg("duplicate column name in statistics definition")));
isn't it easier to have the inner loop go from i+1 to numcols?
I wonder if this is sensible with multi-relation statistics:
+ /*
+ * Store a dependency too, so that statistics are dropped on DROP TABLE
+ */
+ parentobject.classId = RelationRelationId;
+ parentobject.objectId = ObjectIdGetDatum(RelationGetRelid(rel));
+ parentobject.objectSubId = 0;
+ childobject.classId = MvStatisticRelationId;
+ childobject.objectId = statoid;
+ childobject.objectSubId = 0;
I suppose the idea is to drop the stats if any of the rels they are for
is dropped.
Right after that you create a dependency on the schema. Is that
necessary? Since you have the dependency on the relation, the stats
would be dropped by recursion.
Why are you #include'ing builtins.h everywhere?
RelationGetMVStatList() needs a comment.
Please get rid of common.h. It's totally unlike the way we structure
our header files. We don't keep headers in src/backend; they're all in
src/include. One reason is that the latter gets installed as a whole in
include/server, which this file will not be. This file may be necessary
to build some extensions in the future, for example.
In mvstats.h, please mark function prototypes as "extern".
Many files need a pgindent pass.
--
Álvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: multivariate statistics v14
@ 2016-03-21 09:34 Robert Haas <robertmhaas@gmail.com>
parent: Alvaro Herrera <alvherre@2ndquadrant.com>
1 sibling, 1 reply; 58+ messages in thread
From: Robert Haas @ 2016-03-21 09:34 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Tatsuo Ishii <ishii@postgresql.org>; Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>; Jeff Janes <jeff.janes@gmail.com>; pgsql-hackers
On Sun, Mar 20, 2016 at 11:34 PM, Alvaro Herrera
<alvherre@2ndquadrant.com> wrote:
> ObjectProperty[] contains a comment that the ACL is "same as relation",
> but is that still correct, given that now stats may be related to more
> than one relation? Do we even know what the rules for ACLs on
> cross-relation stats are? One very simple way to get around this is to
> dictate that all the rels must have the same owner.
That's not really all that simple - you'd have to forbid changing the
owner of a relation involved in multi-rel statistics, but that's
horrible. Presumably at the very least you'd then have to find some
way of allowing the owner of everything in the group to be changed at
the same time, but that's a whole new innovation. I think this is a
very messy line of attack.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: multivariate statistics v14
@ 2016-03-21 10:08 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Robert Haas <robertmhaas@gmail.com>
0 siblings, 0 replies; 58+ messages in thread
From: Tomas Vondra @ 2016-03-21 10:08 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: Tatsuo Ishii <ishii@postgresql.org>; Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>; Jeff Janes <jeff.janes@gmail.com>; pgsql-hackers
Hi,
On 03/21/2016 10:34 AM, Robert Haas wrote:
> On Sun, Mar 20, 2016 at 11:34 PM, Alvaro Herrera
> <alvherre@2ndquadrant.com> wrote:
>> ObjectProperty[] contains a comment that the ACL is "same as relation",
>> but is that still correct, given that now stats may be related to more
>> than one relation? Do we even know what the rules for ACLs on
>> cross-relation stats are? One very simple way to get around this is to
>> dictate that all the rels must have the same owner.
>
> That's not really all that simple - you'd have to forbid changing
> the owner of a relation involved in multi-rel statistics, but that's
> horrible. Presumably at the very least you'd then have to find some
> way of allowing the owner of everything in the group to be changed
> at the same time, but that's a whole new innovation. I think this is
> a very messy line of attack.
I agree. I don't think we should / need to impose such additional
restrictions (e.g. same owner for all tables).
I think for using the statistics (to compute estimates for a query), it
should be enough that the user can access all the tables it's built on.
Which happens somehow implicitly, and currently it's trivial as each
statistics is built on a single table.
I don't have a clear idea what should we do in the future with multiple
tables (e.g. when the statistics is built on 3 tables, the query is on 2
of them and the user does not have access to the remaining one).
But maybe we need to support ACLs because of ALTER STATISTICS?
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: multivariate statistics v14
@ 2016-03-21 10:30 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Alvaro Herrera <alvherre@2ndquadrant.com>
1 sibling, 0 replies; 58+ messages in thread
From: Tomas Vondra @ 2016-03-21 10:30 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: Tatsuo Ishii <ishii@postgresql.org>; horiguchi.kyotaro@lab.ntt.co.jp; jeff.janes@gmail.com; pgsql-hackers
On 03/21/2016 04:34 AM, Alvaro Herrera wrote:
> Another skim on 0002:
>
> reference.sgml is missing a call to &alterStatistic.
>
> ObjectProperty[] contains a comment that the ACL is "same as relation",
> but is that still correct, given that now stats may be related to more
> than one relation? Do we even know what the rules for ACLs on
> cross-relation stats are? One very simple way to get around this is to
> dictate that all the rels must have the same owner. Perhaps we're not
> considering the multi-relation case yet?
As I wrote in response to Robert's message, I don't think we need ACLs
for statistics - the user should be able to use them when they can
access all the underlying relations (in a query). For ALTER STATISTICS
the (owner || superuser) check should be enough, right?
>
> We have this FIXME comment in do_analyze_rel:
>
> + * FIXME This sample sizing is mostly OK when computing stats for
> + * individual columns, but when computing multi-variate stats
> + * for multivariate stats (histograms, mcv, ...) it's rather
> + * insufficient. For stats on multiple columns / complex stats
> + * we need larger sample sizes, because we need to build more
> + * detailed stats (more MCV items / histogram buckets) to get
> + * good accuracy. Maybe it'd be appropriate to use samples
> + * proportional to the table (say, 0.5% - 1%) instead of a
> + * fixed size might be more appropriate. Also, this should be
> + * bound to the requested statistics size - e.g. number of MCV
> + * items or histogram buckets should require several sample
> + * rows per item/bucket (so the sample should be k*size).
>
> Maybe this merits more discussion. Right now we have an upper bound on
> how much to scan for analyze; if we introduce the idea of scanning a
> percentage of the relation, the time to analyze very large relations
> could increase significantly. Do we have an idea of what to do for
> this? For instance, a rule that would make me comfortable would say to
> scan a sample 3x the current size when you have a mvstats on 3 columns;
> then the size of fraction to scan is still bounded. But does that
> actually work? From the wording of this comment, I assume you don't
> actually know.
Yeah. I think more discussion is needed, because I myself am not sure
the FIXME is actually correct. For now I think we're OK with using the
same logic as statistics on a single column (300 * target).
>
> In this block (CreateStatistics)
> + /* look for duplicities */
> + for (i = 0; i < numcols; i++)
> + for (j = 0; j < numcols; j++)
> + if ((i != j) && (attnums[i] == attnums[j]))
> + ereport(ERROR,
> + (errcode(ERRCODE_UNDEFINED_COLUMN),
> + errmsg("duplicate column name in statistics definition")));
>
> isn't it easier to have the inner loop go from i+1 to numcols?
It probably is.
>
> I wonder if this is sensible with multi-relation statistics:
> + /*
> + * Store a dependency too, so that statistics are dropped on DROP TABLE
> + */
> + parentobject.classId = RelationRelationId;
> + parentobject.objectId = ObjectIdGetDatum(RelationGetRelid(rel));
> + parentobject.objectSubId = 0;
> + childobject.classId = MvStatisticRelationId;
> + childobject.objectId = statoid;
> + childobject.objectSubId = 0;
>
> I suppose the idea is to drop the stats if any of the rels they are for
> is dropped.
What do you mean by sensible? I mean, we don't support multiple tables
at this point (except for choosing a syntax that should allow that), but
the code assumes a single relation on a few places (like this one).
>
> Right after that you create a dependency on the schema. Is that
> necessary? Since you have the dependency on the relation, the stats
> would be dropped by recursion.
Hmmmm, that's probably right. Also, now that I think about it, it
probably gets broken after ALTER STATISTICS ... SET SCHEMA, because the
code does not remove the old dependency (and does not create a new one).
>
> Why are you #include'ing builtins.h everywhere?
Stupidity.
>
> RelationGetMVStatList() needs a comment.
OK.
>
> Please get rid of common.h. It's totally unlike the way we structure
> our header files. We don't keep headers in src/backend; they're all in
> src/include. One reason is that the latter gets installed as a whole in
> include/server, which this file will not be. This file may be necessary
> to build some extensions in the future, for example.
OK, I'll rework that and move it to src/include/.
>
> In mvstats.h, please mark function prototypes as "extern".
>
> Many files need a pgindent pass.
OK.
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: multivariate statistics v14
@ 2016-03-22 05:53 Jeff Janes <jeff.janes@gmail.com>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
1 sibling, 1 reply; 58+ messages in thread
From: Jeff Janes @ 2016-03-22 05:53 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Tatsuo Ishii <ishii@postgresql.org>; Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>; Alvaro Herrera <alvherre@2ndquadrant.com>; pgsql-hackers
On Sun, Mar 20, 2016 at 4:34 PM, Tomas Vondra
<tomas.vondra@2ndquadrant.com> wrote:
>
>
> D'oh. Thanks for reporting. Attached is v16, hopefully fixing the few
> remaining whitespace issues.
Hi Tomas,
I'm trying out v16 against a common problem, where postgresql thinks
it is likely top stop early during a "order by (index express) limit
1" but it doesn't actually stop early due to cross-column
correlations. But the multivariate statistics don't seem to help. Am
I doing this wrong, or just expecting too much?
jjanes=# create table foo as select x, floor(x/(10000000/500))::int as
y from generate_series(1,10000000) f(x);
jjanes=# create index on foo (x,y);
jjanes=# create index on foo (y,x);
jjanes=# create statistics jjj on foo (x,y) with (dependencies,histogram);
jjanes=# vacuum analyze ;
jjanes=# explain (analyze, timing off) select x from foo where y
between 478 and 480 order by x limit 1;
QUERY PLAN
-------------------------------------------------------------------------------------------------------------------
Limit (cost=0.43..4.92 rows=1 width=4) (actual rows=1 loops=1)
-> Index Only Scan using foo_x_y_idx on foo (cost=0.43..210156.55
rows=46812 width=4) (actual rows=1 loops=1)
Index Cond: ((y >= 478) AND (y <= 480))
Heap Fetches: 0
Planning time: 0.311 ms
Execution time: 478.917 ms
Here is walks up the index on x, until it meets the first row meeting
the qualification on y. It thinks it will get to stop early and be
very fast, but it doesn't.
If I add an dummy addition to the ORDER BY, to force it not to talk
the index, I get a plan which uses the other index and is actually
much faster, but is planned to be several hundred times slower:
jjanes=# explain (analyze, timing off) select x from foo where y
between 478 and 480 order by x+0 limit 1;
QUERY PLAN
---------------------------------------------------------------------------------------------------------------------------
Limit (cost=1803.77..1803.77 rows=1 width=8) (actual rows=1 loops=1)
-> Sort (cost=1803.77..1920.80 rows=46812 width=8) (actual rows=1 loops=1)
Sort Key: ((x + 0))
Sort Method: top-N heapsort Memory: 25kB
-> Index Only Scan using foo_y_x_idx on foo
(cost=0.43..1569.70 rows=46812 width=8) (actual rows=60000 loops=1)
Index Cond: ((y >= 478) AND (y <= 480))
Heap Fetches: 0
Planning time: 0.175 ms
Execution time: 20.264 ms
(I use the "timing off" option, because without it the second plan
spends most of its time calling "gettimeofday")
Cheers,
Jeff
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: multivariate statistics v14
@ 2016-03-22 08:13 Tatsuo Ishii <ishii@postgresql.org>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 1 reply; 58+ messages in thread
From: Tatsuo Ishii @ 2016-03-22 08:13 UTC (permalink / raw)
To: tomas.vondra@2ndquadrant.com; +Cc: jeff.janes@gmail.com; alvherre@2ndquadrant.com; pgsql-hackers
>> Do you have any other missing parts in this work? I am asking
>> because I wonder if you want to push this into 9.6 or rather 9.7.
>
> I think the first few parts of the patch series, namely:
>
> * shared infrastructure (0002)
> * functional dependencies (0003)
> * MCV lists (0004)
> * histograms (0005)
>
> might make it into 9.6. I believe the code for building and storing
> the different kinds of stats is reasonably solid. What probably needs
> more thorough review are the changes in clauselist_selectivity(), but
> the code in these parts is reasonably simple as it only supports using
> a single multi-variate statistics per relation.
>
> The part (0006) that allows using multiple statistics (i.e. selects
> which of the available stats to use and in what order) is probably the
> most complex part of the whole patch, and I myself do have some
> questions about some aspects of it. I don't think this part might get
> into 9.6 at this point (although it'd be nice if we managed to do
> that).
Hum. So without 0006 or beyond, there's not much benefit for the
PostgreSQL users, and you are not too confident about 0006 or
beyond. Then I would think it is a little bit hard to justify in
putting 000[2-5] into 9.6. I really like this feature and would like
to see in PostgreSQL someday, but I'm not sure if we should put the
patches (0002-0005) into PostgreSQL now. Please let me know if there's
some reaons we should put the patches into PostgreSQL now.
Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: multivariate statistics v14
@ 2016-03-22 09:23 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Jeff Janes <jeff.janes@gmail.com>
0 siblings, 0 replies; 58+ messages in thread
From: Tomas Vondra @ 2016-03-22 09:23 UTC (permalink / raw)
To: Jeff Janes <jeff.janes@gmail.com>; +Cc: Tatsuo Ishii <ishii@postgresql.org>; Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>; Alvaro Herrera <alvherre@2ndquadrant.com>; pgsql-hackers
Hi,
On 03/22/2016 06:53 AM, Jeff Janes wrote:
> On Sun, Mar 20, 2016 at 4:34 PM, Tomas Vondra
> <tomas.vondra@2ndquadrant.com> wrote:
>>
>>
>> D'oh. Thanks for reporting. Attached is v16, hopefully fixing the few
>> remaining whitespace issues.
>
> Hi Tomas,
>
> I'm trying out v16 against a common problem, where postgresql thinks
> it is likely top stop early during a "order by (index express) limit
> 1" but it doesn't actually stop early due to cross-column
> correlations. But the multivariate statistics don't seem to help. Am
> I doing this wrong, or just expecting too much?
Yes, I think you're expecting a too much from the current patch.
I've been thinking about perhaps addressing cases like this in the
future, but it requires tracking position within the table somehow (e.g.
by means of including ctid in the table, or something like that), and
the current patch does not implement that.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: multivariate statistics v14
@ 2016-03-22 09:44 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Tatsuo Ishii <ishii@postgresql.org>
0 siblings, 1 reply; 58+ messages in thread
From: Tomas Vondra @ 2016-03-22 09:44 UTC (permalink / raw)
To: Tatsuo Ishii <ishii@postgresql.org>; +Cc: jeff.janes@gmail.com; alvherre@2ndquadrant.com; pgsql-hackers
Hello,
On 03/22/2016 09:13 AM, Tatsuo Ishii wrote:
>>> Do you have any other missing parts in this work? I am asking
>>> because I wonder if you want to push this into 9.6 or rather 9.7.
>>
>> I think the first few parts of the patch series, namely:
>>
>> * shared infrastructure (0002)
>> * functional dependencies (0003)
>> * MCV lists (0004)
>> * histograms (0005)
>>
>> might make it into 9.6. I believe the code for building and storing
>> the different kinds of stats is reasonably solid. What probably needs
>> more thorough review are the changes in clauselist_selectivity(), but
>> the code in these parts is reasonably simple as it only supports using
>> a single multi-variate statistics per relation.
>>
>> The part (0006) that allows using multiple statistics (i.e. selects
>> which of the available stats to use and in what order) is probably the
>> most complex part of the whole patch, and I myself do have some
>> questions about some aspects of it. I don't think this part might get
>> into 9.6 at this point (although it'd be nice if we managed to do
>> that).
>
> Hum. So without 0006 or beyond, there's not much benefit for the
> PostgreSQL users, and you are not too confident about 0006 or
> beyond. Then I would think it is a little bit hard to justify in
> putting 000[2-5] into 9.6. I really like this feature and would like
> to see in PostgreSQL someday, but I'm not sure if we should put the
> patches (0002-0005) into PostgreSQL now. Please let me know if there's
> some reaons we should put the patches into PostgreSQL now.
I don't think so. While being able to combine multiple statistics is
certainly useful, I'm convinced that the initial patched add enough
value on their own, even if the 0006 patch gets committed later.
A lot of queries will be just fine with the "single multivariate
statistics" limitation, either because it's using less than 8 columns,
or because only 8 columns are actually correlated. (FWIW the 8 column
limit is mostly arbitrary, it may get increased if needed.)
I haven't really mentioned the aspects of 0006 that I think need more
discussion, but it's mostly about the question whether combining the
statistics by using the overlapping clauses as "conditions" is the right
thing to do (or whether a more expensive approach is needed). None of
that however invalidates the preceding patches.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: multivariate statistics v14
@ 2016-03-22 10:41 Tatsuo Ishii <ishii@postgresql.org>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 1 reply; 58+ messages in thread
From: Tatsuo Ishii @ 2016-03-22 10:41 UTC (permalink / raw)
To: tomas.vondra@2ndquadrant.com; +Cc: jeff.janes@gmail.com; alvherre@2ndquadrant.com; pgsql-hackers
>> Hum. So without 0006 or beyond, there's not much benefit for the
>> PostgreSQL users, and you are not too confident about 0006 or
>> beyond. Then I would think it is a little bit hard to justify in
>> putting 000[2-5] into 9.6. I really like this feature and would like
>> to see in PostgreSQL someday, but I'm not sure if we should put the
>> patches (0002-0005) into PostgreSQL now. Please let me know if there's
>> some reaons we should put the patches into PostgreSQL now.
>
> I don't think so. While being able to combine multiple statistics is
> certainly useful, I'm convinced that the initial patched add enough
Can you please elaborate a little bit more how combining multiple
statistics is useful?
Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: multivariate statistics v14
@ 2016-03-22 12:37 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Tatsuo Ishii <ishii@postgresql.org>
0 siblings, 1 reply; 58+ messages in thread
From: Tomas Vondra @ 2016-03-22 12:37 UTC (permalink / raw)
To: Tatsuo Ishii <ishii@postgresql.org>; +Cc: jeff.janes@gmail.com; alvherre@2ndquadrant.com; pgsql-hackers
Hi,
On 03/22/2016 11:41 AM, Tatsuo Ishii wrote:
>>> Hum. So without 0006 or beyond, there's not much benefit for the
>>> PostgreSQL users, and you are not too confident about 0006 or
>>> beyond. Then I would think it is a little bit hard to justify in
>>> putting 000[2-5] into 9.6. I really like this feature and would
>>> like to see in PostgreSQL someday, but I'm not sure if we should
>>> put the patches (0002-0005) into PostgreSQL now. Please let me
>>> know if there's some reaons we should put the patches into
>>> PostgreSQL now.
>>
>> I don't think so. While being able to combine multiple statistics
>> is certainly useful, I'm convinced that the initial patched add
>> enough
>
> Can you please elaborate a little bit more how combining multiple
> statistics is useful?
Sure.
The goal of multivariate statistics is to approximate a probability
distribution on a group of columns. The larger the number of columns,
the less accurate the statistics will be (with respect to individual
columns), assuming fixed size of the sample in ANALYZE, and fixed
statistics size.
For example, if you add a column to multivariate histogram, you'll do
some "bucket splits" by this dimension, thus reducing the accuracy for
the other columns. You may of course allow larger statistics (e.g.
histograms with more buckets), but that also requires larger samples,
and so on.
Now, let's assume you have a query like this:
WHERE (a=1) AND (b=2) AND (c=3) AND (d=4)
and that "a" and "b" are correlated, and "c" and "d" are correlated, but
that otherwise the columns are independent. It'd be a bit silly to
require building statistics on (a,b,c,d), when two statistics on each of
the column pairs would be cheaper and also more accurate.
That's of course a trivial case - independent groups of correlated
columns. But I'd say this is actually a pretty common case, and I do
believe there's not much controversy that we should support it.
Another reason to allow multiple statistics is that columns in one group
may be a good fit for MCV list (which works well for discrete values),
while the other group may be a good candidate for histogram (which works
well for continuous values). This can't be solved by first building a
MCV and then a histogram on the group.
The question of course is what to do if the groups are not independent.
The patch does that by assuming the statistics overlap, and uses
conditions on the columns included in both statistics to combine them
using conditional probabilities. I do believe this works quite well, but
this is perhaps the part that needs further discussion. There are other
ways to combine the statistics, but I do expect them to be considerably
more expensive.
Is this a sufficient explanation?
Of course, there's a fair amount of additional complexity that I have
not mentioned here (e.g. selecting the right combination of stats).
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: multivariate statistics v14
@ 2016-03-22 12:46 Tatsuo Ishii <ishii@postgresql.org>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 1 reply; 58+ messages in thread
From: Tatsuo Ishii @ 2016-03-22 12:46 UTC (permalink / raw)
To: tomas.vondra@2ndquadrant.com; +Cc: ishii@postgresql.org; jeff.janes@gmail.com; alvherre@2ndquadrant.com; pgsql-hackers
> On 03/22/2016 11:41 AM, Tatsuo Ishii wrote:
>>>> Hum. So without 0006 or beyond, there's not much benefit for the
>>>> PostgreSQL users, and you are not too confident about 0006 or
>>>> beyond. Then I would think it is a little bit hard to justify in
>>>> putting 000[2-5] into 9.6. I really like this feature and would
>>>> like to see in PostgreSQL someday, but I'm not sure if we should
>>>> put the patches (0002-0005) into PostgreSQL now. Please let me
>>>> know if there's some reaons we should put the patches into
>>>> PostgreSQL now.
>>>
>>> I don't think so. While being able to combine multiple statistics
>>> is certainly useful, I'm convinced that the initial patched add
>>> enough
>>
>> Can you please elaborate a little bit more how combining multiple
>> statistics is useful?
>
> Sure.
>
> The goal of multivariate statistics is to approximate a probability
> distribution on a group of columns. The larger the number of columns,
> the less accurate the statistics will be (with respect to individual
> columns), assuming fixed size of the sample in ANALYZE, and fixed
> statistics size.
>
> For example, if you add a column to multivariate histogram, you'll do
> some "bucket splits" by this dimension, thus reducing the accuracy for
> the other columns. You may of course allow larger statistics
> (e.g. histograms with more buckets), but that also requires larger
> samples, and so on.
>
> Now, let's assume you have a query like this:
>
> WHERE (a=1) AND (b=2) AND (c=3) AND (d=4)
>
> and that "a" and "b" are correlated, and "c" and "d" are correlated,
> but that otherwise the columns are independent. It'd be a bit silly to
> require building statistics on (a,b,c,d), when two statistics on each
> of the column pairs would be cheaper and also more accurate.
>
> That's of course a trivial case - independent groups of correlated
> columns. But I'd say this is actually a pretty common case, and I do
> believe there's not much controversy that we should support it.
>
> Another reason to allow multiple statistics is that columns in one
> group may be a good fit for MCV list (which works well for discrete
> values), while the other group may be a good candidate for histogram
> (which works well for continuous values). This can't be solved by
> first building a MCV and then a histogram on the group.
>
> The question of course is what to do if the groups are not
> independent. The patch does that by assuming the statistics overlap,
> and uses conditions on the columns included in both statistics to
> combine them using conditional probabilities. I do believe this works
> quite well, but this is perhaps the part that needs further
> discussion. There are other ways to combine the statistics, but I do
> expect them to be considerably more expensive.
>
> Is this a sufficient explanation?
>
> Of course, there's a fair amount of additional complexity that I have
> not mentioned here (e.g. selecting the right combination of stats).
Sorry, maybe I did not explain clearyly. My question is, if put
patches only 0002 to 0005 into 9.6, does it still give any visible
benefit to users?
Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: multivariate statistics v14
@ 2016-03-22 13:49 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Tatsuo Ishii <ishii@postgresql.org>
0 siblings, 1 reply; 58+ messages in thread
From: Tomas Vondra @ 2016-03-22 13:49 UTC (permalink / raw)
To: Tatsuo Ishii <ishii@postgresql.org>; +Cc: jeff.janes@gmail.com; alvherre@2ndquadrant.com; pgsql-hackers
Hi,
On 03/22/2016 01:46 PM, Tatsuo Ishii wrote:
...
> Sorry, maybe I did not explain clearly. My question is, if put
> patches only 0002 to 0005 into 9.6, does it still give any visible
> benefit to users?
The users will be able to define statistics with the limitation that
only a single one (the one covering the most columns referenced by the
clauses) can be used when estimating a query. Which is not perfect, but
I think it's a valuable improvement.
It might also be possible to split 0006 into smaller pieces, for example
implementing the "non-overlapping statistics" case first and then
extending it to more complicated cases. That might increase the change
of getting at least some of that into 9.6 ...
But considering it's not clear whether the initial chunks are likely to
make it into 9.6 - I kinda expect a fair amount of comments from TL
about the preceding parts, who mentioned he might look at the patch this
week. So I'm not sure splitting 0006 into smaller pieces makes sense at
this point.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: multivariate statistics v14
@ 2016-03-23 01:53 Tatsuo Ishii <ishii@postgresql.org>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 1 reply; 58+ messages in thread
From: Tatsuo Ishii @ 2016-03-23 01:53 UTC (permalink / raw)
To: tomas.vondra@2ndquadrant.com; +Cc: jeff.janes@gmail.com; alvherre@2ndquadrant.com; pgsql-hackers
> The users will be able to define statistics with the limitation that
> only a single one (the one covering the most columns referenced by the
> clauses) can be used when estimating a query. Which is not perfect,
> but I think it's a valuable improvement.
>
> It might also be possible to split 0006 into smaller pieces, for
> example implementing the "non-overlapping statistics" case first and
> then extending it to more complicated cases. That might increase the
> change of getting at least some of that into 9.6 ...
>
> But considering it's not clear whether the initial chunks are likely
> to make it into 9.6 - I kinda expect a fair amount of comments from TL
> about the preceding parts, who mentioned he might look at the patch
> this week. So I'm not sure splitting 0006 into smaller pieces makes
> sense at this point.
Thanks for the explanation. I will look into patch 0001 to 0005 so
that they could get into 9.6.
In the mean time after applying patch 0001 to 0005 of v16, I get this
while compiling SGML docs.
openjade:ref/create_statistics.sgml:281:26:X: reference to non-existent ID "SQL-ALTERSTATISTICS"
openjade:ref/drop_statistics.sgml:86:26:X: reference to non-existent ID "SQL-ALTERSTATISTICS"
Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: multivariate statistics v14
@ 2016-03-23 01:57 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Tatsuo Ishii <ishii@postgresql.org>
0 siblings, 1 reply; 58+ messages in thread
From: Tomas Vondra @ 2016-03-23 01:57 UTC (permalink / raw)
To: Tatsuo Ishii <ishii@postgresql.org>; +Cc: jeff.janes@gmail.com; alvherre@2ndquadrant.com; pgsql-hackers
On 03/23/2016 02:53 AM, Tatsuo Ishii wrote:
>> The users will be able to define statistics with the limitation that
>> only a single one (the one covering the most columns referenced by the
>> clauses) can be used when estimating a query. Which is not perfect,
>> but I think it's a valuable improvement.
>>
>> It might also be possible to split 0006 into smaller pieces, for
>> example implementing the "non-overlapping statistics" case first and
>> then extending it to more complicated cases. That might increase the
>> change of getting at least some of that into 9.6 ...
>>
>> But considering it's not clear whether the initial chunks are likely
>> to make it into 9.6 - I kinda expect a fair amount of comments from TL
>> about the preceding parts, who mentioned he might look at the patch
>> this week. So I'm not sure splitting 0006 into smaller pieces makes
>> sense at this point.
>
> Thanks for the explanation. I will look into patch 0001 to 0005 so
> that they could get into 9.6.
>
> In the mean time after applying patch 0001 to 0005 of v16, I get this
> while compiling SGML docs.
>
> openjade:ref/create_statistics.sgml:281:26:X: reference to non-existent ID "SQL-ALTERSTATISTICS"
> openjade:ref/drop_statistics.sgml:86:26:X: reference to non-existent ID "SQL-ALTERSTATISTICS"
I believe this is because reference.sgml is missing a call to
&alterStatistic (per report by Alvaro Herrera).
thanks
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: multivariate statistics v14
@ 2016-03-23 02:01 Tatsuo Ishii <ishii@postgresql.org>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 1 reply; 58+ messages in thread
From: Tatsuo Ishii @ 2016-03-23 02:01 UTC (permalink / raw)
To: tomas.vondra@2ndquadrant.com; +Cc: jeff.janes@gmail.com; alvherre@2ndquadrant.com; pgsql-hackers
>> Thanks for the explanation. I will look into patch 0001 to 0005 so
>> that they could get into 9.6.
>>
>> In the mean time after applying patch 0001 to 0005 of v16, I get this
>> while compiling SGML docs.
>>
>> openjade:ref/create_statistics.sgml:281:26:X: reference to
>> non-existent ID "SQL-ALTERSTATISTICS"
>> openjade:ref/drop_statistics.sgml:86:26:X: reference to non-existent
>> ID "SQL-ALTERSTATISTICS"
>
> I believe this is because reference.sgml is missing a call to
> &alterStatistic (per report by Alvaro Herrera).
Ok, I will patch reference.sgml.
Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: multivariate statistics v14
@ 2016-03-23 02:57 Tatsuo Ishii <ishii@postgresql.org>
parent: Tatsuo Ishii <ishii@postgresql.org>
0 siblings, 1 reply; 58+ messages in thread
From: Tatsuo Ishii @ 2016-03-23 02:57 UTC (permalink / raw)
To: tomas.vondra@2ndquadrant.com; +Cc: jeff.janes@gmail.com; alvherre@2ndquadrant.com; pgsql-hackers
>> I believe this is because reference.sgml is missing a call to
>> &alterStatistic (per report by Alvaro Herrera).
>
> Ok, I will patch reference.sgml.
Here are some comments on docs.
- There's no docs for pg_mv_statistic (should be added to "49. System
Catalogs")
- The word "multivariate statistics" or something like that should
appear in the index.
- There are some explanation how to deal with multivariate statistics
in "14.1 Using Explain" and "14.2 Statistics used by the Planner"
section.
I am now looking into the create statistics doc to see if the example
appearing in it is working. I will get back if I find any.
Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: multivariate statistics v14
@ 2016-03-23 03:01 Tatsuo Ishii <ishii@postgresql.org>
parent: Tatsuo Ishii <ishii@postgresql.org>
0 siblings, 1 reply; 58+ messages in thread
From: Tatsuo Ishii @ 2016-03-23 03:01 UTC (permalink / raw)
To: ishii@postgresql.org; +Cc: tomas.vondra@2ndquadrant.com; jeff.janes@gmail.com; alvherre@2ndquadrant.com; pgsql-hackers
>>> I believe this is because reference.sgml is missing a call to
>>> &alterStatistic (per report by Alvaro Herrera).
>>
>> Ok, I will patch reference.sgml.
>
> Here are some comments on docs.
>
> - There's no docs for pg_mv_statistic (should be added to "49. System
> Catalogs")
>
> - The word "multivariate statistics" or something like that should
> appear in the index.
>
> - There are some explanation how to deal with multivariate statistics
Oops. Should read "There should be some explanations".
> in "14.1 Using Explain" and "14.2 Statistics used by the Planner"
> section.
>
> I am now looking into the create statistics doc to see if the example
> appearing in it is working. I will get back if I find any.
>
> Best regards,
> --
> Tatsuo Ishii
> SRA OSS, Inc. Japan
> English: http://www.sraoss.co.jp/index_en.php
> Japanese:http://www.sraoss.co.jp
>
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: multivariate statistics v14
@ 2016-03-23 05:20 Tatsuo Ishii <ishii@postgresql.org>
parent: Tatsuo Ishii <ishii@postgresql.org>
0 siblings, 1 reply; 58+ messages in thread
From: Tatsuo Ishii @ 2016-03-23 05:20 UTC (permalink / raw)
To: tomas.vondra@2ndquadrant.com; +Cc: jeff.janes@gmail.com; alvherre@2ndquadrant.com; pgsql-hackers
>> I am now looking into the create statistics doc to see if the example
>> appearing in it is working. I will get back if I find any.
I have the ref doc: CREATE STATISTICS
There are nice examples how the multivariate statistics gives better
row number estimation. So I gave them a try.
"Create table t1 with two functionally dependent columns,
i.e. knowledge of a value in the first column is sufficient for
determining the value in the other column" The example creates table
"t1", then populates it using generate_series. After CREATE
STATISTICS, ANALYZE and EXPLAIN. I expected the EXPLAIN demonstrates
how result rows estimation is enhanced by using the multivariate
statistics.
Here is the EXPLAIN output using the multivariate statistics:
EXPLAIN ANALYZE SELECT * FROM t1 WHERE (a = 1) AND (b = 1);
QUERY PLAN
---------------------------------------------------------------------------------------------------
Seq Scan on t1 (cost=0.00..19425.00 rows=98 width=8) (actual time=76.876..76.876 rows=0 loops=1)
Filter: ((a = 1) AND (b = 1))
Rows Removed by Filter: 1000000
Planning time: 0.146 ms
Execution time: 76.896 ms
(5 rows)
Here is the EXPLAIN output without the multivariate statistics:
EXPLAIN ANALYZE SELECT * FROM t1 WHERE (a = 1) AND (b = 1);
QUERY PLAN
--------------------------------------------------------------------------------------------------
Seq Scan on t1 (cost=0.00..19425.00 rows=1 width=8) (actual time=78.867..78.867 rows=0 loops=1)
Filter: ((a = 1) AND (b = 1))
Rows Removed by Filter: 1000000
Planning time: 0.102 ms
Execution time: 78.885 ms
(5 rows)
It seems the row numbers estimation (98) using the multivariate
statistics is actually *worse* than the one (1) not using the
statistics because the actual row number is 0.
Next example (using table "t2") is much better than the case using t1.
Here is the EXPLAIN output using the multivariate statistics:
EXPLAIN ANALYZE SELECT * FROM t2 WHERE (a = 1) AND (b = 1);
QUERY PLAN
--------------------------------------------------------------------------------------------------------
Seq Scan on t2 (cost=0.00..19425.00 rows=9633 width=8) (actual time=0.012..75.350 rows=10000 loops=1)
Filter: ((a = 1) AND (b = 1))
Rows Removed by Filter: 990000
Planning time: 0.107 ms
Execution time: 75.680 ms
(5 rows)
Here is the EXPLAIN output without the multivariate statistics:
EXPLAIN ANALYZE SELECT * FROM t2 WHERE (a = 1) AND (b = 1);
QUERY PLAN
------------------------------------------------------------------------------------------------------
Seq Scan on t2 (cost=0.00..19425.00 rows=91 width=8) (actual time=0.008..76.614 rows=10000 loops=1)
Filter: ((a = 1) AND (b = 1))
Rows Removed by Filter: 990000
Planning time: 0.067 ms
Execution time: 76.935 ms
(5 rows)
This time it seems the row numbers estimation (9633) using the
multivariate statistics is much better than the one (91) not using the
statistics because the actual row number is 10000.
The last example (using table "t3") seems no effect by multivariate statistics.
Here is the EXPLAIN output using the multivariate statistics:
EXPLAIN ANALYZE SELECT * FROM t3 WHERE (a < 500) AND (b > 500);
QUERY PLAN
-----------------------------------------------------------------------------------------------------------
Seq Scan on t3 (cost=0.00..20407.65 rows=111123 width=16) (actual time=0.154..132.509 rows=6002 loops=1)
Filter: ((a < '500'::double precision) AND (b > '500'::double precision))
Rows Removed by Filter: 993998
Planning time: 0.080 ms
Execution time: 132.735 ms
(5 rows)
EXPLAIN ANALYZE SELECT * FROM t3 WHERE (a < 400) AND (b > 600);
QUERY PLAN
----------------------------------------------------------------------------------------------------------
Seq Scan on t3 (cost=0.00..20407.65 rows=111123 width=16) (actual time=110.518..110.518 rows=0 loops=1)
Filter: ((a < '400'::double precision) AND (b > '600'::double precision))
Rows Removed by Filter: 1000000
Planning time: 0.052 ms
Execution time: 110.531 ms
(5 rows)
Here is the EXPLAIN output without the multivariate statistics:
EXPLAIN ANALYZE SELECT * FROM t3 WHERE (a < 500) AND (b > 500);
QUERY PLAN
-----------------------------------------------------------------------------------------------------------
Seq Scan on t3 (cost=0.00..20407.65 rows=111123 width=16) (actual time=0.149..129.718 rows=5999 loops=1)
Filter: ((a < '500'::double precision) AND (b > '500'::double precision))
Rows Removed by Filter: 994001
Planning time: 0.058 ms
Execution time: 129.893 ms
(5 rows)
EXPLAIN ANALYZE SELECT * FROM t3 WHERE (a < 400) AND (b > 600);
QUERY PLAN
----------------------------------------------------------------------------------------------------------
Seq Scan on t3 (cost=0.00..20407.65 rows=111123 width=16) (actual time=108.015..108.015 rows=0 loops=1)
Filter: ((a < '400'::double precision) AND (b > '600'::double precision))
Rows Removed by Filter: 1000000
Planning time: 0.037 ms
Execution time: 108.027 ms
(5 rows)
This time it seems the row numbers estimation (111123) using the
multivariate statistics is same as same as the one (111123) not
using the statistics because the actual row number is 5999 or 0.
In summary, the only case which shows the effect of the multivariate
statistics is the "t2" case. So I don't see why other examples are
shown in the manual. Am I missing something?
Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: multivariate statistics v14
@ 2016-03-23 13:21 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Tatsuo Ishii <ishii@postgresql.org>
0 siblings, 1 reply; 58+ messages in thread
From: Tomas Vondra @ 2016-03-23 13:21 UTC (permalink / raw)
To: Tatsuo Ishii <ishii@postgresql.org>; +Cc: jeff.janes@gmail.com; alvherre@2ndquadrant.com; pgsql-hackers
On 03/23/2016 06:20 AM, Tatsuo Ishii wrote:
>>> I am now looking into the create statistics doc to see if the example
>>> appearing in it is working. I will get back if I find any.
>
> I have the ref doc: CREATE STATISTICS
>
> There are nice examples how the multivariate statistics gives better
> row number estimation. So I gave them a try.
>
> "Create table t1 with two functionally dependent columns,
> i.e. knowledge of a value in the first column is sufficient for
> determining the value in the other column" The example creates table
> "t1", then populates it using generate_series. After CREATE
> STATISTICS, ANALYZE and EXPLAIN. I expected the EXPLAIN demonstrates
> how result rows estimation is enhanced by using the multivariate
> statistics.
>
> Here is the EXPLAIN output using the multivariate statistics:
>
> EXPLAIN ANALYZE SELECT * FROM t1 WHERE (a = 1) AND (b = 1);
> QUERY PLAN
> ---------------------------------------------------------------------------------------------------
> Seq Scan on t1 (cost=0.00..19425.00 rows=98 width=8) (actual time=76.876..76.876 rows=0 loops=1)
> Filter: ((a = 1) AND (b = 1))
> Rows Removed by Filter: 1000000
> Planning time: 0.146 ms
> Execution time: 76.896 ms
> (5 rows)
>
> Here is the EXPLAIN output without the multivariate statistics:
>
> EXPLAIN ANALYZE SELECT * FROM t1 WHERE (a = 1) AND (b = 1);
> QUERY PLAN
> --------------------------------------------------------------------------------------------------
> Seq Scan on t1 (cost=0.00..19425.00 rows=1 width=8) (actual time=78.867..78.867 rows=0 loops=1)
> Filter: ((a = 1) AND (b = 1))
> Rows Removed by Filter: 1000000
> Planning time: 0.102 ms
> Execution time: 78.885 ms
> (5 rows)
>
> It seems the row numbers estimation (98) using the multivariate
> statistics is actually *worse* than the one (1) not using the
> statistics because the actual row number is 0.
Yes, there's a mistake in the first query, because the conditions
actually are not compatible. I.e. (i/100)=1 and (i/500)=1 have no
overlapping rows, clearly. It should be
EXPLAIN ANALYZE SELECT * FROM t1 WHERE (a = 1) AND (b = 0);
instead. Will fix.
>
> Next example (using table "t2") is much better than the case using t1.
>
> Here is the EXPLAIN output using the multivariate statistics:
>
> EXPLAIN ANALYZE SELECT * FROM t2 WHERE (a = 1) AND (b = 1);
> QUERY PLAN
> --------------------------------------------------------------------------------------------------------
> Seq Scan on t2 (cost=0.00..19425.00 rows=9633 width=8) (actual time=0.012..75.350 rows=10000 loops=1)
> Filter: ((a = 1) AND (b = 1))
> Rows Removed by Filter: 990000
> Planning time: 0.107 ms
> Execution time: 75.680 ms
> (5 rows)
>
> Here is the EXPLAIN output without the multivariate statistics:
>
> EXPLAIN ANALYZE SELECT * FROM t2 WHERE (a = 1) AND (b = 1);
> QUERY PLAN
> ------------------------------------------------------------------------------------------------------
> Seq Scan on t2 (cost=0.00..19425.00 rows=91 width=8) (actual time=0.008..76.614 rows=10000 loops=1)
> Filter: ((a = 1) AND (b = 1))
> Rows Removed by Filter: 990000
> Planning time: 0.067 ms
> Execution time: 76.935 ms
> (5 rows)
>
> This time it seems the row numbers estimation (9633) using the
> multivariate statistics is much better than the one (91) not using the
> statistics because the actual row number is 10000.
>
> The last example (using table "t3") seems no effect by multivariate statistics.
Yes. There's a typo in the example - it analyzes the wrong table (t2
instead of t3). Once I fix that, the estimates are much better.
> In summary, the only case which shows the effect of the multivariate
> statistics is the "t2" case. So I don't see why other examples are
> shown in the manual. Am I missing something?
No, thanks for spotting those mistakes. I'll fix them and submit a new
version of the patch - either later today or perhaps tomorrow.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: multivariate statistics v14
@ 2016-03-23 18:23 Petr Jelinek <petr@2ndquadrant.com>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 1 reply; 58+ messages in thread
From: Petr Jelinek @ 2016-03-23 18:23 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Tatsuo Ishii <ishii@postgresql.org>; +Cc: jeff.janes@gmail.com; alvherre@2ndquadrant.com; pgsql-hackers
Hi,
I'll add couple of code comments from my first cursory read through
(this is huge):
0002:
there is some whitespace noise between the varlistentries in
alter_statistics.sgml
+ parentobject.classId = RelationRelationId;
+ parentobject.objectId = ObjectIdGetDatum(RelationGetRelid(rel));
+ parentobject.objectSubId = 0;
+ childobject.classId = MvStatisticRelationId;
+ childobject.objectId = statoid;
+ childobject.objectSubId = 0;
I wonder if this (several places similar code) would be simpler done
using ObjectAddressSet()
The common.h in backend/utils/mvstat is slightly weird header file
placement and naming.
0004:
+/* used for merging bitmaps - AND (min), OR (max) */
+#define MAX(x, y) (((x) > (y)) ? (x) : (y))
+#define MIN(x, y) (((x) < (y)) ? (x) : (y))
Huh? We have Max and Min macros defined in c.h
+ values[Anum_pg_mv_statistic_stamcv - 1] = PointerGetDatum(data);
Why the double space (that's actually in several places in several of
the patches).
I don't really understand why 0008 and 0009 are separate patches and
aren't part of one of the other patches. But otherwise good job on
splitting the functionality into patchset.
--
Petr Jelinek http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: multivariate statistics v14
@ 2016-03-24 17:12 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Petr Jelinek <petr@2ndquadrant.com>
0 siblings, 1 reply; 58+ messages in thread
From: Tomas Vondra @ 2016-03-24 17:12 UTC (permalink / raw)
To: Petr Jelinek <petr@2ndquadrant.com>; Tatsuo Ishii <ishii@postgresql.org>; +Cc: jeff.janes@gmail.com; alvherre@2ndquadrant.com; pgsql-hackers
Hi,
attached is v17 of the patch series, with these changes:
* rebase to current master (the AM patch caused some conflicts)
* add alterStatistics to reference.sgml (Alvaro)
* move the sample size discussion to README.stats (Alvaro)
* tweak the inner for loop in CREATE STATISTICS (Alvaro)
* use ObjectAddressSet() to create dependencies in statscmds.c (Petr)
* fix whitespace in alterStatistics.sgml (Petr)
* replace custom MIN/MAX with Min/Max in c.h (Petr)
* fix examples in createStatistics.sgml (Tatsuo)
A few more comments inline:
On 03/23/2016 07:23 PM, Petr Jelinek wrote:
>
> The common.h in backend/utils/mvstat is slightly weird header file
> placement and naming.
>
True. I plan to move this header to
src/include/catalog/pg_mv_statistic_fn.h
which is what the other catalogs do (as pointed by Alvaro). Or do you
think another location/name would be more appropriate?
>
> + values[Anum_pg_mv_statistic_stamcv - 1] = PointerGetDatum(data);
>
> Why the double space (that's actually in several places in several of
> the patches).
To align the whole block like this:
nulls[Anum_pg_mv_statistic_stadeps -1] = true;
nulls[Anum_pg_mv_statistic_stamcv -1] = true;
nulls[Anum_pg_mv_statistic_stahist -1] = true;
nulls[Anum_pg_mv_statistic_standist -1] = true;
But I won't fight for this too hard, if it breaks rules somehow.
>
> I don't really understand why 0008 and 0009 are separate patches and
> aren't part of one of the other patches. But otherwise good job on
> splitting the functionality into patchset.
That is mostly because both 0007 and 0008 tweak the GROUP BY estimates,
but 0008 is not really part of this patch (it's discussed separately in
another thread). I admit it may be a bit confusing.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[application/x-compressed-tar] multivariate-stats-v17.tgz (144.9K, ../../5b18d0a1-4dc7-78c7-784d-e1497e001675@2ndquadrant.com/2-multivariate-stats-v17.tgz)
download
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: multivariate statistics v14
@ 2016-03-24 17:45 Alvaro Herrera <alvherre@2ndquadrant.com>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 1 reply; 58+ messages in thread
From: Alvaro Herrera @ 2016-03-24 17:45 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Petr Jelinek <petr@2ndquadrant.com>; Tatsuo Ishii <ishii@postgresql.org>; jeff.janes@gmail.com; pgsql-hackers
Tomas Vondra wrote:
> >+ values[Anum_pg_mv_statistic_stamcv - 1] = PointerGetDatum(data);
> >
> >Why the double space (that's actually in several places in several of
> >the patches).
>
> To align the whole block like this:
>
> nulls[Anum_pg_mv_statistic_stadeps -1] = true;
> nulls[Anum_pg_mv_statistic_stamcv -1] = true;
> nulls[Anum_pg_mv_statistic_stahist -1] = true;
> nulls[Anum_pg_mv_statistic_standist -1] = true;
>
> But I won't fight for this too hard, if it breaks rules somehow.
Yeah, it will be undone by pgindent. I suggest you pgindent all the
patches in the series. With some clever patch vs. patch -R application,
you can do it without having to resolve any conflicts when pgindent
modifies code that a patch further up in the series modifies again.
--
Álvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: multivariate statistics v14
@ 2016-03-25 15:32 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Alvaro Herrera <alvherre@2ndquadrant.com>
0 siblings, 1 reply; 58+ messages in thread
From: Tomas Vondra @ 2016-03-25 15:32 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: Petr Jelinek <petr@2ndquadrant.com>; Tatsuo Ishii <ishii@postgresql.org>; jeff.janes@gmail.com; pgsql-hackers
On 03/24/2016 06:45 PM, Alvaro Herrera wrote:
> Tomas Vondra wrote:
>
>>> + values[Anum_pg_mv_statistic_stamcv - 1] = PointerGetDatum(data);
>>>
>>> Why the double space (that's actually in several places in several of
>>> the patches).
>>
>> To align the whole block like this:
>>
>> nulls[Anum_pg_mv_statistic_stadeps -1] = true;
>> nulls[Anum_pg_mv_statistic_stamcv -1] = true;
>> nulls[Anum_pg_mv_statistic_stahist -1] = true;
>> nulls[Anum_pg_mv_statistic_standist -1] = true;
>>
>> But I won't fight for this too hard, if it breaks rules somehow.
>
> Yeah, it will be undone by pgindent. I suggest you pgindent all the
> patches in the series. With some clever patch vs. patch -R application,
> you can do it without having to resolve any conflicts when pgindent
> modifies code that a patch further up in the series modifies again.
>
I could do that, but isn't that a bit pointless? I thought pgindent is
run regularly on the whole codebase, not for individual patches. Sure,
it'll tweak the formatting on a few places in the patch (including the
code discussed above, as you pointed out), but there are many other such
places coming from other committed patches.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: multivariate statistics v14
@ 2016-03-25 21:26 Tom Lane <tgl@sss.pgh.pa.us>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 1 reply; 58+ messages in thread
From: Tom Lane @ 2016-03-25 21:26 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Alvaro Herrera <alvherre@2ndquadrant.com>; Petr Jelinek <petr@2ndquadrant.com>; Tatsuo Ishii <ishii@postgresql.org>; jeff.janes@gmail.com; pgsql-hackers
Tomas Vondra <tomas.vondra@2ndquadrant.com> writes:
> I could do that, but isn't that a bit pointless? I thought pgindent is
> run regularly on the whole codebase, not for individual patches. Sure,
> it'll tweak the formatting on a few places in the patch (including the
> code discussed above, as you pointed out), but there are many other such
> places coming from other committed patches.
One point of running pgindent for yourself is to make sure you haven't set
up any code in a way that will look horrible after pgindent gets done with
it.
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: multivariate statistics v14
@ 2016-03-26 02:02 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 2 replies; 58+ messages in thread
From: Tomas Vondra @ 2016-03-26 02:02 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Alvaro Herrera <alvherre@2ndquadrant.com>; Petr Jelinek <petr@2ndquadrant.com>; Tatsuo Ishii <ishii@postgresql.org>; jeff.janes@gmail.com; pgsql-hackers
On 03/25/2016 10:26 PM, Tom Lane wrote:
> Tomas Vondra <tomas.vondra@2ndquadrant.com> writes:
>> I could do that, but isn't that a bit pointless? I thought pgindent is
>> run regularly on the whole codebase, not for individual patches. Sure,
>> it'll tweak the formatting on a few places in the patch (including the
>> code discussed above, as you pointed out), but there are many other such
>> places coming from other committed patches.
>
> One point of running pgindent for yourself is to make sure you
> haven't set up any code in a way that will look horrible after
> pgindent gets done with it.
Fair point. Attached is v18 of the patch, after pgindent cleanup.
FWIW, most of the tweaks were minor things like (! x) instead of (!x)
and so on. I also had to fix a few comments with internal formatting,
because pgindent decided to reformat the text using tabs etc.
There are a few places where I reverted the pgindent formatting, because
it seemed a bit too weird - the first one are the lists of function
prototypes in common.h/mvstat.h, the second one are function calls to
_greedy/_exhaustive methods.
None of those places would however qualify as 'horrible' in my opinion,
and the _greedy/_exhaustive functions are in the 0006 part, so fixing
that is not of immediate importance I think.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[application/x-compressed-tar] multivariate-stats-v18.tgz (144.6K, ../../ef0d75d9-f8fd-5a81-850e-e5e42579fba8@2ndquadrant.com/2-multivariate-stats-v18.tgz)
download
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: multivariate statistics v14
@ 2016-03-26 09:18 Tatsuo Ishii <ishii@postgresql.org>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
1 sibling, 1 reply; 58+ messages in thread
From: Tatsuo Ishii @ 2016-03-26 09:18 UTC (permalink / raw)
To: tomas.vondra@2ndquadrant.com; +Cc: tgl@sss.pgh.pa.us; alvherre@2ndquadrant.com; petr@2ndquadrant.com; jeff.janes@gmail.com; pgsql-hackers
> Fair point. Attached is v18 of the patch, after pgindent cleanup.
Here are some feedbacks to v18 patch.
1) regarding examples in create_statistics manual
Here are numbers I got. "with statistics" referrers to the case where
multivariate statistics are used. "without statistics" referrers to the
case where multivariate statistics are not used. The numbers denote
estimated_rows/actual_rows. Thus closer to 1.0 is better. Some numbers
are shown as a fraction to avoid 0 division. In my understanding case
1, 3, 4 showed that multivariate statistics superior.
with statistics without statistics
case1 0.98 0.01
case2 98/0 1/0
case3 1.05 0.01
case4 1/0 103/0
case5 18.50 18.33
case6 111123/0 1111123/0
2) following comments by me are not addressed in the v18 patch.
> - There's no docs for pg_mv_statistic (should be added to "49. System
> Catalogs")
>
> - The word "multivariate statistics" or something like that should
> appear in the index.
>
> - There are some explanation how to deal with multivariate statistics
> in "14.1 Using Explain" and "14.2 Statistics used by the Planner"
> section.
Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: multivariate statistics v14
@ 2016-03-26 19:09 Alvaro Herrera <alvherre@2ndquadrant.com>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
1 sibling, 1 reply; 58+ messages in thread
From: Alvaro Herrera @ 2016-03-26 19:09 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Petr Jelinek <petr@2ndquadrant.com>; Tatsuo Ishii <ishii@postgresql.org>; jeff.janes@gmail.com; pgsql-hackers
Tomas Vondra wrote:
> There are a few places where I reverted the pgindent formatting, because it
> seemed a bit too weird - the first one are the lists of function prototypes
> in common.h/mvstat.h, the second one are function calls to
> _greedy/_exhaustive methods.
Function prototypes being weird is something that we've learned to
accept. There's no point in undoing pgindent decisions there, because
the next run will re-apply them anyway. Best not to fight it.
What you should definitely look into fixing is the formatting of
comments, if the result is too horrible. You can prevent it from
messing those by adding dashes /*----- at the beginning of the comment.
--
Álvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: multivariate statistics v14
@ 2016-03-28 08:42 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Tatsuo Ishii <ishii@postgresql.org>
0 siblings, 2 replies; 58+ messages in thread
From: Tomas Vondra @ 2016-03-28 08:42 UTC (permalink / raw)
To: Tatsuo Ishii <ishii@postgresql.org>; +Cc: tgl@sss.pgh.pa.us; alvherre@2ndquadrant.com; petr@2ndquadrant.com; jeff.janes@gmail.com; pgsql-hackers
Hi,
On 03/26/2016 10:18 AM, Tatsuo Ishii wrote:
>> Fair point. Attached is v18 of the patch, after pgindent cleanup.
>
> Here are some feedbacks to v18 patch.
>
> 1) regarding examples in create_statistics manual
>
> Here are numbers I got. "with statistics" referrers to the case where
> multivariate statistics are used. "without statistics" referrers to the
> case where multivariate statistics are not used. The numbers denote
> estimated_rows/actual_rows. Thus closer to 1.0 is better. Some numbers
> are shown as a fraction to avoid 0 division. In my understanding case
> 1, 3, 4 showed that multivariate statistics superior.
>
> with statistics without statistics
> case1 0.98 0.01
> case2 98/0 1/0
The case2 shows that functional dependencies assume that the conditions
used in queries won't be incompatible - that's something this type of
statistics can't fix.
> case3 1.05 0.01
> case4 1/0 103/0
> case5 18.50 18.33
> case6 111123/0 1111123/0
The last two lines (case5 + case6) seem a bit suspicious. I believe
those are for the histogram data, and I do get these numbers:
case5 0.93 (5517 / 5949) 42.0 (249943 / 5949)
case6 100/0 100/0
Perhaps you've been using the version before the bugfix, with ANALYZE on
the wrong table?
>
> 2) following comments by me are not addressed in the v18 patch.
>
>> - There's no docs for pg_mv_statistic (should be added to "49. System
>> Catalogs")
>>
>> - The word "multivariate statistics" or something like that should
>> appear in the index.
>>
>> - There are some explanation how to deal with multivariate statistics
>> in "14.1 Using Explain" and "14.2 Statistics used by the Planner"
>> section.
Yes, those are valid omissions. I plan to address them, and I'd also
considering adding a section to 65.1 (How the Planner Uses Statistics),
explaining more thoroughly how the planner uses multivariate stats.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: multivariate statistics v14
@ 2016-03-28 08:49 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Alvaro Herrera <alvherre@2ndquadrant.com>
0 siblings, 1 reply; 58+ messages in thread
From: Tomas Vondra @ 2016-03-28 08:49 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Petr Jelinek <petr@2ndquadrant.com>; Tatsuo Ishii <ishii@postgresql.org>; jeff.janes@gmail.com; pgsql-hackers
On 03/26/2016 08:09 PM, Alvaro Herrera wrote:
> Tomas Vondra wrote:
>
>> There are a few places where I reverted the pgindent formatting, because it
>> seemed a bit too weird - the first one are the lists of function prototypes
>> in common.h/mvstat.h, the second one are function calls to
>> _greedy/_exhaustive methods.
>
> Function prototypes being weird is something that we've learned to
> accept. There's no point in undoing pgindent decisions there, because
> the next run will re-apply them anyway. Best not to fight it.
>
> What you should definitely look into fixing is the formatting of
> comments, if the result is too horrible. You can prevent it from
> messing those by adding dashes /*----- at the beginning of the comment.
>
Yep, formatting of some of the comments got slightly broken, but it
wasn't difficult to fix that without the /*------- trick.
I'm not sure about the prototypes though. It was a bit weird because
prototypes in the same header file were formatted very differently.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: multivariate statistics v14
@ 2016-03-28 15:54 Alvaro Herrera <alvherre@2ndquadrant.com>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 0 replies; 58+ messages in thread
From: Alvaro Herrera @ 2016-03-28 15:54 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Petr Jelinek <petr@2ndquadrant.com>; Tatsuo Ishii <ishii@postgresql.org>; jeff.janes@gmail.com; pgsql-hackers
Tomas Vondra wrote:
> I'm not sure about the prototypes though. It was a bit weird because
> prototypes in the same header file were formatted very differently.
Yeah, it is very odd. What happens is that the BSD indent binary does
one thing (return type is in one line and function name in following
line; subsequent argument lines are aligned to opening parens), then the
pgindent perl script changes it (moves function name to same line as
return type, but does not reindent subsequent lines of arguments).
You can imitate the effect by adding an extra newline just before the
function name, reflowing the arguments to align to the (, then deleting
the extra newline. Rather annoying.
--
Álvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: multivariate statistics v14
@ 2016-03-29 15:18 David Steele <david@pgmasters.net>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
1 sibling, 1 reply; 58+ messages in thread
From: David Steele @ 2016-03-29 15:18 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Tatsuo Ishii <ishii@postgresql.org>; +Cc: tgl@sss.pgh.pa.us; alvherre@2ndquadrant.com; petr@2ndquadrant.com; jeff.janes@gmail.com; pgsql-hackers
Hi Tomas,
On 3/28/16 4:42 AM, Tomas Vondra wrote:
> Yes, those are valid omissions. I plan to address them, and I'd also
> considering adding a section to 65.1 (How the Planner Uses Statistics),
> explaining more thoroughly how the planner uses multivariate stats.
It looks you need post a new patch so I have marked this "waiting on
author".
Thanks,
--
-David
david@pgmasters.net
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: multivariate statistics v14
@ 2016-03-30 05:15 Tatsuo Ishii <ishii@postgresql.org>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
1 sibling, 0 replies; 58+ messages in thread
From: Tatsuo Ishii @ 2016-03-30 05:15 UTC (permalink / raw)
To: tomas.vondra@2ndquadrant.com; +Cc: tgl@sss.pgh.pa.us; alvherre@2ndquadrant.com; petr@2ndquadrant.com; jeff.janes@gmail.com; pgsql-hackers
>> with statistics without statistics
>> case1 0.98 0.01
>> case2 98/0 1/0
>
> The case2 shows that functional dependencies assume that the
> conditions used in queries won't be incompatible - that's something
> this type of statistics can't fix.
It would be nice if that's mentioned in the manual to avoid user's
confusion.
>> case3 1.05 0.01
>> case4 1/0 103/0
>> case5 18.50 18.33
>> case6 111123/0 1111123/0
>
> The last two lines (case5 + case6) seem a bit suspicious. I believe
> those are for the histogram data, and I do get these numbers:
>
> case5 0.93 (5517 / 5949) 42.0 (249943 / 5949)
> case6 100/0 100/0
>
> Perhaps you've been using the version before the bugfix, with ANALYZE
> on the wrong table?
You are right. I accidentally ANALYZE t2, not t3. Now I get these
numbers:
case5 1.23 (7367 / 5968) 41.7 (249118 / 5981)
case6 117/0 162092/0
>> 2) following comments by me are not addressed in the v18 patch.
>>
>>> - There's no docs for pg_mv_statistic (should be added to "49. System
>>> Catalogs")
>>>
>>> - The word "multivariate statistics" or something like that should
>>> appear in the index.
>>>
>>> - There are some explanation how to deal with multivariate statistics
>>> in "14.1 Using Explain" and "14.2 Statistics used by the Planner"
>>> section.
>
> Yes, those are valid omissions. I plan to address them, and I'd also
> considering adding a section to 65.1 (How the Planner Uses
> Statistics), explaining more thoroughly how the planner uses
> multivariate stats.
Great.
Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: multivariate statistics v14
@ 2016-04-08 15:55 Robert Haas <robertmhaas@gmail.com>
parent: David Steele <david@pgmasters.net>
0 siblings, 1 reply; 58+ messages in thread
From: Robert Haas @ 2016-04-08 15:55 UTC (permalink / raw)
To: David Steele <david@pgmasters.net>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Tatsuo Ishii <ishii@postgresql.org>; Tom Lane <tgl@sss.pgh.pa.us>; Álvaro Herrera <alvherre@2ndquadrant.com>; Petr Jelinek <petr@2ndquadrant.com>; Jeff Janes <jeff.janes@gmail.com>; pgsql-hackers
On Tue, Mar 29, 2016 at 11:18 AM, David Steele <david@pgmasters.net> wrote:
> On 3/28/16 4:42 AM, Tomas Vondra wrote:
>> Yes, those are valid omissions. I plan to address them, and I'd also
>> considering adding a section to 65.1 (How the Planner Uses Statistics),
>> explaining more thoroughly how the planner uses multivariate stats.
>
> It looks you need post a new patch so I have marked this "waiting on
> author".
Since no new version of this patch has been posted in the last 10
days, it seems clear that there will not be time for this to
reasonably become ready for committer and then get committed in the
few hours remaining before the deadline. That is a bummer, since I
was hoping we would have this feature in this release, but hopefully
we will get it into 9.7. I am marking it Returned with Feedback.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: multivariate statistics v14
@ 2016-04-08 18:55 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Robert Haas <robertmhaas@gmail.com>
0 siblings, 2 replies; 58+ messages in thread
From: Tomas Vondra @ 2016-04-08 18:55 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; David Steele <david@pgmasters.net>; +Cc: Tatsuo Ishii <ishii@postgresql.org>; Tom Lane <tgl@sss.pgh.pa.us>; Álvaro Herrera <alvherre@2ndquadrant.com>; Petr Jelinek <petr@2ndquadrant.com>; Jeff Janes <jeff.janes@gmail.com>; pgsql-hackers
On 04/08/2016 05:55 PM, Robert Haas wrote:
> On Tue, Mar 29, 2016 at 11:18 AM, David Steele <david@pgmasters.net> wrote:
>> On 3/28/16 4:42 AM, Tomas Vondra wrote:
>>> Yes, those are valid omissions. I plan to address them, and I'd also
>>> considering adding a section to 65.1 (How the Planner Uses Statistics),
>>> explaining more thoroughly how the planner uses multivariate stats.
>>
>> It looks you need post a new patch so I have marked this "waiting on
>> author".
>
> Since no new version of this patch has been posted in the last 10
> days, it seems clear that there will not be time for this to
> reasonably become ready for committer and then get committed in the
> few hours remaining before the deadline. That is a bummer, since I
> was hoping we would have this feature in this release, but hopefully
> we will get it into 9.7. I am marking it Returned with Feedback.
>
Well, me to. But my feeling is the patch received entirely insufficient
amount of thorough code review, considering how important part of the
code it touches. I agree docs are an important part of a patch, but
polishing user-level docs would hardly move the patch closer to being
committable (especially when there's ~50kB of READMEs).
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: multivariate statistics v14
@ 2016-04-08 19:03 Robert Haas <robertmhaas@gmail.com>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
1 sibling, 1 reply; 58+ messages in thread
From: Robert Haas @ 2016-04-08 19:03 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: David Steele <david@pgmasters.net>; Tatsuo Ishii <ishii@postgresql.org>; Tom Lane <tgl@sss.pgh.pa.us>; Álvaro Herrera <alvherre@2ndquadrant.com>; Petr Jelinek <petr@2ndquadrant.com>; Jeff Janes <jeff.janes@gmail.com>; pgsql-hackers
On Fri, Apr 8, 2016 at 2:55 PM, Tomas Vondra
<tomas.vondra@2ndquadrant.com> wrote:
> Well, me to. But my feeling is the patch received entirely insufficient
> amount of thorough code review, considering how important part of the code
> it touches. I agree docs are an important part of a patch, but polishing
> user-level docs would hardly move the patch closer to being committable
> (especially when there's ~50kB of READMEs).
I have to admit that I was really hoping Tom would follow through on
his statement that he would look into this one, or that Dean Rasheed
would get involved. I am sure I could do a good review of this patch
given enough time, but I am also sure that it would take an amount of
time that is at least one if not two orders of magnitude more than I
put into any patch this CommitFest. I understand statistics at some
basic level, but I am not an expert on them the way some people here
are.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: multivariate statistics v14
@ 2016-04-08 19:13 Tom Lane <tgl@sss.pgh.pa.us>
parent: Robert Haas <robertmhaas@gmail.com>
0 siblings, 2 replies; 58+ messages in thread
From: Tom Lane @ 2016-04-08 19:13 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; David Steele <david@pgmasters.net>; Tatsuo Ishii <ishii@postgresql.org>; Álvaro Herrera <alvherre@2ndquadrant.com>; Petr Jelinek <petr@2ndquadrant.com>; Jeff Janes <jeff.janes@gmail.com>; pgsql-hackers
Robert Haas <robertmhaas@gmail.com> writes:
> On Fri, Apr 8, 2016 at 2:55 PM, Tomas Vondra
> <tomas.vondra@2ndquadrant.com> wrote:
>> Well, me to. But my feeling is the patch received entirely insufficient
>> amount of thorough code review, considering how important part of the code
>> it touches. I agree docs are an important part of a patch, but polishing
>> user-level docs would hardly move the patch closer to being committable
>> (especially when there's ~50kB of READMEs).
> I have to admit that I was really hoping Tom would follow through on
> his statement that he would look into this one, or that Dean Rasheed
> would get involved.
I'm sorry I didn't get to it, but it's not like I have been slacking
during this commitfest. At some point, you just have to accept that
not everything we could wish will get into 9.6.
I will make it a high priority for 9.7, though.
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: multivariate statistics v14
@ 2016-04-08 19:26 Robert Haas <robertmhaas@gmail.com>
parent: Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 0 replies; 58+ messages in thread
From: Robert Haas @ 2016-04-08 19:26 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; David Steele <david@pgmasters.net>; Tatsuo Ishii <ishii@postgresql.org>; Álvaro Herrera <alvherre@2ndquadrant.com>; Petr Jelinek <petr@2ndquadrant.com>; Jeff Janes <jeff.janes@gmail.com>; pgsql-hackers
On Fri, Apr 8, 2016 at 3:13 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
>> On Fri, Apr 8, 2016 at 2:55 PM, Tomas Vondra
>> <tomas.vondra@2ndquadrant.com> wrote:
>>> Well, me to. But my feeling is the patch received entirely insufficient
>>> amount of thorough code review, considering how important part of the code
>>> it touches. I agree docs are an important part of a patch, but polishing
>>> user-level docs would hardly move the patch closer to being committable
>>> (especially when there's ~50kB of READMEs).
>
>> I have to admit that I was really hoping Tom would follow through on
>> his statement that he would look into this one, or that Dean Rasheed
>> would get involved.
>
> I'm sorry I didn't get to it, but it's not like I have been slacking
> during this commitfest. At some point, you just have to accept that
> not everything we could wish will get into 9.6.
I did not mean to imply otherwise. I'm just explaining why I didn't
spend time on it - I figured I was not the most qualified person, and
of course I have not been slacking either. :-)
> I will make it a high priority for 9.7, though.
Woohoo!
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: multivariate statistics v14
@ 2016-04-08 23:21 Tatsuo Ishii <ishii@postgresql.org>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
1 sibling, 1 reply; 58+ messages in thread
From: Tatsuo Ishii @ 2016-04-08 23:21 UTC (permalink / raw)
To: tomas.vondra@2ndquadrant.com; +Cc: robertmhaas@gmail.com; david@pgmasters.net; ishii@postgresql.org; tgl@sss.pgh.pa.us; alvherre@2ndquadrant.com; petr@2ndquadrant.com; jeff.janes@gmail.com; pgsql-hackers
From: Tomas Vondra <tomas.vondra@2ndquadrant.com>
Subject: Re: [HACKERS] multivariate statistics v14
Date: Fri, 8 Apr 2016 20:55:24 +0200
Message-ID: <5d1d62a6-6228-188c-e079-c1be59942168@2ndquadrant.com>
> On 04/08/2016 05:55 PM, Robert Haas wrote:
>> On Tue, Mar 29, 2016 at 11:18 AM, David Steele <david@pgmasters.net>
>> wrote:
>>> On 3/28/16 4:42 AM, Tomas Vondra wrote:
>>>> Yes, those are valid omissions. I plan to address them, and I'd also
>>>> considering adding a section to 65.1 (How the Planner Uses
>>>> Statistics),
>>>> explaining more thoroughly how the planner uses multivariate stats.
>>>
>>> It looks you need post a new patch so I have marked this "waiting on
>>> author".
>>
>> Since no new version of this patch has been posted in the last 10
>> days, it seems clear that there will not be time for this to
>> reasonably become ready for committer and then get committed in the
>> few hours remaining before the deadline. That is a bummer, since I
>> was hoping we would have this feature in this release, but hopefully
>> we will get it into 9.7. I am marking it Returned with Feedback.
>>
>
> Well, me to. But my feeling is the patch received entirely
> insufficient amount of thorough code review, considering how important
> part of the code it touches. I agree docs are an important part of a
> patch, but polishing user-level docs would hardly move the patch
> closer to being committable (especially when there's ~50kB of
> READMEs).
My feedback regarding docs were:
> - There's no docs for pg_mv_statistic (should be added to "49. System
> Catalogs")
>
> - The word "multivariate statistics" or something like that should
> appear in the index.
>
> - There are some explanation how to deal with multivariate statistics
> in "14.1 Using Explain" and "14.2 Statistics used by the Planner"
> section.
The second and the third point maybe are something like "polishing
user-level" docs, but I don't think the first one is for "user-level".
Also I think without the first one the patch will be never
committable. If someone add a new system catalog, the doc should be
added to "System Catalogs" section, that's our standard, at least in
my understanding.
Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: multivariate statistics v14
@ 2016-04-09 10:00 Simon Riggs <simon@2ndQuadrant.com>
parent: Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 0 replies; 58+ messages in thread
From: Simon Riggs @ 2016-04-09 10:00 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Robert Haas <robertmhaas@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; David Steele <david@pgmasters.net>; Tatsuo Ishii <ishii@postgresql.org>; Álvaro Herrera <alvherre@2ndquadrant.com>; Petr Jelinek <petr@2ndquadrant.com>; Jeff Janes <jeff.janes@gmail.com>; pgsql-hackers
On 8 April 2016 at 20:13, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> I will make it a high priority for 9.7, though.
>
That is my plan also. I've already started reviewing the non-planner parts
anyway, specifically patch 0002.
--
Simon Riggs http://www.2ndQuadrant.com/
<http://www.2ndquadrant.com/;
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: multivariate statistics v14
@ 2016-04-09 11:32 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Tatsuo Ishii <ishii@postgresql.org>
0 siblings, 1 reply; 58+ messages in thread
From: Tomas Vondra @ 2016-04-09 11:32 UTC (permalink / raw)
To: Tatsuo Ishii <ishii@postgresql.org>; +Cc: robertmhaas@gmail.com; david@pgmasters.net; tgl@sss.pgh.pa.us; alvherre@2ndquadrant.com; petr@2ndquadrant.com; jeff.janes@gmail.com; pgsql-hackers
Hi,
On 04/09/2016 01:21 AM, Tatsuo Ishii wrote:
> From: Tomas Vondra <tomas.vondra@2ndquadrant.com>
...
> My feedback regarding docs were:
>> - There's no docs for pg_mv_statistic (should be added to "49. System
>> Catalogs")
>>
>> - The word "multivariate statistics" or something like that should
>> appear in the index.
>>
>> - There are some explanation how to deal with multivariate statistics
>> in "14.1 Using Explain" and "14.2 Statistics used by the Planner"
>> section.
>
> The second and the third point maybe are something like "polishing
> user-level" docs, but I don't think the first one is for "user-level".
> Also I think without the first one the patch will be never
> committable. If someone add a new system catalog, the doc should be
> added to "System Catalogs" section, that's our standard, at least in
> my understanding.
I do apologize if it seemed that I don't value your review, and I do
agree that those changes need to be done, although I still see them
rather as a user-level docs (as opposed to READMEs/comments, which I
think are used by developers much more often).
But I still think it wouldn't move the patch any closer to committable
state, because what it really needs is review whether the catalog
definition makes sense, whether it should be more like pg_statistic, and
so on. Only then it makes sense to describe the catalog structure in the
SGML docs, I think. That's why I added some basic SGML docs for
CREATE/DROP/ALTER STATISTICS, which I expect to be rather stable, and
not the catalog and other low-level stuff (which is commented heavily in
the code anyway).
Had the patch been a Titanic, fixing the SGML docs a few days before the
code freeze would be akin to washing the deck instead of looking for
icebergs on April 15, 1912.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: multivariate statistics v14
@ 2016-04-09 17:37 Tatsuo Ishii <ishii@postgresql.org>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 2 replies; 58+ messages in thread
From: Tatsuo Ishii @ 2016-04-09 17:37 UTC (permalink / raw)
To: tomas.vondra@2ndquadrant.com; +Cc: ishii@postgresql.org; robertmhaas@gmail.com; david@pgmasters.net; tgl@sss.pgh.pa.us; alvherre@2ndquadrant.com; petr@2ndquadrant.com; jeff.janes@gmail.com; pgsql-hackers
> But I still think it wouldn't move the patch any closer to committable
> state, because what it really needs is review whether the catalog
> definition makes sense, whether it should be more like pg_statistic,
> and so on. Only then it makes sense to describe the catalog structure
> in the SGML docs, I think. That's why I added some basic SGML docs for
> CREATE/DROP/ALTER STATISTICS, which I expect to be rather stable, and
> not the catalog and other low-level stuff (which is commented heavily
> in the code anyway).
Without "user-level docs" (now I understand that the term means all
SGML docs for you), it is very hard to find a visible
characteristics/behavior of the patch. CREATE/DROP/ALTER STATISTICS
just defines a user interface, and does not help how it affects to the
planning. The READMEs do not help either.
In this case reviewing your code is something like reviewing a program
which has no specification.
That's the reason why I said before below, but it was never seriously
considered.
>> - There are some explanation how to deal with multivariate statistics
>> in "14.1 Using Explain" and "14.2 Statistics used by the Planner"
>> section.
Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: multivariate statistics v14
@ 2016-04-10 08:25 Simon Riggs <simon@2ndQuadrant.com>
parent: Tatsuo Ishii <ishii@postgresql.org>
1 sibling, 1 reply; 58+ messages in thread
From: Simon Riggs @ 2016-04-10 08:25 UTC (permalink / raw)
To: Tatsuo Ishii <ishii@postgresql.org>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Robert Haas <robertmhaas@gmail.com>; David Steele <david@pgmasters.net>; Tom Lane <tgl@sss.pgh.pa.us>; Álvaro Herrera <alvherre@2ndquadrant.com>; Petr Jelinek <petr@2ndquadrant.com>; Jeff Janes <jeff.janes@gmail.com>; pgsql-hackers
On 9 April 2016 at 18:37, Tatsuo Ishii <ishii@postgresql.org> wrote:
> > But I still think it wouldn't move the patch any closer to committable
> > state, because what it really needs is review whether the catalog
> > definition makes sense, whether it should be more like pg_statistic,
> > and so on. Only then it makes sense to describe the catalog structure
> > in the SGML docs, I think. That's why I added some basic SGML docs for
> > CREATE/DROP/ALTER STATISTICS, which I expect to be rather stable, and
> > not the catalog and other low-level stuff (which is commented heavily
> > in the code anyway).
>
> Without "user-level docs" (now I understand that the term means all
> SGML docs for you), it is very hard to find a visible
> characteristics/behavior of the patch. CREATE/DROP/ALTER STATISTICS
> just defines a user interface, and does not help how it affects to the
> planning. The READMEs do not help either.
>
> In this case reviewing your code is something like reviewing a program
> which has no specification.
>
> That's the reason why I said before below, but it was never seriously
> considered.
>
I would likely have said this myself but didn't even get that far.
Your contribution was useful and went further than anybody else's review,
so thank you.
--
Simon Riggs http://www.2ndQuadrant.com/
<http://www.2ndquadrant.com/;
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: multivariate statistics v14
@ 2016-04-10 18:27 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Tatsuo Ishii <ishii@postgresql.org>
1 sibling, 0 replies; 58+ messages in thread
From: Tomas Vondra @ 2016-04-10 18:27 UTC (permalink / raw)
To: Tatsuo Ishii <ishii@postgresql.org>; +Cc: robertmhaas@gmail.com; david@pgmasters.net; tgl@sss.pgh.pa.us; alvherre@2ndquadrant.com; petr@2ndquadrant.com; jeff.janes@gmail.com; pgsql-hackers
Hello,
On 04/09/2016 07:37 PM, Tatsuo Ishii wrote:
>> But I still think it wouldn't move the patch any closer to committable
>> state, because what it really needs is review whether the catalog
>> definition makes sense, whether it should be more like pg_statistic,
>> and so on. Only then it makes sense to describe the catalog structure
>> in the SGML docs, I think. That's why I added some basic SGML docs for
>> CREATE/DROP/ALTER STATISTICS, which I expect to be rather stable, and
>> not the catalog and other low-level stuff (which is commented heavily
>> in the code anyway).
>
> Without "user-level docs" (now I understand that the term means all
> SGML docs for you), it is very hard to find a visible
> characteristics/behavior of the patch. CREATE/DROP/ALTER STATISTICS
> just defines a user interface, and does not help how it affects to
> the planning. The READMEs do not help either.
>
> In this case reviewing your code is something like reviewing a
> program which has no specification.
I certainly agree that reviewing a patch without the context is hard. My
intent was to provide such context / explanation in the READMEs, but
perhaps I failed to do so with enough detail.
BTW when you say that READMEs do not help either, does that mean you
consider READMEs unsuitable for this type of information in general, or
that the current READMEs lack important information?
>
> That's the reason why I said before below, but it was never
> seriously considered.
>
I've considered it, but my plan was to have detailed READMEs, and then
eventually distill that into something suitable for the SGML (perhaps
without discussion of some implementation details). Maybe that's not the
right approach.
FWIW providing the context is why I started working on a "paper"
explaining both the motivation and implementation, including a bit of
math and figures (which is what we don't have in READMEs or SGML). I
haven't updated it recently, and it probably got buried in the thread,
but perhaps this would be a better way to provide the context?
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 58+ messages in thread
* Re: multivariate statistics v14
@ 2016-04-10 18:29 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Simon Riggs <simon@2ndQuadrant.com>
0 siblings, 0 replies; 58+ messages in thread
From: Tomas Vondra @ 2016-04-10 18:29 UTC (permalink / raw)
To: Simon Riggs <simon@2ndQuadrant.com>; Tatsuo Ishii <ishii@postgresql.org>; +Cc: Robert Haas <robertmhaas@gmail.com>; David Steele <david@pgmasters.net>; Tom Lane <tgl@sss.pgh.pa.us>; Álvaro Herrera <alvherre@2ndquadrant.com>; Petr Jelinek <petr@2ndquadrant.com>; Jeff Janes <jeff.janes@gmail.com>; pgsql-hackers
On 04/10/2016 10:25 AM, Simon Riggs wrote:
> On 9 April 2016 at 18:37, Tatsuo Ishii <ishii@postgresql.org
> <mailto:ishii@postgresql.org>> wrote:
>
> > But I still think it wouldn't move the patch any closer to committable
> > state, because what it really needs is review whether the catalog
> > definition makes sense, whether it should be more like pg_statistic,
> > and so on. Only then it makes sense to describe the catalog structure
> > in the SGML docs, I think. That's why I added some basic SGML docs for
> > CREATE/DROP/ALTER STATISTICS, which I expect to be rather stable, and
> > not the catalog and other low-level stuff (which is commented heavily
> > in the code anyway).
>
> Without "user-level docs" (now I understand that the term means all
> SGML docs for you), it is very hard to find a visible
> characteristics/behavior of the patch. CREATE/DROP/ALTER STATISTICS
> just defines a user interface, and does not help how it affects to the
> planning. The READMEs do not help either.
>
> In this case reviewing your code is something like reviewing a program
> which has no specification.
>
> That's the reason why I said before below, but it was never seriously
> considered.
>
>
> I would likely have said this myself but didn't even get that far.
>
> Your contribution was useful and went further than anybody else's
> review, so thank you.
100% agreed. Thanks for the useful feedback.
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 58+ messages in thread
end of thread, other threads:[~2016-04-10 18:29 UTC | newest]
Thread overview: 58+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2016-03-09 15:02 Re: multivariate statistics v14 Tomas Vondra <tomas.vondra@2ndquadrant.com>
2016-03-09 16:45 ` Jeff Janes <jeff.janes@gmail.com>
2016-03-09 17:21 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2016-03-09 18:09 ` Jeff Janes <jeff.janes@gmail.com>
2016-03-09 18:18 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2016-03-13 07:30 ` Jeff Janes <jeff.janes@gmail.com>
2016-03-13 21:59 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2016-03-16 02:29 ` Tatsuo Ishii <ishii@postgresql.org>
2016-03-16 02:58 ` Tatsuo Ishii <ishii@postgresql.org>
2016-03-16 13:45 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2016-03-22 08:13 ` Tatsuo Ishii <ishii@postgresql.org>
2016-03-22 09:44 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2016-03-22 10:41 ` Tatsuo Ishii <ishii@postgresql.org>
2016-03-22 12:37 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2016-03-22 12:46 ` Tatsuo Ishii <ishii@postgresql.org>
2016-03-22 13:49 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2016-03-23 01:53 ` Tatsuo Ishii <ishii@postgresql.org>
2016-03-23 01:57 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2016-03-23 02:01 ` Tatsuo Ishii <ishii@postgresql.org>
2016-03-23 02:57 ` Tatsuo Ishii <ishii@postgresql.org>
2016-03-23 03:01 ` Tatsuo Ishii <ishii@postgresql.org>
2016-03-23 05:20 ` Tatsuo Ishii <ishii@postgresql.org>
2016-03-23 13:21 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2016-03-23 18:23 ` Petr Jelinek <petr@2ndquadrant.com>
2016-03-24 17:12 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2016-03-24 17:45 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2016-03-25 15:32 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2016-03-25 21:26 ` Tom Lane <tgl@sss.pgh.pa.us>
2016-03-26 02:02 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2016-03-26 09:18 ` Tatsuo Ishii <ishii@postgresql.org>
2016-03-28 08:42 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2016-03-29 15:18 ` David Steele <david@pgmasters.net>
2016-04-08 15:55 ` Robert Haas <robertmhaas@gmail.com>
2016-04-08 18:55 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2016-04-08 19:03 ` Robert Haas <robertmhaas@gmail.com>
2016-04-08 19:13 ` Tom Lane <tgl@sss.pgh.pa.us>
2016-04-08 19:26 ` Robert Haas <robertmhaas@gmail.com>
2016-04-09 10:00 ` Simon Riggs <simon@2ndQuadrant.com>
2016-04-08 23:21 ` Tatsuo Ishii <ishii@postgresql.org>
2016-04-09 11:32 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2016-04-09 17:37 ` Tatsuo Ishii <ishii@postgresql.org>
2016-04-10 08:25 ` Simon Riggs <simon@2ndQuadrant.com>
2016-04-10 18:29 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2016-04-10 18:27 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2016-03-30 05:15 ` Tatsuo Ishii <ishii@postgresql.org>
2016-03-26 19:09 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2016-03-28 08:49 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2016-03-28 15:54 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2016-03-16 08:31 ` Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
2016-03-16 13:32 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2016-03-20 23:00 ` Tatsuo Ishii <ishii@postgresql.org>
2016-03-20 23:34 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2016-03-21 03:34 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2016-03-21 09:34 ` Robert Haas <robertmhaas@gmail.com>
2016-03-21 10:08 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2016-03-21 10:30 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2016-03-22 05:53 ` Jeff Janes <jeff.janes@gmail.com>
2016-03-22 09:23 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox