public inbox for [email protected]help / color / mirror / Atom feed
[PATCH 3/3] Extended statistics on expressions 2+ messages / 2 participants [nested] [flat]
* [PATCH 3/3] Extended statistics on expressions @ 2020-12-03 15:19 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 2+ messages in thread From: Tomas Vondra @ 2020-12-03 15:19 UTC (permalink / raw) Allow defining extended statistics on expressions, not just simple column references. With this commit, it's possible to do things like CREATE TABLE t (a int); CREATE STATISTICS s ON mod(a,10), mod(a,20) FROM t; and the collected statistics will be useful for estimating queries using those expressions in various places, like SELECT * FROM t WHERE mod(a,10) = 0 AND mod(a,20) = 0; or SELECT mod(a,10), mod(a,20) FROM t GROUP BY 1, 2; The commit also adds a new statistics type "expressions" which builds the usual per-column statistics for each expression, allowing better estimates even for queries with just a single expression, which are not affected by multi-column statistics. This achieves the same goal as creating expression indexes, without index maintenance overhead. --- doc/src/sgml/catalogs.sgml | 236 +++ doc/src/sgml/ref/create_statistics.sgml | 94 +- src/backend/catalog/Makefile | 8 +- src/backend/catalog/system_views.sql | 74 + src/backend/commands/statscmds.c | 378 ++++- src/backend/nodes/copyfuncs.c | 14 + src/backend/nodes/equalfuncs.c | 13 + src/backend/nodes/outfuncs.c | 12 + src/backend/optimizer/util/plancat.c | 53 + src/backend/parser/gram.y | 31 +- src/backend/parser/parse_agg.c | 10 + src/backend/parser/parse_expr.c | 6 + src/backend/parser/parse_func.c | 3 + src/backend/parser/parse_utilcmd.c | 120 +- src/backend/statistics/dependencies.c | 366 +++- src/backend/statistics/extended_stats.c | 1486 ++++++++++++++++- src/backend/statistics/mcv.c | 293 +++- src/backend/statistics/mvdistinct.c | 99 +- src/backend/tcop/utility.c | 17 +- src/backend/utils/adt/ruleutils.c | 295 +++- src/backend/utils/adt/selfuncs.c | 407 ++++- src/bin/psql/describe.c | 22 +- src/include/catalog/pg_proc.dat | 8 + src/include/catalog/pg_statistic_ext.h | 4 + src/include/catalog/pg_statistic_ext_data.h | 1 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 16 + src/include/nodes/pathnodes.h | 1 + src/include/parser/parse_node.h | 1 + src/include/parser/parse_utilcmd.h | 2 + .../statistics/extended_stats_internal.h | 40 +- src/include/statistics/statistics.h | 2 + src/test/regress/expected/rules.out | 75 + src/test/regress/expected/stats_ext.out | 674 +++++++- src/test/regress/sql/stats_ext.sql | 310 +++- 35 files changed, 4816 insertions(+), 356 deletions(-) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index 79069ddfab..0c1d9a2b11 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -9362,6 +9362,11 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l <entry>extended planner statistics</entry> </row> + <row> + <entry><link linkend="view-pg-stats-ext-exprs"><structname>pg_stats_ext_exprs</structname></link></entry> + <entry>extended planner statistics for expressions</entry> + </row> + <row> <entry><link linkend="view-pg-tables"><structname>pg_tables</structname></link></entry> <entry>tables</entry> @@ -12924,6 +12929,237 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx </sect1> + <sect1 id="view-pg-stats-ext-exprs"> + <title><structname>pg_stats_ext_exprs</structname></title> + + <indexterm zone="view-pg-stats-ext-exprs"> + <primary>pg_stats_ext_exprs</primary> + </indexterm> + + <para> + The view <structname>pg_stats_ext_exprs</structname> provides access to + the information stored in the <link + linkend="catalog-pg-statistic-ext"><structname>pg_statistic_ext</structname></link> + and <link linkend="catalog-pg-statistic-ext-data"><structname>pg_statistic_ext_data</structname></link> + catalogs. This view allows access only to rows of + <link linkend="catalog-pg-statistic-ext"><structname>pg_statistic_ext</structname></link> and <link linkend="catalog-pg-statistic-ext-data"><structname>pg_statistic_ext_data</structname></link> + that correspond to tables the user has permission to read, and therefore + it is safe to allow public read access to this view. + </para> + + <para> + <structname>pg_stats_ext_exprs</structname> is also designed to present + the information in a more readable format than the underlying catalogs + — at the cost that its schema must be extended whenever the structure + of statistics <link linkend="catalog-pg-statistic"><structname>pg_statistic</structname></link> changes. + </para> + + <table> + <title><structname>pg_stats_ext_exprs</structname> Columns</title> + <tgroup cols="1"> + <thead> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + Column Type + </para> + <para> + Description + </para></entry> + </row> + </thead> + + <tbody> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>schemaname</structfield> <type>name</type> + (references <link linkend="catalog-pg-namespace"><structname>pg_namespace</structname></link>.<structfield>nspname</structfield>) + </para> + <para> + Name of schema containing table + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>tablename</structfield> <type>name</type> + (references <link linkend="catalog-pg-class"><structname>pg_class</structname></link>.<structfield>relname</structfield>) + </para> + <para> + Name of table + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>statistics_schemaname</structfield> <type>name</type> + (references <link linkend="catalog-pg-namespace"><structname>pg_namespace</structname></link>.<structfield>nspname</structfield>) + </para> + <para> + Name of schema containing extended statistic + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>statistics_name</structfield> <type>name</type> + (references <link linkend="catalog-pg-statistic-ext"><structname>pg_statistic_ext</structname></link>.<structfield>stxname</structfield>) + </para> + <para> + Name of extended statistics + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>statistics_owner</structfield> <type>name</type> + (references <link linkend="catalog-pg-authid"><structname>pg_authid</structname></link>.<structfield>rolname</structfield>) + </para> + <para> + Owner of the extended statistics + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>expr</structfield> <type>text</type> + </para> + <para> + Expression the extended statistics is defined on + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>null_frac</structfield> <type>float4</type> + </para> + <para> + Fraction of column entries that are null + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>avg_width</structfield> <type>int4</type> + </para> + <para> + Average width in bytes of column's entries + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>n_distinct</structfield> <type>float4</type> + </para> + <para> + If greater than zero, the estimated number of distinct values in the + column. If less than zero, the negative of the number of distinct + values divided by the number of rows. (The negated form is used when + <command>ANALYZE</command> believes that the number of distinct values is + likely to increase as the table grows; the positive form is used when + the column seems to have a fixed number of possible values.) For + example, -1 indicates a unique column in which the number of distinct + values is the same as the number of rows. + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>most_common_vals</structfield> <type>anyarray</type> + </para> + <para> + A list of the most common values in the column. (Null if + no values seem to be more common than any others.) + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>most_common_freqs</structfield> <type>float4[]</type> + </para> + <para> + A list of the frequencies of the most common values, + i.e., number of occurrences of each divided by total number of rows. + (Null when <structfield>most_common_vals</structfield> is.) + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>histogram_bounds</structfield> <type>anyarray</type> + </para> + <para> + A list of values that divide the column's values into groups of + approximately equal population. The values in + <structfield>most_common_vals</structfield>, if present, are omitted from this + histogram calculation. (This column is null if the column data type + does not have a <literal><</literal> operator or if the + <structfield>most_common_vals</structfield> list accounts for the entire + population.) + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>correlation</structfield> <type>float4</type> + </para> + <para> + Statistical correlation between physical row ordering and + logical ordering of the column values. This ranges from -1 to +1. + When the value is near -1 or +1, an index scan on the column will + be estimated to be cheaper than when it is near zero, due to reduction + of random access to the disk. (This column is null if the column data + type does not have a <literal><</literal> operator.) + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>most_common_elems</structfield> <type>anyarray</type> + </para> + <para> + A list of non-null element values most often appearing within values of + the column. (Null for scalar types.) + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>most_common_elem_freqs</structfield> <type>float4[]</type> + </para> + <para> + A list of the frequencies of the most common element values, i.e., the + fraction of rows containing at least one instance of the given value. + Two or three additional values follow the per-element frequencies; + these are the minimum and maximum of the preceding per-element + frequencies, and optionally the frequency of null elements. + (Null when <structfield>most_common_elems</structfield> is.) + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>elem_count_histogram</structfield> <type>float4[]</type> + </para> + <para> + A histogram of the counts of distinct non-null element values within the + values of the column, followed by the average number of distinct + non-null elements. (Null for scalar types.) + </para></entry> + </row> + </tbody> + </tgroup> + </table> + + <para> + The maximum number of entries in the array fields can be controlled on a + column-by-column basis using the <link linkend="sql-altertable"><command>ALTER + TABLE SET STATISTICS</command></link> + command, or globally by setting the + <xref linkend="guc-default-statistics-target"/> run-time parameter. + </para> + + </sect1> + <sect1 id="view-pg-tables"> <title><structname>pg_tables</structname></title> diff --git a/doc/src/sgml/ref/create_statistics.sgml b/doc/src/sgml/ref/create_statistics.sgml index 4363be50c3..518d99ed8a 100644 --- a/doc/src/sgml/ref/create_statistics.sgml +++ b/doc/src/sgml/ref/create_statistics.sgml @@ -23,7 +23,7 @@ PostgreSQL documentation <synopsis> CREATE STATISTICS [ IF NOT EXISTS ] <replaceable class="parameter">statistics_name</replaceable> [ ( <replaceable class="parameter">statistics_kind</replaceable> [, ... ] ) ] - ON <replaceable class="parameter">column_name</replaceable>, <replaceable class="parameter">column_name</replaceable> [, ...] + ON { <replaceable class="parameter">column_name</replaceable> | ( <replaceable class="parameter">expression</replaceable> ) } [, ...] FROM <replaceable class="parameter">table_name</replaceable> </synopsis> @@ -81,12 +81,15 @@ CREATE STATISTICS [ IF NOT EXISTS ] <replaceable class="parameter">statistics_na <para> A statistics kind to be computed in this statistics object. Currently supported kinds are + <literal>expressions</literal>, which enables expression statistics, <literal>ndistinct</literal>, which enables n-distinct statistics, <literal>dependencies</literal>, which enables functional dependency statistics, and <literal>mcv</literal> which enables most-common values lists. If this clause is omitted, all supported statistics kinds are - included in the statistics object. + included in the statistics object. Expression statistics are included + only when the statistics definition includes complex expressions and + not just simple column references. For more information, see <xref linkend="planner-stats-extended"/> and <xref linkend="multivariate-statistics-examples"/>. </para> @@ -104,6 +107,17 @@ CREATE STATISTICS [ IF NOT EXISTS ] <replaceable class="parameter">statistics_na </listitem> </varlistentry> + <varlistentry> + <term><replaceable class="parameter">expression</replaceable></term> + <listitem> + <para> + The expression to be covered by the computed statistics. In this case + only a single expression is required, in which case only the expression + statistics kind is allowed. The order of expressions is insignificant. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><replaceable class="parameter">table_name</replaceable></term> <listitem> @@ -125,6 +139,22 @@ CREATE STATISTICS [ IF NOT EXISTS ] <replaceable class="parameter">statistics_na reading it. Once created, however, the ownership of the statistics object is independent of the underlying table(s). </para> + + <para> + Creating expression statistics is allowed only when expressions are given. + Expression statistics are per-expression and are similar to creating an + index on the expression, except that they avoid the overhead of the index. + </para> + + <para> + All functions and operators used in a statistics definition must be + <quote>immutable</quote>, that is, their results must depend only on + their arguments and never on any outside influence (such as + the contents of another table or the current time). This restriction + ensures that the behavior of the statistics is well-defined. To use a + user-defined function in a statistics expression, remember to mark + the function immutable when you create it. + </para> </refsect1> <refsect1 id="sql-createstatistics-examples"> @@ -196,6 +226,66 @@ EXPLAIN ANALYZE SELECT * FROM t2 WHERE (a = 1) AND (b = 2); in the table, allowing it to generate better estimates in both cases. </para> + <para> + Create table <structname>t3</structname> with a single timestamp column, + and run a query using an expression on that column. Without the + extended statistics, the planner has no information about data + distribution for reasults of those expression, and uses default + estimates as illustrated by the first query. The planner also does + not realize the value of the second column fully defines the value + of the other column, because date truncated to day still identifies + the month). Then expression and ndistinct statistics are built on + those two columns: + +<programlisting> +CREATE TABLE t3 ( + a timestamp +); + +INSERT INTO t3 SELECT i FROM generate_series('2020-01-01'::timestamp, + '2020-12-31'::timestamp, + '1 minute'::interval) s(i); + +ANALYZE t3; + +-- the number of matching rows will be drastically underestimated: +EXPLAIN ANALYZE SELECT * FROM t3 + WHERE date_trunc('month', a) = '2020-01-01'::timestamp; + +EXPLAIN ANALYZE SELECT * FROM t3 + WHERE date_trunc('day', a) BETWEEN '2020-01-01'::timestamp + AND '2020-06-30'::timestamp; + +EXPLAIN ANALYZE SELECT date_trunc('month', a), date_trunc('day', a) + FROM t3 GROUP BY 1, 2; + +CREATE STATISTICS s3 (expressions, ndistinct) ON date_trunc('month', a), date_trunc('day', a) FROM t3; + +ANALYZE t1; + +-- now the row count estimates are more accurate: +EXPLAIN ANALYZE SELECT * FROM t3 + WHERE date_trunc('month', a) = '2020-01-01'::timestamp; + +EXPLAIN ANALYZE SELECT * FROM t3 + WHERE date_trunc('day', a) BETWEEN '2020-01-01'::timestamp + AND '2020-06-30'::timestamp; + +EXPLAIN ANALYZE SELECT date_trunc('month', a), date_trunc('day', a) + FROM t3 GROUP BY 1, 2; +</programlisting> + + Without expression and ndistinct statistics, the planner would assume + that the two <literal>WHERE</literal> and <literal>GROUP BY</literal> + conditions are independent, and would multiply their selectivities + together to arrive at a much-too-small row count estimate in the first + two queries, and a much-too-high group count estimate in the aggregate + query. This is further exacerbated by the lack of accurate statistics + for the expressions, forcing the planner to use default selectivities. + With such statistics, the planner recognizes that the conditions are + correlated and arrives at much more accurate estimates. + </para> + </refsect1> <refsect1> diff --git a/src/backend/catalog/Makefile b/src/backend/catalog/Makefile index 2519771210..203dfb2911 100644 --- a/src/backend/catalog/Makefile +++ b/src/backend/catalog/Makefile @@ -49,15 +49,15 @@ include $(top_srcdir)/src/backend/common.mk # Note: the order of this list determines the order in which the catalog # header files are assembled into postgres.bki. BKI_BOOTSTRAP catalogs -# must appear first, and there are reputedly other, undocumented ordering -# dependencies. +# must appear first, and pg_statistic before pg_statistic_ext_data, and +# are are reputedly other, undocumented ordering dependencies. CATALOG_HEADERS := \ pg_proc.h pg_type.h pg_attribute.h pg_class.h \ pg_attrdef.h pg_constraint.h pg_inherits.h pg_index.h pg_operator.h \ pg_opfamily.h pg_opclass.h pg_am.h pg_amop.h pg_amproc.h \ pg_language.h pg_largeobject_metadata.h pg_largeobject.h pg_aggregate.h \ - pg_statistic_ext.h pg_statistic_ext_data.h \ - pg_statistic.h pg_rewrite.h pg_trigger.h pg_event_trigger.h pg_description.h \ + pg_statistic.h pg_statistic_ext.h pg_statistic_ext_data.h \ + pg_rewrite.h pg_trigger.h pg_event_trigger.h pg_description.h \ pg_cast.h pg_enum.h pg_namespace.h pg_conversion.h pg_depend.h \ pg_database.h pg_db_role_setting.h pg_tablespace.h \ pg_authid.h pg_auth_members.h pg_shdepend.h pg_shdescription.h \ diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql index b140c210bc..b7f4880091 100644 --- a/src/backend/catalog/system_views.sql +++ b/src/backend/catalog/system_views.sql @@ -264,6 +264,7 @@ CREATE VIEW pg_stats_ext WITH (security_barrier) AS JOIN pg_attribute a ON (a.attrelid = s.stxrelid AND a.attnum = k) ) AS attnames, + pg_get_statisticsobjdef_expressions(s.oid) as exprs, s.stxkind AS kinds, sd.stxdndistinct AS n_distinct, sd.stxddependencies AS dependencies, @@ -290,6 +291,79 @@ CREATE VIEW pg_stats_ext WITH (security_barrier) AS WHERE NOT has_column_privilege(c.oid, a.attnum, 'select') ) AND (c.relrowsecurity = false OR NOT row_security_active(c.oid)); +CREATE VIEW pg_stats_ext_exprs WITH (security_barrier) AS + SELECT cn.nspname AS schemaname, + c.relname AS tablename, + sn.nspname AS statistics_schemaname, + s.stxname AS statistics_name, + pg_get_userbyid(s.stxowner) AS statistics_owner, + stat.expr, + (stat.a).stanullfrac AS null_frac, + (stat.a).stawidth AS avg_width, + (stat.a).stadistinct AS n_distinct, + (CASE + WHEN (stat.a).stakind1 = 1 THEN (stat.a).stavalues1 + WHEN (stat.a).stakind2 = 1 THEN (stat.a).stavalues2 + WHEN (stat.a).stakind3 = 1 THEN (stat.a).stavalues3 + WHEN (stat.a).stakind4 = 1 THEN (stat.a).stavalues4 + WHEN (stat.a).stakind5 = 1 THEN (stat.a).stavalues5 + END) AS most_common_vals, + (CASE + WHEN (stat.a).stakind1 = 1 THEN (stat.a).stanumbers1 + WHEN (stat.a).stakind2 = 1 THEN (stat.a).stanumbers2 + WHEN (stat.a).stakind3 = 1 THEN (stat.a).stanumbers3 + WHEN (stat.a).stakind4 = 1 THEN (stat.a).stanumbers4 + WHEN (stat.a).stakind5 = 1 THEN (stat.a).stanumbers5 + END) AS most_common_freqs, + (CASE + WHEN (stat.a).stakind1 = 2 THEN (stat.a).stavalues1 + WHEN (stat.a).stakind2 = 2 THEN (stat.a).stavalues2 + WHEN (stat.a).stakind3 = 2 THEN (stat.a).stavalues3 + WHEN (stat.a).stakind4 = 2 THEN (stat.a).stavalues4 + WHEN (stat.a).stakind5 = 2 THEN (stat.a).stavalues5 + END) AS histogram_bounds, + (CASE + WHEN (stat.a).stakind1 = 3 THEN (stat.a).stanumbers1[1] + WHEN (stat.a).stakind2 = 3 THEN (stat.a).stanumbers2[1] + WHEN (stat.a).stakind3 = 3 THEN (stat.a).stanumbers3[1] + WHEN (stat.a).stakind4 = 3 THEN (stat.a).stanumbers4[1] + WHEN (stat.a).stakind5 = 3 THEN (stat.a).stanumbers5[1] + END) correlation, + (CASE + WHEN (stat.a).stakind1 = 4 THEN (stat.a).stavalues1 + WHEN (stat.a).stakind2 = 4 THEN (stat.a).stavalues2 + WHEN (stat.a).stakind3 = 4 THEN (stat.a).stavalues3 + WHEN (stat.a).stakind4 = 4 THEN (stat.a).stavalues4 + WHEN (stat.a).stakind5 = 4 THEN (stat.a).stavalues5 + END) AS most_common_elems, + (CASE + WHEN (stat.a).stakind1 = 4 THEN (stat.a).stanumbers1 + WHEN (stat.a).stakind2 = 4 THEN (stat.a).stanumbers2 + WHEN (stat.a).stakind3 = 4 THEN (stat.a).stanumbers3 + WHEN (stat.a).stakind4 = 4 THEN (stat.a).stanumbers4 + WHEN (stat.a).stakind5 = 4 THEN (stat.a).stanumbers5 + END) AS most_common_elem_freqs, + (CASE + WHEN (stat.a).stakind1 = 5 THEN (stat.a).stanumbers1 + WHEN (stat.a).stakind2 = 5 THEN (stat.a).stanumbers2 + WHEN (stat.a).stakind3 = 5 THEN (stat.a).stanumbers3 + WHEN (stat.a).stakind4 = 5 THEN (stat.a).stanumbers4 + WHEN (stat.a).stakind5 = 5 THEN (stat.a).stanumbers5 + END) AS elem_count_histogram + FROM pg_statistic_ext s JOIN pg_class c ON (c.oid = s.stxrelid) + JOIN pg_statistic_ext_data sd ON (s.oid = sd.stxoid) + LEFT JOIN pg_namespace cn ON (cn.oid = c.relnamespace) + LEFT JOIN pg_namespace sn ON (sn.oid = s.stxnamespace) + LEFT JOIN LATERAL ( + SELECT + * + FROM ( + SELECT + unnest(pg_get_statisticsobjdef_expressions(s.oid)) AS expr, + unnest(sd.stxdexpr)::pg_statistic AS a + ) x + ) stat ON sd.stxdexpr IS NOT NULL; + -- unprivileged users may read pg_statistic_ext but not pg_statistic_ext_data REVOKE ALL on pg_statistic_ext_data FROM public; diff --git a/src/backend/commands/statscmds.c b/src/backend/commands/statscmds.c index 3057d89d50..1769d09222 100644 --- a/src/backend/commands/statscmds.c +++ b/src/backend/commands/statscmds.c @@ -29,6 +29,8 @@ #include "commands/comment.h" #include "commands/defrem.h" #include "miscadmin.h" +#include "nodes/nodeFuncs.h" +#include "optimizer/optimizer.h" #include "statistics/statistics.h" #include "utils/builtins.h" #include "utils/fmgroids.h" @@ -42,6 +44,7 @@ static char *ChooseExtendedStatisticName(const char *name1, const char *name2, const char *label, Oid namespaceid); static char *ChooseExtendedStatisticNameAddition(List *exprs); +static bool CheckMutability(Expr *expr); /* qsort comparator for the attnums in CreateStatistics */ @@ -62,7 +65,8 @@ ObjectAddress CreateStatistics(CreateStatsStmt *stmt) { int16 attnums[STATS_MAX_DIMENSIONS]; - int numcols = 0; + int nattnums = 0; + int numcols; char *namestr; NameData stxname; Oid statoid; @@ -74,21 +78,26 @@ CreateStatistics(CreateStatsStmt *stmt) Datum datavalues[Natts_pg_statistic_ext_data]; bool datanulls[Natts_pg_statistic_ext_data]; int2vector *stxkeys; + List *stxexprs = NIL; + Datum exprsDatum; Relation statrel; Relation datarel; Relation rel = NULL; Oid relid; ObjectAddress parentobject, myself; - Datum types[3]; /* one for each possible type of statistic */ + Datum types[4]; /* one for each possible type of statistic */ int ntypes; ArrayType *stxkind; bool build_ndistinct; bool build_dependencies; bool build_mcv; + bool build_expressions; + bool build_expressions_only; bool requested_type = false; int i; ListCell *cell; + ListCell *cell2; Assert(IsA(stmt, CreateStatsStmt)); @@ -183,72 +192,196 @@ CreateStatistics(CreateStatsStmt *stmt) } /* - * Currently, we only allow simple column references in the expression - * list. That will change someday, and again the grammar already supports - * it so we have to enforce restrictions here. For now, we can convert - * the expression list to a simple array of attnums. While at it, enforce - * some constraints. + * Make sure no more than STATS_MAX_DIMENSIONS columns are used. There + * might be duplicates and so on, but we'll deal with those later. + */ + numcols = list_length(stmt->exprs); + if (numcols > STATS_MAX_DIMENSIONS) + ereport(ERROR, + (errcode(ERRCODE_TOO_MANY_COLUMNS), + errmsg("cannot have more than %d columns in statistics", + STATS_MAX_DIMENSIONS))); + + /* + * Convert the expression list to a simple array of attnums. While at + * it, enforce some constraints. */ foreach(cell, stmt->exprs) { Node *expr = (Node *) lfirst(cell); - ColumnRef *cref; - char *attname; + StatsElem *selem; HeapTuple atttuple; Form_pg_attribute attForm; TypeCacheEntry *type; - if (!IsA(expr, ColumnRef)) + /* + * XXX How could we get anything else than a StatsElem, given the + * grammar? But let's keep it as a safety, maybe shall we turn it + * into an assert? + */ + if (!IsA(expr, StatsElem)) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("only simple column references are allowed in CREATE STATISTICS"))); - cref = (ColumnRef *) expr; + errmsg("only simple column references and expressions are allowed in CREATE STATISTICS"))); - if (list_length(cref->fields) != 1) - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("only simple column references are allowed in CREATE STATISTICS"))); - attname = strVal((Value *) linitial(cref->fields)); + selem = (StatsElem *) expr; - atttuple = SearchSysCacheAttName(relid, attname); - if (!HeapTupleIsValid(atttuple)) - ereport(ERROR, - (errcode(ERRCODE_UNDEFINED_COLUMN), - errmsg("column \"%s\" does not exist", - attname))); - attForm = (Form_pg_attribute) GETSTRUCT(atttuple); - - /* Disallow use of system attributes in extended stats */ - if (attForm->attnum <= 0) - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("statistics creation on system columns is not supported"))); + if (selem->name) /* column reference */ + { + char *attname; + attname = selem->name; + + atttuple = SearchSysCacheAttName(relid, attname); + if (!HeapTupleIsValid(atttuple)) + ereport(ERROR, + (errcode(ERRCODE_UNDEFINED_COLUMN), + errmsg("column \"%s\" does not exist", + attname))); + attForm = (Form_pg_attribute) GETSTRUCT(atttuple); + + /* Disallow use of system attributes in extended stats */ + if (attForm->attnum <= 0) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("statistics creation on system columns is not supported"))); + + /* Disallow data types without a less-than operator */ + type = lookup_type_cache(attForm->atttypid, TYPECACHE_LT_OPR); + if (type->lt_opr == InvalidOid) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column \"%s\" cannot be used in statistics because its type %s has no default btree operator class", + attname, format_type_be(attForm->atttypid)))); + + attnums[nattnums] = attForm->attnum; + nattnums++; + ReleaseSysCache(atttuple); + } + else /* expression */ + { + Node *expr = selem->expr; + Oid atttype; + + Assert(expr != NULL); + + /* + * An expression using mutable functions is probably wrong, + * since if you aren't going to get the same result for the + * same data every time, it's not clear what the index entries + * mean at all. + */ + if (CheckMutability((Expr *) expr)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_OBJECT_DEFINITION), + errmsg("functions in statistics expression must be marked IMMUTABLE"))); + + /* + * Disallow data types without a less-than operator + * + * XXX Maybe allow this, but only for EXPRESSIONS stats and + * prevent building e.g. MCV etc. + */ + atttype = exprType(expr); + type = lookup_type_cache(atttype, TYPECACHE_LT_OPR); + if (type->lt_opr == InvalidOid) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("expression cannot be used in statistics because its type %s has no default btree operator class", + format_type_be(atttype)))); + + stxexprs = lappend(stxexprs, expr); + } + } - /* Disallow data types without a less-than operator */ - type = lookup_type_cache(attForm->atttypid, TYPECACHE_LT_OPR); - if (type->lt_opr == InvalidOid) - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("column \"%s\" cannot be used in statistics because its type %s has no default btree operator class", - attname, format_type_be(attForm->atttypid)))); + /* + * Parse the statistics kinds. + */ + build_ndistinct = false; + build_dependencies = false; + build_mcv = false; + build_expressions = false; + foreach(cell, stmt->stat_types) + { + char *type = strVal((Value *) lfirst(cell)); - /* Make sure no more than STATS_MAX_DIMENSIONS columns are used */ - if (numcols >= STATS_MAX_DIMENSIONS) + if (strcmp(type, "ndistinct") == 0) + { + build_ndistinct = true; + requested_type = true; + } + else if (strcmp(type, "dependencies") == 0) + { + build_dependencies = true; + requested_type = true; + } + else if (strcmp(type, "mcv") == 0) + { + build_mcv = true; + requested_type = true; + } + else if (strcmp(type, "expressions") == 0) + { + build_expressions = true; + requested_type = true; + } + else ereport(ERROR, - (errcode(ERRCODE_TOO_MANY_COLUMNS), - errmsg("cannot have more than %d columns in statistics", - STATS_MAX_DIMENSIONS))); + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("unrecognized statistics kind \"%s\"", + type))); + } - attnums[numcols] = attForm->attnum; - numcols++; - ReleaseSysCache(atttuple); + /* + * If no statistic type was specified, build them all (but request + * expression stats only when there actually are any expressions). + */ + if (!requested_type) + { + build_ndistinct = (numcols >= 2); + build_dependencies = (numcols >= 2); + build_mcv = (numcols >= 2); + build_expressions = (list_length(stxexprs) != 0); } + /* Are we building only the expression statistics? */ + build_expressions_only = build_expressions && + (!build_ndistinct) && (!build_dependencies) && (!build_mcv); + + /* + * Check that with explicitly requested expression stats there really + * are some expressions. + */ + if (build_expressions && (list_length(stxexprs) == 0)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_OBJECT_DEFINITION), + errmsg("extended expression statistics require at least one expression"))); + /* - * Check that at least two columns were specified in the statement. The - * upper bound was already checked in the loop above. + * When building only expression stats, all the elements have to be + * expressions. It's pointless to build those stats for regular + * columns, as we already have that in pg_statistic. + * + * XXX This is probably easy to evade by doing "dummy" expression on + * the column, but meh. */ - if (numcols < 2) + if (build_expressions_only && (nattnums > 0)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_OBJECT_DEFINITION), + errmsg("building only extended expression statistics on simple columns not allowed"))); + + /* + * Check that at least two columns were specified in the statement, or + * one when only expression stats were requested. The upper bound was + * already checked in the loop above. + * + * XXX The first check is probably pointless after the one checking for + * expressions. + */ + if (build_expressions_only && (numcols == 0)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_OBJECT_DEFINITION), + errmsg("extended expression statistics require at least 1 column"))); + else if (!build_expressions_only && (numcols < 2)) ereport(ERROR, (errcode(ERRCODE_INVALID_OBJECT_DEFINITION), errmsg("extended statistics require at least 2 columns"))); @@ -258,13 +391,13 @@ CreateStatistics(CreateStatsStmt *stmt) * it does not hurt (it does not affect the efficiency, unlike for * indexes, for example). */ - qsort(attnums, numcols, sizeof(int16), compare_int16); + qsort(attnums, nattnums, sizeof(int16), compare_int16); /* * Check for duplicates in the list of columns. The attnums are sorted so * just check consecutive elements. */ - for (i = 1; i < numcols; i++) + for (i = 1; i < nattnums; i++) { if (attnums[i] == attnums[i - 1]) ereport(ERROR, @@ -272,48 +405,36 @@ CreateStatistics(CreateStatsStmt *stmt) errmsg("duplicate column name in statistics definition"))); } - /* Form an int2vector representation of the sorted column list */ - stxkeys = buildint2vector(attnums, numcols); - /* - * Parse the statistics kinds. + * Check for duplicate expressions. We do two loops, counting the + * occurrences of each expression. This is O(N^2) but we only allow + * small number of expressions and it's not executed often. */ - build_ndistinct = false; - build_dependencies = false; - build_mcv = false; - foreach(cell, stmt->stat_types) + foreach (cell, stxexprs) { - char *type = strVal((Value *) lfirst(cell)); + Node *expr1 = (Node *) lfirst(cell); + int cnt = 0; - if (strcmp(type, "ndistinct") == 0) - { - build_ndistinct = true; - requested_type = true; - } - else if (strcmp(type, "dependencies") == 0) - { - build_dependencies = true; - requested_type = true; - } - else if (strcmp(type, "mcv") == 0) + foreach (cell2, stxexprs) { - build_mcv = true; - requested_type = true; + Node *expr2 = (Node *) lfirst(cell2); + + if (equal(expr1, expr2)) + cnt += 1; } - else + + /* every expression should find at least itself */ + Assert(cnt >= 1); + + if (cnt > 1) ereport(ERROR, - (errcode(ERRCODE_SYNTAX_ERROR), - errmsg("unrecognized statistics kind \"%s\"", - type))); - } - /* If no statistic type was specified, build them all. */ - if (!requested_type) - { - build_ndistinct = true; - build_dependencies = true; - build_mcv = true; + (errcode(ERRCODE_DUPLICATE_COLUMN), + errmsg("duplicate expression in statistics definition"))); } + /* Form an int2vector representation of the sorted column list */ + stxkeys = buildint2vector(attnums, nattnums); + /* construct the char array of enabled statistic types */ ntypes = 0; if (build_ndistinct) @@ -322,9 +443,23 @@ CreateStatistics(CreateStatsStmt *stmt) types[ntypes++] = CharGetDatum(STATS_EXT_DEPENDENCIES); if (build_mcv) types[ntypes++] = CharGetDatum(STATS_EXT_MCV); + if (build_expressions) + types[ntypes++] = CharGetDatum(STATS_EXT_EXPRESSIONS); Assert(ntypes > 0 && ntypes <= lengthof(types)); stxkind = construct_array(types, ntypes, CHAROID, 1, true, TYPALIGN_CHAR); + /* convert the expressions (if any) to a text datum */ + if (stxexprs != NIL) + { + char *exprsString; + + exprsString = nodeToString(stxexprs); + exprsDatum = CStringGetTextDatum(exprsString); + pfree(exprsString); + } + else + exprsDatum = (Datum) 0; + statrel = table_open(StatisticExtRelationId, RowExclusiveLock); /* @@ -344,6 +479,10 @@ CreateStatistics(CreateStatsStmt *stmt) values[Anum_pg_statistic_ext_stxkeys - 1] = PointerGetDatum(stxkeys); values[Anum_pg_statistic_ext_stxkind - 1] = PointerGetDatum(stxkind); + values[Anum_pg_statistic_ext_stxexprs - 1] = exprsDatum; + if (exprsDatum == (Datum) 0) + nulls[Anum_pg_statistic_ext_stxexprs - 1] = true; + /* insert it into pg_statistic_ext */ htup = heap_form_tuple(statrel->rd_att, values, nulls); CatalogTupleInsert(statrel, htup); @@ -366,6 +505,7 @@ CreateStatistics(CreateStatsStmt *stmt) datanulls[Anum_pg_statistic_ext_data_stxdndistinct - 1] = true; datanulls[Anum_pg_statistic_ext_data_stxddependencies - 1] = true; datanulls[Anum_pg_statistic_ext_data_stxdmcv - 1] = true; + datanulls[Anum_pg_statistic_ext_data_stxdexpr - 1] = true; /* insert it into pg_statistic_ext_data */ htup = heap_form_tuple(datarel->rd_att, datavalues, datanulls); @@ -389,12 +529,39 @@ CreateStatistics(CreateStatsStmt *stmt) */ ObjectAddressSet(myself, StatisticExtRelationId, statoid); - for (i = 0; i < numcols; i++) + /* add dependencies for plain column references */ + for (i = 0; i < nattnums; i++) { ObjectAddressSubSet(parentobject, RelationRelationId, relid, attnums[i]); recordDependencyOn(&myself, &parentobject, DEPENDENCY_AUTO); } + /* + * If there are no simply-referenced columns, give the statistics an + * auto dependency on the whole table. In most cases, this will + * be redundant, but it might not be if the statistics expressions + * contain no Vars (which might seem strange but possible). + * + * XXX This is copied from index_create, not sure if it's applicable + * to extended statistics too. + */ + if (!nattnums) + { + ObjectAddressSet(parentobject, RelationRelationId, relid); + recordDependencyOn(&myself, &parentobject, DEPENDENCY_AUTO); + } + + /* + * Store dependencies on anything mentioned in statistics expressions, + * just like we do for index expressions. + */ + if (stxexprs) + recordDependencyOnSingleRelExpr(&myself, + (Node *) stxexprs, + relid, + DEPENDENCY_NORMAL, + DEPENDENCY_AUTO, false, true); + /* * Also add dependencies on namespace and owner. These are required * because the stats object might have a different namespace and/or owner @@ -638,6 +805,7 @@ UpdateStatisticsForTypeChange(Oid statsOid, Oid relationOid, int attnum, replaces[Anum_pg_statistic_ext_data_stxdmcv - 1] = true; nulls[Anum_pg_statistic_ext_data_stxdmcv - 1] = true; + nulls[Anum_pg_statistic_ext_data_stxdexpr - 1] = true; rel = table_open(StatisticExtDataRelationId, RowExclusiveLock); @@ -724,18 +892,26 @@ ChooseExtendedStatisticNameAddition(List *exprs) buf[0] = '\0'; foreach(lc, exprs) { - ColumnRef *cref = (ColumnRef *) lfirst(lc); + StatsElem *selem = (StatsElem *) lfirst(lc); const char *name; /* It should be one of these, but just skip if it happens not to be */ - if (!IsA(cref, ColumnRef)) + if (!IsA(selem, StatsElem)) continue; - name = strVal((Value *) linitial(cref->fields)); + name = selem->name; if (buflen > 0) buf[buflen++] = '_'; /* insert _ between names */ + /* + * FIXME use 'expr' for expressions, which have empty column names. + * For indexes this is handled in ChooseIndexColumnNames, but we + * have no such function for stats. + */ + if (!name) + name = "expr"; + /* * At this point we have buflen <= NAMEDATALEN. name should be less * than NAMEDATALEN already, but use strlcpy for paranoia. @@ -747,3 +923,31 @@ ChooseExtendedStatisticNameAddition(List *exprs) } return pstrdup(buf); } + +/* + * CheckMutability + * Test whether given expression is mutable + * + * FIXME copied from indexcmds.c, maybe use some shared function? + */ +static bool +CheckMutability(Expr *expr) +{ + /* + * First run the expression through the planner. This has a couple of + * important consequences. First, function default arguments will get + * inserted, which may affect volatility (consider "default now()"). + * Second, inline-able functions will get inlined, which may allow us to + * conclude that the function is really less volatile than it's marked. As + * an example, polymorphic functions must be marked with the most volatile + * behavior that they have for any input type, but once we inline the + * function we may be able to conclude that it's not so volatile for the + * particular input type we're dealing with. + * + * We assume here that expression_planner() won't scribble on its input. + */ + expr = expression_planner(expr); + + /* Now we can search for non-immutable functions */ + return contain_mutable_functions((Node *) expr); +} diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 910906f639..befcc104cf 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2924,6 +2924,17 @@ _copyIndexElem(const IndexElem *from) return newnode; } +static StatsElem * +_copyStatsElem(const StatsElem *from) +{ + StatsElem *newnode = makeNode(StatsElem); + + COPY_STRING_FIELD(name); + COPY_NODE_FIELD(expr); + + return newnode; +} + static ColumnDef * _copyColumnDef(const ColumnDef *from) { @@ -5618,6 +5629,9 @@ copyObjectImpl(const void *from) case T_IndexElem: retval = _copyIndexElem(from); break; + case T_StatsElem: + retval = _copyStatsElem(from); + break; case T_ColumnDef: retval = _copyColumnDef(from); break; diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index 687609f59e..d6daaae6e2 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2580,6 +2580,16 @@ _equalIndexElem(const IndexElem *a, const IndexElem *b) return true; } + +static bool +_equalStatsElem(const StatsElem *a, const StatsElem *b) +{ + COMPARE_STRING_FIELD(name); + COMPARE_NODE_FIELD(expr); + + return true; +} + static bool _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { @@ -3673,6 +3683,9 @@ equal(const void *a, const void *b) case T_IndexElem: retval = _equalIndexElem(a, b); break; + case T_StatsElem: + retval = _equalStatsElem(a, b); + break; case T_ColumnDef: retval = _equalColumnDef(a, b); break; diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 9c73c605a4..6ba29a8931 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2919,6 +2919,15 @@ _outIndexElem(StringInfo str, const IndexElem *node) WRITE_ENUM_FIELD(nulls_ordering, SortByNulls); } +static void +_outStatsElem(StringInfo str, const StatsElem *node) +{ + WRITE_NODE_TYPE("STATSELEM"); + + WRITE_STRING_FIELD(name); + WRITE_NODE_FIELD(expr); +} + static void _outQuery(StringInfo str, const Query *node) { @@ -4228,6 +4237,9 @@ outNode(StringInfo str, const void *obj) case T_IndexElem: _outIndexElem(str, obj); break; + case T_StatsElem: + _outStatsElem(str, obj); + break; case T_Query: _outQuery(str, obj); break; diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index 3e94256d34..010cd8d26c 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -35,6 +35,7 @@ #include "foreign/fdwapi.h" #include "miscadmin.h" #include "nodes/makefuncs.h" +#include "nodes/nodeFuncs.h" #include "nodes/supportnodes.h" #include "optimizer/clauses.h" #include "optimizer/cost.h" @@ -1317,6 +1318,7 @@ get_relation_statistics(RelOptInfo *rel, Relation relation) HeapTuple dtup; Bitmapset *keys = NULL; int i; + List *exprs = NIL; htup = SearchSysCache1(STATEXTOID, ObjectIdGetDatum(statOid)); if (!HeapTupleIsValid(htup)) @@ -1335,6 +1337,41 @@ get_relation_statistics(RelOptInfo *rel, Relation relation) for (i = 0; i < staForm->stxkeys.dim1; i++) keys = bms_add_member(keys, staForm->stxkeys.values[i]); + /* + * preprocess expression (if any) + * + * FIXME Should we cache the result somewhere? + */ + { + bool isnull; + Datum datum; + + /* decode expression (if any) */ + datum = SysCacheGetAttr(STATEXTOID, htup, + Anum_pg_statistic_ext_stxexprs, &isnull); + + if (!isnull) + { + char *exprsString; + + exprsString = TextDatumGetCString(datum); + exprs = (List *) stringToNode(exprsString); + pfree(exprsString); + + /* + * Run the expressions through eval_const_expressions. This is not just an + * optimization, but is necessary, because the planner will be comparing + * them to similarly-processed qual clauses, and may fail to detect valid + * matches without this. We must not use canonicalize_qual, however, + * since these aren't qual expressions. + */ + exprs = (List *) eval_const_expressions(NULL, (Node *) exprs); + + /* May as well fix opfuncids too */ + fix_opfuncids((Node *) exprs); + } + } + /* add one StatisticExtInfo for each kind built */ if (statext_is_kind_built(dtup, STATS_EXT_NDISTINCT)) { @@ -1344,6 +1381,7 @@ get_relation_statistics(RelOptInfo *rel, Relation relation) info->rel = rel; info->kind = STATS_EXT_NDISTINCT; info->keys = bms_copy(keys); + info->exprs = exprs; stainfos = lappend(stainfos, info); } @@ -1356,6 +1394,7 @@ get_relation_statistics(RelOptInfo *rel, Relation relation) info->rel = rel; info->kind = STATS_EXT_DEPENDENCIES; info->keys = bms_copy(keys); + info->exprs = exprs; stainfos = lappend(stainfos, info); } @@ -1368,6 +1407,20 @@ get_relation_statistics(RelOptInfo *rel, Relation relation) info->rel = rel; info->kind = STATS_EXT_MCV; info->keys = bms_copy(keys); + info->exprs = exprs; + + stainfos = lappend(stainfos, info); + } + + if (statext_is_kind_built(dtup, STATS_EXT_EXPRESSIONS)) + { + StatisticExtInfo *info = makeNode(StatisticExtInfo); + + info->statOid = statOid; + info->rel = rel; + info->kind = STATS_EXT_EXPRESSIONS; + info->keys = bms_copy(keys); + info->exprs = exprs; stainfos = lappend(stainfos, info); } diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 469de52bc2..fe1f192010 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -233,6 +233,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); WindowDef *windef; JoinExpr *jexpr; IndexElem *ielem; + StatsElem *selem; Alias *alias; RangeVar *range; IntoClause *into; @@ -396,7 +397,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); old_aggr_definition old_aggr_list oper_argtypes RuleActionList RuleActionMulti opt_column_list columnList opt_name_list - sort_clause opt_sort_clause sortby_list index_params + sort_clause opt_sort_clause sortby_list index_params stats_params opt_include opt_c_include index_including_params name_list role_list from_clause from_list opt_array_bounds qualified_name_list any_name any_name_list type_name_list @@ -502,6 +503,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> func_alias_clause %type <sortby> sortby %type <ielem> index_elem index_elem_options +%type <selem> stats_param %type <node> table_ref %type <jexpr> joined_table %type <range> relation_expr @@ -4003,7 +4005,7 @@ ExistingIndex: USING INDEX name { $$ = $3; } CreateStatsStmt: CREATE STATISTICS any_name - opt_name_list ON expr_list FROM from_list + opt_name_list ON stats_params FROM from_list { CreateStatsStmt *n = makeNode(CreateStatsStmt); n->defnames = $3; @@ -4015,7 +4017,7 @@ CreateStatsStmt: $$ = (Node *)n; } | CREATE STATISTICS IF_P NOT EXISTS any_name - opt_name_list ON expr_list FROM from_list + opt_name_list ON stats_params FROM from_list { CreateStatsStmt *n = makeNode(CreateStatsStmt); n->defnames = $6; @@ -4028,6 +4030,29 @@ CreateStatsStmt: } ; +stats_params: stats_param { $$ = list_make1($1); } + | stats_params ',' stats_param { $$ = lappend($1, $3); } + ; + +stats_param: ColId + { + $$ = makeNode(StatsElem); + $$->name = $1; + $$->expr = NULL; + } + | func_expr_windowless + { + $$ = makeNode(StatsElem); + $$->name = NULL; + $$->expr = $1; + } + | '(' a_expr ')' + { + $$ = makeNode(StatsElem); + $$->name = NULL; + $$->expr = $2; + } + ; /***************************************************************************** * diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 783f3fe8f2..12b9e855d5 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -484,6 +484,13 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) else err = _("grouping operations are not allowed in index predicates"); + break; + case EXPR_KIND_STATS_EXPRESSION: + if (isAgg) + err = _("aggregate functions are not allowed in statistics expressions"); + else + err = _("grouping operations are not allowed in statistics expressions"); + break; case EXPR_KIND_ALTER_COL_TRANSFORM: if (isAgg) @@ -906,6 +913,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_INDEX_EXPRESSION: err = _("window functions are not allowed in index expressions"); break; + case EXPR_KIND_STATS_EXPRESSION: + err = _("window functions are not allowed in stats expressions"); + break; case EXPR_KIND_INDEX_PREDICATE: err = _("window functions are not allowed in index predicates"); break; diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 36002f059d..57ba583f74 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -560,6 +560,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_FUNCTION_DEFAULT: case EXPR_KIND_INDEX_EXPRESSION: case EXPR_KIND_INDEX_PREDICATE: + case EXPR_KIND_STATS_EXPRESSION: case EXPR_KIND_ALTER_COL_TRANSFORM: case EXPR_KIND_EXECUTE_PARAMETER: case EXPR_KIND_TRIGGER_WHEN: @@ -1865,6 +1866,9 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_INDEX_PREDICATE: err = _("cannot use subquery in index predicate"); break; + case EXPR_KIND_STATS_EXPRESSION: + err = _("cannot use subquery in statistics expression"); + break; case EXPR_KIND_ALTER_COL_TRANSFORM: err = _("cannot use subquery in transform expression"); break; @@ -3472,6 +3476,8 @@ ParseExprKindName(ParseExprKind exprKind) return "index expression"; case EXPR_KIND_INDEX_PREDICATE: return "index predicate"; + case EXPR_KIND_STATS_EXPRESSION: + return "statistics expression"; case EXPR_KIND_ALTER_COL_TRANSFORM: return "USING"; case EXPR_KIND_EXECUTE_PARAMETER: diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index 23ac2a2fe6..e590e659ad 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2503,6 +2503,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_INDEX_PREDICATE: err = _("set-returning functions are not allowed in index predicates"); break; + case EXPR_KIND_STATS_EXPRESSION: + err = _("set-returning functions are not allowed in stats expressions"); + break; case EXPR_KIND_ALTER_COL_TRANSFORM: err = _("set-returning functions are not allowed in transform expressions"); break; diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 89ee990599..b03b958b14 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -1898,6 +1898,8 @@ generateClonedExtStatsStmt(RangeVar *heapRel, Oid heapRelid, stat_types = lappend(stat_types, makeString("dependencies")); else if (enabled[i] == STATS_EXT_MCV) stat_types = lappend(stat_types, makeString("mcv")); + else if (enabled[i] == STATS_EXT_EXPRESSIONS) + stat_types = lappend(stat_types, makeString("expressions")); else elog(ERROR, "unrecognized statistics kind %c", enabled[i]); } @@ -1905,14 +1907,43 @@ generateClonedExtStatsStmt(RangeVar *heapRel, Oid heapRelid, /* Determine which columns the statistics are on */ for (i = 0; i < statsrec->stxkeys.dim1; i++) { - ColumnRef *cref = makeNode(ColumnRef); + StatsElem *selem = makeNode(StatsElem); AttrNumber attnum = statsrec->stxkeys.values[i]; - cref->fields = list_make1(makeString(get_attname(heapRelid, - attnum, false))); - cref->location = -1; + selem->name = get_attname(heapRelid, attnum, false); + selem->expr = NULL; - def_names = lappend(def_names, cref); + def_names = lappend(def_names, selem); + } + + /* + * Now handle expressions, if there are any. The order does not + * matter for extended stats, so we simply append them after + * simple column references. + */ + datum = SysCacheGetAttr(STATEXTOID, ht_stats, + Anum_pg_statistic_ext_stxexprs, &isnull); + + if (!isnull) + { + ListCell *lc; + List *exprs = NIL; + char *exprsString; + + exprsString = TextDatumGetCString(datum); + exprs = (List *) stringToNode(exprsString); + + foreach(lc, exprs) + { + StatsElem *selem = makeNode(StatsElem); + + selem->name = NULL; + selem->expr = (Node *) lfirst(lc); + + def_names = lappend(def_names, selem); + } + + pfree(exprsString); } /* finally, build the output node */ @@ -1923,6 +1954,7 @@ generateClonedExtStatsStmt(RangeVar *heapRel, Oid heapRelid, stats->relations = list_make1(heapRel); stats->stxcomment = NULL; stats->if_not_exists = false; + stats->transformed = true; /* don't need transformStatsStmt */ /* Clean up */ ReleaseSysCache(ht_stats); @@ -2847,6 +2879,84 @@ transformIndexStmt(Oid relid, IndexStmt *stmt, const char *queryString) return stmt; } +/* + * transformStatsStmt - parse analysis for CREATE STATISTICS + * + * To avoid race conditions, it's important that this function rely only on + * the passed-in relid (and not on stmt->relation) to determine the target + * relation. + */ +CreateStatsStmt * +transformStatsStmt(Oid relid, CreateStatsStmt *stmt, const char *queryString) +{ + ParseState *pstate; + ParseNamespaceItem *nsitem; + ListCell *l; + Relation rel; + + /* Nothing to do if statement already transformed. */ + if (stmt->transformed) + return stmt; + + /* + * We must not scribble on the passed-in CreateStatsStmt, so copy it. (This is + * overkill, but easy.) + */ + stmt = copyObject(stmt); + + /* Set up pstate */ + pstate = make_parsestate(NULL); + pstate->p_sourcetext = queryString; + + /* + * Put the parent table into the rtable so that the expressions can refer + * to its fields without qualification. Caller is responsible for locking + * relation, but we still need to open it. + */ + rel = relation_open(relid, NoLock); + nsitem = addRangeTableEntryForRelation(pstate, rel, + AccessShareLock, + NULL, false, true); + + /* no to join list, yes to namespaces */ + addNSItemToQuery(pstate, nsitem, false, true, true); + + /* take care of any expressions */ + foreach(l, stmt->exprs) + { + StatsElem *selem = (StatsElem *) lfirst(l); + + if (selem->expr) + { + /* Now do parse transformation of the expression */ + selem->expr = transformExpr(pstate, selem->expr, + EXPR_KIND_STATS_EXPRESSION); + + /* We have to fix its collations too */ + assign_expr_collations(pstate, selem->expr); + } + } + + /* + * Check that only the base rel is mentioned. (This should be dead code + * now that add_missing_from is history.) + */ + if (list_length(pstate->p_rtable) != 1) + ereport(ERROR, + (errcode(ERRCODE_INVALID_COLUMN_REFERENCE), + errmsg("statistics expressions and predicates can refer only to the table being indexed"))); + + free_parsestate(pstate); + + /* Close relation */ + table_close(rel, NoLock); + + /* Mark statement as successfully transformed */ + stmt->transformed = true; + + return stmt; +} + /* * transformRuleStmt - diff --git a/src/backend/statistics/dependencies.c b/src/backend/statistics/dependencies.c index b1abcde968..04661e7628 100644 --- a/src/backend/statistics/dependencies.c +++ b/src/backend/statistics/dependencies.c @@ -70,15 +70,18 @@ static void generate_dependencies(DependencyGenerator state); static DependencyGenerator DependencyGenerator_init(int n, int k); static void DependencyGenerator_free(DependencyGenerator state); static AttrNumber *DependencyGenerator_next(DependencyGenerator state); -static double dependency_degree(int numrows, HeapTuple *rows, int k, - AttrNumber *dependency, VacAttrStats **stats, Bitmapset *attrs); +static double dependency_degree(int numrows, HeapTuple *rows, + ExprInfo *exprs, int k, + AttrNumber *dependency, VacAttrStats **stats, + Bitmapset *attrs); static bool dependency_is_fully_matched(MVDependency *dependency, Bitmapset *attnums); static bool dependency_is_compatible_clause(Node *clause, Index relid, AttrNumber *attnum); +static bool dependency_is_compatible_expression(Node *clause, Index relid, + List *statlist, Node **expr); static MVDependency *find_strongest_dependency(MVDependencies **dependencies, - int ndependencies, - Bitmapset *attnums); + int ndependencies, Bitmapset *attnums); static Selectivity clauselist_apply_dependencies(PlannerInfo *root, List *clauses, int varRelid, JoinType jointype, SpecialJoinInfo *sjinfo, @@ -219,8 +222,9 @@ DependencyGenerator_next(DependencyGenerator state) * the last one. */ static double -dependency_degree(int numrows, HeapTuple *rows, int k, AttrNumber *dependency, - VacAttrStats **stats, Bitmapset *attrs) +dependency_degree(int numrows, HeapTuple *rows, ExprInfo *exprs, int k, + AttrNumber *dependency, VacAttrStats **stats, + Bitmapset *attrs) { int i, nitems; @@ -289,8 +293,8 @@ dependency_degree(int numrows, HeapTuple *rows, int k, AttrNumber *dependency, * descriptor. For now that assumption holds, but it might change in the * future for example if we support statistics on multiple tables. */ - items = build_sorted_items(numrows, &nitems, rows, stats[0]->tupDesc, - mss, k, attnums_dep); + items = build_sorted_items(numrows, &nitems, rows, exprs, + stats[0]->tupDesc, mss, k, attnums_dep); /* * Walk through the sorted array, split it into rows according to the @@ -360,7 +364,8 @@ dependency_degree(int numrows, HeapTuple *rows, int k, AttrNumber *dependency, * (c) -> b */ MVDependencies * -statext_dependencies_build(int numrows, HeapTuple *rows, Bitmapset *attrs, +statext_dependencies_build(int numrows, HeapTuple *rows, + ExprInfo *exprs, Bitmapset *attrs, VacAttrStats **stats) { int i, @@ -371,6 +376,9 @@ statext_dependencies_build(int numrows, HeapTuple *rows, Bitmapset *attrs, /* result */ MVDependencies *dependencies = NULL; + /* treat expressions as special attributes with high attnums */ + attrs = add_expressions_to_attributes(attrs, exprs->nexprs); + /* * Transform the bms into an array, to make accessing i-th member easier. */ @@ -398,7 +406,8 @@ statext_dependencies_build(int numrows, HeapTuple *rows, Bitmapset *attrs, MVDependency *d; /* compute how valid the dependency seems */ - degree = dependency_degree(numrows, rows, k, dependency, stats, attrs); + degree = dependency_degree(numrows, rows, exprs, k, dependency, + stats, attrs); /* * if the dependency seems entirely invalid, don't store it @@ -441,6 +450,8 @@ statext_dependencies_build(int numrows, HeapTuple *rows, Bitmapset *attrs, DependencyGenerator_free(DependencyGenerator); } + pfree(attrs); + return dependencies; } @@ -603,6 +614,7 @@ static bool dependency_is_fully_matched(MVDependency *dependency, Bitmapset *attnums) { int j; + bool result = true; /* match by default */ /* * Check that the dependency actually is fully covered by clauses. We have @@ -613,10 +625,13 @@ dependency_is_fully_matched(MVDependency *dependency, Bitmapset *attnums) int attnum = dependency->attributes[j]; if (!bms_is_member(attnum, attnums)) - return false; + { + result = false; + break; + } } - return true; + return result; } /* @@ -927,8 +942,8 @@ dependency_is_compatible_clause(Node *clause, Index relid, AttrNumber *attnum) * (see the comment in dependencies_clauselist_selectivity). */ static MVDependency * -find_strongest_dependency(MVDependencies **dependencies, int ndependencies, - Bitmapset *attnums) +find_strongest_dependency(MVDependencies **dependencies, + int ndependencies, Bitmapset *attnums) { int i, j; @@ -1157,6 +1172,131 @@ clauselist_apply_dependencies(PlannerInfo *root, List *clauses, return s1; } +/* + * Similar to dependency_is_compatible_clause, but don't enforce that the + * expression is a simple Var. OTOH we check that there's at least one + * statistics matching the expression. + */ +static bool +dependency_is_compatible_expression(Node *clause, Index relid, List *statlist, Node **expr) +{ + List *vars; + ListCell *lc, *lc2; + + RestrictInfo *rinfo = (RestrictInfo *) clause; + Node *clause_expr; + + if (!IsA(rinfo, RestrictInfo)) + return false; + + /* Pseudoconstants are not interesting (they couldn't contain a Var) */ + if (rinfo->pseudoconstant) + return false; + + /* Clauses referencing multiple, or no, varnos are incompatible */ + if (bms_membership(rinfo->clause_relids) != BMS_SINGLETON) + return false; + + if (is_opclause(rinfo->clause)) + { + /* If it's an opclause, check for Var = Const or Const = Var. */ + OpExpr *expr = (OpExpr *) rinfo->clause; + + /* Only expressions with two arguments are candidates. */ + if (list_length(expr->args) != 2) + return false; + + /* Make sure non-selected argument is a pseudoconstant. */ + if (is_pseudo_constant_clause(lsecond(expr->args))) + clause_expr = linitial(expr->args); + else if (is_pseudo_constant_clause(linitial(expr->args))) + clause_expr = lsecond(expr->args); + else + return false; + + /* + * If it's not an "=" operator, just ignore the clause, as it's not + * compatible with functional dependencies. + * + * This uses the function for estimating selectivity, not the operator + * directly (a bit awkward, but well ...). + * + * XXX this is pretty dubious; probably it'd be better to check btree + * or hash opclass membership, so as not to be fooled by custom + * selectivity functions, and to be more consistent with decisions + * elsewhere in the planner. + */ + if (get_oprrest(expr->opno) != F_EQSEL) + return false; + + /* OK to proceed with checking "var" */ + } + else if (is_notclause(rinfo->clause)) + { + /* + * "NOT x" can be interpreted as "x = false", so get the argument and + * proceed with seeing if it's a suitable Var. + */ + clause_expr = (Node *) get_notclausearg(rinfo->clause); + } + else + { + /* + * A boolean expression "x" can be interpreted as "x = true", so + * proceed with seeing if it's a suitable Var. + */ + clause_expr = (Node *) rinfo->clause; + } + + /* + * We may ignore any RelabelType node above the operand. (There won't be + * more than one, since eval_const_expressions has been applied already.) + */ + if (IsA(clause_expr, RelabelType)) + clause_expr = (Node *) ((RelabelType *) clause_expr)->arg; + + vars = pull_var_clause(clause_expr, 0); + + foreach (lc, vars) + { + Var *var = (Var *) lfirst(lc); + + /* Ensure Var is from the correct relation */ + if (var->varno != relid) + return false; + + /* We also better ensure the Var is from the current level */ + if (var->varlevelsup != 0) + return false; + + /* Also ignore system attributes (we don't allow stats on those) */ + if (!AttrNumberIsForUserDefinedAttr(var->varattno)) + return false; + } + + foreach (lc, statlist) + { + StatisticExtInfo *info = (StatisticExtInfo *) lfirst(lc); + + /* ignore stats without dependencies */ + if (info->kind != STATS_EXT_DEPENDENCIES) + continue; + + foreach (lc2, info->exprs) + { + Node *stat_expr = (Node *) lfirst(lc2); + + if (equal(clause_expr, stat_expr)) + { + *expr = stat_expr; + return true; + } + } + } + + return false; +} + /* * dependencies_clauselist_selectivity * Return the estimated selectivity of (a subset of) the given clauses @@ -1205,6 +1345,10 @@ dependencies_clauselist_selectivity(PlannerInfo *root, int ndependencies; int i; + /* unique expressions */ + Node **unique_exprs; + int unique_exprs_cnt; + /* check if there's any stats that might be useful for us. */ if (!has_stats_of_kind(rel->statlist, STATS_EXT_DEPENDENCIES)) return 1.0; @@ -1212,6 +1356,10 @@ dependencies_clauselist_selectivity(PlannerInfo *root, list_attnums = (AttrNumber *) palloc(sizeof(AttrNumber) * list_length(clauses)); + /* unique expressions */ + unique_exprs = (Node **) palloc(sizeof(Node *) * list_length(clauses)); + unique_exprs_cnt = 0; + /* * Pre-process the clauses list to extract the attnums seen in each item. * We need to determine if there's any clauses which will be useful for @@ -1222,29 +1370,70 @@ dependencies_clauselist_selectivity(PlannerInfo *root, * * We also skip clauses that we already estimated using different types of * statistics (we treat them as incompatible). + * + * For expressions, we generate attnums higher than MaxHeapAttributeNumber + * so that we can work with attnums only. */ listidx = 0; foreach(l, clauses) { Node *clause = (Node *) lfirst(l); AttrNumber attnum; + Node *expr = NULL; + + /* ignore clause by default */ + list_attnums[listidx] = InvalidAttrNumber; - if (!bms_is_member(listidx, *estimatedclauses) && - dependency_is_compatible_clause(clause, rel->relid, &attnum)) + if (!bms_is_member(listidx, *estimatedclauses)) { - list_attnums[listidx] = attnum; - clauses_attnums = bms_add_member(clauses_attnums, attnum); + if (dependency_is_compatible_clause(clause, rel->relid, &attnum)) + { + list_attnums[listidx] = attnum; + clauses_attnums = bms_add_member(clauses_attnums, attnum); + } + else if (dependency_is_compatible_expression(clause, rel->relid, + rel->statlist, + &expr)) + { + /* special attnum assigned to this expression */ + attnum = InvalidAttrNumber; + + Assert(expr != NULL); + + /* build list of unique expressions, for re-mapping later */ + for (i = 0; i < unique_exprs_cnt; i++) + { + if (equal(unique_exprs[i], expr)) + { + attnum = (i + 1); + break; + } + } + + /* not found in the list, so add it */ + if (attnum == InvalidAttrNumber) + { + attnum = EXPRESSION_ATTNUM(unique_exprs_cnt); + unique_exprs[unique_exprs_cnt++] = expr; + + /* shouldn't have seen this attnum yet */ + Assert(!bms_is_member(attnum, clauses_attnums)); + } + + /* we may add the attnum repeatedly to clauses_attnums */ + clauses_attnums = bms_add_member(clauses_attnums, attnum); + + list_attnums[listidx] = attnum; + } } - else - list_attnums[listidx] = InvalidAttrNumber; listidx++; } /* - * If there's not at least two distinct attnums then reject the whole list - * of clauses. We must return 1.0 so the calling function's selectivity is - * unaffected. + * If there's not at least two distinct attnums and expressions, then + * reject the whole list of clauses. We must return 1.0 so the calling + * function's selectivity is unaffected. */ if (bms_membership(clauses_attnums) != BMS_MULTIPLE) { @@ -1273,25 +1462,138 @@ dependencies_clauselist_selectivity(PlannerInfo *root, { StatisticExtInfo *stat = (StatisticExtInfo *) lfirst(l); Bitmapset *matched; - BMS_Membership membership; + int nmatched; + int nexprs; + MVDependencies *deps; /* skip statistics that are not of the correct type */ if (stat->kind != STATS_EXT_DEPENDENCIES) continue; + /* count matching simple clauses */ matched = bms_intersect(clauses_attnums, stat->keys); - membership = bms_membership(matched); + nmatched = bms_num_members(matched); bms_free(matched); - /* skip objects matching fewer than two attributes from clauses */ - if (membership != BMS_MULTIPLE) + /* count matching expressions */ + nexprs = 0; + for (i = 0; i < unique_exprs_cnt; i++) + { + ListCell *lc; + + foreach (lc, stat->exprs) + { + Node *stat_expr = (Node *) lfirst(lc); + + /* try to match it */ + if (equal(stat_expr, unique_exprs[i])) + nexprs++; + } + } + + /* + * Skip objects matching fewer than two attributes/expressions + * from clauses. + */ + if (nmatched + nexprs < 2) continue; - func_dependencies[nfunc_dependencies] - = statext_dependencies_load(stat->statOid); + deps = statext_dependencies_load(stat->statOid); - total_ndeps += func_dependencies[nfunc_dependencies]->ndeps; - nfunc_dependencies++; + /* + * The expressions may be represented by different attnums in the + * stats, we need to remap them to be consistent with the clauses. + * That will make the later steps (e.g. picking the strongest item + * and so on) much simpler. + * + * When we're at it, we can also remove dependencies referencing + * missing clauses (i.e. expressions that are not in the clauses). + * + * XXX We might also skip clauses referencing missing attnums, not + * just expressions. + */ + if (stat->exprs) + { + int ndeps = 0; + + for (i = 0; i < deps->ndeps; i++) + { + bool skip = false; + MVDependency *dep = deps->deps[i]; + int j; + + for (j = 0; j < dep->nattributes; j++) + { + int idx; + Node *expr; + int k; + AttrNumber unique_attnum = InvalidAttrNumber; + + /* regular attribute, no need to remap */ + if (dep->attributes[j] <= MaxHeapAttributeNumber) + continue; + + /* index of the expression */ + idx = EXPRESSION_INDEX(dep->attributes[j]); + + /* make sure the expression index is valid */ + Assert((idx >= 0) && (idx < list_length(stat->exprs))); + + expr = (Node *) list_nth(stat->exprs, idx); + + /* try to find the expression in the unique list */ + for (k = 0; k < unique_exprs_cnt; k++) + { + /* + * found a matching unique expression, use the attnum + * (derived from index of the unique expression) + */ + if (equal(unique_exprs[k], expr)) + { + unique_attnum = EXPRESSION_ATTNUM(k); + break; + } + } + + /* + * Not found a matching expression, so we can simply + * skip this dependency, because there's no chance it + * will be fully covered. + */ + if (unique_attnum == InvalidAttrNumber) + { + skip = true; + break; + } + + /* otherwise remap it to the new attnum */ + dep->attributes[j] = unique_attnum; + } + + /* if found a matching, */ + if (!skip) + { + /* maybe we've skipped something earlier, so move it */ + if (ndeps != i) + deps->deps[ndeps] = deps->deps[i]; + + ndeps++; + } + } + + deps->ndeps = ndeps; + } + + /* + * It's possible we've removed all dependencies, in which case we + * don't bother adding it to the list. + */ + if (deps->ndeps > 0) + { + func_dependencies[nfunc_dependencies] = deps; + total_ndeps += deps->ndeps; + nfunc_dependencies++; + } } /* if no matching stats could be found then we've nothing to do */ @@ -1300,6 +1602,7 @@ dependencies_clauselist_selectivity(PlannerInfo *root, pfree(func_dependencies); bms_free(clauses_attnums); pfree(list_attnums); + pfree(unique_exprs); return 1.0; } @@ -1347,6 +1650,7 @@ dependencies_clauselist_selectivity(PlannerInfo *root, pfree(func_dependencies); bms_free(clauses_attnums); pfree(list_attnums); + pfree(unique_exprs); return s1; } diff --git a/src/backend/statistics/extended_stats.c b/src/backend/statistics/extended_stats.c index 8d3cd091ad..4e07c6941f 100644 --- a/src/backend/statistics/extended_stats.c +++ b/src/backend/statistics/extended_stats.c @@ -24,6 +24,7 @@ #include "catalog/pg_collation.h" #include "catalog/pg_statistic_ext.h" #include "catalog/pg_statistic_ext_data.h" +#include "executor/executor.h" #include "commands/progress.h" #include "miscadmin.h" #include "nodes/nodeFuncs.h" @@ -35,6 +36,7 @@ #include "statistics/statistics.h" #include "utils/acl.h" #include "utils/array.h" +#include "utils/attoptcache.h" #include "utils/builtins.h" #include "utils/fmgroids.h" #include "utils/lsyscache.h" @@ -42,6 +44,7 @@ #include "utils/rel.h" #include "utils/selfuncs.h" #include "utils/syscache.h" +#include "utils/typcache.h" /* * To avoid consuming too much memory during analysis and/or too much space @@ -66,18 +69,35 @@ typedef struct StatExtEntry Bitmapset *columns; /* attribute numbers covered by the object */ List *types; /* 'char' list of enabled statistic kinds */ int stattarget; /* statistics target (-1 for default) */ + List *exprs; /* expressions */ } StatExtEntry; static List *fetch_statentries_for_relation(Relation pg_statext, Oid relid); -static VacAttrStats **lookup_var_attr_stats(Relation rel, Bitmapset *attrs, +static VacAttrStats **lookup_var_attr_stats(Relation rel, Bitmapset *attrs, List *exprs, int nvacatts, VacAttrStats **vacatts); -static void statext_store(Oid relid, +static void statext_store(Oid statOid, MVNDistinct *ndistinct, MVDependencies *dependencies, - MCVList *mcv, VacAttrStats **stats); + MCVList *mcv, Datum exprs, VacAttrStats **stats); static int statext_compute_stattarget(int stattarget, int natts, VacAttrStats **stats); +typedef struct AnlExprData +{ + Node *expr; /* expression to analyze */ + VacAttrStats *vacattrstat; /* index attrs to analyze */ +} AnlExprData; + +static void compute_expr_stats(Relation onerel, double totalrows, + AnlExprData *exprdata, int nexprs, + HeapTuple *rows, int numrows); +static Datum serialize_expr_stats(AnlExprData *exprdata, int nexprs); +static Datum expr_fetch_func(VacAttrStatsP stats, int rownum, bool *isNull); +static AnlExprData *build_expr_data(List *exprs); +static VacAttrStats *examine_expression(Node *expr); +static ExprInfo *evaluate_expressions(Relation rel, List *exprs, + int numrows, HeapTuple *rows); + /* * Compute requested extended stats, using the rows sampled for the plain * (single-column) stats. @@ -92,7 +112,7 @@ BuildRelationExtStatistics(Relation onerel, double totalrows, { Relation pg_stext; ListCell *lc; - List *stats; + List *statslist; MemoryContext cxt; MemoryContext oldcxt; int64 ext_cnt; @@ -103,10 +123,10 @@ BuildRelationExtStatistics(Relation onerel, double totalrows, oldcxt = MemoryContextSwitchTo(cxt); pg_stext = table_open(StatisticExtRelationId, RowExclusiveLock); - stats = fetch_statentries_for_relation(pg_stext, RelationGetRelid(onerel)); + statslist = fetch_statentries_for_relation(pg_stext, RelationGetRelid(onerel)); /* report this phase */ - if (stats != NIL) + if (statslist != NIL) { const int index[] = { PROGRESS_ANALYZE_PHASE, @@ -114,28 +134,31 @@ BuildRelationExtStatistics(Relation onerel, double totalrows, }; const int64 val[] = { PROGRESS_ANALYZE_PHASE_COMPUTE_EXT_STATS, - list_length(stats) + list_length(statslist) }; pgstat_progress_update_multi_param(2, index, val); } ext_cnt = 0; - foreach(lc, stats) + foreach(lc, statslist) { StatExtEntry *stat = (StatExtEntry *) lfirst(lc); MVNDistinct *ndistinct = NULL; MVDependencies *dependencies = NULL; MCVList *mcv = NULL; + Datum exprstats = (Datum) 0; VacAttrStats **stats; ListCell *lc2; int stattarget; + ExprInfo *exprs; + int min_attrs; /* * Check if we can build these stats based on the column analyzed. If * not, report this fact (except in autovacuum) and move on. */ - stats = lookup_var_attr_stats(onerel, stat->columns, + stats = lookup_var_attr_stats(onerel, stat->columns, stat->exprs, natts, vacattrstats); if (!stats) { @@ -150,9 +173,28 @@ BuildRelationExtStatistics(Relation onerel, double totalrows, continue; } + /* determine the minimum required number of attributes/expressions */ + min_attrs = 1; + foreach(lc2, stat->types) + { + char t = (char) lfirst_int(lc2); + + switch (t) + { + /* expressions only need a single item */ + case STATS_EXT_EXPRESSIONS: + break; + + /* all other statistics kinds require at least two */ + default: + min_attrs = 2; + break; + } + } + /* check allowed number of dimensions */ - Assert(bms_num_members(stat->columns) >= 2 && - bms_num_members(stat->columns) <= STATS_MAX_DIMENSIONS); + Assert(bms_num_members(stat->columns) + list_length(stat->exprs) >= min_attrs && + bms_num_members(stat->columns) + list_length(stat->exprs) <= STATS_MAX_DIMENSIONS); /* compute statistics target for this statistics */ stattarget = statext_compute_stattarget(stat->stattarget, @@ -167,6 +209,9 @@ BuildRelationExtStatistics(Relation onerel, double totalrows, if (stattarget == 0) continue; + /* evaluate expressions (if the statistics has any) */ + exprs = evaluate_expressions(onerel, stat->exprs, numrows, rows); + /* compute statistic of each requested type */ foreach(lc2, stat->types) { @@ -174,21 +219,43 @@ BuildRelationExtStatistics(Relation onerel, double totalrows, if (t == STATS_EXT_NDISTINCT) ndistinct = statext_ndistinct_build(totalrows, numrows, rows, - stat->columns, stats); + exprs, stat->columns, + stats); else if (t == STATS_EXT_DEPENDENCIES) dependencies = statext_dependencies_build(numrows, rows, - stat->columns, stats); + exprs, stat->columns, + stats); else if (t == STATS_EXT_MCV) - mcv = statext_mcv_build(numrows, rows, stat->columns, stats, - totalrows, stattarget); + mcv = statext_mcv_build(numrows, rows, exprs, stat->columns, + stats, totalrows, stattarget); + else if (t == STATS_EXT_EXPRESSIONS) + { + AnlExprData *exprdata; + int nexprs; + + /* should not happen, thanks to checks when defining stats */ + if (!stat->exprs) + elog(ERROR, "requested expression stats, but there are no expressions"); + + exprdata = build_expr_data(stat->exprs); + nexprs = list_length(stat->exprs); + + compute_expr_stats(onerel, totalrows, + exprdata, nexprs, + rows, numrows); + + exprstats = serialize_expr_stats(exprdata, nexprs); + } } /* store the statistics in the catalog */ - statext_store(stat->statOid, ndistinct, dependencies, mcv, stats); + statext_store(stat->statOid, ndistinct, dependencies, mcv, exprstats, stats); /* for reporting progress */ pgstat_progress_update_param(PROGRESS_ANALYZE_EXT_STATS_COMPUTED, ++ext_cnt); + + pfree(exprs); } table_close(pg_stext, RowExclusiveLock); @@ -241,7 +308,7 @@ ComputeExtStatisticsRows(Relation onerel, * analyzed. If not, ignore it (don't report anything, we'll do that * during the actual build BuildRelationExtStatistics). */ - stats = lookup_var_attr_stats(onerel, stat->columns, + stats = lookup_var_attr_stats(onerel, stat->columns, stat->exprs, natts, vacattrstats); if (!stats) @@ -349,6 +416,10 @@ statext_is_kind_built(HeapTuple htup, char type) attnum = Anum_pg_statistic_ext_data_stxdmcv; break; + case STATS_EXT_EXPRESSIONS: + attnum = Anum_pg_statistic_ext_data_stxdexpr; + break; + default: elog(ERROR, "unexpected statistics type requested: %d", type); } @@ -388,6 +459,7 @@ fetch_statentries_for_relation(Relation pg_statext, Oid relid) ArrayType *arr; char *enabled; Form_pg_statistic_ext staForm; + List *exprs = NIL; entry = palloc0(sizeof(StatExtEntry)); staForm = (Form_pg_statistic_ext) GETSTRUCT(htup); @@ -415,10 +487,39 @@ fetch_statentries_for_relation(Relation pg_statext, Oid relid) { Assert((enabled[i] == STATS_EXT_NDISTINCT) || (enabled[i] == STATS_EXT_DEPENDENCIES) || - (enabled[i] == STATS_EXT_MCV)); + (enabled[i] == STATS_EXT_MCV) || + (enabled[i] == STATS_EXT_EXPRESSIONS)); entry->types = lappend_int(entry->types, (int) enabled[i]); } + /* decode expression (if any) */ + datum = SysCacheGetAttr(STATEXTOID, htup, + Anum_pg_statistic_ext_stxexprs, &isnull); + + if (!isnull) + { + char *exprsString; + + exprsString = TextDatumGetCString(datum); + exprs = (List *) stringToNode(exprsString); + + pfree(exprsString); + + /* + * Run the expressions through eval_const_expressions. This is not just an + * optimization, but is necessary, because the planner will be comparing + * them to similarly-processed qual clauses, and may fail to detect valid + * matches without this. We must not use canonicalize_qual, however, + * since these aren't qual expressions. + */ + exprs = (List *) eval_const_expressions(NULL, (Node *) exprs); + + /* May as well fix opfuncids too */ + fix_opfuncids((Node *) exprs); + } + + entry->exprs = exprs; + result = lappend(result, entry); } @@ -427,6 +528,86 @@ fetch_statentries_for_relation(Relation pg_statext, Oid relid) return result; } + +/* + * examine_attribute -- pre-analysis of a single column + * + * Determine whether the column is analyzable; if so, create and initialize + * a VacAttrStats struct for it. If not, return NULL. + */ +static VacAttrStats * +examine_attribute(Node *expr) +{ + HeapTuple typtuple; + VacAttrStats *stats; + int i; + bool ok; + + /* + * Create the VacAttrStats struct. Note that we only have a copy of the + * fixed fields of the pg_attribute tuple. + */ + stats = (VacAttrStats *) palloc0(sizeof(VacAttrStats)); + + /* fake the attribute */ + stats->attr = (Form_pg_attribute) palloc0(ATTRIBUTE_FIXED_PART_SIZE); + stats->attr->attstattarget = -1; + + /* + * When analyzing an expression index, believe the expression tree's type + * not the column datatype --- the latter might be the opckeytype storage + * type of the opclass, which is not interesting for our purposes. (Note: + * if we did anything with non-expression index columns, we'd need to + * figure out where to get the correct type info from, but for now that's + * not a problem.) It's not clear whether anyone will care about the + * typmod, but we store that too just in case. + */ + stats->attrtypid = exprType(expr); + stats->attrtypmod = exprTypmod(expr); + stats->attrcollid = exprCollation(expr); + + typtuple = SearchSysCacheCopy1(TYPEOID, + ObjectIdGetDatum(stats->attrtypid)); + if (!HeapTupleIsValid(typtuple)) + elog(ERROR, "cache lookup failed for type %u", stats->attrtypid); + stats->attrtype = (Form_pg_type) GETSTRUCT(typtuple); + // stats->anl_context = anl_context; + stats->tupattnum = InvalidAttrNumber; + + /* + * The fields describing the stats->stavalues[n] element types default to + * the type of the data being analyzed, but the type-specific typanalyze + * function can change them if it wants to store something else. + */ + for (i = 0; i < STATISTIC_NUM_SLOTS; i++) + { + stats->statypid[i] = stats->attrtypid; + stats->statyplen[i] = stats->attrtype->typlen; + stats->statypbyval[i] = stats->attrtype->typbyval; + stats->statypalign[i] = stats->attrtype->typalign; + } + + /* + * Call the type-specific typanalyze function. If none is specified, use + * std_typanalyze(). + */ + if (OidIsValid(stats->attrtype->typanalyze)) + ok = DatumGetBool(OidFunctionCall1(stats->attrtype->typanalyze, + PointerGetDatum(stats))); + else + ok = std_typanalyze(stats); + + if (!ok || stats->compute_stats == NULL || stats->minrows <= 0) + { + heap_freetuple(typtuple); + pfree(stats->attr); + pfree(stats); + return NULL; + } + + return stats; +} + /* * Using 'vacatts' of size 'nvacatts' as input data, return a newly built * VacAttrStats array which includes only the items corresponding to @@ -435,15 +616,18 @@ fetch_statentries_for_relation(Relation pg_statext, Oid relid) * to the caller that the stats should not be built. */ static VacAttrStats ** -lookup_var_attr_stats(Relation rel, Bitmapset *attrs, +lookup_var_attr_stats(Relation rel, Bitmapset *attrs, List *exprs, int nvacatts, VacAttrStats **vacatts) { int i = 0; int x = -1; + int natts; VacAttrStats **stats; + ListCell *lc; - stats = (VacAttrStats **) - palloc(bms_num_members(attrs) * sizeof(VacAttrStats *)); + natts = bms_num_members(attrs) + list_length(exprs); + + stats = (VacAttrStats **) palloc(natts * sizeof(VacAttrStats *)); /* lookup VacAttrStats info for the requested columns (same attnum) */ while ((x = bms_next_member(attrs, x)) >= 0) @@ -480,6 +664,24 @@ lookup_var_attr_stats(Relation rel, Bitmapset *attrs, i++; } + /* also add info for expressions */ + foreach (lc, exprs) + { + Node *expr = (Node *) lfirst(lc); + + stats[i] = examine_attribute(expr); + + /* + * FIXME We need tuple descriptor later, and we just grab it from + * stats[0]->tupDesc (see e.g. statext_mcv_build). But as coded + * examine_attribute does not set that, so just grab it from the + * first vacatts element. + */ + stats[i]->tupDesc = vacatts[0]->tupDesc; + + i++; + } + return stats; } @@ -491,7 +693,7 @@ lookup_var_attr_stats(Relation rel, Bitmapset *attrs, static void statext_store(Oid statOid, MVNDistinct *ndistinct, MVDependencies *dependencies, - MCVList *mcv, VacAttrStats **stats) + MCVList *mcv, Datum exprs, VacAttrStats **stats) { Relation pg_stextdata; HeapTuple stup, @@ -532,11 +734,17 @@ statext_store(Oid statOid, nulls[Anum_pg_statistic_ext_data_stxdmcv - 1] = (data == NULL); values[Anum_pg_statistic_ext_data_stxdmcv - 1] = PointerGetDatum(data); } + if (exprs != (Datum) 0) + { + nulls[Anum_pg_statistic_ext_data_stxdexpr - 1] = false; + values[Anum_pg_statistic_ext_data_stxdexpr - 1] = exprs; + } /* always replace the value (either by bytea or NULL) */ replaces[Anum_pg_statistic_ext_data_stxdndistinct - 1] = true; replaces[Anum_pg_statistic_ext_data_stxddependencies - 1] = true; replaces[Anum_pg_statistic_ext_data_stxdmcv - 1] = true; + replaces[Anum_pg_statistic_ext_data_stxdexpr - 1] = true; /* there should already be a pg_statistic_ext_data tuple */ oldtup = SearchSysCache1(STATEXTDATASTXOID, ObjectIdGetDatum(statOid)); @@ -741,8 +949,9 @@ build_attnums_array(Bitmapset *attrs, int *numattrs) * can simply pfree the return value to release all of it. */ SortItem * -build_sorted_items(int numrows, int *nitems, HeapTuple *rows, TupleDesc tdesc, - MultiSortSupport mss, int numattrs, AttrNumber *attnums) +build_sorted_items(int numrows, int *nitems, HeapTuple *rows, ExprInfo *exprs, + TupleDesc tdesc, MultiSortSupport mss, + int numattrs, AttrNumber *attnums) { int i, j, @@ -789,8 +998,24 @@ build_sorted_items(int numrows, int *nitems, HeapTuple *rows, TupleDesc tdesc, { Datum value; bool isnull; + int attlen; + + if (attnums[j] <= MaxHeapAttributeNumber) + { + value = heap_getattr(rows[i], attnums[j], tdesc, &isnull); + attlen = TupleDescAttr(tdesc, attnums[j] - 1)->attlen; + } + else + { + int idx = EXPRESSION_INDEX(attnums[j]); + + Assert((idx >= 0) && (idx < exprs->nexprs)); + + value = exprs->values[idx][i]; + isnull = exprs->nulls[idx][i]; - value = heap_getattr(rows[i], attnums[j], tdesc, &isnull); + attlen = get_typlen(exprs->types[idx]); + } /* * If this is a varlena value, check if it's too wide and if yes @@ -801,8 +1026,7 @@ build_sorted_items(int numrows, int *nitems, HeapTuple *rows, TupleDesc tdesc, * on the assumption that those are small (below WIDTH_THRESHOLD) * and will be discarded at the end of analyze. */ - if ((!isnull) && - (TupleDescAttr(tdesc, attnums[j] - 1)->attlen == -1)) + if ((!isnull) && (attlen == -1)) { if (toast_raw_datum_size(value) > WIDTH_THRESHOLD) { @@ -881,7 +1105,8 @@ has_stats_of_kind(List *stats, char requiredkind) */ StatisticExtInfo * choose_best_statistics(List *stats, char requiredkind, - Bitmapset **clause_attnums, int nclauses) + Bitmapset **clause_attnums, List **clause_exprs, + int nclauses) { ListCell *lc; StatisticExtInfo *best_match = NULL; @@ -894,6 +1119,7 @@ choose_best_statistics(List *stats, char requiredkind, StatisticExtInfo *info = (StatisticExtInfo *) lfirst(lc); Bitmapset *matched = NULL; int num_matched; + int num_matched_exprs; int numkeys; /* skip statistics that are not of the correct type */ @@ -920,6 +1146,38 @@ choose_best_statistics(List *stats, char requiredkind, num_matched = bms_num_members(matched); bms_free(matched); + /* + * Collect expressions in remaining (unestimated) expressions, covered + * by an expression in this statistic object. + */ + num_matched_exprs = 0; + for (i = 0; i < nclauses; i++) + { + ListCell *lc3; + + /* ignore incompatible/estimated expressions */ + if (!clause_exprs[i]) + continue; + + /* ignore expressions that are not covered by this object */ + foreach (lc3, clause_exprs[i]) + { + ListCell *lc2; + Node *expr = (Node *) lfirst(lc3); + + foreach(lc2, info->exprs) + { + Node *stat_expr = (Node *) lfirst(lc2); + + if (equal(expr, stat_expr)) + { + num_matched_exprs++; + break; + } + } + } + } + /* * save the actual number of keys in the stats so that we can choose * the narrowest stats with the most matching keys. @@ -931,11 +1189,12 @@ choose_best_statistics(List *stats, char requiredkind, * when it matches the same number of attributes but these stats have * fewer keys than any previous match. */ - if (num_matched > best_num_matched || - (num_matched == best_num_matched && numkeys < best_match_keys)) + if (num_matched + num_matched_exprs > best_num_matched || + ((num_matched + num_matched_exprs) == best_num_matched && + numkeys < best_match_keys)) { best_match = info; - best_num_matched = num_matched; + best_num_matched = num_matched + num_matched_exprs; best_match_keys = numkeys; } } @@ -994,7 +1253,7 @@ statext_is_compatible_clause_internal(PlannerInfo *root, Node *clause, return false; /* Check if the expression has the right shape (one Var, one Const) */ - if (!examine_clause_args(expr->args, &var, NULL, NULL)) + if (!examine_opclause_expression(expr, &var, NULL, NULL)) return false; /* @@ -1150,6 +1409,187 @@ statext_is_compatible_clause_internal(PlannerInfo *root, Node *clause, return false; } +/* + * statext_extract_expression_internal + * FIXME + * + */ +static List * +statext_extract_expression_internal(PlannerInfo *root, Node *clause, Index relid) +{ + /* Look inside any binary-compatible relabeling (as in examine_variable) */ + if (IsA(clause, RelabelType)) + clause = (Node *) ((RelabelType *) clause)->arg; + + /* plain Var references (boolean Vars or recursive checks) */ + if (IsA(clause, Var)) + { + Var *var = (Var *) clause; + + /* Ensure var is from the correct relation */ + if (var->varno != relid) + return NIL; + + /* we also better ensure the Var is from the current level */ + if (var->varlevelsup > 0) + return NIL; + + /* Also skip system attributes (we don't allow stats on those). */ + if (!AttrNumberIsForUserDefinedAttr(var->varattno)) + return NIL; + + return list_make1(clause); + } + + /* (Var op Const) or (Const op Var) */ + if (is_opclause(clause)) + { + RangeTblEntry *rte = root->simple_rte_array[relid]; + OpExpr *expr = (OpExpr *) clause; + Node *expr2 = NULL; + + /* Only expressions with two arguments are considered compatible. */ + if (list_length(expr->args) != 2) + return NIL; + + /* Check if the expression has the right shape (one Expr, one Const) */ + if (!examine_opclause_expression2(expr, &expr2, NULL, NULL)) + return NIL; + + /* + * If it's not one of the supported operators ("=", "<", ">", etc.), + * just ignore the clause, as it's not compatible with MCV lists. + * + * This uses the function for estimating selectivity, not the operator + * directly (a bit awkward, but well ...). + */ + switch (get_oprrest(expr->opno)) + { + case F_EQSEL: + case F_NEQSEL: + case F_SCALARLTSEL: + case F_SCALARLESEL: + case F_SCALARGTSEL: + case F_SCALARGESEL: + /* supported, will continue with inspection of the Var */ + break; + + default: + /* other estimators are considered unknown/unsupported */ + return NIL; + } + + /* + * If there are any securityQuals on the RTE from security barrier + * views or RLS policies, then the user may not have access to all the + * table's data, and we must check that the operator is leak-proof. + * + * If the operator is leaky, then we must ignore this clause for the + * purposes of estimating with MCV lists, otherwise the operator might + * reveal values from the MCV list that the user doesn't have + * permission to see. + */ + if (rte->securityQuals != NIL && + !get_func_leakproof(get_opcode(expr->opno))) + return NIL; + + return list_make1(expr2); + } + + if (IsA(clause, ScalarArrayOpExpr)) + { + RangeTblEntry *rte = root->simple_rte_array[relid]; + ScalarArrayOpExpr *expr = (ScalarArrayOpExpr *) clause; + Node *expr2 = NULL; + + /* Only expressions with two arguments are considered compatible. */ + if (list_length(expr->args) != 2) + return NIL; + + /* Check if the expression has the right shape (one Expr, one Const) */ + if (!examine_clause_args2(expr->args, &expr2, NULL, NULL)) + return NIL; + + /* + * If there are any securityQuals on the RTE from security barrier + * views or RLS policies, then the user may not have access to all the + * table's data, and we must check that the operator is leak-proof. + * + * If the operator is leaky, then we must ignore this clause for the + * purposes of estimating with MCV lists, otherwise the operator might + * reveal values from the MCV list that the user doesn't have + * permission to see. + */ + if (rte->securityQuals != NIL && + !get_func_leakproof(get_opcode(expr->opno))) + return NIL; + + return list_make1(expr2); + } + + /* AND/OR/NOT clause */ + if (is_andclause(clause) || + is_orclause(clause) || + is_notclause(clause)) + { + /* + * AND/OR/NOT-clauses are supported if all sub-clauses are supported + * + * Perhaps we could improve this by handling mixed cases, when some of + * the clauses are supported and some are not. Selectivity for the + * supported subclauses would be computed using extended statistics, + * and the remaining clauses would be estimated using the traditional + * algorithm (product of selectivities). + * + * It however seems overly complex, and in a way we already do that + * because if we reject the whole clause as unsupported here, it will + * be eventually passed to clauselist_selectivity() which does exactly + * this (split into supported/unsupported clauses etc). + */ + BoolExpr *expr = (BoolExpr *) clause; + ListCell *lc; + List *exprs = NIL; + + foreach(lc, expr->args) + { + List *tmp; + + /* + * Had we found incompatible clause in the arguments, treat the + * whole clause as incompatible. + */ + tmp = statext_extract_expression_internal(root, + (Node *) lfirst(lc), + relid); + + if (!tmp) + return NIL; + + exprs = list_concat(exprs, tmp); + } + + return exprs; + } + + /* Var IS NULL */ + if (IsA(clause, NullTest)) + { + NullTest *nt = (NullTest *) clause; + + /* + * Only simple (Var IS NULL) expressions supported for now. Maybe we + * could use examine_variable to fix this? + */ + if (!IsA(nt->arg, Var)) + return NIL; + + return statext_extract_expression_internal(root, (Node *) (nt->arg), + relid); + } + + return NIL; +} + /* * statext_is_compatible_clause * Determines if the clause is compatible with MCV lists. @@ -1163,6 +1603,8 @@ statext_is_compatible_clause_internal(PlannerInfo *root, Node *clause, * * (c) combinations using AND/OR/NOT * + * (d) ScalarArrayOpExprs of the form (Var op ANY (array)) or (Var op ALL (array)) + * * In the future, the range of supported clauses may be expanded to more * complex cases, for example (Var op Var). */ @@ -1225,15 +1667,62 @@ statext_is_compatible_clause(PlannerInfo *root, Node *clause, Index relid, } /* - * statext_mcv_clauselist_selectivity - * Estimate clauses using the best multi-column statistics. + * statext_extract_expression + * Determines if the clause is compatible with extended statistics. * - * Applies available extended (multi-column) statistics on a table. There may - * be multiple applicable statistics (with respect to the clauses), in which - * case we use greedy approach. In each round we select the best statistic on - * a table (measured by the number of attributes extracted from the clauses - * and covered by it), and compute the selectivity for the supplied clauses. - * We repeat this process with the remaining clauses (if any), until none of + * Currently, we only support three types of clauses: + * + * (a) OpExprs of the form (Var op Const), or (Const op Var), where the op + * is one of ("=", "<", ">", ">=", "<=") + * + * (b) (Var IS [NOT] NULL) + * + * (c) combinations using AND/OR/NOT + * + * (d) ScalarArrayOpExprs of the form (Var op ANY (array)) or (Var op ALL (array)) + * + * In the future, the range of supported clauses may be expanded to more + * complex cases, for example (Var op Var). + */ +static List * +statext_extract_expression(PlannerInfo *root, Node *clause, Index relid) +{ + RestrictInfo *rinfo = (RestrictInfo *) clause; + List *exprs; + + if (!IsA(rinfo, RestrictInfo)) + return NIL; + + /* Pseudoconstants are not really interesting here. */ + if (rinfo->pseudoconstant) + return NIL; + + /* clauses referencing multiple varnos are incompatible */ + if (bms_membership(rinfo->clause_relids) != BMS_SINGLETON) + return NIL; + + /* Check the clause and determine what attributes it references. */ + exprs = statext_extract_expression_internal(root, (Node *) rinfo->clause, relid); + + if (!exprs) + return NIL; + + /* FIXME do the same ACL check as in statext_is_compatible_clause */ + + /* If we reach here, the clause is OK */ + return exprs; +} + +/* + * statext_mcv_clauselist_selectivity + * Estimate clauses using the best multi-column statistics. + * + * Applies available extended (multi-column) statistics on a table. There may + * be multiple applicable statistics (with respect to the clauses), in which + * case we use greedy approach. In each round we select the best statistic on + * a table (measured by the number of attributes extracted from the clauses + * and covered by it), and compute the selectivity for the supplied clauses. + * We repeat this process with the remaining clauses (if any), until none of * the available statistics can be used. * * One of the main challenges with using MCV lists is how to extrapolate the @@ -1265,7 +1754,8 @@ statext_mcv_clauselist_selectivity(PlannerInfo *root, List *clauses, int varReli bool is_or) { ListCell *l; - Bitmapset **list_attnums; + Bitmapset **list_attnums; /* attnums extracted from the clause */ + List **list_exprs; /* expressions matched to any statistic */ int listidx; Selectivity sel = (is_or) ? 0.0 : 1.0; @@ -1276,6 +1766,9 @@ statext_mcv_clauselist_selectivity(PlannerInfo *root, List *clauses, int varReli list_attnums = (Bitmapset **) palloc(sizeof(Bitmapset *) * list_length(clauses)); + /* expressions extracted from complex expressions */ + list_exprs = (List **) palloc(sizeof(Node *) * list_length(clauses)); + /* * Pre-process the clauses list to extract the attnums seen in each item. * We need to determine if there's any clauses which will be useful for @@ -1293,11 +1786,100 @@ statext_mcv_clauselist_selectivity(PlannerInfo *root, List *clauses, int varReli Node *clause = (Node *) lfirst(l); Bitmapset *attnums = NULL; + /* the clause is considered incompatible by default */ + list_attnums[listidx] = NULL; + + /* and it's also not covered exactly by the statistic */ + list_exprs[listidx] = NULL; + + /* + * First see if the clause is simple enough to be covered directly + * by the attributes. If not, see if there's at least one statistic + * object using the expression as-is. + */ if (!bms_is_member(listidx, *estimatedclauses) && statext_is_compatible_clause(root, clause, rel->relid, &attnums)) + { + /* simple expression, covered through attnum(s) */ list_attnums[listidx] = attnums; + } else - list_attnums[listidx] = NULL; + { + ListCell *lc; + List *exprs; + + /* + * XXX This is kinda dubious, because we extract the smallest + * clauses - e.g. from (Var op Const) we extract Var. But maybe + * the statistics covers larger expressions, so maybe this will + * skip that. For example give ((a+b) + (c+d)) it's not clear + * if we should extract the whole clause or some smaller parts. + * OTOH we need (Expr op Const) so maybe we only care about the + * clause as a whole? + */ + exprs = statext_extract_expression(root, clause, rel->relid); + + /* complex expression, search for statistic covering all parts */ + foreach(lc, rel->statlist) + { + ListCell *le; + StatisticExtInfo *info = (StatisticExtInfo *) lfirst(lc); + + /* + * Assume all parts are covered by this statistics, we'll + * stop if we found part that is not covered. + */ + bool covered = true; + + /* have we already matched the expression to a statistic? */ + Assert(!list_exprs[listidx]); + + /* no expressions in the statistic */ + if (!info->exprs) + continue; + + foreach(le, exprs) + { + ListCell *lc2; + Node *expr = (Node *) lfirst(le); + bool found = false; + + /* + * Walk the expressions, see if all expressions extracted from + * the clause are covered by the extended statistic object. + */ + foreach (lc2, info->exprs) + { + Node *stat_expr = (Node *) lfirst(lc2); + + if (equal(expr, stat_expr)) + { + found = true; + break; + } + } + + /* found expression not covered by the statistics, stop */ + if (!found) + { + covered = false; + break; + } + } + + /* + * OK, we found a statistics covering this clause, stop looking + * for another one + */ + if (covered) + { + /* XXX should this add the original expression instead? */ + list_exprs[listidx] = exprs; + break; + } + + } + } listidx++; } @@ -1311,7 +1893,8 @@ statext_mcv_clauselist_selectivity(PlannerInfo *root, List *clauses, int varReli /* find the best suited statistics object for these attnums */ stat = choose_best_statistics(rel->statlist, STATS_EXT_MCV, - list_attnums, list_length(clauses)); + list_attnums, list_exprs, + list_length(clauses)); /* * if no (additional) matching stats could be found then we've nothing @@ -1334,11 +1917,13 @@ statext_mcv_clauselist_selectivity(PlannerInfo *root, List *clauses, int varReli { /* * If the clause is compatible with the selected statistics, mark - * it as estimated and add it to the list to estimate. + * it as estimated and add it to the list to estimate. It may be + * either a simple clause, or an expression. */ if (list_attnums[listidx] != NULL && bms_is_subset(list_attnums[listidx], stat->keys)) { + /* simple clause (single Var) */ if (bms_membership(list_attnums[listidx]) == BMS_SINGLETON) simple_clauses = bms_add_member(simple_clauses, list_length(stat_clauses)); @@ -1349,6 +1934,45 @@ statext_mcv_clauselist_selectivity(PlannerInfo *root, List *clauses, int varReli bms_free(list_attnums[listidx]); list_attnums[listidx] = NULL; } + else if (list_exprs[listidx] != NIL) + { + /* are all parts of the expression covered by the statistic? */ + ListCell *lc; + int ncovered = 0; + + foreach (lc, list_exprs[listidx]) + { + ListCell *lc2; + Node *expr = (Node *) lfirst(lc); + bool found = false; + + foreach (lc2, stat->exprs) + { + Node *stat_expr = (Node *) lfirst(lc2); + + if (equal(expr, stat_expr)) + { + found = true; + break; + } + } + + /* count it as covered and continue to the next expression */ + if (found) + ncovered++; + } + + /* all parts of thi expression are covered by this statistics */ + if (ncovered == list_length(list_exprs[listidx])) + { + stat_clauses = lappend(stat_clauses, (Node *) lfirst(l)); + *estimatedclauses = bms_add_member(*estimatedclauses, listidx); + + // bms_free(list_attnums[listidx]); + list_exprs[listidx] = NULL; + } + + } listidx++; } @@ -1587,3 +2211,777 @@ examine_clause_args(List *args, Var **varp, Const **cstp, bool *varonleftp) return true; } + +bool +examine_clause_args2(List *args, Node **exprp, Const **cstp, bool *expronleftp) +{ + Node *expr; + Const *cst; + bool expronleft; + Node *leftop, + *rightop; + + /* enforced by statext_is_compatible_clause_internal */ + Assert(list_length(args) == 2); + + leftop = linitial(args); + rightop = lsecond(args); + + /* strip RelabelType from either side of the expression */ + if (IsA(leftop, RelabelType)) + leftop = (Node *) ((RelabelType *) leftop)->arg; + + if (IsA(rightop, RelabelType)) + rightop = (Node *) ((RelabelType *) rightop)->arg; + + if (IsA(rightop, Const)) + { + expr = (Node *) leftop; + cst = (Const *) rightop; + expronleft = true; + } + else if (IsA(leftop, Const)) + { + expr = (Node *) rightop; + cst = (Const *) leftop; + expronleft = false; + } + else + return false; + + /* return pointers to the extracted parts if requested */ + if (exprp) + *exprp = expr; + + if (cstp) + *cstp = cst; + + if (expronleftp) + *expronleftp = expronleft; + + return true; +} + +bool +examine_opclause_expression(OpExpr *expr, Var **varp, Const **cstp, bool *varonleftp) +{ + Var *var; + Const *cst; + bool varonleft; + Node *leftop, + *rightop; + + /* enforced by statext_is_compatible_clause_internal */ + Assert(list_length(expr->args) == 2); + + leftop = linitial(expr->args); + rightop = lsecond(expr->args); + + /* strip RelabelType from either side of the expression */ + if (IsA(leftop, RelabelType)) + leftop = (Node *) ((RelabelType *) leftop)->arg; + + if (IsA(rightop, RelabelType)) + rightop = (Node *) ((RelabelType *) rightop)->arg; + + if (IsA(leftop, Var) && IsA(rightop, Const)) + { + var = (Var *) leftop; + cst = (Const *) rightop; + varonleft = true; + } + else if (IsA(leftop, Const) && IsA(rightop, Var)) + { + var = (Var *) rightop; + cst = (Const *) leftop; + varonleft = false; + } + else + return false; + + /* return pointers to the extracted parts if requested */ + if (varp) + *varp = var; + + if (cstp) + *cstp = cst; + + if (varonleftp) + *varonleftp = varonleft; + + return true; +} + +bool +examine_opclause_expression2(OpExpr *expr, Node **exprp, Const **cstp, bool *expronleftp) +{ + Node *expr2; + Const *cst; + bool expronleft; + Node *leftop, + *rightop; + + /* enforced by statext_is_compatible_clause_internal */ + Assert(list_length(expr->args) == 2); + + leftop = linitial(expr->args); + rightop = lsecond(expr->args); + + /* strip RelabelType from either side of the expression */ + if (IsA(leftop, RelabelType)) + leftop = (Node *) ((RelabelType *) leftop)->arg; + + if (IsA(rightop, RelabelType)) + rightop = (Node *) ((RelabelType *) rightop)->arg; + + if (IsA(rightop, Const)) + { + expr2 = (Node *) leftop; + cst = (Const *) rightop; + expronleft = true; + } + else if (IsA(leftop, Const)) + { + expr2 = (Node *) rightop; + cst = (Const *) leftop; + expronleft = false; + } + else + return false; + + /* return pointers to the extracted parts if requested */ + if (exprp) + *exprp = expr2; + + if (cstp) + *cstp = cst; + + if (expronleftp) + *expronleftp = expronleft; + + return true; +} + + +/* + * Compute statistics about expressions of a relation. + */ +static void +compute_expr_stats(Relation onerel, double totalrows, + AnlExprData *exprdata, int nexprs, + HeapTuple *rows, int numrows) +{ + MemoryContext expr_context, + old_context; + int ind, + i; + + expr_context = AllocSetContextCreate(CurrentMemoryContext, + "Analyze Expression", + ALLOCSET_DEFAULT_SIZES); + old_context = MemoryContextSwitchTo(expr_context); + + for (ind = 0; ind < nexprs; ind++) + { + AnlExprData *thisdata = &exprdata[ind]; + Node *expr = thisdata->expr; + TupleTableSlot *slot; + EState *estate; + ExprContext *econtext; + Datum *exprvals; + bool *exprnulls; + ExprState *exprstate; + int tcnt; + + /* + * Need an EState for evaluation of expressions. Create it in + * the per-expression context to be sure it gets cleaned up at + * the bottom of the loop. + */ + estate = CreateExecutorState(); + econtext = GetPerTupleExprContext(estate); + + /* Set up expression evaluation state */ + exprstate = ExecPrepareExpr((Expr *) expr, estate); + + /* Need a slot to hold the current heap tuple, too */ + slot = MakeSingleTupleTableSlot(RelationGetDescr(onerel), + &TTSOpsHeapTuple); + + /* Arrange for econtext's scan tuple to be the tuple under test */ + econtext->ecxt_scantuple = slot; + + /* Compute and save index expression values */ + exprvals = (Datum *) palloc(numrows * sizeof(Datum)); + exprnulls = (bool *) palloc(numrows * sizeof(bool)); + + tcnt = 0; + for (i = 0; i < numrows; i++) + { + Datum datum; + bool isnull; + + /* + * Reset the per-tuple context each time, to reclaim any cruft + * left behind by evaluating the predicate or index expressions. + */ + ResetExprContext(econtext); + + /* Set up for predicate or expression evaluation */ + ExecStoreHeapTuple(rows[i], slot, false); + + /* + * FIXME this probably leaks memory. Maybe we should use + * ExecEvalExprSwitchContext but then we need to copy the + * result somewhere else. + */ + datum = ExecEvalExpr(exprstate, + GetPerTupleExprContext(estate), + &isnull); + if (isnull) + { + exprvals[tcnt] = (Datum) 0; + exprnulls[tcnt] = true; + } + else + { + exprvals[tcnt] = (Datum) datum; + exprnulls[tcnt] = false; + } + + tcnt++; + } + + /* + * Now we can compute the statistics for the expression columns. + */ + if (tcnt > 0) + { + // MemoryContextSwitchTo(col_context); + VacAttrStats *stats = thisdata->vacattrstat; + AttributeOpts *aopt = + get_attribute_options(stats->attr->attrelid, + stats->attr->attnum); + + stats->exprvals = exprvals; + stats->exprnulls = exprnulls; + stats->rowstride = 1; + stats->compute_stats(stats, + expr_fetch_func, + tcnt, + tcnt); + + /* + * If the n_distinct option is specified, it overrides the + * above computation. + */ + if (aopt != NULL && aopt->n_distinct != 0.0) + stats->stadistinct = aopt->n_distinct; + + // MemoryContextResetAndDeleteChildren(col_context); + } + + /* And clean up */ + // MemoryContextSwitchTo(expr_context); + + ExecDropSingleTupleTableSlot(slot); + FreeExecutorState(estate); + // MemoryContextResetAndDeleteChildren(expr_context); + } + + MemoryContextSwitchTo(old_context); + MemoryContextDelete(expr_context); +} + + +/* + * Fetch function for analyzing index expressions. + * + * We have not bothered to construct index tuples, instead the data is + * just in Datum arrays. + */ +static Datum +expr_fetch_func(VacAttrStatsP stats, int rownum, bool *isNull) +{ + int i; + + /* exprvals and exprnulls are already offset for proper column */ + i = rownum * stats->rowstride; + *isNull = stats->exprnulls[i]; + return stats->exprvals[i]; +} + +/* + * Build analyze data for a list of expressions. As this is not tied + * directly to a relation (table or index), we have to fake some of + * the data. + */ +static AnlExprData * +build_expr_data(List *exprs) +{ + int idx; + int nexprs = list_length(exprs); + AnlExprData *exprdata; + ListCell *lc; + + exprdata = (AnlExprData *) palloc0(nexprs * sizeof(AnlExprData)); + + idx = 0; + foreach (lc, exprs) + { + Node *expr = (Node *) lfirst(lc); + AnlExprData *thisdata = &exprdata[idx]; + + thisdata->expr = expr; + thisdata->vacattrstat = (VacAttrStats *) palloc(sizeof(VacAttrStats)); + + thisdata->vacattrstat = examine_expression(expr); + idx++; + } + + return exprdata; +} + +/* + * examine_expression -- pre-analysis of a single column + * + * Determine whether the column is analyzable; if so, create and initialize + * a VacAttrStats struct for it. If not, return NULL. + */ +static VacAttrStats * +examine_expression(Node *expr) +{ + HeapTuple typtuple; + VacAttrStats *stats; + int i; + bool ok; + + Assert(expr != NULL); + + /* + * Create the VacAttrStats struct. + */ + stats = (VacAttrStats *) palloc0(sizeof(VacAttrStats)); + + /* + * When analyzing an expression, believe the expression tree's type. + */ + stats->attrtypid = exprType(expr); + stats->attrtypmod = exprTypmod(expr); + + /* + * XXX Do we need to do anything special about the collation, similar + * to what examine_attribute does for expression indexes? + */ + stats->attrcollid = exprCollation(expr); + + /* + * We don't have any pg_attribute for expressions, so let's fake + * something reasonable into attstattarget, which is the only thing + * std_typanalyze needs. + */ + stats->attr = (Form_pg_attribute) palloc(ATTRIBUTE_FIXED_PART_SIZE); + + /* + * FIXME we should probably get the target from the extended stats + * object, or something like that. + */ + stats->attr->attstattarget = default_statistics_target; + + /* initialize some basic fields */ + stats->attr->attrelid = InvalidOid; + stats->attr->attnum = InvalidAttrNumber; + stats->attr->atttypid = stats->attrtypid; + + typtuple = SearchSysCacheCopy1(TYPEOID, + ObjectIdGetDatum(stats->attrtypid)); + if (!HeapTupleIsValid(typtuple)) + elog(ERROR, "cache lookup failed for type %u", stats->attrtypid); + stats->attrtype = (Form_pg_type) GETSTRUCT(typtuple); + stats->anl_context = CurrentMemoryContext; /* XXX should be using something else? */ + stats->tupattnum = InvalidAttrNumber; + + /* + * The fields describing the stats->stavalues[n] element types default to + * the type of the data being analyzed, but the type-specific typanalyze + * function can change them if it wants to store something else. + */ + for (i = 0; i < STATISTIC_NUM_SLOTS; i++) + { + stats->statypid[i] = stats->attrtypid; + stats->statyplen[i] = stats->attrtype->typlen; + stats->statypbyval[i] = stats->attrtype->typbyval; + stats->statypalign[i] = stats->attrtype->typalign; + } + + /* + * Call the type-specific typanalyze function. If none is specified, use + * std_typanalyze(). + */ + if (OidIsValid(stats->attrtype->typanalyze)) + ok = DatumGetBool(OidFunctionCall1(stats->attrtype->typanalyze, + PointerGetDatum(stats))); + else + ok = std_typanalyze(stats); + + if (!ok || stats->compute_stats == NULL || stats->minrows <= 0) + { + heap_freetuple(typtuple); + pfree(stats); + return NULL; + } + + return stats; +} + +/* form an array of pg_statistic rows (per update_attstats) */ +static Datum +serialize_expr_stats(AnlExprData *exprdata, int nexprs) +{ + int exprno; + Oid typOid; + Relation sd; + + ArrayBuildState *astate = NULL; + + sd = table_open(StatisticRelationId, RowExclusiveLock); + + /* lookup OID of composite type for pg_statistic */ + typOid = get_rel_type_id(StatisticRelationId); + if (!OidIsValid(typOid)) + ereport(ERROR, + (errcode(ERRCODE_WRONG_OBJECT_TYPE), + errmsg("relation \"pg_statistic\" does not have a composite type"))); + + for (exprno = 0; exprno < nexprs; exprno++) + { + int i, k; + VacAttrStats *stats = exprdata[exprno].vacattrstat; + + Datum values[Natts_pg_statistic]; + bool nulls[Natts_pg_statistic]; + HeapTuple stup; + + if (!stats->stats_valid) + { + astate = accumArrayResult(astate, + (Datum) 0, + true, + typOid, + CurrentMemoryContext); + continue; + } + + /* + * Construct a new pg_statistic tuple + */ + for (i = 0; i < Natts_pg_statistic; ++i) + { + nulls[i] = false; + } + + values[Anum_pg_statistic_starelid - 1] = ObjectIdGetDatum(InvalidOid); + values[Anum_pg_statistic_staattnum - 1] = Int16GetDatum(InvalidAttrNumber); + values[Anum_pg_statistic_stainherit - 1] = BoolGetDatum(false); + values[Anum_pg_statistic_stanullfrac - 1] = Float4GetDatum(stats->stanullfrac); + values[Anum_pg_statistic_stawidth - 1] = Int32GetDatum(stats->stawidth); + values[Anum_pg_statistic_stadistinct - 1] = Float4GetDatum(stats->stadistinct); + i = Anum_pg_statistic_stakind1 - 1; + for (k = 0; k < STATISTIC_NUM_SLOTS; k++) + { + values[i++] = Int16GetDatum(stats->stakind[k]); /* stakindN */ + } + i = Anum_pg_statistic_staop1 - 1; + for (k = 0; k < STATISTIC_NUM_SLOTS; k++) + { + values[i++] = ObjectIdGetDatum(stats->staop[k]); /* staopN */ + } + i = Anum_pg_statistic_stacoll1 - 1; + for (k = 0; k < STATISTIC_NUM_SLOTS; k++) + { + values[i++] = ObjectIdGetDatum(stats->stacoll[k]); /* stacollN */ + } + i = Anum_pg_statistic_stanumbers1 - 1; + for (k = 0; k < STATISTIC_NUM_SLOTS; k++) + { + int nnum = stats->numnumbers[k]; + + if (nnum > 0) + { + int n; + Datum *numdatums = (Datum *) palloc(nnum * sizeof(Datum)); + ArrayType *arry; + + for (n = 0; n < nnum; n++) + numdatums[n] = Float4GetDatum(stats->stanumbers[k][n]); + /* XXX knows more than it should about type float4: */ + arry = construct_array(numdatums, nnum, + FLOAT4OID, + sizeof(float4), true, TYPALIGN_INT); + values[i++] = PointerGetDatum(arry); /* stanumbersN */ + } + else + { + nulls[i] = true; + values[i++] = (Datum) 0; + } + } + i = Anum_pg_statistic_stavalues1 - 1; + for (k = 0; k < STATISTIC_NUM_SLOTS; k++) + { + if (stats->numvalues[k] > 0) + { + ArrayType *arry; + + arry = construct_array(stats->stavalues[k], + stats->numvalues[k], + stats->statypid[k], + stats->statyplen[k], + stats->statypbyval[k], + stats->statypalign[k]); + values[i++] = PointerGetDatum(arry); /* stavaluesN */ + } + else + { + nulls[i] = true; + values[i++] = (Datum) 0; + } + } + + stup = heap_form_tuple(RelationGetDescr(sd), values, nulls); + + astate = accumArrayResult(astate, + heap_copy_tuple_as_datum(stup, RelationGetDescr(sd)), + false, + typOid, + CurrentMemoryContext); + } + + table_close(sd, RowExclusiveLock); + + return makeArrayResult(astate, CurrentMemoryContext); +} + + +/* + * Loads pg_statistic record from expression statistics for expression + * identified by the supplied index. + */ +HeapTuple +statext_expressions_load(Oid stxoid, int idx) +{ + bool isnull; + Datum value; + HeapTuple htup; + ExpandedArrayHeader *eah; + HeapTupleHeader td; + HeapTupleData tmptup; + HeapTuple tup; + + htup = SearchSysCache1(STATEXTDATASTXOID, ObjectIdGetDatum(stxoid)); + if (!HeapTupleIsValid(htup)) + elog(ERROR, "cache lookup failed for statistics object %u", stxoid); + + value = SysCacheGetAttr(STATEXTDATASTXOID, htup, + Anum_pg_statistic_ext_data_stxdexpr, &isnull); + if (isnull) + elog(ERROR, + "requested statistic kind \"%c\" is not yet built for statistics object %u", + STATS_EXT_DEPENDENCIES, stxoid); + + eah = DatumGetExpandedArray(value); + + deconstruct_expanded_array(eah); + + td = DatumGetHeapTupleHeader(eah->dvalues[idx]); + + /* Build a temporary HeapTuple control structure */ + tmptup.t_len = HeapTupleHeaderGetDatumLength(td); + tmptup.t_data = td; + + tup = heap_copytuple(&tmptup); + + ReleaseSysCache(htup); + + return tup; +} + +/* + * Evaluate the expressions, so that we can use the results to build + * all the requested statistics types. This matters especially for + * expensive expressions, of course. + */ +static ExprInfo * +evaluate_expressions(Relation rel, List *exprs, int numrows, HeapTuple *rows) +{ + /* evaluated expressions */ + ExprInfo *result; + char *ptr; + Size len; + + int i; + int idx; + TupleTableSlot *slot; + EState *estate; + ExprContext *econtext; + List *exprstates = NIL; + int nexprs = list_length(exprs); + ListCell *lc; + + /* allocate everything as a single chunk, so we can free it easily */ + len = MAXALIGN(sizeof(ExprInfo)); + len += MAXALIGN(sizeof(Oid) * nexprs); /* types */ + len += MAXALIGN(sizeof(Oid) * nexprs); /* collations */ + + /* values */ + len += MAXALIGN(sizeof(Datum *) * nexprs); + len += nexprs * MAXALIGN(sizeof(Datum) * numrows); + + /* nulls */ + len += MAXALIGN(sizeof(bool *) * nexprs); + len += nexprs * MAXALIGN(sizeof(bool) * numrows); + + ptr = palloc(len); + + /* set the pointers */ + result = (ExprInfo *) ptr; + ptr += sizeof(ExprInfo); + + /* types */ + result->types = (Oid *) ptr; + ptr += MAXALIGN(sizeof(Oid) * nexprs); + + /* collations */ + result->collations = (Oid *) ptr; + ptr += MAXALIGN(sizeof(Oid) * nexprs); + + /* values */ + result->values = (Datum **) ptr; + ptr += MAXALIGN(sizeof(Datum *) * nexprs); + + /* nulls */ + result->nulls = (bool **) ptr; + ptr += MAXALIGN(sizeof(bool *) * nexprs); + + for (i = 0; i < nexprs; i++) + { + result->values[i] = (Datum *) ptr; + ptr += MAXALIGN(sizeof(Datum) * numrows); + + result->nulls[i] = (bool *) ptr; + ptr += MAXALIGN(sizeof(bool) * numrows); + } + + Assert((ptr - (char *) result) == len); + + result->nexprs = list_length(exprs); + + idx = 0; + foreach (lc, exprs) + { + Node *expr = (Node *) lfirst(lc); + + result->types[idx] = exprType(expr); + result->collations[idx] = exprCollation(expr); + + idx++; + } + + /* + * Need an EState for evaluation of index expressions and + * partial-index predicates. Create it in the per-index context to be + * sure it gets cleaned up at the bottom of the loop. + */ + estate = CreateExecutorState(); + econtext = GetPerTupleExprContext(estate); + + /* Need a slot to hold the current heap tuple, too */ + slot = MakeSingleTupleTableSlot(RelationGetDescr(rel), + &TTSOpsHeapTuple); + + /* Arrange for econtext's scan tuple to be the tuple under test */ + econtext->ecxt_scantuple = slot; + + /* Set up expression evaluation state */ + exprstates = ExecPrepareExprList(exprs, estate); + + for (i = 0; i < numrows; i++) + { + /* + * Reset the per-tuple context each time, to reclaim any cruft + * left behind by evaluating the predicate or index expressions. + */ + ResetExprContext(econtext); + + /* Set up for predicate or expression evaluation */ + ExecStoreHeapTuple(rows[i], slot, false); + + idx = 0; + foreach (lc, exprstates) + { + Datum datum; + bool isnull; + ExprState *exprstate = (ExprState *) lfirst(lc); + + /* + * FIXME this probably leaks memory. Maybe we should use + * ExecEvalExprSwitchContext but then we need to copy the + * result somewhere else. + */ + datum = ExecEvalExpr(exprstate, + GetPerTupleExprContext(estate), + &isnull); + if (isnull) + { + result->values[idx][i] = (Datum) 0; + result->nulls[idx][i] = true; + } + else + { + result->values[idx][i] = (Datum) datum; + result->nulls[idx][i] = false; + } + + idx++; + } + } + + ExecDropSingleTupleTableSlot(slot); + FreeExecutorState(estate); + + return result; +} + +/* + * add_expressions_to_attributes + * add expressions as attributes with high attnums + * + * Treat the expressions as attributes with attnums above the regular + * attnum range. This will allow us to handle everything in the same + * way, and identify expressions in the dependencies. + * + * XXX This always creates a copy of the bitmap. We might optimize this + * by only creating the copy with (nexprs > 0) but then we'd have to track + * this in order to free it (if we want to). Does not seem worth it. + */ +Bitmapset * +add_expressions_to_attributes(Bitmapset *attrs, int nexprs) +{ + int i; + + /* + * Copy the bitmapset and add fake attnums representing expressions, + * starting above MaxHeapAttributeNumber. + */ + attrs = bms_copy(attrs); + + /* start with (MaxHeapAttributeNumber + 1) */ + for (i = 0; i < nexprs; i++) + { + Assert(EXPRESSION_ATTNUM(i) > MaxHeapAttributeNumber); + + attrs = bms_add_member(attrs, EXPRESSION_ATTNUM(i)); + } + + return attrs; +} diff --git a/src/backend/statistics/mcv.c b/src/backend/statistics/mcv.c index fae792a2dd..4abd98eb1d 100644 --- a/src/backend/statistics/mcv.c +++ b/src/backend/statistics/mcv.c @@ -74,7 +74,8 @@ ((ndims) * sizeof(DimensionInfo)) + \ ((nitems) * ITEM_SIZE(ndims))) -static MultiSortSupport build_mss(VacAttrStats **stats, int numattrs); +static MultiSortSupport build_mss(VacAttrStats **stats, int numattrs, + ExprInfo *exprs); static SortItem *build_distinct_groups(int numrows, SortItem *items, MultiSortSupport mss, int *ndistinct); @@ -181,8 +182,9 @@ get_mincount_for_mcv_list(int samplerows, double totalrows) * */ MCVList * -statext_mcv_build(int numrows, HeapTuple *rows, Bitmapset *attrs, - VacAttrStats **stats, double totalrows, int stattarget) +statext_mcv_build(int numrows, HeapTuple *rows, ExprInfo *exprs, + Bitmapset *attrs, VacAttrStats **stats, + double totalrows, int stattarget) { int i, numattrs, @@ -195,14 +197,23 @@ statext_mcv_build(int numrows, HeapTuple *rows, Bitmapset *attrs, MCVList *mcvlist = NULL; MultiSortSupport mss; - attnums = build_attnums_array(attrs, &numattrs); - /* comparator for all the columns */ - mss = build_mss(stats, numattrs); + mss = build_mss(stats, bms_num_members(attrs), exprs); + + /* + * treat expressions as special attributes with high attnums + * + * XXX We do this after build_mss, because that expects the bitmapset + * to only contain simple attributes (with a matching VacAttrStats) + */ + attrs = add_expressions_to_attributes(attrs, exprs->nexprs); + + /* now build the array, with the special expression attnums */ + attnums = build_attnums_array(attrs, &numattrs); /* sort the rows */ - items = build_sorted_items(numrows, &nitems, rows, stats[0]->tupDesc, - mss, numattrs, attnums); + items = build_sorted_items(numrows, &nitems, rows, exprs, + stats[0]->tupDesc, mss, numattrs, attnums); if (!items) return NULL; @@ -338,6 +349,7 @@ statext_mcv_build(int numrows, HeapTuple *rows, Bitmapset *attrs, pfree(items); pfree(groups); + pfree(attrs); return mcvlist; } @@ -347,12 +359,12 @@ statext_mcv_build(int numrows, HeapTuple *rows, Bitmapset *attrs, * build MultiSortSupport for the attributes passed in attrs */ static MultiSortSupport -build_mss(VacAttrStats **stats, int numattrs) +build_mss(VacAttrStats **stats, int numattrs, ExprInfo *exprs) { int i; /* Sort by multiple columns (using array of SortSupport) */ - MultiSortSupport mss = multi_sort_init(numattrs); + MultiSortSupport mss = multi_sort_init(numattrs + exprs->nexprs); /* prepare the sort functions for all the attributes */ for (i = 0; i < numattrs; i++) @@ -368,6 +380,20 @@ build_mss(VacAttrStats **stats, int numattrs) multi_sort_add_dimension(mss, i, type->lt_opr, colstat->attrcollid); } + /* prepare the sort functions for all the expressions */ + for (i = 0; i < exprs->nexprs; i++) + { + TypeCacheEntry *type; + + type = lookup_type_cache(exprs->types[i], TYPECACHE_LT_OPR); + if (type->lt_opr == InvalidOid) /* shouldn't happen */ + elog(ERROR, "cache lookup failed for ordering operator for type %u", + exprs->types[i]); + + multi_sort_add_dimension(mss, numattrs + i, type->lt_opr, + exprs->collations[i]); + } + return mss; } @@ -1541,10 +1567,14 @@ pg_mcv_list_send(PG_FUNCTION_ARGS) * the size to ~1/8. It would also allow us to combine bitmaps simply using * & and |, which should be faster than min/max. The bitmaps are fairly * small, though (thanks to the cap on the MCV list size). + * + * XXX There's a lot of code duplication between branches for simple columns + * and complex expressions. We should refactor it somehow. */ static bool * mcv_get_match_bitmap(PlannerInfo *root, List *clauses, - Bitmapset *keys, MCVList *mcvlist, bool is_or) + Bitmapset *keys, List *exprs, + MCVList *mcvlist, bool is_or) { int i; ListCell *l; @@ -1584,8 +1614,10 @@ mcv_get_match_bitmap(PlannerInfo *root, List *clauses, /* valid only after examine_clause_args returns true */ Var *var; + Node *clause_expr; Const *cst; bool varonleft; + bool expronleft; fmgr_info(get_opcode(expr->opno), &opproc); @@ -1654,6 +1686,89 @@ mcv_get_match_bitmap(PlannerInfo *root, List *clauses, matches[i] = RESULT_MERGE(matches[i], is_or, match); } } + /* extract the expr and const from the expression */ + else if (examine_clause_args2(expr->args, &clause_expr, &cst, &expronleft)) + { + ListCell *lc; + int idx; + Oid collid = exprCollation(clause_expr); + + /* match the attribute to a dimension of the statistic */ + idx = bms_num_members(keys); + + foreach(lc, exprs) + { + Node *stat_expr = (Node *) lfirst(lc); + + if (equal(clause_expr, stat_expr)) + break; + + idx++; + } + + /* index should be valid */ + Assert((idx >= 0) && + (idx < bms_num_members(keys) + list_length(exprs))); + + /* + * Walk through the MCV items and evaluate the current clause. + * We can skip items that were already ruled out, and + * terminate if there are no remaining MCV items that might + * possibly match. + */ + for (i = 0; i < mcvlist->nitems; i++) + { + bool match = true; + MCVItem *item = &mcvlist->items[i]; + + /* + * When the MCV item or the Const value is NULL we can + * treat this as a mismatch. We must not call the operator + * because of strictness. + */ + if (item->isnull[idx] || cst->constisnull) + { + matches[i] = RESULT_MERGE(matches[i], is_or, false); + continue; + } + + /* + * Skip MCV items that can't change result in the bitmap. + * Once the value gets false for AND-lists, or true for + * OR-lists, we don't need to look at more clauses. + */ + if (RESULT_IS_FINAL(matches[i], is_or)) + continue; + + /* + * First check whether the constant is below the lower + * boundary (in that case we can skip the bucket, because + * there's no overlap). + * + * We don't store collations used to build the statistics, + * but we can use the collation for the attribute itself, + * as stored in varcollid. We do reset the statistics + * after a type change (including collation change), so + * this is OK. We may need to relax this after allowing + * extended statistics on expressions. + */ + if (expronleft) + match = DatumGetBool(FunctionCall2Coll(&opproc, + collid, + item->values[idx], + cst->constvalue)); + else + match = DatumGetBool(FunctionCall2Coll(&opproc, + collid, + cst->constvalue, + item->values[idx])); + + /* update the match bitmap with the result */ + matches[i] = RESULT_MERGE(matches[i], is_or, match); + } + } + else + elog(ERROR, "incompatible clause"); } else if (IsA(clause, ScalarArrayOpExpr)) { @@ -1662,8 +1777,10 @@ mcv_get_match_bitmap(PlannerInfo *root, List *clauses, /* valid only after examine_clause_args returns true */ Var *var; + Node *clause_expr; Const *cst; bool varonleft; + bool expronleft; fmgr_info(get_opcode(expr->opno), &opproc); @@ -1761,14 +1878,155 @@ mcv_get_match_bitmap(PlannerInfo *root, List *clauses, matches[i] = RESULT_MERGE(matches[i], is_or, match); } } + /* extract the expr and const from the expression */ + else if (examine_clause_args2(expr->args, &clause_expr, &cst, &expronleft)) + { + ListCell *lc; + int idx; + + ArrayType *arrayval; + int16 elmlen; + bool elmbyval; + char elmalign; + int num_elems; + Datum *elem_values; + bool *elem_nulls; + Oid collid = exprCollation(clause_expr); + + /* ScalarArrayOpExpr has the Var always on the left */ + Assert(expronleft); + + if (!cst->constisnull) + { + arrayval = DatumGetArrayTypeP(cst->constvalue); + get_typlenbyvalalign(ARR_ELEMTYPE(arrayval), + &elmlen, &elmbyval, &elmalign); + deconstruct_array(arrayval, + ARR_ELEMTYPE(arrayval), + elmlen, elmbyval, elmalign, + &elem_values, &elem_nulls, &num_elems); + } + + /* match the attribute to a dimension of the statistic */ + idx = bms_num_members(keys); + + foreach(lc, exprs) + { + Node *stat_expr = (Node *) lfirst(lc); + + if (equal(clause_expr, stat_expr)) + break; + + idx++; + } + + /* index should be valid */ + Assert((idx >= 0) && + (idx < bms_num_members(keys) + list_length(exprs))); + + /* + * Walk through the MCV items and evaluate the current clause. + * We can skip items that were already ruled out, and + * terminate if there are no remaining MCV items that might + * possibly match. + */ + for (i = 0; i < mcvlist->nitems; i++) + { + int j; + bool match = (expr->useOr ? false : true); + MCVItem *item = &mcvlist->items[i]; + + /* + * When the MCV item or the Const value is NULL we can + * treat this as a mismatch. We must not call the operator + * because of strictness. + */ + if (item->isnull[idx] || cst->constisnull) + { + matches[i] = RESULT_MERGE(matches[i], is_or, false); + continue; + } + + /* + * Skip MCV items that can't change result in the bitmap. + * Once the value gets false for AND-lists, or true for + * OR-lists, we don't need to look at more clauses. + */ + if (RESULT_IS_FINAL(matches[i], is_or)) + continue; + + for (j = 0; j < num_elems; j++) + { + Datum elem_value = elem_values[j]; + bool elem_isnull = elem_nulls[j]; + bool elem_match; + + /* NULL values always evaluate as not matching. */ + if (elem_isnull) + { + match = RESULT_MERGE(match, expr->useOr, false); + continue; + } + + /* + * Stop evaluating the array elements once we reach + * match value that can't change - ALL() is the same + * as AND-list, ANY() is the same as OR-list. + */ + if (RESULT_IS_FINAL(match, expr->useOr)) + break; + + elem_match = DatumGetBool(FunctionCall2Coll(&opproc, + collid, + item->values[idx], + elem_value)); + + match = RESULT_MERGE(match, expr->useOr, elem_match); + } + + /* update the match bitmap with the result */ + matches[i] = RESULT_MERGE(matches[i], is_or, match); + } + } + else + elog(ERROR, "incompatible clause"); } else if (IsA(clause, NullTest)) { NullTest *expr = (NullTest *) clause; - Var *var = (Var *) (expr->arg); + Node *clause_expr = (Node *) (expr->arg); /* match the attribute to a dimension of the statistic */ - int idx = bms_member_index(keys, var->varattno); + int idx = -1; + + if (IsA(clause_expr, Var)) + { + /* simple Var, so just lookup using varattno */ + Var *var = (Var *) clause_expr; + + idx = bms_member_index(keys, var->varattno); + } + else + { + ListCell *lc; + + /* expressions are after the simple columns */ + idx = bms_num_members(keys); + + /* expression - lookup in stats expressions */ + foreach(lc, exprs) + { + Node *stat_expr = (Node *) lfirst(lc); + + if (equal(clause_expr, stat_expr)) + break; + + idx++; + } + } + + /* index should be valid */ + Assert((idx >= 0) && (idx < bms_num_members(keys) + list_length(exprs))); /* * Walk through the MCV items and evaluate the current clause. We @@ -1811,7 +2069,7 @@ mcv_get_match_bitmap(PlannerInfo *root, List *clauses, Assert(list_length(bool_clauses) >= 2); /* build the match bitmap for the OR-clauses */ - bool_matches = mcv_get_match_bitmap(root, bool_clauses, keys, + bool_matches = mcv_get_match_bitmap(root, bool_clauses, keys, exprs, mcvlist, is_orclause(clause)); /* @@ -1839,7 +2097,7 @@ mcv_get_match_bitmap(PlannerInfo *root, List *clauses, Assert(list_length(not_args) == 1); /* build the match bitmap for the NOT-clause */ - not_matches = mcv_get_match_bitmap(root, not_args, keys, + not_matches = mcv_get_match_bitmap(root, not_args, keys, exprs, mcvlist, false); /* @@ -1982,7 +2240,8 @@ mcv_clauselist_selectivity(PlannerInfo *root, StatisticExtInfo *stat, mcv = statext_mcv_load(stat->statOid); /* build a match bitmap for the clauses */ - matches = mcv_get_match_bitmap(root, clauses, stat->keys, mcv, false); + matches = mcv_get_match_bitmap(root, clauses, stat->keys, stat->exprs, + mcv, false); /* sum frequencies for all the matching MCV items */ *basesel = 0.0; @@ -2056,7 +2315,7 @@ mcv_clause_selectivity_or(PlannerInfo *root, StatisticExtInfo *stat, /* build the match bitmap for the new clause */ new_matches = mcv_get_match_bitmap(root, list_make1(clause), stat->keys, - mcv, false); + stat->exprs, mcv, false); /* * Sum the frequencies for all the MCV items matching this clause and also diff --git a/src/backend/statistics/mvdistinct.c b/src/backend/statistics/mvdistinct.c index 4b86f0ab2d..552d755ab4 100644 --- a/src/backend/statistics/mvdistinct.c +++ b/src/backend/statistics/mvdistinct.c @@ -37,7 +37,8 @@ #include "utils/typcache.h" static double ndistinct_for_combination(double totalrows, int numrows, - HeapTuple *rows, VacAttrStats **stats, + HeapTuple *rows, ExprInfo *exprs, + int nattrs, VacAttrStats **stats, int k, int *combination); static double estimate_ndistinct(double totalrows, int numrows, int d, int f1); static int n_choose_k(int n, int k); @@ -81,16 +82,21 @@ static void generate_combinations(CombinationGenerator *state); * * This computes the ndistinct estimate using the same estimator used * in analyze.c and then computes the coefficient. + * + * To handle expressions easily, we treat them as special attributes with + * attnums above MaxHeapAttributeNumber, and we assume the expressions are + * placed after all simple attributes. */ MVNDistinct * statext_ndistinct_build(double totalrows, int numrows, HeapTuple *rows, - Bitmapset *attrs, VacAttrStats **stats) + ExprInfo *exprs, Bitmapset *attrs, + VacAttrStats **stats) { MVNDistinct *result; int k; int itemcnt; int numattrs = bms_num_members(attrs); - int numcombs = num_combinations(numattrs); + int numcombs = num_combinations(numattrs + exprs->nexprs); result = palloc(offsetof(MVNDistinct, items) + numcombs * sizeof(MVNDistinctItem)); @@ -98,14 +104,20 @@ statext_ndistinct_build(double totalrows, int numrows, HeapTuple *rows, result->type = STATS_NDISTINCT_TYPE_BASIC; result->nitems = numcombs; + /* treat expressions as special attributes with high attnums */ + attrs = add_expressions_to_attributes(attrs, exprs->nexprs); + + /* make sure there were no clashes */ + Assert(bms_num_members(attrs) == numattrs + exprs->nexprs); + itemcnt = 0; - for (k = 2; k <= numattrs; k++) + for (k = 2; k <= bms_num_members(attrs); k++) { int *combination; CombinationGenerator *generator; /* generate combinations of K out of N elements */ - generator = generator_init(numattrs, k); + generator = generator_init(bms_num_members(attrs), k); while ((combination = generator_next(generator))) { @@ -114,10 +126,32 @@ statext_ndistinct_build(double totalrows, int numrows, HeapTuple *rows, item->attrs = NULL; for (j = 0; j < k; j++) - item->attrs = bms_add_member(item->attrs, - stats[combination[j]]->attr->attnum); + { + AttrNumber attnum = InvalidAttrNumber; + + /* + * The simple attributes are before expressions, so have + * indexes below numattrs. + * */ + if (combination[j] < numattrs) + attnum = stats[combination[j]]->attr->attnum; + else + { + /* make sure the expression index is valid */ + Assert((combination[j] - numattrs) >= 0); + Assert((combination[j] - numattrs) < exprs->nexprs); + + attnum = EXPRESSION_ATTNUM(combination[j] - numattrs); + } + + Assert(attnum != InvalidAttrNumber); + + item->attrs = bms_add_member(item->attrs, attnum); + } + item->ndistinct = ndistinct_for_combination(totalrows, numrows, rows, + exprs, numattrs, stats, k, combination); itemcnt++; @@ -428,6 +462,7 @@ pg_ndistinct_send(PG_FUNCTION_ARGS) */ static double ndistinct_for_combination(double totalrows, int numrows, HeapTuple *rows, + ExprInfo *exprs, int nattrs, VacAttrStats **stats, int k, int *combination) { int i, @@ -467,25 +502,57 @@ ndistinct_for_combination(double totalrows, int numrows, HeapTuple *rows, */ for (i = 0; i < k; i++) { - VacAttrStats *colstat = stats[combination[i]]; + Oid typid; TypeCacheEntry *type; + AttrNumber attnum = InvalidAttrNumber; + TupleDesc tdesc = NULL; + Oid collid = InvalidOid; + + if (combination[i] < nattrs) + { + VacAttrStats *colstat = stats[combination[i]]; + typid = colstat->attrtypid; + attnum = colstat->attr->attnum; + collid = colstat->attrcollid; + tdesc = colstat->tupDesc; + } + else + { + typid = exprs->types[combination[i] - nattrs]; + collid = exprs->collations[combination[i] - nattrs]; + } - type = lookup_type_cache(colstat->attrtypid, TYPECACHE_LT_OPR); + type = lookup_type_cache(typid, TYPECACHE_LT_OPR); if (type->lt_opr == InvalidOid) /* shouldn't happen */ elog(ERROR, "cache lookup failed for ordering operator for type %u", - colstat->attrtypid); + typid); /* prepare the sort function for this dimension */ - multi_sort_add_dimension(mss, i, type->lt_opr, colstat->attrcollid); + multi_sort_add_dimension(mss, i, type->lt_opr, collid); /* accumulate all the data for this dimension into the arrays */ for (j = 0; j < numrows; j++) { - items[j].values[i] = - heap_getattr(rows[j], - colstat->attr->attnum, - colstat->tupDesc, - &items[j].isnull[i]); + /* + * The first nattrs indexes identify simple attributes, higher + * indexes are expressions. + */ + if (combination[i] < nattrs) + items[j].values[i] = + heap_getattr(rows[j], + attnum, + tdesc, + &items[j].isnull[i]); + else + { + int idx = (combination[i] - nattrs); + + /* make sure the expression index is valid */ + Assert((idx >= 0) && (idx < exprs->nexprs)); + + items[j].values[i] = exprs->values[idx][j]; + items[j].isnull[i] = exprs->nulls[idx][j]; + } } } diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c index a42ead7d69..1dfd004376 100644 --- a/src/backend/tcop/utility.c +++ b/src/backend/tcop/utility.c @@ -1834,7 +1834,22 @@ ProcessUtilitySlow(ParseState *pstate, break; case T_CreateStatsStmt: - address = CreateStatistics((CreateStatsStmt *) parsetree); + { + Oid relid; + CreateStatsStmt *stmt = (CreateStatsStmt *) parsetree; + RangeVar *rel = (RangeVar *) linitial(stmt->relations); + + /* + * XXX RangeVarCallbackOwnsRelation not needed needed here, + * to keep the same behavior as before. + */ + relid = RangeVarGetRelid(rel, ShareLock, false); + + /* Run parse analysis ... */ + stmt = transformStatsStmt(relid, stmt, queryString); + + address = CreateStatistics(stmt); + } break; case T_AlterStatsStmt: diff --git a/src/backend/utils/adt/ruleutils.c b/src/backend/utils/adt/ruleutils.c index c2c6df2a4f..f3c0060124 100644 --- a/src/backend/utils/adt/ruleutils.c +++ b/src/backend/utils/adt/ruleutils.c @@ -337,7 +337,8 @@ static char *pg_get_indexdef_worker(Oid indexrelid, int colno, bool attrsOnly, bool keysOnly, bool showTblSpc, bool inherits, int prettyFlags, bool missing_ok); -static char *pg_get_statisticsobj_worker(Oid statextid, bool missing_ok); +static char *pg_get_statisticsobj_worker(Oid statextid, bool columns_only, + bool missing_ok); static char *pg_get_partkeydef_worker(Oid relid, int prettyFlags, bool attrsOnly, bool missing_ok); static char *pg_get_constraintdef_worker(Oid constraintId, bool fullCommand, @@ -1508,7 +1509,26 @@ pg_get_statisticsobjdef(PG_FUNCTION_ARGS) Oid statextid = PG_GETARG_OID(0); char *res; - res = pg_get_statisticsobj_worker(statextid, true); + res = pg_get_statisticsobj_worker(statextid, false, true); + + if (res == NULL) + PG_RETURN_NULL(); + + PG_RETURN_TEXT_P(string_to_text(res)); +} + + +/* + * pg_get_statisticsobjdef_columns + * Get columns and expressions for an extended statistics object + */ +Datum +pg_get_statisticsobjdef_columns(PG_FUNCTION_ARGS) +{ + Oid statextid = PG_GETARG_OID(0); + char *res; + + res = pg_get_statisticsobj_worker(statextid, true, true); if (res == NULL) PG_RETURN_NULL(); @@ -1520,7 +1540,7 @@ pg_get_statisticsobjdef(PG_FUNCTION_ARGS) * Internal workhorse to decompile an extended statistics object. */ static char * -pg_get_statisticsobj_worker(Oid statextid, bool missing_ok) +pg_get_statisticsobj_worker(Oid statextid, bool columns_only, bool missing_ok) { Form_pg_statistic_ext statextrec; HeapTuple statexttup; @@ -1534,7 +1554,12 @@ pg_get_statisticsobj_worker(Oid statextid, bool missing_ok) bool ndistinct_enabled; bool dependencies_enabled; bool mcv_enabled; + bool exprs_enabled; int i; + List *context; + ListCell *lc; + List *exprs = NIL; + bool has_exprs; statexttup = SearchSysCache1(STATEXTOID, ObjectIdGetDatum(statextid)); @@ -1545,75 +1570,91 @@ pg_get_statisticsobj_worker(Oid statextid, bool missing_ok) elog(ERROR, "cache lookup failed for statistics object %u", statextid); } + /* has the statistics expressions? */ + has_exprs = !heap_attisnull(statexttup, Anum_pg_statistic_ext_stxexprs, NULL); + statextrec = (Form_pg_statistic_ext) GETSTRUCT(statexttup); initStringInfo(&buf); - nsp = get_namespace_name(statextrec->stxnamespace); - appendStringInfo(&buf, "CREATE STATISTICS %s", - quote_qualified_identifier(nsp, - NameStr(statextrec->stxname))); + if (!columns_only) + { + nsp = get_namespace_name(statextrec->stxnamespace); + appendStringInfo(&buf, "CREATE STATISTICS %s", + quote_qualified_identifier(nsp, + NameStr(statextrec->stxname))); - /* - * Decode the stxkind column so that we know which stats types to print. - */ - datum = SysCacheGetAttr(STATEXTOID, statexttup, - Anum_pg_statistic_ext_stxkind, &isnull); - Assert(!isnull); - arr = DatumGetArrayTypeP(datum); - if (ARR_NDIM(arr) != 1 || - ARR_HASNULL(arr) || - ARR_ELEMTYPE(arr) != CHAROID) - elog(ERROR, "stxkind is not a 1-D char array"); - enabled = (char *) ARR_DATA_PTR(arr); + /* + * Decode the stxkind column so that we know which stats types to print. + */ + datum = SysCacheGetAttr(STATEXTOID, statexttup, + Anum_pg_statistic_ext_stxkind, &isnull); + Assert(!isnull); + arr = DatumGetArrayTypeP(datum); + if (ARR_NDIM(arr) != 1 || + ARR_HASNULL(arr) || + ARR_ELEMTYPE(arr) != CHAROID) + elog(ERROR, "stxkind is not a 1-D char array"); + enabled = (char *) ARR_DATA_PTR(arr); + + ndistinct_enabled = false; + dependencies_enabled = false; + mcv_enabled = false; + exprs_enabled = false; + + for (i = 0; i < ARR_DIMS(arr)[0]; i++) + { + if (enabled[i] == STATS_EXT_NDISTINCT) + ndistinct_enabled = true; + if (enabled[i] == STATS_EXT_DEPENDENCIES) + dependencies_enabled = true; + if (enabled[i] == STATS_EXT_MCV) + mcv_enabled = true; + if (enabled[i] == STATS_EXT_EXPRESSIONS) + exprs_enabled = true; + } - ndistinct_enabled = false; - dependencies_enabled = false; - mcv_enabled = false; + /* + * If any option is disabled, then we'll need to append the types clause + * to show which options are enabled. We omit the types clause on purpose + * when all options are enabled, so a pg_dump/pg_restore will create all + * statistics types on a newer postgres version, if the statistics had all + * options enabled on the original version. + */ + if (!ndistinct_enabled || !dependencies_enabled || !mcv_enabled || (!exprs_enabled && has_exprs)) + { + bool gotone = false; - for (i = 0; i < ARR_DIMS(arr)[0]; i++) - { - if (enabled[i] == STATS_EXT_NDISTINCT) - ndistinct_enabled = true; - if (enabled[i] == STATS_EXT_DEPENDENCIES) - dependencies_enabled = true; - if (enabled[i] == STATS_EXT_MCV) - mcv_enabled = true; - } + appendStringInfoString(&buf, " ("); - /* - * If any option is disabled, then we'll need to append the types clause - * to show which options are enabled. We omit the types clause on purpose - * when all options are enabled, so a pg_dump/pg_restore will create all - * statistics types on a newer postgres version, if the statistics had all - * options enabled on the original version. - */ - if (!ndistinct_enabled || !dependencies_enabled || !mcv_enabled) - { - bool gotone = false; + if (ndistinct_enabled) + { + appendStringInfoString(&buf, "ndistinct"); + gotone = true; + } - appendStringInfoString(&buf, " ("); + if (dependencies_enabled) + { + appendStringInfo(&buf, "%sdependencies", gotone ? ", " : ""); + gotone = true; + } - if (ndistinct_enabled) - { - appendStringInfoString(&buf, "ndistinct"); - gotone = true; - } + if (mcv_enabled) + { + appendStringInfo(&buf, "%smcv", gotone ? ", " : ""); + gotone = true; + } - if (dependencies_enabled) - { - appendStringInfo(&buf, "%sdependencies", gotone ? ", " : ""); - gotone = true; - } + if (exprs_enabled) + appendStringInfo(&buf, "%sexpressions", gotone ? ", " : ""); - if (mcv_enabled) - appendStringInfo(&buf, "%smcv", gotone ? ", " : ""); + appendStringInfoChar(&buf, ')'); + } - appendStringInfoChar(&buf, ')'); + appendStringInfoString(&buf, " ON "); } - appendStringInfoString(&buf, " ON "); - + /* decode simple column references */ for (colno = 0; colno < statextrec->stxkeys.dim1; colno++) { AttrNumber attnum = statextrec->stxkeys.values[colno]; @@ -1627,14 +1668,150 @@ pg_get_statisticsobj_worker(Oid statextid, bool missing_ok) appendStringInfoString(&buf, quote_identifier(attname)); } - appendStringInfo(&buf, " FROM %s", - generate_relation_name(statextrec->stxrelid, NIL)); + /* + * Get the statistics expressions, if any. (NOTE: we do not use the + * relcache versions of the expressions and predicate, because we want + * to display non-const-folded expressions.) + */ + if (has_exprs) + { + Datum exprsDatum; + bool isnull; + char *exprsString; + + exprsDatum = SysCacheGetAttr(STATEXTOID, statexttup, + Anum_pg_statistic_ext_stxexprs, &isnull); + Assert(!isnull); + exprsString = TextDatumGetCString(exprsDatum); + exprs = (List *) stringToNode(exprsString); + pfree(exprsString); + + /* + * Run the expressions through eval_const_expressions. This is not just an + * optimization, but is necessary, because the planner will be comparing + * them to similarly-processed qual clauses, and may fail to detect valid + * matches without this. We must not use canonicalize_qual, however, + * since these aren't qual expressions. + * + * XXX Not sure if this is really needed, it's not in pg_get_indexdef. In + * fact the comment above suggests we don't want const-folding here. + */ + // exprs = (List *) eval_const_expressions(NULL, (Node *) exprs); + + /* + * May as well fix opfuncids too + * + * XXX Same here. Is this something we want/need? + */ + // fix_opfuncids((Node *) exprs); + + } + else + exprs = NIL; + + context = deparse_context_for(get_relation_name(statextrec->stxrelid), + statextrec->stxrelid); + + foreach (lc, exprs) + { + Node *expr = (Node *) lfirst(lc); + char *str; + int prettyFlags = PRETTYFLAG_INDENT; + + str = deparse_expression_pretty(expr, context, false, false, + prettyFlags, 0); + + if (colno > 0) + appendStringInfoString(&buf, ", "); + + /* Need parens if it's not a bare function call */ + if (looks_like_function(expr)) + appendStringInfoString(&buf, str); + else + appendStringInfo(&buf, "(%s)", str); + + colno++; + } + + if (!columns_only) + appendStringInfo(&buf, " FROM %s", + generate_relation_name(statextrec->stxrelid, NIL)); ReleaseSysCache(statexttup); return buf.data; } +/* + * Generate text array of expressions for statistics object. + */ +Datum +pg_get_statisticsobjdef_expressions(PG_FUNCTION_ARGS) +{ + Oid statextid = PG_GETARG_OID(0); + Form_pg_statistic_ext statextrec; + HeapTuple statexttup; + Datum datum; + bool isnull; + List *context; + ListCell *lc; + List *exprs = NIL; + bool has_exprs; + char *tmp; + ArrayBuildState *astate = NULL; + + statexttup = SearchSysCache1(STATEXTOID, ObjectIdGetDatum(statextid)); + + if (!HeapTupleIsValid(statexttup)) + elog(ERROR, "cache lookup failed for statistics object %u", statextid); + + /* has the statistics expressions? */ + has_exprs = !heap_attisnull(statexttup, Anum_pg_statistic_ext_stxexprs, NULL); + + /* no expressions? we're done */ + if (!has_exprs) + { + ReleaseSysCache(statexttup); + PG_RETURN_NULL(); + } + + statextrec = (Form_pg_statistic_ext) GETSTRUCT(statexttup); + + /* + * Get the statistics expressions, and deparse them into text values. + */ + datum = SysCacheGetAttr(STATEXTOID, statexttup, + Anum_pg_statistic_ext_stxexprs, &isnull); + + Assert(!isnull); + tmp = TextDatumGetCString(datum); + exprs = (List *) stringToNode(tmp); + pfree(tmp); + + context = deparse_context_for(get_relation_name(statextrec->stxrelid), + statextrec->stxrelid); + + foreach (lc, exprs) + { + Node *expr = (Node *) lfirst(lc); + char *str; + int prettyFlags = PRETTYFLAG_INDENT; + + str = deparse_expression_pretty(expr, context, false, false, + prettyFlags, 0); + + astate = accumArrayResult(astate, + PointerGetDatum(cstring_to_text(str)), + false, + TEXTOID, + CurrentMemoryContext); + } + + ReleaseSysCache(statexttup); + + PG_RETURN_DATUM(makeArrayResult(astate, CurrentMemoryContext)); +} + /* * pg_get_partkeydef * diff --git a/src/backend/utils/adt/selfuncs.c b/src/backend/utils/adt/selfuncs.c index 80bd60f876..1a09f18ce1 100644 --- a/src/backend/utils/adt/selfuncs.c +++ b/src/backend/utils/adt/selfuncs.c @@ -3291,6 +3291,88 @@ add_unique_group_var(PlannerInfo *root, List *varinfos, return varinfos; } +/* + * Helper routine for estimate_num_groups: add an item to a list of + * GrouExprInfos, but only if it's not known equal to any of the existing + * entries. + */ +typedef struct +{ + Node *expr; /* expression */ + RelOptInfo *rel; /* relation it belongs to */ + List *varinfos; /* info for variables in this expression */ +} GroupExprInfo; + +static List * +add_unique_group_expr(PlannerInfo *root, List *exprinfos, + Node *expr, List *vars) +{ + GroupExprInfo *exprinfo; + ListCell *lc; + Bitmapset *varnos; + Index varno; + + foreach(lc, exprinfos) + { + exprinfo = (GroupExprInfo *) lfirst(lc); + + /* Drop exact duplicates */ + if (equal(expr, exprinfo->expr)) + return exprinfos; + } + + exprinfo = (GroupExprInfo *) palloc(sizeof(GroupExprInfo)); + + varnos = pull_varnos(expr); + + /* + * Expressions with vars from multiple relations should never get + * here, as we split them to vars. + */ + Assert(bms_num_members(varnos) == 1); + + varno = bms_singleton_member(varnos); + + exprinfo->expr = expr; + exprinfo->varinfos = NIL; + exprinfo->rel = root->simple_rel_array[varno]; + + Assert(exprinfo->rel); + + /* Track vars for this expression. */ + foreach (lc, vars) + { + VariableStatData vardata; + Node *var = (Node *) lfirst(lc); + + /* can we get no vardata for the variable? */ + examine_variable(root, var, 0, &vardata); + + exprinfo->varinfos + = add_unique_group_var(root, exprinfo->varinfos, var, &vardata); + + ReleaseVariableStats(vardata); + } + + /* without a list of variables, use the expression itself */ + if (vars == NIL) + { + VariableStatData vardata; + + /* can we get no vardata for the variable? */ + examine_variable(root, expr, 0, &vardata); + + exprinfo->varinfos + = add_unique_group_var(root, exprinfo->varinfos, + expr, &vardata); + + ReleaseVariableStats(vardata); + } + + return lappend(exprinfos, exprinfo); +} + + /* * estimate_num_groups - Estimate number of groups in a grouped query * @@ -3360,7 +3442,7 @@ double estimate_num_groups(PlannerInfo *root, List *groupExprs, double input_rows, List **pgset) { - List *varinfos = NIL; + List *exprinfos = NIL; double srf_multiplier = 1.0; double numdistinct; ListCell *l; @@ -3398,6 +3480,7 @@ estimate_num_groups(PlannerInfo *root, List *groupExprs, double input_rows, double this_srf_multiplier; VariableStatData vardata; List *varshere; + Relids varnos; ListCell *l2; /* is expression in this grouping set? */ @@ -3434,8 +3517,9 @@ estimate_num_groups(PlannerInfo *root, List *groupExprs, double input_rows, examine_variable(root, groupexpr, 0, &vardata); if (HeapTupleIsValid(vardata.statsTuple) || vardata.isunique) { - varinfos = add_unique_group_var(root, varinfos, - groupexpr, &vardata); + exprinfos = add_unique_group_expr(root, exprinfos, + groupexpr, NIL); + ReleaseVariableStats(vardata); continue; } @@ -3465,6 +3549,19 @@ estimate_num_groups(PlannerInfo *root, List *groupExprs, double input_rows, continue; } + /* + * Are all the variables from the same relation? If yes, search for + * an extended statistic matching this expression exactly. + */ + varnos = pull_varnos((Node *) varshere); + if (bms_membership(varnos) == BMS_SINGLETON) + { + exprinfos = add_unique_group_expr(root, exprinfos, + groupexpr, + varshere); + continue; + } + /* * Else add variables to varinfos list */ @@ -3472,9 +3569,8 @@ estimate_num_groups(PlannerInfo *root, List *groupExprs, double input_rows, { Node *var = (Node *) lfirst(l2); - examine_variable(root, var, 0, &vardata); - varinfos = add_unique_group_var(root, varinfos, var, &vardata); - ReleaseVariableStats(vardata); + exprinfos = add_unique_group_expr(root, exprinfos, + var, NIL); } } @@ -3482,7 +3578,7 @@ estimate_num_groups(PlannerInfo *root, List *groupExprs, double input_rows, * If now no Vars, we must have an all-constant or all-boolean GROUP BY * list. */ - if (varinfos == NIL) + if (exprinfos == NIL) { /* Apply SRF multiplier as we would do in the long path */ numdistinct *= srf_multiplier; @@ -3506,32 +3602,32 @@ estimate_num_groups(PlannerInfo *root, List *groupExprs, double input_rows, */ do { - GroupVarInfo *varinfo1 = (GroupVarInfo *) linitial(varinfos); - RelOptInfo *rel = varinfo1->rel; + GroupExprInfo *exprinfo1 = (GroupExprInfo *) linitial(exprinfos); + RelOptInfo *rel = exprinfo1->rel; double reldistinct = 1; double relmaxndistinct = reldistinct; int relvarcount = 0; - List *newvarinfos = NIL; - List *relvarinfos = NIL; + List *newexprinfos = NIL; + List *relexprinfos = NIL; /* * Split the list of varinfos in two - one for the current rel, one * for remaining Vars on other rels. */ - relvarinfos = lappend(relvarinfos, varinfo1); - for_each_from(l, varinfos, 1) + relexprinfos = lappend(relexprinfos, exprinfo1); + for_each_from(l, exprinfos, 1) { - GroupVarInfo *varinfo2 = (GroupVarInfo *) lfirst(l); + GroupExprInfo *exprinfo2 = (GroupExprInfo *) lfirst(l); - if (varinfo2->rel == varinfo1->rel) + if (exprinfo2->rel == exprinfo1->rel) { /* varinfos on current rel */ - relvarinfos = lappend(relvarinfos, varinfo2); + relexprinfos = lappend(relexprinfos, exprinfo2); } else { - /* not time to process varinfo2 yet */ - newvarinfos = lappend(newvarinfos, varinfo2); + /* not time to process exprinfo2 yet */ + newexprinfos = lappend(newexprinfos, exprinfo2); } } @@ -3547,11 +3643,11 @@ estimate_num_groups(PlannerInfo *root, List *groupExprs, double input_rows, * apply. We apply a fudge factor below, but only if we multiplied * more than one such values. */ - while (relvarinfos) + while (relexprinfos) { double mvndistinct; - if (estimate_multivariate_ndistinct(root, rel, &relvarinfos, + if (estimate_multivariate_ndistinct(root, rel, &relexprinfos, &mvndistinct)) { reldistinct *= mvndistinct; @@ -3561,18 +3657,24 @@ estimate_num_groups(PlannerInfo *root, List *groupExprs, double input_rows, } else { - foreach(l, relvarinfos) + foreach(l, relexprinfos) { - GroupVarInfo *varinfo2 = (GroupVarInfo *) lfirst(l); + ListCell *lc; + GroupExprInfo *exprinfo2 = (GroupExprInfo *) lfirst(l); + + foreach (lc, exprinfo2->varinfos) + { + GroupVarInfo *varinfo2 = (GroupVarInfo *) lfirst(lc); - reldistinct *= varinfo2->ndistinct; - if (relmaxndistinct < varinfo2->ndistinct) - relmaxndistinct = varinfo2->ndistinct; - relvarcount++; + reldistinct *= varinfo2->ndistinct; + if (relmaxndistinct < varinfo2->ndistinct) + relmaxndistinct = varinfo2->ndistinct; + relvarcount++; + } } /* we're done with this relation */ - relvarinfos = NIL; + relexprinfos = NIL; } } @@ -3658,8 +3760,8 @@ estimate_num_groups(PlannerInfo *root, List *groupExprs, double input_rows, numdistinct *= reldistinct; } - varinfos = newvarinfos; - } while (varinfos != NIL); + exprinfos = newexprinfos; + } while (exprinfos != NIL); /* Now we can account for the effects of any SRFs */ numdistinct *= srf_multiplier; @@ -3877,53 +3979,75 @@ estimate_hashagg_tablesize(PlannerInfo *root, Path *path, */ static bool estimate_multivariate_ndistinct(PlannerInfo *root, RelOptInfo *rel, - List **varinfos, double *ndistinct) + List **exprinfos, double *ndistinct) { ListCell *lc; - Bitmapset *attnums = NULL; - int nmatches; + int nmatches_vars; + int nmatches_exprs; Oid statOid = InvalidOid; MVNDistinct *stats; - Bitmapset *matched = NULL; + StatisticExtInfo *matched_info = NULL; /* bail out immediately if the table has no extended statistics */ if (!rel->statlist) return false; - /* Determine the attnums we're looking for */ - foreach(lc, *varinfos) - { - GroupVarInfo *varinfo = (GroupVarInfo *) lfirst(lc); - AttrNumber attnum; - - Assert(varinfo->rel == rel); - - if (!IsA(varinfo->var, Var)) - continue; - - attnum = ((Var *) varinfo->var)->varattno; - - if (!AttrNumberIsForUserDefinedAttr(attnum)) - continue; - - attnums = bms_add_member(attnums, attnum); - } - /* look for the ndistinct statistics matching the most vars */ - nmatches = 1; /* we require at least two matches */ + nmatches_vars = 0; /* we require at least two matches */ + nmatches_exprs = 0; foreach(lc, rel->statlist) { + ListCell *lc2; StatisticExtInfo *info = (StatisticExtInfo *) lfirst(lc); - Bitmapset *shared; - int nshared; + int nshared_vars = 0; + int nshared_exprs = 0; /* skip statistics of other kinds */ if (info->kind != STATS_EXT_NDISTINCT) continue; - /* compute attnums shared by the vars and the statistics object */ - shared = bms_intersect(info->keys, attnums); - nshared = bms_num_members(shared); + /* + * Determine how many expressions (and variables in non-matched + * expressions) match. + */ + foreach(lc2, *exprinfos) + { + ListCell *lc3; + GroupExprInfo *exprinfo = (GroupExprInfo *) lfirst(lc2); + AttrNumber attnum; + + Assert(exprinfo->rel == rel); + + /* simple Var, search in statistics keys directly */ + if (IsA(exprinfo->expr, Var)) + { + attnum = ((Var *) exprinfo->expr)->varattno; + + if (!AttrNumberIsForUserDefinedAttr(attnum)) + continue; + + if (bms_is_member(attnum, info->keys)) + nshared_vars++; + + continue; + } + + /* expression - see if it's in the statistics */ + foreach (lc3, info->exprs) + { + Node *expr = (Node *) lfirst(lc3); + + if (equal(exprinfo->expr, expr)) + { + nshared_exprs++; + nshared_vars += list_length(exprinfo->varinfos); + break; + } + } + } + + if (nshared_vars + nshared_exprs < 2) + continue; /* * Does this statistics object match more columns than the currently @@ -3932,18 +4056,21 @@ estimate_multivariate_ndistinct(PlannerInfo *root, RelOptInfo *rel, * XXX This should break ties using name of the object, or something * like that, to make the outcome stable. */ - if (nshared > nmatches) + if ((nshared_vars > nmatches_vars) || + ((nshared_vars == nmatches_vars) && (nshared_exprs > nmatches_exprs))) { statOid = info->statOid; - nmatches = nshared; - matched = shared; + nmatches_vars = nshared_vars; + nmatches_exprs = nshared_exprs; + matched_info = info; } } /* No match? */ if (statOid == InvalidOid) return false; - Assert(nmatches > 1 && matched != NULL); + + Assert(nmatches_vars + nmatches_exprs > 1); stats = statext_ndistinct_load(statOid); @@ -3956,6 +4083,56 @@ estimate_multivariate_ndistinct(PlannerInfo *root, RelOptInfo *rel, int i; List *newlist = NIL; MVNDistinctItem *item = NULL; + ListCell *lc2; + Bitmapset *matched = NULL; + + /* see what actually matched */ + foreach (lc2, *exprinfos) + { + ListCell *lc3; + int idx; + bool found = false; + + GroupExprInfo *exprinfo = (GroupExprInfo *) lfirst(lc2); + + /* expression - see if it's in the statistics */ + idx = 0; + foreach (lc3, matched_info->exprs) + { + Node *expr = (Node *) lfirst(lc3); + + idx++; + + if (equal(exprinfo->expr, expr)) + { + matched = bms_add_member(matched, MaxHeapAttributeNumber + idx); + found = true; + break; + } + } + + if (found) + continue; + + foreach (lc3, exprinfo->varinfos) + { + GroupVarInfo *varinfo = (GroupVarInfo *) lfirst(lc2); + + /* simple Var, search in statistics keys directly */ + if (IsA(varinfo->var, Var)) + { + AttrNumber attnum = ((Var *) varinfo->var)->varattno; + + if (!AttrNumberIsForUserDefinedAttr(attnum)) + continue; + + if (!bms_is_member(attnum, matched_info->keys)) + continue; + + matched = bms_add_member(matched, attnum); + } + } + } /* Find the specific item that exactly matches the combination */ for (i = 0; i < stats->nitems; i++) @@ -3973,28 +4150,49 @@ estimate_multivariate_ndistinct(PlannerInfo *root, RelOptInfo *rel, if (!item) elog(ERROR, "corrupt MVNDistinct entry"); - /* Form the output varinfo list, keeping only unmatched ones */ - foreach(lc, *varinfos) + /* Form the output exprinfo list, keeping only unmatched ones */ + foreach(lc, *exprinfos) { - GroupVarInfo *varinfo = (GroupVarInfo *) lfirst(lc); + GroupExprInfo *exprinfo = (GroupExprInfo *) lfirst(lc); AttrNumber attnum; + ListCell *lc3; + bool found = false; + + foreach (lc3, matched_info->exprs) + { + Node *expr = (Node *) lfirst(lc3); + + if (equal(exprinfo->expr, expr)) + { + found = true; + break; + } + } + + /* the whole expression was matched, so skip it */ + if (found) + continue; - if (!IsA(varinfo->var, Var)) + if (!IsA(exprinfo->expr, Var)) { - newlist = lappend(newlist, varinfo); + /* + * FIXME Probably should remove varinfos that match the + * selected MVNDistinct item. + */ + newlist = lappend(newlist, exprinfo); continue; } - attnum = ((Var *) varinfo->var)->varattno; + attnum = ((Var *) exprinfo->expr)->varattno; if (!AttrNumberIsForUserDefinedAttr(attnum)) continue; if (!bms_is_member(attnum, matched)) - newlist = lappend(newlist, varinfo); + newlist = lappend(newlist, exprinfo); } - *varinfos = newlist; + *exprinfos = newlist; *ndistinct = item->ndistinct; return true; } @@ -4690,6 +4888,13 @@ get_join_variables(PlannerInfo *root, List *args, SpecialJoinInfo *sjinfo, *join_is_reversed = false; } +/* statext_expressions_load copies the tuple, so just pfree it. */ +static void +ReleaseDummy(HeapTuple tuple) +{ + pfree(tuple); +} + /* * examine_variable * Try to look up statistical data about an expression. @@ -4830,6 +5035,7 @@ examine_variable(PlannerInfo *root, Node *node, int varRelid, * operator we are estimating for. FIXME later. */ ListCell *ilist; + ListCell *slist; foreach(ilist, onerel->indexlist) { @@ -4986,6 +5192,67 @@ examine_variable(PlannerInfo *root, Node *node, int varRelid, if (vardata->statsTuple) break; } + + /* + * Search extended statistics for one with a matching expression. + * There might be multiple ones, so just grab the first one. In + * the future, we might consider + */ + foreach(slist, onerel->statlist) + { + StatisticExtInfo *info = (StatisticExtInfo *) lfirst(slist); + ListCell *expr_item; + int pos; + + /* + * Stop once we've found statistics for the expression (either + * from extended stats, or for an index in the preceding loop). + */ + if (vardata->statsTuple) + break; + + /* skip stats without per-expression stats */ + if (info->kind != STATS_EXT_EXPRESSIONS) + continue; + + pos = 0; + foreach (expr_item, info->exprs) + { + Node *expr = (Node *) lfirst(expr_item); + + Assert(expr); + + /* strip RelabelType before comparing it */ + if (expr && IsA(expr, RelabelType)) + expr = (Node *) ((RelabelType *) expr)->arg; + + /* found a match, see if we can extract pg_statistic row */ + if (equal(node, expr)) + { + HeapTuple t = statext_expressions_load(info->statOid, pos); + + vardata->statsTuple = t; + + /* + * FIXME not sure if we should cache the tuple somewhere? + * It's stored in a cached tuple in the "data" catalog, + * and we just create a new copy every time. + */ + vardata->freefunc = ReleaseDummy; + + /* + * FIXME Hack to make statistic_proc_security_check happy, + * so that this does not get rejected. Probably needs more + * thought, just a hack. + */ + vardata->acl_ok = true; + + break; + } + + pos++; + } + } } } diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 14150d05a9..2d966136ac 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -2680,18 +2680,20 @@ describeOneTableDetails(const char *schemaname, /* print any extended statistics */ if (pset.sversion >= 100000) { + /* + * FIXME this needs to be version-dependent, because older + * versions don't have pg_get_statisticsobjdef_columns. + */ printfPQExpBuffer(&buf, "SELECT oid, " "stxrelid::pg_catalog.regclass, " "stxnamespace::pg_catalog.regnamespace AS nsp, " "stxname,\n" - " (SELECT pg_catalog.string_agg(pg_catalog.quote_ident(attname),', ')\n" - " FROM pg_catalog.unnest(stxkeys) s(attnum)\n" - " JOIN pg_catalog.pg_attribute a ON (stxrelid = a.attrelid AND\n" - " a.attnum = s.attnum AND NOT attisdropped)) AS columns,\n" + "pg_get_statisticsobjdef_columns(oid) AS columns,\n" " 'd' = any(stxkind) AS ndist_enabled,\n" " 'f' = any(stxkind) AS deps_enabled,\n" - " 'm' = any(stxkind) AS mcv_enabled,\n"); + " 'm' = any(stxkind) AS mcv_enabled,\n" + " 'e' = any(stxkind) AS expressions_enabled,\n"); if (pset.sversion >= 130000) appendPQExpBufferStr(&buf, " stxstattarget\n"); @@ -2739,6 +2741,12 @@ describeOneTableDetails(const char *schemaname, if (strcmp(PQgetvalue(result, i, 7), "t") == 0) { appendPQExpBuffer(&buf, "%smcv", gotone ? ", " : ""); + gotone = true; + } + + if (strcmp(PQgetvalue(result, i, 8), "t") == 0) + { + appendPQExpBuffer(&buf, "%sexpressions", gotone ? ", " : ""); } appendPQExpBuffer(&buf, ") ON %s FROM %s", @@ -2746,9 +2754,9 @@ describeOneTableDetails(const char *schemaname, PQgetvalue(result, i, 1)); /* Show the stats target if it's not default */ - if (strcmp(PQgetvalue(result, i, 8), "-1") != 0) + if (strcmp(PQgetvalue(result, i, 9), "-1") != 0) appendPQExpBuffer(&buf, "; STATISTICS %s", - PQgetvalue(result, i, 8)); + PQgetvalue(result, i, 9)); printTableAddFooter(&cont, buf.data); } diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index fc2202b843..26dee513f4 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -3652,6 +3652,14 @@ proname => 'pg_get_statisticsobjdef', provolatile => 's', prorettype => 'text', proargtypes => 'oid', prosrc => 'pg_get_statisticsobjdef' }, +{ oid => '8887', descr => 'extended statistics columns', + proname => 'pg_get_statisticsobjdef_columns', provolatile => 's', + prorettype => 'text', proargtypes => 'oid', + prosrc => 'pg_get_statisticsobjdef_columns' }, +{ oid => '8886', descr => 'extended statistics expressions', + proname => 'pg_get_statisticsobjdef_expressions', provolatile => 's', + prorettype => '_text', proargtypes => 'oid', + prosrc => 'pg_get_statisticsobjdef_expressions' }, { oid => '3352', descr => 'partition key description', proname => 'pg_get_partkeydef', provolatile => 's', prorettype => 'text', proargtypes => 'oid', prosrc => 'pg_get_partkeydef' }, diff --git a/src/include/catalog/pg_statistic_ext.h b/src/include/catalog/pg_statistic_ext.h index 61d402c600..c182f5684c 100644 --- a/src/include/catalog/pg_statistic_ext.h +++ b/src/include/catalog/pg_statistic_ext.h @@ -52,6 +52,9 @@ CATALOG(pg_statistic_ext,3381,StatisticExtRelationId) #ifdef CATALOG_VARLEN char stxkind[1] BKI_FORCE_NOT_NULL; /* statistics kinds requested * to build */ + pg_node_tree stxexprs; /* A list of expression trees for stats + * attributes that are not simple column + * references. */ #endif } FormData_pg_statistic_ext; @@ -77,6 +80,7 @@ DECLARE_INDEX(pg_statistic_ext_relid_index, 3379, on pg_statistic_ext using btre #define STATS_EXT_NDISTINCT 'd' #define STATS_EXT_DEPENDENCIES 'f' #define STATS_EXT_MCV 'm' +#define STATS_EXT_EXPRESSIONS 'e' #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_statistic_ext_data.h b/src/include/catalog/pg_statistic_ext_data.h index c9515df117..4794fcd2dd 100644 --- a/src/include/catalog/pg_statistic_ext_data.h +++ b/src/include/catalog/pg_statistic_ext_data.h @@ -37,6 +37,7 @@ CATALOG(pg_statistic_ext_data,3429,StatisticExtDataRelationId) pg_ndistinct stxdndistinct; /* ndistinct coefficients (serialized) */ pg_dependencies stxddependencies; /* dependencies (serialized) */ pg_mcv_list stxdmcv; /* MCV (serialized) */ + pg_statistic stxdexpr[1]; /* stats for expressions */ #endif diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index 3684f87a88..f42cf15866 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -450,6 +450,7 @@ typedef enum NodeTag T_TypeName, T_ColumnDef, T_IndexElem, + T_StatsElem, T_Constraint, T_DefElem, T_RangeTblEntry, diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ec14fc2036..046df9ddcc 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -2812,8 +2812,24 @@ typedef struct CreateStatsStmt List *relations; /* rels to build stats on (list of RangeVar) */ char *stxcomment; /* comment to apply to stats, or NULL */ bool if_not_exists; /* do nothing if stats name already exists */ + bool transformed; /* true when transformStatsStmt is finished */ } CreateStatsStmt; +/* + * StatsElem - statistics parameters (used in CREATE STATISTICS) + * + * For a plain attribute, 'name' is the name of the referenced table column + * and 'expr' is NULL. For an expression, 'name' is NULL and 'expr' is the + * expression tree. + */ +typedef struct StatsElem +{ + NodeTag type; + char *name; /* name of attribute to index, or NULL */ + Node *expr; /* expression to index, or NULL */ +} StatsElem; + + /* ---------------------- * Alter Statistics Statement * ---------------------- diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index b4059895de..de8fab0506 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -917,6 +917,7 @@ typedef struct StatisticExtInfo RelOptInfo *rel; /* back-link to statistic's table */ char kind; /* statistic kind of this entry */ Bitmapset *keys; /* attnums of the columns covered */ + List *exprs; /* expressions */ } StatisticExtInfo; /* diff --git a/src/include/parser/parse_node.h b/src/include/parser/parse_node.h index d25819aa28..82e5190964 100644 --- a/src/include/parser/parse_node.h +++ b/src/include/parser/parse_node.h @@ -69,6 +69,7 @@ typedef enum ParseExprKind EXPR_KIND_FUNCTION_DEFAULT, /* default parameter value for function */ EXPR_KIND_INDEX_EXPRESSION, /* index expression */ EXPR_KIND_INDEX_PREDICATE, /* index predicate */ + EXPR_KIND_STATS_EXPRESSION, /* extended statistics expression */ EXPR_KIND_ALTER_COL_TRANSFORM, /* transform expr in ALTER COLUMN TYPE */ EXPR_KIND_EXECUTE_PARAMETER, /* parameter value in EXECUTE */ EXPR_KIND_TRIGGER_WHEN, /* WHEN condition in CREATE TRIGGER */ diff --git a/src/include/parser/parse_utilcmd.h b/src/include/parser/parse_utilcmd.h index bc3d66ed88..c864801628 100644 --- a/src/include/parser/parse_utilcmd.h +++ b/src/include/parser/parse_utilcmd.h @@ -26,6 +26,8 @@ extern AlterTableStmt *transformAlterTableStmt(Oid relid, AlterTableStmt *stmt, List **afterStmts); extern IndexStmt *transformIndexStmt(Oid relid, IndexStmt *stmt, const char *queryString); +extern CreateStatsStmt *transformStatsStmt(Oid relid, CreateStatsStmt *stmt, + const char *queryString); extern void transformRuleStmt(RuleStmt *stmt, const char *queryString, List **actions, Node **whereClause); extern List *transformCreateSchemaStmt(CreateSchemaStmt *stmt); diff --git a/src/include/statistics/extended_stats_internal.h b/src/include/statistics/extended_stats_internal.h index 02bf6a0502..5ef358754f 100644 --- a/src/include/statistics/extended_stats_internal.h +++ b/src/include/statistics/extended_stats_internal.h @@ -57,19 +57,35 @@ typedef struct SortItem int count; } SortItem; +/* + * Used to pass pre-computed information about expressions the stats + * object is defined on. + */ +typedef struct ExprInfo +{ + int nexprs; /* number of expressions */ + Oid *collations; /* collation for each expression */ + Oid *types; /* type of each expression */ + Datum **values; /* values for each expression */ + bool **nulls; /* nulls for each expression */ +} ExprInfo; + extern MVNDistinct *statext_ndistinct_build(double totalrows, int numrows, HeapTuple *rows, - Bitmapset *attrs, VacAttrStats **stats); + ExprInfo *exprs, Bitmapset *attrs, + VacAttrStats **stats); extern bytea *statext_ndistinct_serialize(MVNDistinct *ndistinct); extern MVNDistinct *statext_ndistinct_deserialize(bytea *data); extern MVDependencies *statext_dependencies_build(int numrows, HeapTuple *rows, - Bitmapset *attrs, VacAttrStats **stats); + ExprInfo *exprs, Bitmapset *attrs, + VacAttrStats **stats); extern bytea *statext_dependencies_serialize(MVDependencies *dependencies); extern MVDependencies *statext_dependencies_deserialize(bytea *data); extern MCVList *statext_mcv_build(int numrows, HeapTuple *rows, - Bitmapset *attrs, VacAttrStats **stats, + ExprInfo *exprs, Bitmapset *attrs, + VacAttrStats **stats, double totalrows, int stattarget); extern bytea *statext_mcv_serialize(MCVList *mcv, VacAttrStats **stats); extern MCVList *statext_mcv_deserialize(bytea *data); @@ -93,11 +109,18 @@ extern void *bsearch_arg(const void *key, const void *base, extern AttrNumber *build_attnums_array(Bitmapset *attrs, int *numattrs); extern SortItem *build_sorted_items(int numrows, int *nitems, HeapTuple *rows, - TupleDesc tdesc, MultiSortSupport mss, + ExprInfo *exprs, TupleDesc tdesc, + MultiSortSupport mss, int numattrs, AttrNumber *attnums); extern bool examine_clause_args(List *args, Var **varp, Const **cstp, bool *varonleftp); +extern bool examine_clause_args2(List *args, Node **exprp, + Const **cstp, bool *expronleftp); +extern bool examine_opclause_expression(OpExpr *expr, Var **varp, Const **cstp, + bool *varonleftp); +extern bool examine_opclause_expression2(OpExpr *expr, Node **exprp, Const **cstp, + bool *expronleftp); extern Selectivity mcv_combine_selectivities(Selectivity simple_sel, Selectivity mcv_sel, @@ -124,4 +147,13 @@ extern Selectivity mcv_clause_selectivity_or(PlannerInfo *root, Selectivity *overlap_basesel, Selectivity *totalsel); +extern Bitmapset *add_expressions_to_attributes(Bitmapset *attrs, int nexprs); + +/* translate 0-based expression index to attnum and back */ +#define EXPRESSION_ATTNUM(index) \ + (MaxHeapAttributeNumber + (index) + 1) + +#define EXPRESSION_INDEX(attnum) \ + ((attnum) - MaxHeapAttributeNumber - 1) + #endif /* EXTENDED_STATS_INTERNAL_H */ diff --git a/src/include/statistics/statistics.h b/src/include/statistics/statistics.h index c9ed21155c..37e975cd78 100644 --- a/src/include/statistics/statistics.h +++ b/src/include/statistics/statistics.h @@ -121,6 +121,8 @@ extern Selectivity statext_clauselist_selectivity(PlannerInfo *root, extern bool has_stats_of_kind(List *stats, char requiredkind); extern StatisticExtInfo *choose_best_statistics(List *stats, char requiredkind, Bitmapset **clause_attnums, + List **clause_exprs, int nclauses); +extern HeapTuple statext_expressions_load(Oid stxoid, int idx); #endif /* STATISTICS_H */ diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out index 6293ab57bc..d9f8811aef 100644 --- a/src/test/regress/expected/rules.out +++ b/src/test/regress/expected/rules.out @@ -2384,6 +2384,7 @@ pg_stats_ext| SELECT cn.nspname AS schemaname, ( SELECT array_agg(a.attname ORDER BY a.attnum) AS array_agg FROM (unnest(s.stxkeys) k(k) JOIN pg_attribute a ON (((a.attrelid = s.stxrelid) AND (a.attnum = k.k))))) AS attnames, + pg_get_statisticsobjdef_expressions(s.oid) AS exprs, s.stxkind AS kinds, sd.stxdndistinct AS n_distinct, sd.stxddependencies AS dependencies, @@ -2405,6 +2406,80 @@ pg_stats_ext| SELECT cn.nspname AS schemaname, FROM (unnest(s.stxkeys) k(k) JOIN pg_attribute a ON (((a.attrelid = s.stxrelid) AND (a.attnum = k.k)))) WHERE (NOT has_column_privilege(c.oid, a.attnum, 'select'::text))))) AND ((c.relrowsecurity = false) OR (NOT row_security_active(c.oid)))); +pg_stats_ext_exprs| SELECT cn.nspname AS schemaname, + c.relname AS tablename, + sn.nspname AS statistics_schemaname, + s.stxname AS statistics_name, + pg_get_userbyid(s.stxowner) AS statistics_owner, + stat.expr, + (stat.a).stanullfrac AS null_frac, + (stat.a).stawidth AS avg_width, + (stat.a).stadistinct AS n_distinct, + CASE + WHEN ((stat.a).stakind1 = 1) THEN (stat.a).stavalues1 + WHEN ((stat.a).stakind2 = 1) THEN (stat.a).stavalues2 + WHEN ((stat.a).stakind3 = 1) THEN (stat.a).stavalues3 + WHEN ((stat.a).stakind4 = 1) THEN (stat.a).stavalues4 + WHEN ((stat.a).stakind5 = 1) THEN (stat.a).stavalues5 + ELSE NULL::anyarray + END AS most_common_vals, + CASE + WHEN ((stat.a).stakind1 = 1) THEN (stat.a).stanumbers1 + WHEN ((stat.a).stakind2 = 1) THEN (stat.a).stanumbers2 + WHEN ((stat.a).stakind3 = 1) THEN (stat.a).stanumbers3 + WHEN ((stat.a).stakind4 = 1) THEN (stat.a).stanumbers4 + WHEN ((stat.a).stakind5 = 1) THEN (stat.a).stanumbers5 + ELSE NULL::real[] + END AS most_common_freqs, + CASE + WHEN ((stat.a).stakind1 = 2) THEN (stat.a).stavalues1 + WHEN ((stat.a).stakind2 = 2) THEN (stat.a).stavalues2 + WHEN ((stat.a).stakind3 = 2) THEN (stat.a).stavalues3 + WHEN ((stat.a).stakind4 = 2) THEN (stat.a).stavalues4 + WHEN ((stat.a).stakind5 = 2) THEN (stat.a).stavalues5 + ELSE NULL::anyarray + END AS histogram_bounds, + CASE + WHEN ((stat.a).stakind1 = 3) THEN (stat.a).stanumbers1[1] + WHEN ((stat.a).stakind2 = 3) THEN (stat.a).stanumbers2[1] + WHEN ((stat.a).stakind3 = 3) THEN (stat.a).stanumbers3[1] + WHEN ((stat.a).stakind4 = 3) THEN (stat.a).stanumbers4[1] + WHEN ((stat.a).stakind5 = 3) THEN (stat.a).stanumbers5[1] + ELSE NULL::real + END AS correlation, + CASE + WHEN ((stat.a).stakind1 = 4) THEN (stat.a).stavalues1 + WHEN ((stat.a).stakind2 = 4) THEN (stat.a).stavalues2 + WHEN ((stat.a).stakind3 = 4) THEN (stat.a).stavalues3 + WHEN ((stat.a).stakind4 = 4) THEN (stat.a).stavalues4 + WHEN ((stat.a).stakind5 = 4) THEN (stat.a).stavalues5 + ELSE NULL::anyarray + END AS most_common_elems, + CASE + WHEN ((stat.a).stakind1 = 4) THEN (stat.a).stanumbers1 + WHEN ((stat.a).stakind2 = 4) THEN (stat.a).stanumbers2 + WHEN ((stat.a).stakind3 = 4) THEN (stat.a).stanumbers3 + WHEN ((stat.a).stakind4 = 4) THEN (stat.a).stanumbers4 + WHEN ((stat.a).stakind5 = 4) THEN (stat.a).stanumbers5 + ELSE NULL::real[] + END AS most_common_elem_freqs, + CASE + WHEN ((stat.a).stakind1 = 5) THEN (stat.a).stanumbers1 + WHEN ((stat.a).stakind2 = 5) THEN (stat.a).stanumbers2 + WHEN ((stat.a).stakind3 = 5) THEN (stat.a).stanumbers3 + WHEN ((stat.a).stakind4 = 5) THEN (stat.a).stanumbers4 + WHEN ((stat.a).stakind5 = 5) THEN (stat.a).stanumbers5 + ELSE NULL::real[] + END AS elem_count_histogram + FROM (((((pg_statistic_ext s + JOIN pg_class c ON ((c.oid = s.stxrelid))) + JOIN pg_statistic_ext_data sd ON ((s.oid = sd.stxoid))) + LEFT JOIN pg_namespace cn ON ((cn.oid = c.relnamespace))) + LEFT JOIN pg_namespace sn ON ((sn.oid = s.stxnamespace))) + LEFT JOIN LATERAL ( SELECT x.expr, + x.a + FROM ( SELECT unnest(pg_get_statisticsobjdef_expressions(s.oid)) AS expr, + unnest(sd.stxdexpr) AS a) x) stat ON ((sd.stxdexpr IS NOT NULL))); pg_tables| SELECT n.nspname AS schemaname, c.relname AS tablename, pg_get_userbyid(c.relowner) AS tableowner, diff --git a/src/test/regress/expected/stats_ext.out b/src/test/regress/expected/stats_ext.out index dbbe9844b2..63c44e1d70 100644 --- a/src/test/regress/expected/stats_ext.out +++ b/src/test/regress/expected/stats_ext.out @@ -41,14 +41,29 @@ CREATE STATISTICS tst ON a, b FROM nonexistent; ERROR: relation "nonexistent" does not exist CREATE STATISTICS tst ON a, b FROM pg_class; ERROR: column "a" does not exist +CREATE STATISTICS tst ON relname FROM pg_class; +ERROR: extended statistics require at least 2 columns CREATE STATISTICS tst ON relname, relname, relnatts FROM pg_class; ERROR: duplicate column name in statistics definition -CREATE STATISTICS tst ON relnatts + relpages FROM pg_class; -ERROR: only simple column references are allowed in CREATE STATISTICS -CREATE STATISTICS tst ON (relpages, reltuples) FROM pg_class; -ERROR: only simple column references are allowed in CREATE STATISTICS +CREATE STATISTICS tst ON relname, relname, relnatts, relname, relname, relnatts, relname, relname, relnatts FROM pg_class; +ERROR: cannot have more than 8 columns in statistics +CREATE STATISTICS tst ON relname, relname, relnatts, relname, relname, (relname || 'x'), (relnatts + 1), (relname || 'x'), (relname || 'x'), (relnatts + 1) FROM pg_class; +ERROR: cannot have more than 8 columns in statistics +CREATE STATISTICS tst ON (relname || 'x'), (relname || 'x'), (relnatts + 1), (relname || 'x'), (relname || 'x'), (relnatts + 1), (relname || 'x'), (relname || 'x'), (relnatts + 1) FROM pg_class; +ERROR: cannot have more than 8 columns in statistics +CREATE STATISTICS tst ON (relname || 'x'), (relname || 'x'), relnatts FROM pg_class; +ERROR: duplicate expression in statistics definition CREATE STATISTICS tst (unrecognized) ON relname, relnatts FROM pg_class; ERROR: unrecognized statistics kind "unrecognized" +-- incorrect expressions +CREATE STATISTICS tst ON relnatts + relpages FROM pg_class; -- missing parentheses +ERROR: syntax error at or near "+" +LINE 1: CREATE STATISTICS tst ON relnatts + relpages FROM pg_class; + ^ +CREATE STATISTICS tst ON (relpages, reltuples) FROM pg_class; -- tuple expression +ERROR: syntax error at or near "," +LINE 1: CREATE STATISTICS tst ON (relpages, reltuples) FROM pg_class... + ^ -- Ensure stats are dropped sanely, and test IF NOT EXISTS while at it CREATE TABLE ab1 (a INTEGER, b INTEGER, c INTEGER); CREATE STATISTICS IF NOT EXISTS ab1_a_b_stats ON a, b FROM ab1; @@ -148,6 +163,40 @@ CREATE STATISTICS ab1_a_b_stats ON a, b FROM ab1; ANALYZE ab1; DROP TABLE ab1 CASCADE; NOTICE: drop cascades to table ab1c +-- basic test for statistics on expressions +CREATE TABLE ab1 (a INTEGER, b INTEGER, c TIMESTAMP, d TIMESTAMPTZ); +-- expression stats may be built on a single column +CREATE STATISTICS ab1_exprstat_1 (expressions) ON (a+b) FROM ab1; +-- with a single expression, we only enable expression statistics +CREATE STATISTICS ab1_exprstat_2 ON (a+b) FROM ab1; +SELECT stxkind FROM pg_statistic_ext WHERE stxname = 'ab1_exprstat_2'; + stxkind +--------- + {e} +(1 row) + +-- adding anything to the expression builds all statistics kinds +CREATE STATISTICS ab1_exprstat_3 ON (a+b), a FROM ab1; +SELECT stxkind FROM pg_statistic_ext WHERE stxname = 'ab1_exprstat_3'; + stxkind +----------- + {d,f,m,e} +(1 row) + +-- expression must be immutable, but date_trunc on timestamptz is not +CREATE STATISTICS ab1_exprstat_4 (expressions) ON date_trunc('day', d) FROM ab1; +ERROR: functions in statistics expression must be marked IMMUTABLE +-- but on timestamp it should work fine +CREATE STATISTICS ab1_exprstat_5 (expressions) ON (a+b), (a-b), date_trunc('day', c) FROM ab1; +-- insert some data and run analyze, to test that these cases build properly +INSERT INTO ab1 +SELECT + generate_series(1,10), + generate_series(1,10), + generate_series('2020-10-01'::timestamp, '2020-10-10'::timestamp, interval '1 day'), + generate_series('2020-10-01'::timestamptz, '2020-10-10'::timestamptz, interval '1 day'); +ANALYZE ab1; +DROP TABLE ab1; -- Verify supported object types for extended statistics CREATE schema tststats; CREATE TABLE tststats.t (a int, b int, c text); @@ -425,6 +474,40 @@ SELECT * FROM check_estimated_rows('SELECT * FROM functional_dependencies WHERE 1 | 1 (1 row) +-- a => b, a => c, b => c +TRUNCATE functional_dependencies; +DROP STATISTICS func_deps_stat; +-- now do the same thing, but with expressions +INSERT INTO functional_dependencies (a, b, c, filler1) + SELECT i, i, i, i FROM generate_series(1,5000) s(i); +ANALYZE functional_dependencies; +SELECT * FROM check_estimated_rows('SELECT * FROM functional_dependencies WHERE mod(a, 11) = 1 AND mod(b::int, 13) = 1'); + estimated | actual +-----------+-------- + 1 | 35 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM functional_dependencies WHERE mod(a, 11) = 1 AND mod(b::int, 13) = 1 AND mod(c, 7) = 1'); + estimated | actual +-----------+-------- + 1 | 5 +(1 row) + +-- create statistics +CREATE STATISTICS func_deps_stat (expressions, dependencies) ON (mod(a,11)), (mod(b::int, 13)), (mod(c, 7)) FROM functional_dependencies; +ANALYZE functional_dependencies; +SELECT * FROM check_estimated_rows('SELECT * FROM functional_dependencies WHERE mod(a, 11) = 1 AND mod(b::int, 13) = 1'); + estimated | actual +-----------+-------- + 35 | 35 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM functional_dependencies WHERE mod(a, 11) = 1 AND mod(b::int, 13) = 1 AND mod(c, 7) = 1'); + estimated | actual +-----------+-------- + 5 | 5 +(1 row) + -- a => b, a => c, b => c TRUNCATE functional_dependencies; DROP STATISTICS func_deps_stat; @@ -894,6 +977,39 @@ SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE a = 1 AND b = 1 | 1 (1 row) +TRUNCATE mcv_lists; +DROP STATISTICS mcv_lists_stats; +-- random data (no MCV list), but with expression +INSERT INTO mcv_lists (a, b, c, filler1) + SELECT i, i, i, i FROM generate_series(1,5000) s(i); +ANALYZE mcv_lists; +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,37) = 1 AND mod(b::int,41) = 1'); + estimated | actual +-----------+-------- + 1 | 4 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,37) = 1 AND mod(b::int,41) = 1 AND mod(c,47) = 1'); + estimated | actual +-----------+-------- + 1 | 1 +(1 row) + +-- create statistics +CREATE STATISTICS mcv_lists_stats (expressions, mcv) ON (mod(a,37)), (mod(b::int,41)), (mod(c,47)) FROM mcv_lists; +ANALYZE mcv_lists; +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,37) = 1 AND mod(b::int,41) = 1'); + estimated | actual +-----------+-------- + 3 | 4 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,37) = 1 AND mod(b::int,41) = 1 AND mod(c,47) = 1'); + estimated | actual +-----------+-------- + 1 | 1 +(1 row) + -- 100 distinct combinations, all in the MCV list TRUNCATE mcv_lists; DROP STATISTICS mcv_lists_stats; @@ -1119,6 +1235,12 @@ SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE a = 1 OR b = ' 200 | 200 (1 row) +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE a = 1 OR b = ''1'' OR c = 1 OR d IS NOT NULL'); + estimated | actual +-----------+-------- + 200 | 200 +(1 row) + SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE a IN (1, 2, 51, 52) AND b IN ( ''1'', ''2'')'); estimated | actual -----------+-------- @@ -1205,6 +1327,454 @@ SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE a = 1 AND b = 50 | 50 (1 row) +-- 100 distinct combinations, all in the MCV list, but with expressions +TRUNCATE mcv_lists; +DROP STATISTICS mcv_lists_stats; +INSERT INTO mcv_lists (a, b, c, filler1) + SELECT i, i, i, i FROM generate_series(1,5000) s(i); +ANALYZE mcv_lists; +-- without any stats on the expressions, we have to use default selectivities, which +-- is why the estimates here are different from the pre-computed case above +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 AND mod(b::int,50) = 1'); + estimated | actual +-----------+-------- + 1 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE 1 = mod(a,100) AND 1 = mod(b::int,50)'); + estimated | actual +-----------+-------- + 1 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < 1 AND mod(b::int,50) < 1'); + estimated | actual +-----------+-------- + 556 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE 1 > mod(a,100) AND 1 > mod(b::int,50)'); + estimated | actual +-----------+-------- + 556 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) <= 0 AND mod(b::int,50) <= 0'); + estimated | actual +-----------+-------- + 556 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE 0 >= mod(a,100) AND 0 >= mod(b::int,50)'); + estimated | actual +-----------+-------- + 556 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 AND mod(b::int,50) = 1 AND mod(c,25) = 1'); + estimated | actual +-----------+-------- + 1 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < 5 AND mod(b::int,50) < 1 AND mod(c,25) < 5'); + estimated | actual +-----------+-------- + 185 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < 5 AND 1 > mod(b::int,50) AND 5 > mod(c,25)'); + estimated | actual +-----------+-------- + 185 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) <= 4 AND mod(b::int,50) <= 0 AND mod(c,25) <= 4'); + estimated | actual +-----------+-------- + 185 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE 4 >= mod(a,100) AND 0 >= mod(b::int,50) AND 4 >= mod(c,25)'); + estimated | actual +-----------+-------- + 185 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 OR mod(b::int,50) = 1 OR mod(c,25) = 1'); + estimated | actual +-----------+-------- + 75 | 200 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 OR mod(b::int,50) = 1 OR mod(c,25) = 1 OR d IS NOT NULL'); + estimated | actual +-----------+-------- + 75 | 200 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) IN (1, 2, 51, 52) AND mod(b::int,50) IN ( 1, 2)'); + estimated | actual +-----------+-------- + 1 | 200 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) IN (1, 2, 51, 52, NULL) AND mod(b::int,50) IN ( 1, 2, NULL)'); + estimated | actual +-----------+-------- + 1 | 200 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = ANY (ARRAY[1, 2, 51, 52]) AND mod(b::int,50) = ANY (ARRAY[1, 2])'); + estimated | actual +-----------+-------- + 1 | 200 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = ANY (ARRAY[NULL, 1, 2, 51, 52]) AND mod(b::int,50) = ANY (ARRAY[1, 2, NULL])'); + estimated | actual +-----------+-------- + 1 | 200 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) <= ANY (ARRAY[1, 2, 3]) AND mod(b::int,50) IN (1, 2, 3)'); + estimated | actual +-----------+-------- + 53 | 150 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) <= ANY (ARRAY[1, NULL, 2, 3]) AND mod(b::int,50) IN (1, 2, NULL, 3)'); + estimated | actual +-----------+-------- + 53 | 150 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < ALL (ARRAY[4, 5]) AND mod(c,25) > ANY (ARRAY[1, 2, 3])'); + estimated | actual +-----------+-------- + 391 | 100 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < ALL (ARRAY[4, 5]) AND mod(c,25) > ANY (ARRAY[1, 2, 3, NULL])'); + estimated | actual +-----------+-------- + 391 | 100 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < ALL (ARRAY[4, 5]) AND mod(b::int,50) IN (1, 2, 3) AND mod(c,25) > ANY (ARRAY[1, 2, 3])'); + estimated | actual +-----------+-------- + 6 | 100 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < ALL (ARRAY[4, 5]) AND mod(b::int,50) IN (1, 2, NULL, 3) AND mod(c,25) > ANY (ARRAY[1, 2, NULL, 3])'); + estimated | actual +-----------+-------- + 6 | 100 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 OR mod(b::int,50) = 1 OR mod(c,25) = 1 OR d IS NOT NULL'); + estimated | actual +-----------+-------- + 75 | 200 +(1 row) + +-- create statistics with expressions only +CREATE STATISTICS mcv_lists_stats (expressions) ON (mod(a,100)), (mod(b::int,50)), (mod(c,25)) FROM mcv_lists; +ANALYZE mcv_lists; +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 AND mod(b::int,50) = 1'); + estimated | actual +-----------+-------- + 1 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE 1 = mod(a,100) AND 1 = mod(b::int,50)'); + estimated | actual +-----------+-------- + 1 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < 1 AND mod(b::int,50) < 1'); + estimated | actual +-----------+-------- + 1 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE 1 > mod(a,100) AND 1 > mod(b::int,50)'); + estimated | actual +-----------+-------- + 1 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) <= 0 AND mod(b::int,50) <= 0'); + estimated | actual +-----------+-------- + 1 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE 0 >= mod(a,100) AND 0 >= mod(b::int,50)'); + estimated | actual +-----------+-------- + 1 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 AND mod(b::int,50) = 1 AND mod(c,25) = 1'); + estimated | actual +-----------+-------- + 1 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < 5 AND mod(b::int,50) < 1 AND mod(c,25) < 5'); + estimated | actual +-----------+-------- + 1 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < 5 AND 1 > mod(b::int,50) AND 5 > mod(c,25)'); + estimated | actual +-----------+-------- + 1 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) <= 4 AND mod(b::int,50) <= 0 AND mod(c,25) <= 4'); + estimated | actual +-----------+-------- + 1 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE 4 >= mod(a,100) AND 0 >= mod(b::int,50) AND 4 >= mod(c,25)'); + estimated | actual +-----------+-------- + 1 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 OR mod(b::int,50) = 1 OR mod(c,25) = 1'); + estimated | actual +-----------+-------- + 343 | 200 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 OR mod(b::int,50) = 1 OR mod(c,25) = 1 OR d IS NOT NULL'); + estimated | actual +-----------+-------- + 343 | 200 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) IN (1, 2, 51, 52) AND mod(b::int,50) IN ( 1, 2)'); + estimated | actual +-----------+-------- + 8 | 200 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) IN (1, 2, 51, 52, NULL) AND mod(b::int,50) IN ( 1, 2, NULL)'); + estimated | actual +-----------+-------- + 8 | 200 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = ANY (ARRAY[1, 2, 51, 52]) AND mod(b::int,50) = ANY (ARRAY[1, 2])'); + estimated | actual +-----------+-------- + 8 | 200 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = ANY (ARRAY[NULL, 1, 2, 51, 52]) AND mod(b::int,50) = ANY (ARRAY[1, 2, NULL])'); + estimated | actual +-----------+-------- + 8 | 200 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) <= ANY (ARRAY[1, 2, 3]) AND mod(b::int,50) IN (1, 2, 3)'); + estimated | actual +-----------+-------- + 26 | 150 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) <= ANY (ARRAY[1, NULL, 2, 3]) AND mod(b::int,50) IN (1, 2, NULL, 3)'); + estimated | actual +-----------+-------- + 26 | 150 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < ALL (ARRAY[4, 5]) AND mod(c,25) > ANY (ARRAY[1, 2, 3])'); + estimated | actual +-----------+-------- + 10 | 100 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < ALL (ARRAY[4, 5]) AND mod(c,25) > ANY (ARRAY[1, 2, 3, NULL])'); + estimated | actual +-----------+-------- + 10 | 100 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < ALL (ARRAY[4, 5]) AND mod(b::int,50) IN (1, 2, 3) AND mod(c,25) > ANY (ARRAY[1, 2, 3])'); + estimated | actual +-----------+-------- + 1 | 100 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < ALL (ARRAY[4, 5]) AND mod(b::int,50) IN (1, 2, NULL, 3) AND mod(c,25) > ANY (ARRAY[1, 2, NULL, 3])'); + estimated | actual +-----------+-------- + 1 | 100 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 OR mod(b::int,50) = 1 OR mod(c,25) = 1 OR d IS NOT NULL'); + estimated | actual +-----------+-------- + 343 | 200 +(1 row) + +DROP STATISTICS mcv_lists_stats; +-- create statistics with both MCV and expressions +CREATE STATISTICS mcv_lists_stats (expressions, mcv) ON (mod(a,100)), (mod(b::int,50)), (mod(c,25)) FROM mcv_lists; +ANALYZE mcv_lists; +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 AND mod(b::int,50) = 1'); + estimated | actual +-----------+-------- + 50 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE 1 = mod(a,100) AND 1 = mod(b::int,50)'); + estimated | actual +-----------+-------- + 50 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < 1 AND mod(b::int,50) < 1'); + estimated | actual +-----------+-------- + 50 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE 1 > mod(a,100) AND 1 > mod(b::int,50)'); + estimated | actual +-----------+-------- + 50 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) <= 0 AND mod(b::int,50) <= 0'); + estimated | actual +-----------+-------- + 50 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE 0 >= mod(a,100) AND 0 >= mod(b::int,50)'); + estimated | actual +-----------+-------- + 50 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 AND mod(b::int,50) = 1 AND mod(c,25) = 1'); + estimated | actual +-----------+-------- + 50 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < 5 AND mod(b::int,50) < 1 AND mod(c,25) < 5'); + estimated | actual +-----------+-------- + 50 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < 5 AND 1 > mod(b::int,50) AND 5 > mod(c,25)'); + estimated | actual +-----------+-------- + 50 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) <= 4 AND mod(b::int,50) <= 0 AND mod(c,25) <= 4'); + estimated | actual +-----------+-------- + 50 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE 4 >= mod(a,100) AND 0 >= mod(b::int,50) AND 4 >= mod(c,25)'); + estimated | actual +-----------+-------- + 50 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 OR mod(b::int,50) = 1 OR mod(c,25) = 1'); + estimated | actual +-----------+-------- + 200 | 200 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 OR mod(b::int,50) = 1 OR mod(c,25) = 1 OR d IS NOT NULL'); + estimated | actual +-----------+-------- + 200 | 200 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) IN (1, 2, 51, 52) AND mod(b::int,50) IN ( 1, 2)'); + estimated | actual +-----------+-------- + 200 | 200 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) IN (1, 2, 51, 52, NULL) AND mod(b::int,50) IN ( 1, 2, NULL)'); + estimated | actual +-----------+-------- + 200 | 200 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = ANY (ARRAY[1, 2, 51, 52]) AND mod(b::int,50) = ANY (ARRAY[1, 2])'); + estimated | actual +-----------+-------- + 200 | 200 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = ANY (ARRAY[NULL, 1, 2, 51, 52]) AND mod(b::int,50) = ANY (ARRAY[1, 2, NULL])'); + estimated | actual +-----------+-------- + 200 | 200 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) <= ANY (ARRAY[1, 2, 3]) AND mod(b::int,50) IN (1, 2, 3)'); + estimated | actual +-----------+-------- + 150 | 150 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) <= ANY (ARRAY[1, NULL, 2, 3]) AND mod(b::int,50) IN (1, 2, NULL, 3)'); + estimated | actual +-----------+-------- + 150 | 150 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < ALL (ARRAY[4, 5]) AND mod(c,25) > ANY (ARRAY[1, 2, 3])'); + estimated | actual +-----------+-------- + 100 | 100 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < ALL (ARRAY[4, 5]) AND mod(c,25) > ANY (ARRAY[1, 2, 3, NULL])'); + estimated | actual +-----------+-------- + 100 | 100 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < ALL (ARRAY[4, 5]) AND mod(b::int,50) IN (1, 2, 3) AND mod(c,25) > ANY (ARRAY[1, 2, 3])'); + estimated | actual +-----------+-------- + 100 | 100 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < ALL (ARRAY[4, 5]) AND mod(b::int,50) IN (1, 2, NULL, 3) AND mod(c,25) > ANY (ARRAY[1, 2, NULL, 3])'); + estimated | actual +-----------+-------- + 100 | 100 +(1 row) + +-- we can't use the statistic for OR clauses that are not fully covered (missing 'd' attribute) +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 OR mod(b::int,50) = 1 OR mod(c,25) = 1 OR d IS NOT NULL'); + estimated | actual +-----------+-------- + 200 | 200 +(1 row) + -- 100 distinct combinations with NULL values, all in the MCV list TRUNCATE mcv_lists; DROP STATISTICS mcv_lists_stats; @@ -1710,6 +2280,102 @@ SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_multi WHERE a = 0 OR (1 row) DROP TABLE mcv_lists_multi; +-- statistics on integer expressions +CREATE TABLE expr_stats (a int, b int, c int); +INSERT INTO expr_stats SELECT mod(i,10), mod(i,10), mod(i,10) FROM generate_series(1,10000) s(i); +ANALYZE expr_stats; +SELECT * FROM check_estimated_rows('SELECT * FROM expr_stats WHERE (2*a) = 0 AND (3*b) = 0'); + estimated | actual +-----------+-------- + 1 | 1000 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM expr_stats WHERE (a+b) = 0 AND (a-b) = 0'); + estimated | actual +-----------+-------- + 1 | 1000 +(1 row) + +CREATE STATISTICS expr_stats_1 (mcv) ON (a+b), (a-b), (2*a), (3*b) FROM expr_stats; +ANALYZE expr_stats; +SELECT * FROM check_estimated_rows('SELECT * FROM expr_stats WHERE (2*a) = 0 AND (3*b) = 0'); + estimated | actual +-----------+-------- + 1000 | 1000 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM expr_stats WHERE (a+b) = 0 AND (a-b) = 0'); + estimated | actual +-----------+-------- + 1000 | 1000 +(1 row) + +-- FIXME add dependency tracking for expressions, to automatically drop after DROP TABLE +-- (not it fails, when there are no simple column references) +DROP STATISTICS expr_stats_1; +DROP TABLE expr_stats; +-- statistics on a mix columns and expressions +CREATE TABLE expr_stats (a int, b int, c int); +INSERT INTO expr_stats SELECT mod(i,10), mod(i,10), mod(i,10) FROM generate_series(1,10000) s(i); +ANALYZE expr_stats; +SELECT * FROM check_estimated_rows('SELECT * FROM expr_stats WHERE a = 0 AND (2*a) = 0 AND (3*b) = 0'); + estimated | actual +-----------+-------- + 1 | 1000 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM expr_stats WHERE a = 3 AND b = 3 AND (a-b) = 0'); + estimated | actual +-----------+-------- + 1 | 1000 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM expr_stats WHERE a = 0 AND b = 1 AND (a-b) = 0'); + estimated | actual +-----------+-------- + 1 | 0 +(1 row) + +CREATE STATISTICS expr_stats_1 (mcv) ON a, b, (2*a), (3*b), (a+b), (a-b) FROM expr_stats; +ANALYZE expr_stats; +SELECT * FROM check_estimated_rows('SELECT * FROM expr_stats WHERE a = 0 AND (2*a) = 0 AND (3*b) = 0'); + estimated | actual +-----------+-------- + 1000 | 1000 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM expr_stats WHERE a = 3 AND b = 3 AND (a-b) = 0'); + estimated | actual +-----------+-------- + 1000 | 1000 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM expr_stats WHERE a = 0 AND b = 1 AND (a-b) = 0'); + estimated | actual +-----------+-------- + 1 | 0 +(1 row) + +DROP TABLE expr_stats; +-- statistics on expressions with different data types +CREATE TABLE expr_stats (a int, b name, c text); +INSERT INTO expr_stats SELECT mod(i,10), md5(mod(i,10)::text), md5(mod(i,10)::text) FROM generate_series(1,10000) s(i); +ANALYZE expr_stats; +SELECT * FROM check_estimated_rows('SELECT * FROM expr_stats WHERE a = 0 AND (b || c) <= ''z'' AND (c || b) >= ''0'''); + estimated | actual +-----------+-------- + 111 | 1000 +(1 row) + +CREATE STATISTICS expr_stats_1 (mcv) ON a, b, (b || c), (c || b) FROM expr_stats; +ANALYZE expr_stats; +SELECT * FROM check_estimated_rows('SELECT * FROM expr_stats WHERE a = 0 AND (b || c) <= ''z'' AND (c || b) >= ''0'''); + estimated | actual +-----------+-------- + 1000 | 1000 +(1 row) + +DROP TABLE expr_stats; -- Permission tests. Users should not be able to see specific data values in -- the extended statistics, if they lack permission to see those values in -- the underlying table. diff --git a/src/test/regress/sql/stats_ext.sql b/src/test/regress/sql/stats_ext.sql index 7912e733ae..80d513f4b1 100644 --- a/src/test/regress/sql/stats_ext.sql +++ b/src/test/regress/sql/stats_ext.sql @@ -33,10 +33,16 @@ CREATE STATISTICS tst ON a, b; CREATE STATISTICS tst FROM sometab; CREATE STATISTICS tst ON a, b FROM nonexistent; CREATE STATISTICS tst ON a, b FROM pg_class; +CREATE STATISTICS tst ON relname FROM pg_class; CREATE STATISTICS tst ON relname, relname, relnatts FROM pg_class; -CREATE STATISTICS tst ON relnatts + relpages FROM pg_class; -CREATE STATISTICS tst ON (relpages, reltuples) FROM pg_class; +CREATE STATISTICS tst ON relname, relname, relnatts, relname, relname, relnatts, relname, relname, relnatts FROM pg_class; +CREATE STATISTICS tst ON relname, relname, relnatts, relname, relname, (relname || 'x'), (relnatts + 1), (relname || 'x'), (relname || 'x'), (relnatts + 1) FROM pg_class; +CREATE STATISTICS tst ON (relname || 'x'), (relname || 'x'), (relnatts + 1), (relname || 'x'), (relname || 'x'), (relnatts + 1), (relname || 'x'), (relname || 'x'), (relnatts + 1) FROM pg_class; +CREATE STATISTICS tst ON (relname || 'x'), (relname || 'x'), relnatts FROM pg_class; CREATE STATISTICS tst (unrecognized) ON relname, relnatts FROM pg_class; +-- incorrect expressions +CREATE STATISTICS tst ON relnatts + relpages FROM pg_class; -- missing parentheses +CREATE STATISTICS tst ON (relpages, reltuples) FROM pg_class; -- tuple expression -- Ensure stats are dropped sanely, and test IF NOT EXISTS while at it CREATE TABLE ab1 (a INTEGER, b INTEGER, c INTEGER); @@ -95,6 +101,36 @@ CREATE STATISTICS ab1_a_b_stats ON a, b FROM ab1; ANALYZE ab1; DROP TABLE ab1 CASCADE; +-- basic test for statistics on expressions +CREATE TABLE ab1 (a INTEGER, b INTEGER, c TIMESTAMP, d TIMESTAMPTZ); + +-- expression stats may be built on a single column +CREATE STATISTICS ab1_exprstat_1 (expressions) ON (a+b) FROM ab1; + +-- with a single expression, we only enable expression statistics +CREATE STATISTICS ab1_exprstat_2 ON (a+b) FROM ab1; +SELECT stxkind FROM pg_statistic_ext WHERE stxname = 'ab1_exprstat_2'; + +-- adding anything to the expression builds all statistics kinds +CREATE STATISTICS ab1_exprstat_3 ON (a+b), a FROM ab1; +SELECT stxkind FROM pg_statistic_ext WHERE stxname = 'ab1_exprstat_3'; + +-- expression must be immutable, but date_trunc on timestamptz is not +CREATE STATISTICS ab1_exprstat_4 (expressions) ON date_trunc('day', d) FROM ab1; + +-- but on timestamp it should work fine +CREATE STATISTICS ab1_exprstat_5 (expressions) ON (a+b), (a-b), date_trunc('day', c) FROM ab1; + +-- insert some data and run analyze, to test that these cases build properly +INSERT INTO ab1 +SELECT + generate_series(1,10), + generate_series(1,10), + generate_series('2020-10-01'::timestamp, '2020-10-10'::timestamp, interval '1 day'), + generate_series('2020-10-01'::timestamptz, '2020-10-10'::timestamptz, interval '1 day'); +ANALYZE ab1; +DROP TABLE ab1; + -- Verify supported object types for extended statistics CREATE schema tststats; @@ -270,6 +306,29 @@ SELECT * FROM check_estimated_rows('SELECT * FROM functional_dependencies WHERE TRUNCATE functional_dependencies; DROP STATISTICS func_deps_stat; +-- now do the same thing, but with expressions +INSERT INTO functional_dependencies (a, b, c, filler1) + SELECT i, i, i, i FROM generate_series(1,5000) s(i); + +ANALYZE functional_dependencies; + +SELECT * FROM check_estimated_rows('SELECT * FROM functional_dependencies WHERE mod(a, 11) = 1 AND mod(b::int, 13) = 1'); + +SELECT * FROM check_estimated_rows('SELECT * FROM functional_dependencies WHERE mod(a, 11) = 1 AND mod(b::int, 13) = 1 AND mod(c, 7) = 1'); + +-- create statistics +CREATE STATISTICS func_deps_stat (expressions, dependencies) ON (mod(a,11)), (mod(b::int, 13)), (mod(c, 7)) FROM functional_dependencies; + +ANALYZE functional_dependencies; + +SELECT * FROM check_estimated_rows('SELECT * FROM functional_dependencies WHERE mod(a, 11) = 1 AND mod(b::int, 13) = 1'); + +SELECT * FROM check_estimated_rows('SELECT * FROM functional_dependencies WHERE mod(a, 11) = 1 AND mod(b::int, 13) = 1 AND mod(c, 7) = 1'); + +-- a => b, a => c, b => c +TRUNCATE functional_dependencies; +DROP STATISTICS func_deps_stat; + INSERT INTO functional_dependencies (a, b, c, filler1) SELECT mod(i,100), mod(i,50), mod(i,25), i FROM generate_series(1,5000) s(i); @@ -477,6 +536,28 @@ SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE a = 1 AND b = SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE a = 1 AND b = ''1'' AND c = 1'); +TRUNCATE mcv_lists; +DROP STATISTICS mcv_lists_stats; + +-- random data (no MCV list), but with expression +INSERT INTO mcv_lists (a, b, c, filler1) + SELECT i, i, i, i FROM generate_series(1,5000) s(i); + +ANALYZE mcv_lists; + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,37) = 1 AND mod(b::int,41) = 1'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,37) = 1 AND mod(b::int,41) = 1 AND mod(c,47) = 1'); + +-- create statistics +CREATE STATISTICS mcv_lists_stats (expressions, mcv) ON (mod(a,37)), (mod(b::int,41)), (mod(c,47)) FROM mcv_lists; + +ANALYZE mcv_lists; + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,37) = 1 AND mod(b::int,41) = 1'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,37) = 1 AND mod(b::int,41) = 1 AND mod(c,47) = 1'); + -- 100 distinct combinations, all in the MCV list TRUNCATE mcv_lists; DROP STATISTICS mcv_lists_stats; @@ -563,6 +644,8 @@ SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE a = 1 OR b = ' SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE a = 1 OR b = ''1'' OR c = 1 OR d IS NOT NULL'); +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE a = 1 OR b = ''1'' OR c = 1 OR d IS NOT NULL'); + SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE a IN (1, 2, 51, 52) AND b IN ( ''1'', ''2'')'); SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE a IN (1, 2, 51, 52, NULL) AND b IN ( ''1'', ''2'', NULL)'); @@ -600,6 +683,176 @@ ANALYZE mcv_lists; SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE a = 1 AND b = ''1'''); + +-- 100 distinct combinations, all in the MCV list, but with expressions +TRUNCATE mcv_lists; +DROP STATISTICS mcv_lists_stats; + +INSERT INTO mcv_lists (a, b, c, filler1) + SELECT i, i, i, i FROM generate_series(1,5000) s(i); + +ANALYZE mcv_lists; + +-- without any stats on the expressions, we have to use default selectivities, which +-- is why the estimates here are different from the pre-computed case above + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 AND mod(b::int,50) = 1'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE 1 = mod(a,100) AND 1 = mod(b::int,50)'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < 1 AND mod(b::int,50) < 1'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE 1 > mod(a,100) AND 1 > mod(b::int,50)'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) <= 0 AND mod(b::int,50) <= 0'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE 0 >= mod(a,100) AND 0 >= mod(b::int,50)'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 AND mod(b::int,50) = 1 AND mod(c,25) = 1'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < 5 AND mod(b::int,50) < 1 AND mod(c,25) < 5'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < 5 AND 1 > mod(b::int,50) AND 5 > mod(c,25)'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) <= 4 AND mod(b::int,50) <= 0 AND mod(c,25) <= 4'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE 4 >= mod(a,100) AND 0 >= mod(b::int,50) AND 4 >= mod(c,25)'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 OR mod(b::int,50) = 1 OR mod(c,25) = 1'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 OR mod(b::int,50) = 1 OR mod(c,25) = 1 OR d IS NOT NULL'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) IN (1, 2, 51, 52) AND mod(b::int,50) IN ( 1, 2)'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) IN (1, 2, 51, 52, NULL) AND mod(b::int,50) IN ( 1, 2, NULL)'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = ANY (ARRAY[1, 2, 51, 52]) AND mod(b::int,50) = ANY (ARRAY[1, 2])'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = ANY (ARRAY[NULL, 1, 2, 51, 52]) AND mod(b::int,50) = ANY (ARRAY[1, 2, NULL])'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) <= ANY (ARRAY[1, 2, 3]) AND mod(b::int,50) IN (1, 2, 3)'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) <= ANY (ARRAY[1, NULL, 2, 3]) AND mod(b::int,50) IN (1, 2, NULL, 3)'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < ALL (ARRAY[4, 5]) AND mod(c,25) > ANY (ARRAY[1, 2, 3])'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < ALL (ARRAY[4, 5]) AND mod(c,25) > ANY (ARRAY[1, 2, 3, NULL])'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < ALL (ARRAY[4, 5]) AND mod(b::int,50) IN (1, 2, 3) AND mod(c,25) > ANY (ARRAY[1, 2, 3])'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < ALL (ARRAY[4, 5]) AND mod(b::int,50) IN (1, 2, NULL, 3) AND mod(c,25) > ANY (ARRAY[1, 2, NULL, 3])'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 OR mod(b::int,50) = 1 OR mod(c,25) = 1 OR d IS NOT NULL'); + +-- create statistics with expressions only +CREATE STATISTICS mcv_lists_stats (expressions) ON (mod(a,100)), (mod(b::int,50)), (mod(c,25)) FROM mcv_lists; + +ANALYZE mcv_lists; + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 AND mod(b::int,50) = 1'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE 1 = mod(a,100) AND 1 = mod(b::int,50)'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < 1 AND mod(b::int,50) < 1'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE 1 > mod(a,100) AND 1 > mod(b::int,50)'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) <= 0 AND mod(b::int,50) <= 0'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE 0 >= mod(a,100) AND 0 >= mod(b::int,50)'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 AND mod(b::int,50) = 1 AND mod(c,25) = 1'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < 5 AND mod(b::int,50) < 1 AND mod(c,25) < 5'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < 5 AND 1 > mod(b::int,50) AND 5 > mod(c,25)'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) <= 4 AND mod(b::int,50) <= 0 AND mod(c,25) <= 4'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE 4 >= mod(a,100) AND 0 >= mod(b::int,50) AND 4 >= mod(c,25)'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 OR mod(b::int,50) = 1 OR mod(c,25) = 1'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 OR mod(b::int,50) = 1 OR mod(c,25) = 1 OR d IS NOT NULL'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) IN (1, 2, 51, 52) AND mod(b::int,50) IN ( 1, 2)'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) IN (1, 2, 51, 52, NULL) AND mod(b::int,50) IN ( 1, 2, NULL)'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = ANY (ARRAY[1, 2, 51, 52]) AND mod(b::int,50) = ANY (ARRAY[1, 2])'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = ANY (ARRAY[NULL, 1, 2, 51, 52]) AND mod(b::int,50) = ANY (ARRAY[1, 2, NULL])'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) <= ANY (ARRAY[1, 2, 3]) AND mod(b::int,50) IN (1, 2, 3)'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) <= ANY (ARRAY[1, NULL, 2, 3]) AND mod(b::int,50) IN (1, 2, NULL, 3)'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < ALL (ARRAY[4, 5]) AND mod(c,25) > ANY (ARRAY[1, 2, 3])'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < ALL (ARRAY[4, 5]) AND mod(c,25) > ANY (ARRAY[1, 2, 3, NULL])'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < ALL (ARRAY[4, 5]) AND mod(b::int,50) IN (1, 2, 3) AND mod(c,25) > ANY (ARRAY[1, 2, 3])'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < ALL (ARRAY[4, 5]) AND mod(b::int,50) IN (1, 2, NULL, 3) AND mod(c,25) > ANY (ARRAY[1, 2, NULL, 3])'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 OR mod(b::int,50) = 1 OR mod(c,25) = 1 OR d IS NOT NULL'); + +DROP STATISTICS mcv_lists_stats; + +-- create statistics with both MCV and expressions +CREATE STATISTICS mcv_lists_stats (expressions, mcv) ON (mod(a,100)), (mod(b::int,50)), (mod(c,25)) FROM mcv_lists; + +ANALYZE mcv_lists; + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 AND mod(b::int,50) = 1'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE 1 = mod(a,100) AND 1 = mod(b::int,50)'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < 1 AND mod(b::int,50) < 1'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE 1 > mod(a,100) AND 1 > mod(b::int,50)'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) <= 0 AND mod(b::int,50) <= 0'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE 0 >= mod(a,100) AND 0 >= mod(b::int,50)'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 AND mod(b::int,50) = 1 AND mod(c,25) = 1'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < 5 AND mod(b::int,50) < 1 AND mod(c,25) < 5'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < 5 AND 1 > mod(b::int,50) AND 5 > mod(c,25)'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) <= 4 AND mod(b::int,50) <= 0 AND mod(c,25) <= 4'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE 4 >= mod(a,100) AND 0 >= mod(b::int,50) AND 4 >= mod(c,25)'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 OR mod(b::int,50) = 1 OR mod(c,25) = 1'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 OR mod(b::int,50) = 1 OR mod(c,25) = 1 OR d IS NOT NULL'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) IN (1, 2, 51, 52) AND mod(b::int,50) IN ( 1, 2)'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) IN (1, 2, 51, 52, NULL) AND mod(b::int,50) IN ( 1, 2, NULL)'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = ANY (ARRAY[1, 2, 51, 52]) AND mod(b::int,50) = ANY (ARRAY[1, 2])'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = ANY (ARRAY[NULL, 1, 2, 51, 52]) AND mod(b::int,50) = ANY (ARRAY[1, 2, NULL])'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) <= ANY (ARRAY[1, 2, 3]) AND mod(b::int,50) IN (1, 2, 3)'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) <= ANY (ARRAY[1, NULL, 2, 3]) AND mod(b::int,50) IN (1, 2, NULL, 3)'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < ALL (ARRAY[4, 5]) AND mod(c,25) > ANY (ARRAY[1, 2, 3])'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < ALL (ARRAY[4, 5]) AND mod(c,25) > ANY (ARRAY[1, 2, 3, NULL])'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < ALL (ARRAY[4, 5]) AND mod(b::int,50) IN (1, 2, 3) AND mod(c,25) > ANY (ARRAY[1, 2, 3])'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < ALL (ARRAY[4, 5]) AND mod(b::int,50) IN (1, 2, NULL, 3) AND mod(c,25) > ANY (ARRAY[1, 2, NULL, 3])'); + +-- we can't use the statistic for OR clauses that are not fully covered (missing 'd' attribute) +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 OR mod(b::int,50) = 1 OR mod(c,25) = 1 OR d IS NOT NULL'); + -- 100 distinct combinations with NULL values, all in the MCV list TRUNCATE mcv_lists; DROP STATISTICS mcv_lists_stats; @@ -892,6 +1145,59 @@ SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_multi WHERE a = 0 OR DROP TABLE mcv_lists_multi; + +-- statistics on integer expressions +CREATE TABLE expr_stats (a int, b int, c int); +INSERT INTO expr_stats SELECT mod(i,10), mod(i,10), mod(i,10) FROM generate_series(1,10000) s(i); +ANALYZE expr_stats; + +SELECT * FROM check_estimated_rows('SELECT * FROM expr_stats WHERE (2*a) = 0 AND (3*b) = 0'); +SELECT * FROM check_estimated_rows('SELECT * FROM expr_stats WHERE (a+b) = 0 AND (a-b) = 0'); + +CREATE STATISTICS expr_stats_1 (mcv) ON (a+b), (a-b), (2*a), (3*b) FROM expr_stats; +ANALYZE expr_stats; + +SELECT * FROM check_estimated_rows('SELECT * FROM expr_stats WHERE (2*a) = 0 AND (3*b) = 0'); +SELECT * FROM check_estimated_rows('SELECT * FROM expr_stats WHERE (a+b) = 0 AND (a-b) = 0'); + +-- FIXME add dependency tracking for expressions, to automatically drop after DROP TABLE +-- (not it fails, when there are no simple column references) +DROP STATISTICS expr_stats_1; +DROP TABLE expr_stats; + +-- statistics on a mix columns and expressions +CREATE TABLE expr_stats (a int, b int, c int); +INSERT INTO expr_stats SELECT mod(i,10), mod(i,10), mod(i,10) FROM generate_series(1,10000) s(i); +ANALYZE expr_stats; + +SELECT * FROM check_estimated_rows('SELECT * FROM expr_stats WHERE a = 0 AND (2*a) = 0 AND (3*b) = 0'); +SELECT * FROM check_estimated_rows('SELECT * FROM expr_stats WHERE a = 3 AND b = 3 AND (a-b) = 0'); +SELECT * FROM check_estimated_rows('SELECT * FROM expr_stats WHERE a = 0 AND b = 1 AND (a-b) = 0'); + +CREATE STATISTICS expr_stats_1 (mcv) ON a, b, (2*a), (3*b), (a+b), (a-b) FROM expr_stats; +ANALYZE expr_stats; + +SELECT * FROM check_estimated_rows('SELECT * FROM expr_stats WHERE a = 0 AND (2*a) = 0 AND (3*b) = 0'); +SELECT * FROM check_estimated_rows('SELECT * FROM expr_stats WHERE a = 3 AND b = 3 AND (a-b) = 0'); +SELECT * FROM check_estimated_rows('SELECT * FROM expr_stats WHERE a = 0 AND b = 1 AND (a-b) = 0'); + +DROP TABLE expr_stats; + +-- statistics on expressions with different data types +CREATE TABLE expr_stats (a int, b name, c text); +INSERT INTO expr_stats SELECT mod(i,10), md5(mod(i,10)::text), md5(mod(i,10)::text) FROM generate_series(1,10000) s(i); +ANALYZE expr_stats; + +SELECT * FROM check_estimated_rows('SELECT * FROM expr_stats WHERE a = 0 AND (b || c) <= ''z'' AND (c || b) >= ''0'''); + +CREATE STATISTICS expr_stats_1 (mcv) ON a, b, (b || c), (c || b) FROM expr_stats; +ANALYZE expr_stats; + +SELECT * FROM check_estimated_rows('SELECT * FROM expr_stats WHERE a = 0 AND (b || c) <= ''z'' AND (c || b) >= ''0'''); + +DROP TABLE expr_stats; + + -- Permission tests. Users should not be able to see specific data values in -- the extended statistics, if they lack permission to see those values in -- the underlying table. -- 2.26.2 --------------8917F6B7186C9FBB138CED7B-- ^ permalink raw reply [nested|flat] 2+ messages in thread
* Re: [PATCH] New [relation] option engine @ 2024-06-16 14:07 Nikolay Shaplov <[email protected]> 0 siblings, 0 replies; 2+ messages in thread From: Nikolay Shaplov @ 2024-06-16 14:07 UTC (permalink / raw) To: jian he <[email protected]>; +Cc: Peter Smith <[email protected]>; Alvaro Herrera <[email protected]>; Yura Sokolov <[email protected]>; [email protected]; Michael Paquier <[email protected]> В письме от понедельник, 3 июня 2024 г. 10:52:02 MSK пользователь jian he написал: Hi! Thank you for giving attention to my patch, and sorry, it takes me some time to deal with such things. By the way, what is your further intentions concerning this patch? Should I add you as a reviewer, or you are just passing by? Or add you as second author, if you are planning to suggest big changes So I tried to fix all issues you've pointed out: > you've changed the `typedef struct IndexAmRoutine` > then we also need to update doc/src/sgml/indexam.sgml. Oh. You are right. I've fixed that. The description I've created is quite short. I do not know what else I can say there without diving deep into implementation details. Do you have any ideas what can be added there? > some places you use "Spec Set", other places you use "spec set" ... > here the comments, you used "spec_set". > maybe we can make it more consistent? This sounds reasonable. I've changed it to Spec Set and Options Spec Set wherever I found alternative spelling. > typedef enum option_value_status > { > OPTION_VALUE_STATUS_EMPTY, /* Option was just initialized */ > OPTION_VALUE_STATUS_RAW, /* Option just came from syntax analyzer in > * has name, and raw (unparsed) value */ > OPTION_VALUE_STATUS_PARSED, /* Option was parsed and has link to catalog > * entry and proper value */ > OPTION_VALUE_STATUS_FOR_RESET, /* This option came from ALTER xxx RESET */ > } option_value_status; > overall I am not sure about the usage of option_value_status. > using some real example demo different option_value_status usage would be > great. I've added explanatory comment before option_value_status enum explaining what is what. Hope it will make things more clear. If not me know and we will try to find out what comments should be added > > + errmsg("RESET must not include values for parameters"))); > the error message seems not so good? eh... It does seem to be not good, but I've copied it from original code https://github.com/postgres/postgres/blob/ 00ac25a3c365004821e819653c3307acd3294818/src/backend/access/common/ reloptions.c#L1231 I would not dare changing it. I've already changed to many things here. > you declared as: > options_spec_set *get_stdrd_relopt_spec_set(bool is_for_toast); > but you used as: > options_spec_set * get_stdrd_relopt_spec_set(bool is_heap) > > maybe we refactor the usage as > `options_spec_set * get_stdrd_relopt_spec_set(bool is_for_toast)` My bad. I've changed is_for_toast to is_heap once, and I guess I've missed function declaration... Fixed it to be consistent. > + ereport(ERROR, > + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), > + errmsg("value %s out of bounds for option \"%s\"", > + value, option->gen->name), > + errdetail("Valid values are between \"%d\" and \"%d\".", > + optint->min, optint->max))); ..... > I think the above two, change errdetail to errhint would be more > appropriate. Same as above... I've just copied error messages from the original code. Fixing them better go as a separate patch I guess... -- Nikolay Shaplov aka Nataraj Fuzzing Engineer at Postgres Professional Matrix IM: @dhyan:nataraj.su Attachments: [text/x-patch] v8-0001-New-options-engine.patch (206.4K, ../../2726217.uZKlY2gecq@thinkpad-pgpro/2-v8-0001-New-options-engine.patch) download | inline diff: From 91bf99b9dfc03847bf5384713c2eee240de6d278 Mon Sep 17 00:00:00 2001 From: Nikolay Shaplov <[email protected]> Date: Sun, 16 Jun 2024 16:27:16 +0300 Subject: [PATCH v8 1/1] New options engine Add new options engine that will allow to define options directly in any part of code they are needed (anywhere, not only for relation and op. classes) and use unified API for option processing. This will allow: - Unify options code. Now we have core AM reloptions, contrib AM reloptions and local reloptions for op. classes, each using their own API. New engine will allow to use single API for them all. - Move options definition completely to Access Method implementation. Done for index AM, will be done storage AM later. (Before core AM all options were defined in reloptions.c) - All options for AM is defined in one single chunk of code. (Before for core AM you had to edit both reloptions.c and code of AM, this was error prone) - More simple way to define new option stets (we will need partitioned.* options sooner or later, this engine will allow to do it more easy) - Allow to add name=value options anywhere they needed. - More flexible way for defining option behavior (e.g. disable OIDS=true via postvalidate callback, without changing option processing code) What is done, and what is still in progress: - Index AM options now using new option engine the way it should be. - Options for Heap and Toast, are also using new engine, but they are still located in reloptions.c file as they were before. They will be moved to storage AM later. - local_options API that is used for op. class options is now wrapper around new option engine. Later on, build in op. classes will be updated to use new option engine directly, and local_options API can be kept for backward compatibility for a while. This patch does not change postgres behavior. All existing options work the way they did it before. One single error message is changed, and it is more accurate now. ``` -ERROR: unrecognized parameter "not_existing_option" +ERROR: unrecognized parameter "toast.not_existing_option" ``` General concepts of the new engine: - `option_spec` is a C-structure that completely describes single option, how it should be parsed, validated and stored: name, type, acceptable values, offset in bytea representation, postvalidate call back, etc. We have `option_spec_*` for each option type. - `options_spec_set` is a set of `option_spec`s that are available for certain context (e.g. for certain AM reloptions). It is a list of option_spec`s plus a callback for overall validation (e.g. make sure optionA < optionB) - There are four representations of options. The way they came from SQL parser (defList), the way they are stored in DB (TEXT[]), the binary representation that is used by outer code (bytea), and internal representation that is used by option engine while parsing and validating options (Option Value List) - There are functions to convert options from one representation to another (possibly with validation if applicable): `optionsTextArrayToDefList`, `optionsDefListToTextArray`, `optionsTextArrayToBytea` and others. - There are functions for defining `option_spec_set`: `allocateOptionsSpecSet`, `optionsSpecSetAddBool`, `optionsSpecSetAddInt`, etc. Outer code, that would like to use option engine, should define it's own `option_spec_set`, implement getting options from SQL parser, getting and putting them to DB storage, passing binary representation of the option to the code that would actually use them. For parsing, validating and converting options from one representation to another, outer code should use functions from options engine function set. Each function will use information from `option_spec_set` to determinate what options are available and how exactly they should be parsed and validated. This is only a start of option code reworking. Further improvements listed above and other suggestion will follow. --- contrib/bloom/bloom.h | 4 +- contrib/bloom/blutils.c | 102 +- contrib/dblink/dblink.c | 2 +- contrib/file_fdw/file_fdw.c | 2 +- contrib/postgres_fdw/option.c | 2 +- contrib/test_decoding/expected/twophase.out | 3 +- doc/src/sgml/indexam.sgml | 25 +- src/backend/access/brin/brin.c | 46 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/meson.build | 1 + src/backend/access/common/options.c | 1366 ++++++++++ src/backend/access/common/reloptions.c | 2234 ++++------------- src/backend/access/gin/ginutil.c | 46 +- src/backend/access/gist/gist.c | 2 +- src/backend/access/gist/gistutil.c | 52 +- src/backend/access/hash/hash.c | 2 +- src/backend/access/hash/hashutil.c | 34 +- src/backend/access/nbtree/nbtree.c | 35 +- src/backend/access/nbtree/nbtutils.c | 19 +- src/backend/access/spgist/spgutils.c | 41 +- src/backend/commands/createas.c | 11 +- src/backend/commands/foreigncmds.c | 2 +- src/backend/commands/indexcmds.c | 22 +- src/backend/commands/tablecmds.c | 117 +- src/backend/commands/tablespace.c | 16 +- src/backend/foreign/foreign.c | 14 +- src/backend/parser/parse_utilcmd.c | 4 +- src/backend/tcop/utility.c | 29 +- src/backend/utils/cache/attoptcache.c | 4 +- src/backend/utils/cache/relcache.c | 8 +- src/backend/utils/cache/spccache.c | 3 +- src/bin/pg_basebackup/streamutil.c | 2 +- src/include/access/amapi.h | 9 +- src/include/access/brin.h | 2 + src/include/access/brin_internal.h | 2 + src/include/access/gin_private.h | 1 + src/include/access/gist_private.h | 4 +- src/include/access/hash.h | 2 +- src/include/access/nbtree.h | 2 +- src/include/access/options.h | 274 ++ src/include/access/reloptions.h | 205 +- src/include/access/spgist.h | 3 - src/include/access/spgist_private.h | 1 + src/include/commands/tablecmds.h | 2 +- .../modules/dummy_index_am/dummy_index_am.c | 148 +- .../test_oat_hooks/expected/alter_table.out | 16 + src/test/regress/expected/reloptions.out | 13 +- src/test/regress/sql/reloptions.sql | 7 + 48 files changed, 2609 insertions(+), 2333 deletions(-) create mode 100644 src/backend/access/common/options.c create mode 100644 src/include/access/options.h diff --git a/contrib/bloom/bloom.h b/contrib/bloom/bloom.h index fba3ba7771..8e6741bc74 100644 --- a/contrib/bloom/bloom.h +++ b/contrib/bloom/bloom.h @@ -17,6 +17,7 @@ #include "access/generic_xlog.h" #include "access/itup.h" #include "access/xlog.h" +#include "access/options.h" #include "fmgr.h" #include "nodes/pathnodes.h" @@ -206,7 +207,8 @@ extern IndexBulkDeleteResult *blbulkdelete(IndexVacuumInfo *info, void *callback_state); extern IndexBulkDeleteResult *blvacuumcleanup(IndexVacuumInfo *info, IndexBulkDeleteResult *stats); -extern bytea *bloptions(Datum reloptions, bool validate); +extern void *blrelopt_specset(void); +extern void blReloptionPostprocess(void *, bool validate); extern void blcostestimate(PlannerInfo *root, IndexPath *path, double loop_count, Cost *indexStartupCost, Cost *indexTotalCost, Selectivity *indexSelectivity, diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c index 6836129c90..5a7074f431 100644 --- a/contrib/bloom/blutils.c +++ b/contrib/bloom/blutils.c @@ -15,7 +15,7 @@ #include "access/amapi.h" #include "access/generic_xlog.h" -#include "access/reloptions.h" +#include "access/options.h" #include "bloom.h" #include "catalog/index.h" #include "commands/vacuum.h" @@ -34,52 +34,12 @@ PG_FUNCTION_INFO_V1(blhandler); -/* Kind of relation options for bloom index */ -static relopt_kind bl_relopt_kind; - -/* parse table for fillRelOptions */ -static relopt_parse_elt bl_relopt_tab[INDEX_MAX_KEYS + 1]; +/* Catalog of relation options for bloom index */ +static options_spec_set *bl_relopt_specset; static int32 myRand(void); static void mySrand(uint32 seed); -/* - * Module initialize function: initialize info about Bloom relation options. - * - * Note: keep this in sync with makeDefaultBloomOptions(). - */ -void -_PG_init(void) -{ - int i; - char buf[16]; - - bl_relopt_kind = add_reloption_kind(); - - /* Option for length of signature */ - add_int_reloption(bl_relopt_kind, "length", - "Length of signature in bits", - DEFAULT_BLOOM_LENGTH, 1, MAX_BLOOM_LENGTH, - AccessExclusiveLock); - bl_relopt_tab[0].optname = "length"; - bl_relopt_tab[0].opttype = RELOPT_TYPE_INT; - bl_relopt_tab[0].offset = offsetof(BloomOptions, bloomLength); - - /* Number of bits for each possible index column: col1, col2, ... */ - for (i = 0; i < INDEX_MAX_KEYS; i++) - { - snprintf(buf, sizeof(buf), "col%d", i + 1); - add_int_reloption(bl_relopt_kind, buf, - "Number of bits generated for each index column", - DEFAULT_BLOOM_BITS, 1, MAX_BLOOM_BITS, - AccessExclusiveLock); - bl_relopt_tab[i + 1].optname = MemoryContextStrdup(TopMemoryContext, - buf); - bl_relopt_tab[i + 1].opttype = RELOPT_TYPE_INT; - bl_relopt_tab[i + 1].offset = offsetof(BloomOptions, bitSize[0]) + sizeof(int) * i; - } -} - /* * Construct a default set of Bloom options. */ @@ -137,7 +97,7 @@ blhandler(PG_FUNCTION_ARGS) amroutine->amvacuumcleanup = blvacuumcleanup; amroutine->amcanreturn = NULL; amroutine->amcostestimate = blcostestimate; - amroutine->amoptions = bloptions; + amroutine->amreloptspecset = blrelopt_specset; amroutine->amproperty = NULL; amroutine->ambuildphasename = NULL; amroutine->amvalidate = blvalidate; @@ -156,6 +116,15 @@ blhandler(PG_FUNCTION_ARGS) PG_RETURN_POINTER(amroutine); } +void +blReloptionPostprocess(void *data, bool validate) +{ + BloomOptions *opts = (BloomOptions *) data; + /* Convert signature length from # of bits to # to words, rounding up */ + opts->bloomLength = (opts->bloomLength + SIGNWORDBITS - 1) / SIGNWORDBITS; +} + + /* * Fill BloomState structure for particular index. */ @@ -470,24 +439,37 @@ BloomInitMetapage(Relation index, ForkNumber forknum) UnlockReleaseBuffer(metaBuffer); } -/* - * Parse reloptions for bloom index, producing a BloomOptions struct. - */ -bytea * -bloptions(Datum reloptions, bool validate) +void * +blrelopt_specset(void) { - BloomOptions *rdopts; + int i; + char buf[16]; - /* Parse the user-given reloptions */ - rdopts = (BloomOptions *) build_reloptions(reloptions, validate, - bl_relopt_kind, - sizeof(BloomOptions), - bl_relopt_tab, - lengthof(bl_relopt_tab)); + if (bl_relopt_specset) + return bl_relopt_specset; - /* Convert signature length from # of bits to # to words, rounding up */ - if (rdopts) - rdopts->bloomLength = (rdopts->bloomLength + SIGNWORDBITS - 1) / SIGNWORDBITS; - return (bytea *) rdopts; + bl_relopt_specset = allocateOptionsSpecSet(NULL, + sizeof(BloomOptions), false, INDEX_MAX_KEYS + 1); + bl_relopt_specset->postprocess_fun = blReloptionPostprocess; + + optionsSpecSetAddInt(bl_relopt_specset, "length", + "Length of signature in bits", + NoLock, /* No lock as far as ALTER is not + * effective */ + offsetof(BloomOptions, bloomLength), NULL, + DEFAULT_BLOOM_LENGTH, 1, MAX_BLOOM_LENGTH); + + /* Number of bits for each possible index column: col1, col2, ... */ + for (i = 0; i < INDEX_MAX_KEYS; i++) + { + snprintf(buf, 16, "col%d", i + 1); + optionsSpecSetAddInt(bl_relopt_specset, buf, + "Number of bits for corresponding column", + NoLock, /* No lock as far as ALTER is not + * effective */ + offsetof(BloomOptions, bitSize[i]), NULL, + DEFAULT_BLOOM_BITS, 1, MAX_BLOOM_BITS); + } + return bl_relopt_specset; } diff --git a/contrib/dblink/dblink.c b/contrib/dblink/dblink.c index 755293456f..e39c8cd49a 100644 --- a/contrib/dblink/dblink.c +++ b/contrib/dblink/dblink.c @@ -1938,7 +1938,7 @@ PG_FUNCTION_INFO_V1(dblink_fdw_validator); Datum dblink_fdw_validator(PG_FUNCTION_ARGS) { - List *options_list = untransformRelOptions(PG_GETARG_DATUM(0)); + List *options_list = optionsTextArrayToDefList(PG_GETARG_DATUM(0)); Oid context = PG_GETARG_OID(1); ListCell *cell; diff --git a/contrib/file_fdw/file_fdw.c b/contrib/file_fdw/file_fdw.c index 249d82d3a0..26740d3ef7 100644 --- a/contrib/file_fdw/file_fdw.c +++ b/contrib/file_fdw/file_fdw.c @@ -197,7 +197,7 @@ file_fdw_handler(PG_FUNCTION_ARGS) Datum file_fdw_validator(PG_FUNCTION_ARGS) { - List *options_list = untransformRelOptions(PG_GETARG_DATUM(0)); + List *options_list = optionsTextArrayToDefList(PG_GETARG_DATUM(0)); Oid catalog = PG_GETARG_OID(1); char *filename = NULL; DefElem *force_not_null = NULL; diff --git a/contrib/postgres_fdw/option.c b/contrib/postgres_fdw/option.c index 630b304338..e6df6dd53a 100644 --- a/contrib/postgres_fdw/option.c +++ b/contrib/postgres_fdw/option.c @@ -71,7 +71,7 @@ PG_FUNCTION_INFO_V1(postgres_fdw_validator); Datum postgres_fdw_validator(PG_FUNCTION_ARGS) { - List *options_list = untransformRelOptions(PG_GETARG_DATUM(0)); + List *options_list = optionsTextArrayToDefList(PG_GETARG_DATUM(0)); Oid catalog = PG_GETARG_OID(1); ListCell *cell; diff --git a/contrib/test_decoding/expected/twophase.out b/contrib/test_decoding/expected/twophase.out index 517f20bc37..71581fba34 100644 --- a/contrib/test_decoding/expected/twophase.out +++ b/contrib/test_decoding/expected/twophase.out @@ -69,9 +69,10 @@ WHERE locktype = 'relation' AND relation = 'test_prepared1'::regclass; relation | locktype | mode -----------------+----------+--------------------- + test_prepared_1 | relation | AccessShareLock test_prepared_1 | relation | RowExclusiveLock test_prepared_1 | relation | AccessExclusiveLock -(2 rows) +(3 rows) -- The insert should show the newly altered column but not the DDL. SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1'); diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml index e3c1539a1e..31b7d940e7 100644 --- a/doc/src/sgml/indexam.sgml +++ b/doc/src/sgml/indexam.sgml @@ -480,27 +480,12 @@ amcostestimate (PlannerInfo *root, <para> <programlisting> -bytea * -amoptions (ArrayType *reloptions, - bool validate); +void * +amreloptspecset(void); </programlisting> - Parse and validate the reloptions array for an index. This is called only - when a non-null reloptions array exists for the index. - <parameter>reloptions</parameter> is a <type>text</type> array containing entries of the - form <replaceable>name</replaceable><literal>=</literal><replaceable>value</replaceable>. - The function should construct a <type>bytea</type> value, which will be copied - into the <structfield>rd_options</structfield> field of the index's relcache entry. - The data contents of the <type>bytea</type> value are open for the access - method to define; most of the standard access methods use struct - <structname>StdRdOptions</structname>. - When <parameter>validate</parameter> is true, the function should report a suitable - error message if any of the options are unrecognized or have invalid - values; when <parameter>validate</parameter> is false, invalid entries should be - silently ignored. (<parameter>validate</parameter> is false when loading options - already stored in <structname>pg_catalog</structname>; an invalid entry could only - be found if the access method has changed its rules for options, and in - that case ignoring obsolete entries is appropriate.) - It is OK to return NULL if default behavior is wanted. + Returns pointer to Options Spec Set that defines options available for an + index relation. This Spec Set will be used for parsing and validating index + relation options. </para> <para> diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c index d0f3848c95..01e332c033 100644 --- a/src/backend/access/brin/brin.c +++ b/src/backend/access/brin/brin.c @@ -20,7 +20,6 @@ #include "access/brin_pageops.h" #include "access/brin_xlog.h" #include "access/relation.h" -#include "access/reloptions.h" #include "access/relscan.h" #include "access/table.h" #include "access/tableam.h" @@ -279,7 +278,6 @@ brinhandler(PG_FUNCTION_ARGS) amroutine->amvacuumcleanup = brinvacuumcleanup; amroutine->amcanreturn = NULL; amroutine->amcostestimate = brincostestimate; - amroutine->amoptions = brinoptions; amroutine->amproperty = NULL; amroutine->ambuildphasename = NULL; amroutine->amvalidate = brinvalidate; @@ -294,6 +292,7 @@ brinhandler(PG_FUNCTION_ARGS) amroutine->amestimateparallelscan = NULL; amroutine->aminitparallelscan = NULL; amroutine->amparallelrescan = NULL; + amroutine->amreloptspecset = bringetreloptspecset; PG_RETURN_POINTER(amroutine); } @@ -1329,23 +1328,6 @@ brinvacuumcleanup(IndexVacuumInfo *info, IndexBulkDeleteResult *stats) return stats; } -/* - * reloptions processor for BRIN indexes - */ -bytea * -brinoptions(Datum reloptions, bool validate) -{ - static const relopt_parse_elt tab[] = { - {"pages_per_range", RELOPT_TYPE_INT, offsetof(BrinOptions, pagesPerRange)}, - {"autosummarize", RELOPT_TYPE_BOOL, offsetof(BrinOptions, autosummarize)} - }; - - return (bytea *) build_reloptions(reloptions, validate, - RELOPT_KIND_BRIN, - sizeof(BrinOptions), - tab, lengthof(tab)); -} - /* * SQL-callable function to scan through an index and summarize all ranges * that are not currently summarized. @@ -3000,3 +2982,29 @@ brin_fill_empty_ranges(BrinBuildState *state, blkno += state->bs_pagesPerRange; } } + +static options_spec_set *brin_relopt_specset = NULL; + +void * +bringetreloptspecset(void) +{ + if (brin_relopt_specset) + return brin_relopt_specset; + brin_relopt_specset = allocateOptionsSpecSet(NULL, + sizeof(BrinOptions), false, 2); + + optionsSpecSetAddInt(brin_relopt_specset, "pages_per_range", + "Number of pages that each page range covers in a BRIN index", + NoLock, /* since ALTER is not allowed no lock + * needed */ + offsetof(BrinOptions, pagesPerRange), NULL, + BRIN_DEFAULT_PAGES_PER_RANGE, + BRIN_MIN_PAGES_PER_RANGE, + BRIN_MAX_PAGES_PER_RANGE); + optionsSpecSetAddBool(brin_relopt_specset, "autosummarize", + "Enables automatic summarization on this BRIN index", + AccessExclusiveLock, + offsetof(BrinOptions, autosummarize), NULL, + false); + return brin_relopt_specset; +} diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index e78de31265..014d1b1562 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -18,6 +18,7 @@ OBJS = \ detoast.o \ heaptuple.o \ indextuple.o \ + options.o \ printsimple.o \ printtup.o \ relation.o \ diff --git a/src/backend/access/common/meson.build b/src/backend/access/common/meson.build index 14090c728f..0cfd469910 100644 --- a/src/backend/access/common/meson.build +++ b/src/backend/access/common/meson.build @@ -6,6 +6,7 @@ backend_sources += files( 'detoast.c', 'heaptuple.c', 'indextuple.c', + 'options.c', 'printsimple.c', 'printtup.c', 'relation.c', diff --git a/src/backend/access/common/options.c b/src/backend/access/common/options.c new file mode 100644 index 0000000000..8cc790365b --- /dev/null +++ b/src/backend/access/common/options.c @@ -0,0 +1,1366 @@ +/*------------------------------------------------------------------------- + * + * options.c + * An uniform, context-free API for processing name=value options. Used + * to process relation options (reloptions), attribute options, opclass + * options, etc. + * + * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * + * IDENTIFICATION + * src/backend/access/common/options.c + * + *------------------------------------------------------------------------- + */ + +#include "postgres.h" + +#include "access/options.h" +#include "catalog/pg_type.h" +#include "commands/defrem.h" +#include "nodes/makefuncs.h" +#include "utils/builtins.h" +#include "utils/guc.h" +#include "utils/memutils.h" +#include "mb/pg_wchar.h" + + +/* + * OPTIONS SPECIFICATION and OPTION SPECIFICATION SET + * + * Each option is defined via Option Specification object (Option Spec). + * Option Spec should have all information that is needed for processing + * (parsing, validating, converting) of a single option. Implemented via set of + * option_spec_* structures. + * + * A set of Option Specs (Options Spec Set), defines all options available for + * certain object (certain relation kind for example). It is a list of + * Options Specs, plus validation functions that can be used to validate whole + * option set, if needed. Implemented via options_spec_set structure and set of + * optionsSpecSetAdd* functions that are used for adding Option Specs items to + * a Set. + * + * NOTE: we choose term "specification" instead of "definition" because term + * "definition" is used for objects that came from syntax parser. So to avoid + * confusion here we have Option Specifications, and all "definitions" are from + * parser. + */ + +/* + * OPTION VALUES REPRESENTATIONS + * + * Option values usually came from syntax parser in form of defList object, + * stored in pg_catalog as text array, and used when they are stored in memory + * as C-structure. These are different option values representations. Here goes + * brief description of all representations used in the code. + * + * Value List + * + * Value List is an internal representation that is used while converting + * option values between different representation. Value List item is called + * "parsed", when Value's value is converted to a proper data type and + * validated, or is called "unparsed", when Value's value is stored as raw + * string that was obtained from the source without any checks. In conversion + * function names first case is referred as Values, second case is referred as + * RawValues. Value List is implemented as List of option_value C-structures. + * + * defList + * + * Options in form of definition List that comes from syntax parser. (For + * reloptions it is a part of SQL query that goes after WITH, SET or RESET + * keywords). Can be converted to Value List using optionsDefListToRawValues + * and can be obtained from TEXT[] via optionsTextArrayToDefList functions. + * + * TEXT[] + * + * Options in form suitable for storig in TEXT[] field in DB. (E.g. reloptions + * are stores in pg_catalog.pg_class table in reloptions field). Can be + * converted to and from Value List using optionsValuesToTextArray and + * optionsTextArrayToRawValues functions. + * + * Bytea + * + * Option data stored in C-structure with varlena header in the beginning of + * the structure. This representation is used to pass option values to the core + * postgres. It is fast to read, it can be cached and so on. Bytea + * representation can be obtained from Value List using optionsValuesToBytea + * function, and can't be converted back. + */ + +/* + * OPTION STRING VALUE NOTION + * + * Important thing for bytea representation is that all data should be stored + * in one bytea chunk, including values of the string options. This is needed + * as * bytea representation is cached, and can freed, moved or recreated again + * without any notion, so it can't have parts allocated separately. + * + * Thus memory chunk for Bytea option values representation is divided into two + * parts. First goes a C-structure that stores fixed length values. Then goes + * memory area reserved for string values. + * + * For string values C-structure stores offsets (not pointers). These offsets + * can be used to access attached string value: + * + * String_pointer = Bytea_head_pointer + offset. + */ + +static option_spec_basic *allocateOptionSpec(int type, const char *name, + const char *desc, LOCKMODE lockmode, + int struct_offset, bool is_local, + option_value_postvalidate postvalidate_fn); + +static void parse_one_option(option_value *option, bool validate); +static void *optionsAllocateBytea(options_spec_set *spec_set, List *options); + + +static List *optionsDefListToRawValues(List *defList, bool is_for_reset); +static Datum optionsValuesToTextArray(List *options_values); +static List *optionsMergeOptionValues(List *old_options, List *new_options); + +/* + * allocateOptionsSpecSet + * Creates new Options Spec Set object: Allocates memory and initializes + * structure members. + * + * Spec Set items can be add via allocateOptionSpec and optionSpecSetAddItem + * functions or by calling directly any of optionsSpecSetAdd* function + * (preferable way) + * + * namespace - Spec Set can be bind to certain namespace (E.g. + * namespace.option=value). Options from other namespaces will be ignored while + * processing. If set to NULL, no namespace will be used at all. + * + * bytea_size - size of target structure of Bytea options representation + * + * num_items_expected - if you know expected number of Spec Set items set it + * here. Set to -1 in other cases. num_items_expected will be used for + * preallocating memory and will trigger error, if you try to add more items + * than you expected. + */ + +options_spec_set * +allocateOptionsSpecSet(const char *namspace, int bytea_size, bool is_local, + int num_items_expected) +{ + MemoryContext oldcxt; + options_spec_set *spec_set; + + if (!is_local) + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + spec_set = palloc(sizeof(options_spec_set)); + if (namspace) + { + spec_set->namspace = palloc(strlen(namspace) + 1); + strcpy(spec_set->namspace, namspace); + } + else + spec_set->namspace = NULL; + if (num_items_expected > 0) + { + spec_set->num_allocated = num_items_expected; + spec_set->assert_on_realloc = true; + spec_set->definitions = palloc( + spec_set->num_allocated * sizeof(option_spec_basic *)); + } + else + { + spec_set->num_allocated = 0; + spec_set->assert_on_realloc = false; + spec_set->definitions = NULL; + } + spec_set->num = 0; + spec_set->struct_size = bytea_size; + spec_set->postprocess_fun = NULL; + spec_set->is_local = is_local; + if (!is_local) + MemoryContextSwitchTo(oldcxt); + return spec_set; +} + +/* + * allocateOptionSpec + * Allocates a new Option Specifiation object of desired type and + * initialize the type-independent fields + */ +static option_spec_basic * +allocateOptionSpec(int type, const char *name, const char *desc, + LOCKMODE lockmode, int struct_offset, bool is_local, + option_value_postvalidate postvalidate_fn) +{ + MemoryContext oldcxt; + size_t size; + option_spec_basic *newoption; + + if (!is_local) + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + + switch (type) + { + case OPTION_TYPE_BOOL: + size = sizeof(option_spec_bool); + break; + case OPTION_TYPE_INT: + size = sizeof(option_spec_int); + break; + case OPTION_TYPE_REAL: + size = sizeof(option_spec_real); + break; + case OPTION_TYPE_ENUM: + size = sizeof(option_spec_enum); + break; + case OPTION_TYPE_STRING: + size = sizeof(option_spec_string); + break; + default: + elog(ERROR, "unsupported reloption type %d", (int) type); + return NULL; /* keep compiler quiet */ + } + + newoption = palloc(size); + + newoption->name = pstrdup(name); + if (desc) + newoption->desc = pstrdup(desc); + else + newoption->desc = NULL; + newoption->type = type; + newoption->lockmode = lockmode; + newoption->struct_offset = struct_offset; + newoption->postvalidate_fn = postvalidate_fn; + + if (!is_local) + MemoryContextSwitchTo(oldcxt); + + return newoption; +} + +/* + * optionSpecSetAddItem + * Adds pre-created Option Specification object to the Spec Set + */ +static void +optionSpecSetAddItem(option_spec_basic *newoption, + options_spec_set *spec_set) +{ + if (spec_set->num >= spec_set->num_allocated) + { + MemoryContext oldcxt = NULL; + + Assert(!spec_set->assert_on_realloc); + if (!spec_set->is_local) + oldcxt = MemoryContextSwitchTo(TopMemoryContext); + + if (spec_set->num_allocated == 0) + { + spec_set->num_allocated = 8; + spec_set->definitions = palloc( + spec_set->num_allocated * sizeof(option_spec_basic *)); + } + else + { + spec_set->num_allocated *= 2; + spec_set->definitions = repalloc(spec_set->definitions, + spec_set->num_allocated * sizeof(option_spec_basic *)); + } + if (!spec_set->is_local) + MemoryContextSwitchTo(oldcxt); + } + spec_set->definitions[spec_set->num] = newoption; + spec_set->num++; +} + + +/* + * optionsSpecSetAddBool + * Adds boolean Option Specification entry to the Spec Set + */ +void +optionsSpecSetAddBool(options_spec_set *spec_set, const char *name, + const char *desc, LOCKMODE lockmode, int struct_offset, + option_value_postvalidate postvalidate_fn, + bool default_val) +{ + option_spec_bool *spec_set_item; + + spec_set_item = (option_spec_bool *) allocateOptionSpec(OPTION_TYPE_BOOL, + name, desc, lockmode, struct_offset, + spec_set->is_local, postvalidate_fn); + + spec_set_item->default_val = default_val; + + optionSpecSetAddItem((option_spec_basic *) spec_set_item, spec_set); +} + +/* + * optionsSpecSetAddInt + * Adds integer Option Specification entry to the Spec Set + */ +void +optionsSpecSetAddInt(options_spec_set *spec_set, const char *name, + const char *desc, LOCKMODE lockmode, int struct_offset, + option_value_postvalidate postvalidate_fn, + int default_val, int min_val, int max_val) +{ + option_spec_int *spec_set_item; + + spec_set_item = (option_spec_int *) allocateOptionSpec(OPTION_TYPE_INT, + name, desc, lockmode, struct_offset, + spec_set->is_local, postvalidate_fn); + + spec_set_item->default_val = default_val; + spec_set_item->min = min_val; + spec_set_item->max = max_val; + + optionSpecSetAddItem((option_spec_basic *) spec_set_item, spec_set); +} + +/* + * optionsSpecSetAddReal + * Adds float Option Specification entry to the Spec Set + */ +void +optionsSpecSetAddReal(options_spec_set *spec_set, const char *name, + const char *desc, LOCKMODE lockmode, int struct_offset, + option_value_postvalidate postvalidate_fn, + double default_val, double min_val, double max_val) +{ + option_spec_real *spec_set_item; + + spec_set_item = (option_spec_real *) allocateOptionSpec(OPTION_TYPE_REAL, + name, desc, lockmode, struct_offset, + spec_set->is_local, postvalidate_fn); + + spec_set_item->default_val = default_val; + spec_set_item->min = min_val; + spec_set_item->max = max_val; + + optionSpecSetAddItem((option_spec_basic *) spec_set_item, spec_set); +} + +/* + * optionsSpecSetAddEnum + * Adds enum Option Specification entry to the Spec Set + * + * The members array must have a terminating NULL entry. + * + * The detailmsg is shown when unsupported values are passed, and has this + * form: "Valid values are \"foo\", \"bar\", and \"bar\"." + * + * The members array and detailmsg are not copied -- caller must ensure that + * they are valid throughout the life of the process. + */ + +void +optionsSpecSetAddEnum(options_spec_set *spec_set, const char *name, + const char *desc, LOCKMODE lockmode, int struct_offset, + option_value_postvalidate postvalidate_fn, + opt_enum_elt_def *members, int default_val, + const char *detailmsg) +{ + option_spec_enum *spec_set_item; + + spec_set_item = (option_spec_enum *) allocateOptionSpec(OPTION_TYPE_ENUM, + name, desc, lockmode, struct_offset, + spec_set->is_local, postvalidate_fn); + + spec_set_item->default_val = default_val; + spec_set_item->members = members; + spec_set_item->detailmsg = detailmsg; + + optionSpecSetAddItem((option_spec_basic *) spec_set_item, spec_set); +} + +/* + * optionsSpecSetAddString + * Adds string Option Specification entry to the Spec Set + * + * "validator" is an optional function pointer that can be used to test the + * validity of the values. It must elog(ERROR) when the argument string is + * not acceptable for the variable. Note that the default value must pass + * the validation. + */ +void +optionsSpecSetAddString(options_spec_set *spec_set, const char *name, + const char *desc, LOCKMODE lockmode, int struct_offset, + option_value_postvalidate postvalidate_fn, + const char *default_val, validate_string_option validator, + fill_string_option filler) +{ + option_spec_string *spec_set_item; + + /* make sure the validator/default combination is sane */ + if (validator) + (validator) (default_val); + + spec_set_item = (option_spec_string *) allocateOptionSpec( + OPTION_TYPE_STRING, + name, desc, lockmode, struct_offset, + spec_set->is_local, postvalidate_fn); + spec_set_item->validate_cb = validator; + spec_set_item->fill_cb = filler; + + if (default_val) + spec_set_item->default_val = MemoryContextStrdup(TopMemoryContext, + default_val); + else + spec_set_item->default_val = NULL; + optionSpecSetAddItem((option_spec_basic *) spec_set_item, spec_set); +} + +/* optionsDefListToRawValues + * Converts options values from DefList representation into Raw Values + * List. + * + * No parsing is done here except for checking that RESET syntax is correct + * (i.e. does not have =name part of value=name template). Syntax analyzer does + * not see difference between SET and RESET cases, so we should treat it here + * manually + */ +static List * +optionsDefListToRawValues(List *defList, bool is_for_reset) +{ + ListCell *cell; + List *result = NIL; + + foreach(cell, defList) + { + option_value *option_dst; + DefElem *def = (DefElem *) lfirst(cell); + char *value; + + option_dst = palloc(sizeof(option_value)); + + if (def->defnamespace) + { + option_dst->namspace = palloc(strlen(def->defnamespace) + 1); + strcpy(option_dst->namspace, def->defnamespace); + } + else + option_dst->namspace = NULL; + + option_dst->raw_name = palloc(strlen(def->defname) + 1); + strcpy(option_dst->raw_name, def->defname); + + if (is_for_reset) + { + /* + * If this option came from RESET statement we should throw error + * it it brings us name=value data, as syntax analyzer do not + * prevent it + */ + if (def->arg != NULL) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("RESET must not include values for parameters"))); + + option_dst->status = OPTION_VALUE_STATUS_FOR_RESET; + } + else + { + /* + * For SET statement we should treat (name) expression as if it is + * actually (name=true) so do it here manually. In other cases + * just use value as we should use it + */ + option_dst->status = OPTION_VALUE_STATUS_RAW; + if (def->arg != NULL) + value = defGetString(def); + else + value = "true"; + option_dst->raw_value = palloc(strlen(value) + 1); + strcpy(option_dst->raw_value, value); + } + + result = lappend(result, option_dst); + } + return result; +} + +/* + * optionsValuesToTextArray + * Converts options Values List (option_values) into TEXT[] representation + * + * This conversion is usually needed for saving option values into database + * (e.g. to store reloptions in pg_class.reloptions) + */ + +Datum +optionsValuesToTextArray(List *options_values) +{ + ArrayBuildState *astate = NULL; + ListCell *cell; + Datum result; + + foreach(cell, options_values) + { + option_value *option = (option_value *) lfirst(cell); + const char *name; + char *value; + text *t; + int len; + + /* + * Raw value were not cleared while parsing, so instead of converting + * it back, just use it to store value as text + */ + value = option->raw_value; + + Assert(option->status != OPTION_VALUE_STATUS_EMPTY); + + /* + * Name will be taken from option definition, if option were parsed or + * from raw_name if option were not parsed for some reason + */ + if (option->status == OPTION_VALUE_STATUS_PARSED) + name = option->gen->name; + else + name = option->raw_name; + + /* + * Now build "name=value" string and append it to the array + */ + len = VARHDRSZ + strlen(name) + strlen(value) + 1; + t = (text *) palloc(len + 1); + SET_VARSIZE(t, len); + sprintf(VARDATA(t), "%s=%s", name, value); + astate = accumArrayResult(astate, PointerGetDatum(t), false, + TEXTOID, CurrentMemoryContext); + } + if (astate) + result = makeArrayResult(astate, CurrentMemoryContext); + else + result = (Datum) 0; + + return result; +} + +/* + * optionsTextArrayToRawValues + * Converts option values from TEXT[] representation (datum_array) into + * Raw Values List. + * + * Used while fetching options values from DB, as a first step of converting + * them to other representations. + */ +List * +optionsTextArrayToRawValues(Datum array_datum) +{ + List *result = NIL; + + if (PointerIsValid(DatumGetPointer(array_datum))) + { + ArrayType *array = DatumGetArrayTypeP(array_datum); + Datum *options; + int noptions; + int i; + + deconstruct_array_builtin(array, TEXTOID, &options, NULL, &noptions); + + for (i = 0; i < noptions; i++) + { + option_value *option_dst; + char *text_str = VARDATA(options[i]); + int text_len = VARSIZE(options[i]) - VARHDRSZ; + int j; + int name_len = -1; + char *name; + int raw_value_len; + char *raw_value; + + /* + * Find position of '=' sign and treat id as a separator between + * name and value in "name=value" item + */ + for (j = 0; j < text_len; j = j + pg_mblen(text_str)) + { + if (text_str[j] == '=') + { + name_len = j; + break; + } + } + Assert(name_len >= 1); /* Just in case */ + + raw_value_len = text_len - name_len - 1; + + /* + * Copy name from src + */ + name = palloc(name_len + 1); + memcpy(name, text_str, name_len); + name[name_len] = '\0'; + + /* + * Copy value from src + */ + raw_value = palloc(raw_value_len + 1); + memcpy(raw_value, text_str + name_len + 1, raw_value_len); + raw_value[raw_value_len] = '\0'; + + /* + * Create new option_value item + */ + option_dst = palloc(sizeof(option_value)); + option_dst->status = OPTION_VALUE_STATUS_RAW; + option_dst->raw_name = name; + option_dst->raw_value = raw_value; + option_dst->namspace = NULL; + + result = lappend(result, option_dst); + } + } + return result; +} + +/* + * optionsMergeOptionValues + * Updates(or Resets) values from one Options Values List(old_options), + * with values from another Options Values List (new_options) + + * This function is used while ALTERing options of some object. + * If option from new_options list has OPTION_VALUE_STATUS_FOR_RESET flag + * on, option with that name will be excluded result list. + */ +static List * +optionsMergeOptionValues(List *old_options, List *new_options) +{ + List *result = NIL; + ListCell *old_cell; + ListCell *new_cell; + + /* + * First add to result all old options that are not mentioned in new list + */ + foreach(old_cell, old_options) + { + bool found; + const char *old_name; + option_value *old_option; + + old_option = (option_value *) lfirst(old_cell); + if (old_option->status == OPTION_VALUE_STATUS_PARSED) + old_name = old_option->gen->name; + else + old_name = old_option->raw_name; + + /* + * Looking for a new option with same name + */ + found = false; + foreach(new_cell, new_options) + { + option_value *new_option; + const char *new_name; + + new_option = (option_value *) lfirst(new_cell); + if (new_option->status == OPTION_VALUE_STATUS_PARSED) + new_name = new_option->gen->name; + else + new_name = new_option->raw_name; + + if (strcmp(new_name, old_name) == 0) + { + found = true; + break; + } + } + if (!found) + result = lappend(result, old_option); + } + + /* + * Now add all to result all new options that are not designated for reset + */ + foreach(new_cell, new_options) + { + option_value *new_option; + + new_option = (option_value *) lfirst(new_cell); + + if (new_option->status != OPTION_VALUE_STATUS_FOR_RESET) + result = lappend(result, new_option); + } + return result; +} + +/* + * optionsDefListValdateNamespaces + * Checks that defList has only options with namespaces from + * allowed_namspaces array. Items without namspace are also accepted + * + * Used while validation of syntax parser output. Error is thrown if unproper + * namespace is found. + */ +void +optionsDefListValdateNamespaces(List *defList, char **allowed_namspaces) +{ + ListCell *cell; + + foreach(cell, defList) + { + DefElem *def = (DefElem *) lfirst(cell); + + /* + * Checking namespace only for options that have namespaces. Options + * with no namespaces are always accepted + */ + if (def->defnamespace) + { + bool found = false; + int i = 0; + + while (allowed_namspaces[i]) + { + if (strcmp(def->defnamespace, + allowed_namspaces[i]) == 0) + { + found = true; + break; + } + i++; + } + if (!found) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("unrecognized parameter namespace \"%s\"", + def->defnamespace))); + } + } +} + +/* + * optionsDefListFilterNamespaces + * Filter out DefList items that has "namspace" namespace. If "namspace" + * is NULL, only namespaseless options are returned + */ +List * +optionsDefListFilterNamespaces(List *defList, const char *namspace) +{ + ListCell *cell; + List *result = NIL; + + foreach(cell, defList) + { + DefElem *def = (DefElem *) lfirst(cell); + + if ((!namspace && !def->defnamespace) || + (namspace && def->defnamespace && + strcmp(namspace, def->defnamespace) == 0)) + result = lappend(result, def); + } + return result; +} + +/* + * optionsTextArrayToDefList + * Converts option values from TEXT[] representation into DefList + * representation. + */ +List * +optionsTextArrayToDefList(Datum options) +{ + List *result = NIL; + ArrayType *array; + Datum *optiondatums; + int noptions; + int i; + + /* Nothing to do if no options */ + if (!PointerIsValid(DatumGetPointer(options))) + return result; + + array = DatumGetArrayTypeP(options); + + deconstruct_array_builtin(array, TEXTOID, &optiondatums, NULL, &noptions); + + for (i = 0; i < noptions; i++) + { + char *s; + char *p; + Node *val = NULL; + + s = TextDatumGetCString(optiondatums[i]); + p = strchr(s, '='); + if (p) + { + *p++ = '\0'; + val = (Node *) makeString(pstrdup(p)); + } + result = lappend(result, makeDefElem(pstrdup(s), val, -1)); + } + + return result; +} + +/* + * optionsDefListToTextArray + * Converts option values from DefList representation into TEXT[] + * representation. + */ +Datum +optionsDefListToTextArray(List *defList) +{ + ListCell *cell; + Datum result; + ArrayBuildState *astate = NULL; + + foreach(cell, defList) + { + DefElem *def = (DefElem *) lfirst(cell); + const char *name = def->defname; + const char *value; + text *t; + int len; + + if (def->arg != NULL) + value = defGetString(def); + else + value = "true"; + + if (def->defnamespace) + { + /* + * This function is used for backward compatibility in the place + * where namespases are not allowed + */ + Assert(false); /* Should not get here */ + return (Datum) 0; + } + len = VARHDRSZ + strlen(name) + strlen(value) + 1; + t = (text *) palloc(len + 1); + SET_VARSIZE(t, len); + sprintf(VARDATA(t), "%s=%s", name, value); + astate = accumArrayResult(astate, PointerGetDatum(t), false, + TEXTOID, CurrentMemoryContext); + + } + if (astate) + result = makeArrayResult(astate, CurrentMemoryContext); + else + result = (Datum) 0; + return result; +} + + +/* + * optionsParseRawValues + * Transforms RawValues List into [Parsed] Values List. Validation is done + * if validate flag is set. + * + * Options data that come parsed SQL query or DB storage, first converted into + * RawValues (where value are kept in text format), then is parsed into + * Values using this function + * + * Validation is used only for data that came from SQL query. We trust that + * data that came from DB is correct. + * + * If validation is off, all unknown options are kept unparsed so they will + * be stored back to DB until user RESETs them directly. + * + * This function destroys incoming list. + */ +List * +optionsParseRawValues(List *raw_values, options_spec_set *spec_set, + bool validate) +{ + ListCell *cell; + List *result = NIL; + bool *is_set; + int i; + + is_set = palloc0(sizeof(bool) * spec_set->num); + foreach(cell, raw_values) + { + option_value *option = (option_value *) lfirst(cell); + bool found = false; + + /* option values with RESET status does not need parsing */ + Assert(option->status != OPTION_VALUE_STATUS_FOR_RESET); + + /* Should not parse Values Set twice */ + Assert(option->status != OPTION_VALUE_STATUS_PARSED); + + if (validate && option->namspace && (!spec_set->namspace || + strcmp(spec_set->namspace, option->namspace) != 0)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("unrecognized parameter namespace \"%s\"", + option->namspace))); + + for (i = 0; i < spec_set->num; i++) + { + option_spec_basic *opt_spec = spec_set->definitions[i]; + + if (strcmp(option->raw_name, opt_spec->name) == 0) + { + if (validate && is_set[i]) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("parameter \"%s\" specified more than once", + option->raw_name))); + + pfree(option->raw_name); + option->raw_name = NULL; + option->gen = opt_spec; + parse_one_option(option, validate); + is_set[i] = true; + found = true; + break; + } + } + if (validate && !found) + { + if (option->namspace) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("unrecognized parameter \"%s.%s\"", + option->namspace, option->raw_name))); + else + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("unrecognized parameter \"%s\"", + option->raw_name))); + } + result = lappend(result, option); + } + return result; +} + +/* + * parse_one_option + * Function to parse and validate single option. + * + * See optionsParseRawValues for more info. + * + * Link to Option Spec for the option is embedded into "option_value" + */ +static void +parse_one_option(option_value *option, bool validate) +{ + char *value; + bool parsed; + + value = option->raw_value; + + switch (option->gen->type) + { + case OPTION_TYPE_BOOL: + { + parsed = parse_bool(value, &option->values.bool_val); + if (validate && !parsed) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("invalid value for boolean option \"%s\": %s", + option->gen->name, value))); + } + break; + case OPTION_TYPE_INT: + { + option_spec_int *optint = + (option_spec_int *) option->gen; + + parsed = parse_int(value, &option->values.int_val, 0, NULL); + if (validate && !parsed) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("invalid value for integer option \"%s\": %s", + option->gen->name, value))); + if (validate && (option->values.int_val < optint->min || + option->values.int_val > optint->max)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("value %s out of bounds for option \"%s\"", + value, option->gen->name), + errdetail("Valid values are between \"%d\" and \"%d\".", + optint->min, optint->max))); + } + break; + case OPTION_TYPE_REAL: + { + option_spec_real *optreal = + (option_spec_real *) option->gen; + + parsed = parse_real(value, &option->values.real_val, 0, NULL); + if (validate && !parsed) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("invalid value for floating point option \"%s\": %s", + option->gen->name, value))); + if (validate && (option->values.real_val < optreal->min || + option->values.real_val > optreal->max)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("value %s out of bounds for option \"%s\"", + value, option->gen->name), + errdetail("Valid values are between \"%f\" and \"%f\".", + optreal->min, optreal->max))); + } + break; + case OPTION_TYPE_ENUM: + { + option_spec_enum *optenum = + (option_spec_enum *) option->gen; + opt_enum_elt_def *elt; + + parsed = false; + for (elt = optenum->members; elt->string_val; elt++) + { + if (strcmp(value, elt->string_val) == 0) + { + option->values.enum_val = elt->symbol_val; + parsed = true; + break; + } + } + if (!parsed) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("invalid value for enum option \"%s\": %s", + option->gen->name, value), + optenum->detailmsg ? + errdetail_internal("%s", _(optenum->detailmsg)) : 0)); + } + break; + case OPTION_TYPE_STRING: + { + option_spec_string *optstring = + (option_spec_string *) option->gen; + + option->values.string_val = value; + if (validate && optstring->validate_cb) + (optstring->validate_cb) (value); + parsed = true; + } + break; + default: + elog(ERROR, "unsupported reloption type %d", (int) option->gen->type); + parsed = true; /* quiet compiler */ + break; + } + if (validate && option->gen->postvalidate_fn) + option->gen->postvalidate_fn(option); + + if (parsed) + option->status = OPTION_VALUE_STATUS_PARSED; + +} + +/* + * optionsAllocateBytea + * Allocates memory for Bytea options representation + * + * We need special function for this, as string option values are embedded into + * Bytea object, stored at the rear part of the memory chunk. Thus we need to + * allocate extra memory, so all string values would fit in. This function + * calculates required size and do allocation. + * + * See "OPTION STRING VALUE NOTION" at the beginning of the file for better + * understanding. + */ +static void * +optionsAllocateBytea(options_spec_set *spec_set, List *options) +{ + Size size; + int i; + ListCell *cell; + int length; + void *res; + + size = spec_set->struct_size; + + /* Calculate size needed to store all string values for this option */ + for (i = 0; i < spec_set->num; i++) + { + option_spec_basic *opt_spec = spec_set->definitions[i]; + option_spec_string *opt_spec_str; + bool found = false; + option_value *option; + const char *val = NULL; + + /* Not interested in non-string options, skipping */ + if (opt_spec->type != OPTION_TYPE_STRING) + continue; + + /* + * Trying to find option_value that references opt_spec entry + */ + opt_spec_str = (option_spec_string *) opt_spec; + foreach(cell, options) + { + option = (option_value *) lfirst(cell); + if (option->status == OPTION_VALUE_STATUS_PARSED && + strcmp(option->gen->name, opt_spec->name) == 0) + { + found = true; + break; + } + } + if (found) + val = option->values.string_val; + else + val = opt_spec_str->default_val; + + if (opt_spec_str->fill_cb) + length = opt_spec_str->fill_cb(val, NULL); + else if (val) + length = strlen(val) + 1; + else + length = 0; /* "Default Value is NULL" case */ + + /* Add total length of each string values to basic size */ + size += length; + } + + res = palloc0(size); + SET_VARSIZE(res, size); + return res; +} + +/* + * optionsValuesToBytea + * Converts options values from Value List representation to Bytea + * representation. + * + * Fills resulting Bytea with option values according to Option Sec Set. + * + * For understanding processing of the string option please read "OPTION STRING + * VALUE NOTION" at the beginning of the file. + */ +bytea * +optionsValuesToBytea(List *options, options_spec_set *spec_set) +{ + char *data; + char *string_values_buffer; + int i; + + data = optionsAllocateBytea(spec_set, options); + + /* place for string data starts right after original structure */ + string_values_buffer = data + spec_set->struct_size; + + for (i = 0; i < spec_set->num; i++) + { + option_value *found = NULL; + ListCell *cell; + char *item_pos; + option_spec_basic *opt_spec = spec_set->definitions[i]; + + if (opt_spec->struct_offset < 0) + continue; /* This option value should not be stored in + * Bytea for some reason. May be it is + * deprecated and has warning or error in + * postvalidate function */ + + /* Calculate the position of the item inside the structure */ + item_pos = data + opt_spec->struct_offset; + + /* Looking for the corresponding option from options list */ + foreach(cell, options) + { + option_value *option = (option_value *) lfirst(cell); + + if (option->status == OPTION_VALUE_STATUS_RAW) + continue; /* raw can come from db. Just ignore them then */ + Assert(option->status != OPTION_VALUE_STATUS_EMPTY); + + if (strcmp(opt_spec->name, option->gen->name) == 0) + { + found = option; + break; + } + } + /* writing to the proper position either option value or default val */ + switch (opt_spec->type) + { + case OPTION_TYPE_BOOL: + *(bool *) item_pos = found ? + found->values.bool_val : + ((option_spec_bool *) opt_spec)->default_val; + break; + case OPTION_TYPE_INT: + *(int *) item_pos = found ? + found->values.int_val : + ((option_spec_int *) opt_spec)->default_val; + break; + case OPTION_TYPE_REAL: + *(double *) item_pos = found ? + found->values.real_val : + ((option_spec_real *) opt_spec)->default_val; + break; + case OPTION_TYPE_ENUM: + *(int *) item_pos = found ? + found->values.enum_val : + ((option_spec_enum *) opt_spec)->default_val; + break; + + case OPTION_TYPE_STRING: + { + /* + * For string options: writing string value at the string + * buffer after the structure, and storing and offset to + * that value + */ + char *value = NULL; + option_spec_string *opt_spec_str = + (option_spec_string *) opt_spec; + + if (found) + value = found->values.string_val; + else + value = opt_spec_str->default_val; + + if (opt_spec_str->fill_cb) + { + Size size = + opt_spec_str->fill_cb(value, string_values_buffer); + + if (size) + { + *(int *) item_pos = string_values_buffer - data; + string_values_buffer += size; + } + else + *(int *) item_pos = + OPTION_STRING_VALUE_NOT_SET_OFFSET; + } + else + { + *(int *) item_pos = value ? + string_values_buffer - data : + OPTION_STRING_VALUE_NOT_SET_OFFSET; + if (value) + { + strcpy(string_values_buffer, value); + string_values_buffer += strlen(value) + 1; + } + } + } + break; + default: + elog(ERROR, "unsupported reloption type %d", + (int) opt_spec->type); + break; + } + } + return (void *) data; +} + +/* + * optionDefListToTextArray + * Converts options from defList to TEXT[] representation. + * + * Used when new relation (or other object) is created. defList comes from + * SQL syntax parser, TEXT[] goes to DB storage. Options are always validated + * while conversion. + */ +Datum +optionDefListToTextArray(options_spec_set *spec_set, List *defList) +{ + Datum result; + List *new_values; + + /* Parse and validate new values */ + new_values = optionsDefListToRawValues(defList, false); + new_values = optionsParseRawValues(new_values, spec_set, true); + + /* Some checks can be done in postprocess_fun, we should call it */ + if (spec_set->postprocess_fun) + { + bytea *data; + if (defList) + data = optionsValuesToBytea(new_values, spec_set); + else + data = NULL; + + spec_set->postprocess_fun(data, true); + if (data) pfree(data); + } + result = optionsValuesToTextArray(new_values); + return result; +} + +/* + * optionsUpdateTexArrayWithDefList + * Modifies oldOptions values (in TEXT[] representation) with defList + * values. + * + * Old values are appened or replaced with new values if do_reset flag is set + * to false. If do_reset is set to true, defList specify the list of the options + * that should be removed from original list. + */ + +Datum +optionsUpdateTexArrayWithDefList(options_spec_set *spec_set, Datum oldOptions, + List *defList, bool do_reset) +{ + Datum result; + List *new_values; + List *old_values; + List *merged_values; + + /* + * Parse and validate New values + */ + new_values = optionsDefListToRawValues(defList, do_reset); + if (!do_reset) + new_values = optionsParseRawValues(new_values, spec_set, true); + + if (PointerIsValid(DatumGetPointer(oldOptions))) + { + old_values = optionsTextArrayToRawValues(oldOptions); + merged_values = optionsMergeOptionValues(old_values, new_values); + } + else + { + if (do_reset) + merged_values = NULL; /* return nothing */ + else + merged_values = new_values; + } + + /* + * If we have postprocess_fun function defined in spec_set, then there + * might be some custom options checks there, with error throwing. So we + * should do it here to throw these errors while CREATing or ALTERing + * options + */ + if (spec_set->postprocess_fun) + { + bytea *data = optionsValuesToBytea(merged_values, spec_set); + + spec_set->postprocess_fun(data, true); + pfree(data); + } + + /* + * Convert options to TextArray format so caller can store them into + * database + */ + result = optionsValuesToTextArray(merged_values); + return result; +} + +/* + * optionsTextArrayToBytea + * Convert options values from TEXT[] representation into Bytea + * representation + * + * This function uses other conversion function to get desired result. + */ +bytea * +optionsTextArrayToBytea(options_spec_set *spec_set, Datum data, bool validate) +{ + List *values; + bytea *options; + + values = optionsTextArrayToRawValues(data); + values = optionsParseRawValues(values, spec_set, validate); + options = optionsValuesToBytea(values, spec_set); + + if (spec_set->postprocess_fun) + spec_set->postprocess_fun(options, false); + return options; +} diff --git a/src/backend/access/common/reloptions.c b/src/backend/access/common/reloptions.c index d6eb5d8559..7ec05df7b9 100644 --- a/src/backend/access/common/reloptions.c +++ b/src/backend/access/common/reloptions.c @@ -1,7 +1,7 @@ /*------------------------------------------------------------------------- * * reloptions.c - * Core support for relation options (pg_class.reloptions) + * Support for relation options (pg_class.reloptions) * * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group * Portions Copyright (c) 1994, Regents of the University of California @@ -17,13 +17,10 @@ #include <float.h> -#include "access/gist_private.h" -#include "access/hash.h" #include "access/heaptoast.h" #include "access/htup_details.h" -#include "access/nbtree.h" #include "access/reloptions.h" -#include "access/spgist_private.h" +#include "access/options.h" #include "catalog/pg_type.h" #include "commands/defrem.h" #include "commands/tablespace.h" @@ -34,6 +31,7 @@ #include "utils/guc.h" #include "utils/memutils.h" #include "utils/rel.h" +#include "storage/bufmgr.h" /* * Contents of pg_class.reloptions @@ -91,389 +89,8 @@ * value has no effect until the next VACUUM, so no need for stronger lock. */ -static relopt_bool boolRelOpts[] = -{ - { - { - "autosummarize", - "Enables automatic summarization on this BRIN index", - RELOPT_KIND_BRIN, - AccessExclusiveLock - }, - false - }, - { - { - "autovacuum_enabled", - "Enables autovacuum in this relation", - RELOPT_KIND_HEAP | RELOPT_KIND_TOAST, - ShareUpdateExclusiveLock - }, - true - }, - { - { - "user_catalog_table", - "Declare a table as an additional catalog table, e.g. for the purpose of logical replication", - RELOPT_KIND_HEAP, - AccessExclusiveLock - }, - false - }, - { - { - "fastupdate", - "Enables \"fast update\" feature for this GIN index", - RELOPT_KIND_GIN, - AccessExclusiveLock - }, - true - }, - { - { - "security_barrier", - "View acts as a row security barrier", - RELOPT_KIND_VIEW, - AccessExclusiveLock - }, - false - }, - { - { - "security_invoker", - "Privileges on underlying relations are checked as the invoking user, not the view owner", - RELOPT_KIND_VIEW, - AccessExclusiveLock - }, - false - }, - { - { - "vacuum_truncate", - "Enables vacuum to truncate empty pages at the end of this table", - RELOPT_KIND_HEAP | RELOPT_KIND_TOAST, - ShareUpdateExclusiveLock - }, - true - }, - { - { - "deduplicate_items", - "Enables \"deduplicate items\" feature for this btree index", - RELOPT_KIND_BTREE, - ShareUpdateExclusiveLock /* since it applies only to later - * inserts */ - }, - true - }, - /* list terminator */ - {{NULL}} -}; - -static relopt_int intRelOpts[] = -{ - { - { - "fillfactor", - "Packs table pages only to this percentage", - RELOPT_KIND_HEAP, - ShareUpdateExclusiveLock /* since it applies only to later - * inserts */ - }, - HEAP_DEFAULT_FILLFACTOR, HEAP_MIN_FILLFACTOR, 100 - }, - { - { - "fillfactor", - "Packs btree index pages only to this percentage", - RELOPT_KIND_BTREE, - ShareUpdateExclusiveLock /* since it applies only to later - * inserts */ - }, - BTREE_DEFAULT_FILLFACTOR, BTREE_MIN_FILLFACTOR, 100 - }, - { - { - "fillfactor", - "Packs hash index pages only to this percentage", - RELOPT_KIND_HASH, - ShareUpdateExclusiveLock /* since it applies only to later - * inserts */ - }, - HASH_DEFAULT_FILLFACTOR, HASH_MIN_FILLFACTOR, 100 - }, - { - { - "fillfactor", - "Packs gist index pages only to this percentage", - RELOPT_KIND_GIST, - ShareUpdateExclusiveLock /* since it applies only to later - * inserts */ - }, - GIST_DEFAULT_FILLFACTOR, GIST_MIN_FILLFACTOR, 100 - }, - { - { - "fillfactor", - "Packs spgist index pages only to this percentage", - RELOPT_KIND_SPGIST, - ShareUpdateExclusiveLock /* since it applies only to later - * inserts */ - }, - SPGIST_DEFAULT_FILLFACTOR, SPGIST_MIN_FILLFACTOR, 100 - }, - { - { - "autovacuum_vacuum_threshold", - "Minimum number of tuple updates or deletes prior to vacuum", - RELOPT_KIND_HEAP | RELOPT_KIND_TOAST, - ShareUpdateExclusiveLock - }, - -1, 0, INT_MAX - }, - { - { - "autovacuum_vacuum_insert_threshold", - "Minimum number of tuple inserts prior to vacuum, or -1 to disable insert vacuums", - RELOPT_KIND_HEAP | RELOPT_KIND_TOAST, - ShareUpdateExclusiveLock - }, - -2, -1, INT_MAX - }, - { - { - "autovacuum_analyze_threshold", - "Minimum number of tuple inserts, updates or deletes prior to analyze", - RELOPT_KIND_HEAP, - ShareUpdateExclusiveLock - }, - -1, 0, INT_MAX - }, - { - { - "autovacuum_vacuum_cost_limit", - "Vacuum cost amount available before napping, for autovacuum", - RELOPT_KIND_HEAP | RELOPT_KIND_TOAST, - ShareUpdateExclusiveLock - }, - -1, 1, 10000 - }, - { - { - "autovacuum_freeze_min_age", - "Minimum age at which VACUUM should freeze a table row, for autovacuum", - RELOPT_KIND_HEAP | RELOPT_KIND_TOAST, - ShareUpdateExclusiveLock - }, - -1, 0, 1000000000 - }, - { - { - "autovacuum_multixact_freeze_min_age", - "Minimum multixact age at which VACUUM should freeze a row multixact's, for autovacuum", - RELOPT_KIND_HEAP | RELOPT_KIND_TOAST, - ShareUpdateExclusiveLock - }, - -1, 0, 1000000000 - }, - { - { - "autovacuum_freeze_max_age", - "Age at which to autovacuum a table to prevent transaction ID wraparound", - RELOPT_KIND_HEAP | RELOPT_KIND_TOAST, - ShareUpdateExclusiveLock - }, - -1, 100000, 2000000000 - }, - { - { - "autovacuum_multixact_freeze_max_age", - "Multixact age at which to autovacuum a table to prevent multixact wraparound", - RELOPT_KIND_HEAP | RELOPT_KIND_TOAST, - ShareUpdateExclusiveLock - }, - -1, 10000, 2000000000 - }, - { - { - "autovacuum_freeze_table_age", - "Age at which VACUUM should perform a full table sweep to freeze row versions", - RELOPT_KIND_HEAP | RELOPT_KIND_TOAST, - ShareUpdateExclusiveLock - }, -1, 0, 2000000000 - }, - { - { - "autovacuum_multixact_freeze_table_age", - "Age of multixact at which VACUUM should perform a full table sweep to freeze row versions", - RELOPT_KIND_HEAP | RELOPT_KIND_TOAST, - ShareUpdateExclusiveLock - }, -1, 0, 2000000000 - }, - { - { - "log_autovacuum_min_duration", - "Sets the minimum execution time above which autovacuum actions will be logged", - RELOPT_KIND_HEAP | RELOPT_KIND_TOAST, - ShareUpdateExclusiveLock - }, - -1, -1, INT_MAX - }, - { - { - "toast_tuple_target", - "Sets the target tuple length at which external columns will be toasted", - RELOPT_KIND_HEAP, - ShareUpdateExclusiveLock - }, - TOAST_TUPLE_TARGET, 128, TOAST_TUPLE_TARGET_MAIN - }, - { - { - "pages_per_range", - "Number of pages that each page range covers in a BRIN index", - RELOPT_KIND_BRIN, - AccessExclusiveLock - }, 128, 1, 131072 - }, - { - { - "gin_pending_list_limit", - "Maximum size of the pending list for this GIN index, in kilobytes.", - RELOPT_KIND_GIN, - AccessExclusiveLock - }, - -1, 64, MAX_KILOBYTES - }, - { - { - "effective_io_concurrency", - "Number of simultaneous requests that can be handled efficiently by the disk subsystem.", - RELOPT_KIND_TABLESPACE, - ShareUpdateExclusiveLock - }, -#ifdef USE_PREFETCH - -1, 0, MAX_IO_CONCURRENCY -#else - 0, 0, 0 -#endif - }, - { - { - "maintenance_io_concurrency", - "Number of simultaneous requests that can be handled efficiently by the disk subsystem for maintenance work.", - RELOPT_KIND_TABLESPACE, - ShareUpdateExclusiveLock - }, -#ifdef USE_PREFETCH - -1, 0, MAX_IO_CONCURRENCY -#else - 0, 0, 0 -#endif - }, - { - { - "parallel_workers", - "Number of parallel processes that can be used per executor node for this relation.", - RELOPT_KIND_HEAP, - ShareUpdateExclusiveLock - }, - -1, 0, 1024 - }, - - /* list terminator */ - {{NULL}} -}; - -static relopt_real realRelOpts[] = -{ - { - { - "autovacuum_vacuum_cost_delay", - "Vacuum cost delay in milliseconds, for autovacuum", - RELOPT_KIND_HEAP | RELOPT_KIND_TOAST, - ShareUpdateExclusiveLock - }, - -1, 0.0, 100.0 - }, - { - { - "autovacuum_vacuum_scale_factor", - "Number of tuple updates or deletes prior to vacuum as a fraction of reltuples", - RELOPT_KIND_HEAP | RELOPT_KIND_TOAST, - ShareUpdateExclusiveLock - }, - -1, 0.0, 100.0 - }, - { - { - "autovacuum_vacuum_insert_scale_factor", - "Number of tuple inserts prior to vacuum as a fraction of reltuples", - RELOPT_KIND_HEAP | RELOPT_KIND_TOAST, - ShareUpdateExclusiveLock - }, - -1, 0.0, 100.0 - }, - { - { - "autovacuum_analyze_scale_factor", - "Number of tuple inserts, updates or deletes prior to analyze as a fraction of reltuples", - RELOPT_KIND_HEAP, - ShareUpdateExclusiveLock - }, - -1, 0.0, 100.0 - }, - { - { - "seq_page_cost", - "Sets the planner's estimate of the cost of a sequentially fetched disk page.", - RELOPT_KIND_TABLESPACE, - ShareUpdateExclusiveLock - }, - -1, 0.0, DBL_MAX - }, - { - { - "random_page_cost", - "Sets the planner's estimate of the cost of a nonsequentially fetched disk page.", - RELOPT_KIND_TABLESPACE, - ShareUpdateExclusiveLock - }, - -1, 0.0, DBL_MAX - }, - { - { - "n_distinct", - "Sets the planner's estimate of the number of distinct values appearing in a column (excluding child relations).", - RELOPT_KIND_ATTRIBUTE, - ShareUpdateExclusiveLock - }, - 0, -1.0, DBL_MAX - }, - { - { - "n_distinct_inherited", - "Sets the planner's estimate of the number of distinct values appearing in a column (including child relations).", - RELOPT_KIND_ATTRIBUTE, - ShareUpdateExclusiveLock - }, - 0, -1.0, DBL_MAX - }, - { - { - "vacuum_cleanup_index_scale_factor", - "Deprecated B-Tree parameter.", - RELOPT_KIND_BTREE, - ShareUpdateExclusiveLock - }, - -1, 0.0, 1e10 - }, - /* list terminator */ - {{NULL}} -}; - /* values from StdRdOptIndexCleanup */ -static relopt_enum_elt_def StdRdOptIndexCleanupValues[] = +static opt_enum_elt_def StdRdOptIndexCleanupValues[] = { {"auto", STDRD_OPTION_VACUUM_INDEX_CLEANUP_AUTO}, {"on", STDRD_OPTION_VACUUM_INDEX_CLEANUP_ON}, @@ -487,17 +104,8 @@ static relopt_enum_elt_def StdRdOptIndexCleanupValues[] = {(const char *) NULL} /* list terminator */ }; -/* values from GistOptBufferingMode */ -static relopt_enum_elt_def gistBufferingOptValues[] = -{ - {"auto", GIST_OPTION_BUFFERING_AUTO}, - {"on", GIST_OPTION_BUFFERING_ON}, - {"off", GIST_OPTION_BUFFERING_OFF}, - {(const char *) NULL} /* list terminator */ -}; - /* values from ViewOptCheckOption */ -static relopt_enum_elt_def viewCheckOptValues[] = +static opt_enum_elt_def viewCheckOptValues[] = { /* no value for NOT_SET */ {"local", VIEW_OPTION_CHECK_OPTION_LOCAL}, @@ -505,225 +113,9 @@ static relopt_enum_elt_def viewCheckOptValues[] = {(const char *) NULL} /* list terminator */ }; -static relopt_enum enumRelOpts[] = -{ - { - { - "vacuum_index_cleanup", - "Controls index vacuuming and index cleanup", - RELOPT_KIND_HEAP | RELOPT_KIND_TOAST, - ShareUpdateExclusiveLock - }, - StdRdOptIndexCleanupValues, - STDRD_OPTION_VACUUM_INDEX_CLEANUP_AUTO, - gettext_noop("Valid values are \"on\", \"off\", and \"auto\".") - }, - { - { - "buffering", - "Enables buffering build for this GiST index", - RELOPT_KIND_GIST, - AccessExclusiveLock - }, - gistBufferingOptValues, - GIST_OPTION_BUFFERING_AUTO, - gettext_noop("Valid values are \"on\", \"off\", and \"auto\".") - }, - { - { - "check_option", - "View has WITH CHECK OPTION defined (local or cascaded).", - RELOPT_KIND_VIEW, - AccessExclusiveLock - }, - viewCheckOptValues, - VIEW_OPTION_CHECK_OPTION_NOT_SET, - gettext_noop("Valid values are \"local\" and \"cascaded\".") - }, - /* list terminator */ - {{NULL}} -}; - -static relopt_string stringRelOpts[] = -{ - /* list terminator */ - {{NULL}} -}; -static relopt_gen **relOpts = NULL; -static bits32 last_assigned_kind = RELOPT_KIND_LAST_DEFAULT; - -static int num_custom_options = 0; -static relopt_gen **custom_options = NULL; -static bool need_initialization = true; - -static void initialize_reloptions(void); -static void parse_one_reloption(relopt_value *option, char *text_str, - int text_len, bool validate); - -/* - * Get the length of a string reloption (either default or the user-defined - * value). This is used for allocation purposes when building a set of - * relation options. - */ -#define GET_STRING_RELOPTION_LEN(option) \ - ((option).isset ? strlen((option).values.string_val) : \ - ((relopt_string *) (option).gen)->default_len) - -/* - * initialize_reloptions - * initialization routine, must be called before parsing - * - * Initialize the relOpts array and fill each variable's type and name length. - */ -static void -initialize_reloptions(void) -{ - int i; - int j; - - j = 0; - for (i = 0; boolRelOpts[i].gen.name; i++) - { - Assert(DoLockModesConflict(boolRelOpts[i].gen.lockmode, - boolRelOpts[i].gen.lockmode)); - j++; - } - for (i = 0; intRelOpts[i].gen.name; i++) - { - Assert(DoLockModesConflict(intRelOpts[i].gen.lockmode, - intRelOpts[i].gen.lockmode)); - j++; - } - for (i = 0; realRelOpts[i].gen.name; i++) - { - Assert(DoLockModesConflict(realRelOpts[i].gen.lockmode, - realRelOpts[i].gen.lockmode)); - j++; - } - for (i = 0; enumRelOpts[i].gen.name; i++) - { - Assert(DoLockModesConflict(enumRelOpts[i].gen.lockmode, - enumRelOpts[i].gen.lockmode)); - j++; - } - for (i = 0; stringRelOpts[i].gen.name; i++) - { - Assert(DoLockModesConflict(stringRelOpts[i].gen.lockmode, - stringRelOpts[i].gen.lockmode)); - j++; - } - j += num_custom_options; - - if (relOpts) - pfree(relOpts); - relOpts = MemoryContextAlloc(TopMemoryContext, - (j + 1) * sizeof(relopt_gen *)); - - j = 0; - for (i = 0; boolRelOpts[i].gen.name; i++) - { - relOpts[j] = &boolRelOpts[i].gen; - relOpts[j]->type = RELOPT_TYPE_BOOL; - relOpts[j]->namelen = strlen(relOpts[j]->name); - j++; - } - - for (i = 0; intRelOpts[i].gen.name; i++) - { - relOpts[j] = &intRelOpts[i].gen; - relOpts[j]->type = RELOPT_TYPE_INT; - relOpts[j]->namelen = strlen(relOpts[j]->name); - j++; - } - - for (i = 0; realRelOpts[i].gen.name; i++) - { - relOpts[j] = &realRelOpts[i].gen; - relOpts[j]->type = RELOPT_TYPE_REAL; - relOpts[j]->namelen = strlen(relOpts[j]->name); - j++; - } - - for (i = 0; enumRelOpts[i].gen.name; i++) - { - relOpts[j] = &enumRelOpts[i].gen; - relOpts[j]->type = RELOPT_TYPE_ENUM; - relOpts[j]->namelen = strlen(relOpts[j]->name); - j++; - } - - for (i = 0; stringRelOpts[i].gen.name; i++) - { - relOpts[j] = &stringRelOpts[i].gen; - relOpts[j]->type = RELOPT_TYPE_STRING; - relOpts[j]->namelen = strlen(relOpts[j]->name); - j++; - } - - for (i = 0; i < num_custom_options; i++) - { - relOpts[j] = custom_options[i]; - j++; - } - - /* add a list terminator */ - relOpts[j] = NULL; - - /* flag the work is complete */ - need_initialization = false; -} - -/* - * add_reloption_kind - * Create a new relopt_kind value, to be used in custom reloptions by - * user-defined AMs. - */ -relopt_kind -add_reloption_kind(void) -{ - /* don't hand out the last bit so that the enum's behavior is portable */ - if (last_assigned_kind >= RELOPT_KIND_MAX) - ereport(ERROR, - (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED), - errmsg("user-defined relation parameter types limit exceeded"))); - last_assigned_kind <<= 1; - return (relopt_kind) last_assigned_kind; -} - -/* - * add_reloption - * Add an already-created custom reloption to the list, and recompute the - * main parser table. - */ -static void -add_reloption(relopt_gen *newoption) -{ - static int max_custom_options = 0; - - if (num_custom_options >= max_custom_options) - { - MemoryContext oldcxt; - - oldcxt = MemoryContextSwitchTo(TopMemoryContext); - - if (max_custom_options == 0) - { - max_custom_options = 8; - custom_options = palloc(max_custom_options * sizeof(relopt_gen *)); - } - else - { - max_custom_options *= 2; - custom_options = repalloc(custom_options, - max_custom_options * sizeof(relopt_gen *)); - } - MemoryContextSwitchTo(oldcxt); - } - custom_options[num_custom_options++] = newoption; - - need_initialization = true; -} +options_spec_set *get_stdrd_relopt_spec_set(bool is_heap); +void oid_postvalidate(option_value *value); /* * init_local_reloptions @@ -733,9 +125,8 @@ add_reloption(relopt_gen *newoption) void init_local_reloptions(local_relopts *relopts, Size relopt_struct_size) { - relopts->options = NIL; relopts->validators = NIL; - relopts->relopt_struct_size = relopt_struct_size; + relopts->spec_set = allocateOptionsSpecSet(NULL, relopt_struct_size, true, 0); } /* @@ -749,112 +140,6 @@ register_reloptions_validator(local_relopts *relopts, relopts_validator validato relopts->validators = lappend(relopts->validators, validator); } -/* - * add_local_reloption - * Add an already-created custom reloption to the local list. - */ -static void -add_local_reloption(local_relopts *relopts, relopt_gen *newoption, int offset) -{ - local_relopt *opt = palloc(sizeof(*opt)); - - Assert(offset < relopts->relopt_struct_size); - - opt->option = newoption; - opt->offset = offset; - - relopts->options = lappend(relopts->options, opt); -} - -/* - * allocate_reloption - * Allocate a new reloption and initialize the type-agnostic fields - * (for types other than string) - */ -static relopt_gen * -allocate_reloption(bits32 kinds, int type, const char *name, const char *desc, - LOCKMODE lockmode) -{ - MemoryContext oldcxt; - size_t size; - relopt_gen *newoption; - - if (kinds != RELOPT_KIND_LOCAL) - oldcxt = MemoryContextSwitchTo(TopMemoryContext); - else - oldcxt = NULL; - - switch (type) - { - case RELOPT_TYPE_BOOL: - size = sizeof(relopt_bool); - break; - case RELOPT_TYPE_INT: - size = sizeof(relopt_int); - break; - case RELOPT_TYPE_REAL: - size = sizeof(relopt_real); - break; - case RELOPT_TYPE_ENUM: - size = sizeof(relopt_enum); - break; - case RELOPT_TYPE_STRING: - size = sizeof(relopt_string); - break; - default: - elog(ERROR, "unsupported reloption type %d", type); - return NULL; /* keep compiler quiet */ - } - - newoption = palloc(size); - - newoption->name = pstrdup(name); - if (desc) - newoption->desc = pstrdup(desc); - else - newoption->desc = NULL; - newoption->kinds = kinds; - newoption->namelen = strlen(name); - newoption->type = type; - newoption->lockmode = lockmode; - - if (oldcxt != NULL) - MemoryContextSwitchTo(oldcxt); - - return newoption; -} - -/* - * init_bool_reloption - * Allocate and initialize a new boolean reloption - */ -static relopt_bool * -init_bool_reloption(bits32 kinds, const char *name, const char *desc, - bool default_val, LOCKMODE lockmode) -{ - relopt_bool *newoption; - - newoption = (relopt_bool *) allocate_reloption(kinds, RELOPT_TYPE_BOOL, - name, desc, lockmode); - newoption->default_val = default_val; - - return newoption; -} - -/* - * add_bool_reloption - * Add a new boolean reloption - */ -void -add_bool_reloption(bits32 kinds, const char *name, const char *desc, - bool default_val, LOCKMODE lockmode) -{ - relopt_bool *newoption = init_bool_reloption(kinds, name, desc, - default_val, lockmode); - - add_reloption((relopt_gen *) newoption); -} - /* * add_local_bool_reloption * Add a new boolean local reloption @@ -865,47 +150,8 @@ void add_local_bool_reloption(local_relopts *relopts, const char *name, const char *desc, bool default_val, int offset) { - relopt_bool *newoption = init_bool_reloption(RELOPT_KIND_LOCAL, - name, desc, - default_val, 0); - - add_local_reloption(relopts, (relopt_gen *) newoption, offset); -} - - -/* - * init_real_reloption - * Allocate and initialize a new integer reloption - */ -static relopt_int * -init_int_reloption(bits32 kinds, const char *name, const char *desc, - int default_val, int min_val, int max_val, - LOCKMODE lockmode) -{ - relopt_int *newoption; - - newoption = (relopt_int *) allocate_reloption(kinds, RELOPT_TYPE_INT, - name, desc, lockmode); - newoption->default_val = default_val; - newoption->min = min_val; - newoption->max = max_val; - - return newoption; -} - -/* - * add_int_reloption - * Add a new integer reloption - */ -void -add_int_reloption(bits32 kinds, const char *name, const char *desc, int default_val, - int min_val, int max_val, LOCKMODE lockmode) -{ - relopt_int *newoption = init_int_reloption(kinds, name, desc, - default_val, min_val, - max_val, lockmode); - - add_reloption((relopt_gen *) newoption); + optionsSpecSetAddBool(relopts->spec_set, name, desc, NoLock, offset, NULL, + default_val); } /* @@ -919,47 +165,8 @@ add_local_int_reloption(local_relopts *relopts, const char *name, const char *desc, int default_val, int min_val, int max_val, int offset) { - relopt_int *newoption = init_int_reloption(RELOPT_KIND_LOCAL, - name, desc, default_val, - min_val, max_val, 0); - - add_local_reloption(relopts, (relopt_gen *) newoption, offset); -} - -/* - * init_real_reloption - * Allocate and initialize a new real reloption - */ -static relopt_real * -init_real_reloption(bits32 kinds, const char *name, const char *desc, - double default_val, double min_val, double max_val, - LOCKMODE lockmode) -{ - relopt_real *newoption; - - newoption = (relopt_real *) allocate_reloption(kinds, RELOPT_TYPE_REAL, - name, desc, lockmode); - newoption->default_val = default_val; - newoption->min = min_val; - newoption->max = max_val; - - return newoption; -} - -/* - * add_real_reloption - * Add a new float reloption - */ -void -add_real_reloption(bits32 kinds, const char *name, const char *desc, - double default_val, double min_val, double max_val, - LOCKMODE lockmode) -{ - relopt_real *newoption = init_real_reloption(kinds, name, desc, - default_val, min_val, - max_val, lockmode); - - add_reloption((relopt_gen *) newoption); + optionsSpecSetAddInt(relopts->spec_set, name, desc, NoLock, offset, NULL, + default_val, min_val, max_val); } /* @@ -969,142 +176,28 @@ add_real_reloption(bits32 kinds, const char *name, const char *desc, * 'offset' is offset of double-typed field. */ void -add_local_real_reloption(local_relopts *relopts, const char *name, - const char *desc, double default_val, - double min_val, double max_val, int offset) -{ - relopt_real *newoption = init_real_reloption(RELOPT_KIND_LOCAL, - name, desc, - default_val, min_val, - max_val, 0); - - add_local_reloption(relopts, (relopt_gen *) newoption, offset); -} - -/* - * init_enum_reloption - * Allocate and initialize a new enum reloption - */ -static relopt_enum * -init_enum_reloption(bits32 kinds, const char *name, const char *desc, - relopt_enum_elt_def *members, int default_val, - const char *detailmsg, LOCKMODE lockmode) -{ - relopt_enum *newoption; - - newoption = (relopt_enum *) allocate_reloption(kinds, RELOPT_TYPE_ENUM, - name, desc, lockmode); - newoption->members = members; - newoption->default_val = default_val; - newoption->detailmsg = detailmsg; - - return newoption; -} - - -/* - * add_enum_reloption - * Add a new enum reloption - * - * The members array must have a terminating NULL entry. - * - * The detailmsg is shown when unsupported values are passed, and has this - * form: "Valid values are \"foo\", \"bar\", and \"bar\"." - * - * The members array and detailmsg are not copied -- caller must ensure that - * they are valid throughout the life of the process. - */ -void -add_enum_reloption(bits32 kinds, const char *name, const char *desc, - relopt_enum_elt_def *members, int default_val, - const char *detailmsg, LOCKMODE lockmode) -{ - relopt_enum *newoption = init_enum_reloption(kinds, name, desc, - members, default_val, - detailmsg, lockmode); - - add_reloption((relopt_gen *) newoption); -} - -/* - * add_local_enum_reloption - * Add a new local enum reloption - * - * 'offset' is offset of int-typed field. - */ -void -add_local_enum_reloption(local_relopts *relopts, const char *name, - const char *desc, relopt_enum_elt_def *members, - int default_val, const char *detailmsg, int offset) -{ - relopt_enum *newoption = init_enum_reloption(RELOPT_KIND_LOCAL, - name, desc, - members, default_val, - detailmsg, 0); - - add_local_reloption(relopts, (relopt_gen *) newoption, offset); -} - -/* - * init_string_reloption - * Allocate and initialize a new string reloption - */ -static relopt_string * -init_string_reloption(bits32 kinds, const char *name, const char *desc, - const char *default_val, - validate_string_relopt validator, - fill_string_relopt filler, - LOCKMODE lockmode) -{ - relopt_string *newoption; - - /* make sure the validator/default combination is sane */ - if (validator) - (validator) (default_val); - - newoption = (relopt_string *) allocate_reloption(kinds, RELOPT_TYPE_STRING, - name, desc, lockmode); - newoption->validate_cb = validator; - newoption->fill_cb = filler; - if (default_val) - { - if (kinds == RELOPT_KIND_LOCAL) - newoption->default_val = strdup(default_val); - else - newoption->default_val = MemoryContextStrdup(TopMemoryContext, default_val); - newoption->default_len = strlen(default_val); - newoption->default_isnull = false; - } - else - { - newoption->default_val = ""; - newoption->default_len = 0; - newoption->default_isnull = true; - } +add_local_real_reloption(local_relopts *relopts, const char *name, + const char *desc, double default_val, + double min_val, double max_val, int offset) +{ + optionsSpecSetAddReal(relopts->spec_set, name, desc, NoLock, offset, NULL, + default_val, min_val, max_val); - return newoption; } /* - * add_string_reloption - * Add a new string reloption + * add_local_enum_reloption + * Add a new local enum reloption * - * "validator" is an optional function pointer that can be used to test the - * validity of the values. It must elog(ERROR) when the argument string is - * not acceptable for the variable. Note that the default value must pass - * the validation. + * 'offset' is offset of int-typed field. */ void -add_string_reloption(bits32 kinds, const char *name, const char *desc, - const char *default_val, validate_string_relopt validator, - LOCKMODE lockmode) +add_local_enum_reloption(local_relopts *relopts, const char *name, + const char *desc, opt_enum_elt_def *members, + int default_val, const char *detailmsg, int offset) { - relopt_string *newoption = init_string_reloption(kinds, name, desc, - default_val, - validator, NULL, - lockmode); - - add_reloption((relopt_gen *) newoption); + optionsSpecSetAddEnum(relopts->spec_set, name, desc, NoLock, offset, NULL, + members, default_val, detailmsg); } /* @@ -1120,247 +213,8 @@ add_local_string_reloption(local_relopts *relopts, const char *name, validate_string_relopt validator, fill_string_relopt filler, int offset) { - relopt_string *newoption = init_string_reloption(RELOPT_KIND_LOCAL, - name, desc, - default_val, - validator, filler, - 0); - - add_local_reloption(relopts, (relopt_gen *) newoption, offset); -} - -/* - * Transform a relation options list (list of DefElem) into the text array - * format that is kept in pg_class.reloptions, including only those options - * that are in the passed namespace. The output values do not include the - * namespace. - * - * This is used for three cases: CREATE TABLE/INDEX, ALTER TABLE SET, and - * ALTER TABLE RESET. In the ALTER cases, oldOptions is the existing - * reloptions value (possibly NULL), and we replace or remove entries - * as needed. - * - * If acceptOidsOff is true, then we allow oids = false, but throw error when - * on. This is solely needed for backwards compatibility. - * - * Note that this is not responsible for determining whether the options - * are valid, but it does check that namespaces for all the options given are - * listed in validnsps. The NULL namespace is always valid and need not be - * explicitly listed. Passing a NULL pointer means that only the NULL - * namespace is valid. - * - * Both oldOptions and the result are text arrays (or NULL for "default"), - * but we declare them as Datums to avoid including array.h in reloptions.h. - */ -Datum -transformRelOptions(Datum oldOptions, List *defList, const char *namspace, - char *validnsps[], bool acceptOidsOff, bool isReset) -{ - Datum result; - ArrayBuildState *astate; - ListCell *cell; - - /* no change if empty list */ - if (defList == NIL) - return oldOptions; - - /* We build new array using accumArrayResult */ - astate = NULL; - - /* Copy any oldOptions that aren't to be replaced */ - if (PointerIsValid(DatumGetPointer(oldOptions))) - { - ArrayType *array = DatumGetArrayTypeP(oldOptions); - Datum *oldoptions; - int noldoptions; - int i; - - deconstruct_array_builtin(array, TEXTOID, &oldoptions, NULL, &noldoptions); - - for (i = 0; i < noldoptions; i++) - { - char *text_str = VARDATA(oldoptions[i]); - int text_len = VARSIZE(oldoptions[i]) - VARHDRSZ; - - /* Search for a match in defList */ - foreach(cell, defList) - { - DefElem *def = (DefElem *) lfirst(cell); - int kw_len; - - /* ignore if not in the same namespace */ - if (namspace == NULL) - { - if (def->defnamespace != NULL) - continue; - } - else if (def->defnamespace == NULL) - continue; - else if (strcmp(def->defnamespace, namspace) != 0) - continue; - - kw_len = strlen(def->defname); - if (text_len > kw_len && text_str[kw_len] == '=' && - strncmp(text_str, def->defname, kw_len) == 0) - break; - } - if (!cell) - { - /* No match, so keep old option */ - astate = accumArrayResult(astate, oldoptions[i], - false, TEXTOID, - CurrentMemoryContext); - } - } - } - - /* - * If CREATE/SET, add new options to array; if RESET, just check that the - * user didn't say RESET (option=val). (Must do this because the grammar - * doesn't enforce it.) - */ - foreach(cell, defList) - { - DefElem *def = (DefElem *) lfirst(cell); - - if (isReset) - { - if (def->arg != NULL) - ereport(ERROR, - (errcode(ERRCODE_SYNTAX_ERROR), - errmsg("RESET must not include values for parameters"))); - } - else - { - text *t; - const char *value; - Size len; - - /* - * Error out if the namespace is not valid. A NULL namespace is - * always valid. - */ - if (def->defnamespace != NULL) - { - bool valid = false; - int i; - - if (validnsps) - { - for (i = 0; validnsps[i]; i++) - { - if (strcmp(def->defnamespace, validnsps[i]) == 0) - { - valid = true; - break; - } - } - } - - if (!valid) - ereport(ERROR, - (errcode(ERRCODE_INVALID_PARAMETER_VALUE), - errmsg("unrecognized parameter namespace \"%s\"", - def->defnamespace))); - } - - /* ignore if not in the same namespace */ - if (namspace == NULL) - { - if (def->defnamespace != NULL) - continue; - } - else if (def->defnamespace == NULL) - continue; - else if (strcmp(def->defnamespace, namspace) != 0) - continue; - - /* - * Flatten the DefElem into a text string like "name=arg". If we - * have just "name", assume "name=true" is meant. Note: the - * namespace is not output. - */ - if (def->arg != NULL) - value = defGetString(def); - else - value = "true"; - - /* - * This is not a great place for this test, but there's no other - * convenient place to filter the option out. As WITH (oids = - * false) will be removed someday, this seems like an acceptable - * amount of ugly. - */ - if (acceptOidsOff && def->defnamespace == NULL && - strcmp(def->defname, "oids") == 0) - { - if (defGetBoolean(def)) - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("tables declared WITH OIDS are not supported"))); - /* skip over option, reloptions machinery doesn't know it */ - continue; - } - - len = VARHDRSZ + strlen(def->defname) + 1 + strlen(value); - /* +1 leaves room for sprintf's trailing null */ - t = (text *) palloc(len + 1); - SET_VARSIZE(t, len); - sprintf(VARDATA(t), "%s=%s", def->defname, value); - - astate = accumArrayResult(astate, PointerGetDatum(t), - false, TEXTOID, - CurrentMemoryContext); - } - } - - if (astate) - result = makeArrayResult(astate, CurrentMemoryContext); - else - result = (Datum) 0; - - return result; -} - - -/* - * Convert the text-array format of reloptions into a List of DefElem. - * This is the inverse of transformRelOptions(). - */ -List * -untransformRelOptions(Datum options) -{ - List *result = NIL; - ArrayType *array; - Datum *optiondatums; - int noptions; - int i; - - /* Nothing to do if no options */ - if (!PointerIsValid(DatumGetPointer(options))) - return result; - - array = DatumGetArrayTypeP(options); - - deconstruct_array_builtin(array, TEXTOID, &optiondatums, NULL, &noptions); - - for (i = 0; i < noptions; i++) - { - char *s; - char *p; - Node *val = NULL; - - s = TextDatumGetCString(optiondatums[i]); - p = strchr(s, '='); - if (p) - { - *p++ = '\0'; - val = (Node *) makeString(p); - } - result = lappend(result, makeDefElem(s, val, -1)); - } - - return result; + optionsSpecSetAddString(relopts->spec_set, name, desc, NoLock, offset, + NULL, default_val, validator, filler); } /* @@ -1377,12 +231,13 @@ untransformRelOptions(Datum options) */ bytea * extractRelOptions(HeapTuple tuple, TupleDesc tupdesc, - amoptions_function amoptions) + amreloptspecset_function amoptionsspecsetfn) { bytea *options; bool isnull; Datum datum; Form_pg_class classForm; + options_spec_set *spec_set; datum = fastgetattr(tuple, Anum_pg_class_reloptions, @@ -1396,544 +251,326 @@ extractRelOptions(HeapTuple tuple, TupleDesc tupdesc, /* Parse into appropriate format; don't error out here */ switch (classForm->relkind) { - case RELKIND_RELATION: case RELKIND_TOASTVALUE: + spec_set = get_toast_relopt_spec_set(); + break; + case RELKIND_RELATION: case RELKIND_MATVIEW: - options = heap_reloptions(classForm->relkind, datum, false); + spec_set = get_heap_relopt_spec_set(); break; case RELKIND_PARTITIONED_TABLE: - options = partitioned_table_reloptions(datum, false); + spec_set = get_partitioned_relopt_spec_set(); break; case RELKIND_VIEW: - options = view_reloptions(datum, false); + spec_set = get_view_relopt_spec_set(); break; case RELKIND_INDEX: case RELKIND_PARTITIONED_INDEX: - options = index_reloptions(amoptions, datum, false); + if (amoptionsspecsetfn) + spec_set = amoptionsspecsetfn(); + else + spec_set = NULL; break; case RELKIND_FOREIGN_TABLE: - options = NULL; + spec_set = NULL; break; default: Assert(false); /* can't get here */ - options = NULL; /* keep compiler quiet */ + spec_set = NULL; /* keep compiler quiet */ break; } - + if (spec_set) + options = optionsTextArrayToBytea(spec_set, datum, 0); + else + options = NULL; return options; } -static void -parseRelOptionsInternal(Datum options, bool validate, - relopt_value *reloptions, int numoptions) +void +oid_postvalidate(option_value *value) { - ArrayType *array = DatumGetArrayTypeP(options); - Datum *optiondatums; - int noptions; - int i; - - deconstruct_array_builtin(array, TEXTOID, &optiondatums, NULL, &noptions); - - for (i = 0; i < noptions; i++) - { - char *text_str = VARDATA(optiondatums[i]); - int text_len = VARSIZE(optiondatums[i]) - VARHDRSZ; - int j; - - /* Search for a match in reloptions */ - for (j = 0; j < numoptions; j++) - { - int kw_len = reloptions[j].gen->namelen; - - if (text_len > kw_len && text_str[kw_len] == '=' && - strncmp(text_str, reloptions[j].gen->name, kw_len) == 0) - { - parse_one_reloption(&reloptions[j], text_str, text_len, - validate); - break; - } - } - - if (j >= numoptions && validate) - { - char *s; - char *p; - - s = TextDatumGetCString(optiondatums[i]); - p = strchr(s, '='); - if (p) - *p = '\0'; - ereport(ERROR, - (errcode(ERRCODE_INVALID_PARAMETER_VALUE), - errmsg("unrecognized parameter \"%s\"", s))); - } - } - - /* It's worth avoiding memory leaks in this function */ - pfree(optiondatums); - - if (((void *) array) != DatumGetPointer(options)) - pfree(array); + if (value->values.bool_val) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("tables declared WITH OIDS are not supported"))); } /* - * Interpret reloptions that are given in text-array format. + * Relation options and Lock levels: + * + * The default choice for any new option should be AccessExclusiveLock. + * In some cases the lock level can be reduced from there, but the lock + * level chosen should always conflict with itself to ensure that multiple + * changes aren't lost when we attempt concurrent changes. + * The choice of lock level depends completely upon how that parameter + * is used within the server, not upon how and when you'd like to change it. + * Safety first. Existing choices are documented here, and elsewhere in + * backend code where the parameters are used. + * + * In general, anything that affects the results obtained from a SELECT must be + * protected by AccessExclusiveLock. * - * options is a reloption text array as constructed by transformRelOptions. - * kind specifies the family of options to be processed. + * Autovacuum related parameters can be set at ShareUpdateExclusiveLock + * since they are only used by the AV procs and don't change anything + * currently executing. * - * The return value is a relopt_value * array on which the options actually - * set in the options array are marked with isset=true. The length of this - * array is returned in *numrelopts. Options not set are also present in the - * array; this is so that the caller can easily locate the default values. + * Fillfactor can be set because it applies only to subsequent changes made to + * data blocks, as documented in heapio.c * - * If there are no options of the given kind, numrelopts is set to 0 and NULL - * is returned (unless options are illegally supplied despite none being - * defined, in which case an error occurs). + * n_distinct options can be set at ShareUpdateExclusiveLock because they + * are only used during ANALYZE, which uses a ShareUpdateExclusiveLock, + * so the ANALYZE will not be affected by in-flight changes. Changing those + * values has no affect until the next ANALYZE, so no need for stronger lock. * - * Note: values of type int, bool and real are allocated as part of the - * returned array. Values of type string are allocated separately and must - * be freed by the caller. - */ -static relopt_value * -parseRelOptions(Datum options, bool validate, relopt_kind kind, - int *numrelopts) -{ - relopt_value *reloptions = NULL; - int numoptions = 0; - int i; - int j; - - if (need_initialization) - initialize_reloptions(); - - /* Build a list of expected options, based on kind */ - - for (i = 0; relOpts[i]; i++) - if (relOpts[i]->kinds & kind) - numoptions++; - - if (numoptions > 0) - { - reloptions = palloc(numoptions * sizeof(relopt_value)); - - for (i = 0, j = 0; relOpts[i]; i++) - { - if (relOpts[i]->kinds & kind) - { - reloptions[j].gen = relOpts[i]; - reloptions[j].isset = false; - j++; - } - } + * Planner-related parameters can be set with ShareUpdateExclusiveLock because + * they only affect planning and not the correctness of the execution. Plans + * cannot be changed in mid-flight, so changes here could not easily result in + * new improved plans in any case. So we allow existing queries to continue + * and existing plans to survive, a small price to pay for allowing better + * plans to be introduced concurrently without interfering with users. + * + * Setting parallel_workers is safe, since it acts the same as + * max_parallel_workers_per_gather which is a USERSET parameter that doesn't + * affect existing plans or queries. + */ + + +options_spec_set * +get_stdrd_relopt_spec_set(bool is_heap) +{ + options_spec_set *stdrd_relopt_spec_set = allocateOptionsSpecSet( + is_heap ? NULL : "toast", sizeof(StdRdOptions), false, 0); + + if (is_heap) + optionsSpecSetAddInt(stdrd_relopt_spec_set, "fillfactor", + "Packs table pages only to this percentag", + ShareUpdateExclusiveLock, /* since it applies only + * to later inserts */ + offsetof(StdRdOptions, fillfactor), NULL, + HEAP_DEFAULT_FILLFACTOR, HEAP_MIN_FILLFACTOR, 100); + + optionsSpecSetAddBool(stdrd_relopt_spec_set, "autovacuum_enabled", + "Enables autovacuum in this relation", + ShareUpdateExclusiveLock, + offsetof(StdRdOptions, autovacuum) + + offsetof(AutoVacOpts, enabled), + NULL, true); + + optionsSpecSetAddInt(stdrd_relopt_spec_set, "autovacuum_vacuum_threshold", + "Minimum number of tuple updates or deletes prior to vacuum", + ShareUpdateExclusiveLock, + offsetof(StdRdOptions, autovacuum) + + offsetof(AutoVacOpts, vacuum_threshold), + NULL, -1, 0, INT_MAX); + + if (is_heap) + optionsSpecSetAddInt(stdrd_relopt_spec_set, "autovacuum_analyze_threshold", + "Minimum number of tuple updates or deletes prior to vacuum", + ShareUpdateExclusiveLock, + offsetof(StdRdOptions, autovacuum) + + offsetof(AutoVacOpts, analyze_threshold), + NULL, -1, 0, INT_MAX); + + optionsSpecSetAddInt(stdrd_relopt_spec_set, "autovacuum_vacuum_cost_limit", + "Vacuum cost amount available before napping, for autovacuum", + ShareUpdateExclusiveLock, + offsetof(StdRdOptions, autovacuum) + + offsetof(AutoVacOpts, vacuum_cost_limit), + NULL, -1, 0, 10000); + + optionsSpecSetAddInt(stdrd_relopt_spec_set, "autovacuum_freeze_min_age", + "Minimum age at which VACUUM should freeze a table row, for autovacuum", + ShareUpdateExclusiveLock, + offsetof(StdRdOptions, autovacuum) + + offsetof(AutoVacOpts, freeze_min_age), + NULL, -1, 0, 1000000000); + + optionsSpecSetAddInt(stdrd_relopt_spec_set, "autovacuum_freeze_max_age", + "Age at which to autovacuum a table to prevent transaction ID wraparound", + ShareUpdateExclusiveLock, + offsetof(StdRdOptions, autovacuum) + + offsetof(AutoVacOpts, freeze_max_age), + NULL, -1, 100000, 2000000000); + + optionsSpecSetAddInt(stdrd_relopt_spec_set, "autovacuum_freeze_table_age", + "Age at which VACUUM should perform a full table sweep to freeze row versions", + ShareUpdateExclusiveLock, + offsetof(StdRdOptions, autovacuum) + + offsetof(AutoVacOpts, freeze_table_age), + NULL, -1, 0, 2000000000); + + optionsSpecSetAddInt(stdrd_relopt_spec_set, "autovacuum_multixact_freeze_min_age", + "Minimum multixact age at which VACUUM should freeze a row multixact's, for autovacuum", + ShareUpdateExclusiveLock, + offsetof(StdRdOptions, autovacuum) + + offsetof(AutoVacOpts, multixact_freeze_min_age), + NULL, -1, 0, 1000000000); + + optionsSpecSetAddInt(stdrd_relopt_spec_set, "autovacuum_multixact_freeze_max_age", + "Multixact age at which to autovacuum a table to prevent multixact wraparound", + ShareUpdateExclusiveLock, + offsetof(StdRdOptions, autovacuum) + + offsetof(AutoVacOpts, multixact_freeze_max_age), + NULL, -1, 10000, 2000000000); + + optionsSpecSetAddInt(stdrd_relopt_spec_set, "autovacuum_multixact_freeze_table_age", + "Age of multixact at which VACUUM should perform a full table sweep to freeze row versions", + ShareUpdateExclusiveLock, + offsetof(StdRdOptions, autovacuum) + + offsetof(AutoVacOpts, multixact_freeze_table_age), + NULL, -1, 0, 2000000000); + + optionsSpecSetAddInt(stdrd_relopt_spec_set, "log_autovacuum_min_duration", + "Sets the minimum execution time above which autovacuum actions will be logged", + ShareUpdateExclusiveLock, + offsetof(StdRdOptions, autovacuum) + + offsetof(AutoVacOpts, log_min_duration), + NULL, -1, -1, INT_MAX); + + optionsSpecSetAddReal(stdrd_relopt_spec_set, "autovacuum_vacuum_cost_delay", + "Vacuum cost delay in milliseconds, for autovacuum", + ShareUpdateExclusiveLock, + offsetof(StdRdOptions, autovacuum) + + offsetof(AutoVacOpts, vacuum_cost_delay), + NULL, -1, 0.0, 100.0); + + optionsSpecSetAddReal(stdrd_relopt_spec_set, "autovacuum_vacuum_scale_factor", + "Number of tuple updates or deletes prior to vacuum as a fraction of reltuples", + ShareUpdateExclusiveLock, + offsetof(StdRdOptions, autovacuum) + + offsetof(AutoVacOpts, vacuum_scale_factor), + NULL, -1, 0.0, 100.0); + + optionsSpecSetAddReal(stdrd_relopt_spec_set, "autovacuum_vacuum_insert_scale_factor", + "Number of tuple inserts prior to vacuum as a fraction of reltuples", + ShareUpdateExclusiveLock, + offsetof(StdRdOptions, autovacuum) + + offsetof(AutoVacOpts, vacuum_ins_scale_factor), + NULL, -1, 0.0, 100.0); + if (is_heap) + { + optionsSpecSetAddReal(stdrd_relopt_spec_set, "autovacuum_analyze_scale_factor", + "Number of tuple inserts, updates or deletes prior to analyze as a fraction of reltuples", + ShareUpdateExclusiveLock, + offsetof(StdRdOptions, autovacuum) + + offsetof(AutoVacOpts, analyze_scale_factor), + NULL, -1, 0.0, 100.0); + + optionsSpecSetAddInt(stdrd_relopt_spec_set, "toast_tuple_target", + "Sets the target tuple length at which external columns will be toasted", + ShareUpdateExclusiveLock, + offsetof(StdRdOptions, toast_tuple_target), + NULL, TOAST_TUPLE_TARGET, 128, + TOAST_TUPLE_TARGET_MAIN); + + optionsSpecSetAddBool(stdrd_relopt_spec_set, "user_catalog_table", + "Declare a table as an additional catalog table, e.g. for the purpose of logical replication", + AccessExclusiveLock, + offsetof(StdRdOptions, user_catalog_table), + NULL, false); + + optionsSpecSetAddInt(stdrd_relopt_spec_set, "parallel_workers", + "Number of parallel processes that can be used per executor node for this relation.", + ShareUpdateExclusiveLock, + offsetof(StdRdOptions, parallel_workers), + NULL, -1, 0, 1024); } - /* Done if no options */ - if (PointerIsValid(DatumGetPointer(options))) - parseRelOptionsInternal(options, validate, reloptions, numoptions); - - *numrelopts = numoptions; - return reloptions; -} - -/* Parse local unregistered options. */ -static relopt_value * -parseLocalRelOptions(local_relopts *relopts, Datum options, bool validate) -{ - int nopts = list_length(relopts->options); - relopt_value *values = palloc(sizeof(*values) * nopts); - ListCell *lc; - int i = 0; + optionsSpecSetAddEnum(stdrd_relopt_spec_set, "vacuum_index_cleanup", + "Controls index vacuuming and index cleanup", + ShareUpdateExclusiveLock, + offsetof(StdRdOptions, vacuum_index_cleanup), + NULL, StdRdOptIndexCleanupValues, + STDRD_OPTION_VACUUM_INDEX_CLEANUP_AUTO, + gettext_noop("Valid values are \"on\", \"off\", and \"auto\".")); - foreach(lc, relopts->options) - { - local_relopt *opt = lfirst(lc); - - values[i].gen = opt->option; - values[i].isset = false; - - i++; - } + optionsSpecSetAddBool(stdrd_relopt_spec_set, "vacuum_truncate", + "Enables vacuum to truncate empty pages at the end of this table", + ShareUpdateExclusiveLock, + offsetof(StdRdOptions, vacuum_truncate), + NULL, true); - if (options != (Datum) 0) - parseRelOptionsInternal(options, validate, values, nopts); + if (is_heap) + optionsSpecSetAddBool(stdrd_relopt_spec_set, "oids", + "Backward compatibility option. Will do nothing when false, will throw error when true", + NoLock, + -1, /* Do not actually store it's value */ + &oid_postvalidate, false); - return values; + return stdrd_relopt_spec_set; } -/* - * Subroutine for parseRelOptions, to parse and validate a single option's - * value - */ -static void -parse_one_reloption(relopt_value *option, char *text_str, int text_len, - bool validate) -{ - char *value; - int value_len; - bool parsed; - bool nofree = false; - if (option->isset && validate) - ereport(ERROR, - (errcode(ERRCODE_INVALID_PARAMETER_VALUE), - errmsg("parameter \"%s\" specified more than once", - option->gen->name))); - - value_len = text_len - option->gen->namelen - 1; - value = (char *) palloc(value_len + 1); - memcpy(value, text_str + option->gen->namelen + 1, value_len); - value[value_len] = '\0'; - - switch (option->gen->type) - { - case RELOPT_TYPE_BOOL: - { - parsed = parse_bool(value, &option->values.bool_val); - if (validate && !parsed) - ereport(ERROR, - (errcode(ERRCODE_INVALID_PARAMETER_VALUE), - errmsg("invalid value for boolean option \"%s\": %s", - option->gen->name, value))); - } - break; - case RELOPT_TYPE_INT: - { - relopt_int *optint = (relopt_int *) option->gen; - - parsed = parse_int(value, &option->values.int_val, 0, NULL); - if (validate && !parsed) - ereport(ERROR, - (errcode(ERRCODE_INVALID_PARAMETER_VALUE), - errmsg("invalid value for integer option \"%s\": %s", - option->gen->name, value))); - if (validate && (option->values.int_val < optint->min || - option->values.int_val > optint->max)) - ereport(ERROR, - (errcode(ERRCODE_INVALID_PARAMETER_VALUE), - errmsg("value %s out of bounds for option \"%s\"", - value, option->gen->name), - errdetail("Valid values are between \"%d\" and \"%d\".", - optint->min, optint->max))); - } - break; - case RELOPT_TYPE_REAL: - { - relopt_real *optreal = (relopt_real *) option->gen; - - parsed = parse_real(value, &option->values.real_val, 0, NULL); - if (validate && !parsed) - ereport(ERROR, - (errcode(ERRCODE_INVALID_PARAMETER_VALUE), - errmsg("invalid value for floating point option \"%s\": %s", - option->gen->name, value))); - if (validate && (option->values.real_val < optreal->min || - option->values.real_val > optreal->max)) - ereport(ERROR, - (errcode(ERRCODE_INVALID_PARAMETER_VALUE), - errmsg("value %s out of bounds for option \"%s\"", - value, option->gen->name), - errdetail("Valid values are between \"%f\" and \"%f\".", - optreal->min, optreal->max))); - } - break; - case RELOPT_TYPE_ENUM: - { - relopt_enum *optenum = (relopt_enum *) option->gen; - relopt_enum_elt_def *elt; - - parsed = false; - for (elt = optenum->members; elt->string_val; elt++) - { - if (pg_strcasecmp(value, elt->string_val) == 0) - { - option->values.enum_val = elt->symbol_val; - parsed = true; - break; - } - } - if (validate && !parsed) - ereport(ERROR, - (errcode(ERRCODE_INVALID_PARAMETER_VALUE), - errmsg("invalid value for enum option \"%s\": %s", - option->gen->name, value), - optenum->detailmsg ? - errdetail_internal("%s", _(optenum->detailmsg)) : 0)); - - /* - * If value is not among the allowed string values, but we are - * not asked to validate, just use the default numeric value. - */ - if (!parsed) - option->values.enum_val = optenum->default_val; - } - break; - case RELOPT_TYPE_STRING: - { - relopt_string *optstring = (relopt_string *) option->gen; - - option->values.string_val = value; - nofree = true; - if (validate && optstring->validate_cb) - (optstring->validate_cb) (value); - parsed = true; - } - break; - default: - elog(ERROR, "unsupported reloption type %d", option->gen->type); - parsed = true; /* quiet compiler */ - break; - } +static options_spec_set *heap_relopt_spec_set = NULL; - if (parsed) - option->isset = true; - if (!nofree) - pfree(value); +options_spec_set * +get_heap_relopt_spec_set(void) +{ + if (heap_relopt_spec_set) + return heap_relopt_spec_set; + heap_relopt_spec_set = get_stdrd_relopt_spec_set(true); + return heap_relopt_spec_set; } /* - * Given the result from parseRelOptions, allocate a struct that's of the - * specified base size plus any extra space that's needed for string variables. - * - * "base" should be sizeof(struct) of the reloptions struct (StdRdOptions or - * equivalent). + * These toast options are can't be set via SQL, but we should set them + * to their defaults in binary representation, to make postgres work properly */ -static void * -allocateReloptStruct(Size base, relopt_value *options, int numoptions) +static void +toast_options_postprocess(void *data, bool validate) { - Size size = base; - int i; - - for (i = 0; i < numoptions; i++) + if (data) { - relopt_value *optval = &options[i]; + StdRdOptions *toast_options = (StdRdOptions *) data; - if (optval->gen->type == RELOPT_TYPE_STRING) - { - relopt_string *optstr = (relopt_string *) optval->gen; - - if (optstr->fill_cb) - { - const char *val = optval->isset ? optval->values.string_val : - optstr->default_isnull ? NULL : optstr->default_val; - - size += optstr->fill_cb(val, NULL); - } - else - size += GET_STRING_RELOPTION_LEN(*optval) + 1; - } + toast_options->fillfactor = 100; + toast_options->autovacuum.analyze_threshold = -1; + toast_options->autovacuum.analyze_scale_factor = -1; } - - return palloc0(size); } -/* - * Given the result of parseRelOptions and a parsing table, fill in the - * struct (previously allocated with allocateReloptStruct) with the parsed - * values. - * - * rdopts is the pointer to the allocated struct to be filled. - * basesize is the sizeof(struct) that was passed to allocateReloptStruct. - * options, of length numoptions, is parseRelOptions' output. - * elems, of length numelems, is the table describing the allowed options. - * When validate is true, it is expected that all options appear in elems. - */ -static void -fillRelOptions(void *rdopts, Size basesize, - relopt_value *options, int numoptions, - bool validate, - const relopt_parse_elt *elems, int numelems) +static options_spec_set *toast_relopt_spec_set = NULL; +options_spec_set * +get_toast_relopt_spec_set(void) { - int i; - int offset = basesize; + if (toast_relopt_spec_set) + return toast_relopt_spec_set; - for (i = 0; i < numoptions; i++) - { - int j; - bool found = false; + toast_relopt_spec_set = get_stdrd_relopt_spec_set(false); + toast_relopt_spec_set->postprocess_fun = toast_options_postprocess; - for (j = 0; j < numelems; j++) - { - if (strcmp(options[i].gen->name, elems[j].optname) == 0) - { - relopt_string *optstring; - char *itempos = ((char *) rdopts) + elems[j].offset; - char *string_val; - - switch (options[i].gen->type) - { - case RELOPT_TYPE_BOOL: - *(bool *) itempos = options[i].isset ? - options[i].values.bool_val : - ((relopt_bool *) options[i].gen)->default_val; - break; - case RELOPT_TYPE_INT: - *(int *) itempos = options[i].isset ? - options[i].values.int_val : - ((relopt_int *) options[i].gen)->default_val; - break; - case RELOPT_TYPE_REAL: - *(double *) itempos = options[i].isset ? - options[i].values.real_val : - ((relopt_real *) options[i].gen)->default_val; - break; - case RELOPT_TYPE_ENUM: - *(int *) itempos = options[i].isset ? - options[i].values.enum_val : - ((relopt_enum *) options[i].gen)->default_val; - break; - case RELOPT_TYPE_STRING: - optstring = (relopt_string *) options[i].gen; - if (options[i].isset) - string_val = options[i].values.string_val; - else if (!optstring->default_isnull) - string_val = optstring->default_val; - else - string_val = NULL; - - if (optstring->fill_cb) - { - Size size = - optstring->fill_cb(string_val, - (char *) rdopts + offset); - - if (size) - { - *(int *) itempos = offset; - offset += size; - } - else - *(int *) itempos = 0; - } - else if (string_val == NULL) - *(int *) itempos = 0; - else - { - strcpy((char *) rdopts + offset, string_val); - *(int *) itempos = offset; - offset += strlen(string_val) + 1; - } - break; - default: - elog(ERROR, "unsupported reloption type %d", - options[i].gen->type); - break; - } - found = true; - break; - } - } - if (validate && !found) - elog(ERROR, "reloption \"%s\" not found in parse table", - options[i].gen->name); - } - SET_VARSIZE(rdopts, offset); + return toast_relopt_spec_set; } /* - * Option parser for anything that uses StdRdOptions. + * Do not allow to set any option on partitioned table */ -bytea * -default_reloptions(Datum reloptions, bool validate, relopt_kind kind) +static void +partitioned_options_postprocess(void *data, bool validate) { - static const relopt_parse_elt tab[] = { - {"fillfactor", RELOPT_TYPE_INT, offsetof(StdRdOptions, fillfactor)}, - {"autovacuum_enabled", RELOPT_TYPE_BOOL, - offsetof(StdRdOptions, autovacuum) + offsetof(AutoVacOpts, enabled)}, - {"autovacuum_vacuum_threshold", RELOPT_TYPE_INT, - offsetof(StdRdOptions, autovacuum) + offsetof(AutoVacOpts, vacuum_threshold)}, - {"autovacuum_vacuum_insert_threshold", RELOPT_TYPE_INT, - offsetof(StdRdOptions, autovacuum) + offsetof(AutoVacOpts, vacuum_ins_threshold)}, - {"autovacuum_analyze_threshold", RELOPT_TYPE_INT, - offsetof(StdRdOptions, autovacuum) + offsetof(AutoVacOpts, analyze_threshold)}, - {"autovacuum_vacuum_cost_limit", RELOPT_TYPE_INT, - offsetof(StdRdOptions, autovacuum) + offsetof(AutoVacOpts, vacuum_cost_limit)}, - {"autovacuum_freeze_min_age", RELOPT_TYPE_INT, - offsetof(StdRdOptions, autovacuum) + offsetof(AutoVacOpts, freeze_min_age)}, - {"autovacuum_freeze_max_age", RELOPT_TYPE_INT, - offsetof(StdRdOptions, autovacuum) + offsetof(AutoVacOpts, freeze_max_age)}, - {"autovacuum_freeze_table_age", RELOPT_TYPE_INT, - offsetof(StdRdOptions, autovacuum) + offsetof(AutoVacOpts, freeze_table_age)}, - {"autovacuum_multixact_freeze_min_age", RELOPT_TYPE_INT, - offsetof(StdRdOptions, autovacuum) + offsetof(AutoVacOpts, multixact_freeze_min_age)}, - {"autovacuum_multixact_freeze_max_age", RELOPT_TYPE_INT, - offsetof(StdRdOptions, autovacuum) + offsetof(AutoVacOpts, multixact_freeze_max_age)}, - {"autovacuum_multixact_freeze_table_age", RELOPT_TYPE_INT, - offsetof(StdRdOptions, autovacuum) + offsetof(AutoVacOpts, multixact_freeze_table_age)}, - {"log_autovacuum_min_duration", RELOPT_TYPE_INT, - offsetof(StdRdOptions, autovacuum) + offsetof(AutoVacOpts, log_min_duration)}, - {"toast_tuple_target", RELOPT_TYPE_INT, - offsetof(StdRdOptions, toast_tuple_target)}, - {"autovacuum_vacuum_cost_delay", RELOPT_TYPE_REAL, - offsetof(StdRdOptions, autovacuum) + offsetof(AutoVacOpts, vacuum_cost_delay)}, - {"autovacuum_vacuum_scale_factor", RELOPT_TYPE_REAL, - offsetof(StdRdOptions, autovacuum) + offsetof(AutoVacOpts, vacuum_scale_factor)}, - {"autovacuum_vacuum_insert_scale_factor", RELOPT_TYPE_REAL, - offsetof(StdRdOptions, autovacuum) + offsetof(AutoVacOpts, vacuum_ins_scale_factor)}, - {"autovacuum_analyze_scale_factor", RELOPT_TYPE_REAL, - offsetof(StdRdOptions, autovacuum) + offsetof(AutoVacOpts, analyze_scale_factor)}, - {"user_catalog_table", RELOPT_TYPE_BOOL, - offsetof(StdRdOptions, user_catalog_table)}, - {"parallel_workers", RELOPT_TYPE_INT, - offsetof(StdRdOptions, parallel_workers)}, - {"vacuum_index_cleanup", RELOPT_TYPE_ENUM, - offsetof(StdRdOptions, vacuum_index_cleanup)}, - {"vacuum_truncate", RELOPT_TYPE_BOOL, - offsetof(StdRdOptions, vacuum_truncate)} - }; - - return (bytea *) build_reloptions(reloptions, validate, kind, - sizeof(StdRdOptions), - tab, lengthof(tab)); + if (data && validate) + ereport(ERROR, + errcode(ERRCODE_WRONG_OBJECT_TYPE), + errmsg("cannot specify storage parameters for a partitioned table"), + errhint("Specify storage parameters for its leaf partitions instead.")); } -/* - * build_reloptions - * - * Parses "reloptions" provided by the caller, returning them in a - * structure containing the parsed options. The parsing is done with - * the help of a parsing table describing the allowed options, defined - * by "relopt_elems" of length "num_relopt_elems". - * - * "validate" must be true if reloptions value is freshly built by - * transformRelOptions(), as opposed to being read from the catalog, in which - * case the values contained in it must already be valid. - * - * NULL is returned if the passed-in options did not match any of the options - * in the parsing table, unless validate is true in which case an error would - * be reported. - */ -void * -build_reloptions(Datum reloptions, bool validate, - relopt_kind kind, - Size relopt_struct_size, - const relopt_parse_elt *relopt_elems, - int num_relopt_elems) -{ - int numoptions; - relopt_value *options; - void *rdopts; - - /* parse options specific to given relation option kind */ - options = parseRelOptions(reloptions, validate, kind, &numoptions); - Assert(numoptions <= num_relopt_elems); - /* if none set, we're done */ - if (numoptions == 0) - { - Assert(options == NULL); - return NULL; - } +static options_spec_set *partitioned_relopt_spec_set = NULL; - /* allocate and fill the structure */ - rdopts = allocateReloptStruct(relopt_struct_size, options, numoptions); - fillRelOptions(rdopts, relopt_struct_size, options, numoptions, - validate, relopt_elems, num_relopt_elems); +options_spec_set * +get_partitioned_relopt_spec_set(void) +{ + if (partitioned_relopt_spec_set) + return partitioned_relopt_spec_set; + partitioned_relopt_spec_set = get_stdrd_relopt_spec_set(true); - pfree(options); + /* No options for now, so Spec Set is empty */ + partitioned_relopt_spec_set->postprocess_fun = + partitioned_options_postprocess; - return rdopts; + return partitioned_relopt_spec_set; } /* @@ -1944,158 +581,165 @@ build_reloptions(Datum reloptions, bool validate, void * build_local_reloptions(local_relopts *relopts, Datum options, bool validate) { - int noptions = list_length(relopts->options); - relopt_parse_elt *elems = palloc(sizeof(*elems) * noptions); - relopt_value *vals; void *opts; - int i = 0; ListCell *lc; + List *values; + + values = optionsTextArrayToRawValues(options); + values = optionsParseRawValues(values, relopts->spec_set, validate); + opts = optionsValuesToBytea(values, relopts->spec_set); + + /* + * Kind of ugly conversion here for backward compatibility. Would be + * removed while moving opclass options to options.c API + */ - foreach(lc, relopts->options) + if (validate && relopts->validators) { - local_relopt *opt = lfirst(lc); + int val_count = list_length(values); + int i; + option_value *val_array; - elems[i].optname = opt->option->name; - elems[i].opttype = opt->option->type; - elems[i].offset = opt->offset; + val_array = palloc(sizeof(option_value) * val_count); - i++; - } + i = 0; + foreach(lc, values) + { + option_value *val = lfirst(lc); - vals = parseLocalRelOptions(relopts, options, validate); - opts = allocateReloptStruct(relopts->relopt_struct_size, vals, noptions); - fillRelOptions(opts, relopts->relopt_struct_size, vals, noptions, validate, - elems, noptions); + memcpy(&(val_array[i]), val, sizeof(option_value)); + i++; + } - if (validate) foreach(lc, relopts->validators) - ((relopts_validator) lfirst(lc)) (opts, vals, noptions); + ((relopts_validator) lfirst(lc)) (opts, val_array, val_count); - if (elems) - pfree(elems); + pfree(val_array); + } return opts; } /* - * Option parser for partitioned tables - */ -bytea * -partitioned_table_reloptions(Datum reloptions, bool validate) -{ - if (validate && reloptions) - ereport(ERROR, - errcode(ERRCODE_WRONG_OBJECT_TYPE), - errmsg("cannot specify storage parameters for a partitioned table"), - errhint("Specify storage parameters for its leaf partitions instead.")); - return NULL; -} - -/* - * Option parser for views + * get_view_relopt_spec_set + * Returns an options catalog for view relation. */ -bytea * -view_reloptions(Datum reloptions, bool validate) -{ - static const relopt_parse_elt tab[] = { - {"security_barrier", RELOPT_TYPE_BOOL, - offsetof(ViewOptions, security_barrier)}, - {"security_invoker", RELOPT_TYPE_BOOL, - offsetof(ViewOptions, security_invoker)}, - {"check_option", RELOPT_TYPE_ENUM, - offsetof(ViewOptions, check_option)} - }; - - return (bytea *) build_reloptions(reloptions, validate, - RELOPT_KIND_VIEW, - sizeof(ViewOptions), - tab, lengthof(tab)); -} +static options_spec_set *view_relopt_spec_set = NULL; -/* - * Parse options for heaps, views and toast tables. - */ -bytea * -heap_reloptions(char relkind, Datum reloptions, bool validate) +options_spec_set * +get_view_relopt_spec_set(void) { - StdRdOptions *rdopts; + if (view_relopt_spec_set) + return view_relopt_spec_set; - switch (relkind) - { - case RELKIND_TOASTVALUE: - rdopts = (StdRdOptions *) - default_reloptions(reloptions, validate, RELOPT_KIND_TOAST); - if (rdopts != NULL) - { - /* adjust default-only parameters for TOAST relations */ - rdopts->fillfactor = 100; - rdopts->autovacuum.analyze_threshold = -1; - rdopts->autovacuum.analyze_scale_factor = -1; - } - return (bytea *) rdopts; - case RELKIND_RELATION: - case RELKIND_MATVIEW: - return default_reloptions(reloptions, validate, RELOPT_KIND_HEAP); - default: - /* other relkinds are not supported */ - return NULL; - } -} + view_relopt_spec_set = allocateOptionsSpecSet(NULL, + sizeof(ViewOptions), false, 3); + optionsSpecSetAddBool(view_relopt_spec_set, "security_barrier", + "View acts as a row security barrier", + AccessExclusiveLock, + offsetof(ViewOptions, security_barrier), NULL, false); -/* - * Parse options for indexes. - * - * amoptions index AM's option parser function - * reloptions options as text[] datum - * validate error flag - */ -bytea * -index_reloptions(amoptions_function amoptions, Datum reloptions, bool validate) -{ - Assert(amoptions != NULL); + optionsSpecSetAddBool(view_relopt_spec_set, "security_invoker", + "Privileges on underlying relations are checked as the invoking user, not the view owner", + AccessExclusiveLock, + offsetof(ViewOptions, security_invoker), NULL, false); - /* Assume function is strict */ - if (!PointerIsValid(DatumGetPointer(reloptions))) - return NULL; + optionsSpecSetAddEnum(view_relopt_spec_set, "check_option", + "View has WITH CHECK OPTION defined (local or cascaded)", + AccessExclusiveLock, + offsetof(ViewOptions, check_option), NULL, + viewCheckOptValues, + VIEW_OPTION_CHECK_OPTION_NOT_SET, + gettext_noop("Valid values are \"local\" and \"cascaded\".")); - return amoptions(reloptions, validate); + return view_relopt_spec_set; } /* - * Option parser for attribute reloptions + * get_attribute_options_spec_set + * Returns an Options Spec Set for heap attributes */ -bytea * -attribute_reloptions(Datum reloptions, bool validate) +static options_spec_set *attribute_options_spec_set = NULL; + +options_spec_set * +get_attribute_options_spec_set(void) { - static const relopt_parse_elt tab[] = { - {"n_distinct", RELOPT_TYPE_REAL, offsetof(AttributeOpts, n_distinct)}, - {"n_distinct_inherited", RELOPT_TYPE_REAL, offsetof(AttributeOpts, n_distinct_inherited)} - }; - - return (bytea *) build_reloptions(reloptions, validate, - RELOPT_KIND_ATTRIBUTE, - sizeof(AttributeOpts), - tab, lengthof(tab)); + if (attribute_options_spec_set) + return attribute_options_spec_set; + + attribute_options_spec_set = allocateOptionsSpecSet(NULL, + sizeof(AttributeOpts), false, 2); + + optionsSpecSetAddReal(attribute_options_spec_set, "n_distinct", + "Sets the planner's estimate of the number of distinct values appearing in a column (excluding child relations).", + ShareUpdateExclusiveLock, + offsetof(AttributeOpts, n_distinct), NULL, + 0, -1.0, DBL_MAX); + + optionsSpecSetAddReal(attribute_options_spec_set, + "n_distinct_inherited", + "Sets the planner's estimate of the number of distinct values appearing in a column (including child relations).", + ShareUpdateExclusiveLock, + offsetof(AttributeOpts, n_distinct_inherited), NULL, + 0, -1.0, DBL_MAX); + + return attribute_options_spec_set; } + /* - * Option parser for tablespace reloptions - */ -bytea * -tablespace_reloptions(Datum reloptions, bool validate) + * get_tablespace_options_spec_set + * Returns an Options Spec Set for tablespaces +*/ +static options_spec_set *tablespace_options_spec_set = NULL; + +options_spec_set * +get_tablespace_options_spec_set(void) { - static const relopt_parse_elt tab[] = { - {"random_page_cost", RELOPT_TYPE_REAL, offsetof(TableSpaceOpts, random_page_cost)}, - {"seq_page_cost", RELOPT_TYPE_REAL, offsetof(TableSpaceOpts, seq_page_cost)}, - {"effective_io_concurrency", RELOPT_TYPE_INT, offsetof(TableSpaceOpts, effective_io_concurrency)}, - {"maintenance_io_concurrency", RELOPT_TYPE_INT, offsetof(TableSpaceOpts, maintenance_io_concurrency)} - }; - - return (bytea *) build_reloptions(reloptions, validate, - RELOPT_KIND_TABLESPACE, - sizeof(TableSpaceOpts), - tab, lengthof(tab)); + if (tablespace_options_spec_set) + return tablespace_options_spec_set; + + tablespace_options_spec_set = allocateOptionsSpecSet(NULL, + sizeof(TableSpaceOpts), false, 4); + optionsSpecSetAddReal(tablespace_options_spec_set, + "random_page_cost", + "Sets the planner's estimate of the cost of a nonsequentially fetched disk page", + ShareUpdateExclusiveLock, + offsetof(TableSpaceOpts, random_page_cost), + NULL, -1, 0.0, DBL_MAX); + + optionsSpecSetAddReal(tablespace_options_spec_set, "seq_page_cost", + "Sets the planner's estimate of the cost of a sequentially fetched disk page", + ShareUpdateExclusiveLock, + offsetof(TableSpaceOpts, seq_page_cost), + NULL, -1, 0.0, DBL_MAX); + + optionsSpecSetAddInt(tablespace_options_spec_set, "effective_io_concurrency", + "Number of simultaneous requests that can be handled efficiently by the disk subsystem", + ShareUpdateExclusiveLock, + offsetof(TableSpaceOpts, effective_io_concurrency), + NULL, +#ifdef USE_PREFETCH + -1, 0, MAX_IO_CONCURRENCY +#else + 0, 0, 0 +#endif + ); + + optionsSpecSetAddInt(tablespace_options_spec_set, + "maintenance_io_concurrency", + "Number of simultaneous requests that can be handled efficiently by the disk subsystem for maintenance work.", + ShareUpdateExclusiveLock, + offsetof(TableSpaceOpts, maintenance_io_concurrency), + NULL, +#ifdef USE_PREFETCH + -1, 0, MAX_IO_CONCURRENCY +#else + 0, 0, 0 +#endif + ); + return tablespace_options_spec_set; } /* @@ -2105,33 +749,55 @@ tablespace_reloptions(Datum reloptions, bool validate) * for a longer explanation of how this works. */ LOCKMODE -AlterTableGetRelOptionsLockLevel(List *defList) +AlterTableGetRelOptionsLockLevel(Relation rel, List *defList) { LOCKMODE lockmode = NoLock; ListCell *cell; + options_spec_set *spec_set = NULL; if (defList == NIL) return AccessExclusiveLock; - if (need_initialization) - initialize_reloptions(); + switch (rel->rd_rel->relkind) + { + case RELKIND_TOASTVALUE: + spec_set = get_toast_relopt_spec_set(); + break; + case RELKIND_RELATION: + case RELKIND_MATVIEW: + spec_set = get_heap_relopt_spec_set(); + break; + case RELKIND_INDEX: + spec_set = rel->rd_indam->amreloptspecset(); + break; + case RELKIND_VIEW: + spec_set = get_view_relopt_spec_set(); + break; + case RELKIND_PARTITIONED_TABLE: + spec_set = get_partitioned_relopt_spec_set(); + break; + default: + Assert(false); /* can't get here */ + break; + } + Assert(spec_set); /* No spec set - no reloption change. Should + * never get here */ foreach(cell, defList) { DefElem *def = (DefElem *) lfirst(cell); + int i; - for (i = 0; relOpts[i]; i++) + for (i = 0; i < spec_set->num; i++) { - if (strncmp(relOpts[i]->name, - def->defname, - relOpts[i]->namelen + 1) == 0) - { - if (lockmode < relOpts[i]->lockmode) - lockmode = relOpts[i]->lockmode; - } + option_spec_basic *gen = spec_set->definitions[i]; + + if (pg_strcasecmp(gen->name, + def->defname) == 0) + if (lockmode < gen->lockmode) + lockmode = gen->lockmode; } } - return lockmode; } diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c index 5747ae6a4c..3a67719ab9 100644 --- a/src/backend/access/gin/ginutil.c +++ b/src/backend/access/gin/ginutil.c @@ -16,7 +16,7 @@ #include "access/gin_private.h" #include "access/ginxlog.h" -#include "access/reloptions.h" +#include "access/options.h" #include "access/xloginsert.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -27,6 +27,7 @@ #include "utils/index_selfuncs.h" #include "utils/rel.h" #include "utils/typcache.h" +#include "utils/guc.h" /* @@ -69,7 +70,6 @@ ginhandler(PG_FUNCTION_ARGS) amroutine->amvacuumcleanup = ginvacuumcleanup; amroutine->amcanreturn = NULL; amroutine->amcostestimate = gincostestimate; - amroutine->amoptions = ginoptions; amroutine->amproperty = NULL; amroutine->ambuildphasename = NULL; amroutine->amvalidate = ginvalidate; @@ -84,6 +84,7 @@ ginhandler(PG_FUNCTION_ARGS) amroutine->amestimateparallelscan = NULL; amroutine->aminitparallelscan = NULL; amroutine->amparallelrescan = NULL; + amroutine->amreloptspecset = gingetreloptspecset; PG_RETURN_POINTER(amroutine); } @@ -598,21 +599,6 @@ ginExtractEntries(GinState *ginstate, OffsetNumber attnum, return entries; } -bytea * -ginoptions(Datum reloptions, bool validate) -{ - static const relopt_parse_elt tab[] = { - {"fastupdate", RELOPT_TYPE_BOOL, offsetof(GinOptions, useFastUpdate)}, - {"gin_pending_list_limit", RELOPT_TYPE_INT, offsetof(GinOptions, - pendingListCleanupSize)} - }; - - return (bytea *) build_reloptions(reloptions, validate, - RELOPT_KIND_GIN, - sizeof(GinOptions), - tab, lengthof(tab)); -} - /* * Fetch index's statistical data into *stats * @@ -699,3 +685,29 @@ ginUpdateStats(Relation index, const GinStatsData *stats, bool is_build) END_CRIT_SECTION(); } + +static options_spec_set *gin_relopt_specset = NULL; + +void * +gingetreloptspecset(void) +{ + if (gin_relopt_specset) + return gin_relopt_specset; + + gin_relopt_specset = allocateOptionsSpecSet(NULL, + sizeof(GinOptions), false, 2); + + optionsSpecSetAddBool(gin_relopt_specset, "fastupdate", + "Enables \"fast update\" feature for this GIN index", + AccessExclusiveLock, + offsetof(GinOptions, useFastUpdate), NULL, + GIN_DEFAULT_USE_FASTUPDATE); + + optionsSpecSetAddInt(gin_relopt_specset, "gin_pending_list_limit", + "Maximum size of the pending list for this GIN index, in kilobytes", + AccessExclusiveLock, + offsetof(GinOptions, pendingListCleanupSize), NULL, + -1, 64, MAX_KILOBYTES); + + return gin_relopt_specset; +} diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c index ed4ffa63a7..b5a1521a83 100644 --- a/src/backend/access/gist/gist.c +++ b/src/backend/access/gist/gist.c @@ -91,7 +91,6 @@ gisthandler(PG_FUNCTION_ARGS) amroutine->amvacuumcleanup = gistvacuumcleanup; amroutine->amcanreturn = gistcanreturn; amroutine->amcostestimate = gistcostestimate; - amroutine->amoptions = gistoptions; amroutine->amproperty = gistproperty; amroutine->ambuildphasename = NULL; amroutine->amvalidate = gistvalidate; @@ -106,6 +105,7 @@ gisthandler(PG_FUNCTION_ARGS) amroutine->amestimateparallelscan = NULL; amroutine->aminitparallelscan = NULL; amroutine->amparallelrescan = NULL; + amroutine->amreloptspecset = gistgetreloptspecset; PG_RETURN_POINTER(amroutine); } diff --git a/src/backend/access/gist/gistutil.c b/src/backend/access/gist/gistutil.c index 78e98d68b1..865da66455 100644 --- a/src/backend/access/gist/gistutil.c +++ b/src/backend/access/gist/gistutil.c @@ -17,7 +17,7 @@ #include "access/gist_private.h" #include "access/htup_details.h" -#include "access/reloptions.h" +#include "access/options.h" #include "common/pg_prng.h" #include "storage/indexfsm.h" #include "utils/float.h" @@ -907,20 +907,6 @@ gistPageRecyclable(Page page) return false; } -bytea * -gistoptions(Datum reloptions, bool validate) -{ - static const relopt_parse_elt tab[] = { - {"fillfactor", RELOPT_TYPE_INT, offsetof(GiSTOptions, fillfactor)}, - {"buffering", RELOPT_TYPE_ENUM, offsetof(GiSTOptions, buffering_mode)} - }; - - return (bytea *) build_reloptions(reloptions, validate, - RELOPT_KIND_GIST, - sizeof(GiSTOptions), - tab, lengthof(tab)); -} - /* * gistproperty() -- Check boolean properties of indexes. * @@ -1055,3 +1041,39 @@ gistGetFakeLSN(Relation rel) return GetFakeLSNForUnloggedRel(); } } + +/* values from GistOptBufferingMode */ +static opt_enum_elt_def gistBufferingOptValues[] = +{ + {"auto", GIST_OPTION_BUFFERING_AUTO}, + {"on", GIST_OPTION_BUFFERING_ON}, + {"off", GIST_OPTION_BUFFERING_OFF}, + {(const char *) NULL} /* list terminator */ +}; + +static options_spec_set *gist_relopt_specset = NULL; + +void * +gistgetreloptspecset(void) +{ + if (gist_relopt_specset) + return gist_relopt_specset; + + gist_relopt_specset = allocateOptionsSpecSet(NULL, + sizeof(GiSTOptions), false, 2); + + optionsSpecSetAddInt(gist_relopt_specset, "fillfactor", + "Packs gist index pages only to this percentage", + NoLock, /* No ALTER, no lock */ + offsetof(GiSTOptions, fillfactor), NULL, + GIST_DEFAULT_FILLFACTOR, GIST_MIN_FILLFACTOR, 100); + + optionsSpecSetAddEnum(gist_relopt_specset, "buffering", + "Enables buffering build for this GiST index", + NoLock, /* No ALTER, no lock */ + offsetof(GiSTOptions, buffering_mode), NULL, + gistBufferingOptValues, + GIST_OPTION_BUFFERING_AUTO, + gettext_noop("Valid values are \"on\", \"off\", and \"auto\".")); + return gist_relopt_specset; +} diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c index 01d06b7c32..aae9541762 100644 --- a/src/backend/access/hash/hash.c +++ b/src/backend/access/hash/hash.c @@ -89,7 +89,6 @@ hashhandler(PG_FUNCTION_ARGS) amroutine->amvacuumcleanup = hashvacuumcleanup; amroutine->amcanreturn = NULL; amroutine->amcostestimate = hashcostestimate; - amroutine->amoptions = hashoptions; amroutine->amproperty = NULL; amroutine->ambuildphasename = NULL; amroutine->amvalidate = hashvalidate; @@ -104,6 +103,7 @@ hashhandler(PG_FUNCTION_ARGS) amroutine->amestimateparallelscan = NULL; amroutine->aminitparallelscan = NULL; amroutine->amparallelrescan = NULL; + amroutine->amreloptspecset = hashgetreloptspecset; PG_RETURN_POINTER(amroutine); } diff --git a/src/backend/access/hash/hashutil.c b/src/backend/access/hash/hashutil.c index 20028f5cd1..bb02c08a3d 100644 --- a/src/backend/access/hash/hashutil.c +++ b/src/backend/access/hash/hashutil.c @@ -15,7 +15,7 @@ #include "postgres.h" #include "access/hash.h" -#include "access/reloptions.h" +#include "access/options.h" #include "access/relscan.h" #include "port/pg_bitutils.h" #include "utils/lsyscache.h" @@ -271,19 +271,6 @@ _hash_checkpage(Relation rel, Buffer buf, int flags) } } -bytea * -hashoptions(Datum reloptions, bool validate) -{ - static const relopt_parse_elt tab[] = { - {"fillfactor", RELOPT_TYPE_INT, offsetof(HashOptions, fillfactor)}, - }; - - return (bytea *) build_reloptions(reloptions, validate, - RELOPT_KIND_HASH, - sizeof(HashOptions), - tab, lengthof(tab)); -} - /* * _hash_get_indextuple_hashkey - get the hash index tuple's hash key value */ @@ -619,3 +606,22 @@ _hash_kill_items(IndexScanDesc scan) else _hash_relbuf(rel, buf); } + +static options_spec_set *hash_relopt_specset = NULL; + +void * +hashgetreloptspecset(void) +{ + if (hash_relopt_specset) + return hash_relopt_specset; + + hash_relopt_specset = allocateOptionsSpecSet(NULL, + sizeof(HashOptions), false, 1); + optionsSpecSetAddInt(hash_relopt_specset, "fillfactor", + "Packs hash index pages only to this percentage", + NoLock, /* No ALTER -- no lock */ + offsetof(HashOptions, fillfactor), NULL, + HASH_DEFAULT_FILLFACTOR, HASH_MIN_FILLFACTOR, 100); + + return hash_relopt_specset; +} diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c index 686a3206f7..200189a4d0 100644 --- a/src/backend/access/nbtree/nbtree.c +++ b/src/backend/access/nbtree/nbtree.c @@ -21,6 +21,7 @@ #include "access/nbtree.h" #include "access/relscan.h" #include "access/xloginsert.h" +#include "access/options.h" #include "commands/progress.h" #include "commands/vacuum.h" #include "miscadmin.h" @@ -133,7 +134,6 @@ bthandler(PG_FUNCTION_ARGS) amroutine->amvacuumcleanup = btvacuumcleanup; amroutine->amcanreturn = btcanreturn; amroutine->amcostestimate = btcostestimate; - amroutine->amoptions = btoptions; amroutine->amproperty = btproperty; amroutine->ambuildphasename = btbuildphasename; amroutine->amvalidate = btvalidate; @@ -148,6 +148,7 @@ bthandler(PG_FUNCTION_ARGS) amroutine->amestimateparallelscan = btestimateparallelscan; amroutine->aminitparallelscan = btinitparallelscan; amroutine->amparallelrescan = btparallelrescan; + amroutine->amreloptspecset = btgetreloptspecset; PG_RETURN_POINTER(amroutine); } @@ -1445,3 +1446,35 @@ btcanreturn(Relation index, int attno) { return true; } + +static options_spec_set *bt_relopt_specset = NULL; + +void * +btgetreloptspecset(void) +{ + if (bt_relopt_specset) + return bt_relopt_specset; + + bt_relopt_specset = allocateOptionsSpecSet(NULL, + sizeof(BTOptions), false, 3); + + optionsSpecSetAddInt(bt_relopt_specset, "fillfactor", + "Packs btree index pages only to this percentage", + ShareUpdateExclusiveLock, /* affects inserts only */ + offsetof(BTOptions, fillfactor), NULL, + BTREE_DEFAULT_FILLFACTOR, BTREE_MIN_FILLFACTOR, 100); + + optionsSpecSetAddReal(bt_relopt_specset, "vacuum_cleanup_index_scale_factor", + "Number of tuple inserts prior to index cleanup as a fraction of reltuples", + ShareUpdateExclusiveLock, + offsetof(BTOptions, vacuum_cleanup_index_scale_factor), + NULL, -1, 0.0, 1e10); + + optionsSpecSetAddBool(bt_relopt_specset, "deduplicate_items", + "Enables \"deduplicate items\" feature for this btree index", + ShareUpdateExclusiveLock, /* affects inserts only */ + offsetof(BTOptions, deduplicate_items), NULL, + true); + + return bt_relopt_specset; +} diff --git a/src/backend/access/nbtree/nbtutils.c b/src/backend/access/nbtree/nbtutils.c index 05f0d23dc0..65658a5023 100644 --- a/src/backend/access/nbtree/nbtutils.c +++ b/src/backend/access/nbtree/nbtutils.c @@ -18,7 +18,7 @@ #include <time.h> #include "access/nbtree.h" -#include "access/reloptions.h" +#include "storage/lock.h" #include "access/relscan.h" #include "commands/progress.h" #include "lib/qunique.h" @@ -4557,23 +4557,6 @@ BTreeShmemInit(void) Assert(found); } -bytea * -btoptions(Datum reloptions, bool validate) -{ - static const relopt_parse_elt tab[] = { - {"fillfactor", RELOPT_TYPE_INT, offsetof(BTOptions, fillfactor)}, - {"vacuum_cleanup_index_scale_factor", RELOPT_TYPE_REAL, - offsetof(BTOptions, vacuum_cleanup_index_scale_factor)}, - {"deduplicate_items", RELOPT_TYPE_BOOL, - offsetof(BTOptions, deduplicate_items)} - }; - - return (bytea *) build_reloptions(reloptions, validate, - RELOPT_KIND_BTREE, - sizeof(BTOptions), - tab, lengthof(tab)); -} - /* * btproperty() -- Check boolean properties of indexes. * diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c index 3f793125f7..86a48d2eff 100644 --- a/src/backend/access/spgist/spgutils.c +++ b/src/backend/access/spgist/spgutils.c @@ -17,7 +17,7 @@ #include "access/amvalidate.h" #include "access/htup_details.h" -#include "access/reloptions.h" +#include "access/options.h" #include "access/spgist_private.h" #include "access/toast_compression.h" #include "access/transam.h" @@ -76,7 +76,6 @@ spghandler(PG_FUNCTION_ARGS) amroutine->amvacuumcleanup = spgvacuumcleanup; amroutine->amcanreturn = spgcanreturn; amroutine->amcostestimate = spgcostestimate; - amroutine->amoptions = spgoptions; amroutine->amproperty = spgproperty; amroutine->ambuildphasename = NULL; amroutine->amvalidate = spgvalidate; @@ -91,6 +90,7 @@ spghandler(PG_FUNCTION_ARGS) amroutine->amestimateparallelscan = NULL; amroutine->aminitparallelscan = NULL; amroutine->amparallelrescan = NULL; + amroutine->amreloptspecset = spggetreloptspecset; PG_RETURN_POINTER(amroutine); } @@ -733,22 +733,6 @@ SpGistInitMetapage(Page page) ((char *) metadata + sizeof(SpGistMetaPageData)) - (char *) page; } -/* - * reloptions processing for SPGiST - */ -bytea * -spgoptions(Datum reloptions, bool validate) -{ - static const relopt_parse_elt tab[] = { - {"fillfactor", RELOPT_TYPE_INT, offsetof(SpGistOptions, fillfactor)}, - }; - - return (bytea *) build_reloptions(reloptions, validate, - RELOPT_KIND_SPGIST, - sizeof(SpGistOptions), - tab, lengthof(tab)); -} - /* * Get the space needed to store a non-null datum of the indicated type * in an inner tuple (that is, as a prefix or node label). @@ -1348,3 +1332,24 @@ spgproperty(Oid index_oid, int attno, return true; } + +static options_spec_set *spgist_relopt_specset = NULL; + +void * +spggetreloptspecset(void) +{ + + if (spgist_relopt_specset) + return spgist_relopt_specset; + + spgist_relopt_specset = allocateOptionsSpecSet(NULL, + sizeof(SpGistOptions), false, 1); + + optionsSpecSetAddInt(spgist_relopt_specset, "fillfactor", + "Packs spgist index pages only to this percentage", + ShareUpdateExclusiveLock, /* affects only inserts */ + offsetof(SpGistOptions, fillfactor), NULL, + SPGIST_DEFAULT_FILLFACTOR, SPGIST_MIN_FILLFACTOR, 100); + + return spgist_relopt_specset; +} diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c index 62050f4dc5..936f640fdf 100644 --- a/src/backend/commands/createas.c +++ b/src/backend/commands/createas.c @@ -85,6 +85,7 @@ create_ctas_internal(List *attrList, IntoClause *into) Datum toast_options; static char *validnsps[] = HEAP_RELOPT_NAMESPACES; ObjectAddress intoRelationAddr; + List *toastDefList; /* This code supports both CREATE TABLE AS and CREATE MATERIALIZED VIEW */ is_matview = (into->viewQuery != NULL); @@ -119,14 +120,12 @@ create_ctas_internal(List *attrList, IntoClause *into) CommandCounterIncrement(); /* parse and validate reloptions for the toast table */ - toast_options = transformRelOptions((Datum) 0, - create->options, - "toast", - validnsps, - true, false); - (void) heap_reloptions(RELKIND_TOASTVALUE, toast_options, true); + optionsDefListValdateNamespaces(create->options, validnsps); + toastDefList = optionsDefListFilterNamespaces(create->options, "toast"); + toast_options = optionDefListToTextArray(get_toast_relopt_spec_set(), + toastDefList); NewRelationCreateToastTable(intoRelationAddr.objectId, toast_options); /* Create the "view" part of a materialized view. */ diff --git a/src/backend/commands/foreigncmds.c b/src/backend/commands/foreigncmds.c index cf61bbac1f..556cad13ed 100644 --- a/src/backend/commands/foreigncmds.c +++ b/src/backend/commands/foreigncmds.c @@ -112,7 +112,7 @@ transformGenericOptions(Oid catalogId, List *options, Oid fdwvalidator) { - List *resultOptions = untransformRelOptions(oldOptions); + List *resultOptions = optionsTextArrayToDefList(oldOptions); ListCell *optcell; Datum result; diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 309389e20d..649aa915b6 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -19,6 +19,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/reloptions.h" +#include "access/options.h" #include "access/sysattr.h" #include "access/tableam.h" #include "access/xact.h" @@ -559,7 +560,7 @@ DefineIndex(Oid tableId, IndexAmRoutine *amRoutine; bool amcanorder; bool amissummarizing; - amoptions_function amoptions; + amreloptspecset_function amreloptspecsetfn; bool partitioned; bool safe_index; Datum reloptions; @@ -870,7 +871,7 @@ DefineIndex(Oid tableId, accessMethodName))); amcanorder = amRoutine->amcanorder; - amoptions = amRoutine->amoptions; + amreloptspecsetfn = amRoutine->amreloptspecset; amissummarizing = amRoutine->amsummarizing; pfree(amRoutine); @@ -885,10 +886,18 @@ DefineIndex(Oid tableId, /* * Parse AM-specific options, convert to text array form, validate. */ - reloptions = transformRelOptions((Datum) 0, stmt->options, - NULL, NULL, false, false); - (void) index_reloptions(amoptions, reloptions, true); + if (stmt->options && !amreloptspecsetfn) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("access method %s does not support options", + accessMethodName))); + + if (amreloptspecsetfn) + reloptions = optionDefListToTextArray(amreloptspecsetfn(), + stmt->options); + else + reloptions = (Datum) 0; /* * Prepare arguments for index_create, primarily an IndexInfo structure. @@ -2195,8 +2204,7 @@ ComputeIndexAttrs(IndexInfo *indexInfo, Assert(attn < nkeycols); opclassOptions[attn] = - transformRelOptions((Datum) 0, attribute->opclassopts, - NULL, NULL, false, false); + optionsDefListToTextArray(attribute->opclassopts); } else opclassOptions[attn] = (Datum) 0; diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 7b6c69b7a5..e145bf7957 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -20,6 +20,7 @@ #include "access/heapam_xlog.h" #include "access/multixact.h" #include "access/reloptions.h" +#include "access/options.h" #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" @@ -699,7 +700,6 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, ListCell *listptr; AttrNumber attnum; bool partitioned; - static char *validnsps[] = HEAP_RELOPT_NAMESPACES; Oid ofTypeId; ObjectAddress address; LOCKMODE parentLockmode; @@ -842,19 +842,36 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, /* * Parse and validate reloptions, if any. */ - reloptions = transformRelOptions((Datum) 0, stmt->options, NULL, validnsps, - true, false); switch (relkind) { case RELKIND_VIEW: - (void) view_reloptions(reloptions, true); + reloptions = optionDefListToTextArray(get_view_relopt_spec_set(), + stmt->options); break; case RELKIND_PARTITIONED_TABLE: - (void) partitioned_table_reloptions(reloptions, true); - break; + { + /* If it is not all listed above, then it if heap */ + char *namespaces[] = HEAP_RELOPT_NAMESPACES; + List *heapDefList; + + optionsDefListValdateNamespaces(stmt->options, namespaces); + heapDefList = optionsDefListFilterNamespaces(stmt->options, NULL); + reloptions = optionDefListToTextArray( + get_partitioned_relopt_spec_set(), heapDefList); + break; + } default: - (void) heap_reloptions(relkind, reloptions, true); + { + /* If it is not all listed above, is should be heap */ + char *namespaces[] = HEAP_RELOPT_NAMESPACES; + List *heapDefList; + + optionsDefListValdateNamespaces(stmt->options, namespaces); + heapDefList = optionsDefListFilterNamespaces(stmt->options, NULL); + reloptions = optionDefListToTextArray(get_heap_relopt_spec_set(), + heapDefList); + } } if (stmt->ofTypename) @@ -4366,7 +4383,7 @@ void AlterTableInternal(Oid relid, List *cmds, bool recurse) { Relation rel; - LOCKMODE lockmode = AlterTableGetLockLevel(cmds); + LOCKMODE lockmode = AlterTableGetLockLevel(relid, cmds); rel = relation_open(relid, lockmode); @@ -4408,7 +4425,7 @@ AlterTableInternal(Oid relid, List *cmds, bool recurse) * otherwise we might end up with an inconsistent dump that can't restore. */ LOCKMODE -AlterTableGetLockLevel(List *cmds) +AlterTableGetLockLevel(Oid relid, List *cmds) { /* * This only works if we read catalog tables using MVCC snapshots. @@ -4629,9 +4646,14 @@ AlterTableGetLockLevel(List *cmds) * getTables() */ case AT_ResetRelOptions: /* Uses MVCC in getIndexes() and * getTables() */ - cmd_lockmode = AlterTableGetRelOptionsLockLevel((List *) cmd->def); - break; + { + Relation rel = relation_open(relid, AccessShareLock); + cmd_lockmode = AlterTableGetRelOptionsLockLevel(rel, + castNode(List, cmd->def)); + relation_close(rel, AccessShareLock); + break; + } case AT_AttachPartition: cmd_lockmode = ShareUpdateExclusiveLock; break; @@ -8720,12 +8742,13 @@ ATExecSetOptions(Relation rel, const char *colName, Node *options, /* Generate new proposed attoptions (text array) */ datum = SysCacheGetAttr(ATTNAME, tuple, Anum_pg_attribute_attoptions, &isnull); - newOptions = transformRelOptions(isnull ? (Datum) 0 : datum, - castNode(List, options), NULL, NULL, - false, isReset); - /* Validate new options */ - (void) attribute_reloptions(newOptions, true); + if (isnull) + datum = (Datum) 0; + newOptions = optionsUpdateTexArrayWithDefList( + get_attribute_options_spec_set(), + datum, castNode(List, options), + isReset); /* Build new tuple. */ memset(repl_null, false, sizeof(repl_null)); memset(repl_repl, false, sizeof(repl_repl)); @@ -14857,12 +14880,13 @@ ATExecSetRelOptions(Relation rel, List *defList, AlterTableType operation, HeapTuple tuple; HeapTuple newtuple; Datum datum; - bool isnull; Datum newOptions; Datum repl_val[Natts_pg_class]; bool repl_null[Natts_pg_class]; bool repl_repl[Natts_pg_class]; - static char *validnsps[] = HEAP_RELOPT_NAMESPACES; + List *optionsDefList; + options_spec_set *optionsSpecSet; + char *heap_namespaces[] = HEAP_RELOPT_NAMESPACES; if (defList == NIL && operation != AT_ReplaceRelOptions) return; /* nothing to do */ @@ -14882,38 +14906,56 @@ ATExecSetRelOptions(Relation rel, List *defList, AlterTableType operation, * there were none before. */ datum = (Datum) 0; - isnull = true; } else { + bool isnull; + /* Get the old reloptions */ datum = SysCacheGetAttr(RELOID, tuple, Anum_pg_class_reloptions, &isnull); + if (isnull) + datum = (Datum) 0; } /* Generate new proposed reloptions (text array) */ - newOptions = transformRelOptions(isnull ? (Datum) 0 : datum, - defList, NULL, validnsps, false, - operation == AT_ResetRelOptions); /* Validate */ + + optionsSpecSet = NULL; switch (rel->rd_rel->relkind) { case RELKIND_RELATION: - case RELKIND_TOASTVALUE: case RELKIND_MATVIEW: - (void) heap_reloptions(rel->rd_rel->relkind, newOptions, true); + optionsDefListValdateNamespaces(defList, heap_namespaces); + optionsDefList = optionsDefListFilterNamespaces(defList, NULL); + optionsSpecSet = get_heap_relopt_spec_set(); break; case RELKIND_PARTITIONED_TABLE: - (void) partitioned_table_reloptions(newOptions, true); + optionsDefListValdateNamespaces(defList, heap_namespaces); + optionsDefList = optionsDefListFilterNamespaces(defList, NULL); + optionsSpecSet = get_partitioned_relopt_spec_set(); break; case RELKIND_VIEW: - (void) view_reloptions(newOptions, true); + optionsDefList = defList; + optionsSpecSet = get_view_relopt_spec_set(); break; case RELKIND_INDEX: case RELKIND_PARTITIONED_INDEX: - (void) index_reloptions(rel->rd_indam->amoptions, newOptions, true); + if (!rel->rd_indam->amreloptspecset) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("index %s does not support options", + RelationGetRelationName(rel)))); + optionsDefList = defList; + optionsSpecSet = rel->rd_indam->amreloptspecset(); break; + case RELKIND_TOASTVALUE: + /* Should never get here */ + /* TOAST options are never altered directly */ + Assert(0); + /* FALLTHRU */ + /* If we get here in prod. error is the best option */ default: ereport(ERROR, (errcode(ERRCODE_WRONG_OBJECT_TYPE), @@ -14923,11 +14965,15 @@ ATExecSetRelOptions(Relation rel, List *defList, AlterTableType operation, break; } + newOptions = optionsUpdateTexArrayWithDefList(optionsSpecSet, datum, + optionsDefList, + operation == AT_ResetRelOptions); + /* Special-case validation of view options */ if (rel->rd_rel->relkind == RELKIND_VIEW) { Query *view_query = get_view_query(rel); - List *view_options = untransformRelOptions(newOptions); + List *view_options = optionsTextArrayToDefList(newOptions); ListCell *cell; bool check_option = false; @@ -15002,20 +15048,23 @@ ATExecSetRelOptions(Relation rel, List *defList, AlterTableType operation, * pretend there were none before. */ datum = (Datum) 0; - isnull = true; } else { + bool isnull; + /* Get the old reloptions */ datum = SysCacheGetAttr(RELOID, tuple, Anum_pg_class_reloptions, &isnull); + if (isnull) + datum = (Datum) 0; } - newOptions = transformRelOptions(isnull ? (Datum) 0 : datum, - defList, "toast", validnsps, false, - operation == AT_ResetRelOptions); - - (void) heap_reloptions(RELKIND_TOASTVALUE, newOptions, true); + optionsDefList = optionsDefListFilterNamespaces(defList, "toast"); + newOptions = optionsUpdateTexArrayWithDefList( + get_toast_relopt_spec_set(), + datum, optionsDefList, + operation == AT_ResetRelOptions); memset(repl_val, 0, sizeof(repl_val)); memset(repl_null, false, sizeof(repl_null)); diff --git a/src/backend/commands/tablespace.c b/src/backend/commands/tablespace.c index 113b480731..01aace1f1a 100644 --- a/src/backend/commands/tablespace.c +++ b/src/backend/commands/tablespace.c @@ -333,10 +333,9 @@ CreateTableSpace(CreateTableSpaceStmt *stmt) nulls[Anum_pg_tablespace_spcacl - 1] = true; /* Generate new proposed spcoptions (text array) */ - newOptions = transformRelOptions((Datum) 0, - stmt->options, - NULL, NULL, false, false); - (void) tablespace_reloptions(newOptions, true); + newOptions = optionDefListToTextArray(get_tablespace_options_spec_set(), + stmt->options); + if (newOptions != (Datum) 0) values[Anum_pg_tablespace_spcoptions - 1] = newOptions; else @@ -1052,11 +1051,12 @@ AlterTableSpaceOptions(AlterTableSpaceOptionsStmt *stmt) /* Generate new proposed spcoptions (text array) */ datum = heap_getattr(tup, Anum_pg_tablespace_spcoptions, RelationGetDescr(rel), &isnull); - newOptions = transformRelOptions(isnull ? (Datum) 0 : datum, - stmt->options, NULL, NULL, false, - stmt->isReset); - (void) tablespace_reloptions(newOptions, true); + if (isnull) + datum = (Datum) 0; + newOptions = optionsUpdateTexArrayWithDefList( + get_tablespace_options_spec_set(), + datum, stmt->options, stmt->isReset); /* Build new tuple. */ memset(repl_null, false, sizeof(repl_null)); memset(repl_repl, false, sizeof(repl_repl)); diff --git a/src/backend/foreign/foreign.c b/src/backend/foreign/foreign.c index f4f35728b4..307195a148 100644 --- a/src/backend/foreign/foreign.c +++ b/src/backend/foreign/foreign.c @@ -79,7 +79,7 @@ GetForeignDataWrapperExtended(Oid fdwid, bits16 flags) if (isnull) fdw->options = NIL; else - fdw->options = untransformRelOptions(datum); + fdw->options = optionsTextArrayToDefList(datum); ReleaseSysCache(tp); @@ -166,7 +166,7 @@ GetForeignServerExtended(Oid serverid, bits16 flags) if (isnull) server->options = NIL; else - server->options = untransformRelOptions(datum); + server->options = optionsTextArrayToDefList(datum); ReleaseSysCache(tp); @@ -238,7 +238,7 @@ GetUserMapping(Oid userid, Oid serverid) if (isnull) um->options = NIL; else - um->options = untransformRelOptions(datum); + um->options = optionsTextArrayToDefList(datum); ReleaseSysCache(tp); @@ -275,7 +275,7 @@ GetForeignTable(Oid relid) if (isnull) ft->options = NIL; else - ft->options = untransformRelOptions(datum); + ft->options = optionsTextArrayToDefList(datum); ReleaseSysCache(tp); @@ -308,7 +308,7 @@ GetForeignColumnOptions(Oid relid, AttrNumber attnum) if (isnull) options = NIL; else - options = untransformRelOptions(datum); + options = optionsTextArrayToDefList(datum); ReleaseSysCache(tp); @@ -516,7 +516,7 @@ pg_options_to_table(PG_FUNCTION_ARGS) List *options; ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo; - options = untransformRelOptions(array); + options = optionsTextArrayToDefList(array); rsinfo = (ReturnSetInfo *) fcinfo->resultinfo; /* prepare the result set */ @@ -614,7 +614,7 @@ is_conninfo_option(const char *option, Oid context) Datum postgresql_fdw_validator(PG_FUNCTION_ARGS) { - List *options_list = untransformRelOptions(PG_GETARG_DATUM(0)); + List *options_list = optionsTextArrayToDefList(PG_GETARG_DATUM(0)); Oid catalog = PG_GETARG_OID(1); ListCell *cell; diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index d5c2b2ff0b..d0df5822bd 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -1726,7 +1726,7 @@ generateClonedIndexStmt(RangeVar *heapRel, Relation source_idx, /* Add the operator class name, if non-default */ iparam->opclass = get_opclass(indclass->values[keyno], keycoltype); iparam->opclassopts = - untransformRelOptions(get_attoptions(source_relid, keyno + 1)); + optionsTextArrayToDefList(get_attoptions(source_relid, keyno + 1)); iparam->ordering = SORTBY_DEFAULT; iparam->nulls_ordering = SORTBY_NULLS_DEFAULT; @@ -1790,7 +1790,7 @@ generateClonedIndexStmt(RangeVar *heapRel, Relation source_idx, datum = SysCacheGetAttr(RELOID, ht_idxrel, Anum_pg_class_reloptions, &isnull); if (!isnull) - index->options = untransformRelOptions(datum); + index->options = optionsTextArrayToDefList(datum); /* If it's a partial index, decompile and append the predicate */ datum = SysCacheGetAttr(INDEXRELID, ht_idx, diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c index fa66b8017e..bafa48199b 100644 --- a/src/backend/tcop/utility.c +++ b/src/backend/tcop/utility.c @@ -1156,6 +1156,7 @@ ProcessUtilitySlow(ParseState *pstate, CreateStmt *cstmt = (CreateStmt *) stmt; Datum toast_options; static char *validnsps[] = HEAP_RELOPT_NAMESPACES; + List *toastDefList; /* Remember transformed RangeVar for LIKE */ table_rv = cstmt->relation; @@ -1179,15 +1180,16 @@ ProcessUtilitySlow(ParseState *pstate, * parse and validate reloptions for the toast * table */ - toast_options = transformRelOptions((Datum) 0, - cstmt->options, - "toast", - validnsps, - true, - false); - (void) heap_reloptions(RELKIND_TOASTVALUE, - toast_options, - true); + + optionsDefListValdateNamespaces( + ((CreateStmt *) stmt)->options, + validnsps); + + toastDefList = optionsDefListFilterNamespaces( + ((CreateStmt *) stmt)->options, "toast"); + + toast_options = optionDefListToTextArray( + get_toast_relopt_spec_set(), toastDefList); NewRelationCreateToastTable(address.objectId, toast_options); @@ -1296,9 +1298,12 @@ ProcessUtilitySlow(ParseState *pstate, * lock on (for example) a relation on which we have no * permissions. */ - lockmode = AlterTableGetLockLevel(atstmt->cmds); - relid = AlterTableLookupRelation(atstmt, lockmode); - + relid = AlterTableLookupRelation(atstmt, AccessShareLock); + if (OidIsValid(relid)) + { + lockmode = AlterTableGetLockLevel(relid, atstmt->cmds); + relid = AlterTableLookupRelation(atstmt, lockmode); + } if (OidIsValid(relid)) { AlterTableUtilityContext atcontext; diff --git a/src/backend/utils/cache/attoptcache.c b/src/backend/utils/cache/attoptcache.c index af978ccd4b..b295191d12 100644 --- a/src/backend/utils/cache/attoptcache.c +++ b/src/backend/utils/cache/attoptcache.c @@ -16,6 +16,7 @@ */ #include "postgres.h" +#include "access/options.h" #include "access/reloptions.h" #include "utils/attoptcache.h" #include "utils/catcache.h" @@ -149,7 +150,8 @@ get_attribute_options(Oid attrelid, int attnum) opts = NULL; else { - bytea *bytea_opts = attribute_reloptions(datum, false); + bytea *bytea_opts = optionsTextArrayToBytea( + get_attribute_options_spec_set(), datum, 0); opts = MemoryContextAlloc(CacheMemoryContext, VARSIZE(bytea_opts)); diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c index cc9b0c6524..761811d940 100644 --- a/src/backend/utils/cache/relcache.c +++ b/src/backend/utils/cache/relcache.c @@ -463,7 +463,7 @@ static void RelationParseRelOptions(Relation relation, HeapTuple tuple) { bytea *options; - amoptions_function amoptsfn; + amreloptspecset_function amoptspecsetfn; relation->rd_options = NULL; @@ -478,11 +478,11 @@ RelationParseRelOptions(Relation relation, HeapTuple tuple) case RELKIND_VIEW: case RELKIND_MATVIEW: case RELKIND_PARTITIONED_TABLE: - amoptsfn = NULL; + amoptspecsetfn = NULL; break; case RELKIND_INDEX: case RELKIND_PARTITIONED_INDEX: - amoptsfn = relation->rd_indam->amoptions; + amoptspecsetfn = relation->rd_indam->amreloptspecset; break; default: return; @@ -493,7 +493,7 @@ RelationParseRelOptions(Relation relation, HeapTuple tuple) * we might not have any other for pg_class yet (consider executing this * code for pg_class itself) */ - options = extractRelOptions(tuple, GetPgClassDescriptor(), amoptsfn); + options = extractRelOptions(tuple, GetPgClassDescriptor(), amoptspecsetfn); /* * Copy parsed data into CacheMemoryContext. To guard against the diff --git a/src/backend/utils/cache/spccache.c b/src/backend/utils/cache/spccache.c index ec63cdc8e5..ea9552ca67 100644 --- a/src/backend/utils/cache/spccache.c +++ b/src/backend/utils/cache/spccache.c @@ -149,7 +149,8 @@ get_tablespace(Oid spcid) opts = NULL; else { - bytea *bytea_opts = tablespace_reloptions(datum, false); + bytea *bytea_opts = optionsTextArrayToBytea( + get_tablespace_options_spec_set(), datum, 0); opts = MemoryContextAlloc(CacheMemoryContext, VARSIZE(bytea_opts)); memcpy(opts, bytea_opts, VARSIZE(bytea_opts)); diff --git a/src/bin/pg_basebackup/streamutil.c b/src/bin/pg_basebackup/streamutil.c index feee451d59..a611f9da2c 100644 --- a/src/bin/pg_basebackup/streamutil.c +++ b/src/bin/pg_basebackup/streamutil.c @@ -832,7 +832,7 @@ AppendStringCommandOption(PQExpBuffer buf, bool use_new_option_syntax, { AppendPlainCommandOption(buf, use_new_option_syntax, option_name); - if (option_value != NULL) + if (option_value !=NULL) { size_t length = strlen(option_value); char *escaped_value = palloc(1 + 2 * length); diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h index f25c9d58a7..0ef48df437 100644 --- a/src/include/access/amapi.h +++ b/src/include/access/amapi.h @@ -140,10 +140,6 @@ typedef void (*amcostestimate_function) (struct PlannerInfo *root, double *indexCorrelation, double *indexPages); -/* parse index reloptions */ -typedef bytea *(*amoptions_function) (Datum reloptions, - bool validate); - /* report AM, index, or index column property */ typedef bool (*amproperty_function) (Oid index_oid, int attno, IndexAMProperty prop, const char *propname, @@ -190,6 +186,9 @@ typedef void (*ammarkpos_function) (IndexScanDesc scan); /* restore marked scan position */ typedef void (*amrestrpos_function) (IndexScanDesc scan); +/* get catalog of reloptions definitions */ +typedef void *(*amreloptspecset_function) (); + /* * Callback function signatures - for parallel index scans. */ @@ -272,7 +271,6 @@ typedef struct IndexAmRoutine amvacuumcleanup_function amvacuumcleanup; amcanreturn_function amcanreturn; /* can be NULL */ amcostestimate_function amcostestimate; - amoptions_function amoptions; amproperty_function amproperty; /* can be NULL */ ambuildphasename_function ambuildphasename; /* can be NULL */ amvalidate_function amvalidate; @@ -284,6 +282,7 @@ typedef struct IndexAmRoutine amendscan_function amendscan; ammarkpos_function ammarkpos; /* can be NULL */ amrestrpos_function amrestrpos; /* can be NULL */ + amreloptspecset_function amreloptspecset; /* can be NULL */ /* interface functions to support parallel index scans */ amestimateparallelscan_function amestimateparallelscan; /* can be NULL */ diff --git a/src/include/access/brin.h b/src/include/access/brin.h index 1c786281dd..a552003224 100644 --- a/src/include/access/brin.h +++ b/src/include/access/brin.h @@ -37,6 +37,8 @@ typedef struct BrinStatsData #define BRIN_DEFAULT_PAGES_PER_RANGE 128 +#define BRIN_MIN_PAGES_PER_RANGE 1 +#define BRIN_MAX_PAGES_PER_RANGE 131072 #define BrinGetPagesPerRange(relation) \ (AssertMacro(relation->rd_rel->relkind == RELKIND_INDEX && \ relation->rd_rel->relam == BRIN_AM_OID), \ diff --git a/src/include/access/brin_internal.h b/src/include/access/brin_internal.h index a5a9772621..fd99dfae80 100644 --- a/src/include/access/brin_internal.h +++ b/src/include/access/brin_internal.h @@ -14,6 +14,7 @@ #include "access/amapi.h" #include "storage/bufpage.h" #include "utils/typcache.h" +#include "access/options.h" /* @@ -109,6 +110,7 @@ extern IndexBulkDeleteResult *brinbulkdelete(IndexVacuumInfo *info, extern IndexBulkDeleteResult *brinvacuumcleanup(IndexVacuumInfo *info, IndexBulkDeleteResult *stats); extern bytea *brinoptions(Datum reloptions, bool validate); +extern void *bringetreloptspecset(void); /* brin_validate.c */ extern bool brinvalidate(Oid opclassoid); diff --git a/src/include/access/gin_private.h b/src/include/access/gin_private.h index 3013a44bae..5ba3a7578b 100644 --- a/src/include/access/gin_private.h +++ b/src/include/access/gin_private.h @@ -109,6 +109,7 @@ extern Datum *ginExtractEntries(GinState *ginstate, OffsetNumber attnum, extern OffsetNumber gintuple_get_attrnum(GinState *ginstate, IndexTuple tuple); extern Datum gintuple_get_key(GinState *ginstate, IndexTuple tuple, GinNullCategory *category); +extern void *gingetreloptspecset(void); /* gininsert.c */ extern IndexBuildResult *ginbuild(Relation heap, Relation index, diff --git a/src/include/access/gist_private.h b/src/include/access/gist_private.h index 7b8749c8db..4579f1b5ff 100644 --- a/src/include/access/gist_private.h +++ b/src/include/access/gist_private.h @@ -22,6 +22,7 @@ #include "storage/buffile.h" #include "utils/hsearch.h" #include "access/genam.h" +#include "access/options.h" /* * Maximum number of "halves" a page can be split into in one operation. @@ -388,6 +389,7 @@ typedef enum GistOptBufferingMode GIST_OPTION_BUFFERING_OFF, } GistOptBufferingMode; + /* * Storage type for GiST's reloptions */ @@ -479,7 +481,7 @@ extern void gistadjustmembers(Oid opfamilyoid, #define GIST_MIN_FILLFACTOR 10 #define GIST_DEFAULT_FILLFACTOR 90 -extern bytea *gistoptions(Datum reloptions, bool validate); +extern void *gistgetreloptspecset(void); extern bool gistproperty(Oid index_oid, int attno, IndexAMProperty prop, const char *propname, bool *res, bool *isnull); diff --git a/src/include/access/hash.h b/src/include/access/hash.h index 9c7d81525b..6fa680afe3 100644 --- a/src/include/access/hash.h +++ b/src/include/access/hash.h @@ -380,7 +380,6 @@ extern IndexBulkDeleteResult *hashbulkdelete(IndexVacuumInfo *info, void *callback_state); extern IndexBulkDeleteResult *hashvacuumcleanup(IndexVacuumInfo *info, IndexBulkDeleteResult *stats); -extern bytea *hashoptions(Datum reloptions, bool validate); extern bool hashvalidate(Oid opclassoid); extern void hashadjustmembers(Oid opfamilyoid, Oid opclassoid, @@ -474,6 +473,7 @@ extern BlockNumber _hash_get_newblock_from_oldbucket(Relation rel, Bucket old_bu extern Bucket _hash_get_newbucket_from_oldbucket(Relation rel, Bucket old_bucket, uint32 lowmask, uint32 maxbucket); extern void _hash_kill_items(IndexScanDesc scan); +extern void *hashgetreloptspecset(void); /* hash.c */ extern void hashbucketcleanup(Relation rel, Bucket cur_bucket, diff --git a/src/include/access/nbtree.h b/src/include/access/nbtree.h index 7493043348..0ec72c7dd7 100644 --- a/src/include/access/nbtree.h +++ b/src/include/access/nbtree.h @@ -1298,7 +1298,7 @@ extern void _bt_end_vacuum(Relation rel); extern void _bt_end_vacuum_callback(int code, Datum arg); extern Size BTreeShmemSize(void); extern void BTreeShmemInit(void); -extern bytea *btoptions(Datum reloptions, bool validate); +extern void *btgetreloptspecset(void); extern bool btproperty(Oid index_oid, int attno, IndexAMProperty prop, const char *propname, bool *res, bool *isnull); diff --git a/src/include/access/options.h b/src/include/access/options.h new file mode 100644 index 0000000000..c87dc9e8d9 --- /dev/null +++ b/src/include/access/options.h @@ -0,0 +1,274 @@ +/*------------------------------------------------------------------------- + * + * options.h + * An uniform, context-free API for processing name=value options. Used + * to process relation options (reloptions), attribute options, opclass + * options, etc. + * + * Note: the functions dealing with text-array options values declare + * them as Datum, not ArrayType *, to avoid needing to include array.h + * into a lot of low-level code. + * + * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * + * src/include/access/options.h + * + *------------------------------------------------------------------------- + */ + +#ifndef OPTIONS_H +#define OPTIONS_H + +#include "storage/lock.h" +#include "nodes/pg_list.h" + + +/* supported option types */ +typedef enum option_type +{ + OPTION_TYPE_BOOL, + OPTION_TYPE_INT, + OPTION_TYPE_REAL, + OPTION_TYPE_ENUM, + OPTION_TYPE_STRING, +} option_type; + +/* + * Each Option Value item passes through certain life cycle. Option Value + * Status specifies at what point of life cycle this Option Value is now. + * Option Value Statuses are: + * EMPTY - Option Value structure have been just created + * RAW - Option Value have been read, but has not been parsed or validated yet. + * Use raw_value structure member to access Raw Value + * PARSED - Option Value have been parsed, you should use proper union member + * to access Parsed Option Value, according to Option Type + * FOR_RESET - Specifies that this Option is designated for resetting to + * default value + */ +typedef enum option_value_status +{ + OPTION_VALUE_STATUS_EMPTY, /* Option was just initialized */ + OPTION_VALUE_STATUS_RAW, /* Option just came from syntax analyzer in + * has name, and raw (unparsed) value */ + OPTION_VALUE_STATUS_PARSED, /* Option was parsed and has link to Option + * Spec Set entry and proper value */ + OPTION_VALUE_STATUS_FOR_RESET, /* This option came from ALTER xxx RESET */ +} option_value_status; + +/* + * opt_enum_elt_def -- One member of the array of acceptable values + * of an enum reloption. + */ +typedef struct opt_enum_elt_def +{ + const char *string_val; + int symbol_val; +} opt_enum_elt_def; + + +typedef struct option_value option_value; + +/* Function that would be called after option validation. + * Might be needed for custom warnings, errors, or for changing + * option value after being validated, etc. + */ +typedef void (*option_value_postvalidate) (option_value *value); + +/* generic structure to store Option Spec information */ +typedef struct option_spec_basic +{ + const char *name; /* must be first (used as list termination + * marker) */ + const char *desc; + LOCKMODE lockmode; + option_type type; + int struct_offset; /* offset of the value in Bytea representation */ + option_value_postvalidate postvalidate_fn; +} option_spec_basic; + + +/* reloptions records for specific variable types */ +typedef struct option_spec_bool +{ + option_spec_basic base; + bool default_val; +} option_spec_bool; + +typedef struct option_spec_int +{ + option_spec_basic base; + int default_val; + int min; + int max; +} option_spec_int; + +typedef struct option_spec_real +{ + option_spec_basic base; + double default_val; + double min; + double max; +} option_spec_real; + +typedef struct option_spec_enum +{ + option_spec_basic base; + opt_enum_elt_def *members; /* Null terminated array of allowed names and + * corresponding values */ + int default_val; /* Default value, may differ from values in + * members array */ + const char *detailmsg; +} option_spec_enum; + +/* validation routines for strings */ +typedef void (*validate_string_option) (const char *value); +typedef Size (*fill_string_option) (const char *value, void *ptr); + +/* + * When storing sting reloptions, we should deal with special case when + * option value is not set. For fixed length options, we just copy default + * option value into the binary structure. For varlen value, there can be + * "not set" special case, with no default value offered. + * In this case we will set offset value to -1, so code that use reloptions + * can deal this case. For better readability it was defined as a constant. + */ +#define OPTION_STRING_VALUE_NOT_SET_OFFSET -1 + +typedef struct option_spec_string +{ + option_spec_basic base; + validate_string_option validate_cb; + fill_string_option fill_cb; + char *default_val; +} option_spec_string; + +typedef void (*postprocess_bytea_options_function) (void *data, bool validate); + +typedef struct options_spec_set +{ + option_spec_basic **definitions; + int num; /* Number of spec_set items in use */ + int num_allocated; /* Number of spec_set items allocated */ + bool assert_on_realloc; /* If number of items of the spec_set were + * strictly set to certain value assert on + * adding more items */ + bool is_local; /* If true specset is in local memory context */ + Size struct_size; /* Size of a structure for options in binary + * representation */ + postprocess_bytea_options_function postprocess_fun; /* This function is + * called after options + * were converted in + * Bytea representation. + * Can be used for extra + * validation etc. */ + char *namspace; /* Spec Set is used for options from this + * namespase */ +} options_spec_set; + + +/* holds an option value parsed or unparsed */ +typedef struct option_value +{ + option_spec_basic *gen; + char *namspace; + option_value_status status; + char *raw_value; /* allocated separately */ + char *raw_name; + union + { + bool bool_val; + int int_val; + double real_val; + int enum_val; + char *string_val; /* allocated separately */ + } values; +} option_value; + + +/* + * Options Spec Set related functions + */ +extern options_spec_set *allocateOptionsSpecSet(const char *namspace, + int bytea_size, bool is_local, int num_items_expected); + +extern void optionsSpecSetAddBool(options_spec_set *spec_set, const char *name, + const char *desc, LOCKMODE lockmode, + int struct_offset, + option_value_postvalidate postvalidate_fn, + bool default_val); + +extern void optionsSpecSetAddInt(options_spec_set *spec_set, const char *name, + const char *desc, LOCKMODE lockmode, + int struct_offset, + option_value_postvalidate postvalidate_fn, + int default_val, int min_val, int max_val); + +extern void optionsSpecSetAddReal(options_spec_set *spec_set, const char *name, + const char *desc, LOCKMODE lockmode, + int struct_offset, + option_value_postvalidate postvalidate_fn, + double default_val, double min_val, double max_val); + +extern void optionsSpecSetAddEnum(options_spec_set *spec_set, const char *name, + const char *desc, LOCKMODE lockmode, + int struct_offset, + option_value_postvalidate postvalidate_fn, + opt_enum_elt_def *members, int default_val, + const char *detailmsg); + + +extern void optionsSpecSetAddString(options_spec_set *spec_set, + const char *name, const char *desc, + LOCKMODE lockmode, int struct_offset, + option_value_postvalidate postvalidate_fn, + const char *default_val, + validate_string_option validator, + fill_string_option filler); + + +/* + * This macro allows to get string option value from bytea representation. + * "optstruct" - is a structure that is stored in bytea options representation + * "member" - member of this structure that has string option value + * (actually string values are stored in bytea after the structure, and + * and "member" will contain an offset to this value. This macro do all + * the math + */ +#define GET_STRING_OPTION(optstruct, member) \ + ((optstruct)->member == OPTION_STRING_VALUE_NOT_SET_OFFSET ? NULL : \ + (char *)(optstruct) + (optstruct)->member) + +/* + * Functions related to option conversion, parsing, manipulation + * and validation + */ +extern void optionsDefListValdateNamespaces(List *defList, + char **allowed_namspaces); +extern List *optionsDefListFilterNamespaces(List *defList, + const char *namspace); +extern List *optionsTextArrayToDefList(Datum options); +extern Datum optionsDefListToTextArray(List *defList); + +/* + * Meta functions that uses functions above to get options for relations, + * tablespaces, views and so on + */ + +extern bytea *optionsTextArrayToBytea(options_spec_set *spec_set, Datum data, + bool validate); +extern Datum optionsUpdateTexArrayWithDefList(options_spec_set *spec_set, + Datum oldOptions, List *defList, bool do_reset); +extern Datum optionDefListToTextArray(options_spec_set *spec_set, + List *defList); + +/* Internal functions */ + +extern List *optionsTextArrayToRawValues(Datum array_datum); +extern List *optionsParseRawValues(List *raw_values, + options_spec_set *spec_set, bool validate); +extern bytea *optionsValuesToBytea(List *options, options_spec_set *spec_set); + + +#endif /* OPTIONS_H */ diff --git a/src/include/access/reloptions.h b/src/include/access/reloptions.h index 81829b8270..f80e1e840c 100644 --- a/src/include/access/reloptions.h +++ b/src/include/access/reloptions.h @@ -1,14 +1,9 @@ /*------------------------------------------------------------------------- * * reloptions.h - * Core support for relation and tablespace options (pg_class.reloptions ++ * Support for relation view and tablespace options (pg_class.reloptions * and pg_tablespace.spcoptions) * - * Note: the functions dealing with text-array reloptions values declare - * them as Datum, not ArrayType *, to avoid needing to include array.h - * into a lot of low-level code. - * - * * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group * Portions Copyright (c) 1994, Regents of the University of California * @@ -22,178 +17,40 @@ #include "access/amapi.h" #include "access/htup.h" #include "access/tupdesc.h" +#include "access/options.h" #include "nodes/pg_list.h" #include "storage/lock.h" -/* types supported by reloptions */ -typedef enum relopt_type -{ - RELOPT_TYPE_BOOL, - RELOPT_TYPE_INT, - RELOPT_TYPE_REAL, - RELOPT_TYPE_ENUM, - RELOPT_TYPE_STRING, -} relopt_type; - -/* kinds supported by reloptions */ -typedef enum relopt_kind -{ - RELOPT_KIND_LOCAL = 0, - RELOPT_KIND_HEAP = (1 << 0), - RELOPT_KIND_TOAST = (1 << 1), - RELOPT_KIND_BTREE = (1 << 2), - RELOPT_KIND_HASH = (1 << 3), - RELOPT_KIND_GIN = (1 << 4), - RELOPT_KIND_GIST = (1 << 5), - RELOPT_KIND_ATTRIBUTE = (1 << 6), - RELOPT_KIND_TABLESPACE = (1 << 7), - RELOPT_KIND_SPGIST = (1 << 8), - RELOPT_KIND_VIEW = (1 << 9), - RELOPT_KIND_BRIN = (1 << 10), - RELOPT_KIND_PARTITIONED = (1 << 11), - /* if you add a new kind, make sure you update "last_default" too */ - RELOPT_KIND_LAST_DEFAULT = RELOPT_KIND_PARTITIONED, - /* some compilers treat enums as signed ints, so we can't use 1 << 31 */ - RELOPT_KIND_MAX = (1 << 30) -} relopt_kind; - /* reloption namespaces allowed for heaps -- currently only TOAST */ #define HEAP_RELOPT_NAMESPACES { "toast", NULL } -/* generic struct to hold shared data */ -typedef struct relopt_gen -{ - const char *name; /* must be first (used as list termination - * marker) */ - const char *desc; - bits32 kinds; - LOCKMODE lockmode; - int namelen; - relopt_type type; -} relopt_gen; - -/* holds a parsed value */ -typedef struct relopt_value -{ - relopt_gen *gen; - bool isset; - union - { - bool bool_val; - int int_val; - double real_val; - int enum_val; - char *string_val; /* allocated separately */ - } values; -} relopt_value; - -/* reloptions records for specific variable types */ -typedef struct relopt_bool -{ - relopt_gen gen; - bool default_val; -} relopt_bool; - -typedef struct relopt_int -{ - relopt_gen gen; - int default_val; - int min; - int max; -} relopt_int; - -typedef struct relopt_real -{ - relopt_gen gen; - double default_val; - double min; - double max; -} relopt_real; - /* - * relopt_enum_elt_def -- One member of the array of acceptable values - * of an enum reloption. + * backward compatibility aliases so local reloption code of custom validator + * can work. */ -typedef struct relopt_enum_elt_def -{ - const char *string_val; - int symbol_val; -} relopt_enum_elt_def; +typedef option_value relopt_value; +typedef fill_string_option fill_string_relopt; +typedef validate_string_option validate_string_relopt; +#define GET_STRING_RELOPTION(optstruct, member) \ + GET_STRING_OPTION(optstruct, member) -typedef struct relopt_enum -{ - relopt_gen gen; - relopt_enum_elt_def *members; - int default_val; - const char *detailmsg; - /* null-terminated array of members */ -} relopt_enum; -/* validation routines for strings */ -typedef void (*validate_string_relopt) (const char *value); -typedef Size (*fill_string_relopt) (const char *value, void *ptr); +/* + * relopts_validator functions is left for backward compatibility for using + * with local reloptions. Should not be used elsewhere + */ /* validation routine for the whole option set */ -typedef void (*relopts_validator) (void *parsed_options, relopt_value *vals, int nvals); - -typedef struct relopt_string -{ - relopt_gen gen; - int default_len; - bool default_isnull; - validate_string_relopt validate_cb; - fill_string_relopt fill_cb; - char *default_val; -} relopt_string; - -/* This is the table datatype for build_reloptions() */ -typedef struct -{ - const char *optname; /* option's name */ - relopt_type opttype; /* option's datatype */ - int offset; /* offset of field in result struct */ -} relopt_parse_elt; - -/* Local reloption definition */ -typedef struct local_relopt -{ - relopt_gen *option; /* option definition */ - int offset; /* offset of parsed value in bytea structure */ -} local_relopt; +typedef void (*relopts_validator) (void *parsed_options, relopt_value *vals, + int nvals); /* Structure to hold local reloption data for build_local_reloptions() */ typedef struct local_relopts { - List *options; /* list of local_relopt definitions */ List *validators; /* list of relopts_validator callbacks */ - Size relopt_struct_size; /* size of parsed bytea structure */ + options_spec_set *spec_set; /* Spec Set to store options info */ } local_relopts; -/* - * Utility macro to get a value for a string reloption once the options - * are parsed. This gets a pointer to the string value itself. "optstruct" - * is the StdRdOptions struct or equivalent, "member" is the struct member - * corresponding to the string option. - */ -#define GET_STRING_RELOPTION(optstruct, member) \ - ((optstruct)->member == 0 ? NULL : \ - (char *)(optstruct) + (optstruct)->member) - -extern relopt_kind add_reloption_kind(void); -extern void add_bool_reloption(bits32 kinds, const char *name, const char *desc, - bool default_val, LOCKMODE lockmode); -extern void add_int_reloption(bits32 kinds, const char *name, const char *desc, - int default_val, int min_val, int max_val, - LOCKMODE lockmode); -extern void add_real_reloption(bits32 kinds, const char *name, const char *desc, - double default_val, double min_val, double max_val, - LOCKMODE lockmode); -extern void add_enum_reloption(bits32 kinds, const char *name, const char *desc, - relopt_enum_elt_def *members, int default_val, - const char *detailmsg, LOCKMODE lockmode); -extern void add_string_reloption(bits32 kinds, const char *name, const char *desc, - const char *default_val, validate_string_relopt validator, - LOCKMODE lockmode); extern void init_local_reloptions(local_relopts *relopts, Size relopt_struct_size); extern void register_reloptions_validator(local_relopts *relopts, @@ -210,7 +67,7 @@ extern void add_local_real_reloption(local_relopts *relopts, const char *name, int offset); extern void add_local_enum_reloption(local_relopts *relopts, const char *name, const char *desc, - relopt_enum_elt_def *members, + opt_enum_elt_def *members, int default_val, const char *detailmsg, int offset); extern void add_local_string_reloption(local_relopts *relopts, const char *name, @@ -219,29 +76,17 @@ extern void add_local_string_reloption(local_relopts *relopts, const char *name, validate_string_relopt validator, fill_string_relopt filler, int offset); -extern Datum transformRelOptions(Datum oldOptions, List *defList, - const char *namspace, char *validnsps[], - bool acceptOidsOff, bool isReset); -extern List *untransformRelOptions(Datum options); extern bytea *extractRelOptions(HeapTuple tuple, TupleDesc tupdesc, - amoptions_function amoptions); -extern void *build_reloptions(Datum reloptions, bool validate, - relopt_kind kind, - Size relopt_struct_size, - const relopt_parse_elt *relopt_elems, - int num_relopt_elems); + amreloptspecset_function amoptions_def_set); extern void *build_local_reloptions(local_relopts *relopts, Datum options, bool validate); -extern bytea *default_reloptions(Datum reloptions, bool validate, - relopt_kind kind); -extern bytea *heap_reloptions(char relkind, Datum reloptions, bool validate); -extern bytea *view_reloptions(Datum reloptions, bool validate); -extern bytea *partitioned_table_reloptions(Datum reloptions, bool validate); -extern bytea *index_reloptions(amoptions_function amoptions, Datum reloptions, - bool validate); -extern bytea *attribute_reloptions(Datum reloptions, bool validate); -extern bytea *tablespace_reloptions(Datum reloptions, bool validate); -extern LOCKMODE AlterTableGetRelOptionsLockLevel(List *defList); +options_spec_set *get_heap_relopt_spec_set(void); +options_spec_set *get_toast_relopt_spec_set(void); +options_spec_set *get_partitioned_relopt_spec_set(void); +options_spec_set *get_view_relopt_spec_set(void); +options_spec_set *get_attribute_options_spec_set(void); +options_spec_set *get_tablespace_options_spec_set(void); +extern LOCKMODE AlterTableGetRelOptionsLockLevel(Relation rel, List *defList); #endif /* RELOPTIONS_H */ diff --git a/src/include/access/spgist.h b/src/include/access/spgist.h index d6a4953120..7b113d7a76 100644 --- a/src/include/access/spgist.h +++ b/src/include/access/spgist.h @@ -189,9 +189,6 @@ typedef struct spgLeafConsistentOut } spgLeafConsistentOut; -/* spgutils.c */ -extern bytea *spgoptions(Datum reloptions, bool validate); - /* spginsert.c */ extern IndexBuildResult *spgbuild(Relation heap, Relation index, struct IndexInfo *indexInfo); diff --git a/src/include/access/spgist_private.h b/src/include/access/spgist_private.h index 2e9c757b30..a4d809cc29 100644 --- a/src/include/access/spgist_private.h +++ b/src/include/access/spgist_private.h @@ -529,6 +529,7 @@ extern OffsetNumber SpGistPageAddNewItem(SpGistState *state, Page page, extern bool spgproperty(Oid index_oid, int attno, IndexAMProperty prop, const char *propname, bool *res, bool *isnull); +extern void *spggetreloptspecset(void); /* spgdoinsert.c */ extern void spgUpdateNodeLink(SpGistInnerTuple tup, int nodeN, diff --git a/src/include/commands/tablecmds.h b/src/include/commands/tablecmds.h index 85cbad3d0c..1c7bb7f925 100644 --- a/src/include/commands/tablecmds.h +++ b/src/include/commands/tablecmds.h @@ -36,7 +36,7 @@ extern Oid AlterTableLookupRelation(AlterTableStmt *stmt, LOCKMODE lockmode); extern void AlterTable(AlterTableStmt *stmt, LOCKMODE lockmode, struct AlterTableUtilityContext *context); -extern LOCKMODE AlterTableGetLockLevel(List *cmds); +extern LOCKMODE AlterTableGetLockLevel(Oid relid, List *cmds); extern void ATExecChangeOwner(Oid relationOid, Oid newOwnerId, bool recursing, LOCKMODE lockmode); diff --git a/src/test/modules/dummy_index_am/dummy_index_am.c b/src/test/modules/dummy_index_am/dummy_index_am.c index 18185d0206..a8744fc415 100644 --- a/src/test/modules/dummy_index_am/dummy_index_am.c +++ b/src/test/modules/dummy_index_am/dummy_index_am.c @@ -14,7 +14,7 @@ #include "postgres.h" #include "access/amapi.h" -#include "access/reloptions.h" +#include "access/options.h" #include "catalog/index.h" #include "commands/vacuum.h" #include "nodes/pathnodes.h" @@ -23,11 +23,7 @@ PG_MODULE_MAGIC; -/* parse table for fillRelOptions */ -relopt_parse_elt di_relopt_tab[6]; - -/* Kind of relation options for dummy index */ -relopt_kind di_relopt_kind; +void _PG_init(void); typedef enum DummyAmEnum { @@ -47,7 +43,7 @@ typedef struct DummyIndexOptions int option_string_null_offset; } DummyIndexOptions; -relopt_enum_elt_def dummyAmEnumValues[] = +static opt_enum_elt_def dummyAmEnumValues[] = { {"one", DUMMY_AM_ENUM_ONE}, {"two", DUMMY_AM_ENUM_TWO}, @@ -61,77 +57,82 @@ PG_FUNCTION_INFO_V1(dihandler); * Validation function for string relation options. */ static void -validate_string_option(const char *value) +divalidate_string_option(const char *value) { ereport(NOTICE, (errmsg("new option value for string parameter %s", value ? value : "NULL"))); } -/* - * This function creates a full set of relation option types, - * with various patterns. - */ -static void -create_reloptions_table(void) +static options_spec_set *di_relopt_specset = NULL; +void *digetreloptspecset(void); + +void * +digetreloptspecset(void) { - di_relopt_kind = add_reloption_kind(); - - add_int_reloption(di_relopt_kind, "option_int", - "Integer option for dummy_index_am", - 10, -10, 100, AccessExclusiveLock); - di_relopt_tab[0].optname = "option_int"; - di_relopt_tab[0].opttype = RELOPT_TYPE_INT; - di_relopt_tab[0].offset = offsetof(DummyIndexOptions, option_int); - - add_real_reloption(di_relopt_kind, "option_real", - "Real option for dummy_index_am", - 3.1415, -10, 100, AccessExclusiveLock); - di_relopt_tab[1].optname = "option_real"; - di_relopt_tab[1].opttype = RELOPT_TYPE_REAL; - di_relopt_tab[1].offset = offsetof(DummyIndexOptions, option_real); - - add_bool_reloption(di_relopt_kind, "option_bool", - "Boolean option for dummy_index_am", - true, AccessExclusiveLock); - di_relopt_tab[2].optname = "option_bool"; - di_relopt_tab[2].opttype = RELOPT_TYPE_BOOL; - di_relopt_tab[2].offset = offsetof(DummyIndexOptions, option_bool); - - add_enum_reloption(di_relopt_kind, "option_enum", - "Enum option for dummy_index_am", - dummyAmEnumValues, - DUMMY_AM_ENUM_ONE, - "Valid values are \"one\" and \"two\".", - AccessExclusiveLock); - di_relopt_tab[3].optname = "option_enum"; - di_relopt_tab[3].opttype = RELOPT_TYPE_ENUM; - di_relopt_tab[3].offset = offsetof(DummyIndexOptions, option_enum); - - add_string_reloption(di_relopt_kind, "option_string_val", - "String option for dummy_index_am with non-NULL default", - "DefaultValue", &validate_string_option, - AccessExclusiveLock); - di_relopt_tab[4].optname = "option_string_val"; - di_relopt_tab[4].opttype = RELOPT_TYPE_STRING; - di_relopt_tab[4].offset = offsetof(DummyIndexOptions, - option_string_val_offset); + if (di_relopt_specset) + return di_relopt_specset; + + di_relopt_specset = allocateOptionsSpecSet(NULL, + sizeof(DummyIndexOptions), false, 6); + + optionsSpecSetAddInt( + di_relopt_specset, "option_int", + "Integer option for dummy_index_am", + AccessExclusiveLock, + offsetof(DummyIndexOptions, option_int), NULL, + 10, -10, 100 + ); + + + optionsSpecSetAddReal( + di_relopt_specset, "option_real", + "Real option for dummy_index_am", + AccessExclusiveLock, + offsetof(DummyIndexOptions, option_real), NULL, + 3.1415, -10, 100 + ); + + optionsSpecSetAddBool( + di_relopt_specset, "option_bool", + "Boolean option for dummy_index_am", + AccessExclusiveLock, + offsetof(DummyIndexOptions, option_bool), NULL, true + ); + + optionsSpecSetAddEnum(di_relopt_specset, "option_enum", + "Enum option for dummy_index_am", + AccessExclusiveLock, + offsetof(DummyIndexOptions, option_enum), NULL, + dummyAmEnumValues, + DUMMY_AM_ENUM_ONE, + "Valid values are \"one\" and \"two\"." + ); + + optionsSpecSetAddString(di_relopt_specset, "option_string_val", + "String option for dummy_index_am with non-NULL default", + AccessExclusiveLock, + offsetof(DummyIndexOptions, option_string_val_offset), NULL, + "DefaultValue", &divalidate_string_option, NULL + ); /* * String option for dummy_index_am with NULL default, and without * description. */ - add_string_reloption(di_relopt_kind, "option_string_null", - NULL, /* description */ - NULL, &validate_string_option, - AccessExclusiveLock); - di_relopt_tab[5].optname = "option_string_null"; - di_relopt_tab[5].opttype = RELOPT_TYPE_STRING; - di_relopt_tab[5].offset = offsetof(DummyIndexOptions, - option_string_null_offset); + + optionsSpecSetAddString(di_relopt_specset, "option_string_null", + NULL, /* description */ + AccessExclusiveLock, + offsetof(DummyIndexOptions, option_string_null_offset), NULL, + NULL, &divalidate_string_option, NULL + ); + + return di_relopt_specset; } + /* * Build a new index. */ @@ -216,19 +217,6 @@ dicostestimate(PlannerInfo *root, IndexPath *path, double loop_count, *indexPages = 1; } -/* - * Parse relation options for index AM, returning a DummyIndexOptions - * structure filled with option values. - */ -static bytea * -dioptions(Datum reloptions, bool validate) -{ - return (bytea *) build_reloptions(reloptions, validate, - di_relopt_kind, - sizeof(DummyIndexOptions), - di_relopt_tab, lengthof(di_relopt_tab)); -} - /* * Validator for index AM. */ @@ -308,7 +296,6 @@ dihandler(PG_FUNCTION_ARGS) amroutine->amvacuumcleanup = divacuumcleanup; amroutine->amcanreturn = NULL; amroutine->amcostestimate = dicostestimate; - amroutine->amoptions = dioptions; amroutine->amproperty = NULL; amroutine->ambuildphasename = NULL; amroutine->amvalidate = divalidate; @@ -322,12 +309,7 @@ dihandler(PG_FUNCTION_ARGS) amroutine->amestimateparallelscan = NULL; amroutine->aminitparallelscan = NULL; amroutine->amparallelrescan = NULL; + amroutine->amreloptspecset = digetreloptspecset; PG_RETURN_POINTER(amroutine); } - -void -_PG_init(void) -{ - create_reloptions_table(); -} diff --git a/src/test/modules/test_oat_hooks/expected/alter_table.out b/src/test/modules/test_oat_hooks/expected/alter_table.out index 8cbacca2c9..8319cd0694 100644 --- a/src/test/modules/test_oat_hooks/expected/alter_table.out +++ b/src/test/modules/test_oat_hooks/expected/alter_table.out @@ -82,6 +82,8 @@ ALTER TABLE test_oat_schema.test_oat_tab ENABLE ROW LEVEL SECURITY; NOTICE: in process utility: superuser attempting ALTER TABLE NOTICE: in object access: superuser attempting namespace search (subId=0x0) [report on violation, allowed] NOTICE: in object access: superuser finished namespace search (subId=0x0) [report on violation, allowed] +NOTICE: in object access: superuser attempting namespace search (subId=0x0) [report on violation, allowed] +NOTICE: in object access: superuser finished namespace search (subId=0x0) [report on violation, allowed] NOTICE: in object access: superuser attempting alter (subId=0x0) [explicit without auxiliary object] NOTICE: in object access: superuser finished alter (subId=0x0) [explicit without auxiliary object] NOTICE: in process utility: superuser finished ALTER TABLE @@ -89,6 +91,8 @@ ALTER TABLE test_oat_schema.test_oat_tab DISABLE ROW LEVEL SECURITY; NOTICE: in process utility: superuser attempting ALTER TABLE NOTICE: in object access: superuser attempting namespace search (subId=0x0) [report on violation, allowed] NOTICE: in object access: superuser finished namespace search (subId=0x0) [report on violation, allowed] +NOTICE: in object access: superuser attempting namespace search (subId=0x0) [report on violation, allowed] +NOTICE: in object access: superuser finished namespace search (subId=0x0) [report on violation, allowed] NOTICE: in object access: superuser attempting alter (subId=0x0) [explicit without auxiliary object] NOTICE: in object access: superuser finished alter (subId=0x0) [explicit without auxiliary object] NOTICE: in process utility: superuser finished ALTER TABLE @@ -96,6 +100,8 @@ ALTER TABLE test_oat_schema.test_oat_tab FORCE ROW LEVEL SECURITY; NOTICE: in process utility: superuser attempting ALTER TABLE NOTICE: in object access: superuser attempting namespace search (subId=0x0) [report on violation, allowed] NOTICE: in object access: superuser finished namespace search (subId=0x0) [report on violation, allowed] +NOTICE: in object access: superuser attempting namespace search (subId=0x0) [report on violation, allowed] +NOTICE: in object access: superuser finished namespace search (subId=0x0) [report on violation, allowed] NOTICE: in object access: superuser attempting alter (subId=0x0) [explicit without auxiliary object] NOTICE: in object access: superuser finished alter (subId=0x0) [explicit without auxiliary object] NOTICE: in process utility: superuser finished ALTER TABLE @@ -103,6 +109,8 @@ ALTER TABLE test_oat_schema.test_oat_tab NO FORCE ROW LEVEL SECURITY; NOTICE: in process utility: superuser attempting ALTER TABLE NOTICE: in object access: superuser attempting namespace search (subId=0x0) [report on violation, allowed] NOTICE: in object access: superuser finished namespace search (subId=0x0) [report on violation, allowed] +NOTICE: in object access: superuser attempting namespace search (subId=0x0) [report on violation, allowed] +NOTICE: in object access: superuser finished namespace search (subId=0x0) [report on violation, allowed] NOTICE: in object access: superuser attempting alter (subId=0x0) [explicit without auxiliary object] NOTICE: in object access: superuser finished alter (subId=0x0) [explicit without auxiliary object] NOTICE: in process utility: superuser finished ALTER TABLE @@ -111,6 +119,8 @@ ALTER TABLE test_oat_schema.test_oat_tab DISABLE RULE test_oat_notify; NOTICE: in process utility: superuser attempting ALTER TABLE NOTICE: in object access: superuser attempting namespace search (subId=0x0) [report on violation, allowed] NOTICE: in object access: superuser finished namespace search (subId=0x0) [report on violation, allowed] +NOTICE: in object access: superuser attempting namespace search (subId=0x0) [report on violation, allowed] +NOTICE: in object access: superuser finished namespace search (subId=0x0) [report on violation, allowed] NOTICE: in object access: superuser attempting alter (subId=0x0) [explicit without auxiliary object] NOTICE: in object access: superuser finished alter (subId=0x0) [explicit without auxiliary object] NOTICE: in object access: superuser attempting alter (subId=0x0) [explicit without auxiliary object] @@ -120,6 +130,8 @@ ALTER TABLE test_oat_schema.test_oat_tab ENABLE RULE test_oat_notify; NOTICE: in process utility: superuser attempting ALTER TABLE NOTICE: in object access: superuser attempting namespace search (subId=0x0) [report on violation, allowed] NOTICE: in object access: superuser finished namespace search (subId=0x0) [report on violation, allowed] +NOTICE: in object access: superuser attempting namespace search (subId=0x0) [report on violation, allowed] +NOTICE: in object access: superuser finished namespace search (subId=0x0) [report on violation, allowed] NOTICE: in object access: superuser attempting alter (subId=0x0) [explicit without auxiliary object] NOTICE: in object access: superuser finished alter (subId=0x0) [explicit without auxiliary object] NOTICE: in object access: superuser attempting alter (subId=0x0) [explicit without auxiliary object] @@ -130,6 +142,8 @@ ALTER TABLE test_oat_schema.test_oat_tab DISABLE TRIGGER test_oat_trigger; NOTICE: in process utility: superuser attempting ALTER TABLE NOTICE: in object access: superuser attempting namespace search (subId=0x0) [report on violation, allowed] NOTICE: in object access: superuser finished namespace search (subId=0x0) [report on violation, allowed] +NOTICE: in object access: superuser attempting namespace search (subId=0x0) [report on violation, allowed] +NOTICE: in object access: superuser finished namespace search (subId=0x0) [report on violation, allowed] NOTICE: in object access: superuser attempting alter (subId=0x0) [explicit without auxiliary object] NOTICE: in object access: superuser finished alter (subId=0x0) [explicit without auxiliary object] NOTICE: in object access: superuser attempting alter (subId=0x0) [explicit without auxiliary object] @@ -139,6 +153,8 @@ ALTER TABLE test_oat_schema.test_oat_tab ENABLE TRIGGER test_oat_trigger; NOTICE: in process utility: superuser attempting ALTER TABLE NOTICE: in object access: superuser attempting namespace search (subId=0x0) [report on violation, allowed] NOTICE: in object access: superuser finished namespace search (subId=0x0) [report on violation, allowed] +NOTICE: in object access: superuser attempting namespace search (subId=0x0) [report on violation, allowed] +NOTICE: in object access: superuser finished namespace search (subId=0x0) [report on violation, allowed] NOTICE: in object access: superuser attempting alter (subId=0x0) [explicit without auxiliary object] NOTICE: in object access: superuser finished alter (subId=0x0) [explicit without auxiliary object] NOTICE: in object access: superuser attempting alter (subId=0x0) [explicit without auxiliary object] diff --git a/src/test/regress/expected/reloptions.out b/src/test/regress/expected/reloptions.out index 9de19b4e3f..67f0a828f9 100644 --- a/src/test/regress/expected/reloptions.out +++ b/src/test/regress/expected/reloptions.out @@ -175,7 +175,7 @@ SELECT reloptions FROM pg_class WHERE oid = :toast_oid; -- Fail on non-existent options in toast namespace CREATE TABLE reloptions_test2 (i int) WITH (toast.not_existing_option = 42); -ERROR: unrecognized parameter "not_existing_option" +ERROR: unrecognized parameter "toast.not_existing_option" -- Mix TOAST & heap DROP TABLE reloptions_test; CREATE TABLE reloptions_test (s VARCHAR) WITH @@ -194,6 +194,17 @@ SELECT reloptions FROM pg_class WHERE oid = ( {autovacuum_vacuum_cost_delay=23} (1 row) +-- Can reset option that is not allowed, but for some reason is already set +UPDATE pg_class + SET reloptions = '{fillfactor=13,autovacuum_enabled=false,illegal_option=4}' + WHERE oid = 'reloptions_test'::regclass; +ALTER TABLE reloptions_test RESET (illegal_option); +SELECT reloptions FROM pg_class WHERE oid = 'reloptions_test'::regclass; + reloptions +------------------------------------------ + {fillfactor=13,autovacuum_enabled=false} +(1 row) + -- -- CREATE INDEX, ALTER INDEX for btrees -- diff --git a/src/test/regress/sql/reloptions.sql b/src/test/regress/sql/reloptions.sql index 24fbe0b478..9b404295c0 100644 --- a/src/test/regress/sql/reloptions.sql +++ b/src/test/regress/sql/reloptions.sql @@ -112,6 +112,13 @@ SELECT reloptions FROM pg_class WHERE oid = 'reloptions_test'::regclass; SELECT reloptions FROM pg_class WHERE oid = ( SELECT reltoastrelid FROM pg_class WHERE oid = 'reloptions_test'::regclass); +-- Can reset option that is not allowed, but for some reason is already set +UPDATE pg_class + SET reloptions = '{fillfactor=13,autovacuum_enabled=false,illegal_option=4}' + WHERE oid = 'reloptions_test'::regclass; +ALTER TABLE reloptions_test RESET (illegal_option); +SELECT reloptions FROM pg_class WHERE oid = 'reloptions_test'::regclass; + -- -- CREATE INDEX, ALTER INDEX for btrees -- -- 2.39.2 [application/pgp-signature] signature.asc (488B, ../../2726217.uZKlY2gecq@thinkpad-pgpro/3-signature.asc) download ^ permalink raw reply [nested|flat] 2+ messages in thread
end of thread, other threads:[~2024-06-16 14:07 UTC | newest] Thread overview: 2+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2020-12-03 15:19 [PATCH 3/3] Extended statistics on expressions Tomas Vondra <[email protected]> 2024-06-16 14:07 Re: [PATCH] New [relation] option engine Nikolay Shaplov <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox